[Piggidragon]

Large all-reduces under -sm tensor now go through the copy engines instead of NCCL, on two GPUs without peer access. Prompt processing gets 15–17% faster, decode is unchanged, and the result is at least as precise as NCCL's.

Why

Without peer access, NCCL moves a reduction through host memory with an SM on each GPU. That costs twice:

• A lost SM: the stream-k MMQ tiles launch exactly one block per SM. With one SM taken by NCCL, the last block has to wait for a second wave. In an nsys trace on the RTX 3060, matrix multiplications that overlapped an NCCL kernel took 1.4–1.8× as long (median, by grid size). NCCL_MAX_NCHANNELS=1 did not help: one SM is already enough to cause the second wave.
• One direction at a time: NCCL does not use both directions of a link at once. The 3060 sits on PCIe 3.0 x4: 3.3 GB/s each way alone, 2.8 + 2.8 GB/s with both directions in use. NCCL reached about 2 GB/s per AllReduce (2.6 MB in 1.29 ms).

Change

ggml/src/ggml-cuda/allreduce-ce.cu, about 300 lines:

• Send: each GPU converts its F32 part to BF16 (the same wire precision as NCCL's large-tensor path) and copies it to pinned memory in 512 KiB chunks, on its own copy stream.
• Receive: on a second copy stream, each GPU copies the peer's chunks back as soon as they arrive. Both directions of both links are busy, and no SM is.
• Sum: a small kernel adds the two BF16 values in F32, in the same order on both GPUs, so both get bit-identical results. This rounds once less than NCCL, which rounds the BF16 sum to BF16 again.
• Ordering: the GPUs are ordered with events only, and the host never waits. Two slots alternate, so the next piece can be converted while the previous one is still being transferred.
• Memory: the buffers (16 MiB VRAM, 8 MiB pinned memory per GPU) are allocated with the first large reduction, per context that has one. An MTP draft context processes the prompt batches too, so with MTP it is 32 MiB of VRAM per GPU in total. If the allocation fails, NCCL takes over.

With NCCL compiled in, two GPUs and no peer access, reductions are now routed like this:

Size	Path
below 512 KiB	internal pipeline (#14)
from 512 KiB	copy engines (this PR)
unsupported by both (not F32, not contiguous)	NCCL

With peer access everything stays on NCCL.

Environment variables:

• GGML_CUDA_AR_CE=0: NCCL for the large reductions.
• GGML_CUDA_AR_CE_MIN_BYTES: the threshold. Used with 1 below to send every reduction through the copy engines in the tests.

docs/multi-gpu.md now describes the three paths and their variables, and adds --attn-split (#15) to the argument table.

Results

Qwen3.8-27B UD-Q4_K_S, RTX 4070 (PCIe 4.0 x16) + RTX 3060 (PCIe 3.0 x4), -sm tensor -ts 0.625/0.375 -as 1/3, q5_1 cache, native FA, -ub 512, t/s:

test	NCCL (GGML_CUDA_AR_CE=0)	copy engines	change
pp512	710.5	831.5	+17.0%
pp2048	700.5	817.8	+16.7%
pp512 @ d16384	638.7	733.2	+14.8%
pp2048 @ d16384	638.2	732.1	+14.7%
pp3	97.5	97.5	same
tg32	36.9	36.9	same

llama-server with the production configuration (262144 tokens of q5_1 cache in 2 slots, MTP):

prompt	prefill NCCL	prefill copy engines	peak VRAM (4070 / 3060)
10k tokens	637 t/s	733 t/s	11730 / 11694 → 11748 / 11718 MiB
95k tokens	511 t/s	570 t/s	same

Precision

llama-perplexity --kl-divergence, wikitext, 40 chunks of 512, against F32 reductions everywhere (GGML_CUDA_ALLREDUCE=internal GGML_CUDA_AR_BF16_THRESHOLD=0):

	mean KLD	maximum KLD	same top token
NCCL (BF16 sum)	0.00262 ± 0.00022	1.88	97.53%
copy engines (F32 sum of BF16 values)	0.00243 ± 0.00012	1.07	97.81%

Tests

• test-llama-archs: run with the defaults, and with GGML_CUDA_AR_CE_MIN_BYTES=1 (every reduction through the copy engines, including the -as 1/0 configs with empty slices). 321 OK Meta rows each. The only failure is maion-coder MoE in "Mixed batch", which also fails on a single GPU.
• Qwen3.5-0.8B: the same greedy output with all reductions through the copy engines, with NCCL only, and with the defaults. The log confirms that the copy-engine buffers are allocated.

Review

A review (reviewer-xhigh) found no bug in the reduction itself:

• Hazards: every RAW/WAR hazard was traced across the 6 streams and both devices, and holds.
• Stress test: a harness with CUDA graphs and no host syncs, comparing both GPUs bitwise and against an exact CPU emulation, showed 0 mismatches, including inf/NaN/-0 and empty slices.
• Sanitizer and trace: compute-sanitizer memcheck was clean. An nsys trace shows both link directions busy.

Four smaller points are fixed in the second commit:

• MUSA build: the file did not compile for MUSA, because cudaHostAlloc is missing there. It now has the same guard and stubs as allreduce.cu.
• Docs, peer access: the docs said GGML_CUDA_AR_NCCL_MIN_BYTES also enables the copy engines with peer access. They stay off there, as intended.
• Memory, MTP: an MTP draft allocates its own buffers, see Memory above.
• Stale reference: a header comment referred to lanes that this PR does not have.

The copy engines are now also set up when the internal pipeline is not, which needs Volta or newer.

🤖 Generated with Claude Code