[Piggidragon]

Adds --allreduce host|p2p for -sm tensor on two CUDA GPUs:

• host: the paths of cuda : reduce small tensors through the internal AllReduce when NCCL has no peer access #14 (small reductions) and cuda : reduce large tensors through the copy engines when NCCL has no peer access #16 (large reductions) through host memory.
• p2p: new paths with peer access.

Without the flag, NCCL does every reduction, as upstream. Before, #14 and #16 were always on and switched by environment variables. After merging, a setup that used them needs --allreduce host (or p2p), or it falls back to NCCL.

The README's fork section is rewritten as one list (what, benefit, how to enable), with an empty table for upstream vs this fork. The server's RAM prompt cache changes are the exception that needs no flag, and the README says so.

Why

On an RTX 4070 + RTX 3060 (PCIe 3.0 x4) with BAR1 P2P, before this PR:

• Small reductions (decode): with peer access, every reduction went to NCCL. NCCL over P2P is slower than the internal pipeline through host memory: tg32 dropped from 36.9 to 34.2 t/s as soon as peer access was there.
• Large reductions with --split-overlap: NCCL's kernels hold an SM on each GPU, which slows down the other ubatch's compute. Prompt processing dropped from 1180 to 865 t/s.
• Large reductions in general: the 3060's x4 link limits them whether or not they go through host memory.

Change

--allreduce p2p

• Small reductions (below 512 KiB): allreduce-p2p.cu. One kernel per GPU, with 8 blocks.
  • Each block writes its slice of the own contribution (BF16, as the internal pipeline does) into the other GPU's receive buffer. It then writes its token into that GPU's flag, after __threadfence_system.
  • It polls only its own flag, in its own memory, and adds f32(bf16(own)) + f32(bf16(peer)). Both GPUs get the same bits.
  • Two slots alternate, without host synchronization. A GPU writes slot k%2 of call k only after its kernel for call k-1 saw the other GPU's arrival for call k-1. The other GPU signals that arrival after its kernel for call k-2 has read the slot (stream order).
  • Tokens are compared in unsigned arithmetic and wrap at 2^32.
• Large reductions: the copy-engine AllReduce of cuda : reduce large tensors through the copy engines when NCCL has no peer access #16 gets a peer mode. Each GPU copies its BF16 contribution straight into the other GPU's buffer with cudaMemcpyPeerAsync, with no pinned memory.
• Peer access: both paths enable it themselves. Without peer access (or on HIP/MUSA) p2p falls back to host with a warning.

The flag

• Plumbing: llama_context_params.allreduce_type (enum llama_allreduce_type) is passed to the backends through a new reg proc, ggml_backend_comm_set_mode. It is called before a context creates its meta backends.
• Tools: --allreduce in common, and -ar in llama-bench.
• Environment variables:
  • Removed: GGML_CUDA_AR_CE, GGML_CUDA_AR_CE_P2P and GGML_CUDA_AR_P2P_SMALL.
  • Kept for tuning: the two size thresholds, GGML_CUDA_AR_NCCL_MIN_BYTES and GGML_CUDA_AR_CE_MIN_BYTES.
  • GGML_CUDA_ALLREDUCE still overrides the flag.
• Docs: recipe 7 in docs/multi-gpu.md describes the flag.

Results

Qwen3.8-27B UD-Q4_K_S, -sm tensor -ts 0.625/0.375 -as 1/3, q5_1, native FA, -ub 256 --split-overlap, patched libcuda with BAR1 P2P, t/s:

test	default (NCCL over P2P)	host	p2p
pp2048	866	1178	1181
pp3	88.8	97.4	99.5
tg32	34.2	36.9	37.6

Earlier run at 16k depth (same configuration), host vs p2p: pp2048 1046 vs 1067, pp3 77.6 vs 79.2, tg32 35.77 vs 36.48.

In an nsys trace of decode, a small reduction takes 10 µs on the 3060 (median), against 17 µs for the internal pipeline and 41 µs for NCCL without P2P.

Tests

• test-llama-archs: the new configs Meta -ar host and Meta -ar p2p -so pass on every architecture, with peer access. No failures in the run.

  • The configs lower the copy-engine threshold to 4 KiB, so that decode takes the small path and the ubatches go through the copy engines; the test models' reductions are otherwise all below 512 KiB.

• Peer access against none: llama-perplexity --kl-divergence (6 × 2048) gives mean KLD 0 and same top token 100%. The paths use the same arithmetic, so the output is bit-identical.

• Review, round 2 (reviewer-high): two low findings, both fixed:

  • the README claimed no fork change was active without a flag, which is wrong for the server's RAM prompt cache;
  • the copy-engine path was not covered by the new test configs.

• Review, round 1 (reviewer-xhigh) found three issues, all fixed:

  • the token comparison overflowed at 2^31;
  • P2P was used without an opt-in;
  • a threshold gap left some reductions between the two paths.

  Also checked in that review:

  • memory ordering, slot reuse with lanes and an MTP context, and event hazards;
  • compute-sanitizer memcheck clean;
  • server output byte-identical with and without peer access.

🤖 Generated with Claude Code