llama, cuda : opt-in AllReduce paths with --allreduce host|p2p

README.md

@@ -1,55 +1,45 @@
 # What this fork adds
 
-Branch `master` mirrors [ggml-org/llama.cpp](https://github.com/ggml-org/llama.cpp). Branch `main` is upstream plus the two features below. The rest of this file is the upstream README.
-
-## Quantized-native CUDA FlashAttention
-
-The CUDA MMA FlashAttention kernel reads a quantized K/V cache in place, instead of casting the visible attention window to F16 first.
-
-- It is opt-in with `--flash-attn-native-quants` (`-fanq 1` in `llama-bench`). Without the flag everything runs as upstream.
-- Cache types: `q4_0`, `q4_1`, `q5_0`, `q5_1` and `q8_0`, the same type for K and V. Head sizes 64, 128, 256 and 512, at every query batch size, on Turing (`sm_75`) and newer.
-- Decode and the small batches of speculative decoding take tiles of 8 Q heads, so each K/V row is read once per KV head instead of once per Q head.
-- It is faster at long context. Qwen3.8-27B at 16k context:
-  - token generation +6-11% on an RTX 3060 or 4070;
-  - 2-token speculative steps +14-31%.
-
-  On both GPUs at 64k context the gain is up to +39%. On D=128 models, 4-token batches run up to twice as fast.
-- The F16 copy of the window no longer takes compute buffer: 1.6 GiB less for Qwen3.8-27B at 262k context.
-- `q4_0` and `q8_0` are always compiled. `q4_1`, `q5_0` and `q5_1` are compiled with their pair in `GGML_CUDA_FA_QUANTS` (for example `-DGGML_CUDA_FA_QUANTS="q5_1-q5_1"`) or with `all`.
-- Results differ from the F16 path in rounding only.
-
-Details, validation and all measurements: [docs/quantized-native-flash-attention.md](docs/quantized-native-flash-attention.md).
-
-## Prompt cache hierarchy: slots, RAM, disk
-
-`--cache-disk PATH` adds a disk level below the RAM prompt cache (`--cache-ram`).
-
-- Prompts that RAM evicts are written to disk in the background, and are loaded back when a later request shares their prefix.
-- On exit and before sleeping, the server also writes its slots and RAM entries. Conversations survive restarts and model swaps.
-
-How prompts are stored:
-
-- A prompt is stored as KV blocks of `--cache-block-size` tokens (default 2048), keyed by a hash of their prefix.
-- Prompts that share a prefix share its blocks, in RAM and on disk.
-- A growing conversation writes only its new blocks. In one 25k-token conversation with Qwen3.8-27B, the next turn wrote 0.16 GB instead of 2.3 GB.
-
-The new libllama functions `llama_state_seq_get_data_range()` and `llama_state_seq_set_data_range()` provide the blocks. All model types are supported:
-
-- Attention, sliding-window, recurrent and hybrid models work.
-- Models that roll back only to context checkpoints (hybrid, recurrent, sliding-window) are stored at their newest checkpoint.
-- Caches without state ranges store whole states.
-
-Size limit and sharing:
-
-- `--cache-disk-size` caps the directory (default 32 GiB). The least recently used prompt is removed first, and 5% of the file system always stays free.
-- File names carry a fingerprint of the model files and the KV settings, so several models can share one directory.
-
-Two more server changes come with the disk level:
-
-- Slot selection: if a cached prompt shares at least 256 more tokens with a request than the most similar slot does, the server restores the cached prompt instead of processing those tokens again. One case is a slot that holds another conversation with the same system prompt.
-- SIGINT and SIGTERM are handled on their own thread. A sleeping server now stops cleanly and writes its cache.
-
-Details: [tools/server/README.md](tools/server/README.md#prompt-cache-on-disk).
+Branch `master` mirrors [ggml-org/llama.cpp](https://github.com/ggml-org/llama.cpp). Branch `main` is upstream plus the features below. Each one is off until its flag enables it, except the changes to the server's RAM prompt cache, which are always on (see the prompt cache item). The rest of this file is the upstream README.
+
+## Features
+
+- **Quantized-native FlashAttention (CUDA).** The FlashAttention kernel reads a quantized K/V cache in place instead of first copying the attention window to F16.
+  - Benefit: faster generation at long context (Qwen3.8-27B: +6-11% at 16k, up to +39% at 64k), and no F16 copy in the compute buffer (1.6 GiB less at 262k context).
+  - Enable: `--flash-attn-native-quants` (`-fanq 1` in llama-bench) with a quantized cache, the same type for K and V (`-ctk q5_1 -ctv q5_1`). `q4_1`, `q5_0` and `q5_1` need `-DGGML_CUDA_FA_QUANTS=<pair>` or `all` at build time. [Details](docs/quantized-native-flash-attention.md)
+- **Prompt cache on disk.** A disk level below the RAM prompt cache. Prompts are stored as KV blocks keyed by their prefix, so conversations with a shared prefix share blocks, and a growing conversation writes only its new blocks.
+  - Benefit: conversations survive RAM eviction, restarts and model swaps, and resume without processing the prompt again. The next turn of a 25k-token conversation writes 0.16 GB instead of 2.3 GB.
+  - Enable: `--cache-disk PATH` (server), sized with `--cache-disk-size` (MiB). [Details](tools/server/README.md#prompt-cache-on-disk)
+  - Always on, with the RAM prompt cache (`--cache-ram`, on by default):
+    - The RAM cache stores prompts as KV blocks of `--cache-block-size` tokens (default 2048).
+    - A cached prompt keeps only its newest 3 context checkpoints.
+    - The server restores a cached prompt into a slot when that prompt shares at least 256 more tokens with the request than the slot does.
+    - SIGINT and SIGTERM are handled on their own thread.
+- **Attention split** (`--split-mode tensor`). The attention heads and their KV cache get their own split across the GPUs, separate from `--tensor-split`.
+  - Benefit: with GPUs of different speed, the faster one takes more of the weights while the other one holds more of the KV cache.
+  - Enable: `--attn-split 1,3` (`-as`), next to e.g. `--tensor-split 0.625,0.375`. [Details](docs/multi-gpu.md)
+- **AllReduce by size** (`--split-mode tensor`, two CUDA GPUs). Small reductions go through a kernel with a handshake instead of NCCL, and large ones through the copy engines. With `p2p`, the GPUs write directly into each other's memory.
+  - Benefit: lower latency per decode step, and prompt processing that keeps every SM free for compute.
+  - Enable: `--allreduce host` (no peer access needed) or `--allreduce p2p` (needs peer access between the GPUs; falls back to `host` without it). [Details](docs/multi-gpu.md#7-two-gpus-over-pcie---allreduce)
+- **Split overlap** (`--split-mode tensor`). Large ubatches alternate between two schedulers, so the AllReduce of one ubatch overlaps with the compute of the next.
+  - Benefit: faster prompt processing when the link between the GPUs is slow (Qwen3.8-27B on two PCIe GPUs: 733 -> 947 t/s at 10k context).
+  - Enable: `--split-overlap`, with half the usual `--ubatch-size` (each scheduler has its own compute buffers). Best together with `--allreduce host` or `p2p`. [Details](docs/multi-gpu.md#8-overlapping-the-allreduce-with-compute---split-overlap)
+
+## Upstream vs this fork
+
+Setup: RTX 4070 (12 GB, PCIe 4.0 x16) + RTX 3060 (12 GB, PCIe 3.0 x4), Qwen3.8-27B UD-Q4_K_S with its MTP head (2 draft tokens), `llama-server` with 2 slots of 131k context, q5_1 K/V cache.
+
+- Upstream: `-sm tensor -fa on -ctk q5_1 -ctv q5_1`, the best split that fits.
+- This fork: additionally `--flash-attn-native-quants -ts 0.625,0.375 --attn-split 1,3 --allreduce p2p --split-overlap -ub 256 --cache-disk ...`.
+
+| | upstream | this fork |
+|---|--:|--:|
+| prompt processing, 10k context (t/s) | | |
+| prompt processing, 95k context (t/s) | | |
+| generation, 10k context (t/s) | | |
+| generation, 95k context (t/s) | | |
+| resume a 25k-token conversation after a restart (s) | | |
+| VRAM after a 95k request, 4070 / 3060 (MiB) | | |
 
 ---
 

common/arg.cpp

@@ -1779,6 +1779,27 @@ common_params_context common_params_parser_init(common_params & params, llama_ex
             params.split_overlap = value;
         }
     ).set_env("LLAMA_ARG_SPLIT_OVERLAP"));
+    add_opt(common_arg(
+        {"--allreduce"}, "{default,host,p2p}",
+        "with --split-mode tensor on two CUDA GPUs: how the GPUs exchange their partial results (docs/multi-gpu.md)\n"
+        "- default: NCCL\n"
+        "- host: by size, through host memory: small reductions through a kernel with a handshake in pinned memory, "
+        "large ones through the copy engines (no SM busy, both directions of the link at once)\n"
+        "- p2p: by size, with CUDA peer access between the GPUs: small reductions through a kernel that writes into "
+        "the other GPU, large ones through the copy engines from GPU to GPU; host without peer access. Some "
+        "platforms report peer access but corrupt the transfers (IOMMU)",
+        [](common_params & params, const std::string & value) {
+            if (value == "default") {
+                params.allreduce_type = LLAMA_ALLREDUCE_TYPE_DEFAULT;
+            } else if (value == "host") {
+                params.allreduce_type = LLAMA_ALLREDUCE_TYPE_HOST;
+            } else if (value == "p2p") {
+                params.allreduce_type = LLAMA_ALLREDUCE_TYPE_P2P;
+            } else {
+                throw std::invalid_argument("invalid value for --allreduce: " + value);
+            }
+        }
+    ).set_env("LLAMA_ARG_ALLREDUCE"));
     add_opt(common_arg(
         {"--cache-idle-slots"},
         {"--no-cache-idle-slots"},

common/common.cpp

@@ -1734,6 +1734,7 @@ struct llama_context_params common_context_params_to_llama(const common_params &
     cparams.pooling_type      = params.pooling_type;
     cparams.attention_type    = params.attention_type;
     cparams.flash_attn_type   = params.flash_attn_type;
+    cparams.allreduce_type    = params.allreduce_type;
     cparams.cb_eval           = params.cb_eval;
     cparams.cb_eval_user_data = params.cb_eval_user_data;
     cparams.offload_kqv       = !params.no_kv_offload;

common/common.h

@@ -505,6 +505,7 @@ struct common_params {
     enum llama_pooling_type      pooling_type      = LLAMA_POOLING_TYPE_UNSPECIFIED; // pooling type for embeddings
     enum llama_attention_type    attention_type    = LLAMA_ATTENTION_TYPE_UNSPECIFIED; // attention type for embeddings
     enum llama_flash_attn_type   flash_attn_type   = LLAMA_FLASH_ATTN_TYPE_AUTO; // whether to use Flash Attention
+    enum llama_allreduce_type    allreduce_type    = LLAMA_ALLREDUCE_TYPE_DEFAULT; // split mode tensor: AllReduce paths
 
     bool flash_attn_native_quants = false; // read a quantized K/V cache in place instead of casting it to F16
     bool split_overlap            = false; // split mode tensor: overlap the AllReduce of a ubatch with the compute of the next

docs/multi-gpu.md

@@ -113,28 +113,38 @@ GGML_CUDA_P2P=1 llama-cli -m model.gguf -sm tensor
 
 P2P requires driver support (usually restricted to workstation/datacenter GPUs) and **may cause crashes or corrupted outputs on some motherboards or BIOS configurations** (e.g. when IOMMU is enabled). If you see instability after enabling it, unset the variable.
 
-### 7. Two GPUs without peer access
+`--allreduce p2p` (recipe 7) uses peer access for the AllReduce of `--split-mode tensor` without this variable: it enables peer access between the two GPUs itself, with the same caveat.
 
-With `--split-mode tensor`, two GPUs that cannot access each other's memory (the usual case for consumer cards) exchange their partial results through host memory. With NCCL compiled in, each reduction takes one of three paths:
+### 7. Two GPUs over PCIe (`--allreduce`)
 
-| Reduction size | Path | Why |
-|---|---|---|
-| below 512 KiB (decode, small batches) | internal pipeline, a single kernel with a handshake through pinned memory | lower latency than NCCL |
-| from 512 KiB (prompt processing) | copy engines: each GPU copies its part to pinned memory in chunks and the other one copies them back as they arrive | no SM is busy, both directions of a link are used at once |
-| anything the two above do not handle | NCCL | |
+With `--split-mode tensor`, two GPUs exchange their partial results after every layer. By default NCCL does all of it. With NCCL compiled in, `--allreduce` picks a path for each reduction by its size instead:
+
+| `--allreduce` | Reduction size | Path | Why |
+|---|---|---|---|
+| `host` | below 512 KiB (decode, small batches) | internal pipeline: a single kernel with a handshake through pinned memory | lower latency than NCCL |
+| `host` | from 512 KiB (prompt processing) | copy engines: each GPU copies its part to pinned memory in chunks, the other copies them back as they arrive | no SM is busy, both directions of a link are used at once |
+| `p2p` | below 512 KiB | peer-to-peer kernel: each GPU writes its part and a flag into the memory of the other and polls only its own | lower latency than NCCL and the internal pipeline |
+| `p2p` | from 512 KiB | copy engines from GPU to GPU | as above, without the copy through host memory |
+| `default` | all | NCCL | |
 
-With peer access, NCCL handles every reduction.
+`host` works without peer access, which consumer cards usually lack. `p2p` needs peer access between the GPUs (on consumer cards only with a patched driver) and falls back to `host` without it. Some platforms report peer access but corrupt or hang the transfers (IOMMU, see recipe 6). Over NVLink, keep `default`: NCCL uses the link natively.
 
-Environment variables:
+```bash
+llama-server -m model.gguf -sm tensor --allreduce host --split-overlap -ub 256
+```
+
+The paths matter most with `--split-overlap` (recipe 8): NCCL keeps an SM on each GPU busy, which slows down the compute of the other ubatch. Over PCIe, the bandwidth of the slower link limits a large reduction with or without peer access, so peer access mostly lowers the latency of the small ones. Measured with an RTX 4070 (PCIe 4.0 x16) and an RTX 3060 (PCIe 3.0 x4), Qwen3.8-27B, `--split-overlap -ub 256`, `default`/`host`/`p2p`: prompt processing 866/1178/1181 t/s, 3-token batches 88.8/97.4/99.5 t/s, generation 34.2/36.9/37.6 t/s (`default`: NCCL over peer access).
+
+For tuning, two environment variables move the thresholds:
 
 | Variable | Effect |
 |---|---|
-| `GGML_CUDA_ALLREDUCE=nccl` | NCCL for every reduction |
-| `GGML_CUDA_AR_NCCL_MIN_BYTES=N` | size from which the internal pipeline is no longer used, and the default of `GGML_CUDA_AR_CE_MIN_BYTES` (0: NCCL for every reduction). Setting it also enables the internal pipeline with peer access; the copy engines stay off there |
-| `GGML_CUDA_AR_CE=0` | NCCL instead of the copy engines for large reductions |
-| `GGML_CUDA_AR_CE_MIN_BYTES=N` | size from which the copy engines are used |
+| `GGML_CUDA_AR_NCCL_MIN_BYTES=N` | size below which the small path is used (default 512 KiB, the most the peer-to-peer kernel handles; 0: NCCL for every reduction) |
+| `GGML_CUDA_AR_CE_MIN_BYTES=N` | size from which the copy engines are used (default: where the small path ends) |
+
+`GGML_CUDA_ALLREDUCE=nccl|internal|none` overrides `--allreduce`.
 
-The copy engines need 16 MiB of VRAM and 8 MiB of pinned memory per GPU and per context that has large reductions. They are allocated with the first large reduction. An MTP draft context processes the prompt batches as well, so it has its own buffers: 32 MiB of VRAM per GPU in total.
+The copy engines need 16 MiB of VRAM per GPU, plus 8 MiB of pinned memory without peer access, per context that has large reductions. They are allocated with the first large reduction. An MTP draft context processes the prompt batches as well, so it has its own buffers: 32 MiB of VRAM per GPU in total. The peer-to-peer kernel needs 512 KiB of VRAM per GPU and context.
 
 ### 8. Overlapping the AllReduce with compute (`--split-overlap`)
 
@@ -154,7 +164,7 @@ Notes:
 
 - Each scheduler reserves the compute buffers for a full ubatch. With half the `--ubatch-size`, the memory stays the same, and two ubatches are in flight at once. On the GPUs below, 256 per scheduler was faster than 128, 384 or 512.
 - Each scheduler captures its own CUDA graphs, which takes some more VRAM at run time (about 50 MiB here).
-- The overlap works best with the copy-engine AllReduce of recipe 7: NCCL kernels hold an SM each, and the compute of the other ubatch then runs slower.
+- The overlap works best with `--allreduce host` or `p2p` (recipe 7): NCCL kernels hold an SM each, and the compute of the other ubatch then runs slower.
 
 Qwen3.8-27B on an RTX 4070 (PCIe 4.0 x16) + RTX 3060 (PCIe 3.0 x4), `-ts 0.625,0.375 -as 1,3`, 262144 tokens of q5_1 cache in 2 slots, with MTP, prompt processing in t/s:
 

ggml/include/ggml-backend.h

@@ -210,6 +210,9 @@ extern "C" {
     typedef void * (*ggml_backend_comm_init_t)(ggml_backend_t * backends, size_t n_backends);
     typedef void   (*ggml_backend_comm_free_t)(void * comm_ctx);
     typedef bool   (*ggml_backend_comm_allreduce_tensor_t)(void * comm_ctx, struct ggml_tensor ** tensors);
+    // the AllReduce paths of the comm contexts created after the call: "default" or a backend-specific mode (CUDA: "host",
+    // "p2p"), returns false for an unknown mode
+    typedef bool   (*ggml_backend_comm_set_mode_t)(const char * mode);
 
     // Split buffer type for tensor parallelism (old)
     typedef ggml_backend_buffer_type_t   (*ggml_backend_split_buffer_type_t)(int main_device, const float * tensor_split);