[Piggidragon]

Stacked on #16 (copy-engine AllReduce). This PR is the last commit; once #16 is merged, the base changes to main.

With --split-overlap, the AllReduce of one ubatch under -sm tensor runs while the next ubatch computes. Prompt processing gets 29–39% faster, the output is identical, and decode is unchanged.

Why

Every layer under -sm tensor ends with an AllReduce, and the GPUs wait for it. On an RTX 3060 at PCIe 3.0 x4, that was 42% of the 3060's prompt-processing time, never overlapped with compute.

How

Consecutive ubatches of at least 64 tokens alternate between two schedulers ("lanes"). Each lane has its own meta backend, and so its own streams, AllReduce context and compute buffers.

• Ordering between the lanes: ggml_backend_meta_share_lanes() links the two meta backends. When the graph is rebuilt, each subgraph gets the list of state tensors it reads or writes: KV cache and recurrent state, i.e. tensors in buffers that are neither weights nor compute buffers, identified by the root of their views. On each device, a subgraph then waits by event for the subgraphs of the other lane that last touched one of those tensors, from any of its graphs and at any index. The AllReduce after a subgraph is not waited for, so it overlaps with the other lane's compute. Subgraphs without state (FFN) do not wait at all.
• Small ubatches: smaller ubatches and draft contexts (MTP) stay on lane 0. Decode is therefore unaffected. Lanes are used only with all layers, the KV cache and the experts on the devices, and on devices with events.
• Shared buffers: the lanes share the weight and KV buffers. The meta buffer maps the views of a graph in a rotating pair of containers, and a rebuild clears the previous one. So a lane that becomes active builds and allocates its graph again instead of reusing it. Without that, test-llama-archs failed its mixed-batch check (NMSE 1e-2) and gpt2 hit an illegal memory access. With it, every check matches plain Meta exactly. It costs one graph build per prompt ubatch, a few ms against ~250 ms of compute.
• Memory: each lane reserves the compute buffers of a full ubatch, so half the --ubatch-size takes the same memory as before. The memory breakdown counts both lanes. Each lane captures its own CUDA graphs, about 50 MiB more at run time.
• Plumbing: llama_context_params.split_overlap, --split-overlap / LLAMA_ARG_SPLIT_OVERLAP, llama-bench -so, a Meta -so config in test-llama-archs, and recipe 8 in docs/multi-gpu.md.

Results

Qwen3.8-27B UD-Q4_K_S, RTX 4070 (PCIe 4.0 x16) + RTX 3060 (PCIe 3.0 x4), -sm tensor -ts 0.625/0.375 -as 1/3, q5_1 cache, native FA, copy-engine AllReduce (#16).

llama-bench, t/s:

test	-ub 512	-ub 256 -so 1	change
pp2048	817.1	1180.3	+44%
pp2048 @ d16384	731.4	1047.0	+43%
pp3	97.5	97.3	same
tg32	36.9	36.9	same

Each lane's ubatch size in the prototype (pp2048, t/s): 128 → 1119, 256 → 1179, 384 → 1066, 512 → 965. Smaller pieces pipeline better, until the fixed cost per ubatch wins.

llama-server with the production configuration (262144 tokens of q5_1 cache in 2 slots, MTP, -ub 256):

scenario	without overlap	--split-overlap	output
10k-token prompt	774 t/s	947 t/s	identical
continuation (10382 tokens from the cache)			identical
two 5k prompts in parallel	612 / 576 t/s	702 / 709 t/s	identical
93k-token prompt	598 t/s	734 t/s	identical
VRAM peak (4070 / 3060)	11510 / 11486 MiB	11750 / 11724 MiB

Against #16 at -ub 512 (733 and 570 t/s for the 10k and 95k prompts), the overlap adds 29%.

Review

A review (reviewer-xhigh) found 4 issues; all are fixed in d7e2304:

• High: the waits were wrong for some architectures. Waiting only for the same subgraph index of the other lane is not enough.
  • Gemma4 layers that share another layer's KV read it many subgraphs later. The other lane could overwrite the SWA cells in between (with swa_full = false).
  • The LFM2 short-conv state is read one or two subgraphs before it is written.
  • The reviewer reproduced wrong logits for both. Qwen3.5/3.8 was not affected.
  • Fix: the waits now follow the actual state tensors, as described above. With the reviewer's fixtures (Gemma4: 30 layers, 25 sharing KV, ub 512, 3000 tokens; LFM2: n_embd 4096, ub 2048, 6144 tokens), Meta and Meta -so now give bitwise identical logits in every run.
• Medium: the test could not catch a missing wait. The tiny test models never overlap, so Meta -so passed even with the waits removed. The fixtures above did reproduce the race and are what verified the fix; the test-llama-archs config only covers the plumbing.
• Low: alternation reset every call. The lane alternation started over in every decode(), so one large ubatch per call always ran on lane 1. It now carries over between calls.
• Low: devices without events aborted. ggml_backend_meta_share_lanes() now returns false there, and the lanes stay off with a warning.

With the fix, prompt processing is faster than before (1138 → 1180 t/s, at 16k 984 → 1047): subgraphs without state no longer wait. The table above shows the new numbers. #16's review fixes are merged into this branch.

Tests

• test-llama-archs: the new Meta -so config passes every check that plain Meta passes: 163 rows, 308 OK cells each. The ubatches of 64 tokens alternate between the lanes there. The only failure is maion-coder MoE in "Mixed batch", the same as Meta and a single GPU.
• test-arg-parser: passes.

🤖 Generated with Claude Code