ggml-cuda : round growing pool requests up to a power of two

[michal-zurkowski]

Overview

ggml_cuda_pool_leg::alloc reuses a cached block only when it is >= size, so a smaller cached block can serve neither this request nor any larger later one, and nothing shrinks the pool in normal operation. A caller whose request grows every step therefore misses the cache and strands its predecessor once per step. qsa-prefill.cu is such a caller: it sizes packed K/V scratch from the KV cache, which gains one ubatch per prefill step (8 MiB per 8192-token ubatch).

This PR rounds the request up to a power of two when the pool's largest block sits just below it, so the series reuses one block per octave instead. One file, +21/-1. Nothing is freed.

Mechanism

Not freeing is the point: pool scratch is returned to the pool during a CUDA graph capture while its pointer is already baked into a captured kernel node (ggml-cuda.cu:5244, :5502), and replay never re-enters the allocator, so releasing a cached block could invalidate a pointer a later replay still uses. The gate matters too: rounding unconditionally also inflates large batch-sized scratch, which at ubatch 16384 cancelled the entire saving. f = 2 is the optimum rather than the obvious choice, since a series growing to M costs about f*M live plus M/(f-1) stranded and f + 1/(f-1) is minimised at f = 2. The OOM retry falls back to the unrounded size, so it can never fail where it would previously have succeeded.

Measurements

Kernel:  7.2.3-1-cachyos-server
ROCm:    10.2.0 (develop-3636-58eba734dd20+pm4), gfx1151
Power:   STAPM 132 W, PPT fast 140 W, PPT slow 140 W, APU 100 W (ryzenadj -i)
Build:   -DGGML_HIP=ON -DAMDGPU_TARGETS=gfx1151 -DCMAKE_BUILD_TYPE=Release -DGGML_CUDA_GRAPHS=ON
         -DGGML_HIP_NO_VMM=ON -DGGML_HIP_MMQ_MFMA=ON -DGGML_NATIVE=ON -DGGML_OPENMP=ON
Model:   unsloth/Qwen3.8-Flash-Next-GGUF, PLE-BF16-Q4_K_XL

Both binaries are master plus a temporary patch that widens QSA block indices to 32 bits, which is what lets the prefill reach 512Ki; that patch is not part of this PR and is identical in both. They differ only by this commit: separate source and build directories, prefixes and RPATHs, same session, no runtime switch.

Memory

GTT growth during one prefill, MiB. llama-server, native 262144 context for the 256Ki rows, 524288 with YaRN x2 for the 512Ki rows.

depth	ubatch	unpatched	this PR	saved
256Ki	4096	12,774	2,344	82%
256Ki	8192	7,072	2,556	64%
256Ki	12288	4,022	2,508	38%
256Ki	16384	4,146	2,974	28%
512Ki	4096	15,582 a	4,403	72%
512Ki	8192	13,072 a	4,604	65%
512Ki	12288	11,312 a	5,049	55%
512Ki	16384	7,504 a	5,022	33%

a the unpatched run never finished: it was stopped by the probe's own guard when free system memory fell below 2048 MiB, between 0.58 and 0.75 of the prompt. Those figures are lower bounds, so the saving is understated. Every PR run completed the full 504,281-token prefill, with 12.8 to 19.1 GB of GTT still free. GTT and system RAM are one pool on this part, so the ceiling is real.

Unpatched cost falls as ubatch grows, because a larger ubatch means fewer, larger growth steps. The PR column does not move, because its footprint is set by the number of power-of-two size classes, not by how many steps were taken to reach them.

Memory, multi-turn

The same effect with the cache grown by conversation turns instead of one long prefill, at ubatch 16384 and native context, cache_prompt on. Turn sizes are generated once and replayed against both builds, so the two runs see an identical sequence. Measured prefix reuse was 97.5% and 98.4%.

turn size	turns	depth	unpatched	this PR	saved
fixed 3000 tok	80	240,267	12,792	2,476	81%
random 256-4096 tok	123	240,466	12,236	2,446	80%

A turn smaller than one ubatch makes the turn, not the ubatch, the growth step. At ubatch 16384 the unpatched build goes from 4,146 MiB over 16 steps to 12,792 MiB over 80, while the PR stays at 2,476.

Throughput

Prompt processing, tok/s, 256Ki prefill. The 512Ki rows have no unpatched figure because no unpatched run finished.

depth	ubatch	unpatched	this PR
256Ki	4096	800.7	808.5
256Ki	8192	871.9	874.5
256Ki	12288	861.3	863.2
256Ki	16384	896.1	900.0

Correctness

test-backend-ops, full suite, both builds: 29695/29695, rc=0 each.

Complete final logits after prefill, compared with cmp between the two builds: byte-identical on all eight prompts, token ids identical. Four corpora (tests/corpus/correctness-{prose,code,structured,numeric}.txt) at native length, and each repeated deterministically to 210,000 chars, giving prompt depths of 40,347 / 72,811 / 85,508 / 170,815 tokens.

Requirements

• I have read and agree with the contributing guidelines
• Strix Halo specific / justified by measurements on Strix Halo: the leg pool is what runs here (GGML_HIP_NO_VMM=ON, gfx1151), the only caller with a growing request is this fork's QSA prefill, and GTT and system RAM being one pool is what turns stranding into a failed run.
• AI usage disclosure: AGENT-AUTHORED. An agent (Claude Opus 5, via Claude Code) wrote the patch, built both trees, ran the measurements and drafted this text. It also produced several wrong versions first: one that released superseded blocks with cudaFree, which is unsafe against CUDA graph replay; an env-var gate that was only test scaffolding; an unconditional rounding that inflated batch-sized scratch and cancelled its own saving at ubatch 16384; and a first round of measurements taken against a master that was 86 commits stale. Each of those was caught by the repository owner or by review, and redone. The owner set the objective, chose the base, and directed the corrections.
• The CUDA graph replay hazard that rules out freeing pooled blocks was reasoned from the code, not reproduced; the claim here is only that this version cannot hit it, because it frees nothing.