[rcfa]

Overview

Three Vulkan passes size the shared scratch (prealloc_x) from the current context:

• the flash-attention K/V dequant/contiguize copy: K + V in f16
• the fused QSA top-k of the masked per-cell path (ggml_vk_topk_qsa): n_kv x rows F32
• the top-k of the QSA block selection on the selected-key path (ggml_vk_topk), which is master's default on
  Vulkan since vulkan : selected-key flash attention for QSA block selection #119

All three set the scratch to the exact need, and that need grows with every prompt ubatch. So the scratch was freed and
allocated again as the context grew. With Qwen3.8-Flash-Next and 174k cells in the cache, it grew to 172, 173, 174 and
176 MiB during one 4096-token prompt on master. On the masked path it was 341, 342, ... 348 MiB, once per ubatch.

This PR sizes all three with one helper, ggml_vk_scratch_size():

• A need of 64 MiB or more is rounded up to 64 MiB, so the scratch is reallocated about once per 64 MiB of growth.
• A need below 64 MiB keeps its exact size. Ordinary models and ordinary top-k calls hold no extra memory.
• The rounded size never exceeds max_buffer_size, and never falls below the need.

The fused QSA top-k now also binds only the scratch it uses (scratch_size), not the whole shared buffer.

Measurements

Device:     Ryzen AI 9 HX PRO 370 / Radeon 890M (gfx1150, Strix Point - NOT Strix Halo) / Minisforum N5 PRO, BIOS 1.05
Memory:     96 GB DDR5-5600 (2 x 48 GB SO-DIMM, ECC)
Power:      BIOS defaults; amdgpu power_dpm_force_performance_level=auto; cpufreq governor powersave (amd-pstate)
BIOS:       UMA split 48 GB
Kernel:     7.1.5+kali-amd64, amdgpu params: amdgpu.gttsize=73728 ttm.pages_limit=8650752
Backend:    Vulkan RADV, Mesa 26.1.6 (libvulkan1 1.4.357)
Build:      -DGGML_VULKAN=ON -DLLAMA_CURL=OFF -DCMAKE_BUILD_TYPE=Release -DCMAKE_BUILD_RPATH_USE_ORIGIN=ON (gcc 16.2)
Baseline:   4e0d9177597bb47e2dae3bd7b00835f0a1f3b35a (master, with #115 and #119), built and run in the same session
Change:     542a402641c3d209873d949c05fedfb995284ae9
Model:      unsloth/Qwen3.8-Flash-Next-GGUF, UD-IQ4_XS

Update, master 6aa81b8 (2026-10-11): the Flash-Next numbers below were measured on master 4e0d917. On
6aa81b8 the 174k-resident probe no longer grows the scratch with or without this PR: since #117 the block selection
scores only the tracked conversation's own blocks. What the PR still does on this master is in the comments:

• On RADV, master copies K/V into the scratch for a quantised cache, or for an f16 cache with more than one KV head.
• For Qwen2.5-1.5B at a 98k-token depth, this PR cuts the scratch's reallocations of 64 MiB or more from 232 to 14,
  at +41 MiB peak.
• The speed does not change measurably.

Device memory near a full context.

• Setup: llama-server -c 262144 -np 4 -kvu, MTP n=1, GGML_VK_MEMORY_LOGGER=1.
• One 58k-token conversation is saved and restored into slots 1-3, so 174k cells are resident.
• Slot 0 then reads a 4096-token prompt and generates 32 tokens.

The table counts the allocations and frees during that probe, with the logger's own device total:

build	alloc/free events	of those >= 64 MiB	scratch	device total before -> peak	probe prompt speed
master	46	24	+172M, then -172M +173M, -173M +174M, -174M +176M	72.59 -> 72.59 GiB	109 t/s
this PR	40	18	+192M once	72.59 -> 72.59 GiB	111 t/s

The remaining large events are a compute buffer growing with the graph (809 -> 811 -> ... -> 824 MiB). That buffer belongs to the graph allocator's reserve, which this PR does not change. On the masked path (#115, measured on 307c50d), the same rounding took the scratch from every ubatch to once.

A small model keeps its exact sizes. Qwen2.5-1.5B-Instruct Q4_K_M with a q8_0 cache, llama-bench -fa 1 -ctk q8_0 -ctv q8_0 -p 2048 -n 0 -d 8192, GGML_VK_MEMORY_LOGGER=1:

• With this PR, the device allocations and the peak (1402.9 MiB) are identical to master.
• An earlier version of this PR rounded unconditionally and held 51 MiB more at peak.

Correctness:

• The change only rounds up allocation sizes and narrows one descriptor range. The four repository corpora go through llama-server (1 slot, no MTP), each as is (570-876 tokens) and repeated to 12.7k-53.4k tokens. Compared on prompt tokens, the complete next-token distribution after prefill, and 64 greedy steps with top-20 log-probabilities, the output is bit-identical to master on all eight prompts.
• test-backend-ops -b Vulkan0: TOP_K 579/579, ARGSORT 98/98, TOPK_QSA 4/4, FLASH_ATTN_EXT 5344/5344, QSA_PREFILL 26/26, QSA_DECODE 92/92, QSA_PREFILL_MASKLESS 6/6, QSA_DECODE_MASKLESS 25/25.

Requirements

• I have read and agree with the contributing guidelines
• This change is Strix Halo specific, or justified by measurements on Strix Halo: it concerns the fork's Vulkan
  flash-attention copy and QSA top-k passes. Our measurements come from Strix Point (gfx1150), not Strix Halo.
• AI usage disclosure: AGENT-AUTHORED. Claude Opus 5.5 found the churn with the memory logger, wrote the change and
  this description, and ran every measurement. Every commit carries Assisted-by: Claude Opus 5.5.
• What was NOT verified:
  • Strix Halo (gfx1151) hardware.
  • The clamp to max_buffer_size: on our device the limit is far above these sizes, so it never applies.
  • Other models with a context-sized scratch of 64 MiB or more.

[dzannotti]

Tests pass on Strix Halo (RADV, master + this PR): TOPK_QSA 4/4, FLASH_ATTN_EXT 5344/5344. Three issues with the rounding:

• No clamp. At ggml-vulkan.cpp:9737 and :12969, a need that fits exactly can round past max_buffer_size when the limit isn't a multiple of 64 MiB, and ggml_vk_create_buffer then throws. Something like std::max(need, std::min(padded, device->max_buffer_size)).
• The top-k descriptor binds the whole buffer (:12982, prealloc_x->size). After rounding, that can exceed maxStorageBufferRange when the exact size wouldn't. Bind scratch_size instead.
• Ungated. The FA dequant copy is used by every Vulkan user, so small models now hold up to 64 MiB extra. Suggest rounding only once the need is at least 64 MiB.

Comment generated via automated review of clanker.

[rcfa]

Thanks, all three are addressed in cf62cc8d7. The branch is rebased onto master 4e0d917, so it no longer carries #115.

• Clamp and gate: both sites now use one helper, ggml_vk_scratch_size(). A need below 64 MiB keeps its exact size. From 64 MiB, the need is rounded up to 64 MiB, but never past max_buffer_size, and never below the need itself.
• Descriptor: the top-k binds scratch_size, not the whole shared buffer.

Re-measuring on master turned up one more thing. Since #119, Flash-Next on Vulkan takes the selected-key path, and there the scratch grows in neither of these two passes. It grows in ggml_vk_topk, the block selection's top-k: 172, 173, 174 and 176 MiB during one 4096-token prompt next to 174k resident cells. So on master the PR changed nothing for this model. The new commit 542a40264 sizes that scratch with the same helper. Its passes bind dbl_buf_size subranges, so the larger buffer is never bound.

Checked on gfx1150 (RADV):

• Ops: TOP_K 579/579, ARGSORT 98/98, TOPK_QSA 4/4, FLASH_ATTN_EXT 5344/5344, QSA_PREFILL(_MASKLESS) and QSA_DECODE(_MASKLESS) all pass.
• Flash-Next near a full context: the scratch is allocated once at 192 MiB instead of four times. Events of 64 MiB or more drop from 24 to 18; the rest is the compute buffer's growth in the graph allocator. The device total is unchanged.
• Small model: Qwen2.5-1.5B with a q8_0 cache, llama-bench -fa 1 -ctk q8_0 -ctv q8_0 -p 2048 -d 8192. Device allocations and peak (1402.9 MiB) are now identical to master. The previous version held a 64 MiB copy, +51 MiB at peak.
• Corpora: bit-identical to master on all eight prompts.

The description is updated.

Assisted-by: Claude Opus 5.5

[dzannotti]

@rcfa ready to merge after a last rebase

[rcfa]

Rebased onto master 6aa81b8 (head 04c6faa). There was one conflict: #159 now reserves the V^T region in the same flash-attention scratch (need = K + V + V^T), so the rounding now applies to that need. Every region is still bound with its exact size.

Checked on gfx1150 (RADV):

• Ops: TOP_K 579/579, ARGSORT 98/98, TOPK_QSA 4/4, FLASH_ATTN_EXT 5359/5359, QSA_PREFILL 26/26, QSA_DECODE 92/92, QSA_PREFILL_MASKLESS 13/13, QSA_DECODE_MASKLESS 33/33.
• Corpora: bit-identical to master on all eight prompts.

One thing to know before merging. On this master, Flash-Next's scratch no longer grows in our memory probe (174k resident cells, a 4096-token prompt). It peaks at 6 MiB on both builds. The compute-buffer churn is gone too, probably through #164. So the probe is identical with and without this PR, and the Flash-Next numbers in the description, measured on 4e0d917, no longer reproduce. I added a note to the description. The change still sizes the three scratches as described, but we have not measured another model where they reach 64 MiB on this master. Merge or close it as you see fit.

Assisted-by: Claude Opus 5.5

[rcfa]

What this PR still does on master 6aa81b8, measured on gfx1150 (RADV).

On devices without coopmat2 (RADV here), master copies K and V into the shared scratch for every prompt ubatch in two cases: when the cache is quantised, and when it is f16 with more than one KV head. In the second case the cache view is head-interleaved, so fa_kv_contig makes it contiguous, and since #159 the V^T region is added as well. The scratch is sized to the exact need, which grows with the context. Once the need passes 64 MiB, master frees and reallocates the scratch on every ubatch. This PR rounds it up in 64 MiB steps.

Test: Qwen2.5-1.5B-Instruct Q4_K_M with its default f16 cache (2 KV heads), llama-bench -fa 1 -p 2048 -n 0 -d 98304, GGML_VK_MEMORY_LOGGER=1:

	master	this PR
device allocations and frees of 64 MiB or more	232 (the scratch grows 65 -> 147 MiB, one step per ubatch)	14 (128, then 192 MiB)
peak device total	4137 MiB	4178 MiB
pp2048 at d98304, 6 samples each (master, PR, PR, master)	356.3 ± 1.1 t/s	356.9 ± 0.8 t/s

For this model the scratch needs about 1.5 KiB per cell, so it passes 64 MiB at about 43k cells. A model with more KV heads gets there sooner. The PR removes the reallocation churn at the cost of up to 64 MiB more scratch. On our machine it does not change the speed measurably.

The other two sites:

• Fused QSA top-k: it runs only on the masked path, which Vulkan no longer takes by default now that it has the selected-key kernels (vulkan : selected-key flash attention for QSA block selection #119).
• Generic top-k: this is the QSA block selection. Since qsa : enable the incremental QSA cache on Vulkan #117 it scores only the tracked conversation's own blocks. That is why our earlier probe, a short conversation next to 174k resident cells, no longer grows it. We are checking whether a single long Flash-Next conversation still takes it past 64 MiB, and will add the result here.

Assisted-by: Claude Opus 5.5

[rcfa]

Follow-up on the generic top-k: with one long Flash-Next conversation, the scratch still grows on master. Test: llama-bench -p 2048 -n 0 -d 98304 (a single sequence of about 100k cells) with GGML_VK_MEMORY_LOGGER=1.

• master: the scratch is reallocated about 37 times as it grows from 64 to 100 MiB, at about 1 KiB per cell. We did not instrument this run; that growth rate matches the block-selection top-k we traced on 4e0d917.
• this PR: two allocations, 64 MiB and then 128 MiB.
• Compute buffers: the 160 compute-buffer events in that run are identical on both builds.
• Speed: pp2048 at d98304, run in the order master, PR, PR, master: 176.7 vs 177.5 t/s (2 samples each). There is no measurable change.

So the PR applies to Flash-Next too, once a single conversation passes about 64k tokens. As with the f16-cache case above, it removes allocation churn, not time.

Assisted-by: Claude Opus 5.5