Repository navigation
hip/gfx1151: fix mmap load hang, enable QSA get_rows rows %32, long-context session support - #166
Draft
gaetan-puleo wants to merge 398 commits into
Draft
gaetan-puleo wants to merge 398 commits into
gaetan-puleo wants to merge 398 commits into
Conversation
The gather rejected anything but f16 K/V, so a DSv4 run with -ctk q8_0 -ctv q8_0 took the
dense fallback for every batch 2..63 decode - the whole small-batch path, silently off. Two
community test sets were run that way before anyone noticed, which is what prompted this.
The restriction was incidental rather than essential. The gather RELOCATES rows; it never
reads a value out of one. Addressing K as raw 4-byte words instead of f16 elements makes the
same shader serve any type whose row is a whole number of words, which every block-quantised
type here satisfies at head size 512. The union shader needed no change at all - it only ever
touched top_k indices.
Zeroing an unused row now writes zero BYTES. That decodes to zero for the block-quantised
types as well as for f16, because a zero scale zeroes the block; those rows are -inf in the
mask regardless, so zeroing only keeps a garbage dot product out of the softmax as a NaN.
The type gate asks ggml_vk_fa_kv_native rather than just "is it quantised". While the compact
scratch is active the dequant/contiguize pass is disabled by construction, and the assert
guarding that combination aborts rather than falling back, so admitting a type flash-attention
has no native shader for would crash instead of degrade.
test-backend-ops perf, kv=11008 n_kv_raw=2304 n_top_k=512 ov=60, gather off vs on:
batch f16 dense -> gathered q8_0 dense -> gathered
2 2192 -> 645 us 3.40x 3917 -> 1110 us 3.53x
4 2203 -> 726 us 3.03x 3924 -> 1257 us 3.12x
8 2213 -> 910 us 2.43x 3936 -> 1567 us 2.51x
16 2219 -> 2232 us declined 3930 -> 3952 us declined
Two things worth stating because they contradict what I expected going in.
Compaction does NOT pay more on quantised K/V. The reasoning was that a gathered set is
contiguous and would dodge the strided-read and channel-aliasing taxes that hurt quantised
reads more; the measured ratios differ by 3 to 4%, inside noise. It pays the same proportion
from a worse starting point.
That worse starting point is the real result: q8_0 attention is about 1.78x SLOWER than f16
here, dense (3917 vs 2192) and gathered (1110 vs 645) alike. Halving the bytes does not pay
for the dequant work in the inner loop at this head size. So this change makes q8_0 K/V
survivable for DSv4 rather than advisable - f16 is still the faster cache by a wide margin,
and that inverts the usual q8_0-KV-buys-speed rule of thumb for this model.
13318 FLASH_ATTN_EXT cases pass, including 8 new q8_0/q4_0 top-k cases at the shapes a DSv4
decode actually hits. test_flash_attn_ext_top_k takes a type_K parameter for them.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Assisted-by: Claude (Opus 5)
Relocating quantised rows verbatim leaves flash attention to decode them in its inner loop,
which it does once per query block that reads a row rather than once per row. Measured, that
costs 0.15 us per KV row attended - 0.1481 / 0.1510 / 0.1512 us/row at batch 8 / 4 / 2 and
0.1565 dense, so a constant across a 3.6x span of row counts and across both regimes. It is
what made q8_0 attention 1.78x slower than f16 while reading half the bytes.
The gather is the natural place to pay it instead: it already touches every selected row
exactly once. Decoding there makes the compact scratch f16, flash attention takes its f16
path, and the redundancy disappears. No extra pass - the pass already existed.
test-backend-ops perf, kv=11008 n_kv_raw=2304 n_top_k=512 ov=60, q8_0 K/V:
batch dense verbatim gather decoded gather f16 gather
2 3917 us 1110 us 630 us 640 us
4 3924 us 1257 us 715 us 727 us
8 3936 us 1567 us 891 us 908 us
q8_0 now edges f16 by 1.5 to 1.8% rather than trailing it by 78%: the FA does identical f16
work either way, and the gather's scattered read side moves half the bytes. Against the dense
fallback a q8_0 cache took before any of this work, batch 8 is 4.41x.
Flash attention picks its pipeline from what the SCRATCH holds, not from the source tensor, so
k_type_eff/v_type_eff now also key off the compact state - the same mechanism use_dequant_kv
already used, which is why that variable existed in this form.
q8_0 only, deliberately. It is the type both community test sets used and the one worth having;
every other quantised type keeps the verbatim path, which is correct for them and validated.
Generalising means one shader variant per type through dequant_funcs.glsl, which is only worth
the shader-permutation cost if this shape of win reproduces elsewhere.
Not covered: the per-token gather still relocates verbatim, so a batch where the union declines
but the per-token form fits keeps paying the inline decode. Same fix applies, and the same
measurement will say whether it is worth a second shader.
13318 FLASH_ATTN_EXT cases pass.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Assisted-by: Claude (Opus 5)
q4_0 turns out to want this more than q8_0 did. Its inline-decode penalty is the same shape - a constant per KV row attended, flat across batch and across dense vs gathered - but larger: 0.172 to 0.179 us/row against q8_0's 0.148 to 0.153. Unpacking nibbles costs more ALU than the halved bytes save, which is also why q4_0 was the SLOWEST of the three types before this. kv=11008 n_kv_raw=2304 n_top_k=512 ov=60, batch 2 / 4 / 8 type dense verbatim gather decoded gather f16 2192/2203/2213 - 640/726/902 q8_0 3917/3924/3936 1110/1257/1567 628/712/888 q4_0 4134/4141/4145 1189/1342/1671 626/711/886 All three converge, because after the gather they run the same f16 attention; what is left is the scattered-read side, where fewer bytes now wins. Against the dense fallback each type took before any of this work, batch 8 is 2.45x for f16, 4.43x for q8_0 and 4.68x for q4_0. The interesting part is what the first attempt got wrong. Generalising via dequantize4() from dequant_funcs.glsl looked obvious and passed on q8_0, then failed q4_0 with NMSE 1.17. That helper is permutation AGNOSTIC by design: mul_mat_vec only ever feeds it into a dot product, where any consistent element order gives the same sum, so for q4_0 it returns nibbles in packed order rather than element order. q8_0 passed only because its layout happens to be contiguous. Materialising a row into memory is precisely the use where the order matters, so each type's true mapping is now written out - for q4_0, byte j carries element j in its low nibble and element j+16 in its high one, exactly as dequant_q4_0.comp materialises it. That is also why this stops at two types rather than looping over the whole native list: every type needs its mapping stated and tested, an #error guards anything else reaching the shader, and q8_0 and q4_0 are the two actually used as a KV cache. q4_1/q5_0/q5_1/iq4_nl keep the verbatim gather, which is correct for them. 13318 FLASH_ATTN_EXT cases pass, including q4_0 and q8_0 top-k cases at the shapes a DSv4 decode hits, plus q4_0 perf cells alongside the q8_0 ones. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Assisted-by: Claude (Opus 5)
Generalising the gather from f16 elements to raw words (6b2cade) replaced [[unroll]] for (uint i = 0; i < HEAD_SIZE / LANES; ++i) with a loop bounded by the row_words push constant, which cannot be unrolled and pays a bounds check per iteration. Half as many, twice as wide accesses were still 6% slower on the smallest op the gather serves. Bisected over the 19 beta3 commits with the tip's test file overlaid at every point, so the instrument stayed fixed while beta3 grew its own perf cases. nb=1 at kv=8192: 54.58 us at the pre-beta3 base, flat through commit 16 (54.84), 58.24 at 6b2cade, 58.23 at the tip. Step 4*LANES instead. Each access stays contiguous across the lanes, so coalescing is unchanged, and an f16 row at head size 512 is 256 words == 4*LANES: one iteration, no loop overhead. The tail loop carries the quantised row sizes, which are not multiples of 4*LANES (q8_0 is 136 words, q4_0 is 72). nb=1 against the pre-beta3 base, f16 K/V: -0.07% at kv=8192, -2.83% at kv=35584, mean -1.29% across the six shapes, so the regression is gone rather than reduced. Against beta3 as merged: -4.8% to -6.2%. The nb>=2 gains are untouched, worst cell -0.51%. 13318 FLASH_ATTN_EXT cases pass. Worth noting the gather itself was never the cost at nb=1: with GGML_VK_FA_TOPK_GATHER=0 that cell is 249.84 us against 58.14 us with it on. Assisted-by: Claude Opus 5
The union gather decodes q8_0/q4_0 on the way in, but it exists only for n_batch > 1: a single token has nothing to deduplicate, so plain autoregressive decode takes the per-token gather, which relocated rows verbatim and left flash attention paying the same inline decode the union work measured at ~0.15 us per KV row attended. flash_attn_gather_dq.comp closes that: same row mapping as flash_attn_gather.comp, same per-type element mapping as flash_attn_gather_union_dq.comp (q4_0 materialised in element order, not packed order). The compact scratch becomes f16 and flash attention takes its f16 path, keyed off the compact state exactly as the union path already is. test-backend-ops perf, kv=11008 n_kv_raw=2304 n_top_k=512 ov=60, nb=1: type verbatim gather decoded gather q8_0 226.9 us 102.9 us q4_0 258.4 us 101.8 us f16 114.5 us 105.5 us (f16 never decodes; its two runs bound the harness spread) Both quantised types land on the f16 op time instead of trailing it 2.0-2.3x: after the gather the attention work is identical f16, and the gather's scattered read side moves fewer bytes. q8_0 and q4_0 only, as in the union: each type's element mapping must be stated and tested, an #error guards anything else reaching the shader, and every other type keeps the verbatim gather. nb=1 rows are added to the top-k perf grids so this regime stays measured. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Assisted-by: Claude (Opus 5)
ggml_vk_flash_attn_top_k declined any K/V but f16, so a DSv4 run with a quantised cache took dense FA for every prefill batch - O(kv) against the sparse path's O(n_kv_raw + n_top_k), a penalty that grows with depth. A 128GB community run measured it end to end: prefill -5.4/-22.2/-39.4/-52.9% against f16 at source depths 17k/33k/67k/134k with -ctk q8_0. The sparse shaders stay f16-only. A quantised cache is dequantised once per op into the prealloc_x f16 scratch by the dense path's own fused dequant+transpose pipeline, and the shaders read the scratch. K and V are the same tensor in this path, so one pass covers both at half the dense path's scratch footprint. Types without a dequant-transpose pipeline, oversized caches, and GGML_VK_FA_DEQUANT=0 keep the dense fallback. test-backend-ops perf at nb=1024, kv 5504/11008/19200/35584 (the community depths in compressed-K rows): q8_0 was 60.9 -> 427.6 ms dense (1.30x -> 8.27x of f16, growing with kv) and lands within +0.8% of f16's flat 46.9-51.7 ms with the scratch; the dequant pass itself costs under 1%. 36/36 FLASH_ATTN_EXT top-k cases pass, including 6 new quantised prefill cases (nb=64 ns=1/2, nb=128 split form). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Assisted-by: Claude (Opus 5)
The K tile is invariant across the head loop but coopMatLoad sits inside it, so the shader re-read the same 8 MatrixA fragments from shared memory once per head tile - 4x at N_HEAD=64. Each wave64 fragment load moves 1 KiB of LDS traffic because the 16x16 f16 fragment is replicated 4x across the subgroup. Same loads of the same data feeding the same coopMatMulAdd sequence, so the output is bit-identical; the fragments just live in registers (8 of them, 16 f16 per lane) instead of being re-fetched. gfx1151, test-backend-ops perf, kv 8704/33280/131584 x batch 1-15: 16-20% faster at every shape, no spills, no regression. Batch 1 is ordinary DSv4 decode and gets 12-18% of that. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Assisted-by: Claude (Opus 5)
…ecode CM shader Builds directly on pepuscz's PR #6 and issue #10. Their per-kernel table - 43.8 us scalar against 27.9 us small CM per 1k scanned rows at batch 5, and the same ratio at every depth - is what showed the indexer's cost is a per-tile constant rather than anything to do with tokens or bytes. Once the cost is per tile, the thing to minimise is the tile count, and that is what this changes. The finding is downstream of their work; only the choice of shader differs. The decode CM shader puts 16 HEADS in the coopmat N dimension and dispatches one workgroup per token, so it issues 4*n_batch tiles per 16 KV rows. That is the arithmetic minimum - (64 heads x n_batch tokens) / 16 columns - because 64 heads fill its 16 columns exactly, with no remainder at any batch. The small CM shader puts 16 TOKENS in N and pays a flat 64 tiles however small the batch is, so at batch 5 eleven of its sixteen columns are padding. Measured cost tracks tile count: at kv=131584 batch 1 costs 1.95 us per 1k scanned rows and batch 5 costs 26.28, i.e. 13.5x for 5x the work, and both shaders sit at 6.6-7.8 ns per tile. The decode CM shader body has no n_batch == 1 assumption - token is gl_WorkGroupID.y and it indexes q/w/mask/dst by it - so the old gate was an artefact of where it was written. gfx1151, kv=131584 (526k source tokens), us/run, shipped vs this: batch 2 2335.6 -> 419.0 (5.6x, was on the scalar path) batch 3 3492.3 -> 631.8 (5.5x, was on the scalar path) batch 4 3267.9 -> 840.8 (3.9x) batch 5 3465.6 -> 1119.4 (3.1x) <- DSpark n-max 4 verify shape batch 8 3704.9 -> 1764.1 (2.1x) batch 15 4467.9 -> 3231.6 (1.4x) Batch 16 and 32 are unchanged, which confirms the arms are isolated. PR #6's route is kept as the opt-out arm rather than deleted, and is promoted from opt-in to default-on so that one variable is enough to reach it: default decode CM for the whole 2-15 window ..._DECODE_CM_BATCH=0 small CM for 4-15, scalar for 2-3 (PR #6) ..._DECODE_CM_BATCH=0 SMALL_CM=0 scalar for 2-15 (pre-PR #6 baseline) Verified all three route as documented and pass 29/29. Keeping it is not just courtesy: the tile-count argument is hardware independent, but decode CM re-reads the K tile once per token and that part is bandwidth dependent, so the crossover need not sit in the same place on other devices, and these numbers are from one gfx1151 box. A (head, token) packing shader was also tried and is strictly worse - it reaches 4b tiles only when the batch divides 16, and it divides the workgroup count by the batch. Kept out of tree. Numerics: decode CM sums the 64 heads in tiles of 16 rather than one at a time, so f32 accumulation order differs from the small CM path. Cleared by KLD A/B on trunc10 (-c 8192, -b/-ub small so every indexer dispatch routes through the window, 8 chunks wikitext-2, same binary, only the env var differing): -ub 8 mean KLD 0.000000, max 6.0e-5, RMS dp 0.000%, same-top 99.994%, PPL 10507558.3156 identical in both arms -ub 5 mean KLD 0.000000, max 8.8e-4 but 99.9% 4.9e-5 (one tail event, no argmax flip), RMS dp 0.000%, same-top 100.000% Same class as PR #6's own measured numbers (max 5.5e-5, same-top 100.000%) and 45x tighter than the FA_WAVE32 change already shipped. Adds the eval coverage the 2-15 window never had (batches 4/8/15 at kv=256) and the perf grid these numbers came from. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Assisted-by: Claude (Opus 5)
Same defect as the decode CM shader, larger blast radius. `k_sh[wave]` is written
once above the head loop and never touched again, but `coopMatLoad` sits inside
both the head_base and head_local loops, so each wave re-reads the same 8 MatrixA
fragments N_HEAD times: 512 loads where 8 would do. Each wave64 fragment load
moves 1 KiB of shared-memory traffic because the 16x16 f16 fragment is replicated
4x across the subgroup.
Same loads of the same data feeding the same coopMatMulAdd order, so the output is
bit-identical; the fragments live in registers (8 of them, 16 f16 per lane) instead
of being re-fetched.
This source compiles to both the prefill pipeline and the small-batch one, so both
gain. gfx1151, test-backend-ops perf, us/run:
prefill (lightning_indexer_cm_f16)
kv 8704 nb 2048 27492.5 -> 18207.0 1.51x
kv 33280 nb 2048 104101.9 -> 68947.8 1.51x
kv 131584 nb 2048 421239.8 -> 281540.5 1.50x
kv 131584 nb 16 3456.2 -> 2329.2 1.48x
kv 131584 nb 32 6858.6 -> 4623.6 1.48x
small batch (lightning_indexer_cm_small_f16)
kv 131584 nb 5 3433.1 -> 2783.3 1.23x
kv 131584 nb 8 3682.3 -> 3105.1 1.19x
kv 131584 nb 15 4474.4 -> 4209.5 1.06x
1.48-1.54x on prefill at every depth and batch measured, flat. The small-batch
variant gains less, which says the 1-wave configuration is bound by its Q staging
and epilogue rather than by fragment traffic.
No regression at any shape, so the 8 held fragments (64 VGPRs per lane) neither
spill nor cost occupancy even in the prefill pipeline, which runs 512 threads at
62.7 KiB of shared memory and was already pinned to one workgroup per CU.
LIGHTNING_INDEXER 29/29.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Assisted-by: Claude (Opus 5)
…ation constant
The indexer pipelines hardcoded nh=64 (DeepSeek-V4's geometry) in supports_op and
as a shader constant, so any other head count fell to the CPU backend.
Qwen3.8-Flash-Next (qwen4_exp, released today) carries the same indexer at nh=4.
N_HEAD becomes specialization constant 1, one pipeline per supported head count
(LI_NH_VALUES = {4, 32, 64}), selected by q->ne[1] at dispatch. It must be a
specialization constant, not a push constant: the head loop's trip count has to
stay visible to the compiler - as a push constant the loop cannot unroll and
decode costs ~55% (measured, gfx1151, kv=131584, and why this commit is not the
simpler design).
nh=4 does not fill a 16-wide head tile, so the tail tile stages q as zeros
(relu(0)*w = 0, padded heads drop out of the sum) and the weight read clamps its
index. Both guards are gated on N_HEAD % TILE != 0, a specialization-constant
expression, so at nh=64 they fold away at pipeline compile and the codegen is
equivalent to the previous shader - measured, because relying on the compiler to
range-prove head < N_HEAD instead still cost ~45%:
kv=131584, us/run before after
decode b1 226.8 219.6
decode b5 1058.8 1007.4
decode b15 3174.0 2966.8
prefill b16 2329.2 2230.2
prefill b2048 281540.5 281169.1
No regression at nh=64; the small decode improvement is within a cautious
reading of run-to-run variance and is not claimed.
test-backend-ops LIGHTNING_INDEXER: 44/44, up from 29/29 - the suite's existing
nh=32 cases had been silently reporting "not supported" under the old gate and
now run against the CPU reference.
Head size stays pinned at 128: the coopmat tiling and f16vec4 staging both
assume it, and every known indexer model (DSv4, GLM-DSA, Qwen4-exp) uses 128.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Assisted-by: Claude (Opus 5)
The scalar64 shader kept its hardcoded 64-head loop when the pipelines went per-head-count; nh 4 and 32 would have run with the wrong trip count. Assisted-by: Claude Fable 5
The GEMM2 P load depends only on bc_chunk, but sat inside the hsv_tile loop, so all four fragments were re-read from shared memory once per tile (2 tiles at HSV=128, 4 at HSV=256). Load them once into a coopmat array before the loop. Psh is not written again until the next KV block, so the fragments stay valid across the barriers inside it. Measured on gfx1151 (RADV), Qwen3-Coder-30B-A3B, pp2048 f16 KV: d8192 +6.9%, d16384 +8.1%, d32768 +9.2%. VGPRs and LDS unchanged, 0 spilled. FLASH_ATTN_EXT suite 5105/5105. Assisted-by: Claude Fable 5
Psh held P as [kv][query]. The GEMM2 UseA load therefore had to request ColumnMajor, and RADV only attaches an alignment hint to the internal column-major case, which for UseA means the RowMajor request. The load was emitted as 16 separate 16-bit shared reads per fragment. Store P as [query][kv] instead and request RowMajor. The producer now writes four scalar components rather than one vec4; those go to disjoint bytes, so there is no read-modify-write and no partially written vec4. Also updates the host shared-memory estimator, which mirrored the old stride and would otherwise disagree with the shader. Perf-neutral on gfx1151 (within 1% at hd128 and hd256), but it shrinks Psh: LDS 16384 -> 15360 B and code size 13896 -> 13596 at hd128, VGPRs unchanged at 96 with 0 spilled. Kept because it is free and removes a scalar shared-memory access pattern. FLASH_ATTN_EXT suite 5105/5105. Assisted-by: Claude Fable 5
get_fa_tuning_params_coopmat1 took device->subgroup_size unconditionally, so on a 64-wide device the coopmat1 FA path always ran wave64. The sibling scalar path already does AMD-specific wave selection; the coopmat1 path never got the equivalent. Narrowing is free exactly when it does not add an iteration to the O-accumulation loop, which runs ceil((HSV/4) / threads_per_rowgroup) per row. On a 64-wide device the test reduces to hsv <= 128. Above it the narrow subgroup issues 1.5x to 1.8x the instructions for the same SIMD passes and hd256 measures 6 to 18 percent slower, so the rule declines. Gated behind GGML_VK_FA_WAVE32 (=1 rule, =2 forces the pin regardless of head size, diagnostic only). Off by default. Measured on gfx1151 (RADV), model-level pp2048, Qwen3-Coder-30B-A3B: d0 +2.5%, d8192 +8.4%, d16384 +10.1%, d32768 +11.3%. Op-level across head sizes, largest where wave64 wastes the most lanes: hsv=64 +12.3%, hsv=96 +7.4%, hsv=128 +6.5%, hsv=256 correctly declined. Full test-backend-ops suite 15884/15884 at =0, =1 and =2; the pin was confirmed to engage in-band via pipeline VGPR statistics rather than inferred from the env var being set. Assisted-by: Claude Fable 5
Split from the fork's flag-flip commit (f7d804ee7): GGML_VK_FA_WAVE32 now defaults to 1 (apply the HSV-based narrowing rule); =0 opts out and =2 keeps its diagnostic force meaning. The subgroup_size_control device guard is unchanged, so devices without a 32-wide subgroup mode are unaffected. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Assisted-by: Claude (Opus 5)
Select the matmul_id tile by expected per-expert token count (nei1*nei0/n_expert) instead of aggregate nei1, which always picked the widest tile and left most N-lanes empty at MoE prefill (~16 rows/expert vs BN=64). Gated on GGML_VK_MMID_SMALLN=1, default off pending cross-model validation. Was null (+0.7%) standalone pre-row-lists: smaller tiles meant more workgroups each re-paying the per-WG id scan. With the Stage 1 row-list prepass the scan is gone and the occupancy win materializes. Strix Halo (RADV gfx1151), Qwen3.6-35B-A3B UD-Q5_K_XL, clean window, pp512 4-way (rowlists x smalln): 914.7 / 937.9 / 1005.3 / 1063.6 t/s (combined +16.3% vs baseline). MUL_MAT_ID q5_K 2.33->4.43 TFLOPS (1.91x), q6_K 1.99->3.57 (1.79x) cumulative. Hot pipelines verified via probe: matmul_id_*_f32 (y_f32 path, quantize_y does not engage for these quants); smalln flips tile _m -> _s. 790/790 MUL_MAT_ID both rowlists configs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Assisted-by: Claude (Opus 5)
Two env-gated overrides of the mmid small-tile config (KHR coopmat branch only, scoped to the mul_mat_id quant pipelines; dense untouched): - GGML_VK_MMID_TILE16: BN/WN 32->16. NEGATIVE (-3.8% e2e): experts with n_e>16 split into two column tiles and re-stream their full weight matrix; A-traffic scales with sum(ceil(n_e/BN)), so BN must not drop below the mean per-expert n. Kept as a documented dead end. - GGML_VK_MMID_BM64: BM 32->64, BLOCK_SIZE 64->128 (two warps). +1.3% e2e on top of rowlists+smalln: same A-traffic, half the ir-tiles so half the per-expert B re-reads, larger WGs hide latency. Strix Halo, Qwen3.6-35B-A3B UD-Q5_K_XL pp512, clean window, all on top of RL+SN control 1065.4 +/- 3.3: BM64 1078.9 +/- 3.5, TILE16 1024.6, BM64+TILE16 1007.5. Cumulative vs pre-Stage-1 baseline: 914.7 -> 1078.9 (+17.9%). 790/790 MUL_MAT_ID for all configs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Assisted-by: Claude (Opus 5)
…ated) Extends the BM64 idea to the medium tile: GGML_VK_MMID_M128=1 raises the mmid m-tile to BM=128 / BLOCK_SIZE=4*subgroup (four warps), same A-traffic, half the ir-tiles and per-expert B re-reads. The medium tile is what the per-expert-n heuristic selects at n~64 (e.g. ub2048 on 256-expert/top-8 models, or ub1024 at 128 experts). Strix Halo, Qwen3.6-35B-A3B UD-Q5_K_XL pp2048, drain-verified window: ub2048 stack 1039.9 +/- 1.1 -> +M128 1091.6 +/- 2.6 (+5.0%); inert in the s-tile regime (ub1024 1137.2 vs 1147.9 ref). ub1024 remains the throughput sweet spot for this model. 790/790 MUL_MAT_ID. Also measured this session, NOT kept: caching counts+row lists across the three expert matmuls of a layer (they share one ids tensor) was correctness-clean but perf-null (1066.3 vs 1070.9 baseline) - the prepass dispatches and barriers are already free on this queue; reverted rather than carry the invalidation surface. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Assisted-by: Claude (Opus 5)
Force required subgroup size 32 on the KHR-coopmat mmid quant pipelines (pipeline_dequant_mul_mat_mat_id[*] only; dense untouched). Hypothesis: RDNA3.5 WMMA is wave32-native, so RADV may lower KHR_coopmat better at wave32 than at the reported default of 64. Mechanics: the cm1 path in mul_mm.comp derives its warp grid from the real subgroup (warp_i = gl_SubgroupID, tiw = gl_SubgroupInvocationID) while NUM_WARPS = BLOCK_SIZE/WARP (spec constants) sizes coopmat_stage[] and ballots_sh[] and warp_r/warp_c assume NUM_WARPS == (BM/WM)*(BN/WN). Forcing sg32 with WARP=64 would over-run those shared arrays and leave warp_c outside the tile, so the gate (a) sets WARP=32 in shadowed copies of the s/m/l mmid warptiles (composes after the BM64/M128 shadows) and (b) halves WM (or WN, keeping WM>=TM, WN>=TN) until the doubled subgroup count exactly tiles BM x BN again, asserting both invariants. BLOCK_SIZE is kept, so workgroup shape, load loops and shmem match the wave64 stack; per-lane accumulator footprint is also unchanged (half the lanes per subgroup, half the (WM/TM)*(WN/TN) fragments). The required size is passed via a scoped CREATE_MM redefinition adding a trailing required_subgroup_size arg (fp16-branch pattern), gated on subgroup_size_control covering 32 since ggml_vk_create_pipeline_func silently drops the required size otherwise. Correctness (test-backend-ops -o MUL_MAT_ID -b Vulkan0): 790/790 plain, 790/790 WAVE32=1, 790/790 WAVE32+SMALLN+BM64. Perf, Qwen3.6-35B-A3B-UD-Q5_K_XL, fa=1 b/ub=512 ctk/ctv=q8_0 pp512 r=3, atomic window, canary clean: stack (SMALLN+BM64), wave64 default: 1076.61 +/- 3.71 t/s stack + WAVE32: 1106.73 +/- 2.52 t/s (+2.8%) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Assisted-by: Claude (Opus 5)
Convert contiguous f32 activations (B) to f16 for quantized MUL_MAT_ID on KHR_coopmat devices and run the matmul_id_subgroup_<quant>_f16 kernels instead of the f32-B ones. Halves B bytes and buf_b shared memory. The _f16 SPIR-V already exists for every quant type; this adds a parallel env-gated pipeline array (pipeline_dequant_mul_mat_mat_id_f16b), extends the getter to return it for src1=F16 on non-coopmat2 devices, relaxes the src1-type assert, and forces the existing y_non_contig convert-to- prealloc_y plumbing (same as coopmat2). Default OFF, zero behavior change when unset. Measured on Radeon 8060S (RADV gfx1151), Qwen3.6-35B-A3B-UD-Q5_K_XL, -fa 1 -b 512 -ub 512 -ctk q8_0 -ctv q8_0 -p 512 -r 3, stacked on GGML_VK_MMID_SMALLN=1 GGML_VK_MMID_BM64=1: stack (f32 B, canary): pp512 1075.04 +/- 8.66 t/s stack + F16B: pp512 1100.66 +/- 2.37 t/s (+2.4%) test-backend-ops test -o MUL_MAT_ID -b Vulkan0: 790/790 with and without GGML_VK_MMID_F16B=1. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Assisted-by: Claude (Opus 5)
Assisted-by: Claude (Opus 5)
The MMQ J config is chosen from n, so a wave-partitioning error in one J tile only shows up when n sits on that tile's boundary. The neighbouring n picks a different J and passes, which hides it. The existing quantized cases stop at n=129 and the general MUL_MAT set jumps 64 -> 4096, so nothing lands on 256 or 512 and the whole class went untested. Sweep n over 255/256/257/511/512/513 for q8_0, q4_0, q4_K, q5_K and q6_K, in both MUL_MAT and MUL_MAT_ID. On an RDNA3.5 build that runs the J128 kernel with 16 wave32 waves over a 128-row tile these fail for every n that selects J128 and pass at n=513, which selects J112. A build predating that config passes the whole sweep. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> (cherry picked from commit b54cd8a) Assisted-by: Claude (Opus 5)
30d8bb0 raised mul_mat_id_param_count to 6 for the fused MUL epilogue and gave every mul_mat_id shader a binding 5 (FusedScale), including the coopmat2 one, which binds it purely for descriptor-layout parity. The coopmat2 pipeline creation block was left passing a literal 5, so two things go wrong there: - the pipelines are created with 5 descriptors while the shader declares 6 - PARAMCOUNT == mul_mat_id_param_count doubles as the "this is mul_mat_id" argument to ggml_vk_mul_mm_cm2_spec, so it went false and every coopmat2 mul_mat_id pipeline was specialized as a plain matmul, dropping the trailing spec constant Only reachable where device->coopmat2 is true. gfx1151 does not take that path and the v0.5 release predates the constant bump, so neither is affected. Not validated on hardware - no coopmat2 device here. The block does compile: built with the pinned shaderc (GL_NV_cooperative_matrix2 supported). The two OCP FP4 sites stay behind GL_EXT_float_e2m1, which that glslc does not support. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Assisted-by: Claude (Opus 5)
Split from the fork's flag-flip commit (f7d804ee7): GGML_VK_MMID_F16B, BM64, M128, WAVE32 and SMALLN now default on with =0 opt-out. The wave32 gate keeps its subgroup_size_control device guard, so devices without a 32-wide subgroup mode are unaffected. GGML_VK_MMID_TILE16 stays opt-in (documented negative on gfx1151). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Assisted-by: Claude (Opus 5)
RDNA3.x WMMA is wave32-native, so a wave64 subgroup issues each coopmat op as two halves. GGML_VK_MMID_WAVE32 already exploits this for mul_mat_id and leaves the dense pipelines at the driver default; this gives dense the same treatment, gated on measurement rather than a flag. BLOCK_SIZE is kept, so the subgroup count doubles and WM (or WN) halves until the warp grid tiles BM x BN again. Only the quantised tiles are retiled: on gfx1151 a standalone MUL_MAT microbench at both dense FFN shapes reads q6_K +5.2..+10.8%, q8_0 +5.4..+8.4%, q4_K +0.7..+9.1%, q4_0 -1.5..+1.8%, while f16 reads -6.7..+6.4% and bf16 ~0 - the float paths are bandwidth-bound on the weight stream, not issue-bound. The win tracks inline dequant instruction count (q6_K 3907 -> 3433 instructions, identical 192 VGPRs and 8 subgroups/SIMD). The required subgroup size is now the tile's own WARP for every dense coopmat pipeline. The cm1 shaders derive their warp grid from gl_SubgroupID and size shared arrays as NUM_WARPS = BLOCK_SIZE / WARP, so WARP and the real subgroup must agree; leaving that to the driver made the agreement incidental. Scoped to AMD coopmat1 on a wave64 default; other vendors keep the driver default. GGML_VK_DENSE_WAVE32=0 disables, =2 also retiles the float tiles. Qwen3-32B Q6_K_XL pp2048 +7.2% at ub256 / +3.9% at ub2048, Qwen3.8-27B +5.3% / +4.8%. PPL unchanged: 6.9496 +/- 0.24246 in both arms, all 20 per-chunk values identical, since the retile changes which warp owns an output sub-tile and not the K-reduction order within an element. Assisted-by: Claude Opus 5 (cherry picked from commit 448994e9637405610ced3e0ead02c7b6fa688314)
All default OFF, so the same binary A/Bs each change. GGML_VK_CONCAT_TRANSPOSE: delta-net does ggml_transpose() into a dim-0 ggml_concat(), which the generic concat reads fully de-coalesced. Route that shape through a 32x32 shared-memory tile transpose. CONCAT 11877 -> 957 us/op. GGML_VK_MMID_SCALE_EPILOGUE: apply the following MUL's per-(expert,token) broadcast scale as mul_mat_id writes out, removing a 134 MB write plus read back. Prefill only; the existing fusion is gated to mat-vec. Not implemented in the coopmat2 shader, so it is refused there. GGML_VK_FUSE_UNARY_MUL: silu(x)*y is two nodes in the delta-net path; run it as the existing swiglu split. 750 -> 443 us/op. GGML_VK_MMID_WG256: the RADV tuning gives the dense large tile 256 threads on a 128x128 tile but left the mul_mat_id variants at 128. Qwen3.6-35B-A3B UD-Q4_K_XL, gfx1151, pp2048 at ub2048: 1223.75 -> 1733.64 t/s. test-backend-ops CONCAT/MUL_MAT_ID/UNARY/MUL pass; generated text is unchanged with each flag on, and the disabled paths are bit-identical to before. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> (cherry picked from commit 6f7a49e) Assisted-by: Claude (Opus 5)
The fused epilogue derives the scale index from row_ids as [token * nei0 + expert_slot], which carries no 4th dimension, but the gate admitted any ne[3] as long as the scale and the matmul agreed. A tensor with ne[3] > 1 would read the wrong scale for every batch past the first and return quietly wrong results. test-backend-ops never generates such a case, so the suite passed throughout; found by reading the gate against the shader. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> (cherry picked from commit 016e906) Assisted-by: Claude (Opus 5)
The delta-net conv-state path transposes straight into a dim-0 concat, so the generic concat kernel walks src1 with a conv_channels * 4 byte stride. On qwen35 that is 40960 B, which is 160 * 256 B with 160 % 16 == 0, so every read lands on the same one of the 16 memory channels: 13.7 GB/s against 138.9 GB/s for the tiled path. The tiled-transpose route has been behind GGML_VK_CONCAT_TRANSPOSE=1 since it landed. Turn it on by default and keep GGML_VK_CONCAT_TRANSPOSE=0 as the opt-out. Qwen3.8-27B pp2048: +0.4% at ub 256, +4.7% at ub 1024, +7.2% at ub 2048. Assisted-by: Claude Opus 5
The scheduler's async input copies between graph splits land in the compute ctx on devices without a separate transfer queue, so the perf logger's fresh-ctx assert fired under partial offload (--n-cpu-moe). Assisted-by: Claude Fable 5
vulkan : vec4 store for int8 MMQ on RDNA3, and MoE router weights in the MUL_MAT_ID epilogue
server : add --spec-draft-max-slots to stop drafting under concurrency
qsa : enable the incremental QSA cache on Vulkan
…ontext session support Measured on Strix Halo (Ryzen AI MAX+ 395, gfx1151, ROCm 10) with Qwen3.8-Flash-Next-UD-Q4_K_XL, fork master 6aa81b8, GGML_HIP=ON, -two full-day stability runs (4 concurrent sessions) and one 994k-token single-slot session. - llama-model-loader: stage device-bound tensor copies coming from a file mapping through a pinned staging buffer. A hipMemcpy from file-backed (non-pinned) host memory never completes on this platform and rocr::core::BusyWaitSignal::WaitRelaxed spins at 100% CPU forever, hanging --load-mode mmap at 76 GiB uploaded. With the staging path mmap loads take ~27 s and process Pss_Anon drops from 34.9 GiB to 16 MiB (MemAvailable 8 -> 44 GiB). - getrows: dequantize per 32-value block for IQ4_NL / MXFP4 / Q4_0_ROCMFP4(+FAST) and relax the supports_op gate to ne[0] % 32, so lookup tables with head_dim % QK_K != 0 (per_layer_token_embd, 160-element rows) no longer fall into an oversized CPU buffer. - qsa-prefill/qsa-decode: raise the KV-cell ceiling from 262140/262144 to 1048576; beyond it the sole fallback was a GGML_ABORT, killing the server mid-prefill at ~262144 tokens (reached at 810 tok/s). - server: LLAMA_SERVER_ALLOW_CTX_BEYOND_TRAIN=1 lifts the slot cap at n_ctx_train when the admin knowingly runs a yarn-extended context. - ggml-cuda: GGML_CUDA_POOL_FREE_MIN_MB returns oversized scratch blocks to the driver instead of ratcheting the pool; measured -2.7 GiB of permanent GTT growth at ~1M tokens, d_gtt = 0 MB. Assisted-by: OpenCode (GLM)
- ggml-cuda: drop the dead !pool_pool_full() gate in pool free (both branches already freed the block); document that the knob applies to the legacy non-VMM pool (GGML_HIP_NO_VMM default ON). - llama-model-loader: gate the pinned staging buffer on use_mmap || lazy.any() so non-mmap loads never allocate it; guard the new device-copy branch with cur->buffer. - getrows/dequantize: cross-reference the %32 shape gate and the nibble-verbatim invariant of the 32-block wrappers. - tools/server/README.md: document LLAMA_SERVER_ALLOW_CTX_BEYOND_TRAIN, GGML_CUDA_POOL_FREE_MIN_MB, QSA_DIRECT, GGML_QSA_RANKSORT. Assisted-by: OpenCode (GLM)
Compiler flagged stage_mapping as unused: the device-copy branch tested mapped_stage_buf only. Use the flag (it implies the buffer is valid), keeping the buffer check available for safety. Assisted-by: OpenCode (GLM)
The QSA-family MXFP4 and ROCmFP4 quant paths are measured ROCm kernels; they keep the original QK_K get_rows kernel and the QK_K supports_op gate byte-for-byte. Only IQ4_NL (per_layer_token_embd, 160-element rows) needs the relaxed shape: single-type gate + the 32-grain kernel route, matching what the earlier PR text claimed without the collateral dispatch change. Assisted-by: OpenCode (GLM)
- getrows.cu: the 32-grain launcher comment claimed multi-type routing; state IQ4_NL-only and that MXFP4/ROCmFP4 stay on the QK_K kernel. - server-context.cpp: describe the cap lift as env-only (README is the reference), dropping the over-promising metadata parenthetical. Assisted-by: OpenCode (GLM)
The 16 MiB pinned staging buffer is held by a unique_ptr with ggml_backend_buffer_free as deleter for the whole loading loop, so a check_tensors throw or a ggml_backend_tensor_set failure inside the tensor loop can no longer leak it. Assisted-by: OpenCode (GLM)
supports_op refused the gated DSV4_HC_PRE and the identity-comb DSV4_HC_POST on RDNA3.5 (comment said 'the unfused graph goes through the hc_mix fusion, which is faster'). The sturdier consequence measured on strix-halo-001 with Qwen3.8-Flash-Next (parallel >= 1) is that the graph keeps a CPU split: the fused nodes land on the CPU backend, and, since the hc ops are dissecting every layer, the server keeps multi-GiB host compute buffers (ROCm_Host 1.84 GiB pinned plus ~7 GiB anonymous buffers) and decode throughput collapses to 0.3-1.6 tok/s at long context while the GPU idles. The CUDA kernels for both forms already exist (dsv4-hc.cu dsv4_hc_pre_f32 / dsv4_hc_post_f32, gated and plain). Enabling the gated/identity forms keeps the whole graph on the GPU. Earlier in-tree numbers (Qwen3.8-Flash-Next Q2_0, gfx1151: pp2048 908.7 fused vs 832.5, tg128 30.2 vs 30.7) already showed the per-op cost is a few percent at prefill; the CPU split cost dwarfs it. Operators wanting the older split path still can: LLAMA_FUSED_DSV4_HC_PRE=0 keeps the unfused hyper-connection graph. Assisted-by: OpenCode (GLM)
The host-resident PLE gather keeps a std::vector<float> sized to the largest gathered batch (worst case on a 4-slot cont-batched ubatch: n_heads * n_tokens * ple_head_dim * float). The capacity persists to the next gather; shrink_to_fit after the tensor_set releases it to the allocator between gathers so idle servers do not sit on the peak-tuned host staging. Both host-gather paths (shared head-vocab table and ple_disk pread gather) release theirs. Assisted-by: OpenCode (GLM)
gaetan-puleo
marked this pull request as draft
October 11, 2026 13:15
gaetan-puleo
marked this pull request as ready for review
October 11, 2026 13:35
Replaces the per-gather unconditional shrink_to_fit with a high- watermark: the vector grows via resize (capacity retained), and the peak is only released after watermark_age (=8) consecutive gathers that are less than half the peak. Bursty cont-batched 4-slot loads keep their capacity (no realloc churn on the hot path); idle servers return the multi-hundred-MB staging after the aging window. Also fixes the stale shrink_age comment. Assisted-by: OpenCode (GLM)
…te, HC flip, gather shrink Maintainer review findings on this campaign PR, acted on: - qsa-prefill/qsa-decode: restoring the base 262140/262144 ceilings. The ruled-16-bit block ids are the real limit: past 262140 keys block 65535 doubles as the padding marker and higher blocks wrap to the start of the cache, silently corrupting attention past that depth. Widening the ids is #124's job (already open, and it lowers the model-side limit to 1M); this 994k run was survival-throughput only, with no output-quality gate. - server-context.cpp: LLAMA_SERVER_ALLOW_CTX_BEYOND_TRAIN removed. The cap exists to gate un-scaled over-context runs, and only those reach the item-1 corruption path. YaRN configurations do not need the env (base raises n_ctx_train to yarn-orig-ctx / freq_scale). - getrows.cu/dequantize.cuh: the 32-grain kernel is withdrawn; supports_op for IQ4_NL returns to the base QK_K gate. The offloaded PLE gains the campaign claimed for it were the file-mmap-to-pinned-host move that item 1 (staging) already covers, and the gather buffer was at most tens of MiB (the real 3.3 GiB runtime growth is traced to ggml_backend_sched_new's hv_tensor_copies allocation with the server's per-slot sampling reserve, not the kernel work). - qwen4exp.cpp: the embd_buf watermark is dropped with it; capacity never regrew past ~40 MiB. - tools/server/README.md: the env-knob table no longer splits the generated options table; non-ASCII removed; LLAMA_SERVER_ALLOW_CTX_ BEYOND_TRAIN row deleted. Kept from the campaign: the mmap pin-staged loader fix (the HSA busy-wait crash), the pool-free knob (with a large-threshold note: frees during HIP graph capture should be avoided at small thresholds), the RAII staging owner, and the env-knob docs. Assisted-by: OpenCode (GLM)
…bound to 1M The maskless selected-key route was capped at 262140 keys because qsa_prefill stores whole-block ids as uint16 and uses 0xFFFF as the padding sentinel: 65535 real blocks * ratio 4 = 262140, and at n_kv = 262144 the 65536th block id collides with the sentinel. A maskless ubatch above 512 tokens at any depth past 262140 has no kernel that accepts it, so fattn aborts the instance (#85, and the repeated aborts at k [256,265984,2,1] under -c 524288). Store the union's block ids as uint32 with 0xFFFFFFFF as the sentinel and raise the bound to GGML_QSA_MAX_KEYS = 1048576, the max context the Qwen3.8-Flash-Next card supports with RoPE/YaRN. This replaces the earlier cap-the-gate approach proposed in review on this PR: the fatal band disappears entirely and per-slot context can go past 262140 without falling back to the masked route. - qsa-prefill.cu: ublk becomes uint32 (umask stays uint16: 4 bits per query slot for QSA3_G=4 queries over ratio=4 keys). The descriptor load now reads one uint4 per chunk instead of a uint2 of two packed id-halves; cap is always a multiple of 4, so (g*cap + 4c)*4 stays 16B-aligned. blk_of() tests the sentinel per element; pad slots still resolve to block 0 with no mask bits and are discarded exactly as before. - qsa-decode.cu: depth bound follows the same constant. Its ids are gathered directly (no block packing); the per-row unique table is sized by ids->ne[0] (<= 2560), not by nk, so depth was never structurally limited. - common.cuh: one shared GGML_QSA_MAX_KEYS so the two kernel gates cannot drift apart again; qwen4exp.cpp mirrors it (llama/ must not include ggml-cuda). - qwen4exp.cpp: QSA_MASKLESS_MAX_KV = 1048576 replaces the bogus 16777216, which was checked against neither kernel bound. Verified on the host only: the CPU build of libllama/ggml compiles, test-qsa-prefix and test-qsa-fast pass, and a host harness reproducing the writer/reader lines proves the old and new encodings agree for every id < 65535, that the old one mis-serves 65535/65536/66495 (sentinel aliasing and truncation) where the new one is exact, that padding still resolves to a discarded block 0, and 20000 randomised lists round-trip. NOT run on gfx1151: no ROCm/HIP toolchain here, so the kernel itself is unbuilt and unmeasured. Expect +16 MiB of pool per ubatch at ids=2051, n_q=4096 (uint32 ublk doubles that buffer). Assisted-by: Piclaw (Qwen3.8-Flash)
|
Prefill regression from Setup: Ryzen AI Max+ 395 (gfx1151, 128 GB), HIP, Qwen3.8-Flash-Next UD-IQ4_XS (expert gate/up IQ3_S, down IQ4_NL), MTP draft,
So on this model the fused path costs about 17% of pp2048 at every depth, not ~8%. Decode was unchanged within noise. We did not see the CPU split or decode stall that the guards were removed for (2 slots at 262144 each, all layers on GPU with the guards in place), so it may depend on Would you consider keeping the guards by default and making the fused form opt-in (or the reverse only when the graph would otherwise split)? Happy to run more shapes if useful. |
Collaborator
|
Verdict at d7116c8: FAIL as pushed - the description lists the DSV4 HC flip as withdrawn, but the Setup: base Loader staging: fixes a real hang (measured)llama-server
QSA (#124 cherry-pick): exact, same speed as our patch (measured)
The runtime-indexed DSV4 HC supports_op flip: still in the code, regression (measured)llama-bench,
Pool knob (measured)
Docs (code reading)
Not verified
Suggestion: drop the DSV4 HC hunk (so the code matches the description), fix the README table, and guard or drop the pool knob. The loader staging and the #124 part are good to go. Run by an AI agent (Claude) at the direction of the reviewer. |
gaetan-puleo
changed the title
gaetan-puleo
changed the title
gaetan-puleo
changed the title
gaetan-puleo
changed the title
gaetan-puleo
marked this pull request as draft
October 12, 2026 05:11
|
I've been running this (with #124 ) for almost 24 hours without any new issues under various coding workloads model = q38fn |
3 of 5 tasks
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR carries the fixes measured during a long-session stability campaign on strix-halo-001
(AMD Ryzen AI MAX+
gfx1151, ROCm 10) againstmaster@6aa81b88, built withGGML_HIP=ON,GPU_TARGETS=gfx1151, running Qwen3.8-Flash-Next-UD-Q4_K_XL only. One PR (per request) covering onemeasured scenario: stable ROCm sessions with mmap-based loading across many concurrent/long sessions.
After the maintainer-grade review on this campaign, the PR was narrowed to the fixes that hold under
review; the ambitious long-context items were withdrawn (details below).
Kept (4 areas)
1.
llama-model-loader.cpp- mmap load hang (crash, severity: high)A direct
hipMemcpyfrom file-backed (non-pinned) host memory never completes on this platform;rocr::core::BusyWaitSignal::WaitRelaxed(HSA) busy-waits at 100% CPU forever (gdb backtrace recorded),with the loader frozen after ~76 GiB uploaded,
read_bytesfrozen. Reproduced on every--load-mode mmapstart.Fix: in
load_all_data(), device-bound copies from a file mapping are staged through a pinned buffer inchunks (the same mechanism the non-mmap path already uses).
auto)--load-mode mmap)Pss_AnonMemAvailableat 4 slots x 150k2.
llama-model-loader.cpp- RAII owner for the staging bufferThe 16 MiB pinned staging buffer is owned by a
unique_ptrwithggml_backend_buffer_freeas deleter forthe whole loading loop, so a
check_tensorsthrow or a failedggml_backend_tensor_setinside the tensorloop cannot leak it.
3.
ggml-cuda.cu-GGML_CUDA_POOL_FREE_MIN_MBreleases oversized scratchThe device pool caches freed blocks forever; at long-context workloads transient prefill/decode
intermediates ratcheted the pool and never returned it. With the knob set (512 MiB in these runs), blocks
at or above the threshold are
cudaFreed on release (legacy/non-VMM pool only). Measured GTT delta= 0 MB across prefill stages in the campaign server runs.
4.
tools/server/README.md- document implementation knobsA dedicated subsection (placed after the generated options table, ASCII-clean) documents
GGML_CUDA_POOL_FREE_MIN_MB, plus the pre-existingQSA_DIRECT,GGML_QSA_RANKSORTandHC_NO_MIX_FUSEinternals.
QSA depth limit fixed in this PR (integrated from #124)
Selected-key block ids widened to uint32 (
qsa-prefill.cu/qsa-decode.cu/common.cuh,model-side gate in
qwen4exp.cpp): the uint16 ids were the actual limit (past ~262140 keys block 65535doubles as the padding marker and higher blocks wrap to the cache start - silent corruption). The
kernels now store block ids as
uint32with0xFFFFFFFFas the padding sentinel and takeGGML_QSA_MAX_KEYS = 1048576; the maskless model-side gate mirrors it (QSA_MASKLESS_MAX_KV).Note: this commit is the content of cuda : widen QSA selected-key block ids to uint32 and lift the depth bound to 1M #124 (
fix/qsa-maskless-nkv-cap, commit86bd254,byte-identical kernel change, adapted here to the newer
qwen4exp_use_block_selectionsignature).If cuda : widen QSA selected-key block ids to uint32 and lift the depth bound to 1M #124 merges first, this PR rebases cleanly; otherwise the same one-line-owner strategy applies.
Output-quality gate this time: needle test at ~700k depth, asked at ~854k context, single slot on
yarn (
-c 1000000 --parallel 1, no env override needed): the hiddenCORDIAL-SEVENstring isretrieved verbatim (
STAGE_B_CONTENT: 'CORDIAL-SEVEN'); the control run without the needle answersnone(no hallucination). Prefill ~923 t/s to 854k tokens; decode 16.6 t/s at that depth; GTT flatat 110.5 GiB throughout; test-backend-ops
GET_ROWS 543/543,DSV4 38/38.LLAMA_SERVER_ALLOW_CTX_BEYOND_TRAIN=1(server-context.cpp): not needed for YaRN - base alreadyraises
n_ctx_traintoyarn-orig-ctx / freq_scale; the env only opened unscaled over-context runs,which is what the cap exists to block (and how this campaign reached the faulty QSA depth). Removed.
IQ4_NL
get_rowsat rows % 32 (getrows.cu,dequantize.cuh,supports_op): withdrawn - the_32dequant helper lacked the per-thread lane-offset+coverage semantics, giving racy/partial writes(stock
GET_ROWSiq4_nl verified failing at-j 4, while base passes543/543). The PLE offloadpath also does not use that kernel on this device, so the correct effect (an mmap-backed file mapping
instead of an anonymous copy) is provided by the item 1 staging fix already.
fused DSV4 HC pre/post on RDNA3.5 (
supports_op): withdrawn - the base already opts out of thefused form when it would run on the CPU, and the removed in-tree comment measured the fused path ~8%
slower at prefill (fresh bench on this campaign rig: fused default
pp2048913 vs unfused-fusion1016 t/s;
tg12824.5 vs 26.4).PLE gather
embd_bufhigh-watermark (qwen4exp.cpp): withdrawn - the gather staging is at most~40 MiB (
ple_n_heads=16, ple_head_dim=160, ubatch tokens*heads); the "multi-GiB" runtime growth onthe campaign rig was traced to
ggml_backend_sched_newallocatinghv_tensor_copieswith the server'sper-slot sampling-node reserve (see campaign note), not this buffer.
Campaign note (for a follow-up issue, not this PR)
With
--parallel 4, the sampled server sampler chain reports ~4M nodes/slot (n_candidates = 128*1024probe), so
sched_reserverequests a ~2^24-node hash:hv_tensor_copiesthen allocateshash_size x 3 backends x 4 copies x 8 B~ 3.2 GiB + keys (~256 MiB) of anonymous process memory percontext (backtrace on
ggml_backend_sched_new). Capping the sampler-node estimate to the real decodebudget would drop that to a ~262k-entry hash (~16 MiB).
Validation (this rig, after narrowing)
ninja llama-server llama-bench test-backend-opsclean (0 errors).test-backend-ops -b ROCm0:GET_ROWS543/543,DSV4_HC_PRE/POST/COMB38/38.llama-benchon Qwen3.8-Flash-Next Q4_K_XL:pp2048 1016.4 +/- 21.0,tg128 26.44(unfused path).mmap), health OK, PONG completion, plus a 4-worker x 2-stageconcurrent smoke (perfect prefix-cache reuse) with host
Pss_Anonat reuse-ready levels.Assisted-by: OpenCode (GLM)