Repository navigation
Conversation
The K tile is invariant across the head loop but coopMatLoad sits inside it, so the shader re-read the same 8 MatrixA fragments from shared memory once per head tile - 4x at N_HEAD=64. Each wave64 fragment load moves 1 KiB of LDS traffic because the 16x16 f16 fragment is replicated 4x across the subgroup. Same loads of the same data feeding the same coopMatMulAdd sequence, so the output is bit-identical; the fragments just live in registers (8 of them, 16 f16 per lane) instead of being re-fetched. gfx1151, test-backend-ops perf, kv 8704/33280/131584 x batch 1-15: 16-20% faster at every shape, no spills, no regression. Batch 1 is ordinary DSv4 decode and gets 12-18% of that. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Assisted-by: Claude (Opus 5)
…ecode CM shader Builds directly on pepuscz's PR #6 and issue #10. Their per-kernel table - 43.8 us scalar against 27.9 us small CM per 1k scanned rows at batch 5, and the same ratio at every depth - is what showed the indexer's cost is a per-tile constant rather than anything to do with tokens or bytes. Once the cost is per tile, the thing to minimise is the tile count, and that is what this changes. The finding is downstream of their work; only the choice of shader differs. The decode CM shader puts 16 HEADS in the coopmat N dimension and dispatches one workgroup per token, so it issues 4*n_batch tiles per 16 KV rows. That is the arithmetic minimum - (64 heads x n_batch tokens) / 16 columns - because 64 heads fill its 16 columns exactly, with no remainder at any batch. The small CM shader puts 16 TOKENS in N and pays a flat 64 tiles however small the batch is, so at batch 5 eleven of its sixteen columns are padding. Measured cost tracks tile count: at kv=131584 batch 1 costs 1.95 us per 1k scanned rows and batch 5 costs 26.28, i.e. 13.5x for 5x the work, and both shaders sit at 6.6-7.8 ns per tile. The decode CM shader body has no n_batch == 1 assumption - token is gl_WorkGroupID.y and it indexes q/w/mask/dst by it - so the old gate was an artefact of where it was written. gfx1151, kv=131584 (526k source tokens), us/run, shipped vs this: batch 2 2335.6 -> 419.0 (5.6x, was on the scalar path) batch 3 3492.3 -> 631.8 (5.5x, was on the scalar path) batch 4 3267.9 -> 840.8 (3.9x) batch 5 3465.6 -> 1119.4 (3.1x) <- DSpark n-max 4 verify shape batch 8 3704.9 -> 1764.1 (2.1x) batch 15 4467.9 -> 3231.6 (1.4x) Batch 16 and 32 are unchanged, which confirms the arms are isolated. PR #6's route is kept as the opt-out arm rather than deleted, and is promoted from opt-in to default-on so that one variable is enough to reach it: default decode CM for the whole 2-15 window ..._DECODE_CM_BATCH=0 small CM for 4-15, scalar for 2-3 (PR #6) ..._DECODE_CM_BATCH=0 SMALL_CM=0 scalar for 2-15 (pre-PR #6 baseline) Verified all three route as documented and pass 29/29. Keeping it is not just courtesy: the tile-count argument is hardware independent, but decode CM re-reads the K tile once per token and that part is bandwidth dependent, so the crossover need not sit in the same place on other devices, and these numbers are from one gfx1151 box. A (head, token) packing shader was also tried and is strictly worse - it reaches 4b tiles only when the batch divides 16, and it divides the workgroup count by the batch. Kept out of tree. Numerics: decode CM sums the 64 heads in tiles of 16 rather than one at a time, so f32 accumulation order differs from the small CM path. Cleared by KLD A/B on trunc10 (-c 8192, -b/-ub small so every indexer dispatch routes through the window, 8 chunks wikitext-2, same binary, only the env var differing): -ub 8 mean KLD 0.000000, max 6.0e-5, RMS dp 0.000%, same-top 99.994%, PPL 10507558.3156 identical in both arms -ub 5 mean KLD 0.000000, max 8.8e-4 but 99.9% 4.9e-5 (one tail event, no argmax flip), RMS dp 0.000%, same-top 100.000% Same class as PR #6's own measured numbers (max 5.5e-5, same-top 100.000%) and 45x tighter than the FA_WAVE32 change already shipped. Adds the eval coverage the 2-15 window never had (batches 4/8/15 at kv=256) and the perf grid these numbers came from. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Assisted-by: Claude (Opus 5)
Same defect as the decode CM shader, larger blast radius. `k_sh[wave]` is written
once above the head loop and never touched again, but `coopMatLoad` sits inside
both the head_base and head_local loops, so each wave re-reads the same 8 MatrixA
fragments N_HEAD times: 512 loads where 8 would do. Each wave64 fragment load
moves 1 KiB of shared-memory traffic because the 16x16 f16 fragment is replicated
4x across the subgroup.
Same loads of the same data feeding the same coopMatMulAdd order, so the output is
bit-identical; the fragments live in registers (8 of them, 16 f16 per lane) instead
of being re-fetched.
This source compiles to both the prefill pipeline and the small-batch one, so both
gain. gfx1151, test-backend-ops perf, us/run:
prefill (lightning_indexer_cm_f16)
kv 8704 nb 2048 27492.5 -> 18207.0 1.51x
kv 33280 nb 2048 104101.9 -> 68947.8 1.51x
kv 131584 nb 2048 421239.8 -> 281540.5 1.50x
kv 131584 nb 16 3456.2 -> 2329.2 1.48x
kv 131584 nb 32 6858.6 -> 4623.6 1.48x
small batch (lightning_indexer_cm_small_f16)
kv 131584 nb 5 3433.1 -> 2783.3 1.23x
kv 131584 nb 8 3682.3 -> 3105.1 1.19x
kv 131584 nb 15 4474.4 -> 4209.5 1.06x
1.48-1.54x on prefill at every depth and batch measured, flat. The small-batch
variant gains less, which says the 1-wave configuration is bound by its Q staging
and epilogue rather than by fragment traffic.
No regression at any shape, so the 8 held fragments (64 VGPRs per lane) neither
spill nor cost occupancy even in the prefill pipeline, which runs 512 threads at
62.7 KiB of shared memory and was already pinned to one workgroup per CU.
LIGHTNING_INDEXER 29/29.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Assisted-by: Claude (Opus 5)
…ation constant
The indexer pipelines hardcoded nh=64 (DeepSeek-V4's geometry) in supports_op and
as a shader constant, so any other head count fell to the CPU backend.
Qwen3.8-Flash-Next (qwen4_exp, released today) carries the same indexer at nh=4.
N_HEAD becomes specialization constant 1, one pipeline per supported head count
(LI_NH_VALUES = {4, 32, 64}), selected by q->ne[1] at dispatch. It must be a
specialization constant, not a push constant: the head loop's trip count has to
stay visible to the compiler - as a push constant the loop cannot unroll and
decode costs ~55% (measured, gfx1151, kv=131584, and why this commit is not the
simpler design).
nh=4 does not fill a 16-wide head tile, so the tail tile stages q as zeros
(relu(0)*w = 0, padded heads drop out of the sum) and the weight read clamps its
index. Both guards are gated on N_HEAD % TILE != 0, a specialization-constant
expression, so at nh=64 they fold away at pipeline compile and the codegen is
equivalent to the previous shader - measured, because relying on the compiler to
range-prove head < N_HEAD instead still cost ~45%:
kv=131584, us/run before after
decode b1 226.8 219.6
decode b5 1058.8 1007.4
decode b15 3174.0 2966.8
prefill b16 2329.2 2230.2
prefill b2048 281540.5 281169.1
No regression at nh=64; the small decode improvement is within a cautious
reading of run-to-run variance and is not claimed.
test-backend-ops LIGHTNING_INDEXER: 44/44, up from 29/29 - the suite's existing
nh=32 cases had been silently reporting "not supported" under the old gate and
now run against the CPU reference.
Head size stays pinned at 128: the coopmat tiling and f16vec4 staging both
assume it, and every known indexer model (DSv4, GLM-DSA, Qwen4-exp) uses 128.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Assisted-by: Claude (Opus 5)
The scalar64 shader kept its hardcoded 64-head loop when the pipelines went per-head-count; nh 4 and 32 would have run with the wrong trip count. Assisted-by: Claude Fable 5
The GEMM2 P load depends only on bc_chunk, but sat inside the hsv_tile loop, so all four fragments were re-read from shared memory once per tile (2 tiles at HSV=128, 4 at HSV=256). Load them once into a coopmat array before the loop. Psh is not written again until the next KV block, so the fragments stay valid across the barriers inside it. Measured on gfx1151 (RADV), Qwen3-Coder-30B-A3B, pp2048 f16 KV: d8192 +6.9%, d16384 +8.1%, d32768 +9.2%. VGPRs and LDS unchanged, 0 spilled. FLASH_ATTN_EXT suite 5105/5105. Assisted-by: Claude Fable 5
Psh held P as [kv][query]. The GEMM2 UseA load therefore had to request ColumnMajor, and RADV only attaches an alignment hint to the internal column-major case, which for UseA means the RowMajor request. The load was emitted as 16 separate 16-bit shared reads per fragment. Store P as [query][kv] instead and request RowMajor. The producer now writes four scalar components rather than one vec4; those go to disjoint bytes, so there is no read-modify-write and no partially written vec4. Also updates the host shared-memory estimator, which mirrored the old stride and would otherwise disagree with the shader. Perf-neutral on gfx1151 (within 1% at hd128 and hd256), but it shrinks Psh: LDS 16384 -> 15360 B and code size 13896 -> 13596 at hd128, VGPRs unchanged at 96 with 0 spilled. Kept because it is free and removes a scalar shared-memory access pattern. FLASH_ATTN_EXT suite 5105/5105. Assisted-by: Claude Fable 5
get_fa_tuning_params_coopmat1 took device->subgroup_size unconditionally, so on a 64-wide device the coopmat1 FA path always ran wave64. The sibling scalar path already does AMD-specific wave selection; the coopmat1 path never got the equivalent. Narrowing is free exactly when it does not add an iteration to the O-accumulation loop, which runs ceil((HSV/4) / threads_per_rowgroup) per row. On a 64-wide device the test reduces to hsv <= 128. Above it the narrow subgroup issues 1.5x to 1.8x the instructions for the same SIMD passes and hd256 measures 6 to 18 percent slower, so the rule declines. Gated behind GGML_VK_FA_WAVE32 (=1 rule, =2 forces the pin regardless of head size, diagnostic only). Off by default. Measured on gfx1151 (RADV), model-level pp2048, Qwen3-Coder-30B-A3B: d0 +2.5%, d8192 +8.4%, d16384 +10.1%, d32768 +11.3%. Op-level across head sizes, largest where wave64 wastes the most lanes: hsv=64 +12.3%, hsv=96 +7.4%, hsv=128 +6.5%, hsv=256 correctly declined. Full test-backend-ops suite 15884/15884 at =0, =1 and =2; the pin was confirmed to engage in-band via pipeline VGPR statistics rather than inferred from the env var being set. Assisted-by: Claude Fable 5
Split from the fork's flag-flip commit (f7d804ee7): GGML_VK_FA_WAVE32 now defaults to 1 (apply the HSV-based narrowing rule); =0 opts out and =2 keeps its diagnostic force meaning. The subgroup_size_control device guard is unchanged, so devices without a 32-wide subgroup mode are unaffected. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Assisted-by: Claude (Opus 5)
Select the matmul_id tile by expected per-expert token count (nei1*nei0/n_expert) instead of aggregate nei1, which always picked the widest tile and left most N-lanes empty at MoE prefill (~16 rows/expert vs BN=64). Gated on GGML_VK_MMID_SMALLN=1, default off pending cross-model validation. Was null (+0.7%) standalone pre-row-lists: smaller tiles meant more workgroups each re-paying the per-WG id scan. With the Stage 1 row-list prepass the scan is gone and the occupancy win materializes. Strix Halo (RADV gfx1151), Qwen3.6-35B-A3B UD-Q5_K_XL, clean window, pp512 4-way (rowlists x smalln): 914.7 / 937.9 / 1005.3 / 1063.6 t/s (combined +16.3% vs baseline). MUL_MAT_ID q5_K 2.33->4.43 TFLOPS (1.91x), q6_K 1.99->3.57 (1.79x) cumulative. Hot pipelines verified via probe: matmul_id_*_f32 (y_f32 path, quantize_y does not engage for these quants); smalln flips tile _m -> _s. 790/790 MUL_MAT_ID both rowlists configs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Assisted-by: Claude (Opus 5)
Two env-gated overrides of the mmid small-tile config (KHR coopmat branch only, scoped to the mul_mat_id quant pipelines; dense untouched): - GGML_VK_MMID_TILE16: BN/WN 32->16. NEGATIVE (-3.8% e2e): experts with n_e>16 split into two column tiles and re-stream their full weight matrix; A-traffic scales with sum(ceil(n_e/BN)), so BN must not drop below the mean per-expert n. Kept as a documented dead end. - GGML_VK_MMID_BM64: BM 32->64, BLOCK_SIZE 64->128 (two warps). +1.3% e2e on top of rowlists+smalln: same A-traffic, half the ir-tiles so half the per-expert B re-reads, larger WGs hide latency. Strix Halo, Qwen3.6-35B-A3B UD-Q5_K_XL pp512, clean window, all on top of RL+SN control 1065.4 +/- 3.3: BM64 1078.9 +/- 3.5, TILE16 1024.6, BM64+TILE16 1007.5. Cumulative vs pre-Stage-1 baseline: 914.7 -> 1078.9 (+17.9%). 790/790 MUL_MAT_ID for all configs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Assisted-by: Claude (Opus 5)
…ated) Extends the BM64 idea to the medium tile: GGML_VK_MMID_M128=1 raises the mmid m-tile to BM=128 / BLOCK_SIZE=4*subgroup (four warps), same A-traffic, half the ir-tiles and per-expert B re-reads. The medium tile is what the per-expert-n heuristic selects at n~64 (e.g. ub2048 on 256-expert/top-8 models, or ub1024 at 128 experts). Strix Halo, Qwen3.6-35B-A3B UD-Q5_K_XL pp2048, drain-verified window: ub2048 stack 1039.9 +/- 1.1 -> +M128 1091.6 +/- 2.6 (+5.0%); inert in the s-tile regime (ub1024 1137.2 vs 1147.9 ref). ub1024 remains the throughput sweet spot for this model. 790/790 MUL_MAT_ID. Also measured this session, NOT kept: caching counts+row lists across the three expert matmuls of a layer (they share one ids tensor) was correctness-clean but perf-null (1066.3 vs 1070.9 baseline) - the prepass dispatches and barriers are already free on this queue; reverted rather than carry the invalidation surface. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Assisted-by: Claude (Opus 5)
Force required subgroup size 32 on the KHR-coopmat mmid quant pipelines (pipeline_dequant_mul_mat_mat_id[*] only; dense untouched). Hypothesis: RDNA3.5 WMMA is wave32-native, so RADV may lower KHR_coopmat better at wave32 than at the reported default of 64. Mechanics: the cm1 path in mul_mm.comp derives its warp grid from the real subgroup (warp_i = gl_SubgroupID, tiw = gl_SubgroupInvocationID) while NUM_WARPS = BLOCK_SIZE/WARP (spec constants) sizes coopmat_stage[] and ballots_sh[] and warp_r/warp_c assume NUM_WARPS == (BM/WM)*(BN/WN). Forcing sg32 with WARP=64 would over-run those shared arrays and leave warp_c outside the tile, so the gate (a) sets WARP=32 in shadowed copies of the s/m/l mmid warptiles (composes after the BM64/M128 shadows) and (b) halves WM (or WN, keeping WM>=TM, WN>=TN) until the doubled subgroup count exactly tiles BM x BN again, asserting both invariants. BLOCK_SIZE is kept, so workgroup shape, load loops and shmem match the wave64 stack; per-lane accumulator footprint is also unchanged (half the lanes per subgroup, half the (WM/TM)*(WN/TN) fragments). The required size is passed via a scoped CREATE_MM redefinition adding a trailing required_subgroup_size arg (fp16-branch pattern), gated on subgroup_size_control covering 32 since ggml_vk_create_pipeline_func silently drops the required size otherwise. Correctness (test-backend-ops -o MUL_MAT_ID -b Vulkan0): 790/790 plain, 790/790 WAVE32=1, 790/790 WAVE32+SMALLN+BM64. Perf, Qwen3.6-35B-A3B-UD-Q5_K_XL, fa=1 b/ub=512 ctk/ctv=q8_0 pp512 r=3, atomic window, canary clean: stack (SMALLN+BM64), wave64 default: 1076.61 +/- 3.71 t/s stack + WAVE32: 1106.73 +/- 2.52 t/s (+2.8%) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Assisted-by: Claude (Opus 5)
Convert contiguous f32 activations (B) to f16 for quantized MUL_MAT_ID on KHR_coopmat devices and run the matmul_id_subgroup_<quant>_f16 kernels instead of the f32-B ones. Halves B bytes and buf_b shared memory. The _f16 SPIR-V already exists for every quant type; this adds a parallel env-gated pipeline array (pipeline_dequant_mul_mat_mat_id_f16b), extends the getter to return it for src1=F16 on non-coopmat2 devices, relaxes the src1-type assert, and forces the existing y_non_contig convert-to- prealloc_y plumbing (same as coopmat2). Default OFF, zero behavior change when unset. Measured on Radeon 8060S (RADV gfx1151), Qwen3.6-35B-A3B-UD-Q5_K_XL, -fa 1 -b 512 -ub 512 -ctk q8_0 -ctv q8_0 -p 512 -r 3, stacked on GGML_VK_MMID_SMALLN=1 GGML_VK_MMID_BM64=1: stack (f32 B, canary): pp512 1075.04 +/- 8.66 t/s stack + F16B: pp512 1100.66 +/- 2.37 t/s (+2.4%) test-backend-ops test -o MUL_MAT_ID -b Vulkan0: 790/790 with and without GGML_VK_MMID_F16B=1. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Assisted-by: Claude (Opus 5)
Assisted-by: Claude (Opus 5)
The MMQ J config is chosen from n, so a wave-partitioning error in one J tile only shows up when n sits on that tile's boundary. The neighbouring n picks a different J and passes, which hides it. The existing quantized cases stop at n=129 and the general MUL_MAT set jumps 64 -> 4096, so nothing lands on 256 or 512 and the whole class went untested. Sweep n over 255/256/257/511/512/513 for q8_0, q4_0, q4_K, q5_K and q6_K, in both MUL_MAT and MUL_MAT_ID. On an RDNA3.5 build that runs the J128 kernel with 16 wave32 waves over a 128-row tile these fail for every n that selects J128 and pass at n=513, which selects J112. A build predating that config passes the whole sweep. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> (cherry picked from commit b54cd8a) Assisted-by: Claude (Opus 5)
30d8bb0 raised mul_mat_id_param_count to 6 for the fused MUL epilogue and gave every mul_mat_id shader a binding 5 (FusedScale), including the coopmat2 one, which binds it purely for descriptor-layout parity. The coopmat2 pipeline creation block was left passing a literal 5, so two things go wrong there: - the pipelines are created with 5 descriptors while the shader declares 6 - PARAMCOUNT == mul_mat_id_param_count doubles as the "this is mul_mat_id" argument to ggml_vk_mul_mm_cm2_spec, so it went false and every coopmat2 mul_mat_id pipeline was specialized as a plain matmul, dropping the trailing spec constant Only reachable where device->coopmat2 is true. gfx1151 does not take that path and the v0.5 release predates the constant bump, so neither is affected. Not validated on hardware - no coopmat2 device here. The block does compile: built with the pinned shaderc (GL_NV_cooperative_matrix2 supported). The two OCP FP4 sites stay behind GL_EXT_float_e2m1, which that glslc does not support. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Assisted-by: Claude (Opus 5)
Split from the fork's flag-flip commit (f7d804ee7): GGML_VK_MMID_F16B, BM64, M128, WAVE32 and SMALLN now default on with =0 opt-out. The wave32 gate keeps its subgroup_size_control device guard, so devices without a 32-wide subgroup mode are unaffected. GGML_VK_MMID_TILE16 stays opt-in (documented negative on gfx1151). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Assisted-by: Claude (Opus 5)
RDNA3.x WMMA is wave32-native, so a wave64 subgroup issues each coopmat op as two halves. GGML_VK_MMID_WAVE32 already exploits this for mul_mat_id and leaves the dense pipelines at the driver default; this gives dense the same treatment, gated on measurement rather than a flag. BLOCK_SIZE is kept, so the subgroup count doubles and WM (or WN) halves until the warp grid tiles BM x BN again. Only the quantised tiles are retiled: on gfx1151 a standalone MUL_MAT microbench at both dense FFN shapes reads q6_K +5.2..+10.8%, q8_0 +5.4..+8.4%, q4_K +0.7..+9.1%, q4_0 -1.5..+1.8%, while f16 reads -6.7..+6.4% and bf16 ~0 - the float paths are bandwidth-bound on the weight stream, not issue-bound. The win tracks inline dequant instruction count (q6_K 3907 -> 3433 instructions, identical 192 VGPRs and 8 subgroups/SIMD). The required subgroup size is now the tile's own WARP for every dense coopmat pipeline. The cm1 shaders derive their warp grid from gl_SubgroupID and size shared arrays as NUM_WARPS = BLOCK_SIZE / WARP, so WARP and the real subgroup must agree; leaving that to the driver made the agreement incidental. Scoped to AMD coopmat1 on a wave64 default; other vendors keep the driver default. GGML_VK_DENSE_WAVE32=0 disables, =2 also retiles the float tiles. Qwen3-32B Q6_K_XL pp2048 +7.2% at ub256 / +3.9% at ub2048, Qwen3.8-27B +5.3% / +4.8%. PPL unchanged: 6.9496 +/- 0.24246 in both arms, all 20 per-chunk values identical, since the retile changes which warp owns an output sub-tile and not the K-reduction order within an element. Assisted-by: Claude Opus 5 (cherry picked from commit 448994e9637405610ced3e0ead02c7b6fa688314)
All default OFF, so the same binary A/Bs each change. GGML_VK_CONCAT_TRANSPOSE: delta-net does ggml_transpose() into a dim-0 ggml_concat(), which the generic concat reads fully de-coalesced. Route that shape through a 32x32 shared-memory tile transpose. CONCAT 11877 -> 957 us/op. GGML_VK_MMID_SCALE_EPILOGUE: apply the following MUL's per-(expert,token) broadcast scale as mul_mat_id writes out, removing a 134 MB write plus read back. Prefill only; the existing fusion is gated to mat-vec. Not implemented in the coopmat2 shader, so it is refused there. GGML_VK_FUSE_UNARY_MUL: silu(x)*y is two nodes in the delta-net path; run it as the existing swiglu split. 750 -> 443 us/op. GGML_VK_MMID_WG256: the RADV tuning gives the dense large tile 256 threads on a 128x128 tile but left the mul_mat_id variants at 128. Qwen3.6-35B-A3B UD-Q4_K_XL, gfx1151, pp2048 at ub2048: 1223.75 -> 1733.64 t/s. test-backend-ops CONCAT/MUL_MAT_ID/UNARY/MUL pass; generated text is unchanged with each flag on, and the disabled paths are bit-identical to before. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> (cherry picked from commit 6f7a49e) Assisted-by: Claude (Opus 5)
The fused epilogue derives the scale index from row_ids as [token * nei0 + expert_slot], which carries no 4th dimension, but the gate admitted any ne[3] as long as the scale and the matmul agreed. A tensor with ne[3] > 1 would read the wrong scale for every batch past the first and return quietly wrong results. test-backend-ops never generates such a case, so the suite passed throughout; found by reading the gate against the shader. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> (cherry picked from commit 016e906) Assisted-by: Claude (Opus 5)
The delta-net conv-state path transposes straight into a dim-0 concat, so the generic concat kernel walks src1 with a conv_channels * 4 byte stride. On qwen35 that is 40960 B, which is 160 * 256 B with 160 % 16 == 0, so every read lands on the same one of the 16 memory channels: 13.7 GB/s against 138.9 GB/s for the tiled path. The tiled-transpose route has been behind GGML_VK_CONCAT_TRANSPOSE=1 since it landed. Turn it on by default and keep GGML_VK_CONCAT_TRANSPOSE=0 as the opt-out. Qwen3.8-27B pp2048: +0.4% at ub 256, +4.7% at ub 1024, +7.2% at ub 2048. Assisted-by: Claude Opus 5
The scheduler's async input copies between graph splits land in the compute ctx on devices without a separate transfer queue, so the perf logger's fresh-ctx assert fired under partial offload (--n-cpu-moe). Assisted-by: Claude Fable 5
Nodes with no flops estimate (large copies, set_rows, mask fills) can pack a command buffer whose execution time grows with context length until it exceeds the amdgpu ring timeout (10s on the compute ring), causing the ring resets and DeviceLost reported at long context. Add a bytes-per-submit cap (default 8 GiB, GGML_VK_MAX_MB_PER_SUBMIT to override, 0 disables) alongside the existing flops and node-count gates. Assisted-by: Claude Fable 5
ggml-org#22789 replaced the fixed 30-entry split input array with a growable one and, in the same edit, changed the split-cutting heuristic from the constant to split->inputs_capacity: - if (split->n_inputs == GGML_SCHED_MAX_SPLIT_INPUTS) { + if (split->n_inputs >= split->inputs_capacity) { inputs_capacity starts at GGML_SCHED_MAX_SPLIT_INPUTS but doubles on demand and is never reset for the life of the sched, so once a split slot grows, the scheduler stops cutting there and the cut point ratchets up for every later graph build. Longer splits mean every cross-backend input copy is materialised at the split's start and stays live to its last use inside it, which raises the peak the compute-buffer allocator has to cover - n_copies times over under pipeline parallelism. Only multi-backend configurations can reach this. Keep the growable array, which is what fixes the original >30-input assert, and cut on the constant again as before ggml-org#22789. >= rather than == so the check keeps firing for splits that did have to grow. DeepSeek-V4-Flash UD-IQ3_XXS, gfx1151, -c 400000 -ub 2048 -fa 1 --fit off: Vulkan0 compute buffer 4714.00 MiB and 9157 graph nodes, byte-identical to the unpatched tree, and neither run grows a split past 30 inputs. Expected - one Vulkan device plus the CPU backend cannot exercise the path on this box. The reported case is 3 devices with pipeline parallelism. test-backend-ops -o FLASH_ATTN_EXT, run alone on gfx1151: 13257/13295 on both this and the unpatched tree, with the same 38 failing cases (identical case list, all type_K=q8_0 prec=def kv_view=1). Pre-existing on the branch, not touched by this change. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Assisted-by: Claude (Opus 5)
The _f16 SPIR-V and pipeline_dequant_mul_mat_mat_f16 already exist, but are only populated in the coopmat2 branch. GGML_VK_DENSE_F16B populates them for coopmat1 too and routes B through the existing convert-to-prealloc_y path. Off by default: it helps large dense models and costs a little elsewhere. gfx1151, pp2048: Qwen3.8-27B and Qwen3-32B (hidden 5120) +5 to +7% for both q6_K and q8_0 weights, Qwen2.5-7B -1.2%, Qwen3-Coder-30B MoE -0.5%. Decode is untouched, ne1==1 does not reach this path. Numerically identical: mul_mm stages B into shared FLOAT_TYPE either way, so the f32-B kernel already rounds B to f16. Wikitext PPL matches to 4 dp. Assisted-by: Claude Opus 5
=auto restricts the f16-B path to ne10 == 5120, the only reduction width measured to gain, so it cannot fire on the widths that lose. =1 keeps the old all-shapes behaviour as a manual override. gfx1151, Qwen3.8-27B UD-Q6_K_XL pp2048, auto vs off: +5.8 / +6.0 / +5.9 / +5.3% at ub 256/512/1024/2048, which is 97% of the all-shapes win at ub256 and 82-86% above it. Qwen3-Coder-30B MoE is untouched, the gate never fires. The width equality is a stopgap until a per-shape predicate is derived; it will silently do nothing for a dense model of another width. Assisted-by: Claude Opus 5
The dense wave32 change rewrites s/m/l_warptile_mmq in place (WARP=32, WM/WN halved). The mul_mat_id tile block copied those rewritten vectors, so with GGML_VK_MMID_WAVE32=0 the mmid pipelines carried a WARP=32 spec constant but no required subgroup size and ran at wave64 with the wrong warp grid: test-backend-ops MUL_MAT_ID 700/7432 for every quantised type at n >= 16 (garbage output, not an error). Default settings were not affected; the opt-out was. Snapshot the dense tiles before the wave32 rewrite and build the mmid tiles from the snapshot. When the mmid wave32 pin is not applied, assert the coverage invariant and require the tile's own WARP as the subgroup size where the driver honours it, so the two can never disagree. Assisted-by: Claude (Opus 5)
Decode dispatches carry N = gqa_ratio query rows and the narrow subgroup loses there: Qwen3-Coder-30B q8_0 KV tg64 at d8192/d32768 measured 3.5 to 4 percent slower with the pin than without, while the prefill gain it was introduced for (up to 10 percent at d8192) needs the multi-row shapes. Apply the rule only when n_rows >= 32. Assisted-by: Claude (Opus 5)
… fusions rms_norm.comp runs one 512-thread workgroup per row, so on 128-wide rows (the GDN head norms) 3/4 of the lanes idle and every row pays a shared-memory tree reduction. rms_norm_small.comp gives each row one subgroup (subgroupAdd), for ne00 <= 256, f32 in/out, contiguous rows. It handles plain RMS_NORM and RMS_NORM_MUL, plus two fusions only it has: - RMS_NORM_MUL_MUL: the gated norm, normalised * gamma * gate (qwen3.5/qwen4exp build_norm_gated). The gate chain (sigmoid/silu of z) sits between the two MULs in the graph, so graph_optimize now schedules it ahead of the norm and keeps the three nodes adjacent. The products are the same f32 multiplies as the separate ops. - RMS_NORM_SCALE: build_gdn_l2_norm, scale(rms_norm(x, eps/n), 1/sqrt(n)). The factor is applied on the store after rounding the normalised value to f32, the same two roundings as the SCALE op. graph_optimize keeps the SCALE behind its norm. The kernel strides over rows, so the grid can be capped at 65535 workgroups and a subgroup size other than the one the host assumed still covers every row. GGML_VK_NORM_SMALL=0 restores the old path (and disables both fusions); GGML_VK_FUSE_NORM_EXTRA=0 keeps the kernel but disables the two fusions. gfx1151 (RADV), same binary with GGML_VK_NORM_SMALL off/on, ABBA x2, -r 2 (mean of 8): Flash-Next REAP-320 pp2048 -ub 2048 773.3 -> 785.0 t/s (+1.5%), Qwen3.6-35B-A3B pp2048 -ub 2048 2100.4 -> 2145.5 (+2.1%), Qwen3.8-27B pp512 -ub 256 430.5 -> 431.6 (flat). Norm ops 640/640 in all three gate settings. PPL moves only by the reduction order (subgroupAdd vs the shared-memory tree). Only tested on gfx1151. Assisted-by: Claude Opus 5.5
updated readme.md to remove halo-box fork
vulkan : multi-row coopmat1 flash attention for prefill, with a V^T scratch
The cm1 int8 MMQ epilogue wrote its f32 result one element at a time. On the RDNA3/3.5 wave32 accumulator layout lane l holds column l % 16 and rows (l / 16) + 2e, so every store instruction scattered 4-byte writes over 16 columns. Two subgroupShuffleXor(16) per four elements now give each lane four adjacent rows of its column, and the lane writes them as one vec4. This covers MUL_MAT and MUL_MAT_ID. - The packed path needs stride_d (and batch_stride_d or the batch offset) to be a multiple of 4. Otherwise the per-element store runs, for example with odd M. A vec4 that crosses the row end falls back to scalar stores, so nothing is written past M. - It is enabled by spec constant 12 (PACKSTORE) of mul_mmq_cm1.comp. GGML_VK_MMQ_PACKSTORE=0 turns it off. RDNA4 (blocked layout) and other tile shapes keep the old store. - test-backend-ops: new int8 MMQ cases with ragged M/N (M = 20, 36, 132, 199, 68x2 batch; N = 67, 129) for MUL_MAT and MUL_MAT_ID. Results on gfx1151 (RADV): - MUL_MAT + MUL_MAT_ID 14980/14983 with the gate on and off; the 3 failures are the same pre-existing CONCAT k=5120 cases in both arms. - Positive controls: swapping lanes inside the vec4 fails thousands of MUL_MAT and MUL_MAT_ID cases with the gate on and none with it off. Dropping the row-end check fails the ragged-M cases, while odd M = 199 passes because it takes the scalar fallback. - llama-bench, same binary with the gate off/on, ABBA x2, -r 2 (mean of 8): Flash-Next REAP-320 pp2048 -ub 2048 769.5 -> 796.3 t/s (+3.5%), Qwen3.6-35B-A3B pp2048 -ub 2048 2100.5 -> 2187.1 (+4.1%), Qwen3.8-27B pp512 -ub 256 430.9 -> 441.5 (+2.4%). Every gate-on sample beat every gate-off sample. - Flash-Next perplexity is identical per chunk with the gate on and off. Assisted-by: Claude Opus 5.5
MUL_MAT_ID followed by MUL with the router weights ([1, n_expert_used, n_tokens]) was fused only on the mat-vec path. At prefill the expert outputs were written as f32, read back by a separate MUL, scaled and written again. The int8 MMQ coopmat1 MUL_MAT_ID pipelines get a sixth binding holding the weights. Bit 1 of the hoist_row_ids push constant tells the epilogue to scale each output row by its (expert slot, token) weight before the store. The product is the same f32 multiply the separate MUL does, so results are identical. The fusion is admitted only when ggml_vk_mul_mat_id_q_f16 will take the int8 MMQ path (checked against the pipeline's binding count); other MUL_MAT_ID GEMMs keep the separate MUL. GGML_VK_MMID_EPILOGUE=0 disables it. The packed vec4 store path scales its four rows by the same column weight. test-backend-ops: 9 new MUL_MAT_ID_FUSION cases (q8_0, iq4_nl, q4_K; n_used 10/4; ragged and odd M). MUL_MAT_ID + MUL_MAT_ID_FUSION + MOE_PREFILL 7561/7561 with the gate on and off. Scaling by twice the weight in the new epilogue fails every GEMM-path fusion case with the gate on and none with it off. gfx1151 (RADV), pp2048 -ub 2048, same binary with the gate off/on, ABBA x2, -r 2 (mean of 8): Flash-Next REAP-320 775.7 -> 800.2 t/s (+3.2%), Qwen3.6-35B-A3B 2100.4 -> 2207.5 (+5.1%). Every gate-on sample beat every gate-off sample. Perplexity is identical in every chunk. Assisted-by: Claude Opus 5.5
…th chunk The compute buffer is split into chunks of at most the backend's max size (1 GiB on Vulkan), but the chunk list was a fixed array of 16, and the 16th chunk was made unbounded so it could take whatever was left. A compute buffer past ~16 chunks therefore put everything remaining in one chunk that can be larger than the device allows: Flash-Next at -c 262144 on a 128 GB Strix Halo needed a 19.5 GB compute buffer and failed on a 4.4 GiB chunk, over RADV's 4 GiB buffer limit (issue 22). The chunk lists of the allocator and of the virtual buffer now grow (16, then doubling), and every chunk stays at the max size unless one tensor alone is larger. Graphs that fit in 16 chunks are laid out exactly as before. test_not_enough_chunks now also checks that every chunk stays within max_size; it fails on the old layout. Assisted-by: Claude Opus 5.5
vulkan : GDN conv straight from x and the state, and a short-row RMS norm with two fusions
ggml-alloc : grow the chunk list on demand instead of an unbounded 16th chunk
vulkan : vec4 store for int8 MMQ on RDNA3, and MoE router weights in the MUL_MAT_ID epilogue
server : add --spec-draft-max-slots to stop drafting under concurrency
qsa : enable the incremental QSA cache on Vulkan
Two passes size the shared scratch (prealloc_x) from the current context: the flash-attention K/V dequant/contiguize copy (K + V in f16) and the fused QSA top-k (n_kv x rows F32). Both grow by one ubatch of cells with every prompt ubatch, and both sized the scratch to the exact need, so it was freed and allocated again on every ubatch: with Qwen3.8-Flash-Next and 174k cells in the cache, 341, 342, ... 348 MiB, once per ubatch. Rounding only one of them is not enough: the top-k runs first in each layer and sets the exact size, so the copy then finds the scratch large enough. Round both up to 64 MiB: one reallocation per ~64 ubatches. Assisted-by: Claude Opus 5.5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… buffer limit Follow-up to the 64 MiB rounding of the flash-attention K/V copy and the fused QSA top-k scratch (review on #118): - a need below 64 MiB keeps its exact size, so a model whose K/V copy is small holds no extra memory - the rounded size never exceeds max_buffer_size: a need that fits exactly no longer rounds up into a buffer the device cannot create - the top-k binds the scratch it uses (scratch_size), not the whole shared buffer, so its descriptor range stays within maxStorageBufferRange whenever the exact need does Assisted-by: Claude Opus 5.5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On master, Flash-Next takes the selected-key path on Vulkan (#119), where the QSA block selection is a GGML_OP_TOP_K over the KV cells. ggml_vk_topk sizes its double buffer in the shared scratch (prealloc_x) to the exact need, so the scratch was still reallocated as the context grew: 172, 173, 174, 176 MiB during one 4096-token prompt next to 174k resident cells. Size it with ggml_vk_scratch_size() like the other context-sized scratches; below 64 MiB (any ordinary top-k) nothing changes. Its passes bind dbl_buf_size subranges, not the whole buffer. Assisted-by: Claude Opus 5.5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Merge upstream llama.cpp (2026-10-08) and the final halo-box/llama.cpp master
quantize : choose quant types to hit a bpw or file size target (upstream ggml-org#15550)
vulkan : grow the context-sized scratch in 64 MiB steps
Collaborator
|
Please don't merge this 🙏 , I am working on a way to create better merge request for merging upstream |
quantize : choose quant types to hit a bpw or file size target (re-open of #156)
3 of 5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Re-opens #155, which was reverted in #168 because agents stopped working on master after it merged. The content is unchanged; this PR is where the breakage gets found and fixed before it lands again.
halo-box/llama.cpp is archived, so this fork now syncs with ggml-org/llama.cpp directly. This PR does that in two real merges, per SYNC.md:
Afterwards strix is 0 commits behind upstream. Merge with "Create a merge commit", not squash, or the upstream history is lost and the next sync conflicts on every commit again.
Draft until two things are done (see the end of this description): the remaining backend-ops runs pass, and someone runs Qwen3.8-Flash on HIP and Vulkan.
Resolution rules
MTP follows upstream's design. Everywhere else, strix's side wins where the two collide, upstream's non-conflicting changes stay, and upstream fixes are ported into strix's code. Strix contributors: please check how your areas came out.
qwen4exp / QSA
qwen4exp.cppand itsmodels.hstruct: selected-key/WMMA block selection, incremental pooled keys, PLE prefetch and the HIP prefill work.llama-memory-hybrid-idxcarries both APIs: strix'sqsa_*for qwen4exp, and upstream's k-pool for the new GLM5Next model (GLM-5.3-Flash). strix's contextset_input_qsa, which upstream had deleted, is restored.MTP
graph_mtp. It builds the same draft as upstream's Qwen4Exp: add MTP ggml-org/llama.cpp#29761: eh_proj of [enorm(e); hnorm(h)], the draft's ownnextn.hc_head_*mixer, sparse QSA attention. It also loads shared sidecars that ship without embeddings or an LM head.Recurrent state (
llama-graph)Upstream's single fixed-size state gather (ggml-org#29856) is combined with strix's in-place gated delta net. In-place-capable consumers keep the full gather, and their state view sits next to the consumer whether or not the rows are in place. In-place and gathered ubatches then build the same graph. Without that, every switch between them reallocated the graph: a speed cost in normal use, and an abort under
GGML_SCHED_DEBUG_REALLOCin both upstream's shared-seq test and strix's seq-cp graph-reuse test.CUDA / HIP
get_J_maxhelper upstream removed.Vulkan
flash_attn.comp,flash_attn_cm2.comp,lightning_indexer.compandtopk_nary_search.comp, with upstream's FA shared-memory out-of-bounds fix (vulkan: fix Flash Attention shmem write out of bounds ggml-org/llama.cpp#29988) ported in.Allocation
alloc_buffer_nrefactor replaces strix's backport of the view-init fix. Its planner flushed on zero-size tensors, which left views of a tensor larger thanmax_sizeuninitialised. That is fixed here, caught by strix'stest_view_init_after_max_size_split, and is worth upstreaming.Server and tests
common_batch/llama_process.Known regression to port later
None open. ggml-org#29856 was reverted in the first merge commit and restored in the second, so review the two together.
Verified
test-jinja-py, because the image has nojinja2.test-recurrent-state-rollback-dsv4: passes on both backends here, but fails intermittently, as it does on pristine strix (3 of 4 runs).test-backend-opsagainst CPU on ROCm0: TOP_K, ARGSORT, RMS_NORM, GATED_DELTA_NET, PAD and ROLL have no failures.Before taking this out of draft
test-backend-opsruns pass: MUL_MAT, MUL_MAT_ID and FLASH_ATTN_EXT on ROCm0; TOP_K, FLASH_ATTN_EXT, LIGHTNING_INDEXER and GATED_DELTA_NET on Vulkan0.--spec-type draft-mtp, including a shared-sidecar MTP file), compared against current master for prompt-processing and token-generation speed and output. Nothing has been benchmarked yet.🤖 Generated with Claude Code