Repository navigation
Conversation
A small draft model prefills the prompt and decodes a few lookahead tokens. The softmax attention of those tokens over the prompt is captured with a ggml eval callback, smoothed, max-reduced over heads and layers and pooled into chunks. Only the top scoring chunks go into the target model's KV cache. This is lossy. Adds llama_set_eval_callback, common/speculative-prefill, the --spec-prefill-* options, the standalone example and the llama-server hookup. Not ported from that PR: the llama-bench changes and the python eval scripts. Measured on Strix Halo / Vulkan with a Qwen3.5-2B estimator, TTFT at p = 0.15: Qwen3.8-27B 5604 -> 1587 ms, Flash-Next FP4 4920 -> 1645 ms, and 54.9 -> 11.9 s on a 12k prompt. Decode rate is unchanged. See docs/speculative-prefill.md. Assisted-by: Claude Opus 5
Use 32 waves for the production non-KDA H32/S128 Gated DeltaNet shape on gfx1151. This reduces kernel latency by 1.34% and improves Qwen3.6 IQ3_XXS PP2048 by up to 0.97% while preserving exact four-corpus logits and decode output. Assisted-by: OpenCode
With 64 KiB of LDS per CU only 1-2 MMQ blocks are resident on gfx1151, so the global-load latency of every 256-wide K iteration is exposed (PMC: VALU busy 6%, 53% of wave-cycles waiting on data). Stage the next iteration's activation tile (and, for Q8_0, the weight tile) in registers while the current one is consumed. Same loads, same LDS layout, same arithmetic: outputs are bit-identical to the previous kernels. Whitelisted to the specializations where the extra registers do not spill: Q8_0 J48/J128, Q6_K J32, Q5_K J32, Q4_K J48, IQ2_S/IQ3_XXS J128 (all others keep the original loop; no spill regression in any kernel). test-backend-ops perf, MUL_MAT_ID 256 experts top-8, 2048 tokens (includes the quantize/ids overhead): Q8_0 gate/up 2717 -> 2551 us, Q6_K 3600 -> 3481, Q5_K 2571 -> 2510, Q4_K 2776 -> 2701, IQ2_S 3830 -> 3743, IQ3_XXS 3544 -> 3457. Also adds these production shapes to the perf test list.
Two exact-preserving changes to the gated delta net recurrence: * warp_reduce_sum compiled to 5 dependent ds_bpermute round trips per reduction (10 per token); on RDNA3/4 use v_permlanex16 + DPP row_xmask with the same xor-16/8/4/2/1 pairing, so results are bit-identical. * For the non-KDA S=128 prompt-processing path add a tiled kernel: q/k/g/ beta/v of a 16-token tile are loaded cooperatively into LDS and each warp owns 4 state columns, so k/q are read once per token for 4 columns and 4x fewer waves compete for issue slots (the kernel was issue-bound with ~50 VALU per token per wave for 4 state elements). FMA contractions are spelled out to match the original kernel's code generation. Bit-identical to the previous kernels on H=32/64, T=1/17/64/2048 (op-level dump comparison); test-backend-ops GATED_DELTA_NET 43/43. gfx1151, test-backend-ops perf: H32 T2048 5029 -> 2350 us, H64 T2048 9289 -> 4533 us, H32 T64 158 -> 93 us, T=1 13.3 -> 12.6 us. Together with the MMQ prefetch, Qwen3.6-35B-A3B PP2048 (llama-bench -b 2048 -ub 2048): Q8_0 +10.9%, UD-Q6_K_XL +9.4%, UD-Q4_K_XL +9.2%, UD-IQ3_XXS +7.6%.
Assisted-by: Codex Assisted-by: Claude Opus 5
The fused weighted expert sum used a volatile product to prevent FMA contraction and preserve the unfused graph's rounding. On AMD this spills every product to scratch. Use explicit v_mul_f32/v_add_f32 instructions to retain the exact operation order without scratch traffic, and process four adjacent channels per thread to amortize token indexing and weight loads. The fusion is restricted to contiguous 2048-wide Qwen tensors, so all float4 accesses are aligned. A direct old/new kernel comparison across 1,048,576 random outputs is bit-identical; MUL and ADD backend tests pass. gfx1151 weighted_expert_sum_f32<8>: 813 -> 689 us (-15.3%), scratch 8 -> 0 bytes. Qwen3.6-35B-A3B PP2048 improves by 0.35-1.36% across Q8_0, Q6_K_XL, Q4_K_XL, and IQ3_XXS.
… RDNA3.5 Token generation on gfx1151 launches ~1480 kernels per token with a ~2 us gap between each, so the quantize_q8_1 kernel in front of every MMVQ costs about as much as the quantization itself. Add mul_mat_vec_q_fq for the single-column Q8_0 path: a block of 16 waves (one output row per wave) quantizes the activation vector once into shared memory, replicating quantize_q8_1 exactly (same amax, IEEE division and roundf), and then runs the per-wave K loop of mul_mat_vec_q in the same order against it, with two K iterations of weight loads issued before the quantization and barrier. The two-kernel path is kept for other types, batched decode, and unaligned activations. Dense and MUL_MAT_ID (fused SWIGLU and down) outputs at all Qwen3.6 decode shapes are byte-identical to the previous path; MUL_MAT and MUL_MAT_ID backend tests pass. 16 waves per block and a prefetch depth of 2 were the fastest of the measured variants (8/32 waves, depth 1/4/8/16). Qwen3.6-35B-A3B TG128 on gfx1151: Q8_0 48.83 -> 50.99 t/s (+4.4%) at depth 0 and 42.88 -> 44.47 (+3.7%) at 32K; UD-Q8_K_XL 48.04 -> 49.61 (+3.3%) and 42.24 -> 43.38 (+2.7%). Adds Qwen3.6 batch-1 perf cases to test-backend-ops. Assisted-by: OpenCode
For a single decode token, every linear-attention layer runs 14 graph nodes between the qkv projection and the output gate: concat with the conv state, conv state shift into the cache, state gather, causal conv, SiLU, two L2 norms, alpha bias/softplus/A gate, beta sigmoid, the recurrence, the state copy into the cache, and the gated RMS norm. On gfx1151 these are 11 small kernels (~32 us) separated by ~2 us launch gaps, about 50 us per layer. Match this chain in the CUDA backend and run it as two kernels: one block per value head computes the conv + SiLU of its q/k/v channels (shifting the conv state straight into the cache), the L2 norms, gate and beta, and the recurrence with the state read from and written to the cache in place; a second tiny kernel applies the gated RMS norm (its destination may alias inputs of the first kernel that other blocks are still reading). Every op is spelled with the operation order and FMA contraction of the kernel it replaces, so the outputs, conv state and recurrent state are byte-identical to the unfused graph (checked in place and across cache rows). The fusion requires n_tokens == n_seqs == 1 and S == 128 and falls back otherwise, including the first token of a sequence where the state reset sits inside the chain. Qwen3.6-35B-A3B TG128 on gfx1151 (with the fused activation quantization): Q8_0 48.74 -> 54.18 t/s (+11.2%) at depth 0 and 42.62 -> 46.62 (+9.4%) at 32K; UD-Q8_K_XL 48.02 -> 52.69 (+9.7%) and 42.23 -> 45.71 (+8.2%). Assisted-by: OpenCode
ggml_cuda_op_weighted_expert_sum always staged its result in a pool buffer and copied it into the destination, because the graph allocator places the output over the expert or routing-weight buffers that die inside the fused range (in Qwen3.6 decode the eight routing weights sit inside the output). That copy is one more launch per layer in a decode step that is dominated by launch gaps. When the whole sum fits in one block (n_embd/4 * n_tokens <= 1024, i.e. decode), launch a single block and place a barrier between the reads and the store, so the output can be written in place whatever it aliases. The per-thread arithmetic is unchanged; larger grids keep the staging copy. Greedy 120-token generations with Qwen3.6-35B-A3B are identical. Qwen3.6-35B-A3B UD-Q8_K_XL TG128 on gfx1151: 52.69 -> 52.91 t/s. Assisted-by: OpenCode
The shared expert output is gated by sigmoid(w . y) with a 2048-wide f32 weight vector: a one-row mul_mat_vec_f, a sigmoid and the already fused mul + add + add, i.e. three launches per layer in decode. Match the five nodes and run them in one single-block kernel: the dot product repeats the per-thread fma chain and two-level warp reduction of mul_mat_vec_f<256>, the sigmoid is the same expression, and the elementwise tail uses explicit v_mul_f32/v_add_f32 like shared_mul_add_f32 (plain __fmul_rn/__fadd_rn are contracted into an fma by the HIP compiler). One block reads the whole activation vector before the reduction barrier, so the output may alias it. Outputs are byte-identical to the unfused graph. Qwen3.6-35B-A3B UD-Q8_K_XL TG128 on gfx1151: 52.91 -> 53.42 t/s. Assisted-by: OpenCode
mul_mat_vec_f loops over K with a runtime trip count, so the compiler keeps one 8-byte load per lane in flight and waits for it every iteration. The f32 MoE router (2048 x 256) of Qwen3.6 ran at 131 GB/s on gfx1151 because of this. For the single-column, unfused f32 path on HIP, load four K iterations of weights and activations before consuming them, in the same accumulation order; the remainder loop is unchanged. Outputs are byte-identical. gfx1151 Qwen3.6-35B-A3B decode: router matvec 16.0 -> 10.5 us, alpha/beta projections 3.2 -> 2.6 us; UD-Q8_K_XL TG128 53.42 -> 54.16 t/s. Assisted-by: OpenCode
mul_mat_vec_f picks a 256-thread block for K=512, so every wave issues a single 4-byte bf16 load and then reduces; on gfx1151 the bf16 MoE down projections of Unsloth UD quants (8 x 2048 rows of K=512) run at ~145 GB/s. Add a wave-per-row kernel that emulates the 256-thread block: lane l accumulates the eight virtual warps' lane l in the same fma order, reduces each with the same butterfly and combines them exactly like block_reduce over lanes 0..7 (including the zero-lane adds), with eight loads in flight per lane. Used on RDNA for unfused single-column bf16 with K <= 512; for longer rows the lower wave count loses, so the standard kernel stays. Outputs are byte-identical (dense and MUL_MAT_ID, odd shapes included). gfx1151 Qwen3.6-35B-A3B UD-Q8_K_XL: bf16 down projection 93 -> 67 us, TG128 54.16 -> 54.54 t/s. Assisted-by: OpenCode
Extend mul_mat_vec_q_fq beyond Q8_0 with a generic path that runs the per-wave vec_dot loop of mul_mat_vec_q<type, 1> against the shared-memory activations, and replicate the Q8_1 block sum of quantize_q8_1 in the exact butterfly order for types that read it. Q6_K (scale only) is enabled: it is byte-identical (dense and MUL_MAT_ID) and measured +0.8% on Qwen3.6-35B-A3B UD-Q6_K_XL TG128 (56.64 -> 57.09 t/s). Q4_K/Q5_K are exact as well but neutral without weight prefetch, so they stay on the two-kernel path. Assisted-by: OpenCode
…NA3.5 For mul_mat_id the MMQ tile width J is chosen from ncols_max, which is the total number of token-expert pairs, so the generic path always picks J=128. With 256 experts and 8 active the per-expert row count is only n_tokens/32 (16/32/64 rows for ubatch 512/1024/2048), so 50-87% of every tile was padding. The RDNA3.5 specializations that fixed this were gated on exact shapes and on 32/64 rows per expert, i.e. only ubatch 1024/2048, and only Q8_0/Q6_K used the routed-compact kernel. Replace the shape-specific paths with a per-type rule on rows per expert, and route all of them through mul_mat_q_routed_compact: Q8_0: 16/32/64 rows -> J=48 Q4_K/Q5_K/Q6_K: 16 rows -> J=32, 32 rows -> J=48, 64 rows -> J=32 Values measured on gfx1151 for the 2048x512 and 3072x1024 expert shapes; the previous choices (Q8_0 J48, Q6_K J32 at 64 rows) are unchanged, Q4_K at 64 rows moves from tiled J48 to compact J32 (-7% up-proj, -31% down-proj), Q5_K at 64 rows from tiled J32 to compact J32 (-31% down-proj). MUL_MAT_ID kernel time, 256 experts, 512x2048 / 2048x512, n=512 tokens: Q8_0 2347/2490 us -> 1517/1518 us Q6_K 3918/3931 us -> 1344/1300 us Q5_K 2606/2607 us -> 1087/1084 us Q4_K 2561/2538 us -> 1026/ 991 us n=1024: Q6_K -54%, Q5_K -50%, Q4_K -54% (Q8_0 unchanged). Qwen3.6-35B-A3B on gfx1151, llama-bench -b 2048 -ub 2048: UD-Q8_K_XL PP512 1437 -> 1722 t/s (+19.8%); PP1024/2048/4096 unchanged UD-Q4_K_XL PP512 1439 -> 2086 t/s (+44.9%), PP1024 +25.5%, PP2048 +3.5% UD-Q8_K_XL default ubatch 512: PP512 +22.1%, PP2048 +22.4% TG unchanged. test-backend-ops MUL_MAT_ID: 906/906 passed (adds 16/32 rows-per-expert eval cases and qwen3.6 perf cases).
The previous rule only matched exactly 16/32/64 rows per expert (ubatch
512/1024/2048 with 8 active experts); any other ubatch size, and the
remainder ubatch of every prompt, still fell back to the generic J=128 tiled
grid, which launches ncols_max/J*n_experts mostly-empty blocks.
Use ranges instead, and allow the routed-compact kernel for every selected J:
Q8_0: rows <= 12 -> J16, <= 64 -> J48, else J128
Q4_K/Q5_K/Q6_K: rows <= 12 -> J16, <= 24 -> J32, <= 48 -> J48,
<= 64 -> J32, else J128
The 16/32/64 choices are unchanged from the previous commit. Ranges were
picked from a J sweep over 64..4096 tokens on gfx1151 and validated
end-to-end.
Qwen3.6-35B-A3B on gfx1151, -b 2048 -ub 2048 unless noted, vs cutoff:
PP128 PP256 PP384 PP768 PP1536 PP4096@ub4096
UD-Q8_K_XL +39.7% +61.1% +21.5% +17.2% +12.4% +3.9%
UD-Q4_K_XL +93.5% +75.9% +60.1% +39.1% +18.4% +2.4%
(PP512/1024/2048 as in the previous commit.)
test-backend-ops MUL_MAT_ID: 914/914 passed (adds 4- and 128-row cases).
On gfx1151 the D=256 flash-attention path always fell back to the tile kernel, which is FMA-bound (~8 TFLOPS) and dominates prefill time at deep context (68% of PP time at 32k, 81% at 64k for Qwen3.5-0.8B). - Enable AMD WMMA device code for DKQ=256 (previously DKQ>128 was excluded). - Add a RDNA ncols=64 D=256 config tuned for the 256-VGPR budget: nwarps=8, nstages=1, nbatch_fa=32, nbatch_K2=128, nbatch_V2=64, nbatch_combine=64. Naive settings spill heavily and lose to the tile kernel. - Dispatch to MMA only when Q->ne[1]*gqa_ratio_eff > 32; the tile kernel is still faster for small batches (nb 4/8). - Fix a latent tile_Q reuse race in the combine loop: when DV/2 > nbatch_combine, warps read tile_Q columns written by other warps and the buffer is rewritten by the next batch without a barrier. Qwen3.5-0.8B UD-Q8_K_XL, gfx1151, PP512: depth 0: 9361 -> 9473 t/s (+1.2%) depth 32768: 3211 -> 4105 t/s (+27.9%) depth 65536: 1927 -> 2632 t/s (+36.6%) TG unchanged. test-backend-ops FLASH_ATTN_EXT: 2939/2939 passed.
Co-authored-by: Marshall <assistant@llama.cpp> Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Marshall <assistant@llama.cpp>
Catches up 68 commits from the parent fork. strix/master 732484c halobox/master ca062e9 Two conflicts, both the same shape: our side of the hunk was empty and the parent adds lines, so the incoming text is taken as it is. ggml/src/ggml-backend.cpp one ggml_is_quantized() condition ([TAG_ALLOC_SIZE_EXPAND]) ggml/src/ggml-metal/ggml-metal-tuning.cpp Metal M2 tuning tables flowing through from upstream; irrelevant to this part but harmless Compile-checked (llama-server, CPU backend) against exactly these two bases. This fork's own eight commits are untouched: the RDNA3.5 batched mat-vec chunking, the Q8_1 matvec fix, DFlash draft block widths, both reasoning-budget commits, the CI change and the docs. Note this only reaches the parent fork's level. halo-box/llama.cpp is itself 80 commits behind ggml-org and its own catch-up is blocked on a top-k reconciliation: halo-box#17 and upstream have independently implemented radix top-k for large k, with different shaders, different binding counts (2 vs 5) and different push-constant structs, and upstream has since deleted ggml_vk_topk_small entirely. Claude-Session: https://claude.ai/code/session_01JDKcT3SjBYaJKmpRJqTPmq
This reverts the Vulkan part of "Halo/vulkan topk radix (halo-box#17)" (ca062e9), which ported the EngramHalo HIP radix kernel as topk_radix.comp, so that the next commit can carry upstream's top_k radix select (ggml-org#28032) unchanged. One implementation of the op, the one upstream settled on, keeps this tree closest to upstream and lets the deterministic slot assignment that follows apply as-is. The two shaders solve the same problem; the difference that matters for Qwen 3.8 Flash-Next is order: topk_radix.comp assigns output slots with atomicAdd, so the selected indices come out in a scheduling-dependent order, and the sparse attention sums them in that order (see the deterministic-scan commit for the measured effect). Upstream's variant also carries the QSA indexer fusion. Co-authored-by: Claude (Opus 5)
…l-org#28032) * vulkan: add top-k radix sort shader for k >= 1024 * add Qwen 3.8 Flash Next top-k tests * add top-k qsa fusion * clean up code (cherry picked from commit daef7b6)
The radix select (ggml-org#28032) assigned output slots with atomicAdd, so identical runs emitted the selected indices in a scheduling-dependent order and, at the tie boundary, could drop a different tied element. On Qwen 3.8 Flash-Next the sparse attention sums the selection in that order, the ULP noise cascades through the QSA layers, and identical greedy requests above 2051 prompt tokens diverge (32k prompt x4: 2 to 3 distinct on every unfixed build, upstream included). Replace both atomicAdd slot counters with a per-chunk subgroup-ballot exclusive scan, walking the columns in ascending index order: the output order is fixed (ascending, which also helps gather locality) and the tie fill keeps the lowest-indexed tied elements. The histogram atomicAdd is unchanged. Verified on gfx1151: test-backend-ops TOP_K 453/453; 32k x4 byte-identical (was 2 to 3 distinct); 1024 x16 byte-identical; layer 3 and layer 47 top-k dumps identical in set and order across runs; prefill 337.5 vs 339.6 t/s and decode 26.3 vs 26.2 t/s at 16k depth (within noise). Co-authored-by: Claude (Opus 5) (cherry picked from commit fe6620c88c599b2f3a6e5ff4a6b99bb4773be38e)
The GDN q/k normalization is defined by flash-linear-attention as
l2norm(x) = x * rsqrt(sum(x*x) + eps)
with eps inside the root. Every GDN call site in the tree uses ggml_l2_norm
instead, which is x / max(sqrt(sum(x*x)), eps), i.e.
torch.nn.functional.normalize - its CUDA kernel cites that page.
The clamp never engages at these magnitudes, so in practice llama.cpp
normalizes with no epsilon at all where the reference has one inside the
root.
transformers made the same substitution when it first added Qwen3-Next and
corrected it three days later in huggingface/transformers#40842, 'Fix the
misalignment between the l2norm in GDN of Qwen3-Next and the implementation
in the FLA library'. vLLM and SGLang vendor FLA rather than reimplementing
it, so neither ever had the clamp.
eps keeps coming from the checkpoint, exactly as every call site already
passed it. The references hardcode 1e-6 for this norm; that is a separate
question and the two agree on every GDN checkpoint in the wild.
ggml_l2_norm itself is correct and unchanged, as is rwkv7-base, its original
caller, which passes normalize's own default eps of 1e-12.
No new ggml op: rms_norm already carries eps inside the root, so
rms_norm(x, eps/n) * (1/sqrt(n)) is exactly x * rsqrt(sum(x*x) + eps).
(cherry picked from commit 0dd72fd)
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> (cherry picked from commit fe46328)
On gfx11 (RDNA3) WMMA, x + (-0.0) is not exact in f16, so a fully masked flash-attention column with P == +0.0 still leaks the sign of whatever V the cell last held into the accumulator: the output of a request depends on what the previous request left in the cache. On a 1024-token prompt with 16 identical greedy requests, Qwen2.5-7B (f16 KV) alternates between two continuations and Qwen3.8 Flash-Next gives 5-6 distinct ones. Rather than patching the shader (which costs 8-18% dense prefill at depth on gfx1151 by its mere presence in flash_attn_cm1.comp), keep the invariant that a free cell is always zero: the KV buffers are cleared at construction, and every cell that becomes free is memset again, in every layer, when it is freed (seq_rm, seq_keep; clear() wipes up to a per-stream high-water mark, and stream copies mirror the mark). Caches that mirror another cache's cells (the Flash-Next indexer) register with the owner and are zeroed with it. On UMA the Vulkan backend's tensor memset is a host memset; it happens once per free, never per token, and never as a graph node, so graph reuse cannot replay it. Not covered: masked-out cells that belong to another sequence in a unified multi-sequence cache; only a shader-side fix covers that case. Gate (16 identical greedy requests, 1024-token prompt, 129 tokens, per-position top-8 logprob streams compared): 16/16 identical; the fresh-server output is unchanged byte for byte. Co-authored-by: Claude (Opus 5)
…-09-03 Merge halo-box/llama.cpp master (catch up 68 commits)
…-master Correctness: repeatable output on gfx1151 (freed-cell zeroing, deterministic top-k), plus two upstream picks
…ts (halo-box#37) Co-authored-by: Yoshinao Kadowaki <nuts.kado@gmail.com>
The arithmetic UE4M3 half-scale decode (non-LUT branch) accepted bytes 0x7f-0xff and produced garbage scales, while the CPU decoder (rocmfp4.c) returns 0 for them. Loader validation of those bytes only runs with --check-tensors, so a crafted GGUF decoded differently on GPU and CPU. Guard x > 0x7e -> 0 in both branches to match CPU. Verified bit-identical to the CPU LUT decoder for all 256 scale byte values; nvcc sm_89 compile clean. No behavior change for valid bytes, so existing test-backend-ops results stand. Assisted-by: Hermes Agent (Qwen3.8-Flash-Next)
Resolve tests/CMakeLists.txt by keeping both the qwen35moe and qwen4exp test-recurrent-state-rollback ctest entries. Assisted-by: Claude Opus 5.5
llama : qwen35/qwen35moe GDN graph - 2-D output projection, in-place recurrent state, masked h_nextn fix
…p4-mmq cuda: Q4_0_ROCMFP4 + Q4_0_ROCMFP4_FAST support and MMQ tiles on HIP (gfx1151)
…-29 halo sync Keep ROCmFP4 FAST and MXFP4/NVFP4 W4A4 MMQ declarations. Keep explicit rollback registrations, including Qwen4exp, without duplicate directory runs. Combine the status/all-model harness with scaled-weight seq_cp/graph-reuse coverage. Assisted-by: Codex
…ndency pass Pass the inclusive final node index instead of the exclusive group end. This keeps allocation dependencies in the graph when the group is terminal. Assisted-by: Codex
PR halo-box#75 gated the BF16 WMMA matmul path (mmb) to LLM_ARCH_QWEN4EXP only, which fixed real prefill regressions on some models but also removed the path from models where it is a large win (e.g. Qwen3.8-27B Q8_0). Only Q8_0 and IQ4_NL have a dedicated fast decoder in mmb_dispatch_quant (the special int constants 1 and 0); every other quant falls back to the generic decode. So the benefit tracks the weight quant type, not the arch. This keeps the path opt-in for all architectures but restricts mmb_quant_type() to the quants that actually have a fast decoder, which removes the regressions PR halo-box#75 addressed while keeping the Q8_0 wins. Strix Halo (gfx1151), pp2048, depth 0, -b 2048 -ub 2048 (llama-bench): model / quant mmb off mmb all this Gemma 4 26B Q8_0 2082 2452 2316 Qwen3.8-27B Q8_0 213 499 453 Qwen3.8-27B IQ3_S 378 301 382 Signal-3.8-27B Q4_K_XL 411 276 411
…9-29 Merge halo-box/llama.cpp master into strix (2026-09-29)
…gate hip: gate the MMB matmul path by quant type, not architecture
Collaborator
|
Any reason you are still capping it with this PR? |
server : add --cache-disk for a persistent prompt cache
klucsik
force-pushed
the
fix/qsa-maskless-nkv-cap
branch
2 times, most recently
from
October 2, 2026 06:58
3a1c45d to
81b5f7b
Compare
…bound to 1M The maskless selected-key route was capped at 262140 keys because qsa_prefill stores whole-block ids as uint16 and uses 0xFFFF as the padding sentinel: 65535 real blocks * ratio 4 = 262140, and at n_kv = 262144 the 65536th block id collides with the sentinel. A maskless ubatch above 512 tokens at any depth past 262140 has no kernel that accepts it, so fattn aborts the instance (halo-box#85, and the repeated aborts at k [256,265984,2,1] under -c 524288). Store the union's block ids as uint32 with 0xFFFFFFFF as the sentinel and raise the bound to GGML_QSA_MAX_KEYS = 1048576, the max context the Qwen3.8-Flash-Next card supports with RoPE/YaRN. This replaces the earlier cap-the-gate approach proposed in review on this PR: the fatal band disappears entirely and per-slot context can go past 262140 without falling back to the masked route. - qsa-prefill.cu: ublk becomes uint32 (umask stays uint16: 4 bits per query slot for QSA3_G=4 queries over ratio=4 keys). The descriptor load now reads one uint4 per chunk instead of a uint2 of two packed id-halves; cap is always a multiple of 4, so (g*cap + 4c)*4 stays 16B-aligned. blk_of() tests the sentinel per element; pad slots still resolve to block 0 with no mask bits and are discarded exactly as before. - qsa-decode.cu: depth bound follows the same constant. Its ids are gathered directly (no block packing); the per-row unique table is sized by ids->ne[0] (<= 2560), not by nk, so depth was never structurally limited. - common.cuh: one shared GGML_QSA_MAX_KEYS so the two kernel gates cannot drift apart again; qwen4exp.cpp mirrors it (llama/ must not include ggml-cuda). - qwen4exp.cpp: QSA_MASKLESS_MAX_KV = 1048576 replaces the bogus 16777216, which was checked against neither kernel bound. Verified on the host only: the CPU build of libllama/ggml compiles, test-qsa-prefix and test-qsa-fast pass, and a host harness reproducing the writer/reader lines proves the old and new encodings agree for every id < 65535, that the old one mis-serves 65535/65536/66495 (sentinel aliasing and truncation) where the new one is exact, that padding still resolves to a discarded block 0, and 20000 randomised lists round-trip. NOT run on gfx1151: no ROCm/HIP toolchain here, so the kernel itself is unbuilt and unmeasured. Expect +16 MiB of pool per ubatch at ids=2051, n_q=4096 (uint32 ublk doubles that buffer). Assisted-by: Piclaw (Qwen3.8-Flash)
klucsik
force-pushed
the
fix/qsa-maskless-nkv-cap
branch
from
October 2, 2026 07:00
81b5f7b to
86bd254
Compare
klucsik
changed the title
Author
Good diea, I updated the PR, name and description included. Now it touches more files tho. |
klucsik
force-pushed
the
fix/qsa-maskless-nkv-cap
branch
from
October 2, 2026 08:09
86bd254 to
3a1c45d
Compare
klucsik
force-pushed
the
fix/qsa-maskless-nkv-cap
branch
from
October 2, 2026 08:24
3a1c45d to
86bd254
Compare
klucsik
marked this pull request as draft
October 2, 2026 09:52
Collaborator
|
Are you going forward with this PR? if yes, please rebase it and move it to open. |
gaetan-puleo
added a commit
that referenced
this pull request
Oct 11, 2026
…te, HC flip, gather shrink Maintainer review findings on this campaign PR, acted on: - qsa-prefill/qsa-decode: restoring the base 262140/262144 ceilings. The ruled-16-bit block ids are the real limit: past 262140 keys block 65535 doubles as the padding marker and higher blocks wrap to the start of the cache, silently corrupting attention past that depth. Widening the ids is #124's job (already open, and it lowers the model-side limit to 1M); this 994k run was survival-throughput only, with no output-quality gate. - server-context.cpp: LLAMA_SERVER_ALLOW_CTX_BEYOND_TRAIN removed. The cap exists to gate un-scaled over-context runs, and only those reach the item-1 corruption path. YaRN configurations do not need the env (base raises n_ctx_train to yarn-orig-ctx / freq_scale). - getrows.cu/dequantize.cuh: the 32-grain kernel is withdrawn; supports_op for IQ4_NL returns to the base QK_K gate. The offloaded PLE gains the campaign claimed for it were the file-mmap-to-pinned-host move that item 1 (staging) already covers, and the gather buffer was at most tens of MiB (the real 3.3 GiB runtime growth is traced to ggml_backend_sched_new's hv_tensor_copies allocation with the server's per-slot sampling reserve, not the kernel work). - qwen4exp.cpp: the embd_buf watermark is dropped with it; capacity never regrew past ~40 MiB. - tools/server/README.md: the env-knob table no longer splits the generated options table; non-ASCII removed; LLAMA_SERVER_ALLOW_CTX_ BEYOND_TRAIN row deleted. Kept from the campaign: the mmap pin-staged loader fix (the HSA busy-wait crash), the pool-free knob (with a large-threshold note: frees during HIP graph capture should be avoided at small thresholds), the RAII staging owner, and the env-knob docs. Assisted-by: OpenCode (GLM)
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
17 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
This replaces the cap-the-gate approach this PR originally shipped, per review: instead of routing the fatal band
to the masked path, lift the kernels so the band disappears. Closes #85.
qsa_prefillstored whole-block ids asuint16and used0xFFFFas the padding sentinel, so the servable spacewas
65535 blocks * ratio 4 = 262140keys — and atn_kv = 262144the 65536th block id collides with thesentinel outright. Any maskless ubatch with >512 queries past that depth had no kernel that accepted it, and
fattn.cuaborted the instance. That is no longer theoretical: with-c 524288 -np 1running today, the abortreproduces at
k [256,265984,2,1],q [256,4096,24,1]— four child respawns in 20 minutes on the box.The fix widens the union's block-id list to
uint32(0xFFFFFFFFsentinel) and raises the bound toGGML_QSA_MAX_KEYS = 1048576, the maximum context the Qwen3.8-Flash-Next card supports with RoPE/YaRN:qsa-prefill.cu:ublkbecomesuint32(umaskstaysuint16— it is 4 bits per query slot, unchangedsemantics). The per-chunk descriptor load reads one
uint4instead of auint2of two packed id-halves;cap = (QSA3_G*ns+3)&~3is always a multiple of 4, so(g*cap + 4c)*4stays 16 B-aligned.blk_of()tests thesentinel per element; pad slots still resolve to block 0 with no mask bits and are discarded exactly as before.
qsa-decode.cu: depth bound follows the same constant. It gathers keys directly (no block packing) and itsper-row dedup table is sized by
ids->ne[0] <= 2560, not bynk, so it never had a structural reason for 262144.common.cuh: one sharedGGML_QSA_MAX_KEYSso the two kernel gates cannot drift apart again (the 262140/262144disagreement is what created the abort band).
qwen4exp.cpp:QSA_MASKLESS_MAX_KV = 1048576replaces16777216, which matched neither kernel bound(llama/ mirrors it as a literal because it must not include ggml-cuda headers).
Net effect: the fatal band disappears, and depths above 262140 keep the maskless fast path instead of falling back
to the masked route.
Measurements
Baseline: not measured (no HIP toolchain in the authoring environment).
After: not measured.
What WAS verified, on the authoring host (x86 Debian container, g++ 14):
-DGGML_HIP=OFFbuild oflibllama/ggmlcompiles clean (model-side gate is backend-neutral).test-qsa-prefixandtest-qsa-fastPASS (unchanged behaviour of the prefix/indexer layer).qsa-ids-equiv.cpp) replicating only the touched writer/reader lines proves, 7/7 PASS:the new
uint4descriptor load is 16 B-aligned for every validcap; old and new encodings agree for everyid < 65535 (so all currently-working configs are bit-identical); the old encoding mis-serves
65535/65536/66495(sentinel aliasing and truncation — the live crash range) where the new one is exact;padding still resolves to a discarded block 0; 20 000 randomised lists round-trip.
Cost note:
ublkdoubles (uint16 → uint32): +16 MiB of pool per ubatch atn_q=4096, ids=2051(32 MiB vs 16 MiB).
umaskunchanged. At 1M depth the prefillpackpath (QSA_DIRECT=0) would allocate~2 GiB of packed K/V; the auto policy (
n_q*32 < nk) picks DIRECT for realistic ubatches, so this is the samebehaviour as at 256K, just further out.
Correctness: on-device
test-backend-opsand allama-perplexitybefore/after are needed from thehardware owner. Expect zero numeric drift below 262140 keys by the equivalence test above; ids ≥ 65535 are new,
previously-aborting territory.
Additional information
3a1c45d64) capped the model-sidegate at 262140 instead — same crash prevented, but it forfeits maskless throughput above 262140 keys and cannot
ever serve the card's 1M context. Say the word and I can restore that as the shape of this PR.
q=512case (now served by decode at any depth ≤ 1M) and theq>512case(now served by prefill at any depth ≤ 1M).
-c 524288 -np 1(n_ctx_slot = 524288, kv_unified = 'false'in the load log),maskless ubatch
q=4096at padded depthk->ne[1]=265984-> 4 aborts in 20 minutes, children 33573/58813/38011/36905.The earlier shape
-c 262144 --kv-unified -np 2(qwen4exp: GGML_ABORT ("maskless selected-key attention was not taken by a QSA kernel") on KV-cache exhaustion under --kv-unified with concurrent slots #85, and our 2026-09-30 incident) hits the same clause once thetwo slots together pass 261889 used cells. Both are the same four-cell gap; both are served after this change.
k->ne[1] > 65535*ratiolimit also implicitly guarded theqsa3_merge_kernelblock-count math; block countsat 1M are
262143, well insideint32, and the sentinel at0xFFFFFFFFcan no longer alias a real id.Requirements
belong in halo-box/llama.cpp instead
(Qwen3.8-Flash); the human owns the PR, built nothing yet, and will run the hardware verification. Commit
carries
Assisted-by: Piclaw (Qwen3.8-Flash).container) — no llama-bench, no on-device test-backend-ops, no perplexity, no baseline comparison. The
uint32 descriptor path is proven by a host-replica equivalence harness only. Vulkan/CUDA backends untouched
and untested. YaRN/1M context correctness is model-card territory, not validated here. The +16 MiB/ubatch
pool growth is computed, not measured. Requesting a gfx1151 build + the two tables from the hardware owner.