qwen4exp: GGML_ABORT ("maskless selected-key attention was not taken by a QSA kernel") on KV-cache exhaustion under --kv-unified with concurrent slots #85

Activity

1. joshvoigts commented on Sep 30, 2026

   

   joshvoigts

   on Sep 30, 2026

   More actions

   I'm seeing the same error and crash. Seems to occur near the end (or possibly at the end of the maximum context).

   Build: strix-llama.cpp=12643af64 (master), rocm_systems=7dda3ac6cf (ilintar-experiments)

   gfx1151 / HIP, -c 262144 --kv-unified, n_slots=1, MTP draft (mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf), not --parallel.
2. klucsik commented on Sep 30, 2026

   

   klucsik

   on Sep 30, 2026

   More actions

   Second data point + the exact trigger band (and a proposed gate fix)

   Production, gfx1151 / HIP, build 12643af64 (= master at the time), -c 262144 --kv-unified -np 2 --flash-attn on --cache-type-k f16 --cache-type-v f16 --fit off, MTP+ngram-mod speculation:

   [47401] /opt/llama.cpp/ggml/src/ggml-cuda/fattn.cu:783: flash_attn_ext: maskless selected-key
   attention was not taken by a QSA kernel (node_762): q [256,853,24,1] f32 nb 24576/1024,
   k [256,262144,2,1] f16 nb 2/1024/512, v f16 nb 1024/512, ids [2051,853,1,1] nb 8204,
   src6 0 src7 0 p4 0 dst contiguous 1
   

   The child instance exited with status 1 and the multi-model supervisor respawned it — which is why the container restartCount stays 0 and the Pod never looks unhealthy. Slot occupancy at the abort: 153,781 + 108,067 = 261,848 of 262,144 cells used, i.e. two concurrent slots filling the unified cache from opposite ends.

   The gate and the kernels disagree

   qwen4exp_use_block_selection() (src/models/qwen4exp.cpp:812) admits the maskless route up to n_kv <= 16777216. The kernels that must execute the resulting op are far stricter, and they disagree with each other by 4 cells:

   kernel	depth bound	why	query cap
   qsa_prefill	k->ne[1] <= 262140, %4 == 0	whole blocks packed as uint16 with 0xFFFF reserved as the empty sentinel → 65535 * ratio(4)	q->ne[1] >= 128
   qsa_decode	k->ne[1] <= 262144	no block packing	q->ne[1] <= 512

   llama_kv_cache::get_n_kv() pads the used extent to 256 (n_pad_cur = max(n_pad, 256)), so a 262144-cell cache exposes exactly one depth past the prefill bound: every used count in 261889..262144 reports n_kv = 262144. The rung below it is 261888, which is inside the bound.

   fattn.cu tries decode, then prefill. So the fatal combination is narrow and fully determined:

   maskless route taken (n_stream == 1) AND n_kv == 262144 (cache padded onto the last rung) AND ubatch > 512 tokens → no kernel accepts the op → GGML_ABORT.

   Every prefill clause that the abort string exposes passes on our node (853 ≥ 128, ids->ne[0]=2051 ≤ 2560, ids->nb[1]=8204 so %4==0, k->nb[1]=1024/k->nb[2]=512 so %16==0, f16 K/V, head 256, 24 = 12×2 GQA, dst contiguous, src6/src7 null, p4 0, bias/softcap 0) — the only two clauses not printed are the q->data/k->data/ids->data alignment tests, which are satisfied in practice because these are graph-allocated buffers. So the op is 4 keys of depth away from being servable, and decode can't take it only because of the 512-query cap. Note 262144 % 4 == 0, so this isn't an artifact of the alignment clause: the two caps genuinely differ, and the llama-side gate is checked against neither.

   This also explains #85's own node — and predicts it is partially fixed

   #85 and @joshvoigts's report both show q=512 at depth 262144. On those builds (8c1c282ec, before 9bc06f5f7 "qsa : take 1..512 queries in the per-query decode kernel") the decode cap was 8, so a 512-token ubatch fell through both gates for the same structural reason. After 9bc06f5f7, decode accepts q <= 512 at 262144 — so I'd expect your exact node to be served now; worth re-testing on current master. The band that survives on master is q > 512, which is what we hit.

   Proposed fix: gate on the intersection of the kernel bounds

   --- a/src/models/qwen4exp.cpp
   +++ b/src/models/qwen4exp.cpp
   @@
   +// The selected depth must also fit the kernel that will run the op: qsa_prefill indexes whole blocks as uint16 and
   +// reserves 0xFFFF as the empty sentinel, so at ratio 4 it can address 65535*4 -> 262140 keys. qsa_decode is looser
   +// (262144), but it only takes up to 512 queries, so a deeper ubatch that lands past the prefill bound has no kernel
   +// at all and fattn aborts. Keep this in sync with the k->ne[1] bound in ggml/src/ggml-cuda/qsa-prefill.cu.
   +static constexpr int64_t QSA_MASKLESS_MAX_KV = 262140;
   +static_assert(QSA_MASKLESS_MAX_KV%4 == 0, "qsa_prefill requires a multiple-of-4 key count at its depth bound");
   +
    static bool qwen4exp_use_block_selection(bool blk_bias, int64_t n_stream, int64_t ratio, int64_t n_kv,
            const llama_ubatch & ubatch, const llama_cparams & cparams, const llama_hparams & hparams,
            ggml_type type_k, ggml_type type_v) {
        return blk_bias && n_stream==1 && ratio>1 && hparams.indexer_top_k%ratio==0 &&
   -        n_kv>hparams.indexer_top_k+ratio-1 && n_kv<=16777216 && ubatch.token &&
   +        n_kv>hparams.indexer_top_k+ratio-1 && n_kv<=QSA_MASKLESS_MAX_KV && ubatch.token &&

   Cost is one padded rung: used cells 261889..262144 served by a >512-token maskless ubatch fall back to the masked route (261888 is a multiple of 4, so prefill alignment is respected; decode at q <= 512 and depth 262144 stays on the QSA path). Because get_n_kv() quantises to 256, this is a single rung, not a sliding window.

   Hardening alternatives, if you'd rather not tighten the model-side gate:

   • raise the prefill bound to 262144 by padding the last pack and dropping the %4 requirement, or
   • align the two kernel caps so no reachable depth is accepted by neither, and assert the invariant (n_kv == 262144 ⇒ only q <= 512 may be maskless).

   Relation to #115 and #110

   • qwen4exp : use maskless block selection only where the selected-key kernels run #115 threads a selected_key_attn backend-capability flag into this exact expression, but its description states it "does not change ROCm, so it does not fix qwen4exp: GGML_ABORT ("maskless selected-key attention was not taken by a QSA kernel") on KV-cache exhaustion under --kv-unified with concurrent slots #85". On HIP the flag is always true, so a 262144-depth / >512-query ubatch still aborts. Since it already rewrites this line, it'd be a natural place to also cap n_kv at the selected-key depth bound.
   • llama : log the QSA maskless/masked route decision and why it was refused #110 turns the same condition into a logged route decision — our node's tensors are exactly what its refusal string would print. Complementary: llama : log the QSA maskless/masked route decision and why it was refused #110 tells the operator that the route failed, this prevents the abort and names the clause.

   Rebuild-free workaround

   -c 261888 (256-aligned, so GGML_PAD(n_ctx, 256) leaves it; n_ctx_seq == 261888 <= 262140 under --kv-unified). The cache can never report the fatal rung, for 0.1 % of context. Not yet applied on our side (we have no write access to the ConfigMap), so this is a proposed mitigation rather than a verified one.

   Repro recipe

   Two concurrent sessions on -c 262144 --kv-unified -np 2 --flash-attn on --cache-type-k f16 --cache-type-v f16 --fit off, n_swa == 0. Drive combined used cells past 261888 (we were at 261,848), then send a >512-token incremental prompt to the slot with the shorter prefix so it doesn't split into <=512 chunks. --ubatch-size 1024 makes it deterministic. Full 787 s log available.

   Confirmed present in 12643af64; fattn.cu, qsa-decode.cu and qsa-prefill.cu are byte-identical in current 8b7217718, and no branch in the repo relaxes the 262140 bound (0636c9aee introduced it).
3. klucsik

   added 4 commits that reference this issue on Sep 30, 2026

   llama : cap maskless block selection at the qsa_prefill depth bound (…

   ...

   fae04bd

   llama : cap maskless block selection at the qsa_prefill depth bound (…

   ...

   3a1c45d

   cuda : widen QSA selected-key block ids to uint32 and lift the depth …

   ...

   86bd254

   cuda : widen QSA selected-key block ids to uint32 and lift the depth …

   ...

   81b5f7b
4. gaetan-puleo

   added a commit that references this issue on Oct 11, 2026

   cuda : widen QSA selected-key block ids to uint32 and lift the depth …

   ...

   d7116c8