Misc. bug: draft-MTP acceptance 0.0 on qwen4exp at prompt lengths >= ~4k tokens (0.75 shallow, output unaffected)

[ChrisC381]

Name and Version

Built from the pwilkin strix-halo stack (merged into this repo by #63) at two
pins, both affected:

• pwilkin/llama.cpp branch strix-halo @ be905cf7 (Sep 13 2026 build)
• pwilkin/llama.cpp branch strix-halo @ d67d5883 (Sep 14 2026 build;
  includes the sparse QSA decode rewrite and the incremental indexer state
  with cache-lifecycle invalidation hooks)

Rest of the stack: installer pwilkin/strix-halo @ fa169251,
pwilkin/rocm-systems @ 7dda3ac6c (custom ROCr/HIP runtime), SDK ROCm
7.14.1. Hardware: Radeon 8060S (gfx1151) on Ryzen AI Max+ 395, 128 GB unified
memory, Linux 7.0.0-31-generic.

Operating systems

Linux

Which llama.cpp modules do you know to be affected?

• libllama (core library)
• llama-server

Command line

llama-server \
  -m <qwen4exp target gguf> \
  -md <shared-MTP draft gguf, Q8_0> \
  --spec-type draft-mtp --spec-draft-device ROCm0 --spec-draft-ngl 99 \
  --spec-draft-n-max 2 --spec-draft-p-split 0.10 \
  -dev ROCm0 -ngl 999 -fa on -fit off \
  --load-mode none --lazy-mode on-direct \
  -ctk f16 -ctv f16 \
  -c 262144 -b 16384 -ub 16384 --parallel 1 --metrics

-ctk/-ctv f16 is forced by the QSA kernels (the assert at
qwen4exp.cpp:1309 rejects non-F16 KV). Env vars come from the launcher
profile: HSA_OVERRIDE_GFX_VERSION=11.5.1,
GGML_HIP_ENABLE_UNIFIED_MEMORY=1, ENABLE_RETAINED_PM4=1.

Client side: POST /completion with temperature 0, n_predict 256, a prompt of
N tokens, then read the delta of llamacpp:spec_decode_num_draft_tokens_total
/ llamacpp:spec_decode_num_accepted_tokens_total on /metrics across exactly
that request (the counters are process-lifetime cumulative, so only deltas
are meaningful).

Problem description & steps to reproduce

With a qwen4exp target and the shared-MTP draft, acceptance is 0.0 whenever
the prompt is >= ~4k tokens (threshold not bisected between 200 and 4096);
shallow prompts accept normally. Greedy output is unaffected (canary hashes
byte-identical across spec / no-spec legs at both pins), so this is an
efficiency bug, not corruption: at depth, spec decode runs below the no-spec
control.

Acceptance vs prompt length, same server and session (pin d67d588,
n-max 2; acceptance identical at be905cf):

prompt tokens	acceptance	tok/s (spec)	tok/s (no-spec control, be905cf)
~200	0.754 (765/1015)	37.5	26.0
4096	0.0 (0/2535)	15.2	24.4
32768	0.0 (0/2535)	15.0	20.8
131072	0.0 (0/1521)	14.5	14.1

What we ruled out:

1. Not prefix-cache restore. cache_prompt=false at 4k on d67d588:
   accept 0/251 over 128 generated tokens.
2. Not n-max. With n-max 3 acceptance is 0.0 at every depth including
   shallow; dropping to n-max 2 restores shallow only.
3. Not the draft weights. The draft file is the flash-next profile pin
   mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf, sha256
   5ff54097406a905cf3a724c709124ceb0e3e10235ee862298969e91c96fa96e6,
   byte-identical to the installer pin. The same draft file with the same
   target on a different llama.cpp build (Vulkan backend) accepts 816/1020
   shallow, and 955/955, 955/955 and 573/573 at 4k/32k/128k.
4. Not the engine's spec plumbing at depth in general. The published
   27B IQ4_XS + DFlash2 pair through the same engine at a 31.5k prompt:
   accept 218/220. That pair uses the DFlash2 draft path, not shared-MTP.
5. Not output corruption. Greedy canary sha256 identical across spec /
   no-spec / n-max 2 legs on this engine (a73e540c...).
6. Not a stale build. Identical at be905cf and d67d588, i.e. across
   the sparse QSA decode rewrite and the cache-lifecycle invalidation hooks.

One cell we did not run: this engine + shared-MTP + your own PROJFIX sharded
quant (93 GiB download). Our target was an own quant of the same public
weights, so a target-quant interaction is not fully excluded. But the draft
file is exactly yours and the same pair accepts normally at shallow length,
so the likelier cause is the qwen4exp shared-MTP path at length, not the
draft weights.

First Bad Commit

Not bisected. Present at both be905cf and d67d588.

Relevant log output

Server's own timing line for the cache_prompt=false 4k probe (d67d588),
confirming the same reading from the server side:

Sep 14 09:29:30 ... slot print_timing: id 0 | task 4148 | draft acceptance = 0.00000 (  0 accepted /  251 generated), mean len =  1.00

---

AI disclosure (per AGENTS.md): assembled by an AI agent (Hermes, GLM)
from saved run artifacts on the machine above; every number is quoted from
those artifacts; the one untested cell is named; the operator reviewed this
report before filing.

[rcfa]

A data point from the Vulkan side, on this fork's own master build, which may help narrow this down: with the
shared-MTP draft, acceptance does not collapse at depth here.

Setup - note the differences from yours: a Radeon 890M (gfx1150, Strix Point, not gfx1151), Vulkan (RADV,
Mesa 26.1.6) instead of ROCm, and the Q4_K_M shared draft instead of Q8_0.

GGML_VK_MMV_NO_SPLIT=1 llama-server -m Qwen3.8-Flash-Next-UD-IQ4_XS-00001-of-00003.gguf \
    -ngl 99 --load-mode none --lazy-mode on-direct -c 131072 \
    --spec-type draft-mtp -md mtp-Qwen3.8-Flash-Next-shared-Q4_K_M.gguf --spec-draft-n-max 2 -ngld 99

Fork 8c1c282ec; defaults otherwise (4 slots, unified KV, f16 KV cache, no -fa flag).
GGML_VK_MMV_NO_SPLIT=1 is for an unrelated small-batch issue on gfx1150 (#88).

Method - chat-formatted prompts (/apply-template, then tokenized) of licence text at increasing length,
/completion with temperature 0 and 200 generated tokens; acceptance is read from each response's own
timings.draft_n / timings.draft_n_accepted, so it is per request, not a cumulative counter.

prompt tokens	computed (rest from cache)	acceptance	decode, MTP	decode, no MTP
1066	1066	0.80 (104/130)	15.19 t/s
4066	3516	0.59 (107/182)	11.78 t/s
16065	12515	0.67 (113/169)	11.30 t/s	8.30 t/s
16065	16065 (fresh server)	0.63 (111/175)	11.11 t/s
60066	44517	0.81 (123/152)	9.11 t/s	5.88 t/s

The second 16k row is a fresh server with nothing cached, so acceptance at depth holds whether or not the prefix
came from a checkpoint - the same variable your point 1 ruled out. The no-MTP figures are from a --no-mtp
server at the same depths (different generated text, same length).

So with the same fork code on Vulkan, shared-MTP accepts at 4k-60k, which matches your Vulkan control but
removes "a different llama.cpp build" from the comparison. What still differs from your failing setup: the
ROCm/HIP backend with -fa on and the QSA kernels (with forced f16 KV), gfx1151, the Q8_0 draft, -b/-ub 16384,
and --spec-draft-p-split 0.10. I have no ROCm run to offer, and did not go past 60k.

---

AI disclosure (per AGENTS.md): written by an AI agent (Claude, in Claude Code) from runs it made on the machine
above, and posted at the operator's request; every number is from those runs.

[ChrisC381]

Issue #76 close-as-fixed comment (draft — post when gh auth restored)

Resolved on current master — retested today (Sep 26) with our original failing configuration.

Build: halo-box master 03895887a (post PR #86 + #87), custom ROCr runtime
7dda3ac6c, SDK 7.14.1, Radeon 8060S / gfx1151, 128 GB unified.

Same setup that failed at pins be905cf7 and d67d5883:
target = our self-quantized Qwen3.8-Flash-Next APEXFN TableQ8 (112 GB, mixed
Q8_0/IQ-family tensors), draft = the installer-pinned shared-MTP head
mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf (SHA 5ff54097…), --spec-draft-n-max 2,
-ctk/-ctv f16, -b/-ub 16384, --lazy-mode on-direct.

Per-request acceptance (from timings.draft_n / timings.draft_n_accepted, the
method @rcfa used upthread — thanks, it's much better than cumulative counters):

prompt	old pin d67d588	master 0389588	master, PM4 off
shallow	0.75	0.87	0.87
4k	0.0 (15.2 t/s)	1.0 (850/850, 44.5 t/s)	1.0 (42.8 t/s)
32k	0.0 (15.0 t/s)	1.0 (850/850, 44.1 t/s)	1.0 (41.8 t/s)

Notes:

• The fix is in the code, not environment: acceptance holds 1.0 with
  DEBUG_HIP_GRAPH_PM4 unset and GGML_CUDA_DISABLE_GRAPHS=1 (PM4-off column).
  PM4 graphs are worth ~4% tg at this scale.
• Greedy canary output is byte-identical to our (independent) Vulkan build's
  output on the same weights — the old failure was pure efficiency loss, and
  now both engines agree exactly.
• test-backend-ops full matrix on this master: clean except the known SOLVE_TRI
  k=64 (hipErrorNotSupported 801).

Whatever landed in the Sep 24–25 QSA window (compact graph reuse, shared-block
visibility, maskless multi-seq) cured it — our original report's isolation
ladder (length-thresholded ~4k+, cache-independent) is consistent with a
visibility/window-sizing fix of exactly that class. Thanks for the merge work
that carried it in. Closing as fixed; we'll keep watching master.