[gaetan-puleo]

Summary

Decode-throughput improvements for qwen4exp (Qwen3.8-Flash-Next UD-IQ4_XS) on gfx1151, in two groups: byte-exact Q8_0 MMVQ fixes that raise plain single-token decode, and speculative-decoding (MTP) improvements that raise both single-stream and multi-sequence decode. Nothing here changes model math for the default path except where noted.

All measurements on the same machine, llama-bench/server interleaved A/B, medians.

Results

area	before	after	delta
plain decode tg128 (3 Q8_0 MMVQ kernels)	25.71 t/s	26.39 t/s	+2.65 %
MTP single-stream	36.31 t/s	41.58 t/s	+14.5 %
MTP multi-sequence (-np 3)	37.05 t/s	44.3 t/s	+19.6 % (now level with plain batched decode 45.2)

Changes

Plain decode (byte-exact Q8_0 MMVQ, logits SHA unchanged c283a8f4…)

1. Rows-per-wave for short-K Q8_0 (hc_{attn,ffn}_up, ncols<=512).
2. Deep prefetch for long-K Q8_0 without fusion (ssm_out/attn_output, attn_q).
3. Deep prefetch for the grouped long-K Q8_0 path (attn_qkv+attn_gate).

Speculative decoding
4. Even-length MTP drafts across sequences (STRIX_SPEC_EVEN_DRAFTS, default on). The target's hybrid memory splits a batch into ubatches that need the same token count per sequence, so drafts of different lengths turned one verify into extra whole-model passes (each a fresh graph). All drafting sequences now pad to the longest confident run. Multi-sequence aggregate: 31.4 -> 36.3 t/s (adaptive), fixed-length np=3 unchanged (already even). Single-sequence byte-identical.
5. Vector-kernel chunks for small-batch MoE router and experts (STRIX_F32_VEC_CHUNK_MAX=32, STRIX_MOE_VEC_CHUNK=4 / _MAX_T=16). A 5-16 token verify step's F32 router [2560x512] and its routed experts (past the vector limits: IQ3_S 4, IQ4_NL 6) fell to tiny GEMM/MMQ tiles; they now run the vector kernel over column chunks. Multi-sequence -np 3: 37.0 -> 42.7 t/s.
6. Small products of a verify step off one-workgroup kernels (STRIX_Q_VEC_CHUNK_MAX/ROWS, STRIX_CONCAT_T_MIN). Quant weights of <=1024 output rows (hc_down [10240->320], attn k/v, shared-expert gate/up) chunked through the vector kernel, and the GDN conv concat uses the tiled transpose from 2 tokens. Multi-sequence np=3: +2.6 %.
7. HIP graphs keyed on the graph shape as well as its first node (STRIX_GRAPH_KEY_SHAPE). Every draft graph replays instead of resetting its neighbour (warmup_reset 416 -> 5). Single-stream MTP: 36.45 -> 39.86 t/s (+9.4 %), byte-exact.
8. IQ3_S vec-dot for the HIP MMVQ path. The sign mask is one v_cmp instead of a per-half unpack; byte-exact. Single-stream MTP: 39.86 -> 42.02 t/s (+5.4 %).

Correctness

• test-backend-ops test -b ROCm0 -o MUL_MAT,MUL_MAT_ID: 14577/14577 on the final build.
• The Q8_0 MMVQ changes (1-3), the HIP graph-key change (7) and the IQ3_S vec-dot (8) are byte-exact (identical final logits SHA / identical MTP content sha1).
• The speculative multi-sequence changes (4-6) change which kernel small batches take, not the arithmetic; they are byte-exact on single-sequence and on the measured multi-sequence outputs (identical content sha1).

Not included / could not port

From the same optimization set, these did not apply to this fork (diverged plumbing/backends) and are left out: GDN deferred rollback (lazy recurrent-state records), QSA decode gather + small-batch mask (gated on a direct_indices path we do not expose), and the MMB Q8_0 hyper-connection / MoE GLU3 fusions (our mmb.cu is a different backend). Measured but reverted: Q6_K MMQ at all widths (noise), BF16 MMB from 9 columns (regressed prefill/tg).

Notes

• These kernels are shared with other architectures; measurements here are on qwen4exp.
• STRIX_* env knobs disable each change (default is the new behaviour) for A/B.

[gaetan-puleo]

Superseded: this duplicated the decode branch of #106 and carried MTP commits. Reorganized — general decode (Q8_0 deep-prefetch, HIP graph keys, IQ3_S vec-dot) now stacks on #106; the speculative-decoding changes (even-length MTP drafts, small-batch verify) now live on #105.