Repository navigation
spec: MTP draft-head subset + even drafts / small-batch verify for qwen4exp - #105
Closed
gaetan-puleo wants to merge 4 commits into
Closed
gaetan-puleo wants to merge 4 commits into
gaetan-puleo wants to merge 4 commits into
Conversation
--spec-draft-mtp-vocab was accepted for qwen35/qwen35moe only, so on a qwen4exp MTP sidecar the flag was silently ignored and every drafted token was scored against the full 248k-row Q6_K LM head. Allow LLM_ARCH_QWEN4EXP in mtp_draft_vocab_get and add the row-subset head branch to graph_mtp, mirroring qwen35moe: score build_lora_mm on the runtime subset (rows copied from the existing head at load - no file or weight change) and scatter its logits into a full-vocab -inf row, so token ids and the verification path stay unchanged. Measured on Qwen3.8-Flash-Next UD-IQ4_XS + mtp-...-Q4_K_M sidecar, 128 greedy tokens, same-session A/B: 38.0 -> 41.5 t/s at N=65536 (+8%), identical output and acceptance. N=131072 is slower than the control; N=32768/16384 are about equal with a small acceptance loss. With the subset the draft phase is 340 ms of ~3.4 s and the target verify is the remainder.
The target hybrid memory splits a verify batch into ubatches with an equal number of tokens per sequence, so unequal drafts (e.g. 2,2,1) become extra whole-model passes. The MTP drafter now pads an unconfident sequence and cuts all drafts to the longest confident run; the server cuts every draft to the shortest when a slot drafted nothing (unless one replays accepted tokens). STRIX_SPEC_EVEN_DRAFTS=0 restores the previous behaviour. Single sequence is unchanged (even only applies to >1 drafting sequence). Measured, 4 sequences adaptive drafting, 192 tokens each, same build A/B: even off: 31.39 31.51 31.17 tok/s summed even on : 34.18 36.36 38.24 tok/s summed
A verify step of several conversations carries 5-16 tokens, one more than the vector kernel takes for some types. The F32 MoE router [2560x512] at 9-32 columns now runs MMVF over column chunks (STRIX_F32_VEC_CHUNK_MAX, default 32) instead of the GEMM path; quantized routed experts at 5-16 tokens run MMVQ over chunks (STRIX_MOE_VEC_CHUNK, default 4; STRIX_MOE_VEC_CHUNK_MAX_T, default 16) instead of the tiled MMQ kernel. Batches of 1-4 tokens and >32 columns are unchanged. test-backend-ops MUL_MAT,MUL_MAT_ID: OK (ROCm0, 14576/14577, one flaky q4_1 case which also fails intermittently on the control build). Measured, 3 sequences fixed drafting, 128 tokens each, same build A/B: before: 37.3 / 35.4 tok/s summed (patch 1 only) after : 43.1 / 42.2 tok/s summed Single sequence (4-token verify) byte-exact unchanged.
Quantized weights of at most STRIX_Q_VEC_CHUNK_ROWS (1024) output rows at 9-32 columns (the HC down projection, attention k/v, shared expert) now run MMVQ over 8-column chunks instead of a handful of MMQ tiles (STRIX_Q_VEC_CHUNK_MAX=0 disables). Taller weights stay on MMQ. The GDN conv input concat takes the tiled transpose from 2 tokens on RDNA3.5 F32 (STRIX_CONCAT_T_MIN, default 2) instead of the non-contiguous kernel. The patch's BF16 vec-chunk hunk does not map: our tree has no combined vec_chunks block, MMB is disabled below T=512 and BF16 small batches use MMF. Measured, 3 sequences fixed drafting, 384 tokens each, same build A/B: off: 46.6 / 50.3 tok/s summed on : 48.9 / 50.5 tok/s summed Single sequence byte-exact unchanged.
gaetan-puleo
changed the title
Collaborator
Author
|
Superseded by #108, whose head branch is on halo-box (this one pointed at a fork). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Speculative-decoding (MTP) improvements for
qwen4exp(Qwen3.8-Flash-Next UD-IQ4_XS) on gfx1151. Two kinds of change: the--spec-draft-mtp-vocabdraft-head row subset (previously accepted only forqwen35/qwen35moe), and three fixes that make MTP actually pay off at several sequences and on small verify steps.Measured on the same machine, server interleaved A/B, medians.
Changes
qwen4exp(src/llama-model.cpp,src/models/qwen4exp.cpp).--spec-draft-mtp-vocab <n>scores drafted tokens against only the firstnrows of the target LM head (a load-time row copy, no GGUF/weight change) and scatters them into a full-vocab-infrow. On a qwen4exp MTP sidecar the flag used to be silently ignored, leaving the draft on the full 248k-row Q6_K head. N=65536 is the sweet spot.STRIX_SPEC_EVEN_DRAFTS, default on;common/speculative.cpp,tools/server/server-context.cpp). The target's hybrid memory splits a batch into ubatches that need the same token count per sequence, so drafts of different lengths turned one verify into extra whole-model passes, each a fresh graph. All drafting sequences now pad to the longest confident run. Multi-sequence aggregate: 31.4 -> 36.3 t/s (adaptive).STRIX_F32_VEC_CHUNK_MAX=32,STRIX_MOE_VEC_CHUNK=4/_MAX_T=16;ggml/src/ggml-cuda/ggml-cuda.cu). A 5-16 token verify step's F32 router[2560x512]and its routed experts (past the vector limits: IQ3_S 4, IQ4_NL 6) fell to tiny GEMM/MMQ tiles; they now run the vector kernel over column chunks. Multi-sequence-np 3: 37.0 -> 42.7 t/s.STRIX_Q_VEC_CHUNK_MAX/ROWS,STRIX_CONCAT_T_MIN;ggml-cuda.cu,concat.cu). Quant weights of <=1024 output rows (hc_down [10240->320], attn k/v, shared-expert gate/up) chunked through the vector kernel, and the GDN conv concat uses the tiled transpose from 2 tokens. Multi-sequence np=3: +2.6 %.Results
-np 3)--spec-draft-mtp-vocab 65536Correctness
test-backend-ops test -b ROCm0 -o MUL_MAT,MUL_MAT_ID: 14577/14577.Not included
The adaptive/cost-aware draft path (#102) is intentionally not part of this change.
Note
This branch was split out cleanly: the non-speculative decode kernels (Q8_0 deep-prefetch, HIP graph keys, IQ3_S vec-dot) live on the decode PR #106, not here.