[Model][SM70] Calibrated E4M3 KV cache for Qwen4Exp QSA with MTP

[Leonccaa]

Purpose

Enable calibrated E4M3 KV cache for the Qwen4Exp QSA path with MTP on SM70 (V100), and complete the draft-side scale handling that the staging work left open. It is opt-in (kv_cache_dtype=fp8_e4m3 plus VLLM_QWEN4EXP_QSA_E4M3_MTP=1); without it nothing changes.

On 4×V100 TP4 this raises usable KV capacity by 78.8% and prefill throughput by 31–35%, at a decode cost of 1.3–7.0% that scales with context length. It has served our TP4 V100 deployment, with MTP3 and grouped CPU offload, since 2026-09-19.

Status and history

Rebased onto current main (1e90d17f2c). #647 and #637 are merged, so their commits are gone. Four commits remain:

• 115984a — areslp, E4M3 QSA main KV cache with MTP (phase 2 + 2b), from his staging branch feat/qsa-e4m3-mtp-draft-scales (0fda07e). Its message still says "NOT for merge and no PR": that described the unfinished staging state, and the next commits finish it. @areslp, if you would rather it not land this way, I can squash it into the completion commit with you as co-author.
• 834e158 — draft-side scale finalisation: E4M3 MTP draft loading requires calibrated 2/2 K/V scales and fails closed; the main-model fallback contract is unchanged. Adds the revision-checked scale overlay materialiser (tools/qwen4_exp/materialize_qsa_scale_overlay.py).
• 11f41e1 — FP32 XQA workspace for E4M3. Fixes [Bug][SM70] E4M3 KV: XQA page4 route rejects the fp16 tmp_out that qsa.py allocates (kernel expects fp32 since 8a10215d6) #648, which breaks E4M3 on current main even without MTP. The fix is also offered on its own in [Bugfix][SM70] Allocate the QSA XQA page4 workspace for E4M3 KV #705 so it can land first; whichever lands first, the other drops it on rebase.
• 2c1c3f7 — tests pinning both halves of the scale-overlay gate.

Related work: #702 makes E4M3 without MTP the default first and calls the MTP scale overlay a separate contract; this PR is that contract. #696 (QSA DCP2) merges with this PR without conflicts.

Validation on 4×V100 TP4

Qwen3.8-Flash-Next AWQ g32, max_model_len 262,144, MTP3 with draft_sample_method=greedy,
gpu_memory_utilization 0.96.

Correctness: C1/C2/C4 acceptance arms pass, 64 generated segments with zero corruption,
24/24 main QSA and 2/2 MTP draft scales loaded under strict checking.

KV capacity (engine-reported, same image both arms): FP16 433,401 → E4M3 775,096
tokens, +78.8%.

Throughput, single-variable: identical image, weights, draft, offload spec, gmu,
max_num_seqs, max_model_len, compilation config and seed; the only difference is
kv_cache_dtype for main and draft. Prompts are cut from this repository's own
benchmarks/sonnet.txt plus docs/*.md, each request carries a unique nonce so nothing is
served from the prefix cache (prefix_cache_hit_blocks is 0 on every run), 3 repeats,
medians:

prompt	prefill	decode	tokens/step	MTP acceptance	step p50
1,024	+30.9%	−1.3%	±0.0%	±0.0%	+1.3%
4,096	+34.5%	−5.2%	−3.7%	−5.3%	+1.2%
8,192	+31.6%	−7.0%	−5.7%	−8.5%	+1.4%
32,768	+30.6%	−4.3%	−2.4%	−3.1%	+0.8%

The decode cost is disclosed, not hidden. It is not a constant tax: step time is
essentially unchanged (p50 +0.8–1.4%, p90 +0.7–1.6%), so attention itself is not slower.
The decode delta tracks tokens-per-step, which tracks MTP draft acceptance. At 1k both arms
show identical acceptance (0.582) and decode is within noise; the gap grows with context,
which is what KV quantisation error accumulating along the sequence looks like.

Workload note: MTP acceptance was measured per request and targeted to the 63% ± 3% seen on
real agent traffic on this deployment (measured medians 0.58–0.76). Synthetic prompts are
unusable for this comparison — they pin tokens-per-step near the MTP3 ceiling of 4.0, which
erases the very variable the decode cost flows through.

Not caused by this change

This stack was investigated for an unrelated non-terminating thinking loop on AIME 2026.
A single-variable FP16 control reproduces the loop at essentially the same offset
(reasoning char 731 vs 738, same repeating unit), it also reproduces with speculative
decoding disabled entirely and on an older build, and it does not occur at all under the
model card's recommended thinking-mode sampling (0/20 trajectories). E4M3 KV is not
implicated. Separately, the temperature>0 output corruption seen at max_num_seqs>2 is
the pre-existing draft_sample_method=probabilistic defect in #656, reproducible on FP16 KV.

Re-validation on current main

On main 1e90d17f2c with this PR, 4×V100 TP4, E4M3 KV, MTP3, CUDA graphs, prefix caching and 32 GiB grouped CPU offload, fresh servers:

• Same outputs as the build we deploy: on the fixed gate (six MTP prompts with accepted and drafted counts, five code repeats, and a 30,030-token prompt with prefix-hit repeat and changed suffix), DCP1 output matches that build exactly. Decode is 77.7 tok/s (77.9 on that build), with zero error lines.
• Together with [Model][QSA] Target-only DCP2 for Qwen4Exp QSA on TP4 V100 #696: DCP2 matches DCP1 on the same gate. GPU KV tokens at 262,144/C4/GMU 0.96 go from 775,096 (DCP1, 2.96x at 256K) to 1,178,337 (DCP2, 4.50x).
• Unit tests (V100): this PR's E4M3/MTP, calibration, overlay and weight-loading suites plus the [Bugfix][SM70] Allocate the QSA XQA page4 workspace for E4M3 KV #705 workspace test give 87 passed, 1 skipped.

Known limitations

• Measured on SM70 / V100 TP4 only.
• draft_sample_method must be greedy above max_num_seqs=2 until [Bug][Spec Decode] probabilistic draft sampling corrupts output above max_num_seqs=2 at temperature > 0 #656 is resolved.

Co-authored-by: areslp 100579+areslp@users.noreply.github.com

🤖 Generated with Claude Code