[Perf][SM70] Narrow Qwen3.8 AWQ active grouped decode

[Leonccaa]

Purpose

Reuse the existing SM70 AWQ active-stage operator for Qwen3.8 TP4 small-batch decode (2 to 8 input tokens), and align CUDA Graph warmup with that route so replay hits the same kernels. This is the reduced-surface successor to closed #491: no new operator, Python wrapper or GEMM kernel is added.

• Narrow gate: 512 experts / top-k 10, native or effective group 32, hidden 2560, local intermediate 160 (W13 K2560/N320, W2 K160/N2560), 2 to 8 input tokens. C1 and every unmatched contract keep their existing dispatch.
• Multi-row active-expert segments and offsets-based scheduling are preserved; the active-stage op is called once per stage instead of once per segment.
• The model-specific grouped route defaults on inside that gate. Rollback is VLLM_SM70_AWQ_QWEN38_MOE_COMPACT_GROUPED_DECODE=0 before startup.
• The existing VLLM_SM70_AWQ_TUNE_SMALL_SHAPES=1 autotune is documented as an optional, separately measured configuration and stays globally default-off.
• No precision policy, scheduler, KV or deployment change.

Implementation

Five production files, one test file, one design/evidence document.

• awq_sm70_moe.py: layer contract check _qwen38_active_grouped_layer_contract, runtime admission _use_qwen38_active_grouped_decode (2 to 8 tokens, top-k 10, batched GEMM on, the exact-W2 experiment flags off), and the W13/W2 route switch onto awq_moe_active_dense_stage_sm70_out with the grouped-active exact W2 reduction.
• awq_sm70_warmup.py: warm the active-stage op for the admitted 2 to 8 token shapes with the same admission function, so capture and replay use the same dynamic-offset GEMM.
• awq_sm70_gemm.cu: inside awq_moe_active_dense_stage_sm70_out, a 17-line fast path for exactly these shapes (group 32, 20 to 80 slots in multiples of 10, K2560/N320 or K160/N2560) that runs the offsets-based GEMM in one call instead of the per-segment loop.
• torch_bindings.cpp: the schema now declares active_expert_offsets and active_expert_ids as mutated (Tensor(b!), Tensor(c!)), which the implementation already did.
• envs.py: the new default-on switch.

Review note: the C++ fast path additionally reads the same environment variable with std::getenv, so the op stays self-guarding for direct callers such as warmup and one variable rolls back both sides. The Python admission is the primary gate. If maintainers prefer an explicit argument instead of the environment read, that is a small follow-up and does not change the measured route.

Results

4x Tesla V100 PCIe 32 GB, TP4/MTP0, FP16 activations and KV, chunked prefill with 8192 batched tokens, prefix cache off, frozen input IDs, ignore_eos=false, 320 output tokens, one score per cell and arm. Pure aggregate decode tok/s over the common all-active window.

Implementation-only effect, frozen base 5cceeaad89, autotune off in both arms:

Cell	Grouped OFF	Grouped ON	Change	NVFP4 reference
C1 x 64K	48.65	48.60	-0.09%	54.65
C4 x 64K	110.02	116.20	+5.62%	130.91
C8 x 16K	207.29	211.94	+2.24%	237.58

Additional effect of the existing opt-in autotune, grouped on in both arms, fresh processes, no imported LUT:

Cell	Tuning OFF	Tuning ON	Change
C1 x 64K	48.79	48.59	-0.40%
C4 x 64K	116.13	129.27	+11.32%
C8 x 16K	212.02	242.72	+14.48%

A post-score C4 profile shows the W13 tactic moving from M64x128x32 to M8x256x64 on all four ranks, 5.531 to 2.109 GPU ms per step, with other categories nearly unchanged. Do not add the two tables' percentages.

Integration acceptance on base 755baae1d0 (grouped decode and autotune enabled, shared-expert overlap off): C1 x 64K 51.29, C4 x 64K 131.71, C8 x 16K 247.59 tok/s. All 18 short quality requests stopped at EOS and passed basic answer checks; C4 self-repeats matched complete token IDs and top-5 logprobs 4/4. KV capacity 427,385 tokens, graph memory 0.37 GiB, no OOM. This is integration acceptance on the new base, not a new OFF/ON pair; a same-base OFF/ON pair on current main has not been rerun yet. E2E for these long-prefill cells improves only about 1.4% (C4) and 3.4% (C8) because prefill dominates them; the decode-window gain is the target of this PR.

Precision

• Grouped OFF/ON: 17/18 complete short-output token sequences match. Tuning OFF/ON: 16/18 match; the two C8 differences are wording (and vs +, answer unchanged) and an open Chinese response of 71 vs 70 tokens.
• W13 tile and split-K choices carry small real-weight rounding differences; the separate tuning probe reached a maximum absolute difference of 0.0009765625. No functional failure was found; bitwise equality and broad model-quality non-inferiority are not claimed.
• The cross-batch and cross-process precision sensitivity is a pre-existing mainline property (HC FP16 GEMM and NCCL reduction), explicitly recorded and deferred; this PR does not attempt a determinism fix.

Duplicate-work check

Open-PR searches on 2026-09-06 in 1CatAI/1Cat-vLLM for AWQ grouped decode, "AWQ QPN", "SM70 AWQ" and "#491" found no competing AWQ small-batch route. #504 is NVFP4 batched HC/QSA work; #509 is conversion-cache release; #519 adds an AWQ M=1 operator and does not touch the 2 to 8 token path. Base is main@755baae1d0.

Tests

• CPU: tests/quantization/test_sm70_awq_active_grouped_decode.py, test_sm70_awq_indexed_prefill.py, test_sm70_awq_compact_metadata.py, test_awq_sm70_persistent_cap.py: 70 passed. Boundary cases cover 1/2/8/9/127/128 tokens and disjoint indexed-prefill versus grouped-decode admission.
• Pre-commit on the branch's changed files and git diff --check: clean.
• Frozen-source GPU validation: extension build, 51 GPU-directed tests, 32 dynamic-route operator comparisons. New base: native _C, _moe_C, _C_stable_libtorch built for SM70 from this head, import and SHA-256 gate, then the bounded model acceptance above.

Provenance

Fork review draft: Leonccaa#8 (same commits). The opt-in AWQ M=1 dispatch proposed separately reuses _qwen38_active_grouped_layer_contract from this PR; the M=1 operator itself (#519) is independent.

AI assistance was used. OpenAI Codex implemented the change, tests and evidence under Leon's direction; Claude Code reviewed the diff, re-ran the CPU tests and pre-commit, and prepared this submission. Human review is still required.

🤖 Generated with Claude Code

[Leonccaa]

Fresh matched default/rollback regression (2026-09-06 UTC)

Reviewed head fbaa195f4790fddd11b9141b6c375e154bac2ce4 and tested its switch
in the exact public final stack at #522 head
9c5b69950e62f48840ed0b959e79d9f7a9a6bd12. Both arms use that same rebuilt
binary and restart the engine. This is a configuration-isolated stack test,
not two different binaries or an independent build of each intermediate PR.

Version boundary: #522 subsequently changed its compact-metadata default at
9e23886bdcb91f4fef91bfd2c44a895377e523e4. The OFF/ON arms below explicitly
set 4B and use the earlier frozen head. “Default/rollback” here isolates this
PR's grouped switch; it does not claim to test the newer unspecified 3B default.

Contract: 4 x V100 32GB / TP4, the same AWQ checkpoint and frozen input IDs,
FP16 activation/KV, MTP0, chunk budget 8192, prefix cache off, async scheduling
off, FULL_AND_PIECEWISE graphs, optional small-shape tuning off, QPN M1 off,
4B metadata. Current public Flash/QSA sources; no #494, #509, or #523 overlay.
All four workers' loaded hashes and all 48 AWQ layers were verified. The
native and Python grouped-admission gates both honor the rollback flag.

One warmup then one scored run per cell, output cap 320,
ignore_eos=false, min_tokens=0; no score selection or confidence interval.

Cell	Online pure-decode total tok/s OFF -> ON	Change	Batch E2E seconds OFF -> ON
C1 x 64K	51.3759 -> 51.4864	+0.22%	20.7288 -> 20.9666
C4 x 64K	110.5603 -> 117.4943	+6.27%	91.6303 -> 90.6454
C8 x 16K	206.2690 -> 212.5878	+3.06%	43.3588 -> 42.9427

“Online pure decode” uses one shared engine-output wall interval: after
the last request's first output and through the first request's completion.
Only newly delivered tokens in that interval count; per-request counts are
319 / 294 / 304 for the respective cells. Sampling/runtime overhead is
included. It is neither whole-request TPOT (which includes mixed waiting)
nor a standalone sampling-free llama-bench TG result. ITL p50/p95 ON:
19.463/19.758, 34.095/34.775, 37.638/38.583 ms. All 13 performance outputs
matched complete token IDs across arms.

Quality and numerical limits

The 65 frozen quality records had no per-record score flips, with 45/65
complete output sequences equal. Both arms scored HumanEval 5/5, MBPP 4/5,
IFEval strict prompts 4/5 (instructions 10/12), GSM8K 29/32, local tool-choice
coarse checks 10/12, and needle checks from 1K through 128K 6/6. Code answers
were evaluated only in a CPU-only, read-only, network-disabled sandbox;
IFEval uses the pinned official evaluator. These are bounded regression
subsets, not full benchmark leaderboards or statistical non-inferiority.

Completion changes must not be hidden by unchanged scores: IFEval case
ifeval-0000 and one local tool-choice record changed from natural stop to
the fixed length cap, while their existing scorers still marked them correct.
One already-incorrect MBPP record changed from length to natural stop. Thus
this is not a claim of no added truncation or unconditional quality acceptance.

Same-arm teacher-forced A/A was exactly equal (70 rows in each arm), but
cross-process fixed-prefix logits were not. C1 x 64K maximum absolute logit
difference was 7.509765625, RMS 0.2437164; all 320 argmax values were unchanged.
Physical M1 is outside this new grouped route. Existing QSA allocation-order
and startup-kernel-selection effects are not fully controlled here; #494
remains outside this public freeze. These differences cannot be uniquely
attributed to grouped decode or blanket-excused as small local rounding noise.

Both model/scoring captures completed successfully. Ready live allocation
and KV capacity were unchanged between OFF/ON: 28,605,358,080 allocated bytes
per rank, 563 KV blocks. Sampled startup peaks were close to 32 GiB; successful
execution does not imply spare capacity for co-hosting. No product edits or
new performance optimization were made for this regression.

Startup-peak clarification: neither tested #522 snapshot includes the still-open
#509 loading-cache fix.
That fix already has separate paired AWQ/NVFP4 validation. The near-32-GiB
observations here are from builds without it, not evidence that the fix failed
or that this PR introduced a new loading-peak defect. This long-context stack
has not been remeasured with #509; its earlier reduction cannot simply be
subtracted from these peaks.

AI assistance: Codex performed this regression and prepared the addendum at
Leon's request; this is not independent maintainer approval or upstream GPU CI.