[Perf][SM70] Coalesce AWQ QPN W13 compact metadata loads and enable compact metadata by default

[Leonccaa]

Purpose

Two things, in two commits: read the 3-byte {FP16 scale, u8 zero} W13 metadata of the AWQ M=1 operator (#519) cooperatively instead of byte by byte, without changing dequantization, bias reconstruction, MMA order or any rounding boundary.

Depends on #519 (the operator) and is stacked on #521 (whose design document this PR extends). The commits specific to this PR are the last two, 9c5b69950e (cooperative reader) and 9e23886bdc (default switch); the rest of the diff belongs to #520, #519 and #521 and disappears once they merge.

• Each warp loads one 96-byte metadata tile as 24 aligned 32-bit words and distributes the 3-byte records with shuffles. The last record (byte offset 93) needs no read or shuffle beyond the tile.
• W2 keeps the scalar 3-byte reader, where the cooperative variant measured slower. The 4-byte path is unchanged.
• No metadata representation change, no repack, no added weight copy or buffer, no new flag. Reverting this commit restores the scalar reader.

Operator results

Real layer-1 AWQ weights, TP4 rank 0, M=1, top-k 10, 48 calls per CUDA Graph replay, 7 samples, medians in ms per 48 calls (evidence qwen38-c1-qpn-awq-20260905/evidence-r1 and evidence-r2):

W13 + SwiGLU, 3-byte metadata	hot 10 experts	rotating 480 experts
Scalar reader (#519)	0.6024	0.6578
Cooperative reader (this PR)	0.5573	0.6023
Native NVFP4 M=1 kernel (reference)	0.5472	0.5831

For W2 the cooperative reader measured 0.4492 / 0.4826 against the scalar 0.4015 / 0.4331, so W2 is left scalar. A separate single-layer graph probe of the combined W13/W2 step measured 21.528 us scalar versus 20.616 us cooperative, about 4.2%. These are component observations, not model throughput; the full-model results in #521 use 4-byte metadata and do not exercise this path.

Default switch (second commit)

With the cooperative reader in place, the 3-byte layout no longer costs decode speed on the M=1 route, so VLLM_SM70_AWQ_MOE_COMPACT_METADATA now defaults to 1 for the supported Qwen3.8 TP4 / E512 / native-group-32 shape. The compact layout has been in main since #498 as an opt-in; it frees about 0.88 GiB of resident metadata per GPU, which the engine hands straight to the KV cache.

• Setting 0 keeps the 4-byte layout, as before.
• A default-on request that the build or layer cannot serve (no awq_sm70_prepare_compact op, or a non-Qwen3.8-TP4 shape) falls back to the 4-byte layout with an info log instead of raising. An explicit =1 still fails closed with the same messages as before.
• The decision is a small pure function, _resolve_compact_metadata, covered by unit tests for all cases.
• Numerics do not change: the two layouts decode to identical weights, and the compact layout's outputs were byte-identical to the 4-byte layout at the operator level in both the TurboMind route and the M=1 operator.

Same-day regression on 4x Tesla V100 PCIe 32 GB, TP4/MTP0, FP16 activations and KV, native-group-32 AWQ, max model length 131,328, GPU memory utilization 0.89, the stack #520 + #519 + #521 + this PR, one score per cell:

	4-byte metadata	3-byte metadata
Engine-reported KV cache size	427,385 tokens	497,983 tokens (+16.5%)
Maximum concurrency at 131,328 tokens per request	3.25x	3.79x
C1 x 64K pure decode	59.21 tok/s	59.33 tok/s
C4 x 64K / C8 x 16K pure decode	within 1.0% / 1.5% of each other, inside run-to-run noise

The earlier acceptance of the compact layout on the TurboMind route (#498) measured a warm-state cost of about +1.22% on an 8192-token prefill and +0.27% on 32-token decode against +19.36% KV capacity at fixed memory utilization. Other AWQ MoE models keep the 4-byte layout through the fallback.

Tests

• Scalar and cooperative builds were each tested with the same two portable SM70 GPU tests (tests/kernels/test_sm70_awq_qpn_m1.py), 256 real-weight/dynamic-graph exact comparisons against the validated prototype, 72 independent FP64 stage checks with no rounding-bound violations, and 48 invalid-argument rejections, covering both metadata layouts and all four TP partitions. Outputs of the scalar, cooperative and prototype builds are identical on those inputs.
• The extracted CUDA source is byte-identical to the GPU-validated full native source (SHA-256 55010c76...9481b5).
• Pre-commit on the changed files and git diff --check: clean.
• Default switch: tests/quantization/test_sm70_awq_compact_metadata.py (default on, 0 disables, fallback cases, explicit fail-closed cases) plus the neighbouring AWQ CPU suites: 97 passed. Two unchanged modules still show their pre-existing teardown errors on a GPU-less host (torch.accelerator.empty_cache()), identical before and after this change.

Duplicate-work check

Open-PR searches on 2026-09-06 in 1CatAI/1Cat-vLLM for AWQ compact metadata, "AWQ QPN" and "SM70 AWQ" found no competing AWQ metadata reader. Base is main@755baae1d0.

Provenance

Fork review draft: Leonccaa#11 (same commit).

AI assistance was used. OpenAI Codex implemented and validated the cooperative reader under Leon's direction; Claude Code reviewed the tile and shuffle index arithmetic, implemented the default switch and its tests after Leon decided the default, re-ran pre-commit on this branch, and prepared this submission. The regression measurements come from the run driven by OpenAI Codex. Human review is still required.

🤖 Generated with Claude Code

[Leonccaa]

Fresh regression: reader, layout capacity, and updated default (2026-09-06 UTC)

There are two source snapshots in this addendum. The original operator and
explicit-layout model arms use head 9c5b69950e62f48840ed0b959e79d9f7a9a6bd12.
The follow-up default/fallback checks use the updated head
9e23886bdcb91f4fef91bfd2c44a895377e523e4. Earlier results are not relabeled
as full testing of the later head or its unspecified environment defaults.

The two exact git archives differ only in two Python files, one test and one
design document. All other tracked files, including native sources and build
inputs, are identical. The updated runtime verifies all 2,439 tracked runtime
files and reuses the checked native binaries with that provenance recorded;
this is not described as another native compilation. Loaded _C SHA-256:
c0ed337cbce8c4d47e681ad0c80c1c3c9fa65cf190bd7459841d78ac9cfd7058.
Current public Flash/QSA is used, with no #494, #509, or #523 overlay.

Startup-peak clarification: neither tested #522 snapshot includes the still-open
#509 loading-cache fix.
That fix already has separate paired AWQ/NVFP4 validation. Near-32-GiB startup
observations from this regression therefore describe builds without it, not a
failure of that fix or evidence of a new defect caused by 3B. This long-context
stack has not been remeasured with #509; its earlier reduction cannot simply
be subtracted from these peaks. Loading-cache cleanup and the 3B resident-bank
capacity benefit below are separate mechanisms.

Reader-level evidence

The original exact-head SM70 build passed 2 portable GPU tests, 256
native/reference and dynamic-Graph bitwise checks, 72 independent FP64
stage-pair bounds with zero violations, and 48 invalid-argument checks.
The existing tiny-value relative-only gate exception is retained separately
(absolute 5.960464477539063e-8, relative L2 0.0020175909 versus 0.002);
no threshold was changed or relabeled passed. See the #519 addendum for scope.

A same-layout 3B scalar-reader -> cooperative-reader operator comparison,
using real layer-1/rank-0 checkpoint data, measured 21.808 -> 20.840 us
(12 ABBA pairs, 64 calls per graph), about 4.44% lower component latency.
This is not a whole-model speedup or a rotating-expert benchmark claim.

Layout-level capacity and serving measurements

Contract: 4 x V100 32GB / TP4, frozen AWQ native-g32 weights and prompt token
IDs, FP16 activation/KV, MTP0, max model length 131328, memory utilization
0.89, chunk budget 8192, prefix cache/async scheduling off, CUDA Graphs.
Both following arms have grouped ON, QPN M1 ON and optional tuning OFF;
the explicit change is 4B -> 3B, including automatic KV-pool resizing.

Cell	Online pure-decode total tok/s 4B -> 3B	Change	Batch E2E seconds 4B -> 3B
C1 x 64K	59.2099 -> 59.3330	+0.21%	19.8932 -> 20.1776
C4 x 64K	117.7127 -> 116.5037	-1.03%	90.6700 -> 91.9439
C8 x 16K	212.8465 -> 209.7103	-1.47%	42.9054 -> 43.8104

Each cell has one scored run after warmup, max 320 output tokens,
ignore_eos=false, min_tokens=0. A shared all-active engine-output window
excludes prefill/mixed waiting and counts 319/294/304 new tokens per request;
normal sampling/runtime overhead remains included. These are not isolated
sampling-free TG rates. All 13 performance outputs matched complete token IDs.
One observation per cell cannot establish a stable gain, regression, or that
the differences are statistically just noise. This 4B/3B comparison does not
isolate the same-3B old/new reader at full-model level.

Prepared banks save 943,718,400 bytes = 900 MiB per rank across 48 layers.
KV blocks increase 563 -> 656 and engine-reported capacity 427,385 -> 497,983
tokens (+16.52%). This capacity benefit belongs to the existing compact
layout; the reader change does not itself introduce those saved bytes.
Automatic KV allocation consumes the freed space: ready live allocation is
28,605,358,080 -> 28,613,578,752 bytes, not 900 MiB less total ready allocation.
The reported capacity still does not fit C8 x 65536 input tokens under this
fixed contract. No fresh E4M3-KV capacity result is claimed.

The 65 frozen quality scores and all finish reasons are unchanged in this
4B/3B pair; 47/65 complete sequences match. Both score HumanEval 5/5, MBPP 4/5,
IFEval strict prompts 4/5 (instructions 10/12), GSM8K 29/32, local tool-choice
coarse checks 10/12, and needle checks 1K through 128K 6/6. Code/IFEval scoring
uses the pinned evaluator in a CPU-only, read-only, network-disabled sandbox.
These bounded subsets do not establish statistical quality non-inferiority;
pre-existing length caps and the limits of the tool-choice rubric remain.

Teacher-forced same-arm A/A is exact (70 rows each). Cross-process C1 x 64K
max absolute logit difference is 7.509033203125, RMS 0.2554592, p99 0.892334,
with 0/320 argmax changes. All logits are finite. Allocation-order effects
remain uncontrolled in this public stack without #494, and changing the
layout also changes the KV pool. The full-model differences cannot be uniquely
attributed to the reader or blanket-accepted as small local rounding noise.

Updated implicit-default validation

The default/fallback-specific CPU tests pass (5 tests). The updated exact
runtime passes 211 adjacent SM70 tests, 14 deprecation warnings, 53.40 s.
Expanded CPU-only attempts initially lacked a native NVFP4 op and hit
no-accelerator teardown errors; they were retained and did not reproduce in
the proper native environment. No product or cleanup-fixture edits were made.

The actual default combination completed a separate full capture on the new
head, with QPN OFF and the metadata environment variable absent. All four
ranks verified 48/48 3B layers and 0/48 QPN-enabled layers; KV is again 656
blocks / 497,983 tokens. Compared with the earlier explicit G1Q0T0B4 arm:

Cell	Online pure-decode total tok/s 4B -> implicit 3B	Change	E2E seconds 4B -> implicit 3B
C1 x 64K	51.4864 -> 51.1163	-0.72%	20.9666 -> 21.0347
C4 x 64K	117.4943 -> 116.1633	-1.13%	90.6454 -> 91.6998
C8 x 16K	212.5878 -> 208.5280	-1.91%	42.9427 -> 43.7480

All 13 performance outputs match. This is an explicitly audited cross-source-
snapshot comparison, not the same-head QPN-ON layout pair above. The analysis
retains both heads and permits only the verified default-switch Python delta,
with identical native libraries/checkpoint/input IDs; it does not ignore
manifest mismatches generally. One run per cell gives no confidence interval.

The new default's 65 per-record quality scores are unchanged relative to
the old explicit 4B arm, with 46/65 complete token sequences equal. One
already-incorrect MBPP record (mbpp-0320) changes from natural stop to the
fixed length cap; this is disclosed, not erased by unchanged scores. Teacher
A/A is exact (70 rows each), all logits finite; cross-process C1 x 64K max
absolute difference is 6.962890625, RMS 0.2681553, p99 1.00683594, with 0/320
argmax changes. These unresolved whole-model differences are not a proof of
a reader defect or blanket evidence of harmless rounding. Numerical attribution
remains a separate follow-up, not a new kernel change in this regression.

Separately measured prompt processing

Both 3B and explicit 4B/QPN-OFF PP captures completed on the same updated
head with identical runtime manifests. PP uses prefill-only forwards with one output
token per request and exact per-rank prompt accounting. It is not TTFT-derived.
Time spans synchronized worker input/attention preparation, full model/TP
communication and LM head, stopping before sampling. Sum the slowest rank
duration once per step; exclude tokenization, sampling, detokenization, queue
and inter-step engine scheduling. This is explicitly vLLM model-stage PP,
not the identical llama.cpp executable or continuous host-loop timer.

Submitted cell	4B model-stage seconds	3B model-stage seconds	4B PP tok/s	3B PP tok/s	Change
C1 x 64K	14.4700	14.7700	4529.08	4437.10	-2.03%
C4 x 64K	57.9792	59.4784	4521.35	4407.38	-2.52%
C8 x 16K	27.5727	28.0424	4753.69	4674.06	-1.67%

There are 8/32/16 measured model steps respectively, all prefill-only. C4/C8
describe batch submission, not a claim that every request overlaps in each
prefill step. The 3B PP probe follows the full regression; the 4B reference
is PP-only in a fresh process. Each cell has its own warmup and one score;
neither workload-history identity nor statistical significance is claimed.
The 3B layout's observed capacity benefit therefore is not advertised as a
speedup in every phase. C8 has 16K context, so its rate divided by the 64K C1
rate is not a same-context concurrency-scaling measurement.

The first appended PP warmup after the original complete model captures
failed a harness internal/external request-ID accounting assertion. No PP
score was accepted from that attempt; its overall exit 1 is retained, while
the earlier completed serving/teacher/quality captures and sandbox scores
remain separate. The harness now records the engine's actual returned IDs
and persists raw observations before assertions. Frozen scorers are unchanged.
The replacement PP captures both exited 0 with all three scored cells and
all-rank exact input-token accounting complete.

AI assistance: Codex performed this regression and prepared the addendum at
Leon's request. These results are not independent maintainer approval or
upstream GPU CI. No new optimization or numerical acceptance threshold was
introduced in this regression.