[Perf][SM70] Block-pack the activations of the NVFP4 QPN2 kernels

[Peuqui]

Purpose

The QPN2 kernels (nvfp4_qpn2_sm70.cu) read the activations row-major. The
m8n8k4 A fragment hands lane L the activation row (L & 3) + 4 * (L >> 4),
so a warp needs eight activation rows for every 16-k group, and those rows
lie k * 2 bytes apart: eight cache lines per group. On a Quadro RTX 8000
(64 KB L1) this shows up as a 5x growth in L1 wavefronts over the Tesla V100
(128 KB L1) for the same kernel, and it was the reason for the Turing decode
gap on DFlash2 in our fork (the MMA shape was not: m8n8k4 and m16n8k8
run at the same rate on Turing, measured with a probe against a CPU
reference).

This change adds a small pack kernel that lays the activations out as
[k / 16][rows][16], one contiguous block per 16-k group holding exactly the
rows the fragment scatters over, and a Packed template parameter on the
GEMM and the gated GEMM that reads that layout instead. It is pure data
movement: same values, same order, same fp32 accumulation. Rows beyond M are
zero. The pack is skipped where its own launch costs more than the row
spread: it is taken from M=7 on Turing and from M=8 on Volta.

VLLM_SM70_NVFP4_QPN2_PACK (auto / 0 / 1) is registered in envs.py
for A/B runs and excluded from compile_factors(): the kernels are
bit-identical with and without the pack, and the compiled graph only sees
the opaque op, so the switch must not change the compile hash (see the
note in the test result).

Test Plan

1. pre-commit run --files csrc/sm70_turbomind/ops/nvfp4_qpn2_sm70.cu vllm/envs.py tests/kernels/quantization/test_nvfp4_qpn2_block_pack.py
   and the mypy-3.10 hook on the Python files.
2. New GPU test tests/kernels/quantization/test_nvfp4_qpn2_block_pack.py:
   GEMM and gated GEMM with the switch forced off and forced on, compared
   bit for bit, for M in {1, 2, 4, 5, 7, 8, 9, 15, 16, 17, 32} (one- and
   two-row-tile paths) and across the launch configurations (split-K 8 / 12
   / 16 / 32, one and two accumulator chains, shapes 3584x5120, 8704x5120,
   5120x1536, 5120x4352, 62080x5120). Run on a Tesla V100 and on a Quadro
   RTX 8000.
3. End to end on 2x Quadro RTX 8000, RadixArk/Qwen3.8-27B-NVFP4 through the
   Turing route of [Feature][SM75] Run ModelOpt NVFP4 and FP8 linears on Turing through the SM70 QPN kernels #604, MTP with k = 3..7 (verify M = 4..8), greedy, 5 x 400
   tokens per boot, pack off against pack auto at each k, --attention-backend TRITON_ATTN --kv-cache-dtype float16.

Test Result

1. All applicable hooks passed; mypy-3.10 passed.

2. 58 passed on the Tesla V100, 58 passed on the Quadro RTX 8000.

3. k	verify M	pack off, tok/s	pack auto, tok/s	delta	text SHA / acceptance length
   3	4	68.8	68.9 (not packed, below threshold)	-	equal / 2.920
   4	5	69.7	69.4	-0.4 %	equal / 3.053
   5	6	71.4	71.6	+0.3 %	equal / 3.390
   6	7	66.5	67.8	+1.9 %	equal / 3.419
   7	8	64.4	66.7	+3.7 %	equal / 3.571

   Spans within a boot are at most 0.5 tok/s. Within every pair the 400-token
   text and the acceptance length are identical. The Turing threshold is set
   to M=7 from these numbers (a kernel microbenchmark had suggested 5). On
   our fork the same pack moved DFlash2 (verify M=8) on the RTX 8000 from
   69.2 to 72.7 tok/s with the text unchanged; DFlash2 could not be run here
   because its drafter needs non-causal attention, which TRITON_ATTN does
   not provide and FLASH_ATTN_V100 refuses on sm_75.

   A note on the method, since it cost a day: with the SM70 Flash-V100 0.0.3
   graph, VLLM_USE_AOT_COMPILE=1 and benchmark_combo_kernel=True, every
   fresh compile picks its combo kernels by timing, and the AOT path pins
   that choice in a per-hash Inductor cache directory even with
   VLLM_DISABLE_COMPILE_CACHE=1. Two boots that differ only in the value of
   an unregistered VLLM_ variable get different hashes, different first
   compiles and, at the 400-token scale, different greedy texts (measured:
   twelve boots, two texts, each stable per hash). The switch is therefore
   registered and excluded from the compile factors, so both arms of every
   pair above share one compiled graph and differ only in the pack.

Not a duplicate

Checked on 2026-09-12 against 1CatAI/1Cat-vLLM: gh pr list --state open --search for "qpn2", "block pack", "activation layout", "pre-ampere";
gh pr diff --name-only over every open PR for nvfp4_qpn2_sm70.cu. #561
(shared NVFP4 layout for DFlash2) changes the weight side and keeps the
current prepack as the control layout; this change touches the activation
side only and applies on top of either layout. #604 (mine) is the Turing
route the end-to-end numbers were taken on; the kernel change here is
independent of it.

AI assistance (Claude) was used to port the pack from our fork and run the
measurements; I reviewed every line and ran the tests above.

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

[Peuqui]

A short note so this does not read as superseded by the recent QPN2 work on main.

The gain of this change is on Turing, not on Volta. The activation rows the m8n8k4 fragment reads lie k * 2 bytes apart, which a 128 KB L1 absorbs but a 64 KB L1 does not. On the Quadro RTX 8000 the kernel trunk at M=8 runs 1.45x faster with the pack, on the Tesla V100 1.04x. That matches the V100 TP4 result recorded in sm70_quasar_dflash2_15ms.md for the same [K/16, 8, 16] input layout, well under a millisecond per round: a V100-only screen will not show it.

I rebased the change onto a6f5e83. It conflicted textually with d66797c in nvfp4_qpn2_sm70.cu; the resolution keeps TurboMindLayout and CacheCodes and adds Packed as a further template parameter, including the cached TP4 output-weight call. Re-measured on that rebase plus #604 (the Turing route), 2x Quadro RTX 8000, Qwen3.8-27B-NVFP4, MTP k=7 (verify M=8), greedy, 5 x 400 tokens per boot: pack off 68.47 tok/s, pack auto 71.10 tok/s (+3.8 %), identical text and acceptance length (3.800), pack confirmed active only in the auto boot. This reproduces the +3.7 % in the description.

What I could not measure: the combination with VLLM_SM70_NVFP4_QPN2_SHARED_WEIGHT=1. On our 2x V100 TP2 setup the NVFP4 linears run through the TurboMind GEMM, so neither the pack nor the shared layout is exercised there, and I do not have four V100s free for the TP4 case it targets. The two changes act on different sides (d66797c orders the weight tiles, this packs the activations), but that is a reading of the code, not a measurement.

The rebased branch is pushed.