[Kernel][SM70] Reach 82 tok/s on Qwen3.8 Flash Next NVFP4

[yangzhuxinyzx]

Purpose

Follow up the merged Qwen3.8 Flash Next bring-up in #345 with trace-driven SM70 decode acceleration for the fixed V100 acceptance lane:

• 4x V100-SXM2-32GB, TP4/PP1
• V2 model runner, FP16 activations and KV cache
• MTP/speculative decoding disabled
• 8192 input tokens, 512 output tokens, one request
• FULL_AND_PIECEWISE CUDA graphs
• pinned-host PLE/ngram table

Implementation

• Add exact-shape online QPN8 projection routes for Qwen4Exp GDN and hyperconnections.
• Add fused shared-expert gating, exact E512/K10 routing, TP4 push all-reduce, and the V2 greedy TP-local pair sampler.
• Add a direct NVFP4 QPN-M1 routed-expert path for the exact TP4/B1 Qwen3.8 geometry. It consumes the existing TurboMind-packed expert tensors, broadcasts the input inside W13, and skips the per-layer routed-input replication kernel.
• Keep the route fail-closed to SM70, TP4, B1, H2560, E512/K10, FP16, contiguous int32 IDs. Other shapes retain the existing path.
• Preserve public-main quality policy: the unrelated serialized PP2xTP4 QPN8 defaults remain disabled.

Test plan and result

Full model performance

Matched previous control: 77.370242 tokens/s, 12.924868 ms/token.

New 8192x512 steady decode repeats:

• 82.252144 tokens/s
• 82.275049 tokens/s
• 82.295610 tokens/s

Mean: 82.274268 tokens/s, 12.154469 ms/token. This is +4.904026 tokens/s and -0.770399 ms/token versus the matched control. Repeat range is 0.043466 tokens/s.

Quality and memory

• Arithmetic output remains 42 with unchanged token hash 93def17b...
• Chinese output remains 在标准大气压下，水的沸点是100摄氏度。 with unchanged token hash 11d98c01...
• Peak sampled device memory: 29,752 MiB on each 32,768-MiB V100.
• Model state: 20.86 GiB/rank; available KV memory: 3.73 GiB/rank; graph footprint: 0.16 GiB/rank.
• PLE remains pinned on the host at 11.92 GiB/rank.

Operator and source gates

• Real-weight microbenchmark spans expert IDs 0 through 470 and 16 random inputs. Selected W13 split-K 8 has maximum stage error 3.052e-5; W2 split-K 1 is bitwise exact; final weighted MoE output differs by at most 4.768e-7.
• The SM70 CUDA sidecar compiles with CUDA 12.8 and runs graph-safe on V100.
• The production torch_bindings.cpp translation unit compiles with the official _C macro/include set.
• FakeTensor registration smoke passes for the sidecar namespace.
• Focused NVFP4 suite: 19 passed.
• Ruff format/check, compileall, and git diff checks pass.
• The clean public-main integration worktree repeats the focused 19-test suite and preserves the public default-policy assertions.

Evidence

The exact command contract, trace decomposition, rejected alternatives, microbenchmarks, route markers, memory figures, and result paths are recorded in docs/design/sm70_qwen38_flash_next_nvfp4.md.

This PR remains Draft for public review and CI. The immediate >=80 tokens/s no-MTP gate is met; the longer-term >=100 tokens/s target remains follow-up work.

[yangzhuxinyzx]

Final audit update: #363 has been merged and this branch is now synchronized to latest main at head 98076cfb9537e02c8ed34f479c75136ac8ad868c. The final tree exactly matches the independently projected merge tree 3083c32742609b8ca5f53f3b21775332dc6fb67f.

Repairs applied before merge:

• measured online QPN8 and shared-expert gate fusion now default on, with explicit =0 rollback;
• optional QPN sidecars load consistently with those default-on routes;
• the online QPN8 model_type identity check was replaced with exact TP/device/topology/shape/HC/QSA structural gates;
• V2 TP-local greedy sampling falls back whenever LoRA is configured.

Validation: complete 28-file changed-file pre-commit passed; 23 focused policy/contract tests passed with one CUDA-only numerical test skipped; original retained evidence remains 82.274268 tok/s mean, unchanged arithmetic/Chinese token hashes, W13 max stage error 3.052e-5, W2 bitwise identity, and final weighted MoE max difference 4.768e-7. I will merge after the latest-head remote full pre-commit passes.