[Kernel][WIP] Follow up FlashNext batch HC fusion and QSA route audit

[yangzhuxinyzx]

Purpose

Continue the no-MTP FlashNext batch scaling target (C4/C8/C16: 238/420/728 tok/s) while protecting measured C1 ~81 tok/s and model quality. Integration base: 755baae1d075ee04fa9096b23fc0225b23589a86.

Follow-up to merged #474. Only its unmerged HC pointwise/disjoint-push fusion prototype is carried here; #481's merged single-token work is not duplicated. AI assistance was used. Human review, clean-build and model-quality gates are still required. Do not merge or enable by default based on operator checks alone.

Test Plan

• Screen complete batched HC using rotating real weights, eager/poisoned graph replay, private-channel auxiliary-stream stress.
• Audit frozen-run Flash-V100 import and QSA row admission; microbenchmark native attention against the frozen Triton route before any dispatch change.
• Integrate only profitable local capabilities, then run paired endpoint and dataset/tool/schema quality gates. No concurrency/chunk/KV global binding.

Test Result

• Frozen pre-merge source micro: private HC channel reduces complete HC M4/M8/M16 median by 14.6/14.9/15.9%; auxiliary sum2 stress passes 64 changed-input cycles on all four ranks. Not endpoint savings or model-quality evidence, and not yet a current-main binary test.

• Width checks 1 through 17 and 24 pass against unfused sharded oracle; original replicated GEMM association differences remain unadmitted.

• QSA import fallback reproduced in the old launcher. Default native admission is rows >=64, so this is not proof of a decode regression for C4/C8/C16.

• Fixed a benchmark-only metadata defect: the old Page4 selection generator appended three tail tokens regardless of the actual open-group residue and could select future pages for early MTP rows. Canonical causal-tail CPU cases: 24 passed. Current-main QSA launch/route suite: 27 passed, one GPU-only skip.

• First GPU micro used that noncanonical generator and failed its FP32 oracle before timing. No speed or production-quality claim from that run. Corrected screen calls production index expansion, checks all four residues independently, and retains per-route failures without admitting their timings.

• Canonical QSA GPU screen completed: Triton versus direct XQA M4 53.162 vs 283.080 us, M8 98.955 vs 287.075 us, M16 166.763 vs 289.534 us. Reject native small-batch route; production threshold unchanged. Grouped M8/M16 passes FP32 tolerance but differs between eager/graph, so its timings are not admitted. No claim that those small differences prove model-quality degradation. Frozen old QSA source and pinned FA binary only.

• New-main benchmark sidecar (SM70, CUDA 12.8/Torch 2.10; SHA256 872f4e3a37cdf57c2e5ddd010c7b95cd968e4f8d6f5a64fd79fff87063c51f8a) now passes TP4 GPU packed/view/down-view screens and 64 changed-input auxiliary sum2 cycles at M4/M8/M16. It intentionally omits production M1 bindings and must not be deployed as an endpoint sidecar.

• New packed HC GPU pairs M4/M8/M16: 31.406/32.214/33.819 -> 26.898/26.626/27.885 us. All views save shard storage but retain only 2--6% speedup over their baseline, not the performance choice. Down-view pairs: 31.294/32.059/32.410 -> 26.989/27.309/27.000 us, avoiding a projected 165 MiB of extra weight copies per worker over 96 HC pairs. These are separate-process paired microbenchmarks, not proof of <=1% difference between packed and down-view or endpoint/model-quality results.

• CPU weight-view layout/alias/update suite: 16 passed. No new weight/activation quantization. Original replicated-GEMM association differences and all model-score gates remain unadmitted.

• Runtime integration at aeba8e2d32fb809d7ccd33e284f34c47d1a8423c: default-off VLLM_SM70_QWEN38_BATCH_HC_FP16, model-owned post-quantization up-shard preparation, zero-copy down views, dedicated per-TP communication channel, original M1 delegate and dynamic prefill/unsupported fallbacks. Full native owner sidecar preserves the M1 bindings omitted by the earlier benchmark-only build.

• Focused runtime/dispatcher CPU suites: 60 passed. All staged pre-commit hooks passed.

• Real GatedResidual + vLLM TP4 communicator GPU gate passed M4/M8/M16 routing, exact fallback at M1/prefill/unsupported rows, eight changed-input/poisoned-output graph replays, prefill-first dynamic compile, stable reload pointer and channel destruction. This remains component evidence, not full-model scores or throughput.

• Complete native sidecar SHA256: 3664deec1c713e3c4e0fe2bb5de22cc783a0eb99ebb7539a0376c39c270bdab2. Source plus sidecar, not a clean wheel.

• New full-model HC A/B completed at b64dae6e5b7d14eb4f0e0861d31600fc866c4e48, both arms using the same pinned native owner/FA package and grouped MoE. Original deterministic independent 8K/256 workload, Prefix+Mamba align, 256K max context, chunk 2048, maxseq 16, TP4 V100 and FULL decode graphs. C1 off/on 88.051/88.059 tok/s; C4 213.298/216.496, C8 361.266/364.129, C16 583.926/589.860 tok/s. Step savings 0.277/0.174/0.276 ms: real but small initial A/B deltas, not a repeated CI admission. The 238/420/728 targets remain unmet. Do not present the microbenchmark's ~14--17% HC gain as model throughput.

• Same first-16 GSM8K prompts/official sampling/natural EOS: both 15/16, same missed item, neither truncated. Generated lengths differ; health-check duration is not a speed result. Full coding/tool/schema/PPL quality admission remains pending, and this HC-only A/B does not admit grouped-MoE numerics relative to the original production control.

• Runtime scope audit: both arms' frozen base _C lack two newer M1-only MoE fusion ops and take the same fallback. M4 fused-batch and M8/M16 grouped branches do not depend on those ops. This is source+sidecar evidence, not a clean latest-main wheel. The forced-speed M1 continuation differs at token 12; retained as a diagnostic, not attributed to HC or treated as proof of degraded task quality. No default enabled.

• Concurrent tool/schema client added at b64dae6e5b7d14eb4f0e0861d31600fc866c4e48: reuses BFCL argument/name scoring and JSONSchemaBench validation; fixed 64 BFCL + 16 WashingtonPost cases pass CPU manifest/schema validation. This closes the coverage hole where serial requests would only exercise M1, not the batch candidate. No model/API score is claimed from the dry run.

• Combined runtime/dispatch/client suite: 72 passed. Client checks cover overlap at C1/4/8/16, deterministic seed/order, retained transport errors without retry, malformed names and truncated/invalid JSON. Full staged pre-commit passes.

• Concurrent API subset completed: all 80 HTTP 200, 53 natural tool_calls and 27 stop. Corrected subset scores: BFCL simple 14/16, parallel 12/16, multiple 14/16, irrelevance 11/16; JSON Schema 16/16. BFCL is 51/64, not quality admission without the original production control. Two prior false negatives were an alternative-value dictionary scorer bug, corrected offline without resampling. Dataset hashes/payloads/order are checked; eight real/negative vectors match the pinned official helper and 24 CPU client/scorer tests pass. Scorer repair commit b1d3bd1a0e.

• Reject the original API trace: all captured steps contained prefill and none had graph nodes. A replacement triggered only after eight consecutive confirmed C16 pure-decode outputs and captured 16 x 4 complete graphs, 2211 nodes each. All owned model/API processes exited. No resident API remains.

• Whole-graph attribution (middle 14/rank): envelope 26.789 ms, activity union 24.895 ms, service 27.053 ms, internal gaps 1.894 ms, overlap 2.158 ms. These are profiled diagnostics, not new accepted throughput. Major service families: W13 4.301, W2 2.284, MoE plan/reduce 0.268, QSA non-projection 3.824 ms. Source/stream-shape attribution avoids mislabeling shared cuBLAS kernels as LM-head; unknown kernels remain explicit.

• QSA scorer has a capacity-sized (16,1026) grid while the 8K workload needs only about 512--528 live CTAs. Benchmark-only contiguous grouping passes exact changed-length graph checks in 12 C1/4/8/16 x 8K/128K/256K cases. C16/8K 65.331 -> 48.179 us, but C16/256K regresses 1094.042 -> 1113.856 us. Reject unconditional grouping; no production default changes. A bounded grid-stride benchmark prototype now targets empty CTAs without truncating long-context coverage; GPU admission remains pending behind foreign ownership.

• Grid-stride follow-up is now complete (both jobs exited, owned GPU memory released). Both stage settings pass all 12 exact changed-input/length graph cases, but fail the long-performance gate. Stage-2 C16/8K 65.229 -> 44.954 us, C16/256K 1105.306 -> 1174.477 us for grid 64. Stage-1 grid 256 still regresses C16/256K 1104.128 -> 1118.720 us (1.32%). Compiled baseline/stride shared memory is the same 40960 bytes, registers 224/238, zero spills; no measured occupancy claim. Do not promote either fixed-grid/grouping scheme. The ~0.24-ms projected short scorer saving is too small for the remaining ~5.15-ms full-step gap; next focus is the measured 6.85-ms routed MoE family and sparse-attention forward. All production dispatch/defaults remain unchanged.

• MoE follow-up at eb40e649ac: W2 warp locality gives only ~2.2 us/layer on the captured C16 case; paired W13 gate/up input reuse is slower at every screened width. Tile-major FP16 scale layout saves ~3 us at C16 but both-projection layout regresses distinct M4 by 1.85%. All retained as benchmark-only negative/small results, not new engine throughput. Logical traffic audit from the old 48-layer route cohort is 3.239 GB/step; it is not a measured DRAM counter or current-trace roofline proof.

• New copy-only GDN batch fallback: preserve both original GEMMs and replace four output .contiguous() kernels with one bit-preserving copy. Runtime-source rotating-weight complete projection M4/M8/M16 59.430/59.504/60.774 -> 48.950/49.427/50.509 us; M2/M32/M64 also improve. ~0.37 ms across 36 layers is a projection, not endpoint evidence. Default-off VLLM_SM70_GDN_BATCH_SPLIT_COPY; unchanged M1 and unsupported fallback, no upper batch/maxseq/chunk/KV/model-name gate in this local copy admission.

• GDN copy gates: all 65536 FP16 payloads through all four stores; changed/poisoned graphs and unchanged GEMM results bitwise equal. New public-op CPU/GPU suite 35 passed, including real route-hit tracking at M2/4/8/16/17/32/64 and unchanged M1. Related CPU suite 74 passed, 8 skipped, 3 GPU-only deselected. The initial CPU-only run incorrectly selected three old HC GPU tests that failed at CUDA initialization; no operator failure was masked. Full staged pre-commit passes. No dataset or full-model admission claimed for the new copy path.

• Original-production API quality control completed at eb40e649ac with identical retained 80 cases, payloads, seeds and hashes; grouped MoE, batch HC and the new copy flag explicitly off. Original BFCL 54/64 versus earlier batch candidate 51/64; Schema both 16/16. All 80 control requests succeed and naturally finish. Four candidate losses/one win are retained, not rescored away. Small stochastic subset is not proof of causal degradation (one-sided paired sign p=0.1875), but does not pass noninferiority. Separate grouped-MoE/HC diagnostic ablation is next; do not attribute this signal to the new copy kernel, disabled in both datasets. API/workers exited, GPUs released; no owned API or queued GPU job remains.

• Factorial API quality at b6c18db202: same 80-case manifest, original/MoE-only/HC-only/both BFCL 54/51/52/51 out of 64, Schema 16/16 throughout. Different loss sets and first-token differences prevent causal attribution to either decode flag. A later unchanged original/both replay scores 55/54, retained alongside earlier results rather than replacing them. All natural requests complete; no noninferiority claim.

• Diagnostic-only fixed original continuations: all 3654 tokens reproduced in both arms, raw-logprob reporting and all four ranks' seed/position coverage validated. Decode mean candidate-minus-control NLL -0.0009651, paired per-case bootstrap 95% interval [-0.0031603,+0.0015295], but maximum absolute token delta 2.6824 nats. Matching pure-C16 subset mean +0.0013119. Not a corpus PPL score or model-quality admission. Prefill states and online batches remain confounders. Existing [Bugfix][SM70] Stabilize QSA page4 order under KV relocation #494 planner repair will be validated as a dependency; its entire older qsa.py must not replace current batch/sidecar/gating changes.

• Reject both benchmark-only one-N32-tile-per-CTA W13 variants. Unbounded captured M4/M8/M16 complete MoE 58.886/90.502/125.811 -> 62.989/91.322/128.250 us; bounded-register C16 126.093 -> 139.232 us, despite restoring a 48-warp static resource ceiling. Exact intermediate/final graph cases pass but speed fails. No endpoint reruns or production switches for these failed variants. Recorded in commit 46f2b79022; full staged pre-commit passes.

• Existing [Bugfix][SM70] Stabilize QSA page4 order under KV relocation #494 was built and validated as an artifact-only dependency, not duplicated/merged or used to replace the entire newer QSA file. New FA SO SHA256 b439320b4cd67c0c1c59d41277401a32472770c9995cc82f3b3eb604edb47434. PR regression + current QSA + added 8K/page784 tests: old 25 failed / 24 passed, repaired 49 passed. Repair composes with current two-warp/output-gate/selector code.

• Fixed-package original and MoE+HC API arms: same full 80 cases, BFCL 53/64 each, Schema 16/16 each, all natural requests successful. All 3654 diagnostic trajectory tokens also match; matched pure-C16 mean delta NLL +0.00005284, but a 2.1801-nat token tail remains. Equal aggregate scores do not establish full noninferiority; no attribution of all tails to [Bugfix][SM70] Stabilize QSA page4 order under KV relocation #494. GDN copy and the new attention prototypes remain disabled in these model tests.

• Repaired grouped sparse attention now passes changed-input/slot-map/tail-residue eager/graph checks, but its small-batch launch has only 1/1/2 forward CTAs at C4/C8/C16. C16 is 3690.357 us versus current Triton 166.126 us, so do not lower the production admission threshold.

• A benchmark-only split-KV adapter reuses the native arithmetic across ~80 CTAs. C16 improves to 266.766 us, still slower than Triton. Isolated C16 phase graph service: planner 126.149, pack 7.451, forward 114.710, merge 6.213 us, not an E2E additive closure.

• A second benchmark-only per-query logical-order plan removes hash union, while keeping independent query masks. Paired Triton/candidate C4 53.574/94.070 us, C8 99.203/96.672, C16 167.611/145.221. All exercised eager/graph/FP32 checks pass (max relative L2 <0.000360), but C4 regresses and no model/prefix-sharing/E4M3 quality admission exists. C16 projects only 0.269 ms over 12 layers, not new full-step throughput. Do not ship an exact-C16 switch. Next inspect a smaller native independent-request GQA tile instead of blindly varying split counts.

• Fixed the legacy attention benchmark's positional-call incompatibility with the newer optional output-gate argument; use explicit keywords. Initial failed benchmark/phase harness attempts are retained and not counted as arithmetic or speed evidence. Canonical-index CPU tests 24 passed; staged pre-commit passes. All owned API/micro jobs exited, ports 18201/18202 closed, owned GPU allocations released.

Detailed provenance, commands and raw artifact SHA256 values are in docs/design/sm70_qwen38_nomtp_concurrency.md. The opt-in dispatcher, initial full-model A/B and refreshed C16 attribution are complete; full performance/quality goals remain open. No new production default enabled and no silent relabeling of the old baseline.

[yangzhuxinyzx]

Maintainer-requested partial integration completed: #549 merged only the copy-only batched GDN projection split, with unchanged GEMM arithmetic and latest-main GEMV role plans. 61 targeted tests, exhaustive FP16 payload checks and CUDA Graph replay passed; actual-layer complete-projection timing improved 11–19%. The copy flag defaults on within its existing four-copy fallback, without enabling other opt-in arithmetic. This PR remains open for the independent HC/QSA experimental remainder; it has NOT been merged wholesale or closed.