[heydryft]

The 794 ms decode step, itemised — and the 91 ms nobody had opened

Two peers own the two biggest kernels (fp8_matmul_tiled 524 ms, qtip2b_grouped_gemm 69 ms). This PR takes apart everything else: the 67 ms lm_head, the 43 ms copy storm, and the 91 ms that had never been examined. Plus four correctness defects, three of which are gates that could not fail.

Nothing here is measured on hardware. Every number below is derived from V4-Flash's shipped config.json against H200 peaks (4.8 TB/s, 989 TFLOP/s dense BF16, 67 TFLOP/s FP32 scalar). No speedup is claimed anywhere. Where a derivation cannot close, it is reported as not closing.

---

The budget

Ground truth: 43 layers (2 Standard, 21 CSA/ratio-4, 20 HCA/ratio-128), hidden 4096, 64 heads × head_dim 512, 1 KV head, 256 routed experts / top-6, vocab 129,280, first_k_dense_replace=0 so every layer is MoE. Context-dependent rows are given at C=2048.

item	ms/step	share	owner
fp8_matmul_tiled	524	66.0%	peer
qtip2b_grouped_gemm	69	8.7%	peer
lm_head	67	8.4%	fixed here
copies (ucopy_bf16 + copy2d_bf16)	43	5.4%	partly fixed here
the remainder	91	11.5%	named here

lm_head — 67 ms, and 244 ≈ B = 256 is the whole answer

Bound: max(2·256·4096·129280 / 989 TFLOP/s, 4096·129280·2 B / 4.8 TB/s) = max(0.274, 0.221) = 0.274 ms. Measured 67 ms is 244× over. Back-solving the traffic: 256 × 1.06 GB = 271 GB per step, 271 GB / 67 ms = 4.05 TB/s = 84% of peak HBM bandwidth. The kernel is not slow. It is streaming the weight once per batch row.

Cause: every activation is rank-3 [B, T, hidden]. UnquantLinear::forward took w.broadcast_left(B) — a stride-0 view — and both dispatch arms then treated B as a GEMM batch dim with stride_b = 0. At decode T = 1 that is B independent m = 1 GEMVs over one 1.06 GB weight that does not fit in L2. Fixed by flattening to [B*T, hidden].

The same defect was fixed one level up in BlockwiseFP8Linear by 445b920. That did not cover lm_head, because DeepSeek ships lm_head.weight in BF16 with no weight_scale_inv, so blockwise_fp8::linear_no_bias short-circuits straight to UnquantLinear. The defect is in UnquantLinear; fixing it there is the general case and covers every bias: None rank-3 linear.

CPU is deliberately excluded, and the reason is worth reading. The first push broke v4_xs_history_two_seq_batch_matches_single_sequence on ubuntu and nowhere else. Not a leak: max diff 0.0625 against a 0.0399 tolerance — 1.6× over, where that test documents a real leak as O(1), ~100× over. Probing it: MatMul::matmul casts to F16 on CPU, and before the flatten broadcast_left made the batched case B separate m = 1 GEMMs — the same shape, so the same accumulation order, as the solo run it is compared against. Batched-vs-solo was bit identical, max_diff exactly 0; that 1e-2 tolerance had never been exercised in its life. Flattening makes it m = B, and x86's F16 microkernel blocks m = 2 differently from m = 1 (aarch64 does not, which is why macOS stayed green).

The fix was not to widen the tolerance — widening a leak detector to admit a change made in the same PR is how a gate stops being a gate. The flatten is device-gated instead, on the boundary the derivation always had: the defect is HBM traffic, there is no HBM on the CPU path, and CPU exists here to be a reference oracle the whole V4 batched-vs-solo suite is calibrated against. The fix lands where the 271 GB/step is; the oracle keeps its arithmetic.

The copy storm — 22,794 factorises exactly

22,794  =  84 × 256      +  43 × 30
        =  cache_slots×B    layers × per-layer forward copies

84 = 43 KV + 41 xs: V4 appends one XsRolling compressor-history slot per CSA/HCA layer after the KV entries, and NormalCacheManager derives its loop bound from the cache vector's length. So 94.3% of the strided copies are the per-sequence cache scatter/gather — one strided narrow().contiguous() per sequence per cache slot per step. A second exact factorisation, 256 × (2×43 + 3) + 10, lands on the same family.

That is a layout problem, not a kernel problem, and the fix is architectural (a persistent batched cache with per-row indices — the design paged_attention/ already has and V4 opted out of via KvCacheLayout::Standard). Scoped, not attempted here. What is fixed here is a pure duplication in the same bucket: V4 is MQA and deepseek4.rs calls dsv4_attention(&q, &k_cached, &k_cached, …), so the raw-window narrow and the compressed cat were each done twice over identical bytes — 24.2 GB/step for the concat pair at C=2048, half of it duplicate, plus 5.8 GB/step for the narrow pair.

The 91 ms — named, with the residual stated honestly

Bytes cannot explain it. 91 ms at 4.8 TB/s is 437 GB; every un-owned kernel in the step touches 36 GB in total. Bandwidth/FLOP work in un-owned kernels is 7.5 ms at C=2048 (11.7 ms at C=8192). So the 91 ms is launch count, and the arithmetic is 47,120 un-owned launches × floor:

per-launch floor	launch cost	+ 7.5 ms work	vs 91 ms
1.05 µs (this repo's measured in-graph floor)	49.5	57.0	names 63%, residual 34 ms
1.77 µs (back-solved)	83.5	91.0	names 100%
2.6 µs (this repo's measured on-stream floor)	122.5	130.0	overshoots by 39 ms

I can name 57–91 of the 91 ms, and at the repo's own on-stream floor the arithmetic overshoots. That is not forced to fit. What does not depend on the floor:

~114,200 kernel launches per step, and 110,080 of them — 96.4% — come from two for loops over the 256 sequences, each running once per layer per step:

• layers.rs DeepSeekV2RotaryEmbedding::forward — 66,048 launches/step
• layers.rs forward_inverse_tail — 44,032 launches/step

They move 0.8 GB total (0.17 ms of bandwidth). They are ~100% overhead by construction. The trigger is one condition: seqlen_offsets.len() is 256 at B=256, not 1, so three sites take their per-sequence branch — including deepseek4.rs's fused qk_norm_rope kernel, which declines outright with decline!("per-sequence position offsets") and hands the work to a 1,548-launch-per-layer eager path.

66,048 of those launches are copies and therefore sit inside the already-named 43 ms bucket, which independently implies an effective per-copy-launch cost of ~0.4–0.6 µs once the ~15.7 ms of genuine copy bandwidth is subtracted. Either the effective floor is ~1.8 µs, or the 43 ms copy bucket is understated. Both readings point at the same two loops, and the remedy is the same either way, so the ambiguity does not need resolving to act.

PR #198 is the fix for these two loops, and it is sitting unmerged. It replaces seqlen_offsets.len() == 1 — a test of length — with uniform_offset(), a test of values, in both forward and forward_inverse_tail. That collapses 99,072 launches/step to 301.

But note the boundary the 91 ms analysis exposes: uniform_offset returns None the moment two sequences sit at different positions ([7, 7, 8] → None). A uniform cohort is what the default scheduler admits, so #198 is the right fix for that case — but in a serving mix with staggered arrivals the offsets are not uniform and the 110,080-launch loop fires again. The general fix is one index_select of the cos/sin tables by a [B] position tensor followed by a single batched rope_i, which works for any offsets; the machinery already exists a few functions down in forward_at_positions. That is a follow-up on top of #198, not a criticism of it.

Not double-counted: #198 removes eager_qk's data movement, which is disjoint from the cache-manager copies that make up 94.3% of the strided copies.

Also un-owned, and not GPU time: 172 pageable H2D transfers per step from Tensor::arange/Tensor::full inside the dsv4_attention mask build, plus 256 one-element H2D round trips in inputs_processor.rs. These serialise the host and make the decode forward uncapturable.

The second of those is fixed here. make_completion_chunk built one Tensor::new(.., device) per sequence and cat-ed them — B pageable 1-element H2D transfers on the legacy default stream, each a full round trip, plus one copy2d per cat argument. At B=256 that is 256 synchronous transfers and 256 copy kernels to assemble a [256, 1] tensor holding one kilobyte. It was already named in-file as "CLAUDE.md pitfall #5, still live" and already instrumented with its own sync_span — named, measured, and left running. Rows now stay on the host and cross once: 256 round trips and 256 launches become 1 and 0. Ragged input is refused rather than reshaped, because the dangerous case divides evenly — rows of 1, 2 and 3 tokens reshape cleanly to [3, 2] and hand sequence 0 a token belonging to sequence 1, with a correct shape and somebody else's contents.

Ruled out by reading: the C4 indexer is built and never called — zero use sites in forward; it costs HBM, not time. All v4_*_dbg hooks are env::var_os early-returns. The MoE per-expert loop emits index_select/index_add, not ucopy_bf16 — 43 × 256 = 11,008 is a red herring.

Top remaining un-owned items, all scoped and none attempted here: (1) the two per-sequence RoPE loops beyond #198's uniform case; (2) re-enabling the fused qk_norm_rope kernel above batch 1 by taking a per-row [B] position tensor instead of a scalar; (3) the MoE router GEMM runs in FP32 on CUDA cores — 23.1 GFLOP/step at 67 TFLOP/s, ~15× the cost of the same GEMM on tensor cores, plus a 0.27 GB activation cast to feed it (a peer has a branch on this); (4) V4MHCHead::forward is the one mHC site never fused — 15 launches where the structurally identical hc_pre next door is 1.

---

Correctness

The citation gate had never opened a .rs file. doc_citations.rs claims in its own header to verify that sentences describing the code are true. It scanned ["memory", "arc-tools"] filtered to .md — so it checked the mission record and never once opened a source file, while ~310 citations sat in Rust doc comments pointing into the fastest-moving code in the repo. Widening it needed two things the one-line change did not anticipate:

• A .rs line cannot be scanned whole. Citation-shaped text lives in string literals — this file's own BASELINE is a table of them and its fixtures build citations that are deliberately wrong. Scanning those manufactures SymbolRot, which has no waiver path, so the gate would have been unlandable against its own source. split_rust lexes comments out: nested /* */, \-continued and multi-line raw strings, 'x' char literals kept distinct from 'a lifetimes. A per-line scanner is not enough — the first fixture written here was a multi-line raw string, and a per-line scanner reported every deliberately-broken citation in it as real.
• The widening silently disarmed the one unwaivable verdict. Once .rs files were documents, symbol_exists_anywhere started finding a cited symbol in the comment that cited it. Every SymbolRot degraded to the waivable SymbolMoved, for the whole new corpus, and nothing would have said so. The symbol search now reads comment-free code — the same rule .md already gets.

Census 972 → 1,323 citations, 0 live findings. CITATION_FLOOR 700 → 1,100, deliberately above the .md-only total so a scope shrink fails. The 130 findings that surfaced are all unresolved paths into upstream trees Arc reads and does not vendor, pinned as generated BASELINE rows. No SymbolRot accompanied the widening — all ~310 Rust-comment citations name symbols that still exist.

The rental gate certified "V4 fits in 60 GB" from a run that loaded on CPU. used_bytes = free_before.saturating_sub(free_after) is 0 when nothing became resident; the verdict was total_gb <= target, i.e. 0.0 <= 60.0, i.e. true. It printed PASS, wrote "pass": true into the JSON the rental step reads, and exited 0. The zero case was detected — it pushed a free-form note, which changes no exit code and which nothing reads. Now three-valued: 0 fits, 1 measured-and-does-not-fit, 2 we do not know. Both new functions live outside #[cfg(feature = "cuda")] on purpose — the defect sat in CUDA-gated code no CPU CI job compiles, so a test beside the bug would have been as unreachable as the bug.

Seven cited mission docs existed only on one laptop. Untracked, not gitignored, just never added — including CEILINGS.json, the file this project's rules order an agent to read before quoting any speed number. Committed at their cited paths. The seventh citation was rot, not absence: segmented_allocator.rs cited wave59-CJ-mtp-batch.md §3, which exists nowhere; the claim is real and is in wave45-BW-mtp-batched.md §2. Both fixed, and the quoted formula now matches the document verbatim.

Worth naming: the widened gate did not catch these seven. It matches path.ext:N, and all of them are cited by bare filename or §. That is a real hole in the widening and a follow-up — not a reason to loosen a rule that currently has zero false positives.

PR #199 (the attention mask fix) is shepherded, not merged here. Verified against master: merges cleanly, 0 conflicts, 2 files, all 18 checks green. mergeStateStatus: BLOCKED is REVIEW_REQUIRED only. It needs a human approval — nothing else.

This PR targets master, not #194. It was opened against release/openrouter-ready and CI's Base branch lane rejected it by design: "This PR targets '…', not 'master'. Its lanes compile the head merged into another PR's branch, which is not the tree that will land." Retargeted. Worth flagging that #195 and #198 are red for exactly this reason — their only failing lanes are Base branch and the CI complete roll-up that depends on it. The two loops #198 fixes are 96.4% of the step's kernel launches, and the PR is failing on its base ref, not its content.

---

What has a real test, and what does not

change	test	kind
UnquantLinear flatten	rank-3 vs 2-D agree across four [B,T] splittings ± bias inside an F16 budget (the gap that motivates the device gate); the CPU oracle is pinned bit-for-bit against the pre-change formulation; the extracted shape arithmetic round-trips rank-3, rank-4 and a non-contiguous input	real, CPU
K/V aliasing	same inputs aliased vs un-aliased, both branches, bit-identical output; is_same_tensor is identity not equality	real, CPU
decode cohort H2D	single transfer is byte-identical to the per-row cat, on [B,1] and a wider rectangle; a ragged cohort that divides evenly is refused and the error names the row; empty batch errors	real, CPU
citation gate	2 new fixture tests: exact citations extracted from a fixture with an escaped quote, a raw string containing //, and a lifetime; a dead symbol in a Rust comment still hard-fails	real
rental gate	zero delta is not a pass; 40 GiB still passes and 80 GiB still fails; exit mapping stays three-valued; legacy JSON parses	real
every performance number above	none	UNVERIFIED ON HARDWARE — derivation only

cargo test -p mistralrs-core (all targets) 652 + 11 + 12 + 7 + 17 + 2 passed · -p mistralrs-quant --lib 332 passed · -p arc-cli --bins validate 15 passed · scoped clippy lane green · cargo check --workspace --tests clean · formatting scoped to touched files only.

[github-actions]

Code Metrics Report

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 Language              Files        Lines         Code     Comments       Blanks
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 C Header                  5          305          210           52           43
 CSS                       2         1181         1036           34          111
 CUDA                     78        27867        19381         5568         2918
 Dockerfile                1           39           22            8            9
 JavaScript               16         3546         2676          482          388
 Jinja2                    7          694          656            5           33
 JSON                     74         4600         4597            0            3
 Makefile                  1            6            5            0            1
 Metal Shading Lan|       33        12224         9431         1142         1651
 PowerShell                1          300          227           30           43
 Python                  147        15371        12677          824         1870
 Shell                    42        10258         6849         2734          675
 Plain Text                4         3801            0         2479         1322
 TOML                     33         1498         1294           54          150
 YAML                      3           25           23            2            0
─────────────────────────────────────────────────────────────────────────────────
 HTML                      4         2687         2604           43           40
 |- CSS                    2          543          479           37           27
 |- JavaScript             1         1233         1215           12            6
 (Total)                             4463         4298           92           73
─────────────────────────────────────────────────────────────────────────────────
 Jupyter Notebooks         4          122           83           23           16
 |- Markdown               1           60           30           22            8
 |- Python                 1          122          113            1            8
 (Total)                              304          226           46           32
─────────────────────────────────────────────────────────────────────────────────
 Markdown                204        45330            0        35183        10147
 |- BASH                  72         1655         1203          331          121
 |- C                      3           17           17            0            0
 |- CUDA                   2           84           56           16           12
 |- JSON                  18          708          708            0            0
 |- PowerShell             1            1            1            0            0
 |- Python                23         1008          787          113          108
 |- Rust                  66         2051         1716           77          258
 |- TOML                   6          207          164            0           43
 |- YAML                   5           41           36            5            0
 (Total)                            51102         4688        35725        10689
─────────────────────────────────────────────────────────────────────────────────
 Rust                    678       336264       289507        17400        29357
 |- Markdown             500        30663          471        26449         3743
 (Total)                           366927       289978        43849        33100
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 Total                  1337       504511       358274        93126        53111
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━