[heydryft]

Parent system: ArcInfer / ArcAttention. Three-job lane: one correctness (ship-blocking), two performance. Base: release/openrouter-ready @ 9a5921b.

1. Correctness — the Metal sinks arm still dropped the caller's mask

#199 (7df05f62a, the rebase of 8dbdd1a87) gated the CUDA fused arm on mask.is_none() and gave flash_attn_sinks a mask parameter so it can refuse one. The Metal arm got neither: it engaged on head_dim alone and called flash_attn_sinks_metal — which has no mask parameter at all, so the interface cannot even refuse. Reachable today: GPT-OSS on any macOS build (head_dim 64 is in Metal's fused set {64,80,96,112,128,192,256}, and gpt_oss.rs passes a mask it assert!s is Some).

Measured on a real Metal device (M-series, head_dim 128, F32, metal_sinks_mask_tests):

max|fused − reference(causal only)| = 0.000001
max|fused − reference(masked)|      = 0.860029
max|ref(causal) − ref(masked)|      = 0.860029   <- the mask's whole effect

The maskless kernel tracks the causal-only reference to 1e-6 and misses the masked one by the mask's entire effect — dropped, not approximated. Fix: the same one-line && mask.is_none() gate as CUDA's, plus a once-per-process warn naming the deliberate perf loss.

Mutation checks (values both ways):

Test	Fix in place	Guard/mask reverted
masked_sinks_attention_on_metal_matches_the_masked_reference (real Metal device)	max abs diff 0.00000000 (pass)	0.8600293 (fail)
padding_mask_is_honored_and_dropping_it_changes_the_output (CPU, always runs; padding-column mask — the class no fused kernel can re-derive)	max abs diff 0.0 (pass); teeth: masked-vs-maskless signal 0.9353 > 1e-2	0.9353 vs 0.0 required (fail)

Not done here (out of fence, mistralrs-quant): giving flash_attn_sinks_metal a mask parameter so the Metal kernel can refuse at the interface the way the CUDA one now does. Until then the dispatch gate is the only line of defence on Metal.

2. V≡K duplicate — already landed on this base; premise re-verified

The v_aliases_k dedup shipped in #204 (00f31e3e1): handle-identity test (Tensor::id) at both duplication sites (the raw narrow(..).contiguous() and the union Tensor::cat), with engagement counters (fused_v_stats). Re-verified against the current base:

• deepseek4.rs still passes the same binding twice at three of four call sites (&k, &k @1888; &k_full, &k_full @1949; &k_cached, &k_cached @2043) and genuinely distinct tensors at the fourth (&k_full, &v_full @1874) — the dedup fires on the first three only.
• Existing tests cover exactly what this lane was asked to add: aliased_v_is_bit_identical_and_engages (bit-equality of outputs via to_bits(), counter deltas ≥ +2), alias_predicate_is_handle_identity (a copy() with equal values must NOT alias — the control), distinct_v_is_not_aliased (a different V must change the output). All pass on this branch.

No code change needed; nothing added.

Live call-site trace (does the dedup actually fire?) — per review request, report-only (deepseek4.rs is outside this lane's fence):

Arm	Site (current lines)	K/V handles	Dedup
PagedAttention serving (paged_attn Some + metadata)	&k_full, &v_full @ 1874-1875	new-id — let v = k.copy() @ 1863 (deliberate: reshape_and_cache write-locks and read-locks the same storage, so aliased k/v self-deadlock the RwLock during the cache WRITE), then cache_write_and_gather gathers K and V from two separate pools	never fires
Dummy / imatrix (paged, no metadata)	&k, &k @ 1889-1890	same binding → same TensorId	fires
CUDA-graph decode (has_graph_mode_positions && seq_len 1)	&k_full, &k_full @ 1949-1950 (one append_graph_kv_mqa result)	same-id	fires
Dense NormalCache path	&k_cached, &k_cached @ 2043-2044 (one cached.span(..) result)	same-id	fires

On a box serving with PagedAttention ON, fused_v_stats will read (0, N) — the dedup is a no-op there. The one-line change that fixes it (NOT made, outside fence): pass &k_full, &k_full at deepseek4.rs:1874-1875. The v = k.copy() at :1863 must STAY — it protects the cache write, not the dispatch — but the gathered v_full is byte-identical to k_full (the value pool was written from a byte-copy of k and gathered with the same block tables), and dsv4_attention is purely functional over k/v (narrow/contiguous/cat/matmul, no in-place ops), so handing it k_full twice is safe and bitwise identical. Deeper follow-up (also outside fence): with that in place the V-pool write+gather is dead weight on V4 — skipping it is the real capacity/bandwidth win, but it touches paged_attention internals.

3. Union-mask rebuild — memoized across layers

The union mask is a pure function of nine integers, yet dsv4_attention rebuilt it once per layer per step. The prompt's three arange sites (:759/:762/:786) had partially moved since: a7b0890d9/64fe1d678 already killed the decode-path aranges (all-ones raw mask + the cached compress_positions table). What remained: the prefill path still ran 3 aranges + casts + compares per layer, and both paths still ran the U8 cat + 0/−inf conversion per layer.

This PR makes the validity build lazy (build_valid, character-for-character the old code on a miss) and memoizes the additive [1,1,t_q,n_keys] mask thread-local on the integer key (UnionMaskKey, 4 FIFO slots — one step touches at most 3 keys: the Standard/CSA/HCA classes). Each distinct mask is now built once per step instead of ~once per layer. Deliberately excluded: graph_positions (constant key across steps; ops must stay inside the capture region, not be hoisted into a warmup-owned tensor whose eviction would leave a recorded graph reading freed memory) and ragged cohorts (per-row leads are content but not key).

Rider in sinks.rs: sinks_attn_cpu_varlen rebuilt its per-sequence mask with Tensor::from_vec (an H2D upload per sequence per layer per step on the GPU fall-through) — now a single-slot memo, varlen_causal_mask_cached. The to_vec1 D2H syncs on cu_seqlens in the same function remain: hoisting the host copy into FlashParams is shared plumbing, reported not fixed.

Mutation checks (values both ways):

Test	Fix in place	Memo key-matching broken ("serve newest entry")
memo_hits_without_building_and_misses_on_a_changed_key	pass (hit returns cached handle without running the builder; advanced q0 misses)	fail: TensorId(1) != TensorId(6) — "an advanced position was served the previous position's mask"
memoized_masks_are_bit_identical_to_fresh_builds (ctx 35→36, same mask shape, different admitted block set)	pass, bit-identical to fresh-thread references	fail: ctx=36 output bits diverge from the fresh reference (first lane 1051680768 vs 1051066368)
varlen_mask_memo_hits_and_never_serves_stale	pass	fail on the (5, 8, 3) entry with the key comparison removed

Premises verified / refuted

• Refuted: "the CUDA arm and Metal arm both drop the mask" — the CUDA arm was already fixed on this base by 🔴 fix(ArcAttention): the fused head_dim=512 sinks kernel silently discards the mask — 4-day-old regression, default ON #199; only Metal remained. 8dbdd1a87 is not an ancestor, but its rebase 7df05f62a is.
• Refuted: "Job 2's v_cat is an unconditional byte-identical duplicate" — already fixed by Eight cited defects that were reaching nobody: 4 wrong answers (default-on), 3 capacity, 1 dead loader #204 on this base, with the exact tests requested.
• Partially refuted: the three per-layer aranges at :759/:762/:786 — the decode-path ones were already gone; the prefill-path ones (now :888/:896/:950-area) were real and are fixed here.
• Confirmed: the file's sinks_attn_cpu naming/doc issue — docs corrected (device-agnostic fall-through, not CPU-only); the function is not renamed in this PR as instructed.
• Confirmed: GPT-OSS mask reachability (gpt_oss.rs asserts attention_mask.is_some() and forwards it into the sinks dispatch).

Not done / out of fence

• No edits to deepseek4.rs, layers.rs, scheduler, sampler, or mistralrs-quant.
• No new env flags (nothing to register in capability_reachability.rs).
• CUDA-device runs of the CUDA-gated tests (no CUDA hardware in this lane); the Metal twin ran on real hardware instead, and the CPU tests validate the shared dispatch logic.
• The two pre-existing rustfmt deviations in dsv4_attention.rs (the q0 checked_sub block) and the pre-existing clippy cast lints are untouched — present on base, out of scope.

🤖 Generated with Claude Code

https://claude.ai/code/session_01WK8VgocBUrr5djjE1ZfCNf

[github-actions]

Code Metrics Report

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 Language              Files        Lines         Code     Comments       Blanks
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 C Header                  5          305          210           52           43
 CSS                       2         1181         1036           34          111
 CUDA                     81        29475        20161         6275         3039
 Dockerfile                1           39           22            8            9
 JavaScript               16         3546         2676          482          388
 Jinja2                    7          694          656            5           33
 JSON                     75         4896         4893            0            3
 Makefile                  1            6            5            0            1
 Metal Shading Lan|       33        12224         9431         1142         1651
 PowerShell                1          300          227           30           43
 Python                  147        15371        12677          824         1870
 Shell                    42        10304         6858         2769          677
 Plain Text                4         3801            0         2479         1322
 TOML                     33         1498         1294           54          150
 YAML                      3           25           23            2            0
─────────────────────────────────────────────────────────────────────────────────
 HTML                      4         2687         2604           43           40
 |- CSS                    2          543          479           37           27
 |- JavaScript             1         1233         1215           12            6
 (Total)                             4463         4298           92           73
─────────────────────────────────────────────────────────────────────────────────
 Jupyter Notebooks         4          122           83           23           16
 |- Markdown               1           60           30           22            8
 |- Python                 1          122          113            1            8
 (Total)                              304          226           46           32
─────────────────────────────────────────────────────────────────────────────────
 Markdown                213        48932            0        38081        10851
 |- BASH                  72         1655         1203          331          121
 |- C                      4           19           19            0            0
 |- CUDA                   2           84           56           16           12
 |- JSON                  19          779          779            0            0
 |- PowerShell             1            1            1            0            0
 |- Python                23         1008          787          113          108
 |- Rust                  68         2063         1727           78          258
 |- TOML                   6          207          164            0           43
 |- YAML                   5           41           36            5            0
 (Total)                            54789         4772        38624        11393
─────────────────────────────────────────────────────────────────────────────────
 Rust                    681       340988       293252        18096        29640
 |- Markdown             504        32562          471        28063         4028
 (Total)                           373550       293723        46159        33668
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 Total                  1353       516771       363188        99077        54506
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━