From 2d38ccf0806b7a409b987b7f3766024d9d0855fb Mon Sep 17 00:00:00 2001 From: Nirupam Bhowmick <48842933+heydryft@users.noreply.github.com> Date: Fri, 21 Aug 2026 13:17:08 +0100 Subject: [PATCH] docs(ArcAttention): register the V4 compressed-KV re-RoPE recompute (wave65-CQ) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit V4 decode re-applies compress-theta RoPE to the ENTIRE compressed KV cache on every step, on every one of the 41 compressed layers, to recompute a rotation that `compressed_kv_from_rows`'s own doc comment says "depends on nothing but `j`". The rows are cached pre-RoPE; the rotation on top of them is not. Derived from shapes at cc5487ad3, not measured: ~22 MB/token of redundant traffic at 2048 ctx, 123 launches/token in copies plus 82 in cos/sin gathers. CSA re-rotates an unchanged row set on 3 of every 4 steps, HCA on 127 of 128. Surfaced, not fixed, and the file says why: the row set arrives through two accessors with different aliasing (`advance` vs `compressed_rows_fixed` across `pin_comp_capacity`), and a stale entry is a silently-wrong distant-context branch rather than a slow one — the RUN-161 / ARC_V4_FP8_KV failure shape. Settling it needs a GPU. Filed as a mission record because the repo has GitHub issues disabled. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01UMmjFy8TvsgypxVWNVhhC7 --- .../wave65-CQ-v4-compressed-rope-recompute.md | 129 ++++++++++++++++++ 1 file changed, 129 insertions(+) create mode 100644 memory/mission/wave65-CQ-v4-compressed-rope-recompute.md diff --git a/memory/mission/wave65-CQ-v4-compressed-rope-recompute.md b/memory/mission/wave65-CQ-v4-compressed-rope-recompute.md new file mode 100644 index 000000000..3fcf377e8 --- /dev/null +++ b/memory/mission/wave65-CQ-v4-compressed-rope-recompute.md @@ -0,0 +1,129 @@ +# wave65-CQ — V4 decode re-RoPEs the whole compressed KV cache, every step, every layer + +Parent system: ArcInfer / ArcAttention, with a dependency on ArcKV (`XsRollingCache`). + +Status: **surfaced, not fixed.** Deliberately. See "Why this was not shipped". + +Found while censusing pure-data-movement ops on the V4 batch=1 decode path for +PR #206. Every line number below is read from source at `cc5487ad3`. **No +hardware leg was taken; nothing here is measured on a GPU.** The byte counts are +arithmetic from tensor shapes, the launch counts are from reading the candle +call chain. Both are stated as derivations, not measurements. + +--- + +## What happens + +`Attention::forward` calls `compressed_kv_from_rows` +(`mistralrs-core/src/models/deepseek4.rs:1409-1422`) on **every decode step**, +for **every compressed layer**. That function hands the entire compressed KV row +set to `DeepSeekV2RotaryEmbedding::forward_at_positions` +(`mistralrs-core/src/layers.rs:1826-1843`), which re-applies compress-theta RoPE +to **all `t_c` rows** from scratch: + +```rust +let x_nope = x.narrow(D::Minus1, 0, nope)?; +let x_pe = x.narrow(D::Minus1, nope, rope_dim)?.contiguous()?; // real copy +let positions = positions.to_dtype(DType::U32)?; // no-op; already U32 +let cos = self.cos.index_select(&positions, 0)?; +let sin = self.sin.index_select(&positions, 0)?; +let rotated = candle_nn::rotary_emb::rope_i(&x_pe, &cos, &sin)?; +Tensor::cat(&[&x_nope, &rotated], D::Minus1)?.contiguous() // 2 copies of the full [1,1,t_c,512] +``` + +The rotation applied to compressed row `j` is a pure function of `j`. +`compressed_kv_from_rows`'s own doc comment says so, in as many words: + +> Row `j` sits at absolute position `j * ratio` and its rotation depends on +> nothing but `j`, which is why the rows can be cached pre-RoPE. + +The rows **are** cached pre-RoPE, by the rolling compressor. The rotation on top +of them is not. So it is recomputed for every row on every token, even though +the rolling compressor only ever *appends*, and only appends once every `ratio` +tokens. + +The one thing already memoized here is the position vector itself +(`compressed_row_positions`, `deepseek4.rs:3013-3037`) — the thread-local there +is the pattern the rest of this wants. + +--- + +## Cost (derived from shapes, not measured) + +V4 Flash geometry from `deepseek4.rs:5988-6047`: 43 layers, `compress_ratios` +giving **2 Standard + 41 compressed**; `head_dim` 512, `qk_rope_head_dim` 64. + +At 2 048 ctx: + +| | `t_c` | bytes moved / layer / token | layers | per token | +|---|---|---|---|---| +| CSA (ratio 4) | 512 | ~1.05 MB across the narrow-copy and the two-input cat (read+write) | 21 | **~22 MB** | +| HCA (ratio 128) | 16 | ~33 KB | 20 | ~0.7 MB | + +Launches, on top of the bytes: + +* **123 / token** in copies alone — `narrow(..).contiguous()` plus the two-input + `cat`, x41 layers. +* **82 / token** in the two `index_select` gathers, x41 layers. + +Redundancy rate: for CSA the row set is unchanged on **3 of every 4** steps; for +HCA on **127 of every 128**. Essentially all of the above recomputes a value that +did not change. + +--- + +## What a fix needs + +1. A **post-RoPE cache on `XsRollingCache`** + (`mistralrs-core/src/kv_cache/xs_rolling.rs:465`) alongside the pre-RoPE rows, + invalidated only when `advance` (`:1065`) actually appends. A decode step then + rotates 0 or 1 new rows and concatenates, instead of rotating `t_c`. +2. The `cos`/`sin` gathers are **separately memoizable** — they depend only on + `(t_c, ratio, device)`, exactly like the already-memoized + `compressed_row_positions` (`deepseek4.rs:3013-3037`) and `GRAPH_BLOCK_IDS` + (`dsv4_attention.rs`). Those two are the established in-tree pattern to copy. + +Item 2 is independent of item 1 and much cheaper to land; it is worth doing first. + +--- + +## Why this was not shipped with PR #206 + +Two hazards, neither settleable without a GPU. + +1. **Two row-set accessors, with different aliasing.** `Attention::forward` + (`deepseek4.rs:1522-1534`) takes the result of `state.advance(...)` on the + eager path, but `state.compressed_rows_fixed(rows)` (`xs_rolling.rs:912`) + under graph decode — where the buffer has been pinned to a fixed capacity by + `pin_comp_capacity` (`:867`) and the read is a constant-width narrow of a + buffer whose tail slots are unwritten. A cache keyed on "the rows did not + change" has to be right for **both**, across the pin, and in the case where + `advance` returns `None`. + +2. **A stale entry here is a wrong answer, not a slow one.** It silently corrupts + the distant-context branch: coherent output, degraded long-context recall, and + nothing downstream that catches it. That is the worst failure shape in this + repo's history — cf. the RUN-161 long-context collapse, and the + `ARC_V4_FP8_KV` default-on-without-a-GPU incident recorded on + `v4_fp8_kv_enabled` (`deepseek4.rs:3149`), which killed every request on the + first V4 forward that met real hardware. + +So it wants the treatment PR #206 got: an identity proof written down, an +engagement counter that **reports** rather than assumes, a poison control that +has been observed going red, and an on-GPU A/B before the default flips. Landing +it **default-off behind a flag with the A/B harness attached** is the right first +PR; flipping the default is the second. + +--- + +## Related + +* **PR #206** removes 467 launches/token from the mask block on this same path, + and names this as the single worst remaining item it did not fix. +* `CAPTURE_LANE`: op count prices at the GPU-side micro-kernel floor, so the ~205 + launches here are worth only ~0.2 ms. **The ~22 MB/token of redundant traffic + is the bigger half of this item**, and unlike launch count it is not a retired + lever. +* Filed here rather than as a GitHub issue because `aeonmindai/arc` has issues + disabled (`gh repo view --json hasIssuesEnabled` -> `false`); `memory/mission/` + is the tracker this project actually uses.