Arc → OpenRouter-ready: the single integration PR (everything else merges into this branch)

.github/workflows/cuda_compile_check.yaml

@@ -305,6 +305,16 @@ jobs:
       - name: QTIP beam spill gate (LOCAL:0, all geometries)
         run: bash arc-tools/qtip_beam_res_usage_check.sh sm_${{ matrix.compute_cap }}
 
+      # Same free-instrument class as the QTIP gate above, for the blockwise-
+      # FP8 dense-GEMM lane: asserts all 4 fp8_matmul_tiled instantiations
+      # exist (2 dtypes x 2 POW2_SCALE scale-index arms), prints the LDS width
+      # census that decides whether the bank-conflict cost model describes the
+      # compiled kernel, and HARD-FAILS if the never-executed WMMA GEMM
+      # contains no tensor-core (HMMA/HGMMA) instructions. nvcc + cuobjdump
+      # only; no GPU.
+      - name: FP8 GEMM SASS probe (LDS census + WMMA HMMA presence)
+        run: bash arc-tools/fp8_gemm_sass_check.sh sm_${{ matrix.compute_cap }}
+
       - name: Compile QTIP CUDA kernels (mistralrs-quant)
         run: cargo build -p mistralrs-quant --lib --features cuda
 

.typos.toml

@@ -60,6 +60,13 @@ extend-ignore-re = [
 #   optin         : "opt-in" without the hyphen — CUDA's
 #                   cudaDevAttrMaxSharedMemoryPerBlockOptin, and filenames /
 #                   branches that cannot carry a hyphen there
+#   arange        : candle's `Tensor::arange` (numpy naming), which also
+#                   appears as a word segment inside larger identifiers
+#                   (`cached_arange_f32`, `ARANGE_CACHE`) — the
+#                   extend-identifiers entries below only cover the exact
+#                   identifier `arange`, not segments
+"arange" = "arange"
+"ARANGE" = "ARANGE"
 "nd" = "nd"
 "clen" = "clen"
 "tpos" = "tpos"

Cargo.toml

@@ -44,16 +44,31 @@ rust-version = "1.88"
 # RUN-161: forked candle for the opt-in CUDA caching allocator (graph-capture-safe).
 # Default-off; identical to upstream c3bb5bf when ARC_CANDLE_ALLOC_CACHE unset.
 # Fork of huggingface/candle 0.9.2 (rev c3bb5bf), mirrored to the org for CI.
-# PINNED TO AN EXPLICIT REV, not a branch. This change depends on
-# `drain_alloc_cache_and_free` (fork commit 88d86a20), and a moving branch
-# pointer has broken CI on this repo before — a path/branch dependency that
-# advances under a merged PR turns a green build into a red one with no commit
-# here to blame.
-candle-core = { git = "https://github.com/aeonmindai/candle.git", rev = "89ab14ef1216331a782539f533b99f6816708604" }
-candle-nn = { git = "https://github.com/aeonmindai/candle.git", rev = "89ab14ef1216331a782539f533b99f6816708604" }
-candle-flash-attn-v3 = { git = "https://github.com/aeonmindai/candle.git", rev = "89ab14ef1216331a782539f533b99f6816708604" }
-candle-flash-attn = { git = "https://github.com/aeonmindai/candle.git", rev = "89ab14ef1216331a782539f533b99f6816708604" }
-candle-metal-kernels = { git = "https://github.com/aeonmindai/candle.git", rev = "89ab14ef1216331a782539f533b99f6816708604" }
+# PINNED TO AN EXPLICIT REV, not a branch. This tree needs BOTH lines of the
+# fork's allocator work, and a moving branch pointer has broken CI on this repo
+# before — a path/branch dependency that advances under a merged PR turns a
+# green build into a red one with no commit here to blame.
+#
+# The rev below is `arcgraph/bounded-alloc-plus-prewarm` (9586979db), which is
+# the MERGE of the two, and a strict superset of each:
+#
+#   * 89ab14ef (`perf/alloc-cache-bounded`) — `drain_alloc_cache_and_free`
+#     (88d86a20) plus the bounded caching allocator #213 depends on, its cap
+#     contract test, and "eviction must not run inside the capture window".
+#   * 1fa534db (`arcgraph/prewarm-profile`) — the capture-miss ledger and
+#     profile-driven pre-warm this file's pre-warm caller needs
+#     (`prewarm_alloc_cache`, `reset_capture_misses`, `capture_misses`).
+#
+# Those two revs had DIVERGED: 1fa534db is missing three commits 89ab14ef has.
+# Pinning straight to 1fa534db, as this change originally did, would therefore
+# have silently reverted the bounded allocator and the capture-window eviction
+# fix. Verified by `gh api repos/aeonmindai/candle/compare/...`: 9586979db is
+# ahead of 89ab14ef by 6 and of 1fa534db by 8, behind neither.
+candle-core = { git = "https://github.com/aeonmindai/candle.git", rev = "9586979db9031df6377f7dab1233c4548d976b76" }
+candle-nn = { git = "https://github.com/aeonmindai/candle.git", rev = "9586979db9031df6377f7dab1233c4548d976b76" }
+candle-flash-attn-v3 = { git = "https://github.com/aeonmindai/candle.git", rev = "9586979db9031df6377f7dab1233c4548d976b76" }
+candle-flash-attn = { git = "https://github.com/aeonmindai/candle.git", rev = "9586979db9031df6377f7dab1233c4548d976b76" }
+candle-metal-kernels = { git = "https://github.com/aeonmindai/candle.git", rev = "9586979db9031df6377f7dab1233c4548d976b76" }
 
 axum = "0.8.8"
 anyhow = "1.0.100"

README.md

@@ -28,7 +28,7 @@ Forked from [mistral.rs](https://github.com/EricLBuehler/mistral.rs). Apache 2.0
 | Feature | What it does |
 |---|---|
 | **QTIP 2-bit weights** | Cornell ICLR'25 — trellis quantization with Hadamard incoherence rotation. 8× weight compression vs FP16. Two rungs: `qtip2` (LUT) and `qtip2b` (computed codebook), the latter being what Arc serves. Works on any model. |
-| **TurboQuant KV** *(paged default at head_dim 128)* | Zandieh et al. (Google Research, [arXiv:2504.19874](https://arxiv.org/abs/2504.19874), ICLR'26) — Arc Rust implementation of WHT + Lloyd-Max KV coding. **On by default on the paged path**: a standard-layout head_dim-128 model on CUDA gets K4/V3 KV with no flag, and loses prefix caching with it (`--pa-cache-type auto` opts out). Every other geometry falls back to `auto` with a warning; there is no kernel at head_dim 512, so DeepSeek V4 does not take it. Served end-to-end on a B200 at **55 tok/s with correct output**; **quality has never been evaluated**. Eager (non-paged) KV is a separate path, opt-in via `ARC_TURBOQUANT_KV=1`. |
+| **TurboQuant KV** *(paged default at head_dim 128)* | Zandieh et al. (Google Research, [arXiv:2504.19874](https://arxiv.org/abs/2504.19874), ICLR'26) — Arc Rust implementation of WHT + Lloyd-Max KV coding. **On by default on the paged path**: a standard-layout head_dim-128 model on CUDA gets K4/V3 KV with no flag, and loses prefix caching with it (`--pa-cache-type auto` opts out). The kernels are now instantiated at head_dim 64/128/256/512, but only 128 has ever executed on hardware and only 128 is reachable without asking — every other geometry falls back to `auto` with a warning, so DeepSeek V4's head_dim of 512 is opt-in via an explicit `--pa-cache-type turboquant`. Served end-to-end on a B200 at **55 tok/s with correct output** at head_dim 128; **quality has never been evaluated**, at any width. Eager (non-paged) KV is a separate path, opt-in via `ARC_TURBOQUANT_KV=1`. |
 | **TD-MoE Tucker + whitening** | Whitened Tucker decomposition of the MoE expert pool. The **"lossless 20%"** figure is the **paper's** (NeurIPS'25), not ours — *published, not reproduced by us*. Works on any MoE. |
 | **ArcAttention/Indexer** — sparse attention | DeepSeek V4's Lightning Indexer with FlashMLASparse CUDA kernels (MIT, ported from sgl-project) for top-k attention. Dense O(n²) → sparse O(n·k). |
 | **ArcGraph** — GPU-autonomous decode | Full decode loop (forward + sample + EOS check) on GPU. Zero CPU sync per token. Works on any model. |
@@ -79,7 +79,7 @@ Each layer compresses a different axis, and in principle the wins multiply.
 
 **Read the block above as arithmetic, because that is all it is.** The weight line is measured end-to-end: 284B → **~74 GB** at 2.09 bits/param (V4 Flash trellis bake, serving on one H200 — see [Results](#results)). **The KV lines are not.** They are bytes-per-token ratios computed from the packed format; no run has ever produced a measured KV compression figure, and the "4.27×" once published here is retracted.
 
-What *has* run: the TurboQuant K4/V3 paged path served Qwen3-32B on a B200 end-to-end with correct output (see [Results](#results)) — so "no TurboQuant forward pass has ever been benchmarked", which this section previously asserted, was false. What that run does **not** establish is any of the ratios above, or any quality number. TurboQuant is the paged default at head_dim 128; it has no kernel at V4's head_dim of 512. The xKV pool is not implemented.
+What *has* run: the TurboQuant K4/V3 paged path served Qwen3-32B on a B200 end-to-end with correct output (see [Results](#results)) — so "no TurboQuant forward pass has ever been benchmarked", which this section previously asserted, was false. What that run does **not** establish is any of the ratios above, or any quality number, and it was at head_dim 128 only. TurboQuant is the paged default at head_dim 128; the paged kernels now also cover V4's head_dim of 512, but nothing has ever run at that width, so it stays opt-in. The xKV pool is not implemented.
 
 For long context (1M+ tokens), the bottleneck shifts from weights to attention compute + KV bandwidth. Arc handles that via FlashMLASparse (CUDA kernel, MIT-licensed, ported from sgl-project), turning dense attention's O(n²) into sparse top-k O(n·k) on models with native sparse-attention training (DeepSeek V3.2+ family) and via top-k attention + sink preservation on the rest.
 
@@ -144,9 +144,9 @@ Python bindings: `pip install mistralrs-cuda`.
 
 > **TurboQuant *is* the paged default** — correcting an earlier note here that read "TurboQuant is not the default anywhere", which was false when written. `defaults::PAGED_CACHE_TYPE` is `PagedCacheType::TurboQuant` and `--pa-cache-type` carries no clap default, so leaving it unset on CUDA (where PagedAttention is itself on by default) gives a standard-layout head_dim-128 model TurboQuant K4/V3 KV with nothing asked for — and **silently drops prefix caching**, which no TurboQuant preset supports. Pass `--pa-cache-type auto`, or `PagedCacheType::Auto` as above, to opt out.
 >
-> Off that envelope — MLA layouts, any other head_dim — the default falls back to `Auto` with a warning, while an explicit `--pa-cache-type turboquant` is a hard error instead. The eager (non-paged) KV path is separate and genuinely opt-in via `ARC_TURBOQUANT_KV=1`.
+> Off that envelope the default falls back to `Auto` with a warning: MLA layouts, uninstantiated head dims, and the instantiated-but-unmeasured widths 64/256/512. Those three are reachable only by asking for them — an explicit `--pa-cache-type turboquant` is accepted at any instantiated width, and is a hard error off that set. The eager (non-paged) KV path is separate and genuinely opt-in via `ARC_TURBOQUANT_KV=1`.
 >
-> **Provenance:** this path has served a real model on real hardware — Qwen3-32B, B200, 55 tok/s, correct output — but at b=1, one width, one preset, with **no quality evaluation and no A/B against an unquantized cache**. See [Results](#results) before relying on it.
+> **Provenance:** this path has served a real model on real hardware — Qwen3-32B, B200, 55 tok/s, correct output — but at b=1, **head_dim 128 only**, one preset, with **no quality evaluation and no A/B against an unquantized cache**. Widths 64/256/512 compile; none has ever executed. See [Results](#results) before relying on it.
 
 ## Supported Models