feat(ArcSched/ArcKV): lift the three scheduler caps below the B>=214 gate by heydryft · Pull Request #219 · aeonmindai/arc

heydryft · 2026-08-21T19:07:49Z

Parent system: ArcInfer / ArcSched + ArcKV. Targets release/openrouter-ready, never master.

⚠️ This PR changes serving behavior in two ways by default

--max-seqs default is now 256 (was 32). At B=32 the physics ceiling is ~3,740 tok/s; the 14,000 tok/s aggregate gate needs B ≥ 214, so the old default made the gate unreachable before any kernel mattered. Overridable as always. Memory is the real bound (~222 MiB/seq at ctx-4096 for V4 Flash-class ⇒ ~55 GiB at 256): PagedAttention deployments are protected by block-budget admission; DefaultScheduler admits by count only — on small cards, set --max-seqs explicitly. Wiring the dense path's default through a VRAM-aware admission check is named follow-up (none exists today; not built here per scope).
The prompt-starvation floor defaults ON at 4 (ARC_PREFILL_FLOOR_STEPS, unset → 4, 0 → cleanly OFF, restoring the previous selection key for key). The file's own fixture: behind a 47-seq decode cohort a fresh prompt wins 0 of 24 steps with no floor, 1 in 5 with floor=4. Without this, admitted prompts starve forever behind any large cohort (47 running, 48 waiting and stops).

The Cap-2 pair: mask fixed and shipped, default flip left for the box — deliberately

Turning ragged decode on activates CausalMasker::make_left_padded_causal_mask on every decode step of a ragged cohort. It was a scalar triple loop over B·t_q·(past+t_q) host f32s plus a full-mask H2D copy — 4.2 MB/token at B=256 ctx-4096 (CLAUDE.md pitfall #5 on the hot path). It is now built on the device: broadcast comparisons over a per-device cached arange (grown geometrically, narrowed per call); the only per-step H2D is the [B] lead vector. The host loop survives verbatim as make_left_padded_causal_mask_host, the test oracle, pinned element-for-element.

Note the mask fix is live by default already: on this branch, plain dense models (all-Normal caches) publish batch_can_be_ragged = true with no flag, so ragged dense cohorts — and this mask — are reachable in default serving. The fix is not contingent on the V4 flip.

Why the V4 default stays OFF: the premise "the pair is ARC_V4_XS_PER_SEQ + v4_ragged_rows_enabled_from" is stale on this branch. v4_ragged_rows_enabled_from does not exist; the per-row query-position gate is ragged_row_q0 (models/deepseek4.rs:3178), gated on ARC_MTP_PER_SEQ_KV (via per_sequence_kv_requested), a different flag in a file outside this PR's fence (deepseek4.rs, mtp_pipeline.rs). Flipping only ARC_V4_XS_PER_SEQ would let shorter V4 rows attend compressed blocks they have not reached — silently wrong answers, not a regression you can read off a throughput chart. The supported switch surface already exists (--v4-ragged-decode / v4_ragged_decode config key, ARC_V4_XS_PER_SEQ env fallback read by value — =0/unset is off, 1|true is on; latched once via request_xs_per_sequence). The box run flips --v4-ragged-decode and ARC_MTP_PER_SEQ_KV=1 together on a measured pass. Unifying the two under one switch means touching deepseek4.rs/mtp_pipeline.rs and belongs to whoever owns that lane.

Engagement instruments for the box run

Both pre-existing log lines still fire (untouched by this PR) — grep for:

ArcSched: decode bucketing (default_scheduler.rs, logs ragged_decode + bucket count on transition; a ragged cohort must collapse to 1 bucket)
ArcKV: ragged-decode capability resolved (kv_cache/mod.rs:1681, logs batch_can_be_ragged + xs_per_seq_enabled on transition)

Plus the floor's own once-only line: ARC prefill floor: forced a prompt bucket after N passes.

Capability registry (the switched-off guard)

Updated/added in mistralrs-core/tests/capability_reachability.rs, all Live: the device-built mask (make_left_padded_causal_mask), the scheduler's consumer read (ragged_decode_supported), the CLI/config latch (request_xs_per_sequence), and the floor entry renamed to carry its default-4 semantics.

Tests + mutation checks (values recorded both ways)

Test	Green (as shipped)	Mutated	Red value
`a_formed_cohort_admits_a_newcomer_when_ragged_decode_is_on`	ON arm runs 9/9 incl. the newcomer; OFF arm asserted 8/9 in the same run (fixture guard)	`len` pinned to `cache_bucket_len()` (revert the ragged arm)	`left: 8, right: 9`, `got [0..7]` — newcomer waitlisted again
`the_device_mask_matches_the_host_oracle_element_for_element`	8 cases, 0 mismatches, no NaNs	`lead` dropped from the device build (`0.0`)	`case 0 (tgt=1, past=6, live=[6,2]): 4 of 14 elements differ, first at flat index 7 (host -inf, device 0)`
`the_floor_defaults_on_and_zero_is_the_kill_switch`	unset→`Some(4)`, `"0"`→`None`, `"7"`→`Some(7)`, garbage→`Some(4)`	`floor_from` reverted to the old `.filter(\|n\| *n > 0)` shape	`left: None, right: Some(4)`

Existing starvation tests pass unchanged with the new default (they inject the floor directly, by design). Full lane: 701 core lib tests, 11/11 capability_reachability, 6 CLI tests, scoped clippy (arc-bench arc-engine arc-cuda-graph arc-cli mistralrs-quant) clean, cargo check clean.

Adjacent, observed, NOT fixed (not this PR's lane)

engine/mod.rs (~:465-478): pre_op is In only on membership change, but post_op is Out unconditionally — clone_out_cache runs every step. At B=256 a membership change transiently needs ~2× the batch cache (~119 GB next to 73 GB of weights). Reported to the owning lane.

Also noticed while fixturing: with ragged decode ON, a 1-token fresh prompt keys (0, false, 0) — the same bucket key as the pinned decode cohort — so a prompt could merge into a decode bucket. Not reachable in this PR's defaults (V4 pair off; dense prompts of >1 token key by len()-1); flagged for the ragged-enable session.

🤖 Generated with Claude Code

https://claude.ai/code/session_01WK8VgocBUrr5djjE1ZfCNf

…gate — serve default 256, device-side ragged mask, prefill floor default 4 Parent system: ArcInfer / ArcSched + ArcKV. CAP 1 — admission default. mistralrs-cli's max_seqs default goes 32 -> 256 (arg default, struct default, default_max_seqs(), and the tune config template). At B=32 the physics ceiling is ~3,740 tok/s, so the 14,000 tok/s aggregate gate (B >= 214) was unreachable on the default configuration. Memory implications documented on the flag; PagedAttention admission is block-budgeted, DefaultScheduler admission is count-only (VRAM-aware check for the dense path is named follow-up, not built). CAP 2 — the mask that would make ragged decode read as a regression. CausalMasker::make_left_padded_causal_mask was a scalar triple loop over B * t_q * (past + t_q) host f32s plus a full-mask H2D copy, per decode step while a cohort is ragged — 4.2 MB/token at B=256 ctx-4096 (CLAUDE.md pitfall #5 on the hot path). It is now built on the device from broadcast comparisons; the only per-step H2D is the [B] lead vector, and the index rows come from a per-device cached arange grown geometrically. The host loop is kept verbatim as make_left_padded_causal_mask_host, the test oracle, and the two are pinned element-for-element across mixed lengths, sliding windows, multi-row queries and the degenerate past=0. CAP 3 — the prompt-starvation floor defaults ON at 4. Behind a 47-sequence decode cohort a fresh prompt wins 0 of 24 steps with no floor and 1 in 5 with floor=4 (this file's own fixture). ARC_PREFILL_FLOOR_STEPS=0 is the kill-switch restoring the pre-floor selection key for key; the env mapping is factored into floor_from() and pinned by test on both sides. The V4 ragged-pair default is left OFF deliberately: on this branch the pair is --v4-ragged-decode / ARC_V4_XS_PER_SEQ (the xs cache) plus ARC_MTP_PER_SEQ_KV (the per-row query positions in dsv4_attention via ragged_row_q0). The second gate lives outside this change's file fence, and opening only the first would let shorter V4 rows attend compressed blocks they have not reached — silently. The box run flips the two together on a measured pass. New/updated capability registry entries: the device-built mask, the scheduler's ragged_decode_supported read, the request_xs_per_sequence CLI/config latch, and the floor entry renamed to carry its new default. New tests: a_formed_cohort_admits_a_newcomer_when_ragged_decode_is_on (cohorts could previously only shrink), the_device_mask_matches_the_host_oracle_element_for_element, the_device_mask_and_the_oracle_refuse_the_same_inputs, the_floor_defaults_on_and_zero_is_the_kill_switch. All mutation-checked; values recorded in the test docs and the PR body. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WK8VgocBUrr5djjE1ZfCNf

github-actions · 2026-08-21T19:09:13Z

Code Metrics Report

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 Language              Files        Lines         Code     Comments       Blanks
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 C Header                  5          305          210           52           43
 CSS                       2         1181         1036           34          111
 CUDA                     81        29475        20161         6275         3039
 Dockerfile                1           39           22            8            9
 JavaScript               16         3546         2676          482          388
 Jinja2                    7          694          656            5           33
 JSON                     75         4896         4893            0            3
 Makefile                  1            6            5            0            1
 Metal Shading Lan|       33        12224         9431         1142         1651
 PowerShell                1          300          227           30           43
 Python                  147        15371        12677          824         1870
 Shell                    42        10304         6858         2769          677
 Plain Text                4         3801            0         2479         1322
 TOML                     33         1498         1294           54          150
 YAML                      3           25           23            2            0
─────────────────────────────────────────────────────────────────────────────────
 HTML                      4         2687         2604           43           40
 |- CSS                    2          543          479           37           27
 |- JavaScript             1         1233         1215           12            6
 (Total)                             4463         4298           92           73
─────────────────────────────────────────────────────────────────────────────────
 Jupyter Notebooks         4          122           83           23           16
 |- Markdown               1           60           30           22            8
 |- Python                 1          122          113            1            8
 (Total)                              304          226           46           32
─────────────────────────────────────────────────────────────────────────────────
 Markdown                213        48932            0        38081        10851
 |- BASH                  72         1655         1203          331          121
 |- C                      4           19           19            0            0
 |- CUDA                   2           84           56           16           12
 |- JSON                  19          779          779            0            0
 |- PowerShell             1            1            1            0            0
 |- Python                23         1008          787          113          108
 |- Rust                  68         2063         1727           78          258
 |- TOML                   6          207          164            0           43
 |- YAML                   5           41           36            5            0
 (Total)                            54789         4772        38624        11393
─────────────────────────────────────────────────────────────────────────────────
 Rust                    681       340988       293252        18096        29640
 |- Markdown             504        32562          471        28063         4028
 (Total)                           373550       293723        46159        33668
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 Total                  1353       516771       363188        99077        54506
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

…ucket — prompt-ness joins the BucketKey; typos lane green The collision the PR body flagged as UNRESOLVED is real, and is now fixed and pinned by test. With ragged decode published, decode sequences key the sentinel length 0 — and a 1-token prompt's cache_bucket_len() is len() - 1 == 0, the same number. The probe test showed the prompt riding into the decode cohort through the merged bucket's `<= 1` fast path: scheduled in the same pass (prompt = [4] alongside completion = [0, 1, 2, 3]), bypassing select_running_bucket, the floor's bookkeeping and the one-bucket-per-step invariant. Fix: `is_prompt` becomes a fourth BucketKey component, so a bucket is homogeneous by construction. The coalescing filter requires matching prompt-ness (a decode bucket must never idle to catch up to a prompt bucket), and the floor's is_prompt_bucket now reads the key instead of whichever sequence sits first in the bucket — removing an order-dependence at the same time. Test: a_one_token_prompt_is_not_folded_into_the_pinned_decode_bucket. Green: completion = [0, 1, 2, 3], prompt = [], waiting = 1. Mutated (fourth component pinned false): red with prompt = [4]. All 14 scheduler tests and the full 702-test core lib suite pass. Typos lane: "unparseable" -> "unparsable" (floor doc), and word-level allowlist entries for arange/ARANGE in .typos.toml — candle's Tensor::arange naming appearing as a segment inside cached_arange_f32 / ARANGE_CACHE, which the existing exact-identifier entries do not cover. `typos --config .typos.toml .` is clean repo-wide. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WK8VgocBUrr5djjE1ZfCNf

heydryft · 2026-08-21T19:22:31Z

Follow-up pushed in bcb686f — the UNRESOLVED item at the bottom of the PR body is now closed:

The 1-token-prompt / pinned-decode-bucket collision is REAL and is fixed. The probe test showed the prompt riding into the decode cohort through the merged bucket's <= 1 fast path (prompt = [4] scheduled alongside completion = [0, 1, 2, 3], bypassing select_running_bucket, the floor's bookkeeping and the one-bucket-per-step invariant). Fix: is_prompt is now a fourth BucketKey component — buckets are homogeneous by construction, the coalescing filter requires matching prompt-ness, and the floor's is_prompt_bucket reads the key instead of whichever sequence sits first in the bucket (removing an order-dependence at the same time).

Test: a_one_token_prompt_is_not_folded_into_the_pinned_decode_bucket. Green: completion = [0,1,2,3], prompt = [], waiting = 1. Mutation (fourth component pinned false): red with prompt = [4]. All 14 scheduler tests + full 702-test core lib suite pass.

Also in the same commit, the Typos lane fix: "unparseable" → "unparsable" in the floor doc, and word-level arange/ARANGE allowlist entries in .typos.toml (candle's Tensor::arange naming as a segment inside cached_arange_f32/ARANGE_CACHE, which the existing exact-identifier entries don't cover). typos --config .typos.toml . is clean repo-wide.

…ched; #219 owns the allowlist) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WK8VgocBUrr5djjE1ZfCNf

… the V4 union mask across layers (#221) * fix(ArcAttention): the Metal sinks arm still dropped the caller's mask CORRECTNESS, the other backend. #199 gated the CUDA fused arm on mask.is_none(), but the Metal arm kept engaging on head_dim alone and calling flash_attn_sinks_metal — which takes no mask parameter at all, so unlike the CUDA kernel it cannot even refuse one at the interface. GPT-OSS reaches this arm on every macOS build: head_dim 64 is in Metal's fused set and the model passes a mask it asserts is Some (gpt_oss.rs). MEASURED on a real Metal device (M-series, head_dim 128, F32): max|fused - reference(causal only)| = 0.000001 max|fused - reference(masked)| = 0.860029 max|ref(causal) - ref(masked)| = 0.860029 <- the mask's whole effect The maskless kernel tracks the causal-only reference to 1e-6 and misses the masked one by the mask's entire effect: dropped, not approximated. The fix is the same one line as CUDA's: an explicit mask pins the call to the unfused mask-honoring path, with a once-per-process warn naming the deliberate perf loss. New tests: * metal_sinks_mask_tests (feature = "metal", real device): the baseline above, plus the gate — masked dispatch vs masked reference, diff 0.0. Mutation-checked on hardware: with the guard reverted the gate test fails at max abs diff 0.8600293; restored, 0.00000000. * padding_mask_is_honored_and_dropping_it_changes_the_output (CPU, always runs): a padding-column mask — the class no fused kernel can re-derive — survives the dispatch end to end. Mutation-checked: dropping the mask at the fall-through arm fails it at max abs diff 0.9353 vs 0.0 required. Rider (perf, contained): sinks_attn_cpu_varlen rebuilt its per-sequence mask with Tensor::from_vec — a host-to-device upload per sequence per layer per step on the GPU fall-through (CLAUDE.md pitfall #5). It is a pure function of (q_len, kv_len, window, dtype, device), now memoized in a single slot; uniform batches build it once per shape. varlen_mask_memo_hits_and_never_ serves_stale pins both memo failure modes (mutation-checked: with the key comparison removed it fails on the (5, 8, 3) entry). Also corrects the file docs that called the unfused fall-through "CPU": it is device-agnostic candle math and is exactly what V4's head_dim-512 layers run on CUDA. The to_vec1 D2H syncs on cu_seqlens remain — hoisting the host copy into FlashParams is shared plumbing, reported not fixed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WK8VgocBUrr5djjE1ZfCNf * perf(ArcInfer/ArcAttention): memoize the V4 union mask across layers The union mask is a pure function of nine integers — (q0, t_q, raw_base, t_k, t_c, ratio, window, dtype, device) — yet dsv4_attention rebuilt it once per LAYER per step. On the prefill path that is three Tensor::arange host-to-device uploads plus their casts, compares, the U8 cat and the 0/-inf conversion, every layer (CLAUDE.md pitfall #5: each arange with a GPU device is a CPU-to-GPU sync). a7b0890/64fe1d678 already killed the DECODE-path aranges; this removes the per-layer rebuild wholesale on both paths. Mechanics: the validity build is now lazy (build_valid closure, character-for-character the old code on a miss) and the additive [1, 1, t_q, n_keys] mask is memoized thread-local on the integer key — 4 FIFO slots, since one step touches at most three keys (the Standard / CSA / HCA layer classes). Per step, each distinct mask is built once instead of ~once per layer. Excluded on purpose: * graph_positions (fixed-capacity graph decode): the key would be constant across steps, and the ops must stay inside the capture region rather than be hoisted into a warmup-owned tensor whose later eviction would leave a recorded graph reading freed memory. That build is already arange-free. * ragged cohorts (row_q0): the per-row leads are content but not key. Tests (CPU, validating code not perf — D14): * memo_hits_without_building_and_misses_on_a_changed_key: a hit returns the cached handle WITHOUT running the builder (handle identity is the only observable — a memo that always misses returns correct values while doing nothing); an advanced q0 must miss. * memoized_masks_are_bit_identical_to_fresh_builds: decode at ctx 35 then 36 (ratio 4, window 8) — two masks with the SAME shape and different content (threshold crosses 8 -> 9). Sequential same-thread runs must be bit-identical to fresh-thread (empty-memo) references. Mutation-checked: with the memo's key comparison replaced by "serve the newest entry", both tests fail — the identity test at TensorId(1) != TensorId(6), the end-to-end test with ctx=36's output bits diverging from the fresh reference (first lane 1051680768 vs 1051066368). No behavioral change: a miss runs the exact pre-change build, and the existing 38 dsv4_attention tests (bit-identity, ragged, graph-geometry, absorbed-decode) all pass unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WK8VgocBUrr5djjE1ZfCNf * docs(ArcAttention): reword 'aranges' for the Typos lane (config untouched; #219 owns the allowlist) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WK8VgocBUrr5djjE1ZfCNf --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

heydryft mentioned this pull request Aug 21, 2026

fix(ArcInfer/ArcGraph): cross-request token leak — parked burst tokens no longer outlive their sequence #220

Merged

heydryft merged commit 738f566 into release/openrouter-ready Aug 21, 2026
18 checks passed

Navigation Menu

Repository navigation

feat(ArcSched/ArcKV): lift the three scheduler caps below the B>=214 gate - #219

feat(ArcSched/ArcKV): lift the three scheduler caps below the B>=214 gate#219
heydryft merged 2 commits into
release/openrouter-readyfrom
arcsched/ragged-and-floor

heydryft commented Aug 21, 2026

Uh oh!

github-actions Bot commented Aug 21, 2026

Uh oh!

heydryft commented Aug 21, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

heydryft commented Aug 21, 2026

⚠️ This PR changes serving behavior in two ways by default

The Cap-2 pair: mask fixed and shipped, default flip left for the box — deliberately

Engagement instruments for the box run

Capability registry (the switched-off guard)

Tests + mutation checks (values recorded both ways)

Adjacent, observed, NOT fixed (not this PR's lane)

Uh oh!

github-actions Bot commented Aug 21, 2026

Uh oh!

heydryft commented Aug 21, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant