Review sidecar probes and replays spend the per-key free-tier rate budget the review itself needs

[seonghobae]

What the evidence now shows

With the catalog fixed (#1939, every run selects 4+4+4 across the three free accounts) and the per-attempt trace landing (#1943/#1944/#1945), the failures that remain on the free pool are not routing failures. Two post-#1939 runs, one hour apart, same shape:

• noema-review run 33993637015 (#1687, 21:38Z): preflight ready 1/12 — 429 ×6 (all four OpenRouter routes; deepseek-v4-flash and v4-pro on the primary NVIDIA key while the sibling key's flash was ready), 404 ×4 (gemma-3-12b/4b on both NVIDIA keys), 500 ×1. The one ready route was held 542 s and the caller got 502.
• strix run 33992904674 (#1930, 23:48–00:12Z, artifact 9979385164): ready 1/12 — 429 ×6 again, one TimeoutError, gemma-3 ×4 404; the sole ready route (nvidia_nim_sub deepseek-v4-pro) answered 429 rate_limit_exceeded on the first real request and on all five Strix replays; llm_usage.requests = 0.

Corrected post-#1939 tally of noema-review runs that reached the verdict step: 1 success / 3 failures (three run-level "successes" in that window were closure-event runs with the job skipped). Reading: at those hours there was no free capacity to interleave, and each such job still held a runner for ~25 minutes inside a 234-deep queue.

Mechanism

Every sidecar boot probes all twelve catalog routes with a real completion (REVIEW_PREFLIGHT_BASE_TOKENS = 16, contextual_orchestrator_review_launcher.py:404), i.e. four requests per account per boot, then the review makes its own requests, and Strix replays each scan step up to five times. All of that draws on the same per-key budgets (NVIDIA NIM ≈ 40 requests/min per key, community-reported; OpenRouter free tier is per-key and per-day). With N sidecars booting in the same minute across the organization — required workflows run in every repository — the probes alone can consume the budget the review then needs, and a 429 at probe time is frequently caused by the other sidecars' probes. #1947 stops discarding such routes at preflight, which restores failover depth, but cannot create capacity: a route that is rate-limited for the hour fails fast for everyone.

What this is not

Not a routing defect (#1939 measured working), not a failover defect (#1938 thread: 429 is retryable=True and fails over; provider_errors.py:82), not the launcher strip (#1476 thread). The four permanent gemma-3 404s are a separate discovery-side waste of a third of the catalog.

Levers — all policy values, hence the owner's

1. Probe budget per boot. Stop probing once K routes are ready (or probe one route per account first), instead of twelve unconditional completions; that divides probe spend by up to 3–4 at the cost of a smaller preflight report. This is an amendment to ADR-0005's "bounds request count" contract.
2. Replay budget. Strix's five replays against an already-429 route multiply the burn; a replay policy that stops on a rate-limit answer is a strix-agent/CO decision.
3. Concurrency of sidecar boots. The organization ceiling (60 jobs) is not a per-key limiter; an admission rule per provider key (or a shared rate-limit ledger across sidecars — the same cross-process state contextual-orchestrator#1045 needs for its breaker) would be.
4. Paid headroom. OpenRouter credits lift the free tier's daily cap; NVIDIA's per-key RPM does not change with credits.

What to measure before choosing

With #1944's artifact every failed noema-review now ships contextual-orchestrator-preflight.json and the sanitized stderr. Two numbers decide between (1)–(3): the fraction of probe rejections that are 429 (vs 404/500/timeout), and the number of sidecars whose probes fell in the same minute (created_at of in-progress noema/strix/opencode jobs org-wide). If most 429s cluster at boot storms, (1) and (3) are the levers; if the sole ready route also 429s on the real request in quiet minutes, the key is simply saturated and (4) is the only one.

[seonghobae]

Datum for lever (3), measured while filing (00:20Z): 25 noema-review/strix workflow runs in_progress in this repository alone — each one a sidecar that probed twelve routes at boot and is now making review requests against the same three free accounts. That is the boot-storm denominator; the organization-wide figure is larger since the required workflows run in every repository. The per-key budgets (≈40 requests/min on NVIDIA NIM, per-key/day on OpenRouter free) are shared by all of them.

[seonghobae]

Correction to the body (host 1): "held for 542 s (= one route's full 2 × 3 × 90 s budget)" is wrong — that arithmetic belongs to the orchestrated _invoke/chat() path. The review request goes through the passthrough path (proxy_completion → proxy_send_once, orchestrator.py:4137, :2408-2412): one attempt per candidate, stream=False. The artifacts confirm attempt=1/1 on every provider attempt. So 542 s was one candidate whose upstream held a non-streamed request open for nine minutes before answering 502. Nothing else in this issue changes — the probe/replay budget mechanism and the four levers stand — but it adds a fifth datum for the owner: the only ceiling on a held request is a response-start or total deadline at the gateway (§8), and the artifacts' per-candidate timestamps now show how often that class occurs.

[seonghobae]

Measurement that bears on levers (1) and the "permanent 404s" aside — from the sidecar artifacts, pre- and post-#1939, same repository, same discovery data.

Post-#1939 the served NVIDIA set lost every non-deepseek working route. Discovery (contextual-orchestrator-discovery.json in #1938's Strix artifact 9979583987) lists 21 models per NVIDIA key — deepseek ×2, gemma-3 ×2, gemma-4, llama-3.2 11b/90b, muse-glimmer-30b, minimax-m3, mistral ×2, kimi ×2, nemotron ×4, cosmos, riva, gpt-oss-20b, laguna. The 4+4+4 fill takes each key's first four in catalog order, which is alphabetical by model within the tier: deepseek-ai/deepseek-v4-flash, deepseek-ai/deepseek-v4-pro, google/gemma-3-12b-it, google/gemma-3-4b-it. Two of those four are the permanent 404s, so each NVIDIA key contributes two working routes, both deepseek-v4 — the most-contended models on the key.

Pre-#1939 the same order with the 8+4 fill reached google/gemma-4-31b, meta/llama-3.2-11b-vision-instruct, meta/llama-3.2-90b-vision-instruct, meta/muse-glimmer-30b on the primary key, and three of those were ready:

Strix run (artifact)	created	fill	ready / 12	ready routes
33979406293 (#790)	16:56Z	8+4+0	6	deepseek flash+pro ×2 keys, llama-3.2-11b, muse-glimmer-30b
33979153466 (#789)	16:51Z	8+4+0	6	deepseek flash+pro, llama-3.2-11b, llama-3.2-90b, muse-glimmer-30b, sub deepseek-pro
33978435797 (#1382)	16:37Z	8+4+0	5	deepseek-pro ×2 keys, llama-3.2-11b, llama-3.2-90b, muse-glimmer-30b
33992904674 (#1930)	23:48Z	4+4+4	1	sub deepseek-pro
33992736699 (#1938)	23:47Z	4+4+4	3	deepseek-pro, sub deepseek-flash, sub deepseek-pro

So the interleave traded three ready NVIDIA routes (llama/muse, which nobody else was hammering) for four OpenRouter routes that 429 at preflight — the account diversity went up, the served diversity went down, and the ready count fell from 5–6 to 1–3 before the evening load is counted.

Honest post-#1939 verdict tally (every noema-review run in .github created after 17:30Z with a PR attached, by the Prepare Noema model verdict step's own conclusion): 0 successes, 22 failures (verdict-step durations 199–4273 s, median ≈1500 s). The two run-level "successes" (#1902, runs 33982955696 / 33994883456) are draft skips — PR is draft; Noema verdict preparation skipped., verdict step 0–1 s — which corrects the "1 success" in my comment on contextual-orchestrator#1045 and this issue's opening tally. Pre-#1939, 10:00–16:25Z, the same count was 7 / 14. That is not proof #1939 caused the drop (load rose in parallel), but the served-set change above is a mechanism that does not need load to explain part of it.

Lever that addresses both this and (1) at once, for the owner's call: fill each account's slots lazily from its full ranked candidate list until K routes are ready, instead of probing a fixed first-four. A permanent 404 then costs one cheap probe and is replaced by the next candidate (llama, muse, nemotron …) rather than occupying a slot for the whole run; probing stops at K ready, which is lever (1)'s bound; and no model name needs hard-coding. Failing that, the smaller change is to rank a tier's candidates by model family round-robin (deepseek, google, meta, minimax, …) before the per-account cap, so no key's four are two dead gemma-3 plus the two most-contended deepseek routes. Both stay inside contextual_orchestrator_review_policy.py / the launcher preflight, i.e. this lane; I am not opening a competing change.

---

Generated by Claude Code

[seonghobae]

Correction to my correction (host 1): the first noema-sidecar-evidence artifact (#1661 run 33995553859, 3873 s) shows the review path is _invoke — per silent agent two rounds × three 90 s timeouts (measured 270.3–271.1 s per round, 541 s per agent), not one passthrough attempt; the attempt=1/1 lines were preflight probes. The 542 s in the body is therefore one agent's two rounds, as first written. The probe/replay-budget mechanism and the levers stand; the response-start deadline still matters, but the dominant cost here was 13 silent rounds on two NVIDIA keys repeated across three internal walks, plus two preflight-ready routes never attempted — see the table on #1939.

[seonghobae]

Lever taken (host 1): #1949, stacked on #1947. The catalog becomes a candidate list (24, per-account cap 8, #1939's tier-then-round-robin order) and _preflight_review_agents probes it in order, stopping at 8 ready routes or 16 probes, whichever comes first. A dead candidate then costs one probe instead of a served slot, and each account's list continues past its gemma-3 404s to the llama-3.2 / muse-glimmer entries that were ready in the pre-#1939 artifacts. No model name is hard-coded; the 404s are found by their probe. Report gains candidate_count, target_ready, probe_budget; probed_count now counts probes actually sent. ADR-0029 records it and amends ADR-0003 §2's "twelve-route" sentence.

Two honest numbers for this issue's ledger. Worst case (nothing ready) is now 16 probes per stage where it was 12 — a third more against keys that are already exhausted; that is the price of finding routes past the dead ones, and the shared rate ledger above it stays this issue's lever. Healthy case stops at about 8–12 probes instead of always 12, so the common hour spends less. Your family round-robin fallback is recorded in the ADR as the next layer if the post-merge census shows same-family contention as the residual.

Discriminator per boot after merge: probed_count vs candidate_count and the served set's ready_count, against 6/6/5 of 12 before #1939 and 1–3 after.

[seonghobae]

Slot measurement 2026-09-06T04:30Z (lane jan), for the levers list.

Who held the runners. Of 10 in_progress runs in .github, 7 were strix jobs on the pre-bump pin holding runners since 00:10–03:23Z by job started_at — the five oldest 3 h 45 m–4 h 25 m inside "Run Strix (quick)" (#1231, #1271, #1555 and two push/main scans; #1678 and #1872 since 03:16Z / 03:23Z) — plus 2 noema-review verdict steps (#1872 since 03:22Z, #1280 since 03:47Z) and one OpenCode Review Dispatch (#939). There is no job timeout on either review job by design (strix.yml:345-350, noema-review.yml:261-276), so a job that enters signature 3's bulk-500 loop holds its slot until the 6-hour Actions maximum. Nothing created after the 03:01Z pin bump had started: 11 noema-review runs queued since 03:07Z, so the co#1081 KPI (does the verdict rate move off 0/23?) still has zero observations.

What sat in front of them. 170 queued runs, 85 of them CodeQL Scan Dispatch handlers from sibling repositories rejected at #1929's actor gate (codeql-scan-dispatch.yml:142: actor=opencode-agent[bot], allowlist github-actions[bot]). Each waits ≈65 minutes for a slot, fails in 3–6 s, and the source repository re-dispatches: 231 runs since 01:30Z, two or three per SHA (argos, html4tree and newsdom-api 12 each). Their runner cost is seconds; their cost is queue position.

Pool at 04:21Z (#1913 run 34006939646, artifact 9982569956): 0 of 12 ready. New since the last artifacts: both keys' deepseek-v4-pro — the route that had been carrying reviews — TimeoutError at the 90 s probe bound; flash 429 ×2, gemma-3 404 ×4, OpenRouter 429 ×4, bytez discovery http_status_500. With 0 ready the launcher exits before /healthz and the job fails in 5 minutes at provisioning with the verdict step skipped — the first fail-fast entry in the tally, and the cheapest form of the signature. #1949's walk would not change a 0-ready hour; it bounds the probes spent finding out.

Relief applied under signature 7's three-part test (catalog in #1913; first applied by a peer at 16:36Z yesterday): the two push/main Strix scans of superseded heads d9eb9f79b (created 21:59Z, slot since 00:10Z) and 972b74be2 (22:09Z, slot since 00:15Z) — both ancestors of main@efb892692, whose own push scan is queued — were retired at 04:35:57Z / 04:36:00Z (runs 33994595180, 33995072470, both read completed/cancelled); two runners returned. The five PR scans were left alone: a PR scan is consumed by its check. Expect the freed slots to produce fail-fast provisioning failures while the pool reads 0 — a 5-minute failure with an artifact is still a better use of the slot than a 4-hour hold without one.

---

Generated by Claude Code

[seonghobae]

First Noema artifact under the new pin (414f2297), and it is the no-capacity class (host 1). .github#1661 run 34008191123, job 101424607975 (noema-review, 04:46–04:52Z, head 6047f36f), failed at Provision contextual-orchestrator review sidecar: preflight ready 0 of 12, review sidecar preflight failed, sidecar exited before healthz. From noema-sidecar-evidence (uploaded on a provisioning-step failure too — #1944 covers this path):

routes	answer at preflight
nvidia_nim + nvidia_nim_sub deepseek-v4-flash	429, 429
nvidia_nim + nvidia_nim_sub deepseek-v4-pro	TimeoutError — each probe held for 90.06 s / 90.10 s (04:48:53 → 04:50:23 → 04:51:53)
openrouter ×4 (cohere north-mini, dots-3-note, gemma-4-26b, gemma-4-31b)	429 ×4
nvidia_nim + nvidia_nim_sub gemma-3-12b, gemma-3-4b	404 ×4

So at 04:48Z both NIM keys were rate-limited on flash and silent on pro, and OpenRouter was rate-limited on every free route. Three readings for this issue's ledger:

1. A 16-token probe can cost the full 90 s recv timeout. The two silent NIM probes spent 180 s of a 5-minute job before the walk reached the 404s; preflight wall time is bounded by probes × 90 s, not by request count alone — the same arithmetic #1949 uses for its probe cap.
2. Neither readiness lever applies to a zero-ready hour. #1947's deferral needs at least one ready route; #1949's account skip would have trimmed OpenRouter's last two probes and the 404s cost nothing, but no candidate in this catalog was serving. This is capacity, i.e. the lever above both: probe budget per boot across concurrent sidecars, or paid headroom.
3. No serving-phase data, so the post-fix(opencode): system llvm for cargo-llvm-cov (v2 concurrency) #1081 per-agent cost (~180 s expected on the no-tools path) is still unmeasured; the next Noema run that gets past preflight is the sample.

For the census: classify this run as preflight-fail (capacity), distinct from the 502 walk-end class (#1661's previous run) and the Strix 500 class (#1812, CO#1082).

[seonghobae]

Owner-decision item (policy, not engineering): should the review sidecar's preflight probe have its own deadline, distinct from inference?

What the code does today. The launcher builds a preflight-only ModelClient(max_output_tokens=…, max_retries=0, temperature=…) (contextual_orchestrator_review_launcher.py:981) and never passes timeout, so it inherits the vendored transport default ModelClient.__init__(timeout: int = 90); proxy_send_once (pin 414f2297, orchestrator.py:2451) takes no timeout argument, so the constructor is the only knob. The serving client (:1005) is a separate instance. contextual-orchestrator's own ModelClient.probe() already uses a separate DEFAULT_PROVIDER_PROBE_TIMEOUT = 5.0.

What it costs. #1661 run 34008191123 (first Noema on pin 414f2297): both NVIDIA deepseek-v4-pro probes of a 16-token Reply with just 'OK' request ran to the full recv timeout (TimeoutError at 90.06 s / 90.10 s) — 180 s of preflight for two routes that were never going to serve. With #1949's MAX_PROBES=16, the dead-hour worst case is 16 × 90 s = 24 min of preflight per sidecar start; a 30 s probe deadline would make it 8 min (and 180 s → 60 s on that run).

Why it is not an engineering call. ADR-0005 §Superseding decision: "ADR 0003 governs these operations. Inference, initial ping/preflight, warmup, retry/repair, provider discovery, OpenRouter ZDR lookup, DNS/TLS setup, and local health checks have no fixed wall-clock timeout. Work ends only through an operator action or cancellation of an obsolete PR head." CLAUDE.md (model-path timeouts are policy-fixed; #1889/#1890/#1892 added 900 s caps on real hang evidence and were all reverted; never convert elapsed inference time into a model-failure verdict). A launcher-set timeout=30 that turns a slow probe into rejected is exactly "elapsed time → route verdict", on the operation the ADR names. The 90 s we have today is the vendored transport's recv default, not a deadline this repo set. #1947 (defer on 429/5xx) and #1949 (probe-count cap, account skip on 429 evidence) bound wall time without using time as a verdict, so they stay inside the policy.

Decision requested. (a) Keep ADR-0003/0005 as written — preflight probes stay on the transport default; wall time is bounded only by probe count and evidence-based skips (status quo, #1949). (b) Amend ADR-0003 to state that a preflight probe deadline is a distinct policy value from the inference deadline (precedent: contextual-orchestrator's probe() at 5 s), and set it (30 s proposed; 20 s is tight for the escalated probe, which shares the client). If (b), the change is one constant + one timeout= kwarg at :981 + one test, stacked on #1949 after it lands; a timed-out probe records error_type=TimeoutError with no http_status and is rejected (not deferred), consistent with #1947's "silent probe → reject".

Not implemented pending the decision (raised by a peer's policy check after I proposed it as a stacked commit).