From 3eae4e9e750338e731367b9326dbba37b349aed2 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 5 Sep 2026 02:16:58 +0000 Subject: [PATCH 01/12] docs: confirm review pipeline already routes through orchestrator/free, not NIM directly Audited opencode.jsonc, opencode-review-dispatch.yml's model candidates, and contextual_orchestrator_review_sidecar.sh: Noema/OpenCode review and Strix security review (via pr-review-autofix's sidecar) already route exclusively through contextual-orchestrator's orchestrator/free pool. NVIDIA NIM keys only flow in as bootstrap KV credentials for the vendored gateway's own model discovery; no workflow or script makes a direct HTTP call to a NIM endpoint. Also corrects a stale gap-baseline note: the hourly-cron test flagged not-yet-fixed on 2026-09-04 was in fact fixed by #1877 the same day (test_review_fix_caller_keeps_the_github_daily_recovery_slot). Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_01KPmJErfkcHer4UVEgrQxUX --- CHANGELOG.md | 10 +++++ docs/product-technical-gap-baseline.md | 55 +++++++++++++++++++++++--- 2 files changed, 60 insertions(+), 5 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 0a1552edf4..b021fc4d73 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,6 +11,16 @@ - Raised `hourly-review-repair.yml`'s discovery ceiling from 50 to 200 while rotating deterministic 50-PR deep-inspection windows by hourly run number. The scheduler hydrates only the selected window and stops immediately after its single dispatch, preserving access to newer PRs without quadrupling expensive review/check/comment work. See `docs/doctoring/hourly-review-repair-single-file-consolidation.md`'s 2026-09-03 follow-up. ## [Unreleased] +- **Document (no code change): confirmed Noema/OpenCode/Strix review already routes exclusively through + contextual-orchestrator's `orchestrator/free`, with no direct NVIDIA NIM communication.** Audited + `opencode.jsonc` (only `contextual-orchestrator` enabled, model pinned to `orchestrator/free`), + `opencode-review-dispatch.yml`'s `OPENCODE_MODEL_CANDIDATES`, and + `scripts/ci/contextual_orchestrator_review_sidecar.sh` (NIM keys are forwarded only as bootstrap KV + credentials into the vendored gateway process for model discovery; the actual review call targets the + sidecar's own loopback endpoint with `CONTEXTUAL_ORCHESTRATOR_POOL` hard-locked to `free`). Also fixed a + stale gap-baseline note: `tests/test_pr_review_autofix_nvidia_nim_contract.py`'s hourly-cron test, flagged + not-yet-fixed on 2026-09-04, was fixed by `#1877` the same day. See + `docs/product-technical-gap-baseline.md`'s 2026-09-05 entry for the full audit trail. - **Fix current-main contract drift that blocked the unscoped `agent-review-runtime-quality-ci.yml` "Verify scheduler and contextual-orchestrator review-repair contracts" step (which discovers and diff --git a/docs/product-technical-gap-baseline.md b/docs/product-technical-gap-baseline.md index 4ab5284a15..af6d135a83 100644 --- a/docs/product-technical-gap-baseline.md +++ b/docs/product-technical-gap-baseline.md @@ -3205,10 +3205,55 @@ others) — this fix deliberately stayed scoped to the one file with direct, con starvation rather than a speculative sweep of every remaining occurrence. Worth revisiting each individually if queuing symptoms recur on them specifically. -**Separately found while validating this fix, not yet fixed:** `tests/test_pr_review_autofix_nvidia_nim_contract.py::test_review_fix_caller_runs_once_each_hour` -fails on a clean `origin/main` checkout, independent of this fix — `hourly-review-repair.yml` was renamed to +**Separately found while validating this fix, since fixed:** `tests/test_pr_review_autofix_nvidia_nim_contract.py::test_review_fix_caller_runs_once_each_hour` +failed on a clean `origin/main` checkout, independent of this fix — `hourly-review-repair.yml` was renamed to "Daily Review Recovery" and redesigned from one hourly cron to 17 staggered daily crons (one per target -repository), but this test still asserts the old single hourly `cron: "23 * * * *"`. Same bug class as the +repository), but this test still asserted the old single hourly `cron: "23 * * * *"`. Same bug class as the `test_strix_quick_gate.sh` org-sweep-cron staleness found and fixed on `#1503` the same day: a test left -behind by a workflow redesign. Needs its own fix understanding the new staggered-daily design's actual -intended contract before rewriting the assertion — left for a dedicated follow-up rather than guessed at here. +behind by a workflow redesign. `#1877` ("fix(tests): repair changed-scope drift and stale noema cancel-step +test") rewrote it as `test_review_fix_caller_keeps_the_github_daily_recovery_slot`, asserting the central +repo's actual daily slot (`cron: "21 6 * * *"`) and that no product repository is hard-coded into the +reusable scheduler; merged as `12fe2d19`/`b5efbc27`. A parallel attempt (`#1875`) proposed an equivalent fix +independently and was closed as fully redundant once `#1877` landed the same contract — see that PR's own +closing comment for the redundant-assertion list. Confirmed live: `origin/main` at `8272e4f9` carries the +renamed test with no `test_review_fix_caller_runs_once_each_hour` symbol remaining. + +## Noema/OpenCode/Strix review already routes through contextual-orchestrator's `orchestrator/free`, not NVIDIA NIM directly — 2026-09-05 + +**Status:** Confirmed already implemented; no code change needed. This cycle's directive explicitly named +"Noema/OpenCode 리뷰·태그·PR 충돌 자동 해결·리뷰 반영, Strix 보안 리뷰가 contextual-orchestrator를 통해 +orchestrator/free로 이루어지는 것" (Noema/OpenCode review, tagging, PR-conflict auto-resolution and +review-reflection, and Strix security review, going through contextual-orchestrator's `orchestrator/free`) +as its target, with direct NIM communication called out as a removal target. A targeted audit of the +central review pipeline found no violation: + +- `opencode.jsonc` declares `enabled_providers: ["contextual-orchestrator"]` only; `model`/`small_model` + are pinned to `"contextual-orchestrator/orchestrator/free"`. No `nvidia-nim` provider block exists — it + was deliberately removed (`docs/doctoring/opencode-jsonc-nvidia-nim-block-removal.md`, 2026-08-31). +- `.github/workflows/opencode-review-dispatch.yml`'s `OPENCODE_MODEL_CANDIDATES` names only + `contextual-orchestrator/orchestrator/free` — no paid or auto-selected candidate is dispatched, matching + this cycle's stated rationale ("free+ZDR 조합도 해결 못 하는데 유료 모델 포함 auto는 의미 없다"). +- `scripts/ci/contextual_orchestrator_review_sidecar.sh` (the vendored gateway `pr-review-autofix.yml` + provisions for Noema/OpenCode/Strix repair) treats `NVIDIA_NIM_API_KEY`/`NVIDIA_NIM_API_KEY_SUB` only as + bootstrap credentials forwarded into the vendored orchestrator's own KV for `discover_all_models()` — the + actual review completion call targets the sidecar's own loopback `http://127.0.0.1:18080/v1/chat/completions` + with `model=orchestrator/{pool}`, and any `CONTEXTUAL_ORCHESTRATOR_POOL` other than `free` is hard-rejected + at script level. No `.github`-side HTTP client calls an NVIDIA NIM endpoint directly. +- A prior direct-HTTP NIM resolver (`scripts/ci/select_nvidia_nim_model.py`) was already removed on + 2026-08-30 after confirming zero callers (`docs/doctoring/direct-nvidia-nim-communication-removal.md`). + Legacy `is_nvidia_nim_candidate`/`is_schema_repair_candidate` branches remain in + `scripts/ci/run_opencode_review_model_pool.sh` but are inert dead code — no `nvidia-nim/*` candidate is + ever configured, and `scripts/ci/test_strix_quick_gate.sh` asserts none of the workflow files contain an + `nvidia-nim/` string. +- `scripts/ci/zdr_policy.py` marks `nvidia_nim`/`nvidia_nim_sub` as explicitly **not** ZDR (NVIDIA's trial + ToS trains on submitted data) and ZDR-attests only `openrouter` via its live `/api/v1/endpoints/zdr` feed; + the sidecar's discovered-model catalog is ZDR-prioritized within the fail-closed zero-cost `free` pool + rather than a claim that every route through it is ZDR — this matches, and does not contradict, the + "free+ZDR" framing already in the codebase. + +Noema's own repo (`ContextualWisdomLab/noema`) is architecturally clean by construction here too: it is an +OIDC-to-installation-token credential broker, not an LLM caller, and holds no upstream provider keys — every +LLM path it participates in (production review, hourly product development, naruon judgments) calls +`contextual-orchestrator` per its own `CLAUDE.md`. `contextual-orchestrator`'s own direct NIM calls (inside +`model_discovery.py`/`zdr_policy.py`) are the intended single point of contact, not a violation — the +directive's removal target is *other systems bypassing the gateway*, which does not occur here. From 50de5f6327a88035ce28a911e2a616330dccc1ce Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 5 Sep 2026 08:11:13 +0000 Subject: [PATCH 02/12] docs: correct overbroad "already implemented" claim on NIM-routing gap entry @seonghobae disputed the original "Confirmed already implemented; no code change needed" framing on this PR. Independently re-verified all three of their points against exact file:line evidence before applying this correction: the model-selection/logical-routing layer (opencode.jsonc, opencode-review-dispatch.yml) is correct as originally audited, but the sidecar/egress infrastructure layer four consumers actually run on (scripts/ci/contextual_orchestrator_review_sidecar.sh) still injects five raw provider secrets, clones and runs contextual-orchestrator fresh on the calling runner, performs discovery in-process there, and leaves strix.yml's harden-runner at egress-policy: audit rather than block. Corrects the gap baseline entry and its matching CHANGELOG.md entry to state this accurately, with citations to .github#1759 and contextual-orchestrator#1041 comment 5550412102 for the closing work still needed. Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_01KPmJErfkcHer4UVEgrQxUX --- CHANGELOG.md | 27 ++++++----- docs/product-technical-gap-baseline.md | 62 ++++++++++++++++++++------ 2 files changed, 65 insertions(+), 24 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index b021fc4d73..232dc68fee 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,16 +11,23 @@ - Raised `hourly-review-repair.yml`'s discovery ceiling from 50 to 200 while rotating deterministic 50-PR deep-inspection windows by hourly run number. The scheduler hydrates only the selected window and stops immediately after its single dispatch, preserving access to newer PRs without quadrupling expensive review/check/comment work. See `docs/doctoring/hourly-review-repair-single-file-consolidation.md`'s 2026-09-03 follow-up. ## [Unreleased] -- **Document (no code change): confirmed Noema/OpenCode/Strix review already routes exclusively through - contextual-orchestrator's `orchestrator/free`, with no direct NVIDIA NIM communication.** Audited - `opencode.jsonc` (only `contextual-orchestrator` enabled, model pinned to `orchestrator/free`), - `opencode-review-dispatch.yml`'s `OPENCODE_MODEL_CANDIDATES`, and - `scripts/ci/contextual_orchestrator_review_sidecar.sh` (NIM keys are forwarded only as bootstrap KV - credentials into the vendored gateway process for model discovery; the actual review call targets the - sidecar's own loopback endpoint with `CONTEXTUAL_ORCHESTRATOR_POOL` hard-locked to `free`). Also fixed a - stale gap-baseline note: `tests/test_pr_review_autofix_nvidia_nim_contract.py`'s hourly-cron test, flagged - not-yet-fixed on 2026-09-04, was fixed by `#1877` the same day. See - `docs/product-technical-gap-baseline.md`'s 2026-09-05 entry for the full audit trail. +- **Document (no code change yet, correction to an earlier same-day entry): Noema/OpenCode/Strix review's + model-selection layer routes through contextual-orchestrator's `orchestrator/free`, but the sidecar/egress + infrastructure layer four consumers actually run on does not yet.** This entry originally claimed the + routing was already fully implemented; @seonghobae disputed that framing on `.github#1884` and the dispute + held up under independent re-verification. `opencode.jsonc` (only `contextual-orchestrator` enabled, model + pinned to `orchestrator/free`) and `opencode-review-dispatch.yml`'s `OPENCODE_MODEL_CANDIDATES` are + correct as originally audited. But `scripts/ci/contextual_orchestrator_review_sidecar.sh` — the runtime + path `noema-review.yml`, `strix.yml`, `opencode-review-dispatch.yml`, and `pr-review-autofix.yml` all still + use — still injects all five raw provider secrets (including both NIM keys), clones and runs + `contextual-orchestrator` fresh on the calling runner per invocation, and performs multi-provider model + discovery in-process on that runner; `strix.yml`'s `harden-runner` step is still `egress-policy: audit`, + not `block`. None of the four consumers has migrated onto the newer `orchestrator-free-sidecar` composite + action (`.github#1736`) meant to centralize this. Tracked by `.github#1759` and + `contextual-orchestrator#1041` comment `5550412102`. Also fixed a stale gap-baseline note: + `tests/test_pr_review_autofix_nvidia_nim_contract.py`'s hourly-cron test, flagged not-yet-fixed on + 2026-09-04, was fixed by `#1877` the same day. See `docs/product-technical-gap-baseline.md`'s 2026-09-05 + entry (corrected 2026-09-05) for the full audit trail. - **Fix current-main contract drift that blocked the unscoped `agent-review-runtime-quality-ci.yml` "Verify scheduler and contextual-orchestrator review-repair contracts" step (which discovers and diff --git a/docs/product-technical-gap-baseline.md b/docs/product-technical-gap-baseline.md index d3ea067bc5..50619fae54 100644 --- a/docs/product-technical-gap-baseline.md +++ b/docs/product-technical-gap-baseline.md @@ -3238,14 +3238,19 @@ independently and was closed as fully redundant once `#1877` landed the same con closing comment for the redundant-assertion list. Confirmed live: `origin/main` at `8272e4f9` carries the renamed test with no `test_review_fix_caller_runs_once_each_hour` symbol remaining. -## Noema/OpenCode/Strix review already routes through contextual-orchestrator's `orchestrator/free`, not NVIDIA NIM directly — 2026-09-05 +## Noema/OpenCode/Strix review's model-selection layer routes through contextual-orchestrator's `orchestrator/free`; the sidecar/egress infrastructure layer does not yet — 2026-09-05, corrected 2026-09-05 -**Status:** Confirmed already implemented; no code change needed. This cycle's directive explicitly named -"Noema/OpenCode 리뷰·태그·PR 충돌 자동 해결·리뷰 반영, Strix 보안 리뷰가 contextual-orchestrator를 통해 -orchestrator/free로 이루어지는 것" (Noema/OpenCode review, tagging, PR-conflict auto-resolution and -review-reflection, and Strix security review, going through contextual-orchestrator's `orchestrator/free`) -as its target, with direct NIM communication called out as a removal target. A targeted audit of the -central review pipeline found no violation: +**Status:** Partially implemented; code change still needed. This entry originally read "Confirmed already +implemented; no code change needed." That framing was too broad and has been corrected in place after +@seonghobae disputed it on this entry's own PR (`ContextualWisdomLab/.github#1884`) with a three-part +architectural read; each of the three points was independently re-verified against exact file:line evidence +before this correction, and all three held up. This cycle's directive explicitly named "Noema/OpenCode +리뷰·태그·PR 충돌 자동 해결·리뷰 반영, Strix 보안 리뷰가 contextual-orchestrator를 통해 orchestrator/free로 +이루어지는 것" (Noema/OpenCode review, tagging, PR-conflict auto-resolution and review-reflection, and Strix +security review, going through contextual-orchestrator's `orchestrator/free`) as its target, with direct NIM +communication called out as a removal target. The audit needs to be read as two separate layers, not one: + +**Model-selection / logical-routing layer — confirmed correct, no violation:** - `opencode.jsonc` declares `enabled_providers: ["contextual-orchestrator"]` only; `model`/`small_model` are pinned to `"contextual-orchestrator/orchestrator/free"`. No `nvidia-nim` provider block exists — it @@ -3253,12 +3258,6 @@ central review pipeline found no violation: - `.github/workflows/opencode-review-dispatch.yml`'s `OPENCODE_MODEL_CANDIDATES` names only `contextual-orchestrator/orchestrator/free` — no paid or auto-selected candidate is dispatched, matching this cycle's stated rationale ("free+ZDR 조합도 해결 못 하는데 유료 모델 포함 auto는 의미 없다"). -- `scripts/ci/contextual_orchestrator_review_sidecar.sh` (the vendored gateway `pr-review-autofix.yml` - provisions for Noema/OpenCode/Strix repair) treats `NVIDIA_NIM_API_KEY`/`NVIDIA_NIM_API_KEY_SUB` only as - bootstrap credentials forwarded into the vendored orchestrator's own KV for `discover_all_models()` — the - actual review completion call targets the sidecar's own loopback `http://127.0.0.1:18080/v1/chat/completions` - with `model=orchestrator/{pool}`, and any `CONTEXTUAL_ORCHESTRATOR_POOL` other than `free` is hard-rejected - at script level. No `.github`-side HTTP client calls an NVIDIA NIM endpoint directly. - A prior direct-HTTP NIM resolver (`scripts/ci/select_nvidia_nim_model.py`) was already removed on 2026-08-30 after confirming zero callers (`docs/doctoring/direct-nvidia-nim-communication-removal.md`). Legacy `is_nvidia_nim_candidate`/`is_schema_repair_candidate` branches remain in @@ -3271,9 +3270,44 @@ central review pipeline found no violation: rather than a claim that every route through it is ZDR — this matches, and does not contradict, the "free+ZDR" framing already in the codebase. +**Sidecar / egress infrastructure layer — not yet complete, this is the corrected part:** + +`scripts/ci/contextual_orchestrator_review_sidecar.sh` is the actual runtime egress path for all four +consumers that need it (`noema-review.yml`, `strix.yml`, `opencode-review-dispatch.yml`, +`pr-review-autofix.yml`, per `ContextualWisdomLab/.github#1759`'s migration-order tracking), and it is not +the thin, ZDR-attested gateway call the original framing implied: + +- It still requires all five raw provider secrets — `BYTEZ_API_KEY`, `NVIDIA_NIM_API_KEY`, + `NVIDIA_NIM_API_KEY_SUB`, `OPENROUTER_API_KEY`, `OPENAI_API_KEY` — injected into it at the workflow level + (e.g. `.github/workflows/strix.yml:745-758`), not just `contextual-orchestrator`'s own KV. +- It clones `contextual-orchestrator` source at a pinned SHA and builds/runs it fresh on the calling runner + at request time (`scripts/ci/contextual_orchestrator_review_sidecar.sh` lines 51-63 and 91-98), rather than + invoking a pre-built, immutable released artifact. +- Model discovery (`discover_all_models()`) runs in-process, locally, on that same runner, against all five + injected provider credentials (sidecar script lines 108-109 and 316-328) — i.e. the runner itself performs + the multi-provider discovery the "free+ZDR only" framing above describes as `contextual-orchestrator`'s + internal concern, not something every calling workflow's runner should be doing with raw provider keys. +- None of the four consumers has migrated to the newer composite action `orchestrator-free-sidecar` (added + in `ContextualWisdomLab/.github#1736`) that was meant to centralize this and remove the per-consumer + secret/clone/discovery duplication. +- `strix.yml`'s `step-security/harden-runner` step is still `egress-policy: audit` (`strix.yml:365-367`), not + `block` — consistent with a runner that still needs open egress for the five-secret, in-process discovery + path above, and itself evidence that the egress boundary this directive wants is not closed yet. + +Canonical tracking for closing this gap: `ContextualWisdomLab/.github#1759` (the consumer migration order) +and `ContextualWisdomLab/contextual-orchestrator#1041` comment `5550412102` (six requirements for an +"immutable released gateway/client/schema/egress contract" that must ship before consumers can drop the five +provider secrets and flip `strix.yml`'s runner egress policy to `block`). Until that contract ships and all +four consumers migrate onto it, "Strix 보안 리뷰가 contextual-orchestrator를 통해 orchestrator/free로 +이루어지는 것" is true at the model-selection layer only — the underlying egress path a reviewer of Strix's +actual network behavior would see still runs raw provider secrets and in-process multi-provider discovery on +the calling runner, which is the shape this directive's "NIM 직접 통신은 제거 대상" line is aimed at closing. + Noema's own repo (`ContextualWisdomLab/noema`) is architecturally clean by construction here too: it is an OIDC-to-installation-token credential broker, not an LLM caller, and holds no upstream provider keys — every LLM path it participates in (production review, hourly product development, naruon judgments) calls `contextual-orchestrator` per its own `CLAUDE.md`. `contextual-orchestrator`'s own direct NIM calls (inside `model_discovery.py`/`zdr_policy.py`) are the intended single point of contact, not a violation — the -directive's removal target is *other systems bypassing the gateway*, which does not occur here. +directive's removal target is *other systems bypassing the gateway*, which does not occur in Noema's own +repo. It does occur, in the narrower sidecar-secrets sense above, in the four `.github`-side consumers until +they migrate onto the immutable gateway contract tracked by `#1759`/`contextual-orchestrator#1041`. From cbb61c5a81ddbf2a64f489bb61d72f67e8088f18 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 6 Sep 2026 07:10:10 +0000 Subject: [PATCH 03/12] docs(gap-baseline): record the orchestrator/free retry-stacking root cause, pin advance, and first post-pin measurement Extends the 2026-09-05 NIM-routing entry with a dated follow-up: the _invoke x _send_with_retry retry multiplication (contextual-orchestrator#1081), the ModelClient.single_attempt_transport() fix, the hard-coded sidecar pin that kept the fix out of production until #1951, the rule for which runs count as post-advance evidence, and .github#1661 run 34008191123 as the first run vendoring 414f2297 (pin live; preflight 0/12 ready, so the fix is not yet exercised end to end). This PR's CHANGELOG entry gains the same pointer. Documentation only. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01KPmJErfkcHer4UVEgrQxUX --- CHANGELOG.md | 6 ++- docs/product-technical-gap-baseline.md | 59 ++++++++++++++++++++++++++ 2 files changed, 64 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index c8778e72d9..3c015f0aa4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -64,7 +64,11 @@ `contextual-orchestrator#1041` comment `5550412102`. Also fixed a stale gap-baseline note: `tests/test_pr_review_autofix_nvidia_nim_contract.py`'s hourly-cron test, flagged not-yet-fixed on 2026-09-04, was fixed by `#1877` the same day. See `docs/product-technical-gap-baseline.md`'s 2026-09-05 - entry (corrected 2026-09-05) for the full audit trail. + entry (corrected 2026-09-05) for the full audit trail. 2026-09-06 follow-up in the same entry: records the + `orchestrator/free` retry-stacking root cause (`contextual-orchestrator#1081`), the sidecar pin advance that + delivered it (`#1951`), the rule for what counts as a post-advance run, and the first post-pin measurement + (`.github#1661` run `34008191123`: pin `414f2297…` live, preflight 0 of 12 routes ready, review request never + made) — status "fixed, delivery confirmed, effect unconfirmed". - Include merge-scheduler entrypoint, core, and regression-test changes in the existing runtime-quality workflow's trigger and suite selector. Scheduler workflow edits retain queue checks and also select the full review-repair diff --git a/docs/product-technical-gap-baseline.md b/docs/product-technical-gap-baseline.md index 3ff0c2c7a4..7bb1edec4b 100644 --- a/docs/product-technical-gap-baseline.md +++ b/docs/product-technical-gap-baseline.md @@ -3318,6 +3318,65 @@ directive's removal target is *other systems bypassing the gateway*, which does repo. It does occur, in the narrower sidecar-secrets sense above, in the four `.github`-side consumers until they migrate onto the immutable gateway contract tracked by `#1759`/`contextual-orchestrator#1041`. +### 2026-09-06 follow-up: the `orchestrator/free` pool's retry-stacking defect — root cause, fix, pin advance, first post-pin measurement + +**Status:** Fixed upstream and delivered to the central sidecar; end-to-end effect not yet observed because +provider availability at preflight is now the earlier failure. This is a separate defect from the +sidecar/egress layer gap above (still open under `#1759`) and does not change that gap's status. + +- **Symptom.** Independently observed `noema-review`/`strix`/`opencode-review` incidents on `#1912`, `#1231`, + `#1503`, and `#1198` each spent 9–57+ minutes on one escalated route and surfaced that same route's model + (`deepseek-ai/deepseek-v4-flash-0731`) in the final error, never reaching a cleanly-ready sibling that + preflight had already found. `#1187`'s `noema-review` job `101326875524` (2026-09-05 16:38–17:38Z, old pin) + has the same shape: `HTTP Error 502 ... caller attempts=1, duration=2343.1s, phase=response_error, + served_model=google/gemma-4-31b-it`. +- **Root cause** (`ContextualWisdomLab/contextual-orchestrator#1081`). `TaskOrchestrator._invoke`'s own + retry-then-failover decision for a retryable 5xx (budgeted `1 + tool_retry_attempts` real tries per + candidate) was multiplied by `ModelClient._send_with_retry`'s independent transient-retry-with-backoff loop + underneath it (`max_retries + 1` further tries per call) — up to 6 real network attempts against one + already-flagged-flaky agent before `_invoke` tried the next ranked candidate. Reproduced against unmodified + `contextual-orchestrator` `main` with a real `_send_with_retry` over a flaky `_send` (6 attempts) and + confirmed fixed (≤ 2) by + `tests/test_provider_reliability.py::test_free_pool_failover_does_not_multiply_transport_retries_on_one_flaky_agent`. + A second lane reached the same reading independently from the first sidecar DEBUG trace (`.github#1661` + run `33995553859`: "the review path is `_invoke`: a silent route costs two rounds of three 90 s timeouts", + commit `37a1129a`). +- **Fix.** `ModelClient.single_attempt_transport()`, a thread-local context manager that `_invoke` wraps + around each per-agent `chat()` call so the transport layer makes exactly one attempt per `_invoke` try. + Only which agent gets tried next changes; no per-attempt timeout, `max_retries`, or backoff value moved + (ADR-0003 forbids a wall-clock timeout on the inference path). Merged as + `contextual-orchestrator@414f22973658c4ddc3d4320fcf7acd9b4e8ba991`. +- **Delivery gap found and closed the same night.** Merging the upstream fix had no production effect by + itself: `scripts/ci/contextual_orchestrator_review_sidecar.sh` hard-codes `ORCHESTRATOR_PIN_SHA`'s default + and does not track `contextual-orchestrator` `main`, so every review sidecar kept vendoring + `2e414d15ba58f28597751b625a8a2f00fc9fadcf` (two days older than the fix). `#1951` (merged + `efb8926923de45245338159a489a1b227e81945f`, 2026-09-06 03:01Z) advanced the pin together with + `tests/test_contextual_orchestrator_review_sidecar_contract.py`'s `ORCH_PIN_SHA`, ADR-0003's new 2026-09-06 + amendment, and `CHANGELOG.md`. It was bypass-merged under this cycle's explicit authorization because the + PR's own required reviews ran through the base branch's still-stale sidecar (`pull_request_target` trust + boundary) — the chicken-and-egg case the advance exists to resolve. +- **Verification rule (recorded because it was applied wrongly once).** A rerun of a pre-advance job, and even + a fresh internal retry against an unchanged PR head, replays the trusted-source ref resolved at the original + dispatch, so it still vendors the old pin no matter when it executes (`#1280`'s `noema-review` started + 03:42Z, after the advance, and still logged `vendoring contextual-orchestrator @ 2e414d15…`). The only valid + evidence is a run triggered by a head pushed after 03:00Z whose sidecar log reads + `vendoring contextual-orchestrator @ 414f2297…`. +- **First post-pin measurement.** `.github#1661` run `34008191123`, `noema-review` job `101424607975` + (2026-09-06 04:46Z) is the first such run. It logs `vendoring contextual-orchestrator @ + 414f22973658c4ddc3d4320fcf7acd9b4e8ba991`, so the advance is live for every sidecar consumer. That run then + failed before any review request was made: preflight probed 12 routes and found 0 ready (429 on both NVIDIA + keys' `deepseek-v4-flash` and on three OpenRouter free routes, 90 s `TimeoutError` on both keys' + `deepseek-v4-pro`, 404 on NIM's `gemma-3-12b`/`gemma-3-4b`) and the sidecar exited before `healthz`. The + retry-stacking fix has therefore not yet been exercised end to end; the blocker moved from the gateway's + failover logic to route availability at preflight, which `#1947` (transient-rejected routes kept as deferred + failover), `#1949` / `docs/adr/0029-sidecar-preflight-lazy-fill.md` (lazy fill to a readiness target of 8 + within 16 probes, account skip after two consecutive 429s), and `#1950` (per-traceback exception evidence) + address — all merged 2026-09-06 03:01–05:15Z, after that run's trusted source was resolved. +- **What closes this follow-up.** One `noema-review`, `opencode-review`, or `strix` run on a head pushed after + 05:15Z whose preflight reports `ready_count ≥ 1` and whose review request either succeeds or fails over past + the first stalled route within one `_invoke` try budget (≤ 2 transport attempts per agent in the sidecar + DEBUG trace). Until then this item reads "fixed, delivery confirmed, effect unconfirmed". + ## Items 15/16/17 measurement: `Detect changed scope` gate jobs — 2 of 3 are pure runner overhead — 2026-09-05 **Status:** Measured, not yet fixed. Recorded so the fix is grounded in real numbers rather than the intuition From b35947ea1a893c5a4e3b484fa347a6a52218079a Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 6 Sep 2026 10:00:18 +0000 Subject: [PATCH 04/12] docs(gap-baseline): close the retry-stacking follow-up with its production measurement The 2026-09-06 follow-up now records the confirming evidence rather than leaving the effect unverified. .github#1946 run 34008655765 measured 180 s per gateway request (two 90 s tries on the one ready route) against 540 s under the old pin, and the three post-advance Strix scans show attempt=1/1 throughout, so contextual-orchestrator#1081 is closed. It also separates what remains from what was fixed: capacity (#1948), with a four-run census 08:27-09:11Z in which #1187, #1411 (noema-review and strix) and #1884 all report identical preflight evidence -- 6 probes, 6 x 429 across three accounts, 18 candidates skipped, 0 ready -- and contextual-orchestrator#1082, where a 90 s passthrough timeout surfaces as 500 internal_error with _record_failure unreachable, so the same silent route is re-selected on every retry. Measurements from a second lane's postings on #1948; job logs re-read directly for the four boots cited here. Documentation only. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01KPmJErfkcHer4UVEgrQxUX --- CHANGELOG.md | 5 +++- docs/product-technical-gap-baseline.md | 39 +++++++++++++++++++++----- 2 files changed, 36 insertions(+), 8 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 3c015f0aa4..8840240e19 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -68,7 +68,10 @@ `orchestrator/free` retry-stacking root cause (`contextual-orchestrator#1081`), the sidecar pin advance that delivered it (`#1951`), the rule for what counts as a post-advance run, and the first post-pin measurement (`.github#1661` run `34008191123`: pin `414f2297…` live, preflight 0 of 12 routes ready, review request never - made) — status "fixed, delivery confirmed, effect unconfirmed". + made), then the confirming measurements (`.github#1946` run `34008655765`: 180 s per gateway request against + 540 s under the old pin; three post-advance Strix scans at `attempt=1/1` throughout) — status "fixed, + delivery confirmed, effect confirmed"; the residuals are capacity (`#1948`) and + `contextual-orchestrator#1082`. - Include merge-scheduler entrypoint, core, and regression-test changes in the existing runtime-quality workflow's trigger and suite selector. Scheduler workflow edits retain queue checks and also select the full review-repair diff --git a/docs/product-technical-gap-baseline.md b/docs/product-technical-gap-baseline.md index 7bb1edec4b..28b86c59aa 100644 --- a/docs/product-technical-gap-baseline.md +++ b/docs/product-technical-gap-baseline.md @@ -3320,9 +3320,11 @@ they migrate onto the immutable gateway contract tracked by `#1759`/`contextual- ### 2026-09-06 follow-up: the `orchestrator/free` pool's retry-stacking defect — root cause, fix, pin advance, first post-pin measurement -**Status:** Fixed upstream and delivered to the central sidecar; end-to-end effect not yet observed because -provider availability at preflight is now the earlier failure. This is a separate defect from the -sidecar/egress layer gap above (still open under `#1759`) and does not change that gap's status. +**Status:** Fixed upstream, delivered to the central sidecar, and effect confirmed in production (the +05:23Z–07:27Z measurements below). What remains on the review path is capacity (`#1948`) and a separate +timeout-classification defect on the tool-bearing passthrough path (`contextual-orchestrator#1082`), each +tracked under its own number. This is a separate defect from the sidecar/egress layer gap above (still open +under `#1759`) and does not change that gap's status. - **Symptom.** Independently observed `noema-review`/`strix`/`opencode-review` incidents on `#1912`, `#1231`, `#1503`, and `#1198` each spent 9–57+ minutes on one escalated route and surfaced that same route's model @@ -3372,10 +3374,33 @@ sidecar/egress layer gap above (still open under `#1759`) and does not change th failover), `#1949` / `docs/adr/0029-sidecar-preflight-lazy-fill.md` (lazy fill to a readiness target of 8 within 16 probes, account skip after two consecutive 429s), and `#1950` (per-traceback exception evidence) address — all merged 2026-09-06 03:01–05:15Z, after that run's trusted source was resolved. -- **What closes this follow-up.** One `noema-review`, `opencode-review`, or `strix` run on a head pushed after - 05:15Z whose preflight reports `ready_count ≥ 1` and whose review request either succeeds or fails over past - the first stalled route within one `_invoke` try budget (≤ 2 transport attempts per agent in the sidecar - DEBUG trace). Until then this item reads "fixed, delivery confirmed, effect unconfirmed". +- **Effect confirmed (measured by a second lane on `#1948`, 05:23Z and 07:27Z).** `.github#1946` run + `34008655765`, `noema-review` job `101427555591` (artifact `noema-sidecar-evidence` `9983259344`), the first + Noema run past route preflight on the new pin: each of the three gateway-preflight attempts was served with + exactly two 90 s tries on the one ready route (`attempt=1/1` at 05:11:54 and 05:13:24, then 05:14:55 and + 05:16:25, then 05:17:55 and 05:19:25; circuit `failures 1.0 → 3.0`), i.e. **180 s per gateway request** + against 540 s (6 × 90 s) under the old pin. The three post-advance Strix scans (`#1930` run `34008575120`, + `#1916` run `34008489633`, `#1946` run `34008655751`) show `attempt=1/1` throughout with no `2/3`, and pushed + real work through the pool (up to 2 M input tokens on `#1916`) that no pre-advance scan on it managed. The + 6-per-candidate multiplication is gone; this defect is closed. +- **What remains on the review path is not this defect.** (i) Capacity, `#1948`: per-key free-tier budgets + shared by every concurrently booting sidecar. Four consecutive boots on 2026-09-06 between 08:27Z and + 09:11Z — `#1187` `noema-review` job `101451600433`, `#1411` `noema-review` `101453707588`, `#1411` `strix` + `101453993157`, `#1884` `noema-review` `101455078239` — report byte-identical preflight evidence: + `candidate_count 24, probe_budget 16, target_ready 8, probed 6, ready 0, rejected 6, deferred 0, + skipped 18`, the six rejections being 429 from both NVIDIA keys' `deepseek-v4-flash` and + `deepseek-v4-pro` and from two OpenRouter free routes, each answered within 90 ms. `#1949`'s account skip + then retires the remaining 18 candidates, so a fully rate-limited hour costs six probes and under half a + second instead of the whole budget — the fast-fail shape that walk was designed to produce, and still four + failed reviews. This is the dominant failure class on the review path today, and no change inside the + sidecar can manufacture capacity. (ii) + `contextual-orchestrator#1082`: on the tool-bearing passthrough path a 90 s timeout surfaces to the caller + as `500 internal_error` with `_record_failure` never reached, so the same silent first-ranked route is + re-selected on every retry (176 timeouts across those three Strix scans, ≈ 4.4 of their 5.6 runner-hours). + That is the next root cause in this chain and belongs to the gateway. (iii) `#1948`'s open owner decision + on whether a *preflight probe* deadline is a policy value distinct from the inference deadline + (ADR-0003/0005 forbid a wall-clock timeout on the inference path; the 90 s seen today is the transport's + recv default, not a deadline this repository set). ## Items 15/16/17 measurement: `Detect changed scope` gate jobs — 2 of 3 are pure runner overhead — 2026-09-05 From 6408ff2162411d17da40ac82590b1f4e4b1465b7 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 6 Sep 2026 10:07:13 +0000 Subject: [PATCH 05/12] docs(gap-baseline): attribute the daily-recovery cron slots to their real tests CodeRabbit's finding on b35947ea was valid, and the sentence carried two errors, not one. test_review_fix_caller_keeps_the_github_daily_recovery_slot asserts cron "23 7 * * *" -- the clearfolio row of hourly-review-repair.yml's github.event.schedule lookup table -- plus the absence of the old hourly "23 * * * *" and the reusable-scheduler reference. The central repository's own slot is a different row, cron "21 6 * * *", asserted by test_github_hourly_conflict_repair.py::test_central_repository_has_daily_self_caller; both are pinned as data in test_hourly_review_repair_callers.py. The paragraph attributed the 21 6 value to the #1877 test, conflating the two contracts and leaving a wrong verification record in the baseline. Verified against the workflow's own cron-to-repository mapping and all four tests before rewriting. Documentation only. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01KPmJErfkcHer4UVEgrQxUX --- docs/product-technical-gap-baseline.md | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-) diff --git a/docs/product-technical-gap-baseline.md b/docs/product-technical-gap-baseline.md index 28b86c59aa..7ab5816a09 100644 --- a/docs/product-technical-gap-baseline.md +++ b/docs/product-technical-gap-baseline.md @@ -3233,10 +3233,17 @@ failed on a clean `origin/main` checkout, independent of this fix — `hourly-re repository), but this test still asserted the old single hourly `cron: "23 * * * *"`. Same bug class as the `test_strix_quick_gate.sh` org-sweep-cron staleness found and fixed on `#1503` the same day: a test left behind by a workflow redesign. `#1877` ("fix(tests): repair changed-scope drift and stale noema cancel-step -test") rewrote it as `test_review_fix_caller_keeps_the_github_daily_recovery_slot`, asserting the central -repo's actual daily slot (`cron: "21 6 * * *"`) and that no product repository is hard-coded into the -reusable scheduler; merged as `12fe2d19`/`b5efbc27`. A parallel attempt (`#1875`) proposed an equivalent fix -independently and was closed as fully redundant once `#1877` landed the same contract — see that PR's own +test") rewrote it as `test_review_fix_caller_keeps_the_github_daily_recovery_slot`, which asserts that the +caller still carries a distributed daily slot (`cron: "23 7 * * *"`, the `clearfolio` entry in the +`github.event.schedule` lookup table), that the old single hourly `cron: "23 * * * *"` is gone, and that the +caller still routes through `./.github/workflows/pr-review-fix-scheduler.yml`; merged as +`12fe2d19`/`b5efbc27`. The central repository's own daily slot is a different row of that table +(`cron: "21 6 * * *"` → `ContextualWisdomLab/.github`) and is asserted separately, by +`tests/test_github_hourly_conflict_repair.py::test_central_repository_has_daily_self_caller`; both slots are +also pinned as data in `tests/test_hourly_review_repair_callers.py`. An earlier revision of this paragraph +attributed the `21 6` value to the `#1877` test, conflating the two contracts. A parallel attempt (`#1875`) +proposed an equivalent fix independently and was closed as fully redundant once `#1877` landed the same +contract — see that PR's own closing comment for the redundant-assertion list. Confirmed live (re-verified during this entry's #1884 merge): current `origin/main`'s `tests/test_pr_review_autofix_nvidia_nim_contract.py` carries only `test_review_fix_caller_keeps_the_github_daily_recovery_slot`, with no `test_review_fix_caller_runs_once_each_hour` From cb01a3dae41ad9b099d3a1becf8393ea181fff42 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 6 Sep 2026 12:16:06 +0000 Subject: [PATCH 06/12] docs(gap-baseline): separate the capacity and timeout residuals with a boot that reached serving .github#1187's strix job 101451547867 is the first boot observed past route preflight on this repository: probed 16, ready 6, rejected 8, deferred 2, skipped 4, healthz confirmed after 400s. It provisioned at 08:37-08:49Z, between the 08:27Z and 08:53Z zero-ready boots, so readiness swings minute to minute rather than degrading monotonically, and #1949's lazy fill is visibly doing what it targeted -- one cheap probe each on NIM's permanently 404 gemma-3 entries, then on to gemma-4-31b, six served routes where the fixed first-four slice yielded two. The same boot is contextual-orchestrator#1082 end to end: with six ready routes it scanned 3h21m over two changed files and ended STRIX_PROVIDER_UNAVAILABLE with Vulnerabilities 0. Capacity was not the binding constraint there; the timeout classification was. The two residuals are now separable in this repository's own data. Also retires this entry's own re-run trigger. It proposed spending each held PR's sanctioned re-run once an artifact showed ready_count >= 1; this artifact meets that and still cost 3h21m for no verdict, so the trigger becomes: wait until #1082 lands and its fix reaches the sidecar pin. Adds the fifth zero-ready boot (11:35Z) showing the exhaustion is sustained rather than one burst. Documentation only. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01KPmJErfkcHer4UVEgrQxUX --- docs/product-technical-gap-baseline.md | 21 ++++++++++++++++++--- 1 file changed, 18 insertions(+), 3 deletions(-) diff --git a/docs/product-technical-gap-baseline.md b/docs/product-technical-gap-baseline.md index 7ab5816a09..039cd90f86 100644 --- a/docs/product-technical-gap-baseline.md +++ b/docs/product-technical-gap-baseline.md @@ -3399,12 +3399,27 @@ under `#1759`) and does not change that gap's status. `deepseek-v4-pro` and from two OpenRouter free routes, each answered within 90 ms. `#1949`'s account skip then retires the remaining 18 candidates, so a fully rate-limited hour costs six probes and under half a second instead of the whole budget — the fast-fail shape that walk was designed to produce, and still four - failed reviews. This is the dominant failure class on the review path today, and no change inside the - sidecar can manufacture capacity. (ii) + failed reviews. A fifth boot at 11:35Z (`#1884` `noema-review` job `101473734239`) repeats the same six + numbers two and a half hours later, so the exhaustion is sustained rather than one burst. No change inside + the sidecar can manufacture capacity. Readiness does, however, swing minute to minute rather than degrade + monotonically: `#1187`'s `strix` job `101451547867` provisioned at 08:37–08:49Z, between the 08:27Z and + 08:53Z zero-ready boots, and reported `probed 16, ready 6, rejected 8, deferred 2, skipped 4` with + `healthz and provider-route preflight confirmed after 400s` — six served routes where the pre-`#1949` + fixed first-four slice yielded two, with one cheap probe each spent on NIM's permanently-404 + `gemma-3-12b`/`gemma-3-4b` before the walk continued to `gemma-4-31b`. `#1949` therefore did what it + targeted; the zero-ready hours are capacity, not selection. (ii) `contextual-orchestrator#1082`: on the tool-bearing passthrough path a 90 s timeout surfaces to the caller as `500 internal_error` with `_record_failure` never reached, so the same silent first-ranked route is re-selected on every retry (176 timeouts across those three Strix scans, ≈ 4.4 of their 5.6 runner-hours). - That is the next root cause in this chain and belongs to the gateway. (iii) `#1948`'s open owner decision + That same `#1187` `strix` boot is this class end to end: having reached six ready routes it scanned for + 3 h 21 m over the PR's two changed files and ended `STRIX_PROVIDER_UNAVAILABLE ... exhausted` with + `Vulnerabilities 0` — capacity was not the binding constraint there, the timeout classification was. So + the two residuals are separable in this repository's own data, and this one owns the runner-hours whenever + preflight succeeds. It is the next root cause in this chain and belongs to the gateway. **Consequence for + re-runs:** an earlier revision of this entry proposed spending each held PR's sanctioned re-run once an + artifact showed `ready_count ≥ 1`. That trigger is retired as insufficient — this artifact meets it and + still cost 3 h 21 m for no verdict — in favour of waiting until `#1082` lands and its fix reaches the + sidecar pin. (iii) `#1948`'s open owner decision on whether a *preflight probe* deadline is a policy value distinct from the inference deadline (ADR-0003/0005 forbid a wall-clock timeout on the inference path; the 90 s seen today is the transport's recv default, not a deadline this repository set). From 9c010fcb0fea140c40e0d4eb0b5df4103c2b9c26 Mon Sep 17 00:00:00 2001 From: seonghobae Date: Sun, 6 Sep 2026 13:14:53 +0000 Subject: [PATCH 07/12] docs(gap-baseline): locate the 90s recv default at source and name its upstream fix MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Residual (iii) of the retry-stacking follow-up recorded that the 90 s seen on every stalled review request is "the transport's recv default, not a deadline this repository set", and left open whether a preflight probe deadline is a policy value distinct from the inference deadline. Half of that is no longer open. The default is `ModelClient.__init__(timeout: int = 90)` at `contextual_orchestrator/orchestrator.py:1696`, read at `contextual-orchestrator@414f2297` — the SHA the central sidecar is pinned to — and `contextual-orchestrator#1053` changes exactly that signature to `timeout: float | None = None`, propagating the `None` through `_local_provider_slot`'s deadline arithmetic and, per its diff stat, through `endpoint_race.py`, `cost_router.py`, `batch_routing.py`, `server.py`, and the synchronous embedding path. So the inference-path half has a claimed upstream fix and only the preflight-probe deadline stays open in this repository. The entry also records that `#1053` and `#1082` do not subsume each other — `#1053` removes the 90 s attempt, `#1082` records and classifies a transport failure when one still arrives — and that the sidecar pin must advance past whichever lands last. Both are other lanes' active work; this is a dependency note, not a claim. Also merges `origin/main` (`dd0b96fe`) into the branch, which was `behind`. Verification: full suite 2956 passed, 1 skipped, 21 subtests passed; `git diff --check` clean. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01KPmJErfkcHer4UVEgrQxUX --- CHANGELOG.md | 7 ++++++- docs/product-technical-gap-baseline.md | 17 ++++++++++++++++- 2 files changed, 22 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 26cd337f45..f8f39aa33d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -91,7 +91,12 @@ made), then the confirming measurements (`.github#1946` run `34008655765`: 180 s per gateway request against 540 s under the old pin; three post-advance Strix scans at `attempt=1/1` throughout) — status "fixed, delivery confirmed, effect confirmed"; the residuals are capacity (`#1948`) and - `contextual-orchestrator#1082`. + `contextual-orchestrator#1082`. A further 2026-09-06 note locates the 90 s the residual entry attributes to + "the transport's recv default" at its source — `ModelClient.__init__(timeout: int = 90)`, + `contextual_orchestrator/orchestrator.py:1696` at `contextual-orchestrator@414f2297`, the pinned SHA — and + records `contextual-orchestrator#1053` as the upstream change that removes it (`timeout: float | None = + None`), so the inference-path half of that open question already has a claimed fix upstream and only the + preflight-probe deadline stays open here. - Include merge-scheduler entrypoint, core, and regression-test changes in the existing runtime-quality workflow's trigger and suite selector. Scheduler workflow edits retain queue checks and also select the full review-repair diff --git a/docs/product-technical-gap-baseline.md b/docs/product-technical-gap-baseline.md index 039cd90f86..fc18f1b01a 100644 --- a/docs/product-technical-gap-baseline.md +++ b/docs/product-technical-gap-baseline.md @@ -3422,7 +3422,22 @@ under `#1759`) and does not change that gap's status. sidecar pin. (iii) `#1948`'s open owner decision on whether a *preflight probe* deadline is a policy value distinct from the inference deadline (ADR-0003/0005 forbid a wall-clock timeout on the inference path; the 90 s seen today is the transport's - recv default, not a deadline this repository set). + recv default, not a deadline this repository set). **Located at source, 2026-09-06 13:10Z:** that + default is `ModelClient.__init__(timeout: int = 90)` (`contextual_orchestrator/orchestrator.py:1696`, + read at `contextual-orchestrator@414f2297`, the SHA the central sidecar is pinned to), and + `contextual-orchestrator#1053` ("fix(gateway): remove implicit model request timeout") changes exactly + that signature to `timeout: float | None = None`, propagating the `None` through + `_local_provider_slot`'s deadline arithmetic and, per its own description and diff stat, through + `endpoint_race.py`, `cost_router.py`, `batch_routing.py`, `server.py`, and the synchronous embedding + path (8 files, 129 insertions). So (iii) is not an open design question this repository has to answer + for the *inference* path: the 90 s there is an upstream library default with a claimed upstream fix, and + only the separate question of whether a *preflight probe* may carry a deadline of its own stays open + here. The two upstream changes do not subsume each other — `#1053` removes the 90 s attempt, `#1082` + records and classifies it when it still happens (a stalled socket has no timeout to hit once the default + is `None`, but a reset, truncated read, or upstream-imposed cutoff still arrives) — and the sidecar pin + must advance past whichever lands last before either reaches a review run. `#1053` was pushed to + `661ce8db` at 12:48Z with all 16 checks re-queued and `#1082` sits at `812bf11f` since 03:09Z, both + under other lanes' active work; this entry records the dependency, not a claim on either. ## Items 15/16/17 measurement: `Detect changed scope` gate jobs — 2 of 3 are pure runner overhead — 2026-09-05 From ace6f347109931061ee41b6e0bc4237e3b51620e Mon Sep 17 00:00:00 2001 From: seonghobae Date: Sun, 6 Sep 2026 15:43:08 +0000 Subject: [PATCH 08/12] docs(gap-baseline): record a fourth residual shape the first three do not describe MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `noema-review` failed on this PR's own head at 15:37Z (`9c010fcb`, run 34035522521, job 101501520756): Noema gateway transport failed: HTTPError: HTTP Error 502: Bad Gateway; caller attempts=1, duration=1424.1s, phase=response_error, served_model=deepseek-ai/deepseek-v4-flash-0731 None of residuals (i)-(iii) accounts for it: - Not capacity (i): a route was ready and `deepseek-v4-flash-0731` served, so preflight succeeded rather than exhausting at zero ready. - Not the raw-500 class (ii): the caller received the classified 502 that `contextual-orchestrator#1082` is adding, not an opaque `internal_error`. - Not the 90 s recv default (iii): one caller attempt ran 1424.1 s, about 23.7 minutes, roughly sixteen times that limit, and `phase=response_error` says a response arrived carrying an error status rather than a socket expiring — a different event from the `TimeoutError` in `#1053`'s own 90.054 s noema measurement. The reading this supports is that `caller attempts=1` bounds the caller only; the gateway owns repair and failover (its own warning line says so) and spent those 23.7 minutes walking the pool internally before classifying. Honest limit stated in the entry: this is the job log, not the `noema-sidecar-evidence` artifact (9992218398) that would give the internal attempt count and the distribution of the 23.7 minutes. No per-attempt breakdown is claimed. What the log establishes alone is that a served route plus a classified 502 plus a 23.7-minute wall clock is a real, current combination, so closing (i) and (ii) will not by itself account for it. Verification: full suite 2956 passed, 1 skipped, 21 subtests passed; `git diff --check` clean. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01KPmJErfkcHer4UVEgrQxUX --- CHANGELOG.md | 1 + docs/product-technical-gap-baseline.md | 20 ++++++++++++++++++++ 2 files changed, 21 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index f8f39aa33d..0887402814 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -97,6 +97,7 @@ records `contextual-orchestrator#1053` as the upstream change that removes it (`timeout: float | None = None`), so the inference-path half of that open question already has a claimed fix upstream and only the preflight-probe deadline stays open here. + A 2026-09-06 15:37Z measurement on this PR's own head adds a fourth residual shape none of the first three describes: `noema-review` on `9c010fcb` (run 34035522521) returned `HTTP Error 502: Bad Gateway; caller attempts=1, duration=1424.1s, phase=response_error, served_model=deepseek-ai/deepseek-v4-flash-0731`. A route was ready and served (so not the capacity class), the caller got a classified 502 rather than an opaque `internal_error` (so not the raw-500 class), and one caller attempt ran 23.7 minutes -- roughly sixteen times the 90 s that both residual (iii) and `contextual-orchestrator#1053` treat as the operative limit, with `phase=response_error` indicating a response arrived rather than a socket expiring. Read from the job log only; the `noema-sidecar-evidence` artifact that would give the gateway's internal attempt breakdown is not read, so none is claimed. - Include merge-scheduler entrypoint, core, and regression-test changes in the existing runtime-quality workflow's trigger and suite selector. Scheduler workflow edits retain queue checks and also select the full review-repair diff --git a/docs/product-technical-gap-baseline.md b/docs/product-technical-gap-baseline.md index fc18f1b01a..b35ccd52f7 100644 --- a/docs/product-technical-gap-baseline.md +++ b/docs/product-technical-gap-baseline.md @@ -3438,6 +3438,26 @@ under `#1759`) and does not change that gap's status. must advance past whichever lands last before either reaches a review run. `#1053` was pushed to `661ce8db` at 12:48Z with all 16 checks re-queued and `#1082` sits at `812bf11f` since 03:09Z, both under other lanes' active work; this entry records the dependency, not a claim on either. + (iv) **A fourth shape, measured on this PR's own head at 15:37Z, that none of (i)-(iii) describes.** + `noema-review` on `#1884` `9c010fcb` (run 34035522521, job 101501520756) failed with `Noema gateway + transport failed: HTTPError: HTTP Error 502: Bad Gateway; caller attempts=1, duration=1424.1s, + phase=response_error, served_model=deepseek-ai/deepseek-v4-flash-0731`. Three things separate it from + every sample above. It is **not** the capacity class (i): a route was ready and `deepseek-v4-flash-0731` + actually served, so preflight succeeded. It is **not** the raw-500 class (ii): the caller received a + classified 502 through the path `#1082` is adding, not an opaque `internal_error`. And the single caller + attempt ran **1424.1 s, about 23.7 minutes** — roughly sixteen times the 90 s that (iii) and + `contextual-orchestrator#1053` both treat as the operative limit, so the `ModelClient` default was not + what bounded this request. `phase=response_error` says a response arrived and carried an error status + rather than a socket expiring, which is a different event from the `TimeoutError` in `#1053`'s own + 90.054 s noema measurement. The reading this supports is that `caller attempts=1` bounds the *caller* + only — the gateway owns repair and failover, and it spent those 23.7 minutes walking its pool internally + before returning the classified 502 that its warning line calls out (`gateway owns repair/failover`). + **Honest limit:** this is read from the job log. The `noema-sidecar-evidence` artifact (9992218398, 2366 + bytes) holds the sidecar stderr and the preflight JSON that would give the internal attempt count and + how the 23.7 minutes was distributed; it is not read here, so no per-attempt breakdown is claimed. What + the job log does establish on its own is that a served route plus a classified 502 plus a + 23.7-minute wall clock is a real, current combination, and that closing (i) and (ii) will not by + itself account for it. ## Items 15/16/17 measurement: `Detect changed scope` gate jobs — 2 of 3 are pure runner overhead — 2026-09-05 From 9ee8b6058e2b0ccf06420b0f01346ada02fe6faa Mon Sep 17 00:00:00 2001 From: seonghobae Date: Sun, 6 Sep 2026 15:48:56 +0000 Subject: [PATCH 09/12] docs(gap-baseline): reproduce residual (iv) on a second head and name the shared route MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `#1187` `541cadd1` `noema-review` (run 34036172068, job 101502686002, failed 15:38:45Z) returned the same four fields as the `#1884` sample 65 seconds earlier: HTTP Error 502: Bad Gateway; caller attempts=1, duration=1215.2s, phase=response_error, served_model=deepseek-ai/deepseek-v4-flash-0731 Two pull requests, two heads, 1424.1 s and 1215.2 s, both with a ready route, the same model served, and a classified 502 rather than a timeout. Residual (iv) is a class, not an incident. The shared detail is the model. `deepseek-ai/deepseek-v4-flash-0731` is the same first-ranked route `contextual-orchestrator#1082`'s evidence names as the candidate that stalls and is re-selected — 44 of the 48 timeouts in its `#1930` sample. So (ii) and (iv) may be one unhealthy upstream route observed through two request shapes: on the tool-bearing passthrough walk it expires a socket at 90 s and leaks a raw 500; on the orchestrated walk it is served, held for twenty minutes or more, and classified. Recorded as a hypothesis these logs support but do not establish. Confirming it needs the gateway's internal attempt records from the `noema-sidecar-evidence` artifacts (9992218398, 9992230612), which are not read here. Verification: full suite passed; `git diff --check` clean. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01KPmJErfkcHer4UVEgrQxUX --- CHANGELOG.md | 1 + docs/product-technical-gap-baseline.md | 14 ++++++++++++++ 2 files changed, 15 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 0887402814..0bbe2d8208 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -98,6 +98,7 @@ None`), so the inference-path half of that open question already has a claimed fix upstream and only the preflight-probe deadline stays open here. A 2026-09-06 15:37Z measurement on this PR's own head adds a fourth residual shape none of the first three describes: `noema-review` on `9c010fcb` (run 34035522521) returned `HTTP Error 502: Bad Gateway; caller attempts=1, duration=1424.1s, phase=response_error, served_model=deepseek-ai/deepseek-v4-flash-0731`. A route was ready and served (so not the capacity class), the caller got a classified 502 rather than an opaque `internal_error` (so not the raw-500 class), and one caller attempt ran 23.7 minutes -- roughly sixteen times the 90 s that both residual (iii) and `contextual-orchestrator#1053` treat as the operative limit, with `phase=response_error` indicating a response arrived rather than a socket expiring. Read from the job log only; the `noema-sidecar-evidence` artifact that would give the gateway's internal attempt breakdown is not read, so none is claimed. + Reproduced 65 seconds later on a second head -- `#1187` `541cadd1` (run 34036172068) returned the same four fields at `duration=1215.2s` with the same `served_model` -- which makes it a class rather than an incident. The shared detail is the model: `deepseek-ai/deepseek-v4-flash-0731` is the same first-ranked route `contextual-orchestrator#1082` names as the stalling candidate (44 of 48 timeouts in its `#1930` sample), so (ii) and (iv) may be one unhealthy upstream route seen through two request shapes rather than two problems. That is a hypothesis these logs support but do not establish. - Include merge-scheduler entrypoint, core, and regression-test changes in the existing runtime-quality workflow's trigger and suite selector. Scheduler workflow edits retain queue checks and also select the full review-repair diff --git a/docs/product-technical-gap-baseline.md b/docs/product-technical-gap-baseline.md index b35ccd52f7..191513525e 100644 --- a/docs/product-technical-gap-baseline.md +++ b/docs/product-technical-gap-baseline.md @@ -3458,6 +3458,20 @@ under `#1759`) and does not change that gap's status. the job log does establish on its own is that a served route plus a classified 502 plus a 23.7-minute wall clock is a real, current combination, and that closing (i) and (ii) will not by itself account for it. + **Reproduced on a second head 65 seconds later, which makes it a class rather than an incident.** + `#1187` `541cadd1` `noema-review` (run 34036172068, job 101502686002, failed 15:38:45Z) returned the + same four fields: `HTTP Error 502: Bad Gateway; caller attempts=1, duration=1215.2s, + phase=response_error, served_model=deepseek-ai/deepseek-v4-flash-0731`. Two different pull requests, two + different heads, durations of 1424.1 s and 1215.2 s, and in both cases a route was ready, the same model + served, and the caller received a classified 502 rather than a timeout. **The shared detail worth + pulling out is the model.** `deepseek-ai/deepseek-v4-flash-0731` is the same first-ranked route + `contextual-orchestrator#1082`'s own evidence names as the candidate that stalls and is re-selected — 44 + of the 48 timeouts in its `#1930` sample. So (ii) and (iv) may not be two independent problems so much + as one unhealthy upstream route observed through two request shapes: on the tool-bearing passthrough + walk it expires a socket at 90 s and leaks a raw 500, and on the orchestrated walk it is served, held + for twenty minutes or more, and classified. That is a hypothesis this repository's logs support but do + not establish — confirming it needs the gateway's internal attempt records, which live in the + `noema-sidecar-evidence` artifacts (9992218398 and 9992230612) that are not read here. ## Items 15/16/17 measurement: `Detect changed scope` gate jobs — 2 of 3 are pure runner overhead — 2026-09-05 From a7ce345f6f0480da12cab3239e06b7666bd422b9 Mon Sep 17 00:00:00 2001 From: seonghobae Date: Sun, 6 Sep 2026 15:56:26 +0000 Subject: [PATCH 10/12] docs(gap-baseline): retract two claims in residual (iv) after reading the evidence artifacts MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit I downloaded the `noema-sidecar-evidence` artifacts (9992218398, 9992230612) that the previous two commits explicitly declined to read, and they refute two of the three claims the entry rested on. Both retractions are recorded in the entry rather than edited away. Retraction 1 — nothing was served. `served_model` names the last route *attempted*, not one that answered. Both artifacts end with `provider_attempt_failed agent_id=nvidia_nim_deepseek_ai_deepseek_v4_flash_0731 … error_type=TimeoutError`, then `circuit_failure … failures=1.0 threshold=3`, then `request_failed status=502 code=provider_connection_error`. I read a field name as an outcome. Retraction 2 — the 90 s default IS operative here, so this is not evidence against `contextual-orchestrator#1053`. `caller attempts=1` bounds the caller; the gateway ran 24 matched internal attempts summing to ~11,500 s against a 1,424 s wall clock (8–9× concurrency, so they race), and four of the 24 sit at 89.5–92 s in both runs. I told the `#1053` lane their 90.054 s sample was contradicted; it is not, and that is corrected on `#1053` directly. What survives is the part worth acting on: the durations are bimodal. Besides those four ~90 s attempts and six failing under 10 s, 11 of 24 attempts on `#1884` and 12 of 24 on `#1187` ran past 600 s, to 1,333.7 s and 1,122.9 s, medians 478.3 s and 631.3 s. No 90 s bound explains that second population, and the consequence runs opposite to what I implied earlier: removing the implicit timeout converts the ~90 s population into unbounded waits and leaves the long one untouched, so on this evidence it should make these runs longer. Stated as a two-sample prediction, not a proven regression. Two readings that support the other lanes rather than contradict them: the breaker is told on this path (`circuit_failure` 13 and 11, `circuit_opened` twice each), matching `#1082`'s scoping of its defect to the passthrough walk; and its re-selection concentration is visible here, with `deepseek-v4-flash-0731` taking 23 of ~40 attempts across both NVIDIA keys despite the breaker opening twice. Preflight read `ready_count 6, rejected 8, deferred 2, skipped 4` in both runs, so capacity is ruled out by artifact rather than by inference. Verification: full suite passed; `git diff --check` clean. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01KPmJErfkcHer4UVEgrQxUX --- CHANGELOG.md | 3 +- docs/product-technical-gap-baseline.md | 74 ++++++++++++++------------ 2 files changed, 41 insertions(+), 36 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 0bbe2d8208..d704a7d66a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -97,8 +97,7 @@ records `contextual-orchestrator#1053` as the upstream change that removes it (`timeout: float | None = None`), so the inference-path half of that open question already has a claimed fix upstream and only the preflight-probe deadline stays open here. - A 2026-09-06 15:37Z measurement on this PR's own head adds a fourth residual shape none of the first three describes: `noema-review` on `9c010fcb` (run 34035522521) returned `HTTP Error 502: Bad Gateway; caller attempts=1, duration=1424.1s, phase=response_error, served_model=deepseek-ai/deepseek-v4-flash-0731`. A route was ready and served (so not the capacity class), the caller got a classified 502 rather than an opaque `internal_error` (so not the raw-500 class), and one caller attempt ran 23.7 minutes -- roughly sixteen times the 90 s that both residual (iii) and `contextual-orchestrator#1053` treat as the operative limit, with `phase=response_error` indicating a response arrived rather than a socket expiring. Read from the job log only; the `noema-sidecar-evidence` artifact that would give the gateway's internal attempt breakdown is not read, so none is claimed. - Reproduced 65 seconds later on a second head -- `#1187` `541cadd1` (run 34036172068) returned the same four fields at `duration=1215.2s` with the same `served_model` -- which makes it a class rather than an incident. The shared detail is the model: `deepseek-ai/deepseek-v4-flash-0731` is the same first-ranked route `contextual-orchestrator#1082` names as the stalling candidate (44 of 48 timeouts in its `#1930` sample), so (ii) and (iv) may be one unhealthy upstream route seen through two request shapes rather than two problems. That is a hypothesis these logs support but do not establish. + A 2026-09-06 15:37Z `noema-review` failure on this PR's own head, repeated on `#1187` 65 seconds later, first looked like a fourth residual shape and on reading the `noema-sidecar-evidence` artifacts turned out not to be one. Two claims are retracted in the entry rather than edited away: nothing was served (`served_model` names the last route *attempted*, and both runs end `provider_attempt_failed ... TimeoutError` -> `circuit_failure` -> `request_failed status=502`), and the 90 s `ModelClient` default is operative here (the gateway ran 24 internal attempts summing to ~11,500 s against a 1,424 s wall clock, about 8-9x concurrency, four of them at 89.5-92 s), so this is not evidence against `contextual-orchestrator#1053` and the `#1053` lane was told so directly. What survives is that the durations are bimodal: 11 of 24 attempts on `#1884` and 12 of 24 on `#1187` exceeded 600 s, to 1,333.7 s, with medians of 478.3 s and 631.3 s -- a population no 90 s bound explains. The consequence stated in the entry is that removing the implicit timeout should make these runs longer rather than shorter unless the long population is addressed too, offered as a two-sample prediction rather than a proven regression. Preflight read `ready_count 6` in both runs, so capacity is ruled out by artifact rather than inference, and `circuit_failure`/`circuit_opened` fire on this path, consistent with `#1082` scoping its defect to the passthrough walk. - Include merge-scheduler entrypoint, core, and regression-test changes in the existing runtime-quality workflow's trigger and suite selector. Scheduler workflow edits retain queue checks and also select the full review-repair diff --git a/docs/product-technical-gap-baseline.md b/docs/product-technical-gap-baseline.md index 191513525e..1d6c269952 100644 --- a/docs/product-technical-gap-baseline.md +++ b/docs/product-technical-gap-baseline.md @@ -3438,40 +3438,46 @@ under `#1759`) and does not change that gap's status. must advance past whichever lands last before either reaches a review run. `#1053` was pushed to `661ce8db` at 12:48Z with all 16 checks re-queued and `#1082` sits at `812bf11f` since 03:09Z, both under other lanes' active work; this entry records the dependency, not a claim on either. - (iv) **A fourth shape, measured on this PR's own head at 15:37Z, that none of (i)-(iii) describes.** - `noema-review` on `#1884` `9c010fcb` (run 34035522521, job 101501520756) failed with `Noema gateway - transport failed: HTTPError: HTTP Error 502: Bad Gateway; caller attempts=1, duration=1424.1s, - phase=response_error, served_model=deepseek-ai/deepseek-v4-flash-0731`. Three things separate it from - every sample above. It is **not** the capacity class (i): a route was ready and `deepseek-v4-flash-0731` - actually served, so preflight succeeded. It is **not** the raw-500 class (ii): the caller received a - classified 502 through the path `#1082` is adding, not an opaque `internal_error`. And the single caller - attempt ran **1424.1 s, about 23.7 minutes** — roughly sixteen times the 90 s that (iii) and - `contextual-orchestrator#1053` both treat as the operative limit, so the `ModelClient` default was not - what bounded this request. `phase=response_error` says a response arrived and carried an error status - rather than a socket expiring, which is a different event from the `TimeoutError` in `#1053`'s own - 90.054 s noema measurement. The reading this supports is that `caller attempts=1` bounds the *caller* - only — the gateway owns repair and failover, and it spent those 23.7 minutes walking its pool internally - before returning the classified 502 that its warning line calls out (`gateway owns repair/failover`). - **Honest limit:** this is read from the job log. The `noema-sidecar-evidence` artifact (9992218398, 2366 - bytes) holds the sidecar stderr and the preflight JSON that would give the internal attempt count and - how the 23.7 minutes was distributed; it is not read here, so no per-attempt breakdown is claimed. What - the job log does establish on its own is that a served route plus a classified 502 plus a - 23.7-minute wall clock is a real, current combination, and that closing (i) and (ii) will not by - itself account for it. - **Reproduced on a second head 65 seconds later, which makes it a class rather than an incident.** - `#1187` `541cadd1` `noema-review` (run 34036172068, job 101502686002, failed 15:38:45Z) returned the - same four fields: `HTTP Error 502: Bad Gateway; caller attempts=1, duration=1215.2s, - phase=response_error, served_model=deepseek-ai/deepseek-v4-flash-0731`. Two different pull requests, two - different heads, durations of 1424.1 s and 1215.2 s, and in both cases a route was ready, the same model - served, and the caller received a classified 502 rather than a timeout. **The shared detail worth - pulling out is the model.** `deepseek-ai/deepseek-v4-flash-0731` is the same first-ranked route - `contextual-orchestrator#1082`'s own evidence names as the candidate that stalls and is re-selected — 44 - of the 48 timeouts in its `#1930` sample. So (ii) and (iv) may not be two independent problems so much - as one unhealthy upstream route observed through two request shapes: on the tool-bearing passthrough - walk it expires a socket at 90 s and leaks a raw 500, and on the orchestrated walk it is served, held - for twenty minutes or more, and classified. That is a hypothesis this repository's logs support but do - not establish — confirming it needs the gateway's internal attempt records, which live in the - `noema-sidecar-evidence` artifacts (9992218398 and 9992230612) that are not read here. + (iv) **A `noema-review` failure shape seen on two heads — and, on reading the evidence artifacts, *not* + the separate root cause an earlier revision of this entry claimed.** Two runs 65 seconds apart reported + the same caller-side fields: `#1884` `9c010fcb` (run 34035522521) `HTTP Error 502: Bad Gateway; caller + attempts=1, duration=1424.1s, phase=response_error, served_model=deepseek-ai/deepseek-v4-flash-0731`, + and `#1187` `541cadd1` (run 34036172068) the same at `duration=1215.2s`. From the caller line alone this + looked like a fourth failure mode: a served route, a classified 502, and a wall clock far past 90 s. + **Reading the `noema-sidecar-evidence` artifacts (9992218398, 9992230612) refuted two of the three + claims that reading rested on, and both retractions are recorded here rather than quietly edited away.** + + **Retraction 1 — nothing was served.** `served_model` names the *last route attempted*, not one that + answered. The final three events in both artifacts are `provider_attempt_failed + agent_id=nvidia_nim_deepseek_ai_deepseek_v4_flash_0731 … error_type=TimeoutError transient=True`, then + `circuit_failure … failures=1.0 threshold=3`, then `request_failed status=502 + code=provider_connection_error`. The earlier text read the field name as an outcome. + + **Retraction 2 — the 90 s default is operative on this path, so this is not evidence against + `contextual-orchestrator#1053`.** `caller attempts=1` bounds the caller; the gateway ran **24 matched + internal attempts** whose durations sum to about 11,500 s against a 1,424 s wall clock — roughly **8–9× + concurrency**, so the attempts race rather than run in series. Four of the 24 sit at 89.5–92 s in both + runs, which is exactly the `ModelClient` recv default. An earlier revision told the `#1053` lane their + 90.054 s sample was contradicted; it is not, and that was corrected on `#1053` directly. + + **What survives, and it is the part worth acting on: the attempt durations are bimodal.** Alongside + those four ~90 s attempts and six that fail in under 10 s, **11 of 24 attempts on `#1884` and 12 of 24 + on `#1187` ran longer than 600 s**, to a maximum of 1,333.7 s and 1,122.9 s; the medians are 478.3 s and + 631.3 s. No 90 s bound explains that second population. The consequence for the merge order is the + opposite of what the earlier revision implied: **removing the implicit timeout (`#1053`) converts the + ~90 s population into unbounded waits and leaves the >600 s population untouched, so on this evidence it + should make these runs longer, not shorter, unless the long population is addressed too.** That is a + prediction from two samples, not a proven regression, and it is the reason it is stated rather than + assumed. + + **Two further readings, both consistent with the other lanes' work rather than against it.** The breaker + *is* told on the orchestrated path — `circuit_failure` 13 and 11 times, `circuit_opened` twice in each + run — which matches `#1082`'s own scoping of its defect to the tool-bearing passthrough walk and not to + this one. And the re-selection concentration `#1082` describes is visible here too: + `deepseek-v4-flash-0731` takes 16 attempts on the primary NVIDIA key plus 7 on the `_SUB` key, 23 of + roughly 40, despite the breaker opening twice. Preflight in both runs reported `ready_count 6, rejected + 8, deferred 2, skipped 4` out of 24 candidates, so capacity (i) is ruled out by the artifact rather than + by inference. ## Items 15/16/17 measurement: `Detect changed scope` gate jobs — 2 of 3 are pure runner overhead — 2026-09-05 From 396b4dee276937d2ce768a21d91fa287cbe22bf3 Mon Sep 17 00:00:00 2001 From: seonghobae Date: Sun, 6 Sep 2026 16:07:55 +0000 Subject: [PATCH 11/12] docs(gap-baseline): record a capacity-class run without folding it into the bimodal claim `#1967` `533b86b8` `noema-review` (run 34039136693, job 101508436453, artifact 9992585682) failed 25 minutes after the two runs behind residual (iv), and on reading its artifact it is the capacity class, not that one. Its profile is the inverse: preflight ready_count 1 (not 6), deferred 8, skipped 8 46 HTTPError vs 5 TimeoutError (not 15-17 timeouts) 45 of 51 attempts under 10s, median 0.1s (not 478-631s) circuit_opened 8, span 534.7s (not 20-24 minutes) terminal: request_failed status=429 code=rate_limit_exceeded (not 502) It would have been easy to count this as a third sample of the long-attempt shape, since the failing check and the PR family are the same. It is not one, and the bimodal duration finding stays an explicitly two-sample claim. Two details bear on `#1948`/`#1949`: `postponed_probed_count` is 10, the first boot observed here where `#1949`'s postponement rule actually spent a second pass, and readiness still finished at 1 of a 24-candidate catalog. The rule executed as designed and did not by itself produce a servable pool. `escalations_used` is 0 against 2 in the other two runs, so the priced escalation path is not what differed either. Verification: full suite 2956 passed, 1 skipped, 21 subtests passed; `git diff --check` clean. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01KPmJErfkcHer4UVEgrQxUX --- CHANGELOG.md | 1 + docs/product-technical-gap-baseline.md | 16 ++++++++++++++++ 2 files changed, 17 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index d704a7d66a..11f7f656cc 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -98,6 +98,7 @@ None`), so the inference-path half of that open question already has a claimed fix upstream and only the preflight-probe deadline stays open here. A 2026-09-06 15:37Z `noema-review` failure on this PR's own head, repeated on `#1187` 65 seconds later, first looked like a fourth residual shape and on reading the `noema-sidecar-evidence` artifacts turned out not to be one. Two claims are retracted in the entry rather than edited away: nothing was served (`served_model` names the last route *attempted*, and both runs end `provider_attempt_failed ... TimeoutError` -> `circuit_failure` -> `request_failed status=502`), and the 90 s `ModelClient` default is operative here (the gateway ran 24 internal attempts summing to ~11,500 s against a 1,424 s wall clock, about 8-9x concurrency, four of them at 89.5-92 s), so this is not evidence against `contextual-orchestrator#1053` and the `#1053` lane was told so directly. What survives is that the durations are bimodal: 11 of 24 attempts on `#1884` and 12 of 24 on `#1187` exceeded 600 s, to 1,333.7 s, with medians of 478.3 s and 631.3 s -- a population no 90 s bound explains. The consequence stated in the entry is that removing the implicit timeout should make these runs longer rather than shorter unless the long population is addressed too, offered as a two-sample prediction rather than a proven regression. Preflight read `ready_count 6` in both runs, so capacity is ruled out by artifact rather than inference, and `circuit_failure`/`circuit_opened` fire on this path, consistent with `#1082` scoping its defect to the passthrough walk. + A third run 25 minutes later (`#1967` `533b86b8`, run 34039136693, artifact 9992585682) is the capacity class rather than this one and is recorded so the two are not merged: it ends `429 rate_limit_exceeded` with preflight `ready_count 1`, 46 `HTTPError` against 5 `TimeoutError`, and 45 of 51 attempts under 10 s at a 0.1 s median. The bimodal finding stays a two-sample claim. That run is also the first observed boot where `#1949`'s postponement rule spent a second pass (`postponed_probed_count` 10) and readiness still finished at 1 of 24, with `escalations_used` 0. - Include merge-scheduler entrypoint, core, and regression-test changes in the existing runtime-quality workflow's trigger and suite selector. Scheduler workflow edits retain queue checks and also select the full review-repair diff --git a/docs/product-technical-gap-baseline.md b/docs/product-technical-gap-baseline.md index 1d6c269952..8e1caf0f5b 100644 --- a/docs/product-technical-gap-baseline.md +++ b/docs/product-technical-gap-baseline.md @@ -3479,6 +3479,22 @@ under `#1759`) and does not change that gap's status. 8, deferred 2, skipped 4` out of 24 candidates, so capacity (i) is ruled out by the artifact rather than by inference. + **A third run 25 minutes later is the capacity class, not this one, and is recorded here precisely so + the two are not merged.** `#1967` `533b86b8` `noema-review` (run 34039136693, job 101508436453, artifact + 9992585682) ended `request_failed status=429 code=rate_limit_exceeded`, and its profile is the inverse + of the two above: preflight `ready_count 1` rather than 6, 51 failures of which **46 are `HTTPError` + against 5 `TimeoutError`** rather than 15-17 timeouts, **45 of 51 attempts finishing under 10 seconds** + at a median of 0.1 s rather than 478-631 s, `circuit_opened` 8 times, and a 534.7 s span rather than + 20-24 minutes. The bimodal duration finding above therefore stays a **two-sample** claim; this run does + not extend it and would misrepresent it if counted. What the classes share is only that both end without + a verdict. + + Two details in that third run bear on `#1948`/`#1949`. Its `postponed_probed_count` is **10** -- the + first boot observed here where `#1949`'s postponement rule actually spent a second pass -- and readiness + still finished at 1 of a 24-candidate catalog, with `deferred_count 8` and `skipped_count 8`. The rule + executed as designed and did not by itself produce a servable pool. Its `escalations_used` is 0 against + 2 in the other two runs, so the priced escalation path is not what differed either. + ## Items 15/16/17 measurement: `Detect changed scope` gate jobs — 2 of 3 are pure runner overhead — 2026-09-05 **Status:** Measured, not yet fixed. Recorded so the fix is grounded in real numbers rather than the intuition From 767e740ce6bfd07d1dbe2dc79c8d7f50819eeb6c Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 7 Sep 2026 04:23:40 +0000 Subject: [PATCH 12/12] docs(gap-baseline): record the dispatch-actor outage and its fix, the free-pool single-family collapse, and three retracted claims Adds a 2026-09-07 entry covering three findings from this cycle. The central review pipeline was dark on an identity allowlist rather than on model capability: required workflows run under pull_request_target with a read-only token, mint an OpenCode App token, and dispatch as opencode-agent[bot], while the receiving gate admitted only github-actions[bot]. Every App-token dispatch was rejected before any review job ran, so the required job's promise to rerun after an exact-head verdict could never be kept. The allowlist value was identical in the one authorized run since 2026-08-31 and in the rejected ones, isolating the variable to the sender. Resolved by owner action between 00:07Z and 00:22:49Z. The orchestrator/free pool resolved to a single provider family in three independent runs, which makes the standing NIM-removal target cost-bearing today. Records that free_account_diversity already exists but is catalog-scoped, absent from the uploaded preflight report, and on the account axis rather than the family axis. Records three of this repository's own claims being retracted under an independent count of the same artifact, and names per-request correlation ids as the prerequisite for any further diagnosis. Refs #1915, #1925, #1927, #1929, #1756, #2000. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01KPmJErfkcHer4UVEgrQxUX --- CHANGELOG.md | 3 + docs/product-technical-gap-baseline.md | 108 +++++++++++++++++++++++++ 2 files changed, 111 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 11f7f656cc..492af38f41 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -99,6 +99,9 @@ preflight-probe deadline stays open here. A 2026-09-06 15:37Z `noema-review` failure on this PR's own head, repeated on `#1187` 65 seconds later, first looked like a fourth residual shape and on reading the `noema-sidecar-evidence` artifacts turned out not to be one. Two claims are retracted in the entry rather than edited away: nothing was served (`served_model` names the last route *attempted*, and both runs end `provider_attempt_failed ... TimeoutError` -> `circuit_failure` -> `request_failed status=502`), and the 90 s `ModelClient` default is operative here (the gateway ran 24 internal attempts summing to ~11,500 s against a 1,424 s wall clock, about 8-9x concurrency, four of them at 89.5-92 s), so this is not evidence against `contextual-orchestrator#1053` and the `#1053` lane was told so directly. What survives is that the durations are bimodal: 11 of 24 attempts on `#1884` and 12 of 24 on `#1187` exceeded 600 s, to 1,333.7 s, with medians of 478.3 s and 631.3 s -- a population no 90 s bound explains. The consequence stated in the entry is that removing the implicit timeout should make these runs longer rather than shorter unless the long population is addressed too, offered as a two-sample prediction rather than a proven regression. Preflight read `ready_count 6` in both runs, so capacity is ruled out by artifact rather than inference, and `circuit_failure`/`circuit_opened` fire on this path, consistent with `#1082` scoping its defect to the passthrough walk. A third run 25 minutes later (`#1967` `533b86b8`, run 34039136693, artifact 9992585682) is the capacity class rather than this one and is recorded so the two are not merged: it ends `429 rate_limit_exceeded` with preflight `ready_count 1`, 46 `HTTPError` against 5 `TimeoutError`, and 45 of 51 attempts under 10 s at a 0.1 s median. The bimodal finding stays a two-sample claim. That run is also the first observed boot where `#1949`'s postponement rule spent a second pass (`postponed_probed_count` 10) and readiness still finished at 1 of 24, with `escalations_used` 0. + A 2026-09-07 entry records that the whole central review pipeline was dark on an identity allowlist rather than on model capability, and that it has since been fixed. Required workflows run under `pull_request_target` with a read-only token, so they mint an OpenCode App token and send `repos/.../dispatches` as `opencode-agent[bot]`, while the receiving gate admitted only `github-actions[bot]` via `vars.OPENCODE_REPOSITORY_DISPATCH_ACTOR` -- every App-token dispatch was rejected at `validate-pr-metadata`/`validate-dispatch` with all review jobs `skipped`, so the required job's promise that "the dispatch workflow will rerun this failed job" could never be kept. The variable was identical in the one authorized run since 2026-08-31 (`33998627665`, actor `github-actions[bot]`, the merge-scheduler path) and in the rejected ones, so the defect was two senders against a one-entry allowlist, not a drifted value; `codeql-scan-dispatch.yml`, fed only by the App-token sender, had 0 successes across 2352 runs. Fixed by owner action between 00:07Z and 00:22:49Z on 2026-09-07 (that workflow now shows 5 successes, all `opencode-agent[bot]`). Tracked on `#1927`/`#1929`, with `#1925` a separate defect on the same workflow; no PR diff could carry the fix because an Actions variable is not repository content. + The same entry records the free pool resolving to a single provider family in three independent runs on 2026-09-06 (`ready_count` 6/5/6 of 24 against `target_ready` 8, every ready route `nvidia_nim` or `nvidia_nim_sub`, Bytez failing at discovery with `http_status_500` and both OpenRouter free routes deferred on 429), which makes the standing NIM-removal target cost-bearing today: removing direct NIM communication would leave the free pool empty. It also notes that `free_account_diversity` already exists in `contextual_orchestrator_review_launcher.py::_with_discovery_counts` but is catalog-scoped, absent from the uploaded `strix-plain-chat-preflight-v2` report, and account-axis (`provider_account` is the identity function, correctly so for rate-limit isolation) -- so it reads 2 where outage diversity is 1. Tracked on `#1915`. + Finally, the entry records three of this repository's own claims being retracted under independent count. `#2000` reported a 6 h 00 m 17 s Strix run ending `cancelled` with no verdict; a cross-family session matched the artifact's audit boundary exactly and showed the main route had 837 starts but 564 recorded failures, leaving 273 unclassifiable because the log emits no success-shaped event. "837 consecutive failures", "the caller re-selecting a dead route", and "killed by the 6-hour ceiling" were all withdrawn; with zero `request_id` fields in the evidence, caller retry and independent concurrent requests are indistinguishable, and `provider_exhausted: 0` is an absence rather than a demonstration. What survives is the start distribution (92% on one route), the aggregate counts, and a required check that ended with no verdict -- the state `#1756` sweeps. Emitting a per-request correlation id is named as the prerequisite for any further diagnosis. Refs #1915, #1927, #1929, #2000, #1756. - Include merge-scheduler entrypoint, core, and regression-test changes in the existing runtime-quality workflow's trigger and suite selector. Scheduler workflow edits retain queue checks and also select the full review-repair diff --git a/docs/product-technical-gap-baseline.md b/docs/product-technical-gap-baseline.md index 8e1caf0f5b..d4b27cdc07 100644 --- a/docs/product-technical-gap-baseline.md +++ b/docs/product-technical-gap-baseline.md @@ -3613,3 +3613,111 @@ queries the check-runs API at its own time, order-independently. The implementin their change was safe because they had scoped it narrowly, not because they had checked for the name collision — which is the more useful lesson: **a job name is unique only within one workflow file, and the same name in another file can carry the opposite safety property.** + +## Central review pipeline: one identity mismatch dark org-wide, then fixed — and one of this record's own +claims retracted under independent count — 2026-09-07 + +**The whole central review pipeline was dark on an identity allowlist, not on model capability.** +Required review workflows run under `pull_request_target` with a read-only token, so +`opencode-review.yml` and `codeql-pr.yml` mint a repository-scoped OpenCode App token and send +`repos/.../dispatches` with it. GitHub attributes that dispatch to the App, `opencode-agent[bot]`. The +receiving gate in `opencode-review-dispatch.yml` and `codeql-scan-dispatch.yml` admits only identities +listed in `vars.OPENCODE_REPOSITORY_DISPATCH_ACTOR`, whose live value was `github-actions[bot]` alone. +Every App-token dispatch was therefore rejected at `validate-pr-metadata` / `validate-dispatch` before +any review job ran, with `coverage-source-tree`, `coverage-evidence` and `opencode-review` all +`skipped`. + +**The required job's own error text promises the sequence that authorization then prevented** — *"The +dispatch workflow will rerun this failed job after publishing an authenticated exact-head verdict."* No +rerun could occur, so the required check was a permanent fail-closed rather than the designed wait-state +it reads as. The same shape held on the CodeQL side, where shards sat at `DISPATCH_OUTCOME: success` / +`VERDICT_STATE: pending` for 13 hours. + +**What isolated the variable was the single counterexample.** Between 2026-08-31 and 2026-09-07T00:22Z +exactly one `opencode-review-dispatch.yml` run concluded `success`: run `33998627665` +(`.github#1946@db361daa`), whose `triggering_actor` was `github-actions[bot]` — the merge-scheduler +path, which dispatches from a run whose token may be write-scoped. `ALLOWED_DISPATCH_ACTOR` was +**identical** in that authorized run and in the rejected ones, so the defect was never a drifted +variable; it was two senders against a one-entry allowlist. `codeql-scan-dispatch.yml`, fed only by the +App-token sender, had **0 successes across 2352 runs** — it had never once worked. + +**Resolved by owner action between 2026-09-07T00:07Z and 00:22:49Z.** `codeql-scan-dispatch.yml` went +from 0 successes to 5, all with `triggering_actor = opencode-agent[bot]`, the first being run +`34069630406` (`.github#1653`). The remedy was the allowlist rather than the sender because making the +required job dispatch as `github-actions[bot]` would require `contents: write` on a +`pull_request_target` job, which AGENTS.md and CLAUDE.md forbid; the receiving gate already parsed a +comma-separated list, so the App identity was the value the code was written to accept. Tracked on +`#1927` / `#1929`, with `#1925` recording an independent second defect on the same workflow. No +pull-request diff could carry this fix: it is an Actions variable, and an agent session cannot read or +write one (`GET /repos/.../actions/variables` returns `403` through the agent proxy). **The +generalizable lesson: a required check that fails closed waiting for an out-of-band verdict is +indistinguishable, from the PR page, between "waiting" and "can never arrive". Only the dispatch run's +own authorization step separates them.** + +### The free pool has no provider-family diversity, reproduced three times in one day + +Three independent runs on 2026-09-06 show the `orchestrator/free` catalog resolving to a single provider +family, against a `target_ready` of 8: + +| run | evidence artifact | `candidate_count` | `ready_count` | ready families | +|---|---|---|---|---| +| `noema-review` on `#1884@396b4dee` | `noema-sidecar-evidence` 9994541963 | 24 | 6 | `nvidia_nim`, `nvidia_nim_sub` | +| `strix` on `#1187@541cadd1` | `strix-reports` 9997372950 | 24 | 5 | `nvidia_nim`, `nvidia_nim_sub` | +| `noema-review` on `#1967@533b86b8` | `noema-sidecar-evidence` 9993983422 | 24 | 6 | `nvidia_nim`, `nvidia_nim_sub` | + +In each, the gateway itself was healthy (`"gateway": {"status": "ready", "finish_reason": "stop"}`) — +this is pool composition, not reachability. The two sources that could have supplied diversity failed in +different ways every time: Bytez never entered the pool at all (`provider_discovery_failed +provider=bytez code=http_status_500`, the first line of the sidecar log, so discovery rather than +inference), and both OpenRouter free candidates deferred on `HTTPError` / `429`. The eight rejects are +NVIDIA `404`s on retired hosted models plus timeouts. Because the two NVIDIA credentials are one family, +**removing direct NIM communication today would leave the free pool empty** — which is the concrete cost +of the standing removal target, recorded here as measurement rather than as an argument against it. +Tracked on `#1915`, the free-pool availability acceptance tracker. + +**A diversity metric already exists but cannot see this.** +`scripts/ci/contextual_orchestrator_review_launcher.py::_with_discovery_counts` emits +`free_account_diversity` / `free_pool_account_diversity`, computed over the whole discovery catalog +(`cost_evidence == "free"`) and absent from the `strix-plain-chat-preflight-v2` report that is actually +uploaded when a gate goes dark. Its axis is accounts, not families: +`contextual_orchestrator_review_policy.py::provider_account` is the identity function, which is correct +for its own purpose — the two NVIDIA keys were confirmed on 2026-09-03 to have independent rate limits, +and PR-scoped Strix concurrency was restored on that finding. But on these runs an account count over +the `ready` set reads 2 while outage diversity is 1: both accounts timed out on the same upstream, on +the same models, minutes apart. **Independent rate limits do not buy independent availability, and one +axis must not stand in for the other.** + +### A record of this document's own overclaim, retracted under independent count + +A Strix run on `#1187@541cadd1` ran `15:19:48Z → 21:20:05Z` — 6 h 00 m 17 s — and ended `cancelled`, so +the required check carried no findings and no gate decision. Its sidecar log shows 909 +`provider_attempt` starts, 603 recorded failures (541 `TimeoutError`, 62 `HTTPError`), 536 +`request_failed status=500 code=internal_error`, `provider_exhausted` 0, and 837 of the 909 starts (92%) +on one `agent_id`. This was filed as `#2000`. + +**Three of that issue's claims did not survive review and were retracted.** A cross-family session +re-downloaded the same artifact, matched the audit boundary exactly (stderr SHA256 +`ce5ed2846ecf284674cf5c6d0d80e674e79bfc4e58c27f40c878fde3bc25d70a`), agreed every aggregate, and showed +the main route had **837 starts but 564 recorded failures**, leaving 273 starts with no recorded +outcome. Re-derived independently on this side and confirmed. So "837 consecutive failures" was a number +never counted; the log emits no success-shaped event at all, so those 273 cannot be classified in either +direction. Likewise `attempt=1/1` bounds only the per-call retry budget, and with **zero `request_id` +fields** in the evidence, caller retry, independent concurrent requests, and in-request fallback are +indistinguishable — so "the caller re-selecting a dead route" and "has no exhaustion condition" were +both unproven, `provider_exhausted: 0` being an absence rather than a demonstration. And 6 h 00 m 17 s +is *consistent with* the platform job ceiling without being direct evidence of who cancelled the run. + +**What survives is narrower and still worth acting on**: the attempt-start distribution (92% on one +route, a fact about starts that needs no outcome classification), the aggregate counts, and the outcome +— a required check that ended with no verdict, the state `#1756` is already sweeping. The blocking gap +for any further diagnosis is the one the count exposed: **until a correlation id is emitted per logical +request and carried onto every `provider_attempt` line, every future run reproduces the same ambiguity +and the same argument.** Evidence instrumentation precedes the exhaustion-contract work, not the other +way round. + +**Two method lessons, both earned the hard way in this cycle.** First, failing checks that look alike +are not one class: on a single PR, `opencode-review` was blocked by the dispatch allowlist while +`noema-review` never dispatches at all and was blocked by pool composition — a single claim covering +both was posted and had to be corrected on the PR. Second, before proposing that something be built, +read whether it already exists: a proposal to "add a provider-diversity metric" had to be narrowed to +"give the existing metric a family axis and a `ready` scope" once the source was actually read.