From 9e1fe96521c1de740a12e3ec3990b9e84f7c8e57 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 3 Sep 2026 23:02:59 +0900 Subject: [PATCH 1/7] docs(gap-baseline): root-cause item 4's underlying provider-pool stall Four same-day cross-repo data points (fast-mlsirm#1518, .github#1689, naruon#1539, mightyETL#330) show noema-review's gateway stall growing 649.5s -> 1332.6s -> 1462.9s -> 2161.9s. Traced through contextual-orchestrator's TaskOrchestrator._invoke: the free-tier candidate pool failover loop has no fast-fail path for a candidate recently known to be failing, so pool saturation compounds as serial per-candidate timeout budget rather than a bounded worst case -- and each stalled run holds an Actions job slot for 20-36 minutes, worsening the same 60-job ceiling this doc's queue-congestion entries already track. Proposes a direction (skip via #911's routing observations, or race top candidates) without implementing it -- that's shared production routing code in contextual-orchestrator with a 100% coverage bar and deserves its own PR and owner sign-off. --- docs/product-technical-gap-baseline.md | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/docs/product-technical-gap-baseline.md b/docs/product-technical-gap-baseline.md index 5868e7aad9..8baed54cc4 100644 --- a/docs/product-technical-gap-baseline.md +++ b/docs/product-technical-gap-baseline.md @@ -2714,6 +2714,24 @@ prose" convention already stated in `CLAUDE.md`. **Formerly open, gateway-owned — now fixed, PR open.** The missing model/provider attribution on the real-call failure path (`served_model=unknown` where preflight proves the sidecar can report this detail) is root-caused and fixed: `ContextualWisdomLab/contextual-orchestrator#1037` (branch `fix/invoke-failover-attempt-telemetry`, based on `main` @ `f4e5fc67`, open, not yet merged). Root cause: `TaskOrchestrator._invoke`'s failover loop (`contextual_orchestrator/orchestrator.py:7660-7893`) tracked only the single most recent candidate's failure (`last_upstream_error`/`last_provider_response_error`, overwritten on every new candidate), discarding every earlier candidate's `agent_id`/`model`/`provider_name`/failure reason the moment the loop moved on — so a fully-exhausted pool's raised exception could only ever describe the last agent tried, exactly matching the `served_model=unknown` symptom above. Fix: `ProviderUpstreamError.detail` now conditionally surfaces `attempts` (one record per candidate: `agent_id`/`model`/`provider`/`error_code`/`provider_status`/`retryable`/`retry_attempt`, reusing the existing `_record_tool_fallback` shape — never raw exception text) and `stop_reason`, populated at all 3 of `_invoke`'s existing "candidate exhausted" exit points; `server.py`'s error-message helper surfaces the count/reason; a second, compounding bug (the 413 `request_too_large` handler silently dropping `exc.detail` via a missing 4th `_send_error` argument) was fixed alongside it since it shares the same attribution-loss shape. RED-then-GREEN on 3 new tests, regression guards (`test_detail_and_transport_are_preserved_for_callers`, `test_invoke_preserves_final_classified_failure_across_candidates`, `test_all_agents_failing_raises_after_trying_every_candidate`) confirmed unmodified, full suite green. Zero line-range overlap with the concurrently-active PR #1032 (confirmed via diff comparison — #1032 touches `_orchestrated_provider_completion`'s schema-repair accounting; this touches `_invoke`'s failover loop, a different code path), branched from `main` directly rather than stacked. `.github`-side follow-up still needed once both #1661 and #1037 land: `scripts/ci/noema_review_gate.py`'s `call_llm` catches `urllib.error.HTTPError` without calling `exc.read()`, so it cannot see the response body CO now sends on failure, and `_extract_served_model` only reads a top-level `data.get("model")` while CO nests everything under `error.detail`/`error_detail` — the caller needs its own small patch to actually surface what the gateway now provides. +## Item 4: the underlying provider-pool stall itself, root-caused — 2026-09-03 + +**Status:** Open. The prior entry above explicitly left this unresolved ("This does not fix the underlying 649-second provider stall itself — that remains a real, separate, unresolved question"). This entry answers that question with direct code evidence plus four same-day, cross-repository data points showing the stall duration growing over the course of the day, and proposes a design direction — not yet implemented, and deliberately not implemented in this pass given the change touches shared production routing code in `contextual-orchestrator` that carries a 100% coverage bar and no maintainer/owner discussion of the tradeoff has happened yet. + +**Fresh evidence, four same-day data points, growing.** All four are `noema-review`'s "Prepare Noema model verdict" step failing with `HTTPError: HTTP Error 502: Bad Gateway`, `phase=connecting` (per the entry above, this label covers the whole wait, not literal TCP connect), `served_model=unknown`, `caller attempts=1`: +- `ContextualWisdomLab/.github#1689`: `duration=1462.9s` (24m23s) — run [`33701909483`](https://github.com/ContextualWisdomLab/.github/actions/runs/33701909483/job/100482801962). +- `ContextualWisdomLab/naruon#1539`: `duration=1332.6s` (22m13s) — run [`33702091175`](https://github.com/ContextualWisdomLab/naruon/actions/runs/33702091175/job/100483351489) (found and relayed by a peer session). +- `ContextualWisdomLab/mightyETL#330`: `duration=2161.9s` (36m2s) — run [`33701944041`](https://github.com/ContextualWisdomLab/mightyETL/actions/runs/33701944041/job/100482910533) (found and relayed by the same peer session; that PR's `strix` check also failed separately, log expired `BlobNotFound`, cause not independently confirmed). +- The prior entry's own `fast-mlsirm#1518` data point: `duration=649.5s` (10m49s), earlier the same day. + +Sequence across the day: 649.5s → 1332.6s → 1462.9s → 2161.9s. Not proof of a monotonic trend on four points from three different repositories, but consistent with a shared-capacity resource degrading as the day's cumulative concurrent CI load (8+ sessions, dozens of repositories, all routing `noema-review`/`strix`/`opencode-review` through the same `orchestrator/free` pool) increases. + +**Root cause, from direct code reading of `contextual-orchestrator@464da4715b495b5eaaa593eba3796e2d976ee0c9`'s `contextual_orchestrator/orchestrator.py`.** `TaskOrchestrator._invoke`'s failover loop (`:7762`, `for agent in candidates:`) iterates the **entire ranked free-tier candidate pool** — `free_selected_count` in `.github#1689`'s own sidecar preflight log was 12 — serially, with no overall wall-clock budget for the request as a whole. Each candidate gets an inner retry loop (`:7763`, `retry_limit = min(self.tool_retry_attempts, MAX_TOOL_RETRY_ATTEMPTS)`, `MAX_TOOL_RETRY_ATTEMPTS = 4` in `tool_fallback.py:12`) calling `ModelClient.chat`, whose defaults (`orchestrator.py:1699-1709`) are `timeout=90` seconds per HTTP attempt, `max_retries=2`, full-jitter exponential backoff (`:2091-2095`) capped at `retry_backoff_cap=8.0` seconds between attempts. Worst case for one candidate that hangs the full timeout on every attempt: `90 × 3 attempts + up to ~16s backoff ≈ 286s`. There is no per-candidate circuit breaker in this loop that skips a candidate known-bad from very recent history before spending its full timeout budget on it again — `#911`'s durable routing-observation EWMA data (`model_group.py`, merged this same day) is consumed by `_measured_member_order` for *ranking* candidate order, not for *excluding* a candidate outright, so a saturated pool still pays close to the full per-candidate cost for every member it reaches before either succeeding or exhausting the pool. Five candidates each hitting close to worst case (`5 × ~286s ≈ 1430s`) lands within seconds of the actual `.github#1689` observation (`1462.9s`); the smaller and larger observed values are consistent with fewer or more of the pool's 12 members being simultaneously slow rather than fast-failing (an immediate 429/503 response costs very little; a member that accepts the connection and then hangs costs close to the full 90s×3 budget) — exactly the shape you'd expect from a shared free-tier pool under uneven, fluctuating saturation across many concurrent callers org-wide. + +**Why this is a real gap, not a config bug.** No single number here is wrong in isolation — a 90-second per-attempt timeout, 2 retries, and a 4-attempt candidate budget are all individually reasonable, and `docs/product-goal-directive.md` §8 deliberately forbids capping total model-inference wall-clock (see the entry above and `docs/doctoring/autofix-and-noema-review-model-job-timeout-removal.md`). The gap is structural: **the failover loop has no mechanism to fail fast past a candidate that is very likely to fail again** (state `_group_router`/`#911`'s observation store already tracks, per the problem-solving history in this session), so pool saturation shows up as linearly-compounding serial wait rather than either a fast failure or a bounded worst case. This also directly costs Actions job-slot capacity: every one of these four runs held a `noema-review` job (and its GitHub Actions runner) for 20-36 minutes before failing, worsening the same 60-job organization concurrency ceiling this document's queue-congestion entries already treat as a standing constraint. + +**Proposed direction, not implemented here.** Two independent levers, either alone or combined: (1) let `_invoke`'s candidate loop consult the existing `#911` routing-observation data to *skip* (not just deprioritize) a candidate whose recent observation window shows a failure rate above a threshold, rather than re-spending its full per-attempt timeout; (2) race a small number of top-ranked candidates concurrently (the function already contains a `race_members`/`outcome.winner_endpoint_id` concurrent path a few lines above the serial loop for a different condition not fully traced in this pass — worth checking whether that path can be extended to the free-tier worker case) instead of trying them strictly one after another. Both change shared production routing code with a 100% coverage bar in `contextual-orchestrator` and deserve their own PR, tests, and — given the 2-hour-plus-per-model-call policy this document already defends elsewhere — explicit confirmation that "skip a likely-bad candidate fast" does not become "cap total inference time" through the back door. Not filed as a code change in this session; filed as a tracked gap for the session or maintainer with the most current `contextual-orchestrator` routing context to pick up. + ## Item 41: CodeQL PR `startup_failure` blocking merges org-wide — existing-repo gap closed, future-repo gap open **Problem.** Every ruleset-injected `codeql-pr.yml` run in every repository covered by org ruleset `18156473` (confirmed: bandscope, naruon, aFIPC, pg-erd-cloud, xtrmLLMBatchPython, wardnet, spanning 2026-09-02T20:12:52Z through 2026-09-03T03:15:43Z) concluded `startup_failure` with **zero check runs created** — while every other required workflow in the same PRs at the same time enqueued normally. Example: [wardnet run 33710719228](https://github.com/ContextualWisdomLab/wardnet/actions/runs/33710719228). From ae2192d80e7be5edf7683d0c26bffd0e278a969c Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 3 Sep 2026 23:26:29 +0900 Subject: [PATCH 2/7] docs(gap-baseline): add 5th item-4 stall data point, correct coverage claim contextual-orchestrator#1028's own noema-review check hit the identical gateway stall (duration=2296.6s, the largest yet) while triaging an unrelated autofix event -- notable because it's contextual-orchestrator's own CI blocked by contextual-orchestrator's own gateway saturation. Also corrects an overclaim in the first draft: the repo's real fail-under=100 coverage gate is scoped to nim_benchmark.py, not the whole package, verified against .github/workflows/ci.yml directly. --- docs/product-technical-gap-baseline.md | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/docs/product-technical-gap-baseline.md b/docs/product-technical-gap-baseline.md index 8baed54cc4..e3f9394304 100644 --- a/docs/product-technical-gap-baseline.md +++ b/docs/product-technical-gap-baseline.md @@ -2716,21 +2716,22 @@ prose" convention already stated in `CLAUDE.md`. ## Item 4: the underlying provider-pool stall itself, root-caused — 2026-09-03 -**Status:** Open. The prior entry above explicitly left this unresolved ("This does not fix the underlying 649-second provider stall itself — that remains a real, separate, unresolved question"). This entry answers that question with direct code evidence plus four same-day, cross-repository data points showing the stall duration growing over the course of the day, and proposes a design direction — not yet implemented, and deliberately not implemented in this pass given the change touches shared production routing code in `contextual-orchestrator` that carries a 100% coverage bar and no maintainer/owner discussion of the tradeoff has happened yet. +**Status:** Open. The prior entry above explicitly left this unresolved ("This does not fix the underlying 649-second provider stall itself — that remains a real, separate, unresolved question"). This entry answers that question with direct code evidence plus five same-day, cross-repository data points showing the stall duration growing over the course of the day, and proposes a design direction — not yet implemented, and deliberately not implemented in this pass given the change touches shared production routing code in `contextual-orchestrator` and no maintainer/owner discussion of the tradeoff has happened yet. -**Fresh evidence, four same-day data points, growing.** All four are `noema-review`'s "Prepare Noema model verdict" step failing with `HTTPError: HTTP Error 502: Bad Gateway`, `phase=connecting` (per the entry above, this label covers the whole wait, not literal TCP connect), `served_model=unknown`, `caller attempts=1`: +**Fresh evidence, five same-day data points, growing.** All five are `noema-review`'s "Prepare Noema model verdict" step failing with `HTTPError: HTTP Error 502` (one, the earliest, `500`), `phase=connecting` (per the entry above, this label covers the whole wait, not literal TCP connect), `served_model=unknown`, `caller attempts=1`: +- `ContextualWisdomLab/contextual-orchestrator#1028`: `duration=2296.6s` (38m17s) — run [`33702758879`](https://github.com/ContextualWisdomLab/contextual-orchestrator/actions/runs/33702758879/job/100485372257). Notable: this is `contextual-orchestrator`'s own CI, stalled by `contextual-orchestrator`'s own gateway — the same pool the review sidecar routes through is congested regardless of which repository's PR triggered the request. +- `ContextualWisdomLab/mightyETL#330`: `duration=2161.9s` (36m2s) — run [`33701944041`](https://github.com/ContextualWisdomLab/mightyETL/actions/runs/33701944041/job/100482910533) (found and relayed by a peer session; that PR's `strix` check also failed separately, log expired `BlobNotFound`, cause not independently confirmed). - `ContextualWisdomLab/.github#1689`: `duration=1462.9s` (24m23s) — run [`33701909483`](https://github.com/ContextualWisdomLab/.github/actions/runs/33701909483/job/100482801962). -- `ContextualWisdomLab/naruon#1539`: `duration=1332.6s` (22m13s) — run [`33702091175`](https://github.com/ContextualWisdomLab/naruon/actions/runs/33702091175/job/100483351489) (found and relayed by a peer session). -- `ContextualWisdomLab/mightyETL#330`: `duration=2161.9s` (36m2s) — run [`33701944041`](https://github.com/ContextualWisdomLab/mightyETL/actions/runs/33701944041/job/100482910533) (found and relayed by the same peer session; that PR's `strix` check also failed separately, log expired `BlobNotFound`, cause not independently confirmed). +- `ContextualWisdomLab/naruon#1539`: `duration=1332.6s` (22m13s) — run [`33702091175`](https://github.com/ContextualWisdomLab/naruon/actions/runs/33702091175/job/100483351489) (found and relayed by the same peer session). - The prior entry's own `fast-mlsirm#1518` data point: `duration=649.5s` (10m49s), earlier the same day. -Sequence across the day: 649.5s → 1332.6s → 1462.9s → 2161.9s. Not proof of a monotonic trend on four points from three different repositories, but consistent with a shared-capacity resource degrading as the day's cumulative concurrent CI load (8+ sessions, dozens of repositories, all routing `noema-review`/`strix`/`opencode-review` through the same `orchestrator/free` pool) increases. +Sequence across the day: 649.5s → 1332.6s → 1462.9s → 2161.9s → 2296.6s. Not proof of a monotonic trend on five points from four different repositories, but consistent with a shared-capacity resource degrading as the day's cumulative concurrent CI load (8+ sessions, dozens of repositories, all routing `noema-review`/`strix`/`opencode-review` through the same `orchestrator/free` pool) increases. **Root cause, from direct code reading of `contextual-orchestrator@464da4715b495b5eaaa593eba3796e2d976ee0c9`'s `contextual_orchestrator/orchestrator.py`.** `TaskOrchestrator._invoke`'s failover loop (`:7762`, `for agent in candidates:`) iterates the **entire ranked free-tier candidate pool** — `free_selected_count` in `.github#1689`'s own sidecar preflight log was 12 — serially, with no overall wall-clock budget for the request as a whole. Each candidate gets an inner retry loop (`:7763`, `retry_limit = min(self.tool_retry_attempts, MAX_TOOL_RETRY_ATTEMPTS)`, `MAX_TOOL_RETRY_ATTEMPTS = 4` in `tool_fallback.py:12`) calling `ModelClient.chat`, whose defaults (`orchestrator.py:1699-1709`) are `timeout=90` seconds per HTTP attempt, `max_retries=2`, full-jitter exponential backoff (`:2091-2095`) capped at `retry_backoff_cap=8.0` seconds between attempts. Worst case for one candidate that hangs the full timeout on every attempt: `90 × 3 attempts + up to ~16s backoff ≈ 286s`. There is no per-candidate circuit breaker in this loop that skips a candidate known-bad from very recent history before spending its full timeout budget on it again — `#911`'s durable routing-observation EWMA data (`model_group.py`, merged this same day) is consumed by `_measured_member_order` for *ranking* candidate order, not for *excluding* a candidate outright, so a saturated pool still pays close to the full per-candidate cost for every member it reaches before either succeeding or exhausting the pool. Five candidates each hitting close to worst case (`5 × ~286s ≈ 1430s`) lands within seconds of the actual `.github#1689` observation (`1462.9s`); the smaller and larger observed values are consistent with fewer or more of the pool's 12 members being simultaneously slow rather than fast-failing (an immediate 429/503 response costs very little; a member that accepts the connection and then hangs costs close to the full 90s×3 budget) — exactly the shape you'd expect from a shared free-tier pool under uneven, fluctuating saturation across many concurrent callers org-wide. -**Why this is a real gap, not a config bug.** No single number here is wrong in isolation — a 90-second per-attempt timeout, 2 retries, and a 4-attempt candidate budget are all individually reasonable, and `docs/product-goal-directive.md` §8 deliberately forbids capping total model-inference wall-clock (see the entry above and `docs/doctoring/autofix-and-noema-review-model-job-timeout-removal.md`). The gap is structural: **the failover loop has no mechanism to fail fast past a candidate that is very likely to fail again** (state `_group_router`/`#911`'s observation store already tracks, per the problem-solving history in this session), so pool saturation shows up as linearly-compounding serial wait rather than either a fast failure or a bounded worst case. This also directly costs Actions job-slot capacity: every one of these four runs held a `noema-review` job (and its GitHub Actions runner) for 20-36 minutes before failing, worsening the same 60-job organization concurrency ceiling this document's queue-congestion entries already treat as a standing constraint. +**Why this is a real gap, not a config bug.** No single number here is wrong in isolation — a 90-second per-attempt timeout, 2 retries, and a 4-attempt candidate budget are all individually reasonable, and `docs/product-goal-directive.md` §8 deliberately forbids capping total model-inference wall-clock (see the entry above and `docs/doctoring/autofix-and-noema-review-model-job-timeout-removal.md`). The gap is structural: **the failover loop has no mechanism to fail fast past a candidate that is very likely to fail again** (state `_group_router`/`#911`'s observation store already tracks, per the problem-solving history in this session), so pool saturation shows up as linearly-compounding serial wait rather than either a fast failure or a bounded worst case. This also directly costs Actions job-slot capacity: every one of these five runs held a `noema-review` job (and its GitHub Actions runner) for 10-38 minutes before failing, worsening the same 60-job organization concurrency ceiling this document's queue-congestion entries already treat as a standing constraint — including, per the `contextual-orchestrator#1028` data point, `contextual-orchestrator`'s own required checks. -**Proposed direction, not implemented here.** Two independent levers, either alone or combined: (1) let `_invoke`'s candidate loop consult the existing `#911` routing-observation data to *skip* (not just deprioritize) a candidate whose recent observation window shows a failure rate above a threshold, rather than re-spending its full per-attempt timeout; (2) race a small number of top-ranked candidates concurrently (the function already contains a `race_members`/`outcome.winner_endpoint_id` concurrent path a few lines above the serial loop for a different condition not fully traced in this pass — worth checking whether that path can be extended to the free-tier worker case) instead of trying them strictly one after another. Both change shared production routing code with a 100% coverage bar in `contextual-orchestrator` and deserve their own PR, tests, and — given the 2-hour-plus-per-model-call policy this document already defends elsewhere — explicit confirmation that "skip a likely-bad candidate fast" does not become "cap total inference time" through the back door. Not filed as a code change in this session; filed as a tracked gap for the session or maintainer with the most current `contextual-orchestrator` routing context to pick up. +**Proposed direction, not implemented here.** Two independent levers, either alone or combined: (1) let `_invoke`'s candidate loop consult the existing `#911` routing-observation data to *skip* (not just deprioritize) a candidate whose recent observation window shows a failure rate above a threshold, rather than re-spending its full per-attempt timeout; (2) race a small number of top-ranked candidates concurrently (the function already contains a `race_members`/`outcome.winner_endpoint_id` concurrent path a few lines above the serial loop for a different condition not fully traced in this pass — worth checking whether that path can be extended to the free-tier worker case) instead of trying them strictly one after another. Both change shared production routing code in `contextual-orchestrator` (a large test suite, though — correcting an earlier draft of this claim — the repository's actual `fail-under=100` coverage gate in CI is scoped to `contextual_orchestrator/nim_benchmark.py` specifically, not the whole package) and deserve their own PR, tests, and — given the 2-hour-plus-per-model-call policy this document already defends elsewhere — explicit confirmation that "skip a likely-bad candidate fast" does not become "cap total inference time" through the back door. Not filed as a code change in this session; filed as a tracked gap for the session or maintainer with the most current `contextual-orchestrator` routing context to pick up. ## Item 41: CodeQL PR `startup_failure` blocking merges org-wide — existing-repo gap closed, future-repo gap open From c815307650a614ad8b9d73fa3f85b0a395967364 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Fri, 4 Sep 2026 02:58:05 +0900 Subject: [PATCH 3/7] docs(gap-baseline): correct item 4's proposed fix -- both mechanisms already exist and are structurally inert Traced the actual code: a real circuit breaker (_record_failure/ _circuit_open, wired into _failover_candidates) and a real concurrent racing path (_equivalent_race_members) both already exist in contextual-orchestrator. Neither can help the observed stall: the circuit breaker is in-memory per-process state, but the review sidecar boots a fresh process per CI job and makes exactly one request before exiting, so there's never a second request to benefit from a circuit that only opens after 3 accumulated failures. #911's routing-observation data has the identical structural problem -- also in-memory, also reset per process -- so the skip-fast design this entry originally proposed would have been just as inert, not a fix. Racing is inert for a different reason: it requires candidates to share a group_name and a declared endpoint_equivalence contract proving they're redundant replicas of the same model; the free-tier pool has neither -- its members are different models entirely, so racing them is a real speed-vs-quality product tradeoff, not a mechanical extension. Splits the revised direction into two genuinely different fixes (persist state across the ephemeral process boundary, vs. extend racing to non-equivalent candidates with an explicit tradeoff decision) instead of the two originally proposed, neither of which would have worked. --- docs/product-technical-gap-baseline.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/docs/product-technical-gap-baseline.md b/docs/product-technical-gap-baseline.md index e3f9394304..18f24de349 100644 --- a/docs/product-technical-gap-baseline.md +++ b/docs/product-technical-gap-baseline.md @@ -2731,7 +2731,11 @@ Sequence across the day: 649.5s → 1332.6s → 1462.9s → 2161.9s → 2296.6s. **Why this is a real gap, not a config bug.** No single number here is wrong in isolation — a 90-second per-attempt timeout, 2 retries, and a 4-attempt candidate budget are all individually reasonable, and `docs/product-goal-directive.md` §8 deliberately forbids capping total model-inference wall-clock (see the entry above and `docs/doctoring/autofix-and-noema-review-model-job-timeout-removal.md`). The gap is structural: **the failover loop has no mechanism to fail fast past a candidate that is very likely to fail again** (state `_group_router`/`#911`'s observation store already tracks, per the problem-solving history in this session), so pool saturation shows up as linearly-compounding serial wait rather than either a fast failure or a bounded worst case. This also directly costs Actions job-slot capacity: every one of these five runs held a `noema-review` job (and its GitHub Actions runner) for 10-38 minutes before failing, worsening the same 60-job organization concurrency ceiling this document's queue-congestion entries already treat as a standing constraint — including, per the `contextual-orchestrator#1028` data point, `contextual-orchestrator`'s own required checks. -**Proposed direction, not implemented here.** Two independent levers, either alone or combined: (1) let `_invoke`'s candidate loop consult the existing `#911` routing-observation data to *skip* (not just deprioritize) a candidate whose recent observation window shows a failure rate above a threshold, rather than re-spending its full per-attempt timeout; (2) race a small number of top-ranked candidates concurrently (the function already contains a `race_members`/`outcome.winner_endpoint_id` concurrent path a few lines above the serial loop for a different condition not fully traced in this pass — worth checking whether that path can be extended to the free-tier worker case) instead of trying them strictly one after another. Both change shared production routing code in `contextual-orchestrator` (a large test suite, though — correcting an earlier draft of this claim — the repository's actual `fail-under=100` coverage gate in CI is scoped to `contextual_orchestrator/nim_benchmark.py` specifically, not the whole package) and deserve their own PR, tests, and — given the 2-hour-plus-per-model-call policy this document already defends elsewhere — explicit confirmation that "skip a likely-bad candidate fast" does not become "cap total inference time" through the back door. Not filed as a code change in this session; filed as a tracked gap for the session or maintainer with the most current `contextual-orchestrator` routing context to pick up. +**Correction (2026-09-04): both mechanisms this entry originally proposed already exist in `contextual-orchestrator`, and neither can help the review-sidecar deployment. Traced to the actual, deeper constraint.** A closer read of `TaskOrchestrator` found that (1) a real circuit breaker already exists (`self._circuit`, `_record_failure`/`_circuit_open`, `orchestrator.py:7972-8018`, `circuit_failure_threshold=3`, `circuit_reset_seconds=30.0`) and is already wired into `_failover_candidates` (`:7968`, `healthy = [agent for agent in eligible if not self._circuit_open(agent.id)]`), which is what actually produces `_invoke`'s `candidates` list (`:7683`); and (2) a concurrent-racing path already exists too (`_equivalent_race_members`/`race_members`, `:7410-7433`, `:7707-7750`) and is already checked immediately before the serial loop for the `"text"` capability, the exact capability a chat-completion review request uses. + +**Why both are structurally inert for this failure mode.** The circuit breaker is per-process, in-memory state (`self._circuit` is a plain instance dict) that needs 3 accumulated failures for one candidate before it opens — but the review sidecar boots a **brand-new orchestrator process for every single CI job** (confirmed directly: `contextual_orchestrator_review_sidecar.sh`'s own log line, `"starting review sidecar on 127.0.0.1:18080"`, printed fresh on every job in every job log this document cites), and each sidecar makes exactly one real `/v1/chat/completions` request before the job ends. There is no second request in the same process to benefit from a circuit that only opens after the first one's own candidate-walk already paid most of the cost. `#911`'s routing-observation ledger (`self._group_router`) has the identical structural problem — also in-memory, also reset to empty on every fresh boot, so a skip-fast check reading it (this entry's original item (1)) would have been just as inert as the circuit breaker, not a fix. The racing path is inert for a different, more fundamental reason: `_equivalent_race_members` (`:7414`) requires candidates to share a `group_name` **and** carry a declared `endpoint_equivalence` contract (`EndpointEquivalenceContract`, `hedge_eligible=True`, `execution_policy="immediate_race"`) proving they are interchangeable replicas of the *same* model across redundant endpoints. The free-tier pool's members are declared as ordinary discovered agents with **no `group_name` at all** (confirmed: zero occurrences of `group_name` anywhere in `model_discovery.py` or `provider_catalog_bootstrap.py`) and are genuinely *different models* (DeepSeek, Gemma, Llama variants, per the preflight evidence above) — not redundant copies of one model, so racing them would mean sometimes accepting a materially weaker model's answer over a not-yet-finished stronger one. That is a real product-quality tradeoff, not a mechanical extension of the existing replica-racing design. + +**Revised proposed direction, still not implemented here, now split by tradeoff.** Two genuinely different fixes, not the two originally proposed: (1) **persist** circuit-breaker and/or routing-observation state across the ephemeral sidecar's process boundary (a shared file, or a lightweight external store keyed by pool/model identity) so that congestion learned by one CI job's sidecar benefits the next one within the reset window — this closes the gap without any quality tradeoff, but is real new infrastructure (concurrency-safe cross-process state, cache invalidation, and a decision about whether/how a review sidecar should share state with production traffic at all) rather than a small change; (2) **extend racing to non-equivalent fallback candidates** for the free-tier pool specifically — cheaper to build (reuse the existing concurrent machinery) but requires an explicit, named product decision about accepting a faster-but-possibly-weaker answer over a slower-but-stronger one, which is exactly the kind of tradeoff `docs/product-goal-directive.md` §8's "no fixed inference-time cap" policy was written to prevent being smuggled in as an implementation detail. Neither is implemented in this pass. This is shared production routing code in `contextual-orchestrator`, and — now more than before this correction — the right next step is a design decision from whoever owns that tradeoff, not another unilateral attempt at a small mechanical patch. ## Item 41: CodeQL PR `startup_failure` blocking merges org-wide — existing-repo gap closed, future-repo gap open From 3b487a810fe2d2ef60caebf8444c939055def4cb Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Fri, 4 Sep 2026 05:13:08 +0900 Subject: [PATCH 4/7] docs(gap-baseline): connect item 4's direction (1) to the half-built #911 contextual-orchestrator#911 (open, unmerged) already builds real, tested cross-process persistence infrastructure (routing_observation_ store.py, SQLite-backed, time-windowed) -- exactly what item 4's "persist state across the ephemeral process boundary" direction calls for. But it wires that store into the ranking ledger (_group_router), which per this entry's own earlier finding is never consulted for the free-tier pool's ungrouped candidates anyway. The mechanism that *is* consulted regardless of grouping -- the circuit breaker (self._circuit) -- isn't touched by #911 at all. Flags the concrete next step (route the circuit breaker through the same already-built store) without implementing it, since #911 itself is still unmerged and could change shape before landing. --- docs/product-technical-gap-baseline.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/product-technical-gap-baseline.md b/docs/product-technical-gap-baseline.md index 18f24de349..6d07bbd370 100644 --- a/docs/product-technical-gap-baseline.md +++ b/docs/product-technical-gap-baseline.md @@ -2737,6 +2737,8 @@ Sequence across the day: 649.5s → 1332.6s → 1462.9s → 2161.9s → 2296.6s. **Revised proposed direction, still not implemented here, now split by tradeoff.** Two genuinely different fixes, not the two originally proposed: (1) **persist** circuit-breaker and/or routing-observation state across the ephemeral sidecar's process boundary (a shared file, or a lightweight external store keyed by pool/model identity) so that congestion learned by one CI job's sidecar benefits the next one within the reset window — this closes the gap without any quality tradeoff, but is real new infrastructure (concurrency-safe cross-process state, cache invalidation, and a decision about whether/how a review sidecar should share state with production traffic at all) rather than a small change; (2) **extend racing to non-equivalent fallback candidates** for the free-tier pool specifically — cheaper to build (reuse the existing concurrent machinery) but requires an explicit, named product decision about accepting a faster-but-possibly-weaker answer over a slower-but-stronger one, which is exactly the kind of tradeoff `docs/product-goal-directive.md` §8's "no fixed inference-time cap" policy was written to prevent being smuggled in as an implementation detail. Neither is implemented in this pass. This is shared production routing code in `contextual-orchestrator`, and — now more than before this correction — the right next step is a design decision from whoever owns that tradeoff, not another unilateral attempt at a small mechanical patch. +**A concrete, lower-effort path to direction (1) already exists, half-built, sitting unmerged.** `contextual-orchestrator#911` ("Persist bounded model-group routing observations," open, not merged, `Devin`-reviewed, `2537 passed` at its own head) adds exactly the persistence primitive direction (1) calls for: a new `routing_observation_store.py` (`RoutingObservationStore`, a time-windowed SQLite-backed store keyed by `member_id`/`success`/`latency_seconds`) that lets `ModelGroupRouter`'s ledger survive across separate gateway processes within a configured wall-clock window (`--routing-observation-window-seconds`/`--state-db`). This is real, tested, already-built cross-process persistence infrastructure. **But it persists the wrong ledger for this specific gap**: `#911` wires the store into `_group_router`/`_quality_router` (the *ranking* ledgers `ModelGroupRouter.member_score`/`ranked_member_ids` read) — and, per the entry above, ranking is only ever consulted for candidates that share a `group_name`, which the free-tier pool's discovered agents never have. The mechanism that *is* consulted regardless of grouping — the circuit breaker (`self._circuit`, `_record_failure`/`_circuit_open`) — is not touched by `#911` at all; it stays exactly as in-memory and per-process as before. The concrete, low-risk next step once `#911` lands is to route `self._circuit`'s reads/writes through the same `RoutingObservationStore` mechanism `#911` already built and tested, using its existing `circuit_reset_seconds` (30s) as the natural replay window — reusing tested infrastructure for a second ledger, not building a new persistence layer from scratch. Not attempted in this pass: `#911` itself is unmerged and could still change shape before landing, and building on top of an unmerged PR risks needing a full rebase; flagging this connection (also left as a comment on `#911` itself) so whoever picks up either PR next has the concrete linkage. + ## Item 41: CodeQL PR `startup_failure` blocking merges org-wide — existing-repo gap closed, future-repo gap open **Problem.** Every ruleset-injected `codeql-pr.yml` run in every repository covered by org ruleset `18156473` (confirmed: bandscope, naruon, aFIPC, pg-erd-cloud, xtrmLLMBatchPython, wardnet, spanning 2026-09-02T20:12:52Z through 2026-09-03T03:15:43Z) concluded `startup_failure` with **zero check runs created** — while every other required workflow in the same PRs at the same time enqueued normally. Example: [wardnet run 33710719228](https://github.com/ContextualWisdomLab/wardnet/actions/runs/33710719228). From ee4f8bf4b9082c16f778af37c2460ca626a70753 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Fri, 4 Sep 2026 05:37:22 +0900 Subject: [PATCH 5/7] docs(gap-baseline): add 6th item-4 stall data point (contextual-orchestrator#1032, 2604.3s) --- docs/product-technical-gap-baseline.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/docs/product-technical-gap-baseline.md b/docs/product-technical-gap-baseline.md index 6d07bbd370..e48e804e32 100644 --- a/docs/product-technical-gap-baseline.md +++ b/docs/product-technical-gap-baseline.md @@ -2716,16 +2716,17 @@ prose" convention already stated in `CLAUDE.md`. ## Item 4: the underlying provider-pool stall itself, root-caused — 2026-09-03 -**Status:** Open. The prior entry above explicitly left this unresolved ("This does not fix the underlying 649-second provider stall itself — that remains a real, separate, unresolved question"). This entry answers that question with direct code evidence plus five same-day, cross-repository data points showing the stall duration growing over the course of the day, and proposes a design direction — not yet implemented, and deliberately not implemented in this pass given the change touches shared production routing code in `contextual-orchestrator` and no maintainer/owner discussion of the tradeoff has happened yet. +**Status:** Open. The prior entry above explicitly left this unresolved ("This does not fix the underlying 649-second provider stall itself — that remains a real, separate, unresolved question"). This entry answers that question with direct code evidence plus six same-day, cross-repository data points showing the stall duration growing over the course of the day, and proposes a design direction — not yet implemented, and deliberately not implemented in this pass given the change touches shared production routing code in `contextual-orchestrator` and no maintainer/owner discussion of the tradeoff has happened yet. -**Fresh evidence, five same-day data points, growing.** All five are `noema-review`'s "Prepare Noema model verdict" step failing with `HTTPError: HTTP Error 502` (one, the earliest, `500`), `phase=connecting` (per the entry above, this label covers the whole wait, not literal TCP connect), `served_model=unknown`, `caller attempts=1`: +**Fresh evidence, six same-day data points, growing.** All six are `noema-review`'s "Prepare Noema model verdict" step failing with `HTTPError: HTTP Error 502` (one, the earliest, `500`), `phase=connecting` (per the entry above, this label covers the whole wait, not literal TCP connect), `served_model=unknown`, `caller attempts=1`: +- `ContextualWisdomLab/contextual-orchestrator#1032`: `duration=2604.3s` (43m24s) — run [`33726444754`](https://github.com/ContextualWisdomLab/contextual-orchestrator/actions/runs/33726444754/job/100556361214). No PR-specific defect: only check failing, no other actionable issue found on this PR (unlike `#1028` below, which separately had a real, fixed test bug). - `ContextualWisdomLab/contextual-orchestrator#1028`: `duration=2296.6s` (38m17s) — run [`33702758879`](https://github.com/ContextualWisdomLab/contextual-orchestrator/actions/runs/33702758879/job/100485372257). Notable: this is `contextual-orchestrator`'s own CI, stalled by `contextual-orchestrator`'s own gateway — the same pool the review sidecar routes through is congested regardless of which repository's PR triggered the request. - `ContextualWisdomLab/mightyETL#330`: `duration=2161.9s` (36m2s) — run [`33701944041`](https://github.com/ContextualWisdomLab/mightyETL/actions/runs/33701944041/job/100482910533) (found and relayed by a peer session; that PR's `strix` check also failed separately, log expired `BlobNotFound`, cause not independently confirmed). - `ContextualWisdomLab/.github#1689`: `duration=1462.9s` (24m23s) — run [`33701909483`](https://github.com/ContextualWisdomLab/.github/actions/runs/33701909483/job/100482801962). - `ContextualWisdomLab/naruon#1539`: `duration=1332.6s` (22m13s) — run [`33702091175`](https://github.com/ContextualWisdomLab/naruon/actions/runs/33702091175/job/100483351489) (found and relayed by the same peer session). - The prior entry's own `fast-mlsirm#1518` data point: `duration=649.5s` (10m49s), earlier the same day. -Sequence across the day: 649.5s → 1332.6s → 1462.9s → 2161.9s → 2296.6s. Not proof of a monotonic trend on five points from four different repositories, but consistent with a shared-capacity resource degrading as the day's cumulative concurrent CI load (8+ sessions, dozens of repositories, all routing `noema-review`/`strix`/`opencode-review` through the same `orchestrator/free` pool) increases. +Sequence across the day: 649.5s → 1332.6s → 1462.9s → 2161.9s → 2296.6s → 2604.3s. Not proof of a monotonic trend on six points from four different repositories, but consistent with a shared-capacity resource degrading as the day's cumulative concurrent CI load (8+ sessions, dozens of repositories, all routing `noema-review`/`strix`/`opencode-review` through the same `orchestrator/free` pool) increases. **Root cause, from direct code reading of `contextual-orchestrator@464da4715b495b5eaaa593eba3796e2d976ee0c9`'s `contextual_orchestrator/orchestrator.py`.** `TaskOrchestrator._invoke`'s failover loop (`:7762`, `for agent in candidates:`) iterates the **entire ranked free-tier candidate pool** — `free_selected_count` in `.github#1689`'s own sidecar preflight log was 12 — serially, with no overall wall-clock budget for the request as a whole. Each candidate gets an inner retry loop (`:7763`, `retry_limit = min(self.tool_retry_attempts, MAX_TOOL_RETRY_ATTEMPTS)`, `MAX_TOOL_RETRY_ATTEMPTS = 4` in `tool_fallback.py:12`) calling `ModelClient.chat`, whose defaults (`orchestrator.py:1699-1709`) are `timeout=90` seconds per HTTP attempt, `max_retries=2`, full-jitter exponential backoff (`:2091-2095`) capped at `retry_backoff_cap=8.0` seconds between attempts. Worst case for one candidate that hangs the full timeout on every attempt: `90 × 3 attempts + up to ~16s backoff ≈ 286s`. There is no per-candidate circuit breaker in this loop that skips a candidate known-bad from very recent history before spending its full timeout budget on it again — `#911`'s durable routing-observation EWMA data (`model_group.py`, merged this same day) is consumed by `_measured_member_order` for *ranking* candidate order, not for *excluding* a candidate outright, so a saturated pool still pays close to the full per-candidate cost for every member it reaches before either succeeding or exhausting the pool. Five candidates each hitting close to worst case (`5 × ~286s ≈ 1430s`) lands within seconds of the actual `.github#1689` observation (`1462.9s`); the smaller and larger observed values are consistent with fewer or more of the pool's 12 members being simultaneously slow rather than fast-failing (an immediate 429/503 response costs very little; a member that accepts the connection and then hangs costs close to the full 90s×3 budget) — exactly the shape you'd expect from a shared free-tier pool under uneven, fluctuating saturation across many concurrent callers org-wide. From 8e902c243b953275a3f92befcd9f24fca2d2974c Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Fri, 4 Sep 2026 09:58:39 +0900 Subject: [PATCH 6/7] docs(gap-baseline): fix Devin Review findings on item 4's retry math and stale corrected-premise text - Retry math was wrong: the 286s worst-case only counted ModelClient's own 3-attempt internal retry, missing that _invoke's outer RETRY_SAME_AGENT loop (default tool_retry_attempts=1) re-invokes the whole 3-attempt call up to once more -- 6 total HTTP attempts per candidate, not 3, worst case ~540s not 286s. Verified directly against contextual-orchestrator's source (classify_provider_ transport_failure always returns RETRY_SAME_AGENT for a retryable transport failure). Updated the illustrative candidate-count math to match. - The "Why this is a real gap" paragraph still asserted the original, since-superseded premise (that the loop simply doesn't consult existing tracked state) even though a later "Correction" section in the same entry explains that consulting it wouldn't have helped anyway, for a deeper reason (ephemeral per-process state). Added an explicit forward-pointer so the two sections don't read as contradicting each other. - Restated the one HTTP 500 exception directly on its own bullet (previously only in the intro sentence) to remove any ambiguity a reader scanning the bullets alone might have. - Fixed two stale counts left over from earlier revisions of this entry ("five" runs/"four" repositories -> six runs/five repositories, matching the two data points added after this text was first written). Co-Authored-By: Claude Sonnet 5 --- docs/product-technical-gap-baseline.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/product-technical-gap-baseline.md b/docs/product-technical-gap-baseline.md index e48e804e32..90f9ae270b 100644 --- a/docs/product-technical-gap-baseline.md +++ b/docs/product-technical-gap-baseline.md @@ -2724,13 +2724,13 @@ prose" convention already stated in `CLAUDE.md`. - `ContextualWisdomLab/mightyETL#330`: `duration=2161.9s` (36m2s) — run [`33701944041`](https://github.com/ContextualWisdomLab/mightyETL/actions/runs/33701944041/job/100482910533) (found and relayed by a peer session; that PR's `strix` check also failed separately, log expired `BlobNotFound`, cause not independently confirmed). - `ContextualWisdomLab/.github#1689`: `duration=1462.9s` (24m23s) — run [`33701909483`](https://github.com/ContextualWisdomLab/.github/actions/runs/33701909483/job/100482801962). - `ContextualWisdomLab/naruon#1539`: `duration=1332.6s` (22m13s) — run [`33702091175`](https://github.com/ContextualWisdomLab/naruon/actions/runs/33702091175/job/100483351489) (found and relayed by the same peer session). -- The prior entry's own `fast-mlsirm#1518` data point: `duration=649.5s` (10m49s), earlier the same day. +- The prior entry's own `fast-mlsirm#1518` data point: `duration=649.5s` (10m49s, `HTTP 500` — the one exception the parenthetical above already carves out, restated here directly on its own bullet to remove any ambiguity), earlier the same day. -Sequence across the day: 649.5s → 1332.6s → 1462.9s → 2161.9s → 2296.6s → 2604.3s. Not proof of a monotonic trend on six points from four different repositories, but consistent with a shared-capacity resource degrading as the day's cumulative concurrent CI load (8+ sessions, dozens of repositories, all routing `noema-review`/`strix`/`opencode-review` through the same `orchestrator/free` pool) increases. +Sequence across the day: 649.5s → 1332.6s → 1462.9s → 2161.9s → 2296.6s → 2604.3s. Not proof of a monotonic trend on six points from five different repositories, but consistent with a shared-capacity resource degrading as the day's cumulative concurrent CI load (8+ sessions, dozens of repositories, all routing `noema-review`/`strix`/`opencode-review` through the same `orchestrator/free` pool) increases. -**Root cause, from direct code reading of `contextual-orchestrator@464da4715b495b5eaaa593eba3796e2d976ee0c9`'s `contextual_orchestrator/orchestrator.py`.** `TaskOrchestrator._invoke`'s failover loop (`:7762`, `for agent in candidates:`) iterates the **entire ranked free-tier candidate pool** — `free_selected_count` in `.github#1689`'s own sidecar preflight log was 12 — serially, with no overall wall-clock budget for the request as a whole. Each candidate gets an inner retry loop (`:7763`, `retry_limit = min(self.tool_retry_attempts, MAX_TOOL_RETRY_ATTEMPTS)`, `MAX_TOOL_RETRY_ATTEMPTS = 4` in `tool_fallback.py:12`) calling `ModelClient.chat`, whose defaults (`orchestrator.py:1699-1709`) are `timeout=90` seconds per HTTP attempt, `max_retries=2`, full-jitter exponential backoff (`:2091-2095`) capped at `retry_backoff_cap=8.0` seconds between attempts. Worst case for one candidate that hangs the full timeout on every attempt: `90 × 3 attempts + up to ~16s backoff ≈ 286s`. There is no per-candidate circuit breaker in this loop that skips a candidate known-bad from very recent history before spending its full timeout budget on it again — `#911`'s durable routing-observation EWMA data (`model_group.py`, merged this same day) is consumed by `_measured_member_order` for *ranking* candidate order, not for *excluding* a candidate outright, so a saturated pool still pays close to the full per-candidate cost for every member it reaches before either succeeding or exhausting the pool. Five candidates each hitting close to worst case (`5 × ~286s ≈ 1430s`) lands within seconds of the actual `.github#1689` observation (`1462.9s`); the smaller and larger observed values are consistent with fewer or more of the pool's 12 members being simultaneously slow rather than fast-failing (an immediate 429/503 response costs very little; a member that accepts the connection and then hangs costs close to the full 90s×3 budget) — exactly the shape you'd expect from a shared free-tier pool under uneven, fluctuating saturation across many concurrent callers org-wide. +**Root cause, from direct code reading of `contextual-orchestrator@464da4715b495b5eaaa593eba3796e2d976ee0c9`'s `contextual_orchestrator/orchestrator.py`.** `TaskOrchestrator._invoke`'s failover loop (`:7762`, `for agent in candidates:`) iterates the **entire ranked free-tier candidate pool** — `free_selected_count` in `.github#1689`'s own sidecar preflight log was 12 — serially, with no overall wall-clock budget for the request as a whole. **Correction (Devin Review): the per-candidate cost is roughly double the figure this entry first computed.** Each candidate's `self.client.chat()` call is itself internally retried by `ModelClient` (`orchestrator.py:1699-1709`, `timeout=90` seconds per HTTP attempt, `max_retries=2` → 3 total attempts, full-jitter exponential backoff capped at `retry_backoff_cap=8.0` seconds between attempts) — but `_invoke`'s own outer loop (`:7841`, `action is ToolFallbackAction.RETRY_SAME_AGENT and retry_attempt < retry_limit`) separately re-invokes that *entire* 3-attempt `self.client.chat()` call up to `retry_limit = min(self.tool_retry_attempts, MAX_TOOL_RETRY_ATTEMPTS)` more times, and a retryable transport failure (`classify_provider_transport_failure`, `tool_fallback.py:148-154`) always classifies as `RETRY_SAME_AGENT`. With the default `tool_retry_attempts=1` (`orchestrator.py:3849`), that is 2 outer passes, each covering 3 inner attempts: **up to 6 total HTTP attempts per candidate**, not 3 — worst case ≈ `2 × 3 × 90s ≈ 540s` before backoff (outer-loop backoff is small, `tool_retry_backoff_seconds=0.25` default, `:3850`; inner-loop backoff is likewise a few seconds at most, not the dominant term). There is a real per-candidate circuit breaker in this loop (`self._circuit`/`_circuit_open`, see the correction further below) but it cannot skip a candidate known-bad from very recent history the way the original draft of this entry assumed — the correction below explains why in detail. Roughly 3 candidates each hitting close to the corrected worst case (`3 × ~540s ≈ 1620s`) is in the same order of magnitude as the actual `.github#1689` observation (`1462.9s`) — closer to 2-3 simultaneously-slow candidates than the 5 this entry originally estimated from the uncorrected math; the smaller and larger observed values across this entry's data points are consistent with fewer or more of the pool's 12 members being simultaneously slow rather than fast-failing (an immediate 429/503 response costs very little; a member that accepts the connection and then hangs costs close to the full 6-attempt budget) — exactly the shape you'd expect from a shared free-tier pool under uneven, fluctuating saturation across many concurrent callers org-wide. -**Why this is a real gap, not a config bug.** No single number here is wrong in isolation — a 90-second per-attempt timeout, 2 retries, and a 4-attempt candidate budget are all individually reasonable, and `docs/product-goal-directive.md` §8 deliberately forbids capping total model-inference wall-clock (see the entry above and `docs/doctoring/autofix-and-noema-review-model-job-timeout-removal.md`). The gap is structural: **the failover loop has no mechanism to fail fast past a candidate that is very likely to fail again** (state `_group_router`/`#911`'s observation store already tracks, per the problem-solving history in this session), so pool saturation shows up as linearly-compounding serial wait rather than either a fast failure or a bounded worst case. This also directly costs Actions job-slot capacity: every one of these five runs held a `noema-review` job (and its GitHub Actions runner) for 10-38 minutes before failing, worsening the same 60-job organization concurrency ceiling this document's queue-congestion entries already treat as a standing constraint — including, per the `contextual-orchestrator#1028` data point, `contextual-orchestrator`'s own required checks. +**Why this is a real gap, not a config bug.** No single number here is wrong in isolation — a 90-second per-attempt timeout, 2 internal retries, an outer same-agent retry, and a 4-attempt candidate budget are all individually reasonable, and `docs/product-goal-directive.md` §8 deliberately forbids capping total model-inference wall-clock (see the entry above and `docs/doctoring/autofix-and-noema-review-model-job-timeout-removal.md`). The gap is structural: **the failover loop has no mechanism to fail fast past a candidate that is very likely to fail again.** The correction immediately below explains precisely why not — it is *not* simply that existing state (`self._circuit`, `_group_router`/`#911`'s observation store) goes unconsulted; even consulting it would not help, for a deeper reason tied to how this specific caller is deployed. Read the correction before treating this paragraph's framing as complete. Regardless of mechanism, pool saturation shows up as linearly-compounding serial wait rather than either a fast failure or a bounded worst case. This also directly costs Actions job-slot capacity: every one of these six runs held a `noema-review` job (and its GitHub Actions runner) for 10-43 minutes before failing, worsening the same 60-job organization concurrency ceiling this document's queue-congestion entries already treat as a standing constraint — including, per the `contextual-orchestrator#1028`/`#1032` data points, `contextual-orchestrator`'s own required checks. **Correction (2026-09-04): both mechanisms this entry originally proposed already exist in `contextual-orchestrator`, and neither can help the review-sidecar deployment. Traced to the actual, deeper constraint.** A closer read of `TaskOrchestrator` found that (1) a real circuit breaker already exists (`self._circuit`, `_record_failure`/`_circuit_open`, `orchestrator.py:7972-8018`, `circuit_failure_threshold=3`, `circuit_reset_seconds=30.0`) and is already wired into `_failover_candidates` (`:7968`, `healthy = [agent for agent in eligible if not self._circuit_open(agent.id)]`), which is what actually produces `_invoke`'s `candidates` list (`:7683`); and (2) a concurrent-racing path already exists too (`_equivalent_race_members`/`race_members`, `:7410-7433`, `:7707-7750`) and is already checked immediately before the serial loop for the `"text"` capability, the exact capability a chat-completion review request uses. From 373f2693c655b5840ac3df0938296246a623a054 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Fri, 4 Sep 2026 14:35:03 +0900 Subject: [PATCH 7/7] docs(gap-baseline): add 7th item-4 stall data point (contextual-orchestrator#1028 later push, 3913.0s) --- docs/product-technical-gap-baseline.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/docs/product-technical-gap-baseline.md b/docs/product-technical-gap-baseline.md index 90f9ae270b..01906765e0 100644 --- a/docs/product-technical-gap-baseline.md +++ b/docs/product-technical-gap-baseline.md @@ -2716,9 +2716,10 @@ prose" convention already stated in `CLAUDE.md`. ## Item 4: the underlying provider-pool stall itself, root-caused — 2026-09-03 -**Status:** Open. The prior entry above explicitly left this unresolved ("This does not fix the underlying 649-second provider stall itself — that remains a real, separate, unresolved question"). This entry answers that question with direct code evidence plus six same-day, cross-repository data points showing the stall duration growing over the course of the day, and proposes a design direction — not yet implemented, and deliberately not implemented in this pass given the change touches shared production routing code in `contextual-orchestrator` and no maintainer/owner discussion of the tradeoff has happened yet. +**Status:** Open. The prior entry above explicitly left this unresolved ("This does not fix the underlying 649-second provider stall itself — that remains a real, separate, unresolved question"). This entry answers that question with direct code evidence plus seven same-day, cross-repository data points showing the stall duration growing over the course of the day, and proposes a design direction — not yet implemented, and deliberately not implemented in this pass given the change touches shared production routing code in `contextual-orchestrator` and no maintainer/owner discussion of the tradeoff has happened yet. -**Fresh evidence, six same-day data points, growing.** All six are `noema-review`'s "Prepare Noema model verdict" step failing with `HTTPError: HTTP Error 502` (one, the earliest, `500`), `phase=connecting` (per the entry above, this label covers the whole wait, not literal TCP connect), `served_model=unknown`, `caller attempts=1`: +**Fresh evidence, seven same-day data points, growing.** All seven are `noema-review`'s "Prepare Noema model verdict" step failing with `HTTPError: HTTP Error 502` (one, the earliest, `500`), `phase=connecting` (per the entry above, this label covers the whole wait, not literal TCP connect), `served_model=unknown`, `caller attempts=1`: +- `ContextualWisdomLab/contextual-orchestrator#1028` (later push): `duration=3913.0s` (1h5m13s) — run [`33767309930`](https://github.com/ContextualWisdomLab/contextual-orchestrator/actions/runs/33767309930/job/100688363895). The largest data point yet, on a *later* head of the same PR listed below (`33c5ede0`, after this session's own unrelated test fix landed) — confirms the stall recurs across separate pushes to the same PR, not a one-off tied to a specific commit. - `ContextualWisdomLab/contextual-orchestrator#1032`: `duration=2604.3s` (43m24s) — run [`33726444754`](https://github.com/ContextualWisdomLab/contextual-orchestrator/actions/runs/33726444754/job/100556361214). No PR-specific defect: only check failing, no other actionable issue found on this PR (unlike `#1028` below, which separately had a real, fixed test bug). - `ContextualWisdomLab/contextual-orchestrator#1028`: `duration=2296.6s` (38m17s) — run [`33702758879`](https://github.com/ContextualWisdomLab/contextual-orchestrator/actions/runs/33702758879/job/100485372257). Notable: this is `contextual-orchestrator`'s own CI, stalled by `contextual-orchestrator`'s own gateway — the same pool the review sidecar routes through is congested regardless of which repository's PR triggered the request. - `ContextualWisdomLab/mightyETL#330`: `duration=2161.9s` (36m2s) — run [`33701944041`](https://github.com/ContextualWisdomLab/mightyETL/actions/runs/33701944041/job/100482910533) (found and relayed by a peer session; that PR's `strix` check also failed separately, log expired `BlobNotFound`, cause not independently confirmed). @@ -2726,7 +2727,7 @@ prose" convention already stated in `CLAUDE.md`. - `ContextualWisdomLab/naruon#1539`: `duration=1332.6s` (22m13s) — run [`33702091175`](https://github.com/ContextualWisdomLab/naruon/actions/runs/33702091175/job/100483351489) (found and relayed by the same peer session). - The prior entry's own `fast-mlsirm#1518` data point: `duration=649.5s` (10m49s, `HTTP 500` — the one exception the parenthetical above already carves out, restated here directly on its own bullet to remove any ambiguity), earlier the same day. -Sequence across the day: 649.5s → 1332.6s → 1462.9s → 2161.9s → 2296.6s → 2604.3s. Not proof of a monotonic trend on six points from five different repositories, but consistent with a shared-capacity resource degrading as the day's cumulative concurrent CI load (8+ sessions, dozens of repositories, all routing `noema-review`/`strix`/`opencode-review` through the same `orchestrator/free` pool) increases. +Sequence across the day: 649.5s → 1332.6s → 1462.9s → 2161.9s → 2296.6s → 2604.3s → 3913.0s. Not proof of a monotonic trend on seven points from five different repositories, but consistent with a shared-capacity resource degrading as the day's cumulative concurrent CI load (8+ sessions, dozens of repositories, all routing `noema-review`/`strix`/`opencode-review` through the same `orchestrator/free` pool) increases. **Root cause, from direct code reading of `contextual-orchestrator@464da4715b495b5eaaa593eba3796e2d976ee0c9`'s `contextual_orchestrator/orchestrator.py`.** `TaskOrchestrator._invoke`'s failover loop (`:7762`, `for agent in candidates:`) iterates the **entire ranked free-tier candidate pool** — `free_selected_count` in `.github#1689`'s own sidecar preflight log was 12 — serially, with no overall wall-clock budget for the request as a whole. **Correction (Devin Review): the per-candidate cost is roughly double the figure this entry first computed.** Each candidate's `self.client.chat()` call is itself internally retried by `ModelClient` (`orchestrator.py:1699-1709`, `timeout=90` seconds per HTTP attempt, `max_retries=2` → 3 total attempts, full-jitter exponential backoff capped at `retry_backoff_cap=8.0` seconds between attempts) — but `_invoke`'s own outer loop (`:7841`, `action is ToolFallbackAction.RETRY_SAME_AGENT and retry_attempt < retry_limit`) separately re-invokes that *entire* 3-attempt `self.client.chat()` call up to `retry_limit = min(self.tool_retry_attempts, MAX_TOOL_RETRY_ATTEMPTS)` more times, and a retryable transport failure (`classify_provider_transport_failure`, `tool_fallback.py:148-154`) always classifies as `RETRY_SAME_AGENT`. With the default `tool_retry_attempts=1` (`orchestrator.py:3849`), that is 2 outer passes, each covering 3 inner attempts: **up to 6 total HTTP attempts per candidate**, not 3 — worst case ≈ `2 × 3 × 90s ≈ 540s` before backoff (outer-loop backoff is small, `tool_retry_backoff_seconds=0.25` default, `:3850`; inner-loop backoff is likewise a few seconds at most, not the dominant term). There is a real per-candidate circuit breaker in this loop (`self._circuit`/`_circuit_open`, see the correction further below) but it cannot skip a candidate known-bad from very recent history the way the original draft of this entry assumed — the correction below explains why in detail. Roughly 3 candidates each hitting close to the corrected worst case (`3 × ~540s ≈ 1620s`) is in the same order of magnitude as the actual `.github#1689` observation (`1462.9s`) — closer to 2-3 simultaneously-slow candidates than the 5 this entry originally estimated from the uncorrected math; the smaller and larger observed values across this entry's data points are consistent with fewer or more of the pool's 12 members being simultaneously slow rather than fast-failing (an immediate 429/503 response costs very little; a member that accepts the connection and then hangs costs close to the full 6-attempt budget) — exactly the shape you'd expect from a shared free-tier pool under uneven, fluctuating saturation across many concurrent callers org-wide.