Persist bounded model-group routing observations by seonghobae · Pull Request #911 · ContextualWisdomLab/contextual-orchestrator

seonghobae · 2026-08-28T20:58:42Z

Summary

Add an explicit, opt-in SQLite routing_observations store for transport and quality ledgers.
Replay only a configured wall-clock window across gateway processes; prune expired rows on writes and fail closed on persistence errors.
Keep default process-local behavior, no decay, no cross-model weighting, and no production horizontal-scaling claim.
Document the boundary in the PRD/TRD/gap baseline and ADR 0039.
Preserve successful observations without inventing latency, isolate retention leases from failed startup, contain persistence failures during embedding failover, and bind batch quality to the execution-time routing context.
Treat active unexpired leases as pruning authority; the historical maximum window remains compatibility/audit metadata only.

Validation

uv run pytest -q — 3452 passed, 2 skipped on exact head e80949188f0afa86052f10f5a9b627da1ee1ef0b
focused routing, batching, embedding failover, NIM acceptance, and security metadata regression suite — 190 passed
ruff check --select E9,F63,F7,F82 on changed Python files
semgrep --config p/python on changed runtime files — 0 findings
python -m compileall -q contextual_orchestrator
git diff --check

Stacked on PR #1068 exact head fa5446294ae7ae69f1c2958aa1ab6c071fa760bc; the PR head contains the latest main and was pushed by fast-forward without force.

Summary by CodeRabbit

새로운 기능
- 라우팅 관찰 정보를 선택적으로 시간 창 동안 저장하고 여러 게이트웨이 프로세스에서 공유할 수 있습니다.
- --routing-observation-window-seconds 및 --state-db 옵션으로 설정할 수 있습니다.
- 관리자 상태에서 관찰 정책과 적용 중인 시간 창을 확인할 수 있습니다.
버그 수정
- 빈 seed 및 top_logprobs가 불필요한 비스트리밍 처리를 유발하지 않습니다.
- 저장 실패 시 응답과 장애 조치가 계속됩니다.
- 관찰 데이터 정리가 더 긴 보존 창의 증거를 삭제하지 않습니다.
- 민감한 제공자 오류 정보가 고객 응답에서 자동으로 가려집니다.

coderabbitai · 2026-08-28T20:58:54Z

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

@coderabbitai resume to resume automatic reviews.
@coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

▶️ Resume reviews
🔍 Trigger review

📝 Walkthrough

Walkthrough

옵트인 SQLite 라우팅 관찰 저장소가 컨텍스트 키와 완료 시각을 저장하고 공유합니다. CLI와 오케스트레이터가 저장소를 구성하고 라우팅 경로에 연결합니다. Responses의 빈 seed와 top_logprobs는 생략과 동일하게 처리됩니다. Provider 오류 메시지는 안전한 형태로 정제됩니다.

Changes

라우팅 관찰 지속성

Layer / File(s)	Summary
관찰 저장소 계약과 SQLite 구현 `contextual_orchestrator/routing_observation_store.py`	`RoutingObservation`과 저장소 프로토콜을 추가합니다. 관찰 컨텍스트와 완료 시각을 저장합니다. 공유 데이터베이스의 최대 등록 보존 창을 기준으로 정리합니다.
라우터 관찰 기록과 재생 `contextual_orchestrator/model_group.py`	`ModelGroupRouter`가 멤버별 컨텍스트를 관리하고 관찰값을 저장 및 재생합니다. 멤버 삭제, prior 값, 잠금과 상태 수치를 처리합니다.
오케스트레이터와 CLI 연결 `contextual_orchestrator/orchestrator.py`, `contextual_orchestrator/__main__.py`, `contextual_orchestrator/__init__.py`, `contextual_orchestrator/server.py`	시간 창 옵션을 검증하고 SQLite 저장소를 transport 및 quality 라우터에 연결합니다. 요청, 장애 조치, 스트리밍, 실시간 품질 판정에 관찰 컨텍스트를 전달합니다.
라우팅 관찰 검증 `tests/test_routing_observation_store.py`, `tests/test_measured_routing_evidence.py`	공유 저장소, 컨텍스트 필터, 새로고침, 보존 경계, 재시작 복원, 저장 오류와 장애 조치를 검증합니다.
라우팅 관찰 운영 문서 `README.md`, `CHANGELOG.md`, `docs/model-group-product-technical-spec.md`, `docs/planning/adrs/0032-model-group-cost-aware-discovery.md`, `docs/planning/adrs/0039-time-windowed-routing-observations.md`, `docs/product_planning.md`, `docs/product-technical-gap-baseline.md`, `docs/library_research.md`	SQLite 시간 창의 설정, 공유 보존 의미, 저장 오류 처리와 지원 범위를 문서화합니다.

요청 경로 동작 수정

Layer / File(s)	Summary
Responses 입력 정규화와 provider 오류 진단 처리 `contextual_orchestrator/server.py`, `contextual_orchestrator/provider_errors.py`	빈 `seed`와 `top_logprobs`를 필드 생략으로 정규화합니다. `ProviderUpstreamError` 메시지를 제어 문자 제거, 길이 제한과 민감 정보 필터를 거쳐 처리합니다.
요청 경로 회귀 검증 `tests/test_orchestrated_responses_stream.py`, `tests/test_provider_error_taxonomy.py`, `tests/test_true_streaming.py`	빈 제어 필드가 공급자 전용 경로를 강제하지 않는지 검증합니다. HTTP와 SSE 오류 응답에서 민감한 provider 진단이 제거되는지 검증합니다.

프로젝트 기록 갱신

Layer / File(s)	Summary
ADR 및 기준 문서 갱신 `docs/planning/adrs/0040-streamed-responses-usage-boundary.md`, `docs/product-technical-gap-baseline.md`	스트리밍 ADR 식별자를 `0040`으로 변경합니다. 요청 경로와 라우팅 관찰 검증 기록을 갱신합니다.

Estimated code review effort: 4 (Complex) | ~60 minutes

Merge Risk: 🟡 Moderate · up to 64143

The opt-in durable-routing path can turn successful requests into failures during storage errors, lose routing observations when provider identity changes, and expose raw exception details in service logs. The PR should not merge until these bounded correctness, availability, and observability risks are fixed or explicitly accepted.

Sequence Diagram(s)

sequenceDiagram
  participant CLI
  participant TaskOrchestrator
  participant ModelGroupRouter
  participant SqliteRoutingObservationStore
  participant SQLite

  CLI->>TaskOrchestrator: 시간 창과 state_db 전달
  TaskOrchestrator->>SqliteRoutingObservationStore: 저장소 초기화
  TaskOrchestrator->>ModelGroupRouter: transport 및 quality 라우터 연결
  ModelGroupRouter->>SqliteRoutingObservationStore: context_key와 observed_at을 포함한 관찰값 append
  SqliteRoutingObservationStore->>SQLite: 관찰값 저장 및 공유 최대 보존 창 기준 정리
  TaskOrchestrator->>ModelGroupRouter: refresh()
  ModelGroupRouter->>SqliteRoutingObservationStore: active_contexts로 조회
  SqliteRoutingObservationStore->>SQLite: 현재 컨텍스트 관찰값 조회
  SQLite-->>SqliteRoutingObservationStore: 관찰값 반환
  SqliteRoutingObservationStore-->>ModelGroupRouter: 관찰값 목록 반환

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 47.75% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 111 functions across 11 files. (9 skipped…	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	제목은 모델 그룹 라우팅 관찰값을 제한된 범위로 영속화하는 주요 변경을 정확하고 간결하게 설명합니다.

Full details: Docstring Coverage

Explanation

Docstring coverage is 47.75% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 111 functions across 11 files. (9 skipped: 8 unsupported, 1 too large.)

Fix all pre-merge checks with AI

✨ Finishing Touches 💡 2

📝 Generate docstrings 💡

Create stacked PR
Commit on current branch

🛠️ Fix failing CI checks 💡

Create stacked PR
Commit on current branch

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch feat/durable-routing-observations-20260829

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

github-actions · 2026-08-29T08:35:38Z

Reusable conflict resolver stopped fail-closed. Executable or structured-data conflicts require semantic resolution at exact head b9c3f58ca84d0cf0fa5505ea89de4640bb7070c0 against protected main 9b0a356daa4f6bfcb5f83a314f11a7b273cd2623:\n\n```\ncontextual_orchestrator/orchestrator.py

github-actions · 2026-08-29T08:37:34Z

Reusable conflict resolver stopped fail-closed.

Exact PR head: d703d9d4bc35a3f6ca04afe8577ecdee1ab7af4c
Protected main: 9b0a356daa4f6bfcb5f83a314f11a7b273cd2623

Executable or structured-data conflicts require semantic resolution:

`contextual_orchestrator/orchestrator.py`

  3109         cache_max_entries: int = 256,
  3110         tool_retry_attempts: int = 1,
  3111         tool_retry_backoff_seconds: float = 0.25,
  3112         cache_provider: ResponseCacheProvider | None = None,
  3113         role_effort_catalog: dict[str, ReasoningEffortProfile] | None = None,
  3114         pii_key_name: str = DEFAULT_PII_KEY_NAME,
  3115 <<<<<<< ours
  3116         routing_observation_window_seconds: int | None = None,
  3117 ||||||| base
  3118 =======
  3119         allow_empty_agents: bool = False,
  3120 >>>>>>> theirs
  3121     ) -> None:
  3122         # Optional durable model-group management: stored operator changes overlay the
  3123         # seed agents file at startup (stored rows win by id; stored-new rows append).
  3124         self._pool_store = _AgentPoolStore(agents_db) if agents_db else None
  3125         if self._pool_store is not None:
  3126             stored = {agent.id: agent for agent in self._pool_store.load_all()}
  4268         answer = "".join(parts)
  4269         # Real-time judging after the stream: already-sent bytes cannot be
  4270         # recalled, so the verdict never changes this response -- it feeds the
  4271         # quality ledger so measured accuracy steers future member ordering,
  4272         # and it is persisted for audit.
  4273         latency_seconds = time.perf_counter() - started_at
  4274 <<<<<<< ours
  4275         try:
  4276             verification = self._realtime_route_judge(
  4277                 text=text,
  4278                 answer=answer,
  4279                 served_id=agent.id,
  4280                 latency_seconds=latency_seconds,
  4281                 usage=None,
  4282                 free_only=model_name == self.FREE_MODEL,
  4283             )
  4284         except RoutingObservationPersistenceError:
  4285             _LOGGER.error(
  4286                 "durable routing observation failed after streamed response completion"
  4287             )
  4288             verification = {
  4289                 "accepted": False,
  4290                 "reason": "routing observation persistence failed after stream completion",
  4291                 "verifier_output": "",
  4292             }
  4293 ||||||| base
  4294         verification = self._realtime_route_judge(
  4295             text=text,
  4296             answer=answer,
  4297             served_id=agent.id,
  4298             latency_seconds=latency_seconds,
  4299             usage=None,
  4300             free_only=model_name == self.FREE_MODEL,
  4301         )
  4302 =======
  4303         verification = self._realtime_route_judge(
  4304             text=text,
  4305             answer=answer,
  4306             served_id=agent.id,
  4307             latency_seconds=latency_seconds,
  4308             usage=usage,
  4309             free_only=model_name == self.FREE_MODEL,
  4310         )
  4311         trace_step = {
  4312             "id": 0,
  4313             "role": "worker",
  4314             "agent_id": agent.id,
  4315             "subtask": "Direct route (streamed)",
  4316             "access": [],
  4317             "output": answer,
  4318         }
  4319         if isinstance(usage, dict):
  4320             trace_step["usage"] = usage
  4321 >>>>>>> theirs
  4322         record = self._with_effort_snapshot(
  4323             {
  4324                 "workflow_run_id": workflow_run_id or f"run_{uuid.uuid4().hex}",
  4325                 "created_at": int(time.time()),
  4326                 "mode": "route",
  4327                 "policy_mode": "route",

seonghobae · 2026-09-02T13:36:46Z

Routine staleness triage: this is the PR matching the org's known stale-base marker — its recorded base.sha is c6c3a0c9d6c2ec8f3509ccdc6db24b1562a32e7b, far behind current main (464da4715b495b5eaaa593eba3796e2d976ee0c9), and mergeable_state is genuinely dirty (a real conflict, not just "behind").

I deliberately did not attempt a merge/push here: this session's shared environment shows clear signs of another in-progress process already mid-resolution on this exact branch (an existing local checkout with unpushed commits including a main-merge and uncommitted edits to test files). Duplicating that work risks a race/overwrite, so I'm leaving it alone rather than guessing at a second, possibly-conflicting resolution.

Flagging for visibility in case that other effort stalls or is abandoned: as of this comment the PR branch on GitHub is still unchanged (head 5d43fa0, base still the old marker commit), so it will need the merge-main → resolve-conflicts → verify → push treatment from whoever picks it up next.

Generated by Claude Code

seonghobae · 2026-09-02T18:42:16Z

Re-verified this PR fresh (head still 5d43fa02ea4d86a863e757f5d24f190167fff8fc, mergeable_state: dirty, last real commit 2026-08-31T21:20:11Z) before touching anything, per this session's standard stale-base recipe. Stopping short of that recipe here — no merge, no push — for two reasons visible only in this thread's history, not in PR metadata:

Standing owner hold. Your 2026-09-01 comment on this exact head ("Review only: do not merge, auto-merge, update/rebase the branch, weaken gates, or reuse predecessor-head verdicts") plus the "must not merge as-is... reconcile with fix(routing): select concrete free model groups #971" finding are still in force. fix(routing): select concrete free model groups #971 (canonical owner lane for the routing/timeout contract this PR predates) remains mergeable_state: blocked, explicitly "not merge-ready" as of 2026-09-03 in its own body. That reconciliation precondition hasn't landed, so merging main into this branch now would be exactly the "update/rebase the branch" you asked withheld.
Possible concurrent local work. The 13:36 UTC comment today reported another session's local checkout with an unpushed main-merge and edited test files on this exact branch. I can't confirm from here whether that's still live, so I'm not racing a second resolution against it.

Root-caused both original CI failures from job logs anyway, since that's independent of whether a merge proceeds:

noema-review (job 99731238054): sidecar started cleanly, gateway preflight found a live route (meta/llama-3.2-11b-vision-instruct via nvidia_nim), the LLM review ran and returned a verdict — but scripts/ci/noema_review_gate.py's evidence validator rejected it: Noema reviewed line 1 is not an exact changed-side line. That's the exact-changed-side-line binding check on the LLM's finding failing, not a defect in this PR's diff. Matches what you already noted: central .github#1546 landed a "bounded truncation that preserves changed-line identity" repair for this validator after this run.
opencode-review (job 99705874068): the repository_dispatch to trigger a review fired successfully, but the required-check step then polled pulls/911/reviews for 90 minutes (180×30s) and never saw an APPROVED/CHANGES_REQUESTED review from opencode-agent on this exact head SHA — a pure timeout/no-verdict, not a review finding.

Both are review-infrastructure gaps (consistent with the #868/#857/#906/#912-cycle pattern already flagged in this thread), not code defects this PR needs to fix. No action taken beyond this diagnosis — leaving the merge and any check re-run to whoever completes the #971 reconciliation, per your hold.

Generated by Claude Code

opencode-agent · 2026-09-03T07:20:58Z

Scheduled review-feedback autofix for this PR head.

Head SHA: 5d43fa02ea4d86a863e757f5d24f190167fff8fc

Resolves 8 conflicted files on this 5-day-old branch (CHANGELOG.md, __main__.py, model_group.py, orchestrator.py, provider_errors.py, docs/library_research.md, docs/product-technical-gap-baseline.md, tests/test_embeddings_model_pool_http_honesty.py). The two docs/CHANGELOG conflicts were the usual "both sides append a distinct dated entry" shape (kept both). The code conflicts needed real feature-merging, not a pick-one-side resolution, since main had independently evolved the exact same call sites this branch touches: - model_group.py's observe_success/_apply_success_locked: kept this branch's persist_observation/_apply_success_locked architecture (already used by 5+ other call sites org-wide) but merged in main's independently-added optional-latency support (latency_seconds: float | None, for successes with no honest single-attempt timing, e.g. shared Batch API calls) and its clamped-is-not-None guard -- which required widening _apply_success_locked's own latency param to Optional and adding the matching guard there too, since its original unconditional ewma update would have crashed on None. - orchestrator.py's _orchestrated_provider_completion synthesis/repair retry loop: took main's current while-loop structure (EffortProfileError handling, schema-repair-once logic -- this branch predates that whole refactor) but restored this branch's durable-persistence context-key tracking (synthesis_context_key/repair_context_key, _record_group_failure(_for_agent) wrapper calls) at each of the 4 failure-recording call sites, and fixed one condition (candidate.group_name and not (request_too_large OR capability_mismatch)) this branch had dropped relative to main. - _invoke's return shape: main independently widened it to a 4-tuple (output, served_id, served_model, usage); this branch's competing widening was a named _InvocationResult dataclass (output, served_id, usage, observation_context_key) with a 3-item __iter__ for tuple-unpack compat. Merged: added served_model as _InvocationResult's 3rd field, extended __iter__ to yield it, and fixed both _InvocationResult(...) construction sites plus the __invoke race-path usage extraction (was reading the stale outcome.value[2] as usage; main's caller-side code already used outcome.value[3] a few lines above it -- confirmed via call()'s own `-> tuple[str, str, str, ...]` return annotation that index 2 is now served_model, not usage). - provider_errors.py: kept this branch's _sanitize_provider_message_text extraction (reused at a second call site) but folded in main's independently-added _SAFE_SCHEMA_DIAGNOSTIC check, which the extraction had been missing. Two rounds of full-suite failures beyond the marked conflicts, both fixed: 1. route_once had a leftover `getattr(invocation, ...)` reference to a variable a different call site's resolution had removed -- caused a NameError cascading into ~448 failures across every test whose fixture transitively calls route_once. Fixed by capturing the _InvocationResult object instead of discarding it after unpacking. 2. Two unrelated post-fix failures: an ADR-0039 number collision (this branch's own "time-windowed-routing-observations" ADR vs. an independently-landed main ADR of the same number) -- renumbered to 0042 (next free slot before the 0124 block), fixed 3 cross-references in CHANGELOG.md/library_research.md/product-technical-gap-baseline.md. And a genuinely unrelated main-side token_counting.py rewrite (old heuristic word-count estimator replaced with a strict exact-model-allowlist policy that fails closed for unknown model names) broke this branch's own test_embedding_attempts_keep_their_original_routing_context, which never got the ExactSyntheticCounter workaround every sibling test in the same file already uses for its "mock-planner" fixture model name -- applied the same pattern. Full suite: 3445 passed, 1 skipped. provider_errors.py 100% coverage, model_group.py 98% (pre-existing gaps in unrelated validation branches, confirmed my own added guard is fully covered), interrogate 100%. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

devin-ai-integration

Devin Review found 5 new potential issues.

…only CodeRabbit flagged that the correction paragraph mischaracterized contextual-orchestrator#911 as "EWMA-based candidate-exclusion data." Per .github#1804 (the source PR#1043 already cites), #911's routing-observation EWMA data only reorders candidates by score — it does not exclude or skip a saturated one. Landing #911 alone would not solve the serial-failover stall. Verified directly via the GitHub API that #911 is still open, unmerged, and mergeable_state=blocked, so CodeRabbit's "already merged" premise for this same finding is false and was not acted on. The underlying re-ranking-vs-exclusion mischaracterization was real on its own terms and is fixed here: a separate, not-yet-built durable exclusion/skip mechanism (consuming #911's ranking data once #911 lands) is what's actually still required, not #911 by itself. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BV96rXhqoR3tYZ9AeAVur4

seonghobae · 2026-09-03T20:13:26Z

Found a concrete connection to this PR while investigating a separate, long-standing gateway-stall issue (.github's gap-baseline "item 4": noema-review and other CI-triggered chat-completion requests sometimes stall 10-38 minutes walking the free-tier candidate pool serially before failing, .github#1804).

This PR's new routing_observation_store.py is real, tested, cross-process persistence infrastructure — exactly the kind of fix item 4 needs (its callers, review sidecars, boot a brand-new orchestrator process per CI job and make exactly one request before exiting, so any purely in-memory adaptive mechanism has no chance to learn across requests). But as wired here, it persists _group_router/_quality_router — the ranking ledgers ModelGroupRouter.member_score/ranked_member_ids read — and ranking is only ever consulted for candidates sharing a group_name (_refine_partition, orchestrator.py). The free-tier pool's discovered agents never get a group_name (verified: zero occurrences in model_discovery.py/provider_catalog_bootstrap.py), so this persistence, once merged, would not by itself change anything for that pool's stall.

The mechanism that is consulted regardless of grouping is the circuit breaker (self._circuit, _record_failure/_circuit_open, wired into _failover_candidates at orchestrator.py:7968) — and it isn't touched by this PR at all, staying exactly as in-memory/per-process as before.

Not asking for scope creep on this PR — just flagging a concrete, low-risk follow-up once this lands: route self._circuit's reads/writes through the same RoutingObservationStore this PR already built and tested, using the existing circuit_reset_seconds (30s) as the natural replay window. Reusing tested infrastructure for a second ledger, rather than building a new persistence layer from scratch. Full writeup: .github#1804.

opencode-agent · 2026-09-04T11:11:49Z

Scheduled review-feedback autofix for this PR head.

Head SHA: ba3b299e9084c27b941e4dc61183a82f81dd7248

opencode-agent · 2026-09-04T13:31:45Z

Scheduled review-feedback autofix for this PR head.

Head SHA: ba3b299e9084c27b941e4dc61183a82f81dd7248

opencode-agent · 2026-09-04T16:31:54Z

Scheduled review-feedback autofix for this PR head.

Head SHA: ba3b299e9084c27b941e4dc61183a82f81dd7248

Assisted-by: OpenAI Codex Signed-off-by: Seongho Bae <me@seonghobae.me>

…to codex/pr911-repair

The correction above defers option (a) to a "not-yet-built durable candidate-exclusion/skip mechanism". Reading TaskOrchestrator at pin 414f229 shows a per-agent breaker already exists and still would not exclude a stalled candidate, for two independent reasons: 1. The tool-bearing passthrough re-raises a bare TimeoutError as 500 internal_error before _record_failure (orchestrator.py:8048) runs, so the breaker never counts the failure. Measured: 0 of 21, 0 of 48, 0 of 63 and 0 of 65 passthrough timeouts recorded as circuit_failure, against 9/14 and 10/15 on the no-tools _invoke route-walk. That is #1082's scope. 2. _circuit_open clears state["failures"] to 0.0 once circuit_reset_seconds (30.0) have elapsed since opened_at (orchestrator.py:8036-8038). Against the ~90s attempts these stalls take, a route is re-admitted after 30s and needs three fresh failures to be excluded again. Also records that #911 remains unmerged as of this amendment. The conclusion is unchanged: a fixed wall-clock deadline on the candidate/retry loop is still barred by product-goal-directive section 8. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BV96rXhqoR3tYZ9AeAVur4

_record_success (orchestrator.py:8073-8077 at pin 414f229) pops the agent's circuit state rather than decrementing it, so with a threshold of 3 any single success zeroes the accumulated failure count. A route that alternates failure and success -- what an overloaded provider does -- never reaches the threshold and is never excluded, however long each failure runs. Recorded as a property rather than one log's reading: the same sequence appeared on three separate pull requests on 2026-09-06, each on the single ready route the post-.github#1957 preflight had found. Also corrects how far #911 is from landing. It is not merely unmerged: its base is codex/nim-evidence-successor, which is #1068 (open), itself based on codex/stacked-security-successor. "Once #911 lands" requires that whole stack. The conclusion is unchanged -- a fixed wall-clock deadline on the candidate/retry loop is still barred by product-goal-directive section 8. The count of missing pieces in the not-yet-built exclusion mechanism goes from two to three. Gate: 3392 passed, 3 failed, 2 skipped. All three failures are pre-existing and reproduce identically on origin/main (missing fast_mlsirm, missing OTLP HTTP exporter, and test_spend_analytics asserting mixed vs tokenizer); this change is documentation only. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BV96rXhqoR3tYZ9AeAVur4

seonghobae · 2026-09-07T01:11:02Z

Exact-head RCA/lifecycle correction (2026-09-07): preserved the routing-observation delta and converted e80949188f0afa86052f10f5a9b627da1ee1ef0b to Draft. Hosted Tests job 101164729708 is genuine RED: 6 failed, 3446 passed, 2 skipped. All six are preempted by _require_current_actual_cost_evidence() reading real wall time and rejecting the branch's future-dated NVIDIA evidence before the tests reach their intended malformed-contract/credential/expiry assertions. Protected main@414f22973658c4ddc3d4320fcf7acd9b4e8ba991 already owns the minimal deterministic fixture boundary: its autouse _fresh_backend pins test-only evidence dates to 2000-01-01 / 2999-12-31, leaving production expiry validation intact. Integrate that protected delta non-destructively through the parent stack; do not duplicate or weaken production validation here. Noema also failed with a 503 after 2,295.9s, OpenCode has no authenticated exact-head verdict, and compatibility CodeQL is unresolved. Fresh exact-head GREEN is required before Ready.

feat(routing): persist bounded group observations

ecc93ac

fix(routing): preserve failure ledger updates

9129931