Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
### Contextual-orchestrator pin advance fixes orchestrator/free retry-stacking

- Advanced the central sidecar's pinned immutable CO revision from `2e414d15` to protected `main@414f22973658c4ddc3d4320fcf7acd9b4e8ba991`, carrying contextual-orchestrator#1081's fix into Strix, OpenCode, and Noema. Root cause: `TaskOrchestrator._invoke`'s own retry-then-failover decision for a retryable 5xx (budgeted `1 + tool_retry_attempts` real tries per candidate) was getting multiplied by `ModelClient._send_with_retry`'s independent transient-retry-with-backoff underneath it (`max_retries + 1` further tries per call) -- up to 6 real network attempts against one already-flagged-flaky `orchestrator/free` agent before `_invoke` ever tried the next ranked candidate. Confirmed as the cause of independently observed incidents in #1912, #1231, #1503, and #1198, each spending 9-57+ minutes on one escalated route and surfacing that same route's model in its final error, never reaching a cleanly-ready sibling preflight had already found. The fix (`ModelClient.single_attempt_transport()`) changes only which agent gets tried next; no per-attempt timeout changed. Reproduced the bug directly against unmodified contextual-orchestrator `main` before the fix (6 real attempts) and confirmed the fix resolves it (<=2) before advancing this pin. `docs/adr/0003-contextual-orchestrator-vendored-free-zdr.md`'s 2026-09-06 amendment and `tests/test_contextual_orchestrator_review_sidecar_contract.py`'s `ORCH_PIN_SHA` were updated alongside this pin. All callers still consume an exact SHA; no branch or tag is introduced.

### Noema review ships sidecar evidence on failure

- `noema-review.yml` now uploads `strix_runs/contextual-orchestrator-sidecar.stderr.log` and `strix_runs/contextual-orchestrator-preflight.json` as the `noema-sidecar-evidence` artifact when the verdict phase fails (`if: failure()`, the same pinned `actions/upload-artifact` Strix uses, `if-no-files-found: ignore`, 5-day retention). Until now a failed Noema run left `artifacts=0` -- run `33981136873` spent 3122 s walking six ready routes twice each and ended in HTTP 502 with no per-route trace anywhere but the sidecar's stderr -- so the only diagnosis available was the caller's one-line summary. The stderr file is the sanitizer's bounded allowlist output (`sanitize_contextual_orchestrator_sidecar_stream.py`), the same file Strix already publishes in `strix-reports`; per-attempt route outcomes still need an allowlisted structured line from the orchestrator to appear in it. Refs #1935, #1939.
Expand Down
25 changes: 24 additions & 1 deletion docs/adr/0003-contextual-orchestrator-vendored-free-zdr.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ all five, and auto-optimize routing by cost.

1. **Vendoring, pinned**: `scripts/ci/contextual_orchestrator_review_sidecar.sh`
clones `ContextualWisdomLab/contextual-orchestrator` at an exact SHA
(`2e414d15ba58f28597751b625a8a2f00fc9fadcf` today) into `RUNNER_TEMP`. The
(`414f22973658c4ddc3d4320fcf7acd9b4e8ba991` today) into `RUNNER_TEMP`. The
source's `requirements.lock` is installed with `--require-hashes` and
`--no-deps`, so dependency resolution cannot silently move the reviewed
runtime.
Expand Down Expand Up @@ -256,3 +256,26 @@ all five, and auto-optimize routing by cost.
fault. Accepted-size and tool-schema probes call the pinned client's
deterministic mock response explicitly and therefore perform no provider
call.
- **2026-09-06 amendment: advance the governed runtime pin to fix
`orchestrator/free` retry-stacking.** The vendored pin advances from
`2e414d15ba58f28597751b625a8a2f00fc9fadcf` to
`414f22973658c4ddc3d4320fcf7acd9b4e8ba991`, the commit that merges
`contextual-orchestrator#1081`. That PR fixes `TaskOrchestrator._invoke`'s
per-agent retry-then-failover decision (`RETRY_SAME_AGENT` for a retryable
5xx, budgeted at `1 + tool_retry_attempts` real tries per candidate) getting
multiplied by `ModelClient._send_with_retry`'s own, independent
transient-retry-with-backoff loop underneath it (`max_retries + 1` further
tries per call) — up to `(tool_retry_attempts + 1) × (max_retries + 1)` real
network attempts (6 at production defaults) against one already-flagged-flaky
`orchestrator/free` agent before `_invoke` ever tried the next ranked
candidate. This is the confirmed root cause of independently observed
incidents in `ContextualWisdomLab/.github` PRs #1912, #1231, #1503, and
#1198, each spending 9–57+ minutes on one escalated route and surfacing that
same route's model in its final error, never reaching a cleanly-ready
sibling preflight had already found. The fix adds
`ModelClient.single_attempt_transport()` (a thread-local context manager
mirroring the existing `request_settings()` pattern) that forces
`_send_with_retry`'s retry budget to 0 for the duration of `_invoke`'s own
per-agent attempt; it changes only *which* agent gets tried next, never any
per-attempt timeout, consistent with the 2026-08-31 amendment above. No
other contextual-orchestrator behavior changes with this pin advance.
2 changes: 1 addition & 1 deletion scripts/ci/contextual_orchestrator_review_sidecar.sh
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@
# (fail-closed zero-cost) pool.
set -euo pipefail

ORCHESTRATOR_PIN_SHA="${ORCHESTRATOR_PIN_SHA:-2e414d15ba58f28597751b625a8a2f00fc9fadcf}"
ORCHESTRATOR_PIN_SHA="${ORCHESTRATOR_PIN_SHA:-414f22973658c4ddc3d4320fcf7acd9b4e8ba991}"
ORCHESTRATOR_GIT_URL="${ORCHESTRATOR_GIT_URL:-https://github.com/ContextualWisdomLab/contextual-orchestrator.git}"
# The Strix gate and Noema SSRF guard accept this one process-local origin.
# Keep it fixed so an environment override cannot create an unvalidated sidecar.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@
)

GATEWAY_MODEL = "contextual-orchestrator/orchestrator/free"
ORCH_PIN_SHA = "2e414d15ba58f28597751b625a8a2f00fc9fadcf"
ORCH_PIN_SHA = "414f22973658c4ddc3d4320fcf7acd9b4e8ba991"


def _read(path: Path) -> str:
Expand Down
Loading