From 4913d0aa37b8cf768fcf572dbb18eb5b778c187c Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 30 Aug 2026 04:08:42 +0000 Subject: [PATCH 1/2] fix(sidecar): refresh stale contextual-orchestrator review pin scripts/ci/contextual_orchestrator_review_sidecar.sh's ORCHESTRATOR_PIN_SHA default (b21645116b352967e50fc497b87eb745b9cc8c61) was 103 commits behind contextual-orchestrator main. Observed directly in hosted noema-review job logs (.github PR #1421, contextual-orchestrator PR #857 and others): the vendored sidecar's own preflight against the stale pin fails closed with "gateway preflight returned HTTP 502" (and, on a differently-shaped request, "request_failed status=413 code=request_too_large") before the model pool can run, so opencode-agent/Noema never post a verdict and the required opencode-review/noema-review checks fail on unrelated PRs across both repos. Bumps the pin to 5f2753ace756ddd81049a5221d55e8977572a416 (current contextual-orchestrator main HEAD, confirmed green on Tests/Security/Fuzz) in the three places the contract tests pin it: the sidecar script default, tests/test_contextual_orchestrator_review_sidecar_contract.py's ORCH_PIN_SHA, and docs/adr/0003-contextual-orchestrator-vendored-free-zdr.md's "today" reference. requirements.lock needs no separate sync -- the sidecar installs it fresh from the freshly-checked-out pinned commit, not from a copy embedded in this repo. Also ports the still-unmerged #1414 fix (remove the redundant step-level if: github.event_name guard inside required-workflow-bootstrap, which the bootstrap contract correctly rejects) so this PR's own exact-head-path-policy check isn't blocked by that separate, already-diagnosed, pre-existing gap. This is the second sidecar-pin staleness incident in as many days (see the 2026-08-29 gap-baseline entry for the first). Documented as a gap-baseline entry recommending a scheduled pin-freshness check as a follow-up, since nothing currently keeps this pin near contextual-orchestrator main on an ongoing basis. Full local suite green: 1873 passed / 1 skipped (pytest), full test_strix_quick_gate.sh PASS. Live gateway behavior at the new pin can only be confirmed by a post-merge hosted noema-review/opencode-review run. Co-Authored-By: Claude Claude-Session: https://claude.ai/code/session_015Gs7KmNvH75nxz1sL8mKjw --- .github/workflows/opencode-review.yml | 1 - CHANGELOG.md | 11 ++++++ ...ntextual-orchestrator-vendored-free-zdr.md | 2 +- docs/product-technical-gap-baseline.md | 35 +++++++++++++++++++ .../contextual_orchestrator_review_sidecar.sh | 2 +- ...al_orchestrator_review_sidecar_contract.py | 2 +- tests/test_pingora_edge_workflow_contract.py | 3 +- 7 files changed, 51 insertions(+), 5 deletions(-) diff --git a/.github/workflows/opencode-review.yml b/.github/workflows/opencode-review.yml index f3e3c24996..d66979d406 100644 --- a/.github/workflows/opencode-review.yml +++ b/.github/workflows/opencode-review.yml @@ -197,7 +197,6 @@ jobs: fi - name: Enforce Cloudflare Pingora edge policy - if: ${{ github.event_name == 'pull_request_target' }} env: GITHUB_TOKEN: ${{ github.token }} TARGET_REPOSITORY: ${{ github.event.pull_request.base.repo.full_name || github.repository }} diff --git a/CHANGELOG.md b/CHANGELOG.md index 3eab104fc2..7755e7f1ed 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,17 @@ this file. The format follows Keep a Changelog, and versioned releases follow Semantic Versioning where the repository publishes a release. ## [Unreleased] +- Keep the required OpenCode bootstrap's Pingora policy step unconditional + within its pull-request-only workflow, so the static bootstrap contract does + not depend on event payload fields. (Ported from #1414, not yet merged, to + unblock this PR's own `exact-head-path-policy` check.) +- Bump the vendored `contextual-orchestrator` review-sidecar pin from + `b2164511` (103 commits stale) to current `main` `5f2753a`, so the + gateway's model-discovery/ZDR/pool-selection fixes landed since the old pin + reach `opencode-review`/`noema-review`. The stale pin's discovery logic was + failing the sidecar's own preflight with a gateway 502 before any review + could post, which is why `opencode-review` and `noema-review` were failing + closed on most `contextual-orchestrator` PRs and several `.github` PRs. - Skip trusted base Python lock materialization for exact-head reviews with no Python source or dependency-manifest changes, while preserving the fail-closed wheel-only path when Python coverage is relevant. diff --git a/docs/adr/0003-contextual-orchestrator-vendored-free-zdr.md b/docs/adr/0003-contextual-orchestrator-vendored-free-zdr.md index 3e40886e5f..f1f5883505 100644 --- a/docs/adr/0003-contextual-orchestrator-vendored-free-zdr.md +++ b/docs/adr/0003-contextual-orchestrator-vendored-free-zdr.md @@ -22,7 +22,7 @@ all five, and auto-optimize routing by cost. 1. **Vendoring, pinned**: `scripts/ci/contextual_orchestrator_review_sidecar.sh` clones `ContextualWisdomLab/contextual-orchestrator` at an exact SHA - (`b21645116b352967e50fc497b87eb745b9cc8c61` today) into `RUNNER_TEMP`. The + (`5f2753ace756ddd81049a5221d55e8977572a416` today) into `RUNNER_TEMP`. The source's `requirements.lock` is installed with `--require-hashes` and `--no-deps`, so dependency resolution cannot silently move the reviewed runtime. diff --git a/docs/product-technical-gap-baseline.md b/docs/product-technical-gap-baseline.md index 1072e8b397..e40d09682f 100644 --- a/docs/product-technical-gap-baseline.md +++ b/docs/product-technical-gap-baseline.md @@ -396,6 +396,41 @@ flowchart LR or typed provider result. A green event handler that skips the LLM call is not acceptance evidence. +## 2026-08-30 sidecar pin staleness recurrence + +- Same class of defect as the 2026-08-29 entry above recurred within one day: + `scripts/ci/contextual_orchestrator_review_sidecar.sh`'s + `ORCHESTRATOR_PIN_SHA` default (`b21645116b352967e50fc497b87eb745b9cc8c61`) + was already 103 commits behind `contextual-orchestrator` `main`. Observed + directly in hosted `noema-review` job logs (`.github` PR #1421, + `ContextualWisdomLab/contextual-orchestrator` PR #857 and others): the + vendored sidecar's own preflight against the stale pin fails closed with + `gateway preflight returned HTTP 502` (and, on a differently-shaped request, + `request_failed status=413 code=request_too_large`) before the model pool + can run, so `opencode-agent`/Noema never post a verdict and the required + `opencode-review`/`noema-review` checks fail on unrelated PRs across both + repos. Confirmed via `contextual-orchestrator` main history that + `5f2753ace756ddd81049a5221d55e8977572a416` is the current `main` HEAD and + passes its own Tests/Security/Fuzz gates. +- This PR bumps the pin to `5f2753ace756ddd81049a5221d55e8977572a416` in the + three places the contract tests pin it: the sidecar script default, + `tests/test_contextual_orchestrator_review_sidecar_contract.py`'s + `ORCH_PIN_SHA`, and `docs/adr/0003-contextual-orchestrator-vendored-free-zdr.md`'s + "today" reference. `requirements.lock` needs no separate sync — the sidecar + installs it fresh from the freshly-checked-out pinned commit, not from a + copy embedded in this repo. +- Acceptance remains open the same way the 2026-08-29 entry describes: this + fixes the reproduced local preflight failure and all static contract tests + pass, but only a fresh post-merge hosted `noema-review`/`opencode-review` + run against the new pin is proof the live gateway path actually completes + and posts a verdict. Given this is the second staleness incident in as many + days, the underlying gap is process, not just this one value: nothing + currently keeps this pin near `contextual-orchestrator` `main` on an + ongoing basis. A scheduled or CI-triggered pin-freshness check (e.g., fail + a nightly job once the pin falls more than N commits or M days behind a + green `contextual-orchestrator` main) would close that gap; not implemented + in this PR, left for a follow-up. + ## 5. 실행 루프와 고객의 다음 행동 각 hourly pass는 아래 순서를 유지한다. diff --git a/scripts/ci/contextual_orchestrator_review_sidecar.sh b/scripts/ci/contextual_orchestrator_review_sidecar.sh index bca9d5c00e..1fb2b5aea3 100755 --- a/scripts/ci/contextual_orchestrator_review_sidecar.sh +++ b/scripts/ci/contextual_orchestrator_review_sidecar.sh @@ -14,7 +14,7 @@ # (fail-closed zero-cost) pool. set -euo pipefail -ORCHESTRATOR_PIN_SHA="${ORCHESTRATOR_PIN_SHA:-b21645116b352967e50fc497b87eb745b9cc8c61}" +ORCHESTRATOR_PIN_SHA="${ORCHESTRATOR_PIN_SHA:-5f2753ace756ddd81049a5221d55e8977572a416}" ORCHESTRATOR_GIT_URL="${ORCHESTRATOR_GIT_URL:-https://github.com/ContextualWisdomLab/contextual-orchestrator.git}" # The Strix gate and Noema SSRF guard accept this one process-local origin. # Keep it fixed so an environment override cannot create an unvalidated sidecar. diff --git a/tests/test_contextual_orchestrator_review_sidecar_contract.py b/tests/test_contextual_orchestrator_review_sidecar_contract.py index a1c0746f03..fc7c3b4794 100644 --- a/tests/test_contextual_orchestrator_review_sidecar_contract.py +++ b/tests/test_contextual_orchestrator_review_sidecar_contract.py @@ -40,7 +40,7 @@ ) GATEWAY_MODEL = "contextual-orchestrator/orchestrator/free" -ORCH_PIN_SHA = "b21645116b352967e50fc497b87eb745b9cc8c61" +ORCH_PIN_SHA = "5f2753ace756ddd81049a5221d55e8977572a416" def _read(path: Path) -> str: diff --git a/tests/test_pingora_edge_workflow_contract.py b/tests/test_pingora_edge_workflow_contract.py index 82a85986a0..ad0667cc6b 100644 --- a/tests/test_pingora_edge_workflow_contract.py +++ b/tests/test_pingora_edge_workflow_contract.py @@ -40,7 +40,8 @@ def test_required_workflow_enforces_pingora_without_executing_pr_content() -> No assert '[ -L "$trusted_source_dir/$EXPECTED_FILE" ]' in text assert '[ ! -f "$trusted_source_dir/scripts/ci/pingora_edge_policy.py" ]' in text assert '[ -L "$trusted_source_dir/scripts/ci/pingora_edge_policy.py" ]' in text - assert "if: ${{ github.event_name == 'pull_request_target' }}" in text + assert "pull_request_target:" in text + assert "if: ${{ github.event_name == 'pull_request_target' }}" not in text assert text.index("Verify immutable central policy source") < text.index( "Enforce Cloudflare Pingora edge policy" ) From 6e3c55a7f341ef5d8be85e807dfc39fb907443b6 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 30 Aug 2026 04:28:34 +0000 Subject: [PATCH 2/2] fix(sidecar): exclude evidence-only discovery rows from live selection Devin Review flagged this on PR #1422: the new pin's contextual_orchestrator discover_all_models() marks the entire OpenRouter catalog evidence_only=True (added since the old b2164511 pin) -- by contextual-orchestrator's own design, those rows exist solely to supply ZDR evidence for other providers' models and must never become live inference agents; its own agent_from_discovered() raises ValueError rather than serve one, and is_routable_discovered_model() excludes them. scripts/ci/contextual_orchestrator_review_launcher.py builds its own catalog independently (never calling agent_from_discovered()), and its selection loop had no evidence_only check anywhere in its discovery-to-catalog path, so an evidence-only OpenRouter row could reach this review sidecar's live serving catalog once the pin advances far enough to introduce the field. Adds _routable_discovered_models() (mirroring contextual-orchestrator's own filtering semantics) and applies it before both free-route detection and pool selection. Adds a direct unit test proving the exclusion, and confirmed it fails with the exact reported behavior before this fix. Updates the sidecar contract test's pinned wiring strings to match. Full local suite green: 1874 passed / 1 skipped (pytest, +1 new test); full test_strix_quick_gate.sh PASS. Co-Authored-By: Claude Claude-Session: https://claude.ai/code/session_015Gs7KmNvH75nxz1sL8mKjw --- ...contextual_orchestrator_review_launcher.py | 18 +++++++++-- ...l_orchestrator_review_runtime_preflight.py | 30 +++++++++++++++++++ ...al_orchestrator_review_sidecar_contract.py | 4 ++- 3 files changed, 49 insertions(+), 3 deletions(-) diff --git a/scripts/ci/contextual_orchestrator_review_launcher.py b/scripts/ci/contextual_orchestrator_review_launcher.py index abb9af3b21..aa61ea1c80 100644 --- a/scripts/ci/contextual_orchestrator_review_launcher.py +++ b/scripts/ci/contextual_orchestrator_review_launcher.py @@ -65,6 +65,19 @@ def _has_text_output(model: object) -> bool: return not modalities or "text" in {str(modality).casefold() for modality in modalities} +def _routable_discovered_models(discovered: list[object] | None) -> list[object]: + """Drop evidence-only discovery rows before any live-serving selection. + + Evidence-only rows (e.g. the OpenRouter catalog) exist solely to supply + ZDR evidence for other providers' models; contextual_orchestrator's own + ``agent_from_discovered()`` refuses to turn one into a serving agent. + Filtering here keeps that same invariant in this sidecar's selection path, + which builds its catalog independently rather than calling + ``agent_from_discovered()`` directly. + """ + return [model for model in (discovered or []) if not getattr(model, "evidence_only", False)] + + def _route_identity(model: object) -> tuple[str, str]: """Return the provider/model identity used to bind price evidence.""" @@ -385,10 +398,11 @@ def main(argv: list[str] | None = None) -> int: discovered, _ = discover_all_models() except Exception as exc: # pragma: no cover - provider/networking failure is runtime-only raise SystemExit(f"review sidecar discovery failed: {exc}") from exc - free_models = list(free_discovered_models(discovered)) if discovered else [] + routable_discovered = _routable_discovered_models(discovered) + free_models = list(free_discovered_models(routable_discovered)) if routable_discovered else [] free_route_identities = frozenset(_route_identity(model) for model in free_models) selected_models = [] - for model in discovered or []: + for model in routable_discovered: model_id = getattr(model, "model_id", "") if not is_general_chat_agent_model_id(model_id) or not _has_text_output(model): continue diff --git a/tests/test_contextual_orchestrator_review_runtime_preflight.py b/tests/test_contextual_orchestrator_review_runtime_preflight.py index baa1a8df76..a0fbac1f4a 100644 --- a/tests/test_contextual_orchestrator_review_runtime_preflight.py +++ b/tests/test_contextual_orchestrator_review_runtime_preflight.py @@ -52,6 +52,36 @@ def _openai_text(content: str) -> dict[str, object]: return {"choices": [{"message": {"content": content}}]} +def test_routable_discovered_models_excludes_evidence_only_rows() -> None: + """Evidence-only rows (e.g. OpenRouter) must never enter live selection.""" + namespace = _load_launcher() + routable = namespace.get("_routable_discovered_models") + assert callable(routable), "launcher must expose an evidence-only discovery filter" + + evidence_only_model = SimpleNamespace( + id="openrouter_evidence_only", + provider_name="openrouter", + model_id="some/model", + evidence_only=True, + ) + live_model = SimpleNamespace( + id="nvidia_ready", + provider_name="nvidia_nim", + model_id="ready/free", + evidence_only=False, + ) + no_flag_model = SimpleNamespace( + id="bytez_untagged", provider_name="bytez", model_id="untagged/free" + ) + + assert routable([evidence_only_model, live_model, no_flag_model]) == [ + live_model, + no_flag_model, + ] + assert routable(None) == [] + assert routable([]) == [] + + def test_preflight_mirrors_runtime_request_and_keeps_only_compatible_routes() -> None: """Reject provider errors/malformed replies before the sidecar becomes ready.""" namespace = _load_launcher() diff --git a/tests/test_contextual_orchestrator_review_sidecar_contract.py b/tests/test_contextual_orchestrator_review_sidecar_contract.py index fc7c3b4794..3ddf8952d2 100644 --- a/tests/test_contextual_orchestrator_review_sidecar_contract.py +++ b/tests/test_contextual_orchestrator_review_sidecar_contract.py @@ -281,7 +281,9 @@ def test_launcher_uses_orchestrator_discovery_and_governed_pools() -> None: text = _read(LAUNCHER) assert "from contextual_orchestrator.chat_capability import is_general_chat_agent_model_id" in text assert "from contextual_orchestrator.model_discovery import discover_all_models, free_discovered_models" in text - assert "free_discovered_models(discovered)" in text + assert "routable_discovered = _routable_discovered_models(discovered)" in text + assert "free_discovered_models(routable_discovered)" in text + assert 'getattr(model, "evidence_only", False)' in text assert 'getattr(model, "output_modalities", None)' in text assert 'isinstance(modalities, str)' in text assert '"text" in {str(modality).casefold() for modality in modalities}' in text