docs(orchestrator): confirm orchestrator/free pin, audit gateway enforcement

[seonghobae]

Summary

Scope note (2026-09-03, added after this PR grew well past its original doc-only scope — flagged by Devin's automated review): this PR started as the doc-only orchestrator/free audit below, but has since accumulated substantial workflow/scripts/tests changes across many follow-up commits, primarily the noema-review.yml Actions concurrency-deadlock fix and its dependents. The bullets immediately below are the original scope; the ones after "Since then" reflect what actually landed.

• Verified with actual GitHub Actions run evidence (not just reading YAML/scripts) that OpenCode Review and Strix genuinely invoke the vendored contextual-orchestrator gateway through orchestrator/free, and that a job correctly fails closed (not silently passes) when the gateway/scanner connection itself fails. Evidence: docs/doctoring/contextual-orchestrator-gateway-enforcement-audit-20260902.md.
• Recorded the repo owner's explicit, in-session confirmation (2026-09-02) that orchestrator/free is the intended pin for both OpenCode and Strix across all GitHub Actions usage, closing ADR-0003's 2026-08-31 "open, unreviewed risk" note without erasing that history (docs/adr/0003-contextual-orchestrator-vendored-free-zdr.md, docs/doctoring/contextual-orchestrator-strix-free-diversity-evidence.md, docs/product-goal-directive.md §8).
• Documented (not fixed — this is a branch-protection/security-configuration change, outside this PR's scope) a real, previously-unrecorded gap found while tracing how review evidence reaches a merge decision: .github's own main branch protection omits strix from required_status_checks and has enforce_admins disabled, so an admin-credentialed merge can bypass the review gate entirely — cross-checked against PR fix(strix): remove the 300s LLM_TIMEOUT cap #1658's actual merge. A follow-up task chip was spawned for the owner to review separately.
• PR fix(sidecar): discover Bytez free models and suppress expected 413 #1651 (Bytez discovery sidecar fix) had already merged by the time this session reached it; no action was needed there.

Since then:

• noema-review.yml: fixed an Actions concurrency deadlock — the cleanup logic that cancels superseded runs was originally a step inside the same job-level concurrency group it needed to unblock, so a long-running older-head review (Noema inference has no wall-clock deadline by design) could block a current head's review from ever starting. Extracted cancel-superseded-noema-runs into its own job with no concurrency restriction of its own (mirroring strix.yml/opencode-review.yml's existing shape), scoped the noema-review job's own group by PR number with cancel-in-progress: false (not head SHA — see the job's own extensive in-file comment for the capacity-vs-correctness tradeoff reasoning), and moved actions: write off the review job.
• noema-review.yml (follow-up, 2026-09-03): Devin's review on this same PR found cancel-superseded-noema-runs never ran for repository_dispatch retries even though they share the review job's concurrency group with pull_request_target pushes, so a retry could get stuck behind a stale run with nothing able to cancel it — extended the job's if:/env: to cover both trigger types. Also found (and this pass only partially closes, see docs/doctoring/noema-review-repository-dispatch-cleanup-and-pending-slot-eviction-20260903.md for the full analysis) that GitHub's own single-pending-slot rule can let a delayed, out-of-order trigger for an older head evict the current head's own pending job before it ever runs — added a detection step (not full auto-recovery, which would need either a permission grant this repo's conventions caution against or a broader multi-file trigger redesign) that flags this loudly when it happens. That detection step's first draft was itself run through 3-lens adversarial verification, which caught a real false-positive bug (it couldn't distinguish a genuine eviction from the two routine, non-buggy cancellation paths already in this file) — fixed before this PR, see the doctoring record for the full before/after.
• pr-review-merge-scheduler.yml: carved the in-flight manual org-sweep walk out of the scheduler's cancel-in-progress expression so a second manual dispatch can't cancel an already-spent partial walk.
• hourly-nvidia-nim-review-repair.yml: added a CI-tooling hash-lock verification step, then hardened it twice more on later Devin passes (exact whole-line match instead of a substring search that could match the wrong pinned entry; strips inline comments/environment markers before matching).
• scripts/ci/test_strix_quick_gate.sh: updated a stale assertion that no longer matched the scheduler's actual (and correct) cancel-in-progress expression after the org-sweep carve-out above — this quick gate was failing on every run until fixed here.
• scripts/ci/noema_review_gate.py + associated tests: SSRF hardening and repair-attempt telemetry for the Noema review gateway path.
• docs/product-technical-gap-baseline.md: extensive running updates tracking all of the above plus the standing cross-repo review/fix/merge loop's findings.

Test plan

• PYTHONPATH=. python3 -m pytest tests/test_noema_review_gate.py tests/test_strix_required_smoke_availability.py tests/test_contextual_orchestrator_review_sidecar_contract.py tests/test_strix_contextual_orchestrator_contract.py -q — 152 passed (these are the tests that reference the edited ADR/doctoring files from the original doc-only scope).
• grep-verified no test pins the exact modified prose (only substring/file-existence checks against the ADR).
• Full suite (PYTHONPATH=. python3 -m pytest tests -q) passes on the current head — 2730 passed, 1 skipped, 21 subtests passed.
• coverage run -m pytest tests && coverage report --show-missing — 100% statement and branch coverage (12,225 statements, 4,958 branches).
• interrogate — 100% docstring coverage.
• scripts/ci/test_strix_quick_gate.sh (the required quick gate) — PASS on the current head.
• Every run: block across all three jobs in noema-review.yml (12 steps) independently extracted via PyYAML and checked with bash -n — all pass.
• The new pending-slot-eviction detection step's jq logic manually verified against synthetic Jobs-API fixtures for all four cases (evicted, legitimately-superseded, no-matching-job, still-pending) — each resolves through the correct branch.
• 3-lens adversarial verification (Workflow, one independent agent each on GitHub Actions platform semantics, bash/YAML correctness, and completeness against the original Devin findings) run against the noema-review.yml diff before this push; the one real bug it found (see above) is fixed and re-verified.

🤖 Generated with Claude Code

---

Summary by CodeRabbit

• 문서

  • OpenCode와 Strix의 orchestrator/free 사용 결정 및 잔여 가용성 위험을 최신 상태로 반영했습니다.
  • 게이트웨이 강제 적용, 실제 실행 감사 결과와 CI 동시성·중복 워크플로 조사 내용을 문서화했습니다.
  • 롤아웃 계획과 감사 기록을 확인된 근거에 맞게 정정하고 Markdown 제목 표시 오류를 수정했습니다.

• 개선

  • CI 도구 잠금 파일 변경 시 관련 검토가 실행되도록 조건을 보완했습니다.
  • 고정된 도구 버전과 잠금 정보의 일치를 자동 검증합니다.
  • 외부 게이트웨이 연결 시 안전하지 않은 주소를 차단하고, DNS 확인 실패도 안전하게 거부합니다.
  • 관련 동작을 검증하는 자동화 테스트를 추가했습니다.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

📝 Walkthrough

Walkthrough

OpenCode와 Strix의 orchestrator/free 설정과 운영 감사 기록을 갱신했습니다. Actions requirements 계약 검증을 추가했습니다. Noema 게이트웨이는 DNS 결과를 검증하고 고정 IP로 연결하도록 변경했습니다.

Changes

오케스트레이터 운영 및 감사 기록

Layer / File(s)	Summary
무료 풀 결정 기록 AGENTS.md, docs/adr/..., docs/doctoring/contextual-orchestrator-strix-free-diversity-evidence.md, docs/product-goal-directive.md	OpenCode, Noema, Strix의 orchestrator/free 설정과 소유자 확인을 기록했습니다. free_account_diversity를 모니터링 지표로 유지했습니다.
게이트웨이 강제 감사 docs/doctoring/contextual-orchestrator-gateway-enforcement-audit-20260902.md	Strix의 게이트웨이 호출, 로컬 strix-agent 실행, fail-closed 처리, OpenCode 경로 추론과 branch protection 상태를 기록했습니다.
운영 정책 및 수리 기록 docs/product-technical-gap-baseline.md	repair-not-close 정책, timeout 설계, workflow 통합, Actions 큐 분석, 토큰 감사와 Markdown 헤딩 수정을 기록했습니다.
Strix 교차 PR 동시성 기아 기록 docs/doctoring/strix-cross-pr-concurrency-starvation-20260902.md	교차 PR 축출, 취소 증거, check-then-act 경쟁 및 후속 단계를 기록했습니다.

Requirements 계약 검증

Layer / File(s)	Summary
Requirements 계약 검증 .github/workflows/hourly-nvidia-nim-review-repair.yml, tests/test_pr_review_fix_hourly_contract.py	두 requirements 파일을 workflow 트리거에 추가했습니다. 정확한 == 핀이 hash lock에 있는지 contract 잡과 테스트에서 검증합니다.

Noema 고정 IP 전송

Layer / File(s)	Summary
고정 IP 전송 구현 및 검증 scripts/ci/noema_review_gate.py, tests/test_noema_repair_attempt_telemetry.py, tests/test_noema_review_gate.py, tests/test_noema_review_orchestrator_ssrf.py	DNS 실패와 사용 가능한 IP 부재를 fail-closed로 처리합니다. 검증된 공인 IP로 HTTP 및 HTTPS 연결을 수행하고, TLS SNI와 Host 헤더를 유지하며, 다중 IP 폴백을 검증합니다.

Estimated code review effort: 4 (Complex) | ~60 minutes

Merge Risk: 🟠 High · up to 539e9

The PR changes the CI review gateway’s outbound transport and destination validation. It still permits bearer credentials and review data over plain HTTP, allows destinations in 100.64.0.0/10, and can defer enforcement to unverified proxy policy, creating concrete security risk for privileged CI requests; these issues should be fixed or explicitly accepted before merge.

Sequence Diagram(s)

sequenceDiagram
  participant call_llm
  participant reject_private_llm_url
  participant PinnedHTTPHandlers
  participant LLMGateway
  call_llm->>reject_private_llm_url: URL 검증 및 공인 IP 확인
  reject_private_llm_url-->>call_llm: pinned_ips 반환
  call_llm->>PinnedHTTPHandlers: 고정 연결 handler로 opener 생성
  PinnedHTTPHandlers->>LLMGateway: 원래 host를 유지하고 고정 IP로 연결
  LLMGateway-->>call_llm: HTTP 응답 반환

Loading

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 71.74% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 46 functions across 5 files. (2 skipped: …	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	제목은 orchestrator/free 핀 확인과 게이트웨이 강제 감사라는 PR의 주요 문서화 목적을 정확하고 간결하게 요약합니다.

Full details: Docstring Coverage

Explanation

Docstring coverage is 71.74% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 46 functions across 5 files. (2 skipped: 2 unsupported.)

✨ Finishing Touches 💡 1

📝 Generate docstrings 💡

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch claude/contextual-orchestrator-integration-8ec7f8

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[seonghobae]

ConceptWeave owner-path coordination for .github#1924: this PR is the current active writer touching scripts/ci/noema_review_gate.py and its tests, so I am not opening a competing central repair. Consumer evidence is unchanged ContextualWisdomLab/ConceptWeave#35@a31ae0c2df920f2794f7ddb456795b04797ab472: Noema review 5120903874 emitted REQUEST_CHANGES because it asserted cargo generate-lockfile --locked is unsupported, while the current Cargo command reference explicitly documents --locked for cargo generate-lockfile. Please fold #1924's generic regression into this existing Noema writer if it remains cohesive: an external CLI/library/API capability finding contradicted by supplied authoritative primary/runtime evidence must not be promoted to REQUEST_CHANGES; absent authoritative evidence must remain inconclusive/fail-closed, not auto-approved. Preserve exact-head binding, provenance, orchestrator/free, and adversarial-review requirements. This is coordination only; no source write, review dismissal, or leaf workaround was made.

[seonghobae]

Status of the four failing checks on head 24005da6 (host 1 session, 2026-09-05 ~21:40Z). All nine runs on this head were created at 17:30:21Z, after #1939 merged, so they ran the current sidecar.

CodeQL compatibility analysis (actions / python) — blocked upstream, not by this tree. The dispatched scan cannot obtain a verdict while OPENCODE_REPOSITORY_DISPATCH_ACTOR holds a single identity (#1929; 71 of 71 dispatches since 13:57Z rejected at the actor gate). Nothing in this PR changes that outcome.

noema-review — served, then rejected by the local validator. Sidecar policy summary: free_pool_admitted_routes 62, selected 4 nvidia_nim + 4 nvidia_nim_sub + 4 openrouter (the #1939 split), preflight ready 2/12 — both ready routes OpenRouter (all eight NVIDIA routes rejected: four 404, four no-status). The review was served by dots-studio/dots-3-note-preview:free after three escalations and failed local validation because a cited line entry was an array index, not a source line. That is a model-output defect, caller attempts=1, so the job has been re-run once for a second sample.

strix — three CRITICAL findings, none bound to the tree. All three cite scripts/ci/noema_review_gate.py at lines 3018, 3244-3249, 3256, 3300, 3400; the file on this head has 2062 lines (main: 1828), so every cited location is past end-of-file, and each PoC is a # Simulate… sketch, not an executed probe. Against the real code:

• Command injection via gh api arguments — the module's only subprocess entry is run() (:282-300): argv list, shell=False, and a TypeError on a string command; every gh call passes a list (:313, :483-484, :497, :730/765/786, :1986). No shell is ever reached, and author logins come from GitHub's own API response.
• Token exposure in error messages — run() passes stderr through scrub_sensitive_data (:296; patterns at :264-270 mask token …, Authorization: Bearer …, api_key=…) before it can appear in an exception.
• Path traversal in parse_diff_path (:558-569) — the decoded path is only added to an in-memory locations set (:545, :550) for allowed-location matching; the module contains no open()/Path()/os.path sink on it.

The gate closed on the Vulnerabilities 3 marker alone (strix.yml:957, exit 1 at :1012); it has no check that a finding's path:line exists in the scanned tree, which is filed separately as a gate gap. The strix job has been re-run once for a fresh scan of the same head. #1661 itself does change this file (+246/−12), which is why the finding "intersects" — the delta is the allowed-locations/prefix work, not any of the three claimed sinks.

[seonghobae]

noema-review on 57e48484 (run 33995553859, job 101392181634) failed after 3873 s with a 502 from the gateway after walking two silent NVIDIA deepseek-flash routes for 13 rounds of three 90 s timeouts, three times over; the full per-route trace is in the run's noema-sidecar-evidence artifact and tabulated on #1939. Nothing in this PR's tree is involved — the same head's preflight found 6 ready routes and the selection was 4+4+4 — so there is no fix to push here; it is the provider-side condition tracked in #1948 and contextual-orchestrator#1045. Not re-running now: a re-run into the same hour would repeat the 64-minute walk.

[seonghobae]

Verified by running it. Scoped to head c92434a8 — this approval does not extend past that sha. Both axes raised in the request check out, and the merged tree loses nothing from main.

Gate on the merged tree

check	result
merge-tree vs main	clean, 0 conflict markers outside tests/
pytest	3012 passed, 1 skipped, 21 subtests
coverage (scripts/ci)	100%, 0 missed of 13235
interrogate	100%
collected node-ids	3013, 0 duplicates, 2434 distinct function names

The merged tree against main is 24 files, +4837/−191, and all 191 deleted lines belong to files this change intentionally rewrites. Nothing from main is dropped. Note the raw main…head diff reads 48 files and −1161 because the branch predates today's merges; that figure is not a loss, and the merged tree is the one to read.

Axis (a): the rewritten noema-review.yml keeps today's concurrency contract

The file gains 536 lines and loses 79, and it is asserted from several places in the queue contract. In the merged tree it still parses to a workflow-level group with cancel-in-progress: True, and the contract is live rather than merely present — flipping the flag behind a comment, with yaml.safe_load confirming the parsed value really became False, fails three tests:

test_required_pull_request_workflows_cancel_superseded_runs      <- today's contract, #1979
test_noema_concurrency_and_live_head_cleanup_preserve_current_review
test_noema_triggers_preserve_standalone_pull_request_review

Today's other two contracts also survive by name: test_pr_keyed_scan_workflows_pin_cancellation_as_a_value (#1980) and test_central_run_filter_accepts_the_run_name_github_actually_sends (#1983), plus auditable_repositories (#1989).

Axis (b): the new gate code is covered by execution, not fixtures

noema_review_gate.py gains 246 lines of SSRF hardening — DNS pinning, scheme enforcement, proxy detection. 100% line coverage says those lines ran; it does not say the guards are load-bearing. Deleting each guard so it can never fire:

guard removed	result
scheme must be https	caught, 2 failed
pinned-IP membership filter	caught, 28 failed
"no usable IP" refusal	caught, 2 failed

Each was restored and the tree re-checked clean. The coverage on this module is real: the tests fail when the security behaviour is removed, which is the property that matters and the one line coverage cannot show.

On the count prediction

The def test_ count under-predicted (3003 vs 3012) because four parameterized cases and subtests make passed diverge from collected. That is a real limit of the arithmetic check — it holds for plain function additions only. The duplicate detection it was standing in for is done above directly: 3013 collected, zero duplicate node-ids.

Merge eligibility

I am not the merger here and do not decide it, but recording what I measured: this change is BEHIND under strict protection, and its content is the noema review gate, its workflow, and documentation. It does not repair the queue blockage, so it does not satisfy the narrower "fixes the thing blocking it" reading. The author reached the same conclusion and declined to rule on their own pull request, which is the right call. My evidence establishes readiness, not eligibility.

🤖 Addressed by Claude Code

[seonghobae]

Exact-head CodeQL recovery receipt for c92434a83f570b5374cfcc439807e3f76a6b0956:

• run 34032093067 had terminal compatibility jobs whose log ended with DISPATCH_OUTCOME=success, VERDICT_STATE=pending; this was the intentional first-pass runner-release protocol, not an analysis failure;
• after the repository dispatch-actor authorization repair, only the failed jobs were rerun without changing the PR head;
• new jobs: 101610597633, 101610597845;
• both jobs currently have steps=null and are queued before runner execution.

No GREEN, approval, merge readiness, predecessor-evidence transfer, bypass, or synthetic status is claimed.