Skip to content

fix: pin NOEMA_LLM_MODEL routing alias to orchestrator/free - #535

Draft
seonghobae wants to merge 93 commits into
mainfrom
fix/noema-orchestrator-free-routing-alias
Draft

fix: pin NOEMA_LLM_MODEL routing alias to orchestrator/free#535
seonghobae wants to merge 93 commits into
mainfrom
fix/noema-orchestrator-free-routing-alias

Conversation

@seonghobae

@seonghobae seonghobae commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Scope and ownership

Noema consumes the governed orchestrator/free contract only. ContextualWisdomLab/contextual-orchestrator owns provider/model discovery, routing, provider credentials, retry and failover; this lane retains Noema-owned reviewer/config/privacy/tool/gateway-preflight boundaries without creating local provider authority.

Current exact authority — 2026-09-06 KST

Protected Noema is main@e26d771470a4ece873c367b40b3cd6cb03ac7de3. Ordinary two-parent/non-force restack 5de3fcb2a6acd1b8190ffab95f729fc6b160b0a8 preserves the 45 branch-owned paths and adopts merged #527 OIDC trust. The cross-lane product-gap baseline remains owned only by #547.

Fresh exact-head patch-validator-image 34026886149, required Security Scan 34026886190, reviewer-ci 34026886125, and ci 34026886147 are queued. Predecessor GREEN does not transfer. Keep Draft until unchanged-head terminal gates and zero valid unresolved findings.

The bare contextual-orchestrator alias is treated by contextual-orchestrator's
TaskOrchestrator the same as orchestrator/auto: the full agent pool, including
paid providers, is eligible. Only orchestrator/free restricts a request to the
free/ZDR agent pool (free_only=True, judge_agent_ids scoped to free_ids in
conduct()). Noema's scripts/lib/orchestrator-gateway.mjs hard-enforced the
bare alias as the only accepted NOEMA_LLM_MODEL value, so every Noema/naruon
LLM call (PR review, hourly product development, naruon judgments) could
reach paid providers instead of being restricted to the free/ZDR pool.

Change DEFAULT_ROUTING_ALIAS to "orchestrator/free" and update
resolveOrchestratorModel to hard-reject the old bare alias. Regenerate
contracts/orchestrator-gateway.json and update every test/doc that asserted
the old alias as the canonical value or described routing as "min-cost /
max-performance" (now the fail-closed zero-cost ZDR-first pool). Matches
ContextualWisdomLab/.github's opencode.jsonc, which already pins
contextual-orchestrator/orchestrator/free.

This is a code/doc change only. The live NOEMA_LLM_MODEL GitHub Actions
variable must be updated separately by an org/repo administrator; until then
the hardened preflight fails closed on the old value by design.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BV96rXhqoR3tYZ9AeAVur4
@coderabbitai

coderabbitai Bot commented Sep 2, 2026

Copy link
Copy Markdown

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@seonghobae seonghobae added the bug Something isn't working label Sep 2, 2026 — with ChatGPT Codex Connector

Copy link
Copy Markdown
Contributor Author

Cross-repo consistency check from a session that's spent this pass deep in .github's docs/product-goal-directive.md §8/§10 (the org's canonical LLM-orchestration/CI-pool-pinning directive) and its Devin Review reconciliation — offering this since it's context another same-repo reviewer likely doesn't have loaded.

Two things this PR does that line up cleanly with what the directive requires elsewhere in the org, worth having on record as independent cross-repo confirmation:

  1. Timeout/retry delegation (config.py's _reject_legacy_attempt_controls, AsyncOpenAI(timeout=None, max_retries=0)): this matches §8's "no uniform hardcoded LLM request timeout — default unlimited/null, upstream provider ends the call on its own failure" principle exactly. .github's own Strix/OpenCode CI paths made the identical move (removing repo-authored LLM_TIMEOUT/attempt-budget variables in favor of gateway-owned allocation) for the same reason. Good to see the same discipline applied independently in Noema's own reviewer client.
  2. orchestrator/free as the canonical routing alias, with orchestrator/auto and arbitrary aliases explicitly rejected fail-closed: this is exactly the same posture .github/workflows/strix.yml enforces (CONTEXTUAL_ORCHESTRATOR_POOL accepts only free, auto removed as an accepted value — see .github's docs/product-technical-gap-baseline.md). One nuance worth being aware of if it comes up later: .github's own directive had to explicitly reconcile (this session, today) that the original orchestrator/free switch on strix.yml was an unreviewed agent action, not an owner decision, and that the underlying single-outage-domain availability risk (ADR-0003) is still open — the owner has since separately, explicitly authorized the pin-as-CI-policy (not the risk itself). Doesn't block anything here since this PR's doctoring note doesn't make an owner-authorization claim either way, but flagging it in case Noema's own docs ever need to describe why orchestrator/free is required, not just that it is.

No blocking concerns from this reading — flagging for the record, not asking for changes.


Generated by Claude Code

Copy link
Copy Markdown
Contributor Author

Merge conflict resolution: mainfix/noema-orchestrator-free-routing-alias

Confirmed the reported dirty merge state was real (verified fresh via git fetch + git merge-tree against live main@bbee332, not stale), then resolved it with a normal merge commit (no rebase, no force push).

Merge: git merge --no-edit origin/main at PR head 79a8dbc3b725a5a64276bba15f042d5d425b5e9e (main had advanced to bbee33270b496255d785c766fc009a5f9162a695 via PR #528, a runtime-orchestration bounded-context feature landed independently of this PR). Result: merge commit 82dc4b11b4b76b904d919d1ed177460e53123fb0, pushed non-force.

Conflict scope: only CHANGELOG.md had a real textual conflict — both branches appended a distinct ## Unreleased bullet at the same insertion point (this PR's orchestrator/free routing-alias entry vs. main's Workflow/Task Execution + State & Checkpoint admission entries). Resolved by keeping both entries, in order. Every other file git merge-tree reported as touched by both sides (docs/PRD.md, docs/README.md, docs/adr/README.md, plus the new bounded-context src//test/ files from #528) merged cleanly with no marker conflicts.

No lockfile/package.json changes were part of this merge, so no lockfile regeneration was needed.

Verified this PR's own security-hardening contributions survived the merge intact — did not let main's independent evolution silently drop either side:

  • scripts/lib/orchestrator-gateway.mjs still has "*": "deny" (deny-by-default OpenCode permission model) with the explicit read/edit/glob/grep/list allowlist.
  • orchestrator/free remains the pinned DEFAULT_ROUTING_ALIAS.

Gates run on the merged head (Node 24.19.0, matching devEngines):

  • npm run typecheck — clean.
  • npm run security:scan (npm audit --audit-level=high) — 0 vulnerabilities.
  • npx vitest run test/opencode-tool-capability-boundary.test.ts (the RED/GREEN pair for the deny-by-default permission model) — 1/1 passed, no regression from the merge.
  • npm test (full suite) — 3943-3947 passed; 3 pre-existing failures (test/hourly-product-development-final-candidate-cleanup.test.ts, test/no-heuristic-workflow-authority.test.ts, test/orchestrator-gateway-contract.test.ts) reproduce identically on the unmerged PR head 79a8dbc in this environment (verified in an isolated worktree), so they predate and are unrelated to this merge — consistent with this PR's own "Verification boundary" section noting exact-head CI was not yet GREEN. Not touched or masked by this merge.

PR left in Draft as instructed; no threads resolved (none were addressed by this conflict fix); no merge/approval action taken.


Generated by Claude Code

Comment thread reviewer/tests/test_no_heuristic_gateway_policy.py Fixed
Commit 44e58c9 ("remove local severity admission thresholds") changed
reviewer/noema_reviewer/models.py and gating.py but left four regression
tests, one workflow-authority test, and one CLI contract test unsynced
with the new behavior:

- test_gating.py: the downgrade summary wording changed from "current-head
  checks or MEDIUM-or-higher..." to "unresolved current-head check,
  scanner, or review-thread evidence...", but the test still asserted the
  old plural "current-head checks" substring.
- test_github_io.py: ReviewVerdict.confidence became a read-only property
  (always Confidence.MEDIUM / LegacyConfidence "not-applicable") rather
  than a constructor field, but the test still passed confidence= as a
  kwarg, which pydantic now rejects as extra_forbidden.
- test_verdict_invariants.py: the approval-invariant validator now
  rejects ANY finding (not just MEDIUM+) alongside an APPROVE verdict, per
  models.py's own docstring ("severity ... never a local admission
  threshold") and the already-updated, still-passing
  test_models.py::test_approval_rejects_every_evidence_backed_finding.
  The LOW/INFO "advisory findings remain compatible with approval" test
  encoded the old, removed invariant; flipped it to assert rejection.
- gating.py: added the one missing docstring (nested `identity` helper)
  that was failing the package's 100% interrogate gate, unrelated to but
  blocking the same reviewer-ci check.
- test_no_heuristic_gateway_policy.py: removed a redundant local
  `import noema_reviewer.config as config_module` that duplicated the
  file's existing top-level import; use inspect.getmodule(resolve_config)
  instead (quality nit, no behavior change).

Also fixes two stale `verify` (vitest) fixtures on the same head:

- orchestrator-gateway-contract.test.ts: commit 857b76a added
  requirePublicRepositoryForOpenCode(), which now requires
  GITHUB_EVENT_PATH whenever --write-opencode-config is passed. The
  "prints the gateway identity after a successful CLI preflight" test
  never supplied it, so the success-path assertions failed closed
  (status 1) instead of succeeding. Added a public-visibility event-file
  fixture, matching the pattern already used in
  opencode-private-visibility-boundary.test.ts.
- no-heuristic-workflow-authority.test.ts: the "derives request privacy
  from live repository visibility" test asserted the hourly workflow
  should carry the same gh-api/NOEMA_LLM_ZDR_ONLY pattern as
  central-review.yml. That pattern is specific to the PydanticAI
  reviewer's proved request-level zdr_only transport; OpenCode has no
  such transport (per verify-orchestrator-gateway.mjs's own docstring)
  and instead fails closed by refusing to run entirely for non-public
  repositories, via requirePublicRepositoryForOpenCode -- already
  implemented and covered by opencode-private-visibility-boundary.test.ts.
  Setting NOEMA_LLM_ZDR_ONLY in the hourly workflow would be an unread,
  misleading no-op contrary to this PR's own "no invented ... confidence
  ... policy" principle. Split the test in two: central-review's
  assertions unchanged, hourly's assertions now check the actual
  fail-closed wiring instead of the inapplicable flag pattern.

Verification (reviewer/, Python 3.11, non-root runner, real CI
conditions -- ambient GH_TOKEN unset):
  515 passed, 100.00% line+branch coverage, 100.0% docstring coverage.
Verification (root, vitest): the 2 previously-reported tests pass;
full-suite run confirms 6 other failing files/8 tests are pre-existing
and reproduce identically on the unmodified branch (git stash verified),
unrelated to this change. tsc --noEmit is clean.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BV96rXhqoR3tYZ9AeAVur4

Copy link
Copy Markdown
Contributor Author

Reviewer-ci + verify failures on abae56f4 — root cause and fix

Pushed 9d06400 on top of abae56f4 (non-force). Summary below; full reasoning is in the commit message.

Python reviewer/ package (4 reported pytest failures)

All four traced to one production commit on this branch, 44e58c9 "remove local severity admission thresholds" (preceded by its RED, 519500c), which intentionally:

  • removed Confidence as a settable ReviewVerdict field (replaced with a read-only .confidence property that always returns a LegacyConfidence/"not-applicable" sentinel, since Noema has no calibrated confidence model — see models.py's own docstring),
  • changed the approval-invariant validator from "block only MEDIUM+ findings" to "block APPROVE alongside any finding" (severity is descriptive evidence metadata only, "never a local admission threshold" — same docstring, same wording repeated in agent.py's system prompt), and
  • as a side effect, reworded the downgrade-summary prefix in gating.py (no more severity-threshold language).

44e58c9 updated test_models.py and test_manifest.py accordingly (test_models.py::test_approval_rejects_every_evidence_backed_finding already asserts rejection for every Severity including LOW/INFO, and test_verdict_defaults_are_evidence_only already asserts confidence isn't a field) but missed three other test files:

  1. test_gating.py::test_failed_check_downgrades_approval_with_log_pointer — asserted the old plural substring "current-head checks". Updated to the actual current prefix "unresolved current-head check, scanner, or review-thread evidence".
  2. test_github_io.py::test_render_review_body_marks_findings_and_marker — passed confidence=Confidence.MEDIUM to the constructor, now extra_forbidden since confidence isn't a field. Removed the kwarg; added an assertion that the rendered body shows Confidence: not-applicable (exercising the legacy-sentinel property directly).
  3. test_verdict_invariants.py::test_approval_allows_nonblocking_advisory_findings — directly re-encoded the removed invariant ("LOW/INFO findings remain compatible with approval"), contradicting the sibling test in test_models.py. This is the security-relevant one: I verified the new global-rejection design is deliberate (docstrings + system prompt + the already-updated sibling test agree), not a regression, so I flipped this test to assert ValidationError for LOW/INFO too, matching MEDIUM/HIGH/CRITICAL.

Also fixed gating.py: the package's interrogate docstring gate (100% required) was separately failing on a missing docstring on the nested identity() helper in _enforce_findings — pre-existing on this branch, unrelated to the 4 pytest failures but blocking the same reviewer-ci check-run. Added the one-line docstring.

Folded in the requested quality nit in test_no_heuristic_gateway_policy.py: removed the redundant local import noema_reviewer.config as config_module (duplicate of the top-level import) in favor of inspect.getmodule(resolve_config).

Verification (Python 3.11, non-root runner, GH_TOKEN unset — matching real CI; this sandbox's ambient GH_TOKEN had been masking one branch in github_io.py as a false coverage gap, unrelated to this PR):

  • RED (pre-fix, on abae56f4): exactly the 4 reported failures reproduced.
  • GREEN (post-fix): 515 passed, 0 failed, 100.00% line+branch coverage, 100.0% docstring coverage (interrogate).

verify (vitest) — 2 fixtures on the same head

  1. orchestrator-gateway-contract.test.ts — "prints the gateway identity after a successful CLI preflight": commit 857b76a ("fail closed for private OpenCode routing") added requirePublicRepositoryForOpenCode(), which now requires GITHUB_EVENT_PATH whenever --write-opencode-config is passed. 857b76a didn't update this pre-existing success-path test, so it now fails closed (status 1) for a fixture with no event file. Added a GITHUB_EVENT_PATH fixture with visibility: "public", matching the pattern in opencode-private-visibility-boundary.test.ts.
  2. no-heuristic-workflow-authority.test.ts — "derives request privacy from live repository visibility": asserted the hourly workflow should carry the same gh api .../--jq .visibilityNOEMA_LLM_ZDR_ONLY pattern as central-review.yml. That pattern exists because the PydanticAI reviewer has a proved request-level zdr_only transport. OpenCode does notverify-orchestrator-gateway.mjs's own docstring says so explicitly, and NOEMA_LLM_ZDR_ONLY is never read anywhere in the OpenCode/hourly path. OpenCode's actual (and already fully tested, in opencode-private-visibility-boundary.test.ts) privacy mechanism is stricter: refuse to run at all for non-public repos, via requirePublicRepositoryForOpenCode. Implementing what the test literally asked for would have added an unread NOEMA_LLM_ZDR_ONLY=true env var to the hourly workflow — a phantom signal nothing enforces, which is exactly the kind of invented/uncalibrated signal this PR's own design principle rejects. I split the test: central-review's assertions are unchanged (still correct), and the hourly assertion now checks the actual fail-closed wiring (--write-opencode-configrequirePublicRepositoryForOpenCode → the real "OpenCode inference fails closed for ..." message) instead of the inapplicable flag pattern. If anyone disagrees with this read, flag it here — I did not touch the underlying privacy behavior, only which assertions the test makes.

Verification: both tests RED before (reproduced exactly as reported) → GREEN after. tsc --noEmit clean. Full vitest run: 3940 passed, 0 of the 2 reported failures remain. 6 other test files (8 tests: acquisition-data-room-manifest, acquisition-output-symlink, acquisition-review-regressions, dependency-license-inventory-release-wiring, hourly-product-development-final-candidate-cleanup [readSingleRunBudget is not a function — a real stale-export bug, but pre-existing and out of scope of what was reported], kpi-tail-streaming-integrity) fail identically with or without my changes (confirmed via git stash against unmodified abae56f4) — root/sandbox-specific (symlink handling, SIGTERM reaping, npm-CLI PATH resolution under this container), not present in the 3945/3948 figure from the actual job log, and untouched by this commit. Flagging readSingleRunBudget specifically since it looks like a genuine stale reference someone should look at, just not part of this fix.

Commit: 9d06400. No force-push, no history rewrite, no gate weakened — the LOW/INFO invariant fix makes the gate stricter, not looser.

🤖 Generated with Claude Code

https://claude.ai/code/session_01BV96rXhqoR3tYZ9AeAVur4


Generated by Claude Code

Copy link
Copy Markdown
Contributor Author

Fresh protected-base repair supersedes the stale authority block. Protected main is e1ac9d50f6c646f04be8c137c8acdc7200182fcd. Ordinary two-parent non-force merge ac5efa46ada634f17da0321df0f80d53fecec4d7 preserves the provider-neutral orchestrator/free/privacy/tool-authority delta and inherits only the protected stateless GitHub installation-token regression. Current exact-head workflows ci 33872001753, reviewer-ci 33872001734, required Security Scan 33872001818, and patch-validator-image 33872001775 are queued/non-passing. No predecessor or local result transfers; keep Draft.

Copy link
Copy Markdown
Contributor Author

2026-09-06 KST post-#546 repair finding: protected semantic reviewer truth is now main@85b17014b8d46eacc95e096ca114568c321d0263. This branch overlaps that protected delta in .github/workflows/central-review.yml, CHANGELOG.md, reviewer/noema_reviewer/gating.py, and reviewer gating/GitHub-I/O tests while also owning CO consumer/work-conserving writer changes. A blind tree replacement would drop valid delta from one side. Keep Draft and perform an ordinary semantic three-way restack that preserves both owner deltas, then regenerate exact-head gates. Do not force-push, copy provider/routing authority into Noema, or transfer the pre-#546 reviewer result.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working priority: high High-priority or P1 work status: draft Draft pull request type: bug Defect or incorrect behavior

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants