Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 16 additions & 4 deletions .github/workflows/noema-review.yml
Original file line number Diff line number Diff line change
Expand Up @@ -258,9 +258,22 @@ jobs:
name: noema-review
needs: [admit-current-head]
runs-on: ubuntu-24.04
# Bound shared-runner occupancy while leaving time for trusted setup, the
# 900-second model phase, and exact-head verdict publication.
timeout-minutes: 30
# No job-level timeout-minutes here, deliberately. This job's "Prepare
# Noema model verdict" step calls two_phase.py's call_llm synchronously
# via the contextual-orchestrator gateway and blocks on the model's own
# response -- a job-level wall-clock bound here would cap the model's
# reasoning/tool-use time directly, which docs/product-goal-directive.md
# #8 prohibits ("Model timeout은 application·Agent·Gateway 공통 상한 없이
# 기본 null이다"; "OpenCode·Strix·Noema의 모델당 2시간 이상을 수용한다"). An
# earlier version of this job set timeout-minutes: 210, reasoning it gave
# that step "the same ~180-minute allowance" PR #1707 gave an unrelated
# step -- that reasoning was wrong: #1707's poll_deadline_epoch bounds a
# step that polls GitHub for whether a *separately triggered* review
# process has posted a verdict yet (an async external wait), not a step
# that itself runs the model synchronously. Any fixed cap on a job whose
# body IS the synchronous model call is exactly the fixed inference-time
# cap the policy forbids. See
# docs/doctoring/autofix-and-noema-review-model-job-timeout-removal.md.
if: >-
needs.admit-current-head.outputs.admitted == 'true'
&& (
Expand Down Expand Up @@ -640,7 +653,6 @@ jobs:
- name: Prepare Noema model verdict
if: env.PR_NUMBER != ''
id: noema_prepare
timeout-minutes: 15
env:
GH_TOKEN: ${{ secrets.NOEMA_REVIEW_TOKEN || steps.noema_github_app_token.outputs.token || steps.noema_oidc_token.outputs.token }}
NOEMA_REVIEW_TOKEN_SOURCE: ${{ steps.noema_credential.outputs.source == 'pat' && 'noema-review-pat' || steps.noema_credential.outputs.source == 'github-app' && 'noema-review-github-app' || 'noema-review-app-oidc' }}
Expand Down
Original file line number Diff line number Diff line change
@@ -1,8 +1,5 @@
# Removing job-level timeout-minutes from autofix and noema-review

> Superseded on 2026-09-05: synchronous central model execution now has a
> 900-second bound so a stalled provider cannot retain a shared runner.

## What was wrong

Earlier the same day, `pr-review-autofix.yml`'s `autofix` job (#1714) and
Expand Down
2 changes: 1 addition & 1 deletion docs/product-goal-directive.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@ Per this file's own conflict policy above: this note is the resolution, and `doc

## 8. LLM, orchestration, and embedding

> LLM·오케스트레이션·Embedding LLM이 필요한 테스트는 contextual-orchestrator 기반 OpenCode Agent로 만든다. contextual-orchestrator는 GitHub Secrets의 BYTEZ_API_KEY, NVIDIA_NIM_API_KEY, NVIDIA_NIM_API_KEY_SUB, OPENROUTER_API_KEY, OPENAI_API_KEY를 모두 써 auto model discovery로 최적 모형을 제공한다. embedding·responses·completions, audio, video, image, ommi-modal 등 가용 모델을 폭넓게 지원한다. 가능하면 반입해 쓰고 발견한 해당 저장소 문제도 함께 수정한다. LLM 사용 소프트웨어와 contextual-orchestrator는 Fugu·Conductor·TRINITY 연구를 근거로 단일 모델 라우팅과 심층 다중 Agent 오케스트레이션 사이의 계산량을 배분한다. 워크플로 단계, 재귀 깊이, 작업 분해, 접근 목록으로 test-time compute를 조절하고 역할별 reasoning effort를 다르게 하며 추론 수준 ablation을 수행한다. 정확성을 우선하되 중앙 OpenCode, Strix, Noema의 동기 모델 실행은 900초 안에 완료하거나 실패를 명시해 공유 러너를 반환한다. LLM Chat model은 chat completion API와 responses API를 모두 지원하고 json_object와 json_schema를 모두 처리한다. Embedding은 문단·구문·DOM·송수신자 등 의미 단위를 식별해 chunking한다. 본문에 base64 이미지가 있으면 텍스트 인식, 객체 인식, 태그 설명, 이미지 별도 검색 방법을 연구 근거와 함께 DB 설계에 넣고 원래 삽입 위치를 보존해 그림 맥락까지 검색·표현한다. GitHub Actions scheduler는 contextual-orchestrator 기반 OpenCode Agent로 전환한다. COPILOT_GITHUB_TOKEN은 쓰지 않고 기존 리뷰 Agent 키 체계를 유지한다.
> LLM·오케스트레이션·Embedding LLM이 필요한 테스트는 contextual-orchestrator 기반 OpenCode Agent로 만든다. contextual-orchestrator는 GitHub Secrets의 BYTEZ_API_KEY, NVIDIA_NIM_API_KEY, NVIDIA_NIM_API_KEY_SUB, OPENROUTER_API_KEY, OPENAI_API_KEY를 모두 써 auto model discovery로 최적 모형을 제공한다. embedding·responses·completions, audio, video, image, ommi-modal 등 가용 모델을 폭넓게 지원한다. 가능하면 반입해 쓰고 발견한 해당 저장소 문제도 함께 수정한다. LLM 사용 소프트웨어와 contextual-orchestrator는 Fugu·Conductor·TRINITY 연구를 근거로 단일 모델 라우팅과 심층 다중 Agent 오케스트레이션 사이의 계산량을 배분한다. 워크플로 단계, 재귀 깊이, 작업 분해, 접근 목록으로 test-time compute를 조절하고 역할별 reasoning effort를 다르게 하며 추론 수준 ablation을 수행한다. 속도는 핵심 고려사항이 아니며 정확성을 우선한다. 중앙 OpenCode, Strix, Noema는 모델당 두 시간 이상 걸릴 수 있음을 수용한다. LLM Chat model은 chat completion API와 responses API를 모두 지원하고 json_object와 json_schema를 모두 처리한다. Embedding은 문단·구문·DOM·송수신자 등 의미 단위를 식별해 chunking한다. 본문에 base64 이미지가 있으면 텍스트 인식, 객체 인식, 태그 설명, 이미지 별도 검색 방법을 연구 근거와 함께 DB 설계에 넣고 원래 삽입 위치를 보존해 그림 맥락까지 검색·표현한다. GitHub Actions scheduler는 contextual-orchestrator 기반 OpenCode Agent로 전환한다. COPILOT_GITHUB_TOKEN은 쓰지 않고 기존 리뷰 Agent 키 체계를 유지한다.

**Note (flagged by CodeRabbit on this PR, 2026-08-30):** section 8's quoted text describes `contextual-orchestrator`'s general product capability — broad model/modality support and all-five-secret auto model discovery as a *design principle for the orchestrator itself*. It does not specify, and must not be read as overriding, which pool each CI consumer routes through: that is governed exclusively by `docs/adr/0003-contextual-orchestrator-vendored-free-zdr.md` and its doctoring records — `OpenCode` and `Noema` use the fail-closed, ZDR-prioritized `orchestrator/free` pool; only `Strix` security analysis uses the provider-diverse `orchestrator/auto` pool; private/internal review targets require an attested ZDR-only catalog and never fall back to a non-ZDR provider. Do not loosen any CI consumer's pool or credential scope on the strength of this section's general wording alone.

Expand Down
36 changes: 30 additions & 6 deletions tests/test_noema_orchestrator_workflow_contract.py
Original file line number Diff line number Diff line change
Expand Up @@ -450,14 +450,38 @@ def test_cancel_closed_pr_runs_has_a_bounded_runtime() -> None:
assert timeout < 360


def test_noema_review_bounds_model_and_job_runtime() -> None:
"""Noema must not retain a shared runner beyond its reviewed budgets."""
def test_noema_review_job_has_no_job_level_timeout() -> None:
"""noema-review must not carry a job-level timeout-minutes.

Its "Prepare Noema model verdict" step calls two_phase.py's call_llm
synchronously via the contextual-orchestrator gateway and blocks on the
model's own response -- a job-level wall-clock bound here directly caps
the model's reasoning/tool-use time once elapsed, which
docs/product-goal-directive.md #8 prohibits ("Model timeout은
application·Agent·Gateway 공통 상한 없이 기본 null이다"). An earlier
version of this job set timeout-minutes: 210, reasoning it gave that
step "the same ~180-minute allowance" opencode-review.yml's
poll_deadline_epoch gives an unrelated step -- that reasoning was
itself the mistake: poll_deadline_epoch bounds a step that polls GitHub
for whether a *separately triggered* review process has posted a
verdict yet (an async external wait), not a step that itself runs the
model synchronously. Any fixed cap on a job whose body IS the
synchronous model call is exactly the forbidden inference-time cap. See
docs/doctoring/autofix-and-noema-review-model-job-timeout-removal.md.
"""
workflow = workflow_text("noema-review.yml")
job = workflow.split(" noema-review:\n", 1)[1]

match = re.search(r"^ timeout-minutes: (\d+)$", job, flags=re.MULTILINE)
assert match is not None
assert int(match.group(1)) == 30
assert match is None, (
"noema-review must not declare a job-level timeout-minutes -- its "
"body is a synchronous model call, so any job-level bound caps "
"model inference time, which this org's model-timeout policy forbids"
)

prepare = workflow_step(workflow, "Prepare Noema model verdict")
assert "timeout-minutes: 15" in prepare
assert (
"모델당 두 시간 이상 걸릴 수 있음을 수용한다"
in (Path(__file__).resolve().parents[1] / "docs" / "product-goal-directive.md").read_text(
encoding="utf-8"
)
), "the two-hour-per-model allowance this bound relies on must still be documented"
Loading