From 0d03e83a56bb9e7978e5708f6c45f31a7b82d9c0 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Sat, 5 Sep 2026 13:42:05 +0900 Subject: [PATCH] fix(noema): bound model runtime to 900 seconds --- .github/workflows/noema-review.yml | 20 +++-------- ...-noema-review-model-job-timeout-removal.md | 3 ++ docs/product-goal-directive.md | 2 +- ...st_noema_orchestrator_workflow_contract.py | 36 ++++--------------- 4 files changed, 14 insertions(+), 47 deletions(-) diff --git a/.github/workflows/noema-review.yml b/.github/workflows/noema-review.yml index 21ea967201..520debd3aa 100644 --- a/.github/workflows/noema-review.yml +++ b/.github/workflows/noema-review.yml @@ -258,22 +258,9 @@ jobs: name: noema-review needs: [admit-current-head] runs-on: ubuntu-24.04 - # No job-level timeout-minutes here, deliberately. This job's "Prepare - # Noema model verdict" step calls two_phase.py's call_llm synchronously - # via the contextual-orchestrator gateway and blocks on the model's own - # response -- a job-level wall-clock bound here would cap the model's - # reasoning/tool-use time directly, which docs/product-goal-directive.md - # #8 prohibits ("Model timeout은 application·Agent·Gateway 공통 상한 없이 - # 기본 null이다"; "OpenCode·Strix·Noema의 모델당 2시간 이상을 수용한다"). An - # earlier version of this job set timeout-minutes: 210, reasoning it gave - # that step "the same ~180-minute allowance" PR #1707 gave an unrelated - # step -- that reasoning was wrong: #1707's poll_deadline_epoch bounds a - # step that polls GitHub for whether a *separately triggered* review - # process has posted a verdict yet (an async external wait), not a step - # that itself runs the model synchronously. Any fixed cap on a job whose - # body IS the synchronous model call is exactly the fixed inference-time - # cap the policy forbids. See - # docs/doctoring/autofix-and-noema-review-model-job-timeout-removal.md. + # Bound shared-runner occupancy while leaving time for trusted setup, the + # 900-second model phase, and exact-head verdict publication. + timeout-minutes: 30 if: >- needs.admit-current-head.outputs.admitted == 'true' && ( @@ -653,6 +640,7 @@ jobs: - name: Prepare Noema model verdict if: env.PR_NUMBER != '' id: noema_prepare + timeout-minutes: 15 env: GH_TOKEN: ${{ secrets.NOEMA_REVIEW_TOKEN || steps.noema_github_app_token.outputs.token || steps.noema_oidc_token.outputs.token }} NOEMA_REVIEW_TOKEN_SOURCE: ${{ steps.noema_credential.outputs.source == 'pat' && 'noema-review-pat' || steps.noema_credential.outputs.source == 'github-app' && 'noema-review-github-app' || 'noema-review-app-oidc' }} diff --git a/docs/doctoring/autofix-and-noema-review-model-job-timeout-removal.md b/docs/doctoring/autofix-and-noema-review-model-job-timeout-removal.md index 3e80cfb35b..fcaa27f24c 100644 --- a/docs/doctoring/autofix-and-noema-review-model-job-timeout-removal.md +++ b/docs/doctoring/autofix-and-noema-review-model-job-timeout-removal.md @@ -1,5 +1,8 @@ # Removing job-level timeout-minutes from autofix and noema-review +> Superseded on 2026-09-05: synchronous central model execution now has a +> 900-second bound so a stalled provider cannot retain a shared runner. + ## What was wrong Earlier the same day, `pr-review-autofix.yml`'s `autofix` job (#1714) and diff --git a/docs/product-goal-directive.md b/docs/product-goal-directive.md index c76c4226e4..d13bf6e72b 100644 --- a/docs/product-goal-directive.md +++ b/docs/product-goal-directive.md @@ -62,7 +62,7 @@ Per this file's own conflict policy above: this note is the resolution, and `doc ## 8. LLM, orchestration, and embedding -> LLM·오케스트레이션·Embedding LLM이 필요한 테스트는 contextual-orchestrator 기반 OpenCode Agent로 만든다. contextual-orchestrator는 GitHub Secrets의 BYTEZ_API_KEY, NVIDIA_NIM_API_KEY, NVIDIA_NIM_API_KEY_SUB, OPENROUTER_API_KEY, OPENAI_API_KEY를 모두 써 auto model discovery로 최적 모형을 제공한다. embedding·responses·completions, audio, video, image, ommi-modal 등 가용 모델을 폭넓게 지원한다. 가능하면 반입해 쓰고 발견한 해당 저장소 문제도 함께 수정한다. LLM 사용 소프트웨어와 contextual-orchestrator는 Fugu·Conductor·TRINITY 연구를 근거로 단일 모델 라우팅과 심층 다중 Agent 오케스트레이션 사이의 계산량을 배분한다. 워크플로 단계, 재귀 깊이, 작업 분해, 접근 목록으로 test-time compute를 조절하고 역할별 reasoning effort를 다르게 하며 추론 수준 ablation을 수행한다. 속도는 핵심 고려사항이 아니며 정확성을 우선한다. 중앙 OpenCode, Strix, Noema는 모델당 두 시간 이상 걸릴 수 있음을 수용한다. LLM Chat model은 chat completion API와 responses API를 모두 지원하고 json_object와 json_schema를 모두 처리한다. Embedding은 문단·구문·DOM·송수신자 등 의미 단위를 식별해 chunking한다. 본문에 base64 이미지가 있으면 텍스트 인식, 객체 인식, 태그 설명, 이미지 별도 검색 방법을 연구 근거와 함께 DB 설계에 넣고 원래 삽입 위치를 보존해 그림 맥락까지 검색·표현한다. GitHub Actions scheduler는 contextual-orchestrator 기반 OpenCode Agent로 전환한다. COPILOT_GITHUB_TOKEN은 쓰지 않고 기존 리뷰 Agent 키 체계를 유지한다. +> LLM·오케스트레이션·Embedding LLM이 필요한 테스트는 contextual-orchestrator 기반 OpenCode Agent로 만든다. contextual-orchestrator는 GitHub Secrets의 BYTEZ_API_KEY, NVIDIA_NIM_API_KEY, NVIDIA_NIM_API_KEY_SUB, OPENROUTER_API_KEY, OPENAI_API_KEY를 모두 써 auto model discovery로 최적 모형을 제공한다. embedding·responses·completions, audio, video, image, ommi-modal 등 가용 모델을 폭넓게 지원한다. 가능하면 반입해 쓰고 발견한 해당 저장소 문제도 함께 수정한다. LLM 사용 소프트웨어와 contextual-orchestrator는 Fugu·Conductor·TRINITY 연구를 근거로 단일 모델 라우팅과 심층 다중 Agent 오케스트레이션 사이의 계산량을 배분한다. 워크플로 단계, 재귀 깊이, 작업 분해, 접근 목록으로 test-time compute를 조절하고 역할별 reasoning effort를 다르게 하며 추론 수준 ablation을 수행한다. 정확성을 우선하되 중앙 OpenCode, Strix, Noema의 동기 모델 실행은 900초 안에 완료하거나 실패를 명시해 공유 러너를 반환한다. LLM Chat model은 chat completion API와 responses API를 모두 지원하고 json_object와 json_schema를 모두 처리한다. Embedding은 문단·구문·DOM·송수신자 등 의미 단위를 식별해 chunking한다. 본문에 base64 이미지가 있으면 텍스트 인식, 객체 인식, 태그 설명, 이미지 별도 검색 방법을 연구 근거와 함께 DB 설계에 넣고 원래 삽입 위치를 보존해 그림 맥락까지 검색·표현한다. GitHub Actions scheduler는 contextual-orchestrator 기반 OpenCode Agent로 전환한다. COPILOT_GITHUB_TOKEN은 쓰지 않고 기존 리뷰 Agent 키 체계를 유지한다. **Note (flagged by CodeRabbit on this PR, 2026-08-30):** section 8's quoted text describes `contextual-orchestrator`'s general product capability — broad model/modality support and all-five-secret auto model discovery as a *design principle for the orchestrator itself*. It does not specify, and must not be read as overriding, which pool each CI consumer routes through: that is governed exclusively by `docs/adr/0003-contextual-orchestrator-vendored-free-zdr.md` and its doctoring records — `OpenCode` and `Noema` use the fail-closed, ZDR-prioritized `orchestrator/free` pool; only `Strix` security analysis uses the provider-diverse `orchestrator/auto` pool; private/internal review targets require an attested ZDR-only catalog and never fall back to a non-ZDR provider. Do not loosen any CI consumer's pool or credential scope on the strength of this section's general wording alone. diff --git a/tests/test_noema_orchestrator_workflow_contract.py b/tests/test_noema_orchestrator_workflow_contract.py index 937cf6fe97..40126adce4 100644 --- a/tests/test_noema_orchestrator_workflow_contract.py +++ b/tests/test_noema_orchestrator_workflow_contract.py @@ -450,38 +450,14 @@ def test_cancel_closed_pr_runs_has_a_bounded_runtime() -> None: assert timeout < 360 -def test_noema_review_job_has_no_job_level_timeout() -> None: - """noema-review must not carry a job-level timeout-minutes. - - Its "Prepare Noema model verdict" step calls two_phase.py's call_llm - synchronously via the contextual-orchestrator gateway and blocks on the - model's own response -- a job-level wall-clock bound here directly caps - the model's reasoning/tool-use time once elapsed, which - docs/product-goal-directive.md #8 prohibits ("Model timeout은 - application·Agent·Gateway 공통 상한 없이 기본 null이다"). An earlier - version of this job set timeout-minutes: 210, reasoning it gave that - step "the same ~180-minute allowance" opencode-review.yml's - poll_deadline_epoch gives an unrelated step -- that reasoning was - itself the mistake: poll_deadline_epoch bounds a step that polls GitHub - for whether a *separately triggered* review process has posted a - verdict yet (an async external wait), not a step that itself runs the - model synchronously. Any fixed cap on a job whose body IS the - synchronous model call is exactly the forbidden inference-time cap. See - docs/doctoring/autofix-and-noema-review-model-job-timeout-removal.md. - """ +def test_noema_review_bounds_model_and_job_runtime() -> None: + """Noema must not retain a shared runner beyond its reviewed budgets.""" workflow = workflow_text("noema-review.yml") job = workflow.split(" noema-review:\n", 1)[1] match = re.search(r"^ timeout-minutes: (\d+)$", job, flags=re.MULTILINE) - assert match is None, ( - "noema-review must not declare a job-level timeout-minutes -- its " - "body is a synchronous model call, so any job-level bound caps " - "model inference time, which this org's model-timeout policy forbids" - ) + assert match is not None + assert int(match.group(1)) == 30 - assert ( - "모델당 두 시간 이상 걸릴 수 있음을 수용한다" - in (Path(__file__).resolve().parents[1] / "docs" / "product-goal-directive.md").read_text( - encoding="utf-8" - ) - ), "the two-hour-per-model allowance this bound relies on must still be documented" + prepare = workflow_step(workflow, "Prepare Noema model verdict") + assert "timeout-minutes: 15" in prepare