Skip to content

fix(noema): expose structured failure kind - #1898

Open
seonghobae wants to merge 69 commits into
mainfrom
codex/noema-failure-kind-telemetry
Open

fix(noema): expose structured failure kind#1898
seonghobae wants to merge 69 commits into
mainfrom
codex/noema-failure-kind-telemetry

Conversation

@seonghobae

@seonghobae seonghobae commented Sep 5, 2026

Copy link
Copy Markdown
Contributor

Current integration receipt — 2026-09-05

Exact head: a7afb39d8c9e6b1b20857e378a236f566c94ff36; tree: fdf57270e965050f7c781ef40303af48d5e6dd3c; protected base: f250638827f8252b0d9e5cb2601f4d333f96162f. The existing df0f735f42adbb44d45f3c3a4e503e400b47ed79 failure-kind delta and the merged #1922 prerequisite are preserved by ordinary merges. This same PR branch was pushed without force; no closure, approval, ruleset change, or merge was performed.

Causal repair and evidence boundary

The original patch handles optional error.detail.failure_kind. Protected contextual-orchestrator a080297d2546bb61e89520d637cabc202db331ec already returns error.code=invalid_structured_output for a structured-response error but that path has no failure kind, model, or attempt list. The consumer was discarding this existing classification. It now retains canonical error.code independently, using the same bounded reader and identifier validator in both the single failure annotation and the raised diagnostic.

CO #1004's typed exhaustion detail remains proposed, not a released dependency or proof of the cause of Naruon #1244's historical 502. Noema still makes one gateway request and fails on the returned error; no provider routing, paid fallback, retry, timeout, or semantic-approval rule changed. Identifier format checking is not arbitrary secret detection. Free-form messages, request identifiers, and unrelated body fields remain excluded.

The existing doctoring record and G-02/G-03 baseline carry exact producer/API/log links, alternatives, OWASP APA 7 grounding, regression history, ownership, and remaining delivery gates. CHANGELOG records the observable maintenance benefit without claiming incident resolution.

Verification on the unchanged candidate

  • RED: removing only canonical-code extraction/formatting yielded 8 failed / 43 passed. Restored field matrix: 51 passed, including the sparse current gateway envelope, malformed values, 200/201-character boundaries, CR/LF, escapes, surrogates, absent siblings, one request/annotation, and no unrelated payload text.
  • Focused Noema/edge/environment regression: 176 passed.
  • Final normal full suite: 2940 passed, 1 skipped, 21 subtests passed (115.62 s).
  • Final GITHUB_ACTIONS=true full suite: 2940 passed, 1 skipped, 21 subtests passed (141.22 s). Five measured modules: 1512 statements / 606 branches, 100%; Noema alone: 854 statements / 384 branches, 100%.
  • Noema docstrings: 100%. Ruff 0.15.21 and compilation passed; unused test import found by Ruff was removed. Worktree was clean and head/tree unchanged before and after verification.
  • The existing reviewed-LLVM-19 admission case is skipped on this macOS host because the required tools are absent. It is unverified, not passed.
  • Task-local Python 3.12.13 uses the hash-locked review requirements plus the separately declared pip==26.2.1 tool. The initial missing-pip failure was corrected in the environment, not hidden or converted into a source change. This is not claimed as a fully hash-locked clean install or hosted Python 3.14 execution.
  • Independent read-only review identified the missing sparse-envelope case; it was added and re-reviewed with no remaining actionable finding. This is not a qualifying GitHub approval.

Commands (repository root, task-local environment):

PATH="$PWD/.venv/bin:$PATH" .venv/bin/python -m pytest tests -q -W error -rs
GITHUB_ACTIONS=true PATH="$PWD/.venv/bin:$PATH" .venv/bin/python -m pytest tests -q -W error -rs --cov=scripts.ci.noema_review_gate --cov=scripts.ci.pr_review_conflict_scope --cov=scripts.ci.pr_review_autofix_context --cov=scripts.ci.zdr_policy --cov=scripts.ci.contextual_orchestrator_review_policy --cov-branch --cov-fail-under=100
.venv/bin/python -m interrogate --fail-under 100 scripts/ci/noema_review_gate.py

JUnit SHA-256: normal 9cec553a5a503cbf1ba529eafdb79f8b82951aa978c2fa232816ede3d44608a5; CI be8f30f2c08c451c747f3a2c3def264674cd47c8fb6fc3eca7f77d7ae136c881. Coverage JSON SHA-256: 3ccad4837f674d8028e4ce83cbe83e1a95ea12eb45f283fb0d11ee3ac7e4df7e.

Remaining protected-delivery work

New exact-head quality run 33962985324, Noema run 33962984599, security scans, and independent review are queued/pending after the push. Preserve those handles and re-fetch terminal evidence; do not repeat dispatches or borrow old-head passes. The separate maintainer exception reported for #1922 does not authorize bypass for this PR. Local checks do not prove protected merge, release, real PostgreSQL/browser operation, or a deployed gateway fix.

Historical original receipt

The text below describes the original df0f735 proposal and its 123-test run. Its mention of CO #1004 is proposed-contract context, not a claim that the current protected structured-error path already emits that field.

문제

contextual-orchestrator가 안전한 HTTP 오류 detail에 failure_kind=structured_output_exhausted를 제공해도 중앙 Noema는 해당 필드를 allowlist하지 않아 Actions 로그에서 구조화 응답 실패 원인을 버렸습니다.

수정

  • 기존 bounded HTTP error parser에서 안전한 식별자 규칙으로 failure_kind를 수용합니다.
  • 공개 로그에는 allowlist된 scalar만 기록하며 원문 응답과 임의 필드는 계속 폐기합니다.
  • HTTP 502 회귀 테스트에 structured_output_exhausted 전달과 비밀 비노출을 검증합니다.

검증

  • uv run pytest -q tests/test_noema_review_gate.py tests/test_noema_model_output_edge_coverage.py
  • 123 passed
  • git diff --check

#1004가 제공하는 typed structured-output detail과 중앙 Noema telemetry를 연결합니다.

@opencode-agent
opencode-agent Bot disabled auto-merge September 5, 2026 10:52
seonghobae and others added 7 commits September 5, 2026 20:02
Keep the existing failure_kind delta. Preserve bounded error.code from the protected gateway envelope in both failed-call diagnostics; do not infer failure cause from the HTTP exception label. The expanded field regression reproduced four failures before the fix; focused Noema and declared-pip environment regression: 159 passed.

Co-Authored-By: Codex <noreply@openai.com>
Signed-off-by: Seongho Bae <me@seonghobae.me>
Address the independent sparse-envelope finding while retaining the existing error telemetry implementation. Removing code extraction reproduced 8 failed/43 passed; restored field matrix 51 passed and focused Noema/edge/environment tests 176 passed. Remove one unused test import found by Ruff. Record the protected-source contract, unknown historical 502 cause, logging limits, separate verification stages, and APA logging guidance in the existing doctoring and gap baseline.

Co-Authored-By: Codex <noreply@openai.com>
Signed-off-by: Seongho Bae <me@seonghobae.me>
Non-force merge current protected main into #1641. Reject model-authored runtime, command-output, toolchain-help, and authoritative external-source claims unless a typed out-of-band receipt is explicitly cited. Preserve source-only reasoning, verification directions, exact-line evidence, and fail-closed findings.

Grounded by ConceptWeave #35 review 5120903874 and Noema run 33938445009/job 101256294197, which executed no Cargo or documentation lookup. Removed both purpose-complete temporary writer workflows.

Validation: 268 Noema tests; full 2,922 passed, 1 skipped, 21 subtests.
Resolve the remaining valid #1641 review findings. One unified-diff state machine now emits both exact source text and accepted coordinates. Bounded truncation drops the incomplete final line instead of manufacturing +/- source, so a genuine line equal to the historical omission marker remains reviewable.

Focused RED reproduced both failures before the repair. GREEN: focused 3 passed; Noema 268 passed; full 2,922 passed, 1 skipped, 21 subtests; py_compile and diff checks clean.

Copy link
Copy Markdown
Contributor Author

Dependency-order restack (2026-09-05 UTC)

#1641 and #1898 both modified the canonical Noema validator from protected main@f250638827f8252b0d9e5cb2601f4d333f96162f and had diverged. #1641 is now the dependency root; this later telemetry PR is its ordinary non-force descendant.

The provenance receipt boundary, unified diff parser, whitespace-only evidence guard, and bounded error.code/failure_kind telemetry all coexist on the candidate. No predecessor hosted check/review evidence transfers to this head.

Copy link
Copy Markdown
Contributor Author

Writer handoff — source writer released

The bounded source integration and local verification for this PR are complete. There is no active #1898 source-writing process or verification command in this automation.

  • exact head: a7afb39d8c9e6b1b20857e378a236f566c94ff36
  • tree: fdf57270e965050f7c781ef40303af48d5e6dd3c
  • ordinary merge parents: 0db01c2615457430018a584be1394f57dfdd7038 and evidence-provenance root 294601776987407b2feac45eb949656dd579d655
  • completed local evidence: Noema 319 passed; repository 2,973 passed · 1 skipped · 21 subtests; compilation/diff checks clean

Those results belong to this local execution only. They do not transfer predecessor coverage or replace fresh exact-head hosted checks/review. The originating read-only audit may safely continue without treating this automation as an active competing writer. No source, rerun, model call, approval, or merge was performed in this handoff.

Copy link
Copy Markdown
Contributor Author

Ordinary non-force integration of the updated #1641 dependency root is complete.

Exact #1898 head: 46a01788c181e1228889ed96e792253f82a3c619
Parents:

  1. predecessor fix(noema): expose structured failure kind #1898 a7afb39d8c9e6b1b20857e378a236f566c94ff36
  2. current fix(noema): enforce observed defect-class review evidence #1641 9df1ea4c03521aa69e9ba6b48fa4d940d858fb0c
    Combined tree: 86383910a4b422e1eed69da98b8f80fe3b96c65c

Fresh checkout GREEN on that exact combined head:

  • Noema-related suite: 586 passed
  • full normal: 2976 passed, 1 skipped, 21 subtests
  • full GITHUB_ACTIONS=true: 2976 passed, 1 skipped, 21 subtests
  • full branch coverage: 2976 passed, 1 skipped, 21 subtests; scripts/ci/noema_review_gate.py 953 statements / 444 branches, 100%

This integrates strict outbound-schema/local-validator parity and the corrected invalid-location telemetry fixture without transferring the predecessor 0db01c coverage claim. No workflow rerun, model call, review dismissal, approval, merge, or bypass was issued. #1898 source/verification writer is released at 46a01788c.

Copy link
Copy Markdown
Contributor Author

Protected main adoption is now included through the current #1641 dependency root.

Current #1898 exact head: f1cd87504ec773dac17027b90857cf139111abcc
Parents: prior #1898 46a01788c181e1228889ed96e792253f82a3c619 + current #1641 409638a918e9a224534f7b8779c63cc15ddda801
Tree: 51d7a438ee329708f7304518c83b90c8cb64320e

Fresh exact-head results:

  • full normal: 2977 passed, 1 skipped, 21 subtests
  • full GITHUB_ACTIONS=true: 2977 passed, 1 skipped, 21 subtests
  • branch coverage: 953 statements / 444 branches in noema_review_gate.py, 100%

No predecessor GREEN, historical review, or prior coverage was transferred. No rerun, model call, self-approval, bypass, or protected merge was issued. #1898 source/verification writer is released at f1cd8750.

Wrap the verdict alternatives below the root object so the strict structured-output schema can reject decision/status contradictions without unsupported root anyOf or conditional keywords. Keep the local semantic validator as a second fail-closed boundary.

Always close the bounded HTTPError response after typed telemetry extraction without masking the original transport outcome. Add exact regression, malformed-envelope, and cleanup-failure coverage.
Preserve the bounded typed repair-attempt telemetry while adopting the canonical #1641 decision/status structured-output parity and HTTPError lifetime fix through an ordinary two-parent merge.

Copy link
Copy Markdown
Contributor Author

Non-force successor integration completed at exact head c41cc4534315babc65bfe8b6ec4029b31c42d91a (parents prior #1898@f1cd87504ec773dac17027b90857cf139111abcc + canonical #1641@80fc255d0df2634273b01600f1c3c713b2409557; tree 1475b88939e8eeb8bb4ed223c897165cc3076743).

This ordinary two-parent merge preserves the existing bounded error.code / failure_kind telemetry delta and adopts the canonical decision/status schema parity plus HTTPError lifetime repair. Exact combined-tree GREEN:

  • normal: 3,021 passed · 1 skipped · 21 subtests;
  • CI environment: 3,021 passed · 1 skipped · 21 subtests;
  • coverage run: 3,021 passed · 1 skipped · 21 subtests; noema_review_gate.py 973 statements / 450 branches = 100%.

Historical predecessor results are not transferred to this head. Current-head required checks and independent review remain required. Source writer and local verification commands for #1898 are now released.

정리 중 일반 예외가 원래 실패를 덮지 않게 하되 사용자 중단은 전파한다. 직접 리다이렉트 검사는 자신이 받은 응답을 닫는다.

Commit-Message-Assisted-by: Codex (OpenAI)
Signed-off-by: Seongho Bae <me@seonghobae.me>
Adopt #1641's bounded HTTPError cleanup repair without force-pushing while preserving #1898's typed failure telemetry.

Exact combined-tree verification:
- focused Noema contracts: 233 passed
- repository suite: 3024 passed, 1 skipped, 21 subtests in normal and GITHUB_ACTIONS=true environments

No gate weakening, warning suppression, consumer rerun, or authorization change.

Copy link
Copy Markdown
Contributor Author

Latest #1641 cleanup was adopted at exact head 4cf6febcac772745587c8bd636668b75a9e21e56 as an ordinary two-parent, non-force merge of predecessor #1898 c41cc4534315babc65bfe8b6ec4029b31c42d91a and #1641 ad48dd65c7d0d8b6e0d37f0315302b9c6e138899 (tree 8b3b209f5559e55c2fa879f3c77af05a34e50e2b).

The combined tree preserves #1898's bounded error.code / failure_kind telemetry and #1641's schema/provenance contract plus HTTPError cleanup semantics. The sole CHANGELOG conflict was resolved by retaining both independent entries.

Exact GREEN:

  • focused Noema contracts: 233 passed;
  • full repository suite: 3,024 passed · 1 skipped · 21 subtests in normal and GITHUB_ACTIONS=true environments.

Fresh exact-head hosted checks and independent current-head review remain required. No warning suppression, consumer rerun, authorization change, self-approval, bypass, force-push, or gate weakening was performed.

Source-writer release: no #1898 source mutation or validation command remains active from this owner; further work must reuse this branch after a live compare-guard.

@seonghobae seonghobae added bug Something isn't working priority: medium Normal-priority or P2 work status: needs-review Open pull request requiring current-head review or checks type: bug Defect or incorrect behavior labels Sep 7, 2026 — with ChatGPT Codex Connector
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working priority: medium Normal-priority or P2 work status: needs-review Open pull request requiring current-head review or checks type: bug Defect or incorrect behavior

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant