fix(judge): harden runtime validation and weighted evidence

[seonghobae]

Closes #1235.
Closes #1238.

Defects

This branch now owns two compatible LLM-judge reliability/data-integrity slices on the same mutable llm_judge.py lane.

1. Five production invariants in llm_judge.py and judge_calibration.py used assert. Python removes those statements under -O/-OO, so invalid internal states could proceed or fail incidentally instead of through the package-owned validation contract.
2. Criterion weights were validated individually but not as a finite aggregate before contextual-orchestrator transport, and the direct/plain judge path trusted the model's redundant top-level score instead of deriving the authoritative decision from validated per-criterion evidence and configured weights.

The second slice was originally intended for a stacked successor (#1239), but this exact branch advanced independently from 4163bbb42763282c0b7729c37e8b919895141a5f to e4d170040ed2230bef8133855826fbdfaec29b86 with that bounded implementation already present. Fresh delta inspection and current tests confirm it satisfies #1235/#1238, so the live implementation is adopted here rather than duplicated or force-rebased.

Fix

• replace category-anchor/category-count assertions with explicit ValueError failures;
• replace passed-calibration-result assertions with explicit RuntimeError failures;
• preserve the protected-main .jules/sentinel.md security-learnings artifact referenced by CLAUDE.md;
• compute criterion-weight totals with package-owned stable finite summation at criteria admission and reject aggregate overflow before transport;
• keep validating the redundant model-reported top-level score, but derive the authoritative direct-path score and accept/reject decision from validated criterion scores and configured weights;
• reuse the same weighted-evidence derivation for binary-threshold/category/direct paths while preserving prompt/JSON/category contracts and treating LLM output as fallible rater evidence, not truth.

Regression evidence

• tests/test_judge_runtime_assertion_safety.py rejects production ast.Assert regressions and proves python -O keeps the package-owned validation behavior.
• tests/test_llm_judge.py::test_direct_judge_derives_score_from_weighted_criteria_not_self_report proves a misleading top-level score=0.9 yields the configured weighted score 0.375 and rejection.
• tests/test_llm_judge.py::test_criteria_reject_overflowing_aggregate_weight_before_any_transport proves two individually valid 1e308 weights fail closed before any orchestrator call.

Exact lineage

• runtime assertion replacement: 091d78bbe4d7071237428e94c38823390590c2c6;
• optimization-mode regression: 08c4044dc7c43d933970632d5b172dab2bf75fa6;
• runtime changelog: dc15bed96f35ce44cfb88592bc47f3222deb8c07;
• canonical sentinel restoration: 4163bbb42763282c0b7729c37e8b919895141a5f;
• weighted-evidence implementation now present on the same live branch: e4d170040ed2230bef8133855826fbdfaec29b86.

Current protected base at this update: main@04d0bc21a2a20693bcf16108cd76d394fe844d23.
Current exact head at this update: e4d170040ed2230bef8133855826fbdfaec29b86.

The repository-local CI, Security Scan, SAST Semgrep, and CodeQL workflows are exact-head green. Historical CHANGES_REQUESTED reviews are bound to predecessor heads and remain historical evidence; current integration still requires all applicable same-head central coverage/review/package/SBOM/provenance gates. No self-approval or gate bypass.

No MLSIRM/IRT likelihood, estimator, calibration statistic, uncertainty, or Rust numerical kernel is changed.

[google-labs-jules]

👋 Jules, reporting for duty! I'm here to lend a hand with this pull request.

When you start a review, I'll add a 👀 emoji to each comment to let you know I've read it. I'll focus on feedback directed at me and will do my best to stay out of conversations between you and other bots or reviewers to keep the noise down.

I'll push a commit with your requested changes shortly after. Please note there might be a delay between these steps, but rest assured I'm on the job!

For more direct control, you can switch me to Reactive Mode. When this mode is on, I will only act on comments where you specifically mention me with @jules. You can find this option in the Pull Request section of your global Jules UI settings. You can always switch back!

New to Jules? Learn more at jules.google/docs.

---

For security, I will only act on instructions from the user who triggered this task.

[coderabbitai]

Caution

Review failed

An error occurred during the review process. Please try again later.

📝 Walkthrough

Walkthrough

The changes replace production assert checks with explicit ValueError and RuntimeError exceptions in judge validation and calibration paths. A security note documents the CWE-617 risk of optimized Python execution removing assertions.

Changes

Runtime validation checks

Layer / File(s)	Summary
Judge input validation python/fast_mlsirm/llm_judge.py	Judge construction and binary-threshold execution now raise ValueError when category anchors or required category counts are missing.
Calibration result validation python/fast_mlsirm/judge_calibration.py, .jules/sentinel.md	Calibration summaries now raise RuntimeError when outcomes lack judge results. The security note documents the assertion risk and prevention pattern.

Estimated code review effort: 2 (Simple) | ~10 minutes

Merge Risk: 🔵 Low · up to f24da

The branch still includes an unintended repository artifact that the change description says should be removed; deleting it avoids carrying unrelated automation content into the product, with no other merge-blocking risk identified.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	제목은 Python -O에서도 런타임 검증을 유지하는 핵심 변경을 정확하고 간결하게 설명합니다.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix-assert-validations-7577214956643354730

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[opencode-agent]

Pull request overview

OpenCode cannot approve yet because required coverage evidence did not pass.

Review outcome

1. HIGH .github/workflows/opencode-review.yml:1 - Coverage evidence did not prove required test/docstring evidence

• Problem: The required coverage-evidence job result was failure, so OpenCode cannot establish approval sufficiency for this head.

• Root cause: Automated approval is only valid when the same-head coverage-evidence job proves supported repository test suites passed and configured docstring gates passed or were advisory, or reports not applicable because no supported source files or package manifests exist. Missing, failed, skipped, unavailable, or unsupported-tooling test evidence is a blocker.

• Fix: Install or configure the repository test/docstring evidence tooling when source files or package manifests exist, rerun the current-head coverage-evidence job, and approve only after it reports success with required evidence or explicit no-source not-applicable evidence.

• Regression test: Keep the approval branch checking needs.coverage-evidence.result == success before posting APPROVE, and publish REQUEST_CHANGES when coverage-evidence blocker states such as cancelled, skipped, failed, unsupported-tooling, or below-100 evidence are present.

• Result: REQUEST_CHANGES

• Reason: coverage-evidence result was failure, so required test/docstring evidence was not proven for current head dc15bed96f35ce44cfb88592bc47f3222deb8c07.

• Head SHA: dc15bed96f35ce44cfb88592bc47f3222deb8c07

• Workflow run: 32221092626

• Workflow attempt: 1

Coverage evidence

Coverage evidence job did not run or did not publish coverage evidence.

Changed-File Evidence Map

flowchart LR
  PR["PR changed files"] --> Evidence["OpenCode bounded evidence"]
  Evidence --> S1["Changed file (3 files)"]
  S1 --> I1["repository behavior"]
  I1 --> R1["Review risk: Changed file (3 files)"]
  R1 --> V1["required checks"]
  Evidence --> S2["Docs: 1019-judge-runtime-assertion-safety.md"]
  S2 --> I2["operator or user guidance"]
  I2 --> R2["Review risk: Docs: 1019-judge-runtime-assertion-safety.md"]
  R2 --> V2["docs review"]
  Evidence --> S3["Test: test_judge_runtime_assertion_safety.py"]
  S3 --> I3["regression suite"]
  I3 --> R3["Review risk: Test: test_judge_runtime_assertion_safety.py"]
  R3 --> V3["targeted test run"]

Loading

[opencode-agent]

OpenCode Review Overview

• Head SHA: 9e456b4b0c08ced956bdf0c72228fc6a0f95017d
• Workflow run: 32286166270
• Workflow attempt: 1
• Gate result: REQUEST_CHANGES (approval step)

Pull request overview

OpenCode cannot approve yet because required coverage evidence did not pass.

Review outcome

1. HIGH .github/workflows/opencode-review.yml:1 - Coverage evidence did not prove required test/docstring evidence

• Problem: The required coverage-evidence job result was failure, so OpenCode cannot establish approval sufficiency for this head.

• Root cause: Automated approval is only valid when the same-head coverage-evidence job proves supported repository test suites passed and configured docstring gates passed or were advisory, or reports not applicable because no supported source files or package manifests exist. Missing, failed, skipped, unavailable, or unsupported-tooling test evidence is a blocker.

• Fix: Install or configure the repository test/docstring evidence tooling when source files or package manifests exist, rerun the current-head coverage-evidence job, and approve only after it reports success with required evidence or explicit no-source not-applicable evidence.

• Regression test: Keep the approval branch checking needs.coverage-evidence.result == success before posting APPROVE, and publish REQUEST_CHANGES when coverage-evidence blocker states such as cancelled, skipped, failed, unsupported-tooling, or below-100 evidence are present.

• Result: REQUEST_CHANGES

• Reason: coverage-evidence result was failure, so required test/docstring evidence was not proven for current head 9e456b4b0c08ced956bdf0c72228fc6a0f95017d.

• Head SHA: 9e456b4b0c08ced956bdf0c72228fc6a0f95017d

• Workflow run: 32286166270

• Workflow attempt: 1

Coverage evidence

Coverage Decision

• Result: FAIL
• Test evidence: not proven passing
• Docstring evidence: not proven passing when configured
• Failure count: 2

Changed-File Evidence Map

flowchart LR
  PR["PR changed files"] --> Evidence["OpenCode bounded evidence"]
  Evidence --> S1["Changed file (3 files)"]
  S1 --> I1["repository behavior"]
  I1 --> R1["Review risk: Changed file (3 files)"]
  R1 --> V1["required checks"]
  Evidence --> S2["Docs: 1019-judge-runtime-assertion-safety.md"]
  S2 --> I2["operator or user guidance"]
  I2 --> R2["Review risk: Docs: 1019-judge-runtime-assertion-safety.md"]
  R2 --> V2["docs review"]
  Evidence --> S3["Test: test_judge_runtime_assertion_safety.py"]
  S3 --> I3["regression suite"]
  I3 --> R3["Review risk: Test: test_judge_runtime_assertion_safety.py"]
  R3 --> V3["targeted test run"]

Loading

[seonghobae]

@opencode-agent review
@cwl-noema-review review

Fresh re-review request: ContextualWisdomLab/.github#1136 has merged, fixing the central coverage-evidence bug (_install_trusted_uv() target-triple validation) that caused the prior REQUEST_CHANGES verdict on this PR. Please re-review exact current head dc15bed96f35ce44cfb88592bc47f3222deb8c07 — the underlying infrastructure blocker is now resolved on .github main. Do not transfer evidence from any prior head.

---

Generated by Claude Code

[opencode-agent]

Pull request overview

OpenCode cannot approve yet because required coverage evidence did not pass.

Review outcome

1. HIGH .github/workflows/opencode-review.yml:1 - Coverage evidence did not prove required test/docstring evidence

• Problem: The required coverage-evidence job result was failure, so OpenCode cannot establish approval sufficiency for this head.

• Root cause: Automated approval is only valid when the same-head coverage-evidence job proves supported repository test suites passed and configured docstring gates passed or were advisory, or reports not applicable because no supported source files or package manifests exist. Missing, failed, skipped, unavailable, or unsupported-tooling test evidence is a blocker.

• Fix: Install or configure the repository test/docstring evidence tooling when source files or package manifests exist, rerun the current-head coverage-evidence job, and approve only after it reports success with required evidence or explicit no-source not-applicable evidence.

• Regression test: Keep the approval branch checking needs.coverage-evidence.result == success before posting APPROVE, and publish REQUEST_CHANGES when coverage-evidence blocker states such as cancelled, skipped, failed, unsupported-tooling, or below-100 evidence are present.

• Result: REQUEST_CHANGES

• Reason: coverage-evidence result was failure, so required test/docstring evidence was not proven for current head dc15bed96f35ce44cfb88592bc47f3222deb8c07.

• Head SHA: dc15bed96f35ce44cfb88592bc47f3222deb8c07

• Workflow run: 32224979764

• Workflow attempt: 1

Coverage evidence

Coverage evidence job did not run or did not publish coverage evidence.

Changed-File Evidence Map

flowchart LR
  PR["PR changed files"] --> Evidence["OpenCode bounded evidence"]
  Evidence --> S1["Changed file (3 files)"]
  S1 --> I1["repository behavior"]
  I1 --> R1["Review risk: Changed file (3 files)"]
  R1 --> V1["required checks"]
  Evidence --> S2["Docs: 1019-judge-runtime-assertion-safety.md"]
  S2 --> I2["operator or user guidance"]
  I2 --> R2["Review risk: Docs: 1019-judge-runtime-assertion-safety.md"]
  R2 --> V2["docs review"]
  Evidence --> S3["Test: test_judge_runtime_assertion_safety.py"]
  S3 --> I3["regression suite"]
  I3 --> R3["Review risk: Test: test_judge_runtime_assertion_safety.py"]
  R3 --> V3["targeted test run"]

Loading

[seonghobae]

@opencode-agent Please review exact current head 9e456b4b0c08ced956bdf0c72228fc6a0f95017d against live protected main@04d0bc21a2a20693bcf16108cd76d394fe844d23. The current head is four commits ahead of the predecessor formal reviews and includes the CodeRabbit-requested deletion of .jules/sentinel.md; that inline finding is resolved/outdated. CodeQL, Semgrep, and Security Scan are terminal-success on this head and CI is regenerating. Reacquire current-head coverage/review evidence under current central .github@9e9f59f3ac1e96a960c021b131d768c238f4c21a; do not transfer the predecessor dc15bed... coverage verdict.

[seonghobae]

@opencode-agent review
@cwl-noema-review review

Review current exact head 9e456b4b0c08ced956bdf0c72228fc6a0f95017d under central .github@bbedc1a51ec1a2421f129955c629b3cd0507a4ec. Repository CI, Security Scan, CodeQL, and Semgrep are terminal-success on this SHA; the only CodeRabbit thread is resolved/outdated. Existing formal CHANGES_REQUESTED reviews evaluated predecessor dc15bed96f35ce44cfb88592bc47f3222deb8c07 and pre-#1136 central coverage. Generate fresh same-head coverage/formal review evidence without transferring predecessor verdicts.

[opencode-agent]

Pull request overview

OpenCode cannot approve yet because required coverage evidence did not pass.

Review outcome

1. HIGH .github/workflows/opencode-review.yml:1 - Coverage evidence did not prove required test/docstring evidence

• Problem: The required coverage-evidence job result was failure, so OpenCode cannot establish approval sufficiency for this head.

• Root cause: Automated approval is only valid when the same-head coverage-evidence job proves supported repository test suites passed and configured docstring gates passed or were advisory, or reports not applicable because no supported source files or package manifests exist. Missing, failed, skipped, unavailable, or unsupported-tooling test evidence is a blocker.

• Fix: Install or configure the repository test/docstring evidence tooling when source files or package manifests exist, rerun the current-head coverage-evidence job, and approve only after it reports success with required evidence or explicit no-source not-applicable evidence.

• Regression test: Keep the approval branch checking needs.coverage-evidence.result == success before posting APPROVE, and publish REQUEST_CHANGES when coverage-evidence blocker states such as cancelled, skipped, failed, unsupported-tooling, or below-100 evidence are present.

• Result: REQUEST_CHANGES

• Reason: coverage-evidence result was failure, so required test/docstring evidence was not proven for current head 9e456b4b0c08ced956bdf0c72228fc6a0f95017d.

• Head SHA: 9e456b4b0c08ced956bdf0c72228fc6a0f95017d

• Workflow run: 32251845609

• Workflow attempt: 1

Coverage evidence

Coverage Decision

• Result: FAIL
• Test evidence: not proven passing
• Docstring evidence: not proven passing when configured
• Failure count: 2

Changed-File Evidence Map

flowchart LR
  PR["PR changed files"] --> Evidence["OpenCode bounded evidence"]
  Evidence --> S1["Changed file (3 files)"]
  S1 --> I1["repository behavior"]
  I1 --> R1["Review risk: Changed file (3 files)"]
  R1 --> V1["required checks"]
  Evidence --> S2["Docs: 1019-judge-runtime-assertion-safety.md"]
  S2 --> I2["operator or user guidance"]
  I2 --> R2["Review risk: Docs: 1019-judge-runtime-assertion-safety.md"]
  R2 --> V2["docs review"]
  Evidence --> S3["Test: test_judge_runtime_assertion_safety.py"]
  S3 --> I3["regression suite"]
  I3 --> R3["Review risk: Test: test_judge_runtime_assertion_safety.py"]
  R3 --> V3["targeted test run"]

Loading

[opencode-agent]

Pull request overview

OpenCode cannot approve yet because required coverage evidence did not pass.

Review outcome

1. HIGH .github/workflows/opencode-review.yml:1 - Coverage evidence did not prove required test/docstring evidence

• Problem: The required coverage-evidence job result was failure, so OpenCode cannot establish approval sufficiency for this head.

• Root cause: Automated approval is only valid when the same-head coverage-evidence job proves supported repository test suites passed and configured docstring gates passed or were advisory, or reports not applicable because no supported source files or package manifests exist. Missing, failed, skipped, unavailable, or unsupported-tooling test evidence is a blocker.

• Fix: Install or configure the repository test/docstring evidence tooling when source files or package manifests exist, rerun the current-head coverage-evidence job, and approve only after it reports success with required evidence or explicit no-source not-applicable evidence.

• Regression test: Keep the approval branch checking needs.coverage-evidence.result == success before posting APPROVE, and publish REQUEST_CHANGES when coverage-evidence blocker states such as cancelled, skipped, failed, unsupported-tooling, or below-100 evidence are present.

• Result: REQUEST_CHANGES

• Reason: coverage-evidence result was failure, so required test/docstring evidence was not proven for current head 9e456b4b0c08ced956bdf0c72228fc6a0f95017d.

• Head SHA: 9e456b4b0c08ced956bdf0c72228fc6a0f95017d

• Workflow run: 32286166270

• Workflow attempt: 1

Coverage evidence

Coverage Decision

• Result: FAIL
• Test evidence: not proven passing
• Docstring evidence: not proven passing when configured
• Failure count: 2

Changed-File Evidence Map

flowchart LR
  PR["PR changed files"] --> Evidence["OpenCode bounded evidence"]
  Evidence --> S1["Changed file (3 files)"]
  S1 --> I1["repository behavior"]
  I1 --> R1["Review risk: Changed file (3 files)"]
  R1 --> V1["required checks"]
  Evidence --> S2["Docs: 1019-judge-runtime-assertion-safety.md"]
  S2 --> I2["operator or user guidance"]
  I2 --> R2["Review risk: Docs: 1019-judge-runtime-assertion-safety.md"]
  R2 --> V2["docs review"]
  Evidence --> S3["Test: test_judge_runtime_assertion_safety.py"]
  S3 --> I3["regression suite"]
  I3 --> R3["Review risk: Test: test_judge_runtime_assertion_safety.py"]
  R3 --> V3["targeted test run"]

Loading

[seonghobae]

@opencode-agent review
@cwl-noema-review review

Please re-review exact current head 9e456b4b0c08ced956bdf0c72228fc6a0f95017d against protected main@04d0bc21a2a20693bcf16108cd76d394fe844d23. Exact-head CI, Security Scan, CodeQL, Semgrep, coverage-evidence, package, fuzz, GPU-smoke, and Strix checks are terminal-success; focused optimization-safety tests pass locally (3 passed with the branch PYTHONPATH). Prior OpenCode verdicts evaluated predecessor heads; reacquire same-head approval and verify the removed .jules/sentinel.md artifact remains absent.

[seonghobae]

Exact-head validation for 9e456b4 completed.

• The prior TypeError was an environment-path false signal: pytest had imported /private/tmp/fast-mlsirm-pr-997/python/fast_mlsirm/llm_judge.py, not this PR.
• With PYTHONPATH=/private/tmp/fast-mlsirm-pr-1019/python, the exact PR module was imported and the focused judge suite passed: 76 passed.
• Ruff passed for the changed production/test files.
• No source change is required for this PR; its optimization-safe guards are present and pass under python -O.

Please re-run the current-head OpenCode/coverage review against this exact SHA.

[seonghobae]

Correction for exact-head validation of 9e456b4.

• The prior TypeError was an environment-path false signal: pytest had imported /private/tmp/fast-mlsirm-pr-997/python/fast_mlsirm/llm_judge.py, not this PR.
• With PYTHONPATH=/private/tmp/fast-mlsirm-pr-1019/python, the exact PR module was imported and the focused judge suite passed: 76 passed.
• Ruff passed for the changed production/test files.
• No source change is required for this PR; its optimization-safe guards are present and pass under python -O.

Please re-run the current-head OpenCode/coverage review against this exact SHA.

[opencode-agent]

Scheduled review-feedback autofix for this PR head.

• Head SHA: 9e456b4b0c08ced956bdf0c72228fc6a0f95017d

[coderabbitai]

Caution

Review failed

An error occurred during the review process. Please try again later.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[seonghobae]

Added a successor commit (`e4d1700`) reconstructing #1236 (closed unmerged due to stale branch lineage, finding confirmed valid) as a compatible addition to this branch, plus the aggregate-weight-overflow protection required by #1235:

• Plain scoring path (`category_count=None`) now derives `score` from `criterion_scores` and each `JudgeCriterion.weight` instead of trusting the model's self-reported top-level score — matching the three `category_count`-based paths, which already do this.
• All three weighted-score paths now share a `_finite_sum`/`_weighted_average` helper that rejects a non-finite aggregate criterion weight (e.g. two `1e308` weights overflowing on sum) before any contextual-orchestrator transport call, instead of silently collapsing to an incorrect finite score.

Full repo suite (4568 tests) green on this branch after a from-scratch Rust extension rebuild. Closes #1235, closes #1238.

[devin-ai-integration]

Devin Review found 1 new potential issue.

[coderabbitai]

Caution

Review failed

An error occurred during the review process. Please try again later.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[seonghobae]

@opencode-agent please re-review — this PR's CHANGES_REQUESTED review appears stale (all current checks pass on this head). This is a score-integrity hardening PR; re-approval would help unblock the fast-mlsirm backlog per ContextualWisdomLab/.github#1212.