Repository navigation
Dual-judge runner with Cohen's κ floor + judge-key preflight - #1
Merged
Merged
Conversation
The first real capability of the substrate: every free-text clinical output
the eval rig grades goes through two independent judges. Per-batch Cohen's
kappa is computed against the published kappa_floor (default 0.6, Landis &
Koch substantial-agreement threshold) and the runner exposes a hard
reward_signal_accepted boolean that downstream training callers must check.
This guards the substrate against the binding ceiling we hit on healthcraft
V9: judge hallucination silently propagates into the reward signal unless an
agreement floor is enforced.
Includes:
- judges/base.py — Judge Protocol + JudgeQuery / JudgeVerdict dataclasses
- judges/dual_judge.py — DualJudge runner with Wilson 95% CI on per-judge
pass rates and aggregate kappa
- judges/preflight.py — pre-multi-hour-run key check that NEVER prints the
secret value, only the length (per feedback_no_secret_value_dumps lesson)
- tests using a scripted in-memory judge (no live API calls); pinned to
pass on perfect agreement, fail on κ<floor, fail-closed on prompt_id
mismatch and empty batch
Live judge implementations (Anthropic + OpenAI) land in PR #2 along with
canary requests against each provider.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…ed pytest Three independent fixes bundled because CI failed on all three: 1. secrets-scan: gitleaks/gitleaks-action@v2 now requires a paid license on org-owned repos. Replace with a direct install of the free gitleaks binary and invoke `gitleaks detect --no-banner --redact` from a step. 2. test_smoke.py: the prior Fleiss-style example accidentally produced κ=0.20, not 0.00, because observed agreement was 0.6 and expected was 0.5. Replaced with the canonical independence-table construction (n=20, each cell = 5) which provably yields κ = 0. 3. test_judges_preflight.py: drop the unused `import pytest` (ruff F401). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Two trivial format normalisations: a wrapped print line in preflight that fits on one line, and a redundant paren-group in the _ndtri Beasley-Springer-Moro denominator. No behavior change. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
bandit B101 (assert_used) flags the two paranoid asserts in manifest_check. ruff S101 was configured-out via pyproject but bandit runs separately. Convert to an explicit if/return with exit code 4 so the gate stays meaningful when running under `python -O` too (which strips asserts). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
bGOATnote
added a commit
that referenced
this pull request
May 26, 2026
PRs #1 (dual-judge core) and #2 (live Anthropic + OpenAI judges + canary) are merged. Update the continuation contract so a future agent picks up at the shift-gauntlet / CSGG work, not at the (already-done) scaffold. Notes branch protection is NOT yet enabled on this repo — follow-up once the shift gauntlet ships. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
DualJudge.evaluate(...)— the substrate's first real capability. Runs two independent judges over a batch, computes Cohen's κ + Wilson 95% CI per judge, and exposes a hardreward_signal_acceptedboolean.Why this lands first
This module is the moat. The healthcraft V9 lesson — judge hallucination is the binding ceiling — is what
reward_signal_acceptedguards against. Without this, the substrate is a slower version of every other open-source medical eval rig. With it, every model trained against this substrate has a quantified inter-judge κ on every batch.Test plan
cohens_kappapinned to Cohen (1960) Table 1 worked example (κ = 0.40 ± 0.005)DualJudgeaccepts perfect agreement, rejects κ < floor, raises on prompt_id mismatchreward_signal_accepted = False)🤖 Generated with Claude Code