Repository navigation
Live Anthropic + OpenAI judges + canary preflight - #2
Merged
Merged
Conversation
The substrate now has working judges, not just a runner contract. Both judges
implement the Judge Protocol, return JudgeVerdicts with parsed labels, and
ship with the right per-provider conventions:
AnthropicJudge (default: claude-opus-4-7)
- No temperature/top_p/top_k (would 400 on Opus 4.7)
- No budget_tokens (also removed on 4.7) — adaptive thinking left off
for grading (deterministic-ish, short prompts)
- System prompt sent with cache_control: {"type": "ephemeral"} so prompt
caching activates once the rubric grows past 4096 tokens; below that
the marker is silently inert (no cost, no error)
- Structured output via output_config.format (NOT prefill — prefills
400 on 4.7)
- Lazy SDK import so the module imports cleanly without the optional
[judges] extra
OpenAIJudge (default: gpt-5.4)
- Uses max_completion_tokens, not the legacy max_tokens
(per the scribegoat2 incident memory)
- Strict response_format with the same JSON schema as Anthropic for
apples-to-apples comparison
Canary preflight (judges/preflight.py)
- Real tiny request to each provider whose key is set; fails fast on 401
- Length-only secret logging — never echoes the value
(per feedback_no_secret_value_dumps)
- --strict makes missing keys an error (exit 3); default mode is warning-
only so CI without keys still passes
- run_preflight() accepts an injected canaries dict for hermetic tests
Decision parsing (judges/decision.py)
- Lenient JSON extraction: strict JSON, fenced JSON, or embedded in prose
- Validates label ∈ {0, 1}, rationale is str
- Rejects bool labels explicitly (bool is a subclass of int in Python)
Tests: 51 hermetic tests covering all paths above. No live API calls; both
judges accept an injected client.
CI: unit job now installs .[dev,judges] so the new tests can import the
SDKs (no SDK calls in the test path).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
bGOATnote
added a commit
that referenced
this pull request
May 26, 2026
PRs #1 (dual-judge core) and #2 (live Anthropic + OpenAI judges + canary) are merged. Update the continuation contract so a future agent picks up at the shift-gauntlet / CSGG work, not at the (already-done) scaffold. Notes branch protection is NOT yet enabled on this repo — follow-up once the shift gauntlet ships. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Why this lands now
`DualJudge` shipped last PR but had no real judges to drive — it could only be exercised with scripted in-memory stubs. Both downstream model repos (`medimage-model`, `medimage-model-research`) need live judges to do any real eval. This is the unblock.
Provider-specific gotchas baked into the implementations
Test plan
🤖 Generated with Claude Code