Repository navigation
Rewrite RUBRIC_DELTAS.md against rubric v4 - #12
Merged
Merged
Conversation
The gate has run on rubric_v4.md (upstream 68a872e) since f56f03d, but the deltas doc still described a v3 pin and compared against v3 throughout. - State the pin: file, upstream commit, sha256, RUBRIC_TAG, and that the gate vendors the rubric, not judge_prompt_v4.md. - Re-check every delta against judge.py, prompt.md and the v4 text. Drop the ones v4 now absorbs (abstain, unless, confidence, evidence, covered), drop the factual-correctness exemption the prompt no longer makes, and add the real v4 gaps: no would_have_been escalation, no default floor without a policy file, one-principle-per-finding vs global rule 10, shared finding ids on accept. - Scoped note that v3 is frozen for the published v1 benchmark and gate verdicts are not comparable to it. - README setup step no longer calls the vendored rubric a hand-transcription. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
HumaneBench ·
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The gate moved to HumaneBench rubric v4 in f56f03d (
rubrics/rubric_v4.md, upstream68a872e, sha2566e81d24…, byte-identical to humanebenchmain).RUBRIC_DELTAS.mdstill said it was pinned to v3. Docs only: no change to the code, the prompts or the vendored rubric.What changed
judge_prompt_v4.md.judge.py,prompt.mdand the v4 text:unless, the confidence filter, evidence verification, suppressions reported ascovered. They're listed under "Carried from v4, not deltas".prompt.mdno longer exempts it.covereditem whosewould_have_beenis -1.0humane-policy.tomlis missingreviewprompt.mdsays any floor finding meansdiscuss, but the code requires -1.0.🤖 Generated with Claude Code