Skip to content

Rewrite RUBRIC_DELTAS.md against rubric v4 - #12

Merged
ErikaOnFire merged 1 commit into
mainfrom
docs/rubric-deltas-v4
Sep 24, 2026
Merged

ErikaOnFire merged 1 commit into
mainfrom
docs/rubric-deltas-v4

Conversation

@ErikaOnFire

Copy link
Copy Markdown
Contributor

The gate moved to HumaneBench rubric v4 in f56f03d (rubrics/rubric_v4.md, upstream 68a872e, sha256 6e81d24…, byte-identical to humanebench main). RUBRIC_DELTAS.md still said it was pinned to v3. Docs only: no change to the code, the prompts or the vendored rubric.

What changed

  • New header: which rubric runs, the pinned commit and sha256, and that the gate vendors the rubric spec, not judge_prompt_v4.md.
  • Each delta re-checked against judge.py, prompt.md and the v4 text:
    • Dropped because v4 now has them: abstain by default, unless, the confidence filter, evidence verification, suppressions reported as covered. They're listed under "Carried from v4, not deltas".
    • Dropped because it no longer applies: "global rule 1 is not applied". v4 narrowed the rule and prompt.md no longer exempts it.
    • Added where the gate falls short of v4:
      • no escalation for a non-floor covered item whose would_have_been is -1.0
      • no default floor when humane-policy.toml is missing
      • "single closest principle" versus global rule 10
      • unresolved questions raise the verdict to review
      • one acceptance covers every finding sharing a principle+file id
      • document-mode acceptances aren't version-pinned
      • evidence matching is looser than "verbatim"
  • A short, scoped note: v3 is frozen for the published v1 benchmark, and gate verdicts can't be compared with it.
  • "Not yet reconciled" now lists two open items:
    • prompt.md says any floor finding means discuss, but the code requires -1.0.
    • The prompt never maps v4's outcomes onto the gate's schema.
  • README setup step 2 no longer calls the vendored rubric a hand-transcription.

🤖 Generated with Claude Code

The gate has run on rubric_v4.md (upstream 68a872e) since f56f03d, but the
deltas doc still described a v3 pin and compared against v3 throughout.

- State the pin: file, upstream commit, sha256, RUBRIC_TAG, and that the
  gate vendors the rubric, not judge_prompt_v4.md.
- Re-check every delta against judge.py, prompt.md and the v4 text. Drop
  the ones v4 now absorbs (abstain, unless, confidence, evidence, covered),
  drop the factual-correctness exemption the prompt no longer makes, and
  add the real v4 gaps: no would_have_been escalation, no default floor
  without a policy file, one-principle-per-finding vs global rule 10,
  shared finding ids on accept.
- Scoped note that v3 is frozen for the published v1 benchmark and gate
  verdicts are not comparable to it.
- README setup step no longer calls the vendored rubric a hand-transcription.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown

HumaneBench · advisory

🟢 Clear

This diff migrates documentation from rubric v3 to v4 and clarifies the sync-rubric workflow. It changes no code that affects users and introduces no new capability, data handling, or user-facing behavior.


🟢 clear   🟡 review   🟠 discuss   🔵 needs context   ⚪ accepted by a person. There is no red, because this check does not block anything. Scored against HumaneBench rubric v4, loaded verbatim, plus this repo's humane-policy.toml and the policy documents it names. Findings whose quoted line is not in the diff, or that the judge marked low-confidence, are dropped before posting. Deviations from v4 are in RUBRIC_DELTAS.md. Rubric 68a872e, commit ebec944.

Judged 24 Sep 2026, 08:19 UTC

@ErikaOnFire
ErikaOnFire merged commit 891fbb4 into main Sep 24, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant