From ebec944478774bf0ee1cc86f79bb2d8b35b7afb5 Mon Sep 17 00:00:00 2001 From: Erika Anderson Date: Thu, 24 Sep 2026 01:19:43 -0700 Subject: [PATCH] Rewrite RUBRIC_DELTAS.md against rubric v4 The gate has run on rubric_v4.md (upstream 68a872e) since f56f03d, but the deltas doc still described a v3 pin and compared against v3 throughout. - State the pin: file, upstream commit, sha256, RUBRIC_TAG, and that the gate vendors the rubric, not judge_prompt_v4.md. - Re-check every delta against judge.py, prompt.md and the v4 text. Drop the ones v4 now absorbs (abstain, unless, confidence, evidence, covered), drop the factual-correctness exemption the prompt no longer makes, and add the real v4 gaps: no would_have_been escalation, no default floor without a policy file, one-principle-per-finding vs global rule 10, shared finding ids on accept. - Scoped note that v3 is frozen for the published v1 benchmark and gate verdicts are not comparable to it. - README setup step no longer calls the vendored rubric a hand-transcription. Co-Authored-By: Claude Opus 5.5 (1M context) --- README.md | 5 +- RUBRIC_DELTAS.md | 191 ++++++++++++++++++++++++----------------------- 2 files changed, 99 insertions(+), 97 deletions(-) diff --git a/README.md b/README.md index 953f532..1500de4 100644 --- a/README.md +++ b/README.md @@ -50,8 +50,9 @@ Anything else is dropped by the runner, not by the model. ## Setup 1. New public repo, push this. -2. **`./scripts/sync_rubric.sh`** — the checked-in rubric is a hand-transcription - until you run this once. It replaces it with the real bytes and pins the commit. +2. **`./scripts/sync_rubric.sh --check`**: the checked-in rubric is already + byte-identical to upstream and pinned in `rubrics/VERSION`. This confirms it + has not drifted. Without `--check` it pulls the new bytes and re-pins. 3. `gh secret set ANTHROPIC_API_KEY` 4. `./demo/open_pr.sh 1` and so on, one at a time diff --git a/RUBRIC_DELTAS.md b/RUBRIC_DELTAS.md index aec40ad..b46d05c 100644 --- a/RUBRIC_DELTAS.md +++ b/RUBRIC_DELTAS.md @@ -1,109 +1,110 @@ -# How the PR check differs from HumaneBench rubric v3.0 - -`rubrics/rubric_v3.md` is vendored verbatim from the benchmark repo and is loaded -as the first half of the judge's system prompt. Every deviation is listed here. -There are no others, and none of them are made silently in the rubric file. - -| # | Deviation | Why | +# How the PR check differs from HumaneBench rubric v4 + +## What the gate runs + +| | | +|---|---| +| Rubric | HumaneBench rubric v4, `rubrics/rubric_v4.md` | +| Upstream | `buildinghumanetech/humanebench`, `rubrics/rubric_v4.md` | +| Pinned commit | `68a872eee60bedccb36b38f4ed705eb5a59a2ee9` (recorded in `rubrics/VERSION`) | +| sha256 | `6e81d241c740391042a7d82b3259bc1ccbbfaa02b71284b3a4682d7b8fc1d207` | +| Selected by | `RUBRIC_TAG = "v4"` in `humanebench/judge.py` and `scripts/sync_rubric.sh` | + +The rubric file is vendored byte-for-byte and never hand-edited. It is loaded +verbatim as the first half of the judge's system prompt. The second half is this +repo's adaptation layer, `humanebench/prompt.md` (plus +`humanebench/prompt_document.md` in document mode), followed by +`humane-policy.toml` and the policy documents it names. + +The gate vendors the human-readable **rubric**, not upstream's +`rubrics/judge_prompt_v4.md`. That judge prompt scores one assistant turn and +returns an outcome for each of the eight principles. The gate scores a diff and +returns one verdict, so it uses its own adaptation layer and its own output +schema. Everything below is where that layer, or the runner code in +`humanebench/judge.py`, departs from or adds to rubric v4. Nothing is changed in +the rubric file itself. + +**Scores are not comparable to the published benchmark.** `rubric_v3.md` is +frozen as the rubric of record for the published HumaneBench v1 results. The gate +does not use it. A gate verdict is a v4 judgment of a code change, not a +benchmark score, and must not be compared with a published v3 score. + +## Deliberate differences from v4 + +| # | Difference | Why | |---|---|---| -| 1 | **Unit of judgment.** v3 scores an AI-generated response. The check scores a diff that changes how responses are produced. | A PR is not a conversation. Every finding must name the downstream user-facing behavior, which keeps the mapping explicit rather than implied. | -| 2a | **One verdict for the pull request** (clear / review / discuss), computed in code from the org's floor, with findings folded behind it. v3 scores every principle every time. | An engineer reads the first line and decides whether to read the second. A point-by-point table makes that decision for them, and the decision is no. There is no failing verdict because the check does not block. | -| 2b | **Unresolved questions are a first-class output.** v3 has no equivalent: a response is fully visible, so there is nothing it cannot see. | A diff is not. When a verdict depends on the conversation, the policy or the screen, none of which are in the diff, the judge asks instead of scoring. This is the fix for the single-turn transparency flag that fires on a turn whose predecessor already disclosed. | -| 2c | **Floor and ceiling.** The org names a small set of principles where a **-1.0** finding raises the verdict to discuss. A -0.5 on a floor principle, or a -1.0 off it, stays at review. v3 treats all eight alike. | Severity is the organization's call, not the rubric's, and a short floor people respect beats a long one they route around. | -| 2d | **A policy document can excuse anything except a floor principle.** Below the floor, a document that permits the behavior does not close the question; the runner surfaces it as a conflict between the org's own two documents. Not in v3, which has no notion of org policy. | Without it, an organization with a permissive policy gets a quieter check, which makes this a conformance tool rather than a humane one. Their documents can make the check stricter. Below the floor they cannot make it more permissive. | -| 2e | **Suppressions are an output, not a silence.** Every finding a policy document talked the judge out of is reported in `covered`, with the document named. | Silence and deference look identical from the outside, and only one of them is trustworthy. A team that sees the check name the document that stopped it believes the next thing it says. | -| 2f | **Ask or score, never both.** A finding on the same file and principle as an open question is dropped in code. Not in v3, which has no questions. | Raising a question and filing the finding it depends on tells the engineer you had already decided and the question was decoration. | -| 2g | **The judge is shown the whole of every changed file, after the change**, alongside the diff. Evidence is still verified against changed lines only. | Five lines of context is enough to see what changed and not enough to see what it calls. "The diff does not show this guard" is a fact about the context window, not the code, and the engineer who knows the guard is there stops reading at that sentence. | -| 2h | **Findings may be conditional.** A finding can name the one fact that would make it wrong (`unless`), and the comment renders the reply that closes it. Not in v3, which judges a finished response with nothing left to ask. | A judge that cannot ask a question either guesses or stays silent, and both cost more than asking. CI cannot hold a conversation, so the question is asked where a reply can arrive. | -| 2i | **A person with write access can accept any finding, by name, with a reason.** The acceptance is recorded against their GitHub account and the commit it covers, and stops driving the verdict. Pushing after signing makes it stale and reopens the finding. | A named reviewer who wrote down why beats a wall of unexplained red, for the engineer and for whoever audits this later. The record lives in the PR conversation rather than a ledger the gate writes, because GitHub already stores authorship and edit history and will not let one account post as another. | -| 2j | **Document mode.** The same rubric and the same floor, applied to a PRD, ticket or spec instead of a diff. Absence of a consideration counts as evidence in a document and does not in a diff. | Product people do not open pull requests. By the time a decision reaches a diff it has been made, argued and scheduled, and the engineer implementing it has no standing to reopen it. | -| 2 | **Findings report -1.0 and -0.5. Commendations report +1.0 only.** +0.5 is never surfaced. | A check that praises a refactor is noise. A check that never says anything good reads as a compliance tax, which is how it gets switched off. +1.0 is the only tier that means a protection was actively added. | -| 3 | **Abstain by default.** v3 rates every response against the principles. The check returns clean unless the diff changes the range of experiences a person can have. | Direct response to the Aug 2026 judge validity audit, where ~57% of negative flags landed on mundane exchanges with no real stakes. Most PRs genuinely have no humane surface. | -| 4 | **One principle per finding, max 3 findings.** v3 has no such cap. | Same reason. An eight-row score table on every PR trains engineers to collapse the comment. | -| 5 | **Quoted evidence required, and verified.** Not in v3. | A finding that cannot quote the changed line cannot be argued with, and cannot be fixed. The runner checks the quote against the diff's added and removed lines and discards anything that does not match, so the judge cannot invent its own evidence. | -| 6 | **Confidence field, low-confidence findings dropped by the runner.** Not in v3. | The filter is code, not model judgment. Same audit: 64% of -1.0 scores had reasoning supporting only -0.5. | -| 7 | **Global rule 1 (factual correctness) is not applied.** | It governs the truth of a response's claims. A diff makes no claims. | -| 8 | **Single judge, no sampling controls.** The benchmark uses an ensemble of GPT-5.1, Claude Sonnet 4.5 and Gemini 2.5 Pro and takes the mean severity. | One judge is cheap enough to run on every PR. Ensembling is the obvious upgrade if a design partner wants it, and it is the honest fix for the variance noted below. | - -## Explicit non-violations - -The rubric is about how a capability treats a person, not about which -capabilities exist. The adaptation layer names these so the judge cannot drift -into scoring the feature instead of its treatment: - -- Memory across sessions is legitimate. Undisclosed memory, memory a person - cannot see or delete, and memory used to manufacture a relationship are not. -- Personalization is legitimate; exploiting a known vulnerability is not. -- Notifications are legitimate; ignoring a stated preference is not. -- Warmth is legitimate; parasocial framing and engagement pressure are not. -- Wanting people to return is legitimate; coercion and guilt are not. - -## Scope - -v3 was written about conversational responses. The gate applies it to the whole -stack, not the front end: deletion and retention logic, logging granularity, -ranking weights, queue and retry behavior, permission defaults, experiment -bucketing and cache consistency are all in scope, because each one sets the -range of experiences a person can have. Roughly 5% of a codebase is the surface -someone looks at; the other 95% decides what that surface is able to do. - -## Organization policy - -The gate loads `humane-policy.toml` **and the policy documents it names** alongside -the rubric: privacy policy, safety policy, retention policy, whatever the team -points it at. Where one of those documents permits what a diff does, that is not -a finding, and the comment says which document permits it. - -This is the difference between a check that has read what you wrote and one that -has not. A company that deliberately retains violating messages for safety, and -says so in its privacy policy, should not be told it is violating dignity and -safety by doing the thing it published. - -The gate loads `humane-policy.toml` alongside the rubric. v3 has no equivalent: -it scores a response against principles, full stop. A PR check has to work inside -one team's product, where the principles are shared but the thresholds are not. -So the policy file supplies the numbers and the judge is told, in the prompt, not -to substitute its own. Changes to the policy file are themselves judged. +| 1 | **Unit of judgment.** v4 scores one AI-generated turn. The check scores a diff (or a written proposal) that changes how turns are produced and what a person can do. Every finding has to name the downstream consequence for a person. | A PR is not a conversation. Naming the consequence keeps the mapping from code to experience explicit instead of implied. | +| 2 | **One verdict for the pull request**, not an outcome for each principle. The runner computes `clear` / `review` / `discuss` (or `accepted` when everything raised has been signed for) in code. The model's own `verdict` field is overwritten. There is no per-principle `not_applicable`, no HumaneScore mean and no `coverage` counts. | An engineer reads the first line and decides whether to read the second. There is no failing verdict because the check does not block anything. With no mean to compute, v4's aggregation rules do not apply. | +| 3 | **Findings report -1.0 and -0.5 only. Commendations report +1.0 only**, at most two, and only for a change that actively adds a protection. +0.5 is never surfaced. | A check that praises a refactor is noise. A check that never says anything good reads as a compliance tax. +1.0 is the only tier that means a protection was actively added. | +| 4 | **At most three findings, one principle per finding, and the prompt asks for "the single closest" principle.** This departs from v4 global rule 10, which scores every principle whose scope clause fires and does not pick a primary one. Questions and `covered` items are also capped at three each. | An eight-row table on every PR trains engineers to collapse the comment. The gate accepts some loss of v4's overlap signal in exchange for a comment that gets read. | +| 5 | **Unresolved questions raise the verdict to `review`.** They are the gate's form of v4's `insufficient_context`, each naming a principle, a file and a quoted changed line. v4 says `insufficient_context` never counts as a negative. In the gate it does not count as a finding either, but it does make someone look. | In a diff the missing fact is usually in the conversation, the policy or the screen, and only a person can supply it. A question nobody sees never gets answered. | +| 6 | **Ask or score, enforced in code.** If the judge raises a question and files a finding on the same file and principle, the runner drops the finding and keeps the question. | v4's schema gives each principle one outcome, so the conflict cannot arise there. The gate's schema allows both, so the rule is enforced after the fact. | +| 7 | **Floor from `humane-policy.toml`, and a -1.0 on a floor principle is what raises the verdict to `discuss`.** A -0.5 on a floor principle, or a -1.0 off it, stays at `review`. This is v4 Part 4. The difference: with no `humane-policy.toml`, or none naming a floor, the gate has no floor at all, whereas v4 defaults to Protect Dignity & Safety and Be Transparent and Honest. This repo's policy names those two. | Severity is the organization's call, and the policy file is where it is written down. | +| 8 | **`covered` on a floor principle becomes a document conflict and raises the verdict to `discuss`.** The runner decides this from the floor list, not from a `document_conflict` field. Off the floor, a `covered` item is shown as a one-line "Not flagged" note. **Not implemented:** v4's rule that any `covered` entry whose `would_have_been` is -1.0 escalates wherever it sits. The gate's schema has no `would_have_been` field, so a non-floor `covered` item never changes the verdict. The gate also has no code check that `covered` is empty when no policy document was supplied. It relies on the rubric text for that. | Floor behavior is v4 global rule 7. The two gaps are listed so nobody assumes they are covered. | +| 9 | **The judge sees the whole of every changed file, after the change,** alongside the diff. Evidence must still quote a changed line. | Five lines of context shows what changed but not what it calls. "The diff does not show this guard" is a fact about the context window, not the code. | +| 10 | **Evidence is checked against changed lines only**, meaning lines the diff added or removed, not the whole file. Findings, commendations, questions and `covered` items that fail the check are dropped. The match ignores whitespace and accepts containment in either direction, so it is looser than v4's "verbatim". | v4's evidence discipline, applied to a diff. The judge cannot quote unchanged code as proof of anything. | +| 11 | **Deterministic scope filter before the model.** Lockfiles, tests, docs, fixtures, build output, vendored code, `.github/` and the gate's own directories are removed before judging. A PR that touches only those returns `clear` without a model call. | The cheap, auditable half of v4 gate rule 1 (no stakes, no score). The prompt's abstain rule is the other half. | +| 12 | **The whole stack is in scope, not just copy.** The adaptation layer names these as in scope: deletion and retention, logging granularity, ranking weights, queue and retry behavior, permission defaults, experiment bucketing, and cache consistency. | Each one sets the range of experiences a person can have. v4's mechanical-output exception ("unless the output itself acts on a person") already points this way, and the gate makes it the default reading. | +| 13 | **`humane-policy.toml` supplies thresholds and is itself judged.** Where it sets a number, the judge must use that number. A diff that edits the file is in scope as a change to what the product may do to people. | v4 global rule 7 covers published policy *documents*. The gate also has team-set numbers, and those have to come from the team, not from the judge. | +| 14 | **Small changes get small scores.** Prompt hard rule 7: a constant moving one step is usually -0.5 at most, and often clean. | Diff-specific calibration on top of v4's tier discipline. | +| 15 | **Document mode.** The same rubric and floor applied to a PRD, ticket or spec. Absence of a consideration counts as evidence in a document that presents itself as complete. In a diff it does not, because the guard is usually just out of view. | Product people do not open pull requests, and a proposal is the cheapest point to change course. v4's evidence rules still apply: quote the nearest line that shows the gap. | +| 16 | **Accept-with-reason, GitHub-backed.** This is v4 Part 3. Only OWNER, MEMBER or COLLABORATOR comments count. An acceptance is pinned to the PR head at the time of the comment and goes stale when the branch moves. Two gaps against v4: (a) finding ids come from principle plus file, so two findings on the same principle in the same file share an id and one acceptance covers both, where v4 says an acceptance covers one finding; (b) in document mode an acceptance cannot be pinned to a version, so an edited proposal has to be re-run. | The record lives in the PR conversation because GitHub already stores authorship and edit history and will not let one account post as another. | +| 17 | **Principle names are the display names** ("Protect Dignity & Safety"), not v4's snake_case keys. | `judge.py` enforces the display names as a schema enum, matches floor principles by display name, and derives finding ids from their capital initials. | +| 18 | **Single judge, single run.** The default model is `claude-sonnet-4-5` (`HUMANEBENCH_MODEL`). v4 requires an ensemble, or N-of-M agreement, for any score used to make a decision, and says a single-run score is a signal that must be labeled as one. | That is why the check is advisory and posts a `neutral` check run. It must not gate a merge until it meets v4's bar. See "Known limitations". | + +## Carried from v4, not deltas + +These are v4 rules the adaptation layer restates for the diff setting. They are +listed so no one reads them as local inventions: abstain by default (gate rule +1), `unless` on findings (Part 3), tier discipline (Part 1), confidence with +`low` discarded by the runner, suppressions reported as `covered` (global rule +7), and the explicit non-violations (global rule 8): memory, personalization, +notifications, warmth and retention work are legitimate. Undisclosed or +uncontrollable memory, exploiting a known vulnerability, ignoring a stated +preference, and manufactured dependency are not. The runner enforces the +low-confidence filter on findings in code. ## Known limitations -**Verdicts are not reproducible.** The Messages API currently exposes no -temperature, top_p or top_k, so the same diff can score differently on a re-run. -This is the same instability the Aug 2026 judge validity audit found in the -benchmark itself, where one identical response scored anywhere from +0.5 to -1.0. +**Verdicts are not reproducible.** The Messages API as the gate calls it exposes +no temperature, top_p or top_k, so the same diff can score differently on a +re-run. v4 names the same instability: one identical response scored anywhere +from +0.5 to -1.0 across runs of a single judge. What the check does about it, which is mitigation and not a fix: -- the rubric is pinned, so the text being judged against never moves underneath a verdict +- the rubric is pinned, so the text a verdict was judged against never moves + underneath it - every comment records the rubric commit and the code commit it judged - low-confidence findings are dropped, which removes the least stable band - findings must quote a changed line, so a spurious one is visible as spurious -What would actually fix it: judge each diff N times and report only findings that -appear in a majority of runs, or ensemble across models the way the benchmark does. -Both cost more per PR. Worth doing before anyone lets this gate a merge. +What would actually fix it is what v4 requires for decision-grade scores: judge +each diff N times and report only findings that appear in a majority of runs, or +ensemble across models. Both cost more per PR. Do this before anyone lets the +check gate a merge. + +**Multi-turn effects are out of scope.** v4 still judges one turn at a time and +relies on session rollup to close `insufficient_context`. A diff has no session, +so the gate has no rollup. Engagement harm that only shows across turns is not +something either one measures. ## Not yet reconciled -v3 is a single-turn rubric. Multi-turn effects, which is where most engagement -harm actually lives, are out of scope for both v3 and this check. - -**The rubric is still pinned to v3, and v4 now exists upstream.** -`rubrics/rubric_v4.md` in the benchmark repo rewrites the transparency rule to -five explicit triggers, adds a per-principle applicability gate, and makes -`not_applicable` and `insufficient_context` first-class outcomes — which is most -of what deviations 2b, 3, 4 and 6 above are currently reimplementing locally on -top of v3. Moving the pin is a migration, not a sync: - -- point `scripts/sync_rubric.sh` at `rubric_v4.md` and re-pin `rubrics/VERSION` -- re-run every demo PR before merging, per the rule in the README. **The clean - PRs have to stay clean**, and 3, 6, 8 and 9 are the ones that matter -- delete the local deviations v4 absorbs rather than leaving them stated twice. - Two documents defining the same rule is how one of them goes stale -- v4's default floor is `Protect Dignity & Safety` and `Be Transparent and - Honest`, which is already what `humane-policy.toml` names. No change there -- v4 also changes the judge's output shape (an `outcome` per principle, plus - `covered` and `coverage`). This check computes its own verdict in code and - does not consume that shape, so nothing here breaks — but the rubric text the - judge reads will describe outcomes the check's SCHEMA does not offer, and that - mismatch needs a line in `humanebench/prompt.md` when the pin moves. +- `humanebench/prompt.md` says a finding on a floor principle raises the verdict + to `discuss`. The runner (and v4 Part 4) requires a **-1.0** on a floor + principle. The code is authoritative, and the prompt line should be brought + into line. +- The vendored rubric describes outcomes (`not_applicable`, + `insufficient_context`, `coverage` counts) that the gate's output schema does + not offer. The prompt does not map them explicitly. In practice the judge uses + `unresolved` for `insufficient_context` and returns nothing for + `not_applicable`. Adding one explicit line to `prompt.md` would remove the + ambiguity. + +## Moving the pin + +`.github/workflows/rubric-sync.yml` opens a PR weekly when upstream's +`rubrics/rubric_v4.md` changes. Merging it changes what every future verdict +means. Before merging, re-run the demo pull requests. The clean ones have to +stay clean. Then re-check every row above against the new text.