[ZachFara]

What changed

• Matching from a word's start (tools/eval/runner.py mentions). This applies to cause_words and symptom_words, after lowercasing.
  • No [a-z0-9_] character may come just before the word. That stops "utc" in "outcomes", "view" in "reviews", "ops" in "drops" and "2.0" in "21002.0".
  • A word that ends in a digit must also end there. So "9.2" doesn't match "9.21", and "polars 2" matches "polars 2.0" but not "polars 2026".
  • Any other word may run on, so stems and plurals keep working: "truncat" matches "truncated", "round" matches "rounding", "offset" matches "offsets", "overwr" matches "overwritten".
  • A word that starts with punctuation (".prom") needs nothing before it.
  • No scenario data changed.
• Version words never count toward cause_right (is_version_word). A version word is digits and dots with an optional leading v. It may have one prefix, but only the discovery's library or "python": "9.2", "pytest 9.2" (for pytest), "polars 2" (for polars) and "python 3.14" are version words; "pep 649" and "rfc 3339" are not. That is why is_version_word(word, library) takes the library. The data keeps these words; the grader skips them for cause_right only. Across measured and retired scenarios, the skipped words are exactly 9.2, 2.14, 0.0.45, 0.11, polars 2, 3.14, 3.0, 2.1 and 2.0, the same set as before the prefix rule changed. A test pins that set.
• GRADER_VERSION = 2 is recorded on every per-session JSON (Result.grader_version) and on every ledger line. A result without it is version 1 (agent_eval.grader_version).
• No figure mixes grader versions (agent_eval.by_grader).
  • report, compare and effects print a section per version (== <condition>: N sessions, grader vN).
  • compare gives each version its own verdict and notes when a condition holds more than one.
  • trials groups by version too, with a new last column, grader.
• Cause and fix text is stored on Proposal (cause, fix, defaulting to ""). It goes through the patch excerpt's scrubber and cap. I renamed patch_excerpt to excerpt, since it now handles any text, and updated both callers. Old JSON files and ledger lines without these fields still load.
• Tests (test_eval_grader.py, test_eval_compare.py, test_eval_arms.py):
  • matching: utc/outcomes, view/reviews, ops/drops, 9.2/9.21, polars 2/polars 2.0 and polars 2026, round/rounding, truncat/truncated, offset/offsets, overwr/overwritten, and punctuation edges;
  • grade_proposal itself uses whole-word-start matching for both lists: with symptom_words=("view",) and cause_words=("utc",), a proposal saying "reviews" and "outcomes" gets on_topic and cause_right both False, and one saying "view" and "UTC" gets both True. With substring matching, this test fails;
  • which words count as version words, for which library;
  • a proposal whose only hit is a version word gets cause_right=False;
  • cause and fix are stored scrubbed and capped;
  • a saved JSON and a ledger line both carry grader_version 2;
  • the committed docs/eval/baseline-holdout results report as grader v1;
  • compare gives each version its own verdict;
  • trials and effects split by version;
  • every measured scenario, multi-turn second task and GitHub task has a cause word that is not a version word. All pass, so there is no failing list.
• Docs. I added a "Grading" section to docs/eval/README.md and a grader_version row to its ledger field table. The decision record is docs/decisions/2026-11-08-MIT-661-grader-v2.md, named by date: docs/decisions/README.md says records after 0038 never use a sequence number.
• Existing tests whose expectations changed:
  • test_herding_reads_a_lines_excerpt_and_an_older_lines_whole_patch_by_hash (test_eval_arms.py). Its older ledger line has no grader_version, so it is v1, and the appended lines are v2. It expected compare to pool their herding (2/3), but this ticket says compare must not pool grader versions. The pooled herded(rows) == 2 check is kept, so a line's excerpt and an older line's whole patch still herd by hash. The compare assertion now checks each version's own section: 0/1 for v1 and 0/2 for v2.
  • test_the_ledger_records_the_skill_and_old_lines_still_load (test_eval_skill.py). The old v1 baseline line and the new v2 baseline line are now separate trials groups. The assertion that old lines still load is unchanged.

Why

Ticket: MIT-661. Substring matching gave false hits that start mid-word ("utc" in "outcomes", "9.2" in "9.21"). A bare version also counted as naming the cause, though a version only says when the cause arrived. Because v2 changes what a good proposal means, every result now records which grader scored it, and no figure pools two graders. Storing cause and fix lets later grader changes re-score new results.

How it was tested

• scripts/test-db.sh uv run pytest server/tests/test_eval_*.py server/tests/test_variants.py: 279 passed.
• make check (origin/feature/eval-setup already merged and unchanged since): all checks passed.
• No eval sessions and no containers were run.

Learned

• v1 results can't be re-scored. Committed results and the ledger kept no cause or fix text, so only results from v2 onward can be re-scored.
• Prefix hits still match. A word that is the start of a longer, unrelated word still matches, for example "end" in "endpoint". A list that hits this needs a longer word.
• The tests' example checks still match substrings, deliberately. cause_words_in_shipped_text in test_variants.py and the skill/example check catch a superset of what v2 matches. Their docstrings ("matched the way the grader does") now describe v1.
• Headers changed. trials has a new last column, grader. The section headers of report, compare and effects now end in , grader vN. Substring assertions on the old headers still pass.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Qc1CQTb8r67J32HpkhS43y