MIT-661: eval grader v2, word-start matching, no version-only causes, versioned results

docs/decisions/2026-11-08-MIT-661-grader-v2.md

@@ -0,0 +1,44 @@
+# Eval grader v2: words from a word's start, no version-only causes, versioned results
+Status: Accepted
+Date: 2026-11-08
+
+Context: The eval grader (`tools/eval/runner.py` `grade_proposal`, decision 0023) matched
+`cause_words` and `symptom_words` as case-insensitive substrings of the proposal. Short words
+matched in the middle of unrelated ones: "utc" in "outcomes", "view" in "reviews", "ops" in
+"drops", "9.2" in "9.21", "polars 2" in "polars 2026". A proposal could also get
+`cause_right` from a version alone ("broke in pytest 9.2"), which says when the cause arrived,
+not what it is. Changing the grader changes what "good proposal" means, so results graded
+before and after can't be pooled.
+
+Decision:
+1. **Words match from a word's start** (`runner.mentions`, for `cause_words` and
+   `symptom_words`). After lowercasing, no `[a-z0-9_]` character may come just before the word.
+   A word that ends in a digit must also end there, so "9.2" doesn't match "9.21" and
+   "polars 2" matches "polars 2.0" but not "polars 2026". Any other word may run on, so stems
+   and plurals written for substring matching keep working: "truncat" matches "truncated",
+   "round" "rounding", "offset" "offsets", "overwr" "overwritten". Every false hit above starts
+   mid-word, so the left boundary alone stops them. A word starting with punctuation (".prom")
+   needs nothing before it. Words may hold spaces, dots or hyphens.
+2. **Version words never make the cause right.** A cause word that is only a version (digits
+   and dots, an optional leading `v`), maybe after the discovery's `library` or "python", is a
+   version word: "9.2", "2.14", "0.11", "polars 2" (for polars), "python 3.14"
+   (`runner.is_version_word`). Any other prefix makes it part of a cause, not a version's
+   label: "pep 649", "rfc 3339". The data keeps version words; the grader skips them for
+   `cause_right`. Tests check that every measured scenario and every GitHub task has at least
+   one cause word that is not a version word, and that the skipped words are exactly 9.2, 2.14,
+   0.0.45, 0.11, polars 2, 3.14, 3.0, 2.1 and 2.0.
+3. **Every result records its grader.** `runner.GRADER_VERSION = 2`. Every new ledger line and
+   per-session JSON carries `grader_version`; a result without it was graded by version 1.
+4. **No figure crosses graders.** `agent_eval.py` `report`, `compare` and `effects` print a
+   section per grader version, and `trials` a row per version. `compare` gives each version its
+   own verdict, from that version's results alone, and says so when a condition holds more than
+   one. The pre-registered verdict (the arms record) is read from one version's section.
+5. **Proposals keep their text.** Each graded proposal stores its cause and fix, passed through
+   the same scrubber and cap as the patch excerpt (`runner.excerpt`: secrets redacted, then cut
+   at 4,000 characters), so a later grader can score it again. Older results, without them,
+   still load.
+
+Consequences: v2 rates of good proposals are not comparable with v1 rates; compare like with
+like, or re-run under v2. v1 results can't be re-scored, since they kept no cause or fix. A
+word that legitimately occurs as a prefix of a longer unrelated word ("end" in "endpoint")
+still matches; a scenario list that hits this needs a longer word.

docs/eval/README.md

@@ -63,6 +63,38 @@ Everything needed to measure the agent-facing text again is in the repo: the har
   `--setting-sources project --strict-mcp-config` so no hooks or servers run. Ask what led to the
   decision, not for its reasoning: asking a model to quote its own reasoning trips a safeguard.
 
+## Grading
+
+`runner.grade_proposal` judges each proposal against its scenario's `discovery`. A **good
+proposal** is all three:
+
+- **on topic**: a `symptom_words` entry appears in the symptom;
+- **cause right**: a `cause_words` entry that is not a version word appears in the cause or fix;
+- **pinned**: the library appears with a version in `environment.libraries` (or, for a runtime
+  change, in `environment.language`).
+
+**Grader v2** (2026-11-08, `docs/decisions/2026-11-08-MIT-661-grader-v2.md`) is the current
+grader (`runner.GRADER_VERSION`):
+
+- **Words start at a word's start.** Ignoring case, no letter, digit or underscore may come
+  just before a word, so "utc" is not in "outcomes", "view" not in "reviews", "ops" not in
+  "drops". A word ending in a digit must also end there: "9.2" is not in "9.21", and
+  "polars 2" is in "polars 2.0" but not in "polars 2026". Any other word may run on, so stems
+  and plurals still match: "truncat" in "truncated", "round" in "rounding", "offset" in
+  "offsets". A word starting with punctuation (".prom") needs nothing before it.
+- **Version words never make the cause right.** A cause word that is only a version, maybe after
+  the discovery's `library` or "python" ("9.2", "2.14", "polars 2" for polars, "python 3.14"),
+  stays in the data and is skipped for
+  `cause_right`. Every measured scenario and GitHub task must have another cause word (a test).
+- **Kept text.** Each proposal in a session's JSON keeps its `cause` and `fix`, scrubbed and cut
+  like the patch excerpt, so a later grader can score it again.
+
+Version 1 matched substrings and counted version words. Every new ledger line and session JSON
+records `grader_version`; one without it (everything committed before 2026-11-08) is version 1.
+`report`, `compare`, `trials` and `effects` keep grader versions apart: `report`, `compare` and
+`effects` print a section per version (`compare` a verdict per version), and `trials` a row per
+version, so no figure pools two graders.
+
 ## GitHub tasks
 
 A GitHub task replays a real issue from a real repository: the agent gets the project at the
@@ -201,6 +233,7 @@ Each result and ledger line also records, in every arm and in `run` and `search`
 | `patch_truncated` | true when the diff is not all of the agent's change: a changed file was over 256 KiB, or the diff over 1 MiB |
 | `patch_hash` | 16 hex digits of sha256 over the whole diff of what was read, taken with whitespace removed and blank lines dropped, so whitespace-only differences share a hash |
 | `repeat` | which of `--runs` repeats of its task, arm and model the session was (1 to N); 0 outside `arms`. Unlike `run`, the same repeat pairs across arms |
+| `grader_version` | the grader that judged the session's proposals (`runner.GRADER_VERSION`, see Grading); lines and results without it are version 1, and no report pools two versions |
 
 Test files (under `tests/`, `test/`, or `__tests__/`, or named `test_*`, `*_test.*`,
 `conftest.py`, `*.test.*`, `*.spec.*`) are left out of the patch, as is generated output:
@@ -758,9 +791,12 @@ scenario, just no longer collected. Report `real-polars-explode-empty` on its ow
 a baseline of its own; a hold-out total from before this change (four first-set scenarios
 including whenever) is not the same set as one after it.
 
-**Known limit of the grader.** `cause_words` are matched as substrings, so generic words in
+**Known limit of the grader (v1).** Grader v1 matched `cause_words` as substrings, so generic words in
 other hold-outs' lists also occur in the shipped text: "removed" (both Python 3.14 scenarios),
 "naive" (in "naive_arithmetic_ok", `real-sqlmodel-utc`), "view" (in "reviews",
 `hard-pandas-cow-subset`) and "utc" (in "outcomes", `hard-sqlmodel-legacy-rows`). None of them
 gives a cause away, but the shipped text as a whole can't be held to the example check until
-those lists change, which would change how those scenarios grade.
+those lists change, which would change how those scenarios grade. Grader v2 (see Grading)
+matches only from a word's start, so "view" and "utc" no longer match inside "reviews" and
+"outcomes" ("naive" still starts "naive_arithmetic_ok"); the
+example checks in the tests still match substrings, which catches more.

server/tests/test_eval_arms.py

@@ -336,6 +336,32 @@ def test_old_ledger_lines_still_load_in_trials_and_effects(
     assert "w-a0b4c5d1e0f1g1h0n0s1" in pooled  # the old line is a config of its condition
 
 
+def test_trials_and_effects_never_pool_grader_versions(
+    tmp_path: Path, monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str]
+) -> None:
+    """A line without `grader_version` is v1; a v2 line of the same text is its own group."""
+    ledger = tmp_path / "trials.jsonl"
+    monkeypatch.setattr(tools.agent_eval, "LEDGER", ledger)
+    regraded = {**OLD_LINE, "grader_version": 2, "variant": "w-a1b0c0d0e0f0g0h0n0s0"}
+    same_text = {**OLD_LINE, "grader_version": 2, "good_proposals": 1}
+    lines = [OLD_LINE, same_text, regraded]
+    ledger.write_text("".join(json.dumps(line) + "\n" for line in lines), encoding="utf-8")
+
+    assert tools.agent_eval.by_variant() == 0
+    rows = [line.split() for line in capsys.readouterr().out.splitlines()[1:]]
+    old = [row for row in rows if row[1] == OLD_LINE["variant"]]
+    # Columns: arm, variant, text, trigger, n, search, propose, good, fixed, grader.
+    assert [(row[4], row[7], row[-1]) for row in old] == [("1", "0", "v1"), ("1", "1", "v2")]
+
+    assert tools.agent_eval.effects(argparse.Namespace(condition="cross-final")) == 0
+    first, second = capsys.readouterr().out.split("== cross-final: 2 sessions, grader v2")
+    assert "== cross-final: 1 sessions, grader v1" in first
+    assert "w-a1b0c0d0e0f0g0h0n0s0" not in first
+    assert "w-a1b0c0d0e0f0g0h0n0s0" in second
+    assert "per option" in first
+    assert "per option" in second
+
+
 def test_herding_reads_a_lines_excerpt_and_an_older_lines_whole_patch_by_hash(
     tmp_path: Path, monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str]
 ) -> None:
@@ -356,5 +382,9 @@ def test_herding_reads_a_lines_excerpt_and_an_older_lines_whole_patch_by_hash(
     assert "patch" not in rows[1]
     assert rows[1]["patch_excerpt"] == "-x\n+ y"
     assert tools.agent_eval.herded(rows) == 2
+    # The older line is grader v1 and the new ones v2, and `compare` reports graders apart.
     assert tools.agent_eval.compare(argparse.Namespace(condition="arms")) == 0
-    assert "share their task's top wrong patch 2/3" in capsys.readouterr().out
+    first, second = capsys.readouterr().out.split("== arms: 2 sessions, grader v2")
+    assert "== arms: 1 sessions, grader v1" in first
+    assert "share their task's top wrong patch 0/1" in first
+    assert "share their task's top wrong patch 0/2" in second

server/tests/test_eval_compare.py

@@ -198,6 +198,33 @@ def test_the_verdict_counts_both_folds_of_github_arms_and_never_same_task_lines(
     assert "verdict: continue" in capsys.readouterr().out  # reported apart, never pooled
 
 
+def test_compare_never_pools_grader_versions_and_gives_each_its_own_verdict(
+    tmp_path: Path, monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str]
+) -> None:
+    """Lines without `grader_version` are grader v1; v2 lines of the condition are apart."""
+    ledger = tmp_path / "trials.jsonl"
+    monkeypatch.setattr(tools.agent_eval, "LEDGER", ledger)
+    old = arm_runs("both", [3] * 10, 3) + arm_runs("context7", [1] * 10, 3)  # a clear win
+    equal = [3, 2, 0, 1, 3, 3, 2, 1, 0, 3]
+    new = [
+        {**row, "grader_version": 2}
+        for row in arm_runs("both", equal, 2) + arm_runs("context7", equal, 2)
+    ]
+    ledger.write_text("".join(json.dumps(row) + "\n" for row in old + new), encoding="utf-8")
+
+    assert tools.agent_eval.compare(argparse.Namespace(condition=CONDITION)) == 0
+    out = capsys.readouterr().out
+    assert f"{CONDITION} holds results of graders v1, v2" in out
+    first, second = out.split(f"== {CONDITION}: 40 sessions, grader v2")
+    assert f"== {CONDITION}: 60 sessions, grader v1" in first
+    assert "primary: both vs context7 over 10 tasks" in first
+    assert "verdict: continue" in first
+    assert "verdict: kill" not in first
+    assert "verdict: kill" in second  # the v2 lines alone: equal arms
+    assert "verdict: continue" not in second
+    assert "grader v1" not in second
+
+
 def test_compare_with_no_trials_says_so(
     tmp_path: Path, monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str]
 ) -> None: