feat(tools): durable per-request results and strict stdout classification in the eval harness

[monotophic]

Authored by Opus 5 in Claude Code, analysis in partnership with @monotophic

Revised 2026-09-17 in response to review. This PR is now the durability half only. The
ablation-evidence checker and its release-archive gate have moved to #1356, where their writer lives, and the
SCORE_EVIDENCE identity binding is removed. Title updated to match: the previous one advertised two things that
are no longer here.

The evaluation harness reads scoring output off the engine's stdout, and today it trusts whatever it is handed: a
line it does not recognise is silently absorbed, and results are written only when the whole run finishes, so a
run that dies mid-task leaves nothing behind.

This changes both.

tools/eval_glm.py gains two properties it did not have. Results are appended per request, so a run that dies
mid-task leaves a valid partial file with an INCOMPLETE marker and still prints the accuracy table over the rows
that landed. And every line of the scoring stream is classified against the engine's documented SCORE-mode
output, so an unrecognised line is refused by name instead of silently absorbed.

tools/engine_evidence.py owns the grammar of the engine's two typed startup lines — the == GLM C engine banner
and the loaded in … record — so every consumer agrees by construction instead of each re-deriving the same
regular expressions and the same normalisation. It is the shared foundation the other two PRs in this group also
carry; see the note at the end.

Context. This PR is one of five independent contributions derived from a single locally-verified working tree
(the fp8 container line in #1102, the OpenAI-compatible server hardening in #1353, this harness work, the engine's
ablation scoring mode in #1356, and an engine-transcript check tool in #1357). This one carries the evaluation
harness and the shared preamble grammar. It stands alone — its whole suite is green with the rest of the set
absent (97 cases on a checkout of dev carrying only these five files), and merging or declining the others does
not affect it. The others are proposed separately, each with its own evidence. The set-level explanation and a traceability matrix covering every request across the three PRs are in the #1355 comment: .

Behavioral contract

• Everything the harness did before, it still does on its own inputs: task loading, per-option scoring,
  exact-vs-greedy accounting, and the output format existing consumers read are unchanged, including --dry
  without a vocabulary and the accuracy table on a partial run.
• A run interrupted at any point leaves every already-scored row on disk, complete and parseable, with no
  truncated trailing record.
• A line the engine never prints on the scoring stream is refused by name; it is never absorbed.
• The preamble grammar accepts exactly what the engine really prints, including every kernel name it can emit, and
  rejects what it does not.
• No test here runs a model, and none needs a GPU or a container.

Capstone matrix — one decisive artifact per claim; each would fail if the claim were false.

claim	decisive evidence
the durability guard actually guards durability	delete the per-row out_f.flush() at tools/eval_glm.py:601 and tests/test_eval_glm.py::test_row_is_durable_on_disk_before_the_run_finishes fails. The older test named for the same property — test_result_rows_are_written_and_flushed_incrementally — stays green under that mutation, which is why the new one exists
the guard is aimed at the right line	the same deletion applied to the CSV header flush at :547 does not fail the test. It pins the per-row property, not the presence of a flush() call
the property holds against a REAL engine under a hard kill	an integration run on a Linux host: eval_glm.py driven against a real container, polled until three complete rows were on disk, then SIGKILLed — not a graceful stop, so the interpreter's flush-on-close never ran. A second, independent file handle opened afterwards found 3 complete rows, 9 fields each, no truncated trailing line. The control is what is missing: the file carries no # finished record, which the harness writes last — proof the process really was killed mid-run rather than allowed to finish
rows land on disk while the run is still in progress	the harness's own main() driven end to end with rows observed on disk 1→2→3→4 mid-run, read through a second independent file handle. Stated exactly: the engine is a fake subprocess.Popen returning canned stdout, as this suite's own docstring says — no engine is launched here. The property under test is the harness's write-and-flush behaviour, which is where the defect was; proving it against a live engine is an integration cell, not a unit test
foreign lines are refused, not absorbed	a direct differential pinned by two tests in this branch: test_dev_copy_silently_accepts_a_foreign_numeric_line proves dev's classifier accepts the line 1 1 1, shaped like its own grammar; test_new_copy_refuses_the_same_foreign_line_by_name proves this branch raises on the identical input. Both run against the two implementations, not against a description of them
the preamble grammar is pinned to real output, not to itself	every entry of the kernel roster is asserted against a frozen literal independent of the parser's own table — a same-length rename (neon→neom) fails the suite. An earlier fix that derived the expectation from the roster it policed passed that mutation, which is the reason for the literal
the harness still behaves like dev's where it should	RAN at this head, against dev's own copy of eval_glm.py extracted from origin/dev: the CLI surface is option-for-option identical, and --selftest produces byte-identical output from both, each exiting 0. Scope stated plainly — this covers the accuracy-table and scoring-reduction path; it does not cover --dry against a real snapshot, which needs a container and is not run here

Fuller matrix, what changed since the reviewed head, and origin accounting

What changed since 5e9b1131, the head reviewed on 2026-09-17

change	origin
the stale pack_python.py import-closure paragraph is gone from this body	maintainer-requested — and dev-motion: #1364 fixed it on dev, so this branch's diff for that file is empty
SCORE_EVIDENCE identity binding removed (6 references → 0), and the UNBOUND status with it (4 → 0)	maintainer-requested — nothing on dev reads that variable, so the reader shipped ahead of its writer
tools/check_ablate_evidence.py, its test, and the release.yml gate moved to #1356	maintainer-requested, resolved differently — its writer is #1356's coli-ablate/2 mode, so it moved rather than being parked
the durability guard above	re-review-found — the feature worked; the test named for it could not detect losing it
the kernel-roster literal	re-review-found — the roster was unvalidated data, and the first fix for it was itself insensitive to content

requirement	instrument	result
helper grammar is source-true	the banner and loaded in regular expressions read against the engine's own format strings and the kernel names it can print	match, including the draft upper bound the engine clamps to
incremental durability	kill the harness mid-run and read the file through a second handle	valid partial file, INCOMPLETE marker, table printed, no truncated trailing record
foreign-line refusal	lines the engine never prints on the scoring stream, driven through the classifier main() actually uses	each refused by name; the dev/new differential above is the decisive pair
numeric grammar	both the six-decimal form the engine ships today and the older %.17g form, plus nan/inf/-inf	both accepted; fixtures are non-dyadic
suite runs with nothing else from the set present	python3 -m unittest tests.test_engine_evidence tests.test_eval_glm tests.test_pack_python on dev + these five files	Ran 97 tests in 0.311s … OK

The shared file, disclosed. tools/engine_evidence.py and tests/test_engine_evidence.py — 524 lines, two
files — are byte-identical in this PR and #1357. #1356 no longer carries them: its revision left it importing
nothing from the module, so it was dropped there rather than shipped unused. That duplication is present by design until this PR lands;
the other two then merge dev and shed their copies mechanically. Merging this PR first gives the smallest
review surface for the other two, but any order merges cleanly — all six orderings were built in a scratch
worktree and produce one identical tree with zero conflicts.

A limitation we are disclosing rather than fixing here. The wire-format literals in this tooling are hand
transcriptions of formats printed by c/colibri.c. Drift in the Python module is pinned by the tests above;
drift in the C source is not — no test in this branch reads a C file. That predates this revision and a fix
would reopen the shared file across all three PRs, so it is noted as a follow-up rather than folded in.

Scope note: no file outside the five listed is touched, and no surrounding code was reformatted.

Durable vs current state: the harness properties, the preamble grammar and the refusal contract are durable.
The counts (97 cases) are current state, measured 2026-09-17 against base cc756a63 on macOS (Apple silicon).

[JustVugg]

Conflict resolved on your branch, pushed as 9e96637. No rebase needed on your side -- pull before your next commit.

What I did, so you can check rather than trust it:

• tools/pack_python.py takes dev's version wholesale. It is a strict superset of this branch's: I diffed them, and the only lines unique here were the three your own commit replaced (the old two-edge docstring, the scripts, script_paths init, the old summary print). Nothing of yours was dropped.
• tests/test_pack_python.py takes dev's version and keeps your engine_evidence assertion as an additional case, not a replacement. On dev it could not run, since engine_evidence.py is introduced here. On this branch it pins the edge this branch adds, sitting next to the one that pins the historical bug. Both are real-tree assertions and they defend different things.

12 tests, green locally. CI is running.

One thing I noticed while verifying, worth a decision from you

tools/check_ablate_evidence.py is not in the computed set:

engine_evidence.py           True
eval_glm.py                  True
check_ablate_evidence.py     False

Not a defect in your change: nothing in the code names it, and it takes argparse arguments, so its caller is a person. Static analysis cannot reach a human. tools/k3_tokenizer.py is in exactly the same position and release.yml copies it explicitly for that reason.

So it is a decision, not a bug: if the checker is meant to be usable from a release archive, it wants the same explicit copy plus a test -f gate. If it is a repository-side tool for people working on the engine, it is already right as it is. You know which one you intended.

Thanks again for #1359. Cherry-picking your commit rather than rewriting it was the least I could do after duplicating another contributor's work last week by not checking the PR list first.

[monotophic]

@JustVugg: Thanks for patching the bugs, resolving the conflict on this branch, and the write-up. I checked the merge: pack_python.py at 9e96637 is dev's file with nothing of ours dropped, and test_pack_python.py keeps the engine_evidence case as an added assertion. I have pulled 9e96637 and have not rewritten the branch.

On check_ablate_evidence.py: it should ship in the release archive. It verifies the evidence that eval_glm.py produces, and that harness is in the archive, so anyone running a released engine and producing ablation evidence wants the checker beside it. Added in ba2802a: the same copy and test -f gate that release.yml gives k3_tokenizer.py, plus a parse check, and a real-tree test that pins both halves (the walk does not reach either human-only tool; the workflow copies and gates each).

#1356
and
#1357
stack on this branch; I merged 9e96637 into each the same way rather than rebasing, so their histories stay put.

[Authored by Claude Fable 5.1 in Claude Code, analysis in partnership with @monotophic.]

[monotophic]

Authored by Claude Fable 5.1 in Claude Code, analysis in partnership with @monotophic

Merged current dev (87c54a93) into this branch — no functional change on this PR's side. The only file dev motion touched here is .github/workflows/release.yml, where this PR's three hunks (copying, gating and parse-checking check_ablate_evidence.py in the release archive) sit beside two unrelated dev commits (d46b7154 SHA-pinning of third-party actions; 882c102f deepseek_v41 in the build matrix). All merged automatically. The other seven files are unchanged from what was already under review. This PR's test modules (177 tests) pass on the merged head; make check passes on #1356, which contains this branch.

@JustVugg similar steering request to #1102 and #1353, this PR is part of the set to make the full checkpoint-quality GLM 5.2/5.3 containers work and to establish the OpenAI API instrumentation needed for container quality studies. Let me know if this PR is off target or otherwise not likely to merge. Thanks for all you do, this project is awesome :)

[JustVugg]

Reviewed. The durability half is worth having on its own: appending results per request so an interrupted run leaves something behind, and classifying every stdout line instead of absorbing the unrecognised ones, are both real.

Two things in the body no longer match the diff, and I would rather not merge a PR whose description misdescribes it.

The pack_python.py import-closure gap you describe as a pre-existing defect on dev, with a promise to open a separate issue, is already fixed on dev, merged as #1364. Your diff for that file is empty. What survives is the test additions plus the release.yml gate.

And the evidence half is inert as it stands. c/tools/eval_glm.py sets SCORE_EVIDENCE=1 in the child environment, and I checked the dev tree: nothing reads it, zero files, against a positive control that finds coli_omp_tune in eleven. Your own docstring says so. So the harness reports UNBOUND on every build that exists.

My suggestion is to split: merge the durability fix now, with the body corrected, and bring the binding back when an engine actually writes that record. Carrying a reader ahead of its writer is the same pattern you flagged and offered to drop yourself in #1353.

[JustVugg]

One more thing that applies to the set rather than to this PR, and it is the single change that would most help review.

#1356 and #1357 each contain this PR's engine_evidence.py and check_ablate_evidence.py byte for byte, same blob hashes, along with four identical commits. So the four PRs are not 22,000 lines, they are about 15,300 with 6,700 counted three times.

That also makes two statements stale. #1356 says one round-trip case fails until this PR lands, and #1357 says both modules fail to import without it. Neither is true of the diffs as pushed: each already carries both files.

Could you rebase #1356 and #1357 onto this branch's head, so each diff shows only its own work? It cuts the review surface by nearly a third and makes the bodies true again.

[monotophic]

Thank you — this was clear and constructive feedback. The only two places where we found different numbers, you had been charitable. We did a deep review on the set of three (1355, 1356, and 1357) and also integration testing with 1102 and 1353 on current dev (as of this posting). I think that what we've done is as good or better than your asks, but if you differ on our "improvements" we can deliver your asks directly.

Two things first: the two places we didn't do what you asked to the letter.

No rebase, but with reason

You asked us to rebase #1356 and #1357 onto this branch's head so each diff shows only its own work. We are not
doing that literally, because your own merge commit 9e966378 is an ancestor of all three branches, and a rebase
would force-push over it.

We served the goal a different way: #1356 and #1357 were rebuilt fresh from current dev, carrying only their own
files. Each is a forced update, and the result is the diff-shrink you asked for — the set goes
from 34 files and +16,228 to 21 files and +8,178. 9e966378 is deliberately no longer an ancestor of #1356/#1357;
it stands on this branch, where you pushed it, untouched.

We chose the rebuild over the rebase because it re-parents reviewed content rather than rewriting it. For #1357 the
rebuilt tree is byte-identical to the head our own review cleared (af9c25f057); only ancestry changed. For
#1356 that is no longer the whole story, and we would rather say so: after the rebuild we also removed
tools/engine_evidence.py and its test from it, because this revision left #1356 importing nothing from that
module — so shipping it there would have been 524 lines the PR does not use. Every one of #1356's remaining 12
files is byte-identical to the reviewed content; two files are simply gone.

If you would rather have the rebase, say so and we will do it. It is your history and your call; we did not want to force-push over your commit when there was a functional alternative.

check_ablate_evidence.py: we moved it to where it belonged rather than parking it

You treated the evidence half as a reader without a writer and asked us to park it. We are parking the part that
genuinely is one — the SCORE_EVIDENCE identity binding in eval_glm.py — and you are right about it: nothing on
dev reads that variable.

But check_ablate_evidence.py is not in that position. Its writer is #1356's own coli-ablate/2 engine mode.
Rather than assert that, we tested it at the head we are pushing. The checker refuses an artifact whose schema
line reads coli-ablate/1 — the version your dev mode emits — with
[ablate-evidence] INCOMPLETE: header keys or values are not exact, exit 1; and its own 91-check suite passes
against coli-ablate/2 shapes. So the checker is bound to the schema #1356 introduces. It has a writer; the writer
just lives in the next PR, not this one, so it has moved to #1356.

The full end-to-end round trip was regenerated against the current integrated tree rather than quoted from an
earlier run: an integration cell drove #1356's engine mode over a real container and fed the artifact to this
checker, which validated it against the manifest and the model's own config.json and passed. It is reported with
#1356's evidence.

If you still want it parked, say so and we will park it.

Your asks

you asked	disposition	where it lives now
"the body … no longer match[es] the diff"; pack_python.py "already fixed on dev, merged as #1364"	done — body corrected; the stale import-closure claim is gone. You are exactly right about what survives: the test additions plus the release.yml gate	this PR's body
"the evidence half is inert … nothing reads it, zero files" — split it out	done differently — the SCORE_EVIDENCE identity binding is parked; the durability half you said is "worth having on its own" stays and is now the whole of this PR. check_ablate_evidence.py moved to #1356 rather than parked (see above)	this PR / #1356
"rebase #1356 and #1357 onto this branch's head, so each diff shows only its own work"	done differently — rebuilt fresh from dev instead of rebased, to avoid force-pushing over 9e966378. Literal rebase offered above	#1356, #1357

What changed in each PR, at a glance

PR	files	lines added	moved in	moved out	parked	added on re-review
#1355	8 → 5	+3,315 → +2,153	—	the ablation-evidence checker, its test and the release.yml gate → #1356	the SCORE_EVIDENCE identity binding	a durability guard that fails when the per-row flush is removed; a frozen kernel-roster literal
#1356	15 → 12	+5,449 → +3,706	the ablation-evidence checker, its test and the release.yml gate, from #1355	engine_evidence.py and its test — dropped, no importer left here	—	the cap derived from CTX and checked ahead of all three allocations; removal of a one-implementation dispatch typedef; a real import check in the archive gate; a 2.01× logit_topk_select fix
#1357	11 → 4	+7,464 → +2,319	—	five files returned to the PR that owns each	check_native_mtp_witness.py + its test, 2,613 lines	seven per-ordinal frame-order tests and a frozen order anchor; the coupling test made bidirectional
set	34 → 21	+16,228 → +8,178

Every ask across the set

One row per ask, in your order. Several asks were raised on one PR and resolved on another — that crossing is
the thing this table exists to make visible. The disposition is the same word each PR's own comment uses.

your ask	asked on	resolved in	disposition	detail
"the body … no longer match[es] the diff"; pack_python.py "already fixed on dev, merged as #1364"	#1355	#1355 — this PR's body	done	this comment
"the evidence half is inert … nothing reads it, zero files" — split it out	#1355	#1355 (the SCORE_EVIDENCE binding removed) and #1356 (the checker, its test and the release.yml gate)	done differently	this comment · https://github.com/JustVugg/colibri/pull/1356#issuecomment-5725636433
"rebase #1356 and #1357 onto this branch's head, so each diff shows only its own work"	#1355	#1356 and #1357 — both rebuilt fresh from dev rather than rebased	done differently	this comment
"is ABLATE_SCORE used by anyone outside your own tree?"	#1356	#1356	done	https://github.com/JustVugg/colibri/pull/1356#issuecomment-5725636433
"Could the writer collapse to a plain FILE pointer with the tests driving a tmpfile?"	#1356	#1356 — c/colibri.c; the second structure you named still carries two production-path function pointers	done for AblateWriter, NOT for AblateOutputRun	https://github.com/JustVugg/colibri/pull/1356#issuecomment-5725636433
"I would make the prose match what the cap actually bounds"	#1356	#1356 — c/colibri.c, docs/ENVIRONMENT.md	done, and then some	https://github.com/JustVugg/colibri/pull/1356#issuecomment-5725636433
"send the witness alongside the engine change that emits the record it needs"	#1357	monotophic/colibri codex/parked-mtp-witness @ e0474748 — out of the PR entirely, preserved on a branch, returning with the engine change	parked	https://github.com/JustVugg/colibri/pull/1357#issuecomment-5725733595
"the logprob-gap hardening as its own small PR now"	#1357	#1357 — it is now the whole PR, 4 files	done	https://github.com/JustVugg/colibri/pull/1357#issuecomment-5725733595
"neither transcript tool is referenced by c/Makefile or by any workflow … What is meant to invoke them?"	#1357	#1357 — answered; wiring offered on your word	answered	https://github.com/JustVugg/colibri/pull/1357#issuecomment-5725733595

There is also a c/Makefile interaction between #1356 and #1102 that asks nothing of you; it is noted on both
threads.

What crossed a PR boundary

Without this map, this PR's deletions read as us dropping your work. They are not deletions; they are relocations.

file	was in	now in	why
c/tools/check_ablate_evidence.py	#1355	#1356	its writer is #1356's coli-ablate/2 mode
c/tests/test_check_ablate_evidence.py	#1355	#1356	follows its subject
.github/workflows/release.yml	#1355	#1356	the gate it adds is for the checker above
c/tests/check_native_mtp_witness.py (+ its test)	#1357	parked	2,613 lines waiting on an engine that emits MTPEMIT

Merge order, so you can sweep without reading us

Merging #1355 first gives the smallest review for #1357, which then merges dev and its copy of the shared
foundation drops out mechanically. But any order merges cleanly — we proved it rather than assuming it: all six
orderings of the three branches merge with zero conflicts and produce one identical tree.

One honest exception, and it is ordinary: #1356 and #1102 both add entries to c/Makefile and will conflict
there — only there; the engine sources merge cleanly. Whichever lands second, we re-merge dev and push within a
day.

The duplication that is still there, and why

c/tools/engine_evidence.py and its test — 524 lines, two files — remain in #1355 and #1357, byte-identical.
#1356 no longer carries them at all: this revision moved the manifest helpers into check_ablate_evidence.py,
which left that PR importing nothing from the module, so shipping it there would have been 524 lines of code
nothing in the PR used — the very pattern you objected to. The duplication that remains is one pair across two
PRs, and it drops out when #1355 lands. The 6,700 lines you counted three times are now 524.

Details — origin accounting and evidence

maintainer-requested: body correction; SCORE_EVIDENCE binding parked; the de-duplication.
dev-motion: pack_python.py empty diff (your #1364 fixed it on dev).
re-review-found: the durability guard below.

One thing our own re-review found, disclosed because it is a real gap we shipped. This PR's headline feature is
incremental, per-row durability, and the test named for it could not detect losing it: deleting the per-row
flush() left the suite green. That is a test that manufactures confidence. There is now a guard that fails when
the flush is removed and passes when it is present — verified by running the mutation, not by inspection.

Also disclosed: the transcribed wire-format literals in these tools are hand transcriptions of the C source.
Module drift is pinned by tests; C-source drift is not. We have a follow-up in mind for that and are not
folding it in here.

Authored by Opus 5 in Claude Code, analysis in partnership with @Monotophic