fix(pixel): judge an exit-0 unittest run by the runner's own summary; a coding stop takes precedence over test-command coaching

[Lightheartdevs]

Problem

Fleet, open-prompt 07 photo-renamer ("write a python script that renames all photos in a folder by date taken, and test it") on main d4a61f33:

host (model)	grade	cause
tower3 (Qwen3.5-27B)	PARTIAL 87 s	python3 -m unittest test_photo_renamer -v exited 0 with Ran 11 tests in 0.010s / OK. A passing negative-path test had printed Error: Folder '/nonexistent/folder/path' does not exist. to stdout. execResultHasNonCleanUnittestOutcome matched that line with `/(?:^
strixy (Qwen3.6-35B-A3B)	PARTIAL 941 s	After the coding stop (6 of 6 failed runs), the non-auditable check ran before the coding terminal. The model's ...; echo "EXIT:$?" got "Run it directly", which contradicts "Do not call another tool". The model obeyed the newer message (call 41).

Control on the same tower2 hardware and model: Hermes DELIVERED in 23 s and OpenCode DELIVERED in 19 s (claude-app-compare/runs/val2/07-photo-renamer).

The same heuristic also false-fails:

• Python's default logging (ERROR:root:...);
• stderr messages such as strixy's renamer ERROR: cannot read folder ...;
• -v docstrings that start with "Error when".

(AssertionError text printed by a passing test still fails; see Accepted tradeoffs.)

Change

plugin/tool-loop-guard.mjs:

1. Runner summary verdict, for unittest's own runner only. The summary is trusted only when the command the model sent runs python -m unittest ... directly, alone or after cd /workspace/... (execFingerprintRunsUnittestRunner). If a result text of such a command contains unittest's summary line (Ran N tests in X.XXXs, CRLF-tolerant, ANSI colors stripped), the result is judged by these rules instead of the old text heuristics. It is non-clean if any of these holds:

   • a summary with N=0, or NO TESTS RAN;
   • a summary not followed by OK / OK (...) after at most one blank line;
   • a FAILED ( line, or unittest's failure counts in any case (failed (failures=1));
   • expected failures= or unexpected successes= greater than 0, or their -v forms;
   • a FAIL|ERROR: <test> (<dotted.id>) header directly after unittest's "=" * 70 separator;
   • a line that starts (optionally indented) with an upper-case FAIL word, or with FAIL in any case before :, (, a space, a tab or the line end (fail: negative numbers, Fail: ...);
   • AssertionError anywhere, in any case (Test failed: AssertionError('...'), ✗ add(-1, -2): AssertionError: ...).

   The last two rules are the df3f4bf protection for a check that runs beside unittest in the same exec: a module-level print checker imported by discovery, or a test that catches its own assertion and prints it. Program output such as Error: ..., ERROR: IMG_0002.jpg (corrupt), ERROR:root: logging or a -v docstring is no longer failure evidence for such a command. Without a summary in the result, the old text heuristics apply. execFailed still decides first, and pytest is still judged by exit status only.

2. Anything else keeps the old text heuristics, byte for byte. A summary can be printed by something other than the runner, so these commands never use the summary rules:

   • any and-chain setup that andChainVerificationParams accepts before the test (echo Ran 5 tests ... && ..., cp summary.txt /dev/stdout && ..., mkdir, mv, touch, true);
   • a direct test script (python3 test_calc.py), whose own code runs around unittest.main(): checks after unittest.main(exit=False), or a custom runner that prints a summary of its own;
   • a second process started beside the runner with & (python3 -m unittest test_calc & python3 check_calc.py), which verificationCommandIsAuditable does not refuse. The trailing stderr merge 2>&1 alone is not a second process.

   This holds on the exec path and on the process-poll path.

3. Compaction keeps program output visible. Tool Search compaction (cleanUnittestSummary) replaces a clean result with its summary alone. It still requires the old text heuristics to find nothing, as on d4a61f33. So a result that passes despite program output keeps that output in the transcript.

4. Coding stop first. After codingExhausted, the non-auditable refusal defers to the coding terminal handler: one terminal block, then abortRun. Before a coding stop, the refusal text and its free-correction accounting are unchanged.

Deliberate deviations from the spec (please review)

• Stale-pass invalidation is kept. The spec's literal && !state?.codingExhausted on the branch condition would also skip the branch's latestVerificationStatus = "failed" line. That line invalidates a pass that predates a later write. A piped test command has no verification fingerprint, so afterToolCall does not re-mark it, and the stale passed would survive. I checked this: with the literal condition, the new test "after a coding stop ... still invalidates a stale pass" fails with actual: 'passed'. So only the coaching return is gated on codingExhausted.
• ANSI colors are stripped before the summary check. Python 3.14 colors OK, FAILED and the failure headers when output goes to a terminal (a pty exec). I checked this locally with Python 3.14.4 and FORCE_COLOR=1: \x1b[32mOK\x1b[0m (\x1b[33mskipped=1\x1b[0m). Without stripping, a clean colored run would newly fail. Main also let a colored exit-0 FAILED run pass: \bAssertionError\b has no word boundary after m. Both cases are covered by tests.
• A third fixture (tower2) was added for the spec's replay 4, the custom runner that still passes.

Accepted tradeoffs

• Program output next to a clean summary. For python -m unittest, a check that reports a failure only as Error: or ERROR: lines (for example a module-level checker imported by discovery) passes next to a clean summary. This is the tower3 shape, which cannot be told apart from a passing negative-path test's output. A check that prints FAIL at the start of a line, or AssertionError anywhere, still fails.
• False failures kept on purpose. A passing test that prints AssertionError text, or a program line that starts with FAIL (for example FAIL: IMG_0001.jpg (no EXIF date)), fails the run. Both also failed on d4a61f33: a false failure, never a false pass.
• Not flagged, as on d4a61f33. A checker line that starts with FAILED: or Failed: beside a clean python -m unittest summary passes; it also passes on d4a61f33.
• Stricter than d4a61f33. These shapes passed on main and now fail: a summary not followed by OK within one blank line, including output cut after the Ran line, two blank lines, or a program line before OK. None of these occurs in the replayed fleet sessions.

Review fixes (11c8f711)

finding	fix
A clean summary dropped the df3f4bf protection. Real Python 3.12 discovery of a module-level checker, and a nested custom runner, went from failed on main to passed.	A line-start FAIL or AssertionError fails the result even with a summary. The real Python repro, reviewer probes A, B and D, the Python 3.14 unittest.main(exit=False) capture, and pass-edit-rerun all return to failed.
Tool Search compaction hid that evidence from the model.	Compaction again requires the old text heuristics to be clean. In reviewer probe P the Error: line now stays in the persisted result.
An and-chain echo could forge a summary ahead of a custom runner. A mimic runner's failed (failures=1) was also missed.	Echo-setup chains use the text heuristics (exec and process-poll). Lower-case failure counts fail. Probes C and H return to failed.
Add/add conflict with #6753 on photo-renamer-tower2-d4a61f33.json. The tower2 test also asserted the rm -rf delivery text that #6753 changes.	Fixtures are renamed unittest-verdict-*-d4a61f33.json. The tower2 replay stops at call 8 and asserts only the call-6 and call-8 verdicts.
UNITTEST_FAILURE_HEADER matched program output such as ERROR: IMG_0001.jpg (no EXIF date).	A header counts only directly after the "=" * 70 separator and only as <test> (<dotted.id>). ERROR: IMG_0002.jpg (corrupt) and ERROR: /tmp/x.jpg ([Errno 2] ...) now pass.
Program output after the summary was ignored (info).	Covered by the first fix. The remaining shape is listed under Accepted tradeoffs.
Stricter shapes (info).	Documented under Accepted tradeoffs, with no change.

Found while checking the fixes. #6751 changes the Playground refusal text that tower2 call 0 recorded, so the old exact-refusal check failed with #6751 merged. The replays now pin the refusal text only for the calls this change concerns (strixy 38-41). For every other call they check that the recorded path is followed: refused calls are refused and calls that ran still run.

Re-review fixes (c0a83a78e, 1363e768c)

The re-review found that a failed or non-unittest output could still be read as a passing unittest trailer on 11c8f711.

finding	fix
A checker's fail: ..., Fail: ..., fail(...) or a bare fail line, Test failed: AssertionError('...'), ✗ add(-1, -2): AssertionError: ... or a lower-case assertionerror beside a clean summary passed.	c0a83a78e: FAIL in any case before :, (, a space, a tab or the line end, and AssertionError anywhere in any case, fail the result in summary mode.
A summary written by other and-chain setup (cp expected_summary.txt /dev/stdout && python3 -m unittest -v, also mv, mkdir, touch, true) was trusted.	c0a83a78e: only python -m unittest itself, alone or after cd /workspace/..., is trusted; every and-chain setup keeps the text heuristics (exec and process-poll).
A direct test script trusted a summary its own code printed, and Error:/ERROR: checks after unittest.main(exit=False) passed.	c0a83a78e: python3 test_x.py keeps the text heuristics, with or without a summary.
python3 -m unittest test_calc & python3 check_calc.py is a plain, auditable command, so its summary was trusted: the checker's Error: add(-1, -2) returned 1, expected -3 next to the runner's clean summary passed on 11c8f711 and c0a83a78e, and fails on d4a61f33 and main 0a632581.	1363e768c: a command with & (other than the trailing 2>&1) keeps the text heuristics, on the exec and process-poll paths. The pre-execution gate is unchanged.

Replays (recorded sessions, call by call through createToolLoopGuard)

The fixtures were trimmed from the session JSONLs, with provenance in each file:

• tower3 047d3145, sha256 e861b9f0…, all 8 calls;
• strixy 519cb0fa, sha256 e5134e08…, the two writes, the six exit-1 runs each after its preceding edit, edit 37 and refused calls 38-41;
• tower2 fe9ee859, sha256 8d21ebc1…, calls 0-8 (the fixture keeps all 11).

The replay runs in a real temp workspace, so Playground routing behaves as it does on the host.

	d4a61f3 (control)	this branch
tower3 status after calls 2/4/5/6/7	failed / failed / failed / failed / failed	failed / passed / passed / passed / passed
tower3 verificationForRun	failed, VERIFICATION_FAILED_DELIVERY_PREFIX (= recorded answer.txt)	{status:'passed'}
tower3 replyPayloadSending(973-char answer)	VERIFICATION_FAILED_DELIVERY_PREFIX	undefined: the answer is delivered unchanged (beforeAgentFinalize also undefined)
strixy 38 / 39	CODING_RETRY_EXHAUSTED_REASON ×2 (= recorded)	same
strixy 40 ...; echo "EXIT:$?"	VERIFICATION_COMMAND_NOT_AUDITABLE_REASON (= recorded), no abort	CODING_LOOP_ABORT_REASON, abortRun called
strixy verdict	failed, no run counted as passed	failed, no run counted as passed
tower2 call 8 python3 test_rename.py (custom runner, no summary)	passed	passed

More replay probes:

• Failed-attempt count. The limit is set to 2 for this probe. After tower3 run 1, one more failing run is allowed (count 1). After run 2, two more are allowed (count reset to 0).
• Successful command. Run 2 is counted as a successful command, not a failed one: one identical rerun is allowed, and the next gets CODING_REPEAT_NO_PROGRESS_REASON.
• Process-poll path. The same tower3 output delivered through the process-poll completion also passes.

Fleet-wide replay.

• 11c8f711 (earlier): the reviewer harness replay2.mjs over 63 recorded sessions (the six d4a61f3 open-prompt photo-renamer sessions and 57 qualification sessions from rounds 100-107; 3,350 calls, 318 runs) gave 0 differences against 462c7849, and against d4a61f33 the same 4 sessions as before: tower3 flips to delivered; strixy and two laptop coding journeys reach the coding stop one call earlier with no change in delivered status.
• 1363e768c (this head): reviewer 2's replay-multi harness (each run ends with the real agent_end handler) over 117 recorded sessions: the 60 d4a61f3 open-prompt sessions (6 hosts) and 57 qualification sessions from rounds 100-107; 372 runs, 3,728 calls. Against 11c8f711: 0 differences in delivered status and text, refusals, signals and aborts. Of the 275 recorded unittest commands, 131 are a plain auditable python -m unittest command and none of them contains &, so 1363e768c changes no recorded session.

Tests

New file tests/unittest_trailer_verdict.test.mjs, 89 tests.

The replays are listed above. The unit cases below all exit 0 and run both native and Tool Search wrapped.

Stay passed (python -m unittest):

• ERROR:root: logging;
• strixy's stderr ERROR: cannot read folder;
• a -v docstring that starts with "Error when";
• OK (skipped=2);
• CRLF output;
• program Error: / ERROR: lines after the summary;
• ERROR: IMG_0001.jpg (no EXIF date), ERROR: IMG_0002.jpg (corrupt) and ERROR: /tmp/… ([Errno 2] …);
• Failed (no EXIF date): …;
• Python 3.14 colored output.

Stay failed:

• a FAIL: header with OK;
• a module import ERROR: header after the separator;
• unittest.main(exit=False) with FAILED (failures=1);
• FAILED (errors=1);
• Ran 0 tests ... OK (df3f4bf);
• NO TESTS RAN;
• OK (expected failures=1);
• ... unexpected success;
• a summary cut before its outcome;
• a clean summary followed by a failed one;
• colored FAILED;
• a colored unexpected success;
• the custom runner FAIL: negative numbers\nAssertionError (df3f4bf).

New for the first review:

• a line-start FAIL after the summary;
• a discovered module-level checker (real Python 3.12 output);
• ad-hoc checks after unittest.main(exit=False) (Python 3.14 shape);
• a caught assertion's printed traceback;
• an indented FAIL;
• failed (failures=1) after OK;
• a nested custom runner (python3 test_all.py);
• an echo-forged summary on the exec and process-poll paths, next to an echo before a real unittest run, which is judged as on d4a61f33;
• compaction: a result with program output keeps it, and a clean result is still compacted.

New for the re-review:

• AssertionError text printed by a passing test, mid-line or in lower case, and in a repr or after a mark;
• fail:, Fail:, fail(...) and a bare fail checker line;
• a discovered checker that prints its caught AssertionError (real Python 3.12 output);
• direct test scripts keep the text heuristics with or without a summary, including Error:/ERROR: checks after unittest.main(exit=False) and a script that prints its own summary; the same output from python3 -m unittest passes;
• a summary written by cp/mv to /dev/stdout or /dev/stderr, and by mkdir/touch/true setup chains (exec and process-poll);
• a second process started with &, alone or after cd /workspace/... (exec and process-poll); the trailing 2>&1 alone still passes.

Other cases:

• separate stdout and stderr streams;
• pytest is still judged by exit status only;
• a nonzero exit still decides first;
• the rider, before and after a coding stop.

Unchanged:

• the whole of verification_redirect.test.mjs;
• the existing expected-failure and FAILED (failures=1) tests in tool_loop_guard.test.mjs;
• the df3f4bf fixtures;
• DEFAULT_WEB_TOOL_LIMITS (3 / 6);
• RUN_PROGRESS_LIMITS;
• VERIFICATION_COMMAND_NOT_AUDITABLE_REASON;
• verificationCommand, verificationCommandIsAuditable and andChainVerificationParams;
• canonicalRequestedUnittestParams;
• compactFailedUnittestText.

Red-checks:

• This head's test file on the 11c8f711 guard: 21 of 89 fail, all of them re-review cases.
• The & test on the c0a83a78e guard: fails (88 of 89 pass).
• Earlier: the first 71 tests on the 462c7849 guard, 19 fail; on the d4a61f33 guard, 32 fail.

Full Linux suite (remote-js-test.sh, tower2):

ref	tests	pass	fail
d4a61f33 (main)	3138	3100	0
462c7849 (first commit)	3191	3153	0
11c8f711	3209	3171	0
c0a83a78e	3226	3188	0
1363e768c (this head)	3227	3189	0
1363e768c + main 0a632581 (local merge)	3294	3256	0
1363e768c + main 8c589ef5 (local merge)	3300	3262	0
#6753 head c091bd635 (contains this head) + main 8c589ef5	3328	3290	0

Composition with open PRs (at 11c8f711). I merged each of 19 open Pixel PRs into 11c8f711 locally and ran the new test file each time: #6698, #6702, #6715, #6716, #6722, #6723, #6724, #6726, #6730, #6732, #6741, #6743 and #6746-#6753.

• 17 merge cleanly and pass 71/71.
• fix(pixel): stop whole-file rewrites and test bisection after failed tool results (output efficiency C1-C4) #6716 and fix(pixel): check that owner-requested text is visible when the preview loads #6723 conflict with d4a61f33 itself, independently of this PR.

Credential scan of fixtures. No keys, tokens, passwords, emails or private IPs. The host cwd is stripped. The only long hex strings are the session sha256 values.

Not done here

• Fleet rerun of open-prompt 07 on tower3. It was not run because fleet hosts were read-only for this session. Expected result: DELIVERED with the model answer intact.
• Other hosts in the sweep are not affected by this change:
  • mac-mini: NOTHING, from Playground-routing refusals and then a progress stop;
  • laptop: Pixel delivered the model's answer after 16 tool calls. The NOTHING grade came from the harness trace transport (session-trace.json ok:false, Wsl/Service/0x8007274c), not from Pixel. An earlier version of this description said "code in chat only", which was wrong;
  • tower2's final verdict is the recursive-delete refusal, which fix(pixel): a refused rm -rf keeps a current passing test, and the owner gets a receipt instead of guard text #6753 addresses.
• Pre-existing, not changed here. A passing single-test subset run (python3 -m unittest mod.T.test_one) resets the failed-attempt count, so failing full-suite runs can continue past the limit. This is the same on d4a61f33 and deserves its own change.

🤖 Generated with Claude Code