Skip to content

test: the force-exit test reads its runner's report from a file, so that runner's exit can't drop what it has written - #487

Merged
nissa-seru merged 1 commit into
anima-research:mainfrom
nissa-seru:langston/f168-runner-report-file
Oct 18, 2026
Merged

nissa-seru merged 1 commit into
anima-research:mainfrom
nissa-seru:langston/f168-runner-report-file

Conversation

@nissa-seru

Copy link
Copy Markdown
Contributor

test/force-exit-blocking-stdout.test.ts's runner test now has the runner it spawns write its report to a file, and counts the fixture's bytes there. It used to read that report through a pipe, and that runner's own --test-force-exit can drop what is still queued for the pipe. The helper the test checks doesn't cover the runner's stdout. The test failed three times on macOS Node 20, in a way that fits that cut.

Problem

The test, from #464, spawns node --import <helper> --test --test-force-exit --test-reporter=tap <fixture>. The fixture writes 10,000,000 ~ and exits at once. The test counts the ~ in the spawned runner's report, which it read through a pipe.

It failed on macOS Node 20 in three pull requests' CI that don't touch it:

How the spawned runner's report can lose its tail:

  • Node 20.20.2's runner pipes its reporter to stdout: compose(rootReporter, reporter).pipe(destination) (utils.js, lines 184 to 191).
  • Under --test-force-exit, its root calls process.exit() when the root reporter stream closes (test.js, lines 836 to 845). Nothing waits for the reporter's output to reach stdout, or for stdout's pipe to drain.
  • The fixture's bytes are all in by then. The runner waits for the file's process to exit and for its stdout to end before it reports the file (runner.js, lines 413 to 416).
  • The helper acts only in a test file's process, where NODE_TEST_CONTEXT is child-v8. So it leaves that runner's stdout as Node sets it.

On Linux, with the helper loaded, the report's tail went missing only when the runner's stdout was a slow or small pipe, never when it was a file (Checks). The three macOS shortfalls are a tail's size, and in each macOS job's latest attempt in the 15 CI runs between #464's merge and 03:53Z, the helper delivered all 10,000,000 bytes. macOS wasn't run here.

Changes

  • runnerReceived opens a temporary file and gives it to the spawned runner as stdout. Once the runner exits, it counts the ~ in the file and removes it.
  • A runner that fails to start, or exits with anything but the 0 it gives for this fixture, now makes runnerReceived throw with the end of the runner's stderr and report, where it would have read as a shortfall. A test file that fails to load reports its error in the report, not on the runner's stderr.
  • The file's header says why the report goes to a file, and that the first test is the one that runs the fixture under the runner.

What each part is for

  • The file: Node writes to a file synchronously (Node's "A note on process I/O"). So whatever the runner has handed to its stdout is in the file when it exits, where a pipe's queue of it can be dropped.
  • The throw keeps a runner that never ran the fixture, such as one that can't load the helper, from being read as the helper losing bytes.
  • What stays the same: the fixture's pipe to the runner is the one the helper protects, and it's as before. So the test still measures the helper: without it the fixture's bytes are lost at its exit and the test doesn't skip, and with it they all arrive.

What a caller notices

Nothing outside this repository. The test's count no longer depends on the spawned runner's stdout draining before that runner's forced exit. A short count still fails it.

Tests

Changed: a test file that exits at once after writing more than its pipe holds … delivers all of it to the runner now reads the runner's report from a file.

Checks

At ea0297f28, on Linux with Node 24.22.0 and 20.20.2 unless another is named, with main (77066360) as the base.

  • CI's install, type-check, build and test steps (npm install, npx tsc --noEmit, npm run build, npm test): tsc and the build are clean, and npm test passed all 2975 tests in 194 suites under each Node.

  • The changed file alone, five times under each of Node 24.22.0, 20.20.2 and 24.21.0, CI's Node 24: all three of its tests passed every time, none skipped. So in each run, the runner test's run without the helper lost bytes, and its run with the helper delivered all 10,000,000.

  • Removal: with the helper's setBlocking call taken out of the built helper, the runner test fails under both Nodes.

  • A runner that can't load the helper: with the test's helper path pointed at a missing file, the runner test fails with "the runner exited with 1", and the error carries the report's ERR_MODULE_NOT_FOUND.

  • The two ways of reading the report, a probe on Linux with Node 20.20.2 and 24.21.0. The helper was loaded throughout, and only the spawned runner's stdout changed:

    The runner's stdout Node 20.20.2 Node 24.21.0
    a file 60 of 60 complete 60 of 60 complete
    a socketpair, as libuv makes for a child, with its buffers set to 8192 bytes, read as fast as it comes 10 of 10 short, by 359 to 433 KB 8 of 10 short, by 7 to 44 KB
    the same socketpair at Linux's default buffers 10 of 10 complete 10 of 10 complete
    a pipe read by Python in 8 KiB pieces, at full speed 9 of 10 short 10 of 10 complete
    the same pipe with a 0.5 ms pause between reads 10 of 10 short 5 of 10 short

    The Python pipe's short runs lacked 287 to 456 KB on Node 20.20.2 and 2 to 54 KB on Node 24.21.0, the report's summary missing in each. Of each Node's 60 runs to a file, 55 used the fixture as shipped and 5 a copy that writes the same and ends normally, and the Python-pipe rows used each in half their runs. Without the helper, the file still shows the fixture's loss: with the fixture as shipped, Node 20.20.2 got 219,264 to 255,808 of 10,000,000 bytes, and Node 24.21.0 got 182,720.

Not verified

  • macOS. CI runs this test there, and both failures were there. Everything above ran on Linux. The 8192-byte socketpair cuts far more often than macOS CI does, so it shows what the cut depends on, not how often macOS sees it.

Not covered

  • On Node 20, and less often on Node 24, a CI log can lose the tail of npm test's own report the same way, through the runner's forced exit. By our reading, the run's exit code stays right, since the runner sets it before the exit.

Written by Langston-2479 (Claude Opus 5.5). Kestrel-2409, Ada-1017 and Fiona-2757 found the three failures, and Ada-1017 decided the fix. After a round of our internal review, Greta-2425 (Claude Opus 5.5) reviewed it at ea0297f28.

…hat runner's exit can't drop what it has written

force-exit-blocking-stdout's runner test counts the fixture's bytes in
the report of the runner it spawns. It read that report through a pipe,
and the runner's own --test-force-exit can drop what is still queued for
that pipe. On Node 20.20.2 the root calls process.exit() when its root
reporter stream closes (lib/internal/test_runner/test.js 836-845),
without waiting for the reporter's output to reach stdout or for
stdout's pipe to drain. The helper acts only in a test file's process,
so it leaves that runner's stdout as it is. The runner has read the
fixture's pipe to its end by then: it waits for the child's exit and for
its stdout to end before it reports the file (runner.js 413-416).

The test failed on macOS Node 20 in anima-research#250's CI at ed40f2d (9,953,278 of
10,000,000 bytes; its rerun passed) and in anima-research#480's at 256baf6
(9,945,086). In the 15 CI runs between anima-research#464's merge and 03:53Z, every
job's latest attempt passed it.

The runner now writes its report to a temporary file. Node writes a file
synchronously, so whatever the runner has handed to its stdout is in the
file when it exits; the fixture's pipe to the runner, which the helper
protects, is as before. On Linux, with the helper, a file held the whole
report in 60 of 60 runs under each of Node 20.20.2 and 24.21.0, where a
socketpair with 8192-byte buffers came up short in 10 of 10 and 8 of 10.

runnerReceived now throws, with the end of the runner's stderr and
report, when the runner fails to start or exits otherwise than 0, as it
does for this fixture, so a failed runner no longer reads as a
shortfall. A test file that fails to load reports its error in the
runner's report, not on the runner's stderr.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@greptile-apps

greptile-apps Bot commented Oct 18, 2026

Copy link
Copy Markdown

Required label not found on this PR.

nissa-seru pushed a commit to nissa-seru/agent-framework that referenced this pull request Oct 18, 2026
@nissa-seru
nissa-seru merged commit c31eca6 into anima-research:main Oct 18, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant