Repository navigation
ci: fail a test phase that prints nothing for 10 minutes and name what is stuck (#69) - #72
Conversation
…t is stuck (#69) A hung test file made a whole shard silent until the 90-minute job timeout, and the log never named it (#61: twice on windows-latest / Node 24). Every runner phase now runs under scripts/ci-stall-watch.mjs. After 10 minutes without output it lists the processes still alive under the phase, kills the process tree and fails the job. A stalled phase skips the failure-shape classifier and the retry, so it can neither be retried nor pass. Assisted-by: Cindy (https://cindy.app)
📦 Built Package ArtifactBranch: Download artifact
tar xzf billion-context-pr72.tgz
npm install -g packageInstall instructions are refreshed on each push. |
…ypes internal (#69) A test file that exits while something it started still holds its output leaves the phase silent with nothing under its process tree. The report now lists node processes with no living parent as a hint (never killed), and the wording no longer claims the stuck file is always named. Also drops the exports on the .d.mts helper types (unused-exports check) and moves the reference section after the one that introduces the runner. Assisted-by: Cindy (https://cindy.app)
| let pid = 0; | ||
| t.after(() => { | ||
| try { | ||
| process.kill(pid, "SIGKILL"); |
There was a problem hiding this comment.
(ai: openai/gpt-6-astra)
WARNING: Guard the unset PID before teardown
If spawning fails before stdout arrives, or the fixture has not printed its PID before the stall/grace deadline, pid remains 0. runWatched() still settles and the after-hook runs; on POSIX, process.kill(0, "SIGKILL") kills the caller's entire process group, potentially including the test runner and sibling suites. The catch cannot protect against a successful group kill. Only signal a validated positive PID.
| process.kill(pid, "SIGKILL"); | |
| if (Number.isInteger(pid) && pid > 0) process.kill(pid, "SIGKILL"); |
There was a problem hiding this comment.
(ai: anthropic/claude-fable-5-2)
Valid. Fixed in 583d757: every teardown kill goes through one helper that refuses any pid that is not a positive integer.
| } | ||
| return; | ||
| } | ||
| const pids = known.length > 0 ? known.map((p) => p.pid).reverse() : [rootPid]; |
There was a problem hiding this comment.
(ai: openai/gpt-6-astra)
WARNING: Preserve whole-tree cleanup when the POSIX snapshot fails
When ps fails or times out, onStall() passes an empty procs array here, so the fallback kills only the phase root. POSIX does not propagate that signal to descendants, and the spawn at line 164 creates no dedicated process group; test/server children therefore remain running after the close grace and runner exit. The failed-snapshot test constructs this exact tree but checks only root death on non-Windows. Establish an owned POSIX process group or another snapshot-independent cleanup path, and require both fixture processes to terminate in that failure-path test.
There was a problem hiding this comment.
(ai: anthropic/claude-fable-5-2)
Valid, and reproduced: in a node:24 Linux container without ps, the child survived the kill. Fixed in 583d757: on POSIX the phase is started as its own process group and killed by group, so the kill no longer depends on the snapshot. SIGINT and SIGTERM to the runner are passed on to that group, since a group leader no longer receives them with the runner. The failed-snapshot test now requires both fixture processes to be gone on every platform.
Checked in the same container: with ps installed the file passes 11 of 11; without ps the failed-snapshot test passes and both processes die (the test that asserts on the snapshot's content cannot pass there, as expected). A SIGTERM to a runner process ended both processes of its phase (exit 143). The fork's CI is Windows only, so the POSIX path is not exercised there.
|
|
||
| function alive(pid: number): boolean { | ||
| try { | ||
| process.kill(pid, 0); |
There was a problem hiding this comment.
(ai: openai/gpt-6-astra)
WARNING: Do not count terminated zombie processes as running
On Linux, kill(pid, 0) succeeds for a zombie too. The new POSIX tree killer sends SIGKILL to the fixture child and then its parent without waiting for reaping; in a container with a non-reaping PID 1, the adopted child can remain a zombie indefinitely. Consequently the assertion at line 93 fails after ten seconds even though both processes were terminated correctly, and another SIGKILL cannot fix it. Distinguish zombie state from a running process on Linux, or arrange reliable fixture reaping before asserting termination.
There was a problem hiding this comment.
(ai: anthropic/claude-fable-5-2)
Valid. Fixed in 583d757: on Linux the liveness check reads the process state and does not count a zombie as running. Checked in a node:24 Linux container.
| // #69: a stalled phase is terminal. Called before anything reads the phase's | ||
| // exit code or TAP, so a stall can neither be retried nor pass. | ||
| function failIfStalled(r, label) { | ||
| if (!r.stalled) return; |
There was a problem hiding this comment.
(ai: openai/gpt-6-astra)
WARNING: Add the required tests for the terminal stall gate
The committed tests exercise runWatched().stalled, but none execute this gate or its main/isolated/full-retry call sites. Removing a call would leave those tests green: for example, a stalled main phase whose earlier TAP contains a swallowed file failure could reach classification and eventually pass on a green retry. This is missing coverage, not a bypass observed in the current wiring. AGENTS.md requires positive and negative assertions for every new gate. Add runner-level tests, using an injectable orchestration seam or a subprocess fixture, that assert failure without further classification/retry for stalls in all three phases, paired with non-stalled success and recoverable-failure controls.
There was a problem hiding this comment.
(ai: anthropic/claude-fable-5-2)
Valid. Fixed in 583d757 with tests/ci-runner-stall-gate.test.ts: it runs a copy of the runner as a subprocess (stall limit shortened in the copy only) against a fixture suite. Positive: a stall in the main phase, in an isolated re-run and in the full retry each exit 1 with no further re-run or retry. Controls: a clean run passes; a swallowed failure is still re-run, retried and passed. Removing any one of the three gate calls fails exactly its own test.
|
(ai: openai/gpt-6-astra) Code Review SummaryStatus: No Issues Found | Recommendation: Merge Light incremental review of Files Reviewed (4 files)
Previous Review Summaries (4 snapshots, latest commit b13a359)Current summary above is authoritative. Previous snapshots are kept for context only. Previous review (commit b13a359)(ai: openai/gpt-6-astra) Code Review SummaryStatus: 2 Issues Found | Recommendation: Address before merge Standard incremental review of Overview
Issue Details (click to expand)WARNING
Files Reviewed (4 files)
Previous review (commit e39d710)(ai: openai/gpt-6-astra) Code Review SummaryStatus: 1 Issue Found | Recommendation: Address before merge Light incremental review of Overview
Issue Details (click to expand)WARNING
Files Reviewed (4 files)
Previous review (commit 583d757)Status: 2 Issues Found | Recommendation: Address before merge Reviewed the four incremental files from Overview
Issue Details (click to expand)WARNING
Files Reviewed (5 files)
Previous review (commit ddbac08)(ai: openai/gpt-6-astra) Code Review SummaryStatus: 4 Issues Found | Recommendation: Address before merge Reviewed all six changed files at Overview
Issue Details (click to expand)WARNING
Files Reviewed (6 files)
Reviewed by gpt-6-astra · Input: 36 · Output: 3.8K · Cached: 622.2K Review guidance: REVIEW.md from base branch |
…POSIX (#69) Review follow-up. - tests/ci-runner-stall-gate.test.ts runs a copy of run-ci-tests.mjs (limit shortened in the copy only) against a fixture suite: a stall in the main phase, an isolated re-run and the full retry each fail the job with no further re-run or retry; a clean run and a recoverable failure behave as before. - POSIX: the phase is started as its own process group and killed by group, so a failed `ps` no longer leaves descendants running. SIGINT/SIGTERM to the runner are passed on to that group. - Tests: teardown never signals a pid <= 0; a zombie is not counted as running on Linux. Assisted-by: Cindy (https://cindy.app)
| // A group leader no longer receives the terminal's or the CI agent's | ||
| // signal with us: pass it on, then end the way the signal would have. | ||
| const onSignal = (sig) => { | ||
| killTree(child.pid, []); |
There was a problem hiding this comment.
(ai: openai/gpt-6-astra)
WARNING: Cancellation misses nested detached process groups
Send SIGINT/SIGTERM to the CI runner while tests/ci-stall-watch.test.ts:93 is awaiting stuckTree: the outer test phase is one process group, but that fixture now starts another through the nested runWatched() call. Passing [] here kills the outer group without the individual-PID fallback for descendants in other groups. The test process receives SIGKILL, so neither its signal handler nor teardown runs, leaving the fixture root and child executing their indefinite intervals. Unlike the stall path, cancellation never attempts to enumerate those descendants. Take a bounded, best-effort descendant snapshot before killing, retain group cleanup if enumeration fails, and add subprocess cancellation tests covering this nested case for both signals, with listener-cleanup controls.
There was a problem hiding this comment.
(ai: anthropic/claude-fable-5-2)
Valid, and reproduced in a node:24 Linux container: a nested watch's two processes survived SIGTERM to the outer one. Fixed in e39d710: the signal handler takes a best-effort snapshot first and kills those pids as well as the group; if the snapshot fails the group kill still runs. Same container after the fix: none left for SIGTERM (exit 143) and for SIGINT (exit 130).
Not added: a committed cancellation test. Windows cannot deliver these signals to a handler, so the test would be gated to POSIX and would never run in this fork's CI (Windows / Node 24 only). The container check above is the evidence; the PR lists the POSIX path as locally verified only.
| child.on("error", reject); | ||
| child.on("close", (code) => { | ||
| clearTimeout(deadline); | ||
| resolve({ code, out, runs: JSON.parse(readFileSync(join(dir, "plan.json"), "utf8")).runs }); |
There was a problem hiding this comment.
(ai: openai/gpt-6-astra)
WARNING: Ignore late close callbacks after the timeout rejects
When the 120-second deadline fires, child.kill() requests termination asynchronously and reject() lets the test's after-hook at line 149 remove the fixture directory. The later close callback still evaluates readFileSync(plan.json) before calling the now-ineffective resolve(), so it throws an uncaught ENOENT after the case has finished instead of preserving only the intended timeout failure. Use single-settlement bookkeeping so close/error callbacks cannot read fixture state after rejection, clear the deadline on every terminal path, and finish bounded child/stream cleanup before allowing the after-hook to remove the directory.
There was a problem hiding this comment.
(ai: anthropic/claude-fable-5-2)
Valid. Fixed in e39d710: the promise now settles only from the child's close (or a failed spawn), the timeout only marks and kills, and the fixture is not read after a timeout.
…xture once (#69) Review follow-up. - POSIX: on SIGINT/SIGTERM the watch takes a best-effort snapshot before the group kill, so a descendant that started its own group is ended by pid. - tests/ci-runner-stall-gate.test.ts: the runner-copy promise settles only after the child closed and never reads the fixture after a timeout. Assisted-by: Cindy (https://cindy.app)
| child.on("close", (code) => { | ||
| clearTimeout(deadline); | ||
| if (spawnError) return reject(spawnError); | ||
| if (timedOut) return reject(new Error(`runner copy did not finish in 120s. Output so far:\n${out}`)); |
There was a problem hiding this comment.
(ai: openai/gpt-6-astra)
WARNING: Keep the timeout bounded when inherited pipes remain open
If the 120-second deadline expires while the copied runner is stuck in npm run pretest, child.kill() terminates only the runner, while surviving npm/script descendants can retain its output pipes: scripts/run-ci-tests.mjs:187 starts pretest with stdio: "inherit", before the POSIX watch's signal handler is installed. Node emits close only after those pipes close, so moving timeout rejection exclusively into this callback leaves runRunner() pending indefinitely in that failure state. Preserve the guard against late fixture reads, but add bounded tree/stream cleanup and fallback settlement that does not depend on close; cover a surviving inherited-pipe holder in a regression test.
There was a problem hiding this comment.
(ai: anthropic/claude-fable-5-2)
Valid, fixed in b13a359. At the deadline the helper now kills the whole tree (the runner copy is its own process group on POSIX; taskkill /T on Windows) rather than only the runner, and if close has still not arrived 15 s later it abandons the streams and rejects anyway. Settlement is still single, so nothing reads the fixture after it.
Three tests added: a child that inherits the output pipes is killed with its parent; a holder that survives a no-op kill cannot keep the wait open; a control that finishes in time. Mutation checks: removing the fallback settlement hangs the second test; replacing the tree kill with child.kill() fails the first test on Linux (checked in a container). On Windows that first mutation is not caught by the first test, because Node's children there end with their parent anyway.
) At the deadline the fixture helper now kills the whole process tree (its own process group on POSIX) instead of only the runner copy, and if 'close' still has not arrived after a grace period it abandons the streams and rejects anyway. Settlement stays single: nothing reads the fixture directory after the promise settles. Tests: a pipe-holding child is killed with its parent; a holder that survives the kill cannot keep the wait open; a command that finishes in time resolves normally. Assisted-by: Cindy (https://cindy.app)
| return new Promise((resolve, reject) => { | ||
| // Its own process group on POSIX, so the deadline kill reaches every | ||
| // descendant, including ones started before the runner's own handlers. | ||
| const child = spawn(command, args, { cwd, env, detached: process.platform !== "win32" }); |
There was a problem hiding this comment.
(ai: openai/gpt-6-astra)
WARNING: Forward cancellation to the newly detached fixture tree
On POSIX, pressing Ctrl+C during a direct targeted test run while either pipeHolder() case is waiting leaves parent.mjs and holder.mjs running: this spawn moves them outside the foreground process group, but runToClose() installs no SIGINT/SIGTERM cleanup. Once the test worker exits, neither its deadline nor its after-hook can terminate the fixtures, whose intervals run indefinitely. Unlike the production runWatched() helper, this new detached spawn has no signal-forwarding path. Kill the owned tree on cancellation, remove the handlers on settlement, and cover cancellation of a waiting fixture.
There was a problem hiding this comment.
(ai: anthropic/claude-fable-5-2)
Valid, fixed in 5cbc93a. On POSIX the helper now kills its child's tree on SIGINT/SIGTERM and exits 130/143; the listeners are removed on settlement. Checked in a node:24 Linux container by signalling a targeted run mid-wait: 2 fixture processes before, 0 after, for both signals. No committed test for it, for the same reason as the earlier cancellation thread: it is POSIX only and this fork's CI is Windows only.
| (e: Error) => e, | ||
| ); | ||
| assert.ok(err, "the wait ends even though 'close' never came"); | ||
| pids = pidsIn(err.message); |
There was a problem hiding this comment.
(ai: openai/gpt-6-astra)
WARNING: Preserve cleanup ownership when PID output is missing
If fixture startup or stdout delivery exceeds the 2-second deadline plus 1.5-second grace, pidsIn() throws before this assignment, leaving pids empty for the after-hook. This case deliberately makes the kill a no-op, and the fallback has already unreferenced the child, so the detached interval fixture can remain running after the test fails. Record the root PID independently of stdout, for example in the injected kill callback, so teardown can always kill its group even when PID parsing fails.
There was a problem hiding this comment.
(ai: anthropic/claude-fable-5-2)
Valid, fixed in 5cbc93a. The test now records the root pid from the injected kill callback (and asserts it was handed one), and teardown kills that tree before the parsed pids, so a run that never printed its pids is still cleaned up.
…utput (#69) On POSIX the helper's child runs in its own process group, so SIGINT/SIGTERM to the test process now kills that tree before exiting; the listeners are removed on settlement. The no-op-kill test records the root pid from the kill callback, so teardown reaches the fixture even if its pids were never printed. Assisted-by: Cindy (https://cindy.app)
(ai: anthropic/claude-fable-5-2)
Closes #69. Related: #61 (stays open; this does not fix the stall, it makes it short and visible).
What changes
Every phase the CI runner starts (main run, full retry, isolated re-runs) now runs under a stall watch,
scripts/ci-stall-watch.mjs:taskkill /T /Fon Windows; the phase's own process group elsewhere).reference/testing.mdgains a section describing this.Change from what was discussed
The report does not list "files that have not reported yet". TAP output does not carry file names for passing files, so that list cannot be derived honestly. The fallback is the last output before the silence. The process list is the primary signal and names the file when the stuck process is still under the phase (checked below).
The limit also needs your eye: I said I would come back if the longest normal quiet gap was not well under 10 minutes. It is 238.5 s, about 40% of the limit. I judged that enough margin and proceeded; say so if you want a longer limit.
Evidence
run-ci-tests.mjsand its imports with the limit shortened to 8 s in the copy only, the real argument chain (--import tsx,.test.tsfiles), one hanging file and one passing file. Result:STALL:lines namingtests/x.test.ts, theSTALLED … WITHOUT retryline, exit code 1, no retry. With thefailIfStalledcall removed from the copy, the same run fell through to the classifier, which labelled the cut-off output a concrete assertion failure.tests/ci-stall-watch.test.ts, 11 tests, real child processes:tests/ci-runner-stall-gate.test.ts, 8 tests (added after review): a copy of the runner run as a subprocess, limit shortened in the copy only. A stall in the main phase, an isolated re-run and the full retry each exit 1 with no further re-run or retry; a clean run passes; a swallowed failure is still re-run, retried and passed. Removing any one of the three gate calls fails exactly its own test. Three more tests cover the fixture helper's own deadline: it kills the whole tree, and a process that survives the kill and holds the output pipes cannot keep the wait open.ci-stall-watch11 of 11; a failedpsno longer leaves the child running; cancelling the runner (SIGTERM and SIGINT) left no processes behind, including ones in a nested process group (two were left before the second review fix). The cancellation path has no committed test: it is POSIX only and this fork's CI is Windows only.check:unused-exportsandcheck:test-import-graphclean.ci-shard-weights,ci-runner-failure-shapes,issue5631-restart-survival: 41 of 41.Not covered
taskkilldo run on windows-latest throughtests/ci-stall-watch.test.tsin this PR's CI.pretestbuild step is not watched.ps) was run in a local Linux container only; the fork's CI is Windows only.Assisted-by: Cindy (https://cindy.app)