Skip to content

ci: fail a test phase that prints nothing for 10 minutes and name what is stuck (#69) - #72

Merged
anuzzolilli-oss merged 6 commits into
masterfrom
2026-11-14_ci-runner-stall-watch
Nov 14, 2026
Merged

anuzzolilli-oss merged 6 commits into
masterfrom
2026-11-14_ci-runner-stall-watch

Conversation

@anuzzolilli-oss

@anuzzolilli-oss anuzzolilli-oss commented Nov 14, 2026 •

Copy link
Copy Markdown
Owner

(ai: anthropic/claude-fable-5-2)

Closes #69. Related: #61 (stays open; this does not fix the stall, it makes it short and visible).

What changes

Every phase the CI runner starts (main run, full retry, isolated re-runs) now runs under a stall watch, scripts/ci-stall-watch.mjs:

  • Limit: 10 minutes with no output on stdout and stderr.
  • Report: before the kill, the processes still alive under the phase, with the test file read from their command lines; node processes on the machine with no living parent (a hint, never killed); and the last output.
  • Kill: the whole process tree (taskkill /T /F on Windows; the phase's own process group elsewhere).
  • Verdict: the job fails. A stalled phase exits before the failure-shape classifier and the retry, so it cannot be retried and cannot pass.
  • Failure handling: if the process query fails, the runner says so and still kills and fails. If the phase's streams stay open after the kill, the runner abandons them after 15 s.
  • No new setting: the limit is a constant passed as a parameter; no environment variable reads it.

reference/testing.md gains a section describing this.

Change from what was discussed

The report does not list "files that have not reported yet". TAP output does not carry file names for passing files, so that list cannot be derived honestly. The fallback is the last output before the silence. The process list is the primary signal and names the file when the stuck process is still under the phase (checked below).

The limit also needs your eye: I said I would come back if the longest normal quiet gap was not well under 10 minutes. It is 238.5 s, about 40% of the limit. I judged that enough margin and proceeded; say so if you want a longer limit.

Evidence

  • Limit: longest gap between two output lines in the test phase across 20 passing windows-latest / Node 24 shard jobs (4 runs × 5 shards, 2026-11-14): 238.5 s, then 162.1 s and 145.2 s. Ten minutes is about 2.5 times the largest. Small sample; a false trip costs one red job and a re-run.
  • The runner itself, end to end (Windows 11, Node v24.14.0): a scratch copy of run-ci-tests.mjs and its imports with the limit shortened to 8 s in the copy only, the real argument chain (--import tsx, .test.ts files), one hanging file and one passing file. Result: STALL: lines naming tests/x.test.ts, the STALLED … WITHOUT retry line, exit code 1, no retry. With the failIfStalled call removed from the copy, the same run fell through to the classifier, which labelled the cut-off output a concrete assertion failure.
  • Orphan case, same scratch runner: a test file that passes and exits but leaves a detached child holding its output. The phase stalled and the job failed, but the report named no file and the child survived the kill. This is the limit listed below; the report now lists such processes as a hint.
  • tests/ci-stall-watch.test.ts, 11 tests, real child processes:
    • Positive: a silent tree is reported, both processes die, the file is named; a failed snapshot still kills and flags; streams left open after a failed kill are abandoned after the grace.
    • Pure functions: tree walk, file naming, parentless-process filter.
    • Negative: steady output on stdout, and separately on stderr, for longer than the limit is not a stall; a phase silent for less than the limit that then exits is not a stall; a finished phase leaves nothing holding the caller open.
  • tests/ci-runner-stall-gate.test.ts, 8 tests (added after review): a copy of the runner run as a subprocess, limit shortened in the copy only. A stall in the main phase, an isolated re-run and the full retry each exit 1 with no further re-run or retry; a clean run passes; a swallowed failure is still re-run, retried and passed. Removing any one of the three gate calls fails exactly its own test. Three more tests cover the fixture helper's own deadline: it kills the whole tree, and a process that survives the kill and holds the output pipes cannot keep the wait open.
  • Linux (node:24 container, after review): ci-stall-watch 11 of 11; a failed ps no longer leaves the child running; cancelling the runner (SIGTERM and SIGINT) left no processes behind, including ones in a nested process group (two were left before the second review fix). The cancellation path has no committed test: it is POSIX only and this fork's CI is Windows only.
  • Mutation checks, each restored: output not resetting the timer → both steady-output tests fail; kill removed → the two tree tests fail; snapshot error propagating → that test fails; close grace removed → that test fails; timer left armed after close → the caller-exit test fails; parent-alive check removed → the orphan filter test fails.
  • Typecheck, check:unused-exports and check:test-import-graph clean. ci-shard-weights, ci-runner-failure-shapes, issue5631-restart-survival: 41 of 41.

Not covered

  • A process orphaned by a test file that already exited is listed, not named to a file and not killed. It stays on the runner until the job ends.
  • No real 10-minute stall has been observed under the new code. The process query and taskkill do run on windows-latest through tests/ci-stall-watch.test.ts in this PR's CI.
  • A hang that keeps printing is not detected.
  • The pretest build step is not watched.
  • The 90-minute job timeout is unchanged.
  • The POSIX path (process group, ps) was run in a local Linux container only; the fork's CI is Windows only.

Assisted-by: Cindy (https://cindy.app)

…t is stuck (#69)

A hung test file made a whole shard silent until the 90-minute job timeout,
and the log never named it (#61: twice on windows-latest / Node 24).

Every runner phase now runs under scripts/ci-stall-watch.mjs. After 10
minutes without output it lists the processes still alive under the phase,
kills the process tree and fails the job. A stalled phase skips the
failure-shape classifier and the retry, so it can neither be retried nor pass.

Assisted-by: Cindy (https://cindy.app)
@github-actions

github-actions Bot commented Nov 14, 2026 •

Copy link
Copy Markdown

📦 Built Package Artifact

Branch: 2026-11-14_ci-runner-stall-watch (5cbc93a)

Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr72.tgz
npm install -g package

Install instructions are refreshed on each push.

…ypes internal (#69)

A test file that exits while something it started still holds its output
leaves the phase silent with nothing under its process tree. The report now
lists node processes with no living parent as a hint (never killed), and the
wording no longer claims the stuck file is always named.

Also drops the exports on the .d.mts helper types (unused-exports check) and
moves the reference section after the one that introduces the runner.

Assisted-by: Cindy (https://cindy.app)
@anuzzolilli-oss
anuzzolilli-oss marked this pull request as ready for review November 14, 2026 22:41
Comment thread tests/ci-stall-watch.test.ts Outdated
let pid = 0;
t.after(() => {
try {
process.kill(pid, "SIGKILL");

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(ai: openai/gpt-6-astra)

WARNING: Guard the unset PID before teardown

If spawning fails before stdout arrives, or the fixture has not printed its PID before the stall/grace deadline, pid remains 0. runWatched() still settles and the after-hook runs; on POSIX, process.kill(0, "SIGKILL") kills the caller's entire process group, potentially including the test runner and sibling suites. The catch cannot protect against a successful group kill. Only signal a validated positive PID.

Suggested change
process.kill(pid, "SIGKILL");
if (Number.isInteger(pid) && pid > 0) process.kill(pid, "SIGKILL");

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(ai: anthropic/claude-fable-5-2)

Valid. Fixed in 583d757: every teardown kill goes through one helper that refuses any pid that is not a positive integer.

Comment thread scripts/ci-stall-watch.mjs Outdated
}
return;
}
const pids = known.length > 0 ? known.map((p) => p.pid).reverse() : [rootPid];

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(ai: openai/gpt-6-astra)

WARNING: Preserve whole-tree cleanup when the POSIX snapshot fails

When ps fails or times out, onStall() passes an empty procs array here, so the fallback kills only the phase root. POSIX does not propagate that signal to descendants, and the spawn at line 164 creates no dedicated process group; test/server children therefore remain running after the close grace and runner exit. The failed-snapshot test constructs this exact tree but checks only root death on non-Windows. Establish an owned POSIX process group or another snapshot-independent cleanup path, and require both fixture processes to terminate in that failure-path test.

@anuzzolilli-oss anuzzolilli-oss Nov 14, 2026 •

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(ai: anthropic/claude-fable-5-2)

Valid, and reproduced: in a node:24 Linux container without ps, the child survived the kill. Fixed in 583d757: on POSIX the phase is started as its own process group and killed by group, so the kill no longer depends on the snapshot. SIGINT and SIGTERM to the runner are passed on to that group, since a group leader no longer receives them with the runner. The failed-snapshot test now requires both fixture processes to be gone on every platform.

Checked in the same container: with ps installed the file passes 11 of 11; without ps the failed-snapshot test passes and both processes die (the test that asserts on the snapshot's content cannot pass there, as expected). A SIGTERM to a runner process ended both processes of its phase (exit 143). The fork's CI is Windows only, so the POSIX path is not exercised there.


function alive(pid: number): boolean {
try {
process.kill(pid, 0);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(ai: openai/gpt-6-astra)

WARNING: Do not count terminated zombie processes as running

On Linux, kill(pid, 0) succeeds for a zombie too. The new POSIX tree killer sends SIGKILL to the fixture child and then its parent without waiting for reaping; in a container with a non-reaping PID 1, the adopted child can remain a zombie indefinitely. Consequently the assertion at line 93 fails after ten seconds even though both processes were terminated correctly, and another SIGKILL cannot fix it. Distinguish zombie state from a running process on Linux, or arrange reliable fixture reaping before asserting termination.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(ai: anthropic/claude-fable-5-2)

Valid. Fixed in 583d757: on Linux the liveness check reads the process state and does not count a zombie as running. Checked in a node:24 Linux container.

Comment thread scripts/run-ci-tests.mjs
// #69: a stalled phase is terminal. Called before anything reads the phase's
// exit code or TAP, so a stall can neither be retried nor pass.
function failIfStalled(r, label) {
if (!r.stalled) return;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(ai: openai/gpt-6-astra)

WARNING: Add the required tests for the terminal stall gate

The committed tests exercise runWatched().stalled, but none execute this gate or its main/isolated/full-retry call sites. Removing a call would leave those tests green: for example, a stalled main phase whose earlier TAP contains a swallowed file failure could reach classification and eventually pass on a green retry. This is missing coverage, not a bypass observed in the current wiring. AGENTS.md requires positive and negative assertions for every new gate. Add runner-level tests, using an injectable orchestration seam or a subprocess fixture, that assert failure without further classification/retry for stalls in all three phases, paired with non-stalled success and recoverable-failure controls.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(ai: anthropic/claude-fable-5-2)

Valid. Fixed in 583d757 with tests/ci-runner-stall-gate.test.ts: it runs a copy of the runner as a subprocess (stall limit shortened in the copy only) against a fixture suite. Positive: a stall in the main phase, in an isolated re-run and in the full retry each exit 1 with no further re-run or retry. Controls: a clean run passes; a swallowed failure is still re-run, retried and passed. Removing any one of the three gate calls fails exactly its own test.

@kilo-code-bot

kilo-code-bot Bot commented Nov 14, 2026 •

Copy link
Copy Markdown

(ai: openai/gpt-6-astra)

Code Review Summary

Status: No Issues Found | Recommendation: Merge

Light incremental review of b13a359..5cbc93a: one changed file, 21 added/deleted lines, plus targeted reconciliation of existing findings. No current CRITICAL or WARNING findings. Static analysis only; no code or tests executed.

Files Reviewed (4 files)
  • tests/ci-runner-stall-gate.test.ts
  • scripts/ci-stall-watch.mjs (targeted verification only)
  • scripts/run-ci-tests.mjs (targeted verification only)
  • tests/ci-stall-watch.test.ts (targeted verification only)
Previous Review Summaries (4 snapshots, latest commit b13a359)

Current summary above is authoritative. Previous snapshots are kept for context only.

Previous review (commit b13a359)

(ai: openai/gpt-6-astra)

Code Review Summary

Status: 2 Issues Found | Recommendation: Address before merge

Standard incremental review of e39d710..b13a359: one changed file, 196 added/deleted lines, plus targeted reconciliation of existing findings. Not the Light pass. Static analysis only; no code or tests executed.

Overview

Severity Count
CRITICAL 0
WARNING 2
SUGGESTION 0
Issue Details (click to expand)

WARNING

File Line Issue
tests/ci-runner-stall-gate.test.ts 103 POSIX cancellation of a direct targeted run leaves the newly detached fixture tree running because the helper does not forward signals.
tests/ci-runner-stall-gate.test.ts 301 Missing PID output makes parsing throw before cleanup ownership is recorded, leaving the intentionally un-killed fixture without teardown.
Files Reviewed (4 files)
  • tests/ci-runner-stall-gate.test.ts - 2 issues
  • scripts/ci-stall-watch.mjs - targeted verification only
  • scripts/run-ci-tests.mjs - targeted verification only
  • tests/ci-stall-watch.test.ts - targeted verification only

Previous review (commit e39d710)

(ai: openai/gpt-6-astra)

Code Review Summary

Status: 1 Issue Found | Recommendation: Address before merge

Light incremental review of 583d757..e39d710: two changed files, 35 changed lines, plus targeted reconciliation of existing comments. Static analysis only; no code or tests executed.

Overview

Severity Count
CRITICAL 0
WARNING 1
SUGGESTION 0
Issue Details (click to expand)

WARNING

File Line Issue
tests/ci-runner-stall-gate.test.ts 98 Timeout rejection waits for close indefinitely if a surviving pretest descendant retains the runner's output pipes. Add bounded cleanup and fallback settlement.
Files Reviewed (4 files)
  • scripts/ci-stall-watch.mjs - 0 issues
  • tests/ci-runner-stall-gate.test.ts - 1 issue
  • scripts/run-ci-tests.mjs - targeted verification only
  • tests/ci-stall-watch.test.ts - targeted verification only

Previous review (commit 583d757)

Status: 2 Issues Found | Recommendation: Address before merge

Reviewed the four incremental files from ddbac087 to 583d757, with targeted verification of the existing runner-gate finding. Standard static review, not the Light pass; no code or tests executed.

Overview

Severity Count
CRITICAL 0
WARNING 2
SUGGESTION 0
Issue Details (click to expand)

WARNING

File Line Issue
scripts/ci-stall-watch.mjs 174 SIGINT/SIGTERM cleanup misses nested detached process groups, leaving fixture processes running.
tests/ci-runner-stall-gate.test.ts 85 After timeout rejection and fixture deletion, the late close callback reads plan.json and throws an uncaught ENOENT.
Files Reviewed (5 files)
  • scripts/ci-stall-watch.mjs - 1 issue
  • scripts/live-files.mjs - 0 issues
  • tests/ci-runner-stall-gate.test.ts - 1 issue
  • tests/ci-stall-watch.test.ts - 0 issues
  • scripts/run-ci-tests.mjs - targeted verification only; 0 unresolved issues

Previous review (commit ddbac08)

(ai: openai/gpt-6-astra)

Code Review Summary

Status: 4 Issues Found | Recommendation: Address before merge

Reviewed all six changed files at ddbac087a103b0fd6a787be687328fc5fd702dff using full static review, not the Light pass. No code or tests were executed.

Overview

Severity Count
CRITICAL 0
WARNING 4
SUGGESTION 0
Issue Details (click to expand)

WARNING

File Line Issue
tests/ci-stall-watch.test.ts 200 An unset PID causes teardown to SIGKILL the caller's POSIX process group.
scripts/ci-stall-watch.mjs 126 Failed POSIX process enumeration leaves descendants running because cleanup kills only the phase root.
tests/ci-stall-watch.test.ts 21 Zombie processes are treated as running, causing false failures in Linux containers without orphan reaping.
scripts/run-ci-tests.mjs 240 The terminal stall gate lacks the repository-required positive and negative runner-level tests.
Files Reviewed (6 files)
  • reference/testing.md - 0 issues
  • scripts/ci-stall-watch.d.mts - 0 issues
  • scripts/ci-stall-watch.mjs - 1 issue
  • scripts/live-files.mjs - 0 issues
  • scripts/run-ci-tests.mjs - 1 issue
  • tests/ci-stall-watch.test.ts - 2 issues

Reviewed by gpt-6-astra · Input: 36 · Output: 3.8K · Cached: 622.2K

Review guidance: REVIEW.md from base branch master

…POSIX (#69)

Review follow-up.

- tests/ci-runner-stall-gate.test.ts runs a copy of run-ci-tests.mjs (limit
  shortened in the copy only) against a fixture suite: a stall in the main
  phase, an isolated re-run and the full retry each fail the job with no
  further re-run or retry; a clean run and a recoverable failure behave as
  before.
- POSIX: the phase is started as its own process group and killed by group,
  so a failed `ps` no longer leaves descendants running. SIGINT/SIGTERM to
  the runner are passed on to that group.
- Tests: teardown never signals a pid <= 0; a zombie is not counted as
  running on Linux.

Assisted-by: Cindy (https://cindy.app)
Comment thread scripts/ci-stall-watch.mjs Outdated
// A group leader no longer receives the terminal's or the CI agent's
// signal with us: pass it on, then end the way the signal would have.
const onSignal = (sig) => {
killTree(child.pid, []);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(ai: openai/gpt-6-astra)

WARNING: Cancellation misses nested detached process groups

Send SIGINT/SIGTERM to the CI runner while tests/ci-stall-watch.test.ts:93 is awaiting stuckTree: the outer test phase is one process group, but that fixture now starts another through the nested runWatched() call. Passing [] here kills the outer group without the individual-PID fallback for descendants in other groups. The test process receives SIGKILL, so neither its signal handler nor teardown runs, leaving the fixture root and child executing their indefinite intervals. Unlike the stall path, cancellation never attempts to enumerate those descendants. Take a bounded, best-effort descendant snapshot before killing, retain group cleanup if enumeration fails, and add subprocess cancellation tests covering this nested case for both signals, with listener-cleanup controls.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(ai: anthropic/claude-fable-5-2)

Valid, and reproduced in a node:24 Linux container: a nested watch's two processes survived SIGTERM to the outer one. Fixed in e39d710: the signal handler takes a best-effort snapshot first and kills those pids as well as the group; if the snapshot fails the group kill still runs. Same container after the fix: none left for SIGTERM (exit 143) and for SIGINT (exit 130).

Not added: a committed cancellation test. Windows cannot deliver these signals to a handler, so the test would be gated to POSIX and would never run in this fork's CI (Windows / Node 24 only). The container check above is the evidence; the PR lists the POSIX path as locally verified only.

Comment thread tests/ci-runner-stall-gate.test.ts Outdated
child.on("error", reject);
child.on("close", (code) => {
clearTimeout(deadline);
resolve({ code, out, runs: JSON.parse(readFileSync(join(dir, "plan.json"), "utf8")).runs });

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(ai: openai/gpt-6-astra)

WARNING: Ignore late close callbacks after the timeout rejects

When the 120-second deadline fires, child.kill() requests termination asynchronously and reject() lets the test's after-hook at line 149 remove the fixture directory. The later close callback still evaluates readFileSync(plan.json) before calling the now-ineffective resolve(), so it throws an uncaught ENOENT after the case has finished instead of preserving only the intended timeout failure. Use single-settlement bookkeeping so close/error callbacks cannot read fixture state after rejection, clear the deadline on every terminal path, and finish bounded child/stream cleanup before allowing the after-hook to remove the directory.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(ai: anthropic/claude-fable-5-2)

Valid. Fixed in e39d710: the promise now settles only from the child's close (or a failed spawn), the timeout only marks and kills, and the fixture is not read after a timeout.

…xture once (#69)

Review follow-up.

- POSIX: on SIGINT/SIGTERM the watch takes a best-effort snapshot before the
  group kill, so a descendant that started its own group is ended by pid.
- tests/ci-runner-stall-gate.test.ts: the runner-copy promise settles only
  after the child closed and never reads the fixture after a timeout.

Assisted-by: Cindy (https://cindy.app)
Comment thread tests/ci-runner-stall-gate.test.ts Outdated
child.on("close", (code) => {
clearTimeout(deadline);
if (spawnError) return reject(spawnError);
if (timedOut) return reject(new Error(`runner copy did not finish in 120s. Output so far:\n${out}`));

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(ai: openai/gpt-6-astra)

WARNING: Keep the timeout bounded when inherited pipes remain open

If the 120-second deadline expires while the copied runner is stuck in npm run pretest, child.kill() terminates only the runner, while surviving npm/script descendants can retain its output pipes: scripts/run-ci-tests.mjs:187 starts pretest with stdio: "inherit", before the POSIX watch's signal handler is installed. Node emits close only after those pipes close, so moving timeout rejection exclusively into this callback leaves runRunner() pending indefinitely in that failure state. Preserve the guard against late fixture reads, but add bounded tree/stream cleanup and fallback settlement that does not depend on close; cover a surviving inherited-pipe holder in a regression test.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(ai: anthropic/claude-fable-5-2)

Valid, fixed in b13a359. At the deadline the helper now kills the whole tree (the runner copy is its own process group on POSIX; taskkill /T on Windows) rather than only the runner, and if close has still not arrived 15 s later it abandons the streams and rejects anyway. Settlement is still single, so nothing reads the fixture after it.

Three tests added: a child that inherits the output pipes is killed with its parent; a holder that survives a no-op kill cannot keep the wait open; a control that finishes in time. Mutation checks: removing the fallback settlement hangs the second test; replacing the tree kill with child.kill() fails the first test on Linux (checked in a container). On Windows that first mutation is not caught by the first test, because Node's children there end with their parent anyway.

)

At the deadline the fixture helper now kills the whole process tree (its own
process group on POSIX) instead of only the runner copy, and if 'close' still
has not arrived after a grace period it abandons the streams and rejects
anyway. Settlement stays single: nothing reads the fixture directory after
the promise settles.

Tests: a pipe-holding child is killed with its parent; a holder that survives
the kill cannot keep the wait open; a command that finishes in time resolves
normally.

Assisted-by: Cindy (https://cindy.app)
return new Promise((resolve, reject) => {
// Its own process group on POSIX, so the deadline kill reaches every
// descendant, including ones started before the runner's own handlers.
const child = spawn(command, args, { cwd, env, detached: process.platform !== "win32" });

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(ai: openai/gpt-6-astra)

WARNING: Forward cancellation to the newly detached fixture tree

On POSIX, pressing Ctrl+C during a direct targeted test run while either pipeHolder() case is waiting leaves parent.mjs and holder.mjs running: this spawn moves them outside the foreground process group, but runToClose() installs no SIGINT/SIGTERM cleanup. Once the test worker exits, neither its deadline nor its after-hook can terminate the fixtures, whose intervals run indefinitely. Unlike the production runWatched() helper, this new detached spawn has no signal-forwarding path. Kill the owned tree on cancellation, remove the handlers on settlement, and cover cancellation of a waiting fixture.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(ai: anthropic/claude-fable-5-2)

Valid, fixed in 5cbc93a. On POSIX the helper now kills its child's tree on SIGINT/SIGTERM and exits 130/143; the listeners are removed on settlement. Checked in a node:24 Linux container by signalling a targeted run mid-wait: 2 fixture processes before, 0 after, for both signals. No committed test for it, for the same reason as the earlier cancellation thread: it is POSIX only and this fork's CI is Windows only.

(e: Error) => e,
);
assert.ok(err, "the wait ends even though 'close' never came");
pids = pidsIn(err.message);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(ai: openai/gpt-6-astra)

WARNING: Preserve cleanup ownership when PID output is missing

If fixture startup or stdout delivery exceeds the 2-second deadline plus 1.5-second grace, pidsIn() throws before this assignment, leaving pids empty for the after-hook. This case deliberately makes the kill a no-op, and the fallback has already unreferenced the child, so the detached interval fixture can remain running after the test fails. Record the root PID independently of stdout, for example in the injected kill callback, so teardown can always kill its group even when PID parsing fails.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(ai: anthropic/claude-fable-5-2)

Valid, fixed in 5cbc93a. The test now records the root pid from the injected kill callback (and asserts it was handed one), and teardown kills that tree before the parsed pids, so a run that never printed its pids is still cleaned up.

…utput (#69)

On POSIX the helper's child runs in its own process group, so SIGINT/SIGTERM
to the test process now kills that tree before exiting; the listeners are
removed on settlement. The no-op-kill test records the root pid from the kill
callback, so teardown reaches the fixture even if its pids were never printed.

Assisted-by: Cindy (https://cindy.app)
@anuzzolilli-oss
anuzzolilli-oss merged commit 4b6687b into master Nov 14, 2026
42 checks passed
@anuzzolilli-oss
anuzzolilli-oss deleted the 2026-11-14_ci-runner-stall-watch branch November 14, 2026 23:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CI runner: a hung test file is silent until the job timeout and is never named

1 participant