You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Tests: tests/dsh-native.test.ts stalled a Windows / Node 24 CI shard for 40 minutes with no output #61
On PR #57 (head 7307a283d), CI run 43559914969, first attempt, job test (windows-latest, 24, 1) stopped producing output at 05:08:50 UTC, 52 seconds after the test phase began, and was still silent 40 minutes later. The same leg took under 6 minutes on the previous run. The run was cancelled by hand at 05:48 UTC to release the log; the job timeout is 90 minutes.
Evidence
The log ends at ok 250 - agentless compaction dispatch (sessionId, no initiator) is attributed, the last test of tests/dsh-llm-stream-attribution.test.ts. No failure, no cancellation, no runner message follows.
node --test reports files in alphabetical order (the log starts with tests/accept-encoding-restrict.test.ts), and holds back later files until the one at the head of the order finishes. The next file of this shard in that order is tests/dsh-native.test.ts, and none of its tests appear in the log. So that file's process did not finish in 40 minutes. Its recorded weight in scripts/ci-shard-weights.json is about 150 seconds.
Which test inside tests/dsh-native.test.ts hung is not known: nothing from the file was printed.
Whether another file later in the order was also stuck is not known; their output was held back behind this one.
Cause is not established. The file spawns child processes and opens listeners, so a child that never exits or a wait with no deadline are candidates, not findings.
One occurrence. The file passed in the full local run on 2026-11-13 and on earlier CI runs.
Related gap
scripts/run-ci-tests.mjs has no stall detection, so a hung file is silent until the 90-minute job timeout and the log never names it. A per-phase inactivity message naming the files still running would make the next occurrence diagnosable. That is a runner change and a separate scope from this report.
Reproduction attempt, 2026-11-14: tests/dsh-native.test.ts on Windows 11, Node v24.14.0, run alone twice with a 15-minute limit, using the documented test command. Result: clean 2 of 2. Not reproduced. Two runs is a small sample, and both were without the other files of the shard running alongside.
The same CI job passed on re-run on the same commit (PR #57, run 43559914969, attempt 2: 2105 tests, 0 failed, 237.5 s).
Cause still not established and no change made. The gap noted above stands: the runner prints nothing when a file hangs, so a recurrence will again be silent until the job timeout unless stall reporting is added to scripts/run-ci-tests.mjs.
PR #65 (head dce3f48a4, changes tests/issue411-abort-usage.test.ts only), run 43610934093 attempt 1, job test (windows-latest, 24, 1).
Test phase began 15:36:06 UTC. The last line printed was ok 250 - agentless compaction dispatch (sessionId, no initiator) is attributed at 15:37:01, then nothing for 22 minutes. Cancelled by hand at 15:58:59.
This is the same stall point as the first occurrence: the last file reported is tests/dsh-llm-stream-attribution.test.ts, and the next file of the shard in report order is tests/dsh-native.test.ts, with none of its tests printed.
So this is intermittent and recurring, not a one-off. Still not reproduced locally (2 of 2 clean, run alone). Cause not established.
What I checked and what it does not show
The file has several places that reserve a loopback port, release it, and later listen on that number. I checked in a miniature test whether a port taken in between could explain a silent hang: it does not fit cleanly. Node reports that test as failed (EADDRINUSE, uncaughtException) and prints the result, whereas the CI log shows no result from this file at all. So either the stall happens before the file's first test completes, or the parent holds the file's output back; the log cannot tell which. --test-timeout did not end the miniature case either.
Recommendation
Two stalls in eight attempts will keep costing 20 to 90 minutes each and a manual cancel. The runner (scripts/run-ci-tests.mjs) should end a phase that has printed nothing for several minutes and say so, so the existing isolated re-run can name the file. That is a runner change and needs its own issue and the owner's go-ahead. Suggest raising this issue to p2.
Master 4b6687bc8, Windows 11, Node v24.14.0 locally. No timeout, wait or threshold was changed.
1. The stall sits before the file's first test finishes. I had left open whether the parent was holding this file's output back. It is not: in a two-file miniature under node --test --test-reporter=tap, a file at the head of the report order prints each finished test at once (ok 1 appeared while its second test was still hanging), while a file whose first test hangs prints nothing. Both CI stalls printed nothing from tests/dsh-native.test.ts. So, given it was the file at the head of the order, the process was stuck in module loading, in the file's top-level code, or inside its first test (#3370: dsh incremental conversion is atomic…, line 20), and never completed that test. The 130 or so later tests of the file are not involved.
Inside that first test, waitFor has a 15 s deadline and would have reported a failure. The awaits with no deadline are startMockProxy (listen on port 0), await _stateToolsReadyForTest() and the three await beforeSend(...) calls. Those are where a silent hang can sit; which one, if any, is not known.
2. Not reproduced with its neighbours. The file run together with nine neighbouring shard-1 files at concurrency 4, 12 rounds, 5-minute limit each: 12 of 12 clean, about 13 s per round. With the two earlier solo runs that is 14 clean local runs.
3. CI since the two stalls. Shard 1 has passed on every attempt since (PR #68, five heads of PR #72, and the master run 43655618829 on 4b6687bc8). Still 2 stalls, both on 2026-11-14.
4. Not tested. CI runs Node 24.22.2; every local run here used 24.14.0. The CI runner is also slower and more loaded than this machine (the file's recorded weight is about 150 s there).
Where that leaves it
Cause not established. The stall watch from #69 is now on master: the next occurrence fails the job after 10 minutes and prints the process list, which will show whether the file's own process is alive and what children it has. Copy that list here when it happens. No code change is proposed until then.
The nine files I ran beside tests/dsh-native.test.ts were its neighbours in today's shard 1 (decompress-image-ref-dispatch … e2e-ptc-runcode-shape), not the ones running on the day of the stalls. Both stall logs end in tests/dsh-llm-stream-attribution.test.ts, which is not beside it in today's partition, so the partition has moved since.
I tried to rebuild the shard from the tree and weights at 7307a283d. My reconstruction did not place dsh-native in shard 1 at all, so it is wrong somewhere and I did not get a usable neighbour list. That attempt produced no evidence either way and I stopped it.
So the 12 clean rounds show only that the file passes beside today's neighbours on this machine. The stall-day neighbour set is untested. Point 1 (the file never finished its first test) does not depend on this and stands.
Activity
anuzzolilli-oss commentedon Nov 14, 2026
(ai: anthropic/claude-fable-5-2)
Reproduction attempt, 2026-11-14:
tests/dsh-native.test.tson Windows 11, Node v24.14.0, run alone twice with a 15-minute limit, using the documented test command. Result: clean 2 of 2. Not reproduced. Two runs is a small sample, and both were without the other files of the shard running alongside.The same CI job passed on re-run on the same commit (PR #57, run 43559914969, attempt 2: 2105 tests, 0 failed, 237.5 s).
Cause still not established and no change made. The gap noted above stands: the runner prints nothing when a file hangs, so a recurrence will again be silent until the job timeout unless stall reporting is added to
scripts/run-ci-tests.mjs.anuzzolilli-oss commentedon Nov 14, 2026
(ai: anthropic/claude-fable-5-2)
Second occurrence, 2026-11-14
PR #65 (head
dce3f48a4, changestests/issue411-abort-usage.test.tsonly), run 43610934093 attempt 1, jobtest (windows-latest, 24, 1).ok 250 - agentless compaction dispatch (sessionId, no initiator) is attributedat 15:37:01, then nothing for 22 minutes. Cancelled by hand at 15:58:59.tests/dsh-llm-stream-attribution.test.ts, and the next file of the shard in report order istests/dsh-native.test.ts, with none of its tests printed.So this is intermittent and recurring, not a one-off. Still not reproduced locally (2 of 2 clean, run alone). Cause not established.
What I checked and what it does not show
The file has several places that reserve a loopback port, release it, and later listen on that number. I checked in a miniature test whether a port taken in between could explain a silent hang: it does not fit cleanly. Node reports that test as failed (
EADDRINUSE,uncaughtException) and prints the result, whereas the CI log shows no result from this file at all. So either the stall happens before the file's first test completes, or the parent holds the file's output back; the log cannot tell which.--test-timeoutdid not end the miniature case either.Recommendation
Two stalls in eight attempts will keep costing 20 to 90 minutes each and a manual cancel. The runner (
scripts/run-ci-tests.mjs) should end a phase that has printed nothing for several minutes and say so, so the existing isolated re-run can name the file. That is a runner change and needs its own issue and the owner's go-ahead. Suggest raising this issue to p2.anuzzolilli-oss commentedon Nov 15, 2026
(ai: anthropic/claude-fable-5-2)
Follow-up 2026-11-15: narrowed, not reproduced
Master
4b6687bc8, Windows 11, Node v24.14.0 locally. No timeout, wait or threshold was changed.1. The stall sits before the file's first test finishes. I had left open whether the parent was holding this file's output back. It is not: in a two-file miniature under
node --test --test-reporter=tap, a file at the head of the report order prints each finished test at once (ok 1appeared while its second test was still hanging), while a file whose first test hangs prints nothing. Both CI stalls printed nothing fromtests/dsh-native.test.ts. So, given it was the file at the head of the order, the process was stuck in module loading, in the file's top-level code, or inside its first test (#3370: dsh incremental conversion is atomic…, line 20), and never completed that test. The 130 or so later tests of the file are not involved.Inside that first test,
waitForhas a 15 s deadline and would have reported a failure. The awaits with no deadline arestartMockProxy(listen on port 0),await _stateToolsReadyForTest()and the threeawait beforeSend(...)calls. Those are where a silent hang can sit; which one, if any, is not known.2. Not reproduced with its neighbours. The file run together with nine neighbouring shard-1 files at concurrency 4, 12 rounds, 5-minute limit each: 12 of 12 clean, about 13 s per round. With the two earlier solo runs that is 14 clean local runs.
3. CI since the two stalls. Shard 1 has passed on every attempt since (PR #68, five heads of PR #72, and the master run 43655618829 on
4b6687bc8). Still 2 stalls, both on 2026-11-14.4. Not tested. CI runs Node 24.22.2; every local run here used 24.14.0. The CI runner is also slower and more loaded than this machine (the file's recorded weight is about 150 s there).
Where that leaves it
Cause not established. The stall watch from #69 is now on master: the next occurrence fails the job after 10 minutes and prints the process list, which will show whether the file's own process is alive and what children it has. Copy that list here when it happens. No code change is proposed until then.
anuzzolilli-oss commentedon Nov 15, 2026
(ai: anthropic/claude-fable-5-2)
Correction to point 2 of my previous comment
The nine files I ran beside
tests/dsh-native.test.tswere its neighbours in today's shard 1 (decompress-image-ref-dispatch…e2e-ptc-runcode-shape), not the ones running on the day of the stalls. Both stall logs end intests/dsh-llm-stream-attribution.test.ts, which is not beside it in today's partition, so the partition has moved since.I tried to rebuild the shard from the tree and weights at
7307a283d. My reconstruction did not placedsh-nativein shard 1 at all, so it is wrong somewhere and I did not get a usable neighbour list. That attempt produced no evidence either way and I stopped it.So the 12 clean rounds show only that the file passes beside today's neighbours on this machine. The stall-day neighbour set is untested. Point 1 (the file never finished its first test) does not depend on this and stands.