Skip to content

Tests: tests/dsh-native.test.ts stalled a Windows / Node 24 CI shard for 40 minutes with no output #61

Open

Description

@anuzzolilli-oss

(ai: anthropic/claude-fable-5-2)

What happened

On PR #57 (head 7307a283d), CI run 43559914969, first attempt, job test (windows-latest, 24, 1) stopped producing output at 05:08:50 UTC, 52 seconds after the test phase began, and was still silent 40 minutes later. The same leg took under 6 minutes on the previous run. The run was cancelled by hand at 05:48 UTC to release the log; the job timeout is 90 minutes.

Evidence

  • The log ends at ok 250 - agentless compaction dispatch (sessionId, no initiator) is attributed, the last test of tests/dsh-llm-stream-attribution.test.ts. No failure, no cancellation, no runner message follows.
  • node --test reports files in alphabetical order (the log starts with tests/accept-encoding-restrict.test.ts), and holds back later files until the one at the head of the order finishes. The next file of this shard in that order is tests/dsh-native.test.ts, and none of its tests appear in the log. So that file's process did not finish in 40 minutes. Its recorded weight in scripts/ci-shard-weights.json is about 150 seconds.
  • Runner: windows-latest, Node v24.22.2, 225 files, concurrency 4.

What is and is not known

  • PR test: hold the event loop open during watchdog waits; remove the Node 22 skips #57 changes only three test files. The one in this shard, tests/claude-mcp-entry.test.ts, ran and passed all its tests (numbers 96 to 109 and onward) before the stall. Each file runs in its own process.
  • Which test inside tests/dsh-native.test.ts hung is not known: nothing from the file was printed.
  • Whether another file later in the order was also stuck is not known; their output was held back behind this one.
  • Cause is not established. The file spawns child processes and opens listeners, so a child that never exits or a wait with no deadline are candidates, not findings.
  • One occurrence. The file passed in the full local run on 2026-11-13 and on earlier CI runs.

Related gap

scripts/run-ci-tests.mjs has no stall detection, so a hung file is silent until the 90-minute job timeout and the log never names it. A per-phase inactivity message naming the files still running would make the next occurrence diagnosable. That is a runner change and a separate scope from this report.

Activity

  1. anuzzolilli-oss commented on Nov 14, 2026

    @anuzzolilli-oss
    OwnerAuthor

    (ai: anthropic/claude-fable-5-2)

    Reproduction attempt, 2026-11-14: tests/dsh-native.test.ts on Windows 11, Node v24.14.0, run alone twice with a 15-minute limit, using the documented test command. Result: clean 2 of 2. Not reproduced. Two runs is a small sample, and both were without the other files of the shard running alongside.

    The same CI job passed on re-run on the same commit (PR #57, run 43559914969, attempt 2: 2105 tests, 0 failed, 237.5 s).

    Cause still not established and no change made. The gap noted above stands: the runner prints nothing when a file hangs, so a recurrence will again be silent until the job timeout unless stall reporting is added to scripts/run-ci-tests.mjs.

  2. anuzzolilli-oss commented on Nov 14, 2026

    @anuzzolilli-oss
    OwnerAuthor

    (ai: anthropic/claude-fable-5-2)

    Second occurrence, 2026-11-14

    PR #65 (head dce3f48a4, changes tests/issue411-abort-usage.test.ts only), run 43610934093 attempt 1, job test (windows-latest, 24, 1).

    So this is intermittent and recurring, not a one-off. Still not reproduced locally (2 of 2 clean, run alone). Cause not established.

    What I checked and what it does not show

    The file has several places that reserve a loopback port, release it, and later listen on that number. I checked in a miniature test whether a port taken in between could explain a silent hang: it does not fit cleanly. Node reports that test as failed (EADDRINUSE, uncaughtException) and prints the result, whereas the CI log shows no result from this file at all. So either the stall happens before the file's first test completes, or the parent holds the file's output back; the log cannot tell which. --test-timeout did not end the miniature case either.

    Recommendation

    Two stalls in eight attempts will keep costing 20 to 90 minutes each and a manual cancel. The runner (scripts/run-ci-tests.mjs) should end a phase that has printed nothing for several minutes and say so, so the existing isolated re-run can name the file. That is a runner change and needs its own issue and the owner's go-ahead. Suggest raising this issue to p2.

  3. anuzzolilli-oss commented on Nov 15, 2026

    @anuzzolilli-oss
    OwnerAuthor

    (ai: anthropic/claude-fable-5-2)

    Follow-up 2026-11-15: narrowed, not reproduced

    Master 4b6687bc8, Windows 11, Node v24.14.0 locally. No timeout, wait or threshold was changed.

    1. The stall sits before the file's first test finishes. I had left open whether the parent was holding this file's output back. It is not: in a two-file miniature under node --test --test-reporter=tap, a file at the head of the report order prints each finished test at once (ok 1 appeared while its second test was still hanging), while a file whose first test hangs prints nothing. Both CI stalls printed nothing from tests/dsh-native.test.ts. So, given it was the file at the head of the order, the process was stuck in module loading, in the file's top-level code, or inside its first test (#3370: dsh incremental conversion is atomic…, line 20), and never completed that test. The 130 or so later tests of the file are not involved.

    Inside that first test, waitFor has a 15 s deadline and would have reported a failure. The awaits with no deadline are startMockProxy (listen on port 0), await _stateToolsReadyForTest() and the three await beforeSend(...) calls. Those are where a silent hang can sit; which one, if any, is not known.

    2. Not reproduced with its neighbours. The file run together with nine neighbouring shard-1 files at concurrency 4, 12 rounds, 5-minute limit each: 12 of 12 clean, about 13 s per round. With the two earlier solo runs that is 14 clean local runs.

    3. CI since the two stalls. Shard 1 has passed on every attempt since (PR #68, five heads of PR #72, and the master run 43655618829 on 4b6687bc8). Still 2 stalls, both on 2026-11-14.

    4. Not tested. CI runs Node 24.22.2; every local run here used 24.14.0. The CI runner is also slower and more loaded than this machine (the file's recorded weight is about 150 s there).

    Where that leaves it

    Cause not established. The stall watch from #69 is now on master: the next occurrence fails the job after 10 minutes and prints the process list, which will show whether the file's own process is alive and what children it has. Copy that list here when it happens. No code change is proposed until then.

  4. anuzzolilli-oss commented on Nov 15, 2026

    @anuzzolilli-oss
    OwnerAuthor

    (ai: anthropic/claude-fable-5-2)

    Correction to point 2 of my previous comment

    The nine files I ran beside tests/dsh-native.test.ts were its neighbours in today's shard 1 (decompress-image-ref-dispatch … e2e-ptc-runcode-shape), not the ones running on the day of the stalls. Both stall logs end in tests/dsh-llm-stream-attribution.test.ts, which is not beside it in today's partition, so the partition has moved since.

    I tried to rebuild the shard from the tree and weights at 7307a283d. My reconstruction did not place dsh-native in shard 1 at all, so it is wrong somewhere and I did not get a usable neighbour list. That attempt produced no evidence either way and I stopped it.

    So the 12 clean rounds show only that the file passes beside today's neighbours on this machine. The stall-day neighbour set is untested. Point 1 (the file never finished its first test) does not depend on this and stands.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions