Frontend Unit (windows-latest) times out on slow runners: dev-up.test.mjs PowerShell spawns hit ETIMEDOUT and the job hits its timeout

[Chris0Jeky]

Two occurrences on 2026-09-02, both on docs/skills-only PRs that touch no frontend or scripts/ci file. (1) PR #2369 head 16194cc, run 33580047231: scripts/ci/dev-up.test.mjs 'PowerShell: pipeline cancellation runs transactional cleanup from finally' failed with spawnSync powershell.exe ETIMEDOUT (20.4 s); sibling tests in the same job ran 3-5x slower than on the ubuntu runner and than on PRs #2366/#2367 minutes earlier; passed on the next head. (2) PR #2377 head 7325638, run 33583930535: the whole Frontend Unit (windows-latest) job was cancelled at 25:00 after start (reusable-frontend-unit.yml declares timeout-minutes: 20; the dev-up step runs node --test --test-concurrency=1 --test-timeout=30000 scripts/ci/dev-up.test.mjs). Not flaky by fiat: the dev-up suite launches real PowerShell/API/Vite processes serially and its wall time is dominated by runner speed. Options to evaluate under the CI-control rule (R4): skip the dev-up suite when the diff touches neither scripts/dev-up* nor scripts/ci/dev-up.test.mjs nor the frontend (the Smart CI planner already has path-to-lane policy), raise the step/job budget, or split the dev-up suite into its own advisory Windows lane. Refs ADR-0066 (#2324), #2370.

[Chris0Jeky]

Third occurrence, and the consequence this issue does not yet name: a timed-out Windows leg voids the required E2E gate

Coordinator pass, 2026-09-03. Occurrence (3) matches the body's occurrence (2) exactly, so the diagnosis here is confirmed rather than revised. What is new is what happens downstream, which is more serious than the flake itself.

Occurrence 3

PR #2397, branch issue-1975/paper-dialog-residuals, head b40df4998616e2fc2530132da219130a2c424f49, run 33702035573 attempt 1.

• Frontend Unit / Frontend Unit (windows-latest) started 2026-09-03T01:03:02Z, killed 01:28:02Z. Exactly the 25:00 shape recorded for PR #2377.
• Step 4 Run source launcher regression suite was still in_progress at the kill; steps 5 through 12 never started. Same step, same suite: node --test --test-concurrency=1 --test-timeout=30000 scripts/ci/dev-up.test.mjs at reusable-frontend-unit.yml:38, under timeout-minutes: 20 at reusable-frontend-unit.yml:19.
• Frontend Unit / Frontend Unit (ubuntu-latest) passed in the same run, so it is Windows-only, consistent with the body.
• Not a supersede cancel. run 33702035573 is the newest ci-required.yml run on that branch and b40df4998 is still the branch tip, so nothing cancelled it from outside. Every other job in the run had already completed successfully; this one job was killed by its own budget.

The consequence: the required behavioural gate is skipped, not failed

When that job is killed the whole CI run concludes cancelled, and E2E Smoke in the same run resolves to skipped. At b40df4998 the check-run set contains E2E Smoke | completed/skipped and contains no E2E Smoke / E2E Smoke result at all. Compare #2408, #2405 and #2403 at their heads, which each carry E2E Smoke / E2E Smoke | success.

So the failure mode is not "a required check went red". It is:

1. A slow Windows runner kills one job on its own timeout.
2. The run is cancelled, so the required E2E job never runs and reports skipped.
3. The PR shows MERGEABLE / UNSTABLE with no failing check to investigate, which reads like a transient rather than an absent gate.

That is a gate-integrity property, not a flake property: a PR can reach a state where the required behavioural gate produced no verdict, and nothing in the PR surface says so.

Why it mattered here

#2397 is parked against two named blockers, and blocker 1 is a real product regression in Enter activation whose only proof is tests/e2e/edge-journeys.spec.ts:284. The jsdom suite cannot see it, as recorded on that PR. Alpha pushed 1031d44c9 and b40df4998 aiming at exactly that blocker, and the check that would have proven the fix never ran. The repair sat unproven for roughly 70 minutes and looked merely UNSTABLE.

I re-dispatched run 33702035573 as attempt 2 to produce exact-head evidence.

What this adds to the options already listed

The body's three options (path-scope the dev-up suite, raise the budget, split it into its own advisory Windows lane) all remain right, and option 3 now looks strongest, for a reason the body does not give: the coupling is the defect, independent of the timeout value. Raising the budget makes the hang rarer without removing the property that one advisory-quality Windows leg can void a required behavioural gate. Splitting the dev-up suite into its own lane removes the coupling outright.

Whichever option is taken, the E2E job's dependency on the rest of the run is worth stating explicitly in ci-required.yml so a cancelled sibling cannot silently convert a required gate into a skip. That is a CI-control R4 change and belongs with #2327 (CI-03, required gate and branch-current contract) rather than being done ad hoc.

Milestone

Setting this to v0.3. It is not a release blocker in itself, but it is now demonstrably throttling v0.3 lane throughput and it weakens the evidence contract the release gate depends on.

Not verified: whether the underlying ETIMEDOUT is runner-speed alone or a genuine deadlock in the PowerShell spawn path. Attempt 2's outcome is not in yet at the time of writing. I did not change any workflow file.

[Chris0Jeky]

Escalation: reproduced twice consecutively at the same head. This is now a hard block, not an intermittent flake.

Coordinator pass, 2026-09-03, roughly one hour after my earlier comment. Occurrence 4, and it changes the character of this issue.

The rerun failed identically

I re-dispatched run 33702035573 (PR #2397, head b40df4998616e2fc2530132da219130a2c424f49) as attempt 2 specifically to get exact-head evidence. It died the same way:

	Attempt 1	Attempt 2
Window	01:03:02Z to 01:28:02Z	02:20:21Z to 02:45:21Z
Duration	25:00	25:00
Step still in_progress at kill	4, Run source launcher regression suite	4, Run source launcher regression suite
Steps 5 to 12	pending, never started	pending, never started
E2E Smoke	skipped at 01:28:03Z	skipped at 02:45:21Z

Same step, same duration to the second, two attempts in a row on the same commit.

It is not the branch

Worth stating because "slow runner" invites the assumption that the PR is heavy:

• git diff --name-only origin/main...origin/issue-1975/paper-dialog-residuals returns 28 files, every one under frontend/taskdeck-web/src/. Nothing under scripts/, nothing under .github/. The branch cannot influence scripts/ci/dev-up.test.mjs or reusable-frontend-unit.yml.
• PR #2403 passed Frontend Unit (windows-latest) at fc21159b4 inside the same window, so the lane was not globally down.
• Neither cancellation was a concurrency supersede: run 33702035573 is the newest ci-required.yml run on that branch and b40df4998 is still the tip.

So a branch that touches no CI file, on a lane that was working for other PRs, cannot get through this step in two consecutive attempts.

Why this now blocks delivery rather than annoying it

PR #2397 has a confirmed product regression whose only guard is tests/e2e/edge-journeys.spec.ts:284. The 2026-09-03T00:12:26Z analysis on that PR establishes that jsdom cannot observe the defect class, because it implements no native button activation. Alpha pushed two commits at that blocker. Neither has been evaluated, because e2e-smoke declares needs: frontend-unit (ci-required.yml:223-231) and therefore reported skipped both times.

The result is a PR that cannot distinguish a working fix from a broken one, at 25 minutes per attempt. I have told Alpha to stop pushing at that blocker until this is resolved, and to work its non-E2E blocker instead.

Two consequences for the CI programme

1. The observation ledger. Every run that dies this way is a lost observation from the >=20-PR window that #2327, SC-4 and #2337 section A all depend on. This issue is now spending that budget.
2. The coupling is the defect, independent of the timeout value. Raising timeout-minutes makes the kill rarer without removing the property that one Windows leg of an advisory-quality suite can convert a required behavioural gate into a skipped. Of the three options in the body, splitting the dev-up suite into its own lane is the only one that removes it. Path-scoping it (option 1) also removes it for this class of PR, since a frontend-only diff would not select the dev-up suite at all, and the Smart CI planner already has path-to-lane policy.

.github/workflows/ci-required.yml is currently leased by the beta lane under the 2026-09-03T02:16:57Z #2333 claim, so whichever option is taken, the needs: change belongs with that lease holder rather than being done ad hoc. It is CI-control R4 work and pairs naturally with #2327.

Priority

Raising to Priority I. It is not a release blocker in itself, but it is now the binding constraint on v0.3 lane throughput and it is consuming the evidence budget the private-repository cutover depends on.

Not verified

I did not read the Windows runner's console output beyond step states, so whether the underlying cause is runner slowness or a genuine deadlock in the PowerShell spawn path is still open. Two identical 25:00 kills on one commit argue for a deadlock over slowness, but that is inference. I changed no workflow file.

[Chris0Jeky]

Sequencing note: ci-required.yml is open in PR #2412, and the two hunks do not collide

Checked because this issue's fix needs .github/workflows/ci-required.yml and the platform lane already has that file open.

• #2412 ("Enforce clean release workflow cache boundaries") adds one step to the release-workflow-contract job at roughly line 127: Validate release cache trust contract.
• The fix for this issue touches the e2e-smoke job's needs: list at roughly line 223.

Different hunks, about a hundred lines apart, so git merges them cleanly and neither has to wait for the other. Doing this issue's fix while #2412 is open is fine; just rebase or merge main before pushing if #2412 lands first, and do not reflow the file.

The #2333 lane claim of 2026-09-03T02:16:57Z names ci-required.yml among its owned paths, so the platform lane holds this file either way. That is the correct owner for this fix as well; nothing needs to move.

Also on the board now: this issue is Now on the project, and it is B1 in the current platform pool. It is the only item I put in Now (1 of a maximum 4), because the E2E gate it disables is the binding constraint on #2397 and on any future frontend PR that needs behavioural proof.

[Chris0Jeky]

Four more occurrences, a base rate, one narrowed root cause, and a third failure mode this issue does not name

Coordinator pass, 2026-09-03 09:00Z. I sampled the last 40 ci-required.yml runs (window 03:08Z to 07:53Z) and read the job logs rather than the conclusions. Three things came out of it: the ETIMEDOUT half is narrower than the body suggests, the base rate is lower than the 02:50:54Z escalation implied, and one failure in the window is a different defect that belongs somewhere else.

Base rate over the window

Frontend Unit / Frontend Unit (windows-latest) across 40 runs: 26 success, 3 failure, 9 cancelled, 2 running.

Most of the 9 cancellations are ordinary concurrency supersedes. Two are not, and they are this issue.

Occurrences 5 and 6: the 25:00 kill, confirmed twice more

Run	Job	Branch	Window	Duration
33711890758	100512970289	codex-2205-remove-apply-rate	03:35:09Z to 04:00:09Z	25:00
33710262492	100508010344	codex-2205-remove-apply-rate	03:08:18Z to 03:33:19Z	25:00

Both: step 4 Run source launcher regression suite still in_progress at the kill, steps 5 through 12 pending, never started. Identical to occurrences 2, 3 and 4.

Occurrences 7 and 8: the ETIMEDOUT half is a single test against a single hardcoded budget

Run	Job	PR	Result
33718954936	100534019890	#2416	84/85 pass, 1 fail, 416 s
33715230899	100522959972	#2416	84/85 pass, 1 fail, ~490 s

Both failed on the same single test:

test at scripts\ci\dev-up.test.mjs:1034:3
✖ PowerShell: pipeline cancellation runs transactional cleanup from finally (20077.7784ms / 20251.748ms)
  AssertionError [ERR_ASSERTION]: ifError got unwanted exception:
    spawnSync C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe ETIMEDOUT
      at runPowerShellCancellation (scripts/ci/dev-up.test.mjs:754:14)

This is the same test named in occurrence (1) in the body. That makes it three of three ETIMEDOUT occurrences on one test, not a suite-wide slowness symptom.

The budget it hits is hardcoded:

// scripts/ci/dev-up.test.mjs:774
timeout: 20_000,

All three occurrences report 20.0 s to 20.4 s, so the harness is being cut off exactly at its own budget. That is a budget too tight for a slow runner, not evidence of a hang. Note that this is a different budget from the two 20 s defaults at :718 and :893; only :774 is on the failing path.

The two halves pull against each other

Worth stating before anyone raises :774. The 25:00 kill is total suite wall time against reusable-frontend-unit.yml:19 timeout-minutes: 20. Raising the per-spawn budget makes the slow case slower, so it buys reliability on the ETIMEDOUT half at the cost of pushing the suite closer to the job kill. The path-selection option already named in the body (skip the dev-up suite when the diff touches neither scripts/dev-up*, scripts/ci/dev-up.test.mjs, nor the frontend) avoids that trade entirely, because it removes the suite from most PRs instead of re-budgeting it. On the evidence here that is still the better option, and occurrences 5 to 8 all landed on PRs that changed nothing under scripts/.

I have not proven that raising :774 alone would have turned occurrences 7 and 8 green, and raising a timeout can mask a real hang. That should be measured, not assumed.

A third failure mode, on main, that this issue does not cover

Run 33713409999, job 100517532356, branch main, push of 468d76dc8e7a7859324c40e6369c3caae57fe647 (the merge of PR #2414), 04:00Z. Frontend Unit (windows-latest) failed in 390 s with 83/85 passing:

test at scripts\ci\dev-up.test.mjs:1454:15
✖ mismatch (2913.8764ms)
  AssertionError [ERR_ASSERTION]: failed startup retained state despite complete cleanup
    at assertNoStateAndPortsReleased (scripts/ci/dev-up.test.mjs:889:10)

  expected: null
  actual:   {"schemaVersion":1,"runId":"96b5e572-...","apiPort":61113,"frontend":null,
             "processes":[{"role":"api","pid":1344,"name":"cmd","creationToken":"639240049701425703"}]}

This is not this issue. No timeout is involved: the test failed in 2.9 s, well inside its budget. The launcher was asked to clean up completely after a failed startup and left both its run-state file and a recorded API process behind. That is a fail-closed cleanup property, which is #1898's subject ("Close remaining dev-up fail-closed availability races"), not a slow-runner symptom.

Two things follow. First, main itself went red on that commit, which touches the exit gate's "main green" clause in docs/REVIVAL_PLAN.md:50. Later main pushes at 04:36, 05:15, 05:45, 06:25 and 06:56 were green on this leg, so main recovered without intervention and is not red now. Second, it means the dev-up suite currently has at least two independent defects behind it, and fixing the timeout half will not stop this one.

I saw it once and did not reproduce it. It should be recorded on #1898 with this evidence rather than folded into this issue, and it should not be called flaky until someone has looked at why cleanup left a live process record.

Not verified

I read job logs and the test source; I ran nothing locally and did not reproduce any occurrence. I did not measure whether the 9 cancellations beyond the two named are all ordinary supersedes; I checked those two specifically because their durations were exactly 25:00. I did not check whether :774's budget was ever sufficient on this runner class historically.