Skip to content

ci: compile the unit tests before running them - #469

Merged
foxnne merged 1 commit into
mainfrom
ci/compile-tests-first
Oct 17, 2026
Merged

foxnne merged 1 commit into
mainfrom
ci/compile-tests-first

Conversation

@foxnne

@foxnne foxnne commented Oct 16, 2026 •

Copy link
Copy Markdown
Collaborator

What changes

Tests (Windows) failed about one run in thirty (4 of the last 140 jobs) with test runner failed to respond for 1m… on whichever tiny std-only test was unlucky: fizzy-glass-look-tests, backend-titlebar-tests, fizzy-profile-causes-tests, fizzy-text-doc-state-tests, fizzy-profile-spans-tests, fizzy-settings-plugins-zon-tests. CI now compiles every unit test (zig build check) in a step of its own before zig build test runs them, so no compiler is alive while a test runs.

The cause is in Zig 0.16's std, not in fizzy. processSpawnWindows (std/Io/Threaded.zig) creates each child's pipe ends inheritable and calls CreateProcess with handle inheritance on, with no PROC_THREAD_ATTRIBUTE_HANDLE_LIST and no lock around it. So a child inherits every inheritable handle the build runner holds at that moment, including the pipe ends it is making for its other children. A compiler spawned while a test's pipes exist keeps a copy of that test's stdout write end. After the last test, the runner sends exit and waits for stdout EOF under its 60 s response timeout (std/Build/Step/Run.zig, waitZigTest/requestNextTest). The test has passed and exited, but EOF comes only when the compiler exits, so the step fails with "failed to respond" and reports no test as failed. Every failing run had a test compile that took ≥1 min (fizzy-crash-tests, fizzy-singleton-unix-tests, fizzy-plugin-store-tests). After check, zig build test finds every compile cached and spawns only test binaries, which live for milliseconds.

The transport tests were a red herring. fizzy-native-transport-tests passed (21/21) in every failing run. In the PR #460 failure it ran a minute after the hung tests had started. Its error.Unexpected NTSTATUS=0xc000020d (CONNECTION_RESET) trace comes from the cancel test: its holder closed the accepted socket with the request unread, Windows answers that with a reset, and Zig 0.16's netReadWindows maps every status it doesn't list (this one included, which should be error.ConnectionResetByPeer) to error.Unexpected, dumping a stack trace on stderr. The holder now reads the request first, so the close is a clean FIN and the trace is gone.

How it was found (throwaway workflow on ci/windows-flake-diag, since deleted):

  • 40 test-only zig build test loops on 4 Windows runners: 0 hangs. 96 loops recompiling every test binary each time (from PowerShell, then again exactly as CI runs it, through Git Bash and ci-retry.sh's tee): 0 hangs. Those recompiles take ~4 s each and never produce the minute-long compilers.
  • A standalone repro with no fizzy code: 8 empty tests, each run beside a 75 s ping. On windows-latest, instant-2-tests failed with test runner failed to respond for 1m5.202ms. Without the sleepers it passed, and on ubuntu-latest it passed both ways (posix pipes are O_CLOEXEC).
  • Defender real-time protection is off on the runners, so scanning is ruled out.

Tried and dropped: --test-timeout (it raises the response timeout too, but only hides the wait), -j1 (slow), and retries (CONTRIBUTING: fix the cause).

SDK impact

  • None

Verified

  • macOS: zig build test -Dtest-filter=cancel passes with the transport change.
  • Windows: run 38968162178 plus 5 reruns of Tests (Windows), all green: 563/568 passed (5 skipped) each time. Compile tests did the compiling, and in Run tests every compile was cached (the step takes ~12 s), so no compiler ran beside a test. No NTSTATUS/failed command lines in any of the six logs. At the old ~3% rate, six green runs alone prove little; the evidence is the repro plus that invariant.
  • Linux: Tests (Linux) and the rest of CI green.
  • Web: not affected.

Follow-ups

None. The cause is in Zig 0.16's std; this is fizzy's workaround, and the pitfall in docs/PITFALLS.md keeps the two steps apart.

🤖 Generated with Claude Code

On Windows, Zig 0.16 spawns each child inheriting every inheritable handle the build runner
holds, including the pipe ends it is making for its other children. A compiler spawned while a
test's pipes exist keeps a copy of that test's stdout. The test passes and exits, its stdout
reaches EOF only when the compiler exits, and the runner reports "test runner failed to respond"
a minute later, on whichever tiny test was unlucky (4 of the last 140 Windows jobs). `zig build
check` now compiles every test first, so `zig build test` spawns only test binaries.

The transport cancel test reads the request before it closes the held connection: closing with
bytes unread resets it, and Zig 0.16 reads a reset on Windows as error.Unexpected with a stack
trace on stderr, which looked like the culprit in every failing log.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@foxnne
foxnne marked this pull request as ready for review October 16, 2026 21:50

@foxnne foxnne left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed. zig build check exists (build/app.zig:557, "Compile fizzy unit tests without running them"), so the new CI step compiles exactly what test needs, and the run step then spawns only test binaries. The NativeTransport test reading the request before holding fixes the reset-as-Unexpected noise at its cause, and the pitfall is written where the next person will look. All checks pass. Looks good to merge.

🤖 Generated with Claude Code

@foxnne
foxnne merged commit 7b538aa into main Oct 17, 2026
72 checks passed
@foxnne
foxnne deleted the ci/compile-tests-first branch October 17, 2026 14:48
@foxnne foxnne mentioned this pull request Oct 17, 2026
2 of 3 tasks
foxnne added a commit that referenced this pull request Oct 17, 2026
Part of fizzyedit/plans#5 (`plans/ZIG_0_17_PLAN.md`, step 2): catching
`zig/0.17` up with `main`. It was 40 commits behind, up to #469.

## Please merge this one by fast-forward, not squash

This PR is a merge commit with `main` as its second parent. Squashing it
would drop that parent. The next catch-up would then find the old merge
base, and every line `rewrite.sh` changed in `main`'s code would
conflict again.

Once CI is green, the clean way is a fast-forward of `zig/0.17` to this
commit: `jj bookmark set zig/0.17 -r <this commit>` then `jj git push
--bookmark zig/0.17`. That's a push to the integration branch, not to
`main`. Your call: squash also works, at the cost of re-resolving these
lines next time.

## What changed beyond the merge

- **Conflicts (11 files).** Nearly all were `main`'s `builtin.target`
rename (#435) against the same rename already on `zig/0.17`; those take
`zig/0.17`'s side. These files take `main`'s side, then `rewrite.sh`:
  - `scripts/check-examples.sh`: `main`'s fuller bsdtar fix.
  - `plugins/text/src/Document.zig`: the selection hash.
- `sdk/tape/binary.zig`: the header now written early, and the versioned
field read.
  - `sdk/src/version.zig`: re-recorded (below).
- **`scripts/zig-0.17/rewrite.sh` re-run** over the code `main` brought
in.
- **By hand:**
- `app/state_delta.zig`: a test's multi-character string repeat (`"…" **
100`, gone in 0.17) is now a comptime loop.
- `sdk/tape/binary.zig`: the tape reader's new indexed field loop, in
0.17's `field_names` shape.
- `recorded_sdk_shape_fingerprint` is `0x0b0e62b7eac39f6d`: `main`'s SDK
changes since the switch, on 0.17. It is released with the merge to
`main`, as before.

## SDK impact

- [x] Fingerprint moved: recorded in `sdk/src/version.zig`,
`sdk_version` untouched

## Verified

- [x] macOS (arm64, Zig 0.17.0):
  - `scripts/check-fmt.sh`
  - `zig build`
  - `zig build test`: 95/95 steps, 232/232 tests
  - `zig build check-tests-reached`: 120/120
  - `zig build check-web`: 41/41
  - `zig build test-sdk-version`: 35/35
  - `zig build test-integration`: 77/77 steps, 559/559 tests
- [ ] Windows, Linux, the example repos: CI. `check-examples.sh` wasn't
run locally because the machine is under load from other sessions.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@foxnne foxnne mentioned this pull request Oct 18, 2026
2 of 3 tasks
foxnne added a commit that referenced this pull request Oct 18, 2026
#486 was squash-merged, so zig/0.17 lost main as an ancestor and the next merge would have gone back to #434 as its base. This commit takes #486's original merge commit (46ddcd8) as a second parent and keeps zig/0.17's tree as it is, so main at #469 is the base again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant