build: add a single-binary test target - #798
Open
rustytrees wants to merge 1 commit into
Open
Conversation
`zig build test` compiles one executable per file under `tests/` — 162 of them — and runs 162 fresh processes. That is slow to link, and on a host with a per-binary firewall or endpoint agent it produces a prompt storm per run, because every rebuild is a new executable the agent has not seen. `zig build test-one` links and runs exactly one binary over the same files: 2591 passed, 3 skipped, 0 failed in ~2m20s. `zig build test` is untouched and stays authoritative — one process per file is what actually isolates process-global state, and CI should keep using it. Sharing a process immediately surfaced one latent order dependency, which is fixed here rather than papered over: `custom_theme_test`'s "valid themes file" case asserts on captured CLI output, but `emitPrefixLine` early-returns while the module-global quiet flag is set. Run in its own binary the flag happens to be clear; run after a file that leaves `setQuiet(true)` behind, the capture is empty and the assertion compares against nothing. `boot()` already pins the colour state for exactly this reason — quiet was the one input it missed. That is worth noting on its own: the per-file layout hides this class of bug, and `just test-concurrent` only covers shared *fixture paths*, not shared in-process globals. This target is a cheap way to keep the second axis honest.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Additive build target.
zig build testis untouched.CONTRIBUTING asks for an issue first on anything non-trivial — this is a judgement call about build ergonomics rather than a bug, so I've put the proposal in the PR where the diff is visible. Happy to close it and move the discussion to an issue if you'd rather not review code for something you haven't agreed to in principle.
What and why
zig build testcompiles one executable per file undertests/— 162 of them — and runs 162 fresh processes. Two costs:zig build test-onelinks and runs exactly one binary over the same files: 2591 passed, 3 skipped, 0 failed in ~2m20s.zig build teststays authoritative and unchanged. One process per file is what actually isolates process-global state, and CI should keep using it —test-oneis a local convenience and a second signal, not a replacement.tests/all.zigsays so in its header.It found something
Sharing a process surfaced one latent order dependency, fixed here rather than worked around:
custom_theme_test's "valid themes file" case asserts on captured CLI output, butemitPrefixLineearly-returns while the module-global quiet flag is set. In its own binary the flag happens to be clear; run after a file that leavessetQuiet(true)behind, the capture is empty and the assertion compares against nothing.boot()already pins the colour state for precisely this reason — quiet was the one input it missed.Worth saying plainly: the per-file layout hides this class of bug, and
just test-concurrentcovers shared fixture paths, not shared in-process globals. So this target also guards an axis nothing currently checks. I would expect it to find more of these if it runs regularly.The honest downside
162 files in one process means shared globals, and I only know of the one above. If a future test leaks a global,
test-onewill fail whilezig build teststays green — confusing unless you know why. Two ways to read that: as a maintenance cost, or as the target doing its job. I lean the second, but it is your call, and it is the reason I did not touch the default.tests/all.zigduplicates the file list frombuild.zig'stest_modules, so a new test file needs adding in two places. I considered generating it at build time, but a generated root in the cache cannot resolve@import("foo_test.zig")relative totests/. If drift worries you, ano_*_test-style guard comparing the two lists would fit the existing patterns intests/— say the word and I'll add it.Verification
zig build test-one— 2591 pass, 3 skip, 0 fail, one binaryzig build test— still green, unchangedHope it helps decide the knowledge base.