Skip to content

Run the Rust CI legs in parallel instead of in a line - #46

Merged
teddytennant merged 5 commits into
mainfrom
ci-parallel-rust
Sep 3, 2026
Merged

teddytennant merged 5 commits into
mainfrom
ci-parallel-rust

Conversation

@teddytennant

@teddytennant teddytennant commented Sep 2, 2026

Copy link
Copy Markdown
Owner

A green run took 12m48s and 763s of it was one job. rust ran clippy, the suite, a release build, then clippy and the suite again under --features native, one after another, while everything else in the file had been finished for ten minutes. The macOS leg had the same shape: 105s of cargo check then 237s of cargo test, in a line, on the slowest runner here.

Measured on this branch: 768s to 371s wall clock, and both of the runs here were cold-cached against a warm baseline.

The split

rust became rust-lint, rust-test, rust-test-native, rust-release and a rust gate. suite keeps the two blocking cargo check steps on Darwin and suite-test takes the non-blocking suite.

The release build stops paying for LTO. [profile.release] is lto = true, codegen-units = 1, so a step whose only question is "does this compile with optimisations on" spent 334s linking 729 crates as one unit and then ran nothing. rust-cache drops the workspace crate's artifacts by design, so that link was paid in full on every run, cache hit or not. Overridden by environment, not by editing the profile, since release.yml builds ten assets from it unchanged. The tradeoff is the break only fat LTO finds, mostly duplicate symbols at link time; release.yml catches those before a tag publishes.

save-if on the build jobs the split touched. Six jobs across two platforms and two feature sets writing per-branch copies of the same 729 dependencies is how a 10GB cache budget becomes evictions. Pull requests read a cache and no longer write one, which also drops the ~25s save that would otherwise have been most of what the split cost.

Branch protection matches required checks by name, so "Rust checks" is now a gate job that needs the four legs and goes red when any of them does. No settings change. That adds two failure modes that look green: a leg nobody adds to the gate's needs, and a gate without if: always(), which GitHub skips rather than fails, leaving a PR waiting on a check that will never report. Both are asserted in Workflow invariants, along with the macOS suite still running cargo test somewhere now that it has left the job whose name is protected. Each of the five assertions was checked by mutating a copy of the file and confirming it fails.

Promoting the macOS suite to blocking is two edits now, not one: experimental: false on suite-test, and "macOS suite" added to the required checks. The note above the job says so and the summary step prints it. The experimental key left stranded in suite is removed, since a flag you can flip with no effect is worse than no flag.

The tests

Three tests were waiting on clocks nothing reads.

a_program_wedged_in_a_c_call_does_not_wedge_the_turn and cancelling_reaches_a_program_wedged_in_a_c_call both ran os.execute('sleep 30'). The turn returns in about 3s, which is what they assert, but the program is left to finish in its own time by design, so the test outlives it either way. They were the last two tests to finish in the entire run, which made them the floor.

Shortening the sleep alone would have gutted both, so the bound came down with it: at sleep 10 against the old 15s bound, a turn that waited for the program passes as happily as one the backstop saved. 6s sits between the backstop's grace period and the sleep. Checked by pushing BACKSTOP_GRACE to 8s, where both now fail at 8.3s and 9.0s and the old bound passed them.

a_sandboxed_script_is_bounded_in_time_and_memory is the third. Only the time half of that name is enforced. install_hook reads lua.used_memory(), mlua cannot install its allocator under luajit, and a 0 reading is skipped rather than tripped, so the memory branch cannot fire and the greedy script runs until the clock stops it. The || in the assertion is what hides it. Raising the budget to 60s made the test take 61s, where a working 64 MB cap would end a 1 MB-per-iteration script in under a second. Budget down to 3s with the finding written beside it. Making that bound real is a change to the sandbox and does not belong in this PR.

Numbers

Test execution, holding compile constant:

before after
Linux 49.30s 31.53s
Coverage job 49.90s 32.32s
macOS 94.02s 58.41s

The coverage job restored a full cache-key match on both runs, so that row is the clean one.

Job totals move around more than that: cold-build variance between the two runs here is about 15% either way, and Rust release build came in slower on the second run despite running no tests at all. The execution numbers are the ones that isolate the change.

@teddytennant

Copy link
Copy Markdown
Owner Author

Rebased onto e363a62 and added a rustfmt commit.

Rust lint went red on src/skills/mod.rs:188, which is not from this branch. e363a62 merged with cargo fmt --check already failing on that hunk, so main has been red since, and a PR is tested as base-merged-with-head. Second commit here is cargo fmt output on that one function, nothing else.

Worth a look separately: a required check went red on #45 and it merged anyway.

@teddytennant
teddytennant force-pushed the ci-parallel-rust branch 4 times, most recently from 9198233 to 3f0d684 Compare September 3, 2026 13:12
A green run took 12m48s, and 763s of that was one job. `rust` ran clippy,
the suite, a release build, then clippy and the suite again under
`--features native`, sequentially, while every other job in the file had
been done for ten minutes. Nothing in that sequence needed the step
before it. The macOS leg had the same shape: 105s of `cargo check` then
237s of `cargo test`, in a line, on the slowest runner here.

Split into rust-lint, rust-test, rust-test-native, rust-release, and
suite / suite-test on Darwin.

Two things that were not just reordering:

The release build was paying for a ship build to answer a compile
question. `[profile.release]` is `lto = true, codegen-units = 1`, so the
step spent 334s linking 729 crates as one unit and then ran nothing.
rust-cache drops the workspace crate's artifacts by design, so that link
was paid in full every run. Overridden by environment rather than by
editing the profile, since release.yml builds ten assets from it
unchanged. What that gives up is the break only fat LTO finds, mostly
duplicate symbols at link time, and release.yml still catches those
before a tag publishes.

`save-if` on the build jobs the split touched: six jobs across two
platforms and two feature sets writing per-branch copies of the same 729
dependencies is how the 10GB cache budget turns into evictions. Pull
requests now read a cache and don't write one, which also drops the ~25s
save that would otherwise have been the split's main cost.

Branch protection matches required checks by name, so "Rust checks" is
now a gate job that needs the four legs and is red when any is. That
keeps the split out of the repository settings, and adds two failure
modes that look green: a leg nobody adds to the gate's `needs`, and a
gate without `if: always()`, which is skipped rather than failed when a
leg breaks. Both are asserted in `Workflow invariants`, along with the
macOS suite still existing now that it no longer lives in the job whose
name is protected. Promoting that suite to blocking is two edits now,
not one, and the note above the job says so.

Expect ~5m instead of ~13m.
rustfmt output, no behaviour change. e363a62 landed with cargo fmt --check
red on this hunk, so main has been failing that gate since it merged and
every branch cut from it inherits the failure.
The suite's floor was two tests doing nothing for 30 seconds each.

`a_program_wedged_in_a_c_call_does_not_wedge_the_turn` and
`cancelling_reaches_a_program_wedged_in_a_c_call` both run
`os.execute('sleep 30')`. The turn returns in about 3s, which is what
they assert, but the program is left to finish in its own time by design,
so the test outlives it either way. They were the last two tests to
finish in the whole run.

Shortening the sleep on its own would have gutted both. The sleep and the
bound are a pair: at `sleep 10` against the old 15s bound, a turn that
waited for the program passes just as happily as one the backstop saved.
So the bound comes down to 6s with it, which sits between the backstop's
grace period and the sleep. Checked by pushing BACKSTOP_GRACE to 8s: both
now fail, at 8.3s and 9.0s, where the old bound passed them.

`a_sandboxed_script_is_bounded_in_time_and_memory` is the third, and its
20s was buying nothing. Only the time half of that name is enforced.
install_hook reads lua.used_memory(), mlua cannot install its allocator
under luajit, and a 0 reading is skipped rather than tripped, so the
memory branch cannot fire and the greedy script runs until the clock
stops it. The `||` in the assertion is what hides this. Raising the
budget to 60s made the test take 61s; a working 64 MB cap would have
ended it in under a second on a script allocating 1 MB an iteration.
Budget down to 3s, with the finding written next to it rather than fixed
here, since making that bound real is a change to the sandbox.

Suite goes from 37.7s to 21.3s locally on 16 cores.
Every file directly under tests/ is its own target, so cargo was linking
five test binaries against the whole crate. The five of them run 38 tests
in about 1.5 seconds put together, so what that bought was the linking,
paid five times over on 223k lines and 729 dependencies.

They are modules of tests/integration now. The three `#![cfg(...)]` gates
moved onto their `mod` lines, since a module cannot gate itself out of
its parent, and they mean what they meant before.

mesh_quic needed a real fix, not just a move. It re-executes the test
binary with `--exact child_node_entry_point` to run a mesh node in a
separate process, and as a module that test is `mesh_quic::child_node_
entry_point`, so the hardcoded name matched nothing: the child ran no
test, printed nothing, and the parent failed on a missing socket address
with no hint that the name was the problem. It derives the name from
module_path!() now, so moving the file again costs nothing.

41 tests pass in one binary in 0.49s.
Corrects the previous commit, which claimed the memory bound could not
fire. It fires. The bound was never broken; the test was.

`string.rep('x', 1000000)` builds the same megabyte string on every
iteration, and Lua interns strings, so all of them were one object and
the table grew by a pointer an entry. The loop peaked at 2.3 MB against a
64 MB bound, ran until the clock stopped it, and the `||` in the
assertion accepted that as a pass. `tostring(#t)` varies the string, the
loop allocates for real, and the sandbox refuses it with "exceeded its
memory budget (64 MB)". The assertion asks for memory now, so a time
refusal here fails instead of passing quietly.

The reading was fine all along: mlua cannot install its allocator under
luajit, but `used_memory` falls through to LuaJIT's own LUA_GCCOUNT
rather than returning the 0 the old note warned about. I read both side
by side and they agree to the byte. That note is corrected too.

Case goes from 21s to about 7, and now it tests what it is named for.

macOS suite moves to pushes on main. At 365s against a 129s `macOS
check` it was the slowest job in CI and the only one whose result blocks
nothing, so on a pull request it set the wall clock long after the
verdict anybody waits on had landed. The compile gate is untouched and
still blocking on every pull request; this is the suite alone. Promoting
it to blocking now means deleting the `if` first, or every pull request
waits on a check that never reports.
@teddytennant
teddytennant merged commit 9bc2831 into main Sep 3, 2026
18 checks passed
@teddytennant
teddytennant deleted the ci-parallel-rust branch September 3, 2026 13:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant