ci: parallelize image builds with tests; fix cargo-chef layer caching - #96
Merged
Merged
Conversation
The image matrix was gated on the `test` job (clippy + cargo test), so the two native-arch builds couldn't start until linting and tests finished. Since the test job shares no compiled artifacts with the image builds (host gnu target vs. musl targets), that serialization adds the whole clippy+test duration to the critical path for no reuse benefit. Drop `needs: test` from the image job so tests and the per-arch builds run concurrently. To keep release correctness, the `merge` job (which creates the published multi-arch tag) now also depends on `test`, so a tagged image is never published when linting or tests fail. On a tag push the per-arch jobs may push untagged digests before tests finish, but those stay dangling and unreferenced unless `merge` runs.
…mount The builder stage mounted /build/target (plus the cargo git/registry dirs) as BuildKit `type=cache` mounts. BuildKit cache mounts are NOT part of the image layer and are not exported by `cache-to: type=gha`; setup-buildx-action spins up a fresh builder each CI run, so those mounts start empty every time. cargo-chef relies on the `cook` step's compiled deps being captured so a later source-only build can reuse them. Because the deps were written into a cache mount, they vanished between runs, and on the common case (source changed, deps unchanged) the cook layer was a layer-cache hit and got skipped without repopulating the empty mount -- so the final `cargo build` recompiled every dependency from scratch on each arch, every run. cargo-chef was effectively a no-op in CI. Drop the cache mounts so the cook step's output (compiled deps + downloaded crates) lands in the layer filesystem, where Docker's layer cache captures it and `cache-to: type=gha,mode=max` (scoped per arch) persists it across runs. Now a source-only change restores the cook layer from cache and only rebuilds the workspace crates.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Two structural inefficiencies in the backend release pipeline make CI slower than it needs to be. This addresses both.
1. Image builds were serialized behind clippy + tests
The
imagematrix declaredneeds: test, so neither arch build could start until thetestjob (cargo clippy --release+cargo test --release) finished. But the test job compiles for the host gnu target while the image builds compile for musl (x86_64/aarch64-unknown-linux-musl) — three different target triples that share zero compiled artifacts. So the gate added the entire clippy+test duration to the critical path with no reuse benefit.Change: drop
needs: testfromimageso tests and the two native-arch builds run concurrently. Critical path goes fromtest + buildto roughlymax(test, build).Correctness is preserved by re-gating where it matters: the
mergejob — which creates the published multi-arch tag — nowneeds: [image, test], andreleasestill depends onmerge. So no tagged image is ever published when linting or tests fail. On a tag push the per-arch jobs may push untagged digests before tests finish, but those stay dangling and unreferenced unlessmergeruns.2. cargo-chef was a no-op in CI (deps recompiled from scratch every run)
The
Dockerfilebuilder stage mounted/build/target(and the cargo git/registry dirs) as BuildKittype=cachemounts. But:cache-to: type=gha.setup-buildx-actioncreates a freshdocker-containerbuilder each run, so those mounts start empty every time.cookstep's compiled deps being captured for reuse. On the common case (source changed, deps unchanged), the cook layer was a cache hit and got skipped — without repopulating the empty mount — so the finalcargo buildrecompiled every dependency from scratch, on each arch, every run.Change: remove the cache mounts so the
cookstep's output lands in the layer filesystem, where Docker's layer cache captures it andcache-to: type=gha,mode=max(already scoped per arch) persists it across runs. Now a source-only change restores the cook layer from cache and rebuilds only the workspace crates — which is the whole point of cargo-chef.Verification / caveats
CACHEDinstead of recompiling dependencies.target/dir, so the per-arch GHA caches grow. With two arch scopes this could approach the 10 GB repo cache budget; worth watching. Worst case (eviction) is no worse than today's effectively-cold builds.Each concern is a separate commit if you'd prefer to take them independently.
Generated by Claude Code