Skip to content

ci: parallelize image builds with tests; fix cargo-chef layer caching - #96

Merged
popen2 merged 2 commits into
mainfrom
claude/ci-backend-build-optimization-TIAa6
May 25, 2026
Merged

popen2 merged 2 commits into
mainfrom
claude/ci-backend-build-optimization-TIAa6

Conversation

@popen2

@popen2 popen2 commented May 25, 2026

Copy link
Copy Markdown
Member

Why

Two structural inefficiencies in the backend release pipeline make CI slower than it needs to be. This addresses both.

1. Image builds were serialized behind clippy + tests

The image matrix declared needs: test, so neither arch build could start until the test job (cargo clippy --release + cargo test --release) finished. But the test job compiles for the host gnu target while the image builds compile for musl (x86_64/aarch64-unknown-linux-musl) — three different target triples that share zero compiled artifacts. So the gate added the entire clippy+test duration to the critical path with no reuse benefit.

Change: drop needs: test from image so tests and the two native-arch builds run concurrently. Critical path goes from test + build to roughly max(test, build).

Correctness is preserved by re-gating where it matters: the merge job — which creates the published multi-arch tag — now needs: [image, test], and release still depends on merge. So no tagged image is ever published when linting or tests fail. On a tag push the per-arch jobs may push untagged digests before tests finish, but those stay dangling and unreferenced unless merge runs.

2. cargo-chef was a no-op in CI (deps recompiled from scratch every run)

The Dockerfile builder stage mounted /build/target (and the cargo git/registry dirs) as BuildKit type=cache mounts. But:

  • BuildKit cache mounts are not part of the image layer and are not exported by cache-to: type=gha.
  • setup-buildx-action creates a fresh docker-container builder each run, so those mounts start empty every time.
  • cargo-chef depends on the cook step's compiled deps being captured for reuse. On the common case (source changed, deps unchanged), the cook layer was a cache hit and got skipped — without repopulating the empty mount — so the final cargo build recompiled every dependency from scratch, on each arch, every run.

Change: remove the cache mounts so the cook step's output lands in the layer filesystem, where Docker's layer cache captures it and cache-to: type=gha,mode=max (already scoped per arch) persists it across runs. Now a source-only change restores the cook layer from cache and rebuilds only the workspace crates — which is the whole point of cargo-chef.

Verification / caveats

  • I can't measure Actions run timings or read build logs from where I'm working, so the timing claims are structural, not measured. The cache fix is verifiable from a build log: on a docs-only change, the cook step should now be CACHED instead of recompiling dependencies.
  • The cook layer now carries the dependency target/ dir, so the per-arch GHA caches grow. With two arch scopes this could approach the 10 GB repo cache budget; worth watching. Worst case (eviction) is no worse than today's effectively-cold builds.

Each concern is a separate commit if you'd prefer to take them independently.


Generated by Claude Code

popen2 added 2 commits May 25, 2026 19:14
The image matrix was gated on the `test` job (clippy + cargo test), so
the two native-arch builds couldn't start until linting and tests
finished. Since the test job shares no compiled artifacts with the image
builds (host gnu target vs. musl targets), that serialization adds the
whole clippy+test duration to the critical path for no reuse benefit.

Drop `needs: test` from the image job so tests and the per-arch builds
run concurrently. To keep release correctness, the `merge` job (which
creates the published multi-arch tag) now also depends on `test`, so a
tagged image is never published when linting or tests fail. On a tag
push the per-arch jobs may push untagged digests before tests finish,
but those stay dangling and unreferenced unless `merge` runs.
…mount

The builder stage mounted /build/target (plus the cargo git/registry dirs)
as BuildKit `type=cache` mounts. BuildKit cache mounts are NOT part of the
image layer and are not exported by `cache-to: type=gha`; setup-buildx-action
spins up a fresh builder each CI run, so those mounts start empty every time.

cargo-chef relies on the `cook` step's compiled deps being captured so a
later source-only build can reuse them. Because the deps were written into a
cache mount, they vanished between runs, and on the common case (source
changed, deps unchanged) the cook layer was a layer-cache hit and got skipped
without repopulating the empty mount -- so the final `cargo build` recompiled
every dependency from scratch on each arch, every run. cargo-chef was
effectively a no-op in CI.

Drop the cache mounts so the cook step's output (compiled deps + downloaded
crates) lands in the layer filesystem, where Docker's layer cache captures it
and `cache-to: type=gha,mode=max` (scoped per arch) persists it across runs.
Now a source-only change restores the cook layer from cache and only rebuilds
the workspace crates.
@popen2 popen2 self-assigned this May 25, 2026
@popen2
popen2 merged commit 42c8f26 into main May 25, 2026
7 checks passed
@popen2
popen2 deleted the claude/ci-backend-build-optimization-TIAa6 branch May 25, 2026 17:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant