Fine-tune CI caching: nix phase is ~32 of 35 min in the fixture gate; budget 88% full

[vicondoa]

Summary

CI caching now works but is not tuned. Measured on PR #343, the repository cache sits at 8.78 GB across 19 entries against GitHub's ~10 GB LRU budget, roughly 88% full, and a meaningful share of that is dead weight. Separately, the test-fixture-contracts nix cache is demonstrably working but still leaves 27 derivations to build, and the most likely cause is a size cap that is pruning the store before save.

This issue records the measurements so the tuning can be done against numbers rather than intuition.

What already works

Scoping the nix store cache per job (PR #343) fixed a real defect: all twelve nix jobs previously shared one key, and because actions/cache never overwrites, whichever trivial job finished first froze a 90 MB entry that every other job then "hit". Measured before and after on test-fixture-contracts:

signal	shared key	job-scoped
cache entry size	90 MB	2217 MB
restore step duration	8s	51s
derivations to build	166, then 411	27

last_accessed_at on the 2217 MB entry matches the restore step timestamp exactly, confirming the job consumes it.

Worth stating for anyone reading a build log: cache-nix-action restores the /nix store directory, it is not a substituter. Nix therefore never prints copying path ... from ... for cached content - it simply finds the paths present and drops them from the build plan. The observable evidence is the action's own restore, last_accessed_at, and the derivation count. Absence of substituter lines is not absence of caching.

Current cache inventory

2217MB  nix-Linux-test-fixture-contracts-e28042def...
2094MB  v0-rust-test-rust-Linux-Linux-x64-21820c02-b94d5870
1835MB  v0-rust-test-rust-Linux-Linux-x64-4ece3d53-7027ccd5
1818MB  v0-rust-test-rust-Linux-Linux-x64-4ece3d53-0a5bbf36
 130MB  nix-Linux-test-nix-unit-e28042def...
 126MB  nix-Linux-test-rust-e28042def...              <- stale
 110MB  nix-Linux-flake-eval-x86-e28042def...         <- stale
  95MB  nix-Linux-flake-eval-x86-outputs-e28042def... <- stale
  90MB  nix-Linux-test-drift-e28042def...             <- stale
  90MB  nix-Linux-flake-eval-discover-e28042def...    <- stale
  90MB  nix-Linux-e28042def...                        <- stale (original shared key)
  89MB  nix-Linux-test-flake-aarch64-e28042def...     <- stale

total: 8.78 GB across 19 entries

Findings to tune

1. gc-max-store-size-linux: 4G is probably truncating the fixture store

test-fixture-contracts still builds 27 derivations on a warm cache. The obvious explanation - that a source change invalidated them - was tested and falsified: git diff <seed-commit>..HEAD -- packages/ is empty, so the fixture derivation is byte-identical between the seeding run and the consuming run, and the cache should have covered it entirely.

The remaining hypothesis is the size cap. gc-max-store-size-linux prunes the store before save, so anything beyond the ceiling is dropped and must rebuild. The entry is 2217 MB compressed, consistent with an uncompressed store at or above the 4G cap.

The cap was lowered from 8G to 4G during review specifically to protect the aggregate budget, so raising it is a genuine tradeoff, not a straightforward fix.

To confirm before changing anything: read the 27 derivation names from a completed test-fixture-contracts log. If they are Rust host-tool derivations, the cap is truncating. If they are per-run inputs, the cap is correct and 27 is the floor.

2. Stale per-job nix entries occupy ~730 MB

Seven nix-Linux-* entries exist for jobs that are no longer in NIX_CACHED_JOBS (test-rust, flake-eval-x86, flake-eval-x86-outputs, test-drift, flake-eval-discover, test-flake-aarch64), plus the original shared nix-Linux-<hash> key. They will never be refreshed and will not be read. They occupy budget until LRU reclaims them.

purge-prefixes only purges within a job's own scope, so a job that no longer renders a cache step never purges its own leftovers.

Options: delete them once via gh api --method DELETE, or add a scheduled workflow that reaps nix-* entries whose prefix is not in the current NIX_CACHED_JOBS.

3. rust-cache is the dominant consumer at ~5.7 GB

Three generations at ~1.9 GB each, keyed on Cargo.lock hash. This is the single largest claim on the budget and was not examined during the nix cache work.

Worth asking whether three live generations is the right number, or whether the key should be narrowed so old lockfile generations retire faster.

4. MATRIX_CHECK_SCOPE has no consumer

flake-eval-x86 shards share one ciJobId, so per-job scoping alone would let them clobber each other. A scope_suffix was added and threaded through, but that job is not in NIX_CACHED_JOBS, so the code path is currently dead. It was verified by hand (temporarily adding the job, regenerating, confirming the key renders nix-${{ runner.os }}-flake-eval-x86-${{ matrix.check }}-...) but nothing pins it.

If those shards are ever added to the cached set, that path needs a test. Note the shards each complete in under a minute today, so there is currently no case for caching them.

5. sccache is deliberately absent from CI

Recorded here so it is not re-attempted without knowing the outcome. Enabling sccache in CI breaks the build: its server forwards only a whitelist of environment variables to rustc, and CARGO_BIN_EXE_<name> - which Cargo injects so an integration test can locate the binary it exercises - is not among them. The failure is:

error: environment variable `CARGO_BIN_EXE_d2b-exec-runner` not defined at compile time
   --> d2b-exec-runner/tests/tty_pty_integration.rs:359

It does not reproduce locally because CARGO_INCREMENTAL is on there, so sccache marks those units non-cacheable and passes them through untouched. CI sets CARGO_INCREMENTAL=0, sccache genuinely caches, and the variable goes missing.

sccache remains enabled for local development through packages/.cargo/config.toml and the .cargo/rustc-wrapper.sh shim, where it was verified to cache properly (miss, then 100% hit on an identical rebuild).

If CI sccache is ever wanted, the lever is excluding the CARGO_BIN_EXE_*-dependent test targets from caching - a real tradeoff needing its own decision.

Suggested order

1. Read the 27 derivation names from a completed fixture log and decide whether the 4G cap is the cause.
2. Reap the seven stale nix-Linux-* entries and decide whether reaping should be automated.
3. Re-measure total usage; only then adjust NIX_CACHE_MAX_STORE.
4. Separately review the rust-cache generation count, which is the largest consumer.

Context

Caching was introduced and scoped in PR #343. The .gitignore/scratch-path cleanup is tracked separately in #346.

[vicondoa]

Resolved: finding 1 — the 4G cap is NOT the cause

The 27 derivations have been read from a completed run (job 90860539686, conclusion success). They are not Rust host-tool derivations, so the size cap is not truncating compiled artifacts and should not be raised:

nixos-system-corp-vm-26.05pre-git.drv    closure-info.drv
d2b-corp-vm-closure.json.drv             d2b-sys-work-net-closure.json.drv
d2b-fixture-smoke.drv                    d2b-bundle.json.drv
d2b-processes.json.drv                   d2b-bundle-artifact-inputs.json.drv
initrd-linux-6.18.33.drv                 linux-6.18.33-modules-shrunk.drv
initrd-units.drv                         system-units.drv  user-units.drv
etc.drv  activate.drv  dry-activate.drv  boot.json.drv
unit-sshd.service.drv                    unit-dbus-broker.service.drv
unit-systemd-networkd.service.drv        unit-d2b-load-host-keys.service.drv
sshd.conf-final.drv                      check-sshd-config.drv  dbus-1.drv
X-Restart-Triggers-sshd.drv              X-Restart-Triggers-dbus-broker.drv

Every one is NixOS system assembly — unit files, etc, activation scripts, initrd, the VM toplevel — plus closureInfo and the fixture's own JSON. These are the top of the dependency graph: cheap individually, and they sit above everything expensive, which the cache is now supplying.

So 27 is at or near the floor for this design, and the cache is behaving correctly. Finding 1 should be closed without action. Raising NIX_CACHE_MAX_STORE would consume more of an already 88%-full budget for no benefit.

Revised priority

Timings from the same run:

Nix store cache (restore)              51s
Fixture-dependent Rust contract tests  35m29s
Post Nix store cache (save)             1s

With nix caching working, essentially the whole cost is now the cargo half of that step — cargo nextest run -p d2b-contract-tests, the cargo build -p d2bd harness binary, and the CLI-contract run — not the nix build. That reframes the remaining work: further nix cache tuning has little left to win here, and anyone chasing the fixture gate's duration should measure the cargo phase first.

Findings 2 (≈730 MB of stale entries), 3 (rust-cache at ~5.7 GB, the dominant consumer) and 4 (MATRIX_CHECK_SCOPE untested) stand as written. Given the budget is 88% full and the largest single claim is rust-cache rather than nix, finding 3 is now the highest-value item, followed by finding 2.

[vicondoa]

Correction: the nix phase dominates, not cargo

My previous comment was wrong on both counts and I am retracting it. I read only the first these 27 derivations will be built line and concluded the nix phase was near its floor. The log contains a second wave of 116 derivations, and the split is the opposite of what I reported.

Measured inside the single Fixture-dependent Rust contract tests step (job 90860539686, total 35m29s):

11:44:50  these 27 derivations will be built
11:44:56  building nixos-system-corp-vm
11:45:26  these 116 derivations will be built     <- I missed this entirely
11:46:33  building nixos-system-sys-obs-net
11:55:59  building nixos-system-nixos             <- 9 min after the previous
12:00:42  building nixos-system-corp-full
12:16:41  building d2b-fixture-smoke-full         <- 16 min after the previous
12:16:41  --> cargo nextest run -p d2b-contract-tests
12:19:21  step ends

phase	duration
nix builds	~32 min
cargo (nextest + doctests + CLI contract)	~2.7 min

So roughly 90% of the step is nix, and the cache is not close to its floor. "27 is the floor" and "the remaining cost is cargo" were both wrong.

What is actually being rebuilt

Four separate NixOS system closures, plus repeated closure-info invocations:

• nixos-system-corp-vm (fixture-smoke)
• nixos-system-sys-obs-net
• nixos-system-nixos
• nixos-system-corp-full (fixture-smoke-full)

The two largest gaps are nixos-system-nixos at 11:55:59 and d2b-fixture-smoke-full at 12:16:41, the latter roughly 16 minutes after the closure before it. fixture-smoke-full is on its own close to half the step.

This also reframes the earlier finding that pkgs.closureInfo (nixos-modules/closures-json.nix:24) forces full guest-NixOS realisation: it is not one closure but four, and both fixture variants pay it.

Revised priorities

1. Why are these four system closures not in the cache? They were built during the seeding run and should have been restored. Either gc-max-store-size-linux: 4G is pruning them after all - contrary to what I concluded from the truncated first list - or they are not reachable from the GC roots the action preserves. This needs checking before any other tuning, and it reopens the question I incorrectly closed.
2. fixture-smoke-full deserves separate scrutiny. It adds graphics/video/audio/tpm/usbip and is x86-only. If its marginal contract coverage over fixture-smoke is narrow, running it less often is worth more than any cache change.
3. Findings 2 (stale entries) and 3 (rust-cache at ~5.7 GB) still stand, but rank below the above now that the nix phase is known to be the dominant cost.

Apologies for the noise from the earlier comment - it was a measurement error on my part, from reading a partial log rather than the whole step.

[vicondoa]

resolved in v3 branch