Repository navigation
ci: build the guest-artifact components in parallel and download only what each job uses #703
Description
Parent: #430, workstream 6. Related: #699 and #702.
Summary
The shared guest-artifact job restores or builds the guest components one after another, and every consumer downloads all of them. Once #699 and #702 have reduced cold builds, measure what remains, and build the components in parallel if cold builds still matter.
Today
build-guest-artifactsrestores or builds the kernel, Alpine, Ubuntu, Azure Linux, and Ubuntu distro layer sequentially in theartifactsjob. Warm runs take about 0.4 minutes. In CI: maximize workflow parallelism and reduce turnaround time #430's baseline, 44 of the 73 runs built cold, with a median of 5.3 minutes and a maximum of 11.3.- Every consumer downloads the whole
guest-artifactsartifact, although, for example, the OpenVMM VMM tests boot only the kernel and the Alpine initramfs. - The debug kernel builds separately, in the
debug-kerneljob.
Tasks
- Measure cache misses and build time by component over multiple runs, after ci: key the guest images and OpenVMM VMM tests on the guest binaries' sources, not all of nvx-rs #699 and ci: reuse a pull request's guest images on its dev push #702.
- If cold misses still materially affect p90 or p99 run time, split independent components into parallel producers.
- Make each consumer depend on and download only the components it uses.
- Keep the separate debug-kernel lane and the provenance and determinism checks.
Acceptance criteria
- Per-component measurements recorded here. If components are split, timing with queue time from at least five comparable runs.
- No consumer lacks an artifact that it uses, and the required gates remain fail closed.
doc/ci.mddescribes the producers and handoffs.
Metadata
Metadata
Assignees
Labels
No labels
Activity
ppenna commentedon Oct 21, 2026
Per-component measurements, estimates for #699 and #702, and the split decision
Measured on 2026-10-21 for #711. Data: the jobs API of the 449 CI runs created from 2026-10-13 to 2026-10-21 13:55 UTC, and the logs of the 202
artifactsanddebug-kerneljobs that ran longer than 45 s since 2026-10-15. The jobs API does not list a composite action's steps, so the per-component times come from the step groups in the job logs.The current guest build, with the Rust guest-binary Docker stage and the
-v2image keys, started with #561, merged 2026-10-19 10:52 UTC (#561's own pull-request runs, from 2026-10-16, use it too). The numbers below cover the 200artifactsjobs since then: 47devpushes and 153 pull-request runs.Cache misses and build time by component
build-guest-binaries23), which BuildKit runs beside the base stage; Alpine initramfs 4.5 s; Ubuntu initramfs 34 s; Ubuntu determinism 165 s; distro layer 15 s; Azure Linux initramfs 19 s; kernel 323 s.-v1keys): kernel 5.6 min, Alpine 0.13, Ubuntu 0.63, determinism 1.77, distro 0.30, Azure Linux 0.39. The Rust stage moved about 0.7 min into the Alpine build and 1.1 into the determinism check.guest-artifactsartifact, in p50 5 s on Linux and 7 s on Windows (p90 9–11 s). Per component, in ci: build the guest-artifact components in parallel and download only what each job uses #711's run 39624800851: kernel 7.5 MB, Alpine 8.2, Ubuntu initramfs 38.5, Ubuntu distro layer 43.0, Azure Linux 42.5, scratch template 0.07, debug kernel 7.5. Each download step costs about 1–3 s, so the VMM tests download theirs in 2–6 s, and the jobs that read every image take about 5–10 s more than before.Effect on the run
Of the 145 successful first-attempt runs of that period, 137 ran the OpenVMM VMM tests, 119 of them with cold guest artifacts. The WHP VMM tests started a median 6.0 min after the run started on cold runs, against 0.8 on warm ones, and finished last in 74 of the 119 cold runs; the WHP platform job, behind its benchmark runner, finished last in 33. Examples: cold push 39603556404 (artifacts done at 5.9 min, WHP VMM tests 5.9–26.8 min, run 27.0 min), warm push 39596666239 (0.7–21.9, run 22.0), cold pull request 39602304045 (5.7–28.1, run 28.2).
What #699 and #702 remove
I replayed the 172
artifactsjobs of that period whose commits are available: each run's keys are recomputed from the tree it tested (the pushed commit, or the pull request's head merged intodevat the time), and GitHub's cache scopes are simulated (devplus the run's own pull request; a job saves what it built when it finishes). Against the observed misses, the replay predicts 166 of 172 Alpine outcomes and 171 of 172 kernel outcomes.devpushes (of 47)¹ Approximated by dropping the host-only
nvx-rsfiles from the keys:src/host/**,src/testing/**,src/bin/nvx-rs.rs,src/bin/nvx-fake-openvmm.rs,tests/**,README.md,LICENSE, andartifacts.json. An exact set, such as Cargo's dependency information, may also drop#[cfg(test)]modules.#699 has since merged as #710, on 2026-10-21 at 19:27 UTC. Its keys also leave out the test modules (the
tests.rsfiles, thetestsdirectories, and the managed agent'sloopbackharness), so it should remove at least the misses estimated here; #702 is #713, still open.nvx-rschanges: 37 of 137 (27%).devpushes that rebuild what their pull request built: 10 of 36 cold pushes (28%). Most of the others merged after another merge changednvx-rs, so their pull request's last run had other keys. With ci: key the guest images and OpenVMM VMM tests on the guest binaries' sources, not all of nvx-rs #699, 9 of 27. ci: reuse a pull request's guest images on its dev push #702 should also save the reused images to thedevcache: otherwise later pull-request runs lose thedev-scope hits that the push's build gives them today.devsaved its new keys. Under ci: key the guest images and OpenVMM VMM tests on the guest binaries' sources, not all of nvx-rs #699, the first group still runs the VMM tests.Decision: split into two producers
To see whether the remaining misses matter for the tail, I replayed the 133 successful first-attempt runs of that period: every job keeps its measured queue time and duration, and starts when the jobs it waits for finish. The model reproduces the observed run times within 0.12 min (median).
artifactsand the Ubuntu and Azure Linux images in a newdistro-artifactsjob beside it, uploads one artifact per component, and makes each job download only what it reads. The VMM tests and the debug-kernel jobs wait only forartifacts. The debug-kernel lane, the determinism check, the provenance, and the keys are unchanged.Cold path after the split
At first, #711's runs hit the cache, because #710's
devpush had just built and saved every new key. So, on a temporary branch, I ran ci.yml's two producers as #711 defines them, once on GitHub-hosted runners:save-cache: false;.dockerignore, which every guest key hashes, so that every component missed.The experiment is run 39665813966. The baseline is the single job of #710's
devpush 39661815501, which built every component cold under the same keys an hour earlier:artifactsjobartifactsdistro-artifacts¹ Including the base and guest-binary Docker stages, which
distro-artifactsbuilds again: 0.6 min more.artifactstakes about 1.0 min, as in ci: build the guest-artifact components in parallel and download only what each job uses #711's run below. So the VMM tests and the debug-kernel jobs start about 4 min earlier, while the microVM test and platform jobs wait fordistro-artifactsabout as long as for the single job.#711's own CI, every image cold. #652 changed the guest binaries' sources, and #711, rebased onto it, ran before any
devpush had saved the new keys. In its run 39690060426 of the head46da18e49, every guest image missed the cache, while the kernel and the debug kernel hit:artifactsdistro-artifacts² Including the base and guest-binary Docker stages, as in the experiment.
Both producers started 0.4 min after the run was created.
distro-artifacts(5.7 min). With the single job, they would have waited for every image: a median 5.3 min when it missed, above.distro-artifactsfinished.After #711 merges, the timing with queue time of at least five comparable runs with cold guest images, before and after, belongs here; #711 lists it under its post-merge verification.
Raw per-run data: the 200
artifactsjobs (build seconds per component; empty when the cache hit)