[Pedro Henrique Penna (ppenna)]

Closes #702.
Part of #430.

Summary

A dev push can't restore the caches that its pull request's runs saved, so it rebuilt the guest images that the pull request had already built. Now each guest artifact job of a push installs its components from the merged pull request's latest run of ci.yml when one of their files misses the cache: artifacts the kernel and the Alpine initramfs, and distro-artifacts the Ubuntu and Azure Linux images. It does so only after these checks pass:

• the run came from microsoft/nvx, not a fork;
• the job's record in that run gives the push's own cache keys for the job's components;
• the artifacts' GitHub digests, the files' digests, and the files' provenance check out.

In every other case the job builds as before, and the reuse never fails the run. Consumers still download the push's own same-run artifacts. Each same-repository pull request rehearses the same selection and checks, for both jobs, against the artifacts that its own run uploaded.

This head is rebased onto dev after #711, #712, #710, #709, #734, #707, and #708; see Adaptations.

Why

Measured over the 87 dev pushes from October 10 to 21 whose guest artifact job succeeded, when one job built every guest image (raw data and run links: #702 comment):

	Cold (built at least one image)	Warm
Pushes	50 (57%)	37
artifacts job, median / p90 / max	5.4 / 9.1 / 11.3 min	0.3 / 0.4 / 0.6 min
Guest artifacts ready after run start, median (first attempts)	6.1 min	0.7 min
Run wall time, median (successful first attempts, n = 45 / 35)	34.1 min	27.8 min

• A cold guest artifact job delays everything that waits for it: the OpenVMM VMM tests by the whole build, and the WHP microVM job, beyond its OpenVMM producer and Windows CLI job, by a median of 1.8 min, p90 6.2 min. That happened in 40 of 45 cold first attempts.

• The merged PR's latest run had already produced the same images, under the same keys, less than a day earlier, in 24 of the 50 cold pushes. For example, 39592496133 (nvx-rs: build the NVXS unit tests' epoch identities with one function #678), 39529338356 (nvx-rs: run the control session on the shared NVXS session's host role #641), 39506407533 (nvx-rs: add a neutral guest spawn API and launch managed workloads through it #629), and 39481993344 (nvx-rs: add the sandbox store, launch journal, and runtime identity #628).

• In the other 26:

  • 24 had different keys, because another merge that changed the guest inputs, almost always nvx-rs/, landed between the PR's last run and its merge. ci: key the guest images and VMM tests on the guest binaries' sources #710 has since narrowed the image keys to the guest binaries' sources, so host-only nvx-rs merges no longer change them.
  • 1 PR run was more than a day old.
  • 1 push had no PR run.

  These still build.

Changes

Four commits, each with its own tests and docs:

1. release: validate the guest artifacts of any directory. GUEST_ARTIFACT_COMPONENTS maps each component that a package ships, named as the action's components input names it, to its files. validate_guest_artifacts(directory, components) applies to those components the checks that package already makes to build/: the kernel and Alpine initramfs provenance, and the Ubuntu and Azure Linux manifests against the files and this checkout's sources. The Azure Linux check now takes its initramfs path, and the Ubuntu manifest check moves into a function of its own. package and collect-sources behave as before.
2. ci: record the cache keys and digests of the guest artifacts. When its new record-artifact input names an artifact, build-guest-artifacts ends by running nvx.py record-guest-artifacts, which writes build/guest-artifacts.json, and uploads it under that name for one day.
   • The record holds the exact keys of the selected components' cache entries, taken by name from the key step's outputs, and the SHA-256 of each of their files.
   • artifacts uploads kernel-alpine-record and distro-artifacts uploads ubuntu-azurelinux-record. Both names stay outside ci: build the guest-artifact components in parallel and download only what each job uses #711's guest-* handoff names, because no job of the run downloads a record.
   • The debug kernel and the Ubuntu scratch template, which every run formats anew, are in no record.
3. ci: reuse a pull request's guest images on its dev push. On a push only (reuse: ${{ github.event_name == 'push' }}), after the cache restores and before any build, each of the two jobs runs nvx.py reuse-guest-artifacts when a file of its components is missing. The command does the following:
   • It takes the merged head from the push's merge commit, its second parent, and finds the latest pull_request run of ci.yml for that head whose head_repository is microsoft/nvx. Runs from forks and deleted forks never qualify.
   • It downloads the job's record from that run and requires its keys to equal the push's keys for the job's components exactly. A change to the other job's keys doesn't matter, and a mismatch stops it before it downloads any image.
   • It then downloads the components' artifacts: guest-kernel and guest-alpine, or guest-ubuntu, guest-ubuntu-distro, and guest-azurelinux. It refuses any record or artifact that has expired or lacks a sha256: digest from GitHub, and checks each download against that digest. Every download drops the token on the redirect to storage.
   • It installs the files only when every file matches its SHA-256 in the record and validate_guest_artifacts passes for the job's components. A response that a transport error cuts short, such as with http.client.IncompleteRead, counts as a failed request.
   • It then replaces every file of the job's components, not only the missing ones, because a cache restore that fails partway can leave a truncated file and still report a miss. The installation notes each move before its rename, and when it fails, it puts back the files that it replaced.
   • When the job reuses its components, their builds and the Ubuntu determinism check skip, as on a cache hit; the run that built the images passed that check. The saves still cache the reused files, and the jobs still upload their own artifacts and record.
   • Any other outcome leaves the components to the builds. A two-minute limit stops a stalled selection, download, or check, but never an installation, whose rollback it could cut short. An interval timer enforces it on Linux. It was four minutes, but after ci: build the guest-artifact components in parallel and download only what each job uses #711 a cold Alpine-only artifacts job takes about 1.3 minutes. The step continues on error, prints a notice for an expected reason or a warning for a fault, and writes a job-summary line whose heading names the components.
   • artifacts and distro-artifacts get actions: read.
4. ci: rehearse guest artifact reuse on pull requests. For same-repository pull requests, the new guest-artifact-reuse job (Rehearse guest artifact reuse) runs after artifacts and distro-artifacts succeed.
   • It has one reuse-guest-artifacts --dry-run --head-sha <PR head> step per job, with that job's components, record, and keys, which reach it through the action's new cache-keys output and each job's guest-cache-keys output. Each step selects the latest run for the head, normally its own, then downloads and checks that job's record and artifacts exactly as a push would, and installs nothing.
   • The job follows ci: classify microVM tests, benchmarks, and artifact builds separately #712's classification pattern under run-guest-artifacts, and check-required-ci expects it to succeed for same-repository pull requests that build the guest artifacts. A pull request that breaks the reuse command therefore can't merge.
   • A refusal in the rehearsal warns but doesn't fail, because a transient API error must not block a pull request.

doc/ci.md gains a "Guest artifact reuse" section after "Guest artifact handoff", and a job-graph paragraph. doc/usage.md documents both commands.

Adaptations to dev

• ci: build the guest-artifact components in parallel and download only what each job uses #711 (per-component guest artifacts):
  • The reuse works per guest artifact job: one record, record name, reuse input, and actions: read for each, validation limited to that job's components, and reuse from the PR run's per-component artifacts.
  • COMPONENT_ARTIFACTS maps each component to the artifacts that hand it over, and a test checks it against ci.yml's uploads.
  • ci: build the guest-artifact components in parallel and download only what each job uses #711's step-to-component test now lists the three steps that serve every selected component: the reuse, record, and record upload steps.
  • test_ci_full_runs_do_not_publish_persist_or_save_caches counts 16 push conditions: ci: build the guest-artifact components in parallel and download only what each job uses #711's 14, plus the two reuse arguments. The cache saves stay limited to pushes and pull requests.
• ci: classify microVM tests, benchmarks, and artifact builds separately #712 (classifier outputs):
  • The rehearsal job is in the job classification, in REQUIRED_CI_RESULT_ENVIRONMENTS, and in the required-results policy.
  • The reuse module and both commands are inputs of the microVM tests and the benchmarks under the classifier's every-CLI-module rule, so a change to them selects the guest artifacts and the rehearsal; a test checks this. They aren't inputs of the OpenVMM VMM tests, because on a pull request the reuse doesn't run and the record doesn't change the images.
• ci: key the guest images and VMM tests on the guest binaries' sources #710 (narrower keys): nothing to adapt, because the record takes whatever keys the key step produces for each component.
• ci: queue the dev push runs instead of cancelling the waiting one #734 (queue: max for pushes): each push still runs for its own merge commit, so the selection holds, and no queued push is cancelled before it can reuse. A push that waits more than a day finds the PR's artifacts expired and builds.
• ci: report nightly full-run failures #709 (nightly report section): both sections are kept; the reuse section now follows "Guest artifact handoff".
• microvm tests: record each scenario's duration in a JSON Lines report #707 (scenario duration reports): no interaction; the rebase changed only the context of an import.
• ci: build the Linux OpenVMM VMM tests once for the KVM and MSHV jobs #708 (Linux VMM tests built once): no conflict and no change; its VMM test jobs still wait for artifacts, whose files the reuse checks before it installs them.

Tests

• Unit tests, GuestArtifactReuseTests, with a fake GitHub API that serves each job's record and component artifacts built from the release fixture's valid images, cover the following:
  • the record: the keys and digests of each job's components, and refusal of a missing file, a missing key, an unknown component, or invalid keys;
  • selection: the merged head, the latest same-repository run, and the exclusion of forks, deleted forks, other events, workflows, and heads;
  • artifact selection: expired, other-name, and other-run artifacts, artifacts without a sha256: digest, which it never downloads, and a missing record or component artifact;
  • key mismatches, each of which stops reuse before any image download: a changed, missing, or extra key. A change to another job's component's key doesn't stop reuse;
  • digest and content checks:
    • the GitHub digest of the record and of each artifact;
    • a tampered or missing file, or a file that two artifacts hold;
    • a record without its file, without a file of the components or with another's, or without a valid digest;
    • an unknown or boolean format, a malformed record, and a non-ZIP record or artifact;
  • provenance mismatches against the checkout, for each job;
  • installation, which replaces every file of the components, leaves other files alone, and puts back every file when any rename fails or an interruption follows a successful one;
  • the time limit: an expired deadline, and, on POSIX, an interval timer that interrupts stalled work, stays disarmed during the installation, and ignores a signal after its block. An expiry as the checks end leaves no staging directory behind;
  • API failures on each kind of request, invalid responses, and, on a local server, responses cut short within their Content-Length or a chunk;
  • "not needed" when the cache restored every file of the job's components;
  • the rehearsal of each job, which checks without installing and warns for any refusal;
  • the CLI outputs, reused, the summary, and the annotations, whose headings name the components, including no token, invalid keys, a missing key, and an unknown component;
  • a local HTTP server test that the token reaches the API but not the redirected download.
• A ReleaseTests case covers validate_guest_artifacts for each component:
  • a valid directory, each component in a directory of its own files, and each kind of stale file, which fails its own component and no other;
  • no component, an unknown one, and a missing provenance file.
• Workflow-structure tests check the following:
  • the restore steps of each recorded component use exactly the keys and files that its record names;
  • the action records and uploads last, with a one-day retention;
  • each guest artifact job records under a name of its own, outside guest-*, and the debug kernel job records nothing;
  • the permissions: only the two jobs, the rehearsal, and release get actions: read;
  • only pushes set reuse;
  • the reuse step continues on error and runs after every restore of a recorded component and before every build of one;
  • all six guest builds require reused != 'true', while the saves and the debug kernel ignore it;
  • the artifacts that reuse downloads are those that hand over each component's files;
  • the rehearsal job's needs, condition, permissions, steps, and key plumbing, and that only the required status check waits for it;
  • a change to the reuse, its record, or their commands selects the rehearsal.

Validation

• Windows, every commit: ruff check, ruff format --check, pyright for Linux and Windows, check_markdown_links.py, git diff --check, and the affected test classes.
  • Commits 1 and 2 also ran the full test_nvx_tools and test_markdown_links modules in a git archive export. There, only the two tests that need a Git repository fail.
  • Commits 3 and 4 ran the full modules in the worktree.
• Windows, head eac22eae4, on bd977e35b:
  • The full test_nvx_tools and test_markdown_links modules pass: 619 tests. After the rebase onto microvm tests: record each scenario's duration in a JSON Lines report #707, they pass again: 620 tests. The rebase onto a data/ commit changed no code, and with the fix for the review's truncated-response finding, 621 tests pass. After the rebase onto ci: build the Linux OpenVMM VMM tests once for the KVM and MSHV jobs #708, on 206abd182, 629 tests pass.
  • So do the other validate-nvx steps: compileall, test_adversarial, test_hosts, and nvx.py --help for every listed command and the two new ones.
• WSL Ubuntu, Python 3.14: GuestArtifactReuseTests and CiConfigurationTests pass, including the four POSIX interval-timer tests that Windows skips, and so does the truncated-response test.
• actionlint v1.7.7, the official release binary with a verified checksum: ci.yml has only the known false positive about ci: queue the dev push runs instead of cancelling the waiting one #734's queue key, and the other workflows only the existing unknown-label warning in adversarial.yml.
• Rehearsal on this PR: on the final head, ffd6b59fe, it reported "would reuse" for both jobs in a 16-s job of run 39785367867. On eac22eae4, it selected its own run, 39774840679, and reported "would reuse" for both jobs in a 23-s job: kernel-alpine-record with guest-kernel and guest-alpine, and ubuntu-azurelinux-record with guest-ubuntu, guest-ubuntu-distro, and guest-azurelinux. Before ci: build the guest-artifact components in parallel and download only what each job uses #711, earlier heads reported "would reuse" for the single guest-artifacts artifact: 39626393958 and 39635577644.
• CI: run 39774840679 on eac22eae4 passed: 29 jobs succeeded, including the rehearsal and Required status check, and the 4 push-only jobs were skipped. The rebase onto microvm tests: record each scenario's duration in a JSON Lines report #707 needed no change. Run 39785367867 on the final head, ffd6b59fe, passed too: 30 jobs succeeded, including the rehearsal and Required status check, and the 4 push-only jobs were skipped. The runs for b6a13d63b and 9058b7dea were cancelled when the review fix and the rebase onto ci: build the Linux OpenVMM VMM tests once for the KVM and MSHV jobs #708 superseded them. No run needed a rerun.
• Not possible before merge: the push path end to end; see below.

Expected effect

• The push reuses wherever its PR's run built the same components less than a day earlier: 24 of the 50 cold pushes in the sample, before ci: key the guest images and VMM tests on the guest binaries' sources #710 removed the most common reason for other keys. That lowers the projected cold rate from 57% to about 30% or less, and saves those pushes the guest builds on the path of the VMM tests and the microVM jobs.
• Comparing keys per job makes the reuse even more likely than in that projection, which compared every key at once: a merge that changed only the other job's components no longer prevents it.
• Costs:
  • A job that reuses downloads its components' artifacts and checks them. The rehearsal of both jobs took about 23 s on GitHub-hosted runners, including checkout.
  • A job whose keys differ downloads only its small record before it builds.
  • Each same-repository PR runs one more GitHub-hosted job, which only Required status check waits for.
• There's no change to consumers, the release, the performance gate, or persistence, nor for forks, drafts, path filters, cancellation, or reruns.

Post-merge verification

The reuse runs only on dev pushes:

• Merging this PR likely shows "not needed" in both jobs, because it changes no guest key input. The first push that merges a guest-input change whose PR ran less than a day earlier, with no guest-input merge in between, should show "Guest artifact reuse (…): reused" in the summary of Build kernel and Alpine artifacts or Build Ubuntu and Azure Linux artifacts, and a Build guest artifacts step of seconds, not minutes.
• Over the next five comparable dev pushes, record on ci: reuse a pull request's guest images on its dev push #702, for each guest artifact job, the cold-build rate, the step and job durations, the time when the guest artifacts are ready, and the run wall time, and compare them with the baseline above.
• Confirm that every push that reused components names a same-repository run with the push's own keys, and that every fallback names its reason.

[Copilot]

🟡 Changes recommended

Archive integrity checks can be bypassed, and reuse does not reliably enforce its time limit or handle partial cache restores.

3 open issues

• Enforce a wall-clock deadline across the entire reuse operation · New
• Reject missing or unsupported artifact digests · New
• Replace all files for cache-missed images after partial extraction · New

What changed in this PR

Reduces repeated guest-image builds on dev pushes by reusing verified artifacts from same-repository pull requests.

Changes:

• Records cache keys and file digests, then validates artifacts before reuse.
• Adds push-time reuse and a pull-request dry run.
• Adds validation tests and documentation.

File	Description
scripts/​test_nvx_tools.py	Tests reuse, validation, and workflow wiring.
scripts/​nvx.py	Adds recording and reuse commands.
scripts/​nvx_tools/​release.py	Enables validation of arbitrary image directories.
scripts/​nvx_tools/​guest_artifact_reuse.py	Implements artifact selection, verification, and installation.
doc/​usage.md	Documents both commands.
doc/​ci.md	Explains reuse and fallback behavior.
.github/​workflows/​ci.yml	Wires artifact records and rehearsal job.
.github/​actions/​build-guest-artifacts/​action.yml	Attempts reuse before building missing images.

_{Review effort: Balanced}

[Copilot]

🟡 Changes recommended

An installation-time timeout can delete a cache-restored image that the fallback build skips.

1 open issue

• SIGALRM can omit renamed file from rollback and lose cache backup · New

3 resolved since last review

• Reject missing or unsupported artifact digests
• Enforce a wall-clock deadline across the entire reuse operation
• Replace all files for cache-missed images after partial extraction

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

Cross-run artifact trust and installation recovery warrant human review, and end-to-end push verification remains pending.

No open issues 🎉

1 resolved since last review

• SIGALRM can omit renamed file from rollback and lose cache backup

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

Cross-run artifact trust and installation rollback warrant human review, and the actual dev push reuse path remains unverified end to end.

No open issues 🎉

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

Cross-run artifact trust and rollback need human review, the post-merge path remains unverified, and a confirmed transport-error handling gap remains unresolved.

1 open issue

• Handle truncated GitHub API responses without blocking PRs · New

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

Cross-run artifact trust and installation recovery warrant human review, and the actual dev-push reuse path remains unverified end to end.

No open issues 🎉

1 resolved since last review

• Handle truncated GitHub API responses without blocking PRs

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

Cross-run artifact trust and transactional installation warrant human review, and the production push path still needs end-to-end verification.

No open issues 🎉

_{Review effort: Balanced}