Repository navigation
ci: shorten the runs that restore the VMM tests or skip both OpenVMM suites #982
Description
Follows: #430, which closed with these two targets unmet. Related: #649, #697, and #717.
Summary
#430 cut the median full-matrix run from 31.0 to 24.0 minutes, and to 19.3 without queueing, but missed two of its targets: runs that restore the OpenVMM VMM tests from the cache should take 12 to 15 minutes, and runs that skip both OpenVMM suites at most 15. The OpenVMM suites no longer bound either kind of run. The WHP platform job, the GitHub-hosted Linux OpenVMM unit tests, and the KVM general runners do.
Today
Successful first-attempt ci.yml runs from 2026-10-23 18:52 UTC, when #810 merged, to 2026-10-28 23:32 UTC, timed from the creation of each run's first job to the end of its last job. "Without queueing" means that no self-hosted job waited 2 minutes or more. See the results on #430 for the method and the full-matrix figures.
| Runs | Count | Median | Without queueing | Target |
|---|---|---|---|---|
| VMM tests restored from the cache | 89 | 18.8m | 16.5m (18 runs) | 12-15m |
| Both OpenVMM suites skipped, with benchmarks | 68 | 17.7m | 15.5m (30 runs) | 15m |
| Both OpenVMM suites skipped, without benchmarks | 18 | 12.9m | 12.3m (8 runs) | 15m |
- WHP platform path.
Platform / Windows / WHP / Virtual machinewaits for the wholeNVX CLI / Windowsjob, a median 5.8 minutes, of which buildingnvx-rstakes 0.8 and validating the CLI 4.0. It then waits for WHP's onlybenchmarkrunner and measures for 8.9 minutes, so it ends about 15 minutes into a run before any queueing. It finished last in 67 of the 68 skip-both runs that ran the benchmarks and in 61 of the 89 cache-hit runs. - Hosted Linux unit tests. Since ci: run the OpenVMM unit tests off the hypervisor runners #697,
OpenVMM unit tests / Linuxruns onubuntu-latest, where it takes a median 16.6 minutes. It finished last in 13 of the 26 cache-hit runs that ran the unit tests. - KVM
generalrunners. Each run's longest wait for a KVMgeneralrunner rose from a median 0.1 and a p90 4.1 minutes, from October 10 to 21, to 3.3 and 11.1. With the VMM-test build and leg, two microVM shards, the guest and sandbox job, and the debug-kernel job, a run can need more than the pool's fourgeneralrunners, asdoc/ci.mdnotes.
Tasks
- Evaluate starting each operating system's microVM and platform jobs once its CLI job has handed off
nvx-rs, instead of after the whole CLI validation. They wait today so that a broken CLI does not hold hypervisor runners; measure the gain on WHP first. - Weigh the hosted Linux unit tests again against an MSHV runner, which ran them in 6.5 minutes with sccache (ci: run the OpenVMM unit tests off the hypervisor runners #697), now that they bound cache-hit runs.
- Measure KVM
generalqueueing by job, and either add ageneralrunner or reduce the KVM jobs that a run needs at once. - Measure both targets again over at least five comparable runs each, with queue time.
Acceptance criteria
- Runs that restore the VMM tests have a median of at most 15 minutes without queueing, or the remaining bound is documented.
- Runs that skip both OpenVMM suites have a median of at most 15 minutes, with and without benchmarks, over at least five comparable runs each.
- No required job, gate, or fail-closed behavior is lost, and each performance series is still measured on its one
benchmarkrunner. doc/ci.mddescribes any change to the job graph or the runner pools.
Activity
ci: start the microVM and platform jobs once nvx-rs is built
ci: package the releases beside the platform benchmarks
ci: run the pull requests' debug kernel on WHP
doc: weigh the hosted Linux unit tests against an MSHV runner
ci: start the microVM and platform jobs once nvx-rs is built
ci: package the releases beside the platform benchmarks
ci: run the pull requests' debug kernel on WHP
doc: weigh the hosted Linux unit tests against an MSHV runner
ci: start the microVM and platform jobs once nvx-rs is built
ci: package the releases beside the platform benchmarks
ci: run the pull requests' debug kernel on WHP
doc: weigh the hosted Linux unit tests against an MSHV runner
ci: start the microVM and platform jobs once nvx-rs is built
5 remaining items
ci: run the pull requests' debug kernel on WHP
doc: weigh the hosted Linux unit tests against an MSHV runner
ci: start the microVM and platform jobs once nvx-rs is built
ci: package the releases beside the platform benchmarks
ci: run the pull requests' debug kernel on WHP
doc: weigh the hosted Linux unit tests against an MSHV runner
ci: start the microVM and platform jobs once nvx-rs is built
ci: package the releases beside the platform benchmarks
ci: run the pull requests' debug kernel on MSHV
doc: weigh the hosted Linux unit tests against an MSHV runner
scripts: count the Ubuntu layer among the runtime adapter's reads
ci: run WHP's runtime adapter scenarios in a job of their own
ppenna commentedon Nov 6, 2026
Results after #1241
#1028 did the first three tasks: no hypervisor job waits for the CLI tests any more, the releases are packaged beside the platform jobs, the pull requests' debug kernel runs on MSHV, and
doc/ci.mdrecords why the hosted Linux unit tests stayed hosted. #1074 then stopped CI running OpenVMM's tests, and #1241, which merged on 2026-11-03 at 20:03 UTC, runs WHP's runtime adapter scenarios in a job of their own. This is the fourth task, on the merged graph.Method. #430's, as in #1241: successful first-attempt
ci.ymlruns of pull requests, merge groups, anddevpushes created from 2026-11-03 20:03 UTC to 2026-11-06 00:18 UTC, timed from the creation of each run's first job to the end of its last job. "Without queueing" means that no self-hosted job waited 2 minutes or more. The 12 successful full runs are left out, as in #1241, and so are the 64 of the 157 such runs that ran only GitHub-hosted jobs. All 90 runs that ran the microVM tests ranNVX microVM tests / Windows / WHP / Runtime adapter. CI has run no OpenVMM suite since #1074, so, as in #1241, the runs that restore OpenVMM from its cache stand for those that skipped both suites. A run built OpenVMM when theBuild OpenVMMstep of one of its OpenVMM jobs took 2.3 to 4.1 minutes on Linux or 8.9 to 13.4 on Windows; on a cache hit, it took at most 1.1.Platform / Windows / WHP / Virtual machine, a median 7.8 minutes, ended last in 33 of the 62 runs. Without queueing, the GitHub-hostednvx-rs / ubuntu-latest, a median 10.2 minutes, ended last in 13 of the 30, and the WHP runtime adapter and platform jobs in 6 each. The median was 12.4 minutes on November 3 and 11.9 on November 5, but 19.2 over the 38 runs of November 4, when 58 runs in all ran the benchmarks; 26 of those 38 queued, and the other 12 took 11.5.OpenVMM CI of the pinned revisionended last, since a pull request waits for OpenVMM's own CI of its pin (see OpenVMM CI). No target covers these runs.Queueing. Each run's longest wait for a general runner had a p90 of 8.0 minutes on KVM, 9.7 on MSHV, and 10.8 on WHP, and 32, 30, and 35 of the 93 runs waited 2 minutes or more, so KVM no longer queues the most, against 195 of 286 runs and a p90 of 11.1 in this issue's window. WHP's benchmark runner does: 44 of the 91 runs with benchmarks waited 2 minutes or more for it, at a p90 of 20.0 minutes, against 18 on KVM's and 17 on MSHV's.
Other criteria.
Required status checkneeds, andcheck-required-cichecks the result of, every job that ci: shorten runs that restore the VMM tests or skip both OpenVMM suites #1028 and ci: run WHP's runtime adapter scenarios in a job of their own #1241 added: bothnvx-rsbuilds, the three package jobs, andnvx-microvm-adapter-whp. Each performance series was measured on its one benchmark runner: the 91 platform jobs of each backend all ran onazure-kvm-3,azure-mshv-4, andazure-windows-5.doc/ci.mddescribes thenvx-rshandoff, the package jobs, the pull requests' debug kernel on MSHV, WHP's runtime adapter job, and the runner pools, with the measurements behind them.Both targets are met, the one without benchmarks on the estimate above. Closing as completed.