ci: run the OpenVMM unit tests off the hypervisor runners #697

Activity

1. Pedro Henrique Penna (ppenna)

   self-assigned this

   on Oct 21, 2026
2. Pedro Henrique Penna (ppenna)

   added the

   Feature

   issue type on Oct 21, 2026
3. Pedro Henrique Penna (ppenna)

   added a parent issue on Oct 21, 2026

   • CI: maximize workflow parallelism and reduce turnaround time #430
4. ppenna commented on Oct 21, 2026

   

   Pedro Henrique Penna (ppenna)

   on Oct 21, 2026

   Last edited by ppenna

   ContributorAuthor

   More actions

   Decision

   Move the Linux leg to GitHub-hosted ubuntu-latest, and keep the Windows leg on WHP. PR #717 implements it and updates doc/ci.md. The Linux leg builds without a compilation cache: it takes 12–19 minutes instead of 7, but it finished before the rest of the run in every run measured, so I accept the slowdown rather than spend the repository's full Actions cache on it. No unit test or doctest is lost, and the job, its name, and its fail-closed result in Required status check are unchanged.

   1. Hypervisor runner-minutes per week

   Method: jobs API data for every CI run from #436 (Oct 8, 13:28 UTC) to #660 (Oct 21, 12:04 UTC), 12.9 days, counting every attempt, including cancelled ones, from start to completion.

   Event	Linux leg (MSHV)	Windows leg (WHP)
   Pull requests	109 jobs, 765 min a week	109 jobs, 1,704 min a week
   dev pushes, before #660	54 jobs, 388 min a week	54 jobs, 928 min a week

   #660 changes only dev pushes; pull requests, which #436 already classified, take two thirds of the time. Replaying the current classifier over the window's 145 dev pushes, 46 of the 112 that ran tests would still run the suite (41%); over the 42 merges through #659, 6 of 40 (15%), as this issue says. With 7 nightly full runs (17.3 and 7.2 minutes each):

   Since #660	Linux leg (MSHV)	Windows leg (WHP)
   Estimated minutes a week	870–975	1,960–2,210
   Share of the pool's busy time (replay of Oct 12–21 under #660's rules)	15%	19%
   Measured, Oct 21 12:04–15:50 UTC	15 jobs, 105 min	13 jobs, 214 min

   The first hours after #660 run well above the estimate, because the #430 sub-issue PRs all edit ci.yml, an input of the suite; at 15:50, three Linux unit-test jobs were queued for MSHV runners.

   2. Duration on GitHub-hosted runners

   Method: push-triggered experiment workflows on temporary agents/openvmm-unit-tests-runners-exp* branches, which ci.yml's dev-only push trigger doesn't match. They used only GitHub-hosted runners, restored the OpenVMM inputs that dev pushes cache, saved no cache, and are deleted. Run 39615224333 ran nvx.py test-openvmm-unit twice per OS, once from scratch through an empty local sccache and once more with a fresh target directory and the sccache that the first pass filled, which bounds what any compilation cache could save. Run 39624535618 ran #717's Linux leg, copied verbatim, on three more machines.

   test-openvmm-unit	Linux	Windows
   Self-hosted, 8 vCPUs, local sccache: p50 since Oct 8; hits in 39585909588	6.5 min, 99.6% hits	14.7 min, 77.8% hits
   GitHub-hosted, 4 vCPUs, 16 GB, from scratch	12.1 (EPYC 9V45), 16.2 (Xeon 8370C), 16.4 (EPYC 9V74), 18.1 min (Xeon 8573C, empty sccache)	31.8 min (EPYC 7763)
   GitHub-hosted, warm local sccache	7.6 min	20.7 min
   Tests, all passed	5,294 run, 34 skipped, as on MSHV	4,933 run, 26 skipped, as on WHP
   Doctests in 391 crates, all passed	82 passed, 39 ignored, as on MSHV	82 passed, 33 ignored, as on WHP
   Target directory; sccache	26 GiB; 1.5 GiB	23 GiB; 1.65 GiB, 2.3 GiB after the second build
   Free disk	86 GB on / (145 GB)	147 GB on D:

   The hosted Windows image already disables Defender's real-time monitoring and excludes C:\ and D:\, so that isn't the gap. In #717's own CI run, 39628480713, the Linux leg ran on an EPYC 7763 in 17.3 minutes (a 17.8-minute job) and passed the same 5,294 tests and 82 doctests.

   3. Turnaround

   Method: for each successful first-attempt run since Oct 8 that ran the suite, replace the self-hosted job with a hosted one that starts 0.1 minutes after the job is created (GitHub-hosted jobs queued p50 5 s, p99 82 s in the same window), and check whether it ends after Required status check started.

   Hosted job	Runs delayed	Delay
   Linux, 18.6 min (slowest measurement)	0 of 208	none; the leg had at least 13.7 minutes of slack in every run since #649
   Windows from scratch, 33 min	142 of 206 (69%)	median 6.9 min, max 13.3
   Windows with a warm sccache, 22.5 min	26 of 206 (13%)	median 1.6 min, max 2.8

   Counting the queue time that a hosted runner saves doesn't change this: the Windows leg was never the last job of those 206 runs, and it waited more than 16.5 minutes for a WHP runner in only 2 of them.

   4. Queue relief

   Method: as in #649, a FIFO replay of the observed arrivals and durations of the non-benchmark jobs from Oct 12–21 on today's general pools (4 MSHV, 8 WHP), without the dev-push suite runs that #660 skips. The replay understates real waits (observed since #649: MSHV p90 2.4, p99 7.4 min; WHP p90 3.6, p99 11.5 min), but shows the relative effect.

   Other jobs' queue wait	With the suite	Without it
   MSHV, p99	8.6 min	5.0 min
   MSHV, total over the 9.6 days	507 min	225 min
   WHP, p99	5.7 min	2.2 min
   WHP, total over the 9.6 days	128 min	64 min

   5. Options

   Option	Linux leg	Windows leg	Cost	Caching	Disk, timeouts
   Current self-hosted MSHV and WHP general runners	7.0 min job	16.5 min job	VMs already paid for; ~900 MSHV and ~2,100 WHP min a week	runner-local sccache, 10 GiB, shared with the VMM tests and OpenVMM builds	persistent disks; 90-min timeout
   GitHub-hosted standard runners (chosen for Linux)	12.5–18.6 min job, delays no run	33 min job, delays 69% of runs	free for public repositories	none kept between jobs	see table 2; 90-min timeout kept
   GitHub-hosted larger runners, on paper (needs organization setup)	not needed	~13–20 min (estimated from the 4-vCPU phases)	not free for public repositories: Windows 8-core $0.042/min, 16-core $0.082/min, so ~$430–580 a month for ~125 jobs a week	none kept	larger SSDs
   Self-hosted pool without a hypervisor, on paper	not needed	as today	~$570–605 a month per always-on 8-vCPU Windows VM at Azure list price (D8as_v5, D8s_v5), at ~20% utilization; the same money buys another general WHP runner, which can run any WHP job	runner-local sccache	as today

   6. Caching

   • sccache's GitHub Actions backend stores one cache entry per compilation: ~2,700 per Linux build and ~1,300 per Windows build. The repository may upload 200 cache entries and download 1,500 a minute, shared with every other job, and pull-request entries are scoped to their merge ref.
   • One sccache archive per OS in actions/cache would take 1.5 GiB for Linux and at least 2.3 GiB for Windows, which grows by ~650 MiB per build with its deterministic misses (below). The repository's Actions cache is at its 10 GB limit (9.99 GiB in 182 entries, 7-day retention), so either would evict guest-artifact caches, whose misses cost CI: maximize workflow parallelism and reduce turnaround time #430's critical path ~5 minutes. Raising the limit costs $0.07/GB-month but needs a repository or organization setting.
   • So the Linux leg runs without a compilation cache. If the rest of a run that runs the suite drops below ~19 minutes, which CI: maximize workflow parallelism and reduce turnaround time #430's 12–15-minute target implies, the leg will set the critical path; caching its compilation (7.6 minutes with a warm sccache) or an 8-core larger runner would then pay off.

   7. For #693

   On windows-latest, rebuilding with the sccache that the first build had just filled, in the same directory, still missed 293 of 1,318 cacheable compilations, the same 1,025/293 split as azure-windows-9 in run 39585909588. The Windows misses are deterministic, not evictions from a full 10 GiB cache, so raising SCCACHE_CACHE_SIZE won't remove them.
5. Pedro Henrique Penna (ppenna)

   closed this as completedin #717on Oct 21, 2026
6. Akhil Gupta (akhil9tiet)

   added a commit that references this issue on Oct 22, 2026

   ci: run the Linux OpenVMM unit tests on a GitHub-hosted runner

   ...

   e9369dc