ci: record each microVM scenario's duration in a machine-readable report #694

Activity

1. Pedro Henrique Penna (ppenna)

   self-assigned this

   on Oct 21, 2026
2. Pedro Henrique Penna (ppenna)

   added the

   Feature

   issue type on Oct 21, 2026
3. Pedro Henrique Penna (ppenna)

   added a parent issue on Oct 21, 2026

   • CI: maximize workflow parallelism and reduce turnaround time #430
4. Pedro Henrique Penna (ppenna)

   marked this as blocking on Oct 21, 2026

   • ci: shard the default microVM suite by scenario duration #696
5. ppenna commented on Oct 21, 2026

   

   Pedro Henrique Penna (ppenna)

   on Oct 21, 2026

   Last edited by ppenna

   ContributorAuthor

   More actions

   Per-scenario durations from #707

   #707 makes test-microvm flush its progress lines and write scenario-durations.jsonl, and makes every microVM job upload those reports. Two CI runs of #707 have published them, one artifact per microVM job that ran, nvx-microvm-durations-<os>-<backend>-<kernel>-<run-id>-<run-attempt>, with a report for every test-microvm step:

   • run 1: 39619765285, on f12c2cf57;
   • run 2: 39666359892, on a904264bc, after the rebase onto dev with openvmm: refuse kernel command lines that the guest kernel would truncate #661, which added command-line-limit to the default suite.

   Pull requests skip the MSHV and WHP debug-kernel jobs, which run the same step on dev pushes.

   The reports account for the steps' time

   Step time is from the job log, from the step's ##[group]Run line to the next step's; the reports' spans start at the Python process's start and follow one another without gaps. Run 2, with run 1's residual:

   Job	test-microvm step	Step (log)	Report spans	Setup	Teardown	Residual	Residual, first run
   WHP	Default Alpine suite	586.77 s	586.30 s	0.25 s	1.62 s	+0.47 s	+0.45 s
   WHP	Public managed sandbox acceptance	35.71 s	35.25 s	0.27 s	0.03 s	+0.46 s	+0.44 s
   WHP	Ubuntu guest smoke tests	55.34 s	54.90 s	0.26 s	0.12 s	+0.44 s	+0.43 s
   WHP	Azure Linux guest smoke tests	59.75 s	59.27 s	0.27 s	0.08 s	+0.48 s	+0.44 s
   KVM	Default Alpine suite	457.97 s	457.94 s	0.21 s	0.03 s	+0.03 s	+0.03 s
   KVM	Public managed sandbox acceptance	18.44 s	18.41 s	0.20 s	0.00 s	+0.03 s	+0.03 s
   KVM	Ubuntu guest smoke tests	22.91 s	22.88 s	0.21 s	0.00 s	+0.03 s	+0.03 s
   KVM	Azure Linux guest smoke tests	37.01 s	36.98 s	0.21 s	0.00 s	+0.02 s	+0.03 s
   KVM	Extended Ubuntu guest tests	55.90 s	55.88 s	0.22 s	0.00 s	+0.03 s	+0.03 s
   KVM debug kernel	Same-host restore scenarios	145.01 s	144.97 s	0.21 s	0.01 s	+0.04 s	+0.04 s
   MSHV	Default Alpine suite	421.65 s	421.62 s	0.23 s	0.02 s	+0.03 s	+0.03 s
   MSHV	Public managed sandbox acceptance	19.63 s	19.60 s	0.23 s	0.00 s	+0.03 s	+0.03 s
   MSHV	Ubuntu guest smoke tests	22.26 s	22.23 s	0.22 s	0.00 s	+0.04 s	+0.03 s
   MSHV	Azure Linux guest smoke tests	36.60 s	36.57 s	0.22 s	0.00 s	+0.03 s	+0.03 s

   • The residual is the shell around Python: at most 0.04 s with bash, and about 0.45 s with Windows PowerShell, which starts about 0.2 s before Python and exits after it.
   • Setup, the interpreter's start, imports, and checks before the first scenario, costs 0.2–0.3 s per test-microvm command. Teardown is the time ABI evidence scan of the logs: 1.6–1.8 s for WHP's default suite, at most 0.03 s on Linux.
   • The flushed progress lines now carry each scenario's start: in run 1's KVM debug-kernel log, every Running ... line's timestamp follows its record's started_at by at most 1.2 ms.

   Default Alpine suite, per scenario execution

   Seconds per execution, sorted by WHP; the vCPU counts are those the execution covers. Runners: azure-windows-9, azure-kvm-2, and azure-mshv-3 in run 1, and azure-windows-1, azure-kvm-1, and azure-mshv-1 in run 2.

   42 executions in run 2 (41 in run 1): WHP 584.4 s, KVM 457.7 s, MSHV 421.4 s

   | Scenario (vCPUs) | WHP 1 | WHP 2 | KVM 1 | KVM 2 | MSHV 1 | MSHV 2 | |---|---:|---:|---:|---:|---:|---:| | `restore-downtime` | 84.0 | 83.4 | 69.1 | 69.4 | 70.9 | 71.2 | | `smp-snapshot` 1,2,4,8 | 44.0 | 46.4 | 26.1 | 26.2 | 28.3 | 28.5 | | `ipv6-egress-policy` | 36.5 | 37.1 | 29.2 | 29.2 | 29.7 | 30.0 | | `l3-l4-egress-protocols` | 36.0 | 36.2 | 30.8 | 30.9 | 31.0 | 31.3 | | `restore-processors` 1,2,4,8 | 28.2 | 28.3 | 21.4 | 21.4 | 21.2 | 21.5 | | `host-loopback-policy` | 27.4 | 27.5 | 23.1 | 23.1 | 23.7 | 23.7 | | `console-exit` 4 | 20.3 | 26.5 | 6.2 | 6.2 | 6.6 | 6.7 | | `l3-l4-egress-port-ranges` | 26.4 | 26.4 | 22.2 | 22.1 | 22.2 | 22.6 | | `console-exit` 2 | 25.9 | 24.8 | 6.2 | 6.1 | 6.5 | 6.8 | | `console-exit` 8 | 20.6 | 16.7 | 6.2 | 6.2 | 6.9 | 7.0 | | `l3-l4-egress-policy` | 19.8 | 19.9 | 19.3 | 19.3 | 19.3 | 19.3 | | `managed-lifecycle` | 18.6 | 19.1 | 4.0 | 4.5 | 4.3 | 7.5 | | `smp` 8 | 17.8 | 16.8 | 13.6 | 13.6 | 14.2 | 14.3 | | `network-snapshot` | 17.5 | 17.6 | 17.0 | 17.0 | 16.8 | 16.9 | | `private-network-ingress` | 16.9 | 17.0 | 31.4 | 31.2 | 31.5 | 31.8 | | `snapshot-core` | 16.4 | 16.4 | 13.5 | 13.5 | 12.8 | 12.9 | | `console-exit` 1 | 15.8 | 16.2 | 6.5 | 6.5 | 6.8 | 6.7 | | `endpoint-policy-snapshot` | 12.0 | 12.0 | 10.2 | 10.3 | 10.0 | 10.1 | | `snapshot-tiers` | 12.0 | 12.0 | 6.9 | 6.7 | 6.3 | 6.6 | | `restore-memory` | 10.9 | 11.5 | 4.1 | 4.3 | 4.0 | 4.1 | | `scratch-snapshot` | 8.0 | 8.2 | 3.3 | 3.4 | 3.1 | 3.3 | | `filesystem-snapshot` | 7.1 | 7.1 | 3.2 | 3.3 | 2.7 | 2.9 | | `directional-network-policy` | 6.7 | 6.8 | 5.8 | 5.9 | 5.7 | 5.7 | | `smp` 4 | 6.3 | 6.3 | 4.4 | 4.5 | 4.8 | 4.8 | | `filesystem-shares` | 5.2 | 5.2 | 2.0 | 2.0 | 2.1 | 2.2 | | `filesystem-comma-paths` | 4.7 | 4.9 | 2.6 | 2.5 | 2.8 | 2.9 | | `filesystem-policy` | 4.4 | 4.5 | 1.9 | 1.9 | 1.8 | 2.0 | | `smp` 2 | 4.3 | 4.4 | 2.3 | 2.3 | 2.6 | 2.6 | | `virtio-net` | 4.3 | 4.3 | 3.8 | 3.8 | 3.8 | 3.7 | | `cpu-profile-fallback` | 3.8 | 3.9 | 1.4 | 1.6 | 1.7 | 1.8 | | `console-snapshot` | 3.3 | 3.5 | 49.1 | 48.8 | 1.0 | 1.0 | | `command-line-limit` | | 2.1 | | 1.4 | | 1.4 | | `smp` 1 | 1.9 | 2.0 | 0.9 | 0.9 | 1.0 | 1.0 | | `lifecycle` | 2.0 | 2.0 | 1.8 | 1.7 | 1.7 | 1.6 | | `time-abi-conformance` 8 | 1.6 | 1.6 | 0.7 | 0.8 | 0.8 | 0.8 | | `denied-filesystem-paths` | 1.2 | 1.2 | 0.7 | 0.8 | 0.7 | 0.8 | | `sandbox-blocks` | 1.1 | 1.2 | 0.7 | 0.7 | 0.6 | 0.6 | | `workload-identity` | 1.0 | 1.1 | 0.7 | 0.7 | 0.6 | 0.6 | | `guest-boot` | 0.7 | 0.8 | 0.7 | 0.7 | 0.5 | 0.5 | | `structured-outcome` | 0.8 | 0.8 | 0.7 | 0.6 | 0.5 | 0.5 | | `guest-identity` | 0.8 | 0.8 | 0.7 | 0.7 | 0.5 | 0.5 | | `filesystem-owner` | 0.0 | 0.0 | 0.8 | 0.8 | 0.8 | 0.8 | | **Total** | **576.0** | **584.4** | **455.4** | **457.7** | **412.8** | **421.4** |

   The KVM debug-kernel job's restore scenarios took 144.5 s in run 1 and 144.8 s in run 2: smp 0.9, 2.3, 4.4, and 13.6 s on 1, 2, 4, and 8 vCPUs, smp-snapshot 26.0–26.1 s, restore-processors 21.3–21.4 s, restore-downtime 69.1–69.3 s, and snapshot-tiers 6.8–6.9 s.

   Observations for #696

   • The suite is top-heavy: on WHP, in run 2, the 5 longest executions take 231 s (40%) and the 10 longest 357 s (61%). restore-downtime alone takes 69–84 s, 30 s of which is its fixed downtime window, which bounds any shard that runs it.
   • Across the two runs, on different runners, every KVM and MSHV execution repeats within 0.5 s, except MSHV's managed-lifecycle (4.3 and 7.5 s). WHP varies more: 32 of its 41 executions repeat within 0.5 s, but console-exit on 4 vCPUs took 20.3 and 26.5 s, and smp-snapshot 44.0 and 46.4 s.
   • The four console-exit executions take 83–84 s on WHP but 25–27 s on Linux, and managed-lifecycle, which builds the nvx-rs test with Cargo, about 19 s on WHP but 4–8 s on Linux. A WHP split needs WHP durations.
   • console-snapshot takes 49 s on KVM because it sends 10,000 TX records there, against 1,000 on WHP (3.5 s) and 100 on MSHV (1.0 s); private-network-ingress takes 31 s on Linux but 17 s on WHP. Per-backend durations differ in rank, not only in scale.
   • command-line-limit, new in openvmm: refuse kernel command lines that the guest kernel would truncate #661, takes 2.1 s on WHP and 1.4 s on KVM and MSHV.
   • Against reference run 39585909588, the default suite took 578.5 and 586.8 s instead of 603 s on WHP, and 455.7 and 458.0 s instead of 459 s on KVM. Two runs on different runners are still few, so ci: shard the default microVM suite by scenario duration #696 should aggregate more.
6. Pedro Henrique Penna (ppenna)

   closed this as completedin #707on Oct 22, 2026
7. Pedro Henrique Penna (ppenna)

   added 2 commits that reference this issue on Oct 22, 2026

   microvm tests: split the default suite into duration-balanced shards

   ...

   a054653

   microvm tests: split the default suite into duration-balanced shards

   ...

   80c195c