From 80c195c77762087f3817da9bdac7502d9e1624bf Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 22 Oct 2026 12:09:28 -0300 Subject: [PATCH 1/2] microvm tests: split the default suite into duration-balanced shards The default Alpine suite of test-microvm runs its 36 scenarios one after another: 457.7 seconds on KVM, 416.0 on MSHV, and 601.5 on WHP, as the medians of the scenario duration reports (#694) of the 18 CI runs of October 21 and 22 that the partition lists add up. On WHP its job bounds every run that skips both OpenVMM suites (#696). No scenario depends on another, so CI can run the suite in parallel shards, as long as every scenario runs in exactly one of them and the shards take about as long as each other, whatever number of scenarios each runs. scripts/nvx_tools/microvm_shards.json is that partition: for each backend, the median duration of every default scenario, the CI runs whose reports gave them, and each shard's scenarios with their total. The backends differ too much for one partition: console-snapshot takes 49 seconds on KVM but 1 on MSHV and 4 on WHP, and console-exit 25 on KVM but 87.5 on WHP. The best partition common to all three that a search found left the longest KVM and MSHV shards about 5% above an even split, where one per backend balances each to within a second, so each backend has its own. The new plan-microvm-shards command writes the file. It takes each backend's median of every default scenario over the reports that it is given, sharded or not, of the Alpine guest on the default kernel with the default processor counts, from runs that ended; a report counts for a scenario only when every execution of it passed. Each scenario starts in its current shard, and one without a shard goes, longest first, to the shortest. A shard that is still empty, as one that --shards adds can be, takes the longest scenario of the longest shard that has more than one, since moving a scenario that takes no time into it would shorten no shard. The plan then moves a scenario, or swaps two of different shards, while that shortens the longest shard by a second or more, so that new medians that move within the run-to-run noise leave the shards alone. A scenario that no report measures keeps its duration, or takes an estimate from --seconds, and --shards changes a backend's number of shards. The first plan, from 13 of the runs, started from the two shards that #696 proposed, the restore, snapshot, and time ABI scenarios and the network, filesystem, and lifecycle ones, which were up to 18% apart on KVM, and a single swap per backend balanced each. Planned again from all 18, which add two later dev pushes, another pull request's run, and two runs of this change, every scenario stayed in its shard: 228.8 and 228.9 seconds on KVM, 208.0 and 208.0 on MSHV, and 300.4 and 301.1 on WHP. test-microvm --shard INDEX/COUNT runs one shard of the default suite. It fails before it runs a scenario unless the file puts every default scenario in exactly one shard of every backend and splits the backend's suite into COUNT shards. A scenario that a later change adds, as #661 added command-line-limit, therefore cannot drop out of a sharded run, and a run that counts a different number of shards than the file fails. The option takes no --scenario, --debug-kernel, or other guest. Only the microVM tests use the new module and its data, so the classifier leaves them out of the platform jobs' inputs, and doc/ci.md says so. New tests check that the checked-in file puts each default scenario in exactly one shard of each backend and is the planner's own plan of its durations, and that the shards of each backend together run the default suite that an unsharded run selects, each scenario once. They check that --shard fails before any scenario runs for a new, removed, doubled, or missing scenario, an empty shard, or a count that differs from the file's, and that it rejects other selections. They also cover the planner's balancing, its threshold, its placement of new scenarios and added shards, which it fills even with scenarios that take no time, its determinism, which reports it counts, the command's options and errors, and malformed files. The classification tests cover the new files. doc/usage.md documents the command and the option, and doc/ci.md how the shards are planned and how to plan them again. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/ci.md | 72 ++- doc/usage.md | 51 +- scripts/nvx.py | 12 + scripts/nvx_tools/ci.py | 6 +- scripts/nvx_tools/microvm_shards.json | 290 +++++++++++ scripts/nvx_tools/microvm_shards.py | 673 +++++++++++++++++++++++++ scripts/nvx_tools/microvm_tests.py | 48 +- scripts/test_microvm_tests.py | 685 ++++++++++++++++++++++++++ scripts/test_nvx_tools.py | 13 + 9 files changed, 1842 insertions(+), 8 deletions(-) create mode 100644 scripts/nvx_tools/microvm_shards.json create mode 100644 scripts/nvx_tools/microvm_shards.py diff --git a/doc/ci.md b/doc/ci.md index 7e5466763..9bbd94f34 100644 --- a/doc/ci.md +++ b/doc/ci.md @@ -583,6 +583,72 @@ pool has a single free general runner, either job of a backend can take it first. If the guest and sandbox job does, the default suite waits for it, and the backend's microVM tests take about as long as one job did. +## MicroVM test shards + +`nvx.py test-microvm --shard INDEX/COUNT` runs one of COUNT shards of the +default Alpine suite on the default kernel, which holds the scenarios that +`test-microvm` runs without `--scenario`. The partition is +[`microvm_shards.json`](../scripts/nvx_tools/microvm_shards.json): for each +backend, the median duration of every default scenario, the scenarios of each +shard, and how long each shard takes. The backends differ too much for one +partition to balance them all: `console-snapshot` takes 49.1 seconds on KVM +but 1.0 on MSHV and 3.9 on WHP, and `console-exit` 25.2 seconds on KVM but 87.5 +on WHP, so each backend has a partition of its own. +[`plan-microvm-shards`](usage.md#plan-microvm-shards) plans them from the +[scenario duration reports](usage.md#scenario-duration-report) of CI runs, so +that the shards of a backend take about as long as each other, whatever number +of scenarios each runs. + +The first plan took the medians of the reports of 13 runs from October 21 and +22, 2026. It started each backend from the two shards that +[#696](https://github.com/microsoft/nvx/issues/696) proposed, the restore, +snapshot, and time ABI scenarios and the network, filesystem, and lifecycle +scenarios, which took 248 and 210 seconds on KVM, 203 and 213 on MSHV, and 287 +and 309 on WHP, and on each backend the planner swapped one scenario of the +first shard for one of the second. Planned again from the reports of the 18 +runs that the file lists, which add two later `dev` pushes, another pull +request's run, and two runs of [#763](https://github.com/microsoft/nvx/pull/763), +which added the shards, every scenario stayed in its shard: + +| Backend | Default suite | Shard 1 | Shard 2 | Swapped from the proposed shards | +| --- | ---: | ---: | ---: | --- | +| KVM | 457.7 s | 228.8 s | 228.9 s | `smp` for `filesystem-comma-paths` | +| MSHV | 416.0 s | 208.0 s | 208.0 s | `time-abi-conformance` for `directional-network-policy` | +| WHP | 601.5 s | 300.4 s | 301.1 s | `snapshot-core` for `host-loopback-policy` | + +A sharded run fails before it runs a scenario unless the file puts every +default scenario in exactly one shard of every backend, with its duration, +and splits the backend's suite into the run's COUNT shards. Tests in +`scripts/test_microvm_tests.py` check the same, and that the file is what the +planner makes of its own durations, so a change that adds, removes, or +renames a default scenario fails them until it plans the shards again: + +1. Download the scenario duration reports of recent CI runs that ran the + default suite, such as the last ten `dev` pushes, for each run: + + ```bash + gh run download RUN_ID --repo microsoft/nvx \ + --pattern 'nvx-microvm-durations-*' --dir build/durations/RUN_ID + ``` + +2. Plan the shards from them: + + ```bash + python3 scripts/nvx.py plan-microvm-shards --report build/durations + ``` + + A scenario that no report measures keeps its duration in the file. A new + scenario has none, so give it an estimate, such as + `--seconds command-line-limit=2`, which the report of a local + `test-microvm --scenario command-line-limit` run gives. A regeneration from + CI's reports replaces the estimate once CI has run the scenario. + +3. Commit the file with the change. The planner prints each backend's shards + and the scenarios that changed shard. A scenario stays in its shard unless + moving it, or swapping it for a scenario of another shard, shortens the + longest shard by a second or more, so new medians that only move within + the run-to-run noise leave the shards as they are. + ## Guest artifact handoff Three GitHub-hosted jobs restore or build the guest artifacts side by side, @@ -772,8 +838,10 @@ Every CLI module is an input of the microVM tests, and of the platform jobs, unless none of that suite's jobs runs it, so a new module selects both until the classifier excludes it. `microvm_tests.py` and the modules that only it, `nvx.py sandbox`, or `nvx.py test-aci-edge-sandboxes` use, with the -`microvm_test_scripts/` that it runs in guests, are inputs of the microVM tests -alone, and `performance.py` is an input of the platform jobs alone. The +`microvm_test_scripts/` that it runs in guests and the partition of its +[shards](#microvm-test-shards), `microvm_shards.json`, are inputs of the +microVM tests alone, and `performance.py` is an input of the platform jobs +alone. The modules that both kinds of job run, such as `benchmark.py`, which `microvm_tests.py` imports, `doctor.py`, which every runner validation runs, and `release.py`, whose `verify` command the OpenVMM builds run, select both. diff --git a/doc/usage.md b/doc/usage.md index 5a600b7de..4f485403a 100644 --- a/doc/usage.md +++ b/doc/usage.md @@ -87,6 +87,7 @@ stale build never ships or runs unnoticed. | `build-openvmm-tests` | Build OpenVMM Petri VMM tests without running them. | | `test-openvmm` | Run OpenVMM Petri VMM tests. | | `test-microvm` | Run NVX Linux and device correctness tests through OpenVMM. | +| `plan-microvm-shards` | Split the default microVM suite into shards of balanced scenario durations for CI. | | `doctor` | Qualify this host for the NVX time ABI. | | `test-aci-edge-sandboxes` | Run the `aci_edge_sandboxes` Rust crate lifecycle test on a real hypervisor. | | `test-agent-bundling` | Check the `aci_edge_sandboxes` crate's agent bundle staging without a hypervisor. | @@ -514,6 +515,7 @@ python3 scripts/nvx.py test-microvm [--guest {alpine,ubuntu,azurelinux}] [--scenario SCENARIO]... [--debug-kernel] + [--shard INDEX/COUNT] [--processors {1,2,4,8} ...] [--memory-mib MIB] [--timeout SECONDS] @@ -536,7 +538,14 @@ the Alpine-prompt-specific `console-snapshot` scenario. `--debug-kernel` boots the CI debug kernel, `build/vmlinux-debug`, whose soft-lockup and hung-task detectors the guest's time ABI watcher reports; without `--scenario`, it runs only the same-host restore scenarios `smp`, `smp-snapshot`, -`restore-processors`, `restore-downtime`, and `snapshot-tiers`. The +`restore-processors`, `restore-downtime`, and `snapshot-tiers`. +`--shard INDEX/COUNT`, such as `1/2`, runs shard INDEX of the COUNT shards of +the default Alpine suite on the default kernel, as +[`microvm_shards.json`](../scripts/nvx_tools/microvm_shards.json) splits it for +the backend (see [MicroVM test shards](ci.md#microvm-test-shards)). It takes no +`--scenario`, `--debug-kernel`, or other `--guest`, and fails before it runs a +scenario unless that file puts each default scenario in exactly one shard of +every backend and splits the backend's suite into COUNT shards. The `cpu-profile-fallback` scenario cold boots with OpenVMM's `host-cpu-unknown` test hook, which makes OpenVMM's default CPU profile selection, `auto`, fall back to a host profile on a host that a built-in profile serves, checks the @@ -593,6 +602,46 @@ take them. A failed run ends with the failed scenario's record and the `teardown`; a run that is killed keeps every record that it completed and has no `teardown`. +### `plan-microvm-shards` + +```text +python3 scripts/nvx.py plan-microvm-shards + [--report PATH]... + [--seconds SCENARIO=SECONDS]... + [--shards BACKEND=COUNT]... + [--partition PATH] +``` + +Plans the shards of the default Alpine suite on the default kernel, which CI +runs in parallel (see [MicroVM test shards](ci.md#microvm-test-shards)), and +writes them to [`microvm_shards.json`](../scripts/nvx_tools/microvm_shards.json), +or to `--partition`. The file holds each backend's median duration of every +default scenario, the CI runs whose reports gave them, and each shard's +scenarios with their total duration. + +`--report` reads [scenario duration reports](#scenario-duration-report), or +every report in a directory, such as the one that `gh run download` fills with +a run's `nvx-microvm-durations-*` artifacts. Each backend's duration of a +scenario is the median, to a tenth of a second, of the reports of the default +suite, sharded or not: those of the Alpine guest on the default kernel with +the default processor counts, of runs that ended, passed or not. A report +counts for a scenario only when every execution of the scenario passed, such +as the four of `console-exit`. A scenario that no report measures keeps its +duration in the file. `--seconds` sets a scenario's duration on every backend +instead, such as an estimate of a new scenario that no report measures yet; +the command fails when a default scenario has no duration. + +Each backend keeps its number of shards unless `--shards` changes it. Each +scenario starts in its shard of the file, and a scenario that has none goes, +longest first, to the shard that takes the least time so far. A shard that is +still empty, as one that `--shards` adds can be, takes the longest scenario of +the longest shard that has more than one. The command then moves a scenario +to another shard, or swaps two scenarios of different shards, while that +shortens the longest shard by at least a second, comparing the next longest +where the longest stay alike, and takes the best such change each time. It +prints each backend's shards with their durations, and the scenarios that +changed shard. + ### `doctor` ```text diff --git a/scripts/nvx.py b/scripts/nvx.py index 9c2ab73b8..77df04c56 100644 --- a/scripts/nvx.py +++ b/scripts/nvx.py @@ -86,6 +86,9 @@ ) from nvx_tools.guests import GUEST_NAMES, guest_descriptor from nvx_tools.microvm_tests import configure_parser as configure_microvm_test_parser +from nvx_tools.microvm_tests import ( + configure_shard_parser as configure_microvm_shard_parser, +) from nvx_tools.nightly_ci import configure_parser as configure_nightly_ci_parser from nvx_tools.performance import configure_parser as configure_performance_parser from nvx_tools.release import ( @@ -990,6 +993,15 @@ def parse_args(argv: Sequence[str] | None = None) -> argparse.Namespace: ) configure_microvm_test_parser(microvm_tests) + microvm_shards = subparsers.add_parser( + "plan-microvm-shards", + help=( + "split the default microVM suite into shards of balanced scenario " + "durations for CI" + ), + ) + configure_microvm_shard_parser(microvm_shards) + doctor = subparsers.add_parser( "doctor", help="qualify this host for the NVX time ABI", diff --git a/scripts/nvx_tools/ci.py b/scripts/nvx_tools/ci.py index 3950c0380..3ad17a445 100644 --- a/scripts/nvx_tools/ci.py +++ b/scripts/nvx_tools/ci.py @@ -200,9 +200,9 @@ r"scripts/nvx\.py$|" r"scripts/nvx_tools/(?!(aci_edge_sandboxes_tests|adversarial|" r"adversarial_broker|adversarial_executor|agent_bundling_tests|" - r"create_linux_source_archive|managed_exec_tests|microvm_tests|" - r"openvmm_process|sandbox|sandbox_lifecycle_tests)\.py$|" - r"adversarial_cases/|microvm_test_scripts/))" + r"create_linux_source_archive|managed_exec_tests|microvm_shards|" + r"microvm_tests|openvmm_process|sandbox|sandbox_lifecycle_tests)\.py$|" + r"microvm_shards\.json$|adversarial_cases/|microvm_test_scripts/))" ) # Inputs of the OpenVMM unit and documentation tests: the pinned revision, the # Rust release, the runner setup, and the code and workflow that run them. diff --git a/scripts/nvx_tools/microvm_shards.json b/scripts/nvx_tools/microvm_shards.json new file mode 100644 index 000000000..6b166f430 --- /dev/null +++ b/scripts/nvx_tools/microvm_shards.json @@ -0,0 +1,290 @@ +{ + "schema": "nvx-microvm-shards", + "schema_version": 1, + "runs": [ + "39619765285-1", + "39666359892-1", + "39777444440-1", + "39778084391-1", + "39779640214-1", + "39780305955-1", + "39782906524-1", + "39783750996-1", + "39785367867-1", + "39785687509-1", + "39788615461-1", + "39791725058-1", + "39792079980-1", + "39801222132-1", + "39804644654-1", + "39805377165-1", + "39807968692-1", + "39810434285-1" + ], + "backends": { + "kvm": { + "seconds": { + "command-line-limit": 1.5, + "console-exit": 25.2, + "console-snapshot": 49.1, + "cpu-profile-fallback": 1.5, + "denied-filesystem-paths": 0.8, + "directional-network-policy": 5.8, + "endpoint-policy-snapshot": 10.3, + "filesystem-comma-paths": 2.6, + "filesystem-owner": 0.8, + "filesystem-policy": 1.9, + "filesystem-shares": 2.0, + "filesystem-snapshot": 3.2, + "guest-boot": 0.7, + "guest-identity": 0.7, + "host-loopback-policy": 23.1, + "ipv6-egress-policy": 29.4, + "l3-l4-egress-policy": 19.3, + "l3-l4-egress-port-ranges": 22.4, + "l3-l4-egress-protocols": 31.1, + "lifecycle": 1.7, + "managed-lifecycle": 4.2, + "network-snapshot": 17.0, + "private-network-ingress": 31.3, + "restore-downtime": 69.2, + "restore-memory": 4.1, + "restore-processors": 21.4, + "sandbox-blocks": 0.7, + "scratch-snapshot": 3.4, + "smp": 21.2, + "smp-snapshot": 26.1, + "snapshot-core": 13.4, + "snapshot-tiers": 6.8, + "structured-outcome": 0.6, + "time-abi-conformance": 0.7, + "virtio-net": 3.8, + "workload-identity": 0.7 + }, + "shards": [ + { + "seconds": 228.8, + "scenarios": [ + "console-snapshot", + "cpu-profile-fallback", + "endpoint-policy-snapshot", + "filesystem-comma-paths", + "filesystem-snapshot", + "network-snapshot", + "restore-downtime", + "restore-memory", + "restore-processors", + "scratch-snapshot", + "smp-snapshot", + "snapshot-core", + "snapshot-tiers", + "time-abi-conformance" + ] + }, + { + "seconds": 228.9, + "scenarios": [ + "command-line-limit", + "console-exit", + "denied-filesystem-paths", + "directional-network-policy", + "filesystem-owner", + "filesystem-policy", + "filesystem-shares", + "guest-boot", + "guest-identity", + "host-loopback-policy", + "ipv6-egress-policy", + "l3-l4-egress-policy", + "l3-l4-egress-port-ranges", + "l3-l4-egress-protocols", + "lifecycle", + "managed-lifecycle", + "private-network-ingress", + "sandbox-blocks", + "smp", + "structured-outcome", + "virtio-net", + "workload-identity" + ] + } + ] + }, + "mshv": { + "seconds": { + "command-line-limit": 1.2, + "console-exit": 27.2, + "console-snapshot": 1.0, + "cpu-profile-fallback": 1.7, + "denied-filesystem-paths": 0.6, + "directional-network-policy": 5.7, + "endpoint-policy-snapshot": 10.1, + "filesystem-comma-paths": 2.9, + "filesystem-owner": 0.8, + "filesystem-policy": 1.8, + "filesystem-shares": 2.2, + "filesystem-snapshot": 2.9, + "guest-boot": 0.5, + "guest-identity": 0.5, + "host-loopback-policy": 23.5, + "ipv6-egress-policy": 30.0, + "l3-l4-egress-policy": 19.3, + "l3-l4-egress-port-ranges": 22.5, + "l3-l4-egress-protocols": 31.3, + "lifecycle": 1.6, + "managed-lifecycle": 4.3, + "network-snapshot": 16.8, + "private-network-ingress": 31.6, + "restore-downtime": 71.0, + "restore-memory": 4.0, + "restore-processors": 21.3, + "sandbox-blocks": 0.6, + "scratch-snapshot": 3.2, + "smp": 22.6, + "smp-snapshot": 28.5, + "snapshot-core": 12.9, + "snapshot-tiers": 6.3, + "structured-outcome": 0.5, + "time-abi-conformance": 0.8, + "virtio-net": 3.7, + "workload-identity": 0.6 + }, + "shards": [ + { + "seconds": 208.0, + "scenarios": [ + "console-snapshot", + "cpu-profile-fallback", + "directional-network-policy", + "endpoint-policy-snapshot", + "filesystem-snapshot", + "network-snapshot", + "restore-downtime", + "restore-memory", + "restore-processors", + "scratch-snapshot", + "smp", + "smp-snapshot", + "snapshot-core", + "snapshot-tiers" + ] + }, + { + "seconds": 208.0, + "scenarios": [ + "command-line-limit", + "console-exit", + "denied-filesystem-paths", + "filesystem-comma-paths", + "filesystem-owner", + "filesystem-policy", + "filesystem-shares", + "guest-boot", + "guest-identity", + "host-loopback-policy", + "ipv6-egress-policy", + "l3-l4-egress-policy", + "l3-l4-egress-port-ranges", + "l3-l4-egress-protocols", + "lifecycle", + "managed-lifecycle", + "private-network-ingress", + "sandbox-blocks", + "structured-outcome", + "time-abi-conformance", + "virtio-net", + "workload-identity" + ] + } + ] + }, + "whp": { + "seconds": { + "command-line-limit": 2.3, + "console-exit": 87.5, + "console-snapshot": 3.9, + "cpu-profile-fallback": 4.3, + "denied-filesystem-paths": 1.3, + "directional-network-policy": 7.0, + "endpoint-policy-snapshot": 12.4, + "filesystem-comma-paths": 5.3, + "filesystem-owner": 0.0, + "filesystem-policy": 4.8, + "filesystem-shares": 5.8, + "filesystem-snapshot": 8.1, + "guest-boot": 0.8, + "guest-identity": 0.8, + "host-loopback-policy": 28.2, + "ipv6-egress-policy": 38.6, + "l3-l4-egress-policy": 19.9, + "l3-l4-egress-port-ranges": 27.4, + "l3-l4-egress-protocols": 37.5, + "lifecycle": 2.1, + "managed-lifecycle": 17.8, + "network-snapshot": 17.6, + "private-network-ingress": 17.4, + "restore-downtime": 84.7, + "restore-memory": 12.5, + "restore-processors": 28.8, + "sandbox-blocks": 1.2, + "scratch-snapshot": 8.8, + "smp": 30.9, + "smp-snapshot": 46.6, + "snapshot-core": 17.2, + "snapshot-tiers": 12.0, + "structured-outcome": 0.8, + "time-abi-conformance": 1.6, + "virtio-net": 4.5, + "workload-identity": 1.1 + }, + "shards": [ + { + "seconds": 300.4, + "scenarios": [ + "console-snapshot", + "cpu-profile-fallback", + "endpoint-policy-snapshot", + "filesystem-snapshot", + "host-loopback-policy", + "network-snapshot", + "restore-downtime", + "restore-memory", + "restore-processors", + "scratch-snapshot", + "smp", + "smp-snapshot", + "snapshot-tiers", + "time-abi-conformance" + ] + }, + { + "seconds": 301.1, + "scenarios": [ + "command-line-limit", + "console-exit", + "denied-filesystem-paths", + "directional-network-policy", + "filesystem-comma-paths", + "filesystem-owner", + "filesystem-policy", + "filesystem-shares", + "guest-boot", + "guest-identity", + "ipv6-egress-policy", + "l3-l4-egress-policy", + "l3-l4-egress-port-ranges", + "l3-l4-egress-protocols", + "lifecycle", + "managed-lifecycle", + "private-network-ingress", + "sandbox-blocks", + "snapshot-core", + "structured-outcome", + "virtio-net", + "workload-identity" + ] + } + ] + } + } +} diff --git a/scripts/nvx_tools/microvm_shards.py b/scripts/nvx_tools/microvm_shards.py new file mode 100644 index 000000000..b71e533cd --- /dev/null +++ b/scripts/nvx_tools/microvm_shards.py @@ -0,0 +1,673 @@ +"""The shards of the default microVM suite, which CI runs in parallel. + +doc/ci.md#microvm-test-shards describes the partition, how this module plans +it from scenario duration reports, and when to plan it again. +""" + +from __future__ import annotations + +import argparse +import itertools +import json +import math +import re +import statistics +from collections import Counter +from collections.abc import Callable, Collection, Iterable, Iterator, Mapping, Sequence +from dataclasses import dataclass +from pathlib import Path +from typing import cast + +from . import microvm_durations +from .ci import OPENVMM_TEST_BACKENDS +from .common import ScriptError, strict_json_object + +PARTITION_PATH = Path(__file__).with_name("microvm_shards.json") +SCHEMA = "nvx-microvm-shards" +SCHEMA_VERSION = 1 +# A change must shorten the longest shard, or the next longest where the +# longest stay alike, by at least this many tenths of a second, so that the +# noise of new medians does not reshuffle a partition that is balanced already. +MINIMUM_GAIN_TENTHS = 10 +PLAN_ADVICE = ( + "plan the shards again with `nvx.py plan-microvm-shards` " + "(doc/ci.md#microvm-test-shards)" +) + + +@dataclass(frozen=True) +class BackendShards: + """One backend's partition of the default suite: each scenario's median + duration in seconds, and the scenarios of each shard, in shard order.""" + + seconds: Mapping[str, float] + shards: tuple[tuple[str, ...], ...] + + def shard_seconds(self) -> tuple[float, ...]: + """Return how long each shard's scenarios take in all.""" + return tuple( + sum(_tenths(self.seconds[scenario]) for scenario in shard) / 10 + for shard in self.shards + ) + + +@dataclass(frozen=True) +class Partition: + """The shards of every backend, and the CI runs, as `-`, + whose reports gave the durations.""" + + runs: tuple[str, ...] + backends: Mapping[str, BackendShards] + + +def _tenths(seconds: float) -> int: + return round(seconds * 10) + + +def parse_shard(value: str) -> tuple[int, int]: + """Parse a --shard value, INDEX/COUNT, such as 1/2 for the first of two + shards.""" + match = re.fullmatch(r"([1-9][0-9]*)/([1-9][0-9]*)", value) + if match is None: + raise argparse.ArgumentTypeError("must be INDEX/COUNT, such as 1/2") + index, count = int(match.group(1)), int(match.group(2)) + if index > count: + raise argparse.ArgumentTypeError("INDEX must be at most COUNT") + return index, count + + +def _number(value: object) -> float | None: + if isinstance(value, bool) or not isinstance(value, (int, float)): + return None + return float(value) + + +def _strings(value: object) -> tuple[str, ...] | None: + if not isinstance(value, list): + return None + items = cast(list[object], value) + if not all(isinstance(item, str) for item in items): + return None + return tuple(cast(list[str], items)) + + +def _object(value: object, keys: Collection[str]) -> dict[str, object] | None: + if not isinstance(value, dict): + return None + document = cast(dict[str, object], value) + return document if set(document) == set(keys) else None + + +def _backend_shards(value: object) -> BackendShards | None: + document = _object(value, ("seconds", "shards")) + if document is None: + return None + raw_seconds = document["seconds"] + raw_shards = document["shards"] + if not isinstance(raw_seconds, dict) or not isinstance(raw_shards, list): + return None + seconds: dict[str, float] = {} + for scenario, duration in cast(dict[str, object], raw_seconds).items(): + number = _number(duration) + if number is None or number < 0: + return None + seconds[scenario] = number + shards: list[tuple[str, ...]] = [] + for raw_shard in cast(list[object], raw_shards): + shard = _object(raw_shard, ("seconds", "scenarios")) + if shard is None or _number(shard["seconds"]) is None: + return None + scenarios = _strings(shard["scenarios"]) + if scenarios is None: + return None + shards.append(scenarios) + return BackendShards(seconds, tuple(shards)) + + +def load_partition(path: Path = PARTITION_PATH) -> Partition: + """Read a partition, failing unless its structure is valid; whether it + partitions the default suite is check_partition()'s question.""" + try: + text = path.read_text(encoding="utf-8") + except FileNotFoundError: + raise ScriptError(f"{path} does not exist; {PLAN_ADVICE}") from None + try: + value: object = json.loads(text, object_pairs_hook=strict_json_object) + except ValueError as error: + raise ScriptError(f"{path} is not valid JSON: {error}") from error + document = _object(value, ("schema", "schema_version", "runs", "backends")) + if ( + document is None + or document["schema"] != SCHEMA + or document["schema_version"] != SCHEMA_VERSION + ): + raise ScriptError( + f"{path} is not an {SCHEMA} document of schema version {SCHEMA_VERSION}" + ) + runs = _strings(document["runs"]) + raw_backends = document["backends"] + if runs is None or not isinstance(raw_backends, dict): + raise ScriptError(f"{path} has malformed runs or backends") + backends: dict[str, BackendShards] = {} + for backend, raw in cast(dict[str, object], raw_backends).items(): + shards = _backend_shards(raw) + if shards is None: + raise ScriptError(f"{path} has a malformed partition of {backend}") + backends[backend] = shards + return Partition(runs, backends) + + +def partition_problems(partition: Partition, suite: Sequence[str]) -> list[str]: + """Return how a partition fails to put each scenario of the default suite + in exactly one shard of every backend, with its duration.""" + problems: list[str] = [] + expected = set(suite) + for backend in sorted(set(partition.backends) - set(OPENVMM_TEST_BACKENDS)): + problems.append(f"{backend} is not a backend") + for backend in OPENVMM_TEST_BACKENDS: + shards = partition.backends.get(backend) + if shards is None: + problems.append(f"{backend} has no shards") + continue + if not shards.shards: + problems.append(f"{backend} has no shards") + placements: dict[str, list[int]] = {} + for index, shard in enumerate(shards.shards, 1): + if not shard: + problems.append(f"{backend} shard {index} is empty") + for scenario in shard: + placements.setdefault(scenario, []).append(index) + for scenario in suite: + found = placements.get(scenario, []) + if not found: + problems.append(f"{backend}: {scenario} is in no shard") + elif len(found) > 1: + problems.append( + f"{backend}: {scenario} is in more than one shard: " + + ", ".join(str(index) for index in found) + ) + for scenario in sorted(set(placements) - expected): + problems.append(f"{backend}: {scenario} is not a default scenario") + for scenario in suite: + if scenario not in shards.seconds: + problems.append(f"{backend}: {scenario} has no duration") + for scenario in sorted(set(shards.seconds) - expected): + problems.append( + f"{backend}: {scenario} has a duration but is not a default scenario" + ) + return problems + + +def check_partition( + partition: Partition, suite: Sequence[str], path: Path = PARTITION_PATH +) -> None: + """Fail unless the partition puts each scenario of the default suite in + exactly one shard of every backend.""" + problems = partition_problems(partition, suite) + if problems: + raise ScriptError( + f"{path} does not put each default microVM scenario in exactly one " + f"shard: {'; '.join(problems)}; {PLAN_ADVICE}" + ) + + +def shard_scenarios( + partition: Partition, + backend: str, + shard: tuple[int, int], + suite: Sequence[str], + path: Path = PARTITION_PATH, +) -> tuple[str, ...]: + """Return the scenarios of shard INDEX of COUNT of a backend's default + suite, in suite order, once the partition is checked and has COUNT shards + for the backend.""" + check_partition(partition, suite, path) + index, count = shard + shards = partition.backends[backend].shards + if count != len(shards): + raise ScriptError( + f"{path} splits the {backend} suite into {len(shards)} shards, but " + f"this run is shard {index} of {count}; CI must run one job for each " + "shard (doc/ci.md#microvm-test-shards)" + ) + selected = set(shards[index - 1]) + return tuple(scenario for scenario in suite if scenario in selected) + + +def _report_records(path: Path) -> list[dict[str, object]]: + records: list[dict[str, object]] = [] + for number, line in enumerate(path.read_text(encoding="utf-8").splitlines(), 1): + if not line.strip(): + continue + try: + value: object = json.loads(line, object_pairs_hook=strict_json_object) + except ValueError as error: + raise ScriptError(f"{path}:{number} is not valid JSON: {error}") from error + if not isinstance(value, dict): + raise ScriptError(f"{path}:{number} is not a JSON object") + records.append(cast(dict[str, object], value)) + return records + + +def _report_paths(paths: Iterable[Path]) -> list[Path]: + found: dict[Path, None] = {} + for path in paths: + if path.is_dir(): + for report in sorted(path.rglob(microvm_durations.REPORT_NAME)): + found[report.resolve()] = None + elif path.is_file(): + found[path.resolve()] = None + else: + raise ScriptError(f"{path} is neither a report nor a directory") + return list(found) + + +def read_reports( + paths: Iterable[Path], *, suite: Collection[str], processors: Sequence[int] +) -> tuple[dict[str, dict[str, list[float]]], set[str]]: + """Return each backend's samples of the duration of each scenario of the + default suite, and the CI runs that they come from. + + A sample is one report's time for every execution of a scenario, such as + the four processor counts of console-exit, when none of them failed. Only + the reports of the default suite count: the Alpine guest on the default + kernel with the default processor counts, of a run that ended, failed or + not, since a run that was killed may have recorded part of its last + scenario. + """ + samples: dict[str, dict[str, list[float]]] = {} + runs: set[str] = set() + headers: set[str] = set() + for path in _report_paths(paths): + records = _report_records(path) + if not records: + continue + header = records[0] + if ( + header.get("record") != "run" + or header.get("schema") != microvm_durations.SCHEMA + or header.get("schema_version") != microvm_durations.SCHEMA_VERSION + ): + raise ScriptError( + f"{path} is not a scenario duration report of schema version " + f"{microvm_durations.SCHEMA_VERSION}" + ) + backend = header.get("backend") + if ( + header.get("guest") != "alpine" + or header.get("kernel") != "default" + or header.get("processors") != list(processors) + or not isinstance(backend, str) + or backend not in OPENVMM_TEST_BACKENDS + or not any(record.get("record") == "teardown" for record in records) + ): + continue + # The same report found twice counts once. + identity = json.dumps(header, sort_keys=True) + if identity in headers: + continue + headers.add(identity) + executions: dict[str, list[dict[str, object]]] = {} + for record in records: + scenario = record.get("scenario") + if record.get("record") == "scenario" and scenario in suite: + executions.setdefault(cast(str, scenario), []).append(record) + measured = False + for scenario, records_of_scenario in executions.items(): + durations = [ + _number(record.get("seconds")) for record in records_of_scenario + ] + if any( + record.get("outcome") != "passed" for record in records_of_scenario + ) or any(duration is None for duration in durations): + continue + samples.setdefault(backend, {}).setdefault(scenario, []).append( + sum(cast(list[float], durations)) + ) + measured = True + github = header.get("github") + if measured and isinstance(github, dict): + job = cast(dict[str, object], github) + run_id, attempt = job.get("run_id"), job.get("run_attempt") + if isinstance(run_id, str) and isinstance(attempt, str): + runs.add(f"{run_id}-{attempt}") + return samples, runs + + +def _changes( + scenarios: Sequence[str], shard_of: Mapping[str, int], count: int +) -> Iterator[tuple[tuple[str, int], ...]]: + """Yield every move of a scenario to another shard, and every swap of two + scenarios of different shards, as the scenarios' new shards.""" + for scenario in scenarios: + for shard in range(count): + if shard != shard_of[scenario]: + yield ((scenario, shard),) + for first, second in itertools.combinations(scenarios, 2): + if shard_of[first] != shard_of[second]: + yield ((first, shard_of[second]), (second, shard_of[first])) + + +def _gain(current: Sequence[int], candidate: Sequence[int]) -> int: + """Return by how much a candidate's totals, longest first, undercut the + current ones where they first differ; negative if they exceed them.""" + for before, after in zip(current, candidate, strict=True): + if before != after: + return before - after + return 0 + + +def plan_shards( + seconds: Mapping[str, float], + count: int, + previous: Sequence[Collection[str]] = (), +) -> tuple[tuple[str, ...], ...]: + """Split scenarios of the given durations into `count` balanced shards. + + Each scenario starts in its shard of `previous`, the partition to update; + the others go, longest first, to the shard that takes the least time so + far. A shard that is still empty then takes the longest scenario of the + longest shard that has more than one, since no move into it may shorten + a shard enough, as when its candidates take no time. The plan then moves + a scenario to another shard, or swaps two of different shards, while that + shortens the longest shard by MINIMUM_GAIN_TENTHS or more, comparing the + next longest where the longest stay alike, and takes the best such change + each time; no such change empties a shard. Each shard lists its scenarios + in name order. + """ + tenths = {scenario: _tenths(duration) for scenario, duration in seconds.items()} + scenarios = sorted(tenths) + if not 1 <= count <= len(scenarios): + raise ScriptError( + f"cannot split {len(scenarios)} scenarios into {count} shards" + ) + shard_of: dict[str, int] = {} + for shard, members in enumerate(previous[:count]): + for scenario in sorted(members): + if scenario in tenths and scenario not in shard_of: + shard_of[scenario] = shard + totals = [0] * count + for scenario, shard in shard_of.items(): + totals[shard] += tenths[scenario] + for scenario in sorted( + (scenario for scenario in scenarios if scenario not in shard_of), + key=lambda scenario: (-tenths[scenario], scenario), + ): + shard = min(range(count), key=lambda index: (totals[index], index)) + shard_of[scenario] = shard + totals[shard] += tenths[scenario] + for shard in range(count): + sizes = Counter(shard_of.values()) + if sizes[shard]: + continue + donor = max( + (index for index in range(count) if sizes[index] > 1), + key=lambda index: (totals[index], -index), + ) + scenario = min( + (scenario for scenario in scenarios if shard_of[scenario] == donor), + key=lambda scenario: (-tenths[scenario], scenario), + ) + shard_of[scenario] = shard + totals[donor] -= tenths[scenario] + totals[shard] += tenths[scenario] + while True: + current = sorted(totals, reverse=True) + best: tuple[list[int], tuple[tuple[str, int], ...], list[int]] | None = None + for change in _changes(scenarios, shard_of, count): + trial = list(totals) + for scenario, shard in change: + trial[shard_of[scenario]] -= tenths[scenario] + trial[shard] += tenths[scenario] + candidate = sorted(trial, reverse=True) + if _gain(current, candidate) >= MINIMUM_GAIN_TENTHS and ( + best is None or candidate < best[0] + ): + best = (candidate, change, trial) + if best is None: + break + for scenario, shard in best[1]: + shard_of[scenario] = shard + totals = best[2] + return tuple( + tuple(scenario for scenario in scenarios if shard_of[scenario] == shard) + for shard in range(count) + ) + + +def _run_order(run: str) -> tuple[int, ...]: + return tuple(int(part) if part.isdigit() else 0 for part in run.split("-")) + + +def plan_partition( + previous: Partition | None, + suite: Sequence[str], + *, + samples: Mapping[str, Mapping[str, Sequence[float]]], + runs: Collection[str], + estimates: Mapping[str, float], + counts: Mapping[str, int], +) -> Partition: + """Plan every backend's shards of the default suite. + + A scenario's duration is its estimate, if given, or else the median of + the backend's samples, to a tenth of a second, or else its duration in + `previous`. A backend keeps its number of shards unless `counts` changes + it, and its scenarios start in their shards of `previous`. + """ + backends: dict[str, BackendShards] = {} + missing: list[str] = [] + for backend in OPENVMM_TEST_BACKENDS: + old = previous.backends.get(backend) if previous is not None else None + measured = samples.get(backend, {}) + seconds: dict[str, float] = {} + for scenario in suite: + if scenario in estimates: + seconds[scenario] = round(estimates[scenario], 1) + elif measured.get(scenario): + seconds[scenario] = round(statistics.median(measured[scenario]), 1) + elif old is not None and scenario in old.seconds: + seconds[scenario] = old.seconds[scenario] + else: + missing.append(f"{scenario} on {backend}") + count = counts.get(backend, len(old.shards) if old is not None else 0) + if count == 0: + raise ScriptError( + f"the partition has no {backend} shards; pass --shards {backend}=COUNT" + ) + if missing: + continue + previous_shards = old.shards if old is not None else () + backends[backend] = BackendShards( + seconds, plan_shards(seconds, count, previous_shards) + ) + if missing: + raise ScriptError( + "no report measures, and the partition has no duration of, " + + ", ".join(missing) + + "; pass --report with a report that measures them, or --seconds " + "SCENARIO=SECONDS with an estimate" + ) + planned_runs = ( + tuple(sorted(runs, key=_run_order)) + if samples + else (previous.runs if previous is not None else ()) + ) + return Partition(planned_runs, backends) + + +def partition_json(partition: Partition) -> str: + """Return a partition as its file holds it.""" + document = { + "schema": SCHEMA, + "schema_version": SCHEMA_VERSION, + "runs": list(partition.runs), + "backends": { + backend: { + "seconds": { + scenario: shards.seconds[scenario] + for scenario in sorted(shards.seconds) + }, + "shards": [ + {"seconds": total, "scenarios": list(shard)} + for shard, total in zip( + shards.shards, shards.shard_seconds(), strict=True + ) + ], + } + for backend, shards in partition.backends.items() + }, + } + return json.dumps(document, indent=2) + "\n" + + +def describe_partition(previous: Partition | None, partition: Partition) -> str: + """Return each backend's shards, with their durations and the scenarios + that changed shard.""" + lines: list[str] = [] + for backend, shards in partition.backends.items(): + totals = shards.shard_seconds() + lines.append( + f"{backend}: {len(shards.shards)} shards, " + f"{sum(_tenths(total) for total in totals) / 10:.1f} s in all" + ) + for index, (shard, total) in enumerate( + zip(shards.shards, totals, strict=True), 1 + ): + lines.append(f" shard {index}: {total:.1f} s, {len(shard)} scenarios") + old = previous.backends.get(backend) if previous is not None else None + if old is None: + continue + before = { + scenario: index + for index, shard in enumerate(old.shards, 1) + for scenario in shard + } + for index, shard in enumerate(shards.shards, 1): + for scenario in shard: + if before.get(scenario) != index: + origin = before.get(scenario) + lines.append( + f" {scenario}: " + + (f"shard {origin}" if origin else "new") + + f" -> shard {index}" + ) + return "\n".join(lines) + + +def _estimate(value: str) -> tuple[str, float]: + scenario, separator, seconds = value.partition("=") + try: + duration = float(seconds) + except ValueError: + duration = -1.0 + if not separator or not scenario or not 0 <= duration < math.inf: + raise argparse.ArgumentTypeError("must be SCENARIO=SECONDS, such as smp=20") + return scenario, duration + + +def _shard_count(value: str) -> tuple[str, int]: + backend, separator, count = value.partition("=") + if ( + not separator + or backend not in OPENVMM_TEST_BACKENDS + or not count.isdigit() + or int(count) < 1 + ): + raise argparse.ArgumentTypeError( + "must be BACKEND=COUNT, such as whp=2, for a backend of " + + ", ".join(OPENVMM_TEST_BACKENDS) + ) + return backend, int(count) + + +def configure_parser( + parser: argparse.ArgumentParser, + *, + suite: Callable[[], Sequence[str]], + processors: Sequence[int], +) -> None: + """Configure `plan-microvm-shards` for the default suite that `suite` + returns, with the default processor counts.""" + parser.description = ( + "Plan the shards of the default microVM suite, which CI runs in " + "parallel, from scenario duration reports (doc/ci.md#microvm-test-shards)." + ) + parser.add_argument( + "--report", + type=Path, + action="append", + default=[], + metavar="PATH", + help=( + f"a {microvm_durations.REPORT_NAME} or a directory to search for " + "them, such as one that gh run download filled; repeatable" + ), + ) + parser.add_argument( + "--seconds", + type=_estimate, + action="append", + default=[], + metavar="SCENARIO=SECONDS", + help=( + "use this duration of SCENARIO on every backend, such as an " + "estimate of a scenario that no report measures; repeatable" + ), + ) + parser.add_argument( + "--shards", + type=_shard_count, + action="append", + default=[], + metavar="BACKEND=COUNT", + help=( + "split BACKEND's suite into COUNT shards instead of as many as " + "the partition has; repeatable" + ), + ) + parser.add_argument( + "--partition", + type=Path, + default=PARTITION_PATH, + metavar="PATH", + help="the partition to update (default: the one that CI runs)", + ) + + def handler(args: argparse.Namespace) -> int: + return run(args, suite=suite(), processors=processors) + + parser.set_defaults(handler=handler) + + +def run( + args: argparse.Namespace, *, suite: Sequence[str], processors: Sequence[int] +) -> int: + path: Path = args.partition + estimates = dict(cast(list[tuple[str, float]], args.seconds)) + unknown = sorted(set(estimates) - set(suite)) + if unknown: + raise ScriptError( + "--seconds names scenarios outside the default suite: " + ", ".join(unknown) + ) + previous = load_partition(path) if path.exists() else None + reports = cast(list[Path], args.report) + samples, runs = read_reports(reports, suite=suite, processors=processors) + if reports and not samples: + raise ScriptError( + "no report of the default microVM suite measures a scenario in " + + ", ".join(str(report) for report in reports) + ) + partition = plan_partition( + previous, + suite, + samples=samples, + runs=runs, + estimates=estimates, + counts=dict(cast(list[tuple[str, int]], args.shards)), + ) + check_partition(partition, suite, path) + path.write_text(partition_json(partition), encoding="utf-8", newline="\n") + print(describe_partition(previous, partition)) + print(f"Wrote {path}") + return 0 diff --git a/scripts/nvx_tools/microvm_tests.py b/scripts/nvx_tools/microvm_tests.py index 34a98a301..eaece8403 100644 --- a/scripts/nvx_tools/microvm_tests.py +++ b/scripts/nvx_tools/microvm_tests.py @@ -66,6 +66,8 @@ from .guests import GUEST_NAMES, GuestDescriptor, guest_descriptor from .managed_exec_tests import run_managed_exec_configuration from .microvm_durations import REPORT_NAME, DurationReport +from .microvm_shards import configure_parser as configure_shards_parser +from .microvm_shards import load_partition, parse_shard, shard_scenarios from .nvx_rs import EgressRules, compile_egress_policy from .openvmm_process import OpenvmmProcess, TcpConsole from .sandbox_lifecycle_tests import run_sandbox_lifecycle @@ -356,6 +358,16 @@ ) +def default_suite() -> tuple[str, ...]: + """Return the default suite of the Alpine guest on the default kernel, + which CI runs in the shards of microvm_shards.json.""" + return tuple( + scenario + for scenario in MICROVM_TEST_SCENARIOS + if scenario not in PUBLIC_SANDBOX_SCENARIOS + ) + + def configure_parser(parser: argparse.ArgumentParser) -> None: parser.description = ( "Run NVX-owned Linux and device correctness tests against the public " @@ -386,6 +398,16 @@ def configure_parser(parser: argparse.ArgumentParser) -> None: "scenarios unless --scenario is given" ), ) + parser.add_argument( + "--shard", + type=parse_shard, + metavar="INDEX/COUNT", + help=( + "run shard INDEX of the COUNT shards of the default Alpine suite on " + "the default kernel, as scripts/nvx_tools/microvm_shards.json splits " + "it for the backend, such as 1/2; COUNT must be its number of shards" + ), + ) parser.add_argument( "--processors", type=int, @@ -418,6 +440,13 @@ def configure_parser(parser: argparse.ArgumentParser) -> None: parser.set_defaults(handler=run) +def configure_shard_parser(parser: argparse.ArgumentParser) -> None: + """Configure `plan-microvm-shards` for the default suite.""" + configure_shards_parser( + parser, suite=default_suite, processors=MICROVM_PROCESSOR_COUNTS + ) + + def run_guest_script( command: Sequence[str], script: str, @@ -7519,10 +7548,19 @@ def require_debug_kernel(config: Path) -> None: def run(args: argparse.Namespace) -> int: validate_openvmm_test_backend(args.backend) descriptor = guest_descriptor(args.guest) + debug_kernel = getattr(args, "debug_kernel", False) + shard: tuple[int, int] | None = getattr(args, "shard", None) + if shard is not None and ( + args.scenario is not None or debug_kernel or descriptor.name != "alpine" + ): + raise ScriptError( + "--shard runs a shard of the default Alpine suite on the default " + "kernel, so it takes neither --scenario, --debug-kernel, nor another " + "--guest" + ) if args.memory_mib is None: args.memory_mib = descriptor.default_memory_mib executable = require_file(openvmm_binary_path(), "OpenVMM release binary") - debug_kernel = getattr(args, "debug_kernel", False) if debug_kernel: require_debug_kernel(artifact_path(KernelBuildConstants.DEBUG_CONFIG_NAME)) kernel = require_file( @@ -7545,7 +7583,13 @@ def run(args: argparse.Namespace) -> int: unsupported_scenarios.update(UBUNTU_UNSUPPORTED_SCENARIOS) if not descriptor.sandbox_control: unsupported_scenarios.update(SANDBOX_CONTROL_SCENARIOS) - if args.scenario is None: + if shard is not None: + # Fails unless every default scenario is in exactly one shard of every + # backend, and the backend has as many shards as CI runs. + scenarios = shard_scenarios( + load_partition(), args.backend, shard, default_suite() + ) + elif args.scenario is None: defaults = DEBUG_KERNEL_SCENARIOS if debug_kernel else MICROVM_TEST_SCENARIOS scenarios = tuple( scenario diff --git a/scripts/test_microvm_tests.py b/scripts/test_microvm_tests.py index c5aef3d64..282d9a127 100644 --- a/scripts/test_microvm_tests.py +++ b/scripts/test_microvm_tests.py @@ -36,6 +36,7 @@ common, managed_exec_tests, microvm_durations, + microvm_shards, microvm_tests, nvx_rs, openvmm_process, @@ -10414,5 +10415,689 @@ def test_process_age_counts_from_the_process_start(self): self.assertLess(age, 24 * 3600) +def _write_durations( + path: Path, + backend: str, + executions: list[tuple[str, float, str]], + *, + guest: str = "alpine", + kernel: str = "default", + processors: list[int] | None = None, + run: str | None = "39792079980-1", + teardown: bool = True, +) -> Path: + """Write a scenario duration report of (scenario, seconds, outcome) + executions, as test-microvm writes it.""" + run_id, _, attempt = (run or "").partition("-") + records: list[dict[str, object]] = [ + { + "record": "run", + "schema": microvm_durations.SCHEMA, + "schema_version": microvm_durations.SCHEMA_VERSION, + "started_at": "2026-10-22T14:10:03.792Z", + "guest": guest, + "backend": backend, + "kernel": kernel, + "processors": [1, 2, 4, 8] if processors is None else processors, + "scenarios": list(dict.fromkeys(name for name, _, _ in executions)), + "github": ( + None + if run is None + else {"run_id": run_id, "run_attempt": attempt, "job": "test"} + ), + }, + {"record": "setup", "seconds": 0.25, "outcome": "passed", "error": None}, + *( + { + "record": "scenario", + "scenario": name, + "guest": guest, + "backend": backend, + "kernel": kernel, + "seconds": seconds, + "outcome": outcome, + "error": None, + } + for name, seconds, outcome in executions + ), + ] + if teardown: + records.append( + {"record": "teardown", "seconds": 0.1, "outcome": "passed", "error": None} + ) + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text( + "".join(json.dumps(record) + "\n" for record in records), encoding="utf-8" + ) + return path + + +class MicrovmShardTests(unittest.TestCase): + """The checked-in partition of the default suite into the shards that CI + runs in parallel, the planner that writes it, and test-microvm --shard + (#696).""" + + def _replace( + self, backend: str, shards: tuple[tuple[str, ...], ...] + ) -> microvm_shards.Partition: + """Return the checked-in partition with other shards of a backend.""" + partition = microvm_shards.load_partition() + old = partition.backends[backend] + return microvm_shards.Partition( + partition.runs, + { + **partition.backends, + backend: microvm_shards.BackendShards(old.seconds, shards), + }, + ) + + def _run_shard( + self, + output: Path, + *options: str, + partition: microvm_shards.Partition | None = None, + ) -> list[str]: + """Run test-microvm with every scenario stubbed, and return the + scenarios that its report times.""" + args = nvx.parse_args(["test-microvm", *options, "--output-dir", str(output)]) + + def require(path: Path, _description: str) -> Path: + return path + + with ExitStack() as stack: + stack.enter_context( + patch.object(microvm_tests, "validate_openvmm_test_backend") + ) + stack.enter_context( + patch.object(microvm_tests, "require_file", side_effect=require) + ) + runners = [ + stack.enter_context(patch.object(microvm_tests, name)) + for name in _scenario_runners() + ] + if partition is not None: + stack.enter_context( + patch.object( + microvm_tests, "load_partition", return_value=partition + ) + ) + stack.enter_context(redirect_stdout(io.StringIO())) + self.assertEqual(microvm_tests.run(args), 0) + self.assertTrue(any(runner.called for runner in runners)) + records = _read_report(output / microvm_durations.REPORT_NAME) + executed = { + record["scenario"] for record in records if record["record"] == "scenario" + } + self.assertEqual(executed, set(records[0]["scenarios"])) + return records[0]["scenarios"] + + def test_partition_puts_each_default_scenario_in_one_shard_of_each_backend( + self, + ): + partition = microvm_shards.load_partition() + suite = microvm_tests.default_suite() + self.assertEqual(microvm_shards.partition_problems(partition, suite), []) + self.assertEqual(tuple(partition.backends), microvm_tests.OPENVMM_TEST_BACKENDS) + self.assertTrue(partition.runs) + for backend, shards in partition.backends.items(): + with self.subTest(backend=backend): + self.assertGreaterEqual(len(shards.shards), 2) + self.assertTrue(all(shards.shards)) + self.assertEqual( + sorted(scenario for shard in shards.shards for scenario in shard), + sorted(suite), + ) + self.assertEqual(set(shards.seconds), set(suite)) + + def test_partition_is_the_balanced_plan_of_its_durations(self): + # The file is the planner's output for its own durations: no move or + # swap shortens a longest shard by MINIMUM_GAIN_TENTHS, and each + # shard's duration is the sum of its scenarios'. + partition = microvm_shards.load_partition() + self.assertEqual( + microvm_shards.PARTITION_PATH.read_text(encoding="utf-8"), + microvm_shards.partition_json(partition), + ) + self.assertEqual( + microvm_shards.plan_partition( + partition, + microvm_tests.default_suite(), + samples={}, + runs=(), + estimates={}, + counts={}, + ), + partition, + ) + + def test_default_suite_is_the_suite_that_a_default_run_selects(self): + with tempfile.TemporaryDirectory() as temporary: + selected = self._run_shard(Path(temporary), "--backend", "kvm") + self.assertEqual(selected, list(microvm_tests.default_suite())) + self.assertNotIn("managed-exec-config", selected) + self.assertNotIn("smp-lapic", selected) + + def test_shards_run_each_default_scenario_once_on_each_backend(self): + partition = microvm_shards.load_partition() + suite = microvm_tests.default_suite() + for backend, shards in partition.backends.items(): + ran: list[str] = [] + for index, shard in enumerate(shards.shards, 1): + spec = f"{index}/{len(shards.shards)}" + with ( + self.subTest(backend=backend, shard=spec), + tempfile.TemporaryDirectory() as temporary, + ): + selected = self._run_shard( + Path(temporary), "--backend", backend, "--shard", spec + ) + # In suite order, as an unsharded run orders them. + self.assertEqual( + selected, [scenario for scenario in suite if scenario in shard] + ) + ran.extend(selected) + with self.subTest(backend=backend): + self.assertEqual(sorted(ran), sorted(suite)) + + def test_shard_fails_before_any_scenario_unless_each_is_in_one_shard(self): + partition = microvm_shards.load_partition() + first, second, *_ = partition.backends["kvm"].shards + scenarios = microvm_tests.MICROVM_TEST_SCENARIOS + cases: list[tuple[str, dict[str, Any], str, str]] = [ + # A new default scenario that no shard runs. + ( + "new scenario", + {"scenarios": (*scenarios, "brand-new")}, + "1/2", + r"kvm: brand-new is in no shard; .*whp: brand-new is in no shard", + ), + # A scenario that a change removes but a shard still lists. + ( + "removed scenario", + {"scenarios": tuple(s for s in scenarios if s != "lifecycle")}, + "1/2", + r"kvm: lifecycle is not a default scenario", + ), + ( + "scenario in two shards", + {"partition": self._replace("kvm", (first, (*second, first[0])))}, + "2/2", + rf"kvm: {first[0]} is in more than one shard: 1, 2", + ), + ( + "scenario in no shard", + {"partition": self._replace("kvm", (first[1:], second))}, + "1/2", + rf"kvm: {first[0]} is in no shard", + ), + ( + "empty shard", + {"partition": self._replace("kvm", (first, second, ()))}, + "1/3", + r"kvm shard 3 is empty", + ), + # CI runs fewer or more jobs than the backend has shards. + ("fewer jobs", {}, "1/1", r"splits the kvm suite into 2 shards, but"), + ("more jobs", {}, "3/3", r"this run is shard 3 of 3"), + ] + + def require(path: Path, _description: str) -> Path: + return path + + for name, changes, spec, message in cases: + with ( + self.subTest(case=name), + tempfile.TemporaryDirectory() as temporary, + ExitStack() as stack, + ): + if "scenarios" in changes: + stack.enter_context( + patch.object( + microvm_tests, + "MICROVM_TEST_SCENARIOS", + changes["scenarios"], + ) + ) + if "partition" in changes: + stack.enter_context( + patch.object( + microvm_tests, + "load_partition", + return_value=changes["partition"], + ) + ) + stack.enter_context( + patch.object(microvm_tests, "validate_openvmm_test_backend") + ) + stack.enter_context( + patch.object(microvm_tests, "require_file", side_effect=require) + ) + runners = [ + stack.enter_context(patch.object(microvm_tests, runner)) + for runner in _scenario_runners() + ] + args = nvx.parse_args( + [ + "test-microvm", + "--backend", + "kvm", + "--shard", + spec, + "--output-dir", + temporary, + ] + ) + with self.assertRaisesRegex(common.ScriptError, message) as raised: + microvm_tests.run(args) + self.assertFalse(any(runner.called for runner in runners)) + self.assertFalse( + (Path(temporary) / microvm_durations.REPORT_NAME).exists() + ) + if spec != "1/1" and spec != "3/3": + self.assertIn("plan-microvm-shards", str(raised.exception)) + + def test_shard_takes_only_the_default_alpine_suite(self): + for options in ( + ("--scenario", "smp"), + ("--debug-kernel",), + ("--guest", "ubuntu"), + ): + with self.subTest(options=options): + args = nvx.parse_args( + ["test-microvm", "--backend", "kvm", "--shard", "1/2", *options] + ) + with ( + patch.object(microvm_tests, "validate_openvmm_test_backend"), + patch.object(microvm_tests, "require_file") as require, + self.assertRaisesRegex( + common.ScriptError, "--shard runs a shard of the default" + ), + ): + microvm_tests.run(args) + require.assert_not_called() + self.assertEqual( + nvx.parse_args( + ["test-microvm", "--backend", "whp", "--shard", "2/2"] + ).shard, + (2, 2), + ) + self.assertIsNone(nvx.parse_args(["test-microvm", "--backend", "whp"]).shard) + for value in ("0/2", "3/2", "1", "1/0", "a/b", "1/2/3", "", "01/2"): + with ( + self.subTest(value=value), + self.assertRaises(SystemExit), + patch("sys.stderr", new_callable=io.StringIO), + ): + nvx.parse_args(["test-microvm", "--backend", "kvm", "--shard", value]) + + def test_plan_balances_the_shards_of_the_previous_partition(self): + seconds = {"a": 10.0, "b": 9.0, "c": 2.0, "d": 1.0} + # The best change swaps a and c, which ties with swapping b and d. + self.assertEqual( + microvm_shards.plan_shards(seconds, 2, [("a", "b"), ("c", "d")]), + (("b", "c"), ("a", "d")), + ) + # The order in which durations and shards list scenarios is no input. + self.assertEqual( + microvm_shards.plan_shards( + dict(reversed(seconds.items())), 2, [("b", "a"), ("d", "c")] + ), + (("b", "c"), ("a", "d")), + ) + + def test_plan_keeps_shards_that_no_change_shortens_by_a_second(self): + # Moving c shortens the longest shard by 0.8 s only. + self.assertEqual( + microvm_shards.plan_shards( + {"a": 10.0, "b": 9.0, "c": 0.8}, 2, [("a", "c"), ("b",)] + ), + (("a", "c"), ("b",)), + ) + # By 1 s, it moves; the move comes before the swap of a and b that + # balances the shards as well. + self.assertEqual( + microvm_shards.plan_shards( + {"a": 10.0, "b": 9.0, "c": 1.2}, 2, [("a", "c"), ("b",)] + ), + (("a",), ("b", "c")), + ) + + def test_plan_places_scenarios_without_a_shard_longest_first(self): + # x is no longer a scenario; c, d, and e have no shard yet. + self.assertEqual( + microvm_shards.plan_shards( + {"a": 5.0, "b": 4.0, "c": 3.0, "d": 3.0, "e": 1.0}, + 2, + [("x", "a"), ("b",)], + ), + (("a", "d"), ("b", "c", "e")), + ) + # A new plan starts from no shards at all. + self.assertEqual( + microvm_shards.plan_shards({"a": 5.0, "b": 4.0, "c": 3.0, "d": 2.0}, 2), + (("a", "d"), ("b", "c")), + ) + + def test_plan_fills_added_shards_and_rejects_more_than_scenarios(self): + seconds = {"a": 3.0, "b": 3.0, "c": 3.0} + self.assertEqual( + microvm_shards.plan_shards(seconds, 3, [("a", "b", "c")]), + (("c",), ("a",), ("b",)), + ) + self.assertEqual( + microvm_shards.plan_shards(seconds, 1, [("a",), ("b",), ("c",)]), + (("a", "b", "c"),), + ) + for count in (0, 4): + with ( + self.subTest(count=count), + self.assertRaisesRegex(common.ScriptError, "cannot split 3 scenarios"), + ): + microvm_shards.plan_shards(seconds, count) + + def test_plan_fills_every_shard_where_no_move_shortens_one_enough(self): + # No move of a scenario that takes no time, or less than a second, + # into an empty shard shortens a shard by a second, yet each shard + # takes one: the longest of the longest shard that has more than one. + self.assertEqual( + microvm_shards.plan_shards( + {"a": 5.0, "b": 0.5, "c": 0.0, "d": 0.3}, 4, [("a", "b", "c", "d")] + ), + (("c",), ("a",), ("b",), ("d",)), + ) + # As many shards as the checked-in partition has scenarios. + partition = microvm_shards.load_partition() + suite = microvm_tests.default_suite() + planned = microvm_shards.plan_partition( + partition, + suite, + samples={}, + runs=(), + estimates={}, + counts=dict.fromkeys(partition.backends, len(suite)), + ) + self.assertEqual(microvm_shards.partition_problems(planned, suite), []) + for backend, shards in planned.backends.items(): + with self.subTest(backend=backend): + self.assertEqual( + [len(shard) for shard in shards.shards], [1] * len(suite) + ) + + def test_reports_give_samples_of_the_default_suite_only(self): + suite = ("console-exit", "smp", "lifecycle") + console_exit = [("console-exit", 2.0, "passed")] * 4 + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + first = _write_durations( + root / "1" / microvm_durations.REPORT_NAME, + "kvm", + [*console_exit, ("smp", 5.0, "passed"), ("lifecycle", 3.0, "failed")], + run="1-1", + ) + _write_durations( + root / "2" / "shard" / microvm_durations.REPORT_NAME, + "kvm", + [ + ("console-exit", 3.0, "passed"), + ("console-exit", 3.0, "passed"), + ("console-exit", 3.0, "passed"), + ("console-exit", 3.0, "passed"), + ("smp", 7.0, "passed"), + ("lifecycle", 1.5, "passed"), + ], + run="2-1", + ) + # A console-exit that failed at its second processor count. + _write_durations( + root / "3" / microvm_durations.REPORT_NAME, + "kvm", + [("console-exit", 2.0, "passed"), ("console-exit", 9.0, "failed")], + run="3-1", + ) + # Reports of other runs than the default suite's, and of a run + # that was killed. + smp = [("smp", 99.0, "passed")] + _write_durations( + root / "debug" / microvm_durations.REPORT_NAME, + "kvm", + smp, + kernel="debug", + ) + _write_durations( + root / "ubuntu" / microvm_durations.REPORT_NAME, + "kvm", + smp, + guest="ubuntu", + ) + _write_durations( + root / "one-processor" / microvm_durations.REPORT_NAME, + "kvm", + smp, + processors=[1], + ) + _write_durations( + root / "killed" / microvm_durations.REPORT_NAME, + "kvm", + smp, + teardown=False, + ) + _write_durations( + root / "whp" / microvm_durations.REPORT_NAME, + "whp", + [("smp", 9.0, "passed"), ("managed-exec-config", 5.0, "passed")], + run=None, + ) + samples, runs = microvm_shards.read_reports( + # The same report, found twice, counts once. + [root, first], + suite=suite, + processors=(1, 2, 4, 8), + ) + self.assertEqual( + { + backend: {name: sorted(values) for name, values in found.items()} + for backend, found in samples.items() + }, + { + "kvm": { + "console-exit": [8.0, 12.0], + "smp": [5.0, 7.0], + "lifecycle": [1.5], + }, + "whp": {"smp": [9.0]}, + }, + ) + self.assertEqual(runs, {"1-1", "2-1"}) + + (root / "other").mkdir() + (root / "other" / microvm_durations.REPORT_NAME).write_text( + '{"record":"run","schema":"other","schema_version":1}\n', + encoding="utf-8", + ) + with self.assertRaisesRegex(common.ScriptError, "is not a scenario"): + microvm_shards.read_reports([root], suite=suite, processors=(1,)) + with self.assertRaisesRegex(common.ScriptError, "neither a report"): + microvm_shards.read_reports( + [root / "missing"], suite=suite, processors=(1,) + ) + + def test_plan_command_writes_and_updates_a_partition(self): + scenarios = ("console-exit", "smp", "lifecycle", "managed-exec-config") + + def plan(*options: str) -> str: + args = nvx.parse_args( + ["plan-microvm-shards", "--partition", str(path), *options] + ) + with redirect_stdout(io.StringIO()) as stdout: + self.assertEqual(args.handler(args), 0) + return stdout.getvalue() + + with ( + tempfile.TemporaryDirectory() as temporary, + patch.object(microvm_tests, "MICROVM_TEST_SCENARIOS", scenarios), + ): + root = Path(temporary) + path = root / "shards.json" + reports = root / "reports" + for run, factor in (("7-1", 1.0), ("8-2", 1.5), ("9-1", 2.0)): + for backend in ("kvm", "mshv", "whp"): + _write_durations( + reports / run / backend / microvm_durations.REPORT_NAME, + backend, + [ + *[("console-exit", 2.0 * factor, "passed")] * 4, + ("smp", 6.0 * factor, "passed"), + ("lifecycle", 1.0 * factor, "passed"), + ], + run=run, + ) + with self.assertRaisesRegex( + common.ScriptError, "the partition has no kvm shards" + ): + plan("--report", str(reports)) + output = plan( + "--report", + str(reports), + "--shards", + "kvm=2", + "--shards", + "mshv=1", + "--shards", + "whp=2", + ) + self.assertIn("kvm: 2 shards, 22.5 s in all", output) + text = path.read_text(encoding="utf-8") + self.assertNotIn("\r", text) + partition = microvm_shards.load_partition(path) + self.assertEqual(partition.runs, ("7-1", "8-2", "9-1")) + for backend, count in (("kvm", 2), ("mshv", 1), ("whp", 2)): + shards = partition.backends[backend] + self.assertEqual( + dict(shards.seconds), + {"console-exit": 12.0, "smp": 9.0, "lifecycle": 1.5}, + ) + self.assertEqual(len(shards.shards), count) + self.assertEqual( + partition.backends["kvm"].shards, + (("console-exit",), ("lifecycle", "smp")), + ) + self.assertEqual(partition.backends["kvm"].shard_seconds(), (12.0, 10.5)) + + # Without reports, the plan keeps the durations, runs, and shards. + plan() + self.assertEqual(path.read_text(encoding="utf-8"), text) + # An estimate replaces a duration on every backend. + output = plan("--seconds", "smp=20") + partition = microvm_shards.load_partition(path) + self.assertEqual( + { + backend: shards.seconds["smp"] + for backend, shards in partition.backends.items() + }, + {"kvm": 20.0, "mshv": 20.0, "whp": 20.0}, + ) + self.assertEqual( + partition.backends["kvm"].shards, + (("console-exit", "lifecycle"), ("smp",)), + ) + self.assertIn(" lifecycle: shard 2 -> shard 1", output) + + # A scenario that nothing measures needs an estimate. + with patch.object( + microvm_tests, "MICROVM_TEST_SCENARIOS", (*scenarios, "brand-new") + ): + with self.assertRaisesRegex( + common.ScriptError, + "no report measures, and the partition has no duration of, " + "brand-new on kvm, brand-new on mshv, brand-new on whp", + ): + plan() + output = plan("--seconds", "brand-new=4") + self.assertIn(" brand-new: new -> shard", output) + for options, message in ( + (("--seconds", "unknown=1"), "outside the default suite: unknown"), + (("--report", str(root / "empty")), "neither a report"), + ): + with ( + self.subTest(options=options), + self.assertRaisesRegex(common.ScriptError, message), + ): + plan(*options) + (root / "empty").mkdir() + with self.assertRaisesRegex(common.ScriptError, "no report of the default"): + plan("--report", str(root / "empty")) + for value in ("kvm", "kvm=0", "arm=2", "kvm=x"): + with ( + self.subTest(shards=value), + self.assertRaises(SystemExit), + patch("sys.stderr", new_callable=io.StringIO), + ): + nvx.parse_args(["plan-microvm-shards", "--shards", value]) + for value in ("smp", "smp=-1", "=2", "smp=x", "smp=inf", "smp=nan"): + with ( + self.subTest(seconds=value), + self.assertRaises(SystemExit), + patch("sys.stderr", new_callable=io.StringIO), + ): + nvx.parse_args(["plan-microvm-shards", "--seconds", value]) + + def test_partition_rejects_malformed_files(self): + valid = json.loads(microvm_shards.PARTITION_PATH.read_text(encoding="utf-8")) + cases: list[tuple[str, str, str]] = [ + ("not JSON", "{", "is not valid JSON"), + ( + "duplicate key", + '{"schema": 1, "schema": 2}', + "duplicate JSON property: schema", + ), + ( + "other schema", + json.dumps({**valid, "schema_version": 2}), + "schema version 1", + ), + ( + "malformed shard", + json.dumps( + { + **valid, + "backends": { + **valid["backends"], + "kvm": { + **valid["backends"]["kvm"], + "shards": [{"seconds": 1, "scenarios": [1]}], + }, + }, + } + ), + "malformed partition of kvm", + ), + ( + "negative duration", + json.dumps( + { + **valid, + "backends": { + **valid["backends"], + "whp": { + **valid["backends"]["whp"], + "seconds": {"smp": -1}, + }, + }, + } + ), + "malformed partition of whp", + ), + ] + with tempfile.TemporaryDirectory() as temporary: + path = Path(temporary) / "shards.json" + with self.assertRaisesRegex(common.ScriptError, "does not exist"): + microvm_shards.load_partition(path) + for name, text, message in cases: + with self.subTest(case=name): + path.write_text(text, encoding="utf-8") + with self.assertRaisesRegex(common.ScriptError, message): + microvm_shards.load_partition(path) + + if __name__ == "__main__": unittest.main() diff --git a/scripts/test_nvx_tools.py b/scripts/test_nvx_tools.py index 3d7f105cb..7181c43d0 100644 --- a/scripts/test_nvx_tools.py +++ b/scripts/test_nvx_tools.py @@ -5468,6 +5468,16 @@ def select(*outputs: str) -> ci.CiChanges: microvm_only, microvm_and_benchmarks, ), + ( + ("scripts/nvx_tools/microvm_shards.py",), + microvm_only, + microvm_and_benchmarks, + ), + ( + ("scripts/nvx_tools/microvm_shards.json",), + microvm_only, + microvm_and_benchmarks, + ), (("scripts/nvx_tools/sandbox.py",), microvm_only, microvm_and_benchmarks), ( ("scripts/nvx_tools/microvm_test_scripts/lifecycle.sh",), @@ -5907,6 +5917,8 @@ def job_inputs( { "scripts/nvx_tools/microvm_tests.py", "scripts/nvx_tools/microvm_test_scripts/example", + "scripts/nvx_tools/microvm_shards.py", + "scripts/nvx_tools/microvm_shards.json", "scripts/nvx_tools/doctor.py", "scripts/nvx_tools/host_time_probe.rs", "scripts/nvx_tools/benchmark_scripts/example", @@ -5932,6 +5944,7 @@ def job_inputs( self.assertNotIn(".github/actions/validate-nvx/action.yml", inputs) self.assertNotIn("scripts/nvx_tools/performance.py", microvm_inputs) self.assertNotIn("scripts/nvx_tools/microvm_tests.py", benchmark_inputs) + self.assertNotIn("scripts/nvx_tools/microvm_shards.json", benchmark_inputs) for pattern, inputs in ( (ci.OPENVMM_UNIT_TEST_INPUT_PATHS, unit_inputs), (ci.OPENVMM_VMM_TEST_INPUT_PATHS, vmm_inputs), From 000b1a1d7e50d34d426663a3572ef4f96959511f Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 22 Oct 2026 12:26:41 -0300 Subject: [PATCH 2/2] ci: shard the default microVM suite by scenario duration Each backend ran its default microVM suite in one job, which took 8.1 to 8.6 minutes on KVM, 7.6 to 7.7 on MSHV, and 10.4 to 11.1 on WHP in the first runs after #695 moved the guest and sandbox steps beside it. The WHP job bounds every run that skips both OpenVMM suites, which #430 wants done within 15 minutes, and it starts only when the Windows NVX CLI job ends, about 5.5 minutes into a run. The default-suite job of each backend is now a matrix of two shards, as scripts/nvx_tools/microvm_shards.json splits its suite, and each leg passes test-microvm --shard its matrix.shard and the matrix's strategy.job-total. A new shard input of run-nvx-microvm-tests.yml, with shard-count, selects the shard, and only the default-suite steps pass them on; every other step and job runs as before. A leg fails when the partition has another number of shards for its backend than the matrix runs, so the legs cannot leave a shard out. fail-fast is off, so a failing shard leaves the others to finish and keep their logs. Since a second upload of an artifact name in a run fails, each shard uploads its failure logs as nvx-microvm-tests--shard- and its duration report with -shard- after the kernel. GitHub expands the matrix only once the job's condition holds, so the shards of a backend run together or not at all, under the same fork, draft, path-filter, and cancellation conditions as before, and Re-run failed jobs reruns the failed shards alone. The job keeps its name in the needs of Required status check, Publish development release, and Persist performance baseline, which read its result, and GitHub reports that result as a success only when every leg succeeded. A failed or cancelled shard therefore fails all three closed, as a failed leg of the openvmm-unit-tests or openvmm-vmm-tests-linux matrix does. Two shards per backend is the count at which a shard, about 0.7 minutes of setup and uploads plus half the suite, takes about as long as the guest and sandbox job beside it. In this change's own runs a shard took 4.4 to 4.5 minutes on KVM, 4.2 to 4.9 on MSHV, and 5.7 to 6.4 on WHP, and the guest and sandbox jobs took 3.7 to 4.1, 2.9 to 4.1, and 5.2 to 6.4 since #695. A third shard would end before that job on WHP, and on KVM and MSHV, whose four general runners the two shards, the guest and sandbox job, and the debug-kernel job fill, it would wait for a runner. The debug-kernel jobs, three to five minutes each, stay unsharded and keep their platform rules. A new test checks that each backend's matrix runs exactly the shards of its partition with fail-fast off, that the guest and sandbox and debug-kernel jobs pass no shard, and that the legs' test-microvm commands, parsed with nvx.py's parser, run every default scenario of the backend exactly once, on the Alpine guest and the default kernel with every processor count. The guest and sandbox test now compares that job with the suite's job apart from its shards, and it and the scenario duration test check that every leg of every microVM job uploads its failure logs and its report under a name of its own. The guest artifact test checks the downloads of each leg. doc/ci.md describes the shard jobs, their artifacts and results, the runners that they hold, and how to change their number. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .github/workflows/ci.yml | 38 ++- .github/workflows/run-nvx-microvm-tests.yml | 25 +- doc/ci.md | 84 +++-- doc/usage.md | 9 +- scripts/test_nvx_tools.py | 351 +++++++++++++++----- 5 files changed, 400 insertions(+), 107 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index b6311631c..4790823ea 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -1237,8 +1237,16 @@ jobs: ${{ format('{0}/cargo-home/git', runner.temp) }} key: openvmm-cargo-v1-${{ runner.os }}-${{ runner.arch }}-unit-tests-${{ hashFiles('openvmm/Cargo.lock') }} + # Each backend runs its default suite in parallel shards of about equal + # duration, one leg of this matrix each, as + # scripts/nvx_tools/microvm_shards.json splits it + # (doc/ci.md#microvm-test-shards). A shard fails unless that file puts every + # default scenario in exactly one shard of every backend and splits this + # backend's suite into as many shards as the matrix runs. The job's result, + # which the required status check, the release, and the performance + # persistence read, succeeds only when every shard does. nvx-microvm-tests-kvm: - name: NVX microVM tests / Linux / KVM + name: NVX microVM tests / Linux / KVM / Shard ${{ matrix.shard }} of ${{ strategy.job-total }} needs: [artifacts, build-openvmm-linux-gnu, nvx-cli-linux, openvmm-changes] if: >- ${{ @@ -1260,18 +1268,24 @@ jobs: needs.openvmm-changes.outputs.run-microvm-tests == 'true' ) }} + strategy: + fail-fast: false + matrix: + shard: [1, 2] uses: ./.github/workflows/run-nvx-microvm-tests.yml with: - display-name: Linux / KVM + display-name: Linux / KVM / Shard ${{ matrix.shard }} of ${{ strategy.job-total }} runner: '["self-hosted", "linux", "kvm", "virtual-machine", "general"]' backend: kvm openvmm-artifact: openvmm-linux-gnu nvx-rs-artifact: nvx-rs-linux-musl + shard: ${{ matrix.shard }} + shard-count: ${{ strategy.job-total }} secrets: OPENVMM_DEPLOY_KEY: ${{ secrets.OPENVMM_DEPLOY_KEY }} nvx-microvm-tests-mshv: - name: NVX microVM tests / Linux / MSHV + name: NVX microVM tests / Linux / MSHV / Shard ${{ matrix.shard }} of ${{ strategy.job-total }} needs: [artifacts, build-openvmm-linux-gnu, nvx-cli-linux, openvmm-changes] if: >- ${{ @@ -1293,18 +1307,24 @@ jobs: needs.openvmm-changes.outputs.run-microvm-tests == 'true' ) }} + strategy: + fail-fast: false + matrix: + shard: [1, 2] uses: ./.github/workflows/run-nvx-microvm-tests.yml with: - display-name: Linux / MSHV + display-name: Linux / MSHV / Shard ${{ matrix.shard }} of ${{ strategy.job-total }} runner: '["self-hosted", "linux", "mshv", "virtual-machine", "general"]' backend: mshv openvmm-artifact: openvmm-linux-gnu nvx-rs-artifact: nvx-rs-linux-musl + shard: ${{ matrix.shard }} + shard-count: ${{ strategy.job-total }} secrets: OPENVMM_DEPLOY_KEY: ${{ secrets.OPENVMM_DEPLOY_KEY }} nvx-microvm-tests-whp: - name: NVX microVM tests / Windows / WHP + name: NVX microVM tests / Windows / WHP / Shard ${{ matrix.shard }} of ${{ strategy.job-total }} needs: [artifacts, build-openvmm-windows-msvc, nvx-cli-windows, openvmm-changes] if: >- ${{ @@ -1326,13 +1346,19 @@ jobs: needs.openvmm-changes.outputs.run-microvm-tests == 'true' ) }} + strategy: + fail-fast: false + matrix: + shard: [1, 2] uses: ./.github/workflows/run-nvx-microvm-tests.yml with: - display-name: Windows / WHP + display-name: Windows / WHP / Shard ${{ matrix.shard }} of ${{ strategy.job-total }} runner: '["self-hosted", "windows", "whp", "virtual-machine", "general"]' backend: whp openvmm-artifact: openvmm-windows-msvc nvx-rs-artifact: nvx-rs-windows-msvc + shard: ${{ matrix.shard }} + shard-count: ${{ strategy.job-total }} secrets: OPENVMM_DEPLOY_KEY: ${{ secrets.OPENVMM_DEPLOY_KEY }} diff --git a/.github/workflows/run-nvx-microvm-tests.yml b/.github/workflows/run-nvx-microvm-tests.yml index e66f20d56..e79a811ee 100644 --- a/.github/workflows/run-nvx-microvm-tests.yml +++ b/.github/workflows/run-nvx-microvm-tests.yml @@ -42,6 +42,22 @@ on: required: false type: boolean default: false + shard: + description: >- + Run only this shard, from 1 to shard-count, of the default suite, + as scripts/nvx_tools/microvm_shards.json splits it for the backend + (doc/ci.md#microvm-test-shards); 0 runs all of it. Only for the + default-suite jobs + required: false + type: number + default: 0 + shard-count: + description: >- + How many shards of the default suite the caller runs, one job each; + test-microvm fails unless the partition has as many for the backend + required: false + type: number + default: 0 secrets: OPENVMM_DEPLOY_KEY: description: SSH deploy key used to checkout OpenVMM @@ -166,6 +182,7 @@ jobs: python3 scripts/nvx.py test-microvm --backend "${{ inputs.backend }}" ${{ inputs.debug-kernel && '--debug-kernel' || '' }} + ${{ inputs.shard && format('--shard {0}/{1}', inputs.shard, inputs.shard-count) || '' }} --output-dir "build/test-results/microvm-${{ inputs.backend }}${{ inputs.debug-kernel && '-debug' || '' }}" - name: Run NVX microVM tests on Windows @@ -175,6 +192,7 @@ jobs: python scripts\nvx.py test-microvm --backend "${{ inputs.backend }}" ${{ inputs.debug-kernel && '--debug-kernel' || '' }} + ${{ inputs.shard && format('--shard {0}/{1}', inputs.shard, inputs.shard-count) || '' }} --output-dir "build/test-results/microvm-${{ inputs.backend }}${{ inputs.debug-kernel && '-debug' || '' }}" - name: Run public managed sandbox acceptance on Linux @@ -1254,7 +1272,7 @@ jobs: if: failure() uses: actions/upload-artifact@v7 with: - name: nvx-microvm-tests-${{ inputs.backend }}${{ inputs.debug-kernel && '-debug' || '' }}${{ inputs.guests-and-sandboxes && '-guests' || '' }} + name: nvx-microvm-tests-${{ inputs.backend }}${{ inputs.debug-kernel && '-debug' || '' }}${{ inputs.guests-and-sandboxes && '-guests' || '' }}${{ inputs.shard && format('-shard-{0}', inputs.shard) || '' }} path: | build/test-results/public-exec-${{ inputs.backend }} build/test-results/microvm-${{ inputs.backend }} @@ -1267,12 +1285,13 @@ jobs: # Each job and attempt keeps the scenario duration report of every # test-microvm step (doc/usage.md#scenario-duration-report), whether the - # job passes, fails, or is cancelled. + # job passes, fails, or is cancelled; each shard of the default suite + # keeps its own. - name: Upload NVX microVM scenario durations if: always() uses: actions/upload-artifact@v7 with: - name: nvx-microvm-durations-${{ runner.os }}-${{ inputs.backend }}-${{ inputs.debug-kernel && 'debug' || 'default' }}${{ inputs.guests-and-sandboxes && '-guests' || '' }}-${{ github.run_id }}-${{ github.run_attempt }} + name: nvx-microvm-durations-${{ runner.os }}-${{ inputs.backend }}-${{ inputs.debug-kernel && 'debug' || 'default' }}${{ inputs.guests-and-sandboxes && '-guests' || '' }}${{ inputs.shard && format('-shard-{0}', inputs.shard) || '' }}-${{ github.run_id }}-${{ github.run_attempt }} path: build/test-results/*/scenario-durations.jsonl if-no-files-found: warn retention-days: 30 diff --git a/doc/ci.md b/doc/ci.md index 9bbd94f34..e0dd1310c 100644 --- a/doc/ci.md +++ b/doc/ci.md @@ -37,7 +37,9 @@ behavior through the public OpenVMM CLI, on the NVX Linux kernel plus the selected Alpine, Ubuntu, or Azure Linux initramfs. Both run on the backend's `general` runners, with the same OpenVMM and `nvx-rs` builds. The `nvx-microvm-tests-{kvm,mshv,whp}` jobs run the default suite on the Alpine -initramfs. Beside them, the `nvx-microvm-guests-{kvm,mshv,whp}` jobs run the +initramfs, each in shards of about equal duration that run in parallel, one +runner each (see [MicroVM test shards](#microvm-test-shards)). Beside them, +the `nvx-microvm-guests-{kvm,mshv,whp}` jobs run the guest smoke and sandbox steps, which need no result of that suite: the public managed sandbox acceptance, the Ubuntu guest tests, the one-vCPU Azure Linux smoke set, the Ubuntu sandbox steps, the `aci_edge_sandboxes` lifecycle, and @@ -47,16 +49,18 @@ Linux artifacts, so only the guest and sandbox jobs wait for them (see same time ABI checks. Alpine-control-only scenarios remain explicit and are rejected for the Ubuntu initramfs. The debug-kernel jobs skip the Ubuntu and Azure Linux tests. Each job uploads its failure logs under a name of its own: -`nvx-microvm-tests-` for the default suite, +`nvx-microvm-tests--shard-` for shard `` of the default suite, `nvx-microvm-tests--guests` for the guest and sandbox steps, and `nvx-microvm-tests--debug` for the debug kernel. -Every microVM job, the guest and sandbox and debug-kernel ones too, also -uploads the [scenario duration report](usage.md#scenario-duration-report) of -each `test-microvm` step, whether the job passes, fails, or is cancelled. The +Every microVM job, the guest and sandbox and debug-kernel ones and each shard +too, also uploads the +[scenario duration report](usage.md#scenario-duration-report) of each +`test-microvm` step, whether the job passes, fails, or is cancelled. The 30-day artifact is named -`nvx-microvm-durations---[-guests]--`, -where `` is `default` or `debug` and the guest and sandbox jobs add -`-guests`, so that each job and attempt keeps its own. +`nvx-microvm-durations---[-guests|-shard-]--`, +where `` is `default` or `debug`, the guest and sandbox jobs add +`-guests`, and the shards `-shard-`, so that each job and attempt keeps its +own. Every harness launch, in the tests and the benchmarks, scans the OpenVMM console for the guest's [time ABI](design/3.2-clock-and-time.md) output, and the tests scan every virtio console they read over TCP the same way, to its end. An @@ -449,7 +453,8 @@ lane to pass. The microVM correctness jobs gate every use of benchmark results, so the benchmark action runs no correctness scenario of its own (#286). `Required status check` requires each `nvx-microvm-tests-*` and `nvx-microvm-guests-*` -job that the change schedules to succeed. `Publish development release` and +job that the change schedules to succeed, and an `nvx-microvm-tests-*` job +succeeds only when each of its shards does. `Publish development release` and `Persist performance baseline` run only on `dev` pushes in which every microVM test job succeeded or was skipped. The pull-request `Performance regression gate` reads only the platform jobs' results and publishes nothing. The @@ -564,7 +569,8 @@ repositories pay for, or a self-hosted Windows runner without a hypervisor would cost about as much as another general WHP runner. The WHP microVM tests bound every run that skips both OpenVMM suites, so each -backend runs them in two jobs. In reference run +backend runs them in two jobs, and the first of them in shards (see +[MicroVM test shards](#microvm-test-shards)). In reference run [39585909588](https://github.com/microsoft/nvx/actions/runs/39585909588), the single WHP job took 15.1 minutes: 0.8 minutes of setup, 10.1 minutes of default suite, and 4.2 minutes of guest smoke and sandbox steps, which KVM ran @@ -573,15 +579,18 @@ beside the default suite after a setup of its own, so it holds one more general runner per backend for about three to five minutes. The default suite waits only for `artifacts`, so a cold build of the Ubuntu or Azure Linux images, about 4 minutes, delays only the guest and sandbox job, which takes -about half as long as the default suite. A full-matrix run then fills KVM's -four general runners: one with the Linux VMM-test build and then its KVM leg, -and three with the default suite, guest and sandbox, and debug-kernel jobs. -MSHV's four hold its VMM-test leg and the same three jobs on `dev` pushes and -full runs, which also run the debug kernel there, while a pull request leaves -one free. On WHP, a run uses at most five of the eight general runners. When a -pool has a single free general runner, either job of a backend can take it -first. If the guest and sandbox job does, the default suite waits for it, and -the backend's microVM tests take about as long as one job did. +about as long as a shard of the default suite. A full-matrix run then needs +five of KVM's four general runners at once: one for the Linux VMM-test build +and then its KVM leg, and four for the two shards, the guest and sandbox job, +and the debug-kernel job, so the last of them to start waits for the first to +end, about three minutes later; a run that skips the VMM tests fills the four. +MSHV's four hold the two shards, the guest and sandbox job, and, on `dev` +pushes and full runs, the debug-kernel job, and its VMM-test leg starts after +the Linux build, by when the guest and sandbox job has ended. On WHP, a run +uses at most six of the eight general runners. When a pool has fewer free +general runners than a backend's microVM jobs, the jobs that start last wait +for the first to end, so the backend's microVM tests take up to about twice as +long as one of its jobs, still less than the unsharded suite. ## MicroVM test shards @@ -599,6 +608,25 @@ on WHP, so each backend has a partition of its own. that the shards of a backend take about as long as each other, whatever number of scenarios each runs. +CI runs each shard in a job of its own. The default-suite job of each backend, +`nvx-microvm-tests-`, is a matrix of its shards, which passes each leg +its `matrix.shard` and the matrix's `strategy.job-total` as `--shard +INDEX/COUNT`, on the backend's `general` runners. A leg fails when the +partition gives its backend another number of shards than the matrix runs, +and a test checks that each backend's matrix runs exactly the shards of its +partition, so that the legs run every default scenario exactly once on each +backend, with every processor count. A shard that fails does not cancel the +others (`fail-fast: false`), so each keeps its failure logs and its duration +report, under names of its own, and *Re-run failed jobs* reruns only the +shards that failed. GitHub expands the matrix only once the job's condition +holds, so a backend's shards run together or not at all, and the job's +result, which `Required status check`, `Publish development release`, and +`Persist performance baseline` read, succeeds only when every shard succeeds. +The guest and sandbox jobs and the debug-kernel jobs are not sharded: the +debug-kernel jobs take three to five minutes, about as long as a shard, and +still run on KVM only for pull requests and on every backend for `dev` +pushes and full runs. + The first plan took the medians of the reports of 13 runs from October 21 and 22, 2026. It started each backend from the two shards that [#696](https://github.com/microsoft/nvx/issues/696) proposed, the restore, @@ -616,6 +644,20 @@ which added the shards, every scenario stayed in its shard: | MSHV | 416.0 s | 208.0 s | 208.0 s | `time-abi-conformance` for `directional-network-policy` | | WHP | 601.5 s | 300.4 s | 301.1 s | `snapshot-core` for `host-loopback-policy` | +Each backend runs two shards. A default-suite job spends about 0.7 minutes +outside the suite, setting up and uploading, so in the runs of +[#763](https://github.com/microsoft/nvx/pull/763) a shard job took 4.4 to 4.5 +minutes on KVM, 4.2 to 4.9 on MSHV, and 5.7 to 6.4 on WHP, where the single +default-suite job had taken 8.1 to 8.6, 7.6 to 7.7, and 10.4 to 11.1. That is +about as long as the guest and sandbox jobs beside them, which took 3.7 to +4.1, 2.9 to 4.1, and 5.2 to 6.4 minutes in the runs since +[#695](https://github.com/microsoft/nvx/issues/695) split them off. A third +shard would take about 3.2, 3.0, and 4.0 minutes. On WHP it would end before +the guest and sandbox job, so it would not shorten the backend's tests, and on +KVM, where the two shards, the guest and sandbox job, and the debug-kernel job +fill the four general runners, and on MSHV, where they fill them for `dev` +pushes and full runs, it would mostly wait for a runner. + A sharded run fails before it runs a scenario unless the file puts every default scenario in exactly one shard of every backend, with its duration, and splits the backend's suite into the run's COUNT shards. Tests in @@ -647,7 +689,9 @@ renames a default scenario fails them until it plans the shards again: and the scenarios that changed shard. A scenario stays in its shard unless moving it, or swapping it for a scenario of another shard, shortens the longest shard by a second or more, so new medians that only move within - the run-to-run noise leave the shards as they are. + the run-to-run noise leave the shards as they are. To change a backend's + number of shards, pass `--shards BACKEND=COUNT` and change the matrix of its + default-suite job in `ci.yml` to list the shards from 1 to COUNT. ## Guest artifact handoff diff --git a/doc/usage.md b/doc/usage.md index 4f485403a..0c59e8ce5 100644 --- a/doc/usage.md +++ b/doc/usage.md @@ -542,10 +542,11 @@ only the same-host restore scenarios `smp`, `smp-snapshot`, `--shard INDEX/COUNT`, such as `1/2`, runs shard INDEX of the COUNT shards of the default Alpine suite on the default kernel, as [`microvm_shards.json`](../scripts/nvx_tools/microvm_shards.json) splits it for -the backend (see [MicroVM test shards](ci.md#microvm-test-shards)). It takes no -`--scenario`, `--debug-kernel`, or other `--guest`, and fails before it runs a -scenario unless that file puts each default scenario in exactly one shard of -every backend and splits the backend's suite into COUNT shards. The +the backend; CI runs each shard in a job of its own (see +[MicroVM test shards](ci.md#microvm-test-shards)). It takes no `--scenario`, +`--debug-kernel`, or other `--guest`, and fails before it runs a scenario +unless that file puts each default scenario in exactly one shard of every +backend and splits the backend's suite into COUNT shards. The `cpu-profile-fallback` scenario cold boots with OpenVMM's `host-cpu-unknown` test hook, which makes OpenVMM's default CPU profile selection, `auto`, fall back to a host profile on a host that a built-in profile serves, checks the diff --git a/scripts/test_nvx_tools.py b/scripts/test_nvx_tools.py index 7181c43d0..db9f25c85 100644 --- a/scripts/test_nvx_tools.py +++ b/scripts/test_nvx_tools.py @@ -55,6 +55,7 @@ guest_binaries, guests, microvm_durations, + microvm_shards, nvx_rs, release, sandbox, @@ -73,7 +74,11 @@ UbuntuBuildConstants, ZstdBuildConstants, ) -from nvx_tools.microvm_tests import PUBLIC_SANDBOX_SCENARIOS # noqa: E402 +from nvx_tools.microvm_tests import ( # noqa: E402 + MICROVM_PROCESSOR_COUNTS, + PUBLIC_SANDBOX_SCENARIOS, + default_suite, +) from nvx_tools.time_abi import ( # noqa: E402 CPU_GENERATIONS, OPENVMM_CPU_PROFILES, @@ -252,6 +257,70 @@ def _expression_conjuncts(expression: str) -> list[str]: return conjuncts +def _matrix_legs(job: Mapping[str, Any]) -> list[dict[str, Any]]: + """Return the inputs that each leg of a job that calls a reusable workflow + passes, with the expressions of its matrix and strategy resolved; a job + without a matrix has one leg.""" + strategy: Mapping[str, Any] = job.get("strategy", {}) + matrix: dict[str, list[Any]] = dict(strategy.get("matrix", {})) + if {"include", "exclude"} & set(matrix): + raise AssertionError(f"unsupported matrix: {matrix}") + combinations = [ + dict(zip(matrix, values, strict=True)) + for values in itertools.product(*matrix.values()) + ] + legs: list[dict[str, Any]] = [] + for combination in combinations: + expressions: dict[str, Any] = { + f"${{{{ matrix.{name} }}}}": value for name, value in combination.items() + } + expressions["${{ strategy.job-total }}"] = len(combinations) + + def resolve(value: Any, expressions: dict[str, Any] = expressions) -> Any: + if not isinstance(value, str): + return value + if value in expressions: + return expressions[value] + for expression, replacement in expressions.items(): + value = value.replace(expression, str(replacement)) + return value + + legs.append( + {name: resolve(value) for name, value in job.get("with", {}).items()} + ) + return legs + + +def _resolve_inputs(text: str, inputs: Mapping[str, Any]) -> str: + """Resolve the input expressions of a reusable workflow's step: an input, + `inputs.NAME && 'VALUE' || 'OTHER'`, and `inputs.NAME && format(...) + || 'OTHER'` with inputs as the arguments.""" + + def value(name: str) -> Any: + return inputs[name.removeprefix("inputs.")] + + def formatted(match: re.Match[str]) -> str: + if not value(match.group(1)): + return match.group(4) + arguments = [str(value(name.strip())) for name in match.group(3).split(",")] + return match.group(2).format(*arguments) + + text = re.sub( + r"\$\{\{ (inputs\.[a-z-]+) && format\('([^']*)', ([^)]*)\) " + r"\|\| '([^']*)' \}\}", + formatted, + text, + ) + text = re.sub( + r"\$\{\{ (inputs\.[a-z-]+) && '([^']*)' \|\| '([^']*)' \}\}", + lambda match: match.group(2 if value(match.group(1)) else 3), + text, + ) + return re.sub( + r"\$\{\{ (inputs\.[a-z-]+) \}\}", lambda match: str(value(match.group(1))), text + ) + + def _nvx_command_modules() -> dict[str, set[str]]: """Map each nvx.py command to the scripts/nvx_tools modules that its parser configuration and handler, and the nvx.py functions that they call, name.""" @@ -4531,25 +4600,28 @@ def consume( if callee.startswith("./.github/workflows/"): workflow = load(callee.removeprefix("./.github/workflows/")) triggers = workflow.get(True, workflow.get("on")) - inputs = { - **{ - name: definition.get("default") - for name, definition in triggers["workflow_call"][ - "inputs" - ].items() - }, - **job.get("with", {}), + defaults = { + name: definition.get("default") + for name, definition in triggers["workflow_call"]["inputs"].items() } - contexts: list[dict[str, object]] = [ - { - "runner.os": ( - "Windows" - if "windows" in json.loads(inputs["runner"]) - else "Linux" - ), - **{f"inputs.{name}": value for name, value in inputs.items()}, - } - ] + # Each leg of a matrix, such as each shard of a default + # microVM suite, with the inputs that it passes. + contexts: list[dict[str, object]] = [] + for leg in _matrix_legs(job): + inputs = {**defaults, **leg} + contexts.append( + { + "runner.os": ( + "Windows" + if "windows" in json.loads(inputs["runner"]) + else "Linux" + ), + **{ + f"inputs.{name}": value + for name, value in inputs.items() + }, + } + ) steps = [ step for definition in workflow["jobs"].values() @@ -7278,18 +7350,30 @@ def test_ci_runs_the_steps_after_the_default_microvm_suite_beside_it(self): guests = jobs[f"nvx-microvm-guests-{backend}"] debug = jobs[f"nvx-microvm-debug-{backend}"] name = f"{label} / Guests and sandboxes" + shard = "Shard ${{ matrix.shard }} of ${{ strategy.job-total }}" with self.subTest(backend=backend): - self.assertEqual(suite["name"], f"NVX microVM tests / {label}") + self.assertEqual( + suite["name"], f"NVX microVM tests / {label} / {shard}" + ) self.assertEqual(suite["needs"][0], "artifacts") self.assertNotIn("distro-artifacts", suite["needs"]) # The guest and sandbox job also waits for the Ubuntu and Azure # Linux artifacts, and otherwise equals the suite's job but for - # its name and the selector. + # its name, the selector, and the suite's shards + # (test_ci_runs_each_default_microvm_suite_in_its_shards). conjuncts = _expression_conjuncts(suite["if"]) after = conjuncts.index("needs.artifacts.result == 'success'") + 1 + unsharded = { + key: value for key, value in suite.items() if key != "strategy" + } + inputs = { + key: value + for key, value in suite["with"].items() + if key not in ("shard", "shard-count") + } self.assertEqual( guests | {"if": _expression_conjuncts(guests["if"])}, - suite + unsharded | { "name": f"NVX microVM tests / {name}", "needs": ["artifacts", "distro-artifacts", *suite["needs"][1:]], @@ -7298,7 +7382,7 @@ def test_ci_runs_the_steps_after_the_default_microvm_suite_beside_it(self): "needs.distro-artifacts.result == 'success'", *conjuncts[after:], ], - "with": suite["with"] + "with": inputs | {"display-name": name, "guests-and-sandboxes": True}, }, ) @@ -7403,46 +7487,154 @@ def downloaded(selection: list[str]) -> set[str]: selected(backend, debug=True, guests=None), ) - def artifact(template: str, backend: str, *, debug: bool, guests: bool) -> str: - values = {"debug-kernel": debug, "guests-and-sandboxes": guests} - return re.sub( - r"\$\{\{ inputs\.([a-z-]+) && '([^']*)' \|\| '([^']*)' \}\}", - lambda match: match.group(2 if values[match.group(1)] else 3), - template.replace("${{ inputs.backend }}", backend), - ) - - # A second upload of an artifact name in a run fails, so the default - # suite, the guest and sandbox, and the debug-kernel jobs of a backend - # each upload under a name of their own. - selections = ((False, False), (False, True), (True, False)) + # A second upload of an artifact name in a run fails, so each shard of + # the default suite, the guest and sandbox job, and the debug-kernel + # job of every backend upload under names of their own. + defaults = { + name: definition.get("default") + for name, definition in microvm[True]["workflow_call"]["inputs"].items() + } + legs = [ + {**defaults, **leg} + for job in jobs.values() + if job.get("uses") == "./.github/workflows/run-nvx-microvm-tests.yml" + for leg in _matrix_legs(job) + ] self.assertTrue(uploads) for upload in uploads: - uploaded = [ - artifact(upload["with"]["name"], backend, debug=debug, guests=guests) - for backend in backends - for debug, guests in selections - ] + uploaded = [_resolve_inputs(upload["with"]["name"], leg) for leg in legs] with self.subTest(upload=upload["name"]): + self.assertNotIn("inputs.", " ".join(uploaded)) self.assertEqual(len(set(uploaded)), len(uploaded), uploaded) failure_logs = next( step for step in uploads if step["name"] == "Upload NVX microVM failure logs" ) + shards = len(microvm_shards.load_partition().backends["kvm"].shards) self.assertEqual( + sorted( + _resolve_inputs(failure_logs["with"]["name"], leg) + for leg in legs + if leg["backend"] == "kvm" + ), [ - artifact( - failure_logs["with"]["name"], "kvm", debug=debug, guests=guests - ) - for debug, guests in selections - ], - [ - "nvx-microvm-tests-kvm", - "nvx-microvm-tests-kvm-guests", "nvx-microvm-tests-kvm-debug", + "nvx-microvm-tests-kvm-guests", + *( + f"nvx-microvm-tests-kvm-shard-{index}" + for index in range(1, shards + 1) + ), ], ) + def test_ci_runs_each_default_microvm_suite_in_its_shards(self): + # #696: each backend runs its default suite in the shards that + # microvm_shards.json plans, one leg of the default-suite job's matrix + # each, so that the WHP suite no longer bounds the runs that skip both + # OpenVMM suites. The legs run every shard once, so every default + # scenario runs exactly once per backend, on the Alpine guest and the + # default kernel with every processor count, and the job's result, + # which the required status check, the release, and the persistence + # read, fails when a shard fails. Neither the guest and sandbox jobs + # nor the debug-kernel jobs are sharded. + workflows = BuildConstants.REPO_ROOT / ".github" / "workflows" + jobs = yaml.safe_load((workflows / "ci.yml").read_text(encoding="utf-8"))[ + "jobs" + ] + microvm = yaml.safe_load( + (workflows / "run-nvx-microvm-tests.yml").read_text(encoding="utf-8") + ) + inputs = microvm[True]["workflow_call"]["inputs"] + for name in ("shard", "shard-count"): + with self.subTest(input=name): + self.assertEqual(inputs[name]["type"], "number") + self.assertEqual(inputs[name]["default"], 0) + self.assertIs(inputs[name]["required"], False) + defaults = { + name: definition.get("default") for name, definition in inputs.items() + } + steps = microvm["jobs"]["test"]["steps"] + sharded = { + step["name"]: str(step["run"]) + for step in steps + if "inputs.shard" in str(step.get("run", "")) + } + self.assertEqual( + set(sharded), + {"Run NVX microVM tests on Linux", "Run NVX microVM tests on Windows"}, + ) + partition = microvm_shards.load_partition() + suite = default_suite() + shard = "Shard ${{ matrix.shard }} of ${{ strategy.job-total }}" + backends = { + "kvm": "Linux / KVM", + "mshv": "Linux / MSHV", + "whp": "Windows / WHP", + } + for backend, label in backends.items(): + job = jobs[f"nvx-microvm-tests-{backend}"] + count = len(partition.backends[backend].shards) + system = "Windows" if backend == "whp" else "Linux" + with self.subTest(backend=backend): + # A failed shard does not cancel the others, which keep their + # logs and durations; GitHub expands the matrix after it + # evaluates the condition, so the shards run together or not + # at all. + self.assertEqual( + job["strategy"], + { + "fail-fast": False, + "matrix": {"shard": list(range(1, count + 1))}, + }, + ) + self.assertNotIn("matrix.", job["if"]) + self.assertEqual(job["name"], f"NVX microVM tests / {label} / {shard}") + self.assertEqual(job["with"]["display-name"], f"{label} / {shard}") + self.assertEqual(job["with"]["shard"], "${{ matrix.shard }}") + self.assertEqual( + job["with"]["shard-count"], "${{ strategy.job-total }}" + ) + for kind in ("guests", "debug"): + other = jobs[f"nvx-microvm-{kind}-{backend}"] + self.assertNotIn("strategy", other) + self.assertFalse({"shard", "shard-count"} & set(other["with"])) + + legs = [{**defaults, **leg} for leg in _matrix_legs(job)] + self.assertEqual( + [(leg["shard"], leg["shard-count"]) for leg in legs], + [(index, count) for index in range(1, count + 1)], + ) + ran: list[str] = [] + for leg in legs: + command = _resolve_inputs( + sharded[f"Run NVX microVM tests on {system}"], leg + ) + self.assertNotIn("${{", command) + args = nvx.parse_args( + shlex.split(command.split("nvx.py ", 1)[1].replace("\\", "/")) + ) + self.assertEqual( + (args.backend, args.shard, args.scenario, args.debug_kernel), + (backend, (leg["shard"], count), None, False), + ) + self.assertEqual(args.guest, "alpine") + self.assertEqual(args.processors, list(MICROVM_PROCESSOR_COUNTS)) + ran.extend( + microvm_shards.shard_scenarios( + partition, backend, args.shard, suite + ) + ) + self.assertEqual(sorted(ran), sorted(suite)) + # The guest and sandbox and debug-kernel jobs run test-microvm without + # a shard. + for kind in ("guests", "debug"): + for backend in backends: + leg = {**defaults, **jobs[f"nvx-microvm-{kind}-{backend}"]["with"]} + for step_name, run in sharded.items(): + with self.subTest(job=f"{kind}-{backend}", step=step_name): + self.assertNotIn("--shard", _resolve_inputs(run, leg)) + def test_benchmarks_rely_on_the_microvm_correctness_jobs(self): # #286: the benchmark action ran a second smp-lapic gate before # acceptance. The microVM correctness jobs already gate the required @@ -9425,10 +9617,10 @@ def test_ci_runs_the_public_sandbox_acceptance_on_every_backend(self): self.assertIn("build/test-results/public-exec-${{ inputs.backend }}", upload) def test_every_microvm_job_uploads_its_scenario_durations(self): - # Each microVM job, the guest and sandbox and debug-kernel ones too, - # keeps the duration report of every test-microvm step whether it - # passes, fails, or is cancelled, under a name of its own in each - # attempt (#694, #695). + # Each microVM job, the guest and sandbox and debug-kernel ones and + # each shard of a default suite too, keeps the duration report of + # every test-microvm step whether it passes, fails, or is cancelled, + # under a name of its own in each attempt (#694, #695, #696). workflows = BuildConstants.REPO_ROOT / ".github" / "workflows" steps = { step["name"]: step @@ -9461,16 +9653,18 @@ def test_every_microvm_job_uploads_its_scenario_durations(self): kernel = "${{ inputs.debug-kernel && 'debug' || 'default' }}" part = "${{ inputs.guests-and-sandboxes && '-guests' || '' }}" + shard = "${{ inputs.shard && format('-shard-{0}', inputs.shard) || '' }}" name = upload["with"]["name"] self.assertEqual( name, "nvx-microvm-durations-${{ runner.os }}-${{ inputs.backend }}-" + kernel + part + + shard + "-${{ github.run_id }}-${{ github.run_attempt }}", ) jobs = { - job_name: job["with"] + job_name: job for job_name, job in yaml.safe_load( (workflows / "ci.yml").read_text(encoding="utf-8") )["jobs"].items() @@ -9485,34 +9679,43 @@ def test_every_microvm_job_uploads_its_scenario_durations(self): }, ) artifacts: set[str] = set() - for job_name, inputs in jobs.items(): - system = "Windows" if "windows" in json.loads(inputs["runner"]) else "Linux" - debug = inputs.get("debug-kernel", False) is True - guests = inputs.get("guests-and-sandboxes", False) is True - for attempt in (1, 2): - artifact = ( - name.replace("${{ runner.os }}", system) - .replace("${{ inputs.backend }}", inputs["backend"]) - .replace(kernel, "debug" if debug else "default") - .replace(part, "-guests" if guests else "") - .replace("${{ github.run_id }}", "39585909588") - .replace("${{ github.run_attempt }}", str(attempt)) - ) - with self.subTest(job=job_name, attempt=attempt): - self.assertNotIn("${{", artifact) - self.assertNotIn(artifact, artifacts) - artifacts.add(artifact) + legs = 0 + for job_name, job in jobs.items(): + # Each shard of a default suite is a leg of its job's matrix. + for inputs in _matrix_legs(job): + legs += 1 + system = ( + "Windows" if "windows" in json.loads(inputs["runner"]) else "Linux" + ) + debug = inputs.get("debug-kernel", False) is True + guests = inputs.get("guests-and-sandboxes", False) is True + index = inputs.get("shard", 0) + for attempt in (1, 2): + artifact = ( + name.replace("${{ runner.os }}", system) + .replace("${{ inputs.backend }}", inputs["backend"]) + .replace(kernel, "debug" if debug else "default") + .replace(part, "-guests" if guests else "") + .replace(shard, f"-shard-{index}" if index else "") + .replace("${{ github.run_id }}", "39585909588") + .replace("${{ github.run_attempt }}", str(attempt)) + ) + with self.subTest(job=job_name, shard=index, attempt=attempt): + self.assertNotIn("${{", artifact) + self.assertNotIn(artifact, artifacts) + artifacts.add(artifact) self.assertIn("nvx-microvm-durations-Linux-kvm-debug-39585909588-2", artifacts) - # The default suite and the guest and sandbox steps of a backend each - # keep their own reports. + # Each shard of a backend's default suite and its guest and sandbox + # steps keep their own reports. self.assertLessEqual( { - "nvx-microvm-durations-Windows-whp-default-39585909588-1", + "nvx-microvm-durations-Windows-whp-default-shard-1-39585909588-1", + "nvx-microvm-durations-Windows-whp-default-shard-2-39585909588-1", "nvx-microvm-durations-Windows-whp-default-guests-39585909588-1", }, artifacts, ) - self.assertEqual(len(artifacts), 2 * len(jobs)) + self.assertEqual(len(artifacts), 2 * legs) @unittest.skipUnless(os.name == "nt", "requires Windows PowerShell") def test_windows_cli_validation_stops_at_each_failed_command(self):