Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion doc/ci.md
Original file line number Diff line number Diff line change
Expand Up @@ -1607,7 +1607,7 @@ microcode and the invariant-TSC flags, may read `unknown`.
| Check | Implementation |
| --- | --- |
| H1 | `/dev/kvm` or `/dev/mshv` is readable and writable, and a KVM host has no `/dev/mshv`; on Windows, `WHvGetCapability` reports a hypervisor |
| H2 | Vendor, family, model, stepping, microcode, and OS build from `/proc/cpuinfo` or the Windows registry. Then `openvmm --hypervisor <backend> --cpu-fingerprint <path>` writes the host's CPU fingerprint (by default `nvx-cpu-fingerprint-<backend>.json` in the probe directory; `--cpu-fingerprint` overrides it), checks it against the pinned profile that `auto` selects from OpenVMM's catalog, which shares one profile per generation across backends, and prints one `NVX-CPU-PROFILE:` line. The check never falls back to a host profile, so a CPU on which `auto` falls back fails H2. H2 reports the generation and the profile from that line, so a profile that an OpenVMM pin adds qualifies its hosts without an NVX change. H2 requires exit status 0, `status=pass`, the same backend, and a catalog profile of the reported generation, reports the profile and surface digests, and otherwise fails with OpenVMM's code, for example `[E_PROFILE_HOST_UNKNOWN]` on a CPU that no profile serves or `[E_PROFILE_UNSUPPORTED]` naming every unsupported CPUID bit. On MSHV and WHP, OpenVMM checks the entries outside the profile on a probe partition configured from the profile, as a cold boot does. With `--no-openvmm`, H2 maps the host with NVX's copy of the catalog instead: `skylake-sp` (6/85, steppings 0 to 4, `intel.skylake-sp.v1`), `icelake-sp` (6/106, `intel.icelake-sp.v1`), `emeraldrapids` (6/207, `intel.emeraldrapids.v1`), `alderlake` (6/151 and 6/154, `intel.alderlake.v1`), `milan` (AMD 25/1, `amd.milan.v1`), `genoa` (AMD 25/17, `amd.genoa.v1`), or `turin` (AMD 26/2, `amd.turin.v1`). Any other CPU, including Cascade Lake and Cooper Lake, fails with `E_PROFILE_HOST_UNKNOWN`. A unit test keeps the copy equal to the profiles of the OpenVMM submodule's pinned revision, which it reads from the gitlink's commit, so the NVX CLI jobs validate the CLI after they check out OpenVMM. The host OS's invariant-TSC flags (`constant_tsc nonstop_tsc`, or the CPUID bit on Windows) are recorded as evidence and never fail the check, because they don't decide what a guest observes: Azure WHP hosts show the CPUID bit but cannot offer invariant TSC to partitions, and their guests measure tens of nanoseconds of skew |
| H2 | Vendor, family, model, stepping, microcode, and OS build from `/proc/cpuinfo` or the Windows registry. Then `openvmm --hypervisor <backend> --cpu-fingerprint <path>` writes the host's CPU fingerprint (by default `nvx-cpu-fingerprint-<backend>.json` in the probe directory; `--cpu-fingerprint` overrides it), checks it against the pinned profile that `auto` selects from OpenVMM's catalog, the latest revision of the generation's profile, which every backend shares, and prints one `NVX-CPU-PROFILE:` line. The check never falls back to a host profile, so a CPU on which `auto` falls back fails H2. H2 reports the generation and the profile from that line, so a profile that an OpenVMM pin adds qualifies its hosts without an NVX change. H2 requires exit status 0, `status=pass`, the same backend, and a catalog profile of the reported generation, reports the profile and surface digests, and otherwise fails with OpenVMM's code, for example `[E_PROFILE_HOST_UNKNOWN]` on a CPU that no profile serves or `[E_PROFILE_UNSUPPORTED]` naming every unsupported CPUID bit. On MSHV and WHP, OpenVMM checks the entries outside the profile on a probe partition configured from the profile, as a cold boot does. With `--no-openvmm`, H2 maps the host with NVX's copy of the catalog instead: `skylake-sp` (6/85, steppings 0 to 4, `intel.skylake-sp.v1`), `icelake-sp` (6/106, `intel.icelake-sp.v1`), `emeraldrapids` (6/207, `intel.emeraldrapids.v1`), `alderlake` (6/151 and 6/154, `intel.alderlake.v1`), `milan` (AMD 25/1, `amd.milan.v1`), `genoa` (AMD 25/17, `amd.genoa.v2`), or `turin` (AMD 26/2, `amd.turin.v1`). Any other CPU, including Cascade Lake and Cooper Lake, fails with `E_PROFILE_HOST_UNKNOWN`. A unit test keeps the copy, which records the latest revision of each generation's profile, equal to the profiles of the OpenVMM submodule's pinned revision, which it reads from the gitlink's commit, so the NVX CLI jobs validate the CLI after they check out OpenVMM. The host OS's invariant-TSC flags (`constant_tsc nonstop_tsc`, or the CPUID bit on Windows) are recorded as evidence and never fail the check, because they don't decide what a guest observes: Azure WHP hosts show the CPUID bit but cannot offer invariant TSC to partitions, and their guests measure tens of nanoseconds of skew |
| H3 | `openvmm --x-time-abi-verify` builds the partition and runs the time ABI preflight without running the guest. Its `NVX-TIME-ABI-VERIFY:` line must report `status=ok` for the backend, plausible declared and native TSC rates, the backend's LAPIC rate, and a `cpu_profile` that is a catalog profile, `<vendor>.<generation>.v<revision>` of any generation but a host profile's `host`, which rejects the host profile that `auto` falls back to on a CPU that no built-in profile serves, and, when H2 runs too, a revision of the profile H2 names. A failed preflight reports OpenVMM's code, for example `[E_TSC_SYNC_UNSUPPORTED]` |
| H4 | Samples of the TSC against the host's monotonic clocks, with sleeps between them so the host's CPUs idle: 13 samples 10 s apart, or 3 samples 1 s apart with `--ci-schedule`. Each sample reads the TSC between two reads of a clock, keeping the tightest of 64 brackets; its uncertainty is half the bracket plus half the clock's resolution. A clock that returns the same value to consecutive reads is coarser than one read, so the probe takes its smallest step as its resolution: Hyper-V's reference TSC page advances the Linux clocks in 100 ns steps although `clock_getres` reports 1 ns. The interval stability is judged against a clock that time synchronization never steers, `CLOCK_MONOTONIC_RAW` on Linux and `QueryPerformanceCounter` on Windows: every interval between consecutive samples must be conclusive within 0.25 ppm, and the interval rates must agree within 1 ppm. chrony's frequency updates move `CLOCK_MONOTONIC`'s rate by up to several ppm between seconds on the Azure runners, which says nothing about the TSC. The rate over the whole window must lie within 100 ppm of the rate H3 reports when H3 runs in the same invocation, and H4 reports their deviation to 0.001 ppm. On Linux with a synchronized chronyd, that rate is the rate against `CLOCK_MONOTONIC_RAW` corrected by the frequency that chronyd estimates from its whole sample history, as `chronyc -c tracking` reports it, and every rate within chronyd's skew of that estimate must lie within the limit, so a range that straddles the limit fails as inconclusive; the rate against `CLOCK_MONOTONIC` is evidence only, because it also carries the slew with which chronyd removes its current offset, which reaches hundreds of ppm for seconds after the host's clock jumps (#768). Otherwise the rate is measured against `CLOCK_MONOTONIC` on Linux, and the detail says why chronyd's estimate is missing, or against `QueryPerformanceCounter` on Windows. A Linux host's clocksource is recorded as evidence |
| H5 | Pinned-thread ping-pong rounds over every pair of host CPUs; `max_abs_offset_ns` is at most 1,000, the measurement is conclusive, and no pair stalls |
Expand Down
33 changes: 18 additions & 15 deletions doc/openvmm-upstream-roadmap.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,17 +9,19 @@
- Mapped fork tip: `a0556d7d85ce1fff89f5b45a98f15c5c79cbb6fb`, position 157. The graph, PR
summary, and commit mapping below cover positions 001-157 and the fixups at positions
334-340, 344, and 345.
- NVX's pinned OpenVMM revision: `57d0a18b3b542dce3f0787e60f3c48296b40d30e`, position 347.
Positions 346 and 347, which microsoft/openvmm#4767 proposes for `feature/microvm/dev`, are
two commits on position 345, `e68871cc6`, the head of `feature/microvm/dev` since
microsoft/openvmm#4762 merged positions 344 and 345. Those are two commits on position 343,
`a12d9167f`, which microsoft/openvmm#4749 merged with positions 341 and 342. Positions
341-343 are three commits on position 340, `2b681564d`, which microsoft/openvmm#4735 merged
with position 339. Positions 339 and 340 are two commits on position 338, `8a0d945ed`, which
microsoft/openvmm#4734 merged, and position 338 is one commit on `79a501b63`, the head of the
`feature/microvm/release/v2.3` branch of microsoft/openvmm. The pin is 190 commits past the
mapped tip on the same first-parent chain. Positions 158-333, 341-343, 346, and 347 are not
yet assigned to PRs; see [Commits after the mapped tip](#commits-after-the-mapped-tip).
- NVX's pinned OpenVMM revision: `d2338ea22c36135b468e7f3ae168066b6c9efde9`, position 349.
Positions 348 and 349, which microsoft/openvmm#4769 proposes for `feature/microvm/dev`, are
two commits on position 347, `57d0a18b3`, the head of `feature/microvm/dev` since
microsoft/openvmm#4767 merged positions 346 and 347. Those are two commits on position 345,
`e68871cc6`, which microsoft/openvmm#4762 merged with position 344. Positions 344 and 345 are
two commits on position 343, `a12d9167f`, which microsoft/openvmm#4749 merged with positions
341 and 342. Positions 341-343 are three commits on position 340, `2b681564d`, which
microsoft/openvmm#4735 merged with position 339. Positions 339 and 340 are two commits on
position 338, `8a0d945ed`, which microsoft/openvmm#4734 merged, and position 338 is one
commit on `79a501b63`, the head of the `feature/microvm/release/v2.3` branch of
microsoft/openvmm. The pin is 192 commits past the mapped tip on the same first-parent chain.
Positions 158-333, 341-343, and 346-349 are not yet assigned to PRs; see
[Commits after the mapped tip](#commits-after-the-mapped-tip).
- Upstream main: `22c0485be597f87a76771e1db03eaa823a685ffc` (2026-10-22), the fork base.

## Upstream status
Expand Down Expand Up @@ -309,8 +311,8 @@ The physical branch is one first-parent chain:
-> PR14 fixup [336] -> PR09 fixup [337]
-> PR02 fixup [338] -> PR20 fixup [339] -> PR23 fixup [340]
-> unassigned [341-343] -> PR18 fixups [344-345]
-> unassigned [346-347]
-> 57d0a18b3 (NVX pin)
-> unassigned [346-349]
-> d2338ea22 (NVX pin)
```

For upstream review, PR01-PR08 and PR14 can be rebased independently onto current upstream main,
Expand All @@ -332,8 +334,8 @@ PR18's position 111.

## Commits after the mapped tip

The NVX pin extends the chain past position 157 without merges. Of the 190 later commits,
positions 158-333, 341-343, 346, and 347 are not yet classified or assigned to PRs, positions
The NVX pin extends the chain past position 157 without merges. Of the 192 later commits,
positions 158-333, 341-343, and 346-349 are not yet classified or assigned to PRs, positions
334-337 are the fixups of the `feature/microvm/release/v2.3` rebase, position 338 is a PR02
fixup, positions 339 and 340 are PR20 and PR23 fixups, and positions 344 and 345 are PR18
fixups.
Expand Down Expand Up @@ -381,6 +383,7 @@ This table groups them by topic in chain order; ranges are inclusive, as in the
| 344-345 | `a04cc8a00..e68871cc6` | 2 | A microVM virtio-fs queue worker waits for request admission to reopen instead of ending, so the queues that a gated restore, or the rollback of a failed save, starts behind the input gate serve the guest once input resumes (microsoft/openvmm#4762); fold into 111 |
| 346 | `3e3a454c5` | 1 | A platform snapshot whose NIC was captured with the complete deny-all policy, `--network-egress deny` and `--host-loopback deny` without rules, a proxy, or ingress, is a NIC template, which digest encodings from version 3 on identify |
| 347 | `57d0a18b3` | 1 | `--restore-network-policy` lets a platform restore install the policy of its own options on a NIC template, including private-network ingress, explicit host loopback, and forwards, before any vCPU runs, while the NIC keeps its identity and the contract keeps the template's deny-all (microsoft/openvmm#4767) |
| 348-349 | `689577604..d2338ea22` | 2 | A second revision of the Genoa CPU profile, `amd.genoa.v2`, which `auto` selects: `amd.genoa.v1` without the basic leaves above `0xd` and without `LFENCE` serialization, NoSmmCtlMsr, and the TSA immunities in `0x80000021`, none of which the Microsoft hypervisor of the Azure Genoa VMs fingerprinted so far presents, with the nested-WHP Genoa host of microsoft/nvx#928 as a test fixture (microsoft/openvmm#4769); revises position 282's Genoa profile |

Some of these commits change mapped work rather than add to it. Positions 246-250 remove the
legacy microVM clock paths and change files that PR10 and PR17 also change. Position 252 makes
Expand Down
32 changes: 22 additions & 10 deletions doc/usage.md
Original file line number Diff line number Diff line change
Expand Up @@ -1286,7 +1286,7 @@ is authorized for the organization when it enforces single sign-on.
Every microVM presents a CPU profile to its guest: the complete CPUID surface
of one CPU generation, which OpenVMM pins and every backend shares (see the
[CPU profile design](design/2.2-cpu-and-topology.md#221-cpu-profiles)). By default, OpenVMM selects the
built-in profile of the host's CPU generation:
latest revision of the built-in profile of the host's CPU generation:

| Profile | Generation | CPUs (display family/model) |
| --- | --- | --- |
Expand All @@ -1295,9 +1295,13 @@ built-in profile of the host's CPU generation:
| `intel.emeraldrapids.v1` | `emeraldrapids` | Intel Xeon Scalable, fifth generation (6/207) |
| `intel.alderlake.v1` | `alderlake` | Intel Core, twelfth generation (6/151 and 6/154) |
| `amd.milan.v1` | `milan` | AMD EPYC, third generation (25/1, Milan-X included) |
| `amd.genoa.v1` | `genoa` | AMD EPYC, fourth generation (25/17, Genoa-X included) |
| `amd.genoa.v2` | `genoa` | AMD EPYC, fourth generation (25/17, Genoa-X included) |
| `amd.turin.v1` | `turin` | AMD EPYC, fifth generation (26/2) |

OpenVMM keeps every earlier revision pinned, so that its snapshots still
restore, and `--cpu-profile` still selects it by its ID: `amd.genoa.v1` on
Genoa.

Any other CPU, including Cascade Lake, Sapphire Rapids, Granite Rapids, Tiger
Lake, Raptor Lake, and AMD's Ryzen CPUs, has no built-in profile: on an Intel
or AMD CPU, `--cpu-profile auto` falls back to a host profile, described
Expand All @@ -1306,19 +1310,27 @@ A host of a listed generation can still fail with `E_PROFILE_UNSUPPORTED` if
its SKU or hypervisor lacks a feature of the profile, and on an Intel or AMD
CPU OpenVMM's error then names `--cpu-profile host`. `intel.alderlake.v1`
derives from one Core i9-12900H on WHP, `amd.milan.v1` from one EPYC 7763
Azure VM on WHP, and `amd.genoa.v1` and `amd.turin.v1` from KVM on one
GitHub-hosted Actions runner each, an Azure VM with an EPYC 9V74 or 9V45.
GitHub's runners with an EPYC 7763, 9V74, or 9V45 pass their profiles' full
`test-microvm` suite on KVM. Alder Lake hosts on KVM and MSHV, Milan, Genoa,
and Turin hosts on MSHV, Genoa and Turin hosts on WHP, bare-metal AMD hosts,
and SKUs without the profiles' features are unverified. The AMD profiles leave
Azure VM on WHP, `amd.genoa.v1` and `amd.turin.v1` from KVM on one
GitHub-hosted Actions runner each, an Azure VM with an EPYC 9V74 or 9V45, and
`amd.genoa.v2` from that EPYC 9V74 runner's KVM and the nested MSHV of an
Azure VM with the same CPU. GitHub's runners with an EPYC 7763, 9V74, or 9V45
pass their profiles' full `test-microvm` suite on KVM, the EPYC 9V74 runner
on `amd.genoa.v1`, of which `amd.genoa.v2` presents a subset. Alder Lake
hosts on KVM and MSHV, Milan, Genoa, and Turin hosts on MSHV, Genoa and Turin
hosts on WHP, bare-metal AMD hosts, and SKUs without the profiles' features
are unverified. The AMD profiles leave
out what KVM cannot present, `BTC_NO` and PSFD without a `SPEC_CTRL` control,
and the enumerations of Intel's speculation controls that KVM adds on AMD
hosts. They present none of AMD's speculation controls, `IBRS`, `STIBP`, and
`SSBD`, because the Azure VMs' hypervisors withhold them; `amd.milan.v1`'s
Linux guests therefore use retpolines and report SSB, SRSO, and TSA as
vulnerable on every host. `amd.genoa.v1` pins the TSA immunities that Azure
presents on Genoa, which a bare-metal Genoa host's KVM does not report.
vulnerable on every host. `amd.genoa.v1` pins what Azure presents to KVM on
Genoa: the basic leaves up to `0x1c`, and `LFENCE` serialization,
NoSmmCtlMsr, and the TSA immunities in `0x80000021`. The Microsoft hypervisor
of the Azure Genoa VMs fingerprinted so far, under MSHV and WHP, presents none
of them, and a bare-metal Genoa host's KVM does not report the TSA
immunities. `amd.genoa.v2` leaves them out, so its Linux guests report TSA as
vulnerable, as `amd.milan.v1`'s do.

On an Intel or AMD development host that no built-in profile serves,
`--cpu-profile auto` falls back to a host profile, `intel.host.v1` or
Expand Down
8 changes: 8 additions & 0 deletions nvx-rs/src/host/cli/cpu_generations.json
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,14 @@
"cpus": [{ "family": 25, "model": 17, "steppings": [0, 15] }]
}
},
{
"id": "amd.genoa.v2",
"vendor": "AuthenticAMD",
"generation": {
"name": "genoa",
"cpus": [{ "family": 25, "model": 17, "steppings": [0, 15] }]
}
},
{
"id": "amd.turin.v1",
"vendor": "AuthenticAMD",
Expand Down
28 changes: 23 additions & 5 deletions nvx-rs/src/host/cli/cpu_profile.rs
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,8 @@
//! generation (`doc/usage.md`, "CPU profiles"). `cpu_generations.json` copies the identity of
//! every profile that OpenVMM pins, from its document in `vmm_core/cpu_profile/profiles/` of the
//! OpenVMM submodule: its ID, the vendor of its CPUs, and its generation's name and CPU models,
//! each a display family, a display model, and the first and last of its steppings.
//! each a display family, a display model, and the first and last of its steppings. The revisions
//! of a generation's profile share its generation, which the guidance lists once.
//! `tests/openvmm_cpu_profiles.rs` compares the copy with the submodule's documents.
//! [`HostCpu::current`] identifies the host CPU with CPUID, as OpenVMM does.
//!
Expand Down Expand Up @@ -119,6 +120,19 @@ fn catalog() -> Vec<Profile> {
serde_json::from_str(CATALOG).expect("cpu_generations.json lists the built-in CPU profiles")
}

/// Returns the generations of the built-in profiles in `catalog` as the guidance lists them, such
/// as `milan 25/1, genoa 25/17`: each once, in the order of its first profile, since the revisions
/// of a generation's profile share it.
fn describe_generations(catalog: &[Profile]) -> String {
let mut generations: Vec<String> = Vec::new();
for generation in catalog.iter().map(Profile::describe) {
if !generations.contains(&generation) {
generations.push(generation);
}
}
generations.join(", ")
}

/// A host CPU, as CPUID identifies it. Only x86-64 hosts have CPUID, so elsewhere, nothing
/// identifies one.
#[derive(Debug, Clone, PartialEq, Eq)]
Expand Down Expand Up @@ -288,8 +302,10 @@ pub(crate) fn guidance(
"nvx: no built-in CPU profile serves this host's CPU, {}",
host.describe()
);
let generations: Vec<String> = catalog.iter().map(Profile::describe).collect();
let generations = format!("the built-in profiles cover {}.", generations.join(", "));
let generations = format!(
"the built-in profiles cover {}.",
describe_generations(&catalog)
);
let Some((_, name, issue)) = vendor else {
return Some(format!(
"{unserved}, so a cold boot with --cpu-profile auto, the default, fails on it with \
Expand Down Expand Up @@ -365,12 +381,14 @@ mod tests {
"intel.alderlake.v1",
"amd.milan.v1",
"amd.genoa.v1",
"amd.genoa.v2",
"amd.turin.v1",
]
);
let generations: Vec<String> = catalog.iter().map(Profile::describe).collect();
// The two revisions of Genoa's profile share its generation, which the guidance lists
// once.
assert_eq!(
generations.join(", "),
describe_generations(&catalog),
"skylake-sp 6/85 steppings 0-4, icelake-sp 6/106, emeraldrapids 6/207, alderlake \
6/151 and 6/154, milan 25/1, genoa 25/17, turin 26/2"
);
Expand Down
Loading
Loading