From 15746297a56b1131164d46c4760a8944152cf0f7 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Tue, 13 Oct 2026 15:11:12 -0300 Subject: [PATCH 1/3] doc: add an architecture-only design guide in doc/design-doc Add doc/design-doc, a design guide that describes the NVX microVM and its single-workload guest runtime by architectural concept rather than by implementation component. The guide is derived from doc/design and does not replace it: doc/design and doc/design.md stay unchanged. The guide covers architecture only. It leaves out code paths, command-line options, internal component names, tests, CI and host qualification, performance measurements, implementation status and current limits, and the designs that doc/design marks as Proposed. README.md indexes five chapters. Each chapter page introduces the chapter and links its sections, and each section is a separate file named by its number, such as 2.2-cpu-and-topology.md: - 1. Architecture Overview: goals, non-goals, system components and their responsibilities, trust boundaries, and the compatibility and versioning model. - 2. Virtual Machine Architecture: the machine model and configuration ownership, CPU profiles and topology, memory layout, interrupt and I/O architecture, and device topology, transport, and discovery. - 3. Virtual Devices: the platform control device, clock and time, consoles, block storage, network, and host filesystem. - 4. Guest Runtime Architecture: the guest filesystem and image model, the guest agents and the workload, isolation and resources, and the host-guest control interface. - 5. VM Lifecycle and State: the boot, snapshot, and restore protocols, readiness and runtime control, the snapshot artifact and attachment model, and shutdown and failure semantics. Beyond reorganizing doc/design, the guide: - Records the owner and lifetime of each kind of configuration: one contract version, one VM, one VM process, or one restore. - Explains why the byte channel exists apart from virtio and what the generation ID is for. - Describes end to end how a restored guest refreshes its entropy, from the host's random-number generator to the workload runtime's post-restore hook, and why an instance checkpoint keeps its state. - Keeps processor capacity in the CPU section and moves processor activation to the restore protocol, next to memory activation. - States that the guest TSC runs at its host's rate, and that a restore needs a host rate within 250 ppm of the snapshot's declared rate. - Describes snapshot memory sharing as the pinned OpenVMM implements it: every restored VM maps the immutable memory artifact copy-on-write, and the VMM relies on the host control plane, not file locks, to keep a writable file from reaching more than one VMM. - Adds brief descriptions of many x86 platform terms, such as XSAVE, MSR, and the e820 map, at their first use. - Follows the implementation where doc/design differs from it or leaves it out: the host exports each mapped file of a blockless managed root on its own, the outcome report records the host-loopback action, the control protocol's exit reports have their own outcome categories, apart from its error replies, a restore keeps the guest's cached pages of shared file mappings, a restore may move an authenticated control listener to a fresh private endpoint, and restore repair outside the time ABI fails with status 1. - States the intended rules that issues #552, #553, and #554 track where the implementation does not enforce them yet: the host enforces every host mapping's access, explicit host-loopback access needs an IPv4 network of /29 or larger, and device DMA stays inside the e820 RAM ranges. - Describes sandbox storage with only the distro and scratch block roles, ahead of the removal of the runtime and custom roles that issue #556 tracks: nothing builds or attaches those layers. Their MMIO windows, IRQs 9 and 12, and interrupt-status words are left unused, so no other device moves. - Identifies snapshot blocks only by a caller-supplied storage generation, ahead of the removal of whole-file SHA-256 block identity that issue #557 tracks: neither capture nor restore reads a block to identify it. The guide states each rule only as far as it holds: - Workload isolation differs between the two workload roots. Only the layered sandbox root gives the workload private mount, PID, and UTS namespaces, a chroot, and a restricted device view, and places the guest agents and the workload in sibling cgroups, with a default 16-MiB memory protection for the guest agents and optional memory and process-count limits for the workload. The blockless managed root runs each execution in the guest's initramfs root and namespaces, contained by its credentials and a dedicated execution cgroup. The architecture overview, the chapter 4 introduction, the guest-agent figure, and the enforcement layers make the same distinction. - Restore compatibility differs between pinned and host CPU profiles. A snapshot of a pinned profile needs a VMM build that pins the same revision. No build pins a host profile, so its snapshot carries the only copy, which must be a valid canonical document with the recorded digest. Both need the same effective CPUID. - Snapshot copy-on-write isolates writes, not physical pages: clean pages of a shared memory artifact may remain shared through the host page cache among the VMs that map it. - Port I/O serves the PIC, PIT, and CMOS RTC as well as the platform-control device, whose ports Table 2.5 lists. - Beyond the hypervisor identity leaves of Table 3.5, no leaf returns Hyper-V feature data, and leaf 0x40000081 carries no VS#1 interface signature. Headings are numbered and in title case, and tables and figures carry captions numbered per chapter. Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design-doc/1-architecture-overview.md | 42 ++ doc/design-doc/1.1-goals.md | 16 + doc/design-doc/1.2-non-goals.md | 22 + ...-system-components-and-responsibilities.md | 131 ++++++ doc/design-doc/1.4-trust-boundaries.md | 119 +++++ .../1.5-compatibility-and-versioning-model.md | 81 ++++ .../2-virtual-machine-architecture.md | 24 + .../2.1-machine-model-and-configuration.md | 117 +++++ doc/design-doc/2.2-cpu-and-topology.md | 156 +++++++ doc/design-doc/2.3-memory-layout.md | 63 +++ .../2.4-interrupt-and-io-architecture.md | 114 +++++ ...device-topology-transport-and-discovery.md | 46 ++ doc/design-doc/3-virtual-devices.md | 23 + doc/design-doc/3.1-platform-control-device.md | 140 ++++++ doc/design-doc/3.2-clock-and-time.md | 405 +++++++++++++++++ doc/design-doc/3.3-consoles.md | 117 +++++ doc/design-doc/3.4-block-storage.md | 98 ++++ doc/design-doc/3.5-network.md | 256 +++++++++++ doc/design-doc/3.6-host-filesystem.md | 421 ++++++++++++++++++ .../4-guest-runtime-architecture.md | 26 ++ .../4.1-guest-filesystem-and-image-model.md | 140 ++++++ .../4.2-guest-agents-and-workload-model.md | 77 ++++ .../4.3-isolation-and-resource-model.md | 72 +++ .../4.4-host-guest-control-interface.md | 115 +++++ doc/design-doc/5-vm-lifecycle-and-state.md | 45 ++ doc/design-doc/5.1-boot-protocol.md | 110 +++++ .../5.2-readiness-and-runtime-control.md | 81 ++++ doc/design-doc/5.3-snapshot-protocol.md | 171 +++++++ ...-snapshot-artifact-and-attachment-model.md | 211 +++++++++ doc/design-doc/5.5-restore-protocol.md | 331 ++++++++++++++ .../5.6-shutdown-and-failure-semantics.md | 68 +++ doc/design-doc/README.md | 20 + 32 files changed, 3858 insertions(+) create mode 100644 doc/design-doc/1-architecture-overview.md create mode 100644 doc/design-doc/1.1-goals.md create mode 100644 doc/design-doc/1.2-non-goals.md create mode 100644 doc/design-doc/1.3-system-components-and-responsibilities.md create mode 100644 doc/design-doc/1.4-trust-boundaries.md create mode 100644 doc/design-doc/1.5-compatibility-and-versioning-model.md create mode 100644 doc/design-doc/2-virtual-machine-architecture.md create mode 100644 doc/design-doc/2.1-machine-model-and-configuration.md create mode 100644 doc/design-doc/2.2-cpu-and-topology.md create mode 100644 doc/design-doc/2.3-memory-layout.md create mode 100644 doc/design-doc/2.4-interrupt-and-io-architecture.md create mode 100644 doc/design-doc/2.5-device-topology-transport-and-discovery.md create mode 100644 doc/design-doc/3-virtual-devices.md create mode 100644 doc/design-doc/3.1-platform-control-device.md create mode 100644 doc/design-doc/3.2-clock-and-time.md create mode 100644 doc/design-doc/3.3-consoles.md create mode 100644 doc/design-doc/3.4-block-storage.md create mode 100644 doc/design-doc/3.5-network.md create mode 100644 doc/design-doc/3.6-host-filesystem.md create mode 100644 doc/design-doc/4-guest-runtime-architecture.md create mode 100644 doc/design-doc/4.1-guest-filesystem-and-image-model.md create mode 100644 doc/design-doc/4.2-guest-agents-and-workload-model.md create mode 100644 doc/design-doc/4.3-isolation-and-resource-model.md create mode 100644 doc/design-doc/4.4-host-guest-control-interface.md create mode 100644 doc/design-doc/5-vm-lifecycle-and-state.md create mode 100644 doc/design-doc/5.1-boot-protocol.md create mode 100644 doc/design-doc/5.2-readiness-and-runtime-control.md create mode 100644 doc/design-doc/5.3-snapshot-protocol.md create mode 100644 doc/design-doc/5.4-snapshot-artifact-and-attachment-model.md create mode 100644 doc/design-doc/5.5-restore-protocol.md create mode 100644 doc/design-doc/5.6-shutdown-and-failure-semantics.md create mode 100644 doc/design-doc/README.md diff --git a/doc/design-doc/1-architecture-overview.md b/doc/design-doc/1-architecture-overview.md new file mode 100644 index 000000000..8506b3e02 --- /dev/null +++ b/doc/design-doc/1-architecture-overview.md @@ -0,0 +1,42 @@ +# Architecture Overview + +[Architecture documentation](README.md) + +NVX provides a small, versioned virtual machine for running Linux workloads on Linux or Windows +hosts (Figure 1.1). The guest-visible machine is owned by the NVX machine profile rather than +inferred from the selected hypervisor. KVM, MSHV, and WHP provide execution services, while the +machine profile defines one common boot, CPU, memory, device, time, and lifecycle contract. + +This chapter introduces NVX's purpose, major architectural elements, trust boundaries, and +compatibility principles. + +```mermaid +flowchart TB + Host["Host Orchestrator
configuration, policy, and attachments"] + Vmm["OpenVMM NVX Machine Profile
machine, devices, and lifecycle"] + Backend["Execution backend
KVM, MSHV, or WHP"] + Guest["Guest kernel and guest agents
boot, repair, and supervision"] + Workload["Single isolated workload"] + Artifacts["Immutable snapshot artifacts
and external attachments"] + + Host --> Vmm + Vmm --> Backend + Vmm <--> Guest + Guest --> Workload + Vmm <--> Artifacts +``` + +**Figure 1.1.** High-level view of the NVX architecture. + +## Table of Contents + +- [1.1 Goals](1.1-goals.md): Defines the properties the architecture provides. +- [1.2 Non-Goals](1.2-non-goals.md): Identifies the concerns outside the + architecture's scope. +- [1.3 System Components and Responsibilities](1.3-system-components-and-responsibilities.md): + Identifies the host, VMM, and guest components and assigns responsibility to + each. +- [1.4 Trust Boundaries](1.4-trust-boundaries.md): Describes trusted and + untrusted actors, sensitive data flows, and validation boundaries. +- [1.5 Compatibility and Versioning Model](1.5-compatibility-and-versioning-model.md): + Explains how guest-visible contracts and persisted state evolve. diff --git a/doc/design-doc/1.1-goals.md b/doc/design-doc/1.1-goals.md new file mode 100644 index 000000000..ad6550f1c --- /dev/null +++ b/doc/design-doc/1.1-goals.md @@ -0,0 +1,16 @@ +# 1.1 Goals + +[Architecture documentation](README.md) > +[1. Architecture Overview](1-architecture-overview.md) + +The architecture has the following goals: + +- Boot the same uncompressed Linux-direct kernel on Linux and Windows hosts. +- Keep the guest-visible machine independent of the selected hypervisor. +- Expose only a fixed, allowlisted set of devices. +- Capture a running VM into immutable artifacts that can be restored in a new + process without serializing native host handles. +- Run exactly one workload per microVM. +- Assemble the workload filesystem from an immutable image layer, private + writable scratch, and optional live host shares. +- Keep workload policy and host-resource attachment under host control. diff --git a/doc/design-doc/1.2-non-goals.md b/doc/design-doc/1.2-non-goals.md new file mode 100644 index 000000000..b45e19167 --- /dev/null +++ b/doc/design-doc/1.2-non-goals.md @@ -0,0 +1,22 @@ +# 1.2 Non-Goals + +[Architecture documentation](README.md) > +[1. Architecture Overview](1-architecture-overview.md) + +The architecture deliberately excludes: + +- Firmware boot, caller-defined ACPI and SMBIOS tables or device trees, PCI, vPCI, VMBus, and + arbitrary devices. +- SMT, multiple NUMA nodes, non-x86 guests, and nested virtualization. +- General CPU or memory hotplug after readiness. +- Restore across hypervisor backends or CPU generations, live migration, and + capture-and-continue. +- Scaling of the TSC, trapping of RDTSC, paravirtual clocks, and a virtual PMU. +- Serialization of live network flows, native host handles, or the contents of + a live host filesystem export. +- Multi-container orchestration, including pod infrastructure, pause + containers, dynamic root filesystem injection, and host calls that create + additional containers. +- Confidential computing, because the host is trusted with image content and + guest memory. +- Tenant authorization inside the VMM. diff --git a/doc/design-doc/1.3-system-components-and-responsibilities.md b/doc/design-doc/1.3-system-components-and-responsibilities.md new file mode 100644 index 000000000..bff0d4332 --- /dev/null +++ b/doc/design-doc/1.3-system-components-and-responsibilities.md @@ -0,0 +1,131 @@ +# 1.3 System Components and Responsibilities + +[Architecture documentation](README.md) > +[1. Architecture Overview](1-architecture-overview.md) + +NVX spans three layers (Figure 1.2), and their responsibilities follow the +boundary between policy and mechanism. Host software owns policy, +orchestration, placement, identity, and attachment authorization. The OpenVMM +virtual machine monitor (VMM) owns the virtual platform and transition safety. +Guest software owns workload construction, supervision, and guest-visible +repair, while the workload owns only its execution inside the resulting +sandbox. + +```mermaid +flowchart TB + subgraph HostLayer["Host"] + direction LR + Launcher["NVX launcher"] + LifecycleApi["Sandbox lifecycle API"] + ControlLibrary["Host control library"] + Storage["Snapshot storage"] + Attachments["External attachments"] + end + + subgraph VmmLayer["OpenVMM"] + direction LR + Frontends["Command-line entry
and management RPC"] + Profile["Machine profile"] + Worker["VM worker"] + Chipset["Chipset and
platform devices"] + Virtio["Virtio devices"] + Publisher["Snapshot publisher"] + Backends["Hypervisor backends"] + end + + subgraph GuestLayer["Guest"] + direction LR + Kernel["Linux kernel"] + Agents["Guest agents"] + Workload["Workload"] + end + + Launcher --> Frontends + LifecycleApi --> ControlLibrary + ControlLibrary --> Frontends + Launcher -. control session .-> Virtio + ControlLibrary -. control session .-> Virtio + Frontends --> Profile + Frontends --> Worker + Profile --> Worker + Worker --> Chipset + Worker --> Virtio + Worker --> Publisher + Worker --> Backends + Publisher --> Storage + Attachments --> Virtio + Chipset <--> Kernel + Virtio <--> Kernel + Kernel --> Agents + Agents --> Workload +``` + +**Figure 1.2.** Major host, VMM, and guest components. + +## 1.3.1 Host Components + +- **NVX launcher** runs one-shot microVMs and the managed sandbox lifecycle, + keeps per-sandbox state, prechecks host shares and places them in the guest, + and compiles structured egress-policy files into VMM egress rules. +- **Sandbox lifecycle API** exposes provision, start, exec, stop, and + deprovision operations with request validation, backend capabilities, a + serializable contract model, durable sandbox records, and stable error codes. +- **Host control library** implements the host side of the control protocol, + artifact discovery, blockless managed launch, VMM process supervision, + host-path export with its access policy, delivery of the host mapping table, + and egress-rule translation. +- **Snapshot storage** retains immutable machine state, memory, and optional + paired scratch together with the identities needed to reconstruct external + resources. +- **External attachments** supply process-local resources such as console + endpoints, network endpoints, host filesystem exports, and block files. + +## 1.3.2 VMM Components + +- **Command-line entry and management RPC** are the VMM's frontends. They + construct host attachments, create, capture, and restore microVMs, report + readiness and guest exit, drain the byte channel at exit, and write outcome + reports. +- **Machine profile** selects and validates the machine, defines its ABI + constants, and owns the effective kernel command line. It holds the pinned + CPU profiles, selects or derives each VM's CPU profile, checks backend + support and restore compatibility, and computes each VM's effective CPUID. +- **VM worker** composes and runs the machine. It creates the partition and + its virtual processors, reserves RAM capacity and maps captured memory + privately with copy-on-write, loads the guest through the Linux direct + loader, and places the fixed devices. Its state-unit framework quiesces, + starts, rolls back, and inventories stateful components in dependency order, + and the worker owns snapshot boundaries, management exclusion, clock + contracts, and restore sequencing. +- **Chipset and platform devices** provide the allowlisted chipset: the PIC, the IOAPIC, the PIT, + the RTC, and the byte-channel, shutdown, and + snapshot-request ports. +- **Virtio devices** comprise the consoles, block, network, and host + filesystem devices, with shared interrupt status, queue processing, and + device-private saved state. Their backends include the control-console + broker, which authenticates the local host peer and frames the control + session, the in-process network endpoint, which enforces egress policy, and + the host filesystem server, which enforces each export's access policy. +- **Snapshot publisher** defines the snapshot format and machine contract, + publishes artifacts, and validates them on restore. +- **Hypervisor backends** implement partitions, processor execution, interrupt + injection, CPU contracts, and snapshot clocks for KVM, MSHV, and WHP. + +## 1.3.3 Guest Components + +- **Linux kernel** boots from the fixed Linux-direct handoff, drives the + allowlisted virtual devices, and provides the namespaces, filesystems, + cgroups, and process primitives used by the runtime. +- **Guest agents** mount runtime filesystems, apply boot configuration, mount + live host shares, and assemble the workload filesystem. They construct the + workload's root and credentials, and in the layered sandbox root its + namespaces and device view. They create workload accounts when requested, + supervise one-shot or managed workloads, prepare snapshots, and repair + processors, memory, entropy, and identity after restore. They also announce + the guest's listeners when the host publishes them, check time ABI + conformance, watch for clock violations, discipline the wall clock, and + perform the snapshot time steps. +- **Workload** is the microVM's single tenant: it owns the microVM's lifetime, + resource envelope, and network identity, and it may run any number of + processes, all inside the isolation boundary that the guest agents + construct. It has no direct authority over host resources or VMM policy. diff --git a/doc/design-doc/1.4-trust-boundaries.md b/doc/design-doc/1.4-trust-boundaries.md new file mode 100644 index 000000000..69afe4af6 --- /dev/null +++ b/doc/design-doc/1.4-trust-boundaries.md @@ -0,0 +1,119 @@ +# 1.4 Trust Boundaries + +[Architecture documentation](README.md) > +[1. Architecture Overview](1-architecture-overview.md) + +As Figure 1.3 shows, the host and its orchestrator are trusted with guest +memory, image contents, and attachment policy, while the workload is +untrusted. + +```mermaid +flowchart LR + Control["Trusted host control plane"] + Vmm["VMM trust boundary"] + Agent["Guest agent boundary"] + Workload["Untrusted workload"] + Resources["Host resources"] + + Control -->|"policy and capabilities"| Vmm + Resources -->|"validated attachments"| Vmm + Vmm <-->|"bounded device protocols"| Agent + Agent -->|"isolation, identity,
resources, and supervision"| Workload +``` + +**Figure 1.3.** Trust boundaries of the host, VMM, guest agents, and workload. + +Because each microVM has a single tenant, the virtual machine boundary is also +the boundary between workloads. A workload may use ordinary Linux interfaces +inside the guest, but it gains no authority to reconfigure the virtual +platform or host attachments. + +From the VMM's perspective, guest physical addresses, port and MMIO accesses, virtio descriptors, +packet data, filesystem requests, control-session records, and the timing of a snapshot request are +all untrusted inputs. + +Kernel images, initramfs images, command lines, snapshot artifacts, +restore-time attachments, and host control-console clients also cross +validation boundaries. Authority to provide an input does not make its +structure valid. OpenVMM therefore validates the machine configuration before +opening host resources, validates it again at worker construction, and +validates all persisted state before any restored virtual processor runs. + +## 1.4.1 Defense in Depth + +Invalid configurations are rejected, never silently adjusted. Every contract +is enforced without silent fallbacks: a host, snapshot, or guest that cannot +meet a contract is rejected, or fails fast, with a stable error. The single +fallback, from automatic CPU-profile selection to a host-derived CPU profile, +warns at every cold boot that takes it. + +The VMM uses checked address arithmetic, bounded buffers and tables, typed +validation errors, rate-limited guest-triggerable logs, and fallible quiesce +and restore operations. Native I/O callbacks only record bounded state or +enqueue notifications; they never perform blocking snapshot or host-resource +work. Malformed saved device state is rejected before device workers start, +and no virtual processor runs after a partial restore failure. A failure after +processor stopping begins at a snapshot or post-restore acknowledgement +boundary is terminal: host input stays gated, and the VM is torn down instead +of resuming with uncertain state. + +## 1.4.2 Host Control + +A host control client reaches the guest only through the control-console +broker. The broker admits one local peer whose operating-system identity is +the VMM's user, requires a launcher-provided capability before any host data +reaches the guest, and bounds every record, queue, and credit window in both +directions. Malformed, stale, or out-of-order records fail closed. +Capabilities are process-local authority: they never appear in arguments, +environment variables, logs, endpoint names, snapshots, or attachment +identities. + +## 1.4.3 Device-Enforced Policy + +Host devices enforce policy independently of the guest. Read-only block roles +are advertised as read-only, reject writes in the device, and are opened +without host write access. The network endpoint authorizes the exact owned +copy of every transmitted frame. It accepts inbound connections only on the +host listeners that the host's inbound access allows, for published guest +ports or explicit forwards, and only from the clients that this access admits. +The guest's announcements choose which of its ports are published, never the +host addresses or the clients. The host filesystem server applies each +export's access mode and path policy on every request, whichever guest mount +or link reaches it. Guest mount flags, routes, and links are therefore never +security boundaries. + +## 1.4.4 Images and Snapshot Artifacts + +Image layers are trusted inputs: the overlay filesystem interprets metadata +stored in lower layers, and bytes read from any layer can appear in captured +memory. A snapshot identifies its blocks by a storage generation, which +transfers trust for layer and scratch contents to the storage owner; the VMM +trusts the control plane in the host to authenticate that generation. + +The VMM trusts the host control plane not to give the same writable file, such +as a caller-supplied RAM backing or scratch file, to more than one VMM. It +takes no file locks to detect such sharing, and the RAM backing that it +creates itself is a new file for every VM. Snapshot artifacts are shared by +design: every VM restored from a snapshot maps the same immutable memory +artifact with copy-on-write, so each VM's changes stay private and never reach +the artifact. A capture always publishes new files, and a restored VM is not +captured again. + +Copy-on-write isolates writes, not physical pages. Clean pages of a shared +memory artifact may remain shared through the host page cache among the VMs +that map it, and guest kernel, guest agent, and cached workload pages can all +originate in captured memory. Snapshot copy-on-write is therefore not, by +itself, a tenant authorization or physical-page isolation mechanism. Artifact +access control, sharing scope, backing-file placement, retention, and host +memory deduplication remain deployment responsibilities. + +## 1.4.5 Host-Facing Outputs + +Host-facing outputs are bounded. The optional outcome report is written after +teardown to a previously absent local path. It records only a schema version, +an opaque instance ID, the backend category, the operation and outcome +category with a numeric status, whether the network policy was requested and +applied with its mode, rule counts, and host-loopback action, and per-step +teardown flags. It never contains commands, environment values, paths, +network destinations, proxy details, workload output, credentials, or +free-form error text, and the VMM does not upload it. diff --git a/doc/design-doc/1.5-compatibility-and-versioning-model.md b/doc/design-doc/1.5-compatibility-and-versioning-model.md new file mode 100644 index 000000000..860545f14 --- /dev/null +++ b/doc/design-doc/1.5-compatibility-and-versioning-model.md @@ -0,0 +1,81 @@ +# 1.5 Compatibility and Versioning Model + +[Architecture documentation](README.md) > +[1. Architecture Overview](1-architecture-overview.md) + +Table 1.1 lists the explicit contracts that define NVX compatibility. Each +contract carries its own version or identity, independent of the others. + +**Table 1.1.** Compatibility contracts. + +| Contract | Governs | +| --- | --- | +| Machine ABI | CPU topology, memory layout, device topology, addresses, interrupts, transport, and guest-visible policy. | +| Boot layout | Placement and format of the Linux-direct boot structures: MP tables, bootstrap GDT and page tables, Linux boot-parameter block (zero page), and kernel command line. | +| CPU profile | Guest-visible CPU surface of one vendor and CPU generation, identified by vendor, generation, and revision, and by the digest of its canonical encoding. | +| Time ABI | Hypervisor identity, frequency interfaces, clocksources, timers, downtime semantics, and restore-time repair. The guest reads its version from the CPUID functions reserved for hypervisor information (hypervisor identity leaves). | +| Device-private schemas | Saved queue and device state that must be understood before a device can resume. | +| Snapshot format | Persisted machine contract, manifest, state inventory, memory geometry, attachment requirements, and artifact identity. | +| RAM expansion capability | Recorded RAM capacity, memory-block size, and expansion ranges that restore may activate. | +| Restore packet | Layout and meaning of the one-time record that gives a restored guest its generation, downtime, rate deviation, activation targets, and fresh entropy. | +| Control protocol | Authenticated sessions between the host and the guest agents, request framing, operation semantics, and lifecycle epochs. | +| Management RPC machine profile | Identity of the microVM machine profile in the management RPC. | + +A change to a guest-visible address, interrupt, command-line token, feature +mask, queue shape, time rule, or device behavior requires a new contract +version. A backend-specific difference is valid only when the versioned +contract states it explicitly, as it does for the virtio-net interrupt. + +Guest-visible configuration is immutable for the lifetime represented by a +snapshot. Restore derives the machine from the snapshot manifest; caller input +may reconstruct permitted external resources but cannot silently change the +captured CPU surface, memory capacity, command line, device placement, feature +masks, or network identity. + +Compatibility is exact rather than best effort. Unknown versions, missing or +extra devices, changed queue shapes, incompatible CPU surfaces, malformed +state, or mismatched attachments are rejected before guest execution. A +snapshot with an unsupported manifest version must be recaptured. + +## 1.5.1 Restore Compatibility + +A snapshot restores only: + +- On the backend that captured it. +- On a host whose CPU belongs to the generation of the snapshot's CPU profile. +- On a host whose native TSC rate lies within 250 ppm of the snapshot's + [declared rate](3.2-clock-and-time.md#324-rates). +- With a VMM build that pins the same CPU-profile revision. No build pins a + [host CPU profile](2.2-cpu-and-topology.md#2212-cpu-profile-selection), so + its snapshot instead carries the only copy, which must be a valid canonical + document with the recorded digest. +- With a VMM build that computes the same effective CPUID for that CPU profile + and topology. +- With external resources that reproduce the recorded attachment identities. + +Released CPU-profile revisions are immutable and never deleted, so existing +snapshots stay restorable; a normative change creates a new revision. + +## 1.5.2 Template Compatibility + +A reusable template represents a compatibility class, not an arbitrary fleet +image. The class includes the backend, the CPU profile and generation, the +effective CPUID, the guest kernel and guest agent build, the effective command +line, device roles and geometry, network identity and policy, and consumed +layer identities. Restoring guest memory restores the captured guest agent +build, not a newly deployed initramfs, so deployments rebuild or recertify +templates on relevant rollouts. + +Processor capacity and RAM geometry are immutable, but restore-time activation +lets one template serve several online processor counts and RAM sizes. +Templates are therefore keyed by capacities, base state, and supported +activation ranges. Layer-role presence and device geometry stay fixed: a +platform template accepts a substitute layer only with the same recorded +logical length and block sizes. + +## 1.5.3 Persisted State + +The snapshot manifest is authoritative for a restored process. Native file +descriptors, sockets, named-pipe handles, and hypervisor objects are never part +of the portable state. They are recreated from stable identities and declared +reconstruction policies. diff --git a/doc/design-doc/2-virtual-machine-architecture.md b/doc/design-doc/2-virtual-machine-architecture.md new file mode 100644 index 000000000..20848c9c9 --- /dev/null +++ b/doc/design-doc/2-virtual-machine-architecture.md @@ -0,0 +1,24 @@ +# Virtual Machine Architecture + +[Architecture documentation](README.md) + +The NVX machine profile presents a fixed Linux platform on every supported backend. The machine +profile is an allowlist: it constructs the required machine directly rather than creating a standard +PC and removing devices. + +This chapter defines the backend-independent virtual platform presented to the guest. + +## Table of Contents + +- [2.1 Machine Model and Configuration](2.1-machine-model-and-configuration.md): + Defines the machine identity, invariant configuration, and ownership of + guest-visible settings. +- [2.2 CPU and Topology](2.2-cpu-and-topology.md): Describes the virtual + processor model, topology, feature surface, and processor capacity. +- [2.3 Memory Layout](2.3-memory-layout.md): Defines RAM, reserved regions, MMIO + apertures, and capacity behavior. +- [2.4 Interrupt and I/O Architecture](2.4-interrupt-and-io-architecture.md): + Describes interrupt routing, port I/O, MMIO, and device notification. +- [2.5 Device Topology, Transport, and Discovery](2.5-device-topology-transport-and-discovery.md): + Defines fixed device identities, addresses, interrupts, transport, and + discovery. diff --git a/doc/design-doc/2.1-machine-model-and-configuration.md b/doc/design-doc/2.1-machine-model-and-configuration.md new file mode 100644 index 000000000..d3701baa2 --- /dev/null +++ b/doc/design-doc/2.1-machine-model-and-configuration.md @@ -0,0 +1,117 @@ +# 2.1 Machine Model and Configuration + +[Architecture documentation](README.md) > +[2. Virtual Machine Architecture](2-virtual-machine-architecture.md) + +The machine profile is selected explicitly and owns all guest-visible +configuration. Machine identity is never inferred from a kernel, device, or +hypervisor choice: every frontend, including the command line, the management +RPC, and snapshot configuration, carries it to the VM worker. Validation occurs +before host resources are opened and again at the worker boundary. + +The current machine profile's architectural invariants are: + +- An x86-64 guest with Linux direct boot and Intel MP 1.4 tables, without firmware. +- One NUMA node. +- One socket, one die, one core per virtual processor, no SMT, and xAPIC mode. +- 1, 2, 4, or 8 virtual processors with contiguous APIC IDs starting at zero. +- A fixed low-memory boot layout and a fixed 3-to-4-GiB MMIO aperture. +- The [time ABI](3.2-clock-and-time.md) as the only time path, with one CPU profile. +- A fixed chipset and virtio-mmio device topology. +- No VTL2, confidential-VM isolation, or nested virtualization. +- No paravirtual features from the backend's hypervisor, such as Hyper-V + enlightenments or the KVM paravirtual clock. The time ABI's minimal + hypervisor identity, the same on every backend, is the only hypervisor + interface. + +## 2.1.1 Allowlisted Platform + +The machine profile constructs the required machine directly instead of creating a standard PC and +removing unwanted devices. + +It rejects firmware and boot interfaces: UEFI, PCAT, IGVM, and caller-supplied ACPI, SMBIOS, and +device trees. It also rejects PCI and PCIe, vPCI, VMBus, the DMA controller of the legacy ISA +expansion bus (ISA DMA), IDE, floppy, VMGS, graphics, VGA firmware, debugger resources, and every +device outside the machine profile. The machine profile itself emits the fixed Linux direct +MP-table metadata. + +## 2.1.2 Configuration Ownership + +Table 2.1 lists the owner and lifetime of each kind of configuration. A +lifetime is one of: + +- **Contract version.** The same for every VM of one machine-contract version. +- **VM.** Fixed at cold boot, recorded in every snapshot, and restored from it + unchanged. +- **VM process.** Bound at cold boot and again at every restore, matching the + identities that the VM recorded. +- **Restore.** Selected for each restore, within the limits that the snapshot + records. + +**Table 2.1.** Configuration ownership and lifetime. + +| Configuration | Owner | Lifetime | +| --- | --- | --- | +| Boot layout, device placement, interrupts, and feature masks | Machine profile | Contract version | +| Processor capacity, base RAM and optional RAM capacity, and CPU profile | Host, validated by the machine profile | VM | +| Effective kernel command line | Machine profile, from host policy and caller arguments | VM | +| Workload identity and lifecycle | Host | VM | +| Console, network, block, and host filesystem devices, with their identities and policies | Host | VM | +| Host attachments: console endpoints, the network endpoint, block files, and exported host directories | Host, matching the recorded identities | VM process | +| Online processor and RAM activation targets | Host, within snapshot limits | Restore | + +The host fixes the workload policy of a fresh boot. An optional nonzero +numeric UID and GID become the fixed workload identity, and an optional +one-shot or managed lifecycle selects whether the guest agents run one +workload or stay resident for sequential requests over the control console. +Both are command-line tokens that the machine profile owns and that caller +arguments cannot supply. A managed lifecycle requires a fixed workload +identity and a live, authenticated control console. + +## 2.1.3 Storage Configuration + +Sandbox storage uses block devices with fixed roles rather than arbitrary +disks. The roles are the layers of the sandbox root filesystem: a read-only +`distro` lower layer for the base operating system, beneath one writable +`scratch` layer that holds the sandbox's private changes. +[Block Storage](3.4-block-storage.md) describes each role. + +A disk without a role is rejected, and a disk-backed topology ends with +writable scratch. The only exception is the RAM-backed overlay layout, which +requires exactly one read-only `distro` block and no scratch, keeps the +guest's overlay upper layer in RAM, and supports neither capture nor restore. +Snapshot capture with blocks requires a read-only `distro` layer followed by +writable scratch, both backed by cached, regular raw files with nonzero, +512-byte-aligned geometry. Blockless machines are also snapshot-capable and +carry no sandbox tier metadata. + +Host filesystem sharing uses the fixed filesystem slot rather than adding +devices. The slot exports one host directory, or an aggregate of named +directory or regular-file children, each with its own access mode, beneath a +read-only synthetic root. + +## 2.1.4 Restore-Time Configuration + +Restore is stricter than cold boot. Guest-visible configuration is read from +the snapshot's machine contract. Restore-time input may only: + +- Select the backend kind that the snapshot records. +- Supply required attachments. +- Bind a new directory export to a captured dormant filesystem slot, when the + snapshot records that capability. +- Select processor and RAM activation targets that the contract allows. + +These process-local inputs never change processor capacity, RAM capacity, the +command line, device placement, feature masks, or filesystem and network +identity. Restore always uses the snapshot's CPU profile and never falls back. +Because the captured command line fixes network addressing, workload identity, +and lifecycle, restore rejects inputs that would change them, and a supplied +processor count may only repeat the captured capacity. + +## 2.1.5 Execution Backends + +The execution backend is a property of the host process, not of the guest +platform. Backend code supplies partition creation, virtual processor +execution, memory registration, and interrupt injection while preserving the +same guest-visible contract. KVM and MSHV run on Linux hosts, and WHP runs on +Windows hosts. diff --git a/doc/design-doc/2.2-cpu-and-topology.md b/doc/design-doc/2.2-cpu-and-topology.md new file mode 100644 index 000000000..a456c0a42 --- /dev/null +++ b/doc/design-doc/2.2-cpu-and-topology.md @@ -0,0 +1,156 @@ +# 2.2 CPU and Topology + +[Architecture documentation](README.md) > +[2. Virtual Machine Architecture](2-virtual-machine-architecture.md) + +As Table 2.2 shows, every VM has an immutable processor capacity of 1, 2, 4, +or 8 virtual processors and a deterministic topology. + +**Table 2.2.** Processor topology. + +| Property | Contract | +| --- | --- | +| NUMA nodes | One | +| Sockets | One | +| Dies | One | +| Threads per core | One | +| Cores | Equal to processor capacity | +| APIC mode | xAPIC | +| APIC IDs | Contiguous, starting at zero | + +## 2.2.1 CPU Profiles + +The guest sees a CPU profile rather than host passthrough. A CPU profile is +the complete guest-visible CPU surface of one vendor and CPU generation, +shared by every backend. In every governed CPUID entry the guest sees only +CPU-profile values, except for a fixed set of VMM-owned fields and the fields +that the time ABI owns. One CPU profile gives the same guest behavior on every +backend; it does not make snapshots portable across backends. + +A CPU profile is versioned data with one canonical encoding and a SHA-256 +digest. Its identifier has the form `..v`. It +contains: + +- The CPU vendor and the generation's CPU signatures, as ranges of the + numbers that identify a processor design and its revision (family, model, + and stepping). +- A dense CPUID table covering every basic and extended leaf and each leaf's + secondary indexes (subleaves) up to the CPU profile's maxima, with a value + and a mask per register, where a set mask bit pins the value and a clear bit + marks a VMM-owned or runtime-owned bit. +- The extended processor state save and restore (XSAVE) components that the + guest may enable, such as vector registers, with each component's size, + offset, and flags. +- The guest physical address width. +- Pinned model-specific register (MSR) values with masks. +- Informative provenance. + +CPU profiles are derived by intersecting host fingerprints per register, first +within each backend and then across backends. Released revisions are +immutable and never deleted; a normative change creates a new revision. + +As Table 2.3 shows, every CPUID bit has exactly one owner. + +**Table 2.3.** Ownership of CPUID fields. + +| Owner | Fields | +| --- | --- | +| CPU profile | Every governed value whose mask bit is set, and pinned MSRs | +| VMM | Logical processor count and initial APIC ID, the bit that advertises the APIC's MSR-based mode (x2APIC), core and cache-sharing counts, and the processor topology leaves | +| Guest runtime state | Bits that report whether the guest kernel has enabled XSAVE and memory protection keys (`OSXSAVE` and `OSPKE`), and the XSAVE sizes for the current values of the registers that select the enabled user and supervisor state components (XCR0 and XSS) | +| Time ABI | Leaves `0x40000000` through `0x4fffffff`, `0x15`, `0x16`, and the [CPU time bits](3.2-clock-and-time.md#323-cpu-time-bits) | +| Policy | Zeros for virtualization extensions, Intel's secure enclaves (SGX), processor trace, cache and memory-bandwidth allocation (resource director technology), memory encryption, and other excluded features | + +Because the machine has one socket and one core per virtual processor, every +cache reports the processor count less one as its cores per socket, the L3 +cache is shared by every processor, and each L1 and L2 cache serves one +processor. + +### 2.2.1.1 Effective CPUID + +The effective CPUID is a pure function of the CPU profile, the VM topology, +and the time ABI's identity leaves. It lists exactly the governed entries: +every leaf and subleaf of the CPU profile's table; the topology leaves at the +subleaves that the topology defines, plus the subleaf that terminates them; +and the hypervisor identity leaves with their explicit zero leaves. The VMM +computes it, records it in the snapshot manifest, and recomputes it on +restore. The record therefore never depends on how a backend enumerates or +caches CPUID, and one CPU profile and topology produce the same record on +every backend. Because the record also fixes the VMM's own leaves and +VMM-owned bits, changing how the VMM computes them breaks snapshot +compatibility. + +Entries that only probing past an advertised maximum reaches are reserved: +leaves above the maximum basic or extended leaf, subleaves past a leaf's +maximum or terminator, and XSAVE subleaves of disabled components. Their +values are implementation-defined, and guests must not rely on them; +verification still keeps host features out of them. + +Some fields are informational and are not compared: the processor name that +CPUID reports (the brand string) is generic per generation and names no +frequency, cache descriptor leaves come from reference hosts, and the +microcode revision is not pinned. + +Mitigation-relevant bits follow the hardware. A CPU profile claims an immunity +or a control only if every CPU of its generation has it and every backend can +present it, so guests keep their mitigations enabled. + +### 2.2.1.2 CPU Profile Selection + +Every microVM has a CPU profile. A cold boot selects a pinned CPU profile +explicitly, selects one automatically, or uses a host CPU profile: + +- Automatic selection uses the highest revision of the single pinned profile + whose generation covers the host's vendor, family, model, and stepping. +- On an Intel or AMD CPU that no pinned profile serves, automatic selection + falls back to a host profile and warns at every cold boot that takes the + fallback. +- A CPU of another vendor, an ambiguous catalog match, or a pinned profile + that the backend cannot support fails instead of falling back. + +A host CPU profile is derived at cold boot from a fingerprint of the backend +on the current host, under the same derivation policy as the pinned profiles. +The guest still sees a policy-filtered surface rather than host CPUID: only +permitted data leaves keep data, policy features are cleared, XCR0 stays +within the permitted components, the brand string is generic, and the +generation covers only the host's family, model, and stepping. A host profile +is neither reviewed nor immutable, because firmware, microcode, hypervisor, or +operating-system updates can change it. Its snapshots carry the host +profile's document and restore only on a host of the same CPU model and +stepping whose hypervisor supports it. + +A restore always uses the CPU profile that the snapshot records and never +falls back. + +### 2.2.1.3 CPU Profile Verification + +At partition creation, for cold boot and restore, the VMM verifies that: + +1. The CPU profile is valid, including its canonical encoding, density, + ownership masks, and CPU time bits. +2. The host's vendor, family, model, and stepping belong to the CPU profile's + generation. +3. The backend supports the CPU profile, including every set feature bit, + every limit, the XSAVE layout, the XCR0 and XSS subsets, and every pinned + MSR value, with the time policy bits exempt. +4. On restore, the VMM pins the same CPU profile ID and digest, or the + recorded host profile is a valid canonical document with the recorded + digest, and the recomputed effective CPUID equals the recorded one. +5. The backend presents the effective CPUID on the first virtual processor + under its masks. +6. On backends that pass reserved entries through, no nonzero entry exists + outside the CPU profile's tables, the identity range, and the topology + leaves. + +Every violation is reported together, naming the leaf, subleaf, register, and +bit. + +## 2.2.2 Processor Capacity + +Processor capacity is part of machine identity and stays fixed for the life +of a VM. The processor topology, APIC IDs, MP tables, and saved processor +inventory always describe exactly the capacity, even when a snapshot captures +fewer online processors, so bringing more processors online at restore never +changes the topology. +[Processor Activation](5.5-restore-protocol.md#553-processor-activation) +describes how a restore selects its online processors. diff --git a/doc/design-doc/2.3-memory-layout.md b/doc/design-doc/2.3-memory-layout.md new file mode 100644 index 000000000..080e729ec --- /dev/null +++ b/doc/design-doc/2.3-memory-layout.md @@ -0,0 +1,63 @@ +# 2.3 Memory Layout + +[Architecture documentation](README.md) > +[2. Virtual Machine Architecture](2-virtual-machine-architecture.md) + +As Figure 2.1 shows, guest RAM is continuous in backing-file offset but split +in guest physical address by the fixed MMIO aperture. + +```mermaid +flowchart LR + Low["Low RAM
0x00000000 through 0xbfffffff
up to 3 GiB"] + Gap["Fixed MMIO aperture
0xc0000000 through 0xffffffff
1 GiB"] + High["High RAM
0x100000000 and above"] + Slots["Reserved virtio-mmio slots
0xd0000000 through 0xd0007fff"] + + Low --- Gap + Gap --- High + Gap -. contains .-> Slots +``` + +**Figure 2.1.** Guest physical memory layout. + +Active RAM occupies `[0, min(size, 3 GiB))`. Memory displaced by the one-GiB +MMIO aperture resumes at 4 GiB. There is no high-MMIO or VTL2 aperture. One +memory-layout engine owns this split, using the chipset's fixed MMIO +reservation. Its active RAM ranges are also the snapshot memory-range +inventory and the authoritative PC memory map that tells the Linux kernel +which physical address ranges are RAM and which are reserved (the e820 map). + +Table 2.4 lists the fixed locations of the low-memory boot structures. + +**Table 2.4.** Fixed low-memory boot structures. + +| Guest physical address | Contents | +| ---: | --- | +| `0x0000..0x000f` | Intel MP 1.4 floating pointer | +| `0x0400` | MP configuration table, `180 + 20 * processor count` bytes | +| `0x1000` | Bootstrap GDT | +| `0x2000` | Linux `boot_params` zero page | +| `0x4000..0x9fff` | 4-GiB identity page tables: one table for each of the top two levels of the x86-64 page-table hierarchy (PML4 and PDPT), and four directories of 2-MiB pages | +| `0x20000..0x2ffff` | NUL-terminated kernel command line | +| `0x30000..0x30fff` | Shared virtio interrupt-status page | +| `0x100000` and above | Kernel load segments and ordinary RAM | + +The e820 map reserves the live interrupt-status page and the legacy PC range +from 640 KiB to 1 MiB that holds video memory and firmware (the ISA hole), +splits low and high RAM around the aperture, and never reports the aperture +as RAM. Loader writes and DMA ranges must fit wholly inside a RAM range and +may not cross the MMIO aperture or a reserved boot structure. + +## 2.3.1 Memory Capacity + +Snapshot capture may declare an immutable RAM capacity at least as large as +the active base RAM; both values are then 128-MiB aligned. The layout engine +reserves addresses for the full capacity but publishes only the active base +prefix as RAM, so selecting a larger restore target never moves the MMIO +aperture or any device. The machine contract records the canonical suffix +ranges that are absent from the captured e820 map and memory artifact. A +suffix that crosses the 3-GiB boundary is represented as separate low- and +high-RAM ranges. A snapshot without a declared capacity cannot select a +different RAM size at restore. +[Memory Activation](5.5-restore-protocol.md#554-memory-activation) describes +the restore-time sequence. diff --git a/doc/design-doc/2.4-interrupt-and-io-architecture.md b/doc/design-doc/2.4-interrupt-and-io-architecture.md new file mode 100644 index 000000000..5aa66575a --- /dev/null +++ b/doc/design-doc/2.4-interrupt-and-io-architecture.md @@ -0,0 +1,114 @@ +# 2.4 Interrupt and I/O Architecture + +[Architecture documentation](README.md) > +[2. Virtual Machine Architecture](2-virtual-machine-architecture.md) + +As Figure 2.2 shows, interrupts and I/O follow separate paths. Devices signal +interrupts on interrupt request lines (IRQs) of the PIC and IOAPIC, which +deliver them to each processor's local advanced programmable interrupt +controller (LAPIC). The LAPIC also provides a timer. Processors reach devices +through port I/O and MMIO, while bulk data and interrupt status move through +guest RAM. + +```mermaid +flowchart TB + Vcpu["Virtual processors, each with a LAPIC"] + Ports["Platform-control ports"] + Virtio["Fixed virtio-mmio slots"] + Timers["PIT and RTC"] + Ram["Guest RAM
interrupt-status page
and descriptor chains"] + Controllers["PIC and IOAPIC
16 ISA interrupt lines"] + + Vcpu -->|"port I/O"| Ports + Vcpu -->|"MMIO configuration
and notification"| Virtio + Vcpu <-->|"descriptors and
status words"| Ram + Virtio <-->|"bulk data and
status bits"| Ram + Virtio -->|"dedicated edge IRQs"| Controllers + Timers -->|"IRQ 0 and IRQ 8"| Controllers + Controllers -->|"interrupts"| Vcpu +``` + +**Figure 2.2.** Interrupt and I/O paths. + +## 2.4.1 Architectural Devices + +The base chipset contains exactly this allowlist: + +- A generic PIC and IOAPIC. +- The selected backend's LAPIC. +- An i8253 PIT on IRQ 0. +- A generic programmable CMOS RTC anchored to UTC. +- The raw bidirectional byte channel. +- A status-carrying shutdown port. +- A guest snapshot-request port. + +Serial UARTs, debug-console ports, Hyper-V power management, the legacy +joystick port (gameport), PCI, firmware helpers, and stubs that answer guest +accesses to absent legacy ports (standard-PC missing-port shims) are absent. + +The RTC defaults to binary-coded decimal (BCD), 24-hour fields, with status +register B `0x02`. The PIC, IOAPIC, PIT, RTC, LAPIC, and VM time use common +device and state-unit machinery on every backend. IOAPIC saved state includes +the asserted level of every input line, and routing is reevaluated after +restore, so a level interrupt is neither lost nor treated as an edge while the +backend is reconstructed. The [time ABI](3.2-clock-and-time.md) defines the +behavior of the LAPIC timer, PIT, and RTC. + +## 2.4.2 Interrupt Routing + +The MP table describes processors, the ISA bus, the IOAPIC, and legacy IRQ +routing, and it routes only the 16 ISA interrupts. The machine profile uses no +ACPI interrupt tables. Dedicated virtio interrupts use active-high edge +delivery. No virtio device uses the RTC's exclusive IRQ 8, and no device uses +IRQ 9, 12, or 13 on any backend. + +## 2.4.3 Port I/O + +Port I/O serves the PIC, PIT, and CMOS RTC as well as the minimal +platform-control device. Table 2.5 lists the platform-control ports. + +**Table 2.5.** Platform-control ports. + +| Port | Function | +| ---: | --- | +| `0xe9` | Byte-channel data | +| `0xea` | Byte-channel status and record selector | +| `0xeb` | Time-sample window | +| `0x604` | Shutdown | +| `0x605` | Snapshot request and restore acknowledgement | + +[Platform Control Device](3.1-platform-control-device.md) defines their +behavior. + +## 2.4.4 Shared Interrupt Status + +As Table 2.6 shows, each fixed virtio-mmio slot owns one little-endian, +naturally aligned 32-bit word in the reserved shared-status page. + +**Table 2.6.** Shared interrupt-status words. + +| Slot | Status GPA | +| --- | ---: | +| Network | `0x30000` | +| Host filesystem | `0x30004` | +| Boot console | `0x30008` | +| `distro` block | `0x3000c` | +| `scratch` block | `0x30018` | +| Control console | `0x3001c` | + +A device publishes configuration-change and used-buffer bits with a +sequentially consistent compare-exchange loop and pulses its interrupt only +when the previous word was zero. The guest driver consumes every pending bit +with a sequentially consistent exchange to zero, so steady-state handling +performs no interrupt-status read or acknowledgement through MMIO. A device +update that races the exchange is either included in the exchanged value or +observes zero, stores its bit, and emits a new edge. Device reset and teardown +clear the word. Snapshot quiesce saves a pending value in transport state; +restore writes it back without replaying an edge, and the machine contract +records the interrupt mode, page address, and page size. + +## 2.4.5 Data Movement + +Device configuration and queue notification use MMIO. Bulk console, block, +network, and filesystem data move through descriptor chains in guest RAM +rather than through MMIO windows. diff --git a/doc/design-doc/2.5-device-topology-transport-and-discovery.md b/doc/design-doc/2.5-device-topology-transport-and-discovery.md new file mode 100644 index 000000000..73d351f6e --- /dev/null +++ b/doc/design-doc/2.5-device-topology-transport-and-discovery.md @@ -0,0 +1,46 @@ +# 2.5 Device Topology, Transport, and Discovery + +[Architecture documentation](README.md) > +[2. Virtual Machine Architecture](2-virtual-machine-architecture.md) + +Table 2.7 lists the fixed devices. Every device uses virtio-mmio at a fixed +slot, with packed rings masked. The complete device count, kind, bus, address, +interrupt, feature policy, and queue shape are validated before host resources +are resolved. + +**Table 2.7.** Fixed virtio-mmio devices. + +| Device | Stable identity | MMIO range | IRQ | Presence | +| --- | --- | ---: | ---: | --- | +| Network | `net:microvm0` | `0xd0000000..0xd0000fff` | KVM/MSHV 10, WHP 5 | Optional | +| Host filesystem | `fs:microvm0` | `0xd0001000..0xd0001fff` | 6 | Reserved slot; attachment optional | +| Boot console | `console:microvm-virtio0` | `0xd0002000..0xd0002fff` | 7 | Optional | +| `distro` block | `blk:sandbox:distro` | `0xd0003000..0xd0003fff` | 4 | Optional, read-only | +| `scratch` block | `blk:sandbox:scratch` | `0xd0006000..0xd0006fff` | 11 | Writable final role | +| Control console | `console:microvm-control0` | `0xd0007000..0xd0007fff` | 3 | Optional | + +Placement is explicit rather than allocated sequentially, so an omitted device +never shifts another. All six slots are reserved, and the windows +`0xd0004000..0xd0005fff` and `0xd0008000..0xd0008fff`, IRQs 9, 12, and 13, +and the status words at `0x30010`, `0x30014`, and `0x30020` are unused. + +Every cold-booted microVM instantiates the host filesystem slot, even without +an attachment, so the slot is discoverable before capture and a snapshot can +record an explicit dormant state; a restore keeps the slot only when the +snapshot records it. Several host shares are children of one aggregate +attachment on that slot rather than additional devices. The control slot is +instantiated for a control console, either a live authenticated local +endpoint or an explicitly disconnected console. Other optional devices exist +only when configured. + +The guest discovers fixed devices only from kernel command-line tokens that +the machine profile owns and emits in fixed address order; see +[Effective Command Line](5.1-boot-protocol.md#512-effective-command-line). +User arguments cannot inject alternate device declarations. Stable identity +comes from the machine profile and attachment contract, not from Linux device +enumeration order or names such as `/dev/vdX`; the guest maps each block role +to its device by MMIO address. + +Saved state records the exact device inventory and transport contract. +Restore recreates that inventory in the same order and validates it against +the saved state before a virtual processor runs. diff --git a/doc/design-doc/3-virtual-devices.md b/doc/design-doc/3-virtual-devices.md new file mode 100644 index 000000000..605390259 --- /dev/null +++ b/doc/design-doc/3-virtual-devices.md @@ -0,0 +1,23 @@ +# Virtual Devices + +[Architecture documentation](README.md) + +Virtual devices expose a small set of stable guest interfaces. Each interface has a fixed identity +and placement, bounded state, and explicit behavior during capture and restore. + +This chapter specifies the guest-visible behavior of each device class. + +## Table of Contents + +- [3.1 Platform Control Device](3.1-platform-control-device.md): Covers the + byte-channel, shutdown, and snapshot-request functions. +- [3.2 Clock and Time](3.2-clock-and-time.md): Defines clocksources, timer + behavior, rates, and time continuity. +- [3.3 Consoles](3.3-consoles.md): Separates boot communication from + authenticated runtime control. +- [3.4 Block Storage](3.4-block-storage.md): Defines fixed storage roles, + layering, mutability, and saved state. +- [3.5 Network](3.5-network.md): Defines network identity, connectivity, policy, + inbound access, and packet flow. +- [3.6 Host Filesystem](3.6-host-filesystem.md): Defines the virtio-fs request + path, host-backed exports, access policy, and saved session state. diff --git a/doc/design-doc/3.1-platform-control-device.md b/doc/design-doc/3.1-platform-control-device.md new file mode 100644 index 000000000..2f3584378 --- /dev/null +++ b/doc/design-doc/3.1-platform-control-device.md @@ -0,0 +1,140 @@ +# 3.1 Platform Control Device + +[Architecture documentation](README.md) > +[3. Virtual Devices](3-virtual-devices.md) + +Table 3.1 lists the three port-mapped I/O (PMIO) device functions that the +platform uses. + +**Table 3.1.** Platform-control device functions. + +| Port | Function | Architectural purpose | +| ---: | --- | --- | +| `0xe9..0xeb` | Host/guest byte channel | Early console data, bounded host input, restore metadata, generation identity, and host time samples | +| `0x604` | Shutdown | Guest-requested process termination with a status byte | +| `0x605` | Snapshot request | Guest-initiated capture and post-restore acknowledgement | + +PMIO callbacks perform only bounded state updates and notifications. They +never pause virtual processors, drain devices, hash memory, or write files; +lifecycle work is deferred to the VM controller and the state-unit graph. + +## 3.1.1 Host/Guest Byte Channel + +The byte channel is the simplest path between the guest and the host, and +every microVM has one. It takes the place of the serial port, which the +machine omits, and carries: + +- The guest's console. The kernel writes its boot messages to the channel + from early in its initialization, before it probes any driver, and the + channel is also the kernel console, with host input, unless a + [boot console](3.3-consoles.md#331-boot-console) is configured. +- Failure reports from the guest agents, including failures of the control + console. +- The one-time [restore packet](5.5-restore-protocol.md#555-restore-packet), + which gives a restored guest what it needs to repair its clocks, + processors, and memory, and the fresh random bytes from which it + [refreshes its entropy](5.5-restore-protocol.md#558-entropy-refresh). +- Host time samples, which the guest pairs with its own clock to set and + discipline its wall clock. +- The generation ID, a random identifier unique to each VM process. Every + restored process receives a new one, even when several clones run from one + snapshot, so the guest can detect that it has been restored and + distinguish itself from other clones. + +The channel deliberately does not depend on virtio discovery. The guest finds +its virtio devices only after its kernel has parsed their discovery tokens and +probed their drivers, and the consoles among them are optional. Through the +byte channel, a guest that fails before or during discovery, or whose virtio +devices fail, can still report the failure. The VMM also drains the channel +before it reports a guest-requested exit, so the guest's final output is not +silently lost; see +[Final Output and Exit Status](5.6-shutdown-and-failure-semantics.md#561-final-output-and-exit-status). + +The channel's protocol is also minimal: it needs no feature negotiation, +queues in guest memory, or interrupt line. Each output byte is one port write, +and the guest polls a status port for input. Every access to the channel's +ports is a synchronous exit that the VMM completes before the guest continues, +so a host time that the VMM records during an access lies between two guest +clock reads taken around it. This bounds the error of +[time samples](3.2-clock-and-time.md#329-time-samples) and of the restore +packet's UTC field. + +As Table 3.2 shows, the byte channel's three ports form one device: a data +port, a status and record-selector port, and a time-sample window. + +**Table 3.2.** Byte-channel ports. + +| Port | Access | Behavior | +| ---: | --- | --- | +| `0xe9` | Read and write | Raw byte input and output. A console read consumes one pending byte and zero-fills the rest of the access. With a record selected, a 1-, 2-, or 4-byte read returns that many bytes of the record, little-endian and zero-filled past its end. | +| `0xea` | Read | Status bits, listed in Table 3.3 | +| `0xea` | Write | Record selector: `0xa5` selects the one-time [restore packet](5.5-restore-protocol.md#555-restore-packet), `0xa6` selects the 16-byte generation ID, and `0xa7` latches a fresh [time sample](3.2-clock-and-time.md#329-time-samples) without changing the `0xe9` selection | +| `0xeb` | Read | The next bytes of the latched time sample, zero-filled past its end or when no sample is latched | + +Table 3.3 lists the status bits. + +**Table 3.3.** Byte-channel status bits. + +| Status bit | Meaning | +| ---: | --- | +| 0 | Host input is pending | +| 1 | A fresh restore packet is available | +| 2 | The restore carries a processor target | +| 3 | The restore carries a memory target | +| 4 | The memory target contains one or more expansion ranges | +| 5 | The generation-ID selector is available | +| 6 | The time-sample window exists; always set | +| 7 | Zero | + +The generation ID may be selected repeatedly and is stable for the lifetime of +one VM process. After a restore, the guest agents require it to differ from +the ID that they recorded before capture, or repair fails. A clone's guest +agents also credit it, as part of the restore entropy, to the kernel's +random-number generator and expose it to the workload runtime's post-restore +hook, so that random-number state that clones would otherwise share is +refreshed. [Generations](5.5-restore-protocol.md#556-generations) describes +how a restored guest checks the ID, and +[Entropy Refresh](5.5-restore-protocol.md#558-entropy-refresh) how it obtains +fresh random numbers. + +The time-sample window is separate from the data port because the guest +console driver polls status and reads data from a timer context, so a sample +on the console stream could interleave with input bytes. Guest users of the +selectors serialize their transactions. Every byte the guest reads is a port +exit, so records are small and readable four bytes per exit. + +The receive and transmit buffers are each bounded at 1 MiB. Output overflow +drops the newest bytes and emits a rate-limited warning; input applies +backpressure by not reading the host endpoint while its buffer is full. +Pending bytes are saved, so capture does not silently lose VMM-owned I/O. The +time-sample window is not saved state. + +While host input is gated for a snapshot boundary or post-restore repair, the +channel stops reading its host endpoint, returns zero for console reads, and +clears the input-pending bit. Guest output, the generation ID, any restore +packet, and the time-sample window remain available. + +## 3.1.2 Shutdown + +The first byte written to the shutdown port becomes the process status carried +with the VM power-off request; reads return all ones. Shutdown initiates +orderly VM teardown rather than emulating PC power management. +[Shutdown and Failure Semantics](5.6-shutdown-and-failure-semantics.md) +describes the final drain and status propagation. + +## 3.1.3 Snapshot Request + +A write to the snapshot port requests capture. A zero byte requests fresh +scratch, and a nonzero first byte requests paired scratch; the scratch policy +travels with the deferred boundary request. Reads return all ones, and the +port reports no capture status. + +A write with no configured snapshot destination completes normally, and the +guest continues. With a destination, the device permits at most one pending +transaction, coalesces repeated writes, and defers completion long enough for +the controller to stop the virtual processor at the exact instruction boundary +immediately after the output operation. + +After a gated restore, a write to the same port acknowledges that processor, +memory, time, entropy, and workload state have been repaired. Host input is +released only after the VMM stops at that acknowledgement boundary. diff --git a/doc/design-doc/3.2-clock-and-time.md b/doc/design-doc/3.2-clock-and-time.md new file mode 100644 index 000000000..a57d92614 --- /dev/null +++ b/doc/design-doc/3.2-clock-and-time.md @@ -0,0 +1,405 @@ +# 3.2 Clock and Time + +[Architecture documentation](README.md) > +[3. Virtual Devices](3-virtual-devices.md) + +The NVX time ABI is the single guest-visible time, timer, and clock contract +on every backend and launch path. It is versioned independently of the +machine ABI. Every rule is a hard check: a host or snapshot that cannot meet +the contract is rejected with a stable error code, a guest that observes a +violation fails fast, and no rule falls back to another behavior. + +The time ABI applies the [architecture goal](1.1-goals.md) +of keeping the guest-visible machine independent of the selected hypervisor to +clocks and timers. It provides: + +- The same CPUID identity, synthetic MSRs, CPU time bits, clocksource, tick, + timers, and restore semantics on every backend and launch path. +- No guest kernel patches and no clock parameters on the kernel command line. +- Restore within one backend across hosts of the same CPU generation and CPU + profile whose TSC rates match within a small tolerance, including after a + host reboot. +- Guest monotonic time that advances by the snapshot downtime, and a wall + clock that is set on every restore and disciplined afterwards. +- Kernel stall, lockup, clocksource, and TSC-warp detectors that never trip + across capture and restore. + +The guest sees: + +- A TSC that ticks at a constant rate in every processor power and frequency + state (an invariant TSC) as its only clocksource. +- The declared TSC and LAPIC timer rates through a minimal synthetic + hypervisor identity. +- A one-shot LAPIC timer at the backend's fixed rate. +- A PIT that the guest disables at boot. +- A programmable RTC anchored to UTC. + +An invariant TSC keeps one rate on a given host; it does not give every host +the same rate. The guest TSC runs at its host's native rate, which can differ +between hosts of one CPU generation and is not part of the CPU profile, so the +guest reads the declared rate from a frequency MSR instead of deriving it from +CPUID. A cold boot declares its host's rate, and a snapshot restores only on a +host whose rate lies within 250 ppm of that declaration; see +[Rates](#324-rates). + +Table 3.4 lists the notation that the time ABI uses. + +**Table 3.4.** Time ABI notation. + +| Symbol | Meaning | +| --- | --- | +| `F` | Declared guest TSC rate in Hz | +| `F_s` | Declared rate recorded in a snapshot | +| `F_d` | Native TSC rate of the destination partition in Hz | +| `L` | LAPIC timer rate in Hz | +| `C` | Processor capacity of the machine | +| `D` | Snapshot downtime in nanoseconds | +| `T_c` | Guest TSC of the first processor at the capture anchor | +| `Q` | Host monotonic time from the VM-time cut at quiesce to the capture anchor | +| `g` | Restore generation counter of a VM process | + +## 3.2.1 Hypervisor Identity + +As Table 3.5 shows, every processor sees a minimal Hyper-V frequency and +invariant-TSC identity, configured identically on every backend. Each row +lists the values that one leaf returns in the four general-purpose registers +through which CPUID reports its results (EAX, EBX, ECX, and EDX). + +**Table 3.5.** Hypervisor identity leaves. + +| Leaf | EAX | EBX | ECX | EDX | +| --- | --- | --- | --- | --- | +| `0x40000000` | `0x40000005` (maximum leaf) | `0x7263694d` (`"Micr"`) | `0x666f736f` (`"osof"`) | `0x76482074` (`"t Hv"`) | +| `0x40000001` | `0x31237648` (`"Hv#1"`) | 0 | 0 | 0 | +| `0x40000002` | `0x0058564e` (`"NVX"`) | `0x00010000` (time ABI 1.0) | 0 | 0 | +| `0x40000003` | `0x00008860` | 0 | 0 | `0x00000100` | +| `0x40000004` | 0 | `0xffffffff` | 0 | 0 | +| `0x40000005` | `C` | `C` | 0 | 0 | + +Leaf `0x40000003` grants exactly the privileges that Linux needs to detect the +platform, hypercall-MSR and virtual-processor-number (VP-index) access, plus +frequency-register and invariant-TSC-control access, and advertises only the +frequency registers. +Leaf `0x40000004` recommends nothing and never requests spinlock notification. +Leaves `0x40000006..0x4000000f` and `0x40000080..0x40000082` read as explicit +zeros. Every other leaf in `0x40000006..0x400000ff` returns either zeros or +the vendor's architectural out-of-range result, and no base from `0x40000100` +through `0x4000ff00` carries a hypervisor signature, so Linux selects only the +Hyper-V platform. Beyond the leaves of Table 3.5, no leaf returns Hyper-V +feature data, and leaf `0x40000081` carries no `VS#1` virtualization-stack +interface signature. + +The identity deliberately deviates from the Hyper-V specification: + +- The hypercall MSRs raise a general-protection fault (#GP), there is no + hypercall page, and hypercall instructions are outside the contract. +- The TSC frequency MSR returns the declared rate `F`, which may differ from + the physical rate by up to the [rate tolerance](#324-rates) after a restore. +- The invariant-TSC control can be cleared after it is set. +- The invariant-TSC CPUID bit is set from boot rather than after the control + is written. +- Leaf `0x40000002` carries an NVX signature and the time ABI version instead + of a build number. +- Leaf `0x40000005` reports the processor capacity. + +The guest's only side effect is selecting the Hyper-V platform. No paravirtual +clock, interrupt, TLB, spinlock, steal-time, or page-fault feature exists on +any backend, and the scheduler clock and vDSO use the raw TSC. + +## 3.2.2 MSRs + +As Table 3.6 shows, the VMM serves the identity MSR range +`0x40000000..0x400001ff` identically on every backend. + +**Table 3.6.** Identity MSRs. + +| MSR | Name | Read | Write | +| --- | --- | --- | --- | +| `0x40000002` | VP index | Processor index | #GP | +| `0x40000022` | TSC frequency | `F` | #GP | +| `0x40000023` | APIC frequency | `L` | #GP | +| `0x40000118` | TSC invariant control | Last value written; 0 after reset | 0 or 1 accepted; any other value #GP | +| Any other MSR in the range | | #GP | #GP | + +The TSC invariant control is one partition-wide value. It is guest state, +saved and restored with the VM. Linux writes 1 once at boot without fault +handling, so that write must never fault on any backend. + +The MSRs that hold an offset added to the processor's TSC (`IA32_TSC_ADJUST`, +`0x3b`) and the TSC value at which the LAPIC timer fires in TSC-deadline mode +(`IA32_TSC_DEADLINE`, `0x6e0`) raise #GP on every backend, as do the +cache-control registers of Intel's P6 processor generation (the legacy P6 +L2-cache MSRs). + +As Table 3.7 shows, under an AMD CPU profile, the VMM also serves two +configuration MSRs with fixed values. + +**Table 3.7.** AMD configuration MSRs. + +| MSR | Name | Read | Write | +| --- | --- | --- | --- | +| `0xc0010015` | Hardware configuration (`HWCR`) | `0x1000000`: only the bit that makes the TSC count at a fixed frequency (`TscFreqSel`) | The value read changes nothing; any other value #GP | +| `0xc0011029` | Decode configuration (`DE_CFG`) | `0x2`: only the bit that makes the load-fence instruction hold later instructions until earlier ones complete (`LFENCE` serialization) | The value read changes nothing; any other value #GP | + +## 3.2.3 CPU Time Bits + +Table 3.8 lists the bits that every CPU profile fixes. A CPU profile that +violates them is rejected. + +**Table 3.8.** CPU time bits. + +| Leaf | Register and bits | Value | +| --- | --- | --- | +| `0x1` | ECX[31] hypervisor present | 1 | +| `0x1` | ECX[24] TSC-deadline timer | 0 | +| `0x1` | ECX[15] performance and debug capabilities MSR (PDCM) | 0 | +| `0x1` | EDX[4] TSC | 1 | +| `0x6` | EAX, EBX, ECX, EDX | `0x00000004` (only the always-running APIC timer, ARAT), 0, 0, 0 | +| `0x7.0` | EBX[1] `IA32_TSC_ADJUST` | 0 | +| `0xa` | EAX, EBX, ECX, EDX | 0 (no PMU) | +| `0x15`, `0x16` | EAX, EBX, ECX, EDX | 0, when within the maximum basic leaf | +| `0x80000001` | EDX[27] RDTSC variant that also returns a processor identifier (RDTSCP) | 1 | +| `0x80000007` | EAX, EBX, ECX, EDX | 0, 0, 0, `0x00000100` (invariant TSC only) | + +Leaves `0x15` and `0x16` are zeroed rather than synthesized. Linux takes both +rates from the frequency MSRs, which keeps the CPU profile's CPUID independent +of any host's rate. The invariant-TSC bit is policy rather than a +fingerprinted feature: every CPU profile sets it, and every backend presents +it. + +## 3.2.4 Rates + +The guest TSC runs at the destination's native rate `F_d` and is never scaled. +The declared rate `F` is fixed for the life of a snapshot lineage: a cold boot +declares its native rate, and every restored process keeps `F = F_s`. The +kernel computes its TSC frequency once and never recalibrates. + +At restore, the native rate must lie within 250 ppm of the declared rate, +computed with integer arithmetic of at least 128 bits: + +```text +accept iff |F_d - F_s| * 1_000_000 <= 250 * F_s +``` + +The boundary is accepted. The manifest records the tolerance, and any other +recorded value is rejected. Within the tolerance, the TSC is neither scaled +nor trapped, the frequency MSR keeps returning `F_s`, and guest clocks run +fast or slow by at most the deviation. The restore packet reports that +deviation in the kernel's frequency-adjustment unit, ppm scaled by 2^16, so +that the guest can compensate: + +```text +rate_deviation = round((F_d - F_s) * 1_000_000 * 65_536 / F_s) +``` + +Every rate must also be plausible, `500 MHz <= F <= 10 GHz`. A backend that +cannot report its native rate is rejected at cold boot, capture, and restore +alike; there is no calibration fallback. + +The LAPIC timer rate `L` is a backend constant: 1 GHz on KVM and 200 MHz on +MSHV and WHP. The kernel presets its timer period from the APIC frequency MSR +and never calibrates the LAPIC. On restore, the destination's rate must equal +the recorded rate exactly. + +## 3.2.5 Clocksource, Tick, and Timers + +With this identity, the guest kernel takes the TSC rate from the frequency +MSR, trusts the TSC as reliable, which disables the clocksource watchdog and +the CPU-online warp test, presets the LAPIC timer period, and disables the PIT +at boot. The booted guest must then be in this state: + +- Every CPU reports a constant, nonstop, known-frequency, reliable TSC with + RDTSCP and ARAT, and none reports a TSC-deadline timer or `TSC_ADJUST`. +- The TSC is the current clocksource and, after each CPU's first tick, the + only available one. +- Ticks are tickless-idle with high-resolution timers, and every online CPU's + tick device is the LAPIC timer in one-shot mode. +- No PIT, high-precision event timer (HPET), or broadcast clock-event device + exists. +- The LAPIC timer is never periodic and never in TSC-deadline mode after boot. +- PIT channel 0 never counts after boot, and IRQ 0 has no timer handler. + +Capture and restore reject an armed periodic or TSC-deadline LAPIC timer and a +periodically counting PIT channel 0. Across a downtime `D`, a counting +one-shot LAPIC timer advances by exactly: + +```text +ticks = floor(D * L / 1_000_000_000 / divide) +``` + +where `divide` is the timer's divide configuration. If `ticks` reaches the +current count, the count becomes zero and the timer interrupt is queued unless +the LAPIC register that configures it (its LVT entry) is masked; otherwise, +the count decreases by `ticks`. A queued or pending vector must reach its +processor when the processor is released, even if it is halted. + +## 3.2.6 Backend Obligations + +Each backend implements the same obligations behind one VMM interface. The +guest-visible result is identical; only the mechanism differs. Every backend +must: + +- Present exactly the identity leaves and their explicit zero leaves, with no + other hypervisor signature. +- Present the CPU profile's effective CPUID, including the CPU time bits. +- Route the identity MSRs, and under an AMD CPU profile the two configuration + MSRs, to the VMM. +- Report the native TSC rate and never scale the TSC. +- Run the LAPIC timer at the backend's fixed rate. +- Deliver a pending LAPIC vector after a LAPIC state write, even to a halted + processor. +- Hide the TSC-deadline timer and `TSC_ADJUST`, and fault both MSRs. +- Expose the invariant TSC. +- Offer no paravirtual or synthetic features, and never enable a + hypervisor-specific emulator or its saved state. +- Pair the first processor's TSC with a host time sample within 100 µs for + the capture anchor, keeping the tightest of at most 64 samples. +- Set every instantiated processor's TSC at one host instant and verify it by + read-back before any processor runs. +- Keep live cross-processor skew within 1 µs after release. + +## 3.2.7 Cross-Processor Consistency + +At every guest-observable instant, the TSCs of any two online processors +differ by at most 1 µs, that is `F_s / 1,000,000` cycles. The guest kernel no +longer checks TSC consistency, so the bound is enforced outside the guest. +Cold boot starts every processor's TSC together. Restore writes one target to +every instantiated processor at one host instant and verifies it by read-back, +so the VMM introduces no skew of its own. After release, each backend keeps +live skew within the bound, and a host that cannot hold it does not qualify +for the contract. + +## 3.2.8 Downtime and Continuity + +Guest monotonic time advances by the downtime `D`, the host time elapsed +between the capture anchor and the restore anchor, plus at most the restore +latency `R` from the restore anchor to the first restored instruction. The +guest observes no other discontinuity between the snapshot request and the +first restored instruction. + +The capture anchor records: + +- `T_c`, the first processor's guest TSC. +- Host UTC in nanoseconds since the Unix epoch. +- Suspend-inclusive host monotonic time. +- The host identity. +- The host boot identity. + +A host that cannot provide all of them rejects capture and restore. +As Table 3.9 shows, restore takes the same sample at its own anchor and +selects exactly one downtime source. + +**Table 3.9.** Downtime source selection. + +| Condition | Source | `D` | +| --- | --- | --- | +| Same host identity, boot identity, and host clock kind | Host monotonic time | `monotonic_now - monotonic_capture` | +| Anything else | Host UTC | `utc_now - utc_capture` | + +`D` must satisfy `0 <= D <= 30 days`. On the monotonic path, a UTC delta that +differs from `D` by more than one second is reported as a host wall-clock step +and does not change `D`. The restore packet reports the selected source. + +VM time, the clock of the VMM's emulated devices, and the RTC stop at quiesce, +before the capture anchor. Capture therefore records `Q`, the host monotonic +interval from that cut to the anchor, and restore advances VM time and the RTC +by `D + Q` so that they resume in step with the guest TSC, which kept running +through `Q`. The TSC, the LAPIC timers, and the restore packet use `D`. +`D + Q` must not exceed 30 days. `Q` is best effort: a snapshot without it +restores with `Q = 0` and a warning. + +The restore anchor is the instant of the synchronized TSC set, and guest time +runs from that instant on every backend; no backend defers it to the first +processor run. No backend stops guest time for a capture either, so a capture +that rolls back leaves the guest's clocks continuous, and the guest sees the +quiesce as elapsed time, as it would a long preemption. + +## 3.2.9 Time Samples + +Figure 3.1 shows the layout of the 16-byte sample that writing `0xa7` to the +byte-channel selector latches into the time window. + +```text +offset size field + 0 1 version = 1 + 1 1 flags bit 3 mirrors the restore packet's override flag; other bits zero + 2 2 reserved = 0 + 4 4 generation u32, little-endian, g of this VM process + 8 8 utc_ns u64, little-endian, host UTC at the latch +``` + +**Figure 3.1.** Time-sample layout. + +The VMM cannot observe a running processor's TSC on every backend, so the +guest pairs host UTC with its own clock. It reads its realtime clock as `t0` +immediately before the selector write and as `t1` immediately after it. The +host instant lies in `[t0, t1]`, so: + +```text +theta = utc_ns - (t0 + t1) / 2 uncertainty epsilon = (t1 - t0) / 2 +``` + +The same bracket applies to the UTC sample in the restore packet. +As Table 3.10 shows, the guest accepts a pairing only if `epsilon` is within +the bound for its use. + +**Table 3.10.** Time-pairing uncertainty bounds. + +| Use | Bound | Attempts | When every attempt exceeds the bound | +| --- | --- | --- | --- | +| Restore repair | 1 ms | 3 | Repair fails, and the guest powers off | +| Initial synchronization at boot | 1 ms | 3 | Conformance fails, and the guest powers off | +| Wall-clock discipline | 50 µs | 3 | The poll is skipped; never fatal | + +## 3.2.10 Guest Obligations + +The guest agents perform conformance checks, watch for violations, discipline +the wall clock, and execute the snapshot time steps in every guest mode. +Production paths keep the console quiet; status goes to a state file that the +workload can read through a read-only mount. + +**Conformance.** Before readiness, the guest only takes a time sample within +the bound and steps its clock to host UTC, so the workload starts on host +time. It runs every other boot check asynchronously after readiness, starting +at idle priority, so the checks never delay readiness or take a processor from +the workload. The checks cover the identity leaves and explicit zero leaves, +the CPU time bits on every CPU, the identity MSRs, the kernel's platform, +clocksource, and LAPIC-timer selection, CPU flags, tick devices, the absence +of forbidden clock parameters, stall-detector settings, a running violation +watcher, and kernel W+X integrity. Capture and restore rerun the relevant +subset. Every check is fail-fast: a failure powers the guest off even if the +workload is running. + +**Violation watcher.** The watcher follows the kernel log from before the boot +checks, so no record is missed. TSC instability; clocksource instability, +switches, or skew; TSC warps; `TSC_ADJUST` faults; RCU stalls or starvation; +soft or hard lockups; hung tasks; unchecked MSR accesses; APIC identity +mismatches; and lost log records are violations. The watcher also polls the +kernel's RCU stall count, so a stall is caught even if its record is lost. A +violation is reported as one bounded event line on the kernel log, the byte +channel, and the console, and the guest then powers off through the shutdown +port. + +**Stall suppression.** Restoring after downtime makes kernel time jump, so the +guest suppresses RCU stall detection, and the debug stall detectors, before a +capture request. After restore, it releases the suppression only when the +grace period that was in flight at capture has ended, forcing a full normal +grace period first. If that grace period stays stuck beyond the stall timeout, +the guest releases the suppression anyway so that the kernel reports the real +stall. + +**Wall-clock discipline.** The guest keeps its realtime clock on host UTC +through the kernel's phase-locked loop. It polls every 16 seconds for the +first four polls after boot or a restore, then every 64 seconds, and takes up +to three samples per poll. An offset of at least 128 ms is stepped and counted +as a discontinuity; a smaller one is slewed with error bounds derived from the +offset and its uncertainty, which marks the clock synchronized for +applications. The kernel limits the frequency correction to ±500 ppm, of which +the rate tolerance uses at most 250 ppm. A sample whose generation differs +from the recorded one is discarded, because a restore happened and restore +repair resets the discipline. The discipline follows host wall-clock steps +and never powers off the guest. + +The [snapshot protocol](5.3-snapshot-protocol.md) and +[guest repair](5.5-restore-protocol.md#557-guest-repair) define the snapshot +time steps. diff --git a/doc/design-doc/3.3-consoles.md b/doc/design-doc/3.3-consoles.md new file mode 100644 index 000000000..cd168367d --- /dev/null +++ b/doc/design-doc/3.3-consoles.md @@ -0,0 +1,117 @@ +# 3.3 Consoles + +[Architecture documentation](README.md) > +[3. Virtual Devices](3-virtual-devices.md) + +NVX separates human-oriented boot communication from authenticated workload +control. Both use the single-port virtio-console device with two split queues, +but they are distinct devices with distinct attachment kinds, stable +identities, slots, interrupts, endpoints, saved state, and reconnect policies. + +## 3.3.1 Boot Console + +The optional boot console carries boot messages and an ordinary byte stream. +The kernel console uses it once its driver is ready, and the byte channel +remains the early console. Host RX accepted by the device and partial guest TX +progress are device-private saved state, so restore never replays a +descriptor from byte zero or silently loses VMM-owned data. + +The host attachment is one of: + +- A listener on a Unix socket, a named pipe, or loopback TCP with a fixed + nonzero port, which retains pending guest output until a client connects. +- A client connection. +- The inherited terminal. +- An explicitly disconnected endpoint that discards guest output. + +The snapshot records the attachment's stable ID, canonical endpoint identity, +reconnect policy, requiredness, and timeout, never a native handle. A saved +listener may be recreated at a fresh private endpoint of the same kind, but +its stable ID, attachment kind, backend kind, reconnect policy, required flag, +length, and timeout stay exact. A client attachment keeps its exact saved +identity and requires an explicitly approved restore-time attachment, with a +five-second timeout. An inherited terminal must be supplied again, and a +disconnected attachment stays disconnected. + +## 3.3.2 Control Console + +The control console carries structured operations between the host and the +guest agents. A fresh boot with a control console also requires the boot +console, which makes the control device's guest terminal `hvc2`. The machine +profile owns the token that identifies it, and the boot console remains the +only kernel console. When a control console is present, the machine profile +rejects caller-supplied control terminal, driver probe-order, and +device-discovery tokens, including kernel-equivalent hyphenated spellings, as +well as quotes and the `--` delimiter, so command-line parsing cannot redirect +the guest's terminal discovery. + +The host endpoint is either a local listener or explicitly disconnected. A +live endpoint is protected in three layers: + +- **Private endpoint.** On Linux hosts, it is a Unix socket bound exclusively + with mode `0600` inside an existing, owner-only, non-symlink directory of + the VMM's user. On Windows hosts, it is a named pipe whose protected DACL + admits only LocalSystem and the VMM's user. Every other backend kind is + rejected, and the boot and control endpoints must differ. +- **Peer identity.** The broker reserves its single host slot for a peer + whose operating-system identity is the VMM's user: the Unix peer's effective + UID, or the pipe client's token user SID. Any other peer is disconnected + before its bytes are read. +- **Capability.** The launcher passes a random 32-byte capability through a + one-way pipe whose writers are closed before the VMM starts. The first host + record must present it within a bounded authentication timeout, five seconds + by default. A disconnected control console uses a random capability that no + client can present. + +As Figure 3.2 shows, the broker relays a frozen, versioned record stream +between the guest and the authenticated host. Every record has a fixed-size +header with a magic value, protocol version, record type, broker instance ID, +epoch, direction-local sequence number, and payload length. Attach records +establish the guest and host sides; acknowledgement, data, wait, ready, reset, +authentication-error, and credit records carry the session. + +```mermaid +sequenceDiagram + participant Host as Host client + participant Broker as VMM broker + participant Guest as Guest agents + + Host->>Broker: Connect as the VMM's user + Host->>Broker: Present the capability + Broker->>Guest: Attach the host side + Guest-->>Broker: Acknowledge and grant receive credit + Host->>Broker: Data within the credit + Broker->>Guest: Data + Guest-->>Broker: Data and further credit + Broker-->>Host: Data +``` + +**Figure 3.2.** Control-session establishment and data flow. + +Flow control is credit based: the guest grants byte-counted receive credit for +host data. When credit is exhausted, one complete host record may wait in a +bounded pending slot, and each output direction is bounded in records and +bytes. Malformed lengths, wrong-state records, wrong instance or epoch values, +and duplicate or out-of-order sequence numbers fail closed. A wrong capability +or an authentication timeout detaches the client without advancing the epoch. +A host disconnect advances the epoch, drops unstarted data and unused credit, +finishes any partially transmitted record so the stream stays aligned, and +emits a reset; the next session receives credit only from a fresh guest +acknowledgement. + +Snapshots record only the endpoint identity and a broker policy, either an +authenticated listener or a disconnected console, and never the capability or +the peer identity. Broker saved state omits queued records, host output, and +usable receive credit; it keeps only the guest-side parser, any partially +transmitted guest record, and error counters. To restore an authenticated +listener, the launcher must supply an approved endpoint and a fresh +capability. The endpoint may move to a fresh location, so that concurrent +clones do not collide, but it must meet the private-endpoint rules above, and +the attachment's stable ID, attachment kind, backend kind, reconnect policy, +required flag, length, and timeout stay exact. A disconnected console stays +disconnected. The restored broker receives a fresh instance ID, restarts at +the first epoch with a queued reset, and ignores guest records for the old +instance until the guest acknowledges the new one. + +The guest agents own the workload protocol carried inside the brokered stream; +see [Host-Guest Control Interface](4.4-host-guest-control-interface.md). diff --git a/doc/design-doc/3.4-block-storage.md b/doc/design-doc/3.4-block-storage.md new file mode 100644 index 000000000..f7f96e2c5 --- /dev/null +++ b/doc/design-doc/3.4-block-storage.md @@ -0,0 +1,98 @@ +# 3.4 Block Storage + +[Architecture documentation](README.md) > +[3. Virtual Devices](3-virtual-devices.md) + +As Table 3.11 shows, sandbox storage uses two fixed roles rather than +arbitrary disks. + +**Table 3.11.** Sandbox block roles. + +| Role | Access | Purpose | +| --- | --- | --- | +| `distro` | Read-only | Curated base operating-system layer | +| `scratch` | Read-write | Private ext4 upper and work storage | + +Roles are unique and appear in fixed order; an omitted `distro` role leaves +its slot empty. A disk-backed topology ends with scratch, except in the +RAM-backed overlay layout, which uses exactly one read-only `distro` block. +The role, not Linux enumeration order, defines identity. + +## 3.4.1 Construction and Discovery + +Each attachment must use its role's access mode. The VMM opens the host +storage through a generic disk interface and creates one single-queue +virtio-blk device per role at the role's fixed slot, with no PCI enumeration. +It adds one discovery token per block, which the machine profile owns, to the +kernel command line. The guest kernel creates the device from that token, +negotiates features within the role's fixed set, and places the split +virtqueue in guest RAM. The guest agents identify each role by its MMIO +address rather than by a device name. + +## 3.4.2 Guest-Visible Capabilities + +Each role offers a fixed capability set: virtio 1.0 split rings with indirect +descriptors and event-index notification suppression, a per-request segment +limit, block-size and topology reporting, and a write-back cache with explicit +flush. The `distro` role is also advertised as read-only. Discard is never +offered. The device reports the backing store's capacity and its logical and +physical block sizes. + +## 3.4.3 Request and Completion Path + +The MMIO window is a control plane for configuration and queue notifications; +block data never crosses it. A request is a descriptor chain in guest RAM that +holds a header with the operation and starting sector, any data buffers, and a +status byte. The guest publishes the chain and, unless the device has +suppressed notifications, notifies the queue, which wakes the device's I/O +worker. + +The worker reads requests directly from guest memory and services several +concurrently. It translates each virtio sector address to the backing store's +logical blocks and rejects a read or write that does not start on a +logical-block boundary or whose length is not a whole number of logical +blocks; the physical block size is only an alignment hint for the guest. +Accepted reads and writes become asynchronous scatter-gather I/O. Individual +writes are not forced to stable storage; a guest flush becomes a backend +cache flush. On completion, the device stores a success, I/O-error, or +unsupported status, publishes one used-ring entry, and, unless the guest +suppressed the notification, signals the guest through the role's +shared-status word and edge interrupt. + +## 3.4.4 Host Storage and Access Enforcement + +Because the device depends only on the generic disk interface, its +guest-visible behavior is independent of the host storage type. The backend +sees block offsets and buffers; filesystem semantics remain entirely in the +guest. Read-only access is enforced at three layers: the guest is told that +the device is read-only, the device rejects write and discard requests, and +the VMM opens the host file without write access. Scratch is the only +writable role. + +A snapshot destination or a restore makes every block snapshot-capable. Each +block must then be a nonempty, 512-byte-aligned raw regular file that uses +cached I/O and is not reached through a final symbolic link. The VMM retains +that opened file for flushing and paired-scratch publication. + +## 3.4.5 Snapshot Contract and Quiesce + +The snapshot contract records each block's role, read-only flag, logical +length, logical and physical block sizes, and identity. Every consumed block +shares one identity, a caller-authenticated immutable storage generation; see +[Snapshot Identity](5.4-snapshot-artifact-and-attachment-model.md#545-snapshot-identity). +Platform-tier layers are recorded as unbound, because image binding has not +been consumed, so a restore may supply different layers of the same geometry. +Writable scratch follows one of two policies: + +- **Paired.** Capture publishes the exact scratch image with its length and + identity. Restore rejects a caller-supplied scratch block and materializes + the recorded artifact as a private copy, a reflink copy-on-write clone, or a + single-use claimed continuation. +- **Fresh.** Capture happens before scratch is mounted and publishes no + scratch artifact. Restore requires a new writable file with matching + geometry. + +Transport and queue progress are saved with the other device state. Stopping +a queue prevents new descriptor intake and drains every accepted I/O to one +used-ring completion within the bounded snapshot quiesce timeout. The +controller flushes and copies scratch only after that drain. diff --git a/doc/design-doc/3.5-network.md b/doc/design-doc/3.5-network.md new file mode 100644 index 000000000..28815f3dd --- /dev/null +++ b/doc/design-doc/3.5-network.md @@ -0,0 +1,256 @@ +# 3.5 Network + +[Architecture documentation](README.md) > +[3. Virtual Devices](3-virtual-devices.md) + +As Figure 3.3 shows, the network device is an optional virtio-net NIC backed +by an in-process userspace network stack in the VMM. The stack translates guest +traffic into ordinary host socket operations. It does not attach the guest to +a physical NIC, TAP interface, bridge, or host virtual switch, and the host +needs no interface that holds the guest's gateway address. + +```mermaid +flowchart TB + Guest["Guest application and Linux TCP/IP stack"] + Driver["Guest virtio-net driver"] + Queues["TX/RX virtqueues in guest RAM
MMIO notifications and interrupts"] + Device["VMM virtio-net device
guest TX header prefilter"] + Endpoint["In-process network endpoint
final guest TX authorization"] + Host["Host sockets and DNS resolver"] + Network["Host OS routing, firewall, and network"] + Peer["Remote server or permitted host-local service"] + + Guest <--> Driver + Driver <--> Queues + Queues <--> Device + Device <--> Endpoint + Endpoint <--> Host + Host <--> Network + Network <--> Peer +``` + +**Figure 3.3.** Network data path. + +## 3.5.1 Device and Network Identity + +The NIC occupies the fixed network slot, has one RX/TX queue pair, and +advertises an exact feature mask: the MAC-address feature and virtio +version 1. Its interrupt is the one backend-specific value in the machine +contract: IRQ 10 on KVM and MSHV, and IRQ 5 on WHP. The portable network +profile is mandatory and selects the same in-process data plane on every +backend. + +Table 3.12 shows an example of the static network identity that the host owns. +The identity consists of: + +- An IPv4 prefix from `/1` through `/30`, whose first usable address is the + gateway. +- Deterministic guest and gateway MAC addresses derived from the final three + IPv4 octets. +- A derived IPv6 identity that embeds the guest and gateway IPv4 addresses in + `fd00::/96`, with a prefix of 96 plus the IPv4 prefix length, behind a + gateway that shares the IPv4 gateway's MAC address. + +**Table 3.12.** Example network identity. + +| Setting | Example | +| --- | --- | +| Guest IPv4 address | `10.0.0.2/24` | +| IPv4 gateway | `10.0.0.1` | +| Guest IPv6 address | `fd00::a00:2/120` | +| IPv6 gateway | `fd00::a00:1` | + +The guest configures this identity and its default routes from command-line +tokens that the machine profile owns, without DHCP, router advertisements, +address autoconfiguration, or duplicate address detection. Its kernel-derived +link-local address is its only other IPv6 address, and the egress policy +admits it only as the source of neighbor solicitations. The identity +advertises no routable IPv6 prefix, so external IPv6 reachability depends on +the host's own routes and address-family support. + +## 3.5.2 Packet Forwarding + +Guest transmission follows this path: + +1. The guest network stack builds an Ethernet frame, and the driver places + its buffers in the TX virtqueue. +2. The driver notifies the device, which reads the descriptors and prefilters + the frame headers against the egress policy. +3. The endpoint copies the complete frame into an owned buffer, authorizes + those exact bytes, and passes an allowed frame to the network stack. +4. For a new TCP connection, the stack connects a host TCP socket to the + destination. For UDP, it sends the datagram's payload through a host UDP + socket. + +TCP forwarding is socket proxying, not packet rewriting. The in-process stack +terminates the guest-facing connection, and the host kernel owns the +remote-facing one, with separate sequencing, acknowledgements, and buffering. +A remote peer normally sees the host's network identity. Host routing, VPN +configuration, and host firewall rules still govern the host sockets, and the +host needs no IP forwarding. + +Data arriving on a host socket is associated with its guest-facing flow, +rebuilt as Ethernet, IP, and TCP or UDP packets, and written into buffers that +the guest posted on the RX virtqueue, followed by a used-buffer interrupt. + +ARP and IPv6 neighbor discovery are answered locally rather than broadcast on +a physical LAN. The network profile provides outbound TCP and UDP over IPv4 +and IPv6. The endpoint answers echo requests to either gateway itself, relays +other IPv4 echo requests through a host ICMP socket, and does not relay ICMPv6 +echo to other destinations. It deterministically rejects fragmented IPv4 +packets, IPv6 extension headers, and IPv4-mapped IPv6 destinations. + +## 3.5.3 Egress Policy + +By default, guest egress is allowed and general host ingress is denied. One +canonical egress policy applies before any externally visible transmission. A +generic policy has an explicit default action, `allow` or `deny`, and up to +256 allow rules and 256 deny rules. Each rule selects destinations by IPv4 or +IPv6 address or CIDR, optionally narrowed to TCP, UDP, or ICMP and, for TCP or +UDP, to a destination port or an inclusive port range. Evaluation is +independent of rule order: + +1. A matching deny rule denies the packet. +2. Otherwise, a matching allow rule allows it. +3. Otherwise, the default action decides. + +A rule matches only destinations of its own address family. The canonical +policy also supports IPv4-only host allow lists, host block lists, and exact +TCP endpoint allow lists; these deny all IPv6 traffic and do not combine with +generic rules. + +Filtering is inactive only when the policy allows everything. Otherwise, +packet checks enforce the bound guest MAC and IP identity and validate the +headers, not just the destination, and port-specific rules fail closed on +malformed or fragmented traffic. The device's early header check alone is +insufficient, because guest memory remains mutable after it, so the endpoint +repeats authorization on its complete owned copy of the frame. A guest +therefore cannot rewrite a checked frame into a different destination. Denied +frames are dropped, and their TX descriptors complete without host +transmission. + +The rules are destination-based L3 and L4 rules, not hostname or content +filters. They apply to every frame that the guest transmits, including replies +on forwarded connections, but they are not a receive firewall or a host-wide +firewall. While the host publishes the guest's listeners, the policy also +admits the guest's replies on the flows that the endpoint carries, as +[Host-Local Access and Ingress](#354-host-local-access-and-ingress) describes. + +## 3.5.4 Host-Local Access and Ingress + +By default, and subject to the egress policy, the gateway maps guest TCP and +UDP flows onto host loopback: the IPv4 gateway maps to `127.0.0.1`, and the +IPv6 gateway maps to `::1`. The guest's own loopback remains guest loopback. +Other host-local destinations, such as link-local addresses, also reach host +sockets unless host loopback is denied. + +Denying host loopback disables the gateway-to-loopback mappings, blocks +loopback, unspecified, and link-local destinations, and disallows every +forward. One exact IPv4 gateway TCP proxy endpoint may remain available. It is +bound into the policy digest, admitted before the policy mode is evaluated, +and translated to the same port on host `127.0.0.1`, where the launcher +supplies the proxy service. + +An outbound connection never makes guest services reachable from the host or +its networks. As Table 3.13 shows, inbound access is a separate host decision. + +**Table 3.13.** Inbound access to guest services. + +| Inbound access | Guest listeners published on | Admitted clients | +| --- | --- | --- | +| None, the default | No host address | None | +| Host loopback, under an explicit host-loopback allow | Host loopback, `127.0.0.1` and `::1` | The host | +| Private network, under ingress allow | Every host address | Peers on the host's private networks, and the host under an explicit host-loopback allow | + +An explicit host-loopback allow makes host-loopback access bidirectional: +besides the gateway mapping, it publishes the guest's listeners on host +loopback and admits explicit TCP or UDP forwards from host `127.0.0.1` ports +to guest ports, at most 64 of them. Neither binds a LAN-facing listener. + +The host decision names no ports, and the endpoint owns no listener of its +own, so publishing follows the ports on which the guest listens. The machine +profile passes the ingress mode to the guest in a boot token that callers +cannot supply, and a guest agent then announces the guest's listeners. It +sends each announcement as one UDP datagram to a fixed port of the IPv4 +gateway from a privileged source port, which the workload, holding no +capabilities, cannot bind, so the workload cannot forge an announcement. An +announcement carries a magic value, a version, a flag that marks omitted +listeners, and at most 64 entries in strictly increasing order of protocol and +port, each naming the protocol, TCP or UDP, the address families, and the +port. The endpoint consumes announcements before the egress policy, which +neither authorizes nor forwards them, and each announcement replaces the +published set. For each published port and address family, the endpoint keeps +one host listener at the guest's port number: + +- **Private-network ingress** listens on every host address and relays only + peers in ranges that are not globally reachable: the IPv4 private-use, + shared, link-local, benchmarking, and documentation ranges, and the IPv6 + unique local, site-local, documentation, and benchmarking ranges. It refuses + peers inside the guest's own subnets, which the guest would take for its + gateway or its neighbors. Peers at the host's own addresses, including + loopback, need an explicit host-loopback allow. +- **Host-loopback publishing** without ingress allow listens on the family's + loopback address and admits only the host. It never displaces a host + listener that already holds the port, on loopback or on every host address. + +A port that the host cannot bind, for example because a host process holds +it, is retried with the next announcement. While the guest's listeners are +published, the egress policy also admits ARP requests and neighbor +solicitations for the guest's subnet, where the gateway presents host peers. +It admits a frame that its rules deny when the frame continues a flow that the +endpoint carries: a TCP segment other than an initial SYN, or a UDP datagram +from a published port to a peer that recently sent one to it, which leaves +through the port's host listener. + +The endpoint presents host peers at synthetic addresses inside the guest's +subnet of each family, allocated downward from its last usable address, such +as `10.0.0.254` and `fd00::a00:ff` for the identity in Table 3.12. Explicit +host-loopback access therefore requires an IPv4 network of /29 or larger: a +/30 holds only the gateway and the guest and leaves no address for host peers. +The guest routes its replies to a synthetic address through its NIC rather +than its own loopback and answers from its own address in that subnet, and +the endpoint translates the synthetic address back on transmission. The +egress policy evaluates guest traffic to a synthetic address at the host +address that it stands for, and at the gateway of its family for host +loopback, so a synthetic address reaches no host-loopback port that the rules +deny at the gateway. The device's early header check, which knows no synthetic +addresses, leaves them to the endpoint. + +## 3.5.5 DNS + +The gateway provides a DNS proxy over UDP and TCP port 53. The guest is +pointed at the IPv4 gateway when the egress policy permits DNS to it, and +otherwise at the IPv6 gateway when generic rules permit DNS there. +Exact-endpoint policies never name a gateway, and without a permitted gateway +the guest receives no DNS server. + +A guest query passes the normal egress checks before the gateway handles it. +Upstream resolution uses the host resolver, whose upstream servers are not +guest packet destinations, so guest rules do not check them. A direct guest +query to an external DNS server uses the ordinary outbound path and needs +permission for that server. DNS permission and connection permission are +separate: resolving a name never authorizes a connection to the result. +Denying host loopback does not disable gateway DNS that the egress policy +permits. + +## 3.5.6 Bounded State, Snapshots, and Restore + +Flow, DNS, buffer, and queue state are bounded. At most 128 TCP, 256 UDP, and +16 ICMP guest flows are active at once, and excess flows are rejected before a +host socket is created. At most 256 DNS requests may be pending. A resolver +failure, or a UDP query beyond that limit, receives `SERVFAIL`, and a TCP query +beyond it resets its connection. + +The snapshot records the network profile, identity, and policy digest. +Restore reconstructs a fresh endpoint and requires the same network profile +and policy; native sockets, DNS requests, and flows are never serialized. +Published guest listeners and explicit host forwards are process-local +attachments, so capture and restore reject private-network ingress and +explicit host-loopback access, which create them. + +Capture quiesces the endpoint, drains completion ownership, and requires the +saved queues to contain no unrepresented RX or TX packets. It saves the queue +lifecycle, negotiated features, link state, and endpoint generation, but no +in-flight packet payload. After restore, pre-capture packets are not +replayed, earlier TCP, UDP, ICMP, and DNS state is invalid, and traffic +establishes fresh flows. diff --git a/doc/design-doc/3.6-host-filesystem.md b/doc/design-doc/3.6-host-filesystem.md new file mode 100644 index 000000000..67a7dd8e7 --- /dev/null +++ b/doc/design-doc/3.6-host-filesystem.md @@ -0,0 +1,421 @@ +# 3.6 Host Filesystem + +[Architecture documentation](README.md) > +[3. Virtual Devices](3-virtual-devices.md) + +As Figure 3.4 shows, the host filesystem device shares live host files with +the guest through virtio-fs. The guest kernel's FUSE client turns system calls +on the mounted tag into FUSE requests, which travel through virtqueues in +guest RAM to the host filesystem server, a FUSE server inside the VMM. The +server performs each request on the exported host directory, or on the +children of an aggregate, and returns the reply through the same queue. +Unlike a [block device](3.4-block-storage.md), whose backend sees only +blocks, the server sees names, attributes, and open files: the host owns the +namespace and the data, while the guest owns its VFS objects, permission +checks, and locks. Guest writes reach the host files directly, and host +changes are visible to the guest on its next request. + +```mermaid +flowchart TB + App["Guest application"] + Client["Guest VFS and FUSE client
virtio-fs mount of tag microvm"] + Queues["High-priority and request virtqueues in guest RAM
MMIO notifications and shared-status interrupt"] + Device["VMM virtio-fs device
one worker per queue"] + Session["FUSE session
INIT negotiation and request dispatch"] + Server["Host filesystem server
nodes, handles, access policy, and host identity"] + Volume["Confined host volume
no-follow resolution beneath the export root"] + Export["Exported host directory
or aggregate children"] + + App <--> Client + Client <--> Queues + Queues <--> Device + Device <--> Session + Session <--> Server + Server <--> Volume + Volume <--> Export +``` + +**Figure 3.4.** Host filesystem data path. + +## 3.6.1 Device and FUSE Session + +The device is a virtio-fs device with the fixed tag `microvm`, one +high-priority queue, and one request queue. Its configuration space holds only +the tag and the request-queue count. It offers split rings with indirect +descriptors and event-index notification suppression, but never packed rings +or the DAX shared-memory window, so file data always travels through the +queues. The machine profile rejects alternate tags, extra queues, +shared-memory windows, PCI transport, and aggregates whose children change +after boot. + +The guest opens the FUSE session with `INIT`, and every other request fails +until it succeeds. The device speaks FUSE protocol version 7 and accepts guest +minor versions 27 and newer, but a [restore](#369-restore) requires a saved +session that negotiated minor version 31 or newer. From a fixed set, the +device grants what the guest offers: asynchronous reads, parallel directory +operations, automatic data invalidation, kill-privilege handling, asynchronous +direct I/O, atomic `O_TRUNC`, big writes, extended `INIT`, adaptive +`READDIRPLUS`, and shared mappings of direct-I/O files. It negotiates a 1 MiB +maximum write and nanosecond timestamps, and it never grants writeback +caching, lock forwarding, or DAX mappings. + +Every entry and attribute in a reply carries zero cache lifetimes, and every +file that the guest opens or creates uses direct I/O. The guest therefore +looks up each path component again on every walk, fetches attributes whenever +it needs them, and sends every read and write to the host instead of its page +cache: host changes are visible on the guest's next request, and a guest write +has reached the host file when the write completes. Shared mappings are the +exception, because their pages are cached in the guest and written back +through ordinary write requests. Zero lifetimes also make the guest drop a +name and its inode, and send `FORGET`, as soon as nothing uses them, so it +holds nodes only for objects in use and the directories that lead to them, +such as open files, working directories, and mount and bind points. Because +lock forwarding is not negotiated, the guest kernel keeps POSIX and BSD locks +local, and host processes neither see nor honor them. The guest kernel checks +Unix permissions against the attributes that the host reports before it sends +a request, and the host then checks the operation against the +[host identity](#367-host-identity) that performs it. + +## 3.6.2 Export Forms + +As Table 3.14 shows, the slot takes one of three forms. + +**Table 3.14.** Host filesystem export forms. + +| Form | Guest view | +| --- | --- | +| Dormant | No host server and no active policy, but the device remains discoverable; the session can open, but filesystem requests fail with `ENOSYS` | +| Directory export | One host directory as the filesystem root, with one access mode and path policy | +| Aggregate | A read-only synthetic root, mode `0500` and owned by root, that lists named children; each child is an independent volume with its own access mode and path policy | + +A dormant slot lets a snapshot +[gain an export at restore](#3610-dormant-slot-and-saved-state-versions). A +directory export always exports a directory, because the guest mounts the +device's root. The machine profile announces an attached export to the guest +in boot tokens: its guest location, the tag, its mode, which is read-write +when the export or any child is, and, for an aggregate, an aggregate marker. +On a cold boot, the guest mounts the tag at that location with that mode; for +an aggregate, the guest agents then bind each child at its target, as +[Live Host Shares](4.1-guest-filesystem-and-image-model.md#412-live-host-shares) +describes. + +An aggregate's root lists its children in declaration order and refuses every +change with `EROFS`. A child is a host directory or a single regular file. The +declared kind, not the host object, decides which, and the VMM refuses a host +path of the other kind. A file child serves only its file, as the child +itself: its volume is the file's parent directory, narrowed to the file's +name, which the guest never reaches, so the guest cannot list or change +anything beside the file, a lookup below it fails with `ENOTDIR`, and every +request fails with `EACCES` while anything other than a regular file holds the +name. A file child takes no path policy, because its mode covers the whole +file. + +Child names are unique, use `[A-Za-z0-9._-]`, and are at most 64 bytes long. +An aggregate holds at most 256 children with at most 128 KiB of policy paths, +and child host roots must not equal or contain one another, so no child can +reach another's files under a different policy; a file child may sit beside a +directory child. The children share one FUSE superblock, so the server mixes a +per-child key into each host inode number to make inode-number collisions +between children unlikely. A rename between children fails with `EXDEV`. A +hard link between children fails with `EROFS` when its destination is +read-only and with `EXDEV` otherwise. + +## 3.6.3 Request Path + +A guest operation on the mount becomes one or more FUSE requests: + +1. A system call reaches the guest VFS, whose FUSE client issues the requests + that it needs: `LOOKUP` for each path component, `GETATTR` for attributes, + `OPEN` or `CREATE` for a descriptor, and `READ` or `WRITE` for data. +2. The client places each request in a descriptor chain in guest RAM. + Device-readable buffers hold the FUSE header, with the opcode, request ID, + node ID, and the caller's UID, GID, and PID, followed by the request's + arguments, names, and any write payload, and device-writable buffers + receive the reply. `FORGET` notifications use the high-priority queue and + have no reply buffers; all other requests use the request queue. +3. The client publishes the chain and, unless the device has suppressed + notifications, notifies the queue, which wakes the queue's worker. +4. The worker copies the request out of guest memory and parses its header. + In caller ownership mode, it assumes the caller's host identity. The FUSE + session then dispatches the request to the server. +5. The server resolves the request's node ID to a node, or its handle ID to + an open host object, and applies the access mode, access policy, and table + bounds. +6. The confined host volume performs the operation beneath the export root + without following symbolic links. A Linux host issues the corresponding + system call relative to a pinned parent directory; a Windows host emulates + the same Linux semantics through Windows file APIs, without storing Linux + ownership or mode metadata. +7. The server encodes the result, or a negative error number, after an output + header that carries the request ID, and the worker writes it into the reply + buffers. The worker then publishes one used-ring entry and, unless the + guest suppressed the notification, signals the guest through the slot's + shared-status word and edge interrupt. +8. The client matches the reply to its request by ID and completes the system + call. + +Each queue has one worker, which performs one request at a time to completion +before it takes the next. The request queue therefore serializes guest +requests in ring order, although the guest may post many at once, while +`FORGET` notifications proceed independently on the high-priority queue. The +device does not implement FUSE interrupts, so a posted request always runs to +completion. + +A `READ` fills a host buffer of at most 1 MiB and copies the bytes into the +reply buffers, and a `WRITE` copies its payload out of guest memory before it +writes the host file. Requests on an open handle use its host descriptor +directly, without resolving a path again. A request whose device-readable part +exceeds the 1 MiB write limit plus a 4 KiB header allowance, or whose header +cannot be parsed, completes with no reply, and malformed arguments fail with +`EIO` or a specific error. An operation that the server does not implement, +such as `FLUSH`, `FALLOCATE`, `LSEEK`, or `COPY_FILE_RANGE`, fails with +`ENOSYS`; Linux then stops sending it and either falls back to generic +handling or reports the operation as unsupported. A guest `syncfs` is +acknowledged without flushing the host. + +## 3.6.4 Nodes, Handles, and Listings + +The server identifies host objects to the guest by node ID. Node 1 is the +root: the exported directory or an aggregate's synthetic root. A node records +its volume; the volume-relative names, or aliases, through which the guest has +reached it; its host and guest-visible inode numbers; and a lookup count. It +holds no host descriptor. Every request on a node first walks each recorded +name from the export root again without following links, fails with `ELOOP` +if a name now crosses a symbolic link and with `EACCES` if the policy hides +it, and then operates on the final component without following it. + +Repeated lookups of one host object return the same node and add the new name, +so hard links share one node. The server matches objects by host inode number, +or by name on volumes whose inode numbers are not stable. Every reply that +returns an entry adds one lookup, `FORGET` subtracts the count that the guest +reports, and the node disappears at zero. Renames and removals rewrite or +remove the recorded names at and below the affected path in every node, and +an exchanging rename swaps them. + +`OPEN`, `OPENDIR`, and `CREATE` return handles. A handle owns a host +descriptor, opened without following the final component, which every request +that names the handle uses and `RELEASE` closes. The first directory read on a +handle enumerates the host directory once into a listing of at most 4,096 +entries and 64 KiB of names, without the entries that the policy hides, and +later reads page through that listing; listing a larger directory fails with +`E2BIG`. Each entry's cookie is its position in the listing, so a listing +survives a snapshot. `READDIRPLUS` looks each listed name up again and fails +if it no longer reaches the listed object. + +Figure 3.5 follows one file through a read-write directory export from +creation to removal. The server checks a write against the node's access on +every request, not only at open. A completed write is visible to host +processes but is not forced to stable storage until the guest syncs the file. +A removed file stays reachable through its open handles, and its node remains +until the guest forgets it. + +```mermaid +sequenceDiagram + participant App as Guest application + participant Client as Guest FUSE client + participant Server as Host filesystem server + participant Host as Host filesystem + + App->>Client: Open out.log with O_CREAT and O_RDWR + Client->>Server: LOOKUP(1, out.log) + Server->>Host: No-follow lookup beneath the root + Server-->>Client: ENOENT + Client->>Server: CREATE(1, out.log, flags, mode) + Server->>Host: Create and open in the pinned root directory + Server-->>Client: Node M, handle H, direct I/O + App->>Client: Write 1 MiB at offset 0 + Client->>Server: WRITE(H, offset 0, 1 MiB) + Server->>Host: Positional write through H + Server-->>Client: 1 MiB written + App->>Client: Read 4 KiB at offset 0 + Client->>Server: READ(H, offset 0, 4 KiB) + Server->>Host: Positional read through H + Server-->>Client: 4 KiB of data + App->>Client: Unlink out.log + Client->>Server: LOOKUP(1, out.log), then UNLINK(1, out.log) + Server->>Host: No-follow unlink in the pinned root directory + Server-->>Client: Success, and H still reaches the file + App->>Client: Close + Client->>Server: RELEASE(H) + Server->>Host: Close the descriptor of H + Client->>Server: FORGET(M, n) on the high-priority queue + Server->>Server: Drop node M +``` + +**Figure 3.5.** File lifetime on a read-write directory export. + +The server bounds its state at 4,096 nodes, 4,096 handles, 256 names per node, +16,384 names in total, 1 MiB of names, and 4,096-byte paths. It checks these +bounds before it changes the host, so an operation that would exceed them +fails with `ENOSPC` or `E2BIG` instead of creating host state that the device +could not track or save. + +## 3.6.5 Confinement + +The host root is pinned by its platform object identity, device and inode +number on Linux or volume and file ID on Windows, and revalidated when the +export opens; each aggregate child's root is pinned the same way. Every +operation stays confined to the export: host path resolution follows no +symbolic link or reparse point, never leaves the root, and never follows the +final component. On Windows hosts, guest-created links are reparse points that +path resolution never follows, ancestors that are links or reparse points are +rejected, and reserved device names and objects marked hidden, system, +offline, or recall-on-data-access fail with `EACCES`. A read-write export lets +the guest create symbolic links with exact targets that only the guest +resolves; the host never follows them, and a read-only export rejects them. On +Linux hosts, a rename passes its flags through, so the guest can atomically +exchange two names; Windows hosts reject every rename flag with `EINVAL` +before changing anything. + +## 3.6.6 Access Policy + +Read-only mode, of an export or of a child, rejects mutation in the host +device before any host filesystem operation runs. Path policy then narrows +access within an export: + +- **Denied paths** hide complete subtrees. Denied root object identities also + block aliases through hard links, junctions, and bind mounts. A denied path + may name the root of an export or child when allowed paths inside it remain, + which leaves the root traverse-only. +- **Allowed paths** expose subtrees of denied paths again. The nearest policy + path that contains a path decides its visibility, so denied and allowed + paths can alternate. A hidden directory on the way to an allowed path is + traverse-only: the guest can look it up and list only the entries that lead + to allowed paths, and its object identity is accepted only at its own path. +- **Writable paths**, when present, are the only parts of a read-write export + that the guest can modify. Every namespace mutation checks its directories + and the entries it creates, removes, or replaces, and every object mutation + checks every name that the guest has used for the object. Refused writes + fail with `EROFS`, and a hard link to a read-only object from a writable + path fails with `EXDEV`. + +The policy is unchanged by a second guest mount, and guest mount flags are +never a security boundary. + +## 3.6.7 Host Identity + +By default, the host performs guest operations as the VMM's user. In caller +ownership mode, available on Linux hosts, each request that can access host +files runs as the UID and GID in its FUSE header: the queue worker assumes +that identity for its own thread only, with no supplementary groups or +effective capabilities, and restores its own identity before it handles +anything else. Session negotiation, `FORGET`, and handle release run +unchanged. Guest UID 0 and GID 0 are squashed to the owner and group of the +export root, which therefore must not be UID 0 or GID 0. A request whose +identity cannot be assumed fails with `EPERM` rather than running as the VMM. + +## 3.6.8 Snapshot Contents and Capture + +The exported files are external live state, not part of the snapshot: neither +their contents nor any host descriptor, handle, or cache is captured. A +snapshot instead preserves the protocol state on both sides of the queues. +The guest's FUSE client state, including its mount, node IDs, lookup counts, +and handles, lives in guest RAM and is captured with memory. The device saves +the matching state: + +- The negotiated FUSE session. +- Each node's ID, volume, recorded names, including those of symbolic links, + lookup count, guest-visible inode number, and object identity of device, + inode number, and file type. +- Each handle's ID, node, open flags, object identity, and directory listing + with its cookies. +- The next node and handle IDs. +- The fixed device shape, access mode, ownership mode, root identity, and + access policy, or, for an aggregate, each child's name, kind, mode, root + identity, and policy. + +Device state names objects relative to the export root and never records a +host path. The snapshot's machine contract records the host side instead: the +guest target, access mode, tag, queue counts, direct I/O, cache lifetimes, +canonical host path, policy paths, ownership mode, and, for an aggregate, each +child's name, canonical host path, kind, mode, policy, and root identity. Its +attachment inventory binds the export, with a live-revalidation reconnect +policy, to the root's identity or, for an aggregate, to a digest of its +children's names and root identities in order, so a restore can neither drop, +add, nor reorder a child. + +Capture proceeds as follows: + +1. When capture gates host input, each worker finishes its current request, + publishes the reply, and takes no further request. Descriptors that no + worker has taken stay on the available ring, in guest memory and the saved + queue progress, for the restored device to take. +2. After the processors stop at the boundary, the queues stop, and the save + requires that no request remain in flight. +3. The server inspects every handle through its descriptor and resolves every + recorded name beneath the export root again without following links, + checking that all names of a node reach one object. Capture fails while the + device tracks a node that has no name left, such as a removed file that the + guest holds open, or a name that is gone, hidden, crosses a symbolic link, + or reaches another object than the node's other names. It also fails when + the directory listings of all handles exceed 16,384 entries or 4 MiB of + names. +4. Capture does not flush the host files. Data that the guest wrote before the + boundary through file requests is already in them and reaches stable + storage on the host's schedule unless the guest synced it. Pages of shared + mappings that the guest dirtied but has not yet written back stay in + captured memory instead; only a capture with paired scratch writes them + back first, because the guest agents synchronize every filesystem there. + +Capture and restore inspect and reopen objects as the VMM's user, even in +caller ownership mode, so they fail while the guest holds an object that only +its caller can reach or reopen. + +## 3.6.9 Restore + +A restore rebuilds the device's side of the session before the guest can use +it: + +1. Before the guest runs, the launcher supplies the export again. It must + reproduce the snapshot's target, mode, policy, ownership mode, canonical + host paths, and root identities, and, for an aggregate, the same children + with the same names, kinds, and modes in the same order. Only a dormant + slot may instead gain a new export. +2. Before the transport changes any state, the device validates the saved + state: a known schema version, the fixed device shape, a FUSE session that + the device would negotiate, at minor version 31 or newer, tables within + their bounds, unique IDs below the next IDs, and writable handles only + where the export or child is read-write and every name of the node is + writable. +3. When the restored device first starts, before it services any request, the + server rebuilds the node table with the saved node IDs and lookup counts. + It resolves every recorded name beneath the supplied export without + following links and requires the same object, by device, inode number, and + file type, with the same guest-visible inode number. +4. The server reopens every handle with its saved open flags, except that it + drops `O_CREAT`, `O_EXCL`, and `O_TRUNC`, so a restore never creates or + truncates a file. It verifies the identity and kind of each reopened object + and restores the handle's directory listing. +5. The server restores the negotiated FUSE session without a new `INIT`. + +The guest then continues with its mount, working directories, open +descriptors, mappings, and directory listings, without remounting. Because it +caches no entries or attributes, and no file data outside shared mappings, +its next request observes the host as it is at restore time: the restore +checks only the objects that the guest held, and their contents and every +other name may have changed since capture. Pages of shared mappings are the +exception: they return with captured memory, so they hold the file contents +of capture time until the guest kernel invalidates them, and every restored +guest writes the pages that were dirty at capture back to the live host file. +Identity, not content, binds a restore: a copy of the export, even with +identical contents, fails it, and so does a held file that host software +replaced by renaming a new file over its name. Every restore of one snapshot +attaches the same live host objects; unlike paired scratch, the export is not +copied. Native file descriptors and Windows handles are never serialized. + +## 3.6.10 Dormant Slot and Saved-State Versions + +A dormant capture saves explicit unattached state, with no session, nodes, +handles, root identity, or policy. It requires that the guest have no FUSE +session on the slot, so the tag must not be mounted; unmounting ends the +session. A dormant snapshot may restore with no export or bind one new +directory export, but not an aggregate. The resumed guest then mounts the tag +explicitly, because its boot-time mount step has already run. A snapshot +without this capability cannot add an export, and active state cannot restore +into a dormant slot. + +Capture records the saved-state version that the export requires. Caller +ownership, the access policy of a directory export with allowed or writable +paths, an aggregate, and a file child each require a newer version, so a +release that predates the feature rejects the snapshot rather than restore a +weaker export. diff --git a/doc/design-doc/4-guest-runtime-architecture.md b/doc/design-doc/4-guest-runtime-architecture.md new file mode 100644 index 000000000..2acbe7e88 --- /dev/null +++ b/doc/design-doc/4-guest-runtime-architecture.md @@ -0,0 +1,26 @@ +# Guest Runtime Architecture + +[Architecture documentation](README.md) + +The NVX guest runtime turns one microVM into one workload sandbox. The guest +agents supervise the workload for the life of the VM. In the +[layered sandbox root](4.1-guest-filesystem-and-image-model.md#411-layered-sandbox-root), +they remain outside the workload's namespaces; the +[blockless managed root](4.1-guest-filesystem-and-image-model.md#413-blockless-managed-root) +creates none and contains the workload with its credentials and a cgroup. The +architecture does not introduce pod infrastructure, a pause container, or a +guest-side service for creating arbitrary additional containers. + +This chapter describes how the guest turns the virtual machine into an +isolated workload sandbox. + +## Table of Contents + +- [4.1 Guest Filesystem and Image Model](4.1-guest-filesystem-and-image-model.md): + Explains image layers, writable scratch, overlay assembly, and live shares. +- [4.2 Guest Agents and Workload Model](4.2-guest-agents-and-workload-model.md): + Defines the relationship between the guest agents and the workload. +- [4.3 Isolation and Resource Model](4.3-isolation-and-resource-model.md): + Describes identities, namespaces, capabilities, devices, and cgroups. +- [4.4 Host-Guest Control Interface](4.4-host-guest-control-interface.md): + Defines authenticated runtime operations and their session semantics. diff --git a/doc/design-doc/4.1-guest-filesystem-and-image-model.md b/doc/design-doc/4.1-guest-filesystem-and-image-model.md new file mode 100644 index 000000000..7f53986f3 --- /dev/null +++ b/doc/design-doc/4.1-guest-filesystem-and-image-model.md @@ -0,0 +1,140 @@ +# 4.1 Guest Filesystem and Image Model + +[Architecture documentation](README.md) > +[4. Guest Runtime Architecture](4-guest-runtime-architecture.md) + +As Table 4.1 shows, the guest agents run workloads in one of two roots. + +**Table 4.1.** Workload roots. + +| Root | Construction | Host content | +| --- | --- | --- | +| Layered sandbox root | An overlay of one read-only image layer and private writable scratch | Live host shares bound at declared targets | +| Blockless managed root | The guest's own initramfs root, without block devices | Host directories bound by a host mapping table | + +The guest kernel provides the facilities both roots need: virtio-blk, +virtio-fs, EROFS with compression and extended attributes, overlayfs, cgroup +v2 with the memory and process controllers, namespaces, BPF system calls, and +cgroup BPF programs. + +## 4.1.1 Layered Sandbox Root + +As Figure 4.1 shows, the layered root combines one immutable EROFS `distro` +layer with one private ext4 scratch device. + +```text +/run/nvx/layers/distro (EROFS) ----------> lower directory +/run/nvx/scratch/upper (ext4) -----------> upper directory +/run/nvx/scratch/work (same ext4) ------> work directory + overlay --> /run/nvx/rootfs +virtio-fs tag microvm (optional share) ---> /run/nvx/rootfs/TARGET + or an aggregate ---> /run/nvx/shares, each child bound at + /run/nvx/rootfs/TARGET +``` + +**Figure 4.1.** Layered sandbox root assembly. + +The `distro` layer is required and holds a curated base operating system. One +immutable layer file can serve many sandboxes, and compressed EROFS reduces +stored and cached image bytes, because the guest kernel decompresses blocks as +it reads them. No FUSE daemon or in-guest fetch service is involved in the root +filesystem. + +The guest agents assemble the root in this order: + +1. Mount runtime tmpfs and cgroup v2, create sibling guest-agent and + container cgroups, and move the guest agents into the guest-agent cgroup. +2. Resolve each role's MMIO address through the kernel's platform-device + resources rather than assuming a device name or enumeration order. +3. Check any expected sector count, flush cached block-device buffers, mount + the `distro` layer read-only as EROFS, and check its expected filesystem + UUID. +4. Resolve and flush scratch, mount it as ext4 with `rw,nosuid,nodev`, and + create its upper and work directories. +5. Mount the overlay with the `distro` layer as its lower directory, + metadata-only copy-up, and extended inode numbers. +6. Verify the workload identity in the assembled root, mount each live share + at its target inside that root, and write the workload's machine ID to a + runtime file. +7. Start the workload behind a barrier and place it in the workload cgroup + before releasing it. +8. Wait for the workload, unmount the shares in reverse order and then the + overlay, the `distro` layer, and scratch, and report the workload's status. + +The UUID check detects attachment mistakes; it is not a content-integrity +proof, and neither the guest nor the VMM verifies layer contents. +Metadata-only copy-up interprets overlay metadata stored in the `distro` +layer, so that layer is a trusted input that must not carry untrusted overlay +attributes. A metadata-only change avoids copying file contents, but a data +write still copies the whole lower file into scratch, so scratch capacity must +account for copy-up. Scratch consumes host storage rather than guest RAM and +can run out of space independently of guest memory pressure. + +The initramfs remains the guest agents' root; it is not another workload +layer. The guest agents always keep the overlay upper layer on scratch, so the +machine's RAM-backed overlay layout, which has no scratch, serves only guests +that declare it. + +## 4.1.2 Live Host Shares + +A live share exposes a host directory, such as a source checkout or a tool +cache, or a single host file, such as a settings file, to the workload without +copying it into an image: edits are visible on both sides without staging or +copy-back. + +One shared directory attaches as a directory export. Several shares, or any +shared file, attach as an aggregate whose children are independent volumes +with their own access modes and policies; see +[Host Filesystem](3.6-host-filesystem.md). Each child is named +after its index and a digest of its guest target, so a snapshot pins every +target, and a restore that requests another target fails before the guest +runs. Before launch, the launcher rejects policy paths outside their share or +on a shared file, guest targets or host paths that equal or contain one +another, and shares whose boot tokens exceed the command-line budget. + +The machine profile announces the export through its host filesystem tokens, +and the launcher adds one share token per child that names the child, its +guest target, its mode, and whether it is a file. The guest agents then: + +- Parse every share token and create every target before mounting anything, + so a malformed bootstrap mounts nothing. +- Create each target inside the workload root one component at a time and + refuse a path that crosses a symbolic link, so an image layer cannot + redirect a share outside the root. +- Create a regular file as the last component of a file target and a + directory for any other target. +- Refuse repeated children, overlapping targets, and targets that the + runtime later binds over, such as the machine ID file. +- Mount a single share at its target with its mode and `nosuid,nodev`. +- Mount an aggregate at a root-only location outside the workload root and + bind each child at its target with its mode and `nosuid,nodev`. + +The workload's mount namespace inherits these mounts, and teardown unmounts +the binds in reverse order before the aggregate. Any refusal or mount failure +aborts the sandbox instead of starting the workload without its shares. A +managed sandbox records its shares when it is provisioned and reattaches them +on every start. + +## 4.1.3 Blockless Managed Root + +Without image layers, each workload runs directly in the guest's initramfs +root, inside a dedicated execution cgroup. Host directories reach it through a +host mapping table rather than an image: + +- The host exports the outermost mapped directories, and each mapped file + outside them on its own, as numbered children of one aggregate, so a file + child exposes only its file. The guest agents mount the aggregate at a + root-only location before they accept control traffic. +- The host enforces each mapping's access itself: a child is read-write only + if it holds a read-write mapping, and writes are limited to read-write + mappings. A whole volume is never mapped, and a mapping table that would + need more children than an aggregate holds is refused rather than exported + through the directories of its files, which would expose more than the + files. +- The kernel command line announces the number of mappings, and the host + delivers the table over the + [control interface](4.4-host-guest-control-interface.md). The guest agents accept + entries only in order and up to the announced count, validate each request + before mounting anything, and bind-mount each source at its target with + `nosuid,nodev`, read-only when requested. They refuse to run workloads until + every announced mapping is mounted, and a failed mount keeps them refusing. diff --git a/doc/design-doc/4.2-guest-agents-and-workload-model.md b/doc/design-doc/4.2-guest-agents-and-workload-model.md new file mode 100644 index 000000000..c233f71cc --- /dev/null +++ b/doc/design-doc/4.2-guest-agents-and-workload-model.md @@ -0,0 +1,77 @@ +# 4.2 Guest Agents and Workload Model + +[Architecture documentation](README.md) > +[4. Guest Runtime Architecture](4-guest-runtime-architecture.md) + +As Figure 4.2 shows, the guest agents are the guest's long-lived lifecycle +authority. Their responsibilities are: + +1. Initialize runtime filesystems and cgroup v2. +2. Discover and validate the fixed virtual devices. +3. Assemble the workload filesystem. +4. Apply workload identity, isolation, and resource policy. +5. Start the workload behind a placement barrier. +6. Supervise the workload and reap descendants. +7. Announce the guest's listeners when the host publishes them. +8. Participate in capture and post-restore repair. +9. Unmount resources and report the final workload status. + +```mermaid +flowchart TB + Kernel["Linux kernel"] + Agents["Guest agents
outside workload namespaces
in the layered root"] + Root["Assembled workload root"] + Isolation["Identity, capabilities,
and cgroups, plus namespaces
in the layered root"] + Workload["Workload process tree"] + + Kernel --> Agents + Agents --> Root + Agents --> Isolation + Root --> Workload + Isolation --> Workload + Agents -->|"supervise and reap"| Workload +``` + +**Figure 4.2.** Relationship between the guest agents and the workload. + +In a one-shot lifecycle, the guest agents start one workload, configured by +non-secret kernel command-line tokens, and shut down the VM when it exits. In +a managed lifecycle, the guest agents stay resident and run a sequence of +workloads, one at a time, on request over the authenticated control channel. +Both modes preserve the same separation: the workload is always a child of a +guest agent. The guest agents never replace themselves with the workload, +because supervision, orphan reaping, and the control service all depend on +them remaining alive. + +## 4.2.1 Execution Launch + +For each managed execution, the guest agents pass the execution's parameters +through a sealed anonymous file to a launch helper that runs only after the +workload identity has been applied. The helper enters the requested working +directory, or `/`, sets `PWD` unless the request replaces the environment, +applies the environment, and executes the workload. The working directory is +opened once and kept, so the workload starts in the directory that was +checked even if its path is renamed, replaced, or retargeted in the meantime. +If the workload cannot enter it, the execution fails rather than running +elsewhere. + +## 4.2.2 Containment + +In the blockless managed root, each execution runs in its own cgroup. When the +workload's first process exits, the guest agent kills every process that +remains in that cgroup and, as the guest's init process, reaps the orphans it +inherits. It reports the outcome only after the cgroup's process count reaches +zero. +The process controller charges a process until it is reaped, so no workload +process, not even a zombie, outlives its execution. A failed kill, an +unreadable count, or a settlement timeout is reported as a containment +failure, and every later execution first reaps what earlier workloads left and +verifies that the cgroup is empty, so a session reset cannot bypass the +check. + +A cancellation that names the running execution kills the workload, and a +cancellation for an execution that already finished is ignored. A session +reset means that the host session that could observe the workload is gone: +the guest agent kills the workload, sends nothing more for it, and +acknowledges the new epoch, so the next session never waits for an abandoned +workload. diff --git a/doc/design-doc/4.3-isolation-and-resource-model.md b/doc/design-doc/4.3-isolation-and-resource-model.md new file mode 100644 index 000000000..a360fb779 --- /dev/null +++ b/doc/design-doc/4.3-isolation-and-resource-model.md @@ -0,0 +1,72 @@ +# 4.3 Isolation and Resource Model + +[Architecture documentation](README.md) > +[4. Guest Runtime Architecture](4-guest-runtime-architecture.md) + +## 4.3.1 Identity and Credentials + +The workload runs under a host-selected, non-root numeric UID and GID fixed at +initial boot. The guest agents require an exact user and primary-group match +in the workload root and reject an unavailable identity before starting the +workload. When the boot configuration requests it, they instead create an +account for a host-selected identity that the image lacks, and replace an +image account that uses the UID with another primary group or without a +usable home. This lets a host run workloads under its own user's IDs. + +Before the workload starts, the guest agents clear supplementary groups and +every capability set and enable `no_new_privs`. With every capability cleared, +the workload cannot perform port I/O, so only the guest agents can request a +capture or acknowledge restore repair. + +## 4.3.2 Namespaces and Mounts + +In the layered sandbox root, the workload receives private mount, PID, and UTS +namespaces. Its entry helper makes every mount private and provides: + +- A private `/proc` and a read-only `/sys`. +- A restricted `/dev`, devpts, and shared memory. +- The workload root, entered with `chroot`. +- Only the live shares that the host declared. +- The workload's machine ID, bound read-only from a runtime file so that the + guest agents can refresh it while scratch is frozen. +- The guest time status, bound read-only. + +The guest agents never pivot away from the initramfs, so they keep their own +root and stay outside the workload's namespaces, where they can supervise and +tear down the workload. The control device is never exposed inside the +workload's namespaces. + +The blockless managed root has no such isolation. Each execution runs in the +guest's initramfs root, without a `chroot`, in the same mount, PID, and UTS +namespaces as the guest agents, so it sees the guest's own `/proc`, `/sys`, and +`/dev` and the host mappings bound into that root. Its +[credentials](#431-identity-and-credentials) and its +[execution cgroup](#433-cgroups-and-resources) contain it, but it has no +private mount, process, hostname, or device view. + +## 4.3.3 Cgroups and Resources + +In the layered sandbox root, the guest agents and workload occupy sibling +cgroups. The workload is placed in its cgroup behind a barrier before it is +released. The guest agents' cgroup carries a default memory protection of +16 MiB, a soft reservation that shields the supervisor from reclaim under +workload memory pressure but does not guarantee its survival. The workload +cgroup carries optional memory and process-count limits and is the unit of +capture freeze, termination, and descendant cleanup. + +In the blockless managed root, each execution joins a dedicated cgroup before +the workload runs, for termination and descendant accounting. This path does +not configure a sibling guest-agent cgroup, the 16-MiB memory protection, or +the layered root's optional memory and process-count limits. + +## 4.3.4 Enforcement Layers + +Isolation has two enforcement layers: + +- Guest policy constructs the workload's credentials, mounts, capabilities, + and cgroups and, in the layered sandbox root, its namespaces. +- Host devices independently enforce block mutability, network policy, and + host-filesystem access. + +The guest cannot widen a host policy by remounting a filesystem, changing a +route, creating a link, or issuing a different device request. diff --git a/doc/design-doc/4.4-host-guest-control-interface.md b/doc/design-doc/4.4-host-guest-control-interface.md new file mode 100644 index 000000000..d5e8f4813 --- /dev/null +++ b/doc/design-doc/4.4-host-guest-control-interface.md @@ -0,0 +1,115 @@ +# 4.4 Host-Guest Control Interface + +[Architecture documentation](README.md) > +[4. Guest Runtime Architecture](4-guest-runtime-architecture.md) + +Managed operation uses the dedicated control console rather than the boot +console. The VMM owns the outer transport: bounded framing, same-user local +endpoint authorization, capability authentication, reconnect epochs, and +receive-credit backpressure; see +[Control Console](3.3-consoles.md#332-control-console). The guest agents own +the workload protocol carried inside that stream. Boot diagnostics and kernel +messages stay on the consoles, so framed control traffic never shares an +unstructured byte stream with them. + +## 4.4.1 Operations + +Table 4.2 lists the operations of the control protocol. + +**Table 4.2.** Control operations. + +| Operation | Purpose | +| --- | --- | +| `READY` | Reports readiness, and is the success reply to requests that return no other data | +| `FEATURES` | Reports which control behaviors the guest image provides | +| `MAPS` | Delivers a contiguous range of host mapping-table entries | +| `EXEC` | Runs one workload with a timeout, arguments, and an optional working directory and environment | +| `CANCEL` | Kills the running execution that the request names | +| Shutdown | Shuts the VM down gracefully between executions | + +An execution request carries a 32-bit timeout and length-prefixed arguments, +and optionally an absolute working directory of at most 4,096 bytes and up to +256 `KEY=VALUE` environment entries of at most 4,096 bytes each, with +distinct, non-empty keys. The environment either replaces the default +environment or is layered over it, each entry replacing the default variable +of the same name. Output returns as separate standard output and standard +error streams within the credit window, at most 1 MiB of them combined for +each execution. As Table 4.3 shows, an execution that runs ends with an exit +report, which carries an outcome category and a status. + +**Table 4.3.** Exit outcomes. + +| Outcome | Meaning | +| --- | --- | +| `exit` | The workload exited; the status is its exit status | +| `signal` | A signal ended the workload; the status is 128 plus the signal number | +| `timeout` | The workload exceeded its timeout and was killed; the status is 124 | +| `cancelled` | A cancellation killed the workload; the status is 137 | +| `output-limit` | The workload's combined output exceeded 1 MiB, and the guest agents killed it; the status is 125 | +| `failed` | The guest agents could not supervise the workload; the status is 125 | + +Otherwise, the guest agents reply with an error, which carries one of the +categories in Table 4.4 and a status. An error refuses a request that the +guest agents cannot carry out, or takes the place of the exit report of an +execution that could not start or whose processes the guest agents could not +account for. + +**Table 4.4.** Error categories. + +| Category | Meaning | +| --- | --- | +| `launch-failed` | The guest agents could not prepare the execution | +| `cwd-failed` | The workload could not enter its working directory; the status is the error number | +| `containment-failed` | The guest agents could not prove that every workload process exited | +| `mapping-failed` | A host mapping could not be mounted | +| `mappings-incomplete` | Not every announced host mapping is mounted yet | +| `busy` | Another execution is running | +| `invalid-request` | The request is malformed or invalid in the current configuration | +| `unsupported-operation` | The guest does not implement the operation | + +## 4.4.2 Feature Negotiation + +A successful readiness probe does not prove that an image enforces the +behaviors that a host depends on. As Table 4.5 shows, a `FEATURES` request +therefore returns a four-byte little-endian bit mask of control features. + +**Table 4.5.** Control feature bits. + +| Bit | Feature | +| ---: | --- | +| 0 | Cancellation | +| 1 | Reserved; never set | +| 2 | Workload account creation | +| 3 | Per-execution cgroup containment | +| 4 | Per-execution environment | +| 5 | Working-directory refusal | +| 6 | Host mapping table | + +The blockless managed root provides all six features; the layered sandbox +root provides cancellation, workload account creation, and per-execution +environments. A feature bit is added together with the behavior it names. +Hosts ignore unknown bits and trailing bytes, read an `unsupported-operation` +refusal as no features, and terminate a guest that lacks a feature they need +instead of running workloads without the policy they requested. + +## 4.4.3 Session Semantics + +Requests are typed, length-bounded, ordered, and associated with one VM +instance and epoch. Credit limits bound queued data in both directions. +Malformed, unauthenticated, stale, duplicated, or out-of-order records fail +closed. Only one execution runs at a time; any request other than a +cancellation that arrives during an execution is refused as `busy`. Shutdown +is therefore graceful only between executions: a host cancels a running +execution before it requests shutdown, or else terminates the VM, which fails +the execution. + +The control session is process-local even when guest parser state is +snapshotted. Restore establishes a fresh host capability, peer identity, +instance, and epoch, and the guest agents acknowledge the new instance before +any host data reaches them. A request from before the capture therefore can +never be replayed into a clone or a resumed process. + +Secrets and per-execution environments belong on the authenticated control +channel, not in the kernel command line, image layers, scratch, or snapshot +metadata. Boot configuration that the machine profile owns contains only the +non-secret invariants needed before the control channel exists. diff --git a/doc/design-doc/5-vm-lifecycle-and-state.md b/doc/design-doc/5-vm-lifecycle-and-state.md new file mode 100644 index 000000000..fe7e1a612 --- /dev/null +++ b/doc/design-doc/5-vm-lifecycle-and-state.md @@ -0,0 +1,45 @@ +# VM Lifecycle and State + +[Architecture documentation](README.md) + +As Figure 5.1 shows, the VM lifecycle is a state machine with explicit +ownership and commit boundaries. Cold boot creates a new machine generation. +Snapshot terminates a generation after publishing immutable state. Restore +creates a new process generation from that state and reconstructs process-local +resources. + +This chapter defines state transitions from creation through execution, +capture, restoration, and termination. + +```mermaid +stateDiagram-v2 + [*] --> Constructing + Constructing --> Running: boot or restore succeeds + Running --> Capturing: guest snapshot request + Capturing --> Running: rejected before boundary + Capturing --> Published: publication commits + Published --> [*]: source generation terminates + Published --> Constructing: clone or resume + Running --> Stopping: guest exit or host stop + Constructing --> Failed: validation or reconstruction fails + Capturing --> Failed: boundary becomes uncertain + Stopping --> [*] + Failed --> [*] +``` + +**Figure 5.1.** VM lifecycle states and transitions. + +## Table of Contents + +- [5.1 Boot Protocol](5.1-boot-protocol.md): Defines boot inputs, initial + machine state, and kernel handoff. +- [5.2 Readiness and Runtime Control](5.2-readiness-and-runtime-control.md): + Defines readiness boundaries and supported control operations. +- [5.3 Snapshot Protocol](5.3-snapshot-protocol.md): Defines capture initiation, + quiescence, state collection, and publication. +- [5.4 Snapshot Artifact and Attachment Model](5.4-snapshot-artifact-and-attachment-model.md): + Defines persisted artifacts, identity, sharing scope, and external resources. +- [5.5 Restore Protocol](5.5-restore-protocol.md): Defines admission, + reconstruction, guest repair, entropy refresh, and return to execution. +- [5.6 Shutdown and Failure Semantics](5.6-shutdown-and-failure-semantics.md): + Defines orderly exit, teardown, and terminal failure boundaries. diff --git a/doc/design-doc/5.1-boot-protocol.md b/doc/design-doc/5.1-boot-protocol.md new file mode 100644 index 000000000..290ef835b --- /dev/null +++ b/doc/design-doc/5.1-boot-protocol.md @@ -0,0 +1,110 @@ +# 5.1 Boot Protocol + +[Architecture documentation](README.md) > +[5. VM Lifecycle and State](5-vm-lifecycle-and-state.md) + +Cold boot consumes a host-selected kernel, optional initramfs, initial RAM +size, processor capacity, device attachments, and workload policy. The +machine profile validates these inputs and constructs the complete effective +configuration before opening guest execution. + +## 5.1.1 Linux Direct Loader + +The loader treats the kernel, initramfs, command line, topology, +reservations, and guest addresses as untrusted. It: + +1. Requires a little-endian, uncompressed ELF64 image for x86-64 and rejects + the compressed, self-extracting Linux kernel format (bzImage). +2. Loads validated loadable segments at their physical addresses and zeroes + their BSS tails. +3. Places an optional initramfs at the first 2-MiB boundary after the kernel. +4. Builds the Intel MP 1.4 processor, ISA bus, IOAPIC, and legacy interrupt + entries. +5. Builds Linux `boot_params` with the canonical e820 RAM and reservation map. +6. Installs a bootstrap GDT and 4-GiB identity page tables. +7. Enters the kernel in the 64-bit execution mode of x86-64 (long mode) with + paging enabled. + +No firmware executes, and the loader neither parses nor produces the boot +metadata of Xen's direct-boot entry point (Xen notes or start-info +structures), ACPI tables, the pointer through which an operating system +locates them (RSDP), SMBIOS data, or a device tree. The zero page leaves the +ACPI RSDP address zero. + +Table 5.1 lists the initial processor state. + +**Table 5.1.** Initial processor state. + +| Register | Value | +| --- | --- | +| Instruction pointer (`RIP`) | ELF entry point | +| General-purpose register that carries the boot-parameter pointer (`RSI`) | `0x2000`, the `boot_params` zero page | +| Page-table base (`CR3`) | `0x4000`, the identity page tables | +| Processor control (`CR0`) | Protected mode (`PE`) and paging (`PG`) set | +| Extended processor control (`CR4`) | Physical address extension (`PAE`) set | +| Extended feature enable (`EFER`) | Long mode enabled | + +## 5.1.2 Effective Command Line + +The machine profile, not the caller, owns console, device-discovery, and +host-policy tokens. The base command line is: + +```text +earlycon=xe9 console=hvc0 reboot=t panic=-1 +``` + +When the boot virtio-console is present, the kernel console becomes `hvc1`, +and the byte channel remains the early console. User arguments follow the +base tokens. A fresh boot then appends host-owned tokens in this order: + +1. `nr_cpus=`, so Linux sizes its processor state to the validated + topology. +2. The optional fixed workload identity, `nvx_workload_uid=` and + `nvx_workload_gid=`, and the workload lifecycle, `nvx_lifecycle=`. +3. `nvx_snapshot_tier=` for a capture with sandbox blocks. +4. Device-discovery tokens in fixed address order: network, host filesystem, + boot console, sandbox blocks, and the control console followed by + `nvx_control_tty=hvc2`. +5. Network bootstrap tokens, including a DNS server only when the egress + policy permits DNS to a gateway and, last, the ingress mode when the host + publishes the guest's listeners, followed by the host filesystem bootstrap + tokens. + +Callers may not supply `earlycon=`, `console=`, `virtio_mmio.device=`, +`nr_cpus=`, any network or host filesystem bootstrap token, +`nvx_snapshot_tier=`, `nvx_workload_uid=`, `nvx_workload_gid=`, or +`nvx_lifecycle=`. A machine with a control console also applies the +[control-console rules](3.3-consoles.md#332-control-console). The command +line carries no clock parameter: the guest reads its TSC and LAPIC rates from +the [time ABI](3.2-clock-and-time.md#324-rates) MSRs, and `tsc_early_khz=` or +`lapic_timer_hz=`, from the caller or a snapshot, is rejected. + +Launchers add runtime tokens of their own: one share token per host-share +child, the number of host mappings for a blockless managed root, and a request +to create the workload account. A caller may also declare the RAM-backed +overlay layout with exactly one `nvx_overlay_upper=ramfs` token; any other +form of that token is rejected. + +Embedded NULs are rejected, and the complete NUL-terminated command line must +fit in 64 KiB. The effective string and its SHA-256 digest become part of the +snapshot machine contract. A restore reuses that string verbatim and appends +no tokens. + +## 5.1.3 Cold Boot Sequence + +A cold boot: + +1. Selects the CPU profile and rejects clock tokens in the command line. +2. Creates the partition and preflights the backend before any processor + runs: CPU-profile support, the effective CPUID, identity routing, the + synchronized-set primitive, and the absence of TSC scaling. +3. Reads and checks the native TSC and LAPIC rates, and declares them as `F` + and `L`. +4. Loads the guest and sets the generation counter to zero. +5. Starts the state units and releases the processors. + +A cold boot writes no TSC: every backend starts the processors' TSCs +together. The kernel then discovers the fixed virtio-mmio devices and starts +the guest agents, which step the guest clock to host UTC before readiness, +assemble the workload root, apply isolation, and start either the one-shot +workload or the managed control service. diff --git a/doc/design-doc/5.2-readiness-and-runtime-control.md b/doc/design-doc/5.2-readiness-and-runtime-control.md new file mode 100644 index 000000000..1bbac96a9 --- /dev/null +++ b/doc/design-doc/5.2-readiness-and-runtime-control.md @@ -0,0 +1,81 @@ +# 5.2 Readiness and Runtime Control + +[Architecture documentation](README.md) > +[5. VM Lifecycle and State](5-vm-lifecycle-and-state.md) + +Readiness is an explicit lifecycle boundary, not the observation of arbitrary +console text. A cold boot is usable when the guest agents have initialized the +required devices and either prepared the one-shot workload or established the +managed control path. + +## 5.2.1 Control Frontends + +Two frontends drive the VM worker: command-line launch and the management +RPC. Both construct the same machine profile, and the worker applies the same +validation, snapshot boundaries, and management exclusion to both. The +management RPC creates, captures, and restores machines without sandbox +blocks, a network device, or a control console, attaches the byte channel, +the boot console, and one host filesystem export, publishes readiness, and +reports guest exit. Generic host-initiated save and pulse save-and-restore +operations are unavailable for the microVM: capture is always +guest-requested. + +## 5.2.2 Readiness Endpoint + +An orchestrator may supply a single-use readiness endpoint for a restore: an +existing Unix socket on Linux or named pipe on Windows. The VMM connects to it +before the worker launches. When the restored VM first starts, the worker +writes one fixed readiness record after every state unit has started and +before it releases a restored processor. For a gated restore, host input is +still gated at that point, and guest repair and its acknowledgement follow +the event. A connection, write, or flush failure aborts startup, and the +endpoint is never saved in a snapshot. + +## 5.2.3 Management Exclusion + +While the worker holds the snapshot stop guard, its central dispatcher +immediately rejects management operations that could disturb the boundary, +including memory writes, resume or reset, device changes, and state dumps, +rather than queueing them. Snapshot quiesce, rollback, boundary release, and +memory reads remain available. The exclusion applies at the worker boundary +that every management frontend shares. + +## 5.2.4 Managed Sandbox Lifecycle + +As Table 5.2 shows, host launchers present a persistent sandbox with five +operations. + +**Table 5.2.** Managed sandbox operations. + +| Operation | Effect | +| --- | --- | +| Provision | Records the sandbox's configuration, including its host shares | +| Start | Launches the VMM with a freshly generated control capability, reattaches the recorded shares, and waits for guest readiness | +| Exec | Runs one workload over the control channel | +| Stop | Shuts down the VM and releases its runtime resources | +| Deprovision | Removes the sandbox's state | + +Lifecycle transitions fail closed and take effect one at a time per sandbox, +while executions run beside them. The host identifies the running VMM process +of each sandbox, reconciles VMM processes with recorded state, and recovers +interrupted launches. Teardown leaves no VMM process, control endpoint, +capability, or runtime record, while supplied layers and scratch survive. + +The sandbox lifecycle API exposes the same operations with request +validation, backend capabilities, durable records, and stable error codes. Its +default backend launches the VMM directly with the managed lifecycle and the +blockless managed root. An alternative backend delegates the lifecycle to a +separately supplied native library and an image-backed guest. + +## 5.2.5 Authority + +Runtime control is divided by authority: + +- The host controller may request supported VM lifecycle transitions. +- The VMM serializes transitions that affect machine state. +- The guest agents own workload start, stop, supervision, and capture + preparation. +- The workload cannot invoke host management interfaces directly. + +Operations that could disturb a capture or restore boundary are rejected +while that boundary is active rather than queued for later execution. diff --git a/doc/design-doc/5.3-snapshot-protocol.md b/doc/design-doc/5.3-snapshot-protocol.md new file mode 100644 index 000000000..d400ec0e6 --- /dev/null +++ b/doc/design-doc/5.3-snapshot-protocol.md @@ -0,0 +1,171 @@ +# 5.3 Snapshot Protocol + +[Architecture documentation](README.md) > +[5. VM Lifecycle and State](5-vm-lifecycle-and-state.md) + +Snapshot is guest-requested and stops the source generation. It is not a +host-side memory dump or a capture-and-continue operation. + +## 5.3.1 Capture Configuration + +The snapshot destination is configured at launch or through the management +RPC and must not already exist. When no memory backing file was supplied, the +VMM creates file-backed RAM beside the destination, so memory can be published +without copying. A capture with sandbox blocks requires a tier, and +inconsistent tier, clone or resume, and fresh or paired scratch combinations +are rejected. A capture that binds blocks requires the storage generation that +identifies them; see +[Snapshot Identity](5.4-snapshot-artifact-and-attachment-model.md#545-snapshot-identity). + +## 5.3.2 Guest Preparation + +Before they write the snapshot port, the guest agents: + +1. Wait until the boot conformance checks have completed, then run the + capture checks. +2. Save and suppress RCU stall detection, and the debug stall detectors. +3. For paired scratch, freeze the workload cgroup with a bounded wait, + synchronize, and freeze the mounted scratch filesystem, or, for fresh + scratch, require scratch to be unmounted. +4. Write the capture request, with its scratch policy, to the snapshot port. + +A committed capture terminates the source, so the write returns in the source +only when the capture had no destination or failed before its commit point. +The guest agents tell the cases apart by the restore-packet status bit, which +is clear in a resumed source and set in a restored process. In a resumed +source, they remove their barriers and restore the detector settings, and a +failure to do so powers off the guest. A rejected capture likewise thaws every +guest-owned barrier and then releases the stall-detector suppression. + +## 5.3.3 Capture Transaction + +Figure 5.2 shows the capture transaction from the guest's request to +publication. + +```mermaid +sequenceDiagram + participant Guest + participant Port as Snapshot port + participant Worker as VM worker + participant Units as State units + participant Controller + participant Storage as Snapshot storage + + opt Paired mounted scratch + Guest->>Guest: Freeze workload, sync, and freeze scratch + end + Guest->>Port: Write snapshot request and scratch policy + Port->>Worker: Bounded asynchronous notification + Worker->>Units: Gate host input + Worker->>Worker: Stop vCPU at I/O boundary + Worker-->>Port: Complete the port write + Worker-->>Controller: Boundary ready + Controller->>Controller: Preflight destination and contract + alt No destination or rejected preflight + Controller->>Worker: Release boundary + Worker->>Units: Resume host input + Worker-->>Guest: Continue execution + else Capture accepted + Controller->>Worker: Quiesce for snapshot + Worker->>Units: Stop and save in dependency order + Units-->>Controller: State, inventory, time contract, and CPU profile + Controller->>Storage: Copy state, memory, and optional scratch + Controller->>Storage: Publish with no-replace rename + Controller->>Worker: Terminate committed source + end +``` + +**Figure 5.2.** Capture transaction. + +The VMM performs the capture as one bounded transaction: + +1. Gate host input and defer completion of the snapshot-port write. +2. Stop the virtual processor at the I/O boundary while completing the write, + so saved state starts at the instruction immediately after the output. +3. Preflight the machine profile, destination, backend, external attachments, + and snapshot eligibility; a missing destination or a rejected preflight + releases the boundary and lets the guest continue. +4. Quiesce the remaining device workers and VM time in dependency order, + including the bounded block queue drain. +5. Establish the time contract: require that no LAPIC timer is periodic or in + TSC-deadline mode and that PIT channel 0 is not counting periodically, take + the capture anchor and host identities, and record the declared rates, CPU + profile, effective CPUID, and generation counter. +6. Save the exact state-unit inventory, processor state, device-private state, + and memory, and derive `Q` from the saved VM-time state. +7. Copy and flush state, memory, and any paired scratch into a unique sibling + staging directory. +8. Publish the directory with one no-replace rename. +9. Terminate the source worker and process after publication. + +A failure in step 5 is a rollback-safe capture failure. Deriving `Q` never +fails a capture; when it cannot be derived, it is omitted with a warning. + +## 5.3.4 State-Unit Ordering + +As Figure 5.3 shows, state-unit transitions follow registered dependencies +rather than one global serial list. The partition depends on the device +prerequisite units and on VM time. Save quiesce stops units in reverse +dependency order, and restore and rollback start them in dependency order. +Device prerequisites and VM time may transition in parallel. + +```mermaid +flowchart LR + subgraph Capture["Capture and save"] + direction LR + Gate["Gate host input"] + Boundary["Stop vCPU at
PMIO boundary"] + StopPartition["Stop partition
state unit"] + StopDevices["Stop device
prerequisite units"] + StopTime["Stop VM time"] + Save["Save stopped
state inventory"] + + Gate --> Boundary --> StopPartition + StopPartition --> StopDevices --> Save + StopPartition --> StopTime --> Save + end + + subgraph Restart["Restore or rollback start"] + direction LR + RestoreState["Restore stopped state"] + StartDevices["Start device
prerequisite units"] + StartTime["Start VM time"] + StartPartition["Start partition
and vCPU"] + ResumeInput["Resume host input"] + + RestoreState --> StartDevices --> StartPartition + RestoreState --> StartTime --> StartPartition + StartPartition --> ResumeInput + end + + Save -. committed snapshot .-> RestoreState +``` + +**Figure 5.3.** State-unit ordering during capture, restore, and rollback. + +## 5.3.5 Commit Point + +The directory rename is the commit point. Before it, a capture failure exposes +no final snapshot, and the guest resumes only when rollback is known to be +valid. After it, the source never resumes. This gives the guest an observable +boundary: code after the snapshot request runs once in each restored process +and never in the successfully captured source. A capture driven through the +management RPC likewise halts the source at the committed boundary and +terminates its process, so clients observe the transport closing. + +Establishing the boundary is itself fallible. If host input cannot be gated, +the worker rolls the gate back and releases the write so that the guest +continues; if that rollback is uncertain, the worker terminates. Once +processor stopping has begun, a boundary failure is terminal: host input stays +gated, and the VM is torn down rather than resumed. The post-restore +acknowledgement boundary follows the same rule. + +## 5.3.6 Guest and Host Barriers + +Guest and host barriers have separate jobs. The workload cgroup is frozen in +captured guest state, the VMM gates external input, and a guest agent with +worker threads must park them at a capture-safe point. None of these +substitutes for the others. Device workers and the guest kernel must run +during a gated restore so that processors and memory can be repaired, so the +gate does not imply that interrupts stay disabled. After repair, the guest +agents acknowledge the VMM gate before thawing scratch and then the workload. diff --git a/doc/design-doc/5.4-snapshot-artifact-and-attachment-model.md b/doc/design-doc/5.4-snapshot-artifact-and-attachment-model.md new file mode 100644 index 000000000..5b8c131f3 --- /dev/null +++ b/doc/design-doc/5.4-snapshot-artifact-and-attachment-model.md @@ -0,0 +1,211 @@ +# 5.4 Snapshot Artifact and Attachment Model + +[Architecture documentation](README.md) > +[5. VM Lifecycle and State](5-vm-lifecycle-and-state.md) + +## 5.4.1 Artifact Layout and Publication + +Figure 5.4 shows the files of a published snapshot, and Figure 5.5 shows how +they are published. + +```text +snapshot/ +|-- manifest.bin +|-- state.bin +|-- memory.bin +`-- scratch.img # paired scratch only +``` + +**Figure 5.4.** Snapshot directory layout. + +```mermaid +flowchart LR + State["Stopped VM state"] + Ram["Exact captured base-RAM
file handle"] + Scratch["Optional exact scratch
file handle"] + + subgraph Staging["Unique private sibling staging directory"] + direction TB + StateFile["state.bin
write and flush"] + MemoryFile["memory.bin
exact captured base RAM
from automatic or supplied backing"] + ScratchFile["scratch.img
copy, check length, and flush"] + ManifestFile["manifest.bin
record contract and lengths
write and flush before automatic RAM link"] + end + + SyncStaging["Flush staging directory"] + Rename["No-replace same-parent rename
publication commit point"] + Final["Final snapshot directory"] + SyncParent["Flush parent directory"] + Cleanup["Remove owned staging directory"] + + State --> StateFile --> ManifestFile + Ram --> MemoryFile --> ManifestFile + Scratch -. paired policy .-> ScratchFile --> ManifestFile + ManifestFile --> SyncStaging --> Rename --> Final --> SyncParent + Staging -. failure before commit .-> Cleanup +``` + +**Figure 5.5.** Snapshot publication. + +Publication writes and flushes every artifact into a unique private sibling +staging directory with restrictive permissions, flushes that directory, +commits it with one no-replace rename in the same parent, and then flushes the +parent. A failure before the commit removes the staging directory, and a +pre-existing destination is never deleted or replaced. + +VMM-created RAM backing is published without copying. Once its shared +mappings, the state, and the manifest are durable, the exact open backing file +is hard-linked into the staging directory, and the link is reopened and +checked against the source's file identity and length. Where the filesystem +cannot link the handle, and for caller-supplied backing, publication copies +the memory instead, cloning extents where the filesystem supports it. After +an automatic RAM link commits, the source VM is terminal. A failure after the +staging link is created is rollback-safe only after the private staging +directory has been removed and its parent flushed; otherwise, the VMM +terminates the source rather than resume it with a writable alias of the +artifact. + +## 5.4.2 Manifest + +The manifest is authoritative for: + +- The machine profile and ABI versions. +- The source hypervisor and architecture. +- Captured RAM ranges and file offsets, plus any immutable RAM capacity, the + 128-MiB memory block size, and the canonical expansion ranges. +- Processor topology, processor count, and every APIC identity. +- The effective command line and its digest. +- The exact device order, stable IDs, state-unit names, MMIO and PMIO ranges, + interrupts, transport, feature masks, and queue limits. +- The time contract: the declared TSC and LAPIC rates, the rate tolerance, the + capture anchor, the host clock kind, the host and boot identities, the + generation counter, and `Q`. +- The CPU profile record: the CPU profile's ID, digest, and canonical + document, the effective CPUID, and the capture host's CPU signature. +- Required host attachments and their policies. +- Block roles, access, geometry, layer identities, scratch policy, and + paired-scratch materialization. +- The snapshot tier, clone or resume policy, and consumed configuration + sections. +- The exact lengths of the state and memory artifacts, and the exact length + and identity of paired scratch. + +## 5.4.3 Validation and Integrity + +Snapshot paths and repeated fields are bounded. Restore rejects truncated, +oversized, malformed, wrong-type, path-escaping, symlinked, incompatible, +missing, extra, or reordered state before guest execution. The manifest stores +no checksums, and the VMM never reads an artifact or a block to verify its +contents, so a same-length change to any of them is outside the validation +contract: integrity relies on host access control and on the storage owner +keeping each block's storage generation immutable, and authenticated export +or transport belongs outside the local artifact format. Paired scratch must +match the captured filesystem state, so restore checks it structurally and by +exact size and storage generation. + +Restore opens the snapshot directory once and resolves every artifact, +including the resume claim, relative to that directory handle. It holds the +directory and artifact handles until teardown. Windows hosts open the +artifacts with read sharing only, reject reparse points, and check file +identity and end of file around copy-on-write mapping creation, so writes, +truncation, deletion, and rename stay blocked. Linux hosts keep the exact +descriptors, opened without following symbolic links, and detect metadata +changes before worker handoff, so pathname replacement has no effect. Content +immutability beyond that depends on host access control. + +## 5.4.4 Tiers + +As Table 5.3 shows, snapshots with sandbox blocks declare one of three +lifecycle tiers. + +**Table 5.3.** Snapshot tiers. + +| Tier | Restore policy | Scratch | Consumed configuration | Sharing scope | +| --- | --- | --- | --- | --- | +| `platform` | Clone | Fresh | Invariants only | Trusted platform build | +| `workload-start` | Clone | Paired | Invariants, image binding, and sandbox | Tenant-scoped warm workload or runtime | +| `instance-checkpoint` | Resume | Paired | Invariants, image binding, and sandbox | One live instance with one claimed continuation | + +Tier metadata requires sandbox blocks; blockless snapshots do not declare a +tier. The saved host-owned tier token must agree with the manifest. Platform +lower layers have unbound identities, whereas later tiers bind every consumed +layer. The consumed-section set records which configuration the guest had +already consumed at capture. + +The platform point removes kernel and guest agent initialization from later +launches. A workload-start snapshot also removes runtime initialization, but +it contains workload memory and often cached image data and secrets. +Read-only layer files remain external attachments, yet bytes read from them +can appear in captured memory. Paired scratch is an artifact in the snapshot +directory, not a reference to the caller's original writable file. + +Sharing scope is a deployment requirement, not a tenant-authorization feature +of the VMM. The snapshot controller enforces tier combinations, layer +binding, scratch pairing, and the single-use resume claim. Host artifact +access control, template provenance, and a trustworthy pre-capture guest +determine whether cross-tenant publication is permissible. + +## 5.4.5 Snapshot Identity + +A snapshot binds each consumed block by role, access mode, exact length, and +block geometry, plus one caller-supplied, nonzero storage generation that +every bound block shares. Neither capture nor restore reads a block to +identify it: capture records the generation, and restore requires it and +checks the structural fields. The storage owner therefore vouches for the +bytes and must keep the generation immutable and bound to the exact files. +The generation is not a byte digest, and the VMM does not authenticate it. + +A sealed snapshot is immutable. A clone creates a separate writable +descendant, while a resume atomically claims the snapshot and transfers its +continuation into a live instance, after which it is no longer an available +snapshot. There is no mutable-snapshot state. + +## 5.4.6 Replay Contract + +Restoring replays captured guest state, so capture follows an explicit +contract: never capture live external connections or other state that cannot +be safely reused, never repeat irreversible side effects, and repair runtime +randomness before accepting work. Kernel entropy reseeding cannot reset a +runtime's own random-number generators, caches, or connections, so a +workload-start snapshot requires a runtime post-restore hook. A cloneable +warm point must be single-threaded or hold every peer thread behind a +runtime-owned barrier until repair completes. An instance checkpoint is one +continuation, and a workload-start snapshot is a cloneable starting point; +neither is transparent checkpointing of arbitrary requests. + +## 5.4.7 Host Attachments + +As Table 5.4 shows, machine state contains guest-visible progress, not native +process resources, and each external resource has a stable ID and a +declarative reconstruction policy. + +**Table 5.4.** Host attachment reconstruction. + +| Resource | Saved | Reconstructed or supplied on restore | +| --- | --- | --- | +| Byte channel | Pending RX and TX bytes | Host endpoint, a fresh generation ID, and an optional restore packet | +| Boot console | Queue progress, staged RX, partial TX, endpoint identity, and reconnect policy | An eligible listener at its captured endpoint or an approved fresh endpoint of the same kind, a required client connection, or a supplied inherited terminal | +| Control console | Attachment identity, broker policy, and bounded broker state without queued records, host output, or receive credit; never the capability or peer identity | An approved same-user local endpoint, possibly at a fresh location, with a fresh capability; the broker restarts with a fresh instance and epoch | +| Network | Static identity, queue and packet progress, network profile, and policy digest | A fresh in-process endpoint with the matching policy; published guest listeners and host forwards are rejected | +| Host filesystem | FUSE namespace, handles, cookies, root and object identity, access mode, and path policy, or explicit dormant state | A fresh export with the same canonical path, target, mode, and policy; a dormant slot may stay unattached or bind a new directory export | +| Sandbox block | Queue and device state, roles, access, geometry, layer identities, scratch policy, and paired-scratch materialization | Matching read-only layers, plus private-copy or copy-on-write scratch for clones, direct scratch after an exclusive resume claim, or a new scratch file of the same geometry | +| Readiness endpoint | Nothing | An optional single-use endpoint for one restore | + +Attachment resolution happens before any processor starts. A supplied +attachment must reproduce the saved identity, except that an eligible +listener may replace only its endpoint while keeping its stable ID, attachment +and backend kinds, reconnect policy, required flag, length, and timeout. +Missing privileges, endpoint binding failures, a changed egress policy, +replaced filesystem objects, or a wrong attachment kind fail restore +explicitly. Orchestrators of reusable clones supply fresh private boot-console +and control endpoints, so that independent or concurrent restores never +collide with the terminated source. + +## 5.4.8 Memory Sharing + +Private copy-on-write restore prevents a VM's writes from modifying the +reusable artifact. It does not guarantee that clean physical pages are +unshared across VMs that map the same file, and dropping the guest page cache +before capture does not by itself eliminate shared host physical pages or +cache side channels. Write isolation and page-sharing isolation are therefore +separate properties. diff --git a/doc/design-doc/5.5-restore-protocol.md b/doc/design-doc/5.5-restore-protocol.md new file mode 100644 index 000000000..3cdb40447 --- /dev/null +++ b/doc/design-doc/5.5-restore-protocol.md @@ -0,0 +1,331 @@ +# 5.5 Restore Protocol + +[Architecture documentation](README.md) > +[5. VM Lifecycle and State](5-vm-lifecycle-and-state.md) + +Restore treats the manifest as the source of machine configuration. A failure +of validation or reconstruction rejects the restore before a restored +processor runs. Guest repair runs afterward on the restored processors, and a +failure there powers the guest off, as +[Guest Repair](#557-guest-repair) describes. + +## 5.5.1 Restore Sequence + +Figure 5.6 summarizes the restore flow. + +```mermaid +flowchart LR + Manifest["Bounded manifest"] + Config["Authoritative machine config
and host attachments"] + Verify["Validate state, base RAM, target, and blocks
types, geometry, and identities"] + Cow["Private base-RAM mapping,
fresh expansion, and scratch"] + Machine["Construct partition
and exact devices"] + Validate["Validate destination CPU
and saved device state"] + Restore["Restore time, devices,
partition, and vCPU"] + Run["Start vCPU"] + + Manifest --> Config --> Verify --> Cow --> Machine --> Validate --> Restore --> Run +``` + +**Figure 5.6.** Restore flow. + +Before worker construction, restore: + +1. Opens one exact snapshot directory, reads its bounded manifest, and + requires manifest version 6. +2. Validates the time contract and the CPU profile record. +3. Requires the destination backend to equal the recorded source backend. +4. Requires a matching CPU profile and a host CPU of its generation. +5. Preflights the downtime by sampling the host clocks, selecting the + downtime source, and checking the bounds of `D` and `D + Q`. +6. Derives the authoritative machine configuration and validates its + topology, devices, attachments, command line, and blocks. +7. Resolves console, network, filesystem, policy, and read-only layer + attachments by stable ID or role. +8. Validates the state, captured memory, selected RAM target, block geometry + and identities, and any paired scratch, and claims a resume-policy + snapshot. +9. Generates the restore entropy and generation ID, and sets the generation + counter to the captured value plus one. +10. Maps captured memory privately with copy-on-write, adds fresh private + backing for selected expansion ranges, and materializes scratch according + to its recorded policy. + +In the worker, with every processor stopped, restore: + +11. Creates the partition and the exact device inventory with the selected + active RAM and the snapshot's capacity, declaring `F = F_s` and + `L = L_s`. +12. Preflights the backend before any processor runs or any processor state + is restored: CPU-profile support, the effective CPUID, identity routing, + the synchronized-set primitive, and the absence of TSC scaling. +13. Reads the native TSC and LAPIC rates and applies the + [rate policy](3.2-clock-and-time.md#324-rates). +14. Restores every state unit while stopped: VM time, chipset and virtio + devices, the time platform, the partition, and every processor. +15. Freezes the set of instantiated processors. +16. Takes the restore anchor, computes `D`, and writes the same target to + every instantiated processor's TSC at one host instant, never applying + saved per-processor TSC values: + + ```text + TSC_target = T_c + floor(D * F_s / 1_000_000_000) + ``` + +17. Reads back every processor's TSC, after which guest time runs from the + restore anchor. +18. Advances every counting one-shot LAPIC timer by `D`, and VM time and the + RTC by `D + Q`, while the idle PIT catches up from its saved cursor. +19. Seals the time fields of the restore packet. +20. Starts the state units, with host input gated when the restore requires + an acknowledgement, and publishes the readiness event. +21. Releases the restored processors. +22. For a gated restore, on the guest's acknowledgement, stops at the exact + post-write boundary, releases host input, and only then lets the guest + continue. + +The partition must exist before the VMM can read the CPUID it presents and +compare it with the effective CPUID. This does not expose a partially +restored guest, because these checks and all saved-state validation complete +before a processor runs. + +The management RPC performs the same authoritative restore for snapshots +without sandbox blocks, a network device, or a control console. Its request +may repeat only the machine profile, an exact processor-count assertion, the +byte-channel endpoint, the saved console and filesystem attachments, and guest +power actions, and it may add processor and RAM targets, a gate timeout, and +a readiness endpoint. The restored VM stays paused until the client resumes +it. That resume publishes the readiness event, and a publication failure fails +the resume and tears the VM down. + +## 5.5.2 Memory and Scratch Materialization + +Captured memory is mapped privately with copy-on-write in every restore, +while selected expansion ranges receive fresh private backing. As Table 5.5 +shows, paired scratch records one of three restore policies. + +**Table 5.5.** Paired-scratch restore policies. + +| Policy | Behavior | +| --- | --- | +| Private copy | Attaches an independent copy of the paired scratch | +| Copy-on-write | Attaches a filesystem reflink clone; valid only for a reusable workload-start clone | +| Direct claimed | Attaches the paired scratch itself, only after an instance checkpoint has been durably and exclusively claimed | + +Clone restores therefore dirty independent files and leave the reusable +snapshot immutable, while a direct-claimed restore transfers the single-use +continuation into ordinary live-instance ownership. + +An instance checkpoint is single-use. The first restore attempt that passes +artifact and configuration validation atomically creates and flushes a claim +marker in the snapshot directory. Duplicate and concurrent restores fail +before worker construction, and a later startup failure does not make the +checkpoint reusable. The single-use rule prevents accidental forks; it does +not replace artifact confidentiality, retention policy, or protection against +host-side copying. + +## 5.5.3 Processor Activation + +Restore-time processor activation separates the immutable processor capacity +that a snapshot records from the processors that one restored process brings +online: + +- `C` is the processor capacity, which the processor topology, APIC IDs, MP + tables, and saved processor inventory always contain exactly. +- `B` is the boot-online count that a template records at capture: 1, 2, 4, + or 8, and no larger than `C`. A snapshot that records no boot-online count + rejects activation. +- `N` is the online target that a restore selects, with `B <= N <= C`. It is + process-local and never modifies the snapshot. + +The online set is always the contiguous prefix `0..N-1`. Activation is not a +topology change or a general CPU-hotplug interface: the guest still sees the +capacity-`C` machine, and bringing a suffix processor online after readiness +is outside the contract. + +Saved processor state remains authoritative for the full capacity. Restore +requires exactly one processor state entry for every index in `0..C-1` and +rejects missing, duplicate, or out-of-range entries, including errors in the +dormant suffix. A backend may then instantiate only the requested online +prefix and apply saved state to that prefix alone, or instantiate the full +capacity. A process restored with a reduced prefix can never produce a +complete capacity-`C` inventory, so it cannot be captured again, and access to +an uninstantiated processor fails explicitly. The immutable snapshot remains +reusable at any valid target. + +The restore packet carries the online target `N`, and byte-channel status bit +2 advertises it. While host input remains gated, the guest onlines processors +`B` through `N-1`, verifies that its online set is exactly the requested +prefix, and acknowledges through the snapshot port. + +A fixed-capacity `N`-processor snapshot and a capacity-`C` template with +`B` boot-online processors, restored to `N`, reach the same online prefix but +not the same guest state: the first captured all `N` processors online, +whereas the second activates processors during gated repair. + +## 5.5.4 Memory Activation + +Restore-time memory activation separates the immutable RAM geometry recorded +by a snapshot from the active amount that one restored process uses: + +- `M0` is the captured base RAM, which the memory artifact, the e820 map, and + the saved RAM-range inventory contain exactly. +- `Cmem` is the optional immutable capacity declared at capture, with the + 128-MiB memory block size and every canonical range of `Cmem - M0`. +- `M` is an explicit restore target, 128-MiB aligned, with + `M0 <= M <= Cmem`. Omitting it restores exactly `M0`, and a snapshot + without a declared capacity rejects an explicit target. + +The selected expansion is always a prefix of the canonical ranges in logical +RAM order. Expansion bytes are never read from or written back to the +snapshot, so independent restores may choose different targets without +changing the reusable artifact. All selected ranges are registered with the +hypervisor before any restored processor runs. + +An explicit target sets the restore packet's memory-target flag, and the +packet lists the selected expansion ranges; byte-channel status bits 3 and 4 +mirror the target and the presence of ranges. For a nonempty expansion, host +input stays gated while the guest verifies the memory block size and each +range, adds and onlines the missing memory blocks, verifies their state, and +completes entropy and generation-ID repair before acknowledging. A malformed +range or a failure to add or online memory terminates the restore. An +explicit base-size target has no ranges and gates the restore only when the +tier or a processor target requires it. + +## 5.5.5 Restore Packet + +Figure 5.7 shows the layout of the restore packet, and Table 5.6 lists its +flags. Every restore exposes one restore packet through the byte channel, for +every tier and for untiered snapshots. + +```text +offset size field + 0 3 magic "OVR" + 3 1 version = 4 + 4 1 flags + 5 1 online_vp_count 0 = no processor target, else 1, 2, 4, or 8 + 6 1 memory_range_count 0 unless MEMORY_TARGET + 7 1 reserved = 0 + 8 4 generation u32, little-endian, g >= 1 + 12 4 rate_deviation i32, little-endian, ppm scaled by 2^16 + 16 8 downtime_ns u64, little-endian, D + 24 8 utc_ns u64, little-endian, host UTC at selection + 32 16*n ranges n x { u64 gpa_start, u64 length }, LE + 32+16n 64 entropy +``` + +**Figure 5.7.** Restore packet layout. + +**Table 5.6.** Restore packet flags. + +| Flag bit | Name | Meaning | +| ---: | --- | --- | +| 0 | `DOWNTIME_UTC` | `D` came from the UTC delta; clear means same-boot host monotonic time | +| 1 | `MEMORY_TARGET` | An explicit RAM target was requested; the range count is valid and may be zero | +| 2 | `ACK_REQUIRED` | Host input is gated, and the guest must acknowledge through the snapshot port after repair | +| 3 | `TEST_HOOKS` | Host overrides of the downtime, UTC, rate deviation, time samples, or CPU profile are active | +| 4-7 | | Zero; the guest rejects a packet with any of them set | + +The UTC field is latched when the guest first selects the packet; every other +field is sealed before the first restored processor runs. A packet without +ranges is 96 bytes. The packet is authoritative, and the status bits mirror +it; the acknowledgement flag spares the guest from inferring gating from the +tier and targets. A VM process exposes at most one packet, and reading its +last byte consumes it, so a guest captured again in the same process never +sees an earlier packet. The guest also ignores a packet whose generation +equals its recorded generation. + +## 5.5.6 Generations + +- The generation counter `g` is zero in a cold-booted process and the captured + value plus one in a restored process. It orders restores within a snapshot + lineage, clones of one snapshot share it, and the restore packet and every + time sample carry it, so the guest observes a restore between two samples + as a change of `g`. +- The generation ID is 16 random bytes unique to each VM process, selected + through the byte channel. On restore, it equals the first 16 bytes of the + packet's entropy; on cold boot, it is generated independently. It is never + serialized or restored from device state, and it distinguishes clones. +- After a restore, the guest requires the packet's `g` to equal its recorded + `g` plus one, and the generation ID to differ from its recorded one. Either + mismatch fails repair. + +## 5.5.7 Guest Repair + +A restored guest performs its repair on the readiness path, and nothing else +runs before the acknowledgement: + +1. Read the byte-channel status, whose restore-packet bit is always set after + a restore. +2. Bracket the packet selection with realtime clock reads, read the packet + four bytes per access, and validate its magic, version, reserved bits, + counts, generation, and generation ID. +3. Set the wall clock: derive the offset and its uncertainty from the + bracket, take up to two further time samples if the uncertainty exceeds + 1 ms, step the clock by the offset, and set the frequency correction to the + negated rate deviation, clamped to ±500 ppm. +4. Activate processors and memory as requested. +5. Refresh entropy and identity, as + [Entropy Refresh](#558-entropy-refresh) describes. +6. Acknowledge through the snapshot port when `ACK_REQUIRED` is set. +7. After the acknowledgement, finish asynchronously at idle priority: rerun + the restore conformance checks, wait for the stall-detector grace period, + restore the detector settings, and restart the wall-clock discipline at + its fast cadence. + +A failure in the time ABI's part of the repair, which validates the packet, +steps the wall clock, and restores the stall-detector settings, emits a +violation event and powers off the guest with status 195, and a failed rerun +of the conformance checks powers it off with status 193. A failure in any +other repair step writes an error line and powers off the guest with status 1. +Until the acknowledgement, a gated restore keeps host input gated, so no host +input reaches a guest whose repair failed, and the VMM stops a guest that does +not acknowledge within the restore's gate timeout. + +Validation and resource reconstruction may create process-local objects, but +they never expose a partially restored guest: no virtual processor runs after +either fails. + +## 5.5.8 Entropy Refresh + +Restoring a snapshot replays the guest's captured memory, including the state +of the kernel's random-number generator and of any generators that the +workload runtime keeps. Without a refresh, clones of one snapshot would +produce the same random numbers, and therefore the same keys and identifiers. +To prevent this, every restore delivers fresh entropy, which a clone consumes +before any workload code runs: + +1. Before any processor runs, the VMM draws 64 bytes from the host operating + system's cryptographic random-number generator. +2. The bytes reach the guest in the [restore packet](#555-restore-packet), + and their first 16 bytes also become the process's + [generation ID](#556-generations). +3. For a clone, the guest agents credit all 64 bytes to the kernel's entropy + pool and force the kernel's random-number generator to reseed, so all + later kernel randomness depends on the fresh bytes. +4. They refresh the workload's machine ID, which they derive from the fresh + bytes, and its UTS namespace. +5. The workload runtime's post-restore hook resets the runtime's own + random-number state, which a kernel reseed cannot reach. +6. The guest agents acknowledge the restore and only then thaw scratch and + the workload cgroup. A workload captured in a snapshot stays frozen until + this point, and a platform snapshot is captured before the workload + starts, so no workload code runs with the replayed state. + +As Table 5.7 shows, the refresh depends on the restore path. + +**Table 5.7.** Entropy and identity refresh by restore path. + +| Restore path | Guest random-number generator | Machine identity | +| --- | --- | --- | +| `platform` or `workload-start` tier (clone) | Reseeded before the acknowledgement with the packet's 64 fresh bytes | Machine ID and hostname refreshed; the runtime hook runs if present, and `workload-start` requires it | +| Untiered, with processors or memory to activate | Reseeded before the acknowledgement, as for a clone | Refreshed, as for a clone | +| `instance-checkpoint` tier (resume) | Not reseeded, by design; only the generation ID changes | Preserved | +| Untiered, with nothing to activate | Not reseeded; the fresh entropy is handed to the snapshot's caller | Preserved | + +An instance checkpoint is not reseeded because it is single-use: its random +state continues in exactly one process, and keeping it preserves the +continuity of the resumed instance. Fresh post-restore entropy and the +replacement generation ID are never stored in the reusable snapshot, so +independent restores of one snapshot receive independent bytes; only the +prior generation ID remains in captured guest state, as the comparison token. diff --git a/doc/design-doc/5.6-shutdown-and-failure-semantics.md b/doc/design-doc/5.6-shutdown-and-failure-semantics.md new file mode 100644 index 000000000..1873d6e63 --- /dev/null +++ b/doc/design-doc/5.6-shutdown-and-failure-semantics.md @@ -0,0 +1,68 @@ +# 5.6 Shutdown and Failure Semantics + +[Architecture documentation](README.md) > +[5. VM Lifecycle and State](5-vm-lifecycle-and-state.md) + +Orderly guest shutdown carries the workload status through the platform +shutdown device. The VMM stops virtual processors and devices, drains the +bounded final output path, releases attachments, and reports completion only +after teardown reaches a known state. + +## 5.6.1 Final Output and Exit Status + +Guest-requested exit drains the byte channel and, when present, its host +output relay before reporting completion, within one combined five-second +deadline. Success preserves the guest's exit status; an endpoint, relay, or +timeout failure becomes a process-exit error instead of silently discarding +final output. This guarantee covers the byte channel and its relay, not every +console endpoint. Under the management RPC, a zero guest status completes a +pending wait and stops the server cleanly, while a nonzero status or a failed +drain fails the wait and ends the server with an error. After teardown, the +VMM can write the bounded +[outcome report](1.4-trust-boundaries.md#145-host-facing-outputs). + +As Table 5.8 shows, the guest reports its own fail-fast conditions through +exit statuses, each after a violation event or an error line that names the +cause. + +**Table 5.8.** Guest fail-fast exit statuses. + +| Status | Cause | +| ---: | --- | +| 193 | A time ABI conformance check failed | +| 194 | The violation watcher detected a runtime clock violation | +| 195 | The time ABI's part of post-restore repair failed | +| 125 | Sandbox setup, such as share placement, failed before the workload started | +| 1 | Another snapshot or restore step of the guest agents failed, such as processor activation or the identity refresh | + +A status alone does not identify the cause. In a one-shot lifecycle, the +guest's exit status is otherwise the workload's, which can take any 8-bit +value, and status 1 is also the guest's generic failure status. A host +therefore identifies a fail-fast condition by its status together with the +violation event or error line that precedes it. Statuses in the control +protocol's exit reports and errors, such as 125 for an exceeded output limit, +travel with their own categories instead, as +[Host-Guest Control Interface](4.4-host-guest-control-interface.md#441-operations) +describes. + +## 5.6.2 Failure Classes + +Failures are classified by whether execution can safely continue: + +- Invalid configuration, artifacts, or attachments fail before guest entry. +- Time-contract violations reject a cold boot, capture, or restore with a + stable error code and never fall back. +- A rejected capture preflight may return the still-consistent source to + execution. +- A capture failure after virtual-processor stopping begins is terminal unless + rollback is proven complete. +- A failure after snapshot publication never resumes the source. +- A post-restore repair failure before the acknowledgement, or a missing + acknowledgement, leaves input gated and terminates the restored process. +- Malformed saved state or failed attachment reconstruction prevents any + restored virtual processor from running. + +This fail-closed rule prevents success-shaped recovery from uncertain state. +The architecture prefers terminating one process generation over resuming a +guest whose device ownership, external input, persisted artifacts, or +execution boundary can no longer be established. diff --git a/doc/design-doc/README.md b/doc/design-doc/README.md new file mode 100644 index 000000000..6420c51ab --- /dev/null +++ b/doc/design-doc/README.md @@ -0,0 +1,20 @@ +# NVX Architecture + +This directory describes the architecture of the NVX microVM and its +single-workload guest runtime. It organizes the design by architectural +concept rather than by implementation component. + +Implementation status, current limitations, validation procedures, test +coverage, and delivery plans are intentionally outside this architecture +guide. The existing material under [`../design`](../design/) remains in place +and is not replaced by this directory. + +## Contents + +| Section | Description | +| --- | --- | +| [1. Architecture Overview](1-architecture-overview.md) | Introduces NVX's purpose, major architectural elements, trust boundaries, and compatibility principles. | +| [2. Virtual Machine Architecture](2-virtual-machine-architecture.md) | Defines the backend-independent virtual platform presented to the guest. | +| [3. Virtual Devices](3-virtual-devices.md) | Specifies the guest-visible behavior of each device class. | +| [4. Guest Runtime Architecture](4-guest-runtime-architecture.md) | Describes how the guest turns the virtual machine into an isolated workload sandbox. | +| [5. VM Lifecycle and State](5-vm-lifecycle-and-state.md) | Defines state transitions from creation through execution, capture, restoration, and termination. | From 12411e8c51f24fb7dc1e2cf8a634e3d0572b3a9f Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Wed, 14 Oct 2026 18:16:03 -0300 Subject: [PATCH 2/3] doc: replace doc/design with the architecture guide Make the architecture guide the repository's only design document. Remove the component-oriented design chapters and their index, rename doc/design-doc to doc/design, and retarget every reference to the removed chapters. The removed chapters remain in the history. Removed: - doc/design.md, the index of the old chapters. - doc/design/: goals, configuration boundary, cold boot, machine and device ABI, time ABI, sandbox filesystem and agent architecture, snapshot and restore, snapshot sharing and host storage, host attachment model, concurrency and trust boundaries, validation, current limits, remaining production work, integrated implementation status, and the component responsibility map. Moved and renamed: - doc/design-doc/ becomes doc/design/. Its README no longer says that the old chapters stay in place. - doc/design/copilot-adversarial-testing.md becomes doc/adversarial-testing.md, without its "Design index" breadcrumb. It is the operator document for test-adversarial, with the trust boundary, the executor's containment and credential requirements, artifacts, and CI policy, rather than architecture, so it stays outside the guide. Retargeted documentation: - README.md links the guide's README as the design entry point and the adversarial testing document at its new path. It now also lists the Ubuntu and Azure Linux guest chapters, which only doc/design.md indexed. - doc/guests/: drop the "Design index" breadcrumbs, and link the machine architecture (2), the Linux direct loader (5.1.1), and the time ABI (3.2). The Ubuntu chapter no longer points to the proposed workload-compatibility design, which no longer exists. - doc/build.md, doc/benchmarks.md, doc/ci.md, doc/run.md, and doc/usage.md link the guide's sections: rates (3.2.4), cross-processor consistency (3.2.7), guest obligations (3.2.10), the Linux direct loader (5.1.1), consoles (3.3), network (3.5), shared interrupt status (2.4.4), the layered sandbox root (4.1.1), and CPU profiles (2.2.1). - doc/contribute.md points to doc/ci.md for what the OpenVMM VMM tests and the NVX microVM suite cover, instead of the removed validation chapter. - doc/setup.md, doc/ci.md, and doc/usage.md link doc/adversarial-testing.md. - doc/project-structure.md lists doc/adversarial-testing.md and describes doc/design/ as the architecture guide, and no longer lists doc/design.md. The guide leaves out host qualification, performance budgets, and test schedules, so references to them point to where they now live: - Host qualification checks H1 to H7, and the H4 and H6 schedules: the "Host qualification" section of doc/ci.md, which already describes each check. doc/ci.md and doc/usage.md link the time ABI chapter for the contract and no longer refer to "the spec's" schedules or "spec order". - CPU-time budgets: doc/ci.md says that the budgets come from guest measurements, and CHECK_CPU_BUDGET_US in scripts/nvx_tools/time_abi.py remains the one place that holds them. Retargeted code comments, with no behavior change: - .github/actions/validate-runner/action.yml, scripts/nvx_tools/doctor.py (module docstring, comments, and the doctor's parser description), and scripts/nvx_tools/host_time_probe.rs point to doc/ci.md for host qualification. - guest/common/nvx-snapshot cites the restore protocol (5.5) for the restore packet v4 flags, and names its post-restore work as guest repair steps 4 to 7 of 5.5.7, which were steps 9 to 13 of the removed time ABI chapter. - guest/common/nvx-time.c cites the guide's guest obligations (3.2.10). - scripts/nvx_tools/time_abi.py and scripts/test_guest_time.py cite the guide's clock and time (3.2), CPU profile selection (2.2.1.2), and restore packet (5.5.5) sections. The OpenVMM submodule still cites doc/design/time-abi.md in three comments (vmm_core/virt/src/time_abi/mod.rs, virt_kvm/src/arch/x86_64/time_abi.rs, and virt_whp/src/time_abi.rs). They need a separate OpenVMM change. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .github/actions/validate-runner/action.yml | 2 +- README.md | 11 +- ...rial-testing.md => adversarial-testing.md} | 2 - doc/benchmarks.md | 2 +- doc/build.md | 20 +- doc/ci.md | 19 +- doc/contribute.md | 2 +- doc/design.md | 26 - .../1-architecture-overview.md | 0 doc/{design-doc => design}/1.1-goals.md | 0 doc/{design-doc => design}/1.2-non-goals.md | 0 ...-system-components-and-responsibilities.md | 0 .../1.4-trust-boundaries.md | 0 .../1.5-compatibility-and-versioning-model.md | 0 .../2-virtual-machine-architecture.md | 0 .../2.1-machine-model-and-configuration.md | 0 .../2.2-cpu-and-topology.md | 0 .../2.3-memory-layout.md | 0 .../2.4-interrupt-and-io-architecture.md | 0 ...device-topology-transport-and-discovery.md | 0 .../3-virtual-devices.md | 0 .../3.1-platform-control-device.md | 0 .../3.2-clock-and-time.md | 0 doc/{design-doc => design}/3.3-consoles.md | 0 .../3.4-block-storage.md | 0 doc/{design-doc => design}/3.5-network.md | 0 .../3.6-host-filesystem.md | 0 .../4-guest-runtime-architecture.md | 0 .../4.1-guest-filesystem-and-image-model.md | 0 .../4.2-guest-agents-and-workload-model.md | 0 .../4.3-isolation-and-resource-model.md | 0 .../4.4-host-guest-control-interface.md | 0 .../5-vm-lifecycle-and-state.md | 0 .../5.1-boot-protocol.md | 0 .../5.2-readiness-and-runtime-control.md | 0 .../5.3-snapshot-protocol.md | 0 ...-snapshot-artifact-and-attachment-model.md | 0 .../5.5-restore-protocol.md | 0 .../5.6-shutdown-and-failure-semantics.md | 0 doc/{design-doc => design}/README.md | 3 +- doc/design/code-ownership-map.md | 48 - doc/design/cold-boot.md | 134 - .../concurrency-and-trust-boundaries.md | 35 - doc/design/configuration-boundary.md | 92 - doc/design/current-limits.md | 53 - doc/design/goals.md | 62 - doc/design/host-attachment-model.md | 25 - .../integrated-implementation-status.md | 22 - doc/design/machine-and-device-abi.md | 1217 -------- doc/design/remaining-production-work.md | 67 - ...ndbox-filesystem-and-agent-architecture.md | 495 ---- doc/design/snapshot-and-restore.md | 587 ---- .../snapshot-sharing-and-host-storage.md | 110 - doc/design/time-abi.md | 2464 ----------------- doc/design/validation.md | 184 -- doc/guests/azurelinux-guest.md | 8 +- doc/guests/ubuntu-guest.md | 11 +- doc/project-structure.md | 4 +- doc/run.md | 2 +- doc/setup.md | 2 +- doc/usage.md | 10 +- guest/common/nvx-snapshot | 4 +- guest/common/nvx-time.c | 8 +- scripts/nvx_tools/doctor.py | 11 +- scripts/nvx_tools/host_time_probe.rs | 11 +- scripts/nvx_tools/time_abi.py | 44 +- scripts/test_guest_time.py | 6 +- 67 files changed, 91 insertions(+), 5712 deletions(-) rename doc/{design/copilot-adversarial-testing.md => adversarial-testing.md} (99%) delete mode 100644 doc/design.md rename doc/{design-doc => design}/1-architecture-overview.md (100%) rename doc/{design-doc => design}/1.1-goals.md (100%) rename doc/{design-doc => design}/1.2-non-goals.md (100%) rename doc/{design-doc => design}/1.3-system-components-and-responsibilities.md (100%) rename doc/{design-doc => design}/1.4-trust-boundaries.md (100%) rename doc/{design-doc => design}/1.5-compatibility-and-versioning-model.md (100%) rename doc/{design-doc => design}/2-virtual-machine-architecture.md (100%) rename doc/{design-doc => design}/2.1-machine-model-and-configuration.md (100%) rename doc/{design-doc => design}/2.2-cpu-and-topology.md (100%) rename doc/{design-doc => design}/2.3-memory-layout.md (100%) rename doc/{design-doc => design}/2.4-interrupt-and-io-architecture.md (100%) rename doc/{design-doc => design}/2.5-device-topology-transport-and-discovery.md (100%) rename doc/{design-doc => design}/3-virtual-devices.md (100%) rename doc/{design-doc => design}/3.1-platform-control-device.md (100%) rename doc/{design-doc => design}/3.2-clock-and-time.md (100%) rename doc/{design-doc => design}/3.3-consoles.md (100%) rename doc/{design-doc => design}/3.4-block-storage.md (100%) rename doc/{design-doc => design}/3.5-network.md (100%) rename doc/{design-doc => design}/3.6-host-filesystem.md (100%) rename doc/{design-doc => design}/4-guest-runtime-architecture.md (100%) rename doc/{design-doc => design}/4.1-guest-filesystem-and-image-model.md (100%) rename doc/{design-doc => design}/4.2-guest-agents-and-workload-model.md (100%) rename doc/{design-doc => design}/4.3-isolation-and-resource-model.md (100%) rename doc/{design-doc => design}/4.4-host-guest-control-interface.md (100%) rename doc/{design-doc => design}/5-vm-lifecycle-and-state.md (100%) rename doc/{design-doc => design}/5.1-boot-protocol.md (100%) rename doc/{design-doc => design}/5.2-readiness-and-runtime-control.md (100%) rename doc/{design-doc => design}/5.3-snapshot-protocol.md (100%) rename doc/{design-doc => design}/5.4-snapshot-artifact-and-attachment-model.md (100%) rename doc/{design-doc => design}/5.5-restore-protocol.md (100%) rename doc/{design-doc => design}/5.6-shutdown-and-failure-semantics.md (100%) rename doc/{design-doc => design}/README.md (90%) delete mode 100644 doc/design/code-ownership-map.md delete mode 100644 doc/design/cold-boot.md delete mode 100644 doc/design/concurrency-and-trust-boundaries.md delete mode 100644 doc/design/configuration-boundary.md delete mode 100644 doc/design/current-limits.md delete mode 100644 doc/design/goals.md delete mode 100644 doc/design/host-attachment-model.md delete mode 100644 doc/design/integrated-implementation-status.md delete mode 100644 doc/design/machine-and-device-abi.md delete mode 100644 doc/design/remaining-production-work.md delete mode 100644 doc/design/sandbox-filesystem-and-agent-architecture.md delete mode 100644 doc/design/snapshot-and-restore.md delete mode 100644 doc/design/snapshot-sharing-and-host-storage.md delete mode 100644 doc/design/time-abi.md delete mode 100644 doc/design/validation.md diff --git a/.github/actions/validate-runner/action.yml b/.github/actions/validate-runner/action.yml index 61b643bab..b35157ee7 100644 --- a/.github/actions/validate-runner/action.yml +++ b/.github/actions/validate-runner/action.yml @@ -167,7 +167,7 @@ runs: throw "Hyper-V SVGA firmware was not found: $svgaFirmware" } - # Host time. The time ABI host qualification (doc/design/time-abi.md, + # Host time. The time ABI host qualification (doc/ci.md, # "Host qualification") checks the backend, the CPU fingerprint and # generation, and the TSC rate stability on its short CI schedule, and, # in jobs that downloaded OpenVMM, its CPU profile check (H2) and diff --git a/README.md b/README.md index 13ab71d94..459cfc18a 100644 --- a/README.md +++ b/README.md @@ -71,11 +71,16 @@ python scripts\nvx.py run ### Development -- [Design](doc/design.md) - Current microVM architecture and ABI. +- [Design](doc/design/README.md) - Architecture of the NVX microVM and its single-workload guest + runtime. +- [Ubuntu guest support](doc/guests/ubuntu-guest.md) - How NVX runs Ubuntu userland on its own + kernel. +- [Azure Linux guest support](doc/guests/azurelinux-guest.md) - How NVX runs Azure Linux userland + on its own kernel. - [Project structure](doc/project-structure.md) - Overview of the NVX repository layout. - [Continuous integration](doc/ci.md) - Instructions for running and maintaining NVX CI. -- [Copilot-driven adversarial testing](doc/design/copilot-adversarial-testing.md) - Bounded - adaptive stress and containment campaigns. +- [Copilot-driven adversarial testing](doc/adversarial-testing.md) - Bounded adaptive stress and + containment campaigns. - [Package and source delivery](doc/distribution.md) - Instructions for packaging and distributing NVX. - [Contributing](doc/contribute.md) - Guidelines for contributing to NVX. diff --git a/doc/design/copilot-adversarial-testing.md b/doc/adversarial-testing.md similarity index 99% rename from doc/design/copilot-adversarial-testing.md rename to doc/adversarial-testing.md index 9fd730cdf..077495811 100644 --- a/doc/design/copilot-adversarial-testing.md +++ b/doc/adversarial-testing.md @@ -1,7 +1,5 @@ # Copilot-driven adversarial testing -[Design index](../design.md) - NVX has a separate adversarial harness that uses GitHub Copilot CLI only as an adaptive strategist. Copilot selects deterministic guest-side probes. A typed broker, credential-free executor, independent watchdog, and existing microVM diff --git a/doc/benchmarks.md b/doc/benchmarks.md index 8c1ac6811..a7002fddd 100644 --- a/doc/benchmarks.md +++ b/doc/benchmarks.md @@ -273,7 +273,7 @@ OpenVMM owns `BASE`; each `--cmdline` value below is appended to it. Restore pha command line and resume the one captured in the snapshot. OpenVMM also appends device-discovery tokens (`virtio_mmio.device=...` for an attached mount or NIC) and may replace `console=hvc0` with `console=hvc1` when a virtio console is selected. It appends no clock token: under the -[time ABI](design/time-abi.md#rates) the NVX kernel reads the TSC and LAPIC timer rates from the +[time ABI](design/3.2-clock-and-time.md#324-rates) the NVX kernel reads the TSC and LAPIC timer rates from the hypervisor identity's frequency MSRs and never calibrates either clock, and OpenVMM rejects a supplied `tsc_early_khz=` or `lapic_timer_hz=` token (`E_CMDLINE_CLOCK_TOKEN`). No benchmark command line tunes the clock or clocksource, except the `cold_start_clocksource` variant below. diff --git a/doc/build.md b/doc/build.md index 717913abc..013f37f7d 100644 --- a/doc/build.md +++ b/doc/build.md @@ -171,9 +171,9 @@ the static `nvx-time-probe` guest time probe, and the static `nvx-time` guest time component under `/sbin`. The probe measures TSC and clock read latency, reports the guest's clock, clocksource, TSC flags, and hypervisor identity, and runs the cross-vCPU warp test that the -[time ABI](design/time-abi.md#cross-vcpu-skew-bound) uses to enforce its 1 µs +[time ABI](design/3.2-clock-and-time.md#327-cross-processor-consistency) uses to enforce its 1 µs skew bound. `nvx-time` implements the time ABI's -[guest obligations](design/time-abi.md#guest-obligations): the conformance +[guest obligations](design/3.2-clock-and-time.md#3210-guest-obligations): the conformance checks, the violation watcher, the restore packet v4 and time-sample parsers, the snapshot agent's time steps, and the wall-clock discipline. Before shell-ready it only steps the clock to host UTC; the other boot checks run in @@ -246,12 +246,12 @@ live in `KernelBuildConstants` in | Group | Required settings | Purpose | | --- | --- | --- | -| Direct boot | ACPI and PVH off; MP-table parsing, local APIC, IOAPIC, and virtio-mmio devices from the command line | Boot through the [ACPI-free MP-table loader](design/cold-boot.md#linux-direct-mp-table-loader) | -| Virtio console | Virtio console and HVC driver over virtio-mmio | Drive the [virtio consoles](design/machine-and-device-abi.md#console) | -| Sandbox | virtio-blk; ext4; EROFS with zstd compression; overlayfs that does not always follow redirects; memory and PID cgroups; BPF and cgroup BPF; seccomp filters; Unix-domain sockets; network namespaces and veth | Layer EROFS over ext4 scratch and confine the workload in the [sandbox bootstrap](design/sandbox-filesystem-and-agent-architecture.md#implemented-filesystem-bootstrap) | -| Network | IPv4; IPv6 without the SIT tunnel driver; virtio-net | Configure the dual-stack NIC of the [portable network profile](design/machine-and-device-abi.md#network); SIT would add an `sit0` device that decapsulates IPv6 from IPv4 packets | -| Shared interrupt status | `CONFIG_VIRTIO_MMIO_SHARED_STATUS` from the kernel patches | Read [shared virtio interrupt status](design/machine-and-device-abi.md#fixed-virtio-mmio-transport) | -| Time ABI | Hypervisor-guest and paravirtualization support on; Hyper-V, KVM guest, and cpufreq off | Meet the [time ABI](design/time-abi.md), as described above | +| Direct boot | ACPI and PVH off; MP-table parsing, local APIC, IOAPIC, and virtio-mmio devices from the command line | Boot through the [ACPI-free MP-table loader](design/5.1-boot-protocol.md#511-linux-direct-loader) | +| Virtio console | Virtio console and HVC driver over virtio-mmio | Drive the [virtio consoles](design/3.3-consoles.md) | +| Sandbox | virtio-blk; ext4; EROFS with zstd compression; overlayfs that does not always follow redirects; memory and PID cgroups; BPF and cgroup BPF; seccomp filters; Unix-domain sockets; network namespaces and veth | Layer EROFS over ext4 scratch and confine the workload in the [sandbox bootstrap](design/4.1-guest-filesystem-and-image-model.md#411-layered-sandbox-root) | +| Network | IPv4; IPv6 without the SIT tunnel driver; virtio-net | Configure the dual-stack NIC of the [portable network profile](design/3.5-network.md); SIT would add an `sit0` device that decapsulates IPv6 from IPv4 packets | +| Shared interrupt status | `CONFIG_VIRTIO_MMIO_SHARED_STATUS` from the kernel patches | Read [shared virtio interrupt status](design/2.4-interrupt-and-io-architecture.md#244-shared-interrupt-status) | +| Time ABI | Hypervisor-guest and paravirtualization support on; Hyper-V, KVM guest, and cpufreq off | Meet the [time ABI](design/3.2-clock-and-time.md), as described above | | Hardening | Strict kernel and module RWX with read-only executable memory, the ITS mitigation, an empty modprobe path, trimmed symbol exports, the boot-time W+X audit, and no option built as a module | Keep runtime code read-only, as described above | | Watchdogs | Soft-lockup and hung-task detectors off, or on in the [CI debug kernel](#ci-debug-kernel) | Keep the detectors out of production kernels | @@ -284,8 +284,8 @@ the production RCU stall timeouts and turns off the debug options that detectors are missing from the generated config. CI can shorten the soft-lockup and hung-task thresholds for one boot with `watchdog_thresh=` and `sysctl.kernel.hung_task_timeout_secs=`. The RCU stall timeout stays at the -production value because the time ABI's -[conformance check C8](design/time-abi.md#conformance-checks-and-the-nvx-time-abi-marker) +production value because the time ABI's conformance check C8, which covers +the [stall-detector settings](design/3.2-clock-and-time.md#3210-guest-obligations), requires it. The variant builds in `build/linux-debug`, so it never invalidates the diff --git a/doc/ci.md b/doc/ci.md index 3318730ff..58b24ba6d 100644 --- a/doc/ci.md +++ b/doc/ci.md @@ -35,7 +35,7 @@ one-vCPU smoke set, under the same time ABI checks; the debug-kernel jobs skip it, as they skip the Ubuntu tests. Failure logs from the NVX layer are uploaded per backend. Every harness launch, in the tests and the benchmarks, scans the OpenVMM -console for the guest's [time ABI](design/time-abi.md) output, and the tests +console for the guest's [time ABI](design/3.2-clock-and-time.md) output, and the tests scan every virtio console they read over TCP the same way, to its end. An `NVX-TIME-ABI-VIOLATION` event or a failed `NVX-TIME-ABI` conformance line fails the scenario at once with the guest's code and detail. Guests keep the @@ -69,7 +69,7 @@ online CPUs after each boot or restore, in two rounds with all vCPUs halted for 1 s between them, so that a host without an invariant TSC corrects the guest TSC as idle host CPUs wake (#265). Every pair in every round must stay within the time ABI's 1 µs -[cross-vCPU skew bound](design/time-abi.md#cross-vcpu-skew-bound) for both the +[cross-vCPU skew bound](design/3.2-clock-and-time.md#327-cross-processor-consistency) for both the backward TSC step and the ping-pong offset; a stalled pair or an inconclusive measurement also fails. `smp` boots each requested processor count. `smp-snapshot` captures one snapshot per requested count and restores @@ -148,11 +148,10 @@ boot, restore after a restore, and capture after `snapshot-core`'s released request), the exhaustive check's summary, and the `restore-downtime` stall counts. Where the guest reports a check's CPU time (`cpu_us`), the line also gives its range and how many checks exceed the -backend's [CPU-time budget](design/time-abi.md#performance-expectations-and-acceptance-gate) -for their phase, which the spec sets from guest measurements. The spec exempts -the first capture after a rolled-back capture in the same VM process; CI never -makes one, because each VM process sends OpenVMM at most one snapshot request -and a failed capture fails its scenario. +backend's CPU-time budget for their phase. The budgets come from guest +measurements and exempt the first capture after a rolled-back capture in the +same VM process; CI never makes one, because each VM process sends OpenVMM at +most one snapshot request and a failed capture fails its scenario. `CHECK_CPU_BUDGET_US` in `scripts/nvx_tools/time_abi.py` holds the budgets, one per backend and phase. `elapsed_us` is wall time, including waits behind the workload, and has no budget. CI gates on neither: the performance gate is the @@ -466,12 +465,12 @@ cache keys, and fail when a pattern misses one. ## Host qualification `python3 scripts/nvx.py doctor --backend ` qualifies a host for -the [time ABI](design/time-abi.md#host-qualification). It runs checks H1 to H7 +the [time ABI](design/3.2-clock-and-time.md). It runs checks H1 to H7 in order, prints one `NVX-DOCTOR: check= status= detail="..."` line per check, and exits with status 1 if any check fails. `--checks` selects a subset, and `--summary` appends a Markdown table with the CPU generation, profile, rates, and skew metrics to a file such as `$GITHUB_STEP_SUMMARY`. H4 -and H6 run on the spec's long qualification schedules, which take about three +and H6 run on long qualification schedules, which take about three minutes; `--ci-schedule` selects CI's short ones. A failure that matches a time ABI failure code starts its detail with the code in brackets, for example `[E_PROFILE_HOST_UNKNOWN]`. A check fails, and never passes, when it can't @@ -611,7 +610,7 @@ Normal Actions artifacts contain only the guest-text-free public summary, catalogued case identifiers, and replay manifest. The external provisioner must collect controller transcripts and complete target logs into access-controlled security storage. See -[Copilot-driven adversarial testing](design/copilot-adversarial-testing.md) +[Copilot-driven adversarial testing](adversarial-testing.md) for the architecture and operational contract. ## Copilot environments diff --git a/doc/contribute.md b/doc/contribute.md index 98f63efc1..10245d41c 100644 --- a/doc/contribute.md +++ b/doc/contribute.md @@ -108,7 +108,7 @@ mechanics, such as the golden fixtures of the control protocol, belong in `nvx-rs`, and tests of the ACI Edge Sandboxes contract in `aci_edge_sandboxes`. The `scripts/nvx.py test-aci-edge-sandboxes --backend BACKEND` command runs all ignored `openvmm_e2e` tests on a real hypervisor, -which exercises both crates. [Validation](design/validation.md) +which exercises both crates. [Continuous integration](ci.md) describes what the OpenVMM VMM tests and the NVX microVM suite cover. ## OpenVMM diff --git a/doc/design.md b/doc/design.md deleted file mode 100644 index ebe9fd2c7..000000000 --- a/doc/design.md +++ /dev/null @@ -1,26 +0,0 @@ -# NVX microVM design - -These documents describe the microVM machine profile implemented by the OpenVMM -submodule and the single-container sandbox filesystem and agent architecture -built on it. The implementation is the source of truth. Sections explicitly -marked **Proposed** retain production architecture and rationale that are not -yet implemented; they are not part of the current machine contract. - -1. [Goals](design/goals.md) -2. [Configuration boundary](design/configuration-boundary.md) -3. [Cold boot](design/cold-boot.md) -4. [Machine and device ABI](design/machine-and-device-abi.md) -5. [Time ABI](design/time-abi.md) -6. [Sandbox filesystem and agent architecture](design/sandbox-filesystem-and-agent-architecture.md) -7. [Ubuntu guest support](guests/ubuntu-guest.md) -8. [Azure Linux guest support](guests/azurelinux-guest.md) -9. [Snapshot and restore](design/snapshot-and-restore.md) -10. [Snapshot sharing and host storage](design/snapshot-sharing-and-host-storage.md) -11. [Host attachment model](design/host-attachment-model.md) -12. [Concurrency and trust boundaries](design/concurrency-and-trust-boundaries.md) -13. [Validation](design/validation.md) -14. [Current limits](design/current-limits.md) -15. [Remaining production work](design/remaining-production-work.md) -16. [Integrated implementation status](design/integrated-implementation-status.md) -17. [Component responsibility map](design/code-ownership-map.md) -18. [Copilot-driven adversarial testing](design/copilot-adversarial-testing.md) diff --git a/doc/design-doc/1-architecture-overview.md b/doc/design/1-architecture-overview.md similarity index 100% rename from doc/design-doc/1-architecture-overview.md rename to doc/design/1-architecture-overview.md diff --git a/doc/design-doc/1.1-goals.md b/doc/design/1.1-goals.md similarity index 100% rename from doc/design-doc/1.1-goals.md rename to doc/design/1.1-goals.md diff --git a/doc/design-doc/1.2-non-goals.md b/doc/design/1.2-non-goals.md similarity index 100% rename from doc/design-doc/1.2-non-goals.md rename to doc/design/1.2-non-goals.md diff --git a/doc/design-doc/1.3-system-components-and-responsibilities.md b/doc/design/1.3-system-components-and-responsibilities.md similarity index 100% rename from doc/design-doc/1.3-system-components-and-responsibilities.md rename to doc/design/1.3-system-components-and-responsibilities.md diff --git a/doc/design-doc/1.4-trust-boundaries.md b/doc/design/1.4-trust-boundaries.md similarity index 100% rename from doc/design-doc/1.4-trust-boundaries.md rename to doc/design/1.4-trust-boundaries.md diff --git a/doc/design-doc/1.5-compatibility-and-versioning-model.md b/doc/design/1.5-compatibility-and-versioning-model.md similarity index 100% rename from doc/design-doc/1.5-compatibility-and-versioning-model.md rename to doc/design/1.5-compatibility-and-versioning-model.md diff --git a/doc/design-doc/2-virtual-machine-architecture.md b/doc/design/2-virtual-machine-architecture.md similarity index 100% rename from doc/design-doc/2-virtual-machine-architecture.md rename to doc/design/2-virtual-machine-architecture.md diff --git a/doc/design-doc/2.1-machine-model-and-configuration.md b/doc/design/2.1-machine-model-and-configuration.md similarity index 100% rename from doc/design-doc/2.1-machine-model-and-configuration.md rename to doc/design/2.1-machine-model-and-configuration.md diff --git a/doc/design-doc/2.2-cpu-and-topology.md b/doc/design/2.2-cpu-and-topology.md similarity index 100% rename from doc/design-doc/2.2-cpu-and-topology.md rename to doc/design/2.2-cpu-and-topology.md diff --git a/doc/design-doc/2.3-memory-layout.md b/doc/design/2.3-memory-layout.md similarity index 100% rename from doc/design-doc/2.3-memory-layout.md rename to doc/design/2.3-memory-layout.md diff --git a/doc/design-doc/2.4-interrupt-and-io-architecture.md b/doc/design/2.4-interrupt-and-io-architecture.md similarity index 100% rename from doc/design-doc/2.4-interrupt-and-io-architecture.md rename to doc/design/2.4-interrupt-and-io-architecture.md diff --git a/doc/design-doc/2.5-device-topology-transport-and-discovery.md b/doc/design/2.5-device-topology-transport-and-discovery.md similarity index 100% rename from doc/design-doc/2.5-device-topology-transport-and-discovery.md rename to doc/design/2.5-device-topology-transport-and-discovery.md diff --git a/doc/design-doc/3-virtual-devices.md b/doc/design/3-virtual-devices.md similarity index 100% rename from doc/design-doc/3-virtual-devices.md rename to doc/design/3-virtual-devices.md diff --git a/doc/design-doc/3.1-platform-control-device.md b/doc/design/3.1-platform-control-device.md similarity index 100% rename from doc/design-doc/3.1-platform-control-device.md rename to doc/design/3.1-platform-control-device.md diff --git a/doc/design-doc/3.2-clock-and-time.md b/doc/design/3.2-clock-and-time.md similarity index 100% rename from doc/design-doc/3.2-clock-and-time.md rename to doc/design/3.2-clock-and-time.md diff --git a/doc/design-doc/3.3-consoles.md b/doc/design/3.3-consoles.md similarity index 100% rename from doc/design-doc/3.3-consoles.md rename to doc/design/3.3-consoles.md diff --git a/doc/design-doc/3.4-block-storage.md b/doc/design/3.4-block-storage.md similarity index 100% rename from doc/design-doc/3.4-block-storage.md rename to doc/design/3.4-block-storage.md diff --git a/doc/design-doc/3.5-network.md b/doc/design/3.5-network.md similarity index 100% rename from doc/design-doc/3.5-network.md rename to doc/design/3.5-network.md diff --git a/doc/design-doc/3.6-host-filesystem.md b/doc/design/3.6-host-filesystem.md similarity index 100% rename from doc/design-doc/3.6-host-filesystem.md rename to doc/design/3.6-host-filesystem.md diff --git a/doc/design-doc/4-guest-runtime-architecture.md b/doc/design/4-guest-runtime-architecture.md similarity index 100% rename from doc/design-doc/4-guest-runtime-architecture.md rename to doc/design/4-guest-runtime-architecture.md diff --git a/doc/design-doc/4.1-guest-filesystem-and-image-model.md b/doc/design/4.1-guest-filesystem-and-image-model.md similarity index 100% rename from doc/design-doc/4.1-guest-filesystem-and-image-model.md rename to doc/design/4.1-guest-filesystem-and-image-model.md diff --git a/doc/design-doc/4.2-guest-agents-and-workload-model.md b/doc/design/4.2-guest-agents-and-workload-model.md similarity index 100% rename from doc/design-doc/4.2-guest-agents-and-workload-model.md rename to doc/design/4.2-guest-agents-and-workload-model.md diff --git a/doc/design-doc/4.3-isolation-and-resource-model.md b/doc/design/4.3-isolation-and-resource-model.md similarity index 100% rename from doc/design-doc/4.3-isolation-and-resource-model.md rename to doc/design/4.3-isolation-and-resource-model.md diff --git a/doc/design-doc/4.4-host-guest-control-interface.md b/doc/design/4.4-host-guest-control-interface.md similarity index 100% rename from doc/design-doc/4.4-host-guest-control-interface.md rename to doc/design/4.4-host-guest-control-interface.md diff --git a/doc/design-doc/5-vm-lifecycle-and-state.md b/doc/design/5-vm-lifecycle-and-state.md similarity index 100% rename from doc/design-doc/5-vm-lifecycle-and-state.md rename to doc/design/5-vm-lifecycle-and-state.md diff --git a/doc/design-doc/5.1-boot-protocol.md b/doc/design/5.1-boot-protocol.md similarity index 100% rename from doc/design-doc/5.1-boot-protocol.md rename to doc/design/5.1-boot-protocol.md diff --git a/doc/design-doc/5.2-readiness-and-runtime-control.md b/doc/design/5.2-readiness-and-runtime-control.md similarity index 100% rename from doc/design-doc/5.2-readiness-and-runtime-control.md rename to doc/design/5.2-readiness-and-runtime-control.md diff --git a/doc/design-doc/5.3-snapshot-protocol.md b/doc/design/5.3-snapshot-protocol.md similarity index 100% rename from doc/design-doc/5.3-snapshot-protocol.md rename to doc/design/5.3-snapshot-protocol.md diff --git a/doc/design-doc/5.4-snapshot-artifact-and-attachment-model.md b/doc/design/5.4-snapshot-artifact-and-attachment-model.md similarity index 100% rename from doc/design-doc/5.4-snapshot-artifact-and-attachment-model.md rename to doc/design/5.4-snapshot-artifact-and-attachment-model.md diff --git a/doc/design-doc/5.5-restore-protocol.md b/doc/design/5.5-restore-protocol.md similarity index 100% rename from doc/design-doc/5.5-restore-protocol.md rename to doc/design/5.5-restore-protocol.md diff --git a/doc/design-doc/5.6-shutdown-and-failure-semantics.md b/doc/design/5.6-shutdown-and-failure-semantics.md similarity index 100% rename from doc/design-doc/5.6-shutdown-and-failure-semantics.md rename to doc/design/5.6-shutdown-and-failure-semantics.md diff --git a/doc/design-doc/README.md b/doc/design/README.md similarity index 90% rename from doc/design-doc/README.md rename to doc/design/README.md index 6420c51ab..efd42052f 100644 --- a/doc/design-doc/README.md +++ b/doc/design/README.md @@ -6,8 +6,7 @@ concept rather than by implementation component. Implementation status, current limitations, validation procedures, test coverage, and delivery plans are intentionally outside this architecture -guide. The existing material under [`../design`](../design/) remains in place -and is not replaced by this directory. +guide. ## Contents diff --git a/doc/design/code-ownership-map.md b/doc/design/code-ownership-map.md deleted file mode 100644 index 8ca8c389b..000000000 --- a/doc/design/code-ownership-map.md +++ /dev/null @@ -1,48 +0,0 @@ -# Component responsibility map - -[Design index](../design.md) - -This map assigns each design area to the architectural component responsible -for it. It is a navigation aid, not an implementation index: the components -own their internal structure, and the other design chapters define the -contracts between them. - -| Area | Responsible component | -| --- | --- | -| Machine-profile selection, ABI constants, machine validation, and command-line ownership | OpenVMM machine-profile definitions shared by every launcher and the VM worker | -| CLI options, host-attachment construction, restore preparation, capture orchestration, portb output drain, and outcome reports | OpenVMM command-line entry layer and its VM controller | -| Management-RPC microVM creation, capture, restore, readiness, and guest-exit reporting | OpenVMM management-RPC service | -| Worker composition, fixed virtio placement, snapshot boundaries, management exclusion, clock contracts, and restore sequencing | OpenVMM VM worker | -| Restore-time VP materialization and saved-VP inventory validation | OpenVMM partition unit | -| Restore-time RAM capacity, range selection, split backing, and private copy-on-write mapping | Snapshot machine contract, OpenVMM memory-layout engine, and guest-memory manager | -| Linux direct MP-table loading | OpenVMM Linux direct loader and MP-table builder | -| Base-chipset allowlist and memory-layout defaults | OpenVMM base-chipset manifest builder | -| portb, shutdown, and snapshot PMIO; RTC, PIT, and IOAPIC save and restore | OpenVMM chipset devices | -| Virtio transport, shared interrupt status, and device-private saved state | OpenVMM virtio transport and device models | -| Control-session protocol, authenticated broker, and local peer identity | OpenVMM virtio-console broker and serial socket and named-pipe backends | -| Portable networking, egress policy, and endpoint quiesce | OpenVMM Consomme endpoint, egress policy, and virtio-net device | -| HostFs profile on the single virtio-fs slot: one host directory, or named directory or regular-file children under a read-only synthetic root; per-child access modes, directory-only denied, allowed, and writable paths, file exports that exclude their parent directories, and filesystem saved state | [OpenVMM microVM virtio-fs profile](../../openvmm/vm/devices/virtio/virtiofs/src/microvm/) and [virtio-fs aggregate](../../openvmm/vm/devices/virtio/virtiofs/src/aggregate.rs) | -| State-unit quiesce, start, rollback, inventory, and downtime advance | OpenVMM state-unit framework | -| Snapshot format, machine contract, publication, and artifact validation | OpenVMM snapshot helpers and platform file primitives | -| Backend CPU contracts and snapshot clocks | KVM, MSHV, and WHP backends | -| Guest-visible CPU profiles: the pinned catalog, `--cpu-profile` selection, host profiles and the `auto` fallback, backend support and restore checks, and CPU fingerprints | OpenVMM CPU-profile catalog, with the shared time-ABI layer that builds each VM's effective CPUID | -| Layered sandbox launch and kernel features | NVX sandbox launcher and microVM kernel configuration | -| Managed `sandbox` lifecycle of the NVX CLI: per-sandbox state, fail-closed provision, start, stop, and deprovision transitions that take effect one at a time per sandbox while `exec` runs beside them, the control capability that each start generates and hands to OpenVMM, and identification of the running OpenVMM process | NVX managed sandbox lifecycle and its control-session client | -| Host shares of `run` and `sandbox`: one directory attached directly, or an aggregate for multiple shares or a single regular file; pre-launch checks and the `nvx_share=` tokens that place each child in the guest | [NVX launcher](../../scripts/nvx_tools/sandbox.py) | -| Structured egress-policy files and their translation to OpenVMM egress rules | NVX launcher's egress-policy compiler | -| Edge sandbox lifecycle API, request validation, backend capabilities, and the serializable contract model | NVX `aci_edge_sandboxes` crate and its data-model crate | -| Host side of the control protocol, artifact discovery, blockless managed launch, VM-instance supervision (OpenVMM process reconciliation, interrupted-launch recovery, readiness, the exec pump, and stop), host-path export as aggregate children and their access policy, delivery of the host-mapping table, egress-rule translation, and the fake OpenVMM | [`nvx-rs`](../../nvx-rs/README.md) crate, library `nvx` | -| Durable sandbox records, conversion of contract requests into `nvx` calls, and the mapping of `nvx` errors onto contract error codes | `aci_edge_sandboxes` default OpenVMM backend | -| Image-backed edge sandboxes | `aci_edge_sandboxes` optional agent backend, which delegates to a separately supplied native library | -| Workload namespace and root construction | NVX guest container launch and entry helpers | -| Live share mounts and the bind of each aggregate child at its target | NVX guest init, host-mount helper, and init agent | -| Guest workload, scratch quiesce, managed lifecycle, post-restore CPU/RAM repair, and restore entropy | NVX guest init agent, snapshot helper, and reseed helper | -| Guest time-ABI obligations: conformance checks, violation watcher, wall-clock discipline, and snapshot time steps | NVX guest time component | -| Blockless managed workloads, host-mapping table checks and binds, and workload account creation | NVX guest init and managed agent | -| Host qualification for the time ABI | NVX host doctor, a subset of which CI's runner validation runs | -| OpenVMM control-plane and guest-artifact integration tests | OpenVMM microVM and management-RPC VMM tests | -| NVX Linux and device integration tests | NVX microVM process tests and guest test scripts | -| Edge sandbox contract, fake-OpenVMM, and real-hypervisor lifecycle tests | `aci_edge_sandboxes` crate tests | -| Control-protocol golden fixtures, NVX mechanics unit tests, and fake-OpenVMM instance tests | `nvx-rs` crate tests | -| Copilot adversarial controller and typed broker | NVX adversarial controller and broker | -| Credential-free adversarial executor and independent oracles | NVX adversarial executor and oracle watchdog | diff --git a/doc/design/cold-boot.md b/doc/design/cold-boot.md deleted file mode 100644 index 54616d44f..000000000 --- a/doc/design/cold-boot.md +++ /dev/null @@ -1,134 +0,0 @@ -# Cold boot - -[Design index](../design.md) - -## Linux direct MP-table loader - -The dedicated Linux direct MP-table loader and its shared MP-table builder -treat the kernel, initramfs, command line, topology, reservations, and guest -addresses as untrusted. The loader: - -1. requires a little-endian uncompressed ELF64 image for `EM_X86_64` and - rejects a bzImage; -2. loads validated `PT_LOAD` segments at their physical addresses and zeros - BSS tails; -3. places an optional initramfs at the first 2-MiB boundary after the kernel; -4. builds Intel MP 1.4 processor, ISA bus, IOAPIC, and legacy IRQ entries; -5. builds Linux `boot_params` with the canonical e820 RAM and reservation map; -6. imports a bootstrap GDT and 4-GiB identity page table; and -7. enters the ELF kernel in long mode with paging enabled and `RSI` pointing - to `boot_params`. - -No Xen notes, Xen start-info structures, ACPI tables, RSDP, SMBIOS data, -device tree, or firmware execution environment are parsed or imported. Fixed -virtio devices remain discoverable only through profile-owned command-line -tokens. - -The fixed boot reservations are: - -| Guest physical address | Contents | -| ---: | --- | -| `0x0000..0x000f` | Intel MP 1.4 floating pointer | -| `0x0400...` | MP configuration table (`180 + 20 * vCPU count` bytes) | -| `0x1000` | Linux-direct bootstrap GDT | -| `0x2000` | Linux `boot_params` zero page | -| `0x4000..0x9fff` | Linux-direct 4-GiB identity page tables: one PML4, one PDPT, and four 2-MiB-page directories | -| `0x20000..0x2ffff` | NUL-terminated kernel command line | -| `0x30000..0x30fff` | Shared virtio interrupt-status page | -| `0x100000` and above | Kernel load segments and ordinary RAM | - -The initial state has `RIP` set to the ELF entry, `RSI=0x2000`, `CR3=0x4000`, -`CR0.PE=CR0.PG=1`, `CR4.PAE=1`, and long mode enabled in `EFER`. The zero page -leaves `acpi_rsdp_addr` zero. Its e820 map reserves the live status page and -ISA hole, splits low and high RAM around the fixed 3-to-4-GiB MMIO aperture, -and never reports that aperture as RAM. - -## Memory layout - -Guest RAM is continuous in file offset but split in guest physical address: - -```mermaid -%%{init: {"theme": "base", "themeVariables": {"background": "#ffffff"}}}%% -flowchart LR - Low["Low RAM
0x00000000 through 0xbfffffff
up to 3 GiB"] - Gap["Fixed MMIO gap
0xc0000000 through 0xffffffff
1 GiB"] - High["High RAM
0x100000000 and above"] - Slots["Reserved virtio-mmio slots
0xd0000000 through 0xd0007fff"] - - Low --- Gap - Gap --- High - Gap -. contains .-> Slots -``` - -Active RAM occupies `[0, min(size, 3 GiB))`. Memory displaced by the fixed -one-GiB MMIO aperture resumes at 4 GiB. There is no high-MMIO or VTL2 aperture. -The central OpenVMM memory-layout engine owns this split, using the base -chipset's fixed MMIO reservation; the profile does not maintain a second -allocator. The resulting active RAM ranges are also the authoritative Linux -e820 and snapshot memory-range inventory. - -When capture uses `--memory-capacity`, the layout engine reserves addresses for -the full capacity before publishing only the active base-size prefix as RAM. -Consequently, selecting a larger restore target does not move the MMIO gap or -any device. The machine contract records the canonical suffix ranges that are -absent from the captured e820 RAM map and `memory.bin`; a suffix that crosses -the three-GiB boundary is represented as separate low- and high-RAM ranges. - -Loader writes and DMA ranges must fit wholly inside a RAM range and may not -cross the MMIO gap or a reserved boot structure. - -## Effective command line - -The profile, not the caller, owns console, device-discovery, and host policy -tokens. The base command line is: - -```text -earlycon=xe9 console=hvc0 reboot=t panic=-1 -``` - -When virtio-console is present, `console=hvc0` becomes `console=hvc1`; the raw -portb console remains the early console. User arguments follow the base -tokens. A fresh boot then appends host-owned tokens in this order: - -1. `nr_cpus=`, so Linux sizes processor state to the validated 1, - 2, 4, or 8-vCPU topology; -2. the optional fixed workload identity (`nvx_workload_uid=` and - `nvx_workload_gid=`) and workload lifecycle (`nvx_lifecycle=`); -3. `nvx_snapshot_tier=` for a capture with sandbox blocks; -4. device-discovery tokens in fixed address order: network, the filesystem - slot, boot console, sandbox blocks, and the control console - followed by `nvx_control_tty=hvc2`; and -5. network bootstrap tokens, including gateway DNS only when the egress - policy permits it, which names the IPv4 gateway or else the IPv6 one, and - `virtnet_ingress=allow` last when ingress is allowed, or - `virtnet_ingress=host-loopback` when only explicit host-loopback access - is, followed by - [filesystem bootstrap tokens](machine-and-device-abi.md#filesystem) for - the attached HostFs export or aggregate. - -The command line carries no clock parameter: the guest reads its TSC and -LAPIC rates from the [time ABI](time-abi.md#rates) MSRs, and a -`tsc_early_khz=` or `lapic_timer_hz=` token, from the caller or a snapshot, -fails with `E_CMDLINE_CLOCK_TOKEN`. - -Callers may not supply `earlycon=`, `console=`, `virtio_mmio.device=`, -`virtnet_ip=`, `virtnet_mask=`, `virtnet_gw=`, `virtnet_dns=`, `virtnet_ip6=`, -`virtnet_gw6=`, `virtnet_ingress=`, `virtfs_dir=`, `virtfs_tag=`, `virtfs_mode=`, -`virtfs_aggregate=`, `nvx_snapshot_tier=`, `nr_cpus=`, `nvx_workload_uid=`, -`nvx_workload_gid=`, or `nvx_lifecycle=` tokens. A machine with a control -console also applies the -[control-console command-line rules](machine-and-device-abi.md#control-console). -Embedded NULs are rejected, and the complete NUL-terminated command line must -fit in 64 KiB. The same effective string and its SHA-256 digest become part of -the snapshot machine contract; a restore reuses that string verbatim and -appends no tokens. - -The `nvx_overlay_upper` caller token changes the accepted block layout. -Passing `nvx_overlay_upper=ramfs` exactly once declares that the guest keeps -its overlay upper in RAM. The microVM then requires exactly one read-only -`distro` [sandbox block](machine-and-device-abi.md#fixed-virtio-mmio-transport) -and no other block, so the guest has no writable scratch. This layout supports -neither snapshot capture nor restore. Any other `nvx_overlay_upper` token, -including a bare or repeated one, is rejected. The NVX init agent does not use -this layout; it -[keeps its overlay upper on scratch](sandbox-filesystem-and-agent-architecture.md#implemented-filesystem-bootstrap). diff --git a/doc/design/concurrency-and-trust-boundaries.md b/doc/design/concurrency-and-trust-boundaries.md deleted file mode 100644 index 4bef0dc5a..000000000 --- a/doc/design/concurrency-and-trust-boundaries.md +++ /dev/null @@ -1,35 +0,0 @@ -# Concurrency and trust boundaries - -[Design index](../design.md) - -The guest controls PMIO accesses, virtio descriptors, packet data, FUSE -requests, control-session records, and the timing of a snapshot request. -Kernel, initramfs, command line, snapshot artifacts, restore attachments, and -host control-console clients are also untrusted inputs. - -The implementation therefore uses checked address arithmetic, bounded buffers -and tables, typed validation errors, rate-limited guest-triggerable logs, and -fallible quiesce/restore operations. Native I/O callbacks only record bounded -state or enqueue notifications; they do not perform blocking snapshot or host -resource work. Malformed saved device state is validated before workers start, -and no vCPU runs after a partial restore failure. A failure after vCPU -stopping begins at a snapshot or post-restore acknowledgment boundary is -terminal: host input stays gated and the VM is torn down instead of resuming -with uncertain state. - -A host control client reaches the guest only through the control-console -broker. The broker admits one local peer whose operating-system identity is -the OpenVMM user, requires a launcher-provided capability before any host data -reaches the guest, and bounds every record, queue, and credit window in both -directions; malformed, stale, or out-of-order records fail closed. The -capability never appears in arguments, environment variables, logs, snapshots, -or attachment identities. - -Host-facing outputs are bounded as well. The optional outcome report is written -after teardown to a previously absent local path and records only a schema -version, an opaque instance ID, the backend category, the operation and -outcome category with a numeric status, whether the network policy was -requested and applied with its mode and rule counts, and per-step teardown -flags. It never contains commands, environment values, paths, network -destinations, proxy details, workload output, credentials, or free-form error -text, and OpenVMM does not upload it. diff --git a/doc/design/configuration-boundary.md b/doc/design/configuration-boundary.md deleted file mode 100644 index df43f58ec..000000000 --- a/doc/design/configuration-boundary.md +++ /dev/null @@ -1,92 +0,0 @@ -# Configuration boundary - -[Design index](../design.md) - -Machine identity is explicit rather than inferred from a kernel, device, or -hypervisor choice. OpenVMM carries it through the command line, management -RPC, worker, test harness, and snapshot configuration. The management RPC -exposes one microVM profile with numeric value 2; values 1, 3, and 4 are -reserved, and retired value 1 is rejected before host resources are opened. -Validation occurs before host resources are opened and again at the worker -boundary. - -The microVM requires: - -- an x86-64 guest; -- one NUMA node; -- Linux direct boot with Intel MP 1.4 tables; -- KVM, MSHV, or WHP; -- no VTL2, isolation, nested virtualization, or Hyper-V enlightenments; -- the [time ABI](time-abi.md), its only time path, with one CPU profile; and -- the exact [chipset and device inventory](machine-and-device-abi.md). - -It accepts exactly 1, 2, 4, or 8 vCPUs -in one socket and one die, with one core per vCPU, no SMT, xAPIC mode, and -contiguous APIC IDs starting at zero. Its Linux direct layout places the MP -floating pointer at `0x0`, MP configuration table at `0x400`, GDT at `0x1000`, -zero page at `0x2000`, and reserves `0x30000..0x30fff` for interrupt status. - -The guest sees a [CPU profile](time-abi.md#cpu-profiles), the complete CPUID -surface of one CPU generation, never host passthrough. The host selects it at -cold boot with `--cpu-profile`: a pinned profile ID, `auto`, or `host`, a -development profile derived from the host's CPU. `auto` is the default and is -the only choice of the management RPC; it selects the pinned profile that -serves the host's CPU and falls back, with a warning, to the `host` profile on -an Intel or AMD CPU that no pinned profile serves. The snapshot records the -selected profile. - -Snapshot capture may declare an immutable RAM capacity at least as large as the -active base RAM. When it does, both values must be 128-MiB aligned. A snapshot -without a declared capacity cannot select a different RAM size at restore. - -Only role-bearing block devices are supported; ordinary unroled `--virtio-blk` -is rejected. Blocks end with writable scratch unless the caller passes -`nvx_overlay_upper=ramfs` on the kernel command line, which requires exactly -one read-only `distro` block and no scratch, and rules out snapshot capture and -restore; see [Effective command line](cold-boot.md#effective-command-line). -Snapshot capture with blocks requires one to three -read-only lower layers followed by writable scratch, all backed by cached, -regular raw files with nonzero 512-byte-aligned geometry. Blockless -snapshots are also supported and do not use sandbox tier metadata. The -management RPC constructs only blockless microVMs without a NIC or control -console; it can attach portb, the boot console, and one HostFs export. - -Host filesystem sharing uses the fixed `fs:microvm0` slot instead of adding -devices. The [host-filesystem attachment layer](../../openvmm/openvmm/openvmm_entry/src/microvm/filesystem.rs) -accepts either `--mount` for one host directory or `--mount-aggregate` with -`--mount-child` for named directory or regular-file children. Each child has -its own access mode, and the aggregate root is read-only. The -[filesystem contract](machine-and-device-abi.md#filesystem) defines the -option syntax, export restrictions, and snapshot compatibility rules. - -The microVM rejects UEFI, PCAT, IGVM, caller-supplied ACPI, SMBIOS, device -tree, PCI/PCIe, VPCI, VMBus, ISA DMA, IDE, floppy, VMGS, graphics, VGA -firmware, debugger resources, and devices outside the profile. The profile -itself emits the -[fixed Linux direct MP-table metadata](cold-boot.md#linux-direct-mp-table-loader). -This is an allowlist: the implementation builds a microVM directly instead of -constructing a standard PC and removing unwanted devices. - -The host, not the guest or a workload request, fixes the workload policy of a -fresh boot. An optional nonzero numeric UID and GID become the fixed workload -identity, and an optional one-shot or managed lifecycle selects whether the -guest supervisor runs one workload or stays resident for sequential requests -over the control console. Both are profile-owned command-line tokens that -caller arguments cannot supply. A managed lifecycle requires a fixed workload -identity and a live, authenticated control console. - -Restore is stricter than cold boot. Guest-visible configuration is read from -the snapshot's machine contract. Restore-time input may select the same backend -kind recorded by the snapshot, supply required attachments, bind a new directory -export to a [captured dormant filesystem slot](machine-and-device-abi.md#filesystem) only when -the snapshot records that restore-attachment capability, and select processor -and RAM activation targets explicitly allowed by the contract. Those -process-local targets do not change processor capacity, RAM capacity, command -line, device placement, feature masks, or filesystem and network identity. -Restore always uses the snapshot's CPU profile, never falls back, and requires -the host's CPU to belong to that profile's generation; an explicit -`--cpu-profile` must be `auto`, the snapshot's profile ID, or `host` when the -snapshot used a host profile. The captured command line also fixes the network -addressing, workload identity, and lifecycle, so restore rejects -network-address, workload-identity, and lifecycle options, and a supplied -processor count may only repeat the captured capacity. diff --git a/doc/design/current-limits.md b/doc/design/current-limits.md deleted file mode 100644 index 49bcfb1d4..000000000 --- a/doc/design/current-limits.md +++ /dev/null @@ -1,53 +0,0 @@ -# Current limits - -[Design index](../design.md) - -The current ABI family intentionally does not provide: - -- processor counts other than 1/2/4/8, SMT, multiple NUMA nodes, non-x86 guests, or nested virtualization; -- firmware boot, caller-defined ACPI, SMBIOS, PCI, VPCI, VMBus, arbitrary - post-readiness CPU or memory hotplug, or arbitrary devices; -- cross-hypervisor snapshot restore; -- capture-and-continue, live migration, or saving a reduced-prefix MSHV - restore; -- restore-time RAM shrinking, targets beyond the captured capacity, - non-128-MiB targets, or expansion from snapshots without an opt-in capacity; -- ABI value 1 or boot-layout value 1 snapshots; -- snapshot block media other than cached regular raw files; -- copy-on-write paired-scratch restore on non-Linux hosts or filesystems - without reflink support; -- direct-claimed paired-scratch restore on filesystems without exact-file - hard-link support; -- sandbox-block, network, or control-console construction through the - management RPC, or its restore of snapshots that contain them; -- host networking other than the in-process portable endpoint, ingress from - globally reachable addresses, or live host-loopback publications and port - forwards or private-network ingress in snapshots; -- a production control-session RPC protocol beyond the managed lifecycle - subset; -- replaceable sandbox configuration, a production OCI agent, or a public sandbox snapshot workflow; -- serialization of live network flows or native host handles; -- snapshotting of the contents of a live virtio-fs export; or -- compatibility with standalone NVX `MVMSNAP*` or `WHPSNAP*` files. - -The tier contract and post-restore gate are currently low-level OpenVMM and -guest-agent primitives exercised by the native lifecycle suite. The public -`nvx sandbox` command does not yet orchestrate platform builds, warm-shim -handoff, checkpoints, or sandbox restore. That integration depends on the -replaceable configuration region, production agent, and versioned control -protocol tracked by issues #158, #159, and #160. In particular, a platform -snapshot must be produced only by a trusted pre-image-binding workflow: the -current host validation rejects tenant command-line configuration and records -layer identities as unbound, but cannot prove that arbitrary guest code did -not read an attached layer before requesting capture. - -The authenticated control console, its bounded record framing and broker saved -state, the snapshot RPC exclusion, and the managed lifecycle subset are the -transport foundation for that protocol, not completion of it. The shell -bootstrap does not implement a fleet-safe platform build point, warm-runtime -handoff, arbitrary OCI policy, or production agent resource guarantees. - -Changing a guest-visible address, IRQ, command-line token, feature mask, queue -shape, time policy, or device behavior requires a new microVM ABI version. A -backend-specific difference is valid only when it is explicitly part of that -versioned contract, such as the virtio-net IRQ. diff --git a/doc/design/goals.md b/doc/design/goals.md deleted file mode 100644 index 1aa270f3f..000000000 --- a/doc/design/goals.md +++ /dev/null @@ -1,62 +0,0 @@ -# Goals - -[Design index](../design.md) - -NVX provides a small, versioned virtual machine for running an x86-64 Linux -guest without firmware or a PC platform. Its design has four primary goals: - -- boot the same uncompressed Linux-direct kernel with explicitly selected - Alpine, Ubuntu, or Azure Linux userland on Linux and Windows; -- keep the guest-visible machine independent of the selected hypervisor; -- expose only a fixed, allowlisted set of devices; and -- capture a running VM into immutable artifacts that can be restored in a new - process without serializing host handles. - -The implemented runtime profile is the microVM machine profile. It is selected -explicitly, by `--machine microvm` on the OpenVMM command line or by the -microVM machine profile of the management RPC, and is never inferred from a -kernel, device, or hypervisor choice. It uses fixed sandbox layer and scratch -roles, deterministic SMP topology, and shared virtio-mmio interrupt status with -edge-triggered delivery. Snapshot manifests retain microVM ABI value 2 and -boot-layout value 2, optional restore-time RAM expansion uses machine-contract -capability version 1, and the management RPC uses numeric machine-profile -value 2. KVM and MSHV are supported on Linux and WHP is supported on Windows. -Hypervisor-specific code provides partition creation, vCPU execution, -interrupt injection, and host resource integration. The machine profile owns -the boot protocol, memory map, device topology, command line, and snapshot -compatibility contract. - -```mermaid -%%{init: {"theme": "base", "themeVariables": {"background": "#ffffff"}}}%% -flowchart TB - Inputs["NVX CLI
Linux-direct kernel and selected guest initramfs"] - Profile["OpenVMM microVM
boot, memory, devices, and snapshots"] - Kvm["Linux / KVM"] - Mshv["Linux / MSHV"] - Whp["Windows / WHP"] - Contract["Guest-visible Linux MP-table contract
ABI 2 / boot layout 2"] - - Inputs --> Profile - Profile --> Kvm - Profile --> Mshv - Profile --> Whp - Kvm --> Contract - Mshv --> Contract - Whp --> Contract -``` - -The profile is selected independently from the hypervisor, for example: - -```text -openvmm --machine microvm --hypervisor kvm --kernel vmlinux --initrd initramfs.cpio -openvmm --machine microvm --hypervisor mshv --kernel vmlinux --initrd initramfs.cpio -openvmm --machine microvm --hypervisor whp --kernel vmlinux --initrd initramfs.cpio -openvmm --machine microvm --hypervisor kvm --kernel vmlinux --initrd initramfs.cpio \ - --microvm-sandbox-block distro:file:distro.erofs,ro \ - --microvm-sandbox-block scratch:file:scratch.raw -openvmm --machine microvm --processors 8 --hypervisor kvm \ - --kernel vmlinux --initrd initramfs.cpio -``` - -The supported user entry point in this repository is `python scripts/nvx.py`; -see [Run](../run.md) for complete commands and host-specific options. diff --git a/doc/design/host-attachment-model.md b/doc/design/host-attachment-model.md deleted file mode 100644 index 730cd7b06..000000000 --- a/doc/design/host-attachment-model.md +++ /dev/null @@ -1,25 +0,0 @@ -# Host attachment model - -[Design index](../design.md) - -Snapshot state contains guest-visible progress, not process-local resources. -Each external resource has a stable ID and a declarative reconstruction policy. - -| Resource | Saved | Reconstructed or supplied on restore | -| --- | --- | --- | -| portb | Pending RX/TX bytes | Host serial endpoint, fresh process generation ID, and optional restore packet | -| console | Queue progress, staged RX, partial TX, endpoint identity, and reconnect policy | Eligible listener recreated at its captured or an approved fresh same-kind endpoint, required client connection, or supplied inherited terminal | -| Control console | Distinct attachment identity and broker policy, plus bounded broker state without queued records, host output, or receive credit; never the capability or peer identity | An explicitly approved same-user Unix socket or Windows named pipe, which may use a fresh endpoint identity, plus a fresh launcher-provided capability; the broker restarts with a fresh instance and epoch, keeping only its guest-side parser, any partially written guest record, and counters | -| network | Static identity, queue/packet progress, profile, and policy digest | Fresh in-process Consomme endpoint and matching egress policy; live host-loopback publications and port forwards are rejected | -| filesystem | FUSE namespace, handles, cookies, root/object identity, access mode, and denied, allowed, and writable paths, or explicit dormant state | Fresh host-directory attachment with the same canonical path, target, mode, and denied, allowed, and writable paths; a dormant slot may stay unattached or bind a new attachment | -| sandbox block | Queue/device state, fixed roles, access, geometry, SHA-256 or immutable-generation identities, scratch policy, and paired-scratch materialization | Matching read-only layers plus private-copy or reflink COW scratch for clones, direct scratch after an exclusive resume claim, or a new same-geometry scratch file | -| Readiness endpoint | Nothing | Optional single-use Unix socket or named pipe supplied for one restore | - -Attachment resolution happens before vCPU start. A supplied attachment must -reproduce the saved identity except that an eligible listener may replace only -its endpoint identity while preserving its stable ID, attachment and backend -kinds, reconnect policy, required flag, length, and timeout. Client, inherited, -disconnected, network, filesystem, and block attachments retain their stricter -saved contracts. Missing privileges, endpoint binding failures, changed egress -policy, replaced filesystem objects, or a wrong attachment kind fail restore -explicitly. diff --git a/doc/design/integrated-implementation-status.md b/doc/design/integrated-implementation-status.md deleted file mode 100644 index b8c67e9eb..000000000 --- a/doc/design/integrated-implementation-status.md +++ /dev/null @@ -1,22 +0,0 @@ -# Integrated implementation status - -[Design index](../design.md) - -The implementation and remaining proposals separate as follows: - -| Area | Current implementation | -| --- | --- | -| Base machine | ACPI-free Linux direct boot with Intel MP tables, allowlisted chipset/PMIO, fixed-role sandbox blocks, nine reserved virtio-mmio slots, and 1/2/4/8-vCPU SMP on KVM, MSHV, and WHP, selected explicitly from the command line or the management RPC. Persisted ABI and boot layout remain value 2. | -| Snapshot and restore | Guest-requested version-5 publication, same-backend restore, exact saved inventory, private COW RAM, optional fresh RAM expansion, fresh/paired scratch, three sandbox tiers, input gating, single-use resume claims, a single-use restore readiness event, and management exclusion at the capture boundary. The management RPC drives the same capture and restore for blockless machines without a NIC or control console. Older supported manifests remain readable subject to their recorded capabilities. | -| Resource activation | Opt-in CPU-prefix and 128-MiB-aligned RAM targets within immutable capacity. Explicit MSHV targets materialize only that VP prefix and cannot be saved again; KVM, WHP, and untargeted MSHV retain full VP capacity. | -| Console | Boot virtio-console with private RX/TX state and reconnect policies; bounded portb/host-relay drain at process exit, with guest exit status propagated to management-RPC waiters. The separate control console has authenticated Linux and Windows endpoints, bounded framing with receive credits, per-instance epochs, broker saved state, and managed workload RPC. | -| Network | Static dual-stack identity, fixed transport, the portable in-process Consomme endpoint, directional egress `allow`/`deny`, ingress `deny` or private-network ingress that publishes every port on which the guest listens to clients that are not globally reachable, IPv4 and IPv6 L3/L4 egress rules, host-loopback deny or bidirectional host-loopback access that publishes every port on which the guest listens on host loopback alone, explicit port publishing, an exact proxy exception, and quiesced restore are implemented. Stateful replies remain available under both ingress actions; ingress from globally reachable addresses is refused, and snapshots reject private-network ingress and explicit host-loopback allow. Capture drains packet ownership instead of serializing arbitrary pending packets or host flow state. | -| Guest filesystem | Kernel features and shell bootstrap for EROFS over ext4 scratch, overlayfs, namespace/cgroup isolation, capability stripping, supervision, and low-level snapshot hooks. Not a complete OCI agent or public sandbox restore workflow. | -| Host filesystem | Fixed no-DAX HostFs in a slot that command-line and management-RPC cold boots always expose and that stays dormant without an attachment, live attachment revalidation, server-side denied subtrees with alias-identity enforcement, allowed paths that expose subtrees of denied paths again through traverse-only directories, write narrowing to declared writable paths of a read-write export, guest-created symbolic links on read-write exports with exact targets that the host never follows, and optional caller ownership on Linux hosts, which performs each request as its guest caller's UID and GID with root squashed to the export owner and fails closed with `EPERM`. Provider-backed immutable filesystem generations remain outside the current profile. | -| Workload policy and operations | Host-owned non-root workload identity, one-shot or managed lifecycle selection, bounded local outcome reports, and opt-in lifecycle profiling that does not change ordinary runs. | -| Production services | Replaceable configuration, a Rust guest agent, image conversion/distribution, artifact lifecycle, and tenant-aware snapshot instantiation remain proposals. The current static guest agent and authenticated managed RPC implement only the bounded lifecycle subset. | - -The end-to-end tests establish process-boundary behavior for the available -native host backend. They do not make every future extension a portability -guarantee; the versioned code contract and the -[current limits](current-limits.md) remain authoritative. diff --git a/doc/design/machine-and-device-abi.md b/doc/design/machine-and-device-abi.md deleted file mode 100644 index f608bd142..000000000 --- a/doc/design/machine-and-device-abi.md +++ /dev/null @@ -1,1217 +0,0 @@ -# Machine and device ABI - -[Design index](../design.md) - -## Architectural devices - -The microVM base chipset contains exactly this allowlist: - -- generic PIC and IOAPIC; -- the selected hypervisor's LAPIC; -- i8253 PIT on IRQ 0; -- generic programmable CMOS RTC anchored to UTC; -- raw bidirectional portb; -- status-carrying shutdown port; and -- guest snapshot-request port. - -The RTC defaults to BCD, 24-hour fields with status B `0x02`. PIC, IOAPIC, PIT, -RTC, LAPIC, and VM time use common OpenVMM device and -state-unit machinery on every backend. IOAPIC saved state includes the -asserted level of every input line and reevaluates routing after restore, so a -level interrupt is neither lost nor treated as an edge while reconstructing -the backend. Serial UARTs, debugcon, Hyper-V power management, gameport, PCI, -firmware helpers, and standard-PC missing-port shims are absent. - -The guest's clocks follow [NVX time ABI v1](time-abi.md) on every backend: a -minimal Hyper-V identity whose MSRs declare the TSC and LAPIC rates, served by -the `time-abi` state unit; the TSC as the only clocksource; the LAPIC timer in -one-shot mode at the backend's fixed rate (1 GHz on KVM, 200 MHz on MSHV and -WHP); and a PIT that the guest disables at boot. Capture rejects a periodic or -TSC-deadline LAPIC timer and a periodically counting PIT channel 0. - -```mermaid -%%{init: {"theme": "base", "themeVariables": {"background": "#ffffff"}}}%% -flowchart TB - Guest["x86-64 Linux guest
direct boot, 1/2/4/8 vCPUs"] - - subgraph Abi["microVM machine contract"] - direction LR - Boot["Linux MP-table boot state
and fixed RAM layout"] - Interrupts["PIC, IOAPIC, LAPIC
PIT, RTC, and VM time"] - Pmio["PMIO devices
portb, shutdown, snapshot"] - Virtio["Fixed virtio-mmio
net, fs, boot and control consoles,
and versioned block roles"] - end - - Common["OpenVMM worker
state units and resource resolvers"] - - subgraph Backends["Execution backend"] - direction LR - KvmBackend["KVM"] - MshvBackend["MSHV"] - WhpBackend["WHP"] - end - - Guest --> Boot - Guest --> Interrupts - Guest --> Pmio - Guest --> Virtio - Boot --> Common - Interrupts --> Common - Pmio --> Common - Virtio --> Common - Common --> KvmBackend - Common --> MshvBackend - Common --> WhpBackend -``` - -## PMIO devices - -| Port | Device | Behavior | -| ---: | --- | --- | -| `0xe9` | portb data | Raw byte input and output; console reads consume one pending byte and zero-fill the remaining access width. With the restore packet or generation ID selected, a 1-, 2-, or 4-byte read returns that many bytes of the record, zero-filled past its end. | -| `0xea` | portb status | Bit 0 reports pending host input, bit 1 reports a fresh restore packet, bit 2 reports a processor target, bit 3 reports a memory target, bit 4 reports one or more memory-expansion ranges, bit 5 reports the fixed generation-ID selector, and bit 6 reports the time-sample window, which always exists. Writing `0xa5` after restore selects the one-time [restore packet](time-abi.md#restore-packet). Writing `0xa6` selects the current 16-byte generation ID; it may be selected repeatedly and remains stable for the lifetime of one VM process. Writing `0xa7` latches a fresh [time sample](time-abi.md#time-sample) into the window without changing the `0xe9` selection. | -| `0xeb` | portb time window | A 1-, 2-, or 4-byte read returns the next bytes of the latched time sample, zero-filled past its end or when none is latched. The window is not saved state. | -| `0x604` | shutdown | The first output byte becomes the process status carried with the VM power-off request. Reads return all ones. | -| `0x605` | snapshot request | Reads return all ones. Writes are coalesced and routed asynchronously to the capture controller. Zero requests fresh scratch and a nonzero first byte requests paired scratch. | - -The portb receive and transmit buffers are each bounded at one MiB. Output -overflow drops the newest bytes and emits a rate-limited warning; input applies -backpressure by stopping host reads when its buffer is full. Pending bytes are -saved so capture does not silently lose VMM-owned I/O. While host input is -gated for a snapshot boundary or post-restore repair, portb stops reading its -host endpoint, returns zero for console data reads, and clears the -input-available status bit; guest output, the generation ID, any restore -packet, and the time-sample window remain available. - -Guest-requested process exit drains the portb endpoint and, when present, its -host stdout relay before reporting completion. The combined drain has a -five-second deadline. Success preserves the guest's exit status; endpoint, -relay, or timeout failures become process-exit errors instead of silently -discarding final output. This is a portb/host-relay guarantee, not a general -drain of every virtio-console endpoint. Under the management RPC, a zero guest -status completes a pending wait and stops the server cleanly, while a nonzero -status or failed drain fails the pending wait and ends the server with an -error. - -A snapshot-port write with no configured destination completes normally and -the guest continues. With a destination, the device permits at most one pending -transaction and defers completion long enough for the controller to establish -the exact post-`out` capture boundary. Repeated writes are coalesced. The PMIO -callback itself never pauses vCPUs, drains devices, hashes RAM, or writes files. -The scratch policy travels with the deferred boundary request. After a gated -restore, the same write acknowledges completion of guest repair. - -## Fixed virtio-mmio transport - -All eight fixed address slots are reserved, including the dedicated control -console at `0xd0007000..0xd0007fff` on IRQ 3 (shared status at `0x3001c`). -The window at `0xd0008000..0xd0008fff`, IRQ 13, and the shared-status word at -`0x30020` held a second virtio-fs slot in earlier releases; they are unused, and -a snapshot that recorded that slot no longer restores. -Every microVM cold-booted from the command line or the management RPC -instantiates the virtio-fs slot, so it is discoverable before capture even -without a host attachment; a restore keeps the slot only when the snapshot -recorded it. Several shares are children of one aggregate attachment on that -slot. Other optional devices are instantiated only when configured. -The control slot is instantiated for a control console, either a live -authenticated local endpoint on Linux or Windows or an explicitly disconnected -console. Every device uses virtio-mmio, is discovered only through -profile-owned command-line tokens, and has packed-ring support masked. -The MP table routes only the 16 ISA interrupts, and no device uses IRQ 13 on -KVM, MSHV, or WHP. - -| Device | Stable identity | MMIO range | IRQ | Availability | -| --- | --- | ---: | ---: | --- | -| virtio-net | `net:microvm0` | `0xd0000000..0xd0000fff` | KVM/MSHV 10, WHP 5 | Optional | -| virtio-fs | `fs:microvm0` | `0xd0001000..0xd0001fff` | 6 | Reserved dormant slot; HostFs optional | -| virtio-console | `console:microvm-virtio0` | `0xd0002000..0xd0002fff` | 7 | Optional | -| `distro` virtio-blk | `blk:sandbox:distro` | `0xd0003000..0xd0003fff` | 4 | Optional read-only role | -| `runtime` virtio-blk | `blk:sandbox:runtime` | `0xd0004000..0xd0004fff` | 12 | Optional read-only role | -| `custom` virtio-blk | `blk:sandbox:custom` | `0xd0005000..0xd0005fff` | 9 | Optional read-only role | -| `scratch` virtio-blk | `blk:sandbox:scratch` | `0xd0006000..0xd0006fff` | 11 | Writable final role; required with blocks except in the RAM-backed overlay layout | -| Control virtio-console | `console:microvm-control0` | `0xd0007000..0xd0007fff` | 3 | Optional authenticated local endpoint | - -Explicit placement metadata bypasses the standard sequential MMIO allocator. -The worker validates the complete device count, kind, bus, address, IRQ, and -feature policy before resolving devices. - -`--microvm-sandbox-block ROLE:DISK` assigns each attachment by -role rather than option order: - -| Role | MMIO range | IRQ | Access | -| --- | ---: | ---: | --- | -| `distro` | `0xd0003000..0xd0003fff` | 4 | Read-only | -| `runtime` | `0xd0004000..0xd0004fff` | 12 | Read-only | -| `custom` | `0xd0005000..0xd0005fff` | 9 | Read-only | -| `scratch` | `0xd0006000..0xd0006fff` | 11 | Writable | - -Roles must be unique and supplied in fixed order; omitted lower-layer roles -leave their slots empty. A nonempty topology ends with scratch unless the -[RAM-backed overlay layout](cold-boot.md#effective-command-line) selects exactly -one read-only `distro` block. All four block IRQs use active-high edge -delivery. IRQ 12 avoids the RTC's exclusive IRQ 8. - -The microVM keeps every fixed MMIO address and IRQ number and uses active-high edge -delivery for dedicated virtio IRQs. One little-endian, naturally aligned `u32` -per fixed slot resides in the reserved shared-status page: - -| Slot | Status GPA | -| --- | ---: | -| virtio-net | `0x30000` | -| virtio-fs | `0x30004` | -| virtio-console | `0x30008` | -| `distro` block | `0x3000c` | -| `runtime` block | `0x30010` | -| `custom` block | `0x30014` | -| `scratch` block | `0x30018` | -| Control virtio-console | `0x3001c` | - -OpenVMM publishes config-change and used-buffer bits with a sequentially -consistent compare-exchange loop. It pulses the device IRQ only when the old -word is zero. The specialized Linux driver consumes all pending bits with a -sequentially consistent `xchg` to zero, so steady-state handling performs no -interrupt-status read or interrupt-acknowledgement MMIO access. A host OR that -races the exchange is either included in the exchanged value or observes zero, -stores the bit, and emits a new edge. Device reset and teardown clear the word. -Snapshot quiesce saves the pending value in transport state; restore writes it -back without replaying an edge, and the machine contract records the interrupt -mode, page GPA, and page size. - -### Block - -Block devices are routed directly to virtio-mmio rather than VPCI. Packed rings -are unavailable. Each block has a stable role, MMIO address, IRQ, access mode, -and fixed feature mask. - -#### Construction and discovery - -Each role-bearing attachment must use its role's access mode. OpenVMM opens the -host storage through its generic disk interface and creates one single-queue -virtio-blk device per role at the role's fixed virtio-mmio slot. No PCI bus or -PCI enumeration is involved. - -OpenVMM adds one profile-owned discovery token per block to the kernel command -line. The guest kernel creates the virtio-mmio device from that token, -negotiates features within the role's fixed set, and places the split -virtqueue in guest RAM. Guest device names are not role identities. The sandbox -launcher instead passes each role's MMIO address on the kernel command line, -and the guest agent maps that address to its block device. The agent mounts -lower roles read-only as EROFS and scratch read-write as ext4, then assembles -the [overlay root](sandbox-filesystem-and-agent-architecture.md#implemented-filesystem-bootstrap). - -#### Guest-visible capabilities - -Each role offers a fixed capability set: virtio 1.0 split rings with indirect -descriptors and event-index notification suppression, a per-request segment -limit, block-size and topology reporting, and a write-back cache with explicit -flush. Lower roles are also advertised as read-only. Discard is never offered, -even when the host storage could deallocate blocks. The device reports the -backing store's capacity and its logical and physical block sizes. - -#### Request and completion path - -The MMIO window is a control plane for device configuration and queue -notifications; block data never crosses it. A request is a descriptor chain in -guest RAM that holds a header with the operation and starting sector, any data -buffers, and a status byte. The guest publishes the chain and, unless the -device has suppressed notifications, notifies the queue. OpenVMM receives that -notification through a hypervisor doorbell when available and otherwise by -emulating the MMIO access. Either path wakes the device's I/O worker. - -The worker reads requests directly from guest memory and services several -concurrently. It translates each request's virtio sector address to the -backing store's logical blocks and rejects a read or write that does not start -on a logical-block boundary or whose length is not a whole number of logical -blocks. It does not require alignment to the reported physical block size, -which is only an alignment hint for the guest. Accepted reads and writes -become asynchronous scatter-gather I/O that targets guest buffers directly -when their layout permits. Individual writes are not forced to stable -storage; a guest flush becomes a backend cache flush. - -On completion, the device stores a success, I/O-error, or unsupported status in -the request, publishes one used-ring entry, and, unless the guest suppressed -the notification, signals the guest through the role's -[shared-status word and edge IRQ](#fixed-virtio-mmio-transport). The guest -driver therefore completes block I/O without reading or acknowledging -interrupt status through MMIO. - -#### Host storage and access enforcement - -Because the device depends only on OpenVMM's generic disk interface, its -guest-visible behavior is independent of the host storage type. The backend -sees block offsets and buffers; filesystem semantics remain entirely in the -guest. Without snapshot support, a role accepts any disk backend that OpenVMM -can open. Raw files use asynchronous kernel I/O on Linux, which can transfer -aligned buffers directly to and from guest memory, and positional file I/O on -worker threads on Windows, which copies through an intermediate buffer. - -Read-only access is enforced at three layers: the guest is told that the device -is read-only, the device rejects write and discard requests, and OpenVMM opens -an existing host file without write access. Scratch is the only writable role. - -A snapshot destination or restore makes every block snapshot-capable. Each -must then be a nonempty, 512-byte-aligned raw regular file that uses cached I/O -and is not reached through a final symbolic link. OpenVMM keeps that opened -file for identity hashing, flushing, and paired-scratch publication. - -#### Snapshot contract and quiesce - -The snapshot contract records the role, read-only flag, logical length, -logical and physical block sizes, and identity policy. Each consumed block uses -either whole-file SHA-256 or the caller-authenticated immutable storage -generation supplied at capture. Generation mode requires all bound roles to -carry the same nonzero generation and avoids content scans; the storage -controller remains responsible for binding that generation to the exact -immutable files. Platform-tier layers are recorded as unbound because image -binding has not been consumed; restore may supply different same-geometry -layers. Writable scratch uses one of two artifact policies: - -- **paired**: capture publishes `scratch.img` with its exact length and selected - identity. Restore rejects a caller-supplied scratch block and attaches the - file materialized by the recorded `private-copy`, reflink-only - `copy-on-write`, or single-use `direct-claimed` policy; -- **fresh**: capture occurs before scratch is mounted, publishes no scratch - artifact, and restore requires a new writable file with matching geometry. - -Virtio transport and queue progress are saved with the other device state. -Stopping a queue prevents new descriptor intake and drains every accepted I/O -to one used-ring completion under the bounded snapshot quiesce timeout. The -controller flushes the exact scratch handle and copies it only after that drain. - -### Console - -The optional console is the standard single-port virtio-console device with -two split queues. Host RX accepted by the device and partial guest TX progress -are device-private saved state, so a descriptor is not replayed from byte zero -after restore. Native sockets and handles are not serialized. - -The host attachment is a listener (a Unix socket, a named pipe, or loopback -TCP with a fixed nonzero port), a client connection, the inherited terminal, -or an explicitly disconnected endpoint that discards guest output. A listener -retains pending guest output until a client connects. The snapshot records the -attachment's stable ID, canonical endpoint identity, reconnect policy, -requiredness, and timeout. For a snapshot-capable machine, Unix sockets live -beside the snapshot directory and Windows pipes use OpenVMM's fixed microVM -pipe namespace. A saved listener may be recreated at a fresh private -restore-time endpoint, but its stable attachment ID, attachment kind, backend -kind, reconnect policy, required flag, length, and timeout remain exact. -Client reconnects retain their exact saved identity, require an explicitly -approved restore-time attachment, and have a five-second timeout; inherited -attachments must also be supplied again rather than serialized. Disconnected -attachments remain disconnected. - -### Control console - -The dedicated control console reuses the single-port virtio-console device but -has a distinct device and attachment kind, stable attachment ID -(`console:microvm-control0`), MMIO slot, IRQ, shared-status word, and -saved-state inventory. A fresh boot with a control console requires the boot -virtio-console, making the control device's guest tty `hvc2`; the profile owns -the identifying `nvx_control_tty=hvc2` token, and the boot console remains the -only kernel console. - -When a control console is present, the command-line builder rejects -caller-supplied control-tty, driver-probe-order, and device-discovery tokens, -including kernel-equivalent hyphenated spellings, as well as quotes and the -`--` delimiter. These rules prevent guest tty discovery from being redirected -by command-line parsing. Boot-only command lines retain their existing -behavior. - -The host endpoint is either a local listener or explicitly disconnected. A -live endpoint is protected in three layers: - -- **Private endpoint.** On Linux, it is a Unix socket bound exclusively with - mode `0600` inside an existing, non-symlink, owner-only directory of the - OpenVMM user. On Windows, it is a named pipe whose protected DACL admits only - LocalSystem and the OpenVMM user. TCP, client-connect, terminal, file, - standard-stream, and inherited backends are rejected, and the boot and - control endpoints must differ. -- **Peer identity.** The broker reserves its single host slot only for a peer - whose operating-system identity is the OpenVMM user: the effective UID of - the Unix-socket peer, or the token user SID of the process connected to the - pipe. Any other peer is disconnected before its bytes are read. -- **Capability.** The launcher passes a random 32-byte capability through a - prepared one-way standard-input pipe whose writers are closed before - OpenVMM starts. The first host record must present it within a bounded - authentication timeout, five seconds by default. The capability never - appears in arguments, environment variables, logs, endpoint names, - snapshots, or attachment identities. Because standard input carries it, the - interactive console is disabled, the boot console cannot use the terminal, - and the mode cannot be combined with the management RPC server, a console - relay, or a paused start. - -A disconnected control console uses a random capability that no client can -present. - -OpenVMM brokers a frozen, versioned record stream between the guest and the -authenticated host. Every record has a fixed-size header with a magic value, -protocol version, record type, broker instance ID, epoch, direction-local -sequence, and payload length. Bootstrap attach records establish the guest and -host sides; acknowledgment, data, wait, ready, reset, authentication-error, -and credit records carry the session. Host data toward the guest is limited by -byte-counted receive credit that the guest grants. When credit is exhausted, -one complete host record may wait in a bounded pending slot, and each output -direction is bounded in records and bytes. Malformed lengths, wrong-state -records, wrong instance or epoch values, and duplicate or out-of-order -sequences fail closed. A wrong capability or an authentication timeout -detaches the client without advancing the epoch. An active host disconnect -advances the epoch, drops unstarted data and unused credit, finishes any -partially transmitted record to keep the stream aligned, and emits a reset; -the next session receives credit only from a fresh guest acknowledgment. - -Snapshots record only the endpoint identity and a broker policy, either an -authenticated listener or a disconnected console, never the capability or the -peer UID or SID. Broker saved state omits queued records, host output, and -usable receive credit; restore keeps only the guest-side parser, any partially -transmitted guest record, and error counters. Restore validates the saved -endpoint and requires the launcher to request that same endpoint again with a -fresh capability. The restored broker receives a fresh instance ID, restarts at -the first epoch with a queued reset, and ignores guest records for the old -instance until the guest acknowledges the new one. The management RPC exposes -no control console and explicitly rejects restoring a snapshot that contains -one. - -The control console carries the managed workload lifecycle: a host-owned -lifecycle token selects one-shot or managed operation, and managed operation -requires a fixed workload identity and a live, authenticated control console. -The workload protocol carried inside the brokered data stream belongs to the -guest agent; see -[Control protocol and checkpoint handoff](sandbox-filesystem-and-agent-architecture.md#control-protocol-and-checkpoint-handoff). - -### Network - -#### Architecture and device creation - -Networking is a guest virtio-net NIC backed by the in-process Consomme -userspace network stack in OpenVMM. Consomme translates guest traffic into -ordinary host socket operations; it does not attach the guest to a physical -NIC, TAP interface, bridge, or host virtual switch. The host does not need a -network interface assigned the guest's gateway address. - -The optional NIC occupies the fixed virtio-mmio slot at `0xd0000000`, has one -RX/TX queue pair, and advertises an exact feature mask: the MAC-address feature -and virtio version 1. OpenVMM constructs the virtual device and its Consomme -endpoint, binds the network identity and egress policy to them, and configures -any explicit host port forwards. The guest discovers the device through the -`virtio_mmio.device=` kernel command-line token. - -`--net IPv4/prefix --network-profile portable` accepts prefixes `/1` through -`/30`, derives the first usable address as the gateway, and derives -deterministic guest and gateway MAC addresses from the final three IPv4 -octets. The profile is mandatory and selects the same in-process Consomme data -plane on KVM, MSHV, and WHP; TAP attachments are rejected for this profile. - -```mermaid -%%{init: {"theme": "base", "themeVariables": {"background": "#ffffff"}}}%% -flowchart TB - Guest["Guest application and Linux TCP/IP stack"] - Driver["Guest virtio-net driver"] - Queues["TX/RX virtqueues in guest RAM
MMIO notifications and interrupts"] - Device["OpenVMM virtio-net device
guest TX header prefilter"] - Endpoint["In-process Consomme endpoint
final guest TX authorization"] - Host["Host sockets and DNS resolver"] - Network["Host OS routing, firewall, and network"] - Peer["Remote server or permitted host-local service"] - - Guest <--> Driver - Driver <--> Queues - Queues <--> Device - Device <--> Endpoint - Endpoint <--> Host - Host <--> Network - Network <--> Peer -``` - -#### Guest addressing and bootstrap - -OpenVMM supplies the static IPv4 address, netmask, and gateway through the -`virtnet_ip=`, `virtnet_mask=`, and `virtnet_gw=` kernel command-line tokens. -The [guest initialization code](../../guest/common/init) configures the -interface and its default route from those tokens rather than using DHCP. - -The NIC is dual-stack: its static IPv6 identity embeds the guest and gateway -IPv4 addresses in `fd00::/96`, with a prefix of 96 plus the IPv4 prefix, so -`10.0.0.2/24` becomes `fd00::a00:2/120` behind `fd00::a00:1`. The gateway -shares the IPv4 gateway's MAC address, and the guest configures the identity -from the `virtnet_ip6=` and `virtnet_gw6=` tokens without router -advertisements, autoconfiguration from them, or duplicate address detection. -For example: - -| Setting | Value | -| --- | --- | -| Guest IPv4 address | `10.0.0.2/24` | -| IPv4 gateway | `10.0.0.1` | -| Guest IPv6 address | `fd00::a00:2/120` | -| IPv6 gateway | `fd00::a00:1` | - -The guest kernel's EUI-64 link-local address is its only other IPv6 address, -and egress rules admit it only as the source of Neighbor Solicitations. -The static identity advertises no routable IPv6 prefix. External reachability -depends on the host's routes and address-family support; configuring this -private IPv6 identity does not give an IPv4-only host IPv6 Internet access. - -#### Packet forwarding - -Guest transmission follows this path: - -1. The guest Linux stack constructs an Ethernet frame, and the virtio-net - driver places its buffers in the TX virtqueue in guest RAM. -2. The driver notifies the virtual device through MMIO. OpenVMM reads the - descriptors and prefilters the frame headers against the egress policy. -3. The endpoint copies the complete frame into an owned buffer, authorizes - those exact bytes, and passes an allowed frame to the Consomme stack. -4. For an allowed new TCP connection, Consomme creates a host TCP socket and - connects it to the destination. For UDP, it sends the guest's datagram - payload through a host UDP socket. - -TCP forwarding is socket proxying, not simply rewriting the original TCP -packet. Consomme implements the guest-facing TCP connection, while the host -kernel implements the remote-facing connection. Payload bytes move between -them with separate sequencing, acknowledgments, and buffering. A remote server -normally sees the host's network identity, not the guest's private address. -Host routing, VPN configuration, and applicable host firewall rules still -govern the host sockets; host kernel IP forwarding is not required. - -When data arrives on a host socket, Consomme associates it with the -guest-facing flow and constructs the corresponding Ethernet/IP/TCP or UDP -packets. The endpoint writes them into buffers that the guest has posted on -the RX virtqueue. OpenVMM completes the descriptors and signals the guest -through the virtio interrupt mechanism; the guest driver passes the received -packets into Linux. - -ARP and IPv6 Neighbor Discovery are handled locally by the backend, not -broadcast onto the physical LAN. The portable profile provides outbound TCP and -UDP over IPv4 and IPv6. The backend answers echo requests to the IPv4 or IPv6 -gateway itself and sends other IPv4 echo requests through a host ICMP socket; -it does not relay ICMPv6 echo to other destinations. It deterministically -rejects fragmented IPv4 packets, IPv6 extension headers, and IPv4-mapped IPv6 -destinations rather than forwarding unsupported packet constructions. - -#### Egress policy and enforcement - -By default, guest egress is allowed and general host ingress is denied. One -canonical egress policy applies before externally visible transmission: - -- allow all traffic, the default; -- deny all traffic with `--network-egress deny` and no rules; -- allow only listed IPv4 hosts or CIDRs with `--allow-host`; -- allow IPv4 except listed hosts or CIDRs with `--block-host`; -- allow only exact IPv4 TCP endpoints with `--allow-endpoint`; or -- combine canonical IPv4 or IPv6 CIDR allow and deny rules with an optional - TCP, UDP, or ICMP selector, optional TCP or UDP destination ports, and deny - precedence. - -The legacy `--allow-host`, `--block-host`, and `--allow-endpoint` modes are -mutually exclusive, cannot be combined with generic rules, and deny all IPv6 -traffic. The generic rule mode requires an explicit `--network-egress allow` -or `--network-egress deny` default action and accepts at most 256 allow and -256 deny rules through `--network-egress-allow` and `--network-egress-deny`. -Rules select destinations by address or CIDR, with an optional transport and -TCP or UDP destination port or inclusive port range. Evaluation is independent -of rule order: - -1. A matching deny rule denies the packet. -2. Otherwise, a matching allow rule allows it. -3. Otherwise, the default action decides. - -A rule matches only destinations of its own address family. The IPv6 gateway's -Neighbor Discovery is authorized like the IPv4 gateway's ARP. Filtering is -inactive only when the policy allows all traffic: in the default mode, or with -generic rules that have an allow default and no deny rules. When filtering is -active, packet checks enforce the bound guest MAC and IP identity and validate -the headers, not just the destination. Port-specific traffic fails closed -when malformed or fragmented. - -The virtio-net device validates the bound policy, installs it on the endpoint, -and performs the early header check. That check alone is insufficient because -guest RAM remains mutable. The endpoint therefore repeats authorization on -the complete owned frame before passing it to Consomme, so a guest cannot -rewrite a checked frame into a different transmitted destination. Denied -frames are dropped and their TX descriptors completed without host -transmission. - -These are destination-based L3/L4 rules, not hostname or HTTP-content filters. -They apply to every frame that the guest transmits, including replies on -explicitly forwarded connections, but they are not an independent RX firewall -or host-wide firewall. Responses arriving through the host sockets use the RX -path. Ingress `allow` additionally admits replies on the flows that the -endpoint carries, as [Host-local access and ingress](#host-local-access-and-ingress) -describes. - -#### Host-local access and ingress - -By default, and subject to the egress policy, the gateway maps guest TCP and -UDP flows onto host loopback. The IPv4 gateway maps to `127.0.0.1`, and the -static IPv6 gateway maps to `::1`. For example, a guest connection to -`10.0.0.1:8080` reaches a host service on `127.0.0.1:8080`. The guest's own -`127.0.0.1` still means guest loopback, not host loopback. Guest flows to -other host-local-only destinations, such as link-local addresses, also reach -host sockets unless host loopback is denied. - -`--host-loopback deny` disables the general gateway-to-host-loopback mappings, -blocks loopback, unspecified, and link-local destinations, and disallows every -forward. One exact IPv4 gateway TCP proxy endpoint may remain available under -deny through `--network-proxy`; it is bound into the policy digest, and -OpenVMM translates it to the same port on host `127.0.0.1`, where the -launcher, not OpenVMM, supplies the proxy service. The egress policy admits -TCP frames to that endpoint before evaluating its mode. Gateway DNS is a -separate backend service governed by the egress policy, not by the general -host-loopback mapping. - -An outbound connection does not make guest services reachable from the host -or LAN. Without ingress `allow`, which the paragraphs below describe, the host -reaches guest services through explicit host-loopback access. An explicit -`--host-loopback allow` is bidirectional: besides the gateway mapping, it -publishes the guest's listeners on host loopback, as the paragraphs below -describe, and admits explicit localhost TCP or UDP port forwards, at most 64 of -them. For example: - -```text ---host-loopback allow ---host-loopback-forward tcp:8080:80 -``` - -This binds a real host listener on `127.0.0.1:8080` and forwards connections -to guest port `80`; it does not bind a LAN-facing listener. Each forward -requires explicit `--host-loopback allow`, and omitting `--host-loopback` -keeps the default gateway mapping without publishing guest ports. For -host-loopback peers, the backend uses synthetic peer addresses so the guest -routes replies through its NIC rather than its own loopback, and -reverse-translates those addresses on TX. Synthetic addresses lie inside the -guest's subnet of their family, allocated downward from its last usable -address, such as `10.0.0.254` and `fd00::a00:ff`, so the guest answers them -from its address in that subnet, which the egress policy admits as the guest's -source address. The -endpoint's egress policy evaluates the guest's traffic to a synthetic address -at the host address that it stands for, and at the gateway of its family for -host loopback, so a synthetic address reaches no host-loopback port that the -rules deny at the gateway. The virtio-net prefilter, which knows no synthetic -addresses, leaves them to the endpoint. - -Ingress `allow` publishes the guest's listeners to the host's private -networks. OpenVMM then appends the `virtnet_ingress=allow` token, which callers -may not supply, after the other network tokens, and the guest announces its -listeners in UDP datagrams to port 1021 of the IPv4 gateway from a source port -below 1024. An announcement is the magic `NVXL`, version 1, a flags byte whose -bit 0 marks omitted listeners, a big-endian 16-bit count of at most 64, and -that many four-byte entries in strictly increasing order of protocol and port: -the IP protocol number, 6 for TCP or 17 for UDP, a family mask with 1 for IPv4 -and 2 for IPv6, and the big-endian nonzero port. The endpoint consumes -announcements before the egress policy, which neither authorizes nor forwards -them, and each one replaces the published set. - -For each published port and family, the endpoint listens on that port of every -host address and relays only peers in the IPv4 private-use, shared, -link-local, benchmarking, and documentation ranges or the IPv6 unique local, -site-local, documentation, and benchmarking ranges, none of which is globally -reachable, except peers inside the guest's own subnets, which the guest would -take for its gateway or its neighbors. Peers at the host's own addresses, -including loopback, need an explicit host-loopback allow. - -An explicit host-loopback allow without ingress `allow` publishes the guest's -listeners on host loopback alone. OpenVMM then appends the -`virtnet_ingress=host-loopback` token instead, the guest announces its -listeners in the same way, and for each published port and family the -endpoint listens on that port of the family's loopback address, `127.0.0.1` -or `::1`, and admits only the host. It never displaces a host listener that -holds the port, on loopback or on every host address; on Windows, which would -let a loopback binding take loopback traffic from a listener of the same user -on every address, it first binds the port on every address of the family and, -with a dual-stack socket, on every address of both families, which Windows -binds independently, and keeps those bindings until the loopback binding -holds the port, so no host listener can bind every address in between. A host -listener that binds every address of a published port later fails on Linux; on -Windows, unless it asks for exclusive use of the port, it succeeds and the -publication keeps the port's loopback traffic, as a loopback listener of any -host process would. - -Under ingress allow, or host-loopback allow alone, the policy also admits ARP -requests and Neighbor Solicitations for the guest's subnet, where the gateway -presents host peers. It admits a frame that its rules deny when the frame -continues a flow that the endpoint carries: a TCP segment other than an -initial SYN, or a UDP datagram from a published port to a peer that recently -sent one to it, which leaves through the port's host listener. Either selects -canonical policy encoding version 5, which adds the ingress action, 0 for -allow and 2 for host loopback alone, and leaves the encoding and digest of -other policies unchanged. Snapshot capture and restore reject both, and the -TAP backend rejects policies that allow either. - -#### DNS - -The gateway provides a DNS proxy over UDP and TCP port 53. The -`virtnet_dns=` token names the IPv4 gateway when the egress policy permits -DNS to it, and otherwise the IPv6 gateway when generic rules permit DNS to -that one. Exact-endpoint policies never name a gateway. If no gateway is -named, the token is omitted. The guest initialization code writes the -selected server into `/etc/resolv.conf`. - -A guest query to the gateway passes the normal TX egress checks before -Consomme handles it as a DNS request. Upstream resolution uses the host -resolver. The glibc build that NVX uses with KVM calls the reentrant -`res_nsend` and forces TCP upstream for queries that arrived over TCP. The -statically linked musl build that NVX uses with MSHV calls `res_send`, which -selects its own transport. Both read the host's `/etc/resolv.conf`. Windows -calls `DnsQueryRaw` with the query's transport and the host DNS -configuration. Consomme relays the resolver's response from the gateway -through the guest RX path. - -The host resolver's upstream DNS servers are not guest packet destinations, -so the guest's CIDR rules do not check them. Direct guest queries to an -external DNS server instead use the ordinary outbound socket path and require -permission for that server's address and port. - -DNS permission and connection permission are separate: resolving a hostname -does not authorize a connection to the returned address, and the egress -policy does not filter domain names. Normal DNS behavior should permit both -UDP and TCP port 53 to the chosen gateway. Denying host-loopback access does -not by itself disable gateway DNS that the egress policy permits. - -#### Bounded state, snapshots, and restore - -Flow, DNS, buffer, and packet-queue state are bounded. At most 128 TCP, 256 UDP, -and 16 ICMP guest flows are active at once, and excess flows are rejected -before a host socket is created. At most 256 DNS requests may be pending. -A resolver failure produces a `SERVFAIL` response. At the pending-request -limit, a further UDP query receives `SERVFAIL`, and a further TCP query resets -its connection. - -The snapshot records the profile, network identity, and policy digest. A -policy bound to an IPv6 identity uses canonical encoding version 4, while one -restored from a snapshot that predates IPv6 keeps version 3 and its recorded -digest. The snapshot records the IPv6 identity; a snapshot whose network -predates it restores IPv4-only. Restore reconstructs a fresh endpoint and -requires the profile and policy again; native sockets, DNS requests, and flow -objects are never serialized. Explicit localhost forwards are process-local -attachments, and so are the listeners that ingress `allow` publishes; snapshot -capture and restore reject both. - -Capture quiesces the endpoint, drains completion ownership, and requires the -saved queues to contain no unrepresented RX or TX packets. It then saves the -queue lifecycle, negotiated features, link state, and endpoint generation. -Arbitrary in-flight packet payloads are not serialized. Pre-capture queued -packets are not replayed, old TCP/UDP/ICMP/DNS state is invalidated, and new -traffic must establish fresh post-restore flows. - -### Filesystem - -The filesystem slot shares live host files with the guest through virtio-fs. -The guest kernel's FUSE client turns system calls on the mounted tag into FUSE -requests, which travel through virtqueues in guest RAM to HostFs, a FUSE -server inside OpenVMM. HostFs performs each request on the exported host -directory, or on the children of an aggregate, and returns the reply through -the same queue. Unlike a [block device](#block), whose backend sees only -blocks, HostFs sees names, attributes, and open files: the host owns the -namespace and the data, while the guest owns its VFS objects, permission -checks, and locks. Guest writes reach the host files directly, and host changes -are visible to the guest on its next request. - -```mermaid -%%{init: {"theme": "base", "themeVariables": {"background": "#ffffff"}}}%% -flowchart TB - App["Guest application"] - Client["Guest VFS and FUSE client
virtiofs mount of tag microvm"] - Queues["High-priority and request virtqueues in guest RAM
MMIO notifications and shared-status IRQ 6"] - Device["OpenVMM virtio-fs device
one worker per queue"] - Session["FUSE session
INIT negotiation and opcode dispatch"] - HostFs["HostFs
nodes, handles, access policy, and host identity"] - Volume["Confined host volume
no-follow resolution beneath the export root"] - Export["Exported host directory
or aggregate children"] - - App <--> Client - Client <--> Queues - Queues <--> Device - Device <--> Session - Session <--> HostFs - HostFs <--> Volume - Volume <--> Export -``` - -#### Device and FUSE session - -The slot is a virtio-fs device, device type 26, with the fixed tag `microvm`, -one high-priority queue, and one request queue. Its configuration space holds -only the tag and the request-queue count. It offers split rings with indirect -descriptors and event-index notification suppression. Packed rings and the DAX -shared-memory window are never offered, so file data always travels through -the queues. The explicit profile rejects SectionFs, alternate tags, extra -queues, shared-memory windows, PCI transport, and aggregates whose children -change after boot. - -The guest opens the FUSE session with `INIT`, and every other request fails -until it succeeds. The device speaks FUSE protocol version 7 and accepts guest -minor versions 27 and newer, but the microVM contract is FUSE 7.31 or newer: a -restore rejects a session that negotiated an older minor version. It grants -what the guest offers from a fixed set: asynchronous reads, parallel directory -operations, automatic data invalidation, kill-privilege handling, asynchronous -direct I/O, atomic `O_TRUNC`, big writes with a negotiated page limit, extended -`INIT`, adaptive `READDIRPLUS`, and, through extended `INIT`, shared mappings -of direct-I/O files. It negotiates a 1 MiB maximum write, or 256 pages, and -nanosecond timestamps, and it leaves the guest's background-request limits at -their defaults. It never grants writeback caching, lock forwarding, or DAX -mappings. - -Every entry and attribute in a reply carries zero cache lifetimes, and every -file that `OPEN` or `CREATE` opens uses direct I/O. The guest therefore looks up -each path component again on every walk, fetches attributes with `GETATTR` -whenever it needs them, and sends every read and write to the host instead of -its page cache. Host changes are visible on the guest's next request, and a -guest write has reached the host file when the write completes. Shared -mappings are the exception: their pages are cached in the guest and written -back through ordinary `WRITE` requests. The zero lifetimes also make the guest -drop a name and its inode as soon as nothing uses them and send `FORGET`, so -it holds nodes only for objects in use and the directories that lead to them, -such as open files, working directories, and mount and bind points. Because -lock forwarding is not negotiated, the guest kernel keeps POSIX and BSD locks -local, and host processes neither see nor honor them. Linux virtio-fs mounts -enforce `default_permissions`: the guest kernel checks Unix permissions against -the attributes that the host reports before it sends a request, and the host -filesystem then checks the operation against the -[host identity](#host-identity-of-requests) that performs it. - -#### Attachments and guest mounts - -Without `--mount` or `--mount-aggregate`, the slot is dormant: it has no HostFs -backend or active filesystem policy but remains guest-discoverable, so a -snapshot of it can -[gain a share at restore](#dormant-slot-and-saved-state-versions). A dormant -slot can complete `INIT`, but it serves no files: its filesystem requests fail -with `ENOSYS`. `--mount GUEST_TARGET,HOST_PATH[,ro|rw]` may appear once and -attaches HostFs to one host directory; it exports only a directory, because the -guest mounts the device's root. `--mount-aggregate GUEST_TARGET` with a -repeatable `--mount-child NAME,HOST_PATH[,ro|rw][,file]` instead attaches an -[aggregate](#aggregate-layout) of host directories and regular files. -`--mount-deny`, `--mount-allow`, `--mount-write`, and `--mount-owner` set the -[access policy](#confinement-and-access-policy) and -[host identity](#host-identity-of-requests) of either form. - -The attachment adds one `virtfs_dir=`, `virtfs_tag=`, `virtfs_mode=` token -triplet to the kernel command line, with `virtfs_mode=rw` when the share or any -child is read-write, and `virtfs_aggregate=1` for an aggregate. On a cold boot, -the guest init mounts tag `microvm` at `virtfs_dir` with that mode. For an -aggregate, it then binds each child at the target that an NVX `nvx_share=` -token names, as -[Live host shares](sandbox-filesystem-and-agent-architecture.md#live-host-shares) -describes. Guest mount flags are not a security boundary: HostFs enforces the -mode and policy of the share or child on every request, whichever guest mount -or link reaches it, so a second guest mount of the tag changes nothing. - -#### Aggregate layout - -An aggregate's root is synthetic: a directory with mode `0500`, owned by root, -that lists one child per `--mount-child` under its name, in option order, and -refuses every change with `EROFS`. Below each child is an independent HostFs -volume with its own access mode and denied, allowed, and writable paths. The -`file` flag, not the host object, sets a child's kind, and OpenVMM refuses a -host path of the other kind. A file child serves only its file, as the child -itself: its volume is the file's parent directory, narrowed to the file's name, -which the guest never reaches, so the guest cannot list or change anything -beside the file, a lookup below the file fails with `ENOTDIR`, and every -request fails with `EACCES` while anything but a regular file holds the name. A -file child takes no policy paths, because its mode applies to the whole file. - -Policy paths of an aggregate are absolute and apply to the child whose root -contains them. Child names are unique `[A-Za-z0-9._-]` names of at most 64 -bytes, at most 256 children share at most 128 KiB of policy paths, and their -host roots must not equal or contain one another, so no child can reach the -files of another under a different policy, while a file child may lie beside a -directory child. The children share one FUSE superblock, so HostFs mixes a -per-child key into each host inode number to make `st_ino` collisions between -children unlikely. A rename between children fails with `EXDEV`. A hard link -between children fails with `EROFS` when its destination is read-only, which -Linux also reports first, and otherwise with `EXDEV`. - -#### Request translation and flow - -A guest operation on the mount becomes one or more FUSE requests: - -1. A system call reaches the guest VFS, whose FUSE client issues the requests - that it needs: `LOOKUP` for each path component, `GETATTR` for attributes, - `OPEN` or `CREATE` for a descriptor, and `READ` or `WRITE` for data. -2. The client places each request in a descriptor chain in guest RAM. - Device-readable buffers hold the FUSE input header, with the opcode, request - ID, node ID, and the caller's UID, GID, and PID, followed by the opcode's - arguments, names, and any write payload. Device-writable buffers receive the - reply. `FORGET` notifications use the high-priority queue and have no reply - buffers; all other requests use the request queue. -3. The client publishes the chain and, unless the device has suppressed - notifications, notifies the queue. OpenVMM receives the notification through - a hypervisor doorbell when the backend provides one and the slot has an - attachment, and otherwise by emulating the MMIO write. Either path wakes the - queue's worker. -4. The worker takes the chain, copies the request out of guest memory, and - parses its header. In caller mode, it assumes the caller's host identity. - The FUSE session then dispatches the opcode to HostFs. -5. HostFs resolves the request's node ID to a - [node](#nodes-handles-and-directory-listings) or its handle ID to an open - host object and applies the access mode, access policy, and table bounds. -6. The confined host volume performs the operation beneath the export root - without following symbolic links. A Linux host issues the corresponding - system call relative to a pinned parent directory; a Windows host emulates - the same Linux semantics through Windows file APIs, without storing Linux - ownership or mode metadata. -7. HostFs encodes the result, which is attributes, entries, data, or a negative - errno, after a FUSE output header that carries the request ID, and the worker - writes it into the device-writable buffers. The worker then publishes one - used-ring entry with the reply length and, unless the guest suppressed the - notification, signals the guest through the slot's - [shared-status word and edge IRQ](#fixed-virtio-mmio-transport). -8. The client matches the reply to its request by ID and completes the system - call. - -Each queue has one worker, which performs one request at a time to completion -before it takes the next. The request queue therefore serializes guest requests -in ring order, although the guest may post many at once, while `FORGET` -notifications proceed independently on the high-priority queue. The device -does not implement FUSE interrupts, and the Linux driver sends none, so a -posted request always runs to completion. - -`READ` reads into a host buffer of at most 1 MiB and copies the bytes into the -guest's reply buffers; `WRITE` copies its payload out of guest memory before it -writes the host file. Requests on an open handle use its host descriptor -directly, without resolving a path again. A request whose device-readable part -exceeds the 1 MiB write limit plus a 4 KiB header allowance, or whose header -cannot be parsed, is completed with no reply. Malformed arguments fail with -`EIO` or a specific error. An opcode that HostFs does not implement, such as -`FLUSH`, `FALLOCATE`, `LSEEK`, or `COPY_FILE_RANGE`, fails with `ENOSYS`; Linux -remembers the failure, stops sending that opcode, and either falls back to -generic handling or reports the operation as unsupported. A guest `syncfs` -arrives as `SYNCFS`, which the device acknowledges without flushing the host. - -#### Nodes, handles, and directory listings - -HostFs identifies host objects to the guest by node ID. Node 1 is the root: the -share's root directory or an aggregate's synthetic root. A node records its -volume, the volume-relative names, or aliases, through which the guest has -reached it, its host inode number, its guest-visible inode number, and a lookup -count. It holds no host descriptor. Every request on a node first walks each -recorded name from the export root again without following links, fails with -`ELOOP` if a name now crosses a symbolic link and with `EACCES` if the policy -hides it, and then operates on the final component without following it. - -Repeated lookups of one host object return the same node and add the new name, -so hard links share one node. HostFs matches objects by host inode number, or -by name on volumes, such as FAT, whose inode numbers are not stable. Every reply -that returns an entry, from `LOOKUP`, `CREATE`, `MKDIR`, `MKNOD`, `SYMLINK`, -`LINK`, or each `READDIRPLUS` entry, adds one lookup. `FORGET` subtracts the -count that the guest reports, and the node disappears at zero. `RENAME` and -`UNLINK` rewrite or remove the recorded names at and below the affected path in -every node, and `RENAME_EXCHANGE` swaps them. - -`OPEN`, `OPENDIR`, and `CREATE` return handles. A handle owns a host descriptor, -opened without following the final component, and requests that name the -handle, such as `READ`, `WRITE`, and `FSYNC`, use that descriptor. `RELEASE` and -`RELEASEDIR` close it. The first `READDIR` or `READDIRPLUS` on a directory -handle enumerates the host directory once into a listing of at most 4,096 -entries and 64 KiB of names, without the entries that the policy hides, and -later requests page through that listing; listing a larger directory fails -with `E2BIG`. Each entry's cookie is its position in the listing, so a listing -survives a snapshot. `READDIRPLUS` looks each listed name up again and fails if -it no longer reaches the listed object. - -HostFs bounds its state at 4,096 nodes, 4,096 handles, 256 names per node, -16,384 names in total, 1 MiB of names, and 4,096-byte paths. It checks these -bounds before it changes the host, so an operation that would exceed them -fails with `ENOSPC` or `E2BIG` instead of creating host state that the device -could not track or save. - -#### Operation examples - -These examples use a single read-write share mounted at `/workspace`, whose -root is node 1. The diagram follows one file from creation to removal. - -```mermaid -%%{init: {"theme": "base", "themeVariables": {"background": "#ffffff"}}}%% -sequenceDiagram - box rgb(255, 255, 255) - participant App as Guest application - participant Client as Guest FUSE client - participant HostFs as virtio-fs device and HostFs - participant Host as Host filesystem - end - - rect rgb(255, 255, 255) - App->>Client: open /workspace/out.log with O_CREAT and O_RDWR - Client->>HostFs: LOOKUP(1, out.log) - HostFs->>Host: No-follow lookup beneath the root - HostFs-->>Client: ENOENT - Client->>HostFs: CREATE(1, out.log, flags, mode) - HostFs->>Host: Create and open in the pinned root directory - HostFs-->>Client: Node M, handle H, direct I/O - App->>Client: write 1 MiB at offset 0 - Client->>HostFs: WRITE(H, offset 0, 1 MiB) - HostFs->>Host: Positional write through H - HostFs-->>Client: 1 MiB written - App->>Client: read 4 KiB at offset 0 - Client->>HostFs: READ(H, offset 0, 4 KiB) - HostFs->>Host: Positional read through H - HostFs-->>Client: 4 KiB of data - App->>Client: unlink /workspace/out.log - Client->>HostFs: LOOKUP(1, out.log), then UNLINK(1, out.log) - HostFs->>Host: No-follow unlink in the pinned root directory - HostFs-->>Client: Success, and H still reaches the file - App->>Client: close - Client->>HostFs: RELEASE(H) - HostFs->>Host: Close the descriptor of H - Client->>HostFs: FORGET(M, n) on the high-priority queue - HostFs->>HostFs: Drop node M - end -``` - -**Opening a file.** An application calls -`open("/workspace/src/main.c", O_RDONLY)`. - -1. Because no entry stays cached, the guest's path walk sends - `LOOKUP(1, "src")` and then `LOOKUP(S, "main.c")`, where `S` is the node that - the first reply returned. -2. For each `LOOKUP`, HostFs revalidates the parent node's names, fails with - `EACCES` for a name that the policy hides, and inspects the child beneath - the export root without following it. It returns the existing node of a - known host object or allocates a new one, records the name, adds a lookup, - and replies with the node ID and attributes. -3. The guest checks the caller's permission against those attributes and sends - `OPEN(M, O_RDONLY)` for the file's node `M`. -4. HostFs revalidates `M`, refuses write access or `O_TRUNC` with `EROFS` when - the node is read-only, and opens `src/main.c` beneath the root without - following its final component. It rejects a symbolic link with `ELOOP` and - replies with a new handle `H` and direct I/O. The guest's descriptor now - refers to node `M` and handle `H`. - -**Creating a file.** An application calls -`open("/workspace/out.log", O_WRONLY | O_CREAT | O_EXCL, 0644)`. - -1. `LOOKUP(1, "out.log")` fails with `ENOENT`. -2. The guest sends `CREATE(1, "out.log")` with the flags, the mode with the - caller's umask applied, and the umask. -3. HostFs requires a read-write share or child and, when writable paths exist, - that both the parent directory and the new name lie inside one; otherwise - the request fails with `EROFS`. It checks its table bounds before it touches - the host. -4. HostFs creates and opens `out.log` in the pinned root directory without - following links, as the VMM or, in caller mode, as the caller, and rejects a - symbolic link. It registers a node for the name and a handle for the - descriptor and replies with both, so one round trip creates and opens the - file. - -`mkdir`, `mknod`, and `symlink` pass the same checks through `MKDIR`, `MKNOD`, -and `SYMLINK` and return a node without a handle; `SYMLINK` stores its target -exactly as the guest gives it. - -**Reading.** An application reads 3 MiB from offset 0 of a descriptor whose -handle is `H`. - -1. Direct I/O bypasses the guest page cache, so the guest splits the read into - `READ` requests of at most 1 MiB, each naming `H`, an offset, and a size, - with guest buffers for the data. -2. For each request, HostFs reads from the host descriptor of `H` at the - offset into a host buffer and copies the bytes into the reply buffers after - the output header. A short reply marks the end of the file. -3. The guest completes the read when every reply has arrived. A later read - requests the data again, so it observes any host change. - -**Writing.** An application writes 3 MiB at offset 0 through the same -descriptor. - -1. The guest splits the write into `WRITE` requests of at most 1 MiB, each - carrying `H`, an offset, and the data in device-readable buffers. -2. HostFs copies each request, payload included, out of guest memory and - refuses it with `EROFS` if the node is read-only, which it checks on every - write, not only at open. -3. HostFs writes the data through the host descriptor of `H` at the offset, - as the request's host identity, and replies with the number of bytes - written. -4. When `write` returns, host processes can read the data, but it is not - forced to stable storage. `fsync` and `fdatasync` become `FSYNC`, which - HostFs performs through `H`. Closing the last descriptor sends - `RELEASE(H)`, and HostFs closes the host descriptor. - -**Removing a file.** An application calls `unlink("/workspace/out.log")`. - -1. `LOOKUP(1, "out.log")` returns node `M`. -2. The guest sends `UNLINK(1, "out.log")`. HostFs applies the write checks of - `CREATE` to the root directory and the name, revalidates the root, and - removes the name in the pinned root directory without following it. -3. HostFs removes the name from every node that recorded it. Node `M` remains - while the guest holds lookups on it, and any handle keeps its host - descriptor. -4. When the guest evicts its inode, it sends `FORGET(M, n)` on the - high-priority queue with the lookups that it held. The request has no reply; - HostFs subtracts `n` and drops the node at zero. - -`rmdir` takes the same path through `RMDIR`, which also removes the recorded -names below the directory. - -#### Confinement and access policy - -The host root is pinned by its platform object identity, device and inode -number on Linux or volume and file ID on Windows, and revalidated when the -export is opened; each aggregate child's root is pinned the same way. Every -operation stays confined to the export. On Linux, each host operation opens -its parent directory beneath the root without following a symbolic link or -leaving the root, through `openat2` when the kernel allows it and one component -at a time otherwise, and never follows the final component. On Windows, -guest-created links are WSL-style reparse points that path resolution never -follows, ancestor components that are links or reparse points are rejected, -and the volume refuses DOS device names and objects with the hidden, system, -offline, or recall-on-data-access attribute with `EACCES`. A read-write mapping -lets the guest create symbolic links with exact targets, which only the guest -resolves; a read-only mapping rejects them. On a Linux host, a rename passes -its flags to `renameat2`, so the guest can swap two names with -`RENAME_EXCHANGE`, and the device swaps the names that it tracks for both -objects; a Windows host rejects every rename flag with `EINVAL` before changing -anything. - -Read-only mode, of a share or of an aggregate child, rejects mutation in the -host device before invoking host filesystem operations; read-write mode exposes -only the supported common host contract. Denied host paths are canonicalized -into a bounded relative set and enforced before HostFs operations. Prefix -checks hide complete subtrees, while denied root device/inode identities block -hard-link, junction, and bind-mount aliases. Allowed host paths expose subtrees -of denied paths again; the nearest policy path that contains a path decides -whether the guest can see it, and denied and allowed paths alternate, so -neither kind is redundant. A denied path may name the root of a share or child -when allowed paths inside it remain, which leaves the root traverse-only. -Policy path names may contain spaces, but not begin or end with one. A hidden -directory on the way to an allowed path is traverse-only: the guest can look it -up as a directory and list only the entries that lead to allowed paths, and its -object identity is accepted only at its own path. Writable host paths, when -present, are the only parts of a read-write export that the guest can modify. -Every namespace mutation checks its directories and the entry paths that it -creates, removes, or replaces, and every object mutation checks every name that -the guest has used for the object; refused writes fail with `EROFS`, and a hard -link to a read-only object from a writable path fails with `EXDEV`. - -#### Host identity of requests - -The ownership mode selects the host identity of guest operations. By default, -HostFs performs them as the VMM. With `--mount-owner caller`, a Linux host -performs each request that can access host files as the UID and GID in its -FUSE header: the queue worker switches only its own thread's filesystem UID -and GID, clears the supplementary groups and effective capabilities, and -restores all of them before handling anything else. Session negotiation, -`FORGET`, and handle release run unchanged. Guest UID 0 and GID 0 are squashed -to the owner and group of the export root, which therefore must not be UID 0 or -GID 0. A request whose identity cannot be assumed fails with `EPERM` rather -than running as the VMM. Windows hosts reject the mode. - -#### Snapshot contents - -The exported files are external live state, not part of the VM snapshot: -neither their contents nor any host descriptor, handle, or cache is captured. -A snapshot instead preserves the protocol state on both sides of the queues. -The guest's FUSE client state, including its mount, node IDs, lookup counts, -and handles, lives in guest RAM and is captured with memory. The device keeps -the matching state in device-private saved state: - -- the negotiated FUSE session; -- each node's ID, volume, recorded names, including those of symbolic links, - lookup count, guest-visible inode number, and object identity of device, - inode number, and file type; -- each handle's ID, node, open flags, object identity, and directory listing - with its cookies; -- the next node and handle IDs; and -- the fixed device shape, access mode, ownership mode, and root identity, with - the access policy of a share that has allowed or writable paths, or, for an - aggregate, each child's name, kind, mode, root identity, and policy. - -Device-private state names objects relative to the export root and never -records a host path. The snapshot's machine contract records the host side -instead: the guest target, access mode, tag, queue counts, direct I/O, cache -lifetimes, canonical host path, policy paths, ownership mode, and, for an -aggregate, each child's name, canonical host path, kind, mode, policy, and root -identity. Its [attachment inventory](host-attachment-model.md) binds -`fs:microvm0`, with the `live-revalidate` reconnect policy, to the root's -identity or, for an aggregate, to a digest of its children's names and root -identities in order, so a restore can neither drop, add, nor reorder a child. -The guest binds each child at a target that OpenVMM does not record, so NVX -names each child after its index and a digest of its guest target: the snapshot -then pins each target, and a restore that requests another target fails before -the guest runs. - -#### Snapshot capture - -The device takes part in the [capture boundary](snapshot-and-restore.md#capture-boundary) -as follows: - -1. When capture gates host input, each worker finishes its current request, - publishes the reply, and takes no further request. Descriptors that no - worker has taken stay on the available ring, in guest memory and the saved - queue progress, for the restored device to take. -2. After the vCPUs stop at the boundary, the queues stop, and the save requires - that no request remain in flight. -3. HostFs inspects every handle through its descriptor and resolves every - recorded name beneath the export root again without following links, - checking that all names of a node reach one object. Capture fails while the - device tracks a node that has no name left, such as a removed file that the - guest holds open, or a name that is gone, hidden, crosses a symbolic link, - or reaches another object than the node's other names. It also fails when - the directory listings of all handles exceed 16,384 entries or 4 MiB of - names. -4. Capture does not flush the host files. Data that the guest wrote before the - boundary is already in them and reaches stable storage on the host's - schedule unless the guest synced it. - -Capture and restore inspect and reopen objects as OpenVMM's user, even in -caller mode, so they fail while the guest holds an object that only its caller -can reach or reopen. - -#### Snapshot restore - -A restore rebuilds the device's side of the session before the guest can use -it: - -1. Before the guest runs, the launcher supplies the attachment again. It must - reproduce the snapshot's target, mode, policy, ownership mode, canonical host - paths, and root identities, and, for an aggregate, the same children with - the same names, kinds, and modes in the same order. Only a dormant slot may - instead gain a new attachment. -2. Before the transport changes any state, the device validates the saved - state: a known schema version, the fixed device shape, a FUSE session that - the profile would negotiate at FUSE 7.31 or newer, tables within their - bounds, unique IDs below the next IDs, and writable handles only where the - share or child is read-write and every name of the node is writable. -3. When the restored device first starts, before it services any request, - HostFs rebuilds the node table with the saved node IDs and lookup counts. It - resolves every recorded name beneath the supplied attachment without - following links and requires the same object, by device, inode number, and - file type, with the same guest-visible inode number. -4. HostFs reopens every handle with its saved open flags, except that it drops - `O_CREAT`, `O_EXCL`, and `O_TRUNC`, so a restore never creates or truncates - a file. It verifies the identity and kind of each reopened object and - restores the handle's directory listing. -5. HostFs restores the negotiated FUSE session without a new `INIT`. - -The guest then continues with its mount, working directories, open -descriptors, mappings, and directory listings, without remounting. Because it -caches no entries, attributes, or file data, its next request observes the -host as it is at restore time: the restore checks only the objects that the -guest held, and their contents and every other name may have changed since -capture. A listing in progress continues from the captured listing. Identity, -not content, binds a restore: a copy of the export, even with identical -contents, fails it, and so does a held file that host software replaced by -renaming a new file over its name. Every restore of one snapshot attaches the -same live host objects; unlike paired scratch, the share is not copied. Native -file descriptors and Windows handles are not serialized. - -#### Dormant slot and saved-state versions - -A dormant capture saves explicit unattached state, with no session, nodes, -handles, root identity, or policy. It requires that the guest have no FUSE -session on the slot, so the tag must not be mounted; unmounting ends the -session. A dormant snapshot may restore with no attachment or bind one new -`--mount` of a directory, but not an aggregate. The resumed guest then mounts -tag `microvm` explicitly, because the cold-boot mount hook has already run. -Snapshots without this capability cannot add an attachment, and active state -cannot restore into a dormant slot. - -Capture records the highest saved-state version that applies to the -attachment. Each version marks state that earlier releases cannot enforce, so -they reject the snapshot rather than restore a weaker attachment. A restore -also accepts version 4 from earlier releases. - -| Version | Records | -| ---: | --- | -| 5 | A dormant slot, or a single share with VMM ownership and no allowed or writable paths | -| 6 | Caller ownership, so that a release that predates it cannot restore the attachment as the VMM | -| 7 | The complete access policy of a single share with allowed or writable paths | -| 8 | An aggregate's children and each node's volume; the synthetic root's listing cookies are positions in the recorded child list | -| 9 | An aggregate with a file child, whose kind it records, so that a release that predates file children cannot restore it as a directory | diff --git a/doc/design/remaining-production-work.md b/doc/design/remaining-production-work.md deleted file mode 100644 index 6182ff262..000000000 --- a/doc/design/remaining-production-work.md +++ /dev/null @@ -1,67 +0,0 @@ -# Remaining production work - -[Design index](../design.md) - -The following items are **Proposed**, beyond the low-level implementation: - -| Area | Remaining design or delivery work | -| --- | --- | -| Launch configuration and control | Implement the replaceable region, consumed-section validation, a typed and versioned workload RPC over the authenticated control-console broker, cancellation and restore rehandshake semantics, and a node-version compatibility policy. The broker, its bounded framing, and the managed lifecycle subset are implemented. | -| Production agent | Complete OCI policy, atomic workload placement, orphan supervision, typed errors, PTY/log handling, and capture-safe worker rendezvous. Measure agent CPU/memory reserves. | -| Image conversion | Build the curated-base converter, metadata allowlist, geometry classes, admission validation, and tests for crafted overlay/security attributes. | -| Snapshot orchestration | Integrate trusted placeholder-based platform builds, per-runtime warm shims, checkpoint requests, private scratch acquisition, and capture failure reporting with the public sandbox workflow. | -| Artifact lifecycle | Define tenant-aware placement, retention, refcounts, quotas, distribution, and GC for layers, RAM, state, and paired scratch; invalidate or recertify templates on rollout. | -| Sharing isolation | Validate platform-template provenance and workload memory/device restrictions, per-tenant memory budgets, host deduplication policy, and ReFS physical-frame separation. | -| Metrics and performance | Define readiness and billing metrics; measure launch-to-workload-runnable latency, conversion/cache misses, scratch copy cost, cold versus warm page caches, and activation costs independently. | -| Networking | Design any external endpoint rebinding or tenant resolver/routing extension without silently changing the current portable profile or saved policy. Live flows must reconnect. | - -The sandbox proposal's roughly 10-ms restore goal is a performance target, not -a guarantee established by this document. Report the precise readiness -boundary and backend for every measurement; see [Benchmarks](../benchmarks.md). - -## Workload compatibility and volumes (Proposed) - -Systemd-as-entrypoint images need an explicit, slower compatibility profile: -appropriate container environment, read-only system views, private devices, -mount propagation, an agent-owned parent cgroup with a delegated subtree, and -the required stop signal. Such images may need `CAP_MKNOD` together with a -cgroup device filter. They are not covered by the current drop-all-capabilities -bootstrap, and its private mount policy must not be described as that profile. - -Azure Files, secret/content volumes, and other optional mounts remain undesigned. -Guest-side CIFS, a scratch-backed temporary directory, or an explicit small -content attachment are possible directions, not supported extra device slots -in the fixed ABI. Managed executions deliver bounded environments through a -sealed anonymous file inside the guest; snapshot-time launch configuration and -secret-volume delivery remain distinct proposed features. Scratch cannot -currently grow online; capacity exhaustion and whole-file copy-up remain -workload concerns. - -## Distribution extensions (Proposed) - -P2P distribution can populate the same immutable node blob cache without -changing the guest's filesystem view. Demand fetching missing blob ranges -behind virtio-blk is another possible extension for large images, but it must -define bounded I/O failure, integrity, and snapshot eligibility before replacing -the current cached regular-file backend. Neither facility is implemented. -Adopt them based on measured cache misses and first-run latency, not merely -because a format permits them. - -## Alternatives considered - -| Alternative | Design tradeoff | -| --- | --- | -| Rootfs over virtio-fs/virtiofsd or 9p | Unnecessary per-operation filesystem transport for fixed cold-plugged layers. The existing optional HostFs export serves a different purpose. | -| Nydus/RAFS and an in-guest fetcher | Lazy distribution may be valuable, but an extra fetcher/daemon is not needed for fully staged EROFS blobs. Host-side distribution can evolve independently. | -| virtio-pmem/DAX or general-purpose phram layers | Requires a new memory/device and sharing contract; direct shared-page mappings and image-sized metadata costs need explicit justification. | -| tmpfs scratch | Competes with workload RAM and turns storage growth into memory pressure instead of an independent storage limit. | -| One merged blob or partitioned disk per image combination | Loses independent caching and reuse of distro/runtime layers; adds layout parsing without a need for arbitrary device discovery. | -| Guest device-mapper snapshots | Adds a second copy-on-write mechanism when overlayfs already supplies the writable filesystem view. | -| SquashFS or uncompressed EROFS | EROFS is the selected compressed read-only and overlay-metadata contract. Another format needs a concrete benefit; uncompressed images are primarily relevant to a future DAX path. | -| A general init system in the outer guest | Adds services and boot work that a single workload's supervisor does not need. A systemd workload inside its own namespace is a separate compatibility question. | -| Another vsock or multiport control device | The dedicated single-port control console reuses existing queue and saved-state machinery and already carries the authenticated broker; the production RPC still has to be built on top of it. | -| Configuration in the kernel command line | Size and secrecy limits, and no per-restore replacement of the captured effective command line. | -| Configuration baked into layers, scratch, or initramfs | Forces filesystem access or per-launch image writes and can leave stale configuration in captured RAM. The launch region keeps configuration separate from reusable artifacts. | -| Full layer hashing inside the guest | Reads whole images on the launch path. Current integrity checks belong to host restore; UUID checks in the guest detect attachment mistakes only. | -| Sharing one warm-workload memory file across tenants | COW protects artifact contents, not physical-page isolation. Tenant-scoped instantiation and its memory cost need explicit policy. | -| Replacing the agent with the workload via `exec` | Removes supervision, orphan reaping, and control service. The workload must be a child. | diff --git a/doc/design/sandbox-filesystem-and-agent-architecture.md b/doc/design/sandbox-filesystem-and-agent-architecture.md deleted file mode 100644 index b7a4ebc13..000000000 --- a/doc/design/sandbox-filesystem-and-agent-architecture.md +++ /dev/null @@ -1,495 +0,0 @@ -# Sandbox filesystem and agent architecture - -[Design index](../design.md) - -The sandbox specialization runs exactly one workload container per microVM. -Its lifetime, resource envelope, and network identity belong to that workload, -so it does not need pod infrastructure, a pause container, dynamic rootfs -injection, or a sequence of host RPCs to create additional containers. The -agent remains outside the workload's namespaces and supervises it from the -initramfs. This specialization is for non-confidential, single-workload -sandboxes, not multi-container groups; the host is trusted with image content -and guest memory. - -The implemented foundation is a cold-filesystem bootstrap, low-level snapshot -primitives, and an authenticated control channel with a bounded managed -lifecycle. The production conversion service, replaceable launch -configuration, Rust agent, and production runtime protocol described below -are **Proposed**. They must not be inferred from the presence of block devices -or snapshot-tier metadata alone. - -Two host components launch sandboxes on this foundation. The `nvx sandbox` -command attaches image layers and scratch, from which the guest assembles the -[layered root](#implemented-filesystem-bootstrap). The -[`aci_edge_sandboxes` crate](../../aci_edge_sandboxes/README.md) exposes a -provision, start, exec, stop, and deprovision API. Its default backend launches -OpenVMM directly through the [`nvx-rs` crate](../../nvx-rs/README.md), without the -Python NVX tooling, with the managed lifecycle and no -sandbox blocks, so each workload runs in the Alpine initramfs root, as -[Control protocol and checkpoint handoff](#control-protocol-and-checkpoint-handoff) -describes. Its optional agent backend delegates the lifecycle to a separately -supplied native library and an image-backed edge guest that NVX does not build. - -## Implemented filesystem bootstrap - -The public `nvx sandbox` command accepts one to three role-bearing EROFS lower -images, a preformatted ext4 scratch image, any number of -[live host shares](#live-host-shares) that its kernel command line holds, an -absolute entrypoint, and -individual argument tokens. It supplies non-secret kernel-command-line -configuration for one-shot runs. Managed execution carries bounded arguments -and a per-execution environment over the authenticated control channel; -one-shot environment variables, secrets, and sandbox snapshot orchestration -are not supported by this command. Lower-level OpenVMM capture and restore do -support sandbox blocks. See [Run](../run.md#experimental-single-workload-sandbox). - -The required kernel facilities are already enabled in the NVX microVM kernel -configuration: virtio-blk, virtio-fs, EROFS with compression and xattrs, -overlayfs, cgroup v2, memory and process controllers, namespaces, BPF system -calls, and cgroup BPF programs. Kernel support for a device filter does not -mean the current agent installs one. - -The guest init agent performs the assembly: - -1. mount runtime tmpfs and cgroup2, create sibling `agent` and `container` - cgroups, and move the supervisor into the agent cgroup; -2. resolve each role's MMIO address through the platform resources in - `/proc/iomem` and the virtio block-device sysfs tree, rather than assuming - a `/dev/vdX` name or `virtioN` enumeration order; -3. check an optional expected sector count, flush cached block-device buffers, - mount each lower layer read-only as EROFS, and check its supplied UUID; -4. resolve and flush scratch, mount it as ext4 with `rw,nosuid,nodev`, and - create its `upper` and `work` directories; -5. mount overlayfs with top-first lower layers and `metacopy=on,xino=on`; -6. verify the workload identity in the assembled root, mount each live share - at its target inside that root, and write the workload machine ID to a - runtime-tmpfs file; -7. start a child behind a FIFO barrier and place that child in the workload - cgroup before releasing it; -8. wait for the child, unmount the shares in reverse order and then the - overlay, layers, and scratch, and return its status through the guest exit - helper. - -In a managed sandbox, the managed agent repeats step 7 for each requested -workload, and the teardown in step 8 follows `stop`. - -The assembled view is: - -```text -/run/nvx/layers/custom (optional EROFS) --+ -/run/nvx/layers/runtime (optional EROFS) --+--> lowerdir, top first -/run/nvx/layers/distro (optional EROFS) --+ -/run/nvx/scratch/upper (ext4) -----------> upperdir -/run/nvx/scratch/work (same ext4) ------> workdir - overlay --> /run/nvx/rootfs -virtio-fs tag microvm (optional share) ---> /run/nvx/rootfs/TARGET - or an aggregate ---> /run/nvx/shares, each child bound at - /run/nvx/rootfs/TARGET -``` - -At least one lower role is required by the bootstrap; distro plus runtime is -the intended curated-image shape, not a requirement that all three lower -slots be populated. The initramfs remains the supervisor's root and is not -another container lower layer. - -The container launch helper releases the barrier into private mount, PID, and -UTS namespaces. The container entry helper makes mounts -private, creates private proc, read-only sysfs, `/dev`, devpts, and shared-memory -mounts, binds the workload machine ID read-only, and enters the overlay with -`chroot`. It clears supplementary groups and all capability sets and enables -`no_new_privs`. It does not `pivot_root` away from the initramfs `rootfs`, and -the outer supervisor does not replace itself with the workload. - -The current agent sets `memory.low` to 16 MiB by default and accepts optional -workload `memory.max` and `pids.max`. This is not the stronger production -resource-reservation contract below. FIFO-gated cgroup placement is also not -`clone3(CLONE_INTO_CGROUP)`. The shell supervisor and textual errors remain an -experimental bootstrap, not a complete OCI runtime, typed RPC service, or -systemd-container profile. Workloads do run under one host-owned non-root -numeric identity fixed at initial boot. The agent requires an exact user and -primary-group match in the assembled root, clears supplementary groups and all -capability sets, and rejects an unavailable identity before starting the -workload. - -### Live host shares - -A live share exposes a host directory, such as a source checkout or a tool -cache, or a single host file, such as a settings file, to the workload without -copying it into an image: edits are visible in both directions without staging -or copy-back. - -The microVM has one virtio-fs slot, tag `microvm`. One `--mount` of a directory -attaches a HostFs server to it. Several, or a shared file, attach an aggregate -instead, whose children, named -by their index and a digest of their guest target, are independent HostFs -volumes, so the shares keep independent `ro` or `rw` modes and access -policies. A file child serves only its file, never the file's directory or -siblings, because the guest mounts the device's root, which is a directory. -On every request, whichever guest -mount or link reaches the share, the host enforces the share's mode and its -denied, allowed, and writable paths, and accesses host files as the identity -that `--mount-owner` selects. Guest mount flags are therefore not a security -boundary, and the access policy adds no guest configuration. Before OpenVMM -starts, NVX rejects policy paths outside their share or for a shared file, and -guest targets or host paths that equal or contain one another, because one -share could otherwise hide another or reach its files under a different policy, -and it rejects shares whose kernel command-line tokens exceed the sandbox's -budget. -[Machine and device ABI](machine-and-device-abi.md#filesystem) defines the -device contract. - -OpenVMM appends one `virtfs_dir=`, `virtfs_tag=`, `virtfs_mode=` triplet for -its share, and `virtfs_aggregate=1` for an aggregate, which it mounts -read-write when any child is. NVX then adds one -`nvx_share=NAME,TARGET,MODE[,file]` token per child, where `file` marks a -shared file. Because a child's name holds a digest of its target, a -snapshot pins each target, and a restore that requests another target fails -before the guest runs. The init agent parses every token, and creates every -target, before it mounts anything, so a malformed bootstrap mounts nothing. It -creates each target inside the container root one component at a time and -refuses a path that crosses a symbolic link, so an image layer cannot redirect -a share outside that root. The last component of a shared file's target is a -regular file, which the agent creates when the container root lacks it, and of -any other target a directory. It also refuses a repeated child, overlapping -targets, and targets that the container entry helper later mounts or binds -over, such as `/etc/machine-id`, where the runtime would hide the share or -write into it. A single share -is mounted at its target with its mode and `nosuid,nodev`. An aggregate is -mounted at `/run/nvx/shares`, outside the container root, where its root lists -the children and only root can enter, and each child is bound at its target -with its mode and `nosuid,nodev`. The workload's private mount namespace -inherits the mounts, and teardown unmounts the binds in reverse order before -the aggregate. Any -refusal or mount failure aborts the sandbox with status 125 instead of -starting the workload without its shares. A managed sandbox records its -shares, including whether each is a file, when it is provisioned, and -reattaches them on every start. -[Live host shares](../run.md#live-host-shares) describes -the options and their rules. - -## Image preparation and distribution (Proposed) - -Image conversion belongs off the sandbox start path, in a Linux control-plane -service that can run `mkfs.erofs`. Registry credentials stay in that service, -not on the node's launch path or inside the guest. The host resolves image -defaults and sandbox overrides once; the agent should not implement OCI image -configuration merge semantics or read configuration from a mounted layer. - -The intended artifact ownership is: - -| Artifact | Contents | Delivery and lifetime | -| --- | --- | --- | -| Kernel and initramfs | Platform kernel, agent, and minimal userland | Versioned node deployment, present before sandbox creation | -| `distro` | Curated base OS | Immutable content-addressed node blob cache | -| `runtime` | Language runtime and common libraries | Immutable cache, shared across images using that runtime | -| `custom` | Customer-specific additions, replacements, and deletions | Optional immutable cache entry, scoped to its image | -| Scratch | Writable ext4 upper and work directories | Private node-local file, drawn from a prepared pool or template | - -Curated bases can be recognized by exact OCI layer-digest prefix matching. -The deepest matching base determines the reusable distro/runtime split. An -unrecognized image can be flattened into one custom lower layer; it loses -curated-layer sharing but can still use a compatible image-independent -platform template. Identical file contents with different layer digests do not -establish a prefix match. - -Separate blobs preserve reuse of the common distro and runtime across many -customer images. Pre-merging all combinations, or packaging every combination -as one partitioned disk, would make each combination a separate cache object. -Compressed EROFS reduces distribution size and the host's cached image bytes; -decompression is performed by the guest kernel as blocks are read. A FUSE -daemon and guest-side lazy-pull agent are not required for these rootfs layers. -The optional HostFs devices remain useful for -[live host shares](#live-host-shares) and are a separate feature, not the -container image-delivery path. - -The converter must synthesize OCI deletions as overlay whiteouts and opaque -directory markers. It must apply a deny-by-default metadata policy to every -layer: never copy arbitrary `trusted.overlay.*` or `user.overlay.*` attributes, -and allow and normalize file capabilities, SELinux labels, and ACLs only under -an explicit policy. Device inodes, including whiteouts, need the same treatment. -The current bootstrap mounts supplied EROFS images; it is not this sanitizing -conversion service. - -This trust requirement matters because `metacopy=on` interprets overlay -metadata from lower layers. Metadata-only changes can avoid copying file -contents, but a data write can still copy an entire lower file into scratch. -Scratch capacity must account for that amplification. `xino=on` improves inode -identity but can fall back when the underlying inode encoding overflows; the -chosen filesystem and kernel combination needs validation. - -Scratch should be prepared off the launch path and acquired without `mkfs` -during startup. For the current snapshot-capable microVM it must be a regular -raw file containing ext4, not a VHD/VHDX attachment. A pool or a filesystem -clone can accelerate acquisition, but the VMM also supports independent copies; -block-clone support is not a prerequisite for running NVX. Scratch consumes -host storage rather than guest RAM and can fail with `ENOSPC` independently of -guest memory pressure. Runtime tmpfs is still appropriate for small agent -state, not the workload's general writable layer. - -Blob garbage collection must retain references from both live sandboxes and -snapshots. Paired scratch, state, and RAM need snapshot-scoped retention and -placement. Kernel/agent rollout must rebuild or explicitly recertify templates: -restoring guest RAM restores the captured agent build, not the node's newly -installed initramfs. These cache, pool, and rollout services are host -orchestration responsibilities, not implemented VMM artifact management. - -## Replaceable configuration region (Proposed) - -The production launch contract should use one bounded, versioned, -VMM-populated memory region. Only its location and ordinary invariant kernel -parameters belong on the command line. The command line is size-limited, -readable through `/proc/cmdline`, and captured in RAM; it cannot safely carry -secrets or values that change between clones. - -The configuration is fully resolved by the host and separated by consumption: - -| Section | Contents | First consumed | -| --- | --- | --- | -| Invariants | Role/MMIO mapping, layer geometry, and platform-owned guest network identity such as MAC, address, prefix, and MTU | Before the platform snapshot point | -| Image binding | Expected EROFS UUID for each attached lower role | After the platform snapshot point, before workload launch | -| Sandbox | Entrypoint, arguments, environment and secrets, workdir, identity, capability policy, and tenant resolver/routing settings | After the platform snapshot point | - -Only unconsumed configuration may be replaced. Platform clones may receive -new image binding and sandbox values; workload-start and instance-checkpoint -restores have already consumed them and must not pretend that rewriting a -region changes mounted filesystems or process state. Those later tiers accept -new work through a runtime protocol, not replacement OCI configuration. - -The host must validate the independent launch configuration against the saved -contract before entering a vCPU, including canonical digests for consumed -sections. The agent must validate the header, version, bounds, and invariants -again before using the payload. An EROFS UUID is only an attachment mix-up -check, not a content-integrity proof. Current OpenVMM restore verifies SHA-256 -for consumed read-only layers; a future cache-admission optimization needs an -explicit trusted artifact contract rather than replacing that check with UUIDs. - -The region must be separate from both captured RAM and restore-time expansion -backing, omitted from the Linux direct e820 RAM map and `memory.bin`, and populated -before every launch. Its placement must not overlap any capacity reservation -or device. Restoring RAM must never overwrite newly supplied configuration. -A header with magic, version, bounded length, and corruption detection does not -by itself provide authenticity or secrecy. - -The current platform command-line validator recognizes the exact prospective -token `nvx_config=0xd0010000,65536`; that allowlist entry does not allocate a -region or implement the payload protocol. The current bootstrap still reads -non-secret command-line tokens. The former standalone `phram` mechanism is not -an alternative configuration or layer carrier in this microVM profile. - -Launch metadata must remain fresh even when all configuration sections have -been consumed. A future channel epoch, entropy seed, and clock sample therefore -cannot be frozen by an OCI-section digest. Today restore detection and entropy -repair use the process generation ID and -[portb restore packet](snapshot-and-restore.md#time-and-entropy), -not an implemented configuration-region epoch. Zeroing a region after use is -defense in depth: it does not erase copies in agent buffers or workload memory. -The region and raw-memory devices must never be exposed in the workload's -mount namespace or `/dev`. - -The proposed network split also needs an explicit compatibility contract. -Current portable networking saves the guest identity, bootstrap command line, -and egress policy; it does not implement per-launch tenant DNS/routing swaps or -the old HCN L2Bridge/AF_XDP endpoint translation and readiness handshake. - -## Production agent and launch sequence (Proposed) - -The intended production agent is a small static Rust binary running as PID 1 -from the initramfs. It owns pseudo-filesystem setup, filesystem assembly, -network programming, workload construction, orphan reaping, stdio, and runtime -control. It remains alive after starting the workload; `exec` of the workload -from the supervisor would destroy those responsibilities. - -The cold launch should require no configuration RPC round trip: - -1. the host stages layers and private scratch, constructs all devices, prepares - the network backend, and writes the launch configuration; -2. the VMM boots the kernel and initramfs through Linux direct MP-table mode; -3. the agent sets up pseudo-filesystems and sibling cgroups, consumes only - invariants, and configures the platform-owned guest network; -4. a trusted platform-template build captures here, before image binding, - tenant configuration, or scratch mounts; -5. a cold launch or restored template validates the applicable configuration, - performs required restore repair, discards stale block buffers, and mounts - EROFS, ext4, and the overlay; -6. the agent creates the workload directly in its cgroup and private namespaces, - applies its resolved runtime policy, and starts it under supervision. - -The platform build must attach deterministic non-tenant placeholder layers. -Linux can probe block contents before PID 1 runs, so merely avoiding mounts -does not prove a snapshot contains no image bytes. Restored layers must have -the captured geometry, and cached placeholder blocks must be invalidated -before mounting replacements. Current host validation records platform layers -as unbound but cannot prove which bytes arbitrary guest code read before capture. - -A complete runtime must implement more than the current bootstrap: atomic -`clone3(CLONE_INTO_CGROUP)` placement, the required namespace and mount policy, -masked/read-only paths, user/group and supplementary-group handling, capability -sets, seccomp, rlimits, terminal allocation, signals, and reliable child/orphan -supervision. The root transition must preserve the outer agent's initramfs -view; it cannot assume `pivot_root` works directly from initramfs `rootfs`. - -Production resource policy should give the agent a measured memory reserve -and CPU weight in a sibling cgroup, with workload `memory.max`, -`memory.oom.group=1`, and appropriate process limits. `memory.min` protection -and workload limits must be sized together with VM RAM; the current -`memory.low` setting alone is not a hard survival guarantee. A cgroup device -filter is needed for profiles that retain `CAP_MKNOD`. The workload must not -be able to escape the agent-owned cgroup or undo its freeze; any delegated -subtree must remain below that boundary. - -## Control protocol and checkpoint handoff - -The implemented managed runtime uses the dedicated control virtio-console and -its authenticated broker. Boot diagnostics and kernel `printk` stay on the -existing consoles; framed control traffic does not share an unstructured byte -stream with them. OpenVMM owns the bounded outer framing, same-user local -endpoint authorization, capability authentication, reconnect epochs, and -receive-credit backpressure. The current guest protocol provides readiness, -sequential command execution with bounded arguments, per-execution -environments and output, separate stdout/stderr, timeout and exit categories, -cancellation, and graceful VM shutdown. The control device is never exposed -inside the workload namespaces. - -While a workload runs, the agent keeps reading the control console. A `CANCEL` -request that carries the workload's request ID kills the workload, and the exit -report then uses the `cancelled` category with status 137. Without sandbox -layers, the agent runs each workload directly in the guest's root file system, -inside the `nvx-exec` cgroup; termination writes that cgroup's `cgroup.kill`, -which also reaches processes in other sessions or process groups. When the -workload's first process exits, the agent kills what remains and reaps the -orphans it inherits as PID 1, so no workload process outlives its exec. -It reports the outcome only after the cgroup's `pids.current` reads zero, that -is, once every process of the workload has exited and been reaped. The agent -enables the `pids` controller for this check, because the controller charges a -process until the process is reaped. `cgroup.events` would not do: it stops -counting a killed process as populated before the process becomes a zombie, so -the zombie could remain in `/proc` for the next workload to find. A failed -kill, an unreadable or malformed `pids.current`, or a settlement timeout reports -`containment-failed` instead. Every subsequent direct execution also reaps what -an earlier workload left and then requires verified emptiness, so a -control-session reset cannot bypass a failed containment check. - -An `EXEC` payload holds a 32-bit timeout, a 16-bit argument count, a reserved -16-bit field, and the length-prefixed arguments. A reserved field of 1 -announces an extended header: a 16-bit flag field, a 16-bit environment entry -count, and a 32-bit working-directory length. With flag bit 0, an absolute -working directory of at most 4096 bytes follows the arguments. With flag bit 1, -up to 256 environment entries come last, each a 32-bit length and a -`KEY=VALUE` string of at most 4096 bytes with a distinct, non-empty key, and -replace the workload's environment; flag bit 2, which requires bit 1, layers -them over the default environment instead, each entry replacing the default -variable of the same name. Without bit 1, the entry count must be zero. The -agent passes these fields through a sealed anonymous file to a launch -helper, a copy of the agent that runs once `setpriv` has applied the workload's -identity, inside the container root with sandbox layers. The helper enters the -working directory, or `/` without one, points `PWD` at it unless the request -replaces the environment, applies any environment, and executes the workload; -if it cannot enter the directory or set `PWD`, it writes a diagnostic to the -workload's standard error and exits with status 125. Without sandbox layers, -the workload's child process enters the directory itself, under the workload's -user and group IDs with no supplementary groups or effective capabilities, -before `setpriv` makes that identity permanent. `setpriv` and the helper keep -that directory instead of looking its path up again, so the workload starts in -the directory that was checked even if the path is renamed, replaced, or -retargeted in the meantime. Unless the request replaces the environment, which -controls `PWD` itself, the child also points `PWD` at the directory, and the -agent refuses the request as `launch-failed` if it cannot. If the workload -cannot enter the directory, the agent writes a diagnostic to the workload's -standard error and refuses the request with the `cwd-failed` category and the -error number as status, so nothing runs in another directory. - -Without sandbox layers, the agent maps host directories for workloads from -one aggregate virtio-fs export. The host exports the outermost mapped -directories and the parents of the outermost mapped files as numbered children -of the export, which the init mounts at `/run/nvx/hostfs/root`, and enforces -each mapping's access itself: a child is read-write only if it holds a -read-write mapping, OpenVMM limits writes to its read-write mappings, and the -parent of mapped files hides everything else. Neither a whole volume nor a file -directly in a volume's root, whose parent would be that root, is mapped. Before -it accepts control -traffic, the agent makes `/run/nvx/hostfs` root-only. The kernel command line -announces the number of mappings with an `nvx_maps=COUNT` token, and the host -then sends the mapping table in `MAPS` requests. Each request carries the index -of its first entry and its number of entries, and each entry its flags (bit 0 -selects read-only), the lengths of its source and target, the source, a path -relative to the export such as `0/src`, and the absolute target. The agent -accepts entries only in order and only up to the announced count, checks a -whole request before it mounts anything, bind-mounts each source at its -target, remounting the bind `nosuid,nodev` and, for a read-only entry, -read-only, and answers `READY`, or `ERROR` with the `mapping-failed` category -and the error number. Until it has mounted every announced entry, it refuses -`EXEC` with the `mappings-incomplete` category, and a failed mount keeps it -refusing. Without an `nvx_maps=` token, it refuses `MAPS` as -`invalid-request`, and it ignores the `nvx_map=` tokens of older hosts. -OpenVMM's deny list hides denied paths. A -`nvx_workload_account=create` token lets the managed init create an account for -a host-selected non-root identity that the image lacks, which Linux hosts use -to run workloads under the host user's IDs. An image account that already uses -the UID but names another primary group or lacks a usable home is replaced. -A `CANCEL` for a workload that already finished is ignored. Any other request -during an execution is refused as `busy`. A reset means the host session that -could observe the workload is gone: the agent kills the workload, sends nothing -more for it, and acknowledges the new epoch, so the next session does not wait -for an abandoned workload. - -A host cannot tell from a successful readiness probe whether the image enforces -the behaviors it depends on: an older guest boots and answers, but ignores the -mapping table and `CANCEL` requests. A `FEATURES` request, which carries no -payload, therefore asks which control behaviors the image provides. The agent -answers `READY` with a four-byte little-endian bit mask: `CANCEL` (bit 0), -`WORKLOAD_ACCOUNT` (bit 2, provided by the managed init and reported by the -agent, because both ship in one initramfs), `EXEC_CGROUP` (bit 3), -`EXEC_ENVIRONMENT` (bit 4, which applies an explicit environment to each -execution, either replacing or layering over the bootstrap environment), -`EXEC_CWD` (bit 5, the `cwd-failed` refusal of a working directory that the -workload cannot enter), and `HOST_MAPPING_TABLE` (bit 6, the `MAPS` requests -above). Bit 1 announced the bind mounts that `nvx_map=` kernel tokens listed in -earlier images; it stays unset, so hosts that still send those tokens refuse -current images. Without sandbox layers all six features are provided; with -them, only `CANCEL`, `WORKLOAD_ACCOUNT`, and `EXEC_ENVIRONMENT`. -An agent that predates the request refuses it as `unsupported-operation`, which -a host reads as no features, so a host terminates a guest that lacks a feature -it needs instead of running workloads without the policy it asked for. During -an execution the request is refused as `busy`. A feature bit is added together -with the behavior it names; hosts ignore bits they do not know and trailing -bytes of the answer. - -The remaining production operation families are: - -| Operation | Purpose | -| --- | --- | -| `Ready`, `RestoreHello` | Establish readiness, protocol version, and the current launch identity | -| `Bootstrap` | Explicit configuration when a workflow cannot use the launch region | -| `InteractiveShell` | PTY-backed interactive sessions, including cancellation and resize | -| `Signal`, `ContainerExited` | Out-of-band signal control and OOM reporting | -| `Probe` | Execute health checks | -| `PrepareSnapshot`, `PostRestore`, `Checkpoint` | Agent-coordinated capture and restore hooks | -| `Shutdown` | Graceful workload and VM termination | - -Messages need bounded framing, stream and request identifiers, stable typed -errors, backpressure, and launch-scoped identity. On restore the agent must -discard partial decoder state and requests from the old host connection, -reestablish the session with `RestoreHello`, and fail non-replayable operations -rather than silently executing them twice. Protocol versioning is tied to the -node-deployed agent; transport compatibility alone is not RPC compatibility. - -The proposed workload-facing checkpoint interface is an opt-in, bind-mounted -`SOCK_SEQPACKET` socket, such as `/dev/nvx/checkpoint`, with per-message -`SCM_CREDENTIALS`. The agent owns the PMIO write; the workload does not receive -`CAP_SYS_RAWIO`, `/dev/port`, or unrestricted port-I/O access. Capture requests -and retained artifacts need agent and host rate limits. - -A warm shim requests capture after runtime initialization and receives its -next work item after restore. Python, Node, and Java need runtime-specific -hooks; kernel CRNG reseeding cannot reset their userspace RNGs, caches, or -external connections. Blocking the requesting thread on a socket is not a -barrier for its peers. A cloneable warm point must be single-threaded or hold -all peers behind a runtime-owned barrier until repair is complete. The current -workload-start helper requires a runtime post-restore hook, but there is no -production shim or work-item handoff protocol yet. - -The replay contract is explicit: do not capture live external connections or -state that cannot be safely reused, do not repeat irreversible side effects, -and repair runtime randomness before accepting work. An instance checkpoint -is a single continuation; a workload-start snapshot is a cloneable starting -point, not transparent checkpointing of arbitrary requests. diff --git a/doc/design/snapshot-and-restore.md b/doc/design/snapshot-and-restore.md deleted file mode 100644 index 00f34dffb..000000000 --- a/doc/design/snapshot-and-restore.md +++ /dev/null @@ -1,587 +0,0 @@ -# Snapshot and restore - -[Design index](../design.md) - -## Tier contract - -Sandbox snapshots encode the tier, restore policy, and a consumed -configuration-section bitmask. The validator accepts only these combinations: - -| Tier | Restore policy | Scratch | Consumed sections | Intended author and sharing scope | -| --- | --- | --- | --- | --- | -| `platform` | `clone` | Fresh | Invariants only | Trusted platform build; cross-tenant only with the [pre-image-binding guarantees](sandbox-filesystem-and-agent-architecture.md#production-agent-and-launch-sequence-proposed) | -| `workload-start` | `clone` | Paired | Invariants, image binding, sandbox | Warm workload or runtime shim; tenant-scoped | -| `instance-checkpoint` | `resume` | Paired | Invariants, image binding, sandbox | One live instance; one claimed continuation | - -Tier metadata requires sandbox blocks; blockless snapshots do not declare it. -The saved host-owned `nvx_snapshot_tier=` token must agree with the manifest. -Platform lower layers have empty, unbound identities, whereas later tiers bind -consumed layers either to whole-file SHA-256 identities or to one explicit -caller-authenticated immutable storage generation. The bitmask establishes -when configuration is considered consumed; it is not an implementation of -replaceable payloads or per-section configuration-region validation. - -The platform point removes kernel and agent initialization from subsequent -launches. Workload-start removes runtime initialization as well, but contains -workload memory and often cached image data and secrets. Read-only layer files -remain external attachments; bytes read from them into guest RAM can still -appear in `memory.bin`. Paired scratch is an artifact in the snapshot directory, -not just a reference to the caller's original writable file. - -Sharing scope is a deployment requirement, not a tenant authorization feature -of the VMM. The snapshot controller enforces tier combinations, layer binding, -scratch pairing, and the single-use resume claim. Host artifact access control, -template provenance, and a trustworthy pre-capture guest determine whether -cross-tenant publication is permissible. - -## Capture boundary - -Generic host save and pulse-save/restore RPCs are deliberately unavailable for -the microVM profile. Capture is requested by the guest through PMIO `0x605` and -is coordinated as a bounded transaction. The destination is configured on the -command line or through the management RPC; it must not exist, and OpenVMM -creates file-backed RAM beside it when no memory backing file was supplied. -Capture with sandbox blocks requires -`--snapshot-tier platform|workload-start|instance-checkpoint`; inconsistent -tier, clone/resume, and fresh/paired scratch combinations are rejected. For -the default `sha256` block identity, capture and restore scan each bound block. -An immutable storage owner may instead select -`--snapshot-block-identity generation --snapshot-generation-id <32-HEX>` and -bind every consumed block to that same generation without reading the files -solely to recalculate a digest. This mode trusts the storage controller to keep -the generation immutable and bound to the exact files. For -paired scratch, the guest snapshot helper first freezes the workload cgroup -with a bounded wait, calls `sync`, and freezes the mounted filesystem. Its -fresh-scratch mode instead requires scratch to be unmounted. A rejected capture -thaws every guest-owned barrier and then releases the stall-detector -suppression ([snapshot agent](time-abi.md#snapshot-agent)); failure to thaw or -release terminates the VM. - -1. gate host input and defer completion of the snapshot-port write; -2. stop the vCPU at the I/O boundary while completing the write, so saved state - starts at the instruction immediately after `out`; -3. preflight the profile, destination, backend, external attachments, and - snapshot eligibility; a missing destination or rejected preflight releases - the boundary and lets the guest continue; -4. quiesce the remaining device workers and VM time in dependency order, - including bounded virtio-blk queue drain; -5. save the exact state-unit inventory, processor state, device-private state, - the [time contract and CPU profile record](time-abi.md#snapshot-manifest), - and memory; -6. copy and flush state, memory, and any paired scratch into a unique sibling - staging directory; -7. publish the directory with one no-replace rename; and -8. terminate the source worker and process after publication. - -```mermaid -%%{init: {"theme": "base", "themeVariables": {"background": "#ffffff"}}}%% -sequenceDiagram - box rgb(255, 255, 255) - participant Guest - participant Port as PMIO 0x605 - participant Worker as VM worker - participant Units as State units - participant Controller - participant Storage as Snapshot storage - end - - rect rgb(255, 255, 255) - opt Paired mounted scratch - Guest->>Guest: Freeze workload, sync, and freeze scratch - end - Guest->>Port: OUT snapshot request and scratch policy - Port->>Worker: Bounded asynchronous notification - Worker->>Units: Gate host input - Worker->>Worker: Stop vCPU at I/O boundary - Worker-->>Port: Complete OUT - Worker-->>Controller: Boundary ready - Controller->>Controller: Preflight destination and contract - alt No destination or rejected preflight - Controller->>Worker: Release boundary - Worker->>Units: Resume host input - Worker-->>Guest: Continue execution - else Capture accepted - Controller->>Worker: Quiesce for snapshot - Worker->>Units: Stop and save in dependency order - Units-->>Controller: State, inventory, time contract, and CPU profile - Controller->>Storage: Copy state, memory, and optional scratch - Controller->>Storage: Publish with no-replace rename - Controller->>Worker: Terminate committed source - end - end -``` - -State-unit transitions follow registered dependencies rather than one global -serial list. In the current worker, the partition depends on the device -prerequisite unit and VM time. Save quiesce stops in reverse dependency order; -restore and rollback start in dependency order. Device prerequisites and VM -time may transition in parallel. - -```mermaid -%%{init: {"theme": "base", "themeVariables": {"background": "#ffffff"}}}%% -flowchart LR - subgraph Capture["Capture and save"] - direction LR - Gate["Gate host input"] - Boundary["Stop vCPU at
PMIO boundary"] - StopPartition["Stop partition
state unit"] - StopDevices["Stop device
prerequisite units"] - StopTime["Stop VM time"] - Save["Save stopped
state inventory"] - - Gate --> Boundary --> StopPartition - StopPartition --> StopDevices --> Save - StopPartition --> StopTime --> Save - end - - subgraph Restart["Restore or rollback start"] - direction LR - RestoreState["Restore stopped state"] - StartDevices["Start device
prerequisite units"] - StartTime["Start VM time"] - StartPartition["Start partition
and vCPU"] - ResumeInput["Resume host input"] - - RestoreState --> StartDevices --> StartPartition - RestoreState --> StartTime --> StartPartition - StartPartition --> ResumeInput - end - - Save -. committed snapshot .-> RestoreState -``` - -The directory rename is the commit point. Before it, capture failure exposes no -final snapshot and resumes only when rollback is known to be valid. After it, -the source never resumes. This gives the guest an observable boundary: code -after the snapshot `out` runs once in each restored process and never in the -successfully captured source process. A capture driven through the management -RPC likewise halts the source at the committed boundary and terminates its -process, so clients observe the transport closing. - -Establishing the boundary is itself fallible. If host input cannot be gated, -the worker rolls the gate back and releases the write so the guest continues; -if that rollback is uncertain, the worker terminates. Once vCPU stopping has -begun, a boundary failure is terminal: host input stays gated and the VM is -torn down rather than resumed. The post-restore acknowledgment boundary -follows the same rule. Teardown still depends on the affected backends -responding; there is no bounded shutdown deadline. - -While the worker holds the snapshot stop guard, its central RPC dispatcher -rejects management operations that could disturb the boundary, including -memory writes, resume/reset, device changes, and state dumps. Rejections are -immediate, not queued until capture finishes; infallible RPC forms report a -channel failure. Snapshot quiesce, rollback, boundary release, and memory reads -remain available. The exclusion applies at the worker boundary shared by every -management frontend. - -Guest and host barriers have separate jobs. The workload cgroup is frozen in -captured guest state, the VMM gates external input, and a future multithreaded -agent must park its own RPC/log workers at a capture-safe point. None of these -substitutes for the others. Device workers and guest kernel execution needed -for CPU/RAM repair must run during gated restore; the gate is not a promise -that all interrupts remain disabled. After repair, the helper acknowledges the -VMM gate before thawing scratch and then the workload. The current helper -polls cgroup freeze completion with a bounded loop; an event-driven production -agent and typed capture-failure reporting remain future work. - -## Artifact format and publication - -A published snapshot contains: - -```text -snapshot/ -|-- manifest.bin -|-- state.bin -|-- memory.bin -`-- scratch.img # paired microVM scratch only -``` - -```mermaid -%%{init: {"theme": "base", "themeVariables": {"background": "#ffffff"}}}%% -flowchart LR - State["Stopped VM state"] - Ram["Exact captured base-RAM
file handle"] - Scratch["Optional exact scratch
file handle"] - - subgraph Staging["Unique private sibling staging directory"] - direction TB - StateFile["state.bin
write and flush"] - MemoryFile["memory.bin
exact captured base RAM
from automatic or supplied backing"] - ScratchFile["scratch.img
copy, verify, and flush"] - ManifestFile["manifest.bin
record contract and lengths
write and flush before automatic RAM link"] - end - - SyncStaging["Flush staging directory"] - Rename["No-replace same-parent rename
publication commit point"] - Final["Final snapshot directory"] - SyncParent["Flush parent directory"] - Cleanup["Remove owned staging directory"] - - State --> StateFile --> ManifestFile - Ram --> MemoryFile --> ManifestFile - Scratch -. paired policy .-> ScratchFile --> ManifestFile - ManifestFile --> SyncStaging --> Rename --> Final --> SyncParent - Staging -. failure before commit .-> Cleanup -``` - -The snapshot publisher implements bounded decoding, exact-length publication, -artifact length checks, restrictive creation, flushing, unique staging paths, -and same-parent no-replace publication. Automatic OpenVMM-owned microVM base -RAM is flushed and then hard-linked into the staging directory from the exact -open handle once its shared mappings, state, and manifest are durable. The -link is reopened relative to the staging directory and must match the source's -file identity (device and inode on Linux, volume and file ID on Windows) and -length. Filesystems that cannot link the handle fall back to an independent -copy, and user-supplied backing always takes that copy path. On Linux the copy -clones extents when the filesystem supports reflinks and otherwise copies only -allocated ranges or nonzero data; on Windows it is a dense copy. Automatically -created RAM backing stays sparse on Linux and is dense on Windows, where sparse -files make copy-on-write faults slow. A pre-existing final destination is never -deleted or replaced. - -The source VM is terminal after an automatic RAM link commits. A failure after -creating the staging alias is rollback-safe only after the complete private -staging directory has been synchronously removed and its parent flushed. If -that proof fails, OpenVMM terminates the source rather than resuming it with a -surviving writable alias to the artifact. - -The manifest is authoritative for: - -- profile and ABI version; -- source hypervisor and architecture; -- captured RAM ranges and file offsets, plus any immutable RAM capacity, - 128-MiB block size, and canonical expansion ranges; -- processor topology, vCPU count, and every APIC identity; -- effective command line and digest; -- exact device order, stable IDs, state-unit names, MMIO/PMIO ranges, IRQs, - transport, feature masks, and queue limits; -- the time contract (the declared TSC and LAPIC rates, VP 0's TSC paired with - a host time sample, the host identities, and the generation counter) and the - CPU profile record (the pinned profile and the effective CPUID); -- required host attachments and their policies; -- block roles, access, geometry, layer identities, scratch policy, and paired - scratch restore materialization; -- snapshot tier, clone/resume policy, and consumed configuration sections; -- exact lengths of `state.bin` and `memory.bin`; and -- the exact length and either SHA-256 or storage-generation identity of paired - `scratch.img`. - -Snapshot paths and repeated fields are bounded. Restore rejects truncated, -oversized, malformed, wrong-type, path-escaping, symlinked, incompatible, -missing, extra, or reordered state before guest execution. Every snapshot uses -manifest version 6; any other version or format magic is rejected with -`E_SNAPSHOT_VERSION`, which asks for a recapture. The manifest does not store -or validate embedded checksums for `state.bin` or `memory.bin`; a same-length -change to either payload is therefore outside the validation contract. Paired -scratch is checked because it must match captured filesystem state. SHA-256 -paired scratch is checked byte-for-byte. -Generation-bound paired scratch is checked structurally and by exact size and -generation, so same-length byte verification remains the storage owner's -responsibility. Snapshot -directories rely on host access control, while -authenticated export or transport belongs outside the default local artifact -format. - -Restore opens the snapshot directory once and resolves manifest, state, memory, -paired scratch, and `resume.claim` relative to that directory handle. Windows -opens restore artifacts with read sharing only, rejects reparse points, checks -the file identity and end of file around copy-on-write section creation, and -retains the directory and artifact guards in the VM worker so writes, -truncation, deletion, and rename remain blocked until teardown. Linux retains -the exact directory and regular-file descriptors, opened without following -symbolic links, detects observable metadata changes before worker handoff, and -is therefore immune to pathname replacement; mandatory content immutability -still depends on host access control or an explicit stronger mode such as a -lease, fs-verity, or a verified artifact broker. - -## Authoritative restore - -Restore proceeds in the opposite direction from capture: - -1. open one exact snapshot generation, read its bounded manifest, and derive - the authoritative machine configuration; -2. resolve console, network, filesystem, policy, and read-only layer - attachments by stable ID or role; -3. validate state, captured memory, the selected RAM target, block geometry and - identities, and any paired scratch artifact, and claim a resume-policy - snapshot, before worker construction; -4. create a writable private copy-on-write mapping from the exact opened - `memory.bin` handle, add fresh private backing for selected expansion - ranges, transfer the artifact generation guards to the worker, and - materialize scratch according to its recorded policy as a verified private - copy, a reflink-only copy-on-write clone, an exact-file hard link from a - claimed continuation, or a caller-supplied fresh file; -5. construct the partition and exact device inventory with the selected active - RAM and the snapshot's immutable capacity; -6. run the [time ABI preflight](time-abi.md#restore-algorithm) (backend, CPU - profile and generation, rates, and downtime), compare the destination - topology, device, and queue contract with the saved contract, and check the - partition's effective CPUID against the snapshot's record; -7. restore VM time, chipset and virtio state, partition state, and vCPU state, - then set every VP's TSC at one host instant with read-back, advance the - LAPIC timers by the downtime and VM time, the RTC, and the PIT by the - downtime plus the interval by which VM time stopped before the capture - anchor, and seal the restore packet's time fields; -8. finish reconnecting host resources; -9. start the state units, with external input gated first when tier policy, - processor activation, or nonempty RAM expansion requires guest repair, and - publish the optional readiness event; -10. release the restored vCPUs; and -11. for a gated restore, on the agent's `0x605` acknowledgment, stop at the - exact post-write boundary, release input, and only then let the guest - continue. - -Listener attachments preserve their stable attachment ID, attachment kind, -backend kind, reconnect policy, required flag, length, and timeout. Restore -callers may rebind an eligible listener to a fresh same-kind endpoint. When an -ordinary `recreate-listener` replacement is omitted, OpenVMM reconstructs the -captured pathname; an authenticated control listener still requires an -explicitly approved restore-time attachment with a fresh capability. -Reusable-clone orchestrators must supply fresh private boot-console and -authenticated control-listener paths so independent or concurrent restores do -not collide with the terminated source generation. - -An orchestrator may supply a single-use readiness endpoint: an existing Unix -socket on Linux or named pipe on Windows. OpenVMM connects to it before the -worker launches. When the restored VM first starts, the worker writes one -fixed readiness record after every state unit has started and before it -releases a restored vCPU; for a gated restore, host input is still gated at -that point, and guest repair and its acknowledgment follow the event. A -connection, write, or flush failure aborts startup, and the endpoint is never -saved in a snapshot. - -The management RPC performs the same authoritative restore for snapshots -without sandbox blocks, a NIC, or a control console. Its request may repeat -only the microVM profile, an exact processor-count assertion, the portb -endpoint, the saved console and filesystem attachments, and guest power -actions. The restored VM stays paused until the client resumes it; that resume -publishes the readiness event, and a publication failure fails the resume and -tears the VM down. - -The partition must exist before OpenVMM can read the CPUID it presents and -check it against the effective CPUID. This does not expose a partially -restored guest: these checks and all saved-state validation still complete -before a vCPU runs. - -```mermaid -%%{init: {"theme": "base", "themeVariables": {"background": "#ffffff"}}}%% -flowchart LR - Manifest["Bounded manifest"] - Config["Authoritative machine config
and host attachments"] - Verify["Validate state, base RAM, target, and blocks
types, geometry, and identities"] - Cow["Private base-RAM mapping,
fresh expansion, and scratch"] - Machine["Construct partition
and exact devices"] - Validate["Validate destination CPU
and saved device state"] - Restore["Restore time, devices,
partition, and vCPU"] - Run["Start vCPU"] - - Manifest --> Config --> Verify --> Cow --> Machine --> Validate --> Restore --> Run -``` - -The captured memory artifact is mapped private and copy-on-write across -restores, while selected expansion ranges receive fresh private backing. -Paired scratch records one of three explicit restore policies: - -- `private-copy` preserves the existing behavior and may use reflink or sparse - copying; -- `copy-on-write` requires a filesystem reflink and is valid only for a - reusable workload-start clone; and -- `direct-claimed` attaches paired scratch only after an instance checkpoint - has been durably and exclusively claimed. - -Clone restores therefore dirty an independent file and leave the reusable -snapshot immutable. A direct-claimed restore transfers the single-use -continuation into normal live-instance ownership; it is no longer an available -snapshot. On WHP, a restore whose RAM is entirely the private -mapping of `memory.bin`, with no expansion ranges, prefetch, or VPCI devices, -registers only the first 64 MiB of each mapping with the hypervisor up front. -Later guest accesses register the remainder in 2-MiB chunks, and WHP resolves -the copy-on-write faults itself. Other restores register all guest RAM before -a vCPU runs. An instance checkpoint is single-use: the first -artifact- and configuration-validated restore attempt atomically creates and -flushes `resume.claim`; duplicate and concurrent restores fail before worker -construction, and a later startup failure does not make the checkpoint -reusable. Initial compatibility is same-backend: KVM snapshots restore on -compatible KVM hosts, MSHV on compatible MSHV hosts, and WHP on compatible WHP -hosts, of the CPU generation the snapshot's profile names. Cross-backend -conversion and standalone NVX snapshot import are not supported. - -## Template compatibility and placement - -A template represents a compatibility class, not an arbitrary fleet image. -The class includes the backend, CPU profile and generation, the -[effective CPUID](time-abi.md#cpu-profiles) that the OpenVMM build computes -for that profile and topology, guest kernel and agent build, effective command -line, device roles and geometry, network identity/policy, and consumed layer -identities. Deployment must rebuild or recertify templates on relevant -rollouts: an OpenVMM build that computes a different effective CPUID, for -example a different cache topology, rejects earlier templates with -`E_CPU_SURFACE`. The VMM's actual compatibility and saved-state checks, not a -promise about every build with the same version string, remain authoritative. - -Processor capacity and captured RAM geometry are immutable, but the opt-in -activation contracts below allow one template to serve several online CPU -counts and RAM targets. The template matrix is therefore keyed by capacities, -base state, and supported activation ranges, not necessarily by one artifact -for every final CPU/RAM size. Layer-role presence and device geometry remain -fixed regardless of these activations. - -For a platform template, substituting a layer requires the same recorded -logical length and block sizes. The current contract does not promise a -uniform virtual capacity over arbitrary shorter backing files. Builders may -standardize actual raw-file geometry or maintain separate geometry classes; -larger layers need a compatible template. Guest unbind/rebind cannot bypass -the host's pre-entry geometry checks. The bootstrap's buffer invalidation and -UUID check address stale cached bytes and attachment mix-ups, not permission -to alter the saved machine topology. - -## Restore-time processor activation - -The microVM separates the immutable processor capacity recorded by a snapshot from -the process-local VP set needed by one restore: - -- `C` is the manifest VP capacity. Processor topology, APIC identities, ACPI - and MP tables, and saved VP inventory always contain exactly `C` entries. -- `B` is the boot-online count recorded from an opt-in template's effective - `maxcpus=` command-line token. It must be one of 1, 2, 4, or 8 and no larger - than `C`. A snapshot without this opt-in records zero and rejects activation. -- `N` is an explicit `--restore-processors` target. It must be supported and - satisfy `B <= N <= C`; it is process-local and does not modify the snapshot. - -The requested online set is always the contiguous prefix `0..N-1`. This is not -a topology change or a general CPU-hotplug interface: the guest still sees the -capacity-`C` machine contract, and bringing a suffix VP online after readiness -is unsupported. - -Runtime VP materialization is backend-specific while the guest-visible -contract remains backend-independent: - -| Restore shape | Process-local VP runners and backend binders | Saved VP state | -| --- | --- | --- | -| MSHV with explicit `N` | Instantiate and bind only `0..N-1` | Validate all `C` entries, then apply the prefix | -| MSHV without explicit `N` | Instantiate and bind all `C` | Validate and apply all `C` entries | -| KVM or WHP | Instantiate and bind all `C` | Validate and apply all `C` entries | - -MSHV creates application processors lazily while binding their VP runners, so -discarding suffix binders before that boundary avoids creating processors that -this restore will not run. The synchronized TSC set on MSHV therefore covers -only the created VPs; the hypervisor rejects register access to a VP that was -never created. KVM and WHP deliberately retain their existing full-capacity -construction path, including when an explicit online target is sent to the -guest. - -Saved state remains authoritative despite the narrower MSHV runtime. Before -filtering, restore requires exactly one VP state entry for every index in -`0..C-1` and rejects missing, duplicate, or out-of-range entries, including -errors in the dormant suffix. Only after that validation may an explicit MSHV -restore apply state for `0..N-1`. A reduced-prefix process cannot later produce -a complete capacity-`C` VP inventory, so any attempt to save it fails as -unsupported; dump or debug access to an uninstantiated suffix VP -also fails explicitly instead of indexing nonexistent runtime state. The -original immutable snapshot remains reusable for independent restores at -other valid targets. - -After host-side state restoration, the -[restore packet](time-abi.md#restore-packet) carries `N` to the Alpine agent as -its online VP count, and portb status bit 2 advertises it; a RAM target travels -in the same packet. While external input remains gated, the agent onlines CPUs -from `B` through `N-1`, verifies that the kernel's online-CPU set is exactly the -requested prefix, and acknowledges through PMIO `0x605`. OpenVMM stops at that -post-write boundary, releases host input, and then resumes the guest. - -A fixed-capacity `N`-vCPU snapshot and a capacity-`C`, boot-online-`B` template -restored to `N` therefore reach the same online prefix but do not contain the -same guest state. The fixed snapshot captures all `N` CPUs online; the template -captures only `B` online CPUs and performs CPU activation during gated repair. -Performance comparisons should separate VP binding and worker construction -from guest resume-to-readiness. Matching host-side phase costs do not require -the total restore latencies or tail behavior to match. - -## Restore-time memory activation - -Restore-time memory activation separates the immutable RAM geometry recorded -by a snapshot from the active amount selected for one restored process: - -- `M0` is the captured base RAM. `memory.bin`, the Linux direct e820 map, and - the saved RAM-range inventory contain exactly `M0`. -- `Cmem` is the optional immutable capacity declared by `--memory-capacity` at - capture. The contract records capability version 1, `Cmem`, the 128-MiB Linux - memory-block size, and every canonical GPA range in `Cmem - M0`. -- `M` is an explicit `--restore-memory` target. It must be 128-MiB aligned and - satisfy `M0 <= M <= Cmem`. Omitting it restores exactly `M0`; a snapshot - without the capability rejects an explicit target. - -The management RPC exposes the same capture capacity and per-launch target, -together with the processor target, gate timeout, and readiness endpoint. - -The selected expansion is always a prefix of the contract's canonical ranges -in logical RAM order. OpenVMM maps the captured ranges privately from -`memory.bin`, maps only the selected suffix with fresh private backing, and -registers all selected ranges with the hypervisor before any restored vCPU -runs. Expansion bytes are neither read from nor written back to the snapshot, -so independent restores may choose different valid targets without changing -the reusable artifact. - -An explicit RAM target sets the restore packet's `MEMORY_TARGET` flag, and the -packet lists the selected expansion ranges as `(u64 gpa_start, u64 length)` -pairs ([layout](time-abi.md#restore-packet)). Portb status bit 3 mirrors the -flag and bit 4 indicates that the selected target contains at least one -expansion range. The same packet may carry a processor target. For a nonempty -expansion, OpenVMM keeps external input gated while the Alpine agent verifies -the 128-MiB block size and each range, probes missing blocks through the Linux -memory-block probe interface, onlines them and verifies their online state, and -then completes entropy and generation-ID repair before acknowledging PMIO -`0x605`. Any malformed range or add/online failure terminates the restore. - -An explicit base-size target has a zero range count. Unless the tier or a -processor target gates the restore, the packet's `ACK_REQUIRED` flag is clear, -and the agent emits the deterministic zero-add marker without a restore-gate -transaction. This mechanism is one-shot restore repair, not a general -post-readiness memory-hotplug interface. - -## Time and entropy - -Every microVM uses [NVX time ABI v1](time-abi.md), which defines the guest's -clocks on every backend: - -- The guest sees a minimal Hyper-V identity with the declared TSC and LAPIC - rates in MSRs, a CPU profile pinned for its CPU generation, an invariant TSC - as its clocksource, and the LAPIC timer at the backend's fixed rate. The - command line carries no clock tokens - ([guest-visible contract](time-abi.md#guest-visible-contract)). -- Capture records the time contract and the CPU profile record in the - [manifest](time-abi.md#snapshot-manifest). -- Restore requires the same backend and CPU generation, a native TSC rate - within 250 ppm of the declared rate, and an exactly equal LAPIC rate. It - measures the downtime with same-boot host monotonic time or the host UTC - delta, sets every VP's TSC at one host instant with read-back, and advances - the LAPIC timers, VM time, RTC, and PIT before the first VP runs - ([restore algorithm](time-abi.md#restore-algorithm)). Any failure rejects - the restore with a stable code; there is no fallback. -- After restore, the guest's snapshot agent sets the wall clock from the - restore packet's host UTC sample and re-checks conformance - ([snapshot agent](time-abi.md#snapshot-agent)). -Replaying a snapshot also replays the guest's in-memory random-number-generator -state. Every restore therefore exposes a fresh one-time -[restore packet](time-abi.md#restore-packet) through the portb status/data -protocol; with the time fields and any targets, it carries 64 bytes of fresh -entropy. Every microVM process receives a fresh 16-byte generation ID before -vCPU entry. The ID is not serialized, and the VMM-owned value is not restored -from device state. On restore, the ID is the packet's first 16 entropy bytes; -on cold boot it is generated independently. This keeps the gated repair path -from adding port reads. The Alpine agent keeps the prior ID in its captured -process state, rejects a restored ID that did not change, and exports the -refreshed value to the runtime hook before releasing the gate. - -For clone policy, the Alpine restore path credits the seed, including the -generation ID, to the kernel entropy pool, forces a kernel CRNG reseed, -refreshes machine identity, and requires the workload-start runtime hook to -reset runtime-owned RNG state before accepting work. The workload's -`/etc/machine-id` is a read-only bind of a -runtime-tmpfs file, so the agent can refresh it while scratch remains frozen; -the agent also updates the workload's UTS namespace before acknowledgement. It -then acknowledges the VMM gate before thawing scratch and the workload cgroup. -Resume policy records the fresh generation ID while preserving machine -identity and RNG continuity. Fresh post-restore entropy and the replacement -generation ID are never stored in the reusable snapshot; only the prior ID -remains as the agent's comparison token. diff --git a/doc/design/snapshot-sharing-and-host-storage.md b/doc/design/snapshot-sharing-and-host-storage.md deleted file mode 100644 index f83ee34de..000000000 --- a/doc/design/snapshot-sharing-and-host-storage.md +++ /dev/null @@ -1,110 +0,0 @@ -# Snapshot sharing and host storage - -[Design index](../design.md) - -Private COW restore prevents a VM's writes from modifying the reusable -artifact. It does not guarantee that clean physical pages are unshared across -VMs mapping the same file. Guest kernel, agent, and cached workload pages can -all originate in `memory.bin`; dropping the guest page cache before capture -does not by itself eliminate shared host physical pages or cache side channels. - -## Trust-domain policy (Proposed) - -Production artifact policy must distinguish write isolation from page-sharing -isolation. A workload-start image contains pages addressable by the workload; -sharing those pages across mutually untrusted tenants creates an avoidable -shared-page timing surface. Use tenant-scoped artifacts or independently -instantiated backing files instead of treating COW as tenant isolation. - -A fleet-shared platform template requires a narrower, verified argument: it -contains no tenant image bytes, configuration, secrets, or workload state, and -the workload cannot map the captured agent/kernel memory or configuration -region. Namespace/device policy and agent handling of secrets are part of that -argument. It is not enough to assert that PID 1 had not mounted a layer, or -that a template carries the `platform` label. The current VMM does not prove -these guest provenance and addressability properties. Security review must -also consider agent secret-dependent accesses and host memory deduplication. - -An instance checkpoint has one claimed continuation. Its single-use rule -prevents accidental forks but does not replace artifact confidentiality, -retention policy, or protection against host-side copying. - -## Snapshot identity and admission - -A snapshot binds each consumed sandbox block by role, access mode, exact -length, and block geometry, plus an identity under one policy shared by every -bound block. The [block contract](machine-and-device-abi.md#block) defines -these fields, and the [capture boundary](snapshot-and-restore.md#capture-boundary) -describes the capture options. - -| Identity policy | Selected by | Capture | Restore | Vouches for the bytes | -| --- | --- | --- | --- | --- | -| `sha256` | Default | Hashes each bound block | Rehashes each bound block and rejects a mismatch | OpenVMM | -| `generation` | `--snapshot-block-identity generation --snapshot-generation-id <32-HEX>` | Records the supplied generation for each bound block | Requires one shared nonzero generation and checks the structural fields without reading the blocks | The storage owner, which must keep the generation immutable and bound to the exact files | - -Generation mode trades a full read of every block for trust in the storage -owner. The generation is not a byte digest, and OpenVMM does not authenticate -it. - -A sealed snapshot is immutable. A clone creates a separate writable -descendant, while a resume atomically claims the snapshot and transfers its -continuation into a live instance. The claimed continuation is no longer an -available snapshot; there is no mutable-snapshot state. - -### Trusted storage and transport (Proposed) - -OpenVMM neither issues nor authenticates storage generations, and NVX does not -import or transport snapshots. A deployment that selects generation identity -needs a publisher or storage control plane that authenticates each generation -and binds it to the exact immutable block-object versions. Whole-file SHA-256 -then serves as a diagnostic, export, deduplication, or deep-verification -identity. If an external transport supplies SHA-256, compute it inline while -downloading rather than as a second local-disk pass. An object store ETag or -version ID can identify an immutable object version, but does not necessarily -provide a server-generated SHA-256; publisher metadata must bind any SHA-256 -to that exact version. Upload payloads first and commit the manifest last. -External bytes are an unverified import candidate until admitted. - -## Independent backing files (Proposed) - -Per-tenant instantiation can separate file-backed page caches while retaining -shared disk extents on suitable filesystems: - -| Reference or copy | File identity | Isolation implication | -| --- | --- | --- | -| Hard link or symlink | Same backing file | Does not separate the file-backed page cache | -| Linux reflink | Distinct inode, initially shared extents | Distinct file page caches, subject to host deduplication policy | -| Windows ReFS block clone | Distinct file streams, initially shared extents | Physical-frame separation under COW mappings requires validation on the deployed Windows build | -| Independent full copy | Distinct backing file | Avoids same-file sharing, subject to host deduplication policy | - -XFS/btrfs reflinks and ReFS block clones are optimization options, not mandatory -snapshot storage formats. NTFS and filesystems without cloning can use copies. -The current Windows snapshot fallback is a dense copy, not an implemented -ReFS per-tenant artifact service. KSM/page combining or other memory merging -must not recreate sharing where the deployment relies on separate files; -ReFS on-disk clone semantics alone are not proof of separate physical frames. - -Automatic RAM hard-link publication during capture is a different optimization: -it transfers the exact stopped source backing into one immutable generation. -It must not be confused with creating separate tenant backing files. Placement, -instantiation, access control, and garbage collection remain host-service work. - -Capacity planning must measure clean resident pages per backing generation, -dirty private pages per restored VM, and Windows COW commit charge separately. -A newly instantiated file may have a cold page cache even when the source -template is warm. Measure first restore and steady-state restore separately; -do not assume a near-zero on-disk clone cost means a free memory working set. - -## DAX and other memory-backed layers (Proposed) - -The current profile exposes neither virtio-pmem/DAX layers nor `phram` image -carriers. Retaining virtio-blk keeps read-only enforcement in the device I/O -path and avoids mapping shared image pages directly into the guest. DAX would -need host-enforced write protection, a restricted device surface, and an -explicit trust-domain sharing policy. It would also require suitable -uncompressed EROFS images, giving up the chosen compression benefits. - -Directly mapping an entire image also has guest-physical-address and kernel -metadata costs even when little of it is used. A future small-image carrier -or DAX profile should be justified by measured working sets and security -analysis, not presented as an existing size-based automatic fallback. diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md deleted file mode 100644 index 44fd491f1..000000000 --- a/doc/design/time-abi.md +++ /dev/null @@ -1,2464 +0,0 @@ -# Time ABI - -[Design index](../design.md) - -This document specifies NVX time ABI v1, the single guest-visible time, -timer, and clock contract of the microVM profile on KVM, MSHV, and WHP, as NVX -and its OpenVMM fork (`nanvix/openvmm`) implement it. It replaces the earlier -time rules of [Snapshot and restore](snapshot-and-restore.md#time-and-entropy) -and the clock tokens of [Cold boot](cold-boot.md#effective-command-line). - -Notation: - -| Symbol | Meaning | -| --- | --- | -| `F` | Declared guest TSC rate in Hz, returned by MSR `0x40000022` | -| `F_s` | Declared rate recorded in a snapshot | -| `F_d` | Native TSC rate of the destination partition in Hz | -| `L` | LAPIC timer rate in Hz, returned by MSR `0x40000023` | -| `C` | VP capacity of the machine (1, 2, 4, or 8) | -| `D` | Snapshot downtime in nanoseconds | -| `T_c` | Guest TSC of VP 0 at the capture anchor | -| `Q` | Host monotonic time in nanoseconds from the VM-time cut, where VM time and the RTC stop at quiesce, to the capture anchor | -| `g` | Restore generation counter of a VM process | - -"Reject" means fail with the named error code and never fall back to another -behavior. Every check in this document is a hard check. - -## Scope, goals, and non-goals - -Goals: - -- One time ABI on every backend and for every microVM, whether launched from - the command line or through OpenVMM's management endpoint: the same CPUID - identity, synthetic MSRs, CPU time bits, clocksource, tick, timers, and - restore semantics. -- No guest kernel patches and no clock command-line tokens. -- No silent fallbacks. A host or snapshot that cannot meet the contract is - rejected with a stable error code. The one fallback, from - `--cpu-profile auto` to a host profile on an Intel or AMD CPU that no pinned - profile serves, warns at every cold boot that takes it (see - [CPU profiles](#cpu-profiles)). -- Restore within one backend across hosts of the same CPU generation and CPU - profile, including restores after a host reboot. -- Guest monotonic time advances by the snapshot downtime. Wall clock is set - on every restore and disciplined afterwards. -- RCU-stall, soft-lockup, hung-task, clocksource, and TSC-warp detectors never - trip across capture and restore. A detected violation fails fast. -- No performance regression beyond the gate in - [Performance expectations and acceptance gate](#performance-expectations-and-acceptance-gate). - -Non-goals: - -- Restore across backends or across CPU generations. Both are rejected. -- Restore by an OpenVMM build that computes a different - [effective CPUID](#cpu-profiles) for the snapshot's profile and topology. - It is rejected with `E_CPU_SURFACE`. -- TSC scaling and RDTSC trapping. -- Paravirtual clocks: kvmclock, the Hyper-V reference TSC page and reference - counter, synthetic timers, VMGenID, and VMClock. -- A virtual PMU. -- CPUs of vendors other than Intel and AMD, such as Hygon's. -- Live migration, standard-machine guests, and guests other than the NVX - Linux 6.18 kernel. -- Snapshots older than manifest version 6. They are rejected with a recapture - error. - -The time ABI is versioned independently of the microVM machine ABI: -`time_abi_version` is 1 in the manifest and in CPUID leaf `0x40000002`, while -the microVM ABI version stays 2. Performance history keys its baselines by -microVM ABI version, so the time ABI does not create a new baseline series. - -## Guest-visible contract - -### Hypervisor identity - -Every VP sees a minimal Hyper-V frequency and invariant-TSC identity. OpenVMM -configures exactly these six leaves on every backend: - -| Leaf | EAX | EBX | ECX | EDX | -| --- | --- | --- | --- | --- | -| `0x40000000` | `0x40000005` (maximum leaf) | `0x7263694d` (`"Micr"`) | `0x666f736f` (`"osof"`) | `0x76482074` (`"t Hv"`) | -| `0x40000001` | `0x31237648` (`"Hv#1"`) | 0 | 0 | 0 | -| `0x40000002` | `0x0058564e` (`"NVX"`, build) | `0x00010000` (time ABI 1.0) | 0 | 0 | -| `0x40000003` | `0x00008860` | 0 | 0 | `0x00000100` | -| `0x40000004` | 0 | `0xffffffff` | 0 | 0 | -| `0x40000005` | `C` | `C` | 0 | 0 | - -`0x40000003` EAX grants exactly `AccessHypercallMsrs` (bit 5) and -`AccessVpIndex` (bit 6), which Linux requires to detect the platform, plus -`AccessFrequencyRegs` (bit 11) and `AccessTscInvariantControls` (bit 15). EDX -sets only `FrequencyRegsAvailable` (bit 8). `0x40000004` recommends nothing -and sets the spinlock retry count to "never notify". - -OpenVMM also programs explicit zero results for `0x40000006..=0x4000000f` -and `0x40000080..=0x40000082`, so they read zero on every backend. The -contract promises zeros only for these leaves: every other leaf in -`0x40000006..=0x400000ff` returns either all zeros or the vendor's -architectural out-of-range result (on Intel, the result of the highest basic -leaf for the same subleaf). KVM returns that result for every leaf missing -from its CPUID table, whose 256 entries cannot list the whole range. - -Every leaf beyond the identity leaves that Linux 6.18 reads with this -identity is an explicit zero. Built as NVX builds it (`CONFIG_HYPERVISOR_GUEST` -without `CONFIG_HYPERV`, `CONFIG_KVM_GUEST`, or Xen, ACRN, Jailhouse, and -bhyve guest support), Linux probes only VMware and Hyper-V, and these are all -its reads in the range: - -| Leaf | Reader | Condition | -| --- | --- | --- | -| `0x40000000` | `vmware_platform`, `ms_hyperv_platform`, `ms_hyperv_init_platform` | Always | -| `0x40000003` to `0x40000005` | `ms_hyperv_platform`, `ms_hyperv_init_platform` | Always | -| `0x40000081` | `ms_hyperv_msi_ext_dest_id` | x2APIC available without interrupt remapping | -| `0x40000082` | `ms_hyperv_msi_ext_dest_id` | `0x40000081` EAX is `VS#1`: never | -| `0x4000000a` | `ms_hyperv_init_platform` | Maximum leaf at least `0x4000000a`: never | -| `0x4000000c` | `ms_hyperv_init_platform` | `HV_ISOLATION` in `0x40000003` EBX: never | -| `0x40000010` | `vmware_select_hypercall` | VMware signature at `0x40000000`: never | - -The other explicit zeros serve Hyper-V-aware software that probes further. -No leaf returns Hyper-V feature data, a `VS#1` interface signature at -`0x40000081`, or any hypervisor signature. No base `0x40000100..=0x4000ff00` -(step `0x100`) carries a KVM, Xen, VMware, or other hypervisor signature, so -Linux selects only the Hyper-V platform. - -### MSRs - -OpenVMM serves the identity MSR range `0x40000000..=0x400001ff` identically -on every backend: - -| MSR | Name | Read | Write | -| --- | --- | --- | --- | -| `0x40000002` | `HV_X64_MSR_VP_INDEX` | VP index | #GP | -| `0x40000022` | `HV_X64_MSR_TSC_FREQUENCY` | `F` | #GP | -| `0x40000023` | `HV_X64_MSR_APIC_FREQUENCY` | `L` | #GP | -| `0x40000118` | `HV_X64_MSR_TSC_INVARIANT_CONTROL` | Last value written, 0 after reset | 0 or 1 accepted; any other value #GP | -| Any other MSR in the range | | #GP | #GP | - -`HV_X64_MSR_TSC_INVARIANT_CONTROL` is one partition-wide value. It is guest -state: it is saved with the VM and restored with it. Linux writes 1 once at -boot, on the BSP, with an unchecked `wrmsrq`, so this write must never fault -on any backend. Clearing it after setting it is accepted. - -The CPU time bits hide two architectural timer MSRs, and every backend makes -them raise #GP: `IA32_TSC_ADJUST` (`0x3b`) and `IA32_TSC_DEADLINE` -(`0x6e0`). - -The legacy P6 L2-cache MSRs (`0x88..=0x8a`, `0x116`, `0x118..=0x11b`, and -`0x11e`) also raise #GP on every backend, as MSHV's hypervisor already -makes them do; no profile pins them. KVM and WHP stub them only for Windows -booting from the PCAT BIOS, and standard-machine partitions keep those -stubs. Linux 6.18 never accesses them. A fault on an MSR that the guest -kernel accesses unchecked logs `unchecked MSR access error`, which the boot -check (`C4`) and the violation watcher (`G_UNCHECKED_MSR`) reject. - -Under an AMD CPU profile, OpenVMM also serves two of AMD's configuration -MSRs on every backend, with fixed values -(`virt::time_abi::msr::AMD_CONFIG_MSRS`): - -| MSR | Name | Read | Write | -| --- | --- | --- | --- | -| `0xc0010015` | `HWCR` | `0x1000000`: only `TscFreqSel` (bit 24) | The value read changes nothing; any other value #GP | -| `0xc0011029` | `DE_CFG` | `0x2`: only `LFENCE` serialization (bit 1) | The value read changes nothing; any other value #GP | - -Linux reads `HWCR` with an unchecked `rdmsr` on every AMD CPU with an -invariant TSC, which the time ABI's CPUID always presents, and logs -`[Firmware Bug]: TSC doesn't count with P0 frequency!` unless `TscFreqSel` -is set. It sets `DE_CFG`'s `LFENCE` serialization bit, with safe accessors, -on every CPU whose CPUID does not enumerate it, and relies on a serializing -`LFENCE` to order `RDTSC`. It writes neither MSR on a profile's CPU: only -the instructions-retired counter and CPUID faulting would make it, and the -profiles clear both. KVM's MSR filter denies both MSRs, MSHV intercepts -them, and WHP delivers them as unhandled-MSR exits. Booting Linux 6.18 on 4 -vCPUs on WHP reads `HWCR` once, on the BSP, and `DE_CFG` once per vCPU. -Under an Intel profile, both keep the backend's handling. - -### CPU time bits - -Every CPU profile fixes these bits. A profile that violates them is rejected. - -| Leaf | Register and bits | Value | -| --- | --- | --- | -| `0x1` | ECX[31] hypervisor present | 1 | -| `0x1` | ECX[24] TSC-deadline timer | 0 | -| `0x1` | ECX[15] PDCM | 0 | -| `0x1` | EDX[4] TSC | 1 | -| `0x6` | EAX, EBX, ECX, EDX | `0x00000004` (ARAT only), 0, 0, 0 | -| `0x7.0` | EBX[1] `IA32_TSC_ADJUST` | 0 | -| `0xa` | EAX, EBX, ECX, EDX | 0 (no PMU) | -| `0x15`, `0x16` | EAX, EBX, ECX, EDX | 0, when within the maximum basic leaf | -| `0x80000001` | EDX[27] RDTSCP | 1 | -| `0x80000007` | EAX, EBX, ECX, EDX | 0, 0, 0, `0x00000100` (invariant TSC only) | - -Leaves `0x15` and `0x16` are zeroed rather than synthesized. Linux takes both -rates from the frequency MSRs, and zero leaves keep the profile's CPUID -independent of any host's rate. - -Linux trusts the TSC because of `AccessTscInvariantControls`, not because of -CPUID `0x80000007` EDX[8]; that bit only adds the `nonstop_tsc` flag and -spares each AP a delay-loop calibration of about 150 ms (on Azure MSHV, an -8-vCPU cold boot takes 1,385 ms without the bit and 299 ms with it). The time -bits override the fingerprint: the bit is policy, not a fingerprint feature. -Profiles set it even where the backend cannot offer it to guests through its -feature banks, as on Azure's WHP and nested MSHV, whose host OS sees an -invariant TSC. Every backend exposes it (MSHV and WHP through a CPUID -override), and host qualification backs it with measured invariance (see -[Host qualification](#host-qualification)). - -### Rates - -- **TSC.** The guest TSC runs at the destination's native rate `F_d` and is - never scaled. The declared rate `F` is fixed for the life of a snapshot - lineage: a cold boot declares its native rate, and every restored process - keeps `F = F_s`. OpenVMM does not capture a restored process - (`--snapshot-destination` and `--restore-snapshot` are exclusive, and so - are the management endpoint's capture and restore paths), so a lineage is - one capture and its restores; if re-capture is added, the new snapshot - records `F_s` and the restored process's `g`. The kernel computes - `tsc_khz = floor(F / 1000)` once and never recalibrates. The - [rate policy](#tsc-rate-policy-and-lapic-rate-rule) bounds - `|F_d - F| / F`. -- **LAPIC.** `L` is a per-backend constant: 1,000,000,000 Hz on KVM and - 200,000,000 Hz on MSHV and WHP. Linux sets `lapic_timer_period = L / HZ` - (`0x989680` on KVM, `0x1e8480` on MSHV and WHP, with HZ=100) and never - calibrates the LAPIC. - -### Clocksource, tick, and PIT - -With this identity, Linux 6.18 built without `CONFIG_HYPERV`: - -- reads `F` from MSR `0x40000022`, sets `X86_FEATURE_TSC_KNOWN_FREQ`, and - registers clocksource `tsc` at `device_initcall` with no refinement; -- writes 1 to MSR `0x40000118` and sets `X86_FEATURE_TSC_RELIABLE`, which - disables the clocksource watchdog for `tsc` and the CPU-online TSC warp - test; -- presets `lapic_timer_period` from MSR `0x40000023`, sets `no_timer_check`, - and disables the hard-lockup detector; and -- disables the PIT at boot because the TSC rate is known, ARAT is present, - and the LAPIC period is preset. - -The guest must then be in this state: - -- `/proc/cpuinfo` flags on every CPU include `tsc`, `constant_tsc`, - `nonstop_tsc`, `tsc_known_freq`, `tsc_reliable`, `rdtscp`, `hypervisor`, - and `arat`, and exclude `tsc_deadline_timer` and `tsc_adjust`. -- `current_clocksource` is `tsc`, and `available_clocksource` is `tsc` - alone: Linux lists `refined-jiffies` and `jiffies` only while the reading - CPU's tick is periodic, which ends at that CPU's first tick. -- Ticks are tickless-idle with high-resolution timers. Every online CPU's - tick device is `lapic` in one-shot mode from its first tick onward. No - `pit` or `hpet` clock event device and no broadcast device exist. -- The LAPIC timer is never periodic after boot and never in TSC-deadline mode. -- PIT channel 0 never counts after boot, and IRQ 0 has no timer handler. - -Because the kernel no longer checks cross-vCPU TSC consistency, the -[skew bound](#cross-vcpu-skew-bound) is enforced outside the guest. - -### Deviations from the Hyper-V TLFS - -- `0x40000003` advertises `AccessHypercallMsrs` only for platform detection. - `HV_X64_MSR_GUEST_OS_ID` (`0x40000000`) and `HV_X64_MSR_HYPERCALL` - (`0x40000001`) raise #GP, and there is no hypercall page. The NVX kernel is - built without `CONFIG_HYPERV` and never touches them. `VMCALL` behavior is - outside the contract; the guest never executes it. -- `HV_X64_MSR_TSC_FREQUENCY` returns the declared rate `F`, which differs - from the physical rate `F_d` by up to the rate tolerance after a restore. -- `HV_X64_MSR_TSC_INVARIANT_CONTROL` can be cleared after it is set; Hyper-V - and KVM raise #GP instead. -- On MSHV and WHP, CPUID `0x80000007` EDX[8] is set by the CPU profile from - boot; it does not wait for a write to `HV_X64_MSR_TSC_INVARIANT_CONTROL`. - KVM 6.3 and newer follow the TLFS and hide the bit while KVM's own copy of - the control is 0, so the KVM backend mirrors the guest-visible value into - KVM (see [Backend obligations](#backend-obligations)). Linux writes 1 in - `ms_hyperv_init_platform`, before `identify_boot_cpu` and the APs read the - bit, so the booted guest is identical on every backend. -- `0x40000002` carries an NVX signature and the time ABI version, not a - Hyper-V build number. -- `0x40000005` reports the VP capacity in both EAX and EBX. - -Side effects on Linux are limited to selecting the Hyper-V platform: an -unknown-NMI handler is registered, `pv_info.name` is `Hyper-V`, the -hard-lockup detector is off, and `no_timer_check` is set. No KVM -paravirtual feature (kvmclock, PV EOI, PV IPIs, PV TLB flush, PV spinlocks, -halt polling, steal time, or async page faults) exists on any backend; -`sched_clock` and the vDSO use the raw TSC. - -On KVM, the identity therefore forgoes the paravirtual IPIs, TLB flushes, -halt polling, and steal time that a KVM signature would give a multi-vCPU -guest. That may be why the guest's side of an 8-vCPU cold boot on nested -Azure KVM measured 8.2 [0.4, 15.9] ms slower than the `dev` base's; the -difference is unattributed, and the VMM's gain offsets it (see -[Performance expectations](#performance-expectations-and-acceptance-gate)). -Hyper-V's own enlightenments could restore paravirtual IPIs and TLB flushes -under the identity, but they need the guest's `CONFIG_HYPERV` and hypercall -support in OpenVMM, so they are left for after v1. - -## Backend obligations - -Each backend implements these mechanisms behind one OpenVMM interface. The -guest-visible result is identical; only the mechanism differs. The spikes -(`p1-spike-kvm`, `p1-spike-mshv`, `p1-spike-whp-tsc`) demonstrated every -settled cell on the registered hosts. - -| Obligation | KVM | MSHV | WHP | -| --- | --- | --- | --- | -| Identity CPUID (exact leaves, out-of-range rule) | `KVM_SET_CPUID2` with the identity and explicit zero leaves; every KVM `0x4xxxxxxx` entry removed. Other leaves in the range return KVM's out-of-range result: Intel's under an Intel profile, and zero under an AMD one, as KVM answers a guest whose CPUID vendor is AMD | No synthetic processor features, so the hypervisor reports no `0x400000xx` leaves; CPUID intercept results (`always_override`) for the identity leaves, while the explicit zero leaves already read zero in VP 0's view and get none (see the next row); preflight reads every entry back and requires VP 0 to read zero at six sentinel leaves (`0x40000006`, `0x40000081`, `0x400000ff`, `0x40000100`, `0x40000200`, and `0x4000ff00`) | CPUID exits for the six identity leaves, served from OpenVMM's table; with synthetic features off, WHP returns zero natively for `0x40000006..=0x400000ff` and the bases from `0x40000100`, which preflight asserts | -| Profile CPUID and time bits | `KVM_SET_CPUID2` with the effective CPUID. KVM 6.3 and newer hide invariant TSC while KVM's own `HV_X64_MSR_TSC_INVARIANT_CONTROL` is 0, so the backend mirrors the guest-visible value, which core saves with the VM, into it with a host-initiated `KVM_SET_MSRS` after every accepted guest write and before a VP runs after a restore or reset; without it the guest loses `constant_tsc` and `nonstop_tsc` and boots about 160 ms slower | Processor feature banks derived from the profile (`hv_banks`), plus a CPUID intercept result (`always_override`, subleaf-specific where the entry has a subleaf) for every partition-wide entry of the effective CPUID that VP 0's own view does not already present under its mask. Once VP 0 exists, one bulk `HvCallGetVpCpuidValues` reads every partition-wide entry before any result is registered, and a failed bulk read registers every entry. That leaves 11 of 61 entries on the bare-metal host (the hypervisor bit, one leaf-2 descriptor byte, ARAT, the six identity leaves, and two brand leaves) and 12 of 65 on the nested Azure host (the same plus invariant TSC), at every vCPU count, because VP 0's own view already presents leaf 4's shared L3; registering them, the read included, takes 63 to 71 µs and about 90 µs, against 200 and about 300 µs for every entry. Profile masks never cover runtime-owned or VM-owned bits, so a pinned value that VP 0 presents at reset it presents always. Reserved entries pass through from the hypervisor's guest view, which reads zero at every one with no result registered, and verification requires every candidate to read zero (`E_CPU_UNLISTED`). Per-VP fields cannot be left to the hypervisor: it does not implement `0xB` for these partitions and reads `EDX` as 0 on every VP, so Linux would log `APIC ID mismatch` (`G_APIC_ID_MISMATCH`). The `0xB` and `0x1F` entries are therefore per-VP intercept results with the VP's x2APIC ID in `EDX`, registered as each VP is created: the hypervisor refuses a per-VP result for a VP that does not exist yet (`InvalidVpIndex`). Under an AMD profile, so is `0x8000001E`, with the VP's extended APIC, compute unit, and node IDs (no AMD MSHV host has run it yet). Leaf 1's initial APIC ID is the hypervisor's own, which is correct | Processor feature banks derived from the profile (`hv_banks`), and every effective-CPUID entry outside the hypervisor range as a `CpuidResultList2` result (the banks cannot express the hypervisor bit, ARAT, or, on Azure, invariant TSC); reserved entries pass through from WHP's guest view, which verification requires to be zero (`E_CPU_UNLISTED`). CPUID exits only for the identity leaves and the leaves with per-VP APIC fields (`1`, `0xB`, and, where listed, `0x1F` and `0x8000001E`), 8 exits for every Intel v1 profile and 9 for every AMD v1 profile: each exit-list entry adds about 25 µs to partition setup, and the legacy 269-entry list cost about 7 ms per restore. The exit handler presents the effective CPUID's `0xB` terminator, with the VP's x2APIC ID, instead of zeroing subleaves from 2 | -| Identity MSRs routed to OpenVMM | `KVM_CAP_X86_USER_SPACE_MSR` (`UNKNOWN`, `FILTER`) and `KVM_X86_SET_MSR_FILTER` denying reads and writes of `0x40000000..=0x400001ff`, and under an AMD profile of `HWCR` and `DE_CFG` (see [MSRs](#msrs)); the filter takes precedence over KVM's in-kernel Hyper-V MSRs. Under an Intel profile, a Linux boot takes four MSR exits, all on the BSP, at any vCPU count; under an AMD profile, it also reads `HWCR` once, on the BSP, and `DE_CFG` once on each vCPU, so it takes 4 + 1 + N exits on N vCPUs (counted on WHP; GitHub-hosted runners with EPYC 7763, 9V74, and 9V45 CPUs boot it on KVM) | MSR-index intercepts (`HV_INTERCEPT_TYPE_X64_MSR_INDEX`, `READ_WRITE`) for `0x40000002`, `0x40000022`, `0x40000023`, and `0x40000118`, and under an AMD profile for `HWCR` and `DE_CFG`. The hypervisor itself raises #GP for writes to the three read-only MSRs and for every other MSR in the range. Native synthetic MSRs are forbidden: they pre-empt the intercepts, and the native `HV_X64_MSR_TSC_FREQUENCY` returns the destination's rate instead of `F`. Verified on hypervisor builds 26100.30000 (bare metal) and 26100.9444 (Azure) | `X64MsrExitBitmap` with `UnhandledMsrs` (capability `0x3f` on every host) and the offloaded APIC, no synthetic features and no `hv1_emulator`: the identity MSRs exit to OpenVMM, which raises #GP for every other MSR in the range; `HWCR` and `DE_CFG` exit too, and OpenVMM serves them under an AMD profile | -| Native rate `F_d` | `KVM_GET_TSC_KHZ` × 1000 on VP 0 (1 kHz granularity) | `ProcessorClockFrequency` partition property | `WHvCapabilityCodeProcessorClockFrequency` | -| No TSC scaling | Never `KVM_SET_TSC_KHZ`; every vCPU reports the host rate | No frequency override | No `ProcessorClockFrequency` partition property; the 1 GHz request is removed | -| LAPIC rate `L` | In-kernel LAPIC at 1 GHz; `KVM_CAP_X86_APIC_BUS_CYCLES_NS` never set | The hypervisor's LAPIC at 200 MHz | Offloaded APIC at its fixed 200 MHz, verified at preflight (setting `InterruptClockFrequency` is not supported); the emulated APIC is not used | -| Pending LAPIC vector after a LAPIC state write | Nothing more: `KVM_SET_LAPIC` raises `KVM_REQ_EVENT`, which wakes a halted vCPU for a deliverable vector | Writing the LAPIC state does not wake a halted VP, so the backend asserts the highest pending edge-triggered vector (16 and up) to the VP's own APIC through the hypervisor's interrupt path after the write, and again when the VP next runs; level-triggered vectors stay with the IOAPIC | Writing the offloaded LAPIC state does not reliably wake a halted VP, so the backend asserts the highest pending edge-triggered vector with `WHvRequestInterrupt` to the VP's own APIC ID after the write, and again before the VP next runs | -| TSC-deadline and `TSC_ADJUST` hidden; both MSRs raise #GP | CPUID bits cleared; the MSR filter also denies `IA32_TSC_ADJUST` (`0x3b`) and `IA32_TSC_DEADLINE` (`0x6e0`), which KVM would otherwise serve (reading 0 and ignoring writes to `0x6e0`), and OpenVMM raises #GP | Feature-bank bits `tsc_deadline_tmr_support`, `tsc_adjust_support`, and `a_count_m_count_support` cleared, and CPUID bits cleared; the hypervisor then raises #GP for reads and writes of both MSRs on every CPU | Feature-bank bits `TscDeadlineTmr`, `TscAdjust`, and `ACountMCount` cleared, and CPUID bits cleared; both MSRs exit to OpenVMM, which raises #GP on every CPU | -| Invariant TSC exposed | CPUID bit. KVM hides it from a guest with `"Hv#1"` until KVM's own `HV_X64_MSR_TSC_INVARIANT_CONTROL` is set, so OpenVMM writes 1 to it host-side at vCPU creation | Forced by the profile through a CPUID intercept result: Azure's nested MSHV cannot offer the bit to its guests, although the host OS sees an invariant TSC, and without it each AP pays about 150 ms of calibration. Qualification measures the property instead (`H4`, `H6`) | Set by the CPUID override on every host: on Azure, WHP cannot offer it through the feature banks (bank 1 lacks `TscInvariant`), although the host OS sees an invariant TSC. There the guest TSCs stayed within 60 ns of each other at 8 vCPUs over 60 s and matched the declared rate against host QPC to 0.000 ppm (residual at most 0.07 µs) over 118 s; `H4` and `H6` measure both on every host | -| No paravirtual or synthetic features | No KVM leaves; `KVM_CAP_ENFORCE_PV_FEATURE_CPUID`, so KVM's paravirtual MSRs raise #GP; KVM's in-kernel Hyper-V MSRs are unreachable behind the filter | No synthetic processor features | `--hv` stays rejected for the microVM | -| Capture anchor: VP 0 TSC paired with a host time sample within 100 µs, from at most 64 samples | Host `rdtsc` plus VP 0's `KVM_VCPU_TSC_OFFSET`, bracketed by two host `rdtsc` reads around the host clock reads; up to 16 attempts (0.06 to 0.4 µs) | The tightest of up to 64 brackets `[sample, HvCallGetVpRegisters(VP 0 TSC), sample]`, paired at the bracket midpoint (p50 3.5 µs on bare metal, 6.2 µs on Azure) | The tightest of up to 64 bracketed reads of VP 0's TSC register, paired at the bracket midpoint (2.6 to 6.4 µs on bare metal, 5.4 to 9.7 µs on 8370C runners, 5.8 to 11 µs on 8573C runners) | -| Synchronized TSC set at one host instant | One `KVM_VCPU_TSC_OFFSET` value for every vCPU, `target(t) - (h0 + h1) / 2` from a host clock read `t` bracketed by host `rdtsc` reads `h0` and `h1` (Linux 5.16 or newer); no `IA32_TSC` writes, which Linux 6.6 can discard | Freeze partition time (already frozen since preflight), write the target to every created VP, read back, and clear `TimeFreeze` right after a successful read-back, inside the set rather than at the first VP run (56 to 312 µs for 1 to 8 VPs; equal on 20 of 20 restores) | Suspend partition time, write the target to every VP, read back, and resume with `WHvResumePartitionTime` right after a successful read-back, inside the set. Writing while time runs would skew the VPs by the write latency, about 11 µs per write on bare metal and 25 µs nested; `TscVirtualOffset` is unusable (writes fail) | -| Read-back before any VP runs | Every vCPU's `KVM_VCPU_TSC_OFFSET` equals the written value, and a host `rdtsc` bracket around VP 0's `IA32_TSC` shows no scaling | Every created VP's TSC equals the target while time is frozen | Every VP's TSC equals the target while time is suspended; live reads cannot verify 1 µs (a register read takes 9.5 to 21 µs) | -| Live cross-vCPU skew after release at most 1 µs | Equal offsets: skew is the host's TSC skew, bounded by qualification. Measured at most 63 ns | Measured 0 warps; offsets within 516 ns on dual-socket bare metal and 195 ns on Azure, both bounded by the probe's round trip | Measured at most 70 ns over 60 s on the bare-metal host and on 8370C and 8573C runners | -| VP instantiation | All `C` VPs exist before the set | Every instantiated VP is bound, which creates it, before the set; no VP is created after it | All `C` VPs exist before the set | -| Partition capabilities, so that the identity leaves never enable the Hyper-V emulator or its saved-state elements | Derived from CPUID with the hypervisor range masked: `hv1` and `kvm_clock` are false | Same | Same | -| Unknown MSRs | #GP, including the legacy L2-cache MSR stubs (see [MSRs](#msrs)) | #GP from the hypervisor | #GP, including the legacy L2-cache MSR stubs | -| Removed | `KVM_GET_CLOCK`/`KVM_SET_CLOCK` in the microVM downtime path, kvmclock MSR state, leaf `0x15` synthesis, `KVM_SET_TSC_KHZ`, restore-time `IA32_TSC` writes | BSP-copy TSC alignment, exact-rate equality, leaf `0x15` synthesis | 1 GHz request and fallback, `RestoredTsc` and its RDTSC, RDTSCP, and `IA32_TSC` exits, leaf `0x15` synthesis | - -**KVM common offset.** Restoring each VP's `IA32_TSC` is unreliable on KVM. -Linux 6.6 KVM, which the Azure KVM runners run, treats a host `IA32_TSC` -write within about 1 s of the TSC timeline started at vCPU creation as a -synchronization attempt and discards the written value, the first write of a -restore included; every CI shell snapshot is taken within a second of boot. -With the legacy restore path on nested Azure KVM, a restored guest's monotonic -clock advanced 54.7 ms across a 512.5 ms host interval, losing 458 ms. Linux -6.7 applies the heuristic only after a first user-space write -(`user_set_tsc`). `KVM_VCPU_TSC_OFFSET` sets the offset exactly on every -kernel from 5.16, so the backend never writes `IA32_TSC` and core omits the -saved per-VP TSC values from the VP restore. - -**WHP decision gate: met.** Suspending partition time, writing the target, -verifying the frozen values, and resuming kept every pair of vCPUs within -70 ns over 60 s on bare metal and on both Azure runner generations, so WHP -never traps RDTSC. The emulated clock it replaces cost about 45 µs per guest -timestamp read on bare metal and 66 to 70 µs on Azure (p50), on every vCPU of -an SMP-restored VM for its lifetime. - -## CPU profiles - -A CPU profile is the complete guest-visible CPU surface of one vendor and CPU -generation, shared by every backend. In every governed entry the guest sees -only profile values, never host passthrough, except for a fixed, code-defined -set of VMM-owned fields and the fields the time ABI owns; the other entries -are reserved, and verification keeps host features out of them (see -**Effective CPUID**). Profiles are data: derived mechanically by -intersecting host fingerprints (`openvmm --cpu-fingerprint`, the Firecracker -`cpu-template-helper` workflow) per register, first within each backend and -then across backends, reviewed, checked in at -`vmm_core/cpu_profile/profiles/.json`, embedded in the OpenVMM binary, and -immutable once released. A normative change creates a new revision; released -revisions are never deleted, so old snapshots stay restorable. One surface -serves all three backends: MSHV and WHP derive their processor feature banks -from the profile's CPUID (`cpu_profile::hv_banks`) instead of from the host's, -reading each feature where the profile's vendor enumerates it (AMD CPUs -enumerate the speculation controls in `0x80000008` EBX and SVM in -`0x80000001` ECX, and Intel's `PREFETCHW` feature controls no AMD bit), -and each backend verifies at partition creation that it supports the profile. -Every backend programs the whole surface explicitly and never relies on -hypervisor defaults; WHP's default processor features, for example, omit -SPEC_CTRL, IBPB, STIBP, and SSBD on Skylake and PSFD on Ice Lake. With -explicit banks, Linux in a bare-metal Skylake-SP WHP guest enables IBRS, -IBPB, SSBD, and CPU buffer clearing. On Azure, the L0 hypervisor withholds -the SPEC_CTRL family and MD_CLEAR from every backend on both generations -(each reports `CPUID.7.0:EDX` = `0x20000010`), and the Ice Lake-SP and -Emerald Rapids profiles pin that value. -A shared profile gives the same guest behavior on every backend; it does not -make snapshots portable, because cross-backend restore is rejected -(`E_BACKEND_MISMATCH`). The `IA32_ARCH_CAPABILITIES` bits outside the pinned -mask are the backend's, because MSHV and WHP can neither set nor read them -back. Measured, they differ only in `PSCHANGE_MC_NO` (bit 6) on Skylake-SP, -where KVM and WHP present 1 and MSHV presents 0 on the same hardware. Linux -uses that bit only for `X86_BUG_ITLB_MULTIHIT`, which matters only to KVM -inside the guest, so it changes no mitigation of an NVX guest, which has no -VMX. Only the sysfs `itlb_multihit` line differs. - -**Format.** Schema `openvmm-cpu-profile/v1`. The canonical encoding is -compact canonical JSON (object keys sorted by their UTF-8 bytes, no -whitespace, numbers as hexadecimal strings); the profile digest is the -SHA-256 of that encoding, and decoding accepts only canonical bytes, so a -profile has exactly one encoding and one digest. Pinned files use the same -document in pretty form. - -| Field | Content | -| --- | --- | -| `schema` | `openvmm-cpu-profile/v1` | -| `id` | `..v`, each component `[a-z0-9-]+`, for example `intel.icelake-sp.v1` | -| `description` | Free text | -| `vendor` | The 12-byte CPUID vendor string | -| `generation` | The generation `name` (`skylake-sp`, `icelake-sp`, `emeraldrapids`, `alderlake`, `milan`, `genoa`, or `turin`; `host` for a host profile, see **Host profiles**) and its `cpus`: `(family, model, stepping range)` display signatures; stepping ranges separate model 85's Skylake-SP (0 to 4), Cascade Lake, and Cooper Lake | -| `cpuid` | A dense table of every leaf and subleaf in `[0, max basic]` and `[0x80000000, max extended]` with a value and a mask per register; mask bit 1 pins the value, mask bit 0 marks a VMM-owned or runtime-owned bit | -| `xcr0`, `xss`, `xsave_components` | The XSAVE features the guest may enable, and the size, offset, and flags of every enabled component | -| `physical_address_width` | The guest physical address width | -| `msrs` | Pinned MSR values with masks; v1 pins `IA32_ARCH_CAPABILITIES` only, under the mask of the bits that Hyper-V's feature banks derive plus `ITS_NO`, and AMD profiles pin none, because AMD CPUs have no `IA32_ARCH_CAPABILITIES`. KVM presents it as a feature MSR; MSHV and WHP present it through their banks' `*_NO` bits, because neither has a register to set or read it back | -| `provenance` | The derivation method and the source fingerprints' backends, host counts, and surface digests (informative) | - -VMM-owned bits are a code table, not profile data: `CPUID.1:EBX[31:16]` -(logical count and initial APIC ID), x2APIC (`CPUID.1:ECX[21]`), the core -and cache-sharing counts in `CPUID.4:EAX[31:14]`, and the topology leaves -`0xB` and `0x1F`, which profiles omit. The machine has one socket and one -core per vCPU ([configuration boundary](configuration-boundary.md)), so in -`CPUID.4` every cache reports the vCPU count less one as the cores per -socket (`EAX[31:26]`), and the L3 cache reports it as the logical processors -sharing the cache (`EAX[25:14]`) too; the L1 and L2 caches report 0 there, -one vCPU each. AMD CPUs have no leaf 4, so the table differs by vendor: in -AMD profiles, OpenVMM owns instead the core count and APIC ID size of -`CPUID.80000008:ECX` (bits 7:0 and 15:12), the cache-sharing counts of -`CPUID.8000001D:EAX[25:14]` (the L3 cache shared by every vCPU, the L1 and -L2 caches by one), and the processor topology of `CPUID.8000001E` (each -vCPU's extended APIC, compute unit, and node IDs, the threads per compute -unit, and the nodes per processor). Intel profiles pin those AMD fields as -before, so adding the AMD rows changed no Intel profile's encoding or -digest. Runtime-owned bits mirror control state: `OSXSAVE`, `OSPKE`, -and the XSAVE sizes for the current XCR0 and XSS. -The time ABI owns `0x40000000..=0x4fffffff`, `0x15`, and `0x16`, and every -profile pins the [CPU time bits](#cpu-time-bits). Invariant TSC and ARAT are -pinned set and, with the hypervisor bit, exempt from host-support -verification, because a backend's fingerprint can lack them: WHP's feature -banks cannot express ARAT, and on Azure, WHP and nested MSHV cannot offer -invariant TSC to guests through their feature banks although the host OS -sees it. Host qualification measures them instead. Profiles also pin policy zeros -(VMX and SVM, SGX, PT, RDT, PCONFIG, and the other features listed by the -profiles' derivation policy; in AMD profiles also IBS, LWP, SKINIT, the -performance counter extensions, MONITORX, RDPRU, PPIN, CPPC, INVLPGB, the -instructions-retired counter, SME and SEV, whose leaf `0x8000001F` is -zero, `BTC_NO`, which KVM never enumerates, and PSFD unless a `SPEC_CTRL` -control stays, without which KVM denies a guest `SPEC_CTRL`). The -effective guest CPUID is a pure function of -the profile, the VM topology, and the time ABI's identity leaves. - -**Effective CPUID.** The effective CPUID -lists exactly the governed leaves: every leaf and subleaf of the profile's -`cpuid` table; the topology leaves `0xB` and `0x1F`, when they are within -the maximum basic leaf, at the subleaves the VM's topology defines and the -subleaf that terminates them (an invalid level: `EAX` and `EBX` zero, -`ECX[7:0]` its own number, `ECX[15:8]` zero, and the x2APIC ID in `EDX`), -which Linux reads to end its enumeration; and the identity leaves -`0x40000000..=0x40000005` with the explicit zero leaves. Each -entry holds the four registers and their masks; the per-VP fields hold VP 0's -APIC identity, and the runtime-owned bits have mask 0. OpenVMM computes it, -records it as the manifest's packed binary record, and recomputes it on -restore, so the record never depends on how a backend enumerates or caches -CPUID, and one profile and topology give the same record on every backend. -The record also fixes OpenVMM's own leaves and VMM-owned bits, so changing -how OpenVMM computes them breaks snapshot compatibility: snapshots that -earlier builds captured fail restore with `E_CPU_SURFACE` and must be -recaptured, and the release notes say so. -Its canonical JSON form (`openvmm-effective-cpuid/v1`) serves tools, never -the start paths. At preflight, before any VP runs, each backend reports -VP 0's CPUID for the governed leaves (KVM from the `KVM_SET_CPUID2` table it -programmed, MSHV by reading VP 0, WHP from VP 0's register view), reading an -entry without a subleaf at subleaf 0, and OpenVMM compares the report with -the effective CPUID under its masks. MSHV and WHP also report VP 0's view at -the reserved entries the host's CPUID enumerates (verification step 6). MSHV -reads every reported entry from the hypervisor, the explicit zero leaves of -the identity range included, and preflight also requires VP 0 to read zero -at six sentinel leaves of the range (`0x40000006`, `0x40000081`, -`0x400000ff`, `0x40000100`, `0x40000200`, and `0x4000ff00`; -`E_IDENTITY_ROUTING`). It batches the reads into rep -`HvCallGetVpCpuidValues` calls of up to 128 entries, with the VP's XFEM and -XSS and the registered results applied, and reads one entry per call, with a -warning, if a batched call fails. - -The effective CPUID is the single source of the guest's CPUID. Core passes it -to the backend as `TimeAbiConfig::cpuid`, with the per-VP fields unmasked, -and the backend presents every entry with each VP's own APIC identity (see -[Backend obligations](#backend-obligations)). Banks and pinned MSRs come -from the profile itself, which the backend looks up by -`TimeAbiConfig::cpu_profile`. The governed entries are every entry an -architectural enumeration reaches. - -The other entries, which only probing past an advertised maximum reaches, are -reserved: their values are implementation-defined, and guests must not rely -on them. They are: - -- a leaf above the maximum basic or extended leaf; -- a subleaf past its leaf's reported maximum or terminator; -- a `0xD` subleaf of a component that is not enabled. - -KVM answers them from the effective CPUID: zero, Intel's out-of-range result -(the highest basic leaf's) above the maximum basic leaf under an Intel -profile, or, past the last topology level, an invalid level with the x2APIC -ID. Under an AMD profile, KVM answers every out-of-range leaf with zero, as -it answers any guest whose CPUID vendor is AMD. MSHV and WHP pass them -through from their own guest views. Both read zero at every reserved entry -their sweeps reach (below), so neither programs zero results there, which -would cost about 5 µs per entry at partition setup: about 35 µs on WHP for -the 6 or 7 candidates of a fleet host, and 49 to 63 µs on MSHV for its former -13 to 15. Verification keeps host features out of every entry passed -through: profile verification fails a host whose hypervisor presents a -non-zero entry outside the profile's tables (`E_CPU_UNLISTED`, verification -step 6), checked on VP 0's view at every start. Under the time ABI, the MSHV -and WHP sweeps of VP 0 read zero there, including Intel PT's `0x14.1`, which -the Skylake-SP roots show as non-zero: - -- WHP's hardware sweep covers subleaves 0 to 63 of every indexed leaf and four - leaves past each maximum, on all three host types. -- MSHV's sweep, run with no result registered for any reserved entry, covers - the host-enumerated reserved entries (15 on the bare-metal host and 13 on - the nested Azure one) plus 2,755 and 3,261 probes: the same near range as - WHP's, and far probes where the hypervisor answers by itself (subleaves 64 - to 255 of every indexed leaf, the leaves past the near range up to `0xff` - and `0x800000ff`, the hypervisor range up to `0x400001ff`, and nine distant - leaves). The hypervisor builds were 10.0.26100.30000 and, nested, - 10.0.26100.9444. - -The fleet fingerprints cover only the entries that a guest's own enumeration -reaches. The `E_CPU_SURFACE` comparison and the snapshot record cover only -the governed entries. - -Informational fields: - -- The brand string (`0x80000002..=0x80000004`) is generic per generation: - `Intel(R) Xeon(R) Processor ()` with the generation's display name - (`Skylake-SP`, `Ice Lake-SP`, or `Emerald Rapids`), - `Intel(R) Core(TM) Processor (Alder Lake)`, or - `AMD EPYC Processor ()` with `Milan`, `Genoa`, or `Turin`, - zero-padded, without a - frequency. Every host of a generation presents it whatever its SKU, and - derivation needs no common brand across the source hosts. Host profiles - present `Intel(R) Processor (host profile)` or - `AMD Processor (host profile)`. -- The cache leaves (`0x2` and `0x4`, and AMD's `0x80000005`, `0x80000006`, - and `0x8000001D`, except their VMM-owned counts) come from the reference - hosts; other SKUs of the generation differ, and verification does not - compare them. Neither does it compare the copy of the CPU's signature in - AMD's `0x80000001` EAX, which describes the CPU as leaf 1 does. Address - widths are checked as limits. -- `IA32_UCODE_REV` is not pinned: Linux disables its microcode loader and - skips its microcode checks under a hypervisor, so the value only reaches - `/proc/cpuinfo`, and it is outside the effective-CPUID record and every - restore check. - -Mitigation-relevant bits follow the hardware. A profile sets `ITS_NO` -(`IA32_ARCH_CAPABILITIES` bit 62) or `BHI_CTRL` (`CPUID.7.2:EDX[4]`) only if -every CPU of its generation is immune or has the control and every backend -can present the value; MSHV cannot intercept `IA32_ARCH_CAPABILITIES`. The -v1 profiles clear both: MSHV and WHP cannot present `ITS_NO`, and no host -available to the fleet exposes `BHI_CTRL`. MSHV cannot present `BHI_CTRL` at -all: even on 8573C runners its feature banks have neither `bhi_dis` nor -`bhi_no`, and leaf 7 reports a maximum subleaf of 0. Without `ITS_NO`, the -hypervisor bit makes Linux enable its ITS mitigation. NVX keeps every such -mitigation enabled, and the kernel must not leave its thunk pages writable -and executable (boot check `K1`). - -**Selection.** A microVM always has a profile. A cold boot uses -`--cpu-profile `, `auto` (the default), or `host`. `auto` selects the -highest revision of the single pinned profile whose generation covers the -host's vendor, family, model, and stepping as the VMM's host OS sees them (the -L1 view on Azure). With no match on an Intel or AMD CPU, `auto` falls back to -the host profile that `host` selects (see **Host profiles**); host CPUID -passthrough does not exist. With no match on a CPU of another vendor, such as -Hygon's, or with matches in more than one generation, a catalog defect, -`auto` fails with `E_PROFILE_HOST_UNKNOWN`, naming the host's signature and -the available IDs, and, for the catalog defect on an Intel or AMD CPU, -pointing to `--cpu-profile host`. A cold boot with `auto` whose pinned -profile the backend does not support fails with `E_PROFILE_UNSUPPORTED` -(verification step 3) instead of falling back, because that reveals a -profile or hypervisor problem on a served generation; on an Intel or AMD CPU -its error points to `--cpu-profile host`, because a host profile derives -from what the backend supports. A restore always uses the profile recorded -in the snapshot, and never falls back; an explicit `--cpu-profile` must name -the same profile, or be `host` for a host profile (`E_PROFILE_UNKNOWN`). - -The fallback is never silent. The VM worker logs one warning at the default -log level at every cold boot that falls back, wherever `auto` is selected, -the management endpoint included. It starts with the stable marker -`NVX-CPU-PROFILE-FALLBACK:`, which NVX's tools and log scrapers match, names -the host CPU (vendor, display family, model, and stepping, and brand -string), the host profile and its digest, and the limits of a host profile, -including what that cold boot's fingerprint took, and asks the user to -attach the CPU's fingerprint -(`openvmm --hypervisor --cpu-fingerprint fingerprint.json`) to a -CPU profile request, NVX's issue form `.github/ISSUE_TEMPLATE/cpu-profile.yml`, -through a link that prefills the CPU. Maintainers link each request to #408 -or #409, which explain how fingerprints become a pinned profile. An explicit -`--cpu-profile host` warns too, and with the same request where no pinned -profile serves the CPU. Where the host profile that `auto` fell back to -cannot boot, for example because the backend lacks a CPU feature that the -time ABI requires (`E_PROFILE_UNSUPPORTED`), the error keeps its code and -says that `auto` fell back. - -**Host profiles.** A host profile serves a development host that no pinned -profile serves: `--cpu-profile host` selects one on any Intel or AMD host, -and `auto` falls back to one on such a host that no pinned profile serves. -Before it creates -the partition, the VM worker fingerprints the backend on this host, as -`--cpu-fingerprint` does, and derives `.host.v1` from that one -fingerprint under the pinned profiles' derivation policy -(`cpu_profile::derive_host_profile`). The guest therefore sees a -policy-filtered surface, not the host's CPUID: only the policy's data leaves -keep data, the policy's features are cleared, XCR0 stays within the allowed -components, the brand string is generic, and the generation, named `host`, -covers only the host's family, model, and stepping. Every verification step -below applies to it unchanged. The VM worker hands it to the backend with the -partition's configuration (`cpu_profile::PartitionProfile`), so each VM of a -process holds its own host profile. A host profile is neither reviewed nor -immutable: a microcode, firmware, hypervisor, or OS update can change it. Its -fingerprint costs a probe partition and the host identity at every cold boot, -which #394 measured at 2.6 to 5.5 ms on WHP, 11 to 23 ms on KVM, and 52 ms on -MSHV; the cost varies with the host, and the fallback's warning reports what -each cold boot's fingerprint took. -Its snapshots restore only on a host of the same CPU model and stepping whose -hypervisor supports the profile. Only Intel and AMD CPUs have host profiles, -`intel.host.v1` and `amd.host.v1`; another vendor, such as Hygon, fails -with `E_PROFILE_HOST_UNKNOWN`, and a backend that lacks a CPU -feature that the time ABI requires fails with `E_PROFILE_UNSUPPORTED`. Host -qualification never accepts a host profile, including the one that `auto` -falls back to, so CI and benchmark hosts need a pinned one, and `benchmark` -refuses a CPU that no pinned profile serves, or that it cannot identify. - -The `host-cpu-unknown` test hook makes `auto` treat the host CPU as one that -no pinned profile serves, so that CI's runners, which pinned profiles serve, -exercise the fallback (`test-microvm`'s `cpu-profile-fallback` scenario). - -**Verification.** At partition creation, for cold boot and restore, OpenVMM -reports every violation at once, naming the leaf, subleaf, register, and bit: - -1. The profile is valid: schema, canonical encoding, density, the VMM-owned - and runtime-owned mask tables, and the [CPU time bits](#cpu-time-bits) - (`E_PROFILE_TIME_BITS`; a catalog profile that fails is a build defect - caught by unit tests). -2. The host's vendor, family, model, and stepping are in the profile's - generation (`E_CPU_GENERATION`). -3. The backend supports the profile (`E_PROFILE_UNSUPPORTED`): every set - feature bit is a supported bit, every limit (maximum leaves, address - widths) is within the backend's, every enabled XSAVE component has the - same size, offset, and flags, XCR0 and XSS are subsets, and every pinned - MSR value can be presented (an `ARCH_CAPABILITIES` immunity the host lacks - is a violation). The time policy bits are exempt. Each backend derives - its supported surface from capability queries and the host's CPUID, - without a probe partition, which would add 2.6 to 5.5 ms per cold boot - and restore on WHP; the full fingerprint stays in `--cpu-fingerprint`. -4. On restore only: this OpenVMM pins a profile with the same ID and digest, - and the recorded document is its canonical encoding - (`E_PROFILE_UNKNOWN`, `E_PROFILE_DIGEST`), and the recomputed effective - CPUID equals the recorded one (`E_CPU_SURFACE`). Both are comparisons - with precomputed or recomputed values; nothing is hashed or decoded. The - effective-CPUID record replaces the exact-equality CPU contract. A host - profile has no pinned copy: the snapshot's recorded document is the - profile, which the restore preflight and the VM worker decode and accept - only as the canonical encoding of a valid host profile of the recorded ID - whose SHA-256 is the recorded digest (`E_PROFILE_DIGEST`). Step 2 then - requires the capture host's family, model, and stepping. -5. The backend presents the effective CPUID: VP 0's CPUID for every governed - leaf equals it under its masks (`E_CPU_SURFACE`). -6. On MSHV and WHP, which pass reserved entries through, the hypervisor - presents no non-zero CPUID entry outside the profile's tables, apart from - the identity range and the topology leaves (`E_CPU_UNLISTED`). - - `--cpu-fingerprint` checks what a cold boot checks. On MSHV and WHP, it - creates a second probe partition whose feature banks and XSAVE - features derive from the profile, as a cold boot's partition's do, and - reads its VP 0 at the host's candidates (below). The fingerprint's own - probe partition enables every available feature, so on a CET-capable - host, such as a twelfth-generation Core on WHP, it presents CET's XSAVE - components `0xD.11` and `0xD.12`, which no profile enables. No - CET-capable MSHV host has run it: the Skylake-SP hosts predate CET, and - Azure withholds CET from the roots of its Emerald Rapids VMs. On MSHV, - the configured partition is created as a microVM's is, without x2APIC - or SMT, and reads zero at the Skylake-SP host's 7 candidates. - - At every cold boot and restore, step 5's check covers it on VP 0's - view. The candidates are the entries that the host's CPUID enumerates - outside the profile's tables (`cpu_profile::unlisted_cpuid_candidates`), - each read at subleaf 0 if subleaf-independent. The backend reads them - with the governed leaves: 7 more entries on the Skylake-SP hosts - (`0xF.1`, `0x10.1` to `0x10.3`, `0x12.1`, `0x12.2`, and `0x14.1`), 6 - on the Azure 8370C and 8573C runners, whose roots report no Intel PT, - and 12 on the Core i9-12900H, among them `0xD.11`, `0xD.12`, `0x14.1`, - and `0x20`, which its root reads as non-zero, and 2 on the EPYC 7763 - WHP host, `0xD.11` and `0xD.12`. - The supported surface of step 3 cannot serve: it derives from the - root's CPUID, which shows values that a guest does not see there, so - it would fail sound hosts. - On the Skylake-SP hosts, the MSHV and WHP roots read Intel PT's - `0x14.1` as non-zero. A guest's own enumeration stops at `0x14.0`, - which reads zero, so only the host-derived candidates reach `0x14.1`. - - KVM answers reserved entries from the effective CPUID itself, so it - needs no check. - -The catalog has seven profiles, derived from the fingerprints of three -bare-metal hosts (one per backend), eighteen Azure hosts, and one laptop: - -| ID | Generation | Hosts | Source backends | -| --- | --- | --- | --- | -| `intel.skylake-sp.v1` | 6/85, steppings 0 to 4 | The bare-metal hosts | KVM, MSHV, WHP | -| `intel.icelake-sp.v1` | 6/106 | Xeon Platinum 8370C runners | KVM, MSHV, WHP | -| `intel.emeraldrapids.v1` | 6/207 | Xeon Platinum 8573C runners | MSHV, WHP (GitHub-hosted 8573C runners support it on KVM) | -| `intel.alderlake.v1` | 6/151 and 6/154 | One Core i9-12900H laptop (6/154, stepping 3) | WHP | -| `amd.milan.v1` | 25/1 | One Azure Standard_D16as_v5 VM with an EPYC 7763 (25/1, stepping 1) and nested Hyper-V | WHP | -| `amd.genoa.v1` | 25/17 | One GitHub-hosted Actions runner, an Azure VM with an EPYC 9V74 (25/17, stepping 1) | KVM | -| `amd.turin.v1` | 26/2 | One GitHub-hosted Actions runner, an Azure VM with an EPYC 9V45 (26/2, stepping 1) | KVM | - -The Alder Lake profile serves development hosts, not CI. Derived from one -WHP fingerprint, it pins XCR0 `0x7`, because WHP offers that host only x87, -SSE, and AVX state, and leaves Alder Lake hosts on KVM, other SKUs, and -Alder Lake-S unverified; any of them that lacks one of its features fails -with `E_PROFILE_UNSUPPORTED` until a later revision intersects their -fingerprints. The policy needs nothing for the hybrid cores: it clears the -hybrid flag, `CPUID.(7,0):EDX[15]`, and keeps no data in leaf `0x1A`. - -The AMD profiles serve development hosts, and the GitHub-hosted runners where -Copilot cloud agent sessions run, not CI; no CI microVM runner has an AMD -CPU. Each generation takes every stepping of its model: family 25 model 1 for -Milan, Milan-X's 2 included, which differs only in descriptive fields; -family 25 model 17 for Genoa, Genoa-X's included; and family 26 model 2 for -Turin. -Derived from one nested WHP fingerprint, the Milan profile presents what -that WHP offers its guests: XCR0 `0x7`, TOPOEXT with AMD's cache and -topology leaves, and CLZERO, but none of the speculation controls of -`0x80000008` EBX and nothing in `0x80000021`, which that WHP does not offer. -Linux guests of the profile therefore mitigate Spectre v2 with retpolines and -report SSB, SRSO, and TSA as vulnerable on every host. The policy's AMD rules -keep data in `0x80000005`, `0x8000001D`, `0x8000001E`, and `0x80000021`, keep -AMD's speculation controls and immunities where every source host offers -them, except what KVM cannot present, and clear 3DNowPrefetch, which the root -partition of a nested Azure host does not see although WHP presents it to -guests, so that host's cold-boot support check would fail a profile that -pinned it; Linux infers `PREFETCHW` from long mode. The WHP fingerprint -offers PSFD and `BTC_NO` too, which the profile drops: KVM never enumerates -`BTC_NO`, and Linux marks no CPU of family `0x19` or later as affected by -RETBLEED, which is all the bit decides; PSFD is a bit of `SPEC_CTRL`, -which KVM lets a guest access only beside IBRS, STIBP, or SSBD, and stays -only with one of them, as OpenVMM's KVM backend keeps it in KVM's -supported surface. A Milan host on KVM therefore fails the profile on -neither bit: GitHub's EPYC 7763 runners, Azure VMs whose KVM offers more -than that WHP, pass doctor H1 to H4 and the full default `test-microvm` -suite on it, and the host-profile scenarios on `amd.host.v1`. - -The Genoa and Turin profiles each derive from one KVM fingerprint of a -GitHub-hosted Actions runner, an Azure VM with an EPYC 9V74 or 9V45, and -present what Azure offers that VM's KVM. Genoa's XCR0 is `0x7`, and Turin's -is `0xe7`, with AVX-512, AVX-VNNI, and AVX512-BF16. Both have TOPOEXT, -CLZERO, `LFENCE` serialization in `0x80000021` EAX, and the TSA immunities of -`0x80000021` ECX, which Azure presents. They have none of AMD's speculation -controls, which Azure withholds, and, by the policy, neither PSFD, which -their KVM offers without a `SPEC_CTRL` control, nor `BTC_NO`. KVM also -enumerates Intel's speculation controls and `IA32_ARCH_CAPABILITIES` on AMD -hosts (`CPUID.(7,0):EDX` bits 26, 27, 29, and 31), and emulates the MSR; the -runners' KVM sets bit 29. The policy's AMD rules clear those bits, so no AMD -profile pins `IA32_ARCH_CAPABILITIES`, and a profile derived from KVM alone -pins none of them: no AMD CPU has them, and the Milan host's WHP does not -present them. Every GitHub runner of the two generations that ran them, -four with Genoa and one with Turin, passes doctor H1 to H4 and the full -default `test-microvm` suite on its profile, and the host-profile scenarios. - -Milan hosts on MSHV, Genoa and Turin hosts on MSHV and WHP, bare-metal AMD -hosts, and other SKUs are still unverified, and so are the AMD paths of the -MSHV backend: the AMD MSR contract and MSHV's per-VP `0x8000001E` (#409). -`amd.genoa.v1` fails a Genoa host whose hypervisor does not report the TSA -immunities, such as a bare-metal one whose kernel applies the `VERW` -mitigation. A host that lacks one of a profile's features fails with -`E_PROFILE_UNSUPPORTED`, which names `--cpu-profile host`, until a later -revision intersects their fingerprints, which could also add the speculation -controls, and PSFD beside them, that bare-metal hosts offer. - -Sharing costs nothing on Ice Lake-SP and Emerald Rapids, where every backend -offers the same features. On Skylake-SP, used only by the bare-metal -development hosts, the shared profile drops what one backend alone offers: -PKU, UMIP, and FDP_EXCPTN_ONLY (KVM only), FLUSH_L1D (not on WHP), and the -AMD-alias speculation bits in `0x80000008` EBX (KVM only; Intel guests use -the `7.0` EDX equivalents, which every backend presents). Without FLUSH_L1D, -and with `FB_CLEAR` clear because KVM's feature MSR does not offer it on these -hosts (KVM would accept it in a host-initiated write, but v1 pins only what -every backend's surface offers), -Linux cannot confirm the microcode that makes VERW clear the fill buffers. -It therefore reports MMIO Stale Data as "Vulnerable: Clear CPU buffers -attempted, no microcode" on every backend. Before the profiles, KVM -reported "Mitigation: Clear CPU buffers" through FLUSH_L1D, and MSHV -through `FB_CLEAR`. The guest still clears the CPU buffers with VERW at the -same points (`MD_CLEAR`), as its MDS and TAA status, "Mitigation: Clear CPU -buffers", shows, and the host has the microcode, so only the reported status -differs. - -No profile pins the legacy P6 L2-cache MSRs; they raise #GP on every -backend (see [MSRs](#msrs)). - -## TSC rate policy and LAPIC rate rule - -**TSC rate.** At restore, with integer arithmetic in at least 128 bits: - -```text -accept iff |F_d - F_s| * 1_000_000 <= 250 * F_s -``` - -The boundary is accepted. Beyond it, restore is rejected -(`E_TSC_RATE_TOLERANCE`), reporting both rates and the deviation. The -tolerance is recorded in the manifest as `tsc_tolerance_ppm = 250`; any other -recorded value is rejected. Within the tolerance: - -- the TSC is never scaled, and RDTSC is never trapped; -- MSR `0x40000022` keeps returning `F_s`, so the guest's `tsc_khz` is - unchanged; -- guest clocks run fast or slow by at most the deviation (measured hosts - differ by at most 1 ppm), and the restore packet reports the deviation so - the guest pre-compensates it (see - [Wall-clock discipline](#wall-clock-discipline)). - -Every rate is also checked for plausibility, `500 MHz <= F <= 10 GHz` -(`E_TSC_RATE_IMPLAUSIBLE`). A backend that cannot report its native rate is -rejected at cold boot, capture, and restore alike (`E_TSC_RATE_UNAVAILABLE`); -there is no command-line or calibration fallback. - -**Rate deviation.** The restore packet carries the deviation in the unit of -`adjtimex` frequency, ppm scaled by 2^16, rounded to nearest: - -```text -rate_deviation = round((F_d - F_s) * 1_000_000 * 65_536 / F_s) -``` - -Within the tolerance its magnitude is at most 16,384,000. - -**LAPIC rate.** `L` must equal the backend constant (1 GHz on KVM, 200 MHz on -MSHV and WHP), and on restore `L_d` must equal `L_s` exactly -(`E_LAPIC_RATE_MISMATCH`). A counting-mode one-shot timer advances over a -downtime `D` by exactly - -```text -ticks = floor(D * L / 1_000_000_000 / divide) -``` - -where `divide` is the divide-configuration value (1 to 128). If `ticks` is at -least the current count, the count becomes 0 and the timer interrupt is queued -in the restored LAPIC state unless the LVT is masked; otherwise the current -count decreases by `ticks`. A vector queued this way, or one pending in the -restored state, must reach its VP when the VP is released, even if the VP is -halted. MSHV and WHP do not wake a halted VP for an IRR bit written with its -LAPIC state, so their backends assert the vector again (see -[Backend obligations](#backend-obligations)). Without that, on busy 8-vCPU -guests, 29 of 60 time ABI restores on bare-metal MSHV and 5 of 12 on -bare-metal WHP left a CPU halted with its timer vector pending until another -interrupt woke it, about 2 s later or never; with it, 0 of 300 and 0 of 12. -A periodic or TSC-deadline LAPIC timer is never valid: capture and restore -reject one that is armed (`E_LAPIC_PERIODIC`, `E_LAPIC_TSC_DEADLINE`). - -## Cross-vCPU skew bound - -At every guest-observable instant, the TSCs of any two online VPs differ by -at most 1 µs, that is `F_s / 1,000,000` cycles. Linux no longer checks this -(`TSC_RELIABLE`), so it is enforced at four points: - -1. **VMM synchronized set.** Restore writes one target to every instantiated - VP at one host instant and verifies it by read-back before any VP runs - (`E_TSC_SYNC_READBACK`). The VMM introduces no skew of its own. -2. **Host qualification.** `nvx.py doctor` runs a host TSC skew probe (`H5`) - and the guest warp probe on an idle-inducing schedule (`H6`); a host above - 1 µs is not qualified. -3. **CI warp probe.** Every CI microVM boot and restore scenario, and the - conformance, restore-matrix, and soak runs, execute the guest warp probe - on the CI schedule (see [Warp schedules](#warp-schedules-h6-and-ci)) - after boot and after every restore and fail above 1 µs. -4. **Backend live skew.** Each backend shows that live skew stays within the - bound after release. The spikes measured at most 63 ns of ping-pong - offset and 7.2 ns of backward step on KVM; on MSHV, zero warps and an - equal read-back on 20 of 20 restores, with offsets of at most 516 ns on - dual-socket bare metal (bounded by the probe's round trip) and 195 ns on - Azure; and at most 70 ns on WHP. - -The guest warp probe (`nvx-time-probe warp`, built from -`guest/common/nvx-time-probe.c` and installed as `/sbin/nvx-time-probe`) -runs two tests on every pair of online CPUs, each for at least 100 ms per -pair: - -- `max_backward_ns`: the largest backward TSC step observed when the two CPUs - alternately read the TSC under a shared spinlock (the Linux - `check_tsc_warp` method); a lower bound on skew; and -- `max_abs_offset_ns`: the largest pairwise offset estimated by ping-pong - rounds, `t2 - (t1 + t3) / 2` at the round with the minimum round trip. - -Both must be at most 1,000 ns, and no pair may stall for more than 2 s. -Cycle values convert to nanoseconds with `F`. Each offset estimate is -uncertain by half its pair's minimum round trip, which the probe reports -(`max_uncertainty_ns`); a verdict counts only if every pair's uncertainty is -below the bound (`conclusive=1`), and CI fails an inconclusive one. Round -trips are about 0.25 µs within a socket and 1 µs across sockets, so 100 ms -per pair gives more than 10,000 rounds and an uncertainty of at most about -0.5 µs on dual-socket hosts. - -## Downtime semantics and source selection - -Guest monotonic time advances by the downtime `D`, the host time elapsed -between the capture anchor and the restore anchor, plus at most the restore -latency `R` defined below. The guest observes no other discontinuity between -the snapshot `out` and the first restored instruction. - -Capture records, at the capture anchor: - -- `T_c`, the guest TSC of VP 0; -- host UTC in nanoseconds since the Unix epoch; -- host monotonic time in nanoseconds: `CLOCK_BOOTTIME` on Linux, or - `QueryInterruptTimePrecise` × 100 on Windows (both include host suspend); -- the host identity: `/etc/machine-id` on Linux, or the `MachineGuid` - registry value on Windows, as 16 bytes; and -- the host boot identity: `/proc/sys/kernel/random/boot_id` on Linux, or the - `PrefetchParameters\BootId` registry value on Windows (zero-extended), as - 16 bytes. - -A host that cannot provide all of these rejects capture and restore -(`E_HOST_IDENTITY`). Restore takes the same sample at the restore anchor and -selects exactly one source: - -| Condition | Source | `D` | -| --- | --- | --- | -| Same host identity, same boot identity, and same host clock kind | Host monotonic | `monotonic_now - monotonic_capture` | -| Anything else | UTC | `utc_now - utc_capture` | - -`D` must satisfy `0 <= D <= 30 days` (2,592,000 s); otherwise restore is -rejected (`E_DOWNTIME_NEGATIVE`, `E_DOWNTIME_EXCESSIVE`). On the monotonic -path, a UTC delta that differs from `D` by more than 1 s is logged as a host -wall-clock step and does not change `D`. The packet reports the source, so -the guest and tests can tell the two paths apart. - -VM time, the clock of OpenVMM's emulated devices, and the RTC stop with the -state units at quiesce, before the capture anchor. VM time's keeper reads the -host monotonic clock that the anchor samples at the instant VM time stops and -saves the reading with VM time's state, and capture records `Q`, the host -monotonic time from that VM-time cut to the capture anchor, so a host -wall-clock step cannot enter it. Restore advances VM time and the RTC by -`D + Q`, so that they resume in step with the guest TSC, which kept running -through `Q`; the TSC, the LAPIC timers, and the restore packet still use `D`. -`D + Q` must be at most 30 days (`E_DOWNTIME_EXCESSIVE`). `Q` is best effort: -capture omits it, with a warning, when the saved VM-time state lacks its stop -sample or `Q` would be negative or above 30 days, and the snapshot stays -usable. A snapshot without `Q`, including every snapshot that predates it, -restores with `Q = 0`, and OpenVMM logs a warning. - -The restore anchor is the instant of the synchronized TSC set, and the guest -TSC runs from that instant on every backend: KVM never stops it, and MSHV and -WHP resume partition time right after a successful read-back, inside the -synchronized set (see step 13 of the -[restore algorithm](#restore-algorithm)). No backend defers the resume to the -first VP run. The guest monotonic advance across a restore therefore lies in -`[D, D + R]`, up -to the two anchors' pairing errors (each at most 100 µs), where `R` is the -time from the restore anchor to the first restored instruction; it falls -short of `D + R` only by the frozen write (at most about 0.5 ms). Wall-clock -repair absorbs `R` and the pairing errors for `CLOCK_REALTIME`. The KVM spike -kept guest `CLOCK_MONOTONIC` within −0.09 to +2.8 ms of the host interval -across restores with 0 s and 30 s of downtime at 1 to 8 vCPUs; its positive -part came from a host sample taken before the per-VP TSC save, which the -paired capture anchor removes. MSHV measured −0.05 to −0.55 ms with the -resume at the read-back, at parity with KVM, against −0.9 to −1.6 ms when the -first VP run thawed time; warps and restore latency were unchanged. The -legacy path, which stopped guest time during restore setup, left guest -`CLOCK_MONOTONIC` about 400 ms behind the host on every restore on -nested Azure KVM with 512 MiB (7 to 12 ms on bare metal); the time ABI stays -within 1.5 ms there. - -**Anchor pairing bound.** An anchor that pairs a TSC read with a host time -sample (the capture anchor on every backend, and KVM's restore anchor) -brackets one with the other, and its pairing error is half the bracket. Every -backend uses one bound, 100 µs (`MAX_ANCHOR_PAIRING_NS` in -`virt::time_abi`): it keeps the tightest of at most 64 samples, never accepts -a looser pair, and otherwise fails with `E_TSC_ANCHOR`. A nested hypercall -round trip on Azure takes about 12 µs, so nested MSHV pairs at a median of -6.2 µs and WHP on 8573C runners at up to 11 µs; a 10 µs bound would leave -about 1.5× margin over the median and fail restores at random. Downtime needs -only millisecond accuracy, so a 100 µs pairing error is negligible. - -## Restore algorithm - -Restore runs these steps in order. Any failure rejects the restore before a -restored VP runs, with the named code. - -Before worker construction: - -1. Read the bounded manifest. Its version must be 6 with format magic - `OPENVMM_SNAPSHOT_V6\0` (`E_SNAPSHOT_VERSION`, which asks for a - recapture). -2. Validate the time contract and the CPU profile record: `time_abi_version` - is 1, `tsc_tolerance_ppm` is 250, `F_s` is plausible, `L_s` is a backend - constant, identities are 16 bytes, `capture_generation < 2^32 - 1`, the - profile digest is 32 bytes, and the effective CPUID record is whole, - well-formed entries (`E_MANIFEST_TIME`). A recorded `Q` above 30 days is - rejected (`E_DOWNTIME_EXCESSIVE`). Validation computes no digest. -3. Require the destination backend to equal `source_hypervisor` - (`E_BACKEND_MISMATCH`). -4. Require a pinned profile with the recorded ID and digest, whose canonical - encoding is the embedded document byte for byte (`E_PROFILE_UNKNOWN`, - `E_PROFILE_DIGEST`), both compared with the build's precomputed record, - and the host's CPU generation to be in it (`E_CPU_GENERATION`). -5. Preflight the downtime: sample the host clocks, select the source, and - check the bounds of `D` and `D + Q` (`E_DOWNTIME_*`, `E_HOST_IDENTITY`). - The authoritative `D` is measured again in step 12. -6. Validate the rest of the machine contract as today (topology, devices, - attachments, command line, blocks). The command line must not contain - `tsc_early_khz=` or `lapic_timer_hz=` (`E_CMDLINE_CLOCK_TOKEN`). -7. Generate the restore entropy and generation ID, and set - `g = capture_generation + 1`. - -In the worker, with every VP stopped: - -8. Create the partition with the profile CPUID, the identity leaves, and the - time platform declaring `F = F_s` and `L = L_s`. Preflight the backend - before any VP runs and before any VP state is restored: profile support - and effective CPUID (`E_PROFILE_UNSUPPORTED`, `E_CPU_SURFACE`, - `E_CPU_UNLISTED`), identity routing (`E_IDENTITY_ROUTING`), the - synchronized-set primitive (`E_TSC_SYNC_UNSUPPORTED`), and no scaling - (`E_TSC_SCALING_ACTIVE`). MSHV's preflight freezes partition time, which - probes the primitive, and leaves it frozen, as a reset does: steps 9 to 12 - run with it frozen, and at cold boot the first VP run resumes it. It also - reads VP 0's CPUID at its reset state. -9. Read `F_d` and `L_d` and apply the - [rate policy](#tsc-rate-policy-and-lapic-rate-rule) - (`E_TSC_RATE_UNAVAILABLE`, `E_TSC_RATE_IMPLAUSIBLE`, - `E_TSC_RATE_TOLERANCE`, `E_LAPIC_RATE_UNAVAILABLE`, - `E_LAPIC_RATE_MISMATCH`). -10. Restore every state unit while stopped: VM time, chipset and virtio - devices, the time platform (`HV_X64_MSR_TSC_INVARIANT_CONTROL`), the - partition, and every VP. -11. Freeze the instantiated VP set. Every instantiated VP was bound before - step 10 restored its state, and MSHV creates a VP only when it is bound, - so the set is complete; creating a VP after this point is an internal - error (`E_VP_LATE_CREATION`). The PIT's restore in step 10 already - rejected a channel 0 counting in a periodic mode (`E_PIT_ACTIVE`). -12. Synchronized TSC set. The backend takes the restore anchor (one host - instant) and passes its host sample to the orchestrator, which selects - the downtime source, computes `D` and checks its bounds, and returns - - ```text - TSC_target = T_c + floor(D * F_s / 1_000_000_000) - ``` - - (`E_TSC_TARGET_OVERFLOW` if it exceeds 64 bits). The backend writes - `TSC_target` to every instantiated VP at that instant. Per-VP TSC values - from saved VP state are omitted from the VP restore and never applied. A - restore anchor that pairs a TSC read with host time (KVM's common offset) - takes at most 64 samples and fails with `E_TSC_ANCHOR` if no pair is - within 100 µs; a pairing error shifts every VP's TSC alike, by at most - that amount. A frozen write needs no such pairing. Core never calls the - legacy clock entry points (`set_tsc_frequency_hz`, - `advance_snapshot_time`) on a time ABI partition; KVM rejects both there. -13. Read back: every instantiated VP holds the synchronized value as defined - by its backend (`E_TSC_SYNC_READBACK`). A backend that froze partition - time for the set resumes it right after a successful read-back, before - the synchronized set returns: MSHV clears `TimeFreeze`, and WHP calls - `WHvResumePartitionTime`; KVM's TSC never stops. Guest time therefore - runs from the restore anchor on every backend. A failed read-back leaves - time frozen and fails the restore. -14. Advance every VP's counting-mode LAPIC timer by `D` at `L`, and set every - VP's LAPIC state again, always after step 12 and even when no timer is - armed: KVM derives its timer deadline from the guest TSC when the LAPIC - state is set. The same pass rejects an armed periodic or TSC-deadline - timer (`E_LAPIC_PERIODIC`, `E_LAPIC_TSC_DEADLINE`) before any VP runs, - so restore takes no separate pass over the VPs to check them. -15. Advance VM time by `D + Q` and the RTC's UTC by `D + Q` (milliseconds), - with `Q = 0` for a snapshot that does not record it; a `D + Q` above 30 - days rejects the restore before steps 14 and 15 start - (`E_DOWNTIME_EXCESSIVE`). The PIT catches up from its saved VM-time - cursor; it is idle. Steps 14 and 15 touch disjoint state, because no time - ABI LAPIC reads VM time (see the LAPIC rate row of - [Backend obligations](#backend-obligations)), so OpenVMM runs them - concurrently and waits for both before step 16. -16. Seal the time fields of the restore packet: `D`, its source, the rate - deviation, `g`, and the test-hook flag. -17. Start the state units, with host input gated when the restore requires an - acknowledgement; publish the readiness event; release the VPs. Partition - time has run since step 13, so releasing the VPs does not start it, and - no backend resumes time at the first VP run. - -The guest then repairs its clocks (see -[Snapshot agent](#snapshot-agent)) before acknowledging a gated restore. - -**Cold boot.** A cold boot runs the partition steps without a snapshot, in -this order: - -1. Select the CPU profile (see [CPU profiles](#cpu-profiles)) and reject - `tsc_early_khz=` and `lapic_timer_hz=` in the command line - (`E_CMDLINE_CLOCK_TOKEN`); OpenVMM injects neither. -2. Create the partition and preflight the backend as in step 8, before any - VP runs. -3. Read `F_d` and `L_d`, check them (`E_TSC_RATE_UNAVAILABLE`, - `E_TSC_RATE_IMPLAUSIBLE`, `E_LAPIC_RATE_UNAVAILABLE`, - `E_LAPIC_RATE_MISMATCH`), and declare `F = F_d` and `L = L_d`. -4. Set `g = 0`, start the state units, and release the VPs. - -A cold boot writes no TSC: every backend starts the VPs' TSCs together (the -spikes measured at most 63 ns of skew), and skew enforcement points 2 to 4 -apply. - -## Snapshot manifest - -New snapshots use manifest version 6 with format magic -`OPENVMM_SNAPSHOT_V6\0`. Restore accepts exactly version 6; versions 2 -through 5 and every other value are rejected with `E_SNAPSHOT_VERSION`, whose -message tells the operator to recapture the snapshot. The legacy version -branches and their format constants are deleted. Version 6 is the format's -only version, so standard-machine disk snapshots use it too; they carry no -machine contract and no time records. The manifest's fields 9 and 10, the -version 2 artifact digests, are retired and never reused. - -**Capture.** After every VP has stopped at the snapshot boundary and the -state units are quiesced, capture: - -1. asserts that no LAPIC timer is periodic or in TSC-deadline mode and that - PIT channel 0 is not counting periodically (`E_LAPIC_PERIODIC`, - `E_LAPIC_TSC_DEADLINE`, `E_PIT_ACTIVE`); -2. takes the capture anchor: VP 0's TSC paired with a host time sample taken - within 100 µs of it, keeping the tightest of at most 64 samples - (`E_TSC_ANCHOR`), plus the host identities (`E_HOST_IDENTITY`); -3. records the declared rates, the CPU profile, the effective CPUID, and the - process's generation counter. The CPU profile record is the pinned - profile's precomputed digest and canonical encoding, copied, and the - effective CPUID is a packed binary record: capture encodes no document and - computes no digest; and -4. after saving the state units, records `Q` from VM time's saved stop sample - and the anchor's host monotonic time (see - [Downtime semantics](#downtime-semantics-and-source-selection)). - -Each failure of steps 1 to 3 is a rollback-safe capture failure. Step 4 never -fails a capture: where it cannot derive `Q`, it logs a warning and omits it. - -No backend stops guest time for a capture: the guest TSC keeps running while -the VPs are stopped. So a capture that rolls back leaves the guest's clocks -continuous, and the guest sees the quiesce as elapsed time, as it would a long -preemption. On the bare-metal WHP host, the guest's `CLOCK_MONOTONIC` covered -quiesce windows of 0.39 to 0.65 s within 2.2 ms of the host interval, and -`CLOCK_REALTIME` minus `CLOCK_MONOTONIC` moved by at most 0.14 µs. On the -bare-metal KVM host, `CLOCK_MONOTONIC` stayed within −0.38 to +0.40 ms of the -host over requests of 0.2 to 1.1 s. - -**Machine contract changes.** `SnapshotMachineContract` retires protobuf -fields 11 (`capture_wall_clock`), 12 (`tsc_frequency_hz`), 13 -(`tsc_tolerance_ppm`), 14 (`cpu_contract`), 15 (`cpu_contract_sha256`), 17 -(`clock_policy`), and 20 (`apic_frequency_hz`); their numbers are never -reused. It adds two required fields: - -| Field | Number | Type | -| --- | --- | --- | -| `time` | 31 | `SnapshotTimeContract` | -| `cpu_profile` | 32 | `SnapshotCpuProfile` | - -`SnapshotTimeContract` (package `openvmm.snapshot`): - -| Number | Field | Type | Content | -| --- | --- | --- | --- | -| 1 | `time_abi_version` | `u32` | 1 | -| 2 | `tsc_frequency_hz` | `u64` | `F_s`, the declared rate | -| 3 | `tsc_tolerance_ppm` | `u32` | 250 | -| 4 | `apic_frequency_hz` | `u64` | `L_s` | -| 5 | `capture_tsc` | `u64` | `T_c` | -| 6 | `capture_utc_ns` | `u64` | Host UTC at the capture anchor, nanoseconds since the Unix epoch | -| 7 | `capture_monotonic_ns` | `u64` | Host monotonic time at the capture anchor | -| 8 | `host_clock` | `string` | `linux-boottime` or `windows-interrupt-time` | -| 9 | `host_id` | `bytes` | 16-byte host identity | -| 10 | `host_boot_id` | `bytes` | 16-byte host boot identity | -| 11 | `capture_generation` | `u32` | `g` of the captured process | -| 12 | `vm_time_cut_to_anchor_ns` | optional `u64` | `Q`; absent where capture could not derive it and in snapshots that predate it | - -`SnapshotCpuProfile` (package `openvmm.snapshot`): - -| Number | Field | Type | Content | -| --- | --- | --- | --- | -| 1 | `id` | `string` | Profile ID | -| 2 | `sha256` | `bytes` | Profile digest, 32 bytes: the pinned profile's precomputed digest | -| 3 | `profile` | `bytes` | The pinned profile's canonical encoding, at most 1 MiB | -| 4 | `effective_cpuid` | `bytes` | The [effective CPUID](#cpu-profiles), 1 to 1,024 entries in its order (by leaf, then subleaf). Each entry is 44 bytes: eleven little-endian `u32` values, namely the leaf, then 1 and the subleaf (or 0 and 0 for an entry that applies to every subleaf), then the four registers `EAX` to `EDX`, then their four masks | -| 6 | `capture_cpu_signature` | `u32` | CPUID.1:EAX of the capture host, for diagnostics | - -Field 5, a digest of field 4's earlier canonical JSON form, is retired and -never reused. Restore checks the record by comparison: the digest and the -document against the build's pinned record, and the effective CPUID against -its recomputation, encoded the same way. - -Other rules: - -- The effective command line contains no `tsc_early_khz=` or - `lapic_timer_hz=` token, and platform-tier validation no longer requires - one (`E_CMDLINE_CLOCK_TOKEN`). -- The state-unit inventory gains the time platform unit, `time-abi`, which - saves `HV_X64_MSR_TSC_INVARIANT_CONTROL`. -- `source_hypervisor` keeps its meaning and must match exactly. -- Nothing in the manifest depends on the capture host's native TSC rate - except `F_s`, which is the lineage's declared rate. - -## Restore packet v4 and the time-sample selector - -Every byte the guest reads from portb is a port exit, so the time ABI keeps -records small and lets the guest read four bytes per exit. - -### Portb registers - -| Port | Access | Time ABI v1 behavior | -| ---: | --- | --- | -| `0xe9` | Read | With the restore packet or generation ID selected, a 1-, 2-, or 4-byte read returns that many bytes of the selected record, little-endian, zero-filled past its end. Console input reads are unchanged: one byte per read. | -| `0xea` | Read | Status bits 0 to 5 are unchanged. Bit 6 (`0x40`) is always set: the time-sample window exists. Bit 7 is zero. | -| `0xea` | Write | `0xa5` selects the restore packet and `0xa6` the generation ID, as today. `0xa7` latches a fresh time sample into the window at `0xeb` and does not change the `0xe9` selection. | -| `0xeb` | Read | A 1-, 2-, or 4-byte read returns the next bytes of the latched time sample, zero-filled past its end and when no sample is latched. | - -Port `0xeb` belongs to the portb device, whose range becomes -`0xe9..=0xeb`. The time window is separate from `0xe9` because the console -driver polls `0xea` and reads `0xe9` from a timer thread; a sample on the -console stream could interleave with input bytes. The window is not saved -state: restore empties it. Guest users of selectors `0xa5`, `0xa6`, and -`0xa7` serialize their transactions with an exclusive `flock` on -`/run/nvx/portb.lock`. The snapshot agent holds it from its capture request -through repair step 8, and a discipline poll holds it from its first sample -to the published state, so a capture may wait for an in-flight poll, which -is bounded by three samples and two `adjtimex` calls. - -### Restore packet - -OpenVMM exposes a restore packet on every microVM restore, for every tier and -for untiered snapshots; status bit 1 advertises it. Version 4 replaces -versions 1 through 3, which are no longer produced or accepted. - -```text -offset size field - 0 3 magic "OVR" - 3 1 version = 4 - 4 1 flags - 5 1 online_vp_count 0 = no processor target, else 1, 2, 4, or 8 - 6 1 memory_range_count 0 unless MEMORY_TARGET - 7 1 reserved = 0 - 8 4 generation u32, little-endian, g >= 1 - 12 4 rate_deviation i32, little-endian, ppm scaled by 2^16 - 16 8 downtime_ns u64, little-endian, D - 24 8 utc_ns u64, little-endian, host UTC at selection - 32 16*n ranges n x { u64 gpa_start, u64 length }, LE - 32+16n 64 entropy -``` - -Flags: - -| Bit | Name | Meaning | -| ---: | --- | --- | -| 0 | `DOWNTIME_UTC` | `D` came from the UTC delta; clear means same-boot host monotonic time | -| 1 | `MEMORY_TARGET` | An explicit RAM target was requested; `memory_range_count` is valid and may be 0 | -| 2 | `ACK_REQUIRED` | Host input is gated; the guest must acknowledge through `0x605` after repair | -| 3 | `TEST_HOOKS` | A test hook is active: it can alter the downtime source, `D`, UTC, the rate deviation, the handling of time samples, or the CPU profile | -| 4–7 | | Zero; the guest rejects a packet with any of them set | - -`utc_ns` is latched when the guest first writes selector `0xa5` after the -restore; every other field is sealed before the first restored VP runs (step -16 of the [restore algorithm](#restore-algorithm)). A packet without ranges is -96 bytes, 24 four-byte reads. Status bits 2 to 4 remain and agree with the -packet; the packet is authoritative. `ACK_REQUIRED` replaces the guest's -inference of gating from the tier and targets, so an untiered guest never -writes `0x605` after an ungated restore. - -A VM process exposes at most one packet, and reading its last byte consumes -it: status bits 1 to 4 clear and later `0xa5` writes select nothing. A guest -that is captured again in the same process, including after a rejected -capture, therefore never sees the earlier packet. It also ignores a packet -whose `g` equals its recorded `g`. - -### Time sample - -Writing `0xa7` to `0xea` latches a 16-byte sample into the window at `0xeb`: - -```text -offset size field - 0 1 version = 1 - 1 1 flags bit 3 TEST_HOOKS; other bits zero - 2 2 reserved = 0 - 4 4 generation u32, little-endian, g of this VM process - 8 8 utc_ns u64, little-endian, host UTC at the latch -``` - -The guest pairs `utc_ns` with its own clock: it reads `CLOCK_REALTIME` as -`t0` immediately before the selector write and as `t1` immediately after it. -The host instant lies in `[t0, t1]`, so the offset is - -```text -theta = utc_ns - (t0 + t1) / 2 uncertainty epsilon = (t1 - t0) / 2 -``` - -The same bracket applies to `utc_ns` in the restore packet, around the first -`0xa5` write. A sample costs one write and four reads. OpenVMM cannot observe -a running VP's TSC on every backend, so the guest forms the UTC and TSC pair -itself through this bracket; its `CLOCK_REALTIME` is derived from the TSC. - -### Uncertainty bounds - -The guest accepts a pairing only if `epsilon` is within a bound, retries -otherwise, and then fails with a stable code: - -| Use | Bound | Attempts | When every attempt exceeds the bound | -| --- | --- | --- | --- | -| Restore repair (packet bracket, then time samples) | 1 ms | 3 | `G_REPAIR_SAMPLE`: event and power-off with status 195 | -| Initial synchronization at boot, before shell-ready (time samples) | 1 ms | 3 | `G_CONFORMANCE_C12`: event and power-off with status 193 | -| Discipline poll (time samples) | 50 µs | 3 | `G_SAMPLE_UNCERTAIN`: the poll is skipped and the code is recorded in the state file; never fatal | - -`epsilon` is half the round trip of one PMIO write exit plus two -`CLOCK_REALTIME` reads, so it stays far below both bounds on every backend. -Measured from a guest with a release build of OpenVMM (`ea570db7e`) and a -probe that brackets each selector write with `CLOCK_REALTIME` reads, two runs -of 2,000 samples per boot at 1, 2, and 4 or 8 vCPUs, which made no -difference: - -| Backend | Host | `epsilon` p50 / p99 | Worst sample | -| --- | --- | --- | --- | -| KVM | Bare metal (Linux 7.0) | 4.0 to 4.1 µs / 4 to 15 µs | 33 µs | -| KVM | Nested Azure 8370C (Linux 6.6) | 3.8 to 4.1 µs / 4 to 9 µs | 75 µs | -| MSHV | Bare metal | 6.3 to 6.4 µs / 6.8 to 7.1 µs | 37 µs | -| MSHV | Nested Azure 8573C | 12.7 to 13.0 µs / 15 to 25 µs | 102 µs | -| WHP | Bare metal | 20.7 to 21.3 µs / 29.4 to 30.5 µs | 52 µs | -| WHP | Nested Azure 8370C | 24.8 to 25.2 µs / 36.5 to 41.9 µs | 794 µs | -| WHP | Nested Azure 8573C | 30.5 to 30.8 µs / 46.0 to 46.4 µs | 251 µs | - -The second run of each boot overlaps the console's drain of the first run's -output, and it has the higher KVM p99s. The WHP rows come from the WHP -agent's probe (two runs of 2,000 samples). A sample above the 50 µs -discipline bound is rare except on WHP, whose p99 approaches it; a poll then -retries, and a poll whose three attempts all fail is skipped without harm -(`G_SAMPLE_UNCERTAIN`). - -### Generation counter and generation ID - -- The generation counter `g` is 0 in a cold-booted process and - `capture_generation + 1` in a restored one. It orders restores within a - snapshot lineage; clones of one snapshot share it. The packet and every - time sample carry it, so a guest observes a restore between two samples as - a change of `g`. -- The generation ID is unchanged: 16 random bytes, unique to each VM process, - selected with `0xa6`, and equal to the first 16 entropy bytes when a - restore packet exists. It distinguishes clones. -- After a restore, the guest requires `g` in the packet to equal its - recorded `g` plus one, and the generation ID to differ from the recorded - one. Either mismatch fails repair. - -## Guest obligations - -The guest's time component (`nvx-time` in `guest/common`) performs the -conformance checks, runs the violation watcher and the wall-clock discipline -as one daemon, executes the snapshot agent's time steps, and reports status -on demand. Every guest mode starts it: init's interactive and test paths, -`nvx_exec`, the sandbox agent, and the managed agent. - -Production paths keep the console quiet: the time component prints no -marker or status line, and only a [violation event](#violation-watcher) -reaches the console. Each console byte costs one or two port exits, about 10 -to 12 µs on bare-metal KVM and an estimated 20 to 45 µs nested on Azure, so a -110-byte boot marker and a 117-byte restore marker would cost 1 to 5 ms on -the measured paths. State goes to the [state file](#wall-clock-discipline), -and `nvx-time status` prints it on demand. - -### Conformance checks and the `NVX-TIME-ABI` marker - -Before shell-ready, init does only what boot correctness needs. Right after -mounting `/proc`, `/sys`, and `/dev`, it takes a time sample within the -[uncertainty bound](#uncertainty-bounds) and steps the clock to host UTC -(`C12`), so the workload starts on host time; this step is not counted as a -discontinuity. Init then reports shell-ready (or starts the workload, in -modes without a shell) and runs every other boot check asynchronously, -starting 150 ms after its time ABI boot step (`nvx-time boot`) at -`SCHED_IDLE`, and at normal priority from 100 ms after they start (see -[Snapshot agent](#snapshot-agent)). The boot step is the anchor because -every mode reaches it, whereas a mode that execs an agent reports readiness -later, on the agent's own protocol. In shell mode the boot step comes 15 to -44 ms before shell-ready on Alpine on bare metal, up to 96 ms before it on -the nested Azure WHP runners, and 26 to 84 ms before it on Ubuntu at 1 to -8 vCPUs, so the checks start about 54 to 135 ms after it. Because they start -at `SCHED_IDLE`, a start closer to shell-ready costs the boot path nothing. -The checks start the daemon when they pass. Every check stays -fail-fast: a failure powers the guest off with status 193, even if the -workload is running. A guest that powers off before its checks finish skips them, which -is why CI takes its evidence from `nvx-time status`, which waits for them. - -CPUID is executed on every online CPU (the checker pins itself to each CPU in -turn, or runs one thread pinned to each CPU), and MSRs are read through -`/dev/cpu//msr`. A CPU's VP index is its Linux CPU number, because CPUs -come online as a prefix in APIC-ID order. CPUID tables are per VP on KVM, so -`C1` and `C2` run on every CPU; the MSRs other than the VP index are -partition-wide on every backend, so `C3` reads them on CPU 0 only. - -| ID | Check | Phases | -| --- | --- | --- | -| `C1` | Identity leaves `0x40000000..=0x40000005` equal the [identity table](#hypervisor-identity), with `C` equal to the number of possible CPUs; the explicit zero leaves `0x40000006..=0x4000000f` and `0x40000080..=0x40000082` are zero | boot, restore (new CPUs) | -| `C2` | The [CPU time bits](#cpu-time-bits) | boot, restore (new CPUs) | -| `C3` | MSR `0x40000002` equals the CPU's VP index on every CPU; on CPU 0, `0x40000022` equals `F` with `floor(F / 1000)` equal to the kernel's `cpu MHz` in kHz, `0x40000023` equals 1,000,000,000 or 200,000,000, `0x40000118` equals 1, and reads of `0x40000000`, `0x40000001`, and `0x40000020` fail with `EIO` | boot; restore (CPU 0 `0x40000022` only) | -| `C4` | The kernel log contains `Hypervisor detected: Microsoft Hyper-V`, `Hyper-V: privilege flags low 0x8860,`, `Hyper-V: LAPIC Timer Frequency: 0x989680` or `0x1e8480`, and `clocksource: Switched to clocksource tsc` as its last clocksource switch; it contains no record matching a watcher pattern other than `G_CLOCKSOURCE_SWITCH`, and none containing `Fast TSC calibration`, `Refined TSC clocksource calibration`, `kvm-clock`, or `APIC timer: using supplied frequency` | boot | -| `C5` | `/proc/cpuinfo` flags as listed in [Clocksource, tick, and PIT](#clocksource-tick-and-pit) on every CPU | boot, restore (new CPUs) | -| `C6` | `current_clocksource` is `tsc`; `available_clocksource` lists `tsc` and no other clocksource except `refined-jiffies` and `jiffies`, which Linux lists only before the reading CPU's first tick (after boot it is `tsc` alone) | boot, capture, restore | -| `C7` | `/proc/timer_list`: every online CPU's tick device is `lapic` with `hrtimer_interrupt` in one-shot mode; no `pit` or `hpet` device; no broadcast device | boot, capture, restore (deferred) | -| `C8` | `/sys/bus/vmbus` and `/sys/devices/system/cpu/cpufreq/policy0` are absent; `rcu_cpu_stall_suppress` is 0 and `rcu_cpu_stall_timeout` is 21 | boot | -| `C9` | `/proc/cmdline` contains none of `tsc_early_khz=`, `lapic_timer_hz=`, `notsc`, `nolapic`, `nolapic_timer`, `tsc=unstable`, `hpet=force`, or `clocksource=` with a value other than `tsc` | boot | -| `C10` | The time daemon is running and has recorded no violation; `/sys/kernel/rcu_stall_count` is 0; at boot, the state file is published and the daemon started | boot, capture, restore | -| `C11` | Debug kernel only: `/proc/sys/kernel/soft_watchdog` is 1 and `/proc/sys/kernel/hung_task_timeout_secs` is nonzero | boot | -| `C12` | Before shell-ready: a time sample within the [uncertainty bound](#uncertainty-bounds) is obtained, and the clock is stepped to host UTC | boot | -| `K1` | Kernel integrity, not a clock property: the kernel's boot-time W+X audit logged no `Found insecure W+X mapping` record | boot | - -Capture and restore checks never wait. The boot `C7` check and the deferred -restore `C7` check may poll for at most 200 ms, because a CPU switches to -one-shot mode at its first tick. - -**Exhaustive check (CI only).** The CI conformance suite also runs -`nvx-time exhaustive`, provided by the guest. It is a test tool: it reports -and exits instead of powering off, and production boots never run it. Every -check runs on every online CPU: - -| ID | Check | -| --- | --- | -| `X1` | The explicit zero leaves are zero, and every other leaf in `0x40000006..=0x400000ff` returns all zeros or the Intel out-of-range result (the highest basic leaf's result for the same subleaf), as the [identity rules](#hypervisor-identity) require | -| `X2` | No base `0x40000100..=0x4000ff00` (step `0x100`) carries `KVMKVMKVM` or another hypervisor signature; the NVX kernel has no KVM guest support, so the boot check leaves this static backend property to CI | -| `X3` | Every `C3` MSR | -| `X4` | `0x40000118` accepts writes of 0 and 1, each read back, and rejects 2; the check leaves it at 1 | -| `X5` | Writes to every read-only identity MSR fail | -| `X6` | Reads of `IA32_TSC_ADJUST` and `IA32_TSC_DEADLINE` fail | - -It prints one line per check and CPU, then a summary, and exits with status -0 if every check passed and 1 otherwise: - -```text -NVX-TIME-ABI-EXHAUSTIVE: v=1 check= cpu= status= detail="" -NVX-TIME-ABI-EXHAUSTIVE: v=1 status= cpus= failures= -``` - -**Status command.** Checks print nothing on success; they record their -result in the [state file](#wall-clock-discipline). `/sbin/nvx-time status` -prints it on demand: CI scenarios and the matrix driver run it over the -console when they need evidence, outside measured paths, and production -paths never do. It waits until no recorded check is pending, for at most -30 s. A missing state file or boot record counts as pending, because every -boot runs the boot check, and step 8 of the [snapshot agent](#snapshot-agent) -records the restore check as pending before the workload resumes, so a -`status` run after a restore always waits for that restore's deferred work. -It then prints one line per recorded phase, in the order boot, capture, and -restore, then the runtime line: - -```text -NVX-TIME-ABI: v=1 phase= status= cpus= tsc_hz= lapic_hz= generation= elapsed_us= cpu_us= -NVX-TIME-ABI: v=1 phase=runtime status= generation= discontinuities= offset_ns= uncertainty_ns= rejected_samples= last_sample_error= -``` - -Each phase line reports its own check: - -- `cpus` is the number of online CPUs the check covered. -- `generation` is `g` when the check ran: 0 for boot, the capture's `g` for - capture, and at least 1 for a restore. -- The capture line comes from the checks before the capture request (step - 1), which the snapshot carries, so a restored guest still reports it. -- `status` reports the latest capture, which in a restored guest is the - capture that produced its snapshot, and the restore of this process: a - restored process never captures, and every restore is a new process. -- The runtime line describes the guest at the time `status` runs. - -No other line uses the `NVX-TIME-ABI:` prefix, so a harness can parse every -line by its `phase`. - -`status` exits with status 0 when every phase line reports `ok`. If a check -is still pending after 30 s, its line reports `status=pending`, without -`elapsed_us` or `cpu_us`, and the exit status is 1. - -At boot, the checks' work is the checks and the daemon start, but not the -150 ms delay after the boot step; at capture, -the step 1 checks of the [snapshot agent](#snapshot-agent). At restore, it -is steps 6 to 12 and the step 11 checks, but not step 13's 150 ms delay, the -grace period wait, or the deferred `C7`. `elapsed_us` is that work's wall -time, including any wait for a CPU at `SCHED_IDLE`; it bounds the fail-fast -latency and is not budgeted. `cpu_us` is the CPU time that the `nvx-time` -processes spend on it (`CLOCK_PROCESS_CPUTIME_ID`, summed over their -threads); other processes, such as the shell's children in steps 9 and 10, -are not counted. At restore, `cpu_us` counts only step 13's part: the -daemon's preparation, including its side of the fork that runs the checks, -and the step 11 checks. Steps 6 to 12 are the readiness -path, whose cost the restore latency gate already measures, so `cpu_us` is -the background cost to the workload. It includes interrupts that arrive -while the checks run, and it scales with the host processor's clock: a host -idling at a low frequency runs the checks more slowly, and because the -Hyper-V identity gives the guest no steal-time accounting, host preemption is -charged to the checks too. Fleet validation records every `cpu_us` sample -and checks it after the run against the backend's budget for the phase, with -one exempt sample and a triage rule for samples over budget, both in -[Performance expectations](#performance-expectations-and-acceptance-gate); -CI reports it without gating on it. - -A failed check prints no marker: it emits the violation event with code -`G_CONFORMANCE_` (`G_KERNEL_WX` for `K1`) and powers off with status 193, -so `status` never reports a failure. - -**Report-only mode (test only).** The kernel token `nvx_time_abi=report-only` -makes the checks and the watcher report instead of powering off, so the guest -can be evaluated under an OpenVMM that does not implement the time ABI. -Violations print `NVX-TIME-REPORT-VIOLATION` lines with the event's fields. -`nvx-time status` prints its phase lines with the `NVX-TIME-REPORT` prefix, -`status=fail` for a failed check, and ` failures=` appended, and it exits -with status 1 if any check failed. The state file carries them: in this mode -only, a phase's `_status` key may be `fail`, and `boot_failures`, -`capture_failures`, and `restore_failures` count each check's failures. -These prefixes and keys are reserved for this mode; production images never -set the token, and harnesses never accept a report-only boot as conformant. - -### Violation watcher - -The boot check opens `/dev/kmsg`, seeks it to its end, and only then reads -the whole kernel log once with `syslog(SYSLOG_ACTION_READ_ALL)` for `C4`. The -daemon inherits that descriptor and follows it, so no record is missed; -records logged in between are seen twice, and identical lines are one event. -A log buffer that wrapped before the boot check fails `C4`, because its -required records are missing. The daemon sets `oom_score_adj` to -1000. A -record matches when its message text contains every substring of a row: - -| Code | Substrings | -| --- | --- | -| `G_TSC_UNSTABLE` | `Marking TSC unstable due to ` | -| `G_CLOCKSOURCE_UNSTABLE` | `timekeeping watchdog on CPU` and `as unstable because the skew is too large` | -| `G_CLOCKSOURCE_SWITCH` | `clocksource: Switched to clocksource ` (any switch after the boot check) | -| `G_CLOCKSOURCE_SKEW` | `clocksource: ` and ` ahead of CPU `, or `clocksource: ` and ` behind CPU ` | -| `G_TSC_WARP` | `TSC synchronization [CPU#`, `cycles TSC warp between CPUs`, or `TSC warped randomly between CPUs` | -| `G_TSC_ADJUST` | `TSC ADJUST` | -| `G_RCU_STALL` | `rcu: INFO: ` and one of ` detected stalls on CPUs/tasks`, ` self-detected stall on CPU`, or ` detected expedited stalls` | -| `G_RCU_STARVED` | `rcu: ` and one of ` kthread starved for ` or ` kthread timer wakeup didn't happen for ` | -| `G_SOFT_LOCKUP` | `watchdog: BUG: soft lockup - CPU#` | -| `G_HARD_LOCKUP` | `Watchdog detected hard LOCKUP` | -| `G_HUNG_TASK` | `INFO: task ` and ` blocked for more than ` | -| `G_UNCHECKED_MSR` | `unchecked MSR access error` | -| `G_APIC_ID_MISMATCH` | `APIC ID mismatch` (Linux found a CPUID APIC identity that differs from the CPU's APIC: a backend did not present a per-VP field of the [effective CPUID](#cpu-profiles)) | -| `G_KMSG_OVERRUN` | A read of `/dev/kmsg` fails with `EPIPE` (records were lost) | - -Before the boot check passes, a match is reported by `C4`. Afterwards it is -a runtime violation: the daemon emits the event and powers off with status -194. The daemon also reads `/sys/kernel/rcu_stall_count` at every discipline -poll and reports `G_RCU_STALL` if it is nonzero, so a stall is caught even if -its log record is lost. - -A violation event is one line of at most 512 bytes: - -```text -NVX-TIME-ABI-VIOLATION: v=1 code= source= phase= generation= boottime_ns= detail="" -``` - -The guest writes it to `/dev/kmsg` at priority 2, to the portb data port -`0xe9` (through `/dev/port` or `outb`), and to the console, on every path: -violations are rare, and a diagnosable failure is worth the console bytes. -Consumers treat identical lines as one event. The guest then writes the -status to the shutdown port `0x604` (directly, or through `nvx-exit`); its -first byte becomes the OpenVMM process exit status. - -### Snapshot agent - -`nvx-snapshot` performs these steps for every tier and for untiered -snapshots. Steps marked "debug" apply only to the CI debug kernel, which -builds the soft-lockup and hung-task detectors. - -Before the capture request: - -1. Wait until no boot check is pending, for at most 30 s, as - `nvx-time status` does: a capture right after shell-ready may find the - asynchronous boot checks still running, and the daemon, which `C10` - requires, starts only when they pass. A timeout fails the capture with - `G_CONFORMANCE_C10` (193). Then run the capture checks (`C6`, `C7`, - `C10`), which do not wait and print nothing; they record - `capture_status=ok` in the state file, which the snapshot carries. -2. Save `/sys/module/rcupdate/parameters/rcu_cpu_stall_suppress` and write 1. - Values that an earlier request saved and never restored refuse the - capture. -3. Debug: save and zero `/proc/sys/kernel/soft_watchdog` and - `/proc/sys/kernel/hung_task_timeout_secs`. -4. Apply the existing freezer and scratch barriers. -5. Write the capture request to `0x605`. - -A committed capture terminates the source, so the write returns in the -source VM only when the capture had no destination or failed before its -commit point and OpenVMM resumed the guest. Step 6's status read tells the -cases apart: bit 1 is clear in the source, which has no restore packet and -keeps its generation, and set in a restored process, whose packet carries -the next `g`. Reads of `0x605` return `0xff`; a capture reports no status. -In the source, the agent removes the barriers and then restores the values -saved in steps 2 and 3 (`nvx-time cancel-capture`); if any of these fails, -it powers the guest off with status 1. A restored process runs these steps. -Steps 6 to 10 and 12 are the readiness path: nothing else runs before the -acknowledgement. - -6. Read the status from `0xea`. Bit 1 is always set after a restore. -7. Read `CLOCK_REALTIME` as `t0`, write `0xa5` to `0xea`, read it as `t1`, - and read the packet with four-byte `inl` reads. (`rep insl` would take a - single exit on KVM, but OpenVMM's instruction emulator, which serves - string port I/O on MSHV and WHP, raises #GP for it at CPL 3 although the - TSS I/O permission bitmap grants the port.) Validate the magic, version, - reserved bits, counts, `g`, and the generation ID (`G_REPAIR_PACKET`, - `G_REPAIR_GENERATION`). -8. Set the wall clock. Compute `theta` and `epsilon` from the bracket. If - `epsilon` exceeds 1 ms, take up to two time samples and keep the one with - the smallest uncertainty; if none is within 1 ms, fail - (`G_REPAIR_SAMPLE`). Apply `adjtimex` with `ADJ_SETOFFSET | ADJ_NANO` and - `time = theta`, then `ADJ_FREQUENCY | ADJ_STATUS` with - `freq = -rate_deviation` (clamped to ±500 ppm) and - `status = STA_PLL | STA_NANO` (`G_REPAIR_CLOCK`). Record the restore - discontinuity and the new `g`, with `restore_status=pending` and - `restore_generation=g` in the same state-file update, so that - `nvx-time status` waits for this restore. -9. Run the existing processor and memory activation. -10. Run the existing entropy and identity repair for the restore path - ([restore entropy](#restore-entropy)). The RTC-based wall-clock refresh - is removed; `instance-checkpoint` restores also get step 8. -11. Defer the restore checks to step 13: `C1`, `C2`, and `C5` on newly - onlined CPUs; `C3` for CPU 0; `C6`; and `C10`. They do not wait, and a - failure still emits its violation event and powers off with status 193. -12. If `ACK_REQUIRED` is set, write the acknowledgement (2) to `0x605`. -13. After the acknowledgement, or after step 10 when none is required, signal - the daemon, which finishes the restore asynchronously, starting 150 ms - later at `SCHED_IDLE`: - - it runs the step 11 checks; - - it waits for the [grace period release](#rcu-grace-period-release); - - it restores the values saved in steps 2 and 3 - (`G_REPAIR_SUPPRESSION`); - - it runs the deferred `C7` check and records the restore check in the - state file; - - it restarts the discipline at the fast cadence. - - It prints nothing unless a check fails. - -No restore check runs before the acknowledgement. The post-acknowledgement -checks are `C1`, `C2`, `C3`, `C5`, `C6`, `C7`, and `C10`. - -Steps 6, 7, and 8 run in one helper process. For an untiered restore with no -processor or memory to activate, that helper runs every step through 12 and -signals the daemon, so no other process starts on the readiness path. Each -extra process costs 2 to 4 ms after a restore (fork and exec under demand -faulting, measured on KVM). - -Step 13 starts 150 ms after the acknowledgement (or after step 10, when none -is required), and the asynchronous boot checks start 150 ms after the boot -step. Starting them earlier slows the guest: their checks contend -with the readiness path on the other vCPUs, even at `SCHED_IDLE`. On WHP -(bare metal, 512 MiB, measurement-only builds), checks right after the -acknowledgement added 6 to 12 ms to 2-vCPU restores, whereas a 100 ms -delay left them 0.8 and 1.2 ms over the release guest at 1 and 2 vCPUs. On -bare-metal MSHV, boot checks right after shell-ready added about 3 ms -to 2-vCPU cold boots. KVM showed no clear effect either way. The delay is -150 ms rather than 100 ms because a workload's own activity right after -readiness must not meet the checks either: the guest-exit window of -`network_snapshot_restore_wall` falls 110 to 130 ms after readiness on WHP -and 87 to 97 ms on KVM, which checks starting at 100 ms could overlap. - -Step 13 and the asynchronous boot checks run at `SCHED_IDLE`, so on few -vCPUs they never take a CPU from the restored or starting workload. Work -that has not finished 100 ms after it starts continues at normal priority. -Every check stays fail-fast: a failing check powers the guest off about -150 ms after the acknowledgement or the boot step on an idle guest, and at -most about 250 ms plus the checks' own run time after it on a CPU-bound -one, where `SCHED_IDLE` alone would allow about 0.5 s or more. The `elapsed_us` -and `cpu_us` of both exclude the 150 ms delay. `nvx-time status` waits for -this deferred work, within its 30 s bound. The watcher and the discipline -always run at normal priority, and stall suppression stays set until the -release. Repair failures emit a violation event and power off with status -195. - -### Restore entropy - -Restoring a snapshot also restores the guest kernel's CRNG state, so every -restore packet carries 64 bytes of fresh host entropy. Step 10 uses them -according to the restore path; the tiers follow the clone and resume -policies of [Time and entropy](snapshot-and-restore.md#time-and-entropy): - -| Restore path | Guest CRNG | Machine identity | -| --- | --- | --- | -| `platform` or `workload-start` tier (clone policy) | Reseeded before the acknowledgement: `nvx-reseed` credits the 64 bytes to the kernel entropy pool (`RNDADDENTROPY`) and forces a CRNG reseed (`RNDRESEEDCRNG`) | Machine ID and hostname refreshed; the runtime hook runs if present, and `workload-start` requires it | -| Untiered, with processors or memory to activate | Reseeded before the acknowledgement, as for the clone policy | Refreshed as for the clone policy | -| `instance-checkpoint` tier (resume policy) | Not reseeded, by design: only the generation ID is refreshed | Preserved | -| Untiered, with nothing to activate (`online_vp_count` and `memory_range_count` both 0) | Not reseeded: `nvx-time` writes the 64 bytes to `/run/nvx/restore-entropy` for the caller of `nvx-snapshot` | Preserved | - -Each path's handling is unchanged from before the time ABI. v1 changes only -the delivery: the agent used to read packets v1 to v3 a byte at a time, and -on the last path it did not read the packet, leaving it to its caller. It -now reads packet v4 on every restore, for the time fields, and checks `g` -and the generation-ID change first (step 7). - -On the last path, VMs restored from one snapshot share the snapshot's CRNG -state until something reseeds it, such as a caller that credits -`/run/nvx/restore-entropy` with `nvx-reseed`. Whether that path should -reseed before readiness, given that one snapshot can be restored more than -once, is an open policy question; v1 keeps the existing behavior. - -### RCU grace period release - -Stall suppression is required: on KVM, restores after 30 s of downtime -without it report `rcu_preempt self-detected stall` at 1 and 8 vCPUs, because -jiffies jump by `D`; with it, no stall appears at 1, 2, 4, or 8 vCPUs. - -Stall suppression may be released only after the grace period that was in -flight at capture has ended, or the restored guest reports a false stall. -`membarrier(MEMBARRIER_CMD_GLOBAL)` does not guarantee this: Linux 6.18 skips -`synchronize_rcu()` when one CPU is online, and with `rcupdate.rcu_expedited=1` -it runs an expedited grace period, which does not end the in-flight normal -one. The daemon therefore: - -1. saves `/sys/kernel/rcu_normal` and writes 1, so every synchronous grace - period, expedited or not, waits for a normal grace period; -2. mounts a private `tmpfs` at `/run/nvx/rcu-sync` and unmounts it (on every - unmount, Linux 6.18 `namespace_unlock()` waits in - `synchronize_rcu_expedited()`, which now waits for a full normal grace - period started after the call, on any number of CPUs); -3. restores `/sys/kernel/rcu_normal`; and -4. releases the suppression. - -If step 2 has not returned after `rcu_cpu_stall_timeout` seconds, the daemon -releases the suppression anyway: the grace period is genuinely stuck, and the -kernel then reports the real stall. - -### Wall-clock discipline - -The daemon keeps `CLOCK_REALTIME` on host UTC through the kernel's PLL: - -- **Cadence.** A poll every 16 s for the first four polls after the boot - check or a restore, then every 64 s. -- **Sample.** Up to three time samples per poll; keep the first with - `epsilon <= 50 µs`. Otherwise skip the poll, count a rejected sample, and - set `last_sample_error=G_SAMPLE_UNCERTAIN` in the state file; nothing is - printed. The next accepted sample resets `last_sample_error` to `none`. - A sample whose `g` differs from the recorded `g` is discarded: a restore - happened, and restore repair resets the discipline. -- **Step.** If `|theta| >= 128 ms`, apply `ADJ_SETOFFSET | ADJ_NANO` with - `time = theta`, count a discontinuity, and reapply the frequency and - status. -- **Slew.** Otherwise apply `ADJ_OFFSET | ADJ_STATUS | ADJ_NANO | - ADJ_TIMECONST | ADJ_MAXERROR | ADJ_ESTERROR` with `offset = theta` - nanoseconds, `status = STA_PLL | STA_NANO` (clearing `STA_UNSYNC`), - `constant = 4` at the 16 s cadence or 6 at the 64 s cadence, - `maxerror = ceil((|theta| + epsilon) / 1000)` µs, and - `esterror = ceil(epsilon / 1000)` µs. A clear `STA_UNSYNC` reports the - clock to applications as synchronized, with these error bounds. It also - makes the kernel write the CMOS RTC within a second and then every - 11 minutes, which v1 accepts (see - [Performance expectations](#performance-expectations-and-acceptance-gate)). -- **Bounds.** The kernel limits the frequency to ±500 ppm; the declared rate - tolerance uses at most 250 ppm of it. -- **Convergence.** The boot check and restore repair step the clock, and - restore repair corrects the TSC's rate deviation. Neither corrects the rate - at which the host's own UTC clock runs against its TSC, which host time - synchronization slews by a few ppm. The PLL's frequency integrator absorbs - that rate over about an hour at the 64 s cadence. Until then the offset - settles near 256 s times the rate, behind host UTC when the host's UTC runs - fast and ahead when it runs slow: 1.76 ms behind at the +6.88 ppm of an - Azure 8370C WHP runner, about 3.2 ms ahead at the −12.89 ppm of an 8573C - one, and 4.06 ms behind at the +16.07 ppm of the bare-metal WHP host, - against 4.11 ms predicted. The bare-metal MSHV host, at +14.82 ppm, was - 3.50 ms behind after 12 minutes and still approaching its 3.79 ms plateau. - The bare-metal KVM host, at −5.98 ppm, was 1.45 ms ahead after 12 minutes, - near its 1.53 ms plateau. Where the host's own UTC is being slewed, as on a - nested Azure MSHV host, the offset follows the changing rate. A skipped poll - (`G_SAMPLE_UNCERTAIN`) lets the offset grow at the remaining rate for - another 64 s, and the nested WHP runners, whose sample uncertainty - approaches the 50 µs bound, skip some: the 8573C runner reached 4.1 ms after - two skipped polls in a row. All of these are far below the step threshold. - -The discipline never powers off the guest: a host wall-clock step is -followed, not reported as a violation. - -**State file.** The time component publishes `/run/nvx/time/state` -from the start of the boot checks, replaced atomically with `rename(2)` after -every change, and `nvx-time status` reads it. The sandbox agent bind-mounts -`/run/nvx/time` read-only into the container at the same path. The file holds -`key=value` lines: - -| Key | Content | -| --- | --- | -| `version` | 1 | -| `generation` | `g` | -| `boot_status`, `capture_status`, `restore_status` | `pending` while the phase's check runs, then `ok`; absent until the phase first runs (a failed check powers the guest off). The capture keys hold the latest capture, and the restore keys this process's restore | -| `boot_cpus`, `capture_cpus`, `restore_cpus` | Online CPUs the check covered | -| `boot_elapsed_us`, `capture_elapsed_us`, `restore_elapsed_us` | Its wall time, as `elapsed_us` in the [status command](#conformance-checks-and-the-nvx-time-abi-marker) | -| `boot_cpu_us`, `capture_cpu_us`, `restore_cpu_us` | Its CPU time, as `cpu_us` in the [status command](#conformance-checks-and-the-nvx-time-abi-marker) | -| `capture_generation`, `restore_generation` | `g` when the check ran (the boot check's is 0) | -| `boot_failures`, `capture_failures`, `restore_failures` | [Report-only mode](#conformance-checks-and-the-nvx-time-abi-marker) only: the check's failure count | -| `tsc_hz`, `lapic_hz` | `F` and `L` | -| `discontinuities` | Wall-clock discontinuities since cold boot: restores plus discipline steps | -| `last_discontinuity` | `none`, `restore`, or `step` | -| `last_step_ns` | Signed size of the last step | -| `last_step_realtime_ns` | `CLOCK_REALTIME` right after the last step | -| `last_downtime_ns` | `D` of the last restore | -| `last_downtime_source` | `monotonic` or `utc` | -| `synchronized` | 1 when the last accepted sample is at most 128 s old | -| `offset_ns` | Last measured `theta`. Until the first discipline poll, the boot check's or restore repair's `theta` from before its step | -| `uncertainty_ns` | Last accepted `epsilon` | -| `frequency_ppb` | Current kernel frequency correction | -| `samples`, `rejected_samples` | Sample counters | -| `last_sample_error` | `none` or `G_SAMPLE_UNCERTAIN` | -| `violations` | 0; a violation powers the guest off | - -Workloads that need prompt notice of a step can also arm a `timerfd` with -`TFD_TIMER_CANCEL_ON_SET`, which the kernel cancels on every step. - -## Failure codes - -Codes are stable. OpenVMM errors put the code in brackets at the start of the -message, for example `[E_TSC_RATE_TOLERANCE] destination TSC rate ...`, and -the code survives every error-wrapping layer, including the worker boundary: -the process's fatal-error line leads with the first code in the chain, -`fatal error: [E_TSC_RATE_TOLERANCE] `, followed by the -full cause chain. A rejected cold boot or restore exits the OpenVMM process -with status 1, the existing fatal-error status; tests and orchestrators match -the bracketed code at the start of that line. A capture that a time ABI -check rejects is rollback-safe: the guest continues, and OpenVMM logs the -code. A capture that fails for another reason before its commit point, such -as a failed publication, also rolls back, and its log carries no time ABI -code. - -| VMM code | Condition | Detected at | -| --- | --- | --- | -| `E_SNAPSHOT_VERSION` | Manifest version is not 6; the snapshot must be recaptured | Restore | -| `E_MANIFEST_TIME` | Time contract missing or malformed, `time_abi_version` not 1, or tolerance not 250 | Restore | -| `E_BACKEND_MISMATCH` | Snapshot taken on another backend | Restore | -| `E_PROFILE_UNKNOWN` | Profile ID neither pinned in this OpenVMM nor a host profile, or a restore's explicit `--cpu-profile` names another profile than the snapshot's | Cold boot, restore | -| `E_PROFILE_DIGEST` | The recorded profile digest or embedded document is not the pinned profile's of the same ID, or a host profile's embedded document is not a canonical host profile of the recorded ID and digest | Restore | -| `E_PROFILE_HOST_UNKNOWN` | `--cpu-profile auto` maps a CPU of another vendor than Intel and AMD to no profile, where no host profile can serve it either, or maps the host to profiles of more than one generation; `--cpu-profile host` runs on a CPU of another vendor than Intel and AMD; or `--cpu-fingerprint`'s check, which never falls back to a host profile, maps the host to no pinned profile | Cold boot, `--cpu-fingerprint` | -| `E_PROFILE_TIME_BITS` | The profile is invalid or violates the CPU time bits | Cold boot, restore | -| `E_CPU_GENERATION` | Host CPU vendor, family, model, or stepping not in the profile | Cold boot, restore | -| `E_PROFILE_UNSUPPORTED` | Backend lacks a feature, limit, XSAVE layout, MSR value, or feature-bank bit of the profile, or the host is not qualified | Cold boot, restore | -| `E_CPU_SURFACE` | Recomputed effective CPUID differs from the recorded one, or VP 0's CPUID differs from the effective CPUID under its masks | Cold boot, restore | -| `E_CPU_UNLISTED` | On MSHV or WHP, the hypervisor presents a non-zero CPUID entry outside the profile's tables, a reserved entry that would expose a host feature | Cold boot, restore, `--cpu-fingerprint` | -| `E_IDENTITY_ROUTING` | Backend cannot deliver the identity CPUID or MSRs | Cold boot, restore | -| `E_TSC_SYNC_UNSUPPORTED` | Backend lacks the synchronized TSC set | Cold boot, restore | -| `E_TSC_SCALING_ACTIVE` | The guest TSC would be scaled | Cold boot, restore | -| `E_TSC_RATE_UNAVAILABLE` | Backend cannot report its native TSC rate | Cold boot, capture, restore | -| `E_TSC_RATE_IMPLAUSIBLE` | A rate is outside 500 MHz to 10 GHz | Cold boot, capture, restore | -| `E_TSC_RATE_TOLERANCE` | `abs(F_d - F_s)` exceeds 250 ppm of `F_s` | Restore | -| `E_LAPIC_RATE_UNAVAILABLE` | Backend cannot report its LAPIC rate | Cold boot, capture, restore | -| `E_LAPIC_RATE_MISMATCH` | `L` is not the backend constant, or `L_d` differs from `L_s` | Cold boot, restore | -| `E_HOST_IDENTITY` | Host identity, boot identity, or clocks unavailable | Capture, restore | -| `E_DOWNTIME_NEGATIVE` | Downtime below zero | Restore | -| `E_DOWNTIME_EXCESSIVE` | Downtime above 30 days, or a recorded `Q` or `D + Q` above 30 days | Restore | -| `E_TSC_ANCHOR` | An anchor is unavailable, or none of at most 64 samples pairs within 100 µs | Capture, restore | -| `E_TSC_TARGET_OVERFLOW` | `TSC_target` exceeds 64 bits | Restore | -| `E_TSC_SYNC_READBACK` | A VP does not hold the synchronized value | Restore | -| `E_VP_LATE_CREATION` | A VP was instantiated after the synchronized set | Restore | -| `E_LAPIC_PERIODIC` | A periodic LAPIC timer is armed | Capture, restore | -| `E_LAPIC_TSC_DEADLINE` | TSC-deadline mode or state is present | Capture, restore | -| `E_PIT_ACTIVE` | PIT channel 0 is counting in a periodic mode | Capture, restore | -| `E_GENERATION_EXHAUSTED` | `capture_generation + 1` overflows `u32` | Restore | -| `E_CMDLINE_CLOCK_TOKEN` | `tsc_early_khz=` or `lapic_timer_hz=` in a supplied or saved command line | Cold boot, restore | -| `E_TEST_HOOK` | A malformed or unknown test hook | Cold boot, restore | - -Guest failures power off through `0x604` with a status distinct from the -statuses the guest already uses (0, 1, 37, 125, 126, 127, 128 to 192 for -signals, and 255): - -| Status | Class | Guest codes | -| ---: | --- | --- | -| 193 | Conformance | `G_CONFORMANCE_C1` to `G_CONFORMANCE_C12`, `G_KERNEL_WX` | -| 194 | Runtime violation | `G_TSC_UNSTABLE`, `G_CLOCKSOURCE_UNSTABLE`, `G_CLOCKSOURCE_SWITCH`, `G_CLOCKSOURCE_SKEW`, `G_TSC_WARP`, `G_TSC_ADJUST`, `G_RCU_STALL`, `G_RCU_STARVED`, `G_SOFT_LOCKUP`, `G_HARD_LOCKUP`, `G_HUNG_TASK`, `G_UNCHECKED_MSR`, `G_APIC_ID_MISMATCH`, `G_KMSG_OVERRUN` | -| 195 | Restore repair | `G_REPAIR_PACKET`, `G_REPAIR_GENERATION`, `G_REPAIR_SAMPLE`, `G_REPAIR_CLOCK`, `G_REPAIR_SUPPRESSION` | - -`G_SAMPLE_UNCERTAIN` is the only non-fatal guest code: the discipline records -it and continues (see [Uncertainty bounds](#uncertainty-bounds)). Every fatal -guest code prints its violation event before the power-off, on production -paths too. No other time ABI line reaches the console unless a test runs -`nvx-time status` or `nvx-time exhaustive`. A boot or restore check that -fails after shell-ready or after the acknowledgement still powers off with -193, while the workload may be running. - -A workload can exit with any 8-bit status, so a harness classifies a time -failure by the status together with its `NVX-TIME-ABI-VIOLATION` event. - -## Host qualification - -`nvx.py doctor --backend ` qualifies a host by running every check, -with `H4` and `H6` on their long schedules. The `validate-runner` action runs -`H1`, `H2`, and the short `H4` before every CI job, `H3` in microVM and -platform jobs, and `H6` on the CI schedule in platform jobs, whose benchmarks -boot guests without the scenarios' probes; every CI microVM boot and restore -scenario runs the CI warp schedule. `H5` and `H7` run in `doctor` only. Both -tools print one `NVX-DOCTOR: check= status= detail=...` line -per check and fail if any check fails. In CI, `validate-runner` reports -`nonstop_tsc` in `/proc/cpuinfo` on Linux runners as evidence, and the Linux -runner setup script warns without it; neither gates on it (see [Qualification -gates](#qualification-gates)). The OpenVMM vmm-tests run no warp probe: they -boot OpenVMM's own test guests, whose verdicts don't depend on the cross-vCPU -skew bound. - -| ID | Check | -| --- | --- | -| `H1` | The backend device or API is present and usable | -| `H2` | CPU fingerprint: vendor, family, model, stepping, microcode, host kernel or OS build, the generation name, and the pinned profile that `auto` selects; an unmapped generation fails (`E_PROFILE_HOST_UNKNOWN`), although `auto` falls back to a host profile on such an Intel or AMD CPU. `openvmm --hypervisor --cpu-fingerprint ` writes the fingerprint (`openvmm-cpu-fingerprint/v1`), checks it against the generation's profile, which never falls back to a host profile, and prints one `NVX-CPU-PROFILE:` line (`E_PROFILE_HOST_UNKNOWN`, `E_PROFILE_UNSUPPORTED`, `E_CPU_UNLISTED`), whose generation and profile `H2` reports, so that OpenVMM's catalog alone decides which hosts qualify; NVX's copy of the generation table maps the host only when qualification runs without OpenVMM. The host's invariant TSC (`constant_tsc` and `nonstop_tsc` on Linux, the CPUID bit on Windows) is reported as evidence only | -| `H3` | OpenVMM preflight in verification mode: profile support, identity routing, synchronized TSC set, no scaling, and both rates, without booting a guest. The verified profile must be a pinned one: a host profile fails, including the one that `auto` falls back to on a CPU that no pinned profile serves | -| `H4` | TSC rate stability against the host's clocks: interval agreement against an undisciplined clock (`CLOCK_MONOTONIC_RAW` or `QueryPerformanceCounter`) and the whole-window rate against `F_d` on the disciplined one (see [Rate stability](#rate-stability-h4)). On a Linux host the detail also reports the host clocksource, as evidence only | -| `H5` | Host cross-CPU TSC skew: a pinned-thread probe over all host CPU pairs, `max_abs_offset_ns <= 1000` | -| `H6` | Guest warp probe on the qualification schedule (see [Warp schedules](#warp-schedules-h6-and-ci)) | -| `H7` | Host UTC is synchronized: no `STA_UNSYNC` on Linux; a synchronized `w32tm` source on Windows | - -### Rate stability (`H4`) - -A host probe compares the TSC with two host clocks: - -- **Interval agreement and conclusiveness** use a clock that time - synchronization never steers: `CLOCK_MONOTONIC_RAW` on Linux and - `QueryPerformanceCounter` on Windows. -- **The whole-window rate `r`** and its deviation from `F_d` use the - disciplined `CLOCK_MONOTONIC` on Linux, which is never stepped and whose - long-run rate is the true one, and `QueryPerformanceCounter` on Windows. - -A sample reads the TSC between two reads of a clock and keeps the tightest of -up to 64 such brackets. Its uncertainty is half the bracket plus half the -clock's resolution. The probe reads each clock 100,000 times back to back. -If two consecutive reads ever return the same value, the clock is coarser -than one read, and its resolution is the larger of the reported resolution -and the smallest nonzero step between consecutive reads: 100 ns for Hyper-V's -reference TSC page clock (`hyperv_clocksource_tsc_page`), the clocksource of -MSHV roots and the Azure MSHV runners, which `clock_getres` reports as 1 ns, -and one 10 MHz tick (100 ns) for `QueryPerformanceCounter`. A clock that -advances on every read keeps its reported resolution, because its smallest -step is then only the time one read takes (for example -`CLOCK_MONOTONIC_RAW` on a host whose clocksource is `tsc`). - -The probe sleeps between samples, so the host's CPUs can idle: 3 samples 1 s -apart in `validate-runner`, and 13 samples 10 s apart (120 s) in `doctor`. -For each interval between consecutive samples, it computes against the -undisciplined clock the rate `r_i`, in TSC cycles per second, and its -relative uncertainty `u_i`, the two samples' uncertainties divided by the -interval. `r` is the rate over the whole window against the disciplined -clock. The check passes if: - -- every `u_i` is at most 0.25 ppm; otherwise the measurement is - inconclusive and fails; -- the interval rates agree within 1 ppm (largest minus smallest, relative to - their whole-window rate against the same clock); and -- where `F_d` is known (in `doctor`, and in the `validate-runner` jobs that - run `H3`), `|r - F_d|` is at most 100 ppm of `F_d`. - -The probe's output names both clocks, each with its role (rate or -stability), reported resolution, and observed step. The detail reports `r`, -its deviation from `F_d` in ppm, the agreement, the largest `u_i`, both -clocks and the undisciplined clock's measured resolution, and, on Linux, the -host clocksource, which never gates. - -Interval agreement uses the undisciplined clock because chrony's frequency -updates move `CLOCK_MONOTONIC`'s rate between seconds, on Azure through the -Hyper-V PTP clock: - -- Against `CLOCK_MONOTONIC`, the short schedule failed 3 of 9 runs on the - Azure MSHV runners: their interval rates differed by up to 12.2 ppm. Over - 30 1-s intervals they spread by up to 2.7 ppm, against at most 0.07 ppm - versus `CLOCK_MONOTONIC_RAW`, while chrony's frequency moved by up to - 2.7 ppm. -- Against `CLOCK_MONOTONIC_RAW` and `QueryPerformanceCounter`, 35 - short-schedule runs on the Azure KVM, MSHV, and WHP runners agree within - 0.000 to 0.050 ppm, with the largest `u_i` between 0.04 and 0.106 ppm. The - long schedule agrees within 0.002 ppm on the bare-metal MSHV host, where - `CLOCK_MONOTONIC` gave 0.402 ppm. -- The WHP hosts show 0.000 ppm with a residual of at most 0.07 µs over 118 s. - -A TSC that stops, slows, or is rescaled across idle misses the bound by -orders of magnitude where the host's clocksource does not follow the TSC's -steps: the Hyper-V reference page and `QueryPerformanceCounter` in a VM, -which the hypervisor computes from the TSC but keeps continuous when it -corrects TSC offsets, or the HPET, which has its own oscillator. -Where the host clocksource is `tsc` itself (the Azure KVM runners and the -bare-metal KVM host), every kernel clock derives from the TSC, so no host -clock can catch a TSC step; the guest warp probe (`H6` and the CI schedule) -is the detector there. - -The 100 ppm bound keeps the guest's wall-clock discipline, which steers at -most 500 ppm, clear of the TSC's error plus a restore's rate deviation of up -to 250 ppm. On Windows the clock derives from the same TSC, so the deviation -compares `F_d` with the host's own calibration. - -### Warp schedules (`H6` and CI) - -Each run of the guest warp probe executes -`/sbin/nvx-time-probe warp --bound-ns 1000` over every online vCPU and must -report `max_backward_ns` and `max_abs_offset_ns` at most 1,000, no stall, and -a conclusive verdict. Between runs the guest only sleeps, so every vCPU halts -and the host applies any idle-triggered TSC correction it makes, the source -of the warps in #265: - -- `H6`: a microVM with the host's largest supported vCPU count up to 8 boots, - passes the boot check, and runs the probe five times, with idle gaps of - 0.1, 1, 5, and 1 s; a 1-vCPU microVM then boots and runs it once. -- CI: every microVM boot and restore scenario runs the probe twice, with a - 1 s idle gap, after boot and after every restore. The platform jobs run - `H6` on this schedule before their benchmarks: the larger microVM runs the - probe twice, 1 s apart, and the 1-vCPU microVM runs it once. - -### Qualification gates - -The doctor qualifies alike on every backend, on measured properties only: the -CPU profile (`H2`, `H3`), rate stability (`H4`), and the idle-scheduled warp -probe (`H6`, and the CI schedule in every microVM and platform job). It -records the host OS's invariant-TSC bit and the host clocksource as evidence -and never gates on them, on any backend: - -- On Azure, WHP and nested MSHV cannot offer the invariant-TSC bit to their - guests through their feature banks, although the host OS sees an invariant - TSC on every runner except one 8370C MSHV runner. The profile exposes the - bit through CPUID anyway, so `H4` and `H6` measure what it promises. -- MSHV roots and Azure KVM hosts run `hyperv_clocksource_tsc_page`, not - `tsc`. The guest has no kvmclock, so no guest clock derives from the host - clocksource, and the downtime comes from host monotonic time and UTC, whose - accuracy does not depend on it. -- KVM rewrites a vCPU's TSC offset whenever it loads the vCPU on a host that - marked its TSC unstable, unless the host clocksource is - `hyperv_clocksource_tsc_page`. `H6` measures the warps that this would - cause. - -That 8370C MSHV runner, whose host OS lacks the bit and which showed the -#265 warps before the time ABI, qualifies. With `H4` and `H6` on their long -schedules, `H1` to `H7` pass, and `H6` measures at most 26 ns; its CPU profile -check passes with the same surface digest before and after the MSHV probe -partition fix (#411). Its guests' warp probes, over the full microVM suite on -the production and debug kernels and ten more rounds of `smp`, -`smp-snapshot`, and `restore-processors` (288 runs), measured at most 36 ns of -offset and 4 ns of backward step. Its host-level warp probe saw backward steps -of at most 2.1 ns. - -### Generations and runner placement - -Generation names used in logs, reports, and job summaries are `skylake-sp` -(family 6, model 85), `icelake-sp` (6/106), `emeraldrapids` (6/207), -`alderlake` (6/151 and 6/154), `milan` (25/1), `genoa` (25/17), and `turin` -(26/2); no CI microVM runner has the last four. -`validate-runner` and `nvx.py doctor` detect the generation at run time and -report it, together with the selected profile, `F_d`, `L`, the `H4` rate -metrics, and the skew metrics, in their log and the job summary; an unknown -generation fails qualification explicitly. Runner labels are not used and -runners are not re-registered: per-PR CI captures and restores on the same -runner, and same-generation cross-VM restore is validated by the fleet -restore matrix (`p6-restore-matrix`) on the hosts our account can use. -Routing CI jobs by generation labels is optional future work. - -## Performance expectations and acceptance gate - -**Gate.** Per backend and vCPU count, no p50 is worse than the base-branch -median by more than `max(5%, 2 ms)`. The gate covers 37 cells: all 34 -one-vCPU metrics and `shell_snapshot_restore_512_mib` at 2, 4, and 8 vCPUs. -It is measured on the CI Azure platforms; bare-metal results are -informational. The bare-metal KVM host idles at 800 MHz (`intel_pstate` -powersave without HWP), which inflates its absolute restore latency -(`restored_ms` p50 at one vCPU 39 ms, against 23 ms at full clock) and its -`cpu_us`; the arms of an A/B share the effect. - -Expected effects: - -| Change | Expected effect | -| --- | --- | -| WHP restored SMP without RDTSC emulation | Removes the 1-to-2 vCPU restore jump (120.7 ms to 165.9 ms p50): SMP restores measured 8 to 18 ms faster in the spike and 14 to 31 ms faster on 8370C runners at 2 to 8 vCPUs. A guest timestamp read after an SMP restore drops from 41 to 73 µs to 10 to 16 ns | -| Packet v4 with four-byte reads instead of 83 or more byte reads | Fewer restore exits than reading packets v1 to v3 a byte at a time. The gate's base reads no packet in its untiered restores, so there the time ABI adds 25 port exits per restore (59 against 34 on KVM): the selector write and 24 four-byte reads | -| Wall clock from the packet instead of RTC polling | Removes at least 32 CMOS port exits and the update wait per tiered restore; the gate's base skips the RTC reads in its untiered restores | -| The kernel's RTC write | Because the discipline clears `STA_UNSYNC`, the guest kernel writes the CMOS RTC within a second of each clock step at boot and restore and then every 11 minutes: 26 port exits per write, about 0.1 ms on KVM and more where port exits cost more. Keeping `STA_UNSYNC` set would avoid it but report `TIME_ERROR` to applications | -| No capture-time clocksource waits | Removes the harness's wait for `tsc-early` to become `tsc`: 0.73 to 0.92 s per capture on MSHV and 0.47 to 0.67 s on WHP; outside the gated metrics | -| `no_timer_check` from the Hyper-V identity, no LAPIC calibration, and no `tsc-early` window at cold boot | About 43 to 52 ms (9 to 15%) faster `cold_start_base` and other quiet cold boots on MSHV and WHP (WHP measured 31 to 55 ms on bare metal, and 61 to 71 ms at one vCPU on Azure); KVM unchanged | -| Invariant TSC from the profile where the backend cannot offer it to guests (Azure MSHV and WHP) | Removes each AP's delay calibration of about 150 ms: an 8-vCPU cold boot takes 299 ms instead of 1,385 ms on Azure MSHV, and cold boots are 0.2 to 1.1 s faster at 2 to 8 vCPUs on Azure WHP | -| Fixed restore work: the restore clock (0.7 to 1.1 ms), restore verification, and the backend preflight | A one-vCPU WHP restore is at parity on Azure 8370C runners (p50 −0.6 ms) and about 2 ms slower on bare metal, where no emulation cost is recovered; the counting LAPIC accounts for at most 0.75 ms of it | -| MSHV root-driver costs | Host characteristics, which the time ABI and the legacy path pay alike, not ABI obligations. On the bare-metal MSHV host (hypervisor 26100.30000) three root-driver costs make up 50 of the 70 ms of a one-vCPU, 512 MiB restore and 130 of 150 ms at eight vCPUs: registering guest RAM, which the driver maps no-access 4 KiB at a time (28 ms; 9.5 ms on Azure); creating each application processor after the first: the second takes about 6 ms and each later one 14 to 15 ms, because `MSHV_CREATE_VP` pre-deposits 90 pages, a VP needs about 172 there, and each failed creation deposits one more page (21 and 78 ms at 4 and 8 vCPUs, and the same at cold boot; 1.9 ms at 8 vCPUs on Azure); and the first touch of each 2 MiB chunk after resume (about 22 ms, see below). The first two serialize on the driver's partition mutex. Prefaulting the snapshot's resident chunks before registration (`MADV_POPULATE_WRITE`) cuts the third to 4 to 10 ms but costs about 45 ms itself, so the remedies are driver changes (larger deposits, and read-only mappings for read faults or a smaller fault granule for private file mappings) or guest RAM filled from the snapshot in parallel. The frozen synchronized TSC set adds 40 to 170 µs for 1 to 4 VPs and no per-VP serialized work | -| Boot check and daemon start | Off the cold-boot path. Before shell-ready only the initial time sample and clock step remain (`C12`: one port write and four reads). The other checks and the daemon start run after shell-ready, and `nvx-time status` reports their wall time (`elapsed_us`, not budgeted) and CPU time (`cpu_us`), which each backend budgets per phase (see `cpu_us` budgets below). Wiring v7 measured medians of 1.6 to 3.7 ms at boot and 0.4 to 1.0 ms at restore on MSHV, 3.5 to 5.1 ms and 2.8 to 5.3 ms on KVM (the bare-metal KVM host idles at 800 MHz, which slows the delayed checks), and 2.6 to 4.8 ms and 7.7 to 11.5 ms on WHP at 1 to 8 vCPUs, with captures under 3 ms everywhere | -| Counting LAPIC instead of TSC-deadline on KVM | Different timer-programming exits; covered by the gate | -| v1 profiles without `ITS_NO` | Linux's ITS mitigation at boot, about 6 ms of a one-vCPU cold boot where the host's KVM advertises `ITS_NO` (the bare-metal KVM host: +6.3 [5.5, 7.3] ms against the same build booted with `indirect_target_selection=off`). The gated Azure KVM, MSHV, and WHP guests were already mitigated and are unchanged; restore is unaffected | -| The profile's CPU view at 2 or more vCPUs on KVM | None at the integration head. Before the L3 fix, multi-vCPU cold boots were slower: +9.2 [5.0, 13.7] ms at 2 vCPUs on the bare-metal KVM host against the pre-profile head, and about +25 ms on nested Azure KVM. OpenVMM's `CPUID.4` reported a private L3 cache per vCPU, which made Linux's cache-info initialization wait a 10 ms tick for CPU 1 on many boots. With the L3 shared by the socket, the bare-metal host's cold boots match the pre-profile head: −0.3 [−3.2, +2.7], −0.0 [−2.1, +2.0], and −1.0 [−3.6, +1.7] ms at 2, 4, and 8 vCPUs. The gate's cold boots are one-vCPU; a boot-only comparison against the `dev` base on nested Azure KVM found no significant difference: +1.7 [−2.5, +6.9] ms at 2 vCPUs, +3.7 [−5.4, +13.6] at 4, and +3.3 [−7.0, +11.5] at 8, which excludes the earlier +25 ms. At 8 vCPUs, the time ABI's VMM boots 9.5 [4.6, 16.4] ms faster and its guest 8.2 [0.4, 15.9] ms slower, which cancel; the guest's part may be the KVM paravirtual features that the identity forgoes (see [Hypervisor identity](#hypervisor-identity)). The msr driver's per-CPU hotplug callback (`msr_init`) waits one or two ticks on multi-vCPU boots, as it did before the profiles; it competes for CPU with the asynchronous initramfs unpack and is off the critical path | -| Restore work before the acknowledgement | Only the packet read, the clock set, CPU and memory activation, and entropy and identity repair, in one helper process. The restore checks, the RCU release, and the deferred `C7` start at `SCHED_IDLE` 150 ms after the acknowledgement, after the readiness path | -| Monotonic time that includes the downtime | Right after resume, the guest kernel runs the timer work that came due during the downtime (kworkers, softirqs, and RCU) and first-touches cold pages doing so: at one vCPU about 1.4 ms on MSHV, 2.2 ms on KVM, and 17 ms on WHP (26 ms at two vCPUs). On WHP each restored 4 KiB page that the guest first touches costs about 50 µs of wall time on bare metal and about 70 to 95 µs on the Azure runners: a nested page fault, about 11 µs of it in the hypervisor on bare metal and the rest resolved by the root, which the VMM never sees. WHP's lazily registered 2 MiB chunks add about 90 µs each, all before readiness, and on bare metal a WHP restore's latency after the VPs are released is roughly the pages it touches times 53 µs. It falls inside the gated restore metric whatever the helper's priority. On MSHV the cost is per 2 MiB chunk: the root driver resolves the first touch of each chunk with 512 copy-on-write copies of the private snapshot mapping, about 1.1 ms. A one-vCPU restore touches about 24 chunks, 25 ms of its restore on both the time ABI and the legacy path, and the time ABI's catch-up adds about one chunk | -| No time ABI console lines in production | Each console byte costs one or two port exits, about 10 to 12 µs on bare-metal KVM and an estimated 20 to 45 µs nested on Azure. Without the boot marker (about 110 bytes) and the restore marker (about 117 bytes), boots and restores save 1 to 5 ms, enough to fail the gate on nested Azure KVM; violation events still print | - -**Right after a restore.** Not all post-resume work finishes before readiness. -At resume the guest runs every timer that came due during the downtime: the -kernel's, those that came due during the VMM's restore setup (the legacy path -stopped guest time during that setup, so its guest ran them later; it left out -about 400 ms of a nested Azure KVM restore and 7 to 12 ms on bare-metal KVM, -but only 1 to 2 ms on WHP), and the discipline's poll when the downtime -outlasts the poll period. Restore repair also steps the clock, which the -kernel follows with its RTC write, and the restore checks start 150 ms after -the acknowledgement. Readiness doesn't wait for this work. - -The first process that a restored guest starts pays a separate cost. A plain -(untiered) restore with nothing to activate starts no process before -readiness, while the legacy path's status read forked and executed one there, -which first-touched the guest kernel's fork, exec, and exit paths. Under the -time ABI the first process after readiness touches them instead, about 80 to -110 more pages than on the legacy path, which matters where first touches of -restored RAM are expensive: on the Azure WHP runners, the first process after -readiness takes about 5 to 9 ms longer than on the legacy path, while -readiness itself is unchanged. None of the time ABI's post-resume work, the -kernel's RTC write included, runs while it does, and one small process started -ahead of it removes 64 to 92% of the difference. On bare metal the cost is a -shift: readiness comes 9.5 ms earlier, and readiness plus the first process -takes 4.7 ms less. On the Azure runners, from process launch to exit, the -difference is about +2 to +4 ms or less and isn't significant. - -The gate's teardown metric (`openvmm_snapshot_restore_guest_exit_teardown`) -times such a first process: an exit requested right at readiness. Without a -faster `nvx-exit`, the time ABI would be about 5 to 9 ms (31 to 37%) slower -there than the legacy path on the Azure WHP runners, which would fail the -gate, and 24% slower on bare metal. The metric passes because `nvx-exit` is a -single static executable, which makes every guest exit faster, with or without -the time ABI: with it, the metric measured 20% below the legacy path on bare -metal and 30% below on an Azure 8573C runner. That offsets the cost in the -gate rather than removing it: the first command after a plain restore still -pays it on nested Azure WHP. The remedy, a working-set prefetch that populates -the pages a restore touches before its vCPUs run, is a follow-up after v1. - -On nested Azure KVM, a network restore that follows the shell-snapshot -benchmarks' captures and restores is up to about 12 ms (8.5%) slower, by an -amount that varies from run to run, and is no slower on its own. Of the -difference, 5.7 ms comes before the first good network probe and 2.9 ms in the -teardown after it; the static `nvx-exit` doesn't measurably change it. Neither -arm takes major faults. The time ABI's restore makes about 4% more KVM exits, -26 of them port I/O, mostly the restore packet's read, and more retried EPT -faults and remote TLB flushes, each of which costs more on a nested runner, -depending on the host's state. - -Expected wins are tracked separately and do not relax the gate. - -**`cpu_us` budgets.** Each backend budgets the CPU time of each phase's -checks (boot, capture, and restore) at a + b·(n − 1) ms for n online vCPUs, -per sample. A budget has at least 20% headroom over the largest sample that -fleet validation recorded for its backend, phase, and vCPU count: it is at -least 1.2 times that maximum, taken over bare metal and Azure and over the -production and debug kernels (wiring v7m), without triaged outliers or the -exempt sample below: - -| Backend | Boot | Capture | Restore | Measured on | -| --- | --- | --- | --- | --- | -| KVM | 6 + 2 | 1.5 + 0.4 | 7 + 2 | Bare metal and nested Azure 8370C, 128 MiB | -| MSHV | 3 + 1 | 1 + 0.4 | 2.5 + 0.5 | Bare metal and nested Azure 8573C, 128 and 512 MiB | -| WHP | 25 + 1.5 | 1.2 + 0.4 | 35 + 6 | Bare metal and the Azure 8370C and 8573C runners, 128 and 512 MiB | - -Fleet validation records every sample's `cpu_us` and checks each one -against these budgets after the run. CI reports each phase against its budget -(`_cpu_over_budget`) but does not gate on it, because the A/B gate -above covers latency. The checks' CPU time also counts the host stalls of -their first touches of restored RAM: - -- WHP's restore checks first-touch restored copy-on-write RAM page by page, - about 50 µs each on bare metal and 70 to 95 µs on Azure, which the workload - would otherwise do. Their cost grows with guest memory: medians are 1.3 to - 2.4 times higher at 512 MiB than at 128 MiB at 4 and 8 vCPUs on Azure. So - WHP's restore budget comes from 512 MiB guests, which bound smaller ones; - its Azure maximum is 42.20 ms with 4 vCPUs. Its boot and capture budgets - come from two 128 MiB samples on the nested runners: a 21.84 ms boot with - 2 vCPUs on an 8573C runner (see below) and a 0.99 ms capture with 1 vCPU on - an 8370C one. -- On MSHV, each first touch of a 2 MiB chunk of restored RAM stalls the - toucher about 1.1 ms on bare metal (see MSHV root-driver costs). The - restore budget leaves room for one more stall above every maximum. -- On nested Azure KVM, KVM's restore checks fault in the file-backed restored - RAM 4 KiB at a time: medians of 3.4 to 7.6 ms at 1 to 8 vCPUs over 72 - restores, and fleet validation maxima of 5.53 ms at 1 vCPU and 16.16 ms at - 8. - -KVM's budgets come from 128 MiB guests, the size that CI and the fleet -matrices run. MSHV's hold at 128 and 512 MiB alike: MSHV registers all -guest RAM before the restore, and the checks pay only for the chunks they -touch, which doesn't depend on the guest's memory size. - -**After a rolled-back capture.** A capture flushes file-backed guest RAM -before it stages: this is OpenVMM's pre-staging RAM flush, the same with or -without the time ABI. So a capture that rolls back leaves that RAM clean, and -the guest's first write to each page traps once to re-dirty it. The next -capture's checks pay that for the pages they write: on the bare-metal WHP -host, 0.91 to 1.23 ms at one vCPU and 1.33 to 1.88 ms at eight, against 0.22 -to 0.25 and 0.63 to 0.72 ms before the rollback. On bare-metal MSHV the cost -doesn't show: medians of 0.21 and 0.57 ms at one and eight vCPUs, against 0.20 -and 0.57 ms before. On bare-metal KVM it's small: 0.42 to 0.63 ms at one vCPU -and 1.52 to 2.58 ms at eight, against 0.38 to 0.46 and 2.36 to 2.51 ms after a -rejected request. Later captures are back to normal, and a request that -preflight rejects flushes nothing and costs nothing extra. So the capture -budget doesn't apply to exactly one sample, the first capture after a -rolled-back capture in the same VM, on any backend: validation reports its -`cpu_us` without checking it, and checks the next capture normally. - -**Over-budget samples.** The checks' CPU time includes host preemption (see -[Conformance checks](#conformance-checks-and-the-nvx-time-abi-marker)), so a -host can stretch a single sample past its budget. Validation accepts an -over-budget sample as a triaged outlier only if all three of these hold: - -1. **Isolated:** a rerun of the same case on the same host, with at least 30 - samples of the phase, has none over the budget, and no other sample of - that case on that host is over it. -2. **Stretched:** the sample's `elapsed_us` exceeds the nearest-rank 99th - percentile of the rerun's `elapsed_us` for the phase; host preemption - stretches the wall time along with the CPU time. -3. **Clean:** the same VM shows no violation event, no `NVX-SNAPSHOT-ERROR`, - no exit status 193 to 195, and `violations=0`. - -Any other over-budget sample is a budget failure, fixed in the budget or in -the code. Fleet validation's after-the-run check found: - -- KVM's five 2-vCPU captures on bare metal over the earlier 1 + 0.4 ms were - budget failures, fixed by the capture budget above. -- MSHV's debug-kernel 4-vCPU boot at 47.1 ms, 56.8 ms elapsed, is a triaged - outlier: a rerun of its case had none of 30 samples over, and at most 3.4 ms - elapsed. -- WHP's 1-vCPU capture at 3.91 ms on a nested 8370C runner is a triaged - outlier: a rerun of its case had none of 30 samples over, and at most - 0.99 ms elapsed. -- WHP's 2-vCPU boot at 21.84 ms on a nested 8573C runner, 51.6 ms elapsed, was - a budget failure: a rerun of its case had none of 30 samples over, but up to - 59.0 ms elapsed, because the boot checks there wait across vCPUs for a long - and variable time. The boot budget above covers it. -- KVM's 4-vCPU restore at 14.05 ms on nested Azure KVM (8370C), after - 10 minutes of downtime and 19.0 ms elapsed, is a triaged outlier: a rerun of - its case had none of 30 samples over, and at most 14.4 ms elapsed. - -**Start-up budget.** The time ABI and CPU profile work of a cold boot or -restore costs less than 0.5 ms over the pre-profile head (`e7ec0ca6c`), -measured on the bare-metal KVM host with an attribution harness; the -flip requires it. At the integration head (`e06ed4d4a`), up to the backend -preflight at one vCPU, a restore starts 0.32 ms and a cold boot 0.30 ms -faster than `e7ec0ca6c`. The only added work is the worker's time ABI CPU -checks, about 56 µs over the pre-profile path, and every 90% upper bound is -within +0.31 ms. Process start moves each binary's milestones by about -0.25 ms from run to run, so only differences within one run count. -Neither path encodes, decodes, or hashes a profile document -or an effective CPUID: - -- the pinned profiles are compile-time constants with precomputed canonical - encodings and digests (`cpu_profile::pinned_record`), so selecting one - copies constants; -- capture copies the pinned record and writes the effective CPUID as a packed - binary record; -- restore compares the recorded digest and document with the pinned record, - and the recomputed effective CPUID with the binary record, about 5 µs; -- `verify_support` costs 2 to 10 µs of CPU time, plus the backend's surface. - -On bare-metal KVM, selection, the pinned-record and generation checks, the -effective CPUID merge, `verify_support`, and the unlisted-entry check cost -about 26 µs together (cold medians over 30 fresh processes). The codecs and -digests (`CpuProfile::{encode, decode, digest, digest_string, -to_pretty_json, from_pretty_json}` and `EffectiveCpuid::{encode, decode, -decode_verified, digest}`), `cpu_profile::pinned_profiles`, and the -fingerprint and derive paths are offline-only: tools, tests, and -`--cpu-fingerprint` use them, and no start path may. - -The backends add their own CPUID work: - -- MSHV and WHP enumerate the host CPUID once per process, for the supported - surface and the unlisted-entry candidates, without the hypervisor range, - which neither uses (`cpu_profile::cpuid::enumerate_basic_and_extended`). - Every CPUID instruction exits in a Hyper-V root, so a full enumeration - costs 70 to 100 µs on the bare-metal roots and 0.23 to 0.31 ms on Azure's - nested roots. Skipping the range took MSHV's from 96 to 49 µs on - bare metal and from 305 to 220 µs nested, more than its share of the - queries suggests. KVM takes its surface from `KVM_GET_SUPPORTED_CPUID` and - does not enumerate. MSHV enumerates on a thread that partition creation - starts, so the read leaves the start path; starting the thread costs about - 50 µs in a root partition, and keeping its channel until the partition - drops avoids freeing memory there, which costs 12 to 18 µs. -- MSHV reads VP 0's report in one rep `HvCallGetVpCpuidValues`: 73 µs p50 on - bare metal and 51 µs nested, against 0.51 and 0.82 ms with - one call per entry. WHP has no batched read: its report takes 57 to 60 - native reads, one call each, which with its one host enumeration cost - 0.81 ms on bare metal and 1.48 to 1.86 ms on Azure (medians). Reading - them on a few threads before any VP runs is the allowed lever if a gate - needs it. - -**Attribution.** With `OPENVMM_STARTUP_PROFILE` set, OpenVMM's lifecycle -profile records three exclusive time ABI restore phases: -`restore.time_abi_clock` (restore steps 12 to 16, whose phases OpenVMM also -logs as `time ABI restore clock`: `tsc_set_us`, then `lapic_us` and -`vm_time_us`, which overlap and are each measured from the end of the TSC -set, and `total_us`), `restore.guest_resume` -(from the VP release to the guest's first selection of the restore packet), -and, for a restore with `ACK_REQUIRED`, `restore.guest_repair` (from that -selection to the arrival of the `0x605` acknowledgement). The -`restore.guest_repair_gate` milestone still spans from the VP release to the -release of the acknowledgement boundary. The guest's restore checks run -after the acknowledgement, outside these phases. `nvx-time status` reports -`elapsed_us` and `cpu_us`, which cover the guest's readiness path and those -checks. On -every cold boot and restore, the worker logs `time ABI CPU checks passed` -with `recorded_cpuid_us`, `presented_cpuid_us`, and `profile_support_us`. -Their sum per restore at one vCPU is about 0.15 ms on bare-metal MSHV and -0.12 ms nested, and 0.87 ms on bare-metal WHP and 1.6 to 1.9 ms nested, about -95% of it WHP's per-leaf read-back. OpenVMM's comparison and unlisted-entry -check take 40 to 66 µs of each. - -## Test matrix - -**Unit tests (OpenVMM).** Rate boundary (250 ppm accepted, one hertz beyond -rejected) in both directions; downtime source selection and bounds, including -equal identities with a different clock kind; `TSC_target` arithmetic and -overflow; LAPIC tick arithmetic for every divide value, expiry, masking, and -periodic rejection; identity CPUID and MSR tables, including every #GP case -and `0x40000118` save and restore; capabilities derivation (`hv1` and -`kvm_clock` false); manifest version 6 validation and rejection of versions -2 through 5; packet v4 and time-sample golden vectors and their guest-side -parser; four-byte portb reads; and the generation counter. - -**Conformance.** The boot and restore checks pass at 1, 2, 4, and 8 vCPUs on -all 18 registered hosts, plus the exhaustive CI check and the warp probe on -every backend. Production paths print no marker, so the matrix driver runs -`/sbin/nvx-time status` over the console after shell-ready and after every -restore. CI runs it after every cold boot whose shell is on a console the -harness reads, and after every restore in the `smp-snapshot`, -`restore-processors`, `restore-downtime`, `snapshot-tiers`, and -`cpu-profile-fallback` scenarios, -except `snapshot-tiers`' gate-timeout check, whose guest OpenVMM stops at the -restore gate on purpose. CI's other restores and its guests without a shell, -the managed lifecycle and one-shot workloads, are only scanned for violation -events; a failed check still powers the guest off with status 193. Each query -requires exit status 0 and, after a restore, -a `phase=restore` line with `status=ok`, the restored `generation`, and -`cpus` equal to the online CPUs; after a cold boot, a `phase=boot` line with -`generation=0`. During fleet validation (`p6`), the matrix driver records -each phase's `cpu_us`, and every recorded sample must be within the -backend's budget for that phase in -[Performance expectations](#performance-expectations-and-acceptance-gate); -CI reports it without gating on it. The fleet runs them on -the hosts our SSH account can use, -which for KVM and for MSHV are a bare-metal Skylake-SP host and a nested -Azure VM (8370C for KVM, 8573C for MSHV): the account cannot open `/dev/kvm` -or `/dev/mshv` on -the other KVM and MSHV runners, which only CI jobs exercise. On the Azure WHP -runners, a cold boot with an empty command line, which logs to the console, -keeps `tsc` as its clocksource (#292: 6 of 6 runs with the time ABI, against -0 of 6 before it, which fell back from `tsc-early` to `refined-jiffies`). - -**Restore matrix.** Every case runs with zero violations; rejected cases must -fail with the listed code. A restored case's evidence is the -`nvx-time status` output after the restore, which waits for the deferred -checks. Per-PR CI runs the same-host cases on one runner; -the full matrix, including the cross-VM cases, runs on the hosts our account -can use as the fleet restore matrix. - -| Case | Hosts | Expected | -| --- | --- | --- | -| Same host, immediate | All | Restored; `DOWNTIME_UTC` clear | -| Same host, downtime of 30 s (over the 21 s RCU stall timeout), at 1 and 8 vCPUs, with and without `rcupdate.rcu_expedited=1` | All | Restored; no RCU stall; `rcu_stall_count` stays 0 | -| Same host under DVFS load | The bare-metal KVM host | Restored | -| Simulated host reboot: hooks `force-utc-downtime`, `boot-id-mismatch`, and `dest-rate-offset-ppm=+200`, then `-200` | One host per backend | Restored; `DOWNTIME_UTC` and `TEST_HOOKS` set; rate deviation reported | -| Simulated rate beyond tolerance: `dest-rate-offset-ppm=+251` | One host per backend | `E_TSC_RATE_TOLERANCE` | -| Downtime bounds: `downtime-add-s=2592001`; `force-utc-downtime` with `utc-offset-ms=-`, `n` above the elapsed time | One host per backend | `E_DOWNTIME_EXCESSIVE`; `E_DOWNTIME_NEGATIVE` | -| Sample uncertainty: `sample-delay-us=200`; then `sample-delay-us=3000` on restore and on cold boot | One host per backend | Restored and running, with `last_sample_error=G_SAMPLE_UNCERTAIN` in `nvx-time status`; `G_REPAIR_SAMPLE` (195); `G_CONFORMANCE_C12` (193) | -| Wall-clock convergence, for 12 minutes after a restore and after a cold boot, with the host's UTC rate sampled every 10 s alongside (Linux: `CLOCK_REALTIME` against `CLOCK_MONOTONIC_RAW`; Windows: UTC against QPC) | One host per backend | No step after the initial one; `synchronized=1`; `frequency_ppb` moving toward the window's mean rate; and, at every accepted discipline poll, `abs(theta)` at most 512 s × `abs(r)` + 1 ms, where `r` is the host's mean UTC rate over the 512 s before that poll (before then, over the window so far, at least 60 s), which leaves room for four skipped polls in a row. The checks start at the first discipline poll: the boot check's and restore repair's own samples are taken before their steps (at cold boot, the RTC's whole-second error). Skipped polls are reported with their count and longest run, and fail the case only through that bound. The spread of the host's per-minute rate is reported too: where the host's own UTC is being slewed, a minute's rate can swing through zero, so no single minute sets the bound | -| Across VMs of one generation | Between two 8370C Azure WHP runners, and between two 8573C ones. KVM and MSHV have no usable pair of one generation, so the simulated host reboot covers their cross-host path | Restored | -| Across generations | An 8370C Azure WHP runner to an 8573C one; bare-metal Skylake-SP KVM to nested Azure 8370C KVM; bare-metal Skylake-SP MSHV to nested Azure 8573C MSHV | `E_CPU_GENERATION` | -| Backend that cannot offer invariant TSC to its guests, on a host OS that sees it | The Azure WHP runners and nested Azure MSHV | Restored; the guest has `constant_tsc` and `nonstop_tsc`; `H4` and `H6` pass | -| Across backends | Bare-metal KVM to bare-metal MSHV | `E_BACKEND_MISMATCH` | -| Pre-v1 snapshot | Any | `E_SNAPSHOT_VERSION` | -| Failed capture: in one VM, a request whose destination's parent the host made unwritable to the OpenVMM process after launch, so that creating the staging directory fails after quiesce and OpenVMM rolls back (root's `CAP_DAC_OVERRIDE` and an administrator's backup and restore privileges bypass file permissions, so drop them first and probe with the same credentials, or use an immutable or read-only mount); then, with the parent writable again and after 30 s idle, a second request. Companion: a request whose destination already exists, which preflight rejects before quiesce. Scratch variant: the failed request in a VM with a paired scratch device mounted at `/run/nvx/scratch` and a workload in the `container` cgroup, so that the snapshot agent's freezer and scratch barriers engage | Each bare-metal host at 1 and 8 vCPUs, 3 times each, and the scratch variant on one bare-metal host at 1 vCPU, 3 times; [CI's `snapshot-core`](../ci.md) exercises a request without a destination, which OpenVMM releases before preflight, and asserts that the guest continues once, that `nvx-time status` passes at `generation=0` with no restore, and that `rcu_cpu_stall_suppress` reads 0 | OpenVMM logs the rollback (`microVM snapshot failed before commit; attempting rollback`, then `microVM snapshot rollback succeeded; guest resumed`), with no time ABI code, and each failed request returns in the same VM: status bit 1 clear, `nvx-time status` passing at `generation=0` with `discontinuities` unchanged (from the capture's checks; `cancel-capture` records nothing), the saved values back at their values before the request (`rcu_cpu_stall_suppress`, and on the debug kernel `soft_watchdog` and `hung_task_timeout_secs`) with no saved-values file left, and no 193, 194, or 195 and no clock message; the second request captures, which step 2 of the snapshot agent refuses while the first request's saved values remain, and its snapshot restores; that capture's `cpu_us` is reported but not budgeted (see [`cpu_us` budgets](#performance-expectations-and-acceptance-gate)). The companion logs `microVM snapshot preflight failed; guest continues` with no capture anchor and no rollback, and its guest side is the same. In the scratch variant, the failed request also leaves the workload cgroup thawed (`cgroup.freeze` reads 0), the scratch filesystem taking a write with `fsync` within 1 s, and the workload running, and the real capture and restore with the scratch device pass | -| Processor activation from one boot-online CPU to 2, 4, and 8 | All backends | Restored; warp probe passes | -| Every tier and an untiered snapshot, restored more than once | All backends | Restored; each restore of a snapshot captured at `g = 0` carries `g = 1` and a new generation ID. OpenVMM cannot capture a restored process, so unit tests cover the generation arithmetic beyond one restore, including `E_GENERATION_EXHAUSTED` | -| CPU profile fallback: a cold boot with `auto` and the `host-cpu-unknown` hook, captured, then restored with `auto` and no hook | All backends, in CI's `cpu-profile-fallback` scenario | The cold boot falls back to the host's host profile with one `NVX-CPU-PROFILE-FALLBACK:` warning that links the CPU profile request form, and passes its boot checks on it; the restore takes the snapshot's host profile without a warning, and passes the warp probe and its restore checks | - -No test reboots a host. The orchestrator asks the user before any real -reboot. - -**Reliability.** Repeated snapshot scenarios at 1, 2, 4, and 8 vCPUs on the -hosts the fleet can use (see Conformance), with the warp probe and the -watcher active, report no RCU, soft-lockup, hung-task, clocksource, or warp -message. The CI debug-kernel variant runs the same-host cases with the -soft-lockup and hung-task detectors enabled. - -**Performance.** The gate above, computed by the performance agent from the -CI benchmark matrix. - -Test hooks are hidden OpenVMM options (`--x-time-abi-test-hook `, -repeatable): - -| Hook | Effect | -| --- | --- | -| `force-utc-downtime` | Select the UTC source even on the same host boot | -| `boot-id-mismatch` | Treat the destination boot identity as different | -| `dest-rate-offset-ppm=` | Perturb the measured `F_d` used by the rate policy and the reported deviation; the TSC is never scaled | -| `downtime-add-s=` | Add `n` seconds to the measured `D` before the bounds check | -| `utc-offset-ms=` | Add `n` milliseconds to every destination UTC reading: the downtime sample, the packet, and time samples | -| `sample-delay-us=` | Delay the handling of the first `0xa5` write and of every `0xa7` write by `n` µs, which widens the guest's bracket | -| `host-cpu-unknown` | On a cold boot with `--cpu-profile auto`, treat the host CPU as one that no pinned profile serves, so that `auto` falls back to a host profile, and warns, as on such a CPU | - -Every active hook is logged at warning level and sets `TEST_HOOKS` in the -packet and in every time sample. - -## Removed mechanisms and migration - -Removed from OpenVMM's microVM paths; standard-machine VMs keep their -behavior: - -- Cold-boot clock tokens: `prepare_cold_boot_command_line` no longer injects - `tsc_early_khz=` or `lapic_timer_hz=`, and their propagation, parsing, and - platform-tier validation are deleted. -- The exact-equality CPU contract (`CpuCompatibilityContract`, including its - `tsc_deadline` and `kvm_clock` fields) and host-derived CPUID, replaced by - CPU profiles. -- Leaf `0x15` synthesis on every backend. `tsc_frequency_cpuid_leaves` and - WHP's `add_frequency_leaves` stay for standard-machine VMs, such as WHP VMs - with `--hv`. -- Per-backend downtime paths (`advance_snapshot_time`), per-VP TSC - advancement (`advance_tsc`), TSC-deadline advancement, periodic LAPIC - advancement, wall-clock-only downtime (`calculate_snapshot_downtime`), and - the per-VP TSC writes of saved VP state on restore. -- KVM: the microVM downtime use of `KVM_GET_CLOCK`/`KVM_SET_CLOCK` (the - Hyper-V reference time source keeps them; it requires `hv1`, which the time - ABI never enables), kvmclock MSR state, KVM CPUID leaves, `KVM_SET_TSC_KHZ`, - and restore-time `IA32_TSC` writes. -- MSHV: BSP-copy TSC alignment and exact rate equality. -- WHP: the 1 GHz rate request and its silent fallback, and `RestoredTsc` - with its RDTSC, RDTSCP, and `IA32_TSC` exits. -- Restore packets v1 to v3, and manifest versions 2 to 5 for every snapshot: - standard-machine disk snapshots also move to version 6, so earlier ones - must be recaptured too. -- The effect of `--restore-entropy`: every restore exposes packet v4, which - carries fresh entropy. OpenVMM still accepts the option without effect, - and the harness stops passing it. - -Removed from NVX: - -- Kernel: patch 0004 (`lapic_timer_hz`); `CONFIG_CPU_FREQ`, - `CONFIG_X86_INTEL_PSTATE`, and `CONFIG_SCHED_MC_PRIO` (which selects both - and needs ACPI CPPC data the microVM lacks); and `CONFIG_KVM_GUEST` with - `CONFIG_PARAVIRT_CLOCK` and `CONFIG_HALTPOLL_CPUIDLE`, which are dormant - without a KVM signature. `CONFIG_HYPERVISOR_GUEST` and `CONFIG_PARAVIRT` - stay on and `CONFIG_HYPERV` stays off. `CONFIG_IRQ_TIME_ACCOUNTING` stays - off: it slows WHP cold boots by 15 to 24 ms on bare metal, because the guest - then first-touches about 400 more pages, each a memory-access exit on WHP, - and it changes no `cpu_us` measurably. The CI - debug kernel adds `CONFIG_DEBUG_KERNEL` with the soft-lockup and hung-task - detectors only. -- Guest: RTC polling in `nvx-reseed`, and the restore packet v1 to v3 parser - in `nvx-port-io`. -- Harness: `tsc=reliable` and `no_timer_check` in `BASE_TUNING`; per-backend - `clocksource=` tokens; `stable_clocksource_wait_script` and the - capture-time clocksource waits; the KVM and WHP branches of - `snapshot-core`; `smp-lapic`'s `lapic=notscdeadline` boot, its runs in CI, - and the benchmarks' LAPIC gate, which `smp` makes redundant now that every - guest uses the counting LAPIC (#286; the scenario remains for explicit - local use); `restore-tsc-sync`, `clearcpuid=tsc_adjust`, and the fresh-boot - TSC control, which the warp probe replaces; and the dmesg greps in - `restore-processors.sh`, which the watcher replaces. The - `cold_start_clocksource` benchmark scenario passes `clocksource=tsc` on - every backend. - -Migration impact: - -- Every snapshot from a release before the time ABI (manifest version 5 or - older) is rejected with `E_SNAPSHOT_VERSION`; templates must be - recaptured. -- OpenVMM, the kernel, and the initramfs must be upgraded together. A new - guest on an old OpenVMM fails the boot check (status 193); an old guest's - snapshot agent cannot parse packet v4 and fails closed. -- The guest identifies as Hyper-V on every backend, including KVM. -- CPUID becomes profile-defined, so guests can lose host features that no - profile of their generation pins. -- Hosts that fail qualification cannot run microVMs on a pinned profile until - replaced; an Intel or AMD development host boots on a host profile instead, - which `--cpu-profile auto` falls back to where no pinned profile serves its - CPU, and `--cpu-profile host` selects. A host without an invariant TSC, such - as the 8370C MSHV runner, qualifies when its guests stay within the skew - bound (see [Qualification gates](#qualification-gates)). -- Per-PR CI captures and restores on the same runner. Same-generation - cross-VM restore is validated by the fleet restore matrix on WHP only; no - usable KVM or MSHV pair of one generation exists, so the simulated host - reboot covers those backends. -- The `cold_start_clocksource` metric on KVM changes meaning (from - `kvm-clock` to `tsc`). The change is accepted without an exemption; a gate - failure on it is investigated as a regression. -- Documentation updates: this document, [Snapshot and - restore](snapshot-and-restore.md), [Machine and device - ABI](machine-and-device-abi.md), [Cold boot](cold-boot.md), the - benchmarks and CI guides, and the OpenVMM Guide (see the appendix). - -## Appendix: OpenVMM Guide - -The OpenVMM Guide documents the user-facing contract in -`Guide/src/user_guide/openvmm/snapshots.md`: - -- **Overview:** the time contract and CPU profile in microVM manifests, the - contract's interval from the VM-time cut to the capture anchor, and - manifest version 6 as the only accepted version. -- **Restoring a snapshot:** restore packet v4, the time sample at `0xeb`, the - generation ID, the synchronized TSC set with its read-back, and the VM-time - and RTC advance by the downtime plus the VM-time cut interval. -- **Time and CPU compatibility:** the identity, the CPU profile and - `--cpu-profile`, including host profiles and `auto`'s fallback to them with - its warning, the 250 ppm rate rule, the exact LAPIC rule, the downtime - sources and bounds, and the error codes. -- **Limitations:** the same backend, CPU generation, and profile, a host - profile's snapshot restoring only on the capture host's CPU model and - stepping, and the recapture of older snapshots. - -The CLI reference documents `--cpu-profile`, with `auto`'s fallback and its -warning, `--cpu-fingerprint`, whose check never falls back, and -`--x-time-abi-verify`. The test hooks stay undocumented in the Guide; this -document describes them. diff --git a/doc/design/validation.md b/doc/design/validation.md deleted file mode 100644 index e430b87e7..000000000 --- a/doc/design/validation.md +++ /dev/null @@ -1,184 +0,0 @@ -# Validation - -[Design index](../design.md) - -The implementation is exercised at three levels: - -- loader, command-line, memory-layout, RTC, PMIO, network-policy, snapshot - format, and device-private-state unit tests; -- OpenVMM VMM tests invoked through `nvx.py test-openvmm`: a test-harness - microVM lifecycle test and a management-RPC lifecycle, SMP, and snapshot - test that uses NVX's ACPI-free, MP-enabled x86-64 Linux-direct kernel and - Alpine initramfs; and -- NVX-owned process tests using this repository's Linux kernel and selected - Alpine, Ubuntu, or Azure Linux initramfs through the public OpenVMM CLI. - -Alpine, the only sandbox-control guest, supports every NVX-owned scenario. -Ubuntu and Azure Linux reject the scenarios that -[`test-microvm`](../usage.md#test-microvm) lists, and their guest chapters -list what CI runs on them: [Ubuntu](../guests/ubuntu-guest.md#validation) and -[Azure Linux](../guests/azurelinux-guest.md#validation). - -The harness lifecycle test covers portb I/O, status shutdown, rejection of -host save and pulse save/restore, and a snapshot request that continues -without a configured destination. With NVX's kernel, the management-RPC test -boots 1, 2, 4, and 8 vCPUs, captures a snapshot with an immutable RAM capacity -and a boot-online prefix through the RPC, restores it twice with readiness -signaling and processor and memory targets, and verifies that the artifacts -are unchanged. - -The NVX-owned suite boots the same Linux-direct artifacts on the available native -backend and covers IRQ0/RTC behavior, raw portb I/O, shutdown status, exact -snapshot sequencing, repeated immutable restore, time ABI conformance and -restore downtime, fresh -generation IDs, `getrandom()` output, kernel UUIDs, temporary-file identifiers, -entropy reseed, active console RX/TX, network policy and HTTP traffic, and live -virtio-fs attachment revalidation, including a guest-created symbolic link held -across capture and a capture right after the guest swaps two files and two -directories with `RENAME_EXCHANGE`, which Linux hosts perform and Windows hosts -reject without changing either name. Directional network coverage verifies an -egress request and response with ingress denied, denial of a host connection -to an active guest listener, and complete egress denial on KVM, MSHV, and WHP. -Private-network ingress coverage verifies, with egress denied, that the -guest's TCP and UDP listeners are published at their own ports on the host, -that the host reaches them and gets their replies when host loopback is -allowed, that a closed TCP listener is withdrawn, and that the guest still -cannot open connections of its own; that the host's connections from loopback -and from its own addresses are reset, and its datagrams go unanswered, when -host loopback is denied; that ingress denied publishes nothing; and that a -snapshot with private-network ingress is rejected before boot. Unit tests -classify the peers of other hosts. L3/L4 coverage verifies TCP and UDP -port rules, overlapping deny precedence, default-deny behavior, and pre-boot -rejection of malformed or incomplete rules. Protocol coverage verifies that -TCP, UDP, and ICMP rules without a port, `any` with a port, and `any` alone -each match exactly their traffic, both as allow rules and as deny rules that -take precedence over an allow rule, with the other protocols and the next port -as nearby cases. Port-range coverage verifies that TCP, UDP, and `any` ranges -each match their first and last ports and a port inside them, but neither the -port below nor the port above them nor another protocol, both as allow rules -and as deny rules that take precedence over an allow rule; that a deny rule for -one port inside an allowed range blocks that port alone; that each range -reaches OpenVMM as one native range rule; and that OpenVMM and `nvx.py run` -reject reversed ranges and ranges without a first port before launch. IPv6 -coverage verifies, with MXC-shaped policy files, the guest's derived IPv6 -address and default route, TCP, UDP, and ICMPv6 to the IPv6 gateway, a `deny` -default without rules that blocks both families, port and protocol rules, an -IPv6 port range, deny precedence within `::/0`, a rule without destinations that -admits both families, a `deny` default that IPv4 rules do not widen to IPv6, an -IPv6 deny rule under an `allow` default that leaves IPv4 alone, the DNS server -that each policy gives the guest, the IPv6 path across snapshot restore, -rejection of a restore whose policy differs, and pre-boot rejection of an -invalid IPv6 prefix. The Ubuntu and Azure Linux virtio-net smoke tests also -verify the IPv6 identity. -Host-loopback coverage verifies -general guest-to-host denial with no announcer, an exact proxy exception, and, -under explicit allow with egress denied but for the guest's connection to a -host service, that the guest reaches that service through its gateway and that -the host exchanges TCP lines with a guest server over IPv4 and IPv6 loopback -and through an explicit forward to the same guest port, and UDP datagrams and -their replies with guest servers on published ports over IPv4 and IPv6 -loopback. It also verifies that a guest listener on -guest loopback alone is not published, that no host network address reaches the -guest, and pre-boot rejection of an explicit allow with a snapshot, of a -forward under deny, and of a mismatched proxy. Managed lifecycle coverage -authenticates the -dedicated control channel, runs multiple workloads in one warm VM across -reconnects, preserves guest state, reports execution timeout, and stops the VM -cleanly. Public sandbox coverage drives `nvx.py sandbox` itself: request-scoped -execution options, identity refusal, fail-closed lifecycle transitions, -scratch-backed state across requests and restarts, bounded output and outcome -reports, a crashed OpenVMM's stale runtime state, and teardown that leaves no -OpenVMM process, control endpoint, capability, or runtime record while the -supplied layer and scratch survive. Its guest probe verifies the sandbox's -identity, capability, `no_new_privs`, namespace, device, sysfs, cgroup, -resource-limit, image-layer, and scratch invariants. Sandbox coverage adds -deterministic active block-I/O drain, paired -scratch publication, two private restores, fresh scratch replacement, and -pre-entry rejection of missing, corrupt, mismatched, or wrong-geometry media. -Denied-filesystem coverage verifies listing suppression, allowed writes, -direct and parent-relative denial, symlink/junction alias denial, guest-created -links into the denied subtree, a second -virtio-fs mount, and pre-boot rejection of unsafe path policies. -Caller-ownership coverage verifies that files guest root creates belong to the -export owner, that squashed root cannot chown files or create device nodes, -that a foreign guest identity fails with `EPERM` unless OpenVMM holds -`CAP_SETUID` and `CAP_SETGID`, in which case it owns what it creates, and -pre-boot rejection of a root-owned export or, on Windows, of the mode itself. -Concurrent-share coverage attaches a read-write workspace, a read-only tool -cache, and a data directory as the children of one aggregate on the microVM's -single virtio-fs slot, each with its own denied paths. The data directory -hides its root except for one directory and one file, the only writable path -in it. The guest checks that each child is bound at its target with its -mode, writes to the workspace and to that file, cannot write to the tool -cache, also through a symbolic link, and cannot create names in the hidden -root. Through the aggregate's read-write, root-only mount, OpenVMM still -returns `EROFS` for every mutation of the tool cache, keeps denied paths -hidden, and refuses hard links between children. OpenVMM rejects a repeated -`--mount`, `--mount` beside an aggregate, overlapping children, and a -relative denied path before boot. A snapshot round-trips an open workspace -file, and a restore that drops, swaps, or renames a child, or offers a single -`--mount` instead, fails before the guest resumes. -Access-policy coverage narrows a read-write share's writes to a writable -directory and file, and verifies guest writes, renames, and links inside them -and `EROFS` from OpenVMM for every other mutation, including moves into and -out of the writable paths and mutations through a second mount of the tag, as -well as `EXDEV` for a hard link of a read-only file. It denies a directory -except an allowed child, inside which a denied grandchild stays hidden, and -verifies that the denied directory lists and exposes only the allowed child. -It rejects invalid policies before boot and round-trips the policy through a -snapshot whose restore requires the same policy and keeps a writable handle. -Comma-path coverage shares a read-write `alpha,beta` and a read-only -`2026, final` directory through `nvx.py run`, each alone, as OpenVMM's -`--mount`, and both, as the children of an aggregate. It verifies that each -target shows its exact directory rather than the one beside it that the text -before the last comma names, that each denied subdirectory stays hidden, and -that OpenVMM refuses writes to the read-only directory with `EROFS` even after -guest root remounts it read-write. -The native suite targets KVM, MSHV, and WHP; a passing run on one backend is -not a fresh result for the others. -Coverage also includes -1/2/4/8-vCPU topology, APIC identity, -pinned per-vCPU execution, timer/interrupt progress, reset, cancellation, -count and topology mismatch rejection, and repeated immutable restore. -Restore-time processor coverage captures one capacity-8 template with a -boot-online count of one, restores it at 1/2/4/8 online VPs and without a -target, schedules work on every requested CPU, and verifies that the artifact -is unchanged. Its VP-binding lifecycle records verify that MSHV binds exactly -the requested prefix, while untargeted MSHV restores and all KVM and WHP -restores bind the full capacity. The profiled `snapshot-restore-vcpu` -benchmark reports the same VP-binding and worker-construction phases for -comparison with fixed-capacity restores. -Restore-time memory coverage captures 512 MiB with a 2-GiB capacity, restores -the same artifact at 512 MiB, 1 GiB, and 2 GiB, validates the added-byte count -and expanded allocation, and verifies artifact immutability. Unit coverage -verifies that only an explicit MSHV processor target selects a runtime prefix, -that the complete saved VP inventory is validated before filtering, that -reduced-prefix saves are rejected, that dormant VP access fails cleanly, and -that the MSHV synchronized TSC set targets only created VPs. -Additional unit coverage exercises the control-console slot and attachment -inventory, command-line spoofing rejection, the control-session record -protocol against language-neutral golden vectors and boundary cases, the -broker state machine and its save and restore, peer-identity and capability -admission for local endpoints, workload identity and lifecycle ownership, -the bounded outcome-report schema, egress-policy enforcement in the endpoint -and virtio-net layers, the virtio-fs microVM profile and its access policy, -state-unit quiesce and rollback, management exclusion at the snapshot -boundary, management-RPC guest-exit propagation, output-drain completion and -failures, and the time ABI's rate, downtime, LAPIC, restore-packet, and -manifest rules. Hardware-dependent time ABI tests, such as the synchronized TSC -set, identity routing, and the CPU profile read-back, still require their -native backend; the [time ABI test matrix](time-abi.md#test-matrix) lists the -conformance, restore, soak, and performance coverage. A host-only benchmark -measures snapshot publication, restore preparation, copy-on-write dirtying, and -fresh-process restore preparation without booting a guest. Platform CI and the -benchmark histories in `data/` provide the wider host matrix. - -The separate `test-adversarial` harness adaptively selects tracked -deterministic primitives from these same process tests. Copilot has no tools -or direct runtime access; a typed broker records the selection and a -credential-free executor runs the existing scenario on a disposable target. -Independent filesystem/network canaries, resource heartbeats, complete -teardown, and a fresh post-campaign lifecycle boot add containment and -availability oracles. Unit tests inject canary, network, teardown, timeout, -schema, replay, and Copilot-permission faults. Hardware-backed campaigns remain -backend-specific and do not replace deterministic pull-request coverage. diff --git a/doc/guests/azurelinux-guest.md b/doc/guests/azurelinux-guest.md index d8a72d95f..111601a66 100644 --- a/doc/guests/azurelinux-guest.md +++ b/doc/guests/azurelinux-guest.md @@ -1,12 +1,10 @@ # Azure Linux guest support -[Design index](../design.md) - NVX runs Azure Linux 3.0 userland on its own Linux kernel. It builds one artifact, `initramfs-azurelinux.cpio.gz`, which `nvx.py run --guest azurelinux` boots for an interactive shell or a workload. The artifact boots no Azure Linux kernel, firmware, or systemd, and it does not change the -[machine and device ABI](../design/machine-and-device-abi.md). +[machine and device ABI](../design/2-virtual-machine-architecture.md). [Build](../build.md), [Run](../run.md), and the [command-line reference](../usage.md) cover the commands. This chapter @@ -82,10 +80,10 @@ automated check compares two Azure Linux rebuilds. The Azure Linux initramfs boots like the Alpine one at the machine level: OpenVMM loads the kernel and initramfs through the -[Linux direct MP-table loader](../design/cold-boot.md#linux-direct-mp-table-loader), +[Linux direct MP-table loader](../design/5.1-boot-protocol.md#511-linux-direct-loader), and the common [`/init`](../../guest/common/init) runs as PID 1. It runs the same boot steps as for the other guests, including the -[time ABI](../design/time-abi.md) boot step, and then runs an `nvx_exec` +[time ABI](../design/3.2-clock-and-time.md) boot step, and then runs an `nvx_exec` workload, starts the managed agent, or opens a shell. The managed agent runs the workload directly in the initramfs root, not in a separate layer. On the shell path, `/init` reads the guest identity from `/etc/os-release`, stops on diff --git a/doc/guests/ubuntu-guest.md b/doc/guests/ubuntu-guest.md index b0ca8486c..afebde1a5 100644 --- a/doc/guests/ubuntu-guest.md +++ b/doc/guests/ubuntu-guest.md @@ -1,13 +1,11 @@ # Ubuntu guest support -[Design index](../design.md) - NVX runs Ubuntu userland on its own Linux kernel. It builds two artifacts from one pinned Ubuntu Base root filesystem: an initramfs that `nvx.py run` boots for an interactive Ubuntu shell, and an EROFS `distro` layer that `nvx.py sandbox` runs as a non-root workload. Neither artifact boots an Ubuntu kernel, firmware, or systemd, and neither changes the -[machine and device ABI](../design/machine-and-device-abi.md). +[machine and device ABI](../design/2-virtual-machine-architecture.md). [Build](../build.md), [Run](../run.md), and the [command-line reference](../usage.md) cover the commands. This chapter @@ -97,9 +95,9 @@ order, so a rebuild from the same inputs is byte-identical. The Ubuntu initramfs boots like the Alpine one at the machine level: OpenVMM loads the kernel and initramfs through the -[Linux direct MP-table loader](../design/cold-boot.md#linux-direct-mp-table-loader), +[Linux direct MP-table loader](../design/5.1-boot-protocol.md#511-linux-direct-loader), and the common [`/init`](../../guest/common/init) runs as PID 1. It disables -module loading, runs the [time ABI](../design/time-abi.md) boot step, and then +module loading, runs the [time ABI](../design/3.2-clock-and-time.md) boot step, and then either runs an `nvx_exec` workload or mounts the HostFs share, configures static networking, and starts the managed agent or an interactive shell. For Ubuntu that shell is Bash, whose startup file prints @@ -150,8 +148,7 @@ supplementary groups, no capabilities, and `no_new_privs`. The sandbox rejects systemd entrypoints, and it rejects a distro layer whose manifest lists the `systemd` package. Systemd needs a separate compatibility -profile; see -[workload compatibility](../design/remaining-production-work.md#workload-compatibility-and-volumes-proposed). +profile. ## Provenance and corresponding source diff --git a/doc/project-structure.md b/doc/project-structure.md index a465bc3ad..f97d4acf8 100644 --- a/doc/project-structure.md +++ b/doc/project-structure.md @@ -69,12 +69,12 @@ nvx/ |-- docker/ | `-- Dockerfile Reproducible guest build environment |-- doc/ User and contributor documentation +| |-- adversarial-testing.md Copilot-driven adversarial testing | |-- benchmarks.md Benchmark commands and measurement methodology | |-- build.md Guest and OpenVMM build workflows | |-- ci.md Continuous integration overview | |-- contribute.md Contribution guidelines -| |-- design.md Design index -| |-- design/ MicroVM, sandbox, and snapshot design chapters +| |-- design/ Architecture of the microVM and its guest runtime | |-- distribution.md Packaging and source delivery | |-- guests/ Guest userland chapters (Ubuntu and Azure Linux) | |-- openvmm-upstream-roadmap.md Plan for upstreaming the OpenVMM fork's microVM commits diff --git a/doc/run.md b/doc/run.md index 4b47a2f2a..5690cad05 100644 --- a/doc/run.md +++ b/doc/run.md @@ -751,7 +751,7 @@ Systemd entrypoints are explicitly unsupported and do not relax the non-root, drop-all-capabilities sandbox policy. This is the cold-filesystem bootstrap described in -[the sandbox design](design/sandbox-filesystem-and-agent-architecture.md#implemented-filesystem-bootstrap), not the final +[the sandbox design](design/4.1-guest-filesystem-and-image-model.md#411-layered-sandbox-root), not the final production agent. It accepts no environment variables or secrets and does not expose sandbox snapshot capture or restore, the configuration region, or runtime RPC. Arguments are individual kernel-command-line tokens and therefore cannot diff --git a/doc/setup.md b/doc/setup.md index 75e5ce37f..b44e66bd3 100644 --- a/doc/setup.md +++ b/doc/setup.md @@ -236,7 +236,7 @@ satisfy this prerequisite. Production campaigns also require an administrator-owned, no-argument executor wrapper that provisions a separate disposable target and forwards the bounded executor protocol. See -[Copilot-driven adversarial testing](design/copilot-adversarial-testing.md) +[Copilot-driven adversarial testing](adversarial-testing.md) for the containment and credential requirements. Apply the configured Python and POSIX shell formatters with: diff --git a/doc/usage.md b/doc/usage.md index 1c29cd395..7fc4af44d 100644 --- a/doc/usage.md +++ b/doc/usage.md @@ -318,8 +318,8 @@ python3 scripts/nvx.py doctor [--timeout SECONDS] ``` -Qualifies this host for the [time ABI](design/time-abi.md#host-qualification). -It runs the selected checks in spec order, prints one +Qualifies this host for the [time ABI](design/3.2-clock-and-time.md). +It runs the selected checks in order, prints one `NVX-DOCTOR: check= status= detail="..."` line per check, and exits with status 1 if any check fails. A check fails, and never passes, when it can't read a fact it gates on. [CI](ci.md#host-qualification) describes each @@ -328,7 +328,7 @@ check and the subset that CI runs. | Option | Default | Description | | --- | --- | --- | | `--backend {kvm,mshv,whp}` | required | Select the backend to qualify. | -| `--checks ID ...` | `H1` to `H7` | Run only these checks, from `H1` to `H7`, in spec order. | +| `--checks ID ...` | `H1` to `H7` | Run only these checks, from `H1` to `H7`, in order. | | `--openvmm PATH` | `openvmm/target/release/openvmm[.exe]` | Select the OpenVMM binary for `H2`, `H3`, and `H6`. | | `--kernel PATH` | `build/vmlinux` | Select the guest kernel for `H3` and `H6`. | | `--initrd PATH` | `build/initramfs.cpio.gz` | Select the guest initramfs for `H3` and `H6`. | @@ -421,7 +421,7 @@ must use a separate disposable executor through `--executor-command`; local mode cannot reliably classify a target-host crash. Local target logs are kept under the short `build/adv` state root to avoid Windows path-length failures; the run summary records their absolute location. See -[Copilot-driven adversarial testing](design/copilot-adversarial-testing.md) +[Copilot-driven adversarial testing](adversarial-testing.md) for the trust boundary, wrapper contract, artifacts, and CI policy. ## Download and run @@ -639,7 +639,7 @@ snapshot/configuration limitations. Every microVM presents a CPU profile to its guest: the complete CPUID surface of one CPU generation, which OpenVMM pins and every backend shares (see the -[time ABI](design/time-abi.md#cpu-profiles)). By default, OpenVMM selects the +[CPU profile design](design/2.2-cpu-and-topology.md#221-cpu-profiles)). By default, OpenVMM selects the built-in profile of the host's CPU generation: | Profile | Generation | CPUs (display family/model) | diff --git a/guest/common/nvx-snapshot b/guest/common/nvx-snapshot index 21c641c89..70eef4b00 100644 --- a/guest/common/nvx-snapshot +++ b/guest/common/nvx-snapshot @@ -26,7 +26,7 @@ esac runtime=/run/nvx container_cgroup=/sys/fs/cgroup/container scratch_mount=$runtime/scratch -# Restore packet v4 flags (doc/design/time-abi.md). +# Restore packet v4 flags (doc/design/5.5-restore-protocol.md). PACKET_MEMORY_TARGET=2 PACKET_ACK_REQUIRED=4 container_frozen=false @@ -287,7 +287,7 @@ esac export NVX_VM_GENERATION_ID="$generation_id" entropy=$runtime/restore-entropy -# Restore steps 9 to 13 of the time ABI (doc/design/time-abi.md). nvx-time +# Guest repair steps 4 to 7 (doc/design/5.5-restore-protocol.md). nvx-time # capture has already read restore packet v4, checked its generation, set # the wall clock, and written the entropy. Its metadata is FLAGS, # ONLINE_VP_COUNT, RANGE_COUNT, and the new generation ID, followed by START diff --git a/guest/common/nvx-time.c b/guest/common/nvx-time.c index 222016716..3b806a68e 100644 --- a/guest/common/nvx-time.c +++ b/guest/common/nvx-time.c @@ -3,10 +3,10 @@ // nvx-time: the guest time component of NVX time ABI v1. // -// doc/design/time-abi.md ("Guest obligations") is the contract this helper -// enforces. It runs the conformance checks, the violation watcher and the -// wall-clock discipline (one daemon), and the snapshot agent's time steps. It -// prints nothing on success: checks record their result in the state file, +// doc/design/3.2-clock-and-time.md ("Guest Obligations") is the contract this +// helper enforces. It runs the conformance checks, the violation watcher and +// the wall-clock discipline (one daemon), and the snapshot agent's time steps. +// It prints nothing on success: checks record their result in the state file, // and only violation events reach the console. // // boot initial clock step (C12), then the other boot diff --git a/scripts/nvx_tools/doctor.py b/scripts/nvx_tools/doctor.py index c305d14ac..07223175b 100644 --- a/scripts/nvx_tools/doctor.py +++ b/scripts/nvx_tools/doctor.py @@ -1,6 +1,6 @@ """Host qualification for the NVX time ABI (``nvx.py doctor``). -doc/design/time-abi.md ("Host qualification") defines checks H1 to H7. Each +doc/ci.md ("Host qualification") describes checks H1 to H7. Each check prints one ``NVX-DOCTOR: check= status= detail="..."`` line, and qualification fails closed if any selected check fails. A failure that matches a time ABI failure code starts its detail with that code in @@ -70,7 +70,7 @@ VERIFY_PREFIX = "NVX-TIME-ABI-VERIFY:" CPU_PROFILE_PREFIX = "NVX-CPU-PROFILE:" RUST_ESCAPE = re.compile(r"\\(u\{[0-9a-fA-F]{1,6}\}|.)") -# doc/design/time-abi.md, "Rate stability (H4)". +# H4's rate stability bounds (doc/ci.md, "Host qualification"). RATE_UNCERTAINTY_PPM = 0.25 RATE_AGREEMENT_PPM = 1.0 RATE_TOLERANCE_PPM = 100.0 @@ -88,8 +88,8 @@ class Schedule: warp_gaps: tuple[str, ...] -# doc/design/time-abi.md, "Rate stability (H4)" and "Warp schedules": doctor -# qualifies on the long schedules, and CI on the short ones. +# H4's rate samples and H6's warp gaps (doc/ci.md, "Host qualification"): +# doctor qualifies on the long schedules, and CI on the short ones. QUALIFICATION_SCHEDULE = Schedule(13, 10_000, QUALIFICATION_WARP_GAPS) CI_SCHEDULE = Schedule(3, 1_000, CI_WARP_GAPS) # Qualification gates only on measured properties, alike on every backend: @@ -1135,8 +1135,7 @@ def default_probe_directory() -> Path: def configure_parser(parser: argparse.ArgumentParser) -> None: parser.description = ( - "Qualify this host for the NVX time ABI (doc/design/time-abi.md, " - "'Host qualification')." + "Qualify this host for the NVX time ABI (doc/ci.md, 'Host qualification')." ) parser.add_argument("--backend", choices=OPENVMM_TEST_BACKENDS, required=True) parser.add_argument( diff --git a/scripts/nvx_tools/host_time_probe.rs b/scripts/nvx_tools/host_time_probe.rs index a92102530..4285b1f3c 100644 --- a/scripts/nvx_tools/host_time_probe.rs +++ b/scripts/nvx_tools/host_time_probe.rs @@ -1,11 +1,12 @@ //! Host TSC probe for NVX time ABI host qualification. //! //! `nvx.py doctor` builds this file with `rustc` and runs it for the host -//! checks of doc/design/time-abi.md: H2 (the CPU identity and the host's -//! invariant-TSC bit), H4 (sleep-separated samples of the TSC against the -//! host's monotonic clocks), and H5 (cross-CPU TSC skew between pinned -//! threads). It has no dependencies and prints `NVX-HOST-TIME-PROBE` lines of -//! `key=value` fields; the doctor applies the bounds. +//! checks that doc/ci.md ("Host qualification") describes: H2 (the CPU +//! identity and the host's invariant-TSC bit), H4 (sleep-separated samples of +//! the TSC against the host's monotonic clocks), and H5 (cross-CPU TSC skew +//! between pinned threads). It has no dependencies and prints +//! `NVX-HOST-TIME-PROBE` lines of `key=value` fields; the doctor applies the +//! bounds. use std::process::ExitCode; use std::sync::Arc; diff --git a/scripts/nvx_tools/time_abi.py b/scripts/nvx_tools/time_abi.py index a3c4bae4d..24b2d0903 100644 --- a/scripts/nvx_tools/time_abi.py +++ b/scripts/nvx_tools/time_abi.py @@ -1,10 +1,11 @@ """Harness checks of the NVX time ABI v1 guest obligations. -doc/design/time-abi.md defines what a conforming guest prints and how it fails: -the ``NVX-TIME-ABI`` marker of each conformance check, the -``NVX-TIME-ABI-VIOLATION`` event, and the power-off statuses 193 (conformance), -194 (runtime violation), and 195 (restore repair). It also fixes the warp -probe's 1 us skew bound and the CPU generation names used in reports. +A conforming guest reports each conformance check with an ``NVX-TIME-ABI`` +marker and each runtime violation with an ``NVX-TIME-ABI-VIOLATION`` event, and +fails with the power-off statuses 193 (conformance), 194 (runtime violation), +and 195 (restore repair). doc/design/3.2-clock-and-time.md describes these +guest obligations and the 1 us cross-CPU skew bound that the warp probe checks, +and doc/design/2.2-cpu-and-topology.md the CPU generations named in reports. """ from __future__ import annotations @@ -37,16 +38,15 @@ } MIN_TSC_HZ = 500_000_000 MAX_TSC_HZ = 10_000_000_000 -# doc/design/time-abi.md, "Performance expectations and acceptance gate", -# "cpu_us budgets": each backend has a CPU-time (cpu_us) budget per phase -# (boot, capture, and restore): a base plus an increment per additional online -# CPU, in microseconds, held per sample. These are the spec's final values, each -# at least 1.2 times the largest sample from all fleet validation (bare metal -# and Azure, production and debug kernels). Triaged outliers don't count, and -# neither does the first capture after a rolled-back capture, which the spec -# exempts and CI never makes. This table is the one place the harness reads -# them. The checks' wall time (elapsed_us) has no budget. CI reports both and -# gates on neither: the performance gate is the A/B comparison outside CI. +# Each backend has a CPU-time (cpu_us) budget per phase (boot, capture, and +# restore): a base plus an increment per additional online CPU, in +# microseconds, held per sample. Each budget is at least 1.2 times the largest +# sample from all fleet validation (bare metal and Azure, production and debug +# kernels). Triaged outliers don't count, and neither does the first capture +# after a rolled-back capture, which is exempt and which CI never makes. This +# table is the one place the harness reads them. The checks' wall time +# (elapsed_us) has no budget. CI reports both and gates on neither: the +# performance gate is the A/B comparison outside CI. CHECK_CPU_BUDGET_US: Mapping[str, Mapping[str, tuple[int, int]]] = { "kvm": {"boot": (6_000, 2_000), "capture": (1_500, 400), "restore": (7_000, 2_000)}, "mshv": {"boot": (3_000, 1_000), "capture": (1_000, 400), "restore": (2_500, 500)}, @@ -66,16 +66,17 @@ # The idle-inducing warp schedule: probe rounds separated by halted vCPUs. WARP_PROBE_ROUNDS = 2 WARP_PROBE_IDLE_SECONDS = 1 -# Idle gaps between probe rounds, in seconds (doc/design/time-abi.md, "Warp -# schedules"): CI's schedule after boot and every restore, and H6's. +# Idle gaps between probe rounds, in seconds: CI's schedule after boot and +# every restore, and H6's (doc/ci.md). CI_WARP_GAPS: tuple[str, ...] = (str(WARP_PROBE_IDLE_SECONDS),) * ( WARP_PROBE_ROUNDS - 1 ) QUALIFICATION_WARP_GAPS: tuple[str, ...] = ("0.1", "1", "5", "1") PROFILE_VENDORS: Mapping[str, str] = {"GenuineIntel": "intel", "AuthenticAMD": "amd"} # The generation of every host profile, which `openvmm --cpu-profile host` -# derives from the host it runs on (doc/design/time-abi.md, "Selection"). No -# pinned profile uses it, and host qualification never accepts one. +# derives from the host it runs on (doc/design/2.2-cpu-and-topology.md, "CPU +# Profile Selection"). No pinned profile uses it, and host qualification never +# accepts one. HOST_PROFILE_GENERATION = "host" _PROFILE_ID = re.compile(r"([a-z0-9-]+)\.([a-z0-9-]+)\.v[1-9][0-9]*") @@ -238,7 +239,8 @@ def openvmm_cpu_generations(openvmm_dir: Path) -> tuple[CpuGeneration, ...]: HOST_PROFILE_VENDORS = frozenset({"GenuineIntel", "AuthenticAMD"}) # OpenVMM leads the warning of every cold boot whose `--cpu-profile auto` falls # back to a host profile with this marker, followed by the host CPU and the -# profile with its digest (doc/design/time-abi.md, "Selection"). +# profile with its digest (doc/design/2.2-cpu-and-topology.md, "CPU Profile +# Selection"). CPU_PROFILE_FALLBACK_MARKER = "NVX-CPU-PROFILE-FALLBACK:" _CPU_PROFILE_FALLBACK = re.compile( rf"{re.escape(CPU_PROFILE_FALLBACK_MARKER)} .*?, so --cpu-profile auto fell back" @@ -246,7 +248,7 @@ def openvmm_cpu_generations(openvmm_dir: Path) -> tuple[CpuGeneration, ...]: ) # The issues that track built-in CPU profiles for more Intel and more AMD CPUs. # No issue tracks the CPUs of other vendors, which the time ABI does not serve -# (doc/design/time-abi.md, "Non-goals"). +# (doc/design/2.2-cpu-and-topology.md, "CPU Profile Selection"). CPU_SUPPORT_ISSUES: Mapping[str, tuple[str, str]] = { "GenuineIntel": ("Intel", "https://github.com/microsoft/nvx/issues/408"), "AuthenticAMD": ("AMD", "https://github.com/microsoft/nvx/issues/409"), diff --git a/scripts/test_guest_time.py b/scripts/test_guest_time.py index 7de7daf3b..c60144734 100644 --- a/scripts/test_guest_time.py +++ b/scripts/test_guest_time.py @@ -1,8 +1,10 @@ #!/usr/bin/env python3 """Spec-derived tests for the guest time component, guest/common/nvx-time.c. -The fixtures are encoded here from doc/design/time-abi.md, independently of -the C decoders, so the tests check the guest against the specification. +The fixtures encode the time ABI's guest-visible formats, such as the restore +packet (doc/design/5.5-restore-protocol.md) and the time sample +(doc/design/3.2-clock-and-time.md), independently of the C decoders, so the +tests check the guest against the specification. """ from __future__ import annotations From d20ce7571345d11e90fc91499a18486dc7bcefc4 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Wed, 14 Oct 2026 18:34:08 -0300 Subject: [PATCH 3/3] doc: scope the entropy refresh guarantee to tiered clones Section 5.5.8 said that every clone consumes its fresh entropy before any workload code runs, and that a workload captured in a snapshot stays frozen until the guest acknowledges the restore. Both hold only for clones of platform and workload-start snapshots: the guest freezes the workload cgroup only for paired-scratch captures, and a platform snapshot is captured before the workload starts. An untiered snapshot, which is always blockless, has no such barrier. Its capture freezes nothing, so a process that was running at capture resumes with the replayed random state as soon as the restored processors run, and gating host input does not stop it. An untiered restore with nothing to activate does not reseed at all and hands the entropy to the snapshot's caller, as Table 5.7 records. Scope the guarantee and step 6 to clones of platform and workload-start snapshots, and state that the caller of an untiered snapshot must provide the execution barrier and, where the guest does not reseed, repair the kernel's and the runtime's random state with the entropy handed to it. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/5.5-restore-protocol.md | 18 +++++++++++++----- 1 file changed, 13 insertions(+), 5 deletions(-) diff --git a/doc/design/5.5-restore-protocol.md b/doc/design/5.5-restore-protocol.md index 3cdb40447..bf04443f0 100644 --- a/doc/design/5.5-restore-protocol.md +++ b/doc/design/5.5-restore-protocol.md @@ -292,8 +292,8 @@ Restoring a snapshot replays the guest's captured memory, including the state of the kernel's random-number generator and of any generators that the workload runtime keeps. Without a refresh, clones of one snapshot would produce the same random numbers, and therefore the same keys and identifiers. -To prevent this, every restore delivers fresh entropy, which a clone consumes -before any workload code runs: +To prevent this, every restore delivers fresh entropy, which a clone of a +`platform` or `workload-start` snapshot consumes before any workload code runs: 1. Before any processor runs, the VMM draws 64 bytes from the host operating system's cryptographic random-number generator. @@ -308,9 +308,9 @@ before any workload code runs: 5. The workload runtime's post-restore hook resets the runtime's own random-number state, which a kernel reseed cannot reach. 6. The guest agents acknowledge the restore and only then thaw scratch and - the workload cgroup. A workload captured in a snapshot stays frozen until - this point, and a platform snapshot is captured before the workload - starts, so no workload code runs with the replayed state. + the workload cgroup. A workload captured in a `workload-start` snapshot + stays frozen until this point, and a platform snapshot is captured before + the workload starts, so no workload code runs with the replayed state. As Table 5.7 shows, the refresh depends on the restore path. @@ -329,3 +329,11 @@ continuity of the resumed instance. Fresh post-restore entropy and the replacement generation ID are never stored in the reusable snapshot, so independent restores of one snapshot receive independent bytes; only the prior generation ID remains in captured guest state, as the comparison token. + +An untiered snapshot, which is always blockless, has no capture barrier: its +capture freezes no workload, so every process that was running at capture +resumes with the replayed random state as soon as the restored processors +run, and gating host input does not hold it back. The snapshot's caller must +therefore provide the execution barrier, for example by capturing before any +workload starts, and, where the guest does not reseed, repair the kernel's and +the runtime's random state with the entropy handed to it.