This repository was archived by the owner on Oct 22, 2026. It is now read-only.
Repository navigation
Conversation
Restoring an x86 snapshot is only safe on a partition that presents the guest with the CPU that it was saved on: the same vendor and CPUID results, XSAVE layout, and processor features. There is no canonical representation of that effective CPU that a snapshot could record and a restore could compare. Add `CpuCompatibilityContract`, a protobuf description of the effective CPU: the vendor, every CPUID leaf with its mask, the supported XCR0 and XSS features, the standard and compacted XSAVE sizes and component layout, the x2APIC, CET, SGX, and TSC_AUX features, and the physical address width. `CpuCompatibilityContract::new` builds it from the partition capabilities and the effective CPUID leaves. Add `Partition::cpu_compatibility_contract` for x86 guests, and implement it in the KVM, MSHV, WHP, and OpenHCL partitions from their capabilities and CPUID leaf sets. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: ef640993-0003-4e49-b1ed-1b8b4776d0ef
The saved state of a VP did not include IA32_TSC_DEADLINE, so a guest that arms its local APIC timer in TSC-deadline mode loses the pending deadline across a save and restore. Nothing could move a saved APIC timer forward by the time that the VM spent stopped either. Add a `TscDeadline` VP state element, which is present when the partition supports TSC-deadline mode, and record that support, as reported by CPUID, in the partition capabilities and the CPU compatibility contract. The deadline is not compared after restore, since it can expire between restoring it and reading it back. KVM, WHP, and the hypervisor-backed OpenHCL VPs access the IA32_TSC_DEADLINE register; the OpenHCL SNP and TDX VPs do not implement it. MSHV keeps reporting no TSC-deadline support, since it does not reliably deliver TSC-deadline events to direct-boot guests. Add `Apic::advance_timer`, which advances a one-shot or periodic LAPIC timer by a downtime at a given timer frequency and queues the timer interrupt if the timer expired, and `Apic::advance_tsc_deadline`, which queues the interrupt of an armed TSC deadline that an advanced TSC passed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: ef640993-0003-4e49-b1ed-1b8b4776d0ef
Restoring a snapshot, possibly on another host, requires the guest TSC to keep its saved frequency and to account for the time that the VM spent stopped. Partitions cannot report or request a TSC frequency, though, and processors cannot move their TSC forward. Add defaulted `Partition` hooks that return the effective guest TSC and local APIC timer frequencies, request a TSC frequency, and advance any backend-specific guest clock by a snapshot downtime, and a defaulted `Processor::advance_tsc`, which advances the stopped VTL0 TSC by a number of guest cycles through the VP state access. The defaults report no frequency and ignore frequency requests, so that backends opt in individually. Add `tsc_frequency_cpuid_leaves`, which builds a CPUID leaf 0x15 that exposes an exact TSC frequency, including frequencies that do not fit the 32-bit crystal frequency, without overflowing the frequency computation of the Linux kernel. Detect KVM's paravirtual clock from the hypervisor CPUID leaves, and record it in the partition capabilities and the CPU compatibility contract. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: ef640993-0003-4e49-b1ed-1b8b4776d0ef
KVM partitions did not report their TSC frequency, so guests had to calibrate it, and a restored VM could not be given the TSC frequency that it was saved with. The kvm-clock MSRs of a guest that uses KVM's paravirtual clock were not saved either, so the guest lost its clock configuration across a save and restore. Expose the exact TSC frequency of the BSP in CPUID leaf 0x15, and implement the partition clock hooks: report the TSC frequency and the 1 GHz frequency of the in-kernel local APIC timer, set the TSC frequency of every vCPU, and advance the kvm clock by a snapshot downtime. Advance the TSC of a vCPU by adjusting its TSC offset, since KVM can discard a sub-second write to IA32_TSC as a synchronization attempt. Save and restore the kvm-clock wall-clock and system-time MSRs with the synthetic MSRs when the partition exposes the kvm clock, and make the synthetic MSR and reference time state present for such partitions even without Hyper-V enlightenments. Add the KVM ioctls that get and set the TSC frequency of a vCPU and its TSC offset device attribute. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: ef640993-0003-4e49-b1ed-1b8b4776d0ef
…frequency MSHV partitions did not report their TSC frequency, so guests had to calibrate it, and nothing verified that a snapshot is restored on a host with the TSC frequency that it was saved with. The TSC of each VP is also restored at a different host time, so the counters of a restored SMP guest drift apart. Expose the exact TSC frequency of the partition in CPUID leaf 0x15, and implement the partition clock hooks: report the TSC and local APIC timer frequencies, reject a requested TSC frequency that differs from that of the partition, since MSHV cannot change it, and, after a snapshot downtime, freeze partition time and align the counters of all VPs to the advanced BSP counter. Hide the TSC-deadline timer from the guest, in both the processor features of the partition and CPUID, since MSHV does not reliably deliver its events to direct-boot guests, and report that ordinary state access does not freeze the partition clock. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: ef640993-0003-4e49-b1ed-1b8b4776d0ef
…erent WHP partitions did not report their TSC frequency, so guests had to calibrate it, and nothing verified that a snapshot is restored on a host with the TSC frequency that it was saved with. The TSC of each VP is also restored at a different host time, so the counters of a restored SMP guest disagree with each other. Expose the exact TSC frequency of the partition in CPUID leaf 0x15, and implement the partition clock hooks: report the TSC frequency and the local APIC timer frequency, reject a requested TSC frequency that differs from that of the partition, and, after a snapshot downtime, align the counters of all VPs of an SMP partition with partition time suspended. To keep TSC reads coherent across the VPs, a restored SMP partition that does not use nested virtualization then derives the TSC of each VP from the partition-wide reference time. While this restored-TSC clock is active, WHP exits on RDTSC and on accesses to IA32_TSC, TSC_ADJUST, and IA32_TSC_DEADLINE, and the VP state accessors read and restore the TSC through the clock. A VP whose guest writes its TSC or changes its TSC_ADJUST leaves the clock and gets the native WHP semantics again, and resetting the partition disables the clock. Read IA32_TSC_DEADLINE before the APIC state when saving a VP, so that the deadline cannot expire between the two reads. Add the WHP properties and registers that this needs, and report a failure to inject a general protection fault from an MSR exit instead of panicking. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: ef640993-0003-4e49-b1ed-1b8b4776d0ef
When a VM is restored after it was stopped for some time, the TSC of each VP and its local APIC timer continue from their saved values, so the guest does not observe the elapsed time and its timers fire late. Add `PartitionUnit::advance_tsc`, which advances the TSC of every stopped VP by the downtime at the guest TSC frequency. It also advances the local APIC timer of each VP at the APIC timer frequency, when the backend reports one, and fires an armed TSC deadline that the advanced TSC passed. Each VP adjusts its TSC through `Processor::advance_tsc`, reprograms its APIC and TSC deadline, and then verifies that its counter advanced by at least the requested amount, since a backend could discard the adjustment. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: ef640993-0003-4e49-b1ed-1b8b4776d0ef
Measuring where snapshot capture and restore spend their time requires timing individual lifecycle phases in several OpenVMM components and processes, and correlating the results, without changing the behavior or the output of ordinary runs. Add a `profile` module to `openvmm_defs`. When the `OPENVMM_STARTUP_PROFILE` environment variable is set to a nonempty value other than `0`, a `ProfileSpan` measures an exclusive phase or a cumulative milestone and writes one machine-readable `OPENVMM_SNAPSHOT_PROFILE_V1` record to stderr, with its duration, the time since process start, the process ID, and optional byte and fault counters. Otherwise spans are inert. OpenVMM initializes the process-relative clock at startup; the lifecycle phases add their spans as they are implemented. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: ef640993-0003-4e49-b1ed-1b8b4776d0ef
…ecoding The snapshot manifest identified its format only by a version number: nothing marked the file as an OpenVMM snapshot, or recorded which saved-state schema `state.bin` holds and how long it is. Reading a snapshot also decoded `manifest.bin` and read `state.bin` without any size limit. Introduce manifest version 3, which adds a format magic, the saved-state schema version and protobuf root type, and the length of `state.bin`, and move the format definitions and their validation into a `format` module. `SnapshotManifest` implements `Default` with the current format identifiers, and the callers that build manifests take the fields that they do not set from it. Manifests of the earlier version 2 format, which records SHA-256 digests of `state.bin` and `memory.bin` instead, are accepted as well; version 1 manifests are no longer accepted. Writing a snapshot records the length of `state.bin` and rejects saved state larger than 256 MiB. Reading a snapshot limits `manifest.bin` to 1 MiB, validates the manifest format before it reads `state.bin`, and requires `state.bin` to have the recorded length. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: ef640993-0003-4e49-b1ed-1b8b4776d0ef
Saving a snapshot created the destination directory, wrote the manifest and the saved state into it, and hard-linked the memory backing file as `memory.bin`. A failure could leave a partial snapshot at the destination, nothing was flushed to stable storage, an existing directory was silently reused, and the hard link made the snapshot an alias of live guest RAM that any later write to the backing file corrupted. The backing file also had to be on the same filesystem as the snapshot. Publish snapshots through a private, uniquely named staging directory next to the destination. `state.bin`, `memory.bin`, and `manifest.bin` are written and flushed there, the staging directory is flushed, and it is then renamed to the destination without replacing an existing path. The rename is the commit point, after which the parent directory is flushed. A failure before the commit removes the staging directory and leaves the destination absent; `SnapshotWriteError` distinguishes it from a committed snapshot whose parent directory could not be flushed. `memory.bin` is now an independent copy of guest RAM, made from an open handle so that replacing the path of the backing file cannot substitute other bytes, and `write_snapshot_from_memory_file` accepts that handle directly. The copy clones the file when the filesystem supports it, and otherwise copies only allocated ranges or nonzero data on Linux and copies densely on Windows. The backing file must have the length that the manifest records. Add the snapshot file-system helpers that this needs: private directories, no-replace renames, directory flushes, regular-file access that rejects Windows reparse points, and sparse-aware copies. Update the Guide. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: ef640993-0003-4e49-b1ed-1b8b4776d0ef
Copying guest RAM into `memory.bin` keeps a snapshot independent of the source VM, but it doubles the storage and the capture time for a VM whose backing file OpenVMM itself creates and owns, and whose source is terminated after capture anyway. Add `write_snapshot_from_owned_memory_file`, which publishes such RAM by hard-linking the exact open backing handle into the staging directory as `memory.bin`, after the state and the manifest are written and the handle is flushed. Linux links the descriptor with `linkat(AT_EMPTY_PATH)` or through `/proc/self/fd`, and Windows uses a handle-relative link; the link is then opened relative to the staging directory and must have the device and inode or file ID and the length of the source. Filesystems that cannot link fall back to the independent copy. Once the link may exist, the staging directory aliases live RAM: a failure removes it and flushes its parent before it reports a rollback-safe error, and a failed cleanup is reported as `SnapshotWriteError::CleanupUncertain`, which requires the source to terminate instead of resuming. Size new memory backing files through `initialize_snapshot_memory_backing_file`, which keeps them sparse except on Windows, where a dense file avoids slow copy-on-write faults, and add `open_memory_backing_file_handle`, which returns the sized backing file itself so that a caller can keep the exact handle for publication. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: ef640993-0003-4e49-b1ed-1b8b4776d0ef
Restoring a snapshot maps `memory.bin` as guest RAM, but a file backing could only be mapped shared, so every guest write would go into the snapshot artifact, and the snapshot could be used only once. Add a file mapping mode to the memory manager. With `FileMappingMode::CopyOnWrite`, an existing backing file is mapped with private copy-on-write views, so the guest reads the contents of the file while its writes stay private to the process and never reach the file. Copy-on-write RAM requires an existing file backing, and it is not offered as a shared-memory backing to other processes. The default mode is shared, as before. Also let the memory manager count guest-memory faults and the bytes populated while resolving them, for diagnosing restore performance, and report the totals through `MemoryManager::fault_counters`. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: ef640993-0003-4e49-b1ed-1b8b4776d0ef
…he directory Restore read `manifest.bin` and `state.bin` by joining each name to the snapshot path, and the caller reopened `memory.bin` the same way. Every open resolved the pathname again, following symbolic links, so renaming or replacing the directory, or one of its files, between two opens could combine artifacts of different generations. Unexpected files in the directory were ignored, and the length of `memory.bin` was checked only by the caller. Add `OpenedSnapshot`, which opens the snapshot directory once and then opens each artifact relative to that directory handle. Linux uses `openat` with `O_NOFOLLOW`; Windows uses handle-relative, read-only opens that share only reads and reject reparse points. `OpenedSnapshot::open` reads the bounded manifest and saved state through the opened handles, requires the directory to contain exactly `manifest.bin`, `state.bin`, and `memory.bin` as regular files, checks the exact artifact lengths, and records the generation of the opened memory file: its identity, length, and, on Linux, its modification and change times. Accessors return the manifest and state, check that the memory generation is unchanged, duplicate the exact memory handle for a later mapping, and report logical and allocated artifact sizes for profiling. `into_parts` moves the open directory and artifact handles into the new `SnapshotRestoreGuards`, so a later caller can keep the generation open for the lifetime of a restored VM. `restore::read_snapshot` now takes the expected guest memory size and validates all three artifacts through an `OpenedSnapshot`, and `read_snapshot_manifest`, `read_snapshot_with_memory`, and `read_snapshot_artifacts_with_memory` cover manifest-only reads and reads that also return the exact memory handle. The top-level `read_snapshot` re-export is removed; its callers use `restore::read_snapshot`. The command-line restore path still reopens `memory.bin` as shared, writable guest RAM, until restores map it privately copy-on-write. Document restore validation and the limits of the local format: it does not authenticate same-length payload changes, and Linux file descriptors do not provide mandatory write exclusion. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: ef640993-0003-4e49-b1ed-1b8b4776d0ef
…ion handles `--restore-snapshot` reopened `memory.bin` by path for reading and writing and mapped it as shared guest RAM. Every guest write after restore went into the snapshot artifact, so a snapshot could be restored only once, and copying the directory first was the only way to reuse it. The restored VM also held no handle to the generation it had validated, so the artifacts could be replaced while the VM ran. Restore now opens the snapshot through an `OpenedSnapshot` before the VM configuration is built, in the new `snapshot_restore` module of `openvmm_entry`. Just before the VM worker launches, the opened generation is validated against the VM configuration, a duplicate of the exact opened memory handle becomes a private copy-on-write memory handle (`file_to_copy_on_write_memory_fd`), and the memory generation is checked again after the mapping handle is created. The worker receives the handle with the new `shared_memory_copy_on_write` flag and maps guest RAM privately copy-on-write, so guest writes stay in the process and never reach `memory.bin`, and one snapshot can be restored any number of times. The open directory and artifact handles move to the worker as the new `snapshot_restore_guards` parameter, and the worker keeps them until it exits, which keeps the generation open for the lifetime of the restored VM; on Windows the read-only handles also deny writes to it. The worker takes its restore inputs from the parameters in the new `dispatch/restore` module, which also takes over the saved-state inventory check. The management and petri launches pass the defaults: a shared mapping and no guards. With `OPENVMM_STARTUP_PROFILE` set, restore records the `restore` phases `artifact_open`, `artifact_prepare`, and `cow_section_create`, and a launch records the `startup` milestones `worker_launch` and `worker_construct` and the worker phases `partition_prototype`, `cow_map_view`, `partition_build`, `gpa_registration`, and `partition_finalize`. Document that restores are private and repeatable, and describe the handles that restore retains. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: ef640993-0003-4e49-b1ed-1b8b4776d0ef
An orchestrator that restores a VM from a snapshot has no reliable way to learn when the restore has finished: OpenVMM reports failures only in its log, and a restored vCPU may already be running before the orchestrator can tell whether devices started. Add `--restore-ready-path <PATH>`, which requires `--restore-snapshot`. Before the VM worker launches, OpenVMM connects to the named endpoint, an existing Unix domain socket on Linux or a `//./pipe/...` named pipe on Windows, and passes the connection to the worker as the new `restore_ready_sink` parameter. After the saved state is applied, the worker holds the restored VPs stopped. When the VM first starts, the worker starts its state units, writes exactly `RESTORE_READY_EVENT_V1` (`OPENVMM_RESTORE_READY_V1\n`) to the endpoint and flushes it, and only then releases the VPs, so the event always precedes guest execution. A failure to connect, write, or flush fails the launch, and a failure to write or flush also stops the started units, leaving the VPs stopped. The endpoint is used once and is never saved in a snapshot. The worker keeps this restore state in `dispatch/restore`, which now sequences the start of a restored VM. With `--paused`, the first successful `resume` in the REPL publishes the event. A resume that fails before then ends the REPL with an error and tears the VM down, because the single-use endpoint may already be consumed and a later resume would start the guest without the event. With `OPENVMM_STARTUP_PROFILE` set, restore also records the `restore` phases `saved_state_restore` and `device_start`, the latter with the guest-memory faults and populated bytes counted while the devices start. The management and petri launches pass no readiness endpoint. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: ef640993-0003-4e49-b1ed-1b8b4776d0ef
A WHP partition registers every guest RAM range with WHP when the range is mapped, and asks WHP to deliver GPA access faults on mapped memory to the VMM so that it can populate pages itself. For a VM restored from a private copy-on-write snapshot mapping, both are wasted work: registering a large memory range up front makes startup scale with guest memory, and every first touch of a page costs a VP exit and a populate call even though WHP can resolve the copy-on-write fault itself. Add two partition-creation options to `ProtoPartitionConfig`. `user_mode_memory_faults` keeps the current behavior when set; when it is clear, WHP no longer requests `GpaAccessFaultExit`, and the partition reports that it cannot resolve memory faults, so the memory backing does not defer commits or protections to them. `lazy_memory_registration` makes the WHP memory mapper register only the first 64 MiB of each mapping when it is mapped. A guest access to a range that is not yet registered reaches the VP as an unmapped-GPA access; the mapper then registers the 2-MiB chunk containing it, and the VP retries the access. The mapper tracks the registered ranges, so an overlay page removed from an unregistered range is unmapped rather than remapped, and unmapping a range unregisters only what was registered. Mapping failures in the mapper are now returned instead of panicking. The VM worker and the TMK VMM keep the current behavior: user-mode fault resolution and eager registration. Backends other than WHP ignore both options. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: ef640993-0003-4e49-b1ed-1b8b4776d0ef
…erialized kicks The virtio transports save and restore only transport state: feature negotiation, queue configuration, and queue progress. A device whose state lives outside its queues, such as buffered console or network data, has nowhere to save it, so such devices cannot support save/restore. Restore also trusted the saved queue layout and progress, and a queue notification signaled the queue event directly from the transport, so a kick that raced with queue startup could be lost; this is why the virtio-blk over virtio-mmio boot test was marked unstable. Add device-private saved state to `VirtioDevice`. After all queues stop, `save_device` returns an optional typed `SavedStateBlob`, which the PCI and MMIO transports save as a new `device_state` field; `restore_device` receives it, and `device_state_validator` returns an immutable validator that the transport keeps after the device moves into its task. The defaults preserve the current behavior: devices that support save/restore have no private state, and other devices cannot be saved. The new `device::saved_state` module holds the validator type and a helper that reads the front restored descriptor without consuming it. Restore now validates the saved state before it changes the transport: exactly two feature banks, queue sizes within each queue's device maximum, enabled rings aligned, inside guest RAM, and not overlapping, available and used progress consistent with the rings, progress present for every enabled queue of a DRIVER_OK device, and private state accepted by the device's validator. The restored private state is then staged, and the device task applies it on the first activation: the start of an active device, DRIVER_OK, a device-configuration access, or a queue kick. A configuration access whose activation fails completes with an error, and the staged state is kept. `start_fallible` waits for the activation and the queue starts, so a failure fails the VM start. A save before activation returns the staged state unchanged, and queues that were never started keep their restored progress. A guest reset of a device that is not DRIVER_OK discards the staged state. Queue notifications now reach the device task as serialized `Kick` commands, ordered with private-state activation. Kicks coalesce until the task dequeues them, and a kick for a queue that has not started is staged and delivered once the queue starts, so no notification is lost. The virtio-blk over virtio-mmio boot test is therefore no longer marked unstable. The device task traces the restore lifecycle under the `virtio_restore` target. `SavedStateBlob` becomes `Clone`, so the staged state can be applied and saved again. The virtio-blk persistence test now shuts the guest down with an enlightened shutdown after its save/restore pulse, avoiding the pipette shutdown delay, which depends on the guest timer state that the test just restored. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: ef640993-0003-4e49-b1ed-1b8b4776d0ef
A snapshot of a running VM first gates host input, so that no new external input reaches a device while its state is captured, and resumes the input if the save transaction fails. `ChangeDeviceState` provides `quiesce_input` and `resume_input` for this, but the virtio transports did not implement them, so a virtio device could not take part. Add `quiesce_input` and `resume_input` to `VirtioDevice`, defaulting to success for devices without host input. The PCI and MMIO transports forward both through the device task as the new `QuiesceInput` and `ResumeInput` commands, so they are ordered with the device's other commands, and return the device's error. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: ef640993-0003-4e49-b1ed-1b8b4776d0ef
…t policies A virtio-console always discarded guest output while its host backend was disconnected, and a console resource could not describe how its host endpoint is attached or reconnected. A host that must not lose console output, or that reattaches endpoints after a restore, has no way to ask for that, and the resolver cannot reject an inconsistent endpoint description. Add `VirtioConsoleDisconnectPolicy` and an optional `VirtioConsoleAttachment` to `VirtioConsoleHandle`. With the `Retain` policy, the console keeps pending guest transmit descriptors while the backend is disconnected and forwards them after it reconnects; `Discard` keeps the current behavior and remains the default of `VirtioConsoleDevice::new`. The new `new_with_policy` constructor selects the policy, and the worker now holds its backend in a `ConsoleWorkerMode`. The attachment records a stable ID, the backend kind, the attachment mode, the endpoint identity, and a reconnect policy with its requirement and timeout. The resolver rejects an attachment whose identifiers are empty or oversized, whose mode and reconnect policy conflict, or whose requirement and timeout do not match the policy: a recreated listener or a discarding attachment is optional without a timeout, a reconnecting client is required with a bounded timeout, and an inherited attachment is required without a timeout. Disconnects are also handled consistently: after a read error or end-of-file, the worker completes the backend's disconnect before it waits for a new connection, and a backend write that accepts no bytes is treated as a disconnect instead of a panic. All existing console resources use `Discard` without an attachment. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: ef640993-0003-4e49-b1ed-1b8b4776d0ef
A virtio-console could not be saved: host input already read from the backend lived only in a local buffer of the receive loop, the worker state was removed once both queues stopped, and the progress within the current transmit descriptor was lost with it. Restoring a console would therefore drop accepted input or replay output. Implement device-private save and restore for the direct console mode. The worker state is now created with the device and kept while the queues are stopped, so a stopped device still holds its private state. Host input is staged in the worker state before it is copied to a guest receive buffer, so input accepted from the backend survives worker restarts and snapshots; the receive loop also reads the backend before the guest posts a buffer, completing zero-length buffers as it goes. The saved state, schema version 1, records the console size, the bytes already written from the current transmit descriptor, the staged input, and the disconnect policy. Restore validates the encoded size, the schema version, the policy, and the bounds of every field before decoding further, and requires a restored transmit offset to lie within the front descriptor of the enabled transmit queue, both before the queues restart and when the transmit queue starts. Reset clears the private state. The console also implements `quiesce_input` and `resume_input`: while input is gated, the worker reads no host input, and output continues. `SavedStateBlob::encoded_len` returns the encoded payload length without decoding it, which the console uses to bound its private state. Document that virtio-console now supports save and restore through the transport's device-private payload. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: ef640993-0003-4e49-b1ed-1b8b4776d0ef
To save a network device, its frontend must stop exchanging packets with the endpoint at a well-defined point: every receive and transmit already accepted by the endpoint must complete, and the endpoint must not keep any guest buffer. The `Queue` trait had no operation for this, so a frontend could only drop its queues, losing work in flight. Add `Queue::quiesce`, which finishes the work that a queue already accepted without accepting new work. On success, all resulting receive and transmit completions are available through `rx_poll` and `tx_poll`, the queue retains no buffer IDs or guest-memory segments, and the returned `QueueQuiesceResult` reports how many completions of each kind remain to be polled. `Queue::resume` resumes host receive admission after a rolled-back quiesce. Queues do not support quiesce by default, and resume succeeds by default. `BufferAccess` now requires `Send`, so a quiesce future can hold the frontend's buffer pool across await points; every existing implementation already is. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: ef640993-0003-4e49-b1ed-1b8b4776d0ef
A TAP queue wrote guest packets synchronously and completed every packet, including the ones that the TAP interface dropped because its buffer was full or the interface was down. A packet dropped under backpressure was therefore lost, and the queue could not implement the endpoint quiesce contract, because it could not report which transmits were still in flight. Each `tx_avail` call now writes at most one guest packet. When the TAP interface applies backpressure, the queue keeps the packet, and with it the ownership of its descriptors, until `poll_ready` writes it through the new `AsyncWrite` implementation of `PolledTap`; `tx_poll` then completes it. A partial write or a write error is reported as a fatal transmit error instead of being ignored. Implement `Queue::quiesce` for TAP queues: a quiesced queue drops its free receive buffers, stops reading from the interface and admitting receive buffers, finishes writing its pending packet, and reports the receive and transmit completions that it still holds. `Queue::resume` resumes receiving after a rolled-back quiesce. The TAP backpressure test now checks that every packet is completed exactly once instead of being dropped. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: ef640993-0003-4e49-b1ed-1b8b4776d0ef
A Consomme queue could not stop exchanging packets at a well-defined point for a save, because it always polled the Consomme stack and its control channels and kept accepting receive buffers. Implement `Queue::quiesce` for Consomme queues. A quiesced queue processes the frames that the guest already queued for transmit, drops its posted receive buffers, ignores new ones, and stops polling the Consomme stack and the control channels; `poll_ready` then only reports the completions still pending. Quiesce reports the receive and transmit completions that the queue still holds, and `Queue::resume` restores normal polling after a rolled-back quiesce. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: ef640993-0003-4e49-b1ed-1b8b4776d0ef
Consomme derives the guest and gateway addresses from a CIDR, so a host cannot give a guest an exact, preconfigured IPv4 identity: a specific guest address and prefix, and a gateway whose address and MAC address stay fixed across snapshots and restores. Consomme also always treats traffic to the gateway as ordinary egress, with no way to map it onto host loopback, which a guest with a static identity uses to reach host services. Add `ConsommeParams::set_static_ipv4`, which sets an exact guest address, prefix length, gateway address, and gateway MAC address. The identity must be consistent: a prefix of 1 to 30 bits, a guest address that is neither the network nor the broadcast address, and a gateway at the first host address of the network, distinct from the guest. A static identity stops advertising a routable IPv6 prefix and enables the new `map_gateway_to_host_loopback` parameter, which translates guest TCP and UDP flows addressed to the gateway onto host loopback. With the mapping disabled, such flows are dropped, except TCP to the single port named by the new `gateway_loopback_proxy_port` parameter. Expose the identity to resources: `ConsommeHandle` gains an optional `StaticIpv4Config`, which is mutually exclusive with `cidr`, and optional overrides of host-local access, the gateway mapping, and the gateway proxy port. The resolver rejects a handle with both a CIDR and a static identity, or with an inconsistent identity. Existing handles set none of the new fields. This changes one behavior of every Consomme endpoint: a guest TCP or UDP flow addressed to the gateway, other than the gateway's own DNS and DHCP services, used to be sent to the gateway's address on the host network, where no gateway exists. Such flows are now dropped unless the gateway mapping or the proxy port allows them. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: ef640993-0003-4e49-b1ed-1b8b4776d0ef
virtio-net claimed to support save and restore unconditionally, but the transport saved only queue progress: packets that the endpoint had accepted, or that the device held for the guest, were lost, and nothing tied a restored guest's view of its network identity to the endpoint. Saving a VM with virtio-net therefore silently corrupted the guest's network state. Save and restore are now opt-in. `NicBuilder::save_restore` binds a device to an exact static IPv4 identity and a fixed effective feature contract, and `VirtioNetHandle` gains `save_restore`, `static_ipv4`, and `effective_features` fields for resources; a resource that opts in must supply both. Other devices report that they do not support save and restore. When the transport stops an active queue pair of such a device, the device quiesces the endpoint queues, drains every completion the endpoint still holds so that no descriptor stays in flight, and captures the progress of both queues, which the transport then collects queue by queue. The device-private saved state, schema version 1, records the identity, the guest MAC address, the link state, the effective and negotiated feature banks, the endpoint offloads, the queue-pair lifecycle, and an endpoint generation that each restore advances. Restore validates it against the destination device: the identity must match and use MAC addresses derived from its IPv4 addresses, the effective features and endpoint offloads must be unchanged, the negotiated features must stay within the contract and match the saved banks, and the device must have a single queue pair whose restored queues own no descriptors. Restarting a stopped pair requires the same negotiated features. The device also implements `quiesce_input` and `resume_input`: while input is gated, the endpoint queues finish the work they already accepted but admit no new host receive traffic, and the gate is re-established whenever the queues restart. Workers are now started only by the coordinator, after it establishes the gate. Petri records the PCIe ports of virtio-net devices added without a save and restore contract and skips its startup save/restore test while any of them is configured. Every existing virtio-net resource leaves `save_restore` unset. Document that virtio-net supports save and restore only with a static identity and feature contract. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: ef640993-0003-4e49-b1ed-1b8b4776d0ef
A FUSE session negotiates its protocol version, feature flags, and limits with the guest during FUSE_INIT and keeps them only in memory. A file system device built on it could not be saved and restored, because a restored session would be uninitialized while the guest driver continued with its negotiated protocol. Add `SessionState`, a serializable snapshot of whether a session is initialized and of its negotiated values, which contains no file handle, task, or native resource. `Session::save_state` captures it, and `Session::restore_state` restores it after validating it: an uninitialized state must carry no negotiated values, and an initialized one must use a supported protocol version (7.27 through the implementation's minor version), accept only feature flags that the guest advertised, keep its extended-init flags consistent, and have nonzero write and time-granularity limits. The types are re-exported from the crate root. Two related fixes make the saved values well-defined. FUSE_INIT now records the negotiated minor version as the lower of the guest's and the implementation's, instead of the guest's value, so a session never claims a protocol newer than it implements. Destroying a session also resets its negotiated information, so a later session starts from defaults. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: ef640993-0003-4e49-b1ed-1b8b4776d0ef
A virtio-fs inode records a single host path, which it uses for every later operation on the node. That path went stale in several cases. A hard link gave the inode a second name that nothing recorded, so unlinking the original name left the inode pointing at a missing path even though it was still reachable through the link. Renaming a directory left every open inode below it pointing at the old location. The dedup map, which lets repeated lookups of a host file return one FUSE node, only evicted the renamed or removed path, without updating the inodes that referred to it. Each inode now keeps the set of host paths through which the guest knows it. Lookups and hard links add a path. After an unlink, the paths at or below the removed name are dropped from every inode of the volume, and an inode whose primary path was removed switches to one of its remaining aliases. After a rename, the paths at or below the old name are rewritten under the new name, after first dropping the aliases of any overwritten destination. The dedup map is then rebuilt from the surviving inodes, and a path-keyed inode with no remaining alias is left out of it, so a later create at a vacated path still gets a fresh node. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: ef640993-0003-4e49-b1ed-1b8b4776d0ef
Linux 5.0 added `acpi_rsdp_addr` to the zero page: a 64-bit field at offset 0x70 of `boot_params` through which a boot loader hands the kernel the physical address of the ACPI RSDP, so the kernel need not scan legacy memory for it. `boot_params` still declared those bytes as part of the 16-byte `_pad3`, so a loader could not set the field. Split `_pad3` into `acpi_rsdp_addr` and the remaining 8 bytes of padding, matching the kernel's layout. The structure's size and the offsets of all other fields are unchanged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: ef640993-0003-4e49-b1ed-1b8b4776d0ef
`load_static_elf` allocates a page-aligned chunk buffer for every call and uses it to import the file-backed parts of each segment. A loader that imports several images in a row, such as a kernel followed by an initial RAM disk, therefore allocates and frees that buffer once per image. Split the import into a crate-private `load_static_elf_with_buffer`, which takes the chunk buffer from its caller, and keep `load_static_elf` as a wrapper that allocates a buffer for a single call. Behavior is unchanged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: ef640993-0003-4e49-b1ed-1b8b4776d0ef
A guest booted directly into a Linux kernel, without firmware and without ACPI, learns its processors and interrupt routing from Intel MultiProcessor Specification 1.4 tables. The loader had no way to build them. Add the `mptable` module, whose `build` function returns an MP floating pointer, placed at guest address 0, and a configuration table, placed at 0x400, for a given set of APIC IDs and level-triggered ISA IRQs. The configuration table describes each processor, with the first as the bootstrap processor, one ISA bus, one I/O APIC at 0xfec00000, and the routing of the 15 ISA IRQs other than the cascade, with IRQ 0 on pin 2; IRQs listed as level-triggered are described as active-high, and the others use bus defaults. APIC IDs must be contiguous from zero and fit the 8-bit MP processor entry, IRQs must be in the ISA range, and both structures carry valid checksums. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: ef640993-0003-4e49-b1ed-1b8b4776d0ef
Reject copy-on-write scratch capture where restore cannot materialize it, document direct-claimed hard-link requirements, and cover clone isolation plus exact claimed-file identity. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
A cold boot with --cpu-profile auto, the default, failed with E_PROFILE_HOST_UNKNOWN on every CPU that no pinned profile serves, such as Cascade Lake, Sapphire Rapids, Granite Rapids, Tiger Lake, Meteor Lake, and AMD's Ryzen CPUs, including the Granite Rapids runners of Copilot cloud agent sessions. --cpu-profile host booted such Intel and AMD hosts, but only after the user opted in following the failure (microsoft/nvx#434). select_on_host now falls back there. Where no pinned profile's generation covers an Intel or AMD host CPU (the new cpu_profile::pinned_profile_serves), auto derives the host profile as --cpu-profile host does: the VM worker fingerprints the backend, derives <vendor>.host.v1 under the pinned profiles' policy, and checks its generation. A CPU of another vendor, such as Hygon's, still fails with E_PROFILE_HOST_UNKNOWN, and so does a CPU that pinned profiles of more than one generation serve, a catalog defect, whose error still names --cpu-profile host. auto does not fall back where the backend does not support the CPU's pinned profile (E_PROFILE_UNSUPPORTED): that reveals a profile or hypervisor problem on a served generation, and the error still names --cpu-profile host. Explicit profile IDs, --cpu-profile host, restores, which take the snapshot's profile, and --cpu-fingerprint, whose check never falls back, are unchanged. The fallback is not silent. Each cold boot that takes it logs one warning at the default log level, from the VM worker, so wherever auto is selected, the management endpoint included. It starts with the stable marker NVX-CPU-PROFILE-FALLBACK:, names the host CPU (vendor, display family/model/stepping, and brand string), the host profile and its digest, and the limits of a host profile, including what that cold boot's fingerprint and derivation took, and asks the user to attach the CPU's fingerprint (openvmm --hypervisor <backend> --cpu-fingerprint fingerprint.json) to NVX's CPU profile request form, through a link that prefills the form's title and CPU fields. An explicit --cpu-profile host warns with the same profile, digest, and limits, and with the same request where no pinned profile serves the CPU. Where the fallback cannot boot, its errors keep their codes and say that auto fell back: a failed fingerprint or derivation, such as a backend that lacks a CPU feature that the time ABI requires (E_PROFILE_UNSUPPORTED), through a typed context, and a CPU profile failure of the partition (E_PROFILE_UNSUPPORTED, E_PROFILE_TIME_BITS, E_CPU_SURFACE, or E_CPU_UNLISTED), through the cold boot's hint, which no longer suggests --cpu-profile host there and does not repeat the fallback's context. The host-cpu-unknown time ABI test hook makes auto treat the host CPU as one that no pinned profile serves, so that CI's runners, which pinned profiles serve, exercise the fallback. Like every hook, it sets TEST_HOOKS in the restore packet and the time samples. Unit tests cover the selection on Intel, AMD, and Hygon CPUs with and without the hook, the fallback and its warning, the explicit host profile's warning with and without the request, the request link's encoding, a fallback that cannot boot (a failed fingerprint, a derivation without RDTSCP, and a later E_PROFILE_UNSUPPORTED), the restore of a fallback's snapshot under any hooks, the catalog predicate, and the hook's parsing. The Guide documents the fallback and its warning in the CLI reference and the snapshots page. On a Core i9-12900H with WHP, which intel.alderlake.v1 serves, the hook made a cold boot fall back to intel.host.v1 with the warning; its guest passed the time ABI checks, and its snapshot restored on intel.host.v1 with auto. The package-scoped clippy, documentation, and unit-test gates (cpu_profile, virt, openvmm_core, openvmm_entry, and openvmm_helpers) and xtask fmt pass. Part of microsoft/nvx#434. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Allow exactly one read-only distro block with the explicit nvx_overlay_upper=ramfs kernel token, and reject any other nvx_overlay_upper token, including a bare or repeated one. Keep existing scratch layouts unchanged and reject snapshot or restore for the scratchless profile. Document the layout and its restrictions in the Guide. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
esaurez
force-pushed
the
esaurez/microvm-image-slots
branch
from
October 8, 2026 17:30
5ffe5c2 to
41a595e
Compare
There was a problem hiding this comment.
🟡 Changes recommended
The new capability is exposed in debug logs, and host-control framing has reliability and validation defects.
5 open findings
What changed in this PR
Adds four bind-once, read-only image slots to microVM ABI 2, including fixed virtio topology, authenticated host control, and snapshot/restore support.
Changes:
- Adds empty/bind-once virtio-blk image-slot devices and configuration notifications.
- Extends CLI, management RPC, restore packets, validation, and snapshot contracts.
- Documents image-slot topology and the host-control protocol.
| File | Description |
|---|---|
vm/devices/virtio/virtio/src/transport/task.rs |
Polls device configuration changes. |
vm/devices/virtio/virtio/src/transport/core.rs |
Signals configuration-change interrupts. |
vm/devices/virtio/virtio/src/device.rs |
Adds configuration-change polling API. |
vm/devices/virtio/virtio_resources/src/lib.rs |
Defines image-slot resources and RPCs. |
vm/devices/virtio/virtio_resources/Cargo.toml |
Adds error dependency. |
vm/devices/virtio/virtio_blk/src/resolver.rs |
Resolves image-slot devices. |
vm/devices/virtio/virtio_blk/src/lib.rs |
Implements empty and bind-once media. |
vm/devices/virtio/virtio_blk/src/integration_tests.rs |
Tests empty-slot I/O errors. |
vm/devices/virtio/virtio_blk/Cargo.toml |
Adds image-slot dependencies. |
vm/devices/chipset/src/microvm.rs |
Updates restore-packet tests. |
vm/devices/chipset_resources/src/microvm_time.rs |
Adds image-slot restore targets. |
openvmm/openvmm_ttrpc_vmservice/src/vmservice.proto |
Exposes image-slot launch fields. |
openvmm/openvmm_helpers/src/snapshot/microvm.rs |
Extends snapshot contracts and validation. |
openvmm/openvmm_entry/src/vm_controller/microvm.rs |
Excludes binding during capture. |
openvmm/openvmm_entry/src/vm_controller.rs |
Runs the host-control service. |
openvmm/openvmm_entry/src/ttrpc/mod.rs |
Passes the driver during finalization. |
openvmm/openvmm_entry/src/ttrpc/microvm.rs |
Handles management-RPC slot configuration. |
openvmm/openvmm_entry/src/storage_builder/microvm.rs |
Builds fixed image-slot devices. |
openvmm/openvmm_entry/src/storage_builder.rs |
Stores image-slot configuration. |
openvmm/openvmm_entry/src/serial_io/microvm.rs |
Binds secure host-control endpoints. |
openvmm/openvmm_entry/src/microvm/restore.rs |
Carries and validates restore targets. |
openvmm/openvmm_entry/src/microvm/network.rs |
Updates discovery call sites. |
openvmm/openvmm_entry/src/microvm/mod.rs |
Adds host-control resources. |
openvmm/openvmm_entry/src/microvm/launch.rs |
Propagates image-slot configuration. |
openvmm/openvmm_entry/src/microvm/host_control.rs |
Implements host-control protocol v1. |
openvmm/openvmm_entry/src/microvm/filesystem.rs |
Updates discovery call sites. |
openvmm/openvmm_entry/src/microvm/console.rs |
Shares control authentication. |
openvmm/openvmm_entry/src/microvm/config.rs |
Configures slots and host control. |
openvmm/openvmm_entry/src/cli_args/microvm.rs |
Adds and validates CLI options. |
openvmm/openvmm_defs/src/microvm.rs |
Defines topology and machine validation. |
openvmm/openvmm_core/src/worker/dispatch/microvm.rs |
Assigns slot MMIO and IRQ resources. |
Guide/src/user_guide/openvmm/snapshots.md |
Documents snapshot semantics. |
Guide/src/SUMMARY.md |
Links the host-control page. |
Guide/src/reference/openvmm/management/host_control_protocol.md |
Documents protocol v1. |
Guide/src/reference/openvmm/management/grpc.md |
Documents RPC restore activation. |
Guide/src/reference/openvmm/management/cli.md |
Documents CLI and topology. |
Cargo.lock |
Records dependency updates. |
🧠 Review effort: Balanced
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
microVM L3/L4 egress rules took two forms: IPv4[/PREFIX], which matches every IPv4 protocol, and IPv4[/PREFIX]:tcp|udp:PORT, which matches one TCP or UDP destination port. A rule could not select one protocol on every port, nor ICMP at all, so policies such as "allow TCP to 192.0.2.0/24 on any port" or "deny ICMP to 192.0.2.7" could not be expressed (microsoft/nvx#277). EgressRule now also parses IPv4[/PREFIX]:tcp, IPv4[/PREFIX]:udp, and IPv4[/PREFIX]:icmp. A TCP or UDP rule without a port matches every destination port of its protocol. An ICMP rule matches every ICMP message and takes no port: IPv4[/PREFIX]:icmp:PORT fails with the new ParseEgressRuleError::IcmpPort. EgressTransport gains Icmp, and EgressRule::port is zero for every rule without a port. "any" is not a native protocol: a caller expresses any protocol on one port as one TCP rule and one UDP rule. A rule matches a packet whose destination its prefix contains and, unless the rule is address-only, whose IPv4 protocol is the rule's and, for a port-specific rule, whose TCP or UDP destination port is the rule's. Deny rules still take precedence over allow rules, and the default action still decides the traffic that matches neither list. Every IPv4 fragment carries its protocol, but later fragments carry no transport header. Rules without a port therefore classify fragments by their protocol, and only a port-specific rule still makes a policy reject every fragment, as any TCP or UDP rule did before. Policies of address-only and port-specific rules behave as before, and ICMP and other protocols match only address-only rules unless an ICMP rule selects them. allows_gateway_dns counts a protocol-wide TCP or UDP rule for the gateway as it counts a rule for port 53, and does not count an ICMP rule. ARP for a destination stays denied only by an address-only deny rule, because a protocol-wide deny rule leaves the destination's other protocols reachable. The canonical encoding gives ICMP the transport byte 3 and port-free TCP and UDP rules the port 0, so the snapshot policy digest tells every new rule apart, while the bytes of existing policies, and their encoding version, are unchanged. openvmm_entry's --network-egress-allow and --network-egress-deny document the new forms. The CLI's L3/L4 rule test was nested inside another test function and never ran; it now runs, beside a new test of the protocol-wide forms and of the forms that are rejected before VM resources are opened. Unit tests cover the parser's new forms and errors, each protocol selector as an allow rule and as a deny rule that overrides an allow rule, a protocol-wide rule composed with a port rule in both directions, fragments under port-free and port-specific policies, the canonical encoding of each form, and the gateway DNS and ARP decisions. The Guide documents the forms in the CLI reference and on the Consomme backend page. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
microVM L3/L4 egress rules could select one TCP or UDP destination port,
or every port of a protocol, but not a range of ports. A policy that
admits or blocks a span of ports, such as MXC's network.egress selector
{"protocol": "tcp", "port": 8000, "endPort": 8010}, needed one rule per
port, and each rule list accepts at most 256 rules, so a range of more
than 256 ports could not be expressed at all (microsoft/nvx#278).
EgressRule now also parses IPv4[/PREFIX]:tcp:FIRST-LAST and
IPv4[/PREFIX]:udp:FIRST-LAST, which match every destination port from
FIRST through LAST, both included. Both ports must be in 1 through
65535. A range that ends below its first port fails with the new
ParseEgressRuleError::ReversedPortRange, and one that lacks its first or
last port, or names more than two, with InvalidPortRange. A range of one
port is that port's rule, so FIRST-FIRST and FIRST parse, deduplicate,
and encode alike. The new EgressRule::end_port returns the last port of
a range, the port of a single-port rule, and zero for a rule without a
port.
A port-specific rule matches an unfragmented TCP or UDP packet of its
protocol whose destination port lies in its range. Deny rules still take
precedence over allow rules, so a deny rule for one port inside an
allowed range blocks that port alone. A range restricts ports, so, like
a single-port rule, it makes a policy reject IPv4 fragments, and
allows_gateway_dns counts a TCP or UDP range for the gateway that
contains port 53.
The canonical encoding gives a TCP range the transport byte 4 and a UDP
range the transport byte 5, followed by its first and last ports, so the
snapshot policy digest of every encoding version tells a range apart
from its first port and from every other rule, while the bytes of
existing policies, and their encoding version, are unchanged.
EgressPolicy::bind now also rejects rules that parsing cannot produce,
such as a reversed range or an ICMP rule with a port, with
NonCanonicalRules, so validate catches them after a policy crosses a
boundary.
openvmm_entry's --network-egress-allow and --network-egress-deny
document the range form, and the Guide documents it in the CLI reference
and on the Consomme backend page.
Unit tests cover the parser's range forms and errors; each range's first
and last ports, a port inside it, and the ports next to it, for TCP and
UDP, in allow rules and in deny rules that override an allow rule; a
denied port inside an allowed range; fragments and gateway DNS under
ranges; the canonical encoding of ranges in every encoding version; the
rejection of rule forms that parsing cannot produce; and the CLI's
parsing of ranges, and rejection of malformed ones, before VM resources
are opened.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
esaurez
force-pushed
the
esaurez/microvm-image-slots
branch
from
October 9, 2026 19:09
41a595e to
c713731
Compare
esaurez
added a commit
to microsoft/nvx
that referenced
this pull request
Oct 9, 2026
Pin OpenVMM c7137315 (nanvix/openvmm#116), which adds four optional bind-once read-only image-slot transports to the microVM as part of ABI 2. nanvix/openvmm#116 is stacked on nanvix/openvmm#124 (3796991d), the current dev pin, so this promotion adds exactly its four commits: - c071b869 virtio_blk: add bind-once image slots - 0795babf microvm: add bind-once read-only image slots - 957b1472 microvm: fold image slots into ABI 2 - c7137315 microvm: harden host-control framing, logging, and cleanup A machine that does not declare image slots keeps its device topology, and every machine and snapshot keeps microVM ABI 2, so SOURCE-MANIFEST.json and the benchmark history dimensions are unchanged. - Image slot 0 shares the second virtio-fs slot's MMIO window and shared-status word, and slot 1 its IRQ 13, so a slot-declaring launch attaches at most one live share. - The restore packet carries an image-slot target in header byte 7, flagged by bit 4. nvx-time validates and reports it, never finishes such a restore itself, and nvx-snapshot activates the requested prefix before acknowledging the gated restore. - The shared-status kernel patch maps slots 1 to 3 to their status words. - Managed sandboxes can declare slots at provision time (configuration format 6), boot without a distro layer, and query or bind slots through the authenticated host-control channel. Only a slot-declaring sandbox records a host-control endpoint, and stop and deprovision remove its socket. - The image-slot scenario and boot-cost benchmark hand OpenVMM its capability through a prepared pipe, as the other launchers do (#440). - The Azure Linux initramfs installs nvx-image-slot. - Add an image-slot correctness scenario and a boot-cost benchmark. nvx.py verify passes with this pin. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
esaurez
added a commit
to microsoft/nvx
that referenced
this pull request
Oct 9, 2026
Pin OpenVMM c7137315 (nanvix/openvmm#116), which adds four optional bind-once read-only image-slot transports to the microVM as part of ABI 2. nanvix/openvmm#116 is stacked on nanvix/openvmm#124 (3796991d), the current dev pin, so this promotion adds exactly its four commits: - c071b869 virtio_blk: add bind-once image slots - 0795babf microvm: add bind-once read-only image slots - 957b1472 microvm: fold image slots into ABI 2 - c7137315 microvm: harden host-control framing, logging, and cleanup A machine that does not declare image slots keeps its device topology, and every machine and snapshot keeps microVM ABI 2, so SOURCE-MANIFEST.json and the benchmark history dimensions are unchanged. - Image slot 0 shares the second virtio-fs slot's MMIO window and shared-status word, and slot 1 its IRQ 13, so a slot-declaring launch attaches at most one live share. - The restore packet carries an image-slot target in header byte 7, flagged by bit 4. nvx-time validates and reports it, never finishes such a restore itself, and nvx-snapshot activates the requested prefix before acknowledging the gated restore. - The shared-status kernel patch maps slots 1 to 3 to their status words. - Managed sandboxes can declare slots at provision time (configuration format 6), boot without a distro layer, and query or bind slots through the authenticated host-control channel. Only a slot-declaring sandbox records a host-control endpoint, and stop and deprovision remove its socket. - The image-slot scenario and boot-cost benchmark hand OpenVMM its capability through a prepared pipe, as the other launchers do (#440). - The Azure Linux initramfs installs nvx-image-slot. - Add an image-slot correctness scenario and a boot-cost benchmark. nvx.py verify passes with this pin. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
esaurez
added a commit
to microsoft/nvx
that referenced
this pull request
Oct 9, 2026
Pin OpenVMM c7137315 (nanvix/openvmm#116), which adds four optional bind-once read-only image-slot transports to the microVM as part of ABI 2. nanvix/openvmm#116 is stacked on nanvix/openvmm#124 (3796991d), the current dev pin, so this promotion adds exactly its four commits: - c071b869 virtio_blk: add bind-once image slots - 0795babf microvm: add bind-once read-only image slots - 957b1472 microvm: fold image slots into ABI 2 - c7137315 microvm: harden host-control framing, logging, and cleanup A machine that does not declare image slots keeps its device topology, and every machine and snapshot keeps microVM ABI 2, so SOURCE-MANIFEST.json and the benchmark history dimensions are unchanged. - Image slot 0 shares the second virtio-fs slot's MMIO window and shared-status word, and slot 1 its IRQ 13, so a slot-declaring launch attaches at most one live share. - The restore packet carries an image-slot target in header byte 7, flagged by bit 4. nvx-time validates and reports it, never finishes such a restore itself, and nvx-snapshot activates the requested prefix before acknowledging the gated restore. - The shared-status kernel patch maps slots 1 to 3 to their status words. - Managed sandboxes can declare slots at provision time (configuration format 6), boot without a distro layer, and query or bind slots through the authenticated host-control channel. Only a slot-declaring sandbox records a host-control endpoint, and stop and deprovision remove its socket. - The image-slot scenario and boot-cost benchmark hand OpenVMM its capability through a prepared pipe, as the other launchers do (#440). - The Azure Linux initramfs installs nvx-image-slot. - Add an image-slot correctness scenario and a boot-cost benchmark. nvx.py verify passes with this pin. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
esaurez
added a commit
to microsoft/nvx
that referenced
this pull request
Oct 9, 2026
Pin OpenVMM c7137315 (nanvix/openvmm#116), which adds four optional bind-once read-only image-slot transports to the microVM as part of ABI 2. nanvix/openvmm#116 is stacked on nanvix/openvmm#124 (3796991d), the current dev pin, so this promotion adds exactly its four commits: - c071b869 virtio_blk: add bind-once image slots - 0795babf microvm: add bind-once read-only image slots - 957b1472 microvm: fold image slots into ABI 2 - c7137315 microvm: harden host-control framing, logging, and cleanup A machine that does not declare image slots keeps its device topology, and every machine and snapshot keeps microVM ABI 2, so SOURCE-MANIFEST.json and the benchmark history dimensions are unchanged. - Image slot 0 shares the second virtio-fs slot's MMIO window and shared-status word, and slot 1 its IRQ 13, so a slot-declaring launch attaches at most one live share. - The restore packet carries an image-slot target in header byte 7, flagged by bit 4. nvx-time validates and reports it, never finishes such a restore itself, and nvx-snapshot activates the requested prefix before acknowledging the gated restore. - The shared-status kernel patch maps slots 1 to 3 to their status words. - Managed sandboxes can declare slots at provision time (configuration format 6), boot without a distro layer, and query or bind slots through the authenticated host-control channel. Only a slot-declaring sandbox records a host-control endpoint, and stop and deprovision remove its socket. - The image-slot scenario and boot-cost benchmark hand OpenVMM its capability through a prepared pipe, as the other launchers do (#440). - The Azure Linux initramfs installs nvx-image-slot. - Add an image-slot correctness scenario and a boot-cost benchmark. nvx.py verify passes with this pin. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
A microVM may declare four bind-once read-only virtio-blk image slots, which select microVM ABI 3. A VM without slots keeps ABI 2, its topology, discovery tokens, and snapshot format. Slots image0-image3 use MMIO 0xd0008000-0xd000b000, ISA edge IRQs 1/13/14/15, and shared-status words 0x20-0x2c. The first slot reuses the second virtio-fs slot's window and status word, and the second slot its interrupt, so a VM with image slots attaches at most one filesystem. --microvm-image-slot-boot-count activates a prefix of the slots at cold boot. A restore may activate a longer prefix with --restore-image-slots; the target travels in restore packet version 4 as flag bit 4 and header byte 7, and holds the post-restore gate until the guest acknowledges. The management RPC carries the same target. A host-control endpoint (--microvm-host-control), authenticated with the control-console capability, queries the slots and binds a validated regular file into an empty slot once. A clean close at a frame boundary ends a session normally. Machine validation admits exactly the declared image-slot transports, and the snapshot contract records the slot capacity and the boot prefix. Capture is rejected while a slot is bound. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Keep image slots optional within ABI 2 instead of allocating a separate ABI version. Validate the slot-specific snapshot metadata and discovery independently, including platform-tier snapshots, while preserving the no-slot device layout. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Address review findings in the image-slot host-control service: - Redact the host-control capability when the management RPC logs a CreateVM request. The capability authenticates the endpoint. - Apply the idle timeout to a request frame's payload as well as its length, so an authenticated client that stalls inside a frame cannot hold the single-connection endpoint. - Keep responses within the 64 KiB frame limit. A response that would exceed it, such as an error that echoes a long path, becomes a bounded response_too_large error for the same request. - Reject a management-RPC image_slot_boot_count without image_slots, as the command line already does. - After a committed snapshot, remove the host-control socket with the console sockets and fail the source process if that removal fails, instead of leaving it to the cleanup guard. Drop the unreachable duplicate removal from the error path. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
esaurez
force-pushed
the
esaurez/microvm-image-slots
branch
from
October 9, 2026 20:20
c713731 to
67b641d
Compare
esaurez
added a commit
to microsoft/nvx
that referenced
this pull request
Oct 9, 2026
Pin OpenVMM 67b641d8 (nanvix/openvmm#116), which adds four optional bind-once read-only image-slot transports to the microVM as part of ABI 2. nanvix/openvmm#116 is based on OpenVMM main f4566bab, the current dev pin, so this promotion adds exactly its four commits: - f5c8dab0 virtio_blk: add bind-once image slots - 0a3dc07e microvm: add bind-once read-only image slots - 4eb09487 microvm: fold image slots into ABI 2 - 67b641d8 microvm: harden host-control framing, logging, and cleanup A machine that does not declare image slots keeps its device topology, and every machine and snapshot keeps microVM ABI 2, so SOURCE-MANIFEST.json and the benchmark history dimensions are unchanged. - Image slot 0 shares the second virtio-fs slot's MMIO window and shared-status word, and slot 1 its IRQ 13, so a slot-declaring launch attaches at most one live share. - The restore packet carries an image-slot target in header byte 7, flagged by bit 4. nvx-time validates and reports it, never finishes such a restore itself, and nvx-snapshot activates the requested prefix before acknowledging the gated restore. - The shared-status kernel patch maps slots 1 to 3 to their status words. - Managed sandboxes can declare slots at provision time (configuration format 6), boot without a distro layer, and query or bind slots through the authenticated host-control channel. Only a slot-declaring sandbox records a host-control endpoint, and stop and deprovision remove its socket. - The image-slot scenario and boot-cost benchmark hand OpenVMM its capability through a prepared pipe, as the other launchers do (#440). - The Azure Linux initramfs installs nvx-image-slot. - Add an image-slot correctness scenario and a boot-cost benchmark. nvx.py verify passes with this pin. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
esaurez
added a commit
to microsoft/nvx
that referenced
this pull request
Oct 9, 2026
Pin OpenVMM 67b641d8 (nanvix/openvmm#116), which adds four optional bind-once read-only image-slot transports to the microVM as part of ABI 2. nanvix/openvmm#116 is based on OpenVMM main f4566bab, the current dev pin, so this promotion adds exactly its four commits: - f5c8dab0 virtio_blk: add bind-once image slots - 0a3dc07e microvm: add bind-once read-only image slots - 4eb09487 microvm: fold image slots into ABI 2 - 67b641d8 microvm: harden host-control framing, logging, and cleanup A machine that does not declare image slots keeps its device topology, and every machine and snapshot keeps microVM ABI 2, so SOURCE-MANIFEST.json and the benchmark history dimensions are unchanged. - Image slot 0 shares the second virtio-fs slot's MMIO window and shared-status word, and slot 1 its IRQ 13, so a slot-declaring launch attaches at most one live share. - The restore packet carries an image-slot target in header byte 7, flagged by bit 4. nvx-time validates and reports it, never finishes such a restore itself, and nvx-snapshot activates the requested prefix before acknowledging the gated restore. - The shared-status kernel patch maps slots 1 to 3 to their status words. - Managed sandboxes can declare slots at provision time (configuration format 6), boot without a distro layer, and query or bind slots through the authenticated host-control channel. Only a slot-declaring sandbox records a host-control endpoint, and stop and deprovision remove its socket. - The image-slot scenario and boot-cost benchmark hand OpenVMM its capability through a prepared pipe, as the other launchers do (#440). - The Azure Linux initramfs installs nvx-image-slot. - Add an image-slot correctness scenario and a boot-cost benchmark. nvx.py verify passes with this pin. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
esaurez
added a commit
to microsoft/nvx
that referenced
this pull request
Oct 9, 2026
Pin OpenVMM 67b641d8 (nanvix/openvmm#116), which adds four optional bind-once read-only image-slot transports to the microVM as part of ABI 2. nanvix/openvmm#116 is based on OpenVMM main f4566bab, the current dev pin, so this promotion adds exactly its four commits: - f5c8dab0 virtio_blk: add bind-once image slots - 0a3dc07e microvm: add bind-once read-only image slots - 4eb09487 microvm: fold image slots into ABI 2 - 67b641d8 microvm: harden host-control framing, logging, and cleanup A machine that does not declare image slots keeps its device topology, and every machine and snapshot keeps microVM ABI 2, so SOURCE-MANIFEST.json and the benchmark history dimensions are unchanged. - Image slot 0 shares the second virtio-fs slot's MMIO window and shared-status word, and slot 1 its IRQ 13, so a slot-declaring launch attaches at most one live share. - The restore packet carries an image-slot target in header byte 7, flagged by bit 4. nvx-time validates and reports it, never finishes such a restore itself, and nvx-snapshot activates the requested prefix before acknowledging the gated restore. - The shared-status kernel patch maps slots 1 to 3 to their status words. - Managed sandboxes can declare slots at provision time (configuration format 6), boot without a distro layer, and query or bind slots through the authenticated host-control channel. Only a slot-declaring sandbox records a host-control endpoint, and stop and deprovision remove its socket. - The image-slot scenario and boot-cost benchmark hand OpenVMM its capability through a prepared pipe, as the other launchers do (#440). - The Azure Linux initramfs installs nvx-image-slot. - Add an image-slot correctness scenario, a boot-cost benchmark, and socket-backed host-control client tests. nvx.py verify passes with this pin. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
esaurez
added a commit
to microsoft/nvx
that referenced
this pull request
Oct 9, 2026
Pin OpenVMM 67b641d8 (nanvix/openvmm#116), which adds four optional bind-once read-only image-slot transports to the microVM as part of ABI 2. nanvix/openvmm#116 is based on OpenVMM main f4566bab, the current dev pin, so this promotion adds exactly its four commits: - f5c8dab0 virtio_blk: add bind-once image slots - 0a3dc07e microvm: add bind-once read-only image slots - 4eb09487 microvm: fold image slots into ABI 2 - 67b641d8 microvm: harden host-control framing, logging, and cleanup A machine that does not declare image slots keeps its device topology, and every machine and snapshot keeps microVM ABI 2, so SOURCE-MANIFEST.json and the benchmark history dimensions are unchanged. - Image slot 0 shares the second virtio-fs slot's MMIO window and shared-status word, and slot 1 its IRQ 13, so a slot-declaring launch attaches at most one live share. - The restore packet carries an image-slot target in header byte 7, flagged by bit 4. nvx-time validates and reports it, never finishes such a restore itself, and nvx-snapshot activates the requested prefix before acknowledging the gated restore. - The shared-status kernel patch maps slots 1 to 3 to their status words. - Managed sandboxes can declare slots at provision time (configuration format 6), boot without a distro layer, and query or bind slots through the authenticated host-control channel. Only a slot-declaring sandbox records a host-control endpoint, and stop and deprovision remove its socket. - The image-slot scenario and boot-cost benchmark hand OpenVMM its capability through a prepared pipe, as the other launchers do (#440). - The Azure Linux initramfs installs nvx-image-slot. - Add an image-slot correctness scenario, a boot-cost benchmark, and socket-backed host-control client tests. nvx.py verify passes with this pin. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This was referenced Oct 13, 2026
Author
|
Closing together with microsoft/nvx#406: the image-backed runtime plan in microsoft/nvx#535 leaves multiple image slots out of scope. The branch stays, so existing pins remain fetchable. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds bind-once read-only virtio-blk image slots to the microVM profile. Image slots extend microVM ABI 2 in place, without a separate ABI or backwards-compatibility path because the feature is not deployed. A VM without slots keeps its original device topology, discovery tokens, and snapshot shape.
A VM that declares slots also uses microVM ABI 2, with four permanent slot identities:
image00xd00080000x20image10xd00090000x24image20xd000a0000x28image30xd000b0000x2cimage0reuses the MMIO window and shared-status word of the second virtio-fs share, andimage1reuses its IRQ 13. A VM with image slots therefore attaches at most one virtio-fs share. The CLI, VM configuration, machine validation, and the snapshot contract reject the combination. The discovery token ismicrovm_image_slots=4.Activation (no hotplug)
Bslots (--microvm-image-slot-boot-count,1..4, default 1). Inactive slots are virtio-mmio placeholders (device ID 0).--restore-image-slots N(B <= N <= 4). The target travels in restore packet version 4: flag bit 4 is set and header byte 7, previously reserved zero, holdsN. No portb status bit advertises it. An explicit target always holds the restore gate until the guest verifies the active prefix and acknowledges, through both the CLI and the management RPC. No slot is activated after readiness, and no device is added or removed at runtime.IOERR.Binding
bind_image_slotbinds a validated read-only regular file once. The VMM opens and retains the handle, sets the capacity, and raises a virtio configuration change. Repeating the same identity is idempotent. A different identity on a bound slot, an inactive slot, invalid media, mismatched geometry, and writes are rejected. Identities are 1 to 1024 bytes. There is no unbind, eject, or rebind; media is released when the VM exits.query_image_slotsreports each slot as inactive, empty, or bound with its identity, and is the authority after a host reconnect.Host control
Adds a versioned (v1) host-control protocol over an owner-restricted Unix socket or Windows named pipe. It checks the local peer UID/SID and authenticates with the existing 32-byte control capability. CLI launches read the capability once from stdin; management-RPC launches carry it in the authenticated request, and the management RPC redacts it from its request log. Frames in both directions are limited to 64 KiB: a response that would exceed the limit becomes a
response_too_largeerror. A connection closes after 60 seconds without a request or inside an incomplete frame. A client ends its session by closing the connection after a response: end-of-stream at a frame boundary is a normal close, while a close inside a frame is an error.Snapshots
Slot-enabled ABI-2 snapshots record the capacity and the boot-active count (contract fields 34 and 35), the four stable devices, and empty media state. Without slots, ABI-2 snapshots record zero capacity and boot-active count and have no slot devices. Capture is rejected before the guest is quiesced while any slot is bound; binding and capture share an exclusion gate.
Machine validation
Machine validation admits exactly the four image-slot transports when slots are declared, and raises the device limit by the slot capacity only in that case. Machines without slots keep the original inventory and limit, and standard machines reject an image-slot configuration.
Commits
f5c8dab0virtio_blk: add bind-once image slots0a3dc07emicrovm: add bind-once read-only image slots4eb09487microvm: fold image slots into ABI 267b641d8microvm: harden host-control framing, logging, and cleanupRebase onto
mainThe previous head
41a595e9was based onmain96e2363b. The three commits were first replayed onto #124 so that microsoft/nvx#406 could pin one revision containing both. After #124 and #125 merged, they were replayed ontomainf4566bab. Each commit builds, andrange-diffshows the last move changed no patch.f12dcf5e("Permit RAM-backed microVM overlays without scratch") made sandbox-block validation depend onnvx_overlay_upper=ramfs. VM-configuration validation now applies that rule and then validates the declared image slots; the RAM-overlay test passes the new image-slot argument.67b641d8addresses the review findings on41a595e9:image_slot_boot_countwithoutimage_slotsis rejected, as on the command line.Validation
Current head
67b641d8:cargo nextest run --profile agent -p virtio_blk -p virtio -p virtio_resources -p chipset_resources -p chipset -p openvmm_defs -p openvmm_helpers -p openvmm_entry -p openvmm_ttrpc_vmservice -p net_backend_resourcespassed 583 tests.cargo clippy --all-targets -- -D warnings,cargo fmt --check, andcargo doc --no-depswithRUSTDOCFLAGS=-D warningsare clean for the changed packages.cargo check --testspasses at each commit.cargo testfor the same ten packages passed 621 tests;cargo clippy --all-targets -- -D warningsandcargo fmt --checkare clean.3b885041and874fd540, which pin this head). Host: AMD EPYC 9V74, Linux6.6.148-200.mshv-b1f02da.azl3; static musl OpenVMM fromnvx.py build-openvmm --backend mshvwith Rust 1.95.0 (SHA-256b2c02064…). This host's hypervisor cannot present the built-inamd.genoa.v1profile that--cpu-profile autoselects, so a test-only wrapper passed--cpu-profile hostto cold microVM boots; restores and the CPU-profile fallback scenario ran unchanged.B = 1, rebinds it idempotently, rejects a different identity, and reads it; then it captures an empty-slot snapshot, restores it with four active slots, and binds and reads slots 1, 2, and 3. Every slot IRQ (1, 13, 14, 15) therefore delivers a configuration change.Earlier heads (before the ABI-2 consolidation): Linux/MSHV runs of the NVX image-slot scenario, the default microVM suite, and the image-slot boot benchmark, and microsoft/nvx#406 CI on KVM, MSHV, and WHP, all passed.
Follow-up
Track additional microVM IRQ lines and possible expanded IOAPIC routing in microsoft#123. More IRQ lines alone do not resolve the separate second-filesystem MMIO overlap.