Repository navigation
Snapshot/restore workstream: start sandboxes from captured templates through the shared host and guest libraries #833
Description
Objective. Let a sandbox start by restoring a template instead of booting. A template is a tenant-free VM, captured once after boot and before any tenant input. Each restore becomes a separate sandbox, a clone, with its own image, upper, export, network settings and identity. Both adapters get clones through the shared host and guest libraries that #535 delivered: NVX's own adapter (its CLI and its guest init) and the in-repo test adapter (tests/runtime-adapter). The cold path stays as it is.
Today (dev d04d9fb) and after:
| Area | Today | After |
|---|---|---|
| Starting a sandbox | Every sandbox cold-boots, and its guest assembles the root from boot tokens. | A sandbox can also start as a clone of a template. |
| Snapshot support | nvx-rs run restores a raw OpenVMM snapshot on the terminal, and test scripts drive platform snapshots. nvx::host has no capture, template store or clone restore. |
nvx::host captures templates into a store and restores clones from it, for both adapters. |
| Guest at capture | The shell nvx-snapshot requests captures. The layered init mounts the image, the scratch and the overlay first, so no capture can come before root assembly. |
Template mode captures before root assembly: no image, overlay, workload or application session. |
| Guest after a restore | Shell tools repair time, CPUs, memory and entropy. The guest keeps its captured configuration and binds, and takes the first 32 hex digits of the seed's SHA-256 as its machine ID. | Repair runs in-process, through nvx::guest. Each clone then gets its own configuration in an Activate message, and assembles its root through the same nvx::guest::root operations as a cold boot. |
| Host control | The host attaches, then asks for readiness. NVXC v1 has no Activate. | The host connects after OpenVMM's restore-ready event, sends Activate on a restricted session, checks that Ready reports the activation applied, with its transaction ID, then sends MAPS and Start. |
| Image | A disk's size is its file length, and a restore needs a disk of exactly the captured length. | The clone's image replaces the template's placeholder distro. With zero fill, OpenVMM presents the placeholder's size and reads zeros past a shorter image. |
| Upper | A platform capture needs an unmounted scratch disk, and a RAM upper can't be captured. | Every template has a dormant scratch device, and each clone picks tmpfs or the scratch at activation. A scratch clone brings its own scratch file; with zero fill, a tmpfs clone brings none. |
| CPUs and memory | Restore can grow memory, but a platform snapshot keeps its boot CPU count. | Memory targets per clone, and CPU targets per clone with #801. |
| Network | A restore needs the captured policy, and snapshots refuse ingress, host loopback and forwards. | A per-clone policy on a deny-all template, with #536, including ingress, host loopback and forwards. |
| Exports | A restore keeps the binds that its boot made. A dormant filesystem slot accepts one directory at restore. | Zero or one export per clone, on the dormant slot: one directory at first, then an aggregate of directories and files with #540. MAPS binds paths inside it after Ready. |
| Hashing | A platform capture hashes each read-only disk by default, then discards the hash. | Captures pass OpenVMM's generation identity flags, so capture and restore hash no large content. |
Design.
- Template (nvx-rs: lower, capture and publish platform templates #841): a VM captured after boot and before tenant input. It has a read-only placeholder
distro, whose size is every clone's image capacity; a dormant scratch device; a dormant filesystem slot; no NIC, or one fixed NIC; CPU capacity C, with B CPUs online at boot; boot memory M0 and memory capacity Cmem; and a command line of OpenVMM's platform-allowlisted tokens only. The guest has nothing mounted from the image, the scratch or the filesystem tag, and no workload or application session. Its control session is attached, with no host connected. - Key and store (nvx-rs: the template compatibility key, the template store and leases #839): one canonical, versioned compatibility key, projected from small metadata: the bundle and contract versions, CPU and time, resources, the command line, disk geometry and devices. It never comes from hashes of content. Generations publish into a store in two commits, and a durable lease ties each clone to its generation until the clone's verified exit.
- Clone inputs (nvx-rs: restore clones from a template through Activate, Ready and Start #844, nvx-rs: exports, MAPS and network templates for clones #846): an authorized EROFS or GPT image of at most the capacity; tmpfs or the scratch; zero or one export, with 1 to 4,096 mappings: one directory, or an aggregate of up to 256 directories and files once microvm: at a platform restore, attach an aggregate export to the dormant filesystem slot #540 lands; network settings; memory targets, and CPU targets with openvmm: let platform snapshots activate vCPUs at restore #801; a hostname; a machine-ID mode; optionally, a workload UID and GID; fresh endpoints and a fresh capability.
- Restore, in order: select and lease; plan; OpenVMM restores with the clone's disks, export, targets and policy before any vCPU runs; the guest repairs (nvx-rs: repair a restored clone in the guest, with an owned seed and one acknowledgment #842), and its acknowledgment opens the host-input gate; the host connects after the restore-ready event; the restricted session takes one Activate (nvx-rs: the restricted control state and Activate after a restore, in the guest library and the test adapter's framing #843), a canonical record of at most 1,024 bytes (nvx-rs: a canonical activation record for platform clones #838); the guest activates its root (nvx-rs: activate a clone's root for GPT images and either upper, with its export, network settings and workload view #845); Ready reports that the activation applied, with the transaction ID; MAPS; Start.
- Failures: a failure at any step tears down the VM, with no cold fallback. Only template faults quarantine a generation.
- Rules: no large content is hashed on capture or restore. No device hotplug: every device that a clone uses is already in the template. One image per clone, in
distro.
Stages. Stages 1 to 3 need nothing of NVX's own binaries, so they start and finish with the test adapter. Stage 4 is the NVX library wiring workstream (#816).
| Stage | Issue | Content | Needs |
|---|---|---|---|
| 1 | #834 | The libraries on today's OpenVMM, through the test adapter, under interim rules | The PRs of #822 and #805 that #834 lists, and #860's PR 1, before its VM work |
| 2 | #835 | The OpenVMM changes: zero fill (#856), CPU targets for platform snapshots (#796), a new network policy at restore (#536) and an aggregate export at restore (#540) | — |
| 3 | #836 | Their features in the libraries and the test adapter. Each removes an interim rule. | Stage 1, and each item's OpenVMM change in NVX's pin |
| 4 | #816 | The NVX library wiring workstream: NVX's binaries and tools move onto the shared libraries | That workstream's own order |
| 5 | #837 | NVX's adapter, with every stage 3 feature from the start, then the acceptance | Stage 1; stage 3, unless an OpenVMM change is late (#837's fallback); #817, #818, #820, #821 and #624 |
Dependencies on the NVX library wiring workstream (#816). That workstream keeps this work; the snapshot/restore workstream depends on it explicitly:
| Issue in #816 | Needed by | For |
|---|---|---|
| #822, with #805 | #843, #845, #841, #844, #846, #847 | The test adapter end to end, in stacked PRs that #834 lists: the adapter plan and checked handoff, executions in the VM, Ready, MAPS and Start in order, the export mount, and network policy |
| #860 | #845 | A layered root's resolver, and runtime binds that a teardown can remove (its PR 1) |
| #817 | #852 | NVX's Rust layered init, which gets template mode |
| #818 | #853 | The CLI's image formats and protected paths |
| #820 | #853 | The guest build ID, in the compatibility key |
| #821 | #851 | NVXC v2 with a validated Ready, which embeds #838's ReadyActivation and reserves the Activate request's type |
| #624 | #855 | The acceptance harness |
In the other direction, #821's Ready embeds #838's ReadyActivation, so #838 comes first.
Dependencies on OpenVMM. Four OpenVMM changes, described in #835. Each lets a stage 3 item remove one of stage 1's interim rules:
| Change | Tracking | Needed by | Until it lands |
|---|---|---|---|
Zero fill: a platform restore presents the template's distro capacity and reads zeros past a shorter image; a fresh scratch declared absent reads zeros and fails writes |
#856 | #848 | Images of exactly the placeholder's length; a sparse scratch file for each tmpfs clone |
| CPU targets for platform snapshots, and the authentication timer during a gated restore | #796, fixed by PR #801, which is stacked on #797 | #849 | One template per CPU count; the host connects after the restore-ready event |
| A new network policy at restore, for deny-all templates | #536 | #850 | One template per network policy, with no ingress, host loopback or forwards |
| An aggregate export on the dormant filesystem slot at restore | #540 | #857 | One directory export per clone: an export plan with several children, or a file child, is refused before launch |
Children and order. Each stage's issue lists its PR issues. Stage 1's issues each split their work into stacked PRs, which #834 orders.
| Issue | Stage | Scope | Needs |
|---|---|---|---|
| #838 | 1 | The activation record and Ready's activation | — |
| #839 | 1 | The compatibility key, the template store and leases | — |
| #840 | 1 | Capture preparation, and template mode in the test adapter's guest | — |
| #841 | 1 | Template lowering, capture and publication | #839, #840; #822 |
| #842 | 1 | Repair of a restored clone in the guest | #840 |
| #843 | 1 | The restricted control state, Activate, and Ready's activation and transaction | #838, #842; #822 |
| #844 | 1 | Clone restore through Activate, Ready and Start | #841, #843; #822 |
| #845 | 1 | Root activation: every image format and upper, the export, DNS and the proxy, and the workload view | #844; #822, #860 |
| #846 | 1 | Exports, MAPS and network templates | #845; #822 |
| #847 | 1 | The test adapter's template and clone operations | #846; #805 |
| #856 | 2 | Zero fill: zeros past a shorter image, and an absent scratch | — |
| #536 | 2 | A new network policy at restore; #796 is referenced above | — |
| #540 | 2 | An aggregate export on the dormant filesystem slot at restore | — |
| #848 | 3 | Shorter images, and tmpfs clones without a scratch | Stage 1; #856 |
| #849 | 3 | Per-clone CPU targets | Stage 1; #801 |
| #850 | 3 | Per-clone network policies | Stage 1; #536 |
| #857 | 3 | Aggregate exports for clones | Stage 1; #540 |
| #851 | 5 | Activate, and the restore transaction in Ready, in NVXC v2 | Stage 1; #821 |
| #852 | 5 | Template mode, repair and activation in NVX's init | #851, stage 3 or #837's fallback; #817 |
| #853 | 5 | The CLI's template and clone commands | #852; #818, #820 |
| #854 | 5 | NVX's clone scenarios and documentation | #853 |
| #855 | 5 | Acceptance for both adapters | #854, every stage 3 item; #624 |
Not in scope:
- performance measurements and budgets, and delivering the activation record without RPC (nvx-rs: replace activation RPC delivery with an uncaptured VMM configuration region #544): the performance workstream, to be filed;
- a final RAM handoff, no-replace publication and ReFS: deferred; this workstream uses today's publication, on Linux and NTFS;
- receipts for opened attachments (OpenVMM prerequisite: verify and receipt opened host attachments #548) and pinned file children: later hardening;
- generation identity as OpenVMM's default (microVM: identify snapshot blocks only by storage generation; stop hashing block artifacts #557): an optional cleanup;
- workload-tier snapshots, capture of a populated upper, extra image devices, and snapshots without a tier (Track network restore of non-sandbox snapshots:
nvx runcan't restore a snapshot that has a NIC #800); - more than one image per clone, such as one per container: a later image-slot workstream, which first brings bind-once image slots into NVX's OpenVMM pin. Its templates would boot without an image and keep their slots empty across capture, and its clones would bind slots after Ready, as a cold sandbox would. The record and the key are versioned, so that work adds to them. microVM: remove the runtime and custom sandbox block roles #556 removes the unused
runtimeandcustomroles independently; - device hotplug and capability discovery.
Acceptance. #855's: both adapters run platform clones end to end through the shared code on KVM, MSHV and WHP, for every image format and upper, with bundled serving the release that includes this work.
Related:
- nvx-rs: prove shared host and guest mechanics through two independent adapters #545's restore conformance is carried by nvx-rs: acceptance for platform clones in both adapters #855.
- nvx-rs: prove cross-image platform restore with pre-ready activation RPC #542 and nvx-rs: reuse a platform template across mapped paths and network policies #543, the earlier plan's restore phases, are superseded by stages 1 and 3.
- nvx-rs: check a GPT disk's kernel partitions in the guest #770 isn't needed: a GPT clone rereads its partition table before
nvx::guest::root's existing checks (nvx-rs: activate a clone's root for GPT images and either upper, with its export, network settings and workload view #845). - The guide (doc: describe the managed layered root and platform-clone activation #591) documents each stage with its PRs.
Activity
[-]Platform clones: start sandboxes from captured templates through the shared host and guest libraries[/-][+]Snapshot/restore workstream: start sandboxes from captured templates through the shared host and guest libraries[/+]