From 29395bdf45672a311f1fb4580677f526bd9607cf Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Sun, 11 Oct 2026 14:48:26 -0300 Subject: [PATCH] openvmm: pin the snapshot VM clock cut d43dd87d5 (nanvix/openvmm#128, rebased onto main at 70dde4c65) keeps VM time and the RTC in step with the guest TSC across a microVM snapshot. Capture quiesces the state units, which stops VM time and the RTC, before it takes the time ABI capture anchor, while the guest TSC keeps running. Restore advanced VM time and the RTC by the anchor-to-restore downtime D alone, so they resumed behind the TSC by the interval from that cut to the anchor. VM time's keeper now reads the host monotonic clock that the capture anchor samples at the instant VM time stops and saves the reading with VM time's state, and capture records the interval Q from that cut to the anchor's host monotonic time in the time contract's new optional field 12, vm_time_cut_to_anchor_ns, so that a host wall-clock step cannot enter it. Restore advances VM time and the RTC by D + Q, while the TSC, the LAPIC timers, and the restore packet keep D, and it rejects a recorded Q or a D + Q above 30 days with E_DOWNTIME_EXCESSIVE. Recording Q is best effort: capture logs a warning and omits it when the saved stop sample is missing or Q would be negative or above 30 days, and a snapshot without it, including every earlier one, restores with Q = 0 and a warning. Manifests stay at version 6. The time ABI specification follows: its notation defines Q, and Downtime semantics, capture step 4, the SnapshotTimeContract table, restore steps 2, 5, and 15, E_DOWNTIME_EXCESSIVE, and the Guide appendix describe it. Snapshot and restore's restore sequence advances VM time, the RTC, and the PIT by the downtime plus Q. The upstream roadmap records the new pin, retains the partition halt-state fix at position 309, extends the branch DAG to position 311, and indexes the clock-cut commits at positions 310 and 311. The original reviewed head b435a486b was rebased and amended after review of this pin: Q uses the host monotonic clock rather than UTC, and the primary keeper reads it where it computes the stop time rather than around the asynchronous stop. This preserves the fixes for host wall-clock steps and scheduling delays, including the regression test. Rebase the validated source head 6d6e9c1e8 onto the latest OpenVMM main 70dde4c65 without conflicts. Both commits match in git range-diff and the complete diffs against their bases share stable patch ID 8b563430cc7079186a2019aac1c321ccc5ff8ea2. The new pin retains nanvix/openvmm#129's partition halt-state fix, now part of both default branches, and fast-forwards the current NVX pin by exactly two commits. Rebase this single NVX commit onto dev at d206d618f, preserving the upstream changes and resolving the gitlink and roadmap conflicts with the combined revision. On the rebased source, cargo check, clippy, rustdoc, all 500 unit tests, and xtask fmt pass for the five clock-cut packages and vmm_core. NVX verification and the affected CLI, snapshot, and timing tests pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/snapshot-and-restore.md | 5 +-- doc/design/time-abi.md | 54 ++++++++++++++++++++++-------- doc/openvmm-upstream-roadmap.md | 7 ++-- openvmm | 2 +- 4 files changed, 48 insertions(+), 20 deletions(-) diff --git a/doc/design/snapshot-and-restore.md b/doc/design/snapshot-and-restore.md index 2dff7b11c..00f34dffb 100644 --- a/doc/design/snapshot-and-restore.md +++ b/doc/design/snapshot-and-restore.md @@ -319,8 +319,9 @@ Restore proceeds in the opposite direction from capture: partition's effective CPUID against the snapshot's record; 7. restore VM time, chipset and virtio state, partition state, and vCPU state, then set every VP's TSC at one host instant with read-back, advance the - LAPIC timers, VM time, RTC, and PIT by the downtime, and seal the restore - packet's time fields; + LAPIC timers by the downtime and VM time, the RTC, and the PIT by the + downtime plus the interval by which VM time stopped before the capture + anchor, and seal the restore packet's time fields; 8. finish reconnecting host resources; 9. start the state units, with external input gated first when tier policy, processor activation, or nonempty RAM expansion requires guest repair, and diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index d30bb0180..44fd491f1 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -19,6 +19,7 @@ Notation: | `C` | VP capacity of the machine (1, 2, 4, or 8) | | `D` | Snapshot downtime in nanoseconds | | `T_c` | Guest TSC of VP 0 at the capture anchor | +| `Q` | Host monotonic time in nanoseconds from the VM-time cut, where VM time and the RTC stop at quiesce, to the capture anchor | | `g` | Restore generation counter of a VM process | "Reject" means fail with the named error code and never fall back to another @@ -911,6 +912,20 @@ path, a UTC delta that differs from `D` by more than 1 s is logged as a host wall-clock step and does not change `D`. The packet reports the source, so the guest and tests can tell the two paths apart. +VM time, the clock of OpenVMM's emulated devices, and the RTC stop with the +state units at quiesce, before the capture anchor. VM time's keeper reads the +host monotonic clock that the anchor samples at the instant VM time stops and +saves the reading with VM time's state, and capture records `Q`, the host +monotonic time from that VM-time cut to the capture anchor, so a host +wall-clock step cannot enter it. Restore advances VM time and the RTC by +`D + Q`, so that they resume in step with the guest TSC, which kept running +through `Q`; the TSC, the LAPIC timers, and the restore packet still use `D`. +`D + Q` must be at most 30 days (`E_DOWNTIME_EXCESSIVE`). `Q` is best effort: +capture omits it, with a warning, when the saved VM-time state lacks its stop +sample or `Q` would be negative or above 30 days, and the snapshot stays +usable. A snapshot without `Q`, including every snapshot that predates it, +restores with `Q = 0`, and OpenVMM logs a warning. + The restore anchor is the instant of the synchronized TSC set, and the guest TSC runs from that instant on every backend: KVM never stops it, and MSHV and WHP resume partition time right after a successful read-back, inside the @@ -958,7 +973,8 @@ Before worker construction: is 1, `tsc_tolerance_ppm` is 250, `F_s` is plausible, `L_s` is a backend constant, identities are 16 bytes, `capture_generation < 2^32 - 1`, the profile digest is 32 bytes, and the effective CPUID record is whole, - well-formed entries (`E_MANIFEST_TIME`). Validation computes no digest. + well-formed entries (`E_MANIFEST_TIME`). A recorded `Q` above 30 days is + rejected (`E_DOWNTIME_EXCESSIVE`). Validation computes no digest. 3. Require the destination backend to equal `source_hypervisor` (`E_BACKEND_MISMATCH`). 4. Require a pinned profile with the recorded ID and digest, whose canonical @@ -966,8 +982,8 @@ Before worker construction: `E_PROFILE_DIGEST`), both compared with the build's precomputed record, and the host's CPU generation to be in it (`E_CPU_GENERATION`). 5. Preflight the downtime: sample the host clocks, select the source, and - check the bounds (`E_DOWNTIME_*`, `E_HOST_IDENTITY`). The authoritative - `D` is measured again in step 12. + check the bounds of `D` and `D + Q` (`E_DOWNTIME_*`, `E_HOST_IDENTITY`). + The authoritative `D` is measured again in step 12. 6. Validate the rest of the machine contract as today (topology, devices, attachments, command line, blocks). The command line must not contain `tsc_early_khz=` or `lapic_timer_hz=` (`E_CMDLINE_CLOCK_TOKEN`). @@ -1029,11 +1045,14 @@ In the worker, with every VP stopped: state is set. The same pass rejects an armed periodic or TSC-deadline timer (`E_LAPIC_PERIODIC`, `E_LAPIC_TSC_DEADLINE`) before any VP runs, so restore takes no separate pass over the VPs to check them. -15. Advance VM time by `D` and the RTC's UTC by `D` (milliseconds). The PIT - catches up from its saved VM-time cursor; it is idle. Steps 14 and 15 - touch disjoint state, because no time ABI LAPIC reads VM time (see the - LAPIC rate row of [Backend obligations](#backend-obligations)), so - OpenVMM runs them concurrently and waits for both before step 16. +15. Advance VM time by `D + Q` and the RTC's UTC by `D + Q` (milliseconds), + with `Q = 0` for a snapshot that does not record it; a `D + Q` above 30 + days rejects the restore before steps 14 and 15 start + (`E_DOWNTIME_EXCESSIVE`). The PIT catches up from its saved VM-time + cursor; it is idle. Steps 14 and 15 touch disjoint state, because no time + ABI LAPIC reads VM time (see the LAPIC rate row of + [Backend obligations](#backend-obligations)), so OpenVMM runs them + concurrently and waits for both before step 16. 16. Seal the time fields of the restore packet: `D`, its source, the rate deviation, `g`, and the test-hook flag. 17. Start the state units, with host input gated when the restore requires an @@ -1080,14 +1099,18 @@ state units are quiesced, capture: `E_LAPIC_TSC_DEADLINE`, `E_PIT_ACTIVE`); 2. takes the capture anchor: VP 0's TSC paired with a host time sample taken within 100 µs of it, keeping the tightest of at most 64 samples - (`E_TSC_ANCHOR`), plus the host identities (`E_HOST_IDENTITY`); and + (`E_TSC_ANCHOR`), plus the host identities (`E_HOST_IDENTITY`); 3. records the declared rates, the CPU profile, the effective CPUID, and the process's generation counter. The CPU profile record is the pinned profile's precomputed digest and canonical encoding, copied, and the effective CPUID is a packed binary record: capture encodes no document and - computes no digest. + computes no digest; and +4. after saving the state units, records `Q` from VM time's saved stop sample + and the anchor's host monotonic time (see + [Downtime semantics](#downtime-semantics-and-source-selection)). -Each of these failures is a rollback-safe capture failure. +Each failure of steps 1 to 3 is a rollback-safe capture failure. Step 4 never +fails a capture: where it cannot derive `Q`, it logs a warning and omits it. No backend stops guest time for a capture: the guest TSC keeps running while the VPs are stopped. So a capture that rolls back leaves the guest's clocks @@ -1124,6 +1147,7 @@ reused. It adds two required fields: | 9 | `host_id` | `bytes` | 16-byte host identity | | 10 | `host_boot_id` | `bytes` | 16-byte host boot identity | | 11 | `capture_generation` | `u32` | `g` of the captured process | +| 12 | `vm_time_cut_to_anchor_ns` | optional `u64` | `Q`; absent where capture could not derive it and in snapshots that predate it | `SnapshotCpuProfile` (package `openvmm.snapshot`): @@ -1788,7 +1812,7 @@ code. | `E_LAPIC_RATE_MISMATCH` | `L` is not the backend constant, or `L_d` differs from `L_s` | Cold boot, restore | | `E_HOST_IDENTITY` | Host identity, boot identity, or clocks unavailable | Capture, restore | | `E_DOWNTIME_NEGATIVE` | Downtime below zero | Restore | -| `E_DOWNTIME_EXCESSIVE` | Downtime above 30 days | Restore | +| `E_DOWNTIME_EXCESSIVE` | Downtime above 30 days, or a recorded `Q` or `D + Q` above 30 days | Restore | | `E_TSC_ANCHOR` | An anchor is unavailable, or none of at most 64 samples pairs within 100 µs | Capture, restore | | `E_TSC_TARGET_OVERFLOW` | `TSC_target` exceeds 64 bits | Restore | | `E_TSC_SYNC_READBACK` | A VP does not hold the synchronized value | Restore | @@ -2420,10 +2444,12 @@ Migration impact: The OpenVMM Guide documents the user-facing contract in `Guide/src/user_guide/openvmm/snapshots.md`: -- **Overview:** the time contract and CPU profile in microVM manifests, and +- **Overview:** the time contract and CPU profile in microVM manifests, the + contract's interval from the VM-time cut to the capture anchor, and manifest version 6 as the only accepted version. - **Restoring a snapshot:** restore packet v4, the time sample at `0xeb`, the - generation ID, and the synchronized TSC set with its read-back. + generation ID, the synchronized TSC set with its read-back, and the VM-time + and RTC advance by the downtime plus the VM-time cut interval. - **Time and CPU compatibility:** the identity, the CPU profile and `--cpu-profile`, including host profiles and `auto`'s fallback to them with its warning, the 250 ppm rate rule, the exact LAPIC rule, the downtime diff --git a/doc/openvmm-upstream-roadmap.md b/doc/openvmm-upstream-roadmap.md index 64881a453..73c465058 100644 --- a/doc/openvmm-upstream-roadmap.md +++ b/doc/openvmm-upstream-roadmap.md @@ -5,7 +5,7 @@ - OpenVMM fork base: `e1cdbd916d3aacb216309c8e7779b832546759a1`. - Mapped fork tip: `2728f33ea9a44d2d0fb956d9f01768ae2100fb62`, position 166. The graph, PR summary, and commit mapping below cover positions 001-166. -- NVX's pinned OpenVMM revision: `70dde4c65b16de084744b0fa85aeb581f0b5c6cd`, 143 commits past the +- NVX's pinned OpenVMM revision: `d43dd87d5b9c27109c71100accb5b2790416acb4`, 145 commits past the mapped tip on the same first-parent chain. These commits are not yet assigned to PRs; see [Commits after the mapped tip](#commits-after-the-mapped-tip). - Upstream main: `1fd455b19e7c69c72a2d375d0ed1cb35d901c688` (2026-10-01), 18 commits past @@ -199,7 +199,7 @@ e1cdbd916 -> PR18/PR23 split [162] -> PR18 fixup [163] -> PR25 fixup [164] -> PR27 [165-166] -> 2728f33ea (mapped tip) - -> unassigned [167-309] -> 70dde4c65 (NVX pin) + -> unassigned [167-311] -> d43dd87d5 (NVX pin) ``` For upstream review, PR01-PR08 and PR14 can be rebased independently onto current upstream main. @@ -215,7 +215,7 @@ can be submitted independently if preferred. ## Commits after the mapped tip -The NVX pin extends the chain past position 166 without merges. The 143 later commits are not +The NVX pin extends the chain past position 166 without merges. The 145 later commits are not yet classified or assigned to PRs. This table groups them by topic in chain order; ranges are inclusive, as in the [PR summary](#pr-summary). @@ -236,6 +236,7 @@ inclusive, as in the [PR summary](#pr-summary). | 302-305 | `3072d7a91..7bf0ee28b` | 4 | Several host directories on one virtio-fs slot, their snapshot, and their path policy | | 306-308 | `be5297817..2130f24b5` | 3 | Single host files as aggregate virtio-fs children, their saved kinds, and the `file` flag that requests one | | 309 | `70dde4c65` | 1 | Partition halt state across snapshots | +| 310-311 | `9b2073ded..d43dd87d5` | 2 | Restore advances VM time and the RTC by the downtime plus the recorded interval from their stop at capture to the capture anchor | Some of these commits change mapped work rather than add to it. Positions 255-259 remove the legacy microVM clock paths and change files that PR10 and PR17 also change. Position 261 makes diff --git a/openvmm b/openvmm index 70dde4c65..d43dd87d5 160000 --- a/openvmm +++ b/openvmm @@ -1 +1 @@ -Subproject commit 70dde4c65b16de084744b0fa85aeb581f0b5c6cd +Subproject commit d43dd87d5b9c27109c71100accb5b2790416acb4