fix: read legacy v9 snapshots in the nested KVM runtime by Hygge-Gezelligheid · Pull Request #27 · kvcache-ai/firecracker

Hygge-Gezelligheid · 2026-10-10T14:08:54Z

Allow the nested-KVM runtime to restore legacy v9 snapshots and continue saving v1000 state.

Changes

Decode legacy v9 snapshots on x86_64 and accept the unchanged v9.0.0 layout on aarch64, retaining size, CRC, architecture and version validation.
Query nested-state support before capture and align snapshot inspection and snapshot-editor behavior.
Preserve AMD SVM GIF in header-only nested state, so snapshot restore retains the guest's interrupt-blocking state.
Package x86_64 and aarch64 as 1.15.1-patch-v2.
Preserve IA32_XSS across x86_64 snapshot restore through upstream backport 490577590dc9ddd59ba8b9f13868cd56d0c3f9c2.

Validation

Runtime compatibility checks: 5 x86 MSR tests, 9 x86 snapshot tests and 8 aarch64 snapshot tests passed; aarch64 unit tests ran under QEMU userspace.
Intel nested-KVM API and raw Firecracker matrices: 9/9 each.
AMD nested-KVM API and raw matrices: 9/9 each. After the GIF fix, the targeted interrupt-preservation regression and the three-state raw matrix passed.
The GIF regression fails with the previous runtime and passes with the fix; the focused unit test also covers unchanged VMX decisions.
Native ARM64 KVM with the released runtime passed template restoration, fork isolation, pause/resume and service restart, with the process pool enabled and disabled.
Real old v9 templates and paused records restored, saved as v1000 and restored again. The ARM installation-script upgrade also passed using staged AgentENV packages.
Earlier companion AgentENV Rust integration tests and Docker Compose E2E passed.
Native ARM64 Linux 6.18 lifecycle and real v9-to-v1000 upgrade tests passed with the process pool enabled and disabled.
Empty-cache public-download, cold-start and pause/resume verification passed on Intel x86_64 and native ARM64 with the published patch-v2 assets.

Compatibility and release

Compatibility is old-to-new: v9 input files remain unchanged, and old runtimes cannot read v1000. Nested-KVM hardware validation covers Intel and AMD; the ARM hardware tests cover ordinary KVM.

Release: v1.15.1-patch-v2. Both runtime archives were built natively on their respective architectures from 95c9c9653ae3ed1517a312fd39a22bedc6586e01, matching the release tag. The PR and release source trees differ only in src/cpu-template-helper/build.rs; native builds select the same architecture-specific mock kernel. Each contains firecracker and cpu-template-helper; prebuilt Linux 6.18 kernels are available for both architectures. Installations retaining historical pre-fault patch-v2 binaries must replace that same-version cache.

Companion integration and nested test tools: AgentENV #358.

License Acceptance

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

…hots Decode the historical x86_64 v9 layout into current VM state and accept the unchanged ARM v9 layout while retaining current-format writes. Share validated loading with recovery and inspection tools, and query KVM nested-state support before capture. Cover conversion and rejection with regression tests and an authentic v9 fixture. Signed-off-by: Yicheng Lin <linyc25@mails.tsinghua.edu.cn>

huang-jl · 2026-10-28T03:32:19Z

-    if cfg!(target_arch = "x86_64") {
+    let target_arch = std::env::var("CARGO_CFG_TARGET_ARCH").expect("Missing target architecture");
+    if target_arch == "x86_64" {


Why do we need this change? Was there an issue with the previous cfg!?

I added this to support the x86_64-to-aarch64 cross-compilation workflow I used for preparing the release. It makes build.rs select the mock kernel by the target architecture rather than the build host.

Since we can build the ARM artifacts natively, this build-workflow adaptation doesn’t need to be included in this PR. I’ll drop this change and use native ARM builds.

huang-jl · 2026-10-28T07:27:25Z

Two additional points:

I think we should saved two additional MSRs: MSR_VM_HSAVE_PA and MSR_VM_CR.
In the current Firecracker implementation, msrs_to_save looks potentially incorrect for a restored microVM. You can check it against a microVM booted from scratch vs a microVM restored from snapshot?

Note that MSR_VM_HSAVE_PA and MSR_VM_CR are not listed in KVM_GET_MSR_INDEX_LIST,so add it to whitelist like MSR_IA32_XSS may not work.

Align package, API and changelog versions at 1.15.1-patch-v2. Signed-off-by: Yicheng Lin <linyc25@mails.tsinghua.edu.cn>

IA32_XSS controls which supervisor XSTATE components (CET_U, CET_S, PT, HDC, etc.) are valid for XSAVES/XRSTORS. Without it the guest kernel's context-switch path triggers #GP when it encounters supervisor components in a process's XSAVE buffer that don't match the (now zeroed) XSS register: Bad FPU state detected at restore_fpregs_from_fpstate, \ reinitializing FPU registers. KVM unconditionally includes IA32_XSS in its msrs_to_save_base[] list (it's not CPUID-gated), but Firecracker's serializable MSR allowlist previously excluded it. Add it so snapshots round-trip any guest relying on supervisor XSTATE features. Signed-off-by: Jack Thomson <jackabt@amazon.com> (cherry picked from commit 4905775) Backport the MSR allowlist change and document it in the release changelog. Validated against the AgentENV lifecycle failure on AMD EPYC 9V45: unmodified runtime fails; the XSS candidate passes all 21 active Rust integration tests with the same Linux 6.18 kernel. Signed-off-by: Yicheng Lin <linyc25@mails.tsinghua.edu.cn>

Keep SVM nested-state headers so snapshot restore preserves both GIF values. Cover SVM headers and the existing VMX cases with a focused regression test.

Hygge-Gezelligheid · 2026-10-28T08:56:30Z

Two additional points:

I think we should saved two additional MSRs: MSR_VM_HSAVE_PA and MSR_VM_CR.

In the current Firecracker implementation, msrs_to_save looks potentially incorrect for a restored microVM. You can check it against a microVM booted from scratch vs a microVM restored from snapshot?

Note that MSR_VM_HSAVE_PA and MSR_VM_CR are not listed in KVM_GET_MSR_INDEX_LIST,so add it to whitelist like MSR_IA32_XSS may not work.

I confirm two issues with the experiment:

SVM MSRs: On this host, MSR_VM_HSAVE_PA is returned by KVM_GET_MSR_INDEX_LIST and is already preserved correctly. MSR_VM_CR is absent from the list: setting it to 1 in the guest and restoring a snapshot changes it to 0. Adding it only to the filtering whitelist would therefore not be sufficient.
Repeated snapshot/restore: The first snapshot reads 188 MSR entries, while subsequent snapshots read only 31. A non-default MC0_CTL value survives the first restore but is lost after saving and restoring again. The restore path restores the register values but does not reconstruct msrs_to_save, including entries added during cold-boot configuration.
I plan to:

Explicitly include the required SVM MSRs for supported AMD guests, ensuring both VM_HSAVE_PA and VM_CR are covered without duplicate entries.
Rebuild the ongoing save list during restore by incorporating the MSR indices from the snapshot while retaining the current runtime’s required entries. This should address MC0_CTL and other omitted entries without special-casing each register.

huang-jl · 2026-10-28T09:21:50Z

I plan to:

Explicitly include the required SVM MSRs for supported AMD guests, ensuring both VM_HSAVE_PA and VM_CR are covered without duplicate entries.

Rebuild the ongoing save list during restore by incorporating the MSR indices from the snapshot while retaining the current runtime’s required entries. This should address MC0_CTL and other omitted entries without special-casing each register.

LGTM.

huang-jl

Looks good overall, though I do not reviewed the details closely.

Rebuild the ongoing MSR save set from captured state and guest CPUID, retaining baseline indices and deduplicating additions without reordering. Select VM_CR and HSAVE_PA when the guest exposes SVM, since KVM does not fully enumerate the SVM state registers. Include VIRT_SPEC_CTRL in the serialization whitelist to retain the guest SSBD setting. Add focused whitelist, CPUID-selection and repeated-restore regressions.

Hygge-Gezelligheid mentioned this pull request Oct 10, 2026

feat: integrate nested KVM and support legacy v9 snapshot recovery kvcache-ai/AgentENV#358

Merged

4 tasks done

Hygge-Gezelligheid changed the title ~~Add the Linux 6.18 nested KVM guest profile and real recovery tools~~ feat: add Linux 6.18 nested KVM profile and recovery tools Oct 10, 2026

Hygge-Gezelligheid marked this pull request as ready for review October 12, 2026 06:09

Hygge-Gezelligheid marked this pull request as draft October 13, 2026 07:54

Hygge-Gezelligheid changed the title ~~feat: add Linux 6.18 nested KVM profile and recovery tools~~ fix: read legacy v9 snapshots in the nested KVM runtime Oct 14, 2026

Hygge-Gezelligheid force-pushed the agent/nestedvirt-profile-tools-20261010 branch from 0fd7672 to 721f057 Compare October 15, 2026 07:42

Hygge-Gezelligheid marked this pull request as ready for review October 15, 2026 08:11

Hygge-Gezelligheid mentioned this pull request Oct 21, 2026

review: patch-v2 nested KVM and legacy v9 compatibility Hygge-Gezelligheid/firecracker#1

Draft

Hygge-Gezelligheid force-pushed the agent/nestedvirt-profile-tools-20261010 branch from 721f057 to 1ba4038 Compare October 21, 2026 16:20

yingdi-shan requested a review from huang-jl October 23, 2026 01:59

Hygge-Gezelligheid force-pushed the agent/nestedvirt-profile-tools-20261010 branch from 95c9c96 to 6ccf130 Compare October 26, 2026 04:36

Hygge-Gezelligheid mentioned this pull request Oct 28, 2026

feat: add dirty memory clear API #30

Open

huang-jl reviewed Oct 28, 2026

View reviewed changes

Hygge-Gezelligheid and others added 3 commits October 28, 2026 15:28

build: prepare patch-v2 for x86_64 and aarch64

b57e38b

Align package, API and changelog versions at 1.15.1-patch-v2. Signed-off-by: Yicheng Lin <linyc25@mails.tsinghua.edu.cn>

fix(x86_64): preserve AMD GIF in header-only nested state

7088964

Keep SVM nested-state headers so snapshot restore preserves both GIF values. Cover SVM headers and the existing VMX cases with a focused regression test.

Hygge-Gezelligheid force-pushed the agent/nestedvirt-profile-tools-20261010 branch from 6ccf130 to 7088964 Compare October 28, 2026 07:29

Hygge-Gezelligheid force-pushed the agent/nestedvirt-profile-tools-20261010 branch from fa75825 to d6c517a Compare October 28, 2026 11:24

huang-jl approved these changes Oct 28, 2026

View reviewed changes

Hygge-Gezelligheid force-pushed the agent/nestedvirt-profile-tools-20261010 branch from d6c517a to 9b08a50 Compare October 28, 2026 14:31

Navigation Menu

Repository navigation

fix: read legacy v9 snapshots in the nested KVM runtime - #27

fix: read legacy v9 snapshots in the nested KVM runtime#27
Hygge-Gezelligheid wants to merge 5 commits into
kvcache-ai:v1.15.1-patch-nestedvirtfrom
Hygge-Gezelligheid:agent/nestedvirt-profile-tools-20261010

Hygge-Gezelligheid commented Oct 10, 2026 •

edited

Loading

Uh oh!

huang-jl Oct 28, 2026

Uh oh!

Hygge-Gezelligheid Oct 28, 2026

Uh oh!

huang-jl commented Oct 28, 2026 •

edited

Loading

Uh oh!

Hygge-Gezelligheid commented Oct 28, 2026

Uh oh!

huang-jl commented Oct 28, 2026

Uh oh!

huang-jl left a comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

Hygge-Gezelligheid commented Oct 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Validation

Compatibility and release

License Acceptance

Uh oh!

huang-jl Oct 28, 2026

Choose a reason for hiding this comment

Uh oh!

Hygge-Gezelligheid Oct 28, 2026

Choose a reason for hiding this comment

Uh oh!

huang-jl commented Oct 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Hygge-Gezelligheid commented Oct 28, 2026

Uh oh!

huang-jl commented Oct 28, 2026

Uh oh!

huang-jl left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Hygge-Gezelligheid commented Oct 10, 2026 •

edited

Loading

huang-jl commented Oct 28, 2026 •

edited

Loading