microvm: A platform snapshot can only be restored with the network policy it was captured with #799

A microVM snapshot records its network policy, and a restore must supply the identical policy. Every snapshot also refuses ingress, explicit host-loopback access and port forwards, both at capture and at restore.

For a platform snapshot captured under a complete deny-all policy, these rules are stricter than needed: the guest had no network access, no published listeners and no forwards when it was captured. Each clone of such a snapshot should be able to bring its own policy. OpenVMM already builds the network endpoint from restore-time options on every restore, so the change is mostly in validation.

Background

An orchestrator restores many clones from one platform snapshot. Each clone may need its own:

• egress rules;
• proxy;
• private-network ingress for the listeners the guest announces;
• host-loopback access;
• explicit port forwards.

Today the orchestrator must capture one platform snapshot per distinct policy, which multiplies snapshots and rules out per-clone allow-lists. No clone of any snapshot can use ingress, host loopback or port forwards.

Current behavior

1. Capture records a digest of the policy. SnapshotMicrovmNetwork::new() stores the policy's mode name, the SHA-256 of its canonical bytes, and whether the policy is active. The canonical bytes cover all of these:

   • the guest identity;
   • the mode and its rules;
   • next hops;
   • host loopback and the proxy;
   • IPv6;
   • ingress, from encoding version 5.

2. Restore requires the same policy. effective_microvm_network():

   • takes the addressing from saved state and refuses --net;
   • refuses every network option for a snapshot without a NIC;
   • requires --network-profile portable;
   • builds the restore-time policy from the command-line options, with Options::microvm_egress_policy();
   • calls validate_microvm_network_policy(), which requires the same mode, digest and active flag:

   anyhow::ensure!(
       saved.egress_policy_mode == policy.mode_name()
           && saved.egress_policy_sha256 == digest.as_slice()
           && saved.egress_policy_required == policy.is_active(),
       "restore-time egress policy does not match the snapshot contract"
   );

   The final machine-contract comparison then checks the whole network record again (L1627-L1628).

3. Every snapshot refuses ingress, host loopback and forwards. Option validation rejects each of these whenever --snapshot-destination or --restore-snapshot is given:

   • --host-loopback allow (L731-L739);
   • --network-ingress allow (L1225-L1229);
   • --host-loopback-forward (L1239-L1247). A forward also requires an explicit --host-loopback allow.

   For example:

   anyhow::ensure!(
       self.microvm.network_ingress != Some(MicrovmNetworkActionCli::Allow)
           || (self.microvm.snapshot_destination.is_none() && self.restore_snapshot.is_none()),
       "microVM snapshots do not support --network-ingress allow, whose published guest listeners are live host attachments"
   );

4. The endpoint itself is not snapshot state.

   • microvm_network_endpoint() builds a new consomme endpoint on every restore, from the restore-time policy, forwards and ingress settings. net_consomme has no save or restore code.
   • Ingress works through guest announcements: UDP datagrams to gateway port 1021, which the endpoint consumes in order to bind host listeners (consomme ingress, announcement format).
   • Whether the endpoint consumes announcements depends on the policy, not on the command line.

   So the enforcement machinery already handles any policy at restore. Only the checks in steps 2 and 3 prevent a different one.

Expected behavior

A platform snapshot is policy-replaceable when its saved policy is a complete deny-all:

• deny-all egress with no rules;
• host loopback denied, and no proxy;
• ingress and host-loopback ingress denied;
• no forwards (capture already refuses them).

OpenVMM decides eligibility from the saved contract only, never from the caller's options.

Restoring a policy-replaceable snapshot:

• Accepts any restore-time policy that the portable profile supports at cold boot: egress modes and rules, a proxy, --network-ingress allow, --host-loopback allow and --host-loopback-forward.
• Keeps the NIC identity from saved state: profile, guest and gateway addresses, prefixes, MACs, features and queues. The restore-time policy is validated against that identity the same way as at cold boot.
• Enforces before running. The whole policy is installed and every explicit forward is bound before any restored vCPU runs. A failure to bind fails the restore and releases what was acquired.
• Publishes announced listeners afterwards, under the new policy, with today's ingress behavior: retries, and never displacing an existing host listener.
• Records the new policy. The restore-time policy becomes the VM's effective policy, so any later capture of the clone records it.

Unchanged:

• Exact-policy rule kept: every other snapshot keeps it, both other tiers and platform snapshots captured with any other policy.
• No added NIC: a snapshot without a NIC can't gain one.
• Capture rules kept: capture still refuses ingress, host loopback and forwards, so a platform snapshot never holds live host attachments.
• Command line unchanged: the saved command line is reused verbatim. The guest learns its new network settings (DNS, proxy, whether to announce listeners) from its own runtime after restore, not from virtnet_* tokens.

Proposed changes

1. Compute eligibility once, from saved state.

   • Rebuild the canonical complete-deny-all policy for the saved network identity.
   • Compare its digest with the saved egress_policy_sha256.
   • Also require the platform tier.

   This needs no manifest change. Storing the policy's posture as explicit fields in a new manifest version is an alternative.

2. Apply the same eligibility check at all three gates:

   • Option validation: allow --network-ingress allow, --host-loopback allow and --host-loopback-forward with --restore-snapshot when the snapshot is eligible. This needs the manifest at validation time, or these checks moving into restore preparation.
   • validate_microvm_network_policy(): for an eligible snapshot, skip the equality check, but still validate the restore-time policy against the saved identity and encoding version.
   • Machine-contract comparison: for an eligible snapshot, compare only the identity fields of microvm_network.

3. Order the setup. Make sure the endpoint and every explicit forward are set up before the vCPUs are released, and that a failed forward bind stops the restore and releases what was acquired.

4. Tests:

   • Positive: a deny-all platform snapshot restores with each policy element: allow rules, a proxy, ingress, host loopback and forwards. No packet passes before the policy is installed.
   • Unchanged refusals:
     • a deny-all snapshot of another tier, and a platform snapshot that wasn't a complete deny-all, still refuse a different policy;
     • a snapshot without a NIC still refuses network options.
   • Failure handling: a busy forward port fails the restore cleanly.
   • After restore:
     • announced listeners are published under the new policy;
     • gateway DNS works under the new policy.

5. Docs: update the snapshot guide, and the CLI help for --network-ingress, --host-loopback and --host-loopback-forward, which today says snapshots don't support them.

Workaround today

Capture one platform snapshot per network policy, so the policy becomes part of the snapshot's selection key, or capture without a NIC. Either way, clones can't use ingress, host loopback or port forwards.