Research: evaluate per-VM vsock NIC adapters for vnet/HeatNet and SpiralCore #4607

Agent pickup triage

Category: design/research. This classifies the next task, not completion or a verified current-source pass.

Next bounded step: Use #4626/#4634 as evidence inputs; settle adapter placement and explicit security decisions before any guest-visible NIC prototype or production implementation.

Handoff evidence: comment · PR. PR states at triage: #4698 merged.

Read the linked handoff and coordinate any active owner before implementation. Preserve the original acceptance criteria below; this triage grants no spending, credentials, publication, host reconfiguration, or landing authority.

Context

mvm runs workload microVMs with no network device. Every guest network surface currently crosses an authenticated FlowMux session on vsock to a per-VM mvm-network-endpoint, which applies the signed admission plan, default-deny policy, host-side name resolution, secret handling, resource limits, and audit. Ingress must be declared before boot; dynamic forwarding is refused.

mvmd is the separate fleet orchestrator and may later automate network enrollment, placement, address allocation, and lifecycle. The packet/frame data plane and per-VM enforcement remain owned by mvm; any mvmd implementation work is tracked in its own repository after this research selects an architecture.

SpiralCore is developing warm-cache networking: a forward-only Kuramoto time header, content/temporal roots and witnesses, page/group Merkle manifests, WarmWire frames, and a warm path that sends a 134 B identity frame or a 4,238 B one-page repair instead of resending the full object. The existing prototype has local, emulated-WAN and real remote-WAN evidence. This issue is the starting point for connecting that work to mvm and mvmd; it is not approval of a new protocol, a new guest NIC, or a second network path.

Status: research umbrella. This issue compares adapter shapes and produces a decision. Nothing here authorizes production implementation, a guest-visible NIC, a second data plane, or a change to MVM's security posture.

Relationship to the mailbox and security programs

These efforts share host-to-host transport and resource-isolation questions, but they own different layers:

The possible reusable seam is below mailbox semantics and above a concrete host transport. Reuse is permitted only where the semantics match; mailbox delivery, FlowMux egress, and packet/frame forwarding retain separate authorization, flow-control, audit, and conformance contracts.

Research goal

Determine whether MVM should add a host-mediated adapter, a guest-visible vsock-backed NIC, neither, or both as separately admitted modes, and produce the evidence and contract needed for a later implementation decision. The research covers:

• where the vsock NIC adapter sits for each mvm instance;
• how mvm's host stack, the host's bridge, and vnet/HeatNet connect;
• how mvmd allocates, identifies and orchestrates the network attachment;
• how the first bounded ingress/egress slice is admitted, audited, refused and torn down.

The first slice is expressed as 1-2 inbound/outbound vsock connections:

1. one connection that bridges the local (or global) mvm network;
2. one connection that interfaces exactly with the host or the host's bridge.

Whether these must remain two separate vsock connections, or can be one multiplexed connection with two logical roles, is a design choice for this issue. It should be decided from the actual needs of the mvm stack, the host stack, and vnet/HeatNet rather than assumed up front.

Prototype both adapter shapes

Two candidate shapes must be prototyped and compared; neither becomes a production default through this issue.

A. Host-mediated adapter (guest remains NIC-less)

The guest keeps its current FlowMux/vsock surface. A host-side adapter, or an extension of the per-VM mvm-network-endpoint, maps the guest's network identity onto vnet/HeatNet and owns the bridge/NAT translation. No guest-routable network device is added. The signed plan, default-deny gate, ingress-before-boot rule and audit chain remain authoritative.

B. Guest-visible vsock-backed NIC

The guest receives a virtual network interface whose frames or packets are carried over vsock to a host endpoint or bridge. This is a material security-posture change: it creates a guest network device, changes the admitted seccomp/network surface, and must be covered by an explicit threat-model and owner decision. It must not become an automatic fallback for the NIC-less path.

The prototypes should record: packet/frame model, MTU, vsock ports, address and identity binding, host socket ownership, failure behavior, teardown, audit events, and loopback/WAN measurements. The design review chooses the production shape.

What must be designed

• Adapter placement: guest agent, per-VM mvm-network-endpoint, host bridge, or VMM device emulation. Name the exact process boundaries and which side owns each socket.
• Connection model: one vsock connection with two logical roles versus two separate connections. Define framing, multiplexing, flow control, backpressure, and error semantics for each option.
• Host bridge/NAT: where the vsock-side identity is translated into the host network or bridge; whether the bridge is per-host, per-tenant or per-network; privilege and lifecycle requirements on Linux and macOS.
• vnet versus HeatNet: both are virtual networks; HeatNet adds master/member settings. Decide whether HeatNet is the better fit for a vsock-carried network, and define the membership, address, and control-plane contract. Neither is assumed into mvm's current production model.
• mvmd ownership: address allocation, tenant/network identity, policy propagation, lifecycle, retries, and observability. mvm should enforce the per-VM boundary; mvmd should not become a second data plane.
• Policy and audit: how SpiralCore identity, WarmWire frames or page/group repair map to the signed plan, default-deny decisions, ingress admission, and payload-free audit records.
• Backend parity: both Firecracker/Linux and HVF/macOS are required. The contract must be backend-neutral even if the first proof uses one backend.

Proposed scaffold

This issue scaffolds the work; detailed implementation slices should become linked follow-ups after the design is accepted.

Workstreams

• D1 - Design record/ADR: adapter placement, connection model, frame/packet format, vsock ports, host bridge/NAT, vnet/HeatNet, mvm/mvmd ownership.
• D2 - Threat model and security decision: explicitly decide whether a guest-visible NIC is allowed; document the change to the no-guest-NIC posture or keep the adapter host-mediated.
• D3 - Host-mediated prototype: one guest, one host endpoint, one vnet/HeatNet bridge; prove 1 ingress and 1 egress vsock connection with denial, audit and teardown.
• D4 - Guest-visible prototype: same proof for the virtual NIC shape, including seccomp/network-tier implications.
• D5 - SpiralCore contract: WarmWire/identity/page-group repair semantics over vsock; handshake, versioning, limits, retransmission and fallback rules.
• D6 - mvmd contract: network identity, allocation, lifecycle, policy distribution and cleanup; cross-link the private mvmd issue once available.
• D7 - Cross-backend test plan: Firecracker/Linux and HVF/macOS, allowed and denied routes, host bridge failure, teardown and restart.
• D8 - Design review and production choice: compare prototypes, choose one shape, and open bounded implementation issues.

First proof (after design decisions)

• One backend, one host bridge/NAT path, one guest.
• One admitted ingress route and one admitted egress route.
• The two vsock connection roles exercised, or one multiplexed connection with both roles if that is selected.
• Positive forwarding, unlisted-destination refusal, identity binding, audit evidence, and clean teardown.
• mvmd records the network identity and lifecycle; mvm enforces the per-VM plan.
• Both backends are required for acceptance; if the first proof runs on one, the other is a tracked blocker, not a skip.

Acceptance criteria

• Benchmark: characterize 1,000-microVM egress and shared-vsock saturation #4626 identifies the first measured saturation point before an adapter is justified as a congestion remedy.
• Evaluate HeatNet/UOR as mailbox overlay, transport, or virtual network #4634 identifies whether the evaluated vnet/HeatNet/UOR artifact is an overlay, byte transport, TUN/L3 network, or no suitable production dependency.
• A design record/ADR names the adapter placement, process boundaries, protocol, and mvm/mvmd ownership.
• The threat model states whether the first shape creates a guest network device and what that changes.
• The connection model is decided: one vsock connection with two logical roles, or two separate connections.
• vnet/HeatNet membership, addressing, master/member role, and host bridge/NAT behavior are defined.
• The signed plan, default-deny, ingress-before-boot, audit, and no-hidden-fallback rules are preserved or explicitly changed by owner decision.
• A bounded 1-2 route proof exists on the selected backend with positive, negative, audit, failure and teardown checks.
• Firecracker/Linux and HVF/macOS are both covered, or have explicit blockers with owners.
• The selected shape satisfies Program: host-blind execution and zero-ambient-authority workload containment #4621's applicable capability, credential, host-blind, assurance-tier, and no-silent-downgrade gates.
• The decision states explicitly that Program: signed pass-by-value mailboxes for workload, host, and user messaging #4622 mailboxes remain independently usable without a guest NIC, L3 overlay, or mvmd data plane.
• Follow-up implementation issues are linked from this design issue.

Non-goals

• Implementing the full vnet/HeatNet or SpiralCore networking stack in this issue.
• General guest Internet access or a default route.
• Replacing FlowMux with a second, unaudited network implementation.
• Silently changing mvm's no-guest-NIC security posture.
• Publishing private mvmd internals or credentials.
• Committing to a production default before both prototypes are compared.

Open questions

• Is the intended "vsock NIC adapter" a host-mediated endpoint, a guest-visible virtual interface, or both during the prototype?
• Are 1-2 inbound/outbound vsock connections two fixed connections, or one multiplexed connection with two logical roles?
• What exactly does "local (or global) mvm network" mean in this topology, and is membership managed by mvmd, vnet, HeatNet, or the host?
• Should HeatNet's master/member model be the first target because it may map naturally to a vsock-carried network, or should vnet be the baseline?
• Where does the host bridge/NAT sit, and what privileges and lifecycle does it need on Linux versus macOS?
• Which backend should host the first proof: Firecracker/Linux, HVF/macOS, or both in parallel?
• What is the mvmd contract and where should its follow-up issue live?

References

• mvm README: NIC-less workloads, signed execution plans, host-side egress, audit chain.
• mvm networking guide: FlowMux, ingress before boot, no guest NIC, no raw-stack fallback.
• mvm FlowMux networking guide: guest loopback adapters to authenticated vsock port 5253.
• mvm host source: crates/mvm-hostd/src/bin/mvm-network-endpoint.rs and the per-VM network endpoint.
• mvm guest source: crates/mvm-agentd flowmux and vsock bridge modules.
• SpiralCore warm-cache network prototype: spiralcore-base/warm-cache (UOR-R4), with local, emulated-WAN and real remote-WAN results.
• SpiralCore design record: docs/labs/spiralcore-warm-cache-network-2026-10-20/README.md (UOR-R4).

Related follow-ups should cover adapter implementation, mvmd orchestration, HeatNet/vnet membership, the SpiralCore transport contract, and cross-backend conformance separately. This issue is the shared starting point and decision record.