You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Guest mailbox: one resident registration-based daemon, event-driven, warm-claim compatible (scale to ~1000 microVMs) #5292
Part of #5121 follow-up. The guest mailbox shipped for the mission demo serves each granted VM from its own host process. That is fine at demo scale and costs nothing for VMs without a mailbox grant, but it does not meet the goal of launching on the order of 1,000 microVMs per host.
What it costs per granted VM today
One extra OS process and at least two threads, about 5 MB resident on Firecracker and 8 MB on HVF measured in live witnesses. At 1,000 granted VMs that is 5 to 8 GB and thousands of threads.
Idle polling: the accept loop wakes every 20 ms and the host re-dials an idle guest every 2 s. About 50,000 wakeups per second and 500 dials per second across 1,000 VMs (about 0.2% CPU per process measured on Linux, roughly 2 cores in total). An interim PR replaces the 20 ms poll with a blocking wait.
A granted launch cannot be warm-claimed (warm_eligible_launch excludes it: a standby parent boots without the mvm.mailbox token and the host process is per launch), so it cold-boots and misses the 300 ms launch budget.
Direction (D1 decided; see ADR-057 amendment in PR #5513)
One resident keyless scheduler per host routes, queues and journals sealed messages only, and schedules wakes. It holds no mailbox key and cannot open or forge message contents. It sees routing, size, timing and queue metadata. A compromised key holder exposes only its own VM's mailbox; this is not protection against a compromised host.
VM-scoped key holders own seal/open and mailbox authorization. Existing per-VM HVF and libkrun supervisors are candidates after checking their guest-input boundary. Firecracker has no separate per-VM supervisor, so it needs a lazy VM-scoped helper or a separately reviewed supervisor; the network endpoint is not a holder. A single-tenant key-holding daemon would hold every VM's keys while parsing guest input.
Event-driven socket/process wakes and a bounded idle exit, not per-VM polling; journal and ownership are reconciled after a crash. Mailbox admission on a warm claim is carried through the fail-closed post-restore handshake. Keep the guest-facing outbox frame unchanged.
Acceptance: demonstrate bounded memory, host processes, threads, idle CPU/wakeups and first-message latency for 1,000 granted VMs on an adequately provisioned host, plus a granted warm claim within the existing budget and unchanged witnessed delivery. PR 0 must first record N=100/600/1,000 baseline; the supplied 64 GiB hosts cannot admit the 600/1,000 configured maxima under the harness safety gate. Firecracker's helper may exist per VM only while serving work, not as a resident idle process.
Architecture decision needed before treating this as the default mailbox density implementation. This issue proposes one host/tenant daemon holding derived keys for many mailboxes; a separate local draft design instead proposes a keyless opaque-notice scheduler with VM-scoped key holders and the existing host-dial payload path. The former reduces resident process/thread cost and may improve warm claims, but a compromised daemon can expose keys for every mailbox it holds; the latter preserves a narrower key-custody boundary but needs proven lazy-helper journal/anchor recovery, first-message latency, and warm-claim compatibility. Recommend measuring the current helper baseline at 100/600/1,000 granted running VMs and explicitly choosing the acceptable compromise boundary before replacing the process model. #4635 remains open for real-VM and durable-message latency evidence. No production notifier or pooled daemon from this draft has been shipped or merged.
#5292 design: one resident mailbox daemon per tenant, registration-based, event-driven, warm-claim compatible
Read-only design pass against origin/main (142ba6b). Every file:line below is from that tree. Nothing was built or run. Numbers marked "estimate" are unmeasured; the baseline measurement is PR 0 below and must land before any acceptance number is treated as a target.
0. Recommendation in ten lines
Replace the per-VM mvm-mailbox process with one resident mvm-mailbox daemon per tenant (same boundary as the host-agent daemon, mvm-core/src/config.rs:691). Zero host processes per granted VM; threads independent of VM count.
VMs register and deregister over a 0700 control socket with a host-signed, domain-separated message (the SignedControl precedent, mvm-contract/src/protocol/broker_control.rs:185-222). The registration carries only per-mailbox derived keys. The daemon holds the host signer public key, never the seed, the encryption root or the epoch secret.
No keyless-parser / key-holder split. The broker needed one because it parses rich guest JSON. The mailbox parses a 4-byte length plus OutboxMessage::decode (32-byte recipient, opaque body) and the plaintext queue lives in the same process regardless, so a split protects almost nothing. Confine the daemon instead (jailer, no core dumps, AF_UNIX only). Detail in B2.
The guest-facing frame path stays byte-identical: host dials guest vsock 5257, one frame per connection, GuestSealer, bounded queue 64, replay protector, epoch policy. Per-VM MailboxCore state is split out of MailboxService so the daemon reuses the code, not a copy.
Event loop: mio (already a mvm-hostd dependency) for control socket, per-VM recv listeners and held guest connections; a deadline heap for timers; a small fixed worker pool (default 4) for blocking dial/exchange work. No thread per VM, no fixed-rate polling.
Idle delivery: phase 1 keeps host redial but with a coalesced schedule; phase 2 makes the guest agent hold the accepted connection and answer when the spool becomes non-empty (inotify), so an idle VM costs zero wakeups on both sides and no new backend channel is needed. A guest-dials doorbell (a new standing channel in four backend drivers) is rejected unless the held-connection spike fails (B6).
Warm claim: standby parents boot mailbox-capable but dormant (mvm.mailbox_standby=1: PID 1 pre-creates the 0700 outbox dir, no listener). The claim registers with the daemon before the fork and delivers a mailbox admission through the PostRestore handshake, exactly like the seccomp tier (ChildIdentity -> handle_post_restore -> PostRestoreAck -> require_admitted_mailbox, fail closed). Then drop the exclusion at crates/mvm-cli/src/commands/pool.rs:1020-1022.
Crash blast radius: a daemon crash drops every registered VM's host-side queue (<= 64 unread messages each) and nothing else. The guest spool (<= 256 files) survives, so no sender-side message is lost; delivery resumes after re-registration, which any seed-holding launcher or mvmctl can redo deterministically because the derived keys are re-derivable from the seed plus the epoch secret file that stays in the VM state dir until stop.
Acceptance is anchored to ADR-046's own launch SLO table (specs/adrs/046-secure-message-fabric.md ~L585-596): warm claim to ready p99 <= 50 ms, fabric-enabled vs disabled cold delta p99 <= 5 ms and warm delta p99 <= 2 ms, first local message p99 <= 10 ms, idle CPU event-driven. Concrete 1,000-VM numbers in B10.
A. Exact inventory of what exists today
A1. Host side, per-VM process
What
Where
Behaviour today
Change under the daemon
Binary
crates/mvm-hostd/src/bin/mvm-mailbox.rs:17-28
panic_hook::install, reads all of stdin into a plain Vec (L21, not zeroized), parse, std::process::exit(serve(config))
Becomes daemon entry: config = tenant, control socket, host signer public key path. No per-VM stdin. raw buffers on the control path become Zeroizing.
Cargo/manifest/release
crates/mvm-hostd/Cargo.toml[[bin]] mvm-mailbox (the comment above it says "secret-substitution moat", a stale copy; fix in passing); crates/mvm-cli/src/host_binaries/manifest.rs:95; .github/workflows/release.yml:360,382,560
Shipped and required as a host binary
Keep the binary name and manifest entry; no release plumbing change.
Replaced by ensure_mailbox_daemon(tenant) + register, modeled on host_agent_spawn.rs:134-200 (flock, pid file, spawn_detached_with_config) and :226-250.
Entry points
spawn_for_launchmailbox_process.rs:193-220; serve_started:163-189 (stops the VM when the grant cannot be served); require_servable/require_wired:129-157 (refuse before boot if backend lacks VmCapabilities::guest_mailbox)
Three launch call sites
Same signatures; body becomes register. require_* unchanged.
Each swaps spawn_for_launch/serve_started for the register call; stage names stay for audit continuity.
Warm-claim path
none.crates/mvm-cli/src/commands/pool.rs:1016-1023: warm_eligible_launch excludes a launch when mailbox_cmdline_token(cfg).is_some(); tests at pool.rs:1974-2079 assert it stays cold
Granted launch always cold boots
Remove the exclusion, flip tests, add the claim path (B7).
Liveness
mailbox_process.rs:68-85serve: loop std::thread::sleep(500 ms) (POLL_INTERVAL L46) over vm_state_dir.exists() and endpoint_needed(launcher_is_alive, vm_is_alive) (vm_lifetime.rs:98,124; process_liveness.rs:157state_dir_has_live_process)
A 500 ms keeper loop in every per-VM process: 2 wakeups/s/VM that #5297 did not remove
Gone. Daemon is detached and outlives launchers by design; VM death is learned from deregistration, route failure plus a coalesced liveness sweep (B5).
Dial loop
mailbox_launch.rs:453-496serve_derived_with (spawns a thread): bind retry L465-477, then serve_guest L483 -> Ok: reset attempt, POST_FRAME_PAUSE 10 ms; Err: dial_delay (L65-71: 100 ms x 300 attempts, then 2 s forever); constants L58-62
One thread per VM, ~0.5 dial/s idle after 30 s
Becomes a per-registration schedule entry in the deadline heap; same constants initially (phase 1), removed by the held connection (phase 2).
Route
mailbox_service.rs:374-392serve_guest -> telemetry_collector/mod.rs:981-991dial_route: PortSocket = UnixStream::connect; Mux = dial_mux (L993-1006) = blocking connect_to_port_once(.., 5s) with CONNECT <port> handshake. HostDialRoute at crates/mvm-core/src/protocol/vm_backend.rs:42-47; FC driver/fc.rs:1314 (Mux), HVF driver/hvf.rs:1030 (PortSocket)
Backend-owned, unchanged
Route is carried in Register verbatim. dial_mux is blocking, so dials run on the worker pool, not the reactor (B6).
Queue/replay/admission
mailbox_service.rs:203-224State {queue: VecDeque, accepted, read, replay: InMemoryReplayProtector, policy} behind one Mutex; admit:531-555 (depth check then open_envelope); serve:447-461, serve_peer:464-488, read_frame:514-529 (cap max_frame_bytes), IO timeout 5 s :43
All inside MailboxService, which also owns an accept thread (bind:296-344, wait_readable:272-281)
Split into MailboxCore (State + serve/serve_peer/admit/take, no thread, no listener) and a thin per-VM wrapper that disappears at cutover.
Sealer
mailbox_service.rs:78-152GuestSealer, with_grant:93-104; bind_grant:191-201 verifies grant authority key == derived authority
Per VM
Unchanged; daemon calls bind_grant at registration, so a registrant cannot bind a grant the keys do not match.
Unchanged for readers. Daemon binds one listener per registration at the same path; reader code and the mailbox.read audit entry do not change (constraint 4).
Teardown
crates/mvm-vmm/src/host/vm_helpers.rs:21-34reap_vm_host_helpers: reap_pid_file(mailbox.pid) (L26) then shred mailbox-epoch.secret (L29-33). Callers: mvm-runtime/.../runner/backend.rs:151 (stop), runner/durability.rs:60, mvm-hostd/src/supervisor/wall_clock.rs:116 (supervisor stop); wall_clock.rs:107-110 also shreds on its own path
Kills the process synchronously (keys gone)
Replace L26 with reap_mailbox_registration(state_dir, vm): read a mailbox.tenant ref file (like HOST_AGENT_TENANT_REF, host_agent_spawn.rs:~482-520), send signed Deregister (1 s timeout, CONTROL_IO_TIMEOUT precedent), then shred. Shred stays unconditional.
Launcher mints, process never sees it (only the derived encryption key)
Unchanged. This file is what makes post-crash re-derivation possible (B5).
A2. Guest side (frozen by constraint 2, except the two admission points)
What
Where
Note
Cmdline token
crates/mvm-vmm/src/host/egress_bridge.rs:50-53mailbox_cmdline_token (mvm.mailbox=1 iff admitted plan has a grant); emitted at cmdline.rs:165; HVF/standing channel at mvm-runtime/.../runner/sockets.rs:81
Plan-derived on a cold boot; a standby parent has no plan, hence no token
Outbox provisioning
mvm-agentd/src/guest_bootstrap.rs:109,618-636provision_mailbox_dir: PID 1, root, before privilege drop, gated on the token; mailbox_outbox.rs:128-157 creates /run/mvm/mailbox and /run/mvm/mailbox/outbox, 0700, chown 901
The dir is created as root because the agent "cannot create it under /run/mvm afterwards". WORKLOAD_UID = 901 = agent uid (guest_mount.rs:17), so the agent owns the dir and can later remove outbox but not /run/mvm/mailbox.
Listener
crates/mvm-agentd/src/bin/mvm-guest-agent/mailbox.rs:24-39init_mailbox (cmdline gated, spawned from mvm-guest-agent.rs:1090 post-activation), accept loop :48-65: one connection at a time, forward_outbox per accept, 5 s socket IO timeout
Single-threaded listener
Forward
mailbox_outbox.rs:106-123forward_outbox: oldest spool file, send_outbox_message, unlink unless QueueFull; empty spool = Refusal::Malformed and an immediate close (no frame)
This is why the host must re-dial an idle guest
Spool
mailbox_outbox.rs:166-198write_outbox, 256 files max (MAX_SPOOLED_MESSAGES L33)
The sender-side durability that makes a daemon crash non-lossy
A3. The warm-claim analogue to copy (seccomp tier)
Step
Where
Host builds the post-restore identity
mvm-runtime/src/workload_runner/runner.rs:1045-1050ChildIdentity {token, grant_envelope, seccomp_tier, volumes}; type at mvm-vmm/src/post_restore.rs:285-307; signal builder :314-383 (with_seccomp_tier, request)
Wire
GuestRequest::PostRestore {.. seccomp_tier, volumes} (mvm-agentd/src/vsock/api.rs), PostRestoreAck {.., seccomp_tier, volumes_mounted} mapped at post_restore.rs:150-173 into PostRestoreOutcome
Guest admits before the claim commits
mvm-agentd/src/bin/mvm-guest-agent/handlers.rs:620-709handle_post_restore: admit_restored (L636-639, workload_seccomp.rs:157) before reseed, volumes, clock, init signal
Host verifies and refuses
runner.rs:1081-1102take_fresh_child_identity -> refusal.rs:71-91require_admitted_seccomp_tier, require_late_volumes_mounted; on refusal force_stop at runner.rs:1051-1064
Old agent behaviour
An agent too old to know the field refuses the handshake; the claim fails closed (runner.rs:1092-1097 message)
Registration before the fork
runner.rs:118-120BrokerRegistrar, runner.rs:946-958 registers the child's broker beforefork_standby_child; guard defused at commit runner.rs:1070-1072
Privileged late work
mvm-agentd/src/late_volumes.rs: a one-shot root helper started by PID 1 only when the cmdline carries mvm.late_volume_slots; the first PostRestore consumes it. Only Firecracker overrides late_volume_slots (mvm-backends/src/driver/fc.rs:1069); the default (mvm-vmm/src/driver/traits.rs:169) is the no-slot case, so HVF boots no helper
That last row is why B7 does not reuse the late-volume helper for the outbox mkdir.
A4. Findings from the inventory that change the design
Rate and byte limits are declared but not enforced.default_template sets max_messages_per_minute: 600, max_outstanding_calls: 8, max_queue_contribution_bytes: 1 MiB (mailbox_launch.rs:347-350). git grep max_messages_per_minute origin/main -- crates finds only the definition (mvm-contract/.../mailbox.rs:558), a non-zero validity check (:579) and this template. Nothing reads them at admission (admit, mailbox_service.rs:531-555). Today the only bounds are depth 64 and frame 64 KiB (LAUNCH_LIMITS, mailbox_launch.rs:499-502), i.e. up to 4 MiB of queue per VM, 4 GiB at 1,000 VMs in the worst case. A shared daemon needs these enforced (B4); enforcing them applies the limits the host already minted, so it is a bug fix, not a policy change.
Replay state is unbounded.InMemoryReplayProtector is a BTreeSet that only grows (mvm-core/src/mailbox.rs:90-98). Per-VM process death used to cap it; a resident daemon holding VMs for days cannot. The sealer's sequence/nonce counters are strictly increasing (GuestSealer::seal, mailbox_service.rs:125-137), so a high-water mark plus a bounded recent window is exact for the guest path. The peer-send path (serve_peer) needs the set.
The seal-then-open loop is a self-loopback.serve_guest seals the guest plaintext with host-held keys (GuestSealer::seal) and admit immediately opens it with the same host-held keys. The keys' security value is on the peer send path (a host peer that holds keys can enqueue) and in epoch isolation, not guest confidentiality from the process that holds them. This is central to the key-custody argument in B2.
The per-VM process also polls at 500 ms, independent of the 2 s redial, so idle cost today is roughly 2 timer wakeups plus 0.5 dials per second per VM: about 2,000 wakeups/s and 500 dials/s (each dial also wakes the guest agent) at 1,000 VMs. Estimate from the code, not a measurement; PR 0 measures it.
fork_vm_full.rs:359 and commands/vm/checkpoint.rs:1024 hard-code mailbox: None; a forked/checkpointed copy of a granted VM never gets a mailbox. Out of scope, unchanged.
A5. The density harness (constraint 8)
scripts/measure-hvf-density.sh is the single runner; scripts/measure-firecracker-density.sh just sets MVM_HVF_DENSITY_BACKEND=firecracker and execs it. Post-processing: scripts/density_timings.py (receipt publication latency per wave). A separate in-process lifecycle benchmark lives in crates/mvm-cli/src/bench/ (cold_launch*.rs, harness.rs, report.rs).
Run with granted VMs:
# Linux/KVM box
MVM_HVF_DENSITY_BACKEND=firecracker \
MVM_HVF_DENSITY_MAILBOX=1 \
MVM_HVF_DENSITY_WAVES="1 10 100 600 1000" \
MVM_HVF_DENSITY_ACK_LARGE=1 \ # required above 100 VMs (L142)
MVM_HVF_DENSITY_LAUNCH_SPACING_SECONDS=0 \ # default 1 s would take ~17 min to launch 1000
MVM_HVF_DENSITY_SETTLE_SECONDS=60 \
MVM_HVF_DENSITY_OUT_DIR=/tmp/mvm-fc-density-mailbox \
scripts/measure-firecracker-density.sh
# macOS 26+ Apple Silicon: same without BACKEND (default hvf)
Mechanics: MVM_HVF_DENSITY_MAILBOX=1 (script L17, L138, L300-341) builds mvm-mailbox, exports MVM_MAILBOX_PATH, and passes --mailbox <sha256(name)> per machine run (L562-566). Large waves require configured guest RAM <= 80% of host RAM (usage text). The guest runs sleep 240 by default, so it measures idle granted VMs only.
What it records per wave: rss-{before,settled,after-cleanup}.tsv per component with counts (awk regex at L230 and L400 lists mvm-mailbox), mailbox-threads-*.tsv (L420-431 uses ps -M, which is macOS syntax; on Linux ps -M is an SELinux column, so thread counts there are probably wrong, verify in PR 0), host memory, storage, launch failures, cleanup_ok, receipt publication timings, and wave_N_mailbox_{processes,rss_kib,threads} in summary.tsv (L595-607).
What it does not measure and PR 0 must add: idle CPU and wakeups/s (no sampling), first-message latency (hard-coded unavailable:no_public_receipt_probe, L636-641), warm-claim vs cold split, daemon RSS/threads/fds as a function of N, teardown leaks (mailbox.sock, mailbox-epoch.secret, registrations), and any behaviour above ps snapshots. The regexes at L230/L400/L425/L596 key off the process name mvm-mailbox; keeping the binary name keeps them valid.
B. Design
B1. Process topology: one daemon per tenant
Unit: one mvm-mailbox daemon per tenant, directory host_agent_dir(tenant)-style (mvm-core/src/config.rs:691), control socket 0700-dir-protected like secure_host_agent_socket_dir. mvm local is the single-tenant slice (one daemon per host in practice); mvmd runs the same binary per tenant. This is the "smallest real trust boundary" in the owner's rule: per-VM buys nothing (it is the O(N) problem), host-wide would put different tenants' derived keys and plaintext queues in one address space.
Not a role inside mvm-host-agent. That daemon "holds no keys (the moat)" (mvm-hostd/src/broker/daemon.rs:17-21) and forwards to a signer helper. Mailbox needs derived keys, so it must be a separate process with its own address space; folding it in would break that moat and xtask check-trust-gradient's ledger. It is a separate binary reusing the host-agent's machinery (spawn lock, pid file, idle exit host_agent_idle.rs, control framing host_agent_spawn.rs:608-680).
Sharding: none in v1. One reactor comfortably multiplexes thousands of mostly-idle fds; the cost is crypto on messages, handled by the worker pool. Revisit only if measured p99 at 1,000 VMs misses B10.
Trust-gradient ledger: add a host-tier row (mailbox-daemon; forbidden authorities signing-key, plan-admission, audit-writer; witness fn: the test that a registration carrying a seed-shaped field is refused). audit-writer stays out: the daemon emits no audit entries; mailbox.read stays client-signed (constraint 4).
B2. Trust moat: is a keyless parser / key-holder split needed?
No for v1. Argument, in order of weight:
What guest bytes reach. The only guest-controlled structure is u32 len + OutboxMessage (mvm-core/src/mailbox/frame.rs:60-95: 32-byte recipient then an opaque body, size-checked at MAX_MAILBOX_PAYLOAD_BYTES and max_frame_bytes). The header, capability, routing and keys are host templates (GuestSealer, mailbox_launch.rs:318-411). The guest cannot make the daemon parse JSON, CBOR or any nested format. The broker, by contrast, parses rich guest JSON service calls; that is why it needed a keyless parser. Rust, bounded 64 KiB, fixed-offset decode: the parse surface is smaller than the control-socket JSON the daemon must parse anyway.
The plaintext queue is in the same process either way. A split that keeps keys out of the parser still leaves every queued plaintext message in whichever process owns the queue, and that process must open envelopes to produce it. The marginal secret a compromised daemon gains from also holding the derived keys is the ability to forge or open live-epoch frames for the same tenant's live mailboxes, which is the same set whose queued plaintext it already reads.
Keys are small-radius by construction. Per-launch, per-epoch, derived from a separate random root plus a per-launch secret shredded at stop (mailbox_launch.rs:1-19). A daemon compromise exposes no seed, no encryption root, no past epoch, no other tenant, and nothing outlives Deregister/VM stop.
A split would put a hop on the hot path (parser -> key holder per message) for 1,000 VMs, directly against the warm-delta budget, for the marginal benefit in (2).
Confinement instead (these are real requirements, each with a witness in B9):
Apply the in-process jailer used by the other roles (mvm-hostd jailer module); AF_UNIX only, no exec, no TCP/UDP sockets; check-single-network-path stays green because the daemon opens no new network path.
RLIMIT_CORE=0 and PR_SET_DUMPABLE=0 (Linux) / PT_DENY_ATTACH-equivalent where available (macOS): key material must not reach a core file. Unverified whether the jailer already does this; PR 2 checks and adds.
env_clear, own process group (as mailbox_process.rs:116-124 does today).
All key-carrying buffers Zeroizing; DerivedMailboxKeys dropped right after trust(); deregistration drops GuestSealer + MailboxPolicy (both hold Key/SigningKey copies; verify ed25519-dalekzeroize feature is on, otherwise add it).
Alternative that would make the daemon keyless (owner decision D3): for the guest path, skip seal-then-open entirely and enqueue plaintext after capability/size/rate admission; keep keys only for the peer send path. That removes keys from the daemon for guest traffic but changes the witnessed behaviour (sealed envelopes opened, mission_demo scenarios) and the claim narrative. Not recommended now; revisit with ADR-046's durable store.
B3. Registration protocol and authentication
Transport: AF_UNIX stream at <tenant dir>/mailbox-control.sock, directory 0700, socket 0600. Frames: u32 BE len + JSON, cap 64 KiB (control replies tiny; CONTROL_MAX_FRAME_BYTES precedent host_agent_spawn.rs:45). One request per connection.
Authentication (all three, defense in depth):
Filesystem: 0700 directory the guest cannot reach.
Peer credentials: daemon checks SO_PEERCRED (Linux) / getpeereid (macOS) uid equals its own euid; any other uid is refused before reading.
Signed, domain-separated message: Ed25519 by the host signer over the JCS bytes (mvm_core::protocol::broker_control::sign/verify), verified against the host signer public key the daemon is configured with (host_signer_public_key_path, the precedent at host_agent_spawn.rs:~180). A new type MailboxControlV1 { domain: "mvm.mailbox.control.v1", tenant, issued_at_ms, nonce: [u8;16], op } so a signature for the host-agent control plane (or the HVF instance control, hvf_control.rs, which already does domain separation) cannot be replayed here, and vice versa. Daemon refuses |now - issued_at| > 30 s and a repeated (nonce) inside that window (small LRU).
Messages (#[serde(deny_unknown_fields)], new module mvm-contract/src/protocol/mailbox_control.rs, sign/verify helpers beside mvm-core/src/protocol/broker_control.rs):
Validate vm_id safe and state_dir == vm_state_dir(vm_id) (the existing "unsafe id" rule); bind_grant(grant, trust, sealer, ..) (rejects a grant whose authority key is not the derived one); build MailboxCore; bind the per-VM recv listener at vm_mailbox_socket_at(state_dir) mode 0600; add to the schedule. Idempotent only for identical (mailbox, epoch, keys fingerprint); a different epoch must be strictly greater and replaces; a lower/equal-epoch different-key registration is refused.
Deregister
vm_id, epoch
Remove schedule entry, close and unlink the listener, drop sealer/policy/core (zeroizing), drop queued plaintext. Idempotent.
Status
none
Counts only: registrations, total queued, fds, wakeups, uptime. Never keys, never payload. For doctor and the density harness.
Replies: Ok | Err {message} (host-authored). Register returns after the listener is bound, so a reader that runs right after launch finds the socket (today bind also happens off-thread with retry, mailbox_launch.rs:465-477; this is stricter).
Why the signature covers secrets safely: the registration travels on a same-uid 0700 socket; the signature leaks nothing about the message and the buffer is zeroized after parse. It adds integrity and domain binding, not confidentiality. A same-uid attacker can already read the 0600 seed, so the control plane is not a boundary against same-uid code; it is a boundary against the guest and against cross-tenant confusion, which matches ADR-020's own statement of the control plane's purpose.
B4. Per-VM state, isolation and bounds (constraint 3)
Each registration owns its own MailboxCore { queue, accepted, read, replay, policy }, GuestSealer, route, schedule slot and recv listener. Nothing is shared between VMs except the worker pool and the global budget. Identity of a guest connection is server-derived from the route it arrived on (as in the broker daemon): a frame from VM A's route can only ever reach VM A's core; there is no field in a guest frame that selects a mailbox.
Per-VM bounds, all enforced, all with tests:
Bound
Value
Source
Queue depth
64
existing LAUNCH_LIMITS.max_depth
Frame bytes
64 KiB
existing max_frame_bytes
Queue bytes
1 MiB
already minted as max_queue_contribution_bytes (A4.1), now enforced; refuse with QueueFull semantics (guest keeps the frame, no loss)
Rate
600/min token bucket, burst 16
already minted as max_messages_per_minute; excess answered like QueueFull so the guest keeps the frame and the schedule backs off
In-flight exchanges
1
a VM never occupies more than one worker
Connections to the recv socket
4 concurrent, 5 s IO deadline (IO_TIMEOUT)
extends mailbox_service.rs:43
Replay memory
sequence high-water + bounded window (4096) for the guest path; BTreeSet capped for the peer path
A4.2
Global bounds: total queued bytes <= 256 MiB (refuse Register-time? no: refuse admission with QueueFull); registrations <= RLIMIT_NOFILE * 0.4 (each VM costs 1 listener fd plus at most 1 held guest fd; Register refused with a clear message above that; daemon raises its soft limit to the hard limit at start); worker pool P = min(4, ncpu).
Fairness: run queue is FIFO over ready registrations with in-flight = 1, so a flooder is limited by its own token bucket and cannot starve 999 others. Estimate: sealing + admission is ~3 Ed25519 signs/verifies plus AES-GCM, on the order of 0.3-0.5 ms CPU; at the declared 10 msg/s cap, 1,000 VMs all flooding would need ~3-5 cores. That worst case is bounded by P, so latency degrades for everyone rather than starving; the acceptance test (B9 T9) measures it. If it misses, lower the default declared rate in the template (a one-line, host-minted value) rather than add machinery.
B5. Lifecycle, crash recovery, blast radius
Start: lazy, by the first registrar. ensure_mailbox_daemon(tenant) copies ensure_host_agent_daemon (host_agent_spawn.rs:134-200): create dir, secure socket dir, exclusive flock spawn lock, live-pid check using the start-time-aware marker from #5332 (process_liveness.rsEndpointPid/pid_file_has_live_process) so a recycled pid cannot read as the daemon, spawn_detached_with_config with config {tenant_id, control_socket, host_signer_public_key_path}. No seed path in the config (unlike host-agent's software_chain_key_path). First-granted-launch cost = one daemon spawn (estimate tens of ms); amortized afterwards and eliminated for warm claims by prewarm at standby refill (open point, B8 PR 4).
Idle exit: after N seconds (default 300, reuse host_agent_idle::parse_idle_timeout semantics) with zero registrations the daemon exits; the next Register respawns it. Prevents the leak class host_agent_idle.rs was written for.
Deregister (normal stop): reap_vm_host_helpers (vm_helpers.rs:21) and wall_clock.rs:116 -> signed Deregister (1 s timeout) -> then the unconditional shred_file of mailbox-epoch.secret (vm_helpers.rs:29-33). Order matters: deregister first so the daemon zeroizes before the secret is gone; if the daemon is unreachable, the shred still runs.
Unannounced VM death (launcher SIGKILLed, host crash, supervisor killed): the daemon cannot rely on a reap call. It learns by (a) a dial failing with route-gone (NotFound/ConnectionRefused on the route path) which triggers an immediate liveness probe, and (b) a coalesced liveness sweep: every ~10 s per registration (jittered, batched into <= 4 wakeups/s total) it runs state_dir_has_live_process(state_dir) (process_liveness.rs:157, ~10 us) and drops dead registrations. Honest delta: today the per-VM process is killed synchronously by pid, so keys die at stop; with the daemon, keys of an unannounced-dead VM can live in daemon memory only for up to the sweep interval (10 s; never on disk). Linux pidfd_open / macOS EVFILT_PROC would make it instant and zero-poll; deferred as a follow-up because mio does not expose EVFILT_PROC and it needs raw kevent. (Decision D4.)
Lost: for every registered VM of that tenant, the host-side queue (up to 64 messages, <= 1 MiB) and the replay state, plus all in-memory keys.
Not lost: the guest spool (<= 256 files, mailbox_outbox.rs:33); a frame is unlinked in the guest only after the host answered with a verdict other than QueueFull (forward_outbox L119-121), so a frame that never reached the host stays. Messages already accepted into the host queue but not yet read are lost (same as a per-VM process crash today, but now across all VMs).
Epoch secrets and the seed are untouched on disk, so keys are re-derivable: the registration is build_config (mailbox_process.rs:94) again, deterministically (same seed, root, grant, epoch secret file -> same keys). Nothing in the daemon was the source of truth.
Recovery trigger (no new resident seed holder): (1) any launcher that finds no live daemon starts one under the spawn lock and, as the spawner, re-registers every live granted VM of that tenant it finds (scan VM state dirs for mailbox-epoch.secret + live process + persisted plan with a grant; mvm_hostd::audit::plan_persist::read_plan already exists, used at mvm-client/src/mailbox.rs:20); (2) mvmctl machine mailbox recv and mvmctl doctor do the same lazily when a VM's socket is unreachable but the VM is live. Between crash and trigger, delivery stalls but nothing is lost.
A host-agent-style wrapper (restart on crash) is not proposed: the wrapper holds no seed, so it cannot re-register; adding the seed there is what constraint 5 forbids. Recovery latency is therefore "until the next registrar or reader", which for demo-scale local use is immediate and for fleet use is mvmd's reconcile loop. This is decision D2.
B6. Event loop and idle delivery (no polling)
Reactor (1 thread): mio::Poll (epoll on Linux, kqueue on macOS; mio is already a mvm-hostd dependency for FlowMux, so no new crate). Registered sources: control listener, one recv UnixListener per registration, held guest connections (phase 2), a wake pipe. Timers: BinaryHeap<(Instant, VmKey, Generation)>; poll timeout = next deadline; deadlines coalesced to 250 ms slots so idle wakeups are bounded (<= 4/s) independent of N. No fixed tick. Recv/peer requests are tiny (a length-prefixed JSON recv or one sealed frame): read nonblocking with a per-connection 5 s deadline, never blocking the reactor; Recv reply is queue take().
Workers (P threads): run exchanges that involve blocking I/O or crypto: dial the route (dial_route reused as-is, so the Firecracker CONNECT handshake, up to 5 s, never stalls the reactor), serve the frame (reusing MailboxCore::serve), reply, then return the result to the reactor to reschedule. One in-flight exchange per VM. Rationale for reusing the blocking dial_mux instead of writing a nonblocking CONNECT state machine: reuse-first; the worker cap bounds the damage of paused VMs; a nonblocking variant is a later optimization if the pool shows saturation.
Idle-delivery options compared
A. Host redial, coalesced schedule
B. Guest holds the accepted connection (long-poll)
C. Guest dials host doorbell
Idle host wakeups
N dials per period (1,000 VMs @ 30 s = 33/s)
0 (fd sits in epoll)
0
Idle guest wakeups
1 per period per VM (and each wakes a vCPU)
0 (agent blocks on inotify)
0
First-message latency (idle VM)
up to the period (2 s today; 30 s if lengthened)
**~ ms** (inotify + read)
~ ms
Guest change
none
agent only: hold connection until spool non-empty
agent + new standing channel in every backend driver (fc.rs, hvf.rs, libkrun.rs, qemu.rs, wire_guest_dial_bridges, hvf supervisor config)
Guest wire change
none
none on the frame; only when the guest answers
new port + daemon listener per VM
Warm-claim fit
fine
fine (listener starts post-claim)
needs the channel in the parent's spec; parents carry no plan
Old host vs new agent / old agent vs new host
n/a
old host dials a new agent: sees a held read, times out at 5 s (IO_TIMEOUT) and redials: degrades to A. New host vs old agent: empty-spool close -> daemon falls back to A schedule
n/a
Risk
highest idle cost at 1,000 VMs
HVF/FC bridge cost of one long-lived connection per VM is unverified (does the HVF host-dial bridge spawn a supervisor thread per accepted connection? driver/hvf.rs:290-312HostDialSocket); needs a spike
largest blast radius
Recommendation: A first (phase 1, zero guest change, immediate gain: no thread/process per VM), then B (phase 2). C is rejected unless B's spike shows per-connection supervisor cost. Guest change for B is small: mailbox.rs:48-65 accept loop: on accept, if the spool is empty, block on inotify (Linux guest only; unix_transport_selected() already gates non-vsock) until a non-dot file appears or the peer closes, then forward_outbox; a second accepted connection supersedes the first so a stale held socket (daemon restarted) cannot wedge the single-threaded listener. Bound: never hold longer than 10 min without a keepalive read so a vanished host is eventually noticed.
Daemon side of B: a held connection that becomes readable = a frame is arriving -> hand to a worker to serve; EOF/reset -> reschedule redial with the phase-1 backoff. After each served frame redial immediately (as POST_FRAME_PAUSE does today).
B7. Warm-claim path (constraint 7)
Goal: a granted launch is warm-claimable; the claim adds a registration and one handshake field, nothing else on the critical path.
Parent side (dormant capability, no authority):
New cmdline token mvm.mailbox_standby=1 on every standby parent (set where the factory parent config is derived, runner.rs:1104-1125spawn_standby_captured/factory_parent_config; parent never carries a plan, grant or key; ADR-046's "warm snapshots contain ... never live authority, keys, nonces, sequences" holds).
PID 1 (guest_bootstrap.rs:621-636) provisions the 0700 /run/mvm/mailbox/outbox dir on that token, as it does today for mvm.mailbox=1. No listener. Reuse provision_outbox_dir as-is.
Why not the late-volume helper: it only exists on guests with mvm.late_volume_slots, which only Firecracker sets (A3). HVF would be left cold. Why a dormant directory instead of a helper: the agent (uid 901) owns the dir it needs and only needs to removeoutbox when ungranted; it cannot create it (A2). Keeping a helper path out of this avoids a second privileged one-shot.
The standby compat key (StandbyCompat, vm_backend.rs) must include the capability, so granted launches only claim a capable parent and a pre-existing parent without it keeps serving only ungranted launches (the claim falls back to cold with a typed reason).
Claim side (host):
In claim_standby (runner.rs:677), after broker_registered (runner.rs:946-958) and beforechild_forked: mint the epoch secret into the child dir, derive_launch_keys, ensure_mailbox_daemon(tenant), Register with route = driver.host_dial_route(&child_dir, MAILBOX_PORT). Behind a MailboxRegistrar trait mirroring BrokerRegistrar (runner.rs:118-120) so the runner stays unit-testable with a recording double. Returns a MailboxGuard that deregisters and shreds on drop until defuse() at commit (runner.rs:1070-1072).
Channel plumbing: the child's host-dialed mailbox channel comes from the plan-derived standing_sockets (sockets.rs:81, built from the child's child_cfg) through workload_vsock_ports into fork_request.channels (runner.rs:~965). Firecracker host-dial needs no pre-arm (mux CONNECT); HVF must be verified: handoff_channel_mask(req.channels) (driver/hvf.rs:966) must include the mailbox host-dial socket for a resident handoff, and the child's state dir must be the one the route names. This is the single biggest unknown (R1).
ChildIdentity gains mailbox: Option<MailboxAdmission> (post_restore.rs:285), carried through VsockPostRestoreSignal (:314-383) into GuestRequest::PostRestore. MailboxAdmission carries no key and no secret: just { listen: true } (the guest holds nothing). Ungranted claims send Some({listen:false}) so the guest retracts the dormant dir.
PostRestoreAck gains mailbox: Option<MailboxState>; mapped in post_restore.rs:150-173 into PostRestoreOutcome; refusal.rs gains require_admitted_mailbox(child, requested, outcome) next to require_admitted_seccomp_tier (:71-91) with the same fail-closed semantics; checked in take_fresh_child_identity (runner.rs:1099-1101); on refusal the existing force_stop path (runner.rs:1051-1064) runs and the guard deregisters.
Guest side (agent), in handle_post_restore (handlers.rs:620-709), after admit_restored and before the init signal:
listen:true: verify /run/mvm/mailbox/outbox exists and is owned by 901 (else refuse: the parent was not capable); start the 5257 listener once (refactor init_mailboxmailbox.rs:24-39 into start_listener() guarded by a OnceLock, called from both the cold path and here); report Listening. Second admission with a different value is refused (same shape as restored_admission_refusal, workload_seccomp.rs).
listen:false: remove_dir_all(outbox) best-effort, assert no listener; report Absent. (An empty root-owned-by-nobody-else /run/mvm/mailbox 0700 dir may remain because the agent cannot rmdir its parent; it is unreachable and inert. The cmdline-test invariant "an ungranted guest asserts nothing" is restated as "no 5257 listener and no spool dir".)
Older agent: the unknown PostRestore field is refused, so a granted claim of an old-agent parent fails closed exactly as an old-agent seccomp claim does today (runner.rs:1092-1097); never "claimed without a mailbox".
Remove the exclusion at pool.rs:1020-1022 and its comment (:1016-1023), flip the assertions at pool.rs:1974-2079 (granted becomes warm-eligible when the backend/parent is mailbox-capable; stays cold with a typed reason otherwise).
Latency accounting (estimates; PR 4 measures): derive + mint (HMAC x ~12, 2-3 Ed25519 pubkey derivations, one small file write) <= 1 ms; control round trip + Register (bind listener) ~0.3 ms; PostRestore extra work (spawn one thread, bind a vsock listener) ~0.2 ms; first dial after claim hits a not-yet-admitted listener only if registration precedes admission: harmless (refused, backoff 100 ms), or add an Activate op sent after identity_verified to start dialing only then (cheaper than wasted dials, +0.2 ms, decide in PR 4). Budget: granted warm-claim p50 within +2 ms of ungranted warm (ADR-046 warm delta p99 <= 2 ms), absolute inside the 300 ms launch budget wherever the ungranted claim is.
B8. Migration plan: small PRs, each independently shippable and witnessed
PR
Content
Ships alone because
Witness (hermetic + live)
0 Baseline + harness
Extend scripts/measure-hvf-density.sh / density_timings.py: idle CPU and wakeup sampler (Linux /proc/<pid>/status ctxt-switch deltas and Threads: instead of ps -M; macOS ps -M/top -stats), per-component fd/thread counts, first-message latency probe (a guest command that writes the spool, the harness reads via machine mailbox recv in a tight loop; guest/host clock skew is coarse so also record daemon-side accept timestamps once PR 2 lands), warm-claim split, teardown leak check (mailbox.sock, mailbox-epoch.secret, daemon registrations), ACK_LARGE-aware wave sizes 100/600/1000
Pure tooling
Run today's per-VM process at N=100 / 600 / 1000 granted on the KVM box and N=100 on the Mac; commit the numbers to the #5292 thread. This is the baseline the owner's earlier comment asks for.
1MailboxCore extraction + enforce bounds
Split State/Shared/admit/serve/serve_peer/take out of MailboxService (no thread, no listener); enforce queue bytes + rate + bounded replay (A4.1, A4.2). Per-VM wrapper keeps working unchanged.
No topology change; fixes real unbounded memory
Existing mailbox_service tests unchanged and green; new unit tests per bound (over bytes, over rate, replay window, peer path); mailbox_crypto.rs, replay_protection.rs, mailbox_process_bin.rs, mission_demo unchanged.
2 Daemon + registrar behind a flag
mailbox_control.rs types and sign/verify; mvm-mailbox daemon mode (mio reactor, worker pool, heap, per-registration listener, jailer, Status); ensure_mailbox_daemon/register/deregister in mvm-vmm/src/host/ mirroring host_agent_spawn.rs; MVM_MAILBOX_DAEMON=1 opt-in, per-VM process still default; trust-gradient row
Opt-in, default behaviour unchanged
Hermetic: signature/domain/nonce/skew/uid/unsafe-id/state-dir/epoch-regression refusals; seed-shaped field refused (deny_unknown_fields); idle exit; register -> frame -> recv -> mailbox.read audit end to end; N=64 in-process isolation test (flood VM A, VM B latency and queue unchanged). Live A/B at N=10/100 via harness MAILBOX=1 (flag on vs off) on FC and HVF.
3 Cutover, delete per-VM
Wire plan_admission.rs:2174, transient.rs:194, persistent.rs:523 to register; reap_vm_host_helpers deregisters (vm_helpers.rs:26); wall_clock.rs:116 path; liveness sweep; crash-recovery sweep by spawner and recv; delete spawn_serving, mailbox.pid, the 500 ms keeper loop, MailboxProcessConfig stdin path (no fallback valve: this is a first version, and ADR-020's valve is explicitly "transition only")
Daemon proven in PR 2
kill -9 daemon test (queues lost, guest spool intact, recovery by next registrar, no duplicate delivery); teardown leak test (0 sockets, 0 secret files, registrations 0, daemon RSS back to baseline); N=100/600/1000 live.
4 Warm claim
A3/B7 in full; drop pool.rs:1020-1022; parent token + compat key; MailboxRegistrar/MailboxGuard; PostRestore/Ack fields; require_admitted_mailbox; agent start_listener() refactor and listen:false retraction; Activate decision
Independent of PR 5
Hermetic: runner tests with recording registrar (registered before fork, deregistered on refusal, defused on commit), refusal on missing/wrong ack, old-agent refusal; agent tests (listener once, retraction, owner check). Live: extend crates/mvm-runtime/tests/fc_warm_pool_live.rs and fc_fork_live.rs (they already build ChildIdentity, L449/L477) with a granted claim; HVF resident-handoff live on the Mac. Measure claim p50/p99 granted vs ungranted, and a post-claim message delivery.
5 Held connection
Guest agent long-poll (inotify), daemon fd-in-epoll path, fallback to A on old agents
Pure optimization over PR 3/4
Spike first: HVF bridge cost per held connection (R2). Then idle wakeups at N=1000 (host and guest), first-message latency p50/p99, daemon-restart-with-stale-held-socket test.
6 Prewarm + doctor
ensure_mailbox_daemon at standby refill/bootstrap so the first granted claim pays no spawn; mvmctl doctor line (registrations, daemon alive, last sweep)
Polish
cold first-granted-claim delta <= 5 ms
Doc/ADR changes: amend ADR-020's host-agent section (or add a short ADR) recording that per-tenant resident daemons now also cover the mailbox and that the mailbox daemon holds derived keys; update CLAUDE.md only after PR 3 lands (the "per-VM process" statements in mailbox_process.rs module docs and scripts/measure-hvf-density.sh usage text change at PR 3).
B9. Test and witness plan
Hermetic (every PR, no VM):
T1 control plane: unsigned, wrong key, wrong domain (a host-agent SignedControl replayed here), stale issued_at, repeated nonce, wrong peer uid, deny_unknown_fields (a seed/root/secret field is a parse error, preserving the existing the_config_refuses_unknown_fields/the_config_carries_no_seed... intent from mailbox_process.rs:352-396).
T2 registration validation: unsafe vm_id, state_dir != vm_state_dir(vm_id), grant authority key not derived, epoch regression, identical re-register idempotent.
T3 isolation: two registrations, A floods and violates bounds; B's queue contents, replay state, accepted/read counters and latency unaffected; a frame arriving on A's route can never land in B (assert by construction: route-keyed).
T4 zeroization: after Deregister a test hook confirms the sealer/policy are dropped and the keys buffer zeroed (reuse the Zeroizing/Drop pattern at mailbox_launch.rs:202-209).
T5 crash model: simulate daemon death mid-queue; assert guest spool unchanged, queue lost, re-register restores delivery with identical derived keys (deterministic derivation test), no frame delivered twice.
T6 reactor: N=1000 in-process fake routes, idle for 2 s, assert wakeups <= bound and thread count independent of N (the same style as an_idle_service_blocks_instead_of_polling, mailbox_service.rs:589-594).
T8 warm claim (PR 4): registrar before fork, guard behaviour on every refusal arm, require_admitted_mailbox table (none / wrong / ok), agent admission (once, retract, owner check).
T9 load: worst-case flood of all registrations at the declared rate against a P-thread pool; record p99 Register latency and message latency; fails the PR if control p99 > 50 ms.
Gates to run per CLAUDE.md before each PR (not run here): cargo fmt --all -- --check, nextest, doctests, clippy -D warnings, just check::gated (Linux-gated test files, required-features), xtask check-single-network-path, check-trust-gradient, check-claim-catalog, check-witness-citations (the CLAUDE.md prose that names spawn_for_launch/mailbox.pid must be updated in PR 3 or the gate fails).
Live scale witness (density harness, B10 numbers): Firecracker on the KVM box (up to 1,000; mind the rotational disk and clean up, per the box notes) and HVF on the Mac (as many as RAM allows, 80% rule; Mac is swap-bound, so use 100 here and extrapolate only from the daemon's per-VM slope).
B10. Acceptance numbers for 1,000 granted VMs
Baseline (today, to be measured in PR 0, estimates from issue/code): 1,000 processes, 5-8 GB RSS, 2,000-3,000 threads, ~2,000 timer wakeups/s + ~500 dials/s idle, granted launch cold-only.
<= 2N + 64; Register refused cleanly at 40% of RLIMIT_NOFILE
Idle daemon wakeups
A: <= 40/s (redial 30 s period + sweep); B: <= 10/s
voluntary_ctxt_switches delta / 60 s
Idle daemon CPU
<= 0.1% of one core
Idle guest mailbox wakeups
A: 1 per 30 s per VM; B: 0
guest agent counters
Cold granted launch delta (register)
p99 <= 5 ms over ungranted
ADR-046 cold delta
Granted warm claim
p50 <= ungranted + 2 ms; p99 delta <= 2 ms; absolute inside 300 ms where ungranted is; ready p99 <= 50 ms
ADR-046 warm rows
First message, idle VM
A: <= redial period; B: p50 <= 10 ms, p99 <= 50 ms at 1% concurrent senders
ADR-046 first-message p99 <= 10 ms is the stretch target
Register / Deregister p99 under 1,000-VM flood
<= 50 ms
Teardown of all N
0 mailbox.sock, 0 mailbox-epoch.secret, daemon registrations 0, RSS within 10% of empty baseline, idle exit within window
harness leak check
kill -9 daemon at N=1,000
Recovery of all registrations by the next registrar <= 5 s; lost = host-queued unread only; 0 lost sender-side messages
T5 live
Isolation
One VM at 10x its declared rate does not move any other VM's p99 by > 20%
T3/T9 live
C. Risks, unknowns and owner decisions
C1. Unknowns to resolve by spike (cheap, early)
R1 HVF warm-claim route. For a resident handoff the claimed VM keeps the parent's identity (a claimed VM "is named by its standby id"); the HVF per-port host-dial socket (vm_hvf_vsock_port_socket_at) must exist under the claimed VM's state dir for the mailbox port, but the factory parent's spec carries no plan-derived channels. Check handoff_channel_mask (hvf.rs:966) and the supervisor's HostDialSocket creation (hvf.rs:290-312). If the supervisor cannot add the per-port socket at handoff, HVF granted launches stay cold until it can (FC unaffected).
R2 Held-connection cost on the bridge. Does the HVF supervisor (and the libkrun/QEMU bridges) spend a thread per accepted host-dial connection? A long-lived connection per VM would move the O(N) threads into the supervisors. Measure at N=100 before PR 5.
R3 Key zeroization completeness.GuestSealer/MailboxPolicy hold Key/SigningKey copies; confirm Key zeroizes on drop and that ed25519-dalek is built with zeroize. If not, PR 2 adds it; T4 pins it.
R4 Jailer posture. Whether the in-process jailer already sets RLIMIT_CORE=0 / non-dumpable. If not, PR 2 adds it.
R5 Crypto throughput. The 0.3-0.5 ms/message estimate is unmeasured; PR 1 adds a micro-benchmark. If wrong by 10x the pool size and declared rate need revisiting.
R6 Standby-parent compat on upgrade. Existing parents lack mvm.mailbox_standby=1; the compat key must make them cold-only for granted launches, never claimable-then-refused.
R7 macOS FD and kqueue limits for 1,000 UDS listeners + held connections (default ulimit -n 256 on macOS shells; the docs already warn about TooManyOpenFiles, public/src/content/docs/install/linux.md:109).
R8 ADR-046 overlap. ADR-046's durable fabric store (resident shard worker, "no workload private key", ~L350) is the long-term home of mailbox queues. The registration seam in B3 is store-agnostic on purpose (the daemon's MailboxCore is the in-memory precursor), so this work should not be redone when the durable store lands; keep MailboxCore behind a trait only if the second implementation appears (YAGNI until then).
C2. Decisions the owner must make (ranked, with a recommendation)
D1. Accept per-tenant key custody in one resident process (vs. a keyless scheduler with VM-scoped key holders). This is the decision flagged in the earlier comment on this issue. Per-tenant daemon: holds only live-epoch derived keys and queued plaintext for one tenant; no seed; zeroized at deregister; blast radius = that tenant's currently running granted VMs' mailboxes. VM-scoped key holders keep that to one VM but reintroduce a process, a lazy-spawn latency and journal/anchor recovery on the message path, i.e. exactly the cost #5292 exists to remove, and the plaintext queue exposure is not reduced unless the queue also moves out. Recommend: accept the per-tenant daemon (B1/B2). If the owner will not accept it, the only coherent alternative is D3.
D2. Crash recovery without a resident seed holder. Recommend: recovery is performed by the next seed-holding registrar or reader (launcher, mvmctl machine mailbox recv, doctor), not by a supervising wrapper. Accepts "delivery stalls until the next trigger; nothing sender-side is lost". The alternative (let the existing per-tenant signer helper derive and push keys to the daemon on restart) gives automatic recovery but widens the key-holding helper's API, against the "no seed in any long-lived process other than the existing host signer holders" rule's spirit. Recommend the lazy trigger; revisit only if fleet use needs sub-second recovery.
D3. Keep sealing on the guest path (witnessed behaviour) or drop it. Recommend keep (zero behaviour change, constraint 2 and the "no change to witnessed delivery behaviour" acceptance). Dropping seal-then-open for the guest path would make the daemon keyless for guest traffic but changes claims narrative and mission_demo; do it only together with ADR-046's durable store.
D4. Residency bound of keys after an unannounced VM death. Today: 500 ms (pid-killed synchronously). Daemon with a 10 s sweep: <= 10 s in memory only. Recommend accept 10 s now, schedule pidfd_open / EVFILT_PROC as a follow-up that makes it instant and also removes the sweep's wakeups.
D5. Idle-delivery mechanism: A then B, reject C. Recommend A (PR 3), B (PR 5) gated on spike R2; C (guest-dials doorbell) only if B fails, because it adds a standing channel to four drivers.
D6. Enforcing the already-declared capability limits (max_messages_per_minute 600, max_queue_contribution_bytes 1 MiB, max_outstanding_calls 8). It tightens behaviour for a guest that sends > 10 msg/s or > 1 MiB unread. Recommend enforce (they are the host's own template values; without them 1,000 VMs can hold 4 GiB of queue). If any existing witness (mission_demo, conformance) sends faster, raise the template value instead of dropping enforcement.
D7. Warm parent shape. Recommend every standby parent boots mailbox-capable and dormant (mvm.mailbox_standby=1, empty 0700 dir, no listener) with retraction for ungranted claims, rather than a second class of "mailbox parents" (doubles the standby pool) or a late-mkdir helper (HVF has none). Cost: ungranted guests may carry an empty unreachable /run/mvm/mailbox directory; the "an ungranted guest creates nothing" test restates as "no listener and no spool dir".
D8. No fallback valve. Recommend deleting the per-VM spawn path at PR 3 (this repo's no-backcompat rule for a first version), keeping the flag only for PR 2's A/B measurement. ADR-020's MVM_HOST_AGENT_DAEMON=0 precedent was a transition valve; here the A/B is bounded to one PR.
C3. What this design deliberately does not do
Cross-host transport (#4633); a durable store (ADR-046); changing the guest wire frame; any new network path; mailbox for forked/checkpointed copies (mailbox: None at fork_vm_full.rs:359, checkpoint.rs:1024); sharding the daemon; per-VM CPU accounting.
Burn-down triage note (2026-11-14), no implementation claimed here:
Interim items from this issue have landed since it was filed: #5343 (idle host dial held open instead of 2 s redial) and #5339 (answer only a frame-sending peer). The per-process model, warm-claim exclusion, and the 5-8 MB/process density cost remain as described.
One finding from today's live witnesses that a daemon redesign must not inherit: on current main (0.28.0-rc.3 dev), a granted guest's first outbox frame never reaches the host queue on HVF — fresh grant, mvm.mailbox=1 on the cmdline, frame visible in the guest outbox, machine mailbox recv empty after 10 minutes (details in #5294's latest comment). Whatever serves the vsock dial in the resident-daemon design needs a hermetic delivery test that fails on today's behavior before the density work lands, or the 1,000-VM acceptance would measure a broken pipe.
Suggested slicing for whoever takes this (matches the issue's direction, adds an order): (1) daemon + registration/deregistration for newly launched VMs only, dispatched per launch, with the witnessed delivery/stop/audit checks run against the daemon path; (2) warm-claim provisioning through the post-restore handshake; (3) the 1,000-VM density witness. Keeping the guest-facing outbox frame unchanged makes (1) a host-side swap behind the same checks.
D1 before implementation: one keyless scheduler per host, holding only sealed frames and message metadata (routing, queue, journal, wake scheduling); per-VM key custody for seal/open/authorization. The old per-tenant key-holding daemon design in this thread is superseded. A compromised scheduler must not be able to read or forge mailbox contents; a compromised VM key holder exposes that VM's mailbox. This is a trusted-host, compromised-daemon boundary, not confidentiality from a compromised host.
Can we reuse an existing per-VM process? HVF and libkrun: yes as a candidate, their existing per-VM supervisors can be extended with an isolated VM-scoped key service, subject to verifying their current privilege, guest-input exposure, signed admission and teardown/zeroization; don't assume embedding keys into their VMM address spaces is automatically safe. Firecracker: not with today's topology.FcRunningVm tracks the detached firecracker VMM (crates/mvm-backends/src/driver/fc.rs:1319); fc.rs:975-978 explicitly says there is no mvm-*-supervisor in front of it. The host-agent is shared, and the network/substitution endpoint is not a key holder. The Firecracker VMM parses guest input and is not a safe equivalent. Use a VM-scoped key-holder helper there, or add a narrowly scoped per-VM supervisor only with its own threat review; do not put keys in the shared scheduler or network endpoint.
Can the holder be event-driven with zero idle polling? Yes as a design, not yet a witness. The scheduler can wait on its control/queue FDs and deadlines, signal a VM-scoped holder only when sealed work arrives, and a lazy Firecracker helper can start on demand. The existing guest mailbox already blocks on a held host connection for outbox work (#5343), but current MailboxLaunch still sleeps/retries on errors (mailbox_launch.rs:537-548); key-holder lifecycle and crash recovery need armed process/socket events, with bounded timeout and durable journal reconciliation only after a crash. A VM holder can exit after an idle deadline, never timer-tick per VM. Validate first-message latency, journal/anchor recovery, HVF/libkrun supervision and the Firecracker helper at N=100/600/1000 before promising zero wakeups or warm-claim parity. Current HVF delivery witness on #5294 is broken; fix/witness that pipe before accepting a density result as delivery evidence.
I will keep the seven-PR sequence, revise it around sealed-only scheduling and VM-scoped custody after the measured PR 0 baseline. No code or scale result is claimed by this decision comment.
ADR-057 amendment for the D1 decision is open for review in PR #5513. The D1 checkbox is checked on #5510; PR 0 and PRs 1–6 remain unchecked pending hardware measurements and the revised plan.
Revised seven-PR sequence for D1: sealed-only scheduler, VM-scoped keys
This supersedes the older per-tenant key-holding daemon plan above. It is a staged proposal, not an implementation or a claim that N=600/1,000 is witnessed. D1 is recorded in the issue body and ADR-057 amendment #5513.
PR
Scope
Gate before the next step
0
#5517: cold/required-warm density harness, idle CPU/wakeup sampler, per-VM process/RSS/thread/FD and cleanup evidence. Existing per-VM mailbox-helper baseline at N=100/600/1,000, 256/512 MiB.
A passing granted baseline on each requested shape, or explicitly record which host-memory guard refuses. The supplied amd64 has 64,222 MiB total: its safety gate refuses every 600/1,000 × 256/512 wave. m1 cold/256 N=100 measured here; amd64 cold/256 N=100 had 16 startup failures at 0.5 s spacing and is being retried slower. Do not extrapolate either to 1,000.
1
Extract the current VM's mailbox state/crypto into one VM-scoped holder interface; enforce already-declared queue-byte/rate and bounded replay limits without changing the guest frame or topology.
Per-VM crypto, refusal, replay and teardown tests. The known HVF guest-outbox delivery regression (#5294) must have a separate live witness before any HVF cutover.
2
Add an opt-in keyless host scheduler with sealed-only queue/journal, route metadata, bounded capacity, event-driven wake and VM registration/deregistration. No key, plaintext queue, seed, envelope-opening or audit-writer privilege in its address space.
Wrong-key/domain/epoch/replay/uid and scheduler-compromise tests; durable acceptance only after journal commit, with typed recovery on crash. Existing delivery remains default.
3
Firecracker VM-scoped key holder: a lazy helper (there is no existing separate per-VM supervisor), connected to the sealed scheduler; never use the network endpoint.
Cold granted delivery, host-key isolation, helper exit/restart and journal-anchor recovery on real KVM; no idle helper process.
4
HVF and libkrun VM-scoped key holders in their existing per-VM supervisors, only after checking those supervisors' guest-input and confinement boundary; sealed scheduler connection.
Per-backend cold delivery, stop/zeroization, crash recovery and no extra resident mailbox process; HVF #5294 delivery must already work.
5
Cut over the default cold granted path on witnessed backends, remove the old resident mvm-mailbox per-VM process and idle redial, and make scheduler/holder waits event-driven with bounded deadlines.
N=100 cold A/B against PR 0, idle CPU/wakeups, first-message latency, teardown and restart; no change to signed audit/durable acknowledgement. Keep an unsupported backend fail-closed, never silently fall back to a key-holding scheduler.
6
Warm-claim admission through a dormant standby capability and fail-closed post-restore handshake; register the child holder before claim, revoke on failure, then measure granted warm/first-message and scale.
Explicit required-claim proof, old-agent refusal, journal/anchor recovery, native HVF and Firecracker witnesses. The 600/1,000 gate requires an adequately provisioned host; do not mark acceptance from a 64 GiB box.
Host key custody is limited to each VM's holder; scheduler compromise exposes metadata and possibly availability, but cannot read/forge payloads. A compromised holder exposes its own mailbox. Any PR that changes ownership, admission or audit must name its failure/stop/crash transitions and keep the ADR-001 claim witnesses green. This plan does not authorize implementing the later security-sensitive PRs before the custody ADR review and PR 0 evidence are complete.
On the Apple Silicon m1 runner (macOS 27.0.1, 64 GiB), harness PR #5517 at f0c0a3c328: Alpine sleep 900, 2 vCPUs, 0.5 s launch spacing, 20 s settle, MVM_HVF_DENSITY_IDLE_SAMPLE_SECONDS=15, MVM_HVF_DENSITY_MAILBOX=1. One unmeasured primer and N=1 control preceded the N=100 wave. MVM_HVF_DENSITY_ALLOW_BUSY_HOST=1 bypassed a broad process-argv preflight; CPU figures are shared-host samples.
Settled N=100
Per-VM mailbox helpers
HVF supervisors
Network endpoints
mvmctl
Process count
100
100
200
100
Sum of RSS
945,440 KiB (~923 MiB)
5,642,608 KiB
2,421,136 KiB
5,589,392 KiB
Threads
200
1,200
600
700
Descriptors
4,500
8,400
9,000
4,500
Idle CPU, 15.2 s
72.2 millicores
1,304.1 millicores
109.3 millicores
68.8 millicores
Kernel wakeups/s
490.7
11,161.5
872.7
718.7
The existing mailbox model therefore adds one resident process, two threads, ~9.23 MiB summed RSS and ~4.9 wakeups/s per granted VM at this N. RSS is not unique physical RAM. The separate host-agent/signer rows included shared-host daemons, so do not charge their whole RSS to the wave. Requested VMM/mailbox process counts were present; zero early launch failures; stop/rm status 0, no matching processes after cleanup. Completion receipts and first-message latency are unavailable for this resident workload; this measures process/idle cost, not delivery (the HVF outbox regression #5294 remains open) or a warm claim. Artifacts: m1 /tmp/m5510mb256a/ (summary.tsv, wave-100/rss-settled.tsv, wave-100/cleanup.txt). N=600/1,000 × 256/512 still fail the 64 GiB host's 80% configured-RAM guard and need a larger reference host. The revised seven-PR plan above remains contingent on that gate.
Firecracker PR 0 baseline correction to the provisional plan: the slower amd64 N=100 cold/256 MiB retry (2 s spacing) reached 97/100 VMMs but still had three launch failures (Firecracker API socket >3 s once; missing per-VM flowmux-identity.ext4 twice). Cleanup passed and host memory was ample. The original 0.5 s run reached 84/100. Thus the Firecracker N=100 cold baseline is not accepted at either spacing; no mailbox-granted Firecracker wave should be presented as a 100-VM success until the startup/path race is diagnosed. See #5510 status. This does not change the measured m1 granted table above.
Activity
auser commentedon Nov 11, 2026
Architecture decision needed before treating this as the default mailbox density implementation. This issue proposes one host/tenant daemon holding derived keys for many mailboxes; a separate local draft design instead proposes a keyless opaque-notice scheduler with VM-scoped key holders and the existing host-dial payload path. The former reduces resident process/thread cost and may improve warm claims, but a compromised daemon can expose keys for every mailbox it holds; the latter preserves a narrower key-custody boundary but needs proven lazy-helper journal/anchor recovery, first-message latency, and warm-claim compatibility. Recommend measuring the current helper baseline at 100/600/1,000 granted running VMs and explicitly choosing the acceptable compromise boundary before replacing the process model. #4635 remains open for real-VM and durable-message latency evidence. No production notifier or pooled daemon from this draft has been shipped or merged.
auser commentedon Nov 11, 2026
#5292 design: one resident mailbox daemon per tenant, registration-based, event-driven, warm-claim compatible
Read-only design pass against
origin/main(142ba6b). Everyfile:linebelow is from that tree. Nothing was built or run. Numbers marked "estimate" are unmeasured; the baseline measurement is PR 0 below and must land before any acceptance number is treated as a target.0. Recommendation in ten lines
mvm-mailboxprocess with one residentmvm-mailboxdaemon per tenant (same boundary as the host-agent daemon,mvm-core/src/config.rs:691). Zero host processes per granted VM; threads independent of VM count.SignedControlprecedent,mvm-contract/src/protocol/broker_control.rs:185-222). The registration carries only per-mailbox derived keys. The daemon holds the host signer public key, never the seed, the encryption root or the epoch secret.OutboxMessage::decode(32-byte recipient, opaque body) and the plaintext queue lives in the same process regardless, so a split protects almost nothing. Confine the daemon instead (jailer, no core dumps, AF_UNIX only). Detail in B2.GuestSealer, bounded queue 64, replay protector, epoch policy. Per-VMMailboxCorestate is split out ofMailboxServiceso the daemon reuses the code, not a copy.mio(already amvm-hostddependency) for control socket, per-VM recv listeners and held guest connections; a deadline heap for timers; a small fixed worker pool (default 4) for blocking dial/exchange work. No thread per VM, no fixed-rate polling.mvm.mailbox_standby=1: PID 1 pre-creates the 0700 outbox dir, no listener). The claim registers with the daemon before the fork and delivers a mailbox admission through the PostRestore handshake, exactly like the seccomp tier (ChildIdentity->handle_post_restore->PostRestoreAck->require_admitted_mailbox, fail closed). Then drop the exclusion atcrates/mvm-cli/src/commands/pool.rs:1020-1022.mvmctlcan redo deterministically because the derived keys are re-derivable from the seed plus the epoch secret file that stays in the VM state dir until stop.specs/adrs/046-secure-message-fabric.md~L585-596): warm claim to ready p99 <= 50 ms, fabric-enabled vs disabled cold delta p99 <= 5 ms and warm delta p99 <= 2 ms, first local message p99 <= 10 ms, idle CPU event-driven. Concrete 1,000-VM numbers in B10.A. Exact inventory of what exists today
A1. Host side, per-VM process
crates/mvm-hostd/src/bin/mvm-mailbox.rs:17-28panic_hook::install, reads all of stdin into a plainVec(L21, not zeroized),parse,std::process::exit(serve(config))rawbuffers on the control path becomeZeroizing.crates/mvm-hostd/Cargo.toml[[bin]] mvm-mailbox(the comment above it says "secret-substitution moat", a stale copy; fix in passing);crates/mvm-cli/src/host_binaries/manifest.rs:95;.github/workflows/release.yml:360,382,560mailbox_process.rs:50-60MailboxProcessConfig {grant, keys, socket, route, vm_state_dir},deny_unknown_fieldsRegistermessage (B3).vm_state_dirstays (daemon validates it equalsvm_state_dir(vm_id)).mailbox_launch.rs:185-209DerivedMailboxKeys {sender, authority, encryption, template};Dropzeroizes (L202-209)trust()so onlyGuestSealerandMailboxPolicyhold key material.mailbox_process.rs:94-111build_config:mint_epoch_secret(mailbox_launch.rs:125),derive_launch_keys(L259-287, reads seed viaderived_keysL150-165, encryption root L266), shred on failuremailbox_process.rs:222-242spawn_serving: resolve bin,command()(L116-124:env_clear, null stdout/stderr,process_group(0)), write config, writemailbox.pid(L240)ensure_mailbox_daemon(tenant)+register, modeled onhost_agent_spawn.rs:134-200(flock, pid file,spawn_detached_with_config) and:226-250.spawn_for_launchmailbox_process.rs:193-220;serve_started:163-189(stops the VM when the grant cannot be served);require_servable/require_wired:129-157(refuse before boot if backend lacksVmCapabilities::guest_mailbox)require_*unchanged.crates/mvm-hostd/src/plan_admission.rs:2113(require),:2174(spawn, rollback viaundo_launchstage"mailbox-process");crates/mvm-cli/src/exec/transient.rs:168,194;crates/mvm-client/src/launch/persistent.rs:497,523spawn_for_launch/serve_startedfor the register call; stage names stay for audit continuity.crates/mvm-cli/src/commands/pool.rs:1016-1023:warm_eligible_launchexcludes a launch whenmailbox_cmdline_token(cfg).is_some(); tests atpool.rs:1974-2079assert it stays coldmailbox_process.rs:68-85serve: loopstd::thread::sleep(500 ms)(POLL_INTERVALL46) overvm_state_dir.exists()andendpoint_needed(launcher_is_alive, vm_is_alive)(vm_lifetime.rs:98,124;process_liveness.rs:157state_dir_has_live_process)mailbox_launch.rs:453-496serve_derived_with(spawns a thread): bind retry L465-477, thenserve_guestL483 ->Ok: reset attempt,POST_FRAME_PAUSE10 ms;Err:dial_delay(L65-71: 100 ms x 300 attempts, then 2 s forever); constants L58-62mailbox_service.rs:374-392serve_guest->telemetry_collector/mod.rs:981-991dial_route:PortSocket=UnixStream::connect;Mux=dial_mux(L993-1006) = blockingconnect_to_port_once(.., 5s)withCONNECT <port>handshake.HostDialRouteatcrates/mvm-core/src/protocol/vm_backend.rs:42-47; FCdriver/fc.rs:1314(Mux), HVFdriver/hvf.rs:1030(PortSocket)Registerverbatim.dial_muxis blocking, so dials run on the worker pool, not the reactor (B6).mailbox_service.rs:203-224State {queue: VecDeque, accepted, read, replay: InMemoryReplayProtector, policy}behind oneMutex;admit:531-555(depth check thenopen_envelope);serve:447-461,serve_peer:464-488,read_frame:514-529(capmax_frame_bytes), IO timeout 5 s:43MailboxService, which also owns an accept thread (bind:296-344,wait_readable:272-281)MailboxCore(State + serve/serve_peer/admit/take, no thread, no listener) and a thin per-VM wrapper that disappears at cutover.mailbox_service.rs:78-152GuestSealer,with_grant:93-104;bind_grant:191-201verifies grant authority key == derived authoritybind_grantat registration, so a registrant cannot bind a grant the keys do not match.mvm_core::config::vm_mailbox_socket_at(mvm-core/src/config.rs:876);mailbox_service::recv:254-269;mvm_client::mailbox::recvcrates/mvm-client/src/mailbox.rs:19-45;machine mailbox recvcrates/mvm-cli/src/commands/machine/mailbox.rs:14-20; auditemit_mailbox_readcrates/mvm-hostd/src/audit/emitter/mailbox.rs:11-29(signed client-side, payload-free)mailbox.readaudit entry do not change (constraint 4).crates/mvm-vmm/src/host/vm_helpers.rs:21-34reap_vm_host_helpers:reap_pid_file(mailbox.pid)(L26) then shredmailbox-epoch.secret(L29-33). Callers:mvm-runtime/.../runner/backend.rs:151(stop),runner/durability.rs:60,mvm-hostd/src/supervisor/wall_clock.rs:116(supervisor stop);wall_clock.rs:107-110also shreds on its own pathreap_mailbox_registration(state_dir, vm): read amailbox.tenantref file (likeHOST_AGENT_TENANT_REF,host_agent_spawn.rs:~482-520), send signedDeregister(1 s timeout,CONTROL_IO_TIMEOUTprecedent), then shred. Shred stays unconditional.mvm-core/src/config.rs:882vm_mailbox_secret_at=<state>/mailbox-epoch.secret;mailbox_launch.rs:122-134mint/shredA2. Guest side (frozen by constraint 2, except the two admission points)
crates/mvm-vmm/src/host/egress_bridge.rs:50-53mailbox_cmdline_token(mvm.mailbox=1iff admitted plan has a grant); emitted atcmdline.rs:165; HVF/standing channel atmvm-runtime/.../runner/sockets.rs:81mvm-agentd/src/guest_bootstrap.rs:109,618-636provision_mailbox_dir: PID 1, root, before privilege drop, gated on the token;mailbox_outbox.rs:128-157creates/run/mvm/mailboxand/run/mvm/mailbox/outbox, 0700, chown 901/run/mvmafterwards".WORKLOAD_UID= 901 = agent uid (guest_mount.rs:17), so the agent owns the dir and can later removeoutboxbut not/run/mvm/mailbox.crates/mvm-agentd/src/bin/mvm-guest-agent/mailbox.rs:24-39init_mailbox(cmdline gated, spawned frommvm-guest-agent.rs:1090post-activation), accept loop:48-65: one connection at a time,forward_outboxper accept, 5 s socket IO timeoutmailbox_outbox.rs:106-123forward_outbox: oldest spool file,send_outbox_message, unlink unlessQueueFull; empty spool =Refusal::Malformedand an immediate close (no frame)mailbox_outbox.rs:166-198write_outbox, 256 files max (MAX_SPOOLED_MESSAGESL33)A3. The warm-claim analogue to copy (seccomp tier)
mvm-runtime/src/workload_runner/runner.rs:1045-1050ChildIdentity {token, grant_envelope, seccomp_tier, volumes}; type atmvm-vmm/src/post_restore.rs:285-307; signal builder:314-383(with_seccomp_tier,request)GuestRequest::PostRestore {.. seccomp_tier, volumes}(mvm-agentd/src/vsock/api.rs),PostRestoreAck {.., seccomp_tier, volumes_mounted}mapped atpost_restore.rs:150-173intoPostRestoreOutcomemvm-agentd/src/bin/mvm-guest-agent/handlers.rs:620-709handle_post_restore:admit_restored(L636-639,workload_seccomp.rs:157) before reseed, volumes, clock, init signalrunner.rs:1081-1102take_fresh_child_identity->refusal.rs:71-91require_admitted_seccomp_tier,require_late_volumes_mounted; on refusalforce_stopatrunner.rs:1051-1064runner.rs:1092-1097message)runner.rs:118-120BrokerRegistrar,runner.rs:946-958registers the child's broker beforefork_standby_child; guard defused at commitrunner.rs:1070-1072mvm-agentd/src/late_volumes.rs: a one-shot root helper started by PID 1 only when the cmdline carriesmvm.late_volume_slots; the first PostRestore consumes it. Only Firecracker overrideslate_volume_slots(mvm-backends/src/driver/fc.rs:1069); the default (mvm-vmm/src/driver/traits.rs:169) is the no-slot case, so HVF boots no helperThat last row is why B7 does not reuse the late-volume helper for the outbox mkdir.
A4. Findings from the inventory that change the design
default_templatesetsmax_messages_per_minute: 600,max_outstanding_calls: 8,max_queue_contribution_bytes: 1 MiB(mailbox_launch.rs:347-350).git grep max_messages_per_minute origin/main -- cratesfinds only the definition (mvm-contract/.../mailbox.rs:558), a non-zero validity check (:579) and this template. Nothing reads them at admission (admit,mailbox_service.rs:531-555). Today the only bounds are depth 64 and frame 64 KiB (LAUNCH_LIMITS,mailbox_launch.rs:499-502), i.e. up to 4 MiB of queue per VM, 4 GiB at 1,000 VMs in the worst case. A shared daemon needs these enforced (B4); enforcing them applies the limits the host already minted, so it is a bug fix, not a policy change.InMemoryReplayProtectoris aBTreeSetthat only grows (mvm-core/src/mailbox.rs:90-98). Per-VM process death used to cap it; a resident daemon holding VMs for days cannot. The sealer's sequence/nonce counters are strictly increasing (GuestSealer::seal,mailbox_service.rs:125-137), so a high-water mark plus a bounded recent window is exact for the guest path. The peer-sendpath (serve_peer) needs the set.serve_guestseals the guest plaintext with host-held keys (GuestSealer::seal) andadmitimmediately opens it with the same host-held keys. The keys' security value is on the peersendpath (a host peer that holds keys can enqueue) and in epoch isolation, not guest confidentiality from the process that holds them. This is central to the key-custody argument in B2.fork_vm_full.rs:359andcommands/vm/checkpoint.rs:1024hard-codemailbox: None; a forked/checkpointed copy of a granted VM never gets a mailbox. Out of scope, unchanged.A5. The density harness (constraint 8)
scripts/measure-hvf-density.shis the single runner;scripts/measure-firecracker-density.shjust setsMVM_HVF_DENSITY_BACKEND=firecrackerand execs it. Post-processing:scripts/density_timings.py(receipt publication latency per wave). A separate in-process lifecycle benchmark lives incrates/mvm-cli/src/bench/(cold_launch*.rs,harness.rs,report.rs).Run with granted VMs:
Mechanics:
MVM_HVF_DENSITY_MAILBOX=1(script L17, L138, L300-341) buildsmvm-mailbox, exportsMVM_MAILBOX_PATH, and passes--mailbox <sha256(name)>permachine run(L562-566). Large waves require configured guest RAM <= 80% of host RAM (usage text). The guest runssleep 240by default, so it measures idle granted VMs only.What it records per wave:
rss-{before,settled,after-cleanup}.tsvper component with counts (awk regex at L230 and L400 listsmvm-mailbox),mailbox-threads-*.tsv(L420-431 usesps -M, which is macOS syntax; on Linuxps -Mis an SELinux column, so thread counts there are probably wrong, verify in PR 0), host memory, storage, launch failures,cleanup_ok, receipt publication timings, andwave_N_mailbox_{processes,rss_kib,threads}insummary.tsv(L595-607).What it does not measure and PR 0 must add: idle CPU and wakeups/s (no sampling), first-message latency (hard-coded
unavailable:no_public_receipt_probe, L636-641), warm-claim vs cold split, daemon RSS/threads/fds as a function of N, teardown leaks (mailbox.sock,mailbox-epoch.secret, registrations), and any behaviour abovepssnapshots. The regexes at L230/L400/L425/L596 key off the process namemvm-mailbox; keeping the binary name keeps them valid.B. Design
B1. Process topology: one daemon per tenant
mvm-mailboxdaemon per tenant, directoryhost_agent_dir(tenant)-style (mvm-core/src/config.rs:691), control socket 0700-dir-protected likesecure_host_agent_socket_dir. mvm local is the single-tenant slice (one daemon per host in practice); mvmd runs the same binary per tenant. This is the "smallest real trust boundary" in the owner's rule: per-VM buys nothing (it is the O(N) problem), host-wide would put different tenants' derived keys and plaintext queues in one address space.mvm-host-agent. That daemon "holds no keys (the moat)" (mvm-hostd/src/broker/daemon.rs:17-21) and forwards to a signer helper. Mailbox needs derived keys, so it must be a separate process with its own address space; folding it in would break that moat andxtask check-trust-gradient's ledger. It is a separate binary reusing the host-agent's machinery (spawn lock, pid file, idle exithost_agent_idle.rs, control framinghost_agent_spawn.rs:608-680).mailbox-daemon; forbidden authoritiessigning-key,plan-admission,audit-writer; witnessfn:the test that a registration carrying a seed-shaped field is refused).audit-writerstays out: the daemon emits no audit entries;mailbox.readstays client-signed (constraint 4).B2. Trust moat: is a keyless parser / key-holder split needed?
No for v1. Argument, in order of weight:
u32 len+OutboxMessage(mvm-core/src/mailbox/frame.rs:60-95: 32-byte recipient then an opaque body, size-checked atMAX_MAILBOX_PAYLOAD_BYTESandmax_frame_bytes). The header, capability, routing and keys are host templates (GuestSealer,mailbox_launch.rs:318-411). The guest cannot make the daemon parse JSON, CBOR or any nested format. The broker, by contrast, parses rich guest JSON service calls; that is why it needed a keyless parser. Rust, bounded 64 KiB, fixed-offset decode: the parse surface is smaller than the control-socket JSON the daemon must parse anyway.mailbox_launch.rs:1-19). A daemon compromise exposes no seed, no encryption root, no past epoch, no other tenant, and nothing outlivesDeregister/VM stop.Confinement instead (these are real requirements, each with a witness in B9):
mvm-hostdjailer module); AF_UNIX only, no exec, no TCP/UDP sockets;check-single-network-pathstays green because the daemon opens no new network path.RLIMIT_CORE=0andPR_SET_DUMPABLE=0(Linux) /PT_DENY_ATTACH-equivalent where available (macOS): key material must not reach a core file. Unverified whether the jailer already does this; PR 2 checks and adds.env_clear, own process group (asmailbox_process.rs:116-124does today).Zeroizing;DerivedMailboxKeysdropped right aftertrust(); deregistration dropsGuestSealer+MailboxPolicy(both holdKey/SigningKeycopies; verifyed25519-dalekzeroizefeature is on, otherwise add it).Alternative that would make the daemon keyless (owner decision D3): for the guest path, skip seal-then-open entirely and enqueue plaintext after capability/size/rate admission; keep keys only for the peer
sendpath. That removes keys from the daemon for guest traffic but changes the witnessed behaviour (sealed envelopes opened,mission_demoscenarios) and the claim narrative. Not recommended now; revisit with ADR-046's durable store.B3. Registration protocol and authentication
Transport: AF_UNIX stream at
<tenant dir>/mailbox-control.sock, directory 0700, socket 0600. Frames:u32 BE len+ JSON, cap 64 KiB (control replies tiny;CONTROL_MAX_FRAME_BYTESprecedenthost_agent_spawn.rs:45). One request per connection.Authentication (all three, defense in depth):
SO_PEERCRED(Linux) /getpeereid(macOS) uid equals its own euid; any other uid is refused before reading.mvm_core::protocol::broker_control::sign/verify), verified against the host signer public key the daemon is configured with (host_signer_public_key_path, the precedent athost_agent_spawn.rs:~180). A new typeMailboxControlV1 { domain: "mvm.mailbox.control.v1", tenant, issued_at_ms, nonce: [u8;16], op }so a signature for the host-agent control plane (or the HVF instance control,hvf_control.rs, which already does domain separation) cannot be replayed here, and vice versa. Daemon refuses|now - issued_at| > 30 sand a repeated(nonce)inside that window (small LRU).Messages (
#[serde(deny_unknown_fields)], new modulemvm-contract/src/protocol/mailbox_control.rs, sign/verify helpers besidemvm-core/src/protocol/broker_control.rs):Registervm_id,state_dir,grant: MailboxGrant,route: HostDialRoute,keys: DerivedMailboxKeys,limits: MailboxLimits+ byte/rate boundsvm_idsafe andstate_dir == vm_state_dir(vm_id)(the existing "unsafe id" rule);bind_grant(grant, trust, sealer, ..)(rejects a grant whose authority key is not the derived one); buildMailboxCore; bind the per-VM recv listener atvm_mailbox_socket_at(state_dir)mode 0600; add to the schedule. Idempotent only for identical(mailbox, epoch, keys fingerprint); a different epoch must be strictly greater and replaces; a lower/equal-epoch different-key registration is refused.Deregistervm_id,epochStatusdoctorand the density harness.Replies:
Ok|Err {message}(host-authored).Registerreturns after the listener is bound, so a reader that runs right after launch finds the socket (todaybindalso happens off-thread with retry,mailbox_launch.rs:465-477; this is stricter).Why the signature covers secrets safely: the registration travels on a same-uid 0700 socket; the signature leaks nothing about the message and the buffer is zeroized after parse. It adds integrity and domain binding, not confidentiality. A same-uid attacker can already read the 0600 seed, so the control plane is not a boundary against same-uid code; it is a boundary against the guest and against cross-tenant confusion, which matches ADR-020's own statement of the control plane's purpose.
B4. Per-VM state, isolation and bounds (constraint 3)
Each registration owns its own
MailboxCore { queue, accepted, read, replay, policy },GuestSealer, route, schedule slot and recv listener. Nothing is shared between VMs except the worker pool and the global budget. Identity of a guest connection is server-derived from the route it arrived on (as in the broker daemon): a frame from VM A's route can only ever reach VM A's core; there is no field in a guest frame that selects a mailbox.Per-VM bounds, all enforced, all with tests:
LAUNCH_LIMITS.max_depthmax_frame_bytesmax_queue_contribution_bytes(A4.1), now enforced; refuse withQueueFullsemantics (guest keeps the frame, no loss)max_messages_per_minute; excess answered likeQueueFullso the guest keeps the frame and the schedule backs offIO_TIMEOUT)mailbox_service.rs:43BTreeSetcapped for the peer pathGlobal bounds: total queued bytes <= 256 MiB (refuse
Register-time? no: refuse admission withQueueFull); registrations <=RLIMIT_NOFILE * 0.4(each VM costs 1 listener fd plus at most 1 held guest fd;Registerrefused with a clear message above that; daemon raises its soft limit to the hard limit at start); worker pool P =min(4, ncpu).Fairness: run queue is FIFO over ready registrations with in-flight = 1, so a flooder is limited by its own token bucket and cannot starve 999 others. Estimate: sealing + admission is ~3 Ed25519 signs/verifies plus AES-GCM, on the order of 0.3-0.5 ms CPU; at the declared 10 msg/s cap, 1,000 VMs all flooding would need ~3-5 cores. That worst case is bounded by P, so latency degrades for everyone rather than starving; the acceptance test (B9 T9) measures it. If it misses, lower the default declared rate in the template (a one-line, host-minted value) rather than add machinery.
B5. Lifecycle, crash recovery, blast radius
Start: lazy, by the first registrar.
ensure_mailbox_daemon(tenant)copiesensure_host_agent_daemon(host_agent_spawn.rs:134-200): create dir, secure socket dir, exclusiveflockspawn lock, live-pid check using the start-time-aware marker from #5332 (process_liveness.rsEndpointPid/pid_file_has_live_process) so a recycled pid cannot read as the daemon,spawn_detached_with_configwith config{tenant_id, control_socket, host_signer_public_key_path}. No seed path in the config (unlike host-agent'ssoftware_chain_key_path). First-granted-launch cost = one daemon spawn (estimate tens of ms); amortized afterwards and eliminated for warm claims by prewarm at standby refill (open point, B8 PR 4).Idle exit: after N seconds (default 300, reuse
host_agent_idle::parse_idle_timeoutsemantics) with zero registrations the daemon exits; the nextRegisterrespawns it. Prevents the leak classhost_agent_idle.rswas written for.Deregister (normal stop):
reap_vm_host_helpers(vm_helpers.rs:21) andwall_clock.rs:116-> signedDeregister(1 s timeout) -> then the unconditionalshred_fileofmailbox-epoch.secret(vm_helpers.rs:29-33). Order matters: deregister first so the daemon zeroizes before the secret is gone; if the daemon is unreachable, the shred still runs.Unannounced VM death (launcher SIGKILLed, host crash, supervisor killed): the daemon cannot rely on a reap call. It learns by (a) a dial failing with route-gone (
NotFound/ConnectionRefusedon the route path) which triggers an immediate liveness probe, and (b) a coalesced liveness sweep: every ~10 s per registration (jittered, batched into <= 4 wakeups/s total) it runsstate_dir_has_live_process(state_dir)(process_liveness.rs:157, ~10 us) and drops dead registrations. Honest delta: today the per-VM process is killed synchronously by pid, so keys die at stop; with the daemon, keys of an unannounced-dead VM can live in daemon memory only for up to the sweep interval (10 s; never on disk). Linuxpidfd_open/ macOSEVFILT_PROCwould make it instant and zero-poll; deferred as a follow-up becausemiodoes not exposeEVFILT_PROCand it needs rawkevent. (Decision D4.)Daemon crash / kill -9: blast radius, stated exactly.
mailbox_outbox.rs:33); a frame is unlinked in the guest only after the host answered with a verdict other thanQueueFull(forward_outboxL119-121), so a frame that never reached the host stays. Messages already accepted into the host queue but not yet read are lost (same as a per-VM process crash today, but now across all VMs).build_config(mailbox_process.rs:94) again, deterministically (same seed, root, grant, epoch secret file -> same keys). Nothing in the daemon was the source of truth.mailbox-epoch.secret+ live process + persisted plan with a grant;mvm_hostd::audit::plan_persist::read_planalready exists, used atmvm-client/src/mailbox.rs:20); (2)mvmctl machine mailbox recvandmvmctl doctordo the same lazily when a VM's socket is unreachable but the VM is live. Between crash and trigger, delivery stalls but nothing is lost.B6. Event loop and idle delivery (no polling)
Reactor (1 thread):
mio::Poll(epoll on Linux, kqueue on macOS;miois already amvm-hostddependency for FlowMux, so no new crate). Registered sources: control listener, one recvUnixListenerper registration, held guest connections (phase 2), a wake pipe. Timers:BinaryHeap<(Instant, VmKey, Generation)>; poll timeout = next deadline; deadlines coalesced to 250 ms slots so idle wakeups are bounded (<= 4/s) independent of N. No fixed tick. Recv/peer requests are tiny (a length-prefixed JSONrecvor one sealed frame): read nonblocking with a per-connection 5 s deadline, never blocking the reactor;Recvreply is queuetake().Workers (P threads): run exchanges that involve blocking I/O or crypto: dial the route (
dial_routereused as-is, so the FirecrackerCONNECThandshake, up to 5 s, never stalls the reactor),servethe frame (reusingMailboxCore::serve), reply, then return the result to the reactor to reschedule. One in-flight exchange per VM. Rationale for reusing the blockingdial_muxinstead of writing a nonblocking CONNECT state machine: reuse-first; the worker cap bounds the damage of paused VMs; a nonblocking variant is a later optimization if the pool shows saturation.Idle-delivery options compared
fc.rs,hvf.rs,libkrun.rs,qemu.rs,wire_guest_dial_bridges,hvfsupervisor config)IO_TIMEOUT) and redials: degrades to A. New host vs old agent: empty-spool close -> daemon falls back to A scheduledriver/hvf.rs:290-312HostDialSocket); needs a spikeRecommendation: A first (phase 1, zero guest change, immediate gain: no thread/process per VM), then B (phase 2). C is rejected unless B's spike shows per-connection supervisor cost. Guest change for B is small:
mailbox.rs:48-65accept loop: on accept, if the spool is empty, block oninotify(Linux guest only;unix_transport_selected()already gates non-vsock) until a non-dot file appears or the peer closes, thenforward_outbox; a second accepted connection supersedes the first so a stale held socket (daemon restarted) cannot wedge the single-threaded listener. Bound: never hold longer than 10 min without a keepalive read so a vanished host is eventually noticed.Daemon side of B: a held connection that becomes readable = a frame is arriving -> hand to a worker to
serve; EOF/reset -> reschedule redial with the phase-1 backoff. After each served frame redial immediately (asPOST_FRAME_PAUSEdoes today).B7. Warm-claim path (constraint 7)
Goal: a granted launch is warm-claimable; the claim adds a registration and one handshake field, nothing else on the critical path.
Parent side (dormant capability, no authority):
mvm.mailbox_standby=1on every standby parent (set where the factory parent config is derived,runner.rs:1104-1125spawn_standby_captured/factory_parent_config; parent never carries a plan, grant or key; ADR-046's "warm snapshots contain ... never live authority, keys, nonces, sequences" holds).guest_bootstrap.rs:621-636) provisions the 0700/run/mvm/mailbox/outboxdir on that token, as it does today formvm.mailbox=1. No listener. Reuseprovision_outbox_diras-is.mvm.late_volume_slots, which only Firecracker sets (A3). HVF would be left cold. Why a dormant directory instead of a helper: the agent (uid 901) owns the dir it needs and only needs to removeoutboxwhen ungranted; it cannot create it (A2). Keeping a helper path out of this avoids a second privileged one-shot.StandbyCompat,vm_backend.rs) must include the capability, so granted launches only claim a capable parent and a pre-existing parent without it keeps serving only ungranted launches (the claim falls back to cold with a typed reason).Claim side (host):
claim_standby(runner.rs:677), afterbroker_registered(runner.rs:946-958) and beforechild_forked: mint the epoch secret into the child dir,derive_launch_keys,ensure_mailbox_daemon(tenant),Registerwithroute = driver.host_dial_route(&child_dir, MAILBOX_PORT). Behind aMailboxRegistrartrait mirroringBrokerRegistrar(runner.rs:118-120) so the runner stays unit-testable with a recording double. Returns aMailboxGuardthat deregisters and shreds on drop untildefuse()at commit (runner.rs:1070-1072).standing_sockets(sockets.rs:81, built from the child'schild_cfg) throughworkload_vsock_portsintofork_request.channels(runner.rs:~965). Firecracker host-dial needs no pre-arm (muxCONNECT); HVF must be verified:handoff_channel_mask(req.channels)(driver/hvf.rs:966) must include the mailbox host-dial socket for a resident handoff, and the child's state dir must be the one the route names. This is the single biggest unknown (R1).ChildIdentitygainsmailbox: Option<MailboxAdmission>(post_restore.rs:285), carried throughVsockPostRestoreSignal(:314-383) intoGuestRequest::PostRestore.MailboxAdmissioncarries no key and no secret: just{ listen: true }(the guest holds nothing). Ungranted claims sendSome({listen:false})so the guest retracts the dormant dir.PostRestoreAckgainsmailbox: Option<MailboxState>; mapped inpost_restore.rs:150-173intoPostRestoreOutcome;refusal.rsgainsrequire_admitted_mailbox(child, requested, outcome)next torequire_admitted_seccomp_tier(:71-91) with the same fail-closed semantics; checked intake_fresh_child_identity(runner.rs:1099-1101); on refusal the existingforce_stoppath (runner.rs:1051-1064) runs and the guard deregisters.Guest side (agent), in
handle_post_restore(handlers.rs:620-709), afteradmit_restoredand before the init signal:listen:true: verify/run/mvm/mailbox/outboxexists and is owned by 901 (else refuse: the parent was not capable); start the 5257 listener once (refactorinit_mailboxmailbox.rs:24-39intostart_listener()guarded by aOnceLock, called from both the cold path and here); reportListening. Second admission with a different value is refused (same shape asrestored_admission_refusal,workload_seccomp.rs).listen:false:remove_dir_all(outbox)best-effort, assert no listener; reportAbsent. (An empty root-owned-by-nobody-else/run/mvm/mailbox0700 dir may remain because the agent cannot rmdir its parent; it is unreachable and inert. The cmdline-test invariant "an ungranted guest asserts nothing" is restated as "no 5257 listener and no spool dir".)PostRestorefield is refused, so a granted claim of an old-agent parent fails closed exactly as an old-agent seccomp claim does today (runner.rs:1092-1097); never "claimed without a mailbox".Remove the exclusion at
pool.rs:1020-1022and its comment (:1016-1023), flip the assertions atpool.rs:1974-2079(granted becomes warm-eligible when the backend/parent is mailbox-capable; stays cold with a typed reason otherwise).Latency accounting (estimates; PR 4 measures): derive + mint (HMAC x ~12, 2-3 Ed25519 pubkey derivations, one small file write) <= 1 ms; control round trip +
Register(bind listener) ~0.3 ms; PostRestore extra work (spawn one thread, bind a vsock listener) ~0.2 ms; first dial after claim hits a not-yet-admitted listener only if registration precedes admission: harmless (refused, backoff 100 ms), or add anActivateop sent afteridentity_verifiedto start dialing only then (cheaper than wasted dials, +0.2 ms, decide in PR 4). Budget: granted warm-claim p50 within +2 ms of ungranted warm (ADR-046 warm delta p99 <= 2 ms), absolute inside the 300 ms launch budget wherever the ungranted claim is.B8. Migration plan: small PRs, each independently shippable and witnessed
scripts/measure-hvf-density.sh/density_timings.py: idle CPU and wakeup sampler (Linux/proc/<pid>/statusctxt-switch deltas andThreads:instead ofps -M; macOSps -M/top -stats), per-component fd/thread counts, first-message latency probe (a guest command that writes the spool, the harness reads viamachine mailbox recvin a tight loop; guest/host clock skew is coarse so also record daemon-side accept timestamps once PR 2 lands), warm-claim split, teardown leak check (mailbox.sock,mailbox-epoch.secret, daemon registrations),ACK_LARGE-aware wave sizes 100/600/1000MailboxCoreextraction + enforce boundsState/Shared/admit/serve/serve_peer/takeout ofMailboxService(no thread, no listener); enforce queue bytes + rate + bounded replay (A4.1, A4.2). Per-VM wrapper keeps working unchanged.mailbox_servicetests unchanged and green; new unit tests per bound (over bytes, over rate, replay window, peer path);mailbox_crypto.rs,replay_protection.rs,mailbox_process_bin.rs,mission_demounchanged.mailbox_control.rstypes and sign/verify;mvm-mailboxdaemon mode (mio reactor, worker pool, heap, per-registration listener, jailer,Status);ensure_mailbox_daemon/register/deregisterinmvm-vmm/src/host/mirroringhost_agent_spawn.rs;MVM_MAILBOX_DAEMON=1opt-in, per-VM process still default; trust-gradient rowdeny_unknown_fields); idle exit; register -> frame ->recv->mailbox.readaudit end to end; N=64 in-process isolation test (flood VM A, VM B latency and queue unchanged). Live A/B at N=10/100 via harnessMAILBOX=1(flag on vs off) on FC and HVF.plan_admission.rs:2174,transient.rs:194,persistent.rs:523to register;reap_vm_host_helpersderegisters (vm_helpers.rs:26);wall_clock.rs:116path; liveness sweep; crash-recovery sweep by spawner andrecv; deletespawn_serving,mailbox.pid, the 500 ms keeper loop,MailboxProcessConfigstdin path (no fallback valve: this is a first version, and ADR-020's valve is explicitly "transition only")pool.rs:1020-1022; parent token + compat key;MailboxRegistrar/MailboxGuard;PostRestore/Ackfields;require_admitted_mailbox; agentstart_listener()refactor andlisten:falseretraction;Activatedecisioncrates/mvm-runtime/tests/fc_warm_pool_live.rsandfc_fork_live.rs(they already buildChildIdentity, L449/L477) with a granted claim; HVF resident-handoff live on the Mac. Measure claim p50/p99 granted vs ungranted, and a post-claim message delivery.ensure_mailbox_daemonat standby refill/bootstrap so the first granted claim pays no spawn;mvmctl doctorline (registrations, daemon alive, last sweep)Doc/ADR changes: amend ADR-020's host-agent section (or add a short ADR) recording that per-tenant resident daemons now also cover the mailbox and that the mailbox daemon holds derived keys; update
CLAUDE.mdonly after PR 3 lands (the "per-VM process" statements inmailbox_process.rsmodule docs andscripts/measure-hvf-density.shusage text change at PR 3).B9. Test and witness plan
Hermetic (every PR, no VM):
SignedControlreplayed here), staleissued_at, repeated nonce, wrong peer uid,deny_unknown_fields(aseed/root/secretfield is a parse error, preserving the existingthe_config_refuses_unknown_fields/the_config_carries_no_seed...intent frommailbox_process.rs:352-396).vm_id,state_dir != vm_state_dir(vm_id), grant authority key not derived, epoch regression, identical re-register idempotent.Deregistera test hook confirms the sealer/policy are dropped and the keys buffer zeroed (reuse theZeroizing/Droppattern atmailbox_launch.rs:202-209).an_idle_service_blocks_instead_of_polling,mailbox_service.rs:589-594).require_admitted_mailboxtable (none / wrong / ok), agent admission (once, retract, owner check).Registerlatency and message latency; fails the PR if control p99 > 50 ms.cargo fmt --all -- --check, nextest, doctests, clippy-D warnings,just check::gated(Linux-gated test files,required-features),xtask check-single-network-path,check-trust-gradient,check-claim-catalog,check-witness-citations(the CLAUDE.md prose that namesspawn_for_launch/mailbox.pidmust be updated in PR 3 or the gate fails).Live scale witness (density harness, B10 numbers): Firecracker on the KVM box (up to 1,000; mind the rotational disk and clean up, per the box notes) and HVF on the Mac (as many as RAM allows, 80% rule; Mac is swap-bound, so use 100 here and extrapolate only from the daemon's per-VM slope).
B10. Acceptance numbers for 1,000 granted VMs
Baseline (today, to be measured in PR 0, estimates from issue/code): 1,000 processes, 5-8 GB RSS, 2,000-3,000 threads, ~2,000 timer wakeups/s + ~500 dials/s idle, granted launch cold-only.
Targets after PR 3 (cold) / PR 4 (warm) / PR 5 (idle):
pscount ofmvm-mailbox== 1)mailbox_processesRegisterrefused cleanly at 40% ofRLIMIT_NOFILEvoluntary_ctxt_switchesdelta / 60 smailbox.sock, 0mailbox-epoch.secret, daemon registrations 0, RSS within 10% of empty baseline, idle exit within windowkill -9daemon at N=1,000C. Risks, unknowns and owner decisions
C1. Unknowns to resolve by spike (cheap, early)
vm_hvf_vsock_port_socket_at) must exist under the claimed VM's state dir for the mailbox port, but the factory parent's spec carries no plan-derived channels. Checkhandoff_channel_mask(hvf.rs:966) and the supervisor'sHostDialSocketcreation (hvf.rs:290-312). If the supervisor cannot add the per-port socket at handoff, HVF granted launches stay cold until it can (FC unaffected).GuestSealer/MailboxPolicyholdKey/SigningKeycopies; confirmKeyzeroizes on drop and thated25519-dalekis built withzeroize. If not, PR 2 adds it; T4 pins it.RLIMIT_CORE=0/ non-dumpable. If not, PR 2 adds it.mvm.mailbox_standby=1; the compat key must make them cold-only for granted launches, never claimable-then-refused.ulimit -n256 on macOS shells; the docs already warn aboutTooManyOpenFiles,public/src/content/docs/install/linux.md:109).~L350) is the long-term home of mailbox queues. The registration seam in B3 is store-agnostic on purpose (the daemon'sMailboxCoreis the in-memory precursor), so this work should not be redone when the durable store lands; keepMailboxCorebehind a trait only if the second implementation appears (YAGNI until then).C2. Decisions the owner must make (ranked, with a recommendation)
D1. Accept per-tenant key custody in one resident process (vs. a keyless scheduler with VM-scoped key holders). This is the decision flagged in the earlier comment on this issue. Per-tenant daemon: holds only live-epoch derived keys and queued plaintext for one tenant; no seed; zeroized at deregister; blast radius = that tenant's currently running granted VMs' mailboxes. VM-scoped key holders keep that to one VM but reintroduce a process, a lazy-spawn latency and journal/anchor recovery on the message path, i.e. exactly the cost #5292 exists to remove, and the plaintext queue exposure is not reduced unless the queue also moves out. Recommend: accept the per-tenant daemon (B1/B2). If the owner will not accept it, the only coherent alternative is D3.
D2. Crash recovery without a resident seed holder. Recommend: recovery is performed by the next seed-holding registrar or reader (launcher,
mvmctl machine mailbox recv,doctor), not by a supervising wrapper. Accepts "delivery stalls until the next trigger; nothing sender-side is lost". The alternative (let the existing per-tenant signer helper derive and push keys to the daemon on restart) gives automatic recovery but widens the key-holding helper's API, against the "no seed in any long-lived process other than the existing host signer holders" rule's spirit. Recommend the lazy trigger; revisit only if fleet use needs sub-second recovery.D3. Keep sealing on the guest path (witnessed behaviour) or drop it. Recommend keep (zero behaviour change, constraint 2 and the "no change to witnessed delivery behaviour" acceptance). Dropping seal-then-open for the guest path would make the daemon keyless for guest traffic but changes claims narrative and
mission_demo; do it only together with ADR-046's durable store.D4. Residency bound of keys after an unannounced VM death. Today: 500 ms (pid-killed synchronously). Daemon with a 10 s sweep: <= 10 s in memory only. Recommend accept 10 s now, schedule
pidfd_open/EVFILT_PROCas a follow-up that makes it instant and also removes the sweep's wakeups.D5. Idle-delivery mechanism: A then B, reject C. Recommend A (PR 3), B (PR 5) gated on spike R2; C (guest-dials doorbell) only if B fails, because it adds a standing channel to four drivers.
D6. Enforcing the already-declared capability limits (
max_messages_per_minute600,max_queue_contribution_bytes1 MiB,max_outstanding_calls8). It tightens behaviour for a guest that sends > 10 msg/s or > 1 MiB unread. Recommend enforce (they are the host's own template values; without them 1,000 VMs can hold 4 GiB of queue). If any existing witness (mission_demo, conformance) sends faster, raise the template value instead of dropping enforcement.D7. Warm parent shape. Recommend every standby parent boots mailbox-capable and dormant (
mvm.mailbox_standby=1, empty 0700 dir, no listener) with retraction for ungranted claims, rather than a second class of "mailbox parents" (doubles the standby pool) or a late-mkdir helper (HVF has none). Cost: ungranted guests may carry an empty unreachable/run/mvm/mailboxdirectory; the "an ungranted guest creates nothing" test restates as "no listener and no spool dir".D8. No fallback valve. Recommend deleting the per-VM spawn path at PR 3 (this repo's no-backcompat rule for a first version), keeping the flag only for PR 2's A/B measurement. ADR-020's
MVM_HOST_AGENT_DAEMON=0precedent was a transition valve; here the A/B is bounded to one PR.C3. What this design deliberately does not do
Cross-host transport (#4633); a durable store (ADR-046); changing the guest wire frame; any new network path; mailbox for forked/checkpointed copies (
mailbox: Noneatfork_vm_full.rs:359,checkpoint.rs:1024); sharding the daemon; per-VM CPU accounting.auser commentedon Nov 14, 2026
Burn-down triage note (2026-11-14), no implementation claimed here:
Interim items from this issue have landed since it was filed: #5343 (idle host dial held open instead of 2 s redial) and #5339 (answer only a frame-sending peer). The per-process model, warm-claim exclusion, and the 5-8 MB/process density cost remain as described.
One finding from today's live witnesses that a daemon redesign must not inherit: on current main (0.28.0-rc.3 dev), a granted guest's first outbox frame never reaches the host queue on HVF — fresh grant,
mvm.mailbox=1on the cmdline, frame visible in the guest outbox,machine mailbox recvempty after 10 minutes (details in #5294's latest comment). Whatever serves the vsock dial in the resident-daemon design needs a hermetic delivery test that fails on today's behavior before the density work lands, or the 1,000-VM acceptance would measure a broken pipe.Suggested slicing for whoever takes this (matches the issue's direction, adds an order): (1) daemon + registration/deregistration for newly launched VMs only, dispatched per launch, with the witnessed delivery/stop/audit checks run against the daemon path; (2) warm-claim provisioning through the post-restore handshake; (3) the 1,000-VM density witness. Keeping the guest-facing outbox frame unchanged makes (1) a host-side swap behind the same checks.
auser commentedon Nov 15, 2026
D1 before implementation: one keyless scheduler per host, holding only sealed frames and message metadata (routing, queue, journal, wake scheduling); per-VM key custody for seal/open/authorization. The old per-tenant key-holding daemon design in this thread is superseded. A compromised scheduler must not be able to read or forge mailbox contents; a compromised VM key holder exposes that VM's mailbox. This is a trusted-host, compromised-daemon boundary, not confidentiality from a compromised host.
Can we reuse an existing per-VM process? HVF and libkrun: yes as a candidate, their existing per-VM supervisors can be extended with an isolated VM-scoped key service, subject to verifying their current privilege, guest-input exposure, signed admission and teardown/zeroization; don't assume embedding keys into their VMM address spaces is automatically safe. Firecracker: not with today's topology.
FcRunningVmtracks the detachedfirecrackerVMM (crates/mvm-backends/src/driver/fc.rs:1319);fc.rs:975-978explicitly says there is nomvm-*-supervisorin front of it. The host-agent is shared, and the network/substitution endpoint is not a key holder. The Firecracker VMM parses guest input and is not a safe equivalent. Use a VM-scoped key-holder helper there, or add a narrowly scoped per-VM supervisor only with its own threat review; do not put keys in the shared scheduler or network endpoint.Can the holder be event-driven with zero idle polling? Yes as a design, not yet a witness. The scheduler can wait on its control/queue FDs and deadlines, signal a VM-scoped holder only when sealed work arrives, and a lazy Firecracker helper can start on demand. The existing guest mailbox already blocks on a held host connection for outbox work (#5343), but current
MailboxLaunchstill sleeps/retries on errors (mailbox_launch.rs:537-548); key-holder lifecycle and crash recovery need armed process/socket events, with bounded timeout and durable journal reconciliation only after a crash. A VM holder can exit after an idle deadline, never timer-tick per VM. Validate first-message latency, journal/anchor recovery, HVF/libkrun supervision and the Firecracker helper at N=100/600/1000 before promising zero wakeups or warm-claim parity. Current HVF delivery witness on #5294 is broken; fix/witness that pipe before accepting a density result as delivery evidence.I will keep the seven-PR sequence, revise it around sealed-only scheduling and VM-scoped custody after the measured PR 0 baseline. No code or scale result is claimed by this decision comment.
auser commentedon Nov 15, 2026
ADR-057 amendment for the D1 decision is open for review in PR #5513. The D1 checkbox is checked on #5510; PR 0 and PRs 1–6 remain unchecked pending hardware measurements and the revised plan.
auser commentedon Nov 15, 2026
Revised seven-PR sequence for D1: sealed-only scheduler, VM-scoped keys
This supersedes the older per-tenant key-holding daemon plan above. It is a staged proposal, not an implementation or a claim that N=600/1,000 is witnessed. D1 is recorded in the issue body and ADR-057 amendment #5513.
mvm-mailboxper-VM process and idle redial, and make scheduler/holder waits event-driven with bounded deadlines.Host key custody is limited to each VM's holder; scheduler compromise exposes metadata and possibly availability, but cannot read/forge payloads. A compromised holder exposes its own mailbox. Any PR that changes ownership, admission or audit must name its failure/stop/crash transitions and keep the ADR-001 claim witnesses green. This plan does not authorize implementing the later security-sensitive PRs before the custody ADR review and PR 0 evidence are complete.
auser commentedon Nov 15, 2026
PR 0 partial baseline: 100 granted HVF guests (m1, cold, 256 MiB)
On the Apple Silicon m1 runner (macOS 27.0.1, 64 GiB), harness PR #5517 at
f0c0a3c328: Alpinesleep 900, 2 vCPUs, 0.5 s launch spacing, 20 s settle,MVM_HVF_DENSITY_IDLE_SAMPLE_SECONDS=15,MVM_HVF_DENSITY_MAILBOX=1. One unmeasured primer and N=1 control preceded the N=100 wave.MVM_HVF_DENSITY_ALLOW_BUSY_HOST=1bypassed a broad process-argv preflight; CPU figures are shared-host samples.The existing mailbox model therefore adds one resident process, two threads, ~9.23 MiB summed RSS and ~4.9 wakeups/s per granted VM at this N. RSS is not unique physical RAM. The separate host-agent/signer rows included shared-host daemons, so do not charge their whole RSS to the wave. Requested VMM/mailbox process counts were present; zero early launch failures; stop/rm status 0, no matching processes after cleanup. Completion receipts and first-message latency are unavailable for this resident workload; this measures process/idle cost, not delivery (the HVF outbox regression #5294 remains open) or a warm claim. Artifacts: m1
/tmp/m5510mb256a/(summary.tsv,wave-100/rss-settled.tsv,wave-100/cleanup.txt). N=600/1,000 × 256/512 still fail the 64 GiB host's 80% configured-RAM guard and need a larger reference host. The revised seven-PR plan above remains contingent on that gate.auser commentedon Nov 15, 2026
Firecracker PR 0 baseline correction to the provisional plan: the slower amd64 N=100 cold/256 MiB retry (2 s spacing) reached 97/100 VMMs but still had three launch failures (Firecracker API socket >3 s once; missing per-VM
flowmux-identity.ext4twice). Cleanup passed and host memory was ample. The original 0.5 s run reached 84/100. Thus the Firecracker N=100 cold baseline is not accepted at either spacing; no mailbox-granted Firecracker wave should be presented as a 100-VM success until the startup/path race is diagnosed. See #5510 status. This does not change the measured m1 granted table above.