Repository navigation
HVF on a developer Mac: a launch spends 2 to 4 s in admission, overlay attach, initramfs attach and VMM creation before the guest starts (warm and cold) #5487
Description
Observed (v0.28.0-rc.2 release binary, macOS 27 Apple Silicon, mvmctl run --image curlimages/curl:8.21.0 --hypervisor hvf -- /bin/true, image cached, runtime prepared). A one-command launch takes about 2.7 s end to end on a quiet host for a cold boot and about 3.0 to 3.2 s for a warm claim, so a warm claim is not faster than a cold boot through the CLI. The product's own warm-claim bench reports a sample reaching the guest 3973.8 ms after invocation against the 300 ms ceiling.
Where the time goes (MVM_PHASE_TIMING=1, five cold launches, taken while other sessions' builds held the load average near 33, so absolute numbers are inflated):
| phase | range |
|---|---|
admit_plan (inside admit) |
2.1 to 4.3 s |
attach_overlay |
2.0 to 3.3 s |
attach_initramfs |
0.7 to 1.4 s |
vmm_create (the whole backend start) |
1.4 to 2.5 s |
guest ready |
2.7 ms |
command |
11 ms |
teardown |
0.24 to 0.46 s |
The guest itself is not the cost. In the warm case the claim is also not the cost: warm_claim was 0.8 ms, and the time is the same admission, overlay and initramfs work.
MVM_SPAN_TIMINGS=1 agrees: admit_plan_for_run 1.6 to 2.4 s of self time, attach_runtime_overlay 1.5 to 2.3 s, attach_universal_initramfs_if_cached about 0.8 s, SHA-256 of the image about 0.1 s in total.
What it is not: hashing the image (about 0.1 s), and the size of the audit chain (admit_plan was 2.3 to 3.7 s on homes holding 177, 316, 349 and 1,518 entries, with no trend). A process sample during admission shows the threads mostly blocked, with read the dominant running call, which points at file reads, but the release binary is stripped so I could not attribute them.
Related: the lane prepared-cold of mvmctl bench refuses this launch (lane prepared_cold forbids warm_claim, artifact_hash but the launch performed it), so an OCI --image launch is not currently measurable as a prepared cold launch. This differs from the CI-runner result recorded in #4391 (hard 300 ms ceiling passing, max 158 ms) and the published budgets tracked in #4648 and #4337. I have not established why: the host here was under heavy load, the image is an OCI image rather than whatever the runner uses, and the machines differ.
Not established: the cause inside admit_plan_for_run and attach_runtime_overlay; whether the same launch is fast on an idle host; whether a prepared bundle (.mvmpkg) skips these phases.
Reproduce: MVM_RESIDENCY=cold MVM_PHASE_TIMING=1 MVM_SPAN_TIMINGS=1 mvmctl run --image <cached image> --hypervisor hvf -- /bin/true, then repeat after mvmctl pool warm --image <image> 1 without MVM_RESIDENCY.
Why it matters: the factory-edge demo launches several workloads and each one costs this. A launch has to reach the guest well under a second for it to look like a production launch.
Activity
auser commentedon Nov 14, 2026
Follow-up measurements, and one correction.
Correction: I did not establish that a launch spends CPU-seconds inside the
mvmctlprocess. The figure I had wastimeoutput, which includes child processes (the VMM, the endpoint), so it says nothing about the in-process share. Please ignore it.Ruled out since:
mvmctl env verify-release --tag v0.28.0-rc.2 --bundle <bundle> <archive>on the installed 20 MB guest-runtime archive took 0.20 to 0.29 s including process start (mvmctl --versionalone is 0.19 s); reading and hashing the same file in Python took 0.04 s.A comparison worth explaining. The learning on
fix/bootstrap-warms-source-declarationrecords the physical M1 CI runner atattach_overlay1.3 ms and a whole startup of about 305 to 340 ms with the source runtime declaration cached (228 ms and 528 ms without it), withvmm_createabout 260 ms as the remaining cost. That is a source build. On the signed v0.28.0-rc.2 release here,attach_overlaywas 1.5 to 3.3 s. By reading, the release path (acquire_runtime_overlay_for_launchcallingverify_runtime_release_proof, thenresolve_or_seed_from_default_cache, then re-hashing the overlay and its sidecar) has no equivalent of the declaration cache. I have not shown that this is the difference.Proposed next measurement (nobody has done it): on one idle host, run the same cold launch with
MVM_PHASE_TIMING=1from (a) a source build and (b) the signed release, with the same image and the same prepared home. If (b) is slow and (a) is fast, the release path is the gap and the fix is a cache keyed by artifact identity. If both are slow on this Mac, it is the host or the OCI image path (theprepared-coldlane refuses an OCI--imagelaunch because it performsartifact_hash).auser commentedon Nov 14, 2026
Data point from the M1 physical-cold diagnostic (workflow
prepared-cold-m1-diagnostic.yml, runs 43480860734 and 43482178539, 2026-11-13,m1-runner, macOS 27.0.1, load about 4, mvm 0.28.0-rc.2 release build, HVF, Alpine image). Read from the logs and the run'spreflight.json; nothing was re-run.The ceiling really is missed on a CI runner too. The
prepared_coldlane refused the first warm-up: "sample reached the guest 358.1 ms after invocation" (run 1) and "429.5 ms" (run 2), against the 300 ms ceiling. No report was produced, so no budget was relaxed.Phases of the preflight launch (one launch, not a benchmark sample; total 1,038 ms):
admitadmit_plan26,attach_overlay220)backend_start/vmm_createdriver_boot410.8,activate_workload89.9,endpoint_spawn1.5, others under 1teardownwork.artifact_bytes_hashedwas 54,242,577 for that launch. In the code,attach_overlayrunsacquire_runtime_overlay_for_launch(crates/mvm-client/src/launch/runtime_overlay.rs), which verifies the release proof or checks the source runtime and then callssha256_file_verifiedon the overlay ext4 and its verity sidecar. About 54 MB in about 220 ms is roughly 250 MB/s, well under what hardware SHA-256 gives on Apple silicon, so hashing speed or the lack of a hit on the remembered-digest path is the first thing to check. That is a hypothesis; it is not measured.Differences from the numbers above: this is a CI runner with a fresh
MVM_HOME, not a developer Mac, and the load was low. The ratio between phases looks similar to the one in this issue, with admission, overlay andvmm_createdominating and the guest not the cost.A PR adding finer spans inside
driver_bootandattach_overlayis in progress so one native run can attribute them. It changes nothing that is verified. See also #4391 and #4648.auser commentedon Nov 14, 2026
More data, from another session reading the failed M1
prepared_colddispatches (runs 43480860734 and 43482178539; m1-runner, macOS 27.0.1, load about 4). Warm-up 0 reached the guest at 358.1 ms and 429.5 ms against the 300 ms ceiling, and the run'spreflight.jsonshowsadmit248 ms (attach_overlay220 ms,admit_plan26 ms),vmm_create602 ms (driver_boot411 ms,activate_workload90 ms),teardown174 ms, andwork.artifact_bytes_hashed= 54,242,577, so about 54 MB is hashed per launch.Side by side with the signed v0.28.0-rc.2 on a developer Mac (re-measured at a load average of 130 to 170, three cold launches, same image):
admit_planattach_overlayattach_initramfsvmm_createteardownadmit_planis the widest gap (about 100×) and it hashes nothing large, so hashing speed (the leading hypothesis forattach_overlay) does not explain it. For scale, reading and hashing the 20 MB guest-runtime archive in Python took 0.042 s here, so 54 MB should be about 0.1 s ifsha256_file_verifiedis comparable; one microbenchmark on the same 54 MB would settle the hashing question.Ruled out on the developer Mac: archive signature verification (0.2 s), macOS Keychain lookup (about 0.1 s), CPU contention, audit chain size, file copies, helper processes.
Implication: the runner does not reproduce the developer-Mac numbers, so finer spans (inside
admit_plan_for_run,attach_overlayanddriver_boot) need to be run on a developer Mac, not only on the runner. I can run a branch carrying those spans on that Mac and report back.auser commentedon Nov 14, 2026
Code-reading pass on
mainf0adf5e forattach_overlayandvmm_create. Nothing was run; the call counts are from reading and need the spans to confirm.One candidate cause for the signed-release numbers: the guest-runtime archive is fully re-verified several times per launch, uncached.
verify_release_archive(crates/mvm-build/src/guest_runtime/evidence.rs) reads the whole archive into memory and hashes it, verifies the checksum and archive signatures, writes the bytes back out to a new temp file, then gunzips that file and hashes every member. It is reached from:attach_overlaymvm-client/src/launch/runtime_overlay.rs(acquire_runtime_overlay_for_launch)attach_initramfsmvm-runtime/src/universal_initramfs.rs(attach_universal_initramfs_if_cached)admit_planplan_admission.rs(verify_selected_runtime,verify_assurance_guest_protocol, the assurance binding)vmm_createRuntimeLifecycleLease::bindinmvm-runtime/src/runtime_evidence.rsThat would fit the developer-Mac table:
attach_initramfs(one verification) 0.5 to 0.7 s,admit_plan1.7 to 2.7 s. It would also explain the runner gap, since a source-built runtime takes the cached-declaration branch (check_source_runtime) and never reaches this function.vmm_createhas untraced work before the driver boots.start_workload_custodiedcallslifecycle.bindbeforeprepare_workload_bootcreates the launch trace, so none of this appears underdriver_boot:snapshot_declared_artifactscopies the overlay, its verity file and the initramfs into a per-VM generation, hashing each while copying, andretain_release_proof_incopies the archive the same way;snapshot_fileends each copy withsync_all, andbindthen syncslaunch.jsonand the directory. On macOSsync_allisF_FULLFSYNC, whichaudit/leaf_cache.rsalready avoids for cost. Under concurrent writers this is the kind of call that leaves threads blocked while CPU-bound work is unaffected.retaining runtime generation before spawn), so each launch leaves its copies on disk.The kernel path clones and hashes the private clone instead (
kernel_custody.rsviaVerifiedRestoreFile::prepare).The hashed-bytes counter undercounts.
work.artifact_bytes_hashedis recorded only insha256_reader.verify_release_archive(Sha256::digest),hash_cappedandhash_artifacthash without recording, so the 54 MB figure excludes the archive, its members and the snapshot copies. Counting them would makeprepared_coldrefuse more launches, so that is a separate decision.Not established: the per-step cost of one verification, how much of
vmm_createis flush time, and what is inside the remainingdriver_boot. I found no fixed sleep in the HVF boot wait.I am opening a spans-only PR for
verify_release_archiveand its steps,bind, and the per-file snapshot copy and flush, so oneMVM_SPAN_TIMINGS=1run on a Mac gives calls and time for each. It changes nothing that is verified. If the spans branch already in progress covers these functions, say so here and I will drop the overlap.Generated by Claude Code
auser commentedon Nov 14, 2026
Clean measurement on an idle host: load is not the cause.
A temporary GCP VM (
n2-standard-4, Ubuntu 24.04, SSD persistent disk, KVM, load average 0.1 to 0.4, nothing else running), the signed v0.28.0-rc.2 release installed with the repository installer, Firecracker backend,MVM_RESIDENCY=cold MVM_PHASE_TIMING=1 mvmctl run --image curlimages/curl:8.21.0 -- /bin/true, one warm-up and five measured cold launches:admit_planattach_overlayattach_initramfsbackend_startteardownThe spread is about 10 ms. A cost this steady on an idle machine is deterministic work, not contention, so the 2 to 4 s phases seen on the developer Mac are not explained by load. This is a different backend (Firecracker, not HVF), but
admit_plan_for_runand the overlay and initramfs attach are shared code, and they cost about the same here as on the Mac.Against the M1 runner (source build, declaration cached):
admit_plan26 ms,attach_overlay220 ms. So on idle hardware the signed release is roughly 100× slower inadmit_planand 8× inattach_overlay. That points at the release path (or the OCI--imagepath), not the host. I have not separated those two yet.strace -f -Tof one launch: no outbound network connect at all (only Unix sockets to the VM's own runtime), and no single slow syscall; the longest are waits on the VM and a 250 ms sleep loop on a separate thread that looks like the progress spinner. So admission is not waiting on a server, and the cost is not one blocking call. I withdrew an idea I had that the phase times are multiples of 250 ms; that was a coincidence of the spinner.Next, in progress: building the exact
v0.28.0-rc.2tag with line tables on this VM and profiling one launch withperf, to name the functions. I will post the result here. The VM is temporary (auto-deleted after three hours) and I will delete it when done.auser commentedon Nov 14, 2026
Found it: the signed-release proof is verified about seven times per launch, at about 640 ms each, and that is about 80% of the launch.
Measured on the idle GCP VM (Firecracker, load under 1), with the spans from #5508 built into the exact v0.28.0-rc.2 tree (versions pinned to rc.2 locally so the cached signed runtime applies),
MVM_RESIDENCY=cold MVM_LAUNCH_SAMPLE_JSON=<path> mvmctl run --image curlimages/curl:8.21.0 -- /bin/true. Five launches, identical to within 1%:release_proof*laps (all)release_proof_verifyruntime_evidence.release_proof_verifyattach_overlay.release_proof_lookup+attach_overlay.release_proof_verifyattach_initramfs.release_proof_verifyadmit_plan.grants_budget_and_host_interfacedriver_bootguest_bootabout 239,vmm_startabout 57)admit_plan.sign_and_verify_planadmit_plan.audit_recordadmit_plan.host_signing_key_loadattach_overlay.overlay_sha256,verity_sidecar_sha256,admit_plan.image_sha256Total launch about 5.6 s, so the proof checks are about 4.5 s of it. Signing, audit, key load and all the hashing are negligible, which rules out the earlier hashing and fsync hypotheses.
This also explains the seven 20 MB temp files I saw per launch (a 20 MB guest-runtime archive copied to a temp file each time) and why the M1 runner, which runs a source runtime and a cached declaration, shows 26 ms for
admit_planand 220 ms forattach_overlay: it never runs the release-proof path. It matches the lead in #5506 (5 to 7verify_release_archivecalls on a signed release, 0 on a source runtime).Why each call costs 640 ms when
mvmctl env verify-releaseon the same archive took 0.2 s: the per-call work is more than the signature check (the proof path reads and hashes the archive, verifies the archive and checksum signatures, copies it to a temp file and parses it). #5506's inner laps would split that; I have not.Proposed fix (not implemented): verify once, not seven times. (1) Memoize the result of
verify_runtime_release_prooffor the process, keyed by the archive digest and the proof identity, so the seven call sites share one verification. (2) Persist a verification record across launches, signed by the host key and keyed the same way, exactly like the source-runtime declaration cache, so a launch checks a small record and a file identity instead of redoing the archive check. Expected effect from these numbers: (1) takes about 4.5 s down to about 0.65 s per launch; (2) takes it to about 0. What remains after that is about 323 ms ingrants_budget_and_host_interface, about 300 ms indriver_boot, and about 90 ms of teardown, so a launch would be about 0.7 s before any work on those.Caveats: this is the Firecracker backend on x86_64 with the spans binary, not HVF on the developer Mac; the Mac showed the same phases at the same order of magnitude (
admit_plan1.7 to 2.7 s,attach_overlay1.0 to 1.2 s), and #5508 also adds HVF laps, so the next step is the same spans binary on the Mac. The fix is a change to what is verified when, and needs the owner's review: what the memoized record is bound to must still cover the archive, the proof and the host key. I have not changed any code. The temporary VM stays up until the owners say they do not need it; it deletes itself by 17:42 local time.auser commentedon Nov 14, 2026
Inner split of the 640 ms (idle GCP VM, #5506's spans built into the exact rc.2 tree,
MVM_SPAN_TIMINGS=1). Note for anyone repeating this:git archivestamps files with the commit time, so Cargo may see them as older than an earlier build and recompile nothing;touchthe sources before building.Per launch:
verify_release_archiveverify_runtime_release_proofverify_runtime_for_launch_atverify_assurance_guest_protocolruntime_lifecycle.bindsnapshot_file+sync_snapshotWhere one
verify_release_archive(643 ms) goes:verify_guest_bins_archive(self)read_and_digest_archiveverify_release_archive_bytes(2 per call, the signature checks)write_archive_snapshot(the temp copy)verify_checksum_signature_bytesWhat the 422 ms is:
verify_guest_bins_archiveopens the guest-runtime.tar.gz, gunzips the whole thing (about 62 MB of binaries uncompressed) and SHA-256s every member (hash_capped), then checks the manifest against those digests. It does that on every call, seven times per launch. This also matches the roughly 54 MB hashed per launch the CI runner reported. The CPU here (Cascade Lake) has no SHA hardware instructions, which is why 20 MB read plus 62 MB hashed costs about 420 ms; Apple silicon has them, so each call should be cheaper on the Mac, but the count is the same.So the cost is: seven repetitions of one unchanged check, each of which decompresses and hashes the full runtime. Signatures (about 105 ms) and signing/admission themselves are small.
Options, for design review (claim 20 / AGENTS.md: reuse verified bytes, never cached authorization):
I have not changed any code in the repository. I am going to try option 1 as a throwaway local patch on the VM only, to measure what the launch would cost; I will post the number, not a PR.
auser commentedon Nov 15, 2026
Throwaway experiment (VM only, not a PR): caching just the member check removes 2.5 s of a 5.6 s launch.
On the idle GCP VM I patched my local copy of
verify_release_archiveso thatverify_guest_bins_archive(the 422 ms step that gunzips the guest-runtime archive and SHA-256s every member) is memoized in-process, keyed by the archive digest the signature check has just authenticated. Nothing else changed: the archive is still read and digested, and both signatures are still verified, on every call. Interleaved runs, cold launch, same image:The saving is 2.55 s, which is what the spans predicted (6 avoided repetitions × 422 ms). It is repeatable to under 0.2%.
What is left of the 3.0 s: about 7 × (107 ms read and digest + 105 ms signature checks + 8 ms temp copy) ≈ 1.5 s,
grants_budget_and_host_interfaceabout 0.32 s,driver_bootabout 0.3 s, plus teardown and small items. Verifying the archive once per launch instead of seven times would remove most of the 1.5 s (projected from the spans, not measured): total about 1.7 s. A persisted record would go lower but is a cached authorization and needs the review described above.This is a measurement, not a proposal to merge that patch: it changes when a signed artifact is verified, so it is claim 20 territory. It shows the cost is real, deterministic, and removable by reusing verified results.
auser commentedon Nov 15, 2026
The write-up for this issue is now in the repository as ADR-112 (PR #5514,
specs/adrs/112-verify-the-runtime-release-proof-once-per-launch.md, status Proposed): the measurements and what was ruled out, the decision to verify once per launch, the conditions that keep it the same verification, the alternatives, the tests the implementation needs, and the recipe for repeating the measurement on a temporary cloud VM. The temporary VM used for these measurements has been deleted and no instance, disk or address of ours remains.auser commentedon Nov 15, 2026
Design for the repeated release-proof verification (7 per launch, about 4.5 s of 5.6 s on the idle GCP VM per the #5508 spans) is written up for approval in #5515: verify the whole runtime set once per launch as one typed result consumed by overlay attach, initramfs attach, admission and dispatch, with the verified bytes used instead of re-read paths. No cross-process cache in that change; that is a separate decision.
auser commentedon Nov 15, 2026
Correction to my inner-split comment. I wrote that
verify_guest_bins_archivedecompresses "about 62 MB of binaries". That was wrong: I took 62 MB from a cache directory size that included other files. Measured on the installed rc.2 archive, it is 19.8 MB compressed and 42.4 MB uncompressed, in 77 members. I also wrote that this "matches the roughly 54 MB hashed per launch the CI runner reported". With 42.4 MB it does not match, so please disregard that link; I do not know what the runner's 54 MB is made of. And I said the VM's CPU lacks SHA instructions as the reason a call takes about 420 ms; I did not check the CPU flags, so whether software SHA-256 explains part of it is not established. The span timings themselves (7 verifications per launch, about 640 ms each, 422 ms of that inverify_guest_bins_archive) are unaffected. ADR-112 (PR #5514) and the contract document are corrected.auser commentedon Nov 15, 2026
Correction to my earlier comment here (the M1 diagnostic data point): I suggested hashing speed or a missed remembered-digest hit was the first thing to check for
attach_overlay. That was a hypothesis and it is superseded: the #5508 spans on the idle GCP VM show the cost is the signed release proof being verified 7 times per launch, and hashing of the overlay and sidecar is about 0. The 54 MBartifact_bytes_hashedfigure counts onlysha256_readerbytes (it excludes the release archive'sSha256::digest), and the guest-runtime archive is 19.8 MB compressed / 42.4 MB uncompressed, so the earlier arithmetic should not be relied on. Design for the fix: #5515.