From 7029de368b5a2e0ccfd1790a23bbbcde066bfe1a Mon Sep 17 00:00:00 2001 From: Ari Date: Sat, 14 Nov 2026 16:21:10 -0800 Subject: [PATCH 1/2] docs: ADR-112, verify the signed runtime release proof once per launch A cold launch of the signed v0.28.0-rc.2 release takes about 5.6 s on an idle machine, against a 300 ms gate. About 4.5 s of it is the signed guest-runtime archive being verified from scratch seven times per launch, at about 640 ms each: read and digest, two signature checks, a temporary copy, and decompressing and SHA-256ing all of its members. The cost is deterministic, which is why it looked like load. The ADR records the measurements and what was ruled out, decides to verify once per launch (scoped to a launch, not the process, so a long-lived embedder still verifies at every launch), lists the conditions that keep it the same verification, the alternatives including the persisted record that is left for a separate decision, and the tests the implementation needs. It also records how to repeat the measurement on a temporary cloud VM and clean it up. The contract document's launch-cost section now states the cause and links to the ADR instead of saying it was not established. --- ...e-runtime-release-proof-once-per-launch.md | 236 ++++++++++++++++++ .../host-interface-distributed-execution.md | 48 ++-- 2 files changed, 259 insertions(+), 25 deletions(-) create mode 100644 specs/adrs/112-verify-the-runtime-release-proof-once-per-launch.md diff --git a/specs/adrs/112-verify-the-runtime-release-proof-once-per-launch.md b/specs/adrs/112-verify-the-runtime-release-proof-once-per-launch.md new file mode 100644 index 0000000000..b3928b0e51 --- /dev/null +++ b/specs/adrs/112-verify-the-runtime-release-proof-once-per-launch.md @@ -0,0 +1,236 @@ +# ADR-112 — Verify the signed runtime release proof once per launch + +Backing: preview +Validation: none + +**Status: Proposed.** Direction chosen on 2026-11-14: verify once. Review of the +claim-20 consequences, below, is still to be done before any code lands. + +## Summary, for a reader in a hurry + +- A launch of the signed `v0.28.0-rc.2` release takes about 5.6 s on an idle + machine. The launch gates are 300 ms for a cold launch and 50 ms for a warm + claim, so this is about 19 times over the cold gate. +- About 4.5 s of that, 80%, is one check repeated seven times: the signed + guest-runtime archive is verified from scratch seven times per launch, at about + 640 ms each. Nothing else in a launch is large, and the guest itself is ready + in about 12 ms. +- The cost is deterministic. It does not depend on load, the disk or the + network, so it was invisible in noise and looked like "a slow machine". +- Verifying once per launch removes about 3.9 s of it. A launch-scoped change + that only caches the single costliest step, tried as a throwaway on an idle + VM, already took the launch from 5.58 s to 3.03 s. +- This does not reach 300 ms on its own. It is the largest single step toward + it, and it is what makes the remaining costs visible. +- What changes is when a signed artifact is checked, which is claim 20 territory, + so the exact conditions are written down below and need review. + +## Context + +### What a launch costs today + +Measured on a temporary Google Cloud VM with nothing else running (`n2-standard-4`, +Ubuntu 24.04, SSD, load average under 1, KVM, Firecracker), with the signed +`v0.28.0-rc.2` release installed by the repository installer, a cold launch of a +cached 36 MB OCI image (`MVM_RESIDENCY=cold`). Five runs agreed to within 1%. A build of the same tree with the timing spans, used +for the breakdown below, took about 5.6 s per launch; the figures in this ADR that +say "5.6 s" refer to that build, and the installed release is slower still. + +| Phase | Time | Notes | +| --- | --- | --- | +| `admit_plan` | 2,967 to 2,977 ms | inside admission | +| `attach_overlay` | 1,721 to 1,729 ms | inside admission | +| `attach_initramfs` | 868 to 879 ms | | +| `backend_start` | 1,529 to 1,556 ms | includes two more proof checks | +| guest ready | 11.7 to 13.7 ms | the guest is not the cost | +| teardown | 86 to 94 ms | | +| total | 7,225 to 7,260 ms | the installed signed release | + +The same phases on the developer Mac (HVF, the same signed release, under heavy +load from other builds) were `admit_plan` 1.7 to 2.7 s, `attach_overlay` 1.0 to +1.2 s and `vmm_create` 1.2 to 1.4 s, so the cost is not specific to one backend. +The physical M1 CI runner, which runs a source-built runtime, shows `admit_plan` +26 ms and `attach_overlay` 220 ms: it never takes the path described here. + +### Where the 4.5 s goes + +With the timing spans from #5508 and #5506 built into the exact release tree, +every cold launch showed the same thing, to within 1%: + +| Span | Calls per launch | Total | Each | +| --- | --- | --- | --- | +| `verify_release_archive` | 7 | 4.50 s | 643 ms | +| `verify_runtime_release_proof` | 6 | 3.85 s | 641 ms | +| `verify_runtime_for_launch_at` | 4 | 2.56 s | 640 ms | +| `verify_assurance_guest_protocol` | 2 | 1.28 s | 641 ms | +| `runtime_lifecycle.bind` | 1 | 966 ms | contains about 1.5 verifications | + +The callers are `acquire_runtime_overlay_for_launch` (twice), +`attach_universal_initramfs_if_cached`, `verify_runtime_for_launch_at` (four times, +at admission and again at dispatch) and the lifecycle bind. They each ask the +same question of the same archive and each pay for the whole answer. + +One verification (643 ms) is made of: + +| Step | Each | Share | +| --- | --- | --- | +| `verify_guest_bins_archive`, its own time | 422 ms | 66% | +| `read_and_digest_archive` | 107 ms | 17% | +| the two signature checks (`verify_release_archive_bytes`) | 105 ms | 16% | +| `write_archive_snapshot`, a temporary copy | 8 ms | 1% | + +`verify_guest_bins_archive` opens the guest-runtime `.tar.gz`, decompresses all of +it (about 62 MB of binaries) and SHA-256s every member, then checks the manifest +against those digests. The CPU on the measurement VM has no SHA instructions, so +that costs about 420 ms; Apple silicon has them, so each call is cheaper on a Mac, +but the count is the same. It also explains the seven temporary files of about +20 MB that a launch writes: one copy of the archive per verification. + +### What is not the cause + +Each of these was measured and ruled out on the machines named above. + +- CPU contention: a fixed 0.3 CPU-second task took 0.33 s of wall time at a load + average of 90 on 16 cores. +- Signature verification alone: `mvmctl env verify-release` on the installed + archive took 0.2 s including process start. +- The macOS Keychain: a lookup took about 0.1 s. +- Hashing the image: about 0.1 s in total. +- The size of the audit chain: `admit_plan` was the same on homes holding 177 and + 1,518 entries. +- File writes: a 20 MB copy took under 5 ms. +- Helper processes: only two are started, at the very end. +- Network: a system-call trace of a launch shows no outbound connection. +- Signing, audit and key load: `sign_and_verify_plan` 0.3 ms, + `audit_record` 4.8 ms, `host_signing_key_load` 0.2 ms. + +### Why the cost was hard to see + +It is deterministic. A launch is not waiting on anything; it repeats the same +fixed work, so it costs the same every time on every host of a given speed. It +therefore looked like noise-free "slow hardware" and was attributed to load. The +spans from #5508 and #5506 are what exposed the repetition. + +## Decision + +Verify the signed runtime release proof **once per launch** and share that one +result among every step of that launch that needs it. + +The conditions that make this the same verification, not a weaker one: + +1. **Scope is one launch, not the process.** `mvmctl` starts a new process per + launch, but `mvm-client` and `mvm-hostlib` run in one long-lived process that + starts many VMs. A process-wide cache would turn "verified at every launch" + into "verified at the first launch". The shared result is therefore created at + the start of a launch and dropped at its end, passed through the launch + context or held in a launch-scoped value, never in a `static`. +2. **The shared result owns the verified bytes.** `VerifiedRuntimeManifest` + already owns a private snapshot of the exact signed archive bytes, so that + later use reads what was verified. The shared value keeps that ownership; it + is shared by reference (`Arc`), not re-derived from a path. +3. **The key covers everything the verification depends on:** the release + version, the guest architecture and the identity of the archive, checksum and + bundle files (path, size, modification time and the archive digest). If any of + them differs, the step verifies again. +4. **Failures are not remembered.** A refusal on the first call is the refusal + for that launch. +5. **Nothing is persisted.** No record is written for a later process to trust. +6. **Everything else is unchanged:** the overlay and verity digests are still + compared with the declaration, the plan is still signed and verified, and the + dispatch-time re-check still happens; it reuses the verified result instead of + repeating the archive work. +7. **`MVM_SKIP_COSIGN_VERIFY` keeps its present meaning** for each path that + honours it today, and the Firecracker launch proof remains non-waivable. + +This gives a launch of about 1.7 s on the idle VM (projected from the spans: +5.58 s minus about 3.9 s), against 5.6 s. + +## What this does not do + +- It does not reach 300 ms. After it, the launch is dominated by about 640 ms for + the one verification, about 320 ms in `grants_budget_and_host_interface`, about + 300 ms in `driver_boot`, and teardown. Each is a separate piece of work. +- It does not make a warm claim faster than a cold boot through the CLI. A warm + claim costs the same admission and attach work, which is what this removes. +- It does not change which signatures are accepted or by whom. + +## Alternatives considered + +- **Persist a verification record across launches** (signed by the host key and + keyed by the archive digest, like the source-runtime declaration cache). This + would take the launch to about 1.1 s. It is a cached authorization, and the + repository's rule is to reuse verified bytes and never cached authorization, + so it needs its own decision and a claim note. Not chosen here. +- **Cache only the member check (`verify_guest_bins_archive`) by archive digest.** + Tried as a throwaway on the idle VM: the launch went from 5,574 to 5,583 ms to + 3,032 to 3,038 ms, a saving of 2.55 s, repeatable to under 0.2%. It leaves six + extra reads, six extra signature checks and six extra temporary copies, so it + saves about 65% of what the decided design saves. It is the fallback if the + review wants a smaller change first. The throwaway used a process-wide cache, + which is not what is proposed; see condition 1. +- **Make one verification cheaper** with hardware SHA-256, by not decompressing + twice, or by trusting the signed manifest's member digests once the archive + digest is authenticated. Each changes what is checked and is a separate review. +- **Do nothing.** The launch stays at about 5.6 s, and no launch gate can pass. + +## Consequences + +- Claim 20's guest-runtime path stays "verified at every launch": every launch + still performs the full archive check, once. The ADR-001 row and the boundaries + note need a sentence recording that the check is shared within a launch. +- A launch writes one temporary archive copy instead of seven, which also + narrows the window in which a copy could be tampered with. +- The change touches `crates/mvm-build/src/guest_runtime/evidence.rs`, + `crates/mvm-client/src/launch/runtime_overlay.rs`, + `crates/mvm-runtime/src/runtime_evidence.rs` and + `crates/mvm-runtime/src/universal_initramfs.rs`, so it conflicts with in-flight + spans work in the same files. + +## What the implementation must test + +1. A second request for the same archive in one launch performs no second + archive read, hash or signature check (assert the number of verifications). +2. A different version, architecture, archive, checksum or bundle in the same + launch is verified again. +3. A launch whose archive or bundle changes on disk between two steps verifies + again, and refuses if it no longer verifies. +4. A refusal is not remembered: after a failing first call, the next call refuses + too. +5. Two launches in one process verify twice, once each. +6. A tampered archive still refuses at the first call, with the same error and the + same audit entry as before. +7. Every claim-20 witness named in the ADR-001 ledger still passes unchanged. + +Acceptance, measured on an idle machine: `verify_release_archive` appears once in +the span table, and a cold launch of the cached image falls from about 5.6 s to +about 1.7 s. + +## How this was measured + +The spans are on `main` (`MVM_SPAN_TIMINGS=1`, and `MVM_PHASE_TIMING=1` or +`MVM_LAUNCH_SAMPLE_JSON=` for the named laps). #5506 adds the inner steps of +the proof verification. + +To repeat it on a clean machine: + +1. Create a temporary VM that deletes itself, for example + `gcloud compute instances create --zone= --machine-type=n2-standard-4 --enable-nested-virtualization --min-cpu-platform="Intel Cascade Lake" --image-family=ubuntu-2404-lts-amd64 --image-project=ubuntu-os-cloud --boot-disk-size=60GB --boot-disk-type=pd-ssd --labels=purpose=