Repository navigation
nvx-rs: derive the compatibility keys of templates - #879
Pedro Henrique Penna (ppenna) merged 3 commits into
Conversation
There was a problem hiding this comment.
🔵 Needs a closer look
The persistent compatibility encoding and hand-written manifest decoder need human confirmation of cross-platform behavior and long-term versioning assumptions.
No open issues 🎉
What changed in this PR
Adds metadata-only template compatibility keys to nvx-rs, establishing the first part of #839’s template-store foundation on top of #871’s activation records. Template capture, storage, and restore remain out of scope.
Changes:
- Introduces capture classes, compatibility keys, canonical encodings, and domain-separated identities.
- Adds strict manifest decoding, OpenVMM-generated fixtures, and compatibility conformance tests.
- Centralizes restore contract versions and extends checks against the OpenVMM submodule.
| File | Description |
|---|---|
nvx-rs/tests/vectors/README.md |
Documents template-key vectors and contract checks. |
nvx-rs/tests/openvmm_vectors.rs |
Checks restore contract versions against OpenVMM. |
nvx-rs/tests/fixtures/template/README.md |
Documents manifest fixtures and regeneration. |
nvx-rs/tests/fixtures/template/generate.rs |
Generates fixtures using OpenVMM’s writer. |
nvx-rs/src/lib.rs |
Documents the new template and contract APIs. |
nvx-rs/src/host/template/wire.rs |
Implements strict protobuf wire decoding. |
nvx-rs/src/host/template/test_support.rs |
Supplies manifest and capture-class test builders. |
nvx-rs/src/host/template/mod.rs |
Exposes template APIs and documents encoding rules. |
nvx-rs/src/host/template/manifest/tests.rs |
Tests manifest validation and safe file reading. |
nvx-rs/src/host/template/key/tests.rs |
Tests key derivation, identities, and conformance vectors. |
nvx-rs/src/host/template/key.rs |
Defines capture classes and canonical compatibility keys. |
nvx-rs/src/host/template/canonical.rs |
Implements bounded canonical encoding and decoding. |
nvx-rs/src/host/mod.rs |
Exports and documents the template module. |
nvx-rs/src/guest/time/decoders.rs |
Uses the shared restore-packet version constant. |
nvx-rs/src/contract.rs |
Defines snapshot and restore contract versions. |
nvx-rs/README.md |
Documents template metadata and validation. |
doc/ci.md |
Describes the expanded OpenVMM consistency checks. |
.github/workflows/ci.yml |
Updates comments describing existing checks. |
.gitattributes |
Preserves manifest fixtures as binary files. |
Review effort: Balanced
dc4517f to
3032e6d
Compare
8ef07da to
01aa995
Compare
There was a problem hiding this comment.
🔵 Needs a closer look
The handwritten manifest decoder and persistent compatibility encoding need domain review for parity with OpenVMM’s schema and restore semantics because the pinned submodule sources are unavailable.
No open issues 🎉
Review effort: Balanced
01aa995 to
ff7ddec
Compare
fa45e48 to
f341f09
Compare
A template's compatibility key must hold the version of every contract through which its clones are restored. A runtime identity records the boot and control versions, but none of those of OpenVMM's snapshot format, the schema of the saved device state, the time ABI, and the restore packet. Add them to nvx::contract with the values of the pinned OpenVMM, and derive the guest's restore-packet decoder version from its constant. The ignored test of tests/openvmm_vectors.rs, which the NVX CLI jobs run with the submodule checked out, now also checks each version against the constant in OpenVMM's source that it must match. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: bc335726-838c-4538-a3c4-d6791aba5101
A template is a capture of a microVM from which independent clones are restored. Which clones it can serve, and on which hosts, depends on small metadata: the runtime that captured it, the contracts through which its clones are restored, and the machine that they restore. Stores, adapters, and restores need one key that holds exactly that metadata, which they compare, and whose canonical bytes and identity never vary between hosts or builds. Add nvx::host::template: - TemplateManifest::read reads the manifest that OpenVMM wrote for a capture, at most 1 MiB of a regular file that is not a link or other reparse point, through a strict reader of the protobuf wire format that refuses unknown, mistyped, repeated, and truncated fields. It refuses a capture that is not a template's: a platform-tier capture for clones, with a read-only distro whose identity is unbound, a fresh scratch, and the dormant filesystem slot. - CaptureClass holds the invariant inputs of a capture, known before it, and TemplateKey::new checks that the capture made the machine that its class requested before it derives the key from metadata alone. It never reads or digests an artifact. - A key and a class each have one canonical, versioned encoding, and their identities are domain-separated SHA-256 digests of it. TemplateKey's from_bytes refuses any other encoding. Besides the versions of the snapshot contracts in nvx::contract, a key holds the version of the activation contract, nvx::activation::VERSION, and a key's identity converts without loss to and from the nvx::activation::Identity by which a clone's activation record names its template. The fixtures are manifests that OpenVMM's own writer encoded, so the tests check the reader against OpenVMM's encoding and not only against the test encoder. The vectors of tests/vectors/template-key-v1.txt, which an independent encoder wrote, fix version 1 of the encoding. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: bc335726-838c-4538-a3c4-d6791aba5101
The row of nvx::host::template ended mid-sentence: the conformance vectors of version 1 pin the key's encoding and identity. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: bc335726-838c-4538-a3c4-d6791aba5101
ff7ddec to
0a9fdce
Compare
| @@ -0,0 +1,988 @@ | |||
| //! Builders of the manifests and capture classes of templates, for their tests. | |||
| //! | |||
| //! [`Proto`] writes the protobuf wire format as OpenVMM's `mesh` does, independently of the reader | |||
There was a problem hiding this comment.
This is not actually using protobuf, we should create an issue to streamline the way we are serializing data in NVX, we should have a single way instead of multiple ones scatter around the repos, potentially using a library, let's try to converge to a single canonical way of serializing.
Based on
dev, where #866, #882, #883, #876, #871, #873, and #874 have merged. Stacked above it: #881 → #885 → #886 → #884 → #887 → #888.Which clones a template can serve, and on which hosts, depends on small metadata: the runtime that captured it, the contracts through which its clones are restored, and the machine that they restore. Stores, adapters, and restores need one key that holds exactly that metadata, whose canonical bytes and identity never vary between hosts or builds. This adds
nvx::host::template: the reader of a capture's manifest, the invariant inputs of a capture, and the compatibility key that they derive, from metadata alone. It changes no behavior: nothing captures, stores, or restores a template yet.Part of #839: the first of its four pull requests. The second adds the template store, the third eligibility, leases, tombstones, and retirement, and the fourth the supervisor seam and the lock order.
New public constants
nvx::contractgains the versions of the contracts through which OpenVMM restores a snapshot, which no runtime identity records, each apub const ...: u64besideBOOT_CONTRACT_VERSION,CONTROL_SESSION_PROTOCOL_VERSION, andMICROVM_ABI_VERSION:SNAPSHOT_FORMAT_VERSIONMANIFEST_VERSIONinopenvmm/openvmm_helpers/src/snapshot.rsDEVICE_STATE_SCHEMA_VERSIONSAVED_STATE_SCHEMA_VERSIONinopenvmm/openvmm_helpers/src/snapshot/format.rsTIME_ABI_VERSIONTIME_ABI_VERSIONinvmm_core/virt/src/time_abi/mod.rsRESTORE_PACKET_VERSIONRESTORE_PACKET_VERSIONinvm/devices/chipset_resources/src/microvm_time.rsThe ignored test of
tests/openvmm_vectors.rs, which the NVX CLI jobs run with the submodule checked out, now also checks each against the submodule's source. The guest's restore-packet decoder takes its version fromRESTORE_PACKET_VERSION.What changed
The manifest (
manifest.rs,wire.rs).TemplateManifest::readreads themanifest.binthat OpenVMM wrote for a capture: at mostMAX_MANIFEST_LEN(1 MiB) bytes of a regular file that is not a link or other reparse point, through a strict reader of the protobuf wire format that refuses unknown, mistyped, repeated, and truncated fields. It refuses a manifest of another snapshot format, saved-state schema, machine ABI, boot layout, or time ABI, and a capture that is not a template's: a platform-tier capture for clones, with a read-onlydistrowhose identity is unbound, a fresh scratch, and the dormant filesystem slot. It reads the time contract,TimeContract, and never reads another artifact of the snapshot.The class and the key (
key.rs).CaptureClassholds the invariant inputs of a capture, known before it: the runtime, the guest's build, the backend, the requested CPU profile, the processors and RAM, the geometries of the disks, the network, and the adapter's otherInvariants.TemplateKey::newchecks that the capture made the machine that its class requested, and derives the key, which holds, and nothing else:nvx::activation::VERSION, widened to au64.distroand of the scratch.A capture's own metadata stays out of the key, so the generations of one key differ in it: the capture time, the time contract's anchors, host and boot IDs, and lineage counter, and the artifacts' lengths.
TemplateManifestkeeps them.The encoding (
canonical.rs). A key and a class each have one canonical, versioned encoding, which the module documents value by value. Their identities, aKeyIdentityand aClassIdentity, are SHA-256 digests of it after a domain of their own.TemplateKey::from_bytesreads a key back, and refuses any other encoding and bytes left after it. AKeyIdentityconverts to and from thenvx::activation::Identityby which a clone's activation record names its template's key, without loss.Fixtures and vectors.
tests/fixtures/template/holds two manifests that OpenVMM's own writer encoded, and the program that wrote them, so the tests check the reader against OpenVMM's encoding and not only against the tests' encoder.tests/vectors/template-key-v1.txtfixes version 1 of the encoding: keys, classes, keys of other versions, and invalid keys, which an independent encoder wrote from the documented encoding.Assumptions to review
Invariants: named settings, at most 64, in the order of their names, which the class and the key hold as they are.ClassIdentity, so that the generations of one class can be found before their keys are known.TemplateKey::newleaves the identity of the network policy to OpenVMM, which derives it from the class's network arguments by an encoding of its own. A capture that enforced another policy than its class requested makes a key of its own, which no restore with the class's arguments serves: OpenVMM refuses a restore whose network, its policy included, is not the one that the snapshot records.TemplateManifest::readrefuses a link at the manifest itself. It names the manifest by its path, so the snapshot's directory and those above it must be ones that only the user, or an administrator, may change, as the sandbox store requires of its own.Validation
At
d74181e, rebased ondev'sb57760b, on Windows and on Linux x86_64 (the head,0a9fdce, ondev'sce8c7d6, differs from it only indev's performance baselines):cargo fmt --check;cargo clippywith--all-targets --all-features,--lib, and--lib --no-default-features,-D warnings;cargo testwith all, default, and no features, among themtests/host_modules.rs, the layering ofnvx::host;cargo docwith-D warnings, also of private items on Windows;cargo +1.89 check;tests/openvmm_vectors.rswith--ignored, against the OpenVMM submodule at its pin,e68871c;nvx, against a synthetic bundle;doc/usage.mddocuments every error code thatnvx-rs/README.mdlists, and the Markdown links;cargo fmt --check,cargo clippywith--all-targets --all-features,--lib, and--lib --no-default-features,-D warnings, andtests/host_modules.rs, and, on Linux, the macOS check.Every check passed on both hosts.
A final review by a separate model found three issues, which the head addresses: an expansion range now refuses its file offset, field 3, even when it is zero, as an unknown field; and the documentation now states that
TemplateManifest::readrefuses a link at the manifest itself, and thatTemplateKey::newleaves the network policy's identity to OpenVMM, as above.The restack on #874 resolved conflicts only in
nvx-rs/README.mdandtests/vectors/README.md, where #871 to #874 anddevchanged the same lines. It keeps their text, such as the activation entries and the new listing oftests/, and adds this pull request's lines to it. It changed no code. A final review of the restack by a separate model found no issues.A second restack, on #874's
fa45e48, which drops the configuration identity of activation records, again resolved conflicts only innvx-rs/README.mdandtests/vectors/README.md, in this pull request's rows next to #874's activation entries: it keeps #874's new text and this pull request's rows. It changed no code, and the template key's vectors are unchanged, sincenvx::activation::VERSIONis still 1. A final review of it by a separate model found no issues.The rebase onto
dev, after #874 merged, resolved conflicts only in the crate tree ofnvx-rs/README.mdand the module documentation ofnvx::host, keepingdev's text and adding this pull request's; it changed no code,tests/host_modules.rsplacesnvx::host::templatein the top layer ofnvx::host, without a cycle, the template key's vectors are unchanged, and a final review of it by a separate model found no issues.