Repository navigation
nvx-rs: retire the Python-parity golden tests and make Rust the reference #722
Description
Goal
nvx.py is being retired, so the nvx library and the nvx-rs CLI become the reference for the behavior that #359 ported from Python, rather than the Python code that #359 deleted. Replace the golden tests that pin Python's behavior with tests whose expectations come from NVX's documented behavior, then delete the recorded fixtures and the CPython emulation in nvx::host::python. Keep the golden byte tests of the wire formats, as versioned conformance vectors, because they are the only check against implementations that share no code with nvx-rs.
Background
The retired scripts/nvx_rs_fixtures.py recorded what CPython 3.14 did on POSIX and Windows hosts. #359 deleted it with the code that it recorded, so its records are frozen and are edited by hand when behavior changes:
Fixture in nvx-rs/tests/fixtures/ |
Size | Cases | Checked by |
|---|---|---|---|
python.json |
431 KiB | 822 Unicode ranges, and 2,731 cases of repr, str.split(), str.strip(), int(), uuid.UUID, pathlib, json.loads, and UTF-8 decoding |
src/host/python/golden.rs |
sandbox.json |
459 KiB | 1,412 identities, layers, mount values and targets, child names, mounts, and launches | src/host/sandbox/golden.rs |
sandbox_options.json |
283 KiB | 885 share targets, mounts, exec environments, environment files, and distro manifests | src/host/sandbox/golden.rs |
egress-policy.json |
1,112 KiB | 784 policies, 272 of which compile, 2 unreadable files, and 55 paths | src/host/network/policy/tests.rs |
cli.json |
365 KiB | 141 POSIX and 140 Windows scenarios of nvx.py sandbox, egress compile, and artifacts, in 426 steps |
tests/cli.rs |
At 8b585d7, on Windows, with branch coverage (Linux isn't measured yet):
| Tests | Lines | Branches |
|---|---|---|
| All | 90.21% | 78.02% |
| All but the Python-parity tests | 86.97% | 72.89% |
Only the Python-parity tests reach 546 lines and 147 branches of production code, and the CLI replay reaches 533 and 130 of them. The wire-format byte tests reach no line that other tests miss: what they add is an expectation that doesn't come from the codec that the host, the guest agent, and the fake OpenVMM share.
Decisions
- No new Python-recorded fixtures or scenarios. Until a family is retired, a change may edit its records by hand, but adds no recorded scenarios or fixture files.
- Errors: exit statuses, stable machine-readable error codes (nvx-rs: give CLI errors stable machine-readable codes #724), and stable fragments, such as option names and field paths, are the contract. Message wording may change.
- Accepted input is strict: RFC 8259 JSON without
NaN,Infinity, or duplicate keys; ASCII decimal integers; canonical UUIDs. - Values that become kernel command-line tokens refuse the bytes that the kernel treats as separators or quotes (nvx-rs: kernel command-line checks split parameters differently from Linux #723).
- Host paths that aren't valid Unicode are refused.
- Older
nvx.pyandnvx-rsbuilds need not operate sandboxes that newer builds provisioned:nvx-rskeeps readingconfig.jsonformats 1 to 6, and writes only the newest. - Property tests use
proptest. - Conformance vectors live in
nvx-rs/tests/vectors/, and the protocol design documents link to them. - Callers use
nvx-rsinstead ofnvx.py sandboxandnvx.py run(nvx.py: retire the sandbox and run commands in favor of nvx-rs #725), sonvx-rsdrops--progand its emulation ofargparse's layout.
Plan
The sub-issues are the steps, and their blocked-by relationships record what each step needs.
#728, #729, and #730 take two pull requests each: the first adds the replacement tests without changing behavior, and the second deletes the fixture and makes the changes that the decisions call for. Only the gate of the second needs #726.
Gates of each deletion
- Coverage: every line and branch that only the deleted tests reach is reached by the remaining tests on Linux and Windows, or is deleted in the same change. The pull request lists any exception.
- Mutation testing:
cargo mutantson the family's files misses no mutant with the new tests that the deleted tests catch, except in code that the change deletes. - Traceability: every recorded scenario or fixture section maps to a test, and so do the regressions of Track nvx.py managed sandbox bugs: PID-only liveness check wedges stop, deprovision, and start #369, Track managed sandbox bugs: --dry-run performs managed operations, and layered exec --cwd leaves PWD at / #373, Track egress policy compiler bugs: scoped IPv6 validation, range fragmentation #462, Track managed sandbox lifecycle bugs: on Windows, provision waits out its timeout for a lock file that refuses access for good #464, and Track Rust sandbox exec bugs: async output collection bypasses timeouts #513.
Kept
- The wire-format vectors and constants, the C agent's parity constants while a supported release ships the C agent, and the
NVXLannouncement vectors (nvx-rs: turn the wire-format golden bytes into versioned conformance vectors #727). - The store record fixtures in
tests/fixtures/store/: hand-written records of a versioned on-disk format (nvx-rs: evaluate the uuid crate for launch identities and temporary file names #646). - The byte assertions of
tests/guest_tools.rs: the README makes the guest binaries' command lines, exit statuses, and byte output compatibility contracts.
Related work
- nvx-rs: make the nvx-rs CLI the backend of nvx.py run #638 plans to record
nvx.py runscenarios from Python and requires the golden scenarios to pass, and nvx-rs: share the VMM supervisor and the sandbox store #620 and nvx-rs: host cold integration for image-backed sandboxes #622 require them too. Under these decisions, their scenarios become tests of nvx-rs: replace the golden CLI scenarios with structured scenario tests #728. - nvx-rs: prove shared host and guest mechanics through two independent adapters #545 asks for conformance vectors, nvx-rs: a shared credit and stdio engine for executions #623 plans vectors for exec v2, and nvx-rs: final acceptance for the image-backed runtime #624 asks for a coverage manifest. nvx-rs: turn the wire-format golden bytes into versioned conformance vectors #727 and the traceability tables serve them.
- Open pull requests that edit the fixtures: nvx-rs: separate distro verification from launch planning #653, nvx-rs: prepare the CLI's move onto the store and supervisor #675, nvx-rs: plan sandbox launches with the image format and upper #676, and nvx-rs: make the nvx-rs CLI the backend of nvx.py run #704. nvx-rs: check the golden tests' fixture sections with one function #683, which refactors the golden-test helpers, can merge; the helpers go in nvx-rs: replace the golden CLI scenarios with structured scenario tests #728 to nvx-rs: remove the CPython emulation in nvx::host::python #731.
Done when
- The Python-parity fixtures and
nvx::host::pythonare gone. - The README's "Python parity" and "Golden scenarios" sections describe the tests that replace them.
-
doc/contribute.mdstates the testing policy: a deliberate change of behavior updates a test derived from the specification, never recorded output, and a conformance vector changes only with a protocol version. - Whether the coverage workflow of nvx-rs: measure line, branch, and mutation coverage on Linux and Windows #726 keeps running on a schedule is decided.
Activity
ci: run cargo-mutants on files of nvx-rs on demand
ci: measure what each golden-test family of nvx-rs alone reaches
ci: run cargo-mutants on files of nvx-rs on demand
ci: measure what each golden-test family of nvx-rs alone reaches
41 remaining items
scripts: stop measuring the coverage of the CLI scenarios
nvx-rs: retire the golden CLI scenarios
scripts: retire the nvx-rs coverage workflow
nvx-rs: quote values with Rust's debug formatting
nvx-rs: read the CLI's integers as ASCII decimals in explicit ranges
nvx-rs: read and write the CLI's JSON with serde_json
scripts: retire the nvx-rs coverage workflow
nvx-rs: quote values with Rust's debug formatting
nvx-rs: read the CLI's integers as ASCII decimals in explicit ranges
nvx-rs: read and write the CLI's JSON with serde_json
scripts: retire the nvx-rs coverage workflow
nvx-rs: move the README's sandbox specification tests after its other…
doc: state the testing policy that replaced recorded output
nvx-rs: describe the tests that replaced the golden tests