Skip to content

nvx-rs: retire the Python-parity golden tests and make Rust the reference #722

Closed
7 of 7 issues completed

Description

Goal

nvx.py is being retired, so the nvx library and the nvx-rs CLI become the reference for the behavior that #359 ported from Python, rather than the Python code that #359 deleted. Replace the golden tests that pin Python's behavior with tests whose expectations come from NVX's documented behavior, then delete the recorded fixtures and the CPython emulation in nvx::host::python. Keep the golden byte tests of the wire formats, as versioned conformance vectors, because they are the only check against implementations that share no code with nvx-rs.

Background

The retired scripts/nvx_rs_fixtures.py recorded what CPython 3.14 did on POSIX and Windows hosts. #359 deleted it with the code that it recorded, so its records are frozen and are edited by hand when behavior changes:

Fixture in nvx-rs/tests/fixtures/ Size Cases Checked by
python.json 431 KiB 822 Unicode ranges, and 2,731 cases of repr, str.split(), str.strip(), int(), uuid.UUID, pathlib, json.loads, and UTF-8 decoding src/host/python/golden.rs
sandbox.json 459 KiB 1,412 identities, layers, mount values and targets, child names, mounts, and launches src/host/sandbox/golden.rs
sandbox_options.json 283 KiB 885 share targets, mounts, exec environments, environment files, and distro manifests src/host/sandbox/golden.rs
egress-policy.json 1,112 KiB 784 policies, 272 of which compile, 2 unreadable files, and 55 paths src/host/network/policy/tests.rs
cli.json 365 KiB 141 POSIX and 140 Windows scenarios of nvx.py sandbox, egress compile, and artifacts, in 426 steps tests/cli.rs

At 8b585d7, on Windows, with branch coverage (Linux isn't measured yet):

Tests Lines Branches
All 90.21% 78.02%
All but the Python-parity tests 86.97% 72.89%

Only the Python-parity tests reach 546 lines and 147 branches of production code, and the CLI replay reaches 533 and 130 of them. The wire-format byte tests reach no line that other tests miss: what they add is an expectation that doesn't come from the codec that the host, the guest agent, and the fake OpenVMM share.

Decisions

  • No new Python-recorded fixtures or scenarios. Until a family is retired, a change may edit its records by hand, but adds no recorded scenarios or fixture files.
  • Errors: exit statuses, stable machine-readable error codes (nvx-rs: give CLI errors stable machine-readable codes #724), and stable fragments, such as option names and field paths, are the contract. Message wording may change.
  • Accepted input is strict: RFC 8259 JSON without NaN, Infinity, or duplicate keys; ASCII decimal integers; canonical UUIDs.
  • Values that become kernel command-line tokens refuse the bytes that the kernel treats as separators or quotes (nvx-rs: kernel command-line checks split parameters differently from Linux #723).
  • Host paths that aren't valid Unicode are refused.
  • Older nvx.py and nvx-rs builds need not operate sandboxes that newer builds provisioned: nvx-rs keeps reading config.json formats 1 to 6, and writes only the newest.
  • Property tests use proptest.
  • Conformance vectors live in nvx-rs/tests/vectors/, and the protocol design documents link to them.
  • Callers use nvx-rs instead of nvx.py sandbox and nvx.py run (nvx.py: retire the sandbox and run commands in favor of nvx-rs #725), so nvx-rs drops --prog and its emulation of argparse's layout.

Plan

The sub-issues are the steps, and their blocked-by relationships record what each step needs.

#728, #729, and #730 take two pull requests each: the first adds the replacement tests without changing behavior, and the second deletes the fixture and makes the changes that the decisions call for. Only the gate of the second needs #726.

Gates of each deletion

Kept

Related work

Done when

  • The Python-parity fixtures and nvx::host::python are gone.
  • The README's "Python parity" and "Golden scenarios" sections describe the tests that replace them.
  • doc/contribute.md states the testing policy: a deliberate change of behavior updates a test derived from the specification, never recorded output, and a conformance vector changes only with a protocol version.
  • Whether the coverage workflow of nvx-rs: measure line, branch, and mutation coverage on Linux and Windows #726 keeps running on a schedule is decided.

Activity

  1. 41 remaining items

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

No labels
No labels

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions