Skip to content

nvx-rs: validate GPT disks against the kernel's partitions in the guest - #715

Closed
Enrique Saurez (esaurez) wants to merge 8 commits into
devfrom
esaurez/a3-gpt-device-validation-20261020
Closed

Enrique Saurez (esaurez) wants to merge 8 commits into
devfrom
esaurez/a3-gpt-device-validation-20261020

Conversation

@esaurez

@esaurez Enrique Saurez (esaurez) commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Adds nvx::guest::gpt, which validates a GPT disk that the guest's kernel attached as a block device against the partitions that the kernel created from its table. It reads the table with nvx::gpt (#673) and checks that the kernel's partitions are exactly the table's; only when they differ does it ask the kernel to reread the table, and wait until they match. A guest therefore never uses a partition that the table doesn't describe: not when the kernel read another table, such as an MBR, in its place, and not when the table changed after the guest read it. Resolving the disk by its role and mounting its layers come in later pull requests.

What changed

  • BlockDevice::new(&File) accepts only a whole disk: a block device whose sysfs directory, /sys/dev/block/MAJOR:MINOR, describes the same device number and has no partition attribute. It reads the device's logical block size (BLKSSZGET) and size (BLKGETSIZE64) through ioctls that nvx::guest::sys now wraps, beside the reread (BLKRRPART) and open_at. The module calls the kernel only through sys.
  • BlockDevice::validate(&Expected, timeout) runs these steps, which are also public for a caller that needs them apart:
    1. read_disk reads and checks the table and the caller's Expected values through the device, at its logical block size and size, as Disk::read describes.
    2. await_partitions, without waiting, compares the kernel's partitions, as sysfs lists them, with the table's: one for each entry, with its number, first byte, and size, and no other. Sysfs gives starts and sizes in 512-byte sectors whatever the logical block size, so the comparison is in bytes. The kernel created the partitions from the table when it found the disk, so at boot they match, and the validation ends here.
    3. Only when they differ, reread_partitions asks the kernel to reread the table, which it does before it returns, and await_partitions compares the partitions again until they match or the timeout passes. systemd's partition-table reread likewise leaves the partitions that already match alone.
  • Another program, such as a device manager that probes the disk, may make the kernel drop and create the partitions again at any time. The comparison therefore repeats after pauses that grow from 1 ms to 4 ms, until the partitions match or the timeout passes; the last difference then fails as Mismatch::Missing, Unexpected, or Extent, for the lowest partition number that differs, or as Changed if the partitions changed while they were read. The same wait bounds a caller whose partitions appear late.
  • A partition whose sysfs attributes vanish while they're read counts as gone, so the comparison tries again. Each partition's attributes are read through one descriptor of its sysfs directory, so that a partition that the kernel drops and creates again under its name while it's read is never described by a mix of the two. A malformed attribute, or a sysfs directory that can't be read, fails at once.
  • Each comparison also takes only partitions that the kernel held together. Otherwise a reread between two partitions could combine an old table's first partition with a new table's later partitions, which together could match the table when neither table does. After reading the partitions, the comparison lists the disk's sysfs directories again, which must be the same, and checks each directory again through the descriptor that the first reading opened: a partition's directory must still give its number, which it no longer does once the kernel drops the partition, even to create another under its name, and every other directory must still describe no partition, so that a partition that the kernel finished creating meanwhile isn't left out. A reread drops every partition before it creates any, so partitions that pass come from one reading of the table; a reading that changed counts as Changed, and the comparison tries again. In the unoptimized test program, a comparison of a three-partition disk takes about 56 µs on WHP and 41 µs on KVM, against 50 to 57 µs and 30 µs without the confirmation.
  • Built as optimized as the guest binaries, the validation of the scenario's read-only disk, as the kernel found it at boot, takes 0.73 to 0.85 ms on MSHV and 0.39 to 0.46 ms on WHP. The reread that it no longer needs there takes 1.29 to 1.42 ms on MSHV and 0.59 to 0.65 ms on WHP.
  • ValidatedDisk returns the table and, for each partition, its table entry, the kernel's name of its block device (such as vda2), and its device number: first_partition() for the configuration that the caller reads with Disk::read_first_partition, and lowers() for the layers.
  • nvx::gpt's Disk::p1, Disk::read_p1, and GptError::P1TooLarge, from nvx-rs: add a strict GPT reader for layered root disks #673, become Disk::first_partition, Disk::read_first_partition, and GptError::FirstPartitionTooLarge, so that no name or document uses the shorthand "p1". Nothing else calls them yet.
  • DeviceError names the device operation or the sysfs path that failed, the table's GptError, or the Mismatch.
  • The reread needs CAP_SYS_ADMIN, and fails while any partition of the disk is open. A validation whose partitions already match needs neither, so it also succeeds while a layer is mounted. Nothing is hotplugged: the disk is attached when the VM is created.
Limit Value
Device A whole disk; a partition fails with NotWholeDisk
Kernel partitions Numbers 1 through 255, compared in bytes
Wait One comparison; only when the partitions differ, a reread, then comparisons after pauses from 1 ms, doubling up to 4 ms, until the caller's timeout
Privileges CAP_SYS_ADMIN, for a reread

Running the privileged tests in a guest

  • nvx-rs/tests/gpt_device.rs holds nine tests that need root and a guest's disks, so they're ignored on a host. python3 scripts/nvx.py build-guest-tests builds them on Linux as one static musl test program, build/guest-tests/gpt-device (1.7 MB, without debug information).
  • The new gpt-device scenario of nvx.py test-microvm is an explicit scenario, like smp-lapic: it runs only when named, so neither the default suite nor its shards run it. The host writes GPT images with scripts/nvx_tools/gpt_image.py, attaches a read-only disk and a writable scratch disk, and shares the program and the scratch disk's images over virtio-fs. The guest copies the program to /tmp and runs it as root with --ignored. The release initramfs is unchanged.
  • The tests check that a crafted disk validates, with the first partition's bytes and bound, the partitions' names and device numbers, and the layers' contents read through their device nodes, and validates again after a reread; that expected values that differ fail; that only a whole disk validates; that a disk rewritten with another table validates after the reread, and rereads through a read-only descriptor too; that partitions that the kernel creates late are awaited; that missing, extra, and moved partitions fail; that a disk that the kernel reads as an MBR fails; and that an open partition fails a validation, with EBUSY, only when the validation must reread the table.
  • In CI, a GitHub-hosted job of its own, guest-test-programs, builds the program beside the guest artifact jobs, under their condition, and uploads it as the guest-test-programs artifact. Only each backend's nvx-microvm-guests-* job, which runs the guest and sandbox steps beside the default suite, waits for it, downloads it into build/guest-tests, and runs the scenario; the default suites and the debug-kernel jobs don't. The required status check expects the job whenever the guest artifacts build, and the development release and performance persistence require it, as they do the debug kernel's build, because they accept skipped microVM jobs. gpt_image.py and guest_test_programs.py run only in the microVM jobs and in that job, so the change classifier doesn't count them as inputs of the platform jobs.
  • doc/project-structure.md lists nvx::gpt, the guest library, the guest GPT device tests, and the two CLI modules.

Guest binaries

The release guest binaries keep their sizes and their instructions, against those built from dev at 58a4a8f:

Binary Before (bytes) After (bytes)
nvx-console-pending 389,664 389,664
nvx-device-io 389,664 389,664
nvx-exit 377,376 377,376
nvx-ingress 422,432 422,432
nvx-managed-agent 487,968 487,968
nvx-mmio-write 389,664 389,664
nvx-reseed 401,952 401,952
nvx-time 635,536 635,536
nvx-time-probe 561,872 561,872

Their instructions are unchanged. Seven of them differ in one byte, the line of a panic location in nvx-rs/src/guest/sys/mod.rs, whose lines moved, and nvx-ingress and nvx-time place some of their functions in another order. The commits that rename the first partition and compare the partitions first change none of them.

Part of #621. Refs #622 and #535.

Validation

This head, e88506f, is on dev at 58a4a8f, after #764 and #713. #713 adds its guest-artifact-reuse job where this pull request adds guest-test-programs, to the jobs and results that the required status check expects, so both stay, and its test of the guest artifacts that a dev push reuses now also lists guest-test-programs among the handoffs that no package ships, beside the debug kernel. The Rust gates ran at 838cba70, which differs from this head only in that test, on Windows 11 (MSVC, Rust 1.95.0) and on an Azure Linux 3.0 virtual machine as a non-root user (Rust 1.95.0), in nvx-rs with NVX_BUNDLE=skip:

  • cargo fmt --all --check
  • cargo clippy --locked --all-targets --all-features -- -D warnings
  • cargo clippy --locked --lib --no-default-features -- -D warnings
  • cargo test --locked --all-features, then cargo test --locked. The new unit tests compare tables with kernel partitions, wait against injected observations (late, changing, at the deadline, with a zero timeout, and failing), read a fake sysfs tree with vanished and malformed attributes and with a partition that is created again between two reads of its attributes, confirm readings against partitions that are dropped, created, or created again after them, against a partition whose attributes the kernel finishes creating after them, from each partial state, and against a reading torn by a reread between two partitions, and check the ioctl request numbers and the splitting of device numbers. Without the second listing, without the check of the partitions' directories, or without that of the other directories, some of the confirmation tests fail.
  • cargo doc --locked --no-deps --all-features with RUSTDOCFLAGS=-D warnings
  • cargo +1.89 check --locked --all-targets --all-features, the minimum supported Rust version
  • tests/runtime-adapter in its protocol-only and host-only configurations, and on Linux its musl guest-only one: rustfmt, Clippy, build, tests, and the 1.89 check
  • aci_edge_sandboxes, which builds on nvx, as its CI checks it, with ACI_EDGE_SANDBOXES_AGENT_BUNDLE=skip: rustfmt, Clippy, tests, nvx.py test-agent-bundling, rustdoc, its minimum supported Rust version, and on Linux the macOS check

On Linux only, at 838cba70:

  • the artifact bundling test over a synthetic bundle, as check-nvx-rs runs it
  • cargo check --locked --all-targets --all-features --target aarch64-apple-darwin
  • python3 scripts/nvx.py check-guest-binaries, build-guest-binaries --output DIR, and test-guest-binaries --binaries DIR; and the comparison above against those built from dev at 58a4a8f
  • python3 scripts/nvx.py build-guest-tests
  • scripts/test_microvm_tests.py, Specula's tests, which need fcntl, .github/skills/nvx-host-connect/scripts/test_hosts.py, and the nvx.py help commands. All the validate-nvx modules and scripts/test_adversarial.py passed there at ba08cc6, except dev's test_sleeper_subshell_records_and_reaps_its_sleepers, which requires that no sleep 611.25 process exist on the host: it failed in the full run while another checkout ran it, and passed when it ran alone.

On Windows, at this head: the validate-nvx test modules, scripts/test_adversarial.py, test_hosts.py, the nvx.py help commands, python -m pyright --pythonplatform Linux and --pythonplatform Windows, ruff check and ruff format --check, at the versions in requirements-dev.txt, python scripts/nvx.py verify, and python scripts/check_markdown_links.py.

On real VMs at 838cba70, with the OpenVMM, kernel, and initramfs of dev's development release at a6785cd, whose OpenVMM pin this head keeps, nvx.py test-microvm --scenario gpt-device passed all nine guest tests three times on each of:

  • Windows/WHP, on Windows 11;
  • Linux/MSHV, on the Linux virtual machine, as root and through an OpenVMM wrapper that adds --cpu-profile host, since no built-in CPU profile serves its CPU.

The duration report times the scenario as its own span, 3.6 s on MSHV and 3.5 s on WHP. The timings of the validation above come from the same scenario at 28b7446, the same code before the rebase, with a test program that reports them, built once without and once with optimization.

The full default nvx.py test-microvm suite, its 36 scenarios, passed on Windows/WHP at ba08cc6; the commits since change no scenario but gpt-device, which the suite doesn't run.

CI runs the hypervisor matrix at this head.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Concurrent partition recreation can mix sysfs attributes from different generations and incorrectly pass validation.

1 open issue
What changed in this PR

Adds guest-side GPT device validation on top of #673’s reader, supporting future layered-root assembly without changing release guest images.

Changes:

  • Validates whole disks and compares GPT entries with kernel partition extents.
  • Adds unit tests, privileged guest tests, and a test-program build command.
  • Runs the new opt-in scenario across CI backends and documents its usage.
File Description
scripts/​test_nvx_tools.py Checks guest-test CI wiring.
scripts/​test_microvm_tests.py Tests scenario selection and disk fixtures.
scripts/​test_guest_binaries.py Tests guest-test program builds.
scripts/​nvx.py Adds build-guest-tests.
scripts/​nvx_tools/​microvm_tests.py Adds GPT fixtures and scenario execution.
scripts/​nvx_tools/​microvm_test_scripts/​gpt-device.sh.in Runs privileged tests inside the guest.
scripts/​nvx_tools/​guest_test_programs.py Builds static guest-test executables.
scripts/​nvx_tools/​gpt_image.py Generates GPT and MBR test images.
nvx-rs/​tests/​gpt_device.rs Adds privileged device integration tests.
nvx-rs/​src/​lib.rs Documents guest GPT validation.
nvx-rs/​src/​guest/​sys/​mod.rs Exposes block-device helpers internally.
nvx-rs/​src/​guest/​sys/​block.rs Wraps block-device ioctls.
nvx-rs/​src/​guest/​mod.rs Exports the GPT guest library.
nvx-rs/​src/​guest/​gpt/​tests.rs Tests comparison, waiting, and sysfs parsing.
nvx-rs/​src/​guest/​gpt/​mod.rs Implements device and partition validation.
nvx-rs/​src/​guest/​gpt/​error.rs Defines validation errors.
nvx-rs/​src/​gpt/​tests.rs Shares a standard disk fixture.
nvx-rs/​src/​gpt/​mod.rs Makes test helpers available internally.
nvx-rs/​README.md Documents the API and guest tests.
doc/​usage.md Documents build and scenario commands.
doc/​ci.md Describes guest-library CI coverage.
.github/​workflows/​run-nvx-microvm-tests.yml Runs the scenario and collects logs.
.github/​workflows/​ci.yml Builds and uploads guest-test programs.

Review effort: Balanced

Comment thread nvx-rs/src/guest/gpt/mod.rs Outdated

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Privileged partition rereads and race-sensitive sysfs validation warrant final human review and backend CI confirmation.

No open issues 🎉

1 resolved since last review

Review effort: Balanced

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Concurrent partition replacement can make the scan falsely accept observations from different table generations.

1 open issue

Review effort: Balanced

Comment thread nvx-rs/src/guest/gpt/mod.rs Outdated

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Kernel partition consistency during concurrent rereads and privileged cross-backend behavior warrant final human review.

No open issues 🎉

1 resolved since last review

Review effort: Balanced

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The privileged kernel operations and race-sensitive sysfs consistency guarantees warrant final human review despite the extensive supplied test coverage.

No open issues 🎉

Review effort: Balanced

Base automatically changed from esaurez/a3-gpt-validation-20261020 to dev October 22, 2026 15:25
Copilot AI balanced review requested due to automatic review settings October 22, 2026 16:29
@esaurez
Enrique Saurez (esaurez) force-pushed the esaurez/a3-gpt-device-validation-20261020 branch from c85d39d to c34bbd7 Compare October 22, 2026 16:29

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The kernel-dependent sysfs consistency guarantees and privileged cross-backend integration warrant final human review.

No open issues 🎉

Review effort: Balanced

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The kernel partition-lifetime assumptions and privileged behavior across KVM, MSHV, and WHP warrant final human review.

No open issues 🎉

Review effort: Balanced

Comment thread doc/project-structure.md Outdated
Comment thread nvx-rs/src/guest/gpt/mod.rs Outdated
Copilot AI balanced review requested due to automatic review settings October 22, 2026 18:49

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Concurrent sysfs consistency guarantees and privileged partition rereads across supported backends warrant final human review.

No open issues 🎉

Review effort: Balanced

Enrique Saurez and others added 8 commits October 22, 2026 12:01
Add nvx::guest::gpt, which checks a whole block device in the guest
before its partitions are used. It reads the disk's GPT with nvx::gpt
and the caller's Expected values, has the kernel reread the partition
table, and waits, up to the caller's timeout, until the partitions that
sysfs reports match the table's entries in number, start, and size. The
first difference fails with the partition that differs, so a disk that
the kernel reads as another table, or that changed after it was read,
never reaches the code that would mount its partitions.

BlockDevice::new accepts only a whole disk whose device number sysfs
confirms, and reads its logical block size and size with the block
ioctls that nvx::guest::sys now wraps, along with the rereading of the
partition table.

The privileged checks run in a guest. build-guest-tests builds
nvx-rs/tests/gpt_device.rs as a static test program, and the gpt-device
scenario of test-microvm, which runs only when named, shares it with GPT
images that the host writes and runs it as root. It checks a valid disk,
a rewritten one, late and missing partitions, extra and moved ones, a
disk that the kernel reads as an MBR, and a reread while a partition is
open. CI builds the program in a job of its own beside the guest
artifacts and runs the scenario in every backend's guest and sandbox
job.

Part of #621.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 1b4b569c-a481-43aa-bfd8-c14a9762c371
The kernel may drop a partition and create another under its name
between two reads of its sysfs attributes, so reading them by path
could combine the start of one partition with the size of the other,
and accept an extent that neither has. Open each partition's sysfs
directory once, and read its attributes relative to it: once the
kernel drops the partition, its directory holds no attributes, so the
partition counts as gone and the comparison tries again. Read the
disk's own description the same way, and add sys::open_at for both.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 1b4b569c-a481-43aa-bfd8-c14a9762c371
nvx::gpt takes and returns nvx::uuid::Uuid in place of its own Guid,
so the privileged guest tests build the disk GUID that must fail to
match with it.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 1b4b569c-a481-43aa-bfd8-c14a9762c371
A reading of the kernel's partitions could combine partitions from
before and after a reread of the table: one read before the kernel
dropped them all, and the others read after it created the new
table's. Together they could match the validated table when neither
table does.

After each reading, read the partitions again, and read each partition
again through the sysfs directory that the first reading opened, which
no longer describes it once the kernel dropped it. The partitions that
pass all existed together, so they come from one reading of the table.
A reading that changed counts as a mismatch, and the comparison repeats
until the deadline, as for any other.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 1b4b569c-a481-43aa-bfd8-c14a9762c371
The confirmation of a reading of the kernel's partitions read every
partition a second time, so that a comparison took about two and a
half times as long as a single reading.

Instead, list the disk's sysfs directories again, which must be those
that the first reading opened, and check each of them again through
the descriptor that the first reading opened: a partition's number
must be the same, which shows that the partition still exists, and
another directory must still describe no partition. A reread still
can't tear a reading, since it drops the partitions that the first
reading holds, and the listing shows a partition that the kernel
created under a new name. A directory that described no partition and
now describes one shows a partition that the kernel finished creating
after the first reading.

A partition keeps its number, first byte, and device number while it
exists, so only a program that resizes partitions one at a time could
change a description meanwhile, as it can after the comparison anyway.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 1b4b569c-a481-43aa-bfd8-c14a9762c371
List nvx::gpt, the guest library that validates a GPT disk's block
device, the guest GPT device tests, and the two CLI modules of the
gpt-device scenario, gpt_image.py and guest_test_programs.py, in the
project structure.

Part of #621.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 1b4b569c-a481-43aa-bfd8-c14a9762c371
"p1" was shorthand that each reader had to decode. Name the partition
explicitly throughout: in the API and its errors, the documentation,
and the gpt-device scenario.

- `Disk::p1` and `Disk::read_p1` become `Disk::first_partition` and
  `Disk::read_first_partition`, and `GptError::P1TooLarge` becomes
  `GptError::FirstPartitionTooLarge`.
- `ValidatedDisk::p1` becomes `ValidatedDisk::first_partition`.
- The scenario passes the first partition's bytes to the guest's tests
  as `NVX_GPT_DEVICE_FIRST_PARTITION`.

Tests and comments name the other partitions by their numbers too.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 1b4b569c-a481-43aa-bfd8-c14a9762c371
The kernel creates a disk's partitions from its table when it finds the
disk, so at boot they already match the table that the validation reads,
and the reread only dropped them and created them again. It took 0.48 to
2.2 ms of the 0.57 to 2.4 ms that a validation took on KVM and WHP.
Compare the partitions first, as systemd's partition-table reread
leaves partitions that already match alone, and reread only when they
differ.

A disk whose partitions match therefore also validates while one of them
is open, which a reread refuses.

The pauses between comparisons, which a wait takes only while the
partitions differ, now grow from 1 ms to 4 ms rather than 50 ms: a
comparison of a few partitions takes well under a millisecond.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 1b4b569c-a481-43aa-bfd8-c14a9762c371
Copilot AI balanced review requested due to automatic review settings October 22, 2026 19:02
@esaurez
Enrique Saurez (esaurez) force-pushed the esaurez/a3-gpt-device-validation-20261020 branch from 28b7446 to e88506f Compare October 22, 2026 19:02

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The partition-consistency guarantees rely on subtle kernel and sysfs lifecycle semantics that warrant final human verification.

No open issues 🎉

Review effort: Balanced

@esaurez

Copy link
Copy Markdown
Contributor Author

Superseded by #771, which reduces this to a first iteration: it reads a GPT disk's table through the guest's block device and checks the expected disk GUID. The comparison with the kernel's partitions, the reread, and the microVM tests move to #770, with this pull request and its branch, esaurez/a3-gpt-device-validation-20261020, as the reference. Closing without merging.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants