Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions nvx-rs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -102,6 +102,12 @@ nvx-rs/
| `nvx::guest::root` | The operations of a layered root, for a guest init or agent, whether or not it is PID 1. `bootstrap`: the agent's and the workload's sibling cgroups of a `Layout`, with every process of their parent moved into the agent's, the agent's `memory.low`, and the workload cgroup prepared through `ExecutionCgroup::prepare`, with its `memory.max` and `pids.max`, and user namespaces kept unavailable. `assemble`: the `Assembly`'s block devices, found by their MMIO windows, checked against their roles' access modes and their expected sizes, and flushed, and the `Root` that it mounts, at a runtime directory, a canonical path that no symbolic link leads through, that it holds exclusively until its teardown, where nothing is mounted already, once it has checked everything: the read-only layers of the `Image` that the caller names, which it never probes: an EROFS image, whose expected `nvx::uuid::Uuid` it checks in its superblock, or a GPT disk whose partition table it reads and checks with `nvx::guest::gpt::read_disk`, against the caller's `nvx::gpt::Expected` values, whose first partition it reads, up to the caller's bound, as opaque bytes that it never mounts, and whose later partitions, at least one, hold ext4 layers that it mounts `ro,nosuid,nodev` with `errors=remount-ro`, so that an error that the kernel finds in a layer fails the operation that found it, rather than panic the guest, as the layer's superblock may ask, from the block devices that the kernel created when it found the disk, once their superblocks show that no journal needs a recovery, which their read-only device can't take, partition 2's at the overlay's base and the last partition's at its top, under an overlay mounted `nosuid,nodev` with the one option set of the caller's `OverlayMode`, `metacopy=on,xino=on` or, for lowers that may carry untrusted overlay xattrs, `metacopy=off,redirect_dir=nofollow`, with no fallback, whose upper and work directories are on the `Upper`: an ext4 scratch mounted `nosuid,nodev`, whose superblock it checks for a journal that the caller's `Journal` policy forbids, or a tmpfs mounted `nosuid,nodev` at the kernel's default size. It refuses overlay options longer than `MOUNT_OPTIONS_LIMIT`, the most that a mount passes, which, below `/run/nvx`, fit up to 195 layers of a GPT disk, and an overlay that the kernel's mount table shows read-only or without the mode's options in effect, since the kernel mounts one so, rather than fail, when its upper can't hold its work directory or its extended attributes. The `Root` reports the first partition's bytes, its `Lower` layers, its `UpperKind`, and its `OverlayMode`. `Root::mount_share` and `Root::mount_aggregate`: virtio-fs live shares at targets in the root that it checks, against the targets that `protocol::share` reserves among others, with one aggregate per root, which it attaches through its `mappings::Mapper` beneath the overlay, with NVX's container root's reserved targets: a share's device mounted out of the root, then moved onto its target, and a copy of each child's entry of the aggregate. `Root::map`: a caller's host mappings, through the same mapper. A share, a child, or a mapping that fails latches the root. `Root::teardown`: the shares and mappings in reverse order, then an aggregate's own mount, the overlay, the layers from the top down, and the upper, with every failed unmount reported. Each failure is a typed `Error`. The operations collect no child's status, change neither the caller's root nor its credentials, start no async runtime, and never power the VM off |
| `nvx::guest::gpt` | `read_disk`: the partition table of a GPT disk, read and checked with `nvx::gpt`'s `Disk::read` through the disk's block device, which the caller opened, at the logical block size and size that `BLKSSZGET` and `BLKGETSIZE64` report, with the caller's `Expected` values, such as the disk GUID that a boot parameter names. It only reads the device, at explicit offsets, so it never moves the file's position. It neither compares the partitions that the kernel created with the table nor asks the kernel to reread the table. Each failure is a typed `DeviceError` |

GPT lower layers use `noload,errors=remount-ro`: the kernel neither loads nor
replays their journals, because even a journal whose file system does not
request recovery can cause an unbounded scan. The assembly still refuses any
layer whose superblock requests recovery, before mounting anything. This
policy does not change EROFS layers or writable ext4 scratches.

Lowering functions take the names by which messages refer to their inputs, such
as `EgressRules::name` and `PathList::name`, so callers' messages name their own
fields.
Expand Down
27 changes: 13 additions & 14 deletions nvx-rs/src/guest/root.rs
Original file line number Diff line number Diff line change
Expand Up @@ -191,11 +191,10 @@ const SCRATCH_DIRECTORY_MODE: u32 = 0o755;
/// The tmpfs upper's options: the mode of its root directory, which an ext4 scratch's root has
/// too, rather than tmpfs's world-writable default. Its size is the kernel's default.
const TMPFS_OPTIONS: &[u8] = b"mode=0755";
/// The options of a GPT disk's layers, ext4 file systems: an error that the kernel finds in a
/// layer fails the operation that found it, rather than panic the guest, as the layer's
/// superblock may ask, with its error policy or among its default mount options, both of which
/// these options override. A layer is read-only already.
const LAYER_OPTIONS: &[u8] = b"errors=remount-ro";
/// Do not load a GPT layer's journal: even without `needs_recovery`, loading can scan a corrupt
/// journal indefinitely. `check_layer` still refuses recovery-required layers. Override the
/// superblock's error policy too, so an ext4 error does not request a guest panic.
const LAYER_OPTIONS: &[u8] = b"noload,errors=remount-ro";
/// The characters that a runtime directory may not hold: the separators and escape of the
/// overlay's options, and the C string's terminator.
const RUNTIME_RESERVED: &[u8] = b",:\\=\0";
Expand Down Expand Up @@ -316,12 +315,13 @@ pub enum Image {
/// asks the kernel to reread it. Its first partition holds opaque bytes, which the assembly
/// returns through [`Root::first_partition`], and never probes or mounts. Each later
/// partition, from partition 2 on, holds a layer, an ext4 file system that the assembly
/// mounts at `layers/pN`, for the partition's number N, `ro,nosuid,nodev`, without `noload`,
/// and with `errors=remount-ro`: an error that the kernel finds in a layer fails the
/// operation that found it, rather than panic the guest, as the layer's superblock may ask.
/// mounts at `layers/pN`, for the partition's number N, `ro,nosuid,nodev`, with
/// `noload,errors=remount-ro`: it never loads or replays the journal, whose scan can hang
/// even when the superblock does not request recovery, and overrides a superblock's
/// request to panic on an ext4 error.
/// It refuses, before it mounts anything, a layer whose superblock has the `needs_recovery`
/// feature: the layer's read-only device can't take the recovery, which `noload` or
/// `norecovery` among the superblock's own default mount options would skip. The overlay
/// feature: the layer's read-only device can't take the recovery, and suppressing journal
/// loading must not silently admit such a layer. The overlay
/// stacks the layers with partition 2's at its base and the last partition's at its top. A
/// disk has at least one layer, as [`nvx::gpt`](crate::gpt) requires.
Gpt {
Expand Down Expand Up @@ -1074,12 +1074,11 @@ fn check_no_journal(device: &Path) -> Result<(), Error> {
}

/// Checks that the layer in partition `number` of a GPT disk, on the block device `device`, holds
/// an ext4 file system whose journal needs no recovery, in its superblock's feature flags, and
/// an ext4 file system whose superblock does not request journal recovery, and
/// reads nothing else.
///
/// The layer's device is read-only, so the kernel can't recover its journal. Without `noload`,
/// the kernel refuses to mount such a layer, but with `noload` or `norecovery` among the
/// superblock's own default mount options, it would mount the layer without the recovery.
/// The layer's device is read-only, so it can't take journal recovery. Reject the recovery bit
/// explicitly because our `noload` mount option would otherwise let the kernel skip it.
fn check_layer(number: u8, device: &Path) -> Result<(), Error> {
match block::ext4_features(device) {
Ok(Some(features)) if !features.recovery => Ok(()),
Expand Down
5 changes: 2 additions & 3 deletions nvx-rs/src/guest/root/tests.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1136,7 +1136,7 @@ fn a_gpt_disk_s_layers_are_mounted_under_the_overlay_top_first() {
&format!("{runtime}/layers/p{number}"),
Some("ext4"),
MountFlags::MS_RDONLY | MountFlags::MS_NOSUID | MountFlags::MS_NODEV,
Some("errors=remount-ro"),
Some("noload,errors=remount-ro"),
)
};
// The disk is flushed, then its table read and checked, and the scratch checked, before any
Expand Down Expand Up @@ -1982,8 +1982,7 @@ fn a_layer_that_is_not_ext4_or_whose_journal_needs_a_recovery_is_refused_before_
let assembly = guest.gpt_assembly(Upper::Tmpfs, OverlayMode::TrustedLowers);
guest.assemble(&assembly).unwrap().teardown().unwrap();

// But not one that needs a recovery, which its read-only device can't take, whatever its
// superblock's own default mount options, such as `norecovery`, which would skip it.
// But not one that needs recovery, despite our `noload` or the superblock's own defaults.
let layer = disk.layer_mut(3);
layer[1024 + 0x60] |= 0x04;
layer[1024 + 0x200..][..10].copy_from_slice(b"norecovery");
Expand Down
13 changes: 11 additions & 2 deletions tests/runtime-adapter/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -817,7 +817,7 @@ own:
| `erofs_over_scratch_with_untrusted_lowers` | The image over the scratch, in the untrusted-lowers mode, with a journal that the launch allows | The root, whose overlay has none of the trusting mode's options, copies up a file's data too, and refuses the rename of a lower directory with `EXDEV` |
| `erofs_over_ram` | The image over a tmpfs in RAM | The root, whose upper and work directories are on a tmpfs of the kernel's default size, whose root directory has mode 0755, with no scratch mounted, and no disk but the image's |
| `erofs_over_ram_with_untrusted_lowers` | The image over a tmpfs, in the untrusted-lowers mode | The root over the tmpfs, which copies up a file's data too, and refuses the rename |
| `gpt_over_scratch` | A GPT disk that the scenario writes, with its own GUID, over a copy of the scratch template | A root that the worker assembled: the first partition's bytes, as the disk holds them without their padding; each layer, an ext4 file system, mounted once, `ro,nosuid,nodev` and without `noload`, from its partition; the overlay, with partition 2's layer at its base and partition 3's at its top, in the default mode, which copies up a file's metadata alone and renames a lower directory; a file that reaches the upper and not the top layer; and the disk's and the scratch's disks |
| `gpt_over_scratch` | A GPT disk that the scenario writes, with its own GUID, over a copy of the scratch template | A root that the worker assembled: the first partition's bytes, as the disk holds them without their padding; each layer, an ext4 file system, mounted once, `ro,nosuid,nodev` with journal loading suppressed (`noload`, reported as `norecovery` by the kernel), from its partition; the overlay, with partition 2's layer at its base and partition 3's at its top, in the default mode, which copies up a file's metadata alone and renames a lower directory; a file that reaches the upper and not the top layer; and the disk's and the scratch's disks |
| `gpt_over_scratch_with_untrusted_lowers` | The disk over the scratch, in the untrusted-lowers mode | The root, which copies up a file's data too, and refuses the rename |
| `gpt_over_ram` | The disk over a tmpfs, with a bound that the first partition meets exactly | The root over the tmpfs, with no scratch mounted, and no disk but the GPT disk's |
| `gpt_over_ram_with_untrusted_lowers` | The disk over a tmpfs, in the untrusted-lowers mode | The root over the tmpfs, which copies up a file's data too, and refuses the rename |
Expand All @@ -828,12 +828,14 @@ own:
| `other_disk_guid_fails_the_assembly` | The GPT disk, of which the launch expects another GUID | The assembly's failure, which the launch reports, and a VM that ends with 14 before the guest mounts the scratch |
| `first_partition_over_its_bound_fails_the_assembly` | The GPT disk, with a bound one byte smaller than its first partition | The assembly's failure, which the launch reports, and a VM that ends with 14 before the guest mounts the scratch |
| `corrupt_layer_fails_the_assembly` | The GPT disk, whose top layer is corrupt and asks the kernel to panic on an error | The top layer's mount, which the kernel refuses without a panic, the assembly's failure, which the launch reports, and a VM that ends with 14 before the guest mounts the scratch |
| `cyclic_journal_is_not_loaded` | The GPT disk over a tmpfs, in the untrusted-lowers mode, whose top layer has a cyclic journal but no `needs_recovery` bit | Assembly, reads, copy-up, and teardown complete without scanning the journal |
| `recovery_required_layer_fails_the_assembly` | The same cyclic journal with the top layer's `needs_recovery` bit set | The assembly refuses recovery before mounting anything, despite `noload`, and the VM ends cleanly with 14 |
| `invalid_configuration_fails_its_stage` | A relative runtime directory | The configuration's failure, which the launch reports, and a VM that ends with 11 |
| `busy_root_fails_the_teardown` | The image over the scratch, which the `hold` probe keeps open | The teardown's failure, which the stop reports, and a VM that ends with 15, leaving the scratch mounted |
| `mappings_apply_beneath_the_root` | The image over the scratch, beneath which the `map` and `map-checks` probes map | Through the root: zero mappings that do no work, then a table mounted in order, each mapping with its access, `nosuid`, and `nodev`, whose writes reach the sources of the writable ones only, on targets that the mapper created on the overlay. Through mappers of the adapter's own: a link or a magic link at each component, the wrong kinds, a target that would be created inside another mapping, and the runtime's, the container root's, and the adapter's own reserved targets refused, and a read-only mapping that is never writable. A teardown that unmounts the mappings, then the scratch, cleanly |
| `a_refused_mapping_latches_the_root` | The image over the scratch, in which the `map-refused` probe plants a symbolic link | A mapping through the link that the root refuses, nothing mounted or created, and a root that refuses every later mapping, then tears down cleanly |

A row names the root, the image, whether a GPT disk's top layer is corrupt, the
A row names the root, the image, the GPT disk's top-layer variant, the
identity that the launch expects of the image, the upper, the runtime
directory, the overlay mode, the scratch's journal policy, and the bound of the
first partition, and what the scenario expects: a root that the guest
Expand Down Expand Up @@ -868,6 +870,13 @@ mode, an error that the kernel finds when it mounts the layer, and the error
policy of its superblock asks the kernel to panic on an error, which the
assembly's `errors=remount-ro` overrides.

The journal scenarios append an internal journal to the top layer, mapped by
one extent in reserved inode 8. Its nonempty ring holds empty revoke records
of one sequence, with no commit, so a kernel journal scan would cycle forever.
With the file system's `needs_recovery` bit clear, `noload` bypasses that scan
and the root remains usable. With that bit set, the assembly refuses the
layer before any mount: `noload` is not permission to ignore required recovery.

### Running the VM tests

The VM tests, `tests/vm_restart.rs` and `tests/vm_layered.rs`, share their
Expand Down
Loading
Loading