nvx-rs: add the sandbox store, launch journal, and runtime identity

doc/project-structure.md

@@ -102,11 +102,11 @@ nvx/
 |   |   |-- protocol/            Control-console codec and NVXS session shared by the host and the guest
 |   |   |-- contract.rs          Control-contract versions and release compatibility
 |   |   |-- net/                 CIDR parsing and subtraction, and hostname and guest-network forms
-|   |   |-- host/                Artifacts, launch, network and egress-policy lowering, sessions, VM instances, the sandbox launch profile, and the `nvx-rs` CLI (feature `cli`)
+|   |   |-- host/                Artifacts, launch, network and egress-policy lowering, sessions, VM instances, sandbox stores, the sandbox launch profile, and the `nvx-rs` CLI (feature `cli`)
 |   |   |-- testing/             Fake OpenVMM (feature `testing`)
 |   |   |-- guest/               Programs that run in Linux guests, and the guest's NVXS connection (feature `guest`)
 |   |   `-- bin/                 nvx-rs CLI, nvx-fake-openvmm test double, and the guest binaries
-|   |-- tests/                   Fake-OpenVMM instance lifecycle, CLI, NVXS session, guest-binary, and hypervisor managed-lifecycle tests, and the golden fixtures that Python recorded
+|   |-- tests/                   Fake-OpenVMM instance lifecycle, sandbox store, CLI, NVXS session, guest-binary, and hypervisor managed-lifecycle tests, the store's record fixtures, and the golden fixtures that Python recorded
 |   |-- build.rs                 Stages the bundled artifacts (feature `bundled`)
 |   `-- artifacts.json           Release package pinned for the bundled artifacts
 |-- openvmm/                     OpenVMM Git submodule

nvx-rs/README.md

@@ -30,15 +30,15 @@ nvx-rs/
 │   ├── error.rs     nvx::Error and its ErrorKind
 │   ├── host/        feature `host` (default): artifacts, launch, network and egress policy files,
 │   │                filesystem, NVXS connection, session, process, platform, VM instances,
-│   │                outcome reports, and the sandbox launch profile of `nvx.py`, with the Python
-│   │                semantics they keep; with feature `cli`, the `nvx-rs` CLI
+│   │                sandbox stores, outcome reports, and the sandbox launch profile of `nvx.py`,
+│   │                with the Python semantics they keep; with feature `cli`, the `nvx-rs` CLI
 │   ├── testing/     feature `testing`: the fake OpenVMM
 │   ├── guest/       feature `guest`: the programs that run in Linux guests, and the guest's
 │   │                NVXS connection
 │   └── bin/         nvx-rs, nvx-fake-openvmm, and the guest binaries
-├── tests/           fake-OpenVMM, CLI, NVXS session, process-supervision, and guest-binary
-│                    tests, and the golden fixtures that Python recorded for the ports of its
-│                    code, in fixtures/
+├── tests/           fake-OpenVMM, CLI, NVXS session, process-supervision, store, and guest-binary
+│                    tests, the store's record fixtures, and the golden fixtures that Python
+│                    recorded for the ports of its code, in fixtures/
 ├── build.rs         bundled-release staging
 └── artifacts.json   the NVX release that the `bundled` feature stages
 ```
@@ -55,6 +55,7 @@ nvx-rs/
 | `nvx::host::network` | `NetworkPolicy` and `openvmm_arguments`: the egress default and rules, ingress, host-loopback access and forwards as OpenVMM options. Egress rules are lowered exactly into OpenVMM's IPv4 and IPv6 rules by the lowering of egress policy files, with at most 256 rules for each action |
 | `nvx::host::network::policy` | Egress policy files: `compile_policy_file`, `compile_policy_bytes`, and `compile_policy` compile one to a `CompiledEgressPolicy` of native rules; see [Egress policy files](#egress-policy-files) |
 | `nvx::host::filesystem` | The `ExportPlan` of host paths through OpenVMM's aggregate virtio-fs device, the host-mapping table, the reserved guest targets, and `guest_path` and `resolve_guest_path` (also at `nvx::`) |
+| `nvx::host::store` | `SandboxStore`: the runtime records of one sandbox in a private directory with one lock, which every host that supervises the sandbox shares. The `RuntimeRecord` names the running VM: the `ProcessRef` of its VMM, its control endpoint, and the `RuntimeIdentity` that it was launched with, the identities of its OpenVMM, kernel, and initramfs from install or staging metadata and its boot and control contract versions, which decide whether another build of the host may reattach to it. The `LaunchJournal` of a launch holds a marker written before the spawn, the VMM's `ProcessRef` right after it, and an inherited claim on the VMM's log, which proves that a launch that never published its endpoint has exited, so that recovery ends an interrupted launch and signals only a VMM that it holds through a verified `Process`. Records are JSON in one versioned format, and records in any other format fail rather than being migrated |
 | `nvx::host::instance` | `Instance`: a VM in a directory of its own. It starts OpenVMM with a fresh capability and waits until the guest is ready and has mounted the mapped paths, reconciles recorded state, recovers interrupted launches, runs workloads and pumps their output, and stops the VM with a forced fallback |
 | `nvx::host::nvxs` | `HostConnection`: the host's role of the NVXS session over a platform transport. It attaches with the launch capability, waiting while the broker asks it to, sends application bytes in as many DATA records as they take, and returns each DATA payload from the guest, with deadlines; a receive that times out keeps a partial record |
 | `nvx::host::process` | The detached launch of OpenVMM, and verified process references. A `ProcessRef` names one process by its ID, its start time, and the `BootId` of the host boot during which it started. `Process::open` checks a reference and holds its process through a pidfd on Linux or a process handle on Windows, whether or not the caller started it, so that waits sleep until the process exits and termination never reaches a process that reuses its ID. A reference from an earlier boot names an exited process, and a process whose identity this user cannot check, such as another user's, makes `Process::open` fail rather than report an exit. Linux hosts without pidfds poll the reference instead, and cannot terminate the process |
@@ -527,7 +528,10 @@ requirement and, on Linux, Ctrl-C, and shared protocol definitions plus golden
 byte fixtures cover both sides of every record. The NVXS test drives the
 host's and the guest's roles of the session and their connections through a
 broker that relays records as OpenVMM's does, with messages of an application
-framing of its own that span records. On Linux, a loopback test drives
+framing of its own that span records. The store test crashes orchestrator
+processes at each step of a launch, recovers the VMM stand-ins that they leave
+behind, and loads records that another build wrote, in `tests/fixtures/store/`.
+On Linux, a loopback test drives
 the host `ControlSession` through a broker shim into the real managed agent's
 session, dispatch, and execution engine, and `nvx.py` also lints and tests the
 guest binaries, and runs their integration tests against a build of them:

nvx-rs/src/host/mod.rs

@@ -11,6 +11,9 @@
 //! - [`instance`] supervises one VM instance in a directory of its own: it starts OpenVMM and
 //!   waits until the guest is ready, reconciles recorded state with the host, runs workloads,
 //!   and stops the VM. [`report`] reads OpenVMM's outcome report.
+//! - [`store`] keeps the runtime records of a sandbox in a private directory with one lock: the
+//!   runtime record of its VM, with the VM's runtime identity, and the journal of its launch,
+//!   through which an interrupted launch is recovered without signaling an unverified process.
 //! - [`session`] speaks the authenticated control protocol, and [`nvxs`] runs the host's role of
 //!   the NVXS session, which carries it and any other application protocol, over a platform
 //!   transport. [`platform`] holds the primitives of Linux and Windows hosts. Other hosts compile
@@ -47,6 +50,7 @@ pub(crate) mod python;
 pub mod report;
 pub mod sandbox;
 pub mod session;
+pub mod store;
 #[cfg(unix)]
 #[allow(unsafe_code)]
 mod sys;

nvx-rs/src/host/platform/linux.rs

@@ -1,6 +1,7 @@
-use std::fs::{self, DirBuilder, OpenOptions, Permissions};
+use std::fs::{self, DirBuilder, File, OpenOptions, Permissions, TryLockError};
 use std::io::{self, Read};
-use std::os::unix::fs::{DirBuilderExt, MetadataExt, PermissionsExt};
+use std::os::fd::AsRawFd;
+use std::os::unix::fs::{DirBuilderExt, MetadataExt, OpenOptionsExt, PermissionsExt};
 use std::os::unix::net::UnixStream;
 use std::path::Path;
 use std::process::Command;
@@ -276,12 +277,203 @@ pub fn create_private_dir(path: &Path) -> io::Result<()> {
     fs::set_permissions(path, Permissions::from_mode(0o700))
 }
 
+/// Creates the directory `path` of a sandbox store with mode 0700, or checks that an existing one
+/// is private to the current user, as [`check_store_dir`] does, and returns why it is not, if it
+/// is not.
+///
+/// A new directory takes the setgid bit of its parent, and loses the bits that the umask clears,
+/// even the current user's permission to read it, so its mode is set again, to the 0700 that
+/// OpenVMM requires of the directory of its control socket. An existing directory is never
+/// changed: others could have placed records in it, or still hold its files open, which a change
+/// of its mode would not revoke.
+pub(crate) fn create_store_dir(path: &Path) -> io::Result<Option<String>> {
+    match DirBuilder::new().mode(0o700).create(path) {
+        Ok(()) => {
+            if let Some(directory) = open_plain_dir(path)? {
+                // A descriptor that only locates the directory cannot change its mode itself, but
+                // its link in /proc names exactly the directory that it opened.
+                fs::set_permissions(
+                    Path::new("/proc/self/fd").join(directory.as_raw_fd().to_string()),
+                    Permissions::from_mode(0o700),
+                )?;
+            }
+        }
+        Err(error) if error.kind() == io::ErrorKind::AlreadyExists => {}
+        Err(error) => return Err(error),
+    }
+    check_store_dir(path)
+}
+
+/// Returns why the directory `path` cannot hold a sandbox store, if it cannot: unless it is
+/// private to the current user, as [`create_store_dir`] makes it, belonging to the user with mode
+/// 0700.
+pub(crate) fn check_store_dir(path: &Path) -> io::Result<Option<String>> {
+    let Some(directory) = open_plain_dir(path)? else {
+        return Ok(Some("is not a plain directory".to_owned()));
+    };
+    let metadata = directory.metadata()?;
+    Ok(store_dir_problem(
+        metadata.uid(),
+        metadata.mode(),
+        sys::effective_ids().0,
+    ))
+}
+
+/// Returns why a directory of a sandbox store, which the user `uid` owns with the mode `mode`, is
+/// not private to the user `euid`, if it is not: it must belong to that user, grant no access to
+/// anyone else, and have mode 0700, which OpenVMM requires of the directory of its control
+/// socket.
+pub(crate) fn store_dir_problem(uid: u32, mode: u32, euid: u32) -> Option<String> {
+    let mode = mode & 0o7777;
+    if uid != euid {
+        Some(format!(
+            "belongs to user {uid}, not to the current user {euid}"
+        ))
+    } else if mode & 0o077 != 0 {
+        Some(format!("is accessible to other users (mode {mode:03o})"))
+    } else if mode != 0o700 {
+        Some(format!(
+            "has mode {mode:03o}, not the mode 700 that the directory of a control socket must have"
+        ))
+    } else {
+        None
+    }
+}
+
+/// Returns why the file that `file` holds open, the lock, a record, or the capability of a
+/// sandbox store, is not private to the current user, if it is not: never, on Linux, where no
+/// one else may look up a file in the store's directory, which grants them no access.
+pub(crate) fn private_file_problem(_file: &File) -> io::Result<Option<String>> {
+    Ok(None)
+}
+
+/// Gives the regular file `path` the mode `mode` if it belongs to the current user, and leaves
+/// anything else, which includes a symbolic link or a missing file, unchanged.
+///
+/// The file is opened only to locate it, which needs no permission on the file itself.
+pub(crate) fn restore_private_file_mode(path: &Path, mode: u32) -> io::Result<()> {
+    use nix::fcntl::OFlag;
+
+    let opened = OpenOptions::new()
+        .read(true)
+        .custom_flags((OFlag::O_PATH | OFlag::O_NOFOLLOW).bits())
+        .open(path);
+    let file = match opened {
+        Ok(file) => file,
+        Err(error)
+            if error.kind() == io::ErrorKind::NotFound
+                || error.raw_os_error() == Some(Errno::ELOOP as i32) =>
+        {
+            return Ok(());
+        }
+        Err(error) => return Err(error),
+    };
+    let metadata = file.metadata()?;
+    if !metadata.file_type().is_file() || metadata.uid() != sys::effective_ids().0 {
+        return Ok(());
+    }
+    // A descriptor that only locates the file cannot change its mode itself, but its link in
+    // /proc names exactly the file that it opened.
+    fs::set_permissions(
+        Path::new("/proc/self/fd").join(file.as_raw_fd().to_string()),
+        Permissions::from_mode(mode),
+    )
+}
+
+/// Opens the directory `path` only to locate it, which needs no permission on the directory
+/// itself, or returns `None` if it is a symbolic link or not a directory.
+fn open_plain_dir(path: &Path) -> io::Result<Option<File>> {
+    use nix::fcntl::OFlag;
+
+    let opened = OpenOptions::new()
+        .read(true)
+        .custom_flags((OFlag::O_PATH | OFlag::O_DIRECTORY | OFlag::O_NOFOLLOW).bits())
+        .open(path);
+    match opened {
+        Ok(directory) => Ok(Some(directory)),
+        Err(error)
+            if error.raw_os_error() == Some(Errno::ELOOP as i32)
+                || error.raw_os_error() == Some(Errno::ENOTDIR as i32) =>
+        {
+            Ok(None)
+        }
+        Err(error) => Err(error),
+    }
+}
+
 /// Returns the device and inode of a file or directory.
 pub fn file_identity(path: &Path) -> io::Result<(u64, u64)> {
     let metadata = fs::metadata(path)?;
     Ok((metadata.dev(), metadata.ino()))
 }
 
+/// Returns the device and inode of the file that `file` holds open.
+pub(crate) fn opened_file_identity(file: &File) -> io::Result<(u64, u64)> {
+    let metadata = file.metadata()?;
+    Ok((metadata.dev(), metadata.ino()))
+}
+
+/// Claims a launch's fresh log, which `file` holds open.
+///
+/// The claim is an exclusive `flock` lock. It belongs to the open file description, which the
+/// VMM shares through the standard output and error that it inherits, and only the last close of
+/// that description releases it, so it outlives the caller while the VMM runs.
+pub(crate) fn claim_log(file: &File) -> io::Result<()> {
+    match file.try_lock() {
+        Ok(()) => Ok(()),
+        Err(TryLockError::WouldBlock) => Err(io::Error::other("another process holds the log")),
+        Err(TryLockError::Error(error)) => Err(error),
+    }
+}
+
+/// Returns whether a process still holds the claimed log at `path`, which must be the file
+/// `identity`: whether its `flock` claim is held through another open file description.
+///
+/// The open follows no link and does not wait, as it would for a FIFO, so that only the claimed
+/// file is examined.
+pub(crate) fn log_claim_held(path: &Path, identity: (u64, u64)) -> io::Result<bool> {
+    let file = OpenOptions::new()
+        .read(true)
+        .custom_flags((nix::fcntl::OFlag::O_NOFOLLOW | nix::fcntl::OFlag::O_NONBLOCK).bits())
+        .open(path)?;
+    if opened_file_identity(&file)? != identity {
+        return Err(io::Error::other(format!(
+            "{} is not the claimed log",
+            path.display()
+        )));
+    }
+    // A lock taken here goes when the file closes, at the end of this function.
+    match file.try_lock() {
+        Ok(()) => Ok(false),
+        Err(TryLockError::WouldBlock) => Ok(true),
+        Err(TryLockError::Error(error)) => Err(error),
+    }
+}
+
+/// Returns whether the live process `pid` that started at `start_time`, which serves the control
+/// endpoint of a launch whose log is the file `log`, is that launch's VMM: whether it holds the
+/// log as its standard output.
+///
+/// The endpoint is the same socket path for every launch of a sandbox, so the log, which every
+/// launch creates afresh, tells its VMM apart. The start time is read again after the
+/// descriptor, so the answer cannot describe another process that reused the ID meanwhile.
+pub(crate) fn serves_launch(pid: u32, start_time: u64, log: (u64, u64)) -> io::Result<bool> {
+    if process_start_time(pid)? != Some(start_time) {
+        return Ok(false);
+    }
+    let held = match fs::metadata(format!("/proc/{pid}/fd/1")) {
+        Ok(metadata) => (metadata.dev(), metadata.ino()) == log,
+        Err(error)
+            if error.kind() == io::ErrorKind::NotFound
+                || error.raw_os_error() == Some(Errno::ESRCH as i32) =>
+        {
+            false
+        }
+        Err(error) => return Err(error),
+    };
+    Ok(held && process_start_time(pid)? == Some(start_time))
+}
+
 /// Returns the control endpoint OpenVMM should listen on for a sandbox.
 pub(crate) fn control_endpoint(socket_path: &Path) -> io::Result<String> {
     socket_path

nvx-rs/src/host/platform/other.rs

@@ -1,4 +1,4 @@
-use std::fs;
+use std::fs::{self, File};
 use std::io;
 use std::path::Path;
 use std::process::Command;
@@ -77,6 +77,34 @@ pub fn file_identity(_path: &Path) -> io::Result<(u64, u64)> {
     Err(unsupported())
 }
 
+pub(crate) fn opened_file_identity(_file: &File) -> io::Result<(u64, u64)> {
+    Err(unsupported())
+}
+
+pub(crate) fn claim_log(_file: &File) -> io::Result<()> {
+    Err(unsupported())
+}
+
+pub(crate) fn log_claim_held(_path: &Path, _identity: (u64, u64)) -> io::Result<bool> {
+    Err(unsupported())
+}
+
+pub(crate) fn serves_launch(_pid: u32, _start_time: u64, _log: (u64, u64)) -> io::Result<bool> {
+    Err(unsupported())
+}
+
+pub(crate) fn create_store_dir(_path: &Path) -> io::Result<Option<String>> {
+    Err(unsupported())
+}
+
+pub(crate) fn check_store_dir(_path: &Path) -> io::Result<Option<String>> {
+    Err(unsupported())
+}
+
+pub(crate) fn private_file_problem(_file: &File) -> io::Result<Option<String>> {
+    Err(unsupported())
+}
+
 pub(crate) fn detach(_command: &mut Command, _breakaway_from_job: bool) {}
 
 pub(crate) fn probe_hypervisor(_hypervisor: Hypervisor) -> Result<(), String> {