From 945313c598a6dbcfa327a27e877b38a3e5e5533c Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Fri, 9 Oct 2026 02:21:24 -0300 Subject: [PATCH] doc: describe live shares in the sandbox filesystem bootstrap Since 13d272239, the sandbox init agent has mounted live virtio-fs shares inside the container root, and since c22004254 a sandbox can attach two, each in its own fixed virtio-fs slot with its own access mode and host-enforced policy. The sandbox design chapter still listed an assembly that never mounts a share and a teardown that never unmounts one, and it called the managed agent's mapping source "OpenVMM's single virtio-fs export". Add the share mount and unmount steps, the shares in the assembled view, and virtio-fs among the kernel facilities. A new section explains how a share reaches the workload and why the host enforces its policy, and links to the run guide for the options and to the machine ABI for the device contract. The managed agent still maps host directories from one export, so the chapter says so. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- ...ndbox-filesystem-and-agent-architecture.md | 72 +++++++++++++++---- 1 file changed, 59 insertions(+), 13 deletions(-) diff --git a/doc/design/sandbox-filesystem-and-agent-architecture.md b/doc/design/sandbox-filesystem-and-agent-architecture.md index 80b329f8f..cf318485d 100644 --- a/doc/design/sandbox-filesystem-and-agent-architecture.md +++ b/doc/design/sandbox-filesystem-and-agent-architecture.md @@ -21,7 +21,8 @@ or snapshot-tier metadata alone. ## Implemented filesystem bootstrap The public `nvx sandbox` command accepts one to three role-bearing EROFS lower -images, a preformatted ext4 scratch image, an absolute entrypoint, and +images, a preformatted ext4 scratch image, up to two +[live host shares](#live-host-shares), an absolute entrypoint, and individual argument tokens. It supplies non-secret kernel-command-line configuration for one-shot runs. Managed execution carries bounded arguments and a per-execution environment over the authenticated control channel; @@ -30,10 +31,10 @@ are not supported by this command. Lower-level OpenVMM capture and restore do support sandbox blocks. See [Run](../run.md#experimental-single-workload-sandbox). The required kernel facilities are already enabled in the NVX microVM kernel -configuration: virtio-blk, EROFS with compression and xattrs, overlayfs, -cgroup v2, memory and process controllers, namespaces, BPF system calls, and -cgroup BPF programs. Kernel support for a device filter does not mean the -current agent installs one. +configuration: virtio-blk, virtio-fs, EROFS with compression and xattrs, +overlayfs, cgroup v2, memory and process controllers, namespaces, BPF system +calls, and cgroup BPF programs. Kernel support for a device filter does not +mean the current agent installs one. The guest init agent performs the assembly: @@ -47,10 +48,17 @@ The guest init agent performs the assembly: 4. resolve and flush scratch, mount it as ext4 with `rw,nosuid,nodev`, and create its `upper` and `work` directories; 5. mount overlayfs with top-first lower layers and `metacopy=on,xino=on`; -6. start a child behind a FIFO barrier, place that child in the workload - cgroup before releasing it, and retain a runtime-tmpfs machine-ID file; -7. wait for the child, unmount the overlay, layers, and scratch, and return its - status through the guest exit helper. +6. verify the workload identity in the assembled root, mount each live share + at its target inside that root, and write the workload machine ID to a + runtime-tmpfs file; +7. start a child behind a FIFO barrier and place that child in the workload + cgroup before releasing it; +8. wait for the child, unmount the shares in reverse order and then the + overlay, layers, and scratch, and return its status through the guest exit + helper. + +In a managed sandbox, the managed agent repeats step 7 for each requested +workload, and the teardown in step 8 follows `stop`. The assembled view is: @@ -61,6 +69,8 @@ The assembled view is: /run/nvx/scratch/upper (ext4) -----------> upperdir /run/nvx/scratch/work (same ext4) ------> workdir overlay --> /run/nvx/rootfs +virtio-fs tag microvm (optional share) ---> /run/nvx/rootfs/TARGET +virtio-fs tag microvm1 (optional share) ---> /run/nvx/rootfs/TARGET ``` At least one lower role is required by the bootstrap; distro plus runtime is @@ -87,6 +97,41 @@ primary-group match in the assembled root, clears supplementary groups and all capability sets, and rejects an unavailable identity before starting the workload. +### Live host shares + +A live share exposes a host directory, such as a source checkout or a tool +cache, to the workload without copying it into an image: edits are visible in +both directions without staging or copy-back. + +Each `--mount` attaches its own fixed virtio-fs slot, tag `microvm` for the +first share and `microvm1` for the second, served by its own HostFs server, so +the two shares keep independent `ro` or `rw` modes and access policies. On +every request, whichever guest mount or link reaches the share, the host +enforces the share's mode and its denied, allowed, and writable paths, and +accesses host files as the identity that `--mount-owner` selects. Guest +mount flags are therefore not a security boundary, and the access policy adds +no guest configuration. Before OpenVMM starts, NVX rejects a third share and +policy paths outside their share. It also rejects guest targets or host +directories that equal or contain one another, because one share could +otherwise hide the other or reach its files under a different policy. +[Machine and device ABI](machine-and-device-abi.md#filesystem) defines the +device contract. + +OpenVMM appends one `virtfs_dir=`, `virtfs_tag=`, `virtfs_mode=` triplet per +share, in slot order. The init agent parses every triplet before it mounts +any, so a malformed bootstrap mounts nothing. It creates each target inside +the container root one component at a time and refuses a path that crosses a +symbolic link, so an image layer cannot redirect a share outside that root. +It also refuses a repeated tag, overlapping targets, and targets that the +container entry helper later mounts or binds over, where the runtime would +hide the share or write into it. Each share is mounted with its mode and +`nosuid,nodev`, and the workload's private mount namespace inherits it. Any +refusal or mount failure aborts the sandbox with status 125 instead of +starting the workload without its shares. A managed sandbox records its +shares when it is provisioned and reattaches them on every start. +[Live host-directory shares](../run.md#live-host-directory-shares) describes +the options and their rules. + ## Image preparation and distribution (Proposed) Image conversion belongs off the sandbox start path, in a Linux control-plane @@ -118,8 +163,9 @@ as one partitioned disk, would make each combination a separate cache object. Compressed EROFS reduces distribution size and the host's cached image bytes; decompression is performed by the guest kernel as blocks are read. A FUSE daemon and guest-side lazy-pull agent are not required for these rootfs layers. -The optional HostFs device remains useful for a live host export and is a -separate feature, not the container image-delivery path. +The optional HostFs devices remain useful for +[live host shares](#live-host-shares) and are a separate feature, not the +container image-delivery path. The converter must synthesize OCI deletions as overlay whiteouts and opaque directory markers. It must apply a deny-by-default metadata policy to every @@ -317,8 +363,8 @@ cannot enter the directory, the agent writes a diagnostic to the workload's standard error and refuses the request with the `cwd-failed` category and the error number as status, so nothing runs in another directory. -Without sandbox layers, host directories reach workloads through OpenVMM's -single virtio-fs export. The host exports the deepest directory that contains +Without sandbox layers, the agent maps host directories for workloads from +one virtio-fs export. The host exports the deepest directory that contains every mapped path to `/run/nvx/hostfs/root`; before it accepts control traffic, the agent makes `/run/nvx/hostfs` root-only and bind-mounts each mapped path named by an `nvx_map=SOURCE,TARGET,ro|rw` kernel token (percent-encoded paths,