Skip to content

doc: describe live shares in the sandbox filesystem bootstrap - #444

Merged
Pedro Henrique Penna (ppenna) merged 1 commit into
devfrom
doc-sync/sandbox-live-shares
Oct 9, 2026
Merged

Pedro Henrique Penna (ppenna) merged 1 commit into
devfrom
doc-sync/sandbox-live-shares

Conversation

@ppenna

Copy link
Copy Markdown
Contributor

Summary

doc/design/sandbox-filesystem-and-agent-architecture.md describes the guest
init agent's filesystem assembly, but it predates the sandbox's live
virtio-fs shares. Since 13d2722 (2026-09-30), the init agent has mounted a
share inside the container root, and since c220042 (2026-10-06) a sandbox can
attach two shares, each in its own fixed virtio-fs slot with its own access
mode and host-enforced policy. None of the later edits to this chapter covered
them. The share commits updated run.md, usage.md, ci.md, and the ABI,
status, and validation chapters, but not this one. As a result, the
chapter's assembly never mounts a share, its teardown never unmounts one, and
it still calls the managed agent's mapping source "OpenVMM's single virtio-fs
export".

This PR brings the implemented-bootstrap section in line with the code. It
adds an architecture-level section on live shares that links to
run.md
for options and rules and to
machine-and-device-abi.md
for the device contract, so those facts stay stated in one place. One file
changes, all under doc/.

Changes

NVX paths are relative to the repository root. OpenVMM paths are in the pinned
submodule.

Statement Was Now Evidence
What nvx sandbox accepts Layers, scratch, entrypoint, and arguments Adds "up to two live host shares" MAX_MOUNTS = len(MOUNT_TAGS) with MOUNT_TAGS = ("microvm", "microvm1") (scripts/nvx_tools/sandbox.py). python scripts/nvx.py sandbox --help: "repeat to attach up to 2 shares". scripts/nvx.py accepts --mount only for run and provision.
Required kernel facilities No virtio-fs Adds virtio-fs CONFIG_FUSE_FS=y and CONFIG_VIRTIO_FS=y in kernel/config-microvm. kernel/config-microvm-debug is a fragment appended to it that only enables watchdogs.
Assembly step 6 Barrier start, with the machine-ID file Verify the identity, mount each live share at its target in the assembled root, and write the machine ID guest/common/nvx-init-agent runs the identity checks, then mount_live_shares, then writes $runtime/workload-machine-id, and only then starts the one-shot child or the managed agent.
Assembly step 7 (was step 6) Barrier start and cgroup placement only Same script: nvx-container-launch behind the FIFO, with the PID written to /sys/fs/cgroup/container/cgroup.procs before start is written.
Assembly step 8 (teardown) Unmount overlay, layers, scratch Unmount the shares in reverse order, then the overlay, layers, and scratch teardown() calls unmount_live_shares (most recent first) before it unmounts $rootfs, the layers, and scratch, then calls nvx-exit.
New: managed sandbox note (not stated) The managed agent repeats step 7 per workload, and step 8 follows stop The init agent starts nvx-managed-agent with $rootfs, then calls teardown. With layers, the managed agent writes the child to the container cgroup, then releases the barrier, then runs nvx-container-launch (write_pid_to_cgroup, release_container_barrier, and exec_sandbox in guest/common/nvx-managed-agent.c).
Assembled view Overlay only Adds tags microvm and microvm1 mounted at /run/nvx/rootfs/TARGET mount -t virtiofs -o "$share_mode,nosuid,nodev" "$share_tag" "$share_path" in the init agent. Tags from MOUNT_TAGS.
New section: Live host shares, slots (not documented) Each --mount gets its own fixed virtio-fs slot (microvm, microvm1) served by its own HostFs server microvm_filesystems_from_mounts zips the mounts with MICROVM_FILESYSTEM_SLOTS (openvmm/openvmm_entry/src/microvm/filesystem.rs). openvmm/openvmm_entry/src/microvm/config.rs builds one VirtioFsBackend::HostFs per attached slot with its own policy profile.
Live host shares: host enforcement (not documented) The host enforces the mode and the denied, allowed, and writable paths on every request, whichever guest mount or link reaches the share, and accesses host files as the --mount-owner identity. Guest flags are not a boundary, and the policy adds no guest tokens. The ### Filesystem section of machine-and-device-abi.md. The policy pin 3c03d93 and c0961d8 ("adds no bootstrap tokens and no guest changes"). _MOUNT_COMMAND_LINE_FRAGMENT carries only virtfs_dir, virtfs_tag, and virtfs_mode, and the policy goes to OpenVMM in mounts_openvmm_arguments. The filesystem-shares and filesystem-policy scenarios expect EROFS from OpenVMM even with the tag remounted read-write.
Live host shares: host-side rejections (not documented) NVX rejects a third share and policy paths outside their share before OpenVMM starts, and also overlapping guest targets or host directories validate_mounts and validate_mount_host_paths, called from SandboxLaunch (scripts/nvx_tools/sandbox.py).
Live host shares: guest assembly (not documented) One triplet per share in slot order, all parsed before any mount. Each target is created one component at a time without crossing a symbolic link. The agent refuses repeated tags, overlapping targets, and targets the entry helper mounts or binds over, mounts each share MODE,nosuid,nodev, and lets the private namespace inherit it. Any refusal aborts with status 125. mount_live_shares, mount_live_share, validate_share_target, and fatal (nvx-exit 125) in guest/common/nvx-init-agent. The entry helpers (guest/alpine/nvx-container-enter and guest/common/nvx-container-enter-azurelinux) mount /proc, /sys, /dev, and /.nvx-agent and bind /etc/machine-id. nvx-container-launch runs unshare --mount after the shares are mounted.
Live host shares: managed sandboxes (not documented) Shares are recorded at provisioning and reattached on every start The mount and mounts configuration in formats 2 through 5 (scripts/nvx_tools/sandbox_lifecycle.py). Also stated in run.md.
Proposed image-delivery paragraph "The optional HostFs device" (singular) "The optional HostFs devices", linking to the new section Two slots: MICROVM_FILESYSTEM_SLOTS: [MicrovmFilesystemSlot; 2] (openvmm/openvmm_defs/src/microvm.rs).
Host mappings without sandbox layers "through OpenVMM's single virtio-fs export" "the agent maps host directories for workloads from one virtio-fs export" OpenVMM now has two slots. The mapping protocol still uses one export: HOSTFS_ROOT in guest/common/nvx-managed-agent.c, and aci_edge_sandboxes/src/openvmm/filesystem.rs passes a single --mount /run/nvx/hostfs/root,....

I left the rest of the section unchanged after rechecking it against the
init agent, the container helpers, and the CLI. That includes block
resolution through /proc/iomem, the sector, flush, and UUID checks, the
overlay options, the one-to-three-layer rule, the 16 MiB memory.low
default, and the managed-exec option split. scripts/nvx.py rejects
--cwd and the environment options outside sandbox exec.

OpenVMM SHA: f12dcf5e8f2ea60cc15899dd16852320530670bd, the gitlink on
dev at 33bd2fe5ef6006fb665cb5b870fa03b08e828f13. This PR does not change
the gitlink or the submodule.

Follow-ups

These are out of scope here, or blocked by open PRs or unpublished branches:

Validation

  • python -m unittest scripts.test_nvx_tools.CliTests scripts.test_performance.PerformanceTests:
    Ran 110 tests, OK (skipped=1). The skipped test needs a Linux
    pseudo-terminal, and this ran on Windows.
  • I ran a throwaway checker kept outside the repo. It covered the changed
    file and every tracked Markdown file that links to it: doc/design.md,
    doc/design/cold-boot.md, doc/design/machine-and-device-abi.md,
    doc/design/snapshot-and-restore.md, doc/guests/ubuntu-guest.md, and
    doc/run.md. It confirmed that every relative link and anchor resolves,
    the changed file has one H1 and balanced code fences, and doc/design.md
    links every chapter in doc/design/. Result: 0 errors. Its self-test
    reported a planted broken link, broken anchors, a second H1, an unclosed
    fence, and an unindexed chapter, and nothing else. The new links resolve:
    #live-host-shares (used twice), machine-and-device-abi.md#filesystem,
    and ../run.md#live-host-directory-shares. The chapter's inbound anchors
    are unchanged.
  • git diff --check: clean.
  • python scripts/nvx.py verify: passes.
  • python scripts/nvx.py classify-ci-changes --event-name pull_request on
    this diff: "Only documentation files changed", and every suite is false.
  • The only changed path,
    doc/design/sandbox-filesystem-and-agent-architecture.md, matches
    ^(doc/|.*\.md$). The commit is SSH-signed.

Since 13d2722, the sandbox init agent has mounted live virtio-fs
shares inside the container root, and since c220042 a sandbox can
attach two, each in its own fixed virtio-fs slot with its own access
mode and host-enforced policy. The sandbox design chapter still listed
an assembly that never mounts a share and a teardown that never
unmounts one, and it called the managed agent's mapping source
"OpenVMM's single virtio-fs export".

Add the share mount and unmount steps, the shares in the assembled
view, and virtio-fs among the kernel facilities. A new section explains
how a share reaches the workload and why the host enforces its policy,
and links to the run guide for the options and to the machine ABI for
the device contract. The managed agent still maps host directories
from one export, so the chapter says so.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings October 9, 2026 05:23
@ppenna Pedro Henrique Penna (ppenna) added the documentation Improvements or additions to documentation label Oct 9, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The documentation incorrectly attributes single-share policy-path validation to NVX instead of OpenVMM.

1 open finding
What changed in this PR

Documents live virtio-fs shares in the sandbox filesystem bootstrap architecture.

Changes:

  • Adds share mounting, teardown, security, and lifecycle details.
  • Updates HostFs terminology for multiple filesystem slots.
File Description
doc/​design/​sandbox-filesystem-and-agent-architecture.md Documents live-share bootstrap and managed-sandbox behavior.

🧠 Review effort: Balanced


💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment on lines +113 to +114
no guest configuration. Before OpenVMM starts, NVX rejects a third share and
policy paths outside their share. It also rejects guest targets or host
@ppenna
Pedro Henrique Penna (ppenna) merged commit 1161e3a into dev Oct 9, 2026
28 checks passed
@ppenna
Pedro Henrique Penna (ppenna) deleted the doc-sync/sandbox-live-shares branch October 9, 2026 12:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants