Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 12 additions & 8 deletions .github/workflows/run-nvx-microvm-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -500,7 +500,10 @@ jobs:
--workload-user 65534:65534 \
--memory-mib 256
test "$(ls -A "${share}")" = nvx-host-marker
# A managed sandbox persists the ownership mode and applies it on start.
# A managed sandbox persists the ownership mode and applies it on
# start. Its guest agent reaches the share as guest root, which the
# mode squashes to the share's owner, before any workload runs, so
# that request fails with EPERM too, and the sandbox never starts.
cp --reflink=auto build/ubuntu-smoke-scratch.ext4 "${scratch}"
python3 scripts/nvx.py sandbox provision \
--state-dir "${state}" \
Expand All @@ -519,13 +522,14 @@ jobs:
if config["format"] != 7 or config["export"]["owner"] != "caller":
sys.exit(f"managed sandbox did not persist caller ownership: {config}")
EOF
python3 scripts/nvx.py sandbox start --state-dir "${state}"
python3 scripts/nvx.py sandbox exec \
--state-dir "${state}" \
--entrypoint /sbin/nvx-sandbox-smoke \
--arg /workspace \
--arg eperm
python3 scripts/nvx.py sandbox stop --state-dir "${state}"
if python3 scripts/nvx.py sandbox start --state-dir "${state}"; then
echo "the caller-owned managed sandbox reached its share as the runner" >&2
exit 1
fi
grep -q 'virtio-fs request cannot run as its caller; failing it with EPERM' \
"${state}/openvmm.log"
grep -q 'NVX-MANAGED-ERROR: stage=host-mappings' "${state}/openvmm.log"
test ! -e "${state}/runtime.json"
test "$(ls -A "${share}")" = nvx-host-marker
python3 scripts/nvx.py sandbox deprovision --state-dir "${state}"
rm -rf "${share}" "${scratch}"
Expand Down
36 changes: 25 additions & 11 deletions doc/design/4.1-guest-filesystem-and-image-model.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,9 +27,11 @@ layer with one private ext4 scratch device.
/run/nvx/scratch/upper (ext4) -----------> upper directory
/run/nvx/scratch/work (same ext4) ------> work directory
overlay --> /run/nvx/rootfs
virtio-fs tag microvm (optional share) ---> /run/nvx/rootfs/TARGET
one-shot: virtio-fs tag microvm (optional share) ---> /run/nvx/rootfs/TARGET
or an aggregate ---> /run/nvx/shares, each child bound at
/run/nvx/rootfs/TARGET
managed: the aggregate export ---> /run/nvx/hostfs/root, each mapping of the
table mounted at /run/nvx/rootfs/TARGET
```

**Figure 4.1.** Layered sandbox root assembly.
Expand Down Expand Up @@ -74,10 +76,12 @@ The guest agents assemble the root in this order:
5. Mount the overlay with the `distro` layer as its lower directory,
metadata-only copy-up, and extended inode numbers.
6. Verify the workload identity in the assembled root, mount each live share
at its target inside that root, and write the workload's machine ID to a
runtime file.
of a one-shot sandbox at its target inside that root, or the host's export
of a managed sandbox, and write the workload's machine ID to a runtime
file.
7. Start the workload behind a barrier and place it in the workload cgroup
before releasing it.
before releasing it; a managed sandbox's agent first maps the shares from
the export, as their mapping table orders.
8. Wait for the workload, unmount the shares in reverse order and then the
overlay, the `distro` layer, and scratch, and report the workload's status.

Expand Down Expand Up @@ -110,11 +114,12 @@ after its index and a digest of its guest target, so a snapshot pins every
target, and a restore that requests another target fails before the guest
runs. Before launch, the launcher rejects policy paths outside their share or
on a shared file, guest targets or host paths that equal or contain one
another, and shares whose boot tokens exceed the command-line budget.
another, and one-shot shares whose boot tokens exceed the command-line budget.

The machine profile announces the export through its host filesystem tokens,
and the launcher adds one share token per child that names the child, its
guest target, its mode, and whether it is a file. The guest agents then:
A one-shot sandbox receives its shares at boot. The machine profile announces
the export through its host filesystem tokens, and the launcher adds one share
token per child that names the child, its guest target, its mode, and whether
it is a file. The guest agents then:

- Parse every share token and create every target before mounting anything,
so a malformed bootstrap mounts nothing.
Expand All @@ -131,9 +136,18 @@ guest target, its mode, and whether it is a file. The guest agents then:

The workload's mount namespace inherits these mounts, and teardown unmounts
the binds in reverse order before the aggregate. Any refusal or mount failure
aborts the sandbox instead of starting the workload without its shares. A
managed sandbox records its shares when it is provisioned and reattaches them
on every start.
aborts the sandbox instead of starting the workload without its shares.

A managed sandbox records its shares when it is provisioned, and on every
start receives them as the blockless root receives its host directories (see
4.1.3): the host exports every share as a numbered child of one aggregate,
which the guest agents mount at the same root-only location, and delivers the
mapping table once the managed agent is ready. The agent maps each entry
beneath the assembled root, whose own mount is the overlay, and refuses the
targets that the container runtime keeps, as the boot tokens' targets are
refused. A mapping that fails ends the start, after the guest shuts down as
stop has it do: it unmounts the mappings in reverse order before the export
and the overlay.

## 4.1.3 Blockless Managed Root

Expand Down
7 changes: 4 additions & 3 deletions doc/design/4.4-host-guest-control-interface.md
Original file line number Diff line number Diff line change
Expand Up @@ -86,9 +86,10 @@ therefore returns a four-byte little-endian bit mask of control features.
| 6 | Host mapping table |

The blockless managed root provides all six features; the layered sandbox
root provides cancellation and per-execution environments. Its workloads run
as an account that the image already has, so it does not create accounts. A
feature bit is added together with the behavior it names.
root provides cancellation, per-execution environments, and the host mapping
table. Its workloads run as an account that the image already has, so it does
not create accounts. A feature bit is added together with the behavior it
names.
Hosts ignore unknown bits and trailing bytes, read an `unsupported-operation`
refusal as no features, and terminate a guest that lacks a feature they need
instead of running workloads without the policy they requested.
Expand Down
2 changes: 1 addition & 1 deletion doc/design/5.2-readiness-and-runtime-control.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ operations.
| Operation | Effect |
| --- | --- |
| Provision | Records the sandbox's configuration, including its host shares |
| Start | Launches the VMM with a freshly generated control capability, reattaches the recorded shares, and waits for guest readiness |
| Start | Launches the VMM with a freshly generated control capability and the recorded shares' export, waits for guest readiness, and delivers the shares' mapping table |
| Exec | Runs one workload over the control channel |
| Stop | Shuts down the VM and releases its runtime resources |
| Deprovision | Removes the sandbox's state |
Expand Down
74 changes: 50 additions & 24 deletions doc/run.md
Original file line number Diff line number Diff line change
Expand Up @@ -837,28 +837,49 @@ Guest mount flags are not a security boundary: OpenVMM rejects every write to
an `ro` share with `EROFS`, whichever mount or link inside the guest reaches
it.
After it assembles the container overlay and verifies the workload identity,
the guest agent creates each target inside the container root and mounts the
share there with `nosuid,nodev` before the workload enters its private mount
namespace. With several shares, or a shared file, it mounts the aggregate at
`/run/nvx/shares`, outside the container root, and binds each child at its
target with its mode and `nosuid,nodev`, in order, as the
`nvx_share=NAME,TARGET,MODE[,file]` tokens that NVX adds to the kernel command
line describe. It binds a shared file at a regular file, which it creates when
the container root has nothing at the target. The number of shares is bounded
only by the sandbox's 1024-byte kernel command-line budget. A one-shot
workload exit, a managed `stop`, and any failure after a share is mounted
unmount the binds in reverse order, then the aggregate, before the overlay is
unmounted or the VM powers off. Each target must be an absolute, canonical
path; `/`, `/etc`, `/run`, and `/run/nvx`, and the `/etc/machine-id`,
`/proc`, `/sys`, `/dev`, `/.nvx-agent`, and `/run/nvx/time` trees, are
reserved for the container runtime, which replaces `/etc/machine-id` with the
workload's machine ID, creates `/run` and `/run/nvx`, and binds its time state
over `/run/nvx/time`.
the guest mounts the shares inside the container root with `nosuid,nodev`,
before any workload enters its private mount namespace. How it learns of them
depends on the lifecycle.

A one-shot `sandbox run` delivers them at boot. The guest agent creates each
target inside the container root and mounts the share there. With several
shares, or a shared file, it mounts the aggregate at `/run/nvx/shares`, outside
the container root, and binds each child at its target with its mode, in
order, as the `nvx_share=NAME,TARGET,MODE[,file]` tokens that NVX adds to the
kernel command line describe. It binds a shared file at a regular file, which
it creates when the container root has nothing at the target. The number of
shares is bounded only by the sandbox's 1024-byte kernel command-line budget.

A managed sandbox delivers them once its guest agent is ready. OpenVMM exports
every share as a child of one aggregate, which the guest mounts at
`/run/nvx/hostfs/root`, where only root can enter, and the kernel command line
announces only the number of mappings, as `nvx_maps=COUNT`. Once the guest
agent answers, `start` sends it the mapping table, in order, in as many
requests as it takes. The agent mounts each entry at its target with its mode,
and creates a missing target only on the container overlay, never inside
another mapping, as a directory or, for a shared file, a regular file. It
refuses every workload until it has mounted the whole table. A mapping that
the guest refuses or cannot mount fails `start`, after the guest shuts down as
`stop` has it do, so the sandbox never runs with part of its shares; `start`
terminates OpenVMM only if the guest does not shut down within `--timeout`.
The kernel command line does not bound the number of a managed sandbox's
shares; the 256 children of one aggregate do.

A one-shot workload exit, a managed `stop`, and any failure after a share is
mounted unmount the binds or mappings in reverse order, then the aggregate or
the export, before the overlay is unmounted or the VM powers off, unless the
guest stops answering, or `start` is interrupted, and OpenVMM is terminated
first. Each target
must be an absolute, canonical path; `/`, `/etc`, `/run`, and `/run/nvx`, and
the `/etc/machine-id`, `/proc`, `/sys`, `/dev`, `/.nvx-agent`, and
`/run/nvx/time` trees, are reserved for the container runtime, which replaces
`/etc/machine-id` with the workload's machine ID, creates `/run` and
`/run/nvx`, and binds its time state over `/run/nvx/time`.
The guest refuses a target whose path crosses a symbolic link in a container
layer, a target that a container layer holds as another kind of file than the
share's, a repeated child, and overlapping targets, and any validation or mount
failure aborts the sandbox with status 125 instead of starting the workload
without its shares.
share's, and overlapping targets, and a one-shot guest a repeated child. Any
validation or mount failure aborts a one-shot sandbox with status 125, and
fails a managed `start`, instead of starting the workload without its shares.

An `rw` share supports the symbolic links that package managers and
language toolchains create; see
Expand Down Expand Up @@ -991,7 +1012,10 @@ A service manager can grant the same set, such as systemd's
operation succeeds only if its caller has OpenVMM's user and primary group and
OpenVMM's user belongs to no other group; every other operation fails with
`EPERM`. Hosts commonly grant access to `/dev/kvm` or `/dev/mshv` through such
a group, so `caller` usually needs both capabilities. Grant OpenVMM no other
a group, so `caller` usually needs both capabilities. A managed sandbox's guest
agent maps the shares before any workload runs, as guest root, which `caller`
squashes to the owner of the host directory, so a managed sandbox whose
OpenVMM cannot assume that identity fails to start. Grant OpenVMM no other
capabilities: `CAP_SETUID` lets it assume any host identity, which `caller`
mode confines to its export.

Expand Down Expand Up @@ -1246,9 +1270,11 @@ and should not be used for secrets. These options apply only to managed
private mount, PID, and UTS namespaces of its own, as the first process of its
PID namespace, through the guest agent's execution entry, which also makes
every mount but `/dev` and its devpts instance `nodev`. The workload sees one
machine ID for the life of the VM. On `stop`, the guest agent unmounts the live share,
overlay, layers, and scratch in dependency order before the VM powers off, as it
does when a one-shot workload exits. The legacy operation-less `sandbox`
machine ID for the life of the VM. On `stop`, the guest agent unmounts the
mappings of the live shares, and the guest then unmounts the host export,
overlay, layers, and scratch in dependency order before the VM powers off, as
it unmounts the shares when a one-shot workload exits. The legacy
operation-less `sandbox`
form is `sandbox run`; it remains one-shot and rejects `--state-dir` or any
request to retain VM state.

Expand Down
2 changes: 1 addition & 1 deletion doc/usage.md
Original file line number Diff line number Diff line change
Expand Up @@ -1198,7 +1198,7 @@ wins over a value that `nvx-rs` would refuse, even when that value comes first.
| `--environment-file PATH` | omitted | Read the exact managed `exec` environment from a UTF-8 JSON array of `KEY=VALUE` strings, limited to 1 MiB of input. An empty array requests an empty environment. This option is mutually exclusive with `--environment`; omitting both preserves guest defaults. |
| `--inherit-default-environment` | off | Layer the managed `exec` environment from `--environment` or `--environment-file` over the guest default environment instead of replacing it; an entry replaces the default variable of the same name. Without either option the workload already receives the defaults, so the flag has no effect. |
| `--hypervisor {auto,whp,kvm,mshv}` | `auto` | Select the host hypervisor. |
| `--mount GUEST_TARGET,HOST_PATH[,ro\|rw]` | none | Live-share a host directory, or a single regular host file without its directory, at the absolute target inside the container rootfs for `run` or `provision`; defaults to `ro`. The mode follows the last comma, so `HOST_PATH` may contain commas when the mode is given. Repeat to attach more shares, each with its own target and mode, for example a read-write workspace and a read-only tool cache, as the children of an aggregate that the guest binds at their targets; a shared file always uses the aggregate. Targets and host paths must not overlap, and the shares' kernel command-line tokens count toward the sandbox's 1024-byte budget. An `rw` share accepts guest-created symbolic links, which the host never follows. `/`, `/etc`, `/run`, and `/run/nvx`, and the `/etc/machine-id`, `/proc`, `/sys`, `/dev`, `/.nvx-agent`, and `/run/nvx/time` trees, are reserved. |
| `--mount GUEST_TARGET,HOST_PATH[,ro\|rw]` | none | Live-share a host directory, or a single regular host file without its directory, at the absolute target inside the container rootfs for `run` or `provision`; defaults to `ro`. The mode follows the last comma, so `HOST_PATH` may contain commas when the mode is given. Repeat to attach more shares, each with its own target and mode, for example a read-write workspace and a read-only tool cache, as the children of an aggregate that the guest binds at their targets; a shared file always uses the aggregate, and so does every share of a managed sandbox. Targets and host paths must not overlap. A one-shot `run` describes its shares in kernel command-line tokens that count toward the sandbox's 1024-byte budget; a managed sandbox announces only their number and sends their mapping table once its guest agent is ready, for at most 256 shares. An `rw` share accepts guest-created symbolic links, which the host never follows. `/`, `/etc`, `/run`, and `/run/nvx`, and the `/etc/machine-id`, `/proc`, `/sys`, `/dev`, `/.nvx-agent`, and `/run/nvx/time` trees, are reserved. |
| `--mount-deny HOST_PATH` | none | Hide one existing file or directory inside a `--mount` host directory; relative paths are resolved inside it. It may name the directory itself when `--mount-allow` paths inside it remain. With several shares, it applies to the `--mount` before it, which must share a directory. Repeat to deny multiple paths. |
| `--mount-allow HOST_PATH` | none | Expose one existing file or directory inside a `--mount-deny` path of the same share again, for example a readable `mcp-payloads` directory inside hidden logs. The hidden directories on the way list only the entries that lead to it, and the workload cannot modify them. Relative paths are resolved inside the share; with several shares, it applies to the `--mount` before it. Repeat to allow multiple paths. See [Access policy](run.md#access-policy). |
| `--mount-write HOST_PATH` | none | Make one existing file or directory one of the only writable parts of an `rw` share; the workload's writes anywhere else in the share fail with `EROFS`. Relative paths are resolved inside the share; with several shares, it applies to the `--mount` before it, which must be `rw`. Repeat to declare multiple writable paths. `provision` persists the denied, allowed, and writable paths. |
Expand Down
Loading
Loading