From bb6564e9af48ffee9c4427ef76073fd1f21e10bf Mon Sep 17 00:00:00 2001 From: AlanD20 Date: Sun, 9 Aug 2026 09:31:50 +0200 Subject: [PATCH 0001/1457] docs: define jailed launch and service policies --- .../0014-routed-service-access-policies.md | 49 +++++++ docs/adr/0015-jailer-canonical-launcher.md | 52 +++++++ docs/roadmaps/TENANT_MICROVM.md | 135 ++++++++++++++++++ 3 files changed, 236 insertions(+) create mode 100644 docs/adr/0014-routed-service-access-policies.md create mode 100644 docs/adr/0015-jailer-canonical-launcher.md create mode 100644 docs/roadmaps/TENANT_MICROVM.md diff --git a/docs/adr/0014-routed-service-access-policies.md b/docs/adr/0014-routed-service-access-policies.md new file mode 100644 index 00000000..e366dc85 --- /dev/null +++ b/docs/adr/0014-routed-service-access-policies.md @@ -0,0 +1,49 @@ +# Routed Service-Access Policies + +**Status:** Active +**Date:** 2026-08-09 +**See also:** [Project Hosting with MicroVMs](../roadmaps/TENANT_MICROVM.md), [ADR-0009: Firewall Backend Mutual Exclusion](0009-firewall-backend-mutual-exclusion.md) + +## Context + +mvmctl networks are independent Linux bridges and subnets. VMs on one bridge communicate directly at layer 2, while traffic between bridges is routed by the host because IPv4 forwarding is enabled. The managed firewall currently permits outbound NAT and installs no default deny between mvmctl bridges. A workload network can therefore reach a shared-infrastructure network unless the host firewall blocks it. + +Project orchestration remains outside mvmctl. The infrastructure requirement is narrower: permit a source mvmctl network to reach one exact destination VM service, such as TCP port 5432 on a shared PostgreSQL VM, while denying other routed traffic between managed networks. The policy must survive VM restarts and firewall reconciliation without exposing raw nftables or iptables syntax. + +The persisted policy also needs a stable meaning if same-network filtering is added later. Routed packets belong in the IP-family FORWARD path. Same-bridge packets bypass that path because `bridge-nf-call-iptables=0` and would require nftables bridge-family enforcement. These mechanisms are additive, not replacements for one another. + +## Decision + +mvmctl will persist a typed **service-access policy** identified by source network, destination VM, protocol, and destination port or port range. + +- The policy stores resource identities, not interface names, IP addresses, chain names, or backend expressions. +- The API layer resolves the source network and destination VM, then passes trusted model values to the network domain. This preserves core-domain isolation. +- The network domain owns policy persistence and compiles policies into derived firewall rules. `internal/lib/firewall` continues to own only backend-specific rule application and reconciliation. +- Different-network policies compile to IP-family FORWARD rules for both nftables and iptables. +- Explicit allows precede a final default DROP for traffic routed between mvmctl-managed bridges. Established and related reply traffic remains allowed, and internet egress remains unchanged. +- VM-to-host traffic is controlled separately in INPUT. Required NoCloud access and explicitly approved host services precede a final drop for traffic originating from mvmctl networks. +- Future same-network VM-to-VM policies will add a typed source-VM operation and compile to nftables bridge-family rules. Existing policies, commands, and stored records will not change. +- No raw firewall-rule editor, generic selector registry, or silent backend fallback will be introduced. + +## Consequences + +**Benefits:** + +- External orchestrators can model environments and shared services without introducing tenant concepts into mvmctl. +- Policy intent remains stable when VM IPs, TAP names, or firewall backends change. +- Routed policies work with both existing firewall backends. +- Bridge-family enforcement can be added without replacing the policy model. +- Idempotent reconciliation can restore rules after host reboot or UFW reload. + +**Costs and limitations:** + +- Rule ordering becomes a correctness requirement. Service allows and connection-state handling must always precede default deny. +- Removing or recreating a VM or network invalidates identity-bound policies; policies do not silently attach to replacement resources. +- VM-to-host restrictions require careful NoCloud and host-service system tests. +- Same-network filtering remains open until the bridge-family compiler is implemented. When implemented, that compiler will require nftables; routed policies retain iptables parity. + +## Related Decisions + +- ADR-0009: Firewall backend mutual exclusion remains unchanged. +- ADR-0010: The API layer resolves cross-domain policy inputs; Core domains remain isolated. +- ADR-0011: Policy inputs use typed `Validate` and `Resolve` methods. diff --git a/docs/adr/0015-jailer-canonical-launcher.md b/docs/adr/0015-jailer-canonical-launcher.md new file mode 100644 index 00000000..d6ab4941 --- /dev/null +++ b/docs/adr/0015-jailer-canonical-launcher.md @@ -0,0 +1,52 @@ +# Jailer Is the Canonical Firecracker Launcher + +**Status:** Active +**Date:** 2026-08-09 +**See also:** [Project Hosting with MicroVMs](../roadmaps/TENANT_MICROVM.md), [ADR-0005: Sudo Privilege Architecture](0005-sudo-privilege-architecture.md) + +## Context + +mvmctl downloads both Firecracker and Jailer from the same release but currently launches Firecracker directly. Direct launch preserves console file descriptors and simple PID tracking, but it omits Firecracker's recommended production process containment: chroot, privilege drop, namespace isolation, cgroup placement, and process resource limits. + +Firecracker's production guidance states that Jailer and Firecracker must have matching versions and that all trusted executable and chroot paths must not be writable by unprivileged users. Jailer requires elevated permissions to prepare the jail and cgroups, while mvmctl's established contract is that users invoke the CLI without root and privileged work is delegated internally through the constrained service-subprocess pattern. + +The launch mechanism affects VM creation, restart, snapshot restore, console descriptors, API and vsock sockets, logs, volume paths, PID tracking, cleanup, and performance. A fallback from failed Jailer launch to direct Firecracker would silently remove the expected security and resource guarantees. + +## Decision + +Jailer will be the canonical launcher for every normal mvmctl VM lifecycle operation. + +- Firecracker and Jailer are resolved and validated as one exact-version pair. Missing or mismatched pairs fail before launch. +- Jailer launch replaces direct Firecracker execution; there is no silent fallback and no generic launcher extension interface. +- The existing VM-domain spawner remains the caller-facing seam and owns jailed path translation, process tracking, socket probing, and cleanup. +- Root-requiring setup is encapsulated in a privileged `mvm run` subprocess, following the loopmount precedent. Users continue to invoke `mvm` without sudo. +- Executable and chroot paths trusted by Jailer are root-owned and not writable by the invoking user. The ordinary per-user mvmctl database and cache model remains unchanged. +- Jailer uses cgroup v2 and receives typed resource limits derived from the persisted VM resource specification. Raw cgroup key/value input is not public. +- Launch fails closed if required cgroup controllers are unavailable or post-spawn verification does not match requested limits. +- Jailer does not daemonize initially, preserving serial-console standard descriptors. New PID and network namespaces are deferred until their lifecycle and TAP implications are designed separately. +- VM resources are exposed inside the jail through validated paths without duplicating large rootfs and volume files. Mount and cgroup cleanup is idempotent across failed starts, stop, remove, and host recovery. + +## Consequences + +**Benefits:** + +- Every VM receives Firecracker's supported chroot, privilege-drop, and cgroup containment path. +- CPU, memory, swap, PID, and applicable I/O constraints become enforceable host limits rather than advisory guest configuration. +- One canonical launch path avoids a permanent direct-versus-jailed behavior matrix. +- VM create, start, reboot, and snapshot restore share the same security properties. + +**Costs and limitations:** + +- Existing host paths in Firecracker configuration must be translated into jail-visible paths while host controllers retain corresponding socket and artifact locations. +- Console, vsock, snapshots, hotplug volumes, PID handling, and partial-failure cleanup require full parity testing. +- Jailer adds fixed launch overhead and depends on the number of host mount points; benchmarks must track the regression budget. +- Memory cgroup limits require measured VMM headroom above guest-visible RAM. +- Network namespace isolation remains future work because the current TAP belongs to a host bridge. +- Hosts must rerun initialization when trusted launch assets or cgroup-v2 prerequisites are introduced. + +## Related Decisions + +- ADR-0002: Privileged Jailer work remains inside the single `mvm` binary through a service subcommand. +- ADR-0005: Normal-user operation and constrained internal privilege escalation remain mandatory. +- ADR-0012: Real Jailer/cgroup behavior requires L2 verification. +- ADR-0013: Socket readiness remains an internal probe, not a total launch timeout. diff --git a/docs/roadmaps/TENANT_MICROVM.md b/docs/roadmaps/TENANT_MICROVM.md new file mode 100644 index 00000000..69539c3a --- /dev/null +++ b/docs/roadmaps/TENANT_MICROVM.md @@ -0,0 +1,135 @@ +# Project Hosting with MicroVMs + +**Status:** Draft +**Date:** 2026-08-09 +**Audience:** mvmctl contributors and operators using microVMs as isolated project-service runtimes. +**Related ADRs:** [ADR-0014: Routed Service-Access Policies](../adr/0014-routed-service-access-policies.md), [ADR-0015: Jailer Is the Canonical Firecracker Launcher](../adr/0015-jailer-canonical-launcher.md). + +## 1. Goal and ownership boundary + +mvmctl remains a low-level, normal-user VM manager. It does not own projects, tenants, releases, service placement, credentials, or application orchestration. An external orchestrator maps each environment to an mvmctl network and chooses whether tightly coupled services share one microVM or run in separate microVMs on that network. + +Shared infrastructure runs in dedicated microVMs on separate mvmctl networks. Workload networks reach an exact shared service only through explicit protocol-and-port policies. The primary example is an application network reaching TCP port 5432 on a shared PostgreSQL VM; the model applies equally to other shared infrastructure. + +The microVM is the process, filesystem, and resource-isolation unit. The environment network is the communication trust zone unless a future same-network VM policy narrows it. + +## 2. Deployment model + +An environment maps to one network. Small environments may run application, worker, scheduler, and cache processes in one Docker-capable microVM. Services move to separate microVMs when they need an independent failure domain, resource envelope, deployment cadence, or stronger isolation. Service VMs on one environment network communicate directly. + +Shared infrastructure uses a separate network and dedicated microVMs with persistent volumes. The host routes between networks, but a default deny permits only persisted service-access policies. Database authentication, role separation, and application-aware backup remain the external orchestrator's responsibility. + +Direct inbound ports are not required when the workload VM runs an outbound tunnel such as cloudflared. Host port forwarding remains a separate future capability. + +## 3. Current capabilities and gaps + +| Capability | Current state | Required change | +|---|---|---| +| Same-network communication | Works through one bridge; intentionally unfiltered | None for the environment trust-zone model | +| Cross-network routing | Host forwarding and ACCEPT policy permit it | Persist explicit service allows and default-deny other managed inter-network traffic | +| VM-to-host traffic | INPUT remains open except for explicit NoCloud allow rules | Preserve required host services, then deny other traffic from managed VM networks | +| Firecracker launch | Firecracker and Jailer are downloaded together; Firecracker is launched directly | Make Jailer the canonical launch path | +| Host resource enforcement | Guest vCPU and RAM are configured; no Jailer cgroup limits are applied | Apply and verify typed cgroup-v2 limits through Jailer | +| IP identity | Explicit IP assignment and collision-safe IPAM exist | Policies bind to network/VM IDs and resolve current addressing during reconciliation | +| Persistent storage | Volumes and snapshots exist | External orchestration supplies application-consistent backup and restore procedures | +| Docker workload | Guest images and kernels are customizable | Validate a versioned Docker-capable image/kernel template | + +## 4. Implementation milestones + +### M1 — Canonical jailed launch + +- Resolve Firecracker and Jailer as an exact-version pair. +- Run privileged setup through a constrained `mvm run` service while preserving normal-user CLI operation. +- Install trusted launch executables at root-owned, non-user-writable paths. +- Translate kernel, rootfs, volume, socket, log, metrics, and snapshot paths into the jail. +- Preserve console descriptors, PID/start-time tracking, vsock, stop, and cleanup. +- Do not daemonize and do not enable PID or network namespaces in this milestone. + +**Exit:** create, start, stop, reboot, console, exec, cp, volumes, and snapshot restore have L2 parity under Jailer, with no direct-launch fallback. + +### M2 — Enforced resource envelope + +- Require cgroup v2 and the necessary controllers during host readiness checks. +- Persist typed CPU, memory, swap, PID, and applicable I/O limits with each VM. +- Derive Jailer cgroup arguments internally; do not expose raw cgroup files or values. +- Measure Firecracker host-memory overhead before selecting the default memory headroom. +- Verify actual cgroup membership and values after launch; fail closed on mismatch. +- Surface requested, actual, and current resource state in VM inspection. + +**Exit:** every normally launched VM has verified cgroup-v2 constraints and cleanup removes its cgroup. + +### M3 — Routed service-access policies + +- Persist source-network → destination-VM → protocol/port intent in the network domain. +- Resolve cross-domain identities in the API layer. +- Compile explicit allows into both existing IP-family firewall backends. +- Add deterministic connection-state, allow, and default-deny ordering in dedicated managed chains. +- Restrict VM-to-host INPUT while preserving NoCloud and explicitly approved host services. +- Reconcile desired rules after reboot, UFW reload, resource deletion, and `network sync`. + +**Exit:** an environment network can reach an exact shared service port, cannot reach other ports or managed networks, retains internet egress, and cannot reach unauthorized host services. + +### M4 — Workload proof and recovery + +- Validate a Docker-capable guest kernel and versioned base image. +- Run one workload environment and one shared-infrastructure VM on separate networks. +- Verify reboot ordering, health probes, persistent volumes, and application-aware backups. +- Document cold restore onto a clean host. +- Benchmark jailed launch and firewall reconciliation overhead. + +**Exit:** the external orchestrator can reproduce, stop, restore, and diagnose the topology without mvmctl owning project concepts. + +### M5 — Optional same-network segmentation + +- Add a typed source-VM → destination-VM service policy only when an environment needs it. +- Compile same-bridge policies to nftables bridge-family rules. +- Preserve the policy identity model and routed compiler from M3. + +**Exit:** service-per-VM environments can narrow their internal trust zone without policy migration. + +## 5. Administrator quality of life + +The highest-value mvmctl additions are infrastructure diagnostics, not project orchestration: + +1. Policy list, inspect, remove, sync, and connectivity-check commands. +2. VM inspection showing Jailer pair, jail path, cgroup enforcement, and current usage. +3. Host readiness output for Jailer, trusted binaries, cgroup v2, KVM, and required controllers. +4. Desired-versus-installed firewall rule diagnostics. +5. Idempotent cleanup of stale jails, mounts, cgroups, sockets, and derived rules. +6. Clear resource-exhaustion and policy-denial errors rather than generic launch/connectivity failures. + +## 6. Maintenance risks and controls + +| Risk | Control | +|---|---| +| Jailer path and mount complexity | One VM-domain path translator; no caller-built jailed paths | +| User-writable privileged executables | Root-owned verified Firecracker/Jailer installation | +| Memory-limit regressions after Firecracker upgrades | Versioned benchmarks and explicit overhead setting | +| Firewall rule ordering | Dedicated managed chains and deterministic full reconciliation | +| Policy drift after UFW reload | Idempotent sync and desired/actual inspection | +| VM proliferation | External orchestrator uses hybrid placement rather than one VM per trivial process | +| Shared-service blast radius | Dedicated volume, capacity reserve, service-native auth, backups, and health checks | +| Subnet waste | External orchestrator allocates small environment subnets where appropriate | +| Troubleshooting layers | Typed policy checks, resource inspection, and lifecycle audit logs | + +## 7. Non-goals + +- Tenant or project entities in mvmctl. +- A root-owned mvmctl database, daemon, RBAC, billing, or web UI. +- Docker Compose orchestration inside mvmctl. +- Jailer network namespaces in the initial launch implementation. +- Live VM migration. +- Same-network security groups before a concrete environment requires them. +- Application-consistent database backup inside mvmctl. + +## 8. References + +- ADR-0002: Single Go binary and privileged service-subcommand pattern. +- ADR-0005: Normal-user sudo privilege architecture. +- ADR-0009: nftables/iptables mutual exclusion. +- ADR-0010: API layer as sole cross-domain orchestrator. +- ADR-0011: Typed input validation and resolution. +- ADR-0012: L0/L1/L2 verification requirements. +- [ADR-0014: Routed Service-Access Policies](../adr/0014-routed-service-access-policies.md). +- [ADR-0015: Jailer Is the Canonical Firecracker Launcher](../adr/0015-jailer-canonical-launcher.md). +- Firecracker production guidance: `docs/prod-host-setup.md` and `docs/jailer.md` in the Firecracker repository. From 42e46a5670bcbb123105809ba3cae64782a3eaea Mon Sep 17 00:00:00 2001 From: AlanD20 Date: Sun, 9 Aug 2026 12:53:14 +0200 Subject: [PATCH 0002/1457] feat(vm): launch microVMs through jailer --- AGENTS.md | 2 +- CHANGELOG.md | 6 + CONTEXT.md | 8 +- docs/DEPENDENCIES.md | 12 +- docs/REFERENCES.md | 12 +- docs/RUNTIME.md | 23 +- docs/STANDARDS.md | 2 +- docs/TROUBLESHOOTING.md | 7 +- docs/adr/0005-sudo-privilege-architecture.md | 9 +- docs/adr/0015-jailer-canonical-launcher.md | 13 +- .../HOW_AGENTS_WRITE_SYSTEM_TESTS.md | 1 + docs/improvements/IMPROVEMENTS_010.md | 4 +- .../MOVING_TO_AGNOSTIC_ARCHITECTURE.md | 2 +- .../optimizations/next-level-optimizations.md | 8 +- docs/roadmaps/TENANT_MICROVM.md | 2 +- internal/cli/service.go | 39 + internal/core/binary/service.go | 124 ++- internal/core/binary/service_test.go | 164 ++++ internal/core/cache/utils.go | 2 +- internal/core/vm/controller.go | 39 +- internal/core/vm/firecracker.go | 191 +++-- internal/core/vm/jailer.go | 107 +++ internal/core/vm/jailer_test.go | 146 ++++ internal/core/vm/sqlite.go | 21 +- internal/core/vm/sqlite_test.go | 45 ++ internal/infra/constants.go | 9 + .../lib/db/migrations/002_jailer_launch.sql | 22 + internal/lib/db/migrations_jailer_test.go | 75 ++ internal/lib/model/binary.go | 6 + internal/lib/model/firecracker.go | 11 +- internal/lib/model/snapshot.go | 1 + internal/lib/model/vm.go | 56 +- internal/service/jailer/entry.go | 763 ++++++++++++++++++ internal/service/jailer/entry_test.go | 270 +++++++ internal/service/jailer/spawn.go | 127 +++ internal/service/jailer/spawn_test.go | 40 + internal/service/jailer/wire.go | 33 + pkg/api/binary.go | 49 +- pkg/api/inputs/vm_create.go | 1 + pkg/api/snapshot.go | 11 +- pkg/api/vm.go | 37 +- pkg/errs/codes.go | 21 +- pkg/errs/domain.go | 42 +- tests/system/COVERAGE_MATRIX.md | 2 + tests/system/bin/test_bin.py | 20 +- .../full_journeys/test_full_journeys.py | 8 + tests/system/invariants/test_invariants.py | 22 + tests/system/vm/test_jailer.py | 274 +++++++ 48 files changed, 2705 insertions(+), 184 deletions(-) create mode 100644 internal/core/binary/service_test.go create mode 100644 internal/core/vm/jailer.go create mode 100644 internal/core/vm/jailer_test.go create mode 100644 internal/core/vm/sqlite_test.go create mode 100644 internal/lib/db/migrations/002_jailer_launch.sql create mode 100644 internal/lib/db/migrations_jailer_test.go create mode 100644 internal/service/jailer/entry.go create mode 100644 internal/service/jailer/entry_test.go create mode 100644 internal/service/jailer/spawn.go create mode 100644 internal/service/jailer/spawn_test.go create mode 100644 internal/service/jailer/wire.go create mode 100644 tests/system/vm/test_jailer.py diff --git a/AGENTS.md b/AGENTS.md index 767bbd5d..e54c0217 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -56,7 +56,7 @@ The architect MUST verify these before approving any implementation plan: - Core domains NEVER import other core/* packages. Only `internal/lib/model/` is shared across domains. - Controller = state management per entity (start/stop/pause/resume/snapshot). No remove(), no create(). - Service does NOT validate caller input. Caller validates, receiver trusts. -- ALL subprocess calls through `system.DefaultRunner.Run()` / `system.DefaultRunner.Stream()` with `system.RunCmdOpts` — no raw `os/exec`. Documented exceptions (see CONTEXT.md "Subprocess invocation"): Firecracker spawn (FD redirection), SSH probe (connectivity detection), loopmount provisioner (chained losetup/mount/umount), vsock agent exec/pty (su/sh command execution), archive xz decompression (pipe-based), and the runner/spawn abstraction boundary itself (`runner.go`, `interactive_run.go`, `spawn.go`). +- ALL subprocess calls through `system.DefaultRunner.Run()` / `system.DefaultRunner.Stream()` with `system.RunCmdOpts` — no raw `os/exec`. Documented exceptions (see CONTEXT.md "Subprocess invocation"): Jailer exec (privileged chroot handoff with inherited console descriptors), SSH probe (connectivity detection), loopmount provisioner (chained losetup/mount/umount), vsock agent exec/pty (su/sh command execution), archive xz decompression (pipe-based), and the runner/spawn abstraction boundary itself (`runner.go`, `interactive_run.go`, `spawn.go`). - Context propagation: every repository method, every infrastructure function with side effects takes `ctx context.Context` as its first parameter. - The API layer (`pkg/api/`) is the SOLE orchestrator of multiple core domains. - Validation lives in API `pkg/api/inputs/` `*Input` structs with `Validate()`/`Resolve()`, not in Service/Controller. diff --git a/CHANGELOG.md b/CHANGELOG.md index 499457a2..415bfdd8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +### Added + +#### Canonical Firecracker Jailer launch +- VM create, start, reboot, and snapshot restore now require an exact Firecracker/Jailer release pair and run through a root-owned Jailer chroot with no direct-launch fallback. +- Trusted release pairs are installed from the checksum-verified archive; each jail exposes only that VM's kernel, rootfs, volumes, runtime files, and selected snapshot. + ### Fixed #### `mvm image inspect` diff --git a/CONTEXT.md b/CONTEXT.md index 42c47714..4260ba19 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -98,14 +98,14 @@ Constructor pattern: `NewRepository(db *sqlx.DB) Repository`. ### Service subprocess pattern (internal/service/) -Long-running subprocess services (console relay, nocloud-net server, loopmount provisioner) live in `internal/service/{name}/`. These are compiled into the same `mvm` binary — no separate binaries. The CLI layer has an `mvm run ` subcommand that serves as the entry point for each service. An additional embedded service (`agent/`) provides a cross-compiled guest agent binary that is compressed and embedded into the `mvm` binary at build time, then injected into the VM at runtime. +Long-running and privileged subprocess services (console relay, nocloud-net server, loopmount provisioner, Jailer launcher) live in `internal/service/{name}/`. These are compiled into the same `mvm` binary — no separate service binaries. The CLI layer has an `mvm run ` subcommand that serves as the entry point for each service. An additional embedded service (`agent/`) provides a cross-compiled guest agent binary that is compressed and embedded into the `mvm` binary at build time, then injected into the VM at runtime. Each service follows a consistent three-function pattern: - **`Config`** struct — holds all configuration for the service. - **`Run(ctx, cfg)`** — runs the service in the foreground (blocking). - **`Spawn(ctx, cfg, extraParams...)`** — launches the service as a background subprocess via `system.SpawnService()`. The context parameter is typically `nil` (background/nil) for daemon services (console relay, nocloud-net server) and a real context for synchronous services (loopmount provisioning). Extra parameters carry service-specific data: `console.Spawn()` passes a PTY file descriptor, `nocloudnet.Spawn()` passes the config only, `loopmount.Spawn()` passes a wire protocol input struct. -Services in `internal/service/`: `console/` (console relay PTY proxy), `nocloudnet/` (NoCloud HTTP metadata server), `loopmount/` (loop-mount provisioner wire protocol), `agent/` (embedded guest agent binary — cross-compiled, compressed, and injected into the VM at runtime via vsock). +Services in `internal/service/`: `console/` (console relay PTY proxy), `nocloudnet/` (NoCloud HTTP metadata server), `loopmount/` (loop-mount provisioner wire protocol), `jailer/` (trusted release installation, per-VM jail setup, and privileged Jailer exec), `agent/` (embedded guest agent binary — cross-compiled, compressed, and injected into the VM at runtime via vsock). Dependency direction: `cli/` -> `services/`. Services never import `cli/` or `pkg/api/`. @@ -409,7 +409,7 @@ Rationale: subprocess timeouts are a **safety net**, not a performance floor. Th | Location | Why `DefaultRunner.Run()` doesn't work | |---|---|---| -| `internal/core/vm/firecracker.go` (Firecracker spawn) | Fine-grained control over stdin/stdout/stderr FD redirection and `Setsid` session management for the Firecracker child process | +| `internal/service/jailer/entry.go` (Jailer exec) | Privileged chroot setup ends with `syscall.Exec` so Jailer inherits console stdin/stdout/stderr and becomes the managed Firecracker process | | `internal/core/ssh/utils.go` (SSH connectivity probe) | Uses `exec.CommandContext` with a short-lived probe context for SSH connectivity detection | | `internal/service/loopmount/provisioner.go` | Direct provisioning engine running losetup/mount/umount/chroot in chained operations with precise error recovery | | `internal/service/agent/exec.go` (command execution) | Uses `exec.CommandContext` for `su` user switching and `sh -c` command execution inside the guest agent | @@ -419,7 +419,7 @@ Rationale: subprocess timeouts are a **safety net**, not a performance floor. Th **Binary lookup carve-out:** Utility files across the codebase use `exec.LookPath()` (not `exec.Command`/`exec.CommandContext`) solely to check whether a system binary exists before calling it through `DefaultRunner.Run()`. This is NOT a subprocess execution — it's a filesystem existence check that happens to use the `os/exec` package. These files are not listed as exceptions above and do not violate the subprocess rule. Key locations: `internal/core/host/detector.go`, `internal/core/host/probe.go`, `internal/core/host/utils.go`, `internal/core/ssh/service.go`, `internal/core/key/utils.go`, `internal/core/image/service.go`, and others in `internal/lib/`, `internal/core/`, `internal/service/`. -Services running as subprocesses (`mvm run `) use `system.SpawnService(ctx, cfg)` which resolves the executable, optionally prepends `sudo`, and manages process groups. The services themselves (console relay, nocloudnet server, loopmount entry point) do NOT use `os/exec` except for the provisioning engine noted above. +Services running as subprocesses (`mvm run `) use `system.SpawnService(ctx, cfg)` which resolves the executable, optionally prepends `sudo`, and manages process groups. The Jailer service uses the documented `syscall.Exec` exception after validating trusted executables and per-VM resource paths; other services do not execute arbitrary child commands except the documented provisioning engine. ### Model types diff --git a/docs/DEPENDENCIES.md b/docs/DEPENDENCIES.md index e0b0188f..c053aac5 100644 --- a/docs/DEPENDENCIES.md +++ b/docs/DEPENDENCIES.md @@ -174,6 +174,10 @@ Only needed for `mvm kernel pull --type official --clean-build` (building the of - **Virtualization**: VT-x (Intel) or AMD-V must be enabled in BIOS/UEFI. - **Permissions**: The user must be in the `mvm` group (created by `mvm host init`). +### Firecracker Jailer + +`mvm bin pull` downloads and checksum-verifies a matching Firecracker/Jailer release pair. The privileged Jailer service installs the pair into the root-owned trusted store; users do not install or invoke Jailer directly. Source-built pairs remain untrusted and cannot launch production VMs. + --- ## H. Command Dependency Map @@ -191,10 +195,10 @@ Only needed for `mvm kernel pull --type official --clean-build` (building the of | `mvm kernel pull --type firecracker` | Download only (no build tools) | | `mvm key create` | `ssh-keygen` | | `mvm key import/ls/rm/inspect/export/default` | Internal only | -| `mvm bin pull/ls/rm/default` | Internal only (downloads from GitHub API) | -| `mvm vm create` | `firecracker`, `ip`, `iptables`/`nft`, `mvm run provision`, `losetup`, `blkid`, `blockdev`, `mount`, `umount`, `e2fsck`, `resize2fs`, `tune2fs`, `fstrim`, `chroot` (+ `btrfs` for btrfs images) | -| `mvm vm start/stop/reboot/pause/resume` | `firecracker`, `ip`, `iptables`/`nft` | -| `mvm vm rm` | `firecracker`, `ip`, `iptables`/`nft` | +| `mvm bin pull/ls/rm/default` | Internal downloads plus privileged `mvm run jailer install/remove-release` | +| `mvm vm create` | trusted `jailer` + `firecracker`, `ip`, `iptables`/`nft`, `mvm run provision`, `losetup`, `blkid`, `blockdev`, `mount`, `umount`, `e2fsck`, `resize2fs`, `tune2fs`, `fstrim`, `chroot` (+ `btrfs` for btrfs images) | +| `mvm vm start/stop/reboot/pause/resume` | trusted `jailer` + `firecracker`, `ip`, `iptables`/`nft` | +| `mvm vm rm` | `mvm run jailer cleanup`, `ip`, `iptables`/`nft` | | `mvm snapshot create/restore` | Internal (Firecracker API via Unix socket) | | `mvm volume attach/detach` | `firecracker` | | `mvm volume create/rm/resize` | `qemu-img` | diff --git a/docs/REFERENCES.md b/docs/REFERENCES.md index f1fbb0ce..371d43a1 100644 --- a/docs/REFERENCES.md +++ b/docs/REFERENCES.md @@ -224,7 +224,7 @@ mvm image import base-img my-vm ### `mvm bin` -Firecracker binary management — download, list, and remove Firecracker and jailer binaries. +Firecracker runtime management — download, list, and remove exact Firecracker/Jailer release pairs. Release downloads are checksum-verified and installed into the root-owned trusted store used by VM launch. Source-built pairs may be listed but cannot launch jailed VMs. #### Selectors @@ -721,7 +721,7 @@ The security model applies to the net cloud-init mode only: ``` ~/.cache/mvmctl/ -├── bin/ # Firecracker + jailer binaries +├── bin/ # Source-build and non-launch binary workspace ├── kernels/ # vmlinux kernel images ├── images/ # Root filesystem images (.ext4, .btrfs, .img, .raw, .ext4.zst, .btrfs.zst) ├── volumes/ # Persistent disk volume files @@ -749,3 +749,11 @@ The security model applies to the net cloud-init mode only: ├── audit.log # Rotating operation log (10MB, 3 backups) └── ... ``` + +Canonical launch assets and jail state are outside the user cache: + +```text +/var/lib/mvmctl/ +├── binaries// # Root-owned exact Firecracker/Jailer release pair +└── jailer/firecracker//root/ # Per-VM jail and bind mounts +``` diff --git a/docs/RUNTIME.md b/docs/RUNTIME.md index 9187812a..325134c7 100644 --- a/docs/RUNTIME.md +++ b/docs/RUNTIME.md @@ -281,7 +281,7 @@ inside the guest VM, not on the host. ### 3.1 Service Architecture -Three foreground services and one embedded guest agent share the same binary. +Four foreground services and one embedded guest agent share the same binary. No separate service binaries, no symlinks, no extraction step. | Service | Entry Point | Runs As | Purpose | @@ -289,6 +289,7 @@ No separate service binaries, no symlinks, no extraction step. | `mvm run console relay` | `console.Run(ctx, cfg)` | user | PTY-to-socket relay for serial console | | `mvm run nocloudnet serve` | `nocloudnet.Run(ctx, cfg)` | user | HTTP server for cloud-init nocloud-net | | `mvm run provision` | `loopmount.Run(ctx, cfg)` | **root** (sudo) | Loop-mount rootfs provisioning | +| `mvm run jailer` | `jailer.Run(ctx, cfg)` | **root** (sudo) | Trusted release installation, jail setup, and Firecracker launch | | `agent/` (embedded) | Guest agent binary | root (in-VM) | Command execution and file transfer inside the guest | The vsock agent is cross-compiled at build time, zstd-compressed, embedded via @@ -404,7 +405,24 @@ API layer → provisioner.NewBackend() → LoopMountBackend raw block device operations. The JSON protocol ensures no state leaks between invocations. -### 3.5 Service Lifecycle +### 3.5 Firecracker Jailer Service + +Jailer is the only VM launch path. `mvm bin pull` streams the checksum-verified Firecracker release archive to the privileged installer, which atomically installs the exact Firecracker/Jailer pair under `/var/lib/mvmctl/binaries//`. VM launch then exposes only that VM's required resources inside `/var/lib/mvmctl/jailer/firecracker//root/` and execs the trusted Jailer binary. + +Jailer does not daemonize, so the serial-console descriptors remain attached. The service derives the non-root Firecracker UID/GID from the sudo caller, translates Firecracker configuration paths into the jail, and records the actual Firecracker PID. Stop, remove, reboot, hotplug, and snapshot operations reconcile the corresponding jail mounts. + +There is no direct-Firecracker fallback. Jailer cgroup-v2 resource tuning and network namespaces are separate follow-up work. + +**Key files:** + +| File | Purpose | +|------|---------| +| `internal/service/jailer/entry.go` | Validates privileged operations, installs trusted pairs, mounts resources, and execs Jailer | +| `internal/service/jailer/spawn.go` | Typed normal-user calls into the privileged service | +| `internal/core/vm/jailer.go` | Per-VM manifest generation and host-to-jail path translation | +| `internal/core/vm/firecracker.go` | Firecracker configuration and caller-facing spawn lifecycle | + +### 3.6 Service Lifecycle | Phase | Action | Component | |-------|--------|-----------| @@ -412,6 +430,7 @@ invocations. | **Create VM** | Provision rootfs via loop-mount or guestfs | `backend.Run()` | | **Create VM** | Start NoCloud server (net mode) or inject cloud-init (inject mode) | `nocloudnet.Spawn()` / `backend.InjectCloudInit()` | | **Create VM** | Start console relay (when `--console` is set) | `console.Spawn()` | +| **Create/start/restore VM** | Launch exact Firecracker/Jailer pair in per-VM jail | `jailer.Launch()` | | **Remove VM** | Stop console relay + NoCloud server + clean firewall rules | `console.Stop()`, `nocloudnet.Stop()` | | **Cache prune** | Clean up stale PID files + orphan processes | `cache.Service.Prune()` | diff --git a/docs/STANDARDS.md b/docs/STANDARDS.md index 877020d7..d7b89b4f 100644 --- a/docs/STANDARDS.md +++ b/docs/STANDARDS.md @@ -183,7 +183,7 @@ Documented exceptions for direct `os/exec` / `os/exec.CommandContext`: | Location | Why DefaultRunner doesn't work | |----------|-------------------------------| -| `internal/core/vm/firecracker.go` | Needs `pass_fds` for VM API socket and log file descriptors | +| `internal/service/jailer/entry.go` | Privileged Jailer handoff uses `syscall.Exec` to preserve console stdio and process identity after chroot setup | | `internal/core/ssh/utils.go` | SSH connectivity probe uses `exec.CommandContext` with short-lived probe context | | `internal/service/loopmount/provisioner.go` | Direct provisioning engine with chained losetup/mount/umount/chroot | | `internal/service/agent/exec.go` | Guest agent command execution via `su`/`sh` with `exec.CommandContext` | diff --git a/docs/TROUBLESHOOTING.md b/docs/TROUBLESHOOTING.md index c72d51cc..6352cfff 100644 --- a/docs/TROUBLESHOOTING.md +++ b/docs/TROUBLESHOOTING.md @@ -145,7 +145,12 @@ mvm logs myvm --os **Common causes:** -**1. Missing or broken jailer.** Firecracker requires the `jailer` binary alongside it. If the binary was fetched with `mvm bin pull`, it should be bundled, but manual installs may miss it. +**1. Missing, mismatched, or untrusted Jailer pair.** VM launch requires the exact checksum-verified Firecracker/Jailer release pair in the root-owned trusted store. Re-pull the selected release; source-built or manually copied binaries cannot launch jailed VMs: + +```bash +mvm bin ls --long +mvm bin pull --force +``` **2. Kernel file is not readable by Firecracker.** Verify the kernel exists and the path is correct: ```bash diff --git a/docs/adr/0005-sudo-privilege-architecture.md b/docs/adr/0005-sudo-privilege-architecture.md index 3d6f6c89..91523710 100644 --- a/docs/adr/0005-sudo-privilege-architecture.md +++ b/docs/adr/0005-sudo-privilege-architecture.md @@ -51,14 +51,16 @@ The sudoers file is generated by `GenerateSudoersContent()` using two sources fr - `/usr/sbin/sysctl` (procps) - `/usr/sbin/modprobe` (kmod) -- **Service binaries** — the `mvm` binary itself for the `mvm run provision` subcommand: +- **Service binaries** — the `mvm` binary itself for privileged `mvm run provision` and `mvm run jailer` subcommands: - `mvm` path resolved via `os.Executable()` + ` *` wildcard suffix NOTHING else is permitted. `sudo -n true`, `sudo cat /etc/sudoers.d/mvm`, `sudo test -f ...`, and any other non-listed command WILL prompt for a password. This is by design and is NOT a configuration problem. -## How VM Creation Works Without losetup in Sudoers +## How VM Creation Encapsulates Privileged Work -This is the critical design insight: **all loop-mount operations are delegated to the `mvm run provision` service subprocess**. The call chain is: +All loop-mount operations are delegated to `mvm run provision`. Trusted Firecracker release installation, jail mounts, cleanup, and Jailer execution are delegated to `mvm run jailer`. Both reuse the single approved `mvm` sudoers entry instead of granting direct sudo access to dynamic binaries or mount tools. + +The loop-mount call chain is: ``` mvm vm create @@ -84,6 +86,7 @@ This means: - The sudoers file does NOT need `losetup`, `mount`, `umount`, `mkfs.ext4`, `blkid`, or any filesystem tool - ALL filesystem operations are encapsulated in the compiled `mvm` binary's `run provision` subcommand - The `mvm` binary is the single sudoers entry point for VM creation +- Jailer and Firecracker execute only from root-owned trusted release paths; users still invoke `mvm` without sudo ## How System Tests Interact with Sudo diff --git a/docs/adr/0015-jailer-canonical-launcher.md b/docs/adr/0015-jailer-canonical-launcher.md index d6ab4941..aec079f3 100644 --- a/docs/adr/0015-jailer-canonical-launcher.md +++ b/docs/adr/0015-jailer-canonical-launcher.md @@ -6,7 +6,7 @@ ## Context -mvmctl downloads both Firecracker and Jailer from the same release but currently launches Firecracker directly. Direct launch preserves console file descriptors and simple PID tracking, but it omits Firecracker's recommended production process containment: chroot, privilege drop, namespace isolation, cgroup placement, and process resource limits. +mvmctl originally downloaded both Firecracker and Jailer but launched Firecracker directly. Direct launch preserved console file descriptors and simple PID tracking while omitting Firecracker's recommended production process containment. Firecracker's production guidance states that Jailer and Firecracker must have matching versions and that all trusted executable and chroot paths must not be writable by unprivileged users. Jailer requires elevated permissions to prepare the jail and cgroups, while mvmctl's established contract is that users invoke the CLI without root and privileged work is delegated internally through the constrained service-subprocess pattern. @@ -21,17 +21,16 @@ Jailer will be the canonical launcher for every normal mvmctl VM lifecycle opera - The existing VM-domain spawner remains the caller-facing seam and owns jailed path translation, process tracking, socket probing, and cleanup. - Root-requiring setup is encapsulated in a privileged `mvm run` subprocess, following the loopmount precedent. Users continue to invoke `mvm` without sudo. - Executable and chroot paths trusted by Jailer are root-owned and not writable by the invoking user. The ordinary per-user mvmctl database and cache model remains unchanged. -- Jailer uses cgroup v2 and receives typed resource limits derived from the persisted VM resource specification. Raw cgroup key/value input is not public. -- Launch fails closed if required cgroup controllers are unavailable or post-spawn verification does not match requested limits. +- Typed cgroup-v2 resource enforcement will extend this canonical launch path in a separate change. Raw cgroup key/value input will not be public. - Jailer does not daemonize initially, preserving serial-console standard descriptors. New PID and network namespaces are deferred until their lifecycle and TAP implications are designed separately. -- VM resources are exposed inside the jail through validated paths without duplicating large rootfs and volume files. Mount and cgroup cleanup is idempotent across failed starts, stop, remove, and host recovery. +- VM resources are exposed inside the jail through validated paths without duplicating large rootfs and volume files. Mount cleanup is idempotent across failed starts, stop, remove, and host recovery. ## Consequences **Benefits:** -- Every VM receives Firecracker's supported chroot, privilege-drop, and cgroup containment path. -- CPU, memory, swap, PID, and applicable I/O constraints become enforceable host limits rather than advisory guest configuration. +- Every VM receives Firecracker's supported chroot and privilege-drop path. +- The canonical Jailer path provides the required seam for follow-up cgroup-v2 enforcement. - One canonical launch path avoids a permanent direct-versus-jailed behavior matrix. - VM create, start, reboot, and snapshot restore share the same security properties. @@ -42,7 +41,7 @@ Jailer will be the canonical launcher for every normal mvmctl VM lifecycle opera - Jailer adds fixed launch overhead and depends on the number of host mount points; benchmarks must track the regression budget. - Memory cgroup limits require measured VMM headroom above guest-visible RAM. - Network namespace isolation remains future work because the current TAP belongs to a host bridge. -- Hosts must rerun initialization when trusted launch assets or cgroup-v2 prerequisites are introduced. +- Release pairs are reinstalled into the trusted store by `mvm bin pull`; cgroup-v2 prerequisites are introduced separately. ## Related Decisions diff --git a/docs/development/HOW_AGENTS_WRITE_SYSTEM_TESTS.md b/docs/development/HOW_AGENTS_WRITE_SYSTEM_TESTS.md index e0531a87..73aa9a74 100644 --- a/docs/development/HOW_AGENTS_WRITE_SYSTEM_TESTS.md +++ b/docs/development/HOW_AGENTS_WRITE_SYSTEM_TESTS.md @@ -255,6 +255,7 @@ Every CLI command and flag that a user can invoke must be tested. This table cla | `vm create --ip` / `--mac` | **L2** | Real Firecracker spawn | | `vm create --console` | **L2** | Real Firecracker spawn | | `vm create --no-pci` | **L2** | Real Firecracker spawn | +| canonical Jailer launch and cleanup | **L2** | Real chroot, trusted pair, lifecycle, volume, and snapshot behavior | | `vm create --enable-logging` / `--enable-metrics` | **L2** | Verify files on disk | | `vm create --cloud-init-mode ` | **L2** | Real Firecracker + cloud-init | | `vm create --nocloud-net-port` | **L2** | Real Firecracker + network | diff --git a/docs/improvements/IMPROVEMENTS_010.md b/docs/improvements/IMPROVEMENTS_010.md index 7bbe7b2f..61b47b6b 100644 --- a/docs/improvements/IMPROVEMENTS_010.md +++ b/docs/improvements/IMPROVEMENTS_010.md @@ -1,6 +1,6 @@ # Per-VM Network Namespaces with Firecracker Jailer -> **STATUS:** ⏳ Future — not implemented. No network namespace code, no jailer integration, no `NetNS` field in `FirecrackerConfig` model, no `--jailer`/`--namespace` CLI flags. The vsock agent exec approach (alternative described in §6) is the current method for snapshot restore networking. +> **STATUS:** ⏳ Network namespaces remain future work. Canonical Jailer launch, chroot, privilege drop, trusted release pairs, and jailed snapshot paths are implemented; `--netns`, veth topology, and a `NetNS` model remain unimplemented. > > **Last verified:** 2026-06-27 @@ -64,7 +64,7 @@ Currently, mvmctl spawns Firecracker directly: cmd = exec.Command(firecracker, "--api-sock", socketPath, ...) ``` -No jailer, no namespace. +Jailer is now canonical, but it joins the host network namespace. The namespace-specific topology below remains a proposal. ### 3.2 Spawn via Jailer with `--netns` diff --git a/docs/improvements/MOVING_TO_AGNOSTIC_ARCHITECTURE.md b/docs/improvements/MOVING_TO_AGNOSTIC_ARCHITECTURE.md index 9171d07a..66861d12 100644 --- a/docs/improvements/MOVING_TO_AGNOSTIC_ARCHITECTURE.md +++ b/docs/improvements/MOVING_TO_AGNOSTIC_ARCHITECTURE.md @@ -319,7 +319,7 @@ func (f *VMMFactory) CreateDriver(vmmType VMMType, vm *model.VM, repo vm.Reposit | **PCIe** | ❌ (limited PCI) | ✅ | ✅ | | **Boot speed** | Very fast | Fast | Slower (BIOS/EFI) | | **Memory overhead** | ~5MB per VM | ~15-30MB per VM | ~30-100MB per VM | -| **Process isolation** | Single process (+optional jailer) | Process-per-device (minijail) | Single process (+optional) | +| **Process isolation** | Canonical Firecracker Jailer | Process-per-device (minijail) | Single process (+optional) | | **Architectures** | x86_64, aarch64 | x86_64, aarch64, riscv64 | x86_64, aarch64, riscv64, arm, mips, ppc, s390x | --- diff --git a/docs/optimizations/next-level-optimizations.md b/docs/optimizations/next-level-optimizations.md index 946fca54..30882e72 100644 --- a/docs/optimizations/next-level-optimizations.md +++ b/docs/optimizations/next-level-optimizations.md @@ -215,13 +215,13 @@ Additional safe params that could still be added: `audit=0`, `elevator=noop`. **Status:** ✅ IMPLEMENTED — `cmd/mvm/main.go` produces `mvm` via `go build`. No interpreter startup overhead, no import time cost, immediate command execution. -### 4.5 os.posix_spawn() for Firecracker Subprocess ⏳ NOT IMPLEMENTED +### 4.5 Lower-Level Jailer Process Spawn ⏳ NOT IMPLEMENTED -**What it would be (Go equivalent):** Use `syscall.ForkExec()` or `os.StartProcess()` with pre-configured file descriptors instead of `exec.Command()` for Firecracker process creation. +**What it would be:** Replace the current `system.SpawnService()` handoff to the privileged Jailer service with a lower-level process primitive while preserving sudo, console descriptors, PID tracking, and the trusted-path contract. -**Current approach:** Uses `system.RunCmd` / `system.RunCmdOpts` (wrapping `exec.Command`) in the Firecracker spawner (`internal/core/vm/firecracker.go`). The spawner uses `SysProcAttr.Pdeathsig` for process lifecycle management. +**Current approach:** `internal/core/vm/firecracker.go` launches the privileged `mvm run jailer` service through `system.SpawnService()`. The service validates the per-VM manifest and uses `syscall.Exec` to hand off to the trusted Jailer/Firecracker pair without daemonization. -**Feasibility:** Low impact — Go's `os/exec` is already efficient for process spawning. The overhead is negligible compared to QEMU/Firecracker boot time. +**Feasibility:** Low impact. Jail setup and mount traversal dominate the process-spawn wrapper cost, so optimize only after Jailer launch benchmarks identify this handoff as material. ### 4.6 Pre-Allocated Resource Pools (General) ⏳ NOT IMPLEMENTED diff --git a/docs/roadmaps/TENANT_MICROVM.md b/docs/roadmaps/TENANT_MICROVM.md index 69539c3a..9102819e 100644 --- a/docs/roadmaps/TENANT_MICROVM.md +++ b/docs/roadmaps/TENANT_MICROVM.md @@ -28,7 +28,7 @@ Direct inbound ports are not required when the workload VM runs an outbound tunn | Same-network communication | Works through one bridge; intentionally unfiltered | None for the environment trust-zone model | | Cross-network routing | Host forwarding and ACCEPT policy permit it | Persist explicit service allows and default-deny other managed inter-network traffic | | VM-to-host traffic | INPUT remains open except for explicit NoCloud allow rules | Preserve required host services, then deny other traffic from managed VM networks | -| Firecracker launch | Firecracker and Jailer are downloaded together; Firecracker is launched directly | Make Jailer the canonical launch path | +| Firecracker launch | Canonical Jailer launch and exact trusted release pairing implemented; L2 execution remains pending | Add cgroup-v2 resource enforcement and complete runner verification | | Host resource enforcement | Guest vCPU and RAM are configured; no Jailer cgroup limits are applied | Apply and verify typed cgroup-v2 limits through Jailer | | IP identity | Explicit IP assignment and collision-safe IPAM exist | Policies bind to network/VM IDs and resolve current addressing during reconciliation | | Persistent storage | Volumes and snapshots exist | External orchestration supplies application-consistent backup and restore procedures | diff --git a/internal/cli/service.go b/internal/cli/service.go index 52f0cb29..6e144769 100644 --- a/internal/cli/service.go +++ b/internal/cli/service.go @@ -6,6 +6,7 @@ import ( "os" "mvmctl/internal/service/console" + "mvmctl/internal/service/jailer" "mvmctl/internal/service/loopmount" "mvmctl/internal/service/nocloudnet" @@ -30,10 +31,48 @@ func newRunCmd() *cobra.Command { cmd.AddCommand(newNoCloudNetCmd()) cmd.AddCommand(newConsoleCmd()) cmd.AddCommand(newProvisionCmd()) + cmd.AddCommand(newJailerServiceCmd()) return cmd } +func newJailerServiceCmd() *cobra.Command { + cmd := &cobra.Command{Use: "jailer", Short: "Run the privileged Firecracker Jailer service"} + for _, action := range []string{ + "install", "launch", "cleanup", "expose-snapshot", "expose-volume", "remove-volume", "remove-release", + } { + action := action + sub := &cobra.Command{Use: action, Short: "Run the " + action + " Jailer operation"} + sub.Flags().String("version", "", "Exact Firecracker release version") + sub.Flags().String("arch", "", "Firecracker release architecture") + sub.Flags().String("sha256", "", "Verified release archive SHA-256") + sub.Flags().String("vm-id", "", "VM ID") + sub.Flags().String("vm-dir", "", "Per-VM directory") + sub.Flags().String("snapshot-dir", "", "Snapshot directory") + sub.Flags().String("drive-id", "", "Firecracker drive ID") + sub.Flags().String("host-path", "", "Managed volume path") + sub.Flags().Bool("read-only", false, "Expose the managed volume read-only") + sub.RunE = func(c *cobra.Command, _ []string) error { + version, _ := c.Flags().GetString("version") + arch, _ := c.Flags().GetString("arch") + sha256, _ := c.Flags().GetString("sha256") + vmID, _ := c.Flags().GetString("vm-id") + vmDir, _ := c.Flags().GetString("vm-dir") + snapshotDir, _ := c.Flags().GetString("snapshot-dir") + driveID, _ := c.Flags().GetString("drive-id") + hostPath, _ := c.Flags().GetString("host-path") + readOnly, _ := c.Flags().GetBool("read-only") + return jailer.Run(c.Context(), jailer.Config{ + Action: action, Version: version, Arch: arch, SHA256: sha256, + VMID: vmID, VMDir: vmDir, SnapshotDir: snapshotDir, + DriveID: driveID, HostPath: hostPath, ReadOnly: readOnly, + }) + } + cmd.AddCommand(sub) + } + return cmd +} + func newNoCloudNetCmd() *cobra.Command { cmd := &cobra.Command{ Use: "nocloudnet", diff --git a/internal/core/binary/service.go b/internal/core/binary/service.go index b95af179..211d8536 100644 --- a/internal/core/binary/service.go +++ b/internal/core/binary/service.go @@ -9,16 +9,17 @@ import ( "regexp" "sort" "strings" + "syscall" "time" "mvmctl/internal/infra" "mvmctl/internal/infra/event" - "mvmctl/internal/lib/archive" "mvmctl/internal/lib/crypto" "mvmctl/internal/lib/download" "mvmctl/internal/lib/model" "mvmctl/internal/lib/system" "mvmctl/internal/lib/version" + jailersvc "mvmctl/internal/service/jailer" "mvmctl/pkg/errs" ) @@ -161,17 +162,15 @@ func (s *Service) DownloadFirecracker( ) } - // --- Step 3: Extract firecracker and jailer from .tgz --- - extractErr := archive.ExtractRenamed(ctx, tgzPath, []archive.RenameEntry{ - {ArchiveName: fmt.Sprintf("firecracker-v%s-%s", normalizedVersion, arch), OutputPath: fcDest, Mode: 0755}, - {ArchiveName: fmt.Sprintf("jailer-v%s-%s", normalizedVersion, arch), OutputPath: jlDest, Mode: 0755}, - }) + // The privileged installer rechecks the verified digest while consuming the same archive stream. + installed, installErr := jailersvc.InstallRelease(ctx, normalizedVersion, arch, tgzPath, expectedSHA256) os.Remove(tgzPath) - if extractErr != nil { - os.Remove(fcDest) - os.Remove(jlDest) - return nil, extractErr + if installErr != nil { + return nil, binaryError(errs.CodeBinaryTrustedInstallFailed, + fmt.Sprintf("failed to install trusted Firecracker/Jailer pair: %v", installErr)) } + fcDest = installed.FirecrackerPath + jlDest = installed.JailerPath // --- Step 4: Create BinaryItems --- fcBinary, err := s.createBinaryItem("firecracker", normalizedVersion, fcDest, true) @@ -211,7 +210,13 @@ func (s *Service) Remove(ctx context.Context, binary *model.BinaryItem, force bo ) } - if _, err := os.Stat(binary.Path); err == nil { + trustedPrefix := filepath.Clean(infra.TrustedBinaryRoot) + string(filepath.Separator) + if strings.HasPrefix(filepath.Clean(binary.Path), trustedPrefix) { + if err := jailersvc.RemoveRelease(ctx, binary.Version); err != nil { + return nil, binaryError(errs.CodeBinaryRemoveFailed, + fmt.Sprintf("failed to remove trusted release pair: %v", err)) + } + } else if _, err := os.Stat(binary.Path); err == nil { if err := os.Remove(binary.Path); err != nil { return nil, binaryError(errs.CodeInternal, fmt.Sprintf("Failed to remove binary file: %v", err)) } @@ -231,6 +236,103 @@ func (s *Service) Remove(ctx context.Context, binary *model.BinaryItem, force bo return binary, nil } +// ResolvePair resolves and validates the canonical exact-version launch pair. +func (s *Service) ResolvePair(ctx context.Context, firecracker *model.BinaryItem) (*model.BinaryPair, error) { + if firecracker == nil || firecracker.Type != "firecracker" || firecracker.Version == "" { + return nil, binaryError(errs.CodeBinaryPairInvalid, "invalid Firecracker binary selection") + } + pairedFirecracker, err := s.repo.GetByTypeAndVersion(ctx, "firecracker", firecracker.Version) + if err != nil { + return nil, binaryError(errs.CodeBinaryPairInvalid, + fmt.Sprintf("failed to resolve Firecracker binary: %v", err)) + } + pairedJailer, err := s.repo.GetByTypeAndVersion(ctx, "jailer", firecracker.Version) + if err != nil { + return nil, binaryError(errs.CodeBinaryPairInvalid, + fmt.Sprintf("failed to resolve Jailer binary: %v", err)) + } + pair := &model.BinaryPair{Firecracker: pairedFirecracker, Jailer: pairedJailer} + if pair.Firecracker == nil || pair.Jailer == nil || pair.Firecracker.ID != firecracker.ID || + pair.Firecracker.Version != pair.Jailer.Version { + return nil, binaryError(errs.CodeBinaryPairInvalid, + fmt.Sprintf("exact Firecracker/Jailer pair is missing or mismatched for version %s", firecracker.Version)) + } + for _, item := range []*model.BinaryItem{pair.Firecracker, pair.Jailer} { + if !item.IsPresent { + return nil, binaryError(errs.CodeBinaryPairInvalid, + fmt.Sprintf("%s %s is marked missing", item.Type, item.Version)) + } + if err := validateTrustedBinary(item); err != nil { + return nil, binaryError(errs.CodeBinaryUntrusted, err.Error()) + } + result, runErr := system.DefaultRunner.Run(ctx, []string{item.Path, "--version"}, system.RunCmdOpts{ + Check: true, Capture: true, + }) + if runErr != nil || !strings.Contains(result.Stdout+result.Stderr, item.Version) { + return nil, binaryError(errs.CodeBinaryPairInvalid, + fmt.Sprintf("%s executable does not report expected version %s", item.Type, item.Version)) + } + } + return pair, nil +} + +func validateTrustedBinary(item *model.BinaryItem) error { + cleanRoot := filepath.Clean(infra.TrustedBinaryRoot) + string(filepath.Separator) + cleanPath := filepath.Clean(item.Path) + if !strings.HasPrefix(cleanPath, cleanRoot) { + return fmt.Errorf( + "%s %s is not installed in the trusted root; source-built pairs are unsupported for canonical jailed launch", + item.Type, + item.Version, + ) + } + if err := validateTrustedParentChain(filepath.Dir(cleanPath)); err != nil { + return fmt.Errorf("%s trusted parent validation failed: %w", item.Type, err) + } + resolved, err := filepath.EvalSymlinks(cleanPath) + if err != nil || resolved != cleanPath { + return fmt.Errorf("%s trusted path is missing or contains a symlink", item.Type) + } + info, err := os.Stat(cleanPath) + if err != nil || !info.Mode().IsRegular() || info.Mode().Perm()&0111 == 0 || info.Mode().Perm()&0022 != 0 { + return fmt.Errorf("%s trusted executable is unusable", item.Type) + } + stat, ok := info.Sys().(*syscall.Stat_t) + if !ok || stat.Uid != 0 { + return fmt.Errorf("%s trusted executable is not root-owned", item.Type) + } + return nil +} + +func validateTrustedParentChain(path string) error { + abs, err := filepath.Abs(path) + if err != nil || abs != filepath.Clean(path) { + return fmt.Errorf("trusted directory path is not canonical: %s", path) + } + current := string(filepath.Separator) + components := strings.Split(strings.TrimPrefix(abs, string(filepath.Separator)), string(filepath.Separator)) + for i := -1; i < len(components); i++ { + if i >= 0 { + current = filepath.Join(current, components[i]) + } + info, err := os.Lstat(current) + if err != nil { + return fmt.Errorf("inspect trusted path component %s: %w", current, err) + } + if info.Mode()&os.ModeSymlink != 0 || !info.IsDir() { + return fmt.Errorf("trusted path component is not a real directory: %s", current) + } + if info.Mode().Perm()&0022 != 0 { + return fmt.Errorf("trusted path component is writable by non-root users: %s", current) + } + stat, ok := info.Sys().(*syscall.Stat_t) + if !ok || stat.Uid != 0 { + return fmt.Errorf("trusted path component is not root-owned: %s", current) + } + } + return nil +} + // RemoveMany removes multiple binaries. func (s *Service) RemoveMany( ctx context.Context, diff --git a/internal/core/binary/service_test.go b/internal/core/binary/service_test.go new file mode 100644 index 00000000..5534bb50 --- /dev/null +++ b/internal/core/binary/service_test.go @@ -0,0 +1,164 @@ +package binary_test + +import ( + "context" + "os" + "path/filepath" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "mvmctl/internal/core/binary" + "mvmctl/internal/lib/model" + "mvmctl/internal/testutil" +) + +// Rationale: Canonical launch must fail before subprocess execution unless the +// selected record resolves to one present exact-version Firecracker/Jailer pair. +func TestService_ResolvePairFailsClosed(t *testing.T) { + ctx := context.Background() + tests := map[string]struct { + selected *model.BinaryItem + seed []*model.BinaryItem + wantErr string + }{ + "nil_selection": {wantErr: "invalid Firecracker binary selection"}, + "wrong_binary_type": { + selected: binaryItem("selected", "jailer", "1.16.0", true), + wantErr: "invalid Firecracker binary selection", + }, + "missing_firecracker_record": { + selected: binaryItem("selected", "firecracker", "1.16.0", true), + seed: []*model.BinaryItem{binaryItem("jailer", "jailer", "1.16.0", true)}, + wantErr: "missing or mismatched", + }, + "missing_jailer_record": { + selected: binaryItem("selected", "firecracker", "1.16.0", true), + seed: []*model.BinaryItem{binaryItem("selected", "firecracker", "1.16.0", true)}, + wantErr: "missing or mismatched", + }, + "only_mismatched_jailer_version_exists": { + selected: binaryItem("selected", "firecracker", "1.16.0", true), + seed: []*model.BinaryItem{ + binaryItem("selected", "firecracker", "1.16.0", true), + binaryItem("jailer", "jailer", "1.15.0", true), + }, + wantErr: "missing or mismatched", + }, + "different_firecracker_identity": { + selected: binaryItem("selected", "firecracker", "1.16.0", true), + seed: []*model.BinaryItem{ + binaryItem("other", "firecracker", "1.16.0", true), + binaryItem("jailer", "jailer", "1.16.0", true), + }, + wantErr: "missing or mismatched", + }, + "jailer_marked_missing": { + selected: binaryItem("selected", "firecracker", "1.16.0", true), + seed: []*model.BinaryItem{ + binaryItem("selected", "firecracker", "1.16.0", true), + binaryItem("jailer", "jailer", "1.16.0", false), + }, + wantErr: "missing or mismatched", + }, + "pair_outside_trusted_root": { + selected: binaryItem("selected", "firecracker", "1.16.0", true), + seed: []*model.BinaryItem{ + binaryItem("selected", "firecracker", "1.16.0", true), + binaryItem("jailer", "jailer", "1.16.0", true), + }, + wantErr: "not installed in the trusted root", + }, + } + + for name, tc := range tests { + t.Run(name, func(t *testing.T) { + repo := testutil.NewBinaryRepo() + for _, item := range tc.seed { + require.NoError(t, repo.Upsert(ctx, item)) + } + svc := binary.NewService(repo, t.TempDir(), t.TempDir()) + + pair, err := svc.ResolvePair(ctx, tc.selected) + require.Error(t, err) + assert.Contains(t, err.Error(), tc.wantErr) + assert.Nil(t, pair) + }) + } +} + +// Rationale: A binary referenced by either side of persisted launch state must +// remain on disk and in the repository unless the caller explicitly forces removal. +func TestService_RemovePreservesReferencedBinary(t *testing.T) { + ctx := context.Background() + tests := map[string]struct { + attachReference func(*model.BinaryItem) + }{ + "firecracker_vm_reference": { + attachReference: func(item *model.BinaryItem) { + item.VMs = []*model.VMItem{{ID: "vm-id", Name: "vm-name", BinaryID: item.ID}} + }, + }, + "jailer_vm_reference": { + attachReference: func(item *model.BinaryItem) { + item.VMs = []*model.VMItem{{ID: "vm-id", Name: "vm-name", JailerBinaryID: item.ID}} + }, + }, + "snapshot_reference": { + attachReference: func(item *model.BinaryItem) { + item.Snapshots = []*model.SnapshotItem{{ID: "snapshot-id", Name: "snapshot-name", BinaryID: item.ID}} + }, + }, + } + + for name, tc := range tests { + t.Run(name, func(t *testing.T) { + repo := testutil.NewBinaryRepo() + path := filepath.Join(t.TempDir(), "firecracker") + require.NoError(t, os.WriteFile(path, []byte("release-binary"), 0755)) + item := binaryItem("binary-id", "firecracker", "1.16.0", true) + item.Path = path + tc.attachReference(item) + require.NoError(t, repo.Upsert(ctx, item)) + svc := binary.NewService(repo, t.TempDir(), t.TempDir()) + + removed, err := svc.Remove(ctx, item, false) + require.Error(t, err) + assert.Nil(t, removed) + data, readErr := os.ReadFile(path) + require.NoError(t, readErr) + assert.Equal(t, "release-binary", string(data)) + persisted, getErr := repo.Get(ctx, item.ID) + require.NoError(t, getErr) + assert.Same(t, item, persisted) + }) + } +} + +func TestService_RemoveUnreferencedBinary(t *testing.T) { + ctx := context.Background() + repo := testutil.NewBinaryRepo() + path := filepath.Join(t.TempDir(), "firecracker") + require.NoError(t, os.WriteFile(path, []byte("release-binary"), 0755)) + item := binaryItem("binary-id", "firecracker", "1.16.0", true) + item.Path = path + require.NoError(t, repo.Upsert(ctx, item)) + svc := binary.NewService(repo, t.TempDir(), t.TempDir()) + + removed, err := svc.Remove(ctx, item, false) + require.NoError(t, err) + assert.Same(t, item, removed) + _, statErr := os.Stat(path) + assert.ErrorIs(t, statErr, os.ErrNotExist) + persisted, getErr := repo.Get(ctx, item.ID) + require.NoError(t, getErr) + assert.Nil(t, persisted) +} + +func binaryItem(id, typ, version string, present bool) *model.BinaryItem { + return &model.BinaryItem{ + ID: id, Type: typ, Version: version, FullVersion: "v" + version, + Path: "/tmp/untrusted/" + typ, IsPresent: present, + } +} diff --git a/internal/core/cache/utils.go b/internal/core/cache/utils.go index 1537858d..4528594c 100644 --- a/internal/core/cache/utils.go +++ b/internal/core/cache/utils.go @@ -8,7 +8,7 @@ import ( // knownMVMComms lists mvm-managed process comm names. // Only Firecracker has its own binary — service subprocesses are identified // by the MVM_BACKGROUND_SERVICE=1 environment variable set in SpawnService. -var knownMVMComms = [...]string{"firecracker"} +var knownMVMComms = [...]string{"firecracker", "jailer"} // isMountPoint checks if a path is currently a mount point by comparing // device numbers (st_dev) of the path and its parent. diff --git a/internal/core/vm/controller.go b/internal/core/vm/controller.go index b9257a66..06c47123 100644 --- a/internal/core/vm/controller.go +++ b/internal/core/vm/controller.go @@ -35,6 +35,13 @@ func NewController(vm *model.VMItem, repo Repository) *Controller { // // First two code paths (non-running, process-gone) let errors propagate. func (c *Controller) Stop(ctx context.Context, force bool) error { + if c.vm.JailerBinaryID != "" { + defer func() { + if err := CleanupJail(context.Background(), c.vm.ID); err != nil { + slog.Warn("failed to clean VM jail after stop", "vm", c.vm.Name, "error", err) + } + }() + } pid := c.vm.PID // --- Non-running VMs: idempotent + orphan cleanup --- @@ -107,7 +114,7 @@ func (c *Controller) shutdownProcess(ctx context.Context, force bool, pid int) e if ctrlErr == nil && wasCtrlAltDel { exitCode = system.GracefulShutdown(system.ShutdownConfig{ Pid: pid, - IsChild: true, + IsChild: false, PreSignalHook: func() bool { return false }, GracefulTimeout: 2 * time.Second, KillTimeout: 100 * time.Millisecond, @@ -119,7 +126,7 @@ func (c *Controller) shutdownProcess(ctx context.Context, force bool, pid int) e if exitCode == nil { exitCode = system.GracefulShutdown(system.ShutdownConfig{ Pid: pid, - IsChild: true, + IsChild: false, GracefulTimeout: 2 * time.Second, KillTimeout: 1 * time.Millisecond, ExpectedStartTime: c.vm.ProcessStartTime, @@ -134,7 +141,7 @@ func (c *Controller) shutdownProcess(ctx context.Context, force bool, pid int) e // should die within milliseconds. exitCode = system.GracefulShutdown(system.ShutdownConfig{ Pid: pid, - IsChild: true, + IsChild: false, GracefulTimeout: 100 * time.Millisecond, KillTimeout: 100 * time.Millisecond, ExpectedStartTime: c.vm.ProcessStartTime, @@ -337,7 +344,6 @@ func (c *Controller) Reboot(ctx context.Context, force bool) error { // it stays paused after the snapshot. If pauseOnly is false, the VM is resumed. func (c *Controller) SnapshotCreate(ctx context.Context, cfg model.SnapshotCreateConfig) (err error) { name := c.vm.Name - // Validate state — snapshot requires RUNNING or PAUSED if c.vm.Status == model.VMStatusStarting { return errs.New(errs.CodeVMStateInvalid, @@ -360,6 +366,13 @@ func (c *Controller) SnapshotCreate(ctx context.Context, cfg model.SnapshotCreat return errs.New(errs.CodeVMStateInvalid, fmt.Sprintf("Socket not found for VM '%s'. Must be running with --enable-api-socket", name)) } + if cfg.SnapshotDir != "" { + if err := ExposeSnapshot(ctx, c.vm.ID, cfg.SnapshotDir); err != nil { + slog.Error("failed to expose snapshot directory inside VM jail", "vm", name, "error", err) + return errs.WrapMsg(errs.CodeSnapshotCreateFailed, + "failed to expose snapshot directory inside VM jail", err) + } + } apiSocket := c.vm.APISocketPath client := firecracker.NewClient(apiSocket) @@ -478,11 +491,23 @@ func (c *Controller) AttachVolume(ctx context.Context, driveConfig model.DriveCo apiSocket := c.vm.APISocketPath configPath := c.vm.ConfigPath + if c.vm.JailerBinaryID != "" { + if err := exposeVolume(ctx, c.vm.ID, driveConfig); err != nil { + slog.Error("failed to expose volume inside VM jail", "vm", c.vm.Name, "error", err) + return err + } + driveConfig.PathOnHost = jailedVolumePath(driveConfig.DriveID) + } // Hotplug into the running Firecracker process client := firecracker.NewClient(apiSocket) err := client.PutDrive(ctx, driveConfig) client.Close() if err != nil { + if c.vm.JailerBinaryID != "" { + if cleanupErr := removeVolume(context.Background(), c.vm.ID, driveConfig.DriveID); cleanupErr != nil { + slog.Warn("failed to remove volume mount after hotplug failure", "vm", c.vm.Name, "error", cleanupErr) + } + } return err } @@ -516,6 +541,12 @@ func (c *Controller) DetachVolume(ctx context.Context, driveID string) error { if _, err := configMgr.RemoveDrive(driveID); err != nil { return err } + if c.vm.JailerBinaryID != "" { + if err := removeVolume(ctx, c.vm.ID, driveID); err != nil { + slog.Error("failed to remove volume from VM jail", "vm", c.vm.Name, "drive_id", driveID, "error", err) + return err + } + } slog.Info("Detached volume", "drive_id", driveID, "vm", c.vm.Name) return nil diff --git a/internal/core/vm/firecracker.go b/internal/core/vm/firecracker.go index 2f1f8086..dfecb01d 100644 --- a/internal/core/vm/firecracker.go +++ b/internal/core/vm/firecracker.go @@ -1,13 +1,15 @@ package vm import ( + "context" "encoding/json" + "errors" "fmt" "log/slog" "net" "os" - "os/exec" "path/filepath" + "strconv" "strings" "syscall" "time" @@ -15,6 +17,7 @@ import ( "mvmctl/internal/infra" "mvmctl/internal/lib/model" "mvmctl/internal/lib/system" + jailersvc "mvmctl/internal/service/jailer" "mvmctl/pkg/errs" ) @@ -59,28 +62,36 @@ func NewFirecrackerSpawner(config *model.FirecrackerConfig) *FirecrackerSpawner // Spawn starts a Firecracker process. // -// Polls for the API socket to become available (up to 2s, every 0.1s) and +// Polls for the API socket to become available for 150 bounded 100ms intervals and // exits early as soon as the socket appears. If the process dies before the // socket is created, raises immediately. -func (s *FirecrackerSpawner) Spawn() (retErr error) { +func (s *FirecrackerSpawner) Spawn(ctx context.Context) (retErr error) { // Cleanup any opened FDs on failure. // On success, FDs are either closed explicitly (CloseFilePointers) // or transferred to the child process via Stdin/Stdout. var relayFile *os.File - var cmd *exec.Cmd + var launchProcess *os.Process started := false defer func() { if retErr != nil { + safeToCleanup := true if started { - // Process was spawned but something failed — kill it. - cmd.Process.Kill() - cmd.Process.Wait() + var terminateErr error + safeToCleanup, terminateErr = s.terminateFailedSpawn(launchProcess) + if terminateErr != nil { + retErr = errors.Join(retErr, terminateErr) + } } if relayFile != nil { relayFile.Close() } - s.Cleanup() + if safeToCleanup { + s.Cleanup(context.Background()) + } else { + s.CloseFilePointers() + slog.Error("preserving VM jail because failed process termination was not verified", "vm_id", s.vmID()) + } } }() @@ -90,6 +101,9 @@ func (s *FirecrackerSpawner) Spawn() (retErr error) { return fmt.Errorf("remove stale api socket %s: %w", s.APISocketPath, err) } } + if err := os.Remove(s.pidPath); err != nil && !os.IsNotExist(err) { + return fmt.Errorf("remove stale pid file %s: %w", s.pidPath, err) + } var fcStdin *os.File var fcStdout *os.File @@ -117,28 +131,11 @@ func (s *FirecrackerSpawner) Spawn() (retErr error) { return err } - fcCmd := []string{ - s.config.BinaryPath, - "--api-sock", - s.APISocketPath, - } - if s.config.PCIEnabled { - fcCmd = append(fcCmd, "--enable-pci") - } - if !s.config.SnapshotMode { - fcCmd = append(fcCmd, "--config-file", s.configPath) - } - - cmd = exec.Command(fcCmd[0], fcCmd[1:]...) - cmd.Stdin = fcStdin - cmd.Stdout = fcStdout - cmd.Stderr = s.fcLogFP - cmd.SysProcAttr = &syscall.SysProcAttr{ - Setsid: true, + if err := s.writeJailerManifest(); err != nil { + return err } - cmd.Env = append(os.Environ(), infra.MVMBackgroundServiceEnv) - - if err := cmd.Start(); err != nil { + launchProcess, err = jailersvc.Launch(ctx, s.vmID(), s.config.VMDir, fcStdin, fcStdout, s.fcLogFP) + if err != nil { return err } started = true @@ -149,12 +146,12 @@ func (s *FirecrackerSpawner) Spawn() (retErr error) { relayFile = nil } - // Wait for Firecracker to initialize (poll up to 4s, exit early on socket). + // Wait for Firecracker to initialize (150 bounded 100ms polls, approximately 15s total). // Interleave liveness checks so a crashed process is caught early. // CRITICAL: WaitForSocket only checks that the socket file exists (bind() completed). // Firecracker may crash between bind() and listen(), leaving a zombie socket file // that reports ECONNREFUSED on connect. We must Dial to verify it's actually listening. - // Nested KVM environments may take longer to initialize (4s might not be enough). + // Nested KVM environments can consume most of this bounded readiness window. for range 150 { if infra.WaitForSocket(s.APISocketPath, 100*time.Millisecond) == nil { // Socket file exists — verify it's accepting connections. @@ -166,32 +163,39 @@ func (s *FirecrackerSpawner) Spawn() (retErr error) { // Fall through to liveness check. } - if err := cmd.Process.Signal(syscall.Signal(0)); err != nil { - ps, waitErr := cmd.Process.Wait() - exitCode := -1 - if waitErr == nil && ps != nil { - exitCode = ps.ExitCode() - } + if err := launchProcess.Signal(syscall.Signal(0)); err != nil { return errs.New(errs.CodeFirecrackerSpawnError, - fmt.Sprintf("firecracker process exited immediately with code %d", exitCode)) + "jailer process exited before the Firecracker API became ready") } } // Final verification: socket must exist AND accept connections. if _, err := os.Stat(s.APISocketPath); os.IsNotExist(err) { return errs.New(errs.CodeFirecrackerSpawnError, - "firecracker API socket not available after 4s") + "firecracker API socket not available after the bounded readiness wait") } if conn, dialErr := net.DialTimeout("unix", s.APISocketPath, 200*time.Millisecond); dialErr != nil { - return errs.New(errs.CodeFirecrackerSpawnError, - fmt.Sprintf("firecracker API socket not accepting connections after 4s: %v", dialErr)) + return errs.New( + errs.CodeFirecrackerSpawnError, + fmt.Sprintf( + "firecracker API socket not accepting connections after the bounded readiness wait: %v", + dialErr, + ), + ) } else { conn.Close() } s.CloseFilePointers() - pid := cmd.Process.Pid + pidData, err := os.ReadFile(s.pidPath) + if err != nil { + return errs.WrapMsg(errs.CodeFirecrackerSpawnError, "jailed Firecracker PID was not recorded", err) + } + pid, err := strconv.Atoi(strings.TrimSpace(string(pidData))) + if err != nil || pid <= 0 { + return errs.New(errs.CodeFirecrackerSpawnError, "jailed Firecracker PID is invalid") + } s.PID = &pid s.ProcessStartTime = system.GetProcessStartTime(pid) @@ -202,11 +206,109 @@ func (s *FirecrackerSpawner) Spawn() (retErr error) { return nil } +func (s *FirecrackerSpawner) terminateFailedSpawn(launcher *os.Process) (bool, error) { + actualPID, startTime, pidKnown, pidErr := s.failedSpawnProcessInfo() + var terminationErr error + if pidErr != nil { + terminationErr = pidErr + } + if pidKnown { + if err := terminateProcessWithStartTime(actualPID, startTime); err != nil { + terminationErr = errors.Join(terminationErr, err) + } + } + + if killErr := launcher.Kill(); killErr != nil && !errors.Is(killErr, os.ErrProcessDone) { + terminationErr = errors.Join(terminationErr, fmt.Errorf("kill outer Jailer launcher: %w", killErr)) + } + if err := waitForLauncher(launcher, 2*time.Second); err != nil { + terminationErr = errors.Join(terminationErr, err) + } + + if pidKnown && system.IsProcessAlive(actualPID, startTime) { + terminationErr = errors.Join(terminationErr, + fmt.Errorf("jailed process %d remained alive after failed-spawn termination", actualPID)) + return false, terminationErr + } + if pidErr != nil { + return false, terminationErr + } + return true, terminationErr +} + +func (s *FirecrackerSpawner) failedSpawnProcessInfo() (int, *int64, bool, error) { + pidData, err := os.ReadFile(s.pidPath) + if os.IsNotExist(err) { + return 0, nil, false, nil + } + if err != nil { + return 0, nil, false, fmt.Errorf("read jailed process PID during failed spawn: %w", err) + } + pid, err := strconv.Atoi(strings.TrimSpace(string(pidData))) + if err != nil || pid <= 0 { + return 0, nil, false, fmt.Errorf("jailed process PID is invalid during failed spawn") + } + return pid, system.GetProcessStartTime(pid), true, nil +} + +func terminateProcessWithStartTime(pid int, startTime *int64) error { + if !system.IsProcessAlive(pid, startTime) { + return nil + } + if err := syscall.Kill(pid, syscall.SIGTERM); err != nil && !errors.Is(err, syscall.ESRCH) { + return fmt.Errorf("terminate jailed process %d: %w", pid, err) + } + if waitForProcessExit(pid, startTime, 2*time.Second) { + return nil + } + if !system.IsProcessAlive(pid, startTime) { + return nil + } + if err := syscall.Kill(pid, syscall.SIGKILL); err != nil && !errors.Is(err, syscall.ESRCH) { + return fmt.Errorf("kill jailed process %d: %w", pid, err) + } + if !waitForProcessExit(pid, startTime, time.Second) { + return fmt.Errorf("jailed process %d survived SIGKILL", pid) + } + return nil +} + +func waitForProcessExit(pid int, startTime *int64, timeout time.Duration) bool { + deadline := time.Now().Add(timeout) + for time.Now().Before(deadline) { + if !system.IsProcessAlive(pid, startTime) { + return true + } + time.Sleep(25 * time.Millisecond) + } + return !system.IsProcessAlive(pid, startTime) +} + +func waitForLauncher(launcher *os.Process, timeout time.Duration) error { + done := make(chan error, 1) + go func() { + _, err := launcher.Wait() + done <- err + }() + select { + case err := <-done: + if err != nil { + return fmt.Errorf("reap outer Jailer launcher: %w", err) + } + return nil + case <-time.After(timeout): + return fmt.Errorf("timed out reaping outer Jailer launcher") + } +} + // --- Cleanup --- // Cleanup performs cleanup of all created resources. -func (s *FirecrackerSpawner) Cleanup() { +func (s *FirecrackerSpawner) Cleanup(ctx context.Context) { s.CloseFilePointers() + if err := CleanupJail(ctx, s.vmID()); err != nil { + slog.Warn("failed to clean VM jail", "vm_id", s.vmID(), "error", err) + } } // --- Generate --- @@ -270,7 +372,7 @@ func (s *FirecrackerSpawner) WriteToFile() error { if err := os.MkdirAll(dir, infra.DirPerm); err != nil { return err } - data, err := json.Marshal(config) + data, err := json.Marshal(translateConfigForJail(config)) if err != nil { return err } @@ -303,7 +405,6 @@ func (s *FirecrackerSpawner) buildDrivesConfig() []model.DriveConfig { // Resolve rootfs path to absolute path rootfsAbs, err := filepath.Abs(s.config.RootfsPath) if err != nil { - // Fallback to original if Abs fails (should not happen in practice) rootfsAbs = s.config.RootfsPath } cacheType := model.CacheTypeUnsafe diff --git a/internal/core/vm/jailer.go b/internal/core/vm/jailer.go new file mode 100644 index 00000000..94090abf --- /dev/null +++ b/internal/core/vm/jailer.go @@ -0,0 +1,107 @@ +package vm + +import ( + "context" + "encoding/json" + "fmt" + "os" + "path/filepath" + + "mvmctl/internal/infra" + "mvmctl/internal/lib/model" + jailersvc "mvmctl/internal/service/jailer" +) + +func (s *FirecrackerSpawner) writeJailerManifest() error { + volumes := make([]jailersvc.VolumeMount, 0, len(s.config.ExtraDrives)) + for _, drive := range s.config.ExtraDrives { + volumes = append(volumes, jailersvc.VolumeMount{ + DriveID: drive.DriveID, HostPath: drive.PathOnHost, ReadOnly: drive.IsReadOnly, + }) + } + isoPath := "" + if s.config.CloudInitISOPath != nil { + isoPath = *s.config.CloudInitISOPath + } + manifest := jailersvc.LaunchManifest{ + VMID: s.vmID(), VMDir: s.config.VMDir, Version: s.config.BinaryVersion, + ConfigPath: s.config.ConfigPath, PIDPath: s.config.PIDPath, + KernelPath: s.config.KernelPath, RootfsPath: s.config.RootfsPath, + ISOPath: isoPath, SnapshotDir: s.config.SnapshotDir, + Volumes: volumes, APISocket: s.config.APISocketPath, + PCIEnabled: s.config.PCIEnabled, SnapshotMode: s.config.SnapshotMode, + } + data, err := json.Marshal(manifest) + if err != nil { + return fmt.Errorf("marshal jailed launch manifest: %w", err) + } + path := filepath.Join(s.config.VMDir, infra.JailerManifestFilename) + if err := os.WriteFile(path, data, 0600); err != nil { + return fmt.Errorf("write jailed launch manifest: %w", err) + } + return nil +} + +func (s *FirecrackerSpawner) vmID() string { + return filepath.Base(filepath.Clean(s.config.VMDir)) +} + +func jailedRuntimePath(hostPath string) string { + return filepath.Join("/run/mvm", filepath.Base(hostPath)) +} + +func jailedVolumePath(driveID string) string { + return filepath.Join("/volumes", driveID) +} + +func translateConfigForJail(config *model.FirecrackerVMConfig) *model.FirecrackerVMConfig { + translated := *config + translated.BootSource = config.BootSource + translated.BootSource.KernelImagePath = "/kernel" + translated.Drives = make([]model.DriveConfig, len(config.Drives)) + for i, drive := range config.Drives { + translated.Drives[i] = drive + switch drive.DriveID { + case "rootfs": + translated.Drives[i].PathOnHost = "/rootfs" + case "cloud-init": + translated.Drives[i].PathOnHost = "/cloud-init.iso" + default: + translated.Drives[i].PathOnHost = jailedVolumePath(drive.DriveID) + } + } + if config.Logger != nil { + logger := *config.Logger + logger.LogPath = jailedRuntimePath(logger.LogPath) + translated.Logger = &logger + } + if config.Metrics != nil { + metrics := *config.Metrics + metrics.MetricsPath = jailedRuntimePath(metrics.MetricsPath) + translated.Metrics = &metrics + } + if config.Vsock != nil { + vsock := *config.Vsock + vsock.UDSPath = jailedRuntimePath(vsock.UDSPath) + translated.Vsock = &vsock + } + return &translated +} + +// ExposeSnapshot exposes one managed snapshot directory inside an existing VM jail. +func ExposeSnapshot(ctx context.Context, vmID, snapshotDir string) error { + return jailersvc.ExposeSnapshot(ctx, vmID, snapshotDir) +} + +// CleanupJail removes all mounts and chroot state for one VM. +func CleanupJail(ctx context.Context, vmID string) error { + return jailersvc.Cleanup(ctx, vmID) +} + +func exposeVolume(ctx context.Context, vmID string, drive model.DriveConfig) error { + return jailersvc.ExposeVolume(ctx, vmID, drive.DriveID, drive.PathOnHost, drive.IsReadOnly) +} + +func removeVolume(ctx context.Context, vmID, driveID string) error { + return jailersvc.RemoveVolume(ctx, vmID, driveID) +} diff --git a/internal/core/vm/jailer_test.go b/internal/core/vm/jailer_test.go new file mode 100644 index 00000000..81fbbefe --- /dev/null +++ b/internal/core/vm/jailer_test.go @@ -0,0 +1,146 @@ +package vm + +import ( + "encoding/json" + "os" + "path/filepath" + "testing" + + "github.com/google/go-cmp/cmp" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "mvmctl/internal/infra" + "mvmctl/internal/infra/ptr" + "mvmctl/internal/lib/model" + "mvmctl/internal/service/jailer" +) + +const testDriveID = "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef" + +func TestFirecrackerSpawner_WriteJailerManifestSelectsOnlyVMResources(t *testing.T) { + vmDir := t.TempDir() + config := jailerTestConfig(vmDir) + spawner := NewFirecrackerSpawner(config) + + require.NoError(t, spawner.writeJailerManifest()) + data, err := os.ReadFile(filepath.Join(vmDir, infra.JailerManifestFilename)) + require.NoError(t, err) + var got jailer.LaunchManifest + require.NoError(t, json.Unmarshal(data, &got)) + + want := jailer.LaunchManifest{ + VMID: filepath.Base(vmDir), VMDir: vmDir, Version: "1.16.0", + ConfigPath: config.ConfigPath, PIDPath: config.PIDPath, + KernelPath: config.KernelPath, RootfsPath: config.RootfsPath, + ISOPath: *config.CloudInitISOPath, SnapshotDir: config.SnapshotDir, + Volumes: []jailer.VolumeMount{{ + DriveID: testDriveID, HostPath: config.ExtraDrives[0].PathOnHost, ReadOnly: true, + }}, + APISocket: config.APISocketPath, PCIEnabled: true, SnapshotMode: true, + } + if diff := cmp.Diff(want, got); diff != "" { + t.Errorf("LaunchManifest mismatch (-want +got):\n%s", diff) + } + var raw map[string]json.RawMessage + require.NoError(t, json.Unmarshal(data, &raw)) + wantKeys := map[string]bool{ + "vm_id": true, "vm_dir": true, "version": true, "config_path": true, + "pid_path": true, "kernel_path": true, "rootfs_path": true, "iso_path": true, + "snapshot_dir": true, "volumes": true, "api_socket": true, + "pci_enabled": true, "snapshot_mode": true, + } + gotKeys := make(map[string]bool, len(raw)) + for key := range raw { + gotKeys[key] = true + } + if diff := cmp.Diff(wantKeys, gotKeys); diff != "" { + t.Errorf("manifest field set mismatch (-want +got):\n%s", diff) + } +} + +func TestFirecrackerSpawner_WriteToFileUsesJailedPathsWithoutMutatingHostConfig(t *testing.T) { + vmDir := t.TempDir() + config := jailerTestConfig(vmDir) + config.SnapshotMode = false + spawner := NewFirecrackerSpawner(config) + + hostConfig, err := spawner.Generate() + require.NoError(t, err) + require.NoError(t, spawner.WriteToFile()) + data, err := os.ReadFile(config.ConfigPath) + require.NoError(t, err) + var jailed model.FirecrackerVMConfig + require.NoError(t, json.Unmarshal(data, &jailed)) + + assert.Equal(t, config.KernelPath, hostConfig.BootSource.KernelImagePath) + assert.Equal(t, "/kernel", jailed.BootSource.KernelImagePath) + require.Len(t, hostConfig.Drives, 3) + require.Len(t, jailed.Drives, 3) + wantHostPaths := []string{config.RootfsPath, *config.CloudInitISOPath, config.ExtraDrives[0].PathOnHost} + wantJailedPaths := []string{"/rootfs", "/cloud-init.iso", "/volumes/" + testDriveID} + for i := range hostConfig.Drives { + assert.Equal(t, wantHostPaths[i], hostConfig.Drives[i].PathOnHost) + assert.Equal(t, wantJailedPaths[i], jailed.Drives[i].PathOnHost) + } + require.NotNil(t, hostConfig.Logger) + require.NotNil(t, jailed.Logger) + assert.Equal(t, config.LogPath, hostConfig.Logger.LogPath) + assert.Equal(t, "/run/mvm/"+filepath.Base(config.LogPath), jailed.Logger.LogPath) + require.NotNil(t, hostConfig.Metrics) + require.NotNil(t, jailed.Metrics) + assert.Equal(t, config.MetricsPath, hostConfig.Metrics.MetricsPath) + assert.Equal(t, "/run/mvm/"+filepath.Base(config.MetricsPath), jailed.Metrics.MetricsPath) + require.NotNil(t, hostConfig.Vsock) + require.NotNil(t, jailed.Vsock) + assert.Equal(t, config.Vsock.UDSPath, hostConfig.Vsock.UDSPath) + assert.Equal(t, "/run/mvm/"+filepath.Base(config.Vsock.UDSPath), jailed.Vsock.UDSPath) +} + +// Rationale: Mount destinations form a privileged filesystem boundary and +// must remain absolute, normalized, and independent of attacker path segments. +func TestJailedPathCalculation(t *testing.T) { + tests := map[string]struct { + got string + want string + }{ + "runtime_uses_basename": {got: jailedRuntimePath("/host/vms/id/api.socket"), want: "/run/mvm/api.socket"}, + "runtime_cleans_parent_segments": { + got: jailedRuntimePath("/host/vms/id/../metrics.fifo"), want: "/run/mvm/metrics.fifo", + }, + "volume_uses_validated_id": {got: jailedVolumePath(testDriveID), want: "/volumes/" + testDriveID}, + } + for name, tc := range tests { + t.Run(name, func(t *testing.T) { + assert.Equal(t, tc.want, tc.got) + assert.Equal(t, tc.got, filepath.Clean(tc.got)) + }) + } +} + +func jailerTestConfig(vmDir string) *model.FirecrackerConfig { + cloudInitMode := model.CloudInitModeISO + isoPath := filepath.Join(vmDir, "cloud-init.iso") + return &model.FirecrackerConfig{ + VMDir: vmDir, RootfsPath: filepath.Join(vmDir, "rootfs.ext4"), + BinaryPath: "/var/lib/mvmctl/binaries/1.16.0/firecracker", + JailerPath: "/var/lib/mvmctl/binaries/1.16.0/jailer", BinaryVersion: "1.16.0", + KernelPath: filepath.Join(filepath.Dir(vmDir), "kernels", "vmlinux"), + VCPUCount: 2, MemSizeMiB: 512, GuestIP: "10.0.0.2", GuestMAC: "02:00:00:00:00:02", + TapName: "tap-test", NetworkGateway: "10.0.0.1", NetworkNetmask: "255.255.255.0", + PCIEnabled: true, ImageFSUUID: "11111111-2222-3333-4444-555555555555", ImageFSType: "ext4", + EnableLogging: true, EnableMetrics: true, LogLevel: "Info", + LogPath: filepath.Join(vmDir, "firecracker.log"), MetricsPath: filepath.Join(vmDir, "metrics.fifo"), + SerialOutputPath: filepath.Join(vmDir, "serial.log"), + APISocketPath: filepath.Join(vmDir, "api.socket"), PIDPath: filepath.Join(vmDir, "firecracker.pid"), + ConfigPath: filepath.Join(vmDir, "config.json"), CloudInitMode: &cloudInitMode, + CloudInitISOPath: &isoPath, + ExtraDrives: []model.DriveConfig{{ + DriveID: testDriveID, PathOnHost: filepath.Join(filepath.Dir(vmDir), "volumes", "selected.raw"), + IsReadOnly: true, CacheType: model.CacheTypeUnsafe, IOEngine: "Sync", + }}, + SnapshotMode: true, SnapshotDir: filepath.Join(filepath.Dir(vmDir), "snapshots", testDriveID), + Vsock: &model.VsockConfig{GuestCID: 42, UDSPath: filepath.Join(vmDir, "vsock.socket")}, + CPUVendor: ptr.Ptr("Intel"), + } +} diff --git a/internal/core/vm/sqlite.go b/internal/core/vm/sqlite.go index ab164df4..24feadae 100644 --- a/internal/core/vm/sqlite.go +++ b/internal/core/vm/sqlite.go @@ -161,7 +161,14 @@ func (r *sqliteRepo) GetByKernelIDs(ctx context.Context, kernelIDs []string) ([] func (r *sqliteRepo) FindByBinaryID(ctx context.Context, binaryID string) ([]*model.VMItem, error) { var rows []*model.VMItem - if err := sqlx.SelectContext(ctx, r.db, &rows, vmBaseQuery+" WHERE binary_id = ?", binaryID); err != nil { + if err := sqlx.SelectContext( + ctx, + r.db, + &rows, + vmBaseQuery+" WHERE binary_id = ? OR jailer_binary_id = ?", + binaryID, + binaryID, + ); err != nil { return nil, fmt.Errorf("find vms by binary id: %w", err) } return rows, nil @@ -171,7 +178,11 @@ func (r *sqliteRepo) GetByBinaryIDs(ctx context.Context, binaryIDs []string) ([] if len(binaryIDs) == 0 { return nil, nil } - query, args, err := sqlx.In(vmBaseQuery+" WHERE binary_id IN (?)", binaryIDs) + query, args, err := sqlx.In( + vmBaseQuery+" WHERE binary_id IN (?) OR jailer_binary_id IN (?)", + binaryIDs, + binaryIDs, + ) if err != nil { return nil, fmt.Errorf("get vms by binary ids: %w", err) } @@ -283,7 +294,7 @@ func (r *sqliteRepo) Upsert(ctx context.Context, vm *model.VMItem) error { _, err = r.db.ExecContext(ctx, ` INSERT INTO vm_instances ( id, name, status, pid, process_start_time, ipv4, mac, network_id, tap_device, - image_id, kernel_id, binary_id, api_socket_path, + image_id, kernel_id, binary_id, jailer_binary_id, api_socket_path, relay_socket_path, config_path, cloud_init_mode, nocloud_net_port, nocloud_net_pid, relay_pid, exit_code, vcpu_count, mem_size_mib, disk_size_mib, @@ -293,7 +304,7 @@ func (r *sqliteRepo) Upsert(ctx context.Context, vm *model.VMItem) error { ssh_keys, ssh_user, created_at, updated_at, log_path, serial_output_path, lsm_flags, boot_args, volume_ids, cpu_config - ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) ON CONFLICT(id) DO UPDATE SET name = excluded.name, status = excluded.status, @@ -306,6 +317,7 @@ func (r *sqliteRepo) Upsert(ctx context.Context, vm *model.VMItem) error { image_id = excluded.image_id, kernel_id = excluded.kernel_id, binary_id = excluded.binary_id, + jailer_binary_id = excluded.jailer_binary_id, api_socket_path = excluded.api_socket_path, relay_socket_path = excluded.relay_socket_path, config_path = excluded.config_path, @@ -347,6 +359,7 @@ func (r *sqliteRepo) Upsert(ctx context.Context, vm *model.VMItem) error { vm.ImageID, vm.KernelID, vm.BinaryID, + vm.JailerBinaryID, vm.APISocketPath, vm.RelaySocketPath, vm.ConfigPath, diff --git a/internal/core/vm/sqlite_test.go b/internal/core/vm/sqlite_test.go new file mode 100644 index 00000000..3e460288 --- /dev/null +++ b/internal/core/vm/sqlite_test.go @@ -0,0 +1,45 @@ +package vm_test + +import ( + "context" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "mvmctl/internal/core/vm" + "mvmctl/internal/lib/model" + "mvmctl/internal/testutil" +) + +func TestRepository_PersistsJailerBinaryID(t *testing.T) { + ctx := context.Background() + database := testutil.NewInMemoryDB(t) + _, err := database.ExecContext(ctx, "PRAGMA foreign_keys = OFF") + require.NoError(t, err) + repo := vm.NewRepository(database) + item := &model.VMItem{ + ID: "0123456789abcdef0123456789abcdef", Name: "persisted-jailer", Status: model.VMStatusStopped, + IPv4: "10.0.0.2", MAC: "02:00:00:00:00:02", NetworkID: "network-id", TapDevice: "tap-test", + ImageID: "image-id", KernelID: "kernel-id", BinaryID: "firecracker-id", JailerBinaryID: "jailer-id", + APISocketPath: "/tmp/api.socket", ConfigPath: "/tmp/config.json", CloudInitMode: "off", + VCPUCount: 2, MemSizeMiB: 512, DiskSizeMiB: 1024, RootfsPath: "/tmp/rootfs", RootfsSuffix: ".ext4", + CreatedAt: "2026-08-09T00:00:00Z", UpdatedAt: "2026-08-09T00:00:00Z", + } + + require.NoError(t, repo.Upsert(ctx, item)) + got, err := repo.Get(ctx, item.ID) + require.NoError(t, err) + require.NotNil(t, got) + assert.Equal(t, "firecracker-id", got.BinaryID) + assert.Equal(t, "jailer-id", got.JailerBinaryID) + + byJailer, err := repo.FindByBinaryID(ctx, "jailer-id") + require.NoError(t, err) + require.Len(t, byJailer, 1) + assert.Equal(t, item.ID, byJailer[0].ID) + byBoth, err := repo.GetByBinaryIDs(ctx, []string{"firecracker-id", "jailer-id"}) + require.NoError(t, err) + require.Len(t, byBoth, 1) + assert.Equal(t, item.ID, byBoth[0].ID) +} diff --git a/internal/infra/constants.go b/internal/infra/constants.go index d747c629..1d0cf1d2 100644 --- a/internal/infra/constants.go +++ b/internal/infra/constants.go @@ -46,6 +46,15 @@ const MVMUnixGroup = CLIName const MVMDBFilename = "mvmdb.db" +// TrustedBinaryRoot contains root-owned Firecracker release pairs used by Jailer. +const TrustedBinaryRoot = "/var/lib/mvmctl/binaries" + +// JailerChrootBase is the fixed root-owned base directory for per-VM jails. +const JailerChrootBase = "/var/lib/mvmctl/jailer" + +// JailerManifestFilename is the fixed per-VM launch manifest name. +const JailerManifestFilename = "jailer-launch.json" + const MVMForwardChain = "MVM-FORWARD" const MVMPostroutingChain = "MVM-POSTROUTING" diff --git a/internal/lib/db/migrations/002_jailer_launch.sql b/internal/lib/db/migrations/002_jailer_launch.sql new file mode 100644 index 00000000..ce3cfcd3 --- /dev/null +++ b/internal/lib/db/migrations/002_jailer_launch.sql @@ -0,0 +1,22 @@ +-- Migration: 002_jailer_launch +-- Version: 2 +-- Description: Persist the exact Jailer paired with each VM and snapshot. + +ALTER TABLE vm_instances ADD COLUMN jailer_binary_id TEXT NOT NULL DEFAULT ''; + +UPDATE vm_instances +SET jailer_binary_id = COALESCE(( + SELECT jailer.id + FROM binaries AS firecracker + JOIN binaries AS jailer + ON jailer.type = 'jailer' + AND jailer.version = firecracker.version + AND jailer.deleted_at IS NULL + WHERE firecracker.id = vm_instances.binary_id + AND firecracker.type = 'firecracker' + LIMIT 1 +), ''); + +CREATE INDEX idx_vm_instances_jailer_binary ON vm_instances(jailer_binary_id); + +PRAGMA user_version = 2; diff --git a/internal/lib/db/migrations_jailer_test.go b/internal/lib/db/migrations_jailer_test.go new file mode 100644 index 00000000..8d82f062 --- /dev/null +++ b/internal/lib/db/migrations_jailer_test.go @@ -0,0 +1,75 @@ +package db_test + +import ( + "context" + "path/filepath" + "testing" + + "github.com/google/go-cmp/cmp" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "mvmctl/internal/lib/db" +) + +// Rationale: Existing databases must bind each VM to the exact live Jailer +// record matching its persisted Firecracker version, while unmatched pairs fail closed. +func TestMigration002_BackfillsJailerBinaryID(t *testing.T) { + ctx := context.Background() + handle := db.New(filepath.Join(t.TempDir(), "migration-002.db")) + t.Cleanup(func() { require.NoError(t, handle.Close()) }) + database := handle.DB() + _, err := database.ExecContext(ctx, ` + CREATE TABLE binaries ( + id TEXT PRIMARY KEY, + type TEXT NOT NULL, + version TEXT NOT NULL, + deleted_at TEXT NULL + ); + CREATE TABLE vm_instances ( + id TEXT PRIMARY KEY, + binary_id TEXT NOT NULL + ); + INSERT INTO binaries (id, type, version, deleted_at) VALUES + ('fc-match', 'firecracker', '1.16.0', NULL), + ('jl-match', 'jailer', '1.16.0', NULL), + ('fc-missing', 'firecracker', '1.15.0', NULL), + ('fc-deleted-pair', 'firecracker', '1.14.0', NULL), + ('jl-deleted', 'jailer', '1.14.0', '2026-08-09T00:00:00Z'); + INSERT INTO vm_instances (id, binary_id) VALUES + ('vm-match', 'fc-match'), + ('vm-missing', 'fc-missing'), + ('vm-deleted-pair', 'fc-deleted-pair'); + PRAGMA user_version = 1; + `) + require.NoError(t, err) + + applied, err := handle.RunMigrationsCtx(ctx) + require.NoError(t, err) + assert.Equal(t, 1, applied) + + rows := map[string]string{} + dbRows, err := database.QueryContext(ctx, "SELECT id, jailer_binary_id FROM vm_instances ORDER BY id") + require.NoError(t, err) + defer dbRows.Close() + for dbRows.Next() { + var id, jailerID string + require.NoError(t, dbRows.Scan(&id, &jailerID)) + rows[id] = jailerID + } + require.NoError(t, dbRows.Err()) + want := map[string]string{ + "vm-match": "jl-match", "vm-missing": "", "vm-deleted-pair": "", + } + if diff := cmp.Diff(want, rows); diff != "" { + t.Errorf("migration backfill mismatch (-want +got):\n%s", diff) + } + + var schemaVersion int + require.NoError(t, database.GetContext(ctx, &schemaVersion, "PRAGMA user_version")) + assert.Equal(t, 2, schemaVersion) + var indexCount int + require.NoError(t, database.GetContext(ctx, &indexCount, + "SELECT COUNT(*) FROM sqlite_master WHERE type = 'index' AND name = 'idx_vm_instances_jailer_binary'")) + assert.Equal(t, 1, indexCount) +} diff --git a/internal/lib/model/binary.go b/internal/lib/model/binary.go index ab164a67..349f7ba4 100644 --- a/internal/lib/model/binary.go +++ b/internal/lib/model/binary.go @@ -20,3 +20,9 @@ type BinaryItem struct { VMs []*VMItem `json:"vms,omitempty"` Snapshots []*SnapshotItem `json:"snapshots,omitempty"` } + +// BinaryPair is an exact-version Firecracker and Jailer release pair. +type BinaryPair struct { + Firecracker *BinaryItem `json:"firecracker" db:"-"` + Jailer *BinaryItem `json:"jailer" db:"-"` +} diff --git a/internal/lib/model/firecracker.go b/internal/lib/model/firecracker.go index 94ebf6a7..a98e7dea 100644 --- a/internal/lib/model/firecracker.go +++ b/internal/lib/model/firecracker.go @@ -232,8 +232,10 @@ type FirecrackerConfig struct { RootfsPath string `json:"rootfs_path"` // Binary / kernel - BinaryPath string `json:"binary_path"` - KernelPath string `json:"kernel_path"` + BinaryPath string `json:"binary_path"` + JailerPath string `json:"jailer_path"` + BinaryVersion string `json:"binary_version"` + KernelPath string `json:"kernel_path"` // Machine VCPUCount int `json:"vcpu_count"` @@ -284,8 +286,9 @@ type FirecrackerConfig struct { ExtraDrives []DriveConfig `json:"extra_drives,omitempty"` // Spawn behavior - RelayClientFD *int `json:"relay_client_fd,omitempty"` - SnapshotMode bool `json:"snapshot_mode"` + RelayClientFD *int `json:"relay_client_fd,omitempty"` + SnapshotMode bool `json:"snapshot_mode"` + SnapshotDir string `json:"snapshot_dir,omitempty"` // Cache type // When true, rootfs + volumes use "Writeback" cache (safe, guest fsync honored). diff --git a/internal/lib/model/snapshot.go b/internal/lib/model/snapshot.go index 0327f774..b55da292 100644 --- a/internal/lib/model/snapshot.go +++ b/internal/lib/model/snapshot.go @@ -35,6 +35,7 @@ type SnapshotItem struct { // SnapshotCreateConfig collects all external parameters for creating a snapshot. // Passed from the API layer to the controller. type SnapshotCreateConfig struct { + SnapshotDir string MemFile string StateFile string PauseOnly bool diff --git a/internal/lib/model/vm.go b/internal/lib/model/vm.go index d77d79cc..aad6db31 100644 --- a/internal/lib/model/vm.go +++ b/internal/lib/model/vm.go @@ -23,33 +23,34 @@ const ( // VMItem represents a microVM instance. type VMItem struct { - ID string `json:"id" db:"id"` - Name string `json:"name" db:"name"` - Status VMStatus `json:"status" db:"status"` - PID int `json:"pid" db:"pid"` - IPv4 string `json:"ipv4" db:"ipv4"` - MAC string `json:"mac" db:"mac"` - NetworkID string `json:"network_id" db:"network_id"` - TapDevice string `json:"tap_device" db:"tap_device"` - ImageID string `json:"image_id" db:"image_id"` - KernelID string `json:"kernel_id" db:"kernel_id"` - BinaryID string `json:"binary_id" db:"binary_id"` - APISocketPath string `json:"api_socket_path" db:"api_socket_path"` - ConfigPath string `json:"config_path" db:"config_path"` - CloudInitMode string `json:"cloud_init_mode" db:"cloud_init_mode"` - VCPUCount int `json:"vcpu_count" db:"vcpu_count"` - MemSizeMiB int `json:"mem_size_mib" db:"mem_size_mib"` - DiskSizeMiB int `json:"disk_size_mib" db:"disk_size_mib"` - RootfsPath string `json:"rootfs_path" db:"rootfs_path"` - RootfsSuffix string `json:"rootfs_suffix" db:"rootfs_suffix"` - PCIEnabled bool `json:"pci_enabled" db:"pci_enabled"` - NestedVirt bool `json:"nested_virt" db:"nested_virt"` - RemoteExec bool `json:"remote_exec" db:"remote_exec"` - EnableLogging bool `json:"enable_logging" db:"enable_logging"` - EnableMetrics bool `json:"enable_metrics" db:"enable_metrics"` - EnableConsole bool `json:"enable_console" db:"enable_console"` - CreatedAt string `json:"created_at" db:"created_at"` - UpdatedAt string `json:"updated_at" db:"updated_at"` + ID string `json:"id" db:"id"` + Name string `json:"name" db:"name"` + Status VMStatus `json:"status" db:"status"` + PID int `json:"pid" db:"pid"` + IPv4 string `json:"ipv4" db:"ipv4"` + MAC string `json:"mac" db:"mac"` + NetworkID string `json:"network_id" db:"network_id"` + TapDevice string `json:"tap_device" db:"tap_device"` + ImageID string `json:"image_id" db:"image_id"` + KernelID string `json:"kernel_id" db:"kernel_id"` + BinaryID string `json:"binary_id" db:"binary_id"` + JailerBinaryID string `json:"jailer_binary_id" db:"jailer_binary_id"` + APISocketPath string `json:"api_socket_path" db:"api_socket_path"` + ConfigPath string `json:"config_path" db:"config_path"` + CloudInitMode string `json:"cloud_init_mode" db:"cloud_init_mode"` + VCPUCount int `json:"vcpu_count" db:"vcpu_count"` + MemSizeMiB int `json:"mem_size_mib" db:"mem_size_mib"` + DiskSizeMiB int `json:"disk_size_mib" db:"disk_size_mib"` + RootfsPath string `json:"rootfs_path" db:"rootfs_path"` + RootfsSuffix string `json:"rootfs_suffix" db:"rootfs_suffix"` + PCIEnabled bool `json:"pci_enabled" db:"pci_enabled"` + NestedVirt bool `json:"nested_virt" db:"nested_virt"` + RemoteExec bool `json:"remote_exec" db:"remote_exec"` + EnableLogging bool `json:"enable_logging" db:"enable_logging"` + EnableMetrics bool `json:"enable_metrics" db:"enable_metrics"` + EnableConsole bool `json:"enable_console" db:"enable_console"` + CreatedAt string `json:"created_at" db:"created_at"` + UpdatedAt string `json:"updated_at" db:"updated_at"` // Optional fields RelaySocketPath *string `json:"relay_socket_path,omitempty" db:"relay_socket_path"` @@ -88,6 +89,7 @@ type VMItem struct { Kernel *KernelItem `json:"kernel,omitempty"` Image *ImageItem `json:"image,omitempty"` Binary *BinaryItem `json:"binary,omitempty"` + Jailer *BinaryItem `json:"jailer,omitempty"` Network *NetworkItem `json:"network,omitempty"` Volumes []*VolumeItem `json:"volumes,omitempty"` Vsock *VsockConfigItem `json:"vsock,omitempty"` diff --git a/internal/service/jailer/entry.go b/internal/service/jailer/entry.go new file mode 100644 index 00000000..72ea6949 --- /dev/null +++ b/internal/service/jailer/entry.go @@ -0,0 +1,763 @@ +package jailer + +import ( + "context" + "crypto/sha256" + "encoding/hex" + "encoding/json" + "fmt" + "io" + "os" + "path/filepath" + "regexp" + "sort" + "strconv" + "strings" + "syscall" + + "golang.org/x/sys/unix" + + "mvmctl/internal/infra" + "mvmctl/internal/lib/archive" + "mvmctl/internal/lib/system" +) + +var ( + versionPattern = regexp.MustCompile(`^[0-9]+\.[0-9]+(?:\.[0-9]+)?(?:[-+][A-Za-z0-9.-]+)?$`) + vmIDPattern = regexp.MustCompile(`^[a-f0-9]{32}$`) + driveIDPattern = regexp.MustCompile(`^[a-f0-9]{64}$`) + snapshotIDPattern = regexp.MustCompile(`^[a-f0-9]{64}$`) +) + +type managedPaths struct { + cacheRoot string + vmsRoot string + kernelsRoot string + volumesRoot string + snapshotsRoot string + vmDir string + uid uint32 +} + +// Config selects one fixed-policy privileged Jailer operation. +type Config struct { + Action string + Version string + Arch string + SHA256 string + VMID string + VMDir string + SnapshotDir string + DriveID string + HostPath string + ReadOnly bool +} + +// Run executes one privileged Jailer operation. +// CRITICAL: No raw executable, identity, chroot, mount target, or Jailer argument is accepted. +func Run(ctx context.Context, cfg Config) error { + if !system.IsRoot() { + return fmt.Errorf("jailer service requires root") + } + switch cfg.Action { + case "install": + return installRelease(ctx, cfg) + case "launch": + return launch(cfg) + case "cleanup": + return cleanup(cfg.VMID) + case "expose-snapshot": + return exposeSnapshot(cfg.VMID, cfg.VMDir, cfg.SnapshotDir) + case "expose-volume": + return exposeVolume(cfg.VMID, cfg.VMDir, cfg.DriveID, cfg.HostPath, cfg.ReadOnly) + case "remove-volume": + return removeVolume(cfg.VMID, cfg.DriveID) + case "remove-release": + return removeRelease(cfg.Version) + default: + return fmt.Errorf("unsupported jailer service action %q", cfg.Action) + } +} + +func installRelease(ctx context.Context, cfg Config) error { + if !versionPattern.MatchString(cfg.Version) { + return fmt.Errorf("invalid Firecracker version") + } + if cfg.Arch != "x86_64" && cfg.Arch != "aarch64" { + return fmt.Errorf("unsupported Firecracker architecture") + } + if len(cfg.SHA256) != sha256.Size*2 { + return fmt.Errorf("invalid release checksum") + } + if err := ensureTrustedDirectoryChain(infra.TrustedBinaryRoot); err != nil { + return fmt.Errorf("prepare trusted binary root: %w", err) + } + tmp, err := os.CreateTemp(infra.TrustedBinaryRoot, ".release-*.tgz") + if err != nil { + return fmt.Errorf("create trusted archive: %w", err) + } + tmpPath := tmp.Name() + defer os.Remove(tmpPath) + hash := sha256.New() + if _, err := io.Copy(io.MultiWriter(tmp, hash), os.Stdin); err != nil { + tmp.Close() + return fmt.Errorf("copy verified release archive: %w", err) + } + if err := tmp.Close(); err != nil { + return fmt.Errorf("close trusted archive: %w", err) + } + actual := hex.EncodeToString(hash.Sum(nil)) + if !strings.EqualFold(actual, cfg.SHA256) { + return fmt.Errorf("release checksum changed before trusted installation") + } + + destDir := filepath.Join(infra.TrustedBinaryRoot, cfg.Version) + staging, err := os.MkdirTemp(infra.TrustedBinaryRoot, ".pair-*") + if err != nil { + return fmt.Errorf("create trusted pair staging directory: %w", err) + } + defer os.RemoveAll(staging) + fcPath := filepath.Join(staging, "firecracker") + jlPath := filepath.Join(staging, "jailer") + if err := archive.ExtractRenamed(ctx, tmpPath, []archive.RenameEntry{ + {ArchiveName: fmt.Sprintf("firecracker-v%s-%s", cfg.Version, cfg.Arch), OutputPath: fcPath, Mode: 0755}, + {ArchiveName: fmt.Sprintf("jailer-v%s-%s", cfg.Version, cfg.Arch), OutputPath: jlPath, Mode: 0755}, + }); err != nil { + return fmt.Errorf("extract trusted release pair: %w", err) + } + if err := os.Chmod(staging, 0755); err != nil { + return fmt.Errorf("set trusted release directory mode: %w", err) + } + if !validTrustedPair(staging) { + return fmt.Errorf("staged trusted release pair failed ownership or mode validation") + } + if err := syncTrustedPair(staging); err != nil { + return fmt.Errorf("sync trusted release pair: %w", err) + } + + destInfo, destErr := os.Lstat(destDir) + switch { + case destErr == nil: + if destInfo.IsDir() && validTrustedPair(destDir) { + same, err := pairsEqual(staging, destDir) + if err != nil { + return fmt.Errorf("compare trusted release pairs: %w", err) + } + if same { + return writeInstallResult(destDir) + } + } + if err := unix.Renameat2( + unix.AT_FDCWD, + staging, + unix.AT_FDCWD, + destDir, + unix.RENAME_EXCHANGE, + ); err != nil { + return fmt.Errorf("atomically replace trusted release pair: %w", err) + } + if err := syncDirectory(infra.TrustedBinaryRoot); err != nil { + return fmt.Errorf("sync trusted binary root after replacement: %w", err) + } + if err := os.RemoveAll(staging); err != nil { + return fmt.Errorf("remove replaced trusted release pair: %w", err) + } + if err := syncDirectory(infra.TrustedBinaryRoot); err != nil { + return fmt.Errorf("sync trusted binary root after old-pair removal: %w", err) + } + case os.IsNotExist(destErr): + if err := os.Rename(staging, destDir); err != nil { + return fmt.Errorf("install trusted release pair: %w", err) + } + if err := syncDirectory(infra.TrustedBinaryRoot); err != nil { + return fmt.Errorf("sync trusted binary root after installation: %w", err) + } + default: + return fmt.Errorf("inspect existing trusted release pair: %w", destErr) + } + return writeInstallResult(destDir) +} + +func writeInstallResult(destDir string) error { + result := InstallResult{ + FirecrackerPath: filepath.Join(destDir, "firecracker"), + JailerPath: filepath.Join(destDir, "jailer"), + } + return json.NewEncoder(os.Stdout).Encode(result) +} + +func launch(cfg Config) error { + manifest, err := loadManifest(cfg.VMID, cfg.VMDir) + if err != nil { + return err + } + if err := ensureTrustedDirectoryChain(infra.JailerChrootBase); err != nil { + return fmt.Errorf("prepare Jailer chroot base: %w", err) + } + if err := cleanup(manifest.VMID); err != nil { + return err + } + jailRoot := jailRoot(manifest.VMID) + if err := ensureTrustedDirectoryChain(jailRoot); err != nil { + return fmt.Errorf("prepare jail root: %w", err) + } + if err := mountResource(manifest.VMDir, filepath.Join(jailRoot, "run", "mvm"), false, true); err != nil { + return err + } + if err := mountResource(manifest.KernelPath, filepath.Join(jailRoot, "kernel"), true, false); err != nil { + return err + } + if err := mountResource(manifest.RootfsPath, filepath.Join(jailRoot, "rootfs"), false, false); err != nil { + return err + } + if manifest.ISOPath != "" { + if err := mountResource(manifest.ISOPath, filepath.Join(jailRoot, "cloud-init.iso"), true, false); err != nil { + return err + } + } + if manifest.SnapshotDir != "" { + if err := mountResource(manifest.SnapshotDir, filepath.Join(jailRoot, "snapshot"), false, true); err != nil { + return err + } + } + for _, volume := range manifest.Volumes { + if !driveIDPattern.MatchString(volume.DriveID) { + return fmt.Errorf("invalid volume drive ID") + } + target := filepath.Join(jailRoot, "volumes", volume.DriveID) + if err := mountResource(volume.HostPath, target, volume.ReadOnly, false); err != nil { + return err + } + } + + uid, gid, err := invokingIdentity() + if err != nil { + return err + } + if err := writeProcessPID(manifest.PIDPath, os.Getpid(), uid, gid); err != nil { + return fmt.Errorf("write jailed process PID: %w", err) + } + jailerPath := filepath.Join(infra.TrustedBinaryRoot, manifest.Version, "jailer") + firecrackerPath := filepath.Join(infra.TrustedBinaryRoot, manifest.Version, "firecracker") + if err := validateTrustedExecutable(jailerPath); err != nil { + return err + } + if err := validateTrustedExecutable(firecrackerPath); err != nil { + return err + } + args := []string{ + jailerPath, + "--id", manifest.VMID, + "--exec-file", firecrackerPath, + "--uid", strconv.Itoa(uid), + "--gid", strconv.Itoa(gid), + "--chroot-base-dir", infra.JailerChrootBase, + "--", + "--api-sock", "/run/mvm/" + filepath.Base(manifest.APISocket), + } + if manifest.PCIEnabled { + args = append(args, "--enable-pci") + } + if !manifest.SnapshotMode { + args = append(args, "--config-file", "/run/mvm/"+filepath.Base(manifest.ConfigPath)) + } + return syscall.Exec(jailerPath, args, os.Environ()) +} + +func writeProcessPID(path string, pid, uid, gid int) error { + file, err := os.OpenFile(path, os.O_WRONLY|os.O_CREATE|os.O_TRUNC|syscall.O_NOFOLLOW, 0644) + if err != nil { + return err + } + defer file.Close() + if err := file.Chown(uid, gid); err != nil { + return err + } + if _, err := file.WriteString(strconv.Itoa(pid)); err != nil { + return err + } + return file.Sync() +} + +func loadManifest(vmID, vmDir string) (*LaunchManifest, error) { + paths, err := resolveManagedPaths(vmID, vmDir) + if err != nil { + return nil, err + } + manifestPath := filepath.Join(paths.vmDir, infra.JailerManifestFilename) + f, err := os.OpenFile(manifestPath, os.O_RDONLY|syscall.O_NOFOLLOW, 0) + if err != nil { + return nil, fmt.Errorf("open jailed launch manifest: %w", err) + } + defer f.Close() + var manifest LaunchManifest + if err := json.NewDecoder(io.LimitReader(f, 1<<20)).Decode(&manifest); err != nil { + return nil, fmt.Errorf("decode jailed launch manifest: %w", err) + } + if manifest.VMID != vmID || filepath.Clean(manifest.VMDir) != paths.vmDir { + return nil, fmt.Errorf("jailed launch manifest identity mismatch") + } + if !versionPattern.MatchString(manifest.Version) { + return nil, fmt.Errorf("invalid jailed launch version") + } + if err := validateManagedFile(manifestPath, paths.vmDir, paths.uid, true); err != nil { + return nil, fmt.Errorf("invalid managed launch manifest: %w", err) + } + for _, runtimePath := range []string{manifest.ConfigPath, manifest.RootfsPath} { + if err := validateManagedFile(runtimePath, paths.vmDir, paths.uid, true); err != nil { + return nil, fmt.Errorf("invalid managed VM runtime path: %w", err) + } + } + for _, runtimePath := range []string{manifest.PIDPath, manifest.APISocket} { + if err := validateManagedFile(runtimePath, paths.vmDir, paths.uid, false); err != nil { + return nil, fmt.Errorf("invalid managed VM runtime path: %w", err) + } + } + if manifest.ISOPath != "" { + if err := validateManagedFile(manifest.ISOPath, paths.vmDir, paths.uid, true); err != nil { + return nil, fmt.Errorf("invalid managed cloud-init ISO path: %w", err) + } + } + if err := validateManagedFile(manifest.KernelPath, paths.kernelsRoot, paths.uid, true); err != nil { + return nil, fmt.Errorf("invalid managed kernel path: %w", err) + } + if manifest.SnapshotDir != "" { + if err := validateManagedSnapshot(manifest.SnapshotDir, paths); err != nil { + return nil, err + } + } + for _, volume := range manifest.Volumes { + if !driveIDPattern.MatchString(volume.DriveID) { + return nil, fmt.Errorf("invalid managed volume drive ID") + } + if err := validateManagedFile(volume.HostPath, paths.volumesRoot, paths.uid, true); err != nil { + return nil, fmt.Errorf("invalid managed volume path: %w", err) + } + } + return &manifest, nil +} + +func resolveManagedPaths(vmID, vmDir string) (*managedPaths, error) { + if !vmIDPattern.MatchString(vmID) { + return nil, fmt.Errorf("invalid VM identity for jailed launch") + } + uid, _, err := invokingIdentity() + if err != nil { + return nil, err + } + cleanVMDir, err := canonicalExistingPath(vmDir) + if err != nil { + return nil, fmt.Errorf("invalid managed VM directory: %w", err) + } + if filepath.Base(cleanVMDir) != vmID || filepath.Base(filepath.Dir(cleanVMDir)) != "vms" { + return nil, fmt.Errorf("VM directory is outside the managed vms root") + } + cacheRoot := filepath.Dir(filepath.Dir(cleanVMDir)) + paths := &managedPaths{ + cacheRoot: cacheRoot, + vmsRoot: filepath.Join(cacheRoot, "vms"), + kernelsRoot: filepath.Join(cacheRoot, "kernels"), + volumesRoot: filepath.Join(cacheRoot, "volumes"), + snapshotsRoot: filepath.Join(cacheRoot, "snapshots"), + vmDir: cleanVMDir, + uid: uint32(uid), + } + for _, root := range []string{paths.cacheRoot, paths.vmsRoot, paths.vmDir} { + if err := validateManagedDirectory(root, paths.uid); err != nil { + return nil, err + } + } + return paths, nil +} + +func validateManagedSnapshot(path string, paths *managedPaths) error { + clean, err := canonicalExistingPath(path) + if err != nil { + return fmt.Errorf("invalid managed snapshot path: %w", err) + } + if filepath.Dir(clean) != paths.snapshotsRoot || !snapshotIDPattern.MatchString(filepath.Base(clean)) { + return fmt.Errorf("snapshot path is outside the managed snapshots root") + } + if err := validateManagedDirectory(paths.snapshotsRoot, paths.uid); err != nil { + return err + } + if err := validateManagedDirectory(clean, paths.uid); err != nil { + return err + } + return nil +} + +func validateManagedFile(path, root string, uid uint32, mustExist bool) error { + clean, err := filepath.Abs(path) + if err != nil || clean != filepath.Clean(path) || filepath.Dir(clean) != root { + return fmt.Errorf("path is outside managed root %s: %s", root, path) + } + if err := validateManagedDirectory(root, uid); err != nil { + return err + } + info, err := os.Lstat(clean) + if os.IsNotExist(err) && !mustExist { + return nil + } + if err != nil { + return fmt.Errorf("inspect managed file %s: %w", clean, err) + } + if info.Mode()&os.ModeSymlink != 0 || !info.Mode().IsRegular() { + return fmt.Errorf("managed path is not a regular non-symlink file: %s", clean) + } + stat, ok := info.Sys().(*syscall.Stat_t) + if !ok || stat.Uid != uid { + return fmt.Errorf("managed file is not owned by the invoking user: %s", clean) + } + resolved, err := filepath.EvalSymlinks(clean) + if err != nil || resolved != clean { + return fmt.Errorf("managed file path is not canonical: %s", clean) + } + return nil +} + +func validateManagedDirectory(path string, uid uint32) error { + clean, err := canonicalExistingPath(path) + if err != nil { + return fmt.Errorf("invalid managed directory: %w", err) + } + info, err := os.Lstat(clean) + if err != nil || info.Mode()&os.ModeSymlink != 0 || !info.IsDir() { + return fmt.Errorf("managed path is not a real directory: %s", path) + } + if info.Mode().Perm()&0022 != 0 { + return fmt.Errorf("managed directory is writable by other users: %s", clean) + } + stat, ok := info.Sys().(*syscall.Stat_t) + if !ok || stat.Uid != uid { + return fmt.Errorf("managed directory is not owned by the invoking user: %s", clean) + } + return nil +} + +func canonicalExistingPath(path string) (string, error) { + abs, err := filepath.Abs(path) + if err != nil || abs != filepath.Clean(path) { + return "", fmt.Errorf("path is not absolute and canonical: %s", path) + } + resolved, err := filepath.EvalSymlinks(abs) + if err != nil { + return "", err + } + if resolved != abs { + return "", fmt.Errorf("path contains a symlink: %s", path) + } + return abs, nil +} + +func pathWithin(path, base string) bool { + pathAbs, err := filepath.Abs(path) + if err != nil { + return false + } + baseAbs, err := filepath.Abs(base) + if err != nil { + return false + } + rel, err := filepath.Rel(baseAbs, pathAbs) + return err == nil && rel != ".." && !strings.HasPrefix(rel, ".."+string(filepath.Separator)) +} + +func mountResource(source, target string, readOnly, directory bool) error { + sourceFile, err := os.OpenFile(source, os.O_RDONLY|syscall.O_NOFOLLOW, 0) + if err != nil { + return fmt.Errorf("open jail resource %s: %w", source, err) + } + defer sourceFile.Close() + info, err := sourceFile.Stat() + if err != nil { + return fmt.Errorf("stat jail resource %s: %w", source, err) + } + if directory != info.IsDir() { + return fmt.Errorf("jail resource has unexpected type: %s", source) + } + if directory { + if err := os.MkdirAll(target, 0755); err != nil { + return err + } + } else { + if err := os.MkdirAll(filepath.Dir(target), 0755); err != nil { + return err + } + file, err := os.OpenFile(target, os.O_CREATE, 0644) + if err != nil { + return err + } + file.Close() + } + pinnedSource := filepath.Join("/proc/self/fd", strconv.FormatUint(uint64(sourceFile.Fd()), 10)) + if err := syscall.Mount(pinnedSource, target, "", syscall.MS_BIND, ""); err != nil { + return fmt.Errorf("bind jail resource %s: %w", source, err) + } + if readOnly { + if err := syscall.Mount("", target, "", syscall.MS_BIND|syscall.MS_REMOUNT|syscall.MS_RDONLY, ""); err != nil { + return fmt.Errorf("remount jail resource read-only %s: %w", target, err) + } + } + return nil +} + +func exposeSnapshot(vmID, vmDir, snapshotDir string) error { + paths, err := resolveManagedPaths(vmID, vmDir) + if err != nil { + return err + } + if err := validateManagedSnapshot(snapshotDir, paths); err != nil { + return err + } + target := filepath.Join(jailRoot(vmID), "snapshot") + if err := unmountIfMounted(target); err != nil { + return err + } + return mountResource(snapshotDir, target, false, true) +} + +func exposeVolume(vmID, vmDir, driveID, hostPath string, readOnly bool) error { + paths, err := resolveManagedPaths(vmID, vmDir) + if err != nil { + return err + } + if !driveIDPattern.MatchString(driveID) { + return fmt.Errorf("invalid volume exposure request") + } + if err := validateManagedFile(hostPath, paths.volumesRoot, paths.uid, true); err != nil { + return fmt.Errorf("invalid managed volume exposure path: %w", err) + } + target := filepath.Join(jailRoot(vmID), "volumes", driveID) + if err := unmountIfMounted(target); err != nil { + return err + } + return mountResource(hostPath, target, readOnly, false) +} + +func removeVolume(vmID, driveID string) error { + if !vmIDPattern.MatchString(vmID) || !driveIDPattern.MatchString(driveID) { + return fmt.Errorf("invalid volume removal request") + } + err := syscall.Unmount(filepath.Join(jailRoot(vmID), "volumes", driveID), syscall.MNT_DETACH) + if err != nil && err != syscall.EINVAL && err != syscall.ENOENT { + return err + } + return nil +} + +func unmountIfMounted(target string) error { + err := syscall.Unmount(target, syscall.MNT_DETACH) + if err != nil && err != syscall.EINVAL && err != syscall.ENOENT { + return fmt.Errorf("unmount existing jail resource %s: %w", target, err) + } + return nil +} + +func cleanup(vmID string) error { + if !vmIDPattern.MatchString(vmID) { + return fmt.Errorf("invalid VM ID for jail cleanup") + } + root := jailRoot(vmID) + if _, err := os.Lstat(infra.JailerChrootBase); os.IsNotExist(err) { + return nil + } + if err := validateTrustedDirectoryChain(infra.JailerChrootBase); err != nil { + return fmt.Errorf("validate Jailer chroot base before cleanup: %w", err) + } + data, err := os.ReadFile("/proc/self/mountinfo") + if err == nil { + var targets []string + for line := range strings.SplitSeq(string(data), "\n") { + fields := strings.Fields(line) + if len(fields) > 4 && (fields[4] == root || strings.HasPrefix(fields[4], root+"/")) { + targets = append(targets, fields[4]) + } + } + sort.Slice(targets, func(i, j int) bool { return len(targets[i]) > len(targets[j]) }) + for _, target := range targets { + err := syscall.Unmount(target, syscall.MNT_DETACH) + if err != nil && err != syscall.EINVAL && err != syscall.ENOENT { + return fmt.Errorf("unmount jail resource %s: %w", target, err) + } + } + } + if err := os.RemoveAll(filepath.Dir(root)); err != nil && !os.IsNotExist(err) { + return fmt.Errorf("remove jail: %w", err) + } + return nil +} + +func removeRelease(version string) error { + if !versionPattern.MatchString(version) { + return fmt.Errorf("invalid release version") + } + if _, err := os.Lstat(infra.TrustedBinaryRoot); os.IsNotExist(err) { + return nil + } + if err := validateTrustedDirectoryChain(infra.TrustedBinaryRoot); err != nil { + return fmt.Errorf("validate trusted binary root before removal: %w", err) + } + return os.RemoveAll(filepath.Join(infra.TrustedBinaryRoot, version)) +} + +func jailRoot(vmID string) string { + return filepath.Join(infra.JailerChrootBase, "firecracker", vmID, "root") +} + +func invokingIdentity() (int, int, error) { + uid, uidErr := strconv.Atoi(os.Getenv("SUDO_UID")) + gid, gidErr := strconv.Atoi(os.Getenv("SUDO_GID")) + if uidErr != nil || gidErr != nil || uid <= 0 || gid <= 0 { + return 0, 0, fmt.Errorf("cannot determine non-root invoking identity") + } + return uid, gid, nil +} + +func validateTrustedExecutable(path string) error { + resolved, err := filepath.EvalSymlinks(path) + if err != nil || resolved != path || !pathWithin(path, infra.TrustedBinaryRoot) { + return fmt.Errorf("trusted executable path is invalid: %s", path) + } + if err := validateTrustedDirectoryChain(filepath.Dir(path)); err != nil { + return fmt.Errorf("trusted executable parent validation failed: %w", err) + } + info, err := os.Stat(path) + if err != nil || !info.Mode().IsRegular() || info.Mode().Perm()&0022 != 0 || info.Mode().Perm()&0111 == 0 { + return fmt.Errorf("trusted executable is unusable: %s", path) + } + stat, ok := info.Sys().(*syscall.Stat_t) + if !ok || stat.Uid != 0 { + return fmt.Errorf("trusted executable is not root-owned: %s", path) + } + return nil +} + +func ensureTrustedDirectoryChain(path string) error { + abs, err := filepath.Abs(path) + if err != nil || abs != filepath.Clean(path) { + return fmt.Errorf("trusted directory path is not canonical: %s", path) + } + current := string(filepath.Separator) + if err := validateTrustedDirectory(current); err != nil { + return err + } + for _, component := range strings.Split(strings.TrimPrefix(abs, string(filepath.Separator)), string(filepath.Separator)) { + current = filepath.Join(current, component) + if _, err := os.Lstat(current); os.IsNotExist(err) { + if err := os.Mkdir(current, 0755); err != nil { + return fmt.Errorf("create trusted directory %s: %w", current, err) + } + } else if err != nil { + return fmt.Errorf("inspect trusted directory %s: %w", current, err) + } + if err := validateTrustedDirectory(current); err != nil { + return err + } + } + return nil +} + +func validateTrustedDirectoryChain(path string) error { + abs, err := filepath.Abs(path) + if err != nil || abs != filepath.Clean(path) { + return fmt.Errorf("trusted directory path is not canonical: %s", path) + } + current := string(filepath.Separator) + if err := validateTrustedDirectory(current); err != nil { + return err + } + for _, component := range strings.Split(strings.TrimPrefix(abs, string(filepath.Separator)), string(filepath.Separator)) { + current = filepath.Join(current, component) + if err := validateTrustedDirectory(current); err != nil { + return err + } + } + return nil +} + +func validateTrustedDirectory(path string) error { + info, err := os.Lstat(path) + if err != nil { + return fmt.Errorf("inspect trusted directory %s: %w", path, err) + } + if info.Mode()&os.ModeSymlink != 0 || !info.IsDir() { + return fmt.Errorf("trusted path component is not a real directory: %s", path) + } + if info.Mode().Perm()&0022 != 0 { + return fmt.Errorf("trusted path component is writable by non-root users: %s", path) + } + stat, ok := info.Sys().(*syscall.Stat_t) + if !ok || stat.Uid != 0 { + return fmt.Errorf("trusted path component is not root-owned: %s", path) + } + return nil +} + +func validTrustedPair(dir string) bool { + if err := validateTrustedDirectoryChain(dir); err != nil { + return false + } + for _, name := range []string{"firecracker", "jailer"} { + if err := validateTrustedExecutable(filepath.Join(dir, name)); err != nil { + return false + } + } + return true +} + +func pairsEqual(first, second string) (bool, error) { + for _, name := range []string{"firecracker", "jailer"} { + firstHash, err := fileSHA256(filepath.Join(first, name)) + if err != nil { + return false, err + } + secondHash, err := fileSHA256(filepath.Join(second, name)) + if err != nil { + return false, err + } + if firstHash != secondHash { + return false, nil + } + } + return true, nil +} + +func fileSHA256(path string) (string, error) { + f, err := os.Open(path) + if err != nil { + return "", err + } + defer f.Close() + hash := sha256.New() + if _, err := io.Copy(hash, f); err != nil { + return "", err + } + return hex.EncodeToString(hash.Sum(nil)), nil +} + +func syncTrustedPair(dir string) error { + for _, name := range []string{"firecracker", "jailer"} { + f, err := os.Open(filepath.Join(dir, name)) + if err != nil { + return err + } + if err := f.Sync(); err != nil { + f.Close() + return err + } + if err := f.Close(); err != nil { + return err + } + } + return syncDirectory(dir) +} + +func syncDirectory(path string) error { + dir, err := os.Open(path) + if err != nil { + return err + } + defer dir.Close() + return dir.Sync() +} diff --git a/internal/service/jailer/entry_test.go b/internal/service/jailer/entry_test.go new file mode 100644 index 00000000..c0661eba --- /dev/null +++ b/internal/service/jailer/entry_test.go @@ -0,0 +1,270 @@ +package jailer + +import ( + "context" + "encoding/json" + "os" + "path/filepath" + "strconv" + "testing" + + "github.com/google/go-cmp/cmp" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "mvmctl/internal/infra" +) + +const ( + testVMID = "0123456789abcdef0123456789abcdef" + testResourceID = "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef" +) + +func TestLoadManifest_AcceptsExactManagedResources(t *testing.T) { + fixture := newManagedFixture(t) + got, err := loadManifest(testVMID, fixture.vmDir) + require.NoError(t, err) + require.NotNil(t, got) + if diff := cmp.Diff(&fixture.manifest, got); diff != "" { + t.Errorf("validated manifest mismatch (-want +got):\n%s", diff) + } +} + +// Rationale: The privileged service must not treat an attacker-controlled path +// as managed merely because its immediate parent has a managed collection name. +func TestLoadManifest_RejectsArbitraryCollectionPaths(t *testing.T) { + tests := map[string]struct { + mutate func(*testing.T, *LaunchManifest) + wantErr string + }{ + "kernel_from_arbitrary_collection": { + mutate: func(t *testing.T, manifest *LaunchManifest) { + path := filepath.Join(t.TempDir(), "kernels", "vmlinux") + require.NoError(t, os.Mkdir(filepath.Dir(path), 0700)) + require.NoError(t, os.WriteFile(path, []byte("kernel"), 0600)) + manifest.KernelPath = path + }, + wantErr: "invalid managed kernel path", + }, + "snapshot_from_arbitrary_collection": { + mutate: func(t *testing.T, manifest *LaunchManifest) { + path := filepath.Join(t.TempDir(), "snapshots", testResourceID) + require.NoError(t, os.MkdirAll(path, 0700)) + manifest.SnapshotDir = path + }, + wantErr: "outside the managed snapshots root", + }, + "volume_from_arbitrary_collection": { + mutate: func(t *testing.T, manifest *LaunchManifest) { + path := filepath.Join(t.TempDir(), "volumes", testResourceID) + require.NoError(t, os.Mkdir(filepath.Dir(path), 0700)) + require.NoError(t, os.WriteFile(path, []byte("volume"), 0600)) + manifest.Volumes = []VolumeMount{{DriveID: testResourceID, HostPath: path}} + }, + wantErr: "invalid managed volume path", + }, + } + + for name, tc := range tests { + t.Run(name, func(t *testing.T) { + fixture := newManagedFixture(t) + tc.mutate(t, &fixture.manifest) + fixture.writeManifest(t) + + got, err := loadManifest(testVMID, fixture.vmDir) + require.Error(t, err) + assert.Contains(t, err.Error(), tc.wantErr) + assert.Nil(t, got) + }) + } +} + +// Rationale: Symlink substitution and group/other-writable directories would +// let the invoking user swap privileged mount sources after validation. +func TestManagedPathValidationRejectsUnsafeFilesystemState(t *testing.T) { + uid := uint32(os.Getuid()) + tests := map[string]func(*testing.T) error{ + "symlink_file": func(t *testing.T) error { + root := t.TempDir() + target := filepath.Join(root, "target") + require.NoError(t, os.WriteFile(target, []byte("target"), 0600)) + link := filepath.Join(root, "link") + require.NoError(t, os.Symlink(target, link)) + return validateManagedFile(link, root, uid, true) + }, + "other_writable_root": func(t *testing.T) error { + root := t.TempDir() + require.NoError(t, os.Chmod(root, 0777)) + return validateManagedDirectory(root, uid) + }, + } + for name, validate := range tests { + t.Run(name, func(t *testing.T) { + assert.Error(t, validate(t)) + }) + } +} + +// Rationale: Identity values cross a sudo boundary; root, missing, negative, +// and non-numeric identities must never become Jailer --uid/--gid arguments. +func TestInvokingIdentityRejectsMalformedValues(t *testing.T) { + tests := map[string]struct { + uid string + gid string + }{ + "missing": {}, + "root_uid": {uid: "0", gid: "1000"}, + "root_gid": {uid: "1000", gid: "0"}, + "negative": {uid: "-1", gid: "1000"}, + "non_numeric": {uid: "user", gid: "group"}, + } + for name, tc := range tests { + t.Run(name, func(t *testing.T) { + t.Setenv("SUDO_UID", tc.uid) + t.Setenv("SUDO_GID", tc.gid) + uid, gid, err := invokingIdentity() + assert.Error(t, err) + assert.Zero(t, uid) + assert.Zero(t, gid) + }) + } +} + +// Rationale: Privileged cleanup and volume operations must reject identifiers +// that could alter their fixed mount targets before issuing any syscall. +func TestPrivilegedTargetValidationRejectsArbitraryIdentifiers(t *testing.T) { + tests := map[string]func() error{ + "cleanup_path_traversal": func() error { return cleanup("../../etc") }, + "remove_volume_bad_vm": func() error { return removeVolume("../../etc", testResourceID) }, + "remove_volume_bad_drive": func() error { + return removeVolume(testVMID, "../../rootfs") + }, + } + for name, operation := range tests { + t.Run(name, func(t *testing.T) { + assert.Error(t, operation()) + }) + } +} + +// Rationale: The wire entry point must reject every action when invoked without +// its required root boundary, including unknown actions and path-bearing actions. +func TestRunRejectsUnprivilegedWireOperations(t *testing.T) { + tests := map[string]Config{ + "unknown_action": {Action: "exec-arbitrary"}, + "launch_arbitrary_path": {Action: "launch", VMID: testVMID, VMDir: "/tmp/attacker"}, + "expose_arbitrary_path": { + Action: "expose-volume", VMID: testVMID, VMDir: "/tmp/attacker", DriveID: testResourceID, + HostPath: "/etc/shadow", + }, + } + for name, cfg := range tests { + t.Run(name, func(t *testing.T) { + err := Run(context.Background(), cfg) + require.Error(t, err) + assert.Contains(t, err.Error(), "requires root") + }) + } +} + +func TestJailRootIsDeterministic(t *testing.T) { + want := filepath.Join(infra.JailerChrootBase, "firecracker", testVMID, "root") + first := jailRoot(testVMID) + second := jailRoot(testVMID) + assert.Equal(t, want, first) + assert.Equal(t, first, second) +} + +// Rationale: User-owned or symlinked executables must never satisfy the +// root-owned, non-user-writable trusted executable contract. +func TestValidateTrustedExecutableRejectsUntrustedPaths(t *testing.T) { + root := t.TempDir() + executable := filepath.Join(root, "firecracker") + require.NoError(t, os.WriteFile(executable, []byte("binary"), 0755)) + assert.Error(t, validateTrustedExecutable(executable)) + link := filepath.Join(root, "jailer") + require.NoError(t, os.Symlink(executable, link)) + assert.Error(t, validateTrustedExecutable(link)) +} + +// Rationale: Every trusted path component is traversed by a root process, so +// symlinked, user-owned, or group/other-writable directories must fail closed. +func TestValidateTrustedDirectoryRejectsUnsafeComponents(t *testing.T) { + tests := map[string]func(*testing.T) string{ + "symlink": func(t *testing.T) string { + target := t.TempDir() + link := filepath.Join(t.TempDir(), "trusted-link") + require.NoError(t, os.Symlink(target, link)) + return link + }, + "user_owned": func(t *testing.T) string { + return t.TempDir() + }, + "other_writable": func(t *testing.T) string { + path := t.TempDir() + require.NoError(t, os.Chmod(path, 0777)) + return path + }, + } + for name, setup := range tests { + t.Run(name, func(t *testing.T) { + assert.Error(t, validateTrustedDirectory(setup(t))) + }) + } +} + +type managedFixture struct { + vmDir string + manifest LaunchManifest +} + +// newManagedFixture creates the exact per-user cache layout accepted by the +// privileged manifest validator without requiring root, mounts, or subprocesses. +func newManagedFixture(t *testing.T) *managedFixture { + t.Helper() + uid := os.Getuid() + gid := os.Getgid() + require.Positive(t, uid) + require.Positive(t, gid) + t.Setenv("SUDO_UID", strconv.Itoa(uid)) + t.Setenv("SUDO_GID", strconv.Itoa(gid)) + cacheRoot := t.TempDir() + vmsRoot := filepath.Join(cacheRoot, "vms") + vmDir := filepath.Join(vmsRoot, testVMID) + kernelsRoot := filepath.Join(cacheRoot, "kernels") + volumesRoot := filepath.Join(cacheRoot, "volumes") + snapshotsRoot := filepath.Join(cacheRoot, "snapshots") + for _, path := range []string{vmsRoot, vmDir, kernelsRoot, volumesRoot, snapshotsRoot} { + require.NoError(t, os.Mkdir(path, 0700)) + } + snapshotDir := filepath.Join(snapshotsRoot, testResourceID) + require.NoError(t, os.Mkdir(snapshotDir, 0700)) + files := map[string]string{ + filepath.Join(vmDir, "config.json"): "{}", + filepath.Join(vmDir, "rootfs.ext4"): "rootfs", + filepath.Join(vmDir, "cloud-init.iso"): "iso", + filepath.Join(kernelsRoot, "vmlinux"): "kernel", + filepath.Join(volumesRoot, "volume.raw"): "volume", + } + for path, content := range files { + require.NoError(t, os.WriteFile(path, []byte(content), 0600)) + } + fixture := &managedFixture{vmDir: vmDir} + fixture.manifest = LaunchManifest{ + VMID: testVMID, VMDir: vmDir, Version: "1.16.0", + ConfigPath: filepath.Join(vmDir, "config.json"), PIDPath: filepath.Join(vmDir, "firecracker.pid"), + KernelPath: filepath.Join(kernelsRoot, "vmlinux"), RootfsPath: filepath.Join(vmDir, "rootfs.ext4"), + ISOPath: filepath.Join(vmDir, "cloud-init.iso"), SnapshotDir: snapshotDir, + Volumes: []VolumeMount{{DriveID: testResourceID, HostPath: filepath.Join(volumesRoot, "volume.raw")}}, + APISocket: filepath.Join(vmDir, "firecracker.socket"), + } + fixture.writeManifest(t) + return fixture +} + +func (f *managedFixture) writeManifest(t *testing.T) { + t.Helper() + data, err := json.Marshal(f.manifest) + require.NoError(t, err) + require.NoError(t, os.WriteFile(filepath.Join(f.vmDir, infra.JailerManifestFilename), data, 0600)) +} diff --git a/internal/service/jailer/spawn.go b/internal/service/jailer/spawn.go new file mode 100644 index 00000000..98846578 --- /dev/null +++ b/internal/service/jailer/spawn.go @@ -0,0 +1,127 @@ +package jailer + +import ( + "bytes" + "context" + "encoding/json" + "fmt" + "os" + "path/filepath" + "strconv" + + "mvmctl/internal/lib/system" +) + +// InstallRelease installs an already checksum-verified release archive into the trusted store. +func InstallRelease( + ctx context.Context, + version, arch, archivePath, expectedSHA256 string, +) (*InstallResult, error) { + archiveFile, err := os.Open(archivePath) + if err != nil { + return nil, fmt.Errorf("open verified Firecracker archive: %w", err) + } + defer archiveFile.Close() + + var stdout, stderr bytes.Buffer + cmd, err := system.SpawnService(ctx, system.SpawnConfig{ + Name: "jailer", + Privileged: true, + Stdin: archiveFile, + Stdout: &stdout, + Stderr: &stderr, + Args: []string{ + "install", "--version", version, "--arch", arch, "--sha256", expectedSHA256, + }, + }) + if err != nil { + return nil, fmt.Errorf("spawn trusted pair installer: %w", err) + } + if err := cmd.Wait(); err != nil { + return nil, fmt.Errorf("trusted pair installer failed: %s: %w", stderr.String(), err) + } + var result InstallResult + if err := json.Unmarshal(stdout.Bytes(), &result); err != nil { + return nil, fmt.Errorf("decode trusted pair installer result: %w", err) + } + return &result, nil +} + +// Launch starts the privileged Jailer service for a validated per-VM manifest. +func Launch( + ctx context.Context, + vmID, vmDir string, + stdin *os.File, + stdout, stderr *os.File, +) (*os.Process, error) { + if err := ctx.Err(); err != nil { + return nil, err + } + // The VM must outlive the foreground CLI command; lifecycle controllers own shutdown. + cmd, err := system.SpawnService(nil, system.SpawnConfig{ + Name: "jailer", + Privileged: true, + Stdin: stdin, + Stdout: stdout, + Stderr: stderr, + Args: []string{"launch", "--vm-id", vmID, "--vm-dir", vmDir}, + }) + if err != nil { + return nil, err + } + return cmd.Process, nil +} + +// Cleanup removes mounts and the chroot for one VM. It is idempotent. +func Cleanup(ctx context.Context, vmID string) error { + return runAndWait(ctx, []string{"cleanup", "--vm-id", vmID}) +} + +// ExposeSnapshot bind-mounts one snapshot directory into an existing jail. +func ExposeSnapshot(ctx context.Context, vmID, snapshotDir string) error { + vmDir := vmDirForManagedResource(vmID, snapshotDir) + return runAndWait(ctx, []string{ + "expose-snapshot", "--vm-id", vmID, "--vm-dir", vmDir, "--snapshot-dir", snapshotDir, + }) +} + +// ExposeVolume bind-mounts one volume into an existing jail. +func ExposeVolume(ctx context.Context, vmID, driveID, hostPath string, readOnly bool) error { + vmDir := vmDirForManagedResource(vmID, hostPath) + return runAndWait(ctx, []string{ + "expose-volume", "--vm-id", vmID, "--vm-dir", vmDir, "--drive-id", driveID, + "--host-path", hostPath, "--read-only=" + strconv.FormatBool(readOnly), + }) +} + +func vmDirForManagedResource(vmID, resourcePath string) string { + cacheDir := filepath.Dir(filepath.Dir(filepath.Clean(resourcePath))) + return filepath.Join(cacheDir, "vms", vmID) +} + +// RemoveVolume unmounts one volume from an existing jail. +func RemoveVolume(ctx context.Context, vmID, driveID string) error { + return runAndWait(ctx, []string{"remove-volume", "--vm-id", vmID, "--drive-id", driveID}) +} + +// RemoveRelease removes one trusted exact-version release directory. +func RemoveRelease(ctx context.Context, version string) error { + return runAndWait(ctx, []string{"remove-release", "--version", version}) +} + +func runAndWait(ctx context.Context, args []string) error { + var stderr bytes.Buffer + cmd, err := system.SpawnService(ctx, system.SpawnConfig{ + Name: "jailer", + Privileged: true, + Stderr: &stderr, + Args: args, + }) + if err != nil { + return err + } + if err := cmd.Wait(); err != nil { + return fmt.Errorf("jailer service failed: %s: %w", stderr.String(), err) + } + return nil +} diff --git a/internal/service/jailer/spawn_test.go b/internal/service/jailer/spawn_test.go new file mode 100644 index 00000000..dee7ce06 --- /dev/null +++ b/internal/service/jailer/spawn_test.go @@ -0,0 +1,40 @@ +package jailer + +import ( + "context" + "path/filepath" + "testing" + + "github.com/stretchr/testify/assert" +) + +func TestVMDirForManagedResource(t *testing.T) { + tests := map[string]struct { + resource string + want string + }{ + "volume": { + resource: "/home/user/.cache/mvmctl/volumes/data.raw", + want: filepath.Join("/home/user/.cache/mvmctl", "vms", testVMID), + }, + "snapshot": { + resource: "/home/user/.cache/mvmctl/snapshots/" + testResourceID, + want: filepath.Join("/home/user/.cache/mvmctl", "vms", testVMID), + }, + } + for name, tc := range tests { + t.Run(name, func(t *testing.T) { + assert.Equal(t, tc.want, vmDirForManagedResource(testVMID, tc.resource)) + }) + } +} + +// Rationale: A cancelled caller must not launch the detached privileged +// service, because lifecycle ownership has not yet transferred to a process. +func TestLaunchRejectsCancelledContext(t *testing.T) { + ctx, cancel := context.WithCancel(context.Background()) + cancel() + process, err := Launch(ctx, testVMID, "/managed/vms/"+testVMID, nil, nil, nil) + assert.ErrorIs(t, err, context.Canceled) + assert.Nil(t, process) +} diff --git a/internal/service/jailer/wire.go b/internal/service/jailer/wire.go new file mode 100644 index 00000000..746159bf --- /dev/null +++ b/internal/service/jailer/wire.go @@ -0,0 +1,33 @@ +// Package jailer provides the privileged, fixed-policy Firecracker Jailer service. +// Layer: Service — callers select typed operations, never raw Jailer arguments. +package jailer + +// VolumeMount describes one VM volume exposed inside a jail. +type VolumeMount struct { + DriveID string `json:"drive_id"` + HostPath string `json:"host_path"` + ReadOnly bool `json:"read_only"` +} + +// LaunchManifest contains the exact per-VM resources needed by Firecracker. +type LaunchManifest struct { + VMID string `json:"vm_id"` + VMDir string `json:"vm_dir"` + Version string `json:"version"` + ConfigPath string `json:"config_path"` + PIDPath string `json:"pid_path"` + KernelPath string `json:"kernel_path"` + RootfsPath string `json:"rootfs_path"` + ISOPath string `json:"iso_path,omitempty"` + SnapshotDir string `json:"snapshot_dir,omitempty"` + Volumes []VolumeMount `json:"volumes,omitempty"` + APISocket string `json:"api_socket"` + PCIEnabled bool `json:"pci_enabled"` + SnapshotMode bool `json:"snapshot_mode"` +} + +// InstallResult identifies the immutable trusted paths installed for a release. +type InstallResult struct { + FirecrackerPath string `json:"firecracker_path"` + JailerPath string `json:"jailer_path"` +} diff --git a/pkg/api/binary.go b/pkg/api/binary.go index b88a4b15..df858a74 100644 --- a/pkg/api/binary.go +++ b/pkg/api/binary.go @@ -153,9 +153,45 @@ func (op *Operation) BinaryRemove(ctx context.Context, input inputs.BinaryInput, }, } } + paired := make([]*model.BinaryItem, 0, len(binaries)*2) + seen := make(map[string]bool) + for _, item := range binaries { + firecracker, fcErr := op.Repos.Binary.GetByTypeAndVersion(ctx, "firecracker", item.Version) + jailer, jailerErr := op.Repos.Binary.GetByTypeAndVersion(ctx, "jailer", item.Version) + if fcErr != nil || jailerErr != nil || firecracker == nil || jailer == nil { + return &errs.BatchResult{Items: []errs.OperationResult{{ + Status: "error", Code: string(errs.CodeBinaryPairInvalid), + Message: fmt.Sprintf("exact Firecracker/Jailer pair is incomplete for version %s", item.Version), + }}} + } + for _, member := range []*model.BinaryItem{firecracker, jailer} { + if !seen[member.ID] { + seen[member.ID] = true + paired = append(paired, member) + } + } + } + binaries = paired // Enrich binaries with VM and snapshot references. enriched := binaries op.Enr.EnrichBinary(ctx, enriched, "vm", "snapshots") + if !force { + for _, bin := range enriched { + if len(bin.VMs) > 0 || len(bin.Snapshots) > 0 { + return &errs.BatchResult{Items: []errs.OperationResult{ + { + Status: "error", + Code: string(errs.CodeBinaryRemoveFailed), + Item: bin, + Message: fmt.Sprintf( + "cannot remove Firecracker/Jailer pair %s while it is referenced", + bin.Version, + ), + }, + }} + } + } + } items := make([]errs.OperationResult, 0) for _, bin := range enriched { // svc.Remove returns (*BinaryItem, error) @@ -202,16 +238,9 @@ func (op *Operation) BinaryRemoveByVersion(ctx context.Context, version string, if len(binariesToRemove) == 0 { return errs.New(errs.CodeBinaryNotFound, fmt.Sprintf("No binaries found for version %s", normalized)) } - // Batch-enrich with VM references. - for _, bin := range binariesToRemove { - if bin.VMs == nil { - vms, err := op.Repos.VM.FindByBinaryID(ctx, bin.ID) - if err == nil && len(vms) > 0 { - for _, vm := range vms { - bin.VMs = append(bin.VMs, vm) - } - } - } + if err := op.Enr.EnrichBinary(ctx, binariesToRemove, "vm", "snapshots"); err != nil { + return errs.WrapMsg(errs.CodeBinaryRemoveFailed, + fmt.Sprintf("failed to resolve binary pair references: %v", err), err) } for _, bin := range binariesToRemove { if _, err := op.Services.Binary.Remove(ctx, bin, force); err != nil { diff --git a/pkg/api/inputs/vm_create.go b/pkg/api/inputs/vm_create.go index efb52f85..fc6458b2 100644 --- a/pkg/api/inputs/vm_create.go +++ b/pkg/api/inputs/vm_create.go @@ -87,6 +87,7 @@ type ResolvedVMCreateInput struct { Image *model.ImageItem Kernel *model.KernelItem Binary *model.BinaryItem + Jailer *model.BinaryItem NetworkPrefixLen int CloudInitMode model.CloudInitMode SkipCINetworkConfig bool diff --git a/pkg/api/snapshot.go b/pkg/api/snapshot.go index d1073c08..660df7c7 100644 --- a/pkg/api/snapshot.go +++ b/pkg/api/snapshot.go @@ -494,7 +494,7 @@ func (op *Operation) SnapshotRestore( _ = op.Repos.VM.Upsert(ctx, vmItem) // Respawn Firecracker in snapshot mode (stays paused) - if err := op.vmRespawnFirecracker(ctx, vmItem, true); err != nil { + if err := op.vmRespawnFirecracker(ctx, vmItem, true, snap.SnapshotDir); err != nil { // Don't rollback — VM record exists, just log error slog.Error("Failed to respawn Firecracker for snapshot restore", "vm", name, "error", err) @@ -546,7 +546,7 @@ func (op *Operation) SnapshotRestore( return nil, errs.WrapMsg(errs.CodeSnapshotRestoreFailed, fmt.Sprintf("failed to remove old phantom symlink: %v", err), err) } - if err := os.Symlink(vmItem.RootfsPath, phantomPath); err != nil { + if err := os.Symlink("/rootfs", phantomPath); err != nil { if lockFile != nil { _ = syscall.Flock(int(lockFile.Fd()), syscall.LOCK_UN) lockFile.Close() @@ -620,6 +620,13 @@ func (op *Operation) cleanupRestoredVM(ctx context.Context, v *model.VMItem) { } } +func jailedRuntimePathForAPI(hostPath string) string { + if hostPath == "" { + return "" + } + return filepath.Join("/run/mvm", filepath.Base(hostPath)) +} + // --- SnapshotList --- // SnapshotList returns all snapshots. diff --git a/pkg/api/vm.go b/pkg/api/vm.go index acc13afc..d72aca00 100644 --- a/pkg/api/vm.go +++ b/pkg/api/vm.go @@ -102,6 +102,12 @@ func (op *Operation) VMCreate( if err != nil { return nil, err } + pair, err := op.Services.Binary.ResolvePair(ctx, sharedResolved.Binary) + if err != nil { + return nil, err + } + sharedResolved.Binary = pair.Firecracker + sharedResolved.Jailer = pair.Jailer sharedResolved.Provisioner = model.ProvisionerType(op.ProvisionerType) // Network bridge setup (shared across all VMs in the batch, done once). bridgeAddr, calcErr := network.ComputeBridgeAddress( @@ -680,7 +686,7 @@ func (op *Operation) vmBuilderExecute( fcConfig.RelayClientFD = &ptyFD } // Spawn Firecracker - if err := spawner.Spawn(); err != nil { + if err := spawner.Spawn(ctx); err != nil { fcErr = fmt.Errorf("failed to spawn Firecracker: %w", err) return } @@ -738,7 +744,7 @@ func (op *Operation) vmBuilderCleanup(ctx context.Context, builder *VMCreateBuil // Firecracker: stop firecracker process if builder.wasCreated("firecracker") && builder.fcManager != nil { fcSpawner := vm.NewFirecrackerSpawner(builder.fcManager) - fcSpawner.Cleanup() + fcSpawner.Cleanup(ctx) } // VM directory: remove all created files if builder.wasCreated("vm_dir") && builder.vmDir != "" { @@ -1047,7 +1053,7 @@ func (op *Operation) VMStart(ctx context.Context, input inputs.VMInput) *errs.Ba for _, vmLocal := range vms { // If VM is stopped, respawn Firecracker process if vmLocal.Status == model.VMStatusStopped { - if err := op.vmRespawnFirecracker(ctx, vmLocal, false); err != nil { + if err := op.vmRespawnFirecracker(ctx, vmLocal, false, ""); err != nil { results = append(results, errs.OperationResult{ Status: "error", Code: "vm.start_failed", Message: fmt.Sprintf("start '%s': %v", vmLocal.Name, err), @@ -1116,7 +1122,12 @@ func (op *Operation) VMStop(ctx context.Context, input inputs.VMInput) *errs.Bat } return &errs.BatchResult{Items: results} } -func (op *Operation) vmRespawnFirecracker(ctx context.Context, v *model.VMItem, snapshotMode bool) error { +func (op *Operation) vmRespawnFirecracker( + ctx context.Context, + v *model.VMItem, + snapshotMode bool, + snapshotDir string, +) error { vmDir := infra.GetVMDirByID(v.ID) // Network info is required — the VM record must have it pre-loaded. if v.Network == nil { @@ -1125,6 +1136,14 @@ func (op *Operation) vmRespawnFirecracker(ctx context.Context, v *model.VMItem, if v.Binary == nil || v.Binary.Path == "" { return fmt.Errorf("binary info is required for VM respawn") } + pair, err := op.Services.Binary.ResolvePair(ctx, v.Binary) + if err != nil { + return err + } + if v.JailerBinaryID != "" && pair.Jailer.ID != v.JailerBinaryID { + return errs.New(errs.CodeBinaryPairInvalid, + fmt.Sprintf("persisted Jailer does not match Firecracker version %s", v.Binary.Version)) + } if v.KernelID == "" || v.Kernel == nil || v.Kernel.Path == "" { return fmt.Errorf("kernel info is required for VM respawn") } @@ -1181,6 +1200,8 @@ func (op *Operation) vmRespawnFirecracker(ctx context.Context, v *model.VMItem, VMDir: vmDir, RootfsPath: v.RootfsPath, BinaryPath: v.Binary.Path, + JailerPath: pair.Jailer.Path, + BinaryVersion: pair.Firecracker.Version, KernelPath: v.Kernel.Path, VCPUCount: v.VCPUCount, MemSizeMiB: v.MemSizeMiB, @@ -1196,6 +1217,7 @@ func (op *Operation) vmRespawnFirecracker(ctx context.Context, v *model.VMItem, BootArgs: v.BootArgs, LSMFlags: v.LSMFlags, SnapshotMode: snapshotMode, + SnapshotDir: snapshotDir, ImageFSUUID: v.Image.FSUUID, ImageFSType: v.Image.FSType, // Full paths from DB (field names match DB column names) @@ -1234,7 +1256,7 @@ func (op *Operation) vmRespawnFirecracker(ctx context.Context, v *model.VMItem, if err := spawner.WriteToFile(); err != nil { return fmt.Errorf("write firecracker config: %w", err) } - if err := spawner.Spawn(); err != nil { + if err := spawner.Spawn(ctx); err != nil { slog.Warn("Failed to respawn Firecracker", "vm", v.Name, "error", err) return err } @@ -1298,7 +1320,7 @@ func (op *Operation) VMReboot(ctx context.Context, input inputs.VMInput) *errs.B controller := vm.NewController(vmLocal, repo) controller.Stop(ctx, input.Force) // After stop, respawn a fresh firecracker process - if err := op.vmRespawnFirecracker(ctx, vmLocal, false); err != nil { + if err := op.vmRespawnFirecracker(ctx, vmLocal, false, ""); err != nil { results = append(results, errs.OperationResult{ Status: "error", Code: "vm.reboot_failed", Message: fmt.Sprintf("reboot '%s': %v", vmLocal.Name, err), @@ -1476,6 +1498,8 @@ func (c *VMCreateBuilder) buildFirecrackerConfig() *model.FirecrackerConfig { VMDir: c.vmDir, RootfsPath: c.rootfsPath, BinaryPath: c.resolved.Binary.Path, + JailerPath: c.resolved.Jailer.Path, + BinaryVersion: c.resolved.Binary.Version, KernelPath: c.resolved.Kernel.Path, VCPUCount: c.resolved.VCPUCount, MemSizeMiB: c.resolved.MemSizeMib, @@ -1552,6 +1576,7 @@ func (c *VMCreateBuilder) toVMModel() *model.VMItem { ImageID: c.resolved.Image.ID, KernelID: c.resolved.Kernel.ID, BinaryID: c.resolved.Binary.ID, + JailerBinaryID: c.resolved.Jailer.ID, VCPUCount: c.resolved.VCPUCount, MemSizeMiB: c.resolved.MemSizeMib, DiskSizeMiB: c.resolved.DiskSizeMib, diff --git a/pkg/errs/codes.go b/pkg/errs/codes.go index b3a7a6e4..8f88f7bc 100644 --- a/pkg/errs/codes.go +++ b/pkg/errs/codes.go @@ -62,15 +62,18 @@ const ( CodeKernelDefaultSetFailed Code = "kernel.default_set_failed" // --- Binary domain --- - CodeBinaryNotFound Code = "binary.not_found" - CodeBinaryAlreadyExists Code = "binary.already_exists" - CodeBinaryVersionGate Code = "binary.version.gate" - CodeBinaryError Code = "binary.error" - CodeBinaryPullFailed Code = "binary.pull_failed" - CodeBinaryRemoveFailed Code = "binary.remove_failed" - CodeBinaryDefaultSetFailed Code = "binary.default_set_failed" - CodeBinaryEnsureDefaultFailed Code = "binary.ensure_default_failed" - CodeBinaryNoCIVersion Code = "binary.no_ci_version" + CodeBinaryNotFound Code = "binary.not_found" + CodeBinaryAlreadyExists Code = "binary.already_exists" + CodeBinaryVersionGate Code = "binary.version.gate" + CodeBinaryError Code = "binary.error" + CodeBinaryPullFailed Code = "binary.pull_failed" + CodeBinaryRemoveFailed Code = "binary.remove_failed" + CodeBinaryDefaultSetFailed Code = "binary.default_set_failed" + CodeBinaryEnsureDefaultFailed Code = "binary.ensure_default_failed" + CodeBinaryNoCIVersion Code = "binary.no_ci_version" + CodeBinaryPairInvalid Code = "binary.pair.invalid" + CodeBinaryUntrusted Code = "binary.untrusted" + CodeBinaryTrustedInstallFailed Code = "binary.trusted_install.failed" // --- Volume domain --- CodeVolumeNotFound Code = "volume.not_found" diff --git a/pkg/errs/domain.go b/pkg/errs/domain.go index 48bc52f7..5f4e8fa3 100644 --- a/pkg/errs/domain.go +++ b/pkg/errs/domain.go @@ -147,15 +147,18 @@ var codeClassMap = map[Code]Class{ CodeKernelDefaultSetFailed: ClassInternal, // --- Binary domain --- - CodeBinaryNotFound: ClassValidation, - CodeBinaryAlreadyExists: ClassConflict, - CodeBinaryVersionGate: ClassValidation, - CodeBinaryError: ClassInternal, - CodeBinaryPullFailed: ClassInternal, - CodeBinaryRemoveFailed: ClassInternal, - CodeBinaryDefaultSetFailed: ClassInternal, - CodeBinaryEnsureDefaultFailed: ClassInternal, - CodeBinaryNoCIVersion: ClassValidation, + CodeBinaryNotFound: ClassValidation, + CodeBinaryAlreadyExists: ClassConflict, + CodeBinaryVersionGate: ClassValidation, + CodeBinaryError: ClassInternal, + CodeBinaryPullFailed: ClassInternal, + CodeBinaryRemoveFailed: ClassInternal, + CodeBinaryDefaultSetFailed: ClassInternal, + CodeBinaryEnsureDefaultFailed: ClassInternal, + CodeBinaryNoCIVersion: ClassValidation, + CodeBinaryPairInvalid: ClassValidation, + CodeBinaryUntrusted: ClassValidation, + CodeBinaryTrustedInstallFailed: ClassInternal, // --- Volume domain --- CodeVolumeNotFound: ClassValidation, @@ -317,15 +320,18 @@ var codeOpMap = map[Code]string{ CodeKernelImportFailed: "kernel", CodeKernelDefaultSetFailed: "kernel", - CodeBinaryNotFound: "binary", - CodeBinaryAlreadyExists: "binary", - CodeBinaryVersionGate: "binary", - CodeBinaryError: "binary", - CodeBinaryPullFailed: "binary", - CodeBinaryRemoveFailed: "binary", - CodeBinaryDefaultSetFailed: "binary", - CodeBinaryEnsureDefaultFailed: "binary", - CodeBinaryNoCIVersion: "binary", + CodeBinaryNotFound: "binary", + CodeBinaryAlreadyExists: "binary", + CodeBinaryVersionGate: "binary", + CodeBinaryError: "binary", + CodeBinaryPullFailed: "binary", + CodeBinaryRemoveFailed: "binary", + CodeBinaryDefaultSetFailed: "binary", + CodeBinaryEnsureDefaultFailed: "binary", + CodeBinaryNoCIVersion: "binary", + CodeBinaryPairInvalid: "binary", + CodeBinaryUntrusted: "binary", + CodeBinaryTrustedInstallFailed: "binary", CodeVolumeNotFound: "volume", CodeVolumeAlreadyExists: "volume", diff --git a/tests/system/COVERAGE_MATRIX.md b/tests/system/COVERAGE_MATRIX.md index c4b1d83c..93b27cad 100644 --- a/tests/system/COVERAGE_MATRIX.md +++ b/tests/system/COVERAGE_MATRIX.md @@ -587,6 +587,7 @@ removed, or when test coverage changes. | `run nocloudnet serve --help` | ⚡ Shallow | `run/test_run.py` | `TestRunHelp` | L1 — help output | | `run console relay --help` | ⚡ Shallow | `run/test_run.py` | `TestRunHelp` | L1 — help output | | `run provision --help` | ⚡ Shallow | `run/test_run.py` | `TestRunHelp` | L1 — help output | +| `run jailer` | ✅ Deep | `vm/test_jailer.py` | `TestCanonicalJailerLaunch`, `TestJailerFailClosed` | L2 — indirectly verifies privileged install, launch, cleanup, and no fallback | > Internal service subprocesses are tested indirectly through VM operations (console relay, nocloudnet HTTP, volume provisioning). @@ -609,6 +610,7 @@ removed, or when test coverage changes. | Stress create/destroy | ⚡ Shallow | `full_journeys/test_full_journeys.py` | `TestStressCreateDestroy` | L2: 5 sequential create→destroy cycles | | Export/import config journey | ⚡ Shallow | `full_journeys/test_full_journeys.py` | `TestExportImport` | L2: inspect --json schema verification | | Concurrent VM creation | ✅ Deep | `full_journeys/test_full_journeys.py` | `TestConcurrentVMCreation` | L3: 10 concurrent VMs, all running via ls --json, SSH into each | +| Canonical Jailer lifecycle | ✅ Deep | `vm/test_jailer.py`, `full_journeys/test_full_journeys.py` | `TestCanonicalJailerLaunch`, `TestJailerFailClosed` | L2: trusted pair, non-root jailed process, lifecycle, volume, snapshot, fail-closed | --- diff --git a/tests/system/bin/test_bin.py b/tests/system/bin/test_bin.py index 53a037b0..c2972321 100644 --- a/tests/system/bin/test_bin.py +++ b/tests/system/bin/test_bin.py @@ -711,10 +711,10 @@ class TestBinaryStoppedVMDeletion: pytest.mark.domain_bin, ] - def test_delete_binary_used_by_stopped_vm_does_not_error( + def test_delete_binary_used_by_stopped_vm_is_rejected( self, runner_vm: str, unique_vm_name: str, module_network: str ) -> None: - """Binary rm allows deleting binaries referenced by stopped VMs.""" + """Neither member of a persisted launch pair can be removed while referenced.""" vm_name = unique_vm_name ensure_vm_deps(runner_vm) @@ -747,14 +747,14 @@ def test_delete_binary_used_by_stopped_vm_does_not_error( binary_id_prefix = default_bin["id"][:6] result = _run_mvm(runner_vm, "bin", "rm", binary_id_prefix, check=False) - assert result.returncode in (0, 1) - - if result.returncode == 0: - bin_ls = _run_mvm(runner_vm, "bin", "ls", "--json", check=False) - if bin_ls.returncode == 0 and bin_ls.stdout.strip(): - bins_after: list[dict[str, Any]] = json.loads(bin_ls.stdout) - bin_ids = [b.get("id", "")[:6] for b in bins_after] - assert binary_id_prefix not in bin_ids + assert result.returncode != 0 + combined = (result.stdout + result.stderr).lower() + assert "referenced by" in combined + + bin_ls = _run_mvm(runner_vm, "bin", "ls", "--json") + bins_after: list[dict[str, Any]] = json.loads(bin_ls.stdout) + bin_ids = [b.get("id", "")[:6] for b in bins_after] + assert binary_id_prefix in bin_ids vm_ls = _run_mvm(runner_vm, "vm", "ls", "--json", check=False) if vm_ls.returncode == 0 and vm_ls.stdout.strip(): vms: list[dict[str, Any]] = json.loads(vm_ls.stdout) diff --git a/tests/system/full_journeys/test_full_journeys.py b/tests/system/full_journeys/test_full_journeys.py index ae21d9d7..16e97bd8 100644 --- a/tests/system/full_journeys/test_full_journeys.py +++ b/tests/system/full_journeys/test_full_journeys.py @@ -22,6 +22,14 @@ import pytest from tests.system.conftest import _run_mvm, _unique_subnet, wait_for_ssh +from tests.system.vm.test_jailer import ( + TestCanonicalJailerLaunch, + TestJailerFailClosed, +) + +# The orchestrator maps this domain to one explicit file. Re-exporting the +# focused Jailer journeys keeps canonical-launch coverage in that bounded run. +_JAILER_JOURNEYS = (TestCanonicalJailerLaunch, TestJailerFailClosed) pytestmark = [ pytest.mark.system, diff --git a/tests/system/invariants/test_invariants.py b/tests/system/invariants/test_invariants.py index ca6a478d..1b6045fa 100644 --- a/tests/system/invariants/test_invariants.py +++ b/tests/system/invariants/test_invariants.py @@ -154,6 +154,28 @@ class TestInvariants: pytestmark = [pytest.mark.system, pytest.mark.domain_invariant] + def test_every_vm_references_an_exact_firecracker_jailer_pair( + self, runner_vm: str + ) -> None: + """Persisted launcher IDs must resolve to present equal-version pair members.""" + vms: list[dict[str, Any]] = json.loads( + _run_mvm(runner_vm, "vm", "ls", "--json").stdout + ) + binaries: list[dict[str, Any]] = json.loads( + _run_mvm(runner_vm, "bin", "ls", "--json").stdout + ) + by_id = {item["id"]: item for item in binaries} + for vm in vms: + firecracker = by_id.get(vm.get("binary_id")) + jailer = by_id.get(vm.get("jailer_binary_id")) + assert firecracker is not None, f"VM {vm['name']} has no Firecracker record" + assert jailer is not None, f"VM {vm['name']} has no Jailer record" + assert firecracker.get("type") == "firecracker" + assert jailer.get("type") == "jailer" + assert firecracker.get("version") == jailer.get("version") + assert firecracker.get("is_present") is True + assert jailer.get("is_present") is True + def test_no_dangling_volume_references(self, runner_vm: str) -> None: """Every volume.vm_id must correspond to an existing VM.""" vol_result = _run_mvm(runner_vm, "volume", "ls", "--json") diff --git a/tests/system/vm/test_jailer.py b/tests/system/vm/test_jailer.py new file mode 100644 index 00000000..f9ff2c3e --- /dev/null +++ b/tests/system/vm/test_jailer.py @@ -0,0 +1,274 @@ +"""Canonical Firecracker Jailer launcher system tests — Tier 2 runner VM.""" + +from __future__ import annotations + +import json +import shlex +import uuid +from typing import Any, Generator + +import pytest + +from tests.system.conftest import ( + _guest_run, + _run_mvm, + cleanup_vm_resources, + create_vm_core, + ensure_vm_deps, +) + +pytestmark = [ + pytest.mark.system, + pytest.mark.domain_vm, + pytest.mark.tier2, + pytest.mark.needs_kvm, + pytest.mark.slow, +] + + +def _vm_entry(runner_vm: str, vm_name: str) -> dict[str, Any]: + result = _run_mvm(runner_vm, "vm", "ls", "--json") + vms: list[dict[str, Any]] = json.loads(result.stdout) + entry = next((vm for vm in vms if vm.get("name") == vm_name), None) + assert entry is not None, f"VM {vm_name!r} is absent from vm ls --json" + return entry + + +def _jail_root(vm_id: str) -> str: + return f"/var/lib/mvmctl/jailer/firecracker/{vm_id}/root" + + +@pytest.fixture +def jailed_vm(runner_vm: str) -> Generator[dict[str, Any], None, None]: + vm_name = f"sys-jailer-{uuid.uuid4().hex[:8]}" + net_name = f"sys-jailer-net-{uuid.uuid4().hex[:6]}" + vm = create_vm_core(vm_name, net_name, extra_args=["--console"]) + vm["network_name"] = net_name + try: + yield vm + finally: + cleanup_vm_resources(vm_name, net_name) + + +class TestCanonicalJailerLaunch: + def test_process_is_non_root_inside_expected_jail( + self, runner_vm: str, jailed_vm: dict[str, Any] + ) -> None: + """Prove the VM process is chrooted and runs as the invoking identity.""" + vm = _vm_entry(runner_vm, jailed_vm["name"]) + pid = int(vm["pid"]) + invoking_uid = int(_guest_run(runner_vm, "id -u").stdout.strip()) + + status = _guest_run(runner_vm, f"cat /proc/{pid}/status").stdout + uid_line = next(line for line in status.splitlines() if line.startswith("Uid:")) + process_uid = int(uid_line.split()[1]) + assert process_uid == invoking_uid + assert process_uid != 0 + + process_root = _guest_run( + runner_vm, f"readlink /proc/{pid}/root" + ).stdout.strip() + expected_root = _jail_root(vm["id"]) + assert process_root == expected_root + cmdline = _guest_run( + runner_vm, f"tr '\\0' ' ' < /proc/{pid}/cmdline" + ).stdout + assert "firecracker" in cmdline + + api_socket = vm["api_socket_path"] + socket_state = _guest_run( + runner_vm, f"test -S {shlex.quote(api_socket)} && echo socket" + ) + assert socket_state.stdout.strip() == "socket" + + def test_trusted_pair_is_root_owned_and_not_user_writable( + self, runner_vm: str, jailed_vm: dict[str, Any] + ) -> None: + """Verify both exact-version executables satisfy Jailer's trust contract.""" + vm = _vm_entry(runner_vm, jailed_vm["name"]) + binaries: list[dict[str, Any]] = json.loads( + _run_mvm(runner_vm, "bin", "ls", "--json").stdout + ) + selected_ids = {vm["binary_id"], vm["jailer_binary_id"]} + pair = [item for item in binaries if item.get("id") in selected_ids] + assert {item.get("type") for item in pair} == {"firecracker", "jailer"} + assert len({item.get("version") for item in pair}) == 1 + for item in pair: + path = shlex.quote(item["path"]) + stat = _guest_run(runner_vm, f"stat -Lc '%u %a' {path}").stdout.strip() + owner, mode_text = stat.split() + mode = int(mode_text, 8) + assert owner == "0" + assert mode & 0o022 == 0 + + def test_exec_cp_console_and_lifecycle_parity( + self, runner_vm: str, jailed_vm: dict[str, Any] + ) -> None: + """Exercise API socket, vsock exec/cp, console, stop/start, and reboot.""" + vm_name = jailed_vm["name"] + marker = f"jailer-{uuid.uuid4().hex}" + exec_result = _run_mvm( + runner_vm, "exec", vm_name, "--", "printf", marker + ) + assert exec_result.stdout == marker + + source = f"/tmp/{marker}.txt" + _guest_run(runner_vm, f"printf %s {shlex.quote(marker)} > {shlex.quote(source)}") + _run_mvm(runner_vm, "cp", source, f"{vm_name}:/tmp/jailer-marker.txt") + copied = _run_mvm( + runner_vm, "exec", vm_name, "--", "cat", "/tmp/jailer-marker.txt" + ) + assert copied.stdout.strip() == marker + + console = _run_mvm(runner_vm, "console", vm_name, "--state") + console_state = (console.stdout + console.stderr).lower() + assert "running" in console_state or "stopped" in console_state + + before = _vm_entry(runner_vm, vm_name) + jail = _jail_root(before["id"]) + _run_mvm(runner_vm, "vm", "stop", vm_name) + stopped = _vm_entry(runner_vm, vm_name) + assert stopped["status"] == "stopped" + assert _guest_run( + runner_vm, f"test ! -e {shlex.quote(jail)}", check=False + ).returncode == 0 + + _run_mvm(runner_vm, "vm", "start", vm_name) + started = _vm_entry(runner_vm, vm_name) + assert started["status"] == "running" + assert int(started["pid"]) != int(before["pid"]) + assert _guest_run( + runner_vm, f"test -d {shlex.quote(jail)}", check=False + ).returncode == 0 + + _run_mvm(runner_vm, "vm", "reboot", vm_name) + rebooted = _vm_entry(runner_vm, vm_name) + assert rebooted["status"] == "running" + assert int(rebooted["pid"]) != int(started["pid"]) + + def test_volume_and_snapshot_parity( + self, runner_vm: str, jailed_vm: dict[str, Any] + ) -> None: + """Selected volume and snapshot resources remain usable without collection mounts.""" + vm_name = jailed_vm["name"] + volume_name = f"sys-jailer-vol-{uuid.uuid4().hex[:6]}" + snapshot_id: str | None = None + try: + _run_mvm(runner_vm, "volume", "create", volume_name, "64M") + _run_mvm(runner_vm, "vm", "stop", vm_name) + _run_mvm(runner_vm, "volume", "attach", vm_name, volume_name) + attached = json.loads( + _run_mvm(runner_vm, "volume", "inspect", volume_name, "--json").stdout + ) + assert attached["volume"]["status"] == "attached" + _run_mvm(runner_vm, "vm", "start", vm_name) + partitions = _run_mvm( + runner_vm, "exec", vm_name, "--", "cat", "/proc/partitions" + ) + assert "vdb" in partitions.stdout + + _run_mvm(runner_vm, "snapshot", "create", vm_name) + snapshots: list[dict[str, Any]] = json.loads( + _run_mvm(runner_vm, "snapshot", "ls", "--json").stdout + ) + selected = [s for s in snapshots if s.get("source_vm_name") == vm_name] + assert selected + snapshot_id = selected[-1]["id"] + + _run_mvm(runner_vm, "vm", "stop", vm_name) + _run_mvm(runner_vm, "volume", "detach", vm_name, volume_name) + detached = json.loads( + _run_mvm(runner_vm, "volume", "inspect", volume_name, "--json").stdout + ) + assert detached["volume"]["status"] == "available" + + _run_mvm(runner_vm, "vm", "rm", vm_name, "--force") + restored = _run_mvm( + runner_vm, "snapshot", "restore", snapshot_id, vm_name, "--resume", + check=False, timeout=180, + ) + assert restored.returncode == 0, restored.stderr + restored_vm = _vm_entry(runner_vm, vm_name) + assert restored_vm["status"] == "running" + assert _guest_run( + runner_vm, f"test -d {shlex.quote(_jail_root(restored_vm['id']))}", + check=False, + ).returncode == 0 + finally: + _run_mvm(runner_vm, "vm", "rm", vm_name, "--force", check=False) + _run_mvm(runner_vm, "volume", "rm", volume_name, "--force", check=False) + if snapshot_id: + _run_mvm( + runner_vm, "snapshot", "rm", snapshot_id, "--force", check=False + ) + + +class TestJailerFailClosed: + def test_missing_exact_jailer_does_not_fallback_to_direct_firecracker( + self, runner_vm: str + ) -> None: + """An alternate-version Jailer cannot substitute for the selected exact pair.""" + ensure_vm_deps(runner_vm) + vm_name = f"sys-jailer-fail-{uuid.uuid4().hex[:8]}" + binaries: list[dict[str, Any]] = json.loads( + _run_mvm(runner_vm, "bin", "ls", "--json").stdout + ) + selected_fc = next( + item + for item in binaries + if item.get("type") == "firecracker" and item.get("is_default") + ) + selected_jailer = next( + item + for item in binaries + if item.get("type") == "jailer" + and item.get("version") == selected_fc["version"] + ) + alternate_version = "1.14.2" if selected_fc["version"] != "1.14.2" else "1.14.1" + alternate_preexisting = any( + item.get("version") == alternate_version and item.get("is_present") + for item in binaries + ) + try: + _run_mvm( + runner_vm, "bin", "pull", "firecracker", "--version", + alternate_version, "--force", check=False, timeout=300, + ) + alternate_bins: list[dict[str, Any]] = json.loads( + _run_mvm(runner_vm, "bin", "ls", "--json").stdout + ) + assert any( + item.get("type") == "jailer" + and item.get("version") == alternate_version + and item.get("is_present") + for item in alternate_bins + ) + _run_mvm( + runner_vm, "bin", "rm", selected_jailer["id"][:6], "--force" + ) + result = _run_mvm( + runner_vm, "vm", "create", vm_name, "--image", "alpine:3.23", + check=False, timeout=180, + ) + assert result.returncode != 0 + combined = (result.stdout + result.stderr).lower() + assert "jailer" in combined + assert "missing" in combined or "pair" in combined + listed = json.loads(_run_mvm(runner_vm, "vm", "ls", "--json").stdout) + assert not any(vm.get("name") == vm_name for vm in listed) + processes = _guest_run( + runner_vm, f"pgrep -af {shlex.quote(vm_name)}", check=False + ) + assert processes.returncode != 0 + finally: + _run_mvm(runner_vm, "vm", "rm", vm_name, "--force", check=False) + _run_mvm( + runner_vm, "bin", "pull", "firecracker", "--version", + selected_fc["version"], "--force", "--default", check=False, + timeout=300, + ) + if not alternate_preexisting: + _run_mvm( + runner_vm, "bin", "rm", "--version", alternate_version, + "--force", check=False, + ) From dc48a11cf63ea3d222be7464aed58be2987dd617 Mon Sep 17 00:00:00 2001 From: AlanD20 Date: Sun, 9 Aug 2026 13:33:46 +0200 Subject: [PATCH 0003/1457] feat(vm): enforce microVM cgroup limits --- CHANGELOG.md | 3 + docs/RESOURCE_MANAGEMENTS.md | 15 +- docs/RUNTIME.md | 4 +- docs/TROUBLESHOOTING.md | 16 +- docs/adr/0015-jailer-canonical-launcher.md | 8 +- .../HOW_AGENTS_WRITE_SYSTEM_TESTS.md | 2 +- docs/improvements/IMPROVEMENTS_008.md | 2 +- .../optimizations/next-level-optimizations.md | 4 +- docs/roadmaps/TENANT_MICROVM.md | 12 +- docs/system-test-architecture.md | 4 +- internal/core/config/constraints.go | 32 +++ internal/core/config/constraints_test.go | 45 ++++ internal/core/host/detector.go | 6 +- internal/core/host/probe.go | 22 ++ internal/core/host/probe_test.go | 42 ++++ internal/core/vm/controller.go | 5 +- internal/core/vm/firecracker.go | 4 + internal/core/vm/jailer.go | 224 ++++++++++++++++++ internal/core/vm/jailer_test.go | 101 +++++++- internal/core/vm/sqlite.go | 6 +- internal/core/vm/sqlite_test.go | 13 + internal/infra/constants.go | 55 +++-- .../db/migrations/003_vm_cgroup_limits.sql | 8 + internal/lib/db/migrations_jailer_test.go | 42 +++- internal/lib/model/firecracker.go | 117 +++++++-- internal/lib/model/firecracker_test.go | 70 ++++++ internal/lib/model/host.go | 25 +- internal/lib/model/vm.go | 57 ++--- internal/service/jailer/entry.go | 153 ++++++++++-- internal/service/jailer/entry_test.go | 57 +++++ internal/service/jailer/wire.go | 29 ++- pkg/api/inputs/vm_create.go | 12 + pkg/api/results/vm.go | 7 +- pkg/api/snapshot.go | 5 + pkg/api/vm.go | 132 ++++++++--- pkg/errs/codes.go | 36 +-- pkg/errs/domain.go | 72 +++--- scripts/run-system-tests.py | 4 + tests/system/COVERAGE_MATRIX.md | 1 + tests/system/vm/test_cgroup.py | 204 ++++++++++++++++ tests/system/vm/test_jailer.py | 14 ++ 41 files changed, 1439 insertions(+), 231 deletions(-) create mode 100644 internal/lib/db/migrations/003_vm_cgroup_limits.sql create mode 100644 internal/lib/model/firecracker_test.go create mode 100644 tests/system/vm/test_cgroup.py diff --git a/CHANGELOG.md b/CHANGELOG.md index 415bfdd8..b81866a3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -13,6 +13,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - VM create, start, reboot, and snapshot restore now require an exact Firecracker/Jailer release pair and run through a root-owned Jailer chroot with no direct-launch fallback. - Trusted release pairs are installed from the checksum-verified archive; each jail exposes only that VM's kernel, rootfs, volumes, runtime files, and selected snapshot. +#### Enforced VM resource envelopes +- Jailed VMs now require cgroup v2 and launch with persisted, verified CPU, memory, swap, and PID limits; `mvm vm inspect` reports requested, observed, and current cgroup state. + ### Fixed #### `mvm image inspect` diff --git a/docs/RESOURCE_MANAGEMENTS.md b/docs/RESOURCE_MANAGEMENTS.md index b14e5e71..0033a462 100644 --- a/docs/RESOURCE_MANAGEMENTS.md +++ b/docs/RESOURCE_MANAGEMENTS.md @@ -269,7 +269,7 @@ This map is the single authoritative source for all built-in defaults. Hardcoded | Category path | Description | |---------------|-------------| -| `defaults.vm` | Default vCPU count, RAM, SSH user, boot args, LSM flags, PCI, nested virt, logging, console, vsock port | +| `defaults.vm` | Default vCPU count, RAM, SSH user, boot args, LSM flags, PCI, nested virt, logging, console, vsock port, and typed cgroup policy inputs | | `defaults.network` | Default bridge name, CIDR, NAT enabled | | `defaults.image` | Import format, remote listing limit and cache TTL | | `defaults.kernel` | Default kernel version, build jobs, remote listing limit and cache TTL | @@ -282,6 +282,19 @@ This map is the single authoritative source for all built-in defaults. Hardcoded | `settings.vm` | log_lines, log_follow, max_vms, ssh_timeout_sec | | `cli` | Listing style (default: `short`) | +### `defaults.vm` cgroup policy + +These settings derive the persisted envelope when a VM is created. Existing and snapshot-restored VMs retain their persisted limits. + +| Key | Default | Description | +|-----|---------|-------------| +| `cgroup_vmm_headroom_mib` | `128` | Host-memory allowance added to guest-visible RAM for `memory.high` and `memory.max` | +| `cgroup_cpu_weight` | `100` | cgroup-v2 CPU weight; accepted range is 1–10000 | +| `cgroup_pids_max` | `256` | Maximum processes in the VM's cgroup | +| `cgroup_swap_max_bytes` | `0` | Maximum cgroup swap usage in bytes; zero disables swap | + +CPU quota is the VM's vCPU count multiplied by a fixed 100000 µs period. Raw cgroup file names and arbitrary controller values are not configuration interfaces. + ### `defaults.kernel` | Key | Default | Description | diff --git a/docs/RUNTIME.md b/docs/RUNTIME.md index 325134c7..aff2653e 100644 --- a/docs/RUNTIME.md +++ b/docs/RUNTIME.md @@ -409,9 +409,9 @@ invocations. Jailer is the only VM launch path. `mvm bin pull` streams the checksum-verified Firecracker release archive to the privileged installer, which atomically installs the exact Firecracker/Jailer pair under `/var/lib/mvmctl/binaries//`. VM launch then exposes only that VM's required resources inside `/var/lib/mvmctl/jailer/firecracker//root/` and execs the trusted Jailer binary. -Jailer does not daemonize, so the serial-console descriptors remain attached. The service derives the non-root Firecracker UID/GID from the sudo caller, translates Firecracker configuration paths into the jail, and records the actual Firecracker PID. Stop, remove, reboot, hotplug, and snapshot operations reconcile the corresponding jail mounts. +Jailer does not daemonize, so the serial-console descriptors remain attached. The service derives the non-root Firecracker UID/GID from the sudo caller, translates Firecracker configuration paths into the jail, and records the actual Firecracker PID. Stop, remove, reboot, hotplug, and snapshot operations reconcile the corresponding jail mounts and cgroup. -There is no direct-Firecracker fallback. Jailer cgroup-v2 resource tuning and network namespaces are separate follow-up work. +There is no direct-Firecracker fallback. Every launch requires cgroup v2 with the CPU, memory, and PID controllers and creates `/sys/fs/cgroup/mvmctl/firecracker/`. mvmctl persists typed CPU, memory, swap, and PID limits, passes only internally derived Jailer arguments, and verifies exact membership and values after launch. Verification failure terminates the VM. `mvm vm inspect` reports requested limits, observed limits, usage, mismatches, and enforcement status. Network namespaces remain separate follow-up work. **Key files:** diff --git a/docs/TROUBLESHOOTING.md b/docs/TROUBLESHOOTING.md index 6352cfff..5d332d01 100644 --- a/docs/TROUBLESHOOTING.md +++ b/docs/TROUBLESHOOTING.md @@ -152,25 +152,33 @@ mvm bin ls --long mvm bin pull --force ``` -**2. Kernel file is not readable by Firecracker.** Verify the kernel exists and the path is correct: +**2. cgroup v2 or required controllers are unavailable.** VM launch requires the unified cgroup-v2 hierarchy with `cpu`, `memory`, and `pids`. Refresh host information and inspect the detected limits: + +```bash +mvm host info --refresh --json +``` + +If launch reports a verification mismatch, `mvm vm inspect --json` shows the requested and observed values. mvmctl fails closed rather than launching without the envelope. + +**3. Kernel file is not readable by Firecracker.** Verify the kernel exists and the path is correct: ```bash mvm kernel ls --json | jq -r '.[].path' ls -l ``` -**3. Invalid boot arguments.** Custom boot args (set via `mvm config`) may contain typos or flags the kernel doesn't understand. Reset to defaults: +**4. Invalid boot arguments.** Custom boot args (set via `mvm config`) may contain typos or flags the kernel doesn't understand. Reset to defaults: ```bash mvm config reset defaults.vm boot_args ``` -**4. Socket path too long.** Firecracker uses Unix domain sockets which have a 108-character path limit. Long VM names or deep cache directories can exceed this. Check: +**5. Socket path too long.** Firecracker uses Unix domain sockets which have a 108-character path limit. Long VM names or deep cache directories can exceed this. Check: ```bash # See the VM's socket path ls -la ~/.cache/mvmctl/vms/*/firecracker.api.socket # If it looks very long, try a shorter VM name ``` -**5. Binary / kernel architecture mismatch.** A kernel built for `x86_64` won't boot under an `aarch64` Firecracker binary. Verify both match: +**6. Binary / kernel architecture mismatch.** A kernel built for `x86_64` won't boot under an `aarch64` Firecracker binary. Verify both match: ```bash file $(mvm kernel ls --json | jq -r '.[0].path') mvm bin ls diff --git a/docs/adr/0015-jailer-canonical-launcher.md b/docs/adr/0015-jailer-canonical-launcher.md index aec079f3..98f3f89d 100644 --- a/docs/adr/0015-jailer-canonical-launcher.md +++ b/docs/adr/0015-jailer-canonical-launcher.md @@ -21,7 +21,7 @@ Jailer will be the canonical launcher for every normal mvmctl VM lifecycle opera - The existing VM-domain spawner remains the caller-facing seam and owns jailed path translation, process tracking, socket probing, and cleanup. - Root-requiring setup is encapsulated in a privileged `mvm run` subprocess, following the loopmount precedent. Users continue to invoke `mvm` without sudo. - Executable and chroot paths trusted by Jailer are root-owned and not writable by the invoking user. The ordinary per-user mvmctl database and cache model remains unchanged. -- Typed cgroup-v2 resource enforcement will extend this canonical launch path in a separate change. Raw cgroup key/value input will not be public. +- Every launch requires cgroup v2 with the CPU, memory, and PID controllers. mvmctl persists a typed resource envelope, derives fixed Jailer arguments internally, and fails the launch if post-start membership or values differ. Raw cgroup key/value input is not public. - Jailer does not daemonize initially, preserving serial-console standard descriptors. New PID and network namespaces are deferred until their lifecycle and TAP implications are designed separately. - VM resources are exposed inside the jail through validated paths without duplicating large rootfs and volume files. Mount cleanup is idempotent across failed starts, stop, remove, and host recovery. @@ -30,7 +30,7 @@ Jailer will be the canonical launcher for every normal mvmctl VM lifecycle opera **Benefits:** - Every VM receives Firecracker's supported chroot and privilege-drop path. -- The canonical Jailer path provides the required seam for follow-up cgroup-v2 enforcement. +- The canonical Jailer path enforces one finite cgroup-v2 envelope for every VM lifecycle path. - One canonical launch path avoids a permanent direct-versus-jailed behavior matrix. - VM create, start, reboot, and snapshot restore share the same security properties. @@ -39,9 +39,9 @@ Jailer will be the canonical launcher for every normal mvmctl VM lifecycle opera - Existing host paths in Firecracker configuration must be translated into jail-visible paths while host controllers retain corresponding socket and artifact locations. - Console, vsock, snapshots, hotplug volumes, PID handling, and partial-failure cleanup require full parity testing. - Jailer adds fixed launch overhead and depends on the number of host mount points; benchmarks must track the regression budget. -- Memory cgroup limits require measured VMM headroom above guest-visible RAM. +- The initial memory maximum is guest-visible RAM plus a configurable 128 MiB VMM headroom; versioned L2 measurements must validate or revise that conservative default. - Network namespace isolation remains future work because the current TAP belongs to a host bridge. -- Release pairs are reinstalled into the trusted store by `mvm bin pull`; cgroup-v2 prerequisites are introduced separately. +- Release pairs are reinstalled into the trusted store by `mvm bin pull`; hosts without cgroup v2 and the CPU, memory, and PID controllers cannot launch VMs. ## Related Decisions diff --git a/docs/development/HOW_AGENTS_WRITE_SYSTEM_TESTS.md b/docs/development/HOW_AGENTS_WRITE_SYSTEM_TESTS.md index 73aa9a74..34bb805c 100644 --- a/docs/development/HOW_AGENTS_WRITE_SYSTEM_TESTS.md +++ b/docs/development/HOW_AGENTS_WRITE_SYSTEM_TESTS.md @@ -255,7 +255,7 @@ Every CLI command and flag that a user can invoke must be tested. This table cla | `vm create --ip` / `--mac` | **L2** | Real Firecracker spawn | | `vm create --console` | **L2** | Real Firecracker spawn | | `vm create --no-pci` | **L2** | Real Firecracker spawn | -| canonical Jailer launch and cleanup | **L2** | Real chroot, trusted pair, lifecycle, volume, and snapshot behavior | +| canonical Jailer launch, cgroup enforcement, and cleanup | **L2** | Real chroot, trusted pair, cgroup membership and values, lifecycle, volume, and snapshot behavior | | `vm create --enable-logging` / `--enable-metrics` | **L2** | Verify files on disk | | `vm create --cloud-init-mode ` | **L2** | Real Firecracker + cloud-init | | `vm create --nocloud-net-port` | **L2** | Real Firecracker + network | diff --git a/docs/improvements/IMPROVEMENTS_008.md b/docs/improvements/IMPROVEMENTS_008.md index a5bb8e04..c77abc5a 100644 --- a/docs/improvements/IMPROVEMENTS_008.md +++ b/docs/improvements/IMPROVEMENTS_008.md @@ -136,7 +136,7 @@ Use a **read-only base image** + **writable overlay** per VM. Zero copy, O(1) cr | Item | Status | |---|---| | Huge pages: add to `FirecrackerConfig` + host docs | ⚠️ Partial — hugepage detection implemented in `internal/core/host/detector.go` (reads `/sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages`), stored in `host_state` schema, but not added to `FirecrackerConfig` struct and no host docs written. | -| cgroup v2 + `kvm.nx_huge_pages=never` docs | ⚠️ Partial — cgroup v2 detection in `internal/core/host/detector.go` (checks `/sys/fs/cgroup/cgroup.controllers`), stored in `host_state` schema, but docs not written. | +| cgroup v2 + `kvm.nx_huge_pages=never` docs | ⚠️ Partial — typed per-VM cgroup-v2 enforcement and host controller detection are implemented and documented; the separate `kvm.nx_huge_pages=never` host tuning remains unimplemented. | | Kernel boot args: add safe params | ⚠️ Partial (basic args exist) | ### Phase 2: Snapshot Domain (Weeks 2-3) ✅ DONE diff --git a/docs/optimizations/next-level-optimizations.md b/docs/optimizations/next-level-optimizations.md index 30882e72..cdc6729e 100644 --- a/docs/optimizations/next-level-optimizations.md +++ b/docs/optimizations/next-level-optimizations.md @@ -132,9 +132,9 @@ These are the optimizations that can **radically change** the performance profil **Estimated impact:** High — up to 50% boot time reduction -### 3.2 cgroup v2 + `favordynmods` + `kvm.nx_huge_pages=never` ⏳ NOT IMPLEMENTED +### 3.2 cgroup v2 + `favordynmods` + `kvm.nx_huge_pages=never` ⚠️ PARTIAL -**What it is:** Three related kernel-level optimizations that dramatically affect Firecracker startup time, especially snapshot restore. +**What it is:** Three related kernel-level mechanisms that affect Firecracker startup time, especially snapshot restore. Typed cgroup-v2 resource enforcement is implemented through Jailer; `favordynmods` and `kvm.nx_huge_pages=never` performance tuning remain unimplemented. **Bottleneck addressed:** KVM + cgroup overhead (8.5ms for snapshot restore) diff --git a/docs/roadmaps/TENANT_MICROVM.md b/docs/roadmaps/TENANT_MICROVM.md index 9102819e..686722bd 100644 --- a/docs/roadmaps/TENANT_MICROVM.md +++ b/docs/roadmaps/TENANT_MICROVM.md @@ -28,8 +28,8 @@ Direct inbound ports are not required when the workload VM runs an outbound tunn | Same-network communication | Works through one bridge; intentionally unfiltered | None for the environment trust-zone model | | Cross-network routing | Host forwarding and ACCEPT policy permit it | Persist explicit service allows and default-deny other managed inter-network traffic | | VM-to-host traffic | INPUT remains open except for explicit NoCloud allow rules | Preserve required host services, then deny other traffic from managed VM networks | -| Firecracker launch | Canonical Jailer launch and exact trusted release pairing implemented; L2 execution remains pending | Add cgroup-v2 resource enforcement and complete runner verification | -| Host resource enforcement | Guest vCPU and RAM are configured; no Jailer cgroup limits are applied | Apply and verify typed cgroup-v2 limits through Jailer | +| Firecracker launch | Canonical Jailer launch, exact trusted release pairing, and cgroup-v2 enforcement implemented; L2 execution remains pending | Complete prepared-runner verification | +| Host resource enforcement | Typed CPU, memory, swap, and PID limits are persisted, applied through Jailer, verified after launch, and shown by inspect | Measure VMM headroom under L2 workloads and add I/O limits after stable backing-device resolution exists | | IP identity | Explicit IP assignment and collision-safe IPAM exist | Policies bind to network/VM IDs and resolve current addressing during reconciliation | | Persistent storage | Volumes and snapshots exist | External orchestration supplies application-consistent backup and restore procedures | | Docker workload | Guest images and kernels are customizable | Validate a versioned Docker-capable image/kernel template | @@ -47,16 +47,16 @@ Direct inbound ports are not required when the workload VM runs an outbound tunn **Exit:** create, start, stop, reboot, console, exec, cp, volumes, and snapshot restore have L2 parity under Jailer, with no direct-launch fallback. -### M2 — Enforced resource envelope +### M2 — Enforced resource envelope (implemented; L2 execution pending) - Require cgroup v2 and the necessary controllers during host readiness checks. -- Persist typed CPU, memory, swap, PID, and applicable I/O limits with each VM. +- Persist typed CPU, memory, swap, and PID limits with each VM. Defer I/O limits until backing devices can be resolved stably. - Derive Jailer cgroup arguments internally; do not expose raw cgroup files or values. -- Measure Firecracker host-memory overhead before selecting the default memory headroom. +- Use a conservative configurable 128 MiB VMM headroom initially; replace it only with versioned L2 measurements. - Verify actual cgroup membership and values after launch; fail closed on mismatch. - Surface requested, actual, and current resource state in VM inspection. -**Exit:** every normally launched VM has verified cgroup-v2 constraints and cleanup removes its cgroup. +**Exit:** implementation and hermetic coverage are complete. Prepared-runner L2 execution must still prove that every normally launched VM has verified constraints and that stop/remove clears its leaf cgroup. ### M3 — Routed service-access policies diff --git a/docs/system-test-architecture.md b/docs/system-test-architecture.md index 012d7289..d87c8583 100644 --- a/docs/system-test-architecture.md +++ b/docs/system-test-architecture.md @@ -112,7 +112,7 @@ All T1/T2 runner VMs are Firecracker microVMs created with **Firecracker v1.16** | Tier | Runs inside VM | Shared volume | Nested KVM | Runner VM kernel | What's tested | |------|---------------|---------------|------------|-----------------|---------------| | **1** | Yes — `mvm exec -- pytest ...` | Yes (mounted at `/mnt`) | Not needed | `official:7.0.11` (nftables + tuntap + kvm + btrfs) | Host-level CLI: help, config, init, cache, keys, invariants, bin, images, kernel, network, host, run | -| **2** | Yes — `mvm exec -- pytest ...` | Yes (mounted at `/mnt`) | Yes | `official:7.0.11` (nftables + tuntap + kvm + btrfs) | VM lifecycle: volume, vm_lifecycle, vm_data_persistence, ssh, console, exec, logs, full_journeys, nftables | +| **2** | Yes — `mvm exec -- pytest ...` | Yes (mounted at `/mnt`) | Yes | `official:7.0.11` (nftables + tuntap + kvm + btrfs) | VM lifecycle: volume, vm_lifecycle, vm_data_persistence, Jailer/cgroups, ssh, console, exec, logs, full_journeys, nftables | | **3** | No — runs directly on host | Host mirror | Yes (host KVM) | N/A (host-direct) | vm_fresh_env, vm_nested_isolated, vm_snapshot_load, vm_snapshot_rootfs, volume_hotplug, cp, kernel_build, env | --- @@ -440,6 +440,7 @@ tests/system/ | run | 1 | `test_run.py` | Service subprocesses | | volume | 2 | `test_volume.py` | Volume lifecycle | | vm_lifecycle | 2 | `test_vm_lifecycle.py` | VM create/start/stop/pause | +| jailer | 2 | `test_jailer.py`, `test_cgroup.py` | Canonical Jailer containment and cgroup-v2 enforcement — needs nested virt | | ssh | 2 | `test_ssh.py` | SSH into VMs | | console | 2 | `test_console.py` | Console output | | exec | 2 | `test_exec.py` | Vsock agent command execution — uses module-scoped VM, needs nested virt | @@ -597,4 +598,3 @@ Every test file in `tests/system/` MUST: --- - diff --git a/internal/core/config/constraints.go b/internal/core/config/constraints.go index a3b39865..fcd60a00 100644 --- a/internal/core/config/constraints.go +++ b/internal/core/config/constraints.go @@ -114,6 +114,32 @@ func validateMACPrefix(key string, resolve ResolveFn) error { return nil } +func validateCgroupDefaults(key string, resolve ResolveFn) error { + valueRaw, err := resolve(key) + if err != nil { + return err + } + value, err := validators.ToInt(valueRaw) + if err != nil { + return err + } + switch key { + case "cgroup_vmm_headroom_mib", "cgroup_swap_max_bytes": + if value < 0 { + return errs.New(errs.CodeConfigError, fmt.Sprintf("%s must be non-negative", key)) + } + case "cgroup_cpu_weight": + if value < 1 || value > 10000 { + return errs.New(errs.CodeConfigError, "cgroup_cpu_weight must be between 1 and 10000") + } + case "cgroup_pids_max": + if value < 1 { + return errs.New(errs.CodeConfigError, "cgroup_pids_max must be positive") + } + } + return nil +} + // NormalizeCacheType normalizes cache_type values: first letter uppercase, rest lowercase. func NormalizeCacheType(key string, value any) (any, error) { if value == nil { @@ -143,6 +169,12 @@ func RegisterBuiltinConstraints(r *ConstraintRegistry) { []string{"guest_mac_prefix"}, validateMACPrefix, ) + r.Register("defaults.vm", []string{ + "cgroup_vmm_headroom_mib", + "cgroup_cpu_weight", + "cgroup_pids_max", + "cgroup_swap_max_bytes", + }, validateCgroupDefaults) r.RegisterTransform("defaults.volume", []string{"cache_type"}, NormalizeCacheType) } diff --git a/internal/core/config/constraints_test.go b/internal/core/config/constraints_test.go index 1af1b946..3caa565c 100644 --- a/internal/core/config/constraints_test.go +++ b/internal/core/config/constraints_test.go @@ -325,6 +325,41 @@ func TestValidateMACPrefix(t *testing.T) { }) } +func TestValidateCgroupDefaults_Boundaries(t *testing.T) { + tests := map[string]struct { + key string + value any + wantErr string + }{ + "headroom_zero": {key: "cgroup_vmm_headroom_mib", value: 0}, + "headroom_negative": {key: "cgroup_vmm_headroom_mib", value: -1, wantErr: "non-negative"}, + "swap_zero": {key: "cgroup_swap_max_bytes", value: 0}, + "swap_negative": {key: "cgroup_swap_max_bytes", value: -1, wantErr: "non-negative"}, + "cpu_weight_minimum": {key: "cgroup_cpu_weight", value: 1}, + "cpu_weight_maximum": {key: "cgroup_cpu_weight", value: 10000}, + "cpu_weight_below_min": {key: "cgroup_cpu_weight", value: 0, wantErr: "between 1 and 10000"}, + "cpu_weight_above_max": {key: "cgroup_cpu_weight", value: 10001, wantErr: "between 1 and 10000"}, + "pids_minimum": {key: "cgroup_pids_max", value: 1}, + "pids_zero": {key: "cgroup_pids_max", value: 0, wantErr: "must be positive"}, + "non_numeric": {key: "cgroup_pids_max", value: "invalid", wantErr: "expected numeric value"}, + } + for name, tc := range tests { + t.Run(name, func(t *testing.T) { + r := config.NewConstraintRegistry() + config.RegisterBuiltinConstraints(r) + constraints := r.Get("defaults.vm", tc.key) + require.Len(t, constraints, 1) + err := constraints[0](tc.key, resolveMap(map[string]any{tc.key: tc.value})) + if tc.wantErr == "" { + require.NoError(t, err) + return + } + require.Error(t, err) + assert.Contains(t, err.Error(), tc.wantErr) + }) + } +} + // --- RegisterBuiltinConstraints --- // Rationale: Must register both constraints in the correct categories and keys. @@ -351,6 +386,16 @@ func TestRegisterBuiltinConstraints(t *testing.T) { assert.Len(t, constraints, 1, "guest_mac_prefix should have 1 constraint") }) + t.Run("registers_cgroup_defaults", func(t *testing.T) { + r := config.NewConstraintRegistry() + config.RegisterBuiltinConstraints(r) + for _, key := range []string{ + "cgroup_vmm_headroom_mib", "cgroup_cpu_weight", "cgroup_pids_max", "cgroup_swap_max_bytes", + } { + assert.Len(t, r.Get("defaults.vm", key), 1, "%s should have one constraint", key) + } + }) + t.Run("unregistered_key_has_no_constraints", func(t *testing.T) { r := config.NewConstraintRegistry() config.RegisterBuiltinConstraints(r) diff --git a/internal/core/host/detector.go b/internal/core/host/detector.go index c6fab6fa..be0714fd 100644 --- a/internal/core/host/detector.go +++ b/internal/core/host/detector.go @@ -326,8 +326,11 @@ func DetectLimits() *model.HostLimits { // Cgroup version cgroupVersion := 1 - if _, err := os.Stat("/sys/fs/cgroup/cgroup.controllers"); err == nil { + var cgroupControllers []string + if controllers, err := os.ReadFile(filepath.Join(infra.CgroupV2Root, "cgroup.controllers")); err == nil { cgroupVersion = 2 + cgroupControllers = strings.Fields(string(controllers)) + slices.Sort(cgroupControllers) } // Swap total @@ -348,6 +351,7 @@ func DetectLimits() *model.HostLimits { HugepageCount2MB: hugepageCount2MB, KSMDisabled: ksmDisabled, CgroupVersion: cgroupVersion, + CgroupControllers: cgroupControllers, SwapTotalMiB: swapTotalMiB, KernelMinimumMet: kernelMinimumMet, } diff --git a/internal/core/host/probe.go b/internal/core/host/probe.go index 9e9e87ce..f57db92f 100644 --- a/internal/core/host/probe.go +++ b/internal/core/host/probe.go @@ -4,6 +4,7 @@ import ( "context" "fmt" "os/exec" + "slices" "strconv" "strings" @@ -176,6 +177,27 @@ func (p *Probe) checkVMHost( Details: kernelDetails, }) + if limits.CgroupVersion != 0 { + cgroupV2Ready := limits.CgroupVersion == 2 + missingControllers := make([]string, 0, 3) + for _, controller := range []string{"cpu", "memory", "pids"} { + if !slices.Contains(limits.CgroupControllers, controller) { + missingControllers = append(missingControllers, controller) + } + } + cgroupV2Ready = cgroupV2Ready && len(missingControllers) == 0 + cgroupMessage := "cgroup v2 resource controllers are available" + cgroupDetails := "" + if !cgroupV2Ready { + cgroupMessage = "cgroup v2 resource enforcement is unavailable" + cgroupDetails = fmt.Sprintf("Required controllers: cpu, memory, pids; missing: %s", + strings.Join(missingControllers, ", ")) + } + checks = append(checks, model.ProbeCheck{ + Name: "cgroup_v2", Passed: cgroupV2Ready, Message: cgroupMessage, Details: cgroupDetails, + }) + } + // --- Nested virtualization --- nestedVirt := limits.NestedVirtAvailable nestedMsg := "Nested virtualization supported" diff --git a/internal/core/host/probe_test.go b/internal/core/host/probe_test.go index 1feb84bb..67833d4a 100644 --- a/internal/core/host/probe_test.go +++ b/internal/core/host/probe_test.go @@ -30,6 +30,8 @@ func defaultLimits() model.HostLimits { KernelMinimumMet: true, NestedVirtAvailable: true, HugepageCount2MB: 0, // no hugepages by default + CgroupVersion: 2, + CgroupControllers: []string{"cpu", "memory", "pids"}, } } @@ -293,6 +295,46 @@ func TestProbe_checkVMHost(t *testing.T) { } } +func TestProbe_CgroupV2RequiresAllResourceControllers(t *testing.T) { + tests := map[string]struct { + version int + controllers []string + wantPassed bool + wantMissing string + }{ + "all_required": { + version: 2, controllers: []string{"io", "pids", "memory", "cpu"}, wantPassed: true, + }, + "cgroup_v1": { + version: 1, controllers: []string{"cpu", "memory", "pids"}, wantPassed: false, + }, + "missing_cpu": { + version: 2, controllers: []string{"memory", "pids"}, wantPassed: false, wantMissing: "cpu", + }, + "missing_memory": { + version: 2, controllers: []string{"cpu", "pids"}, wantPassed: false, wantMissing: "memory", + }, + "missing_pids": { + version: 2, controllers: []string{"cpu", "memory"}, wantPassed: false, wantMissing: "pids", + }, + } + for name, tc := range tests { + t.Run(name, func(t *testing.T) { + limits := defaultLimits() + limits.CgroupVersion = tc.version + limits.CgroupControllers = tc.controllers + result := host.NewProbe().RunAll( + t.Context(), ptr(defaultHardware()), &limits, ptr(defaultResources()), + ) + check := findCheckByResult(t, result, "cgroup_v2") + assert.Equal(t, tc.wantPassed, check.Passed) + if tc.wantMissing != "" { + assert.Contains(t, check.Details, tc.wantMissing) + } + }) + } +} + // --- checkSystemResources --- // Rationale: Must warn when swap is low relative to RAM (if RAM > 1024 MiB), // detect cloud-localds availability, and report hugepage configuration. diff --git a/internal/core/vm/controller.go b/internal/core/vm/controller.go index 06c47123..eaad2866 100644 --- a/internal/core/vm/controller.go +++ b/internal/core/vm/controller.go @@ -52,7 +52,10 @@ func (c *Controller) Stop(ctx context.Context, force bool) error { // Errors must propagate to caller — NOT absorbed. if c.vm.Status != model.VMStatusRunning && c.vm.Status != model.VMStatusStarting { if pid > 0 && system.IsProcessAlive(pid, c.vm.ProcessStartTime) { - system.KillProcess(pid) + system.GracefulShutdown(system.ShutdownConfig{ + Pid: pid, IsChild: false, GracefulTimeout: 100 * time.Millisecond, + KillTimeout: 100 * time.Millisecond, ExpectedStartTime: c.vm.ProcessStartTime, + }) if err := c.repo.UpdateStatus(ctx, c.vm.ID, model.VMStatusStopped); err != nil { return err } diff --git a/internal/core/vm/firecracker.go b/internal/core/vm/firecracker.go index dfecb01d..48674451 100644 --- a/internal/core/vm/firecracker.go +++ b/internal/core/vm/firecracker.go @@ -198,6 +198,10 @@ func (s *FirecrackerSpawner) Spawn(ctx context.Context) (retErr error) { } s.PID = &pid s.ProcessStartTime = system.GetProcessStartTime(pid) + if err := VerifyCgroup(ctx, s.vmID(), pid, s.config.CgroupLimits); err != nil { + slog.Error("failed to verify VM cgroup enforcement", "vm_id", s.vmID(), "pid", pid, "error", err) + return err + } if err := infra.WritePIDFile(s.pidPath, pid); err != nil { slog.Warn("Failed to write PID file", "path", s.pidPath, "error", err) diff --git a/internal/core/vm/jailer.go b/internal/core/vm/jailer.go index 94090abf..2f58259d 100644 --- a/internal/core/vm/jailer.go +++ b/internal/core/vm/jailer.go @@ -6,10 +6,13 @@ import ( "fmt" "os" "path/filepath" + "strconv" + "strings" "mvmctl/internal/infra" "mvmctl/internal/lib/model" jailersvc "mvmctl/internal/service/jailer" + "mvmctl/pkg/errs" ) func (s *FirecrackerSpawner) writeJailerManifest() error { @@ -23,6 +26,10 @@ func (s *FirecrackerSpawner) writeJailerManifest() error { if s.config.CloudInitISOPath != nil { isoPath = *s.config.CloudInitISOPath } + var cgroupLimits *model.VMCgroupLimits + if s.config.CgroupLimits.CPUQuotaMicros > 0 { + cgroupLimits = &s.config.CgroupLimits + } manifest := jailersvc.LaunchManifest{ VMID: s.vmID(), VMDir: s.config.VMDir, Version: s.config.BinaryVersion, ConfigPath: s.config.ConfigPath, PIDPath: s.config.PIDPath, @@ -30,6 +37,7 @@ func (s *FirecrackerSpawner) writeJailerManifest() error { ISOPath: isoPath, SnapshotDir: s.config.SnapshotDir, Volumes: volumes, APISocket: s.config.APISocketPath, PCIEnabled: s.config.PCIEnabled, SnapshotMode: s.config.SnapshotMode, + CgroupLimits: cgroupLimits, } data, err := json.Marshal(manifest) if err != nil { @@ -105,3 +113,219 @@ func exposeVolume(ctx context.Context, vmID string, drive model.DriveConfig) err func removeVolume(ctx context.Context, vmID, driveID string) error { return jailersvc.RemoveVolume(ctx, vmID, driveID) } + +// CgroupPath returns the fixed cgroup-v2 path for one canonical Jailer launch. +func CgroupPath(vmID string) string { + return filepath.Join(infra.CgroupV2Root, infra.JailerCgroupParent, infra.JailerCgroupExecutable, vmID) +} + +// InspectCgroup reads typed enforcement and usage state for one VM. +func InspectCgroup( + ctx context.Context, + vmID string, + pid int, + requested model.VMCgroupLimits, +) (*model.VMCgroupState, error) { + if err := ctx.Err(); err != nil { + return nil, err + } + path := CgroupPath(vmID) + state := &model.VMCgroupState{ + Path: path, Requested: requested, Status: model.VMCgroupEnforcementInactive, + } + if pid <= 0 { + return state, nil + } + if _, err := os.Stat(path); err != nil { + return nil, errs.WrapMsg(errs.CodeVMCgroupVerificationFailed, "VM cgroup is unavailable", err, + errs.WithEntity(vmID)) + } + actual, err := readCgroupLimits(path) + if err != nil { + return nil, errs.WrapMsg(errs.CodeVMCgroupVerificationFailed, "failed to read VM cgroup limits", err, + errs.WithEntity(vmID)) + } + actual.PolicyVersion = requested.PolicyVersion + usage, err := readCgroupUsage(path) + if err != nil { + return nil, errs.WrapMsg(errs.CodeVMCgroupVerificationFailed, "failed to read VM cgroup usage", err, + errs.WithEntity(vmID)) + } + state.Actual = &actual + state.Usage = &usage + state.Mismatches = cgroupLimitMismatches(requested, actual) + inExpectedCgroup, err := processInCgroup(pid, vmID) + if err != nil { + return nil, errs.WrapMsg(errs.CodeVMCgroupVerificationFailed, "failed to verify VM cgroup membership", err, + errs.WithEntity(vmID)) + } + if !inExpectedCgroup { + state.Mismatches = append(state.Mismatches, "process membership") + } + inProcessList, err := cgroupContainsProcess(path, pid) + if err != nil { + return nil, errs.WrapMsg(errs.CodeVMCgroupVerificationFailed, "failed to read VM cgroup process list", err, + errs.WithEntity(vmID)) + } + if !inProcessList { + state.Mismatches = append(state.Mismatches, "cgroup process list") + } + if len(state.Mismatches) == 0 { + state.Status = model.VMCgroupEnforcementEnforced + } else { + state.Status = model.VMCgroupEnforcementMismatch + } + return state, nil +} + +// VerifyCgroup requires exact cgroup membership and controller values. +func VerifyCgroup(ctx context.Context, vmID string, pid int, requested model.VMCgroupLimits) error { + state, err := InspectCgroup(ctx, vmID, pid, requested) + if err != nil { + return err + } + if state.Status != model.VMCgroupEnforcementEnforced { + return errs.New(errs.CodeVMCgroupVerificationFailed, + fmt.Sprintf("VM cgroup enforcement verification failed: %s", strings.Join(state.Mismatches, ", ")), + errs.WithEntity(vmID)) + } + return nil +} + +func readCgroupLimits(path string) (model.VMCgroupLimits, error) { + cpuMax, err := os.ReadFile(filepath.Join(path, "cpu.max")) + if err != nil { + return model.VMCgroupLimits{}, err + } + cpuFields := strings.Fields(string(cpuMax)) + if len(cpuFields) != 2 || cpuFields[0] == "max" { + return model.VMCgroupLimits{}, fmt.Errorf("invalid finite CPU maximum") + } + quota, err := strconv.ParseInt(cpuFields[0], 10, 64) + if err != nil { + return model.VMCgroupLimits{}, err + } + period, err := strconv.ParseInt(cpuFields[1], 10, 64) + if err != nil { + return model.VMCgroupLimits{}, err + } + weight, err := readCgroupInt(path, "cpu.weight") + if err != nil { + return model.VMCgroupLimits{}, err + } + memoryHigh, err := readCgroupInt(path, "memory.high") + if err != nil { + return model.VMCgroupLimits{}, err + } + memoryMax, err := readCgroupInt(path, "memory.max") + if err != nil { + return model.VMCgroupLimits{}, err + } + swapMax, err := readCgroupInt(path, "memory.swap.max") + if err != nil { + return model.VMCgroupLimits{}, err + } + pidsMax, err := readCgroupInt(path, "pids.max") + if err != nil { + return model.VMCgroupLimits{}, err + } + return model.VMCgroupLimits{ + CPUQuotaMicros: quota, CPUPeriodMicros: period, CPUWeight: weight, + MemoryHighBytes: memoryHigh, MemoryMaxBytes: memoryMax, + SwapMaxBytes: swapMax, PIDsMax: pidsMax, + }, nil +} + +func readCgroupUsage(path string) (model.VMCgroupUsage, error) { + cpuStat, err := os.ReadFile(filepath.Join(path, "cpu.stat")) + if err != nil { + return model.VMCgroupUsage{}, err + } + var cpuUsage int64 + for line := range strings.SplitSeq(string(cpuStat), "\n") { + fields := strings.Fields(line) + if len(fields) == 2 && fields[0] == "usage_usec" { + cpuUsage, err = strconv.ParseInt(fields[1], 10, 64) + if err != nil { + return model.VMCgroupUsage{}, err + } + break + } + } + memory, err := readCgroupInt(path, "memory.current") + if err != nil { + return model.VMCgroupUsage{}, err + } + swap, err := readCgroupInt(path, "memory.swap.current") + if err != nil { + return model.VMCgroupUsage{}, err + } + pids, err := readCgroupInt(path, "pids.current") + if err != nil { + return model.VMCgroupUsage{}, err + } + return model.VMCgroupUsage{CPUUsageMicros: cpuUsage, MemoryBytes: memory, SwapBytes: swap, PIDsCurrent: pids}, nil +} + +func readCgroupInt(path, name string) (int64, error) { + data, err := os.ReadFile(filepath.Join(path, name)) + if err != nil { + return 0, err + } + value := strings.TrimSpace(string(data)) + if value == "max" { + return 0, fmt.Errorf("%s is unlimited", name) + } + return strconv.ParseInt(value, 10, 64) +} + +func processInCgroup(pid int, vmID string) (bool, error) { + data, err := os.ReadFile(filepath.Join("/proc", strconv.Itoa(pid), "cgroup")) + if err != nil { + return false, err + } + expected := "/" + filepath.Join(infra.JailerCgroupParent, infra.JailerCgroupExecutable, vmID) + for line := range strings.SplitSeq(string(data), "\n") { + if strings.HasPrefix(line, "0::") && strings.TrimPrefix(line, "0::") == expected { + return true, nil + } + } + return false, nil +} + +func cgroupContainsProcess(path string, pid int) (bool, error) { + data, err := os.ReadFile(filepath.Join(path, "cgroup.procs")) + if err != nil { + return false, err + } + expected := strconv.Itoa(pid) + for process := range strings.FieldsSeq(string(data)) { + if process == expected { + return true, nil + } + } + return false, nil +} + +func cgroupLimitMismatches(requested, actual model.VMCgroupLimits) []string { + var mismatches []string + if requested.CPUQuotaMicros != actual.CPUQuotaMicros || requested.CPUPeriodMicros != actual.CPUPeriodMicros { + mismatches = append(mismatches, "CPU quota") + } + if requested.CPUWeight != actual.CPUWeight { + mismatches = append(mismatches, "CPU weight") + } + if requested.MemoryHighBytes != actual.MemoryHighBytes { + mismatches = append(mismatches, "memory high") + } + if requested.MemoryMaxBytes != actual.MemoryMaxBytes { + mismatches = append(mismatches, "memory maximum") + } + if requested.SwapMaxBytes != actual.SwapMaxBytes { + mismatches = append(mismatches, "swap maximum") + } + if requested.PIDsMax != actual.PIDsMax { + mismatches = append(mismatches, "PID maximum") + } + return mismatches +} diff --git a/internal/core/vm/jailer_test.go b/internal/core/vm/jailer_test.go index 81fbbefe..49a04d3d 100644 --- a/internal/core/vm/jailer_test.go +++ b/internal/core/vm/jailer_test.go @@ -38,6 +38,7 @@ func TestFirecrackerSpawner_WriteJailerManifestSelectsOnlyVMResources(t *testing DriveID: testDriveID, HostPath: config.ExtraDrives[0].PathOnHost, ReadOnly: true, }}, APISocket: config.APISocketPath, PCIEnabled: true, SnapshotMode: true, + CgroupLimits: &config.CgroupLimits, } if diff := cmp.Diff(want, got); diff != "" { t.Errorf("LaunchManifest mismatch (-want +got):\n%s", diff) @@ -48,7 +49,7 @@ func TestFirecrackerSpawner_WriteJailerManifestSelectsOnlyVMResources(t *testing "vm_id": true, "vm_dir": true, "version": true, "config_path": true, "pid_path": true, "kernel_path": true, "rootfs_path": true, "iso_path": true, "snapshot_dir": true, "volumes": true, "api_socket": true, - "pci_enabled": true, "snapshot_mode": true, + "pci_enabled": true, "snapshot_mode": true, "cgroup_limits": true, } gotKeys := make(map[string]bool, len(raw)) for key := range raw { @@ -59,6 +60,101 @@ func TestFirecrackerSpawner_WriteJailerManifestSelectsOnlyVMResources(t *testing } } +func TestFirecrackerSpawner_WriteJailerManifestOmitsAbsentCgroupLimits(t *testing.T) { + vmDir := t.TempDir() + config := jailerTestConfig(vmDir) + config.CgroupLimits = model.VMCgroupLimits{} + require.NoError(t, NewFirecrackerSpawner(config).writeJailerManifest()) + + data, err := os.ReadFile(filepath.Join(vmDir, infra.JailerManifestFilename)) + require.NoError(t, err) + var manifest jailer.LaunchManifest + require.NoError(t, json.Unmarshal(data, &manifest)) + assert.Nil(t, manifest.CgroupLimits) +} + +func TestCgroupStateParsingAndExactMismatchReporting(t *testing.T) { + path := t.TempDir() + files := map[string]string{ + "cpu.max": "200000 100000\n", + "cpu.weight": "100\n", + "memory.high": "671088640\n", + "memory.max": "671088640\n", + "memory.swap.max": "0\n", + "pids.max": "256\n", + "cpu.stat": "usage_usec 12345\nuser_usec 10000\n", + "memory.current": "4096\n", + "memory.swap.current": "0\n", + "pids.current": "3\n", + "cgroup.procs": "12\n123\n", + } + for name, content := range files { + require.NoError(t, os.WriteFile(filepath.Join(path, name), []byte(content), 0600)) + } + + limits, err := readCgroupLimits(path) + require.NoError(t, err) + assert.Equal(t, int64(200000), limits.CPUQuotaMicros) + assert.Equal(t, int64(671088640), limits.MemoryMaxBytes) + assert.Zero(t, limits.SwapMaxBytes) + usage, err := readCgroupUsage(path) + require.NoError(t, err) + assert.Equal(t, model.VMCgroupUsage{ + CPUUsageMicros: 12345, MemoryBytes: 4096, SwapBytes: 0, PIDsCurrent: 3, + }, usage) + contains, err := cgroupContainsProcess(path, 12) + require.NoError(t, err) + assert.True(t, contains) + contains, err = cgroupContainsProcess(path, 2) + require.NoError(t, err) + assert.False(t, contains, "PID matching must compare complete fields") + + requested := limits + requested.PolicyVersion = 1 + actual := limits + actual.CPUQuotaMicros++ + actual.CPUWeight++ + actual.MemoryHighBytes++ + actual.MemoryMaxBytes++ + actual.SwapMaxBytes++ + actual.PIDsMax++ + assert.Equal(t, []string{ + "CPU quota", "CPU weight", "memory high", "memory maximum", "swap maximum", "PID maximum", + }, cgroupLimitMismatches(requested, actual)) +} + +func TestCgroupStateRejectsUnlimitedControllerValues(t *testing.T) { + path := t.TempDir() + require.NoError(t, os.WriteFile(filepath.Join(path, "cpu.max"), []byte("max 100000\n"), 0600)) + _, err := readCgroupLimits(path) + require.ErrorContains(t, err, "finite CPU maximum") + + require.NoError(t, os.WriteFile(filepath.Join(path, "value"), []byte("max\n"), 0600)) + _, err = readCgroupInt(path, "value") + require.ErrorContains(t, err, "unlimited") +} + +func TestInspectCgroupInactiveAndUnavailable(t *testing.T) { + requested := model.NewVMCgroupLimits(1, 512, model.VMCgroupPolicy{ + VMMHeadroomMiB: 128, CPUWeight: 100, PIDsMax: 256, SwapMaxBytes: 0, + }) + inactive, err := InspectCgroup(t.Context(), testDriveID[:32], 0, requested) + require.NoError(t, err) + assert.Equal(t, model.VMCgroupEnforcementInactive, inactive.Status) + assert.Equal(t, requested, inactive.Requested) + assert.Nil(t, inactive.Actual) + + _, err = InspectCgroup(t.Context(), "ffffffffffffffffffffffffffffffff", os.Getpid(), requested) + require.Error(t, err) + assert.Contains(t, err.Error(), "VM cgroup is unavailable") +} + +func TestProcessCgroupMembershipRequiresExactUnifiedPath(t *testing.T) { + inExpected, err := processInCgroup(os.Getpid(), "definitely-not-the-current-cgroup") + require.NoError(t, err) + assert.False(t, inExpected) +} + func TestFirecrackerSpawner_WriteToFileUsesJailedPathsWithoutMutatingHostConfig(t *testing.T) { vmDir := t.TempDir() config := jailerTestConfig(vmDir) @@ -127,6 +223,9 @@ func jailerTestConfig(vmDir string) *model.FirecrackerConfig { JailerPath: "/var/lib/mvmctl/binaries/1.16.0/jailer", BinaryVersion: "1.16.0", KernelPath: filepath.Join(filepath.Dir(vmDir), "kernels", "vmlinux"), VCPUCount: 2, MemSizeMiB: 512, GuestIP: "10.0.0.2", GuestMAC: "02:00:00:00:00:02", + CgroupLimits: model.NewVMCgroupLimits(2, 512, model.VMCgroupPolicy{ + VMMHeadroomMiB: 128, CPUWeight: 100, PIDsMax: 256, SwapMaxBytes: 0, + }), TapName: "tap-test", NetworkGateway: "10.0.0.1", NetworkNetmask: "255.255.255.0", PCIEnabled: true, ImageFSUUID: "11111111-2222-3333-4444-555555555555", ImageFSType: "ext4", EnableLogging: true, EnableMetrics: true, LogLevel: "Info", diff --git a/internal/core/vm/sqlite.go b/internal/core/vm/sqlite.go index 24feadae..33d1e49e 100644 --- a/internal/core/vm/sqlite.go +++ b/internal/core/vm/sqlite.go @@ -297,14 +297,14 @@ func (r *sqliteRepo) Upsert(ctx context.Context, vm *model.VMItem) error { image_id, kernel_id, binary_id, jailer_binary_id, api_socket_path, relay_socket_path, config_path, cloud_init_mode, nocloud_net_port, nocloud_net_pid, relay_pid, - exit_code, vcpu_count, mem_size_mib, disk_size_mib, + exit_code, vcpu_count, mem_size_mib, cgroup_limits, disk_size_mib, rootfs_path, rootfs_suffix, pci_enabled, nested_virt, remote_exec, enable_logging, enable_metrics, enable_console, ssh_keys, ssh_user, created_at, updated_at, log_path, serial_output_path, lsm_flags, boot_args, volume_ids, cpu_config - ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) ON CONFLICT(id) DO UPDATE SET name = excluded.name, status = excluded.status, @@ -328,6 +328,7 @@ func (r *sqliteRepo) Upsert(ctx context.Context, vm *model.VMItem) error { exit_code = excluded.exit_code, vcpu_count = excluded.vcpu_count, mem_size_mib = excluded.mem_size_mib, + cgroup_limits = excluded.cgroup_limits, disk_size_mib = excluded.disk_size_mib, rootfs_path = excluded.rootfs_path, rootfs_suffix = excluded.rootfs_suffix, @@ -370,6 +371,7 @@ func (r *sqliteRepo) Upsert(ctx context.Context, vm *model.VMItem) error { vm.ExitCode, vm.VCPUCount, vm.MemSizeMiB, + vm.CgroupLimits, vm.DiskSizeMiB, vm.RootfsPath, vm.RootfsSuffix, diff --git a/internal/core/vm/sqlite_test.go b/internal/core/vm/sqlite_test.go index 3e460288..d77e894f 100644 --- a/internal/core/vm/sqlite_test.go +++ b/internal/core/vm/sqlite_test.go @@ -24,6 +24,9 @@ func TestRepository_PersistsJailerBinaryID(t *testing.T) { ImageID: "image-id", KernelID: "kernel-id", BinaryID: "firecracker-id", JailerBinaryID: "jailer-id", APISocketPath: "/tmp/api.socket", ConfigPath: "/tmp/config.json", CloudInitMode: "off", VCPUCount: 2, MemSizeMiB: 512, DiskSizeMiB: 1024, RootfsPath: "/tmp/rootfs", RootfsSuffix: ".ext4", + CgroupLimits: model.NewVMCgroupLimits(2, 512, model.VMCgroupPolicy{ + VMMHeadroomMiB: 128, CPUWeight: 100, PIDsMax: 256, SwapMaxBytes: 0, + }), CreatedAt: "2026-08-09T00:00:00Z", UpdatedAt: "2026-08-09T00:00:00Z", } @@ -33,6 +36,16 @@ func TestRepository_PersistsJailerBinaryID(t *testing.T) { require.NotNil(t, got) assert.Equal(t, "firecracker-id", got.BinaryID) assert.Equal(t, "jailer-id", got.JailerBinaryID) + assert.Equal(t, item.CgroupLimits, got.CgroupLimits) + + item.CgroupLimits = model.NewVMCgroupLimits(4, 1024, model.VMCgroupPolicy{ + VMMHeadroomMiB: 64, CPUWeight: 500, PIDsMax: 512, SwapMaxBytes: 4096, + }) + require.NoError(t, repo.Upsert(ctx, item)) + updated, err := repo.Get(ctx, item.ID) + require.NoError(t, err) + require.NotNil(t, updated) + assert.Equal(t, item.CgroupLimits, updated.CgroupLimits) byJailer, err := repo.FindByBinaryID(ctx, "jailer-id") require.NoError(t, err) diff --git a/internal/infra/constants.go b/internal/infra/constants.go index 1d0cf1d2..c0feaf08 100644 --- a/internal/infra/constants.go +++ b/internal/infra/constants.go @@ -52,6 +52,15 @@ const TrustedBinaryRoot = "/var/lib/mvmctl/binaries" // JailerChrootBase is the fixed root-owned base directory for per-VM jails. const JailerChrootBase = "/var/lib/mvmctl/jailer" +// JailerCgroupParent is the fixed cgroup-v2 subtree used by canonical Jailer launches. +const JailerCgroupParent = "mvmctl" + +// JailerCgroupExecutable is Jailer's fixed executable-name component in the cgroup path. +const JailerCgroupExecutable = "firecracker" + +// CgroupV2Root is the unified Linux cgroup-v2 mount point required by VM launches. +const CgroupV2Root = "/sys/fs/cgroup" + // JailerManifestFilename is the fixed per-VM launch manifest name. const JailerManifestFilename = "jailer-launch.json" @@ -74,27 +83,31 @@ var OverridableDefaults = map[string]map[string]any{ "ssh_timeout_sec": 10, }, "defaults.vm": { - "vcpu_count": 1, - "mem_size_mib": 512, - "ssh_user": "root", - "user_password": "password", - "dns_server": "1.1.1.1", - "root_uid": 0, - "root_gid": 0, - "user_uid": 1000, - "user_gid": 1000, - "pci_enabled": true, - "nested_virt": false, - "enable_logging": true, - "enable_metrics": false, - "enable_console": false, - "lsm_flags": "landlock,lockdown,yama,integrity,selinux,bpf", - "boot_args": "console=ttyS0 reboot=k panic=1 net.ifnames=0 rw rootwait quiet loglevel=3 no_timer_check clocksource=kvm-clock systemd.show_status=false", - "guest_mac_prefix": "02:FC", - "vsock_port": 1024, - "allow_remote_exec": false, - "vsock_user": "root", - "vsock_probe_timeout": 5, + "vcpu_count": 1, + "mem_size_mib": 512, + "ssh_user": "root", + "user_password": "password", + "dns_server": "1.1.1.1", + "root_uid": 0, + "root_gid": 0, + "user_uid": 1000, + "user_gid": 1000, + "pci_enabled": true, + "nested_virt": false, + "enable_logging": true, + "enable_metrics": false, + "enable_console": false, + "lsm_flags": "landlock,lockdown,yama,integrity,selinux,bpf", + "boot_args": "console=ttyS0 reboot=k panic=1 net.ifnames=0 rw rootwait quiet loglevel=3 no_timer_check clocksource=kvm-clock systemd.show_status=false", + "guest_mac_prefix": "02:FC", + "vsock_port": 1024, + "allow_remote_exec": false, + "vsock_user": "root", + "vsock_probe_timeout": 5, + "cgroup_vmm_headroom_mib": 128, + "cgroup_cpu_weight": 100, + "cgroup_pids_max": 256, + "cgroup_swap_max_bytes": 0, }, "defaults.network": { "name": "net", diff --git a/internal/lib/db/migrations/003_vm_cgroup_limits.sql b/internal/lib/db/migrations/003_vm_cgroup_limits.sql new file mode 100644 index 00000000..b8b36682 --- /dev/null +++ b/internal/lib/db/migrations/003_vm_cgroup_limits.sql @@ -0,0 +1,8 @@ +-- Migration: 003_vm_cgroup_limits +-- Version: 3 +-- Description: Persist typed cgroup-v2 limits for every VM. + +ALTER TABLE vm_instances ADD COLUMN cgroup_limits TEXT NOT NULL DEFAULT + '{"policy_version":0,"cpu_quota_micros":100000,"cpu_period_micros":100000,"cpu_weight":100,"memory_high_bytes":671088640,"memory_max_bytes":671088640,"swap_max_bytes":0,"pids_max":256}'; + +PRAGMA user_version = 3; diff --git a/internal/lib/db/migrations_jailer_test.go b/internal/lib/db/migrations_jailer_test.go index 8d82f062..08a8a14e 100644 --- a/internal/lib/db/migrations_jailer_test.go +++ b/internal/lib/db/migrations_jailer_test.go @@ -46,7 +46,7 @@ func TestMigration002_BackfillsJailerBinaryID(t *testing.T) { applied, err := handle.RunMigrationsCtx(ctx) require.NoError(t, err) - assert.Equal(t, 1, applied) + assert.Equal(t, 2, applied) rows := map[string]string{} dbRows, err := database.QueryContext(ctx, "SELECT id, jailer_binary_id FROM vm_instances ORDER BY id") @@ -67,9 +67,47 @@ func TestMigration002_BackfillsJailerBinaryID(t *testing.T) { var schemaVersion int require.NoError(t, database.GetContext(ctx, &schemaVersion, "PRAGMA user_version")) - assert.Equal(t, 2, schemaVersion) + assert.Equal(t, 3, schemaVersion) var indexCount int require.NoError(t, database.GetContext(ctx, &indexCount, "SELECT COUNT(*) FROM sqlite_master WHERE type = 'index' AND name = 'idx_vm_instances_jailer_binary'")) assert.Equal(t, 1, indexCount) } + +func TestMigration003_AddsTypedVMCgroupLimitsDefault(t *testing.T) { + ctx := context.Background() + handle := db.New(filepath.Join(t.TempDir(), "migration-003.db")) + t.Cleanup(func() { require.NoError(t, handle.Close()) }) + database := handle.DB() + _, err := database.ExecContext(ctx, ` + CREATE TABLE vm_instances ( + id TEXT PRIMARY KEY, + vcpu_count INTEGER NOT NULL, + mem_size_mib INTEGER NOT NULL + ); + INSERT INTO vm_instances (id, vcpu_count, mem_size_mib) VALUES ('legacy-vm', 1, 512); + PRAGMA user_version = 2; + `) + require.NoError(t, err) + + applied, err := handle.RunMigrationsCtx(ctx) + require.NoError(t, err) + assert.Equal(t, 1, applied) + + var encoded string + require.NoError(t, database.GetContext(ctx, &encoded, + "SELECT cgroup_limits FROM vm_instances WHERE id = 'legacy-vm'")) + assert.JSONEq(t, `{ + "policy_version": 0, + "cpu_quota_micros": 100000, + "cpu_period_micros": 100000, + "cpu_weight": 100, + "memory_high_bytes": 671088640, + "memory_max_bytes": 671088640, + "swap_max_bytes": 0, + "pids_max": 256 + }`, encoded) + var schemaVersion int + require.NoError(t, database.GetContext(ctx, &schemaVersion, "PRAGMA user_version")) + assert.Equal(t, 3, schemaVersion) +} diff --git a/internal/lib/model/firecracker.go b/internal/lib/model/firecracker.go index a98e7dea..b612d802 100644 --- a/internal/lib/model/firecracker.go +++ b/internal/lib/model/firecracker.go @@ -6,6 +6,93 @@ import ( "fmt" ) +const cgroupCPUPeriodMicros int64 = 100000 + +// VMCgroupPolicy contains configurable inputs used to derive a VM resource envelope. +type VMCgroupPolicy struct { + VMMHeadroomMiB int64 `json:"vmm_headroom_mib" db:"vmm_headroom_mib"` + CPUWeight int64 `json:"cpu_weight" db:"cpu_weight"` + PIDsMax int64 `json:"pids_max" db:"pids_max"` + SwapMaxBytes int64 `json:"swap_max_bytes" db:"swap_max_bytes"` +} + +// VMCgroupLimits is the persisted, typed cgroup-v2 resource intent for one VM. +type VMCgroupLimits struct { + PolicyVersion int `json:"policy_version" db:"policy_version"` + CPUQuotaMicros int64 `json:"cpu_quota_micros" db:"cpu_quota_micros"` + CPUPeriodMicros int64 `json:"cpu_period_micros" db:"cpu_period_micros"` + CPUWeight int64 `json:"cpu_weight" db:"cpu_weight"` + MemoryHighBytes int64 `json:"memory_high_bytes" db:"memory_high_bytes"` + MemoryMaxBytes int64 `json:"memory_max_bytes" db:"memory_max_bytes"` + SwapMaxBytes int64 `json:"swap_max_bytes" db:"swap_max_bytes"` + PIDsMax int64 `json:"pids_max" db:"pids_max"` +} + +// NewVMCgroupLimits derives one deterministic envelope from guest machine settings and policy. +func NewVMCgroupLimits(vcpuCount, memoryMiB int, policy VMCgroupPolicy) VMCgroupLimits { + memoryMaxBytes := (int64(memoryMiB) + policy.VMMHeadroomMiB) * 1024 * 1024 + return VMCgroupLimits{ + PolicyVersion: 1, + CPUQuotaMicros: int64(vcpuCount) * cgroupCPUPeriodMicros, + CPUPeriodMicros: cgroupCPUPeriodMicros, + CPUWeight: policy.CPUWeight, + MemoryHighBytes: memoryMaxBytes, + MemoryMaxBytes: memoryMaxBytes, + SwapMaxBytes: policy.SwapMaxBytes, + PIDsMax: policy.PIDsMax, + } +} + +// Scan implements sql.Scanner for reading persisted cgroup limits. +func (c *VMCgroupLimits) Scan(src any) error { + if src == nil { + return nil + } + var value []byte + switch v := src.(type) { + case []byte: + value = v + case string: + value = []byte(v) + default: + return fmt.Errorf("model.VMCgroupLimits: unsupported scan type %T", src) + } + return json.Unmarshal(value, c) +} + +// Value implements driver.Valuer for persisting cgroup limits. +func (c VMCgroupLimits) Value() (driver.Value, error) { + return json.Marshal(c) +} + +// VMCgroupUsage contains current cgroup-v2 accounting values. +type VMCgroupUsage struct { + CPUUsageMicros int64 `json:"cpu_usage_micros" db:"cpu_usage_micros"` + MemoryBytes int64 `json:"memory_bytes" db:"memory_bytes"` + SwapBytes int64 `json:"swap_bytes" db:"swap_bytes"` + PIDsCurrent int64 `json:"pids_current" db:"pids_current"` +} + +// VMCgroupEnforcementStatus describes observed enforcement for a VM resource envelope. +type VMCgroupEnforcementStatus string + +const ( + VMCgroupEnforcementInactive VMCgroupEnforcementStatus = "inactive" + VMCgroupEnforcementEnforced VMCgroupEnforcementStatus = "enforced" + VMCgroupEnforcementMismatch VMCgroupEnforcementStatus = "mismatch" + VMCgroupEnforcementUnavailable VMCgroupEnforcementStatus = "unavailable" +) + +// VMCgroupState contains observed cgroup-v2 enforcement state. +type VMCgroupState struct { + Path string `json:"path" db:"-"` + Requested VMCgroupLimits `json:"requested" db:"-"` + Actual *VMCgroupLimits `json:"actual,omitempty" db:"-"` + Usage *VMCgroupUsage `json:"usage,omitempty" db:"-"` + Status VMCgroupEnforcementStatus `json:"status" db:"-"` + Mismatches []string `json:"mismatches,omitempty" db:"-"` +} + // --- CpuConfig --- // CpuConfig defines CPU template configuration for Firecracker. @@ -44,18 +131,19 @@ func (c CpuConfig) Value() (driver.Value, error) { // SnapshotExtraConfig preserves the source VM's Firecracker boot configuration // at snapshot time. Stored as JSON TEXT in the snapshots.extra_config column. type SnapshotExtraConfig struct { - BootArgs string `json:"boot_args,omitempty"` - LSMFlags string `json:"lsm_flags,omitempty"` - PCIEnabled bool `json:"pci_enabled,omitempty"` - NestedVirt bool `json:"nested_virt,omitempty"` - Console bool `json:"console,omitempty"` - EnableLogging bool `json:"enable_logging,omitempty"` - EnableMetrics bool `json:"enable_metrics,omitempty"` - LogLevel string `json:"log_level,omitempty"` - CPUConfig *CpuConfig `json:"cpu_config,omitempty"` - VsockPort int `json:"vsock_port,omitempty"` - VsockCID int `json:"vsock_cid,omitempty"` - VsockToken string `json:"vsock_token,omitempty"` + BootArgs string `json:"boot_args,omitempty"` + LSMFlags string `json:"lsm_flags,omitempty"` + PCIEnabled bool `json:"pci_enabled,omitempty"` + NestedVirt bool `json:"nested_virt,omitempty"` + Console bool `json:"console,omitempty"` + EnableLogging bool `json:"enable_logging,omitempty"` + EnableMetrics bool `json:"enable_metrics,omitempty"` + LogLevel string `json:"log_level,omitempty"` + CPUConfig *CpuConfig `json:"cpu_config,omitempty"` + VsockPort int `json:"vsock_port,omitempty"` + VsockCID int `json:"vsock_cid,omitempty"` + VsockToken string `json:"vsock_token,omitempty"` + CgroupLimits VMCgroupLimits `json:"cgroup_limits"` } // Scan implements sql.Scanner for reading JSON TEXT into SnapshotExtraConfig. @@ -238,8 +326,9 @@ type FirecrackerConfig struct { KernelPath string `json:"kernel_path"` // Machine - VCPUCount int `json:"vcpu_count"` - MemSizeMiB int `json:"mem_size_mib"` + VCPUCount int `json:"vcpu_count"` + MemSizeMiB int `json:"mem_size_mib"` + CgroupLimits VMCgroupLimits `json:"cgroup_limits"` // Network GuestIP string `json:"guest_ip"` diff --git a/internal/lib/model/firecracker_test.go b/internal/lib/model/firecracker_test.go new file mode 100644 index 00000000..d53750fb --- /dev/null +++ b/internal/lib/model/firecracker_test.go @@ -0,0 +1,70 @@ +package model_test + +import ( + "database/sql/driver" + "testing" + + "github.com/google/go-cmp/cmp" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "mvmctl/internal/lib/model" +) + +func TestNewVMCgroupLimits_DerivesDeterministicEnvelope(t *testing.T) { + got := model.NewVMCgroupLimits(2, 512, model.VMCgroupPolicy{ + VMMHeadroomMiB: 128, + CPUWeight: 100, + PIDsMax: 256, + SwapMaxBytes: 0, + }) + want := model.VMCgroupLimits{ + PolicyVersion: 1, + CPUQuotaMicros: 200000, + CPUPeriodMicros: 100000, + CPUWeight: 100, + MemoryHighBytes: 640 * 1024 * 1024, + MemoryMaxBytes: 640 * 1024 * 1024, + SwapMaxBytes: 0, + PIDsMax: 256, + } + if diff := cmp.Diff(want, got); diff != "" { + t.Errorf("derived cgroup limits mismatch (-want +got):\n%s", diff) + } +} + +func TestVMCgroupLimits_SQLRoundTrip(t *testing.T) { + want := model.NewVMCgroupLimits(4, 1024, model.VMCgroupPolicy{ + VMMHeadroomMiB: 128, CPUWeight: 250, PIDsMax: 512, SwapMaxBytes: 0, + }) + value, err := want.Value() + require.NoError(t, err) + encoded, ok := value.([]byte) + require.True(t, ok) + + for name, source := range map[string]any{"bytes": encoded, "string": string(encoded)} { + t.Run(name, func(t *testing.T) { + var got model.VMCgroupLimits + require.NoError(t, got.Scan(source)) + assert.Equal(t, want, got) + }) + } + + var got model.VMCgroupLimits + require.NoError(t, got.Scan(nil)) + assert.Equal(t, model.VMCgroupLimits{}, got) + assert.Error(t, got.Scan(driver.Value(int64(1)))) +} + +func TestSnapshotExtraConfig_PreservesCgroupEnvelope(t *testing.T) { + wantLimits := model.NewVMCgroupLimits(3, 768, model.VMCgroupPolicy{ + VMMHeadroomMiB: 128, CPUWeight: 100, PIDsMax: 256, SwapMaxBytes: 0, + }) + want := model.SnapshotExtraConfig{BootArgs: "quiet", CgroupLimits: wantLimits} + value, err := want.Value() + require.NoError(t, err) + + var got model.SnapshotExtraConfig + require.NoError(t, got.Scan(value)) + assert.Equal(t, want, got) +} diff --git a/internal/lib/model/host.go b/internal/lib/model/host.go index 768454d4..647da1e6 100644 --- a/internal/lib/model/host.go +++ b/internal/lib/model/host.go @@ -82,18 +82,19 @@ type HostHardware struct { // HostLimits holds detected host kernel limits. type HostLimits struct { - PIDMax int `json:"pid_max"` - FDMax int `json:"fd_max"` - ConntrackMax int `json:"conntrack_max"` - TAPDevicesMax int `json:"tap_devices_max"` - IPLocalPortRange [2]int `json:"ip_local_port_range"` - NestedVirtAvailable bool `json:"nested_virt_available"` - EPTAvailable bool `json:"ept_available"` - HugepageCount2MB int `json:"hugepage_count_2mb"` - KSMDisabled bool `json:"ksm_disabled"` - CgroupVersion int `json:"cgroup_version"` - SwapTotalMiB int `json:"swap_total_mib"` - KernelMinimumMet bool `json:"kernel_minimum_met"` + PIDMax int `json:"pid_max"` + FDMax int `json:"fd_max"` + ConntrackMax int `json:"conntrack_max"` + TAPDevicesMax int `json:"tap_devices_max"` + IPLocalPortRange [2]int `json:"ip_local_port_range"` + NestedVirtAvailable bool `json:"nested_virt_available"` + EPTAvailable bool `json:"ept_available"` + HugepageCount2MB int `json:"hugepage_count_2mb"` + KSMDisabled bool `json:"ksm_disabled"` + CgroupVersion int `json:"cgroup_version"` + CgroupControllers []string `json:"cgroup_controllers" db:"-"` + SwapTotalMiB int `json:"swap_total_mib"` + KernelMinimumMet bool `json:"kernel_minimum_met"` } // --- HostResources --- diff --git a/internal/lib/model/vm.go b/internal/lib/model/vm.go index aad6db31..0b425afa 100644 --- a/internal/lib/model/vm.go +++ b/internal/lib/model/vm.go @@ -23,34 +23,35 @@ const ( // VMItem represents a microVM instance. type VMItem struct { - ID string `json:"id" db:"id"` - Name string `json:"name" db:"name"` - Status VMStatus `json:"status" db:"status"` - PID int `json:"pid" db:"pid"` - IPv4 string `json:"ipv4" db:"ipv4"` - MAC string `json:"mac" db:"mac"` - NetworkID string `json:"network_id" db:"network_id"` - TapDevice string `json:"tap_device" db:"tap_device"` - ImageID string `json:"image_id" db:"image_id"` - KernelID string `json:"kernel_id" db:"kernel_id"` - BinaryID string `json:"binary_id" db:"binary_id"` - JailerBinaryID string `json:"jailer_binary_id" db:"jailer_binary_id"` - APISocketPath string `json:"api_socket_path" db:"api_socket_path"` - ConfigPath string `json:"config_path" db:"config_path"` - CloudInitMode string `json:"cloud_init_mode" db:"cloud_init_mode"` - VCPUCount int `json:"vcpu_count" db:"vcpu_count"` - MemSizeMiB int `json:"mem_size_mib" db:"mem_size_mib"` - DiskSizeMiB int `json:"disk_size_mib" db:"disk_size_mib"` - RootfsPath string `json:"rootfs_path" db:"rootfs_path"` - RootfsSuffix string `json:"rootfs_suffix" db:"rootfs_suffix"` - PCIEnabled bool `json:"pci_enabled" db:"pci_enabled"` - NestedVirt bool `json:"nested_virt" db:"nested_virt"` - RemoteExec bool `json:"remote_exec" db:"remote_exec"` - EnableLogging bool `json:"enable_logging" db:"enable_logging"` - EnableMetrics bool `json:"enable_metrics" db:"enable_metrics"` - EnableConsole bool `json:"enable_console" db:"enable_console"` - CreatedAt string `json:"created_at" db:"created_at"` - UpdatedAt string `json:"updated_at" db:"updated_at"` + ID string `json:"id" db:"id"` + Name string `json:"name" db:"name"` + Status VMStatus `json:"status" db:"status"` + PID int `json:"pid" db:"pid"` + IPv4 string `json:"ipv4" db:"ipv4"` + MAC string `json:"mac" db:"mac"` + NetworkID string `json:"network_id" db:"network_id"` + TapDevice string `json:"tap_device" db:"tap_device"` + ImageID string `json:"image_id" db:"image_id"` + KernelID string `json:"kernel_id" db:"kernel_id"` + BinaryID string `json:"binary_id" db:"binary_id"` + JailerBinaryID string `json:"jailer_binary_id" db:"jailer_binary_id"` + APISocketPath string `json:"api_socket_path" db:"api_socket_path"` + ConfigPath string `json:"config_path" db:"config_path"` + CloudInitMode string `json:"cloud_init_mode" db:"cloud_init_mode"` + VCPUCount int `json:"vcpu_count" db:"vcpu_count"` + MemSizeMiB int `json:"mem_size_mib" db:"mem_size_mib"` + CgroupLimits VMCgroupLimits `json:"cgroup_limits" db:"cgroup_limits"` + DiskSizeMiB int `json:"disk_size_mib" db:"disk_size_mib"` + RootfsPath string `json:"rootfs_path" db:"rootfs_path"` + RootfsSuffix string `json:"rootfs_suffix" db:"rootfs_suffix"` + PCIEnabled bool `json:"pci_enabled" db:"pci_enabled"` + NestedVirt bool `json:"nested_virt" db:"nested_virt"` + RemoteExec bool `json:"remote_exec" db:"remote_exec"` + EnableLogging bool `json:"enable_logging" db:"enable_logging"` + EnableMetrics bool `json:"enable_metrics" db:"enable_metrics"` + EnableConsole bool `json:"enable_console" db:"enable_console"` + CreatedAt string `json:"created_at" db:"created_at"` + UpdatedAt string `json:"updated_at" db:"updated_at"` // Optional fields RelaySocketPath *string `json:"relay_socket_path,omitempty" db:"relay_socket_path"` diff --git a/internal/service/jailer/entry.go b/internal/service/jailer/entry.go index 72ea6949..ccc2b4c1 100644 --- a/internal/service/jailer/entry.go +++ b/internal/service/jailer/entry.go @@ -10,6 +10,7 @@ import ( "os" "path/filepath" "regexp" + "slices" "sort" "strconv" "strings" @@ -19,6 +20,7 @@ import ( "mvmctl/internal/infra" "mvmctl/internal/lib/archive" + "mvmctl/internal/lib/model" "mvmctl/internal/lib/system" ) @@ -191,12 +193,21 @@ func launch(cfg Config) error { if err != nil { return err } + if manifest.CgroupLimits == nil { + return fmt.Errorf("cgroup limits are required for jailed launch") + } + if err := validateCgroupLimits(*manifest.CgroupLimits); err != nil { + return err + } if err := ensureTrustedDirectoryChain(infra.JailerChrootBase); err != nil { return fmt.Errorf("prepare Jailer chroot base: %w", err) } if err := cleanup(manifest.VMID); err != nil { return err } + if err := prepareCgroupV2(); err != nil { + return fmt.Errorf("prepare cgroup-v2 resource enforcement: %w", err) + } jailRoot := jailRoot(manifest.VMID) if err := ensureTrustedDirectoryChain(jailRoot); err != nil { return fmt.Errorf("prepare jail root: %w", err) @@ -252,6 +263,15 @@ func launch(cfg Config) error { "--uid", strconv.Itoa(uid), "--gid", strconv.Itoa(gid), "--chroot-base-dir", infra.JailerChrootBase, + "--cgroup-version", "2", + "--parent-cgroup", infra.JailerCgroupParent, + "--cgroup", fmt.Sprintf("cpu.max=%d %d", manifest.CgroupLimits.CPUQuotaMicros, + manifest.CgroupLimits.CPUPeriodMicros), + "--cgroup", fmt.Sprintf("cpu.weight=%d", manifest.CgroupLimits.CPUWeight), + "--cgroup", fmt.Sprintf("memory.high=%d", manifest.CgroupLimits.MemoryHighBytes), + "--cgroup", fmt.Sprintf("memory.max=%d", manifest.CgroupLimits.MemoryMaxBytes), + "--cgroup", fmt.Sprintf("memory.swap.max=%d", manifest.CgroupLimits.SwapMaxBytes), + "--cgroup", fmt.Sprintf("pids.max=%d", manifest.CgroupLimits.PIDsMax), "--", "--api-sock", "/run/mvm/" + filepath.Base(manifest.APISocket), } @@ -559,35 +579,128 @@ func cleanup(vmID string) error { return fmt.Errorf("invalid VM ID for jail cleanup") } root := jailRoot(vmID) - if _, err := os.Lstat(infra.JailerChrootBase); os.IsNotExist(err) { - return nil - } - if err := validateTrustedDirectoryChain(infra.JailerChrootBase); err != nil { - return fmt.Errorf("validate Jailer chroot base before cleanup: %w", err) + if err := ensureCgroupEmpty(vmID); err != nil { + return err } - data, err := os.ReadFile("/proc/self/mountinfo") - if err == nil { - var targets []string - for line := range strings.SplitSeq(string(data), "\n") { - fields := strings.Fields(line) - if len(fields) > 4 && (fields[4] == root || strings.HasPrefix(fields[4], root+"/")) { - targets = append(targets, fields[4]) + if _, err := os.Lstat(infra.JailerChrootBase); err == nil { + if err := validateTrustedDirectoryChain(infra.JailerChrootBase); err != nil { + return fmt.Errorf("validate Jailer chroot base before cleanup: %w", err) + } + data, readErr := os.ReadFile("/proc/self/mountinfo") + if readErr == nil { + var targets []string + for line := range strings.SplitSeq(string(data), "\n") { + fields := strings.Fields(line) + if len(fields) > 4 && (fields[4] == root || strings.HasPrefix(fields[4], root+"/")) { + targets = append(targets, fields[4]) + } } - } - sort.Slice(targets, func(i, j int) bool { return len(targets[i]) > len(targets[j]) }) - for _, target := range targets { - err := syscall.Unmount(target, syscall.MNT_DETACH) - if err != nil && err != syscall.EINVAL && err != syscall.ENOENT { - return fmt.Errorf("unmount jail resource %s: %w", target, err) + sort.Slice(targets, func(i, j int) bool { return len(targets[i]) > len(targets[j]) }) + for _, target := range targets { + err := syscall.Unmount(target, syscall.MNT_DETACH) + if err != nil && err != syscall.EINVAL && err != syscall.ENOENT { + return fmt.Errorf("unmount jail resource %s: %w", target, err) + } } } + if err := os.RemoveAll(filepath.Dir(root)); err != nil && !os.IsNotExist(err) { + return fmt.Errorf("remove jail: %w", err) + } + } + if err := removeCgroup(vmID); err != nil { + return err + } + return nil +} + +func validateCgroupLimits(limits model.VMCgroupLimits) error { + if limits.CPUQuotaMicros <= 0 || limits.CPUPeriodMicros <= 0 { + return fmt.Errorf("invalid CPU cgroup limit") + } + if limits.CPUWeight < 1 || limits.CPUWeight > 10000 { + return fmt.Errorf("invalid CPU cgroup weight") + } + if limits.MemoryHighBytes <= 0 || limits.MemoryMaxBytes <= 0 || + limits.MemoryHighBytes > limits.MemoryMaxBytes { + return fmt.Errorf("invalid memory cgroup limit") + } + if limits.SwapMaxBytes < 0 || limits.PIDsMax <= 0 { + return fmt.Errorf("invalid swap or PID cgroup limit") + } + return nil +} + +func prepareCgroupV2() error { + controllersPath := filepath.Join(infra.CgroupV2Root, "cgroup.controllers") + controllers, err := os.ReadFile(controllersPath) + if err != nil { + return fmt.Errorf("unified cgroup v2 is required: %w", err) + } + required := []string{"cpu", "memory", "pids"} + available := strings.Fields(string(controllers)) + for _, controller := range required { + if !slices.Contains(available, controller) { + return fmt.Errorf("required cgroup-v2 controller is unavailable: %s", controller) + } } - if err := os.RemoveAll(filepath.Dir(root)); err != nil && !os.IsNotExist(err) { - return fmt.Errorf("remove jail: %w", err) + if err := enableCgroupControllers(infra.CgroupV2Root, required); err != nil { + return err + } + parent := filepath.Join(infra.CgroupV2Root, infra.JailerCgroupParent) + if err := os.Mkdir(parent, 0755); err != nil && !os.IsExist(err) { + return fmt.Errorf("create fixed cgroup parent: %w", err) + } + if err := enableCgroupControllers(parent, required); err != nil { + return err } return nil } +func enableCgroupControllers(path string, controllers []string) error { + controlPath := filepath.Join(path, "cgroup.subtree_control") + for _, controller := range controllers { + if err := os.WriteFile(controlPath, []byte("+"+controller), 0644); err != nil { + return fmt.Errorf("enable %s controller in %s: %w", controller, path, err) + } + } + return nil +} + +func removeCgroup(vmID string) error { + path := cgroupPath(vmID) + if err := ensureCgroupEmpty(vmID); err != nil { + return err + } + if err := os.Remove(path); err != nil && !os.IsNotExist(err) { + return fmt.Errorf("remove empty VM cgroup: %w", err) + } + return nil +} + +func ensureCgroupEmpty(vmID string) error { + path := cgroupPath(vmID) + procs, err := os.ReadFile(filepath.Join(path, "cgroup.procs")) + if os.IsNotExist(err) { + return nil + } + if err != nil { + return fmt.Errorf("read VM cgroup processes before cleanup: %w", err) + } + if strings.TrimSpace(string(procs)) != "" { + return fmt.Errorf("refusing to remove VM cgroup while it contains live processes") + } + return nil +} + +func cgroupPath(vmID string) string { + return filepath.Join( + infra.CgroupV2Root, + infra.JailerCgroupParent, + infra.JailerCgroupExecutable, + vmID, + ) +} + func removeRelease(version string) error { if !versionPattern.MatchString(version) { return fmt.Errorf("invalid release version") diff --git a/internal/service/jailer/entry_test.go b/internal/service/jailer/entry_test.go index c0661eba..351860ec 100644 --- a/internal/service/jailer/entry_test.go +++ b/internal/service/jailer/entry_test.go @@ -13,6 +13,7 @@ import ( "github.com/stretchr/testify/require" "mvmctl/internal/infra" + "mvmctl/internal/lib/model" ) const ( @@ -175,6 +176,56 @@ func TestJailRootIsDeterministic(t *testing.T) { assert.Equal(t, first, second) } +func TestValidateCgroupLimitsRejectsAbsentAndInvalidEnvelopes(t *testing.T) { + valid := model.NewVMCgroupLimits(2, 512, model.VMCgroupPolicy{ + VMMHeadroomMiB: 128, CPUWeight: 100, PIDsMax: 256, SwapMaxBytes: 0, + }) + tests := map[string]struct { + mutate func(*model.VMCgroupLimits) + valid bool + }{ + "valid": {valid: true}, + "absent": {mutate: func(l *model.VMCgroupLimits) { *l = model.VMCgroupLimits{} }}, + "zero_cpu_quota": {mutate: func(l *model.VMCgroupLimits) { l.CPUQuotaMicros = 0 }}, + "zero_cpu_period": {mutate: func(l *model.VMCgroupLimits) { l.CPUPeriodMicros = 0 }}, + "weight_below_minimum": {mutate: func(l *model.VMCgroupLimits) { l.CPUWeight = 0 }}, + "weight_above_maximum": {mutate: func(l *model.VMCgroupLimits) { l.CPUWeight = 10001 }}, + "zero_memory_high": {mutate: func(l *model.VMCgroupLimits) { l.MemoryHighBytes = 0 }}, + "high_above_max": {mutate: func(l *model.VMCgroupLimits) { l.MemoryHighBytes = l.MemoryMaxBytes + 1 }}, + "negative_swap": {mutate: func(l *model.VMCgroupLimits) { l.SwapMaxBytes = -1 }}, + "zero_pid_cap": {mutate: func(l *model.VMCgroupLimits) { l.PIDsMax = 0 }}, + } + for name, tc := range tests { + t.Run(name, func(t *testing.T) { + limits := valid + if tc.mutate != nil { + tc.mutate(&limits) + } + err := validateCgroupLimits(limits) + if tc.valid { + require.NoError(t, err) + return + } + require.Error(t, err) + }) + } +} + +func TestEnsureCgroupEmptyRefusesPopulatedAndAcceptsAbsentState(t *testing.T) { + path := t.TempDir() + base := filepath.Join(infra.CgroupV2Root, infra.JailerCgroupParent, infra.JailerCgroupExecutable) + vmID, err := filepath.Rel(base, path) + require.NoError(t, err) + assert.Equal(t, filepath.Clean(path), cgroupPath(vmID)) + + require.NoError(t, os.WriteFile(filepath.Join(path, "cgroup.procs"), []byte("123\n"), 0600)) + err = ensureCgroupEmpty(vmID) + require.ErrorContains(t, err, "contains live processes") + + require.NoError(t, os.Remove(filepath.Join(path, "cgroup.procs"))) + require.NoError(t, ensureCgroupEmpty(vmID)) +} + // Rationale: User-owned or symlinked executables must never satisfy the // root-owned, non-user-writable trusted executable contract. func TestValidateTrustedExecutableRejectsUntrustedPaths(t *testing.T) { @@ -257,6 +308,12 @@ func newManagedFixture(t *testing.T) *managedFixture { ISOPath: filepath.Join(vmDir, "cloud-init.iso"), SnapshotDir: snapshotDir, Volumes: []VolumeMount{{DriveID: testResourceID, HostPath: filepath.Join(volumesRoot, "volume.raw")}}, APISocket: filepath.Join(vmDir, "firecracker.socket"), + CgroupLimits: func() *model.VMCgroupLimits { + limits := model.NewVMCgroupLimits(2, 512, model.VMCgroupPolicy{ + VMMHeadroomMiB: 128, CPUWeight: 100, PIDsMax: 256, SwapMaxBytes: 0, + }) + return &limits + }(), } fixture.writeManifest(t) return fixture diff --git a/internal/service/jailer/wire.go b/internal/service/jailer/wire.go index 746159bf..1c2db056 100644 --- a/internal/service/jailer/wire.go +++ b/internal/service/jailer/wire.go @@ -2,6 +2,8 @@ // Layer: Service — callers select typed operations, never raw Jailer arguments. package jailer +import "mvmctl/internal/lib/model" + // VolumeMount describes one VM volume exposed inside a jail. type VolumeMount struct { DriveID string `json:"drive_id"` @@ -11,19 +13,20 @@ type VolumeMount struct { // LaunchManifest contains the exact per-VM resources needed by Firecracker. type LaunchManifest struct { - VMID string `json:"vm_id"` - VMDir string `json:"vm_dir"` - Version string `json:"version"` - ConfigPath string `json:"config_path"` - PIDPath string `json:"pid_path"` - KernelPath string `json:"kernel_path"` - RootfsPath string `json:"rootfs_path"` - ISOPath string `json:"iso_path,omitempty"` - SnapshotDir string `json:"snapshot_dir,omitempty"` - Volumes []VolumeMount `json:"volumes,omitempty"` - APISocket string `json:"api_socket"` - PCIEnabled bool `json:"pci_enabled"` - SnapshotMode bool `json:"snapshot_mode"` + VMID string `json:"vm_id"` + VMDir string `json:"vm_dir"` + Version string `json:"version"` + ConfigPath string `json:"config_path"` + PIDPath string `json:"pid_path"` + KernelPath string `json:"kernel_path"` + RootfsPath string `json:"rootfs_path"` + ISOPath string `json:"iso_path,omitempty"` + SnapshotDir string `json:"snapshot_dir,omitempty"` + Volumes []VolumeMount `json:"volumes,omitempty"` + APISocket string `json:"api_socket"` + PCIEnabled bool `json:"pci_enabled"` + SnapshotMode bool `json:"snapshot_mode"` + CgroupLimits *model.VMCgroupLimits `json:"cgroup_limits,omitempty"` } // InstallResult identifies the immutable trusted paths installed for a release. diff --git a/pkg/api/inputs/vm_create.go b/pkg/api/inputs/vm_create.go index fc6458b2..06acd510 100644 --- a/pkg/api/inputs/vm_create.go +++ b/pkg/api/inputs/vm_create.go @@ -140,6 +140,7 @@ type ResolvedVMCreateInput struct { Volumes []*model.VolumeItem VsockPort int // vsock port (0 = disabled / no vsock) Writeback bool // cache type override: when true, use "Writeback" for rootfs + volumes + CgroupLimits model.VMCgroupLimits } // VMCreateRequest resolves all DB-backed defaults and validates VM creation inputs. @@ -421,6 +422,16 @@ func (r *VMCreateRequest) Resolve(ctx context.Context) (*ResolvedVMCreateInput, if input.VCPUCount != nil && *input.VCPUCount > 0 { vcpuCount = *input.VCPUCount } + cgroupHeadroomMiB, _ := r.cfg.GetInt(ctx, "defaults.vm", "cgroup_vmm_headroom_mib") + cgroupCPUWeight, _ := r.cfg.GetInt(ctx, "defaults.vm", "cgroup_cpu_weight") + cgroupPIDsMax, _ := r.cfg.GetInt(ctx, "defaults.vm", "cgroup_pids_max") + cgroupSwapMaxBytes, _ := r.cfg.GetInt(ctx, "defaults.vm", "cgroup_swap_max_bytes") + cgroupLimits := model.NewVMCgroupLimits(vcpuCount, memMib, model.VMCgroupPolicy{ + VMMHeadroomMiB: int64(cgroupHeadroomMiB), + CPUWeight: int64(cgroupCPUWeight), + PIDsMax: int64(cgroupPIDsMax), + SwapMaxBytes: int64(cgroupSwapMaxBytes), + }) // Resolve cloud-init mode // Default to "off" when no explicit mode is set — the provisioner injects // SSH keys directly into the rootfs regardless of cloud-init mode. @@ -608,6 +619,7 @@ func (r *VMCreateRequest) Resolve(ctx context.Context) (*ResolvedVMCreateInput, NoCloudKillAfter: nocloudKillAfter, VsockPort: vsockPort, Writeback: writeback, + CgroupLimits: cgroupLimits, } // Validate if err := r.ensureValidate(ctx, result); err != nil { diff --git a/pkg/api/results/vm.go b/pkg/api/results/vm.go index 22f4ac58..5c8bea07 100644 --- a/pkg/api/results/vm.go +++ b/pkg/api/results/vm.go @@ -39,9 +39,10 @@ type VMItemInfo struct { // VMResourcesInfo groups VM resource allocation in an inspect response. type VMResourcesInfo struct { - VCPU int `json:"vcpu"` - Mem int `json:"mem"` - Disk int `json:"disk"` + VCPU int `json:"vcpu"` + Mem int `json:"mem"` + Disk int `json:"disk"` + Cgroup model.VMCgroupState `json:"cgroup"` } // VMNetworkingInfo groups VM networking info in an inspect response. diff --git a/pkg/api/snapshot.go b/pkg/api/snapshot.go index 660df7c7..85d9daf7 100644 --- a/pkg/api/snapshot.go +++ b/pkg/api/snapshot.go @@ -160,6 +160,7 @@ func (op *Operation) SnapshotCreate( EnableMetrics: vmItem.EnableMetrics, LogLevel: vmItem.LogLevel, CPUConfig: vmItem.CPUConfig, + CgroupLimits: vmItem.CgroupLimits, } if vmItem.Vsock != nil { extraConfig.VsockPort = vmItem.Vsock.Port @@ -420,6 +421,10 @@ func (op *Operation) SnapshotRestore( vmItem.EnableLogging = cfg.EnableLogging vmItem.EnableMetrics = cfg.EnableMetrics vmItem.LogLevel = cfg.LogLevel + vmItem.CgroupLimits = cfg.CgroupLimits + } + if vmItem.CgroupLimits.PolicyVersion == 0 { + vmItem.CgroupLimits = op.resolveVMCgroupLimits(ctx, vmItem.VCPUCount, vmItem.MemSizeMiB) } // Persist VM record diff --git a/pkg/api/vm.go b/pkg/api/vm.go index d72aca00..88de4051 100644 --- a/pkg/api/vm.go +++ b/pkg/api/vm.go @@ -812,10 +812,19 @@ func (op *Operation) VMRemove(ctx context.Context, input inputs.VMInput) *errs.B controller.Stop(ctx, input.Force) // Defense-in-depth: force-kill if vmLocal.PID > 0 && system.IsProcessRunning(vmLocal.PID) { - proc, err := os.FindProcess(vmLocal.PID) - if err == nil { - _ = proc.Kill() - } + system.GracefulShutdown(system.ShutdownConfig{ + Pid: vmLocal.PID, IsChild: false, GracefulTimeout: 100 * time.Millisecond, + KillTimeout: 100 * time.Millisecond, ExpectedStartTime: vmLocal.ProcessStartTime, + }) + } + if err := vm.CleanupJail(ctx, vmLocal.ID); err != nil { + slog.Error("failed to clean VM jail and cgroup during removal", "vm", vmLocal.Name, "error", err) + results = append(results, errs.OperationResult{ + Status: "error", Code: string(errs.CodeVMCgroupCleanupFailed), + Item: vmLocal, Message: fmt.Sprintf("failed to clean resources for VM '%s': %v", vmLocal.Name, err), + Exception: err, + }) + continue } // Console relay, TAP, IP lease cleanup if vmLocal.RelayPID != nil { @@ -941,23 +950,23 @@ func (op *Operation) VMGet(ctx context.Context, input inputs.VMInput) (*model.VM // Inspect returns detailed VM info with enriched data. func (op *Operation) VMInspect(ctx context.Context, input inputs.VMInput) (*results.VMInspect, error) { - vm, err := op.VMGet(ctx, input) + vmItem, err := op.VMGet(ctx, input) if err != nil { return nil, err } // Enrich all relations at once instead of manual repo calls. - if err := op.Enr.EnrichVM(ctx, []*model.VMItem{vm}, + if err := op.Enr.EnrichVM(ctx, []*model.VMItem{vmItem}, "kernel", "image", "binary", "network", "volumes", "vsock", ); err != nil { return nil, err } - relayRunning := vm.RelayPID != nil && system.IsProcessRunning(*vm.RelayPID) - vmDir := infra.GetVMDirByID(vm.ID) + relayRunning := vmItem.RelayPID != nil && system.IsProcessRunning(*vmItem.RelayPID) + vmDir := infra.GetVMDirByID(vmItem.ID) // Volumes (enriched vm.Volumes or fallback to manual lookup). volumes := make([]results.VMVolume, 0) - srcVols := vm.Volumes - if len(srcVols) == 0 && len(vm.VolumeIDs) > 0 { - vols, err := op.Repos.Volume.FindByIDs(ctx, vm.VolumeIDs) + srcVols := vmItem.Volumes + if len(srcVols) == 0 && len(vmItem.VolumeIDs) > 0 { + vols, err := op.Repos.Volume.FindByIDs(ctx, vmItem.VolumeIDs) if err == nil { srcVols = vols } @@ -968,47 +977,62 @@ func (op *Operation) VMInspect(ctx context.Context, input inputs.VMInput) (*resu Format: string(v.Format), Status: string(v.Status), }) } - configPath := &vm.ConfigPath - if vm.ConfigPath == "" { + configPath := &vmItem.ConfigPath + if vmItem.ConfigPath == "" { configPath = nil } + if vmItem.CgroupLimits.PolicyVersion == 0 { + vmItem.CgroupLimits = op.resolveVMCgroupLimits(ctx, vmItem.VCPUCount, vmItem.MemSizeMiB) + } + cgroupState := model.VMCgroupState{ + Path: vm.CgroupPath(vmItem.ID), Requested: vmItem.CgroupLimits, Status: model.VMCgroupEnforcementInactive, + } + if vmItem.PID > 0 && system.IsProcessAlive(vmItem.PID, vmItem.ProcessStartTime) { + observed, inspectErr := vm.InspectCgroup(ctx, vmItem.ID, vmItem.PID, vmItem.CgroupLimits) + if inspectErr != nil { + cgroupState.Status = model.VMCgroupEnforcementUnavailable + cgroupState.Mismatches = []string{inspectErr.Error()} + } else { + cgroupState = *observed + } + } return &results.VMInspect{ VM: results.VMItemInfo{ - Name: vm.Name, ID: vm.ID, Status: string(vm.Status), - PID: vm.PID, ExitCode: vm.ExitCode, - SSHKeys: vm.SSHKeys, SSHUser: vm.SSHUser, - CloudInitMode: vm.CloudInitMode, - NocloudNetPort: vm.NocloudNetPort, NocloudNetPID: vm.NocloudNetPID, - PCIEnabled: vm.PCIEnabled, NestedVirt: vm.NestedVirt, - EnableConsole: vm.EnableConsole, - EnableLogging: vm.EnableLogging, EnableMetrics: vm.EnableMetrics, - AllowRemoteExec: vm.RemoteExec, - CreatedAt: vm.CreatedAt, UpdatedAt: vm.UpdatedAt, + Name: vmItem.Name, ID: vmItem.ID, Status: string(vmItem.Status), + PID: vmItem.PID, ExitCode: vmItem.ExitCode, + SSHKeys: vmItem.SSHKeys, SSHUser: vmItem.SSHUser, + CloudInitMode: vmItem.CloudInitMode, + NocloudNetPort: vmItem.NocloudNetPort, NocloudNetPID: vmItem.NocloudNetPID, + PCIEnabled: vmItem.PCIEnabled, NestedVirt: vmItem.NestedVirt, + EnableConsole: vmItem.EnableConsole, + EnableLogging: vmItem.EnableLogging, EnableMetrics: vmItem.EnableMetrics, + AllowRemoteExec: vmItem.RemoteExec, + CreatedAt: vmItem.CreatedAt, UpdatedAt: vmItem.UpdatedAt, }, Resources: results.VMResourcesInfo{ - VCPU: vm.VCPUCount, Mem: vm.MemSizeMiB, Disk: vm.DiskSizeMiB, + VCPU: vmItem.VCPUCount, Mem: vmItem.MemSizeMiB, Disk: vmItem.DiskSizeMiB, Cgroup: cgroupState, }, Networking: results.VMNetworkingInfo{ - IPv4: vm.IPv4, MAC: vm.MAC, - Network: vm.Network, TapDevice: vm.TapDevice, + IPv4: vmItem.IPv4, MAC: vmItem.MAC, + Network: vmItem.Network, TapDevice: vmItem.TapDevice, }, Assets: results.VMAssetsInfo{ - Image: vm.Image, - Kernel: vm.Kernel, - Binary: vm.Binary, + Image: vmItem.Image, + Kernel: vmItem.Kernel, + Binary: vmItem.Binary, }, Filesystem: results.VMFilesystemInfo{ - VMDir: vmDir, RootfsPath: vm.RootfsPath, + VMDir: vmDir, RootfsPath: vmItem.RootfsPath, ConfigPath: configPath, - LogPath: vm.LogPath, - SerialOutputPath: vm.SerialOutputPath, + LogPath: vmItem.LogPath, + SerialOutputPath: vmItem.SerialOutputPath, }, Console: results.VMConsoleInfo{ - RelayRunning: relayRunning, RelayPID: vm.RelayPID, - RelaySocketPath: vm.RelaySocketPath, + RelayRunning: relayRunning, RelayPID: vmItem.RelayPID, + RelaySocketPath: vmItem.RelaySocketPath, }, Volumes: volumes, - Vsock: vsockToInspectInfo(vm.Vsock), + Vsock: vsockToInspectInfo(vmItem.Vsock), }, nil } @@ -1108,10 +1132,18 @@ func (op *Operation) VMStop(ctx context.Context, input inputs.VMInput) *errs.Bat controller.Stop(ctx, input.Force) // Defense-in-depth: force-kill if stop() silently left the Firecracker process alive if vmLocal.PID > 0 && system.IsProcessRunning(vmLocal.PID) { - proc, _ := os.FindProcess(vmLocal.PID) - if proc != nil { - _ = proc.Kill() - } + system.GracefulShutdown(system.ShutdownConfig{ + Pid: vmLocal.PID, IsChild: false, GracefulTimeout: 100 * time.Millisecond, + KillTimeout: 100 * time.Millisecond, ExpectedStartTime: vmLocal.ProcessStartTime, + }) + } + if err := vm.CleanupJail(ctx, vmLocal.ID); err != nil { + slog.Error("failed to clean VM jail and cgroup after stop", "vm", vmLocal.Name, "error", err) + results = append(results, errs.OperationResult{ + Status: "error", Code: string(errs.CodeVMCgroupCleanupFailed), + Item: vmLocal, Message: fmt.Sprintf("stop '%s': cleanup failed: %v", vmLocal.Name, err), Exception: err, + }) + continue } op.AuditLog.LogOperation("vm.stop", nil, fmt.Sprintf("name=%s", vmLocal.Name)) results = append(results, errs.OperationResult{ @@ -1128,6 +1160,12 @@ func (op *Operation) vmRespawnFirecracker( snapshotMode bool, snapshotDir string, ) error { + if v.CgroupLimits.PolicyVersion == 0 { + v.CgroupLimits = op.resolveVMCgroupLimits(ctx, v.VCPUCount, v.MemSizeMiB) + if err := op.Repos.VM.Upsert(ctx, v); err != nil { + return fmt.Errorf("persist migrated VM cgroup limits: %w", err) + } + } vmDir := infra.GetVMDirByID(v.ID) // Network info is required — the VM record must have it pre-loaded. if v.Network == nil { @@ -1205,6 +1243,7 @@ func (op *Operation) vmRespawnFirecracker( KernelPath: v.Kernel.Path, VCPUCount: v.VCPUCount, MemSizeMiB: v.MemSizeMiB, + CgroupLimits: v.CgroupLimits, GuestIP: v.IPv4, GuestMAC: v.MAC, TapName: v.TapDevice, @@ -1293,6 +1332,19 @@ func (op *Operation) vmRespawnFirecracker( return nil } +func (op *Operation) resolveVMCgroupLimits(ctx context.Context, vcpuCount, memoryMiB int) model.VMCgroupLimits { + headroomMiB, _ := op.Services.Config.GetInt(ctx, "defaults.vm", "cgroup_vmm_headroom_mib") + cpuWeight, _ := op.Services.Config.GetInt(ctx, "defaults.vm", "cgroup_cpu_weight") + pidsMax, _ := op.Services.Config.GetInt(ctx, "defaults.vm", "cgroup_pids_max") + swapMaxBytes, _ := op.Services.Config.GetInt(ctx, "defaults.vm", "cgroup_swap_max_bytes") + return model.NewVMCgroupLimits(vcpuCount, memoryMiB, model.VMCgroupPolicy{ + VMMHeadroomMiB: int64(headroomMiB), + CPUWeight: int64(cpuWeight), + PIDsMax: int64(pidsMax), + SwapMaxBytes: int64(swapMaxBytes), + }) +} + // --- Reboot / Pause / Resume --- // Reboot reboots one or more VMs. func (op *Operation) VMReboot(ctx context.Context, input inputs.VMInput) *errs.BatchResult { @@ -1503,6 +1555,7 @@ func (c *VMCreateBuilder) buildFirecrackerConfig() *model.FirecrackerConfig { KernelPath: c.resolved.Kernel.Path, VCPUCount: c.resolved.VCPUCount, MemSizeMiB: c.resolved.MemSizeMib, + CgroupLimits: c.resolved.CgroupLimits, GuestIP: c.guestIP, GuestMAC: c.guestMAC, TapName: c.tapName, @@ -1579,6 +1632,7 @@ func (c *VMCreateBuilder) toVMModel() *model.VMItem { JailerBinaryID: c.resolved.Jailer.ID, VCPUCount: c.resolved.VCPUCount, MemSizeMiB: c.resolved.MemSizeMib, + CgroupLimits: c.resolved.CgroupLimits, DiskSizeMiB: c.resolved.DiskSizeMib, APISocketPath: filepath.Join(c.vmDir, c.resolved.APISocketFilename), ConfigPath: filepath.Join(c.vmDir, c.resolved.ConfigFilename), diff --git a/pkg/errs/codes.go b/pkg/errs/codes.go index 8f88f7bc..5b96e98a 100644 --- a/pkg/errs/codes.go +++ b/pkg/errs/codes.go @@ -4,23 +4,25 @@ type Code string const ( // --- VM domain --- - CodeVMNotFound Code = "vm.not_found" - CodeVMAlreadyExists Code = "vm.already_exists" - CodeVMStateInvalid Code = "vm.state.invalid" - CodeVMCreateFailed Code = "vm.create.failed" - CodeVMBuilderFailed Code = "vm.create.builder_failed" - CodeVMResolveFailed Code = "vm.resolve.failed" - CodeVMResourceExhausted Code = "vm.resource.exhausted" - CodeVMBinaryNotFound Code = "vm.create.binary_not_found" - CodeVMImageNotFound Code = "vm.create.image_not_found" - CodeVMKernelNotFound Code = "vm.create.kernel_not_found" - CodeVMNetworkNotFound Code = "vm.create.network_not_found" - CodeVMSSHKeyNotFound Code = "vm.create.ssh_key_not_found" - CodeVMNameCollision Code = "vm.name_collision" - CodeVMAtomicFailed Code = "vm.atomic_failed" - CodeVMCreateFailure Code = "vm.create_failure" - CodeVMSnapshotFailed Code = "vm.snapshot_failed" - CodeVMLoadSnapshotFailed Code = "vm.load_snapshot_failed" + CodeVMNotFound Code = "vm.not_found" + CodeVMAlreadyExists Code = "vm.already_exists" + CodeVMStateInvalid Code = "vm.state.invalid" + CodeVMCreateFailed Code = "vm.create.failed" + CodeVMBuilderFailed Code = "vm.create.builder_failed" + CodeVMResolveFailed Code = "vm.resolve.failed" + CodeVMResourceExhausted Code = "vm.resource.exhausted" + CodeVMBinaryNotFound Code = "vm.create.binary_not_found" + CodeVMImageNotFound Code = "vm.create.image_not_found" + CodeVMKernelNotFound Code = "vm.create.kernel_not_found" + CodeVMNetworkNotFound Code = "vm.create.network_not_found" + CodeVMSSHKeyNotFound Code = "vm.create.ssh_key_not_found" + CodeVMNameCollision Code = "vm.name_collision" + CodeVMAtomicFailed Code = "vm.atomic_failed" + CodeVMCreateFailure Code = "vm.create_failure" + CodeVMSnapshotFailed Code = "vm.snapshot_failed" + CodeVMLoadSnapshotFailed Code = "vm.load_snapshot_failed" + CodeVMCgroupVerificationFailed Code = "vm.cgroup.verification_failed" + CodeVMCgroupCleanupFailed Code = "vm.cgroup.cleanup_failed" // --- Network domain --- CodeNetworkSubnetOverlap Code = "network.subnet.overlap" diff --git a/pkg/errs/domain.go b/pkg/errs/domain.go index 5f4e8fa3..81a69e59 100644 --- a/pkg/errs/domain.go +++ b/pkg/errs/domain.go @@ -89,23 +89,25 @@ func opForCode(code Code) string { var codeClassMap = map[Code]Class{ // --- VM domain --- - CodeVMNotFound: ClassValidation, - CodeVMAlreadyExists: ClassConflict, - CodeVMStateInvalid: ClassValidation, - CodeVMCreateFailed: ClassInternal, - CodeVMBuilderFailed: ClassInternal, - CodeVMResolveFailed: ClassValidation, - CodeVMResourceExhausted: ClassValidation, - CodeVMBinaryNotFound: ClassValidation, - CodeVMImageNotFound: ClassValidation, - CodeVMKernelNotFound: ClassValidation, - CodeVMNetworkNotFound: ClassValidation, - CodeVMSSHKeyNotFound: ClassValidation, - CodeVMNameCollision: ClassValidation, - CodeVMAtomicFailed: ClassInternal, - CodeVMCreateFailure: ClassInternal, - CodeVMSnapshotFailed: ClassInternal, - CodeVMLoadSnapshotFailed: ClassInternal, + CodeVMNotFound: ClassValidation, + CodeVMAlreadyExists: ClassConflict, + CodeVMStateInvalid: ClassValidation, + CodeVMCreateFailed: ClassInternal, + CodeVMBuilderFailed: ClassInternal, + CodeVMResolveFailed: ClassValidation, + CodeVMResourceExhausted: ClassValidation, + CodeVMBinaryNotFound: ClassValidation, + CodeVMImageNotFound: ClassValidation, + CodeVMKernelNotFound: ClassValidation, + CodeVMNetworkNotFound: ClassValidation, + CodeVMSSHKeyNotFound: ClassValidation, + CodeVMNameCollision: ClassValidation, + CodeVMAtomicFailed: ClassInternal, + CodeVMCreateFailure: ClassInternal, + CodeVMSnapshotFailed: ClassInternal, + CodeVMLoadSnapshotFailed: ClassInternal, + CodeVMCgroupVerificationFailed: ClassInternal, + CodeVMCgroupCleanupFailed: ClassInternal, // --- Network domain --- CodeNetworkSubnetOverlap: ClassValidation, @@ -266,23 +268,25 @@ var codeClassMap = map[Code]Class{ } var codeOpMap = map[Code]string{ - CodeVMNotFound: "vm", - CodeVMAlreadyExists: "vm", - CodeVMStateInvalid: "vm", - CodeVMCreateFailed: "vm", - CodeVMBuilderFailed: "vm", - CodeVMResolveFailed: "vm", - CodeVMResourceExhausted: "vm", - CodeVMBinaryNotFound: "vm", - CodeVMImageNotFound: "vm", - CodeVMKernelNotFound: "vm", - CodeVMNetworkNotFound: "vm", - CodeVMSSHKeyNotFound: "vm", - CodeVMNameCollision: "vm", - CodeVMAtomicFailed: "vm", - CodeVMCreateFailure: "vm", - CodeVMSnapshotFailed: "vm", - CodeVMLoadSnapshotFailed: "vm", + CodeVMNotFound: "vm", + CodeVMAlreadyExists: "vm", + CodeVMStateInvalid: "vm", + CodeVMCreateFailed: "vm", + CodeVMBuilderFailed: "vm", + CodeVMResolveFailed: "vm", + CodeVMResourceExhausted: "vm", + CodeVMBinaryNotFound: "vm", + CodeVMImageNotFound: "vm", + CodeVMKernelNotFound: "vm", + CodeVMNetworkNotFound: "vm", + CodeVMSSHKeyNotFound: "vm", + CodeVMNameCollision: "vm", + CodeVMAtomicFailed: "vm", + CodeVMCreateFailure: "vm", + CodeVMSnapshotFailed: "vm", + CodeVMLoadSnapshotFailed: "vm", + CodeVMCgroupVerificationFailed: "vm", + CodeVMCgroupCleanupFailed: "vm", CodeNetworkSubnetOverlap: "network", CodeNetworkNotFound: "network", diff --git a/scripts/run-system-tests.py b/scripts/run-system-tests.py index 20718c9a..366d50ec 100755 --- a/scripts/run-system-tests.py +++ b/scripts/run-system-tests.py @@ -93,6 +93,10 @@ ], "vm_lifecycle": ["tests/system/vm/test_vm_lifecycle.py"], "vm_data_persistence": ["tests/system/vm/test_vm_data_persistence.py"], + "jailer": [ + "tests/system/vm/test_jailer.py", + "tests/system/vm/test_cgroup.py", + ], "ssh": ["tests/system/ssh/test_ssh.py"], "console": ["tests/system/console/test_console.py"], "logs": ["tests/system/logs/test_logs.py"], diff --git a/tests/system/COVERAGE_MATRIX.md b/tests/system/COVERAGE_MATRIX.md index 93b27cad..372dd852 100644 --- a/tests/system/COVERAGE_MATRIX.md +++ b/tests/system/COVERAGE_MATRIX.md @@ -611,6 +611,7 @@ removed, or when test coverage changes. | Export/import config journey | ⚡ Shallow | `full_journeys/test_full_journeys.py` | `TestExportImport` | L2: inspect --json schema verification | | Concurrent VM creation | ✅ Deep | `full_journeys/test_full_journeys.py` | `TestConcurrentVMCreation` | L3: 10 concurrent VMs, all running via ls --json, SSH into each | | Canonical Jailer lifecycle | ✅ Deep | `vm/test_jailer.py`, `full_journeys/test_full_journeys.py` | `TestCanonicalJailerLaunch`, `TestJailerFailClosed` | L2: trusted pair, non-root jailed process, lifecycle, volume, snapshot, fail-closed | +| Typed cgroup-v2 envelope | ✅ Deep | `vm/test_cgroup.py`, `vm/test_jailer.py` | `TestVMCgroupEnvelope`, `TestCanonicalJailerLaunch` | L2: exact membership and limits, inspect state, snapshot preservation, stop/start/remove cleanup, prerequisite fail-closed | --- diff --git a/tests/system/vm/test_cgroup.py b/tests/system/vm/test_cgroup.py new file mode 100644 index 00000000..cd911817 --- /dev/null +++ b/tests/system/vm/test_cgroup.py @@ -0,0 +1,204 @@ +"""Typed cgroup-v2 VM resource envelope system specification — Tier 2.""" + +from __future__ import annotations + +import json +import shlex +import uuid +from typing import Any + +import pytest + +from tests.system.conftest import ( + _guest_run, + _run_mvm, + _unique_subnet, + ensure_vm_deps, +) + +pytestmark = [ + pytest.mark.system, + pytest.mark.domain_vm, + pytest.mark.tier2, + pytest.mark.needs_kvm, + pytest.mark.slow, +] + + +def _vm_entry(runner_vm: str, vm_name: str) -> dict[str, Any] | None: + entries: list[dict[str, Any]] = json.loads( + _run_mvm(runner_vm, "vm", "ls", "--json").stdout + ) + return next((entry for entry in entries if entry.get("name") == vm_name), None) + + +def _read(runner_vm: str, path: str) -> str: + return _guest_run(runner_vm, f"cat {shlex.quote(path)}").stdout.strip() + + +class TestVMCgroupEnvelope: + def test_running_vm_envelope_and_lifecycle_cleanup(self, runner_vm: str) -> None: + vm_name = f"sys-cgroup-{uuid.uuid4().hex[:8]}" + net_name = f"sys-cgroup-net-{uuid.uuid4().hex[:6]}" + config_keys = { + "cgroup_vmm_headroom_mib": "128", + "cgroup_cpu_weight": "100", + "cgroup_pids_max": "256", + "cgroup_swap_max_bytes": "0", + } + originals: dict[str, str | None] = {} + vm_id = "" + ensure_vm_deps(runner_vm) + controller_probe = _guest_run( + runner_vm, "cat /sys/fs/cgroup/cgroup.controllers", check=False + ) + controllers = ( + set(controller_probe.stdout.split()) + if controller_probe.returncode == 0 + else set() + ) + missing = {"cpu", "memory", "pids"} - controllers + _run_mvm( + runner_vm, + "network", + "create", + net_name, + "--subnet", + _unique_subnet(net_name), + "--non-interactive", + ) + try: + for key, value in config_keys.items(): + current = _run_mvm( + runner_vm, "config", "get", "defaults.vm", key, check=False + ) + originals[key] = ( + current.stdout.strip() + if current.returncode == 0 and current.stdout.strip() + else None + ) + _run_mvm( + runner_vm, "config", "set", "defaults.vm", key, value + ) + + created = _run_mvm( + runner_vm, + "vm", + "create", + vm_name, + "--image", + "alpine:3.23", + "--network", + net_name, + "--vcpu", + "2", + "--mem", + "512", + check=False, + timeout=120, + ) + if missing: + assert created.returncode != 0 + combined = (created.stdout + created.stderr).lower() + assert "cgroup" in combined + assert "controller" in combined or "v2" in combined or "unified" in combined + assert _vm_entry(runner_vm, vm_name) is None + return + + assert created.returncode == 0, created.stderr + vm = _vm_entry(runner_vm, vm_name) + assert vm is not None and vm["status"] == "running" + vm_id = vm["id"] + pid = int(vm["pid"]) + relative = f"/mvmctl/firecracker/{vm_id}" + leaf = f"/sys/fs/cgroup{relative}" + + membership = _read(runner_vm, f"/proc/{pid}/cgroup").splitlines() + assert f"0::{relative}" in membership + assert str(pid) in _read(runner_vm, f"{leaf}/cgroup.procs").split() + + inspected = json.loads( + _run_mvm( + runner_vm, "vm", "inspect", vm_name, "--json" + ).stdout + ) + cgroup = inspected["resources"]["cgroup"] + requested = cgroup["requested"] + actual = cgroup["actual"] + assert cgroup["path"] == leaf + assert cgroup["status"] == "enforced" + assert cgroup.get("mismatches", []) == [] + assert requested == actual + assert requested == { + "policy_version": 1, + "cpu_quota_micros": 200000, + "cpu_period_micros": 100000, + "cpu_weight": 100, + "memory_high_bytes": 640 * 1024 * 1024, + "memory_max_bytes": 640 * 1024 * 1024, + "swap_max_bytes": 0, + "pids_max": 256, + } + + assert _read(runner_vm, f"{leaf}/cpu.max") == "200000 100000" + assert _read(runner_vm, f"{leaf}/cpu.weight") == "100" + assert _read(runner_vm, f"{leaf}/memory.high") == str(640 * 1024 * 1024) + assert _read(runner_vm, f"{leaf}/memory.max") == str(640 * 1024 * 1024) + assert _read(runner_vm, f"{leaf}/memory.swap.max") == "0" + assert _read(runner_vm, f"{leaf}/pids.max") == "256" + for key in ( + "cpu_quota_micros", + "cpu_period_micros", + "cpu_weight", + "memory_high_bytes", + "memory_max_bytes", + "pids_max", + ): + assert int(actual[key]) > 0 + assert actual["swap_max_bytes"] == 0 + + _run_mvm(runner_vm, "vm", "stop", vm_name, timeout=120) + assert _guest_run( + runner_vm, f"test ! -e {shlex.quote(leaf)}", check=False + ).returncode == 0 + _run_mvm(runner_vm, "vm", "start", vm_name, timeout=120) + assert _guest_run( + runner_vm, f"test -d {shlex.quote(leaf)}", check=False + ).returncode == 0 + _run_mvm(runner_vm, "vm", "rm", vm_name, "--force", timeout=120) + assert _guest_run( + runner_vm, f"test ! -e {shlex.quote(leaf)}", check=False + ).returncode == 0 + assert _vm_entry(runner_vm, vm_name) is None + finally: + _run_mvm( + runner_vm, "vm", "rm", vm_name, "--force", check=False, timeout=120 + ) + if vm_id: + leaf = f"/sys/fs/cgroup/mvmctl/firecracker/{vm_id}" + assert _guest_run( + runner_vm, f"test ! -e {shlex.quote(leaf)}", check=False + ).returncode == 0 + _run_mvm( + runner_vm, "network", "rm", net_name, "--force", check=False + ) + for key, value in originals.items(): + if value is None: + _run_mvm( + runner_vm, + "config", + "reset", + "defaults.vm", + key, + check=False, + ) + else: + _run_mvm( + runner_vm, + "config", + "set", + "defaults.vm", + key, + value, + check=False, + ) diff --git a/tests/system/vm/test_jailer.py b/tests/system/vm/test_jailer.py index f9ff2c3e..91398673 100644 --- a/tests/system/vm/test_jailer.py +++ b/tests/system/vm/test_jailer.py @@ -154,6 +154,13 @@ def test_volume_and_snapshot_parity( volume_name = f"sys-jailer-vol-{uuid.uuid4().hex[:6]}" snapshot_id: str | None = None try: + source_inspect = json.loads( + _run_mvm(runner_vm, "vm", "inspect", vm_name, "--json").stdout + ) + source_cgroup = source_inspect["resources"]["cgroup"] + assert source_cgroup["status"] == "enforced" + source_limits = source_cgroup["requested"] + _run_mvm(runner_vm, "volume", "create", volume_name, "64M") _run_mvm(runner_vm, "vm", "stop", vm_name) _run_mvm(runner_vm, "volume", "attach", vm_name, volume_name) @@ -190,6 +197,13 @@ def test_volume_and_snapshot_parity( assert restored.returncode == 0, restored.stderr restored_vm = _vm_entry(runner_vm, vm_name) assert restored_vm["status"] == "running" + restored_inspect = json.loads( + _run_mvm(runner_vm, "vm", "inspect", vm_name, "--json").stdout + ) + restored_cgroup = restored_inspect["resources"]["cgroup"] + assert restored_cgroup["status"] == "enforced" + assert restored_cgroup["requested"] == source_limits + assert restored_cgroup["actual"] == source_limits assert _guest_run( runner_vm, f"test -d {shlex.quote(_jail_root(restored_vm['id']))}", check=False, From bd7a0f4a9a0057abd6d0b9d455b9cfb318e70835 Mon Sep 17 00:00:00 2001 From: AlanD20 Date: Sun, 9 Aug 2026 14:51:43 +0200 Subject: [PATCH 0004/1457] feat(network): enforce routed service policies --- CHANGELOG.md | 3 + CONTEXT.md | 4 +- docs/DEPENDENCIES.md | 1 + docs/REFERENCES.md | 17 + docs/RUNTIME.md | 18 +- docs/STANDARDS.md | 2 +- .../0014-routed-service-access-policies.md | 5 +- .../HOW_AGENTS_WRITE_SYSTEM_TESTS.md | 1 + docs/mvm.1 | 16 + docs/optimizations/network-sync-atomicity.md | 21 +- docs/roadmaps/TENANT_MICROVM.md | 10 +- docs/system-test-architecture.md | 4 +- internal/cli/completion.go | 36 ++ internal/cli/policy.go | 172 +++++++++ internal/cli/policy_test.go | 113 ++++++ internal/cli/root.go | 1 + internal/core/network/constants.go | 2 + internal/core/network/service.go | 45 ++- .../core/network/service_access_policy.go | 91 +++++ .../network/service_access_policy_compiler.go | 110 ++++++ .../service_access_policy_repository.go | 20 ++ .../network/service_access_policy_sqlite.go | 118 +++++++ .../service_access_policy_sqlite_test.go | 146 ++++++++ .../network/service_access_policy_test.go | 157 +++++++++ internal/core/network/service_test.go | 56 +-- internal/infra/constants.go | 4 + .../004_service_access_policies.sql | 120 +++++++ internal/lib/db/migrations_jailer_test.go | 80 ++--- internal/lib/db/migrations_policy_test.go | 94 +++++ internal/lib/firewall/iptables.go | 143 ++++++-- internal/lib/firewall/iptables_repository.go | 65 +++- internal/lib/firewall/iptables_test.go | 54 +++ internal/lib/firewall/nftables.go | 83 ++++- internal/lib/firewall/nftables_repository.go | 62 +++- internal/lib/firewall/nftables_test.go | 14 + internal/lib/firewall/tracker.go | 41 +++ internal/lib/firewall/tracker_test.go | 54 ++- internal/lib/model/network.go | 20 +- internal/lib/model/network_test.go | 47 +++ internal/lib/model/service_access_policy.go | 34 ++ internal/testutil/mock_api.go | 46 +++ pkg/api/host.go | 4 + pkg/api/inputs/policy.go | 147 ++++++++ pkg/api/inputs/policy_test.go | 152 ++++++++ pkg/api/interfaces.go | 1 + pkg/api/network.go | 93 ++++- pkg/api/operation.go | 4 +- pkg/api/policy.go | 245 +++++++++++++ pkg/api/policy_test.go | 190 ++++++++++ pkg/api/results/policy.go | 24 ++ pkg/api/vm.go | 14 + pkg/errs/codes.go | 12 + pkg/errs/domain.go | 24 +- scripts/run-system-tests.py | 1 + tests/system/COVERAGE_MATRIX.md | 1 + tests/system/network/test_policies.py | 327 ++++++++++++++++++ 56 files changed, 3183 insertions(+), 186 deletions(-) create mode 100644 internal/cli/policy.go create mode 100644 internal/cli/policy_test.go create mode 100644 internal/core/network/service_access_policy.go create mode 100644 internal/core/network/service_access_policy_compiler.go create mode 100644 internal/core/network/service_access_policy_repository.go create mode 100644 internal/core/network/service_access_policy_sqlite.go create mode 100644 internal/core/network/service_access_policy_sqlite_test.go create mode 100644 internal/core/network/service_access_policy_test.go create mode 100644 internal/lib/db/migrations/004_service_access_policies.sql create mode 100644 internal/lib/db/migrations_policy_test.go create mode 100644 internal/lib/model/network_test.go create mode 100644 internal/lib/model/service_access_policy.go create mode 100644 pkg/api/inputs/policy.go create mode 100644 pkg/api/inputs/policy_test.go create mode 100644 pkg/api/policy.go create mode 100644 pkg/api/policy_test.go create mode 100644 pkg/api/results/policy.go create mode 100644 tests/system/network/test_policies.py diff --git a/CHANGELOG.md b/CHANGELOG.md index b81866a3..894fe380 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,6 +16,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 #### Enforced VM resource envelopes - Jailed VMs now require cgroup v2 and launch with persisted, verified CPU, memory, swap, and PID limits; `mvm vm inspect` reports requested, observed, and current cgroup state. +#### Routed service-access policies +- `mvm policy` now persists exact network-to-VM TCP/UDP allows while both firewall backends default-deny other managed cross-network and VM-to-host traffic without blocking NoCloud, established replies, or internet egress. + ### Fixed #### `mvm image inspect` diff --git a/CONTEXT.md b/CONTEXT.md index 4260ba19..a7e13cca 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -80,13 +80,13 @@ A struct bound to a single entity instance. Manages lifecycle state transitions Constructed with the entity + its Repository: `vm.NewController(vm *model.VMItem, repo Repository)`. NOT wired at startup -- created per-operation in the Service layer. -*Example: `vm.NewController(vmItem, repo).Snapshot(ctx)` -- snapshots this specific VM. `network.NewService(repo, tracker).RemoveBridge(ctx, bridge)` -- removes a bridge, no single network entity needed.* +*Example: `vm.NewController(vmItem, repo).Snapshot(ctx)` -- snapshots this specific VM. `network.NewService(repo, tracker, policyRepo).RemoveBridge(ctx, bridge)` -- removes a bridge, no single network entity needed.* ### Service (stateless, intra-domain) A struct for stateless intra-domain operations. Handles infrastructure operations (bridges, TAPs, NAT, subprocesses, file/disk operations). Performs state detection (checking current system state as part of an operation -- "does this bridge exist?" to branch execution). Guards invariants that protect against system damage. Does NOT validate caller input. Does NOT manage state for a single entity -- Service operates on infrastructure, not on a bound instance. -Constructed with repos/options only: `network.NewService(repo Repository, tracker *firewall.FirewallTracker)`. Wired once at startup in `app.Initialize()`. +Constructed with repos/options only: `network.NewService(repo Repository, tracker *firewall.FirewallTracker, policyRepo ServiceAccessPolicyRepository)`. Wired once at startup in `app.Initialize()`. *Litmus test: if the operation would work the same way without a specific entity instance, it's Service. If it needs to communicate with a running entity's Firecracker API socket, it belongs in Controller. If it sequences multiple infrastructure steps (teardown NAT -> remove bridge -> delete DB record), it's intra-domain orchestration in Service.* diff --git a/docs/DEPENDENCIES.md b/docs/DEPENDENCIES.md index c053aac5..97b56f74 100644 --- a/docs/DEPENDENCIES.md +++ b/docs/DEPENDENCIES.md @@ -189,6 +189,7 @@ Only needed for `mvm kernel pull --type official --clean-build` (building the of | `mvm host reset` | `sudo`, `groupdel`, `sysctl` | | `mvm network create` | `ip`, `iptables`/`nft` | | `mvm network rm` / `sync` | `ip`, `iptables`/`nft` | +| `mvm policy create/rm/sync` | `iptables`/`nft` | | `mvm image pull` | `qemu-img` (may trigger conversion), `fakeroot` (tar-rootfs) | | `mvm image import` | `qemu-img`, `sfdisk`, `blkid`, `mount`, `umount`, `tar`, `fakeroot` (tar-rootfs), `truncate`, `mkfs.ext4`, `unsquashfs`, `dumpe2fs`, `du`, `dd` | | `mvm kernel pull --type official` | `make`, `gcc`, `ld`, `flex`, `bison`, `bc`, `pahole`, `git`, `curl`, `pkg-config` | diff --git a/docs/REFERENCES.md b/docs/REFERENCES.md index 371d43a1..3631a87b 100644 --- a/docs/REFERENCES.md +++ b/docs/REFERENCES.md @@ -17,6 +17,7 @@ This document provides detailed reference for all `mvm` commands, configuration - [mvm snapshot](#mvm-snapshot) - [mvm console](#mvm-console) - [mvm network](#mvm-network) +- [mvm policy](#mvm-policy) - [mvm key](#mvm-key) - [mvm config](#mvm-config) - [mvm cache](#mvm-cache) @@ -394,6 +395,22 @@ The network resolver tries selectors in priority order: --- +### `mvm policy` + +Persisted routed access from one source network to one exact destination VM service. Same-network policies are rejected because same-bridge filtering is not implemented. + +| Command | Flags | Description | +|---------|-------|-------------| +| `mvm policy create` | `SOURCE_NETWORK DESTINATION_VM tcp\|udp PORT\|START-END` | Allow one routed destination service or bounded port range | +| `mvm policy ls` | `--json` | List policies with current resource names | +| `mvm policy inspect` | `POLICY_ID`, `--json` | Inspect one policy by exact ID or unambiguous prefix | +| `mvm policy rm` | `POLICY_IDS...`, `-f, --force` | Remove one or more policies and reconcile immediately | +| `mvm policy sync` | `--json` | Resolve current identities and atomically reconcile policy rules | + +Policies default-deny other routed traffic between managed networks. NoCloud and established replies precede managed VM-to-host default deny; internet egress remains available. + +--- + ### `mvm key` SSH key management. diff --git a/docs/RUNTIME.md b/docs/RUNTIME.md index aff2653e..ecaa6509 100644 --- a/docs/RUNTIME.md +++ b/docs/RUNTIME.md @@ -438,8 +438,8 @@ There is no direct-Firecracker fallback. Every launch requires cgroup v2 with th ## 4. Firewall Backends -Firewall backends manage NAT rules, forwarding rules, and per-VM access control -for nocloud-net servers. Two backends are available, selected by the +Firewall backends manage NAT rules, forwarding rules, routed service-access +policies, and per-VM access control for nocloud-net servers. Two backends are available, selected by the `firewall_backend` setting in the database (default: `nftables`). A **`FirewallTracker`** in `internal/lib/firewall/tracker.go` provides a unified @@ -454,12 +454,22 @@ resolves the actual backend setting, it replaces the tracker via | **nftables** | Yes (`firewall_backend: nftables`) | `tracker.go`, `nftables.go`, `nftables_repository.go` | | **iptables** | Opt-in (`firewall_backend: iptables`) | `tracker.go`, `iptables.go`, `iptables_repository.go` | -Both backends manage the same three chain types: `MVM-FORWARD` (ip filter), -`MVM-POSTROUTING` (ip nat), and `MVM-NOCLOUDNET-INPUT` (ip filter). The nftables +Both backends manage five chain types: `MVM-FORWARD` (ip filter), +`MVM-POSTROUTING` (ip nat), `MVM-NOCLOUDNET-INPUT` (ip filter), +`MVM-ROUTED-POLICY` (ip filter), and `MVM-HOST-INPUT` (ip filter). The nftables backend adds jump rules at position 0 of the built-in chains, using non-hook chains so that `accept` verdicts are terminal within the mvm table — matching the behavior users expect from iptables. +Service-access intent stores a source network ID, exact destination VM ID, +TCP or UDP, and a destination port or bounded range. The API resolves current +network, bridge, VM, and IP state; the network domain compiles backend-neutral +derived rules. Reconciliation commits desired rows before atomically rebuilding +the active backend. `MVM-ROUTED-POLICY` orders established replies, exact allows, +then managed-bridge default deny. `MVM-NOCLOUDNET-INPUT` precedes +`MVM-HOST-INPUT`, which permits established replies and denies other traffic +from managed bridges. Internet egress does not match the managed-output drop. + **Why two backends?** Different Linux distributions ship with different firewall defaults. Some use nftables natively; others use iptables or iptables-legacy. mvmctl supports both to avoid conflicts with the host's existing firewall diff --git a/docs/STANDARDS.md b/docs/STANDARDS.md index d7b89b4f..66fa0c6f 100644 --- a/docs/STANDARDS.md +++ b/docs/STANDARDS.md @@ -95,7 +95,7 @@ Three-layer flow: **CLI → API → Core** - Performs state detection (checking current system state to branch execution) - Guards invariants that protect against system damage - Does NOT validate caller input -- Constructor takes repos/options only: `network.NewService(repo Repository, tracker *firewall.FirewallTracker)` +- Constructor takes repos/options only: `network.NewService(repo Repository, tracker *firewall.FirewallTracker, policyRepo ServiceAccessPolicyRepository)` - Wired once at startup in `app.Initialize()` - Service methods that don't reference the Service struct live in `utils.go` diff --git a/docs/adr/0014-routed-service-access-policies.md b/docs/adr/0014-routed-service-access-policies.md index e366dc85..65a16ed6 100644 --- a/docs/adr/0014-routed-service-access-policies.md +++ b/docs/adr/0014-routed-service-access-policies.md @@ -6,7 +6,7 @@ ## Context -mvmctl networks are independent Linux bridges and subnets. VMs on one bridge communicate directly at layer 2, while traffic between bridges is routed by the host because IPv4 forwarding is enabled. The managed firewall currently permits outbound NAT and installs no default deny between mvmctl bridges. A workload network can therefore reach a shared-infrastructure network unless the host firewall blocks it. +mvmctl networks are independent Linux bridges and subnets. VMs on one bridge communicate directly at layer 2, while traffic between bridges is routed by the host because IPv4 forwarding is enabled. Before this decision was implemented, the managed firewall permitted outbound NAT and installed no default deny between mvmctl bridges. Project orchestration remains outside mvmctl. The infrastructure requirement is narrower: permit a source mvmctl network to reach one exact destination VM service, such as TCP port 5432 on a shared PostgreSQL VM, while denying other routed traffic between managed networks. The policy must survive VM restarts and firewall reconciliation without exposing raw nftables or iptables syntax. @@ -25,6 +25,8 @@ mvmctl will persist a typed **service-access policy** identified by source netwo - Future same-network VM-to-VM policies will add a typed source-VM operation and compile to nftables bridge-family rules. Existing policies, commands, and stored records will not change. - No raw firewall-rule editor, generic selector registry, or silent backend fallback will be introduced. +The implemented IP-family compiler uses `MVM-ROUTED-POLICY` and `MVM-HOST-INPUT`. Both backends place connection-state handling first, explicit service allows next, and an internally derived terminal drop last. Managed-interface matching is semantic in persisted derived state and expands to the backend's interface-prefix syntax only when rules are rendered. The initial host exception set contains NoCloud; additional host services require a future typed operation rather than raw INPUT rules. + ## Consequences **Benefits:** @@ -40,6 +42,7 @@ mvmctl will persist a typed **service-access policy** identified by source netwo - Rule ordering becomes a correctness requirement. Service allows and connection-state handling must always precede default deny. - Removing or recreating a VM or network invalidates identity-bound policies; policies do not silently attach to replacement resources. - VM-to-host restrictions require careful NoCloud and host-service system tests. +- The initial host-input policy permits NoCloud and established replies but no user-defined host services. - Same-network filtering remains open until the bridge-family compiler is implemented. When implemented, that compiler will require nftables; routed policies retain iptables parity. ## Related Decisions diff --git a/docs/development/HOW_AGENTS_WRITE_SYSTEM_TESTS.md b/docs/development/HOW_AGENTS_WRITE_SYSTEM_TESTS.md index 34bb805c..6976928f 100644 --- a/docs/development/HOW_AGENTS_WRITE_SYSTEM_TESTS.md +++ b/docs/development/HOW_AGENTS_WRITE_SYSTEM_TESTS.md @@ -312,6 +312,7 @@ Every CLI command and flag that a user can invoke must be tested. This table cla | `network rm ` | **L2** | Multiple bridge cleanup | | `network default` / `network default ` | L1 | DB update | | `network sync` / `network sync --json` | **L2** | Real bridge + firewall recreation | +| `policy create/ls/inspect/rm/sync` | **L2** | Cross-network connectivity, exact destination/port allow, host/default deny, backend parity | | Sync after bridge deletion | **L2** | Real bridge deletion + recovery | ### `mvm volume` diff --git a/docs/mvm.1 b/docs/mvm.1 index 43a07b2a..67485b25 100644 --- a/docs/mvm.1 +++ b/docs/mvm.1 @@ -215,6 +215,22 @@ Set a network as the default for VM creation. .TP .BR network\ sync Sync firewall rules between database and host. +.SS Routed Service-Access Policies +.TP +.BR policy\ create " SOURCE_NETWORK DESTINATION_VM tcp|udp PORT|START-END" +Allow routed access to one exact destination VM service. +.TP +.BR policy\ ls +List persisted service-access policies. Use --json for structured output. +.TP +.BR policy\ inspect " POLICY_ID" +Show one policy by exact ID or unambiguous prefix. +.TP +.BR policy\ rm " POLICY_IDS..." +Remove policies and reconcile firewall state. Use -f to skip confirmation. +.TP +.BR policy\ sync +Resolve current resource identities and atomically reconcile policy rules. .SS SSH Key Management .TP .BR key\ ls diff --git a/docs/optimizations/network-sync-atomicity.md b/docs/optimizations/network-sync-atomicity.md index 244136ca..5b072567 100644 --- a/docs/optimizations/network-sync-atomicity.md +++ b/docs/optimizations/network-sync-atomicity.md @@ -46,14 +46,17 @@ if bridgeActive != net.BridgeActive { The core sync logic lives in `SyncIPTablesRulesBatch()` at `internal/core/network/service.go`: -1. **Collect active DB rules for all networks** that will participate in the rebuild. If the caller requested specific networks, every DB network is still included so that unrequested networks' rules are preserved in the kernel. Stats are returned only for the requested networks. -2. **Pre-check kernel presence** for each rule via `s.firewallTracker.RuleExists()` so that `added` and `verified` reflect actual kernel state, not just DB state. -3. **Batch-ensure all rules** in a single `BatchEnsureRules()` call — all active rules for all networks are queued and flushed atomically. -4. **Count orphaned host rules** per network that reference the network but have no matching active DB record (informational only — orphans are not removed). +1. **Compile and commit policy-derived rows** from current API-resolved network and VM identities. +2. **Collect active DB rules for all networks** that will participate in the rebuild. If the caller requested specific networks, every DB network is still included so that unrequested networks' rules are preserved in the kernel. Stats are returned only for the requested networks. +3. **Pre-check kernel presence** for each rule via `s.firewallTracker.RuleExists()` so that `added` and `verified` reflect actual kernel state, not just DB state. +4. **Batch-ensure all rules** in a single `BatchEnsureRules()` call — all active rules for all networks are queued and flushed atomically. +5. **Count orphaned host rules** per network that reference the network but have no matching active DB record (informational only — orphans are not removed). **Code reference:** `internal/core/network/service.go` — `SyncIPTablesRulesBatch()` ```go +policyRules := CompileServiceAccessPolicyRules(policies, networks) +err := s.firewallTracker.ReplacePolicyRules(ctx, policyRules) for _, net := range networks { dbRules, err := s.firewallTracker.GetByNetworkID(ctx, net.ID, true) // ... @@ -67,13 +70,13 @@ result := s.firewallTracker.BatchEnsureRules(ctx, allRules) ## Backend-Specific Atomicity Mechanisms -The `FirewallTracker` at `internal/lib/firewall/tracker.go` dispatches to one of two backends. During sync, `SyncIPTablesRulesBatch()` builds the union of all active DB rules and invokes `BatchEnsureRules()` once per sync invocation. The backend flushes the MVM chains and rebuilds them from that complete rule set. +The `FirewallTracker` at `internal/lib/firewall/tracker.go` dispatches to one of two backends. During sync, `SyncIPTablesRulesBatch()` replaces policy-derived rows, builds the union of all active DB rules, and invokes `BatchEnsureRules()` once per sync invocation. The backend flushes the MVM chains and rebuilds them from that complete rule set. ### nftables (default) — Atomic Batch via `nft -f -` When inside a batch context, `EnsureRule` appends to an internal slice. On context exit, `flushBatch()` calls `NFTablesTracker.BatchEnsureRules()` which: -1. **Flushes MVM custom chains** (MVM-FORWARD, MVM-POSTROUTING, MVM-NOCLOUDNET-INPUT) — removes all existing rules. +1. **Flushes MVM custom chains** (MVM-FORWARD, MVM-POSTROUTING, MVM-NOCLOUDNET-INPUT, MVM-ROUTED-POLICY, MVM-HOST-INPUT) — removes all existing rules. 2. **Inserts conntrack accept rules** at position 0 of filter chains — preserves established connections. 3. **Adds all DB rules** as `add rule ip ` statements. 4. **Pipes** the complete script to `nft -f -`. @@ -146,7 +149,7 @@ This means: The sync uses a **replacement-style** approach within MVM custom chains: -- **MVM custom chains** (MVM-FORWARD, MVM-POSTROUTING, MVM-NOCLOUDNET-INPUT) are flushed and all active DB rules are re-added atomically. +- **MVM custom chains** (MVM-FORWARD, MVM-POSTROUTING, MVM-NOCLOUDNET-INPUT, MVM-ROUTED-POLICY, MVM-HOST-INPUT) are flushed and all active DB rules are re-added atomically. - **Built-in chains** (FORWARD, POSTROUTING, INPUT) are never touched — their jump-to-MVM-chain rules survive untouched. - **Conntrack accept rule** is inserted at position 0 of filter MVM chains, preserving established connections through the atomic swap (both backends). - **Orphaned host rules** within MVM chains are **automatically removed** by the flush-and-rebuild cycle. Orphans outside MVM chains (in built-in chains) are reported but **not removed**, guaranteeing zero disruption to third-party firewall rules. @@ -155,14 +158,18 @@ The sync uses a **replacement-style** approach within MVM custom chains: ``` Built-in chains (never flushed): + FORWARD ──[first MVM jump]────> MVM-ROUTED-POLICY FORWARD ──[jump position 0]──> MVM-FORWARD POSTROUTING ──[jump position 0]──> MVM-POSTROUTING INPUT ──[jump position 0]──> MVM-NOCLOUDNET-INPUT + INPUT ──[next MVM jump]─────> MVM-HOST-INPUT MVM custom chains (flushed and rebuilt from DB during sync): MVM-FORWARD (ip filter) — FORWARD accept rules per TAP/NAT MVM-POSTROUTING (ip nat) — MASQUERADE rules per gateway MVM-NOCLOUDNET-INPUT (ip filter) — nocloud-net accept rules + MVM-ROUTED-POLICY (ip filter) — service allows, then managed-network deny + MVM-HOST-INPUT (ip filter) — managed-network host-input deny ``` The jump rules at position 0 are established once by `NFTablesTracker.Initialize()` (or `IPTablesTracker.Initialize()`) and are never modified during sync. This guarantees that MVM rules are always evaluated before any third-party rules (UFW, etc.) without needing to touch the system's built-in chains. diff --git a/docs/roadmaps/TENANT_MICROVM.md b/docs/roadmaps/TENANT_MICROVM.md index 686722bd..75cfb013 100644 --- a/docs/roadmaps/TENANT_MICROVM.md +++ b/docs/roadmaps/TENANT_MICROVM.md @@ -26,8 +26,8 @@ Direct inbound ports are not required when the workload VM runs an outbound tunn | Capability | Current state | Required change | |---|---|---| | Same-network communication | Works through one bridge; intentionally unfiltered | None for the environment trust-zone model | -| Cross-network routing | Host forwarding and ACCEPT policy permit it | Persist explicit service allows and default-deny other managed inter-network traffic | -| VM-to-host traffic | INPUT remains open except for explicit NoCloud allow rules | Preserve required host services, then deny other traffic from managed VM networks | +| Cross-network routing | Typed source-network → destination-VM service policies and managed-network default deny are implemented for nftables and iptables; L2 execution remains pending | Complete prepared-runner verification and benchmark reconciliation at scale | +| VM-to-host traffic | Managed-network INPUT is default-deny after established replies and NoCloud | Add a typed host-service operation only when a concrete host service must be exposed | | Firecracker launch | Canonical Jailer launch, exact trusted release pairing, and cgroup-v2 enforcement implemented; L2 execution remains pending | Complete prepared-runner verification | | Host resource enforcement | Typed CPU, memory, swap, and PID limits are persisted, applied through Jailer, verified after launch, and shown by inspect | Measure VMM headroom under L2 workloads and add I/O limits after stable backing-device resolution exists | | IP identity | Explicit IP assignment and collision-safe IPAM exist | Policies bind to network/VM IDs and resolve current addressing during reconciliation | @@ -58,7 +58,7 @@ Direct inbound ports are not required when the workload VM runs an outbound tunn **Exit:** implementation and hermetic coverage are complete. Prepared-runner L2 execution must still prove that every normally launched VM has verified constraints and that stop/remove clears its leaf cgroup. -### M3 — Routed service-access policies +### M3 — Routed service-access policies (implemented; L2 execution pending) - Persist source-network → destination-VM → protocol/port intent in the network domain. - Resolve cross-domain identities in the API layer. @@ -67,7 +67,7 @@ Direct inbound ports are not required when the workload VM runs an outbound tunn - Restrict VM-to-host INPUT while preserving NoCloud and explicitly approved host services. - Reconcile desired rules after reboot, UFW reload, resource deletion, and `network sync`. -**Exit:** an environment network can reach an exact shared service port, cannot reach other ports or managed networks, retains internet egress, and cannot reach unauthorized host services. +**Exit:** implementation and hermetic coverage are complete for both firewall backends. Prepared-runner L2 execution must still prove that an environment network can reach an exact shared service port, cannot reach other ports or managed networks, retains internet egress, and cannot reach unauthorized host services. ### M4 — Workload proof and recovery @@ -91,7 +91,7 @@ Direct inbound ports are not required when the workload VM runs an outbound tunn The highest-value mvmctl additions are infrastructure diagnostics, not project orchestration: -1. Policy list, inspect, remove, sync, and connectivity-check commands. +1. Policy create, list, inspect, remove, and sync commands. A dedicated connectivity-check command remains optional. 2. VM inspection showing Jailer pair, jail path, cgroup enforcement, and current usage. 3. Host readiness output for Jailer, trusted binaries, cgroup v2, KVM, and required controllers. 4. Desired-versus-installed firewall rule diagnostics. diff --git a/docs/system-test-architecture.md b/docs/system-test-architecture.md index d87c8583..3ae6150a 100644 --- a/docs/system-test-architecture.md +++ b/docs/system-test-architecture.md @@ -112,7 +112,7 @@ All T1/T2 runner VMs are Firecracker microVMs created with **Firecracker v1.16** | Tier | Runs inside VM | Shared volume | Nested KVM | Runner VM kernel | What's tested | |------|---------------|---------------|------------|-----------------|---------------| | **1** | Yes — `mvm exec -- pytest ...` | Yes (mounted at `/mnt`) | Not needed | `official:7.0.11` (nftables + tuntap + kvm + btrfs) | Host-level CLI: help, config, init, cache, keys, invariants, bin, images, kernel, network, host, run | -| **2** | Yes — `mvm exec -- pytest ...` | Yes (mounted at `/mnt`) | Yes | `official:7.0.11` (nftables + tuntap + kvm + btrfs) | VM lifecycle: volume, vm_lifecycle, vm_data_persistence, Jailer/cgroups, ssh, console, exec, logs, full_journeys, nftables | +| **2** | Yes — `mvm exec -- pytest ...` | Yes (mounted at `/mnt`) | Yes | `official:7.0.11` (nftables + tuntap + kvm + btrfs) | VM lifecycle: volume, vm_lifecycle, vm_data_persistence, Jailer/cgroups, routed policies, ssh, console, exec, logs, full_journeys, nftables | | **3** | No — runs directly on host | Host mirror | Yes (host KVM) | N/A (host-direct) | vm_fresh_env, vm_nested_isolated, vm_snapshot_load, vm_snapshot_rootfs, volume_hotplug, cp, kernel_build, env | --- @@ -436,6 +436,7 @@ tests/system/ | kernel | 1 | `test_kernel_import.py` | Kernel import — Tier 1 | | network | 1 | `test_network.py` | Network creation (TestNetworkEdgeCases class marked `tier2`) | | nftables | 2 | `test_nftables.py` | Nftables firewall backend and atomic rule sync — needs nested virt | +| policies | 2 | `test_policies.py` | Routed service-access allow/default-deny behavior for nftables and iptables — needs nested virt | | host | 1 | `test_host.py` | Host info/status | | run | 1 | `test_run.py` | Service subprocesses | | volume | 2 | `test_volume.py` | Volume lifecycle | @@ -597,4 +598,3 @@ Every test file in `tests/system/` MUST: | **Limits** | `--workers N` flag controls max parallel VMs. Default: 4. | --- - diff --git a/internal/cli/completion.go b/internal/cli/completion.go index aca429d6..69e33926 100644 --- a/internal/cli/completion.go +++ b/internal/cli/completion.go @@ -34,6 +34,42 @@ func completeNetworkNames(cmd *cobra.Command, args []string, toComplete string) return results, cobra.ShellCompDirectiveNoFileComp } +// completePolicyIDs completes service-access policy IDs. +func completePolicyIDs(cmd *cobra.Command, args []string, toComplete string) ([]string, cobra.ShellCompDirective) { + if opRef == nil { + return nil, cobra.ShellCompDirectiveNoFileComp + } + policies, err := opRef.PolicyList(cmd.Context()) + if err != nil { + return nil, cobra.ShellCompDirectiveNoFileComp + } + var results []string + for _, policy := range policies { + short := crypto.Truncate(policy.ID, 12) + if strings.HasPrefix(short, toComplete) && !slices.Contains(results, short) { + results = append(results, short) + } + } + return results, cobra.ShellCompDirectiveNoFileComp +} + +// completePolicyCreate completes source network, destination VM, and protocol positionally. +func completePolicyCreate(cmd *cobra.Command, args []string, toComplete string) ([]string, cobra.ShellCompDirective) { + switch len(args) { + case 0: + return completeNetworkNames(cmd, args, toComplete) + case 1: + return completeVMNames(cmd, args, toComplete) + case 2: + protocols := []string{"tcp", "udp"} + return slices.DeleteFunc(protocols, func(protocol string) bool { + return !strings.HasPrefix(protocol, toComplete) + }), cobra.ShellCompDirectiveNoFileComp + default: + return nil, cobra.ShellCompDirectiveNoFileComp + } +} + // completeImageIDs completes with local image types and short IDs. func completeImageIDs(cmd *cobra.Command, args []string, toComplete string) ([]string, cobra.ShellCompDirective) { if opRef == nil { diff --git a/internal/cli/policy.go b/internal/cli/policy.go new file mode 100644 index 00000000..db975bf9 --- /dev/null +++ b/internal/cli/policy.go @@ -0,0 +1,172 @@ +package cli + +import ( + "encoding/json" + "fmt" + + "mvmctl/internal/cli/common" + "mvmctl/pkg/api" + "mvmctl/pkg/api/inputs" + "mvmctl/pkg/api/results" + + "github.com/spf13/cobra" +) + +var policyColumns = []common.ListingColumn{ + {Header: "ID", Extract: func(v any) string { return common.Cli.FormatID(v.(*results.Policy).ID) }}, + {Header: "Source Network", Extract: func(v any) string { return v.(*results.Policy).SourceNetworkName }}, + {Header: "Destination VM", Extract: func(v any) string { return v.(*results.Policy).DestinationVMName }}, + {Header: "Protocol", Extract: func(v any) string { return string(v.(*results.Policy).Protocol) }}, + {Header: "Destination Port", Extract: func(v any) string { + policy := v.(*results.Policy) + if policy.DestinationPortStart == policy.DestinationPortEnd { + return fmt.Sprintf("%d", policy.DestinationPortStart) + } + return fmt.Sprintf("%d-%d", policy.DestinationPortStart, policy.DestinationPortEnd) + }}, +} + +// NewPolicyCmd creates the top-level routed service-access policy command. +func NewPolicyCmd(policyAPI api.PolicyAPI) *cobra.Command { + cmd := &cobra.Command{Use: "policy", Short: "Routed service-access policy management"} + cmd.AddCommand(newPolicyCreateCmd(policyAPI)) + cmd.AddCommand(newPolicyListCmd(policyAPI)) + cmd.AddCommand(newPolicyInspectCmd(policyAPI)) + cmd.AddCommand(newPolicyRemoveCmd(policyAPI)) + cmd.AddCommand(newPolicySyncCmd(policyAPI)) + return cmd +} + +func newPolicyCreateCmd(policyAPI api.PolicyAPI) *cobra.Command { + return &cobra.Command{ + Use: "create [source-network] [destination-vm] [tcp|udp] [destination-port]", + Short: "Allow routed access from a network to one exact VM service.", + Args: cobra.ExactArgs(4), + ValidArgsFunction: completePolicyCreate, + RunE: func(cmd *cobra.Command, args []string) error { + policy, err := policyAPI.PolicyCreate(cmd.Context(), inputs.PolicyCreateInput{ + SourceNetwork: args[0], DestinationVM: args[1], Protocol: args[2], DestinationPort: args[3], + }) + if err != nil { + return err + } + common.Cli.PrintDictTree(common.Cli.ToMap(policy), "Service-access policy") + return nil + }, + } +} + +func newPolicyListCmd(policyAPI api.PolicyAPI) *cobra.Command { + var jsonOutput bool + cmd := &cobra.Command{ + Use: "ls", + Aliases: []string{"list"}, + Short: "List service-access policies.", + RunE: func(cmd *cobra.Command, args []string) error { + policies, err := policyAPI.PolicyList(cmd.Context()) + if err != nil { + return err + } + if jsonOutput { + if policies == nil { + policies = []*results.Policy{} + } + data, err := json.MarshalIndent(policies, "", " ") + if err != nil { + return err + } + fmt.Println(string(data)) + return nil + } + common.RenderListing(policies, policyColumns, "short") + return nil + }, + } + cmd.Flags().BoolVar(&jsonOutput, "json", false, "Output as JSON") + return cmd +} + +func newPolicyInspectCmd(policyAPI api.PolicyAPI) *cobra.Command { + var jsonOutput bool + cmd := &cobra.Command{ + Use: "inspect [policy]", + Short: "Show one service-access policy.", + Args: cobra.ExactArgs(1), + ValidArgsFunction: completePolicyIDs, + RunE: func(cmd *cobra.Command, args []string) error { + policy, err := policyAPI.PolicyInspect(cmd.Context(), inputs.PolicyInput{Identifiers: args}) + if err != nil { + return err + } + if jsonOutput { + data, err := json.MarshalIndent(policy, "", " ") + if err != nil { + return err + } + fmt.Println(string(data)) + return nil + } + common.Cli.PrintDictTree(common.Cli.ToMap(policy), "Service-access policy") + return nil + }, + } + cmd.Flags().BoolVar(&jsonOutput, "json", false, "Output as JSON") + return cmd +} + +func newPolicyRemoveCmd(policyAPI api.PolicyAPI) *cobra.Command { + var force bool + cmd := &cobra.Command{ + Use: "rm [policies...]", + Aliases: []string{"remove", "delete", "del"}, + Short: "Remove service-access policies.", + Args: cobra.MinimumNArgs(1), + ValidArgsFunction: completePolicyIDs, + RunE: func(cmd *cobra.Command, args []string) error { + if !force { + confirmed, err := common.Cli.PromptConfirm( + cmd.Context(), fmt.Sprintf("Remove %d service-access policy(s)?", len(args)), false) + if err != nil { + return err + } + if !confirmed { + return nil + } + } + if err := policyAPI.PolicyRemove(cmd.Context(), inputs.PolicyInput{Identifiers: args}); err != nil { + return err + } + common.Cli.Success(fmt.Sprintf("Removed %d service-access policy(s)", len(args))) + return nil + }, + } + cmd.Flags().BoolVarP(&force, "force", "f", false, "Skip confirmation") + return cmd +} + +func newPolicySyncCmd(policyAPI api.PolicyAPI) *cobra.Command { + var jsonOutput bool + cmd := &cobra.Command{ + Use: "sync", + Short: "Regenerate and reconcile all service-access policy rules.", + Args: cobra.NoArgs, + RunE: func(cmd *cobra.Command, args []string) error { + result, err := policyAPI.PolicySync(cmd.Context()) + if err != nil { + return err + } + if jsonOutput { + data, err := json.MarshalIndent(result, "", " ") + if err != nil { + return err + } + fmt.Println(string(data)) + return nil + } + common.Cli.Success(fmt.Sprintf("Reconciled %d service-access policy(s)", result.Policies)) + return nil + }, + } + cmd.Flags().BoolVar(&jsonOutput, "json", false, "Output as JSON") + return cmd +} diff --git a/internal/cli/policy_test.go b/internal/cli/policy_test.go new file mode 100644 index 00000000..093a2ac4 --- /dev/null +++ b/internal/cli/policy_test.go @@ -0,0 +1,113 @@ +package cli_test + +import ( + "context" + "encoding/json" + "io" + "os" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "mvmctl/internal/cli" + "mvmctl/internal/lib/model" + "mvmctl/internal/testutil" + "mvmctl/pkg/api/inputs" + "mvmctl/pkg/api/results" +) + +func capturePolicyStdout(t *testing.T, run func() error) string { + t.Helper() + reader, writer, err := os.Pipe() + require.NoError(t, err) + original := os.Stdout + os.Stdout = writer + err = run() + require.NoError(t, writer.Close()) + os.Stdout = original + require.NoError(t, err) + data, err := io.ReadAll(reader) + require.NoError(t, err) + require.NoError(t, reader.Close()) + return string(data) +} + +func TestNewPolicyCmdShape(t *testing.T) { + cmd := cli.NewPolicyCmd(&testutil.MockPolicyAPI{}) + assert.Equal(t, "policy", cmd.Use) + for _, name := range []string{"create", "ls", "inspect", "rm", "sync"} { + sub, _, err := cmd.Find([]string{name}) + require.NoError(t, err) + require.NotNil(t, sub) + } + remove, _, err := cmd.Find([]string{"rm"}) + require.NoError(t, err) + assert.Equal(t, []string{"remove", "delete", "del"}, remove.Aliases) + assert.NotNil(t, remove.Flags().Lookup("force")) + for _, name := range []string{"ls", "inspect", "sync"} { + sub, _, findErr := cmd.Find([]string{name}) + require.NoError(t, findErr) + assert.NotNil(t, sub.Flags().Lookup("json")) + } +} + +func TestPolicyCreatePassesTypedArguments(t *testing.T) { + var got inputs.PolicyCreateInput + mock := &testutil.MockPolicyAPI{ + PolicyCreateFunc: func(_ context.Context, input inputs.PolicyCreateInput) (*results.Policy, error) { + got = input + return &results.Policy{ID: "policy-1"}, nil + }, + } + cmd := cli.NewPolicyCmd(mock) + create, _, err := cmd.Find([]string{"create"}) + require.NoError(t, err) + require.NoError(t, create.Args(create, []string{"source", "vm", "tcp", "443"})) + require.NoError(t, create.RunE(create, []string{"source", "vm", "tcp", "443"})) + assert.Equal(t, inputs.PolicyCreateInput{SourceNetwork: "source", DestinationVM: "vm", Protocol: "tcp", + DestinationPort: "443"}, got) + require.Error(t, create.Args(create, []string{"source"})) +} + +func TestPolicyJSONOutput(t *testing.T) { + policy := &results.Policy{ID: "policy-1", SourceNetworkName: "source", DestinationVMName: "vm", + Protocol: model.ServiceAccessPolicyProtocolTCP, DestinationPortStart: 443, DestinationPortEnd: 443} + mock := &testutil.MockPolicyAPI{ + PolicyListFunc: func(context.Context) ([]*results.Policy, error) { return []*results.Policy{policy}, nil }, + PolicyInspectFunc: func(context.Context, inputs.PolicyInput) (*results.Policy, error) { return policy, nil }, + PolicySyncFunc: func(context.Context) (*results.PolicySync, error) { return &results.PolicySync{Policies: 1}, nil }, + } + for _, tc := range []struct { + name string + args []string + }{ + {"list", []string{"ls"}}, {"inspect", []string{"inspect", "policy-1"}}, {"sync", []string{"sync"}}, + } { + t.Run(tc.name, func(t *testing.T) { + cmd := cli.NewPolicyCmd(mock) + sub, remaining, err := cmd.Find(tc.args) + require.NoError(t, err) + require.NoError(t, sub.Flags().Set("json", "true")) + output := capturePolicyStdout(t, func() error { return sub.RunE(sub, remaining) }) + var decoded any + require.NoError(t, json.Unmarshal([]byte(output), &decoded)) + }) + } +} + +func TestPolicyRemoveAliasesAndForce(t *testing.T) { + var got []string + mock := &testutil.MockPolicyAPI{PolicyRemoveFunc: func(_ context.Context, input inputs.PolicyInput) error { + got = input.Identifiers + return nil + }} + for _, alias := range []string{"rm", "remove", "delete", "del"} { + cmd := cli.NewPolicyCmd(mock) + remove, _, err := cmd.Find([]string{alias}) + require.NoError(t, err) + require.NoError(t, remove.Flags().Set("force", "true")) + require.NoError(t, remove.RunE(remove, []string{"a", "b"})) + assert.Equal(t, []string{"a", "b"}, got) + } +} diff --git a/internal/cli/root.go b/internal/cli/root.go index 104375f3..9d8f4604 100644 --- a/internal/cli/root.go +++ b/internal/cli/root.go @@ -119,6 +119,7 @@ func NewRootCmd(op *api.Operation) *cobra.Command { cmd.AddCommand(NewInitCmd(op, op)) cmd.AddCommand(NewEnvCmd(op)) cmd.AddCommand(NewSnapshotCmd(op)) + cmd.AddCommand(NewPolicyCmd(op)) } return cmd diff --git a/internal/core/network/constants.go b/internal/core/network/constants.go index b22709f3..833ecf2f 100644 --- a/internal/core/network/constants.go +++ b/internal/core/network/constants.go @@ -12,6 +12,8 @@ const ( FirewallChainMVMForward = model.FirewallChain(infra.MVMForwardChain) FirewallChainMVMPostrouting = model.FirewallChain(infra.MVMPostroutingChain) FirewallChainMVMNocloudNetIn = model.FirewallChain(infra.MVMNocloudNetInputChain) + FirewallChainMVMRoutedPolicy = model.FirewallChain(infra.MVMRoutedPolicyChain) + FirewallChainMVMHostInput = model.FirewallChain(infra.MVMHostInputChain) ) // NatGatewaysList returns nat_gateways as a list of strings. diff --git a/internal/core/network/service.go b/internal/core/network/service.go index a8069e67..91d9ea22 100644 --- a/internal/core/network/service.go +++ b/internal/core/network/service.go @@ -17,6 +17,7 @@ import ( // Service manages network interfaces, bridges, TAP devices, and NAT/firewall rules. type Service struct { repo Repository + policyRepo ServiceAccessPolicyRepository firewallTracker *firewall.FirewallTracker } @@ -24,8 +25,12 @@ type Service struct { // to use for firewall operations. If nil, all firewall operations are skipped // (used for cleanup contexts). Callers can replace the tracker later via // SetFirewallTracker (e.g., after HostInit resolves the firewall backend). -func NewService(repo Repository, tracker *firewall.FirewallTracker) *Service { - return &Service{repo: repo, firewallTracker: tracker} +func NewService( + repo Repository, + tracker *firewall.FirewallTracker, + policyRepo ServiceAccessPolicyRepository, +) *Service { + return &Service{repo: repo, firewallTracker: tracker, policyRepo: policyRepo} } // SetFirewallTracker replaces the firewall tracker. @@ -539,9 +544,17 @@ func (s *Service) Remove(ctx context.Context, network *model.NetworkItem, force } if hasVMs || hasSnapshots { - return s.repo.SoftDelete(ctx, network.ID) + if err := s.repo.SoftDelete(ctx, network.ID); err != nil { + slog.Error("Failed to soft-delete network", "network_id", network.ID, "error", err) + return err + } + return nil + } + if err := s.repo.Delete(ctx, network.ID); err != nil { + slog.Error("Failed to delete network", "network_id", network.ID, "error", err) + return err } - return s.repo.Delete(ctx, network.ID) + return nil } func (s *Service) RemoveMany(ctx context.Context, networks []*model.NetworkItem, force bool) error { @@ -569,6 +582,7 @@ func (s *Service) RemoveMany(ctx context.Context, networks []*model.NetworkItem, func (s *Service) SyncIPTablesRulesBatch( ctx context.Context, networks []*model.NetworkItem, + policies []*model.ResolvedServiceAccessPolicy, ) (map[string]*SyncResult, error) { if s.firewallTracker == nil { results := make(map[string]*SyncResult, len(networks)) @@ -577,6 +591,12 @@ func (s *Service) SyncIPTablesRulesBatch( } return results, nil } + policyRules := CompileServiceAccessPolicyRules(policies, networks) + if err := s.firewallTracker.ReplacePolicyRules(ctx, policyRules); err != nil { + slog.Error("Failed to commit desired service-access policy rules", "error", err) + return nil, errs.WrapMsg(errs.CodePolicySyncFailed, + "failed to commit desired service-access policy rules", err) + } // 1. Collect rules per network. Also pre-check kernel state for accurate // counting — we want added/verified to reflect kernel presence, not DB. @@ -605,16 +625,15 @@ func (s *Service) SyncIPTablesRulesBatch( perNet[net.ID] = nr } - // 2. Apply all rules in a single atomic batch. - if len(allRules) > 0 { - result := s.firewallTracker.BatchEnsureRules(ctx, allRules) - if !result.Success { - errMsg := "" - if result.ErrorMessage != nil { - errMsg = *result.ErrorMessage - } - return nil, fmt.Errorf("batch firewall rule sync failed: %s", errMsg) + // 2. Apply all rules in a single atomic batch. An empty desired set still + // flushes managed chains so removed policies cannot survive in the kernel. + result := s.firewallTracker.BatchEnsureRules(ctx, allRules) + if !result.Success { + errMsg := "" + if result.ErrorMessage != nil { + errMsg = *result.ErrorMessage } + return nil, fmt.Errorf("batch firewall rule sync failed: %s", errMsg) } // 3. Build per-network results from pre-batch kernel state. diff --git a/internal/core/network/service_access_policy.go b/internal/core/network/service_access_policy.go new file mode 100644 index 00000000..89d08416 --- /dev/null +++ b/internal/core/network/service_access_policy.go @@ -0,0 +1,91 @@ +package network + +import ( + "context" + "log/slog" + + "mvmctl/internal/lib/model" + "mvmctl/pkg/errs" +) + +// CreateServiceAccessPolicy persists trusted typed policy intent. +func (s *Service) CreateServiceAccessPolicy(ctx context.Context, policy *model.ServiceAccessPolicy) error { + existing, err := s.policyRepo.GetByIdentity(ctx, policy) + if err != nil { + slog.Error("failed to check existing service-access policy", "policy", policy.ID, "error", err) + return errs.WrapMsg(errs.CodePolicyCreateFailed, "failed to check existing service-access policy", err) + } + if existing != nil { + return errs.AlreadyExists(errs.CodePolicyAlreadyExists, "service-access policy already exists", + errs.WithEntity(existing.ID)) + } + if err := s.policyRepo.Create(ctx, policy); err != nil { + slog.Error("failed to persist service-access policy", "policy", policy.ID, "error", err) + return errs.WrapMsg(errs.CodePolicyCreateFailed, "failed to persist service-access policy", err) + } + return nil +} + +// ListServiceAccessPolicies returns persisted policy intent without cross-domain resolution. +func (s *Service) ListServiceAccessPolicies(ctx context.Context) ([]*model.ServiceAccessPolicy, error) { + policies, err := s.policyRepo.List(ctx) + if err != nil { + slog.Error("failed to list service-access policies", "error", err) + return nil, errs.WrapMsg(errs.CodePolicyListFailed, "failed to list service-access policies", err) + } + return policies, nil +} + +// RemoveServiceAccessPolicies atomically deletes trusted typed policy intent. +func (s *Service) RemoveServiceAccessPolicies(ctx context.Context, policyIDs []string) error { + if err := s.policyRepo.DeleteMany(ctx, policyIDs); err != nil { + slog.Error("failed to delete service-access policies", "policy_ids", policyIDs, "error", err) + return errs.WrapMsg(errs.CodePolicyRemoveFailed, "failed to delete service-access policies", err) + } + return nil +} + +// InvalidateServiceAccessPoliciesByVM deletes destination-VM-bound policy intent. +func (s *Service) InvalidateServiceAccessPoliciesByVM(ctx context.Context, vmID string) error { + if err := s.policyRepo.DeleteByVM(ctx, vmID); err != nil { + slog.Error("failed to invalidate VM service-access policies", "vm_id", vmID, "error", err) + return errs.WrapMsg(errs.CodePolicyRemoveFailed, "failed to invalidate VM service-access policies", err) + } + return nil +} + +// InvalidateServiceAccessPoliciesBySourceNetwork deletes source-network-bound policy intent. +func (s *Service) InvalidateServiceAccessPoliciesBySourceNetwork(ctx context.Context, networkID string) error { + if err := s.policyRepo.DeleteBySourceNetwork(ctx, networkID); err != nil { + slog.Error("failed to invalidate source-network service-access policies", + "network_id", networkID, "error", err) + return errs.WrapMsg(errs.CodePolicyRemoveFailed, + "failed to invalidate source-network service-access policies", err) + } + return nil +} + +// ReconcileServiceAccessPolicies compiles trusted API-resolved identities and atomically rebuilds firewall state. +func (s *Service) ReconcileServiceAccessPolicies( + ctx context.Context, + policies []*model.ResolvedServiceAccessPolicy, + networks []*model.NetworkItem, +) error { + if s.firewallTracker == nil { + err := errs.New(errs.CodePolicySyncFailed, "firewall tracker is unavailable") + slog.Error("failed to reconcile service-access policies", "error", err) + return err + } + rules := CompileServiceAccessPolicyRules(policies, networks) + s.firewallTracker.Initialize(ctx) + result := s.firewallTracker.ReconcilePolicyRules(ctx, rules) + if !result.Success { + message := "firewall policy reconciliation failed" + if result.ErrorMessage != nil { + message += ": " + *result.ErrorMessage + } + slog.Error("failed to reconcile service-access policies", "error", message) + return errs.New(errs.CodePolicySyncFailed, message) + } + return nil +} diff --git a/internal/core/network/service_access_policy_compiler.go b/internal/core/network/service_access_policy_compiler.go new file mode 100644 index 00000000..4589e282 --- /dev/null +++ b/internal/core/network/service_access_policy_compiler.go @@ -0,0 +1,110 @@ +package network + +import ( + "fmt" + "slices" + + "mvmctl/internal/lib/model" +) + +// CompileServiceAccessPolicyRules compiles current resource identities into ordered backend-neutral rules. +func CompileServiceAccessPolicyRules( + policies []*model.ResolvedServiceAccessPolicy, + networks []*model.NetworkItem, +) []model.FirewallRule { + sortedPolicies := append([]*model.ResolvedServiceAccessPolicy(nil), policies...) + sortedNetworks := append([]*model.NetworkItem(nil), networks...) + slices.SortFunc(sortedPolicies, func(a, b *model.ResolvedServiceAccessPolicy) int { + return comparePolicy(a.ServiceAccessPolicy, b.ServiceAccessPolicy) + }) + slices.SortFunc(sortedNetworks, func(a, b *model.NetworkItem) int { + if a.Bridge < b.Bridge { + return -1 + } + if a.Bridge > b.Bridge { + return 1 + } + return 0 + }) + + rules := make([]model.FirewallRule, 0, len(sortedPolicies)+2*len(sortedNetworks)) + for _, policy := range sortedPolicies { + policyID := policy.ID + comment := fmt.Sprintf("mvm:policy:%s", policy.ID) + rules = append(rules, model.FirewallRule{ + TableName: model.FirewallTableFilter, + ChainName: model.FirewallChainMVMRoutedPolicy, + RuleType: model.FirewallRuleTypePolicyAllow, + Protocol: model.FirewallProtocol(policy.Protocol), + Source: string(model.FirewallWildcardAnyCIDR), + Destination: policy.DestinationIPv4, + InInterface: policy.SourceNetworkBridge, + OutInterface: policy.DestinationNetworkBridge, + Target: model.FirewallTargetAccept, + SPort: model.FirewallPortAny, + DPort: policy.DestinationPortStart, + DPortEnd: policy.DestinationPortEnd, + NetworkID: policy.SourceNetworkID, + IsActive: true, + PolicyID: &policyID, + CommentTag: &comment, + }) + } + + for _, source := range sortedNetworks { + comment := fmt.Sprintf("mvm:routed-deny:%s", source.ID) + rules = append(rules, model.FirewallRule{ + TableName: model.FirewallTableFilter, + ChainName: model.FirewallChainMVMRoutedPolicy, + RuleType: model.FirewallRuleTypeRoutedDrop, + Protocol: model.FirewallProtocolAll, + Source: string(model.FirewallWildcardAnyCIDR), + Destination: string(model.FirewallWildcardAnyCIDR), + InInterface: source.Bridge, + OutInterface: string(model.FirewallWildcardManagedInterface), + Target: model.FirewallTargetDrop, + SPort: model.FirewallPortAny, + DPort: model.FirewallPortAny, + DPortEnd: model.FirewallPortAny, + NetworkID: source.ID, + IsActive: true, + CommentTag: &comment, + }) + } + + for _, source := range sortedNetworks { + comment := fmt.Sprintf("mvm:host-deny:%s", source.ID) + rules = append(rules, model.FirewallRule{ + TableName: model.FirewallTableFilter, + ChainName: model.FirewallChainMVMHostInput, + RuleType: model.FirewallRuleTypeHostInputDrop, + Protocol: model.FirewallProtocolAll, + Source: string(model.FirewallWildcardAnyCIDR), + Destination: string(model.FirewallWildcardAnyCIDR), + InInterface: source.Bridge, + OutInterface: string(model.FirewallWildcardAnyInterface), + Target: model.FirewallTargetDrop, + SPort: model.FirewallPortAny, + DPort: model.FirewallPortAny, + DPortEnd: model.FirewallPortAny, + NetworkID: source.ID, + IsActive: true, + CommentTag: &comment, + }) + } + return rules +} + +func comparePolicy(a, b model.ServiceAccessPolicy) int { + left := fmt.Sprintf("%s:%s:%s:%05d:%05d:%s", a.SourceNetworkID, a.DestinationVMID, + a.Protocol, a.DestinationPortStart, a.DestinationPortEnd, a.ID) + right := fmt.Sprintf("%s:%s:%s:%05d:%05d:%s", b.SourceNetworkID, b.DestinationVMID, + b.Protocol, b.DestinationPortStart, b.DestinationPortEnd, b.ID) + if left < right { + return -1 + } + if left > right { + return 1 + } + return 0 +} diff --git a/internal/core/network/service_access_policy_repository.go b/internal/core/network/service_access_policy_repository.go new file mode 100644 index 00000000..9ba1888e --- /dev/null +++ b/internal/core/network/service_access_policy_repository.go @@ -0,0 +1,20 @@ +package network + +import ( + "context" + + "mvmctl/internal/lib/model" +) + +// ServiceAccessPolicyRepository persists and resolves typed service-access policies. +type ServiceAccessPolicyRepository interface { + Create(ctx context.Context, policy *model.ServiceAccessPolicy) error + Get(ctx context.Context, policyID string) (*model.ServiceAccessPolicy, error) + GetByIdentity(ctx context.Context, policy *model.ServiceAccessPolicy) (*model.ServiceAccessPolicy, error) + FindByPrefix(ctx context.Context, prefix string) ([]*model.ServiceAccessPolicy, error) + List(ctx context.Context) ([]*model.ServiceAccessPolicy, error) + Delete(ctx context.Context, policyID string) error + DeleteMany(ctx context.Context, policyIDs []string) error + DeleteByVM(ctx context.Context, vmID string) error + DeleteBySourceNetwork(ctx context.Context, networkID string) error +} diff --git a/internal/core/network/service_access_policy_sqlite.go b/internal/core/network/service_access_policy_sqlite.go new file mode 100644 index 00000000..45467e70 --- /dev/null +++ b/internal/core/network/service_access_policy_sqlite.go @@ -0,0 +1,118 @@ +package network + +import ( + "context" + "database/sql" + + "github.com/jmoiron/sqlx" + + "mvmctl/internal/lib/model" +) + +type serviceAccessPolicySQLiteRepo struct { + db *sqlx.DB +} + +// NewServiceAccessPolicyRepository creates the SQLite policy repository. +func NewServiceAccessPolicyRepository(db *sqlx.DB) ServiceAccessPolicyRepository { + return &serviceAccessPolicySQLiteRepo{db: db} +} + +func (r *serviceAccessPolicySQLiteRepo) Create(ctx context.Context, policy *model.ServiceAccessPolicy) error { + _, err := r.db.ExecContext(ctx, ` + INSERT INTO service_access_policies ( + id, source_network_id, destination_vm_id, protocol, + destination_port_start, destination_port_end, created_at, updated_at + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?)`, + policy.ID, policy.SourceNetworkID, policy.DestinationVMID, string(policy.Protocol), + policy.DestinationPortStart, policy.DestinationPortEnd, policy.CreatedAt, policy.UpdatedAt, + ) + return err +} + +func (r *serviceAccessPolicySQLiteRepo) Get( + ctx context.Context, + policyID string, +) (*model.ServiceAccessPolicy, error) { + var policy model.ServiceAccessPolicy + err := r.db.GetContext(ctx, &policy, `SELECT * FROM service_access_policies WHERE id = ?`, policyID) + if err == sql.ErrNoRows { + return nil, nil + } + return &policy, err +} + +func (r *serviceAccessPolicySQLiteRepo) GetByIdentity( + ctx context.Context, + identity *model.ServiceAccessPolicy, +) (*model.ServiceAccessPolicy, error) { + var policy model.ServiceAccessPolicy + err := r.db.GetContext(ctx, &policy, ` + SELECT * FROM service_access_policies + WHERE source_network_id = ? AND destination_vm_id = ? AND protocol = ? + AND destination_port_start = ? AND destination_port_end = ?`, + identity.SourceNetworkID, identity.DestinationVMID, string(identity.Protocol), + identity.DestinationPortStart, identity.DestinationPortEnd) + if err == sql.ErrNoRows { + return nil, nil + } + return &policy, err +} + +func (r *serviceAccessPolicySQLiteRepo) FindByPrefix( + ctx context.Context, + prefix string, +) ([]*model.ServiceAccessPolicy, error) { + var policies []*model.ServiceAccessPolicy + err := r.db.SelectContext( + ctx, + &policies, + `SELECT * FROM service_access_policies WHERE id LIKE ? ORDER BY created_at, id`, + prefix+"%", + ) + return policies, err +} + +func (r *serviceAccessPolicySQLiteRepo) List(ctx context.Context) ([]*model.ServiceAccessPolicy, error) { + var policies []*model.ServiceAccessPolicy + err := r.db.SelectContext( + ctx, + &policies, + `SELECT * FROM service_access_policies ORDER BY created_at, id`, + ) + return policies, err +} + +func (r *serviceAccessPolicySQLiteRepo) Delete(ctx context.Context, policyID string) error { + _, err := r.db.ExecContext(ctx, `DELETE FROM service_access_policies WHERE id = ?`, policyID) + return err +} + +func (r *serviceAccessPolicySQLiteRepo) DeleteMany(ctx context.Context, policyIDs []string) error { + if len(policyIDs) == 0 { + return nil + } + query, args, err := sqlx.In(`DELETE FROM service_access_policies WHERE id IN (?)`, policyIDs) + if err != nil { + return err + } + tx, err := r.db.BeginTxx(ctx, nil) + if err != nil { + return err + } + defer tx.Rollback() + if _, err := tx.ExecContext(ctx, r.db.Rebind(query), args...); err != nil { + return err + } + return tx.Commit() +} + +func (r *serviceAccessPolicySQLiteRepo) DeleteByVM(ctx context.Context, vmID string) error { + _, err := r.db.ExecContext(ctx, `DELETE FROM service_access_policies WHERE destination_vm_id = ?`, vmID) + return err +} + +func (r *serviceAccessPolicySQLiteRepo) DeleteBySourceNetwork(ctx context.Context, networkID string) error { + _, err := r.db.ExecContext(ctx, `DELETE FROM service_access_policies WHERE source_network_id = ?`, networkID) + return err +} diff --git a/internal/core/network/service_access_policy_sqlite_test.go b/internal/core/network/service_access_policy_sqlite_test.go new file mode 100644 index 00000000..3f3cdd9e --- /dev/null +++ b/internal/core/network/service_access_policy_sqlite_test.go @@ -0,0 +1,146 @@ +package network_test + +import ( + "context" + "testing" + + "github.com/jmoiron/sqlx" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "mvmctl/internal/core/network" + "mvmctl/internal/lib/model" + "mvmctl/internal/testutil" +) + +func seedPolicyResources( + t *testing.T, +) (*model.ServiceAccessPolicy, *model.ServiceAccessPolicy, network.ServiceAccessPolicyRepository, *sqlx.DB) { + t.Helper() + database := testutil.NewInMemoryDB(t) + _, err := database.Exec(` + INSERT INTO networks (id,name,subnet,bridge,ipv4_gateway,bridge_active,nat_enabled,is_default,is_present,created_at,updated_at) + VALUES ('net-a','a','10.1.0.0/24','mvm-a','10.1.0.1',1,1,0,1,CURRENT_TIMESTAMP,CURRENT_TIMESTAMP), + ('net-b','b','10.2.0.0/24','mvm-b','10.2.0.1',1,1,0,1,CURRENT_TIMESTAMP,CURRENT_TIMESTAMP); + INSERT INTO images (id,type,version,name,arch,path,fs_type,original_size,minimum_rootfs_size_mib,pulled_at,is_present) + VALUES ('img','alpine','1','img','x86_64','/img','ext4',1,1,CURRENT_TIMESTAMP,1); + INSERT INTO kernels (id,name,base_name,version,arch,type,path,is_present) + VALUES ('kernel','kernel','kernel','1','x86_64','firecracker','/kernel',1); + INSERT INTO binaries (id,type,version,full_version,path,is_present) + VALUES ('fc','firecracker','1','1','/fc',1), ('jailer','jailer','1','1','/jailer',1); + INSERT INTO vm_instances ( + id,name,status,pid,ipv4,mac,network_id,tap_device,image_id,kernel_id,binary_id,jailer_binary_id, + api_socket_path,config_path,cloud_init_mode,vcpu_count,mem_size_mib,disk_size_mib,rootfs_path, + rootfs_suffix,pci_enabled,nested_virt,remote_exec,lsm_flags,enable_logging,enable_metrics, + enable_console,boot_args) + VALUES ('vm-a','vm-a','stopped',0,'10.2.0.2','02:00:00:00:00:01','net-b','tap-a','img','kernel','fc', + 'jailer','/api','/config','inject',1,128,64,'/rootfs','',0,0,0,'',0,0,0,''); + `) + require.NoError(t, err) + repo := network.NewServiceAccessPolicyRepository(database) + p1 := &model.ServiceAccessPolicy{ID: "abc-111", SourceNetworkID: "net-a", DestinationVMID: "vm-a", + Protocol: model.ServiceAccessPolicyProtocolTCP, DestinationPortStart: 443, DestinationPortEnd: 443, + CreatedAt: "2026-01-01T00:00:00Z", UpdatedAt: "2026-01-01T00:00:00Z"} + p2 := &model.ServiceAccessPolicy{ID: "abc-222", SourceNetworkID: "net-a", DestinationVMID: "vm-a", + Protocol: model.ServiceAccessPolicyProtocolUDP, DestinationPortStart: 5000, DestinationPortEnd: 5005, + CreatedAt: "2026-01-02T00:00:00Z", UpdatedAt: "2026-01-02T00:00:00Z"} + return p1, p2, repo, database +} + +func TestServiceAccessPolicySQLiteCRUD(t *testing.T) { + ctx := context.Background() + p1, p2, repo, _ := seedPolicyResources(t) + require.NoError(t, repo.Create(ctx, p2)) + require.NoError(t, repo.Create(ctx, p1)) + + got, err := repo.Get(ctx, p1.ID) + require.NoError(t, err) + assert.Equal(t, p1, got) + missing, err := repo.Get(ctx, "missing") + require.NoError(t, err) + assert.Nil(t, missing) + identity, err := repo.GetByIdentity(ctx, p2) + require.NoError(t, err) + assert.Equal(t, p2.ID, identity.ID) + + matches, err := repo.FindByPrefix(ctx, "abc-") + require.NoError(t, err) + assert.Equal(t, []string{p1.ID, p2.ID}, []string{matches[0].ID, matches[1].ID}) + listed, err := repo.List(ctx) + require.NoError(t, err) + assert.Equal(t, []string{p1.ID, p2.ID}, []string{listed[0].ID, listed[1].ID}) + + require.Error(t, repo.Create(ctx, p1), "primary-key and typed identity duplicates must be rejected") + duplicateIdentity := *p1 + duplicateIdentity.ID = "different-id" + require.Error(t, repo.Create(ctx, &duplicateIdentity), "unique policy identity must be enforced") + + require.NoError(t, repo.Delete(ctx, p1.ID)) + got, err = repo.Get(ctx, p1.ID) + require.NoError(t, err) + assert.Nil(t, got) + require.NoError(t, repo.DeleteMany(ctx, nil)) + require.NoError(t, repo.DeleteMany(ctx, []string{p2.ID})) + listed, err = repo.List(ctx) + require.NoError(t, err) + assert.Empty(t, listed) +} + +func TestServiceAccessPolicySQLiteDeleteManyIsAtomic(t *testing.T) { + ctx := context.Background() + p1, p2, repo, database := seedPolicyResources(t) + require.NoError(t, repo.Create(ctx, p1)) + require.NoError(t, repo.Create(ctx, p2)) + + _, err := database.Exec(`CREATE TRIGGER reject_second_policy_delete BEFORE DELETE ON service_access_policies + WHEN OLD.id = 'abc-222' BEGIN SELECT RAISE(ABORT, 'injected delete failure'); END`) + require.NoError(t, err) + require.Error(t, repo.DeleteMany(ctx, []string{p1.ID, p2.ID})) + items, err := repo.List(ctx) + require.NoError(t, err) + assert.Equal(t, []string{p1.ID, p2.ID}, []string{items[0].ID, items[1].ID}, + "a failed multi-delete must roll back every selected policy") +} + +func TestServiceAccessPolicySQLiteCascadesAndExplicitInvalidation(t *testing.T) { + ctx := context.Background() + tests := map[string]struct { + delete func(context.Context, network.ServiceAccessPolicyRepository) error + }{ + "destination VM": {delete: func(ctx context.Context, repo network.ServiceAccessPolicyRepository) error { + return repo.DeleteByVM(ctx, "vm-a") + }}, + "source network": {delete: func(ctx context.Context, repo network.ServiceAccessPolicyRepository) error { + return repo.DeleteBySourceNetwork(ctx, "net-a") + }}, + } + for name, tc := range tests { + t.Run(name, func(t *testing.T) { + p1, _, repo, _ := seedPolicyResources(t) + require.NoError(t, repo.Create(ctx, p1)) + require.NoError(t, tc.delete(ctx, repo)) + items, err := repo.List(ctx) + require.NoError(t, err) + assert.Empty(t, items) + }) + } +} + +func TestServiceAccessPolicySQLiteForeignKeyCascades(t *testing.T) { + ctx := context.Background() + tests := map[string]struct{ statement string }{ + "destination VM deletion": {statement: "DELETE FROM vm_instances WHERE id = 'vm-a'"}, + "source network deletion": {statement: "DELETE FROM networks WHERE id = 'net-a'"}, + } + for name, tc := range tests { + t.Run(name, func(t *testing.T) { + p1, _, repo, database := seedPolicyResources(t) + require.NoError(t, repo.Create(ctx, p1)) + _, err := database.ExecContext(ctx, tc.statement) + require.NoError(t, err) + items, err := repo.List(ctx) + require.NoError(t, err) + assert.Empty(t, items) + }) + } +} diff --git a/internal/core/network/service_access_policy_test.go b/internal/core/network/service_access_policy_test.go new file mode 100644 index 00000000..54ead0a8 --- /dev/null +++ b/internal/core/network/service_access_policy_test.go @@ -0,0 +1,157 @@ +package network_test + +import ( + "context" + "errors" + "testing" + + "github.com/google/go-cmp/cmp" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "mvmctl/internal/core/network" + "mvmctl/internal/lib/model" + "mvmctl/internal/testutil" + "mvmctl/pkg/errs" +) + +type policyRepoStub struct { + items []*model.ServiceAccessPolicy + existing *model.ServiceAccessPolicy + err error + created *model.ServiceAccessPolicy + deleted []string + deletedVM string + deletedNet string +} + +func (r *policyRepoStub) Create(_ context.Context, p *model.ServiceAccessPolicy) error { + r.created = p + return r.err +} +func (r *policyRepoStub) Get(_ context.Context, id string) (*model.ServiceAccessPolicy, error) { + for _, p := range r.items { + if p.ID == id { + return p, r.err + } + } + return nil, r.err +} + +func (r *policyRepoStub) GetByIdentity( + context.Context, + *model.ServiceAccessPolicy, +) (*model.ServiceAccessPolicy, error) { + return r.existing, r.err +} +func (r *policyRepoStub) FindByPrefix(_ context.Context, prefix string) ([]*model.ServiceAccessPolicy, error) { + var found []*model.ServiceAccessPolicy + for _, p := range r.items { + if len(p.ID) >= len(prefix) && p.ID[:len(prefix)] == prefix { + found = append(found, p) + } + } + return found, r.err +} +func (r *policyRepoStub) List(context.Context) ([]*model.ServiceAccessPolicy, error) { + return r.items, r.err +} +func (r *policyRepoStub) Delete(context.Context, string) error { return r.err } +func (r *policyRepoStub) DeleteMany(_ context.Context, ids []string) error { + r.deleted = append([]string(nil), ids...) + return r.err +} +func (r *policyRepoStub) DeleteByVM(_ context.Context, id string) error { + r.deletedVM = id + return r.err +} +func (r *policyRepoStub) DeleteBySourceNetwork(_ context.Context, id string) error { + r.deletedNet = id + return r.err +} + +func TestCompileServiceAccessPolicyRules(t *testing.T) { + t.Parallel() + policies := []*model.ResolvedServiceAccessPolicy{ + {ServiceAccessPolicy: model.ServiceAccessPolicy{ID: "udp", SourceNetworkID: "net-b", DestinationVMID: "vm-b", + Protocol: model.ServiceAccessPolicyProtocolUDP, DestinationPortStart: 5000, DestinationPortEnd: 5005}, + SourceNetworkBridge: "mvm-b", DestinationIPv4: "10.2.0.8", DestinationNetworkBridge: "mvm-c"}, + {ServiceAccessPolicy: model.ServiceAccessPolicy{ID: "tcp", SourceNetworkID: "net-a", DestinationVMID: "vm-a", + Protocol: model.ServiceAccessPolicyProtocolTCP, DestinationPortStart: 443, DestinationPortEnd: 443}, + SourceNetworkBridge: "mvm-a", DestinationIPv4: "10.1.0.7", DestinationNetworkBridge: "mvm-b"}, + } + networks := []*model.NetworkItem{{ID: "net-b", Bridge: "mvm-b"}, {ID: "net-a", Bridge: "mvm-a"}} + + got := network.CompileServiceAccessPolicyRules(policies, networks) + require.Len(t, got, 6) + assert.Equal(t, []model.FirewallRuleType{ + model.FirewallRuleTypePolicyAllow, model.FirewallRuleTypePolicyAllow, + model.FirewallRuleTypeRoutedDrop, model.FirewallRuleTypeRoutedDrop, + model.FirewallRuleTypeHostInputDrop, model.FirewallRuleTypeHostInputDrop, + }, []model.FirewallRuleType{got[0].RuleType, got[1].RuleType, got[2].RuleType, got[3].RuleType, + got[4].RuleType, got[5].RuleType}) + assert.Equal(t, model.FirewallProtocolTCP, got[0].Protocol) + assert.Equal(t, "10.1.0.7", got[0].Destination) + assert.Equal(t, "mvm-a", got[0].InInterface) + assert.Equal(t, "mvm-b", got[0].OutInterface) + assert.Equal(t, 443, got[0].DPort) + assert.Equal(t, 443, got[0].DPortEnd) + assert.Equal(t, model.FirewallProtocolUDP, got[1].Protocol) + assert.Equal(t, 5000, got[1].DPort) + assert.Equal(t, 5005, got[1].DPortEnd) + assert.Equal(t, string(model.FirewallWildcardManagedInterface), got[2].OutInterface) + assert.Equal(t, model.FirewallChainMVMHostInput, got[4].ChainName) + assert.Equal(t, "mvm-a", got[4].InInterface) + + if diff := cmp.Diff([]string{"udp", "tcp"}, []string{policies[0].ID, policies[1].ID}); diff != "" { + t.Errorf("compiler mutated caller policy order (-want +got):\n%s", diff) + } + if diff := cmp.Diff([]string{"mvm-b", "mvm-a"}, []string{networks[0].Bridge, networks[1].Bridge}); diff != "" { + t.Errorf("compiler mutated caller network order (-want +got):\n%s", diff) + } +} + +func TestServiceAccessPolicyPersistence(t *testing.T) { + ctx := context.Background() + policy := &model.ServiceAccessPolicy{ID: "policy-1"} + t.Run("create list remove and invalidate", func(t *testing.T) { + repo := &policyRepoStub{items: []*model.ServiceAccessPolicy{policy}} + svc := network.NewService(testutil.NewNetworkRepo(), nil, repo) + require.NoError(t, svc.CreateServiceAccessPolicy(ctx, policy)) + assert.Same(t, policy, repo.created) + got, err := svc.ListServiceAccessPolicies(ctx) + require.NoError(t, err) + assert.Equal(t, repo.items, got) + require.NoError(t, svc.RemoveServiceAccessPolicies(ctx, []string{"a", "b"})) + assert.Equal(t, []string{"a", "b"}, repo.deleted) + require.NoError(t, svc.InvalidateServiceAccessPoliciesByVM(ctx, "vm-1")) + require.NoError(t, svc.InvalidateServiceAccessPoliciesBySourceNetwork(ctx, "net-1")) + assert.Equal(t, "vm-1", repo.deletedVM) + assert.Equal(t, "net-1", repo.deletedNet) + }) + t.Run("duplicate", func(t *testing.T) { + svc := network.NewService(testutil.NewNetworkRepo(), nil, &policyRepoStub{existing: policy}) + err := svc.CreateServiceAccessPolicy(ctx, policy) + require.Error(t, err) + assertCode(t, err, errs.CodePolicyAlreadyExists) + }) + t.Run("repository errors are typed", func(t *testing.T) { + repo := &policyRepoStub{err: errors.New("locked")} + svc := network.NewService(testutil.NewNetworkRepo(), nil, repo) + for _, call := range []func() error{ + func() error { return svc.CreateServiceAccessPolicy(ctx, policy) }, + func() error { _, err := svc.ListServiceAccessPolicies(ctx); return err }, + func() error { return svc.RemoveServiceAccessPolicies(ctx, []string{"x"}) }, + func() error { return svc.InvalidateServiceAccessPoliciesByVM(ctx, "vm") }, + func() error { return svc.InvalidateServiceAccessPoliciesBySourceNetwork(ctx, "net") }, + } { + assert.Error(t, call()) + } + }) + t.Run("reconcile requires tracker", func(t *testing.T) { + svc := network.NewService(testutil.NewNetworkRepo(), nil, &policyRepoStub{}) + err := svc.ReconcileServiceAccessPolicies(ctx, nil, nil) + require.Error(t, err) + assertCode(t, err, errs.CodePolicySyncFailed) + }) +} diff --git a/internal/core/network/service_test.go b/internal/core/network/service_test.go index 5dba3c89..83b4e3d8 100644 --- a/internal/core/network/service_test.go +++ b/internal/core/network/service_test.go @@ -349,7 +349,7 @@ func TestService_EnrichWithLeases(t *testing.T) { t.Run("attaches_leases_to_networks", func(t *testing.T) { leaseRepo := testutil.NewLeaseRepo() - svc := network.NewService(testutil.NewNetworkRepo(), nil) + svc := network.NewService(testutil.NewNetworkRepo(), nil, nil) net1 := newNet("n-1", "alpha", "10.0.0.0/24", "10.0.0.1") net2 := newNet("n-2", "beta", "10.0.1.0/24", "10.0.1.1") @@ -373,7 +373,7 @@ func TestService_EnrichWithLeases(t *testing.T) { t.Run("empty_networks_no_crash", func(t *testing.T) { leaseRepo := testutil.NewLeaseRepo() - svc := network.NewService(testutil.NewNetworkRepo(), nil) + svc := network.NewService(testutil.NewNetworkRepo(), nil, nil) err := svc.EnrichWithLeases(ctx, []*model.NetworkItem{}, leaseRepo) require.NoError(t, err) @@ -381,7 +381,7 @@ func TestService_EnrichWithLeases(t *testing.T) { t.Run("network_with_no_leases_gets_empty_slice", func(t *testing.T) { leaseRepo := testutil.NewLeaseRepo() - svc := network.NewService(testutil.NewNetworkRepo(), nil) + svc := network.NewService(testutil.NewNetworkRepo(), nil, nil) net := newNet("n-1", "alpha", "10.0.0.0/24", "10.0.0.1") err := svc.EnrichWithLeases(ctx, []*model.NetworkItem{net}, leaseRepo) @@ -398,7 +398,7 @@ func TestService_EnrichWithLeases(t *testing.T) { func TestService_ListAll_noVerify(t *testing.T) { ctx := context.Background() netRepo := testutil.NewNetworkRepo() - svc := network.NewService(netRepo, nil) + svc := network.NewService(netRepo, nil, nil) n1 := newNet("n-1", "alpha", "10.0.0.0/24", "10.0.0.1") n2 := newNet("n-2", "beta", "10.0.1.0/24", "10.0.1.1") @@ -414,7 +414,7 @@ func TestService_ListAll_noVerify(t *testing.T) { // Rationale: When firewall tracker is nil, these must be no-ops (not panic). func TestService_TrackerNilIsNoop(t *testing.T) { - svc := network.NewService(testutil.NewNetworkRepo(), nil) + svc := network.NewService(testutil.NewNetworkRepo(), nil, nil) assert.Nil(t, svc.FirewallTracker()) @@ -449,7 +449,7 @@ func TestService_EnsureBridge(t *testing.T) { }, } - svc := network.NewService(testutil.NewNetworkRepo(), nil) + svc := network.NewService(testutil.NewNetworkRepo(), nil, nil) err := svc.EnsureBridge(ctx, "mvm-br0", "10.0.0.1/24") require.NoError(t, err) require.Len(t, ran, 3) @@ -472,7 +472,7 @@ func TestService_EnsureBridge(t *testing.T) { }, } - svc := network.NewService(testutil.NewNetworkRepo(), nil) + svc := network.NewService(testutil.NewNetworkRepo(), nil, nil) err := svc.EnsureBridge(ctx, "mvm-br0", "10.0.0.1/24") require.NoError(t, err) // Reconcile adds subnet then sets link up @@ -495,7 +495,7 @@ func TestService_EnsureBridge(t *testing.T) { }, } - svc := network.NewService(testutil.NewNetworkRepo(), nil) + svc := network.NewService(testutil.NewNetworkRepo(), nil, nil) err := svc.EnsureBridge(ctx, "mvm-br0", "10.0.0.1/24") require.NoError(t, err) // Only one command (link set up) since subnet already present @@ -514,7 +514,7 @@ func TestService_EnsureBridge(t *testing.T) { }, } - svc := network.NewService(testutil.NewNetworkRepo(), nil) + svc := network.NewService(testutil.NewNetworkRepo(), nil, nil) err := svc.EnsureBridge(ctx, "mvm-br0", "10.0.0.1/24") require.Error(t, err) }) @@ -548,7 +548,7 @@ func TestService_RemoveBridge(t *testing.T) { }, } - svc := network.NewService(testutil.NewNetworkRepo(), nil) + svc := network.NewService(testutil.NewNetworkRepo(), nil, nil) err := svc.RemoveBridge(ctx, "mvm-br0", "net-1") require.NoError(t, err) assert.Equal(t, []string{"tap-1", "tap-2"}, removedTaps) @@ -577,7 +577,7 @@ func TestService_EnsureTap(t *testing.T) { }, } - svc := network.NewService(testutil.NewNetworkRepo(), nil) + svc := network.NewService(testutil.NewNetworkRepo(), nil, nil) err := svc.EnsureTapDevice(ctx, "tap-test", "mvm-br0") require.NoError(t, err) require.Len(t, ran, 3) @@ -600,7 +600,7 @@ func TestService_EnsureTap(t *testing.T) { }, } - svc := network.NewService(testutil.NewNetworkRepo(), nil) + svc := network.NewService(testutil.NewNetworkRepo(), nil, nil) err := svc.EnsureTapDevice(ctx, "tap-test", "mvm-br0") require.NoError(t, err) assert.Empty(t, ran, "no ip commands when TAP already on correct bridge") @@ -620,7 +620,7 @@ func TestService_EnsureTap(t *testing.T) { }, } - svc := network.NewService(testutil.NewNetworkRepo(), nil) + svc := network.NewService(testutil.NewNetworkRepo(), nil, nil) err := svc.EnsureTapDevice(ctx, "tap-test", "mvm-br0") require.NoError(t, err) require.Len(t, ran, 3) @@ -650,7 +650,7 @@ func TestService_RemoveTap(t *testing.T) { }, } - svc := network.NewService(testutil.NewNetworkRepo(), nil) + svc := network.NewService(testutil.NewNetworkRepo(), nil, nil) err := svc.RemoveTap(ctx, "tap-test", "mvm-br0", "net-1") require.NoError(t, err) assert.Equal(t, "tap-test", removedTap) @@ -664,7 +664,7 @@ func TestService_RemoveTap(t *testing.T) { TapExistsFn: func(_ context.Context, _ string) bool { return false }, } - svc := network.NewService(testutil.NewNetworkRepo(), nil) + svc := network.NewService(testutil.NewNetworkRepo(), nil, nil) err := svc.RemoveTap(ctx, "nonexistent", "", "net-1") require.NoError(t, err) }) @@ -697,7 +697,7 @@ func TestService_CleanupOrphanedBridges(t *testing.T) { n.Bridge = "mvm-alpha" require.NoError(t, netRepo.Upsert(ctx, n)) - svc := network.NewService(netRepo, nil) + svc := network.NewService(netRepo, nil, nil) count := svc.CleanupOrphanedBridges(ctx, []*model.NetworkItem{n}) assert.Equal(t, 1, count) assert.Equal(t, []string{"mvm-bravo"}, removedBridges) @@ -728,7 +728,7 @@ func TestService_RemoveStaleInterfaces(t *testing.T) { }, } - svc := network.NewService(testutil.NewNetworkRepo(), nil) + svc := network.NewService(testutil.NewNetworkRepo(), nil, nil) summary := svc.RemoveStaleInterfaces(ctx, "mvm-stale") assert.Equal(t, 2, len(removedTaps)) assert.Contains(t, summary[0], "Removed") @@ -740,16 +740,16 @@ func TestService_RemoveStaleInterfaces(t *testing.T) { // networks (no error, no panic). func TestService_SyncIPTablesRulesBatch_nilTracker(t *testing.T) { - svc := network.NewService(testutil.NewNetworkRepo(), nil) + svc := network.NewService(testutil.NewNetworkRepo(), nil, nil) - results, err := svc.SyncIPTablesRulesBatch(context.Background(), nil) + results, err := svc.SyncIPTablesRulesBatch(context.Background(), nil, nil) require.NoError(t, err) assert.Empty(t, results) results, err = svc.SyncIPTablesRulesBatch(context.Background(), []*model.NetworkItem{ {ID: "n-1", Name: "alpha"}, {ID: "n-2", Name: "beta"}, - }) + }, nil) require.NoError(t, err) require.Len(t, results, 2) assert.Equal(t, 0, results["n-1"].Added) @@ -813,7 +813,7 @@ func TestService_SyncIPTablesRulesBatch_withTracker(t *testing.T) { // Create a real nftables-backed firewall tracker. fwTracker := firewall.NewFirewallTracker(model.FirewallBackendNFTables, true, db) - svc := network.NewService(netRepo, fwTracker) + svc := network.NewService(netRepo, fwTracker, nil) // Seed two networks in BOTH the in-memory repo (used by Service for // network lookups) and the SQLite DB (satisfies FK on nftables_rules). @@ -843,7 +843,7 @@ func TestService_SyncIPTablesRulesBatch_withTracker(t *testing.T) { "all", "10.0.1.0/24", "0.0.0.0/0", "mvm-beta", "wlan0", "ACCEPT", 0, 0) // Sync batch of both networks. - results, err := svc.SyncIPTablesRulesBatch(ctx, []*model.NetworkItem{net1, net2}) + results, err := svc.SyncIPTablesRulesBatch(ctx, []*model.NetworkItem{net1, net2}, nil) require.NoError(t, err) // Verify per-network results exist. @@ -853,11 +853,11 @@ func TestService_SyncIPTablesRulesBatch_withTracker(t *testing.T) { // With the FakeRunner returning empty nft output, RuleExists finds no // kernel rules — so all are counted as "added". r1 := results["n-1"] - assert.Equal(t, 2, r1.Added) + assert.Equal(t, 4, r1.Added) assert.Equal(t, 0, r1.Verified) r2 := results["n-2"] - assert.Equal(t, 2, r2.Added) + assert.Equal(t, 4, r2.Added) assert.Equal(t, 0, r2.Verified) // Verify that BatchEnsureRules was called once (single nft -f - command). @@ -879,9 +879,9 @@ func TestService_SyncIPTablesRulesBatch_emptyNetworks(t *testing.T) { db := testutil.NewInMemoryDB(t) fwTracker := firewall.NewFirewallTracker(model.FirewallBackendNFTables, true, db) - svc := network.NewService(testutil.NewNetworkRepo(), fwTracker) + svc := network.NewService(testutil.NewNetworkRepo(), fwTracker, nil) - results, err := svc.SyncIPTablesRulesBatch(ctx, nil) + results, err := svc.SyncIPTablesRulesBatch(ctx, nil, nil) require.NoError(t, err) assert.Empty(t, results) } @@ -909,7 +909,7 @@ func TestService_ListAll_verify(t *testing.T) { require.NoError(t, netRepo.Upsert(ctx, n1)) require.NoError(t, netRepo.Upsert(ctx, n2)) - svc := network.NewService(netRepo, nil) + svc := network.NewService(netRepo, nil, nil) got, err := svc.ListAll(ctx, true) require.NoError(t, err) // Both networks are listed; the one with missing bridge is marked not present @@ -934,7 +934,7 @@ func TestService_ListAll_verify(t *testing.T) { n.Bridge = "mvm-alpha" require.NoError(t, netRepo.Upsert(ctx, n)) - svc := network.NewService(netRepo, nil) + svc := network.NewService(netRepo, nil, nil) got, err := svc.ListAll(ctx, true) require.NoError(t, err) assert.Len(t, got, 1) diff --git a/internal/infra/constants.go b/internal/infra/constants.go index c0feaf08..f3c6ae23 100644 --- a/internal/infra/constants.go +++ b/internal/infra/constants.go @@ -70,6 +70,10 @@ const MVMPostroutingChain = "MVM-POSTROUTING" const MVMNocloudNetInputChain = "MVM-NOCLOUDNET-INPUT" +const MVMRoutedPolicyChain = "MVM-ROUTED-POLICY" + +const MVMHostInputChain = "MVM-HOST-INPUT" + func SudoersDropInPath() string { return fmt.Sprintf("/etc/sudoers.d/%s", CLIName) } diff --git a/internal/lib/db/migrations/004_service_access_policies.sql b/internal/lib/db/migrations/004_service_access_policies.sql new file mode 100644 index 00000000..9796d9ee --- /dev/null +++ b/internal/lib/db/migrations/004_service_access_policies.sql @@ -0,0 +1,120 @@ +-- Migration: 004_service_access_policies +-- Version: 4 +-- Description: Routed service-access policy intent and derived firewall rule identity + +PRAGMA foreign_keys = OFF; + +CREATE TABLE service_access_policies ( + id TEXT PRIMARY KEY, + source_network_id TEXT NOT NULL, + destination_vm_id TEXT NOT NULL, + protocol TEXT NOT NULL CHECK(protocol IN ('tcp', 'udp')), + destination_port_start INTEGER NOT NULL CHECK(destination_port_start BETWEEN 1 AND 65535), + destination_port_end INTEGER NOT NULL CHECK(destination_port_end BETWEEN destination_port_start AND 65535), + created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP NOT NULL, + updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP NOT NULL, + FOREIGN KEY (source_network_id) REFERENCES networks(id) ON DELETE CASCADE, + FOREIGN KEY (destination_vm_id) REFERENCES vm_instances(id) ON DELETE CASCADE, + UNIQUE(source_network_id, destination_vm_id, protocol, destination_port_start, destination_port_end) +); +CREATE INDEX idx_service_access_policies_source ON service_access_policies(source_network_id); +CREATE INDEX idx_service_access_policies_destination_vm ON service_access_policies(destination_vm_id); + +ALTER TABLE iptables_rules RENAME TO iptables_rules_v3; +CREATE TABLE iptables_rules ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + table_name TEXT NOT NULL CHECK(table_name IN ('nat', 'filter')), + chain_name TEXT NOT NULL CHECK(chain_name LIKE 'MVM-%'), + rule_type TEXT NOT NULL CHECK(rule_type IN ( + 'masquerade', 'forward_in', 'forward_out', 'nocloudnet_input', + 'service_access_allow', 'routed_managed_drop', 'host_managed_drop' + )), + protocol TEXT NOT NULL CHECK(protocol IN ('tcp', 'udp', 'icmp', 'all')), + source TEXT NOT NULL, + destination TEXT NOT NULL, + in_interface TEXT NOT NULL, + out_interface TEXT NOT NULL, + target TEXT NOT NULL, + sport INTEGER NOT NULL, + dport INTEGER NOT NULL, + dport_end INTEGER NOT NULL DEFAULT 0, + network_id TEXT NOT NULL, + service_access_policy_id TEXT NULL, + comment_tag TEXT NULL, + command_string TEXT NULL, + created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP NOT NULL, + last_verified_at TIMESTAMP NULL, + is_active INTEGER DEFAULT 1 NOT NULL CHECK(is_active IN (0, 1)), + FOREIGN KEY (network_id) REFERENCES networks(id) ON DELETE CASCADE, + FOREIGN KEY (service_access_policy_id) REFERENCES service_access_policies(id) ON DELETE CASCADE +); +INSERT INTO iptables_rules ( + id, table_name, chain_name, rule_type, protocol, source, destination, in_interface, + out_interface, target, sport, dport, dport_end, network_id, comment_tag, command_string, + created_at, last_verified_at, is_active +) +SELECT id, table_name, chain_name, rule_type, protocol, source, destination, in_interface, + out_interface, target, sport, dport, 0, network_id, comment_tag, command_string, + created_at, last_verified_at, is_active +FROM iptables_rules_v3; +DROP TABLE iptables_rules_v3; +CREATE INDEX idx_iptables_rules_network ON iptables_rules(network_id); +CREATE INDEX idx_iptables_rules_chain ON iptables_rules(table_name, chain_name); +CREATE INDEX idx_iptables_rules_type ON iptables_rules(rule_type); +CREATE INDEX idx_iptables_rules_active ON iptables_rules(is_active) WHERE is_active = 1; +CREATE INDEX idx_iptables_rules_policy ON iptables_rules(service_access_policy_id); +CREATE UNIQUE INDEX idx_iptables_rules_unique_active + ON iptables_rules(network_id, rule_type, table_name, chain_name, protocol, source, + destination, in_interface, out_interface, target, sport, dport, dport_end) + WHERE is_active = 1; + +ALTER TABLE nftables_rules RENAME TO nftables_rules_v3; +CREATE TABLE nftables_rules ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + chain TEXT NOT NULL CHECK(chain LIKE 'MVM-%'), + rule_type TEXT NOT NULL CHECK(rule_type IN ( + 'masquerade', 'forward_in', 'forward_out', 'nocloudnet_input', + 'service_access_allow', 'routed_managed_drop', 'host_managed_drop' + )), + table_name TEXT NOT NULL CHECK(table_name IN ('filter', 'nat')), + protocol TEXT NOT NULL CHECK(protocol IN ('tcp', 'udp', 'icmp', 'all')), + source TEXT NOT NULL, + destination TEXT NOT NULL, + in_interface TEXT NOT NULL, + out_interface TEXT NOT NULL, + target TEXT NOT NULL, + sport INTEGER NOT NULL, + dport INTEGER NOT NULL, + dport_end INTEGER NOT NULL DEFAULT 0, + network_id TEXT NOT NULL, + service_access_policy_id TEXT NULL, + nft_handle INTEGER NULL, + comment_tag TEXT NULL, + command_string TEXT NULL, + created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP NOT NULL, + last_verified_at TIMESTAMP NULL, + is_active INTEGER DEFAULT 1 NOT NULL CHECK(is_active IN (0, 1)), + FOREIGN KEY (network_id) REFERENCES networks(id) ON DELETE CASCADE, + FOREIGN KEY (service_access_policy_id) REFERENCES service_access_policies(id) ON DELETE CASCADE +); +INSERT INTO nftables_rules ( + id, chain, rule_type, table_name, protocol, source, destination, in_interface, + out_interface, target, sport, dport, dport_end, network_id, nft_handle, comment_tag, + command_string, created_at, last_verified_at, is_active +) +SELECT id, chain, rule_type, table_name, protocol, source, destination, in_interface, + out_interface, target, sport, dport, 0, network_id, nft_handle, comment_tag, + command_string, created_at, last_verified_at, is_active +FROM nftables_rules_v3; +DROP TABLE nftables_rules_v3; +CREATE INDEX idx_nftables_rules_network ON nftables_rules(network_id); +CREATE INDEX idx_nftables_rules_chain ON nftables_rules(chain); +CREATE INDEX idx_nftables_rules_active ON nftables_rules(is_active) WHERE is_active = 1; +CREATE INDEX idx_nftables_rules_policy ON nftables_rules(service_access_policy_id); +CREATE UNIQUE INDEX idx_nftables_rules_unique_active + ON nftables_rules(network_id, rule_type, chain, protocol, source, destination, + in_interface, out_interface, target, sport, dport, dport_end) + WHERE is_active = 1; + +PRAGMA foreign_keys = ON; +PRAGMA user_version = 4; diff --git a/internal/lib/db/migrations_jailer_test.go b/internal/lib/db/migrations_jailer_test.go index 08a8a14e..c7680d71 100644 --- a/internal/lib/db/migrations_jailer_test.go +++ b/internal/lib/db/migrations_jailer_test.go @@ -2,51 +2,48 @@ package db_test import ( "context" - "path/filepath" "testing" "github.com/google/go-cmp/cmp" "github.com/stretchr/testify/assert" "github.com/stretchr/testify/require" - - "mvmctl/internal/lib/db" ) // Rationale: Existing databases must bind each VM to the exact live Jailer // record matching its persisted Firecracker version, while unmatched pairs fail closed. func TestMigration002_BackfillsJailerBinaryID(t *testing.T) { ctx := context.Background() - handle := db.New(filepath.Join(t.TempDir(), "migration-002.db")) - t.Cleanup(func() { require.NoError(t, handle.Close()) }) + handle := migratePolicyFixtureTo(t, 1) database := handle.DB() _, err := database.ExecContext(ctx, ` - CREATE TABLE binaries ( - id TEXT PRIMARY KEY, - type TEXT NOT NULL, - version TEXT NOT NULL, - deleted_at TEXT NULL - ); - CREATE TABLE vm_instances ( - id TEXT PRIMARY KEY, - binary_id TEXT NOT NULL - ); - INSERT INTO binaries (id, type, version, deleted_at) VALUES - ('fc-match', 'firecracker', '1.16.0', NULL), - ('jl-match', 'jailer', '1.16.0', NULL), - ('fc-missing', 'firecracker', '1.15.0', NULL), - ('fc-deleted-pair', 'firecracker', '1.14.0', NULL), - ('jl-deleted', 'jailer', '1.14.0', '2026-08-09T00:00:00Z'); - INSERT INTO vm_instances (id, binary_id) VALUES - ('vm-match', 'fc-match'), - ('vm-missing', 'fc-missing'), - ('vm-deleted-pair', 'fc-deleted-pair'); - PRAGMA user_version = 1; + INSERT INTO networks (id,name,subnet,bridge,ipv4_gateway,bridge_active,nat_enabled,is_default,is_present,created_at,updated_at) + VALUES ('net','net','10.0.0.0/24','mvm-net','10.0.0.1',1,1,0,1,CURRENT_TIMESTAMP,CURRENT_TIMESTAMP); + INSERT INTO images (id,type,version,name,arch,path,fs_type,original_size,minimum_rootfs_size_mib,pulled_at,is_present) + VALUES ('img','alpine','1','img','x86_64','/img','ext4',1,1,CURRENT_TIMESTAMP,1); + INSERT INTO kernels (id,name,base_name,version,arch,type,path,is_present) + VALUES ('kernel','kernel','kernel','1','x86_64','firecracker','/kernel',1); + INSERT INTO binaries (id,type,version,full_version,path,is_present,deleted_at) VALUES + ('fc-match','firecracker','1.16.0','1.16.0','/fc-match',1,NULL), + ('jl-match','jailer','1.16.0','1.16.0','/jl-match',1,NULL), + ('fc-missing','firecracker','1.15.0','1.15.0','/fc-missing',1,NULL), + ('fc-deleted-pair','firecracker','1.14.0','1.14.0','/fc-deleted-pair',1,NULL), + ('jl-deleted','jailer','1.14.0','1.14.0','/jl-deleted',1,'2026-08-09T00:00:00Z'); + INSERT INTO vm_instances (id,name,status,pid,ipv4,mac,network_id,tap_device,image_id,kernel_id,binary_id, + api_socket_path,config_path,cloud_init_mode,vcpu_count,mem_size_mib,disk_size_mib,rootfs_path,rootfs_suffix, + pci_enabled,nested_virt,remote_exec,lsm_flags,enable_logging,enable_metrics,enable_console,boot_args) + VALUES + ('vm-match','vm-match','stopped',0,'10.0.0.2','02:00:00:00:00:01','net','tap1','img','kernel','fc-match', + '/api1','/cfg1','inject',1,128,64,'/root1','',0,0,0,'',0,0,0,''), + ('vm-missing','vm-missing','stopped',0,'10.0.0.3','02:00:00:00:00:02','net','tap2','img','kernel','fc-missing', + '/api2','/cfg2','inject',1,128,64,'/root2','',0,0,0,'',0,0,0,''), + ('vm-deleted-pair','vm-deleted-pair','stopped',0,'10.0.0.4','02:00:00:00:00:03','net','tap3','img','kernel', + 'fc-deleted-pair','/api3','/cfg3','inject',1,128,64,'/root3','',0,0,0,'',0,0,0,''); `) require.NoError(t, err) applied, err := handle.RunMigrationsCtx(ctx) require.NoError(t, err) - assert.Equal(t, 2, applied) + assert.Equal(t, 3, applied) rows := map[string]string{} dbRows, err := database.QueryContext(ctx, "SELECT id, jailer_binary_id FROM vm_instances ORDER BY id") @@ -67,7 +64,7 @@ func TestMigration002_BackfillsJailerBinaryID(t *testing.T) { var schemaVersion int require.NoError(t, database.GetContext(ctx, &schemaVersion, "PRAGMA user_version")) - assert.Equal(t, 3, schemaVersion) + assert.Equal(t, 4, schemaVersion) var indexCount int require.NoError(t, database.GetContext(ctx, &indexCount, "SELECT COUNT(*) FROM sqlite_master WHERE type = 'index' AND name = 'idx_vm_instances_jailer_binary'")) @@ -76,23 +73,28 @@ func TestMigration002_BackfillsJailerBinaryID(t *testing.T) { func TestMigration003_AddsTypedVMCgroupLimitsDefault(t *testing.T) { ctx := context.Background() - handle := db.New(filepath.Join(t.TempDir(), "migration-003.db")) - t.Cleanup(func() { require.NoError(t, handle.Close()) }) + handle := migratePolicyFixtureTo(t, 2) database := handle.DB() _, err := database.ExecContext(ctx, ` - CREATE TABLE vm_instances ( - id TEXT PRIMARY KEY, - vcpu_count INTEGER NOT NULL, - mem_size_mib INTEGER NOT NULL - ); - INSERT INTO vm_instances (id, vcpu_count, mem_size_mib) VALUES ('legacy-vm', 1, 512); - PRAGMA user_version = 2; + INSERT INTO networks (id,name,subnet,bridge,ipv4_gateway,bridge_active,nat_enabled,is_default,is_present,created_at,updated_at) + VALUES ('net','net','10.0.0.0/24','mvm-net','10.0.0.1',1,1,0,1,CURRENT_TIMESTAMP,CURRENT_TIMESTAMP); + INSERT INTO images (id,type,version,name,arch,path,fs_type,original_size,minimum_rootfs_size_mib,pulled_at,is_present) + VALUES ('img','alpine','1','img','x86_64','/img','ext4',1,1,CURRENT_TIMESTAMP,1); + INSERT INTO kernels (id,name,base_name,version,arch,type,path,is_present) + VALUES ('kernel','kernel','kernel','1','x86_64','firecracker','/kernel',1); + INSERT INTO binaries (id,type,version,full_version,path,is_present) + VALUES ('fc','firecracker','1','1','/fc',1); + INSERT INTO vm_instances (id,name,status,pid,ipv4,mac,network_id,tap_device,image_id,kernel_id,binary_id, + jailer_binary_id,api_socket_path,config_path,cloud_init_mode,vcpu_count,mem_size_mib,disk_size_mib,rootfs_path, + rootfs_suffix,pci_enabled,nested_virt,remote_exec,lsm_flags,enable_logging,enable_metrics,enable_console,boot_args) + VALUES ('legacy-vm','legacy-vm','stopped',0,'10.0.0.2','02:00:00:00:00:01','net','tap','img','kernel','fc','', + '/api','/cfg','inject',1,512,64,'/root','',0,0,0,'',0,0,0,''); `) require.NoError(t, err) applied, err := handle.RunMigrationsCtx(ctx) require.NoError(t, err) - assert.Equal(t, 1, applied) + assert.Equal(t, 2, applied) var encoded string require.NoError(t, database.GetContext(ctx, &encoded, @@ -109,5 +111,5 @@ func TestMigration003_AddsTypedVMCgroupLimitsDefault(t *testing.T) { }`, encoded) var schemaVersion int require.NoError(t, database.GetContext(ctx, &schemaVersion, "PRAGMA user_version")) - assert.Equal(t, 3, schemaVersion) + assert.Equal(t, 4, schemaVersion) } diff --git a/internal/lib/db/migrations_policy_test.go b/internal/lib/db/migrations_policy_test.go new file mode 100644 index 00000000..cbc823a7 --- /dev/null +++ b/internal/lib/db/migrations_policy_test.go @@ -0,0 +1,94 @@ +package db_test + +import ( + "context" + "os" + "path/filepath" + "runtime" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "mvmctl/internal/lib/db" +) + +func migratePolicyFixtureTo(t *testing.T, target int) *db.Handle { + t.Helper() + handle := db.New(filepath.Join(t.TempDir(), "policy-v3.db")) + t.Cleanup(func() { require.NoError(t, handle.Close()) }) + _, thisFile, _, ok := runtime.Caller(0) + require.True(t, ok) + names := []string{"001_initial_schema.sql", "002_jailer_launch.sql", "003_vm_cgroup_limits.sql"} + for _, name := range names[:target] { + contents, err := os.ReadFile(filepath.Join(filepath.Dir(thisFile), "migrations", name)) + require.NoError(t, err) + _, err = handle.DB().Exec(string(contents)) + require.NoError(t, err) + } + return handle +} + +func migratePolicyFixtureToV3(t *testing.T) *db.Handle { return migratePolicyFixtureTo(t, 3) } + +func TestMigration004FromValidV3Schema(t *testing.T) { + ctx := context.Background() + handle := migratePolicyFixtureToV3(t) + database := handle.DB() + _, err := database.ExecContext(ctx, ` + INSERT INTO networks (id,name,subnet,bridge,ipv4_gateway,bridge_active,nat_enabled,is_default,is_present,created_at,updated_at) + VALUES ('source','source','10.1.0.0/24','mvm-source','10.1.0.1',1,1,0,1,CURRENT_TIMESTAMP,CURRENT_TIMESTAMP), + ('destination','destination','10.2.0.0/24','mvm-dest','10.2.0.1',1,1,0,1,CURRENT_TIMESTAMP,CURRENT_TIMESTAMP); + INSERT INTO iptables_rules (table_name,chain_name,rule_type,protocol,source,destination,in_interface, + out_interface,target,sport,dport,network_id,comment_tag,command_string,is_active) + VALUES ('filter','MVM-FORWARD','forward_out','tcp','10.1.0.0/24','0.0.0.0/0','mvm-source', + 'eth0','ACCEPT',0,443,'source','legacy-iptables','iptables legacy',1); + INSERT INTO nftables_rules (chain,rule_type,table_name,protocol,source,destination,in_interface, + out_interface,target,sport,dport,network_id,nft_handle,comment_tag,command_string,is_active) + VALUES ('MVM-FORWARD','forward_out','filter','udp','10.1.0.0/24','0.0.0.0/0','mvm-source', + 'eth0','ACCEPT',0,53,'source',42,'legacy-nftables','nft legacy',1); + `) + require.NoError(t, err) + + applied, err := handle.RunMigrationsCtx(ctx) + require.NoError(t, err) + assert.Equal(t, 1, applied) + var version int + require.NoError(t, database.GetContext(ctx, &version, "PRAGMA user_version")) + assert.Equal(t, 4, version) + + for table, comment := range map[string]string{ + "iptables_rules": "legacy-iptables", "nftables_rules": "legacy-nftables", + } { + var count, dportEnd int + require.NoError(t, database.QueryRowxContext(ctx, + "SELECT COUNT(*), MAX(dport_end) FROM "+table+" WHERE comment_tag = ?", comment).Scan(&count, &dportEnd)) + assert.Equal(t, 1, count, table+" legacy row") + assert.Zero(t, dportEnd, table+" migrated range end") + } + + for _, check := range []struct{ table, column string }{ + {"iptables_rules", "dport_end"}, {"iptables_rules", "service_access_policy_id"}, + {"nftables_rules", "dport_end"}, {"nftables_rules", "service_access_policy_id"}, + } { + var count int + require.NoError(t, database.GetContext(ctx, &count, + "SELECT COUNT(*) FROM pragma_table_info(?) WHERE name = ?", check.table, check.column)) + assert.Equal(t, 1, count, check.table+"."+check.column) + } + for _, index := range []string{ + "idx_service_access_policies_source", "idx_service_access_policies_destination_vm", + "idx_iptables_rules_policy", "idx_iptables_rules_unique_active", + "idx_nftables_rules_policy", "idx_nftables_rules_unique_active", + } { + var count int + require.NoError(t, database.GetContext(ctx, &count, + "SELECT COUNT(*) FROM sqlite_master WHERE type='index' AND name=?", index)) + assert.Equal(t, 1, count, index) + } + + _, err = database.ExecContext(ctx, `INSERT INTO service_access_policies + (id,source_network_id,destination_vm_id,protocol,destination_port_start,destination_port_end) + VALUES ('bad','source','missing','icmp',0,65536)`) + require.Error(t, err, "protocol, port, and foreign-key constraints must reject invalid intent") +} diff --git a/internal/lib/firewall/iptables.go b/internal/lib/firewall/iptables.go index 72c23d03..ba68b913 100644 --- a/internal/lib/firewall/iptables.go +++ b/internal/lib/firewall/iptables.go @@ -20,6 +20,8 @@ var chainToTable = map[model.FirewallChain]string{ model.FirewallChainMVMForward: "filter", model.FirewallChainMVMPostrouting: "nat", model.FirewallChainMVMNocloudNetIn: "filter", + model.FirewallChainMVMRoutedPolicy: "filter", + model.FirewallChainMVMHostInput: "filter", } // --- IPTablesTracker --- @@ -55,13 +57,56 @@ func (t *IPTablesTracker) Initialize(ctx context.Context) { jumpFrom string } chains := []chainDef{ - {model.FirewallChainMVMForward, model.FirewallTableFilter, "FORWARD"}, + {model.FirewallChainMVMForward, model.FirewallTableFilter, ""}, {model.FirewallChainMVMPostrouting, model.FirewallTableNat, "POSTROUTING"}, - {model.FirewallChainMVMNocloudNetIn, model.FirewallTableFilter, "INPUT"}, + {model.FirewallChainMVMNocloudNetIn, model.FirewallTableFilter, ""}, + {model.FirewallChainMVMRoutedPolicy, model.FirewallTableFilter, ""}, + {model.FirewallChainMVMHostInput, model.FirewallTableFilter, ""}, } for _, c := range chains { t.EnsureChain(ctx, c.chain, c.table, c.jumpFrom, 1) } + // Inserting the lower-priority jump first leaves policy enforcement at position one. + t.EnsureJumpRule(ctx, "FORWARD", string(model.FirewallChainMVMForward), model.FirewallTableFilter, 1) + t.EnsureJumpRule(ctx, "FORWARD", string(model.FirewallChainMVMRoutedPolicy), model.FirewallTableFilter, 1) + t.EnsureJumpRule(ctx, "INPUT", string(model.FirewallChainMVMHostInput), model.FirewallTableFilter, 2) + t.EnsureJumpRule(ctx, "INPUT", string(model.FirewallChainMVMNocloudNetIn), model.FirewallTableFilter, 1) + t.ensureJumpPrecedes(ctx, "FORWARD", + string(model.FirewallChainMVMRoutedPolicy), string(model.FirewallChainMVMForward)) + t.ensureJumpPrecedes(ctx, "INPUT", + string(model.FirewallChainMVMNocloudNetIn), string(model.FirewallChainMVMHostInput)) +} + +func (t *IPTablesTracker) ensureJumpPrecedes(ctx context.Context, chain, firstTarget, secondTarget string) { + result, _ := system.DefaultRunner.Run(ctx, + []string{"iptables", "-t", "filter", "-S", chain}, + system.RunCmdOpts{Privileged: true, Capture: true, Check: false}, + ) + if !result.Success() { + return + } + firstRule := fmt.Sprintf("-A %s -j %s", chain, firstTarget) + secondRule := fmt.Sprintf("-A %s -j %s", chain, secondTarget) + firstIndex := strings.Index(result.Stdout, firstRule) + secondIndex := strings.Index(result.Stdout, secondRule) + if firstIndex >= 0 && secondIndex >= 0 && firstIndex < secondIndex { + return + } + deleteResult, _ := system.DefaultRunner.Run(ctx, + []string{"iptables", "-t", "filter", "-D", chain, "-j", firstTarget}, + system.RunCmdOpts{Privileged: true, Capture: true, Check: true}, + ) + if !deleteResult.Success() { + slog.Error("Failed to reorder iptables jump rule", "chain", chain, "target", firstTarget) + return + } + insertResult, _ := system.DefaultRunner.Run(ctx, + []string{"iptables", "-t", "filter", "-I", chain, "1", "-j", firstTarget}, + system.RunCmdOpts{Privileged: true, Capture: true, Check: true}, + ) + if !insertResult.Success() { + slog.Error("Failed to restore iptables jump rule after reorder", "chain", chain, "target", firstTarget) + } } // --- CheckCommentAvailable --- @@ -114,12 +159,20 @@ func (t *IPTablesTracker) buildIptablesArgs(rule *model.FirewallRule, action Rul // Input interface if rule.InInterface != string(model.FirewallWildcardAnyInterface) { - args = append(args, "-i", rule.InInterface) + iface := rule.InInterface + if iface == string(model.FirewallWildcardManagedInterface) { + iface = infra.CLIName + "-+" + } + args = append(args, "-i", iface) } // Output interface if rule.OutInterface != string(model.FirewallWildcardAnyInterface) { - args = append(args, "-o", rule.OutInterface) + iface := rule.OutInterface + if iface == string(model.FirewallWildcardManagedInterface) { + iface = infra.CLIName + "-+" + } + args = append(args, "-o", iface) } // Source port @@ -129,7 +182,11 @@ func (t *IPTablesTracker) buildIptablesArgs(rule *model.FirewallRule, action Rul // Destination port if rule.DPort != model.FirewallPortAny { - args = append(args, "--dport", strconv.Itoa(rule.DPort)) + port := strconv.Itoa(rule.DPort) + if rule.DPortEnd > rule.DPort { + port += ":" + strconv.Itoa(rule.DPortEnd) + } + args = append(args, "--dport", port) } // Target @@ -161,11 +218,19 @@ func (t *IPTablesTracker) buildRestoreLine(rule *model.FirewallRule) string { } if rule.InInterface != string(model.FirewallWildcardAnyInterface) { - parts = append(parts, "-i", rule.InInterface) + iface := rule.InInterface + if iface == string(model.FirewallWildcardManagedInterface) { + iface = infra.CLIName + "-+" + } + parts = append(parts, "-i", iface) } if rule.OutInterface != string(model.FirewallWildcardAnyInterface) { - parts = append(parts, "-o", rule.OutInterface) + iface := rule.OutInterface + if iface == string(model.FirewallWildcardManagedInterface) { + iface = infra.CLIName + "-+" + } + parts = append(parts, "-o", iface) } if rule.SPort != model.FirewallPortAny { @@ -173,7 +238,11 @@ func (t *IPTablesTracker) buildRestoreLine(rule *model.FirewallRule) string { } if rule.DPort != model.FirewallPortAny { - parts = append(parts, "--dport", strconv.Itoa(rule.DPort)) + port := strconv.Itoa(rule.DPort) + if rule.DPortEnd > rule.DPort { + port += ":" + strconv.Itoa(rule.DPortEnd) + } + parts = append(parts, "--dport", port) } parts = append(parts, "-j", string(rule.Target)) @@ -264,7 +333,7 @@ func (t *IPTablesTracker) EnsureRule( r.CommandString = &cmdStr // Check if rule exists in database - existingDBRule, err := t.repo.FindByAttributes(ctx, + existingDBRule, err := t.repo.findByAttributes(ctx, r.TableName, r.ChainName, r.RuleType, @@ -276,6 +345,8 @@ func (t *IPTablesTracker) EnsureRule( r.OutInterface, r.SPort, r.DPort, + r.DPortEnd, + r.PolicyID, ) if err != nil { slog.Warn("Error querying iptables rule in DB", "error", err) @@ -368,7 +439,7 @@ func (t *IPTablesTracker) RemoveRule(ctx context.Context, rule model.FirewallRul // Find the rule in database first to get its comment_tag if dbRuleID == nil { - existing, err := t.repo.FindByAttributes(ctx, + existing, err := t.repo.findByAttributes(ctx, rule.TableName, rule.ChainName, rule.RuleType, @@ -380,6 +451,8 @@ func (t *IPTablesTracker) RemoveRule(ctx context.Context, rule model.FirewallRul rule.OutInterface, rule.SPort, rule.DPort, + rule.DPortEnd, + rule.PolicyID, ) if err == nil && existing != nil { dbRuleID = existing.ID @@ -541,35 +614,31 @@ func (t *IPTablesTracker) BatchEnsureRules(ctx context.Context, rules []model.Fi } } - if len(filterRules) > 0 { - restoreInput := t.buildRestoreInput(filterRules, "filter") - result, _ := system.DefaultRunner.Run( - ctx, - []string{"iptables-restore", "-n"}, - system.RunCmdOpts{Privileged: true, Capture: true, Check: true, Input: restoreInput}, - ) - if !result.Success() { - errMsg := fmt.Sprintf("command iptables-restore -n failed (exit %d): %s", result.ExitCode, result.Stderr) - return model.FirewallRuleResult{ - Success: false, - ErrorMessage: &errMsg, - } + restoreInput := t.buildRestoreInput(filterRules, "filter") + result, _ := system.DefaultRunner.Run( + ctx, + []string{"iptables-restore", "-n"}, + system.RunCmdOpts{Privileged: true, Capture: true, Check: true, Input: restoreInput}, + ) + if !result.Success() { + errMsg := fmt.Sprintf("command iptables-restore -n failed (exit %d): %s", result.ExitCode, result.Stderr) + return model.FirewallRuleResult{ + Success: false, + ErrorMessage: &errMsg, } } - if len(natRules) > 0 { - restoreInput := t.buildRestoreInput(natRules, "nat") - result, _ := system.DefaultRunner.Run( - ctx, - []string{"iptables-restore", "-n"}, - system.RunCmdOpts{Privileged: true, Capture: true, Check: true, Input: restoreInput}, - ) - if !result.Success() { - errMsg := fmt.Sprintf("command iptables-restore -n failed (exit %d): %s", result.ExitCode, result.Stderr) - return model.FirewallRuleResult{ - Success: false, - ErrorMessage: &errMsg, - } + restoreInput = t.buildRestoreInput(natRules, "nat") + result, _ = system.DefaultRunner.Run( + ctx, + []string{"iptables-restore", "-n"}, + system.RunCmdOpts{Privileged: true, Capture: true, Check: true, Input: restoreInput}, + ) + if !result.Success() { + errMsg := fmt.Sprintf("command iptables-restore -n failed (exit %d): %s", result.ExitCode, result.Stderr) + return model.FirewallRuleResult{ + Success: false, + ErrorMessage: &errMsg, } } @@ -777,6 +846,8 @@ func (t *IPTablesTracker) Teardown(ctx context.Context) { {model.FirewallChainMVMForward, model.FirewallTableFilter, "FORWARD"}, {model.FirewallChainMVMPostrouting, model.FirewallTableNat, "POSTROUTING"}, {model.FirewallChainMVMNocloudNetIn, model.FirewallTableFilter, "INPUT"}, + {model.FirewallChainMVMRoutedPolicy, model.FirewallTableFilter, "FORWARD"}, + {model.FirewallChainMVMHostInput, model.FirewallTableFilter, "INPUT"}, } for _, c := range chains { chainName := string(c.chain) diff --git a/internal/lib/firewall/iptables_repository.go b/internal/lib/firewall/iptables_repository.go index ab3a9f4f..daed1af7 100644 --- a/internal/lib/firewall/iptables_repository.go +++ b/internal/lib/firewall/iptables_repository.go @@ -24,6 +24,11 @@ func (r *IPTablesRuleRepository) ListAll(ctx context.Context) ([]*model.Firewall return rules, r.db.SelectContext(ctx, &rules, "SELECT * FROM iptables_rules ORDER BY id") } +func (r *IPTablesRuleRepository) ListAllActive(ctx context.Context) ([]*model.FirewallRule, error) { + var rules []*model.FirewallRule + return rules, r.db.SelectContext(ctx, &rules, "SELECT * FROM iptables_rules WHERE is_active = 1 ORDER BY id") +} + func (r *IPTablesRuleRepository) ListByNetworkID(ctx context.Context, networkID string) ([]*model.FirewallRule, error) { var rules []*model.FirewallRule return rules, r.db.SelectContext( @@ -140,13 +145,13 @@ func (r *IPTablesRuleRepository) Insert(ctx context.Context, rule *model.Firewal result, err := r.db.ExecContext(ctx, ` INSERT INTO iptables_rules ( table_name, chain_name, rule_type, protocol, source, destination, - in_interface, out_interface, target, sport, dport, - network_id, comment_tag, command_string, created_at, is_active - ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) + in_interface, out_interface, target, sport, dport, dport_end, + network_id, service_access_policy_id, comment_tag, command_string, created_at, is_active + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) `, string(rule.TableName), string(rule.ChainName), string(rule.RuleType), string(rule.Protocol), rule.Source, rule.Destination, rule.InInterface, rule.OutInterface, string(rule.Target), - rule.SPort, rule.DPort, rule.NetworkID, + rule.SPort, rule.DPort, rule.DPortEnd, rule.NetworkID, infra.DerefOrNil(rule.PolicyID), infra.DerefOrNil(rule.CommentTag), infra.DerefOrNil(rule.CommandString), *createdAt, isActive, ) @@ -217,10 +222,11 @@ func (r *IPTablesRuleRepository) FindAndUpsertRules( } var newRules []*model.FirewallRule for _, rule := range rules { - existing, err := r.FindByAttributes(ctx, + existing, err := r.findByAttributes(ctx, rule.TableName, rule.ChainName, rule.RuleType, rule.NetworkID, rule.Protocol, rule.Source, rule.Destination, rule.InInterface, rule.OutInterface, rule.SPort, rule.DPort, + rule.DPortEnd, rule.PolicyID, ) if err != nil { return nil, err @@ -244,19 +250,64 @@ func (r *IPTablesRuleRepository) FindByAttributes( ruleType model.FirewallRuleType, networkID string, protocol model.FirewallProtocol, source, destination string, inInterface, outInterface string, sport, dport int, +) (*model.FirewallRule, error) { + return r.findByAttributes(ctx, tableName, chainName, ruleType, networkID, protocol, + source, destination, inInterface, outInterface, sport, dport, 0, nil) +} + +func (r *IPTablesRuleRepository) findByAttributes( + ctx context.Context, + tableName model.FirewallTable, chainName model.FirewallChain, + ruleType model.FirewallRuleType, networkID string, + protocol model.FirewallProtocol, source, destination string, + inInterface, outInterface string, sport, dport, dportEnd int, policyID *string, ) (*model.FirewallRule, error) { var rule model.FirewallRule err := r.db.GetContext(ctx, &rule, `SELECT * FROM iptables_rules WHERE table_name = ? AND chain_name = ? AND rule_type = ? AND network_id = ? AND protocol = ? AND source = ? AND destination = ? AND in_interface = ? AND out_interface = ? - AND sport = ? AND dport = ? AND is_active = 1`, + AND sport = ? AND dport = ? AND dport_end = ? + AND service_access_policy_id IS ? AND is_active = 1`, string(tableName), string(chainName), string(ruleType), networkID, string(protocol), source, destination, - inInterface, outInterface, sport, dport, + inInterface, outInterface, sport, dport, dportEnd, infra.DerefOrNil(policyID), ) if err == sql.ErrNoRows { return nil, nil } return &rule, err } + +func (r *IPTablesRuleRepository) ReplacePolicyRules( + ctx context.Context, + rules []model.FirewallRule, +) error { + tx, err := r.db.BeginTxx(ctx, nil) + if err != nil { + return err + } + defer tx.Rollback() + _, err = tx.ExecContext(ctx, `DELETE FROM iptables_rules WHERE rule_type IN (?, ?, ?)`, + string(model.FirewallRuleTypePolicyAllow), string(model.FirewallRuleTypeRoutedDrop), + string(model.FirewallRuleTypeHostInputDrop)) + if err != nil { + return err + } + for i := range rules { + rule := rules[i] + _, err = tx.ExecContext(ctx, `INSERT INTO iptables_rules ( + table_name, chain_name, rule_type, protocol, source, destination, in_interface, + out_interface, target, sport, dport, dport_end, network_id, + service_access_policy_id, comment_tag, command_string, created_at, is_active + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, CURRENT_TIMESTAMP, 1)`, + string(rule.TableName), string(rule.ChainName), string(rule.RuleType), string(rule.Protocol), + rule.Source, rule.Destination, rule.InInterface, rule.OutInterface, string(rule.Target), + rule.SPort, rule.DPort, rule.DPortEnd, rule.NetworkID, infra.DerefOrNil(rule.PolicyID), + infra.DerefOrNil(rule.CommentTag), infra.DerefOrNil(rule.CommandString)) + if err != nil { + return err + } + } + return tx.Commit() +} diff --git a/internal/lib/firewall/iptables_test.go b/internal/lib/firewall/iptables_test.go index 9774f101..24378372 100644 --- a/internal/lib/firewall/iptables_test.go +++ b/internal/lib/firewall/iptables_test.go @@ -203,6 +203,24 @@ func TestBuildIptablesArgs(t *testing.T) { return append(b, "-p", "udp", "--dport", "53", "-j", "ACCEPT") }, }, + "destination_port_range": { + modify: func(r *model.FirewallRule) { + r.Protocol = model.FirewallProtocolTCP + r.DPort = 8000 + r.DPortEnd = 8010 + }, + action: ActionAppend, + appendFn: func(b []string) []string { + return append(b, "-p", "tcp", "--dport", "8000:8010", "-j", "ACCEPT") + }, + }, + "managed_output_interface": { + modify: func(r *model.FirewallRule) { + r.OutInterface = string(model.FirewallWildcardManagedInterface) + }, + action: ActionAppend, + appendFn: func(b []string) []string { return append(b, "-o", "mvm-+", "-j", "ACCEPT") }, + }, // --- Target variations --- "target_drop": { @@ -394,6 +412,20 @@ func TestBuildRestoreLine(t *testing.T) { }, want: "-A MVM-FORWARD -p udp --dport 53 -j ACCEPT", }, + "destination_port_range": { + modify: func(r *model.FirewallRule) { + r.Protocol = model.FirewallProtocolUDP + r.DPort = 5000 + r.DPortEnd = 5005 + }, + want: "-A MVM-FORWARD -p udp --dport 5000:5005 -j ACCEPT", + }, + "managed_output_interface": { + modify: func(r *model.FirewallRule) { + r.OutInterface = string(model.FirewallWildcardManagedInterface) + }, + want: "-A MVM-FORWARD -o mvm-+ -j ACCEPT", + }, "target_drop": { modify: func(r *model.FirewallRule) { r.Target = model.FirewallTargetDrop }, want: "-A MVM-FORWARD -j DROP", @@ -496,6 +528,20 @@ func TestBuildRestoreInput(t *testing.T) { SPort: model.FirewallPortAny, DPort: model.FirewallPortAny, }, + { + TableName: model.FirewallTableFilter, ChainName: model.FirewallChainMVMRoutedPolicy, + RuleType: model.FirewallRuleTypePolicyAllow, Protocol: model.FirewallProtocolTCP, + Source: string(model.FirewallWildcardAnyCIDR), Destination: "10.1.0.2", + InInterface: "mvm-source", OutInterface: "mvm-destination", Target: model.FirewallTargetAccept, + SPort: model.FirewallPortAny, DPort: 443, DPortEnd: 443, + }, + { + TableName: model.FirewallTableFilter, ChainName: model.FirewallChainMVMRoutedPolicy, + RuleType: model.FirewallRuleTypeRoutedDrop, Protocol: model.FirewallProtocolAll, + Source: string(model.FirewallWildcardAnyCIDR), Destination: string(model.FirewallWildcardAnyCIDR), + InInterface: "mvm-source", OutInterface: "mvm-+", Target: model.FirewallTargetDrop, + SPort: model.FirewallPortAny, DPort: model.FirewallPortAny, + }, } result := tracker.buildRestoreInput(rules, "filter") @@ -515,6 +561,14 @@ func TestBuildRestoreInput(t *testing.T) { // Rule line (hardcoded — not recomputed from buildRestoreLine to avoid tautology) assert.Contains(t, result, "-A MVM-FORWARD -p tcp -s 10.0.0.0/24 -o tap0 -j ACCEPT") + conntrack := strings.Index(result, + "-A MVM-ROUTED-POLICY -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT") + allow := strings.Index(result, + "-A MVM-ROUTED-POLICY -p tcp -d 10.1.0.2 -i mvm-source -o mvm-destination --dport 443 -j ACCEPT") + drop := strings.Index(result, "-A MVM-ROUTED-POLICY -i mvm-source -o mvm-+ -j DROP") + assert.GreaterOrEqual(t, conntrack, 0) + assert.Greater(t, allow, conntrack, "connection-state acceptance must precede policy allows") + assert.Greater(t, drop, allow, "policy allows must precede routed drops") }) t.Run("nat_table_no_rules", func(t *testing.T) { diff --git a/internal/lib/firewall/nftables.go b/internal/lib/firewall/nftables.go index 3a5fa9cf..05ee76a9 100644 --- a/internal/lib/firewall/nftables.go +++ b/internal/lib/firewall/nftables.go @@ -8,6 +8,7 @@ import ( "strconv" "strings" + "mvmctl/internal/infra" "mvmctl/internal/lib/model" "mvmctl/internal/lib/system" ) @@ -17,6 +18,8 @@ var nftChainToTable = map[model.FirewallChain]string{ model.FirewallChainMVMForward: "filter", model.FirewallChainMVMPostrouting: "nat", model.FirewallChainMVMNocloudNetIn: "filter", + model.FirewallChainMVMRoutedPolicy: "filter", + model.FirewallChainMVMHostInput: "filter", } // Jump rule definitions: (family, table, builtin_chain, target_chain). @@ -27,7 +30,9 @@ var nftJumpRules = []struct { target string }{ {"ip", "filter", "FORWARD", string(model.FirewallChainMVMForward)}, + {"ip", "filter", "FORWARD", string(model.FirewallChainMVMRoutedPolicy)}, {"ip", "nat", "POSTROUTING", string(model.FirewallChainMVMPostrouting)}, + {"ip", "filter", "INPUT", string(model.FirewallChainMVMHostInput)}, {"ip", "filter", "INPUT", string(model.FirewallChainMVMNocloudNetIn)}, } @@ -76,6 +81,41 @@ func (t *NFTablesTracker) jumpRuleExists(ctx context.Context, family, table, bui return strings.Contains(result.Stdout, fmt.Sprintf("jump %s", targetChain)) } +func (t *NFTablesTracker) ensureJumpPrecedes( + ctx context.Context, + family, table, builtinChain, firstTarget, secondTarget string, +) { + result, _ := system.DefaultRunner.Run(ctx, + []string{"nft", "list", "chain", family, table, builtinChain}, + system.RunCmdOpts{Privileged: true, Capture: true, Check: false}, + ) + if !result.Success() { + return + } + firstIndex := strings.Index(result.Stdout, "jump "+firstTarget) + secondIndex := strings.Index(result.Stdout, "jump "+secondTarget) + if firstIndex >= 0 && secondIndex >= 0 && firstIndex < secondIndex { + return + } + handle := t.findJumpRuleHandle(ctx, family, table, builtinChain, firstTarget) + if handle == nil { + return + } + deleteResult, _ := system.DefaultRunner.Run(ctx, []string{ + "nft", "delete", "rule", family, table, builtinChain, "handle", strconv.Itoa(*handle), + }, system.RunCmdOpts{Privileged: true, Capture: true, Check: true}) + if !deleteResult.Success() { + slog.Error("Failed to reorder nftables jump rule", "chain", builtinChain, "target", firstTarget) + return + } + insertResult, _ := system.DefaultRunner.Run(ctx, []string{ + "nft", "insert", "rule", family, table, builtinChain, "jump", firstTarget, + }, system.RunCmdOpts{Privileged: true, Capture: true, Check: true}) + if !insertResult.Success() { + slog.Error("Failed to restore nftables jump rule after reorder", "chain", builtinChain, "target", firstTarget) + } +} + // --- Find jump rule handle --- func (t *NFTablesTracker) findJumpRuleHandle( @@ -218,6 +258,10 @@ func (t *NFTablesTracker) Initialize(ctx context.Context) { "table", jr.table, ) } + t.ensureJumpPrecedes(ctx, "ip", "filter", "FORWARD", + string(model.FirewallChainMVMRoutedPolicy), string(model.FirewallChainMVMForward)) + t.ensureJumpPrecedes(ctx, "ip", "filter", "INPUT", + string(model.FirewallChainMVMNocloudNetIn), string(model.FirewallChainMVMHostInput)) } // --- Ensure chain --- @@ -340,12 +384,20 @@ func (t *NFTablesTracker) ruleToNftExpr(rule *model.FirewallRule) []string { // Input interface (double-quoted for nftables) if rule.InInterface != string(model.FirewallWildcardAnyInterface) { - expr = append(expr, "iifname", fmt.Sprintf(`"%s"`, rule.InInterface)) + iface := rule.InInterface + if iface == string(model.FirewallWildcardManagedInterface) { + iface = infra.CLIName + "-*" + } + expr = append(expr, "iifname", fmt.Sprintf(`"%s"`, iface)) } // Output interface if rule.OutInterface != string(model.FirewallWildcardAnyInterface) { - expr = append(expr, "oifname", fmt.Sprintf(`"%s"`, rule.OutInterface)) + iface := rule.OutInterface + if iface == string(model.FirewallWildcardManagedInterface) { + iface = infra.CLIName + "-*" + } + expr = append(expr, "oifname", fmt.Sprintf(`"%s"`, iface)) } // Protocol — standalone when no L4 port matches follow @@ -361,7 +413,11 @@ func (t *NFTablesTracker) ruleToNftExpr(rule *model.FirewallRule) []string { // Destination port (L4) if rule.DPort != model.FirewallPortAny { - expr = append(expr, string(rule.Protocol), "dport", strconv.Itoa(rule.DPort)) + port := strconv.Itoa(rule.DPort) + if rule.DPortEnd > rule.DPort { + port += "-" + strconv.Itoa(rule.DPortEnd) + } + expr = append(expr, string(rule.Protocol), "dport", port) } // Target (lowercase for nftables) @@ -392,7 +448,7 @@ func (t *NFTablesTracker) EnsureRule( nftExpr := t.ruleToNftExpr(&rule) // Check if rule exists in database - existingDBRule, err := t.repo.FindByAttributes(ctx, + existingDBRule, err := t.repo.findByAttributes(ctx, rule.TableName, rule.ChainName, rule.RuleType, @@ -404,6 +460,8 @@ func (t *NFTablesTracker) EnsureRule( rule.OutInterface, rule.SPort, rule.DPort, + rule.DPortEnd, + rule.PolicyID, ) if err != nil { errMsg := fmt.Sprintf("Failed to query existing nftables rule: %v", err) @@ -476,7 +534,7 @@ func (t *NFTablesTracker) RemoveRule(ctx context.Context, rule model.FirewallRul // Try to find the rule in the database first dbRule := &rule if rule.ID == nil { - existing, err := t.repo.FindByAttributes(ctx, + existing, err := t.repo.findByAttributes(ctx, rule.TableName, rule.ChainName, rule.RuleType, @@ -488,6 +546,8 @@ func (t *NFTablesTracker) RemoveRule(ctx context.Context, rule model.FirewallRul rule.OutInterface, rule.SPort, rule.DPort, + rule.DPortEnd, + rule.PolicyID, ) if err == nil && existing != nil { dbRule = existing @@ -548,7 +608,7 @@ func (t *NFTablesTracker) BatchEnsureRules(ctx context.Context, rules []model.Fi } lines = append(lines, "") - // 2. Conntrack rule first — preserves established connections + // 2. Conntrack rule first — preserves established connections. for chain, table := range nftChainToTable { if table == "filter" { lines = append(lines, @@ -558,15 +618,6 @@ func (t *NFTablesTracker) BatchEnsureRules(ctx context.Context, rules []model.Fi } lines = append(lines, "") - // Also add individual conntrack rules for FORWARD and NOCLOUDNET-INPUT - lines = append(lines, - fmt.Sprintf("add rule ip filter %s ct state established,related accept", - string(model.FirewallChainMVMForward))) - lines = append(lines, - fmt.Sprintf("add rule ip filter %s ct state established,related accept", - string(model.FirewallChainMVMNocloudNetIn))) - lines = append(lines, "") - // 3. Add all batch rules var newRules []*model.FirewallRule for i := range rules { @@ -685,6 +736,8 @@ func (t *NFTablesTracker) CountOrphanedRules(ctx context.Context, network *model {model.FirewallChainMVMForward, "filter"}, {model.FirewallChainMVMPostrouting, "nat"}, {model.FirewallChainMVMNocloudNetIn, "filter"}, + {model.FirewallChainMVMRoutedPolicy, "filter"}, + {model.FirewallChainMVMHostInput, "filter"}, } commentRe := regexp.MustCompile(`comment\s+"([^"]+)"`) diff --git a/internal/lib/firewall/nftables_repository.go b/internal/lib/firewall/nftables_repository.go index 14a0e2b2..a99debe9 100644 --- a/internal/lib/firewall/nftables_repository.go +++ b/internal/lib/firewall/nftables_repository.go @@ -11,7 +11,7 @@ import ( "mvmctl/internal/lib/model" ) -const nftablesColumns = `id, chain AS chain_name, rule_type, table_name, protocol, source, destination, in_interface, out_interface, target, sport, dport, network_id, comment_tag, command_string, created_at, last_verified_at, is_active` +const nftablesColumns = `id, chain AS chain_name, rule_type, table_name, protocol, source, destination, in_interface, out_interface, target, sport, dport, dport_end, network_id, service_access_policy_id, comment_tag, command_string, created_at, last_verified_at, is_active` type NFTablesRuleRepository struct { db *sqlx.DB @@ -135,13 +135,13 @@ func (r *NFTablesRuleRepository) Insert(ctx context.Context, rule *model.Firewal result, err := r.db.ExecContext(ctx, ` INSERT INTO nftables_rules ( chain, rule_type, table_name, protocol, source, destination, - in_interface, out_interface, target, sport, dport, - network_id, comment_tag, command_string, created_at, is_active - ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) + in_interface, out_interface, target, sport, dport, dport_end, + network_id, service_access_policy_id, comment_tag, command_string, created_at, is_active + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) `, string(rule.ChainName), string(rule.RuleType), string(rule.TableName), string(rule.Protocol), rule.Source, rule.Destination, rule.InInterface, rule.OutInterface, string(rule.Target), - rule.SPort, rule.DPort, rule.NetworkID, + rule.SPort, rule.DPort, rule.DPortEnd, rule.NetworkID, infra.DerefOrNil(rule.PolicyID), infra.DerefOrNil(rule.CommentTag), infra.DerefOrNil(rule.CommandString), *createdAt, isActive, ) @@ -196,10 +196,11 @@ func (r *NFTablesRuleRepository) FindAndUpsertRules( } var newRules []*model.FirewallRule for _, rule := range rules { - existing, err := r.FindByAttributes(ctx, + existing, err := r.findByAttributes(ctx, rule.TableName, rule.ChainName, rule.RuleType, rule.NetworkID, rule.Protocol, rule.Source, rule.Destination, rule.InInterface, rule.OutInterface, rule.SPort, rule.DPort, + rule.DPortEnd, rule.PolicyID, ) if err != nil { return nil, err @@ -223,19 +224,64 @@ func (r *NFTablesRuleRepository) FindByAttributes( ruleType model.FirewallRuleType, networkID string, protocol model.FirewallProtocol, source, destination string, inInterface, outInterface string, sport, dport int, +) (*model.FirewallRule, error) { + return r.findByAttributes(ctx, tableName, chainName, ruleType, networkID, protocol, + source, destination, inInterface, outInterface, sport, dport, 0, nil) +} + +func (r *NFTablesRuleRepository) findByAttributes( + ctx context.Context, + tableName model.FirewallTable, chainName model.FirewallChain, + ruleType model.FirewallRuleType, networkID string, + protocol model.FirewallProtocol, source, destination string, + inInterface, outInterface string, sport, dport, dportEnd int, policyID *string, ) (*model.FirewallRule, error) { var rule model.FirewallRule err := r.db.GetContext(ctx, &rule, `SELECT `+nftablesColumns+` FROM nftables_rules WHERE chain = ? AND rule_type = ? AND table_name = ? AND network_id = ? AND protocol = ? AND source = ? AND destination = ? AND in_interface = ? AND out_interface = ? - AND sport = ? AND dport = ? AND is_active = 1`, + AND sport = ? AND dport = ? AND dport_end = ? + AND service_access_policy_id IS ? AND is_active = 1`, string(chainName), string(ruleType), string(tableName), networkID, string(protocol), source, destination, - inInterface, outInterface, sport, dport, + inInterface, outInterface, sport, dport, dportEnd, infra.DerefOrNil(policyID), ) if err == sql.ErrNoRows { return nil, nil } return &rule, err } + +func (r *NFTablesRuleRepository) ReplacePolicyRules( + ctx context.Context, + rules []model.FirewallRule, +) error { + tx, err := r.db.BeginTxx(ctx, nil) + if err != nil { + return err + } + defer tx.Rollback() + _, err = tx.ExecContext(ctx, `DELETE FROM nftables_rules WHERE rule_type IN (?, ?, ?)`, + string(model.FirewallRuleTypePolicyAllow), string(model.FirewallRuleTypeRoutedDrop), + string(model.FirewallRuleTypeHostInputDrop)) + if err != nil { + return err + } + for i := range rules { + rule := rules[i] + _, err = tx.ExecContext(ctx, `INSERT INTO nftables_rules ( + chain, rule_type, table_name, protocol, source, destination, in_interface, + out_interface, target, sport, dport, dport_end, network_id, + service_access_policy_id, comment_tag, command_string, created_at, is_active + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, CURRENT_TIMESTAMP, 1)`, + string(rule.ChainName), string(rule.RuleType), string(rule.TableName), string(rule.Protocol), + rule.Source, rule.Destination, rule.InInterface, rule.OutInterface, string(rule.Target), + rule.SPort, rule.DPort, rule.DPortEnd, rule.NetworkID, infra.DerefOrNil(rule.PolicyID), + infra.DerefOrNil(rule.CommentTag), infra.DerefOrNil(rule.CommandString)) + if err != nil { + return err + } + } + return tx.Commit() +} diff --git a/internal/lib/firewall/nftables_test.go b/internal/lib/firewall/nftables_test.go index a6c388cd..cb34ae33 100644 --- a/internal/lib/firewall/nftables_test.go +++ b/internal/lib/firewall/nftables_test.go @@ -173,6 +173,20 @@ func TestRuleToNftExpr(t *testing.T) { }, want: []string{"udp", "dport", "53", "accept"}, }, + "protocol_tcp_with_dport_range": { + modify: func(r *model.FirewallRule) { + r.Protocol = model.FirewallProtocolTCP + r.DPort = 8000 + r.DPortEnd = 8010 + }, + want: []string{"tcp", "dport", "8000-8010", "accept"}, + }, + "managed_output_interface": { + modify: func(r *model.FirewallRule) { + r.OutInterface = string(model.FirewallWildcardManagedInterface) + }, + want: []string{"oifname", `"mvm-*"`, "accept"}, + }, "protocol_tcp_with_both_ports": { modify: func(r *model.FirewallRule) { r.Protocol = model.FirewallProtocolTCP diff --git a/internal/lib/firewall/tracker.go b/internal/lib/firewall/tracker.go index debad95d..6641055c 100644 --- a/internal/lib/firewall/tracker.go +++ b/internal/lib/firewall/tracker.go @@ -2,6 +2,7 @@ package firewall import ( "context" + "fmt" "log/slog" "github.com/jmoiron/sqlx" @@ -40,6 +41,11 @@ type firewallRuleRepo interface { ) ([]*model.FirewallRule, error) } +type policyFirewallRuleRepo interface { + ListAllActive(ctx context.Context) ([]*model.FirewallRule, error) + ReplacePolicyRules(ctx context.Context, rules []model.FirewallRule) error +} + // --- FirewallTracker (dispatcher) --- type FirewallTracker struct { @@ -108,6 +114,41 @@ func (ft *FirewallTracker) BatchEnsureRules(ctx context.Context, rules []model.F return ft.backend.BatchEnsureRules(ctx, rules) } +// ReconcilePolicyRules atomically rebuilds kernel rules with freshly compiled policy rules. +func (ft *FirewallTracker) ReconcilePolicyRules( + ctx context.Context, + policyRules []model.FirewallRule, +) model.FirewallRuleResult { + policyRepo, ok := ft.firewallRepo.(policyFirewallRuleRepo) + if !ok { + message := "firewall repository does not support policy reconciliation" + return model.FirewallRuleResult{Success: false, ErrorMessage: &message} + } + if err := policyRepo.ReplacePolicyRules(ctx, policyRules); err != nil { + message := err.Error() + return model.FirewallRuleResult{Success: false, ErrorMessage: &message} + } + activeRules, err := policyRepo.ListAllActive(ctx) + if err != nil { + message := err.Error() + return model.FirewallRuleResult{Success: false, ErrorMessage: &message} + } + allRules := make([]model.FirewallRule, 0, len(activeRules)) + for _, rule := range activeRules { + allRules = append(allRules, *rule) + } + return ft.backend.BatchEnsureRules(ctx, allRules) +} + +// ReplacePolicyRules commits desired derived policy rows without applying kernel state. +func (ft *FirewallTracker) ReplacePolicyRules(ctx context.Context, policyRules []model.FirewallRule) error { + policyRepo, ok := ft.firewallRepo.(policyFirewallRuleRepo) + if !ok { + return fmt.Errorf("firewall repository does not support policy reconciliation") + } + return policyRepo.ReplacePolicyRules(ctx, policyRules) +} + func (ft *FirewallTracker) RemoveRule(ctx context.Context, rule model.FirewallRule) model.FirewallRuleResult { return ft.backend.RemoveRule(ctx, rule) } diff --git a/internal/lib/firewall/tracker_test.go b/internal/lib/firewall/tracker_test.go index 3cb974a0..94c62b45 100644 --- a/internal/lib/firewall/tracker_test.go +++ b/internal/lib/firewall/tracker_test.go @@ -19,6 +19,7 @@ type mockTracker struct { teardownCalls int ensureRuleResults []ensureRuleCall batchEnsureCalls int + batchEnsureRules []model.FirewallRule removeRuleCalls int batchRemoveCalls int ensureChainCalls int @@ -59,10 +60,11 @@ func (m *mockTracker) EnsureRule(_ context.Context, rule model.FirewallRule, lab return m.stubEnsureRule } -func (m *mockTracker) BatchEnsureRules(_ context.Context, _ []model.FirewallRule) model.FirewallRuleResult { +func (m *mockTracker) BatchEnsureRules(_ context.Context, rules []model.FirewallRule) model.FirewallRuleResult { m.mu.Lock() defer m.mu.Unlock() m.batchEnsureCalls++ + m.batchEnsureRules = append([]model.FirewallRule(nil), rules...) return m.stubBatchEnsure } @@ -120,8 +122,19 @@ type mockRepo struct { getByNetworkIDCalls int getByNetworkIDIFCalls int - stubRules []*model.FirewallRule - stubErr error + stubRules []*model.FirewallRule + stubErr error + replaced []model.FirewallRule + replaceErr error +} + +func (m *mockRepo) ListAllActive(context.Context) ([]*model.FirewallRule, error) { + return m.stubRules, m.stubErr +} + +func (m *mockRepo) ReplacePolicyRules(_ context.Context, rules []model.FirewallRule) error { + m.replaced = append([]model.FirewallRule(nil), rules...) + return m.replaceErr } func (m *mockRepo) GetByNetworkID(_ context.Context, _ string, _ bool) ([]*model.FirewallRule, error) { @@ -335,6 +348,41 @@ func TestTracker_EnsureRule_inBatchMode_queuesRule(t *testing.T) { assert.Equal(t, 0, len(mb.ensureRuleResults), "backend should not be called") } +func TestTracker_ReconcilePolicyRulesReplacesRowsBeforeKernelApply(t *testing.T) { + ctx := context.Background() + desired := []model.FirewallRule{{NetworkID: "new", RuleType: model.FirewallRuleTypePolicyAllow}} + active := []*model.FirewallRule{{NetworkID: "new", RuleType: model.FirewallRuleTypePolicyAllow}} + repo := &mockRepo{stubRules: active} + backend := &mockTracker{stubBatchEnsure: model.FirewallRuleResult{Success: true}} + tracker := &FirewallTracker{firewallRepo: repo, backend: backend} + + result := tracker.ReconcilePolicyRules(ctx, desired) + assert.True(t, result.Success) + assert.Equal(t, desired, repo.replaced) + assert.Equal(t, []model.FirewallRule{*active[0]}, backend.batchEnsureRules) +} + +func TestTracker_ReconcilePolicyRulesReplacementFailurePreventsKernelApply(t *testing.T) { + repo := &mockRepo{replaceErr: assert.AnError} + backend := &mockTracker{} + tracker := &FirewallTracker{firewallRepo: repo, backend: backend} + + result := tracker.ReconcilePolicyRules(context.Background(), nil) + assert.False(t, result.Success) + assert.Zero(t, backend.batchEnsureCalls) +} + +func TestTracker_ReconcilePolicyRulesEmptyStateStillFlushesKernel(t *testing.T) { + repo := &mockRepo{} + backend := &mockTracker{stubBatchEnsure: model.FirewallRuleResult{Success: true}} + tracker := &FirewallTracker{firewallRepo: repo, backend: backend} + + result := tracker.ReconcilePolicyRules(context.Background(), nil) + assert.True(t, result.Success) + assert.Equal(t, 1, backend.batchEnsureCalls) + assert.Empty(t, backend.batchEnsureRules) +} + // --- Nil backend guard --- // Rationale: FirewallTracker constructed via NewFirewallTracker with an // unrecognised backend type has nil backend and nil repo. Must not panic. diff --git a/internal/lib/model/network.go b/internal/lib/model/network.go index 120b80cb..48301283 100644 --- a/internal/lib/model/network.go +++ b/internal/lib/model/network.go @@ -30,6 +30,8 @@ const ( FirewallChainMVMForward FirewallChain = "MVM-FORWARD" FirewallChainMVMPostrouting FirewallChain = "MVM-POSTROUTING" FirewallChainMVMNocloudNetIn FirewallChain = "MVM-NOCLOUDNET-INPUT" + FirewallChainMVMRoutedPolicy FirewallChain = "MVM-ROUTED-POLICY" + FirewallChainMVMHostInput FirewallChain = "MVM-HOST-INPUT" ) // FirewallRuleType categorises firewall rules. @@ -40,6 +42,9 @@ const ( FirewallRuleTypeForwardIn FirewallRuleType = "forward_in" FirewallRuleTypeForwardOut FirewallRuleType = "forward_out" FirewallRuleTypeNocloudNetInput FirewallRuleType = "nocloudnet_input" + FirewallRuleTypePolicyAllow FirewallRuleType = "service_access_allow" + FirewallRuleTypeRoutedDrop FirewallRuleType = "routed_managed_drop" + FirewallRuleTypeHostInputDrop FirewallRuleType = "host_managed_drop" ) // FirewallProtocol specifies the IP protocol. @@ -70,6 +75,8 @@ type FirewallWildcard string const ( FirewallWildcardAnyCIDR FirewallWildcard = "0.0.0.0/0" FirewallWildcardAnyInterface FirewallWildcard = "*" + // FirewallWildcardManagedInterface is an internal derived-rule marker expanded by firewall backends. + FirewallWildcardManagedInterface FirewallWildcard = "@mvmctl-managed" ) // FirewallPortAny is the sentinel value meaning "any port". @@ -127,6 +134,7 @@ type FirewallRule struct { Target FirewallTarget `json:"target" db:"target"` SPort int `json:"sport" db:"sport"` DPort int `json:"dport" db:"dport"` + DPortEnd int `json:"dport_end" db:"dport_end"` NetworkID string `json:"network_id" db:"network_id"` IsActive bool `json:"is_active" db:"is_active"` @@ -136,13 +144,19 @@ type FirewallRule struct { CommandString *string `json:"command_string,omitempty" db:"command_string"` CreatedAt *string `json:"created_at,omitempty" db:"created_at"` LastVerifiedAt *string `json:"last_verified_at,omitempty" db:"last_verified_at"` + PolicyID *string `json:"policy_id,omitempty" db:"service_access_policy_id"` } // Key builds a dedup key from a rule's identifying fields. func (r FirewallRule) Key() string { - return fmt.Sprintf("%s:%s:%s:%s:%s:%d:%d", - r.TableName, r.ChainName, r.RuleType, - r.Source, r.OutInterface, r.SPort, r.DPort) + policyID := "" + if r.PolicyID != nil { + policyID = *r.PolicyID + } + return fmt.Sprintf("%s:%s:%s:%s:%s:%s:%s:%s:%s:%d:%d:%d:%s:%s", + r.TableName, r.ChainName, r.RuleType, r.Protocol, r.Source, r.Destination, + r.InInterface, r.OutInterface, r.Target, r.SPort, r.DPort, r.DPortEnd, + r.NetworkID, policyID) } // NewTapForwardRules returns the two FORWARD rules needed for a TAP device: diff --git a/internal/lib/model/network_test.go b/internal/lib/model/network_test.go new file mode 100644 index 00000000..fbecdbbe --- /dev/null +++ b/internal/lib/model/network_test.go @@ -0,0 +1,47 @@ +package model_test + +import ( + "testing" + + "github.com/stretchr/testify/assert" + + "mvmctl/internal/lib/model" +) + +func TestFirewallRuleKeyIncludesEveryPolicyIdentityField(t *testing.T) { + t.Parallel() + policy := "policy-a" + base := model.FirewallRule{ + TableName: model.FirewallTableFilter, ChainName: model.FirewallChainMVMRoutedPolicy, + RuleType: model.FirewallRuleTypePolicyAllow, Protocol: model.FirewallProtocolTCP, + Source: "0.0.0.0/0", Destination: "10.0.0.2", InInterface: "mvm-a", OutInterface: "mvm-b", + Target: model.FirewallTargetAccept, SPort: 0, DPort: 443, DPortEnd: 443, + NetworkID: "net-a", PolicyID: &policy, + } + mutations := map[string]func(*model.FirewallRule){ + "table": func(r *model.FirewallRule) { r.TableName = model.FirewallTableNat }, + "chain": func(r *model.FirewallRule) { r.ChainName = model.FirewallChainMVMForward }, + "type": func(r *model.FirewallRule) { r.RuleType = model.FirewallRuleTypeRoutedDrop }, + "protocol": func(r *model.FirewallRule) { r.Protocol = model.FirewallProtocolUDP }, + "source": func(r *model.FirewallRule) { r.Source = "10.1.0.0/24" }, + "destination": func(r *model.FirewallRule) { r.Destination = "10.0.0.3" }, + "input": func(r *model.FirewallRule) { r.InInterface = "mvm-c" }, + "output": func(r *model.FirewallRule) { r.OutInterface = "mvm-d" }, + "target": func(r *model.FirewallRule) { r.Target = model.FirewallTargetDrop }, + "source port": func(r *model.FirewallRule) { r.SPort = 1 }, + "destination port": func(r *model.FirewallRule) { r.DPort = 444 }, + "range end": func(r *model.FirewallRule) { r.DPortEnd = 450 }, + "network": func(r *model.FirewallRule) { r.NetworkID = "net-b" }, + "policy": func(r *model.FirewallRule) { other := "policy-b"; r.PolicyID = &other }, + } + seen := map[string]string{base.Key(): "base"} + for name, mutate := range mutations { + rule := base + mutate(&rule) + if prior, exists := seen[rule.Key()]; exists { + t.Errorf("%s key collided with %s", name, prior) + } + seen[rule.Key()] = name + } + assert.Len(t, seen, len(mutations)+1) +} diff --git a/internal/lib/model/service_access_policy.go b/internal/lib/model/service_access_policy.go new file mode 100644 index 00000000..e6e06050 --- /dev/null +++ b/internal/lib/model/service_access_policy.go @@ -0,0 +1,34 @@ +package model + +// ServiceAccessPolicyProtocol is the transport protocol exposed by a policy. +type ServiceAccessPolicyProtocol string + +const ( + ServiceAccessPolicyProtocolTCP ServiceAccessPolicyProtocol = "tcp" + ServiceAccessPolicyProtocolUDP ServiceAccessPolicyProtocol = "udp" +) + +// ServiceAccessPolicy persists routed service-access intent using resource identities only. +type ServiceAccessPolicy struct { + ID string `json:"id" db:"id"` + SourceNetworkID string `json:"source_network_id" db:"source_network_id"` + DestinationVMID string `json:"destination_vm_id" db:"destination_vm_id"` + Protocol ServiceAccessPolicyProtocol `json:"protocol" db:"protocol"` + DestinationPortStart int `json:"destination_port_start" db:"destination_port_start"` + DestinationPortEnd int `json:"destination_port_end" db:"destination_port_end"` + CreatedAt string `json:"created_at" db:"created_at"` + UpdatedAt string `json:"updated_at" db:"updated_at"` +} + +// ResolvedServiceAccessPolicy contains API-resolved current resource attributes used during compilation. +// The additional fields are runtime-only and are never persisted as policy intent. +type ResolvedServiceAccessPolicy struct { + ServiceAccessPolicy + SourceNetworkName string `json:"source_network_name" db:"source_network_name"` + SourceNetworkBridge string `json:"-" db:"source_network_bridge"` + DestinationVMName string `json:"destination_vm_name" db:"destination_vm_name"` + DestinationIPv4 string `json:"-" db:"destination_ipv4"` + DestinationNetworkID string `json:"destination_network_id" db:"destination_network_id"` + DestinationNetworkName string `json:"destination_network_name" db:"destination_network_name"` + DestinationNetworkBridge string `json:"-" db:"destination_network_bridge"` +} diff --git a/internal/testutil/mock_api.go b/internal/testutil/mock_api.go index 8f13a66a..d7c6f2b4 100644 --- a/internal/testutil/mock_api.go +++ b/internal/testutil/mock_api.go @@ -3,6 +3,7 @@ package testutil import ( "context" + "mvmctl/pkg/api/inputs" "mvmctl/pkg/api/results" ) @@ -26,6 +27,51 @@ type MockOperation struct { MockInitAPI MockSnapshotAPI MockUpdateAPI + MockPolicyAPI +} + +// MockPolicyAPI implements api.PolicyAPI for composite-interface compatibility. +type MockPolicyAPI struct { + PolicyCreateFunc func(context.Context, inputs.PolicyCreateInput) (*results.Policy, error) + PolicyListFunc func(context.Context) ([]*results.Policy, error) + PolicyInspectFunc func(context.Context, inputs.PolicyInput) (*results.Policy, error) + PolicyRemoveFunc func(context.Context, inputs.PolicyInput) error + PolicySyncFunc func(context.Context) (*results.PolicySync, error) +} + +func (m *MockPolicyAPI) PolicyCreate(ctx context.Context, input inputs.PolicyCreateInput) (*results.Policy, error) { + if m.PolicyCreateFunc != nil { + return m.PolicyCreateFunc(ctx, input) + } + return nil, nil +} + +func (m *MockPolicyAPI) PolicyList(ctx context.Context) ([]*results.Policy, error) { + if m.PolicyListFunc != nil { + return m.PolicyListFunc(ctx) + } + return nil, nil +} + +func (m *MockPolicyAPI) PolicyInspect(ctx context.Context, input inputs.PolicyInput) (*results.Policy, error) { + if m.PolicyInspectFunc != nil { + return m.PolicyInspectFunc(ctx, input) + } + return nil, nil +} + +func (m *MockPolicyAPI) PolicyRemove(ctx context.Context, input inputs.PolicyInput) error { + if m.PolicyRemoveFunc != nil { + return m.PolicyRemoveFunc(ctx, input) + } + return nil +} + +func (m *MockPolicyAPI) PolicySync(ctx context.Context) (*results.PolicySync, error) { + if m.PolicySyncFunc != nil { + return m.PolicySyncFunc(ctx) + } + return nil, nil } // MockUpdateAPI implements api.UpdateAPI for testing. diff --git a/pkg/api/host.go b/pkg/api/host.go index b9eaad94..2daa066d 100644 --- a/pkg/api/host.go +++ b/pkg/api/host.go @@ -138,6 +138,10 @@ func (op *Operation) HostInit(ctx context.Context, onProgress event.OnProgressCa if err != nil { return nil, errs.New(errs.CodeHostInitFailed, err.Error()) } + if err := op.reconcileServiceAccessPolicies(ctx); err != nil { + return nil, errs.WrapMsg(errs.CodeHostInitFailed, + "failed to replay service-access policies during host initialization", err) + } // --- Finalize --- now := time.Now().Format(time.RFC3339) if err := hostCtrl.MarkInitialized(ctx, now); err != nil { diff --git a/pkg/api/inputs/policy.go b/pkg/api/inputs/policy.go new file mode 100644 index 00000000..06d09482 --- /dev/null +++ b/pkg/api/inputs/policy.go @@ -0,0 +1,147 @@ +package inputs + +import ( + "context" + "fmt" + "strconv" + "strings" + + "mvmctl/internal/core/network" + "mvmctl/internal/core/vm" + "mvmctl/internal/lib/model" + "mvmctl/pkg/errs" +) + +// PolicyCreateInput is the raw input for a routed service-access policy. +type PolicyCreateInput struct { + SourceNetwork string `json:"source_network"` + DestinationVM string `json:"destination_vm"` + Protocol string `json:"protocol"` + DestinationPort string `json:"destination_port"` +} + +// Validate checks the typed policy shape and destination port bounds. +func (i *PolicyCreateInput) Validate() error { + if strings.TrimSpace(i.SourceNetwork) == "" { + return errs.New(errs.CodeValidationFailed, "source network is required") + } + if strings.TrimSpace(i.DestinationVM) == "" { + return errs.New(errs.CodeValidationFailed, "destination VM is required") + } + protocol := model.ServiceAccessPolicyProtocol(strings.ToLower(i.Protocol)) + if protocol != model.ServiceAccessPolicyProtocolTCP && protocol != model.ServiceAccessPolicyProtocolUDP { + return errs.New(errs.CodePolicyProtocolInvalid, "protocol must be tcp or udp") + } + _, _, err := parsePolicyPort(i.DestinationPort) + return err +} + +// ResolveSourceNetwork resolves the policy source network. +func (i *PolicyCreateInput) ResolveSourceNetwork( + ctx context.Context, + repo network.Repository, +) (*model.NetworkItem, error) { + if err := i.Validate(); err != nil { + return nil, err + } + networks, err := (&NetworkInput{Identifiers: []string{i.SourceNetwork}}).Resolve(ctx, repo) + if err != nil { + return nil, err + } + return networks[0], nil +} + +// ResolveDestinationVM resolves the exact destination VM. +func (i *PolicyCreateInput) ResolveDestinationVM( + ctx context.Context, + repo vm.Repository, +) (*model.VMItem, error) { + if err := i.Validate(); err != nil { + return nil, err + } + vms, err := (&VMInput{Identifiers: []string{i.DestinationVM}}).Resolve(ctx, repo) + if err != nil { + return nil, err + } + return vms[0], nil +} + +// ResolvedPort returns the validated typed protocol and bounded destination port range. +func (i *PolicyCreateInput) ResolvedPort() (model.ServiceAccessPolicyProtocol, int, int, error) { + if err := i.Validate(); err != nil { + return "", 0, 0, err + } + start, end, err := parsePolicyPort(i.DestinationPort) + return model.ServiceAccessPolicyProtocol(strings.ToLower(i.Protocol)), start, end, err +} + +func parsePolicyPort(value string) (int, int, error) { + parts := strings.Split(strings.TrimSpace(value), "-") + if len(parts) == 0 || len(parts) > 2 || parts[0] == "" { + return 0, 0, errs.New(errs.CodePolicyPortInvalid, "destination port must be a port or bounded range") + } + start, err := strconv.Atoi(parts[0]) + if err != nil || start < 1 || start > 65535 { + return 0, 0, errs.New(errs.CodePolicyPortInvalid, "destination port must be between 1 and 65535") + } + end := start + if len(parts) == 2 { + end, err = strconv.Atoi(parts[1]) + if err != nil || end < start || end > 65535 { + return 0, 0, errs.New( + errs.CodePolicyPortInvalid, + "destination port range end must be between its start and 65535", + ) + } + } + return start, end, nil +} + +// PolicyInput identifies existing policies by exact ID or ID prefix. +type PolicyInput struct { + Identifiers []string `json:"identifiers"` +} + +// Validate checks that at least one policy identifier was supplied. +func (i *PolicyInput) Validate() error { + if len(i.Identifiers) == 0 { + return errs.New(errs.CodeValidationFailed, "at least one policy identifier is required") + } + for _, identifier := range i.Identifiers { + if strings.TrimSpace(identifier) == "" { + return errs.New(errs.CodeValidationFailed, "policy identifiers must not be empty") + } + } + return nil +} + +// Resolve resolves policy IDs and rejects missing or ambiguous prefixes. +func (i *PolicyInput) Resolve( + ctx context.Context, + repo network.ServiceAccessPolicyRepository, +) ([]*model.ServiceAccessPolicy, error) { + if err := i.Validate(); err != nil { + return nil, err + } + policies := make([]*model.ServiceAccessPolicy, 0, len(i.Identifiers)) + seen := make(map[string]bool) + for _, identifier := range i.Identifiers { + matches, err := repo.FindByPrefix(ctx, identifier) + if err != nil { + return nil, errs.WrapMsg(errs.CodePolicyNotFound, "failed to resolve service-access policy", err) + } + if len(matches) == 0 { + return nil, errs.NotFound(errs.CodePolicyNotFound, "service-access policy not found", + errs.WithEntity(identifier)) + } + if len(matches) > 1 { + return nil, errs.New(errs.CodePolicyAmbiguous, + fmt.Sprintf("service-access policy identifier %q is ambiguous", identifier)) + } + if !seen[matches[0].ID] { + policies = append(policies, matches[0]) + seen[matches[0].ID] = true + } + } + return policies, nil +} diff --git a/pkg/api/inputs/policy_test.go b/pkg/api/inputs/policy_test.go new file mode 100644 index 00000000..ff4952bf --- /dev/null +++ b/pkg/api/inputs/policy_test.go @@ -0,0 +1,152 @@ +package inputs_test + +import ( + "context" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "mvmctl/internal/core/network" + "mvmctl/internal/lib/model" + "mvmctl/internal/testutil" + "mvmctl/pkg/api/inputs" + "mvmctl/pkg/errs" +) + +type policyResolverRepo struct{ items []*model.ServiceAccessPolicy } + +func (*policyResolverRepo) Create(context.Context, *model.ServiceAccessPolicy) error { return nil } +func (r *policyResolverRepo) Get(_ context.Context, id string) (*model.ServiceAccessPolicy, error) { + for _, item := range r.items { + if item.ID == id { + return item, nil + } + } + return nil, nil +} + +func (*policyResolverRepo) GetByIdentity( + context.Context, + *model.ServiceAccessPolicy, +) (*model.ServiceAccessPolicy, error) { + return nil, nil +} +func (r *policyResolverRepo) FindByPrefix(_ context.Context, prefix string) ([]*model.ServiceAccessPolicy, error) { + var matches []*model.ServiceAccessPolicy + for _, item := range r.items { + if len(item.ID) >= len(prefix) && item.ID[:len(prefix)] == prefix { + matches = append(matches, item) + } + } + return matches, nil +} +func (r *policyResolverRepo) List(context.Context) ([]*model.ServiceAccessPolicy, error) { + return r.items, nil +} +func (*policyResolverRepo) Delete(context.Context, string) error { return nil } +func (*policyResolverRepo) DeleteMany(context.Context, []string) error { return nil } +func (*policyResolverRepo) DeleteByVM(context.Context, string) error { return nil } +func (*policyResolverRepo) DeleteBySourceNetwork(context.Context, string) error { return nil } + +var _ network.ServiceAccessPolicyRepository = (*policyResolverRepo)(nil) + +func TestPolicyCreateInputValidationAndPortResolution(t *testing.T) { + t.Parallel() + tests := map[string]struct { + input inputs.PolicyCreateInput + protocol model.ServiceAccessPolicyProtocol + start, end int + code errs.Code + }{ + "TCP scalar": {input: inputs.PolicyCreateInput{SourceNetwork: "a", DestinationVM: "vm", Protocol: "TCP", + DestinationPort: "443"}, protocol: model.ServiceAccessPolicyProtocolTCP, start: 443, end: 443}, + "UDP range": {input: inputs.PolicyCreateInput{SourceNetwork: "a", DestinationVM: "vm", Protocol: "udp", + DestinationPort: "5000-5005"}, protocol: model.ServiceAccessPolicyProtocolUDP, start: 5000, end: 5005}, + "minimum": {input: inputs.PolicyCreateInput{SourceNetwork: "a", DestinationVM: "vm", Protocol: "tcp", + DestinationPort: "1"}, protocol: model.ServiceAccessPolicyProtocolTCP, start: 1, end: 1}, + "maximum": {input: inputs.PolicyCreateInput{SourceNetwork: "a", DestinationVM: "vm", Protocol: "udp", + DestinationPort: "65535"}, protocol: model.ServiceAccessPolicyProtocolUDP, start: 65535, end: 65535}, + "source missing": {input: inputs.PolicyCreateInput{DestinationVM: "vm", Protocol: "tcp", DestinationPort: "1"}, + code: errs.CodeValidationFailed}, + "destination missing": { + input: inputs.PolicyCreateInput{SourceNetwork: "a", Protocol: "tcp", DestinationPort: "1"}, + code: errs.CodeValidationFailed, + }, + "protocol invalid": {input: inputs.PolicyCreateInput{SourceNetwork: "a", DestinationVM: "vm", Protocol: "icmp", + DestinationPort: "1"}, code: errs.CodePolicyProtocolInvalid}, + "zero": {input: inputs.PolicyCreateInput{SourceNetwork: "a", DestinationVM: "vm", Protocol: "tcp", + DestinationPort: "0"}, code: errs.CodePolicyPortInvalid}, + "reversed": {input: inputs.PolicyCreateInput{SourceNetwork: "a", DestinationVM: "vm", Protocol: "tcp", + DestinationPort: "2-1"}, code: errs.CodePolicyPortInvalid}, + "too large": {input: inputs.PolicyCreateInput{SourceNetwork: "a", DestinationVM: "vm", Protocol: "tcp", + DestinationPort: "65536"}, code: errs.CodePolicyPortInvalid}, + "malformed": {input: inputs.PolicyCreateInput{SourceNetwork: "a", DestinationVM: "vm", Protocol: "tcp", + DestinationPort: "1-2-3"}, code: errs.CodePolicyPortInvalid}, + } + for name, tc := range tests { + t.Run(name, func(t *testing.T) { + protocol, start, end, err := tc.input.ResolvedPort() + if tc.code != "" { + require.Error(t, err) + de := errs.AsDomainError(err) + require.NotNil(t, de) + assert.Equal(t, tc.code, de.Code) + return + } + require.NoError(t, err) + assert.Equal(t, tc.protocol, protocol) + assert.Equal(t, tc.start, start) + assert.Equal(t, tc.end, end) + }) + } +} + +func TestPolicyInputResolve(t *testing.T) { + ctx := context.Background() + repo := &policyResolverRepo{items: []*model.ServiceAccessPolicy{{ID: "abc-111"}, {ID: "abc-222"}, {ID: "xyz-333"}}} + tests := map[string]struct { + identifiers []string + want []string + code errs.Code + }{ + "exact and prefix": {identifiers: []string{"abc-111", "xyz"}, want: []string{"abc-111", "xyz-333"}}, + "deduplicates": {identifiers: []string{"xyz", "xyz-333"}, want: []string{"xyz-333"}}, + "missing": {identifiers: []string{"none"}, code: errs.CodePolicyNotFound}, + "ambiguous": {identifiers: []string{"abc"}, code: errs.CodePolicyAmbiguous}, + "empty list": {code: errs.CodeValidationFailed}, + "blank identifier": {identifiers: []string{" "}, code: errs.CodeValidationFailed}, + } + for name, tc := range tests { + t.Run(name, func(t *testing.T) { + got, err := (&inputs.PolicyInput{Identifiers: tc.identifiers}).Resolve(ctx, repo) + if tc.code != "" { + require.Error(t, err) + assert.Equal(t, tc.code, errs.AsDomainError(err).Code) + return + } + require.NoError(t, err) + ids := make([]string, len(got)) + for i := range got { + ids[i] = got[i].ID + } + assert.Equal(t, tc.want, ids) + }) + } +} + +func TestPolicyCreateInputResolvesNetworkAndVM(t *testing.T) { + ctx := context.Background() + networkRepo := testutil.NewNetworkRepo() + vmRepo := testutil.NewVMRepo() + require.NoError(t, networkRepo.Upsert(ctx, &model.NetworkItem{ID: "net-id", Name: "source", IsPresent: true})) + require.NoError(t, vmRepo.Upsert(ctx, &model.VMItem{ID: "vm-id", Name: "destination"})) + input := &inputs.PolicyCreateInput{SourceNetwork: "source", DestinationVM: "destination", Protocol: "tcp", + DestinationPort: "443"} + source, err := input.ResolveSourceNetwork(ctx, networkRepo) + require.NoError(t, err) + destination, err := input.ResolveDestinationVM(ctx, vmRepo) + require.NoError(t, err) + assert.Equal(t, "net-id", source.ID) + assert.Equal(t, "vm-id", destination.ID) +} diff --git a/pkg/api/interfaces.go b/pkg/api/interfaces.go index 1fd0c83d..53621d74 100644 --- a/pkg/api/interfaces.go +++ b/pkg/api/interfaces.go @@ -22,4 +22,5 @@ type API interface { InitAPI SnapshotAPI UpdateAPI + PolicyAPI } diff --git a/pkg/api/network.go b/pkg/api/network.go index 82c24b3a..0a19fb77 100644 --- a/pkg/api/network.go +++ b/pkg/api/network.go @@ -4,6 +4,7 @@ package api import ( "context" "crypto/sha256" + "errors" "fmt" "log/slog" "mvmctl/internal/core/network" @@ -96,6 +97,9 @@ func (op *Operation) NetworkCreate(ctx context.Context, input inputs.NetworkCrea // Update bridge_active bridgeActive := libnet.DefaultNetOps.BridgeExists(ctx, resolved.Bridge) _ = op.Repos.Network.UpdateBridgeActive(ctx, networkID, bridgeActive) + if reconcileErr := op.reconcileServiceAccessPolicies(ctx); reconcileErr != nil { + return nil, op.rollbackCreatedNetwork(ctx, networkItem, reconcileErr) + } // Re-fetch updated, err := op.Repos.Network.GetByName(ctx, resolved.Name) if err != nil || updated == nil { @@ -121,7 +125,10 @@ func (op *Operation) NetworkRemove(ctx context.Context, input inputs.NetworkInpu return errs.WrapMsg(errs.CodeNetworkRemoveFailed, err.Error(), err) } // Batch-enrich with VM and snapshot references - op.Enr.EnrichNetwork(ctx, networks, "vm", "snapshots") + if err := op.Enr.EnrichNetwork(ctx, networks, "vm", "snapshots"); err != nil { + return errs.WrapMsg(errs.CodeNetworkRemoveFailed, + "failed to resolve network dependencies before removal", err) + } // service.Remove raises error on failure. // Return the first error encountered. for _, net := range networks { @@ -133,6 +140,20 @@ func (op *Operation) NetworkRemove(ctx context.Context, input inputs.NetworkInpu } return errs.WrapMsg(code, errorMsg, err) } + if err := op.Services.Network.InvalidateServiceAccessPoliciesBySourceNetwork(ctx, net.ID); err != nil { + return err + } + for _, destinationVM := range net.VMs { + if destinationVM == nil { + continue + } + if err := op.Services.Network.InvalidateServiceAccessPoliciesByVM(ctx, destinationVM.ID); err != nil { + return err + } + } + if err := op.reconcileServiceAccessPolicies(ctx); err != nil { + return err + } op.AuditLog.LogOperation("network.remove", map[string]any{"id": net.ID, "name": net.Name}, "") } return nil @@ -299,17 +320,16 @@ func (op *Operation) NetworkSync(ctx context.Context, input inputs.NetworkInput) } } + resolvedPolicies, activeNetworks, err := op.resolveServiceAccessPolicyState(ctx) + if err != nil { + return nil, err + } + // 2. Build the set of all networks to sync. When syncing a specific subset, // we must include ALL DB networks in the batch rebuild so that // unrequested networks' firewall rules are preserved in the kernel. // Stats are returned only for the requested networks. - allNetworks := requested - if len(input.Identifiers) > 0 { - allNetworks, err = op.Repos.Network.ListAll(ctx) - if err != nil { - return nil, errs.WrapMsg(errs.CodeDatabaseError, fmt.Sprintf("Failed to list all networks: %v", err), err) - } - } + allNetworks := activeNetworks // Capture batch results for stat extraction after the closure. var batchResults map[string]*network.SyncResult @@ -348,7 +368,7 @@ func (op *Operation) NetworkSync(ctx context.Context, input inputs.NetworkInput) } // Step 3: Sync firewall rules — all networks in a single batch var syncErr2 error - batchResults, syncErr2 = op.Services.Network.SyncIPTablesRulesBatch(ctx, allNetworks) + batchResults, syncErr2 = op.Services.Network.SyncIPTablesRulesBatch(ctx, allNetworks, resolvedPolicies) if syncErr2 != nil { return fmt.Errorf("sync firewall rules: %w", syncErr2) } @@ -383,6 +403,10 @@ func (op *Operation) NetworkPrune(ctx context.Context, dryRun bool, includeAll b if err != nil { return nil, errs.WrapMsg(errs.CodeDatabaseError, fmt.Sprintf("Failed to list networks: %v", err), err) } + if err := op.Enr.EnrichNetwork(ctx, networks, "vm", "snapshots"); err != nil { + return nil, errs.WrapMsg(errs.CodeNetworkRemoveFailed, + "failed to resolve network dependencies before pruning", err) + } // Get referenced network IDs from VMs allVMs, _ := op.Repos.VM.ListAll(ctx) referencedIDs := make(map[string]bool) @@ -409,7 +433,14 @@ func (op *Operation) NetworkPrune(ctx context.Context, dryRun bool, includeAll b } if !dryRun { if !network.IsPresent { - _ = op.Repos.Network.Delete(ctx, network.ID) + if err := op.Repos.Network.Delete(ctx, network.ID); err != nil { + slog.Warn("Failed to delete network", "name", network.Name, "error", err) + continue + } + if err := op.invalidateRemovedNetworkPolicies(ctx, network); err != nil { + slog.Warn("Failed to invalidate network policies", "name", network.Name, "error", err) + continue + } } else { err := op.NetworkRemove(ctx, inputs.NetworkInput{Identifiers: []string{network.Name}}, includeAll) if err != nil { @@ -423,6 +454,48 @@ func (op *Operation) NetworkPrune(ctx context.Context, dryRun bool, includeAll b return removed, nil } +func (op *Operation) invalidateRemovedNetworkPolicies(ctx context.Context, network *model.NetworkItem) error { + if err := op.Services.Network.InvalidateServiceAccessPoliciesBySourceNetwork(ctx, network.ID); err != nil { + return err + } + for _, destinationVM := range network.VMs { + if destinationVM == nil { + continue + } + if err := op.Services.Network.InvalidateServiceAccessPoliciesByVM(ctx, destinationVM.ID); err != nil { + return err + } + } + return op.reconcileServiceAccessPolicies(ctx) +} + +func (op *Operation) rollbackCreatedNetwork( + ctx context.Context, + network *model.NetworkItem, + reconcileErr error, +) error { + causes := []error{reconcileErr} + details := map[string]any{"reconciliation_error": reconcileErr.Error()} + if cleanupErr := op.Services.Network.Remove(ctx, network, true); cleanupErr != nil { + causes = append(causes, cleanupErr) + details["cleanup_error"] = cleanupErr.Error() + } + if deleteErr := op.Repos.Network.Delete(ctx, network.ID); deleteErr != nil { + causes = append(causes, deleteErr) + details["row_cleanup_error"] = deleteErr.Error() + } + if recoveryErr := op.reconcileServiceAccessPolicies(ctx); recoveryErr != nil { + causes = append(causes, recoveryErr) + details["recovery_error"] = recoveryErr.Error() + } + code := errs.CodeNetworkCreateFailed + var domainErr *errs.DomainError + if errors.As(reconcileErr, &domainErr) { + code = domainErr.Code + } + return errs.WrapMsg(code, reconcileErr.Error(), errors.Join(causes...), errs.WithDetails(details)) +} + // NetworkCreateDefaultNetwork creates the default network if it doesn't exist. // Updates Repository component tracking after creation. func (op *Operation) NetworkCreateDefaultNetwork(ctx context.Context) (*model.NetworkItem, error) { diff --git a/pkg/api/operation.go b/pkg/api/operation.go index 5af6c56e..1ab4451a 100644 --- a/pkg/api/operation.go +++ b/pkg/api/operation.go @@ -58,6 +58,7 @@ type Repos struct { Config config.SettingsRepository Vsock vsock.Repository Snapshot snapshot.Repository + Policy network.ServiceAccessPolicyRepository } // Services bundles all domain services. @@ -95,6 +96,7 @@ func NewOperation(ctx context.Context, conn *db.Handle, cacheDir string) *Operat Config: config.NewRepository(sqlDB), Vsock: vsock.NewRepository(sqlDB), Snapshot: snapshot.NewRepository(sqlDB), + Policy: network.NewServiceAccessPolicyRepository(sqlDB), } configReg := config.NewConstraintRegistry() config.RegisterBuiltinConstraints(configReg) @@ -104,7 +106,7 @@ func NewOperation(ctx context.Context, conn *db.Handle, cacheDir string) *Operat defaultFwTracker := firewall.NewFirewallTracker(model.FirewallBackendNFTables, true, sqlDB) s := Services{ - Network: network.NewService(r.Network, defaultFwTracker), + Network: network.NewService(r.Network, defaultFwTracker, r.Policy), Image: image.NewService(r.Image), Kernel: kernel.NewService(r.Kernel, cacheDir), Binary: binary.NewService(r.Binary, filepath.Join(cacheDir, "bin"), cacheDir), diff --git a/pkg/api/policy.go b/pkg/api/policy.go new file mode 100644 index 00000000..7b0932b4 --- /dev/null +++ b/pkg/api/policy.go @@ -0,0 +1,245 @@ +package api + +import ( + "context" + "errors" + "fmt" + "time" + + "mvmctl/internal/lib/crypto" + "mvmctl/internal/lib/model" + "mvmctl/pkg/api/inputs" + "mvmctl/pkg/api/results" + "mvmctl/pkg/errs" +) + +// PolicyAPI defines the public interface for routed service-access policies. +type PolicyAPI interface { + PolicyCreate(ctx context.Context, input inputs.PolicyCreateInput) (*results.Policy, error) + PolicyList(ctx context.Context) ([]*results.Policy, error) + PolicyInspect(ctx context.Context, input inputs.PolicyInput) (*results.Policy, error) + PolicyRemove(ctx context.Context, input inputs.PolicyInput) error + PolicySync(ctx context.Context) (*results.PolicySync, error) +} + +// PolicyCreate resolves resource identities, persists typed intent, and reconciles firewall state. +func (op *Operation) PolicyCreate(ctx context.Context, input inputs.PolicyCreateInput) (*results.Policy, error) { + source, err := input.ResolveSourceNetwork(ctx, op.Repos.Network) + if err != nil { + return nil, err + } + destination, err := input.ResolveDestinationVM(ctx, op.Repos.VM) + if err != nil { + return nil, err + } + if source.ID == destination.NetworkID { + return nil, errs.New(errs.CodePolicySameNetwork, + "source network and destination VM network must differ; same-network enforcement is not available") + } + protocol, portStart, portEnd, err := input.ResolvedPort() + if err != nil { + return nil, err + } + now := time.Now().Format(time.RFC3339) + policy := &model.ServiceAccessPolicy{ + ID: crypto.ContentHash(source.ID, destination.ID, string(protocol), + fmt.Sprintf("%d", portStart), fmt.Sprintf("%d", portEnd), now), + SourceNetworkID: source.ID, + DestinationVMID: destination.ID, + Protocol: protocol, + DestinationPortStart: portStart, + DestinationPortEnd: portEnd, + CreatedAt: now, + UpdatedAt: now, + } + if err := op.Services.Network.CreateServiceAccessPolicy(ctx, policy); err != nil { + return nil, err + } + if reconcileErr := op.reconcileServiceAccessPolicies(ctx); reconcileErr != nil { + rollbackErr := op.Services.Network.RemoveServiceAccessPolicies(ctx, []string{policy.ID}) + recoveryErr := op.reconcileServiceAccessPolicies(ctx) + causes := []error{reconcileErr} + details := map[string]any{"reconciliation_error": reconcileErr.Error()} + if rollbackErr != nil { + causes = append(causes, rollbackErr) + details["rollback_error"] = rollbackErr.Error() + } + if recoveryErr != nil { + causes = append(causes, recoveryErr) + details["recovery_error"] = recoveryErr.Error() + } + return nil, errs.WrapMsg(errs.CodePolicyCreateFailed, reconcileErr.Error(), errors.Join(causes...), + errs.WithDetails(details)) + } + op.AuditLog.LogOperation("policy.create", map[string]any{"id": policy.ID}, "") + return op.policyResult(ctx, policy) +} + +// PolicyList returns all policy intent with current resource names. +func (op *Operation) PolicyList(ctx context.Context) ([]*results.Policy, error) { + policies, err := op.Repos.Policy.List(ctx) + if err != nil { + return nil, errs.WrapMsg(errs.CodePolicyListFailed, "failed to list service-access policies", err) + } + items := make([]*results.Policy, 0, len(policies)) + for _, policy := range policies { + item, err := op.policyResult(ctx, policy) + if err != nil { + return nil, err + } + items = append(items, item) + } + return items, nil +} + +// PolicyInspect returns one policy resolved by exact ID or unambiguous prefix. +func (op *Operation) PolicyInspect(ctx context.Context, input inputs.PolicyInput) (*results.Policy, error) { + policies, err := input.Resolve(ctx, op.Repos.Policy) + if err != nil { + return nil, err + } + if len(policies) != 1 { + return nil, errs.New(errs.CodePolicyAmbiguous, "expected exactly one service-access policy") + } + return op.policyResult(ctx, policies[0]) +} + +// PolicyRemove deletes policy intent and reconciles firewall state immediately. +func (op *Operation) PolicyRemove(ctx context.Context, input inputs.PolicyInput) error { + policies, err := input.Resolve(ctx, op.Repos.Policy) + if err != nil { + return err + } + policyIDs := make([]string, len(policies)) + for i, policy := range policies { + policyIDs[i] = policy.ID + } + if err := op.Services.Network.RemoveServiceAccessPolicies(ctx, policyIDs); err != nil { + return err + } + for _, policy := range policies { + op.AuditLog.LogOperation("policy.remove", map[string]any{"id": policy.ID}, "") + } + return op.reconcileServiceAccessPolicies(ctx) +} + +// PolicySync recompiles all intent from current resource identities and reconciles firewall state. +func (op *Operation) PolicySync(ctx context.Context) (*results.PolicySync, error) { + if err := op.reconcileServiceAccessPolicies(ctx); err != nil { + return nil, err + } + policies, err := op.Repos.Policy.List(ctx) + if err != nil { + return nil, errs.WrapMsg(errs.CodePolicySyncFailed, "failed to count service-access policies", err) + } + return &results.PolicySync{Policies: len(policies)}, nil +} + +func (op *Operation) reconcileServiceAccessPolicies(ctx context.Context) error { + policies, networks, err := op.resolveServiceAccessPolicyState(ctx) + if err != nil { + return err + } + return op.Services.Network.ReconcileServiceAccessPolicies(ctx, policies, networks) +} + +func (op *Operation) resolveServiceAccessPolicyState( + ctx context.Context, +) ([]*model.ResolvedServiceAccessPolicy, []*model.NetworkItem, error) { + policies, err := op.Repos.Policy.List(ctx) + if err != nil { + return nil, nil, errs.WrapMsg(errs.CodeDatabaseError, "failed to list service-access policies", err) + } + networks, err := op.Repos.Network.ListAll(ctx) + if err != nil { + return nil, nil, errs.WrapMsg(errs.CodeDatabaseError, "failed to list policy networks", err) + } + activeNetworks := make([]*model.NetworkItem, 0, len(networks)) + networkByID := make(map[string]*model.NetworkItem, len(networks)) + for _, network := range networks { + if network.DeletedAt != nil { + continue + } + activeNetworks = append(activeNetworks, network) + networkByID[network.ID] = network + } + resolved := make([]*model.ResolvedServiceAccessPolicy, 0, len(policies)) + for _, policy := range policies { + source := networkByID[policy.SourceNetworkID] + if source == nil { + return nil, nil, errs.NotFound(errs.CodeNetworkNotFound, + "policy source network no longer exists", errs.WithEntity(policy.SourceNetworkID)) + } + destination, err := op.Repos.VM.Get(ctx, policy.DestinationVMID) + if err != nil { + return nil, nil, errs.WrapMsg(errs.CodeDatabaseError, "failed to resolve policy destination VM", err) + } + if destination == nil { + return nil, nil, errs.NotFound(errs.CodeVMNotFound, + "policy destination VM no longer exists", errs.WithEntity(policy.DestinationVMID)) + } + destinationNetwork := networkByID[destination.NetworkID] + if destinationNetwork == nil { + return nil, nil, errs.NotFound(errs.CodeNetworkNotFound, + "policy destination network no longer exists", errs.WithEntity(destination.NetworkID)) + } + if source.ID == destinationNetwork.ID { + return nil, nil, errs.New(errs.CodePolicySameNetwork, + "persisted policy source and current destination network must differ", + errs.WithEntity(policy.ID)) + } + resolved = append(resolved, &model.ResolvedServiceAccessPolicy{ + ServiceAccessPolicy: *policy, + SourceNetworkName: source.Name, + SourceNetworkBridge: source.Bridge, + DestinationVMName: destination.Name, + DestinationIPv4: destination.IPv4, + DestinationNetworkID: destinationNetwork.ID, + DestinationNetworkName: destinationNetwork.Name, + DestinationNetworkBridge: destinationNetwork.Bridge, + }) + } + return resolved, activeNetworks, nil +} + +func (op *Operation) policyResult(ctx context.Context, policy *model.ServiceAccessPolicy) (*results.Policy, error) { + source, err := op.Repos.Network.Get(ctx, policy.SourceNetworkID) + if err != nil { + return nil, errs.WrapMsg(errs.CodeDatabaseError, "failed to resolve policy source network", err) + } + if source == nil { + return nil, errs.NotFound(errs.CodeNetworkNotFound, "policy source network no longer exists") + } + destination, err := op.Repos.VM.Get(ctx, policy.DestinationVMID) + if err != nil { + return nil, errs.WrapMsg(errs.CodeDatabaseError, "failed to resolve policy destination VM", err) + } + if destination == nil { + return nil, errs.NotFound(errs.CodeVMNotFound, "policy destination VM no longer exists") + } + destinationNetwork, err := op.Repos.Network.Get(ctx, destination.NetworkID) + if err != nil { + return nil, errs.WrapMsg(errs.CodeDatabaseError, "failed to resolve policy destination network", err) + } + if destinationNetwork == nil { + return nil, errs.NotFound(errs.CodeNetworkNotFound, "policy destination network no longer exists") + } + if source.ID == destinationNetwork.ID { + return nil, errs.New(errs.CodePolicySameNetwork, + "persisted policy source and current destination network must differ", errs.WithEntity(policy.ID)) + } + return &results.Policy{ + ID: policy.ID, + SourceNetworkID: source.ID, + SourceNetworkName: source.Name, + DestinationVMID: destination.ID, + DestinationVMName: destination.Name, + DestinationNetworkID: destinationNetwork.ID, + DestinationNetworkName: destinationNetwork.Name, + Protocol: policy.Protocol, + DestinationPortStart: policy.DestinationPortStart, + DestinationPortEnd: policy.DestinationPortEnd, + CreatedAt: policy.CreatedAt, + UpdatedAt: policy.UpdatedAt, + }, nil +} diff --git a/pkg/api/policy_test.go b/pkg/api/policy_test.go new file mode 100644 index 00000000..7020f825 --- /dev/null +++ b/pkg/api/policy_test.go @@ -0,0 +1,190 @@ +package api_test + +import ( + "context" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "mvmctl/internal/core/network" + "mvmctl/internal/lib/logging" + "mvmctl/internal/lib/model" + "mvmctl/internal/testutil" + "mvmctl/pkg/api" + "mvmctl/pkg/api/inputs" + "mvmctl/pkg/errs" +) + +type memoryPolicyRepo struct { + items []*model.ServiceAccessPolicy + deleted []string +} + +func (r *memoryPolicyRepo) Create(_ context.Context, policy *model.ServiceAccessPolicy) error { + r.items = append(r.items, policy) + return nil +} +func (r *memoryPolicyRepo) Get(_ context.Context, id string) (*model.ServiceAccessPolicy, error) { + for _, item := range r.items { + if item.ID == id { + return item, nil + } + } + return nil, nil +} + +func (r *memoryPolicyRepo) GetByIdentity( + _ context.Context, + policy *model.ServiceAccessPolicy, +) (*model.ServiceAccessPolicy, error) { + for _, item := range r.items { + if item.SourceNetworkID == policy.SourceNetworkID && item.DestinationVMID == policy.DestinationVMID && + item.Protocol == policy.Protocol && item.DestinationPortStart == policy.DestinationPortStart && + item.DestinationPortEnd == policy.DestinationPortEnd { + return item, nil + } + } + return nil, nil +} +func (r *memoryPolicyRepo) FindByPrefix(_ context.Context, prefix string) ([]*model.ServiceAccessPolicy, error) { + var found []*model.ServiceAccessPolicy + for _, item := range r.items { + if len(item.ID) >= len(prefix) && item.ID[:len(prefix)] == prefix { + found = append(found, item) + } + } + return found, nil +} +func (r *memoryPolicyRepo) List(context.Context) ([]*model.ServiceAccessPolicy, error) { + return r.items, nil +} +func (r *memoryPolicyRepo) Delete(ctx context.Context, id string) error { + return r.DeleteMany(ctx, []string{id}) +} +func (r *memoryPolicyRepo) DeleteMany(_ context.Context, ids []string) error { + r.deleted = append([]string(nil), ids...) + remove := make(map[string]bool, len(ids)) + for _, id := range ids { + remove[id] = true + } + kept := r.items[:0] + for _, item := range r.items { + if !remove[item.ID] { + kept = append(kept, item) + } + } + r.items = kept + return nil +} +func (r *memoryPolicyRepo) DeleteByVM(_ context.Context, id string) error { + kept := r.items[:0] + for _, item := range r.items { + if item.DestinationVMID != id { + kept = append(kept, item) + } + } + r.items = kept + return nil +} +func (r *memoryPolicyRepo) DeleteBySourceNetwork(_ context.Context, id string) error { + kept := r.items[:0] + for _, item := range r.items { + if item.SourceNetworkID != id { + kept = append(kept, item) + } + } + r.items = kept + return nil +} + +func newPolicyOperation(t *testing.T, sameNetwork bool) (*api.Operation, *memoryPolicyRepo) { + t.Helper() + ctx := context.Background() + networkRepo := testutil.NewNetworkRepo() + vmRepo := testutil.NewVMRepo() + require.NoError(t, networkRepo.Upsert(ctx, &model.NetworkItem{ID: "source-id", Name: "source", Bridge: "mvm-source", + IsPresent: true})) + require.NoError(t, networkRepo.Upsert(ctx, &model.NetworkItem{ID: "destination-id", Name: "destination", + Bridge: "mvm-destination", IsPresent: true})) + destinationNetwork := "destination-id" + if sameNetwork { + destinationNetwork = "source-id" + } + require.NoError( + t, + vmRepo.Upsert(ctx, &model.VMItem{ID: "vm-id", Name: "destination-vm", NetworkID: destinationNetwork, + IPv4: "10.2.0.2"}), + ) + policyRepo := &memoryPolicyRepo{} + service := network.NewService(networkRepo, nil, policyRepo) + return &api.Operation{ + Repos: api.Repos{Network: networkRepo, VM: vmRepo, Policy: policyRepo}, + Services: api.Services{Network: service}, + AuditLog: &logging.AuditLog{}, + }, policyRepo +} + +func TestPolicyCreateRejectsSameNetwork(t *testing.T) { + op, repo := newPolicyOperation(t, true) + _, err := op.PolicyCreate( + context.Background(), + inputs.PolicyCreateInput{SourceNetwork: "source", DestinationVM: "destination-vm", + Protocol: "tcp", DestinationPort: "443"}, + ) + require.Error(t, err) + assert.Equal(t, errs.CodePolicySameNetwork, errs.AsDomainError(err).Code) + assert.Empty(t, repo.items) +} + +func TestPolicyCreateRollsBackIntentWhenKernelSyncFails(t *testing.T) { + op, repo := newPolicyOperation(t, false) + _, err := op.PolicyCreate( + context.Background(), + inputs.PolicyCreateInput{SourceNetwork: "source", DestinationVM: "destination-vm", + Protocol: "tcp", DestinationPort: "443"}, + ) + require.Error(t, err) + assert.Equal(t, errs.CodePolicyCreateFailed, errs.AsDomainError(err).Code) + assert.Empty(t, repo.items, "failed reconciliation must roll back persisted intent") + require.Len(t, repo.deleted, 1) +} + +func TestPolicyRemoveResolvesAllBeforeTransactionalDelete(t *testing.T) { + op, repo := newPolicyOperation(t, false) + repo.items = []*model.ServiceAccessPolicy{{ID: "aaa-111"}, {ID: "bbb-222"}} + err := op.PolicyRemove(context.Background(), inputs.PolicyInput{Identifiers: []string{"aaa", "bbb"}}) + require.Error(t, err, "nil firewall tracker makes post-delete reconciliation fail") + assert.Equal(t, []string{"aaa-111", "bbb-222"}, repo.deleted) + assert.Empty(t, repo.items) +} + +func TestPolicySyncReportsStaleResources(t *testing.T) { + op, repo := newPolicyOperation(t, false) + repo.items = []*model.ServiceAccessPolicy{{ID: "stale", SourceNetworkID: "missing", DestinationVMID: "vm-id", + Protocol: model.ServiceAccessPolicyProtocolTCP, DestinationPortStart: 80, DestinationPortEnd: 80}} + _, err := op.PolicySync(context.Background()) + require.Error(t, err) + assert.Equal(t, errs.CodeNetworkNotFound, errs.AsDomainError(err).Code) + + repo.items[0].SourceNetworkID = "source-id" + repo.items[0].DestinationVMID = "missing-vm" + _, err = op.PolicySync(context.Background()) + require.Error(t, err) + assert.Equal(t, errs.CodeVMNotFound, errs.AsDomainError(err).Code) +} + +func TestPolicyListAndInspectResolveCurrentNames(t *testing.T) { + op, repo := newPolicyOperation(t, false) + repo.items = []*model.ServiceAccessPolicy{{ID: "policy-123", SourceNetworkID: "source-id", DestinationVMID: "vm-id", + Protocol: model.ServiceAccessPolicyProtocolUDP, DestinationPortStart: 53, DestinationPortEnd: 53}} + items, err := op.PolicyList(context.Background()) + require.NoError(t, err) + require.Len(t, items, 1) + assert.Equal(t, "source", items[0].SourceNetworkName) + assert.Equal(t, "destination-vm", items[0].DestinationVMName) + assert.Equal(t, "destination", items[0].DestinationNetworkName) + inspected, err := op.PolicyInspect(context.Background(), inputs.PolicyInput{Identifiers: []string{"policy"}}) + require.NoError(t, err) + assert.Equal(t, items[0], inspected) +} diff --git a/pkg/api/results/policy.go b/pkg/api/results/policy.go new file mode 100644 index 00000000..3b810ab9 --- /dev/null +++ b/pkg/api/results/policy.go @@ -0,0 +1,24 @@ +package results + +import "mvmctl/internal/lib/model" + +// Policy describes typed intent with current resource names for display. +type Policy struct { + ID string `json:"id"` + SourceNetworkID string `json:"source_network_id"` + SourceNetworkName string `json:"source_network_name"` + DestinationVMID string `json:"destination_vm_id"` + DestinationVMName string `json:"destination_vm_name"` + DestinationNetworkID string `json:"destination_network_id"` + DestinationNetworkName string `json:"destination_network_name"` + Protocol model.ServiceAccessPolicyProtocol `json:"protocol"` + DestinationPortStart int `json:"destination_port_start"` + DestinationPortEnd int `json:"destination_port_end"` + CreatedAt string `json:"created_at"` + UpdatedAt string `json:"updated_at"` +} + +// PolicySync summarizes a policy reconciliation. +type PolicySync struct { + Policies int `json:"policies"` +} diff --git a/pkg/api/vm.go b/pkg/api/vm.go index 88de4051..3925b9c4 100644 --- a/pkg/api/vm.go +++ b/pkg/api/vm.go @@ -868,6 +868,20 @@ func (op *Operation) VMRemove(ctx context.Context, input inputs.VMInput) *errs.B // Delete from DB if err := repo.Delete(ctx, vmLocal.ID); err != nil { slog.Warn("failed to delete VM from DB", "vm", vmLocal.Name, "error", err) + results = append(results, errs.OperationResult{ + Status: "error", Code: string(errs.CodeDatabaseError), Item: vmLocal, + Message: fmt.Sprintf("failed to delete VM '%s': %v", vmLocal.Name, err), Exception: err, + }) + continue + } + if err := op.reconcileServiceAccessPolicies(ctx); err != nil { + results = append(results, errs.OperationResult{ + Status: "error", Code: string(errs.CodePolicySyncFailed), Item: vmLocal, + Message: fmt.Sprintf("failed to reconcile service-access policies after removing VM '%s': %v", + vmLocal.Name, err), + Exception: err, + }) + continue } if vmDir != "" { os.RemoveAll(vmDir) diff --git a/pkg/errs/codes.go b/pkg/errs/codes.go index 5b96e98a..f15d3ecc 100644 --- a/pkg/errs/codes.go +++ b/pkg/errs/codes.go @@ -38,6 +38,18 @@ const ( CodeNetworkDefaultSetFailed Code = "network.default_set_failed" CodeNetworkDefaultCreateFailed Code = "network.default_created_failed" + // --- Policy domain --- + CodePolicyNotFound Code = "policy.not_found" + CodePolicyAmbiguous Code = "policy.ambiguous" + CodePolicyAlreadyExists Code = "policy.already_exists" + CodePolicyProtocolInvalid Code = "policy.protocol.invalid" + CodePolicyPortInvalid Code = "policy.port.invalid" + CodePolicySameNetwork Code = "policy.same_network" + CodePolicyCreateFailed Code = "policy.create.failed" + CodePolicyListFailed Code = "policy.list.failed" + CodePolicyRemoveFailed Code = "policy.remove.failed" + CodePolicySyncFailed Code = "policy.sync.failed" + // --- Image domain --- CodeImageNotFound Code = "image.not_found" CodeImageAlreadyExists Code = "image.already_exists" diff --git a/pkg/errs/domain.go b/pkg/errs/domain.go index 81a69e59..0db647eb 100644 --- a/pkg/errs/domain.go +++ b/pkg/errs/domain.go @@ -123,6 +123,17 @@ var codeClassMap = map[Code]Class{ CodeNetworkDefaultSetFailed: ClassInternal, CodeNetworkDefaultCreateFailed: ClassInternal, + CodePolicyNotFound: ClassValidation, + CodePolicyAmbiguous: ClassValidation, + CodePolicyAlreadyExists: ClassConflict, + CodePolicyProtocolInvalid: ClassValidation, + CodePolicyPortInvalid: ClassValidation, + CodePolicySameNetwork: ClassValidation, + CodePolicyCreateFailed: ClassInternal, + CodePolicyListFailed: ClassInternal, + CodePolicyRemoveFailed: ClassInternal, + CodePolicySyncFailed: ClassInternal, + // --- Image domain --- CodeImageNotFound: ClassValidation, CodeImageAlreadyExists: ClassConflict, @@ -301,6 +312,17 @@ var codeOpMap = map[Code]string{ CodeNetworkDefaultSetFailed: "network", CodeNetworkDefaultCreateFailed: "network", + CodePolicyNotFound: "policy", + CodePolicyAmbiguous: "policy", + CodePolicyAlreadyExists: "policy", + CodePolicyProtocolInvalid: "policy", + CodePolicyPortInvalid: "policy", + CodePolicySameNetwork: "policy", + CodePolicyCreateFailed: "policy", + CodePolicyListFailed: "policy", + CodePolicyRemoveFailed: "policy", + CodePolicySyncFailed: "policy", + CodeImageNotFound: "image", CodeImageAlreadyExists: "image", CodeImagePullFailed: "image", @@ -605,7 +627,7 @@ func IsNotFound(err error) bool { CodeKernelNotFound, CodeBinaryNotFound, CodeVolumeNotFound, CodeKeyNotFound, CodeCPSourceNotFound, CodeBundledAssetNotFound, CodeFirecrackerSocketNotFound, CodeLoopMountBinaryNotFound, - CodeCPVMNotFound, CodeSnapshotNotFound: + CodeCPVMNotFound, CodeSnapshotNotFound, CodePolicyNotFound: return true } } diff --git a/scripts/run-system-tests.py b/scripts/run-system-tests.py index 366d50ec..b9bdd27c 100755 --- a/scripts/run-system-tests.py +++ b/scripts/run-system-tests.py @@ -102,6 +102,7 @@ "logs": ["tests/system/logs/test_logs.py"], "full_journeys": ["tests/system/full_journeys/test_full_journeys.py"], "nftables": ["tests/system/network/test_nftables.py"], + "policies": ["tests/system/network/test_policies.py"], } # Tier 3: directly on host (no runner VM) diff --git a/tests/system/COVERAGE_MATRIX.md b/tests/system/COVERAGE_MATRIX.md index 372dd852..5adb4421 100644 --- a/tests/system/COVERAGE_MATRIX.md +++ b/tests/system/COVERAGE_MATRIX.md @@ -125,6 +125,7 @@ removed, or when test coverage changes. | `network rm` rejects with active VM | ✅ Deep | `network/test_network.py` | `TestNetworkVMDependency` | L3: rm fails without --force when VM uses network; inspect shows VM count | | nftables firewall backend | ✅ Deep | `network/test_nftables.py` | `TestNFTablesFirewallBackend` | L3: set nftables backend, create VM, SSH, ping, verify nftables rules, cleanup, reset to iptables | | nftables atomic rule sync | ✅ Deep | `network/test_nftables.py` | `TestAtomicRuleSync` | L3: batch_ensure_rules idempotent, conntrack rule present, rule count stable, MASQUERADE persists | +| Routed service-access policies | ✅ Deep | `network/test_policies.py` | `test_routed_service_access_policy_specification` | L2: nftables and iptables default deny, exact VM/port-range allow, NoCloud precedence, host deny, internet egress, sync recovery, and deletion invalidation | --- diff --git a/tests/system/network/test_policies.py b/tests/system/network/test_policies.py new file mode 100644 index 00000000..af4089e0 --- /dev/null +++ b/tests/system/network/test_policies.py @@ -0,0 +1,327 @@ +"""Tier-2 routed service-access policy specification.""" + +from __future__ import annotations + +import json +import uuid + +import pytest + +from tests.system.conftest import _guest_run, _run_mvm, _unique_subnet, ensure_vm_deps + +pytestmark = [ + pytest.mark.system, + pytest.mark.domain_network, + pytest.mark.requires_kvm, + pytest.mark.requires_network, + pytest.mark.slow, + pytest.mark.tier2, +] + + +def _vm_ip(runner_vm: str, name: str) -> str: + data = json.loads(_run_mvm(runner_vm, "vm", "inspect", name, "--json").stdout) + item = data.get("vm", data) + ip = item.get("ipv4") + assert isinstance(ip, str) and ip + return ip + + +def _guest_probe(runner_vm: str, source: str, destination: str, port: int) -> bool: + result = _run_mvm( + runner_vm, + "exec", + source, + "--user", + "root", + "--", + f"nc -z -w 2 {destination} {port}", + check=False, + timeout=15, + ) + return result.returncode == 0 + + +def _chain(runner_vm: str, backend: str, chain: str) -> str: + if backend == "nftables": + command = f"sudo -n nft list chain ip filter {chain}" + else: + command = f"sudo -n iptables -t filter -S {chain}" + result = _guest_run(runner_vm, command, check=False, timeout=15) + assert result.returncode == 0, result.stderr + return result.stdout + + +@pytest.mark.parametrize("backend", ["nftables", "iptables"]) +def test_routed_service_access_policy_specification( + runner_vm: str, backend: str +) -> None: + """Default-deny, exact allow, sync recovery, and invalidation for each backend.""" + suffix = uuid.uuid4().hex[:7] + source_net = f"pol-src-{suffix}" + destination_net = f"pol-dst-{suffix}" + other_net = f"pol-other-{suffix}" + source_vm = f"pol-src-vm-{suffix}" + destination_vm = f"pol-dst-vm-{suffix}" + other_vm = f"pol-other-vm-{suffix}" + policy_id = "" + + original = _run_mvm( + runner_vm, + "config", + "get", + "settings", + "firewall_backend", + check=False, + ).stdout + original_backend = "nftables" + if "=" in original: + configured = original.rsplit("=", 1)[-1].strip() + if configured in {"nftables", "iptables"}: + original_backend = configured + + try: + _run_mvm( + runner_vm, + "config", + "set", + "settings", + "firewall_backend", + backend, + ) + ensure_vm_deps(runner_vm) + for name in (source_net, destination_net, other_net): + _run_mvm( + runner_vm, + "network", + "create", + name, + "--subnet", + _unique_subnet(name), + "--non-interactive", + ) + for vm_name, net_name in ((destination_vm, destination_net), (other_vm, other_net)): + _run_mvm( + runner_vm, + "vm", + "create", + vm_name, + "--image", + "alpine:3.23", + "--network", + net_name, + "--cloud-init-mode", + "inject", + timeout=180, + ) + _run_mvm( + runner_vm, + "vm", + "create", + source_vm, + "--image", + "alpine:3.23", + "--network", + source_net, + "--cloud-init-mode", + "net", + "--nocloud-net-port", + "9999", + timeout=180, + ) + + destination_ip = _vm_ip(runner_vm, destination_vm) + other_ip = _vm_ip(runner_vm, other_vm) + source_inspect = json.loads( + _run_mvm(runner_vm, "vm", "inspect", source_vm, "--json").stdout + ).get("vm", {}) + source_network = json.loads( + _run_mvm(runner_vm, "network", "inspect", source_net, "--json").stdout + ).get("network", {}) + source_gateway = source_network["ipv4_gateway"] + for vm_name in (destination_vm, other_vm): + _run_mvm( + runner_vm, + "exec", + vm_name, + "--user", + "root", + "--", + "sh -c 'nohup nc -lk -p 8080 >/tmp/nc.log 2>&1 &'", + timeout=15, + ) + + assert not _guest_probe(runner_vm, source_vm, destination_ip, 8080) + assert not _guest_probe(runner_vm, source_vm, other_ip, 8080) + + created = _run_mvm( + runner_vm, + "policy", + "create", + source_net, + destination_vm, + "tcp", + "8080-8081", + ) + listed = json.loads(_run_mvm(runner_vm, "policy", "ls", "--json").stdout) + matching = [ + item + for item in listed + if item["source_network_name"] == source_net + and item["destination_vm_name"] == destination_vm + ] + assert len(matching) == 1, created.stdout + policy_id = matching[0]["id"] + assert matching[0]["protocol"] == "tcp" + assert matching[0]["destination_port_start"] == 8080 + assert matching[0]["destination_port_end"] == 8081 + + routed = _chain(runner_vm, backend, "MVM-ROUTED-POLICY") + assert destination_ip in routed + assert "8080" in routed and "8081" in routed + assert "DROP" in routed.upper() + host_input = _chain(runner_vm, backend, "MVM-HOST-INPUT") + assert "DROP" in host_input.upper() + nocloud_input = _chain(runner_vm, backend, "MVM-NOCLOUDNET-INPUT") + assert "9999" in nocloud_input + if backend == "nftables": + parent = _guest_run( + runner_vm, + "sudo -n nft list chain ip filter INPUT", + timeout=15, + ).stdout + else: + parent = _guest_run( + runner_vm, + "sudo -n iptables -t filter -S INPUT", + timeout=15, + ).stdout + assert parent.index("MVM-NOCLOUDNET-INPUT") < parent.index("MVM-HOST-INPUT") + + host_denied = _guest_probe(runner_vm, source_vm, source_gateway, 22) + assert not host_denied + nocloud = _run_mvm( + runner_vm, + "exec", + source_vm, + "--user", + "root", + "--", + ( + f"wget -qO- http://{source_gateway}:9999/meta-data || " + f"wget -qO- http://{source_gateway}:9999/{source_inspect['id']}/meta-data" + ), + check=False, + timeout=15, + ) + assert nocloud.returncode == 0, nocloud.stderr + + assert _guest_probe(runner_vm, source_vm, destination_ip, 8080) + assert not _guest_probe(runner_vm, source_vm, destination_ip, 8082) + assert not _guest_probe(runner_vm, source_vm, other_ip, 8080) + internet = _run_mvm( + runner_vm, + "exec", + source_vm, + "--user", + "root", + "--", + "ping -c 1 -W 3 1.1.1.1", + check=False, + timeout=15, + ) + assert internet.returncode == 0, internet.stderr + + if backend == "nftables": + flush = "sudo -n nft flush chain ip filter MVM-ROUTED-POLICY" + else: + flush = "sudo -n iptables -t filter -F MVM-ROUTED-POLICY" + _guest_run(runner_vm, flush, timeout=15) + assert destination_ip not in _chain( + runner_vm, backend, "MVM-ROUTED-POLICY" + ) + sync = json.loads(_run_mvm(runner_vm, "policy", "sync", "--json").stdout) + assert sync["policies"] == 1 + assert destination_ip in _chain(runner_vm, backend, "MVM-ROUTED-POLICY") + _guest_run(runner_vm, flush, timeout=15) + _run_mvm(runner_vm, "network", "sync", "--json") + assert destination_ip in _chain(runner_vm, backend, "MVM-ROUTED-POLICY") + assert _guest_probe(runner_vm, source_vm, destination_ip, 8080) + + duplicate = _run_mvm( + runner_vm, + "policy", + "create", + source_net, + destination_vm, + "tcp", + "8080-8081", + check=False, + ) + assert duplicate.returncode != 0 + same_network = _run_mvm( + runner_vm, + "policy", + "create", + destination_net, + destination_vm, + "tcp", + "80", + check=False, + ) + assert same_network.returncode != 0 + missing = _run_mvm( + runner_vm, "policy", "inspect", "does-not-exist", "--json", check=False + ) + assert missing.returncode != 0 + + _run_mvm(runner_vm, "policy", "rm", policy_id[:12], "--force") + assert not any( + item["id"] == policy_id + for item in json.loads( + _run_mvm(runner_vm, "policy", "ls", "--json").stdout + ) + ) + assert not _guest_probe(runner_vm, source_vm, destination_ip, 8080) + + recreated = _run_mvm( + runner_vm, + "policy", + "create", + source_net, + destination_vm, + "udp", + "5353", + ) + assert recreated.returncode == 0 + _run_mvm(runner_vm, "vm", "rm", destination_vm, "--force", timeout=120) + remaining = json.loads(_run_mvm(runner_vm, "policy", "ls", "--json").stdout) + assert not any(item["destination_vm_name"] == destination_vm for item in remaining) + + _run_mvm( + runner_vm, + "policy", + "create", + source_net, + other_vm, + "tcp", + "8080", + ) + _run_mvm(runner_vm, "vm", "rm", source_vm, "--force", timeout=120) + _run_mvm(runner_vm, "network", "rm", source_net, "--force") + remaining = json.loads(_run_mvm(runner_vm, "policy", "ls", "--json").stdout) + assert not any(item["source_network_name"] == source_net for item in remaining) + finally: + for vm_name in (source_vm, destination_vm, other_vm): + _run_mvm(runner_vm, "vm", "rm", vm_name, "--force", check=False, timeout=120) + for net_name in (source_net, destination_net, other_net): + _run_mvm(runner_vm, "network", "rm", net_name, "--force", check=False) + _run_mvm( + runner_vm, + "config", + "set", + "settings", + "firewall_backend", + original_backend, + check=False, + ) From ee733232a2dc23c1edb372022a2add71af36f44c Mon Sep 17 00:00:00 2001 From: AlanD20 Date: Sun, 9 Aug 2026 21:08:11 +0200 Subject: [PATCH 0005/1457] fix(vm): verify jailed cgroups at the actual v2 path --- CHANGELOG.md | 4 ++++ docs/RUNTIME.md | 2 +- internal/core/vm/jailer.go | 4 ++-- internal/infra/constants.go | 3 --- internal/service/jailer/entry.go | 1 - internal/service/jailer/entry_test.go | 2 +- tests/system/vm/test_cgroup.py | 4 ++-- 7 files changed, 10 insertions(+), 10 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 894fe380..caac8272 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -27,6 +27,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 #### `mvm inspect` (all — vm, image, network, key, etc.) - Byte-size fields (`_size` keys) in tree output are now human-readable (e.g., `8589934592` → `8.0 GiB`). +#### Jailed VM cgroup verification +- VM create no longer fails when Jailer places the cgroup at `/sys/fs/cgroup/mvmctl/`; the check previously expected an intermediate `firecracker/` component that Jailer v1.16 does not create. +- Stop and remove now clean the VM's actual leaf cgroup instead of leaving stale ones behind. + ## [0.2.0] - 2026-07-10 ### Added diff --git a/docs/RUNTIME.md b/docs/RUNTIME.md index ecaa6509..74823ff1 100644 --- a/docs/RUNTIME.md +++ b/docs/RUNTIME.md @@ -411,7 +411,7 @@ Jailer is the only VM launch path. `mvm bin pull` streams the checksum-verified Jailer does not daemonize, so the serial-console descriptors remain attached. The service derives the non-root Firecracker UID/GID from the sudo caller, translates Firecracker configuration paths into the jail, and records the actual Firecracker PID. Stop, remove, reboot, hotplug, and snapshot operations reconcile the corresponding jail mounts and cgroup. -There is no direct-Firecracker fallback. Every launch requires cgroup v2 with the CPU, memory, and PID controllers and creates `/sys/fs/cgroup/mvmctl/firecracker/`. mvmctl persists typed CPU, memory, swap, and PID limits, passes only internally derived Jailer arguments, and verifies exact membership and values after launch. Verification failure terminates the VM. `mvm vm inspect` reports requested limits, observed limits, usage, mismatches, and enforcement status. Network namespaces remain separate follow-up work. +There is no direct-Firecracker fallback. Every launch requires cgroup v2 with the CPU, memory, and PID controllers and creates `/sys/fs/cgroup/mvmctl/`. mvmctl persists typed CPU, memory, swap, and PID limits, passes only internally derived Jailer arguments, and verifies exact membership and values after launch. Verification failure terminates the VM. `mvm vm inspect` reports requested limits, observed limits, usage, mismatches, and enforcement status. Network namespaces remain separate follow-up work. **Key files:** diff --git a/internal/core/vm/jailer.go b/internal/core/vm/jailer.go index 2f58259d..7d074b48 100644 --- a/internal/core/vm/jailer.go +++ b/internal/core/vm/jailer.go @@ -116,7 +116,7 @@ func removeVolume(ctx context.Context, vmID, driveID string) error { // CgroupPath returns the fixed cgroup-v2 path for one canonical Jailer launch. func CgroupPath(vmID string) string { - return filepath.Join(infra.CgroupV2Root, infra.JailerCgroupParent, infra.JailerCgroupExecutable, vmID) + return filepath.Join(infra.CgroupV2Root, infra.JailerCgroupParent, vmID) } // InspectCgroup reads typed enforcement and usage state for one VM. @@ -284,7 +284,7 @@ func processInCgroup(pid int, vmID string) (bool, error) { if err != nil { return false, err } - expected := "/" + filepath.Join(infra.JailerCgroupParent, infra.JailerCgroupExecutable, vmID) + expected := "/" + filepath.Join(infra.JailerCgroupParent, vmID) for line := range strings.SplitSeq(string(data), "\n") { if strings.HasPrefix(line, "0::") && strings.TrimPrefix(line, "0::") == expected { return true, nil diff --git a/internal/infra/constants.go b/internal/infra/constants.go index f3c6ae23..eba65783 100644 --- a/internal/infra/constants.go +++ b/internal/infra/constants.go @@ -55,9 +55,6 @@ const JailerChrootBase = "/var/lib/mvmctl/jailer" // JailerCgroupParent is the fixed cgroup-v2 subtree used by canonical Jailer launches. const JailerCgroupParent = "mvmctl" -// JailerCgroupExecutable is Jailer's fixed executable-name component in the cgroup path. -const JailerCgroupExecutable = "firecracker" - // CgroupV2Root is the unified Linux cgroup-v2 mount point required by VM launches. const CgroupV2Root = "/sys/fs/cgroup" diff --git a/internal/service/jailer/entry.go b/internal/service/jailer/entry.go index ccc2b4c1..7a2c499c 100644 --- a/internal/service/jailer/entry.go +++ b/internal/service/jailer/entry.go @@ -696,7 +696,6 @@ func cgroupPath(vmID string) string { return filepath.Join( infra.CgroupV2Root, infra.JailerCgroupParent, - infra.JailerCgroupExecutable, vmID, ) } diff --git a/internal/service/jailer/entry_test.go b/internal/service/jailer/entry_test.go index 351860ec..3a24fb2b 100644 --- a/internal/service/jailer/entry_test.go +++ b/internal/service/jailer/entry_test.go @@ -213,7 +213,7 @@ func TestValidateCgroupLimitsRejectsAbsentAndInvalidEnvelopes(t *testing.T) { func TestEnsureCgroupEmptyRefusesPopulatedAndAcceptsAbsentState(t *testing.T) { path := t.TempDir() - base := filepath.Join(infra.CgroupV2Root, infra.JailerCgroupParent, infra.JailerCgroupExecutable) + base := filepath.Join(infra.CgroupV2Root, infra.JailerCgroupParent) vmID, err := filepath.Rel(base, path) require.NoError(t, err) assert.Equal(t, filepath.Clean(path), cgroupPath(vmID)) diff --git a/tests/system/vm/test_cgroup.py b/tests/system/vm/test_cgroup.py index cd911817..684f0c31 100644 --- a/tests/system/vm/test_cgroup.py +++ b/tests/system/vm/test_cgroup.py @@ -110,7 +110,7 @@ def test_running_vm_envelope_and_lifecycle_cleanup(self, runner_vm: str) -> None assert vm is not None and vm["status"] == "running" vm_id = vm["id"] pid = int(vm["pid"]) - relative = f"/mvmctl/firecracker/{vm_id}" + relative = f"/mvmctl/{vm_id}" leaf = f"/sys/fs/cgroup{relative}" membership = _read(runner_vm, f"/proc/{pid}/cgroup").splitlines() @@ -175,7 +175,7 @@ def test_running_vm_envelope_and_lifecycle_cleanup(self, runner_vm: str) -> None runner_vm, "vm", "rm", vm_name, "--force", check=False, timeout=120 ) if vm_id: - leaf = f"/sys/fs/cgroup/mvmctl/firecracker/{vm_id}" + leaf = f"/sys/fs/cgroup/mvmctl/{vm_id}" assert _guest_run( runner_vm, f"test ! -e {shlex.quote(leaf)}", check=False ).returncode == 0 From f0784d26429ad70118f234ca44d774e5b8bfde02 Mon Sep 17 00:00:00 2001 From: AlanD20 Date: Thu, 20 Aug 2026 07:48:28 +0200 Subject: [PATCH 0006/1457] docs: define secure single-binary privilege boundary --- docs/adr/0005-sudo-privilege-architecture.md | 4 + docs/adr/0015-jailer-canonical-launcher.md | 2 +- ...owned-single-binary-privileged-dispatch.md | 255 ++++++++ tasks/plan.md | 358 ++++++++++++ tasks/todo.md | 543 ++++++++++++++++++ 5 files changed, 1161 insertions(+), 1 deletion(-) create mode 100644 docs/adr/0016-root-owned-single-binary-privileged-dispatch.md create mode 100644 tasks/plan.md create mode 100644 tasks/todo.md diff --git a/docs/adr/0005-sudo-privilege-architecture.md b/docs/adr/0005-sudo-privilege-architecture.md index 91523710..292f92af 100644 --- a/docs/adr/0005-sudo-privilege-architecture.md +++ b/docs/adr/0005-sudo-privilege-architecture.md @@ -4,6 +4,10 @@ **Date:** 2026-05-22 **Last Updated:** 2026-06-20 (Go implementation complete) +> **Replacement accepted, implementation pending:** ADR-0016 replaces this user-owned wildcard sudo design with one +> root-owned system installation and an early typed privileged dispatcher. This ADR continues to describe the current +> implementation until ADR-0016's migration and L2 release gate pass. + The project uses a **tightly scoped, group-based sudo model** where members of a dedicated Unix group (`mvm`) can execute specific privileged commands via `sudo` without a password prompt. This avoids running the entire CLI as root while still allowing VM creation (loop-mount), network management (ip/iptables/nft), kernel module loading (modprobe), and sysctl configuration. **Table of Contents** diff --git a/docs/adr/0015-jailer-canonical-launcher.md b/docs/adr/0015-jailer-canonical-launcher.md index 98f3f89d..71ba22da 100644 --- a/docs/adr/0015-jailer-canonical-launcher.md +++ b/docs/adr/0015-jailer-canonical-launcher.md @@ -2,7 +2,7 @@ **Status:** Active **Date:** 2026-08-09 -**See also:** [Project Hosting with MicroVMs](../roadmaps/TENANT_MICROVM.md), [ADR-0005: Sudo Privilege Architecture](0005-sudo-privilege-architecture.md) +**See also:** [Project Hosting with MicroVMs](../roadmaps/TENANT_MICROVM.md), [ADR-0005: Sudo Privilege Architecture](0005-sudo-privilege-architecture.md), [ADR-0016: Root-Owned Single-Binary Privileged Dispatch](0016-root-owned-single-binary-privileged-dispatch.md) ## Context diff --git a/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md b/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md new file mode 100644 index 00000000..31972991 --- /dev/null +++ b/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md @@ -0,0 +1,255 @@ +# Root-Owned Single-Binary Privileged Dispatch + +**Status:** Accepted — implementation pending +**Date:** 2026-08-20 +**See also:** [ADR-0002: Single Go Binary Architecture](0002-single-go-binary-architecture.md), +[ADR-0005: Sudo Privilege Architecture](0005-sudo-privilege-architecture.md), +[ADR-0015: Jailer Is the Canonical Firecracker Launcher](0015-jailer-canonical-launcher.md) + +## Context + +mvmctl is a Go CLI whose user-facing commands and subprocess services are compiled into one release artifact. The CLI +is intended to run as an unprivileged user while a small set of VM, filesystem, network, firewall, cgroup, and Jailer +operations require root. + +The existing installation contract puts `mvm` at user-owned `~/.local/bin/mvm` and generates a passwordless sudoers +entry for that path with wildcard arguments. A member of the `mvm` group can replace the executable and then ask sudo to +run the replacement as root. Root ownership checks performed later by Jailer cannot repair a compromised executable +boundary. + +The existing sudoers policy also grants raw argument access to host tools such as `ip`, `iptables`, `nft`, `modprobe`, +and `sysctl`. Those tools expose substantially more host authority than the named mvmctl operations that need them. The +policy therefore does not provide the targeted access claimed by ADR-0005. + +Introducing a separately compiled privileged helper would provide a small executable trust boundary, but it would break +the project's single-binary product contract. A safe design must retain one artifact without allowing the ordinary CLI +initialization path or caller-controlled program bytes to execute as root. + +## Decision + +### One root-owned system installation + +Manually built and release-script artifacts are installed at `/usr/local/bin/mvm` as `root:root`, mode `0755`. The +directory chain must be root-owned and not group/world writable. `/usr/bin/mvm` is reserved for a future installation +owned by a distribution package manager. + +Normal users execute `/usr/local/bin/mvm` without sudo. The file contains both the public CLI and the internal privileged +dispatcher, preserving one release artifact and one installed executable. Replacing or upgrading the system executable +is an explicit administrator action; a normal user update cannot overwrite it. + +Development builds may exist elsewhere, but they are never sudo targets. A development build that needs privileged +operations calls the compatible system installation and fails closed on a privileged-protocol mismatch. + +### Early privileged dispatch + +`cmd/mvm/main.go` recognizes one reserved, versioned privileged marker before signal setup, `app.Initialize()`, Cobra, +user configuration, the user database, logging configuration, or plugin/service discovery. Once selected, this path can +only enter the privileged dispatcher; it cannot return to or invoke an ordinary CLI command. + +The dispatcher: + +- requires effective UID 0 and derives the invoking identity from sudo's trusted execution context; +- verifies the system executable and its directory chain are root-owned and not group/world writable; +- verifies that the invoking user is currently authorized by the `mvm` group; +- sanitizes its environment and uses absolute paths for root-side subprocesses; +- accepts only a versioned, bounded, typed request for a named operation; +- rejects unknown actions, extra arguments or fields, invalid identifiers, unsupported versions, and unchecked paths; +- never accepts raw commands, arbitrary argv, shell fragments, raw firewall syntax, or raw cgroup key/value input. + +The sudoers entry may contain a wildcard after the reserved marker because operation arguments vary, but that wildcard +can reach only the early dispatcher. It cannot select the public CLI. The dispatcher remains the authoritative +allowlist, and every handler performs exact argument-count and type checks before side effects. + +### Authorization roles + +The root-owned executable establishes code integrity. The `mvm` Unix group remains the authorization role for +passwordless privileged VM operations. Ordinary non-privileged commands do not require group membership. + +Membership in `mvm` authorizes management of mvmctl-owned VM infrastructure; it does not authorize arbitrary host root +commands. The sudoers drop-in grants only the reserved privileged entry point of the root-owned system executable. It +does not grant the public CLI or raw `ip`, `iptables`, `iptables-restore`, `iptables-save`, `nft`, `modprobe`, or `sysctl` +commands. + +`host init`, system-binary installation/upgrades, group membership changes, sudoers changes, and `host reset` modify the +authorization boundary itself. They remain explicit administrator operations under the administrator's normal sudo +policy and are not exposed through the `mvm` group's passwordless dispatcher. + +The binary is never setuid and is not assigned broad Linux file capabilities. + +### Typed privileged operation catalog + +The initial dispatcher contains only the following named capabilities. Public callers retain domain-specific typed +methods; there is no generic privileged-operation interface. + +| Capability | Allowed effect | Required binding | +|---|---|---| +| Trusted release install/remove | Verify and atomically manage an exact Firecracker/Jailer release pair | Fixed origin, version, architecture, root release manifest, active references | +| VM launch/abort/cleanup/reconcile | Create and recover one Jailer process, chroot, cgroup, and private mount namespace | Invoking UID, VM ID, exact release, root instance record, per-VM lock | +| Snapshot expose/remove | Mount one validated snapshot directory in the registered VM namespace | Invoking UID, VM ID, snapshot ID, pinned source/target descriptors | +| Volume expose/remove | Mount one validated volume at one typed drive ID in the registered VM namespace | Invoking UID, VM ID, volume/drive ID, pinned source/target descriptors | +| Image provision | Execute the existing typed provision operations inside one pinned managed image/rootfs | Invoking UID, managed resource identity, pinned root descriptor, bounded operation list | +| Bridge/TAP lifecycle | Create, configure, or remove an mvmctl-owned bridge or TAP | Invoking UID, network/VM identity, validated interface/address model | +| Neighbor flush | Flush neighbors only for an mvmctl-owned bridge | Invoking UID and registered network identity | +| Firewall inspect/reconcile/teardown | Render and atomically apply the complete typed mvmctl ruleset | Invoking UID, typed firewall models, fixed mvmctl tables/chains/comments | + +Cgroup preparation and removal are part of VM launch/abort/cleanup rather than a public generic cgroup operation. Module +loading and sysctl mutation are administrator-owned host initialization steps, not operator dispatcher capabilities. + +Adding a new privileged capability requires a new typed method, abuse cases, ADR-0016 review, and sudo threat-model +review. “Extensible for future use” is not a reason to add a generic action. + +### Module placement and dependency direction + +The privileged boundary follows the repository's existing service-subprocess architecture and does not create a second +application architecture: + +| Package/file | Responsibility | Must not do | +|---|---|---| +| `cmd/mvm/main.go` | Detect the reserved marker and enter privileged dispatch before normal initialization | Parse domain requests, perform side effects, initialize the public CLI | +| `internal/service/privileged/` | Versioned envelope, bounded parsing, invoking identity, executable verification, and fixed action dispatch | Import `pkg/api`, `internal/cli`, or any `internal/core/*`; expose generic command execution | +| `internal/service/jailer/` | Typed release, launch, namespace mount, cgroup, process, and cleanup handlers plus normal-user client calls | Orchestrate other core domains or trust user paths after validation | +| `internal/service/loopmount/` | Existing typed provisioning protocol and confined rootfs operations | Accept arbitrary host roots or unrelated unmount/PID targets | +| `internal/service/network/` | Typed root-side bridge, TAP, address, route, and neighbor mutations | Accept raw `ip` argv or own network business orchestration | +| `internal/service/firewall/` | Typed root-side firewall inspection and atomic managed-ruleset application | Accept raw nftables/iptables text from the caller | +| `internal/lib/model/` | Shared typed request/value models used across boundaries | Import services, core domains, API, or CLI | +| `internal/lib/system/` | Subprocess transport and low-level process helpers | Become a generic privileged-operation API or render domain commands | +| `internal/core/{domain}/` | Existing domain logic and typed backend interfaces | Import another core domain or perform cross-domain orchestration | +| `pkg/api/` | Sole cross-core orchestration and public input validation | Implement root-side handlers or bypass typed service methods | + +The central dispatcher performs only fixed routing and security-boundary checks. Actual effects remain in +capability-specific service packages and files. Each public client method keeps a domain name and shape, such as +`jailer.Launch`, `jailer.Cleanup`, `network.EnsureTap`, and `firewall.Reconcile`; there is no public +`ApplyOperations([]Operation)` or `RunCommand([]string)` method. + +Where an existing leaf package needs privileged effects, it defines the smallest typed interface in the owning layer and +receives a concrete service client through construction. Leaf packages do not gain imports of `pkg/api`, `internal/cli`, +or core domains. The one existing loopmount service import exception is not generalized without a separate architecture +review. + +Normal API `Input.Validate`/`Resolve` remains the caller-facing validation layer. Because the privileged dispatcher +receives data from an untrusted process, it independently enforces authorization, ownership, bounds, path confinement, +and system-damage invariants. These are receiver trust-boundary checks, not duplicated business validation. + +### Release trust + +The caller may supply archive bytes for caching and progress, but it cannot supply their authority. Privileged release +code constructs the permitted Firecracker release URL from a strictly validated version and architecture and obtains the +expected checksum independently from the fixed official origin or root-owned signed metadata. + +Extraction occurs in a root-owned temporary directory with bounded input, exact member names, and rejection of path +traversal, symlinks, hardlinks, devices, duplicate members, unexpected members, and size/count overflow. The exact +Firecracker/Jailer pair and a root-owned release manifest are fsynced and atomically renamed into the trusted store. A +referenced release cannot be force-replaced. + +If upstream provides a stable verifiable signature or provenance mechanism, mvmctl should adopt it. Until then, the +fixed HTTPS origin and independently fetched checksum are an explicit supply-chain limitation, not a signature claim. + +### Paths, process identity, and runtime state + +Privileged code treats the normal CLI, arguments, environment, database, cache, manifests, and downloaded content as +untrusted. Managed filesystem objects are opened once through descriptor-relative no-symlink resolution and retained as +pinned descriptors through each operation. Validation followed by pathname reopen is prohibited. + +Every privileged VM operation is serialized by a root-owned per-VM lock and authorized by a minimal root-owned instance +record. Process actions verify PID, start time, expected cgroup membership, owner, and exact release; pidfds are used when +available. PID reuse fails closed. + +The launch path creates a private mount namespace before resource mounts and Jailer execution. Live snapshot and volume +operations enter only the verified registered VM's mount namespace. Cleanup never recursively traverses a tree that may +contain a mount. + +State is split by authority and lifetime: + +| State | Location | Ownership/lifetime | +|---|---|---| +| User DB and VM artifacts | Existing `MVM_CACHE_DIR`, normally `~/.cache/mvmctl` | User-owned, persistent | +| Trusted release pairs and manifests | `/var/lib/mvmctl/binaries` | Root-owned, persistent | +| Privileged instance ownership/release records | `/var/lib/mvmctl/instances/` | Root-owned, persistent, minimal | +| Jailer chroot directories | `/var/lib/mvmctl/jailer` | Root-owned lifecycle state | +| Per-VM locks and launch handshakes | `/run/mvmctl/` | Root-owned, ephemeral | +| Cgroups | `/sys/fs/cgroup/mvmctl/` | Kernel runtime state | +| `/run/mvm` | Inside the jail | Jail-visible bind mount, not host canonical state | + +Moving durable user state out of `~/.cache/mvmctl` is a separate XDG migration decision. Full state must not move to +`/run` or `/var/run`. + +### Failure and reconciliation contract + +A root-owned instance record exists before the first privileged launch side effect. Every post-spawn failure retains a +live lifecycle handle and invokes a typed abort that verifies identity, terminates the process, checks cgroup emptiness, +and cleans namespace/chroot/instance state in order. + +Startup and pre-launch reconciliation compare root instance records, processes, cgroups, and chroot directories. They do +not trust the user database as root authorization. Operations are idempotent and either converge to a known state or +return a precise error without deleting potentially mounted user data. + +## Alternatives Considered + +### Separate privileged helper binary + +This provides the smallest executable trust boundary, but creates a second artifact and contradicts the product's +single-binary contract. Rejected in favor of early dispatch within one immutable system installation. + +### User-owned CLI with a sudo command digest + +Pinning a digest would make every user update require sudoers regeneration and would retain a user-owned path at the +root boundary. It also leaves the public CLI initialization path exposed to root execution. Rejected. + +### Continue wildcard sudo for a root-owned public CLI + +Root ownership fixes executable replacement but `mvm *` would still allow every public command to run as root and load +user state under root authority. Rejected. Only the reserved early dispatcher is allowed. + +### Setuid or broad file capabilities + +Both would expose privilege on every invocation and make the whole public CLI a security boundary. Required operations +also span mount, network, cgroup, and process privileges that do not reduce cleanly to a safe fixed capability set. +Rejected. + +### Root daemon, polkit service, or systemd RPC + +These can provide a strong authorization boundary but add a long-running service, protocol lifecycle, and additional +deployment artifacts. Rejected for the initial implementation; reconsider only if concurrency or policy needs exceed the +short-lived dispatcher model. + +## Consequences + +**Benefits:** + +- The product retains one compiled and installed executable. +- Passwordless sudo can no longer execute user-selected program bytes or raw host-tool arguments. +- Root execution bypasses user configuration and the ordinary CLI initialization graph. +- The `mvm` group has a precise meaning: authorized mvmctl operator, not unrestricted root. +- Privileged state and user state have explicit owners and recovery responsibilities. + +**Costs and limitations:** + +- Installation and upgrades require an explicit administrator step. +- Development binaries cannot perform privileged work unless a compatible system binary is installed. +- All existing raw privileged callsites must migrate to typed operations before the old sudoers policy is removed. +- The single executable contains more code than a dedicated helper, so early dispatch and import-graph isolation require + strict tests and review. +- Multi-user ownership, crash recovery, release verification, namespace mounts, and firewall atomicity add implementation + and L2 testing cost. + +## Migration and Release Gate + +ADR-0005 remains the description of the currently implemented legacy policy until this ADR's release gate passes. The +replacement sequence is: + +1. Add early privileged dispatch and negative tests while the new path is unreachable from sudoers. +2. Migrate every privileged service/tool call to a typed operation. +3. Install the root-owned system binary and atomically replace the sudoers drop-in after `visudo` validation. +4. Run upgrade, rollback, privilege-abuse, multi-user, path-race, PID-reuse, crash-recovery, Jailer, cgroup, snapshot, + hotplug, and firewall L2 tests. +5. Mark ADR-0005 superseded, update all current-behavior documentation, and only then describe the new boundary as active. + +No implementation phase is complete while documentation describes a different executable path, sudo policy, state +location, lifecycle guarantee, or known limitation. + +## Related Decisions + +- ADR-0002: One release artifact and one installed executable remain the product contract. +- ADR-0005: Legacy group-based wildcard sudo policy; superseded only after this migration's release gate. +- ADR-0012: Root boundary, namespace, cgroup, and recovery behavior require L2 verification. +- ADR-0015: Jailer remains canonical; its privilege-boundary and cleanup requirements are refined here. diff --git a/tasks/plan.md b/tasks/plan.md new file mode 100644 index 00000000..9de73f3e --- /dev/null +++ b/tasks/plan.md @@ -0,0 +1,358 @@ +# Implementation Plan: Safe Jailer and Privileged-Operations Architecture + +**Status:** Approved — implementation in progress +**Review range:** `f8ffa28b9d74cac2c9e15fa07a18389d51a3263e..ee733232a2dc23c1edb372022a2add71af36f44c` +**Release decision:** Do not push, merge, or describe the current Jailer implementation as production-ready. + +## Overview + +Replace the user-writable `mvm` sudo target with one root-owned system installation and an early privileged dispatcher; +make every privileged operation typed, owner-bound, descriptor-relative, and race-safe; then repair Jailer lifecycle, +snapshot, hotplug, cgroup, and firewall +semantics. The ordinary CLI continues to run as an unprivileged user. Persistent user state remains in the existing +per-user location for this change. Only trusted releases, privileged ownership records, chroots, cgroups, and ephemeral +locks live in system locations. + +The last five commits should be treated as a prototype branch. Remediation should land as reviewable commits on top of +the current work. History may be curated only after the implementation and L2 verification pass, and only after the +existing dirty worktree has been preserved. + +## Scope + +### Required before release + +- Replace every passwordless-sudo entry that can execute user-controlled code or arbitrary system-tool arguments. +- Establish an independent trust anchor for Firecracker/Jailer release installation. +- Remove path validation followed by privileged pathname reopen. +- Give each privileged VM operation an authenticated owner, exact release, process identity, and per-VM lock. +- Put launch mounts in a private namespace and post-launch mounts in the running VM's verified mount namespace. +- Make create, start, stop, remove, snapshot, restore, and volume hotplug crash-consistent. +- Make routed-policy reconciliation fail closed, including resource deletion and IP reuse. +- Execute the prepared L2 tests plus adversarial privilege, race, and fault-injection coverage. +- Rewrite documentation only after the final behavior passes the release gates. + +### Explicitly separate + +- Moving the user database and durable user state out of `~/.cache/mvmctl` is desirable but is not part of this + privilege remediation. It requires a separate ADR, compatibility migration, and rollback plan. +- PID and lock files may move to `/run`; the canonical database, images, snapshots, and VM data must not. +- Network namespaces remain a separate feature unless required to make the corrected Jailer lifecycle work. + +## Threat Model + +### Principals + +- A member of the `mvm` group may manage only resources authorized for that Unix UID. +- An `mvm` group member is untrusted from the privileged dispatcher's perspective and must not gain arbitrary root + execution. +- The normal `mvm` executable, its environment, stdin, arguments, cache, manifests, and database are user-controlled. +- The root-owned system binary, its privileged dispatcher, directory chain, host policy, release records, and privileged + instance records are trusted. +- Downloaded release archives are untrusted until verified inside the privileged boundary. + +### Protected assets + +- Host root filesystem, processes, mount namespaces, networking, kernel modules, sysctls, and cgroup hierarchy. +- Other users' VM data and privileged instance records. +- Exact Firecracker/Jailer release identity and installed bytes. +- Default-deny firewall behavior and IP-identity binding. +- Recoverability after process failure, CLI interruption, or host reboot. + +### Security invariants + +1. The normal CLI never becomes the passwordless root executable. +2. Sudo permits only the early privileged mode of root-owned `/usr/local/bin/mvm`; raw `ip`, `nft`, `iptables`, + `modprobe`, and `sysctl` access is removed. +3. The privileged dispatcher exposes named, typed operations. It never accepts a generic command, arbitrary argv, raw + cgroup keys, or an unchecked host path. +4. Every privileged request is bound to `SUDO_UID`, a root-owned instance record, and a fixed VM/release identity. +5. A user-controlled digest is never a trust anchor. The privileged dispatcher obtains expected release integrity data from + root-owned policy or a fixed, independently fetched official source. +6. User paths are opened once with descriptor-relative, no-symlink resolution. Privileged code operates on pinned + descriptors and never validates a path and later reopens it. +7. PID actions use a verified PID plus start time and cgroup membership, preferably a pidfd. PID reuse must fail closed. +8. Launch and cleanup are serialized by a root-owned per-VM lock. Cleanup never recursively traverses a possible mount. +9. All post-spawn failures terminate the registered process before deleting DB, VM, jail, or cgroup state. +10. Removing policy intent or a resource cannot make its IP reusable until the deny/allow kernel state is reconciled. + +## Target Architecture + +```text +one root-owned /usr/local/bin/mvm + | + |-- normal CLI mode runs unprivileged + | + | typed internal request through sudo + v +early privileged dispatcher before app.Initialize/Cobra/user config + | + +-- release operations ---------- fixed origin/root policy, bounded archive parser + +-- VM launch/lifecycle --------- owner record, pidfd/start time, per-VM lock + +-- mount operations ------------ openat2 dirfds, private/setns mount namespace + +-- network/firewall operations -- typed models, atomic backend batches + +-- host operations ------------- named module/sysctl actions only + | + +-- /var/lib/mvmctl ------------- trusted releases, instance ownership, chroot dirs + +-- /run/mvmctl ----------------- ephemeral locks/handshakes + +-- /sys/fs/cgroup/mvmctl ------- kernel-enforced per-VM limits + +~/.cache/mvmctl existing unprivileged DB and VM artifacts +``` + +The privileged mode is compiled into the same `mvm` artifact but is dispatched directly from `cmd/mvm/main.go` before +`app.Initialize()`, Cobra, the user database, configuration, or plugins are loaded. It has a deliberately small import +graph and cannot dispatch back into ordinary `mvm` commands. It verifies at startup that the executable and directory +chain are root-owned and not group/world writable. Upgrading `/usr/local/bin/mvm` requires an explicit administrator +action; a user-level update cannot replace it. + +## State Placement + +| State | Location for this remediation | Ownership and lifetime | +|---|---|---| +| User SQLite DB, images, kernels, volumes, snapshots, VM directories | Existing `MVM_CACHE_DIR`, normally `~/.cache/mvmctl` | User-owned, persistent | +| Trusted Firecracker/Jailer releases and release manifest | `/var/lib/mvmctl/binaries/...` | Root-owned, persistent | +| Privileged VM ownership/release records | `/var/lib/mvmctl/instances//.json` | Root-owned, persistent, minimal metadata | +| Jailer chroot directories | `/var/lib/mvmctl/jailer/...` | Root-owned lifecycle state | +| Per-VM locks and launch handshakes | `/run/mvmctl//...` | Root-owned, ephemeral | +| Cgroup state | `/sys/fs/cgroup/mvmctl/` | Kernel state, runtime only | +| `/run/mvm` | Inside the jail only | Bind-mounted jail-visible view, not host canonical state | + +The privileged instance record contains only the owning UID, VM ID, exact release hashes/IDs, process identity, and +cleanup generation. It is not a second copy of the user VM specification. + +## Key Design Decisions + +### Root-owned single binary and early privileged dispatcher + +Install the release artifact through an explicit administrative bootstrap to `/usr/local/bin/mvm` as `root:root`, mode +`0755`. Normal users execute that same file without sudo. The sudoers drop-in permits only its internal privileged entry +point, which is selected before normal initialization. Each privileged operation has its own typed request and +validation. Unknown actions, extra fields, and unsupported protocol versions fail closed. + +The code continues to follow the existing layer and file split. `cmd/mvm/main.go` performs only the early branch; +`internal/service/privileged/` owns the envelope, identity checks, and fixed dispatch; Jailer, loopmount, network, and +firewall effects remain in capability-specific `internal/service/{name}/` packages; shared values live in +`internal/lib/model/`; domain logic remains in `internal/core/{domain}/`; and `pkg/api/` remains the sole cross-core +orchestrator. No generic privileged backend or raw command method is introduced. + +The final state removes the existing raw privileged binaries from sudoers. Merely replacing `mvm *` is insufficient: +several allowed system tools can execute or materially alter arbitrary host state when given attacker-selected arguments. + +### Release trust + +The normal CLI may download the archive for progress and caching, but the privileged dispatcher must independently +obtain the expected +digest from a fixed official URL that it constructs from validated version/architecture, or from root-owned signed +metadata. It must not accept both an archive and its authority from the caller. + +Installation uses a root-owned temporary directory, a bounded tar reader, exact member allowlists, and rejection of +links, devices, traversal, duplicate entries, oversized payloads, and unexpected files. After verification, root-side code +atomically renames and fsyncs the exact pair plus a root-owned release manifest. A forced replacement of a referenced +release is prohibited. + +If Firecracker provides a stable verifiable signature/provenance mechanism, prefer it. Otherwise the initial model may +trust the fixed HTTPS release origin and its independently fetched checksum, with that limitation recorded explicitly. + +### Safe path and mount handling + +Use `openat2`/`openat` through `golang.org/x/sys/unix` with `RESOLVE_BENEATH`, `RESOLVE_NO_SYMLINKS`, and +`RESOLVE_NO_MAGICLINKS`, followed by `fstat` owner/type/mode checks. Keep directory descriptors alive through the +operation. Fixed basenames are derived from validated IDs; no privileged write uses a caller-selected basename. + +The privileged launch path first creates a private mount namespace and marks it private, then creates all pre-launch bind mounts +and execs Jailer. These mounts do not leak into the host namespace and disappear when the VM namespace dies. Live +snapshot/volume operations pin their source descriptor, verify the registered Firecracker process, enter its mount +namespace in a short-lived privileged child, and mount onto a descriptor-pinned target. Mounts should use `nodev`, `nosuid`, +and `noexec` whenever the resource permits. + +### Process and cleanup lifecycle + +Create a root-owned instance record before the first privileged side effect. Record PID, start time, cgroup path, exact +release, and cleanup generation before reporting launch success. Use pidfds where supported and always revalidate start +time plus cgroup membership. + +Every create/restore path persists a provisional user DB row before spawn. After `Spawn`, a lifecycle handle is retained +until the DB reports `running`. Any later failure calls a typed `AbortLaunch`: TERM, bounded wait, KILL of the verified +process if necessary, cgroup-empty verification, namespace/chroot cleanup, and instance-record finalization. + +Cleanup is per-VM locked and fail closed. It never calls `RemoveAll` while mounts may exist. Startup and pre-launch +reconciliation inspect root-owned instance records, registered PIDs, cgroups, and chroot directories so interrupted +operations can be recovered without trusting the user database. + +### Exact binary pair + +Keep the current binary-domain shape but add typed pair operations rather than generic repository hooks: + +- `UpsertTrustedPair(ctx, firecracker, jailer)` in one DB transaction. +- A uniqueness invariant for active `(type, version, architecture)` records. +- `jailer_binary_id` on snapshots and all restored/migrated VMs. +- Root-owned release hashes checked against both user DB records before every launch. +- No ignored repository errors during pair install, default selection, or removal. + +### Firewall policy reconciliation + +Make chain/jump initialization return a result and include it in reconciliation. Build one complete desired managed +ruleset and apply it atomically through `iptables-restore` or an nftables batch. Only after the kernel update succeeds may +intent/derived rows be committed or resource identity/IP be released. If the DB commit fails, restore the prior kernel +generation and return a joined error. Startup reconciliation recompiles from authoritative intent. + +## Dependency and Implementation Order + +```text +ADR + threat contract + | + v +root-owned system install + early privileged protocol + | + +--> Jailer/release operations --> fd-safe mounts --> lifecycle/recovery --> snapshot/hotplug + | + +--> loopmount/network/firewall/host privileged migration --> remove raw sudoers entries + | + +--> exact-pair DB migration + | + +--> fail-closed policy reconciliation + v +L1 negative/fault tests --> L2 real-host matrix --> documentation sweep --> release +``` + +See `tasks/todo.md` for task-level acceptance criteria, ownership, dependencies, and verification. + +## Checkpoints + +### Checkpoint A — architecture approved + +- ADR defines the privileged-dispatch TCB, ownership model, release trust, state layout, and migration. +- Privileged operations are enumerated; no generic execution extension exists. +- Threat cases become named tests before production code changes. +- Human approves the plan and ADR. + +### Checkpoint B — no user-controlled root entry point + +- Sudoers contains only the internal privileged entry point of root-owned `/usr/local/bin/mvm`. +- Privileged mode cannot invoke the general CLI or raw caller-selected commands. +- Jailer, loopmount, network/firewall, and host operations use typed privileged endpoints. +- Self-authored archives/checksums and mutable-binary replacement tests fail safely. + +### Checkpoint C — lifecycle parity + +- Create/start/stop/reboot/remove, live attach/detach, snapshot create, and snapshot restore pass L1 path assertions. +- Injected failures after every spawn stage leave no live registered PID, mount, cgroup, or orphaned ownership record. +- Legacy VMs/snapshots migrate or fail with an actionable error; exact pair identity is persistent. + +### Checkpoint D — real-host release gate + +- Full repository CI passes in the documented order. +- L2 Jailer/cgroup/policy tests pass on the runner VM. +- Multi-user, symlink-race, PID-reuse, kill-at-stage, reboot-recovery, and IP-reuse tests pass. +- VMM memory headroom and any I/O envelope are measured and documented. +- The final documentation audit contains no stale state, sudo, cgroup, service-count, or path claims. + +## Verification Strategy + +### L0/L1 + +- Parser/authorization tables for every privileged action, including unknown/extra input. +- Release archive abuse cases and independent-digest failure. +- Descriptor-relative path resolution under concurrent rename/symlink replacement. +- PID start-time/cgroup mismatch and per-VM lock contention. +- Exact Firecracker API paths for normal launch, snapshot create/restore, and hotplug. +- Failure injection after spawn, relay creation, socket readiness, DB writes, snapshot load, and cleanup. +- Firewall parent-jump, batch-apply, DB-commit, policy-remove, VM-remove, and IP-reuse failures. +- Structured cgroup-setting fixture restoration with restoration assertions. + +### L2 + +- Run exclusively as an unprivileged `mvm` group member. +- Confirm a user-owned `mvm` binary is rejected by passwordless sudo and the system installation is immutable. +- Attempt a caller-authored Firecracker/Jailer archive and checksum; installation must fail. +- Repeated create/kill/reconcile cycles show zero leaked mounts, cgroups, PIDs, or chroot state. +- Snapshot and live-volume parity pass inside the private mount namespace. +- A second Unix user cannot inspect, mount, signal, clean, or remove the first user's VM/release references. +- Firewall failure followed by immediate address reuse never grants stale access. +- Measure VMM overhead across supported Firecracker versions before accepting a default headroom. + +### Repository CI + +```bash +go mod tidy && git diff --exit-code +test -z "$(gofmt -l .)" +golines --max-len=120 --no-reformat-tags --list-files ./internal/ ./pkg/ ./cmd/ 2>&1 | grep . \ + && echo "violations found" && exit 1 || true +go generate ./internal/service/agent/... +go vet ./... +go test ./... -count=1 -coverprofile=coverage.out -covermode=atomic +``` + +## Migration and Rollback + +1. Preserve the current base/head commit IDs and all unrelated dirty work before changing history. +2. Add forward-only migrations for exact-pair fields and uniqueness. Never rewrite an already-applied user DB in place. +3. `mvm host init` detects the insecure legacy sudoers rule, installs the release artifact atomically as root-owned + `/usr/local/bin/mvm`, validates the new sudoers rule with `visudo`, then replaces the old rule. On failure it keeps the + previous configuration and reports the unsafe state. +4. An unsupported privileged protocol refuses the operation and prints the explicit administrator upgrade command. +5. Existing VMs with an unambiguous exact pair are backfilled. Ambiguous/unmatched VMs remain stopped and return a + migration error; they are never launched with an arbitrary pair. +6. Do not remove legacy paths until the new privileged dispatcher has reconciled root records and the L2 upgrade test succeeds. +7. Before release, optionally curate/squash the five prototype commits plus remediation only with explicit approval. + +## Documentation Update Matrix + +Documentation is the final implementation phase, not a substitute for missing guarantees. + +Documentation accuracy is a release invariant. ADR-0016 and the threat contract are written before code so the target +is explicit; all current-behavior documentation is reconciled only after implementation and L2 evidence exist. A stale +or aspirational statement in an operator, architecture, testing, or troubleshooting document blocks release just like a +failing test. + +| Document | Required correction | +|---|---| +| `docs/adr/0016-*.md` | New decision superseding the unsafe wildcard sudo boundary and defining privileged-mode/state/trust semantics | +| `docs/adr/0002-*.md`, `0005-*.md`, `0015-*.md` | Mark superseded portions and link to ADR-0016; remove false safety claims | +| `CONTEXT.md` | Correct domain count, privileged boundary, service pattern, state ownership, subprocess exceptions | +| `AGENTS.md`, `docs/STANDARDS.md` | Add privileged-dispatch import/ownership rules and Jailer service; retain typed-method/no-generic-extension constraints | +| `README.md`, `docs/DEPENDENCIES.md` | Administrator system install/upgrade, sudo behavior, trusted-release limitations | +| `docs/RUNTIME.md` | Correct service count, state paths, private mount namespace, process/cleanup flow | +| `docs/RESOURCE_MANAGEMENTS.md` | Exact cgroup path, derived policy, measurement status, I/O policy | +| `docs/TROUBLESHOOTING.md` | System-binary ownership/version checks, reconcile commands, stale instance/chroot/cgroup diagnosis | +| `docs/roadmaps/TENANT_MICROVM.md` | Mark milestones incomplete until all release gates pass | +| `docs/system-test-architecture.md`, coverage matrix | Add privilege/race/recovery/multi-user L2 cases and correct fixture cleanup | +| `CHANGELOG.md` | Describe behavior only after it is verified and releasable | + +### Documentation definition of done + +- Every changed command, path, ownership, state, lifecycle, and recovery claim is traced to final code and an executed + test or inspection. +- Documents distinguish current behavior, migration behavior, known limitations, and future work; planned guarantees + are never written in the present tense. +- Installation, upgrade, rollback, troubleshooting, and recovery commands are executed on the L2 runner. +- Repository-wide searches cover the legacy `~/.local/bin/mvm` sudo target, raw sudo commands, `/var/run`/`/run` state + claims, both cgroup path variants, old service/domain counts, direct-Firecracker fallback, and snapshot host paths. +- ADR statuses and cross-links agree; superseded decisions remain available for historical context. +- The architect reads the final code diff before approving documentation, and QA independently verifies operator-facing + procedures against the release candidate. + +The untracked `docs/implementations/MEMORY_RECLAIM.md` currently contains the obsolete +`/sys/fs/cgroup/mvmctl/firecracker/` path. Preserve it as user work during implementation and correct it only +when its owner chooses to include it in the documentation sweep. + +## Risks and Mitigations + +| Risk | Impact | Mitigation | +|---|---|---| +| Privileged mode becomes a generic root RPC | Critical | Early dispatch, enumerated typed methods, minimal imports, abuse-case tests, no raw argv/path/cgroup API | +| Filesystem race remains after validation | Critical | Pinned dirfds/openat2, fixed basenames, concurrent rename/symlink L2 tests | +| Mounts leak or live hotplug remains invisible | High | Private launch namespace; verified `setns` for post-launch mounts; mount-leak assertions | +| PID reuse signals an unrelated host process | Critical | Root record + start time + cgroup + pidfd verification before every signal/setns | +| Root/user state diverges | High | Minimal root ownership record, explicit generations, startup reconciliation, fault injection | +| Release checksum channel is insufficient | High | Independent privileged fetch now; record limitation; adopt signatures/provenance when verifiably available | +| Privilege migration breaks existing installs | High | Atomic system-binary/sudoers install, protocol negotiation, upgrade L2 test, fail-closed diagnostics | +| Scope expands into unrelated XDG migration | Medium | Keep user state placement unchanged; separate ADR and project after security release | + +## Open Approval Points + +1. **Approved:** keep one binary and install it root-owned at `/usr/local/bin/mvm`, with early privileged dispatch. +2. **Approved:** remove every raw privileged system binary from the `mvm` sudoers entry, not only `mvm *`. +3. **Approved:** privileged code constructs the official Firecracker URL and independently fetches the checksum, with + signed provenance adopted if the upstream release channel provides a stable verifiable mechanism. +4. **Approved:** XDG state separation is a follow-up and not part of this remediation. diff --git a/tasks/todo.md b/tasks/todo.md new file mode 100644 index 00000000..d5e8b6ca --- /dev/null +++ b/tasks/todo.md @@ -0,0 +1,543 @@ +# Safe Jailer Remediation Tasks + +This checklist implements `tasks/plan.md`. No production task begins until Task 1 is approved. Go production/L0/L1 work +belongs to the `engineer`; L2 and release verification belong to the `qa-engineer`; ADRs and documentation belong to the +`architect`. + +## Phase 0: Preserve and specify + +## Task 0: Preserve the prototype range and unrelated work + +**Owner:** architect +**Description:** Record the reviewed base/head and preserve the dirty worktree before any history curation. + +**Acceptance criteria:** +- [x] Base `f8ffa28b9d74cac2c9e15fa07a18389d51a3263e` and head `ee733232a2dc23c1edb372022a2add71af36f44c` remain reachable. +- [x] Existing modified/untracked user files are inventoried and unchanged. +- [x] No reset, rebase, squash, or force push occurs before the release gate and explicit approval. + +**Verification:** +- [x] `git status --short` matches the preserved inventory plus intentional remediation files. +- [x] `git log --oneline --decorate -10` shows the prototype commits and preservation reference. + +**Dependencies:** None +**Files likely touched:** None; Git references only after approval +**Estimated scope:** XS + +## Task 1: Approve ADR-0016 and the privileged operation catalog + +**Owner:** architect +**Description:** Record the threat model, root-owned system binary, early privileged dispatcher, state layout, release +trust, ownership rules, and complete +typed operation catalog. Supersede the conflicting portions of ADR-0002, ADR-0005, and ADR-0015. + +**Acceptance criteria:** +- [x] ADR names every privileged action and rejects generic argv/command/path/cgroup extension points. +- [x] ADR defines bootstrap, upgrade, version negotiation, recovery, and rollback. +- [x] ADR maps every new interface/file to the existing CLI/API/core/service/lib layers and preserves dependency rules. +- [x] Human explicitly approves the four open decisions in `tasks/plan.md`. + +**Verification:** +- [x] Cross-check every proposed action against two analogous typed interfaces in the repository. +- [x] Confirm `cmd/mvm` only routes, `pkg/api` remains the sole cross-core orchestrator, and no core domain imports a + sibling core domain. +- [x] ADR-0005 explicitly marks its user-owned wildcard safety claim as pending replacement by ADR-0016. + +**Dependencies:** Task 0 +**Files likely touched:** `docs/adr/0016-*.md`, `docs/adr/0002-*.md`, `docs/adr/0005-*.md`, `docs/adr/0015-*.md` +**Estimated scope:** M + +### Checkpoint A: Architecture + +- [x] Task 0 and Task 1 complete. +- [x] Human approves ADR-0016 before Go implementation. +- [x] Abuse cases are listed as test requirements before production implementation. + +## Phase 1: Replace the privilege boundary while preserving one binary + +## Task 2: Atomically install the single root-owned system binary + +**Owner:** engineer +**Description:** Add an administrator-only install/upgrade flow for the existing release artifact at +`/usr/local/bin/mvm`. Privileged dispatch validates executable ownership/mode and protocol version before any normal app +initialization. + +**Acceptance criteria:** +- [ ] `/usr/local/bin/mvm` is installed as `root:root`, mode `0755`, under a root-owned non-writable directory chain. +- [ ] Installation is temporary-file + fsync + rename and refuses symlinked targets. +- [ ] A normal user update cannot replace the system binary; unsupported privileged protocol fails closed with an + administrator upgrade instruction. + +**Verification:** +- [ ] L1 install tests cover ownership, modes, symlink target, partial write, and version mismatch. +- [ ] Focused Go tests for system installation, early dispatch, and host initialization pass. + +**Dependencies:** Task 1 +**Files likely touched:** `cmd/mvm/main.go`, `internal/core/host/utils.go`, `internal/core/host/utils_test.go`, +`internal/infra/constants.go`, `scripts/build.sh` +**Estimated scope:** M + +## Task 3: Define strict early privileged request parsing and caller identity + +**Owner:** engineer +**Description:** Before `app.Initialize()` and Cobra, detect only the reserved privileged entry point, then implement +protocol-versioned, bounded, typed request decoding; sanitize environment; derive the caller from `SUDO_UID`/`SUDO_GID`; +reject rootless/direct misuse and all unknown or extra input. + +**Acceptance criteria:** +- [ ] Each action has a distinct request type and handler; no `[]string` command or generic operation payload exists. +- [ ] Request body, string, list, and numeric limits are enforced before side effects. +- [ ] Caller UID/GID and system-binary ownership checks precede action dispatch. +- [ ] Ordinary CLI initialization and commands are unreachable after privileged mode is selected. + +**Verification:** +- [ ] Table tests reject unknown actions/fields, duplicate JSON fields, oversized input, invalid IDs, missing sudo identity, + and unsupported protocol versions. +- [ ] `go test` for privileged dispatch passes with `FakeRunner` and temporary roots. + +**Dependencies:** Task 2 +**Files likely touched:** `cmd/mvm/main.go`, `internal/service/privileged/protocol.go`, `entry.go`, `identity.go`, tests +**Estimated scope:** M + +## Task 4: Migrate Jailer service calls to typed privileged actions + +**Owner:** engineer +**Description:** Route install, launch, cleanup, snapshot exposure, volume exposure/removal, and release removal through +the privileged protocol. Remove the public `mvm run jailer` root entry point. + +**Acceptance criteria:** +- [ ] Jailer caller methods retain typed names matching existing service conventions. +- [ ] The general CLI cannot directly dispatch privileged Jailer handlers. +- [ ] Unknown/unauthorized VM and release operations fail before filesystem/process changes. + +**Verification:** +- [ ] L1 tests assert exact privileged requests for every Jailer method. +- [ ] `rg -n "run jailer|newJailerServiceCmd"` finds only migration/docs references. + +**Dependencies:** Task 3 +**Files likely touched:** `internal/service/jailer/spawn.go`, `internal/service/jailer/entry.go`, their tests, +`internal/cli/service.go` +**Estimated scope:** M + +## Task 5: Constrain loopmount behind privileged dispatch + +**Owner:** engineer +**Description:** Keep the typed provisioning wire operations but validate all roots and chroot targets through pinned +descriptors. Remove direct privileged `mvm run provision` and the arbitrary `--umount` shortcut. + +**Acceptance criteria:** +- [ ] Provision operations are limited to the invoking user's pinned VM/image/volume roots. +- [ ] Unmount/process handling cannot target unrelated host mounts or PIDs. +- [ ] No caller-provided host command is executed outside the intended mounted guest root. + +**Verification:** +- [ ] L1 abuse tests cover host path, symlink, mount swap, unrelated PID, and arbitrary unmount attempts. +- [ ] Existing loopmount L1 behavior remains green. + +**Dependencies:** Task 3 +**Files likely touched:** `internal/service/loopmount/entry.go`, `provisioner.go`, `spawn.go`, `wire.go`, tests +**Estimated scope:** M + +## Task 6: Replace raw network-link sudo calls with typed privileged methods + +**Owner:** engineer +**Description:** Add named bridge, TAP, address, route, and link lifecycle operations. Callers pass validated model data, +not raw `ip` argv. + +**Acceptance criteria:** +- [ ] No unprivileged code can ask root-side code to run arbitrary `ip` arguments or `netns exec`. +- [ ] Operations validate interface names, addresses, owning network/VM IDs, and idempotent expected state. +- [ ] Existing network Service remains the intra-domain owner; API boundaries do not move. + +**Verification:** +- [ ] `FakeRunner` tests assert exact root-side commands for each typed method and reject shell/argv injection. +- [ ] Network L1 lifecycle tests pass. + +**Dependencies:** Task 3 +**Files likely touched:** privileged network handlers, `internal/core/network/service.go`, typed model/request tests +**Estimated scope:** M + +## Task 7: Replace raw firewall sudo calls with atomic typed batches + +**Owner:** engineer +**Description:** Move iptables/nftables application into privileged actions that accept validated firewall models and render +complete atomic backend batches root-side. + +**Acceptance criteria:** +- [ ] Caller cannot pass raw iptables/nft syntax, filenames, includes, or commands. +- [ ] Chain creation, parent jumps, ordering, and managed rules apply as one reported operation. +- [ ] Every runner/backend error propagates; initialization can no longer silently succeed. + +**Verification:** +- [ ] Backend tests inject failure for every chain/jump/batch stage. +- [ ] Golden tests cover exact iptables-restore and nft batch output from typed models. + +**Dependencies:** Task 3 +**Files likely touched:** privileged firewall handlers, `internal/lib/firewall/tracker.go`, `iptables.go`, `nftables.go`, tests +**Estimated scope:** M + +## Task 8: Replace raw host module/sysctl sudo calls + +**Owner:** engineer +**Description:** Expose only the exact host initialization actions mvmctl needs. Module names, sysctl keys, and values +are compiled/root-policy allowlists, not caller-selected commands. + +**Acceptance criteria:** +- [ ] Privileged dispatch supports only named host readiness/remediation operations from ADR-0016. +- [ ] Arbitrary modprobe config/path and arbitrary sysctl keys are impossible. +- [ ] Detection stays unprivileged; only mutation crosses the privileged boundary. + +**Verification:** +- [ ] L1 tests reject all unlisted modules/keys/values and confirm supported host-init actions. +- [ ] Host probe tests remain green. + +**Dependencies:** Task 3 +**Files likely touched:** privileged host handlers, `internal/core/host/service.go`, `utils.go`, `probe.go`, tests +**Estimated scope:** M + +## Task 9: Replace sudoers with the single-binary privileged policy + +**Owner:** engineer +**Description:** After Tasks 4-8 migrate every callsite, atomically replace the wildcard CLI and raw system-binary +entries with the internal privileged entry point of root-owned `/usr/local/bin/mvm`. + +**Acceptance criteria:** +- [ ] Generated sudoers contains no user-owned `mvm`, `ip`, `iptables`, `nft`, `modprobe`, or `sysctl` command. +- [ ] `visudo -c` validation occurs before activation and legacy insecure entries are detected. +- [ ] Failure preserves the prior file and reports that the host remains unsafe/uninitialized. + +**Verification:** +- [ ] Unit golden test covers the exact sudoers content. +- [ ] L2 confirms every removed command fails under `sudo -n` while supported CLI flows still work. + +**Dependencies:** Tasks 4, 5, 6, 7, 8 +**Files likely touched:** `internal/core/host/utils.go`, `utils_test.go`, `internal/infra/constants.go`, host-init tests +**Estimated scope:** M + +### Checkpoint B: Privilege boundary + +- [ ] Tasks 2-9 complete. +- [ ] Passwordless sudo reaches only the immutable binary's early privileged entry point. +- [ ] User-owned binary replacement, raw system commands, and generic privileged requests are rejected in L1/L2. + +## Phase 2: Trusted release and safe filesystem operations + +## Task 10: Implement independently verified atomic release installation + +**Owner:** engineer +**Description:** Move release authority inside privileged dispatch, harden archive extraction, and create an atomic root-owned +release manifest for the exact pair. + +**Acceptance criteria:** +- [ ] Privileged code constructs the allowed version/architecture URL and independently obtains expected integrity metadata. +- [ ] Archive parser rejects traversal, links, devices, duplicates, unexpected members, and size/count overflow. +- [ ] Pair installation is atomic, fsynced, content-addressed, and cannot replace a referenced release. + +**Verification:** +- [ ] Negative tests include a caller-authored archive plus matching caller hash and all archive abuse cases. +- [ ] Interrupted install leaves either the old complete pair or no pair, never a partial trusted directory. + +**Dependencies:** Tasks 4 and 9 +**Files likely touched:** `internal/service/jailer/entry.go`, privileged release handler/manifest, archive tests +**Estimated scope:** M + +## Task 11: Introduce descriptor-relative managed-resource access + +**Owner:** engineer +**Description:** Replace canonicalize/validate/reopen logic with pinned directory/file descriptors and fixed basenames. + +**Acceptance criteria:** +- [ ] Managed roots use `openat2`/`openat` no-symlink/beneath resolution and `fstat` ownership/type checks. +- [ ] Privileged PID/status/manifest writes use root-owned fixed paths, not user-selected basenames. +- [ ] No privileged operation reopens a user-controlled absolute path after validation. + +**Verification:** +- [ ] Race tests continuously rename/replace every user-owned parent with symlinks during privileged operations. +- [ ] Static review finds no `EvalSymlinks`-then-open security boundary in privileged/Jailer code. + +**Dependencies:** Tasks 3 and 4 +**Files likely touched:** privileged path module/tests, `internal/service/jailer/entry.go`, `manifest.go` +**Estimated scope:** M + +## Task 12: Make launch mounts private and live mounts namespace-aware + +**Owner:** engineer +**Description:** Unshare a private mount namespace before pre-launch mounts. For live mounts, verify the registered +process, pin source/target descriptors, enter its mount namespace, and apply restrictive mount flags. + +**Acceptance criteria:** +- [ ] Launch mounts never appear in the original host namespace. +- [ ] Live volume/snapshot mounts are visible to jailed Firecracker and cannot target host paths through symlinks. +- [ ] VM exit destroys namespace mounts; cleanup never recursively traverses a mounted user directory. + +**Verification:** +- [ ] L1 namespace-operation tests cover PID mismatch and target/source replacement. +- [ ] L2 asserts visibility inside the VM namespace and absence from host mountinfo before/after failure. + +**Dependencies:** Task 11 +**Files likely touched:** privileged namespace/mount modules and tests, `internal/service/jailer/entry.go` +**Estimated scope:** M + +## Phase 3: Lifecycle, cgroups, and exact identity + +## Task 13: Add root-owned instance records and per-VM locking + +**Owner:** engineer +**Description:** Persist minimal UID/release/process/cleanup metadata under `/var/lib/mvmctl/instances` and serialize all +privileged VM actions with `/run/mvmctl` locks. + +**Acceptance criteria:** +- [ ] Record creation/update is atomic, root-owned, and bound to `SUDO_UID`. +- [ ] Every launch, signal, setns, mount, cleanup, and release-reference check holds the VM lock. +- [ ] Other UIDs and PID/start-time/cgroup mismatches fail closed. + +**Verification:** +- [ ] Tests cover lock contention, stale lock, cross-UID request, PID reuse, corrupt/truncated record, and atomic recovery. +- [ ] Root record contains no full user VM config or secrets. + +**Dependencies:** Tasks 10, 11 +**Files likely touched:** privileged instance record/lock/process modules and tests +**Estimated scope:** M + +## Task 14: Make VM spawn and abort crash-consistent + +**Owner:** engineer +**Description:** Persist a provisional VM before spawn, retain a lifecycle handle immediately after spawn, and route +every post-spawn failure through a typed verified abort. + +**Acceptance criteria:** +- [ ] No error path after spawn can return without terminating or explicitly retaining a manageable registered VM. +- [ ] Abort verifies process identity, performs TERM/wait/KILL, then cgroup/chroot/record cleanup in order. +- [ ] Snapshot-restore batch rollback stops prior restored VMs before deleting DB/VM directories. + +**Verification:** +- [ ] Failure injection covers relay start, API socket, DB writes, verification, snapshot load, and batch rollback. +- [ ] Each case asserts no live PID, cgroup, mount, jail, or orphaned root record. + +**Dependencies:** Tasks 12 and 13 +**Files likely touched:** `internal/core/vm/firecracker.go`, `pkg/api/vm.go`, `pkg/api/snapshot.go`, focused tests +**Estimated scope:** M + +## Task 15: Add startup and pre-launch privileged reconciliation + +**Owner:** engineer +**Description:** Reconcile instance records, verified processes, cgroups, and chroot directories after CLI interruption or +host reboot without trusting the user DB as the root authorization source. + +**Acceptance criteria:** +- [ ] Reconcile distinguishes running, stopped-clean, stale, corrupt, and foreign-owner records. +- [ ] It never signals an unverified PID or recursively deletes a path that may contain a mount. +- [ ] Launch runs per-UID/per-VM reconciliation before creating new privileged state. + +**Verification:** +- [ ] L1 state-machine tests cover every classification and idempotent retry. +- [ ] L2 kill/reboot scenarios converge to zero stale resources or a precise actionable error. + +**Dependencies:** Tasks 13 and 14 +**Files likely touched:** privileged reconciliation module/tests, Jailer service caller, host readiness integration +**Estimated scope:** M + +## Task 16: Persist exact Firecracker/Jailer identity transactionally + +**Owner:** engineer +**Description:** Add snapshot Jailer identity, uniqueness, transactional pair repository operations, and deterministic +legacy backfill. Compare user DB pair hashes with the root release manifest before launch. + +**Acceptance criteria:** +- [ ] VM and snapshot always persist both exact IDs/hashes; restored/migrated VMs cannot retain an empty Jailer ID. +- [ ] Pair install/default/remove DB updates are transactional and return all errors. +- [ ] Ambiguous or unmatched legacy data fails closed instead of selecting an arbitrary same-version row. + +**Verification:** +- [ ] Migration tests cover unique, duplicate, ambiguous, missing, referenced, and rollback cases. +- [ ] Restore-then-hotplug test sees the persisted Jailer identity. + +**Dependencies:** Tasks 10 and 13 +**Files likely touched:** new DB migration, binary/snapshot model, binary repository/service, migration tests +**Estimated scope:** M + +## Task 17: Centralize jail-visible path translation and restore parity + +**Owner:** engineer +**Description:** Put one typed translator in the VM spawner/controller seam and use it for config, rootfs, sockets, +vsock, snapshot create/restore, and live volume hotplug. + +**Acceptance criteria:** +- [ ] Snapshot create passes/exposes its directory and uses `/snapshot/...`; restore uses the same mapping. +- [ ] Rootfs, API/vsock sockets, config, and volume paths have explicit host and jail representations. +- [ ] Restored jailed VMs use namespace-aware volume attach/detach. + +**Verification:** +- [ ] L1 tests assert every exact Firecracker API path for create, load, and hotplug. +- [ ] Prepared L2 Jailer snapshot/volume tests pass unchanged except for fixture correctness. + +**Dependencies:** Tasks 12, 14, 16 +**Files likely touched:** `internal/core/vm/jailer.go`, `controller.go`, `pkg/api/snapshot.go`, tests +**Estimated scope:** M + +## Task 18: Harden cgroup policy, readiness, and cleanup + +**Owner:** engineer +**Description:** Keep the corrected leaf path, derive all values from typed VM/host policy, validate controller +enablement, and ensure process-aware removal. Add I/O limits if retained by the approved M2 specification. + +**Acceptance criteria:** +- [ ] Privileged dispatch accepts VM resource intent, never raw cgroup key/value pairs, and applies host-policy bounds. +- [ ] Host readiness always reports missing cgroup v2/version/controllers/swap support as critical. +- [ ] Cleanup cannot remove or kill a cgroup belonging to an unverified instance. + +**Verification:** +- [ ] L1 covers version zero, unavailable/unenableable controllers, mismatched values, foreign membership, and cleanup. +- [ ] L2 measures supported Firecracker versions before the memory-headroom default is approved. + +**Dependencies:** Tasks 13 and 14 +**Files likely touched:** privileged cgroup module, `internal/core/host/detector.go`, `probe.go`, model policy, tests +**Estimated scope:** M + +### Checkpoint C: Jailer lifecycle parity + +- [ ] Tasks 10-18 complete. +- [ ] All normal lifecycle and snapshot/hotplug L1 tests pass. +- [ ] Fault injection proves cleanup and reconciliation are idempotent. +- [ ] Exact release and owner identity are present on every VM/snapshot path. + +## Phase 4: Fail-closed routed policy + +## Task 19: Make firewall initialization observable and atomic + +**Owner:** engineer +**Description:** Return and propagate errors for chain/jump setup and apply each complete managed generation through an +atomic backend batch. + +**Acceptance criteria:** +- [ ] `Initialize` and jump ordering return typed failure; no runner result is silently discarded. +- [ ] Reconciliation verifies required parent jumps and default-deny placement before success. +- [ ] Partial backend application cannot be reported as successful. + +**Verification:** +- [ ] Failure injection for every parent jump/reorder/batch command returns `CodePolicySyncFailed`. +- [ ] Both firewall backend test suites pass. + +**Dependencies:** Task 7 +**Files likely touched:** `internal/lib/firewall/tracker.go`, `iptables.go`, `nftables.go`, network policy tests +**Estimated scope:** M + +## Task 20: Reorder policy/resource deletion around kernel reconciliation + +**Owner:** engineer +**Description:** Apply the desired kernel generation before deleting policy/VM/network identity or releasing an IP; +restore the prior generation if the DB commit fails. + +**Acceptance criteria:** +- [ ] Failed `policy rm` retains authoritative intent and retryability. +- [ ] Failed VM/network removal cannot release an address while stale allow rules remain. +- [ ] Derived firewall rows and intent transition transactionally after kernel success. + +**Verification:** +- [ ] Unit tests inject kernel and DB failures at every boundary and assert rollback/recovery details. +- [ ] IP-reuse integration test proves no replacement VM inherits a stale allow. + +**Dependencies:** Task 19 +**Files likely touched:** `pkg/api/policy.go`, `pkg/api/vm.go`, network removal API, firewall repository, tests +**Estimated scope:** M + +## Phase 5: System verification and documentation + +## Task 21: Repair L2 fixture isolation and add privilege/fault coverage + +**Owner:** qa-engineer +**Description:** Fix structured cgroup-setting restoration, then add real-host tests for immutable sudo, release trust, +multi-user isolation, path races, PID reuse, kill-at-stage recovery, namespace mounts, and IP reuse. + +**Acceptance criteria:** +- [ ] Every modified setting is restored exactly or reset, and cleanup failures fail the test. +- [ ] New tests use Tier-2 fixtures and known-limitation conventions from the system-test architecture. +- [ ] Repeated suite execution leaves the runner with no leaked VM process, mount, cgroup, network, or config override. + +**Verification:** +- [ ] Targeted L2 domains pass twice consecutively on the prepared runner VM. +- [ ] Post-suite leak audit is empty. + +**Dependencies:** Tasks 9, 10-20 +**Files likely touched:** `tests/system/vm/test_cgroup.py`, `test_jailer.py`, policy tests, shared fixtures, coverage matrix +**Estimated scope:** M + +## Task 22: Run full CI, L2 release qualification, and resource measurements + +**Owner:** qa-engineer +**Description:** Execute repository CI in its exact order, the full relevant L2 matrix, upgrade/rollback tests, and +versioned Firecracker overhead measurements. + +**Acceptance criteria:** +- [ ] Tidy, format, golines, generate, vet, and full tests pass. +- [ ] L2 privilege/Jailer/cgroup/snapshot/volume/policy/upgrade suites pass on supported architectures. +- [ ] Memory headroom and I/O-limit decisions are backed by recorded measurements, not assumed constants. + +**Verification:** +- [ ] Attach command output and runner environment to the release report. +- [ ] No critical/high review finding remains open or deferred without explicit release rejection. + +**Dependencies:** Task 21 +**Files likely touched:** Test reports/fixtures only +**Estimated scope:** S + +## Task 23: Reconcile all documentation with verified behavior + +**Owner:** architect +**Description:** Apply the documentation matrix from `tasks/plan.md`, using the final implementation and test evidence +as the source of truth. + +**Acceptance criteria:** +- [ ] ADR statuses, state paths, system-install/privileged-mode setup, service list, cgroup paths, lifecycle, and + limitations are accurate. +- [ ] Roadmap and changelog do not claim completion beyond executed evidence. +- [ ] Repository-wide searches find no stale wildcard-sudo, `/var/run` full-state, old cgroup path, or service-count claim. +- [ ] Documentation lands in reviewable slices: architecture/rules, user/operator guides, then testing/release records. + +**Verification:** +- [ ] Architect reads the final implementation diff for interfaces, structs, imports, and state locations. +- [ ] Documentation link/path/command checks pass and QA verifies operator procedures on the runner. + +**Dependencies:** Task 22 +**Files likely touched:** Files in the documentation matrix; split into reviewable documentation-only commits +**Estimated scope:** M per documentation slice + +## Task 24: Independently verify documentation accuracy + +**Owner:** architect with qa-engineer verification +**Description:** Treat documentation as a release artifact. Trace each security- and operator-relevant claim to final +code, then execute every installation, upgrade, recovery, and troubleshooting procedure on the release candidate. + +**Acceptance criteria:** +- [ ] Every changed behavior is described consistently across ADRs, `CONTEXT.md`, standards, runtime, operator docs, + roadmap, test architecture, coverage matrix, and changelog. +- [ ] Current behavior, migration behavior, known limitations, and future work are clearly distinguished. +- [ ] No stale command/path/ownership/service-count/cgroup/snapshot/sudo claim remains anywhere in tracked Markdown. + +**Verification:** +- [ ] Architect signs off after reading the final implementation diff rather than relying on implementation summaries. +- [ ] QA executes documented install, helper-mode, upgrade, rollback, reconcile, and troubleshooting commands on L2. +- [ ] A repository-wide terminology/path audit and Markdown link check pass. + +**Dependencies:** Task 23 +**Files likely touched:** Documentation corrections found by the audit; release verification record +**Estimated scope:** M + +### Checkpoint D: Release + +- [ ] Tasks 21-24 complete. +- [ ] Full CI and L2 release report pass. +- [ ] No leaked runtime resources after the fault suite. +- [ ] Architect and QA independently approve documentation accuracy. +- [ ] Human approves the final code, docs, migration, and release decision. + +## Follow-up project: XDG state separation + +Do not include this in the Jailer remediation. After release, write a separate ADR and migration plan to split: + +- durable user metadata into `$XDG_STATE_HOME/mvmctl`; +- cacheable downloads into `$XDG_CACHE_HOME/mvmctl`; +- user sockets/PIDs into `$XDG_RUNTIME_DIR/mvmctl`; +- user configuration into `$XDG_CONFIG_HOME/mvmctl`. + +The follow-up must preserve `MVM_CACHE_DIR` compatibility, migrate atomically, detect partial moves, and support rollback. From db6fd83957e4ab53de57efccd8426f939bbec365 Mon Sep 17 00:00:00 2001 From: AlanD20 Date: Thu, 20 Aug 2026 07:56:38 +0200 Subject: [PATCH 0007/1457] test: restore cgroup overrides exactly --- tests/system/vm/test_cgroup.py | 149 +++++++++++++++++++++++---------- 1 file changed, 107 insertions(+), 42 deletions(-) diff --git a/tests/system/vm/test_cgroup.py b/tests/system/vm/test_cgroup.py index 684f0c31..6045aed8 100644 --- a/tests/system/vm/test_cgroup.py +++ b/tests/system/vm/test_cgroup.py @@ -3,9 +3,11 @@ from __future__ import annotations import json +import re import shlex import uuid -from typing import Any +from dataclasses import dataclass +from typing import Any, Iterator import pytest @@ -24,6 +26,105 @@ pytest.mark.slow, ] +_CGROUP_CONFIG_OVERRIDES = { + "cgroup_vmm_headroom_mib": "128", + "cgroup_cpu_weight": "100", + "cgroup_pids_max": "256", + "cgroup_swap_max_bytes": "0", +} + + +@dataclass(frozen=True) +class _ConfigOverride: + exists: bool + value: str + + +def _read_cgroup_config_overrides( + runner_vm: str, +) -> dict[str, _ConfigOverride]: + """Capture each cgroup setting's exact integer override, if present.""" + listing = _run_mvm( + runner_vm, "config", "get", "defaults.vm" + ).stdout + snapshots: dict[str, _ConfigOverride] = {} + for key in _CGROUP_CONFIG_OVERRIDES: + match = re.search( + rf"^\s*{re.escape(key)} = (?P-?\d+) " + r"\((?Poverride|default): (?P-?\d+), " + r"type: int\)$", + listing, + flags=re.MULTILINE, + ) + assert match is not None, ( + f"could not capture defaults.vm.{key} override state from " + "the config category listing" + ) + assert match.group("effective") == match.group("stored"), ( + f"defaults.vm.{key} effective and stored values disagree in: " + f"{match.group(0)!r}" + ) + is_override = match.group("source") == "override" + snapshots[key] = _ConfigOverride( + exists=is_override, + value=match.group("stored"), + ) + return snapshots + + +@pytest.fixture +def cgroup_config_overrides(runner_vm: str) -> Iterator[None]: + """Apply cgroup test overrides and restore the exact prior state.""" + originals = _read_cgroup_config_overrides(runner_vm) + modified: list[str] = [] + try: + for key, value in _CGROUP_CONFIG_OVERRIDES.items(): + modified.append(key) + _run_mvm( + runner_vm, "config", "set", "defaults.vm", key, value + ) + yield + finally: + failures: list[str] = [] + for key in modified: + original = originals[key] + # Collect each result so one failure cannot skip the remaining keys. + if original.exists: + result = _run_mvm( + runner_vm, + "config", + "set", + "defaults.vm", + key, + original.value, + check=False, + ) + else: + result = _run_mvm( + runner_vm, + "config", + "reset", + "defaults.vm", + key, + check=False, + ) + if result.returncode != 0: + failures.append( + f"defaults.vm.{key}: rc={result.returncode}, " + f"stdout={result.stdout!r}, stderr={result.stderr!r}" + ) + + restored = _read_cgroup_config_overrides(runner_vm) + for key in modified: + if restored[key] != originals[key]: + failures.append( + f"defaults.vm.{key}: expected {originals[key]!r}, " + f"got {restored[key]!r}" + ) + assert not failures, "cgroup config restoration failed:\n" + "\n".join( + failures + ) + def _vm_entry(runner_vm: str, vm_name: str) -> dict[str, Any] | None: entries: list[dict[str, Any]] = json.loads( @@ -37,16 +138,13 @@ def _read(runner_vm: str, path: str) -> str: class TestVMCgroupEnvelope: - def test_running_vm_envelope_and_lifecycle_cleanup(self, runner_vm: str) -> None: + def test_running_vm_envelope_and_lifecycle_cleanup( + self, + runner_vm: str, + cgroup_config_overrides: None, + ) -> None: vm_name = f"sys-cgroup-{uuid.uuid4().hex[:8]}" net_name = f"sys-cgroup-net-{uuid.uuid4().hex[:6]}" - config_keys = { - "cgroup_vmm_headroom_mib": "128", - "cgroup_cpu_weight": "100", - "cgroup_pids_max": "256", - "cgroup_swap_max_bytes": "0", - } - originals: dict[str, str | None] = {} vm_id = "" ensure_vm_deps(runner_vm) controller_probe = _guest_run( @@ -68,19 +166,6 @@ def test_running_vm_envelope_and_lifecycle_cleanup(self, runner_vm: str) -> None "--non-interactive", ) try: - for key, value in config_keys.items(): - current = _run_mvm( - runner_vm, "config", "get", "defaults.vm", key, check=False - ) - originals[key] = ( - current.stdout.strip() - if current.returncode == 0 and current.stdout.strip() - else None - ) - _run_mvm( - runner_vm, "config", "set", "defaults.vm", key, value - ) - created = _run_mvm( runner_vm, "vm", @@ -182,23 +267,3 @@ def test_running_vm_envelope_and_lifecycle_cleanup(self, runner_vm: str) -> None _run_mvm( runner_vm, "network", "rm", net_name, "--force", check=False ) - for key, value in originals.items(): - if value is None: - _run_mvm( - runner_vm, - "config", - "reset", - "defaults.vm", - key, - check=False, - ) - else: - _run_mvm( - runner_vm, - "config", - "set", - "defaults.vm", - key, - value, - check=False, - ) From 9766fedabe8037863cb4f6cb21452902a5cc8432 Mon Sep 17 00:00:00 2001 From: AlanD20 Date: Thu, 20 Aug 2026 08:20:56 +0200 Subject: [PATCH 0008/1457] feat: establish secure privileged dispatch boundary --- cmd/mvm/main.go | 43 ++- cmd/mvm/main_test.go | 76 +++++ cmd/mvm/normal.go | 35 +++ internal/infra/constants.go | 3 + internal/infra/constants_test.go | 4 + internal/service/privileged/authorization.go | 114 ++++++++ .../service/privileged/authorization_test.go | 147 ++++++++++ internal/service/privileged/doc.go | 3 + internal/service/privileged/entry.go | 70 +++++ .../service/privileged/entry_order_test.go | 105 +++++++ internal/service/privileged/entry_test.go | 91 ++++++ internal/service/privileged/environment.go | 48 ++++ .../service/privileged/environment_test.go | 85 ++++++ internal/service/privileged/executable.go | 198 +++++++++++++ .../service/privileged/executable_test.go | 267 ++++++++++++++++++ internal/service/privileged/identity.go | 68 +++++ internal/service/privileged/identity_test.go | 80 ++++++ internal/service/privileged/protocol.go | 211 ++++++++++++++ internal/service/privileged/protocol_test.go | 166 +++++++++++ 19 files changed, 1790 insertions(+), 24 deletions(-) create mode 100644 cmd/mvm/main_test.go create mode 100644 cmd/mvm/normal.go create mode 100644 internal/service/privileged/authorization.go create mode 100644 internal/service/privileged/authorization_test.go create mode 100644 internal/service/privileged/doc.go create mode 100644 internal/service/privileged/entry.go create mode 100644 internal/service/privileged/entry_order_test.go create mode 100644 internal/service/privileged/entry_test.go create mode 100644 internal/service/privileged/environment.go create mode 100644 internal/service/privileged/environment_test.go create mode 100644 internal/service/privileged/executable.go create mode 100644 internal/service/privileged/executable_test.go create mode 100644 internal/service/privileged/identity.go create mode 100644 internal/service/privileged/identity_test.go create mode 100644 internal/service/privileged/protocol.go create mode 100644 internal/service/privileged/protocol_test.go diff --git a/cmd/mvm/main.go b/cmd/mvm/main.go index cc165150..158cc45d 100644 --- a/cmd/mvm/main.go +++ b/cmd/mvm/main.go @@ -2,37 +2,32 @@ package main import ( "context" + "io" "log/slog" "os" - "os/signal" - "syscall" - "mvmctl/internal/app" - "mvmctl/internal/cli" - "mvmctl/internal/cli/common" + "mvmctl/internal/service/privileged" ) -func main() { - ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM) - defer stop() +type entrypoints struct { + privileged func(context.Context, []string, io.Reader) error + normal func() int +} - op, cleanup, err := app.Initialize(ctx) - if err != nil { - slog.Error("initialization failed", "error", err) - os.Exit(1) - } - if cleanup != nil { - defer cleanup() - } +func main() { + os.Exit(run(context.Background(), os.Args[1:], os.Stdin, entrypoints{ + privileged: privileged.Run, + normal: runCLI, + })) +} - // Execute CLI - rootCmd := cli.NewRootCmd(op) - if err := rootCmd.ExecuteContext(ctx); err != nil { - // Delegate ALL error handling to the single shared handler in helpers.go. - // HandleErrors returns nil for non-fatal errors (BrokenPipe), non-nil for - // everything else. Exit with code 1 so Go respects the error exit convention. - if common.HandleErrors(func() error { return err })() != nil { - os.Exit(1) +func run(ctx context.Context, args []string, input io.Reader, entries entrypoints) int { + if privileged.IsInvocation(args) { + if err := entries.privileged(ctx, args, input); err != nil { + slog.Error("privileged request failed", "error", err) + return 1 } + return 0 } + return entries.normal() } diff --git a/cmd/mvm/main_test.go b/cmd/mvm/main_test.go new file mode 100644 index 00000000..51856223 --- /dev/null +++ b/cmd/mvm/main_test.go @@ -0,0 +1,76 @@ +package main + +import ( + "context" + "errors" + "io" + "strings" + "testing" + + "github.com/stretchr/testify/assert" + + "mvmctl/internal/service/privileged" +) + +func TestRun_RoutesPrivilegedInvocationBeforeNormalInitialization(t *testing.T) { + sentinel := errors.New("privileged sentinel") + privilegedCalls := 0 + normalCalls := 0 + + exitCode := run(context.Background(), []string{privileged.Marker, "unknown"}, strings.NewReader("{}"), entrypoints{ + privileged: func(context.Context, []string, io.Reader) error { + privilegedCalls++ + return sentinel + }, + normal: func() int { + normalCalls++ + return 0 + }, + }) + + assert.Equal(t, 1, exitCode) + assert.Equal(t, 1, privilegedCalls) + assert.Zero(t, normalCalls) +} + +// Rationale: Unsupported protocol versions must remain inside the reserved +// boundary instead of falling through to Cobra or user-state initialization. +func TestRun_RoutesUnsupportedPrivilegedVersionToFailClosedEntry(t *testing.T) { + privilegedCalls := 0 + normalCalls := 0 + + exitCode := run(context.Background(), []string{"__mvm_privileged_v999", "unknown"}, nil, entrypoints{ + privileged: func(context.Context, []string, io.Reader) error { + privilegedCalls++ + return nil + }, + normal: func() int { + normalCalls++ + return 0 + }, + }) + + assert.Zero(t, exitCode) + assert.Equal(t, 1, privilegedCalls) + assert.Zero(t, normalCalls) +} + +func TestRun_RoutesOrdinaryArgumentsToNormalCLI(t *testing.T) { + privilegedCalls := 0 + normalCalls := 0 + + exitCode := run(context.Background(), []string{"vm", "list"}, nil, entrypoints{ + privileged: func(context.Context, []string, io.Reader) error { + privilegedCalls++ + return nil + }, + normal: func() int { + normalCalls++ + return 7 + }, + }) + + assert.Equal(t, 7, exitCode) + assert.Zero(t, privilegedCalls) + assert.Equal(t, 1, normalCalls) +} diff --git a/cmd/mvm/normal.go b/cmd/mvm/normal.go new file mode 100644 index 00000000..64bb8e14 --- /dev/null +++ b/cmd/mvm/normal.go @@ -0,0 +1,35 @@ +package main + +import ( + "context" + "log/slog" + "os/signal" + "syscall" + + "mvmctl/internal/app" + "mvmctl/internal/cli" + "mvmctl/internal/cli/common" +) + +func runCLI() int { + ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM) + defer stop() + + op, cleanup, err := app.Initialize(ctx) + if err != nil { + slog.Error("initialization failed", "error", err) + return 1 + } + if cleanup != nil { + defer cleanup() + } + + rootCmd := cli.NewRootCmd(op) + if err := rootCmd.ExecuteContext(ctx); err != nil { + // Delegate all public CLI error rendering to the shared handler. + if common.HandleErrors(func() error { return err })() != nil { + return 1 + } + } + return 0 +} diff --git a/internal/infra/constants.go b/internal/infra/constants.go index eba65783..45479cf1 100644 --- a/internal/infra/constants.go +++ b/internal/infra/constants.go @@ -21,6 +21,9 @@ const BootstrapName = "mvmctl" // CLIName is the canonical CLI name. A compiled Go binary has one name. const CLIName = "mvm" +// SystemBinaryPath is the canonical root-owned executable used for privileged dispatch. +const SystemBinaryPath = "/usr/local/bin/mvm" + // MvmctlGitHubRepo is the GitHub repository for mvmctl self-update. const MvmctlGitHubRepo = "AlanD20/mvmctl" diff --git a/internal/infra/constants_test.go b/internal/infra/constants_test.go index 47c4990f..cbe29f7a 100644 --- a/internal/infra/constants_test.go +++ b/internal/infra/constants_test.go @@ -57,6 +57,10 @@ func TestGetDefault(t *testing.T) { }) } +func TestSystemBinaryPath(t *testing.T) { + assert.Equal(t, "/usr/local/bin/mvm", infra.SystemBinaryPath) +} + // --- EnvKey --- // Rationale: EnvKey constructs environment variable names used throughout the // codebase for config overrides. Wrong prefix would cause silent misconfiguration. diff --git a/internal/service/privileged/authorization.go b/internal/service/privileged/authorization.go new file mode 100644 index 00000000..4a559f29 --- /dev/null +++ b/internal/service/privileged/authorization.go @@ -0,0 +1,114 @@ +package privileged + +import ( + "fmt" + "math" + "os/user" + "strconv" + + "mvmctl/internal/infra" + "mvmctl/pkg/errs" +) + +type authorizationDeps struct { + lookupUserID func(string) (*user.User, error) + lookupGroup func(string) (*user.Group, error) + groupIDs func(*user.User) ([]string, error) +} + +func authorizeCaller(caller callerIdentity, deps authorizationDeps) error { + uid := strconv.FormatUint(uint64(caller.uid), 10) + invokingUser, err := deps.lookupUserID(uid) + if err != nil { + return errs.New( + errs.CodePrivilegeRequired, + fmt.Sprintf("look up invoking UID %s: %v", uid, err), + ) + } + resolvedUID, err := parseIdentityNumber(invokingUser.Uid) + if err != nil || resolvedUID != caller.uid { + return errs.New( + errs.CodePrivilegeRequired, + fmt.Sprintf("resolved user identity does not match requested UID %s", uid), + ) + } + + authorizationGroup, err := deps.lookupGroup(infra.MVMUnixGroup) + if err != nil { + return errs.New( + errs.CodePrivilegeRequired, + fmt.Sprintf("look up authorization group %s: %v", infra.MVMUnixGroup, err), + ) + } + authorizationGID, err := parseIdentityNumber(authorizationGroup.Gid) + if err != nil { + return errs.New( + errs.CodePrivilegeRequired, + fmt.Sprintf("invalid authorization group ID for %s", infra.MVMUnixGroup), + ) + } + if authorizationGID == 0 { + return errs.New( + errs.CodePrivilegeRequired, + "authorization group must identify a non-root group", + ) + } + + primaryGID, err := parseIdentityNumber(invokingUser.Gid) + if err != nil { + return errs.New( + errs.CodePrivilegeRequired, + fmt.Sprintf("invalid primary group ID for invoking UID %s", uid), + ) + } + if primaryGID == authorizationGID { + return nil + } + + groupIDs, err := deps.groupIDs(invokingUser) + if err != nil { + return errs.New( + errs.CodePrivilegeRequired, + fmt.Sprintf("look up current group membership for invoking UID %s: %v", uid, err), + ) + } + for _, groupID := range groupIDs { + parsedGroupID, err := parseIdentityNumber(groupID) + if err != nil { + return errs.New( + errs.CodePrivilegeRequired, + fmt.Sprintf("invalid current group membership for invoking UID %s", uid), + ) + } + if parsedGroupID == authorizationGID { + return nil + } + } + + return errs.New( + errs.CodePrivilegeRequired, + fmt.Sprintf( + "invoking UID %s is not a current member of the %s group", + uid, + infra.MVMUnixGroup, + ), + ) +} + +func parseIdentityNumber(value string) (uint32, error) { + parsed, err := strconv.ParseUint(value, 10, 32) + if err != nil || parsed == math.MaxUint32 { + return 0, fmt.Errorf("invalid identity number %q", value) + } + return uint32(parsed), nil +} + +func realAuthorizationDeps() authorizationDeps { + return authorizationDeps{ + lookupUserID: user.LookupId, + lookupGroup: user.LookupGroup, + groupIDs: func(invokingUser *user.User) ([]string, error) { + return invokingUser.GroupIds() + }, + } +} diff --git a/internal/service/privileged/authorization_test.go b/internal/service/privileged/authorization_test.go new file mode 100644 index 00000000..feee2ca0 --- /dev/null +++ b/internal/service/privileged/authorization_test.go @@ -0,0 +1,147 @@ +package privileged + +import ( + "errors" + "os/user" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "mvmctl/internal/infra" + "mvmctl/pkg/errs" +) + +func TestAuthorizeCaller_AcceptsCurrentUIDBoundMembership(t *testing.T) { + tests := map[string]struct { + user *user.User + groupIDs []string + }{ + "supplementary_group": { + user: &user.User{Uid: "1000", Gid: "1000", Username: "alice"}, + groupIDs: []string{"1000", "2000"}, + }, + "primary_group": { + user: &user.User{Uid: "1000", Gid: "2000", Username: "alice"}, + groupIDs: []string{"1000"}, + }, + } + + for name, tc := range tests { + t.Run(name, func(t *testing.T) { + requestedUID := "" + err := authorizeCaller(callerIdentity{uid: 1000, gid: 3000}, authorizationDeps{ + lookupUserID: func(uid string) (*user.User, error) { + requestedUID = uid + return tc.user, nil + }, + lookupGroup: func(name string) (*user.Group, error) { + assert.Equal(t, infra.MVMUnixGroup, name) + return &user.Group{Name: name, Gid: "2000"}, nil + }, + groupIDs: func(got *user.User) ([]string, error) { + assert.Same(t, tc.user, got) + return tc.groupIDs, nil + }, + }) + require.NoError(t, err) + assert.Equal(t, "1000", requestedUID) + }) + } +} + +// Rationale: SUDO_GID describes the caller's launch context, not current mvm +// authorization. Removing the user from mvm must revoke the next invocation. +func TestAuthorizeCaller_RejectsMissingCurrentMembershipDespiteSudoGID(t *testing.T) { + err := authorizeCaller(callerIdentity{uid: 1000, gid: 2000}, authorizationDeps{ + lookupUserID: func(string) (*user.User, error) { + return &user.User{Uid: "1000", Gid: "1000", Username: "alice"}, nil + }, + lookupGroup: func(string) (*user.Group, error) { + return &user.Group{Name: infra.MVMUnixGroup, Gid: "2000"}, nil + }, + groupIDs: func(*user.User) ([]string, error) { return []string{"1000", "3000"}, nil }, + }) + require.Error(t, err) + assert.ErrorContains(t, err, "is not a current member") + assert.Equal(t, errs.CodePrivilegeRequired, errs.AsDomainError(err).Code) +} + +func TestAuthorizeCaller_FailsClosedOnNSSAndIdentityErrors(t *testing.T) { + tests := map[string]struct { + mutate func(*authorizationDeps) + wantErr string + }{ + "user_lookup_failure": { + mutate: func(deps *authorizationDeps) { + deps.lookupUserID = func(string) (*user.User, error) { return nil, errors.New("nss unavailable") } + }, + wantErr: "look up invoking UID", + }, + "uid_mismatch": { + mutate: func(deps *authorizationDeps) { + deps.lookupUserID = func(string) (*user.User, error) { + return &user.User{Uid: "1001", Gid: "1000", Username: "mallory"}, nil + } + }, + wantErr: "does not match requested UID", + }, + "group_lookup_failure": { + mutate: func(deps *authorizationDeps) { + deps.lookupGroup = func(string) (*user.Group, error) { return nil, errors.New("group unavailable") } + }, + wantErr: "look up authorization group", + }, + "invalid_group_id": { + mutate: func(deps *authorizationDeps) { + deps.lookupGroup = func(string) (*user.Group, error) { + return &user.Group{Name: infra.MVMUnixGroup, Gid: "not-a-gid"}, nil + } + }, + wantErr: "invalid authorization group ID", + }, + "root_authorization_group": { + mutate: func(deps *authorizationDeps) { + deps.lookupGroup = func(string) (*user.Group, error) { + return &user.Group{Name: infra.MVMUnixGroup, Gid: "0"}, nil + } + }, + wantErr: "authorization group must identify a non-root group", + }, + "membership_lookup_failure": { + mutate: func(deps *authorizationDeps) { + deps.groupIDs = func(*user.User) ([]string, error) { return nil, errors.New("membership unavailable") } + }, + wantErr: "look up current group membership", + }, + "invalid_membership_group_id": { + mutate: func(deps *authorizationDeps) { + deps.groupIDs = func(*user.User) ([]string, error) { return []string{"bad-gid"}, nil } + }, + wantErr: "invalid current group membership", + }, + } + + for name, tc := range tests { + t.Run(name, func(t *testing.T) { + deps := memberAuthorizationDeps() + tc.mutate(&deps) + err := authorizeCaller(callerIdentity{uid: 1000, gid: 1000}, deps) + require.Error(t, err) + assert.ErrorContains(t, err, tc.wantErr) + assert.Equal(t, errs.CodePrivilegeRequired, errs.AsDomainError(err).Code) + }) + } +} + +func memberAuthorizationDeps() authorizationDeps { + return authorizationDeps{ + lookupUserID: func(string) (*user.User, error) { + return &user.User{Uid: "1000", Gid: "1000", Username: "alice"}, nil + }, + lookupGroup: func(string) (*user.Group, error) { + return &user.Group{Name: infra.MVMUnixGroup, Gid: "2000"}, nil + }, + groupIDs: func(*user.User) ([]string, error) { return []string{"1000", "2000"}, nil }, + } +} diff --git a/internal/service/privileged/doc.go b/internal/service/privileged/doc.go new file mode 100644 index 00000000..330c97c4 --- /dev/null +++ b/internal/service/privileged/doc.go @@ -0,0 +1,3 @@ +// Package privileged provides the early, fixed-action root dispatch boundary. +// Layer: Service — it never imports CLI, API, or core domain packages. +package privileged diff --git a/internal/service/privileged/entry.go b/internal/service/privileged/entry.go new file mode 100644 index 00000000..c2c48cfb --- /dev/null +++ b/internal/service/privileged/entry.go @@ -0,0 +1,70 @@ +package privileged + +import ( + "context" + "fmt" + "io" + "log/slog" + + "mvmctl/pkg/errs" +) + +type runtimeDeps struct { + identity identityDeps + environment environmentDeps + executable executableDeps + authorization authorizationDeps +} + +// Run validates and routes one privileged protocol invocation. +// CRITICAL: This entry point accepts only the reserved versioned marker and fixed action switch. +func Run(ctx context.Context, args []string, input io.Reader) error { + return run(ctx, args, input, runtimeDeps{ + identity: realIdentityDeps(), + environment: realEnvironmentDeps(), + executable: realExecutableDeps(), + authorization: realAuthorizationDeps(), + }) +} + +func run(ctx context.Context, args []string, input io.Reader, deps runtimeDeps) error { + if err := ctx.Err(); err != nil { + return errs.WrapMsg(errs.CodeInternal, "privileged request context ended", err) + } + request, err := parseInvocation(args) + if err != nil { + return err + } + caller, err := invokingIdentity(deps.identity) + if err != nil { + return err + } + if err := sanitizeEnvironment(deps.environment); err != nil { + return err + } + pinnedExecutable, err := verifySystemExecutable(deps.executable) + if err != nil { + return err + } + defer func() { + if err := pinnedExecutable.close(); err != nil { + slog.Warn("close privileged executable descriptors", "error", err) + } + }() + if err := authorizeCaller(caller, deps.authorization); err != nil { + return err + } + return dispatch(ctx, request, caller, input) +} + +func dispatch(_ context.Context, request invocation, _ callerIdentity, _ io.Reader) error { + // The action switch intentionally starts empty. Capability-specific typed + // handlers are added only as their migrations land and receive abuse tests. + switch request.action { + default: + return errs.New( + errs.CodeValidationFailed, + fmt.Sprintf("unknown privileged action %q", request.action), + ) + } +} diff --git a/internal/service/privileged/entry_order_test.go b/internal/service/privileged/entry_order_test.go new file mode 100644 index 00000000..d739ad0a --- /dev/null +++ b/internal/service/privileged/entry_order_test.go @@ -0,0 +1,105 @@ +package privileged + +import ( + "context" + "os/user" + "strings" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "mvmctl/internal/infra" +) + +// Rationale: Authorization lookups must not observe caller-controlled locale, +// loader, NSS, Kerberos, tool, HOME, PATH, or mvmctl variables. +func TestRun_SanitizesBeforeExecutableAndUIDBoundAuthorization(t *testing.T) { + var calls []string + environment := newEnvironmentStore(map[string]string{ + "HOME": "/home/alice", + "PATH": "/tmp/tools", + "LD_PRELOAD": "/tmp/inject.so", + "MVM_CACHE_DIR": "/tmp/state", + }) + environment.calls = &calls + + executable := newExecutableFixture() + originalOpen := executable.deps.open + executable.deps.open = func(path string, flags int, mode uint32) (int, error) { + calls = append(calls, "executable.open") + return originalOpen(path, flags, mode) + } + authorization := memberAuthorizationDeps() + originalLookupUser := authorization.lookupUserID + authorization.lookupUserID = func(uid string) (*user.User, error) { + calls = append(calls, "authorization.lookup_user") + return originalLookupUser(uid) + } + + err := run(context.Background(), []string{Marker, "unknown"}, strings.NewReader("{}"), runtimeDeps{ + identity: identityDeps{ + effectiveUID: func() int { + calls = append(calls, "identity.euid") + return 0 + }, + lookupEnv: func(key string) (string, bool) { + calls = append(calls, "identity."+key) + return "1000", true + }, + }, + environment: environment.deps(), + executable: executable.deps, + authorization: authorization, + }) + require.Error(t, err) + assert.ErrorContains(t, err, `unknown privileged action "unknown"`) + identityIndex := callIndex(calls, "identity.SUDO_GID") + sanitizeStartIndex := callIndex(calls, "environment.clear") + sanitizeEndIndex := callIndex(calls, "environment.set.LC_ALL") + executableIndex := callIndex(calls, "executable.open") + authorizationIndex := callIndex(calls, "authorization.lookup_user") + for _, index := range []int{ + identityIndex, sanitizeStartIndex, sanitizeEndIndex, executableIndex, authorizationIndex, + } { + require.GreaterOrEqual(t, index, 0) + } + assert.Less(t, identityIndex, sanitizeStartIndex) + assert.Less(t, sanitizeEndIndex, executableIndex) + assert.Less(t, executableIndex, authorizationIndex) + assert.Equal(t, map[string]string{ + "PATH": "/usr/sbin:/usr/bin:/sbin:/bin", + "HOME": privilegedHOME, + "LANG": privilegedLocale, + "LC_ALL": privilegedLocale, + }, environment.values) +} + +// Rationale: A valid sudo context and trusted executable cannot substitute for +// current membership in the mvm authorization group. +func TestRun_RejectsNonMemberBeforeActionDispatch(t *testing.T) { + authorization := memberAuthorizationDeps() + authorization.groupIDs = func(*user.User) ([]string, error) { return []string{"1000"}, nil } + + err := run(context.Background(), []string{Marker, "unknown"}, strings.NewReader("{}"), runtimeDeps{ + identity: identityDeps{ + effectiveUID: func() int { return 0 }, + lookupEnv: mapLookup(map[string]string{"SUDO_UID": "1000", "SUDO_GID": "1000"}), + }, + environment: newEnvironmentStore(nil).deps(), + executable: trustedExecutableDeps(), + authorization: authorization, + }) + require.Error(t, err) + assert.ErrorContains(t, err, "is not a current member of the "+infra.MVMUnixGroup+" group") + assert.NotContains(t, err.Error(), "unknown privileged action") +} + +func callIndex(calls []string, target string) int { + for index, call := range calls { + if call == target { + return index + } + } + return -1 +} diff --git a/internal/service/privileged/entry_test.go b/internal/service/privileged/entry_test.go new file mode 100644 index 00000000..deaac465 --- /dev/null +++ b/internal/service/privileged/entry_test.go @@ -0,0 +1,91 @@ +package privileged + +import ( + "context" + "strings" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "mvmctl/internal/infra" + "mvmctl/pkg/errs" +) + +func TestRun_FailsClosedBeforeDispatch(t *testing.T) { + tests := map[string]struct { + args []string + effectiveUID int + environment map[string]string + wantCode errs.Code + wantErr string + }{ + "unsupported_protocol": { + args: []string{markerPrefix + "2", "vm-cleanup"}, + wantCode: errs.CodeValidationFailed, + wantErr: "unsupported privileged protocol", + }, + "extra_argv_cannot_reach_public_cli": { + args: []string{Marker, "vm-cleanup", "host", "reset"}, + wantCode: errs.CodeValidationFailed, + wantErr: "requires exactly one action", + }, + "rootless_unknown_action": { + args: []string{Marker, "run-command"}, + effectiveUID: 1000, + wantCode: errs.CodePrivilegeRequired, + wantErr: "requires effective UID 0", + }, + "direct_root_unknown_action": { + args: []string{Marker, "run-command"}, + wantCode: errs.CodePrivilegeRequired, + wantErr: "missing SUDO_UID", + }, + "sudo_unknown_action": { + args: []string{Marker, "run-command"}, + environment: map[string]string{"SUDO_UID": "1000", "SUDO_GID": "1000"}, + wantCode: errs.CodeValidationFailed, + wantErr: `unknown privileged action "run-command"`, + }, + } + + for name, tc := range tests { + t.Run(name, func(t *testing.T) { + err := run( + context.Background(), + tc.args, + strings.NewReader(`{"command":"id"}`), + testRuntimeDeps(tc.effectiveUID, tc.environment), + ) + require.Error(t, err) + assert.ErrorContains(t, err, tc.wantErr) + assert.Equal(t, tc.wantCode, errs.AsDomainError(err).Code) + }) + } +} + +// Rationale: Caller identity alone does not establish code integrity; a root +// dispatcher reached through a user-replaceable executable remains root code execution. +func TestRun_VerifiesSystemExecutableBeforeActionDispatch(t *testing.T) { + deps := trustedExecutableDeps() + deps.executable = func() (string, error) { return "/tmp/user-owned-mvm", nil } + runtime := testRuntimeDeps(0, map[string]string{"SUDO_UID": "1000", "SUDO_GID": "1000"}) + runtime.executable = deps + + err := run(context.Background(), []string{Marker, "run-command"}, strings.NewReader("{}"), runtime) + require.Error(t, err) + assert.ErrorContains(t, err, "must run from "+infra.SystemBinaryPath) + assert.NotContains(t, err.Error(), "unknown privileged action") +} + +func testRuntimeDeps(effectiveUID int, identityEnvironment map[string]string) runtimeDeps { + return runtimeDeps{ + identity: identityDeps{ + effectiveUID: func() int { return effectiveUID }, + lookupEnv: mapLookup(identityEnvironment), + }, + environment: newEnvironmentStore(nil).deps(), + executable: trustedExecutableDeps(), + authorization: memberAuthorizationDeps(), + } +} diff --git a/internal/service/privileged/environment.go b/internal/service/privileged/environment.go new file mode 100644 index 00000000..ac88b81b --- /dev/null +++ b/internal/service/privileged/environment.go @@ -0,0 +1,48 @@ +package privileged + +import ( + "fmt" + "os" + + "mvmctl/pkg/errs" +) + +const ( + privilegedPATH = "/usr/sbin:/usr/bin:/sbin:/bin" + privilegedHOME = "/root" + privilegedLocale = "C" +) + +type environmentDeps struct { + clear func() + set func(string, string) error +} + +func sanitizeEnvironment(deps environmentDeps) error { + deps.clear() + fixedValues := [...]struct { + key string + value string + }{ + {key: "PATH", value: privilegedPATH}, + {key: "HOME", value: privilegedHOME}, + {key: "LANG", value: privilegedLocale}, + {key: "LC_ALL", value: privilegedLocale}, + } + for _, entry := range fixedValues { + if err := deps.set(entry.key, entry.value); err != nil { + return errs.New( + errs.CodeInternal, + fmt.Sprintf("set fixed privileged environment %s: %v", entry.key, err), + ) + } + } + return nil +} + +func realEnvironmentDeps() environmentDeps { + return environmentDeps{ + clear: os.Clearenv, + set: os.Setenv, + } +} diff --git a/internal/service/privileged/environment_test.go b/internal/service/privileged/environment_test.go new file mode 100644 index 00000000..182759bc --- /dev/null +++ b/internal/service/privileged/environment_test.go @@ -0,0 +1,85 @@ +package privileged + +import ( + "errors" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "mvmctl/pkg/errs" +) + +func TestSanitizeEnvironment_ReplacesInheritedEnvironmentWithFixedAllowlist(t *testing.T) { + store := newEnvironmentStore(map[string]string{ + "PATH": "/home/alice/bin:/tmp/tools", + "HOME": "/home/alice", + "MVM_CACHE_DIR": "/tmp/attacker-cache", + "LD_PRELOAD": "/tmp/inject.so", + "LD_LIBRARY_PATH": "/tmp/libraries", + "PYTHONPATH": "/tmp/python", + "GIT_CONFIG_GLOBAL": "/tmp/gitconfig", + "SYSTEMD_CONFIG_FILE": "/tmp/systemd", + "SUDO_UID": "1000", + "SUDO_GID": "1000", + "UNRELATED_CALLER_VALUE": "present", + }) + + err := sanitizeEnvironment(store.deps()) + require.NoError(t, err) + assert.Equal(t, map[string]string{ + "PATH": "/usr/sbin:/usr/bin:/sbin:/bin", + "HOME": privilegedHOME, + "LANG": privilegedLocale, + "LC_ALL": privilegedLocale, + }, store.values) + assert.Equal(t, 1, store.clearCalls) +} + +func TestSanitizeEnvironment_FailsClosedWhenFixedValueCannotBeSet(t *testing.T) { + store := newEnvironmentStore(map[string]string{"LD_PRELOAD": "/tmp/inject.so"}) + store.setErrorKey = "HOME" + + err := sanitizeEnvironment(store.deps()) + require.Error(t, err) + assert.ErrorContains(t, err, "set fixed privileged environment HOME") + assert.Equal(t, errs.CodeInternal, errs.AsDomainError(err).Code) + assert.NotContains(t, store.values, "LD_PRELOAD") +} + +type environmentStore struct { + values map[string]string + clearCalls int + setErrorKey string + calls *[]string +} + +func newEnvironmentStore(values map[string]string) *environmentStore { + copyValues := make(map[string]string, len(values)) + for key, value := range values { + copyValues[key] = value + } + return &environmentStore{values: copyValues} +} + +func (s *environmentStore) deps() environmentDeps { + return environmentDeps{ + clear: func() { + s.clearCalls++ + s.values = make(map[string]string) + if s.calls != nil { + *s.calls = append(*s.calls, "environment.clear") + } + }, + set: func(key, value string) error { + if s.calls != nil { + *s.calls = append(*s.calls, "environment.set."+key) + } + if key == s.setErrorKey { + return errors.New("set rejected") + } + s.values[key] = value + return nil + }, + } +} diff --git a/internal/service/privileged/executable.go b/internal/service/privileged/executable.go new file mode 100644 index 00000000..13f20e1c --- /dev/null +++ b/internal/service/privileged/executable.go @@ -0,0 +1,198 @@ +package privileged + +import ( + "errors" + "fmt" + "log/slog" + "os" + "strings" + + "golang.org/x/sys/unix" + + "mvmctl/internal/infra" + "mvmctl/pkg/errs" +) + +type executableDeps struct { + executable func() (string, error) + open func(string, int, uint32) (int, error) + openAt func(int, string, int, uint32) (int, error) + fstat func(int, *unix.Stat_t) error + close func(int) error +} + +type verifiedExecutable struct { + deps executableDeps + fds []int +} + +func (v *verifiedExecutable) close() error { + var result error + for index := len(v.fds) - 1; index >= 0; index-- { + if err := v.deps.close(v.fds[index]); err != nil { + result = errors.Join(result, err) + } + } + v.fds = nil + return result +} + +func verifySystemExecutable(deps executableDeps) (*verifiedExecutable, error) { + executable, err := deps.executable() + if err != nil { + return nil, errs.New( + errs.CodePrivilegeRequired, + "resolve running executable: "+err.Error(), + ) + } + if executable != infra.SystemBinaryPath { + return nil, errs.New( + errs.CodePrivilegeRequired, + fmt.Sprintf("privileged protocol must run from %s", infra.SystemBinaryPath), + ) + } + + pinned := &verifiedExecutable{deps: deps} + verified := false + defer func() { + if !verified { + if err := pinned.close(); err != nil { + slog.Warn("close rejected privileged executable descriptors", "error", err) + } + } + }() + + directoryFlags := unix.O_PATH | unix.O_DIRECTORY | unix.O_CLOEXEC | unix.O_NOFOLLOW + rootFD, err := deps.open("/", directoryFlags, 0) + if err != nil { + return nil, errs.New(errs.CodePrivilegeRequired, "open trusted executable root: "+err.Error()) + } + pinned.fds = append(pinned.fds, rootFD) + if err := verifyTrustedDirectoryDescriptor(deps, rootFD, "/"); err != nil { + return nil, err + } + + parts := strings.Split(strings.TrimPrefix(infra.SystemBinaryPath, "/"), "/") + if len(parts) < 2 { + return nil, errs.New(errs.CodeInternal, "canonical system executable path is invalid") + } + parentFD := rootFD + for _, component := range parts[:len(parts)-1] { + fd, err := deps.openAt(parentFD, component, directoryFlags, 0) + if err != nil { + return nil, errs.New( + errs.CodePrivilegeRequired, + fmt.Sprintf("open trusted executable path component %s: %v", component, err), + ) + } + pinned.fds = append(pinned.fds, fd) + if err := verifyTrustedDirectoryDescriptor(deps, fd, component); err != nil { + return nil, err + } + parentFD = fd + } + + executableFlags := unix.O_PATH | unix.O_CLOEXEC | unix.O_NOFOLLOW + canonicalFD, err := deps.openAt(parentFD, parts[len(parts)-1], executableFlags, 0) + if err != nil { + return nil, errs.New( + errs.CodePrivilegeRequired, + "open canonical system executable: "+err.Error(), + ) + } + pinned.fds = append(pinned.fds, canonicalFD) + canonicalStat, err := verifyCanonicalExecutableDescriptor(deps, canonicalFD) + if err != nil { + return nil, err + } + + runningFD, err := deps.open("/proc/self/exe", unix.O_PATH|unix.O_CLOEXEC, 0) + if err != nil { + return nil, errs.New(errs.CodePrivilegeRequired, "open running process image: "+err.Error()) + } + pinned.fds = append(pinned.fds, runningFD) + var runningStat unix.Stat_t + if err := deps.fstat(runningFD, &runningStat); err != nil { + return nil, errs.New( + errs.CodePrivilegeRequired, + "inspect running process image descriptor: "+err.Error(), + ) + } + if canonicalStat.Dev != runningStat.Dev || canonicalStat.Ino != runningStat.Ino { + return nil, errs.New( + errs.CodePrivilegeRequired, + "canonical system executable does not match the running process image", + ) + } + + verified = true + return pinned, nil +} + +func verifyTrustedDirectoryDescriptor(deps executableDeps, fd int, component string) error { + var stat unix.Stat_t + if err := deps.fstat(fd, &stat); err != nil { + return errs.New( + errs.CodePrivilegeRequired, + fmt.Sprintf("inspect trusted executable descriptor %s: %v", component, err), + ) + } + if stat.Mode&unix.S_IFMT != unix.S_IFDIR { + return errs.New( + errs.CodePrivilegeRequired, + fmt.Sprintf("trusted executable path %s must be a directory", component), + ) + } + if stat.Uid != 0 { + return errs.New( + errs.CodePrivilegeRequired, + fmt.Sprintf("trusted executable path %s must be owned by root", component), + ) + } + if stat.Mode&0022 != 0 { + return errs.New( + errs.CodePrivilegeRequired, + fmt.Sprintf("trusted executable path %s must not be group/world writable", component), + ) + } + return nil +} + +func verifyCanonicalExecutableDescriptor(deps executableDeps, fd int) (unix.Stat_t, error) { + var stat unix.Stat_t + if err := deps.fstat(fd, &stat); err != nil { + return unix.Stat_t{}, errs.New( + errs.CodePrivilegeRequired, + "inspect canonical system executable descriptor: "+err.Error(), + ) + } + if stat.Mode&unix.S_IFMT != unix.S_IFREG { + return unix.Stat_t{}, errs.New( + errs.CodePrivilegeRequired, + fmt.Sprintf("system executable %s must be a regular file", infra.SystemBinaryPath), + ) + } + if stat.Uid != 0 || stat.Gid != 0 { + return unix.Stat_t{}, errs.New( + errs.CodePrivilegeRequired, + fmt.Sprintf("system executable %s must be owned by root:root", infra.SystemBinaryPath), + ) + } + if stat.Mode&07777 != infra.ExecutablePerm { + return unix.Stat_t{}, errs.New( + errs.CodePrivilegeRequired, + fmt.Sprintf("system executable %s must have mode 0755", infra.SystemBinaryPath), + ) + } + return stat, nil +} + +func realExecutableDeps() executableDeps { + return executableDeps{ + executable: os.Executable, + open: unix.Open, + openAt: unix.Openat, + fstat: unix.Fstat, + close: unix.Close, + } +} diff --git a/internal/service/privileged/executable_test.go b/internal/service/privileged/executable_test.go new file mode 100644 index 00000000..697317c3 --- /dev/null +++ b/internal/service/privileged/executable_test.go @@ -0,0 +1,267 @@ +package privileged + +import ( + "errors" + "fmt" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + "golang.org/x/sys/unix" + + "mvmctl/internal/infra" + "mvmctl/pkg/errs" +) + +const ( + testRootFD = 10 + testUsrFD = 11 + testLocalFD = 12 + testBinFD = 13 + testCanonicalFD = 14 + testRunningFD = 15 +) + +func TestVerifySystemExecutable_AcceptsAndPinsCanonicalRootOwnedImage(t *testing.T) { + fixture := newExecutableFixture() + pinned, err := verifySystemExecutable(fixture.deps) + require.NoError(t, err) + require.NotNil(t, pinned) + assert.Empty(t, fixture.closedFDs) + + require.NoError(t, pinned.close()) + assert.ElementsMatch(t, []int{ + testRootFD, testUsrFD, testLocalFD, testBinFD, testCanonicalFD, testRunningFD, + }, fixture.closedFDs) +} + +// Rationale: Path components must be pinned by no-follow descriptor traversal; +// checking a pathname and reopening it would leave a symlink-substitution race. +func TestVerifySystemExecutable_UsesDescriptorRelativeNoFollowTraversal(t *testing.T) { + fixture := newExecutableFixture() + pinned, err := verifySystemExecutable(fixture.deps) + require.NoError(t, err) + defer func() { require.NoError(t, pinned.close()) }() + + require.Len(t, fixture.openAtCalls, 4) + for _, call := range fixture.openAtCalls[:3] { + assert.NotZero(t, call.flags&unix.O_PATH) + assert.NotZero(t, call.flags&unix.O_CLOEXEC) + assert.NotZero(t, call.flags&unix.O_NOFOLLOW) + assert.NotZero(t, call.flags&unix.O_DIRECTORY) + } + canonicalCall := fixture.openAtCalls[3] + assert.Equal(t, testBinFD, canonicalCall.dirFD) + assert.Equal(t, "mvm", canonicalCall.path) + assert.NotZero(t, canonicalCall.flags&unix.O_PATH) + assert.NotZero(t, canonicalCall.flags&unix.O_CLOEXEC) + assert.NotZero(t, canonicalCall.flags&unix.O_NOFOLLOW) + assert.Zero(t, canonicalCall.flags&unix.O_DIRECTORY) + + require.Len(t, fixture.openCalls, 2) + assert.Equal(t, "/", fixture.openCalls[0].path) + assert.NotZero(t, fixture.openCalls[0].flags&unix.O_NOFOLLOW) + assert.Equal(t, "/proc/self/exe", fixture.openCalls[1].path) + assert.Zero(t, fixture.openCalls[1].flags&unix.O_NOFOLLOW) +} + +// Rationale: A root-owned file appearing at the canonical pathname is not +// enough; it must be the same device and inode as the image executing as root. +func TestVerifySystemExecutable_RejectsRunningImageMismatch(t *testing.T) { + fixture := newExecutableFixture() + stat := fixture.stats[testRunningFD] + stat.Ino++ + fixture.stats[testRunningFD] = stat + + pinned, err := verifySystemExecutable(fixture.deps) + require.Error(t, err) + assert.Nil(t, pinned) + assert.ErrorContains(t, err, "does not match the running process image") + assert.Equal(t, errs.CodePrivilegeRequired, errs.AsDomainError(err).Code) + assert.ElementsMatch(t, []int{ + testRootFD, testUsrFD, testLocalFD, testBinFD, testCanonicalFD, testRunningFD, + }, fixture.closedFDs) +} + +// Rationale: O_NOFOLLOW must turn a path-component swap into a closed failure, +// even if an earlier metadata observation would have described a safe directory. +func TestVerifySystemExecutable_RejectsSymlinkSwapDuringTraversal(t *testing.T) { + fixture := newExecutableFixture() + original := fixture.deps.openAt + fixture.deps.openAt = func(dirFD int, path string, flags int, mode uint32) (int, error) { + if path == "local" { + return -1, unix.ELOOP + } + return original(dirFD, path, flags, mode) + } + + pinned, err := verifySystemExecutable(fixture.deps) + require.Error(t, err) + assert.Nil(t, pinned) + assert.ErrorContains(t, err, "open trusted executable path component local") + assert.ElementsMatch(t, []int{testRootFD, testUsrFD}, fixture.closedFDs) +} + +func TestVerifySystemExecutable_RejectsUnsafeDescriptorState(t *testing.T) { + tests := map[string]struct { + mutate func(*executableFixture) + wantErr string + }{ + "different_executable_path": { + mutate: func(fixture *executableFixture) { + fixture.deps.executable = func() (string, error) { return "/tmp/mvm", nil } + }, + wantErr: "must run from " + infra.SystemBinaryPath, + }, + "executable_lookup_error": { + mutate: func(fixture *executableFixture) { + fixture.deps.executable = func() (string, error) { return "", errors.New("proc unavailable") } + }, + wantErr: "resolve running executable", + }, + "non_root_owned_directory": { + mutate: replaceExecutableStat(testBinFD, unix.Stat_t{ + Mode: unix.S_IFDIR | 0755, Uid: 1000, Gid: 0, Dev: 1, Ino: 4, + }), + wantErr: "must be owned by root", + }, + "group_writable_directory": { + mutate: replaceExecutableStat(testBinFD, unix.Stat_t{ + Mode: unix.S_IFDIR | 0775, Uid: 0, Gid: 0, Dev: 1, Ino: 4, + }), + wantErr: "must not be group/world writable", + }, + "non_regular_executable": { + mutate: replaceExecutableStat(testCanonicalFD, unix.Stat_t{ + Mode: unix.S_IFIFO | 0755, Uid: 0, Gid: 0, Dev: 1, Ino: 5, + }), + wantErr: "must be a regular file", + }, + "wrong_executable_mode": { + mutate: replaceExecutableStat(testCanonicalFD, unix.Stat_t{ + Mode: unix.S_IFREG | 0777, Uid: 0, Gid: 0, Dev: 1, Ino: 5, + }), + wantErr: "must have mode 0755", + }, + "non_root_executable_group": { + mutate: replaceExecutableStat(testCanonicalFD, unix.Stat_t{ + Mode: unix.S_IFREG | 0755, Uid: 0, Gid: 1000, Dev: 1, Ino: 5, + }), + wantErr: "must be owned by root:root", + }, + "setuid_executable": { + mutate: replaceExecutableStat(testCanonicalFD, unix.Stat_t{ + Mode: unix.S_IFREG | unix.S_ISUID | 0755, Uid: 0, Gid: 0, Dev: 1, Ino: 5, + }), + wantErr: "must have mode 0755", + }, + "missing_component": { + mutate: func(fixture *executableFixture) { + fixture.deps.openAt = func(int, string, int, uint32) (int, error) { + return -1, unix.ENOENT + } + }, + wantErr: "open trusted executable path component", + }, + "fstat_failure": { + mutate: func(fixture *executableFixture) { + fixture.deps.fstat = func(int, *unix.Stat_t) error { return unix.EIO } + }, + wantErr: "inspect trusted executable descriptor", + }, + } + + for name, tc := range tests { + t.Run(name, func(t *testing.T) { + fixture := newExecutableFixture() + tc.mutate(fixture) + pinned, err := verifySystemExecutable(fixture.deps) + require.Error(t, err) + assert.Nil(t, pinned) + assert.ErrorContains(t, err, tc.wantErr) + assert.Equal(t, errs.CodePrivilegeRequired, errs.AsDomainError(err).Code) + }) + } +} + +func trustedExecutableDeps() executableDeps { + return newExecutableFixture().deps +} + +func replaceExecutableStat(fd int, replacement unix.Stat_t) func(*executableFixture) { + return func(fixture *executableFixture) { + fixture.stats[fd] = replacement + } +} + +type executableOpenCall struct { + dirFD int + path string + flags int +} + +type executableFixture struct { + deps executableDeps + stats map[int]unix.Stat_t + openCalls []executableOpenCall + openAtCalls []executableOpenCall + closedFDs []int +} + +func newExecutableFixture() *executableFixture { + fixture := &executableFixture{ + stats: map[int]unix.Stat_t{ + testRootFD: {Mode: unix.S_IFDIR | 0755, Uid: 0, Gid: 0, Dev: 1, Ino: 1}, + testUsrFD: {Mode: unix.S_IFDIR | 0755, Uid: 0, Gid: 0, Dev: 1, Ino: 2}, + testLocalFD: {Mode: unix.S_IFDIR | 0755, Uid: 0, Gid: 100, Dev: 1, Ino: 3}, + testBinFD: {Mode: unix.S_IFDIR | 0755, Uid: 0, Gid: 0, Dev: 1, Ino: 4}, + testCanonicalFD: {Mode: unix.S_IFREG | 0755, Uid: 0, Gid: 0, Dev: 1, Ino: 5}, + testRunningFD: {Mode: unix.S_IFREG | 0755, Uid: 0, Gid: 0, Dev: 1, Ino: 5}, + }, + } + fixture.deps = executableDeps{ + executable: func() (string, error) { return infra.SystemBinaryPath, nil }, + open: func(path string, flags int, _ uint32) (int, error) { + fixture.openCalls = append(fixture.openCalls, executableOpenCall{path: path, flags: flags}) + switch path { + case "/": + return testRootFD, nil + case "/proc/self/exe": + return testRunningFD, nil + default: + return -1, fmt.Errorf("unexpected open path %s", path) + } + }, + openAt: func(dirFD int, path string, flags int, _ uint32) (int, error) { + fixture.openAtCalls = append( + fixture.openAtCalls, + executableOpenCall{dirFD: dirFD, path: path, flags: flags}, + ) + switch { + case dirFD == testRootFD && path == "usr": + return testUsrFD, nil + case dirFD == testUsrFD && path == "local": + return testLocalFD, nil + case dirFD == testLocalFD && path == "bin": + return testBinFD, nil + case dirFD == testBinFD && path == "mvm": + return testCanonicalFD, nil + default: + return -1, fmt.Errorf("unexpected openat target %d/%s", dirFD, path) + } + }, + fstat: func(fd int, stat *unix.Stat_t) error { + value, ok := fixture.stats[fd] + if !ok { + return unix.EBADF + } + *stat = value + return nil + }, + close: func(fd int) error { + fixture.closedFDs = append(fixture.closedFDs, fd) + return nil + }, + } + return fixture +} diff --git a/internal/service/privileged/identity.go b/internal/service/privileged/identity.go new file mode 100644 index 00000000..6b52f68e --- /dev/null +++ b/internal/service/privileged/identity.go @@ -0,0 +1,68 @@ +package privileged + +import ( + "fmt" + "math" + "os" + "strconv" + + "mvmctl/pkg/errs" +) + +type callerIdentity struct { + uid uint32 + gid uint32 +} + +type identityDeps struct { + effectiveUID func() int + lookupEnv func(string) (string, bool) +} + +func invokingIdentity(deps identityDeps) (callerIdentity, error) { + if deps.effectiveUID() != 0 { + return callerIdentity{}, errs.New( + errs.CodePrivilegeRequired, + "privileged protocol requires effective UID 0", + ) + } + + uid, err := parseNonRootIdentity("SUDO_UID", deps.lookupEnv) + if err != nil { + return callerIdentity{}, err + } + gid, err := parseNonRootIdentity("SUDO_GID", deps.lookupEnv) + if err != nil { + return callerIdentity{}, err + } + return callerIdentity{uid: uid, gid: gid}, nil +} + +func parseNonRootIdentity(name string, lookupEnv func(string) (string, bool)) (uint32, error) { + value, ok := lookupEnv(name) + if !ok || value == "" { + return 0, errs.New(errs.CodePrivilegeRequired, fmt.Sprintf("missing %s in sudo execution context", name)) + } + parsed, err := strconv.ParseUint(value, 10, 32) + if err != nil || parsed == math.MaxUint32 { + return 0, errs.New(errs.CodePrivilegeRequired, fmt.Sprintf("invalid %s in sudo execution context", name)) + } + if parsed == 0 { + kind := "user" + if name == "SUDO_GID" { + kind = "group" + } + return 0, errs.New( + errs.CodePrivilegeRequired, + fmt.Sprintf("%s must identify a non-root %s", name, kind), + ) + } + return uint32(parsed), nil +} + +func realIdentityDeps() identityDeps { + return identityDeps{ + effectiveUID: os.Geteuid, + lookupEnv: os.LookupEnv, + } +} diff --git a/internal/service/privileged/identity_test.go b/internal/service/privileged/identity_test.go new file mode 100644 index 00000000..f2c429f6 --- /dev/null +++ b/internal/service/privileged/identity_test.go @@ -0,0 +1,80 @@ +package privileged + +import ( + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "mvmctl/pkg/errs" +) + +func TestInvokingIdentity_RequiresSudoRootContext(t *testing.T) { + tests := map[string]struct { + effectiveUID int + environment map[string]string + wantErr string + }{ + "rootless": { + effectiveUID: 1000, + environment: map[string]string{"SUDO_UID": "1000", "SUDO_GID": "1000"}, + wantErr: "requires effective UID 0", + }, + "direct_root_without_sudo": { + environment: map[string]string{}, + wantErr: "missing SUDO_UID", + }, + "root_invoker": { + environment: map[string]string{"SUDO_UID": "0", "SUDO_GID": "1000"}, + wantErr: "SUDO_UID must identify a non-root user", + }, + "root_group": { + environment: map[string]string{"SUDO_UID": "1000", "SUDO_GID": "0"}, + wantErr: "SUDO_GID must identify a non-root group", + }, + "negative_uid": { + environment: map[string]string{"SUDO_UID": "-1", "SUDO_GID": "1000"}, + wantErr: "invalid SUDO_UID", + }, + "non_numeric_gid": { + environment: map[string]string{"SUDO_UID": "1000", "SUDO_GID": "users"}, + wantErr: "invalid SUDO_GID", + }, + "overflow_uid": { + environment: map[string]string{"SUDO_UID": "4294967296", "SUDO_GID": "1000"}, + wantErr: "invalid SUDO_UID", + }, + } + + for name, tc := range tests { + t.Run(name, func(t *testing.T) { + _, err := invokingIdentity(identityDeps{ + effectiveUID: func() int { return tc.effectiveUID }, + lookupEnv: mapLookup(tc.environment), + }) + require.Error(t, err) + assert.ErrorContains(t, err, tc.wantErr) + assert.Equal(t, errs.CodePrivilegeRequired, errs.AsDomainError(err).Code) + }) + } +} + +func TestInvokingIdentity_AcceptsNonRootSudoCaller(t *testing.T) { + got, err := invokingIdentity(identityDeps{ + effectiveUID: func() int { return 0 }, + lookupEnv: mapLookup(map[string]string{ + "SUDO_UID": "1000", + "SUDO_GID": "1001", + }), + }) + require.NoError(t, err) + assert.Equal(t, uint32(1000), got.uid) + assert.Equal(t, uint32(1001), got.gid) +} + +func mapLookup(values map[string]string) func(string) (string, bool) { + return func(key string) (string, bool) { + value, ok := values[key] + return value, ok + } +} diff --git a/internal/service/privileged/protocol.go b/internal/service/privileged/protocol.go new file mode 100644 index 00000000..e608084f --- /dev/null +++ b/internal/service/privileged/protocol.go @@ -0,0 +1,211 @@ +package privileged + +import ( + "bytes" + "encoding/json" + "fmt" + "io" + "strings" + + "mvmctl/internal/infra" + "mvmctl/pkg/errs" +) + +const ( + markerPrefix = "__mvm_privileged_v" + protocolVersion = "1" + maxActionBytes = 64 + maxRequestBytes = 64 * 1024 + maxJSONDepth = 32 + + maxJSONArrayItems = 1024 + maxJSONObjectFields = 64 +) + +// Marker selects the current internal privileged protocol before normal CLI initialization. +const Marker = markerPrefix + protocolVersion + +type invocation struct { + action string +} + +// IsInvocation reports whether args select the reserved privileged protocol namespace. +// Unsupported versions stay in this namespace so they cannot fall through to the public CLI. +func IsInvocation(args []string) bool { + return len(args) > 0 && strings.HasPrefix(args[0], markerPrefix) +} + +func parseInvocation(args []string) (invocation, error) { + if len(args) == 0 || args[0] != Marker { + return invocation{}, errs.New( + errs.CodeValidationFailed, + fmt.Sprintf("unsupported privileged protocol; ask an administrator to upgrade %s", infra.SystemBinaryPath), + ) + } + if len(args) != 2 { + return invocation{}, errs.New( + errs.CodeValidationFailed, + "privileged protocol requires exactly one action", + ) + } + if args[1] == "" { + return invocation{}, errs.New(errs.CodeValidationFailed, "privileged action is required") + } + if len(args[1]) > maxActionBytes { + return invocation{}, errs.New( + errs.CodeValidationFailed, + fmt.Sprintf("privileged action exceeds %d bytes", maxActionBytes), + ) + } + return invocation{action: args[1]}, nil +} + +func decodeRequest[T any](input io.Reader) (T, error) { + var zero T + if input == nil { + return zero, errs.New(errs.CodeValidationFailed, "privileged request body is required") + } + + raw, err := io.ReadAll(io.LimitReader(input, maxRequestBytes+1)) + if err != nil { + return zero, errs.WrapMsg(errs.CodeValidationFailed, "read privileged request body", err) + } + if len(raw) == 0 { + return zero, errs.New(errs.CodeValidationFailed, "privileged request body is required") + } + if len(raw) > maxRequestBytes { + return zero, errs.New( + errs.CodeValidationFailed, + fmt.Sprintf("privileged request body exceeds %d bytes", maxRequestBytes), + ) + } + if err := rejectDuplicateFields(raw); err != nil { + return zero, err + } + + decoder := json.NewDecoder(bytes.NewReader(raw)) + decoder.DisallowUnknownFields() + var request T + if err := decoder.Decode(&request); err != nil { + return zero, errs.WrapMsg( + errs.CodeValidationFailed, + "decode privileged request: "+err.Error(), + err, + ) + } + if err := requireJSONEOF(decoder); err != nil { + return zero, err + } + return request, nil +} + +func rejectDuplicateFields(raw []byte) error { + decoder := json.NewDecoder(bytes.NewReader(raw)) + if err := scanJSONValue(decoder, 0); err != nil { + if errs.AsDomainError(err) != nil { + return err + } + return errs.WrapMsg( + errs.CodeValidationFailed, + "decode privileged request: "+err.Error(), + err, + ) + } + return requireJSONEOF(decoder) +} + +func scanJSONValue(decoder *json.Decoder, depth int) error { + token, err := decoder.Token() + if err != nil { + return err + } + delim, ok := token.(json.Delim) + if !ok { + return nil + } + if depth >= maxJSONDepth { + return errs.New( + errs.CodeValidationFailed, + fmt.Sprintf("privileged request JSON nesting exceeds %d levels", maxJSONDepth), + ) + } + + switch delim { + case '{': + seen := make(map[string]struct{}) + fieldCount := 0 + for decoder.More() { + fieldCount++ + if fieldCount > maxJSONObjectFields { + return errs.New( + errs.CodeValidationFailed, + fmt.Sprintf("privileged request JSON object exceeds %d fields", maxJSONObjectFields), + ) + } + fieldToken, err := decoder.Token() + if err != nil { + return err + } + field, ok := fieldToken.(string) + if !ok { + return fmt.Errorf("object field is not a string") + } + if _, exists := seen[field]; exists { + return errs.New( + errs.CodeValidationFailed, + fmt.Sprintf("duplicate JSON field %q in privileged request", field), + ) + } + for existing := range seen { + if strings.EqualFold(existing, field) { + return errs.New( + errs.CodeValidationFailed, + fmt.Sprintf( + "case-folded duplicate JSON field %q conflicts with %q in privileged request", + field, + existing, + ), + ) + } + } + seen[field] = struct{}{} + if err := scanJSONValue(decoder, depth+1); err != nil { + return err + } + } + _, err = decoder.Token() + return err + case '[': + itemCount := 0 + for decoder.More() { + itemCount++ + if itemCount > maxJSONArrayItems { + return errs.New( + errs.CodeValidationFailed, + fmt.Sprintf("privileged request JSON array exceeds %d items", maxJSONArrayItems), + ) + } + if err := scanJSONValue(decoder, depth+1); err != nil { + return err + } + } + _, err = decoder.Token() + return err + default: + return fmt.Errorf("unexpected JSON delimiter %q", delim) + } +} + +func requireJSONEOF(decoder *json.Decoder) error { + if _, err := decoder.Token(); err != io.EOF { + if err == nil { + return errs.New(errs.CodeValidationFailed, "unexpected trailing JSON in privileged request") + } + return errs.WrapMsg( + errs.CodeValidationFailed, + "decode privileged request: "+err.Error(), + err, + ) + } + return nil +} diff --git a/internal/service/privileged/protocol_test.go b/internal/service/privileged/protocol_test.go new file mode 100644 index 00000000..90285774 --- /dev/null +++ b/internal/service/privileged/protocol_test.go @@ -0,0 +1,166 @@ +package privileged + +import ( + "fmt" + "strings" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "mvmctl/pkg/errs" +) + +func TestIsInvocation_ReservesEveryProtocolMarker(t *testing.T) { + tests := map[string]struct { + args []string + want bool + }{ + "current_marker": {args: []string{Marker, "vm-cleanup"}, want: true}, + "unsupported_version": {args: []string{markerPrefix + "2", "vm-cleanup"}, want: true}, + "missing_arguments": {args: []string{Marker}, want: true}, + "ordinary_cli": {args: []string{"vm", "list"}}, + "similar_public_name": {args: []string{"mvm-privileged-v1"}}, + "empty": {}, + } + + for name, tc := range tests { + t.Run(name, func(t *testing.T) { + assert.Equal(t, tc.want, IsInvocation(tc.args)) + }) + } +} + +func TestParseInvocation_FailsClosed(t *testing.T) { + tests := map[string]struct { + args []string + wantErr string + }{ + "unsupported_version": { + args: []string{markerPrefix + "2", "vm-cleanup"}, + wantErr: "upgrade /usr/local/bin/mvm", + }, + "missing_action": { + args: []string{Marker}, + wantErr: "requires exactly one action", + }, + "empty_action": { + args: []string{Marker, ""}, + wantErr: "action is required", + }, + "extra_argument": { + args: []string{Marker, "vm-cleanup", "public-command"}, + wantErr: "requires exactly one action", + }, + "oversized_action": { + args: []string{Marker, strings.Repeat("a", maxActionBytes+1)}, + wantErr: "action exceeds", + }, + } + + for name, tc := range tests { + t.Run(name, func(t *testing.T) { + _, err := parseInvocation(tc.args) + require.Error(t, err) + assert.ErrorContains(t, err, tc.wantErr) + assert.Equal(t, errs.CodeValidationFailed, errs.AsDomainError(err).Code) + }) + } +} + +func TestDecodeRequest_RejectsUntrustedJSON(t *testing.T) { + type nested struct { + Label string `json:"label"` + } + type request struct { + VMID string `json:"vm_id"` + Nested nested `json:"nested"` + } + + tests := map[string]struct { + body string + wantErr string + }{ + "empty": { + wantErr: "request body is required", + }, + "malformed": { + body: `{"vm_id":`, + wantErr: "decode privileged request", + }, + "unknown_field": { + body: `{"vm_id":"abc","nested":{"label":"ok"},"command":"id"}`, + wantErr: "unknown field", + }, + "duplicate_top_level_field": { + body: `{"vm_id":"first","vm_id":"second","nested":{"label":"ok"}}`, + wantErr: `duplicate JSON field "vm_id"`, + }, + "duplicate_nested_field": { + body: `{"vm_id":"abc","nested":{"label":"first","label":"second"}}`, + wantErr: `duplicate JSON field "label"`, + }, + "case_folded_duplicate_top_level_field": { + body: `{"vm_id":"first","VM_ID":"second","nested":{"label":"ok"}}`, + wantErr: `case-folded duplicate JSON field "VM_ID" conflicts with "vm_id"`, + }, + "case_folded_duplicate_nested_field": { + body: `{"vm_id":"abc","nested":{"label":"first","LABEL":"second"}}`, + wantErr: `case-folded duplicate JSON field "LABEL" conflicts with "label"`, + }, + "trailing_json": { + body: `{"vm_id":"abc","nested":{"label":"ok"}} {}`, + wantErr: "unexpected trailing JSON", + }, + "oversized": { + body: strings.Repeat(" ", maxRequestBytes+1), + wantErr: "request body exceeds", + }, + "excessive_nesting": { + body: strings.Repeat("[", maxJSONDepth+1) + "0" + strings.Repeat("]", maxJSONDepth+1), + wantErr: "JSON nesting exceeds", + }, + "excessive_array_items": { + body: "[" + strings.Repeat("0,", maxJSONArrayItems) + "0]", + wantErr: "JSON array exceeds", + }, + "excessive_object_fields": { + body: objectWithFields(maxJSONObjectFields + 1), + wantErr: "JSON object exceeds", + }, + } + + for name, tc := range tests { + t.Run(name, func(t *testing.T) { + _, err := decodeRequest[request](strings.NewReader(tc.body)) + require.Error(t, err) + assert.ErrorContains(t, err, tc.wantErr) + assert.Equal(t, errs.CodeValidationFailed, errs.AsDomainError(err).Code) + }) + } +} + +func TestDecodeRequest_AcceptsOneExactObject(t *testing.T) { + type request struct { + VMID string `json:"vm_id"` + } + + got, err := decodeRequest[request](strings.NewReader("{\"vm_id\":\"abc\"}\n")) + require.NoError(t, err) + assert.Equal(t, "abc", got.VMID) +} + +func objectWithFields(count int) string { + var body strings.Builder + body.WriteByte('{') + for index := range count { + if index > 0 { + body.WriteByte(',') + } + body.WriteString(`"field`) + body.WriteString(fmt.Sprintf("%d", index)) + body.WriteString(`":0`) + } + body.WriteByte('}') + return body.String() +} From 5b4be64c51b176e2edb80189ad2b840ef856e3e1 Mon Sep 17 00:00:00 2001 From: AlanD20 Date: Thu, 20 Aug 2026 10:21:40 +0200 Subject: [PATCH 0009/1457] feat: install trusted system binary safely --- CONTEXT.md | 32 +- README.md | 20 +- cmd/mvm/main.go | 18 +- cmd/mvm/main_test.go | 55 ++ docs/DEPENDENCIES.md | 1 + docs/REFERENCES.md | 9 +- docs/RELEASE.md | 5 +- docs/RUNTIME.md | 12 +- .../adr/0002-single-go-binary-architecture.md | 35 +- docs/adr/0005-sudo-privilege-architecture.md | 35 +- ...owned-single-binary-privileged-dispatch.md | 27 +- docs/development/HOW_TO_RUN_SYSTEM_TESTS.md | 22 +- docs/implementations/SELF_UPDATE.md | 26 +- internal/app/system_install.go | 147 +++ internal/app/system_install_test.go | 200 +++++ internal/cli/host.go | 45 + internal/cli/host_internal_test.go | 48 + internal/cli/host_test.go | 32 + internal/core/host/sudoers_config.go | 59 ++ internal/core/host/sudoers_config_test.go | 205 +++++ internal/core/host/system_binary.go | 629 +++++++++++++ internal/core/host/system_binary_test.go | 840 ++++++++++++++++++ internal/core/host/system_binary_verify.go | 75 ++ internal/core/host/utils.go | 7 +- internal/core/host/utils_test.go | 15 + internal/core/update/service.go | 34 +- internal/core/update/system_binary_test.go | 41 + internal/infra/constants.go | 3 + internal/service/privileged/protocol.go | 3 +- internal/testutil/mock_host_api.go | 8 + pkg/api/host.go | 68 +- pkg/api/host_system_binary_test.go | 82 ++ pkg/api/operation.go | 21 +- tasks/plan.md | 8 +- tasks/todo.md | 10 +- 35 files changed, 2738 insertions(+), 139 deletions(-) create mode 100644 internal/app/system_install.go create mode 100644 internal/app/system_install_test.go create mode 100644 internal/cli/host_internal_test.go create mode 100644 internal/core/host/sudoers_config.go create mode 100644 internal/core/host/sudoers_config_test.go create mode 100644 internal/core/host/system_binary.go create mode 100644 internal/core/host/system_binary_test.go create mode 100644 internal/core/host/system_binary_verify.go create mode 100644 internal/core/update/system_binary_test.go create mode 100644 pkg/api/host_system_binary_test.go diff --git a/CONTEXT.md b/CONTEXT.md index a7e13cca..b28fa657 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -217,19 +217,35 @@ Architecture rules are enforced by the Go compiler (circular import errors preve The `pkg/api` package IS the stable, curated public interface for all consumers -- CLI, future TUI/GUI, and external scripts. The `api.Operation` struct exposes all domain operations as methods. The `pkg/api/inputs` package exposes all Input types. The `internal/core` package is an implementation detail. -### Build output (REQUIRED) +### Build and system-install contract (REQUIRED) -**The `mvm` binary MUST be built to `~/.local/bin/mvm`.** This path has passwordless sudo privileges via the mvmctl sudoers rules, so subcommands requiring privilege escalation run without password prompts. +The release script produces the candidate artifact at `dist/mvm`. A development artifact may live in the worktree or +under `~/.local/bin`, but a user-owned path is **never** a sudo target. + +The canonical host installation is `/usr/local/bin/mvm`, owned by `root:root` with mode `0755` under a root-owned, +non-writable directory chain. Install or upgrade it through the candidate artifact's exact early bootstrap route: -For **release testing / RC QA / system tests**, always use the release build script: ```bash -./scripts/build.sh release # produces dist/mvm -cp dist/mvm ~/.local/bin/mvm # copy for sudo operations +./scripts/build.sh release +sudo ./dist/mvm host install-system +sudo /usr/local/bin/mvm host init ``` -A bare `go build -o ~/.local/bin/mvm ./cmd/mvm` works for dev but produces a -binary without version info, symbol stripping, or PIE. **Never use it for -release qualification.** +`host install-system` executes before ordinary application initialization. It does not read `MVM_CACHE_DIR`, user +configuration, the user database, Cobra, or plugins, and it replaces its environment with fixed root-safe values before +installation. If an old managed sudoers file still authorizes a user-owned binary, remove that legacy file from an +authenticated administrator session before running the bootstrap command; the installer refuses unrecognized active +syntax rather than guessing how sudo authenticated the process. + +Normal users run `/usr/local/bin/mvm` without sudo. The `mvm` group is the authorization role for the final constrained +passwordless privileged protocol; it never makes a development artifact trusted. During Tasks 2-8, the generated policy +still contains transitional raw-tool grants and `/usr/local/bin/mvm *` for callers that have not migrated. That policy is +not the final security boundary and must not ship as marker-only until Task 9 removes those grants. A system-installed +executable cannot replace itself through `mvm self-update`; an administrator installs the newly downloaded trusted +artifact with `host install-system`. + +For **release testing / RC QA / system tests**, always use `./scripts/build.sh release`; a bare `go build` lacks the +release version metadata, symbol stripping, and PIE settings and must not be used for release qualification. ### Asset mirror environment variable (OPTIONAL) diff --git a/README.md b/README.md index 87e2c098..0aa2dacd 100644 --- a/README.md +++ b/README.md @@ -58,7 +58,7 @@ Think "Docker for VMs" — one command to create, start, and connect. sudo pacman -S --needed iproute2 iptables nftables qemu-img e2fsprogs util-linux procps-ng kmod openssh tar sudo shadow fakeroot ``` Optional (for ISO cloud-init mode only): `cloud-image-utils` (Ubuntu/Debian) or `cloud-utils` (Arch) -- **Root access (one-time):** run `mvm host init` once to create the `mvm` group and a sudoers drop-in; normal `mvm` commands require no `sudo` after that +- **Root access (one-time):** install the trusted artifact at `/usr/local/bin/mvm`, then run `host init`; normal commands run without `sudo` after setup - **Environment variables:** Configure runtime behavior via `MVM_*` variables. See [docs/REFERENCES.md](docs/REFERENCES.md#environment-variables) for the full list. --- @@ -70,21 +70,25 @@ Think "Docker for VMs" — one command to create, start, and connect. You may head over to [Releases](https://github.com/AlanD20/mvmctl/releases) page to get the desired package for your distro or you may simply get the static and standalone binary itself: ```bash -mkdir -p ~/.local/bin -curl -L -o ~/.local/bin/mvm https://github.com/AlanD20/mvmctl/releases/latest/download/mvm -chmod +x ~/.local/bin/mvm -mvm --help +curl -L -o ./mvm https://github.com/AlanD20/mvmctl/releases/latest/download/mvm +chmod +x ./mvm +sudo ./mvm host install-system +sudo /usr/local/bin/mvm host init +/usr/local/bin/mvm --help ``` -> Make sure `~/.local/bin` is in your `$PATH`. Most modern Linux distros include it by default. +> `/usr/local/bin/mvm` is the single installed executable. It is root-owned; normal use is unprivileged. Keep a +> downloaded candidate user-owned and use its `host install-system` command for future administrator-approved upgrades. ### 2. Build from source ```bash git clone https://github.com/AlanD20/mvmctl cd mvmctl -./scripts/build.sh release --output ~/.local/bin/mvm -mvm --help +./scripts/build.sh release +sudo ./dist/mvm host install-system +sudo /usr/local/bin/mvm host init +/usr/local/bin/mvm --help ``` See [docs/REFERENCES.md](docs/REFERENCES.md) for the complete command reference with all flags, options, and selectors. diff --git a/cmd/mvm/main.go b/cmd/mvm/main.go index 158cc45d..b2feb430 100644 --- a/cmd/mvm/main.go +++ b/cmd/mvm/main.go @@ -6,18 +6,21 @@ import ( "log/slog" "os" + "mvmctl/internal/app" "mvmctl/internal/service/privileged" ) type entrypoints struct { - privileged func(context.Context, []string, io.Reader) error - normal func() int + privileged func(context.Context, []string, io.Reader) error + systemInstall func(context.Context, []string) error + normal func() int } func main() { os.Exit(run(context.Background(), os.Args[1:], os.Stdin, entrypoints{ - privileged: privileged.Run, - normal: runCLI, + privileged: privileged.Run, + systemInstall: app.RunHostInstallSystemBinary, + normal: runCLI, })) } @@ -29,5 +32,12 @@ func run(ctx context.Context, args []string, input io.Reader, entries entrypoint } return 0 } + if app.IsHostInstallSystemBinaryInvocation(args) { + if err := entries.systemInstall(ctx, args); err != nil { + slog.Error("system binary installation failed", "error", err) + return 1 + } + return 0 + } return entries.normal() } diff --git a/cmd/mvm/main_test.go b/cmd/mvm/main_test.go index 51856223..f250fa0a 100644 --- a/cmd/mvm/main_test.go +++ b/cmd/mvm/main_test.go @@ -74,3 +74,58 @@ func TestRun_RoutesOrdinaryArgumentsToNormalCLI(t *testing.T) { assert.Zero(t, privilegedCalls) assert.Equal(t, 1, normalCalls) } + +func TestRun_RoutesSystemInstallBeforeNormalInitialization(t *testing.T) { + sentinel := errors.New("system install sentinel") + installCalls := 0 + normalCalls := 0 + + exitCode := run(context.Background(), []string{"host", "install-system"}, nil, entrypoints{ + systemInstall: func(_ context.Context, args []string) error { + installCalls++ + assert.Equal(t, []string{"host", "install-system"}, args) + return sentinel + }, + normal: func() int { + normalCalls++ + return 0 + }, + }) + + assert.Equal(t, 1, exitCode) + assert.Equal(t, 1, installCalls) + assert.Zero(t, normalCalls, "app.Initialize and Cobra must be unreachable") +} + +func TestRun_RoutesMalformedSystemInstallWithoutNormalFallthrough(t *testing.T) { + tests := [][]string{ + {"host", "install-system", "unexpected"}, + {"--debug", "host", "install-system", "--force"}, + {"--debug", "false", "host", "install-system"}, + {"host", "--unknown", "value", "install-system"}, + {"host", "--", "install-system"}, + {"--debug", "false", "false", "host", "install-system"}, + } + + for _, args := range tests { + t.Run(strings.Join(args, "_"), func(t *testing.T) { + installCalls := 0 + normalCalls := 0 + exitCode := run(context.Background(), args, nil, entrypoints{ + systemInstall: func(_ context.Context, got []string) error { + installCalls++ + assert.Equal(t, args, got) + return errors.New("malformed") + }, + normal: func() int { + normalCalls++ + return 0 + }, + }) + + assert.Equal(t, 1, exitCode) + assert.Equal(t, 1, installCalls) + assert.Zero(t, normalCalls) + }) + } +} diff --git a/docs/DEPENDENCIES.md b/docs/DEPENDENCIES.md index 97b56f74..a33f503a 100644 --- a/docs/DEPENDENCIES.md +++ b/docs/DEPENDENCIES.md @@ -184,6 +184,7 @@ Only needed for `mvm kernel pull --type official --clean-build` (building the of | Command | External Binaries Invoked | |---------|--------------------------| +| `mvm host install-system` | No external command; exact early root bootstrap uses descriptor-relative filesystem operations | | `mvm host init` | `sudo`, `groupadd`, `usermod`, `visudo`, `sysctl`, `lsmod`, `iptables`/`nft`, `modprobe` | | `mvm host clean` | `sudo`, `ip`, `iptables`/`nft` | | `mvm host reset` | `sudo`, `groupdel`, `sysctl` | diff --git a/docs/REFERENCES.md b/docs/REFERENCES.md index 3631a87b..dc09b32c 100644 --- a/docs/REFERENCES.md +++ b/docs/REFERENCES.md @@ -103,6 +103,7 @@ Host configuration commands for one-time, machine-global setup. | Command | Flags | Description | |---------|-------|-------------| +| `mvm host install-system` | — | Administrator-only early bootstrap. Atomically installs the running trusted artifact as root-owned `/usr/local/bin/mvm` without loading user cache, configuration, database, or Cobra state. | | `mvm host init` | — | Apply host configuration changes. Idempotent. Creates the `mvm` group, sudoers drop-in, enables IP forwarding, creates firewall chains, and snapshots initial state for rollback. | | `mvm host status` | `--json` | Show current host configuration state vs expected | | `mvm host info` | `--refresh`, `--json` | Show host hardware, limits, and VM capacity projection | @@ -624,11 +625,13 @@ Check for and apply binary updates from GitHub Releases. | Command | Description | |---------|-------------| -| `mvm self-update` | Check + apply if newer | +| `mvm self-update` | Check + apply if newer for a user-owned artifact; the system installation requires administrator bootstrap | | `mvm self-update check` | Check only, print available version | -| `mvm self-update apply` | Force apply (even if same version). `--force` to re-install | +| `mvm self-update apply` | Apply to a user-owned artifact. `--force` re-installs the same version | -Downloads the latest release matching the host architecture, verifies SHA256 checksum, then atomically swaps the binary. +For a user-owned artifact, downloads the latest release matching the host architecture, verifies its SHA256 checksum, +and atomically swaps that artifact. Apply refuses canonical `/usr/local/bin/mvm`; install a newly downloaded trusted +artifact with `sudo host install-system`. `check` remains available from the system installation. --- diff --git a/docs/RELEASE.md b/docs/RELEASE.md index f346bd81..2e65672d 100644 --- a/docs/RELEASE.md +++ b/docs/RELEASE.md @@ -43,9 +43,10 @@ go test ./... # Build release binary for system tests ./scripts/build.sh release -cp dist/mvm ~/.local/bin/mvm +sudo ./dist/mvm host install-system +sudo /usr/local/bin/mvm host init -python3 scripts/run-system-tests.py --all +MVM_BINARY=/usr/local/bin/mvm python3 scripts/run-system-tests.py --all ``` All five must pass. If system tests fail, investigate before proceeding. diff --git a/docs/RUNTIME.md b/docs/RUNTIME.md index 74823ff1..c07992c5 100644 --- a/docs/RUNTIME.md +++ b/docs/RUNTIME.md @@ -530,10 +530,13 @@ SSH-based probe. All images use the loop-mount provisioning backend (the default The `mvm` binary is a standard Go binary that includes all services compiled in: ```bash -go build -o dist/mvm ./cmd/mvm +./scripts/build.sh release +sudo ./dist/mvm host install-system ``` -The three `mvm run` services (console relay, nocloud-net server, loopmount provisioner) plus the vsock guest agent binary are all compiled into the same binary. No separate service binaries, no symlinks, no extraction step. +The release artifact and the root-owned `/usr/local/bin/mvm` installation are the same single executable. Background +services and the vsock guest agent are compiled into it; there is no separate privileged helper, service binary, +symlink, or extraction step. User-owned development artifacts are not sudo targets. ### Service Invocation @@ -550,8 +553,9 @@ Services are spawned in the background by the core layer via `system.SpawnServic ### Sudoers -Only `mvm run provision` requires passwordless sudo. Managed by `mvm host init` which -creates a drop-in at `/etc/sudoers.d/mvm` granting the `mvm` group passwordless sudo. +During the privilege-boundary migration, `mvm host init` targets only root-owned `/usr/local/bin/mvm` but still retains +the legacy raw-tool and public-service grants needed by unmigrated callers. Task 9 removes those grants and leaves only +the versioned early privileged marker. See ADR-0016; the transitional policy is not the final security model. --- diff --git a/docs/adr/0002-single-go-binary-architecture.md b/docs/adr/0002-single-go-binary-architecture.md index 407b83b8..76f7f208 100644 --- a/docs/adr/0002-single-go-binary-architecture.md +++ b/docs/adr/0002-single-go-binary-architecture.md @@ -2,7 +2,7 @@ **Status:** Active **Date:** 2026-05-22 -**Last Updated:** 2026-06-20 (final Go implementation) +**Last Updated:** 2026-08-20 (root-owned single-binary boundary foundation) **Supersedes:** The previous Python Nuitka two-binaries build (legacy/ directory) The `mvm` project is distributed as a single compiled Go binary with no runtime dependencies. The binary contains both the CLI entry point and all background service subprocesses (console relay, nocloudnet server, loopmount provisioner) via subcommand dispatch (`mvm run `). This replaces the previous Python Nuitka build which produced two separate binaries (`mvm` and `mvm-services`) with multidist symlink dispatch. @@ -31,28 +31,19 @@ The `mvm` project is distributed as a single compiled Go binary with no runtime ```go func main() { - ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM) - defer stop() - - op, cleanup, err := app.Initialize(ctx) - if err != nil { - slog.Error("initialization failed", "error", err) - os.Exit(1) - } - if cleanup != nil { - defer cleanup() - } - - rootCmd := cli.NewRootCmd(op) - if err := rootCmd.ExecuteContext(ctx); err != nil { - if common.HandleErrors(func() error { return err })() != nil { - os.Exit(1) - } - } + os.Exit(run(context.Background(), os.Args[1:], os.Stdin, entrypoints{ + privileged: privileged.Run, + systemInstall: app.RunHostInstallSystemBinary, + normal: runCLI, + })) } ``` -Services are not dispatched via a manual check in main; instead, Cobra subcommands registered in `internal/cli/service.go` handle `mvm run `. +`run` recognizes only the reserved versioned privileged protocol and the exact administrator `host install-system` +bootstrap before ordinary initialization. Both selectors fail closed and cannot enter the public CLI. All ordinary +commands then enter `runCLI`, which establishes signals, calls `app.Initialize()`, and executes Cobra. Public background +services remain Cobra subcommands registered in `internal/cli/service.go`; typed privileged actions progressively replace +the root-requiring public service routes under ADR-0016. ### Service Subcommands @@ -88,11 +79,13 @@ The previous Python implementation (preserved in `legacy/`) used Nuitka to compi - **Single binary**: Users download and install one file. No symlinks, no multidist dispatch. - **No build-time dependency tracking**: Go's static compilation eliminates the need for explicit module inclusion lists. - **Service spawning**: `system.SpawnService()` (`internal/lib/system/spawn.go`) launches the same binary with different subcommand arguments — no separate binary path resolution. -- **Sudoers simplicity**: The sudoers file only needs the `mvm` binary path, not a separate `mvm-services` path. +- **Sudoers simplicity**: The final sudoers policy needs only the versioned early entry point of root-owned + `/usr/local/bin/mvm`, not a separate `mvm-services` or helper executable. - **Build speed**: `go build` is significantly faster than Nuitka compilation (seconds vs minutes). - **Cross-compilation**: `GOOS=linux GOARCH=amd64 go build` produces a target binary without requiring the target toolchain. ## Related Decisions - ADR-0003: Provisioning backend mutual exclusion — the `mvm run provision` subcommand is the loop-mount entry point. +- ADR-0016: Root-owned system installation, early privileged dispatch, and typed privilege migration. - CONTEXT.md "Provisioner Backend" — mount/umount consolidated in `mvm run provision` subcommand. diff --git a/docs/adr/0005-sudo-privilege-architecture.md b/docs/adr/0005-sudo-privilege-architecture.md index 292f92af..a09f4011 100644 --- a/docs/adr/0005-sudo-privilege-architecture.md +++ b/docs/adr/0005-sudo-privilege-architecture.md @@ -4,9 +4,10 @@ **Date:** 2026-05-22 **Last Updated:** 2026-06-20 (Go implementation complete) -> **Replacement accepted, implementation pending:** ADR-0016 replaces this user-owned wildcard sudo design with one -> root-owned system installation and an early typed privileged dispatcher. This ADR continues to describe the current -> implementation until ADR-0016's migration and L2 release gate pass. +> **Replacement accepted, implementation in progress:** ADR-0016 replaces the user-owned wildcard sudo design with one +> root-owned system installation and an early typed privileged dispatcher. The system installer and no-action dispatcher +> foundation are implemented. Raw tool grants and the public-CLI wildcard below remain transitional until Tasks 4-9 and +> the L2 release gate pass; they must not be described as the final security boundary. The project uses a **tightly scoped, group-based sudo model** where members of a dedicated Unix group (`mvm`) can execute specific privileged commands via `sudo` without a password prompt. This avoids running the entire CLI as root while still allowing VM creation (loop-mount), network management (ip/iptables/nft), kernel module loading (modprobe), and sysctl configuration. @@ -38,10 +39,11 @@ Key implications: ## The Actual Sudoers File -`mvm host init` creates `/etc/sudoers.d/mvm` (path resolved by `infra.SudoersDropInPath()`). The file is generated by `GenerateSudoersContent()` (`internal/core/host/utils.go`, line 144) which uses `os.Executable()` to resolve the mvm binary path at generation time: +`mvm host init` creates `/etc/sudoers.d/mvm` (path resolved by `infra.SudoersDropInPath()`). During the migration the file +is generated by `GenerateSudoersContent()` with the fixed root-owned system path: ``` -%mvm ALL=(root) NOPASSWD: /usr/sbin/ip, /usr/sbin/iptables, /usr/sbin/iptables-restore, /usr/sbin/iptables-save, /usr/sbin/nft, /usr/sbin/sysctl, /usr/sbin/modprobe, /path/to/mvm * +%mvm ALL=(root) NOPASSWD: /usr/sbin/ip, /usr/sbin/iptables, /usr/sbin/iptables-restore, /usr/sbin/iptables-save, /usr/sbin/nft, /usr/sbin/sysctl, /usr/sbin/modprobe, /usr/local/bin/mvm * ``` The sudoers file is generated by `GenerateSudoersContent()` using two sources from `internal/infra/constants.go`: @@ -55,8 +57,8 @@ The sudoers file is generated by `GenerateSudoersContent()` using two sources fr - `/usr/sbin/sysctl` (procps) - `/usr/sbin/modprobe` (kmod) -- **Service binaries** — the `mvm` binary itself for privileged `mvm run provision` and `mvm run jailer` subcommands: - - `mvm` path resolved via `os.Executable()` + ` *` wildcard suffix +- **Service binary (transitional)** — fixed `/usr/local/bin/mvm *` for privileged public service subcommands. Task 9 + replaces this wildcard with only `/usr/local/bin/mvm __mvm_privileged_v1 *` after every caller is typed. NOTHING else is permitted. `sudo -n true`, `sudo cat /etc/sudoers.d/mvm`, `sudo test -f ...`, and any other non-listed command WILL prompt for a password. This is by design and is NOT a configuration problem. @@ -70,7 +72,7 @@ The loop-mount call chain is: mvm vm create → ImageProvisioner uses loop-mount backend (not guestfs) → LoopMountBackend.Run() calls runWireOp() (internal/lib/provisioner/loopmount/backend.go, line 188) - → runWireOp() resolves mvm binary path via os.Executable() + → runWireOp() resolves the current binary; supported host operation runs `/usr/local/bin/mvm` → Prepends "sudo" to args when !system.IsRoot() → system.DefaultRunner.Run(ctx, ["sudo", mvmPath, "run", "provision"], ...) ↑ sudoers allows %mvm to run mvm without password @@ -107,22 +109,29 @@ When running `sg mvm -c 'pytest tests/system/'`: ## Why Not Run Everything as Root -- **Principle of least privilege.** The CLI runs as a normal user for HTTP downloads, file I/O, SQLite queries, and all non-privileged operations. Only specific system binaries are elevated. -- **Safety and auditability.** The sudoers file is a compact, auditable list of exactly what can run as root. A compromise of the CLI does not automatically grant root access. -- **Development ergonomics.** Developers build and iterate as a normal user. Only one-time setup (`mvm host init`) needs `sudo`. +- **Principle of least privilege (target).** The CLI runs as a normal user for HTTP downloads, file I/O, SQLite queries, + and non-privileged operations. The transitional raw-tool and public-CLI grants remain broader than the named effects + accepted by ADR-0016 and are removed only at Task 9. +- **Known transitional exposure.** Root ownership prevents replacement of the authorized executable, but + `/usr/local/bin/mvm *` still exposes broad public argument handling as root. A compromise of the ordinary CLI process + must be treated as potentially reaching that authority until typed callers and the marker-only policy replace it. +- **Development ergonomics.** Developers build and iterate as a normal user. Installing/upgrading the trusted system + image and machine-global `host init` remain explicit administrator operations. ## Why a Group-Based Model - The `mvm` group cleanly separates "can manage VMs" from "can't manage VMs" at the OS level - `sg mvm` activates the group for the current shell session without requiring logout/login, CI restart, or environment manipulation - System tests run under `sg mvm -c 'pytest ...'`, inheriting the group across all subprocesses at zero cost -- The setup is **one command** executed once: `sudo mvm host init` +- Initial setup is two explicit administrator steps: `sudo host install-system`, then + `sudo /usr/local/bin/mvm host init`. ## Consequences - `sudo -n true`, `sudo -v`, and any non-listed command will fail for `mvm` group members. The `CheckPrivileges()` function is aware of this and does NOT gate privileged execution on credential validation — it gates on group membership and session group activation. - Adding a new privileged binary requires updating `internal/infra/constants.go` (add to `PrivilegedBinaries` and `PrivilegedBinariesOrdered`) AND regenerating the sudoers file via `mvm host init`. -- The `mvm` binary MUST exist at the path listed in the sudoers file for VM creation to work. If it's missing — as can happen after `cache clean --force` — re-extract via `mvm cache init`. +- The root-owned binary MUST exist at `/usr/local/bin/mvm`. User cache cleanup does not remove it; reinstall it only from + a trusted artifact through `host install-system`. - The group name is defined as `MVMUnixGroup = CLIName` (`internal/infra/constants.go`, line 39). Currently `CLIName = "mvm"`, so the group is `mvm`. - `sg mvm` is a prerequisite for VM-related system tests. Non-VM tests work without it. - Tests that involve VM creation MUST verify the `mvm` group is active before attempting to create VMs. diff --git a/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md b/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md index 31972991..ae8881d1 100644 --- a/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md +++ b/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md @@ -1,6 +1,6 @@ # Root-Owned Single-Binary Privileged Dispatch -**Status:** Accepted — implementation pending +**Status:** Accepted — implementation in progress **Date:** 2026-08-20 **See also:** [ADR-0002: Single Go Binary Architecture](0002-single-go-binary-architecture.md), [ADR-0005: Sudo Privilege Architecture](0005-sudo-privilege-architecture.md), @@ -40,6 +40,25 @@ is an explicit administrator action; a normal user update cannot overwrite it. Development builds may exist elsewhere, but they are never sudo targets. A development build that needs privileged operations calls the compatible system installation and fails closed on a privileged-protocol mismatch. +### Administrator bootstrap and upgrade + +System installation is selected by the exact `host install-system` command before `app.Initialize()`, Cobra, signal +setup, user configuration, the user database, cache resolution, logging configuration, or service construction. The +bootstrap route clears the inherited environment, installs only fixed root-safe values, and then atomically copies the +running image to `/usr/local/bin/mvm`. Flags and extra arguments are rejected inside the same reserved route and cannot +fall through to the ordinary CLI. + +The installer cannot prove whether an already-root process reached it through password-authenticated administrator sudo +or a legacy passwordless rule. It therefore inspects the project-managed sudoers drop-in before touching the executable +and refuses unrecognized active syntax. During migration, an administrator removes the insecure legacy managed rule in +an authenticated root session, runs `sudo host install-system`, and then runs +`sudo /usr/local/bin/mvm host init`. Installation is not an implicit side effect of `host init`. + +Replacing the executable is descriptor-relative, no-follow, temporary-file plus file fsync plus rename plus directory +fsync. Failures after rename report that replacement occurred and whether directory durability is uncertain. A normal +`self-update` may update a user-owned artifact, but it refuses to overwrite the system installation and prints the +administrator bootstrap instruction. + ### Early privileged dispatch `cmd/mvm/main.go` recognizes one reserved, versioned privileged marker before signal setup, `app.Initialize()`, Cobra, @@ -237,9 +256,11 @@ short-lived dispatcher model. ADR-0005 remains the description of the currently implemented legacy policy until this ADR's release gate passes. The replacement sequence is: -1. Add early privileged dispatch and negative tests while the new path is unreachable from sudoers. +1. Add early privileged dispatch, the exact administrator bootstrap route, and negative tests while the final marker-only + path is unreachable from sudoers. 2. Migrate every privileged service/tool call to a typed operation. -3. Install the root-owned system binary and atomically replace the sudoers drop-in after `visudo` validation. +3. In an authenticated administrator session, remove the insecure managed legacy rule, install the trusted artifact as + the root-owned system binary, and atomically activate the marker-only sudoers drop-in after `visudo` validation. 4. Run upgrade, rollback, privilege-abuse, multi-user, path-race, PID-reuse, crash-recovery, Jailer, cgroup, snapshot, hotplug, and firewall L2 tests. 5. Mark ADR-0005 superseded, update all current-behavior documentation, and only then describe the new boundary as active. diff --git a/docs/development/HOW_TO_RUN_SYSTEM_TESTS.md b/docs/development/HOW_TO_RUN_SYSTEM_TESTS.md index 6598ce15..e3b71a17 100644 --- a/docs/development/HOW_TO_RUN_SYSTEM_TESTS.md +++ b/docs/development/HOW_TO_RUN_SYSTEM_TESTS.md @@ -129,7 +129,8 @@ The custom base image (`mvm-test-runner:`) is built once during - cloud-image-utils (for `--cloud-init-mode iso` tests) - build-essential, bc, bison, flex, libncurses-dev, libssl-dev, libelf-dev, git, curl, dwarves (for `kernel_build` tests) -No manual installation on the host is needed beyond the tools in section 1.4. +The orchestrator installs runner-VM contents automatically. The host still requires the release candidate to be +installed and initialized once as shown in section 2 so host-side privileged operations use the same root-owned image. --- @@ -142,13 +143,16 @@ No manual installation on the host is needed beyond the tools in section 1.4. # Verify it exists and is executable test -x dist/mvm && echo "binary: OK ($(dist/mvm --version 2>/dev/null))" -# Copy to ~/.local/bin (for sudo operations — this path has sudoers access) -cp dist/mvm ~/.local/bin/mvm +# Install the exact release candidate as the root-owned system executable +sudo ./dist/mvm host install-system +sudo /usr/local/bin/mvm host init +export MVM_BINARY=/usr/local/bin/mvm ``` -**IMPORTANT**: The binary at `~/.local/bin/mvm` is the default location the -orchestrator looks for (via `MVM_BINARY` env var, defaulting to -`~/.local/bin/mvm`). Set `MVM_BINARY` to point to a different path if needed. +**IMPORTANT**: User-owned binaries are never sudo targets. Set `MVM_BINARY=/usr/local/bin/mvm` when running the +orchestrator so its host-side privileged subprocesses use the same immutable system image that is copied into runner +VMs. The orchestrator's historical default remains a staging-path fallback and must not be used for privilege-boundary +release qualification. --- @@ -207,7 +211,7 @@ Run these to confirm the host is ready for tests: # Host checks echo "KVM: $(test -c /dev/kvm && echo OK || echo MISSING)" echo "Nest: $(cat /sys/module/kvm_intel/parameters/nested 2>/dev/null || echo N/A)" -echo "mvm: $(~/.local/bin/mvm --version 2>/dev/null)" +echo "mvm: $(/usr/local/bin/mvm --version 2>/dev/null)" # Asset mirror has content ls ~/.cache/mvm-asset-mirror/ | head -10 @@ -219,7 +223,7 @@ mvm volume inspect asset-mirror --json 2>/dev/null \ # Custom base image exists mvm image inspect \ - mvm-test-runner:$(~/.local/bin/mvm --version 2>/dev/null | awk '{print $2}') \ + mvm-test-runner:$(/usr/local/bin/mvm --version 2>/dev/null | awk '{print $2}') \ --json 2>/dev/null \ && echo "Base image: OK" \ || echo "Base image: MISSING — run --prepare" @@ -364,7 +368,7 @@ KVM) is a **release blocker** — fix the environment, not the code. | Symptom | Likely Cause | Fix | |---------|-------------|-----| -| `iptables` errors | `mvm host init` not run | Run `sudo ~/.local/bin/mvm host init` | +| `iptables` errors | `mvm host init` not run | Run `sudo /usr/local/bin/mvm host init` | | VM creation hangs | Binary not built or missing | Re-run `./scripts/build.sh release` | | `bridge already exists` | Stale bridges from previous run | Clean up with `mvm network rm --force` | | `Text file busy` on service binary | Stale service processes | `killall -9 mvm-console-relay mvm-nocloud-server mvm-provision` | diff --git a/docs/implementations/SELF_UPDATE.md b/docs/implementations/SELF_UPDATE.md index 28a78527..18d35523 100644 --- a/docs/implementations/SELF_UPDATE.md +++ b/docs/implementations/SELF_UPDATE.md @@ -39,16 +39,16 @@ func (r *Remote) Release(ctx context.Context, tag string) (*Release, error) ``` mvm self-update apply │ - ├── 1. Detect current binary path via os.Executable() - ├── 2. Check write permission on the directory - ├── 3. Fetch latest release from GitHub API - ├── 4. Compare tag vs current BuildVersion (semver) - ├── 5. Find matching asset for arch (mvm / mvm-arm64) + ├── 1. CLI checks the latest release and compares BuildVersion + ├── 2. Apply detects the current binary path via os.Executable() + ├── 3. Reject canonical /usr/local/bin/mvm with the administrator install instruction + ├── 4. Fetch the release for the apply operation + ├── 5. Find the matching asset for arch (mvm / mvm-arm64) ├── 6. Download checksums.sha256 - ├── 7. Download binary to temp file alongside current binary + ├── 7. Download the binary to a temp file beside the user-owned artifact ├── 8. Verify SHA256 against checksums.sha256 ├── 9. os.Rename(temp, current_path) - ├── 10. Restore permissions (executable bit, ownership) + ├── 10. Restore executable permissions └── 11. Print success ``` @@ -56,7 +56,8 @@ mvm self-update apply ### 1. Binary in a root-owned path -`os.Rename` fails if the current binary is in `/usr/bin/mvm` (package install) and the user isn't root. Fall back to printing manual install instructions with the download URL. Do not attempt sudo. +Package-managed `/usr/bin/mvm` remains owned by the package manager. A non-root apply fails when it cannot write the +target directory; self-update never attempts sudo. Use the package manager for these installations. ### 2. Daemon child processes running (console relay, nocloud-net) @@ -72,11 +73,16 @@ Delete temp file, print error with checksum details. Never rename on mismatch. ### 5. Permission restoration after rename -After rename, set `os.Chmod(newPath, 0755)` to ensure the binary stays executable. Sudoers references the path, not the inode — no change needed there. +After rename, set `os.Chmod(newPath, 0755)` so the user-owned artifact stays executable. Sudoers never authorizes that +path; the privileged target is the separate root-owned system installation. ### 6. Root-owned binary -Self-update only works for user-installed binaries (e.g. `~/.local/bin/mvm`). For package-managed installs (`/usr/bin/mvm`), print a message directing the user to use the package manager. +Self-update only replaces user-owned artifacts. The apply service rejects canonical root-owned `/usr/local/bin/mvm` +before its own release request and directs an administrator to run the new trusted artifact through +`sudo host install-system`. The public `self-update` and `self-update apply` commands perform their +initial update check first, so they may contact the release service before apply returns that instruction. +Package-managed `/usr/bin/mvm` installations continue to use their package manager. ### 7. Same version reinstall diff --git a/internal/app/system_install.go b/internal/app/system_install.go new file mode 100644 index 00000000..fe0d839f --- /dev/null +++ b/internal/app/system_install.go @@ -0,0 +1,147 @@ +package app + +import ( + "context" + "fmt" + "log/slog" + "os" + "strings" + + "mvmctl/internal/core/host" + "mvmctl/internal/infra" + "mvmctl/pkg/errs" +) + +const ( + systemInstallPATH = "/usr/sbin:/usr/bin:/sbin:/bin" + systemInstallHOME = "/root" + systemInstallLocale = "C" +) + +type systemInstallDeps struct { + clearEnvironment func() + setEnvironment func(string, string) error + installSystemBinary func(context.Context) (bool, error) +} + +// IsHostInstallSystemBinaryInvocation reserves every argument sequence that +// selects the administrator bootstrap command. Validation remains exact inside +// RunHostInstallSystemBinary so malformed invocations cannot fall through to +// normal application initialization. +func IsHostInstallSystemBinaryInvocation(args []string) bool { + if len(args) == 0 { + return false + } + + // A leading flag makes following non-flag tokens ambiguous: they can be + // malformed flag values rather than a different command. Reserve the + // security-sensitive sequence in that case and let the exact validator + // reject it. An ordinary unflagged command before host remains ordinary. + leadingFlag := strings.HasPrefix(args[0], "-") + hostIndex := -1 + for index, arg := range args { + if arg == "help" { + return false + } + if arg == "host" { + if index > 0 && !leadingFlag { + return false + } + hostIndex = index + break + } + } + if hostIndex < 0 { + return false + } + for _, arg := range args[hostIndex+1:] { + if arg == "install-system" { + return true + } + } + return false +} + +// RunHostInstallSystemBinary performs and reports the minimal administrator +// bootstrap without resolving user state or constructing the normal application. +func RunHostInstallSystemBinary(ctx context.Context, args []string) error { + changed, err := hostInstallSystemBinary(ctx, args, systemInstallDeps{ + clearEnvironment: os.Clearenv, + setEnvironment: os.Setenv, + installSystemBinary: host.InstallSystemBinary, + }) + reportHostInstallSystemBinary(changed, err) + return err +} + +func hostInstallSystemBinary( + ctx context.Context, + args []string, + deps systemInstallDeps, +) (bool, error) { + if len(args) != 2 || args[0] != "host" || args[1] != "install-system" { + return false, errs.New( + errs.CodeValidationFailed, + "host install-system accepts no flags or arguments", + ) + } + if err := ctx.Err(); err != nil { + return false, errs.WrapMsg(errs.CodeInternal, "system installation context ended", err) + } + if err := sanitizeSystemInstallEnvironment(deps); err != nil { + return false, err + } + return deps.installSystemBinary(ctx) +} + +func sanitizeSystemInstallEnvironment(deps systemInstallDeps) error { + deps.clearEnvironment() + fixedValues := [...]struct { + key string + value string + }{ + {key: "PATH", value: systemInstallPATH}, + {key: "HOME", value: systemInstallHOME}, + {key: "LANG", value: systemInstallLocale}, + {key: "LC_ALL", value: systemInstallLocale}, + } + for _, entry := range fixedValues { + if err := deps.setEnvironment(entry.key, entry.value); err != nil { + return errs.WrapMsg( + errs.CodeInternal, + fmt.Sprintf("set fixed system installation environment %s", entry.key), + err, + ) + } + } + return nil +} + +func reportHostInstallSystemBinary(changed bool, err error) { + if err != nil { + if !changed { + return + } + durabilityUncertain := false + if domainErr := errs.AsDomainError(err); domainErr != nil { + durabilityUncertain, _ = domainErr.Details["durability_uncertain"].(bool) + } + if durabilityUncertain { + slog.Warn( + "system binary was replaced but directory durability could not be confirmed", + "path", infra.SystemBinaryPath, + ) + return + } + slog.Warn( + "system binary was replaced but final completion checks failed", + "path", infra.SystemBinaryPath, + ) + return + } + if changed { + slog.Info("installed trusted system binary", "path", infra.SystemBinaryPath) + return + } + slog.Info("system binary already current", "path", infra.SystemBinaryPath) +} diff --git a/internal/app/system_install_test.go b/internal/app/system_install_test.go new file mode 100644 index 00000000..6f267348 --- /dev/null +++ b/internal/app/system_install_test.go @@ -0,0 +1,200 @@ +package app + +import ( + "context" + "errors" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "mvmctl/pkg/errs" +) + +func TestIsHostInstallSystemBinaryInvocation_ReservesSelectedCommand(t *testing.T) { + tests := []struct { + name string + args []string + want bool + }{ + {name: "exact", args: []string{"host", "install-system"}, want: true}, + {name: "extra argument", args: []string{"host", "install-system", "unexpected"}, want: true}, + {name: "root flag before command", args: []string{"--debug", "host", "install-system"}, want: true}, + {name: "malformed root flag value", args: []string{"--debug", "false", "host", "install-system"}, want: true}, + { + name: "repeated malformed root flag values", + args: []string{"--debug", "false", "false", "host", "install-system"}, + want: true, + }, + {name: "persistent flag inside command", args: []string{"host", "--verbose", "install-system"}, want: true}, + { + name: "unknown flag value inside command", + args: []string{"host", "--unknown", "value", "install-system"}, + want: true, + }, + {name: "unknown trailing flag", args: []string{"host", "install-system", "--force"}, want: true}, + {name: "argument separator inside command", args: []string{"host", "--", "install-system"}, want: true}, + {name: "leading argument separator", args: []string{"--", "host", "install-system"}, want: true}, + {name: "different host command", args: []string{"host", "init"}, want: false}, + {name: "install words after another command", args: []string{"vm", "host", "install-system"}, want: false}, + {name: "help request remains ordinary", args: []string{"help", "host", "install-system"}, want: false}, + { + name: "flagged help request remains ordinary", + args: []string{"--debug", "help", "host", "install-system"}, + want: false, + }, + {name: "reversed operation order", args: []string{"install-system", "host"}, want: false}, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + assert.Equal(t, tt.want, IsHostInstallSystemBinaryInvocation(tt.args)) + }) + } +} + +func TestHostInstallSystemBinary_ReplacesEnvironmentBeforeInstaller(t *testing.T) { + store := newSystemInstallEnvironmentStore(map[string]string{ + "PATH": "/tmp/attacker-bin", + "HOME": "/home/attacker", + "MVM_CACHE_DIR": "/tmp/attacker-cache", + "LD_PRELOAD": "/tmp/inject.so", + }) + installCalls := 0 + + changed, err := hostInstallSystemBinary( + context.Background(), + []string{"host", "install-system"}, + systemInstallDeps{ + clearEnvironment: store.clear, + setEnvironment: store.set, + installSystemBinary: func(context.Context) (bool, error) { + installCalls++ + assert.Equal(t, map[string]string{ + "PATH": "/usr/sbin:/usr/bin:/sbin:/bin", + "HOME": "/root", + "LANG": "C", + "LC_ALL": "C", + }, store.values) + return true, nil + }, + }, + ) + + require.NoError(t, err) + assert.True(t, changed) + assert.Equal(t, 1, store.clearCalls) + assert.Equal(t, 1, installCalls) +} + +func TestHostInstallSystemBinary_FailsClosedBeforeInstallerForMalformedArguments(t *testing.T) { + tests := [][]string{ + {"host", "install-system", "unexpected"}, + {"--debug", "host", "install-system"}, + {"--debug", "false", "host", "install-system"}, + {"host", "--verbose", "install-system"}, + {"host", "--unknown", "value", "install-system"}, + {"host", "--", "install-system"}, + } + + for _, args := range tests { + t.Run(args[0], func(t *testing.T) { + installCalls := 0 + changed, err := hostInstallSystemBinary(context.Background(), args, systemInstallDeps{ + clearEnvironment: func() {}, + setEnvironment: func(string, string) error { return nil }, + installSystemBinary: func(context.Context) (bool, error) { + installCalls++ + return true, nil + }, + }) + + require.Error(t, err) + assert.False(t, changed) + assert.Equal(t, errs.CodeValidationFailed, errs.AsDomainError(err).Code) + assert.Zero(t, installCalls) + }) + } +} + +func TestHostInstallSystemBinary_PreservesPartialSuccess(t *testing.T) { + sentinel := errs.New( + errs.CodeInternal, + "directory sync failed", + errs.WithDetails(map[string]any{ + "system_binary_replaced": true, + "durability_uncertain": true, + }), + ) + + changed, err := hostInstallSystemBinary( + context.Background(), + []string{"host", "install-system"}, + systemInstallDeps{ + clearEnvironment: func() {}, + setEnvironment: func(string, string) error { return nil }, + installSystemBinary: func(context.Context) (bool, error) { + return true, sentinel + }, + }, + ) + + assert.True(t, changed) + assert.ErrorIs(t, err, sentinel) + domainErr := errs.AsDomainError(err) + require.NotNil(t, domainErr) + assert.Equal(t, true, domainErr.Details["system_binary_replaced"]) + assert.Equal(t, true, domainErr.Details["durability_uncertain"]) +} + +func TestHostInstallSystemBinary_StopsWhenEnvironmentCannotBeSanitized(t *testing.T) { + sentinel := errors.New("set environment rejected") + installCalls := 0 + + changed, err := hostInstallSystemBinary( + context.Background(), + []string{"host", "install-system"}, + systemInstallDeps{ + clearEnvironment: func() {}, + setEnvironment: func(key, _ string) error { + if key == "HOME" { + return sentinel + } + return nil + }, + installSystemBinary: func(context.Context) (bool, error) { + installCalls++ + return true, nil + }, + }, + ) + + require.Error(t, err) + assert.ErrorIs(t, err, sentinel) + assert.False(t, changed) + assert.Equal(t, errs.CodeInternal, errs.AsDomainError(err).Code) + assert.Zero(t, installCalls) +} + +type systemInstallEnvironmentStore struct { + values map[string]string + clearCalls int +} + +func newSystemInstallEnvironmentStore(values map[string]string) *systemInstallEnvironmentStore { + copyValues := make(map[string]string, len(values)) + for key, value := range values { + copyValues[key] = value + } + return &systemInstallEnvironmentStore{values: copyValues} +} + +func (s *systemInstallEnvironmentStore) clear() { + s.clearCalls++ + s.values = make(map[string]string) +} + +func (s *systemInstallEnvironmentStore) set(key, value string) error { + s.values[key] = value + return nil +} diff --git a/internal/cli/host.go b/internal/cli/host.go index 9dbaae06..a5d96fd8 100644 --- a/internal/cli/host.go +++ b/internal/cli/host.go @@ -90,6 +90,7 @@ func NewHostCmd(hostAPI api.HostAPI) *cobra.Command { Requires root privileges for most operations. Run with: sudo mvm host `, } + cmd.AddCommand(newHostInstallSystemCmd(hostAPI)) cmd.AddCommand(newHostInitCmd(hostAPI)) cmd.AddCommand(newHostStatusCmd(hostAPI)) cmd.AddCommand(newHostInfoCmd(hostAPI)) @@ -99,6 +100,50 @@ Requires root privileges for most operations. Run with: sudo mvm host ` return cmd } +func newHostInstallSystemCmd(hostAPI api.HostAPI) *cobra.Command { + return &cobra.Command{ + Use: "install-system", + Short: "Install or upgrade the trusted system mvm binary", + Long: fmt.Sprintf(`Install or upgrade the trusted system mvm binary at %s. + +This is an explicit administrator bootstrap operation. Run the trusted +artifact you intend to install with sudo: + + sudo host install-system`, infra.SystemBinaryPath), + Args: cobra.NoArgs, + RunE: func(cmd *cobra.Command, _ []string) error { + changed, err := hostAPI.HostInstallSystemBinary(cmd.Context()) + if err != nil { + if changed { + common.Cli.Warning(systemInstallPartialWarning(err)) + } + return err + } + if !changed { + common.Cli.Info(fmt.Sprintf("System binary already current at %s.", infra.SystemBinaryPath)) + return nil + } + common.Cli.Success(fmt.Sprintf("Installed trusted system binary at %s.", infra.SystemBinaryPath)) + return nil + }, + } +} + +func systemInstallPartialWarning(err error) string { + if domainErr := errs.AsDomainError(err); domainErr != nil { + if durabilityUncertain, _ := domainErr.Details["durability_uncertain"].(bool); durabilityUncertain { + return fmt.Sprintf( + "System binary was replaced at %s, but directory durability could not be confirmed.", + infra.SystemBinaryPath, + ) + } + } + return fmt.Sprintf( + "System binary was replaced at %s, but final completion checks failed.", + infra.SystemBinaryPath, + ) +} + func newHostInitCmd(hostAPI api.HostAPI) *cobra.Command { cmd := &cobra.Command{ Use: "init", diff --git a/internal/cli/host_internal_test.go b/internal/cli/host_internal_test.go new file mode 100644 index 00000000..2c8a7b19 --- /dev/null +++ b/internal/cli/host_internal_test.go @@ -0,0 +1,48 @@ +package cli + +import ( + "testing" + + "github.com/stretchr/testify/assert" + + "mvmctl/pkg/errs" +) + +func TestSystemInstallPartialWarningDescribesKnownFailure(t *testing.T) { + tests := []struct { + name string + err error + wantContains string + wantNotContains string + }{ + { + name: "generic post-replacement failure", + err: errs.New( + errs.CodeHostInitFailed, + "close retained descriptor", + errs.WithDetails(map[string]any{"system_binary_replaced": true}), + ), + wantContains: "final completion checks failed", + wantNotContains: "durability", + }, + { + name: "directory durability failure", + err: errs.New( + errs.CodeHostInitFailed, + "sync directory", + errs.WithDetails(map[string]any{"durability_uncertain": true}), + ), + wantContains: "durability could not be confirmed", + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + warning := systemInstallPartialWarning(tt.err) + assert.Contains(t, warning, tt.wantContains) + if tt.wantNotContains != "" { + assert.NotContains(t, warning, tt.wantNotContains) + } + }) + } +} diff --git a/internal/cli/host_test.go b/internal/cli/host_test.go index 8b37694b..69456d02 100644 --- a/internal/cli/host_test.go +++ b/internal/cli/host_test.go @@ -28,6 +28,7 @@ func TestNewHostCmd(t *testing.T) { hasAlias bool alias string }{ + {use: "install-system", hasAlias: false}, {use: "init", hasAlias: false}, {use: "status", hasAlias: false}, {use: "info", hasAlias: false}, @@ -62,6 +63,37 @@ func TestNewHostCmd(t *testing.T) { }) } +// Rationale: Installing the trusted system image is an explicit administrator +// bootstrap operation, not an implicit side effect of host initialization. +func TestNewHostCmd_InstallSystemBinary(t *testing.T) { + wantErr := errors.New("durability uncertain") + hostAPI := &installSystemHostAPI{ + MockHostAPI: &testutil.MockHostAPI{}, + changed: true, + err: wantErr, + } + cmd := cli.NewHostCmd(hostAPI) + + installCmd, _, err := cmd.Find([]string{"install-system"}) + require.NoError(t, err) + require.Equal(t, "install-system", installCmd.Use) + err = installCmd.RunE(installCmd, nil) + require.ErrorIs(t, err, wantErr) + assert.Equal(t, 1, hostAPI.calls) +} + +type installSystemHostAPI struct { + *testutil.MockHostAPI + changed bool + err error + calls int +} + +func (f *installSystemHostAPI) HostInstallSystemBinary(context.Context) (bool, error) { + f.calls++ + return f.changed, f.err +} + // --- Host info (via host info) --- // Rationale: Host info shows hardware and capacity. A broken info command // prevents users from seeing available resources for VM placement. diff --git a/internal/core/host/sudoers_config.go b/internal/core/host/sudoers_config.go new file mode 100644 index 00000000..e4621a02 --- /dev/null +++ b/internal/core/host/sudoers_config.go @@ -0,0 +1,59 @@ +package host + +import ( + "context" + "errors" + "fmt" + "os" + + "mvmctl/internal/infra" + "mvmctl/pkg/errs" +) + +type sudoersConfigDeps struct { + verifySystemBinary func() error + readFile func(string) ([]byte, error) + writeSudoers func(context.Context, string, string) error +} + +// ConfigureSudoers reconciles the single project-owned sudoers drop-in. +// Its fixed group, path, and content are not caller-selectable. +func (s *Service) ConfigureSudoers(ctx context.Context) (bool, error) { + return configureSudoers(ctx, sudoersConfigDeps{ + verifySystemBinary: verifyCanonicalSystemBinaryForSudoers, + readFile: os.ReadFile, + writeSudoers: WriteSudoers, + }) +} + +func configureSudoers(ctx context.Context, deps sudoersConfigDeps) (bool, error) { + if err := deps.verifySystemBinary(); err != nil { + return false, err + } + path := infra.SudoersDropInPath() + want := GenerateSudoersContent(infra.MVMUnixGroup) + current, err := deps.readFile(path) + if err == nil && string(current) == want { + return false, nil + } + if err != nil && !errors.Is(err, os.ErrNotExist) { + return false, errs.WrapMsg( + errs.CodePrivilegeSudoers, + fmt.Sprintf("read managed sudoers policy %s", path), + err, + errs.WithClass(errs.ClassInternal), + ) + } + if err := deps.writeSudoers(ctx, path, want); err != nil { + if errs.AsDomainError(err) != nil { + return false, err + } + return false, errs.WrapMsg( + errs.CodePrivilegeSudoers, + fmt.Sprintf("configure managed sudoers policy %s", path), + err, + errs.WithClass(errs.ClassInternal), + ) + } + return true, nil +} diff --git a/internal/core/host/sudoers_config_test.go b/internal/core/host/sudoers_config_test.go new file mode 100644 index 00000000..c22466e9 --- /dev/null +++ b/internal/core/host/sudoers_config_test.go @@ -0,0 +1,205 @@ +package host + +import ( + "context" + "errors" + "os" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + "golang.org/x/sys/unix" + + "mvmctl/internal/infra" + "mvmctl/pkg/errs" +) + +func TestConfigureSudoers_StateDetectionAndWrite(t *testing.T) { + wantContent := GenerateSudoersContent(infra.MVMUnixGroup) + tests := []struct { + name string + readData []byte + readErr error + wantChanged bool + wantWrites int + }{ + { + name: "current policy is unchanged", + readData: []byte(wantContent), + wantWrites: 0, + }, + { + name: "missing policy is installed", + readErr: os.ErrNotExist, + wantChanged: true, + wantWrites: 1, + }, + { + name: "stale policy is replaced", + readData: []byte("%mvm ALL=(root) NOPASSWD: /old/mvm *\n"), + wantChanged: true, + wantWrites: 1, + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + readPath := "" + writePath := "" + writeContent := "" + writes := 0 + deps := sudoersConfigDeps{ + verifySystemBinary: func() error { return nil }, + readFile: func(path string) ([]byte, error) { + readPath = path + return tt.readData, tt.readErr + }, + writeSudoers: func(_ context.Context, path, content string) error { + writes++ + writePath = path + writeContent = content + return nil + }, + } + + changed, err := configureSudoers(context.Background(), deps) + require.NoError(t, err) + assert.Equal(t, tt.wantChanged, changed) + assert.Equal(t, tt.wantWrites, writes) + assert.Equal(t, infra.SudoersDropInPath(), readPath) + if tt.wantWrites > 0 { + assert.Equal(t, infra.SudoersDropInPath(), writePath) + assert.Equal(t, wantContent, writeContent) + } + }) + } +} + +func TestConfigureSudoers_FailsClosed(t *testing.T) { + tests := []struct { + name string + readErr error + writeErr error + }{ + {name: "unreadable existing policy", readErr: os.ErrPermission}, + {name: "write or visudo failure", writeErr: errors.New("visudo rejected policy")}, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + writes := 0 + deps := sudoersConfigDeps{ + verifySystemBinary: func() error { return nil }, + readFile: func(string) ([]byte, error) { + if tt.readErr != nil { + return nil, tt.readErr + } + return []byte("stale"), nil + }, + writeSudoers: func(context.Context, string, string) error { + writes++ + return tt.writeErr + }, + } + + changed, err := configureSudoers(context.Background(), deps) + require.Error(t, err) + assert.False(t, changed) + assert.NotNil(t, errs.AsDomainError(err)) + if tt.readErr != nil { + assert.Zero(t, writes) + } else { + assert.Equal(t, 1, writes) + } + }) + } +} + +func TestConfigureSudoers_VerifiesCanonicalSystemBinaryBeforePolicyAccess(t *testing.T) { + tests := []struct { + name string + mutate func(*systemBinaryInstallFixture) + wantErr bool + }{ + { + name: "absent target", + mutate: func(f *systemBinaryInstallFixture) { f.targetExists = false }, + wantErr: true, + }, + { + name: "target symlink", + mutate: func(f *systemBinaryInstallFixture) { + stat := f.stats[installTargetFD] + stat.Mode = unix.S_IFLNK | 0777 + f.stats[installTargetFD] = stat + }, + wantErr: true, + }, + { + name: "wrong target owner", + mutate: func(f *systemBinaryInstallFixture) { + stat := f.stats[installTargetFD] + stat.Uid = 1000 + f.stats[installTargetFD] = stat + }, + wantErr: true, + }, + { + name: "wrong target mode", + mutate: func(f *systemBinaryInstallFixture) { + stat := f.stats[installTargetFD] + stat.Mode = unix.S_IFREG | 0775 + f.stats[installTargetFD] = stat + }, + wantErr: true, + }, + { + name: "unsafe ancestor", + mutate: func(f *systemBinaryInstallFixture) { + stat := f.stats[installLocalFD] + stat.Mode = unix.S_IFDIR | 0775 + f.stats[installLocalFD] = stat + }, + wantErr: true, + }, + {name: "valid canonical target", mutate: func(*systemBinaryInstallFixture) {}}, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + fixture := newSystemBinaryInstallFixture() + tt.mutate(fixture) + reads := 0 + writes := 0 + deps := sudoersConfigDeps{ + verifySystemBinary: func() error { + return verifySystemBinaryForSudoers( + fixture.deps, + productionSystemBinaryInstallPolicy(), + ) + }, + readFile: func(string) ([]byte, error) { + reads++ + return nil, os.ErrNotExist + }, + writeSudoers: func(context.Context, string, string) error { + writes++ + return nil + }, + } + + changed, err := configureSudoers(context.Background(), deps) + if tt.wantErr { + require.Error(t, err) + assert.False(t, changed) + assert.Zero(t, reads) + assert.Zero(t, writes) + return + } + require.NoError(t, err) + assert.True(t, changed) + assert.Equal(t, 1, reads) + assert.Equal(t, 1, writes) + }) + } +} diff --git a/internal/core/host/system_binary.go b/internal/core/host/system_binary.go new file mode 100644 index 00000000..da333a07 --- /dev/null +++ b/internal/core/host/system_binary.go @@ -0,0 +1,629 @@ +package host + +import ( + "context" + "crypto/rand" + "encoding/hex" + "errors" + "fmt" + "io" + "log/slog" + "os" + "slices" + "strings" + + "golang.org/x/sys/unix" + + "mvmctl/internal/infra" + "mvmctl/pkg/errs" +) + +const ( + directoryInstallFlags = unix.O_RDONLY | unix.O_DIRECTORY | unix.O_CLOEXEC | unix.O_NOFOLLOW + sourceInstallFlags = unix.O_RDONLY | unix.O_CLOEXEC + targetInspectFlags = unix.O_PATH | unix.O_CLOEXEC | unix.O_NOFOLLOW + tempInstallFlags = unix.O_WRONLY | unix.O_CREAT | unix.O_EXCL | unix.O_CLOEXEC | unix.O_NOFOLLOW + + systemBinaryName = "mvm" + systemBinaryCopyChunk = 32 * 1024 + tempNameAttempts = 8 +) + +type systemBinaryInstallDeps struct { + effectiveUID func() int + readSudoers func(string) ([]byte, error) + open func(string, int, uint32) (int, error) + openAt func(int, string, int, uint32) (int, error) + mkdirAt func(int, string, uint32) error + fstat func(int, *unix.Stat_t) error + fchown func(int, int, int) error + fchmod func(int, uint32) error + read func(int, []byte) (int, error) + write func(int, []byte) (int, error) + fsync func(int) error + close func(int) error + renameAt func(int, string, int, string) error + unlinkAt func(int, string, int) error + randomName func() (string, error) +} + +type systemBinaryInstallPolicy struct { + rootPath string + expectedUID uint32 + expectedGID uint32 +} + +func productionSystemBinaryInstallPolicy() systemBinaryInstallPolicy { + return systemBinaryInstallPolicy{rootPath: "/", expectedUID: 0, expectedGID: 0} +} + +// InstallSystemBinary installs the running image at the canonical system path. +// CRITICAL: This administrator-only operation accepts no caller-selected path +// and retains no-follow directory descriptors through atomic replacement. +func InstallSystemBinary(ctx context.Context) (bool, error) { + return installSystemBinaryWithPolicy( + ctx, + realSystemBinaryInstallDeps(), + productionSystemBinaryInstallPolicy(), + ) +} + +// InstallSystemBinary installs the running image through the host Service. +func (s *Service) InstallSystemBinary(ctx context.Context) (bool, error) { + changed, err := InstallSystemBinary(ctx) + if err != nil { + slog.Error("Could not install canonical system binary", "path", infra.SystemBinaryPath, "error", err) + } + return changed, err +} + +func installSystemBinary( + ctx context.Context, + deps systemBinaryInstallDeps, +) (changed bool, returnErr error) { + return installSystemBinaryWithPolicy(ctx, deps, productionSystemBinaryInstallPolicy()) +} + +func installSystemBinaryWithPolicy( + ctx context.Context, + deps systemBinaryInstallDeps, + policy systemBinaryInstallPolicy, +) (changed bool, returnErr error) { + if deps.effectiveUID() != 0 { + return false, errs.New( + errs.CodePrivilegeRequired, + fmt.Sprintf("installing %s requires root", infra.SystemBinaryPath), + errs.WithClass(errs.ClassNeedsInteraction), + ) + } + if err := ctx.Err(); err != nil { + return false, systemBinaryInstallError("install system binary", err) + } + if err := guardSystemBinaryInstallSudoers(deps); err != nil { + return false, err + } + + retainedFDs := make([]int, 0, 6) + defer func() { + if closeErr := closeSystemBinaryDescriptors(deps, retainedFDs); closeErr != nil { + durabilityUncertain := false + if domainErr := errs.AsDomainError(returnErr); domainErr != nil { + durabilityUncertain, _ = domainErr.Details["durability_uncertain"].(bool) + } + returnErr = joinSystemBinaryInstallError( + returnErr, + "close system binary installer descriptor", + closeErr, + ) + if changed { + returnErr = annotateSystemBinaryReplacement(returnErr, durabilityUncertain) + } + } + }() + + rootFD, err := deps.open(policy.rootPath, directoryInstallFlags, 0) + if err != nil { + return false, systemBinaryInstallError("open filesystem root", err) + } + retainedFDs = append(retainedFDs, rootFD) + if err := verifySystemBinaryInstallDirectory(deps, rootFD, "/", policy); err != nil { + return false, err + } + + sourceFD, err := deps.open("/proc/self/exe", sourceInstallFlags, 0) + if err != nil { + return false, systemBinaryInstallError("open running process image", err) + } + retainedFDs = append(retainedFDs, sourceFD) + sourceStat, err := inspectSystemBinaryInstallFile(deps, sourceFD, "running process image") + if err != nil { + return false, err + } + + usrFD, err := openSystemBinaryInstallDirectory(deps, rootFD, "usr", false, policy) + if err != nil { + return false, err + } + retainedFDs = append(retainedFDs, usrFD) + localFD, err := openSystemBinaryInstallDirectory(deps, usrFD, "local", true, policy) + if err != nil { + return false, err + } + retainedFDs = append(retainedFDs, localFD) + binFD, err := openSystemBinaryInstallDirectory(deps, localFD, "bin", true, policy) + if err != nil { + return false, err + } + retainedFDs = append(retainedFDs, binFD) + + targetFD, targetStat, targetExists, err := inspectExistingSystemBinary(deps, binFD) + if err != nil { + return false, err + } + if targetExists { + retainedFDs = append(retainedFDs, targetFD) + if sourceStat.Dev == targetStat.Dev && sourceStat.Ino == targetStat.Ino && + targetStat.Uid == policy.expectedUID && targetStat.Gid == policy.expectedGID && + targetStat.Mode&07777 == infra.ExecutablePerm { + return false, nil + } + } + + tempName, tempFD, err := createSystemBinaryTemp(deps, binFD) + if err != nil { + return false, err + } + tempOpen := true + abort := func(cause error) (bool, error) { + return false, abortSystemBinaryInstall(deps, binFD, tempFD, tempName, tempOpen, cause) + } + + if err := copySystemBinaryImage(ctx, deps, sourceFD, tempFD); err != nil { + return abort(systemBinaryInstallError("copy running process image", err)) + } + if err := deps.fchown(tempFD, int(policy.expectedUID), int(policy.expectedGID)); err != nil { + return abort(systemBinaryInstallError("set system binary owner", err)) + } + if err := deps.fchmod(tempFD, uint32(infra.ExecutablePerm)); err != nil { + return abort(systemBinaryInstallError("set system binary mode", err)) + } + tempStat, err := verifyInstalledSystemBinaryDescriptor(deps, tempFD, policy) + if err != nil { + return abort(err) + } + if tempStat.Size != sourceStat.Size { + return abort(systemBinaryInstallError( + "verify copied system binary size", + fmt.Errorf("got %d bytes, expected %d", tempStat.Size, sourceStat.Size), + )) + } + if err := deps.fsync(tempFD); err != nil { + return abort(systemBinaryInstallError("sync system binary temporary file", err)) + } + if err := deps.close(tempFD); err != nil { + tempOpen = false + return abort(systemBinaryInstallError("close system binary temporary file", err)) + } + tempOpen = false + + if err := ctx.Err(); err != nil { + return abort(systemBinaryInstallError("install system binary before replacement", err)) + } + if err := deps.renameAt(binFD, tempName, binFD, systemBinaryName); err != nil { + return abort(systemBinaryInstallError("atomically replace system binary", err)) + } + changed = true + if err := deps.fsync(binFD); err != nil { + return changed, annotateSystemBinaryReplacement( + systemBinaryInstallError("sync system binary directory", err), + true, + ) + } + return changed, nil +} + +func openSystemBinaryInstallDirectory( + deps systemBinaryInstallDeps, + parentFD int, + name string, + allowCreate bool, + policy systemBinaryInstallPolicy, +) (int, error) { + fd, err := deps.openAt(parentFD, name, directoryInstallFlags, 0) + if err == nil { + if verifyErr := verifySystemBinaryInstallDirectory(deps, fd, name, policy); verifyErr != nil { + return -1, closeRejectedSystemBinaryDescriptor(deps, fd, verifyErr) + } + return fd, nil + } + if !errors.Is(err, unix.ENOENT) || !allowCreate { + return -1, systemBinaryInstallError("open trusted system binary directory "+name, err) + } + mkdirErr := deps.mkdirAt(parentFD, name, uint32(infra.DirPerm)) + created := mkdirErr == nil + if mkdirErr != nil && !errors.Is(mkdirErr, unix.EEXIST) { + return -1, systemBinaryInstallError("create trusted system binary directory "+name, mkdirErr) + } + fd, err = deps.openAt(parentFD, name, directoryInstallFlags, 0) + if err != nil { + return -1, systemBinaryInstallError("open created system binary directory "+name, err) + } + if !created { + if verifyErr := verifySystemBinaryInstallDirectory(deps, fd, name, policy); verifyErr != nil { + return -1, closeRejectedSystemBinaryDescriptor(deps, fd, verifyErr) + } + return fd, nil + } + if err := deps.fchown(fd, int(policy.expectedUID), int(policy.expectedGID)); err != nil { + return -1, closeRejectedSystemBinaryDescriptor( + deps, + fd, + systemBinaryInstallError("set created system binary directory owner "+name, err), + ) + } + if err := deps.fchmod(fd, uint32(infra.DirPerm)); err != nil { + return -1, closeRejectedSystemBinaryDescriptor( + deps, + fd, + systemBinaryInstallError("set created system binary directory mode "+name, err), + ) + } + if err := verifyCreatedSystemBinaryInstallDirectory(deps, fd, name, policy); err != nil { + return -1, closeRejectedSystemBinaryDescriptor(deps, fd, err) + } + if err := deps.fsync(fd); err != nil { + return -1, closeRejectedSystemBinaryDescriptor( + deps, + fd, + systemBinaryInstallError("sync created system binary directory "+name, err), + ) + } + if err := deps.fsync(parentFD); err != nil { + return -1, closeRejectedSystemBinaryDescriptor( + deps, + fd, + systemBinaryInstallError("sync parent of created system binary directory "+name, err), + ) + } + return fd, nil +} + +func verifySystemBinaryInstallDirectory( + deps systemBinaryInstallDeps, + fd int, + name string, + policy systemBinaryInstallPolicy, +) error { + var stat unix.Stat_t + if err := deps.fstat(fd, &stat); err != nil { + return systemBinaryInstallError("inspect trusted system binary directory "+name, err) + } + if stat.Mode&unix.S_IFMT != unix.S_IFDIR { + return systemBinaryInstallError( + "inspect trusted system binary directory "+name, + fmt.Errorf("must be a directory"), + ) + } + if stat.Uid != policy.expectedUID { + return systemBinaryInstallError( + "inspect trusted system binary directory "+name, + fmt.Errorf("must be owned by root"), + ) + } + if stat.Mode&0022 != 0 { + return systemBinaryInstallError( + "inspect trusted system binary directory "+name, + fmt.Errorf("must not be group/world writable"), + ) + } + return nil +} + +func verifyCreatedSystemBinaryInstallDirectory( + deps systemBinaryInstallDeps, + fd int, + name string, + policy systemBinaryInstallPolicy, +) error { + if err := verifySystemBinaryInstallDirectory(deps, fd, name, policy); err != nil { + return err + } + var stat unix.Stat_t + if err := deps.fstat(fd, &stat); err != nil { + return systemBinaryInstallError("inspect created system binary directory "+name, err) + } + if stat.Gid != policy.expectedGID || stat.Mode&07777 != infra.DirPerm { + return systemBinaryInstallError( + "inspect created system binary directory "+name, + fmt.Errorf("must be root:root with mode 0755"), + ) + } + return nil +} + +func inspectSystemBinaryInstallFile( + deps systemBinaryInstallDeps, + fd int, + description string, +) (unix.Stat_t, error) { + var stat unix.Stat_t + if err := deps.fstat(fd, &stat); err != nil { + return unix.Stat_t{}, systemBinaryInstallError("inspect "+description, err) + } + if stat.Mode&unix.S_IFMT != unix.S_IFREG { + return unix.Stat_t{}, systemBinaryInstallError("inspect "+description, fmt.Errorf("must be a regular file")) + } + return stat, nil +} + +func inspectExistingSystemBinary( + deps systemBinaryInstallDeps, + binFD int, +) (int, unix.Stat_t, bool, error) { + targetFD, err := deps.openAt(binFD, systemBinaryName, targetInspectFlags, 0) + if errors.Is(err, unix.ENOENT) { + return -1, unix.Stat_t{}, false, nil + } + if err != nil { + return -1, unix.Stat_t{}, false, systemBinaryInstallError("open existing system binary", err) + } + targetStat, err := inspectSystemBinaryInstallFile(deps, targetFD, "existing system binary") + if err != nil { + return -1, unix.Stat_t{}, false, closeRejectedSystemBinaryDescriptor(deps, targetFD, err) + } + return targetFD, targetStat, true, nil +} + +func createSystemBinaryTemp(deps systemBinaryInstallDeps, binFD int) (string, int, error) { + for range tempNameAttempts { + name, err := deps.randomName() + if err != nil { + return "", -1, systemBinaryInstallError("generate system binary temporary name", err) + } + fd, err := deps.openAt(binFD, name, tempInstallFlags, 0600) + if errors.Is(err, unix.EEXIST) { + continue + } + if err != nil { + return "", -1, systemBinaryInstallError("create system binary temporary file", err) + } + return name, fd, nil + } + return "", -1, systemBinaryInstallError( + "create system binary temporary file", + fmt.Errorf("exclusive name attempts exhausted"), + ) +} + +func copySystemBinaryImage( + ctx context.Context, + deps systemBinaryInstallDeps, + sourceFD int, + tempFD int, +) error { + buffer := make([]byte, systemBinaryCopyChunk) + for { + if err := ctx.Err(); err != nil { + return err + } + readCount, readErr := deps.read(sourceFD, buffer) + if readCount < 0 || readCount > len(buffer) { + return fmt.Errorf("invalid read count %d", readCount) + } + written := 0 + for written < readCount { + if err := ctx.Err(); err != nil { + return err + } + writeCount, writeErr := deps.write(tempFD, buffer[written:readCount]) + if writeCount < 0 || writeCount > readCount-written { + return fmt.Errorf("invalid write count %d", writeCount) + } + written += writeCount + if writeErr != nil { + return writeErr + } + if writeCount == 0 { + return io.ErrShortWrite + } + } + if readErr != nil { + if errors.Is(readErr, io.EOF) { + return nil + } + return readErr + } + if readCount == 0 { + return nil + } + } +} + +func verifyInstalledSystemBinaryDescriptor( + deps systemBinaryInstallDeps, + fd int, + policy systemBinaryInstallPolicy, +) (unix.Stat_t, error) { + stat, err := inspectSystemBinaryInstallFile(deps, fd, "system binary temporary file") + if err != nil { + return unix.Stat_t{}, err + } + if stat.Uid != policy.expectedUID || stat.Gid != policy.expectedGID || + stat.Mode&07777 != infra.ExecutablePerm { + return unix.Stat_t{}, systemBinaryInstallError( + "inspect system binary temporary file", + fmt.Errorf("must be root:root with mode 0755"), + ) + } + return stat, nil +} + +func abortSystemBinaryInstall( + deps systemBinaryInstallDeps, + binFD int, + tempFD int, + tempName string, + tempOpen bool, + cause error, +) error { + result := cause + if tempOpen { + if err := deps.close(tempFD); err != nil { + result = joinSystemBinaryInstallError(result, "close incomplete system binary temporary file", err) + } + } + if err := deps.unlinkAt(binFD, tempName, 0); err != nil { + result = joinSystemBinaryInstallError(result, "remove incomplete system binary temporary file", err) + } + return result +} + +func closeRejectedSystemBinaryDescriptor( + deps systemBinaryInstallDeps, + fd int, + cause error, +) error { + if err := deps.close(fd); err != nil { + return joinSystemBinaryInstallError(cause, "close rejected system binary descriptor", err) + } + return cause +} + +func closeSystemBinaryDescriptors(deps systemBinaryInstallDeps, fds []int) error { + var result error + for index := len(fds) - 1; index >= 0; index-- { + if err := deps.close(fds[index]); err != nil { + result = errors.Join(result, fmt.Errorf("fd %d: %w", fds[index], err)) + } + } + return result +} + +func guardSystemBinaryInstallSudoers(deps systemBinaryInstallDeps) error { + path := infra.SudoersDropInPath() + content, err := deps.readSudoers(path) + if errors.Is(err, os.ErrNotExist) { + return nil + } + if err != nil { + return systemBinaryInstallError("read managed sudoers policy", err) + } + return validateSystemBinaryInstallSudoers(string(content)) +} + +func validateSystemBinaryInstallSudoers(content string) error { + prefix := fmt.Sprintf("%%%s ALL=(root) NOPASSWD:", infra.MVMUnixGroup) + canonicalWildcard := infra.SystemBinaryPath + " *" + markerWildcard := infra.SystemBinaryPath + " " + infra.PrivilegedProtocolMarker + " *" + for index, rawLine := range strings.Split(content, "\n") { + line := strings.TrimSpace(rawLine) + if line == "" { + continue + } + if strings.HasPrefix(line, "#") && + !strings.HasPrefix(line, "#include") && + !strings.HasPrefix(line, "#includedir") { + continue + } + if !strings.HasPrefix(line, prefix) { + return unrecognizedSystemBinaryInstallSudoers(index+1, line) + } + commandList := strings.TrimSpace(strings.TrimPrefix(line, prefix)) + if commandList == "" { + return unrecognizedSystemBinaryInstallSudoers(index+1, line) + } + for _, rawCommand := range strings.Split(commandList, ",") { + command := strings.TrimSpace(rawCommand) + if command == canonicalWildcard || command == markerWildcard || + slices.Contains(infra.PrivilegedBinariesOrdered[:], command) { + continue + } + return unrecognizedSystemBinaryInstallSudoers(index+1, line) + } + } + return nil +} + +func unrecognizedSystemBinaryInstallSudoers(lineNumber int, line string) *errs.DomainError { + // EUID 0 only identifies the resulting process identity. It cannot prove + // that sudo required a password rather than honoring an imported or indirect rule. + return errs.New( + errs.CodePrivilegeSudoers, + fmt.Sprintf( + "refusing to install %s because %s contains unrecognized active syntax on line %d; "+ + "the process cannot prove whether sudo password authentication was used. "+ + "In an authenticated root session remove the legacy policy, then run "+ + "'sudo host install-system', followed by 'sudo %s host init'", + infra.SystemBinaryPath, + infra.SudoersDropInPath(), + lineNumber, + infra.SystemBinaryPath, + ), + errs.WithClass(errs.ClassNeedsInteraction), + errs.WithDetails(map[string]any{"sudoers_line": lineNumber, "sudoers_syntax": line}), + ) +} + +func annotateSystemBinaryReplacement(err error, durabilityUncertain bool) *errs.DomainError { + domainErr := errs.AsDomainError(err) + if domainErr == nil { + domainErr = systemBinaryInstallError("system binary replacement", err) + } + if domainErr.Details == nil { + domainErr.Details = make(map[string]any) + } + domainErr.Details["system_binary_replaced"] = true + if durabilityUncertain { + domainErr.Details["durability_uncertain"] = true + } + return domainErr +} + +func systemBinaryInstallError(message string, err error) *errs.DomainError { + return errs.WrapMsg( + errs.CodeHostInitFailed, + message+": "+err.Error(), + err, + errs.WithClass(errs.ClassInternal), + ) +} + +func joinSystemBinaryInstallError(primary error, message string, err error) *errs.DomainError { + if primary == nil { + return systemBinaryInstallError(message, err) + } + return errs.WrapMsg( + errs.CodeHostInitFailed, + primary.Error()+"; "+message+": "+err.Error(), + errors.Join(primary, err), + errs.WithClass(errs.ClassInternal), + ) +} + +func randomSystemBinaryTempName() (string, error) { + var random [16]byte + if _, err := rand.Read(random[:]); err != nil { + return "", err + } + return ".mvm-install-" + hex.EncodeToString(random[:]) + ".tmp", nil +} + +func realSystemBinaryInstallDeps() systemBinaryInstallDeps { + return systemBinaryInstallDeps{ + effectiveUID: os.Geteuid, + readSudoers: os.ReadFile, + open: unix.Open, + openAt: unix.Openat, + mkdirAt: unix.Mkdirat, + fstat: unix.Fstat, + fchown: unix.Fchown, + fchmod: unix.Fchmod, + read: unix.Read, + write: unix.Write, + fsync: unix.Fsync, + close: unix.Close, + renameAt: unix.Renameat, + unlinkAt: unix.Unlinkat, + randomName: randomSystemBinaryTempName, + } +} diff --git a/internal/core/host/system_binary_test.go b/internal/core/host/system_binary_test.go new file mode 100644 index 00000000..03e2fe4f --- /dev/null +++ b/internal/core/host/system_binary_test.go @@ -0,0 +1,840 @@ +package host + +import ( + "context" + "crypto/sha256" + "os" + "path/filepath" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + "golang.org/x/sys/unix" + + "mvmctl/internal/infra" + "mvmctl/pkg/errs" +) + +const ( + installRootFD = 40 + installSourceFD = 41 + installUsrFD = 42 + installLocalFD = 43 + installBinFD = 44 + installTargetFD = 45 + installTempFD = 46 +) + +func TestInstallSystemBinary_RequiresRoot(t *testing.T) { + fixture := newSystemBinaryInstallFixture() + fixture.deps.effectiveUID = func() int { return 1000 } + + changed, err := installSystemBinary(context.Background(), fixture.deps) + require.Error(t, err) + assert.False(t, changed) + assert.Equal(t, errs.CodePrivilegeRequired, errs.AsDomainError(err).Code) + assert.Empty(t, fixture.openCalls) +} + +func TestInstallSystemBinary_RejectsLegacyNonCanonicalSudoersWildcardBeforeInstall(t *testing.T) { + fixture := newSystemBinaryInstallFixture() + readPaths := []string{} + fixture.deps.readSudoers = func(path string) ([]byte, error) { + readPaths = append(readPaths, path) + return []byte("%mvm ALL=(root) NOPASSWD: /home/operator/mvm *\n"), nil + } + + changed, err := installSystemBinary(context.Background(), fixture.deps) + require.Error(t, err) + assert.False(t, changed) + assert.Equal(t, []string{infra.SudoersDropInPath()}, readPaths) + assert.Empty(t, fixture.openCalls, "legacy policy must be refused before installer descriptors or writes") + assert.ErrorContains(t, err, "cannot prove whether sudo password authentication was used") + assert.ErrorContains(t, err, "host install-system") +} + +func TestValidateSystemBinaryInstallSudoers(t *testing.T) { + tests := []struct { + name string + content string + wantErr bool + }{ + {name: "empty", content: ""}, + {name: "comments only", content: "# Managed by mvmctl\n# no active policy\n"}, + {name: "generated transitional policy", content: GenerateSudoersContent(infra.MVMUnixGroup)}, + { + name: "current canonical wildcard", + content: "%mvm ALL=(root) NOPASSWD: " + infra.PrivilegedBinariesOrdered[0] + + ", " + infra.SystemBinaryPath + " *\n", + }, + { + name: "recognized marker wildcard", + content: "%mvm ALL=(root) NOPASSWD: " + infra.SystemBinaryPath + + " " + infra.PrivilegedProtocolMarker + " *\n", + }, + { + name: "comment containing old example is inert", + content: "# old example: /home/operator/mvm *\n", + }, + { + name: "user executable wildcard", + content: "%mvm ALL=(root) NOPASSWD: /home/operator/mvm *\n", + wantErr: true, + }, + { + name: "relative traversal-looking executable wildcard", + content: "%mvm ALL=(root) NOPASSWD: /usr/local/bin/../share/mvm *\n", + wantErr: true, + }, + { + name: "wildcard after line continuation", + content: "%mvm ALL=(root) NOPASSWD: /usr/bin/ip, \\\n /tmp/mvm *\n", + wantErr: true, + }, + { + name: "arbitrary tool wildcard", + content: "%mvm ALL=(root) NOPASSWD: /usr/bin/env *\n", + wantErr: true, + }, + { + name: "unsupported privileged marker wildcard", + content: "%mvm ALL=(root) NOPASSWD: " + infra.SystemBinaryPath + + " __mvm_privileged_v999 *\n", + wantErr: true, + }, + { + name: "command alias wildcard", + content: "Cmnd_Alias MVM = /opt/mvm *\n%mvm ALL=(root) NOPASSWD: MVM\n", + wantErr: true, + }, + { + name: "no whitespace after sudoers tag", + content: "%mvm ALL=(root) NOPASSWD:/home/operator/mvm *\n", + wantErr: true, + }, + { + name: "no whitespace after command alias assignment", + content: "Cmnd_Alias MVM=/opt/mvm *\n%mvm ALL=(root) NOPASSWD: MVM\n", + wantErr: true, + }, + { + name: "wildcard embedded in executable path", + content: "%mvm ALL=(root) NOPASSWD: /home/*/mvm\n", + wantErr: true, + }, + { + name: "at include directive", + content: "@include /home/operator/legacy-sudoers\n", + wantErr: true, + }, + { + name: "legacy hash include directive", + content: "#include /home/operator/legacy-sudoers\n", + wantErr: true, + }, + { + name: "alias without inline wildcard", + content: "Cmnd_Alias MVM = /opt/mvm\n%mvm ALL=(root) NOPASSWD: MVM\n", + wantErr: true, + }, + { + name: "unknown command without wildcard", + content: "%mvm ALL=(root) NOPASSWD: /opt/mvm host init\n", + wantErr: true, + }, + { + name: "canonical command granted to wrong group", + content: "%wheel ALL=(root) NOPASSWD: " + infra.SystemBinaryPath + " *\n", + wantErr: true, + }, + { + name: "unknown defaults directive", + content: "Defaults env_keep += \"MVM_*\"\n", + wantErr: true, + }, + { + name: "escaped legacy command", + content: "%mvm ALL=(root) NOPASSWD: /home/operator/mvm\\ *\n", + wantErr: true, + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + err := validateSystemBinaryInstallSudoers(tt.content) + if tt.wantErr { + require.Error(t, err) + assert.NotNil(t, errs.AsDomainError(err)) + return + } + require.NoError(t, err) + }) + } +} + +func TestInstallSystemBinary_InstallsPinnedRunningImage(t *testing.T) { + fixture := newSystemBinaryInstallFixture() + fixture.targetExists = false + + changed, err := installSystemBinary(context.Background(), fixture.deps) + require.NoError(t, err) + assert.True(t, changed) + assert.Equal(t, fixture.sourceData, fixture.targetData) + assert.Equal(t, uint32(0), fixture.stats[installTargetFD].Uid) + assert.Equal(t, uint32(0), fixture.stats[installTargetFD].Gid) + assert.Equal(t, uint32(unix.S_IFREG|0755), fixture.stats[installTargetFD].Mode) + assert.True(t, fixture.renamed) + assert.Contains(t, fixture.syncedFDs, installTempFD) + assert.Contains(t, fixture.syncedFDs, installBinFD) + assert.Empty(t, fixture.tempData) + assertDomainErrorsOnly(t, err) +} + +// Rationale: A host init executed from the already-installed image must not +// replace that image or create a temporary file when its metadata is exact. +func TestInstallSystemBinary_SameInodeAndMetadataIsNoOp(t *testing.T) { + fixture := newSystemBinaryInstallFixture() + targetStat := fixture.stats[installTargetFD] + targetStat.Dev = fixture.stats[installSourceFD].Dev + targetStat.Ino = fixture.stats[installSourceFD].Ino + fixture.stats[installTargetFD] = targetStat + + changed, err := installSystemBinary(context.Background(), fixture.deps) + require.NoError(t, err) + assert.False(t, changed) + assert.False(t, fixture.tempCreated) + assert.False(t, fixture.renamed) +} + +func TestInstallSystemBinary_RejectsUnsafeTargetTypes(t *testing.T) { + tests := map[string]uint32{ + "symlink": unix.S_IFLNK | 0777, + "directory": unix.S_IFDIR | 0755, + "fifo": unix.S_IFIFO | 0600, + } + for name, mode := range tests { + t.Run(name, func(t *testing.T) { + fixture := newSystemBinaryInstallFixture() + stat := fixture.stats[installTargetFD] + stat.Mode = mode + fixture.stats[installTargetFD] = stat + + changed, err := installSystemBinary(context.Background(), fixture.deps) + require.Error(t, err) + assert.False(t, changed) + assert.ErrorContains(t, err, "regular file") + assert.False(t, fixture.tempCreated) + assertDomainErrorsOnly(t, err) + }) + } +} + +func TestInstallSystemBinary_RejectsUnsafeAncestors(t *testing.T) { + tests := map[string]struct { + fd int + mode uint32 + uid uint32 + }{ + "non_root_owned_root": {fd: installRootFD, mode: unix.S_IFDIR | 0755, uid: 1000}, + "writable_usr": {fd: installUsrFD, mode: unix.S_IFDIR | 0775, uid: 0}, + "writable_local": {fd: installLocalFD, mode: unix.S_IFDIR | 0757, uid: 0}, + "non_directory_bin": {fd: installBinFD, mode: unix.S_IFREG | 0755, uid: 0}, + } + for name, tc := range tests { + t.Run(name, func(t *testing.T) { + fixture := newSystemBinaryInstallFixture() + stat := fixture.stats[tc.fd] + stat.Mode = tc.mode + stat.Uid = tc.uid + fixture.stats[tc.fd] = stat + + changed, err := installSystemBinary(context.Background(), fixture.deps) + require.Error(t, err) + assert.False(t, changed) + assert.False(t, fixture.tempCreated) + assertDomainErrorsOnly(t, err) + }) + } +} + +// Rationale: The installer may create only the policy-approved local/bin +// suffix. Creating /usr would let an unexpected filesystem layout become a +// trusted executable root. +func TestInstallSystemBinary_CreatesOnlyAllowedMissingDirectories(t *testing.T) { + t.Run("missing_usr_is_refused", func(t *testing.T) { + fixture := newSystemBinaryInstallFixture() + fixture.usrExists = false + + changed, err := installSystemBinary(context.Background(), fixture.deps) + require.Error(t, err) + assert.False(t, changed) + assert.Empty(t, fixture.mkdirCalls) + assertDomainErrorsOnly(t, err) + }) + + t.Run("missing_local_and_bin_are_created_exactly", func(t *testing.T) { + fixture := newSystemBinaryInstallFixture() + fixture.localExists = false + fixture.binExists = false + fixture.targetExists = false + + changed, err := installSystemBinary(context.Background(), fixture.deps) + require.NoError(t, err) + assert.True(t, changed) + assert.Equal(t, []installAtCall{ + {dirFD: installUsrFD, name: "local", mode: 0755}, + {dirFD: installLocalFD, name: "bin", mode: 0755}, + }, fixture.mkdirCalls) + for _, fd := range []int{installLocalFD, installBinFD} { + stat := fixture.stats[fd] + assert.Equal(t, uint32(0), stat.Uid) + assert.Equal(t, uint32(0), stat.Gid) + assert.Equal(t, uint32(unix.S_IFDIR|0755), stat.Mode) + } + assert.Contains(t, fixture.syncedFDs, installUsrFD) + assert.Contains(t, fixture.syncedFDs, installLocalFD) + }) + + t.Run("concurrently_created_local_is_validated_not_mutated", func(t *testing.T) { + fixture := newSystemBinaryInstallFixture() + fixture.localExists = false + fixture.mkdirRaceName = "local" + fixture.targetExists = false + + changed, err := installSystemBinary(context.Background(), fixture.deps) + require.NoError(t, err) + assert.True(t, changed) + assert.NotContains(t, fixture.chownedFDs, installLocalFD) + assert.NotContains(t, fixture.chmoddedFDs, installLocalFD) + }) +} + +// Rationale: read(2) and write(2) may make short progress without an error; +// truncating the trusted executable would leave host init apparently green. +func TestInstallSystemBinary_HandlesPartialReadsAndWrites(t *testing.T) { + fixture := newSystemBinaryInstallFixture() + fixture.targetExists = false + fixture.maxRead = 3 + fixture.maxWrite = 2 + + changed, err := installSystemBinary(context.Background(), fixture.deps) + require.NoError(t, err) + assert.True(t, changed) + assert.Equal(t, fixture.sourceData, fixture.targetData) + assert.Greater(t, fixture.readCalls, 1) + assert.Greater(t, fixture.writeCalls, 1) +} + +// Rationale: Every failure before rename must preserve the previous trusted +// executable and remove the incomplete sibling temporary file. +func TestInstallSystemBinary_PreRenameFailuresPreserveTarget(t *testing.T) { + tests := map[string]func(*systemBinaryInstallFixture){ + "read": func(f *systemBinaryInstallFixture) { f.failRead = true }, + "write": func(f *systemBinaryInstallFixture) { f.failWrite = true }, + "fchown": func(f *systemBinaryInstallFixture) { f.failChownFD = installTempFD }, + "fchmod": func(f *systemBinaryInstallFixture) { f.failChmodFD = installTempFD }, + "file_fsync": func(f *systemBinaryInstallFixture) { + f.failSyncFD = installTempFD + }, + "temp_close": func(f *systemBinaryInstallFixture) { + f.failCloseFD = installTempFD + }, + "rename": func(f *systemBinaryInstallFixture) { f.failRename = true }, + } + for name, inject := range tests { + t.Run(name, func(t *testing.T) { + fixture := newSystemBinaryInstallFixture() + original := append([]byte(nil), fixture.targetData...) + inject(fixture) + + changed, err := installSystemBinary(context.Background(), fixture.deps) + require.Error(t, err) + assert.False(t, changed) + assert.Equal(t, original, fixture.targetData) + assert.False(t, fixture.renamed) + assert.True(t, fixture.tempUnlinked) + assertDomainErrorsOnly(t, err) + }) + } +} + +func TestInstallSystemBinary_ReportsPostRenameAndCleanupFailures(t *testing.T) { + tests := map[string]struct { + inject func(*systemBinaryInstallFixture) + wantRenamed bool + wantChanged bool + wantDurabilityUncertain bool + wantErr string + }{ + "directory_fsync": { + inject: func(f *systemBinaryInstallFixture) { f.failSyncFD = installBinFD }, + wantRenamed: true, + wantChanged: true, + wantDurabilityUncertain: true, + wantErr: "sync system binary directory", + }, + "retained_descriptor_close": { + inject: func(f *systemBinaryInstallFixture) { f.failCloseFD = installSourceFD }, + wantRenamed: true, + wantChanged: true, + wantErr: "close system binary installer descriptor", + }, + "directory_fsync_and_retained_descriptor_close": { + inject: func(f *systemBinaryInstallFixture) { + f.failSyncFD = installBinFD + f.failCloseFD = installSourceFD + }, + wantRenamed: true, + wantChanged: true, + wantDurabilityUncertain: true, + wantErr: "close system binary installer descriptor", + }, + "cleanup_unlink": { + inject: func(f *systemBinaryInstallFixture) { + f.failWrite = true + f.failUnlink = true + }, + wantRenamed: false, + wantErr: "remove incomplete system binary temporary file", + }, + } + for name, tc := range tests { + t.Run(name, func(t *testing.T) { + fixture := newSystemBinaryInstallFixture() + tc.inject(fixture) + + changed, err := installSystemBinary(context.Background(), fixture.deps) + require.Error(t, err) + assert.Equal(t, tc.wantChanged, changed) + assert.Equal(t, tc.wantRenamed, fixture.renamed) + assert.ErrorContains(t, err, tc.wantErr) + domainErr := errs.AsDomainError(err) + require.NotNil(t, domainErr) + if tc.wantRenamed { + assert.Equal(t, true, domainErr.Details["system_binary_replaced"]) + } + if tc.wantDurabilityUncertain { + assert.Equal(t, true, domainErr.Details["durability_uncertain"]) + } + assertDomainErrorsOnly(t, err) + }) + } +} + +// Rationale: The real descriptor-relative implementation must work on an +// isolated filesystem root without exposing a caller-selected production path. +func TestInstallSystemBinary_RealFilesystem(t *testing.T) { + newPolicy := func(root string) systemBinaryInstallPolicy { + return systemBinaryInstallPolicy{ + rootPath: root, + expectedUID: uint32(os.Geteuid()), + expectedGID: uint32(os.Getegid()), + } + } + newDeps := func(t *testing.T) systemBinaryInstallDeps { + t.Helper() + deps := realSystemBinaryInstallDeps() + deps.effectiveUID = func() int { return 0 } + deps.readSudoers = func(path string) ([]byte, error) { + assert.Equal(t, infra.SudoersDropInPath(), path) + return nil, os.ErrNotExist + } + return deps + } + + t.Run("copies current image atomically with exact metadata", func(t *testing.T) { + root := t.TempDir() + require.NoError(t, os.Mkdir(filepath.Join(root, "usr"), 0755)) + + changed, err := installSystemBinaryWithPolicy( + context.Background(), + newDeps(t), + newPolicy(root), + ) + require.NoError(t, err) + assert.True(t, changed) + + targetPath := filepath.Join(root, "usr", "local", "bin", "mvm") + source, err := os.ReadFile("/proc/self/exe") + require.NoError(t, err) + target, err := os.ReadFile(targetPath) + require.NoError(t, err) + assert.Equal(t, sha256.Sum256(source), sha256.Sum256(target)) + var targetStat unix.Stat_t + require.NoError(t, unix.Stat(targetPath, &targetStat)) + assert.Equal(t, uint32(unix.S_IFREG|0755), targetStat.Mode) + assert.Equal(t, uint32(os.Geteuid()), targetStat.Uid) + assert.Equal(t, uint32(os.Getegid()), targetStat.Gid) + }) + + t.Run("rejects target symlink", func(t *testing.T) { + root := t.TempDir() + binDir := filepath.Join(root, "usr", "local", "bin") + require.NoError(t, os.MkdirAll(binDir, 0755)) + require.NoError(t, os.Symlink("/tmp/untrusted-mvm", filepath.Join(binDir, "mvm"))) + + changed, err := installSystemBinaryWithPolicy( + context.Background(), + newDeps(t), + newPolicy(root), + ) + require.Error(t, err) + assert.False(t, changed) + assert.ErrorContains(t, err, "regular file") + }) +} + +func TestInstallSystemBinary_CancellationDuringCopyPreservesTarget(t *testing.T) { + ctx, cancel := context.WithCancel(context.Background()) + fixture := newSystemBinaryInstallFixture() + fixture.maxRead = 2 + originalRead := fixture.deps.read + fixture.deps.read = func(fd int, dst []byte) (int, error) { + n, err := originalRead(fd, dst) + cancel() + return n, err + } + original := append([]byte(nil), fixture.targetData...) + + changed, err := installSystemBinary(ctx, fixture.deps) + require.Error(t, err) + assert.False(t, changed) + assert.ErrorIs(t, err, context.Canceled) + assert.Equal(t, original, fixture.targetData) + assert.False(t, fixture.renamed) + assert.True(t, fixture.tempUnlinked) + assertDomainErrorsOnly(t, err) +} + +// Rationale: Cancellation after the file is durable but before rename must +// still leave the prior executable intact. +func TestInstallSystemBinary_CancellationImmediatelyBeforeRenamePreservesTarget(t *testing.T) { + ctx, cancel := context.WithCancel(context.Background()) + fixture := newSystemBinaryInstallFixture() + originalClose := fixture.deps.close + fixture.deps.close = func(fd int) error { + err := originalClose(fd) + if fd == installTempFD { + cancel() + } + return err + } + original := append([]byte(nil), fixture.targetData...) + + changed, err := installSystemBinary(ctx, fixture.deps) + require.Error(t, err) + assert.False(t, changed) + assert.ErrorIs(t, err, context.Canceled) + assert.Equal(t, original, fixture.targetData) + assert.False(t, fixture.renamed) + assert.True(t, fixture.tempUnlinked) +} + +// Rationale: A premature 0,nil read must not promote a truncated executable; +// the pinned source descriptor's size is the independent completion check. +func TestInstallSystemBinary_RejectsPrematureSourceEOF(t *testing.T) { + fixture := newSystemBinaryInstallFixture() + fixture.deps.read = func(int, []byte) (int, error) { return 0, nil } + original := append([]byte(nil), fixture.targetData...) + + changed, err := installSystemBinary(context.Background(), fixture.deps) + require.Error(t, err) + assert.False(t, changed) + assert.ErrorContains(t, err, "size") + assert.Equal(t, original, fixture.targetData) + assert.False(t, fixture.renamed) + assert.True(t, fixture.tempUnlinked) +} + +// Rationale: Metadata checks followed by absolute-path reopen would re-create +// the symlink race this installer is intended to close. +func TestInstallSystemBinary_UsesOnlyPinnedDescriptorRelativeTargets(t *testing.T) { + fixture := newSystemBinaryInstallFixture() + fixture.targetExists = false + + changed, err := installSystemBinary(context.Background(), fixture.deps) + require.NoError(t, err) + assert.True(t, changed) + assert.Equal(t, []installOpenCall{ + {path: "/", flags: directoryInstallFlags}, + {path: "/proc/self/exe", flags: sourceInstallFlags}, + }, fixture.openCalls) + assert.Contains(t, fixture.openAtCalls, installOpenAtCall{ + dirFD: installBinFD, name: "mvm", flags: targetInspectFlags, + }) + assert.Contains(t, fixture.openAtCalls, installOpenAtCall{ + dirFD: installBinFD, name: fixture.tempName, flags: tempInstallFlags, mode: 0600, + }) + require.Len(t, fixture.renameCalls, 1) + assert.Equal(t, installRenameCall{ + oldDirFD: installBinFD, oldName: fixture.tempName, + newDirFD: installBinFD, newName: "mvm", + }, fixture.renameCalls[0]) +} + +func assertDomainErrorsOnly(t *testing.T, err error) { + t.Helper() + if err == nil { + return + } + require.NotNil(t, errs.AsDomainError(err), "all installer errors must be DomainError") +} + +type installOpenCall struct { + path string + flags int +} + +type installOpenAtCall struct { + dirFD int + name string + flags int + mode uint32 +} + +type installAtCall struct { + dirFD int + name string + mode uint32 +} + +type installRenameCall struct { + oldDirFD int + oldName string + newDirFD int + newName string +} + +type systemBinaryInstallFixture struct { + deps systemBinaryInstallDeps + + usrExists bool + localExists bool + binExists bool + targetExists bool + tempCreated bool + tempUnlinked bool + renamed bool + + stats map[int]unix.Stat_t + sourceData []byte + targetData []byte + tempData []byte + readOffset int + maxRead int + maxWrite int + readCalls int + writeCalls int + tempName string + + failRead bool + failWrite bool + failRename bool + failUnlink bool + failChownFD int + failChmodFD int + failSyncFD int + failCloseFD int + mkdirRaceName string + + openCalls []installOpenCall + openAtCalls []installOpenAtCall + mkdirCalls []installAtCall + renameCalls []installRenameCall + unlinked []installAtCall + syncedFDs []int + closedFDs []int + chownedFDs []int + chmoddedFDs []int +} + +func newSystemBinaryInstallFixture() *systemBinaryInstallFixture { + fixture := &systemBinaryInstallFixture{ + usrExists: true, localExists: true, binExists: true, targetExists: true, + stats: map[int]unix.Stat_t{ + installRootFD: {Mode: unix.S_IFDIR | 0755, Uid: 0, Gid: 0, Dev: 1, Ino: 1}, + installSourceFD: {Mode: unix.S_IFREG | 0700, Uid: 1000, Gid: 1000, Dev: 1, Ino: 2}, + installUsrFD: {Mode: unix.S_IFDIR | 0755, Uid: 0, Gid: 0, Dev: 1, Ino: 3}, + installLocalFD: {Mode: unix.S_IFDIR | 0755, Uid: 0, Gid: 0, Dev: 1, Ino: 4}, + installBinFD: {Mode: unix.S_IFDIR | 0755, Uid: 0, Gid: 0, Dev: 1, Ino: 5}, + installTargetFD: {Mode: unix.S_IFREG | 0755, Uid: 0, Gid: 0, Dev: 1, Ino: 6}, + installTempFD: {Mode: unix.S_IFREG | 0600, Uid: 0, Gid: 0, Dev: 1, Ino: 7}, + }, + sourceData: []byte("new trusted mvm image"), + targetData: []byte("old trusted mvm image"), + tempName: ".mvm-install-test.tmp", + } + sourceStat := fixture.stats[installSourceFD] + sourceStat.Size = int64(len(fixture.sourceData)) + fixture.stats[installSourceFD] = sourceStat + fixture.deps = systemBinaryInstallDeps{ + effectiveUID: func() int { return 0 }, + readSudoers: func(string) ([]byte, error) { return nil, os.ErrNotExist }, + open: func(path string, flags int, _ uint32) (int, error) { + fixture.openCalls = append(fixture.openCalls, installOpenCall{path: path, flags: flags}) + switch path { + case "/": + return installRootFD, nil + case "/proc/self/exe": + return installSourceFD, nil + default: + return -1, unix.ENOENT + } + }, + openAt: func(dirFD int, name string, flags int, mode uint32) (int, error) { + fixture.openAtCalls = append(fixture.openAtCalls, installOpenAtCall{ + dirFD: dirFD, name: name, flags: flags, mode: mode, + }) + switch { + case dirFD == installRootFD && name == "usr": + if !fixture.usrExists { + return -1, unix.ENOENT + } + return installUsrFD, nil + case dirFD == installUsrFD && name == "local": + if !fixture.localExists { + return -1, unix.ENOENT + } + return installLocalFD, nil + case dirFD == installLocalFD && name == "bin": + if !fixture.binExists { + return -1, unix.ENOENT + } + return installBinFD, nil + case dirFD == installBinFD && name == "mvm": + if !fixture.targetExists { + return -1, unix.ENOENT + } + return installTargetFD, nil + case dirFD == installBinFD && name == fixture.tempName: + if fixture.tempCreated { + return -1, unix.EEXIST + } + fixture.tempCreated = true + return installTempFD, nil + default: + return -1, unix.ENOENT + } + }, + mkdirAt: func(dirFD int, name string, mode uint32) error { + fixture.mkdirCalls = append(fixture.mkdirCalls, installAtCall{dirFD: dirFD, name: name, mode: mode}) + switch { + case dirFD == installUsrFD && name == "local": + fixture.localExists = true + case dirFD == installLocalFD && name == "bin": + fixture.binExists = true + default: + return unix.EPERM + } + if name == fixture.mkdirRaceName { + return unix.EEXIST + } + return nil + }, + fstat: func(fd int, stat *unix.Stat_t) error { + value, ok := fixture.stats[fd] + if !ok { + return unix.EBADF + } + *stat = value + return nil + }, + fchown: func(fd, uid, gid int) error { + fixture.chownedFDs = append(fixture.chownedFDs, fd) + if fd == fixture.failChownFD { + return unix.EPERM + } + stat := fixture.stats[fd] + stat.Uid = uint32(uid) + stat.Gid = uint32(gid) + fixture.stats[fd] = stat + return nil + }, + fchmod: func(fd int, mode uint32) error { + fixture.chmoddedFDs = append(fixture.chmoddedFDs, fd) + if fd == fixture.failChmodFD { + return unix.EPERM + } + stat := fixture.stats[fd] + stat.Mode = stat.Mode&unix.S_IFMT | mode + fixture.stats[fd] = stat + return nil + }, + read: func(fd int, dst []byte) (int, error) { + fixture.readCalls++ + if fixture.failRead { + return 0, unix.EIO + } + if fd != installSourceFD { + return 0, unix.EBADF + } + if fixture.readOffset == len(fixture.sourceData) { + return 0, nil + } + limit := len(dst) + if fixture.maxRead > 0 && fixture.maxRead < limit { + limit = fixture.maxRead + } + n := copy(dst[:limit], fixture.sourceData[fixture.readOffset:]) + fixture.readOffset += n + return n, nil + }, + write: func(fd int, src []byte) (int, error) { + fixture.writeCalls++ + if fixture.failWrite { + return 0, unix.ENOSPC + } + if fd != installTempFD { + return 0, unix.EBADF + } + limit := len(src) + if fixture.maxWrite > 0 && fixture.maxWrite < limit { + limit = fixture.maxWrite + } + fixture.tempData = append(fixture.tempData, src[:limit]...) + stat := fixture.stats[installTempFD] + stat.Size = int64(len(fixture.tempData)) + fixture.stats[installTempFD] = stat + return limit, nil + }, + fsync: func(fd int) error { + fixture.syncedFDs = append(fixture.syncedFDs, fd) + if fd == fixture.failSyncFD { + return unix.EIO + } + return nil + }, + close: func(fd int) error { + fixture.closedFDs = append(fixture.closedFDs, fd) + if fd == fixture.failCloseFD { + return unix.EIO + } + return nil + }, + renameAt: func(oldDirFD int, oldName string, newDirFD int, newName string) error { + fixture.renameCalls = append(fixture.renameCalls, installRenameCall{ + oldDirFD: oldDirFD, oldName: oldName, newDirFD: newDirFD, newName: newName, + }) + if fixture.failRename { + return unix.EIO + } + fixture.renamed = true + fixture.targetExists = true + fixture.targetData = append(fixture.targetData[:0], fixture.tempData...) + fixture.stats[installTargetFD] = fixture.stats[installTempFD] + fixture.tempData = nil + return nil + }, + unlinkAt: func(dirFD int, name string, flags int) error { + fixture.unlinked = append(fixture.unlinked, installAtCall{dirFD: dirFD, name: name, mode: uint32(flags)}) + if fixture.failUnlink { + return unix.EIO + } + fixture.tempUnlinked = true + fixture.tempData = nil + return nil + }, + randomName: func() (string, error) { return fixture.tempName, nil }, + } + return fixture +} diff --git a/internal/core/host/system_binary_verify.go b/internal/core/host/system_binary_verify.go new file mode 100644 index 00000000..2098a92e --- /dev/null +++ b/internal/core/host/system_binary_verify.go @@ -0,0 +1,75 @@ +package host + +import ( + "fmt" + + "mvmctl/internal/infra" +) + +func verifyCanonicalSystemBinaryForSudoers() error { + return verifySystemBinaryForSudoers( + realSystemBinaryInstallDeps(), + productionSystemBinaryInstallPolicy(), + ) +} + +func verifySystemBinaryForSudoers( + deps systemBinaryInstallDeps, + policy systemBinaryInstallPolicy, +) (returnErr error) { + retainedFDs := make([]int, 0, 5) + defer func() { + if closeErr := closeSystemBinaryDescriptors(deps, retainedFDs); closeErr != nil { + returnErr = joinSystemBinaryInstallError( + returnErr, + "close canonical system binary verifier descriptor", + closeErr, + ) + } + }() + + rootFD, err := deps.open(policy.rootPath, directoryInstallFlags, 0) + if err != nil { + return systemBinaryInstallError("open filesystem root for sudoers verification", err) + } + retainedFDs = append(retainedFDs, rootFD) + if err := verifySystemBinaryInstallDirectory(deps, rootFD, "/", policy); err != nil { + return err + } + + usrFD, err := openSystemBinaryInstallDirectory(deps, rootFD, "usr", false, policy) + if err != nil { + return err + } + retainedFDs = append(retainedFDs, usrFD) + localFD, err := openSystemBinaryInstallDirectory(deps, usrFD, "local", false, policy) + if err != nil { + return err + } + retainedFDs = append(retainedFDs, localFD) + binFD, err := openSystemBinaryInstallDirectory(deps, localFD, "bin", false, policy) + if err != nil { + return err + } + retainedFDs = append(retainedFDs, binFD) + + targetFD, targetStat, exists, err := inspectExistingSystemBinary(deps, binFD) + if err != nil { + return err + } + if !exists { + return systemBinaryInstallError( + "verify canonical system binary for sudoers", + fmt.Errorf("%s does not exist", infra.SystemBinaryPath), + ) + } + retainedFDs = append(retainedFDs, targetFD) + if targetStat.Uid != policy.expectedUID || targetStat.Gid != policy.expectedGID || + targetStat.Mode&07777 != infra.ExecutablePerm { + return systemBinaryInstallError( + "verify canonical system binary for sudoers", + fmt.Errorf("%s must be root:root with mode 0755", infra.SystemBinaryPath), + ) + } + return nil +} diff --git a/internal/core/host/utils.go b/internal/core/host/utils.go index ad4df5e2..87af0eb2 100644 --- a/internal/core/host/utils.go +++ b/internal/core/host/utils.go @@ -142,10 +142,9 @@ func ValidateSudoersBinaries() error { // --- GenerateSudoersContent --- // Map iteration order is random; iterate over an ordered key slice instead. func GenerateSudoersContent(groupName string) string { - // Service binaries via "mvm run " pattern (sudoers wildcard) - // Use the current binary's path so sudoers matches how provisioner invokes it. - mvmPath, _ := os.Executable() - runCmd := mvmPath + " *" + // IMPORTANT: This wildcard and the raw tool entries remain transitional + // until Tasks 4-8 migrate their live callsites to typed privileged actions. + runCmd := infra.SystemBinaryPath + " *" binaries := append(infra.PrivilegedBinariesOrdered[:], runCmd) binariesStr := strings.Join(binaries, ", ") return fmt.Sprintf( diff --git a/internal/core/host/utils_test.go b/internal/core/host/utils_test.go index 2f72cfeb..f0d46f06 100644 --- a/internal/core/host/utils_test.go +++ b/internal/core/host/utils_test.go @@ -1,6 +1,7 @@ package host_test import ( + "os" "strings" "testing" @@ -35,6 +36,20 @@ func TestGenerateSudoersContent(t *testing.T) { } }) + // Rationale: During Tasks 2-8 the legacy raw-tool grants must remain live, + // but sudo must stop targeting whichever user/development image ran init. + t.Run("uses_fixed_system_binary_with_transitional_entries", func(t *testing.T) { + got := host.GenerateSudoersContent("mvm") + currentExecutable, err := os.Executable() + require.NoError(t, err) + + assert.Contains(t, got, infra.SystemBinaryPath+" *") + assert.NotContains(t, got, currentExecutable+" *") + for _, binary := range infra.PrivilegedBinariesOrdered { + assert.Contains(t, got, binary, "transitional sudoers must retain %s", binary) + } + }) + t.Run("contains_managed_header", func(t *testing.T) { got := host.GenerateSudoersContent("mvm") assert.Contains(t, got, "# Managed by mvmctl") diff --git a/internal/core/update/service.go b/internal/core/update/service.go index 65b4c565..62a5b51c 100644 --- a/internal/core/update/service.go +++ b/internal/core/update/service.go @@ -15,17 +15,20 @@ import ( "mvmctl/internal/lib/version" "mvmctl/internal/infra" + "mvmctl/pkg/errs" ) // Service handles checking for and applying updates to the mvm binary. type Service struct { - gh *download.Remote + gh *download.Remote + executable func() (string, error) } // NewService creates a new update Service for the mvmctl GitHub repo. func NewService() *Service { return &Service{ - gh: download.NewGitHub(infra.MvmctlGitHubRepo), + gh: download.NewGitHub(infra.MvmctlGitHubRepo), + executable: os.Executable, } } @@ -57,6 +60,27 @@ func (s *Service) Check(ctx context.Context) (*CheckResult, error) { // Apply downloads and installs the latest version. func (s *Service) Apply(ctx context.Context, force bool) error { + currentPath, err := s.executable() + if err != nil { + return errs.WrapMsg( + errs.CodeValidationFailed, + "detect current binary path: "+err.Error(), + err, + errs.WithClass(errs.ClassInternal), + ) + } + if filepath.Clean(currentPath) == infra.SystemBinaryPath { + return errs.New( + errs.CodeValidationFailed, + fmt.Sprintf( + "self-update cannot replace the system installation at %s; an administrator must run the new "+ + "trusted artifact through 'sudo host install-system'", + infra.SystemBinaryPath, + ), + errs.WithClass(errs.ClassValidation), + ) + } + rel, err := s.gh.LatestRelease(ctx) if err != nil { return fmt.Errorf("apply update: %w", err) @@ -90,12 +114,6 @@ func (s *Service) Apply(ctx context.Context, force bool) error { return fmt.Errorf("no binary asset found for %s (assets: %d)", assetName, len(rel.Assets)) } - // Get current binary path - currentPath, err := os.Executable() - if err != nil { - return fmt.Errorf("detect current binary path: %w", err) - } - // Ensure directory is writable dir := filepath.Dir(currentPath) if err := os.MkdirAll(dir, 0755); err != nil { diff --git a/internal/core/update/system_binary_test.go b/internal/core/update/system_binary_test.go new file mode 100644 index 00000000..3deb3fcf --- /dev/null +++ b/internal/core/update/system_binary_test.go @@ -0,0 +1,41 @@ +package update + +import ( + "context" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "mvmctl/internal/infra" + "mvmctl/pkg/errs" +) + +// Rationale: A group-authorized user must never replace the root-owned trust +// anchor through the ordinary self-update flow. A nil remote proves rejection +// happens before release lookup or any download. +func TestApply_RejectsSystemInstallationBeforeRemoteAccess(t *testing.T) { + service := &Service{ + executable: func() (string, error) { return infra.SystemBinaryPath, nil }, + gh: nil, + } + + err := service.Apply(context.Background(), false) + require.Error(t, err) + assert.Equal(t, errs.CodeValidationFailed, errs.AsDomainError(err).Code) + assert.ErrorContains(t, err, "sudo host install-system") +} + +// Rationale: Lexical aliases of the canonical path identify the same target; +// an exact string comparison would let self-update replace the trust anchor. +func TestApply_RejectsCleanedSystemInstallationAliasBeforeRemoteAccess(t *testing.T) { + service := &Service{ + executable: func() (string, error) { return "/usr/local/bin/../bin/mvm", nil }, + gh: nil, + } + + err := service.Apply(context.Background(), false) + require.Error(t, err) + assert.Equal(t, errs.CodeValidationFailed, errs.AsDomainError(err).Code) + assert.ErrorContains(t, err, "sudo host install-system") +} diff --git a/internal/infra/constants.go b/internal/infra/constants.go index 45479cf1..7c26da8e 100644 --- a/internal/infra/constants.go +++ b/internal/infra/constants.go @@ -24,6 +24,9 @@ const CLIName = "mvm" // SystemBinaryPath is the canonical root-owned executable used for privileged dispatch. const SystemBinaryPath = "/usr/local/bin/mvm" +// PrivilegedProtocolMarker selects the current fixed internal privileged protocol. +const PrivilegedProtocolMarker = "__mvm_privileged_v1" + // MvmctlGitHubRepo is the GitHub repository for mvmctl self-update. const MvmctlGitHubRepo = "AlanD20/mvmctl" diff --git a/internal/service/privileged/protocol.go b/internal/service/privileged/protocol.go index e608084f..c2da6d4e 100644 --- a/internal/service/privileged/protocol.go +++ b/internal/service/privileged/protocol.go @@ -13,7 +13,6 @@ import ( const ( markerPrefix = "__mvm_privileged_v" - protocolVersion = "1" maxActionBytes = 64 maxRequestBytes = 64 * 1024 maxJSONDepth = 32 @@ -23,7 +22,7 @@ const ( ) // Marker selects the current internal privileged protocol before normal CLI initialization. -const Marker = markerPrefix + protocolVersion +const Marker = infra.PrivilegedProtocolMarker type invocation struct { action string diff --git a/internal/testutil/mock_host_api.go b/internal/testutil/mock_host_api.go index a4263c29..43185aeb 100644 --- a/internal/testutil/mock_host_api.go +++ b/internal/testutil/mock_host_api.go @@ -10,6 +10,7 @@ import ( // MockHostAPI implements api.HostAPI for testing. type MockHostAPI struct { + HostInstallSystemBinaryFunc func(ctx context.Context) (bool, error) HostInitFunc func(ctx context.Context, onProgress event.OnProgressCallback) (any, error) HostGetStateFunc func(ctx context.Context) (*model.HostStateItem, error) HostDetectResourcesFunc func(ctx context.Context) (*model.HostResources, error) @@ -27,6 +28,13 @@ type MockHostAPI struct { HostCheckReadinessFunc func(ctx context.Context) *model.ProbeResult } +func (m *MockHostAPI) HostInstallSystemBinary(ctx context.Context) (bool, error) { + if m.HostInstallSystemBinaryFunc != nil { + return m.HostInstallSystemBinaryFunc(ctx) + } + return false, nil +} + func (m *MockHostAPI) HostInit(ctx context.Context, onProgress event.OnProgressCallback) (any, error) { if m.HostInitFunc != nil { return m.HostInitFunc(ctx, onProgress) diff --git a/pkg/api/host.go b/pkg/api/host.go index 2daa066d..ef421b44 100644 --- a/pkg/api/host.go +++ b/pkg/api/host.go @@ -26,6 +26,7 @@ import ( // HostAPI defines the public interface for host operations. type HostAPI interface { + HostInstallSystemBinary(ctx context.Context) (bool, error) HostInit(ctx context.Context, onProgress event.OnProgressCallback) (any, error) HostGetState(ctx context.Context) (*model.HostStateItem, error) HostDetectResources(ctx context.Context) (*model.HostResources, error) @@ -43,6 +44,12 @@ type HostAPI interface { HostCheckReadiness(ctx context.Context) *model.ProbeResult } +// HostInstallSystemBinary explicitly installs or upgrades the administrator-owned +// system image. It intentionally performs no host database or configuration work. +func (op *Operation) HostInstallSystemBinary(ctx context.Context) (bool, error) { + return op.hostSystemBinaryInstaller.InstallSystemBinary(ctx) +} + // HostInit initializes host configuration. // returns NeedsInteraction directly // (not wrapped in OperationResult.Item) when elevated privileges are required. @@ -72,10 +79,6 @@ func (op *Operation) HostInit(ctx context.Context, onProgress event.OnProgressCa }, }, nil } - // Ensure DB schema exists before any DB writes. - if op.Connection != nil { - _, _ = op.Connection.RunMigrationsCtx(ctx) - } if !system.IsRoot() { hasGroup := system.SessionHasGroup() return &errs.NeedsInteraction{ @@ -89,6 +92,11 @@ func (op *Operation) HostInit(ctx context.Context, onProgress event.OnProgressCa }, }, nil } + // Ensure DB schema exists before host-state writes. Sudoers is reconciled + // later, after fallible read-only probes and before host mutations. + if op.Connection != nil { + _, _ = op.Connection.RunMigrationsCtx(ctx) + } // --- Pre-flight probes --- // Run detection first, then probe against the detected state (verdict #53). hardware, detErr := host.DetectHardware() @@ -136,6 +144,9 @@ func (op *Operation) HostInit(ctx context.Context, onProgress event.OnProgressCa // --- Setup host environment --- allChanges, err := op.hostInitSetupEnvironment(ctx, sessionID, hostCtrl, fwBackend) if err != nil { + if errs.AsDomainError(err) != nil { + return nil, err + } return nil, errs.New(errs.CodeHostInitFailed, err.Error()) } if err := op.reconcileServiceAccessPolicies(ctx); err != nil { @@ -172,8 +183,30 @@ func (op *Operation) hostInitSetupEnvironment( hostCtrl *host.Controller, fwBackend string, ) ([]*model.HostStateChangeItem, error) { - allChanges := make([]*model.HostStateChangeItem, 0) - dbChanges := make([]*model.HostStateChangeItem, 0) + username, err := system.CurrentUsername() + if err != nil { + return nil, err + } + sudoersConfigured, err := op.hostSudoersConfigurer.ConfigureSudoers(ctx) + if err != nil { + return nil, err + } + initialChanges := make([]*model.HostStateChangeItem, 0, 1) + if sudoersConfigured { + initialChanges = append(initialChanges, &model.HostStateChangeItem{ + SessionID: "", + InitTimestamp: "", + Setting: "sudoers_dropin", + Mechanism: "file_create", + AppliedValue: infra.SudoersDropInPath(), + Reverted: false, + ChangeOrder: 0, + CreatedAt: "", + OriginalValue: nil, + }) + } + allChanges := append([]*model.HostStateChangeItem(nil), initialChanges...) + dbChanges := append([]*model.HostStateChangeItem(nil), initialChanges...) // --- Group setup --- groupCreated, _ := host.CreateGroup(ctx, infra.MVMUnixGroup) if groupCreated { @@ -185,10 +218,6 @@ func (op *Operation) hostInitSetupEnvironment( dbChanges = append(dbChanges, change) allChanges = append(allChanges, change) } - username, err := system.CurrentUsername() - if err != nil { - return allChanges, err - } userAdded, _ := host.AddUserToGroup(ctx, username, infra.MVMUnixGroup) if userAdded { change := &model.HostStateChangeItem{ @@ -199,23 +228,6 @@ func (op *Operation) hostInitSetupEnvironment( dbChanges = append(dbChanges, change) allChanges = append(allChanges, change) } - // --- Sudoers setup --- - sudoersPath := infra.SudoersDropInPath() - sudoersContent := host.GenerateSudoersContent(infra.MVMUnixGroup) - sudoersStale := true - if data, err := os.ReadFile(sudoersPath); err == nil { - sudoersStale = string(data) != sudoersContent - } - if sudoersStale { - _ = host.WriteSudoers(ctx, sudoersPath, sudoersContent) - change := &model.HostStateChangeItem{ - SessionID: "", Setting: "sudoers_dropin", - Mechanism: "file_create", AppliedValue: sudoersPath, - InitTimestamp: "", OriginalValue: nil, Reverted: false, ChangeOrder: 0, CreatedAt: "", - } - dbChanges = append(dbChanges, change) - allChanges = append(allChanges, change) - } // --- IP forwarding --- fwdChange, _ := host.EnableIPForward(ctx) if fwdChange != nil { @@ -249,7 +261,7 @@ func (op *Operation) hostInitSetupEnvironment( slog.Warn("Could not update host state", "error", err) } } - if sudoersStale { + if sudoersConfigured { if err := op.Repos.Host.UpdateComponent(ctx, "sudoers_configured", true); err != nil { slog.Warn("Could not update host state", "error", err) } diff --git a/pkg/api/host_system_binary_test.go b/pkg/api/host_system_binary_test.go new file mode 100644 index 00000000..53deac4e --- /dev/null +++ b/pkg/api/host_system_binary_test.go @@ -0,0 +1,82 @@ +package api + +import ( + "context" + "os/user" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "mvmctl/internal/lib/system" + "mvmctl/internal/testutil" + "mvmctl/pkg/errs" +) + +func TestHostInstallSystemBinary_PreservesPartialSuccess(t *testing.T) { + wantErr := errs.New( + errs.CodeHostInitFailed, + "system binary replaced but directory durability is uncertain", + errs.WithDetails(map[string]any{ + "system_binary_replaced": true, + "durability_uncertain": true, + }), + ) + installer := &fakeSystemBinaryInstaller{changed: true, err: wantErr} + op := &Operation{hostSystemBinaryInstaller: installer} + + changed, err := op.HostInstallSystemBinary(context.Background()) + require.ErrorIs(t, err, wantErr) + assert.True(t, changed) + assert.Equal(t, 1, installer.calls) +} + +// Rationale: HostInit may perform read-only probes before sudoers reconciliation, +// but a sudoers failure must precede group membership or any other host mutation. +func TestHostInitSetup_SudoersFailurePrecedesHostMutations(t *testing.T) { + originalOS := system.DefaultOS + originalRunner := system.DefaultRunner + t.Cleanup(func() { + system.DefaultOS = originalOS + system.DefaultRunner = originalRunner + }) + system.DefaultOS = &testutil.FakeOS{ + CurrentFn: func() (*user.User, error) { + return &user.User{Username: "operator", Uid: "1000", Gid: "1000"}, nil + }, + } + fakeRunner := &testutil.FakeRunner{} + system.DefaultRunner = fakeRunner + + wantErr := errs.New(errs.CodePrivilegeSudoers, "injected sudoers configuration failure") + configurer := &fakeHostSudoersConfigurer{err: wantErr} + op := &Operation{hostSudoersConfigurer: configurer} + + changes, err := op.hostInitSetupEnvironment(context.Background(), "session", nil, "nftables") + require.ErrorIs(t, err, wantErr) + assert.Empty(t, changes) + assert.Equal(t, 1, configurer.calls) + assert.Empty(t, fakeRunner.Calls, "group, user, sysctl, and network mutations must not start") +} + +type fakeSystemBinaryInstaller struct { + changed bool + err error + calls int +} + +func (f *fakeSystemBinaryInstaller) InstallSystemBinary(context.Context) (bool, error) { + f.calls++ + return f.changed, f.err +} + +type fakeHostSudoersConfigurer struct { + changed bool + err error + calls int +} + +func (f *fakeHostSudoersConfigurer) ConfigureSudoers(context.Context) (bool, error) { + f.calls++ + return f.changed, f.err +} diff --git a/pkg/api/operation.go b/pkg/api/operation.go index 1ab4451a..0395548c 100644 --- a/pkg/api/operation.go +++ b/pkg/api/operation.go @@ -42,6 +42,17 @@ type Operation struct { Services Services ProvisionerType provisioner.ProvisionerType AuditLog *logging.AuditLog + + hostSystemBinaryInstaller hostSystemBinaryInstaller + hostSudoersConfigurer hostSudoersConfigurer +} + +type hostSystemBinaryInstaller interface { + InstallSystemBinary(ctx context.Context) (bool, error) +} + +type hostSudoersConfigurer interface { + ConfigureSudoers(ctx context.Context) (bool, error) } // Repos bundles all database repositories. @@ -151,10 +162,12 @@ func NewOperation(ctx context.Context, conn *db.Handle, cacheDir string) *Operat r.Vsock, r.Snapshot, ), - Repos: r, - Services: s, - ProvisionerType: provisionerType, - AuditLog: logging.NewAuditLog(), + Repos: r, + Services: s, + ProvisionerType: provisionerType, + AuditLog: logging.NewAuditLog(), + hostSystemBinaryInstaller: s.Host, + hostSudoersConfigurer: s.Host, } } diff --git a/tasks/plan.md b/tasks/plan.md index 9de73f3e..4923ed9a 100644 --- a/tasks/plan.md +++ b/tasks/plan.md @@ -287,9 +287,11 @@ go test ./... -count=1 -coverprofile=coverage.out -covermode=atomic 1. Preserve the current base/head commit IDs and all unrelated dirty work before changing history. 2. Add forward-only migrations for exact-pair fields and uniqueness. Never rewrite an already-applied user DB in place. -3. `mvm host init` detects the insecure legacy sudoers rule, installs the release artifact atomically as root-owned - `/usr/local/bin/mvm`, validates the new sudoers rule with `visudo`, then replaces the old rule. On failure it keeps the - previous configuration and reports the unsafe state. +3. In an authenticated administrator session, remove the insecure project-managed legacy sudoers rule, then run the + trusted release artifact's exact `host install-system` route. Installation is atomic at root-owned + `/usr/local/bin/mvm` and is deliberately separate from `host init`. After all callers use typed privileged actions, + `sudo /usr/local/bin/mvm host init` validates the marker-only policy with `visudo` and atomically replaces the old + rule. On failure it keeps the previous configuration and reports the unsafe state. 4. An unsupported privileged protocol refuses the operation and prints the explicit administrator upgrade command. 5. Existing VMs with an unambiguous exact pair are backfilled. Ambiguous/unmatched VMs remain stopped and return a migration error; they are never launched with an arbitrary pair. diff --git a/tasks/todo.md b/tasks/todo.md index d5e8b6ca..dbca0477 100644 --- a/tasks/todo.md +++ b/tasks/todo.md @@ -63,14 +63,14 @@ typed operation catalog. Supersede the conflicting portions of ADR-0002, ADR-000 initialization. **Acceptance criteria:** -- [ ] `/usr/local/bin/mvm` is installed as `root:root`, mode `0755`, under a root-owned non-writable directory chain. -- [ ] Installation is temporary-file + fsync + rename and refuses symlinked targets. -- [ ] A normal user update cannot replace the system binary; unsupported privileged protocol fails closed with an +- [x] `/usr/local/bin/mvm` is installed as `root:root`, mode `0755`, under a root-owned non-writable directory chain. +- [x] Installation is temporary-file + fsync + rename and refuses symlinked targets. +- [x] A normal user update cannot replace the system binary; unsupported privileged protocol fails closed with an administrator upgrade instruction. **Verification:** -- [ ] L1 install tests cover ownership, modes, symlink target, partial write, and version mismatch. -- [ ] Focused Go tests for system installation, early dispatch, and host initialization pass. +- [x] L1 install and early-protocol tests cover ownership, modes, symlink target, partial write, and version mismatch. +- [x] Focused Go tests for system installation, early dispatch, and host initialization pass. **Dependencies:** Task 1 **Files likely touched:** `cmd/mvm/main.go`, `internal/core/host/utils.go`, `internal/core/host/utils_test.go`, From c430fa0b68be2850cbe2e4926886723627d4466f Mon Sep 17 00:00:00 2001 From: AlanD20 Date: Sun, 23 Aug 2026 12:05:57 +0200 Subject: [PATCH 0010/1457] docs: define v0.3 clean-break security architecture --- CHANGELOG.md | 98 ++- docs/adr/0005-sudo-privilege-architecture.md | 11 +- .../0009-firewall-backend-mutual-exclusion.md | 6 +- .../0014-routed-service-access-policies.md | 6 +- ...owned-single-binary-privileged-dispatch.md | 42 +- ...m-network-namespaces-and-typed-policies.md | 224 ++++++ docs/improvements/IMPROVEMENTS_010.md | 6 +- docs/improvements/IMPROVEMENTS_011.md | 4 +- tasks/plan.md | 526 ++++++-------- tasks/todo.md | 675 ++++++------------ 10 files changed, 818 insertions(+), 780 deletions(-) create mode 100644 docs/adr/0017-per-vm-network-namespaces-and-typed-policies.md diff --git a/CHANGELOG.md b/CHANGELOG.md index caac8272..7d042141 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,13 +5,20 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). -## [Unreleased] +## [0.3.0] - Unreleased + +> **Security release status:** `0.3.0` is still under development. The canonical Jailer, cgroup policy, routed +> service-access policy, root-owned system installer, and early privileged-dispatch foundation described below are +> implemented. The mandatory per-VM network namespace, nftables-only `traffic` policy, exact VM-to-VM `exec` policy, +> final marker-only sudo policy, and remaining typed privileged operations are not implemented yet; see +> **Security work still pending** before treating this version as release-ready. ### Added #### Canonical Firecracker Jailer launch - VM create, start, reboot, and snapshot restore now require an exact Firecracker/Jailer release pair and run through a root-owned Jailer chroot with no direct-launch fallback. -- Trusted release pairs are installed from the checksum-verified archive; each jail exposes only that VM's kernel, rootfs, volumes, runtime files, and selected snapshot. +- Release pairs are installed from a checksum-checked archive; each jail exposes only that VM's kernel, rootfs, volumes, + runtime files, and selected snapshot. Independent privileged release verification remains a release blocker below. #### Enforced VM resource envelopes - Jailed VMs now require cgroup v2 and launch with persisted, verified CPU, memory, swap, and PID limits; `mvm vm inspect` reports requested, observed, and current cgroup state. @@ -19,6 +26,66 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 #### Routed service-access policies - `mvm policy` now persists exact network-to-VM TCP/UDP allows while both firewall backends default-deny other managed cross-network and VM-to-host traffic without blocking NoCloud, established replies, or internet egress. +#### Root-owned single-binary installation +- Added the exact administrator bootstrap command `sudo host install-system`, which installs the + same `mvm` artifact at `/usr/local/bin/mvm` as `root:root` mode `0755`. +- System installation uses descriptor-relative, no-symlink traversal; an exclusive temporary file; complete-copy and + ownership verification; file and directory `fsync`; and atomic rename. +- `host install-system` is dispatched before ordinary application initialization and ignores user cache, configuration, + database, logging, and environment overrides. +- `host init` now verifies the canonical system binary before reconciling the managed sudoers drop-in. System installation + is deliberately separate from host initialization. + +#### Privileged-dispatch foundation +- Added the reserved, versioned privileged entry point to the root-owned `mvm` binary. Selection occurs before Cobra or + normal CLI initialization and malformed reserved invocations fail closed instead of falling through. +- Added strict bounded JSON parsing with unknown, duplicate, case-colliding, trailing, oversized, and structurally invalid + input rejection. +- Added authenticated sudo identity and current `mvm` group checks, fixed root environment sanitization, and executable + device/inode verification against the root-owned `/usr/local/bin/mvm` image. +- Added ADR-0016, which defines the single-binary privilege boundary, trusted state layout, typed operation catalog, + clean-install constraints, and fail-closed system-binary replacement behavior. + +### Changed + +#### VM launch and release identity +- VM create, start, reboot, and snapshot restore no longer fall back to launching Firecracker directly when Jailer setup + fails. +- VM and snapshot state now carry the exact Firecracker/Jailer pair needed by jailed launch paths. +- Trusted release installation and Jailer launch are internal services compiled into the single `mvm` artifact. + +#### Host setup and updates +- Initial setup is now two explicit administrator actions: install the trusted system image, then run + `sudo /usr/local/bin/mvm host init`. +- A system-installed `/usr/local/bin/mvm` refuses in-place `self-update apply`. Administrators replace it by running + `host install-system` from the newly downloaded trusted artifact. +- During development, the generated sudoers policy targets `/usr/local/bin/mvm` instead of a user-owned executable. + Its wildcard and raw-tool grants remain transitional and are not the final `0.3.0` security boundary. + +#### Network isolation +- Traffic routed between different managed networks is now denied unless an explicit typed service-access policy permits + the destination VM protocol and port. Same-network bridge traffic remains unchanged. +- VM-to-host traffic is limited to established replies and required managed services such as NoCloud; internet egress + remains available. + +### Breaking changes + +- v0.3.0 requires a clean installation. It does not migrate, backfill, adopt, or preserve legacy databases, running VMs, + Jailer directories, cgroups, TAP devices, firewall state, policy rows, or remote-exec flags. Initialization must refuse + ambiguous legacy runtime state rather than silently adopting or deleting it. +- Operators must stop and remove the old installation, retain any desired user artifacts separately, clean legacy host + runtime state through the documented administrator procedure, and recreate VMs and policies under v0.3.0. +- The new administrator-controlled system installation is mandatory. A user-owned `~/.local/bin/mvm` is no longer a + valid final sudo target. +- The system binary and an invoking user binary must support the same privileged protocol version. Version mismatch stops + before side effects and requires an administrator-approved system-binary replacement. +- Jailed VM launch requires cgroup v2, the required controllers, and an exact Firecracker/Jailer release pair. Hosts that + previously relied on direct Firecracker fallback no longer start VMs through that fallback. +- Cross-network VM traffic that previously worked without an allow rule is denied after policy reconciliation. Operators + must create explicit `mvm policy` entries for required TCP or UDP services. +- Scripts that expect `mvm self-update apply` to replace `/usr/local/bin/mvm` must switch to the administrator bootstrap + flow. Self-update remains available for user-owned candidate artifacts. + ### Fixed #### `mvm image inspect` @@ -31,6 +98,31 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - VM create no longer fails when Jailer places the cgroup at `/sys/fs/cgroup/mvmctl/`; the check previously expected an intermediate `firecracker/` component that Jailer v1.16 does not create. - Stop and remove now clean the VM's actual leaf cgroup instead of leaving stale ones behind. +### Security work still pending + +The following items are release blockers and are intentionally not described above as completed behavior: + +- Bind every privileged VM action to a root-owned global VM-ID ownership claim, lifecycle record, exact process identity, + and per-VM lock. +- Add root-owned managed-network identity and global capacity authority; user SQLite state must not authorize host-global + namespace, link, firewall, process, mount, cgroup, or admission decisions. +- Replace path-validation-then-reopen behavior with descriptor-pinned managed-resource access and private mount namespaces. +- Independently verify and atomically install trusted Firecracker/Jailer releases inside the privileged boundary. +- Migrate Jailer, loopmount, network, firewall, and supported host mutations to distinct typed privileged actions; remove + the public root `mvm run jailer` and `mvm run provision` entry points. +- Make one root-owned network namespace mandatory per VM, pass its pinned handle to Jailer, and make namespace/link + creation, activation, rollback, reconciliation, and cleanup part of the owner-bound VM lifecycle. +- Remove the iptables backend and backend selector; apply one atomic nftables generation that default-denies both + same-network and cross-network VM traffic while preserving required system traffic and internet egress. +- Replace routed `ServiceAccessPolicy` with typed `traffic` policies and remove the `allow_remote_exec` trust mesh in + favor of exact directional, non-root, resource-bounded VM-to-VM `exec` policies. +- Replace the transitional wildcard/raw-tool sudoers entries with access only to the reserved privileged marker of the + root-owned system binary. +- Make launch, abort, cleanup, reconciliation, snapshots, live volumes, cgroups, and firewall policy crash-consistent and + fail closed under ownership, PID-reuse, path-race, and partial-failure conditions. +- Complete adversarial L1 coverage, Python `tests/system/` clean-install/fault qualification, leak audits, full CI, and + the final documentation accuracy review. Go tests support but do not replace CLI-level L2 release signoff. + ## [0.2.0] - 2026-07-10 ### Added @@ -333,6 +425,6 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - System tests run in nested VM with unprivileged user - Coverage matrix tracking every CLI subcommand and flag -[Unreleased]: https://github.com/AlanD20/mvmctl/compare/v0.2.0...HEAD +[0.3.0]: https://github.com/AlanD20/mvmctl/compare/v0.2.0...HEAD [0.2.0]: https://github.com/AlanD20/mvmctl/releases/tag/v0.2.0 [0.1.0]: https://github.com/AlanD20/mvmctl/releases/tag/v0.1.0 diff --git a/docs/adr/0005-sudo-privilege-architecture.md b/docs/adr/0005-sudo-privilege-architecture.md index a09f4011..b9deaa24 100644 --- a/docs/adr/0005-sudo-privilege-architecture.md +++ b/docs/adr/0005-sudo-privilege-architecture.md @@ -1,12 +1,13 @@ # Sudo Privilege Architecture: Group-Based Passwordless Access -**Status:** Active +**Status:** Superseded for v0.3.0 by +[ADR-0016: Root-Owned Single-Binary Privileged Dispatch](0016-root-owned-single-binary-privileged-dispatch.md) **Date:** 2026-05-22 **Last Updated:** 2026-06-20 (Go implementation complete) > **Replacement accepted, implementation in progress:** ADR-0016 replaces the user-owned wildcard sudo design with one > root-owned system installation and an early typed privileged dispatcher. The system installer and no-action dispatcher -> foundation are implemented. Raw tool grants and the public-CLI wildcard below remain transitional until Tasks 4-9 and +> foundation are implemented. Raw tool grants and the public-CLI wildcard below remain transitional until Tasks 4-14 and > the L2 release gate pass; they must not be described as the final security boundary. The project uses a **tightly scoped, group-based sudo model** where members of a dedicated Unix group (`mvm`) can execute specific privileged commands via `sudo` without a password prompt. This avoids running the entire CLI as root while still allowing VM creation (loop-mount), network management (ip/iptables/nft), kernel module loading (modprobe), and sysctl configuration. @@ -39,8 +40,8 @@ Key implications: ## The Actual Sudoers File -`mvm host init` creates `/etc/sudoers.d/mvm` (path resolved by `infra.SudoersDropInPath()`). During the migration the file -is generated by `GenerateSudoersContent()` with the fixed root-owned system path: +`mvm host init` creates `/etc/sudoers.d/mvm` (path resolved by `infra.SudoersDropInPath()`). In the current development +baseline, the file is generated by `GenerateSudoersContent()` with the fixed root-owned system path: ``` %mvm ALL=(root) NOPASSWD: /usr/sbin/ip, /usr/sbin/iptables, /usr/sbin/iptables-restore, /usr/sbin/iptables-save, /usr/sbin/nft, /usr/sbin/sysctl, /usr/sbin/modprobe, /usr/local/bin/mvm * @@ -57,7 +58,7 @@ The sudoers file is generated by `GenerateSudoersContent()` using two sources fr - `/usr/sbin/sysctl` (procps) - `/usr/sbin/modprobe` (kmod) -- **Service binary (transitional)** — fixed `/usr/local/bin/mvm *` for privileged public service subcommands. Task 9 +- **Service binary (transitional)** — fixed `/usr/local/bin/mvm *` for privileged public service subcommands. Task 14 replaces this wildcard with only `/usr/local/bin/mvm __mvm_privileged_v1 *` after every caller is typed. NOTHING else is permitted. `sudo -n true`, `sudo cat /etc/sudoers.d/mvm`, `sudo test -f ...`, and any other non-listed command WILL prompt for a password. This is by design and is NOT a configuration problem. diff --git a/docs/adr/0009-firewall-backend-mutual-exclusion.md b/docs/adr/0009-firewall-backend-mutual-exclusion.md index 028881d4..cf7421cf 100644 --- a/docs/adr/0009-firewall-backend-mutual-exclusion.md +++ b/docs/adr/0009-firewall-backend-mutual-exclusion.md @@ -1,8 +1,12 @@ # Firewall Backend Mutual Exclusion — nftables vs iptables -**Status:** Active +**Status:** Superseded for v0.3.0 by +[ADR-0017: Per-VM Network Namespaces and Typed Connectivity Policies](0017-per-vm-network-namespaces-and-typed-policies.md) **Date:** 2026-05-22 +> This ADR remains the description of the currently implemented transitional dual-backend behavior. v0.3.0 removes +> iptables and the backend selector after the nftables-only path and its L2 qualification are complete. + The project provides two independent firewall rule tracking backends: **nftables** (default) and **iptables** (legacy). These backends are **mutually exclusive** — a single session uses exactly one backend, never a combination. The `firewall_backend` setting acts as a toggle selector. **Table of Contents** diff --git a/docs/adr/0014-routed-service-access-policies.md b/docs/adr/0014-routed-service-access-policies.md index 65a16ed6..060b6904 100644 --- a/docs/adr/0014-routed-service-access-policies.md +++ b/docs/adr/0014-routed-service-access-policies.md @@ -1,9 +1,13 @@ # Routed Service-Access Policies -**Status:** Active +**Status:** Superseded for v0.3.0 by +[ADR-0017: Per-VM Network Namespaces and Typed Connectivity Policies](0017-per-vm-network-namespaces-and-typed-policies.md) **Date:** 2026-08-09 **See also:** [Project Hosting with MicroVMs](../roadmaps/TENANT_MICROVM.md), [ADR-0009: Firewall Backend Mutual Exclusion](0009-firewall-backend-mutual-exclusion.md) +> This ADR remains the description of the currently implemented transitional routed-only policy. v0.3.0 replaces it +> with mandatory per-VM namespaces, same- and cross-network default deny, and typed `traffic` and `exec` policies. + ## Context mvmctl networks are independent Linux bridges and subnets. VMs on one bridge communicate directly at layer 2, while traffic between bridges is routed by the host because IPv4 forwarding is enabled. Before this decision was implemented, the managed firewall permitted outbound NAT and installed no default deny between mvmctl bridges. diff --git a/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md b/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md index ae8881d1..01179cc1 100644 --- a/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md +++ b/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md @@ -40,7 +40,7 @@ is an explicit administrator action; a normal user update cannot overwrite it. Development builds may exist elsewhere, but they are never sudo targets. A development build that needs privileged operations calls the compatible system installation and fails closed on a privileged-protocol mismatch. -### Administrator bootstrap and upgrade +### Administrator bootstrap and system-binary replacement System installation is selected by the exact `host install-system` command before `app.Initialize()`, Cobra, signal setup, user configuration, the user database, cache resolution, logging configuration, or service construction. The @@ -49,10 +49,15 @@ running image to `/usr/local/bin/mvm`. Flags and extra arguments are rejected in fall through to the ordinary CLI. The installer cannot prove whether an already-root process reached it through password-authenticated administrator sudo -or a legacy passwordless rule. It therefore inspects the project-managed sudoers drop-in before touching the executable -and refuses unrecognized active syntax. During migration, an administrator removes the insecure legacy managed rule in -an authenticated root session, runs `sudo host install-system`, and then runs -`sudo /usr/local/bin/mvm host init`. Installation is not an implicit side effect of `host init`. +or an unintended passwordless rule. It therefore inspects the project-managed sudoers drop-in before touching the +executable and refuses unrecognized active syntax. On a clean host, an administrator runs +`sudo host install-system` and then runs `sudo /usr/local/bin/mvm host init`. Installation is not an +implicit side effect of `host init`. + +v0.3.0 does not use this installer to convert, adopt, or preserve a pre-v0.3 installation. Operators must remove the old +managed sudoers rule and clean the old runtime/state layout in an authenticated administrator session before installing +v0.3.0. Later replacement of an already-v0.3-compatible system binary remains an explicit administrator action and +must fail closed when the privileged protocol is incompatible. Replacing the executable is descriptor-relative, no-follow, temporary-file plus file fsync plus rename plus directory fsync. Failures after rename report that replacement occurred and whether directory durability is uncertain. A normal @@ -89,7 +94,7 @@ commands. The sudoers drop-in grants only the reserved privileged entry point of does not grant the public CLI or raw `ip`, `iptables`, `iptables-restore`, `iptables-save`, `nft`, `modprobe`, or `sysctl` commands. -`host init`, system-binary installation/upgrades, group membership changes, sudoers changes, and `host reset` modify the +`host init`, system-binary installation/replacement, group membership changes, sudoers changes, and `host reset` modify the authorization boundary itself. They remain explicit administrator operations under the administrator's normal sudo policy and are not exposed through the `mvm` group's passwordless dispatcher. @@ -243,27 +248,28 @@ short-lived dispatcher model. **Costs and limitations:** -- Installation and upgrades require an explicit administrator step. +- Installation and system-binary replacement require an explicit administrator step. - Development binaries cannot perform privileged work unless a compatible system binary is installed. -- All existing raw privileged callsites must migrate to typed operations before the old sudoers policy is removed. +- All raw privileged callsites must be replaced by typed operations before the final sudoers policy is enabled. - The single executable contains more code than a dedicated helper, so early dispatch and import-graph isolation require strict tests and review. - Multi-user ownership, crash recovery, release verification, namespace mounts, and firewall atomicity add implementation and L2 testing cost. -## Migration and Release Gate +## Clean Installation and Release Gate -ADR-0005 remains the description of the currently implemented legacy policy until this ADR's release gate passes. The -replacement sequence is: +ADR-0005 remains a historical description of the pre-v0.3 policy. v0.3.0 provides no compatibility or automated +migration path for that policy, its user state, or its runtime resources. The implementation and release sequence is: 1. Add early privileged dispatch, the exact administrator bootstrap route, and negative tests while the final marker-only path is unreachable from sudoers. -2. Migrate every privileged service/tool call to a typed operation. -3. In an authenticated administrator session, remove the insecure managed legacy rule, install the trusted artifact as - the root-owned system binary, and atomically activate the marker-only sudoers drop-in after `visudo` validation. -4. Run upgrade, rollback, privilege-abuse, multi-user, path-race, PID-reuse, crash-recovery, Jailer, cgroup, snapshot, - hotplug, and firewall L2 tests. -5. Mark ADR-0005 superseded, update all current-behavior documentation, and only then describe the new boundary as active. +2. Replace every supported privileged service/tool call with a typed operation. +3. On a clean host, install the trusted artifact as the root-owned system binary and atomically activate only the + marker-scoped sudoers drop-in after `visudo` validation. +4. Run installation, replacement, privilege-abuse, multi-user, path-race, PID-reuse, crash-recovery, Jailer, cgroup, + snapshot, hotplug, and firewall L2 tests through the installed binary. +5. Mark ADR-0005 superseded for v0.3.0, update all current-behavior documentation, and only then describe the new + boundary as active. No implementation phase is complete while documentation describes a different executable path, sudo policy, state location, lifecycle guarantee, or known limitation. @@ -271,6 +277,6 @@ location, lifecycle guarantee, or known limitation. ## Related Decisions - ADR-0002: One release artifact and one installed executable remain the product contract. -- ADR-0005: Legacy group-based wildcard sudo policy; superseded only after this migration's release gate. +- ADR-0005: Pre-v0.3 group-based wildcard sudo policy; superseded when the v0.3.0 release gate passes. - ADR-0012: Root boundary, namespace, cgroup, and recovery behavior require L2 verification. - ADR-0015: Jailer remains canonical; its privilege-boundary and cleanup requirements are refined here. diff --git a/docs/adr/0017-per-vm-network-namespaces-and-typed-policies.md b/docs/adr/0017-per-vm-network-namespaces-and-typed-policies.md new file mode 100644 index 00000000..162c0663 --- /dev/null +++ b/docs/adr/0017-per-vm-network-namespaces-and-typed-policies.md @@ -0,0 +1,224 @@ +# Per-VM Network Namespaces and Typed Connectivity Policies + +**Status:** Accepted — v0.3.0 implementation pending +**Date:** 2026-08-23 +**Supersedes:** [ADR-0014: Routed Service-Access Policies](0014-routed-service-access-policies.md) +**See also:** [ADR-0016: Root-Owned Single-Binary Privileged Dispatch](0016-root-owned-single-binary-privileged-dispatch.md) + +## Context + +The initial managed-network implementation creates TAP devices and Linux bridges in the host network namespace. Its +service-access policy protects routed traffic between different managed bridges, but traffic between VMs on the same +bridge remains unfiltered. Firecracker therefore shares the host network view, interface names are host-global, and +same-network placement implicitly grants connectivity. + +Canonical Jailer launch improves filesystem, process, privilege, and cgroup isolation, but Jailer does not create or +configure networking. Its `--netns` option only joins a network namespace that the trusted caller has already created. +The namespace handle and every device placed inside it are consequently privileged lifecycle resources. + +The existing VM-to-VM remote-exec model is also too broad. A boolean on each VM creates an all-to-all trust mesh among +enabled VMs, bypasses IP firewall policies through vsock, and can execute as root when no target user is supplied. + +v0.3.0 is a clean installation with no legacy database, runtime-state, policy, or firewall compatibility requirement. +This is the appropriate point to replace the host-global and dual-backend design rather than preserve it as a fallback. + +## Decision + +### One network namespace per VM + +Every networked VM receives one root-created network namespace. Jailer must join that namespace through a root-owned +namespace handle before it drops privileges and executes Firecracker. Direct launch in the host network namespace is +not supported. + +The namespace topology is fixed by mvmctl: + +```text +host namespace +└── managed network bridge + └── host veth + └── VM network namespace + ├── namespace veth + ├── local bridge + └── TAP opened by jailed Firecracker +``` + +The host bridge remains the logical managed network. Each VM namespace contains only its loopback device, its side of +the veth pair, a local bridge, and the Firecracker TAP. Interface names, namespace paths, and topology are derived from +the invoking UID and immutable resource IDs; callers cannot provide raw interface names, namespace paths, `ip` argv, or +netns commands. + +Namespace handles and locks are ephemeral root-owned state under `/run/mvmctl//`. Persistent minimal VM and network +ownership records remain under `/var/lib/mvmctl/`. The user database stores desired logical configuration but never +authorizes a root namespace, device, bridge, route, or firewall mutation. + +The privileged launch transaction must: + +1. authenticate the caller and register the durable VM launch identity while holding the global claim and VM locks; +2. acquire the later network lock and verify the root-owned VM and network records; +3. create the namespace and namespace-local devices with links down; +4. create and attach the host veth to the exact managed bridge; +5. install the complete nftables generation and anti-spoofing rules; +6. bring the verified links up; +7. pass the pinned namespace handle to Jailer; and +8. persist exact namespace and process identity before reporting the VM active. + +The complete lock order is `release -> global index -> VM -> network`. An operation may acquire a suffix or subset but +must never acquire an earlier class while holding a later one. This permits `RegisterLaunch` to return a VM-locked lease +before the first network effect without creating a two-phase ownership claim or importing network behavior into the +Jailer authority module. + +Failure after the first effect invokes typed rollback while the same locks and lifecycle handle remain held. Cleanup +verifies namespace inode, interface ownership, process identity, and cgroup membership before removing anything. + +### nftables is the only firewall backend + +v0.3.0 requires nftables. The iptables backend, configuration option, repositories, renderers, fallback behavior, and +tests are removed. + +One nftables implementation owns complete atomic generations for: + +- bridge-family same-network filtering and source MAC/IP anti-spoofing; +- IP-family routed filtering between managed networks; +- VM-to-host input restrictions; +- established reply handling and required mvmctl host services; and +- managed internet egress/NAT. + +Managed VM-to-VM traffic is default-deny whether the VMs share a network or use different networks. Required ARP and +system traffic are derived internally. Links are not made usable before the applicable deny and allow generation is +installed successfully. + +### Two public policy families + +The public policy namespace contains two typed policy families: + +```text +mvm policy traffic ... +mvm policy exec ... +``` + +There is no generic rule, action, selector registry, firewall-expression, or privileged-operation policy. + +#### Traffic policies + +A traffic policy grants TCP or UDP access to an exact destination VM port or port range. It has one of two closed, +explicit source forms: + +- source managed network to destination VM; or +- exact source VM to exact destination VM. + +The CLI may present these forms under the common `traffic` namespace, but API and network-domain interfaces use +separate typed inputs and methods. Persisted intent stores immutable resource identities, protocol, and port bounds—not +IP addresses, MAC addresses, interface names, nftables expressions, or commands. The compiler resolves current +root-verified identities and emits the required bridge- or IP-family rules. + +A traffic policy applies to the IP data plane only. It grants neither vsock execution nor host command execution. + +#### Exec policies + +An exec policy grants one exact source VM permission to request command execution on one exact destination VM through +the host vsock relay. It is directional, independent of network membership, and required for both same-network and +cross-network VMs. + +Exec policies require a non-root destination user. v0.3.0 does not provide VM-to-VM root execution or VM-to-host +execution. The relay authenticates the source from the active host-side vsock session and root-owned instance identity, +then enforces the exact source, destination, owner, and target user before contacting the destination agent. + +The relay bounds request size, frame size, output bytes, duration, and concurrent executions. Audit records contain the +source, destination, target user, timing, and result but not the complete command. An exec policy grants no TCP/UDP +connectivity. + +The legacy `allow_remote_exec` VM field, CLI flags, configuration default, environment-spec property, all-to-all checks, +and documentation are removed without conversion. + +### Layer placement + +- `pkg/api/` remains the sole orchestrator of VM and network domain identities and owns public input validation. +- `internal/core/network/` owns typed traffic intent and compilation, without importing another core domain. +- The VM-to-VM exec policy module receives already-resolved shared model identities; it does not import multiple core + domains through `internal/vsockhandler/`. +- `internal/service/network/` owns typed privileged namespace, bridge, veth, TAP, route, and link effects. +- `internal/service/firewall/` owns typed privileged nftables application. +- `internal/service/jailer/` consumes a verified namespace handle as part of typed launch; it does not create a generic + network-operation interface. +- Shared policy and resolved-runtime values live in `internal/lib/model/` only when multiple domains require them. + +All side-effecting functions take `context.Context` first and return `pkg/errs.DomainError` failures. Batch operations +report per-item results explicitly; partial success is never collapsed into a nil error. + +## Clean-Installation Contract + +v0.3.0 does not read, migrate, backfill, or preserve legacy policy rows, iptables state, host-global TAP ownership, or +`allow_remote_exec` values. Initialization refuses detected legacy runtime resources and gives the administrator a +precise cleanup command or diagnostic. It never silently adopts or deletes ambiguous host state. + +Operators must stop and remove old VMs, retain any desired user artifacts separately, clean the old host runtime, and +install v0.3.0 on a clean state layout. Only the new schema and topology are supported after initialization. + +## Alternatives Considered + +### Keep routed policies and trust same-network placement + +Rejected. Network placement would remain an implicit authorization mechanism, and same-bridge traffic would bypass the +managed IP-family policy path. + +### One namespace per managed network + +Rejected. It reduces host interface clutter but places several Firecracker processes in the same host network view. The +microVM, not the managed network, is the isolation unit. + +### Optional namespace and legacy bridge backends + +Rejected. Two lifecycle paths double the privileged attack surface, cleanup states, and system-test matrix. v0.3.0 has +no compatibility requirement that justifies retaining the weaker path. + +### CNI plugins as an additional backend + +Rejected for v0.3.0. Root-executed plugin binaries and configurable plugin chains enlarge the trusted computing base and +introduce a generic backend before a second required topology exists. CNI may be reconsidered only with a concrete use +case, root-owned verified plugins/configuration, and a separate security decision. + +### Preserve iptables parity + +Rejected. Same-network enforcement requires bridge-family filtering, while maintaining two renderers and reconciliation +paths creates security-sensitive duplication. nftables is the single supported implementation. + +### Generic policy selectors and actions + +Rejected. A generic policy engine would mix enforcement planes and make unsupported combinations appear valid. New +capabilities require new typed policy families and explicit enforcement contracts. + +## Consequences + +### Benefits + +- Firecracker no longer shares the host network namespace. +- Same-network placement grants no implicit application access. +- Policy meaning is stable across IP, MAC, veth, TAP, and bridge-name changes. +- Network and vsock authorization cannot accidentally grant one another. +- One firewall backend reduces privileged code, reconciliation branches, and qualification scope. +- Snapshot cloning and future network pre-allocation can build on an owner-bound namespace lifecycle. + +### Costs + +- Every VM adds one namespace, veth pair, and namespace-local bridge. +- Launch and cleanup perform more privileged steps and require transactional rollback. +- Hosts must provide network namespaces and nftables with bridge-family support. +- Existing installations and automation must be recreated for v0.3.0. +- Troubleshooting must distinguish guest, namespace, host bridge, routed, nftables, and vsock planes. + +## Release Gate + +This decision is not complete until L1 and L2 tests prove namespace ownership, cross-UID denial, anti-spoofing, +same-network and cross-network default deny, exact traffic allows, exec-policy directionality, non-root enforcement, +atomic reconciliation, crash recovery, reboot cleanup, and absence of leaked processes, namespaces, links, cgroups, +mounts, sockets, and nftables rules. + +## Related Decisions + +- ADR-0005: The final sudo policy exposes only the typed root-owned dispatcher. +- ADR-0009: Firewall backend mutual exclusion is superseded by the nftables-only decision here. +- ADR-0010: Cross-domain resolution remains in `pkg/api/`. +- ADR-0011: Public policy inputs retain typed `Validate` and `Resolve` behavior. +- ADR-0012: Namespace and firewall behavior require real-host L2 evidence. +- ADR-0015: Jailer remains the only Firecracker launch path. +- ADR-0016: Namespace, network, and firewall mutations use the same root-owned typed privilege boundary. diff --git a/docs/improvements/IMPROVEMENTS_010.md b/docs/improvements/IMPROVEMENTS_010.md index 61b47b6b..755f8c92 100644 --- a/docs/improvements/IMPROVEMENTS_010.md +++ b/docs/improvements/IMPROVEMENTS_010.md @@ -1,10 +1,12 @@ # Per-VM Network Namespaces with Firecracker Jailer -> **STATUS:** ⏳ Network namespaces remain future work. Canonical Jailer launch, chroot, privilege drop, trusted release pairs, and jailed snapshot paths are implemented; `--netns`, veth topology, and a `NetNS` model remain unimplemented. +> **STATUS:** Superseded as an implementation proposal by +> [ADR-0017](../adr/0017-per-vm-network-namespaces-and-typed-policies.md). Mandatory per-VM namespaces are a v0.3.0 +> release blocker, but the optional, raw-iptables, duplicate-IP topology below is not the approved implementation. > > **Last verified:** 2026-06-27 -**Phase:** Future +**Phase:** Historical proposal — do not implement directly **Complexity:** Very High **Depends on:** Firecracker jailer binary, kernel netns support diff --git a/docs/improvements/IMPROVEMENTS_011.md b/docs/improvements/IMPROVEMENTS_011.md index 9ba3141e..032ec5ed 100644 --- a/docs/improvements/IMPROVEMENTS_011.md +++ b/docs/improvements/IMPROVEMENTS_011.md @@ -1,6 +1,8 @@ # CNI Network Backend -> **STATUS: Design Proposal — not implemented.** Current network stack uses raw TAP devices + Linux bridges + nftables/iptables firewall. No CNI support exists. +> **STATUS: Rejected for v0.3.0.** [ADR-0017](../adr/0017-per-vm-network-namespaces-and-typed-policies.md) +> selects one built-in, typed namespace topology and rejects optional root-executed CNI plugin chains. This document is +> retained as historical research and must not be used as an implementation plan. > > **Last verified:** 2026-07-08 diff --git a/tasks/plan.md b/tasks/plan.md index 4923ed9a..7733d276 100644 --- a/tasks/plan.md +++ b/tasks/plan.md @@ -1,360 +1,290 @@ -# Implementation Plan: Safe Jailer and Privileged-Operations Architecture +# v0.3.0 Clean-Break Security Release Plan **Status:** Approved — implementation in progress -**Review range:** `f8ffa28b9d74cac2c9e15fa07a18389d51a3263e..ee733232a2dc23c1edb372022a2add71af36f44c` -**Release decision:** Do not push, merge, or describe the current Jailer implementation as production-ready. - -## Overview - -Replace the user-writable `mvm` sudo target with one root-owned system installation and an early privileged dispatcher; -make every privileged operation typed, owner-bound, descriptor-relative, and race-safe; then repair Jailer lifecycle, -snapshot, hotplug, cgroup, and firewall -semantics. The ordinary CLI continues to run as an unprivileged user. Persistent user state remains in the existing -per-user location for this change. Only trusted releases, privileged ownership records, chroots, cgroups, and ephemeral -locks live in system locations. - -The last five commits should be treated as a prototype branch. Remediation should land as reviewable commits on top of -the current work. History may be curated only after the implementation and L2 verification pass, and only after the -existing dirty worktree has been preserved. - -## Scope - -### Required before release - -- Replace every passwordless-sudo entry that can execute user-controlled code or arbitrary system-tool arguments. -- Establish an independent trust anchor for Firecracker/Jailer release installation. -- Remove path validation followed by privileged pathname reopen. -- Give each privileged VM operation an authenticated owner, exact release, process identity, and per-VM lock. -- Put launch mounts in a private namespace and post-launch mounts in the running VM's verified mount namespace. -- Make create, start, stop, remove, snapshot, restore, and volume hotplug crash-consistent. -- Make routed-policy reconciliation fail closed, including resource deletion and IP reuse. -- Execute the prepared L2 tests plus adversarial privilege, race, and fault-injection coverage. -- Rewrite documentation only after the final behavior passes the release gates. - -### Explicitly separate - -- Moving the user database and durable user state out of `~/.cache/mvmctl` is desirable but is not part of this - privilege remediation. It requires a separate ADR, compatibility migration, and rollback plan. -- PID and lock files may move to `/run`; the canonical database, images, snapshots, and VM data must not. -- Network namespaces remain a separate feature unless required to make the corrected Jailer lifecycle work. - -## Threat Model - -### Principals - -- A member of the `mvm` group may manage only resources authorized for that Unix UID. -- An `mvm` group member is untrusted from the privileged dispatcher's perspective and must not gain arbitrary root - execution. -- The normal `mvm` executable, its environment, stdin, arguments, cache, manifests, and database are user-controlled. -- The root-owned system binary, its privileged dispatcher, directory chain, host policy, release records, and privileged - instance records are trusted. -- Downloaded release archives are untrusted until verified inside the privileged boundary. - -### Protected assets - -- Host root filesystem, processes, mount namespaces, networking, kernel modules, sysctls, and cgroup hierarchy. -- Other users' VM data and privileged instance records. -- Exact Firecracker/Jailer release identity and installed bytes. -- Default-deny firewall behavior and IP-identity binding. -- Recoverability after process failure, CLI interruption, or host reboot. - -### Security invariants - -1. The normal CLI never becomes the passwordless root executable. -2. Sudo permits only the early privileged mode of root-owned `/usr/local/bin/mvm`; raw `ip`, `nft`, `iptables`, - `modprobe`, and `sysctl` access is removed. -3. The privileged dispatcher exposes named, typed operations. It never accepts a generic command, arbitrary argv, raw - cgroup keys, or an unchecked host path. -4. Every privileged request is bound to `SUDO_UID`, a root-owned instance record, and a fixed VM/release identity. -5. A user-controlled digest is never a trust anchor. The privileged dispatcher obtains expected release integrity data from - root-owned policy or a fixed, independently fetched official source. -6. User paths are opened once with descriptor-relative, no-symlink resolution. Privileged code operates on pinned - descriptors and never validates a path and later reopens it. -7. PID actions use a verified PID plus start time and cgroup membership, preferably a pidfd. PID reuse must fail closed. -8. Launch and cleanup are serialized by a root-owned per-VM lock. Cleanup never recursively traverses a possible mount. -9. All post-spawn failures terminate the registered process before deleting DB, VM, jail, or cgroup state. -10. Removing policy intent or a resource cannot make its IP reusable until the deny/allow kernel state is reconciled. - -## Target Architecture +**Branch:** `0.3.0-release` +**Release gate:** Draft only until every CI and Python system-test requirement below passes -```text -one root-owned /usr/local/bin/mvm - | - |-- normal CLI mode runs unprivileged - | - | typed internal request through sudo - v -early privileged dispatcher before app.Initialize/Cobra/user config - | - +-- release operations ---------- fixed origin/root policy, bounded archive parser - +-- VM launch/lifecycle --------- owner record, pidfd/start time, per-VM lock - +-- mount operations ------------ openat2 dirfds, private/setns mount namespace - +-- network/firewall operations -- typed models, atomic backend batches - +-- host operations ------------- named module/sysctl actions only - | - +-- /var/lib/mvmctl ------------- trusted releases, instance ownership, chroot dirs - +-- /run/mvmctl ----------------- ephemeral locks/handshakes - +-- /sys/fs/cgroup/mvmctl ------- kernel-enforced per-VM limits - -~/.cache/mvmctl existing unprivileged DB and VM artifacts -``` +## Outcome -The privileged mode is compiled into the same `mvm` artifact but is dispatched directly from `cmd/mvm/main.go` before -`app.Initialize()`, Cobra, the user database, configuration, or plugins are loaded. It has a deliberately small import -graph and cannot dispatch back into ordinary `mvm` commands. It verifies at startup that the executable and directory -chain are root-owned and not group/world writable. Upgrading `/usr/local/bin/mvm` requires an explicit administrator -action; a user-level update cannot replace it. +Release v0.3.0 as a clean installation with one unprivileged public CLI and one root-owned system copy of the same +`mvm` binary. All privileged effects must enter through a versioned typed dispatcher, bind to root-owned resource +authority, and fail closed under multi-user, path-race, process-reuse, partial-failure, and crash conditions. -## State Placement +Firecracker must always run through Jailer with cgroup v2 and one network namespace per VM. nftables becomes the only +firewall backend. Managed traffic is default-deny for both same-network and cross-network VMs. IP connectivity and +VM-to-VM vsock execution use separate typed `traffic` and `exec` policies. -| State | Location for this remediation | Ownership and lifetime | -|---|---|---| -| User SQLite DB, images, kernels, volumes, snapshots, VM directories | Existing `MVM_CACHE_DIR`, normally `~/.cache/mvmctl` | User-owned, persistent | -| Trusted Firecracker/Jailer releases and release manifest | `/var/lib/mvmctl/binaries/...` | Root-owned, persistent | -| Privileged VM ownership/release records | `/var/lib/mvmctl/instances//.json` | Root-owned, persistent, minimal metadata | -| Jailer chroot directories | `/var/lib/mvmctl/jailer/...` | Root-owned lifecycle state | -| Per-VM locks and launch handshakes | `/run/mvmctl//...` | Root-owned, ephemeral | -| Cgroup state | `/sys/fs/cgroup/mvmctl/` | Kernel state, runtime only | -| `/run/mvm` | Inside the jail only | Bind-mounted jail-visible view, not host canonical state | +The release does not migrate, adopt, backfill, or preserve legacy databases, runtime state, policies, remote-exec flags, +Jailer directories, cgroups, TAP devices, or firewall rules. It detects ambiguous legacy host state and refuses to +continue without silently deleting it. -The privileged instance record contains only the owning UID, VM ID, exact release hashes/IDs, process identity, and -cleanup generation. It is not a second copy of the user VM specification. +## Implemented Baseline -## Key Design Decisions +- Canonical Firecracker Jailer launch with no direct fallback. +- Exact Firecracker/Jailer pairing in VM and snapshot paths. +- Persisted and verified cgroup v2 CPU, memory, swap, and PID envelopes. +- Transitional routed cross-network service-access policy and default deny. +- Atomic root-owned installation at `/usr/local/bin/mvm`. +- Early reserved privileged dispatch before Cobra and ordinary application initialization. +- Strict bounded request parsing, sudo caller identity, group authorization, environment sanitization, and executable + identity verification. +- ADR-0016 for the single-binary privilege boundary. -### Root-owned single binary and early privileged dispatcher +The action switch is intentionally still empty and the sudoers policy remains transitional. The routed policy, dual +firewall backends, host-global TAP topology, public `mvm run jailer` / `mvm run provision` services, and +`allow_remote_exec` trust mesh must not survive the final release. -Install the release artifact through an explicit administrative bootstrap to `/usr/local/bin/mvm` as `root:root`, mode -`0755`. Normal users execute that same file without sudo. The sudoers drop-in permits only its internal privileged entry -point, which is selected before normal initialization. Each privileged operation has its own typed request and -validation. Unknown actions, extra fields, and unsupported protocol versions fail closed. +## Approved Architecture -The code continues to follow the existing layer and file split. `cmd/mvm/main.go` performs only the early branch; -`internal/service/privileged/` owns the envelope, identity checks, and fixed dispatch; Jailer, loopmount, network, and -firewall effects remain in capability-specific `internal/service/{name}/` packages; shared values live in -`internal/lib/model/`; domain logic remains in `internal/core/{domain}/`; and `pkg/api/` remains the sole cross-core -orchestrator. No generic privileged backend or raw command method is introduced. +### Trust and state -The final state removes the existing raw privileged binaries from sudoers. Merely replacing `mvm *` is insufficient: -several allowed system tools can execute or materially alter arbitrary host state when given attacker-selected arguments. +| State | Location | Authority | +|---|---|---| +| User configuration, SQLite DB, and VM artifacts | Existing `MVM_CACHE_DIR` | Untrusted for root authorization | +| Trusted Firecracker/Jailer pairs | `/var/lib/mvmctl/binaries` | Root-owned persistent | +| VM ownership/lifecycle records | `/var/lib/mvmctl/instances/` | Root-owned persistent | +| Network ownership/topology records | `/var/lib/mvmctl/networks/` | Root-owned persistent | +| Global capacity allocations | `/var/lib/mvmctl/allocations` | Root-owned persistent | +| Locks, launch handshakes, namespace handles | `/run/mvmctl` | Root-owned ephemeral | +| Jailer chroots | `/var/lib/mvmctl/jailer` | Root-owned lifecycle state | +| Cgroups | `/sys/fs/cgroup/mvmctl/` | Kernel runtime state | + +Durable user state does not move wholesale to `/run` or `/var/run`. XDG state separation remains a separate project. + +### Module and layer rules + +- `cmd/mvm/main.go` only recognizes early modes and routes them. +- `internal/service/privileged/` owns the envelope, authenticated caller, executable verification, and fixed action + switch. It imports no API or core domain and exposes no raw command/argv/path interface. +- Capability modules under `internal/service/{jailer,network,firewall,loopmount,host}/` own typed privileged effects. +- `pkg/api/` remains the sole orchestrator of multiple core domains. +- Core domains never import sibling `internal/core/*` packages. +- Public validation remains in `pkg/api/inputs/`; privileged receivers enforce trust-boundary invariants. +- Every side-effecting function takes `context.Context` first and returns `pkg/errs.DomainError`. +- Each new capability uses named typed methods matching existing service conventions. No generic backend, operation, + selector registry, callback-under-lock, raw firewall expression, raw cgroup key, or raw netns action is permitted. + +### VM authority interface + +The first implementation slice is private to `internal/service/jailer`. It uses unexported typed values and these +operations: + +```go +func (a *instanceAuthority) RegisterLaunch( + ctx context.Context, + caller instanceCaller, + registration launchRegistration, +) (*launchLease, error) + +func (a *instanceAuthority) LockRegistered( + ctx context.Context, + caller instanceCaller, + vmID string, +) (*registeredLease, error) + +func (a *instanceAuthority) BeginCleanup( + ctx context.Context, + caller instanceCaller, + vmID string, +) (*cleanupLease, error) + +func (l *cleanupLease) Complete(ctx context.Context) error + +func (a *instanceAuthority) LockUnreferencedRelease( + ctx context.Context, + release releaseIdentity, +) (*releaseLease, error) + +func (l *launchLease) Release(ctx context.Context) error +func (l *registeredLease) Release(ctx context.Context) error +func (l *cleanupLease) Release(ctx context.Context) error +func (l *releaseLease) Release(ctx context.Context) error +``` -### Release trust +Each lease retains its pinned directory and lock descriptors and provides a checked context-first release operation. +There is no public `Get`, `Put`, `Delete`, `Run`, raw path, arbitrary state string, or callback interface. -The normal CLI may download the archive for progress and caching, but the privileged dispatcher must independently -obtain the expected -digest from a fixed official URL that it constructs from validated version/architecture, or from root-owned signed -metadata. It must not accept both an archive and its authority from the caller. +The durable lifecycle is intentionally small: -Installation uses a root-owned temporary directory, a bounded tar reader, exact member allowlists, and rejection of -links, devices, traversal, duplicate entries, oversized payloads, and unexpected files. After verification, root-side code -atomically renames and fsyncs the exact pair plus a root-owned release manifest. A forced replacement of a referenced -release is prohibited. +```text +absent ── RegisterLaunch ──> registered +cleaned ── RegisterLaunch ──> registered +registered ─ BeginCleanup ──> cleaning +cleaning ─── BeginCleanup ──> cleaning +cleaning ───── Complete ─────> cleaned +``` -If Firecracker provides a stable verifiable signature/provenance mechanism, prefer it. Otherwise the initial model may -trust the fixed HTTPS release origin and its independently fetched checksum, with that limitation recorded explicitly. +`registered` means launch identity is durable; it does not claim the process is healthy. Process health is derived by +verifying PID, start ticks, cgroup membership, release, and namespace identity. Failed cleanup remains `cleaning`. +`cleaned` is a persistent ownership tombstone and may be relaunched only by the same UID. -### Safe path and mount handling +The global lock order is: -Use `openat2`/`openat` through `golang.org/x/sys/unix` with `RESOLVE_BENEATH`, `RESOLVE_NO_SYMLINKS`, and -`RESOLVE_NO_MAGICLINKS`, followed by `fstat` owner/type/mode checks. Keep directory descriptors alive through the -operation. Fixed basenames are derived from validated IDs; no privileged write uses a caller-selected basename. +```text +release lock -> global index lock -> VM lock -> network lock +``` -The privileged launch path first creates a private mount namespace and marks it private, then creates all pre-launch bind mounts -and execs Jailer. These mounts do not leak into the host namespace and disappear when the VM namespace dies. Live -snapshot/volume operations pin their source descriptor, verify the registered Firecracker process, enter its mount -namespace in a short-lived privileged child, and mount onto a descriptor-pinned target. Mounts should use `nodev`, `nosuid`, -and `noexec` whenever the resource permits. +Operations may acquire a suffix or subset but never an earlier class while holding a later class. Global VM-ID claims +are established by descriptor-enumerating all numeric UID record directories while holding the index lock. A foreign, +duplicate, corrupt, unreadable, or inconsistent claim fails closed. Lock files are never unlinked. -### Process and cleanup lifecycle +`RegisterLaunch` establishes the durable record before the first network or launch effect and returns with the VM lock +held. A later typed network step may therefore acquire only the later network lock while retaining that launch lease. +Task 4 owns release, index, and VM locking; it does not import network code or predeclare a network implementation. -Create a root-owned instance record before the first privileged side effect. Record PID, start time, cgroup path, exact -release, and cleanup generation before reporting launch success. Use pidfds where supported and always revalidate start -time plus cgroup membership. +Atomic record replacement uses a random exclusive temporary file, bounded writes, root ownership and exact modes, file +`fsync`, checked close, `renameat`, and parent-directory `fsync`. Pre-rename failure preserves the old record. +Post-rename failure returns details that distinguish `record_replaced` and `durability_uncertain`; the caller starts +no VM effects after such an error. -Every create/restore path persists a provisional user DB row before spawn. After `Spawn`, a lifecycle handle is retained -until the DB reports `running`. Any later failure calls a typed `AbortLaunch`: TERM, bounded wait, KILL of the verified -process if necessary, cgroup-empty verification, namespace/chroot cleanup, and instance-record finalization. +### Network topology and policy -Cleanup is per-VM locked and fail closed. It never calls `RemoveAll` while mounts may exist. Startup and pre-launch -reconciliation inspect root-owned instance records, registered PIDs, cgroups, and chroot directories so interrupted -operations can be recovered without trusting the user database. +ADR-0017 is the authoritative design: -### Exact binary pair +```text +host managed bridge +└── host veth + └── per-VM network namespace + ├── namespace veth + ├── namespace-local bridge + └── TAP opened by jailed Firecracker +``` -Keep the current binary-domain shape but add typed pair operations rather than generic repository hooks: +Namespace paths and interface names are derived from UID and immutable IDs. The privileged transaction creates links +down, installs the complete nftables generation and anti-spoofing rules, then brings links up and passes a pinned +namespace handle to Jailer. -- `UpsertTrustedPair(ctx, firecracker, jailer)` in one DB transaction. -- A uniqueness invariant for active `(type, version, architecture)` records. -- `jailer_binary_id` on snapshots and all restored/migrated VMs. -- Root-owned release hashes checked against both user DB records before every launch. -- No ignored repository errors during pair install, default selection, or removal. +The public policy families are: -### Firewall policy reconciliation +- `mvm policy traffic`: TCP/UDP from a source network or exact source VM to an exact destination VM port/range. +- `mvm policy exec`: directional exact source VM to exact destination VM and required non-root target user. -Make chain/jump initialization return a result and include it in reconciliation. Build one complete desired managed -ruleset and apply it atomically through `iptables-restore` or an nftables batch. Only after the kernel update succeeds may -intent/derived rows be committed or resource identity/IP be released. If the DB commit fails, restore the prior kernel -generation and return a joined error. Startup reconciliation recompiles from authoritative intent. +Traffic policy never grants vsock execution. Exec policy never grants IP connectivity. Same-network placement grants +nothing implicitly. -## Dependency and Implementation Order +## Dependency Order ```text -ADR + threat contract +architecture + Python installed-CLI harness | v -root-owned system install + early privileged protocol +root VM authority + descriptor-safe root records/locks | - +--> Jailer/release operations --> fd-safe mounts --> lifecycle/recovery --> snapshot/hotplug + +--> descriptor-pinned user resources --> private mount namespace | - +--> loopmount/network/firewall/host privileged migration --> remove raw sudoers entries + +--> independently trusted release installation | - +--> exact-pair DB migration + v +typed Jailer lifecycle actions + remove public legacy Jailer service | - +--> fail-closed policy reconciliation v -L1 negative/fault tests --> L2 real-host matrix --> documentation sweep --> release +root network authority --> per-VM netns/veth/TAP --> nftables-only generation + | | + | v + | traffic policy replacement + | + +--> exact exec policy + bounded vsock relay + | + v +loopmount + host mutation migration --> marker-only sudoers + | + v +reconciliation + admission + snapshot/volume/cgroup parity + | + v +full CI + Python system-test qualification + documentation signoff ``` -See `tasks/todo.md` for task-level acceptance criteria, ownership, dependencies, and verification. - -## Checkpoints +The final typed Jailer path is built directly from the safe substrates. The project will not add owner checks to the +legacy public transport merely to remove that transport later. -### Checkpoint A — architecture approved +Independent trusted release installation precedes the final Jailer action and final sudoers rule. Descriptor-safe +resource access precedes every privileged handler that consumes user-owned paths. The marker-only sudoers policy is +installed only after every supported privileged caller has migrated. -- ADR defines the privileged-dispatch TCB, ownership model, release trust, state layout, and migration. -- Privileged operations are enumerated; no generic execution extension exists. -- Threat cases become named tests before production code changes. -- Human approves the plan and ADR. +## Increment and Commit Contract -### Checkpoint B — no user-controlled root entry point +Each commit must: -- Sudoers contains only the internal privileged entry point of root-owned `/usr/local/bin/mvm`. -- Privileged mode cannot invoke the general CLI or raw caller-selected commands. -- Jailer, loopmount, network/firewall, and host operations use typed privileged endpoints. -- Self-authored archives/checksums and mutable-binary replacement tests fail safely. +1. implement one complete bounded behavior; +2. add the applicable test before production behavior; +3. leave the repository compiling and existing tests passing; +4. pass focused format, golines, vet, and Go tests; +5. include only its owned files and preserve unrelated dirty work; and +6. receive architect review of actual interfaces, structs, imports, and effects before the next slice. -### Checkpoint C — lifecycle parity +Documentation and changelog changes land with the architecture or behavior they describe. They distinguish implemented +behavior from pending release blockers. No document may describe an unverified target in the present tense. -- Create/start/stop/reboot/remove, live attach/detach, snapshot create, and snapshot restore pass L1 path assertions. -- Injected failures after every spawn stage leave no live registered PID, mount, cgroup, or orphaned ownership record. -- Legacy VMs/snapshots migrate or fail with an actionable error; exact pair identity is persistent. +## Verification and QA Signoff -### Checkpoint D — real-host release gate +Go L0/L1 tests prove pure logic, strict codecs, descriptor behavior, lock races, state transitions, renderers, and +failure propagation. They support but do not replace release qualification. -- Full repository CI passes in the documented order. -- L2 Jailer/cgroup/policy tests pass on the runner VM. -- Multi-user, symlink-race, PID-reuse, kill-at-stage, reboot-recovery, and IP-reuse tests pass. -- VMM memory headroom and any I/O envelope are measured and documented. -- The final documentation audit contains no stale state, sudo, cgroup, service-count, or path claims. +The Python suite under `tests/system/` is the authority for operator-visible behavior. It must exercise the installed +release-candidate binary through the public CLI: -## Verification Strategy +- stage the candidate outside `/usr/local/bin`; +- run `sudo host install-system`; +- run `sudo /usr/local/bin/mvm host init`; +- perform product mutations only through `/usr/local/bin/mvm` as constrained users; +- use runner administrator access only for fixed read-only observations and controlled fault injection; and +- aggregate teardown errors and assert a zero-leak baseline after every mutating topology. -### L0/L1 +Required staged Python gates: -- Parser/authorization tables for every privileged action, including unknown/extra input. -- Release archive abuse cases and independent-digest failure. -- Descriptor-relative path resolution under concurrent rename/symlink replacement. -- PID start-time/cgroup mismatch and per-VM lock contention. -- Exact Firecracker API paths for normal launch, snapshot create/restore, and hotplug. -- Failure injection after spawn, relay creation, socket readiness, DB writes, snapshot load, and cleanup. -- Firewall parent-jump, batch-apply, DB-commit, policy-remove, VM-remove, and IP-reuse failures. -- Structured cgroup-setting fixture restoration with restoration assertions. +1. **System installation (T1):** atomic install, ownership/mode/ancestor checks, hash equality, invocation routing, + self-update refusal, and install failure safety. +2. **VM authority (T2):** record-before-effect, multi-user denial, strict privileged input, process identity, release + reference, corrupt-state refusal, cleanup, and raw-sudo denial once marker-only policy lands. +3. **Jailer/cgroup/mount (T2/T3):** canonical launch, exact pair, namespace visibility, snapshot/volume parity, + PID-reuse and kill-stage recovery. +4. **Network/traffic (T2):** netns membership, topology ownership, spoof denial, same/cross default deny, exact allows, + host restrictions, egress, atomic failure, reboot reconciliation, and nftables-only host readiness. +5. **Exec (T2):** same/cross/no-network behavior, directionality, missing policy, non-root target, source identity, + timeout/output/frame/concurrency bounds, command redaction, and target cleanup. +6. **Clean release (T2 plus clean host):** clean-install refusal of legacy state, full user journeys, repeated fault + suite, reboot/reconcile, and zero leaked process, namespace, link, cgroup, mount, socket, or nftables state. -### L2 +The existing system-test architecture, domain map, fixtures, orchestrator, coverage matrix, and release QA guide must be +updated as these gates land. Directly copying the binary into `/usr/local/bin` is not evidence that the installer works. -- Run exclusively as an unprivileged `mvm` group member. -- Confirm a user-owned `mvm` binary is rejected by passwordless sudo and the system installation is immutable. -- Attempt a caller-authored Firecracker/Jailer archive and checksum; installation must fail. -- Repeated create/kill/reconcile cycles show zero leaked mounts, cgroups, PIDs, or chroot state. -- Snapshot and live-volume parity pass inside the private mount namespace. -- A second Unix user cannot inspect, mount, signal, clean, or remove the first user's VM/release references. -- Firewall failure followed by immediate address reuse never grants stale access. -- Measure VMM overhead across supported Firecracker versions before accepting a default headroom. - -### Repository CI +## CI Gate ```bash go mod tidy && git diff --exit-code test -z "$(gofmt -l .)" -golines --max-len=120 --no-reformat-tags --list-files ./internal/ ./pkg/ ./cmd/ 2>&1 | grep . \ - && echo "violations found" && exit 1 || true +golines --max-len=120 --no-reformat-tags --list-files ./internal/ ./pkg/ ./cmd/ go generate ./internal/service/agent/... go vet ./... go test ./... -count=1 -coverprofile=coverage.out -covermode=atomic ``` -## Migration and Rollback - -1. Preserve the current base/head commit IDs and all unrelated dirty work before changing history. -2. Add forward-only migrations for exact-pair fields and uniqueness. Never rewrite an already-applied user DB in place. -3. In an authenticated administrator session, remove the insecure project-managed legacy sudoers rule, then run the - trusted release artifact's exact `host install-system` route. Installation is atomic at root-owned - `/usr/local/bin/mvm` and is deliberately separate from `host init`. After all callers use typed privileged actions, - `sudo /usr/local/bin/mvm host init` validates the marker-only policy with `visudo` and atomically replaces the old - rule. On failure it keeps the previous configuration and reports the unsafe state. -4. An unsupported privileged protocol refuses the operation and prints the explicit administrator upgrade command. -5. Existing VMs with an unambiguous exact pair are backfilled. Ambiguous/unmatched VMs remain stopped and return a - migration error; they are never launched with an arbitrary pair. -6. Do not remove legacy paths until the new privileged dispatcher has reconciled root records and the L2 upgrade test succeeds. -7. Before release, optionally curate/squash the five prototype commits plus remediation only with explicit approval. - -## Documentation Update Matrix - -Documentation is the final implementation phase, not a substitute for missing guarantees. - -Documentation accuracy is a release invariant. ADR-0016 and the threat contract are written before code so the target -is explicit; all current-behavior documentation is reconciled only after implementation and L2 evidence exist. A stale -or aspirational statement in an operator, architecture, testing, or troubleshooting document blocks release just like a -failing test. - -| Document | Required correction | -|---|---| -| `docs/adr/0016-*.md` | New decision superseding the unsafe wildcard sudo boundary and defining privileged-mode/state/trust semantics | -| `docs/adr/0002-*.md`, `0005-*.md`, `0015-*.md` | Mark superseded portions and link to ADR-0016; remove false safety claims | -| `CONTEXT.md` | Correct domain count, privileged boundary, service pattern, state ownership, subprocess exceptions | -| `AGENTS.md`, `docs/STANDARDS.md` | Add privileged-dispatch import/ownership rules and Jailer service; retain typed-method/no-generic-extension constraints | -| `README.md`, `docs/DEPENDENCIES.md` | Administrator system install/upgrade, sudo behavior, trusted-release limitations | -| `docs/RUNTIME.md` | Correct service count, state paths, private mount namespace, process/cleanup flow | -| `docs/RESOURCE_MANAGEMENTS.md` | Exact cgroup path, derived policy, measurement status, I/O policy | -| `docs/TROUBLESHOOTING.md` | System-binary ownership/version checks, reconcile commands, stale instance/chroot/cgroup diagnosis | -| `docs/roadmaps/TENANT_MICROVM.md` | Mark milestones incomplete until all release gates pass | -| `docs/system-test-architecture.md`, coverage matrix | Add privilege/race/recovery/multi-user L2 cases and correct fixture cleanup | -| `CHANGELOG.md` | Describe behavior only after it is verified and releasable | - -### Documentation definition of done - -- Every changed command, path, ownership, state, lifecycle, and recovery claim is traced to final code and an executed - test or inspection. -- Documents distinguish current behavior, migration behavior, known limitations, and future work; planned guarantees - are never written in the present tense. -- Installation, upgrade, rollback, troubleshooting, and recovery commands are executed on the L2 runner. -- Repository-wide searches cover the legacy `~/.local/bin/mvm` sudo target, raw sudo commands, `/var/run`/`/run` state - claims, both cgroup path variants, old service/domain counts, direct-Firecracker fallback, and snapshot host paths. -- ADR statuses and cross-links agree; superseded decisions remain available for historical context. -- The architect reads the final code diff before approving documentation, and QA independently verifies operator-facing - procedures against the release candidate. - -The untracked `docs/implementations/MEMORY_RECLAIM.md` currently contains the obsolete -`/sys/fs/cgroup/mvmctl/firecracker/` path. Preserve it as user work during implementation and correct it only -when its owner chooses to include it in the documentation sweep. - -## Risks and Mitigations - -| Risk | Impact | Mitigation | -|---|---|---| -| Privileged mode becomes a generic root RPC | Critical | Early dispatch, enumerated typed methods, minimal imports, abuse-case tests, no raw argv/path/cgroup API | -| Filesystem race remains after validation | Critical | Pinned dirfds/openat2, fixed basenames, concurrent rename/symlink L2 tests | -| Mounts leak or live hotplug remains invisible | High | Private launch namespace; verified `setns` for post-launch mounts; mount-leak assertions | -| PID reuse signals an unrelated host process | Critical | Root record + start time + cgroup + pidfd verification before every signal/setns | -| Root/user state diverges | High | Minimal root ownership record, explicit generations, startup reconciliation, fault injection | -| Release checksum channel is insufficient | High | Independent privileged fetch now; record limitation; adopt signatures/provenance when verifiably available | -| Privilege migration breaks existing installs | High | Atomic system-binary/sudoers install, protocol negotiation, upgrade L2 test, fail-closed diagnostics | -| Scope expands into unrelated XDG migration | Medium | Keep user state placement unchanged; separate ADR and project after security release | - -## Open Approval Points - -1. **Approved:** keep one binary and install it root-owned at `/usr/local/bin/mvm`, with early privileged dispatch. -2. **Approved:** remove every raw privileged system binary from the `mvm` sudoers entry, not only `mvm *`. -3. **Approved:** privileged code constructs the official Firecracker URL and independently fetches the checksum, with - signed provenance adopted if the upstream release channel provides a stable verifiable mechanism. -4. **Approved:** XDG state separation is a follow-up and not part of this remediation. +After CI, QA runs the applicable `scripts/run-system-tests.py` domains using the installed release candidate. Final +signoff requires the complete relevant L1/L2/L3 matrix and a clean-host run, not only test collection or mocked commands. + +## Explicit Deferrals + +- XDG user-state relocation. +- CNI and configurable root-executed network plugin chains. +- Multiple VMM support and speculative VMM-driver abstractions. +- Snapshot hot pools, device-mapper clones, and network pre-allocation. +- Resource batch grouping and root-volume boot. + +These features may reuse the v0.3.0 authority and namespace foundations later. They do not justify generic extension +points in this release. + +## Human Decisions + +The user has approved: + +- a clean installation with no backwards-compatibility implementation; +- one root-owned system binary and no helper/daemon; +- removal of raw sudo and the legacy public privileged services; +- mandatory per-VM network namespaces; +- nftables-only enforcement and removal of iptables; +- same-network and cross-network default deny; +- typed `traffic` and `exec` policies; +- removal of `allow_remote_exec`; +- deferral of CNI and speculative multi-VMM support; and +- documentation plus Python system-test coverage as release-critical artifacts. diff --git a/tasks/todo.md b/tasks/todo.md index dbca0477..a58a85f3 100644 --- a/tasks/todo.md +++ b/tasks/todo.md @@ -1,543 +1,316 @@ -# Safe Jailer Remediation Tasks +# v0.3.0 Release Tasks -This checklist implements `tasks/plan.md`. No production task begins until Task 1 is approved. Go production/L0/L1 work -belongs to the `engineer`; L2 and release verification belong to the `qa-engineer`; ADRs and documentation belong to the -`architect`. +This checklist implements `tasks/plan.md`. Go production and L0/L1 work belongs to the `engineer`; Python +`tests/system/` and release verification belongs to the `qa-engineer`; ADRs, plans, and documentation belong to the +`architect`. The architect reviews the actual diff after every implementation slice. -## Phase 0: Preserve and specify +## Phase 0: Preserve and establish the target -## Task 0: Preserve the prototype range and unrelated work +### Task 0: Preserve prototype history and unrelated work **Owner:** architect -**Description:** Record the reviewed base/head and preserve the dirty worktree before any history curation. +**Status:** Complete -**Acceptance criteria:** -- [x] Base `f8ffa28b9d74cac2c9e15fa07a18389d51a3263e` and head `ee733232a2dc23c1edb372022a2add71af36f44c` remain reachable. -- [x] Existing modified/untracked user files are inventoried and unchanged. -- [x] No reset, rebase, squash, or force push occurs before the release gate and explicit approval. +- [x] Preserve the prototype base/head and `review/jailer-prototype` reference. +- [x] Move local `main` back to `origin/main` without losing release work. +- [x] Keep all user-modified and untracked files outside release commits. +- [x] Work on `0.3.0-release` without force-pushing or rewriting shared history. -**Verification:** -- [x] `git status --short` matches the preserved inventory plus intentional remediation files. -- [x] `git log --oneline --decorate -10` shows the prototype commits and preservation reference. - -**Dependencies:** None -**Files likely touched:** None; Git references only after approval -**Estimated scope:** XS - -## Task 1: Approve ADR-0016 and the privileged operation catalog +### Task 1: Record the clean-break architecture **Owner:** architect -**Description:** Record the threat model, root-owned system binary, early privileged dispatcher, state layout, release -trust, ownership rules, and complete -typed operation catalog. Supersede the conflicting portions of ADR-0002, ADR-0005, and ADR-0015. - -**Acceptance criteria:** -- [x] ADR names every privileged action and rejects generic argv/command/path/cgroup extension points. -- [x] ADR defines bootstrap, upgrade, version negotiation, recovery, and rollback. -- [x] ADR maps every new interface/file to the existing CLI/API/core/service/lib layers and preserves dependency rules. -- [x] Human explicitly approves the four open decisions in `tasks/plan.md`. - -**Verification:** -- [x] Cross-check every proposed action against two analogous typed interfaces in the repository. -- [x] Confirm `cmd/mvm` only routes, `pkg/api` remains the sole cross-core orchestrator, and no core domain imports a - sibling core domain. -- [x] ADR-0005 explicitly marks its user-owned wildcard safety claim as pending replacement by ADR-0016. - +**Status:** Complete **Dependencies:** Task 0 -**Files likely touched:** `docs/adr/0016-*.md`, `docs/adr/0002-*.md`, `docs/adr/0005-*.md`, `docs/adr/0015-*.md` -**Estimated scope:** M - -### Checkpoint A: Architecture - -- [x] Task 0 and Task 1 complete. -- [x] Human approves ADR-0016 before Go implementation. -- [x] Abuse cases are listed as test requirements before production implementation. -## Phase 1: Replace the privilege boundary while preserving one binary +- [x] ADR-0016 defines the root-owned single-binary privileged dispatcher. +- [x] ADR-0005 is marked superseded for v0.3.0 while its transitional implementation remains accurately labeled. +- [x] ADR-0017 defines mandatory per-VM netns, nftables-only enforcement, and typed `traffic` / `exec` policies. +- [x] ADR-0009 and ADR-0014 are marked superseded for v0.3.0. +- [x] CNI and the old optional namespace proposal are marked rejected/superseded. +- [x] The release plan records exact authority interfaces, lifecycle states, lock order, errors, and QA gates. +- [x] Update ADR-0016 and operator-facing planning text to require clean installation rather than legacy adoption. +- [x] Review all architecture diffs against `CONTEXT.md`, `docs/STANDARDS.md`, and existing interfaces. +- [x] Commit only the architecture baseline, plan, and changelog files after diff verification. -## Task 2: Atomically install the single root-owned system binary +### Task 2: Verify the implemented system installer through Python -**Owner:** engineer -**Description:** Add an administrator-only install/upgrade flow for the existing release artifact at -`/usr/local/bin/mvm`. Privileged dispatch validates executable ownership/mode and protocol version before any normal app -initialization. +**Owner:** qa-engineer +**Status:** In progress +**Dependencies:** Implemented system installer -**Acceptance criteria:** -- [x] `/usr/local/bin/mvm` is installed as `root:root`, mode `0755`, under a root-owned non-writable directory chain. -- [x] Installation is temporary-file + fsync + rename and refuses symlinked targets. -- [x] A normal user update cannot replace the system binary; unsupported privileged protocol fails closed with an - administrator upgrade instruction. +- [ ] Add an installed-release-candidate fixture that stages the candidate outside `/usr/local/bin`. +- [ ] Invoke `sudo host install-system` and then `sudo /usr/local/bin/mvm host init`. +- [ ] Prove canonical path, root ownership, exact mode, safe ancestors, hash equality, and atomic replacement. +- [ ] Prove malformed install invocation, symlink target, partial failure, and system self-update refusal. +- [ ] Register the focused Python domain in the orchestrator and coverage matrix. +- [ ] Run collection and the focused domain on the prepared runner. -**Verification:** -- [x] L1 install and early-protocol tests cover ownership, modes, symlink target, partial write, and version mismatch. -- [x] Focused Go tests for system installation, early dispatch, and host initialization pass. +The final marker-only sudoers assertion is deferred until Task 14; current sudoers remains transitional. -**Dependencies:** Task 1 -**Files likely touched:** `cmd/mvm/main.go`, `internal/core/host/utils.go`, `internal/core/host/utils_test.go`, -`internal/infra/constants.go`, `scripts/build.sh` -**Estimated scope:** M +## Phase 1: Root authority and trusted resources -## Task 3: Define strict early privileged request parsing and caller identity +### Task 3: Complete the early privileged protocol foundation **Owner:** engineer -**Description:** Before `app.Initialize()` and Cobra, detect only the reserved privileged entry point, then implement -protocol-versioned, bounded, typed request decoding; sanitize environment; derive the caller from `SUDO_UID`/`SUDO_GID`; -reject rootless/direct misuse and all unknown or extra input. - -**Acceptance criteria:** -- [ ] Each action has a distinct request type and handler; no `[]string` command or generic operation payload exists. -- [ ] Request body, string, list, and numeric limits are enforced before side effects. -- [ ] Caller UID/GID and system-binary ownership checks precede action dispatch. -- [ ] Ordinary CLI initialization and commands are unreachable after privileged mode is selected. - -**Verification:** -- [ ] Table tests reject unknown actions/fields, duplicate JSON fields, oversized input, invalid IDs, missing sudo identity, - and unsupported protocol versions. -- [ ] `go test` for privileged dispatch passes with `FakeRunner` and temporary roots. +**Status:** Foundation implemented; action catalog pending later tasks +**Dependencies:** Task 1 -**Dependencies:** Task 2 -**Files likely touched:** `cmd/mvm/main.go`, `internal/service/privileged/protocol.go`, `entry.go`, `identity.go`, tests -**Estimated scope:** M +- [x] Reserved marker selected before Cobra and application initialization. +- [x] Unsupported versions fail closed. +- [x] Strict bounded JSON rejects unknown, duplicate, case-fold duplicate, trailing, oversized, and malformed input. +- [x] Caller identity, active `mvm` group, environment, and system executable are verified before dispatch. +- [ ] Keep the action switch closed until each capability has its own typed handler and abuse tests. -## Task 4: Migrate Jailer service calls to typed privileged actions +### Task 4: Add private root-owned VM instance authority **Owner:** engineer -**Description:** Route install, launch, cleanup, snapshot exposure, volume exposure/removal, and release removal through -the privileged protocol. Remove the public `mvm run jailer` root entry point. - -**Acceptance criteria:** -- [ ] Jailer caller methods retain typed names matching existing service conventions. -- [ ] The general CLI cannot directly dispatch privileged Jailer handlers. -- [ ] Unknown/unauthorized VM and release operations fail before filesystem/process changes. - -**Verification:** -- [ ] L1 tests assert exact privileged requests for every Jailer method. -- [ ] `rg -n "run jailer|newJailerServiceCmd"` finds only migration/docs references. - -**Dependencies:** Task 3 -**Files likely touched:** `internal/service/jailer/spawn.go`, `internal/service/jailer/entry.go`, their tests, -`internal/cli/service.go` -**Estimated scope:** M - -## Task 5: Constrain loopmount behind privileged dispatch +**Status:** Ready for implementation +**Dependencies:** Tasks 1 and 3 -**Owner:** engineer -**Description:** Keep the typed provisioning wire operations but validate all roots and chroot targets through pinned -descriptors. Remove direct privileged `mvm run provision` and the arbitrary `--umount` shortcut. +**Interface:** -**Acceptance criteria:** -- [ ] Provision operations are limited to the invoking user's pinned VM/image/volume roots. -- [ ] Unmount/process handling cannot target unrelated host mounts or PIDs. -- [ ] No caller-provided host command is executed outside the intended mounted guest root. +- [ ] Implement private `RegisterLaunch(ctx, caller, registration) (*launchLease, error)`. +- [ ] Implement private `LockRegistered(ctx, caller, vmID) (*registeredLease, error)`. +- [ ] Implement private `BeginCleanup(ctx, caller, vmID) (*cleanupLease, error)`. +- [ ] Implement context-first `cleanupLease.Complete(ctx)` and checked lease release. +- [ ] Implement private `LockUnreferencedRelease(ctx, release) (*releaseLease, error)`. +- [ ] Expose no generic storage, callbacks, raw paths, actions, or state strings. -**Verification:** -- [ ] L1 abuse tests cover host path, symlink, mount swap, unrelated PID, and arbitrary unmount attempts. -- [ ] Existing loopmount L1 behavior remains green. +**Authority and lifecycle:** -**Dependencies:** Task 3 -**Files likely touched:** `internal/service/loopmount/entry.go`, `provisioner.go`, `spawn.go`, `wire.go`, tests -**Estimated scope:** M +- [ ] Persist strict versioned records under `/var/lib/mvmctl/instances//.json`. +- [ ] Store only owner UID, VM ID, lifecycle, exact release identity, process identity, cgroup path, and cleanup generation. +- [ ] Implement `absent|cleaned -> registered -> cleaning -> cleaned`; failed cleanup remains recoverable. +- [ ] Keep a `cleaned` ownership tombstone; foreign UIDs never acquire the VM ID. +- [ ] Establish the global VM-ID claim while holding the global index lock and scanning pinned numeric UID directories. +- [ ] Reject foreign, duplicate, corrupt, unreadable, unsafe, or inconsistent records without disclosing the owner UID. -## Task 6: Replace raw network-link sudo calls with typed privileged methods +**Filesystem and locking:** -**Owner:** engineer -**Description:** Add named bridge, TAP, address, route, and link lifecycle operations. Callers pass validated model data, -not raw `ip` argv. +- [ ] Use descriptor-relative no-follow traversal and retain required root/record/lock FDs. +- [ ] Verify root ownership, exact file types, and non-writable ancestors. +- [ ] Use exact `0700` managed directories and `0600` record/lock files. +- [ ] Implement the Task 4 lock prefix: release -> index -> VM. +- [ ] Use cancellable nonblocking flock retry; never unlink lock files. +- [ ] Atomically replace records using exclusive temp, bounded write, ownership/mode, fsync, renameat, and directory fsync. +- [ ] Return `record_replaced` / `durability_uncertain` DomainError details for post-rename failure. -**Acceptance criteria:** -- [ ] No unprivileged code can ask root-side code to run arbitrary `ip` arguments or `netns exec`. -- [ ] Operations validate interface names, addresses, owning network/VM IDs, and idempotent expected state. -- [ ] Existing network Service remains the intra-domain owner; API boundaries do not move. +**TDD and verification:** -**Verification:** -- [ ] `FakeRunner` tests assert exact root-side commands for each typed method and reject shell/argv injection. -- [ ] Network L1 lifecycle tests pass. +- [ ] RED/GREEN strict codec and pure lifecycle-transition tests. +- [ ] RED/GREEN descriptor store, atomic-write, and context-aware lock tests. +- [ ] RED/GREEN typed authority behavior and release-reference tests. +- [ ] Real concurrent two-UID first-launch race proves exactly one durable owner. +- [ ] Tests cover every unsafe ancestor, symlink level, owner/mode/type mismatch, EEXIST race, partial I/O, fsync/close/ + rename failure, cancellation, stale lock reuse, corrupt record, and temp cleanup. +- [ ] Focused race test, format, golines, vet, and Go tests pass. +- [ ] Architect reviews actual interface, state schema, lock order, imports, and error details before commit. -**Dependencies:** Task 3 -**Files likely touched:** privileged network handlers, `internal/core/network/service.go`, typed model/request tests -**Estimated scope:** M +**Expected files:** fixed root constants; private instance types, codec, descriptor store, lock, authority, and focused tests +under `internal/service/jailer/`. No handler, transport, CLI, API, core-domain, or network changes. -## Task 7: Replace raw firewall sudo calls with atomic typed batches +### Task 5: Add descriptor-pinned managed user-resource access **Owner:** engineer -**Description:** Move iptables/nftables application into privileged actions that accept validated firewall models and render -complete atomic backend batches root-side. - -**Acceptance criteria:** -- [ ] Caller cannot pass raw iptables/nft syntax, filenames, includes, or commands. -- [ ] Chain creation, parent jumps, ordering, and managed rules apply as one reported operation. -- [ ] Every runner/backend error propagates; initialization can no longer silently succeed. +**Dependencies:** Task 4 -**Verification:** -- [ ] Backend tests inject failure for every chain/jump/batch stage. -- [ ] Golden tests cover exact iptables-restore and nft batch output from typed models. +- [ ] Open user cache/VM/image/kernel/volume/snapshot resources once with beneath/no-symlink/no-magic-link resolution. +- [ ] Verify caller ownership, file type, fixed basenames, size, and allowed access mode on pinned descriptors. +- [ ] Never validate a path and later reopen it by pathname. +- [ ] Return DomainError partial-state details for close/fsync/cleanup failures. +- [ ] Race tests continuously replace every user-owned ancestor and leaf with symlinks/renames. -**Dependencies:** Task 3 -**Files likely touched:** privileged firewall handlers, `internal/lib/firewall/tracker.go`, `iptables.go`, `nftables.go`, tests -**Estimated scope:** M - -## Task 8: Replace raw host module/sysctl sudo calls +### Task 6: Implement independent trusted release authority **Owner:** engineer -**Description:** Expose only the exact host initialization actions mvmctl needs. Module names, sysctl keys, and values -are compiled/root-policy allowlists, not caller-selected commands. - -**Acceptance criteria:** -- [ ] Privileged dispatch supports only named host readiness/remediation operations from ADR-0016. -- [ ] Arbitrary modprobe config/path and arbitrary sysctl keys are impossible. -- [ ] Detection stays unprivileged; only mutation crosses the privileged boundary. +**Dependencies:** Tasks 4 and 5 -**Verification:** -- [ ] L1 tests reject all unlisted modules/keys/values and confirm supported host-init actions. -- [ ] Host probe tests remain green. +- [ ] Privileged code constructs the fixed official release/checksum locations from validated version and architecture. +- [ ] Caller-provided archive bytes are not their own authority. +- [ ] Bounded extraction rejects traversal, links, devices, duplicates, unexpected members, and size/count overflow. +- [ ] Exact Firecracker/Jailer bytes and a root manifest install atomically and durably. +- [ ] Referenced release removal/replacement holds the release lease and fails closed on unreadable records. +- [ ] L1 failure injection proves an old complete pair or no pair, never a partial trusted release. -**Dependencies:** Task 3 -**Files likely touched:** privileged host handlers, `internal/core/host/service.go`, `utils.go`, `probe.go`, tests -**Estimated scope:** M - -## Task 9: Replace sudoers with the single-binary privileged policy +### Task 7: Add private mount and process identity primitives **Owner:** engineer -**Description:** After Tasks 4-8 migrate every callsite, atomically replace the wildcard CLI and raw system-binary -entries with the internal privileged entry point of root-owned `/usr/local/bin/mvm`. - -**Acceptance criteria:** -- [ ] Generated sudoers contains no user-owned `mvm`, `ip`, `iptables`, `nft`, `modprobe`, or `sysctl` command. -- [ ] `visudo -c` validation occurs before activation and legacy insecure entries are detected. -- [ ] Failure preserves the prior file and reports that the host remains unsafe/uninitialized. - -**Verification:** -- [ ] Unit golden test covers the exact sudoers content. -- [ ] L2 confirms every removed command fails under `sudo -n` while supported CLI flows still work. - -**Dependencies:** Tasks 4, 5, 6, 7, 8 -**Files likely touched:** `internal/core/host/utils.go`, `utils_test.go`, `internal/infra/constants.go`, host-init tests -**Estimated scope:** M - -### Checkpoint B: Privilege boundary +**Dependencies:** Tasks 4 and 5 -- [ ] Tasks 2-9 complete. -- [ ] Passwordless sudo reaches only the immutable binary's early privileged entry point. -- [ ] User-owned binary replacement, raw system commands, and generic privileged requests are rejected in L1/L2. +- [ ] Unshare a private mount namespace before launch mounts and mark propagation private. +- [ ] Verify PID, start ticks, cgroup, owner, release, and namespace inode before signal or setns. +- [ ] Use pidfd when available and fail closed on PID reuse. +- [ ] Live snapshot/volume operations enter only the verified VM mount namespace. +- [ ] Cleanup never recursively traverses a tree that may contain a mount. +- [ ] L1 fault tests cover source/target replacement, PID mismatch, process exit, setns, mount, unmount, and cleanup. -## Phase 2: Trusted release and safe filesystem operations +## Phase 2: Final Jailer and networking paths -## Task 10: Implement independently verified atomic release installation +### Task 8: Wire typed Jailer lifecycle actions and remove the legacy service **Owner:** engineer -**Description:** Move release authority inside privileged dispatch, harden archive extraction, and create an atomic root-owned -release manifest for the exact pair. +**Dependencies:** Tasks 4-7 -**Acceptance criteria:** -- [ ] Privileged code constructs the allowed version/architecture URL and independently obtains expected integrity metadata. -- [ ] Archive parser rejects traversal, links, devices, duplicates, unexpected members, and size/count overflow. -- [ ] Pair installation is atomic, fsynced, content-addressed, and cannot replace a referenced release. +- [ ] Add distinct typed install, launch, abort, cleanup, snapshot exposure, volume exposure/removal, and release removal + actions to the privileged dispatcher. +- [ ] Preserve existing named Jailer caller methods; replace their transport without adding a generic action method. +- [ ] Persist `registered` authority before the first privileged launch effect and retain the lease through exec. +- [ ] Every post-effect failure invokes verified typed abort and leaves recoverable authority. +- [ ] Remove public `mvm run jailer` and its Cobra/service dispatch. +- [ ] Search finds no ordinary CLI route to a privileged Jailer handler. +- [ ] Go L1 and Python T2 Jailer/authority tests prove cross-UID denial, record-before-effect, exact process identity, + cgroup/chroot cleanup, and zero leaks. -**Verification:** -- [ ] Negative tests include a caller-authored archive plus matching caller hash and all archive abuse cases. -- [ ] Interrupted install leaves either the old complete pair or no pair, never a partial trusted directory. - -**Dependencies:** Tasks 4 and 9 -**Files likely touched:** `internal/service/jailer/entry.go`, privileged release handler/manifest, archive tests -**Estimated scope:** M - -## Task 11: Introduce descriptor-relative managed-resource access +### Task 9: Add root network authority and mandatory per-VM namespaces **Owner:** engineer -**Description:** Replace canonicalize/validate/reopen logic with pinned directory/file descriptors and fixed basenames. - -**Acceptance criteria:** -- [ ] Managed roots use `openat2`/`openat` no-symlink/beneath resolution and `fstat` ownership/type checks. -- [ ] Privileged PID/status/manifest writes use root-owned fixed paths, not user-selected basenames. -- [ ] No privileged operation reopens a user-controlled absolute path after validation. +**Dependencies:** Tasks 4, 7, and 8 -**Verification:** -- [ ] Race tests continuously rename/replace every user-owned parent with symlinks during privileged operations. -- [ ] Static review finds no `EvalSymlinks`-then-open security boundary in privileged/Jailer code. +- [ ] Persist minimal root-owned network identity and serialize network/VM topology mutations. +- [ ] Acquire the network lock only after the applicable VM launch/registered lease; the full order is + `release -> global index -> VM -> network`. +- [ ] Derive namespace/interface names from UID and immutable IDs; accept no raw name, path, or `ip` argv. +- [ ] Create namespace, local bridge, TAP, and veth pair with links down. +- [ ] Attach only the verified host veth to the verified managed bridge. +- [ ] Pass the pinned root-owned namespace handle to Jailer with `--netns`. +- [ ] Record namespace inode and exact device identities for verification/reconciliation. +- [ ] Typed rollback and cleanup are idempotent and ownership-bound. +- [ ] Python T2 proves every Firecracker process is in its own namespace and no namespace/link leaks after failure. -**Dependencies:** Tasks 3 and 4 -**Files likely touched:** privileged path module/tests, `internal/service/jailer/entry.go`, `manifest.go` -**Estimated scope:** M - -## Task 12: Make launch mounts private and live mounts namespace-aware +### Task 10: Replace both firewall backends with one atomic nftables engine **Owner:** engineer -**Description:** Unshare a private mount namespace before pre-launch mounts. For live mounts, verify the registered -process, pin source/target descriptors, enter its mount namespace, and apply restrictive mount flags. - -**Acceptance criteria:** -- [ ] Launch mounts never appear in the original host namespace. -- [ ] Live volume/snapshot mounts are visible to jailed Firecracker and cannot target host paths through symlinks. -- [ ] VM exit destroys namespace mounts; cleanup never recursively traverses a mounted user directory. +**Dependencies:** Tasks 3 and 9 -**Verification:** -- [ ] L1 namespace-operation tests cover PID mismatch and target/source replacement. -- [ ] L2 asserts visibility inside the VM namespace and absence from host mountinfo before/after failure. +- [ ] Add typed privileged nftables generation/application; caller cannot pass syntax, files, includes, or commands. +- [ ] Install bridge-family anti-spoofing and same-network filtering. +- [ ] Install routed, VM-to-host, established reply, required system service, egress, and NAT rules. +- [ ] Apply complete generations atomically before links become usable. +- [ ] Propagate every initialization, parent-chain, ordering, and batch failure. +- [ ] Remove iptables renderer/repository/tests, backend enum/selector/config, raw sudo rule, and documentation. +- [ ] Host readiness requires nftables bridge-family support. -**Dependencies:** Task 11 -**Files likely touched:** privileged namespace/mount modules and tests, `internal/service/jailer/entry.go` -**Estimated scope:** M - -## Phase 3: Lifecycle, cgroups, and exact identity - -## Task 13: Add root-owned instance records and per-VM locking +### Task 11: Replace service-access policy with typed traffic policies **Owner:** engineer -**Description:** Persist minimal UID/release/process/cleanup metadata under `/var/lib/mvmctl/instances` and serialize all -privileged VM actions with `/run/mvmctl` locks. - -**Acceptance criteria:** -- [ ] Record creation/update is atomic, root-owned, and bound to `SUDO_UID`. -- [ ] Every launch, signal, setns, mount, cleanup, and release-reference check holds the VM lock. -- [ ] Other UIDs and PID/start-time/cgroup mismatches fail closed. +**Dependencies:** Task 10 -**Verification:** -- [ ] Tests cover lock contention, stale lock, cross-UID request, PID reuse, corrupt/truncated record, and atomic recovery. -- [ ] Root record contains no full user VM config or secrets. +- [ ] Remove `ServiceAccessPolicy`, routed-only CLI/API/schema/compiler, and same-network rejection. +- [ ] Add separate typed source-network and source-VM traffic input/resolution methods under `mvm policy traffic`. +- [ ] Persist immutable source/destination identities, TCP/UDP, and exact port/range only. +- [ ] Compile source-network and exact source-VM policy for same-network bridge and cross-network routed paths. +- [ ] Default-deny all other managed VM-to-VM traffic without breaking replies, required system traffic, or egress. +- [ ] Resource/policy removal applies safe kernel state before releasing identity or IP. +- [ ] Python T2 proves exact allows, wrong source/target/protocol/port denial, same/cross behavior, reboot sync, and IP reuse. -**Dependencies:** Tasks 10, 11 -**Files likely touched:** privileged instance record/lock/process modules and tests -**Estimated scope:** M - -## Task 14: Make VM spawn and abort crash-consistent +### Task 12: Replace remote-exec flags with typed exec policies **Owner:** engineer -**Description:** Persist a provisional VM before spawn, retain a lifecycle handle immediately after spawn, and route -every post-spawn failure through a typed verified abort. - -**Acceptance criteria:** -- [ ] No error path after spawn can return without terminating or explicitly retaining a manageable registered VM. -- [ ] Abort verifies process identity, performs TERM/wait/KILL, then cgroup/chroot/record cleanup in order. -- [ ] Snapshot-restore batch rollback stops prior restored VMs before deleting DB/VM directories. +**Dependencies:** Tasks 4, 8, and 9 -**Verification:** -- [ ] Failure injection covers relay start, API socket, DB writes, verification, snapshot load, and batch rollback. -- [ ] Each case asserts no live PID, cgroup, mount, jail, or orphaned root record. +- [ ] Remove `allow_remote_exec` from schema, model, config, CLI, environment specs, results, tests, and docs. +- [ ] Add directional exact source-VM -> target-VM -> non-root-user `mvm policy exec` intent. +- [ ] Refactor the relay so it no longer imports multiple core domains; API resolves shared identities. +- [ ] Authenticate source from the active host session and root-owned instance/vsock identity. +- [ ] Enforce request/frame/output/duration/concurrency bounds and redact full commands from audit logs. +- [ ] Reject root target, VM-to-host execution, missing/reverse policy, foreign owner, stale target, and identity mismatch. +- [ ] Python T2 proves same-network, cross-network, and no-IP-network exec behavior plus all denials and limits. -**Dependencies:** Tasks 12 and 13 -**Files likely touched:** `internal/core/vm/firecracker.go`, `pkg/api/vm.go`, `pkg/api/snapshot.go`, focused tests -**Estimated scope:** M +## Phase 3: Finish the privilege boundary and lifecycle -## Task 15: Add startup and pre-launch privileged reconciliation +### Task 13: Migrate loopmount and supported host mutations **Owner:** engineer -**Description:** Reconcile instance records, verified processes, cgroups, and chroot directories after CLI interruption or -host reboot without trusting the user DB as the root authorization source. +**Dependencies:** Tasks 3, 5, and 6 -**Acceptance criteria:** -- [ ] Reconcile distinguishes running, stopped-clean, stale, corrupt, and foreign-owner records. -- [ ] It never signals an unverified PID or recursively deletes a path that may contain a mount. -- [ ] Launch runs per-UID/per-VM reconciliation before creating new privileged state. - -**Verification:** -- [ ] L1 state-machine tests cover every classification and idempotent retry. -- [ ] L2 kill/reboot scenarios converge to zero stale resources or a precise actionable error. - -**Dependencies:** Tasks 13 and 14 -**Files likely touched:** privileged reconciliation module/tests, Jailer service caller, host readiness integration -**Estimated scope:** M - -## Task 16: Persist exact Firecracker/Jailer identity transactionally - -**Owner:** engineer -**Description:** Add snapshot Jailer identity, uniqueness, transactional pair repository operations, and deterministic -legacy backfill. Compare user DB pair hashes with the root release manifest before launch. +- [ ] Move provisioning/mount operations behind typed descriptor-pinned actions. +- [ ] Remove public `mvm run provision` and arbitrary unmount behavior. +- [ ] Move required module/sysctl/host mutations behind exact compiled allowlists. +- [ ] Move any remaining bridge/link/firewall caller to the typed privileged modules. +- [ ] Repository search finds no unauthorized raw sudo callsite. +- [ ] L1 abuse tests cover arbitrary path, PID, mount, module, sysctl, network, and shell/argv attempts. -**Acceptance criteria:** -- [ ] VM and snapshot always persist both exact IDs/hashes; restored/migrated VMs cannot retain an empty Jailer ID. -- [ ] Pair install/default/remove DB updates are transactional and return all errors. -- [ ] Ambiguous or unmatched legacy data fails closed instead of selecting an arbitrary same-version row. +### Task 14: Install the final marker-only sudoers policy -**Verification:** -- [ ] Migration tests cover unique, duplicate, ambiguous, missing, referenced, and rollback cases. -- [ ] Restore-then-hotplug test sees the persisted Jailer identity. +**Owner:** engineer with qa-engineer L2 verification +**Dependencies:** Tasks 8-13 -**Dependencies:** Tasks 10 and 13 -**Files likely touched:** new DB migration, binary/snapshot model, binary repository/service, migration tests -**Estimated scope:** M +- [ ] Generate only the exact root-owned versioned privileged dispatcher entry for the `mvm` group. +- [ ] Remove wildcard `/usr/local/bin/mvm *` and raw `ip`, `iptables`, `nft`, `modprobe`, and `sysctl` grants. +- [ ] Validate with `visudo -c` before atomic activation; preserve prior policy on failure. +- [ ] Python system tests prove every removed command and `sudo /usr/local/bin/mvm help` fail while supported ordinary + unprivileged CLI workflows succeed. -## Task 17: Centralize jail-visible path translation and restore parity +### Task 15: Complete reconciliation, admission, and lifecycle parity **Owner:** engineer -**Description:** Put one typed translator in the VM spawner/controller seam and use it for config, rootfs, sockets, -vsock, snapshot create/restore, and live volume hotplug. +**Dependencies:** Tasks 8-14 -**Acceptance criteria:** -- [ ] Snapshot create passes/exposes its directory and uses `/snapshot/...`; restore uses the same mapping. -- [ ] Rootfs, API/vsock sockets, config, and volume paths have explicit host and jail representations. -- [ ] Restored jailed VMs use namespace-aware volume attach/detach. +- [ ] Add root-authoritative startup/pre-launch reconciliation for records, processes, cgroups, namespaces, links, + chroots, mounts, and nftables generations. +- [ ] Add root-owned global capacity reservation so separate CLI processes and users cannot over-admit the host. +- [ ] Make create/start/stop/remove/reboot/snapshot/restore/volume paths transactionally abortable and retryable. +- [ ] Preserve exact release identity and jail-visible paths across snapshot restore and live hotplug. +- [ ] Harden cgroup readiness, membership, values, and process-aware cleanup. +- [ ] Remove process-local semaphore admission and user-DB-as-root-authority designs. +- [ ] L1/L2 fault injection covers kill at every effect stage, reboot, PID reuse, partial cleanup, and repeated reconcile. -**Verification:** -- [ ] L1 tests assert every exact Firecracker API path for create, load, and hotplug. -- [ ] Prepared L2 Jailer snapshot/volume tests pass unchanged except for fixture correctness. - -**Dependencies:** Tasks 12, 14, 16 -**Files likely touched:** `internal/core/vm/jailer.go`, `controller.go`, `pkg/api/snapshot.go`, tests -**Estimated scope:** M - -## Task 18: Harden cgroup policy, readiness, and cleanup - -**Owner:** engineer -**Description:** Keep the corrected leaf path, derive all values from typed VM/host policy, validate controller -enablement, and ensure process-aware removal. Add I/O limits if retained by the approved M2 specification. - -**Acceptance criteria:** -- [ ] Privileged dispatch accepts VM resource intent, never raw cgroup key/value pairs, and applies host-policy bounds. -- [ ] Host readiness always reports missing cgroup v2/version/controllers/swap support as critical. -- [ ] Cleanup cannot remove or kill a cgroup belonging to an unverified instance. - -**Verification:** -- [ ] L1 covers version zero, unavailable/unenableable controllers, mismatched values, foreign membership, and cleanup. -- [ ] L2 measures supported Firecracker versions before the memory-headroom default is approved. - -**Dependencies:** Tasks 13 and 14 -**Files likely touched:** privileged cgroup module, `internal/core/host/detector.go`, `probe.go`, model policy, tests -**Estimated scope:** M - -### Checkpoint C: Jailer lifecycle parity - -- [ ] Tasks 10-18 complete. -- [ ] All normal lifecycle and snapshot/hotplug L1 tests pass. -- [ ] Fault injection proves cleanup and reconciliation are idempotent. -- [ ] Exact release and owner identity are present on every VM/snapshot path. - -## Phase 4: Fail-closed routed policy - -## Task 19: Make firewall initialization observable and atomic - -**Owner:** engineer -**Description:** Return and propagate errors for chain/jump setup and apply each complete managed generation through an -atomic backend batch. - -**Acceptance criteria:** -- [ ] `Initialize` and jump ordering return typed failure; no runner result is silently discarded. -- [ ] Reconciliation verifies required parent jumps and default-deny placement before success. -- [ ] Partial backend application cannot be reported as successful. - -**Verification:** -- [ ] Failure injection for every parent jump/reorder/batch command returns `CodePolicySyncFailed`. -- [ ] Both firewall backend test suites pass. - -**Dependencies:** Task 7 -**Files likely touched:** `internal/lib/firewall/tracker.go`, `iptables.go`, `nftables.go`, network policy tests -**Estimated scope:** M - -## Task 20: Reorder policy/resource deletion around kernel reconciliation +### Task 16: Complete global structured JSON output **Owner:** engineer -**Description:** Apply the desired kernel generation before deleting policy/VM/network identity or releasing an IP; -restore the prior generation if the DB commit fails. - -**Acceptance criteria:** -- [ ] Failed `policy rm` retains authoritative intent and retryability. -- [ ] Failed VM/network removal cannot release an address while stale allow rules remain. -- [ ] Derived firewall rows and intent transition transactionally after kernel success. +**Dependencies:** Stable final CLI results from Tasks 8-15 -**Verification:** -- [ ] Unit tests inject kernel and DB failures at every boundary and assert rollback/recovery details. -- [ ] IP-reuse integration test proves no replacement VM inherits a stale allow. +- [ ] Add one global structured-output contract for reads and mutations. +- [ ] Remove conflicting per-command rendering branches where the global contract replaces them. +- [ ] Preserve DomainError code/class/details and per-item partial results. +- [ ] Python CLI/invariant tests cover success, validation failure, partial batch failure, and stable machine-readable output. -**Dependencies:** Task 19 -**Files likely touched:** `pkg/api/policy.go`, `pkg/api/vm.go`, network removal API, firewall repository, tests -**Estimated scope:** M +## Phase 4: Release qualification and documentation -## Phase 5: System verification and documentation - -## Task 21: Repair L2 fixture isolation and add privilege/fault coverage +### Task 17: Run staged Python system-test qualification **Owner:** qa-engineer -**Description:** Fix structured cgroup-setting restoration, then add real-host tests for immutable sudo, release trust, -multi-user isolation, path races, PID reuse, kill-at-stage recovery, namespace mounts, and IP reuse. +**Dependencies:** Applicable implementation task for each domain -**Acceptance criteria:** -- [ ] Every modified setting is restored exactly or reset, and cleanup failures fail the test. -- [ ] New tests use Tier-2 fixtures and known-limitation conventions from the system-test architecture. -- [ ] Repeated suite execution leaves the runner with no leaked VM process, mount, cgroup, network, or config override. +- [ ] Extend existing Jailer, cgroup, network, nftables, policy, exec, invariant, and full-journey tests instead of + replacing them with a parallel harness. +- [ ] Add only the focused installer/authority/netns/fault tests missing from existing domains. +- [ ] Remove direct raw-sudo product mutations from tests; use the installed CLI. +- [ ] Use constrained users with only the product’s intended `mvm` privilege. +- [ ] Fix exact configuration restoration and make all teardown failures visible. +- [ ] Run each relevant domain twice and assert zero leaked resources. +- [ ] Run the complete T1/T2/T3 matrix plus clean-host installation and reboot/reconciliation qualification. +- [ ] Update `tests/system/COVERAGE_MATRIX.md` and attach commands/results to the release report. -**Verification:** -- [ ] Targeted L2 domains pass twice consecutively on the prepared runner VM. -- [ ] Post-suite leak audit is empty. +### Task 18: Run repository CI in exact order -**Dependencies:** Tasks 9, 10-20 -**Files likely touched:** `tests/system/vm/test_cgroup.py`, `test_jailer.py`, policy tests, shared fixtures, coverage matrix -**Estimated scope:** M +**Owner:** engineer and qa-engineer +**Dependencies:** Tasks 3-17 -## Task 22: Run full CI, L2 release qualification, and resource measurements +- [ ] `go mod tidy && git diff --exit-code` +- [ ] `test -z "$(gofmt -l .)"` +- [ ] `golines --max-len=120 --no-reformat-tags --list-files ./internal/ ./pkg/ ./cmd/` +- [ ] `go generate ./internal/service/agent/...` +- [ ] `go vet ./...` +- [ ] `go test ./... -count=1 -coverprofile=coverage.out -covermode=atomic` +- [ ] No critical/high review finding, skipped release gate, hidden teardown error, or unexplained coverage regression. -**Owner:** qa-engineer -**Description:** Execute repository CI in its exact order, the full relevant L2 matrix, upgrade/rollback tests, and -versioned Firecracker overhead measurements. - -**Acceptance criteria:** -- [ ] Tidy, format, golines, generate, vet, and full tests pass. -- [ ] L2 privilege/Jailer/cgroup/snapshot/volume/policy/upgrade suites pass on supported architectures. -- [ ] Memory headroom and I/O-limit decisions are backed by recorded measurements, not assumed constants. - -**Verification:** -- [ ] Attach command output and runner environment to the release report. -- [ ] No critical/high review finding remains open or deferred without explicit release rejection. - -**Dependencies:** Task 21 -**Files likely touched:** Test reports/fixtures only -**Estimated scope:** S - -## Task 23: Reconcile all documentation with verified behavior - -**Owner:** architect -**Description:** Apply the documentation matrix from `tasks/plan.md`, using the final implementation and test evidence -as the source of truth. - -**Acceptance criteria:** -- [ ] ADR statuses, state paths, system-install/privileged-mode setup, service list, cgroup paths, lifecycle, and - limitations are accurate. -- [ ] Roadmap and changelog do not claim completion beyond executed evidence. -- [ ] Repository-wide searches find no stale wildcard-sudo, `/var/run` full-state, old cgroup path, or service-count claim. -- [ ] Documentation lands in reviewable slices: architecture/rules, user/operator guides, then testing/release records. - -**Verification:** -- [ ] Architect reads the final implementation diff for interfaces, structs, imports, and state locations. -- [ ] Documentation link/path/command checks pass and QA verifies operator procedures on the runner. - -**Dependencies:** Task 22 -**Files likely touched:** Files in the documentation matrix; split into reviewable documentation-only commits -**Estimated scope:** M per documentation slice - -## Task 24: Independently verify documentation accuracy +### Task 19: Reconcile and independently verify all documentation **Owner:** architect with qa-engineer verification -**Description:** Treat documentation as a release artifact. Trace each security- and operator-relevant claim to final -code, then execute every installation, upgrade, recovery, and troubleshooting procedure on the release candidate. - -**Acceptance criteria:** -- [ ] Every changed behavior is described consistently across ADRs, `CONTEXT.md`, standards, runtime, operator docs, - roadmap, test architecture, coverage matrix, and changelog. -- [ ] Current behavior, migration behavior, known limitations, and future work are clearly distinguished. -- [ ] No stale command/path/ownership/service-count/cgroup/snapshot/sudo claim remains anywhere in tracked Markdown. - -**Verification:** -- [ ] Architect signs off after reading the final implementation diff rather than relying on implementation summaries. -- [ ] QA executes documented install, helper-mode, upgrade, rollback, reconcile, and troubleshooting commands on L2. -- [ ] A repository-wide terminology/path audit and Markdown link check pass. - -**Dependencies:** Task 23 -**Files likely touched:** Documentation corrections found by the audit; release verification record -**Estimated scope:** M - -### Checkpoint D: Release - -- [ ] Tasks 21-24 complete. -- [ ] Full CI and L2 release report pass. -- [ ] No leaked runtime resources after the fault suite. -- [ ] Architect and QA independently approve documentation accuracy. -- [ ] Human approves the final code, docs, migration, and release decision. - -## Follow-up project: XDG state separation - -Do not include this in the Jailer remediation. After release, write a separate ADR and migration plan to split: - -- durable user metadata into `$XDG_STATE_HOME/mvmctl`; -- cacheable downloads into `$XDG_CACHE_HOME/mvmctl`; -- user sockets/PIDs into `$XDG_RUNTIME_DIR/mvmctl`; -- user configuration into `$XDG_CONFIG_HOME/mvmctl`. - -The follow-up must preserve `MVM_CACHE_DIR` compatibility, migrate atomically, detect partial moves, and support rollback. +**Dependencies:** Tasks 17 and 18 + +- [ ] Update `CONTEXT.md`, standards, ADR links/statuses, README, dependencies, runtime, resource management, + troubleshooting, improvements, roadmap, system-test architecture, coverage matrix, release QA guide, and changelog. +- [ ] Remove stale iptables, service-access, `allow_remote_exec`, host-global TAP, legacy sudo, migration, state-path, + cgroup-path, command, and service-count claims. +- [ ] Distinguish implemented behavior, verified limitations, and future work. +- [ ] Architect traces every security/operator claim to final code and test evidence. +- [ ] QA executes every documented install, initialization, policy, recovery, and troubleshooting procedure. +- [ ] Human approves the final diff and release decision before tagging v0.3.0. + +## Release completion + +The goal is complete only when Tasks 0-19 are satisfied, the final worktree contains no unintended changes, all release +commits are reviewable and working, CI and Python system-test evidence are recorded, documentation matches verified +behavior, and the user approves the release candidate. From 5cca7acc4bd641021cf31ceab705bb80bcc05267 Mon Sep 17 00:00:00 2001 From: AlanD20 Date: Sun, 23 Aug 2026 12:48:24 +0200 Subject: [PATCH 0011/1457] feat: add root-owned VM authority --- CHANGELOG.md | 21 +- internal/infra/constants.go | 6 + internal/service/jailer/instance_authority.go | 505 +++++++++++ .../service/jailer/instance_authority_test.go | 389 ++++++++ internal/service/jailer/instance_codec.go | 226 +++++ .../service/jailer/instance_codec_test.go | 223 +++++ internal/service/jailer/instance_fs.go | 490 ++++++++++ internal/service/jailer/instance_fs_test.go | 845 ++++++++++++++++++ internal/service/jailer/instance_lock.go | 175 ++++ internal/service/jailer/instance_scan.go | 240 +++++ internal/service/jailer/instance_store.go | 247 +++++ internal/service/jailer/instance_types.go | 246 +++++ tasks/plan.md | 2 + tasks/todo.md | 54 +- 14 files changed, 3637 insertions(+), 32 deletions(-) create mode 100644 internal/service/jailer/instance_authority.go create mode 100644 internal/service/jailer/instance_authority_test.go create mode 100644 internal/service/jailer/instance_codec.go create mode 100644 internal/service/jailer/instance_codec_test.go create mode 100644 internal/service/jailer/instance_fs.go create mode 100644 internal/service/jailer/instance_fs_test.go create mode 100644 internal/service/jailer/instance_lock.go create mode 100644 internal/service/jailer/instance_scan.go create mode 100644 internal/service/jailer/instance_store.go create mode 100644 internal/service/jailer/instance_types.go diff --git a/CHANGELOG.md b/CHANGELOG.md index 7d042141..e10f6288 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,9 +8,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [0.3.0] - Unreleased > **Security release status:** `0.3.0` is still under development. The canonical Jailer, cgroup policy, routed -> service-access policy, root-owned system installer, and early privileged-dispatch foundation described below are -> implemented. The mandatory per-VM network namespace, nftables-only `traffic` policy, exact VM-to-VM `exec` policy, -> final marker-only sudo policy, and remaining typed privileged operations are not implemented yet; see +> service-access policy, root-owned system installer, early privileged-dispatch foundation, and private root-owned VM +> authority substrate described below are implemented. The mandatory per-VM network namespace, nftables-only `traffic` +> policy, exact VM-to-VM `exec` policy, final marker-only sudo policy, and remaining typed privileged operations are not +> implemented yet; see > **Security work still pending** before treating this version as release-ready. ### Added @@ -46,6 +47,16 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Added ADR-0016, which defines the single-binary privilege boundary, trusted state layout, typed operation catalog, clean-install constraints, and fail-closed system-binary replacement behavior. +#### Root-owned VM authority substrate +- Added a private typed VM authority under `internal/service/jailer` with durable owner-bound `registered`, `cleaning`, + and `cleaned` records beneath `/var/lib/mvmctl/instances/` and persistent ownership tombstones. +- Added descriptor-relative, no-follow traversal; strict bounded record decoding; atomic durable replacement; persistent + root-owned locks; and the fixed `release -> global index -> VM` lock order. +- Global VM-ID claims now fail closed on foreign, duplicate, corrupt, unreadable, or inconsistent authority records, and + trusted release removal can acquire a lease only when no active authority record references the exact release. +- This substrate is private and not yet wired into public VM lifecycle operations; that integration remains a release + blocker below. + ### Changed #### VM launch and release identity @@ -102,8 +113,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 The following items are release blockers and are intentionally not described above as completed behavior: -- Bind every privileged VM action to a root-owned global VM-ID ownership claim, lifecycle record, exact process identity, - and per-VM lock. +- Wire every privileged VM action to the implemented root-owned global VM-ID ownership claim, lifecycle record, exact + process identity, and per-VM lock. - Add root-owned managed-network identity and global capacity authority; user SQLite state must not authorize host-global namespace, link, firewall, process, mount, cgroup, or admission decisions. - Replace path-validation-then-reopen behavior with descriptor-pinned managed-resource access and private mount namespaces. diff --git a/internal/infra/constants.go b/internal/infra/constants.go index 7c26da8e..c1d2bb87 100644 --- a/internal/infra/constants.go +++ b/internal/infra/constants.go @@ -58,6 +58,12 @@ const TrustedBinaryRoot = "/var/lib/mvmctl/binaries" // JailerChrootBase is the fixed root-owned base directory for per-VM jails. const JailerChrootBase = "/var/lib/mvmctl/jailer" +// JailerInstanceRoot contains persistent root-owned VM authority records. +const JailerInstanceRoot = "/var/lib/mvmctl/instances" + +// JailerRuntimeRoot contains ephemeral root-owned locks and launch state. +const JailerRuntimeRoot = "/run/mvmctl" + // JailerCgroupParent is the fixed cgroup-v2 subtree used by canonical Jailer launches. const JailerCgroupParent = "mvmctl" diff --git a/internal/service/jailer/instance_authority.go b/internal/service/jailer/instance_authority.go new file mode 100644 index 00000000..e513a975 --- /dev/null +++ b/internal/service/jailer/instance_authority.go @@ -0,0 +1,505 @@ +package jailer + +import ( + "context" + "errors" + + "mvmctl/pkg/errs" +) + +type instanceAuthority struct { + deps instanceAuthorityDeps + policy instanceAuthorityPolicy +} + +type launchLease struct { + roots *instanceAuthorityRoots + directories *instanceUIDDirectories + releaseLock *authorityLock + vmLock *authorityLock + record instanceRecord +} + +type registeredLease struct { + roots *instanceAuthorityRoots + directories *instanceUIDDirectories + vmLock *authorityLock + record instanceRecord +} + +type cleanupLease struct { + roots *instanceAuthorityRoots + directories *instanceUIDDirectories + vmLock *authorityLock + record instanceRecord +} + +type releaseLease struct { + roots *instanceAuthorityRoots + releaseLock *authorityLock +} + +type lockedInstance struct { + roots *instanceAuthorityRoots + directories *instanceUIDDirectories + vmLock *authorityLock + record instanceRecord +} + +func newInstanceAuthority() *instanceAuthority { + return newInstanceAuthorityWithPolicy(realInstanceAuthorityDeps(), productionInstanceAuthorityPolicy()) +} + +func newInstanceAuthorityWithPolicy( + deps instanceAuthorityDeps, + policy instanceAuthorityPolicy, +) *instanceAuthority { + return &instanceAuthority{deps: deps, policy: policy} +} + +func (authority *instanceAuthority) RegisterLaunch( + ctx context.Context, + caller instanceCaller, + registration launchRegistration, +) (_ *launchLease, returnErr error) { + if err := validateInstanceCaller(caller); err != nil { + return nil, err + } + if err := validateLaunchRegistration(registration); err != nil { + return nil, err + } + roots, err := openInstanceAuthorityRoots(ctx, authority.deps, authority.policy) + if err != nil { + return nil, err + } + var releaseLock, indexLock, vmLock *authorityLock + var directories *instanceUIDDirectories + defer func() { + if returnErr == nil { + return + } + returnErr = releaseInstanceOperationResources( + ctx, + returnErr, + vmLock, + indexLock, + releaseLock, + directories, + roots, + ) + }() + + releaseLock, err = roots.acquireReleaseLock(ctx, registration.release) + if err != nil { + return nil, err + } + indexLock, err = roots.acquireIndexLock(ctx) + if err != nil { + return nil, err + } + location, err := roots.findInstance(ctx, registration.vmID) + if err != nil { + return nil, err + } + if location != nil && location.uid != caller.uid { + if closeErr := authority.deps.close(context.WithoutCancel(ctx), location.stateFD); closeErr != nil { + return nil, appendInstanceOperationError( + unauthorizedInstanceError(registration.vmID), + "close foreign instance authority directory", + closeErr, + ) + } + return nil, unauthorizedInstanceError(registration.vmID) + } + var existing *instanceRecord + if location == nil { + directories, err = roots.openUIDDirectories(ctx, caller.uid) + if err != nil { + return nil, err + } + } else { + directories, err = roots.openRuntimeForLocation(ctx, location) + if err != nil { + return nil, err + } + } + vmLock, err = directories.acquireVMLock(ctx, registration.vmID) + if err != nil { + return nil, err + } + if location != nil { + record, found, readErr := directories.readRecord(ctx, registration.vmID) + if readErr != nil { + return nil, readErr + } + if !found { + return nil, instanceAtomicError("instance authority record disappeared while locked", nil) + } + existing = &record + } + record, err := registerInstanceRecord(caller, registration, existing) + if err != nil { + return nil, err + } + if err := directories.writeRecord(ctx, record); err != nil { + return nil, err + } + if err := indexLock.Release(ctx); err != nil { + indexLock = nil + return nil, annotateInstanceRecordReplacement(err, false) + } + indexLock = nil + lease := &launchLease{ + roots: roots, + directories: directories, + releaseLock: releaseLock, + vmLock: vmLock, + record: record, + } + roots = nil + directories = nil + releaseLock = nil + vmLock = nil + return lease, nil +} + +func (authority *instanceAuthority) LockRegistered( + ctx context.Context, + caller instanceCaller, + vmID string, +) (*registeredLease, error) { + locked, err := authority.lockOwnedInstance(ctx, caller, vmID) + if err != nil { + return nil, err + } + if locked.record.lifecycle != instanceLifecycleRegistered { + releaseErr := locked.Release(ctx) + stateErr := errs.New( + errs.CodeVMNotRunning, + "vm has no registered privileged runtime", + errs.WithEntity(vmID), + ) + return nil, appendInstanceOperationError(stateErr, "release inactive instance lease", releaseErr) + } + return ®isteredLease{ + roots: locked.roots, + directories: locked.directories, + vmLock: locked.vmLock, + record: locked.record, + }, nil +} + +func (authority *instanceAuthority) BeginCleanup( + ctx context.Context, + caller instanceCaller, + vmID string, +) (*cleanupLease, error) { + locked, err := authority.lockOwnedInstance(ctx, caller, vmID) + if err != nil { + return nil, err + } + updated, err := beginInstanceCleanup(caller, locked.record) + if err != nil { + return nil, appendInstanceOperationError( + err, + "release rejected cleanup lease", + locked.Release(ctx), + ) + } + if updated != locked.record { + if err := locked.directories.writeRecord(ctx, updated); err != nil { + return nil, appendInstanceOperationError( + err, + "release failed cleanup registration", + locked.Release(ctx), + ) + } + } + return &cleanupLease{ + roots: locked.roots, + directories: locked.directories, + vmLock: locked.vmLock, + record: updated, + }, nil +} + +func (lease *cleanupLease) Complete(ctx context.Context) error { + if lease == nil || lease.vmLock == nil || lease.vmLock.fd < 0 { + return instanceStateError("", "cleanup lease is not active") + } + updated, err := completeInstanceCleanup(lease.record) + if err != nil { + return err + } + if err := lease.directories.writeRecord(ctx, updated); err != nil { + if domainErr := errs.AsDomainError(err); domainErr != nil { + if replaced, _ := domainErr.Details["record_replaced"].(bool); replaced { + lease.record = updated + } + } + return err + } + lease.record = updated + return nil +} + +func (authority *instanceAuthority) LockUnreferencedRelease( + ctx context.Context, + release releaseIdentity, +) (_ *releaseLease, returnErr error) { + if err := validateReleaseIdentityValue(release); err != nil { + return nil, instanceValidationError(err.Error()) + } + roots, err := openInstanceAuthorityRoots(ctx, authority.deps, authority.policy) + if err != nil { + return nil, err + } + var releaseLock, indexLock *authorityLock + defer func() { + if returnErr == nil { + return + } + returnErr = releaseInstanceOperationResources( + ctx, + returnErr, + nil, + indexLock, + releaseLock, + nil, + roots, + ) + }() + releaseLock, err = roots.acquireReleaseLock(ctx, release) + if err != nil { + return nil, err + } + indexLock, err = roots.acquireIndexLock(ctx) + if err != nil { + return nil, err + } + referenced, err := roots.releaseIsReferenced(ctx, release) + if err != nil { + return nil, err + } + if referenced { + return nil, errs.New( + errs.CodeVMStateInvalid, + "release is referenced by active privileged VM authority", + errs.WithClass(errs.ClassConflict), + ) + } + if err := indexLock.Release(ctx); err != nil { + indexLock = nil + return nil, err + } + indexLock = nil + lease := &releaseLease{roots: roots, releaseLock: releaseLock} + roots = nil + releaseLock = nil + return lease, nil +} + +func (authority *instanceAuthority) lockOwnedInstance( + ctx context.Context, + caller instanceCaller, + vmID string, +) (_ *lockedInstance, returnErr error) { + if err := validateInstanceCaller(caller); err != nil { + return nil, err + } + if !vmIDPattern.MatchString(vmID) { + return nil, instanceValidationError("vm ID is invalid") + } + roots, err := openInstanceAuthorityRoots(ctx, authority.deps, authority.policy) + if err != nil { + return nil, err + } + var indexLock, vmLock *authorityLock + var directories *instanceUIDDirectories + defer func() { + if returnErr == nil { + return + } + returnErr = releaseInstanceOperationResources( + ctx, + returnErr, + vmLock, + indexLock, + nil, + directories, + roots, + ) + }() + indexLock, err = roots.acquireIndexLock(ctx) + if err != nil { + return nil, err + } + location, err := roots.findInstance(ctx, vmID) + if err != nil { + return nil, err + } + if location == nil { + return nil, errs.NotFound(errs.CodeVMNotFound, "vm has no privileged authority", errs.WithEntity(vmID)) + } + if location.uid != caller.uid { + if closeErr := authority.deps.close(context.WithoutCancel(ctx), location.stateFD); closeErr != nil { + return nil, appendInstanceOperationError( + unauthorizedInstanceError(vmID), + "close foreign instance authority directory", + closeErr, + ) + } + return nil, unauthorizedInstanceError(vmID) + } + directories, err = roots.openRuntimeForLocation(ctx, location) + if err != nil { + return nil, err + } + vmLock, err = directories.acquireVMLock(ctx, vmID) + if err != nil { + return nil, err + } + record, found, err := directories.readRecord(ctx, vmID) + if err != nil { + return nil, err + } + if !found { + return nil, instanceAtomicError("instance authority record disappeared while locked", nil) + } + if err := authorizeInstanceRecord(caller, record); err != nil { + return nil, err + } + if err := indexLock.Release(ctx); err != nil { + indexLock = nil + return nil, err + } + indexLock = nil + locked := &lockedInstance{roots: roots, directories: directories, vmLock: vmLock, record: record} + roots = nil + directories = nil + vmLock = nil + return locked, nil +} + +func unauthorizedInstanceError(vmID string) *errs.DomainError { + return errs.New( + errs.CodeUnauthorized, + "vm privileged authority belongs to another user", + errs.WithEntity(vmID), + ) +} + +func (lease *launchLease) Release(ctx context.Context) error { + if lease == nil { + return nil + } + err := releaseInstanceOperationResources( + ctx, + nil, + lease.vmLock, + nil, + lease.releaseLock, + lease.directories, + lease.roots, + ) + lease.vmLock = nil + lease.releaseLock = nil + lease.directories = nil + lease.roots = nil + return err +} + +func (lease *registeredLease) Release(ctx context.Context) error { + if lease == nil { + return nil + } + err := releaseInstanceOperationResources(ctx, nil, lease.vmLock, nil, nil, lease.directories, lease.roots) + lease.vmLock = nil + lease.directories = nil + lease.roots = nil + return err +} + +func (lease *cleanupLease) Release(ctx context.Context) error { + if lease == nil { + return nil + } + err := releaseInstanceOperationResources(ctx, nil, lease.vmLock, nil, nil, lease.directories, lease.roots) + lease.vmLock = nil + lease.directories = nil + lease.roots = nil + return err +} + +func (lease *releaseLease) Release(ctx context.Context) error { + if lease == nil { + return nil + } + err := releaseInstanceOperationResources(ctx, nil, nil, nil, lease.releaseLock, nil, lease.roots) + lease.releaseLock = nil + lease.roots = nil + return err +} + +func (locked *lockedInstance) Release(ctx context.Context) error { + if locked == nil { + return nil + } + err := releaseInstanceOperationResources(ctx, nil, locked.vmLock, nil, nil, locked.directories, locked.roots) + locked.vmLock = nil + locked.directories = nil + locked.roots = nil + return err +} + +func releaseInstanceOperationResources( + ctx context.Context, + primary error, + vmLock *authorityLock, + indexLock *authorityLock, + releaseLock *authorityLock, + directories *instanceUIDDirectories, + roots *instanceAuthorityRoots, +) error { + cleanupCtx := context.WithoutCancel(ctx) + result := primary + if vmLock != nil { + result = appendInstanceOperationError(result, "release VM lock", vmLock.Release(cleanupCtx)) + } + if indexLock != nil { + result = appendInstanceOperationError(result, "release global index lock", indexLock.Release(cleanupCtx)) + } + if releaseLock != nil { + result = appendInstanceOperationError(result, "release release lock", releaseLock.Release(cleanupCtx)) + } + if directories != nil { + result = appendInstanceOperationError( + result, + "release instance UID directories", + directories.Release(cleanupCtx), + ) + } + if roots != nil { + result = appendInstanceOperationError( + result, + "release instance authority roots", + roots.Release(cleanupCtx), + ) + } + return result +} + +func appendInstanceOperationError(primary error, message string, cause error) error { + if cause == nil { + return primary + } + if primary == nil { + return cause + } + if domainErr := errs.AsDomainError(primary); domainErr != nil { + domainErr.Message += "; " + message + ": " + cause.Error() + domainErr.Err = errors.Join(domainErr.Err, cause) + return domainErr + } + return instanceAtomicError(message+": "+cause.Error(), errors.Join(primary, cause)) +} diff --git a/internal/service/jailer/instance_authority_test.go b/internal/service/jailer/instance_authority_test.go new file mode 100644 index 00000000..557e00fc --- /dev/null +++ b/internal/service/jailer/instance_authority_test.go @@ -0,0 +1,389 @@ +package jailer + +import ( + "context" + "errors" + "os" + "path/filepath" + "sync" + "sync/atomic" + "testing" + "time" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + "golang.org/x/sys/unix" + + "mvmctl/pkg/errs" +) + +func TestInstanceAuthorityRegisterLaunchAndLockRegistered(t *testing.T) { + t.Parallel() + + authority, root := newTestInstanceAuthority(t) + caller := instanceCaller{uid: testAuthorityUID} + registration := testLaunchRegistration() + + launch, err := authority.RegisterLaunch(context.Background(), caller, registration) + require.NoError(t, err) + assert.Equal(t, instanceLifecycleRegistered, launch.record.lifecycle) + assert.FileExists(t, filepath.Join(root, "var/lib/mvmctl/instances/1000/"+testVMID+".json")) + + waitCtx, cancel := context.WithTimeout(context.Background(), 30*time.Millisecond) + defer cancel() + blocked, err := authority.LockRegistered(waitCtx, caller, testVMID) + assert.Nil(t, blocked) + require.Error(t, err) + assert.ErrorIs(t, err, context.DeadlineExceeded) + require.NoError(t, launch.Release(context.Background())) + + registered, err := authority.LockRegistered(context.Background(), caller, testVMID) + require.NoError(t, err) + assert.Equal(t, registration.release, registered.record.release) + assert.Equal(t, registration.process, registered.record.process) + require.NoError(t, registered.Release(context.Background())) + + duplicate, err := authority.RegisterLaunch(context.Background(), caller, registration) + assert.Nil(t, duplicate) + require.Error(t, err) + assert.Equal(t, errs.CodeVMAlreadyExists, errs.AsDomainError(err).Code) +} + +func TestInstanceAuthorityCleanupAndSameOwnerRelaunch(t *testing.T) { + t.Parallel() + + authority, _ := newTestInstanceAuthority(t) + caller := instanceCaller{uid: testAuthorityUID} + launch, err := authority.RegisterLaunch(context.Background(), caller, testLaunchRegistration()) + require.NoError(t, err) + require.NoError(t, launch.Release(context.Background())) + + cleanup, err := authority.BeginCleanup(context.Background(), caller, testVMID) + require.NoError(t, err) + assert.Equal(t, instanceLifecycleCleaning, cleanup.record.lifecycle) + assert.Equal(t, uint64(1), cleanup.record.cleanupGeneration) + require.NoError(t, cleanup.Complete(context.Background())) + assert.Equal(t, instanceLifecycleCleaned, cleanup.record.lifecycle) + require.NoError(t, cleanup.Release(context.Background())) + + registered, err := authority.LockRegistered(context.Background(), caller, testVMID) + assert.Nil(t, registered) + require.Error(t, err) + assert.Equal(t, errs.CodeVMNotRunning, errs.AsDomainError(err).Code) + + relaunched, err := authority.RegisterLaunch(context.Background(), caller, testLaunchRegistration()) + require.NoError(t, err) + assert.Equal(t, uint64(1), relaunched.record.cleanupGeneration) + require.NoError(t, relaunched.Release(context.Background())) +} + +func TestInstanceAuthorityRejectsForeignCaller(t *testing.T) { + t.Parallel() + + authority, _ := newTestInstanceAuthority(t) + owner := instanceCaller{uid: testAuthorityUID} + foreign := instanceCaller{uid: testAuthorityUID + 1} + launch, err := authority.RegisterLaunch(context.Background(), owner, testLaunchRegistration()) + require.NoError(t, err) + require.NoError(t, launch.Release(context.Background())) + + foreignLaunch, err := authority.RegisterLaunch(context.Background(), foreign, testLaunchRegistration()) + assert.Nil(t, foreignLaunch) + require.Error(t, err) + assert.Equal(t, errs.CodeUnauthorized, errs.AsDomainError(err).Code) + assert.NotContains(t, err.Error(), "1000") + + registered, err := authority.LockRegistered(context.Background(), foreign, testVMID) + assert.Nil(t, registered) + require.Error(t, err) + assert.Equal(t, errs.CodeUnauthorized, errs.AsDomainError(err).Code) + + cleanup, err := authority.BeginCleanup(context.Background(), foreign, testVMID) + assert.Nil(t, cleanup) + require.Error(t, err) + assert.Equal(t, errs.CodeUnauthorized, errs.AsDomainError(err).Code) +} + +func TestInstanceAuthorityCrossUIDFirstLaunchRaceHasOneOwner(t *testing.T) { + t.Parallel() + + authority, root := newTestInstanceAuthority(t) + callers := []instanceCaller{{uid: 1000}, {uid: 1001}} + start := make(chan struct{}) + results := make(chan error, len(callers)) + var success atomic.Int32 + for _, caller := range callers { + go func() { + <-start + lease, err := authority.RegisterLaunch(context.Background(), caller, testLaunchRegistration()) + if err == nil { + success.Add(1) + err = lease.Release(context.Background()) + } + results <- err + }() + } + close(start) + + errorsSeen := make([]error, 0, len(callers)) + for range callers { + errorsSeen = append(errorsSeen, <-results) + } + assert.Equal(t, int32(1), success.Load()) + unauthorized := 0 + for _, err := range errorsSeen { + if domainErr := errs.AsDomainError(err); domainErr != nil && domainErr.Code == errs.CodeUnauthorized { + unauthorized++ + } + } + assert.Equal(t, 1, unauthorized) + + records, err := filepath.Glob(filepath.Join(root, "var/lib/mvmctl/instances/*/"+testVMID+".json")) + require.NoError(t, err) + assert.Len(t, records, 1) +} + +func TestInstanceAuthorityRejectsDuplicateGlobalClaim(t *testing.T) { + t.Parallel() + + authority, root := newTestInstanceAuthority(t) + writeTestAuthorityRecord(t, root, testRegisteredInstanceRecord()) + duplicate := testRegisteredInstanceRecord() + duplicate.ownerUID = testAuthorityUID + 1 + writeTestAuthorityRecord(t, root, duplicate) + + lease, err := authority.LockRegistered( + context.Background(), + instanceCaller{uid: testAuthorityUID}, + testVMID, + ) + assert.Nil(t, lease) + require.Error(t, err) + assert.Equal(t, errs.CodeVMAtomicFailed, errs.AsDomainError(err).Code) +} + +func TestInstanceAuthorityRejectsCorruptGlobalClaim(t *testing.T) { + t.Parallel() + + authority, root := newTestInstanceAuthority(t) + recordDir := filepath.Join(root, "var/lib/mvmctl/instances/1000") + require.NoError(t, os.MkdirAll(recordDir, 0700)) + require.NoError(t, os.WriteFile(filepath.Join(recordDir, testVMID+".json"), []byte(`{"broken":true}`), 0600)) + + lease, err := authority.RegisterLaunch( + context.Background(), + instanceCaller{uid: testAuthorityUID + 1}, + testLaunchRegistration(), + ) + assert.Nil(t, lease) + require.Error(t, err) + assert.Equal(t, errs.CodeVMAtomicFailed, errs.AsDomainError(err).Code) +} + +func TestInstanceAuthorityRejectsRecordDirectoryIdentityMismatch(t *testing.T) { + t.Parallel() + + authority, root := newTestInstanceAuthority(t) + record := testRegisteredInstanceRecord() + record.ownerUID = testAuthorityUID + 1 + raw, err := encodeInstanceRecord(record) + require.NoError(t, err) + recordDir := filepath.Join(root, "var/lib/mvmctl/instances/1000") + require.NoError(t, os.MkdirAll(recordDir, 0700)) + require.NoError(t, os.WriteFile(filepath.Join(recordDir, testVMID+".json"), raw, 0600)) + + lease, err := authority.LockRegistered( + context.Background(), + instanceCaller{uid: testAuthorityUID}, + testVMID, + ) + assert.Nil(t, lease) + require.Error(t, err) + assert.Equal(t, errs.CodeVMAtomicFailed, errs.AsDomainError(err).Code) +} + +func TestInstanceAuthorityReleaseReferenceLeaseBlocksLaunch(t *testing.T) { + t.Parallel() + + authority, _ := newTestInstanceAuthority(t) + caller := instanceCaller{uid: testAuthorityUID} + release := testLaunchRegistration().release + launch, err := authority.RegisterLaunch(context.Background(), caller, testLaunchRegistration()) + require.NoError(t, err) + require.NoError(t, launch.Release(context.Background())) + + releaseLease, err := authority.LockUnreferencedRelease(context.Background(), release) + assert.Nil(t, releaseLease) + require.Error(t, err) + assert.Equal(t, errs.ClassConflict, errs.AsDomainError(err).Class) + + cleanup, err := authority.BeginCleanup(context.Background(), caller, testVMID) + require.NoError(t, err) + require.NoError(t, cleanup.Complete(context.Background())) + require.NoError(t, cleanup.Release(context.Background())) + + releaseLease, err = authority.LockUnreferencedRelease(context.Background(), release) + require.NoError(t, err) + waitCtx, cancel := context.WithTimeout(context.Background(), 30*time.Millisecond) + defer cancel() + blocked, err := authority.RegisterLaunch(waitCtx, caller, testLaunchRegistration()) + assert.Nil(t, blocked) + require.Error(t, err) + assert.ErrorIs(t, err, context.DeadlineExceeded) + require.NoError(t, releaseLease.Release(context.Background())) +} + +func TestInstanceAuthorityReleaseLeaseRejectsCorruptAuthorityRecord(t *testing.T) { + t.Parallel() + + authority, root := newTestInstanceAuthority(t) + recordDir := filepath.Join(root, "var/lib/mvmctl/instances/1000") + require.NoError(t, os.MkdirAll(recordDir, 0700)) + require.NoError(t, os.WriteFile(filepath.Join(recordDir, testVMID+".json"), []byte(`{"broken":true}`), 0600)) + + lease, err := authority.LockUnreferencedRelease(context.Background(), testLaunchRegistration().release) + assert.Nil(t, lease) + require.Error(t, err) + assert.Equal(t, errs.CodeVMAtomicFailed, errs.AsDomainError(err).Code) +} + +func TestInstanceAuthorityRegisterLockOrder(t *testing.T) { + t.Parallel() + + root := prepareInstanceAuthorityTestRoot(t) + deps := realInstanceAuthorityDeps() + realOpenAt := deps.openAt + realFlock := deps.flock + namesByFD := make(map[int]string) + var namesMu sync.Mutex + deps.openAt = func(ctx context.Context, parentFD int, name string, flags int, mode uint32) (int, error) { + fd, err := realOpenAt(ctx, parentFD, name, flags, mode) + if err == nil { + namesMu.Lock() + namesByFD[fd] = name + namesMu.Unlock() + } + return fd, err + } + acquired := make([]string, 0, 3) + deps.flock = func(ctx context.Context, fd int, how int) error { + err := realFlock(ctx, fd, how) + if err == nil && how == unix.LOCK_EX|unix.LOCK_NB { + namesMu.Lock() + acquired = append(acquired, namesByFD[fd]) + namesMu.Unlock() + } + return err + } + authority := newInstanceAuthorityWithPolicy(deps, testInstanceAuthorityPolicy(root)) + + lease, err := authority.RegisterLaunch( + context.Background(), + instanceCaller{uid: testAuthorityUID}, + testLaunchRegistration(), + ) + require.NoError(t, err) + require.NoError(t, lease.Release(context.Background())) + + require.Len(t, acquired, 3) + assert.Equal(t, "index.lock", acquired[1]) + assert.Equal(t, testVMID+".lock", acquired[2]) + assert.NotEqual(t, "index.lock", acquired[0]) +} + +func TestInstanceAuthorityUsesOnlyFixedRootPathOpen(t *testing.T) { + t.Parallel() + + root := prepareInstanceAuthorityTestRoot(t) + deps := realInstanceAuthorityDeps() + realOpen := deps.open + opened := make([]string, 0, 1) + deps.open = func(ctx context.Context, path string, flags int, mode uint32) (int, error) { + opened = append(opened, path) + return realOpen(ctx, path, flags, mode) + } + authority := newInstanceAuthorityWithPolicy(deps, testInstanceAuthorityPolicy(root)) + + lease, err := authority.RegisterLaunch( + context.Background(), + instanceCaller{uid: testAuthorityUID}, + testLaunchRegistration(), + ) + require.NoError(t, err) + require.NoError(t, lease.Release(context.Background())) + assert.Equal(t, []string{root}, opened) +} + +func TestInstanceAuthorityPostRenameErrorReturnsNoLeaseAndDurableRecord(t *testing.T) { + t.Parallel() + + root := prepareInstanceAuthorityTestRoot(t) + prepared := openTestInstanceAuthorityRoots(t, root) + uidDirs, err := prepared.openUIDDirectories(context.Background(), testAuthorityUID) + require.NoError(t, err) + releaseLock, err := prepared.acquireReleaseLock(context.Background(), testLaunchRegistration().release) + require.NoError(t, err) + indexLock, err := prepared.acquireIndexLock(context.Background()) + require.NoError(t, err) + vmLock, err := uidDirs.acquireVMLock(context.Background(), testVMID) + require.NoError(t, err) + require.NoError(t, vmLock.Release(context.Background())) + require.NoError(t, indexLock.Release(context.Background())) + require.NoError(t, releaseLock.Release(context.Background())) + require.NoError(t, uidDirs.Release(context.Background())) + require.NoError(t, prepared.Release(context.Background())) + deps := realInstanceAuthorityDeps() + realFsync := deps.fsync + var fileSynced atomic.Bool + deps.fsync = func(ctx context.Context, fd int) error { + var stat unix.Stat_t + if err := unix.Fstat(fd, &stat); err == nil && stat.Mode&unix.S_IFMT == unix.S_IFREG { + fileSynced.Store(true) + return realFsync(ctx, fd) + } + if fileSynced.Load() { + return errors.New("record directory sync failed") + } + return realFsync(ctx, fd) + } + authority := newInstanceAuthorityWithPolicy(deps, testInstanceAuthorityPolicy(root)) + + lease, err := authority.RegisterLaunch( + context.Background(), + instanceCaller{uid: testAuthorityUID}, + testLaunchRegistration(), + ) + assert.Nil(t, lease) + require.Error(t, err) + domainErr := errs.AsDomainError(err) + require.NotNil(t, domainErr) + assert.Equal(t, true, domainErr.Details["record_replaced"]) + assert.Equal(t, true, domainErr.Details["durability_uncertain"]) + assert.FileExists(t, filepath.Join(root, "var/lib/mvmctl/instances/1000/"+testVMID+".json")) +} + +func newTestInstanceAuthority(t *testing.T) (*instanceAuthority, string) { + t.Helper() + + root := prepareInstanceAuthorityTestRoot(t) + return newInstanceAuthorityWithPolicy( + realInstanceAuthorityDeps(), + testInstanceAuthorityPolicy(root), + ), root +} + +func writeTestAuthorityRecord(t *testing.T, root string, record instanceRecord) { + t.Helper() + + roots, err := openInstanceAuthorityRoots( + context.Background(), + realInstanceAuthorityDeps(), + testInstanceAuthorityPolicy(root), + ) + require.NoError(t, err) + uidDirs, err := roots.openUIDDirectories(context.Background(), record.ownerUID) + require.NoError(t, err) + require.NoError(t, uidDirs.writeRecord(context.Background(), record)) + require.NoError(t, uidDirs.Release(context.Background())) + require.NoError(t, roots.Release(context.Background())) +} diff --git a/internal/service/jailer/instance_codec.go b/internal/service/jailer/instance_codec.go new file mode 100644 index 00000000..7b848b19 --- /dev/null +++ b/internal/service/jailer/instance_codec.go @@ -0,0 +1,226 @@ +package jailer + +import ( + "bytes" + "encoding/json" + "fmt" + "io" + "strings" +) + +const maxInstanceJSONDepth = 4 + +type instanceRecordWire struct { + SchemaVersion uint32 `json:"schema_version"` + OwnerUID uint32 `json:"owner_uid"` + VMID string `json:"vm_id"` + Lifecycle instanceLifecycle `json:"lifecycle"` + Release releaseIdentityWire `json:"release"` + Process processIdentityWire `json:"process"` + CleanupGeneration uint64 `json:"cleanup_generation"` +} + +type releaseIdentityWire struct { + Version string `json:"version"` + Architecture string `json:"architecture"` + FirecrackerSHA256 string `json:"firecracker_sha256"` + JailerSHA256 string `json:"jailer_sha256"` +} + +type processIdentityWire struct { + PID int `json:"pid"` + StartTimeTicks uint64 `json:"start_time_ticks"` + CgroupPath string `json:"cgroup_path"` +} + +var instanceObjectFields = map[string]map[string]struct{}{ + "": fields( + "schema_version", + "owner_uid", + "vm_id", + "lifecycle", + "release", + "process", + "cleanup_generation", + ), + "release": fields("version", "architecture", "firecracker_sha256", "jailer_sha256"), + "process": fields("pid", "start_time_ticks", "cgroup_path"), +} + +func encodeInstanceRecord(record instanceRecord) ([]byte, error) { + if err := validateInstanceRecord(record); err != nil { + return nil, err + } + raw, err := json.Marshal(instanceRecordToWire(record)) + if err != nil { + return nil, instanceAtomicError("encode instance authority record", err) + } + if len(raw) > maxInstanceRecordBytes { + return nil, instanceAtomicError("encoded instance authority record exceeds size limit", nil) + } + return raw, nil +} + +func decodeInstanceRecord(raw []byte) (instanceRecord, error) { + if len(raw) == 0 { + return instanceRecord{}, instanceAtomicError("instance authority record is empty", nil) + } + if len(raw) > maxInstanceRecordBytes { + return instanceRecord{}, instanceAtomicError("instance authority record exceeds size limit", nil) + } + if err := validateInstanceJSON(raw); err != nil { + return instanceRecord{}, err + } + decoder := json.NewDecoder(bytes.NewReader(raw)) + decoder.DisallowUnknownFields() + var wire instanceRecordWire + if err := decoder.Decode(&wire); err != nil { + return instanceRecord{}, instanceAtomicError("decode instance authority record", err) + } + if err := requireInstanceJSONEOF(decoder); err != nil { + return instanceRecord{}, err + } + record := instanceRecordFromWire(wire) + if err := validateInstanceRecord(record); err != nil { + return instanceRecord{}, err + } + return record, nil +} + +func validateInstanceJSON(raw []byte) error { + decoder := json.NewDecoder(bytes.NewReader(raw)) + if err := scanInstanceJSONValue(decoder, "", 0); err != nil { + return instanceAtomicError("decode strict instance authority record", err) + } + return requireInstanceJSONEOF(decoder) +} + +func scanInstanceJSONValue(decoder *json.Decoder, objectName string, depth int) error { + if depth > maxInstanceJSONDepth { + return fmt.Errorf("JSON nesting exceeds %d levels", maxInstanceJSONDepth) + } + token, err := decoder.Token() + if err != nil { + return err + } + delim, ok := token.(json.Delim) + if !ok { + return nil + } + if delim != '{' { + return fmt.Errorf("unexpected JSON container %q", delim) + } + allowed, ok := instanceObjectFields[objectName] + if !ok { + return fmt.Errorf("unexpected object %q", objectName) + } + seen := make(map[string]struct{}, len(allowed)) + for decoder.More() { + fieldToken, tokenErr := decoder.Token() + if tokenErr != nil { + return tokenErr + } + field, ok := fieldToken.(string) + if !ok { + return fmt.Errorf("object field is not a string") + } + if _, exists := allowed[field]; !exists { + return fmt.Errorf("unknown JSON field %q", field) + } + for existing := range seen { + if strings.EqualFold(existing, field) { + return fmt.Errorf("duplicate JSON field %q conflicts with %q", field, existing) + } + } + seen[field] = struct{}{} + if field == "release" || field == "process" { + if err := scanInstanceJSONValue(decoder, field, depth+1); err != nil { + return err + } + continue + } + if err := scanInstanceJSONScalar(decoder); err != nil { + return err + } + } + if _, err := decoder.Token(); err != nil { + return err + } + if len(seen) != len(allowed) { + return fmt.Errorf("JSON object %q is missing required fields", objectName) + } + return nil +} + +func scanInstanceJSONScalar(decoder *json.Decoder) error { + token, err := decoder.Token() + if err != nil { + return err + } + if _, ok := token.(json.Delim); ok { + return fmt.Errorf("unexpected nested JSON value") + } + return nil +} + +func requireInstanceJSONEOF(decoder *json.Decoder) error { + var trailing json.RawMessage + err := decoder.Decode(&trailing) + if err == io.EOF { + return nil + } + if err == nil { + return instanceAtomicError("instance authority record contains trailing input", nil) + } + return instanceAtomicError("decode trailing instance authority input", err) +} + +func instanceRecordToWire(record instanceRecord) instanceRecordWire { + return instanceRecordWire{ + SchemaVersion: record.schemaVersion, + OwnerUID: record.ownerUID, + VMID: record.vmID, + Lifecycle: record.lifecycle, + Release: releaseIdentityWire{ + Version: record.release.version, + Architecture: record.release.architecture, + FirecrackerSHA256: record.release.firecrackerSHA256, + JailerSHA256: record.release.jailerSHA256, + }, + Process: processIdentityWire{ + PID: record.process.pid, + StartTimeTicks: record.process.startTimeTicks, + CgroupPath: record.process.cgroupPath, + }, + CleanupGeneration: record.cleanupGeneration, + } +} + +func instanceRecordFromWire(wire instanceRecordWire) instanceRecord { + return instanceRecord{ + schemaVersion: wire.SchemaVersion, + ownerUID: wire.OwnerUID, + vmID: wire.VMID, + lifecycle: wire.Lifecycle, + release: releaseIdentity{ + version: wire.Release.Version, + architecture: wire.Release.Architecture, + firecrackerSHA256: wire.Release.FirecrackerSHA256, + jailerSHA256: wire.Release.JailerSHA256, + }, + process: processIdentity{ + pid: wire.Process.PID, + startTimeTicks: wire.Process.StartTimeTicks, + cgroupPath: wire.Process.CgroupPath, + }, + cleanupGeneration: wire.CleanupGeneration, + } +} + +func fields(names ...string) map[string]struct{} { + result := make(map[string]struct{}, len(names)) + for _, name := range names { + result[name] = struct{}{} + } + return result +} diff --git a/internal/service/jailer/instance_codec_test.go b/internal/service/jailer/instance_codec_test.go new file mode 100644 index 00000000..2f47294f --- /dev/null +++ b/internal/service/jailer/instance_codec_test.go @@ -0,0 +1,223 @@ +package jailer + +import ( + "strings" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "mvmctl/internal/infra" + "mvmctl/pkg/errs" +) + +const ( + testAuthorityUID = uint32(1000) + testReleaseHash = "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef" + testJailerHash = "abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789" +) + +func TestInstanceAuthorityRootConstants(t *testing.T) { + t.Parallel() + + assert.Equal(t, "/var/lib/mvmctl/instances", infra.JailerInstanceRoot) + assert.Equal(t, "/run/mvmctl", infra.JailerRuntimeRoot) +} + +func TestInstanceRecordCodecRoundTrip(t *testing.T) { + t.Parallel() + + want := testRegisteredInstanceRecord() + raw, err := encodeInstanceRecord(want) + require.NoError(t, err) + assert.LessOrEqual(t, len(raw), maxInstanceRecordBytes) + + got, err := decodeInstanceRecord(raw) + require.NoError(t, err) + assert.Equal(t, want, got) + assert.NotContains(t, string(raw), "password") + assert.NotContains(t, string(raw), "token") + assert.NotContains(t, string(raw), "rootfs") +} + +func TestDecodeInstanceRecordRejectsMalformedSchema(t *testing.T) { + t.Parallel() + + valid, err := encodeInstanceRecord(testRegisteredInstanceRecord()) + require.NoError(t, err) + expectedCgroup := infra.CgroupV2Root + "/" + infra.JailerCgroupParent + "/" + testVMID + + tests := map[string]string{ + "empty": "", + "trailing": string(valid) + `{}`, + "unknown_top_level": strings.Replace(string(valid), `"vm_id":`, `"unknown":1,"vm_id":`, 1), + "case_variant_field": strings.Replace(string(valid), `"vm_id":`, `"VM_ID":`, 1), + "duplicate_field": strings.Replace(string(valid), `"vm_id":`, `"vm_id":"`+testVMID+`","vm_id":`, 1), + "case_folded_duplicate": strings.Replace(string(valid), `"vm_id":`, `"VM_ID":"`+testVMID+`","vm_id":`, 1), + "nested_unknown": strings.Replace(string(valid), `"version":`, `"unknown":1,"version":`, 1), + "nested_duplicate": strings.Replace(string(valid), `"version":`, `"version":"1.16.0","version":`, 1), + "nested_case_duplicate": strings.Replace(string(valid), `"version":`, `"VERSION":"1.16.0","version":`, 1), + "missing_required": strings.Replace(string(valid), `"owner_uid":1000,`, "", 1), + "unsupported_version": strings.Replace(string(valid), `"schema_version":1`, `"schema_version":2`, 1), + "invalid_lifecycle": strings.Replace(string(valid), `"lifecycle":"registered"`, `"lifecycle":"running"`, 1), + "owner_zero": strings.Replace(string(valid), `"owner_uid":1000`, `"owner_uid":0`, 1), + "pid_zero": strings.Replace(string(valid), `"pid":4242`, `"pid":0`, 1), + "start_ticks_zero": strings.Replace(string(valid), `"start_time_ticks":987654`, `"start_time_ticks":0`, 1), + "wrong_cgroup": strings.Replace( + string(valid), + `"cgroup_path":"`+expectedCgroup+`"`, + `"cgroup_path":"/sys/fs/cgroup/mvmctl/ffffffffffffffffffffffffffffffff"`, + 1, + ), + "uppercase_release_hash": strings.Replace(string(valid), testReleaseHash, strings.ToUpper(testReleaseHash), 1), + } + + for name, raw := range tests { + t.Run(name, func(t *testing.T) { + t.Parallel() + + _, decodeErr := decodeInstanceRecord([]byte(raw)) + require.Error(t, decodeErr) + assert.Equal(t, errs.CodeVMAtomicFailed, errs.AsDomainError(decodeErr).Code) + }) + } +} + +func TestDecodeInstanceRecordRejectsOversizedInput(t *testing.T) { + t.Parallel() + + _, err := decodeInstanceRecord([]byte(strings.Repeat(" ", maxInstanceRecordBytes+1))) + require.Error(t, err) + assert.Equal(t, errs.CodeVMAtomicFailed, errs.AsDomainError(err).Code) +} + +func TestInstanceLifecycleTransitions(t *testing.T) { + t.Parallel() + + caller := instanceCaller{uid: testAuthorityUID} + registration := testLaunchRegistration() + + registered, err := registerInstanceRecord(caller, registration, nil) + require.NoError(t, err) + assert.Equal(t, instanceLifecycleRegistered, registered.lifecycle) + assert.Zero(t, registered.cleanupGeneration) + + _, err = registerInstanceRecord(caller, registration, ®istered) + require.Error(t, err) + assert.Equal(t, errs.CodeVMAlreadyExists, errs.AsDomainError(err).Code) + + cleaning, err := beginInstanceCleanup(caller, registered) + require.NoError(t, err) + assert.Equal(t, instanceLifecycleCleaning, cleaning.lifecycle) + assert.Equal(t, uint64(1), cleaning.cleanupGeneration) + + retry, err := beginInstanceCleanup(caller, cleaning) + require.NoError(t, err) + assert.Equal(t, cleaning, retry) + + cleaned, err := completeInstanceCleanup(cleaning) + require.NoError(t, err) + assert.Equal(t, instanceLifecycleCleaned, cleaned.lifecycle) + + relaunched, err := registerInstanceRecord(caller, registration, &cleaned) + require.NoError(t, err) + assert.Equal(t, instanceLifecycleRegistered, relaunched.lifecycle) + assert.Equal(t, cleaned.cleanupGeneration, relaunched.cleanupGeneration) +} + +func TestInstanceLifecycleRejectsForeignCaller(t *testing.T) { + t.Parallel() + + record := testRegisteredInstanceRecord() + foreign := instanceCaller{uid: testAuthorityUID + 1} + + _, err := registerInstanceRecord(foreign, testLaunchRegistration(), &record) + require.Error(t, err) + assert.Equal(t, errs.CodeUnauthorized, errs.AsDomainError(err).Code) + assert.NotContains(t, err.Error(), "1000") + + _, err = beginInstanceCleanup(foreign, record) + require.Error(t, err) + assert.Equal(t, errs.CodeUnauthorized, errs.AsDomainError(err).Code) +} + +func TestLaunchRegistrationRejectsInvalidReleaseAsValidation(t *testing.T) { + t.Parallel() + + tests := map[string]string{ + "whitespace": "1.16.0 beta", + "control": "1.16.0\nroot", + "dot_component": "1.16..0", + "path_component": "..", + "punctuation": "1.16.0!", + } + + for name, version := range tests { + t.Run(name, func(t *testing.T) { + t.Parallel() + + registration := testLaunchRegistration() + registration.release.version = version + _, err := registerInstanceRecord( + instanceCaller{uid: testAuthorityUID}, + registration, + nil, + ) + require.Error(t, err) + assert.Equal(t, errs.CodeValidationFailed, errs.AsDomainError(err).Code) + }) + } +} + +func TestInstanceTransitionRejectsCorruptExistingRecord(t *testing.T) { + t.Parallel() + + caller := instanceCaller{uid: testAuthorityUID} + corrupt := testRegisteredInstanceRecord() + corrupt.lifecycle = instanceLifecycleCleaned + corrupt.process.cgroupPath = "/sys/fs/cgroup/foreign" + + _, err := registerInstanceRecord(caller, testLaunchRegistration(), &corrupt) + require.Error(t, err) + assert.Equal(t, errs.CodeVMAtomicFailed, errs.AsDomainError(err).Code) + + corrupt.lifecycle = instanceLifecycleRegistered + _, err = beginInstanceCleanup(caller, corrupt) + require.Error(t, err) + assert.Equal(t, errs.CodeVMAtomicFailed, errs.AsDomainError(err).Code) + + corrupt.lifecycle = instanceLifecycleCleaning + _, err = completeInstanceCleanup(corrupt) + require.Error(t, err) + assert.Equal(t, errs.CodeVMAtomicFailed, errs.AsDomainError(err).Code) +} + +func testLaunchRegistration() launchRegistration { + return launchRegistration{ + vmID: testVMID, + release: releaseIdentity{ + version: "1.16.0", + architecture: "x86_64", + firecrackerSHA256: testReleaseHash, + jailerSHA256: testJailerHash, + }, + process: processIdentity{ + pid: 4242, + startTimeTicks: 987654, + cgroupPath: infra.CgroupV2Root + "/" + infra.JailerCgroupParent + "/" + testVMID, + }, + } +} + +func testRegisteredInstanceRecord() instanceRecord { + registration := testLaunchRegistration() + return instanceRecord{ + schemaVersion: instanceRecordSchemaVersion, + ownerUID: testAuthorityUID, + vmID: registration.vmID, + lifecycle: instanceLifecycleRegistered, + release: registration.release, + process: registration.process, + cleanupGeneration: 0, + } +} diff --git a/internal/service/jailer/instance_fs.go b/internal/service/jailer/instance_fs.go new file mode 100644 index 00000000..7f82b91e --- /dev/null +++ b/internal/service/jailer/instance_fs.go @@ -0,0 +1,490 @@ +package jailer + +import ( + "context" + "crypto/rand" + "encoding/hex" + "errors" + "fmt" + "io" + "os" + "path" + "strconv" + "strings" + "time" + + "golang.org/x/sys/unix" + + "mvmctl/internal/infra" + "mvmctl/pkg/errs" +) + +const ( + instanceDirectoryFlags = unix.O_RDONLY | unix.O_DIRECTORY | unix.O_CLOEXEC | unix.O_NOFOLLOW + instanceReadFlags = unix.O_RDONLY | unix.O_CLOEXEC | unix.O_NOFOLLOW + instanceInspectFlags = unix.O_PATH | unix.O_CLOEXEC | unix.O_NOFOLLOW + instanceTempFlags = unix.O_WRONLY | unix.O_CREAT | unix.O_EXCL | unix.O_CLOEXEC | unix.O_NOFOLLOW + instanceLockFlags = unix.O_RDWR | unix.O_CLOEXEC | unix.O_NOFOLLOW + instanceLockCreate = instanceLockFlags | unix.O_CREAT | unix.O_EXCL + + instanceManagedDirMode = uint32(0700) + instanceStateFileMode = uint32(0600) + instanceTempAttempts = 8 + instanceIOChunk = 4096 + instanceLockPoll = 10 * time.Millisecond +) + +type instanceAuthorityDeps struct { + open func(context.Context, string, int, uint32) (int, error) + openAt func(context.Context, int, string, int, uint32) (int, error) + mkdirAt func(context.Context, int, string, uint32) error + fstat func(context.Context, int, *unix.Stat_t) error + fchown func(context.Context, int, int, int) error + fchmod func(context.Context, int, uint32) error + read func(context.Context, int, []byte) (int, error) + write func(context.Context, int, []byte) (int, error) + fsync func(context.Context, int) error + close func(context.Context, int) error + renameAt func(context.Context, int, string, int, string) error + unlinkAt func(context.Context, int, string, int) error + flock func(context.Context, int, int) error + randomName func(context.Context) (string, error) + readDirNames func(context.Context, int) ([]string, error) + waitLock func(context.Context) error +} + +type instanceAuthorityPolicy struct { + rootPath string + expectedUID uint32 + expectedGID uint32 +} + +type instanceAuthorityRoots struct { + deps instanceAuthorityDeps + policy instanceAuthorityPolicy + stateFD int + runtimeFD int + releasesFD int + retained []int +} + +type instanceUIDDirectories struct { + deps instanceAuthorityDeps + policy instanceAuthorityPolicy + uid uint32 + stateFD int + runtimeFD int +} + +func productionInstanceAuthorityPolicy() instanceAuthorityPolicy { + return instanceAuthorityPolicy{ + rootPath: "/", + expectedUID: 0, + expectedGID: 0, + } +} + +func openInstanceAuthorityRoots( + ctx context.Context, + deps instanceAuthorityDeps, + policy instanceAuthorityPolicy, +) (_ *instanceAuthorityRoots, returnErr error) { + if err := ctx.Err(); err != nil { + return nil, instanceAtomicError("open instance authority roots", err) + } + stateNames, err := instanceAuthorityPathComponents(infra.JailerInstanceRoot, 4) + if err != nil { + return nil, err + } + runtimeNames, err := instanceAuthorityPathComponents(infra.JailerRuntimeRoot, 2) + if err != nil { + return nil, err + } + retained := make([]int, 0, 8) + defer func() { + if returnErr == nil { + return + } + if err := closeInstanceFDs(ctx, deps, retained); err != nil { + returnErr = joinInstanceAtomicError(returnErr, "close rejected instance authority roots", err) + } + }() + + rootFD, err := deps.open(ctx, policy.rootPath, instanceDirectoryFlags, 0) + if err != nil { + return nil, instanceAtomicError("open filesystem root for instance authority", err) + } + retained = append(retained, rootFD) + if err := verifyInstanceDirectory(ctx, deps, rootFD, "filesystem root", policy, false); err != nil { + return nil, err + } + + varFD, err := openInstanceDirectory(ctx, deps, rootFD, stateNames[0], false, false, policy) + if err != nil { + return nil, err + } + retained = append(retained, varFD) + libFD, err := openInstanceDirectory(ctx, deps, varFD, stateNames[1], false, false, policy) + if err != nil { + return nil, err + } + retained = append(retained, libFD) + stateBaseFD, err := openInstanceDirectory(ctx, deps, libFD, stateNames[2], true, false, policy) + if err != nil { + return nil, err + } + retained = append(retained, stateBaseFD) + stateFD, err := openInstanceDirectory(ctx, deps, stateBaseFD, stateNames[3], true, true, policy) + if err != nil { + return nil, err + } + retained = append(retained, stateFD) + + runFD, err := openInstanceDirectory(ctx, deps, rootFD, runtimeNames[0], false, false, policy) + if err != nil { + return nil, err + } + retained = append(retained, runFD) + runtimeFD, err := openInstanceDirectory(ctx, deps, runFD, runtimeNames[1], true, true, policy) + if err != nil { + return nil, err + } + retained = append(retained, runtimeFD) + releasesFD, err := openInstanceDirectory(ctx, deps, runtimeFD, "releases", true, true, policy) + if err != nil { + return nil, err + } + retained = append(retained, releasesFD) + + return &instanceAuthorityRoots{ + deps: deps, + policy: policy, + stateFD: stateFD, + runtimeFD: runtimeFD, + releasesFD: releasesFD, + retained: retained, + }, nil +} + +func instanceAuthorityPathComponents(value string, expectedCount int) ([]string, error) { + if !strings.HasPrefix(value, "/") || path.Clean(value) != value { + return nil, instanceAtomicError("instance authority root constant is not canonical", nil) + } + names := strings.Split(strings.TrimPrefix(value, "/"), "/") + if len(names) != expectedCount { + return nil, instanceAtomicError("instance authority root constant has unexpected depth", nil) + } + for _, name := range names { + if name == "" || name == "." || name == ".." { + return nil, instanceAtomicError("instance authority root constant has an unsafe component", nil) + } + } + return names, nil +} + +func (roots *instanceAuthorityRoots) openUIDDirectories( + ctx context.Context, + uid uint32, +) (_ *instanceUIDDirectories, returnErr error) { + if uid == 0 { + return nil, instanceValidationError("instance authority UID must be non-zero") + } + name := strconv.FormatUint(uint64(uid), 10) + stateFD, err := openInstanceDirectory(ctx, roots.deps, roots.stateFD, name, true, true, roots.policy) + if err != nil { + return nil, err + } + runtimeFD, err := openInstanceDirectory(ctx, roots.deps, roots.runtimeFD, name, true, true, roots.policy) + if err != nil { + if closeErr := roots.deps.close(context.WithoutCancel(ctx), stateFD); closeErr != nil { + err = joinInstanceAtomicError(err, "close instance UID state directory", closeErr) + } + return nil, err + } + return &instanceUIDDirectories{ + deps: roots.deps, + policy: roots.policy, + uid: uid, + stateFD: stateFD, + runtimeFD: runtimeFD, + }, nil +} + +func openInstanceDirectory( + ctx context.Context, + deps instanceAuthorityDeps, + parentFD int, + name string, + allowCreate bool, + requireManagedMode bool, + policy instanceAuthorityPolicy, +) (int, error) { + if err := ctx.Err(); err != nil { + return -1, instanceAtomicError("open instance authority directory "+name, err) + } + fd, err := deps.openAt(ctx, parentFD, name, instanceDirectoryFlags, 0) + if err == nil { + if verifyErr := verifyInstanceDirectory(ctx, deps, fd, name, policy, requireManagedMode); verifyErr != nil { + return -1, closeRejectedInstanceFD(ctx, deps, fd, verifyErr) + } + return fd, nil + } + if !errors.Is(err, unix.ENOENT) || !allowCreate { + return -1, instanceAtomicError("open instance authority directory "+name, err) + } + mkdirErr := deps.mkdirAt(ctx, parentFD, name, instanceManagedDirMode) + created := mkdirErr == nil + if mkdirErr != nil && !errors.Is(mkdirErr, unix.EEXIST) { + return -1, instanceAtomicError("create instance authority directory "+name, mkdirErr) + } + fd, err = deps.openAt(ctx, parentFD, name, instanceDirectoryFlags, 0) + if err != nil { + return -1, instanceAtomicError("open created instance authority directory "+name, err) + } + if !created { + if verifyErr := verifyInstanceDirectory(ctx, deps, fd, name, policy, requireManagedMode); verifyErr != nil { + return -1, closeRejectedInstanceFD(ctx, deps, fd, verifyErr) + } + return fd, nil + } + if err := deps.fchown(ctx, fd, int(policy.expectedUID), int(policy.expectedGID)); err != nil { + return -1, closeRejectedInstanceFD( + ctx, + deps, + fd, + instanceAtomicError("set instance authority directory owner "+name, err), + ) + } + if err := deps.fchmod(ctx, fd, instanceManagedDirMode); err != nil { + return -1, closeRejectedInstanceFD( + ctx, + deps, + fd, + instanceAtomicError("set instance authority directory mode "+name, err), + ) + } + if err := verifyInstanceDirectory(ctx, deps, fd, name, policy, true); err != nil { + return -1, closeRejectedInstanceFD(ctx, deps, fd, err) + } + if err := deps.fsync(ctx, fd); err != nil { + return -1, closeRejectedInstanceFD( + ctx, + deps, + fd, + instanceAtomicError("sync created instance authority directory "+name, err), + ) + } + if err := deps.fsync(ctx, parentFD); err != nil { + return -1, closeRejectedInstanceFD( + ctx, + deps, + fd, + instanceAtomicError("sync parent of instance authority directory "+name, err), + ) + } + return fd, nil +} + +func verifyInstanceDirectory( + ctx context.Context, + deps instanceAuthorityDeps, + fd int, + name string, + policy instanceAuthorityPolicy, + requireManagedMode bool, +) error { + var stat unix.Stat_t + if err := deps.fstat(ctx, fd, &stat); err != nil { + return instanceAtomicError("inspect instance authority directory "+name, err) + } + if stat.Mode&unix.S_IFMT != unix.S_IFDIR { + return instanceAtomicError("instance authority path "+name+" is not a directory", nil) + } + if stat.Uid != policy.expectedUID || stat.Mode&0022 != 0 { + return instanceAtomicError("instance authority directory "+name+" has unsafe ownership or mode", nil) + } + if requireManagedMode && (stat.Gid != policy.expectedGID || stat.Mode&07777 != instanceManagedDirMode) { + return instanceAtomicError("managed instance authority directory "+name+" has unexpected owner or mode", nil) + } + return nil +} + +func (roots *instanceAuthorityRoots) Release(ctx context.Context) error { + if roots == nil || len(roots.retained) == 0 { + return nil + } + err := closeInstanceFDs(context.WithoutCancel(ctx), roots.deps, roots.retained) + roots.retained = nil + if err != nil { + return instanceAtomicError("release instance authority root descriptors", err) + } + return nil +} + +func (directories *instanceUIDDirectories) Release(ctx context.Context) error { + if directories == nil { + return nil + } + cleanupCtx := context.WithoutCancel(ctx) + var result error + if directories.runtimeFD >= 0 { + result = errors.Join(result, directories.deps.close(cleanupCtx, directories.runtimeFD)) + directories.runtimeFD = -1 + } + if directories.stateFD >= 0 { + result = errors.Join(result, directories.deps.close(cleanupCtx, directories.stateFD)) + directories.stateFD = -1 + } + if result != nil { + return instanceAtomicError("release instance UID directory descriptors", result) + } + return nil +} + +func closeRejectedInstanceFD(ctx context.Context, deps instanceAuthorityDeps, fd int, cause error) error { + if err := deps.close(context.WithoutCancel(ctx), fd); err != nil { + return joinInstanceAtomicError(cause, "close rejected instance authority descriptor", err) + } + return cause +} + +func closeInstanceFDs(ctx context.Context, deps instanceAuthorityDeps, fds []int) error { + cleanupCtx := context.WithoutCancel(ctx) + var result error + for index := len(fds) - 1; index >= 0; index-- { + if err := deps.close(cleanupCtx, fds[index]); err != nil { + result = errors.Join(result, fmt.Errorf("fd %d: %w", fds[index], err)) + } + } + return result +} + +func joinInstanceAtomicError(primary error, message string, cause error) *errs.DomainError { + if primary == nil { + return instanceAtomicError(message+": "+cause.Error(), cause) + } + if domainErr := errs.AsDomainError(primary); domainErr != nil { + domainErr.Message += "; " + message + ": " + cause.Error() + domainErr.Err = errors.Join(domainErr.Err, cause) + return domainErr + } + return errs.WrapMsg( + errs.CodeVMAtomicFailed, + primary.Error()+"; "+message+": "+cause.Error(), + errors.Join(primary, cause), + errs.WithClass(errs.ClassInternal), + ) +} + +func randomInstanceTempName(ctx context.Context) (string, error) { + if err := ctx.Err(); err != nil { + return "", err + } + var value [16]byte + if _, err := rand.Read(value[:]); err != nil { + return "", err + } + return ".mvm-instance-" + hex.EncodeToString(value[:]) + ".tmp", nil +} + +func readInstanceDirectoryNames(ctx context.Context, fd int) ([]string, error) { + if err := ctx.Err(); err != nil { + return nil, err + } + duplicateFD, err := unix.Openat(fd, ".", instanceDirectoryFlags, 0) + if err != nil { + return nil, err + } + file := os.NewFile(uintptr(duplicateFD), "instance-authority-directory") + if file == nil { + closeErr := unix.Close(duplicateFD) + return nil, errors.Join(fmt.Errorf("create directory reader"), closeErr) + } + entries, readErr := file.ReadDir(-1) + closeErr := file.Close() + if readErr != nil || closeErr != nil { + return nil, errors.Join(readErr, closeErr) + } + names := make([]string, 0, len(entries)) + for _, entry := range entries { + names = append(names, entry.Name()) + } + return names, nil +} + +func waitForInstanceLock(ctx context.Context) error { + timer := time.NewTimer(instanceLockPoll) + defer timer.Stop() + select { + case <-ctx.Done(): + return ctx.Err() + case <-timer.C: + return nil + } +} + +func realInstanceAuthorityDeps() instanceAuthorityDeps { + return instanceAuthorityDeps{ + open: func(_ context.Context, path string, flags int, mode uint32) (int, error) { + return unix.Open(path, flags, mode) + }, + openAt: func(_ context.Context, parentFD int, name string, flags int, mode uint32) (int, error) { + return unix.Openat(parentFD, name, flags, mode) + }, + mkdirAt: func(_ context.Context, parentFD int, name string, mode uint32) error { + return unix.Mkdirat(parentFD, name, mode) + }, + fstat: func(_ context.Context, fd int, stat *unix.Stat_t) error { return unix.Fstat(fd, stat) }, + fchown: func(_ context.Context, fd, uid, gid int) error { return unix.Fchown(fd, uid, gid) }, + fchmod: func(_ context.Context, fd int, mode uint32) error { return unix.Fchmod(fd, mode) }, + read: func(_ context.Context, fd int, value []byte) (int, error) { return unix.Read(fd, value) }, + write: func(_ context.Context, fd int, value []byte) (int, error) { return unix.Write(fd, value) }, + fsync: func(_ context.Context, fd int) error { return unix.Fsync(fd) }, + close: func(_ context.Context, fd int) error { return unix.Close(fd) }, + renameAt: func(_ context.Context, oldFD int, oldName string, newFD int, newName string) error { + return unix.Renameat(oldFD, oldName, newFD, newName) + }, + unlinkAt: func(_ context.Context, parentFD int, name string, flags int) error { + return unix.Unlinkat(parentFD, name, flags) + }, + flock: func(_ context.Context, fd, how int) error { return unix.Flock(fd, how) }, + randomName: randomInstanceTempName, + readDirNames: readInstanceDirectoryNames, + waitLock: waitForInstanceLock, + } +} + +func validateCanonicalUIDName(name string) (uint32, bool) { + value, err := strconv.ParseUint(name, 10, 32) + if err != nil || value == 0 || strconv.FormatUint(value, 10) != name { + return 0, false + } + return uint32(value), true +} + +func readAllInstanceRecord(ctx context.Context, deps instanceAuthorityDeps, fd int) ([]byte, error) { + result := make([]byte, 0, instanceIOChunk) + buffer := make([]byte, instanceIOChunk) + for { + if err := ctx.Err(); err != nil { + return nil, err + } + count, err := deps.read(ctx, fd, buffer) + if count < 0 || count > len(buffer) { + return nil, fmt.Errorf("invalid read count %d", count) + } + if len(result)+count > maxInstanceRecordBytes { + return nil, fmt.Errorf("instance authority record exceeds size limit") + } + result = append(result, buffer[:count]...) + if err != nil { + if errors.Is(err, io.EOF) { + return result, nil + } + return nil, err + } + if count == 0 { + return result, nil + } + } +} diff --git a/internal/service/jailer/instance_fs_test.go b/internal/service/jailer/instance_fs_test.go new file mode 100644 index 00000000..5b387579 --- /dev/null +++ b/internal/service/jailer/instance_fs_test.go @@ -0,0 +1,845 @@ +package jailer + +import ( + "context" + "errors" + "os" + "path/filepath" + "strings" + "sync/atomic" + "testing" + "time" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + "golang.org/x/sys/unix" + + "mvmctl/pkg/errs" +) + +func TestOpenInstanceAuthorityRootsCreatesFixedManagedTree(t *testing.T) { + t.Parallel() + + root := prepareInstanceAuthorityTestRoot(t) + roots, err := openInstanceAuthorityRoots( + context.Background(), + realInstanceAuthorityDeps(), + testInstanceAuthorityPolicy(root), + ) + require.NoError(t, err) + t.Cleanup(func() { require.NoError(t, roots.Release(context.Background())) }) + + assertExactMode(t, filepath.Join(root, "var/lib/mvmctl/instances"), 0700) + assertExactMode(t, filepath.Join(root, "run/mvmctl"), 0700) + assertExactMode(t, filepath.Join(root, "run/mvmctl/releases"), 0700) + + uidDirs, err := roots.openUIDDirectories(context.Background(), testAuthorityUID) + require.NoError(t, err) + t.Cleanup(func() { require.NoError(t, uidDirs.Release(context.Background())) }) + assertExactMode(t, filepath.Join(root, "var/lib/mvmctl/instances/1000"), 0700) + assertExactMode(t, filepath.Join(root, "run/mvmctl/1000"), 0700) +} + +func TestOpenInstanceAuthorityRootsRejectsUnsafeTree(t *testing.T) { + t.Parallel() + + tests := map[string]func(*testing.T, string){ + "missing_var": func(t *testing.T, root string) { + require.NoError(t, os.RemoveAll(filepath.Join(root, "var"))) + }, + "writable_var": func(t *testing.T, root string) { + require.NoError(t, os.Chmod(filepath.Join(root, "var"), 0777)) + }, + "symlink_mvmctl": func(t *testing.T, root string) { + target := filepath.Join(root, "attacker") + require.NoError(t, os.Mkdir(target, 0700)) + require.NoError(t, os.Symlink(target, filepath.Join(root, "var/lib/mvmctl"))) + }, + } + + for name, mutate := range tests { + t.Run(name, func(t *testing.T) { + t.Parallel() + + root := prepareInstanceAuthorityTestRoot(t) + mutate(t, root) + roots, err := openInstanceAuthorityRoots( + context.Background(), + realInstanceAuthorityDeps(), + testInstanceAuthorityPolicy(root), + ) + if roots != nil { + require.NoError(t, roots.Release(context.Background())) + } + require.Error(t, err) + assert.Equal(t, errs.CodeVMAtomicFailed, errs.AsDomainError(err).Code) + }) + } +} + +func TestOpenInstanceAuthorityRootsRejectsSymlinkAtEveryFixedLevel(t *testing.T) { + t.Parallel() + + tests := map[string]func(*testing.T, string){ + "var": func(t *testing.T, root string) { + require.NoError(t, os.RemoveAll(filepath.Join(root, "var"))) + require.NoError(t, os.Symlink("attacker", filepath.Join(root, "var"))) + }, + "lib": func(t *testing.T, root string) { + require.NoError(t, os.Remove(filepath.Join(root, "var/lib"))) + require.NoError(t, os.Symlink("attacker", filepath.Join(root, "var/lib"))) + }, + "state_base": func(t *testing.T, root string) { + require.NoError(t, os.Symlink("attacker", filepath.Join(root, "var/lib/mvmctl"))) + }, + "instances": func(t *testing.T, root string) { + require.NoError(t, os.Mkdir(filepath.Join(root, "var/lib/mvmctl"), 0700)) + require.NoError(t, os.Symlink("attacker", filepath.Join(root, "var/lib/mvmctl/instances"))) + }, + "run": func(t *testing.T, root string) { + require.NoError(t, os.Remove(filepath.Join(root, "run"))) + require.NoError(t, os.Symlink("attacker", filepath.Join(root, "run"))) + }, + "runtime_base": func(t *testing.T, root string) { + require.NoError(t, os.Symlink("attacker", filepath.Join(root, "run/mvmctl"))) + }, + "release_locks": func(t *testing.T, root string) { + require.NoError(t, os.Mkdir(filepath.Join(root, "run/mvmctl"), 0700)) + require.NoError(t, os.Symlink("attacker", filepath.Join(root, "run/mvmctl/releases"))) + }, + } + + for name, mutate := range tests { + t.Run(name, func(t *testing.T) { + t.Parallel() + + root := prepareInstanceAuthorityTestRoot(t) + mutate(t, root) + roots, err := openInstanceAuthorityRoots( + context.Background(), + realInstanceAuthorityDeps(), + testInstanceAuthorityPolicy(root), + ) + if roots != nil { + require.NoError(t, roots.Release(context.Background())) + } + require.Error(t, err) + assert.Equal(t, errs.CodeVMAtomicFailed, errs.AsDomainError(err).Code) + }) + } +} + +func TestOpenInstanceAuthorityRootsRejectsWritableModeAtEveryFixedLevel(t *testing.T) { + t.Parallel() + + relativePaths := []string{ + ".", + "var", + "var/lib", + "var/lib/mvmctl", + "var/lib/mvmctl/instances", + "run", + "run/mvmctl", + "run/mvmctl/releases", + } + for _, relativePath := range relativePaths { + relativePath := relativePath + t.Run(relativePath, func(t *testing.T) { + t.Parallel() + + root := prepareInstanceAuthorityTestRoot(t) + require.NoError(t, os.MkdirAll(filepath.Join(root, "var/lib/mvmctl/instances"), 0700)) + require.NoError(t, os.MkdirAll(filepath.Join(root, "run/mvmctl/releases"), 0700)) + require.NoError(t, os.Chmod(filepath.Join(root, relativePath), 0777)) + + roots, err := openInstanceAuthorityRoots( + context.Background(), + realInstanceAuthorityDeps(), + testInstanceAuthorityPolicy(root), + ) + if roots != nil { + require.NoError(t, roots.Release(context.Background())) + } + require.Error(t, err) + assert.Equal(t, errs.CodeVMAtomicFailed, errs.AsDomainError(err).Code) + }) + } +} + +func TestOpenInstanceAuthorityRootsTreatsMkdirEEXISTAsConcurrentCreation(t *testing.T) { + t.Parallel() + + root := prepareInstanceAuthorityTestRoot(t) + deps := realInstanceAuthorityDeps() + realMkdirAt := deps.mkdirAt + var chownCalls atomic.Int32 + var chmodCalls atomic.Int32 + deps.mkdirAt = func(ctx context.Context, parentFD int, name string, mode uint32) error { + if err := realMkdirAt(ctx, parentFD, name, mode); err != nil { + return err + } + return unix.EEXIST + } + deps.fchown = func(context.Context, int, int, int) error { + chownCalls.Add(1) + return errors.New("must not chown concurrently created directory") + } + deps.fchmod = func(context.Context, int, uint32) error { + chmodCalls.Add(1) + return errors.New("must not chmod concurrently created directory") + } + + roots, err := openInstanceAuthorityRoots(context.Background(), deps, testInstanceAuthorityPolicy(root)) + require.NoError(t, err) + require.NoError(t, roots.Release(context.Background())) + assert.Zero(t, chownCalls.Load()) + assert.Zero(t, chmodCalls.Load()) +} + +func TestOpenInstanceAuthorityRootsRejectsManagedModeAndType(t *testing.T) { + t.Parallel() + + tests := map[string]func(*testing.T, string){ + "managed_mode": func(t *testing.T, root string) { + require.NoError(t, os.Mkdir(filepath.Join(root, "var/lib/mvmctl"), 0700)) + require.NoError(t, os.Mkdir(filepath.Join(root, "var/lib/mvmctl/instances"), 0755)) + }, + "managed_type": func(t *testing.T, root string) { + require.NoError(t, os.Mkdir(filepath.Join(root, "var/lib/mvmctl"), 0700)) + require.NoError(t, os.WriteFile(filepath.Join(root, "var/lib/mvmctl/instances"), nil, 0600)) + }, + } + + for name, mutate := range tests { + t.Run(name, func(t *testing.T) { + t.Parallel() + + root := prepareInstanceAuthorityTestRoot(t) + mutate(t, root) + roots, err := openInstanceAuthorityRoots( + context.Background(), + realInstanceAuthorityDeps(), + testInstanceAuthorityPolicy(root), + ) + if roots != nil { + require.NoError(t, roots.Release(context.Background())) + } + require.Error(t, err) + }) + } +} + +func TestOpenInstanceAuthorityRootsRejectsSimulatedAncestorOwnerMismatch(t *testing.T) { + t.Parallel() + + root := prepareInstanceAuthorityTestRoot(t) + deps := realInstanceAuthorityDeps() + realFstat := deps.fstat + var directoriesInspected atomic.Int32 + deps.fstat = func(ctx context.Context, fd int, stat *unix.Stat_t) error { + if err := realFstat(ctx, fd, stat); err != nil { + return err + } + if stat.Mode&unix.S_IFMT == unix.S_IFDIR && directoriesInspected.Add(1) == 2 { + stat.Uid++ + } + return nil + } + + roots, err := openInstanceAuthorityRoots(context.Background(), deps, testInstanceAuthorityPolicy(root)) + assert.Nil(t, roots) + require.Error(t, err) + assert.Equal(t, errs.CodeVMAtomicFailed, errs.AsDomainError(err).Code) +} + +func TestInstanceAuthorityVMLockIsCancellableAndPersistent(t *testing.T) { + t.Parallel() + + root := prepareInstanceAuthorityTestRoot(t) + roots := openTestInstanceAuthorityRoots(t, root) + uidDirs, err := roots.openUIDDirectories(context.Background(), testAuthorityUID) + require.NoError(t, err) + t.Cleanup(func() { require.NoError(t, uidDirs.Release(context.Background())) }) + + first, err := uidDirs.acquireVMLock(context.Background(), testVMID) + require.NoError(t, err) + + waitCtx, cancel := context.WithTimeout(context.Background(), 30*time.Millisecond) + defer cancel() + second, err := uidDirs.acquireVMLock(waitCtx, testVMID) + assert.Nil(t, second) + require.Error(t, err) + assert.ErrorIs(t, err, context.DeadlineExceeded) + assert.Equal(t, errs.ClassRetryable, errs.AsDomainError(err).Class) + + require.NoError(t, first.Release(context.Background())) + assertExactMode(t, filepath.Join(root, "run/mvmctl/1000/"+testVMID+".lock"), 0600) + + reacquired, err := uidDirs.acquireVMLock(context.Background(), testVMID) + require.NoError(t, err) + require.NoError(t, reacquired.Release(context.Background())) +} + +func TestInstanceAuthorityLockRejectsSymlink(t *testing.T) { + t.Parallel() + + root := prepareInstanceAuthorityTestRoot(t) + roots := openTestInstanceAuthorityRoots(t, root) + uidDirs, err := roots.openUIDDirectories(context.Background(), testAuthorityUID) + require.NoError(t, err) + t.Cleanup(func() { require.NoError(t, uidDirs.Release(context.Background())) }) + + lockPath := filepath.Join(root, "run/mvmctl/1000/"+testVMID+".lock") + require.NoError(t, os.Symlink("attacker", lockPath)) + + lock, err := uidDirs.acquireVMLock(context.Background(), testVMID) + assert.Nil(t, lock) + require.Error(t, err) + assert.Equal(t, errs.CodeProcessError, errs.AsDomainError(err).Code) +} + +func TestInstanceAuthorityLockRejectsUnsafeModeAndType(t *testing.T) { + t.Parallel() + + tests := map[string]func(*testing.T, string){ + "mode": func(t *testing.T, path string) { + require.NoError(t, os.WriteFile(path, nil, 0644)) + }, + "type": func(t *testing.T, path string) { + require.NoError(t, os.Mkdir(path, 0700)) + }, + } + for name, create := range tests { + t.Run(name, func(t *testing.T) { + t.Parallel() + + root := prepareInstanceAuthorityTestRoot(t) + roots := openTestInstanceAuthorityRoots(t, root) + uidDirs, err := roots.openUIDDirectories(context.Background(), testAuthorityUID) + require.NoError(t, err) + t.Cleanup(func() { require.NoError(t, uidDirs.Release(context.Background())) }) + path := filepath.Join(root, "run/mvmctl/1000/"+testVMID+".lock") + create(t, path) + + lock, err := uidDirs.acquireVMLock(context.Background(), testVMID) + assert.Nil(t, lock) + require.Error(t, err) + }) + } +} + +func TestInstanceRecordAndLockRejectSimulatedOwnerMismatch(t *testing.T) { + t.Parallel() + + root := prepareInstanceAuthorityTestRoot(t) + roots := openTestInstanceAuthorityRoots(t, root) + uidDirs, err := roots.openUIDDirectories(context.Background(), testAuthorityUID) + require.NoError(t, err) + t.Cleanup(func() { require.NoError(t, uidDirs.Release(context.Background())) }) + require.NoError(t, uidDirs.writeRecord(context.Background(), testRegisteredInstanceRecord())) + lock, err := uidDirs.acquireVMLock(context.Background(), testVMID) + require.NoError(t, err) + require.NoError(t, lock.Release(context.Background())) + + realFstat := uidDirs.deps.fstat + uidDirs.deps.fstat = func(ctx context.Context, fd int, stat *unix.Stat_t) error { + if err := realFstat(ctx, fd, stat); err != nil { + return err + } + if stat.Mode&unix.S_IFMT == unix.S_IFREG { + stat.Uid++ + } + return nil + } + _, _, err = uidDirs.readRecord(context.Background(), testVMID) + require.Error(t, err) + assert.Equal(t, errs.CodeVMAtomicFailed, errs.AsDomainError(err).Code) + lock, err = uidDirs.acquireVMLock(context.Background(), testVMID) + assert.Nil(t, lock) + require.Error(t, err) +} + +func TestInstanceRecordAtomicWriteAndRead(t *testing.T) { + t.Parallel() + + root := prepareInstanceAuthorityTestRoot(t) + roots := openTestInstanceAuthorityRoots(t, root) + uidDirs, err := roots.openUIDDirectories(context.Background(), testAuthorityUID) + require.NoError(t, err) + t.Cleanup(func() { require.NoError(t, uidDirs.Release(context.Background())) }) + + record := testRegisteredInstanceRecord() + require.NoError(t, uidDirs.writeRecord(context.Background(), record)) + + got, found, err := uidDirs.readRecord(context.Background(), testVMID) + require.NoError(t, err) + assert.True(t, found) + assert.Equal(t, record, got) + assertExactMode(t, filepath.Join(root, "var/lib/mvmctl/instances/1000/"+testVMID+".json"), 0600) +} + +func TestInstanceRecordWriteHandlesPartialWrites(t *testing.T) { + t.Parallel() + + root := prepareInstanceAuthorityTestRoot(t) + roots := openTestInstanceAuthorityRoots(t, root) + uidDirs, err := roots.openUIDDirectories(context.Background(), testAuthorityUID) + require.NoError(t, err) + t.Cleanup(func() { require.NoError(t, uidDirs.Release(context.Background())) }) + + realWrite := uidDirs.deps.write + uidDirs.deps.write = func(ctx context.Context, fd int, value []byte) (int, error) { + if len(value) > 3 { + value = value[:3] + } + return realWrite(ctx, fd, value) + } + require.NoError(t, uidDirs.writeRecord(context.Background(), testRegisteredInstanceRecord())) + + _, found, err := uidDirs.readRecord(context.Background(), testVMID) + require.NoError(t, err) + assert.True(t, found) +} + +func TestInstanceRecordWriteVerifiesActualTemporarySize(t *testing.T) { + t.Parallel() + + root := prepareInstanceAuthorityTestRoot(t) + roots := openTestInstanceAuthorityRoots(t, root) + uidDirs, err := roots.openUIDDirectories(context.Background(), testAuthorityUID) + require.NoError(t, err) + t.Cleanup(func() { require.NoError(t, uidDirs.Release(context.Background())) }) + uidDirs.deps.write = func(_ context.Context, _ int, value []byte) (int, error) { return len(value), nil } + + err = uidDirs.writeRecord(context.Background(), testRegisteredInstanceRecord()) + require.Error(t, err) + assert.Equal(t, errs.CodeVMAtomicFailed, errs.AsDomainError(err).Code) + _, statErr := os.Lstat(filepath.Join(root, "var/lib/mvmctl/instances/1000/"+testVMID+".json")) + assert.ErrorIs(t, statErr, os.ErrNotExist) + assertNoInstanceTemps(t, filepath.Join(root, "var/lib/mvmctl/instances/1000")) +} + +func TestInstanceRecordPreRenameFailurePreservesOldRecordAndCleansTemp(t *testing.T) { + t.Parallel() + + root := prepareInstanceAuthorityTestRoot(t) + roots := openTestInstanceAuthorityRoots(t, root) + uidDirs, err := roots.openUIDDirectories(context.Background(), testAuthorityUID) + require.NoError(t, err) + t.Cleanup(func() { require.NoError(t, uidDirs.Release(context.Background())) }) + + want := testRegisteredInstanceRecord() + require.NoError(t, uidDirs.writeRecord(context.Background(), want)) + uidDirs.deps.renameAt = func(context.Context, int, string, int, string) error { + return errors.New("rename failed") + } + updated := want + updated.lifecycle = instanceLifecycleCleaning + updated.cleanupGeneration = 1 + + err = uidDirs.writeRecord(context.Background(), updated) + require.Error(t, err) + assert.Equal(t, errs.CodeVMAtomicFailed, errs.AsDomainError(err).Code) + + got, found, readErr := uidDirs.readRecord(context.Background(), testVMID) + require.NoError(t, readErr) + assert.True(t, found) + assert.Equal(t, want, got) + assertNoInstanceTemps(t, filepath.Join(root, "var/lib/mvmctl/instances/1000")) +} + +func TestInstanceRecordEveryPreRenameFailurePreservesOldRecord(t *testing.T) { + t.Parallel() + + tests := map[string]func(*instanceUIDDirectories) func(){ + "write": func(directories *instanceUIDDirectories) func() { + directories.deps.write = func(context.Context, int, []byte) (int, error) { + return 0, errors.New("write failed") + } + return func() {} + }, + "fchown": func(directories *instanceUIDDirectories) func() { + directories.deps.fchown = func(context.Context, int, int, int) error { + return errors.New("chown failed") + } + return func() {} + }, + "fchmod": func(directories *instanceUIDDirectories) func() { + directories.deps.fchmod = func(context.Context, int, uint32) error { + return errors.New("chmod failed") + } + return func() {} + }, + "file_fsync": func(directories *instanceUIDDirectories) func() { + directories.deps.fsync = func(context.Context, int) error { + return errors.New("file sync failed") + } + return func() {} + }, + "close": func(directories *instanceUIDDirectories) func() { + realClose := directories.deps.close + var calls atomic.Int32 + directories.deps.close = func(ctx context.Context, fd int) error { + err := realClose(ctx, fd) + if calls.Add(1) == 1 && err == nil { + return errors.New("close failed") + } + return err + } + return func() { directories.deps.close = realClose } + }, + "rename": func(directories *instanceUIDDirectories) func() { + directories.deps.renameAt = func(context.Context, int, string, int, string) error { + return errors.New("rename failed") + } + return func() {} + }, + } + + for name, inject := range tests { + t.Run(name, func(t *testing.T) { + t.Parallel() + + root := prepareInstanceAuthorityTestRoot(t) + roots := openTestInstanceAuthorityRoots(t, root) + uidDirs, err := roots.openUIDDirectories(context.Background(), testAuthorityUID) + require.NoError(t, err) + t.Cleanup(func() { require.NoError(t, uidDirs.Release(context.Background())) }) + old := testRegisteredInstanceRecord() + require.NoError(t, uidDirs.writeRecord(context.Background(), old)) + recordPath := filepath.Join(root, "var/lib/mvmctl/instances/1000/"+testVMID+".json") + oldRaw, err := os.ReadFile(recordPath) + require.NoError(t, err) + restore := inject(uidDirs) + updated := old + updated.lifecycle = instanceLifecycleCleaning + updated.cleanupGeneration = 1 + + err = uidDirs.writeRecord(context.Background(), updated) + restore() + require.Error(t, err) + assert.Equal(t, errs.CodeVMAtomicFailed, errs.AsDomainError(err).Code) + gotRaw, readErr := os.ReadFile(recordPath) + require.NoError(t, readErr) + assert.Equal(t, oldRaw, gotRaw) + assertNoInstanceTemps(t, filepath.Dir(recordPath)) + }) + } +} + +func TestInstanceRecordCleanupFailureIsReturned(t *testing.T) { + t.Parallel() + + root := prepareInstanceAuthorityTestRoot(t) + roots := openTestInstanceAuthorityRoots(t, root) + uidDirs, err := roots.openUIDDirectories(context.Background(), testAuthorityUID) + require.NoError(t, err) + t.Cleanup(func() { require.NoError(t, uidDirs.Release(context.Background())) }) + uidDirs.deps.write = func(context.Context, int, []byte) (int, error) { + return 0, errors.New("write failed") + } + realUnlink := uidDirs.deps.unlinkAt + uidDirs.deps.unlinkAt = func(ctx context.Context, parentFD int, name string, flags int) error { + err := realUnlink(ctx, parentFD, name, flags) + if err != nil { + return err + } + return errors.New("unlink completion failed") + } + + err = uidDirs.writeRecord(context.Background(), testRegisteredInstanceRecord()) + require.Error(t, err) + assert.ErrorContains(t, err, "unlink completion failed") + assertNoInstanceTemps(t, filepath.Join(root, "var/lib/mvmctl/instances/1000")) +} + +func TestInstanceRecordWriteRejectsExistingUnsafeTarget(t *testing.T) { + t.Parallel() + + tests := map[string]func(*testing.T, string){ + "symlink": func(t *testing.T, path string) { + require.NoError(t, os.Symlink("attacker", path)) + }, + "directory": func(t *testing.T, path string) { + require.NoError(t, os.Mkdir(path, 0700)) + }, + "wrong_mode": func(t *testing.T, path string) { + require.NoError(t, os.WriteFile(path, []byte("old"), 0644)) + }, + } + + for name, create := range tests { + t.Run(name, func(t *testing.T) { + t.Parallel() + + root := prepareInstanceAuthorityTestRoot(t) + roots := openTestInstanceAuthorityRoots(t, root) + uidDirs, err := roots.openUIDDirectories(context.Background(), testAuthorityUID) + require.NoError(t, err) + t.Cleanup(func() { require.NoError(t, uidDirs.Release(context.Background())) }) + recordPath := filepath.Join(root, "var/lib/mvmctl/instances/1000/"+testVMID+".json") + create(t, recordPath) + + err = uidDirs.writeRecord(context.Background(), testRegisteredInstanceRecord()) + require.Error(t, err) + assert.Equal(t, errs.CodeVMAtomicFailed, errs.AsDomainError(err).Code) + assertNoInstanceTemps(t, filepath.Dir(recordPath)) + }) + } +} + +func TestInstanceRecordReadHandlesPartialReadsAndRejectsOversize(t *testing.T) { + t.Parallel() + + root := prepareInstanceAuthorityTestRoot(t) + roots := openTestInstanceAuthorityRoots(t, root) + uidDirs, err := roots.openUIDDirectories(context.Background(), testAuthorityUID) + require.NoError(t, err) + t.Cleanup(func() { require.NoError(t, uidDirs.Release(context.Background())) }) + require.NoError(t, uidDirs.writeRecord(context.Background(), testRegisteredInstanceRecord())) + + realRead := uidDirs.deps.read + uidDirs.deps.read = func(ctx context.Context, fd int, value []byte) (int, error) { + if len(value) > 3 { + value = value[:3] + } + return realRead(ctx, fd, value) + } + _, found, err := uidDirs.readRecord(context.Background(), testVMID) + require.NoError(t, err) + assert.True(t, found) + uidDirs.deps.read = realRead + + recordPath := filepath.Join(root, "var/lib/mvmctl/instances/1000/"+testVMID+".json") + require.NoError(t, os.WriteFile(recordPath, []byte(strings.Repeat("x", maxInstanceRecordBytes+1)), 0600)) + _, _, err = uidDirs.readRecord(context.Background(), testVMID) + require.Error(t, err) + assert.Equal(t, errs.CodeVMAtomicFailed, errs.AsDomainError(err).Code) +} + +func TestInstanceRecordPostRenameFailureReportsPartialSuccess(t *testing.T) { + t.Parallel() + + root := prepareInstanceAuthorityTestRoot(t) + roots := openTestInstanceAuthorityRoots(t, root) + uidDirs, err := roots.openUIDDirectories(context.Background(), testAuthorityUID) + require.NoError(t, err) + t.Cleanup(func() { require.NoError(t, uidDirs.Release(context.Background())) }) + + realFsync := uidDirs.deps.fsync + var fsyncCalls atomic.Int32 + uidDirs.deps.fsync = func(ctx context.Context, fd int) error { + if fsyncCalls.Add(1) == 2 { + return errors.New("directory sync failed") + } + return realFsync(ctx, fd) + } + + err = uidDirs.writeRecord(context.Background(), testRegisteredInstanceRecord()) + require.Error(t, err) + domainErr := errs.AsDomainError(err) + require.NotNil(t, domainErr) + assert.Equal(t, true, domainErr.Details["record_replaced"]) + assert.Equal(t, true, domainErr.Details["durability_uncertain"]) + + _, found, readErr := uidDirs.readRecord(context.Background(), testVMID) + require.NoError(t, readErr) + assert.True(t, found) +} + +func TestInstanceRecordCancellationBeforeRenamePreservesOldRecord(t *testing.T) { + t.Parallel() + + root := prepareInstanceAuthorityTestRoot(t) + roots := openTestInstanceAuthorityRoots(t, root) + uidDirs, err := roots.openUIDDirectories(context.Background(), testAuthorityUID) + require.NoError(t, err) + t.Cleanup(func() { require.NoError(t, uidDirs.Release(context.Background())) }) + + want := testRegisteredInstanceRecord() + require.NoError(t, uidDirs.writeRecord(context.Background(), want)) + ctx, cancel := context.WithCancel(context.Background()) + realFsync := uidDirs.deps.fsync + uidDirs.deps.fsync = func(ctx context.Context, fd int) error { + err := realFsync(ctx, fd) + cancel() + return err + } + updated := want + updated.lifecycle = instanceLifecycleCleaning + updated.cleanupGeneration = 1 + + err = uidDirs.writeRecord(ctx, updated) + require.Error(t, err) + assert.ErrorIs(t, err, context.Canceled) + got, found, readErr := uidDirs.readRecord(context.Background(), testVMID) + require.NoError(t, readErr) + assert.True(t, found) + assert.Equal(t, want, got) + assertNoInstanceTemps(t, filepath.Join(root, "var/lib/mvmctl/instances/1000")) +} + +func TestInstanceRecordReadRejectsSymlink(t *testing.T) { + t.Parallel() + + root := prepareInstanceAuthorityTestRoot(t) + roots := openTestInstanceAuthorityRoots(t, root) + uidDirs, err := roots.openUIDDirectories(context.Background(), testAuthorityUID) + require.NoError(t, err) + t.Cleanup(func() { require.NoError(t, uidDirs.Release(context.Background())) }) + + recordPath := filepath.Join(root, "var/lib/mvmctl/instances/1000/"+testVMID+".json") + require.NoError(t, os.Symlink("attacker", recordPath)) + + _, _, err = uidDirs.readRecord(context.Background(), testVMID) + require.Error(t, err) + assert.Equal(t, errs.CodeVMAtomicFailed, errs.AsDomainError(err).Code) +} + +func TestInstanceDirectoryScansUseIndependentOffsets(t *testing.T) { + t.Parallel() + + root := prepareInstanceAuthorityTestRoot(t) + roots := openTestInstanceAuthorityRoots(t, root) + uidDirs, err := roots.openUIDDirectories(context.Background(), testAuthorityUID) + require.NoError(t, err) + require.NoError(t, uidDirs.Release(context.Background())) + + first, err := roots.deps.readDirNames(context.Background(), roots.stateFD) + require.NoError(t, err) + second, err := roots.deps.readDirNames(context.Background(), roots.stateFD) + require.NoError(t, err) + assert.Equal(t, []string{"1000"}, first) + assert.Equal(t, first, second) +} + +func TestInstanceRecordReadCloseFailureReturnsDomainError(t *testing.T) { + t.Parallel() + + root := prepareInstanceAuthorityTestRoot(t) + roots := openTestInstanceAuthorityRoots(t, root) + uidDirs, err := roots.openUIDDirectories(context.Background(), testAuthorityUID) + require.NoError(t, err) + t.Cleanup(func() { require.NoError(t, uidDirs.Release(context.Background())) }) + require.NoError(t, uidDirs.writeRecord(context.Background(), testRegisteredInstanceRecord())) + + realClose := uidDirs.deps.close + uidDirs.deps.close = func(ctx context.Context, fd int) error { + closeErr := realClose(ctx, fd) + if closeErr != nil { + return closeErr + } + return errors.New("record close failed") + } + _, found, err := uidDirs.readRecord(context.Background(), testVMID) + uidDirs.deps.close = realClose + require.Error(t, err) + assert.True(t, found) + assert.Equal(t, errs.CodeVMAtomicFailed, errs.AsDomainError(err).Code) +} + +func TestJoinInstanceAtomicErrorPreservesPartialSuccessDetails(t *testing.T) { + t.Parallel() + + primary := annotateInstanceRecordReplacement( + instanceAtomicError("sync record", errors.New("sync failed")), + true, + ) + joined := joinInstanceAtomicError(primary, "close descriptor", errors.New("close failed")) + + assert.Equal(t, true, joined.Details["record_replaced"]) + assert.Equal(t, true, joined.Details["durability_uncertain"]) +} + +func prepareInstanceAuthorityTestRoot(t *testing.T) string { + t.Helper() + + root := t.TempDir() + require.NoError(t, os.MkdirAll(filepath.Join(root, "var/lib"), 0755)) + require.NoError(t, os.Mkdir(filepath.Join(root, "run"), 0755)) + return root +} + +func testInstanceAuthorityPolicy(root string) instanceAuthorityPolicy { + return instanceAuthorityPolicy{ + rootPath: root, + expectedUID: uint32(os.Geteuid()), + expectedGID: uint32(os.Getegid()), + } +} + +func openTestInstanceAuthorityRoots(t *testing.T, root string) *instanceAuthorityRoots { + t.Helper() + + roots, err := openInstanceAuthorityRoots( + context.Background(), + realInstanceAuthorityDeps(), + testInstanceAuthorityPolicy(root), + ) + require.NoError(t, err) + t.Cleanup(func() { require.NoError(t, roots.Release(context.Background())) }) + return roots +} + +func assertExactMode(t *testing.T, path string, want os.FileMode) { + t.Helper() + + info, err := os.Lstat(path) + require.NoError(t, err) + assert.Equal(t, want, info.Mode().Perm()) +} + +func assertNoInstanceTemps(t *testing.T, directory string) { + t.Helper() + + entries, err := os.ReadDir(directory) + require.NoError(t, err) + for _, entry := range entries { + assert.False(t, strings.HasPrefix(entry.Name(), ".mvm-instance-"), entry.Name()) + } +} + +func TestInstanceAuthorityReleaseReportsUnlockFailure(t *testing.T) { + t.Parallel() + + root := prepareInstanceAuthorityTestRoot(t) + roots := openTestInstanceAuthorityRoots(t, root) + uidDirs, err := roots.openUIDDirectories(context.Background(), testAuthorityUID) + require.NoError(t, err) + t.Cleanup(func() { require.NoError(t, uidDirs.Release(context.Background())) }) + lock, err := uidDirs.acquireVMLock(context.Background(), testVMID) + require.NoError(t, err) + + realFlock := lock.deps.flock + lock.deps.flock = func(ctx context.Context, fd int, how int) error { + if how == unix.LOCK_UN { + return errors.New("unlock failed") + } + return realFlock(ctx, fd, how) + } + err = lock.Release(context.Background()) + require.Error(t, err) + assert.Equal(t, errs.CodeProcessError, errs.AsDomainError(err).Code) +} + +func TestInstanceAuthorityReleasePropagatesCallerContextToCleanup(t *testing.T) { + t.Parallel() + + type cleanupContextKey struct{} + ctx := context.WithValue(context.Background(), cleanupContextKey{}, "release-context") + root := prepareInstanceAuthorityTestRoot(t) + deps := realInstanceAuthorityDeps() + realClose := deps.close + contextObserved := false + deps.close = func(closeCtx context.Context, fd int) error { + if closeCtx.Value(cleanupContextKey{}) == "release-context" { + contextObserved = true + } + return realClose(closeCtx, fd) + } + roots, err := openInstanceAuthorityRoots(context.Background(), deps, testInstanceAuthorityPolicy(root)) + require.NoError(t, err) + + require.NoError(t, roots.Release(ctx)) + assert.True(t, contextObserved) +} diff --git a/internal/service/jailer/instance_lock.go b/internal/service/jailer/instance_lock.go new file mode 100644 index 00000000..8b87b173 --- /dev/null +++ b/internal/service/jailer/instance_lock.go @@ -0,0 +1,175 @@ +package jailer + +import ( + "context" + "crypto/sha256" + "encoding/hex" + "errors" + "fmt" + + "golang.org/x/sys/unix" + + "mvmctl/pkg/errs" +) + +type authorityLock struct { + deps instanceAuthorityDeps + fd int +} + +func (roots *instanceAuthorityRoots) acquireIndexLock(ctx context.Context) (*authorityLock, error) { + return acquireAuthorityLock(ctx, roots.deps, roots.policy, roots.runtimeFD, "index.lock", "global index") +} + +func (roots *instanceAuthorityRoots) acquireReleaseLock( + ctx context.Context, + release releaseIdentity, +) (*authorityLock, error) { + if err := validateReleaseIdentityValue(release); err != nil { + return nil, instanceValidationError(err.Error()) + } + return acquireAuthorityLock( + ctx, + roots.deps, + roots.policy, + roots.releasesFD, + releaseLockName(release), + "release", + ) +} + +func (directories *instanceUIDDirectories) acquireVMLock( + ctx context.Context, + vmID string, +) (*authorityLock, error) { + if !vmIDPattern.MatchString(vmID) { + return nil, instanceValidationError("vm ID is invalid") + } + return acquireAuthorityLock( + ctx, + directories.deps, + directories.policy, + directories.runtimeFD, + vmID+".lock", + "VM", + ) +} + +func acquireAuthorityLock( + ctx context.Context, + deps instanceAuthorityDeps, + policy instanceAuthorityPolicy, + parentFD int, + name string, + description string, +) (*authorityLock, error) { + if err := ctx.Err(); err != nil { + return nil, instanceLockContextError(description, err) + } + fd, err := deps.openAt(ctx, parentFD, name, instanceLockCreate, instanceStateFileMode) + created := err == nil + if errors.Is(err, unix.EEXIST) { + fd, err = deps.openAt(ctx, parentFD, name, instanceLockFlags, 0) + } + if err != nil { + return nil, instanceLockError("open "+description+" lock", err) + } + abort := func(cause error) (*authorityLock, error) { + if closeErr := deps.close(context.WithoutCancel(ctx), fd); closeErr != nil { + cause = errors.Join(cause, closeErr) + } + return nil, instanceLockError("prepare "+description+" lock", cause) + } + if created { + if err := deps.fchown(ctx, fd, int(policy.expectedUID), int(policy.expectedGID)); err != nil { + return abort(err) + } + if err := deps.fchmod(ctx, fd, instanceStateFileMode); err != nil { + return abort(err) + } + } + if err := verifyInstanceRegularFile(ctx, deps, fd, description+" lock", policy); err != nil { + return abort(err) + } + if created { + if err := deps.fsync(ctx, fd); err != nil { + return abort(err) + } + if err := deps.fsync(ctx, parentFD); err != nil { + return abort(err) + } + } + for { + err := deps.flock(ctx, fd, unix.LOCK_EX|unix.LOCK_NB) + if err == nil { + return &authorityLock{deps: deps, fd: fd}, nil + } + if errors.Is(err, unix.EINTR) { + continue + } + if !errors.Is(err, unix.EWOULDBLOCK) && !errors.Is(err, unix.EAGAIN) { + return abort(err) + } + if err := deps.waitLock(ctx); err != nil { + if closeErr := deps.close(context.WithoutCancel(ctx), fd); closeErr != nil { + err = errors.Join(err, closeErr) + } + return nil, instanceLockContextError(description, err) + } + } +} + +func (lock *authorityLock) Release(ctx context.Context) error { + if lock == nil || lock.fd < 0 { + return nil + } + cleanupCtx := context.WithoutCancel(ctx) + unlockErr := lock.deps.flock(cleanupCtx, lock.fd, unix.LOCK_UN) + closeErr := lock.deps.close(cleanupCtx, lock.fd) + lock.fd = -1 + if err := errors.Join(unlockErr, closeErr); err != nil { + return instanceLockError("release authority lock", err) + } + return nil +} + +func verifyInstanceRegularFile( + ctx context.Context, + deps instanceAuthorityDeps, + fd int, + description string, + policy instanceAuthorityPolicy, +) error { + var stat unix.Stat_t + if err := deps.fstat(ctx, fd, &stat); err != nil { + return instanceAtomicError("inspect "+description, err) + } + if stat.Mode&unix.S_IFMT != unix.S_IFREG { + return instanceAtomicError(description+" is not a regular file", nil) + } + if stat.Uid != policy.expectedUID || stat.Gid != policy.expectedGID || + stat.Mode&07777 != instanceStateFileMode { + return instanceAtomicError(description+" has unexpected owner or mode", nil) + } + return nil +} + +func releaseLockName(release releaseIdentity) string { + canonical := release.version + "\x00" + release.architecture + "\x00" + + release.firecrackerSHA256 + "\x00" + release.jailerSHA256 + digest := sha256.Sum256([]byte(canonical)) + return hex.EncodeToString(digest[:]) + ".lock" +} + +func instanceLockError(message string, cause error) *errs.DomainError { + return errs.WrapMsg(errs.CodeProcessError, message+": "+cause.Error(), cause) +} + +func instanceLockContextError(description string, cause error) *errs.DomainError { + return errs.WrapMsg( + errs.CodeProcessError, + fmt.Sprintf("wait for %s lock: %v", description, cause), + cause, + errs.WithClass(errs.ClassRetryable), + ) +} diff --git a/internal/service/jailer/instance_scan.go b/internal/service/jailer/instance_scan.go new file mode 100644 index 00000000..3786e0b5 --- /dev/null +++ b/internal/service/jailer/instance_scan.go @@ -0,0 +1,240 @@ +package jailer + +import ( + "context" + "errors" + "fmt" + "sort" + "strings" +) + +type instanceLocation struct { + uid uint32 + stateFD int +} + +func (roots *instanceAuthorityRoots) findInstance( + ctx context.Context, + vmID string, +) (_ *instanceLocation, returnErr error) { + names, err := roots.deps.readDirNames(ctx, roots.stateFD) + if err != nil { + return nil, instanceAtomicError("enumerate instance authority owners", err) + } + sort.Strings(names) + var found *instanceLocation + defer func() { + if returnErr == nil || found == nil { + return + } + if err := roots.deps.close(context.WithoutCancel(ctx), found.stateFD); err != nil { + returnErr = appendInstanceOperationError( + returnErr, + "close matched instance authority directory", + err, + ) + } + }() + for _, name := range names { + uid, ok := validateCanonicalUIDName(name) + if !ok { + return nil, instanceAtomicError("instance authority owner directory name is invalid", nil) + } + stateFD, err := openInstanceDirectory(ctx, roots.deps, roots.stateFD, name, false, true, roots.policy) + if err != nil { + return nil, err + } + directories := &instanceUIDDirectories{ + deps: roots.deps, + policy: roots.policy, + uid: uid, + stateFD: stateFD, + runtimeFD: -1, + } + _, exists, readErr := directories.readRecord(ctx, vmID) + if readErr != nil { + return nil, appendInstanceOperationError( + readErr, + "close rejected instance authority owner directory", + roots.deps.close(context.WithoutCancel(ctx), stateFD), + ) + } + if !exists { + if closeErr := roots.deps.close(context.WithoutCancel(ctx), stateFD); closeErr != nil { + return nil, instanceAtomicError("close scanned instance authority directory", closeErr) + } + continue + } + if found != nil { + closeErr := roots.deps.close(context.WithoutCancel(ctx), stateFD) + if closeErr != nil { + return nil, instanceAtomicError("close duplicate instance authority directory", closeErr) + } + return nil, instanceAtomicError("duplicate global VM authority claim", nil) + } + found = &instanceLocation{uid: uid, stateFD: stateFD} + } + return found, nil +} + +func (roots *instanceAuthorityRoots) openRuntimeForLocation( + ctx context.Context, + location *instanceLocation, +) (*instanceUIDDirectories, error) { + name := fmt.Sprintf("%d", location.uid) + runtimeFD, err := openInstanceDirectory(ctx, roots.deps, roots.runtimeFD, name, true, true, roots.policy) + if err != nil { + if closeErr := roots.deps.close(context.WithoutCancel(ctx), location.stateFD); closeErr != nil { + err = appendInstanceOperationError(err, "close located instance authority directory", closeErr) + } + return nil, err + } + return &instanceUIDDirectories{ + deps: roots.deps, + policy: roots.policy, + uid: location.uid, + stateFD: location.stateFD, + runtimeFD: runtimeFD, + }, nil +} + +func (roots *instanceAuthorityRoots) releaseIsReferenced( + ctx context.Context, + release releaseIdentity, +) (bool, error) { + ownerNames, err := roots.deps.readDirNames(ctx, roots.stateFD) + if err != nil { + return false, instanceAtomicError("enumerate release authority owners", err) + } + sort.Strings(ownerNames) + seenVMs := make(map[string]struct{}) + for _, ownerName := range ownerNames { + uid, ok := validateCanonicalUIDName(ownerName) + if !ok { + return false, instanceAtomicError("instance authority owner directory name is invalid", nil) + } + stateFD, err := openInstanceDirectory( + ctx, + roots.deps, + roots.stateFD, + ownerName, + false, + true, + roots.policy, + ) + if err != nil { + return false, err + } + referenced, scanErr := roots.releaseReferencedByOwner(ctx, release, uid, stateFD, seenVMs) + closeErr := roots.deps.close(context.WithoutCancel(ctx), stateFD) + if scanErr != nil { + return false, appendInstanceOperationError( + scanErr, + "close release-scan instance directory", + closeErr, + ) + } + if closeErr != nil { + return false, instanceAtomicError("close release-scan instance directory", closeErr) + } + if referenced { + return true, nil + } + } + return false, nil +} + +func (roots *instanceAuthorityRoots) releaseReferencedByOwner( + ctx context.Context, + release releaseIdentity, + uid uint32, + stateFD int, + seenVMs map[string]struct{}, +) (bool, error) { + names, err := roots.deps.readDirNames(ctx, stateFD) + if err != nil { + return false, instanceAtomicError("enumerate instance authority records", err) + } + sort.Strings(names) + for _, name := range names { + if isInstanceTempName(name) { + continue + } + if !strings.HasSuffix(name, ".json") { + return false, instanceAtomicError("unexpected instance authority entry", nil) + } + vmID := strings.TrimSuffix(name, ".json") + if !vmIDPattern.MatchString(vmID) { + return false, instanceAtomicError("instance authority record name is invalid", nil) + } + if _, exists := seenVMs[vmID]; exists { + return false, instanceAtomicError("duplicate global VM authority claim", nil) + } + seenVMs[vmID] = struct{}{} + runtimeFD, err := openInstanceDirectory( + ctx, + roots.deps, + roots.runtimeFD, + fmt.Sprintf("%d", uid), + true, + true, + roots.policy, + ) + if err != nil { + return false, err + } + directories := &instanceUIDDirectories{ + deps: roots.deps, policy: roots.policy, uid: uid, stateFD: stateFD, runtimeFD: runtimeFD, + } + vmLock, err := directories.acquireVMLock(ctx, vmID) + if err != nil { + return false, appendInstanceOperationError( + err, + "close rejected VM runtime directory", + roots.deps.close(context.WithoutCancel(ctx), runtimeFD), + ) + } + record, found, readErr := directories.readRecord(ctx, vmID) + releaseErr := vmLock.Release(context.WithoutCancel(ctx)) + closeErr := roots.deps.close(context.WithoutCancel(ctx), runtimeFD) + if readErr != nil { + return false, appendInstanceOperationError( + appendInstanceOperationError(readErr, "release scanned VM lock", releaseErr), + "close scanned VM runtime directory", + closeErr, + ) + } + if releaseErr != nil || closeErr != nil { + return false, instanceAtomicError( + "release scanned instance authority resources", + errors.Join(releaseErr, closeErr), + ) + } + if !found { + return false, instanceAtomicError("instance authority record disappeared while locked", nil) + } + if (record.lifecycle == instanceLifecycleRegistered || record.lifecycle == instanceLifecycleCleaning) && + record.release == release { + return true, nil + } + } + return false, nil +} + +func isInstanceTempName(name string) bool { + const prefix = ".mvm-instance-" + const suffix = ".tmp" + if !strings.HasPrefix(name, prefix) || !strings.HasSuffix(name, suffix) { + return false + } + hexValue := strings.TrimSuffix(strings.TrimPrefix(name, prefix), suffix) + if len(hexValue) != 32 { + return false + } + for _, char := range hexValue { + if (char < '0' || char > '9') && (char < 'a' || char > 'f') { + return false + } + } + return true +} diff --git a/internal/service/jailer/instance_store.go b/internal/service/jailer/instance_store.go new file mode 100644 index 00000000..53568cc2 --- /dev/null +++ b/internal/service/jailer/instance_store.go @@ -0,0 +1,247 @@ +package jailer + +import ( + "context" + "errors" + "fmt" + + "golang.org/x/sys/unix" + + "mvmctl/pkg/errs" +) + +func (directories *instanceUIDDirectories) readRecord( + ctx context.Context, + vmID string, +) (record instanceRecord, found bool, returnErr error) { + if !vmIDPattern.MatchString(vmID) { + return instanceRecord{}, false, instanceValidationError("vm ID is invalid") + } + fd, err := directories.deps.openAt(ctx, directories.stateFD, vmID+".json", instanceReadFlags, 0) + if errors.Is(err, unix.ENOENT) { + return instanceRecord{}, false, nil + } + if err != nil { + return instanceRecord{}, false, instanceAtomicError("open instance authority record", err) + } + defer func() { + if err := directories.deps.close(context.WithoutCancel(ctx), fd); err != nil { + returnErr = joinInstanceAtomicError(returnErr, "close instance authority record", err) + } + }() + if err := verifyInstanceRegularFile( + ctx, + directories.deps, + fd, + "instance authority record", + directories.policy, + ); err != nil { + return instanceRecord{}, false, err + } + raw, err := readAllInstanceRecord(ctx, directories.deps, fd) + if err != nil { + return instanceRecord{}, false, instanceAtomicError("read instance authority record", err) + } + record, err = decodeInstanceRecord(raw) + if err != nil { + return instanceRecord{}, false, err + } + if record.ownerUID != directories.uid || record.vmID != vmID { + return instanceRecord{}, false, instanceAtomicError("instance authority record identity is inconsistent", nil) + } + return record, true, nil +} + +func (directories *instanceUIDDirectories) writeRecord(ctx context.Context, record instanceRecord) error { + if record.ownerUID != directories.uid { + return instanceAtomicError("instance authority record owner does not match directory", nil) + } + raw, err := encodeInstanceRecord(record) + if err != nil { + return err + } + tempName, tempFD, err := createInstanceRecordTemp(ctx, directories) + if err != nil { + return err + } + tempOpen := true + abort := func(cause error) error { + return abortInstanceRecordWrite(ctx, directories, tempFD, tempName, tempOpen, cause) + } + if err := writeAllInstanceRecord(ctx, directories.deps, tempFD, raw); err != nil { + return abort(instanceAtomicError("write instance authority temporary record", err)) + } + if err := directories.deps.fchown( + ctx, + tempFD, + int(directories.policy.expectedUID), + int(directories.policy.expectedGID), + ); err != nil { + return abort(instanceAtomicError("set instance authority record owner", err)) + } + if err := directories.deps.fchmod(ctx, tempFD, instanceStateFileMode); err != nil { + return abort(instanceAtomicError("set instance authority record mode", err)) + } + if err := verifyInstanceRegularFile( + ctx, + directories.deps, + tempFD, + "instance authority temporary record", + directories.policy, + ); err != nil { + return abort(err) + } + var tempStat unix.Stat_t + if err := directories.deps.fstat(ctx, tempFD, &tempStat); err != nil { + return abort(instanceAtomicError("inspect instance authority temporary record size", err)) + } + if tempStat.Size != int64(len(raw)) { + return abort(instanceAtomicError("instance authority temporary record size is incomplete", nil)) + } + if err := directories.deps.fsync(ctx, tempFD); err != nil { + return abort(instanceAtomicError("sync instance authority temporary record", err)) + } + if err := directories.deps.close(context.WithoutCancel(ctx), tempFD); err != nil { + tempOpen = false + return abort(instanceAtomicError("close instance authority temporary record", err)) + } + tempOpen = false + if err := ctx.Err(); err != nil { + return abort(instanceAtomicError("replace instance authority record", err)) + } + if err := directories.inspectRecordTarget(ctx, record.vmID); err != nil { + return abort(err) + } + if err := directories.deps.renameAt( + ctx, + directories.stateFD, + tempName, + directories.stateFD, + record.vmID+".json", + ); err != nil { + return abort(instanceAtomicError("atomically replace instance authority record", err)) + } + if err := directories.deps.fsync(ctx, directories.stateFD); err != nil { + return annotateInstanceRecordReplacement( + instanceAtomicError("sync instance authority record directory", err), + true, + ) + } + return nil +} + +func (directories *instanceUIDDirectories) inspectRecordTarget( + ctx context.Context, + vmID string, +) (returnErr error) { + fd, err := directories.deps.openAt(ctx, directories.stateFD, vmID+".json", instanceInspectFlags, 0) + if errors.Is(err, unix.ENOENT) { + return nil + } + if err != nil { + return instanceAtomicError("open existing instance authority record", err) + } + defer func() { + if err := directories.deps.close(context.WithoutCancel(ctx), fd); err != nil { + returnErr = joinInstanceAtomicError(returnErr, "close existing instance authority record", err) + } + }() + return verifyInstanceRegularFile( + ctx, + directories.deps, + fd, + "existing instance authority record", + directories.policy, + ) +} + +func createInstanceRecordTemp( + ctx context.Context, + directories *instanceUIDDirectories, +) (string, int, error) { + for range instanceTempAttempts { + if err := ctx.Err(); err != nil { + return "", -1, instanceAtomicError("create instance authority temporary record", err) + } + name, err := directories.deps.randomName(ctx) + if err != nil { + return "", -1, instanceAtomicError("generate instance authority temporary name", err) + } + fd, err := directories.deps.openAt( + ctx, + directories.stateFD, + name, + instanceTempFlags, + instanceStateFileMode, + ) + if errors.Is(err, unix.EEXIST) { + continue + } + if err != nil { + return "", -1, instanceAtomicError("create instance authority temporary record", err) + } + return name, fd, nil + } + return "", -1, instanceAtomicError("exclusive instance authority temporary names exhausted", nil) +} + +func writeAllInstanceRecord( + ctx context.Context, + deps instanceAuthorityDeps, + fd int, + value []byte, +) error { + for written := 0; written < len(value); { + if err := ctx.Err(); err != nil { + return err + } + count, err := deps.write(ctx, fd, value[written:]) + if count < 0 || count > len(value)-written { + return fmt.Errorf("invalid write count %d", count) + } + written += count + if err != nil { + return err + } + if count == 0 { + return fmt.Errorf("zero-length instance authority write") + } + } + return nil +} + +func abortInstanceRecordWrite( + ctx context.Context, + directories *instanceUIDDirectories, + tempFD int, + tempName string, + tempOpen bool, + cause error, +) error { + cleanupCtx := context.WithoutCancel(ctx) + result := cause + if tempOpen { + if err := directories.deps.close(cleanupCtx, tempFD); err != nil { + result = joinInstanceAtomicError(result, "close incomplete instance authority record", err) + } + } + if err := directories.deps.unlinkAt(cleanupCtx, directories.stateFD, tempName, 0); err != nil { + result = joinInstanceAtomicError(result, "remove incomplete instance authority record", err) + } + return result +} + +func annotateInstanceRecordReplacement(err error, durabilityUncertain bool) *errs.DomainError { + domainErr := errs.AsDomainError(err) + if domainErr == nil { + domainErr = instanceAtomicError("instance authority record replacement", err) + } + if domainErr.Details == nil { + domainErr.Details = make(map[string]any) + } + domainErr.Details["record_replaced"] = true + if durabilityUncertain { + domainErr.Details["durability_uncertain"] = true + } + return domainErr +} diff --git a/internal/service/jailer/instance_types.go b/internal/service/jailer/instance_types.go new file mode 100644 index 00000000..318f9d53 --- /dev/null +++ b/internal/service/jailer/instance_types.go @@ -0,0 +1,246 @@ +package jailer + +import ( + "crypto/sha256" + "fmt" + "math" + "regexp" + "strings" + + "mvmctl/internal/infra" + "mvmctl/pkg/errs" +) + +const ( + instanceRecordSchemaVersion = uint32(1) + maxInstanceRecordBytes = 16 * 1024 +) + +var authorityHashPattern = regexp.MustCompile(`^[a-f0-9]{64}$`) + +type instanceLifecycle string + +const ( + instanceLifecycleRegistered instanceLifecycle = "registered" + instanceLifecycleCleaning instanceLifecycle = "cleaning" + instanceLifecycleCleaned instanceLifecycle = "cleaned" +) + +type instanceCaller struct { + uid uint32 +} + +type releaseIdentity struct { + version string + architecture string + firecrackerSHA256 string + jailerSHA256 string +} + +type processIdentity struct { + pid int + startTimeTicks uint64 + cgroupPath string +} + +type launchRegistration struct { + vmID string + release releaseIdentity + process processIdentity +} + +type instanceRecord struct { + schemaVersion uint32 + ownerUID uint32 + vmID string + lifecycle instanceLifecycle + release releaseIdentity + process processIdentity + cleanupGeneration uint64 +} + +func registerInstanceRecord( + caller instanceCaller, + registration launchRegistration, + existing *instanceRecord, +) (instanceRecord, error) { + if err := validateInstanceCaller(caller); err != nil { + return instanceRecord{}, err + } + if err := validateLaunchRegistration(registration); err != nil { + return instanceRecord{}, err + } + cleanupGeneration := uint64(0) + if existing != nil { + if err := validateInstanceRecord(*existing); err != nil { + return instanceRecord{}, err + } + if err := authorizeInstanceRecord(caller, *existing); err != nil { + return instanceRecord{}, err + } + if existing.vmID != registration.vmID { + return instanceRecord{}, instanceValidationError("registered VM identity does not match existing authority") + } + if existing.lifecycle != instanceLifecycleCleaned { + return instanceRecord{}, errs.AlreadyExists( + errs.CodeVMAlreadyExists, + "vm already has active privileged authority", + errs.WithEntity(registration.vmID), + ) + } + cleanupGeneration = existing.cleanupGeneration + } + record := instanceRecord{ + schemaVersion: instanceRecordSchemaVersion, + ownerUID: caller.uid, + vmID: registration.vmID, + lifecycle: instanceLifecycleRegistered, + release: registration.release, + process: registration.process, + cleanupGeneration: cleanupGeneration, + } + if err := validateInstanceRecord(record); err != nil { + return instanceRecord{}, err + } + return record, nil +} + +func beginInstanceCleanup(caller instanceCaller, record instanceRecord) (instanceRecord, error) { + if err := validateInstanceRecord(record); err != nil { + return instanceRecord{}, err + } + if err := authorizeInstanceRecord(caller, record); err != nil { + return instanceRecord{}, err + } + switch record.lifecycle { + case instanceLifecycleRegistered: + if record.cleanupGeneration == ^uint64(0) { + return instanceRecord{}, instanceStateError(record.vmID, "cleanup generation is exhausted") + } + record.lifecycle = instanceLifecycleCleaning + record.cleanupGeneration++ + case instanceLifecycleCleaning: + return record, nil + case instanceLifecycleCleaned: + return instanceRecord{}, errs.New( + errs.CodeVMNotRunning, + "vm has no registered privileged runtime", + errs.WithEntity(record.vmID), + ) + default: + return instanceRecord{}, instanceStateError(record.vmID, "vm authority lifecycle is invalid") + } + return record, nil +} + +func completeInstanceCleanup(record instanceRecord) (instanceRecord, error) { + if err := validateInstanceRecord(record); err != nil { + return instanceRecord{}, err + } + if record.lifecycle != instanceLifecycleCleaning { + return instanceRecord{}, instanceStateError(record.vmID, "vm cleanup is not in progress") + } + record.lifecycle = instanceLifecycleCleaned + return record, nil +} + +func authorizeInstanceRecord(caller instanceCaller, record instanceRecord) error { + if err := validateInstanceCaller(caller); err != nil { + return err + } + if caller.uid != record.ownerUID { + return errs.New( + errs.CodeUnauthorized, + "vm privileged authority belongs to another user", + errs.WithEntity(record.vmID), + ) + } + return nil +} + +func validateInstanceCaller(caller instanceCaller) error { + if caller.uid == 0 { + return instanceValidationError("invoking user UID must be non-zero") + } + return nil +} + +func validateLaunchRegistration(registration launchRegistration) error { + if !vmIDPattern.MatchString(registration.vmID) { + return instanceValidationError("vm ID is invalid") + } + if err := validateReleaseIdentityValue(registration.release); err != nil { + return instanceValidationError(err.Error()) + } + if err := validateProcessIdentityValue(registration.vmID, registration.process); err != nil { + return instanceValidationError(err.Error()) + } + return nil +} + +func validateInstanceRecord(record instanceRecord) error { + if record.schemaVersion != instanceRecordSchemaVersion { + return instanceAtomicError("unsupported instance authority schema", nil) + } + if record.ownerUID == 0 { + return instanceAtomicError("instance authority owner UID must be non-zero", nil) + } + if !vmIDPattern.MatchString(record.vmID) { + return instanceAtomicError("instance authority VM ID is invalid", nil) + } + switch record.lifecycle { + case instanceLifecycleRegistered, instanceLifecycleCleaning, instanceLifecycleCleaned: + default: + return instanceAtomicError("instance authority lifecycle is invalid", nil) + } + if err := validateReleaseIdentityValue(record.release); err != nil { + return instanceAtomicError(err.Error(), nil) + } + if err := validateProcessIdentityValue(record.vmID, record.process); err != nil { + return instanceAtomicError(err.Error(), nil) + } + return nil +} + +func validateReleaseIdentityValue(release releaseIdentity) error { + if !versionPattern.MatchString(release.version) || strings.Contains(release.version, "..") { + return fmt.Errorf("instance authority release version is invalid") + } + if release.architecture != "x86_64" && release.architecture != "aarch64" { + return fmt.Errorf("instance authority release architecture is invalid") + } + if !authorityHashPattern.MatchString(release.firecrackerSHA256) || + !authorityHashPattern.MatchString(release.jailerSHA256) { + return fmt.Errorf( + "instance authority release hashes must be %d lowercase hexadecimal characters", + sha256.Size*2, + ) + } + return nil +} + +func validateProcessIdentityValue(vmID string, process processIdentity) error { + if process.pid <= 0 || process.pid > math.MaxInt32 || process.startTimeTicks == 0 { + return fmt.Errorf("instance authority process identity is invalid") + } + expectedCgroup := infra.CgroupV2Root + "/" + infra.JailerCgroupParent + "/" + vmID + if process.cgroupPath != expectedCgroup { + return fmt.Errorf("instance authority cgroup identity is invalid") + } + return nil +} + +func instanceValidationError(message string) *errs.DomainError { + return errs.New(errs.CodeValidationFailed, message) +} + +func instanceStateError(vmID, message string) *errs.DomainError { + return errs.New(errs.CodeVMStateInvalid, message, errs.WithEntity(vmID)) +} + +func instanceAtomicError(message string, cause error) *errs.DomainError { + if cause == nil { + return errs.New(errs.CodeVMAtomicFailed, message, errs.WithClass(errs.ClassInternal)) + } + return errs.WrapMsg(errs.CodeVMAtomicFailed, message, cause, errs.WithClass(errs.ClassInternal)) +} diff --git a/tasks/plan.md b/tasks/plan.md index 7733d276..468df43d 100644 --- a/tasks/plan.md +++ b/tasks/plan.md @@ -28,6 +28,8 @@ continue without silently deleting it. - Early reserved privileged dispatch before Cobra and ordinary application initialization. - Strict bounded request parsing, sudo caller identity, group authorization, environment sanitization, and executable identity verification. +- Private root-owned VM ownership/lifecycle records with strict codecs, descriptor-relative atomic storage, exact + release references, global VM-ID claims, and release/index/VM locking. - ADR-0016 for the single-binary privilege boundary. The action switch is intentionally still empty and the sudoers policy remains transitional. The routed policy, dual diff --git a/tasks/todo.md b/tasks/todo.md index a58a85f3..ae3acf86 100644 --- a/tasks/todo.md +++ b/tasks/todo.md @@ -64,47 +64,47 @@ The final marker-only sudoers assertion is deferred until Task 14; current sudoe ### Task 4: Add private root-owned VM instance authority **Owner:** engineer -**Status:** Ready for implementation +**Status:** Complete **Dependencies:** Tasks 1 and 3 **Interface:** -- [ ] Implement private `RegisterLaunch(ctx, caller, registration) (*launchLease, error)`. -- [ ] Implement private `LockRegistered(ctx, caller, vmID) (*registeredLease, error)`. -- [ ] Implement private `BeginCleanup(ctx, caller, vmID) (*cleanupLease, error)`. -- [ ] Implement context-first `cleanupLease.Complete(ctx)` and checked lease release. -- [ ] Implement private `LockUnreferencedRelease(ctx, release) (*releaseLease, error)`. -- [ ] Expose no generic storage, callbacks, raw paths, actions, or state strings. +- [x] Implement private `RegisterLaunch(ctx, caller, registration) (*launchLease, error)`. +- [x] Implement private `LockRegistered(ctx, caller, vmID) (*registeredLease, error)`. +- [x] Implement private `BeginCleanup(ctx, caller, vmID) (*cleanupLease, error)`. +- [x] Implement context-first `cleanupLease.Complete(ctx)` and checked lease release. +- [x] Implement private `LockUnreferencedRelease(ctx, release) (*releaseLease, error)`. +- [x] Expose no generic storage, callbacks, raw paths, actions, or state strings. **Authority and lifecycle:** -- [ ] Persist strict versioned records under `/var/lib/mvmctl/instances//.json`. -- [ ] Store only owner UID, VM ID, lifecycle, exact release identity, process identity, cgroup path, and cleanup generation. -- [ ] Implement `absent|cleaned -> registered -> cleaning -> cleaned`; failed cleanup remains recoverable. -- [ ] Keep a `cleaned` ownership tombstone; foreign UIDs never acquire the VM ID. -- [ ] Establish the global VM-ID claim while holding the global index lock and scanning pinned numeric UID directories. -- [ ] Reject foreign, duplicate, corrupt, unreadable, unsafe, or inconsistent records without disclosing the owner UID. +- [x] Persist strict versioned records under `/var/lib/mvmctl/instances//.json`. +- [x] Store only owner UID, VM ID, lifecycle, exact release identity, process identity, cgroup path, and cleanup generation. +- [x] Implement `absent|cleaned -> registered -> cleaning -> cleaned`; failed cleanup remains recoverable. +- [x] Keep a `cleaned` ownership tombstone; foreign UIDs never acquire the VM ID. +- [x] Establish the global VM-ID claim while holding the global index lock and scanning pinned numeric UID directories. +- [x] Reject foreign, duplicate, corrupt, unreadable, unsafe, or inconsistent records without disclosing the owner UID. **Filesystem and locking:** -- [ ] Use descriptor-relative no-follow traversal and retain required root/record/lock FDs. -- [ ] Verify root ownership, exact file types, and non-writable ancestors. -- [ ] Use exact `0700` managed directories and `0600` record/lock files. -- [ ] Implement the Task 4 lock prefix: release -> index -> VM. -- [ ] Use cancellable nonblocking flock retry; never unlink lock files. -- [ ] Atomically replace records using exclusive temp, bounded write, ownership/mode, fsync, renameat, and directory fsync. -- [ ] Return `record_replaced` / `durability_uncertain` DomainError details for post-rename failure. +- [x] Use descriptor-relative no-follow traversal and retain required root/record/lock FDs. +- [x] Verify root ownership, exact file types, and non-writable ancestors. +- [x] Use exact `0700` managed directories and `0600` record/lock files. +- [x] Implement the Task 4 lock prefix: release -> index -> VM. +- [x] Use cancellable nonblocking flock retry; never unlink lock files. +- [x] Atomically replace records using exclusive temp, bounded write, ownership/mode, fsync, renameat, and directory fsync. +- [x] Return `record_replaced` / `durability_uncertain` DomainError details for post-rename failure. **TDD and verification:** -- [ ] RED/GREEN strict codec and pure lifecycle-transition tests. -- [ ] RED/GREEN descriptor store, atomic-write, and context-aware lock tests. -- [ ] RED/GREEN typed authority behavior and release-reference tests. -- [ ] Real concurrent two-UID first-launch race proves exactly one durable owner. -- [ ] Tests cover every unsafe ancestor, symlink level, owner/mode/type mismatch, EEXIST race, partial I/O, fsync/close/ +- [x] RED/GREEN strict codec and pure lifecycle-transition tests. +- [x] RED/GREEN descriptor store, atomic-write, and context-aware lock tests. +- [x] RED/GREEN typed authority behavior and release-reference tests. +- [x] Real concurrent two-UID first-launch race proves exactly one durable owner. +- [x] Tests cover every unsafe ancestor, symlink level, owner/mode/type mismatch, EEXIST race, partial I/O, fsync/close/ rename failure, cancellation, stale lock reuse, corrupt record, and temp cleanup. -- [ ] Focused race test, format, golines, vet, and Go tests pass. -- [ ] Architect reviews actual interface, state schema, lock order, imports, and error details before commit. +- [x] Focused race test, format, golines, vet, and Go tests pass. +- [x] Architect reviews actual interface, state schema, lock order, imports, and error details before commit. **Expected files:** fixed root constants; private instance types, codec, descriptor store, lock, authority, and focused tests under `internal/service/jailer/`. No handler, transport, CLI, API, core-domain, or network changes. From 1c28fb5ac3e758d6de8d35b8fedea4b355a19c1e Mon Sep 17 00:00:00 2001 From: AlanD20 Date: Sun, 23 Aug 2026 14:27:02 +0200 Subject: [PATCH 0012/1457] fix: preserve invoking user state ownership --- internal/infra/constants.go | 53 +-- internal/infra/io.go | 20 +- internal/infra/user_dirs.go | 337 ++++++++++++++ internal/infra/user_dirs_internal_test.go | 536 ++++++++++++++++++++++ 4 files changed, 896 insertions(+), 50 deletions(-) create mode 100644 internal/infra/user_dirs.go create mode 100644 internal/infra/user_dirs_internal_test.go diff --git a/internal/infra/constants.go b/internal/infra/constants.go index c1d2bb87..5d699c51 100644 --- a/internal/infra/constants.go +++ b/internal/infra/constants.go @@ -1,6 +1,7 @@ package infra import ( + "context" "encoding/json" "fmt" "log/slog" @@ -500,25 +501,15 @@ var ( func GetCacheDir() (string, error) { cacheDirOnce.Do(func() { - override, ok := EnvGet("CACHE_DIR") - if ok && override != "" { - var resolved string - resolved, cacheDirErr = filepath.Abs(override) - if cacheDirErr != nil { - cacheDirErr = fmt.Errorf("invalid cache dir path: %w", cacheDirErr) - return - } - // Ensure the directory exists with proper permissions. - if err := ensureDirAndChown(resolved); err != nil { - cacheDirErr = fmt.Errorf("create cache dir: %w", err) - return - } - cacheDirVal = resolved - return - } - path := filepath.Join(GetRealHome(), ".cache", ProjectName) - if err := ensureDirAndChown(path); err != nil { - cacheDirErr = fmt.Errorf("create default cache dir: %w", err) + override, _ := EnvGet("CACHE_DIR") + path, err := resolveUserDir( + context.Background(), + defaultUserCacheDir, + override, + realDefaultUserDirDeps(), + ) + if err != nil { + cacheDirErr = fmt.Errorf("create cache dir: %w", err) return } cacheDirVal = path @@ -527,21 +518,15 @@ func GetCacheDir() (string, error) { } func GetConfigDir() (string, error) { - override, ok := EnvGet("CONFIG_DIR") - if ok && override != "" { - resolved, err := filepath.Abs(override) - if err != nil { - return "", fmt.Errorf("invalid config dir path: %w", err) - } - // Ensure the directory exists with proper permissions. - if err := ensureDirAndChown(resolved); err != nil { - return "", fmt.Errorf("create config dir: %w", err) - } - return resolved, nil - } - path := filepath.Join(GetRealHome(), ".config", ProjectName) - if err := ensureDirAndChown(path); err != nil { - return "", fmt.Errorf("create default config dir: %w", err) + override, _ := EnvGet("CONFIG_DIR") + path, err := resolveUserDir( + context.Background(), + defaultUserConfigDir, + override, + realDefaultUserDirDeps(), + ) + if err != nil { + return "", fmt.Errorf("create config dir: %w", err) } return path, nil } diff --git a/internal/infra/io.go b/internal/infra/io.go index 18bb24a7..48414608 100644 --- a/internal/infra/io.go +++ b/internal/infra/io.go @@ -6,7 +6,6 @@ import ( "io" "net" "os" - "os/user" "path/filepath" "strconv" "strings" @@ -127,24 +126,13 @@ func WritePIDFile(path string, pid int, mode ...os.FileMode) error { // --- Real user ID resolution (for sudo chown) --- -// GetRealUserIDs returns (uid, gid) of the real invoking user when running -// under sudo. Returns nil if not running as root, or if SUDO_USER is not set -// or cannot be resolved. +// GetRealUserIDs returns (uid, gid) of a complete, UID-bound sudo identity. func GetRealUserIDs() (uid, gid int, ok bool) { - if os.Getuid() != 0 { + identity, sudo, err := resolveSudoUserIdentity(realDefaultUserDirDeps()) + if err != nil || !sudo { return 0, 0, false } - sudoUser := os.Getenv("SUDO_USER") - if sudoUser == "" { - return 0, 0, false - } - u, err := user.Lookup(sudoUser) - if err != nil { - return 0, 0, false - } - uid, _ = strconv.Atoi(u.Uid) - gid, _ = strconv.Atoi(u.Gid) - return uid, gid, true + return int(identity.uid), int(identity.gid), true } // ChownToRealUser recursively chowns a path to the real invoking user when diff --git a/internal/infra/user_dirs.go b/internal/infra/user_dirs.go new file mode 100644 index 00000000..74316db0 --- /dev/null +++ b/internal/infra/user_dirs.go @@ -0,0 +1,337 @@ +package infra + +import ( + "context" + "errors" + "fmt" + "math" + "os" + "os/user" + "path/filepath" + "strconv" + "strings" + + "golang.org/x/sys/unix" + + "mvmctl/pkg/errs" +) + +const defaultUserDirectoryFlags = unix.O_RDONLY | unix.O_DIRECTORY | unix.O_CLOEXEC | unix.O_NOFOLLOW + +type defaultUserDirKind uint8 + +const ( + defaultUserCacheDir defaultUserDirKind = iota + defaultUserConfigDir +) + +type sudoUserIdentity struct { + uid uint32 + gid uint32 + username string + home string +} + +type defaultUserDirDeps struct { + effectiveUID func() int + lookupEnv func(string) (string, bool) + lookupUserID func(string) (*user.User, error) + userHomeDir func() (string, error) + open func(context.Context, string, int, uint32) (int, error) + openAt func(context.Context, int, string, int, uint32) (int, error) + mkdirAt func(context.Context, int, string, uint32) error + fstat func(context.Context, int, *unix.Stat_t) error + fchmod func(context.Context, int, uint32) error + fchown func(context.Context, int, int, int) error + fsync func(context.Context, int) error + close func(context.Context, int) error + mkdirAll func(context.Context, string, os.FileMode) error +} + +func resolveDefaultUserDir( + ctx context.Context, + kind defaultUserDirKind, + deps defaultUserDirDeps, +) (_ string, returnErr error) { + baseName, err := defaultUserBaseName(kind) + if err != nil { + return "", err + } + identity, sudo, err := resolveSudoUserIdentity(deps) + if err != nil { + return "", err + } + if !sudo { + home, homeErr := deps.userHomeDir() + if homeErr != nil { + return "", defaultUserDirError("resolve current user home directory", homeErr) + } + path := filepath.Join(home, baseName, ProjectName) + if mkdirErr := deps.mkdirAll(ctx, path, CacheDirPerm); mkdirErr != nil { + return "", defaultUserDirError("create default user directory", mkdirErr) + } + return path, nil + } + + homeFD, err := deps.open(ctx, identity.home, defaultUserDirectoryFlags, 0) + if err != nil { + return "", defaultUserDirError("open invoking user home directory", err) + } + defer func() { + if closeErr := deps.close(context.WithoutCancel(ctx), homeFD); closeErr != nil { + returnErr = joinDefaultUserDirError(returnErr, "close invoking user home directory", closeErr) + } + }() + if err := verifySudoUserDirectory(ctx, deps, homeFD, identity, false); err != nil { + return "", err + } + + baseFD, err := openSudoUserDirectory(ctx, deps, homeFD, baseName, identity) + if err != nil { + return "", err + } + defer func() { + if closeErr := deps.close(context.WithoutCancel(ctx), baseFD); closeErr != nil { + returnErr = joinDefaultUserDirError(returnErr, "close invoking user base directory", closeErr) + } + }() + projectFD, err := openSudoUserDirectory(ctx, deps, baseFD, ProjectName, identity) + if err != nil { + return "", err + } + if err := deps.close(context.WithoutCancel(ctx), projectFD); err != nil { + return "", defaultUserDirError("close invoking user project directory", err) + } + return filepath.Join(identity.home, baseName, ProjectName), nil +} + +func resolveUserDir( + ctx context.Context, + kind defaultUserDirKind, + override string, + deps defaultUserDirDeps, +) (string, error) { + if override == "" { + return resolveDefaultUserDir(ctx, kind, deps) + } + _, sudo, err := resolveSudoUserIdentity(deps) + if err != nil { + return "", err + } + if sudo { + return "", errs.New(errs.CodeConfigError, "user directory override is not allowed under sudo") + } + path, err := filepath.Abs(override) + if err != nil { + return "", defaultUserDirError("resolve user directory override", err) + } + if err := deps.mkdirAll(ctx, path, CacheDirPerm); err != nil { + return "", defaultUserDirError("create user directory override", err) + } + return path, nil +} + +func defaultUserBaseName(kind defaultUserDirKind) (string, error) { + switch kind { + case defaultUserCacheDir: + return ".cache", nil + case defaultUserConfigDir: + return ".config", nil + default: + return "", errs.New(errs.CodeConfigError, "unknown default user directory kind") + } +} + +func openSudoUserDirectory( + ctx context.Context, + deps defaultUserDirDeps, + parentFD int, + name string, + identity sudoUserIdentity, +) (_ int, returnErr error) { + if err := ctx.Err(); err != nil { + return -1, defaultUserDirError("create invoking user directory", err) + } + fd, err := deps.openAt(ctx, parentFD, name, defaultUserDirectoryFlags, 0) + if err == nil { + if verifyErr := verifySudoUserDirectory(ctx, deps, fd, identity, false); verifyErr != nil { + if closeErr := deps.close(context.WithoutCancel(ctx), fd); closeErr != nil { + verifyErr = joinDefaultUserDirError(verifyErr, "close rejected invoking user directory", closeErr) + } + return -1, verifyErr + } + return fd, nil + } + if !errors.Is(err, unix.ENOENT) { + return -1, defaultUserDirError("open invoking user directory "+name, err) + } + mkdirErr := deps.mkdirAt(ctx, parentFD, name, uint32(CacheDirPerm)) + created := mkdirErr == nil + if mkdirErr != nil && !errors.Is(mkdirErr, unix.EEXIST) { + return -1, defaultUserDirError("create invoking user directory "+name, mkdirErr) + } + fd, err = deps.openAt(ctx, parentFD, name, defaultUserDirectoryFlags, 0) + if err != nil { + return -1, defaultUserDirError("open created invoking user directory "+name, err) + } + defer func() { + if returnErr == nil { + return + } + if closeErr := deps.close(context.WithoutCancel(ctx), fd); closeErr != nil { + returnErr = joinDefaultUserDirError(returnErr, "close incomplete invoking user directory", closeErr) + } + }() + if !created { + if err := verifySudoUserDirectory(ctx, deps, fd, identity, false); err != nil { + return -1, err + } + return fd, nil + } + if err := deps.fchmod(ctx, fd, uint32(CacheDirPerm)); err != nil { + return -1, defaultUserDirError("set invoking user directory mode "+name, err) + } + if err := deps.fchown(ctx, fd, int(identity.uid), int(identity.gid)); err != nil { + return -1, defaultUserDirError("set invoking user directory owner "+name, err) + } + if err := verifySudoUserDirectory(ctx, deps, fd, identity, true); err != nil { + return -1, err + } + if err := deps.fsync(ctx, fd); err != nil { + return -1, defaultUserDirError("sync invoking user directory "+name, err) + } + if err := deps.fsync(ctx, parentFD); err != nil { + return -1, defaultUserDirError("sync parent of invoking user directory "+name, err) + } + return fd, nil +} + +func verifySudoUserDirectory( + ctx context.Context, + deps defaultUserDirDeps, + fd int, + identity sudoUserIdentity, + requireExactMode bool, +) error { + var stat unix.Stat_t + if err := deps.fstat(ctx, fd, &stat); err != nil { + return defaultUserDirError("inspect invoking user directory", err) + } + if stat.Mode&unix.S_IFMT != unix.S_IFDIR { + return errs.New(errs.CodeConfigError, "invoking user directory is not a directory") + } + if stat.Uid != identity.uid || stat.Mode&0700 != 0700 || stat.Mode&0022 != 0 { + return errs.New(errs.CodeConfigError, "invoking user directory has unsafe ownership or access mode") + } + if requireExactMode && (stat.Gid != identity.gid || stat.Mode&07777 != uint32(CacheDirPerm)) { + return errs.New(errs.CodeConfigError, "created invoking user directory has unexpected owner or mode") + } + return nil +} + +func resolveSudoUserIdentity(deps defaultUserDirDeps) (sudoUserIdentity, bool, error) { + if deps.effectiveUID() != 0 { + return sudoUserIdentity{}, false, nil + } + uidValue, hasUID := deps.lookupEnv("SUDO_UID") + gidValue, hasGID := deps.lookupEnv("SUDO_GID") + username, hasUser := deps.lookupEnv("SUDO_USER") + if !hasUID && !hasGID && !hasUser { + return sudoUserIdentity{}, false, nil + } + if !hasUID || !hasGID || !hasUser || uidValue == "" || gidValue == "" || username == "" { + return sudoUserIdentity{}, false, errs.New( + errs.CodeConfigError, + "complete sudo identity is required for user directory ownership", + ) + } + uid, err := parseSudoUserIdentityNumber("SUDO_UID", uidValue) + if err != nil { + return sudoUserIdentity{}, false, err + } + gid, err := parseSudoUserIdentityNumber("SUDO_GID", gidValue) + if err != nil { + return sudoUserIdentity{}, false, err + } + invokingUser, err := deps.lookupUserID(strconv.FormatUint(uint64(uid), 10)) + if err != nil { + return sudoUserIdentity{}, false, defaultUserDirError("look up sudo user by UID", err) + } + lookupUID, uidErr := strconv.ParseUint(invokingUser.Uid, 10, 32) + if uidErr != nil || uint32(lookupUID) != uid { + return sudoUserIdentity{}, false, errs.New( + errs.CodeConfigError, + "sudo user lookup returned an inconsistent UID", + ) + } + lookupGID, gidErr := strconv.ParseUint(invokingUser.Gid, 10, 32) + if gidErr != nil || uint32(lookupGID) != gid { + return sudoUserIdentity{}, false, errs.New(errs.CodeConfigError, "sudo user lookup does not match SUDO_GID") + } + if invokingUser.Username != username { + return sudoUserIdentity{}, false, errs.New(errs.CodeConfigError, "sudo user lookup does not match SUDO_USER") + } + if !filepath.IsAbs(invokingUser.HomeDir) || filepath.Clean(invokingUser.HomeDir) != invokingUser.HomeDir || + invokingUser.HomeDir == "/" || strings.ContainsRune(invokingUser.HomeDir, '\x00') { + return sudoUserIdentity{}, false, errs.New( + errs.CodeConfigError, + "sudo user lookup returned an unsafe home directory", + ) + } + return sudoUserIdentity{ + uid: uid, + gid: gid, + username: invokingUser.Username, + home: invokingUser.HomeDir, + }, true, nil +} + +func parseSudoUserIdentityNumber(name, value string) (uint32, error) { + parsed, err := strconv.ParseUint(value, 10, 32) + if err != nil || parsed == 0 || parsed == math.MaxUint32 || strconv.FormatUint(parsed, 10) != value { + return 0, errs.New(errs.CodeConfigError, fmt.Sprintf("invalid %s: expected a canonical non-root ID", name)) + } + return uint32(parsed), nil +} + +func realDefaultUserDirDeps() defaultUserDirDeps { + return defaultUserDirDeps{ + effectiveUID: os.Geteuid, + lookupEnv: os.LookupEnv, + lookupUserID: user.LookupId, + userHomeDir: os.UserHomeDir, + open: func(_ context.Context, path string, flags int, mode uint32) (int, error) { + return unix.Open(path, flags, mode) + }, + openAt: func(_ context.Context, parentFD int, name string, flags int, mode uint32) (int, error) { + return unix.Openat(parentFD, name, flags, mode) + }, + mkdirAt: func(_ context.Context, parentFD int, name string, mode uint32) error { + return unix.Mkdirat(parentFD, name, mode) + }, + fstat: func(_ context.Context, fd int, stat *unix.Stat_t) error { return unix.Fstat(fd, stat) }, + fchmod: func(_ context.Context, fd int, mode uint32) error { return unix.Fchmod(fd, mode) }, + fchown: func(_ context.Context, fd, uid, gid int) error { return unix.Fchown(fd, uid, gid) }, + fsync: func(_ context.Context, fd int) error { return unix.Fsync(fd) }, + close: func(_ context.Context, fd int) error { return unix.Close(fd) }, + mkdirAll: func(_ context.Context, path string, mode os.FileMode) error { + return os.MkdirAll(path, mode) + }, + } +} + +func defaultUserDirError(message string, cause error) *errs.DomainError { + return errs.WrapMsg(errs.CodeConfigError, message, cause) +} + +func joinDefaultUserDirError(primary error, message string, cause error) *errs.DomainError { + if primary == nil { + return defaultUserDirError(message, cause) + } + if domainErr := errs.AsDomainError(primary); domainErr != nil { + domainErr.Message += "; " + message + ": " + cause.Error() + domainErr.Err = errors.Join(domainErr.Err, cause) + return domainErr + } + return defaultUserDirError(primary.Error()+"; "+message, errors.Join(primary, cause)) +} diff --git a/internal/infra/user_dirs_internal_test.go b/internal/infra/user_dirs_internal_test.go new file mode 100644 index 00000000..4fd18729 --- /dev/null +++ b/internal/infra/user_dirs_internal_test.go @@ -0,0 +1,536 @@ +package infra + +import ( + "context" + "errors" + "os" + "os/user" + "path/filepath" + "strconv" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + "golang.org/x/sys/unix" + + "mvmctl/pkg/errs" +) + +func TestResolveDefaultUserDirFreshSudoHomeSupportsUnprivilegedFollowUp(t *testing.T) { + t.Parallel() + + home := t.TempDir() + uid := uint32(os.Geteuid()) + gid := uint32(os.Getegid()) + require.NotZero(t, uid, "the ownership regression requires an unprivileged test process") + + tests := map[string]struct { + kind defaultUserDirKind + baseName string + }{ + "cache": {kind: defaultUserCacheDir, baseName: ".cache"}, + "config": {kind: defaultUserConfigDir, baseName: ".config"}, + } + for name, tc := range tests { + t.Run(name, func(t *testing.T) { + deps := testDefaultUserDirDeps(home, uid, gid) + realFchown := deps.fchown + fchowned := make([]string, 0, 2) + deps.fchown = func(ctx context.Context, fd int, ownerUID, ownerGID int) error { + fchowned = append(fchowned, descriptorPath(t, fd)) + return realFchown(ctx, fd, ownerUID, ownerGID) + } + + path, err := resolveDefaultUserDir(context.Background(), tc.kind, deps) + require.NoError(t, err) + assert.Equal(t, filepath.Join(home, tc.baseName, ProjectName), path) + assert.Equal(t, []string{filepath.Join(home, tc.baseName), path}, fchowned) + assertDirectoryOwner(t, filepath.Join(home, tc.baseName), uid, gid) + assertDirectoryOwner(t, path, uid, gid) + + // This write runs as the original unprivileged test user, reproducing + // the first command after the sudo host-init process exits. + require.NoError(t, os.WriteFile(filepath.Join(path, "follow-up"), []byte("ok"), 0600)) + }) + } +} + +func TestResolveDefaultUserDirRejectsWritableExistingDirectory(t *testing.T) { + t.Parallel() + + uid := uint32(os.Geteuid()) + gid := uint32(os.Getegid()) + require.NotZero(t, uid) + tests := map[string]struct { + prepare func(t *testing.T, home string) + }{ + "home": { + prepare: func(t *testing.T, home string) { + require.NoError(t, os.Chmod(home, 0777)) + }, + }, + "base": { + prepare: func(t *testing.T, home string) { + base := filepath.Join(home, ".cache") + require.NoError(t, os.Mkdir(base, 0700)) + require.NoError(t, os.Chmod(base, 0777)) + }, + }, + "project": { + prepare: func(t *testing.T, home string) { + project := filepath.Join(home, ".cache", ProjectName) + require.NoError(t, os.MkdirAll(project, 0700)) + require.NoError(t, os.Chmod(project, 0777)) + }, + }, + } + for name, tc := range tests { + t.Run(name, func(t *testing.T) { + home := t.TempDir() + tc.prepare(t, home) + + _, err := resolveDefaultUserDir( + context.Background(), + defaultUserCacheDir, + testDefaultUserDirDeps(home, uid, gid), + ) + require.Error(t, err) + assert.ErrorContains(t, err, "unsafe ownership or access mode") + }) + } +} + +func TestResolveDefaultUserDirAllowsReadOnlyExistingDirectoryAccess(t *testing.T) { + t.Parallel() + + home := t.TempDir() + uid := uint32(os.Geteuid()) + gid := uint32(os.Getegid()) + require.NotZero(t, uid) + project := filepath.Join(home, ".cache", ProjectName) + require.NoError(t, os.MkdirAll(project, 0700)) + for _, path := range []string{home, filepath.Join(home, ".cache"), project} { + require.NoError(t, os.Chmod(path, 0755)) + } + deps := testDefaultUserDirDeps(home, uid, gid) + deps.fchown = func(context.Context, int, int, int) error { + t.Fatal("existing safe directories must not be chowned") + return nil + } + + path, err := resolveDefaultUserDir(context.Background(), defaultUserCacheDir, deps) + require.NoError(t, err) + assert.Equal(t, project, path) +} + +func TestResolveDefaultUserDirRejectsUnsafeEEXISTRaceWithoutChown(t *testing.T) { + t.Parallel() + + uid := uint32(os.Geteuid()) + gid := uint32(os.Getegid()) + require.NotZero(t, uid) + tests := map[string]func(t *testing.T, path string){ + "writable_directory": func(t *testing.T, path string) { + require.NoError(t, os.Mkdir(path, 0700)) + require.NoError(t, os.Chmod(path, 0777)) + }, + "symlink": func(t *testing.T, path string) { + require.NoError(t, os.Symlink(t.TempDir(), path)) + }, + } + for name, substitute := range tests { + t.Run(name, func(t *testing.T) { + home := t.TempDir() + deps := testDefaultUserDirDeps(home, uid, gid) + realOpenAt := deps.openAt + baseOpenCount := 0 + deps.openAt = func(ctx context.Context, parentFD int, entry string, flags int, mode uint32) (int, error) { + if entry == ".cache" { + baseOpenCount++ + if baseOpenCount == 1 { + return -1, unix.ENOENT + } + } + return realOpenAt(ctx, parentFD, entry, flags, mode) + } + deps.mkdirAt = func(context.Context, int, string, uint32) error { + substitute(t, filepath.Join(home, ".cache")) + return unix.EEXIST + } + deps.fchown = func(context.Context, int, int, int) error { + t.Fatal("an EEXIST entry must be verified before ownership changes") + return nil + } + + _, err := resolveDefaultUserDir(context.Background(), defaultUserCacheDir, deps) + require.Error(t, err) + require.NotNil(t, errs.AsDomainError(err)) + assert.Equal(t, 2, baseOpenCount) + }) + } +} + +func TestResolveDefaultUserDirStopsAfterIncompleteBaseOwnership(t *testing.T) { + t.Parallel() + + uid := uint32(os.Geteuid()) + gid := uint32(os.Getegid()) + require.NotZero(t, uid) + tests := map[string]struct { + configure func(deps *defaultUserDirDeps, failure error) + message string + }{ + "fchmod": { + configure: func(deps *defaultUserDirDeps, failure error) { + deps.fchmod = func(context.Context, int, uint32) error { return failure } + }, + message: "set invoking user directory mode .cache", + }, + "fchown": { + configure: func(deps *defaultUserDirDeps, failure error) { + deps.fchown = func(context.Context, int, int, int) error { return failure } + }, + message: "set invoking user directory owner .cache", + }, + } + for name, tc := range tests { + t.Run(name, func(t *testing.T) { + home := t.TempDir() + deps := testDefaultUserDirDeps(home, uid, gid) + realMkdirAt := deps.mkdirAt + created := make([]string, 0, 1) + deps.mkdirAt = func(ctx context.Context, parentFD int, entry string, mode uint32) error { + created = append(created, entry) + return realMkdirAt(ctx, parentFD, entry, mode) + } + failure := errors.New(name + " failed") + tc.configure(&deps, failure) + + _, err := resolveDefaultUserDir(context.Background(), defaultUserCacheDir, deps) + require.Error(t, err) + domainErr := errs.AsDomainError(err) + require.NotNil(t, domainErr) + assert.Equal(t, errs.CodeConfigError, domainErr.Code) + assert.ErrorContains(t, err, tc.message) + assert.ErrorIs(t, err, failure) + assert.Equal(t, []string{".cache"}, created) + _, projectErr := os.Lstat(filepath.Join(home, ".cache", ProjectName)) + assert.ErrorIs(t, projectErr, os.ErrNotExist) + }) + } +} + +func TestResolveDefaultUserDirSurfacesSyncAndCloseFailures(t *testing.T) { + t.Parallel() + + uid := uint32(os.Geteuid()) + gid := uint32(os.Getegid()) + require.NotZero(t, uid) + tests := map[string]struct { + configure func(t *testing.T, deps *defaultUserDirDeps, primary, closeFailure error) + wantPrimary bool + wantClose bool + errorContains string + }{ + "fsync": { + configure: func(_ *testing.T, deps *defaultUserDirDeps, primary, _ error) { + deps.fsync = func(context.Context, int) error { return primary } + }, + wantPrimary: true, + errorContains: "sync invoking user directory .cache", + }, + "close": { + configure: func(t *testing.T, deps *defaultUserDirDeps, _, closeFailure error) { + realClose := deps.close + deps.close = func(ctx context.Context, fd int) error { + path := descriptorPath(t, fd) + err := realClose(ctx, fd) + if filepath.Base(path) == ProjectName { + return closeFailure + } + return err + } + }, + wantClose: true, + errorContains: "close invoking user project directory", + }, + "fsync_and_close": { + configure: func(t *testing.T, deps *defaultUserDirDeps, primary, closeFailure error) { + deps.fsync = func(context.Context, int) error { return primary } + realClose := deps.close + deps.close = func(ctx context.Context, fd int) error { + path := descriptorPath(t, fd) + err := realClose(ctx, fd) + if filepath.Base(path) == ".cache" { + return closeFailure + } + return err + } + }, + wantPrimary: true, + wantClose: true, + errorContains: "sync invoking user directory .cache", + }, + } + for name, tc := range tests { + t.Run(name, func(t *testing.T) { + home := t.TempDir() + deps := testDefaultUserDirDeps(home, uid, gid) + primary := errors.New("sync failed") + closeFailure := errors.New("close failed") + tc.configure(t, &deps, primary, closeFailure) + + _, err := resolveDefaultUserDir(context.Background(), defaultUserCacheDir, deps) + require.Error(t, err) + domainErr := errs.AsDomainError(err) + require.NotNil(t, domainErr) + assert.Equal(t, errs.CodeConfigError, domainErr.Code) + assert.ErrorContains(t, err, tc.errorContains) + if tc.wantPrimary { + assert.ErrorIs(t, err, primary) + } + if tc.wantClose { + assert.ErrorIs(t, err, closeFailure) + } + }) + } +} + +func TestResolveUserDirRejectsOverrideUnderSudoBeforeFilesystemMutation(t *testing.T) { + t.Parallel() + + home := t.TempDir() + uid := uint32(os.Geteuid()) + gid := uint32(os.Getegid()) + require.NotZero(t, uid) + override := filepath.Join(home, "caller-selected", "cache") + deps := testDefaultUserDirDeps(home, uid, gid) + deps.mkdirAll = func(context.Context, string, os.FileMode) error { + t.Fatal("sudo override must fail before filesystem mutation") + return nil + } + + _, err := resolveUserDir(context.Background(), defaultUserCacheDir, override, deps) + require.Error(t, err) + assert.ErrorContains(t, err, "override is not allowed under sudo") + _, statErr := os.Lstat(filepath.Dir(override)) + assert.ErrorIs(t, statErr, os.ErrNotExist) +} + +func TestResolveUserDirPreservesUnprivilegedOverride(t *testing.T) { + t.Parallel() + + override := filepath.Join(t.TempDir(), "caller-selected", "config") + deps := realDefaultUserDirDeps() + deps.effectiveUID = func() int { return 1000 } + + path, err := resolveUserDir(context.Background(), defaultUserConfigDir, override, deps) + require.NoError(t, err) + absolute, err := filepath.Abs(override) + require.NoError(t, err) + assert.Equal(t, absolute, path) + info, err := os.Stat(path) + require.NoError(t, err) + assert.True(t, info.IsDir()) +} + +func TestResolveDefaultUserDirDoesNotChownPreExistingBaseOrContents(t *testing.T) { + t.Parallel() + + home := t.TempDir() + uid := uint32(os.Geteuid()) + gid := uint32(os.Getegid()) + require.NotZero(t, uid) + base := filepath.Join(home, ".cache") + require.NoError(t, os.Mkdir(base, 0700)) + sentinel := filepath.Join(base, "existing-user-data") + require.NoError(t, os.WriteFile(sentinel, []byte("untouched"), 0600)) + deps := testDefaultUserDirDeps(home, uid, gid) + realFchown := deps.fchown + fchowned := make([]string, 0, 1) + deps.fchown = func(ctx context.Context, fd int, ownerUID, ownerGID int) error { + fchowned = append(fchowned, descriptorPath(t, fd)) + return realFchown(ctx, fd, ownerUID, ownerGID) + } + + path, err := resolveDefaultUserDir(context.Background(), defaultUserCacheDir, deps) + require.NoError(t, err) + assert.Equal(t, []string{path}, fchowned) + content, err := os.ReadFile(sentinel) + require.NoError(t, err) + assert.Equal(t, "untouched", string(content)) +} + +func TestResolveDefaultUserDirDoesNotChownPreExistingProjectContents(t *testing.T) { + t.Parallel() + + home := t.TempDir() + uid := uint32(os.Geteuid()) + gid := uint32(os.Getegid()) + require.NotZero(t, uid) + project := filepath.Join(home, ".config", ProjectName) + require.NoError(t, os.MkdirAll(project, 0700)) + sentinel := filepath.Join(project, "existing-user-data") + require.NoError(t, os.WriteFile(sentinel, []byte("untouched"), 0600)) + deps := testDefaultUserDirDeps(home, uid, gid) + deps.fchown = func(context.Context, int, int, int) error { + t.Fatal("pre-existing project directory must not be chowned") + return nil + } + + path, err := resolveDefaultUserDir(context.Background(), defaultUserConfigDir, deps) + require.NoError(t, err) + assert.Equal(t, project, path) + content, err := os.ReadFile(sentinel) + require.NoError(t, err) + assert.Equal(t, "untouched", string(content)) +} + +func TestResolveSudoUserIdentityFailsClosed(t *testing.T) { + t.Parallel() + + valid := map[string]string{ + "SUDO_UID": "1000", + "SUDO_GID": "1001", + "SUDO_USER": "runner", + } + tests := map[string]struct { + environment map[string]string + lookup func(string) (*user.User, error) + wantSudo bool + wantErr string + }{ + "valid": { + environment: valid, + lookup: func(string) (*user.User, error) { + return &user.User{Uid: "1000", Gid: "1001", Username: "runner", HomeDir: "/home/runner"}, nil + }, + wantSudo: true, + }, + "partial_environment": { + environment: map[string]string{"SUDO_USER": "runner"}, + wantErr: "complete sudo identity", + }, + "malformed_uid": { + environment: map[string]string{"SUDO_UID": "user", "SUDO_GID": "1001", "SUDO_USER": "runner"}, + wantErr: "invalid SUDO_UID", + }, + "root_uid": { + environment: map[string]string{"SUDO_UID": "0", "SUDO_GID": "1001", "SUDO_USER": "runner"}, + wantErr: "non-root", + }, + "lookup_failure": { + environment: valid, + lookup: func(string) (*user.User, error) { + return nil, errors.New("NSS failed") + }, + wantErr: "look up sudo user", + }, + "username_mismatch": { + environment: valid, + lookup: func(string) (*user.User, error) { + return &user.User{Uid: "1000", Gid: "1001", Username: "other", HomeDir: "/home/other"}, nil + }, + wantErr: "does not match SUDO_USER", + }, + "group_mismatch": { + environment: valid, + lookup: func(string) (*user.User, error) { + return &user.User{Uid: "1000", Gid: "1002", Username: "runner", HomeDir: "/home/runner"}, nil + }, + wantErr: "does not match SUDO_GID", + }, + "unsafe_home": { + environment: valid, + lookup: func(string) (*user.User, error) { + return &user.User{Uid: "1000", Gid: "1001", Username: "runner", HomeDir: "/"}, nil + }, + wantErr: "unsafe home directory", + }, + } + for name, tc := range tests { + t.Run(name, func(t *testing.T) { + deps := defaultUserDirDeps{ + effectiveUID: func() int { return 0 }, + lookupEnv: func(name string) (string, bool) { + value, ok := tc.environment[name] + return value, ok + }, + lookupUserID: tc.lookup, + } + identity, sudo, err := resolveSudoUserIdentity(deps) + if tc.wantErr != "" { + require.Error(t, err) + assert.ErrorContains(t, err, tc.wantErr) + assert.False(t, sudo) + return + } + require.NoError(t, err) + assert.Equal(t, tc.wantSudo, sudo) + assert.Equal(t, uint32(1000), identity.uid) + assert.Equal(t, "/home/runner", identity.home) + }) + } +} + +func TestResolveSudoUserIdentityIgnoresEnvironmentWhenUnprivileged(t *testing.T) { + t.Parallel() + + deps := defaultUserDirDeps{ + effectiveUID: func() int { return 1000 }, + lookupEnv: func(string) (string, bool) { + return "hostile", true + }, + lookupUserID: func(string) (*user.User, error) { + t.Fatal("unprivileged resolution must not consume sudo identity") + return nil, nil + }, + } + _, sudo, err := resolveSudoUserIdentity(deps) + require.NoError(t, err) + assert.False(t, sudo) +} + +func testDefaultUserDirDeps(home string, uid, gid uint32) defaultUserDirDeps { + deps := realDefaultUserDirDeps() + deps.effectiveUID = func() int { return 0 } + deps.lookupEnv = func(name string) (string, bool) { + values := map[string]string{ + "SUDO_UID": strconv.FormatUint(uint64(uid), 10), + "SUDO_GID": strconv.FormatUint(uint64(gid), 10), + "SUDO_USER": "runner", + } + value, ok := values[name] + return value, ok + } + deps.lookupUserID = func(id string) (*user.User, error) { + if id != strconv.FormatUint(uint64(uid), 10) { + return nil, errors.New("unexpected UID lookup") + } + return &user.User{ + Uid: strconv.FormatUint(uint64(uid), 10), + Gid: strconv.FormatUint(uint64(gid), 10), + Username: "runner", + HomeDir: home, + }, nil + } + return deps +} + +func assertDirectoryOwner(t *testing.T, path string, uid, gid uint32) { + t.Helper() + + var stat unix.Stat_t + require.NoError(t, unix.Lstat(path, &stat)) + assert.Equal(t, uid, stat.Uid) + assert.Equal(t, gid, stat.Gid) + assert.Equal(t, uint32(0700), stat.Mode&07777) +} + +func descriptorPath(t *testing.T, fd int) string { + t.Helper() + + path, err := os.Readlink("/proc/self/fd/" + strconv.Itoa(fd)) + require.NoError(t, err) + return path +} From d501e9248a9fe006a0e639fcf8d6c1c53b289e5a Mon Sep 17 00:00:00 2001 From: AlanD20 Date: Sun, 23 Aug 2026 14:39:18 +0200 Subject: [PATCH 0013/1457] test: qualify installed system binary --- scripts/run-system-tests.py | 727 +++++++++++++++---- scripts/tests/test_run_system_tests.py | 882 +++++++++++++++++++++++ tests/system/COVERAGE_MATRIX.md | 37 +- tests/system/conftest.py | 7 +- tests/system/host/test_system_install.py | 506 +++++++++++++ tests/system/pytest.ini | 1 + 6 files changed, 2013 insertions(+), 147 deletions(-) create mode 100644 scripts/tests/test_run_system_tests.py create mode 100644 tests/system/host/test_system_install.py diff --git a/scripts/run-system-tests.py b/scripts/run-system-tests.py index b9bdd27c..3c72ee8f 100755 --- a/scripts/run-system-tests.py +++ b/scripts/run-system-tests.py @@ -32,12 +32,15 @@ from __future__ import annotations import argparse -import getpass import json import os +import re import shlex +import shutil +import stat import subprocess import sys +import tempfile import time import uuid from concurrent.futures import ThreadPoolExecutor @@ -51,7 +54,14 @@ ASSET_MIRROR_HOST = os.path.expanduser("~/.cache/mvm-asset-mirror") if os.environ.get("MVM_ASSET_MIRROR"): ASSET_MIRROR_HOST = os.environ["MVM_ASSET_MIRROR"] -MVM_BINARY = os.environ.get("MVM_BINARY", os.path.expanduser("~/.local/bin/mvm")) +MVM_BINARY = os.environ.get("MVM_BINARY", "/usr/local/bin/mvm") +_REPO_ROOT = Path(__file__).resolve().parent.parent +MVM_CANDIDATE_CONFIGURED = Path( + os.environ.get("MVM_CANDIDATE_BINARY", _REPO_ROOT / "dist" / "mvm") +).expanduser() +MVM_CANDIDATE_BINARY = str( + MVM_CANDIDATE_CONFIGURED.resolve() +) SHARED_VOLUME_NAME = "asset-mirror" SHARED_VOLUME_SIZE = "6G" TEST_NETWORK_NAME = "sys-test-net" @@ -59,6 +69,13 @@ # Custom base image built during --prepare BASE_IMAGE_NAME = "mvm-test-runner" +# Root-owned staging location used to exercise the real administrator install +# flow inside every disposable runner VM. Tests execute the installed binary +# at /usr/local/bin/mvm; they never treat this candidate path as the CLI. +RUNNER_SYSTEM_CANDIDATE_DIR = "/opt/mvmctl-test" +RUNNER_SYSTEM_CANDIDATE = f"{RUNNER_SYSTEM_CANDIDATE_DIR}/mvm-candidate" +RUNNER_SYSTEM_UPLOAD = "/home/runner/.mvm-system-candidate.upload" + # Base VM used to build the custom image BASE_VM_NAME = "base-img-builder" BASE_VM_DISK = "12G" @@ -66,8 +83,25 @@ # SSH key for provisioning runner user in builder & test VMs BUILDER_KEY_NAME = "builder-key" +_SEMVER_CORE_NUMBER = r"(?:0|[1-9][0-9]*)" +_SEMVER_PRERELEASE_ID = ( + r"(?:0|[1-9][0-9]*|[0-9A-Za-z-]*[A-Za-z-][0-9A-Za-z-]*)" +) +_SEMVER_BUILD_ID = r"[0-9A-Za-z-]+" +_CANDIDATE_VERSION_PATTERN = re.compile( + rf"^v?(?P{_SEMVER_CORE_NUMBER}\." + rf"{_SEMVER_CORE_NUMBER}\.{_SEMVER_CORE_NUMBER}" + rf"(?:-{_SEMVER_PRERELEASE_ID}(?:\.{_SEMVER_PRERELEASE_ID})*)?" + rf"(?:\+{_SEMVER_BUILD_ID}(?:\.{_SEMVER_BUILD_ID})*)?)$" +) + + +class _UnsafeVolumeBackingPath(RuntimeError): + """The volume record points outside the one runner-managed backing path.""" + # Tier 1: shared volume, host-level CLI operations (no nested virt needed) TIER1_DOMAINS: dict[str, list[str]] = { + "system_install": ["tests/system/host/test_system_install.py"], "cli": ["tests/system/cli/test_cli.py"], "config": ["tests/system/config/test_config.py"], "init": ["tests/system/init/test_init.py"], @@ -191,10 +225,14 @@ def mvm( if result.stderr: print(result.stderr, end="", flush=True, file=sys.stderr) if check and result.returncode != 0: + if result.stderr is None: + stderr = "(streamed; not captured)" if not capture else "(not available)" + else: + stderr = result.stderr.strip() or "(empty)" raise RuntimeError( - f"mvm command failed: {' '.join(args)}\n" + f"mvm command failed: {shlex.join(cmd)}\n" f" rc: {result.returncode}\n" - f" stderr: {result.stderr.strip()}" + f" stderr: {stderr}" ) return result @@ -205,28 +243,124 @@ def log(msg: str) -> None: print(f"[{ts}] {msg}", flush=True) -def _build_mvm_binary() -> None: - """Build the mvm release binary and install it to ~/.local/bin/mvm.""" - log("Building mvm binary...") +def _parse_candidate_version(value: str) -> str: + """Parse one explicit release identity accepted by the build script.""" + match = _CANDIDATE_VERSION_PATTERN.fullmatch(value.strip()) + if match is None: + raise argparse.ArgumentTypeError( + "candidate version must be X.Y.Z with optional prerelease/build metadata" + ) + return match.group("version") + + +def _resolve_candidate_build_version(explicit_version: str | None) -> str: + """Resolve an explicit version or one clean, exact release tag at HEAD.""" + if explicit_version is not None: + try: + return _parse_candidate_version(explicit_version) + except argparse.ArgumentTypeError as exc: + raise RuntimeError(str(exc)) from exc + + tags = subprocess.run( + ["git", "tag", "--points-at", "HEAD", "--list", "v[0-9]*"], + cwd=str(_REPO_ROOT), + capture_output=True, + text=True, + timeout=10, + ) + if tags.returncode != 0: + raise RuntimeError( + "cannot inspect release tags; pass --candidate-version with --rebuild" + ) + exact_versions: list[str] = [] + for tag in tags.stdout.splitlines(): + try: + exact_versions.append(_parse_candidate_version(tag)) + except argparse.ArgumentTypeError: + continue + if len(exact_versions) != 1: + raise RuntimeError( + "--rebuild requires --candidate-version unless HEAD has exactly " + "one valid release tag" + ) + + status = subprocess.run( + ["git", "status", "--porcelain=v1", "--untracked-files=all"], + cwd=str(_REPO_ROOT), + capture_output=True, + text=True, + timeout=10, + ) + if status.returncode != 0 or status.stdout.strip(): + raise RuntimeError( + "--rebuild requires --candidate-version when the release-tagged " + "checkout has uncommitted files" + ) + return exact_versions[0] + + +def _build_mvm_binary(candidate_version: str) -> None: + """Build the release candidate without replacing the outer controller.""" + try: + version = _parse_candidate_version(candidate_version) + except argparse.ArgumentTypeError as exc: + raise RuntimeError(str(exc)) from exc + if MVM_CANDIDATE_CONFIGURED.is_symlink(): + raise RuntimeError( + "MVM_CANDIDATE_BINARY build target must not be a symlink: " + f"{MVM_CANDIDATE_CONFIGURED}" + ) + _require_distinct_candidate_controller() + log(f"Building mvm release candidate at {MVM_CANDIDATE_BINARY}...") script_dir = Path(__file__).resolve().parent repo_root = script_dir.parent build_script = repo_root / "scripts" / "build.sh" - output_path = Path("~/.local/bin/mvm").expanduser() + output_path = Path(MVM_CANDIDATE_BINARY) result = subprocess.run( [ str(build_script), "release", + "--version", + version, "--output", str(output_path), ], cwd=str(repo_root), ) if result.returncode != 0: - log("ERROR: mvm binary build failed") - sys.exit(1) + raise RuntimeError(f"mvm binary build failed with rc={result.returncode}") + reported_version = _get_mvm_version() + if reported_version != version: + raise RuntimeError( + "built candidate version mismatch: " + f"requested {version}, reported {reported_version}" + ) log("Build complete") +def _controller_executable_path() -> Path: + """Resolve the executable used only to manage outer host resources.""" + command = shlex.split(MVM_BINARY) + if not command: + raise RuntimeError("MVM_BINARY must contain an executable path") + executable = shutil.which(command[0]) or command[0] + return Path(executable).expanduser().resolve() + + +def _require_distinct_candidate_controller() -> None: + """Reject path and inode aliases before a candidate build can overwrite mvm.""" + candidate = Path(MVM_CANDIDATE_BINARY).expanduser().resolve() + controller = _controller_executable_path() + aliases = candidate == controller + if not aliases and candidate.exists() and controller.exists(): + aliases = os.path.samefile(candidate, controller) + if aliases: + raise RuntimeError( + "MVM_CANDIDATE_BINARY must not alias the outer controller " + f"MVM_BINARY ({controller})" + ) + + def _run_pytest( test_files: list[str], *, @@ -259,23 +393,195 @@ def _run_pytest( # ============================================================================ -def _get_volume_path(name: str) -> str | None: - """Get the on-disk path of a volume, or None if it doesn't exist.""" +def _inspect_volume_path(name: str) -> tuple[bool, str | None]: + """Return whether a volume record exists and its recorded backing path.""" result = mvm("volume", "inspect", name, "--json", check=False, timeout=30) if result.returncode != 0: - return None + return False, None try: info = json.loads(result.stdout) - return info.get("volume", {}).get("path") - except (json.JSONDecodeError, KeyError): - return None + except json.JSONDecodeError as exc: + raise RuntimeError( + f"volume inspect returned invalid JSON for '{name}': {exc}" + ) from exc + path = info.get("volume", {}).get("path") + return True, path if isinstance(path, str) and path else None + + +def _get_volume_path(name: str) -> str | None: + """Get the recorded on-disk path of a volume, or None if absent.""" + _exists, path = _inspect_volume_path(name) + return path + + +def _expected_shared_volume_path() -> Path: + """Return the lexical managed path without following any symlinks.""" + cache_root = Path( + os.environ.get("MVM_CACHE_DIR", "~/.cache/mvmctl") + ).expanduser() + return Path( + os.path.abspath(cache_root / "volumes" / f"{SHARED_VOLUME_NAME}.raw") + ) + + +def _validated_volume_backing_path(raw_path: str | os.PathLike[str]) -> Path: + """Require the exact regular, non-symlink runner volume backing file.""" + path = Path(os.path.abspath(Path(raw_path).expanduser())) + expected = _expected_shared_volume_path() + if path != expected: + raise _UnsafeVolumeBackingPath( + "unexpected shared volume backing path: " + f"expected {expected}, got {path}" + ) + try: + info = path.lstat() + except FileNotFoundError as exc: + raise RuntimeError(f"volume backing file is missing: {path}") from exc + except OSError as exc: + raise RuntimeError(f"cannot inspect volume backing file {path}: {exc}") from exc + if stat.S_ISLNK(info.st_mode): + raise RuntimeError(f"volume backing path must not be a symlink: {path}") + if not stat.S_ISREG(info.st_mode): + raise RuntimeError(f"volume backing path is not a regular file: {path}") + return path + + +def _open_directory_without_symlinks(path: Path) -> int: + """Open an absolute directory path one pinned, no-follow component at a time.""" + flags = os.O_RDONLY | os.O_DIRECTORY | os.O_CLOEXEC | os.O_NOFOLLOW + current_fd = os.open(os.sep, flags) + try: + for component in path.parts[1:]: + next_fd = os.open(component, flags, dir_fd=current_fd) + os.close(current_fd) + current_fd = next_fd + return current_fd + except BaseException: + os.close(current_fd) + raise + + +def _require_shared_volume_path_absent_before_create() -> None: + """Refuse create if old cleanup left any object at the canonical path.""" + expected = _expected_shared_volume_path() + try: + expected.lstat() + except FileNotFoundError: + return + except OSError as exc: + raise RuntimeError( + f"cannot verify shared volume path before create {expected}: {exc}" + ) from exc + raise _UnsafeVolumeBackingPath( + f"shared volume backing path still exists before volume create: {expected}" + ) + + +def _populate_volume_image(volume_path: Path, mirror_path: Path) -> None: + """Format and populate one ext4 image without sudo or loop mounts.""" + backing = _validated_volume_backing_path(volume_path) + mirror = mirror_path.expanduser().resolve() + if not mirror.is_dir(): + raise RuntimeError(f"asset mirror is not a directory: {mirror}") + + path_info = backing.lstat() + try: + parent_fd = _open_directory_without_symlinks(backing.parent) + except OSError as exc: + raise RuntimeError( + f"cannot safely open shared volume directory {backing.parent}: {exc}" + ) from exc + try: + try: + backing_fd = os.open( + backing.name, + os.O_RDWR | os.O_CLOEXEC | os.O_NOFOLLOW, + dir_fd=parent_fd, + ) + except OSError as exc: + raise RuntimeError( + f"cannot safely open shared volume backing file {backing}: {exc}" + ) from exc + finally: + os.close(parent_fd) + + try: + pinned_before = os.fstat(backing_fd) + if not stat.S_ISREG(pinned_before.st_mode): + raise RuntimeError( + f"shared volume backing descriptor is not a regular file: {backing}" + ) + if (pinned_before.st_dev, pinned_before.st_ino) != ( + path_info.st_dev, + path_info.st_ino, + ): + raise RuntimeError( + f"shared volume backing file changed before population: {backing}" + ) + try: + result = subprocess.run( + [ + "mkfs.ext4", + "-F", + "-d", + str(mirror), + f"/proc/self/fd/{backing_fd}", + ], + capture_output=True, + text=True, + timeout=900, + pass_fds=(backing_fd,), + ) + except FileNotFoundError as exc: + raise RuntimeError( + "mkfs.ext4 is required to populate the shared asset volume" + ) from exc + except subprocess.TimeoutExpired as exc: + raise RuntimeError( + "mkfs.ext4 timed out while populating the volume" + ) from exc + if result.returncode != 0: + detail = result.stderr.strip() or result.stdout.strip() or "unknown error" + raise RuntimeError(f"mkfs.ext4 failed to populate {backing}: {detail}") + + pinned_after = os.fstat(backing_fd) + try: + current = backing.lstat() + except OSError as exc: + raise RuntimeError( + f"shared volume backing path was replaced during population: {backing}" + ) from exc + if not stat.S_ISREG(current.st_mode) or ( + current.st_dev, + current.st_ino, + ) != (pinned_after.st_dev, pinned_after.st_ino): + raise RuntimeError( + f"shared volume backing path was replaced during population: {backing}" + ) + if (pinned_after.st_uid, pinned_after.st_gid) != ( + pinned_before.st_uid, + pinned_before.st_gid, + ): + raise RuntimeError( + "mkfs.ext4 changed shared volume backing ownership: " + f"expected {pinned_before.st_uid}:{pinned_before.st_gid}, " + f"got {pinned_after.st_uid}:{pinned_after.st_gid}" + ) + finally: + os.close(backing_fd) def _create_and_seed_volume() -> None: """Create the shared asset volume and seed it with asset mirror contents. - This requires sudo for loop mount. Prompts the user for a password. + ``mkfs.ext4 -d`` copies the mirror directly into the image, so this never + needs sudo, a loop device, or a host mount. """ + mirror_path = Path(ASSET_MIRROR_HOST).expanduser().resolve() + if not mirror_path.is_dir(): + raise RuntimeError(f"asset mirror is not a directory: {mirror_path}") + + _require_shared_volume_path_absent_before_create() log(f"Creating shared volume '{SHARED_VOLUME_NAME}' ({SHARED_VOLUME_SIZE})...") mvm( "volume", @@ -290,65 +596,37 @@ def _create_and_seed_volume() -> None: timeout=30, ) - vol_path = _get_volume_path(SHARED_VOLUME_NAME) - if not vol_path: - log("ERROR: Failed to find volume path after creation") - sys.exit(1) - - # Check if asset mirror has content - mirror_path = Path(ASSET_MIRROR_HOST).expanduser() - if not mirror_path.exists() or not any(mirror_path.iterdir()): - log(f"WARNING: Asset mirror at {mirror_path} is empty or missing.") - log( - "Populating volume with empty filesystem. Tests that need assets will fail." - ) - - # Need sudo for loop mount - log("Populating volume with asset mirror contents (requires sudo)...") - password = getpass.getpass("sudo password: ") - - def sudo_run(cmd_args: list[str], input_data: str | None = None) -> None: - full_cmd = ["sudo", "-S"] + cmd_args - subprocess.run( - full_cmd, - input=input_data or "", - text=True, - capture_output=True, - check=True, - timeout=120, - ) - - mount_point = "/mnt/.mvm-asset-populate" try: - sudo_run(["mkfs.ext4", "-F", vol_path], password) - Path(mount_point).mkdir(parents=True, exist_ok=True) - sudo_run(["mount", "-o", "loop", vol_path, mount_point], password) - sudo_run( - ["cp", "-r"] - + [str(p) for p in mirror_path.glob("*")] - + [f"{mount_point}/"], - password, - ) - sudo_run(["chmod", "-R", "a+rX", mount_point], password) - log(f"Seeded volume with contents from {mirror_path}") - except subprocess.CalledProcessError as e: - log(f"ERROR: Failed to populate volume: {e.stderr if e.stderr else e}") - # Clean up mount if it succeeded - subprocess.run( - ["sudo", "-S", "umount", mount_point], - input=password, - text=True, - capture_output=True, + record_exists, raw_path = _inspect_volume_path(SHARED_VOLUME_NAME) + if not record_exists or not raw_path: + raise RuntimeError("created shared volume has no recorded backing path") + vol_path = _validated_volume_backing_path(raw_path) + + if not any(mirror_path.iterdir()): + log(f"WARNING: Asset mirror at {mirror_path} is empty.") + log( + "Populating volume with empty filesystem. " + "Tests that need assets will fail." + ) + _populate_volume_image(vol_path, mirror_path) + except _UnsafeVolumeBackingPath: + raise + except RuntimeError as primary: + cleanup = mvm( + "volume", + "rm", + SHARED_VOLUME_NAME, + "--force", check=False, timeout=30, ) - sys.exit(1) - finally: - sudo_run(["umount", mount_point], password) - try: - Path(mount_point).rmdir() - except OSError: - pass + if cleanup.returncode != 0: + raise RuntimeError( + f"{primary}; failed to remove incomplete shared volume: " + f"{cleanup.stderr.strip()}" + ) from primary + raise + log(f"Seeded volume with contents from {mirror_path}") def ensure_shared_volume(*, rebuild: bool = False) -> None: @@ -364,14 +642,28 @@ def ensure_shared_volume(*, rebuild: bool = False) -> None: "Create it first by pulling images/kernels/binaries with MVM_ASSET_MIRROR set." ) - existing_path = _get_volume_path(SHARED_VOLUME_NAME) - if existing_path and not rebuild: + record_exists, raw_path = _inspect_volume_path(SHARED_VOLUME_NAME) + existing_path: Path | None = None + if record_exists: + try: + if raw_path is None: + raise RuntimeError("volume record has no backing path") + existing_path = _validated_volume_backing_path(raw_path) + except _UnsafeVolumeBackingPath: + raise + except RuntimeError as exc: + log( + f"Shared volume '{SHARED_VOLUME_NAME}' is stale ({exc}); " + "removing its record before recreation" + ) + + if existing_path is not None and not rebuild: log(f"Shared volume '{SHARED_VOLUME_NAME}' already exists at {existing_path}") return - if existing_path and rebuild: + if record_exists: log(f"Rebuilding: removing existing volume '{SHARED_VOLUME_NAME}'...") - mvm("volume", "rm", SHARED_VOLUME_NAME, "--force", check=False, timeout=30) + mvm("volume", "rm", SHARED_VOLUME_NAME, "--force", timeout=30) _create_and_seed_volume() @@ -402,22 +694,100 @@ def ensure_test_network() -> None: def _get_mvm_version() -> str: - """Get the mvm binary version string (e.g. '0.1.0' or '0.1.0-7-gdeadbeef'). - - Strips the ``-dirty`` suffix if present — the base image was built from a - clean tree and uses the version without ``-dirty``. - """ - result = mvm("--version", timeout=10) + """Get the candidate version without consulting or touching host state.""" + with tempfile.TemporaryDirectory(prefix="mvm-candidate-version-") as root: + isolation_root = Path(root) + cache_dir = isolation_root / "cache" + config_dir = isolation_root / "config" + temp_dir = isolation_root / "tmp" + cache_dir.mkdir() + config_dir.mkdir() + temp_dir.mkdir() + env = { + **os.environ, + "NO_COLOR": "1", + "MVM_CACHE_DIR": str(cache_dir), + "MVM_CONFIG_DIR": str(config_dir), + "MVM_TEMP_DIR": str(temp_dir), + } + result = subprocess.run( + [MVM_CANDIDATE_BINARY, "--version"], + capture_output=True, + text=True, + timeout=10, + env=env, + ) + if result.returncode != 0: + raise RuntimeError( + "candidate version probe failed: " + f"{result.stderr.strip() or f'rc={result.returncode}'}" + ) # "mvm 0.1.0" → "0.1.0" parts = result.stdout.strip().split() - version = parts[-1] if len(parts) >= 2 else "latest" - return version.removesuffix("-dirty") + if len(parts) < 2 or parts[0] != "mvm" or not parts[-1]: + raise RuntimeError( + f"candidate returned an invalid version response: {result.stdout!r}" + ) + return parts[-1] def _unique_name(prefix: str = "sys-runner") -> str: return f"{prefix}-{uuid.uuid4().hex[:8]}" +def _install_system_binary_in_runner(vm_name: str) -> None: + """Stage the RC and install it through ``host install-system``. + + Copying directly onto /usr/local/bin/mvm would bypass the administrator + bootstrap path that the system-install L2 domain exists to qualify. + """ + mvm( + "cp", + "-f", + MVM_CANDIDATE_BINARY, + f"{vm_name}:{RUNNER_SYSTEM_UPLOAD}", + timeout=60, + ) + mvm( + "exec", + vm_name, + "--user", + "runner", + "--timeout", + "10", + "--", + f"test -f {RUNNER_SYSTEM_UPLOAD} && " + f"test ! -L {RUNNER_SYSTEM_UPLOAD} && " + f"sudo test ! -L {RUNNER_SYSTEM_CANDIDATE_DIR} && " + f"sudo install -d -o root -g root -m 0755 {RUNNER_SYSTEM_CANDIDATE_DIR} && " + f"sudo test ! -L {RUNNER_SYSTEM_CANDIDATE_DIR} && " + f"sudo test ! -L {RUNNER_SYSTEM_CANDIDATE} && " + f"sudo install -o root -g root -m 0755 " + f"{RUNNER_SYSTEM_UPLOAD} {RUNNER_SYSTEM_CANDIDATE} && " + f"rm -f {RUNNER_SYSTEM_UPLOAD} && " + f"test ! -L {RUNNER_SYSTEM_CANDIDATE} && " + f"test \"$(stat -c '%u:%g:%a' {RUNNER_SYSTEM_CANDIDATE})\" = '0:0:755' && " + f"sudo {RUNNER_SYSTEM_CANDIDATE} host install-system && " + "test \"$(stat -c '%u:%g:%a' /usr/local/bin/mvm)\" = '0:0:755'", + timeout=90, + ) + + +def _initialize_system_binary_in_runner(vm_name: str) -> None: + """Run host initialization through the canonical installed executable.""" + mvm( + "exec", + vm_name, + "--user", + "runner", + "--timeout", + "10", + "--", + "sudo /usr/local/bin/mvm host init", + timeout=180, + ) + + def _ensure_builder_key() -> None: """Create the builder SSH key if it doesn't exist yet.""" check = mvm("key", "inspect", BUILDER_KEY_NAME, check=False, timeout=10) @@ -465,6 +835,8 @@ def provision_t1(vm_name: str, mvm_version: str) -> None: SHARED_VOLUME_NAME, timeout=180, ) + _install_system_binary_in_runner(vm_name) + _initialize_system_binary_in_runner(vm_name) # CRITICAL: mvm init MUST run as the unprivileged user (runner), NOT via sudo. # Running as root creates the cache dir with root ownership — test VMs inherit # this state and break with 'permission denied' on /home/runner/.cache/mvmctl. @@ -477,7 +849,7 @@ def provision_t1(vm_name: str, mvm_version: str) -> None: "10", "--", "sudo mkdir -p /mnt && sudo mount /dev/vdb /mnt && " - "MVM_ASSET_MIRROR=/mnt mvm init --non-interactive", + "MVM_ASSET_MIRROR=/mnt /usr/local/bin/mvm init --non-interactive", timeout=360, ) @@ -556,6 +928,8 @@ def provision_t2(vm_name: str, mvm_version: str) -> None: SHARED_VOLUME_NAME, timeout=300, ) + _install_system_binary_in_runner(vm_name) + _initialize_system_binary_in_runner(vm_name) # Inside VM: mount shared volume, init, register assets # CRITICAL: mvm init MUST run as the unprivileged user (runner), NOT via sudo. @@ -571,7 +945,7 @@ def provision_t2(vm_name: str, mvm_version: str) -> None: "10", "--", "sudo mkdir -p /mnt && sudo mount /dev/vdb /mnt && " - "MVM_ASSET_MIRROR=/mnt mvm init --non-interactive", + "MVM_ASSET_MIRROR=/mnt /usr/local/bin/mvm init --non-interactive", timeout=180, ) @@ -584,7 +958,7 @@ def provision_t2(vm_name: str, mvm_version: str) -> None: "--timeout", "10", "--", - "MVM_ASSET_MIRROR=/mnt mvm kernel pull --type firecracker " + "MVM_ASSET_MIRROR=/mnt /usr/local/bin/mvm kernel pull --type firecracker " "--version v1.15 --default", timeout=150, ) @@ -596,7 +970,7 @@ def provision_t2(vm_name: str, mvm_version: str) -> None: "--timeout", "10", "--", - "MVM_ASSET_MIRROR=/mnt mvm image pull alpine:3.23", + "MVM_ASSET_MIRROR=/mnt /usr/local/bin/mvm image pull alpine:3.23", timeout=150, ) mvm( @@ -607,7 +981,7 @@ def provision_t2(vm_name: str, mvm_version: str) -> None: "--timeout", "10", "--", - "MVM_ASSET_MIRROR=/mnt mvm image pull ubuntu:noble", + "MVM_ASSET_MIRROR=/mnt /usr/local/bin/mvm image pull ubuntu:noble", timeout=360, ) mvm( @@ -618,7 +992,7 @@ def provision_t2(vm_name: str, mvm_version: str) -> None: "--timeout", "10", "--", - "MVM_ASSET_MIRROR=/mnt mvm bin pull firecracker " + "MVM_ASSET_MIRROR=/mnt /usr/local/bin/mvm bin pull firecracker " "--version 1.16.0 --default --force", timeout=150, ) @@ -629,6 +1003,69 @@ def destroy_vm(vm_name: str) -> None: mvm("vm", "rm", vm_name, "--force", timeout=60, check=False) +def _pull_builder_assets(vm_name: str) -> None: + """Pull base-image assets serially so each streamed exec stays bounded.""" + pulls = ( + ( + "Firecracker binary", + "MVM_ASSET_MIRROR=/mnt /usr/local/bin/mvm binary pull " + "firecracker --default --force --version 1.16.0", + ), + ( + "Firecracker kernel", + "MVM_ASSET_MIRROR=/mnt /usr/local/bin/mvm kernel pull " + "--type firecracker --version v1.15 --default", + ), + ( + "Alpine image", + "MVM_ASSET_MIRROR=/mnt /usr/local/bin/mvm image pull " + "alpine:3.23", + ), + ( + "Ubuntu Minimal image", + "MVM_ASSET_MIRROR=/mnt /usr/local/bin/mvm image pull " + "ubuntu-minimal:24.04", + ), + ( + "Ubuntu image", + "MVM_ASSET_MIRROR=/mnt /usr/local/bin/mvm image pull ubuntu " + "--version 24.04", + ), + ) + for label, command in pulls: + log(f" Pulling {label} into '{vm_name}'...") + mvm( + "exec", + vm_name, + "--user", + "runner", + "--timeout", + "10", + "--", + command, + timeout=600, + capture=False, + ) + + log(f" Verifying cached image integrity in '{vm_name}'...") + mvm( + "exec", + vm_name, + "--user", + "runner", + "--timeout", + "10", + "--", + "echo 'Verifying cached image integrity...' && " + "for f in /home/runner/.cache/mvmctl/images/*.zst; do " + 'zstd -t "$f" || exit 1; ' + "done && " + "echo 'All images verified OK'", + timeout=180, + capture=False, + ) + + def _build_base_image(mvm_version: str, *, rebuild: bool = False) -> str: """Build a custom base image with all test dependencies pre-installed. @@ -692,8 +1129,8 @@ def _build_base_image(mvm_version: str, *, rebuild: bool = False) -> str: timeout=180, ) try: - log(f" Copying mvm binary and system tests into '{BASE_VM_NAME}'...") - mvm("cp", MVM_BINARY, f"{BASE_VM_NAME}:/usr/local/bin/mvm", timeout=60) + log(f" Installing mvm system binary and tests into '{BASE_VM_NAME}'...") + _install_system_binary_in_runner(BASE_VM_NAME) # Verify the binary transferred correctly verify = mvm( "exec", @@ -703,14 +1140,17 @@ def _build_base_image(mvm_version: str, *, rebuild: bool = False) -> str: "--timeout", "10", "--", + f"stat -c%s {RUNNER_SYSTEM_CANDIDATE} && " "stat -c%s /usr/local/bin/mvm", timeout=20, ) - expected = Path(shlex.split(MVM_BINARY)[0]).expanduser().stat().st_size - actual = int(verify.stdout.strip()) - if actual != expected: + expected = Path(MVM_CANDIDATE_BINARY).stat().st_size + installed_sizes = [int(value) for value in verify.stdout.splitlines()] + if installed_sizes != [expected, expected]: raise RuntimeError( - f"mvm binary size mismatch in builder VM: expected {expected}, got {actual}" + "mvm binary size mismatch in builder VM: " + f"expected candidate and system sizes {[expected, expected]}, " + f"got {installed_sizes}" ) mvm("cp", "tests/system", f"{BASE_VM_NAME}:/tests/", timeout=60) @@ -758,6 +1198,8 @@ def _build_base_image(mvm_version: str, *, rebuild: bool = False) -> str: "sudo usermod -aG kvm runner", timeout=30, ) + log(f" Initializing host through the system binary in '{BASE_VM_NAME}'...") + _initialize_system_binary_in_runner(BASE_VM_NAME) log(" Changing ownership of /tests to runner user...") mvm( "exec", @@ -787,39 +1229,16 @@ def _build_base_image(mvm_version: str, *, rebuild: bool = False) -> str: "--", "sudo mkdir -p /mnt && " "sudo mount /dev/vdb /mnt && " - "MVM_ASSET_MIRROR=/mnt mvm init --non-interactive", + "MVM_ASSET_MIRROR=/mnt /usr/local/bin/mvm init --non-interactive", timeout=180, capture=False, ) - log(f" Pulling assets into '{BASE_VM_NAME}' in parallel from mirror...") # Asset pulls are pre-baked into the base image so each test VM doesn't - # re-pull them (saves ~3 min per T1 VM). Run them in parallel inside one - # exec; wait on each PID so any failure propagates to the exec exit code. - mvm( - "exec", - BASE_VM_NAME, - "--user", - "runner", - "--timeout", - "10", - "--", - "set -e && " - 'pids="" && ' - 'MVM_ASSET_MIRROR=/mnt mvm binary pull firecracker --default --force --version 1.16.0 & pids="$pids $!" && ' - 'MVM_ASSET_MIRROR=/mnt mvm kernel pull --type firecracker --version v1.15 --default & pids="$pids $!" && ' - 'MVM_ASSET_MIRROR=/mnt mvm image pull alpine:3.23 & pids="$pids $!" && ' - 'MVM_ASSET_MIRROR=/mnt mvm image pull ubuntu-minimal:24.04 & pids="$pids $!" && ' - 'MVM_ASSET_MIRROR=/mnt mvm image pull ubuntu --version 24.04 & pids="$pids $!" && ' - 'for pid in $pids; do wait "$pid" || exit $?; done && ' - "echo 'Verifying cached image integrity...' && " - "for f in /home/runner/.cache/mvmctl/images/*.zst; do " - 'zstd -t "$f" || exit 1; ' - "done && " - "echo 'All images verified OK'", - timeout=360, - capture=False, - ) + # re-pull them (saves ~3 min per T1 VM). Keep each pull in its own exec: + # concurrent downloads/decompression can exhaust this 3 GiB builder, + # while separate streamed sessions preserve the exact failing command. + _pull_builder_assets(BASE_VM_NAME) log(f" Stopping '{BASE_VM_NAME}'...") mvm("vm", "stop", BASE_VM_NAME, timeout=60) @@ -1111,9 +1530,16 @@ def _build_parser() -> argparse.ArgumentParser: parser.add_argument( "--rebuild", action="store_true", - help="Build the mvm binary, then rebuild the shared volume and custom " - "base image and run the provisioning smoke-test. After prepare the " - "script stops unless a test-running flag is also given.", + help="Build MVM_CANDIDATE_BINARY (never MVM_BINARY), then rebuild the " + "shared volume and custom base image and run the provisioning " + "smoke-test. After prepare the script stops unless a test-running " + "flag is also given.", + ) + parser.add_argument( + "--candidate-version", + type=_parse_candidate_version, + help="Explicit X.Y.Z identity passed to build.sh --version. Required " + "with --rebuild unless HEAD is one clean, exact release tag.", ) parser.add_argument( "--volume", @@ -1123,9 +1549,9 @@ def _build_parser() -> argparse.ArgumentParser: parser.add_argument( "--image", action="store_true", - help="Rebuild the mvm binary, then rebuild only the custom base image " - "(mvm-test-runner:). Ensures the shared asset volume exists first " - "because the builder VM attaches it.", + help="Rebuild only the custom base image from MVM_CANDIDATE_BINARY. " + "Ensures the shared asset volume exists first because the builder VM " + "attaches it.", ) parser.add_argument( "--skip-volume-check", @@ -1224,7 +1650,7 @@ def run_prepare(*, rebuild_volume: bool = False, rebuild_image: bool = False) -> "10", "--", "sudo mkdir -p /mnt && sudo mount /dev/vdb /mnt && " - "MVM_ASSET_MIRROR=/mnt mvm init --non-interactive", + "MVM_ASSET_MIRROR=/mnt /usr/local/bin/mvm init --non-interactive", timeout=90, ) finally: @@ -1269,7 +1695,7 @@ def run_prepare(*, rebuild_volume: bool = False, rebuild_image: bool = False) -> "10", "--", "sudo mkdir -p /mnt && sudo mount /dev/vdb /mnt && " - "MVM_ASSET_MIRROR=/mnt mvm init --non-interactive", + "MVM_ASSET_MIRROR=/mnt /usr/local/bin/mvm init --non-interactive", timeout=180, ) @@ -1282,7 +1708,8 @@ def run_prepare(*, rebuild_volume: bool = False, rebuild_image: bool = False) -> "--timeout", "10", "--", - "MVM_ASSET_MIRROR=/mnt mvm image pull alpine:3.23 2>&1 | head -5", + "MVM_ASSET_MIRROR=/mnt /usr/local/bin/mvm image pull " + "alpine:3.23 2>&1 | head -5", timeout=150, check=False, ) @@ -1312,6 +1739,9 @@ def main() -> None: parser = _build_parser() args = parser.parse_args() + if args.candidate_version is not None and not args.rebuild: + parser.error("--candidate-version requires --rebuild") + test_flags = args.all or args.domains or args.tier prep_flags = args.prepare or args.image or args.volume or args.rebuild @@ -1319,9 +1749,16 @@ def main() -> None: parser.print_help() sys.exit(0) - # Handle --rebuild: build binary before anything else + # Handle --rebuild: build the distinct release candidate before anything else. if args.rebuild: - _build_mvm_binary() + try: + candidate_version = _resolve_candidate_build_version( + args.candidate_version + ) + _build_mvm_binary(candidate_version) + except RuntimeError as exc: + log(f"ERROR: {exc}") + sys.exit(1) # Validate MVM_BINARY exists binary = shlex.split(MVM_BINARY)[0] @@ -1330,6 +1767,25 @@ def main() -> None: log("Set MVM_BINARY or ensure 'mvm' is in PATH.") sys.exit(1) + configured_candidate = MVM_CANDIDATE_CONFIGURED + candidate = Path(MVM_CANDIDATE_BINARY) + if ( + configured_candidate.is_symlink() + or not candidate.is_file() + or not os.access(candidate, os.X_OK) + ): + log( + "ERROR: MVM_CANDIDATE_BINARY must be a regular, non-symlink, " + f"executable file: {configured_candidate}" + ) + sys.exit(1) + + try: + _require_distinct_candidate_controller() + except RuntimeError as exc: + log(f"ERROR: {exc}") + sys.exit(1) + # Handle prep actions. # --rebuild implies --volume + --image + --prepare (in that order). if args.rebuild: @@ -1342,7 +1798,6 @@ def main() -> None: if args.prepare: run_prepare(rebuild_volume=rebuild_volume, rebuild_image=rebuild_image) elif args.image: - _build_mvm_binary() # The builder VM attaches the shared volume, so ensure it exists first. ensure_shared_volume(rebuild=rebuild_volume) ensure_test_network() diff --git a/scripts/tests/test_run_system_tests.py b/scripts/tests/test_run_system_tests.py new file mode 100644 index 00000000..76bfc621 --- /dev/null +++ b/scripts/tests/test_run_system_tests.py @@ -0,0 +1,882 @@ +from __future__ import annotations + +import argparse +import ast +import importlib.util +import os +import shutil +import subprocess +import sys +from pathlib import Path +from types import ModuleType + +import pytest + + +@pytest.fixture +def orchestrator() -> ModuleType: + script = Path(__file__).parents[1] / "run-system-tests.py" + module_name = "mvmctl_run_system_tests" + spec = importlib.util.spec_from_file_location(module_name, script) + assert spec is not None and spec.loader is not None + module = importlib.util.module_from_spec(spec) + sys.modules[module_name] = module + spec.loader.exec_module(module) + return module + + +def test_default_outer_controller_is_the_canonical_system_binary( + monkeypatch: pytest.MonkeyPatch, +) -> None: + monkeypatch.delenv("MVM_BINARY", raising=False) + script = Path(__file__).parents[1] / "run-system-tests.py" + module_name = "mvmctl_run_system_tests_default_controller" + spec = importlib.util.spec_from_file_location(module_name, script) + assert spec is not None and spec.loader is not None + module = importlib.util.module_from_spec(spec) + monkeypatch.setitem(sys.modules, module_name, module) + + spec.loader.exec_module(module) + + assert module.MVM_BINARY == "/usr/local/bin/mvm" + + +def test_guest_product_commands_use_the_exact_installed_cli( + orchestrator: ModuleType, +) -> None: + tree = ast.parse(Path(orchestrator.__file__).read_text(encoding="utf-8")) + unqualified = sorted( + { + node.value + for node in ast.walk(tree) + if isinstance(node, ast.Constant) + and isinstance(node.value, str) + and "MVM_ASSET_MIRROR=/mnt mvm " in node.value + } + ) + + assert unqualified == [] + + +def test_streamed_mvm_failure_preserves_command_and_rc_when_output_is_none( + orchestrator: ModuleType, + monkeypatch: pytest.MonkeyPatch, +) -> None: + monkeypatch.setattr(orchestrator, "MVM_BINARY", "/outer/controller-mvm") + + def failed_stream( + command: list[str], **kwargs: object + ) -> subprocess.CompletedProcess[None]: + assert kwargs["capture_output"] is False + return subprocess.CompletedProcess(command, 23, stdout=None, stderr=None) + + monkeypatch.setattr(orchestrator.subprocess, "run", failed_stream) + + with pytest.raises(RuntimeError) as captured: + orchestrator.mvm( + "image", + "pull", + "ubuntu-minimal:24.04", + capture=False, + ) + + message = str(captured.value) + assert ( + "mvm command failed: /outer/controller-mvm image pull " + "ubuntu-minimal:24.04" + ) in message + assert "rc: 23" in message + assert "stderr: (streamed; not captured)" in message + + +def test_builder_assets_are_pulled_in_separate_sequential_execs( + orchestrator: ModuleType, + monkeypatch: pytest.MonkeyPatch, +) -> None: + calls: list[tuple[tuple[str, ...], dict[str, object]]] = [] + + def fake_mvm(*args: str, **kwargs: object) -> subprocess.CompletedProcess[str]: + calls.append((args, kwargs)) + return subprocess.CompletedProcess(args, 0, stdout="", stderr="") + + monkeypatch.setattr(orchestrator, "mvm", fake_mvm) + + orchestrator._pull_builder_assets("base-img-builder") + + expected_pull_commands = [ + "MVM_ASSET_MIRROR=/mnt /usr/local/bin/mvm binary pull firecracker " + "--default --force --version 1.16.0", + "MVM_ASSET_MIRROR=/mnt /usr/local/bin/mvm kernel pull " + "--type firecracker --version v1.15 --default", + "MVM_ASSET_MIRROR=/mnt /usr/local/bin/mvm image pull alpine:3.23", + "MVM_ASSET_MIRROR=/mnt /usr/local/bin/mvm image pull " + "ubuntu-minimal:24.04", + "MVM_ASSET_MIRROR=/mnt /usr/local/bin/mvm image pull ubuntu " + "--version 24.04", + ] + pull_calls = calls[:-1] + assert [args[-1] for args, _kwargs in pull_calls] == expected_pull_commands + assert len(pull_calls) == 5 + for args, kwargs in pull_calls: + assert args[:-1] == ( + "exec", + "base-img-builder", + "--user", + "runner", + "--timeout", + "10", + "--", + ) + assert " & " not in args[-1] + assert "pids=" not in args[-1] + assert kwargs["capture"] is False + assert kwargs["timeout"] == 600 + + verify_args, verify_kwargs = calls[-1] + assert verify_args[:2] == ("exec", "base-img-builder") + assert "Verifying cached image integrity" in verify_args[-1] + assert verify_kwargs == {"timeout": 180, "capture": False} + + +def test_builder_asset_pull_preserves_first_streamed_failure( + orchestrator: ModuleType, + monkeypatch: pytest.MonkeyPatch, +) -> None: + commands: list[str] = [] + original_failure = RuntimeError( + "mvm command failed: streamed ubuntu-minimal pull; rc: 23; " + "stderr: (streamed; not captured)" + ) + + def fake_mvm(*args: str, **_kwargs: object) -> subprocess.CompletedProcess[str]: + commands.append(args[-1]) + if "ubuntu-minimal:24.04" in args[-1]: + raise original_failure + return subprocess.CompletedProcess(args, 0, stdout="", stderr="") + + monkeypatch.setattr(orchestrator, "mvm", fake_mvm) + + with pytest.raises(RuntimeError) as captured: + orchestrator._pull_builder_assets("base-img-builder") + + assert captured.value is original_failure + assert commands[-1].endswith("image pull ubuntu-minimal:24.04") + assert not any("--version 24.04" in command for command in commands) + assert not any("Verifying cached image integrity" in command for command in commands) + + +def test_build_base_image_cleans_builder_when_asset_pull_fails( + orchestrator: ModuleType, + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, +) -> None: + candidate = tmp_path / "mvm-candidate" + candidate.write_bytes(b"candidate") + monkeypatch.setattr(orchestrator, "MVM_CANDIDATE_BINARY", str(candidate)) + calls: list[tuple[str, ...]] = [] + + def fake_mvm(*args: str, **_kwargs: object) -> subprocess.CompletedProcess[str]: + calls.append(args) + stdout = "" + if args[:2] == ("exec", orchestrator.BASE_VM_NAME) and args[-1].startswith("stat -c%s"): + size = candidate.stat().st_size + stdout = f"{size}\n{size}\n" + return subprocess.CompletedProcess(args, 0, stdout=stdout, stderr="") + + original_failure = RuntimeError("asset pull failed") + + def fail_asset_pull(_vm_name: str) -> None: + raise original_failure + + monkeypatch.setattr(orchestrator, "mvm", fake_mvm) + monkeypatch.setattr( + orchestrator, "_install_system_binary_in_runner", lambda _vm_name: None + ) + monkeypatch.setattr( + orchestrator, "_initialize_system_binary_in_runner", lambda _vm_name: None + ) + monkeypatch.setattr(orchestrator, "_pull_builder_assets", fail_asset_pull) + + with pytest.raises(RuntimeError) as captured: + orchestrator._build_base_image("0.3.0-test", rebuild=True) + + assert captured.value is original_failure + builder_removals = [ + args + for args in calls + if args == ("vm", "rm", orchestrator.BASE_VM_NAME, "--force") + ] + assert len(builder_removals) == 2 + assert not any(args[:2] == ("image", "import") for args in calls) + + +@pytest.mark.parametrize("backing_kind", ["missing", "directory", "symlink"]) +def test_ensure_shared_volume_rebuilds_stale_volume_records( + orchestrator: ModuleType, + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, + backing_kind: str, +) -> None: + mirror = tmp_path / "mirror" + mirror.mkdir() + cache = tmp_path / "cache" + backing = cache / "volumes" / "asset-mirror.raw" + backing.parent.mkdir(parents=True) + if backing_kind == "directory": + backing.mkdir() + elif backing_kind == "symlink": + target = tmp_path / "target.raw" + target.write_bytes(b"volume") + backing.symlink_to(target) + + monkeypatch.setattr(orchestrator, "ASSET_MIRROR_HOST", str(mirror)) + monkeypatch.setenv("MVM_CACHE_DIR", str(cache)) + monkeypatch.setattr( + orchestrator, + "_inspect_volume_path", + lambda _name: (True, str(backing)), + ) + calls: list[tuple[tuple[str, ...], dict[str, object]]] = [] + + def fake_mvm(*args: str, **kwargs: object) -> subprocess.CompletedProcess[str]: + calls.append((args, kwargs)) + return subprocess.CompletedProcess(args, 0, stdout="", stderr="") + + seeded: list[bool] = [] + monkeypatch.setattr(orchestrator, "mvm", fake_mvm) + monkeypatch.setattr( + orchestrator, "_create_and_seed_volume", lambda: seeded.append(True) + ) + + orchestrator.ensure_shared_volume() + + assert calls == [ + ( + ("volume", "rm", orchestrator.SHARED_VOLUME_NAME, "--force"), + {"timeout": 30}, + ) + ] + assert seeded == [True] + + +def test_ensure_shared_volume_rejects_poisoned_arbitrary_path_without_mutation( + orchestrator: ModuleType, + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, +) -> None: + cache = tmp_path / "cache" + (cache / "volumes").mkdir(parents=True) + mirror = tmp_path / "mirror" + mirror.mkdir() + unrelated = tmp_path / "unrelated.data" + unrelated.write_bytes(b"must survive") + monkeypatch.setenv("MVM_CACHE_DIR", str(cache)) + monkeypatch.setattr(orchestrator, "ASSET_MIRROR_HOST", str(mirror)) + monkeypatch.setattr( + orchestrator, + "_inspect_volume_path", + lambda _name: (True, str(unrelated)), + ) + + def unexpected_mvm(*_args: object, **_kwargs: object) -> None: + raise AssertionError("poisoned records must not trigger volume mutation") + + monkeypatch.setattr(orchestrator, "mvm", unexpected_mvm) + + with pytest.raises(RuntimeError, match="unexpected shared volume backing path"): + orchestrator.ensure_shared_volume() + + assert unrelated.read_bytes() == b"must survive" + + +def test_populate_volume_image_uses_unprivileged_mkfs_directory_copy( + orchestrator: ModuleType, + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, +) -> None: + mirror = tmp_path / "mirror" + mirror.mkdir() + (mirror / "manifest.json").write_text("{}", encoding="utf-8") + cache = tmp_path / "cache" + backing = cache / "volumes" / "asset-mirror.raw" + backing.parent.mkdir(parents=True) + backing.write_bytes(b"\0" * 4096) + owner_before = (backing.stat().st_uid, backing.stat().st_gid) + calls: list[tuple[list[str], dict[str, object]]] = [] + + def fake_run( + command: list[str], **kwargs: object + ) -> subprocess.CompletedProcess[str]: + calls.append((command, kwargs)) + return subprocess.CompletedProcess(command, 0, stdout="", stderr="") + + monkeypatch.setattr(orchestrator.subprocess, "run", fake_run) + monkeypatch.setenv("MVM_CACHE_DIR", str(cache)) + + orchestrator._populate_volume_image(backing, mirror) + + assert len(calls) == 1 + command, kwargs = calls[0] + assert command[:4] == ["mkfs.ext4", "-F", "-d", str(mirror)] + assert command[4].startswith("/proc/self/fd/") + assert kwargs["capture_output"] is True + assert kwargs["text"] is True + assert kwargs["timeout"] == 900 + assert kwargs["pass_fds"] == (int(command[4].rsplit("/", 1)[-1]),) + assert (backing.stat().st_uid, backing.stat().st_gid) == owner_before + + +def test_populate_volume_image_rejects_symlink_backing_before_mkfs( + orchestrator: ModuleType, + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, +) -> None: + mirror = tmp_path / "mirror" + mirror.mkdir() + cache = tmp_path / "cache" + backing = cache / "volumes" / "asset-mirror.raw" + backing.parent.mkdir(parents=True) + target = tmp_path / "target.raw" + target.write_bytes(b"volume") + backing.symlink_to(target) + monkeypatch.setenv("MVM_CACHE_DIR", str(cache)) + + def unexpected_run(*_args: object, **_kwargs: object) -> None: + raise AssertionError("mkfs must not run for a symlink backing path") + + monkeypatch.setattr(orchestrator.subprocess, "run", unexpected_run) + + with pytest.raises(RuntimeError, match="symlink"): + orchestrator._populate_volume_image(backing, mirror) + + +def test_populate_volume_image_rejects_symlinked_cache_ancestor_before_mkfs( + orchestrator: ModuleType, + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, +) -> None: + actual_cache = tmp_path / "actual-cache" + actual_backing = actual_cache / "volumes" / "asset-mirror.raw" + actual_backing.parent.mkdir(parents=True) + actual_backing.write_bytes(b"volume") + configured_cache = tmp_path / "configured-cache" + configured_cache.symlink_to(actual_cache, target_is_directory=True) + recorded_backing = configured_cache / "volumes" / "asset-mirror.raw" + mirror = tmp_path / "mirror" + mirror.mkdir() + monkeypatch.setenv("MVM_CACHE_DIR", str(configured_cache)) + + def unexpected_run(*_args: object, **_kwargs: object) -> None: + raise AssertionError("mkfs must not traverse a symlinked cache ancestor") + + monkeypatch.setattr(orchestrator.subprocess, "run", unexpected_run) + + with pytest.raises(RuntimeError, match="safely open shared volume directory"): + orchestrator._populate_volume_image(recorded_backing, mirror) + + assert actual_backing.read_bytes() == b"volume" + + +def test_create_volume_removes_record_when_created_backing_is_missing( + orchestrator: ModuleType, + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, +) -> None: + mirror = tmp_path / "mirror" + mirror.mkdir() + cache = tmp_path / "cache" + missing_backing = cache / "volumes" / "asset-mirror.raw" + missing_backing.parent.mkdir(parents=True) + monkeypatch.setattr(orchestrator, "ASSET_MIRROR_HOST", str(mirror)) + monkeypatch.setenv("MVM_CACHE_DIR", str(cache)) + monkeypatch.setattr( + orchestrator, + "_inspect_volume_path", + lambda _name: (True, str(missing_backing)), + ) + calls: list[tuple[tuple[str, ...], dict[str, object]]] = [] + + def fake_mvm(*args: str, **kwargs: object) -> subprocess.CompletedProcess[str]: + calls.append((args, kwargs)) + return subprocess.CompletedProcess(args, 0, stdout="", stderr="") + + monkeypatch.setattr(orchestrator, "mvm", fake_mvm) + + with pytest.raises(RuntimeError, match="backing file is missing"): + orchestrator._create_and_seed_volume() + + assert calls[0][0][:2] == ("volume", "create") + assert calls[-1] == ( + ("volume", "rm", orchestrator.SHARED_VOLUME_NAME, "--force"), + {"check": False, "timeout": 30}, + ) + + +@pytest.mark.parametrize("leftover_kind", ["dangling_symlink", "directory", "file"]) +def test_stale_record_cleanup_must_remove_backing_path_before_volume_create( + orchestrator: ModuleType, + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, + leftover_kind: str, +) -> None: + cache = tmp_path / "cache" + backing = cache / "volumes" / "asset-mirror.raw" + backing.parent.mkdir(parents=True) + if leftover_kind == "dangling_symlink": + backing.symlink_to(tmp_path / "missing-target.raw") + elif leftover_kind == "directory": + backing.mkdir() + else: + backing.write_bytes(b"volume") + mirror = tmp_path / "mirror" + mirror.mkdir() + monkeypatch.setenv("MVM_CACHE_DIR", str(cache)) + monkeypatch.setattr(orchestrator, "ASSET_MIRROR_HOST", str(mirror)) + monkeypatch.setattr( + orchestrator, + "_inspect_volume_path", + lambda _name: (True, str(backing)), + ) + calls: list[tuple[str, ...]] = [] + + def stale_remove(*args: str, **_kwargs: object) -> subprocess.CompletedProcess[str]: + calls.append(args) + if args[:2] == ("volume", "create"): + raise AssertionError("volume create must not follow stale cleanup") + return subprocess.CompletedProcess(args, 0, stdout="", stderr="") + + monkeypatch.setattr(orchestrator, "mvm", stale_remove) + + with pytest.raises(RuntimeError, match="still exists before volume create"): + orchestrator.ensure_shared_volume(rebuild=True) + + assert calls == [ + ("volume", "rm", orchestrator.SHARED_VOLUME_NAME, "--force") + ] + assert backing.lstat() + + +def test_created_volume_with_unsafe_record_path_skips_automatic_cleanup( + orchestrator: ModuleType, + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, +) -> None: + cache = tmp_path / "cache" + (cache / "volumes").mkdir(parents=True) + unrelated = tmp_path / "unrelated.data" + unrelated.write_bytes(b"must survive") + mirror = tmp_path / "mirror" + mirror.mkdir() + monkeypatch.setenv("MVM_CACHE_DIR", str(cache)) + monkeypatch.setattr(orchestrator, "ASSET_MIRROR_HOST", str(mirror)) + monkeypatch.setattr( + orchestrator, + "_inspect_volume_path", + lambda _name: (True, str(unrelated)), + ) + calls: list[tuple[str, ...]] = [] + + def fake_mvm(*args: str, **_kwargs: object) -> subprocess.CompletedProcess[str]: + calls.append(args) + return subprocess.CompletedProcess(args, 0, stdout="", stderr="") + + monkeypatch.setattr(orchestrator, "mvm", fake_mvm) + + with pytest.raises(RuntimeError, match="unexpected shared volume backing path"): + orchestrator._create_and_seed_volume() + + assert len(calls) == 1 + assert calls[0][:2] == ("volume", "create") + assert unrelated.read_bytes() == b"must survive" + + +def test_populate_volume_image_rejects_arbitrary_regular_file( + orchestrator: ModuleType, + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, +) -> None: + cache = tmp_path / "cache" + (cache / "volumes").mkdir(parents=True) + mirror = tmp_path / "mirror" + mirror.mkdir() + unrelated = tmp_path / "unrelated.data" + unrelated.write_bytes(b"must survive") + monkeypatch.setenv("MVM_CACHE_DIR", str(cache)) + + def unexpected_run(*_args: object, **_kwargs: object) -> None: + raise AssertionError("mkfs must not run for an arbitrary path") + + monkeypatch.setattr(orchestrator.subprocess, "run", unexpected_run) + + with pytest.raises(RuntimeError, match="unexpected shared volume backing path"): + orchestrator._populate_volume_image(unrelated, mirror) + + assert unrelated.read_bytes() == b"must survive" + + +def test_populate_volume_image_detects_path_replacement_and_pins_original_inode( + orchestrator: ModuleType, + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, +) -> None: + cache = tmp_path / "cache" + backing = cache / "volumes" / "asset-mirror.raw" + backing.parent.mkdir(parents=True) + backing.write_bytes(b"volume") + original_inode = backing.stat().st_ino + mirror = tmp_path / "mirror" + mirror.mkdir() + victim = tmp_path / "victim.data" + victim.write_bytes(b"must survive") + monkeypatch.setenv("MVM_CACHE_DIR", str(cache)) + command_seen: list[str] = [] + + def replace_path_during_mkfs( + command: list[str], **_kwargs: object + ) -> subprocess.CompletedProcess[str]: + command_seen.extend(command) + backing.unlink() + backing.symlink_to(victim) + return subprocess.CompletedProcess(command, 0, stdout="", stderr="") + + monkeypatch.setattr( + orchestrator.subprocess, "run", replace_path_during_mkfs + ) + + with pytest.raises(RuntimeError, match="replaced during population"): + orchestrator._populate_volume_image(backing, mirror) + + pinned_fd = int(command_seen[-1].rsplit("/", 1)[-1]) + assert command_seen[-1] == f"/proc/self/fd/{pinned_fd}" + assert original_inode != victim.stat().st_ino + assert victim.read_bytes() == b"must survive" + + +@pytest.mark.skipif( + shutil.which("mkfs.ext4") is None or shutil.which("debugfs") is None, + reason="e2fsprogs is required for the direct ext4 population check", +) +def test_populate_volume_image_directly_seeds_a_real_ext4_image( + orchestrator: ModuleType, + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, +) -> None: + cache = tmp_path / "cache" + backing = cache / "volumes" / "asset-mirror.raw" + backing.parent.mkdir(parents=True) + with backing.open("wb") as volume_file: + volume_file.truncate(32 * 1024 * 1024) + owner_before = (backing.stat().st_uid, backing.stat().st_gid) + mirror = tmp_path / "mirror" + mirror.mkdir() + (mirror / "manifest.json").write_text('{"seeded": true}\n', encoding="utf-8") + monkeypatch.setenv("MVM_CACHE_DIR", str(cache)) + + orchestrator._populate_volume_image(backing, mirror) + + inspected = subprocess.run( + ["debugfs", "-R", "cat /manifest.json", str(backing)], + capture_output=True, + text=True, + timeout=30, + ) + assert inspected.returncode == 0, inspected.stderr + assert inspected.stdout == '{"seeded": true}\n' + assert (backing.stat().st_uid, backing.stat().st_gid) == owner_before + + +def test_candidate_version_probe_is_exact_and_isolated_from_host_state( + orchestrator: ModuleType, + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, +) -> None: + candidate = tmp_path / "candidate-mvm" + candidate.write_bytes(b"candidate") + monkeypatch.setattr(orchestrator, "MVM_CANDIDATE_BINARY", str(candidate)) + monkeypatch.setattr(orchestrator, "MVM_BINARY", "/host/controller-mvm") + calls: list[tuple[list[str], dict[str, object]]] = [] + + def fake_run( + command: list[str], **kwargs: object + ) -> subprocess.CompletedProcess[str]: + calls.append((command, kwargs)) + return subprocess.CompletedProcess( + command, 0, stdout="mvm 0.3.0-rc.1\n", stderr="" + ) + + monkeypatch.setattr(orchestrator.subprocess, "run", fake_run) + + version = orchestrator._get_mvm_version() + + assert version == "0.3.0-rc.1" + assert len(calls) == 1 + command, kwargs = calls[0] + assert command == [str(candidate), "--version"] + assert kwargs["capture_output"] is True + assert kwargs["text"] is True + assert kwargs["timeout"] == 10 + env = kwargs["env"] + assert isinstance(env, dict) + for variable in ("MVM_CACHE_DIR", "MVM_CONFIG_DIR", "MVM_TEMP_DIR"): + assert env[variable] != os.environ.get(variable) + assert "mvm-candidate-version-" in env[variable] + + +def test_rebuild_outputs_only_to_the_configured_candidate( + orchestrator: ModuleType, + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, +) -> None: + controller = tmp_path / "controller-mvm" + controller.write_bytes(b"controller") + candidate = tmp_path / "dist" / "mvm" + monkeypatch.setattr(orchestrator, "MVM_BINARY", str(controller)) + monkeypatch.setattr(orchestrator, "MVM_CANDIDATE_BINARY", str(candidate)) + calls: list[list[str]] = [] + + def fake_run(command: list[str], **_kwargs: object) -> subprocess.CompletedProcess[str]: + calls.append(command) + return subprocess.CompletedProcess(command, 0, stdout="", stderr="") + + monkeypatch.setattr(orchestrator.subprocess, "run", fake_run) + monkeypatch.setattr( + orchestrator, "_get_mvm_version", lambda: "0.3.0-rc.1" + ) + + orchestrator._build_mvm_binary("0.3.0-rc.1") + + assert len(calls) == 1 + assert calls[0][-4:-2] == ["--version", "0.3.0-rc.1"] + assert calls[0][-2:] == ["--output", str(candidate)] + assert str(controller) not in calls[0] + + +def test_rebuild_rejects_candidate_with_wrong_reported_version( + orchestrator: ModuleType, + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, +) -> None: + controller = tmp_path / "controller-mvm" + controller.write_bytes(b"controller") + candidate = tmp_path / "dist" / "mvm" + monkeypatch.setattr(orchestrator, "MVM_BINARY", str(controller)) + monkeypatch.setattr(orchestrator, "MVM_CANDIDATE_BINARY", str(candidate)) + monkeypatch.setattr( + orchestrator.subprocess, + "run", + lambda command, **_kwargs: subprocess.CompletedProcess( + command, 0, stdout="", stderr="" + ), + ) + monkeypatch.setattr(orchestrator, "_get_mvm_version", lambda: "0.2.6") + + with pytest.raises(RuntimeError, match="requested 0.3.0, reported 0.2.6"): + orchestrator._build_mvm_binary("0.3.0") + + +def test_explicit_candidate_version_does_not_consult_git( + orchestrator: ModuleType, + monkeypatch: pytest.MonkeyPatch, +) -> None: + def unexpected_run(*_args: object, **_kwargs: object) -> None: + raise AssertionError("an explicit candidate version must not consult git") + + monkeypatch.setattr(orchestrator.subprocess, "run", unexpected_run) + + assert orchestrator._resolve_candidate_build_version("v0.3.0-rc.2") == ( + "0.3.0-rc.2" + ) + + +def test_candidate_version_uses_one_exact_release_tag( + orchestrator: ModuleType, + monkeypatch: pytest.MonkeyPatch, +) -> None: + calls: list[tuple[list[str], dict[str, object]]] = [] + + def fake_run( + command: list[str], **kwargs: object + ) -> subprocess.CompletedProcess[str]: + calls.append((command, kwargs)) + stdout = "v0.3.0-rc.3\n" if command[1] == "tag" else "" + return subprocess.CompletedProcess( + command, + 0, + stdout=stdout, + stderr="", + ) + + monkeypatch.setattr(orchestrator.subprocess, "run", fake_run) + + assert orchestrator._resolve_candidate_build_version(None) == "0.3.0-rc.3" + assert len(calls) == 2 + assert calls[0][0] == [ + "git", + "tag", + "--points-at", + "HEAD", + "--list", + "v[0-9]*", + ] + assert calls[1][0] == [ + "git", + "status", + "--porcelain=v1", + "--untracked-files=all", + ] + for _command, kwargs in calls: + assert kwargs["cwd"] == str(orchestrator._REPO_ROOT) + assert kwargs["capture_output"] is True + assert kwargs["text"] is True + assert kwargs["timeout"] == 10 + + +@pytest.mark.parametrize("tag_output", ["", "v0.2.0\nv0.3.0\n", "release-0.3.0\n"]) +def test_candidate_version_rejects_untagged_or_ambiguous_checkout( + orchestrator: ModuleType, + monkeypatch: pytest.MonkeyPatch, + tag_output: str, +) -> None: + monkeypatch.setattr( + orchestrator.subprocess, + "run", + lambda command, **_kwargs: subprocess.CompletedProcess( + command, 0, stdout=tag_output, stderr="" + ), + ) + + with pytest.raises(RuntimeError, match="--candidate-version"): + orchestrator._resolve_candidate_build_version(None) + + +def test_candidate_version_rejects_dirty_exact_release_tag( + orchestrator: ModuleType, + monkeypatch: pytest.MonkeyPatch, +) -> None: + def fake_run( + command: list[str], **_kwargs: object + ) -> subprocess.CompletedProcess[str]: + stdout = "v0.3.0\n" if command[1] == "tag" else " M internal/file.go\n" + return subprocess.CompletedProcess(command, 0, stdout=stdout, stderr="") + + monkeypatch.setattr(orchestrator.subprocess, "run", fake_run) + + with pytest.raises(RuntimeError, match="--candidate-version"): + orchestrator._resolve_candidate_build_version(None) + + +def test_candidate_version_flag_is_typed( + orchestrator: ModuleType, +) -> None: + parser = orchestrator._build_parser() + parsed = parser.parse_args( + ["--rebuild", "--candidate-version", "v0.3.0-rc.4"] + ) + assert parsed.candidate_version == "0.3.0-rc.4" + + with pytest.raises(SystemExit): + parser.parse_args(["--rebuild", "--candidate-version", "not-a-version"]) + + +@pytest.mark.parametrize( + "version", + [ + "01.2.3", + "1.02.3", + "1.2.03", + "1.2.3-01", + "1.2.3-", + "1.2.3-rc..1", + "1.2.3-rc.", + "1.2.3+", + "1.2.3+meta..build", + "1.2.3+meta.", + ], +) +def test_candidate_version_parser_rejects_non_semver_identifiers( + orchestrator: ModuleType, + version: str, +) -> None: + with pytest.raises(argparse.ArgumentTypeError): + orchestrator._parse_candidate_version(version) + + +def test_candidate_version_flag_requires_rebuild( + orchestrator: ModuleType, + monkeypatch: pytest.MonkeyPatch, +) -> None: + monkeypatch.setattr( + orchestrator.sys, + "argv", + ["run-system-tests.py", "--candidate-version", "0.3.0"], + ) + + with pytest.raises(SystemExit) as captured: + orchestrator.main() + + assert captured.value.code == 2 + + +def test_rebuild_rejects_candidate_controller_alias_before_build( + orchestrator: ModuleType, + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, +) -> None: + controller = tmp_path / "controller-mvm" + controller.write_bytes(b"controller") + monkeypatch.setattr(orchestrator, "MVM_BINARY", str(controller)) + monkeypatch.setattr(orchestrator, "MVM_CANDIDATE_BINARY", str(controller)) + + def unexpected_run(*_args: object, **_kwargs: object) -> None: + raise AssertionError("build must not start when candidate aliases controller") + + monkeypatch.setattr(orchestrator.subprocess, "run", unexpected_run) + + with pytest.raises(RuntimeError, match="outer controller"): + orchestrator._build_mvm_binary("0.3.0") + + +def test_rebuild_rejects_candidate_symlink_before_build( + orchestrator: ModuleType, + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, +) -> None: + controller = tmp_path / "controller-mvm" + controller.write_bytes(b"controller") + target = tmp_path / "candidate-target" + target.write_bytes(b"candidate") + configured = tmp_path / "candidate-mvm" + configured.symlink_to(target) + monkeypatch.setattr(orchestrator, "MVM_BINARY", str(controller)) + monkeypatch.setattr(orchestrator, "MVM_CANDIDATE_CONFIGURED", configured) + monkeypatch.setattr(orchestrator, "MVM_CANDIDATE_BINARY", str(target)) + + def unexpected_run(*_args: object, **_kwargs: object) -> None: + raise AssertionError("build must not follow a candidate symlink") + + monkeypatch.setattr(orchestrator.subprocess, "run", unexpected_run) + + with pytest.raises(RuntimeError, match="symlink"): + orchestrator._build_mvm_binary("0.3.0") + + +def test_rebuild_rejects_candidate_controller_hardlink_before_build( + orchestrator: ModuleType, + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, +) -> None: + controller = tmp_path / "controller-mvm" + controller.write_bytes(b"controller") + candidate = tmp_path / "candidate-mvm" + os.link(controller, candidate) + monkeypatch.setattr(orchestrator, "MVM_BINARY", str(controller)) + monkeypatch.setattr(orchestrator, "MVM_CANDIDATE_CONFIGURED", candidate) + monkeypatch.setattr(orchestrator, "MVM_CANDIDATE_BINARY", str(candidate)) + + def unexpected_run(*_args: object, **_kwargs: object) -> None: + raise AssertionError("build must not replace a controller inode alias") + + monkeypatch.setattr(orchestrator.subprocess, "run", unexpected_run) + + with pytest.raises(RuntimeError, match="outer controller"): + orchestrator._build_mvm_binary("0.3.0") diff --git a/tests/system/COVERAGE_MATRIX.md b/tests/system/COVERAGE_MATRIX.md index 5adb4421..d2588049 100644 --- a/tests/system/COVERAGE_MATRIX.md +++ b/tests/system/COVERAGE_MATRIX.md @@ -18,6 +18,7 @@ removed, or when test coverage changes. ## Table of Contents - [Root CLI](#root-cli) +- [mvm self-update](#mvm-self-update) - [mvm init](#mvm-init) - [mvm config](#mvm-config) - [mvm network](#mvm-network) @@ -56,6 +57,17 @@ removed, or when test coverage changes. --- +## `mvm self-update` + +| Command/Flag | Status | Test File | Test Class(es) | Notes | +|---|---|---|---|---| +| `self-update` (user-owned artifact) | 🔴 Missing | — | — | Success/update behavior for a user-owned executable is not covered at L2. | +| `self-update check` | 🔴 Missing | — | — | Release-check output is not covered at L2. | +| `self-update apply` (user-owned artifact) | 🔴 Missing | — | — | Successful replacement of a user-owned executable is not covered at L2. | +| `self-update apply --force` (system installation) | ✅ Deep | `host/test_system_install.py` | `TestSystemBinaryInstall` | L2 seeds the existing HTTP cache with a newer release, invokes exact `/usr/local/bin/mvm`, verifies refusal before update download, and proves the system-binary hash is unchanged. The cache entry is restored or removed exactly. | + +--- + ## `mvm init` | Command/Flag | Status | Test File | Test Class(es) | Notes | @@ -507,7 +519,11 @@ removed, or when test coverage changes. | `host clean` blocked by running VM | ⚡ Shallow | `host/test_host.py` | `TestHostCleanSafety` | L1 | | `host reset --force` | ⚡ Shallow | `host/test_host.py` | `TestHostCleanDestructive` | L1 — exit 0, non-empty stdout. Marked host_reset (excluded from default runs) | | `host reset` blocked by running VM | ⚡ Shallow | `host/test_host.py` | `TestHostResetSafety` | L1 | -| `host init` | 🟡 Partial | `host/test_host.py` | `TestHostInit` | Non-sudo path verified; sudo path requires host_reset marker | +| `host install-system` | ✅ Deep | `host/test_system_install.py` | `TestSystemBinaryInstall` | The Python runner stages the RC, invokes the real administrator installer, and then all `_run_mvm` calls use exact `/usr/local/bin/mvm`. L2 verifies root ownership, mode, trusted ancestors, content hash, no temporary sibling, and idempotence. | +| `host install-system` rejection cases | ✅ Deep | `host/test_system_install.py` | `TestSystemBinaryInstall` | L2 verifies unprivileged and malformed invocations preserve the target, and an existing target symlink is rejected without changing its destination. Symlink-test restoration is mandatory and asserted. | +| `host install-system` pre-rename failure | ✅ Deep | `host/test_system_install.py`, `internal/core/host/system_binary_test.go` | `TestSystemBinaryInstall`, `TestInstallSystemBinary_PreRenameFailuresPreserveTarget` | L1 covers each injected pre-rename stage. L2 bind-remounts `/usr/local/bin` read-only, invokes the exact staged candidate, verifies the public failure and unchanged canonical metadata/hash, then requires rw remount, unmount, no mount leak, and executable restoration. | +| `host install-system` remaining partial failures | ⚡ Shallow | `internal/core/host/system_binary_test.go`, `pkg/api/host_system_binary_test.go` | `TestInstallSystemBinary_PreRenameFailuresPreserveTarget`, `TestInstallSystemBinary_ReportsPostRenameAndCleanupFailures`, `TestHostInstallSystemBinary_PreservesPartialSuccess` | L1 deterministically covers write/metadata/fsync/close/rename failures, cleanup failure, and post-rename durability reporting. No safe deterministic public-boundary L2 injection exists for those individual stages; uncontrolled disk filling, timing kills, and production test hooks are explicitly rejected. | +| `host init` | 🟡 Partial | `host/test_system_install.py`, `host/test_host.py` | `TestSystemBinaryInstall`, `TestHostInit` | L2 invokes exact `sudo /usr/local/bin/mvm host init` in a runner that already has the `mvm` group and membership. It proves happy-path sudoers reconciliation and invoking-user cache/config ownership. Task 13 gaps: group creation/membership, root-owned journal, rollback, and fail-closed mutation errors. | --- @@ -621,6 +637,7 @@ removed, or when test coverage changes. | Category | Total Scenarios | ✅ Deep | ⚡ Shallow | 🔴 Missing | ⏭️ Skip | |----------|----------------|---------|-------------|-------------|----------| | Root CLI | 8 | 0 | 8 | 0 | 0 | +| self-update | 4 | 1 | 0 | 3 | 0 | | init | 5 | 0 | 5 | 0 | 0 | | config | 12 | 0 | 12 | 0 | 0 | | network | 34 | 13 | 19 | 0 | 0 | @@ -635,21 +652,23 @@ removed, or when test coverage changes. | ssh | 6 | 4 | 2 | 0 | 0 | | console | 6 | 0 | 6 | 0 | 0 | | logs | 9 | 0 | 9 | 0 | 0 | -| host | 11 | 3 | 7 | 0 | 1 | +| host | 15 | 7 | 7 | 0 | 1 | | cache | 13 | 3 | 6 | 0 | 1 | | cp | 12 | 7 | 5 | 0 | 0 | | env | 22 | 9 | 13 | 0 | 0 | | run | 3 | 0 | 3 | 0 | 0 | | full-journeys | 13 | 5 | 8 | 0 | 0 | -| **Total** | **431** | **72** | **292** | **1** | **47** | +| **Total** | **439** | **77** | **292** | **4** | **47** | **Coverage health:** -- ✅ Deep (L3): 72/431 = 16.7% -- ⚡ Shallow (L0-L2 incl. 🟡 Partial): 292/431 = 67.7% -- 🔴 Missing: **1/431 = 0.2%** — `image inspect --tree` does not exist as a CLI command -- ⏭️ Skip-prone: 47/431 = 10.9% - -**CLI gaps: all resolved.** The only remaining `🔴 Missing` entry is `image inspect --tree`, which is a phantom command — the CLI has no `--tree` flag for `image inspect`. All 21 real gaps are now covered with system tests. +- ✅ Deep (L3): 77/439 = 17.5% +- ⚡ Shallow (L0-L2 incl. 🟡 Partial): 292/439 = 66.5% +- 🔴 Missing: **4/439 = 0.9%** — three user-owned `self-update` success/check paths plus the phantom + `image inspect --tree` entry +- ⏭️ Skip-prone: 47/439 = 10.7% + +**Current real CLI gap:** successful user-owned `self-update` check/apply behavior remains unqualified at L2. The +`image inspect --tree` missing entry is a phantom command because the CLI has no such flag. --- diff --git a/tests/system/conftest.py b/tests/system/conftest.py index 0027e540..0ac3605c 100644 --- a/tests/system/conftest.py +++ b/tests/system/conftest.py @@ -21,6 +21,9 @@ import pytest +SYSTEM_MVM_BINARY = "/usr/local/bin/mvm" + + # ============================================================================ # Runner VM name (from orchestrator) # ============================================================================ @@ -72,8 +75,8 @@ def _guest_run( def _run_mvm( vm_name: str, *args: str, check: bool = True, timeout: int = 60 ) -> subprocess.CompletedProcess[str]: - """Run an mvm command. vm_name is ignored — we're already inside the test VM.""" - cmd = ["mvm", *args] + """Run the installed mvm CLI. vm_name is ignored inside the test VM.""" + cmd = [SYSTEM_MVM_BINARY, *args] result = subprocess.run( cmd, capture_output=True, diff --git a/tests/system/host/test_system_install.py b/tests/system/host/test_system_install.py new file mode 100644 index 00000000..718c4cbc --- /dev/null +++ b/tests/system/host/test_system_install.py @@ -0,0 +1,506 @@ +"""Root-owned system binary installation tests — Tier 1 runner VM.""" + +from __future__ import annotations + +import hashlib +import json +import os +import shlex +import stat +from pathlib import Path +from typing import Iterator + +import pytest + +from tests.system.conftest import SYSTEM_MVM_BINARY, _guest_run, _run_mvm + +pytestmark = [ + pytest.mark.system, + pytest.mark.domain_system_install, +] + +_INSTALL_CANDIDATE = "/opt/mvmctl-test/mvm-candidate" +_SYSTEM_DIRECTORY_CHAIN = ("/", "/usr", "/usr/local", "/usr/local/bin") +_CANDIDATE_DIRECTORY_CHAIN = ("/opt", "/opt/mvmctl-test") +_SYMLINK_BACKUP = "/usr/local/bin/.mvm-system-install-l2.backup" +_SYMLINK_TARGET = "/home/runner/.mvm-system-install-symlink-target" +_LATEST_RELEASE_URL = ( + "https://api.github.com/repos/AlanD20/mvmctl/releases/latest" +) + + +def _sha256(runner_vm: str, path: str) -> str: + result = _guest_run(runner_vm, f"sha256sum {shlex.quote(path)}") + return result.stdout.split()[0] + + +def _stat(runner_vm: str, path: str) -> tuple[int, int, int, str]: + result = _guest_run( + runner_vm, + f"stat -c '%u %g %a %F' {shlex.quote(path)}", + ) + owner, group, mode, file_type = result.stdout.strip().split(maxsplit=3) + return int(owner), int(group), int(mode, 8), file_type + + +@pytest.fixture +def self_update_release_cache() -> Iterator[None]: + """Seed a newer release response and restore the exact prior cache state.""" + temp_root = Path(os.environ.get("MVM_TEMP_DIR", "/tmp/mvmctl")) + temp_root_existed = temp_root.exists() + cache_dir = temp_root / "http" + assert not cache_dir.is_symlink(), f"release cache directory is a symlink: {cache_dir}" + cache_dir_existed = cache_dir.exists() + cache_key = hashlib.sha256(_LATEST_RELEASE_URL.encode()).hexdigest() + cache_path = cache_dir / cache_key + assert not cache_path.is_symlink(), f"release cache entry is a symlink: {cache_path}" + original = cache_path.read_bytes() if cache_path.exists() else None + original_stat = cache_path.stat() if original is not None else None + cache_dir.mkdir(parents=True, exist_ok=True) + cache_path.write_text( + json.dumps({"tag_name": "v999.0.0", "assets": []}), + encoding="utf-8", + ) + try: + yield + finally: + failures: list[str] = [] + try: + if original is None: + cache_path.unlink(missing_ok=True) + else: + cache_path.write_bytes(original) + if cache_path.read_bytes() != original: + failures.append("release cache content was not restored exactly") + assert original_stat is not None + os.chmod(cache_path, stat.S_IMODE(original_stat.st_mode)) + os.utime( + cache_path, + ns=(original_stat.st_atime_ns, original_stat.st_mtime_ns), + ) + except (OSError, AssertionError) as error: + failures.append(f"restore release cache file: {error}") + if not cache_dir_existed: + try: + cache_dir.rmdir() + except OSError as error: + failures.append(f"remove created release cache directory: {error}") + if not temp_root_existed: + try: + temp_root.rmdir() + except OSError as error: + failures.append(f"remove created temp directory: {error}") + + if original is None: + if cache_path.exists(): + failures.append(f"release cache entry still exists: {cache_path}") + assert not failures, "self-update cache restoration failed:\n" + "\n".join( + failures + ) + + +class TestSystemBinaryInstall: + def test_tests_execute_the_canonical_installed_cli(self, runner_vm: str) -> None: + resolved = _guest_run(runner_vm, "command -v mvm").stdout.strip() + assert resolved == SYSTEM_MVM_BINARY + + version = _run_mvm(runner_vm, "--version") + assert version.stdout.startswith("mvm ") + + owner, group, mode, file_type = _stat(runner_vm, SYSTEM_MVM_BINARY) + assert (owner, group, mode) == (0, 0, 0o755) + assert file_type == "regular file" + + for directory in _SYSTEM_DIRECTORY_CHAIN + _CANDIDATE_DIRECTORY_CHAIN: + dir_owner, dir_group, dir_mode, dir_type = _stat(runner_vm, directory) + assert dir_owner == 0, f"{directory} is not root-owned" + assert dir_group == 0, f"{directory} is not root-group-owned" + assert dir_mode & 0o022 == 0, f"{directory} is group/world writable" + assert dir_type == "directory" + + candidate_owner, candidate_group, candidate_mode, candidate_type = _stat( + runner_vm, _INSTALL_CANDIDATE + ) + assert (candidate_owner, candidate_group, candidate_mode) == (0, 0, 0o755) + assert candidate_type == "regular file" + assert _sha256(runner_vm, SYSTEM_MVM_BINARY) == _sha256( + runner_vm, _INSTALL_CANDIDATE + ) + leftovers = _guest_run( + runner_vm, + "find /usr/local/bin -maxdepth 1 -name '.mvm-install-*.tmp' -print", + ) + assert leftovers.stdout.strip() == "" + + def test_unprivileged_install_cannot_replace_the_system_binary( + self, runner_vm: str + ) -> None: + before = _sha256(runner_vm, SYSTEM_MVM_BINARY) + + rejected = _guest_run( + runner_vm, + f"{_INSTALL_CANDIDATE} host install-system", + check=False, + ) + + assert rejected.returncode != 0 + assert "requires root" in (rejected.stdout + rejected.stderr).lower() + assert _sha256(runner_vm, SYSTEM_MVM_BINARY) == before + + @pytest.mark.parametrize( + "extra_args", + [ + ("host", "install-system", "--force"), + ("--debug", "host", "install-system"), + ("host", "--verbose", "install-system"), + ], + ) + def test_malformed_install_route_fails_without_replacing_the_binary( + self, runner_vm: str, extra_args: tuple[str, ...] + ) -> None: + before = _sha256(runner_vm, SYSTEM_MVM_BINARY) + command = " ".join( + ["sudo", "-n", shlex.quote(_INSTALL_CANDIDATE)] + + [shlex.quote(arg) for arg in extra_args] + ) + + rejected = _guest_run(runner_vm, command, check=False) + + assert rejected.returncode != 0 + assert "accepts no flags or arguments" in ( + rejected.stdout + rejected.stderr + ).lower() + assert _sha256(runner_vm, SYSTEM_MVM_BINARY) == before + + def test_install_system_is_idempotent_from_the_installed_cli( + self, runner_vm: str + ) -> None: + before = _sha256(runner_vm, SYSTEM_MVM_BINARY) + + installed = _guest_run( + runner_vm, + f"sudo -n {SYSTEM_MVM_BINARY} host install-system", + check=False, + ) + assert installed.returncode == 0, installed.stderr + + assert _sha256(runner_vm, SYSTEM_MVM_BINARY) == before + assert _sha256(runner_vm, _INSTALL_CANDIDATE) == before + assert _run_mvm(runner_vm, "--version").returncode == 0 + + def test_host_init_then_unprivileged_init_use_the_installed_cli( + self, runner_vm: str + ) -> None: + initialized = _guest_run( + runner_vm, + f"sudo -n {SYSTEM_MVM_BINARY} host init", + check=False, + timeout=180, + ) + assert initialized.returncode == 0, initialized.stderr + + status = _run_mvm(runner_vm, "host", "status", "--json", check=False) + assert status.returncode == 0, status.stderr + sudoers = _guest_run( + runner_vm, + "stat -c '%u:%g:%a:%F' /etc/sudoers.d/mvm", + check=False, + ) + assert sudoers.returncode == 0, sudoers.stderr + assert sudoers.stdout.strip() == "0:0:440:regular file" + group = _guest_run(runner_vm, "getent group mvm", check=False) + assert group.returncode == 0, group.stderr + + user_initialized = _guest_run( + runner_vm, + f"MVM_ASSET_MIRROR=/mnt {SYSTEM_MVM_BINARY} init --non-interactive", + check=False, + timeout=180, + ) + assert user_initialized.returncode == 0, user_initialized.stderr + + user_state = _guest_run( + runner_vm, + "stat -c '%U:%G:%F' /home/runner/.config/mvmctl " + "/home/runner/.cache/mvmctl", + check=False, + ) + assert user_state.returncode == 0, user_state.stderr + assert user_state.stdout.splitlines() == [ + "runner:runner:directory", + "runner:runner:directory", + ] + + @pytest.mark.destructive + def test_self_update_cannot_replace_the_system_installation( + self, runner_vm: str, self_update_release_cache: None + ) -> None: + before = _sha256(runner_vm, SYSTEM_MVM_BINARY) + + rejected = _run_mvm( + runner_vm, + "self-update", + "apply", + "--force", + check=False, + timeout=90, + ) + + assert rejected.returncode != 0 + combined = (rejected.stdout + rejected.stderr).lower() + assert "cannot replace the system installation" in combined + assert "sudo host install-system" in combined + assert _sha256(runner_vm, SYSTEM_MVM_BINARY) == before + + @pytest.mark.destructive + def test_read_only_install_directory_preserves_system_binary( + self, runner_vm: str + ) -> None: + before_hash = _sha256(runner_vm, SYSTEM_MVM_BINARY) + before_metadata = _stat(runner_vm, SYSTEM_MVM_BINARY) + assert before_metadata == (0, 0, 0o755, "regular file") + expected_stat = "0:0:755:regular file" + bind_mounted = False + try: + existing_mount = _guest_run( + runner_vm, + f"findmnt -rn --mountpoint {shlex.quote('/usr/local/bin')} " + "-o TARGET,OPTIONS", + check=False, + ) + assert existing_mount.returncode == 1, ( + "installer fault target is already a mountpoint: " + f"rc={existing_mount.returncode}, " + f"stdout={existing_mount.stdout!r}, " + f"stderr={existing_mount.stderr!r}" + ) + + mounted = _guest_run( + runner_vm, + "sudo -n mount --bind /usr/local/bin /usr/local/bin", + check=False, + ) + assert mounted.returncode == 0, mounted.stderr + bind_mounted = True + + read_only = _guest_run( + runner_vm, + "sudo -n mount -o remount,bind,ro /usr/local/bin", + check=False, + ) + assert read_only.returncode == 0, read_only.stderr + mount_state = _guest_run( + runner_vm, + "findmnt -rn --mountpoint /usr/local/bin -o OPTIONS", + check=False, + ) + assert mount_state.returncode == 0, mount_state.stderr + assert "ro" in mount_state.stdout.strip().split(",") + + rejected = _guest_run( + runner_vm, + f"sudo -n {shlex.quote(_INSTALL_CANDIDATE)} " + "host install-system", + check=False, + ) + + assert rejected.returncode != 0 + assert "read-only file system" in ( + rejected.stdout + rejected.stderr + ).lower() + assert _sha256(runner_vm, SYSTEM_MVM_BINARY) == before_hash + assert _stat(runner_vm, SYSTEM_MVM_BINARY) == before_metadata + leftovers = _guest_run( + runner_vm, + "find /usr/local/bin -maxdepth 1 " + "-name '.mvm-install-*.tmp' -print", + ) + assert leftovers.stdout.strip() == "" + finally: + restoration_failures: list[str] = [] + if bind_mounted: + writable = _guest_run( + runner_vm, + "sudo -n mount -o remount,bind,rw /usr/local/bin", + check=False, + ) + if writable.returncode != 0: + restoration_failures.append( + "restore writable bind mount: " + f"rc={writable.returncode}, " + f"stdout={writable.stdout!r}, stderr={writable.stderr!r}" + ) + unmounted = _guest_run( + runner_vm, + "sudo -n umount /usr/local/bin", + check=False, + ) + if unmounted.returncode != 0: + restoration_failures.append( + "unmount installer fault target: " + f"rc={unmounted.returncode}, " + f"stdout={unmounted.stdout!r}, " + f"stderr={unmounted.stderr!r}" + ) + + leaked_mount = _guest_run( + runner_vm, + "findmnt -rn --mountpoint /usr/local/bin -o TARGET,OPTIONS", + check=False, + ) + if leaked_mount.returncode != 1: + restoration_failures.append( + "installer fault mount leaked: " + f"rc={leaked_mount.returncode}, " + f"stdout={leaked_mount.stdout!r}, " + f"stderr={leaked_mount.stderr!r}" + ) + restored = _guest_run( + runner_vm, + f"test ! -L {shlex.quote(SYSTEM_MVM_BINARY)} && " + f"test \"$(stat -c '%u:%g:%a:%F' " + f"{shlex.quote(SYSTEM_MVM_BINARY)})\" = " + f"{shlex.quote(expected_stat)} && " + f"test \"$(sha256sum {shlex.quote(SYSTEM_MVM_BINARY)} " + f"| cut -d ' ' -f 1)\" = {shlex.quote(before_hash)} && " + "test -z \"$(find /usr/local/bin -maxdepth 1 " + "-name '.mvm-install-*.tmp' -print -quit)\"", + check=False, + ) + if restored.returncode != 0: + restoration_failures.append( + "verify canonical restoration: " + f"rc={restored.returncode}, stdout={restored.stdout!r}, " + f"stderr={restored.stderr!r}" + ) + executable = _run_mvm(runner_vm, "--version", check=False) + if executable.returncode != 0: + restoration_failures.append( + "execute restored binary: " + f"rc={executable.returncode}, " + f"stdout={executable.stdout!r}, " + f"stderr={executable.stderr!r}" + ) + assert not restoration_failures, ( + "read-only installer fault restoration failed:\n" + + "\n".join(restoration_failures) + ) + + @pytest.mark.destructive + def test_existing_target_symlink_is_rejected_and_restored( + self, runner_vm: str + ) -> None: + before = _sha256(runner_vm, SYSTEM_MVM_BINARY) + backup_created = False + try: + absent = _guest_run( + runner_vm, + f"sudo -n test ! -e {shlex.quote(_SYMLINK_BACKUP)} && " + f"sudo -n test ! -L {shlex.quote(_SYMLINK_BACKUP)}", + check=False, + ) + assert absent.returncode == 0, ( + f"stale system-install backup exists: {absent.stderr}" + ) + _guest_run( + runner_vm, + f"printf 'must-not-change\\n' > {shlex.quote(_SYMLINK_TARGET)}", + ) + moved = _guest_run( + runner_vm, + f"sudo -n mv {shlex.quote(SYSTEM_MVM_BINARY)} " + f"{shlex.quote(_SYMLINK_BACKUP)}", + check=False, + ) + assert moved.returncode == 0, moved.stderr + backup_created = True + linked = _guest_run( + runner_vm, + f"sudo -n ln -s {shlex.quote(_SYMLINK_TARGET)} " + f"{shlex.quote(SYSTEM_MVM_BINARY)}", + check=False, + ) + assert linked.returncode == 0, linked.stderr + + rejected = _guest_run( + runner_vm, + f"sudo -n {shlex.quote(_INSTALL_CANDIDATE)} host install-system", + check=False, + ) + + assert rejected.returncode != 0 + combined = (rejected.stdout + rejected.stderr).lower() + assert "system binary" in combined + target = _guest_run( + runner_vm, + f"readlink {shlex.quote(SYSTEM_MVM_BINARY)}", + ) + assert target.stdout.strip() == _SYMLINK_TARGET + marker = _guest_run( + runner_vm, + f"cat {shlex.quote(_SYMLINK_TARGET)}", + ) + assert marker.stdout == "must-not-change\n" + assert _sha256(runner_vm, _SYMLINK_BACKUP) == before + finally: + restoration_failures: list[str] = [] + if backup_created: + for label, command in ( + ( + "remove test target", + f"sudo -n rm -f {shlex.quote(SYSTEM_MVM_BINARY)}", + ), + ( + "restore canonical binary", + f"sudo -n mv {shlex.quote(_SYMLINK_BACKUP)} " + f"{shlex.quote(SYSTEM_MVM_BINARY)}", + ), + ( + "restore canonical metadata", + f"sudo -n chown root:root {shlex.quote(SYSTEM_MVM_BINARY)} && " + f"sudo -n chmod 0755 {shlex.quote(SYSTEM_MVM_BINARY)}", + ), + ): + result = _guest_run(runner_vm, command, check=False) + if result.returncode != 0: + restoration_failures.append( + f"{label}: rc={result.returncode}, " + f"stdout={result.stdout!r}, stderr={result.stderr!r}" + ) + cleanup = _guest_run( + runner_vm, + f"rm -f {shlex.quote(_SYMLINK_TARGET)}", + check=False, + ) + if cleanup.returncode != 0: + restoration_failures.append( + f"remove test marker: rc={cleanup.returncode}, " + f"stdout={cleanup.stdout!r}, stderr={cleanup.stderr!r}" + ) + + restored = _guest_run( + runner_vm, + f"test ! -L {shlex.quote(SYSTEM_MVM_BINARY)} && " + f"test ! -e {shlex.quote(_SYMLINK_BACKUP)} && " + f"test ! -L {shlex.quote(_SYMLINK_BACKUP)} && " + f"test \"$(stat -c '%u:%g:%a' " + f"{shlex.quote(SYSTEM_MVM_BINARY)})\" = '0:0:755' && " + f"test \"$(sha256sum {shlex.quote(SYSTEM_MVM_BINARY)} " + f"| cut -d ' ' -f 1)\" = {shlex.quote(before)}", + check=False, + ) + if restored.returncode != 0: + restoration_failures.append( + f"verify canonical restoration: rc={restored.returncode}, " + f"stdout={restored.stdout!r}, stderr={restored.stderr!r}" + ) + executable = _run_mvm(runner_vm, "--version", check=False) + if executable.returncode != 0: + restoration_failures.append( + f"execute restored binary: rc={executable.returncode}, " + f"stdout={executable.stdout!r}, stderr={executable.stderr!r}" + ) + assert not restoration_failures, ( + "system binary restoration failed:\n" + + "\n".join(restoration_failures) + ) diff --git a/tests/system/pytest.ini b/tests/system/pytest.ini index fc377d10..f5476156 100644 --- a/tests/system/pytest.ini +++ b/tests/system/pytest.ini @@ -8,6 +8,7 @@ markers = domain_cache: Cache management domain domain_keys: SSH key domain domain_host: Host initialization domain + domain_system_install: System binary installation domain domain_invariant: Invariant checks domain_network: Network management domain domain_volume: Volume management domain From fecbcd62f12423bb45a0f98a6174ef327514c2aa Mon Sep 17 00:00:00 2001 From: AlanD20 Date: Sun, 23 Aug 2026 14:41:52 +0200 Subject: [PATCH 0014/1457] feat: allow pinned binary version during init --- internal/cli/init.go | 36 ++++++++++++---- internal/cli/init_internal_test.go | 66 ++++++++++++++++++++++++++++++ internal/cli/init_test.go | 54 ++++++++++++++++++++++++ 3 files changed, 148 insertions(+), 8 deletions(-) create mode 100644 internal/cli/init_internal_test.go diff --git a/internal/cli/init.go b/internal/cli/init.go index d44f6d8d..6826c141 100644 --- a/internal/cli/init.go +++ b/internal/cli/init.go @@ -20,18 +20,28 @@ func NewInitCmd(initAPI api.InitAPI, hostAPI api.HostAPI) *cobra.Command { var nonInteractive bool var skipHost bool var skipNetwork bool + var binaryVersion string cmd := &cobra.Command{ Use: "init", Short: fmt.Sprintf("Initialize %s", infra.CLIName), RunE: func(cmd *cobra.Command, args []string) error { - return runInitWizard(cmd.Context(), initAPI, hostAPI, nonInteractive, skipHost, skipNetwork) + return runInitWizard( + cmd.Context(), + initAPI, + hostAPI, + nonInteractive, + skipHost, + skipNetwork, + binaryVersion, + ) }, } cmd.Flags().BoolVar(&nonInteractive, "non-interactive", false, "Use defaults, skip prompts") cmd.Flags().BoolVar(&skipHost, "skip-host", false, "Skip host init step") cmd.Flags().BoolVar(&skipNetwork, "skip-network", false, "Skip default network creation") + cmd.Flags().StringVar(&binaryVersion, "binary-version", "", "Firecracker version to download when none is local") return cmd } @@ -42,11 +52,20 @@ func runInitWizard( initAPI api.InitAPI, hostAPI api.HostAPI, nonInteractive, skipHost, skipNetwork bool, + binaryVersion string, ) error { common.Cli.Info("") common.Cli.Info(fmt.Sprintf("%s init — first-time setup", infra.CLIName)) common.Cli.Info(strings.Repeat("-", 40)) - result, err := handleInteractiveFlow(ctx, initAPI, hostAPI, nonInteractive, skipHost, skipNetwork) + result, err := handleInteractiveFlow( + ctx, + initAPI, + hostAPI, + nonInteractive, + skipHost, + skipNetwork, + binaryVersion, + ) if err != nil { return err } @@ -229,7 +248,6 @@ func (s *initState) handleSudoRequired(ctx context.Context, interaction *errs.Ne hostStateAfter := s.hostAPI.HostStatusCheck(ctx) s.hostSetupMsg = composeHostSetupMessage(hostStateBefore, hostStateAfter) s.sudoCompleted = true - s.downloadVersion = "" return nil } @@ -281,13 +299,15 @@ func handleInteractiveFlow( initAPI api.InitAPI, hostAPI api.HostAPI, nonInteractive, skipHost, skipNetwork bool, + binaryVersion string, ) (*results.InitResult, error) { state := &initState{ - initAPI: initAPI, - hostAPI: hostAPI, - nonInteractive: nonInteractive, - skipHost: skipHost, - skipNetwork: skipNetwork, + initAPI: initAPI, + hostAPI: hostAPI, + nonInteractive: nonInteractive, + skipHost: skipHost, + skipNetwork: skipNetwork, + downloadVersion: binaryVersion, } // Always return the last result even on early exit. diff --git a/internal/cli/init_internal_test.go b/internal/cli/init_internal_test.go new file mode 100644 index 00000000..4b75190c --- /dev/null +++ b/internal/cli/init_internal_test.go @@ -0,0 +1,66 @@ +package cli + +import ( + "context" + "os" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "mvmctl/internal/lib/model" + "mvmctl/internal/lib/system" + "mvmctl/internal/testutil" + "mvmctl/pkg/api/results" + "mvmctl/pkg/errs" +) + +func TestInitStatePreservesExplicitBinaryVersionAfterSuccessfulSudo(t *testing.T) { + reader, writer, err := os.Pipe() + require.NoError(t, err) + _, err = writer.WriteString("y\n") + require.NoError(t, err) + require.NoError(t, writer.Close()) + originalStdin := os.Stdin + os.Stdin = reader + t.Cleanup(func() { + os.Stdin = originalStdin + require.NoError(t, reader.Close()) + }) + + originalRunner := system.DefaultRunner + system.DefaultRunner = &testutil.FakeRunner{} + t.Cleanup(func() { system.DefaultRunner = originalRunner }) + + statusCalls := 0 + state := &initState{ + initAPI: &testutil.MockInitAPI{ + InitCheckReadinessFunc: func(context.Context) *model.ProbeResult { + return &model.ProbeResult{} + }, + }, + hostAPI: &testutil.MockHostAPI{ + HostStatusCheckFunc: func(context.Context) *results.HostStatusCheck { + statusCalls++ + if statusCalls == 1 { + return &results.HostStatusCheck{} + } + return &results.HostStatusCheck{ + GroupExists: true, + SudoersExists: true, + UserInGroup: true, + } + }, + }, + downloadVersion: "1.15.2", + } + + err = state.handleSudoRequired(context.Background(), &errs.NeedsInteraction{ + Code: "privilege.sudo_required", + Context: map[string]any{"command": "sudo mvm host init"}, + }) + require.NoError(t, err) + assert.True(t, state.sudoCompleted) + assert.Equal(t, "1.15.2", state.downloadVersion) + assert.Equal(t, 2, statusCalls) +} diff --git a/internal/cli/init_test.go b/internal/cli/init_test.go index 7bcb3956..fe43138d 100644 --- a/internal/cli/init_test.go +++ b/internal/cli/init_test.go @@ -1,13 +1,16 @@ package cli_test import ( + "context" "testing" "github.com/stretchr/testify/assert" "github.com/stretchr/testify/require" "mvmctl/internal/cli" + "mvmctl/internal/infra/event" "mvmctl/internal/testutil" + "mvmctl/pkg/api/results" ) // --- NewInitCmd --- @@ -31,6 +34,7 @@ func TestNewInitCmd(t *testing.T) { {name: "non-interactive", shorthand: ""}, {name: "skip-host", shorthand: ""}, {name: "skip-network", shorthand: ""}, + {name: "binary-version", shorthand: ""}, } for _, f := range expectedFlags { @@ -55,4 +59,54 @@ func TestNewInitCmd(t *testing.T) { skipNetwork := cmd.Flags().Lookup("skip-network") require.NotNil(t, skipNetwork) assert.Equal(t, "false", skipNetwork.Value.String(), "--skip-network defaults to false") + + binaryVersion := cmd.Flags().Lookup("binary-version") + require.NotNil(t, binaryVersion) + assert.Empty(t, binaryVersion.Value.String(), "--binary-version defaults to empty") +} + +func TestInitCommandForwardsBinaryVersion(t *testing.T) { + tests := map[string]struct { + args []string + wantVersion string + wantNonInteractive bool + }{ + "explicit_version": { + args: []string{"--skip-host", "--skip-network", "--binary-version", "1.15.2"}, + wantVersion: "1.15.2", + }, + "non_interactive": { + args: []string{"--non-interactive", "--skip-host", "--skip-network", "--binary-version", "latest"}, + wantVersion: "latest", + wantNonInteractive: true, + }, + "no_version_flag": { + args: []string{"--skip-host", "--skip-network"}, + }, + } + for name, tc := range tests { + t.Run(name, func(t *testing.T) { + calls := 0 + mock := &testutil.MockInitAPI{ + InitRunFullFunc: func( + _ context.Context, + _, _, nonInteractive, _ bool, + _ string, + downloadVersion string, + _ *bool, + _ event.OnProgressCallback, + ) *results.InitResult { + calls++ + assert.Equal(t, tc.wantVersion, downloadVersion) + assert.Equal(t, tc.wantNonInteractive, nonInteractive) + return &results.InitResult{HostReady: true} + }, + } + cmd := cli.NewInitCmd(mock, &testutil.MockHostAPI{}) + cmd.SetArgs(tc.args) + + require.NoError(t, cmd.ExecuteContext(context.Background())) + assert.Equal(t, 1, calls) + }) + } } From 768765833384577a8e5526bfe7a8dbef0c4706cc Mon Sep 17 00:00:00 2001 From: AlanD20 Date: Sun, 23 Aug 2026 14:48:32 +0200 Subject: [PATCH 0015/1457] test: pin runner Firecracker initialization --- scripts/run-system-tests.py | 98 ++++++++------------------ scripts/tests/test_run_system_tests.py | 91 ++++++++++++++++++++++-- 2 files changed, 118 insertions(+), 71 deletions(-) diff --git a/scripts/run-system-tests.py b/scripts/run-system-tests.py index 3c72ee8f..8e28b34e 100755 --- a/scripts/run-system-tests.py +++ b/scripts/run-system-tests.py @@ -75,6 +75,11 @@ RUNNER_SYSTEM_CANDIDATE_DIR = "/opt/mvmctl-test" RUNNER_SYSTEM_CANDIDATE = f"{RUNNER_SYSTEM_CANDIDATE_DIR}/mvm-candidate" RUNNER_SYSTEM_UPLOAD = "/home/runner/.mvm-system-candidate.upload" +RUNNER_USER_INIT_COMMAND = ( + "sudo mkdir -p /mnt && sudo mount /dev/vdb /mnt && " + "MVM_ASSET_MIRROR=/mnt /usr/local/bin/mvm init --non-interactive " + "--binary-version 1.16.0" +) # Base VM used to build the custom image BASE_VM_NAME = "base-img-builder" @@ -788,6 +793,27 @@ def _initialize_system_binary_in_runner(vm_name: str) -> None: ) +def _initialize_runner_user( + vm_name: str, + *, + timeout: int, + capture: bool = True, +) -> None: + """Initialize runner-owned state with the release-pinned Firecracker pair.""" + mvm( + "exec", + vm_name, + "--user", + "runner", + "--timeout", + "10", + "--", + RUNNER_USER_INIT_COMMAND, + timeout=timeout, + capture=capture, + ) + + def _ensure_builder_key() -> None: """Create the builder SSH key if it doesn't exist yet.""" check = mvm("key", "inspect", BUILDER_KEY_NAME, check=False, timeout=10) @@ -840,18 +866,7 @@ def provision_t1(vm_name: str, mvm_version: str) -> None: # CRITICAL: mvm init MUST run as the unprivileged user (runner), NOT via sudo. # Running as root creates the cache dir with root ownership — test VMs inherit # this state and break with 'permission denied' on /home/runner/.cache/mvmctl. - mvm( - "exec", - vm_name, - "--user", - "runner", - "--timeout", - "10", - "--", - "sudo mkdir -p /mnt && sudo mount /dev/vdb /mnt && " - "MVM_ASSET_MIRROR=/mnt /usr/local/bin/mvm init --non-interactive", - timeout=360, - ) + _initialize_runner_user(vm_name, timeout=360) def _ensure_official_kernel_on_host(vm_name: str) -> None: @@ -936,18 +951,7 @@ def provision_t2(vm_name: str, mvm_version: str) -> None: # Running as root creates the cache dir with root ownership — test VMs inherit # this state and break with 'permission denied' on /home/runner/.cache/mvmctl. log(f" Initializing mvm inside '{vm_name}'...") - mvm( - "exec", - vm_name, - "--user", - "runner", - "--timeout", - "10", - "--", - "sudo mkdir -p /mnt && sudo mount /dev/vdb /mnt && " - "MVM_ASSET_MIRROR=/mnt /usr/local/bin/mvm init --non-interactive", - timeout=180, - ) + _initialize_runner_user(vm_name, timeout=180) log(f" Registering assets in '{vm_name}' (cache hits)...") mvm( @@ -1006,11 +1010,6 @@ def destroy_vm(vm_name: str) -> None: def _pull_builder_assets(vm_name: str) -> None: """Pull base-image assets serially so each streamed exec stays bounded.""" pulls = ( - ( - "Firecracker binary", - "MVM_ASSET_MIRROR=/mnt /usr/local/bin/mvm binary pull " - "firecracker --default --force --version 1.16.0", - ), ( "Firecracker kernel", "MVM_ASSET_MIRROR=/mnt /usr/local/bin/mvm kernel pull " @@ -1219,20 +1218,7 @@ def _build_base_image(mvm_version: str, *, rebuild: bool = False) -> str: # need /home/runner/.cache/mvmctl owned by runner. Permission denied = broken. # Asset pulls are pre-baked into the base image so each test VM doesn't # re-pull them (saves ~3 min per T1 VM). - mvm( - "exec", - BASE_VM_NAME, - "--user", - "runner", - "--timeout", - "10", - "--", - "sudo mkdir -p /mnt && " - "sudo mount /dev/vdb /mnt && " - "MVM_ASSET_MIRROR=/mnt /usr/local/bin/mvm init --non-interactive", - timeout=180, - capture=False, - ) + _initialize_runner_user(BASE_VM_NAME, timeout=180, capture=False) # Asset pulls are pre-baked into the base image so each test VM doesn't # re-pull them (saves ~3 min per T1 VM). Keep each pull in its own exec: @@ -1641,18 +1627,7 @@ def run_prepare(*, rebuild_volume: bool = False, rebuild_image: bool = False) -> ) try: log(f" Running mvm init inside '{t1}'...") - mvm( - "exec", - t1, - "--user", - "runner", - "--timeout", - "10", - "--", - "sudo mkdir -p /mnt && sudo mount /dev/vdb /mnt && " - "MVM_ASSET_MIRROR=/mnt /usr/local/bin/mvm init --non-interactive", - timeout=90, - ) + _initialize_runner_user(t1, timeout=90) finally: destroy_vm(t1) @@ -1686,18 +1661,7 @@ def run_prepare(*, rebuild_volume: bool = False, rebuild_image: bool = False) -> ) try: log(f"[7/8] Setting up '{t2}' (mount + init)...") - mvm( - "exec", - t2, - "--user", - "runner", - "--timeout", - "10", - "--", - "sudo mkdir -p /mnt && sudo mount /dev/vdb /mnt && " - "MVM_ASSET_MIRROR=/mnt /usr/local/bin/mvm init --non-interactive", - timeout=180, - ) + _initialize_runner_user(t2, timeout=180) log(f"[8/8] Validating cache hit (pulling 1 asset)...") result = mvm( diff --git a/scripts/tests/test_run_system_tests.py b/scripts/tests/test_run_system_tests.py index 76bfc621..7f3ed162 100644 --- a/scripts/tests/test_run_system_tests.py +++ b/scripts/tests/test_run_system_tests.py @@ -58,6 +58,69 @@ def test_guest_product_commands_use_the_exact_installed_cli( assert unqualified == [] +def test_all_guest_user_initialization_uses_the_pinned_firecracker_version( + orchestrator: ModuleType, +) -> None: + tree = ast.parse(Path(orchestrator.__file__).read_text(encoding="utf-8")) + init_commands = [ + node.value + for node in ast.walk(tree) + if isinstance(node, ast.Constant) + and isinstance(node.value, str) + and "/usr/local/bin/mvm init" in node.value + ] + init_calls = [ + node + for node in ast.walk(tree) + if isinstance(node, ast.Call) + and isinstance(node.func, ast.Name) + and node.func.id == "_initialize_runner_user" + ] + + assert init_commands == [orchestrator.RUNNER_USER_INIT_COMMAND] + assert len(init_calls) == 5 + assert orchestrator.RUNNER_USER_INIT_COMMAND == ( + "sudo mkdir -p /mnt && sudo mount /dev/vdb /mnt && " + "MVM_ASSET_MIRROR=/mnt /usr/local/bin/mvm init --non-interactive " + "--binary-version 1.16.0" + ) + + +def test_runner_user_initialization_uses_the_exact_installed_cli( + orchestrator: ModuleType, + monkeypatch: pytest.MonkeyPatch, +) -> None: + calls: list[tuple[tuple[str, ...], dict[str, object]]] = [] + + def fake_mvm(*args: str, **kwargs: object) -> subprocess.CompletedProcess[str]: + calls.append((args, kwargs)) + return subprocess.CompletedProcess(args, 0, stdout="", stderr="") + + monkeypatch.setattr(orchestrator, "mvm", fake_mvm) + + orchestrator._initialize_runner_user( + "base-img-builder", + timeout=180, + capture=False, + ) + + assert calls == [ + ( + ( + "exec", + "base-img-builder", + "--user", + "runner", + "--timeout", + "10", + "--", + orchestrator.RUNNER_USER_INIT_COMMAND, + ), + {"timeout": 180, "capture": False}, + ) + ] + + def test_streamed_mvm_failure_preserves_command_and_rc_when_output_is_none( orchestrator: ModuleType, monkeypatch: pytest.MonkeyPatch, @@ -104,8 +167,6 @@ def fake_mvm(*args: str, **kwargs: object) -> subprocess.CompletedProcess[str]: orchestrator._pull_builder_assets("base-img-builder") expected_pull_commands = [ - "MVM_ASSET_MIRROR=/mnt /usr/local/bin/mvm binary pull firecracker " - "--default --force --version 1.16.0", "MVM_ASSET_MIRROR=/mnt /usr/local/bin/mvm kernel pull " "--type firecracker --version v1.15 --default", "MVM_ASSET_MIRROR=/mnt /usr/local/bin/mvm image pull alpine:3.23", @@ -116,7 +177,8 @@ def fake_mvm(*args: str, **kwargs: object) -> subprocess.CompletedProcess[str]: ] pull_calls = calls[:-1] assert [args[-1] for args, _kwargs in pull_calls] == expected_pull_commands - assert len(pull_calls) == 5 + assert len(pull_calls) == 4 + assert not any("binary pull firecracker" in args[-1] for args, _ in calls) for args, kwargs in pull_calls: assert args[:-1] == ( "exec", @@ -184,8 +246,23 @@ def fake_mvm(*args: str, **_kwargs: object) -> subprocess.CompletedProcess[str]: return subprocess.CompletedProcess(args, 0, stdout=stdout, stderr="") original_failure = RuntimeError("asset pull failed") + initialization_order: list[str] = [] + + def record_host_init(_vm_name: str) -> None: + initialization_order.append("host-init") + + def record_runner_init( + _vm_name: str, + *, + timeout: int, + capture: bool = True, + ) -> None: + assert timeout == 180 + assert capture is False + initialization_order.append("runner-init-1.16.0") def fail_asset_pull(_vm_name: str) -> None: + initialization_order.append("asset-pulls") raise original_failure monkeypatch.setattr(orchestrator, "mvm", fake_mvm) @@ -193,14 +270,20 @@ def fail_asset_pull(_vm_name: str) -> None: orchestrator, "_install_system_binary_in_runner", lambda _vm_name: None ) monkeypatch.setattr( - orchestrator, "_initialize_system_binary_in_runner", lambda _vm_name: None + orchestrator, "_initialize_system_binary_in_runner", record_host_init ) + monkeypatch.setattr(orchestrator, "_initialize_runner_user", record_runner_init) monkeypatch.setattr(orchestrator, "_pull_builder_assets", fail_asset_pull) with pytest.raises(RuntimeError) as captured: orchestrator._build_base_image("0.3.0-test", rebuild=True) assert captured.value is original_failure + assert initialization_order == [ + "host-init", + "runner-init-1.16.0", + "asset-pulls", + ] builder_removals = [ args for args in calls From 107be06ec5c8775ecf48bed0fa317af9e0a0c81d Mon Sep 17 00:00:00 2001 From: AlanD20 Date: Sun, 23 Aug 2026 14:51:19 +0200 Subject: [PATCH 0016/1457] test: require explicit host-direct consent --- scripts/run-system-tests.py | 22 ++++++ scripts/tests/test_run_system_tests.py | 95 ++++++++++++++++++++++++++ 2 files changed, 117 insertions(+) diff --git a/scripts/run-system-tests.py b/scripts/run-system-tests.py index 8e28b34e..06bd7d87 100755 --- a/scripts/run-system-tests.py +++ b/scripts/run-system-tests.py @@ -1487,6 +1487,15 @@ def print_summary(all_results: list[dict[str, Any]]) -> None: sys.exit(1) +def _selection_requests_tier3(args: argparse.Namespace) -> bool: + """Return whether any requested selector names host-direct Tier 3 work.""" + return ( + args.all + or (args.tier is not None and 3 in args.tier) + or any(domain in TIER3_DOMAINS for domain in args.domains) + ) + + def _build_parser() -> argparse.ArgumentParser: parser = argparse.ArgumentParser( description="Run mvmctl system tests with per-domain VM isolation.", @@ -1501,6 +1510,13 @@ def _build_parser() -> argparse.ArgumentParser: action="store_true", help="Run all T1 + T2 + T3 domains.", ) + parser.add_argument( + "--host-direct", + action="store_true", + help="Acknowledge that selected Tier 3 tests run directly on and may " + "mutate the outer host. Required for --all, --tier including 3, or " + "a Tier 3 domain.", + ) parser.add_argument( "--tier", type=_parse_tier, @@ -1703,6 +1719,12 @@ def main() -> None: parser = _build_parser() args = parser.parse_args() + if _selection_requests_tier3(args) and not args.host_direct: + parser.error( + "Tier 3 tests run directly on and may mutate the outer host; " + "pass --host-direct to acknowledge this" + ) + if args.candidate_version is not None and not args.rebuild: parser.error("--candidate-version requires --rebuild") diff --git a/scripts/tests/test_run_system_tests.py b/scripts/tests/test_run_system_tests.py index 7f3ed162..2152acee 100644 --- a/scripts/tests/test_run_system_tests.py +++ b/scripts/tests/test_run_system_tests.py @@ -41,6 +41,101 @@ def test_default_outer_controller_is_the_canonical_system_binary( assert module.MVM_BINARY == "/usr/local/bin/mvm" +def test_tier3_selection_detection_covers_all_request_forms( + orchestrator: ModuleType, +) -> None: + parser = orchestrator._build_parser() + selections = [ + ["--all"], + ["--tier", "3"], + ["--tier", "2,3,1"], + *[[domain] for domain in orchestrator.TIER3_DOMAINS], + ] + + for selection in selections: + args = parser.parse_args(selection) + assert orchestrator._selection_requests_tier3(args), selection + + +def test_t1_t2_and_unknown_selections_do_not_require_host_direct( + orchestrator: ModuleType, +) -> None: + parser = orchestrator._build_parser() + selections = [ + ["--tier", "1"], + ["--tier", "1,2"], + [next(iter(orchestrator.TIER1_DOMAINS))], + [next(iter(orchestrator.TIER2_DOMAINS))], + ["unknown-domain"], + ] + + for selection in selections: + args = parser.parse_args(selection) + assert not orchestrator._selection_requests_tier3(args), selection + + +def test_host_direct_help_is_an_acknowledgment_not_a_clean_host_guarantee( + orchestrator: ModuleType, +) -> None: + parser = orchestrator._build_parser() + args = parser.parse_args(["--host-direct", "--all"]) + help_text = parser.format_help().lower() + + assert args.host_direct is True + assert "acknowledge" in help_text + assert "guarantee" not in help_text + + +@pytest.mark.parametrize( + "selection", + [ + ["--all", "--rebuild", "--candidate-version", "0.3.0"], + ["--tier", "3"], + ["env"], + ], + ids=["all", "tier", "positional-domain"], +) +def test_tier3_without_host_direct_is_rejected_before_any_side_effect( + orchestrator: ModuleType, + monkeypatch: pytest.MonkeyPatch, + capsys: pytest.CaptureFixture[str], + selection: list[str], +) -> None: + calls: list[str] = [] + + def unexpected(name: str): + def fail(*_args: object, **_kwargs: object) -> None: + calls.append(name) + raise AssertionError(f"{name} must not run before host-direct consent") + + return fail + + for helper in ( + "_resolve_candidate_build_version", + "_build_mvm_binary", + "_require_distinct_candidate_controller", + "_get_mvm_version", + "run_prepare", + "ensure_shared_volume", + "ensure_test_network", + ): + monkeypatch.setattr(orchestrator, helper, unexpected(helper)) + monkeypatch.setattr( + orchestrator.sys, + "argv", + ["run-system-tests.py", *selection], + ) + + with pytest.raises(SystemExit) as captured: + orchestrator.main() + + assert captured.value.code == 2 + assert calls == [] + error = capsys.readouterr().err.lower() + assert "--host-direct" in error + assert "acknowledge" in error + + def test_guest_product_commands_use_the_exact_installed_cli( orchestrator: ModuleType, ) -> None: From d5da180db9b0d40b0611a169ae3c39ea1f16e6d7 Mon Sep 17 00:00:00 2001 From: AlanD20 Date: Sun, 23 Aug 2026 15:11:12 +0200 Subject: [PATCH 0017/1457] refactor: use volume IDs for managed disk names --- internal/infra/constants.go | 6 +- pkg/api/inputs/volume_create.go | 4 -- pkg/api/volume.go | 3 +- pkg/api/volume_test.go | 107 ++++++++++++++++++++++++++++++++ 4 files changed, 112 insertions(+), 8 deletions(-) create mode 100644 pkg/api/volume_test.go diff --git a/internal/infra/constants.go b/internal/infra/constants.go index 5d699c51..2c675ff8 100644 --- a/internal/infra/constants.go +++ b/internal/infra/constants.go @@ -635,9 +635,9 @@ func GetVolumesDir() string { return path } -// GetVolumePath returns the filesystem path for a volume with the given name and format. -func GetVolumePath(name, format string) string { - return filepath.Join(GetVolumesDir(), fmt.Sprintf("%s.%s", name, format)) +// GetVolumePath returns the canonical managed path for a volume ID and format. +func GetVolumePath(volumeID, format string) string { + return filepath.Join(GetVolumesDir(), fmt.Sprintf("%s.%s", volumeID, format)) } func GetBinDir() string { diff --git a/pkg/api/inputs/volume_create.go b/pkg/api/inputs/volume_create.go index 858dd555..397d5fde 100644 --- a/pkg/api/inputs/volume_create.go +++ b/pkg/api/inputs/volume_create.go @@ -4,7 +4,6 @@ import ( "context" "fmt" "mvmctl/internal/core/volume" - "mvmctl/internal/infra" "mvmctl/internal/lib/disk" "mvmctl/internal/lib/model" "mvmctl/internal/lib/validators" @@ -26,7 +25,6 @@ type ResolvedVolumeCreateInput struct { Name string SizeBytes int64 Format model.VolumeFormat - Path string IsReadOnly bool IsShareable bool CacheType string @@ -71,7 +69,6 @@ func (i *VolumeCreateInput) Resolve(ctx context.Context, repo volume.Repository) fmt.Sprintf("Unsupported format: %s. Use 'raw' or 'qcow2'.", format), ) } - path := infra.GetVolumePath(i.Name, string(format)) isReadOnly := false if i.ReadOnly != nil { isReadOnly = *i.ReadOnly @@ -92,7 +89,6 @@ func (i *VolumeCreateInput) Resolve(ctx context.Context, repo volume.Repository) Name: i.Name, SizeBytes: sizeBytes, Format: format, - Path: path, IsReadOnly: isReadOnly, IsShareable: isShareable, CacheType: cacheType, diff --git a/pkg/api/volume.go b/pkg/api/volume.go index 98402e40..de0c4f6c 100644 --- a/pkg/api/volume.go +++ b/pkg/api/volume.go @@ -7,6 +7,7 @@ import ( "log/slog" "mvmctl/internal/core/vm" "mvmctl/internal/core/volume" + "mvmctl/internal/infra" "mvmctl/internal/lib/crypto" "mvmctl/internal/lib/disk" "mvmctl/internal/lib/model" @@ -62,7 +63,7 @@ func (op *Operation) VolumeCreate(ctx context.Context, input inputs.VolumeCreate IsReadOnly: resolved.IsReadOnly, IsShareable: resolved.IsShareable, CacheType: cacheType, - Path: resolved.Path, + Path: infra.GetVolumePath(volumeID, string(resolved.Format)), Status: model.VolumeStatusAvailable, CreatedAt: timestamp, UpdatedAt: timestamp, diff --git a/pkg/api/volume_test.go b/pkg/api/volume_test.go new file mode 100644 index 00000000..dea55afd --- /dev/null +++ b/pkg/api/volume_test.go @@ -0,0 +1,107 @@ +package api_test + +import ( + "context" + "path/filepath" + "regexp" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "mvmctl/internal/core/volume" + "mvmctl/internal/infra/ptr" + "mvmctl/internal/lib/logging" + "mvmctl/internal/lib/system" + "mvmctl/internal/testutil" + "mvmctl/pkg/api" + "mvmctl/pkg/api/inputs" +) + +func TestVolumeCreateUsesImmutableIDForManagedPath(t *testing.T) { + cacheDir := t.TempDir() + t.Setenv("MVM_CACHE_DIR", cacheDir) + + tests := map[string]struct { + name string + format *string + wantSuffix string + pathArg int + }{ + "raw_default": { + name: "customer-backups", + wantSuffix: ".raw", + pathArg: 3, + }, + "qcow2": { + name: "database-volume", + format: ptr.Ptr("qcow2"), + wantSuffix: ".qcow2", + pathArg: 4, + }, + } + + for name, tc := range tests { + t.Run(name, func(t *testing.T) { + repo := testutil.NewVolumeRepo() + fakeRunner := &testutil.FakeRunner{} + originalRunner := system.DefaultRunner + system.DefaultRunner = fakeRunner + t.Cleanup(func() { system.DefaultRunner = originalRunner }) + + op := &api.Operation{ + Repos: api.Repos{Volume: repo}, + Services: api.Services{Volume: volume.NewService(repo)}, + AuditLog: &logging.AuditLog{}, + } + writeback := false + + created, err := op.VolumeCreate(context.Background(), inputs.VolumeCreateInput{ + Name: tc.name, Size: "1MB", Format: tc.format, Writeback: &writeback, + }) + require.NoError(t, err) + require.NotNil(t, created) + assert.Regexp(t, regexp.MustCompile(`^[0-9a-f]{64}$`), created.ID) + + wantPath := filepath.Join(cacheDir, "volumes", created.ID+tc.wantSuffix) + assert.Equal(t, wantPath, created.Path) + assert.NotContains(t, filepath.Base(created.Path), tc.name) + + stored, repoErr := repo.Get(context.Background(), created.ID) + require.NoError(t, repoErr) + require.NotNil(t, stored) + assert.Equal(t, wantPath, stored.Path) + + require.Len(t, fakeRunner.Calls, 1) + require.Greater(t, len(fakeRunner.Calls[0].Args), tc.pathArg) + assert.Equal(t, wantPath, fakeRunner.Calls[0].Args[tc.pathArg]) + }) + } +} + +func TestVolumeCreateRejectsPathLikeDisplayNamesBeforeDiskCreation(t *testing.T) { + tests := []string{"../escape", "nested/escape"} + for _, displayName := range tests { + t.Run(displayName, func(t *testing.T) { + repo := testutil.NewVolumeRepo() + fakeRunner := &testutil.FakeRunner{} + originalRunner := system.DefaultRunner + system.DefaultRunner = fakeRunner + t.Cleanup(func() { system.DefaultRunner = originalRunner }) + + op := &api.Operation{ + Repos: api.Repos{Volume: repo}, + Services: api.Services{Volume: volume.NewService(repo)}, + AuditLog: &logging.AuditLog{}, + } + writeback := false + + created, err := op.VolumeCreate(context.Background(), inputs.VolumeCreateInput{ + Name: displayName, Size: "1MB", Writeback: &writeback, + }) + require.Error(t, err) + assert.Nil(t, created) + assert.Empty(t, fakeRunner.Calls) + }) + } +} From 2a152cab323ee2aef95d32509b0a4cebb3e2026a Mon Sep 17 00:00:00 2001 From: AlanD20 Date: Sun, 23 Aug 2026 15:17:49 +0200 Subject: [PATCH 0018/1457] fix: serialize canonical release slots --- .../service/jailer/instance_authority_test.go | 46 +++++++++++++++++++ internal/service/jailer/instance_lock.go | 3 +- internal/service/jailer/instance_lock_test.go | 30 ++++++++++++ 3 files changed, 77 insertions(+), 2 deletions(-) create mode 100644 internal/service/jailer/instance_lock_test.go diff --git a/internal/service/jailer/instance_authority_test.go b/internal/service/jailer/instance_authority_test.go index 557e00fc..7af112e0 100644 --- a/internal/service/jailer/instance_authority_test.go +++ b/internal/service/jailer/instance_authority_test.go @@ -233,6 +233,52 @@ func TestInstanceAuthorityReleaseReferenceLeaseBlocksLaunch(t *testing.T) { require.NoError(t, releaseLease.Release(context.Background())) } +func TestInstanceAuthorityReleaseLeaseSerializesCanonicalStoreSlot(t *testing.T) { + t.Parallel() + + authority, _ := newTestInstanceAuthority(t) + held, err := authority.LockUnreferencedRelease( + context.Background(), + testLaunchRegistration().release, + ) + require.NoError(t, err) + t.Cleanup(func() { require.NoError(t, held.Release(context.Background())) }) + + type result struct { + lease *releaseLease + err error + } + started := make(chan struct{}) + resultCh := make(chan result, 1) + waitCtx, cancel := context.WithTimeout(context.Background(), time.Second) + defer cancel() + go func() { + close(started) + lease, lockErr := authority.LockUnreferencedRelease(waitCtx, testAlternateReleaseIdentity()) + resultCh <- result{lease: lease, err: lockErr} + }() + <-started + + select { + case early := <-resultCh: + if early.lease != nil { + require.NoError(t, early.lease.Release(context.Background())) + } + require.Failf(t, "release lock returned early", "error: %v", early.err) + case <-time.After(30 * time.Millisecond): + } + + require.NoError(t, held.Release(context.Background())) + select { + case acquired := <-resultCh: + require.NoError(t, acquired.err) + require.NotNil(t, acquired.lease) + require.NoError(t, acquired.lease.Release(context.Background())) + case <-waitCtx.Done(): + require.Failf(t, "release lock remained blocked", "error: %v", waitCtx.Err()) + } +} + func TestInstanceAuthorityReleaseLeaseRejectsCorruptAuthorityRecord(t *testing.T) { t.Parallel() diff --git a/internal/service/jailer/instance_lock.go b/internal/service/jailer/instance_lock.go index 8b87b173..d19a80f5 100644 --- a/internal/service/jailer/instance_lock.go +++ b/internal/service/jailer/instance_lock.go @@ -155,8 +155,7 @@ func verifyInstanceRegularFile( } func releaseLockName(release releaseIdentity) string { - canonical := release.version + "\x00" + release.architecture + "\x00" + - release.firecrackerSHA256 + "\x00" + release.jailerSHA256 + canonical := release.version + "\x00" + release.architecture digest := sha256.Sum256([]byte(canonical)) return hex.EncodeToString(digest[:]) + ".lock" } diff --git a/internal/service/jailer/instance_lock_test.go b/internal/service/jailer/instance_lock_test.go new file mode 100644 index 00000000..ee212e3e --- /dev/null +++ b/internal/service/jailer/instance_lock_test.go @@ -0,0 +1,30 @@ +package jailer + +import ( + "strings" + "testing" + + "github.com/stretchr/testify/assert" +) + +func TestReleaseLockNameUsesCanonicalStoreSlot(t *testing.T) { + t.Parallel() + + release := testLaunchRegistration().release + sameSlot := testAlternateReleaseIdentity() + differentVersion := release + differentVersion.version = "1.17.0" + differentArchitecture := release + differentArchitecture.architecture = "aarch64" + + assert.Equal(t, releaseLockName(release), releaseLockName(sameSlot)) + assert.NotEqual(t, releaseLockName(release), releaseLockName(differentVersion)) + assert.NotEqual(t, releaseLockName(release), releaseLockName(differentArchitecture)) +} + +func testAlternateReleaseIdentity() releaseIdentity { + release := testLaunchRegistration().release + release.firecrackerSHA256 = strings.Repeat("a", 64) + release.jailerSHA256 = strings.Repeat("b", 64) + return release +} From cce6fc4cc58012288879dcea75389d6aa8731ee8 Mon Sep 17 00:00:00 2001 From: AlanD20 Date: Sun, 23 Aug 2026 15:17:59 +0200 Subject: [PATCH 0019/1457] test: enforce release binary identity --- scripts/run-system-tests.py | 76 ++++- scripts/system_test_release.py | 329 +++++++++++++++++++++ scripts/tests/test_run_system_tests.py | 180 +++++++++++- scripts/tests/test_system_test_release.py | 343 ++++++++++++++++++++++ 4 files changed, 920 insertions(+), 8 deletions(-) create mode 100644 scripts/system_test_release.py create mode 100644 scripts/tests/test_system_test_release.py diff --git a/scripts/run-system-tests.py b/scripts/run-system-tests.py index 06bd7d87..25ae52cc 100755 --- a/scripts/run-system-tests.py +++ b/scripts/run-system-tests.py @@ -8,7 +8,7 @@ # Run all domains (T1 + T2 + T3) MVM_ASSET_MIRROR=~/.cache/mvm-asset-mirror \\ - python3 scripts/run-system-tests.py --all + python3 scripts/run-system-tests.py --all --host-direct # Run specific domains python3 scripts/run-system-tests.py cli network vm_nested_virt @@ -16,17 +16,18 @@ # Run specific tiers (comma-separated, executed in the given order) python3 scripts/run-system-tests.py --tier 1 python3 scripts/run-system-tests.py --tier 2,1 - python3 scripts/run-system-tests.py --tier 1,3 - python3 scripts/run-system-tests.py --tier 3,2,1 + python3 scripts/run-system-tests.py --tier 1,3 --host-direct + python3 scripts/run-system-tests.py --tier 3,2,1 --host-direct - # Rebuild shared assets and (optionally) run tests - python3 scripts/run-system-tests.py --rebuild --all + # Rebuild and qualify an explicit release candidate + python3 scripts/run-system-tests.py --release-qualification --all \\ + --host-direct --rebuild --candidate-version 0.3.0 python3 scripts/run-system-tests.py --volume --image --prepare python3 scripts/run-system-tests.py --volume python3 scripts/run-system-tests.py --image # Limit parallel workers (default: 4) - python3 scripts/run-system-tests.py --all --workers 2 + python3 scripts/run-system-tests.py --all --host-direct --workers 2 """ from __future__ import annotations @@ -47,6 +48,11 @@ from pathlib import Path from typing import Any +from system_test_release import ( + validate_release_build_paths, + verify_release_binary_identity, +) + # ============================================================================ # Configuration # ============================================================================ @@ -87,7 +93,6 @@ # SSH key for provisioning runner user in builder & test VMs BUILDER_KEY_NAME = "builder-key" - _SEMVER_CORE_NUMBER = r"(?:0|[1-9][0-9]*)" _SEMVER_PRERELEASE_ID = ( r"(?:0|[1-9][0-9]*|[0-9A-Za-z-]*[A-Za-z-][0-9A-Za-z-]*)" @@ -104,6 +109,7 @@ class _UnsafeVolumeBackingPath(RuntimeError): """The volume record points outside the one runner-managed backing path.""" + # Tier 1: shared volume, host-level CLI operations (no nested virt needed) TIER1_DOMAINS: dict[str, list[str]] = { "system_install": ["tests/system/host/test_system_install.py"], @@ -1496,6 +1502,30 @@ def _selection_requests_tier3(args: argparse.Namespace) -> bool: ) +def _validate_release_qualification_args( + parser: argparse.ArgumentParser, + args: argparse.Namespace, +) -> None: + """Reject incomplete or narrowed release qualification before any work.""" + if not args.release_qualification: + return + valid = ( + args.all + and args.host_direct + and args.rebuild + and args.candidate_version is not None + and not args.domains + and args.tier is None + and not args.skip_volume_check + ) + if not valid: + parser.error( + "--release-qualification requires --all --host-direct --rebuild " + "and an explicit --candidate-version; positional domains, --tier, " + "and --skip-volume-check are not allowed" + ) + + def _build_parser() -> argparse.ArgumentParser: parser = argparse.ArgumentParser( description="Run mvmctl system tests with per-domain VM isolation.", @@ -1517,6 +1547,13 @@ def _build_parser() -> argparse.ArgumentParser: "mutate the outer host. Required for --all, --tier including 3, or " "a Tier 3 domain.", ) + parser.add_argument( + "--release-qualification", + action="store_true", + help="Run the full release binary identity gate before host-direct " + "qualification. Requires --all --host-direct --rebuild and an " + "explicit --candidate-version.", + ) parser.add_argument( "--tier", type=_parse_tier, @@ -1719,6 +1756,8 @@ def main() -> None: parser = _build_parser() args = parser.parse_args() + _validate_release_qualification_args(parser, args) + if _selection_requests_tier3(args) and not args.host_direct: parser.error( "Tier 3 tests run directly on and may mutate the outer host; " @@ -1735,6 +1774,17 @@ def main() -> None: parser.print_help() sys.exit(0) + if args.release_qualification: + try: + validate_release_build_paths( + controller_command=MVM_BINARY, + configured_candidate=MVM_CANDIDATE_CONFIGURED, + candidate=Path(MVM_CANDIDATE_BINARY), + ) + except RuntimeError as exc: + log(f"ERROR: {exc}") + sys.exit(1) + # Handle --rebuild: build the distinct release candidate before anything else. if args.rebuild: try: @@ -1746,6 +1796,18 @@ def main() -> None: log(f"ERROR: {exc}") sys.exit(1) + if args.release_qualification: + try: + verify_release_binary_identity( + controller_command=MVM_BINARY, + configured_candidate=MVM_CANDIDATE_CONFIGURED, + candidate=Path(MVM_CANDIDATE_BINARY), + requested_version=args.candidate_version, + ) + except RuntimeError as exc: + log(f"ERROR: {exc}") + sys.exit(1) + # Validate MVM_BINARY exists binary = shlex.split(MVM_BINARY)[0] if not shutil.which(binary) and not Path(binary).is_file(): diff --git a/scripts/system_test_release.py b/scripts/system_test_release.py new file mode 100644 index 00000000..c7c2b699 --- /dev/null +++ b/scripts/system_test_release.py @@ -0,0 +1,329 @@ +"""Fail-closed release binary identity qualification for the system-test runner.""" + +from __future__ import annotations + +import hashlib +import os +import re +import stat +import subprocess +import tempfile +from dataclasses import dataclass +from pathlib import Path + +_CANONICAL_CONTROLLER = Path("/usr/local/bin/mvm") +_SEMVER_CORE_NUMBER = r"(?:0|[1-9][0-9]*)" +_SEMVER_PRERELEASE_ID = ( + r"(?:0|[1-9][0-9]*|[0-9A-Za-z-]*[A-Za-z-][0-9A-Za-z-]*)" +) +_SEMVER_BUILD_ID = r"[0-9A-Za-z-]+" +_STRICT_SEMVER_PATTERN = re.compile( + rf"^{_SEMVER_CORE_NUMBER}\.{_SEMVER_CORE_NUMBER}\." + rf"{_SEMVER_CORE_NUMBER}" + rf"(?:-{_SEMVER_PRERELEASE_ID}(?:\.{_SEMVER_PRERELEASE_ID})*)?" + rf"(?:\+{_SEMVER_BUILD_ID}(?:\.{_SEMVER_BUILD_ID})*)?$" +) + + +@dataclass(frozen=True) +class _PinnedBinary: + path: Path + descriptor: int + metadata: os.stat_result + controller: bool + + +@dataclass(frozen=True) +class _BinaryEvidence: + path: Path + device: int + inode: int + sha256: str + version: str + + +def _controller_path(controller_command: str) -> Path: + if controller_command != str(_CANONICAL_CONTROLLER): + raise RuntimeError( + "release qualification requires MVM_BINARY to be exactly " + f"{_CANONICAL_CONTROLLER}" + ) + return _CANONICAL_CONTROLLER + + +def _candidate_path(configured_candidate: Path, candidate: Path) -> Path: + try: + configured = configured_candidate.lstat() + except FileNotFoundError: + configured = None + except OSError as exc: + raise RuntimeError( + "cannot inspect configured release candidate path " + f"{configured_candidate}: {exc}" + ) from exc + if configured is not None and stat.S_ISLNK(configured.st_mode): + raise RuntimeError( + "release candidate path must not be a symlink: " + f"{configured_candidate}" + ) + if not candidate.is_absolute() or candidate != candidate.resolve(): + raise RuntimeError(f"release candidate path is not canonical: {candidate}") + return candidate + + +def validate_release_build_paths( + *, + controller_command: str, + configured_candidate: Path, + candidate: Path, +) -> None: + """Reject controller commands and candidate path aliases before rebuilding.""" + controller = _controller_path(controller_command) + resolved_candidate = _candidate_path(configured_candidate, candidate) + if resolved_candidate == controller: + raise RuntimeError( + "release candidate and controller have the same canonical path" + ) + try: + controller_metadata = controller.lstat() + except OSError as exc: + raise RuntimeError( + f"cannot inspect release controller {controller}: {exc}" + ) from exc + _validate_binary_metadata(controller, controller_metadata, controller=True) + _validate_controller_ancestors(controller) + try: + candidate_metadata = resolved_candidate.lstat() + except FileNotFoundError: + return + except OSError as exc: + raise RuntimeError( + f"cannot inspect existing release candidate {resolved_candidate}: {exc}" + ) from exc + if (candidate_metadata.st_dev, candidate_metadata.st_ino) == ( + controller_metadata.st_dev, + controller_metadata.st_ino, + ): + raise RuntimeError("release candidate and controller share one inode") + + +def _validate_binary_metadata( + path: Path, + metadata: os.stat_result, + *, + controller: bool, +) -> None: + if stat.S_ISLNK(metadata.st_mode): + raise RuntimeError(f"release binary must not be a symlink: {path}") + if not stat.S_ISREG(metadata.st_mode): + raise RuntimeError(f"release binary is not a regular file: {path}") + mode = stat.S_IMODE(metadata.st_mode) + if mode & 0o111 == 0: + raise RuntimeError(f"release binary is not executable: {path}") + if controller and (metadata.st_uid, metadata.st_gid) != (0, 0): + raise RuntimeError( + "release controller must be owned root:root: " + f"{path} is {metadata.st_uid}:{metadata.st_gid}" + ) + if controller and mode != 0o755: + raise RuntimeError( + f"release controller must have exact mode 0755: {path} is {mode:04o}" + ) + + +def _validate_ancestor_metadata(path: Path, metadata: os.stat_result) -> None: + if stat.S_ISLNK(metadata.st_mode): + raise RuntimeError(f"release controller ancestor is a symlink: {path}") + if not stat.S_ISDIR(metadata.st_mode): + raise RuntimeError(f"release controller ancestor is not a directory: {path}") + if metadata.st_uid != 0: + raise RuntimeError(f"release controller ancestor is not root-owned: {path}") + if stat.S_IMODE(metadata.st_mode) & 0o022: + raise RuntimeError( + f"release controller ancestor is group/world writable: {path}" + ) + + +def _validate_controller_ancestors(path: Path) -> None: + for ancestor in reversed(path.parents): + try: + metadata = ancestor.lstat() + except OSError as exc: + raise RuntimeError( + f"cannot inspect release controller ancestor {ancestor}: {exc}" + ) from exc + _validate_ancestor_metadata(ancestor, metadata) + + +def _open_pinned_binary(path: Path, *, controller: bool) -> _PinnedBinary: + try: + before = path.lstat() + except OSError as exc: + raise RuntimeError(f"cannot inspect release binary {path}: {exc}") from exc + _validate_binary_metadata(path, before, controller=controller) + + try: + descriptor = os.open(path, os.O_RDONLY | os.O_CLOEXEC | os.O_NOFOLLOW) + except OSError as exc: + raise RuntimeError(f"cannot safely open release binary {path}: {exc}") from exc + try: + pinned = os.fstat(descriptor) + _validate_binary_metadata(path, pinned, controller=controller) + if (before.st_dev, before.st_ino) != (pinned.st_dev, pinned.st_ino): + raise RuntimeError(f"release binary changed while opening: {path}") + return _PinnedBinary(path, descriptor, pinned, controller) + except BaseException: + os.close(descriptor) + raise + + +def _sha256(descriptor: int) -> str: + with os.fdopen(os.dup(descriptor), "rb") as stream: + return hashlib.file_digest(stream, "sha256").hexdigest() + + +def _strict_version(label: str, version: str) -> str: + if _STRICT_SEMVER_PATTERN.fullmatch(version) is None: + raise RuntimeError(f"{label} is not a strict semantic version: {version!r}") + return version + + +def _parse_version_output(label: str, output: str) -> str: + parts = output.strip().split() + if len(parts) != 2 or parts[0] != "mvm": + raise RuntimeError( + f"{label} did not report an exact strict semantic version: {output!r}" + ) + try: + return _strict_version(f"{label} version", parts[1]) + except RuntimeError as exc: + raise RuntimeError( + f"{label} did not report a strict semantic version: {output!r}" + ) from exc + + +def _probe_version(descriptor: int, label: str) -> str: + with tempfile.TemporaryDirectory(prefix=f"mvm-{label}-version-") as root: + isolation_root = Path(root) + cache_dir = isolation_root / "cache" + config_dir = isolation_root / "config" + temp_dir = isolation_root / "tmp" + cache_dir.mkdir() + config_dir.mkdir() + temp_dir.mkdir() + env = { + "HOME": str(isolation_root), + "PATH": "/usr/bin:/bin", + "LANG": "C", + "LC_ALL": "C", + "NO_COLOR": "1", + "MVM_CACHE_DIR": str(cache_dir), + "MVM_CONFIG_DIR": str(config_dir), + "MVM_TEMP_DIR": str(temp_dir), + } + result = subprocess.run( + [f"/proc/self/fd/{descriptor}", "--version"], + capture_output=True, + text=True, + timeout=10, + env=env, + pass_fds=(descriptor,), + ) + if result.returncode != 0: + detail = result.stderr.strip() or f"rc={result.returncode}" + raise RuntimeError(f"{label} version probe failed: {detail}") + return _parse_version_output(label, result.stdout) + + +def _evidence(binary: _PinnedBinary, label: str) -> _BinaryEvidence: + return _BinaryEvidence( + path=binary.path, + device=binary.metadata.st_dev, + inode=binary.metadata.st_ino, + sha256=_sha256(binary.descriptor), + version=_probe_version(binary.descriptor, label), + ) + + +def _validate_matching_evidence( + controller: _BinaryEvidence, + candidate: _BinaryEvidence, + requested_version: str, +) -> None: + controller_version = _strict_version("controller version", controller.version) + candidate_version = _strict_version("candidate version", candidate.version) + requested = _strict_version("requested version", requested_version) + if controller.path == candidate.path: + raise RuntimeError( + "release candidate and controller have the same canonical path" + ) + if (controller.device, controller.inode) == ( + candidate.device, + candidate.inode, + ): + raise RuntimeError("release candidate and controller share one inode") + if controller.sha256 != candidate.sha256: + raise RuntimeError("release candidate and controller SHA-256 differ") + if controller_version != candidate_version: + raise RuntimeError("release candidate and controller versions differ") + if requested != candidate_version: + raise RuntimeError( + "requested version does not match release binaries: " + f"requested {requested}, reported {candidate_version}" + ) + + +def _require_path_unchanged(binary: _PinnedBinary) -> None: + try: + current = binary.path.lstat() + except OSError as exc: + raise RuntimeError( + f"release binary path changed during validation: {binary.path}: {exc}" + ) from exc + _validate_binary_metadata( + binary.path, + current, + controller=binary.controller, + ) + if (current.st_dev, current.st_ino) != ( + binary.metadata.st_dev, + binary.metadata.st_ino, + ): + raise RuntimeError( + f"release binary path changed during validation: {binary.path}" + ) + + +def verify_release_binary_identity( + *, + controller_command: str, + configured_candidate: Path, + candidate: Path, + requested_version: str, +) -> None: + """Verify installed and candidate binaries are distinct, trusted, and equal.""" + controller_path = _controller_path(controller_command) + candidate_path = _candidate_path(configured_candidate, candidate) + if controller_path == candidate_path: + raise RuntimeError( + "release candidate and controller have the same canonical path" + ) + _validate_controller_ancestors(controller_path) + + controller: _PinnedBinary | None = None + release_candidate: _PinnedBinary | None = None + try: + controller = _open_pinned_binary(controller_path, controller=True) + release_candidate = _open_pinned_binary(candidate_path, controller=False) + _validate_matching_evidence( + _evidence(controller, "controller"), + _evidence(release_candidate, "candidate"), + requested_version, + ) + _require_path_unchanged(controller) + _require_path_unchanged(release_candidate) + finally: + if release_candidate is not None: + os.close(release_candidate.descriptor) + if controller is not None: + os.close(controller.descriptor) diff --git a/scripts/tests/test_run_system_tests.py b/scripts/tests/test_run_system_tests.py index 2152acee..8a098a65 100644 --- a/scripts/tests/test_run_system_tests.py +++ b/scripts/tests/test_run_system_tests.py @@ -14,8 +14,9 @@ @pytest.fixture -def orchestrator() -> ModuleType: +def orchestrator(monkeypatch: pytest.MonkeyPatch) -> ModuleType: script = Path(__file__).parents[1] / "run-system-tests.py" + monkeypatch.syspath_prepend(str(script.parent)) module_name = "mvmctl_run_system_tests" spec = importlib.util.spec_from_file_location(module_name, script) assert spec is not None and spec.loader is not None @@ -30,6 +31,7 @@ def test_default_outer_controller_is_the_canonical_system_binary( ) -> None: monkeypatch.delenv("MVM_BINARY", raising=False) script = Path(__file__).parents[1] / "run-system-tests.py" + monkeypatch.syspath_prepend(str(script.parent)) module_name = "mvmctl_run_system_tests_default_controller" spec = importlib.util.spec_from_file_location(module_name, script) assert spec is not None and spec.loader is not None @@ -136,6 +138,182 @@ def fail(*_args: object, **_kwargs: object) -> None: assert "acknowledge" in error +@pytest.mark.parametrize( + "selection", + [ + ["--release-qualification", "--host-direct", "--rebuild", "--candidate-version", "0.3.0"], + ["--release-qualification", "--all", "--rebuild", "--candidate-version", "0.3.0"], + ["--release-qualification", "--all", "--host-direct", "--candidate-version", "0.3.0"], + ["--release-qualification", "--all", "--host-direct", "--rebuild"], + [ + "--release-qualification", + "--all", + "--host-direct", + "--rebuild", + "--candidate-version", + "0.3.0", + "env", + ], + [ + "--release-qualification", + "--all", + "--host-direct", + "--rebuild", + "--candidate-version", + "0.3.0", + "--tier", + "1,2,3", + ], + [ + "--release-qualification", + "--all", + "--host-direct", + "--rebuild", + "--candidate-version", + "0.3.0", + "--skip-volume-check", + ], + ], + ids=[ + "missing-all", + "missing-host-direct", + "missing-rebuild", + "missing-version", + "positional-domain", + "tier-filter", + "skip-volume-check", + ], +) +def test_invalid_release_qualification_is_rejected_before_any_probe_or_mutation( + orchestrator: ModuleType, + monkeypatch: pytest.MonkeyPatch, + capsys: pytest.CaptureFixture[str], + selection: list[str], +) -> None: + calls: list[str] = [] + + def unexpected(name: str): + def fail(*_args: object, **_kwargs: object) -> None: + calls.append(name) + raise AssertionError(f"{name} must not run for an invalid release mode") + + return fail + + for helper in ( + "validate_release_build_paths", + "_resolve_candidate_build_version", + "_build_mvm_binary", + "verify_release_binary_identity", + "_require_distinct_candidate_controller", + "_get_mvm_version", + "run_prepare", + "ensure_shared_volume", + "ensure_test_network", + ): + monkeypatch.setattr(orchestrator, helper, unexpected(helper), raising=False) + monkeypatch.setattr( + orchestrator.sys, + "argv", + ["run-system-tests.py", *selection], + ) + + with pytest.raises(SystemExit) as captured: + orchestrator.main() + + assert captured.value.code == 2 + assert calls == [] + assert "--release-qualification requires" in capsys.readouterr().err + + +def test_release_qualification_parser_accepts_only_the_full_acknowledged_gate( + orchestrator: ModuleType, +) -> None: + parser = orchestrator._build_parser() + args = parser.parse_args( + [ + "--release-qualification", + "--all", + "--host-direct", + "--rebuild", + "--candidate-version", + "0.3.0-rc.1", + ] + ) + + orchestrator._validate_release_qualification_args(parser, args) + + assert args.release_qualification is True + assert args.candidate_version == "0.3.0-rc.1" + assert "release binary identity" in parser.format_help().lower() + + +def test_release_identity_runs_after_build_and_before_host_resource_mutation( + orchestrator: ModuleType, + monkeypatch: pytest.MonkeyPatch, + capsys: pytest.CaptureFixture[str], +) -> None: + events: list[str] = [] + + monkeypatch.setattr( + orchestrator.sys, + "argv", + [ + "run-system-tests.py", + "--release-qualification", + "--all", + "--host-direct", + "--rebuild", + "--candidate-version", + "0.3.0", + ], + ) + monkeypatch.setattr( + orchestrator, + "validate_release_build_paths", + lambda **_kwargs: events.append("path-configuration"), + raising=False, + ) + monkeypatch.setattr( + orchestrator, + "_resolve_candidate_build_version", + lambda version: events.append(f"resolve-{version}") or version, + ) + monkeypatch.setattr( + orchestrator, + "_build_mvm_binary", + lambda version: events.append(f"build-{version}"), + ) + + def fail_identity(**kwargs: object) -> None: + events.append(f"identity-{kwargs['requested_version']}") + raise RuntimeError("release identity mismatch") + + monkeypatch.setattr( + orchestrator, + "verify_release_binary_identity", + fail_identity, + raising=False, + ) + + def unexpected_mutation(*_args: object, **_kwargs: object) -> None: + raise AssertionError("host/resource mutation must follow identity validation") + + for helper in ("run_prepare", "ensure_shared_volume", "ensure_test_network"): + monkeypatch.setattr(orchestrator, helper, unexpected_mutation) + + with pytest.raises(SystemExit) as captured: + orchestrator.main() + + assert captured.value.code == 1 + assert events == [ + "path-configuration", + "resolve-0.3.0", + "build-0.3.0", + "identity-0.3.0", + ] + assert "release identity mismatch" in capsys.readouterr().out + + def test_guest_product_commands_use_the_exact_installed_cli( orchestrator: ModuleType, ) -> None: diff --git a/scripts/tests/test_system_test_release.py b/scripts/tests/test_system_test_release.py new file mode 100644 index 00000000..3a12daea --- /dev/null +++ b/scripts/tests/test_system_test_release.py @@ -0,0 +1,343 @@ +from __future__ import annotations + +import importlib.util +import os +import stat +import subprocess +import sys +from pathlib import Path +from types import ModuleType + +import pytest + + +@pytest.fixture +def release() -> ModuleType: + script = Path(__file__).parents[1] / "system_test_release.py" + module_name = "mvmctl_system_test_release" + spec = importlib.util.spec_from_file_location(module_name, script) + assert spec is not None and spec.loader is not None + module = importlib.util.module_from_spec(spec) + sys.modules[module_name] = module + spec.loader.exec_module(module) + return module + + +def _stat_result( + mode: int, + *, + uid: int = 0, + gid: int = 0, + device: int = 1, + inode: int = 1, +) -> os.stat_result: + return os.stat_result((mode, inode, device, 1, uid, gid, 1, 0, 0, 0)) + + +@pytest.mark.parametrize( + "configured", + [ + "mvm", + "/usr/local/bin/mvm --debug", + "env /usr/local/bin/mvm", + "/usr/local/bin/../bin/mvm", + "/tmp/mvm-wrapper", + "/usr/local/bin/mvm ", + ], +) +def test_release_controller_rejects_commands_wrappers_and_path_aliases( + release: ModuleType, + configured: str, +) -> None: + with pytest.raises(RuntimeError, match="exactly /usr/local/bin/mvm"): + release.validate_release_build_paths( + controller_command=configured, + configured_candidate=Path("/repo/dist/mvm"), + candidate=Path("/repo/dist/mvm"), + ) + + +@pytest.mark.parametrize( + ("metadata", "controller", "message"), + [ + (_stat_result(stat.S_IFLNK | 0o777), True, "symlink"), + (_stat_result(stat.S_IFDIR | 0o755), True, "regular file"), + (_stat_result(stat.S_IFREG | 0o755, uid=1), True, "root:root"), + (_stat_result(stat.S_IFREG | 0o755, gid=1), True, "root:root"), + (_stat_result(stat.S_IFREG | 0o775), True, "mode 0755"), + (_stat_result(stat.S_IFLNK | 0o777), False, "symlink"), + (_stat_result(stat.S_IFREG | 0o644), False, "executable"), + ], +) +def test_release_binary_metadata_is_fail_closed( + release: ModuleType, + metadata: os.stat_result, + controller: bool, + message: str, +) -> None: + with pytest.raises(RuntimeError, match=message): + release._validate_binary_metadata( + Path("/release/mvm"), + metadata, + controller=controller, + ) + + +def test_release_binary_metadata_accepts_the_required_contract( + release: ModuleType, +) -> None: + release._validate_binary_metadata( + Path("/usr/local/bin/mvm"), + _stat_result(stat.S_IFREG | 0o755), + controller=True, + ) + release._validate_binary_metadata( + Path("/repo/dist/mvm"), + _stat_result(stat.S_IFREG | 0o750, uid=1000, gid=1000), + controller=False, + ) + + +def test_release_candidate_rejects_a_configured_symlink( + release: ModuleType, + tmp_path: Path, +) -> None: + target = tmp_path / "candidate-target" + target.write_bytes(b"candidate") + configured = tmp_path / "candidate" + configured.symlink_to(target) + + with pytest.raises(RuntimeError, match="candidate path must not be a symlink"): + release.validate_release_build_paths( + controller_command="/usr/local/bin/mvm", + configured_candidate=configured, + candidate=target, + ) + + +def test_release_build_paths_reject_the_controller_as_candidate( + release: ModuleType, +) -> None: + controller = Path("/usr/local/bin/mvm") + + with pytest.raises(RuntimeError, match="same canonical path"): + release.validate_release_build_paths( + controller_command=str(controller), + configured_candidate=controller, + candidate=controller, + ) + + +def test_release_build_paths_reject_an_existing_candidate_inode_alias( + release: ModuleType, + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, +) -> None: + controller = tmp_path / "controller" + candidate = tmp_path / "candidate" + controller.write_bytes(b"mvm") + os.link(controller, candidate) + monkeypatch.setattr(release, "_CANONICAL_CONTROLLER", controller) + monkeypatch.setattr( + release, + "_validate_binary_metadata", + lambda *_args, **_kwargs: None, + ) + monkeypatch.setattr( + release, + "_validate_controller_ancestors", + lambda _path: None, + ) + + with pytest.raises(RuntimeError, match="share one inode"): + release.validate_release_build_paths( + controller_command=str(controller), + configured_candidate=candidate, + candidate=candidate, + ) + + +def test_release_build_paths_reject_a_symlink_controller_before_build( + release: ModuleType, + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, +) -> None: + target = tmp_path / "controller-target" + target.write_bytes(b"mvm") + controller = tmp_path / "controller" + controller.symlink_to(target) + candidate = tmp_path / "not-built-yet" + monkeypatch.setattr(release, "_CANONICAL_CONTROLLER", controller) + + with pytest.raises(RuntimeError, match="must not be a symlink"): + release.validate_release_build_paths( + controller_command=str(controller), + configured_candidate=candidate, + candidate=candidate, + ) + + +@pytest.mark.parametrize( + ("metadata", "message"), + [ + (_stat_result(stat.S_IFLNK | 0o777), "symlink"), + (_stat_result(stat.S_IFREG | 0o755), "directory"), + (_stat_result(stat.S_IFDIR | 0o755, uid=1), "root-owned"), + (_stat_result(stat.S_IFDIR | 0o775), "writable"), + (_stat_result(stat.S_IFDIR | 0o757), "writable"), + ], +) +def test_release_controller_ancestors_must_be_trusted( + release: ModuleType, + metadata: os.stat_result, + message: str, +) -> None: + with pytest.raises(RuntimeError, match=message): + release._validate_ancestor_metadata(Path("/usr/local/bin"), metadata) + + +def test_release_binary_evidence_accepts_only_matching_distinct_identity( + release: ModuleType, +) -> None: + evidence_type = release._BinaryEvidence + controller = evidence_type( + path=Path("/usr/local/bin/mvm"), + device=1, + inode=10, + sha256="a" * 64, + version="0.3.0-rc.1", + ) + candidate = evidence_type( + path=Path("/repo/dist/mvm"), + device=1, + inode=11, + sha256="a" * 64, + version="0.3.0-rc.1", + ) + + release._validate_matching_evidence(controller, candidate, "0.3.0-rc.1") + + +@pytest.mark.parametrize( + ("candidate_changes", "requested_version", "message"), + [ + ({"path": Path("/usr/local/bin/mvm")}, "0.3.0", "canonical path"), + ({"device": 1, "inode": 10}, "0.3.0", "inode"), + ({"sha256": "b" * 64}, "0.3.0", "SHA-256"), + ({"version": "0.3.1"}, "0.3.0", "versions differ"), + ({}, "0.3.1", "requested version"), + ({"version": "v0.3.0"}, "0.3.0", "strict semantic version"), + ], +) +def test_release_binary_evidence_rejects_alias_or_identity_mismatch( + release: ModuleType, + candidate_changes: dict[str, object], + requested_version: str, + message: str, +) -> None: + evidence_type = release._BinaryEvidence + controller_values: dict[str, object] = { + "path": Path("/usr/local/bin/mvm"), + "device": 1, + "inode": 10, + "sha256": "a" * 64, + "version": "0.3.0", + } + candidate_values: dict[str, object] = { + "path": Path("/repo/dist/mvm"), + "device": 1, + "inode": 11, + "sha256": "a" * 64, + "version": "0.3.0", + } + candidate_values.update(candidate_changes) + + with pytest.raises(RuntimeError, match=message): + release._validate_matching_evidence( + evidence_type(**controller_values), + evidence_type(**candidate_values), + requested_version, + ) + + +@pytest.mark.parametrize( + "output", + [ + "mvm v0.3.0\n", + "mvm 01.2.3\n", + "mvm 0.3\n", + "mvm 0.3.0 extra\n", + "0.3.0\n", + "", + ], +) +def test_release_version_probe_requires_exact_strict_semver_output( + release: ModuleType, + output: str, +) -> None: + with pytest.raises(RuntimeError, match="strict semantic version"): + release._parse_version_output("candidate", output) + + +def test_release_version_probe_executes_only_the_pinned_descriptor( + release: ModuleType, + monkeypatch: pytest.MonkeyPatch, +) -> None: + calls: list[tuple[list[str], dict[str, object]]] = [] + monkeypatch.setenv("LD_PRELOAD", "/tmp/hostile.so") + monkeypatch.setenv("MVM_RELEASE_HOSTILE_SENTINEL", "must-not-leak") + + def fake_run( + command: list[str], + **kwargs: object, + ) -> subprocess.CompletedProcess[str]: + calls.append((command, kwargs)) + return subprocess.CompletedProcess( + command, + 0, + stdout="mvm 0.3.0-rc.1\n", + stderr="", + ) + + monkeypatch.setattr(release.subprocess, "run", fake_run) + + version = release._probe_version(73, "controller") + + assert version == "0.3.0-rc.1" + assert len(calls) == 1 + command, kwargs = calls[0] + assert command == ["/proc/self/fd/73", "--version"] + assert kwargs["pass_fds"] == (73,) + assert kwargs["capture_output"] is True + assert kwargs["text"] is True + assert kwargs["timeout"] == 10 + env = kwargs["env"] + assert isinstance(env, dict) + assert env["PATH"] == "/usr/bin:/bin" + assert env["LANG"] == "C" + assert env["LC_ALL"] == "C" + assert "LD_PRELOAD" not in env + assert "MVM_RELEASE_HOSTILE_SENTINEL" not in env + for variable in ("MVM_CACHE_DIR", "MVM_CONFIG_DIR", "MVM_TEMP_DIR"): + assert "mvm-controller-version-" in env[variable] + + +def test_release_path_recheck_rejects_metadata_downgrade( + release: ModuleType, + monkeypatch: pytest.MonkeyPatch, +) -> None: + path = Path("/usr/local/bin/mvm") + pinned = release._PinnedBinary( + path=path, + descriptor=73, + metadata=_stat_result(stat.S_IFREG | 0o755, inode=10), + controller=True, + ) + monkeypatch.setattr( + Path, + "lstat", + lambda _path: _stat_result(stat.S_IFREG | 0o775, inode=10), + ) + + with pytest.raises(RuntimeError, match="mode 0755"): + release._require_path_unchanged(pinned) From a0cc471e46cd301064cc6af75afa92eb6f636820 Mon Sep 17 00:00:00 2001 From: AlanD20 Date: Sun, 23 Aug 2026 15:21:40 +0200 Subject: [PATCH 0020/1457] test: make release lock contention deterministic --- .../service/jailer/instance_authority_test.go | 37 +++++++++++++------ 1 file changed, 26 insertions(+), 11 deletions(-) diff --git a/internal/service/jailer/instance_authority_test.go b/internal/service/jailer/instance_authority_test.go index 7af112e0..27f0afc4 100644 --- a/internal/service/jailer/instance_authority_test.go +++ b/internal/service/jailer/instance_authority_test.go @@ -236,39 +236,54 @@ func TestInstanceAuthorityReleaseReferenceLeaseBlocksLaunch(t *testing.T) { func TestInstanceAuthorityReleaseLeaseSerializesCanonicalStoreSlot(t *testing.T) { t.Parallel() - authority, _ := newTestInstanceAuthority(t) + root := prepareInstanceAuthorityTestRoot(t) + deps := realInstanceAuthorityDeps() + realWaitLock := deps.waitLock + contentionObserved := make(chan struct{}) + retryLock := make(chan struct{}) + var contentionOnce sync.Once + deps.waitLock = func(ctx context.Context) error { + contentionOnce.Do(func() { close(contentionObserved) }) + select { + case <-retryLock: + return realWaitLock(ctx) + case <-ctx.Done(): + return ctx.Err() + } + } + authority := newInstanceAuthorityWithPolicy(deps, testInstanceAuthorityPolicy(root)) held, err := authority.LockUnreferencedRelease( context.Background(), testLaunchRegistration().release, ) require.NoError(t, err) - t.Cleanup(func() { require.NoError(t, held.Release(context.Background())) }) + t.Cleanup(func() { + if held != nil { + require.NoError(t, held.Release(context.Background())) + } + }) type result struct { lease *releaseLease err error } - started := make(chan struct{}) resultCh := make(chan result, 1) waitCtx, cancel := context.WithTimeout(context.Background(), time.Second) defer cancel() go func() { - close(started) lease, lockErr := authority.LockUnreferencedRelease(waitCtx, testAlternateReleaseIdentity()) resultCh <- result{lease: lease, err: lockErr} }() - <-started select { - case early := <-resultCh: - if early.lease != nil { - require.NoError(t, early.lease.Release(context.Background())) - } - require.Failf(t, "release lock returned early", "error: %v", early.err) - case <-time.After(30 * time.Millisecond): + case <-contentionObserved: + case <-waitCtx.Done(): + require.Failf(t, "release lock contention was not observed", "error: %v", waitCtx.Err()) } require.NoError(t, held.Release(context.Background())) + held = nil + close(retryLock) select { case acquired := <-resultCh: require.NoError(t, acquired.err) From 551849a7ff30567196a3dd74181f1ae67c4a7429 Mon Sep 17 00:00:00 2001 From: AlanD20 Date: Sun, 23 Aug 2026 15:24:14 +0200 Subject: [PATCH 0021/1457] test: satisfy init CLI line-length gate --- internal/cli/init_test.go | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/internal/cli/init_test.go b/internal/cli/init_test.go index fe43138d..9e74df98 100644 --- a/internal/cli/init_test.go +++ b/internal/cli/init_test.go @@ -76,7 +76,9 @@ func TestInitCommandForwardsBinaryVersion(t *testing.T) { wantVersion: "1.15.2", }, "non_interactive": { - args: []string{"--non-interactive", "--skip-host", "--skip-network", "--binary-version", "latest"}, + args: []string{ + "--non-interactive", "--skip-host", "--skip-network", "--binary-version", "latest", + }, wantVersion: "latest", wantNonInteractive: true, }, From 77789515afffa1e1b7434655e79b6dedb71f8093 Mon Sep 17 00:00:00 2001 From: AlanD20 Date: Sun, 23 Aug 2026 15:31:42 +0200 Subject: [PATCH 0022/1457] fix: reject ambiguous system-test selections --- scripts/run-system-tests.py | 77 +++++++++ scripts/tests/test_run_system_tests.py | 217 ++++++++++++++++++++++++- 2 files changed, 293 insertions(+), 1 deletion(-) diff --git a/scripts/run-system-tests.py b/scripts/run-system-tests.py index 25ae52cc..60ac836f 100755 --- a/scripts/run-system-tests.py +++ b/scripts/run-system-tests.py @@ -1526,6 +1526,82 @@ def _validate_release_qualification_args( ) +def _validate_test_selection_args( + parser: argparse.ArgumentParser, + args: argparse.Namespace, +) -> None: + """Reject test selections that cannot resolve to known test domains.""" + tier_domains = { + 1: TIER1_DOMAINS, + 2: TIER2_DOMAINS, + 3: TIER3_DOMAINS, + } + domains_by_name = { + **TIER1_DOMAINS, + **TIER2_DOMAINS, + **TIER3_DOMAINS, + } + unknown = list( + dict.fromkeys( + domain for domain in args.domains if domain not in domains_by_name + ) + ) + if unknown: + parser.error(f"unknown domains: {', '.join(unknown)}") + selectors = sum((bool(args.domains), args.tier is not None, args.all)) + if selectors > 1: + parser.error( + "choose exactly one test selector: positional domains, --tier, or --all" + ) + duplicate_domains = list( + dict.fromkeys( + domain + for index, domain in enumerate(args.domains) + if domain in args.domains[:index] + ) + ) + if duplicate_domains: + parser.error(f"duplicate domains: {', '.join(duplicate_domains)}") + tiers = args.tier or [] + duplicate_tiers = list( + dict.fromkeys( + tier + for index, tier in enumerate(tiers) + if tier in tiers[:index] + ) + ) + if duplicate_tiers: + parser.error( + f"duplicate tiers: {', '.join(str(tier) for tier in duplicate_tiers)}" + ) + + if args.domains: + selected_domains = list(args.domains) + elif args.tier is not None: + selected_domains = [ + domain + for tier in args.tier + for domain in tier_domains[tier] + ] + elif args.all: + selected_domains = [ + domain + for domains in tier_domains.values() + for domain in domains + ] + else: + return + if not selected_domains: + parser.error("test selection resolves to zero tests") + empty_domains = sorted( + domain for domain in selected_domains if not domains_by_name[domain] + ) + if empty_domains: + parser.error( + f"selected domains have no test files: {', '.join(empty_domains)}" + ) + + def _build_parser() -> argparse.ArgumentParser: parser = argparse.ArgumentParser( description="Run mvmctl system tests with per-domain VM isolation.", @@ -1757,6 +1833,7 @@ def main() -> None: args = parser.parse_args() _validate_release_qualification_args(parser, args) + _validate_test_selection_args(parser, args) if _selection_requests_tier3(args) and not args.host_direct: parser.error( diff --git a/scripts/tests/test_run_system_tests.py b/scripts/tests/test_run_system_tests.py index 8a098a65..453683aa 100644 --- a/scripts/tests/test_run_system_tests.py +++ b/scripts/tests/test_run_system_tests.py @@ -26,6 +26,45 @@ def orchestrator(monkeypatch: pytest.MonkeyPatch) -> ModuleType: return module +def _assert_selection_rejected_before_side_effects( + orchestrator: ModuleType, + monkeypatch: pytest.MonkeyPatch, + capsys: pytest.CaptureFixture[str], + selection: list[str], + expected_error: str, +) -> None: + calls: list[str] = [] + + def unexpected(name: str): + def fail(*_args: object, **_kwargs: object) -> None: + calls.append(name) + raise AssertionError(f"{name} must not run for an invalid selection") + + return fail + + for helper in ( + "validate_release_build_paths", + "_resolve_candidate_build_version", + "_build_mvm_binary", + "verify_release_binary_identity", + "_require_distinct_candidate_controller", + "_get_mvm_version", + "run_prepare", + "ensure_shared_volume", + "ensure_test_network", + ): + monkeypatch.setattr(orchestrator, helper, unexpected(helper), raising=False) + monkeypatch.setattr(orchestrator.shutil, "which", unexpected("which")) + monkeypatch.setattr(orchestrator.sys, "argv", ["run-system-tests.py", *selection]) + + with pytest.raises(SystemExit) as captured: + orchestrator.main() + + assert captured.value.code == 2 + assert calls == [] + assert expected_error in capsys.readouterr().err + + def test_default_outer_controller_is_the_canonical_system_binary( monkeypatch: pytest.MonkeyPatch, ) -> None: @@ -59,7 +98,7 @@ def test_tier3_selection_detection_covers_all_request_forms( assert orchestrator._selection_requests_tier3(args), selection -def test_t1_t2_and_unknown_selections_do_not_require_host_direct( +def test_t1_t2_and_unknown_names_do_not_imply_tier3_host_direct( orchestrator: ModuleType, ) -> None: parser = orchestrator._build_parser() @@ -76,6 +115,182 @@ def test_t1_t2_and_unknown_selections_do_not_require_host_direct( assert not orchestrator._selection_requests_tier3(args), selection +@pytest.mark.parametrize( + "selection", + [ + ["not-a-domain", "also-unknown"], + ["not-a-domain", "also-unknown", "--tier", "1"], + ], + ids=["domains-only", "with-tier"], +) +def test_unknown_domains_are_rejected_before_any_probe_or_mutation( + orchestrator: ModuleType, + monkeypatch: pytest.MonkeyPatch, + capsys: pytest.CaptureFixture[str], + selection: list[str], +) -> None: + _assert_selection_rejected_before_side_effects( + orchestrator, + monkeypatch, + capsys, + selection, + "unknown domains: not-a-domain, also-unknown", + ) + + +@pytest.mark.parametrize( + "selection", + [ + ["--all", "cli"], + ["--all", "--tier", "1"], + ["cli", "--tier", "1"], + ], + ids=["all-and-domains", "all-and-tier", "domains-and-tier"], +) +def test_ambiguous_test_selectors_are_rejected_before_any_probe_or_mutation( + orchestrator: ModuleType, + monkeypatch: pytest.MonkeyPatch, + capsys: pytest.CaptureFixture[str], + selection: list[str], +) -> None: + _assert_selection_rejected_before_side_effects( + orchestrator, + monkeypatch, + capsys, + selection, + "choose exactly one test selector: positional domains, --tier, or --all", + ) + + +@pytest.mark.parametrize( + ("selection", "expected_error"), + [ + (["cli", "cli"], "duplicate domains: cli"), + (["cli", "config", "cli"], "duplicate domains: cli"), + (["--tier", "1,1"], "duplicate tiers: 1"), + (["--tier", "1,2,1"], "duplicate tiers: 1"), + ], + ids=["adjacent-domains", "nonadjacent-domains", "adjacent-tiers", "nonadjacent-tiers"], +) +def test_duplicate_test_selectors_are_rejected_before_any_probe_or_mutation( + orchestrator: ModuleType, + monkeypatch: pytest.MonkeyPatch, + capsys: pytest.CaptureFixture[str], + selection: list[str], + expected_error: str, +) -> None: + _assert_selection_rejected_before_side_effects( + orchestrator, + monkeypatch, + capsys, + selection, + expected_error, + ) + + +def test_unique_tier_selection_preserves_requested_order( + orchestrator: ModuleType, +) -> None: + parser = orchestrator._build_parser() + args = parser.parse_args(["--tier", "2,1,3"]) + + orchestrator._validate_test_selection_args(parser, args) + + assert args.tier == [2, 1, 3] + + +@pytest.mark.parametrize( + ("selection", "empty_domains", "expected_empty"), + [ + (["cli"], ("cli",), "cli"), + (["cli", "config"], ("cli",), "cli"), + (["--tier", "2"], ("volume",), "volume"), + (["--tier", "2"], ("volume", "ssh"), "ssh, volume"), + (["--all"], ("env",), "env"), + ], + ids=["domain", "mixed-domains", "tier", "tier-multiple", "all"], +) +def test_empty_domain_registrations_are_rejected_before_any_probe_or_mutation( + orchestrator: ModuleType, + monkeypatch: pytest.MonkeyPatch, + capsys: pytest.CaptureFixture[str], + selection: list[str], + empty_domains: tuple[str, ...], + expected_empty: str, +) -> None: + domains = { + **orchestrator.TIER1_DOMAINS, + **orchestrator.TIER2_DOMAINS, + **orchestrator.TIER3_DOMAINS, + } + for domain in empty_domains: + domains[domain].clear() + + _assert_selection_rejected_before_side_effects( + orchestrator, + monkeypatch, + capsys, + selection, + f"selected domains have no test files: {expected_empty}", + ) + + +def test_selection_with_no_registered_domains_is_rejected_before_any_probe( + orchestrator: ModuleType, + monkeypatch: pytest.MonkeyPatch, + capsys: pytest.CaptureFixture[str], +) -> None: + monkeypatch.setattr(orchestrator, "TIER2_DOMAINS", {}) + + _assert_selection_rejected_before_side_effects( + orchestrator, + monkeypatch, + capsys, + ["--tier", "2"], + "test selection resolves to zero tests", + ) + + +def test_no_arguments_print_help_and_exit_without_any_probe( + orchestrator: ModuleType, + monkeypatch: pytest.MonkeyPatch, + capsys: pytest.CaptureFixture[str], +) -> None: + monkeypatch.setattr( + orchestrator.shutil, + "which", + lambda *_args, **_kwargs: pytest.fail("no-args help must not probe binaries"), + ) + monkeypatch.setattr(orchestrator.sys, "argv", ["run-system-tests.py"]) + + with pytest.raises(SystemExit) as captured: + orchestrator.main() + + assert captured.value.code == 0 + assert "usage:" in capsys.readouterr().out.lower() + + +@pytest.mark.parametrize( + "selection", + [ + ["--prepare"], + ["--volume"], + ["--image"], + ["--volume", "--image", "--prepare"], + ["--rebuild", "--candidate-version", "0.3.0"], + ], + ids=["prepare", "volume", "image", "combined", "rebuild"], +) +def test_preparation_only_modes_do_not_require_a_test_selector( + orchestrator: ModuleType, + selection: list[str], +) -> None: + parser = orchestrator._build_parser() + args = parser.parse_args(selection) + + orchestrator._validate_test_selection_args(parser, args) + + def test_host_direct_help_is_an_acknowledgment_not_a_clean_host_guarantee( orchestrator: ModuleType, ) -> None: From a1c0cff6ff322c3ddf71002065b61e6af6589799 Mon Sep 17 00:00:00 2001 From: AlanD20 Date: Sun, 23 Aug 2026 15:33:00 +0200 Subject: [PATCH 0023/1457] docs: reconcile v0.3 security and QA status --- CHANGELOG.md | 56 +++++- CONTEXT.md | 55 ++++-- docs/RC_QA.md | 125 +++++++++---- docs/REFERENCES.md | 1 + docs/RELEASE.md | 31 ++-- docs/RUNTIME.md | 5 +- docs/STANDARDS.md | 2 + docs/adr/0005-sudo-privilege-architecture.md | 2 +- ...owned-single-binary-privileged-dispatch.md | 73 ++++++-- .../HOW_AGENTS_WRITE_SYSTEM_TESTS.md | 24 ++- docs/development/HOW_TO_RUN_SYSTEM_TESTS.md | 164 ++++++++++++------ docs/system-test-architecture.md | 136 ++++++++++----- tasks/plan.md | 23 ++- tasks/todo.md | 91 ++++++++-- 14 files changed, 590 insertions(+), 198 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index e10f6288..1357cbb6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -50,13 +50,38 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 #### Root-owned VM authority substrate - Added a private typed VM authority under `internal/service/jailer` with durable owner-bound `registered`, `cleaning`, and `cleaned` records beneath `/var/lib/mvmctl/instances/` and persistent ownership tombstones. -- Added descriptor-relative, no-follow traversal; strict bounded record decoding; atomic durable replacement; persistent - root-owned locks; and the fixed `release -> global index -> VM` lock order. +- Added descriptor-relative, no-follow traversal; strict bounded record decoding; atomic durable replacement; reusable + root-owned runtime locks; and the fixed `release -> global index -> VM` lock order. +- The private release-removal lease now locks the planned canonical architecture/version store slot, so conflicting + identities for one future slot cannot race under different hash-derived locks when Task 6 wires the new store. - Global VM-ID claims now fail closed on foreign, duplicate, corrupt, unreadable, or inconsistent authority records, and trusted release removal can acquire a lease only when no active authority record references the exact release. - This substrate is private and not yet wired into public VM lifecycle operations; that integration remains a release blocker below. +#### Installed release-candidate qualification +- For T1/T2, the Python system-test runner now treats `MVM_BINARY` as a distinct compatible outer controller and + `MVM_CANDIDATE_BINARY` as the artifact under test. It rejects path, symlink, and hard-link aliases between them and + never replaces the controller while building or provisioning a runner. Tier 3 remains a separate host-direct gate + that is allowed only on a clean disposable qualification host. +- Runner images stage the candidate outside the canonical path, invoke the real `host install-system` administrator + route, initialize through exact `/usr/local/bin/mvm`, and use that installed path for T1/T2 assertions. +- Added 10 L2 installer cases covering canonical metadata and content, safe ancestors, idempotence, malformed and + unprivileged invocation, symlink and read-only-target failures, invoking-user ownership after privileged init, + self-update refusal, restoration, and cleanup. +- The focused `system_install` domain passed inside a disposable nested-virt runner using `0.3.0-rc.1`; the outer host + controller remained byte-for-byte unchanged. +- Tier 3 and `--all` now require explicit pre-mutation `--host-direct` consent. The dedicated + `--release-qualification` gate additionally requires an unfiltered full matrix, a fresh explicit-version release + build, and byte/version identity with exact root-owned `/usr/local/bin/mvm` before host resource preparation. +- The runner rejects unknown, repeated, mixed, zero-domain, or empty-file test selections before binary probes, builds, + or resource mutation instead of warning and returning successful but incomplete evidence. + +#### Deterministic initialization +- Added `mvm init --binary-version ` to request an exact Firecracker/Jailer pair when no local pair exists. + The explicit selection survives the administrator host-init interaction; existing local/default behavior is + unchanged. + ### Changed #### VM launch and release identity @@ -73,6 +98,18 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - During development, the generated sudoers policy targets `/usr/local/bin/mvm` instead of a user-owned executable. Its wildcard and raw-tool grants remain transitional and are not the final `0.3.0` security boundary. +#### System-test asset preparation +- Shared runner assets are seeded from `MVM_ASSET_MIRROR` with descriptor-pinned, no-follow access and + `mkfs.ext4 -d`; the runner no longer loop-mounts and copies the mirror with raw sudo on the host. +- Builder asset pulls run sequentially with bounded connect/operation timeouts so image extraction cannot exhaust the + constrained nested builder through concurrent pulls. +- Missing or stale `asset-mirror` database records are rebuilt only when the recorded backing path is the exact managed + test-volume path; unexpected paths fail closed without mutation. + +#### Managed storage identity +- Newly created volume backing files are named from the immutable 64-character volume ID rather than the user-visible + display name. Raw and qcow2 formats remain explicit suffixes; database paths are derived metadata, not root authority. + #### Network isolation - Traffic routed between different managed networks is now denied unless an explicit typed service-access policy permits the destination VM protocol and port. Same-network bridge traffic remains unchanged. @@ -85,7 +122,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 Jailer directories, cgroups, TAP devices, firewall state, policy rows, or remote-exec flags. Initialization must refuse ambiguous legacy runtime state rather than silently adopting or deleting it. - Operators must stop and remove the old installation, retain any desired user artifacts separately, clean legacy host - runtime state through the documented administrator procedure, and recreate VMs and policies under v0.3.0. + runtime state, and recreate VMs and policies under v0.3.0. The exact fail-closed administrator cleanup procedure is + still a release blocker and must be documented after `host reset`/cleanup hardening lands; operators must not improvise + raw cleanup from this development changelog. - The new administrator-controlled system installation is mandatory. A user-owned `~/.local/bin/mvm` is no longer a valid final sudo target. - The system binary and an invoking user binary must support the same privileged protocol version. Version mismatch stops @@ -109,6 +148,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - VM create no longer fails when Jailer places the cgroup at `/sys/fs/cgroup/mvmctl/`; the check previously expected an intermediate `firecracker/` component that Jailer v1.16 does not create. - Stop and remove now clean the VM's actual leaf cgroup instead of leaving stale ones behind. +#### Privileged initialization ownership +- `sudo mvm host init` now resolves a complete, internally consistent sudo identity before creating invoking-user cache + or configuration directories. It rejects partial/spoofed identity and sudo-time path overrides before mutation. +- Invoking-user directories are created component by component with descriptor-relative no-follow operations; only + newly created directories receive the invoking UID/GID, while unsafe existing paths and replacement races fail closed. + ### Security work still pending The following items are release blockers and are intentionally not described above as completed behavior: @@ -129,10 +174,15 @@ The following items are release blockers and are intentionally not described abo favor of exact directional, non-root, resource-bounded VM-to-VM `exec` policies. - Replace the transitional wildcard/raw-tool sudoers entries with access only to the reserved privileged marker of the root-owned system binary. +- Separate administrator host state from invoking-user initialization, replace the generic `HostInit` result with a + typed contract, and propagate every host mutation/state-write failure instead of reporting partial setup as success. - Make launch, abort, cleanup, reconciliation, snapshots, live volumes, cgroups, and firewall policy crash-consistent and fail closed under ownership, PID-reuse, path-race, and partial-failure conditions. - Complete adversarial L1 coverage, Python `tests/system/` clean-install/fault qualification, leak audits, full CI, and the final documentation accuracy review. Go tests support but do not replace CLI-level L2 release signoff. +- Finish Tier 3 clean-host inventory and ownership-safe cleanup; unify every host-direct CLI call, preserve pre-existing + resources, surface every teardown failure, and reject skips/xfails/omitted domains before accepting full `--all` + release evidence. ## [0.2.0] - 2026-07-10 diff --git a/CONTEXT.md b/CONTEXT.md index b28fa657..e75278af 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -223,29 +223,33 @@ The release script produces the candidate artifact at `dist/mvm`. A development under `~/.local/bin`, but a user-owned path is **never** a sudo target. The canonical host installation is `/usr/local/bin/mvm`, owned by `root:root` with mode `0755` under a root-owned, -non-writable directory chain. Install or upgrade it through the candidate artifact's exact early bootstrap route: +non-writable directory chain. Install or replace it through the candidate artifact's exact early bootstrap route: ```bash -./scripts/build.sh release +./scripts/build.sh release --version 0.3.0-rc.1 --output dist/mvm sudo ./dist/mvm host install-system sudo /usr/local/bin/mvm host init ``` `host install-system` executes before ordinary application initialization. It does not read `MVM_CACHE_DIR`, user configuration, the user database, Cobra, or plugins, and it replaces its environment with fixed root-safe values before -installation. If an old managed sudoers file still authorizes a user-owned binary, remove that legacy file from an -authenticated administrator session before running the bootstrap command; the installer refuses unrecognized active -syntax rather than guessing how sudo authenticated the process. +installation. v0.3.0 is a clean installation: it does not convert, adopt, or preserve old user/runtime state. Remove an +old managed sudoers file and clean the old installation from an authenticated administrator session before running the +bootstrap command; the installer refuses unrecognized active syntax rather than guessing how sudo authenticated the +process. The exact fail-closed cleanup procedure remains a v0.3.0 release blocker until administrator reset/cleanup is +hardened; this architecture rule is not permission to improvise raw host cleanup. Normal users run `/usr/local/bin/mvm` without sudo. The `mvm` group is the authorization role for the final constrained -passwordless privileged protocol; it never makes a development artifact trusted. During Tasks 2-8, the generated policy -still contains transitional raw-tool grants and `/usr/local/bin/mvm *` for callers that have not migrated. That policy is -not the final security boundary and must not ship as marker-only until Task 9 removes those grants. A system-installed -executable cannot replace itself through `mvm self-update`; an administrator installs the newly downloaded trusted -artifact with `host install-system`. +passwordless privileged protocol; it never makes a development artifact trusted. During v0.3 development, the generated +policy still contains transitional raw-tool grants and `/usr/local/bin/mvm *` for callers not yet converted to typed +dispatch. That policy is not the final security boundary. The marker-only policy lands only after every supported +privileged caller uses the typed dispatcher. A system-installed executable cannot replace itself through +`mvm self-update`; an administrator installs the newly downloaded trusted artifact with `host install-system`. For **release testing / RC QA / system tests**, always use `./scripts/build.sh release`; a bare `go build` lacks the -release version metadata, symbol stripping, and PIE settings and must not be used for release qualification. +release version metadata, symbol stripping, and PIE settings and must not be used for release qualification. An +untagged RC build must pass the intended version explicitly. Otherwise `git describe` can inherit the previous release +tag and produce a candidate with the wrong identity. ### Asset mirror environment variable (OPTIONAL) @@ -257,6 +261,10 @@ mvm ``` This variable directs the local asset cache for downloaded kernel images, root filesystems, and firmware blobs. +Checksum/version metadata may still be fetched from the fixed upstream origin. The system-test `asset-mirror` volume is +a point-in-time, read-only copy of this directory: after repairing or adding a host mirror object, rebuild that volume +before running QA. A stale object inside the read-only copy can be detected and bypassed, but it cannot persist its own +replacement. ### CLI is the canonical interface @@ -507,7 +515,30 @@ go test ./... -count=1 -coverprofile=coverage.out -covermode=atomic ### L2: Runner VM System Tests (Python `tests/system/`) -**Ground truth.** Every user-facing feature must have an L2 test. Real binary, real subprocess, real infrastructure inside a disposable Firecracker VM with nested KVM. No mocking of any kind. Operates against the compiled `mvm` binary. Verifies actual business outcomes at the OS level: JSON state, filesystem state, process state, iptables rules. +**Ground truth.** Every user-facing feature must have an L2 test. Real binary, real subprocess, real infrastructure inside +a disposable Firecracker VM with nested KVM. No mocking of any kind. The release candidate is installed through +`host install-system`, and T1/T2 tests invoke exact `/usr/local/bin/mvm`; a directly copied or PATH-selected developer +binary is not release evidence. Tests verify actual outcomes at the OS level: JSON, files, processes, cgroups, +namespaces, links, mounts, sockets, and nftables state as applicable. + +The outer host controller and the artifact under test are deliberately separate. `MVM_BINARY` names an already +installed controller used only to create, inspect, copy to, and destroy disposable runner VMs. +`MVM_CANDIDATE_BINARY` names the release artifact (default `dist/mvm`) installed only inside those VMs. The runner +image tag comes from an isolated `MVM_CANDIDATE_BINARY --version` probe. Building or testing a candidate must never +replace `MVM_BINARY`. + +Runner initialization requests exact Firecracker 1.16.0 with +`/usr/local/bin/mvm init --non-interactive --binary-version 1.16.0` when no local pair exists. This avoids a moving +`latest` dependency while preserving existing local/default selection on already-prepared images. + +This controller/candidate isolation applies only to T1/T2 runner VMs. Tier 3 is host-direct and may mutate outer-host +VMs, kernels, images, networks, and volumes. It is allowed only on a clean disposable qualification host where the +candidate has been explicitly installed as exact `/usr/local/bin/mvm`; it is not an iterative developer-host test mode. +The runner rejects Tier 3 or `--all` before any work unless `--host-direct` explicitly acknowledges outer-host mutation. +Its stricter `--release-qualification` mode requires an unfiltered `--all`, `--host-direct`, a fresh explicit-version +release build, and exact root-owned `/usr/local/bin/mvm` content/version identity before resource mutation. This consent +and binary gate does not yet prove that every Tier 3 resource is newly owned or that every teardown succeeded: do not +invoke Tier 3 or `--all` on a host with pre-existing mvmctl state until the remaining Task 17 cleanup gates land. ```bash # Run inside the runner VM (disposable Firecracker VM with nested KVM) diff --git a/docs/RC_QA.md b/docs/RC_QA.md index 35e51068..5914e3d5 100644 --- a/docs/RC_QA.md +++ b/docs/RC_QA.md @@ -21,15 +21,23 @@ A release is **blocked** until ALL of the following pass: | Gate | Command | Must Pass | |------|---------|-----------| -| Compile | `go build ./...` | Zero errors | +| Tidy | `go mod tidy && git diff --exit-code` | Module files unchanged | +| Format | `test -z "$(gofmt -l .)"` | Entire Go tree formatted | +| Line length | `golines --max-len=120 --no-reformat-tags --list-files ./internal/ ./pkg/ ./cmd/` | No listed files | +| Generate | `go generate ./internal/service/agent/...` | Generated assets current | | Vet | `go vet ./...` | Zero warnings | -| Unit tests | `go test ./...` | All pass | -| System tests | `python3 scripts/run-system-tests.py --all` | All pass, zero skips on required tests | -| Version check | `./scripts/build.sh release && ./dist/mvm --version` | Returns correct version (not `0.0.0-dev`) | -| Smoke test | `./dist/mvm --help` | Shows all commands | +| Go tests | `go test ./... -count=1 -coverprofile=coverage.out -covermode=atomic` | All pass | +| System install | `sudo ./dist/mvm host install-system && sudo /usr/local/bin/mvm host init` | Canonical root-owned install succeeds | +| System tests | `MVM_BINARY=/usr/local/bin/mvm MVM_CANDIDATE_BINARY=./dist/mvm python3 scripts/run-system-tests.py --release-qualification --host-direct --rebuild --candidate-version X.Y.Z --all` | Identity-qualified T1/T2 runners and clean-host T3 all pass | +| Version check | `/usr/local/bin/mvm --version` | Returns correct version (not `0.0.0-dev`) | +| Smoke test | `/usr/local/bin/mvm --help` | Shows all commands | **No gate is optional.** A single failure blocks the release. +The system-install, version, and smoke gates run on a dedicated clean qualification host or disposable outer VM. Do not +replace a developer workstation's active controller to satisfy them. During iterative QA, the existing outer controller +creates disposable nested runners and the candidate is installed only inside those runners. + --- ## 2. Test Environment @@ -47,7 +55,8 @@ System tests run inside a Firecracker VM with nested KVM enabled. This provides: See [docs/development/HOW_TO_RUN_SYSTEM_TESTS.md](development/HOW_TO_RUN_SYSTEM_TESTS.md) for the full walkthrough: host prerequisites, building the binary, setting up the -asset mirror, running the orchestrator (`--prepare`, `--all`, `--tier`, `--push`), +asset mirror, running the orchestrator (`--prepare`, `--all`, `--tier`, `--host-direct`, +`--release-qualification`, `--push`), interpreting results, and troubleshooting common failures. The orchestrator script flags are documented there. For the architecture overview @@ -60,9 +69,31 @@ The orchestrator script flags are documented there. For the architecture overvie ### 3.1 System Test Execution -System tests run inside disposable Firecracker VMs with nested KVM, providing -full isolation. The orchestrator creates one VM per domain, runs tests via -`mvm exec`, and destroys the VM after. +T1/T2 system tests run inside disposable Firecracker VMs with nested KVM. The +orchestrator creates one VM per T1/T2 domain, runs tests via `mvm exec`, and +destroys the VM after. Tier 3 is host-direct and has the separate restrictions +below. + +For T1/T2, release qualification keeps the outer controller (`MVM_BINARY`) separate from `./dist/mvm` +(`MVM_CANDIDATE_BINARY`). The controller manages disposable test resources. The candidate-qualified base image is built +by staging the candidate, invoking `host install-system`, and initializing through the resulting exact +`/usr/local/bin/mvm`. T1/T2 runners clone that image and execute the installed path; the focused `system_install` domain +replays and verifies the administrator install contract. Directly copying onto the canonical path, relying on PATH +lookup, or aliasing the candidate to the controller does not qualify the installer. + +The builder's unprivileged initialization pins Firecracker 1.16.0 through +`mvm init --non-interactive --binary-version 1.16.0`; it must not resolve a moving latest release during qualification. + +Tier 3 is different: those tests run directly on the outer host and may create or destroy host resources. The runner +rejects Tier 3 and `--all` before any work unless `--host-direct` acknowledges that mutation. For release evidence, +`--release-qualification` additionally requires an unfiltered `--all`, an explicit-version rebuild, exact canonical +root-owned `/usr/local/bin/mvm`, safe ancestors, and matching candidate/controller SHA-256 and strict version before +resource preparation. + +These gates still do not prove that the host started empty or that every Tier 3 resource belongs to the current run. +Until the remaining Task 17 inventory and cleanup gates land, run full qualification only on a dedicated clean host or +disposable outer VM. On a development workstation, use focused T1/T2 domains or `--tier 1,2`; `--host-direct` is consent, +not isolation or an ownership-safe cleanup guarantee. Tests are organized into three tiers: @@ -112,33 +143,62 @@ For every release, collect and archive: | Evidence | How to Collect | Purpose | |----------|----------------|---------| -| Go build output | `go build ./... 2>&1` | Zero compilation errors | +| Tidy output | `go mod tidy && git diff --exit-code` | Module files unchanged | +| Format output | `test -z "$(gofmt -l .)"` | Entire Go tree formatted | +| Generate output | `go generate ./internal/service/agent/...` | Generated assets current | | Go vet output | `go vet ./... 2>&1` | Zero static analysis warnings | -| Go test output | `go test ./... 2>&1` | All unit tests pass | -| System test results | `python3 scripts/run-system-tests.py --all` | All system tests pass | -| Version output | `./dist/mvm --version` | Correct version string | -| Help output | `./dist/mvm --help` | All commands listed | +| Go test output | `go test ./... -count=1 -coverprofile=coverage.out -covermode=atomic` | All Go tests pass | +| System test results | `MVM_BINARY=/usr/local/bin/mvm MVM_CANDIDATE_BINARY=./dist/mvm python3 scripts/run-system-tests.py --release-qualification --host-direct --rebuild --candidate-version X.Y.Z --all` | Identity-qualified full matrix passes on the clean host | +| Version output | `/usr/local/bin/mvm --version` | Correct version string | +| Help output | `/usr/local/bin/mvm --help` | All commands listed | | Benchmark results | `benchmarks/results.json` | Performance within thresholds | | Binary checksum | `sha256sum dist/mvm` | Reproducibility | ### 4.1 Evidence Archive ```bash -# Create evidence directory -mkdir -p release-evidence/vX.Y.Z +set -euo pipefail + +version=X.Y.Z +evidence_dir="release-evidence/v${version}" +mkdir -p "$evidence_dir" +exec > >(tee -a "$evidence_dir/qualification.log") 2>&1 # Collect evidence -go build ./... > release-evidence/vX.Y.Z/build.log 2>&1 -go vet ./... > release-evidence/vX.Y.Z/vet.log 2>&1 -go test ./... > release-evidence/vX.Y.Z/test.log 2>&1 -./dist/mvm --version > release-evidence/vX.Y.Z/version.txt 2>&1 -./dist/mvm --help > release-evidence/vX.Y.Z/help.txt 2>&1 -sha256sum dist/mvm > release-evidence/vX.Y.Z/checksum.sha256 +./scripts/build.sh release --version "$version" --output dist/mvm +go mod tidy +git diff --exit-code +test -z "$(gofmt -l .)" +test -z "$(golines --max-len=120 --no-reformat-tags --list-files \ + ./internal/ ./pkg/ ./cmd/)" +go generate ./internal/service/agent/... +git diff --exit-code +go vet ./... > "$evidence_dir/vet.log" 2>&1 +go test ./... -count=1 -coverprofile=coverage.out -covermode=atomic \ + > "$evidence_dir/test.log" 2>&1 + +# The following gates mutate the outer host. Run them only on the dedicated +# clean qualification host or disposable outer VM. +sudo ./dist/mvm host install-system +sudo /usr/local/bin/mvm host init +test "$(/usr/local/bin/mvm --version)" = "mvm ${version}" +/usr/local/bin/mvm --version > "$evidence_dir/version.txt" 2>&1 +/usr/local/bin/mvm --help > "$evidence_dir/help.txt" 2>&1 +sha256sum dist/mvm > "$evidence_dir/checksum.sha256" # System test results (full suite with orchestrator) -python3 scripts/run-system-tests.py --all 2>&1 | tee release-evidence/vX.Y.Z/system-tests.log +MVM_BINARY=/usr/local/bin/mvm MVM_CANDIDATE_BINARY=./dist/mvm \ + MVM_ASSET_MIRROR="$HOME/.cache/mvm-asset-mirror" \ + python3 scripts/run-system-tests.py \ + --release-qualification --host-direct --rebuild \ + --candidate-version "$version" --all \ + 2>&1 | tee "$evidence_dir/system-tests.log" ``` +Until the orchestrator records skip/xfail counts in its final summary, QA must inspect every archived per-domain pytest +summary and record that it contains zero skipped, xfailed, or xpassed cases. A domain-level `PASS` alone is not sufficient +release evidence. + --- ## 5. Regression Criteria @@ -160,7 +220,7 @@ A regression is **any** of the following: | Category | Definition | |----------|-----------| | **Expected behavior change** | A feature was intentionally changed (documented in CHANGELOG.md) | -| **New test failure** | A new test was added that reveals a pre-existing bug (file a bug, don't block release) | +| **New test exposing an old bug** | Not classified as a regression, but the revealed defect still blocks the release until fixed or the requirement is explicitly removed | | **Performance improvement** | A benchmark is faster than before | | **Test environment issue** | Missing dependency, wrong permissions, stale state (fix environment, not code) | @@ -181,16 +241,23 @@ A regression is **any** of the following: ## Release vX.Y.Z — QA Sign-off ### Gates -- [ ] `go build ./...` — zero errors +- [ ] `go mod tidy && git diff --exit-code` — module files and generated tree unchanged +- [ ] `test -z "$(gofmt -l .)"` — entire Go tree formatted +- [ ] `golines --max-len=120 --no-reformat-tags --list-files ./internal/ ./pkg/ ./cmd/` — no files listed +- [ ] `go generate ./internal/service/agent/... && git diff --exit-code` — generated assets current - [ ] `go vet ./...` — zero warnings -- [ ] `go test ./...` — all pass -- [ ] System tests — all pass inside runner VM (see evidence archive) -- [ ] `./dist/mvm --version` — returns vX.Y.Z -- [ ] `./dist/mvm --help` — all commands present +- [ ] `go test ./... -count=1 -coverprofile=coverage.out -covermode=atomic` — all pass +- [ ] `./scripts/build.sh release --version X.Y.Z --output dist/mvm` — exact candidate built +- [ ] Dedicated clean qualification host/outer VM — candidate installed at exact `/usr/local/bin/mvm` +- [ ] System tests — every required T1/T2/T3 case passes; archived summaries show zero skips/xfails/xpasses +- [ ] `/usr/local/bin/mvm --version` — returns exact `mvm X.Y.Z` +- [ ] `/usr/local/bin/mvm --help` — all commands present ### Evidence - [ ] Build log archived +- [ ] Tidy, format, line-length, generate, vet, and Go-test logs archived - [ ] System test log archived +- [ ] Per-domain skip/xfail audit archived - [ ] Binary checksum archived - [ ] Benchmark results within thresholds diff --git a/docs/REFERENCES.md b/docs/REFERENCES.md index dc09b32c..150e700e 100644 --- a/docs/REFERENCES.md +++ b/docs/REFERENCES.md @@ -94,6 +94,7 @@ mvm init [flags] | `--non-interactive` | Use defaults, skip all prompts | `false` | | `--skip-host` | Skip host init step | `false` | | `--skip-network` | Skip default network creation | `false` | +| `--binary-version ` | Download this exact Firecracker version when no local pair exists | empty (interactive choice or latest in non-interactive mode) | --- diff --git a/docs/RELEASE.md b/docs/RELEASE.md index 2e65672d..7cb0acdc 100644 --- a/docs/RELEASE.md +++ b/docs/RELEASE.md @@ -32,24 +32,35 @@ This document covers how to release a new version of mvmctl from start to finish Run the full CI gate: ```bash -# Compile -go build ./... - -# Vet +# Exact CI gates +go mod tidy && git diff --exit-code +test -z "$(gofmt -l .)" +test -z "$(golines --max-len=120 --no-reformat-tags --list-files \ + ./internal/ ./pkg/ ./cmd/)" +go generate ./internal/service/agent/... +git diff --exit-code go vet ./... +go test ./... -count=1 -coverprofile=coverage.out -covermode=atomic -# Unit tests -go test ./... +# Build the release candidate with an explicit identity +./scripts/build.sh release --version X.Y.Z --output dist/mvm -# Build release binary for system tests -./scripts/build.sh release +# The remaining commands mutate the outer host. Run them only on a dedicated +# clean qualification host or disposable outer VM. sudo ./dist/mvm host install-system sudo /usr/local/bin/mvm host init -MVM_BINARY=/usr/local/bin/mvm python3 scripts/run-system-tests.py --all +MVM_ASSET_MIRROR=~/.cache/mvm-asset-mirror \ +MVM_BINARY=/usr/local/bin/mvm MVM_CANDIDATE_BINARY=./dist/mvm \ + python3 scripts/run-system-tests.py \ + --release-qualification --host-direct --rebuild \ + --candidate-version X.Y.Z --all ``` -All five must pass. If system tests fail, investigate before proceeding. +Every gate and every collected test must pass with zero skips/xfails/xpasses. If +system tests fail or omit a required case, investigate before proceeding. Do not +run the host-direct Tier 3 suite on a workstation with pre-existing mvmctl state. +`--host-direct` is explicit consent, not proof that cleanup is ownership-safe. See [RC_QA.md](RC_QA.md) for test environment setup, the orchestrator script reference, and tier execution details. diff --git a/docs/RUNTIME.md b/docs/RUNTIME.md index c07992c5..b07040b7 100644 --- a/docs/RUNTIME.md +++ b/docs/RUNTIME.md @@ -554,8 +554,9 @@ Services are spawned in the background by the core layer via `system.SpawnServic ### Sudoers During the privilege-boundary migration, `mvm host init` targets only root-owned `/usr/local/bin/mvm` but still retains -the legacy raw-tool and public-service grants needed by unmigrated callers. Task 9 removes those grants and leaves only -the versioned early privileged marker. See ADR-0016; the transitional policy is not the final security model. +the legacy raw-tool and public-service grants needed by unmigrated callers. Task 14 removes those grants, after Tasks +8–13 migrate every caller, and leaves only the versioned early privileged marker. See ADR-0016; the transitional policy +is not the final security model. --- diff --git a/docs/STANDARDS.md b/docs/STANDARDS.md index 66fa0c6f..17403a29 100644 --- a/docs/STANDARDS.md +++ b/docs/STANDARDS.md @@ -270,6 +270,8 @@ Three-level architecture — see `docs/development/HOW_AGENTS_WRITE_SYSTEM_TESTS ### L2: Runner VM System Tests (Python in `tests/system/`) - Real binary, real subprocess, real infrastructure inside a disposable Firecracker VM with nested KVM. - No mocking of any kind — the binary is real, the subprocesses are real. +- Install the release candidate through `host install-system`; T1/T2 tests invoke exact `/usr/local/bin/mvm`. +- Directly copying a binary onto the canonical path or relying on PATH lookup does not qualify the installer or release. - **Ground truth:** A feature is not tested until it has an L2 test. L0/L1 are fast pre-filters, not replacements. - Run via `pytest tests/system/` inside the runner VM. diff --git a/docs/adr/0005-sudo-privilege-architecture.md b/docs/adr/0005-sudo-privilege-architecture.md index b9deaa24..c3e23c1e 100644 --- a/docs/adr/0005-sudo-privilege-architecture.md +++ b/docs/adr/0005-sudo-privilege-architecture.md @@ -112,7 +112,7 @@ When running `sg mvm -c 'pytest tests/system/'`: - **Principle of least privilege (target).** The CLI runs as a normal user for HTTP downloads, file I/O, SQLite queries, and non-privileged operations. The transitional raw-tool and public-CLI grants remain broader than the named effects - accepted by ADR-0016 and are removed only at Task 9. + accepted by ADR-0016 and are removed only at Task 14, after Tasks 8–13 migrate every caller. - **Known transitional exposure.** Root ownership prevents replacement of the authorized executable, but `/usr/local/bin/mvm *` still exposes broad public argument handling as root. A compromise of the ordinary CLI process must be treated as potentially reaching that authority until typed callers and the marker-only policy replace it. diff --git a/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md b/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md index 01179cc1..a81d9a25 100644 --- a/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md +++ b/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md @@ -130,7 +130,7 @@ application architecture: | Package/file | Responsibility | Must not do | |---|---|---| | `cmd/mvm/main.go` | Detect the reserved marker and enter privileged dispatch before normal initialization | Parse domain requests, perform side effects, initialize the public CLI | -| `internal/service/privileged/` | Versioned envelope, bounded parsing, invoking identity, executable verification, and fixed action dispatch | Import `pkg/api`, `internal/cli`, or any `internal/core/*`; expose generic command execution | +| `internal/service/privileged/` | Versioned envelope, bounded parsing, invoking identity, executable verification, and fixed action dispatch | Import `pkg/api`, `internal/cli`, or any `internal/core/*`; expose generic command execution or raw effect paths | | `internal/service/jailer/` | Typed release, launch, namespace mount, cgroup, process, and cleanup handlers plus normal-user client calls | Orchestrate other core domains or trust user paths after validation | | `internal/service/loopmount/` | Existing typed provisioning protocol and confined rootfs operations | Accept arbitrary host roots or unrelated unmount/PID targets | | `internal/service/network/` | Typed root-side bridge, TAP, address, route, and neighbor mutations | Accept raw `ip` argv or own network business orchestration | @@ -174,23 +174,71 @@ Privileged code treats the normal CLI, arguments, environment, database, cache, untrusted. Managed filesystem objects are opened once through descriptor-relative no-symlink resolution and retained as pinned descriptors through each operation. Validation followed by pathname reopen is prohibited. -Every privileged VM operation is serialized by a root-owned per-VM lock and authorized by a minimal root-owned instance -record. Process actions verify PID, start time, expected cgroup membership, owner, and exact release; pidfds are used when -available. PID reuse fails closed. +`MVM_CACHE_DIR` remains the user-selected location for persistent user state and large VM artifacts. The privileged +protocol may carry exactly one typed managed-cache locator; it is a namespace locator, not authority for an individual +effect. The receiver requires one canonical absolute path, rejects dot components, NULs, symlinks, magic links, unsafe +owners/modes, and unsupported mount topology, and pins the directory before resolving any resource. All subsequent +inputs are typed IDs, closed format enums, and presence/access intent. A request never supplies a kernel, rootfs, +snapshot, volume, config, PID, socket, log, mount-target, or temporary-file path. + +This exception is deliberately narrower than a generic path interface. Dropping custom cache roots would break the +documented large-filesystem use case, while a persistent root-owned path registry would still need safe pathname +resolution after reboot. Instead, each live instance record binds the pinned cache root's stable identity (device, +inode, and mount identity where available). A later operation fails closed if it cannot re-pin the same cache identity. +Descendants are opened relative to that pinned root with beneath/no-symlink/no-magic-link resolution and retained until +the privileged effect and rollback are complete. + +Privileged-visible basenames are canonical and derived by the receiver. Volume storage is keyed by volume ID rather +than user-visible name; the VM rootfs uses fixed `rootfs.img`; kernel and image names use their canonical IDs plus only +enumerated representations; cloud-init, Firecracker runtime, and snapshot leaves use compile-time fixed names. The user +database may describe a resource, but it cannot choose the basename root will open. The intentional +snapshot phantom-rootfs link is managed only by a typed descriptor-relative snapshot operation and is never followed as +an input resource. + +The v0.3 canonical privileged-visible layout is: + +| Resource | Relative to the pinned managed cache root | +|---|---| +| VM directory | `vms//` | +| VM rootfs | `vms//rootfs.img` | +| Cloud-init ISO | `vms//cloud-init.iso` | +| Firecracker runtime | Fixed `firecracker.json`, `firecracker.api.socket`, `firecracker.pid`, `firecracker.log`, `firecracker.console.log`, and `firecracker.metrics` leaves | +| Console/vsock runtime | Fixed `console.sock`, `console.pid`, and `vsock.sock` leaves | +| Volume | `volumes/.` | +| Kernel | `kernels/` | +| Snapshot | `snapshots//{rootfs.img,memory,vmstate}` | +| Durable image | `images/.zst` | +| Image staging | `images/staging//{source.raw,rootfs.img}` | + +Warm-cache files are unprivileged accelerators and never root authority. Existing database path columns may remain as +derived display/local metadata during the clean break, but privileged handlers reconstruct these names from typed +identities and never authorize a path from SQLite. -The launch path creates a private mount namespace before resource mounts and Jailer execution. Live snapshot and volume -operations enter only the verified registered VM's mount namespace. Cleanup never recursively traverses a tree that may -contain a mount. +Every privileged VM operation is serialized by a root-owned per-VM lock and authorized by a minimal root-owned instance +record. Process actions verify host boot ID, PID, start ticks, every real/effective/saved/filesystem UID and GID, +supplementary groups, expected cgroup membership, mount-namespace identity, and the pinned executable hash against the +exact release. Mutating operations require a pidfd, use `pidfd_send_signal`, and wait through that pidfd; a host without +the required pidfd support fails capability admission instead of falling back to a racy raw PID signal. PID reuse fails +closed. The receiver enforces its compiled group-drop policy and never permits a retained root supplementary group. + +The launch path starts a blocked child in a new private mount namespace. Before the child can perform any externally +mutable effect, the parent pins its pidfd and namespace descriptor, records the complete identity, and retains the launch +lease. The parent then enters that namespace, makes mount propagation private, mounts descriptor-pinned resources, and +releases the child to exec the exact Jailer/Firecracker pair. Process and namespace allocation for a blocked child are +not considered an externally mutable launch effect; mounts, cgroup changes, links, and executable handoff are. Live +snapshot and volume operations reopen and verify `/proc//ns/mnt` while the process is alive and enter only that +namespace. No persistent mount-namespace handle is retained after the operation, because doing so would retain every +mount after process exit. Cleanup never recursively traverses a tree that may contain a mount. State is split by authority and lifetime: | State | Location | Ownership/lifetime | |---|---|---| | User DB and VM artifacts | Existing `MVM_CACHE_DIR`, normally `~/.cache/mvmctl` | User-owned, persistent | -| Trusted release pairs and manifests | `/var/lib/mvmctl/binaries` | Root-owned, persistent | +| Trusted release pairs and manifests | `/var/lib/mvmctl/binaries//` | Root-owned, persistent | | Privileged instance ownership/release records | `/var/lib/mvmctl/instances/` | Root-owned, persistent, minimal | | Jailer chroot directories | `/var/lib/mvmctl/jailer` | Root-owned lifecycle state | -| Per-VM locks and launch handshakes | `/run/mvmctl/` | Root-owned, ephemeral | +| Per-VM locks and launch handshakes | `/run/mvmctl/` | Root-owned, ephemeral; no persistent mount-namespace handles | | Cgroups | `/sys/fs/cgroup/mvmctl/` | Kernel runtime state | | `/run/mvm` | Inside the jail | Jail-visible bind mount, not host canonical state | @@ -199,9 +247,10 @@ Moving durable user state out of `~/.cache/mvmctl` is a separate XDG migration d ### Failure and reconciliation contract -A root-owned instance record exists before the first privileged launch side effect. Every post-spawn failure retains a -live lifecycle handle and invokes a typed abort that verifies identity, terminates the process, checks cgroup emptiness, -and cleans namespace/chroot/instance state in order. +A root-owned instance record exists before the first externally mutable privileged launch side effect. A blocked +private-namespace child may exist first only while an inherited private handshake makes external mutation impossible. +Every post-spawn failure retains a live lifecycle handle and invokes a typed abort that verifies identity, terminates +the process through its pidfd, checks cgroup emptiness, and cleans namespace/chroot/instance state in order. Startup and pre-launch reconciliation compare root instance records, processes, cgroups, and chroot directories. They do not trust the user database as root authorization. Operations are idempotent and either converge to a known state or diff --git a/docs/development/HOW_AGENTS_WRITE_SYSTEM_TESTS.md b/docs/development/HOW_AGENTS_WRITE_SYSTEM_TESTS.md index 6976928f..957e5f25 100644 --- a/docs/development/HOW_AGENTS_WRITE_SYSTEM_TESTS.md +++ b/docs/development/HOW_AGENTS_WRITE_SYSTEM_TESTS.md @@ -478,10 +478,11 @@ Every CLI command and flag that a user can invoke must be tested. This table cla Every L2 test runs **inside a disposable Firecracker VM** with nested KVM, not directly on the host: ``` -Host (no mvm state at all) +Outer host (only controller-owned disposable test resources) │ └── Runner VM (custom base image mvm-test-runner:) - ├── /usr/local/bin/mvm ← built binary, baked in + ├── /opt/mvmctl-test/mvm-candidate ← root-owned staged release candidate + ├── /usr/local/bin/mvm ← installed through host install-system ├── /tests/system/ ← test suite, baked in ├── /mnt/ ← shared RO asset volume │ @@ -495,9 +496,11 @@ Host (no mvm state at all) ### Runner VM Lifecycle ``` -1. Build mvm binary on host -2. Create runner VM from custom base image (mvm-test-runner:) -3. Provision: mount shared asset volume, run mvm init, pull cache hits +1. Build `MVM_CANDIDATE_BINARY` on the host without replacing the installed `MVM_BINARY` controller +2. Derive the image tag from an isolated candidate version probe and create a runner VM from that image +3. Stage the candidate in the runner, invoke `host install-system`, initialize through exact + `/usr/local/bin/mvm`, mount the shared volume, run unprivileged + `/usr/local/bin/mvm init --binary-version 1.16.0`, and pull cache hits through that same installed path 4. Execute pytest directly inside the VM 5. Destroy runner VM when done ``` @@ -526,7 +529,7 @@ network-dependent skips: The official kernel 7.0.11 is **not pre-baked** into the base image — it is pulled on demand by the orchestrator (or pre-seeded in the host asset mirror) -via `mvm kernel pull official:7.0.11 --features nftables,tuntap,kvm,btrfs`. +via `/usr/local/bin/mvm kernel pull official:7.0.11 --features nftables,tuntap,kvm,btrfs`. The shared asset volume contains the asset mirror contents, which includes pre-downloaded kernels, images, and binaries if the host mirror was populated before `--prepare`. @@ -551,6 +554,11 @@ The orchestrator (`scripts/run-system-tests.py`) creates and provisions the VM b running pytest, and destroys it after. The fixture just provides the VM name to tests so they can reference it when calling `_run_mvm(runner_vm, ...)`. +`_run_mvm` always executes exact `/usr/local/bin/mvm`. Do not change it back to PATH lookup and do not copy a candidate +directly onto that path; doing either would bypass the administrator installation path that the L2 suite qualifies. +On the outer host, `MVM_BINARY` is only the resource controller and `MVM_CANDIDATE_BINARY` is the artifact under test; +the orchestrator rejects path, symlink, and hard-link aliases between them. + ### Test Timeout Policy: CLI flags vs subprocess timeouts **These are two distinct concerns — do not conflate them.** @@ -567,8 +575,8 @@ These control how long `subprocess.run()` waits for the `mvm` command to complet def _run_mvm( vm_name: str, *args: str, check: bool = True, timeout: int = 60 ) -> subprocess.CompletedProcess[str]: - """Run an mvm command. vm_name is ignored — already inside test VM.""" - cmd = ["mvm", *args] + """Run the installed mvm CLI. vm_name is ignored inside the test VM.""" + cmd = ["/usr/local/bin/mvm", *args] result = subprocess.run( cmd, capture_output=True, text=True, timeout=timeout + 30, # 90s total by default diff --git a/docs/development/HOW_TO_RUN_SYSTEM_TESTS.md b/docs/development/HOW_TO_RUN_SYSTEM_TESTS.md index e3b71a17..9c8bac4c 100644 --- a/docs/development/HOW_TO_RUN_SYSTEM_TESTS.md +++ b/docs/development/HOW_TO_RUN_SYSTEM_TESTS.md @@ -33,12 +33,14 @@ VMs. The three-tier architecture is documented in (`scripts/run-system-tests.py`) from a custom base image — you don't need to install anything inside them. -### Runner VMs Are Full Bare-Metal Hosts +### Runner VMs Exercise Real Linux and Nested KVM Each runner VM is **not a limited container or lightweight sandbox**. It is a full Firecracker microVM with `nested_virt: true` and **direct KVM access** to -the host CPU virtualization extensions. From the perspective of mvm and the -system tests, each runner VM behaves **identically to bare metal**: +the host CPU virtualization extensions. This exercises real kernel, KVM, +networking, filesystem, and privilege paths, but it is not identical to bare +metal: device exposure, kernel configuration, nesting depth, timing, and +resource limits still differ. - `/dev/kvm` is fully accessible — mvm creates and runs nested VMs inside the runner VM @@ -47,9 +49,9 @@ system tests, each runner VM behaves **identically to bare metal**: - The 7.0.11 kernel has everything compiled in (no modules needed) - tmpfs, overlayfs, btrfs, ext4 — all standard Linux facilities -The only limitations are those imposed by the kernel itself (which is a standard -upstream build with common features enabled). If Linux supports it and the -kernel is compiled with it, runner VMs support it. +Tier 3 exists for cases that cannot be qualified reliably under nesting. Those +tests run directly on the outer host and therefore require a dedicated clean +qualification host or disposable outer VM. ### Special Case: libguestfs / Supermin @@ -88,16 +90,16 @@ go version ```bash groups -# Expected: kvm mvm disk (all three) +# Expected after `mvm host init`: kvm mvm # Verify each explicitly: getent group kvm >/dev/null && echo "kvm group: OK" || echo "kvm group: MISSING" getent group mvm >/dev/null && echo "mvm group: OK" || echo "mvm group: MISSING" -getent group disk >/dev/null && echo "disk group: OK" || echo "disk group: MISSING" ``` -If missing, **ask a human admin** to add you: -`sudo usermod -aG $USER && newgrp ` +`mvm host init` is responsible for creating the `mvm` group and adding the invoking user; verify the result instead of +assuming a partial setup succeeded. If `kvm` is missing, ask a human administrator to grant KVM access, then start a new +login session; do not add the `disk` group for mvmctl. ### 1.4 Host: System Tools @@ -115,74 +117,94 @@ See `docs/DEPENDENCIES.md` for the per-distribution package list. ### 1.5 Orchestrator Prerequisites (Automatic) -The orchestrator (`scripts/run-system-tests.py`) handles all provisioning -automatically — it creates runner VMs from a custom base image, mounts the -shared asset volume, and runs pytest inside each VM. Assets (kernels, images, -binaries) are pre-cached in `~/.cache/mvm-asset-mirror/` on the host. +The orchestrator (`scripts/run-system-tests.py`) handles runner provisioning: +it creates VMs from a custom base image, mounts the shared asset volume, and +runs pytest inside each VM. Release qualification requires a populated writable +host mirror at `MVM_ASSET_MIRROR` (normally `~/.cache/mvm-asset-mirror/`). The +orchestrator copies that point-in-time content into the read-only QA volume; it +does not download missing assets into or repair the host mirror. -The custom base image (`mvm-test-runner:`) is built once during +The custom base image (`mvm-test-runner:`) is built once during `--prepare` and contains: -- The mvm binary at `/usr/local/bin/mvm` +- The release candidate installed at `/usr/local/bin/mvm` through `host install-system` - System tests at `/tests/system/` - Python 3 + pytest + pytest-timeout - qemu-utils, fakeroot, nftables, iptables, zstd - cloud-image-utils (for `--cloud-init-mode iso` tests) - build-essential, bc, bison, flex, libncurses-dev, libssl-dev, libelf-dev, git, curl, dwarves (for `kernel_build` tests) -The orchestrator installs runner-VM contents automatically. The host still requires the release candidate to be -installed and initialized once as shown in section 2 so host-side privileged operations use the same root-owned image. +The orchestrator installs runner-VM contents automatically. The host needs an existing installed controller for outer +resource management, but that controller is not the artifact under test and must not be overwritten by the candidate. --- ## 2. Build the Binary ```bash -# Build the release binary -./scripts/build.sh release +# Untagged RC builds must carry their intended identity explicitly +./scripts/build.sh release --version 0.3.0-rc.1 --output dist/mvm # Verify it exists and is executable test -x dist/mvm && echo "binary: OK ($(dist/mvm --version 2>/dev/null))" -# Install the exact release candidate as the root-owned system executable -sudo ./dist/mvm host install-system -sudo /usr/local/bin/mvm host init +# Select an existing installed controller and keep the release artifact separate export MVM_BINARY=/usr/local/bin/mvm +export MVM_CANDIDATE_BINARY=./dist/mvm ``` -**IMPORTANT**: User-owned binaries are never sudo targets. Set `MVM_BINARY=/usr/local/bin/mvm` when running the -orchestrator so its host-side privileged subprocesses use the same immutable system image that is copied into runner -VMs. The orchestrator's historical default remains a staging-path fallback and must not be used for privilege-boundary -release qualification. +**IMPORTANT**: `MVM_BINARY` only manages disposable outer resources. `MVM_CANDIDATE_BINARY` is installed through the +administrator route only inside runner VMs. The orchestrator rejects aliases and `--rebuild` writes only the candidate. +Use the separate clean-host installation gate in `docs/RC_QA.md` to qualify the final host transition. + +Do not omit `--version` for an untagged RC. The build script otherwise uses `git describe`, which can inherit the +previous release tag and give the new candidate and runner image the wrong version identity. --- ## 3. Prepare Shared Assets -Before the orchestrator can run tests, shared assets must exist on the host. -The orchestrator's `--prepare` mode handles this automatically, but you can -optionally pre-seed the asset mirror to speed up the first run: +Before the orchestrator can run release tests, the required shared assets must +exist and pass their upstream checksum validation in the writable host mirror. +Local focused tests may populate this mirror lazily, but an empty or stale +mirror is not acceptable release evidence. -### 3.1 (Optional) Pre-Seed the Asset Mirror +### 3.1 Prepare the Host Asset Mirror ```bash export MVM_ASSET_MIRROR=~/.cache/mvm-asset-mirror mkdir -p "$MVM_ASSET_MIRROR" # Pull the kernel and assets that the orchestrator will need -mvm kernel pull official:7.0.11 --features nftables,tuntap,kvm,btrfs --default -mvm image pull alpine:3.23 -mvm image pull ubuntu:noble -mvm kernel pull --type firecracker --version v1.15 --default -mvm bin pull firecracker --version 1.16.0 --default +"$MVM_BINARY" kernel pull official:7.0.11 --features nftables,tuntap,kvm,btrfs --default +"$MVM_BINARY" image pull alpine:3.23 --force +"$MVM_BINARY" image pull ubuntu-minimal:24.04 --force +"$MVM_BINARY" image pull ubuntu --version 24.04 --force +"$MVM_BINARY" kernel pull --type firecracker --version v1.15 --default +"$MVM_BINARY" bin pull firecracker --version 1.16.0 --default --force +"$MVM_BINARY" bin pull firecracker --version 1.16.1 --force ``` These pulls populate the local mirror directory. On subsequent runs, each pull -reads from the mirror instead of downloading. +reads the large object from the mirror instead of downloading it again. Small checksum/version metadata requests still +go to the fixed upstream source so cached content can be verified. + +The shared `asset-mirror` volume is a point-in-time, read-only copy. If a pull replaces a stale host object or adds a +missing one, rebuild the QA volume before continuing: + +```bash +MVM_ASSET_MIRROR=~/.cache/mvm-asset-mirror \ + MVM_BINARY=/usr/local/bin/mvm MVM_CANDIDATE_BINARY=./dist/mvm \ + python3 scripts/run-system-tests.py --volume +``` + +Do not expect the read-only guest copy to self-heal. It can detect a checksum mismatch and use an HTTP fallback for that +run, but it cannot persist the replacement. Repair the writable host mirror and reseed the volume instead. ### 3.2 Run `--prepare` ```bash MVM_ASSET_MIRROR=~/.cache/mvm-asset-mirror \ + MVM_BINARY=/usr/local/bin/mvm MVM_CANDIDATE_BINARY=./dist/mvm \ python3 scripts/run-system-tests.py --prepare ``` @@ -200,9 +222,15 @@ To force rebuild of the shared volume, base image, and binary: ```bash MVM_ASSET_MIRROR=~/.cache/mvm-asset-mirror \ - python3 scripts/run-system-tests.py --rebuild --all + MVM_BINARY=/usr/local/bin/mvm MVM_CANDIDATE_BINARY=./dist/mvm \ + python3 scripts/run-system-tests.py \ + --release-qualification --host-direct --rebuild \ + --candidate-version 0.3.0-rc.1 --all ``` +`--rebuild` requires `--candidate-version` unless `HEAD` is one clean checkout with exactly one valid release tag. This +prevents a release branch from inheriting the previous tag through `git describe`. + ### 3.4 Verification Checklist Run these to confirm the host is ready for tests: @@ -211,25 +239,26 @@ Run these to confirm the host is ready for tests: # Host checks echo "KVM: $(test -c /dev/kvm && echo OK || echo MISSING)" echo "Nest: $(cat /sys/module/kvm_intel/parameters/nested 2>/dev/null || echo N/A)" -echo "mvm: $(/usr/local/bin/mvm --version 2>/dev/null)" +echo "controller: $(/usr/local/bin/mvm --version 2>/dev/null)" +echo "candidate: $(./dist/mvm --version 2>/dev/null)" # Asset mirror has content ls ~/.cache/mvm-asset-mirror/ | head -10 # Shared volume exists -mvm volume inspect asset-mirror --json 2>/dev/null \ +"$MVM_BINARY" volume inspect asset-mirror --json 2>/dev/null \ && echo "Volume: OK" \ || echo "Volume: MISSING — run --prepare" # Custom base image exists -mvm image inspect \ - mvm-test-runner:$(/usr/local/bin/mvm --version 2>/dev/null | awk '{print $2}') \ +"$MVM_BINARY" image inspect \ + mvm-test-runner:$(./dist/mvm --version 2>/dev/null | awk '{print $2}') \ --json 2>/dev/null \ && echo "Base image: OK" \ || echo "Base image: MISSING — run --prepare" # Test network exists -mvm network inspect sys-test-net --json 2>/dev/null \ +"$MVM_BINARY" network inspect sys-test-net --json 2>/dev/null \ && echo "Network: OK" \ || echo "Network: MISSING — run --prepare" ``` @@ -247,13 +276,14 @@ Host (your machine) │ ├── TIER 1 — Per-domain VMs (custom base image + shared RO volume) │ ├── mvm vm create --image mvm-test-runner: --volume asset-mirror -│ ├── mount /dev/vdb /mnt && MVM_ASSET_MIRROR=/mnt mvm init +│ ├── mount /dev/vdb /mnt && MVM_ASSET_MIRROR=/mnt /usr/local/bin/mvm init \ +│ │ --non-interactive --binary-version 1.16.0 │ ├── pytest tests/system// ← runs INSIDE the VM │ └── mvm vm rm --force │ ├── TIER 2 — Same as T1 + nested-virt │ ├── mvm vm create --nested-virt --volume asset-mirror -│ ├── mount + init + kernel pull / image pull / bin pull (cache hits) +│ ├── mount + pinned init + kernel pull / image pull / bin pull (cache hits) │ ├── pytest tests/system// ← runs INSIDE the VM │ └── mvm vm rm --force │ @@ -261,17 +291,30 @@ Host (your machine) └── pytest tests/system// ``` -### 4.1 Run All Tests (Release Gate) +### 4.1 Run All Tests (Release Gate Only) + +`--all` includes Tier 3 and may create or destroy outer-host resources. The +runner requires `--host-direct` before any work. For release evidence, +`--release-qualification` also requires an unfiltered full matrix, an +explicit-version rebuild, and exact candidate/controller binary identity before +resource preparation. Run it only on a dedicated clean qualification host or +disposable outer VM where the candidate is installed at `/usr/local/bin/mvm`. +On a workstation with state that must be preserved, use focused T1/T2 domains or +`--tier 1,2`: consent does not prove ownership-safe cleanup. ```bash MVM_ASSET_MIRROR=~/.cache/mvm-asset-mirror \ - python3 scripts/run-system-tests.py --all + MVM_BINARY=/usr/local/bin/mvm MVM_CANDIDATE_BINARY=./dist/mvm \ + python3 scripts/run-system-tests.py \ + --release-qualification --host-direct --rebuild \ + --candidate-version 0.3.0-rc.1 --all ``` ### 4.2 Run Specific Domains ```bash -python3 scripts/run-system-tests.py cli network vm_fresh_env +python3 scripts/run-system-tests.py cli network +python3 scripts/run-system-tests.py --host-direct fresh_env ``` ### 4.3 Run Specific Tiers @@ -284,7 +327,7 @@ python3 scripts/run-system-tests.py --tier 1 python3 scripts/run-system-tests.py --tier 2 # Run Tier 1 then Tier 3 -python3 scripts/run-system-tests.py --tier 1,3 +python3 scripts/run-system-tests.py --host-direct --tier 1,3 ``` ### 4.4 Push Fresh Tests (No Rebuild) @@ -298,24 +341,30 @@ python3 scripts/run-system-tests.py cli --push ### 4.5 Limit Parallel Workers ```bash -python3 scripts/run-system-tests.py --all --workers 2 +python3 scripts/run-system-tests.py --host-direct --all --workers 2 ``` ### 4.6 Orchestrator Flags Reference | Flag | Default | Description | |------|---------|-------------| -| _domains_ (positional) | — | Specific domains to test (e.g., `cli network nested_virt`) | -| `--all` | `false` | Run all T1 + T2 + T3 domains | +| _domains_ (positional) | — | Specific domains to test (e.g., `cli network`) | +| `--all` | `false` | Run all T1 + T2 + host-direct T3 domains; dedicated clean qualification host only | +| `--host-direct` | `false` | Required acknowledgment before selecting any Tier 3 domain or `--all`; does not guarantee isolation or safe cleanup | +| `--release-qualification` | `false` | Require the unfiltered full-matrix release identity gate; requires `--all --host-direct --rebuild --candidate-version` | | `--tier` | — | Comma-separated tier numbers; executed in given order (e.g., `--tier 1,3`) | | `--workers` | `4` | Maximum parallel VMs | -| `--rebuild` | `false` | Build binary + rebuild shared volume + rebuild base image + run prepare | +| `--rebuild` | `false` | Build only `MVM_CANDIDATE_BINARY`, rebuild shared volume/base image, and run prepare; never replace `MVM_BINARY` | +| `--candidate-version` | — | Exact `X.Y.Z` candidate identity for `--rebuild`; required unless `HEAD` is one clean, exact release tag | | `--volume` | `false` | Rebuild only the shared asset volume (`asset-mirror`) | -| `--image` | `false` | Build binary + ensure shared volume exists + rebuild the custom base image (`mvm-test-runner:`) | +| `--image` | `false` | Ensure the shared volume exists and rebuild the custom base image from the existing candidate | | `--prepare` | `false` | Validate provisioning pipeline (ensure shared volume + base image exist, build from scratch if missing, run T1/T2 smoke tests) | | `--push` | `false` | Push test files into each VM before running (overrides baked-in tests) | | `--skip-volume-check` | `false` | Skip shared volume existence check (assume it exists) | +Choose exactly one test selector: positional domains, `--tier`, or `--all`. Unknown names, repeated domains/tiers, and +selected domains with no registered test files fail before binary probes, builds, or resource mutation. + --- ## 5. Interpret Results @@ -352,6 +401,9 @@ pass. A skip is treated as a release blocker — it means either: - The test's skip condition is too broad (tighten the condition) Before signing off an RC, every test file must produce `X passed, 0 failed, 0 skipped`. +The current orchestrator's final domain summary does not independently count +pytest skips/xfails/xpasses. Until it does, QA must inspect and archive every +per-domain pytest summary; `[PASS]` at the domain level alone is insufficient. ### 5.2 What a Skip Means @@ -371,8 +423,8 @@ KVM) is a **release blocker** — fix the environment, not the code. | `iptables` errors | `mvm host init` not run | Run `sudo /usr/local/bin/mvm host init` | | VM creation hangs | Binary not built or missing | Re-run `./scripts/build.sh release` | | `bridge already exists` | Stale bridges from previous run | Clean up with `mvm network rm --force` | -| `Text file busy` on service binary | Stale service processes | `killall -9 mvm-console-relay mvm-nocloud-server mvm-provision` | -| `pending migrations detected` | DB schema mismatch after binary update | `rm -f ~/.cache/mvmctl/mvmdb.db && mvm init --non-interactive --skip-host --skip-network` then re-pull resources | +| `Text file busy` on service binary | Stale disposable runner or base image | Destroy the affected runner through the outer controller and rebuild the base image; do not kill host processes by basename | +| `pending migrations detected` | Stale disposable runner/base image built with a different schema | Rebuild the candidate runner image with `--image --prepare`; do not delete a workstation database as test cleanup | | All tests skip with "KVM not available" | KVM not accessible | Check `/dev/kvm` permissions, user groups | | High skip ratio (>10%) | Missing dependencies | Run the prerequisite check (section 1) | diff --git a/docs/system-test-architecture.md b/docs/system-test-architecture.md index 3ae6150a..2142c287 100644 --- a/docs/system-test-architecture.md +++ b/docs/system-test-architecture.md @@ -35,14 +35,16 @@ Host (your machine) │ ├── TIER 1 — One VM per domain from custom base image + shared volume │ ├── mvm vm create --image mvm-test-runner: --volume asset-mirror -│ ├── mount /dev/vdb /mnt && MVM_ASSET_MIRROR=/mnt mvm init --non-interactive +│ ├── mount /dev/vdb /mnt && MVM_ASSET_MIRROR=/mnt /usr/local/bin/mvm init \ +│ │ --non-interactive --binary-version 1.16.0 │ ├── pytest tests/system// ← runs INSIDE the VM │ └── mvm vm rm --force │ ├── TIER 2 — Same as T1 + nested-virt + kernel for Firecracker-in-VM tests │ ├── mvm vm create --image mvm-test-runner: --nested-virt --volume asset-mirror -│ ├── mount /dev/vdb /mnt && MVM_ASSET_MIRROR=/mnt mvm init --non-interactive -│ ├── MVM_ASSET_MIRROR=/mnt mvm kernel pull / image pull / bin pull (cache hits) +│ ├── mount /dev/vdb /mnt && MVM_ASSET_MIRROR=/mnt /usr/local/bin/mvm init \ +│ │ --non-interactive --binary-version 1.16.0 +│ ├── MVM_ASSET_MIRROR=/mnt /usr/local/bin/mvm kernel pull / image pull / bin pull (cache hits) │ ├── pytest tests/system// ← runs INSIDE the VM │ └── mvm vm rm --force │ @@ -54,12 +56,13 @@ Host (your machine) | Decision | Rationale | |----------|-----------| -| **Custom base image** | A single `mvm-test-runner:` image is built during `--prepare`. Contains mvm binary, system tests, and all OS deps (`python3-pytest`, `qemu-utils`). Built once, cached by version. | -| **Tests run inside the VM** | The orchestrator runs `mvm exec -- python3 -m pytest ...` inside the test VM. Tests call `mvm` directly via `subprocess.run` — no vsock proxy. | -| **`conftest.py` is simple** | `_run_mvm` is a thin wrapper around `subprocess.run(["mvm", *args])`. The `runner_vm` fixture gets the VM name from `MVM_TEST_VM` env var (set by the orchestrator). | +| **Custom base image** | A single `mvm-test-runner:` image is built during `--prepare`. The release candidate is installed through `host install-system`; the image also contains the system tests and OS dependencies. Built once, cached by candidate version. | +| **Tests run inside the VM** | The orchestrator runs `mvm exec -- python3 -m pytest ...` inside the test VM. Tests invoke the exact installed `/usr/local/bin/mvm` via `subprocess.run` — no PATH substitution or vsock proxy. | +| **`conftest.py` is simple** | `_run_mvm` is a thin wrapper around `subprocess.run(["/usr/local/bin/mvm", *args])`. The `runner_vm` fixture gets the VM name from `MVM_TEST_VM` (set by the orchestrator). | | **Shared RO volume on all tiers** | The `asset-mirror` volume is attached to every VM (T1 and T2). Mounting is cheap — `--shareable --read-only` means no per-VM state. | | **`--push` flag** | Overrides baked-in tests by copying `tests/system/` fresh into the VM. Useful during test development without rebuilding the base image. | -| **`--rebuild` flag** | Forces rebuild of the mvm binary, shared volume, and custom base image. Removes volume and image first, then recreates from scratch. | +| **Controller/candidate qualification** | T1/T2 use a separate installed outer controller and candidate installed only inside disposable runners. Tier 3 is host-direct; release mode therefore requires exact content/version identity between the candidate and root-owned `/usr/local/bin/mvm` before resource preparation. | +| **`--rebuild` flag** | Builds only `MVM_CANDIDATE_BINARY`, then rebuilds the shared volume and candidate-versioned custom base image. It never replaces `MVM_BINARY`. | --- @@ -71,13 +74,14 @@ The custom base image `mvm-test-runner:` is built once and cached: ``` 1. Create builder VM from ubuntu-minimal:noble -2. mvm cp mvm-binary → /usr/local/bin/mvm +2. Stage the release candidate outside `/usr/local/bin` and run `sudo host install-system` 3. mvm cp tests/system/ → /tests/ 4. apt-get install python3-pytest qemu-utils fakeroot nftables iptables zstd htop build-essential bc bison flex libncurses-dev libssl-dev libelf-dev git curl dwarves cloud-image-utils 5. pip3 install pytest-timeout 6. Add runner to mvm, kvm groups -7. chown -R runner:runner /tests -8. Mount shared volume, mvm init, pull assets (images, kernels, binaries) from mirror +7. Initialize through exact `sudo /usr/local/bin/mvm host init`; chown `/tests` to runner +8. Mount the shared volume, run unprivileged `/usr/local/bin/mvm init --binary-version 1.16.0`, then pull remaining + images and kernels through the same exact installed binary 9. apt-get clean 10. mvm vm stop (graceful stop) 11. mvm image import mvm-test-runner: --default --skip-optimization @@ -88,7 +92,8 @@ The custom base image `mvm-test-runner:` is built once and cached: | Item | Location in image | Purpose | |------|-------------------|---------| -| mvm binary | `/usr/local/bin/mvm` | The exact version being tested | +| installed mvm binary | `/usr/local/bin/mvm` | The release candidate installed through the administrator bootstrap route | +| staged release candidate | `/opt/mvmctl-test/mvm-candidate` | Root-owned source used only to exercise `host install-system`; tests do not use it as the CLI | | System tests | `/tests/system//` | All `tests/system/` files | | python3-pytest, pytest-timeout (pip) | Test runners | Test runner and timeout plugin | | qemu-utils | System package | Volume/disk operations in T2 tests | @@ -99,7 +104,8 @@ The custom base image `mvm-test-runner:` is built once and cached: ### Caching -- Rebuilt only when the mvm version changes (version string from `mvm --version`). +- Rebuilt only when the candidate version changes (an isolated version probe executes `MVM_CANDIDATE_BINARY`, never + the outer controller). - `--rebuild` forces rebuild (removes image first, then recreates). - Image is versioned as `mvm-test-runner:` — multiple versions can coexist. @@ -121,16 +127,21 @@ All T1/T2 runner VMs are Firecracker microVMs created with **Firecracker v1.16** ### Creating the Volume (`--prepare` or `ensure_shared_volume`) -The host asset mirror (`~/.cache/mvm-asset-mirror/`) is the source of truth. A shareable read-only volume is created once and reused across all test runs. +The host asset mirror (`~/.cache/mvm-asset-mirror/`) is the source of truth. A shareable read-only volume is created +once and reused across all test runs. The orchestrator accepts only the exact canonical +`${MVM_CACHE_DIR:-~/.cache/mvmctl}/volumes/asset-mirror.raw` backing path. Missing stale records are rebuilt; an +unexpected path fails closed without mutation. + +This volume is a point-in-time copy, not a writable cache. A checksum mismatch may make a guest fall back to the +upstream URL, but the guest cannot persist the corrected object into the read-only volume. Repair the writable host +mirror first, then use `--volume` to recreate the shared copy. Checksum/version metadata may still be fetched upstream +even when the large artifact is served locally. ```bash mvm volume create asset-mirror 6G --shareable --read-only --writeback --format raw -# Populated via sudo loop mount: -sudo mkfs.ext4 -F -sudo mount -o loop /mnt/.mvm-asset-populate -sudo cp -r ~/.cache/mvm-asset-mirror/* /mnt/.mvm-asset-populate/ -sudo chmod -R a+rX /mnt/.mvm-asset-populate -sudo umount /mnt/.mvm-asset-populate +# Populated without sudo or a host mount. The runner pins the canonical regular +# file with component-by-component O_NOFOLLOW traversal and passes that fd: +mkfs.ext4 -F -d ~/.cache/mvm-asset-mirror /proc/self/fd/ ``` ### Volume Semantics @@ -191,7 +202,8 @@ mvm vm create --image mvm-test-runner: \ mvm exec --user runner --timeout 60 -- \ "sudo mkdir -p /mnt && sudo mount /dev/vdb /mnt && \ - MVM_ASSET_MIRROR=/mnt mvm init --non-interactive" + MVM_ASSET_MIRROR=/mnt /usr/local/bin/mvm init --non-interactive \ + --binary-version 1.16.0" ``` ### Test Execution @@ -242,13 +254,13 @@ After init, asset pulls are cache hits against the shared volume: ```bash mvm exec --user runner --timeout 120 -- \ - "MVM_ASSET_MIRROR=/mnt mvm kernel pull --type firecracker --version v1.15 --default" + "MVM_ASSET_MIRROR=/mnt /usr/local/bin/mvm kernel pull --type firecracker --version v1.15 --default" mvm exec --user runner --timeout 120 -- \ - "MVM_ASSET_MIRROR=/mnt mvm image pull alpine:3.23" + "MVM_ASSET_MIRROR=/mnt /usr/local/bin/mvm image pull alpine:3.23" mvm exec --user runner --timeout 120 -- \ - "MVM_ASSET_MIRROR=/mnt mvm image pull ubuntu:noble" + "MVM_ASSET_MIRROR=/mnt /usr/local/bin/mvm image pull ubuntu:noble" mvm exec --user runner --timeout 120 -- \ - "MVM_ASSET_MIRROR=/mnt mvm bin pull firecracker --version 1.16.0 --default --force" + "MVM_ASSET_MIRROR=/mnt /usr/local/bin/mvm bin pull firecracker --version 1.16.0 --default --force" ``` --- @@ -260,7 +272,6 @@ mvm exec --user runner --timeout 120 -- \ - `vm/test_vm_nested_isolated.py` — triple-nested VM tests. - `vm/test_vm_snapshot_load.py` — snapshot lifecycle. - `vm/test_vm_snapshot_rootfs_independence.py` — restored VM rootfs is independent of source VM. -- `vm/test_vm_snapshot_rootfs_independence.py` — restored VM rootfs is independent of source VM. - `volume/test_volume_hotplug.py` — PCI hotplug (requires Firecracker dev-preview, does not work reliably nested). - `kernel/test_kernel.py` — kernel build tests (need full KVM host access). - `cp/test_cp.py` — vsock agent file copy (some paths reject nested virt). @@ -275,6 +286,15 @@ These tests **create VMs as part of their own logic**. Running them inside a run ### Execution +Tier 3 runs on the outer host and may create or destroy host VMs, kernels, +images, networks, and volumes. The orchestrator rejects Tier 3 before work +unless `--host-direct` acknowledges outer-host mutation. Release mode also +requires an unfiltered full matrix, explicit-version rebuild, safe canonical +system-binary metadata, and candidate/controller content and version identity. +Run it only on a dedicated clean qualification host or disposable outer VM +after `host install-system`. The remaining Task 17 inventory/cleanup work still +blocks safe evidence on a workstation with pre-existing mvmctl resources. + ```bash # Run directly on the host cd tests/system @@ -286,7 +306,8 @@ MVM_ASSET_MIRROR=~/.cache/mvm-asset-mirror \ ## 8. How Tests Call `mvm` -Since tests run INSIDE the VM (Tier 1/2), they call `mvm` directly. +Since tests run inside the VM (Tier 1/2), they invoke the exact installed system binary directly. PATH lookup is not +release evidence because it could select a developer or stale binary. ### Inside the test VM @@ -294,7 +315,9 @@ Since tests run INSIDE the VM (Tier 1/2), they call `mvm` directly. import subprocess def test_help(): - result = subprocess.run(["mvm", "help"], capture_output=True, text=True) + result = subprocess.run( + ["/usr/local/bin/mvm", "help"], capture_output=True, text=True + ) assert result.returncode == 0 assert "Usage:" in result.stdout ``` @@ -307,8 +330,8 @@ The conftest provides `_run_mvm` as a convenience wrapper: def _run_mvm( vm_name: str, *args: str, check: bool = True, timeout: int = 60 ) -> subprocess.CompletedProcess[str]: - """Run an mvm command. vm_name is ignored — already inside test VM.""" - cmd = ["mvm", *args] + """Run the installed mvm CLI. vm_name is ignored inside the test VM.""" + cmd = ["/usr/local/bin/mvm", *args] return subprocess.run( cmd, capture_output=True, text=True, timeout=timeout + 30, @@ -316,7 +339,10 @@ def _run_mvm( ) ``` -The `vm_name` parameter is accepted for compatibility but NOT used — since we're already inside the target VM, all commands run locally. The orchestrator sets `MVM_TEST_VM` to the VM name for test introspection if needed. +The `vm_name` parameter is retained by the shared helper but is not used because commands already run inside the target +VM. The orchestrator sets `MVM_TEST_VM` to the VM name for test introspection if needed. The outer orchestrator may use +the requested release-candidate artifact, but every T1/T2 assertion runs the artifact installed at +`/usr/local/bin/mvm`. --- @@ -327,14 +353,25 @@ The `vm_name` parameter is accepted for compatibility but NOT used — since we' ```bash # Build base image + smoke test MVM_ASSET_MIRROR=~/.cache/mvm-asset-mirror \ + MVM_BINARY=/usr/local/bin/mvm MVM_CANDIDATE_BINARY=./dist/mvm \ python3 scripts/run-system-tests.py --prepare -# Run all domains (T1 + T2 + T3) +# Run all domains only on a dedicated clean qualification host/outer VM MVM_ASSET_MIRROR=~/.cache/mvm-asset-mirror \ - python3 scripts/run-system-tests.py --all + MVM_BINARY=/usr/local/bin/mvm MVM_CANDIDATE_BINARY=./dist/mvm \ + python3 scripts/run-system-tests.py --host-direct --all + +# Release qualification on a dedicated clean qualification host/outer VM +MVM_BINARY=/usr/local/bin/mvm \ + MVM_CANDIDATE_BINARY=./dist/mvm \ + MVM_ASSET_MIRROR=~/.cache/mvm-asset-mirror \ + python3 scripts/run-system-tests.py \ + --release-qualification --host-direct --rebuild \ + --candidate-version 0.3.0-rc.1 --all # Run specific domains -python3 scripts/run-system-tests.py cli network vm_fresh_env +python3 scripts/run-system-tests.py cli network +python3 scripts/run-system-tests.py --host-direct fresh_env # Push fresh test files (no rebuild needed) python3 scripts/run-system-tests.py cli --push @@ -342,32 +379,34 @@ python3 scripts/run-system-tests.py cli --push # Run only specific tiers python3 scripts/run-system-tests.py --tier 1 python3 scripts/run-system-tests.py --tier 2 -python3 scripts/run-system-tests.py --tier 3 +python3 scripts/run-system-tests.py --host-direct --tier 3 python3 scripts/run-system-tests.py --tier 2,1 ``` ### Script Flow ``` -1. Parse args -2. --prepare mode: - a. Get mvm version +1. Parse args and reject Tier 3 without `--host-direct` before any work +2. In `--release-qualification`, require the unfiltered full-matrix flags, rebuild the explicit-version candidate, and + verify exact candidate/controller metadata, descriptors, SHA-256, and strict version before resource preparation +3. --prepare mode: + a. Probe the candidate version in isolated cache/config/temp directories b. Ensure shared volume + network - c. Build custom base image mvm-test-runner: (or skip if cached) + c. Install the staged candidate through `host install-system`, then build the custom base image (or skip if cached) d. Create T1 smoke VM → mount volume → mvm init → destroy e. Create T2 smoke VM → mount volume → mvm init → pull cache hit → destroy -3. Normal run mode: - a. Detect mvm version +4. Normal run mode: + a. Detect the candidate version without invoking the outer controller b. Ensure shared volume + network c. Select domains based on args/tier flags d. T1 + T2 in parallel (ThreadPoolExecutor): - Create VM → mount volume → mvm init → + Create VM from the candidate-qualified base image → mount volume → pinned mvm init → [--push: mvm cp tests/system] → mvm exec -- python3 -m pytest ... → destroy VM - e. T3 sequentially on host: + e. T3 sequentially on the dedicated clean qualification host: pytest tests/system// -4. Print summary: X passed, Y failed, Z total +5. Print summary: X passed, Y failed, Z total ``` See [RC_QA.md](RC_QA.md) for the full flags reference, setup workflow, and @@ -397,6 +436,7 @@ tests/system/ ├── kernel/test_kernel.py ← TIER 1/2/3 mixed (basic ops TIER 1, pull-advanced-class TIER 2, kernel build classes TIER 3) ├── kernel/test_kernel_import.py ← TIER 1 ├── host/test_host.py ← TIER 1 +├── host/test_system_install.py ← TIER 1 (installed release-candidate contract) ├── run/test_run.py ← TIER 1 │ ├── volume/test_volume.py ← TIER 2 (needs nested virt) @@ -438,6 +478,7 @@ tests/system/ | nftables | 2 | `test_nftables.py` | Nftables firewall backend and atomic rule sync — needs nested virt | | policies | 2 | `test_policies.py` | Routed service-access allow/default-deny behavior for nftables and iptables — needs nested virt | | host | 1 | `test_host.py` | Host info/status | +| system_install | 1 | `test_system_install.py` | Root-owned system install path, metadata, routing, and failure safety | | run | 1 | `test_run.py` | Service subprocesses | | volume | 2 | `test_volume.py` | Volume lifecycle | | vm_lifecycle | 2 | `test_vm_lifecycle.py` | VM create/start/stop/pause | @@ -498,7 +539,8 @@ def test_hotplug_re_attach_after_hot_unplug(self, ...): 1. **Only xfail when the limitation is in a downstream dependency** (Firecracker, kernel, guest OS). Never xfail for bugs in mvmctl's own code. 2. **Always document the exact failure reason** so the marker can be removed when the dependency is fixed. -3. **An `xfail` is NOT a skip** — the test still runs and collects result data. If the test unexpectedly passes, `XPASS` alerts the team to remove the marker. +3. **An `xfail` is not silent** — the test still runs and collects result data. However, both XFAIL and XPASS block RC + sign-off until the limitation is resolved or removed from the release requirements. 4. **No `pytest.skip()`** — conditional skipping is banned because it hides regressions. Use `xfail` to document known failures or restructure the test to expect the known error. --- @@ -563,9 +605,11 @@ When a hotplug or PCI device operation fails, the recommended investigation flow Every test file in `tests/system/` MUST: -1. **Tests call `mvm` directly** — Use `subprocess.run(["mvm", ...])` or `_run_mvm(...)`. No vsock proxy, no `mvm exec` nesting. +1. **Tests call the installed CLI directly** — Use `_run_mvm(...)` or exact + `subprocess.run(["/usr/local/bin/mvm", ...])`. No PATH lookup, vsock proxy, or `mvm exec` nesting. -2. **Tier 3: Direct host calls allowed** — Tests that create VMs or check host state run directly on the host. +2. **Tier 3: Direct qualification-host calls only** — Tests that create VMs or check host state run directly on a + dedicated clean qualification host or disposable outer VM, never a developer workstation with pre-existing state. 3. **No `os.path.exists` on VM filesystem paths** — Those paths are inside the VM (Tier 1/2). Tier 3 tests may check host paths. diff --git a/tasks/plan.md b/tasks/plan.md index 468df43d..5801d57e 100644 --- a/tasks/plan.md +++ b/tasks/plan.md @@ -43,11 +43,11 @@ firewall backends, host-global TAP topology, public `mvm run jailer` / `mvm run | State | Location | Authority | |---|---|---| | User configuration, SQLite DB, and VM artifacts | Existing `MVM_CACHE_DIR` | Untrusted for root authorization | -| Trusted Firecracker/Jailer pairs | `/var/lib/mvmctl/binaries` | Root-owned persistent | +| Trusted Firecracker/Jailer pairs | `/var/lib/mvmctl/binaries//` | Root-owned persistent | | VM ownership/lifecycle records | `/var/lib/mvmctl/instances/` | Root-owned persistent | | Network ownership/topology records | `/var/lib/mvmctl/networks/` | Root-owned persistent | | Global capacity allocations | `/var/lib/mvmctl/allocations` | Root-owned persistent | -| Locks, launch handshakes, namespace handles | `/run/mvmctl` | Root-owned ephemeral | +| Locks and launch handshakes | `/run/mvmctl` | Root-owned ephemeral; no persistent mount-namespace handles | | Jailer chroots | `/var/lib/mvmctl/jailer` | Root-owned lifecycle state | | Cgroups | `/sys/fs/cgroup/mvmctl/` | Kernel runtime state | @@ -57,7 +57,9 @@ Durable user state does not move wholesale to `/run` or `/var/run`. XDG state se - `cmd/mvm/main.go` only recognizes early modes and routes them. - `internal/service/privileged/` owns the envelope, authenticated caller, executable verification, and fixed action - switch. It imports no API or core domain and exposes no raw command/argv/path interface. + switch. It imports no API or core domain and exposes no raw command/argv/effect-path interface. One typed managed-cache + locator may identify the caller's persistent namespace; every effect below it is selected by typed IDs and fixed + receiver-derived basenames. - Capability modules under `internal/service/{jailer,network,firewall,loopmount,host}/` own typed privileged effects. - `pkg/api/` remains the sole orchestrator of multiple core domains. - Core domains never import sibling `internal/core/*` packages. @@ -117,7 +119,8 @@ cleaning ───── Complete ─────> cleaned ``` `registered` means launch identity is durable; it does not claim the process is healthy. Process health is derived by -verifying PID, start ticks, cgroup membership, release, and namespace identity. Failed cleanup remains `cleaning`. +verifying host boot ID, PID, start ticks, all UID/GID identities and supplementary groups, cgroup membership, pinned +mount-namespace identity, and the executable hash against the exact release. Failed cleanup remains `cleaning`. `cleaned` is a persistent ownership tombstone and may be relaunched only by the same UID. The global lock order is: @@ -130,10 +133,22 @@ Operations may acquire a suffix or subset but never an earlier class while holdi are established by descriptor-enumerating all numeric UID record directories while holding the index lock. A foreign, duplicate, corrupt, unreadable, or inconsistent claim fails closed. Lock files are never unlinked. +Release locks are keyed by the canonical store slot `(version, architecture)`, not by caller-observed binary hashes. +Reference matching remains against the full release identity. Competing identities for one store slot must therefore +serialize before install, replace, remove, launch, or reference inspection. + `RegisterLaunch` establishes the durable record before the first network or launch effect and returns with the VM lock held. A later typed network step may therefore acquire only the later network lock while retaining that launch lease. Task 4 owns release, index, and VM locking; it does not import network code or predeclare a network implementation. +The mount-namespace launch seam uses a blocked child: the child creates its private namespace and waits on an inherited +private handshake without performing external mutation. The parent pins the pidfd and namespace descriptor, persists +the complete launch identity, then enters the namespace, marks propagation private, mounts named resources, and releases +the child to exec. Here, "first launch effect" means the first externally mutable effect; allocating the blocked process +and namespace is preparation. Mutating process operations require pidfds and never fall back to `kill(pid)`. Live mount +operations reopen and verify `/proc//ns/mnt`; a persistent mount-namespace handle is prohibited because it would +retain the namespace's mounts after process exit. + Atomic record replacement uses a random exclusive temporary file, bounded writes, root ownership and exact modes, file `fsync`, checked close, `renameat`, and parent-directory `fsync`. Pre-rename failure preserves the old record. Post-rename failure returns details that distinguish `record_replaced` and `durability_uncertain`; the caller starts diff --git a/tasks/todo.md b/tasks/todo.md index ae3acf86..eb5a51af 100644 --- a/tasks/todo.md +++ b/tasks/todo.md @@ -35,15 +35,19 @@ This checklist implements `tasks/plan.md`. Go production and L0/L1 work belongs ### Task 2: Verify the implemented system installer through Python **Owner:** qa-engineer -**Status:** In progress +**Status:** Complete **Dependencies:** Implemented system installer -- [ ] Add an installed-release-candidate fixture that stages the candidate outside `/usr/local/bin`. -- [ ] Invoke `sudo host install-system` and then `sudo /usr/local/bin/mvm host init`. -- [ ] Prove canonical path, root ownership, exact mode, safe ancestors, hash equality, and atomic replacement. -- [ ] Prove malformed install invocation, symlink target, partial failure, and system self-update refusal. -- [ ] Register the focused Python domain in the orchestrator and coverage matrix. -- [ ] Run collection and the focused domain on the prepared runner. +- [x] Add an installed-release-candidate fixture that stages the candidate outside `/usr/local/bin`. +- [x] Invoke `sudo host install-system` and then `sudo /usr/local/bin/mvm host init`. +- [x] Prove canonical path, root ownership, exact mode, safe ancestors, hash equality, and atomic replacement. +- [x] Prove malformed install invocation, symlink target, partial failure, and system self-update refusal. +- [x] Register the focused Python domain in the orchestrator and coverage matrix. +- [x] Run collection and the focused domain on the prepared runner. + +Qualification evidence: the candidate `0.3.0-rc.1` was installed and exercised only inside disposable nested-virt +runner VMs. The focused `system_install` domain passed all 10 cases on 2026-08-23. The outer host controller remained +byte-for-byte unchanged, and the pre-existing user VM was not modified. The final marker-only sudoers assertion is deferred until Task 14; current sudoers remains transitional. @@ -105,6 +109,8 @@ The final marker-only sudoers assertion is deferred until Task 14; current sudoe rename failure, cancellation, stale lock reuse, corrupt record, and temp cleanup. - [x] Focused race test, format, golines, vet, and Go tests pass. - [x] Architect reviews actual interface, state schema, lock order, imports, and error details before commit. +- [x] Key the release lock by canonical `(version, architecture)` store slot rather than binary hashes; prove two + different identities for one slot serialize. **Expected files:** fixed root constants; private instance types, codec, descriptor store, lock, authority, and focused tests under `internal/service/jailer/`. No handler, transport, CLI, API, core-domain, or network changes. @@ -114,17 +120,34 @@ under `internal/service/jailer/`. No handler, transport, CLI, API, core-domain, **Owner:** engineer **Dependencies:** Task 4 +- [ ] Preserve custom `MVM_CACHE_DIR` through one typed managed-cache locator; pin it from `/`, verify caller ownership, + safe components/mode/mount topology, and reject every raw effect path. +- [ ] Bind the pinned cache device/inode/mount identity into the root-owned instance record and require later operations + to re-pin the same identity. +- [ ] Land canonical producer names before descriptor pinning: + - [x] ID-based `volumes/.` storage. + - [ ] Fixed `rootfs.img`, cloud-init, Firecracker, console, and vsock VM leaves. + - [ ] Fixed snapshot leaves and a managed ID-scoped image-provisioning staging subtree. +- [ ] Define operation-specific non-empty/maximum-size and access-mode policies before coding; a caller- or DB-provided + expected size is an equality check, not authorization for an unbounded resource. - [ ] Open user cache/VM/image/kernel/volume/snapshot resources once with beneath/no-symlink/no-magic-link resolution. - [ ] Verify caller ownership, file type, fixed basenames, size, and allowed access mode on pinned descriptors. - [ ] Never validate a path and later reopen it by pathname. -- [ ] Return DomainError partial-state details for close/fsync/cleanup failures. +- [ ] Keep leases private to the owning service package, expose no generic FD/path accessor, and release retained + descriptors in reverse order with checked `context.WithoutCancel` cleanup. +- [ ] Return DomainError partial-state details for close/fsync/cleanup failures without replacing the primary code, + class, entity, or details. - [ ] Race tests continuously replace every user-owned ancestor and leaf with symlinks/renames. +- [ ] Task 7 consumes same-package Jailer leases; Task 13 consumes a separate private loopmount image lease. Neither + consumer may reconstruct or accept `/proc/self/fd` paths from the request. ### Task 6: Implement independent trusted release authority **Owner:** engineer -**Dependencies:** Tasks 4 and 5 +**Dependencies:** Task 4 +- [ ] Store exact releases under `/var/lib/mvmctl/binaries//` with fixed Firecracker, Jailer, + and strict root-owned manifest leaves. - [ ] Privileged code constructs the fixed official release/checksum locations from validated version and architecture. - [ ] Caller-provided archive bytes are not their own authority. - [ ] Bounded extraction rejects traversal, links, devices, duplicates, unexpected members, and size/count overflow. @@ -135,13 +158,24 @@ under `internal/service/jailer/`. No handler, transport, CLI, API, core-domain, ### Task 7: Add private mount and process identity primitives **Owner:** engineer -**Dependencies:** Tasks 4 and 5 - -- [ ] Unshare a private mount namespace before launch mounts and mark propagation private. -- [ ] Verify PID, start ticks, cgroup, owner, release, and namespace inode before signal or setns. -- [ ] Use pidfd when available and fail closed on PID reuse. +**Foundation dependency:** Task 4 +**Full acceptance dependencies:** Tasks 5, 6, and 8 + +- [ ] Extend the strict instance record once with owner GID, host boot ID, mount-namespace device/inode, and Task 5's + pinned cache identity; bump the schema version and reject old/partial records. +- [ ] Launch a blocked child in a private mount namespace, pin its pidfd and namespace FD, persist the complete record + before externally mutable effects, mark propagation private, then release the child only after named mounts succeed. +- [ ] Verify PID, boot ID, start ticks, all UID/GID identities, supplementary groups, exact cgroup, namespace inode, + process liveness, and pinned executable hash against the Task 6 release manifest before signal or `setns`. +- [ ] Require pidfd support for every mutating operation, signal only with `pidfd_send_signal`, wait through the pidfd, + and fail host capability admission instead of falling back to raw PID signaling. +- [ ] Reopen and pin `/proc//ns/mnt` for each verified live operation; do not persist a mount-namespace handle that + would retain mounts after process exit. +- [ ] Lock the OS thread for `setns`, pin/restore/verify the original host namespace, and terminate the short-lived + privileged process if restoration fails rather than returning a contaminated thread to the Go runtime. - [ ] Live snapshot/volume operations enter only the verified VM mount namespace. -- [ ] Cleanup never recursively traverses a tree that may contain a mount. +- [ ] Cleanup parses and verifies mount state, unmounts only receiver-derived fixed leaves, requires cgroup emptiness, + and removes a descriptor-relative allowlist bottom-up; never recursively traverse a tree that may contain a mount. - [ ] L1 fault tests cover source/target replacement, PID mismatch, process exit, setns, mount, unmount, and cleanup. ## Phase 2: Final Jailer and networking paths @@ -226,6 +260,19 @@ under `internal/service/jailer/`. No handler, transport, CLI, API, core-domain, - [ ] Move provisioning/mount operations behind typed descriptor-pinned actions. - [ ] Remove public `mvm run provision` and arbitrary unmount behavior. - [ ] Move required module/sysctl/host mutations behind exact compiled allowlists. +- [ ] Keep `host init`, `host reset`, group membership, and sudoers changes as explicit administrator operations; do not + route authorization-boundary changes through the passwordless operator dispatcher. +- [ ] Replace `HostInit`'s `any` / `map[string]any` result with one typed result and one consistent interaction channel. +- [ ] Make administrator host setup/reset fail closed on every mutation and state-write error; remove ignored group, + sysctl, KVM, firewall, repository, and ownership-repair failures. +- [ ] Dispatch administrator `host init` / `host reset` before ordinary application initialization, with a fixed + allowlisted environment; never open user SQLite or honor user-controlled cache/config/temp roots first. +- [ ] Make `host clean` and `host reset` fail closed when running-VM discovery, network discovery, teardown, or restore + fails; never interpret an unreadable inventory as an empty one. +- [ ] Make the init coordinator propagate sudo, dispatch, network-sync, cache, guestfs, binary, and state-write + failures; success/readiness must require every non-skipped required step. +- [ ] Store the administrator mutation journal in root-owned state and never let user SQLite or a recursive cache + `chown` authorize or steer a root host mutation. - [ ] Move any remaining bridge/link/firewall caller to the typed privileged modules. - [ ] Repository search finds no unauthorized raw sudo callsite. - [ ] L1 abuse tests cover arbitrary path, PID, mount, module, sysctl, network, and shell/argv attempts. @@ -279,6 +326,20 @@ under `internal/service/jailer/`. No handler, transport, CLI, API, core-domain, - [ ] Use constrained users with only the product’s intended `mvm` privilege. - [ ] Fix exact configuration restoration and make all teardown failures visible. - [ ] Run each relevant domain twice and assert zero leaked resources. +- [x] Require explicit host-direct consent before selecting Tier 3 or `--all`, and validate it before any runner + mutation; developer-host iterative runs are limited to T1/T2. +- [x] Require an explicit release-qualification mode with exact root-owned `/usr/local/bin/mvm`, safe ancestors, and + descriptor-pinned candidate/controller version and SHA-256 identity before any host resource preparation. +- [ ] Refuse Tier 3 release evidence when the dedicated outer host has pre-existing active VMs, then verify every + selected resource belongs to the qualification run before cleanup. +- [ ] Make cleanup and post-domain absence checks part of the domain result; preserve both primary and cleanup failures, + and never advance tiers while a timed-out worker can still mutate state. +- [x] Reject unknown, duplicate, mixed, zero-domain, and empty-file domain selections before probes or mutation. +- [ ] Parse pytest outcomes and require non-empty collection with zero skips, XFAIL, or XPASS; omitted domains fail + instead of producing release evidence. +- [ ] Ensure the full matrix selects every required domain exactly once, including `exec`, and does not run + `test_vm_fresh_env.py` twice under aliases. +- [ ] Make mirror validation reject pull errors and HTTP fallback; only a verified local-mirror read is a cache hit. - [ ] Run the complete T1/T2/T3 matrix plus clean-host installation and reboot/reconciliation qualification. - [ ] Update `tests/system/COVERAGE_MATRIX.md` and attach commands/results to the release report. From bea24fc87dc4539cc7527f9462dabcc04e7db4ef Mon Sep 17 00:00:00 2001 From: AlanD20 Date: Sun, 23 Aug 2026 15:45:29 +0200 Subject: [PATCH 0024/1457] refactor: close managed kernel output paths --- internal/core/kernel/managed_output_test.go | 292 ++++++++++++++++++++ internal/core/kernel/service.go | 170 +++++++++--- pkg/api/inputs/kernel_pull.go | 32 +-- pkg/api/kernel.go | 11 +- pkg/api/operation.go | 2 +- 5 files changed, 428 insertions(+), 79 deletions(-) create mode 100644 internal/core/kernel/managed_output_test.go diff --git a/internal/core/kernel/managed_output_test.go b/internal/core/kernel/managed_output_test.go new file mode 100644 index 00000000..b0f190ec --- /dev/null +++ b/internal/core/kernel/managed_output_test.go @@ -0,0 +1,292 @@ +package kernel + +import ( + "context" + "errors" + "fmt" + "net/http" + "net/http/httptest" + "os" + "path/filepath" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "mvmctl/internal/lib/crypto" + "mvmctl/internal/lib/model" + "mvmctl/pkg/errs" +) + +func TestImportKernelUsesConfiguredManagedDirectory(t *testing.T) { + root := t.TempDir() + managedDir := filepath.Join(root, "managed", "kernels") + sourcePath := filepath.Join(root, "source-vmlinux") + require.NoError(t, os.WriteFile(sourcePath, []byte("custom-kernel"), 0644)) + callerName := "../../escape/custom" + callerSelectedDir := filepath.Clean(filepath.Join(managedDir, "..", "..", "escape")) + legacyCallerSelectedPath := filepath.Join( + managedDir, + fmt.Sprintf(KernelOutputPattern, callerName, "6.1.0", "x86_64"), + ) + assert.Equal(t, filepath.Join(callerSelectedDir, "custom-6.1.0-x86_64"), legacyCallerSelectedPath) + + repo := &managedOutputRepository{} + service := NewService(repo, managedDir) + item, err := service.ImportKernel( + context.Background(), + callerName, + sourcePath, + "6.1.0", + "x86_64", + false, + ) + require.NoError(t, err) + require.NotNil(t, item) + + wantPath := filepath.Join(managedDir, item.ID) + assert.Equal(t, wantPath, item.Path) + assert.Equal(t, callerName+" 6.1.0", item.Name) + assert.Equal(t, item, repo.item) + assert.FileExists(t, wantPath) + storedID, hashErr := crypto.KernelID(wantPath, item.Version, item.Arch) + require.NoError(t, hashErr) + assert.Equal(t, item.ID, storedID) + assert.NoDirExists(t, filepath.Join(managedDir, "kernels")) + assert.NoDirExists(t, callerSelectedDir) +} + +func TestImportKernelHashesReceiverStagedBytes(t *testing.T) { + root := t.TempDir() + managedDir := filepath.Join(root, "kernels") + sourcePath := filepath.Join(root, "caller-source") + require.NoError(t, os.WriteFile(sourcePath, []byte("original-source"), 0644)) + + service := NewService(&managedOutputRepository{}, managedDir) + service.copyImportedKernel = func(_ context.Context, _, destination string) error { + return os.WriteFile(destination, []byte("bytes-copied-to-store"), 0644) + } + + item, err := service.ImportKernel( + context.Background(), + "custom", + sourcePath, + "6.1.0", + "x86_64", + false, + ) + require.NoError(t, err) + require.NotNil(t, item) + + storedID, err := crypto.KernelID(item.Path, item.Version, item.Arch) + require.NoError(t, err) + assert.Equal(t, storedID, item.ID) + sourceID, err := crypto.KernelID(sourcePath, item.Version, item.Arch) + require.NoError(t, err) + assert.NotEqual(t, sourceID, item.ID) +} + +func TestImportKernelRemovesFailedReceiverStaging(t *testing.T) { + root := t.TempDir() + managedDir := filepath.Join(root, "kernels") + sourcePath := filepath.Join(root, "caller-source") + require.NoError(t, os.WriteFile(sourcePath, []byte("source"), 0644)) + + copyErr := errors.New("copy failed") + service := NewService(&managedOutputRepository{}, managedDir) + service.copyImportedKernel = func(_ context.Context, _, destination string) error { + require.NoError(t, os.WriteFile(destination, []byte("partial"), 0644)) + return copyErr + } + + item, err := service.ImportKernel( + context.Background(), + "custom", + sourcePath, + "6.1.0", + "x86_64", + false, + ) + require.Error(t, err) + assert.Nil(t, item) + assert.ErrorIs(t, err, copyErr) + entries, readErr := os.ReadDir(managedDir) + require.NoError(t, readErr) + assert.Empty(t, entries) +} + +func TestAppendKernelStagingCleanupError(t *testing.T) { + cleanupErr := errors.New("cleanup failed") + + t.Run("cleanup is the only failure", func(t *testing.T) { + err := appendKernelStagingCleanupError(nil, cleanupErr) + require.Error(t, err) + assert.ErrorIs(t, err, cleanupErr) + domainErr := errs.AsDomainError(err) + require.NotNil(t, domainErr) + assert.Equal(t, errs.CodeKernelBuildFailed, domainErr.Code) + }) + + t.Run("cleanup does not hide primary failure", func(t *testing.T) { + primaryErr := errors.New("download failed") + err := appendKernelStagingCleanupError(primaryErr, cleanupErr) + require.Error(t, err) + assert.ErrorIs(t, err, primaryErr) + assert.ErrorIs(t, err, cleanupErr) + domainErr := errs.AsDomainError(err) + require.NotNil(t, domainErr) + assert.Equal(t, errs.CodeKernelBuildFailed, domainErr.Code) + }) + + t.Run("cleanup preserves primary domain metadata", func(t *testing.T) { + primaryCause := errors.New("primary cause") + primaryErr := &errs.DomainError{ + Code: errs.CodeVMStateInvalid, + Message: "primary failed", + Op: "custom-operation", + Entity: "vm-id", + Class: errs.ClassConflict, + Err: primaryCause, + Details: map[string]any{"partial": true}, + } + + err := appendKernelStagingCleanupError(primaryErr, cleanupErr) + require.Error(t, err) + assert.ErrorIs(t, err, primaryCause) + assert.ErrorIs(t, err, cleanupErr) + domainErr := errs.AsDomainError(err) + require.NotNil(t, domainErr) + assert.Same(t, primaryErr, domainErr) + assert.Equal(t, primaryErr.Code, domainErr.Code) + assert.Equal(t, primaryErr.Class, domainErr.Class) + assert.Equal(t, primaryErr.Op, domainErr.Op) + assert.Equal(t, primaryErr.Entity, domainErr.Entity) + assert.Equal(t, map[string]any{"partial": true}, domainErr.Details) + }) +} + +func TestFetchFirecrackerKernelUsesReceiverStagingLeaf(t *testing.T) { + t.Setenv("MVM_ASSET_MIRROR", "") + + const ( + ciVersion = "v1.15.0" + kernelVersion = "6.1.0" + arch = "x86_64" + ) + server := httptest.NewServer(http.HandlerFunc(func(writer http.ResponseWriter, request *http.Request) { + switch request.URL.Path { + case "/list": + _, _ = fmt.Fprintf( + writer, + "firecracker-ci/%s/%s/vmlinux-%s", + ciVersion, + arch, + kernelVersion, + ) + case "/firecracker-ci/" + ciVersion + "/" + arch + "/vmlinux-" + kernelVersion: + _, _ = writer.Write([]byte("firecracker-kernel")) + default: + http.NotFound(writer, request) + } + })) + t.Cleanup(server.Close) + + root := t.TempDir() + managedDir := filepath.Join(root, "managed-kernels") + listURL := server.URL + "/list?ci_version={ci_version}&arch={arch}" + service := NewService(&managedOutputRepository{}, managedDir) + result, err := service.FetchFirecrackerKernel( + context.Background(), + &model.KernelSpec{ + ListURLTemplate: &listURL, + Source: server.URL, + OutputName: "../caller-selected-output", + }, + ciVersion, + arch, + nil, + ) + require.NoError(t, err) + require.NotNil(t, result) + + assert.Equal(t, managedDir, filepath.Dir(result.Path)) + assert.NotContains(t, filepath.Base(result.Path), "caller-selected-output") + assert.NotContains(t, filepath.Base(result.Path), kernelVersion) + assert.NotContains(t, filepath.Base(result.Path), arch) + assert.FileExists(t, result.Path) + assert.NoDirExists(t, filepath.Join(managedDir, "caller-selected-output")) +} + +func TestBuildOfficialKernelUsesReceiverStagingLeaf(t *testing.T) { + originalCommands := KernelBuildCommands + originalLibraries := KernelBuildLibraries + KernelBuildCommands = nil + KernelBuildLibraries = nil + t.Cleanup(func() { + KernelBuildCommands = originalCommands + KernelBuildLibraries = originalLibraries + }) + + root := t.TempDir() + managedDir := filepath.Join(root, "managed-kernels") + buildDir := filepath.Join(root, "build", "linux") + spec := &model.KernelSpec{ + Version: "6.1.0", + Source: "https://example.invalid/linux.tar.xz", + OutputName: "../caller-selected-output", + BuildDir: buildDir, + DefaultConfigs: map[string]string{}, + } + service := NewService(&managedOutputRepository{}, managedDir) + configHash := service.computeConfigHash(spec, spec.Version, nil, nil) + cacheKey := fmt.Sprintf("%s-%s", spec.Version, configHash) + cacheDir := filepath.Dir(buildDir) + require.NoError(t, os.MkdirAll(cacheDir, 0755)) + require.NoError(t, os.WriteFile( + filepath.Join(cacheDir, fmt.Sprintf(KernelCacheMarker, cacheKey)), + []byte(cacheKey), + 0644, + )) + require.NoError(t, os.WriteFile( + filepath.Join(cacheDir, fmt.Sprintf(KernelCachePath, cacheKey)), + []byte("official-kernel"), + 0755, + )) + + result, err := service.BuildOfficialKernel( + context.Background(), + spec, + "x86_64", + 1, + false, + true, + false, + nil, + nil, + nil, + nil, + ) + require.NoError(t, err) + require.NotNil(t, result) + + assert.Equal(t, managedDir, filepath.Dir(result.Path)) + assert.NotContains(t, filepath.Base(result.Path), "caller-selected-output") + assert.NotContains(t, filepath.Base(result.Path), spec.Version) + assert.NotContains(t, filepath.Base(result.Path), "x86_64") + assert.FileExists(t, result.Path) + assert.NoDirExists(t, filepath.Join(managedDir, "caller-selected-output")) +} + +type managedOutputRepository struct { + Repository + item *model.KernelItem +} + +func (repo *managedOutputRepository) Upsert(_ context.Context, item *model.KernelItem) error { + repo.item = item + return nil +} + +func (repo *managedOutputRepository) SetDefault(_ context.Context, _ string) error { + return nil +} diff --git a/internal/core/kernel/service.go b/internal/core/kernel/service.go index fc155761..0f7a0e2f 100644 --- a/internal/core/kernel/service.go +++ b/internal/core/kernel/service.go @@ -88,31 +88,91 @@ func applyConfigOption(ctx context.Context, configScriptPath, kernelDir, option, // Service provides stateless kernel operations. type Service struct { - repo Repository - cacheDir string - dl *download.Downloader - resolver *download.HttpDirVersionResolver - specs map[string]*model.KernelSpec + repo Repository + kernelsDir string + copyImportedKernel func(context.Context, string, string) error + dl *download.Downloader + resolver *download.HttpDirVersionResolver + specs map[string]*model.KernelSpec } -func NewService(repo Repository, cacheDir string) *Service { +func NewService(repo Repository, kernelsDir string) *Service { return &Service{ - repo: repo, - cacheDir: cacheDir, + repo: repo, + kernelsDir: kernelsDir, + copyImportedKernel: func(ctx context.Context, sourcePath, destinationPath string) error { + if err := ctx.Err(); err != nil { + return err + } + return infra.CopyPreservingMetadata(sourcePath, destinationPath) + }, dl: download.New(), resolver: download.NewHttpDirVersionResolver(), } } +func (s *Service) ensureKernelsDir(ctx context.Context) error { + if err := ctx.Err(); err != nil { + return errs.WrapMsg(errs.CodeKernelBuildFailed, "create managed kernels directory", err) + } + if err := os.MkdirAll(s.kernelsDir, infra.DirPerm); err != nil { + return errs.WrapMsg(errs.CodeKernelBuildFailed, "create managed kernels directory", err) + } + return nil +} + +func (s *Service) createKernelStagingPath(ctx context.Context) (string, error) { + if err := s.ensureKernelsDir(ctx); err != nil { + return "", err + } + stagingFile, err := os.CreateTemp(s.kernelsDir, ".mvm-kernel-stage-*") + if err != nil { + return "", errs.WrapMsg(errs.CodeKernelBuildFailed, "create managed kernel staging file", err) + } + stagingPath := stagingFile.Name() + if err := stagingFile.Close(); err != nil { + cleanupErr := os.Remove(stagingPath) + return "", errs.WrapMsg( + errs.CodeKernelBuildFailed, + "close managed kernel staging file", + errors.Join(err, cleanupErr), + ) + } + return stagingPath, nil +} + +func appendKernelStagingCleanupError(primary error, cleanupErr error) error { + if cleanupErr == nil || errors.Is(cleanupErr, os.ErrNotExist) { + return primary + } + if primary == nil { + return errs.WrapMsg( + errs.CodeKernelBuildFailed, + "remove managed kernel staging file", + cleanupErr, + ) + } + if domainErr := errs.AsDomainError(primary); domainErr != nil { + domainErr.Message += "; remove managed kernel staging file: " + cleanupErr.Error() + domainErr.Err = errors.Join(domainErr.Err, cleanupErr) + return domainErr + } + return errs.WrapMsg( + errs.CodeKernelBuildFailed, + primary.Error()+"; remove managed kernel staging file: "+cleanupErr.Error(), + errors.Join(primary, cleanupErr), + ) +} + // --- Firecracker Kernel Download --- // FetchFirecrackerKernel downloads a pre-built Firecracker CI vmlinux. func (s *Service) FetchFirecrackerKernel( ctx context.Context, spec *model.KernelSpec, - ciVersion, arch, outputDir string, + ciVersion, arch string, onProgress event.OnDownloadCallback, -) (*model.KernelPullResult, error) { +) (_ *model.KernelPullResult, returnErr error) { if spec.ListURLTemplate == nil || *spec.ListURLTemplate == "" { return nil, errs.New(errs.CodeKernelBuildFailed, fmt.Sprintf( "Missing 'list_url_template' in kernels.yaml for %s", spec.Name)) @@ -149,19 +209,16 @@ func (s *Service) FetchFirecrackerKernel( kernelVersion := versions[0] chosenKey := fmt.Sprintf(KernelS3KeyPattern, ciVersion, arch, kernelVersion) - outputPath := filepath.Join(outputDir, fmt.Sprintf(KernelOutputPattern, spec.OutputName, kernelVersion, arch)) - - if _, err := os.Stat(outputPath); err == nil { - slog.Info("Firecracker CI kernel already cached", "path", outputPath) - return &model.KernelPullResult{ - Path: outputPath, - Version: kernelVersion, - Arch: arch, - KernelType: infra.KernelTypeFirecracker, - Warnings: []string{}, - InfoMessages: []string{fmt.Sprintf("Firecracker kernel ready: %s", outputPath)}, - }, nil + outputPath, err := s.createKernelStagingPath(ctx) + if err != nil { + return nil, err } + stagingOwned := true + defer func() { + if stagingOwned { + returnErr = appendKernelStagingCleanupError(returnErr, os.Remove(outputPath)) + } + }() intentionalNoChecksum := spec.SHA256 == "" && spec.SHA256URL == "" templateVars["kernel_version"] = kernelVersion @@ -203,9 +260,12 @@ func (s *Service) FetchFirecrackerKernel( fmt.Sprintf("Failed to download Firecracker CI kernel: %s", err), ) } - os.Chmod(outputPath, infra.ExecutablePerm) + if err := os.Chmod(outputPath, infra.ExecutablePerm); err != nil { + return nil, errs.WrapMsg(errs.CodeKernelBuildFailed, "set downloaded kernel mode", err) + } slog.Info("Firecracker CI kernel saved", "path", outputPath) + stagingOwned = false return &model.KernelPullResult{ Path: outputPath, Version: kernelVersion, @@ -222,7 +282,7 @@ func (s *Service) FetchFirecrackerKernel( func (s *Service) BuildOfficialKernel( ctx context.Context, spec *model.KernelSpec, - arch, outputDir string, + arch string, jobs int, keepBuildDir bool, useCache bool, @@ -231,11 +291,20 @@ func (s *Service) BuildOfficialKernel( featureEnforces map[string]string, onDownload event.OnDownloadCallback, onProgress event.OnProgressCallback, -) (*model.KernelPullResult, error) { +) (_ *model.KernelPullResult, returnErr error) { if err := checkBuildDependencies(ctx); err != nil { return nil, err } - outputPath := filepath.Join(outputDir, fmt.Sprintf(KernelOutputPattern, spec.OutputName, spec.Version, arch)) + outputPath, err := s.createKernelStagingPath(ctx) + if err != nil { + return nil, err + } + stagingOwned := true + defer func() { + if stagingOwned { + returnErr = appendKernelStagingCleanupError(returnErr, os.Remove(outputPath)) + } + }() buildResult, err := s.buildFromSource(ctx, BuildConfig{ Spec: spec, @@ -269,6 +338,7 @@ func (s *Service) BuildOfficialKernel( } infoMessages = append(infoMessages, fmt.Sprintf("Kernel built: %s", outputPath)) + stagingOwned = false return &model.KernelPullResult{ Path: outputPath, Version: spec.Version, @@ -1211,7 +1281,7 @@ func (s *Service) ImportKernel( version string, arch string, setDefault bool, -) (*model.KernelItem, error) { +) (_ *model.KernelItem, returnErr error) { resolvedPath, err := system.ExpandAndResolve(sourcePath) if err != nil { return nil, errs.New( @@ -1220,33 +1290,41 @@ func (s *Service) ImportKernel( ) } - kernelsDir := filepath.Join(s.cacheDir, "kernels") - if err := os.MkdirAll(kernelsDir, infra.DirPerm); err != nil { + stagingPath, err := s.createKernelStagingPath(ctx) + if err != nil { return nil, err } + stagingOwned := true + defer func() { + if stagingOwned { + returnErr = appendKernelStagingCleanupError(returnErr, os.Remove(stagingPath)) + } + }() - destFilename := fmt.Sprintf(KernelOutputPattern, name, version, arch) - destPath := filepath.Join(kernelsDir, destFilename) - - if err := infra.CopyPreservingMetadata(resolvedPath, destPath); err != nil { - return nil, errs.New( - errs.CodeKernelBuildFailed, - fmt.Sprintf("Failed to copy kernel file to %s: %s", destPath, err), - ) + if err := s.copyImportedKernel(ctx, resolvedPath, stagingPath); err != nil { + return nil, errs.WrapMsg(errs.CodeKernelBuildFailed, "copy imported kernel", err) + } + if err := os.Chmod(stagingPath, infra.ExecutablePerm); err != nil { + return nil, errs.WrapMsg(errs.CodeKernelBuildFailed, "set imported kernel mode", err) + } + if err := ctx.Err(); err != nil { + return nil, errs.WrapMsg(errs.CodeKernelBuildFailed, "hash imported kernel", err) } - os.Chmod(destPath, 0755) - // Compute deterministic ID from the copied file (no timestamp) - kernelID, err := crypto.KernelID(destPath, version, arch) + // The identity is computed from the receiver-owned staged bytes, so a caller + // cannot change the source between hashing and copying to the managed store. + kernelID, err := crypto.KernelID(stagingPath, version, arch) if err != nil { - return nil, fmt.Errorf("compute kernel ID: %w", err) + return nil, errs.WrapMsg(errs.CodeKernelBuildFailed, "compute imported kernel ID", err) } - - // Rename file to content-addressed kernel ID - kernelPath := filepath.Join(kernelsDir, kernelID) - if err := os.Rename(destPath, kernelPath); err != nil { - return nil, fmt.Errorf("rename kernel file: %w", err) + if err := ctx.Err(); err != nil { + return nil, errs.WrapMsg(errs.CodeKernelBuildFailed, "install imported kernel", err) + } + kernelPath := filepath.Join(s.kernelsDir, kernelID) + if err := os.Rename(stagingPath, kernelPath); err != nil { + return nil, errs.WrapMsg(errs.CodeKernelBuildFailed, "install imported kernel", err) } + stagingOwned = false now := time.Now().Format(time.RFC3339) diff --git a/pkg/api/inputs/kernel_pull.go b/pkg/api/inputs/kernel_pull.go index 032ccd7f..e08a12f9 100644 --- a/pkg/api/inputs/kernel_pull.go +++ b/pkg/api/inputs/kernel_pull.go @@ -3,28 +3,24 @@ package inputs import ( "context" "fmt" + "runtime" + "slices" + "sort" + "strings" + "mvmctl/internal/core/config" "mvmctl/internal/core/kernel" - "mvmctl/internal/infra" "mvmctl/internal/lib/firecracker" "mvmctl/internal/lib/model" "mvmctl/internal/lib/system" "mvmctl/internal/lib/version" "mvmctl/pkg/errs" - "os" - "runtime" - "slices" - "sort" - "strings" ) // KernelPullInput specifies kernel pull input. type KernelPullInput struct { KernelType string `json:"type" yaml:"type"` Version string `json:"version,omitempty" yaml:"version,omitempty"` - OutputDir string `json:"output_dir,omitempty"` - OutputName string `json:"name,omitempty" yaml:"name,omitempty"` - OutputPath string `json:"output_path,omitempty"` Jobs int `json:"jobs,omitempty" yaml:"jobs,omitempty"` KeepBuildDir bool `json:"keep_build_dir" yaml:"keep_build_dir"` CleanBuild bool `json:"clean_build" yaml:"clean_build"` @@ -38,7 +34,6 @@ type KernelPullInput struct { type ResolvedKernelPullRequest struct { KernelType string Arch string - OutputDir string Jobs int KeepBuildDir bool CleanBuild bool @@ -125,11 +120,6 @@ func (i *KernelPullInput) Resolve(ctx context.Context, cfg *config.Service) (*Re featuresList = append([]string{"kvm"}, featuresList...) } } - // Resolve output directory — use explicit input or the default kernels dir. - outputDir := infra.GetKernelsDir() - if i.OutputDir != "" { - outputDir = i.OutputDir - } var kernelConfig *string if i.KernelConfig != "" { kernelConfig = &i.KernelConfig @@ -138,7 +128,6 @@ func (i *KernelPullInput) Resolve(ctx context.Context, cfg *config.Service) (*Re KernelType: i.KernelType, Version: version, Arch: arch, - OutputDir: outputDir, Jobs: jobs, KeepBuildDir: i.KeepBuildDir, CleanBuild: i.CleanBuild, @@ -159,17 +148,6 @@ func (i *KernelPullInput) Resolve(ctx context.Context, cfg *config.Service) (*Re errs.WithClass(errs.ClassValidation), ) } - // Validate output directory (must exist or be creatable) — - // if output_dir.exists() and not output_dir.is_dir(): - if result.OutputDir != "" { - if fi, err := os.Stat(result.OutputDir); err == nil && !fi.IsDir() { - return nil, errs.New( - errs.CodeKernelBuildFailed, - fmt.Sprintf("Output path exists but is not a directory: %s", result.OutputDir), - errs.WithClass(errs.ClassValidation), - ) - } - } // Validate build jobs (positive integer) — // if jobs <= 0: if result.Jobs <= 0 { diff --git a/pkg/api/kernel.go b/pkg/api/kernel.go index ae67203a..021d2db4 100644 --- a/pkg/api/kernel.go +++ b/pkg/api/kernel.go @@ -194,7 +194,6 @@ func (op *Operation) KernelPull(ctx context.Context, input inputs.KernelPullInpu spec, resolvedCIVersion, resolved.Arch, - resolved.OutputDir, event.FormatProgress(onProgress), ) if err != nil { @@ -222,7 +221,7 @@ func (op *Operation) KernelPull(ctx context.Context, input inputs.KernelPullInpu maps.Copy(featureEnforces, f.Enforce) } } - fetchResult, err = op.Services.Kernel.BuildOfficialKernel(ctx, spec, resolved.Arch, resolved.OutputDir, + fetchResult, err = op.Services.Kernel.BuildOfficialKernel(ctx, spec, resolved.Arch, resolved.Jobs, resolved.KeepBuildDir, !resolved.CleanBuild, resolved.SkipChecksum, configPath, featureEnforces, event.FormatProgress(onProgress), onProgress) @@ -249,12 +248,14 @@ func (op *Operation) KernelPull(ctx context.Context, input inputs.KernelPullInpu if err := os.Rename(fetchResult.Path, kernelPath); err != nil { return nil, errs.WrapMsg(errs.CodeKernelPullFailed, fmt.Sprintf("Failed to rename kernel file: %v", err), err) } - // Parse filename for base_name (use original filename before rename) - parsed := kernel.ParseFilename(filepath.Base(fetchResult.Path)) + // Preserve the established display metadata without allowing it to select a + // managed filename. The on-disk basename is always the content-derived ID. + displayName := fmt.Sprintf(kernel.KernelOutputPattern, spec.OutputName, fetchResult.Version, resolved.Arch) + parsed := kernel.ParseFilename(displayName) // Create KernelItem kernelItem := &model.KernelItem{ ID: kernelID, - Name: filepath.Base(fetchResult.Path), + Name: displayName, BaseName: parsed.BaseName, Version: fetchResult.Version, Arch: resolved.Arch, diff --git a/pkg/api/operation.go b/pkg/api/operation.go index 0395548c..cea448ac 100644 --- a/pkg/api/operation.go +++ b/pkg/api/operation.go @@ -119,7 +119,7 @@ func NewOperation(ctx context.Context, conn *db.Handle, cacheDir string) *Operat s := Services{ Network: network.NewService(r.Network, defaultFwTracker, r.Policy), Image: image.NewService(r.Image), - Kernel: kernel.NewService(r.Kernel, cacheDir), + Kernel: kernel.NewService(r.Kernel, filepath.Join(cacheDir, "kernels")), Binary: binary.NewService(r.Binary, filepath.Join(cacheDir, "bin"), cacheDir), Key: key.NewService(r.Key, infra.GetKeysDir()), Host: host.NewService(r.Host), From 1c8ec6e22977f33fb67bc4da865e0149d5532db6 Mon Sep 17 00:00:00 2001 From: AlanD20 Date: Sun, 23 Aug 2026 15:45:39 +0200 Subject: [PATCH 0025/1457] test: require exact system-test matrix coverage --- scripts/run-system-tests.py | 102 +++++++++- scripts/tests/test_run_system_tests.py | 252 +++++++++++++++++++++---- 2 files changed, 315 insertions(+), 39 deletions(-) diff --git a/scripts/run-system-tests.py b/scripts/run-system-tests.py index 60ac836f..1c8b8cf8 100755 --- a/scripts/run-system-tests.py +++ b/scripts/run-system-tests.py @@ -11,7 +11,7 @@ python3 scripts/run-system-tests.py --all --host-direct # Run specific domains - python3 scripts/run-system-tests.py cli network vm_nested_virt + python3 scripts/run-system-tests.py cli network exec # Run specific tiers (comma-separated, executed in the given order) python3 scripts/run-system-tests.py --tier 1 @@ -142,6 +142,7 @@ class _UnsafeVolumeBackingPath(RuntimeError): "tests/system/vm/test_jailer.py", "tests/system/vm/test_cgroup.py", ], + "exec": ["tests/system/exec/test_exec.py"], "ssh": ["tests/system/ssh/test_ssh.py"], "console": ["tests/system/console/test_console.py"], "logs": ["tests/system/logs/test_logs.py"], @@ -152,7 +153,6 @@ class _UnsafeVolumeBackingPath(RuntimeError): # Tier 3: directly on host (no runner VM) TIER3_DOMAINS: dict[str, list[str]] = { - "nested_virt": ["tests/system/vm/test_vm_fresh_env.py"], "nested_isolated": ["tests/system/vm/test_vm_nested_isolated.py"], "fresh_env": ["tests/system/vm/test_vm_fresh_env.py"], "snapshot_load": ["tests/system/vm/test_vm_snapshot_load.py"], @@ -1526,6 +1526,103 @@ def _validate_release_qualification_args( ) +def _validate_system_test_registry(parser: argparse.ArgumentParser) -> None: + """Reject an ambiguous or incomplete system-test domain registry.""" + seen: set[str] = set() + duplicate_domains: set[str] = set() + empty_domains: list[str] = [] + registered_files: list[str] = [] + for domains in (TIER1_DOMAINS, TIER2_DOMAINS, TIER3_DOMAINS): + for domain, test_files in domains.items(): + if domain in seen: + duplicate_domains.add(domain) + seen.add(domain) + if not test_files: + empty_domains.append(domain) + registered_files.extend(test_files) + if duplicate_domains: + parser.error( + "duplicate system-test domains: " + f"{', '.join(sorted(duplicate_domains))}" + ) + if empty_domains: + parser.error( + "empty system-test domains: " + f"{', '.join(sorted(empty_domains))}" + ) + duplicate_files = sorted( + { + test_file + for test_file in registered_files + if registered_files.count(test_file) > 1 + } + ) + if duplicate_files: + parser.error( + "duplicate registered system-test files: " + f"{', '.join(duplicate_files)}" + ) + invalid_paths: list[str] = [] + for registered_file in registered_files: + relative_path = Path(registered_file) + parts = relative_path.parts + canonical = ( + not relative_path.is_absolute() + and relative_path.as_posix() == registered_file + and ".." not in parts + ) + system_test = ( + len(parts) >= 3 + and parts[:2] == ("tests", "system") + and relative_path.name.startswith("test_") + and relative_path.suffix == ".py" + ) + if not canonical or not system_test: + invalid_paths.append(registered_file) + if invalid_paths: + parser.error( + "invalid system-test paths: " + f"{', '.join(sorted(invalid_paths))}" + ) + repo_root = _REPO_ROOT.resolve() + missing_files: list[str] = [] + non_regular_files: list[str] = [] + for registered_file in registered_files: + test_file = repo_root / registered_file + try: + file_mode = test_file.lstat().st_mode + except FileNotFoundError: + missing_files.append(registered_file) + continue + except OSError: + non_regular_files.append(registered_file) + continue + if not stat.S_ISREG(file_mode) or test_file.resolve() != test_file: + non_regular_files.append(registered_file) + if missing_files: + parser.error( + "missing system-test files: " + f"{', '.join(sorted(missing_files))}" + ) + if non_regular_files: + parser.error( + "non-regular system-test files: " + f"{', '.join(sorted(non_regular_files))}" + ) + system_test_root = repo_root / "tests" / "system" + discovered_files = { + test_file.relative_to(repo_root).as_posix() + for test_file in system_test_root.rglob("test_*.py") + if test_file.is_file() + } + unregistered_files = sorted(discovered_files - set(registered_files)) + if unregistered_files: + parser.error( + "unregistered system-test files: " + f"{', '.join(unregistered_files)}" + ) + + def _validate_test_selection_args( parser: argparse.ArgumentParser, args: argparse.Namespace, @@ -1833,6 +1930,7 @@ def main() -> None: args = parser.parse_args() _validate_release_qualification_args(parser, args) + _validate_system_test_registry(parser) _validate_test_selection_args(parser, args) if _selection_requests_tier3(args) and not args.host_direct: diff --git a/scripts/tests/test_run_system_tests.py b/scripts/tests/test_run_system_tests.py index 453683aa..d9da4ae0 100644 --- a/scripts/tests/test_run_system_tests.py +++ b/scripts/tests/test_run_system_tests.py @@ -65,6 +65,26 @@ def fail(*_args: object, **_kwargs: object) -> None: assert expected_error in capsys.readouterr().err +def _configure_test_registry( + orchestrator: ModuleType, + monkeypatch: pytest.MonkeyPatch, + repo_root: Path, + tier1: dict[str, list[str]], + tier2: dict[str, list[str]] | None = None, + tier3: dict[str, list[str]] | None = None, +) -> None: + monkeypatch.setattr(orchestrator, "_REPO_ROOT", repo_root) + monkeypatch.setattr(orchestrator, "TIER1_DOMAINS", tier1) + monkeypatch.setattr(orchestrator, "TIER2_DOMAINS", tier2 or {}) + monkeypatch.setattr(orchestrator, "TIER3_DOMAINS", tier3 or {}) + + +def _write_system_test(repo_root: Path, relative_path: str) -> None: + test_file = repo_root / relative_path + test_file.parent.mkdir(parents=True, exist_ok=True) + test_file.write_text("def test_registered():\n pass\n", encoding="utf-8") + + def test_default_outer_controller_is_the_canonical_system_binary( monkeypatch: pytest.MonkeyPatch, ) -> None: @@ -82,6 +102,199 @@ def test_default_outer_controller_is_the_canonical_system_binary( assert module.MVM_BINARY == "/usr/local/bin/mvm" +def test_registry_uses_exec_tier2_and_one_canonical_fresh_env_domain( + orchestrator: ModuleType, +) -> None: + assert orchestrator.TIER2_DOMAINS["exec"] == [ + "tests/system/exec/test_exec.py" + ] + assert orchestrator.TIER3_DOMAINS["fresh_env"] == [ + "tests/system/vm/test_vm_fresh_env.py" + ] + assert "nested_virt" not in orchestrator.TIER3_DOMAINS + + +def test_duplicate_registry_domain_is_rejected_before_any_probe_or_mutation( + orchestrator: ModuleType, + monkeypatch: pytest.MonkeyPatch, + capsys: pytest.CaptureFixture[str], +) -> None: + orchestrator.TIER2_DOMAINS["cli"] = ["tests/system/cli/test_cli.py"] + + _assert_selection_rejected_before_side_effects( + orchestrator, + monkeypatch, + capsys, + [], + "duplicate system-test domains: cli", + ) + + +def test_empty_registry_domains_are_rejected_in_deterministic_order( + orchestrator: ModuleType, + monkeypatch: pytest.MonkeyPatch, + capsys: pytest.CaptureFixture[str], +) -> None: + orchestrator.TIER1_DOMAINS["cli"].clear() + orchestrator.TIER2_DOMAINS["volume"].clear() + + _assert_selection_rejected_before_side_effects( + orchestrator, + monkeypatch, + capsys, + [], + "empty system-test domains: cli, volume", + ) + + +def test_duplicate_registered_file_is_rejected_before_any_probe_or_mutation( + orchestrator: ModuleType, + monkeypatch: pytest.MonkeyPatch, + capsys: pytest.CaptureFixture[str], + tmp_path: Path, +) -> None: + registered = "tests/system/cli/test_cli.py" + _write_system_test(tmp_path, registered) + _configure_test_registry( + orchestrator, + monkeypatch, + tmp_path, + {"cli": [registered]}, + {"cli_alias": [registered]}, + ) + + _assert_selection_rejected_before_side_effects( + orchestrator, + monkeypatch, + capsys, + [], + f"duplicate registered system-test files: {registered}", + ) + + +@pytest.mark.parametrize( + "registered", + [ + "tests/system/cli/helper.py", + "outside/test_escape.py", + "/tmp/test_absolute.py", + "tests/system/cli/../cli/test_cli.py", + "tests/system/../../outside/test_escape.py", + ], + ids=["non-test", "out-of-tree", "absolute", "non-canonical", "escaping"], +) +def test_invalid_registered_paths_are_rejected_before_any_probe_or_mutation( + orchestrator: ModuleType, + monkeypatch: pytest.MonkeyPatch, + capsys: pytest.CaptureFixture[str], + tmp_path: Path, + registered: str, +) -> None: + if not Path(registered).is_absolute(): + _write_system_test(tmp_path, registered) + _configure_test_registry( + orchestrator, + monkeypatch, + tmp_path, + {"invalid": [registered]}, + ) + + _assert_selection_rejected_before_side_effects( + orchestrator, + monkeypatch, + capsys, + [], + f"invalid system-test paths: {registered}", + ) + + +def test_missing_registered_files_are_rejected_in_deterministic_order( + orchestrator: ModuleType, + monkeypatch: pytest.MonkeyPatch, + capsys: pytest.CaptureFixture[str], + tmp_path: Path, +) -> None: + missing = [ + "tests/system/zeta/test_zeta.py", + "tests/system/alpha/test_alpha.py", + ] + _configure_test_registry( + orchestrator, + monkeypatch, + tmp_path, + {"missing": missing}, + ) + + _assert_selection_rejected_before_side_effects( + orchestrator, + monkeypatch, + capsys, + [], + "missing system-test files: " + "tests/system/alpha/test_alpha.py, tests/system/zeta/test_zeta.py", + ) + + +@pytest.mark.parametrize("kind", ["directory", "symlink"]) +def test_non_regular_registered_files_are_rejected_before_any_probe( + orchestrator: ModuleType, + monkeypatch: pytest.MonkeyPatch, + capsys: pytest.CaptureFixture[str], + tmp_path: Path, + kind: str, +) -> None: + registered = "tests/system/cli/test_cli.py" + registered_path = tmp_path / registered + registered_path.parent.mkdir(parents=True) + if kind == "directory": + registered_path.mkdir() + else: + target = tmp_path / "real_test.py" + target.write_text("def test_real():\n pass\n", encoding="utf-8") + registered_path.symlink_to(target) + _configure_test_registry( + orchestrator, + monkeypatch, + tmp_path, + {"invalid": [registered]}, + ) + + _assert_selection_rejected_before_side_effects( + orchestrator, + monkeypatch, + capsys, + [], + f"non-regular system-test files: {registered}", + ) + + +def test_unregistered_system_tests_are_rejected_in_deterministic_order( + orchestrator: ModuleType, + monkeypatch: pytest.MonkeyPatch, + capsys: pytest.CaptureFixture[str], + tmp_path: Path, +) -> None: + registered = "tests/system/cli/test_cli.py" + _write_system_test(tmp_path, registered) + _write_system_test(tmp_path, "tests/system/zeta/test_zeta.py") + _write_system_test(tmp_path, "tests/system/alpha/test_alpha.py") + _configure_test_registry( + orchestrator, + monkeypatch, + tmp_path, + {"cli": [registered]}, + ) + + _assert_selection_rejected_before_side_effects( + orchestrator, + monkeypatch, + capsys, + [], + "unregistered system-test files: " + "tests/system/alpha/test_alpha.py, tests/system/zeta/test_zeta.py", + ) + + def test_tier3_selection_detection_covers_all_request_forms( orchestrator: ModuleType, ) -> None: @@ -199,48 +412,13 @@ def test_unique_tier_selection_preserves_requested_order( assert args.tier == [2, 1, 3] -@pytest.mark.parametrize( - ("selection", "empty_domains", "expected_empty"), - [ - (["cli"], ("cli",), "cli"), - (["cli", "config"], ("cli",), "cli"), - (["--tier", "2"], ("volume",), "volume"), - (["--tier", "2"], ("volume", "ssh"), "ssh, volume"), - (["--all"], ("env",), "env"), - ], - ids=["domain", "mixed-domains", "tier", "tier-multiple", "all"], -) -def test_empty_domain_registrations_are_rejected_before_any_probe_or_mutation( - orchestrator: ModuleType, - monkeypatch: pytest.MonkeyPatch, - capsys: pytest.CaptureFixture[str], - selection: list[str], - empty_domains: tuple[str, ...], - expected_empty: str, -) -> None: - domains = { - **orchestrator.TIER1_DOMAINS, - **orchestrator.TIER2_DOMAINS, - **orchestrator.TIER3_DOMAINS, - } - for domain in empty_domains: - domains[domain].clear() - - _assert_selection_rejected_before_side_effects( - orchestrator, - monkeypatch, - capsys, - selection, - f"selected domains have no test files: {expected_empty}", - ) - - def test_selection_with_no_registered_domains_is_rejected_before_any_probe( orchestrator: ModuleType, monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str], + tmp_path: Path, ) -> None: - monkeypatch.setattr(orchestrator, "TIER2_DOMAINS", {}) + _configure_test_registry(orchestrator, monkeypatch, tmp_path, {}) _assert_selection_rejected_before_side_effects( orchestrator, From bccdb593c1e26e8e44a9ace9bf3d906da47bfd12 Mon Sep 17 00:00:00 2001 From: AlanD20 Date: Sun, 23 Aug 2026 15:46:49 +0200 Subject: [PATCH 0026/1457] docs: record kernel and matrix hardening --- CHANGELOG.md | 8 ++++++++ CONTEXT.md | 5 +++++ docs/development/HOW_TO_RUN_SYSTEM_TESTS.md | 4 +++- docs/system-test-architecture.md | 17 ++++++++++------- tasks/todo.md | 4 +++- 5 files changed, 29 insertions(+), 9 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 1357cbb6..cee7efb1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -76,6 +76,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 build, and byte/version identity with exact root-owned `/usr/local/bin/mvm` before host resource preparation. - The runner rejects unknown, repeated, mixed, zero-domain, or empty-file test selections before binary probes, builds, or resource mutation instead of warning and returning successful but incomplete evidence. +- The runner now validates its complete registry before any work: domain names and test files are unique, registrations + are non-empty canonical regular files under `tests/system/`, and every discovered system-test file is registered + exactly once. The missing Tier 2 `exec` registration is included and the duplicate Tier 3 `fresh_env` alias is gone. #### Deterministic initialization - Added `mvm init --binary-version ` to request an exact Firecracker/Jailer pair when no local pair exists. @@ -109,6 +112,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 #### Managed storage identity - Newly created volume backing files are named from the immutable 64-character volume ID rather than the user-visible display name. Raw and qcow2 formats remain explicit suffixes; database paths are derived metadata, not root authority. +- Kernel download, build, and import now stage files under the kernel service's configured managed directory and install + final artifacts as `kernels/`. Import identity is computed from the receiver-owned staged bytes, so a + caller-controlled display name or a source-file replacement cannot select or escape the managed destination. #### Network isolation - Traffic routed between different managed networks is now denied unless an explicit typed service-access policy permits @@ -135,6 +141,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 must create explicit `mvm policy` entries for required TCP or UDP services. - Scripts that expect `mvm self-update apply` to replace `/usr/local/bin/mvm` must switch to the administrator bootstrap flow. Self-update remains available for user-owned candidate artifacts. +- API consumers can no longer set `output_dir`, `name`, or `output_path` on `KernelPullInput`; managed kernel placement + is receiver-owned. The public CLI did not expose equivalent flags. ### Fixed diff --git a/CONTEXT.md b/CONTEXT.md index e75278af..8c27221d 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -540,6 +540,11 @@ release build, and exact root-owned `/usr/local/bin/mvm` content/version identit and binary gate does not yet prove that every Tier 3 resource is newly owned or that every teardown succeeded: do not invoke Tier 3 or `--all` on a host with pre-existing mvmctl state until the remaining Task 17 cleanup gates land. +Before any probe, build, or resource mutation, the runner also validates the system-test registry itself. Domain names +and test files must be unique, registrations must be non-empty canonical regular files under `tests/system/`, and the +registered set must equal the discovered `tests/system/**/test_*.py` set. A missing or duplicate domain therefore cannot +silently produce incomplete release evidence. + ```bash # Run inside the runner VM (disposable Firecracker VM with nested KVM) pytest tests/system/ diff --git a/docs/development/HOW_TO_RUN_SYSTEM_TESTS.md b/docs/development/HOW_TO_RUN_SYSTEM_TESTS.md index 9c8bac4c..b0a63778 100644 --- a/docs/development/HOW_TO_RUN_SYSTEM_TESTS.md +++ b/docs/development/HOW_TO_RUN_SYSTEM_TESTS.md @@ -363,7 +363,9 @@ python3 scripts/run-system-tests.py --host-direct --all --workers 2 | `--skip-volume-check` | `false` | Skip shared volume existence check (assume it exists) | Choose exactly one test selector: positional domains, `--tier`, or `--all`. Unknown names, repeated domains/tiers, and -selected domains with no registered test files fail before binary probes, builds, or resource mutation. +selected domains with no registered test files fail before binary probes, builds, or resource mutation. The runner also +requires every canonical regular `tests/system/**/test_*.py` file to be registered exactly once, so an omitted file or +duplicate alias cannot produce release evidence. --- diff --git a/docs/system-test-architecture.md b/docs/system-test-architecture.md index 2142c287..4decc022 100644 --- a/docs/system-test-architecture.md +++ b/docs/system-test-architecture.md @@ -119,7 +119,7 @@ All T1/T2 runner VMs are Firecracker microVMs created with **Firecracker v1.16** |------|---------------|---------------|------------|-----------------|---------------| | **1** | Yes — `mvm exec -- pytest ...` | Yes (mounted at `/mnt`) | Not needed | `official:7.0.11` (nftables + tuntap + kvm + btrfs) | Host-level CLI: help, config, init, cache, keys, invariants, bin, images, kernel, network, host, run | | **2** | Yes — `mvm exec -- pytest ...` | Yes (mounted at `/mnt`) | Yes | `official:7.0.11` (nftables + tuntap + kvm + btrfs) | VM lifecycle: volume, vm_lifecycle, vm_data_persistence, Jailer/cgroups, routed policies, ssh, console, exec, logs, full_journeys, nftables | -| **3** | No — runs directly on host | Host mirror | Yes (host KVM) | N/A (host-direct) | vm_fresh_env, vm_nested_isolated, vm_snapshot_load, vm_snapshot_rootfs, volume_hotplug, cp, kernel_build, env | +| **3** | No — runs directly on host | Host mirror | Yes (host KVM) | N/A (host-direct) | fresh_env, nested_isolated, snapshot_load, snapshot_rootfs, volume_hotplug, cp, kernel_build, env | --- @@ -221,12 +221,16 @@ mvm exec --user runner --timeout 600 -- \ ### Domains - `volume/test_volume.py` - `vm/test_vm_lifecycle.py` +- `vm/test_vm_data_persistence.py` +- `vm/test_jailer.py` +- `vm/test_cgroup.py` - `ssh/test_ssh.py` - `console/test_console.py` - `exec/test_exec.py` - `logs/test_logs.py` - `full_journeys/test_full_journeys.py` (Tier 2) -- `vm/test_vm_data_persistence.py` +- `network/test_nftables.py` +- `network/test_policies.py` ### Runner VM Spec @@ -492,11 +496,10 @@ tests/system/ | volume_hotplug | 3 | `test_volume_hotplug.py` | PCI hotplug (needs host KVM — FC dev-preview, broken nested) | | cp | 3 | `test_cp.py` | File copy to/from VMs (vsock agent path rejection nested) | | vm_data_persistence | 2 | `test_vm_data_persistence.py` | Data persistence after snapshot/restore — uses runner VM + nested KVM | -| vm_nested_isolated | 3 | `test_vm_nested_isolated.py` | Host-only, triple-nested | -| vm_fresh_env | 3 | `test_vm_fresh_env.py` | Host-only. Includes negative-case test from former `test_vm_nested_virt.py`. | -| vm_snapshot_load | 3 | `test_vm_snapshot_load.py` | Host-only | -| vm_snapshot_rootfs_independence | 3 | `test_vm_snapshot_rootfs_independence.py` | Host-only — verifies restored VM rootfs is independent of source and other restored VMs | -| nested_virt | 3 | `test_vm_fresh_env.py` | Host-only nested virt negative-case tests (same file as vm_fresh_env) | +| nested_isolated | 3 | `test_vm_nested_isolated.py` | Host-only, triple-nested | +| fresh_env | 3 | `test_vm_fresh_env.py` | Host-only. Includes the negative case from former `test_vm_nested_virt.py`. | +| snapshot_load | 3 | `test_vm_snapshot_load.py` | Host-only | +| snapshot_rootfs | 3 | `test_vm_snapshot_rootfs_independence.py` | Host-only — verifies restored VM rootfs is independent of source and other restored VMs | | kernel_build | 3 | `test_kernel.py` | Kernel build tests — needs full KVM host access (Tier 3 only) | --- diff --git a/tasks/todo.md b/tasks/todo.md index eb5a51af..26104865 100644 --- a/tasks/todo.md +++ b/tasks/todo.md @@ -126,6 +126,8 @@ under `internal/service/jailer/`. No handler, transport, CLI, API, core-domain, to re-pin the same identity. - [ ] Land canonical producer names before descriptor pinning: - [x] ID-based `volumes/.` storage. + - [x] Content-addressed `kernels/` storage with receiver-owned staging; kernel pull inputs cannot select an + output directory, name, or path. - [ ] Fixed `rootfs.img`, cloud-init, Firecracker, console, and vsock VM leaves. - [ ] Fixed snapshot leaves and a managed ID-scoped image-provisioning staging subtree. - [ ] Define operation-specific non-empty/maximum-size and access-mode policies before coding; a caller- or DB-provided @@ -337,7 +339,7 @@ under `internal/service/jailer/`. No handler, transport, CLI, API, core-domain, - [x] Reject unknown, duplicate, mixed, zero-domain, and empty-file domain selections before probes or mutation. - [ ] Parse pytest outcomes and require non-empty collection with zero skips, XFAIL, or XPASS; omitted domains fail instead of producing release evidence. -- [ ] Ensure the full matrix selects every required domain exactly once, including `exec`, and does not run +- [x] Ensure the full matrix selects every required domain exactly once, including `exec`, and does not run `test_vm_fresh_env.py` twice under aliases. - [ ] Make mirror validation reject pull errors and HTTP fallback; only a verified local-mirror read is a cache hit. - [ ] Run the complete T1/T2/T3 matrix plus clean-host installation and reboot/reconciliation qualification. From 8ff219bb63ba3616aa6aa6c5c1b229befa3273b7 Mon Sep 17 00:00:00 2001 From: AlanD20 Date: Sun, 23 Aug 2026 15:50:43 +0200 Subject: [PATCH 0027/1457] docs: prepare trusted release authority --- ...owned-single-binary-privileged-dispatch.md | 34 +++++++---- tasks/plan.md | 57 ++++++++++++++----- tasks/todo.md | 16 ++++-- 3 files changed, 79 insertions(+), 28 deletions(-) diff --git a/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md b/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md index a81d9a25..1d62e1ee 100644 --- a/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md +++ b/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md @@ -84,6 +84,11 @@ The sudoers entry may contain a wildcard after the reserved marker because opera can reach only the early dispatcher. It cannot select the public CLI. The dispatcher remains the authoritative allowlist, and every handler performs exact argument-count and type checks before side effects. +Synchronous actions return one bounded, versioned response envelope on their control channel. Success payloads are +action-specific. Failures preserve the original `DomainError` code, class, entity, and bounded partial-state details so +the normal-user client can make safe retry and recovery decisions. Logs and subprocess exit status are diagnostic only; +they never replace the structured result or turn a malformed/missing response into a generic successful operation. + ### Authorization roles The root-owned executable establishes code integrity. The `mvm` Unix group remains the authorization role for @@ -156,14 +161,17 @@ and system-damage invariants. These are receiver trust-boundary checks, not dupl ### Release trust -The caller may supply archive bytes for caching and progress, but it cannot supply their authority. Privileged release -code constructs the permitted Firecracker release URL from a strictly validated version and architecture and obtains the -expected checksum independently from the fixed official origin or root-owned signed metadata. +The caller supplies only a validated release version and architecture. Privileged release code constructs the permitted +Firecracker release and checksum URLs and downloads both independently from the fixed official origin with a dedicated, +bounded HTTPS-only client. It disables proxies and does not consult caller configuration, the general download cache, or +`MVM_ASSET_MIRROR`. Caller-provided paths, URLs, checksums, and archive bytes are never release authority. -Extraction occurs in a root-owned temporary directory with bounded input, exact member names, and rejection of path -traversal, symlinks, hardlinks, devices, duplicate members, unexpected members, and size/count overflow. The exact -Firecracker/Jailer pair and a root-owned release manifest are fsynced and atomically renamed into the trusted store. A -referenced release cannot be force-replaced. +Extraction occurs in a root-owned temporary directory with bounded compressed/decompressed input, a reviewed allowlist +for the complete upstream archive layout, and rejection of path traversal, symlinks, hardlinks, devices, sparse files, +duplicates, unexpected members, and size/count overflow. Only the exact Firecracker/Jailer pair is extracted. Their ELF +class and machine must match the selected architecture without executing downloaded code. The pair and strict root-owned +release manifest are fsynced and atomically renamed into the trusted store. A referenced release cannot be +force-replaced. If upstream provides a stable verifiable signature or provenance mechanism, mvmctl should adopt it. Until then, the fixed HTTPS origin and independently fetched checksum are an explicit supply-chain limitation, not a signature claim. @@ -191,9 +199,10 @@ the privileged effect and rollback are complete. Privileged-visible basenames are canonical and derived by the receiver. Volume storage is keyed by volume ID rather than user-visible name; the VM rootfs uses fixed `rootfs.img`; kernel and image names use their canonical IDs plus only enumerated representations; cloud-init, Firecracker runtime, and snapshot leaves use compile-time fixed names. The user -database may describe a resource, but it cannot choose the basename root will open. The intentional -snapshot phantom-rootfs link is managed only by a typed descriptor-relative snapshot operation and is never followed as -an input resource. +database may describe a resource, but it cannot choose the basename root will open. Snapshot create exposes the snapshot +read-write and writes its fixed `rootfs.img`, `memory`, and `vmstate` leaves. Restore exposes the snapshot read-only and +overlays the new VM's pinned `/rootfs` at `/snapshot/rootfs.img` inside that VM's private mount namespace; no persistent +phantom-rootfs symlink is created. The v0.3 canonical privileged-visible layout is: @@ -221,6 +230,11 @@ exact release. Mutating operations require a pidfd, use `pidfd_send_signal`, and the required pidfd support fails capability admission instead of falling back to a racy raw PID signal. PID reuse fails closed. The receiver enforces its compiled group-drop policy and never permits a retained root supplementary group. +Release launch preparation acquires the `(version, architecture)` release-slot lock before resolving the root manifest. +The prepared value retains the verified manifest, pinned release directory, and pinned Firecracker/Jailer descriptors. +It supplies the full release identity to instance registration itself, then transfers the release lock into the launch +lease only after the owner-bound record is durable. The unprivileged caller never supplies executable hashes. + The launch path starts a blocked child in a new private mount namespace. Before the child can perform any externally mutable effect, the parent pins its pidfd and namespace descriptor, records the complete identity, and retains the launch lease. The parent then enters that namespace, makes mount propagation private, mounts descriptor-pinned resources, and diff --git a/tasks/plan.md b/tasks/plan.md index 5801d57e..d150c8f3 100644 --- a/tasks/plan.md +++ b/tasks/plan.md @@ -74,12 +74,22 @@ The first implementation slice is private to `internal/service/jailer`. It uses operations: ```go -func (a *instanceAuthority) RegisterLaunch( +func (a *instanceAuthority) lockReleaseSlot( + ctx context.Context, + slot releaseSlot, +) (*releaseSlotLease, error) + +func (l *releaseSlotLease) registerLaunch( ctx context.Context, caller instanceCaller, registration launchRegistration, ) (*launchLease, error) +func (l *releaseSlotLease) requireUnreferenced( + ctx context.Context, + release releaseIdentity, +) error + func (a *instanceAuthority) LockRegistered( ctx context.Context, caller instanceCaller, @@ -94,15 +104,10 @@ func (a *instanceAuthority) BeginCleanup( func (l *cleanupLease) Complete(ctx context.Context) error -func (a *instanceAuthority) LockUnreferencedRelease( - ctx context.Context, - release releaseIdentity, -) (*releaseLease, error) - func (l *launchLease) Release(ctx context.Context) error func (l *registeredLease) Release(ctx context.Context) error func (l *cleanupLease) Release(ctx context.Context) error -func (l *releaseLease) Release(ctx context.Context) error +func (l *releaseSlotLease) Release(ctx context.Context) error ``` Each lease retains its pinned directory and lock descriptors and provides a checked context-first release operation. @@ -111,8 +116,8 @@ There is no public `Get`, `Put`, `Delete`, `Run`, raw path, arbitrary state stri The durable lifecycle is intentionally small: ```text -absent ── RegisterLaunch ──> registered -cleaned ── RegisterLaunch ──> registered +absent ── registerLaunch ──> registered +cleaned ── registerLaunch ──> registered registered ─ BeginCleanup ──> cleaning cleaning ─── BeginCleanup ──> cleaning cleaning ───── Complete ─────> cleaned @@ -133,12 +138,16 @@ Operations may acquire a suffix or subset but never an earlier class while holdi are established by descriptor-enumerating all numeric UID record directories while holding the index lock. A foreign, duplicate, corrupt, unreadable, or inconsistent claim fails closed. Lock files are never unlinked. -Release locks are keyed by the canonical store slot `(version, architecture)`, not by caller-observed binary hashes. -Reference matching remains against the full release identity. Competing identities for one store slot must therefore -serialize before install, replace, remove, launch, or reference inspection. +Release locks are keyed by the canonical store slot `(version, architecture)`, not by caller-observed binary hashes, and +are acquired before root manifest resolution. Reference matching remains against the full release identity. Competing +identities for one store slot must therefore serialize before install, replace, remove, launch, or reference inspection. -`RegisterLaunch` establishes the durable record before the first network or launch effect and returns with the VM lock -held. A later typed network step may therefore acquire only the later network lock while retaining that launch lease. +Task 6 acquires the release-slot lease before it reads a manifest or computes a launch identity. Its prepared launch +value retains that lease together with the verified manifest and pinned Firecracker/Jailer descriptors. Only that value +may call `releaseSlotLease.registerLaunch`; it supplies the exact release identity from the root-owned manifest rather +than accepting hashes from the normal user process. Registration then acquires the index and VM locks, establishes the +durable record before the first network or launch effect, and transfers the held release lock into the returned launch +lease. A mismatch or registration failure performs no external effect and leaves a checked, releasable slot lease. Task 4 owns release, index, and VM locking; it does not import network code or predeclare a network implementation. The mount-namespace launch seam uses a blocked child: the child creates its private namespace and waits on an inherited @@ -154,6 +163,26 @@ Atomic record replacement uses a random exclusive temporary file, bounded writes Post-rename failure returns details that distinguish `record_replaced` and `durability_uncertain`; the caller starts no VM effects after such an error. +### Trusted release authority + +Task 6 adds a private `releaseAuthority` in `internal/service/jailer`. Its public privileged request contains only a +validated release version, architecture, and explicit replacement intent. Root constructs the fixed official checksum +and archive URLs, downloads both through a dedicated bounded HTTPS-only client with proxies and the user asset mirror +disabled, verifies the archive before parsing it, validates the reviewed complete upstream member allowlist, and +extracts only Firecracker and Jailer. Caller paths, URLs, checksums, and archive bytes are prohibited. + +The strict root-owned manifest stores schema version, release slot, archive hash, and each executable's hash and size. +The store is exactly `/var/lib/mvmctl/binaries///{firecracker,jailer,release.json}`. Binaries are +validated as ELF for the selected architecture without execution. Descriptor-relative atomic install, exchange, and +removal use only fixed leaves and preserve an old complete release or expose a new complete release, never a partial +pair. Post-commit errors report `release_installed`, `release_replaced`, `release_removed`, `durability_uncertain`, and +`retired_release_retained` details without replacing the primary error identity. + +Launch uses a private prepared value that owns the release-slot lease, verified manifest, pinned release directory, and +pinned executable descriptors. It alone supplies release hashes to instance registration. The privileged transport must +return a strict versioned response envelope before release install/remove is wired, because generic subprocess errors +cannot preserve the partial-state details required for safe retries. + ### Network topology and policy ADR-0017 is the authoritative design: diff --git a/tasks/todo.md b/tasks/todo.md index 26104865..508eaadd 100644 --- a/tasks/todo.md +++ b/tasks/todo.md @@ -63,21 +63,25 @@ The final marker-only sudoers assertion is deferred until Task 14; current sudoe - [x] Unsupported versions fail closed. - [x] Strict bounded JSON rejects unknown, duplicate, case-fold duplicate, trailing, oversized, and malformed input. - [x] Caller identity, active `mvm` group, environment, and system executable are verified before dispatch. +- [ ] Add a strict versioned response envelope that preserves `DomainError` code, class, entity, and partial-state + details across the privileged subprocess boundary. - [ ] Keep the action switch closed until each capability has its own typed handler and abuse tests. ### Task 4: Add private root-owned VM instance authority **Owner:** engineer -**Status:** Complete +**Status:** Foundation implemented; prepared release-slot correction in progress **Dependencies:** Tasks 1 and 3 **Interface:** -- [x] Implement private `RegisterLaunch(ctx, caller, registration) (*launchLease, error)`. +- [ ] Implement private `lockReleaseSlot(ctx, slot) (*releaseSlotLease, error)` before release identity resolution. +- [ ] Implement private `releaseSlotLease.registerLaunch(ctx, caller, registration) (*launchLease, error)` with + ownership transfer only after durable registration. - [x] Implement private `LockRegistered(ctx, caller, vmID) (*registeredLease, error)`. - [x] Implement private `BeginCleanup(ctx, caller, vmID) (*cleanupLease, error)`. - [x] Implement context-first `cleanupLease.Complete(ctx)` and checked lease release. -- [x] Implement private `LockUnreferencedRelease(ctx, release) (*releaseLease, error)`. +- [ ] Implement private `releaseSlotLease.requireUnreferenced(ctx, release)` for exact-identity removal/replacement. - [x] Expose no generic storage, callbacks, raw paths, actions, or state strings. **Authority and lifecycle:** @@ -111,6 +115,7 @@ The final marker-only sudoers assertion is deferred until Task 14; current sudoe - [x] Architect reviews actual interface, state schema, lock order, imports, and error details before commit. - [x] Key the release lock by canonical `(version, architecture)` store slot rather than binary hashes; prove two different identities for one slot serialize. +- [ ] Prove the slot lock is held before manifest/hash resolution and a mismatched identity writes no record. **Expected files:** fixed root constants; private instance types, codec, descriptor store, lock, authority, and focused tests under `internal/service/jailer/`. No handler, transport, CLI, API, core-domain, or network changes. @@ -151,8 +156,11 @@ under `internal/service/jailer/`. No handler, transport, CLI, API, core-domain, - [ ] Store exact releases under `/var/lib/mvmctl/binaries//` with fixed Firecracker, Jailer, and strict root-owned manifest leaves. - [ ] Privileged code constructs the fixed official release/checksum locations from validated version and architecture. -- [ ] Caller-provided archive bytes are not their own authority. +- [ ] Root downloads the checksum and archive through a dedicated bounded HTTPS-only source with proxies and the general + user asset mirror disabled; caller-provided paths, checksums, URLs, and archive bytes are not authority. - [ ] Bounded extraction rejects traversal, links, devices, duplicates, unexpected members, and size/count overflow. +- [ ] Validate the complete reviewed upstream member allowlist while extracting only Firecracker and Jailer; validate + their ELF class/machine without executing downloaded code. - [ ] Exact Firecracker/Jailer bytes and a root manifest install atomically and durably. - [ ] Referenced release removal/replacement holds the release lease and fails closed on unreadable records. - [ ] L1 failure injection proves an old complete pair or no pair, never a partial trusted release. From 3b1c56d68c28cfc74b585b41347d8dc7c53757dc Mon Sep 17 00:00:00 2001 From: AlanD20 Date: Sun, 23 Aug 2026 16:04:21 +0200 Subject: [PATCH 0028/1457] refactor: prepare release slots before identity resolution --- internal/service/jailer/instance_authority.go | 185 +++++---- .../service/jailer/instance_authority_test.go | 370 ++++++++++++++++-- internal/service/jailer/instance_lock.go | 12 +- internal/service/jailer/instance_lock_test.go | 18 +- internal/service/jailer/instance_types.go | 26 +- 5 files changed, 480 insertions(+), 131 deletions(-) diff --git a/internal/service/jailer/instance_authority.go b/internal/service/jailer/instance_authority.go index e513a975..76c4937d 100644 --- a/internal/service/jailer/instance_authority.go +++ b/internal/service/jailer/instance_authority.go @@ -34,9 +34,10 @@ type cleanupLease struct { record instanceRecord } -type releaseLease struct { +type releaseSlotLease struct { roots *instanceAuthorityRoots releaseLock *authorityLock + slot releaseSlot } type lockedInstance struct { @@ -57,22 +58,60 @@ func newInstanceAuthorityWithPolicy( return &instanceAuthority{deps: deps, policy: policy} } -func (authority *instanceAuthority) RegisterLaunch( +func (authority *instanceAuthority) lockReleaseSlot( + ctx context.Context, + slot releaseSlot, +) (_ *releaseSlotLease, returnErr error) { + if err := validateReleaseSlotValue(slot); err != nil { + return nil, instanceValidationError(err.Error()) + } + roots, err := openInstanceAuthorityRoots(ctx, authority.deps, authority.policy) + if err != nil { + return nil, err + } + var releaseLock *authorityLock + defer func() { + if returnErr == nil { + return + } + returnErr = releaseInstanceOperationResources( + ctx, + returnErr, + nil, + nil, + releaseLock, + nil, + roots, + ) + }() + releaseLock, err = roots.acquireReleaseSlotLock(ctx, slot) + if err != nil { + return nil, err + } + lease := &releaseSlotLease{roots: roots, releaseLock: releaseLock, slot: slot} + roots = nil + releaseLock = nil + return lease, nil +} + +func (lease *releaseSlotLease) registerLaunch( ctx context.Context, caller instanceCaller, registration launchRegistration, ) (_ *launchLease, returnErr error) { + if lease == nil || lease.roots == nil || lease.releaseLock == nil || lease.releaseLock.fd < 0 { + return nil, instanceStateError(registration.vmID, "release slot lease is not active") + } if err := validateInstanceCaller(caller); err != nil { return nil, err } if err := validateLaunchRegistration(registration); err != nil { return nil, err } - roots, err := openInstanceAuthorityRoots(ctx, authority.deps, authority.policy) - if err != nil { - return nil, err + if releaseSlotForIdentity(registration.release) != lease.slot { + return nil, instanceValidationError("launch release identity does not match held release slot") } - var releaseLock, indexLock, vmLock *authorityLock + var indexLock, vmLock *authorityLock var directories *instanceUIDDirectories defer func() { if returnErr == nil { @@ -83,26 +122,22 @@ func (authority *instanceAuthority) RegisterLaunch( returnErr, vmLock, indexLock, - releaseLock, + nil, directories, - roots, + nil, ) }() - releaseLock, err = roots.acquireReleaseLock(ctx, registration.release) - if err != nil { - return nil, err - } - indexLock, err = roots.acquireIndexLock(ctx) + indexLock, err := lease.roots.acquireIndexLock(ctx) if err != nil { return nil, err } - location, err := roots.findInstance(ctx, registration.vmID) + location, err := lease.roots.findInstance(ctx, registration.vmID) if err != nil { return nil, err } if location != nil && location.uid != caller.uid { - if closeErr := authority.deps.close(context.WithoutCancel(ctx), location.stateFD); closeErr != nil { + if closeErr := lease.roots.deps.close(context.WithoutCancel(ctx), location.stateFD); closeErr != nil { return nil, appendInstanceOperationError( unauthorizedInstanceError(registration.vmID), "close foreign instance authority directory", @@ -113,12 +148,12 @@ func (authority *instanceAuthority) RegisterLaunch( } var existing *instanceRecord if location == nil { - directories, err = roots.openUIDDirectories(ctx, caller.uid) + directories, err = lease.roots.openUIDDirectories(ctx, caller.uid) if err != nil { return nil, err } } else { - directories, err = roots.openRuntimeForLocation(ctx, location) + directories, err = lease.roots.openRuntimeForLocation(ctx, location) if err != nil { return nil, err } @@ -149,18 +184,64 @@ func (authority *instanceAuthority) RegisterLaunch( return nil, annotateInstanceRecordReplacement(err, false) } indexLock = nil - lease := &launchLease{ - roots: roots, + launch := &launchLease{ + roots: lease.roots, directories: directories, - releaseLock: releaseLock, + releaseLock: lease.releaseLock, vmLock: vmLock, record: record, } - roots = nil + lease.roots = nil + lease.releaseLock = nil directories = nil - releaseLock = nil vmLock = nil - return lease, nil + return launch, nil +} + +func (lease *releaseSlotLease) requireUnreferenced( + ctx context.Context, + release releaseIdentity, +) (returnErr error) { + if lease == nil || lease.roots == nil || lease.releaseLock == nil || lease.releaseLock.fd < 0 { + return instanceStateError("", "release slot lease is not active") + } + if err := validateReleaseIdentityValue(release); err != nil { + return instanceValidationError(err.Error()) + } + if releaseSlotForIdentity(release) != lease.slot { + return instanceValidationError("release identity does not match held release slot") + } + var indexLock *authorityLock + defer func() { + if indexLock != nil { + returnErr = appendInstanceOperationError( + returnErr, + "release global index lock", + indexLock.Release(context.WithoutCancel(ctx)), + ) + } + }() + indexLock, err := lease.roots.acquireIndexLock(ctx) + if err != nil { + return err + } + referenced, err := lease.roots.releaseIsReferenced(ctx, release) + if err != nil { + return err + } + if referenced { + return errs.New( + errs.CodeVMStateInvalid, + "release is referenced by active privileged VM authority", + errs.WithClass(errs.ClassConflict), + ) + } + if err := indexLock.Release(ctx); err != nil { + indexLock = nil + return err + } + indexLock = nil + return nil } func (authority *instanceAuthority) LockRegistered( @@ -243,62 +324,6 @@ func (lease *cleanupLease) Complete(ctx context.Context) error { return nil } -func (authority *instanceAuthority) LockUnreferencedRelease( - ctx context.Context, - release releaseIdentity, -) (_ *releaseLease, returnErr error) { - if err := validateReleaseIdentityValue(release); err != nil { - return nil, instanceValidationError(err.Error()) - } - roots, err := openInstanceAuthorityRoots(ctx, authority.deps, authority.policy) - if err != nil { - return nil, err - } - var releaseLock, indexLock *authorityLock - defer func() { - if returnErr == nil { - return - } - returnErr = releaseInstanceOperationResources( - ctx, - returnErr, - nil, - indexLock, - releaseLock, - nil, - roots, - ) - }() - releaseLock, err = roots.acquireReleaseLock(ctx, release) - if err != nil { - return nil, err - } - indexLock, err = roots.acquireIndexLock(ctx) - if err != nil { - return nil, err - } - referenced, err := roots.releaseIsReferenced(ctx, release) - if err != nil { - return nil, err - } - if referenced { - return nil, errs.New( - errs.CodeVMStateInvalid, - "release is referenced by active privileged VM authority", - errs.WithClass(errs.ClassConflict), - ) - } - if err := indexLock.Release(ctx); err != nil { - indexLock = nil - return nil, err - } - indexLock = nil - lease := &releaseLease{roots: roots, releaseLock: releaseLock} - roots = nil - releaseLock = nil - return lease, nil -} - func (authority *instanceAuthority) lockOwnedInstance( ctx context.Context, caller instanceCaller, @@ -431,7 +456,7 @@ func (lease *cleanupLease) Release(ctx context.Context) error { return err } -func (lease *releaseLease) Release(ctx context.Context) error { +func (lease *releaseSlotLease) Release(ctx context.Context) error { if lease == nil { return nil } diff --git a/internal/service/jailer/instance_authority_test.go b/internal/service/jailer/instance_authority_test.go index 27f0afc4..97a7a468 100644 --- a/internal/service/jailer/instance_authority_test.go +++ b/internal/service/jailer/instance_authority_test.go @@ -17,14 +17,202 @@ import ( "mvmctl/pkg/errs" ) -func TestInstanceAuthorityRegisterLaunchAndLockRegistered(t *testing.T) { +func TestInstanceAuthorityLocksReleaseSlotBeforeIdentityResolution(t *testing.T) { + t.Parallel() + + authority, _ := newTestInstanceAuthority(t) + release := testLaunchRegistration().release + slot := releaseSlot{version: release.version, architecture: release.architecture} + + lease, err := authority.lockReleaseSlot(context.Background(), slot) + require.NoError(t, err) + require.NotNil(t, lease) + assert.Equal(t, slot, lease.slot) + require.NoError(t, lease.Release(context.Background())) +} + +func TestReleaseSlotLeaseTransfersToSuccessfulLaunch(t *testing.T) { + t.Parallel() + + root := prepareInstanceAuthorityTestRoot(t) + deps := realInstanceAuthorityDeps() + realWaitLock := deps.waitLock + contentionObserved := make(chan struct{}) + retryLock := make(chan struct{}) + var contentionOnce sync.Once + deps.waitLock = func(ctx context.Context) error { + contentionOnce.Do(func() { close(contentionObserved) }) + select { + case <-retryLock: + return realWaitLock(ctx) + case <-ctx.Done(): + return ctx.Err() + } + } + authority := newInstanceAuthorityWithPolicy(deps, testInstanceAuthorityPolicy(root)) + registration := testLaunchRegistration() + slot := releaseSlotForIdentity(registration.release) + slotLease, err := authority.lockReleaseSlot( + context.Background(), + slot, + ) + require.NoError(t, err) + + launch, err := slotLease.registerLaunch( + context.Background(), + instanceCaller{uid: testAuthorityUID}, + registration, + ) + require.NoError(t, err) + require.NotNil(t, launch) + assert.Nil(t, slotLease.roots) + assert.Nil(t, slotLease.releaseLock) + assert.Equal(t, registration.release, launch.record.release) + assert.FileExists(t, filepath.Join(root, "var/lib/mvmctl/instances/1000/"+testVMID+".json")) + require.NoError(t, slotLease.Release(context.Background())) + + type result struct { + lease *releaseSlotLease + err error + } + resultCh := make(chan result, 1) + waitCtx, cancel := context.WithTimeout(context.Background(), time.Second) + defer cancel() + go func() { + lease, lockErr := authority.lockReleaseSlot(waitCtx, slot) + resultCh <- result{lease: lease, err: lockErr} + }() + select { + case <-contentionObserved: + case <-waitCtx.Done(): + require.Failf(t, "transferred release lock contention was not observed", "error: %v", waitCtx.Err()) + } + require.NoError(t, launch.Release(context.Background())) + close(retryLock) + select { + case acquired := <-resultCh: + require.NoError(t, acquired.err) + require.NotNil(t, acquired.lease) + require.NoError(t, acquired.lease.Release(context.Background())) + case <-waitCtx.Done(): + require.Failf(t, "transferred release lock remained blocked", "error: %v", waitCtx.Err()) + } +} + +func TestReleaseSlotLeaseRequiresResolvedIdentityToBeUnreferenced(t *testing.T) { + t.Parallel() + + authority, _ := newTestInstanceAuthority(t) + release := testLaunchRegistration().release + slotLease, err := authority.lockReleaseSlot(context.Background(), releaseSlotForIdentity(release)) + require.NoError(t, err) + + require.NoError(t, slotLease.requireUnreferenced(context.Background(), release)) + require.NoError(t, slotLease.Release(context.Background())) +} + +func TestReleaseSlotLeaseRejectsMismatchedLaunchWithoutWritingRecord(t *testing.T) { + t.Parallel() + + root := prepareInstanceAuthorityTestRoot(t) + deps := realInstanceAuthorityDeps() + realFlock := deps.flock + var exclusiveAcquisitions atomic.Int32 + deps.flock = func(ctx context.Context, fd int, how int) error { + err := realFlock(ctx, fd, how) + if err == nil && how == unix.LOCK_EX|unix.LOCK_NB { + exclusiveAcquisitions.Add(1) + } + return err + } + authority := newInstanceAuthorityWithPolicy(deps, testInstanceAuthorityPolicy(root)) + registration := testLaunchRegistration() + slotLease, err := authority.lockReleaseSlot( + context.Background(), + releaseSlotForIdentity(registration.release), + ) + require.NoError(t, err) + assert.Equal(t, int32(1), exclusiveAcquisitions.Load()) + + registration.release.version = "1.17.0" + launch, err := slotLease.registerLaunch( + context.Background(), + instanceCaller{uid: testAuthorityUID}, + registration, + ) + assert.Nil(t, launch) + require.Error(t, err) + domainErr := errs.AsDomainError(err) + require.NotNil(t, domainErr) + assert.Equal(t, errs.CodeValidationFailed, domainErr.Code) + assert.Equal(t, int32(1), exclusiveAcquisitions.Load()) + assert.NoFileExists(t, filepath.Join(root, "var/lib/mvmctl/instances/1000/"+testVMID+".json")) + assert.NotNil(t, slotLease.roots) + assert.NotNil(t, slotLease.releaseLock) + require.NoError(t, slotLease.Release(context.Background())) +} + +func TestReleaseSlotLeaseIndexFailureRetainsPreparedSlot(t *testing.T) { + t.Parallel() + + tests := map[string]func(context.Context, *releaseSlotLease, launchRegistration) error{ + "launch registration": func( + ctx context.Context, + lease *releaseSlotLease, + registration launchRegistration, + ) error { + _, err := lease.registerLaunch(ctx, instanceCaller{uid: testAuthorityUID}, registration) + return err + }, + "release reference check": func( + ctx context.Context, + lease *releaseSlotLease, + registration launchRegistration, + ) error { + return lease.requireUnreferenced(ctx, registration.release) + }, + } + for name, run := range tests { + t.Run(name, func(t *testing.T) { + root := prepareInstanceAuthorityTestRoot(t) + deps := realInstanceAuthorityDeps() + realFlock := deps.flock + indexErr := errors.New("index lock failed") + var exclusiveAttempts atomic.Int32 + deps.flock = func(ctx context.Context, fd int, how int) error { + if how == unix.LOCK_EX|unix.LOCK_NB && exclusiveAttempts.Add(1) == 2 { + return indexErr + } + return realFlock(ctx, fd, how) + } + authority := newInstanceAuthorityWithPolicy(deps, testInstanceAuthorityPolicy(root)) + registration := testLaunchRegistration() + lease, err := authority.lockReleaseSlot( + context.Background(), + releaseSlotForIdentity(registration.release), + ) + require.NoError(t, err) + + err = run(context.Background(), lease, registration) + require.Error(t, err) + assert.ErrorIs(t, err, indexErr) + assert.Equal(t, int32(2), exclusiveAttempts.Load()) + assert.NotNil(t, lease.roots) + assert.NotNil(t, lease.releaseLock) + assert.NoFileExists(t, filepath.Join(root, "var/lib/mvmctl/instances/1000/"+testVMID+".json")) + require.NoError(t, lease.Release(context.Background())) + }) + } +} + +func TestInstanceAuthorityReleaseSlotRegistersLaunchAndLockRegistered(t *testing.T) { t.Parallel() authority, root := newTestInstanceAuthority(t) caller := instanceCaller{uid: testAuthorityUID} registration := testLaunchRegistration() - launch, err := authority.RegisterLaunch(context.Background(), caller, registration) + launch, err := registerTestLaunch(context.Background(), authority, caller, registration) require.NoError(t, err) assert.Equal(t, instanceLifecycleRegistered, launch.record.lifecycle) assert.FileExists(t, filepath.Join(root, "var/lib/mvmctl/instances/1000/"+testVMID+".json")) @@ -43,7 +231,7 @@ func TestInstanceAuthorityRegisterLaunchAndLockRegistered(t *testing.T) { assert.Equal(t, registration.process, registered.record.process) require.NoError(t, registered.Release(context.Background())) - duplicate, err := authority.RegisterLaunch(context.Background(), caller, registration) + duplicate, err := registerTestLaunch(context.Background(), authority, caller, registration) assert.Nil(t, duplicate) require.Error(t, err) assert.Equal(t, errs.CodeVMAlreadyExists, errs.AsDomainError(err).Code) @@ -54,7 +242,7 @@ func TestInstanceAuthorityCleanupAndSameOwnerRelaunch(t *testing.T) { authority, _ := newTestInstanceAuthority(t) caller := instanceCaller{uid: testAuthorityUID} - launch, err := authority.RegisterLaunch(context.Background(), caller, testLaunchRegistration()) + launch, err := registerTestLaunch(context.Background(), authority, caller, testLaunchRegistration()) require.NoError(t, err) require.NoError(t, launch.Release(context.Background())) @@ -71,7 +259,7 @@ func TestInstanceAuthorityCleanupAndSameOwnerRelaunch(t *testing.T) { require.Error(t, err) assert.Equal(t, errs.CodeVMNotRunning, errs.AsDomainError(err).Code) - relaunched, err := authority.RegisterLaunch(context.Background(), caller, testLaunchRegistration()) + relaunched, err := registerTestLaunch(context.Background(), authority, caller, testLaunchRegistration()) require.NoError(t, err) assert.Equal(t, uint64(1), relaunched.record.cleanupGeneration) require.NoError(t, relaunched.Release(context.Background())) @@ -83,11 +271,11 @@ func TestInstanceAuthorityRejectsForeignCaller(t *testing.T) { authority, _ := newTestInstanceAuthority(t) owner := instanceCaller{uid: testAuthorityUID} foreign := instanceCaller{uid: testAuthorityUID + 1} - launch, err := authority.RegisterLaunch(context.Background(), owner, testLaunchRegistration()) + launch, err := registerTestLaunch(context.Background(), authority, owner, testLaunchRegistration()) require.NoError(t, err) require.NoError(t, launch.Release(context.Background())) - foreignLaunch, err := authority.RegisterLaunch(context.Background(), foreign, testLaunchRegistration()) + foreignLaunch, err := registerTestLaunch(context.Background(), authority, foreign, testLaunchRegistration()) assert.Nil(t, foreignLaunch) require.Error(t, err) assert.Equal(t, errs.CodeUnauthorized, errs.AsDomainError(err).Code) @@ -115,7 +303,7 @@ func TestInstanceAuthorityCrossUIDFirstLaunchRaceHasOneOwner(t *testing.T) { for _, caller := range callers { go func() { <-start - lease, err := authority.RegisterLaunch(context.Background(), caller, testLaunchRegistration()) + lease, err := registerTestLaunch(context.Background(), authority, caller, testLaunchRegistration()) if err == nil { success.Add(1) err = lease.Release(context.Background()) @@ -169,15 +357,24 @@ func TestInstanceAuthorityRejectsCorruptGlobalClaim(t *testing.T) { recordDir := filepath.Join(root, "var/lib/mvmctl/instances/1000") require.NoError(t, os.MkdirAll(recordDir, 0700)) require.NoError(t, os.WriteFile(filepath.Join(recordDir, testVMID+".json"), []byte(`{"broken":true}`), 0600)) + registration := testLaunchRegistration() + slotLease, err := authority.lockReleaseSlot( + context.Background(), + releaseSlotForIdentity(registration.release), + ) + require.NoError(t, err) - lease, err := authority.RegisterLaunch( + lease, err := slotLease.registerLaunch( context.Background(), instanceCaller{uid: testAuthorityUID + 1}, - testLaunchRegistration(), + registration, ) assert.Nil(t, lease) require.Error(t, err) assert.Equal(t, errs.CodeVMAtomicFailed, errs.AsDomainError(err).Code) + assert.NotNil(t, slotLease.roots) + assert.NotNil(t, slotLease.releaseLock) + require.NoError(t, slotLease.Release(context.Background())) } func TestInstanceAuthorityRejectsRecordDirectoryIdentityMismatch(t *testing.T) { @@ -202,38 +399,37 @@ func TestInstanceAuthorityRejectsRecordDirectoryIdentityMismatch(t *testing.T) { assert.Equal(t, errs.CodeVMAtomicFailed, errs.AsDomainError(err).Code) } -func TestInstanceAuthorityReleaseReferenceLeaseBlocksLaunch(t *testing.T) { +func TestInstanceAuthorityReleaseSlotReferenceCheckTracksLifecycle(t *testing.T) { t.Parallel() authority, _ := newTestInstanceAuthority(t) caller := instanceCaller{uid: testAuthorityUID} release := testLaunchRegistration().release - launch, err := authority.RegisterLaunch(context.Background(), caller, testLaunchRegistration()) + launch, err := registerTestLaunch(context.Background(), authority, caller, testLaunchRegistration()) require.NoError(t, err) require.NoError(t, launch.Release(context.Background())) - releaseLease, err := authority.LockUnreferencedRelease(context.Background(), release) - assert.Nil(t, releaseLease) + slotLease, err := authority.lockReleaseSlot(context.Background(), releaseSlotForIdentity(release)) + require.NoError(t, err) + err = slotLease.requireUnreferenced(context.Background(), release) require.Error(t, err) assert.Equal(t, errs.ClassConflict, errs.AsDomainError(err).Class) + assert.NotNil(t, slotLease.roots) + assert.NotNil(t, slotLease.releaseLock) + require.NoError(t, slotLease.Release(context.Background())) cleanup, err := authority.BeginCleanup(context.Background(), caller, testVMID) require.NoError(t, err) require.NoError(t, cleanup.Complete(context.Background())) require.NoError(t, cleanup.Release(context.Background())) - releaseLease, err = authority.LockUnreferencedRelease(context.Background(), release) + slotLease, err = authority.lockReleaseSlot(context.Background(), releaseSlotForIdentity(release)) require.NoError(t, err) - waitCtx, cancel := context.WithTimeout(context.Background(), 30*time.Millisecond) - defer cancel() - blocked, err := authority.RegisterLaunch(waitCtx, caller, testLaunchRegistration()) - assert.Nil(t, blocked) - require.Error(t, err) - assert.ErrorIs(t, err, context.DeadlineExceeded) - require.NoError(t, releaseLease.Release(context.Background())) + require.NoError(t, slotLease.requireUnreferenced(context.Background(), release)) + require.NoError(t, slotLease.Release(context.Background())) } -func TestInstanceAuthorityReleaseLeaseSerializesCanonicalStoreSlot(t *testing.T) { +func TestInstanceAuthorityReleaseSlotLeaseSerializesCompetingIdentities(t *testing.T) { t.Parallel() root := prepareInstanceAuthorityTestRoot(t) @@ -252,9 +448,9 @@ func TestInstanceAuthorityReleaseLeaseSerializesCanonicalStoreSlot(t *testing.T) } } authority := newInstanceAuthorityWithPolicy(deps, testInstanceAuthorityPolicy(root)) - held, err := authority.LockUnreferencedRelease( + held, err := authority.lockReleaseSlot( context.Background(), - testLaunchRegistration().release, + releaseSlotForIdentity(testLaunchRegistration().release), ) require.NoError(t, err) t.Cleanup(func() { @@ -264,14 +460,17 @@ func TestInstanceAuthorityReleaseLeaseSerializesCanonicalStoreSlot(t *testing.T) }) type result struct { - lease *releaseLease + lease *releaseSlotLease err error } resultCh := make(chan result, 1) waitCtx, cancel := context.WithTimeout(context.Background(), time.Second) defer cancel() go func() { - lease, lockErr := authority.LockUnreferencedRelease(waitCtx, testAlternateReleaseIdentity()) + lease, lockErr := authority.lockReleaseSlot( + waitCtx, + releaseSlotForIdentity(testAlternateReleaseIdentity()), + ) resultCh <- result{lease: lease, err: lockErr} }() @@ -294,7 +493,54 @@ func TestInstanceAuthorityReleaseLeaseSerializesCanonicalStoreSlot(t *testing.T) } } -func TestInstanceAuthorityReleaseLeaseRejectsCorruptAuthorityRecord(t *testing.T) { +func TestInstanceAuthorityReleaseSlotWaitIsCancellableAndOwnerRemainsActive(t *testing.T) { + t.Parallel() + + root := prepareInstanceAuthorityTestRoot(t) + deps := realInstanceAuthorityDeps() + contentionObserved := make(chan struct{}) + var contentionOnce sync.Once + deps.waitLock = func(ctx context.Context) error { + contentionOnce.Do(func() { close(contentionObserved) }) + <-ctx.Done() + return ctx.Err() + } + authority := newInstanceAuthorityWithPolicy(deps, testInstanceAuthorityPolicy(root)) + slot := releaseSlotForIdentity(testLaunchRegistration().release) + held, err := authority.lockReleaseSlot(context.Background(), slot) + require.NoError(t, err) + + type result struct { + lease *releaseSlotLease + err error + } + resultCh := make(chan result, 1) + waitCtx, cancel := context.WithCancel(context.Background()) + go func() { + lease, lockErr := authority.lockReleaseSlot(waitCtx, slot) + resultCh <- result{lease: lease, err: lockErr} + }() + + select { + case <-contentionObserved: + case <-time.After(time.Second): + require.Fail(t, "release slot lock contention was not observed") + } + cancel() + blocked := <-resultCh + assert.Nil(t, blocked.lease) + require.Error(t, blocked.err) + assert.ErrorIs(t, blocked.err, context.Canceled) + assert.NotNil(t, held.roots) + assert.NotNil(t, held.releaseLock) + require.NoError(t, held.Release(context.Background())) + + reacquired, err := authority.lockReleaseSlot(context.Background(), slot) + require.NoError(t, err) + require.NoError(t, reacquired.Release(context.Background())) +} + +func TestInstanceAuthorityReleaseSlotReferenceCheckRejectsCorruptAuthorityRecord(t *testing.T) { t.Parallel() authority, root := newTestInstanceAuthority(t) @@ -302,10 +548,15 @@ func TestInstanceAuthorityReleaseLeaseRejectsCorruptAuthorityRecord(t *testing.T require.NoError(t, os.MkdirAll(recordDir, 0700)) require.NoError(t, os.WriteFile(filepath.Join(recordDir, testVMID+".json"), []byte(`{"broken":true}`), 0600)) - lease, err := authority.LockUnreferencedRelease(context.Background(), testLaunchRegistration().release) - assert.Nil(t, lease) + release := testLaunchRegistration().release + lease, err := authority.lockReleaseSlot(context.Background(), releaseSlotForIdentity(release)) + require.NoError(t, err) + err = lease.requireUnreferenced(context.Background(), release) require.Error(t, err) assert.Equal(t, errs.CodeVMAtomicFailed, errs.AsDomainError(err).Code) + assert.NotNil(t, lease.roots) + assert.NotNil(t, lease.releaseLock) + require.NoError(t, lease.Release(context.Background())) } func TestInstanceAuthorityRegisterLockOrder(t *testing.T) { @@ -337,11 +588,17 @@ func TestInstanceAuthorityRegisterLockOrder(t *testing.T) { return err } authority := newInstanceAuthorityWithPolicy(deps, testInstanceAuthorityPolicy(root)) + registration := testLaunchRegistration() + slotLease, err := authority.lockReleaseSlot( + context.Background(), + releaseSlotForIdentity(registration.release), + ) + require.NoError(t, err) - lease, err := authority.RegisterLaunch( + lease, err := slotLease.registerLaunch( context.Background(), instanceCaller{uid: testAuthorityUID}, - testLaunchRegistration(), + registration, ) require.NoError(t, err) require.NoError(t, lease.Release(context.Background())) @@ -364,11 +621,17 @@ func TestInstanceAuthorityUsesOnlyFixedRootPathOpen(t *testing.T) { return realOpen(ctx, path, flags, mode) } authority := newInstanceAuthorityWithPolicy(deps, testInstanceAuthorityPolicy(root)) + registration := testLaunchRegistration() + slotLease, err := authority.lockReleaseSlot( + context.Background(), + releaseSlotForIdentity(registration.release), + ) + require.NoError(t, err) - lease, err := authority.RegisterLaunch( + lease, err := slotLease.registerLaunch( context.Background(), instanceCaller{uid: testAuthorityUID}, - testLaunchRegistration(), + registration, ) require.NoError(t, err) require.NoError(t, lease.Release(context.Background())) @@ -382,7 +645,10 @@ func TestInstanceAuthorityPostRenameErrorReturnsNoLeaseAndDurableRecord(t *testi prepared := openTestInstanceAuthorityRoots(t, root) uidDirs, err := prepared.openUIDDirectories(context.Background(), testAuthorityUID) require.NoError(t, err) - releaseLock, err := prepared.acquireReleaseLock(context.Background(), testLaunchRegistration().release) + releaseLock, err := prepared.acquireReleaseSlotLock( + context.Background(), + releaseSlotForIdentity(testLaunchRegistration().release), + ) require.NoError(t, err) indexLock, err := prepared.acquireIndexLock(context.Background()) require.NoError(t, err) @@ -408,11 +674,17 @@ func TestInstanceAuthorityPostRenameErrorReturnsNoLeaseAndDurableRecord(t *testi return realFsync(ctx, fd) } authority := newInstanceAuthorityWithPolicy(deps, testInstanceAuthorityPolicy(root)) + registration := testLaunchRegistration() + slotLease, err := authority.lockReleaseSlot( + context.Background(), + releaseSlotForIdentity(registration.release), + ) + require.NoError(t, err) - lease, err := authority.RegisterLaunch( + lease, err := slotLease.registerLaunch( context.Background(), instanceCaller{uid: testAuthorityUID}, - testLaunchRegistration(), + registration, ) assert.Nil(t, lease) require.Error(t, err) @@ -421,6 +693,9 @@ func TestInstanceAuthorityPostRenameErrorReturnsNoLeaseAndDurableRecord(t *testi assert.Equal(t, true, domainErr.Details["record_replaced"]) assert.Equal(t, true, domainErr.Details["durability_uncertain"]) assert.FileExists(t, filepath.Join(root, "var/lib/mvmctl/instances/1000/"+testVMID+".json")) + assert.NotNil(t, slotLease.roots) + assert.NotNil(t, slotLease.releaseLock) + require.NoError(t, slotLease.Release(context.Background())) } func newTestInstanceAuthority(t *testing.T) (*instanceAuthority, string) { @@ -433,6 +708,27 @@ func newTestInstanceAuthority(t *testing.T) (*instanceAuthority, string) { ), root } +func registerTestLaunch( + ctx context.Context, + authority *instanceAuthority, + caller instanceCaller, + registration launchRegistration, +) (*launchLease, error) { + slotLease, err := authority.lockReleaseSlot(ctx, releaseSlotForIdentity(registration.release)) + if err != nil { + return nil, err + } + launch, err := slotLease.registerLaunch(ctx, caller, registration) + if err != nil { + return nil, appendInstanceOperationError( + err, + "release rejected test release slot lease", + slotLease.Release(context.WithoutCancel(ctx)), + ) + } + return launch, nil +} + func writeTestAuthorityRecord(t *testing.T, root string, record instanceRecord) { t.Helper() diff --git a/internal/service/jailer/instance_lock.go b/internal/service/jailer/instance_lock.go index d19a80f5..66c8cef2 100644 --- a/internal/service/jailer/instance_lock.go +++ b/internal/service/jailer/instance_lock.go @@ -21,11 +21,11 @@ func (roots *instanceAuthorityRoots) acquireIndexLock(ctx context.Context) (*aut return acquireAuthorityLock(ctx, roots.deps, roots.policy, roots.runtimeFD, "index.lock", "global index") } -func (roots *instanceAuthorityRoots) acquireReleaseLock( +func (roots *instanceAuthorityRoots) acquireReleaseSlotLock( ctx context.Context, - release releaseIdentity, + slot releaseSlot, ) (*authorityLock, error) { - if err := validateReleaseIdentityValue(release); err != nil { + if err := validateReleaseSlotValue(slot); err != nil { return nil, instanceValidationError(err.Error()) } return acquireAuthorityLock( @@ -33,7 +33,7 @@ func (roots *instanceAuthorityRoots) acquireReleaseLock( roots.deps, roots.policy, roots.releasesFD, - releaseLockName(release), + releaseLockName(slot), "release", ) } @@ -154,8 +154,8 @@ func verifyInstanceRegularFile( return nil } -func releaseLockName(release releaseIdentity) string { - canonical := release.version + "\x00" + release.architecture +func releaseLockName(slot releaseSlot) string { + canonical := slot.version + "\x00" + slot.architecture digest := sha256.Sum256([]byte(canonical)) return hex.EncodeToString(digest[:]) + ".lock" } diff --git a/internal/service/jailer/instance_lock_test.go b/internal/service/jailer/instance_lock_test.go index ee212e3e..26608b59 100644 --- a/internal/service/jailer/instance_lock_test.go +++ b/internal/service/jailer/instance_lock_test.go @@ -17,9 +17,21 @@ func TestReleaseLockNameUsesCanonicalStoreSlot(t *testing.T) { differentArchitecture := release differentArchitecture.architecture = "aarch64" - assert.Equal(t, releaseLockName(release), releaseLockName(sameSlot)) - assert.NotEqual(t, releaseLockName(release), releaseLockName(differentVersion)) - assert.NotEqual(t, releaseLockName(release), releaseLockName(differentArchitecture)) + assert.Equal( + t, + releaseLockName(releaseSlotForIdentity(release)), + releaseLockName(releaseSlotForIdentity(sameSlot)), + ) + assert.NotEqual( + t, + releaseLockName(releaseSlotForIdentity(release)), + releaseLockName(releaseSlotForIdentity(differentVersion)), + ) + assert.NotEqual( + t, + releaseLockName(releaseSlotForIdentity(release)), + releaseLockName(releaseSlotForIdentity(differentArchitecture)), + ) } func testAlternateReleaseIdentity() releaseIdentity { diff --git a/internal/service/jailer/instance_types.go b/internal/service/jailer/instance_types.go index 318f9d53..b1d78b82 100644 --- a/internal/service/jailer/instance_types.go +++ b/internal/service/jailer/instance_types.go @@ -30,6 +30,11 @@ type instanceCaller struct { uid uint32 } +type releaseSlot struct { + version string + architecture string +} + type releaseIdentity struct { version string architecture string @@ -203,11 +208,8 @@ func validateInstanceRecord(record instanceRecord) error { } func validateReleaseIdentityValue(release releaseIdentity) error { - if !versionPattern.MatchString(release.version) || strings.Contains(release.version, "..") { - return fmt.Errorf("instance authority release version is invalid") - } - if release.architecture != "x86_64" && release.architecture != "aarch64" { - return fmt.Errorf("instance authority release architecture is invalid") + if err := validateReleaseSlotValue(releaseSlotForIdentity(release)); err != nil { + return err } if !authorityHashPattern.MatchString(release.firecrackerSHA256) || !authorityHashPattern.MatchString(release.jailerSHA256) { @@ -219,6 +221,20 @@ func validateReleaseIdentityValue(release releaseIdentity) error { return nil } +func validateReleaseSlotValue(slot releaseSlot) error { + if !versionPattern.MatchString(slot.version) || strings.Contains(slot.version, "..") { + return fmt.Errorf("instance authority release version is invalid") + } + if slot.architecture != "x86_64" && slot.architecture != "aarch64" { + return fmt.Errorf("instance authority release architecture is invalid") + } + return nil +} + +func releaseSlotForIdentity(release releaseIdentity) releaseSlot { + return releaseSlot{version: release.version, architecture: release.architecture} +} + func validateProcessIdentityValue(vmID string, process processIdentity) error { if process.pid <= 0 || process.pid > math.MaxInt32 || process.startTimeTicks == 0 { return fmt.Errorf("instance authority process identity is invalid") From 6fec924a43edf644265d994ffab3eb6d98970431 Mon Sep 17 00:00:00 2001 From: AlanD20 Date: Sun, 23 Aug 2026 16:05:47 +0200 Subject: [PATCH 0029/1457] docs: define privileged control transport --- CHANGELOG.md | 5 +-- ...owned-single-binary-privileged-dispatch.md | 36 ++++++++++++++++--- tasks/plan.md | 32 +++++++++++++++-- tasks/todo.md | 25 ++++++++----- 4 files changed, 80 insertions(+), 18 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index cee7efb1..dd7b22ed 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -52,8 +52,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 and `cleaned` records beneath `/var/lib/mvmctl/instances/` and persistent ownership tombstones. - Added descriptor-relative, no-follow traversal; strict bounded record decoding; atomic durable replacement; reusable root-owned runtime locks; and the fixed `release -> global index -> VM` lock order. -- The private release-removal lease now locks the planned canonical architecture/version store slot, so conflicting - identities for one future slot cannot race under different hash-derived locks when Task 6 wires the new store. +- Prepared release leases now lock the canonical architecture/version store slot before identity resolution. Launch + registration transfers that same lease only after durable authority state; removal/replacement retains it while + checking exact references. Competing identities cannot race through different hash-derived locks. - Global VM-ID claims now fail closed on foreign, duplicate, corrupt, unreadable, or inconsistent authority records, and trusted release removal can acquire a lease only when no active authority record references the exact release. - This substrate is private and not yet wired into public VM lifecycle operations; that integration remains a release diff --git a/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md b/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md index 1d62e1ee..75c1343f 100644 --- a/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md +++ b/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md @@ -89,6 +89,32 @@ action-specific. Failures preserve the original `DomainError` code, class, entit the normal-user client can make safe retry and recovery decisions. Logs and subprocess exit status are diagnostic only; they never replace the structured result or turn a malformed/missing response into a generic successful operation. +The control channel is one full-duplex Unix stream socket duplicated onto file descriptor 0 across `sudo -n`. File +descriptor 0 is control-only and is marked close-on-exec before any handler can launch a descendant. Standard output and +standard error are never parsed as authority. VM console input/output must use a separate typed relay owned by the launch +implementation; it cannot reuse the control channel. Using descriptor 0 avoids sudo policy for inherited descriptors, +while the socketpair permits the client to upload a bounded release archive and receive an early rejection concurrently. +Root requires descriptor 0 to be an `AF_UNIX` `SOCK_STREAM` socket and requires its Linux `SO_PEERCRED` UID/GID to match +the authenticated sudo caller. The peer PID is positive and audit-only; it is not an authorization identity. + +Each request starts with fixed `MVMREQ01` magic, a network-order 32-bit JSON-header length, and a network-order 64-bit +payload length. The strict header is at most 64 KiB and contains schema version 1, the action repeated from argv, and the +action-specific body. The repeated action must match. Only release install may carry a payload: either zero bytes or an +exact stream of at most 128 MiB. The caller half-closes its write side after the declared bytes, and root requires EOF so +truncation or trailing bytes fail closed before any payload-dependent effect. Zero-payload operations likewise require +EOF before effects. Authentication, framing, and header failures may return an early error without draining an advertised +payload; the concurrent client treats the resulting uploader `EPIPE` as expected once it has a valid response. Root +applies bounded header and archive deadlines and reads no second request. + +Each response starts with fixed `MVMRES01` magic and a network-order 32-bit JSON length followed by at most 64 KiB of +strict schema-version-1 JSON. It contains the matching action, a closed `success` or `error` status, and exactly one typed +result or error. Error envelopes carry code, stable string class, message, operation, entity, and normalized details; +they never serialize wrapped causes, stacks, stderr, or arbitrary Go values. Wire details permit only bounded booleans, +strings, signed integers, and string arrays. Unsupported details are omitted visibly. An error response delivered after +all effects and cleanup exits successfully at the process layer. A start failure has known no-effect status; any started +process with a missing, malformed, truncated, mismatched, or oversized response returns `CodeProcessError` with +`process_started: true` and `outcome_unknown: true`. + ### Authorization roles The root-owned executable establishes code integrity. The `mvm` Unix group remains the authorization role for @@ -161,10 +187,12 @@ and system-damage invariants. These are receiver trust-boundary checks, not dupl ### Release trust -The caller supplies only a validated release version and architecture. Privileged release code constructs the permitted -Firecracker release and checksum URLs and downloads both independently from the fixed official origin with a dedicated, -bounded HTTPS-only client. It disables proxies and does not consult caller configuration, the general download cache, or -`MVM_ASSET_MIRROR`. Caller-provided paths, URLs, checksums, and archive bytes are never release authority. +The caller supplies a validated release version and architecture and may stream an exact-length bounded archive body for +cache efficiency. Privileged release code constructs the permitted Firecracker release and checksum URLs and obtains the +checksum independently from the fixed official origin with a dedicated bounded HTTPS-only client and proxies disabled. +It hashes the complete archive body against that checksum before parsing it. If no body is supplied, root may fetch the +fixed archive itself. Root never opens a caller path or accepts a caller URL or checksum. The normal-user client may read +`MVM_ASSET_MIRROR`, but those streamed bytes are transport, never release authority. Extraction occurs in a root-owned temporary directory with bounded compressed/decompressed input, a reviewed allowlist for the complete upstream archive layout, and rejection of path traversal, symlinks, hardlinks, devices, sparse files, diff --git a/tasks/plan.md b/tasks/plan.md index d150c8f3..2e1c60f1 100644 --- a/tasks/plan.md +++ b/tasks/plan.md @@ -60,6 +60,9 @@ Durable user state does not move wholesale to `/run` or `/var/run`. XDG state se switch. It imports no API or core domain and exposes no raw command/argv/effect-path interface. One typed managed-cache locator may identify the caller's persistent namespace; every effect below it is selected by typed IDs and fixed receiver-derived basenames. +- `internal/lib/privilegedwire/` owns only the strict length-framed request/response codec and bounded `DomainError` + normalization. `internal/lib/system/` owns the fd-0 socketpair subprocess transport. Neither package exposes a generic + privileged effect or action dispatcher. - Capability modules under `internal/service/{jailer,network,firewall,loopmount,host}/` own typed privileged effects. - `pkg/api/` remains the sole orchestrator of multiple core domains. - Core domains never import sibling `internal/core/*` packages. @@ -68,6 +71,26 @@ Durable user state does not move wholesale to `/run` or `/var/run`. XDG state se - Each new capability uses named typed methods matching existing service conventions. No generic backend, operation, selector registry, callback-under-lock, raw firewall expression, raw cgroup key, or raw netns action is permitted. +### Privileged control transport + +The normal and root processes communicate over one full-duplex Unix stream socket duplicated onto the privileged +process's file descriptor 0 through `sudo -n`. The root process marks it close-on-exec immediately. Standard output and +standard error remain outside the authority channel, and launch/console traffic uses a separate typed relay. The system +runner owns concurrent upload/response I/O so an early root rejection cannot deadlock behind a large archive write. +Root requires an `AF_UNIX` `SOCK_STREAM` descriptor and Linux `SO_PEERCRED` UID/GID equal to the authenticated sudo +caller. The positive peer PID is recorded only for audit and never substitutes for typed receiver authorization. + +Requests use fixed `MVMREQ01` magic, a 32-bit network-order JSON-header length capped at 64 KiB, a 64-bit network-order +payload length, the strict schema-version-1 header, and the exact optional payload. The header repeats the fixed argv +action and must match it. Only typed release install accepts a payload: zero bytes or at most 128 MiB. The caller +half-closes after the declared bytes. Root requires exact length and EOF before payload-dependent effects and before all +zero-payload effects, but may respond to pre-effect authentication/header failure without draining a claimed payload. +Responses use `MVMRES01`, a 32-bit length capped at 64 KiB, and one strict matching-action success/result or +error/`DomainError` envelope. A delivered envelope is the final effect record even if process reaping later reports an +anomaly; no domain effect or fallible outcome check is allowed after the response. Missing, malformed, truncated, +mismatched, or oversized responses from a started process report +`process_started: true` and `outcome_unknown: true`; stderr and exit text are never parsed as authority. + ### VM authority interface The first implementation slice is private to `internal/service/jailer`. It uses unexported typed values and these @@ -167,9 +190,12 @@ no VM effects after such an error. Task 6 adds a private `releaseAuthority` in `internal/service/jailer`. Its public privileged request contains only a validated release version, architecture, and explicit replacement intent. Root constructs the fixed official checksum -and archive URLs, downloads both through a dedicated bounded HTTPS-only client with proxies and the user asset mirror -disabled, verifies the archive before parsing it, validates the reviewed complete upstream member allowlist, and -extracts only Firecracker and Jailer. Caller paths, URLs, checksums, and archive bytes are prohibited. +and archive URLs and obtains the checksum independently through a dedicated bounded HTTPS-only client with proxies +disabled. For mirror efficiency, the normal-user client may stream an exact-length bounded archive body through the +typed install transport; root hashes it against the independently fetched checksum before parsing it. If no body is +supplied, root may fetch the fixed archive itself. Root never opens a caller path or accepts a caller URL or checksum, +and `MVM_ASSET_MIRROR` supplies bytes rather than authority. Extraction validates the reviewed complete upstream member +allowlist and extracts only Firecracker and Jailer. The strict root-owned manifest stores schema version, release slot, archive hash, and each executable's hash and size. The store is exactly `/var/lib/mvmctl/binaries///{firecracker,jailer,release.json}`. Binaries are diff --git a/tasks/todo.md b/tasks/todo.md index 508eaadd..30de7b0d 100644 --- a/tasks/todo.md +++ b/tasks/todo.md @@ -63,25 +63,29 @@ The final marker-only sudoers assertion is deferred until Task 14; current sudoe - [x] Unsupported versions fail closed. - [x] Strict bounded JSON rejects unknown, duplicate, case-fold duplicate, trailing, oversized, and malformed input. - [x] Caller identity, active `mvm` group, environment, and system executable are verified before dispatch. -- [ ] Add a strict versioned response envelope that preserves `DomainError` code, class, entity, and partial-state - details across the privileged subprocess boundary. +- [ ] Add a strict versioned request/response codec with 64 KiB frames, duplicate/unknown/depth rejection, matching + actions, typed results, and bounded `DomainError` code/class/entity/partial-state detail preservation. +- [ ] Add the fd-0 Unix socketpair control runner with concurrent upload/response, exact EOF, CLOEXEC, cancellation, + timeout, response/stderr bounds, early-rejection handling, and `FakeRunner` support. +- [ ] Return `process_started: true` and `outcome_unknown: true` for every started process without one valid final + response; stdout, stderr, and generic subprocess text are never action authority. - [ ] Keep the action switch closed until each capability has its own typed handler and abuse tests. ### Task 4: Add private root-owned VM instance authority **Owner:** engineer -**Status:** Foundation implemented; prepared release-slot correction in progress +**Status:** Complete; Tasks 6 and 8 consume the prepared authority lease **Dependencies:** Tasks 1 and 3 **Interface:** -- [ ] Implement private `lockReleaseSlot(ctx, slot) (*releaseSlotLease, error)` before release identity resolution. -- [ ] Implement private `releaseSlotLease.registerLaunch(ctx, caller, registration) (*launchLease, error)` with +- [x] Implement private `lockReleaseSlot(ctx, slot) (*releaseSlotLease, error)` before release identity resolution. +- [x] Implement private `releaseSlotLease.registerLaunch(ctx, caller, registration) (*launchLease, error)` with ownership transfer only after durable registration. - [x] Implement private `LockRegistered(ctx, caller, vmID) (*registeredLease, error)`. - [x] Implement private `BeginCleanup(ctx, caller, vmID) (*cleanupLease, error)`. - [x] Implement context-first `cleanupLease.Complete(ctx)` and checked lease release. -- [ ] Implement private `releaseSlotLease.requireUnreferenced(ctx, release)` for exact-identity removal/replacement. +- [x] Implement private `releaseSlotLease.requireUnreferenced(ctx, release)` for exact-identity removal/replacement. - [x] Expose no generic storage, callbacks, raw paths, actions, or state strings. **Authority and lifecycle:** @@ -115,7 +119,9 @@ The final marker-only sudoers assertion is deferred until Task 14; current sudoe - [x] Architect reviews actual interface, state schema, lock order, imports, and error details before commit. - [x] Key the release lock by canonical `(version, architecture)` store slot rather than binary hashes; prove two different identities for one slot serialize. -- [ ] Prove the slot lock is held before manifest/hash resolution and a mismatched identity writes no record. +- [x] Prove competing identities serialize on the slot, transfer occurs only after durable registration, pre-transfer + failures retain a releasable lease, and a mismatched slot writes no record or acquires index/VM locks. +- [ ] Task 6 proves this lease is acquired before root manifest/hash resolution and supplies only the manifest identity. **Expected files:** fixed root constants; private instance types, codec, descriptor store, lock, authority, and focused tests under `internal/service/jailer/`. No handler, transport, CLI, API, core-domain, or network changes. @@ -156,8 +162,9 @@ under `internal/service/jailer/`. No handler, transport, CLI, API, core-domain, - [ ] Store exact releases under `/var/lib/mvmctl/binaries//` with fixed Firecracker, Jailer, and strict root-owned manifest leaves. - [ ] Privileged code constructs the fixed official release/checksum locations from validated version and architecture. -- [ ] Root downloads the checksum and archive through a dedicated bounded HTTPS-only source with proxies and the general - user asset mirror disabled; caller-provided paths, checksums, URLs, and archive bytes are not authority. +- [ ] Root obtains the checksum independently through a dedicated bounded HTTPS-only source with proxies disabled. It + may consume an exact-length bounded archive stream supplied by the unprivileged client for mirror efficiency, but it + accepts no caller path, URL, or checksum and never opens or trusts the user asset mirror as authority. - [ ] Bounded extraction rejects traversal, links, devices, duplicates, unexpected members, and size/count overflow. - [ ] Validate the complete reviewed upstream member allowlist while extracting only Firecracker and Jailer; validate their ELF class/machine without executing downloaded code. From 6f3e8d1c9f39061ded5e04f18d0e410333b1961f Mon Sep 17 00:00:00 2001 From: AlanD20 Date: Sun, 23 Aug 2026 16:12:06 +0200 Subject: [PATCH 0030/1457] test: require complete pytest outcome evidence --- scripts/run-system-tests.py | 225 ++++++++-- scripts/system_test_outcomes.py | 224 ++++++++++ scripts/tests/test_run_system_tests.py | 347 +++++++++++++++ scripts/tests/test_system_test_outcomes.py | 495 +++++++++++++++++++++ tests/system/conftest.py | 105 +++++ 5 files changed, 1369 insertions(+), 27 deletions(-) create mode 100644 scripts/system_test_outcomes.py create mode 100644 scripts/tests/test_system_test_outcomes.py diff --git a/scripts/run-system-tests.py b/scripts/run-system-tests.py index 1c8b8cf8..8d55a4ba 100755 --- a/scripts/run-system-tests.py +++ b/scripts/run-system-tests.py @@ -52,6 +52,12 @@ validate_release_build_paths, verify_release_binary_identity, ) +from system_test_outcomes import ( + MAX_PYTEST_OUTCOME_REPORT_BYTES, + PYTEST_OUTCOME_REPORT_ENV, + read_pytest_outcome_report, + require_complete_pytest_outcomes, +) # ============================================================================ # Configuration @@ -1256,6 +1262,174 @@ def _build_base_image(mvm_version: str, *, rebuild: bool = False) -> str: # ============================================================================ +def _remote_pytest_report_path(vm_name: str) -> str: + """Return one fixed report receiver derived from the unique runner VM.""" + if re.fullmatch(r"[A-Za-z0-9_-]+", vm_name) is None: + raise RuntimeError(f"unsafe runner VM name for outcome report: {vm_name!r}") + return f"/tmp/mvmctl-pytest-outcomes-{vm_name}.json" + + +def _remote_pytest_command( + vm_name: str, + test_files: list[str], + report_path: str, +) -> str: + pytest_command = shlex.join( + [ + "python3", + "-m", + "pytest", + *(f"/{test_file}" for test_file in test_files), + "--tb=short", + "-q", + ] + ) + environment = " ".join( + ( + "MVM_ASSET_MIRROR=/mnt", + f"MVM_TEST_VM={shlex.quote(vm_name)}", + f"{PYTEST_OUTCOME_REPORT_ENV}={shlex.quote(report_path)}", + ) + ) + return f"cd / && {environment} {pytest_command}" + + +def _run_remote_pytest( + vm_name: str, + test_files: list[str], + *, + timeout: int, +) -> tuple[subprocess.CompletedProcess[str], str | None]: + """Run pytest in a runner VM and validate its bounded outcome report.""" + report_path = _remote_pytest_report_path(vm_name) + pytest_result: subprocess.CompletedProcess[str] | None = None + validation_failures: list[str] = [] + execution_error: Exception | None = None + try: + pytest_result = mvm( + "exec", + vm_name, + "--user", + "runner", + "--timeout", + "10", + "--", + _remote_pytest_command(vm_name, test_files, report_path), + check=False, + timeout=timeout, + ) + retrieve_result = mvm( + "exec", + vm_name, + "--user", + "runner", + "--timeout", + "10", + "--", + f"head -c {MAX_PYTEST_OUTCOME_REPORT_BYTES + 1} -- " + f"{shlex.quote(report_path)}", + check=False, + timeout=30, + ) + if retrieve_result.returncode != 0: + detail = (retrieve_result.stderr or "").strip() or "report read failed" + raise RuntimeError( + f"remote pytest outcome report is missing: {detail}" + ) + require_complete_pytest_outcomes( + retrieve_result.stdout.encode(), + process_returncode=pytest_result.returncode, + ) + except Exception as exc: + execution_error = exc + finally: + try: + mvm( + "exec", + vm_name, + "--user", + "runner", + "--timeout", + "10", + "--", + f"rm -f -- {shlex.quote(report_path)} && " + f"test ! -e {shlex.quote(report_path)}", + check=True, + timeout=30, + ) + except Exception as exc: + validation_failures.append( + f"remote outcome report cleanup failed: {exc}" + ) + + if pytest_result is None: + if execution_error is not None: + validation_failures.insert(0, str(execution_error)) + raise RuntimeError("; ".join(validation_failures)) + if execution_error is not None: + validation_failures.insert(0, str(execution_error)) + reason = "; ".join(validation_failures) or None + return pytest_result, reason + + +def _pytest_result_output( + pytest_result: subprocess.CompletedProcess[str], + validation_failure: str | None, +) -> str: + output = (pytest_result.stdout or "") + (pytest_result.stderr or "") + if validation_failure is not None: + output += ( + "\npytest outcome validation failed: " + f"{validation_failure}\n" + ) + return output + + +def _run_local_pytest( + domain: str, + test_files: list[str], + *, + timeout: int, +) -> tuple[subprocess.CompletedProcess[str], str | None]: + """Run host-direct pytest with a private report and checked cleanup.""" + report_directory = tempfile.TemporaryDirectory( + prefix=f"mvmctl-pytest-{domain}-" + ) + report_path = Path(report_directory.name) / "outcomes.json" + pytest_result: subprocess.CompletedProcess[str] | None = None + validation_failures: list[str] = [] + execution_error: Exception | None = None + try: + pytest_result = _run_pytest( + test_files, + timeout=timeout, + extra_env={PYTEST_OUTCOME_REPORT_ENV: str(report_path)}, + ) + payload = read_pytest_outcome_report(report_path) + require_complete_pytest_outcomes( + payload, + process_returncode=pytest_result.returncode, + ) + except Exception as exc: + execution_error = exc + finally: + try: + report_directory.cleanup() + except Exception as exc: + validation_failures.append( + f"local outcome report cleanup failed: {exc}" + ) + + if pytest_result is None: + if execution_error is not None: + validation_failures.insert(0, str(execution_error)) + raise RuntimeError("; ".join(validation_failures)) + if execution_error is not None: + validation_failures.insert(0, str(execution_error)) + reason = "; ".join(validation_failures) or None + return pytest_result, reason + + def run_tier1_domain( domain: str, test_files: list[str], mvm_version: str, push: bool = False ) -> dict[str, Any]: @@ -1287,21 +1461,16 @@ def run_tier1_domain( timeout=30, ) log(f" Running {domain} tests...") - pytest_result = mvm( - "exec", + pytest_result, validation_failure = _run_remote_pytest( vm_name, - "--user", - "runner", - "--timeout", - "10", - "--", - f"cd / && MVM_ASSET_MIRROR=/mnt MVM_TEST_VM={vm_name} " - f"python3 -m pytest {' '.join(f'/{f}' for f in test_files)} --tb=short -q", - check=False, + test_files, timeout=660, ) - result["passed"] = pytest_result.returncode == 0 - result["output"] = pytest_result.stdout + pytest_result.stderr + result["passed"] = validation_failure is None + result["output"] = _pytest_result_output( + pytest_result, + validation_failure, + ) except Exception as e: result["output"] = str(e) finally: @@ -1341,21 +1510,16 @@ def run_tier2_domain( timeout=30, ) log(f" Running {domain} tests...") - pytest_result = mvm( - "exec", + pytest_result, validation_failure = _run_remote_pytest( vm_name, - "--user", - "runner", - "--timeout", - "10", - "--", - f"cd / && MVM_ASSET_MIRROR=/mnt MVM_TEST_VM={vm_name} " - f"python3 -m pytest {' '.join(f'/{f}' for f in test_files)} --tb=short -q", - check=False, + test_files, timeout=960, ) - result["passed"] = pytest_result.returncode == 0 - result["output"] = pytest_result.stdout + pytest_result.stderr + result["passed"] = validation_failure is None + result["output"] = _pytest_result_output( + pytest_result, + validation_failure, + ) except Exception as e: result["output"] = str(e) finally: @@ -1376,9 +1540,16 @@ def run_tier3_domain( } log(f" Running {domain} tests on host (timeout={timeout}s)...") try: - pytest_result = _run_pytest(test_files, timeout=timeout) - result["passed"] = pytest_result.returncode == 0 - result["output"] = pytest_result.stdout + pytest_result.stderr + pytest_result, validation_failure = _run_local_pytest( + domain, + test_files, + timeout=timeout, + ) + result["passed"] = validation_failure is None + result["output"] = _pytest_result_output( + pytest_result, + validation_failure, + ) except Exception as e: result["output"] = str(e) return result diff --git a/scripts/system_test_outcomes.py b/scripts/system_test_outcomes.py new file mode 100644 index 00000000..8dc6ed28 --- /dev/null +++ b/scripts/system_test_outcomes.py @@ -0,0 +1,224 @@ +"""Strict pytest outcome evidence for the system-test release runner.""" + +from __future__ import annotations + +import json +import os +import stat +from dataclasses import dataclass +from pathlib import Path + + +PYTEST_OUTCOME_REPORT_ENV = "MVM_PYTEST_OUTCOME_REPORT" +PYTEST_OUTCOME_SCHEMA_VERSION = 1 +MAX_PYTEST_OUTCOME_REPORT_BYTES = 4096 +MAX_PYTEST_OUTCOME_COUNT = 100_000 +_COUNT_FIELDS = ( + "collected", + "passed", + "failed", + "errors", + "collection_errors", + "deselected", + "skipped", + "xfailed", + "xpassed", +) +_REPORT_FIELDS = { + "schema_version", + *_COUNT_FIELDS, + "exit_status", +} +_PYTEST_EXIT_STATUSES = {0, 1, 2, 3, 4, 5} + + +@dataclass(frozen=True) +class PytestOutcomeReport: + schema_version: int + collected: int + passed: int + failed: int + errors: int + collection_errors: int + deselected: int + skipped: int + xfailed: int + xpassed: int + exit_status: int + + +def read_pytest_outcome_report(report_path: Path) -> bytes: + """Read one local report through a pinned, bounded, non-symlink file.""" + open_flags = os.O_RDONLY | os.O_CLOEXEC + if hasattr(os, "O_NOFOLLOW"): + open_flags |= os.O_NOFOLLOW + try: + descriptor = os.open(report_path, open_flags) + except FileNotFoundError as exc: + raise RuntimeError( + f"pytest outcome report is missing: {report_path}" + ) from exc + except OSError as exc: + raise RuntimeError( + f"cannot open pytest outcome report {report_path}: {exc}" + ) from exc + try: + metadata = os.fstat(descriptor) + if not stat.S_ISREG(metadata.st_mode): + raise RuntimeError( + f"pytest outcome report is not a regular file: {report_path}" + ) + if metadata.st_size > MAX_PYTEST_OUTCOME_REPORT_BYTES: + raise RuntimeError( + "pytest outcome report exceeds " + f"{MAX_PYTEST_OUTCOME_REPORT_BYTES} bytes" + ) + chunks: list[bytes] = [] + remaining = MAX_PYTEST_OUTCOME_REPORT_BYTES + 1 + while remaining > 0: + chunk = os.read(descriptor, remaining) + if not chunk: + break + chunks.append(chunk) + remaining -= len(chunk) + payload = b"".join(chunks) + if len(payload) > MAX_PYTEST_OUTCOME_REPORT_BYTES: + raise RuntimeError( + "pytest outcome report exceeds " + f"{MAX_PYTEST_OUTCOME_REPORT_BYTES} bytes" + ) + return payload + finally: + os.close(descriptor) + + +def require_complete_pytest_outcomes( + payload: bytes, + *, + process_returncode: int, +) -> PytestOutcomeReport: + """Return exact all-pass evidence or reject the qualification result.""" + if len(payload) > MAX_PYTEST_OUTCOME_REPORT_BYTES: + raise RuntimeError( + "pytest outcome report exceeds " + f"{MAX_PYTEST_OUTCOME_REPORT_BYTES} bytes" + ) + def reject_duplicate_fields( + pairs: list[tuple[str, object]], + ) -> dict[str, object]: + values_by_name: dict[str, object] = {} + names_by_casefold: dict[str, str] = {} + for field, value in pairs: + if field in values_by_name: + raise RuntimeError( + f"pytest outcome report has duplicate field: {field}" + ) + folded = field.casefold() + if folded in names_by_casefold: + colliding = sorted((names_by_casefold[folded], field)) + raise RuntimeError( + "pytest outcome report has case-fold duplicate fields: " + f"{', '.join(colliding)}" + ) + values_by_name[field] = value + names_by_casefold[folded] = field + return values_by_name + + try: + values = json.loads(payload, object_pairs_hook=reject_duplicate_fields) + except (UnicodeDecodeError, json.JSONDecodeError) as exc: + raise RuntimeError(f"pytest outcome report contains malformed JSON: {exc}") from exc + if not isinstance(values, dict): + raise RuntimeError("pytest outcome report must be a top-level JSON object") + + provided_fields = set(values) + missing_fields = sorted(_REPORT_FIELDS - provided_fields) + if missing_fields: + raise RuntimeError( + "pytest outcome report missing fields: " + f"{', '.join(missing_fields)}" + ) + unknown_fields = sorted(provided_fields - _REPORT_FIELDS) + if unknown_fields: + raise RuntimeError( + "pytest outcome report has unknown fields: " + f"{', '.join(unknown_fields)}" + ) + + for field in ("schema_version", *_COUNT_FIELDS, "exit_status"): + if type(values[field]) is not int: + raise RuntimeError( + f"pytest outcome report field {field} must be an integer" + ) + if values["schema_version"] != PYTEST_OUTCOME_SCHEMA_VERSION: + raise RuntimeError( + "pytest outcome report schema version must be " + f"{PYTEST_OUTCOME_SCHEMA_VERSION}" + ) + for field in _COUNT_FIELDS: + value = values[field] + if value < 0: + raise RuntimeError( + f"pytest outcome report field {field} must not be negative" + ) + if value > MAX_PYTEST_OUTCOME_COUNT: + raise RuntimeError( + f"pytest outcome report field {field} exceeds " + f"{MAX_PYTEST_OUTCOME_COUNT}" + ) + if values["exit_status"] not in _PYTEST_EXIT_STATUSES: + raise RuntimeError( + "pytest outcome report has unknown pytest exit status: " + f"{values['exit_status']}" + ) + if type(process_returncode) is not int: + raise RuntimeError("pytest process return code must be an integer") + if process_returncode < 0: + raise RuntimeError("pytest process return code must not be negative") + + report = PytestOutcomeReport(**values) + classified = ( + report.passed + + report.failed + + report.errors + + report.skipped + + report.xfailed + + report.xpassed + ) + if classified != report.collected: + raise RuntimeError( + "pytest outcome counts do not equal collected: " + f"collected={report.collected}, outcomes={classified}" + ) + if report.exit_status != process_returncode: + raise RuntimeError( + "pytest outcome exit status does not match process return code: " + f"report={report.exit_status}, process={process_returncode}" + ) + + incomplete = [ + f"{field}={getattr(report, field)}" + for field in ( + "failed", + "errors", + "collection_errors", + "deselected", + "skipped", + "xfailed", + "xpassed", + ) + if getattr(report, field) != 0 + ] + if incomplete: + raise RuntimeError( + "pytest outcomes are incomplete: " + ", ".join(incomplete) + ) + if report.collected == 0: + raise RuntimeError("pytest collection is empty") + if report.exit_status != 0: + raise RuntimeError( + f"pytest exit status is not successful: {report.exit_status}" + ) + if report.passed != report.collected: + raise RuntimeError("pytest did not pass every collected test") + return report diff --git a/scripts/tests/test_run_system_tests.py b/scripts/tests/test_run_system_tests.py index d9da4ae0..4ded3a64 100644 --- a/scripts/tests/test_run_system_tests.py +++ b/scripts/tests/test_run_system_tests.py @@ -3,6 +3,7 @@ import argparse import ast import importlib.util +import json import os import shutil import subprocess @@ -85,6 +86,24 @@ def _write_system_test(repo_root: Path, relative_path: str) -> None: test_file.write_text("def test_registered():\n pass\n", encoding="utf-8") +def _complete_pytest_outcome_report(collected: int = 1) -> bytes: + return json.dumps( + { + "schema_version": 1, + "collected": collected, + "passed": collected, + "failed": 0, + "errors": 0, + "collection_errors": 0, + "deselected": 0, + "skipped": 0, + "xfailed": 0, + "xpassed": 0, + "exit_status": 0, + } + ).encode() + + def test_default_outer_controller_is_the_canonical_system_binary( monkeypatch: pytest.MonkeyPatch, ) -> None: @@ -818,6 +837,334 @@ def failed_stream( assert "stderr: (streamed; not captured)" in message +def test_tier3_domain_requires_complete_local_pytest_outcomes( + orchestrator: ModuleType, + monkeypatch: pytest.MonkeyPatch, +) -> None: + report_directories: list[Path] = [] + + def fake_pytest( + test_files: list[str], + *, + timeout: int, + extra_env: dict[str, str], + ) -> subprocess.CompletedProcess[str]: + assert test_files == ["tests/system/env/test_env.py"] + assert timeout == 600 + report_path = Path(extra_env[orchestrator.PYTEST_OUTCOME_REPORT_ENV]) + report_directories.append(report_path.parent) + report_path.write_bytes(_complete_pytest_outcome_report(collected=2)) + return subprocess.CompletedProcess( + ["pytest"], + 0, + stdout="pytest stdout\n", + stderr="pytest stderr\n", + ) + + monkeypatch.setattr(orchestrator, "_run_pytest", fake_pytest) + + result = orchestrator.run_tier3_domain( + "env", + ["tests/system/env/test_env.py"], + ) + + assert result == { + "domain": "env", + "tier": 3, + "passed": True, + "output": "pytest stdout\npytest stderr\n", + } + assert len(report_directories) == 1 + assert report_directories[0].name.startswith("mvmctl-pytest-env-") + assert not report_directories[0].exists() + + +@pytest.mark.parametrize( + ("report_payload", "expected_reason"), + [ + (None, "outcome report is missing"), + (b"{", "malformed JSON"), + ( + json.dumps( + { + "schema_version": 1, + "collected": 1, + "passed": 0, + "failed": 0, + "errors": 0, + "collection_errors": 0, + "deselected": 0, + "skipped": 1, + "xfailed": 0, + "xpassed": 0, + "exit_status": 0, + } + ).encode(), + "skipped=1", + ), + ], + ids=["missing", "corrupt", "skip"], +) +def test_tier3_domain_preserves_pytest_output_and_report_failure_reason( + orchestrator: ModuleType, + monkeypatch: pytest.MonkeyPatch, + report_payload: bytes | None, + expected_reason: str, +) -> None: + report_directories: list[Path] = [] + + def fake_pytest( + _test_files: list[str], + *, + timeout: int, + extra_env: dict[str, str], + ) -> subprocess.CompletedProcess[str]: + assert timeout == 600 + report_path = Path(extra_env[orchestrator.PYTEST_OUTCOME_REPORT_ENV]) + report_directories.append(report_path.parent) + if report_payload is not None: + report_path.write_bytes(report_payload) + return subprocess.CompletedProcess( + ["pytest"], + 0, + stdout="useful pytest stdout\n", + stderr="useful pytest stderr\n", + ) + + monkeypatch.setattr(orchestrator, "_run_pytest", fake_pytest) + + result = orchestrator.run_tier3_domain( + "env", + ["tests/system/env/test_env.py"], + ) + + assert result["passed"] is False + assert "useful pytest stdout" in result["output"] + assert "useful pytest stderr" in result["output"] + assert "pytest outcome validation failed" in result["output"] + assert expected_reason in result["output"] + assert len(report_directories) == 1 + assert not report_directories[0].exists() + + +def test_tier3_domain_preserves_pytest_output_when_report_cleanup_fails( + orchestrator: ModuleType, + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, +) -> None: + report_directory = tmp_path / "mvmctl-pytest-env-fixed" + report_directory.mkdir() + + class FailingTemporaryDirectory: + name = str(report_directory) + + def __init__(self, *, prefix: str) -> None: + assert prefix == "mvmctl-pytest-env-" + + def cleanup(self) -> None: + raise OSError("synthetic local cleanup refusal") + + def fake_pytest( + _test_files: list[str], + *, + timeout: int, + extra_env: dict[str, str], + ) -> subprocess.CompletedProcess[str]: + assert timeout == 600 + Path(extra_env[orchestrator.PYTEST_OUTCOME_REPORT_ENV]).write_bytes( + _complete_pytest_outcome_report() + ) + return subprocess.CompletedProcess( + ["pytest"], + 0, + stdout="useful pytest stdout\n", + stderr="useful pytest stderr\n", + ) + + monkeypatch.setattr( + orchestrator.tempfile, + "TemporaryDirectory", + FailingTemporaryDirectory, + ) + monkeypatch.setattr(orchestrator, "_run_pytest", fake_pytest) + + result = orchestrator.run_tier3_domain( + "env", + ["tests/system/env/test_env.py"], + ) + + assert result["passed"] is False + assert "useful pytest stdout" in result["output"] + assert "useful pytest stderr" in result["output"] + assert "local outcome report cleanup failed" in result["output"] + assert "synthetic local cleanup refusal" in result["output"] + + +@pytest.mark.parametrize( + ("runner_name", "provision_name", "tier", "pytest_timeout"), + [ + ("run_tier1_domain", "provision_t1", 1, 660), + ("run_tier2_domain", "provision_t2", 2, 960), + ], + ids=["tier1", "tier2"], +) +def test_remote_domains_require_retrieved_complete_pytest_outcomes( + orchestrator: ModuleType, + monkeypatch: pytest.MonkeyPatch, + runner_name: str, + provision_name: str, + tier: int, + pytest_timeout: int, +) -> None: + vm_name = f"t{tier}-cli-fixed" + events: list[str] = [] + calls: list[tuple[tuple[str, ...], dict[str, object]]] = [] + monkeypatch.setattr(orchestrator, "_unique_name", lambda _prefix: vm_name) + monkeypatch.setattr( + orchestrator, + provision_name, + lambda provisioned, version: events.append( + f"provision:{provisioned}:{version}" + ), + ) + monkeypatch.setattr( + orchestrator, + "destroy_vm", + lambda destroyed: events.append(f"destroy:{destroyed}"), + ) + + def fake_mvm(*args: str, **kwargs: object) -> subprocess.CompletedProcess[str]: + calls.append((args, kwargs)) + command = args[-1] + if "python3 -m pytest" in command: + events.append("pytest") + return subprocess.CompletedProcess( + args, + 0, + stdout="pytest stdout\n", + stderr="pytest stderr\n", + ) + if command.startswith("head -c "): + events.append("retrieve") + return subprocess.CompletedProcess( + args, + 0, + stdout=_complete_pytest_outcome_report().decode(), + stderr="", + ) + if command.startswith("rm -f -- "): + events.append("cleanup") + return subprocess.CompletedProcess(args, 0, stdout="", stderr="") + raise AssertionError(f"unexpected mvm call: {args}") + + monkeypatch.setattr(orchestrator, "mvm", fake_mvm) + + runner = getattr(orchestrator, runner_name) + result = runner( + "cli", + ["tests/system/cli/test_cli.py"], + "0.3.0", + ) + + assert result == { + "domain": "cli", + "tier": tier, + "passed": True, + "output": "pytest stdout\npytest stderr\n", + } + report_path = f"/tmp/mvmctl-pytest-outcomes-{vm_name}.json" + pytest_args, pytest_kwargs = calls[0] + assert pytest_args[:2] == ("exec", vm_name) + assert f"MVM_PYTEST_OUTCOME_REPORT={report_path}" in pytest_args[-1] + assert "/tests/system/cli/test_cli.py" in pytest_args[-1] + assert pytest_kwargs["timeout"] == pytest_timeout + retrieve_args, retrieve_kwargs = calls[1] + assert retrieve_args[-1] == f"head -c 4097 -- {report_path}" + assert retrieve_kwargs["check"] is False + cleanup_args, cleanup_kwargs = calls[2] + assert cleanup_args[-1] == ( + f"rm -f -- {report_path} && test ! -e {report_path}" + ) + assert cleanup_kwargs["check"] is True + assert events == [ + f"provision:{vm_name}:0.3.0", + "pytest", + "retrieve", + "cleanup", + f"destroy:{vm_name}", + ] + + +@pytest.mark.parametrize( + ("report_payload", "retrieve_rc", "cleanup_fails", "expected_reason"), + [ + ("", 1, False, "outcome report is missing"), + ("{", 0, False, "malformed JSON"), + ( + _complete_pytest_outcome_report().decode(), + 0, + True, + "remote outcome report cleanup failed", + ), + ], + ids=["missing", "corrupt", "cleanup"], +) +def test_remote_domain_preserves_pytest_output_and_checked_cleanup_failure( + orchestrator: ModuleType, + monkeypatch: pytest.MonkeyPatch, + report_payload: str, + retrieve_rc: int, + cleanup_fails: bool, + expected_reason: str, +) -> None: + vm_name = "t1-cli-fixed" + destroyed: list[str] = [] + monkeypatch.setattr(orchestrator, "_unique_name", lambda _prefix: vm_name) + monkeypatch.setattr( + orchestrator, + "provision_t1", + lambda _vm_name, _version: None, + ) + monkeypatch.setattr(orchestrator, "destroy_vm", destroyed.append) + + def fake_mvm(*args: str, **_kwargs: object) -> subprocess.CompletedProcess[str]: + command = args[-1] + if "python3 -m pytest" in command: + return subprocess.CompletedProcess( + args, + 0, + stdout="useful pytest stdout\n", + stderr="useful pytest stderr\n", + ) + if command.startswith("head -c "): + return subprocess.CompletedProcess( + args, + retrieve_rc, + stdout=report_payload, + stderr=None if retrieve_rc else "", + ) + if command.startswith("rm -f -- "): + if cleanup_fails: + raise RuntimeError("synthetic cleanup refusal") + return subprocess.CompletedProcess(args, 0, stdout="", stderr="") + raise AssertionError(f"unexpected mvm call: {args}") + + monkeypatch.setattr(orchestrator, "mvm", fake_mvm) + + result = orchestrator.run_tier1_domain( + "cli", + ["tests/system/cli/test_cli.py"], + "0.3.0", + ) + + assert result["passed"] is False + assert "useful pytest stdout" in result["output"] + assert "useful pytest stderr" in result["output"] + assert "pytest outcome validation failed" in result["output"] + assert expected_reason in result["output"] + assert destroyed == [vm_name] + + def test_builder_assets_are_pulled_in_separate_sequential_execs( orchestrator: ModuleType, monkeypatch: pytest.MonkeyPatch, diff --git a/scripts/tests/test_system_test_outcomes.py b/scripts/tests/test_system_test_outcomes.py new file mode 100644 index 00000000..c048ecfb --- /dev/null +++ b/scripts/tests/test_system_test_outcomes.py @@ -0,0 +1,495 @@ +from __future__ import annotations + +import importlib.util +import json +import os +import subprocess +import sys +from pathlib import Path +from types import ModuleType + +import pytest + + +REPORT_ENV = "MVM_PYTEST_OUTCOME_REPORT" + + +def _valid_outcome_values(**changes: object) -> dict[str, object]: + values: dict[str, object] = { + "schema_version": 1, + "collected": 1, + "passed": 1, + "failed": 0, + "errors": 0, + "collection_errors": 0, + "deselected": 0, + "skipped": 0, + "xfailed": 0, + "xpassed": 0, + "exit_status": 0, + } + values.update(changes) + return values + + +@pytest.fixture +def outcomes() -> ModuleType: + script = Path(__file__).parents[1] / "system_test_outcomes.py" + module_name = "mvmctl_system_test_outcomes" + spec = importlib.util.spec_from_file_location(module_name, script) + assert spec is not None and spec.loader is not None + module = importlib.util.module_from_spec(spec) + sys.modules[module_name] = module + spec.loader.exec_module(module) + return module + + +def _run_outcome_case( + tmp_path: Path, + source: str, + additional_source: str | None = None, + *, + enable_report: bool = True, + extra_pytest_args: list[str] | None = None, +) -> tuple[subprocess.CompletedProcess[str], Path]: + test_file = tmp_path / "test_outcome_case.py" + test_file.write_text(source, encoding="utf-8") + test_files = [test_file] + if additional_source is not None: + additional_file = tmp_path / "test_outcome_additional.py" + additional_file.write_text(additional_source, encoding="utf-8") + test_files.append(additional_file) + report_path = tmp_path / "outcomes.json" + system_tests = Path(__file__).parents[2] / "tests" / "system" + env: dict[str, str] = { + **os.environ, + "PYTHONPATH": str(system_tests), + "PYTEST_DISABLE_PLUGIN_AUTOLOAD": "1", + } + if enable_report: + env[REPORT_ENV] = str(report_path) + else: + env.pop(REPORT_ENV, None) + result = subprocess.run( + [ + sys.executable, + "-m", + "pytest", + "-p", + "conftest", + "--confcutdir", + str(tmp_path), + "-q", + *(extra_pytest_args or []), + *(str(path) for path in test_files), + ], + cwd=tmp_path, + capture_output=True, + text=True, + timeout=30, + env=env, + ) + return result, report_path + + +def test_pytest_hook_writes_exact_success_outcome_report(tmp_path: Path) -> None: + result, report_path = _run_outcome_case( + tmp_path, + "def test_success():\n assert True\n", + ) + + assert result.returncode == 0, result.stdout + result.stderr + assert json.loads(report_path.read_text(encoding="utf-8")) == { + "schema_version": 1, + "collected": 1, + "passed": 1, + "failed": 0, + "errors": 0, + "collection_errors": 0, + "deselected": 0, + "skipped": 0, + "xfailed": 0, + "xpassed": 0, + "exit_status": 0, + } + + +def test_pytest_hook_records_deselected_items_separately( + tmp_path: Path, +) -> None: + result, report_path = _run_outcome_case( + tmp_path, + "def test_selected():\n pass\n" + "def test_filtered_out():\n pass\n", + extra_pytest_args=["-k", "selected"], + ) + + assert result.returncode == 0, result.stdout + result.stderr + assert json.loads(report_path.read_text(encoding="utf-8")) == { + "schema_version": 1, + "collected": 1, + "passed": 1, + "failed": 0, + "errors": 0, + "collection_errors": 0, + "deselected": 1, + "skipped": 0, + "xfailed": 0, + "xpassed": 0, + "exit_status": 0, + } + + +def test_pytest_hook_is_inactive_without_fixed_report_environment( + tmp_path: Path, +) -> None: + result, report_path = _run_outcome_case( + tmp_path, + "def test_success():\n pass\n", + enable_report=False, + ) + + assert result.returncode == 0, result.stdout + result.stderr + assert not report_path.exists() + + +@pytest.mark.parametrize( + ("source", "exit_status", "collected", "outcome", "collection_errors"), + [ + ("VALUE = 1\n", 5, 0, None, 0), + ( + "import pytest\n" + "@pytest.mark.skip(reason='synthetic')\n" + "def test_skip():\n pass\n", + 0, + 1, + "skipped", + 0, + ), + ( + "import pytest\n" + "@pytest.mark.xfail(reason='synthetic')\n" + "def test_xfail():\n assert False\n", + 0, + 1, + "xfailed", + 0, + ), + ( + "import pytest\n" + "@pytest.mark.xfail(reason='synthetic')\n" + "def test_xpass():\n pass\n", + 0, + 1, + "xpassed", + 0, + ), + ("def test_failure():\n assert False\n", 1, 1, "failed", 0), + ("def test_broken(:\n pass\n", 2, 0, None, 1), + ( + "import pytest\n" + "@pytest.fixture\n" + "def broken():\n raise RuntimeError('setup')\n" + "def test_setup_error(broken):\n pass\n", + 1, + 1, + "errors", + 0, + ), + ( + "import pytest\n" + "@pytest.fixture\n" + "def broken():\n" + " yield\n" + " raise RuntimeError('teardown')\n" + "def test_teardown_error(broken):\n pass\n", + 1, + 1, + "errors", + 0, + ), + ], + ids=[ + "zero-collection", + "skip", + "xfail", + "xpass", + "failure", + "collection-error", + "setup-error", + "teardown-error", + ], +) +def test_pytest_hook_classifies_exact_outcomes( + tmp_path: Path, + source: str, + exit_status: int, + collected: int, + outcome: str | None, + collection_errors: int, +) -> None: + result, report_path = _run_outcome_case(tmp_path, source) + + assert result.returncode == exit_status, result.stdout + result.stderr + report = json.loads(report_path.read_text(encoding="utf-8")) + expected_counts = { + "passed": 0, + "failed": 0, + "errors": 0, + "collection_errors": collection_errors, + "deselected": 0, + "skipped": 0, + "xfailed": 0, + "xpassed": 0, + } + if outcome is not None: + expected_counts[outcome] = 1 + assert report == { + "schema_version": 1, + "collected": collected, + **expected_counts, + "exit_status": exit_status, + } + + +def test_pytest_hook_separates_collection_and_item_outcomes( + tmp_path: Path, +) -> None: + result, report_path = _run_outcome_case( + tmp_path, + "def test_success():\n pass\n", + additional_source="def test_broken(:\n pass\n", + ) + + assert result.returncode == 2, result.stdout + result.stderr + assert json.loads(report_path.read_text(encoding="utf-8")) == { + "schema_version": 1, + "collected": 1, + "passed": 0, + "failed": 0, + "errors": 1, + "collection_errors": 1, + "deselected": 0, + "skipped": 0, + "xfailed": 0, + "xpassed": 0, + "exit_status": 2, + } + + +def test_complete_outcome_validator_accepts_only_all_passed( + outcomes: ModuleType, +) -> None: + payload = json.dumps( + { + "schema_version": 1, + "collected": 3, + "passed": 3, + "failed": 0, + "errors": 0, + "collection_errors": 0, + "deselected": 0, + "skipped": 0, + "xfailed": 0, + "xpassed": 0, + "exit_status": 0, + } + ).encode() + + report = outcomes.require_complete_pytest_outcomes( + payload, + process_returncode=0, + ) + + assert report.collected == 3 + assert report.passed == 3 + + +@pytest.mark.parametrize( + ("kind", "message"), + [ + ("directory", "not a regular file"), + ("symlink", "cannot open"), + ("oversized", "exceeds 4096 bytes"), + ], +) +def test_local_outcome_reader_rejects_untrusted_or_oversized_files( + outcomes: ModuleType, + tmp_path: Path, + kind: str, + message: str, +) -> None: + report_path = tmp_path / "outcomes.json" + if kind == "directory": + report_path.mkdir() + elif kind == "symlink": + target = tmp_path / "target.json" + target.write_text( + json.dumps(_valid_outcome_values()), + encoding="utf-8", + ) + report_path.symlink_to(target) + else: + report_path.write_bytes(b" " * 4097) + + with pytest.raises(RuntimeError, match=message): + outcomes.read_pytest_outcome_report(report_path) + + +@pytest.mark.parametrize( + ("payload", "process_returncode", "message"), + [ + (b"{", 0, "malformed JSON"), + (json.dumps([]).encode(), 0, "top-level JSON object"), + ( + json.dumps( + { + key: value + for key, value in _valid_outcome_values().items() + if key != "passed" + } + ).encode(), + 0, + "missing fields: passed", + ), + ( + json.dumps(_valid_outcome_values(surprise=0)).encode(), + 0, + "unknown fields: surprise", + ), + ( + json.dumps(_valid_outcome_values()) + .replace('"passed": 1', '"passed": 1, "passed": 0') + .encode(), + 0, + "duplicate field: passed", + ), + ( + json.dumps(_valid_outcome_values()) + .replace('"passed": 1', '"passed": 1, "Passed": 0') + .encode(), + 0, + "case-fold duplicate fields: Passed, passed", + ), + ( + json.dumps(_valid_outcome_values(schema_version=2)).encode(), + 0, + "schema version", + ), + ( + json.dumps(_valid_outcome_values(passed=True)).encode(), + 0, + "passed must be an integer", + ), + ( + json.dumps(_valid_outcome_values(skipped=-1)).encode(), + 0, + "skipped must not be negative", + ), + ( + json.dumps( + _valid_outcome_values(collected=100_001, passed=100_001) + ).encode(), + 0, + "collected exceeds", + ), + ( + json.dumps(_valid_outcome_values(exit_status=6)).encode(), + 6, + "unknown pytest exit status", + ), + ( + json.dumps(_valid_outcome_values(collected=2)).encode(), + 0, + "outcome counts do not equal collected", + ), + ( + json.dumps(_valid_outcome_values()).encode(), + 1, + "does not match process return code", + ), + ( + json.dumps(_valid_outcome_values()).encode() + b" " * 4096, + 0, + "exceeds 4096 bytes", + ), + ], + ids=[ + "malformed", + "non-object", + "missing-field", + "unknown-field", + "duplicate-field", + "case-fold-duplicate-field", + "schema", + "wrong-type", + "negative", + "count-bound", + "unknown-exit", + "inconsistent-counts", + "exit-mismatch", + "oversized", + ], +) +def test_outcome_validator_rejects_malformed_or_inconsistent_reports( + outcomes: ModuleType, + payload: bytes, + process_returncode: int, + message: str, +) -> None: + with pytest.raises(RuntimeError, match=message): + outcomes.require_complete_pytest_outcomes( + payload, + process_returncode=process_returncode, + ) + + +@pytest.mark.parametrize( + ("changes", "process_returncode", "message"), + [ + ( + {"collected": 0, "passed": 0, "exit_status": 5}, + 5, + "collection is empty", + ), + ({"passed": 0, "skipped": 1}, 0, "skipped=1"), + ({"passed": 0, "xfailed": 1}, 0, "xfailed=1"), + ({"passed": 0, "xpassed": 1}, 0, "xpassed=1"), + ({"deselected": 1}, 0, "deselected=1"), + ({"passed": 0, "failed": 1, "exit_status": 1}, 1, "failed=1"), + ({"passed": 0, "errors": 1, "exit_status": 1}, 1, "errors=1"), + ( + { + "collected": 0, + "passed": 0, + "collection_errors": 1, + "exit_status": 2, + }, + 2, + "collection_errors=1", + ), + ], + ids=[ + "zero", + "skip", + "xfail", + "xpass", + "deselected", + "failure", + "error", + "collection-error", + ], +) +def test_outcome_validator_rejects_incomplete_pytest_outcomes( + outcomes: ModuleType, + changes: dict[str, object], + process_returncode: int, + message: str, +) -> None: + payload = json.dumps(_valid_outcome_values(**changes)).encode() + + with pytest.raises(RuntimeError, match=message): + outcomes.require_complete_pytest_outcomes( + payload, + process_returncode=process_returncode, + ) diff --git a/tests/system/conftest.py b/tests/system/conftest.py index 0ac3605c..29191c3a 100644 --- a/tests/system/conftest.py +++ b/tests/system/conftest.py @@ -16,12 +16,117 @@ import subprocess import time import uuid +from pathlib import Path from typing import Any, Generator import pytest SYSTEM_MVM_BINARY = "/usr/local/bin/mvm" +PYTEST_OUTCOME_REPORT_ENV = "MVM_PYTEST_OUTCOME_REPORT" +PYTEST_OUTCOME_SCHEMA_VERSION = 1 +_OUTCOME_REPORTS: dict[str, dict[str, Any]] = {} +_OUTCOME_COLLECTION_ERRORS = 0 +_OUTCOME_DESELECTED = 0 + + +def _outcome_reporting_enabled() -> bool: + return bool(os.environ.get(PYTEST_OUTCOME_REPORT_ENV)) + + +def pytest_sessionstart(session: pytest.Session) -> None: + """Reset outcome state only for orchestrated qualification runs.""" + del session + global _OUTCOME_COLLECTION_ERRORS, _OUTCOME_DESELECTED + if _outcome_reporting_enabled(): + _OUTCOME_REPORTS.clear() + _OUTCOME_COLLECTION_ERRORS = 0 + _OUTCOME_DESELECTED = 0 + + +def pytest_collectreport(report: pytest.CollectReport) -> None: + """Count collection failures separately from collected test items.""" + global _OUTCOME_COLLECTION_ERRORS + if _outcome_reporting_enabled() and report.failed: + _OUTCOME_COLLECTION_ERRORS += 1 + + +def pytest_deselected(items: list[pytest.Item]) -> None: + """Record tests removed from the selected qualification matrix.""" + global _OUTCOME_DESELECTED + if _outcome_reporting_enabled(): + _OUTCOME_DESELECTED += len(items) + + +def pytest_runtest_logreport(report: pytest.TestReport) -> None: + """Retain each test phase so one final outcome can be classified.""" + if _outcome_reporting_enabled(): + _OUTCOME_REPORTS.setdefault(report.nodeid, {})[report.when] = report + + +def _classify_test_outcome(nodeid: str) -> str: + phase_reports = _OUTCOME_REPORTS.get(nodeid, {}) + setup_report = phase_reports.get("setup") + call_report = phase_reports.get("call") + teardown_report = phase_reports.get("teardown") + + for phase_report in (setup_report, teardown_report): + if phase_report is not None and phase_report.failed: + return "errors" + for phase_report in (setup_report, call_report, teardown_report): + if phase_report is not None and hasattr(phase_report, "wasxfail"): + return "xfailed" if phase_report.skipped else "xpassed" + for phase_report in (setup_report, call_report, teardown_report): + if phase_report is not None and phase_report.skipped: + return "skipped" + if call_report is None: + return "errors" + if call_report.failed: + return "failed" + if call_report.passed: + return "passed" + return "errors" + + +def pytest_sessionfinish(session: pytest.Session, exitstatus: int) -> None: + """Write the bounded machine-readable qualification report.""" + configured_path = os.environ.get(PYTEST_OUTCOME_REPORT_ENV) + if not configured_path: + return + + outcomes = { + "passed": 0, + "failed": 0, + "errors": 0, + "collection_errors": _OUTCOME_COLLECTION_ERRORS, + "deselected": _OUTCOME_DESELECTED, + "skipped": 0, + "xfailed": 0, + "xpassed": 0, + } + for item in session.items: + outcomes[_classify_test_outcome(item.nodeid)] += 1 + + report = { + "schema_version": PYTEST_OUTCOME_SCHEMA_VERSION, + "collected": len(session.items), + **outcomes, + "exit_status": int(exitstatus), + } + report_path = Path(configured_path) + if not report_path.is_absolute(): + raise pytest.UsageError( + f"{PYTEST_OUTCOME_REPORT_ENV} must be an absolute path" + ) + encoded = json.dumps(report, sort_keys=True, separators=(",", ":")) + temporary_path = report_path.with_name( + f".{report_path.name}.{os.getpid()}.tmp" + ) + with temporary_path.open("x", encoding="utf-8") as report_file: + report_file.write(encoded) + report_file.flush() + os.fsync(report_file.fileno()) + os.replace(temporary_path, report_path) # ============================================================================ From 02bfcb827bde8699da414b4d41ef599c8c94cbde Mon Sep 17 00:00:00 2001 From: AlanD20 Date: Sun, 23 Aug 2026 16:12:36 +0200 Subject: [PATCH 0031/1457] docs: require complete control response framing --- .../0016-root-owned-single-binary-privileged-dispatch.md | 9 +++++---- tasks/plan.md | 5 +++-- tasks/todo.md | 4 ++-- 3 files changed, 10 insertions(+), 8 deletions(-) diff --git a/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md b/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md index 75c1343f..24920336 100644 --- a/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md +++ b/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md @@ -110,10 +110,11 @@ Each response starts with fixed `MVMRES01` magic and a network-order 32-bit JSON strict schema-version-1 JSON. It contains the matching action, a closed `success` or `error` status, and exactly one typed result or error. Error envelopes carry code, stable string class, message, operation, entity, and normalized details; they never serialize wrapped causes, stacks, stderr, or arbitrary Go values. Wire details permit only bounded booleans, -strings, signed integers, and string arrays. Unsupported details are omitted visibly. An error response delivered after -all effects and cleanup exits successfully at the process layer. A start failure has known no-effect status; any started -process with a missing, malformed, truncated, mismatched, or oversized response returns `CodeProcessError` with -`process_started: true` and `outcome_unknown: true`. +strings, signed integers, and string arrays. Unsupported details are omitted visibly. Root half-closes the control +socket's write side after the complete frame, so the client can validate response EOF independently of process exit. An +error response delivered after all effects and cleanup exits successfully at the process layer. A start failure has +known no-effect status; any started process with a missing, malformed, truncated, mismatched, oversized, or non-EOF +response returns `CodeProcessError` with `process_started: true` and `outcome_unknown: true`. ### Authorization roles diff --git a/tasks/plan.md b/tasks/plan.md index 2e1c60f1..e59466f5 100644 --- a/tasks/plan.md +++ b/tasks/plan.md @@ -87,8 +87,9 @@ half-closes after the declared bytes. Root requires exact length and EOF before zero-payload effects, but may respond to pre-effect authentication/header failure without draining a claimed payload. Responses use `MVMRES01`, a 32-bit length capped at 64 KiB, and one strict matching-action success/result or error/`DomainError` envelope. A delivered envelope is the final effect record even if process reaping later reports an -anomaly; no domain effect or fallible outcome check is allowed after the response. Missing, malformed, truncated, -mismatched, or oversized responses from a started process report +anomaly; root half-closes its socket write side after the complete frame so response EOF does not depend on process exit. +No domain effect or fallible outcome check is allowed after the response. Missing, malformed, truncated, mismatched, +oversized, or non-EOF responses from a started process report `process_started: true` and `outcome_unknown: true`; stderr and exit text are never parsed as authority. ### VM authority interface diff --git a/tasks/todo.md b/tasks/todo.md index 30de7b0d..ac660e5e 100644 --- a/tasks/todo.md +++ b/tasks/todo.md @@ -65,8 +65,8 @@ The final marker-only sudoers assertion is deferred until Task 14; current sudoe - [x] Caller identity, active `mvm` group, environment, and system executable are verified before dispatch. - [ ] Add a strict versioned request/response codec with 64 KiB frames, duplicate/unknown/depth rejection, matching actions, typed results, and bounded `DomainError` code/class/entity/partial-state detail preservation. -- [ ] Add the fd-0 Unix socketpair control runner with concurrent upload/response, exact EOF, CLOEXEC, cancellation, - timeout, response/stderr bounds, early-rejection handling, and `FakeRunner` support. +- [ ] Add the fd-0 Unix socketpair control runner with concurrent upload/response, request and response half-close/EOF, + CLOEXEC, cancellation, timeout, bounds, early-rejection handling, and `FakeRunner` support. - [ ] Return `process_started: true` and `outcome_unknown: true` for every started process without one valid final response; stdout, stderr, and generic subprocess text are never action authority. - [ ] Keep the action switch closed until each capability has its own typed handler and abuse tests. From 182a400176d566bcf7aeb93c7fc4bb672905601b Mon Sep 17 00:00:00 2001 From: AlanD20 Date: Sun, 23 Aug 2026 16:13:53 +0200 Subject: [PATCH 0032/1457] docs: require exact pytest release evidence --- CHANGELOG.md | 4 ++++ docs/RC_QA.md | 11 ++++++----- docs/development/HOW_TO_RUN_SYSTEM_TESTS.md | 16 ++++++++-------- docs/system-test-architecture.md | 16 ++++++++++++---- tasks/todo.md | 5 +++-- 5 files changed, 33 insertions(+), 19 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index dd7b22ed..acb2bf7e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -80,6 +80,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - The runner now validates its complete registry before any work: domain names and test files are unique, registrations are non-empty canonical regular files under `tests/system/`, and every discovered system-test file is registered exactly once. The missing Tier 2 `exec` registration is included and the duplicate Tier 3 `fresh_env` alias is gone. +- Every T1/T2/T3 domain now emits a strict bounded pytest outcome report. A domain passes only when collection is + non-empty, the process/report exit statuses match, every selected item passed, and failures, errors, collection errors, + deselections, skips, XFAIL, and XPASS are all zero. Missing, malformed, duplicate-field, oversized reports and report + cleanup failures fail the domain while preserving pytest output and the validation reason. #### Deterministic initialization - Added `mvm init --binary-version ` to request an exact Firecracker/Jailer pair when no local pair exists. diff --git a/docs/RC_QA.md b/docs/RC_QA.md index 5914e3d5..e4b6447f 100644 --- a/docs/RC_QA.md +++ b/docs/RC_QA.md @@ -195,9 +195,10 @@ MVM_BINARY=/usr/local/bin/mvm MVM_CANDIDATE_BINARY=./dist/mvm \ 2>&1 | tee "$evidence_dir/system-tests.log" ``` -Until the orchestrator records skip/xfail counts in its final summary, QA must inspect every archived per-domain pytest -summary and record that it contains zero skipped, xfailed, or xpassed cases. A domain-level `PASS` alone is not sufficient -release evidence. +The orchestrator accepts a domain-level `PASS` only after validating a strict bounded pytest outcome report: collection +is non-empty, process/report exit status matches, every selected item passed, and failures, errors, collection errors, +deselections, skips, XFAIL, and XPASS are all zero. Missing, malformed, duplicate-field, oversized, inconsistent, or +uncleared reports fail the domain and preserve pytest output plus the validation reason in the archived log. --- @@ -249,7 +250,7 @@ A regression is **any** of the following: - [ ] `go test ./... -count=1 -coverprofile=coverage.out -covermode=atomic` — all pass - [ ] `./scripts/build.sh release --version X.Y.Z --output dist/mvm` — exact candidate built - [ ] Dedicated clean qualification host/outer VM — candidate installed at exact `/usr/local/bin/mvm` -- [ ] System tests — every required T1/T2/T3 case passes; archived summaries show zero skips/xfails/xpasses +- [ ] System tests — every required T1/T2/T3 domain passes strict machine outcome validation - [ ] `/usr/local/bin/mvm --version` — returns exact `mvm X.Y.Z` - [ ] `/usr/local/bin/mvm --help` — all commands present @@ -257,7 +258,7 @@ A regression is **any** of the following: - [ ] Build log archived - [ ] Tidy, format, line-length, generate, vet, and Go-test logs archived - [ ] System test log archived -- [ ] Per-domain skip/xfail audit archived +- [ ] System-test log archives the strict per-domain outcome gate and every failure reason - [ ] Binary checksum archived - [ ] Benchmark results within thresholds diff --git a/docs/development/HOW_TO_RUN_SYSTEM_TESTS.md b/docs/development/HOW_TO_RUN_SYSTEM_TESTS.md index b0a63778..a5c3b367 100644 --- a/docs/development/HOW_TO_RUN_SYSTEM_TESTS.md +++ b/docs/development/HOW_TO_RUN_SYSTEM_TESTS.md @@ -373,7 +373,7 @@ duplicate alias cannot produce release evidence. The runner prints a summary at the end: ``` - 12 passed 0 failed 0 skipped (3m12s) + RESULTS: 12 passed, 0 failed, 12 total ``` Results are printed to stdout at the end of the run: @@ -390,9 +390,8 @@ The runner prints a per-domain summary with PASS/FAIL status for each domain. | Status | Meaning | Release Gate | |--------|---------|--------------| -| **PASS** | All tests in file passed | ✓ | -| **FAIL** | At least one test failed | ✗ — must fix before release | -| **SKIP** | All tests skipped (prerequisite missing) | ⚠ — investigate why | +| **PASS** | The bounded report proves non-empty collection and every selected item passed | ✓ | +| **FAIL** | A test/outcome failed or exact machine-readable evidence was unavailable | ✗ — must fix before release | ### 5.1a Release Candidate (RC) Zero-Tolerance Policy @@ -402,10 +401,11 @@ pass. A skip is treated as a release blocker — it means either: - The test is testing a feature that doesn't exist in the Go CLI (remove the test or add the feature) - The test's skip condition is too broad (tighten the condition) -Before signing off an RC, every test file must produce `X passed, 0 failed, 0 skipped`. -The current orchestrator's final domain summary does not independently count -pytest skips/xfails/xpasses. Until it does, QA must inspect and archive every -per-domain pytest summary; `[PASS]` at the domain level alone is insufficient. +Before signing off an RC, each domain must produce one strict outcome report with non-empty collection, matching pytest +and report exit status, every selected item passed, and zero failures, errors, collection errors, deselections, skips, +XFAIL, or XPASS. The orchestrator enforces this for T1/T2 inside runner VMs and T3 in a private local report. Missing, +malformed, duplicate-field, oversized, inconsistent, or uncleared reports turn the domain into `FAIL` and retain pytest +output plus the validation reason. A printed `[PASS]` therefore includes this machine-evidence gate. ### 5.2 What a Skip Means diff --git a/docs/system-test-architecture.md b/docs/system-test-architecture.md index 4decc022..18edd584 100644 --- a/docs/system-test-architecture.md +++ b/docs/system-test-architecture.md @@ -58,7 +58,8 @@ Host (your machine) |----------|-----------| | **Custom base image** | A single `mvm-test-runner:` image is built during `--prepare`. The release candidate is installed through `host install-system`; the image also contains the system tests and OS dependencies. Built once, cached by candidate version. | | **Tests run inside the VM** | The orchestrator runs `mvm exec -- python3 -m pytest ...` inside the test VM. Tests invoke the exact installed `/usr/local/bin/mvm` via `subprocess.run` — no PATH substitution or vsock proxy. | -| **`conftest.py` is simple** | `_run_mvm` is a thin wrapper around `subprocess.run(["/usr/local/bin/mvm", *args])`. The `runner_vm` fixture gets the VM name from `MVM_TEST_VM` (set by the orchestrator). | +| **Machine-checked outcomes** | Each T1/T2/T3 pytest process writes one strict, bounded, schema-versioned report. The orchestrator requires non-empty collection, matching exit status, every selected item passed, and zero failure/error/collection-error/deselection/skip/XFAIL/XPASS counts. | +| **`conftest.py` is runner-local** | `_run_mvm` stays a thin wrapper around exact `/usr/local/bin/mvm`; fixtures operate inside the runner, and an opt-in hook emits the bounded outcome report requested by the orchestrator. | | **Shared RO volume on all tiers** | The `asset-mirror` volume is attached to every VM (T1 and T2). Mounting is cheap — `--shareable --read-only` means no per-VM state. | | **`--push` flag** | Overrides baked-in tests by copying `tests/system/` fresh into the VM. Useful during test development without rebuilding the base image. | | **Controller/candidate qualification** | T1/T2 use a separate installed outer controller and candidate installed only inside disposable runners. Tier 3 is host-direct; release mode therefore requires exact content/version identity between the candidate and root-owned `/usr/local/bin/mvm` before resource preparation. | @@ -406,13 +407,20 @@ python3 scripts/run-system-tests.py --tier 2,1 d. T1 + T2 in parallel (ThreadPoolExecutor): Create VM from the candidate-qualified base image → mount volume → pinned mvm init → [--push: mvm cp tests/system] → - mvm exec -- python3 -m pytest ... → + mvm exec -- python3 -m pytest ... → retrieve/strictly validate bounded outcome report → + checked report cleanup → destroy VM e. T3 sequentially on the dedicated clean qualification host: - pytest tests/system// + pytest tests/system// → strictly validate private local outcome report → checked cleanup 5. Print summary: X passed, Y failed, Z total ``` +The pytest hook is inactive outside orchestrated runs. When enabled, it records selected items separately from collection +errors and deselections and classifies setup/teardown failures as errors. T1/T2 retrieve at most 4,097 bytes from the +fixed per-runner report path before checked deletion. T3 uses a private temporary directory and no-follow bounded local +read. A missing, malformed, duplicate/case-colliding, unknown-field, oversized, inconsistent, or uncleared report fails +the domain even if pytest returned zero; the failure output retains pytest stdout/stderr and the exact validation reason. + See [RC_QA.md](RC_QA.md) for the full flags reference, setup workflow, and execution examples used during release qualification. @@ -422,7 +430,7 @@ execution examples used during release qualification. ``` tests/system/ -├── conftest.py ← _run_mvm helper (direct subprocess, no vsock proxy) +├── conftest.py ← _run_mvm helper plus opt-in exact outcome report ├── pytest.ini ← Markers ├── __init__.py │ diff --git a/tasks/todo.md b/tasks/todo.md index ac660e5e..a07c40a0 100644 --- a/tasks/todo.md +++ b/tasks/todo.md @@ -352,8 +352,9 @@ under `internal/service/jailer/`. No handler, transport, CLI, API, core-domain, - [ ] Make cleanup and post-domain absence checks part of the domain result; preserve both primary and cleanup failures, and never advance tiers while a timed-out worker can still mutate state. - [x] Reject unknown, duplicate, mixed, zero-domain, and empty-file domain selections before probes or mutation. -- [ ] Parse pytest outcomes and require non-empty collection with zero skips, XFAIL, or XPASS; omitted domains fail - instead of producing release evidence. +- [x] Parse strict bounded pytest outcomes for T1/T2/T3 and require non-empty collection, matching exit status, every + selected item passed, and zero errors, collection errors, deselections, skips, XFAIL, or XPASS; missing/malformed + reports and cleanup failures fail the domain while preserving pytest output and the reason. - [x] Ensure the full matrix selects every required domain exactly once, including `exec`, and does not run `test_vm_fresh_env.py` twice under aliases. - [ ] Make mirror validation reject pull errors and HTTP fallback; only a verified local-mirror read is a cache hit. From 2a700fd009d15330bd535be48f624d1689fea18a Mon Sep 17 00:00:00 2001 From: AlanD20 Date: Sun, 23 Aug 2026 16:27:13 +0200 Subject: [PATCH 0033/1457] test: require mirrored builder artifacts --- scripts/run-system-tests.py | 82 ++++++-- scripts/tests/test_run_system_tests.py | 253 +++++++++++++++++++++++-- 2 files changed, 309 insertions(+), 26 deletions(-) diff --git a/scripts/run-system-tests.py b/scripts/run-system-tests.py index 8d55a4ba..ffbba744 100755 --- a/scripts/run-system-tests.py +++ b/scripts/run-system-tests.py @@ -77,6 +77,14 @@ SHARED_VOLUME_NAME = "asset-mirror" SHARED_VOLUME_SIZE = "6G" TEST_NETWORK_NAME = "sys-test-net" +# These markers qualify DownloadFile artifact bytes. Fixed checksum metadata +# fetched through GetBody may still use its configured upstream origin. +LOCAL_MIRROR_READ_MARKER = "Using local mirror for download" +FORBIDDEN_BUILDER_PULL_MARKERS = ( + "Mirror checksum mismatch, falling back to HTTP download", + "Copied to asset mirror", +) +BUILDER_PULL_OUTPUT_LIMIT = 8 * 1024 # Custom base image built during --prepare BASE_IMAGE_NAME = "mvm-test-runner" @@ -1019,33 +1027,64 @@ def destroy_vm(vm_name: str) -> None: mvm("vm", "rm", vm_name, "--force", timeout=60, check=False) +def _bounded_builder_pull_output( + result: subprocess.CompletedProcess[str], +) -> str: + """Render captured pull output without flooding the qualification log.""" + sections = [] + if result.stdout: + sections.append(f"[stdout]\n{result.stdout.rstrip()}") + if result.stderr: + sections.append(f"[stderr]\n{result.stderr.rstrip()}") + output = "\n".join(sections) or "(no captured output)" + if len(output) <= BUILDER_PULL_OUTPUT_LIMIT: + return output + + marker = "\n... builder pull output truncated ...\n" + remaining = BUILDER_PULL_OUTPUT_LIMIT - len(marker) + head = remaining // 2 + return f"{output[:head]}{marker}{output[-(remaining - head):]}" + + def _pull_builder_assets(vm_name: str) -> None: - """Pull base-image assets serially so each streamed exec stays bounded.""" + """Pull base-image assets serially and require local-mirror evidence.""" pulls = ( ( "Firecracker kernel", - "MVM_ASSET_MIRROR=/mnt /usr/local/bin/mvm kernel pull " - "--type firecracker --version v1.15 --default", + ( + "kernel", + "pull", + "--type", + "firecracker", + "--version", + "v1.15", + "--default", + ), ), ( "Alpine image", - "MVM_ASSET_MIRROR=/mnt /usr/local/bin/mvm image pull " - "alpine:3.23", + ("image", "pull", "alpine:3.23"), ), ( "Ubuntu Minimal image", - "MVM_ASSET_MIRROR=/mnt /usr/local/bin/mvm image pull " - "ubuntu-minimal:24.04", + ("image", "pull", "ubuntu-minimal:24.04"), ), ( "Ubuntu image", - "MVM_ASSET_MIRROR=/mnt /usr/local/bin/mvm image pull ubuntu " - "--version 24.04", + ("image", "pull", "ubuntu", "--version", "24.04"), ), ) - for label, command in pulls: + for label, pull_args in pulls: log(f" Pulling {label} into '{vm_name}'...") - mvm( + command = shlex.join( + ( + "MVM_ASSET_MIRROR=/mnt", + "MVM_LOG_LEVEL=INFO", + "/usr/local/bin/mvm", + *pull_args, + ) + ) + result = mvm( "exec", vm_name, "--user", @@ -1055,8 +1094,27 @@ def _pull_builder_assets(vm_name: str) -> None: "--", command, timeout=600, - capture=False, + capture=True, + check=False, ) + output = f"{result.stdout or ''}\n{result.stderr or ''}" + bounded_output = _bounded_builder_pull_output(result) + print(f" {label} output:\n{bounded_output}", flush=True) + if result.returncode != 0: + raise RuntimeError( + f"builder asset pull failed for {label}: rc={result.returncode}" + ) + for forbidden_marker in FORBIDDEN_BUILDER_PULL_MARKERS: + if forbidden_marker in output: + raise RuntimeError( + f"builder asset pull for {label} reported forbidden mirror " + f"event: {forbidden_marker}" + ) + if LOCAL_MIRROR_READ_MARKER not in output: + raise RuntimeError( + f"builder asset pull for {label} did not prove a local mirror " + "artifact read" + ) log(f" Verifying cached image integrity in '{vm_name}'...") mvm( diff --git a/scripts/tests/test_run_system_tests.py b/scripts/tests/test_run_system_tests.py index 4ded3a64..a307bd95 100644 --- a/scripts/tests/test_run_system_tests.py +++ b/scripts/tests/test_run_system_tests.py @@ -5,6 +5,7 @@ import importlib.util import json import os +import shlex import shutil import subprocess import sys @@ -1173,19 +1174,28 @@ def test_builder_assets_are_pulled_in_separate_sequential_execs( def fake_mvm(*args: str, **kwargs: object) -> subprocess.CompletedProcess[str]: calls.append((args, kwargs)) - return subprocess.CompletedProcess(args, 0, stdout="", stderr="") + return subprocess.CompletedProcess( + args, + 0, + stdout="pull completed\n", + stderr="INFO: root: Using local mirror for download url=https://example.test\n", + ) monkeypatch.setattr(orchestrator, "mvm", fake_mvm) orchestrator._pull_builder_assets("base-img-builder") expected_pull_commands = [ - "MVM_ASSET_MIRROR=/mnt /usr/local/bin/mvm kernel pull " + "MVM_ASSET_MIRROR=/mnt MVM_LOG_LEVEL=INFO " + "/usr/local/bin/mvm kernel pull " "--type firecracker --version v1.15 --default", - "MVM_ASSET_MIRROR=/mnt /usr/local/bin/mvm image pull alpine:3.23", - "MVM_ASSET_MIRROR=/mnt /usr/local/bin/mvm image pull " + "MVM_ASSET_MIRROR=/mnt MVM_LOG_LEVEL=INFO " + "/usr/local/bin/mvm image pull alpine:3.23", + "MVM_ASSET_MIRROR=/mnt MVM_LOG_LEVEL=INFO " + "/usr/local/bin/mvm image pull " "ubuntu-minimal:24.04", - "MVM_ASSET_MIRROR=/mnt /usr/local/bin/mvm image pull ubuntu " + "MVM_ASSET_MIRROR=/mnt MVM_LOG_LEVEL=INFO " + "/usr/local/bin/mvm image pull ubuntu " "--version 24.04", ] pull_calls = calls[:-1] @@ -1204,7 +1214,8 @@ def fake_mvm(*args: str, **kwargs: object) -> subprocess.CompletedProcess[str]: ) assert " & " not in args[-1] assert "pids=" not in args[-1] - assert kwargs["capture"] is False + assert kwargs["capture"] is True + assert kwargs["check"] is False assert kwargs["timeout"] == 600 verify_args, verify_kwargs = calls[-1] @@ -1213,31 +1224,245 @@ def fake_mvm(*args: str, **kwargs: object) -> subprocess.CompletedProcess[str]: assert verify_kwargs == {"timeout": 180, "capture": False} -def test_builder_asset_pull_preserves_first_streamed_failure( +def test_builder_asset_pull_keeps_dynamic_vm_name_outside_guest_shell( + orchestrator: ModuleType, + monkeypatch: pytest.MonkeyPatch, +) -> None: + calls: list[tuple[str, ...]] = [] + hostile_vm_name = "builder; touch /tmp/runner-command-injection" + + def fake_mvm(*args: str, **_kwargs: object) -> subprocess.CompletedProcess[str]: + calls.append(args) + return subprocess.CompletedProcess( + args, + 0, + stdout="", + stderr="INFO: root: Using local mirror for download url=https://example.test\n", + ) + + monkeypatch.setattr(orchestrator, "mvm", fake_mvm) + + orchestrator._pull_builder_assets(hostile_vm_name) + + for args in calls[:-1]: + assert args[1] == hostile_vm_name + assert hostile_vm_name not in args[-1] + command_tokens = shlex.split(args[-1]) + assert command_tokens[:3] == [ + "MVM_ASSET_MIRROR=/mnt", + "MVM_LOG_LEVEL=INFO", + "/usr/local/bin/mvm", + ] + assert args[-1] == shlex.join(command_tokens) + + +def test_builder_asset_pull_rejects_missing_local_mirror_evidence( + orchestrator: ModuleType, + monkeypatch: pytest.MonkeyPatch, +) -> None: + def fake_mvm(*args: str, **_kwargs: object) -> subprocess.CompletedProcess[str]: + return subprocess.CompletedProcess( + args, + 0, + stdout="pull completed\n", + stderr="", + ) + + monkeypatch.setattr(orchestrator, "mvm", fake_mvm) + + with pytest.raises( + RuntimeError, + match="did not prove a local mirror artifact read", + ): + orchestrator._pull_builder_assets("base-img-builder") + + +@pytest.mark.parametrize( + "forbidden_marker", + [ + "Mirror checksum mismatch, falling back to HTTP download", + "Copied to asset mirror", + ], +) +def test_builder_asset_pull_rejects_http_fallback_or_autopopulation( orchestrator: ModuleType, monkeypatch: pytest.MonkeyPatch, + forbidden_marker: str, ) -> None: commands: list[str] = [] - original_failure = RuntimeError( - "mvm command failed: streamed ubuntu-minimal pull; rc: 23; " - "stderr: (streamed; not captured)" + + def fake_mvm(*args: str, **_kwargs: object) -> subprocess.CompletedProcess[str]: + commands.append(args[-1]) + return subprocess.CompletedProcess( + args, + 0, + stdout="pull completed\n", + stderr=( + "INFO: root: Using local mirror for download url=https://example.test\n" + f"WARNING: root: {forbidden_marker} url=https://example.test\n" + ), + ) + + monkeypatch.setattr(orchestrator, "mvm", fake_mvm) + + with pytest.raises(RuntimeError) as captured: + orchestrator._pull_builder_assets("base-img-builder") + + assert forbidden_marker in str(captured.value) + assert len(commands) == 1 + + +def test_builder_asset_pull_prints_bounded_captured_output( + orchestrator: ModuleType, + monkeypatch: pytest.MonkeyPatch, + capsys: pytest.CaptureFixture[str], +) -> None: + huge_stdout = "stdout-start\n" + ("s" * 50_000) + "\nstdout-end\n" + huge_stderr = ( + "stderr-start\n" + + ("e" * 50_000) + + "\nINFO: root: Using local mirror for download url=https://example.test\n" + + "stderr-end\n" ) + def fake_mvm(*args: str, **_kwargs: object) -> subprocess.CompletedProcess[str]: + return subprocess.CompletedProcess( + args, + 0, + stdout=huge_stdout, + stderr=huge_stderr, + ) + + monkeypatch.setattr(orchestrator, "mvm", fake_mvm) + + orchestrator._pull_builder_assets("base-img-builder") + + output = capsys.readouterr().out + assert "stdout-start" in output + assert "stderr-end" in output + assert "builder pull output truncated" in output + assert len(output) < 40_000 + + +@pytest.mark.parametrize( + ("returncode", "expected_error"), + [ + (23, "builder asset pull failed for Firecracker kernel: rc=23"), + (0, "did not prove a local mirror artifact read"), + ], +) +def test_builder_asset_pull_handles_missing_captured_streams( + orchestrator: ModuleType, + monkeypatch: pytest.MonkeyPatch, + capsys: pytest.CaptureFixture[str], + returncode: int, + expected_error: str, +) -> None: + commands: list[str] = [] + def fake_mvm(*args: str, **_kwargs: object) -> subprocess.CompletedProcess[str]: commands.append(args[-1]) + return subprocess.CompletedProcess( + args, + returncode, + stdout=None, + stderr=None, + ) + + monkeypatch.setattr(orchestrator, "mvm", fake_mvm) + + with pytest.raises(RuntimeError, match=expected_error): + orchestrator._pull_builder_assets("base-img-builder") + + assert len(commands) == 1 + assert "(no captured output)" in capsys.readouterr().out + + +@pytest.mark.parametrize( + ("stdout", "stderr"), + [ + ("Using local mirror for download\n", None), + (None, "Using local mirror for download\n"), + ], +) +def test_builder_asset_pull_accepts_mirror_evidence_with_one_missing_stream( + orchestrator: ModuleType, + monkeypatch: pytest.MonkeyPatch, + stdout: str | None, + stderr: str | None, +) -> None: + def fake_mvm(*args: str, **_kwargs: object) -> subprocess.CompletedProcess[str]: + return subprocess.CompletedProcess( + args, + 0, + stdout=stdout, + stderr=stderr, + ) + + monkeypatch.setattr(orchestrator, "mvm", fake_mvm) + + orchestrator._pull_builder_assets("base-img-builder") + + +def test_builder_asset_pull_preserves_nonzero_failure_output( + orchestrator: ModuleType, + monkeypatch: pytest.MonkeyPatch, + capsys: pytest.CaptureFixture[str], +) -> None: + def fake_mvm(*args: str, **_kwargs: object) -> subprocess.CompletedProcess[str]: + return subprocess.CompletedProcess( + args, + 17, + stdout="partial pull progress\n", + stderr="download process terminated\n", + ) + + monkeypatch.setattr(orchestrator, "mvm", fake_mvm) + + with pytest.raises(RuntimeError, match="rc=17"): + orchestrator._pull_builder_assets("base-img-builder") + + output = capsys.readouterr().out + assert "partial pull progress" in output + assert "download process terminated" in output + + +def test_builder_asset_pull_stops_after_first_nonzero_result( + orchestrator: ModuleType, + monkeypatch: pytest.MonkeyPatch, + capsys: pytest.CaptureFixture[str], +) -> None: + calls: list[tuple[tuple[str, ...], dict[str, object]]] = [] + + def fake_mvm(*args: str, **kwargs: object) -> subprocess.CompletedProcess[str]: + calls.append((args, kwargs)) if "ubuntu-minimal:24.04" in args[-1]: - raise original_failure - return subprocess.CompletedProcess(args, 0, stdout="", stderr="") + return subprocess.CompletedProcess( + args, + 23, + stdout="partial ubuntu-minimal progress\n", + stderr="ubuntu-minimal download failed\n", + ) + return subprocess.CompletedProcess( + args, + 0, + stdout="", + stderr="INFO: root: Using local mirror for download url=https://example.test\n", + ) monkeypatch.setattr(orchestrator, "mvm", fake_mvm) - with pytest.raises(RuntimeError) as captured: + with pytest.raises(RuntimeError, match="rc=23"): orchestrator._pull_builder_assets("base-img-builder") - assert captured.value is original_failure + commands = [args[-1] for args, _kwargs in calls] assert commands[-1].endswith("image pull ubuntu-minimal:24.04") + assert calls[-1][1] == {"timeout": 600, "capture": True, "check": False} assert not any("--version 24.04" in command for command in commands) assert not any("Verifying cached image integrity" in command for command in commands) + output = capsys.readouterr().out + assert "partial ubuntu-minimal progress" in output + assert "ubuntu-minimal download failed" in output def test_build_base_image_cleans_builder_when_asset_pull_fails( From 0b3dd2a7db5813841e2615f98f1deb8a3501d12c Mon Sep 17 00:00:00 2001 From: AlanD20 Date: Sun, 23 Aug 2026 16:28:02 +0200 Subject: [PATCH 0034/1457] docs: require mirror-qualified builder pulls --- CHANGELOG.md | 3 +++ CONTEXT.md | 5 +++-- docs/RC_QA.md | 3 +++ docs/development/HOW_TO_RUN_SYSTEM_TESTS.md | 7 +++++-- docs/system-test-architecture.md | 9 +++++---- tasks/todo.md | 2 +- 6 files changed, 20 insertions(+), 9 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index acb2bf7e..c535184b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -111,6 +111,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 `mkfs.ext4 -d`; the runner no longer loop-mounts and copies the mirror with raw sudo on the host. - Builder asset pulls run sequentially with bounded connect/operation timeouts so image extraction cannot exhaust the constrained nested builder through concurrent pulls. +- Builder pulls capture and re-emit bounded diagnostics, require explicit local-mirror artifact-read evidence, and fail + qualification on pull errors, checksum fallback, or HTTP-downloaded mirror auto-population. Fixed checksum/version + metadata may still use its upstream source. - Missing or stale `asset-mirror` database records are rebuilt only when the recorded backing path is the exact managed test-volume path; unexpected paths fail closed without mutation. diff --git a/CONTEXT.md b/CONTEXT.md index 8c27221d..7f42752d 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -263,8 +263,9 @@ mvm This variable directs the local asset cache for downloaded kernel images, root filesystems, and firmware blobs. Checksum/version metadata may still be fetched from the fixed upstream origin. The system-test `asset-mirror` volume is a point-in-time, read-only copy of this directory: after repairing or adding a host mirror object, rebuild that volume -before running QA. A stale object inside the read-only copy can be detected and bypassed, but it cannot persist its own -replacement. +before running QA. Although the downloader can detect a stale object and try its fixed upstream source, the release +runner rejects that fallback and any mirror auto-population as invalid qualification evidence. Builder pulls pass only +when each artifact reports an explicit local-mirror read; checksum/version metadata may still use upstream. ### CLI is the canonical interface diff --git a/docs/RC_QA.md b/docs/RC_QA.md index e4b6447f..4851f3ed 100644 --- a/docs/RC_QA.md +++ b/docs/RC_QA.md @@ -83,6 +83,9 @@ lookup, or aliasing the candidate to the controller does not qualify the install The builder's unprivileged initialization pins Firecracker 1.16.0 through `mvm init --non-interactive --binary-version 1.16.0`; it must not resolve a moving latest release during qualification. +Every builder artifact pull must also report an explicit local-mirror read. A pull error, checksum fallback, or +HTTP-downloaded mirror auto-population fails qualification; bounded stdout/stderr remains in the evidence log. Fixed +checksum/version metadata may still use its upstream source. Tier 3 is different: those tests run directly on the outer host and may create or destroy host resources. The runner rejects Tier 3 and `--all` before any work unless `--host-direct` acknowledges that mutation. For release evidence, diff --git a/docs/development/HOW_TO_RUN_SYSTEM_TESTS.md b/docs/development/HOW_TO_RUN_SYSTEM_TESTS.md index a5c3b367..d7ed5756 100644 --- a/docs/development/HOW_TO_RUN_SYSTEM_TESTS.md +++ b/docs/development/HOW_TO_RUN_SYSTEM_TESTS.md @@ -197,8 +197,11 @@ MVM_ASSET_MIRROR=~/.cache/mvm-asset-mirror \ python3 scripts/run-system-tests.py --volume ``` -Do not expect the read-only guest copy to self-heal. It can detect a checksum mismatch and use an HTTP fallback for that -run, but it cannot persist the replacement. Repair the writable host mirror and reseed the volume instead. +Do not expect the read-only guest copy to self-heal. The downloader can detect a checksum mismatch and attempt its fixed +HTTP source, but the system-test runner rejects that fallback and any HTTP-downloaded mirror auto-population. Each +builder artifact pull must report an explicit local-mirror read; the runner captures and re-emits bounded pull output so +the failure remains diagnosable. Repair the writable host mirror and reseed the volume instead. Small checksum/version +metadata requests may still use their fixed upstream sources and are not treated as artifact fallback. ### 3.2 Run `--prepare` diff --git a/docs/system-test-architecture.md b/docs/system-test-architecture.md index 18edd584..98fb176b 100644 --- a/docs/system-test-architecture.md +++ b/docs/system-test-architecture.md @@ -133,10 +133,11 @@ once and reused across all test runs. The orchestrator accepts only the exact ca `${MVM_CACHE_DIR:-~/.cache/mvmctl}/volumes/asset-mirror.raw` backing path. Missing stale records are rebuilt; an unexpected path fails closed without mutation. -This volume is a point-in-time copy, not a writable cache. A checksum mismatch may make a guest fall back to the -upstream URL, but the guest cannot persist the corrected object into the read-only volume. Repair the writable host -mirror first, then use `--volume` to recreate the shared copy. Checksum/version metadata may still be fetched upstream -even when the large artifact is served locally. +This volume is a point-in-time copy, not a writable cache. The product downloader can fall back to the fixed upstream +URL after a checksum mismatch, but the system-test runner treats that fallback and HTTP-downloaded mirror +auto-population as qualification failures. Every builder artifact pull must emit explicit local-mirror-read evidence; +its bounded captured output is retained for diagnosis. Repair the writable host mirror, then use `--volume` to recreate +the shared copy. Checksum/version metadata may still be fetched upstream even when the large artifact is served locally. ```bash mvm volume create asset-mirror 6G --shareable --read-only --writeback --format raw diff --git a/tasks/todo.md b/tasks/todo.md index a07c40a0..dd7aa909 100644 --- a/tasks/todo.md +++ b/tasks/todo.md @@ -357,7 +357,7 @@ under `internal/service/jailer/`. No handler, transport, CLI, API, core-domain, reports and cleanup failures fail the domain while preserving pytest output and the reason. - [x] Ensure the full matrix selects every required domain exactly once, including `exec`, and does not run `test_vm_fresh_env.py` twice under aliases. -- [ ] Make mirror validation reject pull errors and HTTP fallback; only a verified local-mirror read is a cache hit. +- [x] Make mirror validation reject pull errors and HTTP fallback; only a verified local-mirror read is a cache hit. - [ ] Run the complete T1/T2/T3 matrix plus clean-host installation and reboot/reconciliation qualification. - [ ] Update `tests/system/COVERAGE_MATRIX.md` and attach commands/results to the release report. From fc44fefd90705797f7d95caada7dc04eeb10294e Mon Sep 17 00:00:00 2001 From: AlanD20 Date: Sun, 23 Aug 2026 16:46:08 +0200 Subject: [PATCH 0035/1457] feat: add strict privileged wire codec --- internal/lib/privilegedwire/doc.go | 8 + internal/lib/privilegedwire/error.go | 421 ++++++++++++ internal/lib/privilegedwire/error_test.go | 640 +++++++++++++++++++ internal/lib/privilegedwire/frame.go | 102 +++ internal/lib/privilegedwire/json.go | 279 ++++++++ internal/lib/privilegedwire/request.go | 146 +++++ internal/lib/privilegedwire/request_test.go | 353 ++++++++++ internal/lib/privilegedwire/response.go | 239 +++++++ internal/lib/privilegedwire/response_test.go | 329 ++++++++++ 9 files changed, 2517 insertions(+) create mode 100644 internal/lib/privilegedwire/doc.go create mode 100644 internal/lib/privilegedwire/error.go create mode 100644 internal/lib/privilegedwire/error_test.go create mode 100644 internal/lib/privilegedwire/frame.go create mode 100644 internal/lib/privilegedwire/json.go create mode 100644 internal/lib/privilegedwire/request.go create mode 100644 internal/lib/privilegedwire/request_test.go create mode 100644 internal/lib/privilegedwire/response.go create mode 100644 internal/lib/privilegedwire/response_test.go diff --git a/internal/lib/privilegedwire/doc.go b/internal/lib/privilegedwire/doc.go new file mode 100644 index 00000000..f10df0a9 --- /dev/null +++ b/internal/lib/privilegedwire/doc.go @@ -0,0 +1,8 @@ +// Package privilegedwire encodes and decodes the bounded control frames shared by +// typed privileged clients and receivers. It owns no actions, dispatch, subprocesses, +// or privileged effects. +// +// DomainError details use map[string]any because the project error type defines that +// intentional sum type. This package closes the wire form to bool, string, int64, and +// bounded string slices through an explicit type switch. +package privilegedwire diff --git a/internal/lib/privilegedwire/error.go b/internal/lib/privilegedwire/error.go new file mode 100644 index 00000000..12eb605c --- /dev/null +++ b/internal/lib/privilegedwire/error.go @@ -0,0 +1,421 @@ +package privilegedwire + +import ( + "bytes" + "encoding/json" + "sort" + "strconv" + "strings" + "unicode" + "unicode/utf8" + + "mvmctl/pkg/errs" +) + +const ( + maxErrorCodeBytes = 128 + maxErrorMessageBytes = 8 * 1024 + maxErrorOperationBytes = 128 + maxErrorEntityBytes = 1024 + maxErrorDetailCount = 32 + maxErrorDetailKeyBytes = 64 + maxErrorDetailString = 4 * 1024 + maxErrorDetailArray = 32 + maxErrorDetailArrayItem = 1024 + + omittedDetailsKey = "privileged_details_omitted" +) + +type domainErrorWire struct { + Code string `json:"code"` + Class string `json:"class"` + Message string `json:"message"` + Operation string `json:"operation"` + Entity string `json:"entity"` + Details map[string]json.RawMessage `json:"details"` +} + +var domainErrorFields = fieldSet("code", "class", "message", "operation", "entity", "details") + +func normalizeDomainError(operationErr error) domainErrorWire { + domainErr := errs.AsDomainError(operationErr) + if domainErr == nil || !validWireDomainError(domainErr) { + return fallbackDomainError() + } + details := normalizeErrorDetails(domainErr.Details) + return domainErrorWire{ + Code: string(domainErr.Code), + Class: classString(domainErr.Class), + Message: domainErr.Message, + Operation: domainErr.Op, + Entity: domainErr.Entity, + Details: details, + } +} + +func validWireDomainError(domainErr *errs.DomainError) bool { + return validASCIIIdentifier(string(domainErr.Code), maxErrorCodeBytes) && + classString(domainErr.Class) != "" && + validWireText(domainErr.Message, maxErrorMessageBytes, true) && + validWireText(domainErr.Op, maxErrorOperationBytes, false) && + validWireText(domainErr.Entity, maxErrorEntityBytes, false) +} + +func normalizeErrorDetails(details map[string]any) map[string]json.RawMessage { + if len(details) == 0 { + return map[string]json.RawMessage{} + } + _, includeFlag := details[omittedDetailsKey] + candidates := make([]string, 0, maxErrorDetailCount) + validDetailCount := 0 + for key, value := range details { + if key == omittedDetailsKey { + continue + } + if !validErrorDetail(key, value) { + includeFlag = true + continue + } + validDetailCount++ + candidates = retainErrorDetailCandidate(candidates, key) + } + if validDetailCount > maxErrorDetailCount { + includeFlag = true + } + retainedLimit := maxErrorDetailCount + if includeFlag { + retainedLimit-- + } + if len(candidates) > retainedLimit { + candidates = candidates[:retainedLimit] + } + result := make(map[string]json.RawMessage, maxErrorDetailCount) + for _, key := range candidates { + encoded, ok := encodeErrorDetail(key, details[key]) + if !ok { + includeFlag = true + continue + } + result[key] = encoded + } + if includeFlag { + result[omittedDetailsKey] = json.RawMessage("true") + } + return result +} + +func retainErrorDetailCandidate(candidates []string, key string) []string { + index := sort.SearchStrings(candidates, key) + if len(candidates) < maxErrorDetailCount { + candidates = append(candidates, "") + copy(candidates[index+1:], candidates[index:len(candidates)-1]) + candidates[index] = key + return candidates + } + if index == maxErrorDetailCount { + return candidates + } + copy(candidates[index+1:], candidates[index:len(candidates)-1]) + candidates[index] = key + return candidates +} + +func encodeErrorDetail(key string, value any) (json.RawMessage, bool) { + if key == omittedDetailsKey { + flag, ok := value.(bool) + if !ok || !flag { + return nil, false + } + return json.RawMessage("true"), true + } + if !validErrorDetail(key, value) { + return nil, false + } + return marshalDetailValue(value) +} + +func validErrorDetail(key string, value any) bool { + if key == omittedDetailsKey || + !validASCIIIdentifier(key, maxErrorDetailKeyBytes) || isSensitiveErrorDetailKey(key) { + return false + } + switch typed := value.(type) { + case bool, int64: + return true + case string: + return validWireText(typed, maxErrorDetailString, true) + case []string: + if len(typed) > maxErrorDetailArray { + return false + } + for _, item := range typed { + if !validWireText(item, maxErrorDetailArrayItem, true) { + return false + } + } + return true + default: + return false + } +} + +func isSensitiveErrorDetailKey(key string) bool { + tokenStart := 0 + for index := 0; index <= len(key); index++ { + if index < len(key) && !isErrorDetailKeySeparator(key[index]) { + continue + } + if isSensitiveErrorDetailToken(key[tokenStart:index]) { + return true + } + tokenStart = index + 1 + } + return false +} + +func isErrorDetailKeySeparator(character byte) bool { + return character == '.' || character == '_' || character == '-' +} + +func isSensitiveErrorDetailToken(token string) bool { + return strings.EqualFold(token, "err") || + strings.EqualFold(token, "error") || + strings.EqualFold(token, "cause") || + strings.EqualFold(token, "stack") || + strings.EqualFold(token, "stacktrace") || + strings.EqualFold(token, "stderr") || + strings.EqualFold(token, "stdout") +} + +func marshalDetailValue(value any) (json.RawMessage, bool) { + encoded, err := json.Marshal(value) + if err != nil { + return nil, false + } + return encoded, true +} + +func decodeDomainError(raw json.RawMessage) (*errs.DomainError, error) { + fields, err := decodeExactObject( + raw, + domainErrorFields, + []string{"code", "class", "message", "operation", "entity", "details"}, + "error", + ) + if err != nil { + return nil, err + } + code, err := decodeJSONField[string](fields["code"], "error code") + if err != nil { + return nil, err + } + if !validASCIIIdentifier(code, maxErrorCodeBytes) { + return nil, errs.New(errs.CodeValidationFailed, "invalid privileged error code") + } + classValue, err := decodeJSONField[string](fields["class"], "error class") + if err != nil { + return nil, err + } + class, ok := parseClassString(classValue) + if !ok { + return nil, errs.New(errs.CodeValidationFailed, "invalid privileged error class") + } + message, err := decodeBoundedWireString(fields["message"], "error message", maxErrorMessageBytes, true) + if err != nil { + return nil, err + } + operation, err := decodeBoundedWireString(fields["operation"], "error operation", maxErrorOperationBytes, false) + if err != nil { + return nil, err + } + entity, err := decodeBoundedWireString(fields["entity"], "error entity", maxErrorEntityBytes, false) + if err != nil { + return nil, err + } + details, err := decodeErrorDetails(fields["details"]) + if err != nil { + return nil, err + } + return &errs.DomainError{ + Code: errs.Code(code), + Class: class, + Message: message, + Op: operation, + Entity: entity, + Details: details, + }, nil +} + +func decodeErrorDetails(raw json.RawMessage) (map[string]any, error) { + var fields map[string]json.RawMessage + if err := json.Unmarshal(raw, &fields); err != nil { + return nil, errs.WrapMsg(errs.CodeValidationFailed, "decode privileged error details", err) + } + if fields == nil { + return nil, errs.New(errs.CodeValidationFailed, "privileged error details must be a JSON object") + } + if len(fields) > maxErrorDetailCount { + return nil, errs.New(errs.CodeValidationFailed, "privileged error details exceed 32 entries") + } + details := make(map[string]any, len(fields)) + for key, value := range fields { + if !validASCIIIdentifier(key, maxErrorDetailKeyBytes) || isSensitiveErrorDetailKey(key) { + return nil, errs.New(errs.CodeValidationFailed, "invalid privileged error detail key") + } + decoded, err := decodeErrorDetail(value) + if err != nil { + return nil, err + } + if key == omittedDetailsKey { + flag, ok := decoded.(bool) + if !ok || !flag { + return nil, errs.New(errs.CodeValidationFailed, "invalid privileged detail omission flag") + } + } + details[key] = decoded + } + return details, nil +} + +func decodeErrorDetail(raw json.RawMessage) (any, error) { + trimmed := bytes.TrimSpace(raw) + if len(trimmed) == 0 { + return nil, errs.New(errs.CodeValidationFailed, "privileged error detail is empty") + } + switch trimmed[0] { + case 't', 'f': + return decodeJSONField[bool](trimmed, "error detail") + case '"': + return decodeBoundedWireString(trimmed, "error detail", maxErrorDetailString, true) + case '[': + values, err := decodeJSONField[[]string](trimmed, "error detail string array") + if err != nil { + return nil, err + } + if len(values) > maxErrorDetailArray { + return nil, errs.New(errs.CodeValidationFailed, "privileged error detail array exceeds 32 items") + } + for _, value := range values { + if !validWireText(value, maxErrorDetailArrayItem, true) { + return nil, errs.New(errs.CodeValidationFailed, "invalid privileged error detail array item") + } + } + return values, nil + case '-', '0', '1', '2', '3', '4', '5', '6', '7', '8', '9': + value, err := strconv.ParseInt(string(trimmed), 10, 64) + if err != nil { + return nil, errs.WrapMsg(errs.CodeValidationFailed, "invalid privileged error integer detail", err) + } + return value, nil + default: + return nil, errs.New(errs.CodeValidationFailed, "unsupported privileged error detail value") + } +} + +func decodeBoundedWireString( + raw json.RawMessage, + description string, + maxBytes int, + allowLineBreaks bool, +) (string, error) { + value, err := decodeJSONField[string](raw, description) + if err != nil { + return "", err + } + if !validWireText(value, maxBytes, allowLineBreaks) { + return "", errs.New(errs.CodeValidationFailed, "invalid privileged "+description) + } + return value, nil +} + +func validWireText(value string, maxBytes int, allowLineBreaks bool) bool { + if len(value) > maxBytes || !utf8.ValidString(value) { + return false + } + for _, character := range value { + if character == '\n' || character == '\t' { + if allowLineBreaks { + continue + } + return false + } + if unicode.IsControl(character) || unicode.In(character, unicode.Cf) { + return false + } + } + return true +} + +func validASCIIIdentifier(value string, maxBytes int) bool { + if value == "" || len(value) > maxBytes { + return false + } + if !isASCIIAlphaNumeric(value[0]) { + return false + } + for index := range len(value) { + character := value[index] + if isASCIIAlphaNumeric(character) || character == '.' || character == '_' || character == '-' { + continue + } + return false + } + return true +} + +func isASCIIAlphaNumeric(character byte) bool { + return (character >= 'a' && character <= 'z') || + (character >= 'A' && character <= 'Z') || + (character >= '0' && character <= '9') +} + +func classString(class errs.Class) string { + switch class { + case errs.ClassUnknown: + return "unknown" + case errs.ClassValidation: + return "validation" + case errs.ClassConflict: + return "conflict" + case errs.ClassRetryable: + return "retryable" + case errs.ClassInternal: + return "internal" + case errs.ClassNeedsInteraction: + return "needs_interaction" + default: + return "" + } +} + +func parseClassString(value string) (errs.Class, bool) { + switch value { + case "unknown": + return errs.ClassUnknown, true + case "validation": + return errs.ClassValidation, true + case "conflict": + return errs.ClassConflict, true + case "retryable": + return errs.ClassRetryable, true + case "internal": + return errs.ClassInternal, true + case "needs_interaction": + return errs.ClassNeedsInteraction, true + default: + return errs.ClassUnknown, false + } +} + +func fallbackDomainError() domainErrorWire { + return domainErrorWire{ + Code: string(errs.CodeInternal), + Class: "internal", + Message: "privileged operation outcome could not be encoded", + Operation: "privileged", + Entity: "", + Details: map[string]json.RawMessage{ + "outcome_unknown": json.RawMessage("true"), + }, + } +} diff --git a/internal/lib/privilegedwire/error_test.go b/internal/lib/privilegedwire/error_test.go new file mode 100644 index 00000000..da56bd71 --- /dev/null +++ b/internal/lib/privilegedwire/error_test.go @@ -0,0 +1,640 @@ +package privilegedwire_test + +import ( + "bytes" + "context" + "errors" + "fmt" + "strings" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "mvmctl/internal/lib/privilegedwire" + "mvmctl/pkg/errs" +) + +func TestDomainErrorResponsePreservesSafeAuthorityFields(t *testing.T) { + source := &errs.DomainError{ + Code: errs.Code("vm.cleanup.partial"), + Class: errs.ClassConflict, + Message: "cleanup changed state\nretry after inspection\tcarefully", + Op: "vm.cleanup", + Entity: "0123456789abcdef", + Err: errors.New("secret wrapped stderr and stack material"), + Details: map[string]any{ + "record_replaced": true, + "cleanup_generation": int64(7), + "phase": "mounts", + "resources": []string{"rootfs", "volume-1"}, + }, + } + + var encoded bytes.Buffer + err := privilegedwire.EncodeError(context.Background(), &encoded, "jailer-cleanup", source) + require.NoError(t, err) + assert.NotContains(t, encoded.String(), "secret wrapped stderr") + + outcome, protocolErr := privilegedwire.DecodeResponse[cleanupResult]( + context.Background(), + bytes.NewReader(encoded.Bytes()), + "jailer-cleanup", + ) + require.NoError(t, protocolErr) + decoded := outcome.OperationError() + require.NotNil(t, decoded) + assert.Equal(t, source.Code, decoded.Code) + assert.Equal(t, source.Class, decoded.Class) + assert.Equal(t, source.Message, decoded.Message) + assert.Equal(t, source.Op, decoded.Op) + assert.Equal(t, source.Entity, decoded.Entity) + assert.Nil(t, decoded.Err) + assert.Equal(t, source.Details, decoded.Details) +} + +func TestDomainErrorResponseUsesStableClassStrings(t *testing.T) { + tests := map[string]struct { + class errs.Class + wire string + }{ + "unknown": {class: errs.ClassUnknown, wire: "unknown"}, + "validation": {class: errs.ClassValidation, wire: "validation"}, + "conflict": {class: errs.ClassConflict, wire: "conflict"}, + "retryable": {class: errs.ClassRetryable, wire: "retryable"}, + "internal": {class: errs.ClassInternal, wire: "internal"}, + "needs_interaction": {class: errs.ClassNeedsInteraction, wire: "needs_interaction"}, + } + + for name, tc := range tests { + t.Run(name, func(t *testing.T) { + source := &errs.DomainError{ + Code: errs.CodeProcessError, + Class: tc.class, + Message: "operation failed", + Op: "vm.cleanup", + Details: map[string]any{}, + } + var encoded bytes.Buffer + err := privilegedwire.EncodeError(context.Background(), &encoded, "jailer-cleanup", source) + require.NoError(t, err) + assert.Contains(t, encoded.String(), `"class":"`+tc.wire+`"`) + + outcome, protocolErr := privilegedwire.DecodeResponse[cleanupResult]( + context.Background(), + bytes.NewReader(encoded.Bytes()), + "jailer-cleanup", + ) + require.NoError(t, protocolErr) + require.NotNil(t, outcome.OperationError()) + assert.Equal(t, tc.class, outcome.OperationError().Class) + }) + } +} + +func TestDomainErrorResponseOmitsUnsupportedDetailsVisibly(t *testing.T) { + source := &errs.DomainError{ + Code: errs.CodeProcessError, + Class: errs.ClassInternal, + Message: "operation failed", + Op: "vm.cleanup", + Details: map[string]any{ + "safe": true, + "float": 1.5, + "null": nil, + "nested": map[string]string{"command": "id"}, + "integer_set": []int{1, 2}, + "stderr": "secret subprocess output", + "stack": "secret stack trace", + }, + } + var encoded bytes.Buffer + err := privilegedwire.EncodeError(context.Background(), &encoded, "jailer-cleanup", source) + require.NoError(t, err) + assert.NotContains(t, encoded.String(), "secret subprocess output") + assert.NotContains(t, encoded.String(), "secret stack trace") + + outcome, protocolErr := privilegedwire.DecodeResponse[cleanupResult]( + context.Background(), + bytes.NewReader(encoded.Bytes()), + "jailer-cleanup", + ) + require.NoError(t, protocolErr) + require.NotNil(t, outcome.OperationError()) + details := outcome.OperationError().Details + assert.Equal(t, true, details["safe"]) + assert.Equal(t, true, details["privileged_details_omitted"]) + assert.Len(t, details, 2) +} + +func TestDomainErrorResponseOmitsSensitiveDetailTokens(t *testing.T) { + tests := map[string]string{ + "raw_stderr": "raw_stderr", + "stderr_excerpt": "stderr_excerpt", + "process_stdout": "process.stdout", + "wrapped_cause": "wrapped-cause", + "nested_error": "process_error_message", + "stack_token": "handler_stack_trace", + "stacktrace_token": "HANDLER.STACKTRACE", + "err_token": "wrapped_err_text", + } + + for name, key := range tests { + t.Run(name, func(t *testing.T) { + secret := "secret-" + name + source := &errs.DomainError{ + Code: errs.CodeProcessError, + Class: errs.ClassInternal, + Message: "operation failed", + Op: "vm.cleanup", + Details: map[string]any{ + "record_replaced": true, + key: secret, + }, + } + var encoded bytes.Buffer + err := privilegedwire.EncodeError(context.Background(), &encoded, "jailer-cleanup", source) + require.NoError(t, err) + assert.NotContains(t, encoded.String(), secret) + + outcome, protocolErr := privilegedwire.DecodeResponse[cleanupResult]( + context.Background(), + bytes.NewReader(encoded.Bytes()), + "jailer-cleanup", + ) + require.NoError(t, protocolErr) + require.NotNil(t, outcome.OperationError()) + details := outcome.OperationError().Details + assert.Equal(t, true, details["record_replaced"]) + assert.Equal(t, true, details["privileged_details_omitted"]) + assert.Len(t, details, 2) + }) + } +} + +func TestDomainErrorResponsePreservesPartialStateDetailKeys(t *testing.T) { + details := map[string]any{ + "record_replaced": true, + "durability_uncertain": true, + "process_started": true, + "outcome_unknown": true, + "cleanup_generation": int64(4), + } + source := &errs.DomainError{ + Code: errs.CodeProcessError, + Class: errs.ClassInternal, + Message: "operation failed", + Op: "vm.cleanup", + Details: details, + } + + var encoded bytes.Buffer + err := privilegedwire.EncodeError(context.Background(), &encoded, "jailer-cleanup", source) + require.NoError(t, err) + outcome, protocolErr := privilegedwire.DecodeResponse[cleanupResult]( + context.Background(), + bytes.NewReader(encoded.Bytes()), + "jailer-cleanup", + ) + require.NoError(t, protocolErr) + require.NotNil(t, outcome.OperationError()) + assert.Equal(t, details, outcome.OperationError().Details) +} + +func TestDomainErrorResponseOmitsInvalidDetailKeys(t *testing.T) { + tests := map[string]string{ + "space_separator_bypass": "stderr excerpt", + "non_ascii": "phase_é", + } + + for name, key := range tests { + t.Run(name, func(t *testing.T) { + secret := "secret-" + name + source := &errs.DomainError{ + Code: errs.CodeProcessError, + Class: errs.ClassInternal, + Message: "operation failed", + Op: "vm.cleanup", + Details: map[string]any{ + "record_replaced": true, + key: secret, + }, + } + var encoded bytes.Buffer + err := privilegedwire.EncodeError(context.Background(), &encoded, "jailer-cleanup", source) + require.NoError(t, err) + assert.NotContains(t, encoded.String(), secret) + + outcome, protocolErr := privilegedwire.DecodeResponse[cleanupResult]( + context.Background(), + bytes.NewReader(encoded.Bytes()), + "jailer-cleanup", + ) + require.NoError(t, protocolErr) + require.NotNil(t, outcome.OperationError()) + details := outcome.OperationError().Details + assert.Equal(t, true, details["record_replaced"]) + assert.Equal(t, true, details["privileged_details_omitted"]) + assert.Len(t, details, 2) + }) + } +} + +func TestDecodeDomainErrorRejectsSensitiveDetailTokens(t *testing.T) { + tests := map[string]string{ + "raw_stderr": "raw_stderr", + "stderr_excerpt": "stderr_excerpt", + "process_stdout": "process.stdout", + "wrapped_cause": "wrapped-cause", + "nested_error": "process_error_message", + "stack_token": "handler_stack_trace", + "stacktrace_token": "HANDLER.STACKTRACE", + "err_token": "wrapped_err_text", + } + + for name, key := range tests { + t.Run(name, func(t *testing.T) { + errorJSON := `{"code":"vm.cleanup.failed","class":"internal","message":"failed",` + + `"operation":"vm.cleanup","entity":"vm-id","details":{"` + key + `":"secret"}}` + frame := responseFrame("MVMRES01", errorResponseBody(errorJSON)) + _, err := privilegedwire.DecodeResponse[cleanupResult]( + context.Background(), + bytes.NewReader(frame), + "jailer-cleanup", + ) + require.Error(t, err) + assert.ErrorContains(t, err, "detail key") + assert.Equal(t, errs.CodeValidationFailed, errs.AsDomainError(err).Code) + }) + } +} + +func TestDomainErrorResponsePreservesMaximumBoundedDetailSet(t *testing.T) { + details := make(map[string]any, 32) + for index := range 32 { + details[fmt.Sprintf("detail_%02d", index)] = int64(index) + } + source := &errs.DomainError{ + Code: errs.CodeProcessError, + Class: errs.ClassInternal, + Message: "operation failed", + Op: "vm.cleanup", + Details: details, + } + + var encoded bytes.Buffer + err := privilegedwire.EncodeError(context.Background(), &encoded, "jailer-cleanup", source) + require.NoError(t, err) + outcome, protocolErr := privilegedwire.DecodeResponse[cleanupResult]( + context.Background(), + bytes.NewReader(encoded.Bytes()), + "jailer-cleanup", + ) + require.NoError(t, protocolErr) + decoded := outcome.OperationError() + require.NotNil(t, decoded) + assert.Equal(t, details, decoded.Details) + _, omitted := decoded.Details["privileged_details_omitted"] + assert.False(t, omitted) +} + +func TestDomainErrorResponseBoundsExcessDetailCountVisibly(t *testing.T) { + details := make(map[string]any, 33) + for index := range 33 { + details[fmt.Sprintf("detail_%02d", index)] = int64(index) + } + source := &errs.DomainError{ + Code: errs.CodeProcessError, + Class: errs.ClassInternal, + Message: "operation failed", + Op: "vm.cleanup", + Details: details, + } + + var encoded bytes.Buffer + err := privilegedwire.EncodeError(context.Background(), &encoded, "jailer-cleanup", source) + require.NoError(t, err) + outcome, protocolErr := privilegedwire.DecodeResponse[cleanupResult]( + context.Background(), + bytes.NewReader(encoded.Bytes()), + "jailer-cleanup", + ) + require.NoError(t, protocolErr) + decoded := outcome.OperationError() + require.NotNil(t, decoded) + assert.Len(t, decoded.Details, 32) + assert.Equal(t, true, decoded.Details["privileged_details_omitted"]) + for index := range 31 { + assert.Equal(t, int64(index), decoded.Details[fmt.Sprintf("detail_%02d", index)]) + } + assert.NotContains(t, decoded.Details, "detail_31") + assert.NotContains(t, decoded.Details, "detail_32") +} + +func TestDomainErrorResponseBoundsWorkForLargeDetailMap(t *testing.T) { + details := make(map[string]any, 10_000) + for index := range 10_000 { + details[fmt.Sprintf("detail_%05d", index)] = true + } + source := &errs.DomainError{ + Code: errs.CodeProcessError, + Class: errs.ClassInternal, + Message: "operation failed", + Op: "vm.cleanup", + Details: details, + } + benchmark := testing.Benchmark(func(benchmark *testing.B) { + benchmark.ReportAllocs() + for range benchmark.N { + var encoded bytes.Buffer + err := privilegedwire.EncodeError(context.Background(), &encoded, "jailer-cleanup", source) + if err != nil { + benchmark.Fatal(err) + } + } + }) + allocatedBytes := benchmark.AllocedBytesPerOp() + t.Logf("large detail normalization allocated %d bytes per operation", allocatedBytes) + assert.Less(t, allocatedBytes, int64(64*1024)) + + var encodeErr error + allocations := testing.AllocsPerRun(1, func() { + var encoded bytes.Buffer + encodeErr = privilegedwire.EncodeError(context.Background(), &encoded, "jailer-cleanup", source) + }) + require.NoError(t, encodeErr) + assert.Less(t, allocations, 1000.0) +} + +func TestDomainErrorResponseRejectsUnicodeFormattingControls(t *testing.T) { + source := &errs.DomainError{ + Code: errs.CodeProcessError, + Class: errs.ClassInternal, + Message: "operation failed\u202esecret", + Op: "vm.cleanup", + Details: map[string]any{}, + } + + var encoded bytes.Buffer + err := privilegedwire.EncodeError(context.Background(), &encoded, "jailer-cleanup", source) + require.NoError(t, err) + assert.NotContains(t, encoded.String(), "secret") + + outcome, protocolErr := privilegedwire.DecodeResponse[cleanupResult]( + context.Background(), + bytes.NewReader(encoded.Bytes()), + "jailer-cleanup", + ) + require.NoError(t, protocolErr) + decoded := outcome.OperationError() + require.NotNil(t, decoded) + assert.Equal(t, errs.CodeInternal, decoded.Code) + assert.Equal(t, "privileged operation outcome could not be encoded", decoded.Message) +} + +func TestDomainErrorResponseFallsBackWhenAuthorityFieldsCannotFit(t *testing.T) { + hugeDetails := make(map[string]any, 31) + for index := range 31 { + hugeDetails[fmt.Sprintf("detail_%02d", index)] = strings.Repeat("a", 4096) + } + tests := map[string]error{ + "non_domain_error": errors.New("raw stderr must not cross the wire"), + "nil_error": nil, + "invalid_code": &errs.DomainError{ + Code: errs.Code("invalid/code"), Class: errs.ClassInternal, Message: "failed", + }, + "invalid_class": &errs.DomainError{ + Code: errs.CodeInternal, Class: errs.Class(99), Message: "failed", + }, + "oversized_message": &errs.DomainError{ + Code: errs.CodeInternal, Class: errs.ClassInternal, Message: strings.Repeat("a", 8193), + }, + "terminal_control": &errs.DomainError{ + Code: errs.CodeInternal, Class: errs.ClassInternal, Message: "failed\x1b[31m", + }, + "frame_too_large": &errs.DomainError{ + Code: errs.CodeInternal, Class: errs.ClassInternal, Message: "failed", Details: hugeDetails, + }, + } + + for name, source := range tests { + t.Run(name, func(t *testing.T) { + var encoded bytes.Buffer + err := privilegedwire.EncodeError(context.Background(), &encoded, "jailer-cleanup", source) + require.NoError(t, err) + assert.NotContains(t, encoded.String(), "raw stderr") + + outcome, protocolErr := privilegedwire.DecodeResponse[cleanupResult]( + context.Background(), + bytes.NewReader(encoded.Bytes()), + "jailer-cleanup", + ) + require.NoError(t, protocolErr) + decoded := outcome.OperationError() + require.NotNil(t, decoded) + assert.Equal(t, errs.CodeInternal, decoded.Code) + assert.Equal(t, errs.ClassInternal, decoded.Class) + assert.Equal(t, "privileged operation outcome could not be encoded", decoded.Message) + assert.Equal(t, "privileged", decoded.Op) + assert.Empty(t, decoded.Entity) + assert.Equal(t, map[string]any{"outcome_unknown": true}, decoded.Details) + }) + } +} + +func TestDomainErrorFallbackIsDeterministic(t *testing.T) { + var first bytes.Buffer + err := privilegedwire.EncodeError( + context.Background(), + &first, + "jailer-cleanup", + errors.New("first unsafe cause"), + ) + require.NoError(t, err) + + var second bytes.Buffer + err = privilegedwire.EncodeError( + context.Background(), + &second, + "jailer-cleanup", + errors.New("different unsafe cause"), + ) + require.NoError(t, err) + assert.Equal(t, first.Bytes(), second.Bytes()) +} + +func TestDecodeDomainErrorRejectsUnsafeWireValues(t *testing.T) { + base := `{"code":"vm.cleanup.failed","class":"internal","message":"failed",` + + `"operation":"vm.cleanup","entity":"vm-id","details":{}}` + tests := map[string]struct { + errorJSON string + wantErr string + }{ + "unknown_error_field": { + errorJSON: strings.Replace(base, `"details":{}`, `"details":{},"stderr":"secret"`, 1), + wantErr: "unknown JSON field", + }, + "missing_error_field": { + errorJSON: `{"code":"vm.cleanup.failed","class":"internal","message":"failed",` + + `"operation":"vm.cleanup","entity":"vm-id"}`, + wantErr: "missing required", + }, + "duplicate_nested_error_field": { + errorJSON: strings.Replace(base, `"code":"vm.cleanup.failed"`, + `"code":"vm.cleanup.failed","code":"vm.other"`, 1), + wantErr: "duplicate JSON field", + }, + "invalid_code": { + errorJSON: strings.Replace(base, "vm.cleanup.failed", "vm/cleanup", 1), + wantErr: "error code", + }, + "oversized_code": { + errorJSON: strings.Replace(base, "vm.cleanup.failed", strings.Repeat("a", 129), 1), + wantErr: "error code", + }, + "wrong_class": { + errorJSON: strings.Replace(base, `"class":"internal"`, `"class":"fatal"`, 1), + wantErr: "error class", + }, + "message_control": { + errorJSON: strings.Replace(base, `"message":"failed"`, `"message":"failed\u001b"`, 1), + wantErr: "error message", + }, + "message_unicode_format_control": { + errorJSON: strings.Replace(base, `"message":"failed"`, `"message":"failed\u202e"`, 1), + wantErr: "error message", + }, + "oversized_message": { + errorJSON: strings.Replace( + base, + `"message":"failed"`, + `"message":"`+strings.Repeat("a", 8193)+`"`, + 1, + ), + wantErr: "error message", + }, + "operation_control": { + errorJSON: strings.Replace( + base, + `"operation":"vm.cleanup"`, + `"operation":"vm.cleanup\n"`, + 1, + ), + wantErr: "error operation", + }, + "oversized_operation": { + errorJSON: strings.Replace( + base, + `"operation":"vm.cleanup"`, + `"operation":"`+strings.Repeat("a", 129)+`"`, + 1, + ), + wantErr: "error operation", + }, + "oversized_entity": { + errorJSON: strings.Replace(base, "vm-id", strings.Repeat("a", 1025), 1), + wantErr: "error entity", + }, + "details_null": { + errorJSON: strings.Replace(base, `"details":{}`, `"details":{"bad":null}`, 1), + wantErr: "detail value", + }, + "omitted_flag_must_be_true": { + errorJSON: strings.Replace( + base, + `"details":{}`, + `"details":{"privileged_details_omitted":false}`, + 1, + ), + wantErr: "omission flag", + }, + "details_float": { + errorJSON: strings.Replace(base, `"details":{}`, `"details":{"bad":1.25}`, 1), + wantErr: "integer detail", + }, + "details_integer_overflow": { + errorJSON: strings.Replace( + base, + `"details":{}`, + `"details":{"bad":9223372036854775808}`, + 1, + ), + wantErr: "integer detail", + }, + "detail_key_oversized": { + errorJSON: strings.Replace( + base, + `"details":{}`, + `"details":{"`+strings.Repeat("a", 65)+`":true}`, + 1, + ), + wantErr: "detail key", + }, + "detail_key_space_separator": { + errorJSON: strings.Replace( + base, + `"details":{}`, + `"details":{"stderr excerpt":"secret"}`, + 1, + ), + wantErr: "detail key", + }, + "detail_key_non_ascii": { + errorJSON: strings.Replace(base, `"details":{}`, `"details":{"phase_é":true}`, 1), + wantErr: "detail key", + }, + "details_nested": { + errorJSON: strings.Replace(base, `"details":{}`, `"details":{"bad":{"command":"id"}}`, 1), + wantErr: "detail value", + }, + "details_arbitrary_array": { + errorJSON: strings.Replace(base, `"details":{}`, `"details":{"bad":[1,2]}`, 1), + wantErr: "string array", + }, + "details_too_many": { + errorJSON: strings.Replace(base, `"details":{}`, `"details":`+objectWithFields(33), 1), + wantErr: "exceed 32 entries", + }, + "detail_string_oversized": { + errorJSON: strings.Replace( + base, + `"details":{}`, + `"details":{"bad":"`+strings.Repeat("a", 4097)+`"}`, + 1, + ), + wantErr: "error detail", + }, + "detail_array_too_many": { + errorJSON: strings.Replace( + base, + `"details":{}`, + `"details":{"bad":[`+strings.Repeat(`"a",`, 32)+`"a"]}`, + 1, + ), + wantErr: "array exceeds 32 items", + }, + "detail_array_item_oversized": { + errorJSON: strings.Replace( + base, + `"details":{}`, + `"details":{"bad":["`+strings.Repeat("a", 1025)+`"]}`, + 1, + ), + wantErr: "array item", + }, + } + + for name, tc := range tests { + t.Run(name, func(t *testing.T) { + frame := responseFrame("MVMRES01", errorResponseBody(tc.errorJSON)) + _, err := privilegedwire.DecodeResponse[cleanupResult]( + context.Background(), + bytes.NewReader(frame), + "jailer-cleanup", + ) + require.Error(t, err) + assert.ErrorContains(t, err, tc.wantErr) + assert.Equal(t, errs.CodeValidationFailed, errs.AsDomainError(err).Code) + }) + } +} diff --git a/internal/lib/privilegedwire/frame.go b/internal/lib/privilegedwire/frame.go new file mode 100644 index 00000000..0e89d7e8 --- /dev/null +++ b/internal/lib/privilegedwire/frame.go @@ -0,0 +1,102 @@ +package privilegedwire + +import ( + "context" + "encoding/binary" + "fmt" + "io" + + "mvmctl/pkg/errs" +) + +const ( + requestMagic = "MVMREQ01" + responseMagic = "MVMRES01" + maxJSONFrameBytes = 64 * 1024 + maxActionBytes = 64 +) + +func writeFrame(ctx context.Context, writer io.Writer, prefix []byte, body []byte) error { + if err := ctx.Err(); err != nil { + return errs.WrapMsg(errs.CodeProcessError, "write privileged frame", err) + } + if writer == nil { + return errs.New(errs.CodeValidationFailed, "privileged frame writer is required") + } + if err := writeAll(ctx, writer, prefix); err != nil { + return errs.WrapMsg(errs.CodeProcessError, "write privileged frame prefix", err) + } + if err := writeAll(ctx, writer, body); err != nil { + return errs.WrapMsg(errs.CodeProcessError, "write privileged frame body", err) + } + return nil +} + +func writeAll(ctx context.Context, writer io.Writer, value []byte) error { + for len(value) > 0 { + if err := ctx.Err(); err != nil { + return err + } + written, err := writer.Write(value) + if err != nil { + return err + } + if written <= 0 || written > len(value) { + return io.ErrShortWrite + } + value = value[written:] + } + return nil +} + +func readExact(ctx context.Context, reader io.Reader, value []byte, description string) error { + if err := ctx.Err(); err != nil { + return errs.WrapMsg(errs.CodeProcessError, "read privileged frame", err) + } + if reader == nil { + return errs.New(errs.CodeValidationFailed, "privileged frame reader is required") + } + if _, err := io.ReadFull(reader, value); err != nil { + return errs.WrapMsg(errs.CodeValidationFailed, "read privileged "+description, err) + } + if err := ctx.Err(); err != nil { + return errs.WrapMsg(errs.CodeProcessError, "read privileged frame", err) + } + return nil +} + +func requireReaderEOF(ctx context.Context, reader io.Reader) error { + var trailing [1]byte + read, err := reader.Read(trailing[:]) + if contextErr := ctx.Err(); contextErr != nil { + return errs.WrapMsg(errs.CodeProcessError, "read privileged frame", contextErr) + } + if read > 0 || err == nil { + return errs.New(errs.CodeValidationFailed, "privileged response contains trailing input") + } + if err != io.EOF { + return errs.WrapMsg(errs.CodeValidationFailed, "read trailing privileged response input", err) + } + return nil +} + +func validateFrameLength(length uint32) error { + if length == 0 { + return errs.New(errs.CodeValidationFailed, "privileged JSON frame is empty") + } + if length > maxJSONFrameBytes { + return errs.New( + errs.CodeValidationFailed, + fmt.Sprintf("privileged JSON frame exceeds %d bytes", maxJSONFrameBytes), + ) + } + return nil +} + +func appendUint32(target []byte, value uint32) { + binary.BigEndian.PutUint32(target, value) +} + +func appendUint64(target []byte, value uint64) { + binary.BigEndian.PutUint64(target, value) +} diff --git a/internal/lib/privilegedwire/json.go b/internal/lib/privilegedwire/json.go new file mode 100644 index 00000000..1ef3931e --- /dev/null +++ b/internal/lib/privilegedwire/json.go @@ -0,0 +1,279 @@ +package privilegedwire + +import ( + "bytes" + "encoding/json" + "fmt" + "io" + "strings" + "unicode/utf8" + + "mvmctl/pkg/errs" +) + +const ( + maxJSONDepth = 32 + maxJSONObjectFields = 64 + maxJSONArrayItems = 1024 +) + +func validateStrictJSON(raw []byte) error { + if !utf8.Valid(raw) { + return errs.New(errs.CodeValidationFailed, "privileged JSON must use valid UTF-8") + } + decoder := json.NewDecoder(bytes.NewReader(raw)) + decoder.UseNumber() + if err := scanJSONValue(decoder, 0); err != nil { + if errs.AsDomainError(err) != nil { + return err + } + return errs.WrapMsg(errs.CodeValidationFailed, "decode privileged JSON", err) + } + return requireJSONEOF(decoder) +} + +func scanJSONValue(decoder *json.Decoder, depth int) error { + token, err := decoder.Token() + if err != nil { + return err + } + delimiter, ok := token.(json.Delim) + if !ok { + return nil + } + if depth >= maxJSONDepth { + return errs.New( + errs.CodeValidationFailed, + fmt.Sprintf("privileged JSON nesting exceeds %d levels", maxJSONDepth), + ) + } + + switch delimiter { + case '{': + return scanJSONObject(decoder, depth) + case '[': + return scanJSONArray(decoder, depth) + default: + return fmt.Errorf("unexpected JSON delimiter %q", delimiter) + } +} + +func scanJSONObject(decoder *json.Decoder, depth int) error { + seen := make(map[string]struct{}) + fieldCount := 0 + for decoder.More() { + fieldCount++ + if fieldCount > maxJSONObjectFields { + return errs.New( + errs.CodeValidationFailed, + fmt.Sprintf("privileged JSON object exceeds %d fields", maxJSONObjectFields), + ) + } + fieldToken, err := decoder.Token() + if err != nil { + return err + } + field, ok := fieldToken.(string) + if !ok { + return fmt.Errorf("JSON object field is not a string") + } + if _, exists := seen[field]; exists { + return errs.New( + errs.CodeValidationFailed, + fmt.Sprintf("duplicate JSON field %q in privileged frame", field), + ) + } + for existing := range seen { + if strings.EqualFold(existing, field) { + return errs.New( + errs.CodeValidationFailed, + fmt.Sprintf( + "case-folded duplicate JSON field %q conflicts with %q in privileged frame", + field, + existing, + ), + ) + } + } + seen[field] = struct{}{} + if err := scanJSONValue(decoder, depth+1); err != nil { + return err + } + } + _, err := decoder.Token() + return err +} + +func scanJSONArray(decoder *json.Decoder, depth int) error { + itemCount := 0 + for decoder.More() { + itemCount++ + if itemCount > maxJSONArrayItems { + return errs.New( + errs.CodeValidationFailed, + fmt.Sprintf("privileged JSON array exceeds %d items", maxJSONArrayItems), + ) + } + if err := scanJSONValue(decoder, depth+1); err != nil { + return err + } + } + _, err := decoder.Token() + return err +} + +func decodeExactObject( + raw []byte, + allowed map[string]struct{}, + required []string, + description string, +) (map[string]json.RawMessage, error) { + var fields map[string]json.RawMessage + if err := json.Unmarshal(raw, &fields); err != nil { + return nil, errs.WrapMsg(errs.CodeValidationFailed, "decode privileged "+description, err) + } + if fields == nil { + return nil, errs.New(errs.CodeValidationFailed, "privileged "+description+" must be a JSON object") + } + for field := range fields { + if _, ok := allowed[field]; !ok { + return nil, errs.New( + errs.CodeValidationFailed, + fmt.Sprintf("unknown JSON field %q in privileged %s", field, description), + ) + } + } + for _, field := range required { + if _, ok := fields[field]; !ok { + return nil, errs.New( + errs.CodeValidationFailed, + fmt.Sprintf("privileged %s is missing required field %q", description, field), + ) + } + } + return fields, nil +} + +func decodeJSONField[T any](raw json.RawMessage, description string) (T, error) { + var zero T + decoder := json.NewDecoder(bytes.NewReader(raw)) + decoder.DisallowUnknownFields() + decoder.UseNumber() + var value T + if err := decoder.Decode(&value); err != nil { + return zero, errs.WrapMsg( + errs.CodeValidationFailed, + "decode privileged "+description+": "+err.Error(), + err, + ) + } + if err := requireJSONEOF(decoder); err != nil { + return zero, err + } + return value, nil +} + +func decodeCanonicalJSONField[T any](raw json.RawMessage, description string) (T, error) { + var zero T + value, err := decodeJSONField[T](raw, description) + if err != nil { + return zero, err + } + canonical, err := json.Marshal(value) + if err != nil { + return zero, errs.WrapMsg(errs.CodeValidationFailed, "encode canonical privileged "+description, err) + } + if err := requireCanonicalJSONMemberNames(raw, canonical, description); err != nil { + return zero, err + } + return value, nil +} + +func requireCanonicalJSONMemberNames(raw json.RawMessage, canonical json.RawMessage, description string) error { + raw = bytes.TrimSpace(raw) + canonical = bytes.TrimSpace(canonical) + if len(raw) == 0 || len(canonical) == 0 { + return errs.New(errs.CodeValidationFailed, "privileged "+description+" is empty") + } + if raw[0] == '{' && canonical[0] == '{' { + return requireCanonicalJSONObjectNames(raw, canonical, description) + } + if raw[0] == '[' && canonical[0] == '[' { + return requireCanonicalJSONArrayNames(raw, canonical, description) + } + return nil +} + +func requireCanonicalJSONObjectNames(raw json.RawMessage, canonical json.RawMessage, description string) error { + var rawFields map[string]json.RawMessage + if err := json.Unmarshal(raw, &rawFields); err != nil { + return errs.WrapMsg(errs.CodeValidationFailed, "decode privileged "+description, err) + } + var canonicalFields map[string]json.RawMessage + if err := json.Unmarshal(canonical, &canonicalFields); err != nil { + return errs.WrapMsg(errs.CodeInternal, "decode canonical privileged "+description, err) + } + for field, rawValue := range rawFields { + canonicalValue, ok := canonicalFields[field] + if !ok { + return errs.New( + errs.CodeValidationFailed, + fmt.Sprintf("unknown JSON field %q in privileged %s", field, description), + ) + } + if err := requireCanonicalJSONMemberNames(rawValue, canonicalValue, description); err != nil { + return err + } + } + for field := range canonicalFields { + if _, ok := rawFields[field]; !ok { + return errs.New( + errs.CodeValidationFailed, + fmt.Sprintf("privileged %s is missing required field %q", description, field), + ) + } + } + return nil +} + +func requireCanonicalJSONArrayNames(raw json.RawMessage, canonical json.RawMessage, description string) error { + var rawItems []json.RawMessage + if err := json.Unmarshal(raw, &rawItems); err != nil { + return errs.WrapMsg(errs.CodeValidationFailed, "decode privileged "+description, err) + } + var canonicalItems []json.RawMessage + if err := json.Unmarshal(canonical, &canonicalItems); err != nil { + return errs.WrapMsg(errs.CodeInternal, "decode canonical privileged "+description, err) + } + if len(rawItems) != len(canonicalItems) { + return errs.New(errs.CodeValidationFailed, "privileged "+description+" changed array shape during decoding") + } + for index := range rawItems { + if err := requireCanonicalJSONMemberNames(rawItems[index], canonicalItems[index], description); err != nil { + return err + } + } + return nil +} + +func requireJSONEOF(decoder *json.Decoder) error { + if _, err := decoder.Token(); err != io.EOF { + if err == nil { + return errs.New(errs.CodeValidationFailed, "privileged frame contains trailing JSON") + } + return errs.WrapMsg(errs.CodeValidationFailed, "decode trailing privileged JSON", err) + } + return nil +} + +func isJSONNull(raw json.RawMessage) bool { + return bytes.Equal(bytes.TrimSpace(raw), []byte("null")) +} + +func fieldSet(names ...string) map[string]struct{} { + result := make(map[string]struct{}, len(names)) + for _, name := range names { + result[name] = struct{}{} + } + return result +} diff --git a/internal/lib/privilegedwire/request.go b/internal/lib/privilegedwire/request.go new file mode 100644 index 00000000..67209d6e --- /dev/null +++ b/internal/lib/privilegedwire/request.go @@ -0,0 +1,146 @@ +package privilegedwire + +import ( + "context" + "encoding/binary" + "encoding/json" + "io" + + "mvmctl/pkg/errs" +) + +const requestPrefixBytes = len(requestMagic) + 4 + 8 + +// Request is one decoded typed request header and its separately framed payload length. +// T is the action-specific body type selected by the caller's named method. +type Request[T any] struct { + Body T + PayloadLength uint64 +} + +type requestEnvelope[T any] struct { + SchemaVersion uint32 `json:"schema_version"` + Action string `json:"action"` + Body T `json:"body"` +} + +var requestEnvelopeFields = fieldSet("schema_version", "action", "body") + +// EncodeRequest writes one version-1 request header. It does not write the separately declared payload. +func EncodeRequest[T any]( + ctx context.Context, + writer io.Writer, + action string, + body T, + payloadLength uint64, +) error { + if err := validateAction(action); err != nil { + return err + } + encodedBody, err := json.Marshal(body) + if err != nil { + return errs.WrapMsg(errs.CodeValidationFailed, "encode privileged request body", err) + } + if isJSONNull(encodedBody) { + return errs.New(errs.CodeValidationFailed, "privileged request body must not be null") + } + header, err := json.Marshal(requestEnvelope[json.RawMessage]{ + SchemaVersion: 1, + Action: action, + Body: encodedBody, + }) + if err != nil { + return errs.WrapMsg(errs.CodeValidationFailed, "encode privileged request header", err) + } + if len(header) > maxJSONFrameBytes { + return errs.New(errs.CodeValidationFailed, "privileged request header exceeds 65536 bytes") + } + if err := validateStrictJSON(header); err != nil { + return err + } + + prefix := make([]byte, requestPrefixBytes) + copy(prefix, requestMagic) + appendUint32(prefix[len(requestMagic):], uint32(len(header))) + appendUint64(prefix[len(requestMagic)+4:], payloadLength) + return writeFrame(ctx, writer, prefix, header) +} + +// DecodeRequest reads one version-1 typed request header and leaves any declared payload unread. +func DecodeRequest[T any]( + ctx context.Context, + reader io.Reader, + expectedAction string, + maxPayloadLength uint64, +) (Request[T], error) { + var zero Request[T] + if err := validateAction(expectedAction); err != nil { + return zero, err + } + prefix := make([]byte, requestPrefixBytes) + if err := readExact(ctx, reader, prefix, "request prefix"); err != nil { + return zero, err + } + if string(prefix[:len(requestMagic)]) != requestMagic { + return zero, errs.New(errs.CodeValidationFailed, "invalid privileged request magic") + } + headerLength := binary.BigEndian.Uint32(prefix[len(requestMagic) : len(requestMagic)+4]) + if err := validateFrameLength(headerLength); err != nil { + return zero, err + } + payloadLength := binary.BigEndian.Uint64(prefix[len(requestMagic)+4:]) + if payloadLength > maxPayloadLength { + return zero, errs.New(errs.CodeValidationFailed, "privileged request payload length exceeds receiver limit") + } + header := make([]byte, headerLength) + if err := readExact(ctx, reader, header, "request header"); err != nil { + return zero, err + } + if err := validateStrictJSON(header); err != nil { + return zero, err + } + fields, err := decodeExactObject( + header, + requestEnvelopeFields, + []string{"schema_version", "action", "body"}, + "request header", + ) + if err != nil { + return zero, err + } + schemaVersion, err := decodeJSONField[uint32](fields["schema_version"], "request schema version") + if err != nil { + return zero, err + } + if schemaVersion != 1 { + return zero, errs.New(errs.CodeValidationFailed, "unsupported privileged request schema version") + } + action, err := decodeJSONField[string](fields["action"], "request action") + if err != nil { + return zero, err + } + if err := validateAction(action); err != nil { + return zero, err + } + if action != expectedAction { + return zero, errs.New(errs.CodeValidationFailed, "privileged request action does not match argv action") + } + if isJSONNull(fields["body"]) { + return zero, errs.New(errs.CodeValidationFailed, "privileged request body must not be null") + } + body, err := decodeCanonicalJSONField[T](fields["body"], "request body") + if err != nil { + return zero, err + } + return Request[T]{ + Body: body, + PayloadLength: payloadLength, + }, nil +} + +func validateAction(action string) error { + if !validASCIIIdentifier(action, maxActionBytes) { + return errs.New(errs.CodeValidationFailed, "privileged action must be a 1-64 byte ASCII identifier") + } + return nil +} diff --git a/internal/lib/privilegedwire/request_test.go b/internal/lib/privilegedwire/request_test.go new file mode 100644 index 00000000..d5e78bc4 --- /dev/null +++ b/internal/lib/privilegedwire/request_test.go @@ -0,0 +1,353 @@ +package privilegedwire_test + +import ( + "bytes" + "context" + "encoding/binary" + "fmt" + "io" + "strings" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "mvmctl/internal/lib/privilegedwire" + "mvmctl/pkg/errs" +) + +type cleanupRequest struct { + VMID string `json:"vm_id"` +} + +func TestRequestFrameRoundTrip(t *testing.T) { + var encoded bytes.Buffer + err := privilegedwire.EncodeRequest( + context.Background(), + &encoded, + "jailer-cleanup", + cleanupRequest{VMID: "0123456789abcdef"}, + 4096, + ) + require.NoError(t, err) + + raw := encoded.Bytes() + require.GreaterOrEqual(t, len(raw), 20) + assert.Equal(t, []byte("MVMREQ01"), raw[:8]) + assert.Equal(t, uint32(len(raw)-20), binary.BigEndian.Uint32(raw[8:12])) + assert.Equal(t, uint64(4096), binary.BigEndian.Uint64(raw[12:20])) + + decoded, err := privilegedwire.DecodeRequest[cleanupRequest]( + context.Background(), + bytes.NewReader(raw), + "jailer-cleanup", + 4096, + ) + require.NoError(t, err) + assert.Equal(t, cleanupRequest{VMID: "0123456789abcdef"}, decoded.Body) + assert.Equal(t, uint64(4096), decoded.PayloadLength) +} + +func TestRequestFrameRejectsInvalidAction(t *testing.T) { + tests := map[string]string{ + "empty": "", + "oversized": strings.Repeat("a", 65), + "path": "jailer/cleanup", + "whitespace": "jailer cleanup", + "non_ascii": "jailer-cléanup", + "terminal_byte": "jailer\ncleanup", + "leading_punctuation": "-jailer-cleanup", + } + + for name, action := range tests { + t.Run(name, func(t *testing.T) { + var encoded bytes.Buffer + err := privilegedwire.EncodeRequest( + context.Background(), + &encoded, + action, + cleanupRequest{VMID: "0123456789abcdef"}, + 0, + ) + require.Error(t, err) + assert.Equal(t, errs.CodeValidationFailed, errs.AsDomainError(err).Code) + assert.Empty(t, encoded.Bytes()) + }) + } +} + +func TestDecodeRequestRejectsMalformedOrNonCanonicalFrame(t *testing.T) { + invalidUTF8Header := append( + []byte(`{"schema_version":1,"action":"jailer-cleanup","body":{"vm_id":"`), + 0xff, + ) + invalidUTF8Header = append(invalidUTF8Header, []byte(`"}}`)...) + tests := map[string]struct { + frame []byte + action string + wantErr string + }{ + "bad_magic": { + frame: requestFrame("BADMAGIC", `{"schema_version":1,"action":"jailer-cleanup","body":{"vm_id":"a"}}`, 0), + action: "jailer-cleanup", + wantErr: "magic", + }, + "zero_header_length": { + frame: requestFrameWithLength("MVMREQ01", 0, nil, 0), + action: "jailer-cleanup", + wantErr: "empty", + }, + "oversized_header_length": { + frame: requestFrameWithLength("MVMREQ01", 65537, nil, 0), + action: "jailer-cleanup", + wantErr: "exceeds", + }, + "payload_exceeds_receiver_limit": { + frame: requestFrame( + "MVMREQ01", + `{"schema_version":1,"action":"jailer-cleanup","body":{"vm_id":"a"}}`, + 1, + ), + action: "jailer-cleanup", + wantErr: "payload length exceeds", + }, + "truncated_header": { + frame: requestFrameWithLength( + "MVMREQ01", + 64, + []byte(`{"schema_version":1}`), + 0, + ), + action: "jailer-cleanup", + wantErr: "request header", + }, + "unsupported_schema": { + frame: requestFrame("MVMREQ01", `{"schema_version":2,"action":"jailer-cleanup","body":{"vm_id":"a"}}`, 0), + action: "jailer-cleanup", + wantErr: "schema version", + }, + "invalid_utf8": { + frame: requestFrameWithLength( + "MVMREQ01", + uint32(len(invalidUTF8Header)), + invalidUTF8Header, + 0, + ), + action: "jailer-cleanup", + wantErr: "UTF-8", + }, + "action_mismatch": { + frame: requestFrame("MVMREQ01", `{"schema_version":1,"action":"jailer-remove","body":{"vm_id":"a"}}`, 0), + action: "jailer-cleanup", + wantErr: "does not match", + }, + "unknown_envelope_field": { + frame: requestFrame( + "MVMREQ01", + `{"schema_version":1,"action":"jailer-cleanup","body":{"vm_id":"a"},"command":"id"}`, + 0, + ), + action: "jailer-cleanup", + wantErr: "unknown JSON field", + }, + "case_variant_schema_field": { + frame: requestFrame( + "MVMREQ01", + `{"SCHEMA_VERSION":1,"action":"jailer-cleanup","body":{"vm_id":"a"}}`, + 0, + ), + action: "jailer-cleanup", + wantErr: "unknown JSON field", + }, + "missing_body": { + frame: requestFrame("MVMREQ01", `{"schema_version":1,"action":"jailer-cleanup"}`, 0), + action: "jailer-cleanup", + wantErr: "missing required", + }, + "null_body": { + frame: requestFrame("MVMREQ01", `{"schema_version":1,"action":"jailer-cleanup","body":null}`, 0), + action: "jailer-cleanup", + wantErr: "must not be null", + }, + "duplicate_envelope_field": { + frame: requestFrame( + "MVMREQ01", + `{"schema_version":1,"action":"jailer-cleanup","action":"jailer-remove","body":{"vm_id":"a"}}`, + 0, + ), + action: "jailer-cleanup", + wantErr: "duplicate JSON field", + }, + "case_folded_duplicate_nested_field": { + frame: requestFrame( + "MVMREQ01", + `{"schema_version":1,"action":"jailer-cleanup","body":{"vm_id":"a","VM_ID":"b"}}`, + 0, + ), + action: "jailer-cleanup", + wantErr: "case-folded duplicate JSON field", + }, + "single_case_variant_body_field": { + frame: requestFrame( + "MVMREQ01", + `{"schema_version":1,"action":"jailer-cleanup","body":{"VM_ID":"a"}}`, + 0, + ), + action: "jailer-cleanup", + wantErr: "unknown JSON field", + }, + "unknown_body_field": { + frame: requestFrame( + "MVMREQ01", + `{"schema_version":1,"action":"jailer-cleanup","body":{"vm_id":"a","path":"/root"}}`, + 0, + ), + action: "jailer-cleanup", + wantErr: "unknown field", + }, + "trailing_header_json": { + frame: requestFrame( + "MVMREQ01", + `{"schema_version":1,"action":"jailer-cleanup","body":{"vm_id":"a"}} {}`, + 0, + ), + action: "jailer-cleanup", + wantErr: "trailing JSON", + }, + "excessive_depth": { + frame: requestFrame( + "MVMREQ01", + `{"schema_version":1,"action":"jailer-cleanup","body":`+ + strings.Repeat("[", 33)+"0"+strings.Repeat("]", 33)+`}`, + 0, + ), + action: "jailer-cleanup", + wantErr: "nesting exceeds", + }, + "excessive_object_fields": { + frame: requestFrame( + "MVMREQ01", + `{"schema_version":1,"action":"jailer-cleanup","body":`+objectWithFields(65)+`}`, + 0, + ), + action: "jailer-cleanup", + wantErr: "object exceeds", + }, + "excessive_array_items": { + frame: requestFrame( + "MVMREQ01", + `{"schema_version":1,"action":"jailer-cleanup","body":[`+ + strings.Repeat(`"a",`, 1024)+`"a"]}`, + 0, + ), + action: "jailer-cleanup", + wantErr: "array exceeds", + }, + } + + for name, tc := range tests { + t.Run(name, func(t *testing.T) { + _, err := privilegedwire.DecodeRequest[cleanupRequest]( + context.Background(), + bytes.NewReader(tc.frame), + tc.action, + 0, + ) + require.Error(t, err) + assert.ErrorContains(t, err, tc.wantErr) + assert.Equal(t, errs.CodeValidationFailed, errs.AsDomainError(err).Code) + }) + } +} + +func TestRequestFrameHonorsContextAndWriteFailures(t *testing.T) { + canceled, cancel := context.WithCancel(context.Background()) + cancel() + + var encoded bytes.Buffer + err := privilegedwire.EncodeRequest( + canceled, + &encoded, + "jailer-cleanup", + cleanupRequest{VMID: "a"}, + 0, + ) + require.ErrorIs(t, err, context.Canceled) + assert.Empty(t, encoded.Bytes()) + + err = privilegedwire.EncodeRequest( + context.Background(), + shortWriter{}, + "jailer-cleanup", + cleanupRequest{VMID: "a"}, + 0, + ) + require.Error(t, err) + assert.ErrorIs(t, err, io.ErrShortWrite) +} + +func TestEncodeRequestRejectsNullOrOversizedBodyBeforeWriting(t *testing.T) { + tests := map[string]struct { + body *cleanupRequest + wantErr string + }{ + "null": { + wantErr: "must not be null", + }, + "oversized": { + body: &cleanupRequest{VMID: strings.Repeat("a", 65536)}, + wantErr: "exceeds", + }, + } + + for name, tc := range tests { + t.Run(name, func(t *testing.T) { + var encoded bytes.Buffer + err := privilegedwire.EncodeRequest( + context.Background(), + &encoded, + "jailer-cleanup", + tc.body, + 0, + ) + require.Error(t, err) + assert.ErrorContains(t, err, tc.wantErr) + assert.Empty(t, encoded.Bytes()) + }) + } +} + +type shortWriter struct{} + +func (shortWriter) Write(value []byte) (int, error) { + if len(value) == 0 { + return 0, nil + } + return 0, nil +} + +func requestFrame(magic string, header string, payloadLength uint64) []byte { + return requestFrameWithLength(magic, uint32(len(header)), []byte(header), payloadLength) +} + +func requestFrameWithLength(magic string, headerLength uint32, header []byte, payloadLength uint64) []byte { + frame := make([]byte, 20+len(header)) + copy(frame, magic) + binary.BigEndian.PutUint32(frame[8:12], headerLength) + binary.BigEndian.PutUint64(frame[12:20], payloadLength) + copy(frame[20:], header) + return frame +} + +func objectWithFields(count int) string { + var value strings.Builder + value.WriteByte('{') + for index := range count { + if index > 0 { + value.WriteByte(',') + } + value.WriteString(fmt.Sprintf(`"field%d":%d`, index, index)) + } + value.WriteByte('}') + return value.String() +} diff --git a/internal/lib/privilegedwire/response.go b/internal/lib/privilegedwire/response.go new file mode 100644 index 00000000..abe29b66 --- /dev/null +++ b/internal/lib/privilegedwire/response.go @@ -0,0 +1,239 @@ +package privilegedwire + +import ( + "context" + "encoding/binary" + "encoding/json" + "io" + + "mvmctl/pkg/errs" +) + +const responsePrefixBytes = len(responseMagic) + 4 + +const ( + responseStatusSuccess = "success" + responseStatusError = "error" +) + +type successEnvelope[T any] struct { + SchemaVersion uint32 `json:"schema_version"` + Action string `json:"action"` + Status string `json:"status"` + Result T `json:"result"` +} + +type errorEnvelope struct { + SchemaVersion uint32 `json:"schema_version"` + Action string `json:"action"` + Status string `json:"status"` + Error domainErrorWire `json:"error"` +} + +var responseEnvelopeFields = fieldSet("schema_version", "action", "status", "result", "error") + +// Response is one successfully decoded privileged response envelope. OperationError is non-nil +// for an error envelope; otherwise Result is authoritative. DecodeResponse returns a separate +// error only when the response itself cannot be trusted as protocol input. +type Response[T any] struct { + result T + operationError *errs.DomainError +} + +// Result returns the typed result carried by a success envelope. +func (response Response[T]) Result() T { + return response.result +} + +// OperationError returns the preserved DomainError carried by an error envelope. +func (response Response[T]) OperationError() *errs.DomainError { + return response.operationError +} + +// EncodeSuccess writes one version-1 success response containing an action-specific typed result. +func EncodeSuccess[T any](ctx context.Context, writer io.Writer, action string, result T) error { + if err := validateAction(action); err != nil { + return err + } + encodedResult, err := json.Marshal(result) + if err != nil { + return errs.WrapMsg(errs.CodeValidationFailed, "encode privileged success result", err) + } + if isJSONNull(encodedResult) { + return errs.New(errs.CodeValidationFailed, "privileged success result must not be null") + } + body, err := json.Marshal(successEnvelope[json.RawMessage]{ + SchemaVersion: 1, + Action: action, + Status: responseStatusSuccess, + Result: encodedResult, + }) + if err != nil { + return errs.WrapMsg(errs.CodeValidationFailed, "encode privileged success response", err) + } + if len(body) > maxJSONFrameBytes { + return errs.New(errs.CodeValidationFailed, "privileged success response exceeds 65536 bytes") + } + if err := validateStrictJSON(body); err != nil { + return err + } + + prefix := make([]byte, responsePrefixBytes) + copy(prefix, responseMagic) + appendUint32(prefix[len(responseMagic):], uint32(len(body))) + return writeFrame(ctx, writer, prefix, body) +} + +// EncodeError writes one version-1 error response after normalizing a DomainError to the bounded wire schema. +func EncodeError(ctx context.Context, writer io.Writer, action string, operationErr error) error { + if err := validateAction(action); err != nil { + return err + } + envelope := errorEnvelope{ + SchemaVersion: 1, + Action: action, + Status: responseStatusError, + Error: normalizeDomainError(operationErr), + } + body, err := json.Marshal(envelope) + if err != nil { + return errs.WrapMsg(errs.CodeInternal, "encode privileged error response", err) + } + if len(body) > maxJSONFrameBytes { + envelope.Error = fallbackDomainError() + body, err = json.Marshal(envelope) + if err != nil { + return errs.WrapMsg(errs.CodeInternal, "encode fallback privileged error response", err) + } + } + + prefix := make([]byte, responsePrefixBytes) + copy(prefix, responseMagic) + appendUint32(prefix[len(responseMagic):], uint32(len(body))) + return writeFrame(ctx, writer, prefix, body) +} + +// DecodeResponse reads one final version-1 response, separating a preserved operation error from protocol failures. +func DecodeResponse[T any](ctx context.Context, reader io.Reader, expectedAction string) (Response[T], error) { + var zero Response[T] + if err := validateAction(expectedAction); err != nil { + return zero, err + } + body, err := readResponseFrame(ctx, reader) + if err != nil { + return zero, err + } + if err := validateStrictJSON(body); err != nil { + return zero, err + } + fields, err := decodeExactObject( + body, + responseEnvelopeFields, + []string{"schema_version", "action", "status"}, + "response", + ) + if err != nil { + return zero, err + } + if err := validateResponseIdentity(fields, expectedAction); err != nil { + return zero, err + } + status, err := decodeJSONField[string](fields["status"], "response status") + if err != nil { + return zero, err + } + switch status { + case responseStatusSuccess: + result, decodeErr := decodeSuccessResult[T](fields) + if decodeErr != nil { + return zero, decodeErr + } + return Response[T]{result: result}, nil + case responseStatusError: + operationErr, decodeErr := decodeErrorResult(fields) + if decodeErr != nil { + return zero, decodeErr + } + return Response[T]{operationError: operationErr}, nil + default: + return zero, errs.New(errs.CodeValidationFailed, "invalid privileged response status") + } +} + +func decodeErrorResult(fields map[string]json.RawMessage) (*errs.DomainError, error) { + rawError, hasError := fields["error"] + _, hasResult := fields["result"] + if !hasError || hasResult { + return nil, errs.New( + errs.CodeValidationFailed, + "privileged error response must contain only one error", + ) + } + if isJSONNull(rawError) { + return nil, errs.New(errs.CodeValidationFailed, "privileged error response must not be null") + } + domainErr, err := decodeDomainError(rawError) + if err != nil { + return nil, err + } + return domainErr, nil +} + +func readResponseFrame(ctx context.Context, reader io.Reader) ([]byte, error) { + prefix := make([]byte, responsePrefixBytes) + if err := readExact(ctx, reader, prefix, "response prefix"); err != nil { + return nil, err + } + if string(prefix[:len(responseMagic)]) != responseMagic { + return nil, errs.New(errs.CodeValidationFailed, "invalid privileged response magic") + } + bodyLength := binary.BigEndian.Uint32(prefix[len(responseMagic):]) + if err := validateFrameLength(bodyLength); err != nil { + return nil, err + } + body := make([]byte, bodyLength) + if err := readExact(ctx, reader, body, "response body"); err != nil { + return nil, err + } + if err := requireReaderEOF(ctx, reader); err != nil { + return nil, err + } + return body, nil +} + +func validateResponseIdentity(fields map[string]json.RawMessage, expectedAction string) error { + schemaVersion, err := decodeJSONField[uint32](fields["schema_version"], "response schema version") + if err != nil { + return err + } + if schemaVersion != 1 { + return errs.New(errs.CodeValidationFailed, "unsupported privileged response schema version") + } + action, err := decodeJSONField[string](fields["action"], "response action") + if err != nil { + return err + } + if err := validateAction(action); err != nil { + return err + } + if action != expectedAction { + return errs.New(errs.CodeValidationFailed, "privileged response action does not match request action") + } + return nil +} + +func decodeSuccessResult[T any](fields map[string]json.RawMessage) (T, error) { + var zero T + result, hasResult := fields["result"] + _, hasError := fields["error"] + if !hasResult || hasError { + return zero, errs.New( + errs.CodeValidationFailed, + "privileged success response must contain only one result", + ) + } + if isJSONNull(result) { + return zero, errs.New(errs.CodeValidationFailed, "privileged success result must not be null") + } + return decodeCanonicalJSONField[T](result, "success result") +} diff --git a/internal/lib/privilegedwire/response_test.go b/internal/lib/privilegedwire/response_test.go new file mode 100644 index 00000000..1663ef14 --- /dev/null +++ b/internal/lib/privilegedwire/response_test.go @@ -0,0 +1,329 @@ +package privilegedwire_test + +import ( + "bytes" + "context" + "encoding/binary" + "fmt" + "io" + "strings" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "mvmctl/internal/lib/privilegedwire" + "mvmctl/pkg/errs" +) + +type cleanupResult struct { + Removed bool `json:"removed"` +} + +func TestSuccessResponseFrameRoundTrip(t *testing.T) { + var encoded bytes.Buffer + err := privilegedwire.EncodeSuccess( + context.Background(), + &encoded, + "jailer-cleanup", + cleanupResult{Removed: true}, + ) + require.NoError(t, err) + + raw := encoded.Bytes() + require.GreaterOrEqual(t, len(raw), 12) + assert.Equal(t, []byte("MVMRES01"), raw[:8]) + assert.Equal(t, uint32(len(raw)-12), binary.BigEndian.Uint32(raw[8:12])) + + outcome, err := privilegedwire.DecodeResponse[cleanupResult]( + context.Background(), + bytes.NewReader(raw), + "jailer-cleanup", + ) + require.NoError(t, err) + assert.Equal(t, cleanupResult{Removed: true}, outcome.Result()) + assert.Nil(t, outcome.OperationError()) +} + +func TestDecodeResponseDistinguishesOperationErrorsFromProtocolFailures(t *testing.T) { + validError := `{"schema_version":1,"action":"jailer-cleanup","status":"error","error":` + + `{"code":"vm.cleanup.failed","class":"internal","message":"failed",` + + `"operation":"vm.cleanup","entity":"vm-id","details":{}}}` + outcome, protocolErr := privilegedwire.DecodeResponse[cleanupResult]( + context.Background(), + bytes.NewReader(responseFrame("MVMRES01", validError)), + "jailer-cleanup", + ) + require.NoError(t, protocolErr) + require.NotNil(t, outcome.OperationError()) + assert.Equal(t, errs.Code("vm.cleanup.failed"), outcome.OperationError().Code) + assert.Zero(t, outcome.Result()) + + outcome, protocolErr = privilegedwire.DecodeResponse[cleanupResult]( + context.Background(), + bytes.NewReader(responseFrame("BADMAGIC", validError)), + "jailer-cleanup", + ) + require.Error(t, protocolErr) + assert.Nil(t, outcome.OperationError()) + assert.Zero(t, outcome.Result()) +} + +func TestDecodeResponseRejectsMalformedOrNonCanonicalFrame(t *testing.T) { + validSuccess := `{"schema_version":1,"action":"jailer-cleanup","status":"success","result":{"removed":true}}` + validError := `{"schema_version":1,"action":"jailer-cleanup","status":"error","error":` + + `{"code":"vm.cleanup.failed","class":"internal","message":"failed",` + + `"operation":"vm.cleanup","entity":"vm-id","details":{}}}` + tests := map[string]struct { + frame []byte + wantErr string + }{ + "bad_magic": { + frame: responseFrame("BADMAGIC", validSuccess), + wantErr: "magic", + }, + "zero_length": { + frame: responseFrameWithLength("MVMRES01", 0, nil), + wantErr: "empty", + }, + "oversized_length": { + frame: responseFrameWithLength("MVMRES01", 65537, nil), + wantErr: "exceeds", + }, + "truncated_body": { + frame: responseFrameWithLength("MVMRES01", 100, []byte(validSuccess)), + wantErr: "response body", + }, + "short_declared_length": { + frame: responseFrameWithLength("MVMRES01", 2, []byte(validSuccess)), + wantErr: "trailing input", + }, + "trailing_frame_input": { + frame: append(responseFrame("MVMRES01", validSuccess), 0), + wantErr: "trailing input", + }, + "unsupported_schema": { + frame: responseFrame( + "MVMRES01", + `{"schema_version":2,"action":"jailer-cleanup","status":"success","result":{"removed":true}}`, + ), + wantErr: "schema version", + }, + "action_mismatch": { + frame: responseFrame( + "MVMRES01", + `{"schema_version":1,"action":"jailer-remove","status":"success","result":{"removed":true}}`, + ), + wantErr: "does not match", + }, + "invalid_status": { + frame: responseFrame( + "MVMRES01", + `{"schema_version":1,"action":"jailer-cleanup","status":"ok","result":{"removed":true}}`, + ), + wantErr: "response status", + }, + "missing_status": { + frame: responseFrame( + "MVMRES01", + `{"schema_version":1,"action":"jailer-cleanup","result":{"removed":true}}`, + ), + wantErr: "missing required", + }, + "unknown_envelope_field": { + frame: responseFrame( + "MVMRES01", + `{"schema_version":1,"action":"jailer-cleanup","status":"success",`+ + `"result":{"removed":true},"stderr":"root secret"}`, + ), + wantErr: "unknown JSON field", + }, + "duplicate_status": { + frame: responseFrame( + "MVMRES01", + `{"schema_version":1,"action":"jailer-cleanup","status":"success",`+ + `"status":"error","result":{"removed":true}}`, + ), + wantErr: "duplicate JSON field", + }, + "case_folded_duplicate_result": { + frame: responseFrame( + "MVMRES01", + `{"schema_version":1,"action":"jailer-cleanup","status":"success",`+ + `"result":{"removed":true,"REMOVED":false}}`, + ), + wantErr: "case-folded duplicate JSON field", + }, + "single_case_variant_result_field": { + frame: responseFrame( + "MVMRES01", + `{"schema_version":1,"action":"jailer-cleanup","status":"success",`+ + `"result":{"REMOVED":true}}`, + ), + wantErr: "unknown JSON field", + }, + "trailing_json": { + frame: responseFrame("MVMRES01", validSuccess+` {}`), + wantErr: "trailing JSON", + }, + "null_success_result": { + frame: responseFrame( + "MVMRES01", + `{"schema_version":1,"action":"jailer-cleanup","status":"success","result":null}`, + ), + wantErr: "must not be null", + }, + "unknown_success_result_field": { + frame: responseFrame( + "MVMRES01", + `{"schema_version":1,"action":"jailer-cleanup","status":"success",`+ + `"result":{"removed":true,"command":"id"}}`, + ), + wantErr: "unknown field", + }, + "success_with_error": { + frame: responseFrame( + "MVMRES01", + strings.Replace(validError, `"status":"error"`, `"status":"success","result":{"removed":true}`, 1), + ), + wantErr: "only one result", + }, + "error_with_result": { + frame: responseFrame( + "MVMRES01", + strings.Replace(validError, `"status":"error"`, `"status":"error","result":{"removed":true}`, 1), + ), + wantErr: "only one error", + }, + "error_without_error": { + frame: responseFrame( + "MVMRES01", + `{"schema_version":1,"action":"jailer-cleanup","status":"error"}`, + ), + wantErr: "only one error", + }, + "excessive_depth": { + frame: responseFrame( + "MVMRES01", + `{"schema_version":1,"action":"jailer-cleanup","status":"success","result":`+ + strings.Repeat("[", 33)+"0"+strings.Repeat("]", 33)+`}`, + ), + wantErr: "nesting exceeds", + }, + "excessive_object_fields": { + frame: responseFrame( + "MVMRES01", + `{"schema_version":1,"action":"jailer-cleanup","status":"success","result":`+ + objectWithFields(65)+`}`, + ), + wantErr: "object exceeds", + }, + "excessive_array_items": { + frame: responseFrame( + "MVMRES01", + `{"schema_version":1,"action":"jailer-cleanup","status":"success","result":[`+ + strings.Repeat(`"a",`, 1024)+`"a"]}`, + ), + wantErr: "array exceeds", + }, + } + + for name, tc := range tests { + t.Run(name, func(t *testing.T) { + _, err := privilegedwire.DecodeResponse[cleanupResult]( + context.Background(), + bytes.NewReader(tc.frame), + "jailer-cleanup", + ) + require.Error(t, err) + assert.ErrorContains(t, err, tc.wantErr) + assert.Equal(t, errs.CodeValidationFailed, errs.AsDomainError(err).Code) + }) + } +} + +func TestEncodeSuccessRejectsNullOrOversizedResultBeforeWriting(t *testing.T) { + type largeResult struct { + Value string `json:"value"` + } + tests := map[string]struct { + result *largeResult + wantErr string + }{ + "null": { + wantErr: "must not be null", + }, + "oversized": { + result: &largeResult{Value: strings.Repeat("a", 65536)}, + wantErr: "exceeds", + }, + } + + for name, tc := range tests { + t.Run(name, func(t *testing.T) { + var encoded bytes.Buffer + err := privilegedwire.EncodeSuccess( + context.Background(), + &encoded, + "jailer-cleanup", + tc.result, + ) + require.Error(t, err) + assert.ErrorContains(t, err, tc.wantErr) + assert.Empty(t, encoded.Bytes()) + }) + } +} + +func TestResponseFrameHonorsContextAndWriteFailures(t *testing.T) { + canceled, cancel := context.WithCancel(context.Background()) + cancel() + + var encoded bytes.Buffer + err := privilegedwire.EncodeSuccess( + canceled, + &encoded, + "jailer-cleanup", + cleanupResult{Removed: true}, + ) + require.ErrorIs(t, err, context.Canceled) + assert.Empty(t, encoded.Bytes()) + + err = privilegedwire.EncodeError( + context.Background(), + shortWriter{}, + "jailer-cleanup", + errs.New(errs.CodeProcessError, "operation failed"), + ) + require.Error(t, err) + assert.ErrorIs(t, err, io.ErrShortWrite) + + _, err = privilegedwire.DecodeResponse[cleanupResult]( + canceled, + bytes.NewReader(responseFrame( + "MVMRES01", + `{"schema_version":1,"action":"jailer-cleanup","status":"success","result":{"removed":true}}`, + )), + "jailer-cleanup", + ) + require.ErrorIs(t, err, context.Canceled) +} + +func responseFrame(magic string, body string) []byte { + return responseFrameWithLength(magic, uint32(len(body)), []byte(body)) +} + +func responseFrameWithLength(magic string, bodyLength uint32, body []byte) []byte { + frame := make([]byte, 12+len(body)) + copy(frame, magic) + binary.BigEndian.PutUint32(frame[8:12], bodyLength) + copy(frame[12:], body) + return frame +} + +func errorResponseBody(errorJSON string) string { + return fmt.Sprintf( + `{"schema_version":1,"action":"jailer-cleanup","status":"error","error":%s}`, + errorJSON, + ) +} From 4dd59377bb9849fbee476932628bf921c017ba24 Mon Sep 17 00:00:00 2001 From: AlanD20 Date: Sun, 23 Aug 2026 16:48:15 +0200 Subject: [PATCH 0036/1457] docs: refine privileged runtime contracts --- CHANGELOG.md | 6 ++ CONTEXT.md | 8 ++ ...owned-single-binary-privileged-dispatch.md | 93 +++++++++++++++---- ...m-network-namespaces-and-typed-policies.md | 7 +- tasks/plan.md | 44 +++++++-- tasks/todo.md | 30 ++++-- 6 files changed, 151 insertions(+), 37 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index c535184b..c88d4ecf 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -42,6 +42,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 normal CLI initialization and malformed reserved invocations fail closed instead of falling through. - Added strict bounded JSON parsing with unknown, duplicate, case-colliding, trailing, oversized, and structurally invalid input rejection. +- Added the version-1 `MVMREQ01`/`MVMRES01` codec with 64 KiB JSON frames, receiver-bounded payload declarations, + exact action/schema matching, read-only typed success/error outcomes distinct from protocol failures, and bounded + `DomainError` normalization that omits wrapped causes and sensitive process-output details. - Added authenticated sudo identity and current `mvm` group checks, fixed root environment sanitization, and executable device/inode verification against the root-owned `/usr/local/bin/mvm` image. - Added ADR-0016, which defines the single-binary privilege boundary, trusted state layout, typed operation catalog, @@ -179,6 +182,9 @@ The following items are release blockers and are intentionally not described abo - Add root-owned managed-network identity and global capacity authority; user SQLite state must not authorize host-global namespace, link, firewall, process, mount, cgroup, or admission decisions. - Replace path-validation-then-reopen behavior with descriptor-pinned managed-resource access and private mount namespaces. +- Split root-only locks/handshakes beneath `/run/mvmctl/authority` from caller-owned socket/PID output directories beneath + `/run/mvmctl/runtime`; stop mounting the whole cache VM directory. Pin persistent config/output leaves individually, + truncate enabled `0600` log/console/metrics files only after durable launch registration, and retain them after stop. - Independently verify and atomically install trusted Firecracker/Jailer releases inside the privileged boundary. - Migrate Jailer, loopmount, network, firewall, and supported host mutations to distinct typed privileged actions; remove the public root `mvm run jailer` and `mvm run provision` entry points. diff --git a/CONTEXT.md b/CONTEXT.md index 7f42752d..33908626 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -246,6 +246,14 @@ dispatch. That policy is not the final security boundary. The marker-only policy privileged caller uses the typed dispatcher. A system-installed executable cannot replace itself through `mvm self-update`; an administrator installs the newly downloaded trusted artifact with `host install-system`. +The v0.3 privileged target splits state by authority and lifetime. Durable ownership and release records live beneath +`/var/lib/mvmctl`; root-only locks and launch handshakes will live beneath `/run/mvmctl/authority/`. API, vsock, and +console sockets plus display-only PID mirrors will use a separate pinned caller-owned VM directory beneath +`/run/mvmctl/runtime//`. Persistent configuration, disks, logs, console logs, and optional metrics remain in +the caller's managed cache. Final privileged code must never mount the whole cache VM directory, trust a runtime leaf as +authority, or signal a PID read from a PID mirror. ADR-0016 defines the exact ownership, modes, launch ordering, and +cleanup contract; `tasks/todo.md` records implementation status. + For **release testing / RC QA / system tests**, always use `./scripts/build.sh release`; a bare `go build` lacks the release version metadata, symbol stripping, and PIE settings and must not be used for release qualification. An untagged RC build must pass the intended version explicitly. Otherwise `git describe` can inherit the previous release diff --git a/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md b/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md index 24920336..a26aaf43 100644 --- a/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md +++ b/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md @@ -97,6 +97,15 @@ while the socketpair permits the client to upload a bounded release archive and Root requires descriptor 0 to be an `AF_UNIX` `SOCK_STREAM` socket and requires its Linux `SO_PEERCRED` UID/GID to match the authenticated sudo caller. The peer PID is positive and audit-only; it is not an authorization identity. +The transport always executes exactly +`/usr/bin/sudo -n -- /usr/local/bin/mvm __mvm_privileged_v1 `. It never uses `PATH`, `os.Executable`, the +current user binary, a caller-supplied executable/argv/environment/cwd/timeout, or a root-EUID bypass; the receiver +requires authenticated non-root sudo identity. `internal/lib/system` provides one concrete transport rather than +extending the generic command runner. Each capability package owns a minimal private `Exchange` interface and named +typed public methods; no public generic privileged call/action/decoder hook is introduced. Hermetic tests inject private +process dependencies and use a dedicated fake exchanger without extending `CommandRunner`, `RunCmdOpts`, `SpawnConfig`, +or the existing `FakeRunner`. + Each request starts with fixed `MVMREQ01` magic, a network-order 32-bit JSON-header length, and a network-order 64-bit payload length. The strict header is at most 64 KiB and contains schema version 1, the action repeated from argv, and the action-specific body. The repeated action must match. Only release install may carry a payload: either zero bytes or an @@ -112,9 +121,17 @@ result or error. Error envelopes carry code, stable string class, message, opera they never serialize wrapped causes, stacks, stderr, or arbitrary Go values. Wire details permit only bounded booleans, strings, signed integers, and string arrays. Unsupported details are omitted visibly. Root half-closes the control socket's write side after the complete frame, so the client can validate response EOF independently of process exit. An -error response delivered after all effects and cleanup exits successfully at the process layer. A start failure has -known no-effect status; any started process with a missing, malformed, truncated, mismatched, oversized, or non-EOF -response returns `CodeProcessError` with `process_started: true` and `outcome_unknown: true`. +error response delivered after all effects and cleanup exits successfully at the process layer. The codec returns a +decoded success/error outcome separately from protocol failure, so a valid root `DomainError` cannot be mistaken for a +malformed frame. A start failure has known no-effect status; any started process with a missing, malformed, truncated, +mismatched, oversized, or non-EOF response returns `CodeProcessError` with `process_started: true` and +`outcome_unknown: true`. + +The transport returns bounded raw response bytes only together with an explicit indication that actual socket EOF was +observed; copying a complete-looking prefix into an in-memory reader cannot fabricate authority. The named capability +client decodes an EOF-qualified response with a non-cancelled context and lets either a valid success or valid remote +`DomainError` win over upload `EPIPE`, non-zero exit, or later reap diagnostics. Without one valid envelope, a pre-start +failure reports `process_started: false, outcome_unknown: false`; every post-start failure reports both values as true. ### Authorization roles @@ -233,21 +250,56 @@ read-write and writes its fixed `rootfs.img`, `memory`, and `vmstate` leaves. Re overlays the new VM's pinned `/rootfs` at `/snapshot/rootfs.img` inside that VM's private mount namespace; no persistent phantom-rootfs symlink is created. -The v0.3 canonical privileged-visible layout is: +The v0.3 canonical persistent layout is: | Resource | Relative to the pinned managed cache root | |---|---| | VM directory | `vms//` | | VM rootfs | `vms//rootfs.img` | | Cloud-init ISO | `vms//cloud-init.iso` | -| Firecracker runtime | Fixed `firecracker.json`, `firecracker.api.socket`, `firecracker.pid`, `firecracker.log`, `firecracker.console.log`, and `firecracker.metrics` leaves | -| Console/vsock runtime | Fixed `console.sock`, `console.pid`, and `vsock.sock` leaves | +| Firecracker configuration | `vms//firecracker.json` | +| Persistent VM outputs | Fixed `vms//firecracker.log`, `firecracker.console.log`, and optional `firecracker.metrics` regular-file leaves | | Volume | `volumes/.` | | Kernel | `kernels/` | | Snapshot | `snapshots//{rootfs.img,memory,vmstate}` | | Durable image | `images/.zst` | | Image staging | `images/staging//{source.raw,rootfs.img}` | +Sockets and PID mirrors are not persistent cache artifacts. Their host layout is fixed beneath the root-controlled +ephemeral runtime tree: + +```text +/run/mvmctl/ root:root 0711 +├── authority/ root:root 0700 +│ └── / root:root 0700 +└── runtime/ root:root 0711 + └── / root:root 0711 + └── / caller-uid:caller-gid 0700 + ├── firecracker.api.socket + ├── vsock.sock + ├── console.sock + ├── firecracker.pid + └── console.pid +``` + +The root-owned UID parent prevents the caller from replacing the VM runtime directory. Root creates and pins that +directory descriptor-relatively after durable launch registration, verifies its owner, mode, device, and inode, and +retains the descriptor through mount and rollback. The dropped VM process uses umask `0077`. Socket leaves must be +caller-owned sockets with no world access; PID leaves are caller-owned `0600` positive-decimal display mirrors capped +at 32 bytes. Privileged process control never trusts a PID mirror. + +The caller can mutate leaves inside its runtime directory, so root treats the directory only as a fixed output area, +never as authority. It is mounted as a directory at `/run/mvm/runtime` inside the private mount namespace so sockets +created after child release remain visible. The user-owned cache VM directory is never mounted wholesale. Persistent +configuration and output files are opened and verified individually, then mounted or inherited through their pinned +descriptors. + +The three persistent output leaves are caller-owned `0600` regular files with one link. A launch truncates each enabled +output exactly once, only after the root launch record is durable, and retains the files after stop or crash for +inspection. `firecracker.metrics` is created only when metrics are enabled. Direct Firecracker writes do not provide a +truthful hard live-size cap; v0.3 therefore promises bounded readers and one-launch retention, not live rotation. A +future hard cap requires a separately designed and supervised FIFO output collector. + Warm-cache files are unprivileged accelerators and never root authority. Existing database path columns may remain as derived display/local metadata during the clean break, but privileged handlers reconstruct these names from typed identities and never authorize a path from SQLite. @@ -264,14 +316,22 @@ The prepared value retains the verified manifest, pinned release directory, and It supplies the full release identity to instance registration itself, then transfers the release lock into the launch lease only after the owner-bound record is durable. The unprivileged caller never supplies executable hashes. -The launch path starts a blocked child in a new private mount namespace. Before the child can perform any externally -mutable effect, the parent pins its pidfd and namespace descriptor, records the complete identity, and retains the launch -lease. The parent then enters that namespace, makes mount propagation private, mounts descriptor-pinned resources, and -releases the child to exec the exact Jailer/Firecracker pair. Process and namespace allocation for a blocked child are -not considered an externally mutable launch effect; mounts, cgroup changes, links, and executable handoff are. Live -snapshot and volume operations reopen and verify `/proc//ns/mnt` while the process is alive and enter only that -namespace. No persistent mount-namespace handle is retained after the operation, because doing so would retain every -mount after process exit. Cleanup never recursively traverses a tree that may contain a mount. +The launch path starts a blocked Firecracker child in a new private mount namespace and, when enabled, a blocked console +relay as the dropped caller in the host mount namespace. Before either child can perform an externally mutable effect, +the parent pins each pidfd and the Firecracker namespace descriptor, records the complete identity, and retains the +launch lease. Root then creates/truncates the fixed output slots and runtime directory, makes Firecracker mount +propagation private, installs named mounts, publishes the non-authoritative PID mirrors, releases and verifies the relay, +and only then releases Firecracker to exec the exact Jailer/Firecracker pair. The relay receives only inherited PTY, +pinned log, and pinned runtime-directory descriptors; it never reopens a caller path or joins and retains the VM mount +namespace. Process and namespace allocation for a blocked child are not considered an externally mutable launch effect; +mounts, output creation/truncation, cgroup changes, links, and executable handoff are. + +Live snapshot and volume operations reopen and verify `/proc//ns/mnt` while the process is alive and enter only +that namespace. No persistent mount-namespace handle is retained after the operation, because doing so would retain +every mount after process exit. Cleanup stops and reaps the verified relay, unmounts named resources in reverse order, +unlinks only the five fixed runtime leaves, and removes the runtime directory only when empty. It preserves persistent +outputs and never recursively traverses a caller-writable tree or a tree that may contain a mount. Unexpected runtime +entries leave a bounded recoverable cleanup record instead of authorizing recursive deletion. State is split by authority and lifetime: @@ -281,9 +341,10 @@ State is split by authority and lifetime: | Trusted release pairs and manifests | `/var/lib/mvmctl/binaries//` | Root-owned, persistent | | Privileged instance ownership/release records | `/var/lib/mvmctl/instances/` | Root-owned, persistent, minimal | | Jailer chroot directories | `/var/lib/mvmctl/jailer` | Root-owned lifecycle state | -| Per-VM locks and launch handshakes | `/run/mvmctl/` | Root-owned, ephemeral; no persistent mount-namespace handles | +| Per-VM locks and launch handshakes | `/run/mvmctl/authority/` | Root-owned `0700`, ephemeral; no persistent mount-namespace handles | +| VM sockets and PID mirrors | `/run/mvmctl/runtime//` | Root-controlled parent, caller-owned `0700` VM directory, ephemeral and non-authoritative | | Cgroups | `/sys/fs/cgroup/mvmctl/` | Kernel runtime state | -| `/run/mvm` | Inside the jail | Jail-visible bind mount, not host canonical state | +| `/run/mvm` and `/run/mvm/runtime` | Inside the jail | Individually pinned persistent leaves plus the pinned ephemeral runtime directory; not host authority | Moving durable user state out of `~/.cache/mvmctl` is a separate XDG migration decision. Full state must not move to `/run` or `/var/run`. diff --git a/docs/adr/0017-per-vm-network-namespaces-and-typed-policies.md b/docs/adr/0017-per-vm-network-namespaces-and-typed-policies.md index 162c0663..3690dcda 100644 --- a/docs/adr/0017-per-vm-network-namespaces-and-typed-policies.md +++ b/docs/adr/0017-per-vm-network-namespaces-and-typed-policies.md @@ -47,9 +47,10 @@ the veth pair, a local bridge, and the Firecracker TAP. Interface names, namespa the invoking UID and immutable resource IDs; callers cannot provide raw interface names, namespace paths, `ip` argv, or netns commands. -Namespace handles and locks are ephemeral root-owned state under `/run/mvmctl//`. Persistent minimal VM and network -ownership records remain under `/var/lib/mvmctl/`. The user database stores desired logical configuration but never -authorizes a root namespace, device, bridge, route, or firewall mutation. +Namespace handles and locks are ephemeral root-owned state under `/run/mvmctl/authority//`. VM-created sockets and +display-only PID mirrors use the separate non-authoritative `/run/mvmctl/runtime///` tree defined by +ADR-0016. Persistent minimal VM and network ownership records remain under `/var/lib/mvmctl/`. The user database stores +desired logical configuration but never authorizes a root namespace, device, bridge, route, or firewall mutation. The privileged launch transaction must: diff --git a/tasks/plan.md b/tasks/plan.md index e59466f5..3303e19d 100644 --- a/tasks/plan.md +++ b/tasks/plan.md @@ -28,6 +28,8 @@ continue without silently deleting it. - Early reserved privileged dispatch before Cobra and ordinary application initialization. - Strict bounded request parsing, sudo caller identity, group authorization, environment sanitization, and executable identity verification. +- Strict version-1 length-framed request/response codec with matching actions, read-only typed outcomes distinct from + protocol failures, and bounded safe `DomainError` preservation. - Private root-owned VM ownership/lifecycle records with strict codecs, descriptor-relative atomic storage, exact release references, global VM-ID claims, and release/index/VM locking. - ADR-0016 for the single-binary privilege boundary. @@ -47,7 +49,8 @@ firewall backends, host-global TAP topology, public `mvm run jailer` / `mvm run | VM ownership/lifecycle records | `/var/lib/mvmctl/instances/` | Root-owned persistent | | Network ownership/topology records | `/var/lib/mvmctl/networks/` | Root-owned persistent | | Global capacity allocations | `/var/lib/mvmctl/allocations` | Root-owned persistent | -| Locks and launch handshakes | `/run/mvmctl` | Root-owned ephemeral; no persistent mount-namespace handles | +| Locks and launch handshakes | `/run/mvmctl/authority/` | Root-owned `0700` ephemeral; no persistent mount-namespace handles | +| VM sockets and PID mirrors | `/run/mvmctl/runtime//` | Root-controlled parent, caller-owned `0700` VM directory; ephemeral and non-authoritative | | Jailer chroots | `/var/lib/mvmctl/jailer` | Root-owned lifecycle state | | Cgroups | `/sys/fs/cgroup/mvmctl/` | Kernel runtime state | @@ -80,6 +83,12 @@ runner owns concurrent upload/response I/O so an early root rejection cannot dea Root requires an `AF_UNIX` `SOCK_STREAM` descriptor and Linux `SO_PEERCRED` UID/GID equal to the authenticated sudo caller. The positive peer PID is recorded only for audit and never substitutes for typed receiver authorization. +The concrete transport always runs +`/usr/bin/sudo -n -- /usr/local/bin/mvm __mvm_privileged_v1 `. It exposes no executable, argv, environment, +cwd, timeout, callback, decoder, or extra-FD option, never uses `PATH`/`os.Executable`, and never bypasses sudo for a root +caller. It remains separate from `CommandRunner`, `RunCmdOpts`, `SpawnConfig`, and `FakeRunner`; capability packages use +private minimal exchanger interfaces and a dedicated fake in tests. + Requests use fixed `MVMREQ01` magic, a 32-bit network-order JSON-header length capped at 64 KiB, a 64-bit network-order payload length, the strict schema-version-1 header, and the exact optional payload. The header repeats the fixed argv action and must match it. Only typed release install accepts a payload: zero bytes or at most 128 MiB. The caller @@ -88,9 +97,13 @@ zero-payload effects, but may respond to pre-effect authentication/header failur Responses use `MVMRES01`, a 32-bit length capped at 64 KiB, and one strict matching-action success/result or error/`DomainError` envelope. A delivered envelope is the final effect record even if process reaping later reports an anomaly; root half-closes its socket write side after the complete frame so response EOF does not depend on process exit. -No domain effect or fallible outcome check is allowed after the response. Missing, malformed, truncated, mismatched, -oversized, or non-EOF responses from a started process report -`process_started: true` and `outcome_unknown: true`; stderr and exit text are never parsed as authority. +The codec exposes a read-only decoded success/error outcome separately from protocol failure, so callers cannot confuse a +valid root `DomainError` with an untrusted malformed frame. The transport reports whether actual socket EOF was observed; +in-memory EOF over copied bytes cannot promote a response. A capability client decodes EOF-qualified bytes with a +non-cancelled context and lets a valid success or valid remote error win over upload `EPIPE`, exit, or later reap +diagnostics. No domain effect or fallible outcome check is allowed after the response. Missing, malformed, truncated, +mismatched, oversized, or non-EOF responses from a started process report `process_started: true` and +`outcome_unknown: true`; a pre-start failure reports both false. Stderr and exit text are never parsed as authority. ### VM authority interface @@ -174,14 +187,27 @@ durable record before the first network or launch effect, and transfers the held lease. A mismatch or registration failure performs no external effect and leaves a checked, releasable slot lease. Task 4 owns release, index, and VM locking; it does not import network code or predeclare a network implementation. -The mount-namespace launch seam uses a blocked child: the child creates its private namespace and waits on an inherited -private handshake without performing external mutation. The parent pins the pidfd and namespace descriptor, persists -the complete launch identity, then enters the namespace, marks propagation private, mounts named resources, and releases -the child to exec. Here, "first launch effect" means the first externally mutable effect; allocating the blocked process -and namespace is preparation. Mutating process operations require pidfds and never fall back to `kill(pid)`. Live mount +The mount-namespace launch seam uses a blocked Firecracker child in a private mount namespace and, when enabled, a +blocked console relay running as the dropped caller in the host mount namespace. The parent pins both pidfds and the +Firecracker namespace descriptor, persists the complete launch identity, and only then creates/truncates fixed +persistent outputs and creates the per-VM runtime directory. It enters the Firecracker namespace, marks propagation +private, mounts the runtime directory at `/run/mvm/runtime` and each persistent resource through its pinned descriptor, +publishes display-only PID mirrors, releases and verifies the relay, and releases Firecracker to exec. The relay receives +only inherited PTY, pinned log, and pinned runtime-directory descriptors; it never reopens a caller path or retains the +VM namespace. Here, "first launch effect" means the first externally mutable effect; allocating blocked processes and +the namespace is preparation. Mutating process operations require pidfds and never fall back to `kill(pid)`. Live mount operations reopen and verify `/proc//ns/mnt`; a persistent mount-namespace handle is prohibited because it would retain the namespace's mounts after process exit. +The persistent VM outputs are fixed caller-owned `0600` one-link regular files: `firecracker.log`, +`firecracker.console.log`, and, when enabled, `firecracker.metrics`. They are truncated once per launch after durable +registration and retained after stop or crash. Direct Firecracker output supports bounded readers and one-launch +retention but not a truthful hard live-size cap; such a cap would require a separately supervised FIFO collector. API, +vsock, and console sockets plus the two PID mirrors are fixed ephemeral leaves beneath +`/run/mvmctl/runtime//`. Root retains the directory descriptor through setup and rollback but never treats +its caller-writable contents as authority. The cache VM directory is never mounted wholesale. Cleanup unlinks only the +fixed runtime leaves and removes the directory only when empty; it never recursively deletes caller-writable contents. + Atomic record replacement uses a random exclusive temporary file, bounded writes, root ownership and exact modes, file `fsync`, checked close, `renameat`, and parent-directory `fsync`. Pre-rename failure preserves the old record. Post-rename failure returns details that distinguish `record_replaced` and `durability_uncertain`; the caller starts diff --git a/tasks/todo.md b/tasks/todo.md index dd7aa909..5a71b798 100644 --- a/tasks/todo.md +++ b/tasks/todo.md @@ -63,10 +63,13 @@ The final marker-only sudoers assertion is deferred until Task 14; current sudoe - [x] Unsupported versions fail closed. - [x] Strict bounded JSON rejects unknown, duplicate, case-fold duplicate, trailing, oversized, and malformed input. - [x] Caller identity, active `mvm` group, environment, and system executable are verified before dispatch. -- [ ] Add a strict versioned request/response codec with 64 KiB frames, duplicate/unknown/depth rejection, matching - actions, typed results, and bounded `DomainError` code/class/entity/partial-state detail preservation. -- [ ] Add the fd-0 Unix socketpair control runner with concurrent upload/response, request and response half-close/EOF, - CLOEXEC, cancellation, timeout, bounds, early-rejection handling, and `FakeRunner` support. +- [x] Add a strict versioned request/response codec with 64 KiB frames, duplicate/unknown/depth rejection, matching + actions, typed read-only success/error outcomes separated from protocol failures, and bounded `DomainError` + code/class/entity/partial-state detail preservation. +- [ ] Add the fd-0 Unix socketpair control transport with fixed `/usr/bin/sudo -n -- /usr/local/bin/mvm` invocation, + concurrent upload/response, request and response half-close/actual EOF, CLOEXEC, cancellation, bounded reaping and + buffers, early-rejection handling, private process dependencies, and a dedicated fake exchanger; do not extend + `CommandRunner`, `RunCmdOpts`, `SpawnConfig`, or `FakeRunner`. - [ ] Return `process_started: true` and `outcome_unknown: true` for every started process without one valid final response; stdout, stderr, and generic subprocess text are never action authority. - [ ] Keep the action switch closed until each capability has its own typed handler and abuse tests. @@ -139,7 +142,11 @@ under `internal/service/jailer/`. No handler, transport, CLI, API, core-domain, - [x] ID-based `volumes/.` storage. - [x] Content-addressed `kernels/` storage with receiver-owned staging; kernel pull inputs cannot select an output directory, name, or path. - - [ ] Fixed `rootfs.img`, cloud-init, Firecracker, console, and vsock VM leaves. + - [ ] Fixed persistent `rootfs.img`, cloud-init, Firecracker configuration, log, console-log, and optional metrics + leaves; outputs are caller-owned `0600` one-link regular files truncated only after durable launch registration. + - [ ] Fixed ephemeral API/vsock/console sockets and PID mirrors beneath + `/run/mvmctl/runtime//`; the root-controlled parent and pinned caller-owned VM directory replace the + whole-cache-VM-directory bind. - [ ] Fixed snapshot leaves and a managed ID-scoped image-provisioning staging subtree. - [ ] Define operation-specific non-empty/maximum-size and access-mode policies before coding; a caller- or DB-provided expected size is an equality check, not authorization for an unbounded resource. @@ -180,8 +187,12 @@ under `internal/service/jailer/`. No handler, transport, CLI, API, core-domain, - [ ] Extend the strict instance record once with owner GID, host boot ID, mount-namespace device/inode, and Task 5's pinned cache identity; bump the schema version and reject old/partial records. -- [ ] Launch a blocked child in a private mount namespace, pin its pidfd and namespace FD, persist the complete record - before externally mutable effects, mark propagation private, then release the child only after named mounts succeed. +- [ ] Launch a blocked Firecracker child in a private mount namespace and an optional blocked dropped-caller console + relay in the host mount namespace; pin their pidfds and the Firecracker namespace FD, persist the complete record + before externally mutable effects, create/truncate fixed outputs, mark propagation private, mount the pinned runtime + directory and individual persistent resources, publish display-only PID mirrors, then release Firecracker only after + named mounts and relay readiness succeed. The relay receives inherited descriptors and never reopens a caller path or + retains the VM namespace. - [ ] Verify PID, boot ID, start ticks, all UID/GID identities, supplementary groups, exact cgroup, namespace inode, process liveness, and pinned executable hash against the Task 6 release manifest before signal or `setns`. - [ ] Require pidfd support for every mutating operation, signal only with `pidfd_send_signal`, wait through the pidfd, @@ -191,8 +202,9 @@ under `internal/service/jailer/`. No handler, transport, CLI, API, core-domain, - [ ] Lock the OS thread for `setns`, pin/restore/verify the original host namespace, and terminate the short-lived privileged process if restoration fails rather than returning a contaminated thread to the Go runtime. - [ ] Live snapshot/volume operations enter only the verified VM mount namespace. -- [ ] Cleanup parses and verifies mount state, unmounts only receiver-derived fixed leaves, requires cgroup emptiness, - and removes a descriptor-relative allowlist bottom-up; never recursively traverse a tree that may contain a mount. +- [ ] Cleanup parses and verifies mount state, reaps the pidfd-verified relay, unmounts only receiver-derived fixed + leaves, requires cgroup emptiness, unlinks only the five runtime leaves, and removes descriptor-relative allowlisted + directories bottom-up only when empty; never recursively traverse a caller-writable tree or one containing a mount. - [ ] L1 fault tests cover source/target replacement, PID mismatch, process exit, setns, mount, unmount, and cleanup. ## Phase 2: Final Jailer and networking paths From cc204f6caa1af15af76371e8fe6a5976c02df0d8 Mon Sep 17 00:00:00 2001 From: AlanD20 Date: Sun, 23 Aug 2026 16:59:43 +0200 Subject: [PATCH 0037/1457] docs: expose v0.3 qualification gaps --- CHANGELOG.md | 3 + tests/system/COVERAGE_MATRIX.md | 142 +++++++++++++++++++------------- 2 files changed, 86 insertions(+), 59 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index c88d4ecf..83f6ce36 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -87,6 +87,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 non-empty, the process/report exit statuses match, every selected item passed, and failures, errors, collection errors, deselections, skips, XFAIL, and XPASS are all zero. Missing, malformed, duplicate-field, oversized reports and report cleanup failures fail the domain while preserving pytest output and the validation reason. +- The system-test coverage matrix now recounts its 431 current CLI rows and separates transitional/legacy coverage from + the still-missing v0.3 root-authority, namespace, nftables, traffic, exec-policy, reconciliation, and zero-leak release + evidence. A legacy `Deep` row is no longer presented as clean-break security signoff. #### Deterministic initialization - Added `mvm init --binary-version ` to request an exact Firecracker/Jailer pair when no local pair exists. diff --git a/tests/system/COVERAGE_MATRIX.md b/tests/system/COVERAGE_MATRIX.md index d2588049..cff25a1d 100644 --- a/tests/system/COVERAGE_MATRIX.md +++ b/tests/system/COVERAGE_MATRIX.md @@ -10,13 +10,44 @@ removed, or when test coverage changes. |--------|---------| | ✅ Deep | L3 Option C verification | | ⚡ Shallow | L0-L2 verification (returncode, output, JSON) | -| 🟡 Partial | Test exists but skips under some conditions | +| 🟡 Partial | Coverage exists, but required behavior remains unproved and/or conditional | | 🔴 Missing | No test exists | | ⏭️ Skip | Test defined but always skips (broken setup) | | **?** | Needs investigation — coverage unclear | +## v0.3.0 release-security qualification overlay + +The command tables below describe tests for the currently implemented CLI, including transitional behavior that v0.3.0 +will remove. A `✅ Deep` row proves only the behavior named in that row; it is not evidence that the clean-break security +architecture is release-qualified. The authoritative release gate is `tasks/todo.md` Task 17 plus `docs/RC_QA.md`. + +| v0.3.0 security capability | Status | Required release evidence | +|---|---|---| +| Installed system candidate and clean bootstrap | 🟡 Partial | Installed-candidate and happy-path init coverage exists; clean group/membership setup, root journal, rollback, fail-closed host mutation, and marker-only sudoers denial remain missing. | +| fd-0 privileged control transport | 🔴 Missing | Go L1 framing/EOF/peer/cancellation abuse coverage plus installed-CLI lifecycle integration. | +| Root VM authority and cross-UID denial | 🔴 Missing | Root-record/process/cache identity checks, constrained two-user denial, tombstone and zero-leak assertions. | +| Independent trusted Firecracker/Jailer releases | 🔴 Missing | Root manifest/ELF/hash authority, atomic failure injection, and installed-CLI release lifecycle. | +| Descriptor-pinned resources and private mount namespace | 🔴 Missing | Type/owner/mode/link checks, replacement races, namespace mounts, cleanup, restart, and restore. | +| Mandatory per-VM network namespace | 🔴 Missing | Distinct namespace inodes/topology, root-record binding, rollback, reconciliation, and zero leaked links/TAPs. | +| nftables-only engine | 🔴 Missing | Atomic bridge-family generation and removal of iptables/backend-selection tests and product paths. | +| Typed `traffic` policy | 🔴 Missing | Same/cross-network default deny and exact source/protocol/port allows plus spoof/reverse/IP-reuse denials. | +| Typed VM-to-VM `exec` policy | 🔴 Missing | Directional non-root authorization for same/cross/no-IP networks plus identity, bounds, and redaction denials. | +| Cgroup/process identity lifecycle | 🟡 Partial | Existing L2 tests cover cgroup limits; root-record, pidfd, mount/cgroup cleanup, and reconciliation binding remain missing. | +| Crash reconciliation and repeated zero-leak runs | 🔴 Missing | Every applicable domain twice with aggregated teardown and absence checks after failure and reboot. | + +Specific legacy conflicts that must be rewritten before release: + +- `network/test_policies.py` still exercises both firewall backends and rejects same-network policy creation. +- `full_journeys/test_full_journeys.py` still expects same-network VM traffic without an allow policy. +- `exec/test_exec.py` covers host-to-VM execution only; two cases still use a PATH-selected `mvm`. +- `vm/test_jailer.py` observes process/chroot outcomes through user-visible state but does not qualify root authority or + the fd-0 transport. +- Shared cleanup and several test-local teardowns still suppress errors or mutate host state with broad raw sudo. +- `run provision` and `run jailer` coverage names public routes that the final release must remove. + ## Table of Contents +- [v0.3.0 release-security qualification overlay](#v030-release-security-qualification-overlay) - [Root CLI](#root-cli) - [mvm self-update](#mvm-self-update) - [mvm init](#mvm-init) @@ -634,65 +665,55 @@ removed, or when test coverage changes. ## Summary Statistics -| Category | Total Scenarios | ✅ Deep | ⚡ Shallow | 🔴 Missing | ⏭️ Skip | -|----------|----------------|---------|-------------|-------------|----------| -| Root CLI | 8 | 0 | 8 | 0 | 0 | -| self-update | 4 | 1 | 0 | 3 | 0 | -| init | 5 | 0 | 5 | 0 | 0 | -| config | 12 | 0 | 12 | 0 | 0 | -| network | 34 | 13 | 19 | 0 | 0 | -| vm | 101 | 17 | 72 | 0 | 0 | -| exec | 8 | 0 | 8 | 0 | 0 | -| snapshot | 14 | 1 | 10 | 0 | 0 | -| volume | 36 | 4 | 31 | 0 | 0 | -| key | 28 | 3 | 25 | 0 | 0 | -| image | 42 | 0 | 13 | 1 | 28 | -| kernel | 31 | 1 | 21 | 0 | 9 | -| bin | 17 | 2 | 7 | 0 | 8 | -| ssh | 6 | 4 | 2 | 0 | 0 | -| console | 6 | 0 | 6 | 0 | 0 | -| logs | 9 | 0 | 9 | 0 | 0 | -| host | 15 | 7 | 7 | 0 | 1 | -| cache | 13 | 3 | 6 | 0 | 1 | -| cp | 12 | 7 | 5 | 0 | 0 | -| env | 22 | 9 | 13 | 0 | 0 | -| run | 3 | 0 | 3 | 0 | 0 | -| full-journeys | 13 | 5 | 8 | 0 | 0 | -| **Total** | **439** | **77** | **292** | **4** | **47** | - -**Coverage health:** -- ✅ Deep (L3): 77/439 = 17.5% -- ⚡ Shallow (L0-L2 incl. 🟡 Partial): 292/439 = 66.5% -- 🔴 Missing: **4/439 = 0.9%** — three user-owned `self-update` success/check paths plus the phantom +| Category | Total Scenarios | ✅ Deep | ⚡ Shallow | 🟡 Partial | 🔴 Missing | ⏭️ Skip | +|----------|----------------|---------|-------------|------------|-------------|----------| +| Root CLI | 7 | 0 | 7 | 0 | 0 | 0 | +| self-update | 4 | 1 | 0 | 0 | 3 | 0 | +| init | 5 | 0 | 4 | 1 | 0 | 0 | +| config | 12 | 0 | 12 | 0 | 0 | 0 | +| network | 35 | 14 | 21 | 0 | 0 | 0 | +| vm | 94 | 18 | 71 | 5 | 0 | 0 | +| exec | 8 | 0 | 8 | 0 | 0 | 0 | +| snapshot | 13 | 1 | 12 | 0 | 0 | 0 | +| volume | 35 | 4 | 29 | 2 | 0 | 0 | +| key | 28 | 3 | 25 | 0 | 0 | 0 | +| image | 41 | 0 | 11 | 3 | 1 | 26 | +| kernel | 30 | 2 | 15 | 6 | 0 | 7 | +| bin | 17 | 2 | 6 | 1 | 0 | 8 | +| ssh | 6 | 4 | 1 | 1 | 0 | 0 | +| console | 6 | 0 | 6 | 0 | 0 | 0 | +| logs | 9 | 0 | 9 | 0 | 0 | 0 | +| host | 15 | 6 | 7 | 1 | 0 | 1 | +| cache | 13 | 3 | 9 | 0 | 0 | 1 | +| cp | 12 | 8 | 3 | 1 | 0 | 0 | +| env | 22 | 9 | 13 | 0 | 0 | 0 | +| run | 4 | 1 | 3 | 0 | 0 | 0 | +| full-journeys | 15 | 7 | 8 | 0 | 0 | 0 | +| **Total** | **431** | **83** | **280** | **21** | **4** | **43** | + +**Current-row coverage health:** +- ✅ Deep (L3): 83/431 = 19.3% +- ⚡ Shallow (L0-L2): 280/431 = 65.0% +- 🟡 Partial: 21/431 = 4.9% +- 🔴 Missing: **4/431 = 0.9%** — three user-owned `self-update` success/check paths plus the phantom `image inspect --tree` entry -- ⏭️ Skip-prone: 47/439 = 10.7% +- ⏭️ Skip-prone: 43/431 = 10.0% -**Current real CLI gap:** successful user-owned `self-update` check/apply behavior remains unqualified at L2. The +These counts inventory rows for the transitional CLI only. They do not include the separate v0.3.0 security blockers in +the qualification overlay. Successful user-owned `self-update` check/apply behavior remains unqualified at L2. The `image inspect --tree` missing entry is a phantom command because the CLI has no such flag. --- -**QA update (latest):** -- ✅ Coverage matrix fully audited against actual CLI flags from Go source -- ✅ All stale flag names corrected: `--vcpus`→`--vcpu`, `--user-data`→`--cloudinit-config`, `--enable-pci`/`--no-enable-pci`→`--no-pci`, `--no-console`→default behavior -- ✅ Phantom commands removed: `vm export`, `vm import`, `cp --user`, `cp --key`, `--firecracker-bin` -- ✅ Missing command categories added: `mvm snapshot` (top-level), `mvm env`, `mvm run`, `vm exec`, `completion powershell`, `init --skip-host`, `key import` -- ✅ Missing flags documented: `vm create --console`, `--force`, `--vsock-port`; `config reset --force`; `volume create --shareable`; `image import --version` -- ✅ Snapshot moved from under `vm` to its own top-level section -- ✅ Cross-referenced ALL test classes against actual test code -- ✅ Fixed wrong class references: `TestVMCreate`→🔴 Missing (tests don't exist), `TestKeyAddOverwrite`→`TestKeyImportOverwrite`, `TestImageLifecycle`→`TestImageRemove`, `TestImageInspectTree`→🔴 Missing (doesn't exist), `TestEnvApply`→🔴 Missing (doesn't exist) -- ✅ Fixed `TestVMNocloudNetPort` reference (was incorrectly attributed to `TestVMCloudInit` only) -- ✅ Recalculated summary statistics (420 total scenarios, 22 🔴 Missing) -- ✅ Added all undocumented test classes (34+) across network nftables, volume hotplug/detach, image/kernel/bin dependency, env ls/diff, full-journeys (13 classes), and cache prune/clean - -**June 27, 2026 — Mass gap-fill:** -- ✅ **VM (8 new tests)**: `--console` (L2), `--count N` (L2), `--atomic --count N` (L2), `--count --ip`/`--mac`/`-1` error (L1), `--force` (L2), `--vsock-port` (L2) -- ✅ **exec (8 tests, new file)**: Basic command (L2), interactive shell (L2), `--port` (L2), `--timeout` (L1), `--user` (L2), nonexistent VM (L1), invalid port (L1), empty command (L2) -- ✅ **snapshot (5 new tests)**: `--name` (L2), `--pause` (L2), `inspect` (L1), `inspect --json` (L2), `restore --network` (L2) -- ✅ **volume (1 new test)**: `--shareable` (L2) -- ✅ **key (1 new test)**: `import ` (L2) -- ✅ **image (1 new test)**: `import --version` (L2) -- ✅ **21 of 22 gaps closed** (only `image inspect --tree` remains 🔴 — phantom flag) +**QA audit — 2026-08-23:** + +- Recounted the current command rows directly: 431 total, with the exact status totals above. +- Added the v0.3.0 security overlay so legacy CLI breadth is not represented as clean-break release evidence. +- Confirmed that Jailer/cgroup tests do not yet verify root records, exact process identity, cross-UID denial, private + mount/network namespaces, or complete cleanup. +- Confirmed that policy/full-journey tests still encode superseded dual-firewall and implicit same-network behavior. +- Confirmed that VM-to-VM typed exec, final marker-only sudoers denial, repeated zero-leak runs, and aggregated teardown + failure reporting remain missing. **System test machine requirements:** To run the full system test suite with zero skips, the dedicated test machine must have: @@ -707,10 +728,13 @@ To run the full system test suite with zero skips, the dedicated test machine mu | Network access | Image pull, bin pull, remote listing | Required for HTTP downloads | | `/dev/kvm` | All VM creation tests | KVM-capable CPU + `kvm` kernel module | | `mvm` group membership | All privileged operations | `sudo usermod -aG mvm $USER` | -| `~/.local/bin/mvm` binary | host clean/reset tests | Build with `scripts/build.sh release` | +| Exact `/usr/local/bin/mvm` plus separate candidate | v0.3 qualification | Install the release candidate through `host install-system`; keep the outer controller separate. | -**Skip behavior on dedicated vs developer machines:** -- **Dedicated machine** (all deps installed): Skips nearly never trigger. Skip ratio ≈ 0%. -- **Developer machine** (missing deps): Tests skip gracefully with clear `# Skip-reason:` explaining what to install. +**Prerequisite behavior on dedicated vs developer machines:** +- **Dedicated qualification machine:** every prerequisite is mandatory; release evidence requires zero skips, XFAILs, + XPASSes, deselections, collection errors, or omitted domains. +- **Developer machine:** a focused run may fail prerequisite admission when dependencies are absent. Such a run is useful + for diagnosis but cannot serve as release evidence. -**431 scenarios documented — 1 remaining 🔴 (phantom `image inspect --tree` flag, not a real CLI command). 21 real gaps closed.** +**431 transitional CLI scenarios are documented. Four rows are marked missing; the v0.3.0 qualification overlay lists +additional release-security gaps that must close before signoff.** From f3a1ba12995be42417c3bbbaddb4c53f0b52d271 Mon Sep 17 00:00:00 2001 From: AlanD20 Date: Sun, 23 Aug 2026 17:13:59 +0200 Subject: [PATCH 0038/1457] test: fail domains on runner cleanup errors --- scripts/run-system-tests.py | 28 ++++- scripts/tests/test_run_system_tests.py | 148 +++++++++++++++++++++++++ 2 files changed, 171 insertions(+), 5 deletions(-) diff --git a/scripts/run-system-tests.py b/scripts/run-system-tests.py index ffbba744..34283638 100755 --- a/scripts/run-system-tests.py +++ b/scripts/run-system-tests.py @@ -1022,9 +1022,27 @@ def provision_t2(vm_name: str, mvm_version: str) -> None: ) -def destroy_vm(vm_name: str) -> None: +def destroy_vm(vm_name: str, *, check: bool = True) -> None: """Destroy a runner VM.""" - mvm("vm", "rm", vm_name, "--force", timeout=60, check=False) + mvm("vm", "rm", vm_name, "--force", timeout=60, check=check) + + +def _record_runner_destruction( + result: dict[str, Any], + vm_name: str, +) -> None: + """Make mandatory runner destruction part of the domain result.""" + try: + destroy_vm(vm_name) + except Exception as exc: + result["passed"] = False + cleanup_failure = f"runner VM destruction failed for {vm_name}: {exc}" + primary_output = result["output"].rstrip() + result["output"] = ( + f"{primary_output}\n{cleanup_failure}" + if primary_output + else cleanup_failure + ) def _bounded_builder_pull_output( @@ -1172,7 +1190,7 @@ def _build_base_image(mvm_version: str, *, rebuild: bool = False) -> str: return img_tag # Clean up any leftover builder VM from a previous aborted run - destroy_vm(BASE_VM_NAME) + destroy_vm(BASE_VM_NAME, check=False) log(f" Creating builder VM '{BASE_VM_NAME}'...") mvm( @@ -1532,7 +1550,7 @@ def run_tier1_domain( except Exception as e: result["output"] = str(e) finally: - destroy_vm(vm_name) + _record_runner_destruction(result, vm_name) return result @@ -1581,7 +1599,7 @@ def run_tier2_domain( except Exception as e: result["output"] = str(e) finally: - destroy_vm(vm_name) + _record_runner_destruction(result, vm_name) return result diff --git a/scripts/tests/test_run_system_tests.py b/scripts/tests/test_run_system_tests.py index a307bd95..774739a6 100644 --- a/scripts/tests/test_run_system_tests.py +++ b/scripts/tests/test_run_system_tests.py @@ -1166,6 +1166,154 @@ def fake_mvm(*args: str, **_kwargs: object) -> subprocess.CompletedProcess[str]: assert destroyed == [vm_name] +def test_destroy_vm_requires_success( + orchestrator: ModuleType, + monkeypatch: pytest.MonkeyPatch, +) -> None: + calls: list[tuple[tuple[str, ...], dict[str, object]]] = [] + + def fake_mvm(*args: str, **kwargs: object) -> subprocess.CompletedProcess[str]: + calls.append((args, kwargs)) + return subprocess.CompletedProcess(args, 0, stdout="", stderr="") + + monkeypatch.setattr(orchestrator, "mvm", fake_mvm) + + orchestrator.destroy_vm("t1-cli-fixed") + + assert calls == [ + ( + ("vm", "rm", "t1-cli-fixed", "--force"), + {"timeout": 60, "check": True}, + ) + ] + + +@pytest.mark.parametrize( + ("runner_name", "provision_name", "tier"), + [ + ("run_tier1_domain", "provision_t1", 1), + ("run_tier2_domain", "provision_t2", 2), + ], + ids=["tier1", "tier2"], +) +def test_remote_domain_destruction_failure_overrides_success( + orchestrator: ModuleType, + monkeypatch: pytest.MonkeyPatch, + runner_name: str, + provision_name: str, + tier: int, +) -> None: + vm_name = f"t{tier}-cli-fixed" + events: list[str] = [] + monkeypatch.setattr(orchestrator, "_unique_name", lambda _prefix: vm_name) + monkeypatch.setattr( + orchestrator, + provision_name, + lambda provisioned, version: events.append( + f"provision:{provisioned}:{version}" + ), + ) + + def fake_pytest( + tested_vm: str, + _test_files: list[str], + *, + timeout: int, + ) -> tuple[subprocess.CompletedProcess[str], None]: + events.append(f"pytest:{tested_vm}:{timeout}") + return ( + subprocess.CompletedProcess( + ["pytest"], + 0, + stdout="useful pytest stdout\n", + stderr="", + ), + None, + ) + + def fail_destroy(destroyed: str) -> None: + events.append(f"destroy:{destroyed}") + raise RuntimeError("synthetic runner destruction refusal") + + monkeypatch.setattr(orchestrator, "_run_remote_pytest", fake_pytest) + monkeypatch.setattr(orchestrator, "destroy_vm", fail_destroy) + + result = getattr(orchestrator, runner_name)( + "cli", + ["tests/system/cli/test_cli.py"], + "0.3.0", + ) + + assert result["passed"] is False + assert "useful pytest stdout" in result["output"] + assert "runner VM destruction failed" in result["output"] + assert "synthetic runner destruction refusal" in result["output"] + assert events == [ + f"provision:{vm_name}:0.3.0", + f"pytest:{vm_name}:{660 if tier == 1 else 960}", + f"destroy:{vm_name}", + ] + + +@pytest.mark.parametrize( + ("runner_name", "provision_name", "tier"), + [ + ("run_tier1_domain", "provision_t1", 1), + ("run_tier2_domain", "provision_t2", 2), + ], + ids=["tier1", "tier2"], +) +def test_remote_domain_preserves_primary_failure_when_destruction_fails( + orchestrator: ModuleType, + monkeypatch: pytest.MonkeyPatch, + runner_name: str, + provision_name: str, + tier: int, +) -> None: + vm_name = f"t{tier}-cli-fixed" + events: list[str] = [] + monkeypatch.setattr(orchestrator, "_unique_name", lambda _prefix: vm_name) + monkeypatch.setattr( + orchestrator, + provision_name, + lambda provisioned, version: events.append( + f"provision:{provisioned}:{version}" + ), + ) + + def fail_pytest( + tested_vm: str, + _test_files: list[str], + *, + timeout: int, + ) -> tuple[subprocess.CompletedProcess[str], str | None]: + events.append(f"pytest:{tested_vm}:{timeout}") + raise RuntimeError("synthetic primary pytest failure") + + def fail_destroy(destroyed: str) -> None: + events.append(f"destroy:{destroyed}") + raise RuntimeError("synthetic runner destruction refusal") + + monkeypatch.setattr(orchestrator, "_run_remote_pytest", fail_pytest) + monkeypatch.setattr(orchestrator, "destroy_vm", fail_destroy) + + result = getattr(orchestrator, runner_name)( + "cli", + ["tests/system/cli/test_cli.py"], + "0.3.0", + ) + + assert result["passed"] is False + assert "synthetic primary pytest failure" in result["output"] + assert "runner VM destruction failed" in result["output"] + assert "synthetic runner destruction refusal" in result["output"] + assert events == [ + f"provision:{vm_name}:0.3.0", + f"pytest:{vm_name}:{660 if tier == 1 else 960}", + f"destroy:{vm_name}", + ] + + def test_builder_assets_are_pulled_in_separate_sequential_execs( orchestrator: ModuleType, monkeypatch: pytest.MonkeyPatch, From dc2473b5e2f291cba8a1fc9bcfa8e260e71a6828 Mon Sep 17 00:00:00 2001 From: AlanD20 Date: Sun, 23 Aug 2026 17:21:14 +0200 Subject: [PATCH 0039/1457] test: contain timed-out domain workers --- scripts/run-system-tests.py | 63 ++++++++--- scripts/tests/test_run_system_tests.py | 139 +++++++++++++++++++++++++ 2 files changed, 187 insertions(+), 15 deletions(-) diff --git a/scripts/run-system-tests.py b/scripts/run-system-tests.py index 34283638..9072bdff 100755 --- a/scripts/run-system-tests.py +++ b/scripts/run-system-tests.py @@ -44,7 +44,7 @@ import tempfile import time import uuid -from concurrent.futures import ThreadPoolExecutor +from concurrent.futures import Future, ThreadPoolExecutor from pathlib import Path from typing import Any @@ -1636,6 +1636,34 @@ def run_tier3_domain( # ============================================================================ +def _timed_out_domain_result( + future: Future[dict[str, Any]], + domain: str, + tier: int, + timeout: int, +) -> dict[str, Any]: + """Build one failed result after the timed-out worker is contained.""" + output = [f"Domain timed out after {timeout}s"] + if future.cancelled(): + output.append("Worker was canceled before it started") + else: + try: + late_result = future.result() + except Exception as exc: + output.append(f"Late worker error: {exc}") + else: + late_output = str(late_result.get("output", "")).rstrip() + if late_output: + output.append(f"Late worker output:\n{late_output}") + + return { + "domain": domain, + "tier": tier, + "passed": False, + "output": "\n".join(output), + } + + def run_domains( domains: dict[str, list[str]], tier: int, @@ -1652,6 +1680,7 @@ def run_domains( results: list[dict[str, Any]] = [] # Overall timeout per domain: 15 minutes. domain_timeout = 900 + timed_out: dict[Future[dict[str, Any]], str] = {} pool = ThreadPoolExecutor(max_workers=workers) try: @@ -1669,21 +1698,15 @@ def run_domains( return_when=concurrent.futures.FIRST_COMPLETED, ) if not done: - # Timeout reached — no futures completed within domain_timeout + # Running threads cannot be force-canceled safely. Cancel work + # that has not started, then join active workers before moving on. log( - f" [TIMEOUT] {len(remaining)} domain(s) timed out after {domain_timeout}s — moving on" + f" [TIMEOUT] {len(remaining)} domain(s) timed out after " + f"{domain_timeout}s — containing workers before continuing" ) for future in remaining: - domain = future_map[future] - results.append( - { - "domain": domain, - "tier": tier, - "passed": False, - "output": f"Domain timed out after {domain_timeout}s", - } - ) - log(f" [TIMEOUT] {domain}") + future.cancel() + timed_out[future] = future_map[future] break for future in done: domain = future_map[future] @@ -1705,8 +1728,18 @@ def run_domains( ) log(f" [ERROR] {domain}: {e}") finally: - # Don't wait for timed-out threads — they'll be daemon-killed on exit - pool.shutdown(wait=False, cancel_futures=True) + pool.shutdown(wait=True, cancel_futures=True) + + for future, domain in sorted(timed_out.items(), key=lambda item: item[1]): + domain_result = _timed_out_domain_result( + future, + domain, + tier, + domain_timeout, + ) + results.append(domain_result) + log(f" [TIMEOUT] {domain}") + _print_failure(domain, domain_result) return results diff --git a/scripts/tests/test_run_system_tests.py b/scripts/tests/test_run_system_tests.py index 774739a6..6d2fdee9 100644 --- a/scripts/tests/test_run_system_tests.py +++ b/scripts/tests/test_run_system_tests.py @@ -2,6 +2,7 @@ import argparse import ast +import concurrent.futures import importlib.util import json import os @@ -9,6 +10,7 @@ import shutil import subprocess import sys +import threading from pathlib import Path from types import ModuleType @@ -1314,6 +1316,143 @@ def fail_destroy(destroyed: str) -> None: ] +def test_timed_out_parallel_worker_is_joined_and_late_failures_are_preserved( + orchestrator: ModuleType, + monkeypatch: pytest.MonkeyPatch, +) -> None: + worker_started = threading.Event() + release_worker = threading.Event() + worker_finished = threading.Event() + timeout_reported = threading.Event() + started_domains: list[str] = [] + + def runner( + domain: str, + _test_files: list[str], + _mvm_version: str, + _push: bool, + ) -> dict[str, object]: + started_domains.append(domain) + worker_started.set() + assert release_worker.wait(timeout=2) + worker_finished.set() + return { + "domain": domain, + "tier": 1, + "passed": False, + "output": ( + "late primary pytest failure\n" + "runner VM destruction failed: late cleanup refusal" + ), + } + + def force_timeout( + futures: set[concurrent.futures.Future[dict[str, object]]], + *, + timeout: int, + return_when: str, + ) -> tuple[set[concurrent.futures.Future[dict[str, object]]], set[concurrent.futures.Future[dict[str, object]]]]: + assert timeout == 900 + assert return_when == concurrent.futures.FIRST_COMPLETED + assert worker_started.wait(timeout=2) + return set(), set(futures) + + def capture_log(message: str) -> None: + if "[TIMEOUT]" in message: + timeout_reported.set() + + monkeypatch.setattr(concurrent.futures, "wait", force_timeout) + monkeypatch.setattr(orchestrator, "log", capture_log) + + with concurrent.futures.ThreadPoolExecutor(max_workers=1) as caller_pool: + invocation = caller_pool.submit( + orchestrator.run_domains, + {"slow": ["slow.py"], "queued": ["queued.py"]}, + 1, + runner, + 1, + "0.3.0", + ) + assert timeout_reported.wait(timeout=2) + assert worker_finished.is_set() is False + release_worker.set() + results = invocation.result(timeout=2) + + assert worker_finished.is_set() is True + assert started_domains == ["slow"] + by_domain = {result["domain"]: result for result in results} + assert set(by_domain) == {"queued", "slow"} + assert by_domain["slow"]["passed"] is False + assert "Domain timed out after 900s" in by_domain["slow"]["output"] + assert "late primary pytest failure" in by_domain["slow"]["output"] + assert "runner VM destruction failed: late cleanup refusal" in by_domain["slow"]["output"] + assert by_domain["queued"]["passed"] is False + assert "Domain timed out after 900s" in by_domain["queued"]["output"] + assert "canceled before it started" in by_domain["queued"]["output"] + + +def test_timed_out_parallel_worker_preserves_late_exception( + orchestrator: ModuleType, + monkeypatch: pytest.MonkeyPatch, +) -> None: + worker_started = threading.Event() + release_worker = threading.Event() + timeout_reported = threading.Event() + + def runner( + _domain: str, + _test_files: list[str], + _mvm_version: str, + _push: bool, + ) -> dict[str, object]: + worker_started.set() + assert release_worker.wait(timeout=2) + raise RuntimeError("late worker exception") + + def force_timeout( + futures: set[concurrent.futures.Future[dict[str, object]]], + *, + timeout: int, + return_when: str, + ) -> tuple[set[concurrent.futures.Future[dict[str, object]]], set[concurrent.futures.Future[dict[str, object]]]]: + assert timeout == 900 + assert return_when == concurrent.futures.FIRST_COMPLETED + assert worker_started.wait(timeout=2) + return set(), set(futures) + + def capture_log(message: str) -> None: + if "[TIMEOUT]" in message: + timeout_reported.set() + + monkeypatch.setattr(concurrent.futures, "wait", force_timeout) + monkeypatch.setattr(orchestrator, "log", capture_log) + + with concurrent.futures.ThreadPoolExecutor(max_workers=1) as caller_pool: + invocation = caller_pool.submit( + orchestrator.run_domains, + {"slow": ["slow.py"]}, + 1, + runner, + 1, + "0.3.0", + ) + assert timeout_reported.wait(timeout=2) + release_worker.set() + results = invocation.result(timeout=2) + + assert results == [ + { + "domain": "slow", + "tier": 1, + "passed": False, + "output": ( + "Domain timed out after 900s\n" + "Late worker error: late worker exception" + ), + } + ] + + def test_builder_assets_are_pulled_in_separate_sequential_execs( orchestrator: ModuleType, monkeypatch: pytest.MonkeyPatch, From d7ef1474ade21d7e01d8186888d128376fdca34f Mon Sep 17 00:00:00 2001 From: AlanD20 Date: Wed, 26 Aug 2026 11:38:52 +0200 Subject: [PATCH 0040/1457] docs: queue reboot lifecycle reconciliation --- CHANGELOG.md | 3 +++ ...owned-single-binary-privileged-dispatch.md | 15 +++++++++++ docs/improvements/IMPROVEMENTS_009.md | 14 +++++++--- tasks/plan.md | 27 +++++++++++++++++++ tasks/todo.md | 14 ++++++++++ 5 files changed, 69 insertions(+), 4 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 83f6ce36..7e4981ae 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -203,6 +203,9 @@ The following items are release blockers and are intentionally not described abo typed contract, and propagate every host mutation/state-write failure instead of reporting partial setup as success. - Make launch, abort, cleanup, reconciliation, snapshots, live volumes, cgroups, and firewall policy crash-consistent and fail closed under ownership, PID-reuse, path-race, and partial-failure conditions. +- Reconcile `mvm vm ls`, `mvm vm ps`, and `mvm vm inspect` against root-owned process identity after host reboot instead + of reporting stale SQLite status. Preserve the configured vsock device across start, reboot, and snapshot restore so + host-to-guest `mvm exec` remains available after every supported relaunch path. - Complete adversarial L1 coverage, Python `tests/system/` clean-install/fault qualification, leak audits, full CI, and the final documentation accuracy review. Go tests support but do not replace CLI-level L2 release signoff. - Finish Tier 3 clean-host inventory and ownership-safe cleanup; unify every host-direct CLI call, preserve pre-existing diff --git a/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md b/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md index a26aaf43..60428445 100644 --- a/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md +++ b/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md @@ -360,6 +360,21 @@ Startup and pre-launch reconciliation compare root instance records, processes, not trust the user database as root authorization. Operations are idempotent and either converge to a known state or return a precise error without deleting potentially mounted user data. +Read-side status is also reconciled. Before `mvm vm ls`, `mvm vm ps`, or `mvm vm inspect` filters or renders a status, +the normal process obtains a bounded, typed batch observation from root authority. Root verifies the recorded host boot +ID, PID, start ticks, process credentials, cgroup, namespace, and executable identity. A host boot-ID mismatch proves the +old process identity is not live without contacting the guest. The read path never opens one sudo process per VM and +never dials the Firecracker API or guest agent as a liveness tick. It may update the user database as a projection only +after it receives a complete authoritative result; SQLite remains non-authoritative. A boot-ID mismatch projects +`stopped`; a missing or mismatched process during the same boot projects `crashed`. Failure to obtain an authoritative +observation fails the read instead of returning stale status. + +Every create, start, reboot, and snapshot-restore path prepares the same complete typed relaunch state. If the VM has a +persisted vsock configuration, the root launch request includes only its validated CID; root derives the fixed runtime +UDS leaf. The normal process retains the agent port and token for the post-launch client probe instead of exposing them +to root. A repository failure aborts launch instead of silently omitting the device. Root removes an old fixed vsock +socket only after authority proves that no live registered process owns it. + ## Alternatives Considered ### Separate privileged helper binary diff --git a/docs/improvements/IMPROVEMENTS_009.md b/docs/improvements/IMPROVEMENTS_009.md index 7465340f..43d41050 100644 --- a/docs/improvements/IMPROVEMENTS_009.md +++ b/docs/improvements/IMPROVEMENTS_009.md @@ -10,7 +10,7 @@ > | Resource accounting table + admission control | ❌ Not implemented — no `resource_accounting` or `vm_resource_allocations` tables in schema | > | Event log table + `mvm events` command | ❌ Not implemented — no `event_log` table in schema, no events CLI command. Audit logging exists (`internal/lib/logging/audit.go`) but writes flat file, not queryable event table. | > | Rate-limited VM creation | ❌ Not implemented — no semaphore/concurrency limiter in VM creation pipeline | -> | `mvm host reconcile` for crash recovery | ❌ Not implemented — no such CLI command exists | +> | Root-authoritative crash/status reconciliation | ❌ Not implemented — promoted to v0.3.0 Task 15 | > | sysctl tuning in `mvm host init` | ⚠️ Partial — `EnableIPForward()` + `PersistSysctl()` write `net.ipv4.ip_forward=1`. No tier system or multi-sysctl profile. | > | WAL checkpointing + backup rotation | ❌ Not implemented — no periodic checkpoint or snapshot rotation logic | > @@ -157,13 +157,19 @@ Default: 5 concurrent creates. Configurable via `settings.max_parallel_vm_create --- -## 6. Crash Recovery (`mvm host reconcile`) (Not Implemented) +## 6. Crash and status reconciliation (Not Implemented) -Scans for three types of orphaned state: +The v0.3.0 design in ADR-0016 and `tasks/plan.md` supersedes the original user-database/PID scan proposed here. Root +authority verifies boot ID and complete process identity. `mvm vm ls`, `mvm vm ps`, and `mvm vm inspect` reconcile in +bounded batches before filtering or rendering, so a host reboot cannot leave a VM displayed as running. + +This is not a guest heartbeat. List commands do not dial Firecracker, SSH, or the vsock agent, and they do not start one +sudo process or goroutine per VM. Guest readiness remains the responsibility of the command that needs it. The original +orphan classes remain part of the broader reconciliation operation: | Phase | What It Checks | Action | |-------|---------------|--------| -| 1 | Running VMs with dead PIDs | Mark as `crashed`, release resources | +| 1 | Authority records with no verified live process | Reconcile observed status and recover owned resources | | 2 | Active network leases with no VM | Release lease back to pool | | 3 | TAP devices not owned by any VM | Delete TAP via `system.RunCmd` | diff --git a/tasks/plan.md b/tasks/plan.md index 3303e19d..60a30906 100644 --- a/tasks/plan.md +++ b/tasks/plan.md @@ -261,6 +261,33 @@ The public policy families are: Traffic policy never grants vsock execution. Exec policy never grants IP connectivity. Same-network placement grants nothing implicitly. +### Daemonless status reconciliation and relaunch parity + +`mvm vm ls`, `mvm vm ps`, and `mvm vm inspect` must report observed runtime state rather than replaying a stale SQLite +status. Before status filtering or rendering, the API obtains a bounded batch observation from the root VM authority. +The receiver verifies boot ID and the complete recorded process identity. A host reboot therefore classifies every old +process identity as non-live without waiting for a VM, Firecracker API, SSH, or agent timeout. The user database may be +updated after the authoritative observation, but it never decides whether a privileged process is live. A boot-ID +mismatch projects `stopped`; a missing or mismatched process in the current boot projects `crashed`. The list API must +return an error when authority cannot complete the observation, not silently return stale rows. + +This is a local metadata operation, not a heartbeat. The implementation must not start one sudo process, open one +control connection, or launch one goroutine per VM. L1 benchmarks cover 1,000 authority records and set a regression +budget from the reviewed baseline. L2 proves correctness after a real host reboot. Guest readiness remains an explicit +probe performed by commands such as `mvm exec` and `mvm ssh`, not by list output. + +All relaunch paths use one complete typed relaunch state. Start, reboot, and snapshot restore must load the persisted +vsock relation when present and distinguish a valid absence from a repository failure. The root launch request contains +only the validated CID and derives the fixed runtime UDS leaf. The normal process retains the agent port and token for +later agent client probes. Receiver-owned cleanup removes a stale fixed socket only after process authority proves no +live instance owns it. L1 verifies the generated Firecracker config. L2 runs `mvm exec` after stop/start, reboot, host +reboot plus start, and snapshot restore. + +The 2026-08-26 read-only reproduction isolated the current defect: `VMStart` and `VMReboot` omit the `vsock` relation, +while `vmRespawnFirecracker` emits a vsock device only when `VMItem.Vsock` is populated. The restarted Firecracker config +therefore had no vsock entry, its old host UDS remained stale, and the active guest agent had no host endpoint. This +evidence defines the regression test. It is not approval for a best-effort one-line enrichment patch. + ## Dependency Order ```text diff --git a/tasks/todo.md b/tasks/todo.md index 5a71b798..cf9144f5 100644 --- a/tasks/todo.md +++ b/tasks/todo.md @@ -324,12 +324,24 @@ under `internal/service/jailer/`. No handler, transport, CLI, API, core-domain, - [ ] Add root-authoritative startup/pre-launch reconciliation for records, processes, cgroups, namespaces, links, chroots, mounts, and nftables generations. +- [ ] Reconcile observed runtime status before `mvm vm ls`, `mvm vm ps`, and `mvm vm inspect` filter or render. Use a + bounded typed root-authority batch, make a host boot-ID mismatch immediately non-live, update SQLite only as a + `stopped` projection, classify a same-boot missing/mismatched process as `crashed`, and never perform a per-VM sudo, + Firecracker API, SSH, or guest-agent liveness tick. Change the list API to return an error rather than stale rows when + authority cannot complete the observation. +- [ ] Benchmark 1,000 authority records in L1 and enforce the reviewed regression budget without one goroutine, process, + or control connection per VM. - [ ] Add root-owned global capacity reservation so separate CLI processes and users cannot over-admit the host. - [ ] Make create/start/stop/remove/reboot/snapshot/restore/volume paths transactionally abortable and retryable. +- [ ] Build one complete typed relaunch state for create/start/reboot/snapshot restore. Preserve a configured vsock CID, + let root derive the fixed runtime UDS leaf, retain agent port/token only in the normal process, distinguish valid + absence from lookup failure, and clean an old fixed socket only after verified process absence. - [ ] Preserve exact release identity and jail-visible paths across snapshot restore and live hotplug. - [ ] Harden cgroup readiness, membership, values, and process-aware cleanup. - [ ] Remove process-local semaphore admission and user-DB-as-root-authority designs. - [ ] L1/L2 fault injection covers kill at every effect stage, reboot, PID reuse, partial cleanup, and repeated reconcile. +- [ ] L1 proves relaunch emits the vsock device and L2 proves `mvm exec` after stop/start, reboot, host reboot plus start, + and snapshot restore. L2 also proves `ls`/`ps` stop reporting stale running state after a host reboot. ### Task 16: Complete global structured JSON output @@ -371,6 +383,8 @@ under `internal/service/jailer/`. No handler, transport, CLI, API, core-domain, `test_vm_fresh_env.py` twice under aliases. - [x] Make mirror validation reject pull errors and HTTP fallback; only a verified local-mirror read is a cache hit. - [ ] Run the complete T1/T2/T3 matrix plus clean-host installation and reboot/reconciliation qualification. +- [ ] Add the installed-CLI reboot lifecycle journey: stale `running` state is reconciled before list filtering, then + stop/start and reboot both restore SSH and host-to-guest `mvm exec` without a daemon or leaked vsock socket. - [ ] Update `tests/system/COVERAGE_MATRIX.md` and attach commands/results to the release report. ### Task 18: Run repository CI in exact order From 76ba2adcb76ba8d6a216d424650431ab86b35b40 Mon Sep 17 00:00:00 2001 From: AlanD20 Date: Wed, 26 Aug 2026 12:20:46 +0200 Subject: [PATCH 0041/1457] docs: centralize agent instructions --- .opencode/agent/architect.md | 152 +++------------- .opencode/agent/doc-tracer.md | 312 --------------------------------- .opencode/agent/engineer.md | 177 +++---------------- .opencode/agent/qa-engineer.md | 144 +++------------ AGENTS.md | 126 ++++++++----- 5 files changed, 161 insertions(+), 750 deletions(-) delete mode 100644 .opencode/agent/doc-tracer.md diff --git a/.opencode/agent/architect.md b/.opencode/agent/architect.md index bc9c2bc5..745cb68f 100644 --- a/.opencode/agent/architect.md +++ b/.opencode/agent/architect.md @@ -1,13 +1,7 @@ --- description: >- - Use when you need deep technical discussion, architectural brainstorming, - critical analysis of design decisions, or to manage the full domain - implementation lifecycle. Challenges assumptions, pushes back on weak - decisions, explores alternatives, and orchestrates work by spawning - subagents (`engineer` for production Go code, `qa-engineer` for test - and release, `explore` for research). Never writes production Go code - itself — but OWNS all documentation directly (CONTEXT.md, AGENTS.md, - docs/, .opencode/, *.md). Plans, analyzes, delegates, and writes docs. + Plans mvmctl architecture, delegates implementation, owns documentation, + and reviews completed diffs. Never writes production code or tests. mode: all temperature: 0.65 permission: @@ -22,6 +16,7 @@ permission: "git diff *": allow "git status *": allow "git log *": allow + "git show *": allow "go build *": allow "go vet *": allow "rm *": deny @@ -31,7 +26,6 @@ permission: "git reset --hard *": deny "git restore *": deny "git stash *": deny - "git show *": deny "git branch -D *": deny "git rebase --abort *": deny "git merge --abort *": deny @@ -43,128 +37,32 @@ permission: "git worktree prune *": deny --- -You are the **architect** for the mvmctl project. You are the user's main point of -contact. You do NOT write production Go code — you think, analyze, plan, delegate -implementation to specialized subagents, and OWN all documentation. +You are the mvmctl project architect and the user's primary contact. -## CRITICAL: Question vs Implementation +Read `AGENTS.md` first. It defines the instruction order, role ownership, approval gate, host safety rules, pstack +policy, verification expectations, and architecture constraints. Read only the task-specific references that it names. -A user asking a question — even a leading, suggestive, or "what about X?" question — is ASKING A QUESTION. They want an ANSWER: clarification, reasoning, evidence, trade-offs, counterarguments, or a simple factual response. They do NOT want implementation. They do NOT want you to spawn `engineer` or `qa-engineer`. They do NOT want you to write code. They do NOT want you to assume they want "well actually let me explore that". +## Responsibilities -**Exception:** If answering the question requires reading multiple large files or deep codebase exploration, you may spawn `explore` for reading only. Do not spawn any agent that writes code or tests. +- Answer questions with evidence. A question does not authorize implementation. +- Inspect existing interfaces and two or three analogous paths before proposing a design. +- Present the files, trade-offs, effect boundaries, and verification plan. Wait for explicit approval before dispatching + implementation. +- Delegate production Go code and L0 or L1 Go tests to `engineer`. +- Delegate Python system tests, the system-test orchestrator, coverage, and release verification to the OpenCode `qa` + agent. +- Write plans, ADRs, and documentation directly. Do not write executable scripts, production code, or tests. +- Read every implementation diff before accepting it. Check interfaces, structs, imports, subprocess boundaries, and + ownership against `AGENTS.md` and the applicable ADRs. -**Unless the user explicitly says "implement", "build", "write code", "do it", "make it", "go ahead", or gives unambiguous approval to execute, YOU ANSWER THE QUESTION AND STOP.** +## Safety -If you are unsure whether a prompt is a question or an instruction, default to ANSWERING. Err on the side of talking, not doing. +Preserve all unrelated worktree changes. Candidate builds stay under `dist/`, the worktree, or a task-specific temporary +directory. Never direct an agent to replace an installed host binary, change sudoers, or run Tier 3 unless the user has +explicitly authorized that exact action. -## CRITICAL: Lost context → STOP +Use the controller and candidate separation from `docs/development/HOW_TO_RUN_SYSTEM_TESTS.md`. Do not invent another +test or release workflow in this prompt. -When implementing, watch for signals that you lack context: -- The user hints at regrouping or realigning ("let's step back", "that's not what I meant", "you're missing context"). -- You realize you're guessing about requirements, architecture, or intent. -- The user corrects you on the same task more than once. - -When any of these happen, you are in a broken state. STOP ALL WORK immediately. Do not attempt to fix, redo, or push forward. Switch to planning mode: - -1. Acknowledge the misalignment. -2. Summarize what you think the goal is. -3. Ask the user to confirm or correct your understanding. -4. Propose a concrete plan before doing any work. -5. Wait for explicit approval ("go ahead", "yes", "do it") before resuming implementation. - -You lost the plot — get realigned first. - -## Your role - -1. **Primary interface** — You are the only agent that talks to the user. Subagents report - to you, you report to the user. -2. **Brainstormer** — Challenge assumptions, push back on weak decisions, explore alternatives. -3. **Orchestrator** — Spawn `engineer` for production Go code, `qa-engineer` for test/release, - `explore` for research. Never write production Go code yourself. -4. **Documentation + scripts owner** — You write and maintain ALL documentation directly. This includes: - `CONTEXT.md`, `AGENTS.md`, `docs/**/*.md`, `.opencode/**/*.md`, `README.md`, `CHANGELOG.md`, - and any other `.md` files. You also own `scripts/` (build, dev, and utility scripts). - You have the deepest project knowledge — docs and scripts are architect territory, not engineer territory. -5. **Domain implementation manager** — Full lifecycle: **catalog** (read existing code/docs to understand what exists) → **plan** (propose approach) → **user approval** (wait for explicit "go ahead") → **execute** (spawn subagents). -6. **Deep thinker** — Engage in thorough analysis of architectural decisions and trade-offs. - Consult `CONTEXT.md` and `docs/adr/` for domain language and architectural decisions. - -## Agent boundaries - -| Work | Delegate to | -|---|---|---| -| Production Go code (`cmd/`, `internal/`, `pkg/`, `go.mod`) | `engineer` agent | -| Tests (`*_test.go`, `tests/system/`) + release | `qa-engineer` agent | -| Research / codebase exploration | `explore` agent | -| All documentation (`*.md`, `docs/`, `.opencode/`, `CONTEXT.md`, `AGENTS.md`) + `scripts/` | **architect** (directly — no delegation) | - -## Subagent spawning rules - -1. **Tell the agent its role** — Open every spawn with: "You are the `engineer`/`qa-engineer`/`explore` agent." -2. **State WHAT, not HOW** — Describe the goal and constraints. Do NOT include Go code, - type hints, or implementation details. The subagent knows its patterns. -3. **List files** — Source files to read + target files to modify. -4. **ALWAYS include build output path** — Every engineer/qa-engineer spawn MUST include in the goal: - `"Build to ~/.local/bin/mvm (./scripts/build.sh --output ~/.local/bin/mvm)"` + - `"Set MVM_ASSET_MIRROR=~/.cache/mvm-asset-mirror before running mvm"`. - These are NOT optional. The subagents do NOT have this info in their own instructions. - For `explore` spawns, build/env directives are optional (not needed for research). -5. **Repeat critical boundary** — "Do not touch any file outside the target list." -6. **Scope: only your changes** — Use `git diff` and `git status` ONLY to review files you added/modified. Never inspect, stash, clean, revert, or touch other people's untracked/staged/unstaged changes. They are not your concern. - -## File reading policy - -- **Read files directly** for quick checks (one function, small analysis). -- **Delegate reading to subagents** for multiple large files or deep exploration. -- You focus on thinking and deciding. Subagents focus on reading and doing. - -## Running the binary (REQUIRED) - -The mvm binary MUST be built to `~/.local/bin/mvm` — this path has passwordless sudo via sudoers rules. -The `MVM_ASSET_MIRROR=~/.cache/mvm-asset-mirror` env var MUST be set before any `mvm` command. -You MUST propagate both of these requirements to every subagent spawn. See Subagent Spawning Rule #4. - -```bash -# Build (exact command - do not deviate): -./scripts/build.sh --output ~/.local/bin/mvm - -# Run (exact env - do not deviate): -MVM_ASSET_MIRROR=~/.cache/mvm-asset-mirror mvm -``` - -If a subagent does NOT produce a binary at `~/.local/bin/mvm` or does NOT set the env var, -the task is incomplete. Reject the result and re-spawn with explicit instructions. - -## Plan approval protocol - -The architect MUST verify these before approving any implementation plan: - -1. **Patterns cross-check** — Read the EXISTING interface/pattern the proposal touches. Does the new method match the naming and shape of existing methods? (e.g., `Backend.SetupSSH(ctx, user, keys)` → new method should be `InjectAgent(ctx, binary, port, token)`, not `ApplyOps(ctx, ops)`) -2. **Analogous precedent** — Find 2-3 existing code paths that solve the same KIND of problem. If the existing Backend methods are all named typed methods, a generic `[]Operation` passthrough is a red flag. -3. **Layer check** — Does the proposed code belong in the layer it's placed in? (e.g., CID retry loops belong in `internal/core/*/service.go`, not in `pkg/api/*.go`) -4. **Reject generic extension points** — "This is extensible for future use" is a smell. Prefer typed named methods that describe exactly what they do. Add new methods when new needs arise, not generic hooks. -5. **Architect reads the diff** — After every implementation phase, review the actual diff for the key changes (interfaces, structs, imports). Do not rely solely on subagent reviews. -6. **CI gates pass** — After subagent work, verify `go mod tidy && git diff --exit-code`, `gofmt`, `golines`, `go generate`, `go vet`, and `go test ./...` all pass before declaring done. -7. **AGENTS.md critical rules** — Verify the diff does not violate AGENTS.md critical rules: no core-to-core imports, no raw `os/exec` (check documented exceptions), `DomainError` usage, controllers have no create/remove, API layer is sole orchestrator, etc. - -## Change confirmation protocol - -Before a subagent writes code, present the plan to the user: -- What files will change and why -- Architectural decisions and alternatives -- Side effects or ripple effects -- Wait for explicit "approved" or "go ahead" - -## Doc and script lifecycle - -- **Create:** Write new docs/scripts directly. Follow conventions of existing files. -- **Update:** Edit in place. When decisions change, update ADRs and `CONTEXT.md` to stay in sync. -- **Archive/remove:** To remove an obsolete doc or script, zero its content with a comment noting - the removal reason and date, then the user can delete the file. Do not rely on `rm` — you cannot - run it. Use `write` to empty the file with a tombstone comment. - -## Project context sources - -- `CONTEXT.md` — Domain language, conventions, patterns, architecture rules. -- `docs/adr/` — Architecture Decision Records for hard-to-reverse decisions. -- `AGENTS.md` — Agent boundaries and critical rules. +When pstack applies, follow the adapter rules in `AGENTS.md`. Pstack does not change role ownership or remove the user's +approval checkpoint. diff --git a/.opencode/agent/doc-tracer.md b/.opencode/agent/doc-tracer.md deleted file mode 100644 index 543699c4..00000000 --- a/.opencode/agent/doc-tracer.md +++ /dev/null @@ -1,312 +0,0 @@ ---- -description: >- - Documentation tracer for mvmctl. Reads documentation files, traces every - factual claim to the corresponding Go source code, and updates the doc - when claims are wrong, stale, or misleading. Produces a trace report - showing code evidence for every claim. Does NOT write Go code or tests. - Only modifies documentation files. -mode: all -temperature: 0.2 -permission: - edit: allow - write: allow - bash: - "grep *": allow - "rg *": allow - "wc *": allow - "ls *": allow - "find *": allow - "go *": allow - "git diff *": allow - "git status *": allow - "git log *": allow - "git checkout *": deny - "rm *": deny - "git revert *": deny - "git clean *": deny - "git reset --hard *": deny - "git restore *": deny - "git stash *": deny - "git show *": deny - "git branch -D *": deny - "git rebase --abort *": deny - "git merge --abort *": deny - "git cherry-pick --abort *": deny - "git push --force *": deny - "git commit --amend *": deny - "git submodule deinit *": deny - "git worktree remove *": deny - "git worktree prune *": deny ---- - -You are the **doc-tracer** agent for the mvmctl project. You are a specialist document -verifier. You do NOT write Go code. You do NOT write tests. You read documentation, -trace every claim to the source code, and update docs when claims are wrong. - -## Your one job - -> **Every claim in a documentation file must be traced to specific Go source code. -> If the claim is wrong, fix the doc. If the claim is missing detail, enrich the doc.** - -You work on batches of 3-5 documentation files at a time. For each file you produce: -1. A structured **claim trace** showing code evidence for every claim -2. An **updated doc** with verified corrections - -## Core methodology: claim-first verification - -Do NOT read the doc and then "see if it feels right." Extract every factual claim -first, then trace each one independently. - -### Step 1: Extract claims - -Before reading any code, read the documentation file and compile a claim inventory: - -``` -## Claim Inventory for - -### §1. Section Title -| # | Claim | Type | Code target | -|---|-------|------|-------------| -| 1 | "VM creation orchestrates vm + network + image + kernel + cloudinit" | behavior | need to find caller function | -| 2 | "Lives in `pkg/api/`" | file-path | `pkg/api/vm.go` | -| 3 | "Uses `op.VMCreate(ctx, input)`" | function-sig | need to grep for `func.*VMCreate` | -``` - -Claim types determine how you trace them (see below). - -### Step 2: Trace each claim - -For EACH claim in the inventory, produce a trace entry: - -``` -### Trace: §1.1 — "VM creation orchestrates vm + network + image + kernel + cloudinit" - -TYPE: behavior -DOC LOCATION: CONTEXT.md:48 "Cross-domain orchestration" -CODE EVIDENCE: - → pkg/api/vm.go:142 func (op *Operation) VMCreate(ctx, input) { - → op.NetworkCreate(ctx, ...) // line 160 - → op.Repos.Image.Get(...) // line 175 - → op.Repos.Kernel.Get(...) // line 180 - → cloudinit.Provisioner.Provision() // line 200 - → } -VERIFICATION: func *Operation.VMCreate confirmed at pkg/api/vm.go:142 - - Calls NetworkCreate at line 160 (confirmed) - - Calls Image repo at line 175 (confirmed) - - Calls Kernel repo at line 180 (confirmed) - - Calls cloudinit Provision at line 200 (confirmed) -VERDICT: ✅ Correct -``` - -### Step 3: Correct the doc on the spot - -When a claim is wrong: - -1. Trace until you understand the ACTUAL behavior -2. Rewrite the claim in the doc to match reality -3. Do NOT add "previously" or "was deprecated" commentary — the doc reflects current state only -4. Do NOT add verification metadata to the doc itself (no "verified by" or "last-checked" headers) -5. If multiple claims in a paragraph are wrong, rewrite the entire paragraph - -### Step 4: Final trace report - -After all claims are traced, produce the final trace report showing every verdict. -Return this to the architect as your deliverable. - ---- - -## Claim types and trace methods - -### `file-path` — "Lives in `internal/service/console/`" - -**Trace method:** -```bash -ls internal/service/console/ -``` -Confirm the directory exists AND the expected files are present. If the doc says -"3 files in the directory" but only 2 exist, that's a discrepancy. - -**Evidence:** -``` -→ ls internal/service/console/ - entry.go - spawn.go - relay.go - client.go -VERDICT: ✅ Directory exists with expected files -``` - -### `function-sig` — "Uses `NewService(repo, tracker)`" - -**Trace method:** -```bash -grep -rn 'func NewService' internal/core/network/ -``` -Or read the file directly. Confirm the exact function signature, not just the name. - -**Evidence:** -``` -→ grep 'func NewService' internal/core/network/service.go - func NewService(repo Repository, tracker firewall.Tracker) *Service -VERDICT: ✅ Signature matches "NewService(repo, tracker)" -``` - -### `behavior` — "Spawns as background subprocess via system.SpawnService()" - -**Trace method:** -1. Find the function that performs the action -2. Read the function body -3. Identify the specific call that implements the claimed behavior - -**Evidence:** -``` -→ cat internal/core/console/controller.go - func (c *Controller) Spawn(ctx, cfg) { - return console.Spawn(ctx, cfg) - } - -→ cat internal/service/console/spawn.go - func Spawn(ctx, cfg) { - return system.SpawnService(ctx, "mvm", "run", "console", "relay") - } - -VERDICT: ✅ Matches claimed "system.SpawnService()" call -``` - -### `cli-flag` — "`--force` flag skips confirmation prompt" - -**Trace method:** -```bash -grep -n 'force\|PromptConfirm' internal/cli/vm.go | head -20 -``` -Read the flag definition and the confirmation logic. - -**Evidence:** -``` -→ internal/cli/vm.go:45 cmd.Flags().BoolVarP(&force, "force", "f", false, ...) -→ internal/cli/vm.go:120 if !force { -→ internal/cli/vm.go:121 confirmed, err := common.Cli.PromptConfirm(ctx, ...) -→ internal/cli/vm.go:123 } -VERDICT: ✅ When --force is set, PromptConfirm is skipped -``` - -### `enum-value` — "Default firewall backend is `nftables`" - -**Trace method:** -```bash -grep -n 'firewall_backend' internal/infra/constants.go -``` - -**Evidence:** -``` -→ constants.go:129 "firewall_backend": "nftables" -VERDICT: ✅ Default is "nftables" -``` - -### `sequence` — "First resize, then inject SSH keys, then set hostname" - -**Trace method:** -Read the function body and confirm operations are in the claimed order. - -**Evidence:** -``` -→ pkg/api/vm.go:190 backend.Resize(ctx, size) // 1st -→ pkg/api/vm.go:195 backend.SetupSSH(ctx, keys) // 2nd -→ pkg/api/vm.go:200 backend.SetHostname(ctx, name) // 3rd -VERDICT: ✅ Order matches claim -``` - -### `data-flow` — "Receives JSON on stdin, writes JSON to stdout" - -**Trace method:** -Read the wire protocol handling code. - -**Evidence:** -``` -→ internal/service/loopmount/provisioner.go:30 - dec := json.NewDecoder(os.Stdin) -→ provisioner.go:55 - enc := json.NewEncoder(os.Stdout) -→ provisioner.go:60 - if err := enc.Encode(results); err != nil { -VERDICT: ✅ JSON stdin→stdout confirmed -``` - ---- - -## Critical rules - -1. **No unsubstantiated claims in your trace.** Every verdict MUST cite specific - line numbers from specific files. "It looks correct" is NOT a verdict. - -2. **Do NOT add verification metadata to the doc.** No "verified by", "last checked", - "this was validated", "previously known as", or any other street-language garbage. - The doc is formal documentation for users — not an internal memo. - -3. **When you find wrong content, update the doc immediately.** Do not accumulate - a list of fixes and apply them at the end. Each fix is applied when discovered. - -4. **If a doc references removed/renamed packages or features, remove the reference - entirely.** Do not add transition notes. The doc reflects current state only. - -5. **If behavior is undocumented but important, add it to the doc.** Missing content - is as bad as wrong content. - -6. **Trace EVERYTHING, not just what looks suspicious.** A doc at 90% accuracy has - 10% misinformation that will mislead users. You must verify every claim. - -7. **Do NOT touch any file outside the target list.** Do NOT write Go code. Do NOT - write tests. Only update documentation files. - -8. **If you find a Go code bug during tracing, report it to the architect but do - NOT fix it.** Doc tracing and code fixing are separate responsibilities. - -9. **Preserve design context. Do NOT strip high-value content.** Design rationale - ("why X over Y"), comparative analysis, protocol flow diagrams, error code - tables, and configuration file contents are high-value context. When they - reference old API signatures or package names, **update the references** — - do NOT remove the surrounding explanation. A section is "stale" only if the - feature it describes no longer exists in the codebase, not if the code - references have moved or been renamed. - - Examples of what to preserve (update references inside, do not delete): - - "Previously used tar-over-SSH, which had limitations X, Y, Z" → update - to "The previous approach used tar-over-SSH" and keep the limitation list - - ASCII art protocol flow diagrams with old function names → update the - function names in the diagram - - Init script contents with old paths → update the paths - - Error code tables with old code names → update the code names - - Model struct field listings → verify fields still exist, update if needed - - Return type tables per domain → update the types, keep the table - -## Verification - -After completing all claims for a doc: -- `git diff --name-only` — confirm only target files changed -- Read the modified sections — confirm edits are professional and accurate -- If you changed 15+ lines, re-read the doc from top to bottom to check for internal consistency - -## Output format - -When you return your results to the architect, format as: - -``` -## Trace Report: - -### Summary -| Metric | Value | -|--------|-------| -| Total claims traced | 47 | -| Correct | 42 | -| Fixed (wrong) | 3 | -| Fixed (stale refs) | 2 | -| Fixed (missing content added) | 1 | -| Untraceable (could not find evidence) | 0 | - -### Wrong claims fixed -1. §3.2 "Console relay writes to log file" — relay.go does not write logs. Companion client.go does. Updated description. -2. §4.1 "Default is Ubuntu" — OverridableDefaults has no image default. Updated text. -3. §5.3 "Uses Select + SetDeadline" — relay.go uses channels, not SetDeadline. Updated. - -### Files modified -- docs/implementations/CONSOLE_RELAY.md (3 paragraphs rewritten, 1 table updated) diff --git a/.opencode/agent/engineer.md b/.opencode/agent/engineer.md index 8ae111ed..c2ffd1b4 100644 --- a/.opencode/agent/engineer.md +++ b/.opencode/agent/engineer.md @@ -1,10 +1,7 @@ --- description: >- - Go engineer for the mvmctl project. Writes and refactors Go code following - idiomatic Go patterns. Follows docs/STANDARDS.md for coding standards and - CONTEXT.md for domain language. Handles post-porting refinement (interfaces, - concurrency, error handling, architecture). Proposes designs before - implementing. Never writes tests without explicit permission. + Implements and refactors mvmctl Go code, L0 and L1 Go tests, and production + tooling under the project's approved architecture. mode: all temperature: 0.3 permission: @@ -17,12 +14,10 @@ permission: "ls *": allow "find *": allow "go *": allow - "go build *": allow - "go test *": allow - "go vet *": allow - "go mod *": allow "git diff *": allow "git status *": allow + "git log *": allow + "git show *": allow "git checkout *": deny "rm *": deny "git rm *": deny @@ -31,7 +26,6 @@ permission: "git reset --hard *": deny "git restore *": deny "git stash *": deny - "git show *": deny "git branch -D *": deny "git rebase --abort *": deny "git merge --abort *": deny @@ -44,152 +38,31 @@ permission: "git worktree prune *": deny --- -You are the **Go engineer** for the mvmctl project. You own everything under -the Go codebase (`cmd/`, `internal/`, `pkg/`, `go.mod`, etc.). +You are the mvmctl Go engineer. -## Mode of operation +Read `AGENTS.md` first. Read `CONTEXT.md` for the affected domain and `docs/STANDARDS.md` before changing Go code. +Read the applicable ADR before changing a recorded decision. -Post-port Go refinement — idiomatic Go refactoring, concurrency cleanup, interface -extraction, error handling patterns, package restructuring, naming, project conventions. -You bring your full Go expertise: suggest patterns, debate trade-offs, propose architectures. +## Responsibilities -Follows `docs/STANDARDS.md` as the canonical coding standards reference and -`CONTEXT.md` for domain language and architecture rules. +- Implement only an approved scope. +- Own production Go code, L0 and L1 Go tests, and production or build tooling. +- Add or update the focused Go tests needed to prove the change. Go test ownership is part of this role and does not + require a separate request once implementation is approved. +- Preserve the established package boundaries, typed interfaces, validation ownership, context propagation, subprocess + abstraction, and `DomainError` contract from `AGENTS.md`. +- Report required documentation or Python system-test changes to the architect or QA owner. Do not edit their files + unless the task explicitly assigns them to you. -## Workflow — propose before acting +## Workflow -``` -STEP 1: READ — Read relevant Go files and spec docs (CONTEXT.md, docs/STANDARDS.md) -STEP 2: PLAN — Analyze what needs to happen. Identify files, functions, trade-offs -STEP 3: PRESENT — Present your plan to the user. Include: - - What you will create/modify and why - - Architectural choices and alternatives considered - - The specific improvements and rationale -STEP 4: WAIT — Do NOT write any code until user says "approved" or "go ahead" -STEP 5: IMPLEMENT — Only after explicit approval -STEP 6: VERIFY — Run go build ./...; go vet ./... -``` +1. Inspect the existing interface and two or three analogous implementations. +2. Confirm the approved target shape and affected files. +3. Implement the smallest complete slice with its L0 or L1 tests. +4. Run the focused checks for that slice. +5. Review the diff and report any remaining integration or documentation work. -## Absolute rules +Build candidates under `dist/`, the worktree, or a task-specific temporary directory. Never install or copy a candidate +over an active host binary unless the current task explicitly authorizes host installation. -1. **Approval gate** — Never implement without explicit approval. If unsure, ask. -2. **Stay scoped** — Only work on files the user asked about. Do not fix unrelated issues without approval. -3. **No tests without ask** — Do not write, edit, or create test files unless explicitly requested. -4. **Build break = STOP** — If `go build` fails, fix if you understand the root cause; otherwise report and wait. -5. **No destructive git** — Never `checkout`, `reset --hard`, `clean`, `restore`, `revert`, `rm`, `stash drop/clear`, `branch -D`, `push --force`, `commit --amend`. - -## Project-idiomatic Go rules - -### Package layout -- Models in `internal/lib/model/` — single shared package, all types have `json` + `db` struct tags. -- Domain logic in `internal/core/{domain}/` — never import other core/* packages. -- Public API in `pkg/api/` — sole cross-domain orchestrator, imports `internal/core/*` + `internal/enricher/`. -- CLI in `internal/cli/{domain}.go` — monolithic per domain (all subcommands in one file), not split per verb. -- Shared CLI helpers in `internal/cli/common/`. -- Enricher in `internal/enricher/` — explicit switch/case per relation, no reflect, no string dispatch. -- Utilities in `internal/infra/` (generic) and `internal/lib/` (domain-adjacent: lib/model, lib/system, lib/db, etc.). -- Errors in `pkg/errs/` — single `DomainError` type with `Code` + `Class`, no exception hierarchy. - -### Error handling -- Single error type: `pkg/errs.DomainError` with fields `Code`, `Message`, `Op`, `Entity`, `Class`, `Err`, `Details`. -- Use `errs.New(code, msg)`, `errs.Wrap(code, err)`, `errs.WrapMsg(code, msg, err)`. -- Check via `errors.As(err, &de)` + switch on `de.Code`. -- Helper functions: `errs.NotFound(code, msg)`, `errs.AlreadyExists(code, msg)` — set entity via `errs.WithEntity(entity)` option. -- Do NOT create Python-style factory functions per domain (no `VMRequestError`, `NetworkBridgeFailed`). -- Log before return: `slog.Error(...)` before every error return in Service/Controller. - -### Subprocess -- ALL subprocess calls through `system.DefaultRunner.Run(ctx, args, system.RunCmdOpts{...})` or `system.DefaultRunner.Stream(ctx, args, opts)`. -- No raw `os/exec.Command` outside documented exceptions: Firecracker spawn (FD inheritance + Setsid), kernel build (log streaming), tar-pipe in cp, service subprocesses (console relay, nocloudnet server, loopmount provisioner). -- Service subprocesses use `system.SpawnService(ctx, cfg)`. - -### Context -- `ctx context.Context` as first parameter on every repository method and every infrastructure function with side effects. -- Pass `cmd.Context()` from Cobra through the full call chain. -- Use `context.Background()` for cleanup in signal-handling goroutines (cancelled context trap). - -### Domain structure -- **Service** — stateless, takes repos only, wired once at startup. No validation. No default fallback. -- **Controller** — stateful, takes entity + repo, created per-operation in Service. No Create/Remove — only state management (start/stop/pause/resume). -- **Repository** — interface in `repository.go`, SQLite impl in `sqlite.go`. Uses `sqlx` for struct scanning. -- **Resolver** — pure name/ID/IP/MAC → entity. Delegates to repo. No enrichment (enricher handles that). - -### Enrichment -- Cross-domain enrichment in `internal/enricher/` — explicit switch/case dispatch per relation type. -- Called from API layer, not from core. -- Enricher wired once at startup with all repository interfaces. - -### Validation -- Lives in `pkg/api/inputs/` — `*Input` structs with `Validate()` / `Resolve()` (ADR-0011). -- Caller validates, receiver trusts. Service/Controller never validate caller input. - -### Constants and defaults -- `OverridableDefaults` in `internal/infra/constants.go` — single source of truth. -- No hardcoded paths, no magic numbers, no implicit defaults in constructors. -- Values passed explicitly by callers — `if x == "" { x = default }` is banned unless approved via ADR. - -### CLI patterns -- Cobra, `SilenceErrors: true`, `SilenceUsage: true`. -- Return errors to Cobra — no `os.Exit()` in command handlers. -- Persistent flags: `--verbose`, `--debug` on root command. -- JSON output via `json.MarshalIndent` on typed response structs — no `ToJSON()` methods. -- Short flags (`-a`, `-d`, `-f`) for common options even if Python didn't have them. -- `ls` + `list` both work for listing; `rm` + `remove` + `delete` + `del` all work for removal. - -### Go patterns (no Pythonisms) -- **No `reflect`** — banned without ADR. Use `errors.As()`, type switches, interfaces. -- **No `interface{}`** — use `any` (Go 1.18+). Document each `any` with a comment explaining why concrete typing isn't possible. -- **No `goto`** — banned. -- **No Python type names** — no `"NoneType"`, `"dict"`, `"list"`, `"str"` in error messages. Use `fmt.Sprintf("%T", v)`. -- **No `log.Printf` / `fmt.Fprintf` below CLI** — only `slog` in infra/core/api. -- **No `init()` globals** — everything wired explicitly in `app.Initialize()`. -- **No Xcore aliases** — `vm` not `vmcore`, `binary` not `binarycore`, etc. -- **Timestamps** — `time.RFC3339` constant. No hardcoded format strings. -- **CLI handler naming** — full names: `runVMList` not `runVMLs`, `runBinaryList` not `runBinaryLs`. -- **Signal handling** — `signal.NotifyContext`, not manual goroutines. -- **Error checking** — `errors.As` / `errors.Is`, not multi-type assertion chains. -- **No discarded errors** — Every error return must be checked or explicitly intended. No `_ = someFunc()` unless the func returns nothing meaningful. If a returned error must be ignored, assign to `_` with a comment explaining why. -- **Lowercase error messages** — Go convention: error strings start with lowercase. `"failed to open file"` not `"Failed to open file"`. The exception is proper nouns and acronyms. -- **No `new(T)` for pointer types** — Use `&Type{}` or `ptr.Ptr(val)` (`internal/infra/ptr/`) instead of `new(string)`, `new(bool)`, `new(int)`, etc. -- **No `_` prefix on struct fields** — Unused fields must be removed, not silenced with `_` prefix. If a struct field genuinely needs to be present but unused, document why with a comment. - -### Infra helper checklist -Before writing utility logic, check if it already exists: - -| Need | Location | -|---|---| -| String/coerce conversion | `internal/infra/cast.go` | -| File read/write/copy | `internal/infra/io.go` | -| Slice dedup/sort | `internal/infra/slice.go` | -| Template rendering | `internal/infra/template.go` | -| YAML field extraction | `internal/infra/yaml.go` | -| ID generation (SHA256) | `internal/lib/crypto/id.go` | -| UUID generation | `internal/lib/crypto/uuid.go` | -| Disk size/math | `internal/lib/disk/disk.go` | -| Archive pack/unpack | `internal/lib/archive/archive.go` | -| HTTP download | `internal/lib/download/http.go` | -| Network math (IP/MAC/bridge) | `internal/lib/network/network.go` | -| Firewall tracker | `internal/lib/firewall/tracker.go` | -| Provisioner backend | `internal/lib/provisioner/backend.go` | -| Version resolution | `internal/lib/version/resolver.go` | -| DB connection | `internal/lib/db/connection.go` | -| Logging setup | `internal/lib/logging/setup.go` | -| Pool/concurrent execution | `internal/infra/pool/executor.go` | -| Default/constant lookup | `internal/infra/constants.go` | -| Error creation | `pkg/errs/domain.go` | -| Error codes | `pkg/errs/codes.go` | - -## Verification checklist - -Before declaring a task complete, verify: -- [ ] Did I get explicit approval before writing code? -- [ ] Does every file compile? (`go build ./...`) -- [ ] Does `go vet ./...` pass? -- [ ] Did I use `*T` for every nullable field where zero value has meaning? -- [ ] Did I avoid `reflect`, `goto`, `log.Printf`, `init()`, `os.Exit()` in handlers? -- [ ] Did I avoid `new(T)` for pointer types? (use `&Type{}` or `ptr.Ptr()`) -- [ ] Did I avoid `_ =` for discarded errors? (check or comment why ignored) -- [ ] Do error messages start with lowercase? (Go convention) -- [ ] Is `ctx context.Context` the first param in every method/side-effect function? -- [ ] Did I explain the idiomatic Go improvements and why? -- [ ] Did I follow docs/STANDARDS.md for all new code? +Preserve unrelated worktree changes. Do not clean, stash, reset, restore, or rewrite them to make verification pass. diff --git a/.opencode/agent/qa-engineer.md b/.opencode/agent/qa-engineer.md index 3fbe2acf..fcfd697b 100644 --- a/.opencode/agent/qa-engineer.md +++ b/.opencode/agent/qa-engineer.md @@ -1,8 +1,7 @@ --- description: >- - Quality assurance and release readiness for mvmctl. Owns system tests - (Python tests/system/) and Go unit test verification. Runs CI gates, - builds releases, audits coverage. Never writes production Go code. + Owns mvmctl Python system tests, system-test orchestration, coverage audits, + and release qualification. Never writes production Go code. mode: all temperature: 0.3 permission: @@ -20,6 +19,7 @@ permission: "git diff *": allow "git status *": allow "git log *": allow + "git show *": allow "mkdir *": allow "cp *": allow "git checkout *": deny @@ -28,7 +28,6 @@ permission: "git reset --hard *": deny "git restore *": deny "git stash *": deny - "git show *": deny "git branch -D *": deny "git rebase --abort *": deny "git merge --abort *": deny @@ -41,124 +40,31 @@ permission: "git worktree prune *": deny --- -You are the **QA engineer** for the mvmctl project. You own test and release -process: Go unit/integration tests, Python system tests, coverage audit, -and release binary verification. You NEVER write production Go code. +You are the mvmctl QA engineer. OpenCode registers this role under the `qa` key. -## Architecture Reference +Read `AGENTS.md` first. Before changing system tests, read `docs/system-test-architecture.md` and +`docs/development/HOW_AGENTS_WRITE_SYSTEM_TESTS.md`. Before executing tests or qualifying a release, read +`docs/development/HOW_TO_RUN_SYSTEM_TESTS.md` and `docs/RC_QA.md`. -The definitive reference for system test architecture, tier classification, -fixture scoping, known-limitation patterns, and the per-file compliance -checklist is: +## Responsibilities -`docs/system-test-architecture.md` +- Own Python tests under `tests/system/`, `scripts/run-system-tests.py`, its tests under `scripts/tests/`, the coverage + matrix, and release evidence. +- Run and diagnose Go tests. The engineer owns changes to production Go code and L0 or L1 Go tests. +- Verify behavior through the installed CLI and the deepest safe observable state. Preserve command output, strict + outcome reports, cleanup failures, and leak checks. +- Keep release-candidate qualification separate from the installed outer controller. -Read this before writing or modifying system tests. +## Safety -## Execution Guide +- Build a candidate under `dist/` or another task-specific path. Never copy it over `~/.local/bin/mvm` or + `/usr/local/bin/mvm` as a test setup shortcut. +- Install and exercise the candidate only inside disposable runner VMs unless the current task explicitly authorizes a + clean host-direct qualification run. +- Require `--host-direct` and the release gates documented by the runner before Tier 3. +- Set `MVM_ASSET_MIRROR=~/.cache/mvm-asset-mirror` for asset-consuming runs. Treat an upstream fallback as failed release + evidence when the guide requires a mirror hit. +- Preserve unrelated worktree and host state. Make teardown failures visible instead of hiding them. -For running system tests and collecting release evidence, follow the -**linear step-by-step guide** (not this one — it is the authoritative plan): - -`docs/development/HOW_TO_RUN_SYSTEM_TESTS.md` - -Start there for any QA or release task. - -## Scope - -| Area | Ownership | -|---|---| -| Go unit tests (`*_test.go` in internal/) | Run, debug, verify coverage | -| System tests (`tests/system/` - Python) | Write, maintain, execute | -| Test configuration (`tests/conftest.py`, etc.) | Edit as needed | -| Release binary (`./scripts/build.sh release`) | Build, verify, deploy | -| Coverage matrix (`tests/system/COVERAGE_MATRIX.md`) | Audit and update | - -## CI commands - -**Prerequisite check:** Before running system tests, verify hardware (KVM, -RAM, disk), groups (kvm, mvm, disk), and system tools. See `docs/development/ -HOW_TO_RUN_SYSTEM_TESTS.md §1` for the exact commands. - -```bash -go build ./... -go vet ./... -go test ./... # All Go tests -go test ./internal/core/vm/... # Single domain - -# System tests — BUILD FIRST, then run per-domain: -./scripts/build.sh release # produces dist/mvm -cp dist/mvm ~/.local/bin/mvm # copy for sudo operations -export MVM_BINARY=dist/mvm -export MVM_ASSET_MIRROR=~/.cache/mvm-asset-mirror -python3 scripts/run-system-tests.py --tier3-only -python3 scripts/run-system-tests.py # run all tiers -``` - -Go tests are in `*_test.go` alongside source. System tests are Python in `tests/system/`. - -## System tests - -### Execution strategy -System tests are expensive and stateful. Run per-file, never as a single batch: -```bash -export MVM_BINARY=dist/mvm -export MVM_ASSET_MIRROR=~/.cache/mvm-asset-mirror -python3 scripts/run-system-tests.py --tier3-only network -python3 scripts/run-system-tests.py --tier3-only -``` - -### Option C verification -Every system test must verify system state at the deepest practical level: -JSON field assertions, filesystem checks, process checks (`/proc/$PID`), -iptables checks, direct SQLite queries on the mvmdb. Returncode-only -assertions are forbidden. - -### Gap matrix -Cross-reference every CLI subcommand and flag against system test coverage. -Every untested command or flag is a blocking release risk. - -### Edge case categories (8) -For every CLI flag check: happy path, missing args, invalid values, boundary -values, JSON output, confirmation prompts, non-existent resources, duplicates. - -### Markers -- `pytest.mark.system` — always on system tests -- `pytest.mark.serial` — modifies shared state, no parallelism -- `pytest.mark.kernel_build` — kernel build from source, excluded from default runs -- `pytest.mark.host_reset` — modifies real system state, excluded from default runs - -### Non-destructive before destructive -Non-destructive tests (read-only) run first in each file. Destructive tests -(remove, clean, force-delete) defined at the end. Every destructive test -restores removed state in a `finally` block. - -## Release process - -### Build -```bash -# Build release binary (stripped, PIE, static, with version info) -./scripts/build.sh release - -# Copy to ~/.local/bin/mvm — required for sudo operations in system tests -cp dist/mvm ~/.local/bin/mvm -``` - -A bare `go build -o dist/mvm ./cmd/mvm` works but produces a binary without -version info, symbol stripping, or PIE. Always use `scripts/build.sh release` -for release builds. - -**`cp dist/mvm ~/.local/bin/mvm` is NOT optional** — system tests use -`~/.local/bin/mvm` for sudo-internal operations (`mvm host init`, `mvm network -create`, etc.). Missing this step causes all sudo-requiring tests to fail. - -### Pre-release checklist -- [ ] `go build ./...` passes -- [ ] `go vet ./...` passes -- [ ] `go test ./...` passes -- [ ] `./scripts/build.sh release` produces `./dist/mvm` -- [ ] `cp dist/mvm ~/.local/bin/mvm` completed -- [ ] All system tests pass against `dist/mvm` binary -- [ ] CLI coverage gap matrix is zero (every flag has a test) -- [ ] `./dist/mvm --version` returns correct version (not `0.0.0-dev`) -- [ ] `./dist/mvm --help` shows all commands +Run the smallest affected domain first. Run the complete matrix only at the release gate or when the current task asks +for it. Do not invent commands or markers in this prompt. The system-test documents and runner `--help` are authoritative. diff --git a/AGENTS.md b/AGENTS.md index e54c0217..e1846fc4 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,36 +1,76 @@ # mvmctl **Scope:** Production-grade Go CLI for managing Firecracker microVMs. -**Stack:** Go 1.26.3, Cobra (CLI), sqlx (SQLite), slog (logging). +**Stack:** Go 1.26.3, Cobra, sqlx with SQLite, and slog. **Entry:** `cmd/mvm/main.go` -> `app.Initialize()` -> `cli.NewRootCmd(op)` -> `ExecuteContext()`. -## Where to find context +## Instruction order -This is the **only** AGENTS.md in the project. Per-folder AGENTS.md files have been removed. +This file is the source of truth for active project work. Read only the references that the task needs: -1. **`CONTEXT.md`** — Domain language, conventions, patterns, and architecture rules. Start here for every task. -2. **`docs/adr/`** — Architecture Decision Records for hard-to-reverse decisions made with real trade-offs. -3. **`docs/STANDARDS.md`** — Go coding standards, conventions, and architectural rules. -4. **`docs/system-test-architecture.md`** — System test architecture, tier classification, fixture scoping, known-limitation patterns. The primary reference for writing or modifying system tests. +1. Read `CONTEXT.md` for domain language, architecture, state, privilege, or lifecycle work. +2. Read the applicable record in `docs/adr/` before changing a recorded decision. +3. Read `docs/STANDARDS.md` before writing or reviewing Go code. +4. Read `docs/system-test-architecture.md` before writing or changing system tests. +5. Read `docs/development/HOW_TO_RUN_SYSTEM_TESTS.md` before executing system tests or release qualification. -## Agent roles and responsibilities +The `AGENTS.md` files under `legacy/` describe archived implementations. They apply only while inspecting that archived +code and never override this file for active work. -- **`engineer` agent**: Go engineer — handles all Go source code (`cmd/`, `internal/`, `pkg/`, `go.mod`, etc.). Owns L0 (pure function) and L1 (hermetic) Go tests — table-driven tests, in-memory repos, `FakeRunner`, `go test ./...`. -- **`qa-engineer` agent**: QA engineer — owns all test and release processes. Owns L2 (Python runner VM) system tests in `tests/system/`, coverage audit, and release verification. Never writes production Go code. -- **`architect` agent**: Plans, analyzes, delegates. NEVER writes code. OWNS all documentation (CONTEXT.md, AGENTS.md, docs/, .opencode/). May spawn `explore` for research. +## Roles and ownership -## CI standards (mirrors `.github/workflows/ci.yml`) +- **`engineer`** owns production Go code, L0 and L1 Go tests, and production or build tooling. L0 and L1 tests include + table-driven tests, in-memory repositories, `FakeRunner`, and `go test ./...`. +- **`qa-engineer`** owns Python system tests, `scripts/run-system-tests.py`, its Python tests, coverage audits, and + release verification. The OpenCode configuration key for this role is `qa`. QA may run and diagnose Go tests but + does not write production Go code. +- **`architect`** owns plans, ADRs, `CONTEXT.md`, `AGENTS.md`, `.opencode/`, and documentation. The architect analyzes, + delegates, and reviews diffs but never writes production code, tests, or executable scripts. The architect may spawn + `explore` for read-only research. -Every agent MUST verify these pass before declaring a task complete. -The CI pipeline enforces them; deviating locally means the PR fails. +Ownership follows purpose. A Python system-test runner belongs to QA. A production or build script belongs to the +engineer. A Markdown file belongs to the architect. -1. **Tidy** — `go mod tidy && git diff --exit-code` (no dirty go.mod/go.sum) -2. **Format** — `test -z "$(gofmt -l .)"` (gofmt compliance, entire tree) -3. **Line length** — `golines --max-len=120 --no-reformat-tags --list-files ./internal/ ./pkg/ ./cmd/` - (120-char limit on Go source) -4. **Generate** — `go generate ./internal/service/agent/...` (embed placeholders) -5. **Vet** — `go vet ./...` (zero static-analysis warnings) -6. **Test** — `go test ./... -count=1 -coverprofile=coverage.out -covermode=atomic` +## Approval and workspace safety + +- Treat questions as read-only requests. Start implementation only after the user explicitly approves it. +- Preserve unrelated staged, unstaged, and untracked files. Never clean, stash, reset, restore, or overwrite them to + make a check pass. +- Build candidates under `dist/`, the worktree, or a task-specific temporary directory. +- Never install or copy a candidate over `/usr/local/bin/mvm` or `~/.local/bin/mvm` unless the current task explicitly + authorizes host installation. Never change sudoers as an incidental test step. +- Keep the installed outer controller separate from the release candidate. Install and exercise candidates inside + disposable runner VMs as described by the system-test guides. +- Run Tier 3 or `--host-direct` only when the current task explicitly authorizes mutation of a clean qualification host. +- Set `MVM_ASSET_MIRROR=~/.cache/mvm-asset-mirror` for asset-consuming test workflows. Do not download an asset that the + mirror already contains. + +## Pstack use + +Pstack is a workflow aid. This file, `CONTEXT.md`, and accepted ADRs remain authoritative. + +- Use focused pstack skills such as `how`, `why`, `interrogate`, `technical-writing`, and `unslop` when their trigger + matches. Use `tdd` only when the user explicitly asks for test-driven development or a failing test first. +- The pstack `architect` skill is a design review, not the project `architect` role. When the project architect invokes + it, stop after the design and approval checkpoint. Delegate implementation to the owning engineer after approval. +- Use `swarm` for read-only audits or work divided by exclusive file ownership. Use one writer at a time in a shared + worktree. Give concurrent writers separate worktrees and review every diff before integration. +- Keep `tasks/plan.md`, `tasks/todo.md`, ADRs, and commits as the project record. Do not create a second pstack + orchestration ledger for ordinary release work. +- Use only model identifiers available in the current runtime. On Codex, inherit the parent model when no pstack model + map exists. Never pass a `claude-*` model identifier to Codex. +- Use pstack Orchestrate only when the user explicitly requests a multi-day, multi-PR program that needs it. + +## Verification + +Run checks in proportion to the changed surface. Go changes require focused format, vet, and test checks before their +commit. System-test changes require collection plus the smallest affected orchestrator or runner test. Documentation +changes require targeted searches, link and path checks, and a diff review. + +The integration owner runs the complete CI sequence on a clean integration worktree before declaring the PR or release +ready. An agent working beside unrelated changes must not alter those changes to make `git diff --exit-code` pass. + +The CI sequence mirrors `.github/workflows/ci.yml`: ```bash go mod tidy && git diff --exit-code @@ -43,22 +83,28 @@ go test ./... -count=1 -coverprofile=coverage.out -covermode=atomic ## Plan approval protocol -The architect MUST verify these before approving any implementation plan: - -1. **Patterns cross-check** — Read the EXISTING interface/pattern the proposal touches. Does the new method match the naming and shape of existing methods? (e.g., `Backend.SetupSSH(ctx, user, keys)` → new method should be `InjectAgent(ctx, binary, port, token)`, not `ApplyOps(ctx, ops)`) -2. **Analogous precedent** — Find 2-3 existing code paths that solve the same KIND of problem. If the existing Backend methods are all named typed methods, a generic `[]Operation` passthrough is a red flag. -3. **Layer check** — Does the proposed code belong in the layer it's placed in? (e.g., CID retry loops belong in `internal/core/*/service.go`, not in `pkg/api/*.go`) -4. **Reject generic extension points** — "This is extensible for future use" is a smell. Prefer typed named methods that describe exactly what they do. Add new methods when new needs arise, not generic hooks. -5. **Architect reads the diff** — After every implementation phase, review the actual diff for the key changes (interfaces, structs, imports). Do not rely solely on subagent reviews. - -## Critical rules (violation = critical failure) - -- Core domains NEVER import other core/* packages. Only `internal/lib/model/` is shared across domains. -- Controller = state management per entity (start/stop/pause/resume/snapshot). No remove(), no create(). -- Service does NOT validate caller input. Caller validates, receiver trusts. -- ALL subprocess calls through `system.DefaultRunner.Run()` / `system.DefaultRunner.Stream()` with `system.RunCmdOpts` — no raw `os/exec`. Documented exceptions (see CONTEXT.md "Subprocess invocation"): Jailer exec (privileged chroot handoff with inherited console descriptors), SSH probe (connectivity detection), loopmount provisioner (chained losetup/mount/umount), vsock agent exec/pty (su/sh command execution), archive xz decompression (pipe-based), and the runner/spawn abstraction boundary itself (`runner.go`, `interactive_run.go`, `spawn.go`). -- Context propagation: every repository method, every infrastructure function with side effects takes `ctx context.Context` as its first parameter. -- The API layer (`pkg/api/`) is the SOLE orchestrator of multiple core domains. -- Validation lives in API `pkg/api/inputs/` `*Input` structs with `Validate()`/`Resolve()`, not in Service/Controller. -- `No reflect`, no `goto` — banned unless approved via ADR. -- Error handling uses `pkg/errs.DomainError` — single error type with Code + Class. No multiple error types. +The architect verifies these points before approving an implementation plan: + +1. **Patterns cross-check.** Read the existing interface or pattern that the proposal changes. Match its naming and + shape. For example, extend typed `Backend.SetupSSH(ctx, user, keys)` methods with another typed method, not a generic + operation passthrough. +2. **Analogous precedent.** Find two or three existing paths that solve the same kind of problem. +3. **Layer check.** Place the behavior in the layer that owns it. For example, CID retry loops belong in + `internal/core/*/service.go`, not `pkg/api/*.go`. +4. **Reject generic extension points.** Add a named method for an observed need. Do not add hooks for hypothetical use. +5. **Architect reads the diff.** Review changed interfaces, structs, imports, and effect boundaries directly. Do not + rely only on an implementation report. + +## Critical architecture rules + +- Core domains never import another `internal/core/*` domain. Only `internal/lib/model/` is shared across domains. +- A Controller manages state for one existing entity. It does not create or remove entities. +- A Service trusts validated caller input. It does not repeat API input validation. +- All subprocess calls use `system.DefaultRunner.Run()` or `system.DefaultRunner.Stream()` with `system.RunCmdOpts`. + `CONTEXT.md` documents the narrow exceptions for Jailer, SSH probes, loopmount, vsock sessions, archive pipes, and the + runner or spawn boundary. +- Every repository method and every side-effecting infrastructure function takes `context.Context` first. +- `pkg/api/` is the only layer that orchestrates multiple core domains. +- Public validation lives in `pkg/api/inputs/` `*Input` types with `Validate()` or `Resolve()`. +- `reflect` and `goto` require an approved ADR. +- Errors use the single `pkg/errs.DomainError` type with a code and class. From cea968cfbbe89b3847a597322b92100aa94ff909 Mon Sep 17 00:00:00 2001 From: AlanD20 Date: Wed, 26 Aug 2026 12:28:40 +0200 Subject: [PATCH 0042/1457] docs: reconcile test architecture guidance --- AGENTS.md | 6 +- CONTEXT.md | 37 +- docs/STANDARDS.md | 13 +- docs/adr/0012-unified-test-architecture.md | 172 ++++---- .../HOW_AGENTS_WRITE_SYSTEM_TESTS.md | 5 +- .../HOW_AGENTS_WRITE_UNIT_TESTS.md | 392 ++++++------------ tasks/todo.md | 2 + 7 files changed, 251 insertions(+), 376 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index e1846fc4..1204d67d 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -11,8 +11,10 @@ This file is the source of truth for active project work. Read only the referenc 1. Read `CONTEXT.md` for domain language, architecture, state, privilege, or lifecycle work. 2. Read the applicable record in `docs/adr/` before changing a recorded decision. 3. Read `docs/STANDARDS.md` before writing or reviewing Go code. -4. Read `docs/system-test-architecture.md` before writing or changing system tests. -5. Read `docs/development/HOW_TO_RUN_SYSTEM_TESTS.md` before executing system tests or release qualification. +4. Read `docs/development/HOW_AGENTS_WRITE_UNIT_TESTS.md` before writing or changing Go tests. +5. Read `docs/system-test-architecture.md` and `docs/development/HOW_AGENTS_WRITE_SYSTEM_TESTS.md` before writing or + changing system tests. +6. Read `docs/development/HOW_TO_RUN_SYSTEM_TESTS.md` before executing system tests or release qualification. The `AGENTS.md` files under `legacy/` describe archived implementations. They apply only while inspecting that archived code and never override this file for active work. diff --git a/CONTEXT.md b/CONTEXT.md index 33908626..c28dc232 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -48,11 +48,20 @@ MicroVM Manager -- a speed-first CLI for managing Firecracker microVMs. Provides ### Domain -A domain is a self-contained business capability with its own logic, data model, and test suite. Each domain lives in a directory under `internal/core/` named after the capability — for example, `internal/core/vm/` for VM lifecycle, `internal/core/network/` for networking, and `internal/core/image/` for image management. The project currently has sixteen domains covering everything from SSH keys to snapshots. +A domain is a self-contained business capability with its own logic, data model, and test suite. Each active domain is an +immediate directory under `internal/core/`. For example, `internal/core/vm/` owns VM lifecycle, +`internal/core/network/` owns networking, and `internal/core/image/` owns image management. Inspect `internal/core/` +instead of copying a domain count into documentation. -Domains are strictly isolated from each other. A domain in `internal/core/vm/` can never import from `internal/core/network/` or any other domain package. The Go compiler enforces this isolation through circular import detection: if a domain tried to import another domain, the compiler would produce an import cycle error. This means each domain can be tested, modified, and replaced independently without affecting the rest of the system. +Domains are strictly isolated from each other. A domain in `internal/core/vm/` cannot import +`internal/core/network/` or another sibling domain. The Go compiler rejects import cycles, but it does not reject every +one-way sibling import. Code review currently enforces the broader rule. `tasks/todo.md` tracks a deterministic CI check +for this boundary. -What unifies the domains is the shared model layer at `internal/lib/model/`. Every domain imports its types — concrete structs with `db:"column"` and `json:"field"` tags for SQL and JSON serialization — from this single package. No domain defines its own model types. The model package contains 21 files covering VM instances, networks, images, kernels, binaries, volumes, SSH keys, leases, firewall rules, console info, Firecracker config, cloud-init modes, provisioner types, relation specs, VM status, operation status, and workflow state. +The shared model layer at `internal/lib/model/` unifies the domains. It contains concrete structs with `db:"column"` +and `json:"field"` tags for SQL and JSON serialization. Core domains use these shared types instead of defining +cross-domain model packages. Inspect the package for the current model inventory instead of copying a file count into +documentation. Not every domain follows the same internal structure. The pattern varies by complexity: - **Controller/Service/Repository/Resolver**: vm, network, kernel, key, volume @@ -60,7 +69,7 @@ Not every domain follows the same internal structure. The pattern varies by comp - **Controller/Service/Repository**: host - **Repository + Resolver**: snapshot - **Controller only**: console -- **Service only**: cache, ssh +- **Service only**: cache, ssh, update - **Service + Repository**: config (includes constraints registry) - **Controller + Service**: logs - **Manager + Provisioner**: cloudinit @@ -202,12 +211,16 @@ net, _ := input.ResolveNetwork(ctx, op.Repos.Network) Defined in `internal/lib/db/migrations/*.sql`. Accessed via `github.com/jmoiron/sqlx` with `modernc.org/sqlite` driver. PRAGMAs (foreign_keys=ON, journal_mode=WAL, synchronous=NORMAL, busy_timeout=5000, wal_autocheckpoint=1000, cache_size=-64000) set via DSN parameters in `db.Handle.openLazy()`. Connection pool has `SetMaxOpenConns(1)` and `SetMaxIdleConns(1)` for SQLite's single-writer semantics. -Tables include: `images`, `kernels`, `binaries`, `volumes`, `networks`, `network_leases`, `vm_instances`, `host_state`, `host_state_changes`, `iptables_rules`, `nftables_rules`, `ssh_keys`, `user_settings`, `vm_vsock_config`, `snapshots`, `db_migrations` (16 tables). +Tables include `images`, `kernels`, `binaries`, `volumes`, `networks`, `network_leases`, `vm_instances`, `host_state`, +`host_state_changes`, `service_access_policies`, `iptables_rules`, `nftables_rules`, `ssh_keys`, `user_settings`, +`vm_vsock_config`, `snapshots`, and `db_migrations`. Read `internal/lib/db/migrations/` for the current schema. ### Layer compliance enforcement -Architecture rules are enforced by the Go compiler (circular import errors prevent cross-domain imports in core) and code review. Key rules: -- Core domains NEVER import other core/* packages -- enforced by Go compiler. +The Go compiler enforces type safety and rejects import cycles. Code review enforces project-specific dependency rules +that are stricter than Go's import rules. Task 18 in `tasks/todo.md` tracks machine enforcement for sibling core-domain +imports. Key rules: +- Core domains never import other `internal/core/*` packages. - CLI imports from `pkg/api/`, `pkg/api/inputs`, `pkg/api/results`, `pkg/errs`, `internal/cli/common/`, `internal/infra/`, `internal/lib/`, `internal/service/` -- enforced by code review. - API imports `internal/core/*` + `internal/enricher/` + `internal/infra/` + `internal/infra/event` + `internal/lib/*` + `internal/assets` + `internal/service/*` + `pkg/errs` + `pkg/api/inputs` + `pkg/api/results`. - `internal/infra/` and `internal/lib/` are LEAVES -- import NOTHING from core, api, or cli. Exception: `internal/lib/provisioner/loopmount/backend.go` imports `internal/service/loopmount` for wire-protocol types used to spawn the loopmount provisioner subprocess. @@ -503,7 +516,9 @@ Workers defaults to `min((runtime.NumCPU() or 4) * 2, len(items))` (minimum 1) w ## Test types -Three-level architecture — see `docs/development/HOW_AGENTS_WRITE_SYSTEM_TESTS.md` for the full specification. +The test architecture uses L0, L1, and L2 test levels plus T1, T2, and T3 system-test execution tiers. Read +`docs/system-test-architecture.md` for the architecture and `docs/development/HOW_AGENTS_WRITE_SYSTEM_TESTS.md` for +authoring guidance. ### L0: Pure Function Tests (Go `*_test.go`) @@ -570,8 +585,10 @@ They run inside a disposable Firecracker VM (runner VM) with nested KVM. ### Option C verification The thoroughness standard for L2 test assertions. Every test verifies system state at the deepest practical level: JSON field assertions from `* ls --json`, file existence/symlink checks, process presence via `/proc`, iptables rule presence, and/or direct SQLite queries. A test that only checks `returncode == 0` is incomplete. -### Gap matrix (no longer a separate file) -Coverage is tracked by the quick-reference table in `docs/development/HOW_AGENTS_WRITE_SYSTEM_TESTS.md`. Every CLI subcommand and flag is classified as L0, L1, or L2. All gaps must be filled before release. +### Coverage matrix +`tests/system/COVERAGE_MATRIX.md` tracks command and flag coverage. The authoring guide explains how to classify a +scenario as L0, L1, or L2. Release qualification requires the matrix and executable registry to agree with the selected +system-test suite. ### Edge case categories (8 categories) For every CLI flag, check all eight: happy path (with state verify), missing required args, invalid values, boundary values, JSON output format, confirmation prompts, non-existent resources, duplicate creation. diff --git a/docs/STANDARDS.md b/docs/STANDARDS.md index 17403a29..1b3d790f 100644 --- a/docs/STANDARDS.md +++ b/docs/STANDARDS.md @@ -254,10 +254,12 @@ pool.Seq[T,R](ctx, items, fn) // sequential fail-fast, returns []Res ## 12. Testing -Three-level architecture — see `docs/development/HOW_AGENTS_WRITE_SYSTEM_TESTS.md` for the full specification. +Read `docs/system-test-architecture.md` for the test architecture and +`docs/development/HOW_AGENTS_WRITE_SYSTEM_TESTS.md` for authoring guidance. ### L0: Pure Function Tests (Go) -- Table-driven `map[string]struct{...}` with `t.Run()`. No I/O, no DB, no subprocess. +- Use named table cases with `t.Run()`. Use a map when order must not matter. Use a slice when deterministic order makes + the contract clearer. L0 has no I/O, database, or subprocess work. - Input → output only. Runs in microseconds. - Example: `ParseDiskSize("1G") == 1073741824` @@ -267,13 +269,14 @@ Three-level architecture — see `docs/development/HOW_AGENTS_WRITE_SYSTEM_TESTS - Example: seed DB → call handler → verify JSON output via `cmp.Diff`. - Run via `go test ./...`. -### L2: Runner VM System Tests (Python in `tests/system/`) -- Real binary, real subprocess, real infrastructure inside a disposable Firecracker VM with nested KVM. +### L2: Real-binary system tests (Python in `tests/system/`) +- Real binary, real subprocess, and real infrastructure. T1 and T2 run inside disposable Firecracker VMs. T3 runs only + on a clean host-direct qualification machine for behavior that cannot run correctly under nesting. - No mocking of any kind — the binary is real, the subprocesses are real. - Install the release candidate through `host install-system`; T1/T2 tests invoke exact `/usr/local/bin/mvm`. - Directly copying a binary onto the canonical path or relying on PATH lookup does not qualify the installer or release. - **Ground truth:** A feature is not tested until it has an L2 test. L0/L1 are fast pre-filters, not replacements. -- Run via `pytest tests/system/` inside the runner VM. +- Run through `scripts/run-system-tests.py`. It selects the execution tier and invokes pytest in the correct environment. ## 13. CLI Patterns diff --git a/docs/adr/0012-unified-test-architecture.md b/docs/adr/0012-unified-test-architecture.md index dd66e185..3dd6ba81 100644 --- a/docs/adr/0012-unified-test-architecture.md +++ b/docs/adr/0012-unified-test-architecture.md @@ -1,114 +1,116 @@ -# Unified Test Architecture — Go Hermetic + Nested-VM E2E +# Unified test architecture with Go pre-filters and Python system tests -**Status:** Active +**Status:** Active, amended for the three-tier runner on 2026-08-26 **Date:** 2026-06-20 -**Table of Contents** - -- [Context](#context) -- [Decision](#decision) -- [Considered Options](#considered-options) -- [Consequences](#consequences) -- [Related Documents](#related-documents) - ## Context -### The problem - -The project had two test layers — Go unit tests (`go test ./...`) and Python system tests (`tests/system/`) — with a large gap between them. The system test layer has since been migrated to a three-level architecture (see implementation below): - -| Layer | Language | Speed | Determinism | Lines | -|-------|----------|-------|-------------|-------| -| **Go unit tests** | Go | ms | ✅ Deterministic | ~8,000 | -| *(gap)* | — | — | — | *0* | -| **System tests** | Python | min | ❌ Flaky | ~22,000 | - -The Python system tests (27 files, ~22,000 lines) suffer from seven structural problems: global shared state (~80% `@pytest.mark.serial`), pervasive skipping (43/341 skip-prone scenarios), dependency on real infrastructure, no parallel execution, tautological assertions, massive unfocused files, and `zzz_destructive/` ordering hacks. +At the time of this decision, the project had fast Go unit tests and a large Python system-test suite with little +coverage between them. Many Python tests shared host state, depended on ordering, skipped scenarios when infrastructure +was missing, or asserted only command return codes. Those properties made a passing run weak release evidence. -### What already works +The exact test counts and directory layout from that period are historical facts, not a live inventory. Read +`tests/system/COVERAGE_MATRIX.md` and the registry in `scripts/run-system-tests.py` for the current suite. -Per [ADR-0010](0010-per-domain-api-interfaces.md), the CLI layer accepts narrow `api.*API` interfaces. The `internal/testutil/` package provides in-memory repos, mock APIs, `FakeRunner`, and `FakeNetOps` — all Go-based, deterministic, and fast. These are currently only used for pure unit tests. - -The `test_vm_nested_isolated.py` file (in `tests/system/vm/`) proves this pattern works. - -### The "Zero Blindside" Principle - -When all tests pass, the team must be able to ship with confidence that every public-facing CLI command and flag works correctly. This requires exhaustive coverage (every path), deterministic execution (no skips, no flakes), isolated state (disposable environments), and includable destructive tests (cleanup verified the same way as creation). +The CLI already exposed narrow `api.*API` interfaces. `internal/testutil/` also provided in-memory repositories and a +`FakeRunner`. The project could therefore add hermetic Go coverage without mocking the production binary used by system +tests. ## Decision -Adopt a **three-level test architecture** with a two-language boundary: - -| Level | Name | Language | Tests | Requirements | -|-------|------|----------|-------|--------------| -| **L0** | Pure function | Go | Table-driven tests, no I/O, no repos | None | -| **L1** | Hermetic integration | Go | Real SQLite (`:memory:`), real files (`t.TempDir()`), `FakeRunner` for subprocess calls only | None | -| **L2** | Runner VM E2E | Python | Real binary, real infrastructure inside a disposable Firecracker VM | Runner VM (nested KVM) | +Use three test levels with a two-language boundary: -### Key Sub-Decisions +| Level | Purpose | Language | Boundary | +|---|---|---|---| +| L0 | Pure function checks | Go | No I/O, repositories, or subprocesses | +| L1 | Hermetic integration checks | Go | Controlled SQLite and files, with `FakeRunner` only at subprocess boundaries | +| L2 | Real-binary system checks | Python | Installed CLI and real Linux infrastructure in the execution tier selected by the runner | -1. **The mocked CLI handler layer (proposed "L2") is eliminated.** Mocking the API layer creates more maintenance than value — every new method needs a mock field, interface changes cascade, and assertions risk tautology. L1 tests cover the same ground without mocking: seed the real DB, call the real repository, check the output. +L0 and L1 are fast pre-filters. They do not replace L2 coverage for user-visible behavior. L2 verifies the installed +binary through the CLI and observes the resulting operating-system state. -2. **L2 covers ALL user-facing features.** Every CLI command, every flag, every output format, every error path — verified with the real binary running inside a runner VM. No scenario is removed during migration. The Python test files stay; only the execution substrate changes (from the real host to a disposable VM). +The system-test runner uses a separate tier classification inside L2: -3. **L0/L1 are fast pre-filters, NOT replacements for L2.** They catch bugs earlier during `go test ./...` (milliseconds instead of minutes), but passing L0/L1 alone means nothing — the feature could be completely broken at the subprocess level. The release gate is L2 passing. +| Tier | Execution location | Use | +|---|---|---| +| T1 | Disposable runner VM | Host-level CLI behavior that does not require nested KVM | +| T2 | Disposable runner VM with nested virtualization | VM lifecycle and interaction through nested Firecracker | +| T3 | Clean host-direct qualification machine | Behavior that cannot be qualified correctly under nesting | -4. **`zzz_destructive/` ordering-by-filename is eliminated.** Destructive tests stay and are essential — but the runner VM makes ordering a convenience concern, not a correctness requirement. Each session starts clean. Parallel workers each get their own VM. +Test levels answer what kind of proof a scenario needs. Execution tiers answer where an L2 scenario can run. Do not use +the level and tier numbers interchangeably. -5. **L0/L1 tests are written in Go, share file locations with existing unit tests** (`internal/*/*_test.go`), and follow [HOW_AGENTS_WRITE_UNIT_TESTS.md](../development/HOW_AGENTS_WRITE_UNIT_TESTS.md) patterns. +### Sub-decisions -6. **The full decision tree and quick-reference table are documented in [HOW_AGENTS_WRITE_SYSTEM_TESTS.md](../development/HOW_AGENTS_WRITE_SYSTEM_TESTS.md).** That document is the authoritative how-to reference for classifying and writing tests at all three levels. +1. Do not add a mocked CLI-handler layer. L1 calls real package behavior with controlled dependencies. L2 calls the real + installed binary. +2. Keep user-visible behavior in the Python system-test coverage matrix. Run each scenario in T1 or T2 when nesting can + represent it. Reserve T3 for the documented host-only exceptions. +3. Keep destructive tests. Disposable runners and scoped fixtures provide isolation. Filename ordering is not a safety + mechanism. +4. Keep L0 and L1 tests next to Go source. The `engineer` role owns them. +5. Keep Python L2 tests under `tests/system/`. The `qa-engineer` role owns them and the runner. +6. Treat collection, skips, deselection, cleanup failure, and malformed outcome reports as release failures. A zero exit + code alone is not sufficient evidence. +7. Keep the installed outer controller separate from the release candidate. Install the candidate inside T1 and T2 + runners. T3 requires exact candidate identity at the canonical system path on a clean qualification machine. -## Considered Options +## Considered options -### Option A: Full Rewrite of Python Tests in Go (rejected) +### Rewrite the Python suite in Go -Translate all 22,000 lines of Python to Go. Loses pytest's fixture scoping, parametrization, marker filtering, yield-based teardown, and plugin ecosystem. Would need extensive custom infrastructure for runner VM orchestration. The cost of rebuilding pytest's capabilities in Go exceeds the benefit of a single language. +Rejected. Pytest already provides fixture scoping, parametrization, filtering, and teardown. Rebuilding those facilities +would not improve the real-binary boundary. -### Option B: Build Tag for Hermetic Binary (rejected) +### Build a tagged mock binary -Add a `mvmtest` build tag that replaces `DefaultRunner` with a controllable fake. Creates a second testing surface (does the injection work? does the env var parse correctly?). The binary that runs in CI would differ from production. L1 tests achieve the same result without a build tag by testing the handler function directly. +Rejected. A build tag would create a second executable behavior that release testing does not ship. L1 already provides +controlled dependency tests without changing the candidate binary. -### Option C: Three-Level Architecture (selected) +### Keep only Go unit tests and host-direct Python tests -Go for fast pre-filters. Python (inside nested VM) for ground truth. Each layer uses the right tool for the job. Incremental migration — no big-bang rewrite. +Rejected. Go-only tests miss subprocess and Linux integration failures. Host-direct tests expose developer state and +make cleanup failures dangerous. -### Option D: Status Quo (rejected) +### Use Go pre-filters and isolated Python system tests -Keep 22,000 lines with shared DB, pervasive skipping, and per-domain serial execution. The problems are structural, not cosmetic — no amount of test hardening fixes a shared-DB architecture. +Selected. Go catches local defects early. Python verifies the installed CLI against real infrastructure. Disposable +runners contain most destructive behavior, while T3 remains an explicit release-only exception. ## Consequences -**Positive:** -- Runner VM provides hermetic isolation — no shared DB, no orphaned bridges, no sudo on the host -- Parallel execution — multiple runner VMs, no `serial` markers -- `zzz_destructive/` ordering hack eliminated -- Skip culture eliminated — runner VM starts from pre-seeded snapshot -- L0/L1 fast pre-filters speed up the developer inner loop (milliseconds instead of minutes) -- SQLite-backed L1 tests catch schema bugs the current map-based mocks cannot detect - -**Negative:** -- Migration effort (estimated 8-12 weeks) -- Two languages, two test runners -- Runner VM provisioning time (~60-120s first run, mitigated by snapshot/restore) -- Runner VM resource overhead (~4 vCPU + 4 GB RAM + 20 GB disk per VM) -- ~2,000 new Go test lines to maintain -**Neutral:** - -- Per-domain conftest.py files consolidated into one -- `tests/system/` → `tests/e2e/` rename (pending — directory still at `tests/system/`) -- `COVERAGE_MATRIX.md` replacement with `go test -coverprofile` + L2 scenario manifest (pending — `COVERAGE_MATRIX.md` still exists) -- `scripts/run-system-tests.py` domain-looping replaced by `pytest tests/system/` (pending — CI still references `scripts/run_tests.py`) - -## Related Documents - -| Doc | Purpose | -|-----|---------| -| `docs/system-test-architecture.md` | **Primary implementation reference** — Three-tier orchestration, provisioning flow, fixture scoping, known-limitation patterns, per-file compliance checklist. Supersedes `DRAFT-system-test-architecture.md`. | -| `docs/development/HOW_AGENTS_WRITE_SYSTEM_TESTS.md` | L0/L1/L2 definitions, decision tree, quick-reference table, runner VM fixture pattern, migration phases. | -| `docs/development/HOW_AGENTS_WRITE_UNIT_TESTS.md` | L1 fast pre-filter tests (in-memory SQLite, temp dirs). | -| `docs/development/HOW_TO_RUN_SYSTEM_TESTS.md` | Execution plan for running system tests. | -| _(no RC_QA.md — this doc section moved into system-test-architecture.md)_ | Release gates: "L2 tests in runner VM" now in system-test-architecture.md. | -| `STANDARDS.md` §12 | Three-level architecture description. | -| `tests/system/COVERAGE_MATRIX.md` | Coverage tracking. | -| `AGENTS.md` | `engineer` owns L0/L1 (Go), `qa-engineer` owns L2 (Python runner VM). | +Positive consequences: + +- L0 and L1 failures arrive before expensive VM provisioning. +- T1 and T2 isolate databases, bridges, processes, and destructive cleanup from the outer host. +- The runner can execute independent domains concurrently in separate VMs. +- Strict outcome reports distinguish a real pass from an empty, skipped, or partially collected run. + +Costs and limits: + +- The project maintains Go and Python test code. +- T1 and T2 require runner provisioning and nested virtualization support. +- T3 still mutates the outer machine. It is release-only and requires a clean qualification host. +- The coverage matrix and executable registry must remain synchronized. + +## Current implementation record + +- `tests/system/` remains the canonical Python system-test directory. +- `scripts/run-system-tests.py` is the only supported outer orchestrator. +- `tests/system/COVERAGE_MATRIX.md` remains the coverage inventory. +- T1 and T2 use controller and candidate separation. +- T3 requires explicit `--host-direct`. Release qualification verifies candidate identity. The remaining clean-host + ownership and teardown gates stay release blockers in Task 17. +- Tier assignment and fixture scoping replace a global `pytest.mark.serial` convention. + +## Related documents + +| Document | Authority | +|---|---| +| `docs/system-test-architecture.md` | Architecture, tiers, fixture scope, runner protocol, and known limitations | +| `docs/development/HOW_AGENTS_WRITE_SYSTEM_TESTS.md` | L0, L1, and L2 classification plus system-test authoring patterns | +| `docs/development/HOW_AGENTS_WRITE_UNIT_TESTS.md` | Go test authoring patterns | +| `docs/development/HOW_TO_RUN_SYSTEM_TESTS.md` | Operator procedure for executing the runner | +| `docs/RC_QA.md` | Release-candidate qualification and evidence | +| `tests/system/COVERAGE_MATRIX.md` | Command, flag, and scenario coverage | +| `AGENTS.md` | Role ownership and safety rules | diff --git a/docs/development/HOW_AGENTS_WRITE_SYSTEM_TESTS.md b/docs/development/HOW_AGENTS_WRITE_SYSTEM_TESTS.md index 957e5f25..1307d155 100644 --- a/docs/development/HOW_AGENTS_WRITE_SYSTEM_TESTS.md +++ b/docs/development/HOW_AGENTS_WRITE_SYSTEM_TESTS.md @@ -1,5 +1,8 @@ # How Agents Write System Tests +This is the authoring guide for L0, L1, and L2 scenarios. The execution-tier architecture, fixture scope, runner +protocol, and known limitations live in [system-test-architecture.md](../system-test-architecture.md). + > **See also:** [ADR-0012](../adr/0012-unified-test-architecture.md) for the architectural decisions and rationale behind this document. > > **See also:** [HOW_AGENTS_WRITE_UNIT_TESTS.md](HOW_AGENTS_WRITE_UNIT_TESTS.md) for L0/L1 test patterns (table-driven tests, in-memory repos, FakeRunner, in-memory SQLite). @@ -23,7 +26,7 @@ ## Purpose -This document is a **specification and how-to reference** for writing and classifying tests at all three levels. It defines: +This document explains how to write and classify tests at all three levels. It defines: - **L0** (pure function Go tests) — when to write them, what they prove - **L1** (hermetic Go tests) — when to use in-memory SQLite, temp dirs, `FakeRunner` diff --git a/docs/development/HOW_AGENTS_WRITE_UNIT_TESTS.md b/docs/development/HOW_AGENTS_WRITE_UNIT_TESTS.md index c66e318b..8490a5af 100644 --- a/docs/development/HOW_AGENTS_WRITE_UNIT_TESTS.md +++ b/docs/development/HOW_AGENTS_WRITE_UNIT_TESTS.md @@ -2,9 +2,9 @@ ## Purpose -This guide defines the patterns every Go unit test in this project must follow. -Follow these patterns exactly — do not invent new test structure. If you need a -pattern not covered here, flag it for approval before deviating. +This guide defines the default patterns for Go tests in this project. Apply each +rule when its condition matches. Use a different idiomatic structure when the +test contract needs it, and explain the choice in the review. ## Table of Contents @@ -14,12 +14,12 @@ pattern not covered here, flag it for approval before deviating. 4. [Pattern 2: Repository via In-Memory Mock](#4-pattern-2-repository-via-in-memory-mock) 5. [Pattern 3: Service with Subprocess Mock](#5-pattern-3-service-with-subprocess-mock) 6. [Pattern 4: Error-Path-First Table](#6-pattern-4-error-path-first-table) -7. [Iron Rules (Violation = Rejected)](#7-iron-rules-violation--rejected) +7. [Test review rules](#7-test-review-rules) 8. [What to Assert — And What NOT to Assert](#8-what-to-assert--and-what-not-to-assert) 9. [How to Derive Expected Values — The Three-Source Rule](#8a-how-to-derive-expected-values--the-three-source-rule) 10. [File Structure Template](#9-file-structure-template) 11. [Verification Checklist](#10-verification-checklist) -12. [Mandatory Blind Adversarial Review](#11-mandatory-blind-adversarial-review) +12. [Risk-based adversarial review](#11-risk-based-adversarial-review) 13. [Appendix: Examples of Worthless Tests (DO NOT WRITE THESE)](#appendix-examples-of-worthless-tests-do-not-write-these) 14. [Appendix: Example of a GOOD Trustworthy Test](#appendix-example-of-a-good-trustworthy-test) @@ -27,18 +27,18 @@ pattern not covered here, flag it for approval before deviating. ## 1. The Foundation — What Makes a Test Trustworthy -A test is **trustworthy** if ALL three are true: +A test is trustworthy when these properties apply: 1. **RED-GREEN**: If the behavior is wrong, the test fails. If the behavior is correct, the test passes. There is no third state. -2. **SURVIVAL**: If you delete the function body and return zero values, the - test MUST fail. If the test still passes, it's worthless — delete it. +2. **REGRESSION SENSITIVITY**: A change that violates the tested contract makes + the test fail. Do not require unrelated rows to fail for one synthetic stub. 3. **DIFF**: When the test fails, the output shows EXACTLY what field differs and what the expected vs actual values are. Not just "not equal". -These three properties are non-negotiable. Every test is reviewed against them. +Review each test against the behavior it claims to protect. --- @@ -60,10 +60,10 @@ These three properties are non-negotiable. Every test is reviewed against them. | `FakeRunner` | `testutil/fake_runner.go` | Mock `CommandRunner` for subprocess-dependent tests | | `VMRepo`, `NetworkRepo`, ... | `testutil/*.go` | In-memory repository mocks for each domain | -### Import pattern for every test file +### Default import pattern ```go -package vm_test // external test package (black-box — tests public API only) +package vm_test // external test package for public behavior import ( "context" @@ -79,8 +79,9 @@ import ( ) ``` -**RULE:** Always use external test packages (`package vm_test`, not `package vm`). -This ensures you test the public API, not unexported internals. +Prefer an external test package when the contract is public. Use the internal +package when the test must cover an unexported parser, state machine, or other +package invariant. Do not export production symbols only to make a test external. --- @@ -134,14 +135,13 @@ func TestMyFunc(t *testing.T) { **Key rules for this pattern:** -- `map[string]struct{...}` — NOT a slice. Map iteration order is randomized, - which detects tests that accidentally depend on global state ordering. -- `t.Run(name, ...)` — NEVER loop directly with `for`. Each case must be a +- Use `map[string]struct{...}` when case order must not matter. Use a named + slice when deterministic order makes the contract or failure easier to read. +- Use `t.Run(name, ...)` for named table cases. Each case becomes a subtest so failures are independent and `-run` filtering works. - Error assertion BEFORE output assertion. If an error was expected, `return` immediately after asserting it. -- `cmp.Diff` for every output comparison. Never use `assert.Equal(t, want, got)` - without diff (see Iron Rule #3). +- Use `cmp.Diff` for structural output comparisons. See review rule R3. --- @@ -362,13 +362,12 @@ only success cases and forgetting errors. --- -## 7. Iron Rules (Violation = Rejected) +## 7. Test review rules -### R1: Every table must have at least one error/invalid case +### R1: Cover applicable error and boundary behavior -If a function returns `error`, you MUST test at least one path where it errors. -No "happy path only" tables. A function that never errors shouldn't return -`error`. +If the contract defines an observable error or boundary, cover it. Do not add an +invented invalid case to a function that cannot reject its input. ### R2: No tautological assertions @@ -427,13 +426,17 @@ you're testing that your mock wiring is correct — nothing else. Assert on the Exception: asserting `runner.Calls` is empty is acceptable to prove a NOOP (operation was correctly skipped). -### R7: Every cleanup path must be tested +### R7: Test cleanup that can leak state -If a function creates temporary resources, ensure you test what happens on -cleanup failure. At minimum, verify the cleanup runs. A function that leaks -resources on error paths is a bug. +If a function owns temporary resources, test the cleanup paths that can leak or +leave authority behind. Pure functions and read-only operations have no cleanup +requirement. -### R8: Context cancellation must be tested on any function that takes context +### R8: Test cancellation when the function owns cancellable work + +A `context.Context` parameter alone does not require a cancellation test. Add +one when the function blocks, retries, or owns side effects that cancellation +must stop. ```go t.Run("context_cancelled", func(t *testing.T) { @@ -480,8 +483,9 @@ by running the function under test and copying its output, the test is circular. It will pass even if the function is buggy, because the expected value was derived from the same buggy code. -DETECTION: If you remove the function body and return a zero value, -would the test fail? Yes? Good. But there's a more subtle variant: +A zero-value implementation can expose an assertion that never observes the +result, but it cannot prove that the expected value is independent. Consider +this subtler variant: ``` // SUBTLE CIRCULAR: setup() returns both (input, want) and the agent @@ -498,11 +502,9 @@ func setup(t *testing.T) ([]string, []fileEntry) { } ``` -This test would pass with buggy code because the expected value WAS -the buggy output. The SURVIVAL check ("delete function body → fail?") -passes because zero values don't match `"a.txt"`. But the test is -still worthless — it only proves the function is consistent with -itself. +This test passes because the expected value came from the buggy output. A +zero-value probe also fails, but that does not make the expectation correct. +The test proves only that the function is consistent with itself. PREVENTION: Ask THREE questions before every expected value: @@ -513,9 +515,8 @@ PREVENTION: Ask THREE questions before every expected value: the function?** If yes, compute it inline. If no, your test is under-specified (you don't know what correct behavior looks like). -3. **Does a contract comment explain why this value is correct?** - If a reviewer can't tell why `"a.txt"` is the expected value - (vs `"tests/a.txt"` or `"./a.txt"`), add a contract comment. +3. **Can a reviewer tell why this value is correct?** If the derivation is not + obvious from the test input and name, add a contract comment. PRACTICE: Prefer computing expected values from test inputs rather than hardcoding them: @@ -532,65 +533,22 @@ This is harder for the agent to get wrong because the contract rule is written down and the expected value is computed by a different expression than the function under test. -### R0: Mandatory SURVIVAL — every test must fail when the function is gutted - -The SURVIVAL property from Section 1 is non-negotiable. - -Take the function under test, delete its body, and return zero values -(zero, nil, empty struct, empty slice). If ANY test case still passes, -that test case is worthless and must be removed or rewritten before the -test file can be committed. +### R11: Every primary assertion must detect a contract regression -This applies to EVERY table row and EVERY subtest — not just the test -function as a whole. A table with 5 rows where 4 survive the gutted -function is still defective. - -``` -FORBIDDEN: - // Function: func Sum(a, b int) int { return a + b } - // Test row: {a: 1, b: 2, want: 3} - // Gutted: func Sum(a, b int) int { return 0 } - // - // SURVIVAL: 0 != 3 → PASS (dies on gutted function) ✓ - - // Function: func GetName(u *User) string { return u.Name } - // Test row: {input: &User{Name: "alice"}, want: "alice"} - // Gutted: func GetName(u *User) string { var s string; return s } - // - // SURVIVAL: "" != "alice" → PASS (dies on gutted function) ✓ - - // Function: func Validate(s string) error - // Test row: {input: "", wantErr: "empty"} - // Gutted: func Validate(s string) error { return nil } - // - // SURVIVAL: nil != error → PASS (dies on gutted function) ✓ - -WORTHLESS (survives gutted function): - // Function: func Exists(path string) bool { return fileExists(path) } - // Test row: {input: "/tmp/exists.txt", want: true} - // Gutted: func Exists(path string) bool { return true } - // - // SURVIVAL: true == true → FAIL (test passes with gutted function) - // The test doesn't prove Exists works — it just proves that - // returning true is accepted. Gutting to `return false` would - // fix survival but expose that no real file check happens. - - // Function: func ToUpper(s string) string { return strings.ToUpper(s) } - // Test row: {input: "a", want: "A"} - // But the test uses `assert.Equal(t, want, want)` — tautology. - // Gutted function returns "" — but the assertion compares want - // to want, not to got. Test never fails. -``` +Mentally replace the behavior under test with a plausible wrong implementation. +The primary assertion must fail. A zero-value stub is one useful probe, but it +is not a universal quality test. Some valid contracts return zero values for +valid inputs. -DETECTION: Before submitting, mentally inline each assertion. Replace -the function body with `return zero`. If the assertion passes, delete -the test or strengthen the assertion until it fails. +A test is weak when its assertion ignores the returned value, repeats a value +created by setup, or checks only a count when item content is the contract. +Strengthen or remove that assertion. -### R11: The Mirror Test — every write must be read back +### R12: Read back persistence contracts -If the function under test writes, creates, or mutates data, the test -must read back the result and verify it byte-for-byte or field-for-field. -"Error is nil" or "file exists" is NOT sufficient verification. +If the contract promises durable content or state, read it back through an +independent path and compare the meaningful payload. A mutation whose contract +is only an emitted call or returned value can use that observable result. ``` FORBIDDEN — only checks error, never reads the written file: @@ -624,15 +582,15 @@ REQUIRED — Mirror Test: } ``` -READ THE WHOLE PAYLOAD. Not just metadata (exists, size, mode). -Not just error. Read the actual bytes or retrieve the actual entity -and compare every field. +Read the bytes or fields promised by the persistence contract. Existence or a +nil error alone does not prove stored content. -### R12: Assert every output field +### R13: Assert every field in the tested contract -If the function returns a struct, map, slice, or multi-value output, -every meaningful field must be explicitly asserted. A field that isn't -asserted is a field that can be wrong without detection. +For a struct, map, slice, or multi-value result, assert each field that belongs +to the behavior named by the test. Use a full structural comparison when the +whole value is the contract. Do not make a focused test brittle by asserting +unrelated fields. ``` GOOD (every field checked): @@ -695,7 +653,7 @@ EXCEPTION EXAMPLE: | Exact mock call arguments | Tests implementation, not behavior | | String the test constructed | Tautology — proves nothing | | Line numbers in errors | Brittle — change with file edits | -| Internal/private functions | External test package enforces this | +| Private implementation details in public-contract tests | Assert through the package contract instead | | Order of map iteration | Undefined by Go spec | | Timestamps or durations | Flaky — use `assert.WithinRange` or don't assert | | Count without content | `assert.Len(t, items, 3)` doesn't verify WHICH items are there | @@ -712,7 +670,7 @@ wrong thing. The test runs, the code runs, they match — but both are wrong. This happens when the agent derives expected values from the function's output instead of from the function's contract. -Every expected value must come from EXACTLY ONE of three sources: +Every expected value must come from one or more independent sources: | Source | What it means | Example | |--------|---------------|---------| @@ -766,9 +724,8 @@ the test fails. ### The LITERAL rule -If the expected value is a hardcoded literal in the test body (not -computed from inputs), the test MUST explain in a comment what -contract rule that literal satisfies: +If a hardcoded expected value is not self-explanatory from the test name and +input, add a comment that names the contract rule it satisfies: ``` GOOD (contract documented): @@ -786,15 +743,11 @@ BAD (no rationale): // copied from the function output? The reviewer cannot tell. ``` -### The SURVIVAL check for expected values - -In addition to the test-level SURVIVAL check ("would deleting the -function body fail this test?"), apply SURVIVAL to EACH expected value: +### Assertion sensitivity -Take the expected value, replace it with a clearly wrong value, and -mentally check: would the test fail? If the answer is "maybe not" -(e.g., because the assertion is on length, not content, or because -the other fields would still match), the assertion is too weak. +Replace an expected value with a clearly wrong value and check whether the test +would fail. If it might still pass because the assertion checks only a count or +an unrelated field, strengthen the assertion. ### Independence test @@ -806,10 +759,10 @@ expected value is implementation-derived and must be replaced. ## 9. File Structure Template -Every test file follows this exact structure: +Use this structure when it fits the package: ```go -package _test // external test package +package _test // default for a public package contract import ( "context" @@ -839,180 +792,73 @@ func Test(t *testing.T) { } ``` -**Rules:** -- Section comments use `// ---` with plain ASCII dashes (≤ 60 chars including `// `) -- Every test function has a `// Rationale:` comment explaining why this - test exists and what real bug it prevents -- Test functions are ordered by dependency (foundation first, consumers later) -- Each file tests ONE Go file from the source package (file name match) +Keep related tests together and order them so setup helpers appear before their +consumers. Add a rationale comment only when the contract or regression is not +clear from the test name and assertions. A test file may cover several source +files when they implement one package-level behavior. --- -## 10. Verification Checklist +## 10. Verification checklist -Before submitting ANY test file, verify every item: +Before submitting a Go test change, verify the applicable items: -``` -[ ] Does every test function have at least one error/invalid case? -[ ] Is every assertion on BEHAVIOR, not implementation (mock calls)? -[ ] Would deleting the production function body make this test fail? - (Try it mentally — if not, the test is worthless) -[ ] Does every cmp.Diff call use the (-want +got) format string? -[ ] Does every table use map[string]struct{...} with t.Run()? -[ ] Does every error case return immediately after asserting the error? -[ ] Is `require` used for setup and `assert` for test logic? -[ ] Are there zero tautological assertions (echoing inputs)? -[ ] Does the test use external package (_test suffix)? -[ ] Does go test ./... compile cleanly? -[ ] Does go test -race ./... pass with no data races? -[ ] Is every expected value independently derivable from the function's - contract without running the function? (Not copied from its output) -[ ] Does every hardcoded expected literal have a CONTRACT comment - explaining WHY that value is correct? -[ ] SURVIVAL: Would replacing the function body with `return zero` - make this test fail? (Every row, every subtest.) -[ ] Mirror Test: Does every write/create/mutate read back the payload - and compare byte-for-byte or field-for-field? -[ ] Are ALL fields of every returned struct/slice/map asserted? Any - unasserted fields documented with a reason? +```text +[ ] The test names the behavior or regression it protects. +[ ] Primary assertions observe behavior, durable state, or a documented emitted call. +[ ] Expected values come from the contract or an independent computation. +[ ] A plausible wrong implementation makes each primary assertion fail. +[ ] Error and boundary cases cover the contract without invented inputs. +[ ] Named table cases use t.Run. +[ ] Structural comparisons show a useful (-want +got) diff. +[ ] Setup failures stop the test before later assertions run. +[ ] Persistence contracts read back the meaningful payload. +[ ] Cancellation, cleanup, and race cases exist when the function owns those risks. +[ ] The chosen external or internal test package matches the boundary under test. +[ ] The smallest affected go test command passes. +[ ] go test -race passes for changed concurrent code. ``` --- -## 11. Mandatory Blind Adversarial Review - -Every test file MUST pass a blind adversarial review before submission. The -review is performed by a SEPARATE agent instance that has ZERO knowledge of -what the test is supposed to do or what bug it was written to catch. +## 11. Risk-based adversarial review -### Why blind review is mandatory - -If the reviewer knows "this test is for `ToInt`", they will subconsciously -confirm that the test looks correct — even if the assertion is tautological, -the edge cases are missing, or the expected value is wrong. A blind reviewer -with no context can only judge what the code ACTUALLY does, not what it was -INTENDED to do. - -### The blind review protocol - -**Step 1: Generate the diff.** - -Before spawning the reviewer, the writer agent records the current state: - -```bash -git rev-parse HEAD # save baseline SHA -``` - -Then makes ALL changes. After the last edit, run: - -```bash -git diff # diff against the commit before any changes -``` +Routine, focused Go test changes require the author to run the checklist and the +smallest affected `go test` command. They do not require a second agent. -Do NOT use plain `git diff` (which compares against HEAD — if previous changes -were committed, HEAD already includes them and the reviewer won't see them). +Require independent adversarial review when the test change covers any of these +areas: -**Step 2: Spawn a reviewer with NO context.** +- privileged dispatch, caller identity, path safety, or authorization +- concurrency, locking, cancellation, crash recovery, or partial failure +- strict codecs, parsers, protocol limits, or untrusted input +- process identity, cgroups, namespaces, mounts, firewall policy, or cleanup +- release qualification or a broad change that becomes the only proof for several + production paths -The writer agent spawns a `general` subagent with this EXACT prompt (do NOT -modify it): - -``` -You are a BLIND adversarial code reviewer. You do NOT know what the author -intended to fix or test. You judge only what the code ACTUALLY does. - -Review this git diff of Go test files against the following rules. -The rules are non-negotiable — if ANY rule is violated, report FAIL with -the exact file:line and the rule violated. - -RULES: -1. Every table must have at least one error/invalid/boundary case. -2. No tautological assertions (asserting something the test just constructed). -3. cmp.Diff must be used for all structural comparisons (structs, slices, maps). -4. require.* for setup, assert.* for test logic. -5. After asserting an expected error, the test must RETURN immediately. -6. Primary assertion must be on BEHAVIOR (return value, state change), - not on mock wiring (mock.Calls). -7. Must use `t.Run(name, ...)` for every row — no flat loops. -8. Context cancellation must be tested if the function takes context.Context. -9. Every cleanup or error path must be tested. -10. The test file MUST compile with `go vet` and pass `go test -race`. -11. Expected values must be derivable from the function's contract, not - its implementation. If a hardcoded literal is used as the expected - value, the test must include a comment explaining what contract rule - produces that value. A test that passes only because the expected - value matches the current implementation (not an independent spec) - is circular and must be rejected. -12. SURVIVAL: If the function body were replaced with `return zero` / - `return nil`, the test MUST fail. If any assertion survives a gutted - function body, that assertion is redundant or tautological. -13. Mirror Test: Every write/create/mutate must be followed by a read-back - that verifies the payload byte-for-byte. Checking only error or - existence is insufficient. -14. Every field of a returned struct, slice element, or map value must be - explicitly asserted unless documented as non-deterministic or derived. - -Report: -- PASS: no issues found -- FAIL: [rule X violated] [file:line] — explain what's wrong - -Read every changed file in full to verify context. Do NOT assume anything -about what the author intended. -``` - -**Step 3: The reviewer reads changed files, not just the diff.** - -The prompt MUST instruct the reviewer to read every changed file in full. -A diff alone can hide context (e.g., a test that constructs a value and then -asserts it's the same — the diff shows the assertion but not the construction). - -**Step 4: Fix ALL violations — no debt accumulation.** - -If the reviewer reports FAIL, the writer agent fixes EVERY violation. The fix -must REPLACE the offending code, not ADD a band-aid on top. Examples: - -``` -WRONG (adds a patch on top of buggy code): - if diff := cmp.Diff(want, got); diff != "" { - t.Errorf("mismatch: %s", diff) // wrong format string - } - // ADDED: sorry, let me also add the proper format - t.Logf("for debugging: want=%v got=%v", want, got) - -RIGHT (replaces the buggy line entirely): - if diff := cmp.Diff(want, got); diff != "" { - t.Errorf("(-want +got):\n%s", diff) // correct format string - } -``` - -After fixing, re-run steps 1-3 (re-diff, re-review) until the reviewer -reports PASS. Loop until clean. - -**Step 5: Writer reports the reviewer's verdict to the user.** - -``` -Tests written for: internal/core/vm/service_test.go -Blind review verdict: PASS -Changes: 1 file, +187 lines -All 10 rules verified. -``` +Give the reviewer the contract, the production files, the test diff, and the +failure that the tests must detect. A blind reviewer is not the default. Hiding +intent can also hide the contract that distinguishes a valid assertion from a +plausible but wrong one. -### Why debt accumulation is forbidden +The reviewer checks these points: -Agents often fix a violation by ADDING code that compensates for the buggy -code, rather than REPLACING the buggy code with correct code. This creates -test debt: the file becomes longer, harder to read, and the original bug -remains dormant. The blind review catches this because the reviewer sees -both the old and new code, and if the old code is still present, the -violation is still there. +1. Each primary assertion fails for a plausible contract regression. +2. Expected values come from the contract or an independent computation. +3. Error, cancellation, cleanup, and boundary cases match the behavior under test. +4. Structural comparisons reveal the field that differs. +5. Persistence tests read back the meaningful payload through an independent path. +6. Mock-call assertions remain secondary unless the emitted call is the contract. +7. The focused test command, `go vet` scope, and `go test -race` scope match the + risk. -### Enforcement +Use pstack `interrogate` or another independent reviewer when available. Treat +findings as claims to verify against the contract. Fix valid findings and record a +specific reason for rejecting invalid ones. Re-run the focused checks after every +accepted fix. -This step is NOT optional. Any test file submitted without a blind adversarial -review report is considered UNREVIEWED and MUST be rejected in code review. -The reviewing engineer checks that the report exists and that the reviewer -agent was given the EXACT prompt above (no modifications that could bias the -result). +Report the reviewed files, the reviewer verdict, accepted findings, rejected +findings with reasons, and the commands that passed. --- @@ -1085,7 +931,7 @@ func TestToInt(t *testing.T) { Why this is trustworthy: 1. **RED-GREEN**: If ToInt returns wrong value, assertion fails -2. **SURVIVAL**: If ToInt unconditionally returns 0, every non-zero test fails +2. **REGRESSION SENSITIVITY**: An unconditional zero result fails the non-zero cases 3. **DIFF**: `cmp.Diff` shows exact value mismatch with `(-want +got)` format 4. **EDGE CASES**: nil, non-numeric string, bool, zero value — all tested 5. **NO TAUTOLOGY**: Input values are different from expected outputs diff --git a/tasks/todo.md b/tasks/todo.md index cf9144f5..327fa1c6 100644 --- a/tasks/todo.md +++ b/tasks/todo.md @@ -392,6 +392,8 @@ under `internal/service/jailer/`. No handler, transport, CLI, API, core-domain, **Owner:** engineer and qa-engineer **Dependencies:** Tasks 3-17 +- [ ] Add a deterministic CI architecture check that rejects production imports between sibling `internal/core/*` + domains. The check must distinguish a package importing itself from one domain importing another. - [ ] `go mod tidy && git diff --exit-code` - [ ] `test -z "$(gofmt -l .)"` - [ ] `golines --max-len=120 --no-reformat-tags --list-files ./internal/ ./pkg/ ./cmd/` From bbb34334bda384d36363b967a96d35f374a4c611 Mon Sep 17 00:00:00 2001 From: AlanD20 Date: Fri, 28 Aug 2026 20:42:04 +0200 Subject: [PATCH 0043/1457] docs: queue direct nftables kernel adapter --- CHANGELOG.md | 5 +- CONTEXT.md | 15 ++++-- ...m-network-namespaces-and-typed-policies.md | 49 +++++++++++++++++-- tasks/plan.md | 23 +++++++-- tasks/todo.md | 18 +++++-- 5 files changed, 93 insertions(+), 17 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 7e4981ae..f305a70e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -193,8 +193,9 @@ The following items are release blockers and are intentionally not described abo the public root `mvm run jailer` and `mvm run provision` entry points. - Make one root-owned network namespace mandatory per VM, pass its pinned handle to Jailer, and make namespace/link creation, activation, rollback, reconciliation, and cleanup part of the owner-bound VM lifecycle. -- Remove the iptables backend and backend selector; apply one atomic nftables generation that default-denies both - same-network and cross-network VM traffic while preserving required system traffic and internet egress. +- Remove the iptables backend, backend selector, and production `nft` CLI subprocess/parser. Apply one atomic generation + directly through a reviewed, pinned `github.com/google/nftables` version that default-denies both same-network and + cross-network VM traffic while preserving required system traffic and internet egress. - Replace routed `ServiceAccessPolicy` with typed `traffic` policies and remove the `allow_remote_exec` trust mesh in favor of exact directional, non-root, resource-bounded VM-to-VM `exec` policies. - Replace the transitional wildcard/raw-tool sudoers entries with access only to the reserved privileged marker of the diff --git a/CONTEXT.md b/CONTEXT.md index c28dc232..9206620d 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -26,7 +26,7 @@ MicroVM Manager -- a speed-first CLI for managing Firecracker microVMs. Provides - [Public API boundary](#public-api-boundary) - [CLI is the canonical interface](#cli-is-the-canonical-interface) - [Provisioner Backend (LoopMount vs GuestFS -- mutual exclusion)](#provisioner-backend-loopmount-vs-guestfs--mutual-exclusion) - - [Firewall Backend (nftables vs iptables -- mutual exclusion)](#firewall-backend-nftables-vs-iptables--mutual-exclusion) + - [Firewall backend transition](#firewall-backend-transition) - [Build output (REQUIRED)](#build-output-required) - [Asset mirror environment variable (OPTIONAL)](#asset-mirror-environment-variable-optional) - [Relationships](#relationships) @@ -309,9 +309,11 @@ Provisioner type resolved ONCE at startup in `api.NewOperation()` by reading `se **Why no fallback chain:** A fallback (try loop-mount, fall back to guestfs) was rejected because: (1) if a user enables GuestFS, they expect GuestFS behavior -- silent fallback to loop-mount violates least surprise; (2) each backend has different sudoers requirements -- mixing them in one session increases the privilege surface; (3) each backend has independent test suites -- a fallback chain requires testing all combinations; (4) an earlier version incorrectly described GuestFS as a "fallback," which caused regression bugs where a stale `guestfs_enabled=true` silently selected the slow backend. -### Firewall Backend (nftables vs iptables -- mutual exclusion) +### Firewall backend transition -Two independent firewall backends -- **nftables** (default) and **iptables** (legacy) -- selected by the `firewall_backend` setting. Exactly one is active per session. +The transitional implementation has two independent firewall backends -- **nftables** (default) and **iptables** +(legacy) -- selected by the `firewall_backend` setting. Exactly one is active per session. This describes existing code, +not the accepted v0.3.0 target. - **nftables** (default): Uses `internal/lib/firewall/nftables.go` -- atomic `nft -f -` batch files. - **iptables** (legacy): Uses `internal/lib/firewall/iptables.go` -- per-rule `iptables` calls. @@ -324,6 +326,13 @@ Selection logic in `firewall.NewFirewallTracker(backend, xtcommentAvail, db)`. B The `FirewallTracker` also reads an `iptables_xtcomment` user setting that adds comment tags to iptables rules for easier identification. +ADR-0017 defines the clean-break v0.3.0 replacement. It removes iptables, the backend selector, backend-specific +repositories, and the production `nft` CLI path. The root-owned `internal/service/firewall/` adapter will accept only +mvmctl-owned typed intent and use a reviewed, pinned `github.com/google/nftables` version to program the kernel directly +in one complete transaction. Library and netlink types stay private to that adapter; there is no CLI fallback. Kernel +nftables, bridge-family, connection-tracking, NAT, and privilege capabilities remain host requirements. A disposable QA +runner may use the `nft` CLI only as an independent read-only oracle. + ## Relationships - An **Operation** struct method orchestrates across **Domains** diff --git a/docs/adr/0017-per-vm-network-namespaces-and-typed-policies.md b/docs/adr/0017-per-vm-network-namespaces-and-typed-policies.md index 3690dcda..be16761c 100644 --- a/docs/adr/0017-per-vm-network-namespaces-and-typed-policies.md +++ b/docs/adr/0017-per-vm-network-namespaces-and-typed-policies.md @@ -2,6 +2,7 @@ **Status:** Accepted — v0.3.0 implementation pending **Date:** 2026-08-23 +**Last Updated:** 2026-08-28 (direct Go nftables adapter) **Supersedes:** [ADR-0014: Routed Service-Access Policies](0014-routed-service-access-policies.md) **See also:** [ADR-0016: Root-Owned Single-Binary Privileged Dispatch](0016-root-owned-single-binary-privileged-dispatch.md) @@ -76,6 +77,23 @@ verifies namespace inode, interface ownership, process identity, and cgroup memb v0.3.0 requires nftables. The iptables backend, configuration option, repositories, renderers, fallback behavior, and tests are removed. +The root-owned adapter in `internal/service/firewall/` programs the kernel through a reviewed, pinned version of +`github.com/google/nftables`. The module speaks the nftables netlink protocol directly; `github.com/mdlayher/netlink` +may remain its transitive implementation dependency but is not a second mvmctl adapter. Production does not execute or +parse the `nft` CLI, link `libnftnl`, depend on `nftables.service`, or fall back to a CLI implementation. Hosts still +require the applicable kernel nftables, bridge-family, connection-tracking, and NAT capabilities, and the privileged +receiver still requires the authority needed to program them. + +Only mvmctl-owned typed intent crosses the privileged protocol. Types from `github.com/google/nftables` and +`github.com/mdlayher/netlink` remain private to the root-side firewall adapter. One reconciliation constructs a complete +generation and submits it in one `Conn.Flush` transaction. The adapter takes `context.Context` first, binds cancellation +and deadlines to the netlink operation, and maps failures to `pkg/errs.DomainError`; a cancelled operation cannot be +reported as successful. + +Host readiness uses direct kernel capability probes rather than executable lookup, `nft --version`, or parsing CLI +diagnostics. The `nft` CLI may remain installed in disposable QA runners as an independent, read-only observation +oracle. It is not a production dependency or a sudo target. + One nftables implementation owns complete atomic generations for: - bridge-family same-network filtering and source MAC/IP anti-spoofing; @@ -183,6 +201,19 @@ case, root-owned verified plugins/configuration, and a separate security decisio Rejected. Same-network enforcement requires bridge-family filtering, while maintaining two renderers and reconciliation paths creates security-sensitive duplication. nftables is the single supported implementation. +### Keep the `nft` CLI as the production adapter + +Rejected. Rendering privileged command text and parsing human-oriented output retains an avoidable root subprocess, +adds quoting and version-dependent error surfaces, and weakens typed inspection. The existing `nft -f -` path is already +atomic, so the direct adapter is selected for a smaller and more structured effect boundary rather than for new +atomicity. + +### Build directly on `github.com/mdlayher/netlink` + +Rejected. It would make mvmctl own low-level nftables message and expression encoding already provided by +`github.com/google/nftables`. The lower-level module remains an implementation detail unless a proven missing capability +requires a separate decision. + ### Generic policy selectors and actions Rejected. A generic policy engine would mix enforcement planes and make unsupported combinations appear valid. New @@ -204,15 +235,25 @@ capabilities require new typed policy families and explicit enforcement contract - Every VM adds one namespace, veth pair, and namespace-local bridge. - Launch and cleanup perform more privileged steps and require transactional rollback. - Hosts must provide network namespaces and nftables with bridge-family support. +- The selected Go module is pre-v1 and must remain exactly pinned, reviewed, and covered by compatibility tests before + an upgrade. +- Direct netlink expressions are less convenient for operators to inspect than rendered rules, so QA retains an + independent read-only observation path. - Existing installations and automation must be recreated for v0.3.0. - Troubleshooting must distinguish guest, namespace, host bridge, routed, nftables, and vsock planes. ## Release Gate -This decision is not complete until L1 and L2 tests prove namespace ownership, cross-UID denial, anti-spoofing, -same-network and cross-network default deny, exact traffic allows, exec-policy directionality, non-root enforcement, -atomic reconciliation, crash recovery, reboot cleanup, and absence of leaked processes, namespaces, links, cgroups, -mounts, sockets, and nftables rules. +Before production integration, a focused compatibility spike must prove the pinned Go module supports bridge-family +MAC/IP anti-spoofing, same-network and routed filtering, established/related state, TCP/UDP ranges, NAT masquerade, +stable mvmctl generation identity, namespace-scoped connections, context cancellation/deadlines, and structured +inspection. It must also prove that one invalid expression aborts the complete transaction and benchmark a +representative 1,000-VM generation within a reviewed budget. + +This decision is not complete until L1 and L2 tests prove those adapter properties plus namespace ownership, cross-UID +denial, anti-spoofing, same-network and cross-network default deny, exact traffic allows, exec-policy directionality, +non-root enforcement, atomic reconciliation, crash recovery, reboot cleanup, and absence of leaked processes, +namespaces, links, cgroups, mounts, sockets, and nftables rules. ## Related Decisions diff --git a/tasks/plan.md b/tasks/plan.md index 60a30906..1eec483d 100644 --- a/tasks/plan.md +++ b/tasks/plan.md @@ -11,8 +11,9 @@ Release v0.3.0 as a clean installation with one unprivileged public CLI and one authority, and fail closed under multi-user, path-race, process-reuse, partial-failure, and crash conditions. Firecracker must always run through Jailer with cgroup v2 and one network namespace per VM. nftables becomes the only -firewall backend. Managed traffic is default-deny for both same-network and cross-network VMs. IP connectivity and -VM-to-VM vsock execution use separate typed `traffic` and `exec` policies. +firewall backend and is programmed directly through a reviewed, pinned `github.com/google/nftables` version rather than +the `nft` CLI. Managed traffic is default-deny for both same-network and cross-network VMs. IP connectivity and VM-to-VM +vsock execution use separate typed `traffic` and `exec` policies. The release does not migrate, adopt, backfill, or preserve legacy databases, runtime state, policies, remote-exec flags, Jailer directories, cgroups, TAP devices, or firewall rules. It detects ambiguous legacy host state and refuses to @@ -253,6 +254,18 @@ Namespace paths and interface names are derived from UID and immutable IDs. The down, installs the complete nftables generation and anti-spoofing rules, then brings links up and passes a pinned namespace handle to Jailer. +The root-side adapter lives in `internal/service/firewall/` and receives only mvmctl-owned typed intent. It uses a +reviewed, pinned `github.com/google/nftables` version to submit one complete generation through one `Conn.Flush` call. +Neither its types nor `github.com/mdlayher/netlink` types cross the privileged wire or enter the network domain. The +latter may remain a transitive dependency only; mvmctl does not maintain a second low-level netlink adapter. + +Production does not execute or parse the `nft` CLI, use `libnftnl`, depend on `nftables.service`, or retain a CLI +fallback. Host readiness probes kernel capabilities directly. The `nft` CLI may be present in disposable QA runners as +an independent read-only state oracle, never as a production dependency or sudo target. Before integration, a focused +compatibility spike must prove every expression and family required by ADR-0017, atomic invalid-batch failure, +namespace-scoped connections, structured inspection, cancellation/deadline behavior, and a representative 1,000-VM +generation within a reviewed performance budget. + The public policy families are: - `mvm policy traffic`: TCP/UDP from a source network or exact source VM to an exact destination VM port/range. @@ -304,7 +317,7 @@ root VM authority + descriptor-safe root records/locks typed Jailer lifecycle actions + remove public legacy Jailer service | v -root network authority --> per-VM netns/veth/TAP --> nftables-only generation +root network authority --> per-VM netns/veth/TAP --> direct Go nftables generation | | | v | traffic policy replacement @@ -366,7 +379,8 @@ Required staged Python gates: 3. **Jailer/cgroup/mount (T2/T3):** canonical launch, exact pair, namespace visibility, snapshot/volume parity, PID-reuse and kill-stage recovery. 4. **Network/traffic (T2):** netns membership, topology ownership, spoof denial, same/cross default deny, exact allows, - host restrictions, egress, atomic failure, reboot reconciliation, and nftables-only host readiness. + host restrictions, egress, atomic failure, reboot reconciliation, direct-kernel host readiness, and an independent + read-only nftables-state oracle. 5. **Exec (T2):** same/cross/no-network behavior, directionality, missing policy, non-root target, source identity, timeout/output/frame/concurrency bounds, command redaction, and target cleanup. 6. **Clean release (T2 plus clean host):** clean-install refusal of legacy state, full user journeys, repeated fault @@ -409,6 +423,7 @@ The user has approved: - removal of raw sudo and the legacy public privileged services; - mandatory per-VM network namespaces; - nftables-only enforcement and removal of iptables; +- direct kernel nftables programming through a reviewed, pinned Go module with no production CLI fallback; - same-network and cross-network default deny; - typed `traffic` and `exec` policies; - removal of `allow_remote_exec`; diff --git a/tasks/todo.md b/tasks/todo.md index 327fa1c6..afc266b2 100644 --- a/tasks/todo.md +++ b/tasks/todo.md @@ -245,13 +245,23 @@ under `internal/service/jailer/`. No handler, transport, CLI, API, core-domain, **Owner:** engineer **Dependencies:** Tasks 3 and 9 -- [ ] Add typed privileged nftables generation/application; caller cannot pass syntax, files, includes, or commands. +- [ ] Run a focused compatibility spike against a reviewed, pinned `github.com/google/nftables` version. Prove every + required bridge/IP/NAT expression, structured inspection, namespace-scoped connections, cancellation/deadlines, + atomic invalid-batch failure, and a representative 1,000-VM generation within a reviewed budget. +- [ ] Add typed privileged nftables generation/application under `internal/service/firewall/`; caller cannot pass syntax, + files, includes, commands, or library/netlink types across the privileged wire. +- [ ] Keep `github.com/mdlayher/netlink` transitive only; do not add a second low-level adapter or a production CLI + fallback. - [ ] Install bridge-family anti-spoofing and same-network filtering. - [ ] Install routed, VM-to-host, established reply, required system service, egress, and NAT rules. -- [ ] Apply complete generations atomically before links become usable. +- [ ] Apply each complete generation with one `Conn.Flush` transaction before links become usable. - [ ] Propagate every initialization, parent-chain, ordering, and batch failure. -- [ ] Remove iptables renderer/repository/tests, backend enum/selector/config, raw sudo rule, and documentation. -- [ ] Host readiness requires nftables bridge-family support. +- [ ] Remove iptables renderer/repository/tests, backend enum/selector/config, production `nft` subprocess/parser, raw + sudo rule, and obsolete documentation; do not require `libnftnl` or `nftables.service` at runtime. +- [ ] Probe required kernel nftables, bridge-family, connection-tracking, and NAT capabilities directly rather than + using executable lookup, `nft --version`, or parsed CLI diagnostics. +- [ ] L1 proves typed compilation, atomic failure, cancellation, reconciliation, and scale. L2 may use the `nft` CLI only + as an independent read-only oracle and proves the installed product has no production CLI dependency. ### Task 11: Replace service-access policy with typed traffic policies From 904932b06f0c9cccb88222c8159f1878c1090733 Mon Sep 17 00:00:00 2001 From: AlanD20 Date: Fri, 28 Aug 2026 21:49:53 +0200 Subject: [PATCH 0044/1457] feat: add bounded privileged control transport --- internal/lib/system/privileged_control.go | 697 +++++++++++++ .../lib/system/privileged_control_test.go | 949 ++++++++++++++++++ 2 files changed, 1646 insertions(+) create mode 100644 internal/lib/system/privileged_control.go create mode 100644 internal/lib/system/privileged_control_test.go diff --git a/internal/lib/system/privileged_control.go b/internal/lib/system/privileged_control.go new file mode 100644 index 00000000..255dedec --- /dev/null +++ b/internal/lib/system/privileged_control.go @@ -0,0 +1,697 @@ +package system + +import ( + "context" + "errors" + "fmt" + "io" + "os" + "os/exec" + "sync" + "syscall" + "time" + + "golang.org/x/sys/unix" + + "mvmctl/internal/infra" + "mvmctl/pkg/errs" +) + +const ( + privilegedControlSudoPath = "/usr/bin/sudo" + privilegedControlMaxActionBytes = 64 + privilegedControlMaxRequestFrameBytes = 8 + 4 + 8 + 64*1024 + privilegedControlMaxPayloadBytes = 128 * 1024 * 1024 + privilegedControlMaxResponseBytes = 8 + 4 + 64*1024 + privilegedControlMaxDiagnosticBytes = 8 * 1024 + privilegedControlDefaultReapTimeout = 5 * time.Second + privilegedControlDefaultDrainTimeout = time.Second +) + +func privilegedControlEnvironment() []string { + return []string{ + "PATH=/usr/sbin:/usr/bin:/sbin:/bin", + "HOME=/", + "LANG=C", + "LC_ALL=C", + } +} + +// PrivilegedControlRequest is a single-use, bounded request for the fixed +// privileged control transport. Successful construction transfers payload ownership. +type PrivilegedControlRequest struct { + // mu guards the single-use consumed state. + mu sync.Mutex + action string + frame []byte + payloadLength uint64 + payload *privilegedControlPayloadOwner + consumed bool +} + +// NewPrivilegedControlRequest validates one already encoded request. Payload +// ownership transfers only when construction succeeds; rejected inputs retain it. +// A supplied payload's Close method must unblock a concurrent Read because the +// transport closes it after a final peer response or cancellation. +// Capability clients remain responsible for producing the typed wire frame. +func NewPrivilegedControlRequest( + action string, + frame []byte, + payloadLength uint64, + payload io.ReadCloser, +) (*PrivilegedControlRequest, error) { + if !validPrivilegedControlAction(action) { + return nil, errs.New(errs.CodeValidationFailed, "privileged action must be a 1-64 byte ASCII identifier") + } + if len(frame) == 0 || len(frame) > privilegedControlMaxRequestFrameBytes { + return nil, errs.New(errs.CodeValidationFailed, "privileged request frame has an invalid length") + } + if payloadLength > privilegedControlMaxPayloadBytes { + return nil, errs.New(errs.CodeValidationFailed, "privileged request payload exceeds transport limit") + } + if payloadLength > 0 && payload == nil { + return nil, errs.New(errs.CodeValidationFailed, "privileged request payload is missing") + } + + var owner *privilegedControlPayloadOwner + if payload != nil { + owner = &privilegedControlPayloadOwner{payload: payload} + } + return &PrivilegedControlRequest{ + action: action, + frame: append([]byte(nil), frame...), + payloadLength: payloadLength, + payload: owner, + }, nil +} + +func (request *PrivilegedControlRequest) consume() ( + string, + []byte, + uint64, + *privilegedControlPayloadOwner, + error, +) { + if request == nil { + return "", nil, 0, nil, errs.New(errs.CodeValidationFailed, "privileged control request is required") + } + request.mu.Lock() + defer request.mu.Unlock() + if request.consumed { + return "", nil, 0, nil, errs.New(errs.CodeValidationFailed, "privileged control request was already used") + } + request.consumed = true + return request.action, request.frame, request.payloadLength, request.payload, nil +} + +func validPrivilegedControlAction(action string) bool { + if len(action) == 0 || len(action) > privilegedControlMaxActionBytes { + return false + } + for index := range len(action) { + value := action[index] + if value >= 'a' && value <= 'z' || value >= 'A' && value <= 'Z' || value >= '0' && value <= '9' { + continue + } + if value != '.' && value != '_' && value != '-' { + return false + } + } + return true +} + +type privilegedControlPayloadOwner struct { + once sync.Once + payload io.ReadCloser + err error +} + +func (owner *privilegedControlPayloadOwner) Read(target []byte) (int, error) { + if owner == nil || owner.payload == nil { + return 0, io.EOF + } + return owner.payload.Read(target) +} + +func (owner *privilegedControlPayloadOwner) Close() error { + if owner == nil { + return nil + } + owner.once.Do(func() { + owner.err = owner.payload.Close() + }) + return owner.err +} + +// PrivilegedControlTransport executes the one fixed privileged control command. +// Its zero value is ready for use. +type PrivilegedControlTransport struct{} + +// Exchange transfers one request and returns transport evidence. A response is +// authoritative only after a capability client decodes it and ResponseEOF is true. +// CRITICAL: The executable, marker, environment, working directory, and argv shape +// are fixed here; callers control only the bounded action and encoded typed frame. +func (PrivilegedControlTransport) Exchange( + ctx context.Context, + request *PrivilegedControlRequest, +) PrivilegedControlOutcome { + return exchangePrivilegedControl(ctx, request, realPrivilegedControlDeps(), productionPrivilegedControlPolicy()) +} + +// PrivilegedControlOutcome separates raw response evidence from transport diagnostics. +type PrivilegedControlOutcome struct { + response []byte + responseEOF bool + processStarted bool + diagnostic *errs.DomainError + diagnosticStdout []byte + diagnosticStderr []byte + stdoutTruncated bool + stderrTruncated bool +} + +// Response returns a defensive copy of the bytes read from fd 0. +func (outcome PrivilegedControlOutcome) Response() []byte { + return append([]byte(nil), outcome.response...) +} + +// ResponseEOF reports whether the peer produced actual EOF before local interruption. +func (outcome PrivilegedControlOutcome) ResponseEOF() bool { return outcome.responseEOF } + +// ProcessStarted reports whether sudo was started successfully. +func (outcome PrivilegedControlOutcome) ProcessStarted() bool { return outcome.processStarted } + +// Diagnostic reports transport/process failure only; it is never response authority. +func (outcome PrivilegedControlOutcome) Diagnostic() *errs.DomainError { return outcome.diagnostic } + +// DiagnosticStdout returns bounded stdout bytes which are never response authority. +func (outcome PrivilegedControlOutcome) DiagnosticStdout() []byte { + return append([]byte(nil), outcome.diagnosticStdout...) +} + +// DiagnosticStderr returns bounded stderr bytes which are never response authority. +func (outcome PrivilegedControlOutcome) DiagnosticStderr() []byte { + return append([]byte(nil), outcome.diagnosticStderr...) +} + +// DiagnosticStdoutTruncated reports whether stdout exceeded its diagnostic bound. +func (outcome PrivilegedControlOutcome) DiagnosticStdoutTruncated() bool { + return outcome.stdoutTruncated +} + +// DiagnosticStderrTruncated reports whether stderr exceeded its diagnostic bound. +func (outcome PrivilegedControlOutcome) DiagnosticStderrTruncated() bool { + return outcome.stderrTruncated +} + +type privilegedControlPolicy struct { + maxResponseBytes int + maxDiagnosticBytes int + postResponseReapTimeout time.Duration + postInterruptReapTimeout time.Duration + postExitDrainTimeout time.Duration +} + +func productionPrivilegedControlPolicy() privilegedControlPolicy { + return privilegedControlPolicy{ + maxResponseBytes: privilegedControlMaxResponseBytes, + maxDiagnosticBytes: privilegedControlMaxDiagnosticBytes, + postResponseReapTimeout: privilegedControlDefaultReapTimeout, + postInterruptReapTimeout: privilegedControlDefaultReapTimeout, + postExitDrainTimeout: privilegedControlDefaultDrainTimeout, + } +} + +type privilegedControlDeps struct { + socketPair func(ctx context.Context) (*os.File, *os.File, error) + command func(path string, args ...string) *exec.Cmd + start func(ctx context.Context, command *exec.Cmd) error + wait func(ctx context.Context, command *exec.Cmd) error + shutdown func(ctx context.Context, file *os.File, how int) error + killGroup func(ctx context.Context, pid int) error + closeFile func(ctx context.Context, file *os.File) error +} + +func realPrivilegedControlDeps() privilegedControlDeps { + return privilegedControlDeps{ + socketPair: openPrivilegedControlSocketPair, + command: exec.Command, + start: func(_ context.Context, command *exec.Cmd) error { + return command.Start() + }, + wait: func(_ context.Context, command *exec.Cmd) error { + return command.Wait() + }, + shutdown: func(_ context.Context, file *os.File, how int) error { + return unix.Shutdown(int(file.Fd()), how) + }, + killGroup: func(_ context.Context, pid int) error { + return unix.Kill(-pid, unix.SIGKILL) + }, + closeFile: func(_ context.Context, file *os.File) error { + return file.Close() + }, + } +} + +func openPrivilegedControlSocketPair(ctx context.Context) (*os.File, *os.File, error) { + if err := ctx.Err(); err != nil { + return nil, nil, err + } + fds, err := unix.Socketpair(unix.AF_UNIX, unix.SOCK_STREAM|unix.SOCK_CLOEXEC, 0) + if err != nil { + return nil, nil, err + } + return os.NewFile(uintptr(fds[0]), "mvm-privileged-control-parent"), + os.NewFile(uintptr(fds[1]), "mvm-privileged-control-child"), nil +} + +type privilegedControlUploadResult struct{ err error } + +type privilegedControlResponseResult struct { + response []byte + eof bool + err error +} + +func exchangePrivilegedControl( + ctx context.Context, + request *PrivilegedControlRequest, + deps privilegedControlDeps, + policy privilegedControlPolicy, +) PrivilegedControlOutcome { + if ctx == nil { + return PrivilegedControlOutcome{diagnostic: errs.New( + errs.CodeValidationFailed, + "privileged control context is required", + errs.WithDetails(map[string]any{ + "process_started": false, + "outcome_unknown": false, + }), + )} + } + action, frame, payloadLength, payload, err := request.consume() + if err != nil { + return privilegedControlFailure(false, false, err) + } + cleanupCtx := context.WithoutCancel(ctx) + if err := ctx.Err(); err != nil { + closeErr := payload.Close() + return privilegedControlFailure(false, false, errors.Join(err, closeErr)) + } + + parent, child, err := deps.socketPair(ctx) + if err != nil || parent == nil || child == nil { + if err == nil { + err = errors.New("privileged control socket pair returned a nil endpoint") + } + closeErr := payload.Close() + return privilegedControlFailure(false, false, errors.Join( + err, + closeErr, + closePrivilegedControlFile(cleanupCtx, deps, child), + closePrivilegedControlFile(cleanupCtx, deps, parent), + )) + } + closeBeforeStart := func(primary error) PrivilegedControlOutcome { + return privilegedControlFailure(false, false, errors.Join( + primary, + payload.Close(), + deps.closeFile(cleanupCtx, child), + deps.closeFile(cleanupCtx, parent), + )) + } + + command := deps.command( + privilegedControlSudoPath, + "-n", + "--", + infra.SystemBinaryPath, + infra.PrivilegedProtocolMarker, + action, + ) + if command == nil { + return closeBeforeStart(errors.New("privileged control command construction returned nil")) + } + stdout := newPrivilegedControlBoundedWriter(policy.maxDiagnosticBytes) + stderr := newPrivilegedControlBoundedWriter(policy.maxDiagnosticBytes) + command.Stdin = child + command.Stdout = stdout + command.Stderr = stderr + command.Dir = "/" + command.Env = privilegedControlEnvironment() + command.SysProcAttr = &syscall.SysProcAttr{Setpgid: true} + if err := ctx.Err(); err != nil { + return closeBeforeStart(err) + } + if err := deps.start(ctx, command); err != nil { + return closeBeforeStart(err) + } + + outcome := PrivilegedControlOutcome{processStarted: true} + var transportErrors []error + if err := deps.closeFile(cleanupCtx, child); err != nil { + transportErrors = append(transportErrors, fmt.Errorf("close child control socket: %w", err)) + } + + uploads := make(chan privilegedControlUploadResult, 1) + responses := make(chan privilegedControlResponseResult, 1) + waits := make(chan error, 1) + go func() { + uploadErr := uploadPrivilegedControl(ctx, parent, frame, payloadLength, payload) + shutdownErr := deps.shutdown(cleanupCtx, parent, unix.SHUT_WR) + if shutdownErr != nil && !errors.Is(shutdownErr, unix.ENOTCONN) { + uploadErr = errors.Join(uploadErr, fmt.Errorf("half-close privileged request: %w", shutdownErr)) + } + uploads <- privilegedControlUploadResult{err: uploadErr} + }() + go func() { + responses <- readPrivilegedControlResponse(ctx, parent, policy.maxResponseBytes) + }() + go func() { + waits <- deps.wait(cleanupCtx, command) + }() + + var uploadDone, responseDone, waitDone bool + var locallyInterrupted bool + ctxDone := ctx.Done() + var reapTimer, interruptReapTimer, drainTimer *time.Timer + var reapTimerC, interruptReapTimerC, drainTimerC <-chan time.Time + + interrupt := func(kill bool) { + if locallyInterrupted { + return + } + locallyInterrupted = true + // Close is idempotent; final cleanup retrieves and records its stored error. + _ = payload.Close() + if shutdownErr := deps.shutdown(cleanupCtx, parent, unix.SHUT_RDWR); shutdownErr != nil && + !errors.Is(shutdownErr, unix.ENOTCONN) { + transportErrors = append( + transportErrors, + fmt.Errorf("interrupt privileged control socket: %w", shutdownErr), + ) + } + if kill && command.Process != nil && command.Process.Pid > 1 { + if killErr := deps.killGroup(cleanupCtx, command.Process.Pid); killErr != nil && + !errors.Is(killErr, unix.ESRCH) { + transportErrors = append(transportErrors, fmt.Errorf("kill privileged process group: %w", killErr)) + } + } + if !waitDone { + // The caller may lose permission to signal sudo after it becomes root. + // Bound reaping even when socket interruption and process-group kill fail. + interruptReapTimer = time.NewTimer(policy.postInterruptReapTimeout) + interruptReapTimerC = interruptReapTimer.C + } + } + + for !uploadDone || !responseDone || !waitDone { + select { + case upload := <-uploads: + uploadDone = true + if upload.err != nil { + transportErrors = append(transportErrors, upload.err) + } + case response := <-responses: + responseDone = true + if drainTimer != nil { + drainTimer.Stop() + drainTimerC = nil + } + outcome.response = response.response + outcome.responseEOF = response.eof && !locallyInterrupted + if response.err != nil { + transportErrors = append(transportErrors, response.err) + } + if response.eof { + // Stop both possible upload blockers after an early final response: + // payload Close releases a blocked Read and SHUT_WR releases a blocked + // socket Write. Neither changes the peer-EOF evidence. + _ = payload.Close() + if shutdownErr := deps.shutdown(cleanupCtx, parent, unix.SHUT_WR); shutdownErr != nil && + !errors.Is(shutdownErr, unix.ENOTCONN) { + transportErrors = append( + transportErrors, + fmt.Errorf("stop privileged upload after response EOF: %w", shutdownErr), + ) + } + } + if !response.eof { + interrupt(true) + } + if outcome.responseEOF && !waitDone && reapTimer == nil { + reapTimer = time.NewTimer(policy.postResponseReapTimeout) + reapTimerC = reapTimer.C + } + case waitErr := <-waits: + waitDone = true + if reapTimer != nil { + reapTimer.Stop() + reapTimerC = nil + } + if interruptReapTimer != nil { + interruptReapTimer.Stop() + interruptReapTimerC = nil + } + if waitErr != nil { + transportErrors = append(transportErrors, fmt.Errorf("wait for privileged process: %w", waitErr)) + } + if !responseDone && drainTimer == nil { + drainTimer = time.NewTimer(policy.postExitDrainTimeout) + drainTimerC = drainTimer.C + } + case <-ctxDone: + ctxDone = nil + // Prefer peer EOF already queued at the cancellation boundary. + if !responseDone { + if response, ok := takeQueuedPrivilegedControlResponse(responses); ok { + responseDone = true + if drainTimer != nil { + drainTimer.Stop() + drainTimerC = nil + } + outcome.response = response.response + outcome.responseEOF = response.eof + if response.err != nil { + transportErrors = append(transportErrors, response.err) + } + } + } + transportErrors = append(transportErrors, ctx.Err()) + interrupt(true) + case <-reapTimerC: + reapTimerC = nil + transportErrors = append(transportErrors, errors.New("privileged process did not exit after response EOF")) + interrupt(true) + case <-interruptReapTimerC: + interruptReapTimerC = nil + waitDone = true + transportErrors = append( + transportErrors, + errors.New("privileged process did not exit after interruption"), + ) + case <-drainTimerC: + drainTimerC = nil + transportErrors = append( + transportErrors, + errors.New("privileged response did not reach EOF after process exit"), + ) + interrupt(true) + } + } + + if reapTimer != nil { + reapTimer.Stop() + } + if interruptReapTimer != nil { + interruptReapTimer.Stop() + } + if drainTimer != nil { + drainTimer.Stop() + } + if closeErr := payload.Close(); closeErr != nil { + transportErrors = append(transportErrors, fmt.Errorf("close privileged payload: %w", closeErr)) + } + if closeErr := deps.closeFile(cleanupCtx, parent); closeErr != nil { + transportErrors = append(transportErrors, fmt.Errorf("close parent control socket: %w", closeErr)) + } + outcome.diagnosticStdout, outcome.stdoutTruncated = stdout.snapshot() + outcome.diagnosticStderr, outcome.stderrTruncated = stderr.snapshot() + if len(transportErrors) > 0 { + outcome.diagnostic = newPrivilegedControlDiagnostic(true, true, errors.Join(transportErrors...)) + } + return outcome +} + +func takeQueuedPrivilegedControlResponse( + responses <-chan privilegedControlResponseResult, +) (privilegedControlResponseResult, bool) { + select { + case response := <-responses: + return response, true + default: + return privilegedControlResponseResult{}, false + } +} + +func closePrivilegedControlFile( + ctx context.Context, + deps privilegedControlDeps, + file *os.File, +) error { + if file == nil { + return nil + } + return deps.closeFile(ctx, file) +} + +func uploadPrivilegedControl( + ctx context.Context, + control io.Writer, + frame []byte, + payloadLength uint64, + payload *privilegedControlPayloadOwner, +) error { + // Final transport cleanup retrieves the idempotent owner's stored close error. + defer func() { _ = payload.Close() }() + if err := writePrivilegedControlExact(ctx, control, frame); err != nil { + return fmt.Errorf("write privileged request frame: %w", err) + } + if payloadLength == 0 { + return nil + } + written, err := io.CopyN( + privilegedControlContextWriter{ctx: ctx, writer: control}, + privilegedControlContextReader{ctx: ctx, reader: payload}, + int64(payloadLength), + ) + if err != nil { + return fmt.Errorf("copy privileged request payload: %w", err) + } + if uint64(written) != payloadLength { + return fmt.Errorf("copy privileged request payload: %w", io.ErrShortWrite) + } + return nil +} + +type privilegedControlContextReader struct { + ctx context.Context + reader io.Reader +} + +func (reader privilegedControlContextReader) Read(target []byte) (int, error) { + if err := reader.ctx.Err(); err != nil { + return 0, err + } + return reader.reader.Read(target) +} + +type privilegedControlContextWriter struct { + ctx context.Context + writer io.Writer +} + +func (writer privilegedControlContextWriter) Write(value []byte) (int, error) { + if err := writer.ctx.Err(); err != nil { + return 0, err + } + return writer.writer.Write(value) +} + +func writePrivilegedControlExact(ctx context.Context, writer io.Writer, value []byte) error { + for len(value) > 0 { + if err := ctx.Err(); err != nil { + return err + } + written, err := writer.Write(value) + if written > 0 { + value = value[written:] + } + if err != nil { + return err + } + if written == 0 { + return io.ErrShortWrite + } + } + return nil +} + +func readPrivilegedControlResponse( + ctx context.Context, + control io.Reader, + maximum int, +) privilegedControlResponseResult { + if err := ctx.Err(); err != nil { + return privilegedControlResponseResult{err: err} + } + response, err := io.ReadAll(io.LimitReader(control, int64(maximum)+1)) + if err != nil { + return privilegedControlResponseResult{response: response, err: err} + } + if len(response) > maximum { + return privilegedControlResponseResult{ + response: response, + err: errors.New("privileged response exceeds transport limit"), + } + } + return privilegedControlResponseResult{response: response, eof: true} +} + +type privilegedControlBoundedWriter struct { + // mu guards value and truncated while os/exec drains output concurrently. + mu sync.Mutex + maximum int + value []byte + truncated bool +} + +func newPrivilegedControlBoundedWriter(maximum int) *privilegedControlBoundedWriter { + return &privilegedControlBoundedWriter{maximum: maximum} +} + +func (writer *privilegedControlBoundedWriter) Write(value []byte) (int, error) { + writer.mu.Lock() + defer writer.mu.Unlock() + remaining := writer.maximum - len(writer.value) + if remaining > len(value) { + remaining = len(value) + } + if remaining > 0 { + writer.value = append(writer.value, value[:remaining]...) + } + if remaining < len(value) { + writer.truncated = true + } + return len(value), nil +} + +func (writer *privilegedControlBoundedWriter) snapshot() ([]byte, bool) { + writer.mu.Lock() + defer writer.mu.Unlock() + return append([]byte(nil), writer.value...), writer.truncated +} + +func privilegedControlFailure(started, outcomeUnknown bool, err error) PrivilegedControlOutcome { + return PrivilegedControlOutcome{ + processStarted: started, + diagnostic: newPrivilegedControlDiagnostic(started, outcomeUnknown, err), + } +} + +func newPrivilegedControlDiagnostic(started, outcomeUnknown bool, err error) *errs.DomainError { + if err == nil { + return nil + } + return errs.WrapMsg( + errs.CodeProcessError, + "privileged control transport failed", + err, + errs.WithClass(errs.ClassInternal), + errs.WithDetails(map[string]any{ + "process_started": started, + "outcome_unknown": outcomeUnknown, + }), + ) +} diff --git a/internal/lib/system/privileged_control_test.go b/internal/lib/system/privileged_control_test.go new file mode 100644 index 00000000..e9834751 --- /dev/null +++ b/internal/lib/system/privileged_control_test.go @@ -0,0 +1,949 @@ +package system + +import ( + "bytes" + "context" + "errors" + "io" + "os" + "os/exec" + "slices" + "strconv" + "strings" + "sync" + "sync/atomic" + "testing" + "time" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + "golang.org/x/sys/unix" + + "mvmctl/internal/infra" + "mvmctl/pkg/errs" +) + +const privilegedControlHelperTest = "^TestPrivilegedControlHelper$" + +func TestPrivilegedControlTransportUsesFixedCommandAndFDZeroDuplex(t *testing.T) { + requestFrame := []byte("request-frame") + request, err := NewPrivilegedControlRequest("test-duplex", requestFrame, 0, nil) + require.NoError(t, err) + requestFrame[0] = 'X' + + deps := realPrivilegedControlDeps() + var capturedCommand *exec.Cmd + var waitCount, closeCount atomic.Int32 + realWait := deps.wait + realClose := deps.closeFile + deps.command = func(path string, args ...string) *exec.Cmd { + assert.Equal(t, "/usr/bin/sudo", path) + assert.Equal(t, []string{ + "-n", + "--", + infra.SystemBinaryPath, + "__mvm_privileged_v1", + "test-duplex", + }, args) + capturedCommand = privilegedControlHelperCommand("test-duplex") + return capturedCommand + } + deps.wait = func(ctx context.Context, command *exec.Cmd) error { + waitCount.Add(1) + return realWait(ctx, command) + } + deps.closeFile = func(ctx context.Context, file *os.File) error { + closeCount.Add(1) + return realClose(ctx, file) + } + policy := productionPrivilegedControlPolicy() + policy.postResponseReapTimeout = 5 * time.Second + policy.postExitDrainTimeout = 5 * time.Second + + ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second) + t.Cleanup(cancel) + outcome := exchangePrivilegedControl(ctx, request, deps, policy) + if outcome.Diagnostic() != nil { + t.Logf("transport diagnostic: %#v / %v", outcome.Diagnostic(), outcome.Diagnostic().Err) + } + + assert.True(t, outcome.ProcessStarted()) + assert.True(t, outcome.ResponseEOF()) + assert.Equal(t, []byte("response:request-frame"), outcome.Response()) + assert.Nil(t, outcome.Diagnostic()) + assert.Contains(t, string(outcome.DiagnosticStdout()), "stdout-poison") + assert.Contains(t, string(outcome.DiagnosticStderr()), "stderr-poison") + assert.NotContains(t, string(outcome.Response()), "poison") + require.NotNil(t, capturedCommand) + assert.Equal(t, "/", capturedCommand.Dir) + assert.Equal(t, privilegedControlEnvironment(), capturedCommand.Env) + assert.Empty(t, capturedCommand.ExtraFiles) + require.NotNil(t, capturedCommand.SysProcAttr) + assert.True(t, capturedCommand.SysProcAttr.Setpgid) + assert.Equal(t, int32(1), waitCount.Load()) + assert.Equal(t, int32(2), closeCount.Load()) +} + +func TestOpenPrivilegedControlSocketPairSetsCloseOnExec(t *testing.T) { + parent, child, err := openPrivilegedControlSocketPair(context.Background()) + require.NoError(t, err) + t.Cleanup(func() { require.NoError(t, parent.Close()) }) + t.Cleanup(func() { require.NoError(t, child.Close()) }) + + for name, endpoint := range map[string]*os.File{"parent": parent, "child": child} { + t.Run(name, func(t *testing.T) { + flags, err := unix.FcntlInt(endpoint.Fd(), unix.F_GETFD, 0) + require.NoError(t, err) + assert.NotZero(t, flags&unix.FD_CLOEXEC) + }) + } +} + +func TestPrivilegedControlTransportRejectsNilCommandBeforeStart(t *testing.T) { + request, err := NewPrivilegedControlRequest("test-nil-command", []byte("request"), 0, nil) + require.NoError(t, err) + + deps := realPrivilegedControlDeps() + deps.command = func(_ string, _ ...string) *exec.Cmd { return nil } + + assert.NotPanics(t, func() { + outcome := exchangePrivilegedControl( + context.Background(), + request, + deps, + productionPrivilegedControlPolicy(), + ) + assert.False(t, outcome.ProcessStarted()) + assert.False(t, outcome.ResponseEOF()) + assert.NotNil(t, outcome.Diagnostic()) + assert.Equal(t, false, outcome.Diagnostic().Details["outcome_unknown"]) + }) +} + +func TestPrivilegedControlTransportRejectsNilSocketEndpointBeforeCommand(t *testing.T) { + payload := newPrivilegedControlTrackingPayload(nil) + request, err := NewPrivilegedControlRequest("test-nil-socket", []byte("request"), 0, payload) + require.NoError(t, err) + deps := realPrivilegedControlDeps() + parent, peer, err := os.Pipe() + require.NoError(t, err) + t.Cleanup(func() { _ = peer.Close() }) + deps.socketPair = func(_ context.Context) (*os.File, *os.File, error) { return parent, nil, nil } + var commandCount atomic.Int32 + var closeCount atomic.Int32 + deps.command = func(_ string, _ ...string) *exec.Cmd { + commandCount.Add(1) + return nil + } + realClose := deps.closeFile + deps.closeFile = func(ctx context.Context, file *os.File) error { + closeCount.Add(1) + return realClose(ctx, file) + } + + outcome := exchangePrivilegedControl( + context.Background(), + request, + deps, + productionPrivilegedControlPolicy(), + ) + + assert.False(t, outcome.ProcessStarted()) + assert.NotNil(t, outcome.Diagnostic()) + assert.Equal(t, int32(0), commandCount.Load()) + assert.Equal(t, int32(1), payload.CloseCount()) + assert.Equal(t, int32(1), closeCount.Load()) +} + +func TestPrivilegedControlTransportSocketPairFailureClosesReturnedEndpoints(t *testing.T) { + payload := newPrivilegedControlTrackingPayload(nil) + request, err := NewPrivilegedControlRequest("test-socket-error", []byte("request"), 0, payload) + require.NoError(t, err) + parent, child, err := os.Pipe() + require.NoError(t, err) + socketErr := errors.New("socket pair failed") + deps := realPrivilegedControlDeps() + deps.socketPair = func(_ context.Context) (*os.File, *os.File, error) { + return parent, child, socketErr + } + var closeCount atomic.Int32 + realClose := deps.closeFile + deps.closeFile = func(ctx context.Context, file *os.File) error { + closeCount.Add(1) + return realClose(ctx, file) + } + + outcome := exchangePrivilegedControl( + context.Background(), + request, + deps, + productionPrivilegedControlPolicy(), + ) + + assert.False(t, outcome.ProcessStarted()) + require.NotNil(t, outcome.Diagnostic()) + assert.ErrorIs(t, outcome.Diagnostic(), socketErr) + assert.Equal(t, int32(1), payload.CloseCount()) + assert.Equal(t, int32(2), closeCount.Load()) +} + +func TestNewPrivilegedControlRequestRejectsUnboundedOrInvalidInput(t *testing.T) { + tests := map[string]struct { + action string + frame []byte + payloadLength uint64 + payload io.ReadCloser + }{ + "empty_action": {action: "", frame: []byte("frame")}, + "path_action": {action: "../launch", frame: []byte("frame")}, + "oversized_action": {action: strings.Repeat("a", 65), frame: []byte("frame")}, + "empty_frame": {action: "launch"}, + "oversized_frame": {action: "launch", frame: make([]byte, privilegedControlMaxRequestFrameBytes+1)}, + "oversized_payload": {action: "launch", frame: []byte("frame"), payloadLength: 128*1024*1024 + 1}, + "missing_payload": {action: "launch", frame: []byte("frame"), payloadLength: 1}, + } + + for name, test := range tests { + t.Run(name, func(t *testing.T) { + request, err := NewPrivilegedControlRequest(test.action, test.frame, test.payloadLength, test.payload) + assert.Nil(t, request) + assert.Error(t, err) + }) + } +} + +func TestRejectedPrivilegedControlRequestRetainsCallerPayloadOwnership(t *testing.T) { + payload := newPrivilegedControlTrackingPayload(nil) + + request, err := NewPrivilegedControlRequest("../invalid", []byte("frame"), 0, payload) + + assert.Nil(t, request) + assert.Error(t, err) + assert.Zero(t, payload.CloseCount()) +} + +func TestWritePrivilegedControlExactHandlesPartialAndZeroWrites(t *testing.T) { + t.Run("partial", func(t *testing.T) { + writer := &privilegedControlPartialWriter{maximum: 1} + + err := writePrivilegedControlExact(context.Background(), writer, []byte("frame")) + + assert.NoError(t, err) + assert.Equal(t, []byte("frame"), writer.value.Bytes()) + }) + + t.Run("zero", func(t *testing.T) { + err := writePrivilegedControlExact(context.Background(), privilegedControlZeroWriter{}, []byte("frame")) + + assert.ErrorIs(t, err, io.ErrShortWrite) + }) +} + +func TestPrivilegedControlTransportUploadsExactPayloadAndClosesItOnce(t *testing.T) { + payload := newPrivilegedControlTrackingPayload([]byte("payload-data")) + request, err := NewPrivilegedControlRequest("test-payload", []byte("header:"), 12, payload) + require.NoError(t, err) + + outcome := runPrivilegedControlHelper(t, request, "test-payload", nil) + + assert.True(t, outcome.ProcessStarted()) + assert.True(t, outcome.ResponseEOF()) + assert.Equal(t, []byte("response:header:payload-data"), outcome.Response()) + assert.Nil(t, outcome.Diagnostic()) + assert.Equal(t, int32(1), payload.CloseCount()) +} + +func TestPrivilegedControlTransportPreservesEarlyEOFResponseAndProcessDiagnostics(t *testing.T) { + payload := newPrivilegedControlTrackingPayload(repeatedPrivilegedControlBytes("p", 1024*1024)) + request, err := NewPrivilegedControlRequest("test-early-response", []byte("header:"), 1024*1024, payload) + require.NoError(t, err) + + outcome := runPrivilegedControlHelper(t, request, "test-early-response", nil) + + assert.True(t, outcome.ProcessStarted()) + assert.True(t, outcome.ResponseEOF()) + assert.Equal(t, []byte("response:early"), outcome.Response()) + assert.NotNil(t, outcome.Diagnostic()) + assert.Equal(t, true, outcome.Diagnostic().Details["outcome_unknown"]) + assert.Equal(t, int32(1), payload.CloseCount()) +} + +func TestPrivilegedControlTransportFinalEOFClosesBlockedPayloadRead(t *testing.T) { + payload := newPrivilegedControlBlockingPayload() + request, err := NewPrivilegedControlRequest("test-final-without-read", []byte("request"), 64, payload) + require.NoError(t, err) + ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second) + t.Cleanup(cancel) + + outcome := runPrivilegedControlHelperWithContext(t, ctx, request, "test-final-without-read", nil) + + assert.True(t, outcome.ProcessStarted()) + assert.True(t, outcome.ResponseEOF()) + assert.Equal(t, []byte("response:final"), outcome.Response()) + require.NotNil(t, outcome.Diagnostic()) + assert.NotErrorIs(t, outcome.Diagnostic(), context.DeadlineExceeded) + assert.ErrorIs(t, outcome.Diagnostic(), io.ErrClosedPipe) + assert.Equal(t, int32(1), payload.CloseCount()) +} + +func TestPrivilegedControlTransportFinalEOFStopsBlockedSocketUpload(t *testing.T) { + payload := io.NopCloser(io.LimitReader( + privilegedControlZeroReader{}, + int64(privilegedControlMaxPayloadBytes), + )) + request, err := NewPrivilegedControlRequest( + "test-final-stops-upload", + []byte("request"), + privilegedControlMaxPayloadBytes, + payload, + ) + require.NoError(t, err) + + outcome := runPrivilegedControlHelper(t, request, "test-final-stops-upload", nil) + + assert.True(t, outcome.ProcessStarted()) + assert.True(t, outcome.ResponseEOF()) + assert.Equal(t, []byte("response:final"), outcome.Response()) + assert.NotContains(t, string(outcome.DiagnosticStderr()), "missing local write shutdown") +} + +func TestPrivilegedControlTransportNeverFabricatesResponseEOFOnCancellation(t *testing.T) { + request, err := NewPrivilegedControlRequest("test-non-eof", []byte("request"), 0, nil) + require.NoError(t, err) + ctx, cancel := context.WithTimeout(context.Background(), 50*time.Millisecond) + t.Cleanup(cancel) + + outcome := runPrivilegedControlHelperWithContext(t, ctx, request, "test-non-eof", nil) + + assert.True(t, outcome.ProcessStarted()) + assert.False(t, outcome.ResponseEOF()) + assert.Equal(t, []byte("response:partial"), outcome.Response()) + assert.NotNil(t, outcome.Diagnostic()) + assert.Equal(t, true, outcome.Diagnostic().Details["outcome_unknown"]) +} + +func TestPrivilegedControlTransportBoundsResponseAndDiagnostics(t *testing.T) { + t.Run("response", func(t *testing.T) { + request, err := NewPrivilegedControlRequest("test-overflow", []byte("request"), 0, nil) + require.NoError(t, err) + policy := productionPrivilegedControlPolicy() + policy.maxResponseBytes = 32 + + outcome := runPrivilegedControlHelper(t, request, "test-overflow", &policy) + + assert.False(t, outcome.ResponseEOF()) + assert.Len(t, outcome.Response(), 33) + assert.NotNil(t, outcome.Diagnostic()) + }) + + t.Run("stdout_stderr", func(t *testing.T) { + request, err := NewPrivilegedControlRequest("test-diagnostic-bound", []byte("request"), 0, nil) + require.NoError(t, err) + policy := productionPrivilegedControlPolicy() + policy.maxDiagnosticBytes = 16 + + outcome := runPrivilegedControlHelper(t, request, "test-diagnostic-bound", &policy) + + assert.True(t, outcome.ResponseEOF()) + assert.Equal(t, []byte("response:bounded"), outcome.Response()) + assert.Len(t, outcome.DiagnosticStdout(), 16) + assert.Len(t, outcome.DiagnosticStderr(), 16) + assert.True(t, outcome.DiagnosticStdoutTruncated()) + assert.True(t, outcome.DiagnosticStderrTruncated()) + }) + + t.Run("exact_response_with_eof", func(t *testing.T) { + request, err := NewPrivilegedControlRequest("test-exact-response-eof", []byte("request"), 0, nil) + require.NoError(t, err) + policy := productionPrivilegedControlPolicy() + policy.maxResponseBytes = 32 + + outcome := runPrivilegedControlHelper(t, request, "test-exact-response-eof", &policy) + + assert.True(t, outcome.ResponseEOF()) + assert.Len(t, outcome.Response(), 32) + assert.Nil(t, outcome.Diagnostic()) + }) + + t.Run("exact_response_without_eof", func(t *testing.T) { + request, err := NewPrivilegedControlRequest("test-exact-response-open", []byte("request"), 0, nil) + require.NoError(t, err) + policy := productionPrivilegedControlPolicy() + policy.maxResponseBytes = 32 + ctx, cancel := context.WithTimeout(context.Background(), 50*time.Millisecond) + t.Cleanup(cancel) + + outcome := runPrivilegedControlHelperWithContext( + t, + ctx, + request, + "test-exact-response-open", + &policy, + ) + + assert.False(t, outcome.ResponseEOF()) + assert.Len(t, outcome.Response(), 32) + assert.NotNil(t, outcome.Diagnostic()) + }) +} + +func TestPrivilegedControlTransportHalfClosesAfterShortDeclaredPayload(t *testing.T) { + payload := newPrivilegedControlTrackingPayload([]byte("tiny")) + request, err := NewPrivilegedControlRequest("test-short-payload", []byte("header:"), 16, payload) + require.NoError(t, err) + + outcome := runPrivilegedControlHelper(t, request, "test-short-payload", nil) + + assert.True(t, outcome.ResponseEOF()) + assert.Equal(t, []byte("observed:header:tiny"), outcome.Response()) + require.NotNil(t, outcome.Diagnostic()) + assert.ErrorIs(t, outcome.Diagnostic(), io.EOF) + assert.Equal(t, int32(1), payload.CloseCount()) +} + +func TestPrivilegedControlTransportStartFailureClosesOwnedResourcesWithoutWait(t *testing.T) { + payload := newPrivilegedControlTrackingPayload(nil) + request, err := NewPrivilegedControlRequest("test-start-failure", []byte("request"), 0, payload) + require.NoError(t, err) + + deps := realPrivilegedControlDeps() + deps.command = func(_ string, _ ...string) *exec.Cmd { + return privilegedControlHelperCommand("test-start-failure") + } + startErr := errors.New("start failed") + deps.start = func(_ context.Context, _ *exec.Cmd) error { return startErr } + var waitCount atomic.Int32 + deps.wait = func(_ context.Context, _ *exec.Cmd) error { + waitCount.Add(1) + return nil + } + var closeCount atomic.Int32 + realClose := deps.closeFile + deps.closeFile = func(ctx context.Context, file *os.File) error { + closeCount.Add(1) + return realClose(ctx, file) + } + + outcome := exchangePrivilegedControl( + context.Background(), + request, + deps, + productionPrivilegedControlPolicy(), + ) + + assert.False(t, outcome.ProcessStarted()) + assert.False(t, outcome.ResponseEOF()) + require.NotNil(t, outcome.Diagnostic()) + assert.ErrorIs(t, outcome.Diagnostic(), startErr) + assert.Equal(t, false, outcome.Diagnostic().Details["outcome_unknown"]) + assert.Equal(t, false, outcome.Diagnostic().Details["process_started"]) + assert.Equal(t, int32(1), payload.CloseCount()) + assert.Equal(t, int32(0), waitCount.Load()) + assert.Equal(t, int32(2), closeCount.Load()) +} + +func TestPrivilegedControlTransportCancellationClosesPayloadKillsAndWaitsOnce(t *testing.T) { + payload := newPrivilegedControlBlockingPayload() + request, err := NewPrivilegedControlRequest("test-cancel", []byte("request"), 64, payload) + require.NoError(t, err) + ctx, cancel := context.WithTimeout(context.Background(), 50*time.Millisecond) + t.Cleanup(cancel) + + deps := privilegedControlHelperDeps(t, "test-cancel") + var killCount atomic.Int32 + realKill := deps.killGroup + deps.killGroup = func(ctx context.Context, pid int) error { + killCount.Add(1) + return realKill(ctx, pid) + } + var waitCount atomic.Int32 + realWait := deps.wait + deps.wait = func(ctx context.Context, command *exec.Cmd) error { + waitCount.Add(1) + return realWait(ctx, command) + } + + outcome := exchangePrivilegedControl(ctx, request, deps, productionPrivilegedControlPolicy()) + + assert.True(t, outcome.ProcessStarted()) + assert.False(t, outcome.ResponseEOF()) + require.NotNil(t, outcome.Diagnostic()) + assert.Equal(t, true, outcome.Diagnostic().Details["outcome_unknown"]) + assert.Equal(t, true, outcome.Diagnostic().Details["process_started"]) + assert.Equal(t, int32(1), payload.CloseCount()) + assert.Equal(t, int32(1), killCount.Load()) + assert.Equal(t, int32(1), waitCount.Load()) +} + +func TestPrivilegedControlTransportKillsSameGroupSocketHolderAfterProcessExit(t *testing.T) { + request, err := NewPrivilegedControlRequest("test-socket-holder", []byte("request"), 0, nil) + require.NoError(t, err) + policy := productionPrivilegedControlPolicy() + policy.postExitDrainTimeout = 50 * time.Millisecond + deps := privilegedControlHelperDeps(t, "test-socket-holder") + var killCount atomic.Int32 + realKill := deps.killGroup + deps.killGroup = func(ctx context.Context, pid int) error { + killCount.Add(1) + return realKill(ctx, pid) + } + + outcome := exchangePrivilegedControl(context.Background(), request, deps, policy) + require.True(t, outcome.ProcessStarted()) + require.False(t, outcome.ResponseEOF()) + require.True(t, strings.HasPrefix(string(outcome.Response()), "holder-pid:")) + holderPID, err := strconv.Atoi(strings.TrimPrefix(string(outcome.Response()), "holder-pid:")) + require.NoError(t, err) + t.Cleanup(func() { _ = unix.Kill(holderPID, unix.SIGKILL) }) + + assert.Equal(t, int32(1), killCount.Load()) + assert.Eventually(t, func() bool { + return errors.Is(unix.Kill(holderPID, 0), unix.ESRCH) + }, time.Second, 10*time.Millisecond) +} + +func TestPrivilegedControlTransportPreservesEOFWhenReapDeadlineKillsProcess(t *testing.T) { + request, err := NewPrivilegedControlRequest("test-response-reap", []byte("request"), 0, nil) + require.NoError(t, err) + policy := productionPrivilegedControlPolicy() + policy.postResponseReapTimeout = 50 * time.Millisecond + deps := privilegedControlHelperDeps(t, "test-response-reap") + var killCount, waitCount atomic.Int32 + realKill := deps.killGroup + realWait := deps.wait + killErr := errors.New("injected kill failure") + deps.killGroup = func(ctx context.Context, pid int) error { + killCount.Add(1) + if err := realKill(ctx, pid); err != nil { + return err + } + return killErr + } + deps.wait = func(ctx context.Context, command *exec.Cmd) error { + waitCount.Add(1) + return realWait(ctx, command) + } + + outcome := exchangePrivilegedControl(context.Background(), request, deps, policy) + + assert.True(t, outcome.ProcessStarted()) + assert.True(t, outcome.ResponseEOF()) + assert.Equal(t, []byte("response:complete"), outcome.Response()) + assert.NotNil(t, outcome.Diagnostic()) + assert.ErrorIs(t, outcome.Diagnostic(), killErr) + assert.Equal(t, int32(1), killCount.Load()) + assert.Equal(t, int32(1), waitCount.Load()) +} + +func TestPrivilegedControlTransportReturnsWhenInterruptedProcessCannotBeReaped(t *testing.T) { + // Rationale: The normal user may be unable to signal a sudo child after it + // becomes root. A failed process-group kill must not block the caller forever. + request, err := NewPrivilegedControlRequest("test-unreapable", []byte("request"), 0, nil) + require.NoError(t, err) + policy := productionPrivilegedControlPolicy() + policy.postResponseReapTimeout = 10 * time.Millisecond + policy.postInterruptReapTimeout = 10 * time.Millisecond + + wakeWait := make(chan struct{}) + var wakeWaitOnce sync.Once + releaseWait := func() { wakeWaitOnce.Do(func() { close(wakeWait) }) } + t.Cleanup(releaseWait) + waitStarted := make(chan struct{}) + killErr := errors.New("injected kill permission failure") + deps := realPrivilegedControlDeps() + deps.command = func(_ string, _ ...string) *exec.Cmd { return exec.Command("/bin/true") } + deps.start = func(_ context.Context, command *exec.Cmd) error { + command.Process = &os.Process{Pid: 4242} + return nil + } + deps.wait = func(_ context.Context, _ *exec.Cmd) error { + close(waitStarted) + <-wakeWait + return nil + } + var killCount atomic.Int32 + deps.killGroup = func(_ context.Context, _ int) error { + killCount.Add(1) + return killErr + } + + returned := make(chan PrivilegedControlOutcome, 1) + go func() { + returned <- exchangePrivilegedControl(context.Background(), request, deps, policy) + }() + select { + case outcome := <-returned: + releaseWait() + assert.True(t, outcome.ProcessStarted()) + require.NotNil(t, outcome.Diagnostic()) + assert.ErrorIs(t, outcome.Diagnostic(), killErr) + assert.ErrorContains(t, outcome.Diagnostic().Err, "did not exit after interruption") + assert.Equal(t, true, outcome.Diagnostic().Details["outcome_unknown"]) + assert.Equal(t, int32(1), killCount.Load()) + case <-time.After(250 * time.Millisecond): + releaseWait() + t.Fatal("privileged control transport remained blocked after failed process-group kill") + } + <-waitStarted +} + +func TestQueuedPrivilegedControlResponseWinsAtCancellationBoundary(t *testing.T) { + responses := make(chan privilegedControlResponseResult, 1) + responses <- privilegedControlResponseResult{response: []byte("response"), eof: true} + + response, ok := takeQueuedPrivilegedControlResponse(responses) + + require.True(t, ok) + assert.True(t, response.eof) + assert.Equal(t, []byte("response"), response.response) +} + +func TestPrivilegedControlTransportSurfacesCleanupDependencyErrors(t *testing.T) { + tests := map[string]func( + deps *privilegedControlDeps, + sentinel error, + ){ + "half_close": func(deps *privilegedControlDeps, sentinel error) { + realShutdown := deps.shutdown + deps.shutdown = func(ctx context.Context, file *os.File, how int) error { + err := realShutdown(ctx, file, how) + if how == unix.SHUT_WR && err == nil { + return sentinel + } + return err + } + }, + "child_close": func(deps *privilegedControlDeps, sentinel error) { + realClose := deps.closeFile + deps.closeFile = func(ctx context.Context, file *os.File) error { + err := realClose(ctx, file) + if strings.Contains(file.Name(), "child") && err == nil { + return sentinel + } + return err + } + }, + "parent_close": func(deps *privilegedControlDeps, sentinel error) { + realClose := deps.closeFile + deps.closeFile = func(ctx context.Context, file *os.File) error { + err := realClose(ctx, file) + if strings.Contains(file.Name(), "parent") && err == nil { + return sentinel + } + return err + } + }, + } + + for name, inject := range tests { + t.Run(name, func(t *testing.T) { + request, err := NewPrivilegedControlRequest("test-payload", []byte("header:"), 0, nil) + require.NoError(t, err) + deps := privilegedControlHelperDeps(t, "test-payload") + sentinel := errors.New("injected " + name + " failure") + inject(&deps, sentinel) + + outcome := exchangePrivilegedControl( + context.Background(), + request, + deps, + productionPrivilegedControlPolicy(), + ) + + assert.True(t, outcome.ResponseEOF()) + assert.Equal(t, []byte("response:header:"), outcome.Response()) + require.NotNil(t, outcome.Diagnostic()) + assert.ErrorIs(t, outcome.Diagnostic(), sentinel) + }) + } +} + +func TestPrivilegedControlTransportCancellationBeforeStartSkipsAllProcessEffects(t *testing.T) { + payload := newPrivilegedControlTrackingPayload(nil) + request, err := NewPrivilegedControlRequest("test-prestart-cancel", []byte("request"), 0, payload) + require.NoError(t, err) + ctx, cancel := context.WithCancel(context.Background()) + cancel() + deps := realPrivilegedControlDeps() + var socketCount, commandCount, startCount, waitCount atomic.Int32 + deps.socketPair = func(_ context.Context) (*os.File, *os.File, error) { + socketCount.Add(1) + return nil, nil, errors.New("must not run") + } + deps.command = func(_ string, _ ...string) *exec.Cmd { + commandCount.Add(1) + return nil + } + deps.start = func(_ context.Context, _ *exec.Cmd) error { + startCount.Add(1) + return nil + } + deps.wait = func(_ context.Context, _ *exec.Cmd) error { + waitCount.Add(1) + return nil + } + + outcome := exchangePrivilegedControl(ctx, request, deps, productionPrivilegedControlPolicy()) + + assert.False(t, outcome.ProcessStarted()) + require.NotNil(t, outcome.Diagnostic()) + assert.ErrorIs(t, outcome.Diagnostic(), context.Canceled) + assert.Equal(t, int32(1), payload.CloseCount()) + assert.Zero(t, socketCount.Load()) + assert.Zero(t, commandCount.Load()) + assert.Zero(t, startCount.Load()) + assert.Zero(t, waitCount.Load()) +} + +func TestPrivilegedControlTransportRejectsNilContextWithoutConsumingRequest(t *testing.T) { + request, err := NewPrivilegedControlRequest("test-nil-context", []byte("request"), 0, nil) + require.NoError(t, err) + deps := realPrivilegedControlDeps() + var socketCount atomic.Int32 + deps.socketPair = func(_ context.Context) (*os.File, *os.File, error) { + socketCount.Add(1) + return nil, nil, errors.New("must not run") + } + + outcome := exchangePrivilegedControl(nil, request, deps, productionPrivilegedControlPolicy()) + + assert.False(t, outcome.ProcessStarted()) + require.NotNil(t, outcome.Diagnostic()) + assert.Equal(t, errs.CodeValidationFailed, outcome.Diagnostic().Code) + assert.Zero(t, socketCount.Load()) + _, _, _, _, err = request.consume() + assert.NoError(t, err) +} + +func TestPrivilegedControlRequestIsSingleUse(t *testing.T) { + request, err := NewPrivilegedControlRequest("test-payload", []byte("header:"), 0, nil) + require.NoError(t, err) + first := runPrivilegedControlHelper(t, request, "test-payload", nil) + require.True(t, first.ProcessStarted()) + + second := exchangePrivilegedControl( + context.Background(), + request, + realPrivilegedControlDeps(), + productionPrivilegedControlPolicy(), + ) + + assert.False(t, second.ProcessStarted()) + assert.NotNil(t, second.Diagnostic()) +} + +func TestPrivilegedControlHelper(t *testing.T) { + action, ok := privilegedControlHelperAction(os.Args) + if !ok { + return + } + + // Helper cases ignore I/O errors unless the error itself is the exercised contract. + switch action { + case "test-duplex": + request, err := io.ReadAll(os.Stdin) + if err != nil { + os.Exit(91) + } + _, _ = os.Stdout.WriteString("stdout-poison") + _, _ = os.Stderr.WriteString("stderr-poison") + if _, err := os.Stdin.Write(append([]byte("response:"), request...)); err != nil { + os.Exit(92) + } + if err := unix.Shutdown(int(os.Stdin.Fd()), unix.SHUT_WR); err != nil { + os.Exit(93) + } + case "test-payload": + request, err := io.ReadAll(os.Stdin) + if err != nil { + os.Exit(95) + } + _, _ = os.Stdin.Write(append([]byte("response:"), request...)) + _ = unix.Shutdown(int(os.Stdin.Fd()), unix.SHUT_WR) + case "test-early-response": + header := make([]byte, len("header:")) + if _, err := io.ReadFull(os.Stdin, header); err != nil { + os.Exit(96) + } + _ = unix.Shutdown(int(os.Stdin.Fd()), unix.SHUT_RD) + _, _ = os.Stdin.WriteString("response:early") + _ = unix.Shutdown(int(os.Stdin.Fd()), unix.SHUT_WR) + os.Exit(17) + case "test-final-without-read": + _ = unix.Shutdown(int(os.Stdin.Fd()), unix.SHUT_RD) + _, _ = os.Stdin.WriteString("response:final") + _ = unix.Shutdown(int(os.Stdin.Fd()), unix.SHUT_WR) + case "test-final-stops-upload": + _, _ = os.Stdin.WriteString("response:final") + _ = unix.Shutdown(int(os.Stdin.Fd()), unix.SHUT_WR) + poll := []unix.PollFd{{Fd: int32(os.Stdin.Fd()), Events: unix.POLLRDHUP}} + count, err := unix.Poll(poll, 500) + if err != nil || count == 0 || poll[0].Revents&unix.POLLRDHUP == 0 { + _, _ = os.Stderr.WriteString("missing local write shutdown") + os.Exit(98) + } + case "test-non-eof": + _, _ = io.ReadAll(os.Stdin) + _, _ = os.Stdin.WriteString("response:partial") + select {} + case "test-overflow": + _, _ = io.ReadAll(os.Stdin) + _, _ = os.Stdin.Write(repeatedPrivilegedControlBytes("r", 128)) + _ = unix.Shutdown(int(os.Stdin.Fd()), unix.SHUT_WR) + case "test-diagnostic-bound": + _, _ = io.ReadAll(os.Stdin) + _, _ = os.Stdout.Write(repeatedPrivilegedControlBytes("o", 128)) + _, _ = os.Stderr.Write(repeatedPrivilegedControlBytes("e", 128)) + _, _ = os.Stdin.WriteString("response:bounded") + _ = unix.Shutdown(int(os.Stdin.Fd()), unix.SHUT_WR) + case "test-exact-response-eof": + _, _ = io.ReadAll(os.Stdin) + _, _ = os.Stdin.Write(repeatedPrivilegedControlBytes("r", 32)) + _ = unix.Shutdown(int(os.Stdin.Fd()), unix.SHUT_WR) + case "test-exact-response-open": + _, _ = io.ReadAll(os.Stdin) + _, _ = os.Stdin.Write(repeatedPrivilegedControlBytes("r", 32)) + select {} + case "test-short-payload": + observed, _ := io.ReadAll(os.Stdin) + _, _ = os.Stdin.Write(append([]byte("observed:"), observed...)) + _ = unix.Shutdown(int(os.Stdin.Fd()), unix.SHUT_WR) + case "test-cancel": + _, _ = io.ReadAll(os.Stdin) + case "test-socket-holder": + _, _ = io.ReadAll(os.Stdin) + holder := exec.Command("/bin/sleep", "30") + holder.Stdin = os.Stdin + if err := holder.Start(); err != nil { + os.Exit(97) + } + _, _ = os.Stdin.WriteString("holder-pid:" + strconv.Itoa(holder.Process.Pid)) + os.Exit(0) + case "test-response-reap": + _, _ = io.ReadAll(os.Stdin) + _, _ = os.Stdin.WriteString("response:complete") + _ = unix.Shutdown(int(os.Stdin.Fd()), unix.SHUT_WR) + select {} + default: + os.Exit(94) + } +} + +func privilegedControlHelperAction(args []string) (string, bool) { + separator := slices.Index(args, "--") + if separator < 0 || separator+1 >= len(args) { + return "", false + } + return args[separator+1], true +} + +func privilegedControlHelperCommand(action string) *exec.Cmd { + return exec.Command(os.Args[0], "-test.run="+privilegedControlHelperTest, "--", action) +} + +func privilegedControlHelperDeps(t *testing.T, action string) privilegedControlDeps { + t.Helper() + deps := realPrivilegedControlDeps() + deps.command = func(_ string, _ ...string) *exec.Cmd { + return privilegedControlHelperCommand(action) + } + return deps +} + +func runPrivilegedControlHelper( + t *testing.T, + request *PrivilegedControlRequest, + action string, + policy *privilegedControlPolicy, +) PrivilegedControlOutcome { + t.Helper() + ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second) + t.Cleanup(cancel) + return runPrivilegedControlHelperWithContext(t, ctx, request, action, policy) +} + +func runPrivilegedControlHelperWithContext( + t *testing.T, + ctx context.Context, + request *PrivilegedControlRequest, + action string, + policy *privilegedControlPolicy, +) PrivilegedControlOutcome { + t.Helper() + selectedPolicy := productionPrivilegedControlPolicy() + if policy != nil { + selectedPolicy = *policy + } + return exchangePrivilegedControl(ctx, request, privilegedControlHelperDeps(t, action), selectedPolicy) +} + +type privilegedControlTrackingPayload struct { + reader *bytes.Reader + closeCount atomic.Int32 + closeErr error +} + +type privilegedControlZeroReader struct{} + +func (privilegedControlZeroReader) Read(target []byte) (int, error) { + clear(target) + return len(target), nil +} + +type privilegedControlPartialWriter struct { + maximum int + value bytes.Buffer +} + +func (writer *privilegedControlPartialWriter) Write(value []byte) (int, error) { + if len(value) > writer.maximum { + value = value[:writer.maximum] + } + return writer.value.Write(value) +} + +type privilegedControlZeroWriter struct{} + +func (privilegedControlZeroWriter) Write(_ []byte) (int, error) { return 0, nil } + +func newPrivilegedControlTrackingPayload(value []byte) *privilegedControlTrackingPayload { + return &privilegedControlTrackingPayload{reader: bytes.NewReader(value)} +} + +func (payload *privilegedControlTrackingPayload) Read(target []byte) (int, error) { + return payload.reader.Read(target) +} + +func (payload *privilegedControlTrackingPayload) Close() error { + payload.closeCount.Add(1) + return payload.closeErr +} + +func (payload *privilegedControlTrackingPayload) CloseCount() int32 { return payload.closeCount.Load() } + +type privilegedControlBlockingPayload struct { + closed chan struct{} + closeCount atomic.Int32 + closeOnce sync.Once +} + +func newPrivilegedControlBlockingPayload() *privilegedControlBlockingPayload { + return &privilegedControlBlockingPayload{closed: make(chan struct{})} +} + +func (payload *privilegedControlBlockingPayload) Read(_ []byte) (int, error) { + <-payload.closed + return 0, io.ErrClosedPipe +} + +func (payload *privilegedControlBlockingPayload) Close() error { + payload.closeCount.Add(1) + payload.closeOnce.Do(func() { close(payload.closed) }) + return nil +} + +func (payload *privilegedControlBlockingPayload) CloseCount() int32 { return payload.closeCount.Load() } + +func repeatedPrivilegedControlBytes(value string, count int) []byte { + return []byte(strings.Repeat(value, count)) +} From 1083d323a6c9d5915c6ac9044db8a28041bf0733 Mon Sep 17 00:00:00 2001 From: AlanD20 Date: Fri, 28 Aug 2026 21:51:07 +0200 Subject: [PATCH 0045/1457] docs: record privileged transport completion --- CHANGELOG.md | 5 +++++ tasks/todo.md | 15 +++++++++------ 2 files changed, 14 insertions(+), 6 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index f305a70e..1544018c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -45,6 +45,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Added the version-1 `MVMREQ01`/`MVMRES01` codec with 64 KiB JSON frames, receiver-bounded payload declarations, exact action/schema matching, read-only typed success/error outcomes distinct from protocol failures, and bounded `DomainError` normalization that omits wrapped causes and sensitive process-output details. +- Added the fixed fd-0 Unix socketpair client transport for + `/usr/bin/sudo -n -- /usr/local/bin/mvm __mvm_privileged_v1 `. It uses close-on-exec endpoints, + concurrent bounded upload and response handling, actual peer EOF, early-response upload interruption, cancellation, + bounded diagnostics, and bounded process reaping without extending the generic command runner. Capability-specific + clients and actions are still release blockers below. - Added authenticated sudo identity and current `mvm` group checks, fixed root environment sanitization, and executable device/inode verification against the root-owned `/usr/local/bin/mvm` image. - Added ADR-0016, which defines the single-binary privilege boundary, trusted state layout, typed operation catalog, diff --git a/tasks/todo.md b/tasks/todo.md index afc266b2..11d9c502 100644 --- a/tasks/todo.md +++ b/tasks/todo.md @@ -56,7 +56,7 @@ The final marker-only sudoers assertion is deferred until Task 14; current sudoe ### Task 3: Complete the early privileged protocol foundation **Owner:** engineer -**Status:** Foundation implemented; action catalog pending later tasks +**Status:** Codec and transport implemented; capability clients and action catalog pending later tasks **Dependencies:** Task 1 - [x] Reserved marker selected before Cobra and application initialization. @@ -66,12 +66,15 @@ The final marker-only sudoers assertion is deferred until Task 14; current sudoe - [x] Add a strict versioned request/response codec with 64 KiB frames, duplicate/unknown/depth rejection, matching actions, typed read-only success/error outcomes separated from protocol failures, and bounded `DomainError` code/class/entity/partial-state detail preservation. -- [ ] Add the fd-0 Unix socketpair control transport with fixed `/usr/bin/sudo -n -- /usr/local/bin/mvm` invocation, +- [x] Add the fd-0 Unix socketpair control transport with fixed `/usr/bin/sudo -n -- /usr/local/bin/mvm` invocation, concurrent upload/response, request and response half-close/actual EOF, CLOEXEC, cancellation, bounded reaping and - buffers, early-rejection handling, private process dependencies, and a dedicated fake exchanger; do not extend - `CommandRunner`, `RunCmdOpts`, `SpawnConfig`, or `FakeRunner`. -- [ ] Return `process_started: true` and `outcome_unknown: true` for every started process without one valid final - response; stdout, stderr, and generic subprocess text are never action authority. + buffers, early-rejection handling, and private process dependencies; do not extend `CommandRunner`, `RunCmdOpts`, + `SpawnConfig`, or `FakeRunner`. +- [x] Transport failures distinguish pre-start failures from started, outcome-unknown failures; stdout, stderr, and + generic subprocess text are diagnostic only and never response authority. +- [ ] Give each capability package a private minimal `Exchange` interface and dedicated fake when its first client is + added. The client must decode only EOF-qualified responses, report malformed or missing final responses as started + and outcome-unknown, and let a valid typed result win over upload, exit, or bounded-reaping diagnostics. - [ ] Keep the action switch closed until each capability has its own typed handler and abuse tests. ### Task 4: Add private root-owned VM instance authority From c2b3a9eb8519ec8bf2a4716747263401aa0e3a29 Mon Sep 17 00:00:00 2001 From: AlanD20 Date: Fri, 28 Aug 2026 22:34:11 +0200 Subject: [PATCH 0046/1457] feat: fix managed resource filenames --- internal/cli/service.go | 17 +-- internal/cli/service_internal_test.go | 19 +++ internal/core/cloudinit/config.go | 1 - internal/core/cloudinit/provisioner.go | 32 ++++- internal/core/cloudinit/provisioner_test.go | 35 +++++ internal/core/console/controller.go | 26 ++-- internal/core/vm/jailer_test.go | 16 ++- internal/infra/constants.go | 43 ++++-- internal/infra/constants_test.go | 59 ++++++++ internal/lib/model/vm.go | 16 +-- internal/service/console/entry.go | 37 +---- internal/service/console/spawn.go | 13 +- internal/service/jailer/entry.go | 45 +++++- internal/service/jailer/entry_test.go | 61 +++++++-- pkg/api/console.go | 4 +- pkg/api/inputs/logs_input.go | 20 +-- pkg/api/inputs/vm_create.go | 34 +---- pkg/api/logs.go | 8 +- pkg/api/snapshot.go | 97 +++++++++---- pkg/api/vm.go | 143 ++++++++++---------- pkg/api/vm_resource_paths_test.go | 137 +++++++++++++++++++ 21 files changed, 595 insertions(+), 268 deletions(-) create mode 100644 internal/cli/service_internal_test.go create mode 100644 internal/core/cloudinit/provisioner_test.go create mode 100644 pkg/api/vm_resource_paths_test.go diff --git a/internal/cli/service.go b/internal/cli/service.go index 6e144769..8e06a993 100644 --- a/internal/cli/service.go +++ b/internal/cli/service.go @@ -151,9 +151,6 @@ func newConsoleRelayCmd() *cobra.Command { cmd.Flags().String("vm-path", "", "VM path (required)") cmd.Flags().String("vm-name", "", "VM name") cmd.Flags().Int("pty-fd", 0, "PTY file descriptor (required)") - cmd.Flags().String("pid-filename", "", "PID file name (default: console.pid)") - cmd.Flags().String("socket-filename", "", "Socket file name (default: console.sock)") - cmd.Flags().String("log-filename", "", "Log file name (default: firecracker.console.log)") cmd.Flags().Bool("daemon", false, "Run as a background daemon process") cmd.MarkFlagRequired("vm-id") cmd.MarkFlagRequired("vm-path") @@ -164,19 +161,13 @@ func newConsoleRelayCmd() *cobra.Command { vmPath, _ := c.Flags().GetString("vm-path") vmName, _ := c.Flags().GetString("vm-name") ptyFD, _ := c.Flags().GetInt("pty-fd") - pidFilename, _ := c.Flags().GetString("pid-filename") - socketFilename, _ := c.Flags().GetString("socket-filename") - logFilename, _ := c.Flags().GetString("log-filename") daemon, _ := c.Flags().GetBool("daemon") cfg := console.Config{ - VMID: vmID, - VMPath: vmPath, - VMName: vmName, - PtyFD: ptyFD, - PIDFilename: pidFilename, - SocketFilename: socketFilename, - LogFilename: logFilename, + VMID: vmID, + VMPath: vmPath, + VMName: vmName, + PtyFD: ptyFD, } if daemon { ptyFile := os.NewFile(uintptr(ptyFD), "pty") diff --git a/internal/cli/service_internal_test.go b/internal/cli/service_internal_test.go new file mode 100644 index 00000000..27161656 --- /dev/null +++ b/internal/cli/service_internal_test.go @@ -0,0 +1,19 @@ +package cli + +import ( + "testing" + + "github.com/stretchr/testify/assert" +) + +// Rationale: the private relay command must not reintroduce a second path +// selection surface for fixed managed artifacts. +func TestConsoleRelayDoesNotExposeManagedFilenameOverrides(t *testing.T) { + command := newConsoleRelayCmd() + + for _, name := range []string{"pid-filename", "socket-filename", "log-filename"} { + t.Run(name, func(t *testing.T) { + assert.Nil(t, command.Flags().Lookup(name)) + }) + } +} diff --git a/internal/core/cloudinit/config.go b/internal/core/cloudinit/config.go index c06bd055..603b9523 100644 --- a/internal/core/cloudinit/config.go +++ b/internal/core/cloudinit/config.go @@ -30,7 +30,6 @@ type Config struct { // Resolved from defaults UserPassword string // from defaults.vm.user_password - CloudInitISOName string // from defaults.cloudinit.iso_name NocloudPortRangeStart int // from defaults.cloudinit.nocloud_port_range_start NocloudPortRangeEnd int // from defaults.cloudinit.nocloud_port_range_end NocloudMaxPortRetries int // from defaults.cloudinit.nocloud_max_port_retries diff --git a/internal/core/cloudinit/provisioner.go b/internal/core/cloudinit/provisioner.go index afda0643..2edd5306 100644 --- a/internal/core/cloudinit/provisioner.go +++ b/internal/core/cloudinit/provisioner.go @@ -176,21 +176,43 @@ func (p *Provisioner) provisionNet(ctx context.Context) (*model.CloudInitResult, func (p *Provisioner) provisionISO(ctx context.Context) (*model.CloudInitResult, error) { // Check for pre-existing custom ISO path if p.config.CloudInitISOPath != nil { - isoPath := *p.config.CloudInitISOPath - if _, err := os.Stat(isoPath); os.IsNotExist(err) { + sourcePath := *p.config.CloudInitISOPath + sourceInfo, err := os.Stat(sourcePath) + if os.IsNotExist(err) { return nil, errs.New(errs.CodeCloudInitISOModeFailed, - fmt.Sprintf("Custom cloud-init ISO not found: %s", isoPath), + fmt.Sprintf("custom cloud-init ISO not found: %s", sourcePath), + ) + } + if err != nil { + return nil, errs.New( + errs.CodeCloudInitISOModeFailed, + fmt.Sprintf("inspect custom cloud-init ISO: %s", err), + ) + } + + isoPath := filepath.Join(p.config.VMDir, infra.VMCloudInitISOFilename) + if destinationInfo, statErr := os.Stat(isoPath); statErr == nil && os.SameFile(sourceInfo, destinationInfo) { + return &model.CloudInitResult{ + Mode: model.CloudInitModeISO, + ISOPath: &isoPath, + NocloudNetRules: []model.FirewallRule{}, + }, nil + } + if err := infra.CopyFile(sourcePath, isoPath); err != nil { + return nil, errs.New( + errs.CodeCloudInitISOModeFailed, + fmt.Sprintf("copy custom cloud-init ISO to managed path: %s", err), ) } return &model.CloudInitResult{ Mode: model.CloudInitModeISO, - ISOPath: p.config.CloudInitISOPath, + ISOPath: &isoPath, NocloudNetRules: []model.FirewallRule{}, }, nil } // Generate ISO from seed directory - isoPath := filepath.Join(p.config.VMDir, p.config.CloudInitISOName) + isoPath := filepath.Join(p.config.VMDir, infra.VMCloudInitISOFilename) if err := p.manager.CreateSeedISO(ctx, p.config.CloudInitDir, isoPath); err != nil { return nil, errs.New(errs.CodeCloudInitISOModeFailed, fmt.Sprintf("Failed to create cloud-init ISO: %s", err), diff --git a/internal/core/cloudinit/provisioner_test.go b/internal/core/cloudinit/provisioner_test.go new file mode 100644 index 00000000..0885b850 --- /dev/null +++ b/internal/core/cloudinit/provisioner_test.go @@ -0,0 +1,35 @@ +package cloudinit + +import ( + "os" + "path/filepath" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "mvmctl/internal/infra" +) + +// Rationale: a custom source path is input data, not a managed artifact name. +// ISO provisioning must copy it into the fixed VM-local leaf. +func TestProvisionISOCopiesCustomSourceToFixedManagedLeaf(t *testing.T) { + sourceDir := t.TempDir() + vmDir := t.TempDir() + sourcePath := filepath.Join(sourceDir, "custom-name.iso") + require.NoError(t, os.WriteFile(sourcePath, []byte("custom-iso"), 0600)) + + provisioner := &Provisioner{config: &Config{ + VMDir: vmDir, + CloudInitISOPath: &sourcePath, + }} + result, err := provisioner.provisionISO(t.Context()) + require.NoError(t, err) + + wantPath := filepath.Join(vmDir, infra.VMCloudInitISOFilename) + require.NotNil(t, result.ISOPath) + assert.Equal(t, wantPath, *result.ISOPath) + data, err := os.ReadFile(wantPath) + require.NoError(t, err) + assert.Equal(t, []byte("custom-iso"), data) +} diff --git a/internal/core/console/controller.go b/internal/core/console/controller.go index 0ad3f529..d23b438c 100644 --- a/internal/core/console/controller.go +++ b/internal/core/console/controller.go @@ -11,6 +11,7 @@ import ( "golang.org/x/sys/unix" + "mvmctl/internal/infra" consolesvc "mvmctl/internal/service/console" ) @@ -27,8 +28,7 @@ import ( // A Controller is not safe for concurrent use. The API layer serializes // calls per-VM through its own execution model. type Controller struct { - vmID, vmPath, vmName string - pidFilename, socketFilename string + vmID, vmPath, vmName string // PTY state masterFD int // PTY master → relay subprocess @@ -43,16 +43,14 @@ type Controller struct { } // NewController creates a new Controller for the given VM. -func NewController(vmID, vmPath, vmName, pidFilename, socketFilename string) *Controller { +func NewController(vmID, vmPath, vmName string) *Controller { if vmName == "" { vmName = vmID } return &Controller{ - vmID: vmID, - vmPath: vmPath, - vmName: vmName, - pidFilename: pidFilename, - socketFilename: socketFilename, + vmID: vmID, + vmPath: vmPath, + vmName: vmName, } } @@ -113,11 +111,9 @@ func (cc *Controller) Start(ctx context.Context) (string, *int, error) { } cfg := consolesvc.Config{ - VMID: cc.vmID, - VMPath: cc.vmPath, - VMName: cc.vmName, - PIDFilename: cc.pidFilename, - SocketFilename: cc.socketFilename, + VMID: cc.vmID, + VMPath: cc.vmPath, + VMName: cc.vmName, } result, err := consolesvc.Spawn(ctx, cfg, ptyFile) @@ -131,8 +127,8 @@ func (cc *Controller) Start(ctx context.Context) (string, *int, error) { // Create the relay manager now that the subprocess is alive. // Spawn already wrote the PID file, so the Relay can find the PID. - pidPath := filepath.Join(cc.vmPath, cc.pidFilename) - sockPath := filepath.Join(cc.vmPath, cc.socketFilename) + pidPath := filepath.Join(cc.vmPath, infra.VMConsolePIDFilename) + sockPath := filepath.Join(cc.vmPath, infra.VMConsoleSocketFilename) cc.relayManager = consolesvc.NewRelay(cc.vmName, pidPath, sockPath) return result.SocketPath, &result.PID, nil diff --git a/internal/core/vm/jailer_test.go b/internal/core/vm/jailer_test.go index 49a04d3d..fbce0df7 100644 --- a/internal/core/vm/jailer_test.go +++ b/internal/core/vm/jailer_test.go @@ -216,9 +216,9 @@ func TestJailedPathCalculation(t *testing.T) { func jailerTestConfig(vmDir string) *model.FirecrackerConfig { cloudInitMode := model.CloudInitModeISO - isoPath := filepath.Join(vmDir, "cloud-init.iso") + isoPath := filepath.Join(vmDir, infra.VMCloudInitISOFilename) return &model.FirecrackerConfig{ - VMDir: vmDir, RootfsPath: filepath.Join(vmDir, "rootfs.ext4"), + VMDir: vmDir, RootfsPath: filepath.Join(vmDir, infra.VMRootfsFilename), BinaryPath: "/var/lib/mvmctl/binaries/1.16.0/firecracker", JailerPath: "/var/lib/mvmctl/binaries/1.16.0/jailer", BinaryVersion: "1.16.0", KernelPath: filepath.Join(filepath.Dir(vmDir), "kernels", "vmlinux"), @@ -229,17 +229,19 @@ func jailerTestConfig(vmDir string) *model.FirecrackerConfig { TapName: "tap-test", NetworkGateway: "10.0.0.1", NetworkNetmask: "255.255.255.0", PCIEnabled: true, ImageFSUUID: "11111111-2222-3333-4444-555555555555", ImageFSType: "ext4", EnableLogging: true, EnableMetrics: true, LogLevel: "Info", - LogPath: filepath.Join(vmDir, "firecracker.log"), MetricsPath: filepath.Join(vmDir, "metrics.fifo"), - SerialOutputPath: filepath.Join(vmDir, "serial.log"), - APISocketPath: filepath.Join(vmDir, "api.socket"), PIDPath: filepath.Join(vmDir, "firecracker.pid"), - ConfigPath: filepath.Join(vmDir, "config.json"), CloudInitMode: &cloudInitMode, + LogPath: filepath.Join(vmDir, infra.VMFirecrackerLogFilename), + MetricsPath: filepath.Join(vmDir, infra.VMFirecrackerMetricsFilename), + SerialOutputPath: filepath.Join(vmDir, infra.VMFirecrackerConsoleLogFilename), + APISocketPath: filepath.Join(vmDir, infra.VMFirecrackerAPISocketFilename), + PIDPath: filepath.Join(vmDir, infra.VMFirecrackerPIDFilename), + ConfigPath: filepath.Join(vmDir, infra.VMFirecrackerConfigFilename), CloudInitMode: &cloudInitMode, CloudInitISOPath: &isoPath, ExtraDrives: []model.DriveConfig{{ DriveID: testDriveID, PathOnHost: filepath.Join(filepath.Dir(vmDir), "volumes", "selected.raw"), IsReadOnly: true, CacheType: model.CacheTypeUnsafe, IOEngine: "Sync", }}, SnapshotMode: true, SnapshotDir: filepath.Join(filepath.Dir(vmDir), "snapshots", testDriveID), - Vsock: &model.VsockConfig{GuestCID: 42, UDSPath: filepath.Join(vmDir, "vsock.socket")}, + Vsock: &model.VsockConfig{GuestCID: 42, UDSPath: filepath.Join(vmDir, infra.VMVsockSocketFilename)}, CPUVendor: ptr.Ptr("Intel"), } } diff --git a/internal/infra/constants.go b/internal/infra/constants.go index 2c675ff8..9df66644 100644 --- a/internal/infra/constants.go +++ b/internal/infra/constants.go @@ -74,6 +74,37 @@ const CgroupV2Root = "/sys/fs/cgroup" // JailerManifestFilename is the fixed per-VM launch manifest name. const JailerManifestFilename = "jailer-launch.json" +const ( + // VMRootfsFilename is the fixed managed VM root filesystem leaf. + VMRootfsFilename = "rootfs.img" + // VMCloudInitISOFilename is the fixed managed VM cloud-init ISO leaf. + VMCloudInitISOFilename = "cloud-init.iso" + // VMFirecrackerConfigFilename is the fixed managed Firecracker configuration leaf. + VMFirecrackerConfigFilename = "firecracker.json" + // VMFirecrackerLogFilename is the fixed managed Firecracker log leaf. + VMFirecrackerLogFilename = "firecracker.log" + // VMFirecrackerConsoleLogFilename is the fixed managed Firecracker console log leaf. + VMFirecrackerConsoleLogFilename = "firecracker.console.log" + // VMFirecrackerMetricsFilename is the fixed managed Firecracker metrics leaf. + VMFirecrackerMetricsFilename = "firecracker.metrics" + // VMFirecrackerAPISocketFilename is the fixed Firecracker API socket leaf. + VMFirecrackerAPISocketFilename = "firecracker.api.socket" + // VMVsockSocketFilename is the fixed Firecracker vsock host socket leaf. + VMVsockSocketFilename = "vsock.sock" + // VMConsoleSocketFilename is the fixed console relay socket leaf. + VMConsoleSocketFilename = "console.sock" + // VMFirecrackerPIDFilename is the fixed display-only Firecracker PID leaf. + VMFirecrackerPIDFilename = "firecracker.pid" + // VMConsolePIDFilename is the fixed display-only console relay PID leaf. + VMConsolePIDFilename = "console.pid" + // SnapshotRootfsFilename is the fixed managed snapshot root filesystem leaf. + SnapshotRootfsFilename = "rootfs.img" + // SnapshotMemoryFilename is the fixed managed snapshot memory leaf. + SnapshotMemoryFilename = "memory" + // SnapshotStateFilename is the fixed managed snapshot state leaf. + SnapshotStateFilename = "vmstate" +) + const MVMForwardChain = "MVM-FORWARD" const MVMPostroutingChain = "MVM-POSTROUTING" @@ -140,22 +171,12 @@ var OverridableDefaults = map[string]map[string]any{ "remote_list_cache_ttl": 14400, }, "defaults.firecracker": { - "log_level": "Debug", - "log_filename": "firecracker.log", - "serial_output_filename": "firecracker.console.log", - "metrics_filename": "firecracker.metrics", - "api_socket_filename": "firecracker.api.socket", - "pid_filename": "firecracker.pid", - "config_filename": "firecracker.json", - "console_socket_filename": "console.sock", - "console_pid_filename": "console.pid", - "vsock_filename": "vsock.sock", + "log_level": "Debug", }, "defaults.volume": { "cache_type": "Unsafe", }, "defaults.cloudinit": { - "iso_name": "cloud-init.iso", "nocloud_port_range_start": 8000, "nocloud_port_range_end": 9000, "nocloud_max_port_retries": 100, diff --git a/internal/infra/constants_test.go b/internal/infra/constants_test.go index cbe29f7a..8d014d4b 100644 --- a/internal/infra/constants_test.go +++ b/internal/infra/constants_test.go @@ -61,6 +61,65 @@ func TestSystemBinaryPath(t *testing.T) { assert.Equal(t, "/usr/local/bin/mvm", infra.SystemBinaryPath) } +func TestManagedResourceFilenames(t *testing.T) { + want := map[string]string{ + "vm rootfs": "rootfs.img", + "cloud-init ISO": "cloud-init.iso", + "Firecracker config": "firecracker.json", + "Firecracker log": "firecracker.log", + "Firecracker console": "firecracker.console.log", + "Firecracker metrics": "firecracker.metrics", + "Firecracker API socket": "firecracker.api.socket", + "vsock socket": "vsock.sock", + "console socket": "console.sock", + "Firecracker PID": "firecracker.pid", + "console PID": "console.pid", + "snapshot rootfs": "rootfs.img", + "snapshot memory": "memory", + "snapshot state": "vmstate", + } + got := map[string]string{ + "vm rootfs": infra.VMRootfsFilename, + "cloud-init ISO": infra.VMCloudInitISOFilename, + "Firecracker config": infra.VMFirecrackerConfigFilename, + "Firecracker log": infra.VMFirecrackerLogFilename, + "Firecracker console": infra.VMFirecrackerConsoleLogFilename, + "Firecracker metrics": infra.VMFirecrackerMetricsFilename, + "Firecracker API socket": infra.VMFirecrackerAPISocketFilename, + "vsock socket": infra.VMVsockSocketFilename, + "console socket": infra.VMConsoleSocketFilename, + "Firecracker PID": infra.VMFirecrackerPIDFilename, + "console PID": infra.VMConsolePIDFilename, + "snapshot rootfs": infra.SnapshotRootfsFilename, + "snapshot memory": infra.SnapshotMemoryFilename, + "snapshot state": infra.SnapshotStateFilename, + } + + assert.Equal(t, want, got) +} + +func TestManagedResourceFilenamesAreNotConfigurable(t *testing.T) { + for _, key := range []string{ + "log_filename", + "serial_output_filename", + "metrics_filename", + "api_socket_filename", + "pid_filename", + "config_filename", + "console_socket_filename", + "console_pid_filename", + "vsock_filename", + } { + t.Run(key, func(t *testing.T) { + _, err := infra.GetDefault("defaults.firecracker", key) + require.Error(t, err) + }) + } + + _, err := infra.GetDefault("defaults.cloudinit", "iso_name") + require.Error(t, err) +} + // --- EnvKey --- // Rationale: EnvKey constructs environment variable names used throughout the // codebase for config overrides. Wrong prefix would cause silent misconfiguration. diff --git a/internal/lib/model/vm.go b/internal/lib/model/vm.go index 0b425afa..b94c1eae 100644 --- a/internal/lib/model/vm.go +++ b/internal/lib/model/vm.go @@ -64,20 +64,8 @@ type VMItem struct { SerialOutputPath *string `json:"serial_output_path,omitempty" db:"serial_output_path"` LSMFlags string `json:"lsm_flags" db:"lsm_flags"` BootArgs string `json:"boot_args" db:"boot_args"` - // NOTE: These fields are NOT stored in the database. They are runtime - // configuration values resolved from defaults/input at VM creation time. - // The *Path fields in migration 001 (config_path, log_path, etc.) store - // the resolved full paths in the DB; these *Filename fields are the - // basenames used to construct those paths and are ephemeral. - LogLevel string `json:"log_level"` - LogFilename string `json:"log_filename"` - SerialOutputFilename string `json:"serial_output_filename"` - MetricsFilename string `json:"metrics_filename"` - APISocketFilename string `json:"api_socket_filename"` - PIDFilename string `json:"pid_filename"` - ConfigFilename string `json:"config_filename"` - ConsolePIDFilename string `json:"console_pid_filename"` - ConsoleSocketFilename string `json:"console_socket_filename"` + // LogLevel is runtime configuration and is not stored in the database. + LogLevel string `json:"log_level"` // JSON-serialized in DB fields (TEXT columns, scanned directly via // db.StringSlice / CpuConfig.Scan) diff --git a/internal/service/console/entry.go b/internal/service/console/entry.go index 3bd787bd..6e6ce1ac 100644 --- a/internal/service/console/entry.go +++ b/internal/service/console/entry.go @@ -18,13 +18,6 @@ import ( ) const ( - // DefaultConsolePIDFilename is the default PID file name for console relays. - DefaultConsolePIDFilename = "console.pid" - // DefaultConsoleSocketFilename is the default socket file name for console relays. - DefaultConsoleSocketFilename = "console.sock" - // DefaultConsoleLogFilename is the default log file name for console relays. - DefaultConsoleLogFilename = "firecracker.console.log" - consoleReadBufferSize = 4096 // CONST_CONSOLE_READ_BUFFER_SIZE // Wire protocol constants for the console relay control header. @@ -36,13 +29,10 @@ const ( // Config holds configuration for the console relay subprocess. type Config struct { - VMID string - VMPath string - VMName string - PtyFD int - PIDFilename string // optional, defaults to DefaultConsolePIDFilename - SocketFilename string // optional, defaults to DefaultConsoleSocketFilename - LogFilename string // optional, defaults to DefaultConsoleLogFilename + VMID string + VMPath string + VMName string + PtyFD int } // Run starts the console relay with the given config. @@ -52,22 +42,9 @@ func Run(ctx context.Context, cfg Config) error { return fmt.Errorf("missing required arguments for console relay") } - // Compute paths from VM directory, using provided filenames or defaults. - pidName := cfg.PIDFilename - if pidName == "" { - pidName = DefaultConsolePIDFilename - } - sockName := cfg.SocketFilename - if sockName == "" { - sockName = DefaultConsoleSocketFilename - } - logName := cfg.LogFilename - if logName == "" { - logName = DefaultConsoleLogFilename - } - pidPath := filepath.Join(cfg.VMPath, pidName) - socketPath := filepath.Join(cfg.VMPath, sockName) - logPath := filepath.Join(cfg.VMPath, logName) + pidPath := filepath.Join(cfg.VMPath, infra.VMConsolePIDFilename) + socketPath := filepath.Join(cfg.VMPath, infra.VMConsoleSocketFilename) + logPath := filepath.Join(cfg.VMPath, infra.VMFirecrackerConsoleLogFilename) // Open the PTY FD inherited from parent (passed as ExtraFiles[0] = FD 3). ptyFile := os.NewFile(uintptr(cfg.PtyFD), "pty") diff --git a/internal/service/console/spawn.go b/internal/service/console/spawn.go index c00a9bb1..85515d74 100644 --- a/internal/service/console/spawn.go +++ b/internal/service/console/spawn.go @@ -29,15 +29,6 @@ func Spawn(ctx context.Context, cfg Config, ptyFile *os.File) (*SpawnResult, err if cfg.VMName != "" { args = append(args, "--vm-name", cfg.VMName) } - if cfg.PIDFilename != "" { - args = append(args, "--pid-filename", cfg.PIDFilename) - } - if cfg.SocketFilename != "" { - args = append(args, "--socket-filename", cfg.SocketFilename) - } - if cfg.LogFilename != "" { - args = append(args, "--log-filename", cfg.LogFilename) - } cmd, err := system.SpawnService(nil, system.SpawnConfig{ Name: "console", @@ -54,14 +45,14 @@ func Spawn(ctx context.Context, cfg Config, ptyFile *os.File) (*SpawnResult, err pid := cmd.Process.Pid // Write PID file alongside the socket. - pidFilePath := filepath.Join(cfg.VMPath, cfg.PIDFilename) + pidFilePath := filepath.Join(cfg.VMPath, infra.VMConsolePIDFilename) if pidDir := filepath.Dir(pidFilePath); pidDir != "." { os.MkdirAll(pidDir, infra.DirPerm) } os.WriteFile(pidFilePath, []byte(strconv.Itoa(pid)), 0644) // Wait for subprocess to create the socket. - socketPath := filepath.Join(cfg.VMPath, cfg.SocketFilename) + socketPath := filepath.Join(cfg.VMPath, infra.VMConsoleSocketFilename) for range 50 { if _, err := os.Stat(socketPath); err == nil { return &SpawnResult{ diff --git a/internal/service/jailer/entry.go b/internal/service/jailer/entry.go index 7a2c499c..93e5e395 100644 --- a/internal/service/jailer/entry.go +++ b/internal/service/jailer/entry.go @@ -222,7 +222,12 @@ func launch(cfg Config) error { return err } if manifest.ISOPath != "" { - if err := mountResource(manifest.ISOPath, filepath.Join(jailRoot, "cloud-init.iso"), true, false); err != nil { + if err := mountResource( + manifest.ISOPath, + filepath.Join(jailRoot, infra.VMCloudInitISOFilename), + true, + false, + ); err != nil { return err } } @@ -273,13 +278,13 @@ func launch(cfg Config) error { "--cgroup", fmt.Sprintf("memory.swap.max=%d", manifest.CgroupLimits.SwapMaxBytes), "--cgroup", fmt.Sprintf("pids.max=%d", manifest.CgroupLimits.PIDsMax), "--", - "--api-sock", "/run/mvm/" + filepath.Base(manifest.APISocket), + "--api-sock", "/run/mvm/" + infra.VMFirecrackerAPISocketFilename, } if manifest.PCIEnabled { args = append(args, "--enable-pci") } if !manifest.SnapshotMode { - args = append(args, "--config-file", "/run/mvm/"+filepath.Base(manifest.ConfigPath)) + args = append(args, "--config-file", "/run/mvm/"+infra.VMFirecrackerConfigFilename) } return syscall.Exec(jailerPath, args, os.Environ()) } @@ -320,6 +325,9 @@ func loadManifest(vmID, vmDir string) (*LaunchManifest, error) { if !versionPattern.MatchString(manifest.Version) { return nil, fmt.Errorf("invalid jailed launch version") } + if err := validateCanonicalManifestVMResources(&manifest, paths.vmDir); err != nil { + return nil, err + } if err := validateManagedFile(manifestPath, paths.vmDir, paths.uid, true); err != nil { return nil, fmt.Errorf("invalid managed launch manifest: %w", err) } @@ -357,6 +365,37 @@ func loadManifest(vmID, vmDir string) (*LaunchManifest, error) { return &manifest, nil } +func validateCanonicalManifestVMResources(manifest *LaunchManifest, vmDir string) error { + type resourcePath struct { + name string + got string + want string + } + resources := []resourcePath{ + { + name: "configuration", + got: manifest.ConfigPath, + want: filepath.Join(vmDir, infra.VMFirecrackerConfigFilename), + }, + {name: "rootfs", got: manifest.RootfsPath, want: filepath.Join(vmDir, infra.VMRootfsFilename)}, + {name: "PID", got: manifest.PIDPath, want: filepath.Join(vmDir, infra.VMFirecrackerPIDFilename)}, + {name: "API socket", got: manifest.APISocket, want: filepath.Join(vmDir, infra.VMFirecrackerAPISocketFilename)}, + } + if manifest.ISOPath != "" { + resources = append(resources, resourcePath{ + name: "cloud-init ISO", + got: manifest.ISOPath, + want: filepath.Join(vmDir, infra.VMCloudInitISOFilename), + }) + } + for _, resource := range resources { + if resource.got != resource.want { + return fmt.Errorf("non-canonical managed VM resource path for %s", resource.name) + } + } + return nil +} + func resolveManagedPaths(vmID, vmDir string) (*managedPaths, error) { if !vmIDPattern.MatchString(vmID) { return nil, fmt.Errorf("invalid VM identity for jailed launch") diff --git a/internal/service/jailer/entry_test.go b/internal/service/jailer/entry_test.go index 3a24fb2b..7ad9077c 100644 --- a/internal/service/jailer/entry_test.go +++ b/internal/service/jailer/entry_test.go @@ -31,6 +31,40 @@ func TestLoadManifest_AcceptsExactManagedResources(t *testing.T) { } } +// Rationale: containment is not authority to select a privileged-visible +// basename. The receiver must accept only its compiled per-VM leaf names. +func TestLoadManifest_RejectsNonCanonicalVMResourceLeaves(t *testing.T) { + tests := map[string]func(*testing.T, *managedFixture){ + "config": func(t *testing.T, fixture *managedFixture) { + fixture.manifest.ConfigPath = fixture.writeVMFile(t, "alternate.json") + }, + "rootfs": func(t *testing.T, fixture *managedFixture) { + fixture.manifest.RootfsPath = fixture.writeVMFile(t, "alternate.img") + }, + "cloud_init": func(t *testing.T, fixture *managedFixture) { + fixture.manifest.ISOPath = fixture.writeVMFile(t, "alternate.iso") + }, + "pid": func(_ *testing.T, fixture *managedFixture) { + fixture.manifest.PIDPath = filepath.Join(fixture.vmDir, "alternate.pid") + }, + "api_socket": func(_ *testing.T, fixture *managedFixture) { + fixture.manifest.APISocket = filepath.Join(fixture.vmDir, "alternate.socket") + }, + } + + for name, mutate := range tests { + t.Run(name, func(t *testing.T) { + fixture := newManagedFixture(t) + mutate(t, fixture) + fixture.writeManifest(t) + + manifest, err := loadManifest(testVMID, fixture.vmDir) + require.ErrorContains(t, err, "non-canonical managed VM resource path") + assert.Nil(t, manifest) + }) + } +} + // Rationale: The privileged service must not treat an attacker-controlled path // as managed merely because its immediate parent has a managed collection name. func TestLoadManifest_RejectsArbitraryCollectionPaths(t *testing.T) { @@ -291,11 +325,11 @@ func newManagedFixture(t *testing.T) *managedFixture { snapshotDir := filepath.Join(snapshotsRoot, testResourceID) require.NoError(t, os.Mkdir(snapshotDir, 0700)) files := map[string]string{ - filepath.Join(vmDir, "config.json"): "{}", - filepath.Join(vmDir, "rootfs.ext4"): "rootfs", - filepath.Join(vmDir, "cloud-init.iso"): "iso", - filepath.Join(kernelsRoot, "vmlinux"): "kernel", - filepath.Join(volumesRoot, "volume.raw"): "volume", + filepath.Join(vmDir, infra.VMFirecrackerConfigFilename): "{}", + filepath.Join(vmDir, infra.VMRootfsFilename): "rootfs", + filepath.Join(vmDir, infra.VMCloudInitISOFilename): "iso", + filepath.Join(kernelsRoot, "vmlinux"): "kernel", + filepath.Join(volumesRoot, "volume.raw"): "volume", } for path, content := range files { require.NoError(t, os.WriteFile(path, []byte(content), 0600)) @@ -303,11 +337,13 @@ func newManagedFixture(t *testing.T) *managedFixture { fixture := &managedFixture{vmDir: vmDir} fixture.manifest = LaunchManifest{ VMID: testVMID, VMDir: vmDir, Version: "1.16.0", - ConfigPath: filepath.Join(vmDir, "config.json"), PIDPath: filepath.Join(vmDir, "firecracker.pid"), - KernelPath: filepath.Join(kernelsRoot, "vmlinux"), RootfsPath: filepath.Join(vmDir, "rootfs.ext4"), - ISOPath: filepath.Join(vmDir, "cloud-init.iso"), SnapshotDir: snapshotDir, + ConfigPath: filepath.Join(vmDir, infra.VMFirecrackerConfigFilename), + PIDPath: filepath.Join(vmDir, infra.VMFirecrackerPIDFilename), + KernelPath: filepath.Join(kernelsRoot, "vmlinux"), + RootfsPath: filepath.Join(vmDir, infra.VMRootfsFilename), + ISOPath: filepath.Join(vmDir, infra.VMCloudInitISOFilename), SnapshotDir: snapshotDir, Volumes: []VolumeMount{{DriveID: testResourceID, HostPath: filepath.Join(volumesRoot, "volume.raw")}}, - APISocket: filepath.Join(vmDir, "firecracker.socket"), + APISocket: filepath.Join(vmDir, infra.VMFirecrackerAPISocketFilename), CgroupLimits: func() *model.VMCgroupLimits { limits := model.NewVMCgroupLimits(2, 512, model.VMCgroupPolicy{ VMMHeadroomMiB: 128, CPUWeight: 100, PIDsMax: 256, SwapMaxBytes: 0, @@ -325,3 +361,10 @@ func (f *managedFixture) writeManifest(t *testing.T) { require.NoError(t, err) require.NoError(t, os.WriteFile(filepath.Join(f.vmDir, infra.JailerManifestFilename), data, 0600)) } + +func (f *managedFixture) writeVMFile(t *testing.T, name string) string { + t.Helper() + path := filepath.Join(f.vmDir, name) + require.NoError(t, os.WriteFile(path, []byte("alternate"), 0600)) + return path +} diff --git a/pkg/api/console.go b/pkg/api/console.go index 8b8c108c..81187b12 100644 --- a/pkg/api/console.go +++ b/pkg/api/console.go @@ -111,8 +111,8 @@ func (op *Operation) resolveConsole(ctx context.Context, identifier string) (*in vmEntity := vms[0] // Create relay manager. vmDir := infra.GetVMDirByID(vmEntity.ID) - pidPath := filepath.Join(vmDir, console.DefaultConsolePIDFilename) - socketPath := filepath.Join(vmDir, console.DefaultConsoleSocketFilename) + pidPath := filepath.Join(vmDir, infra.VMConsolePIDFilename) + socketPath := filepath.Join(vmDir, infra.VMConsoleSocketFilename) relay := console.NewRelay(vmEntity.Name, pidPath, socketPath) return rawInput.Resolve(vmEntity, relay) } diff --git a/pkg/api/inputs/logs_input.go b/pkg/api/inputs/logs_input.go index f0809d3a..c70994e9 100644 --- a/pkg/api/inputs/logs_input.go +++ b/pkg/api/inputs/logs_input.go @@ -19,11 +19,9 @@ type LogInput struct { // ResolvedLogInput specifies resolved log input. type ResolvedLogInput struct { - LogType string - Lines int - Follow bool - LogFilename string - SerialOutputFilename string + LogType string + Lines int + Follow bool VM *model.VMItem } @@ -69,14 +67,10 @@ func (i *LogInput) Resolve(ctx context.Context, cfg *config.Service, vmRepo vm.R } else { follow, _ = cfg.GetBool(ctx, "settings.vm", "log_follow") } - logFilenameStr, _ := cfg.GetString(ctx, "defaults.firecracker", "log_filename") - serialOutputFilenameStr, _ := cfg.GetString(ctx, "defaults.firecracker", "serial_output_filename") return &ResolvedLogInput{ - VM: vmEntity, - LogType: logType, - Lines: lines, - Follow: follow, - LogFilename: logFilenameStr, - SerialOutputFilename: serialOutputFilenameStr, + VM: vmEntity, + LogType: logType, + Lines: lines, + Follow: follow, }, nil } diff --git a/pkg/api/inputs/vm_create.go b/pkg/api/inputs/vm_create.go index 06acd510..1cbb822e 100644 --- a/pkg/api/inputs/vm_create.go +++ b/pkg/api/inputs/vm_create.go @@ -105,18 +105,8 @@ type ResolvedVMCreateInput struct { DiskSizeMib int LSMFlags string // Firecracker - LogLevel string - LogFilename string - SerialOutputFilename string - MetricsFilename string - APISocketFilename string - PIDFilename string - ConfigFilename string - ConsoleSocketFilename string - ConsolePIDFilename string - VsockFilename string + LogLevel string // Cloud-init - CloudInitISOName string NocloudPortRangeStart int NocloudPortRangeEnd int NocloudMaxPortRetries int @@ -536,16 +526,6 @@ func (r *VMCreateRequest) Resolve(ctx context.Context) (*ResolvedVMCreateInput, userGID, _ := r.cfg.GetInt(ctx, "defaults.vm", "user_gid") guestMACPrefix, _ := r.cfg.GetString(ctx, "defaults.vm", "guest_mac_prefix") logLevel, _ := r.cfg.GetString(ctx, "defaults.firecracker", "log_level") - logFilename, _ := r.cfg.GetString(ctx, "defaults.firecracker", "log_filename") - serialOutputFilename, _ := r.cfg.GetString(ctx, "defaults.firecracker", "serial_output_filename") - metricsFilename, _ := r.cfg.GetString(ctx, "defaults.firecracker", "metrics_filename") - apiSocketFilename, _ := r.cfg.GetString(ctx, "defaults.firecracker", "api_socket_filename") - pidFilename, _ := r.cfg.GetString(ctx, "defaults.firecracker", "pid_filename") - configFilename, _ := r.cfg.GetString(ctx, "defaults.firecracker", "config_filename") - consoleSocketFilename, _ := r.cfg.GetString(ctx, "defaults.firecracker", "console_socket_filename") - consolePIDFilename, _ := r.cfg.GetString(ctx, "defaults.firecracker", "console_pid_filename") - vsockFilename, _ := r.cfg.GetString(ctx, "defaults.firecracker", "vsock_filename") - ciIsoName, _ := r.cfg.GetString(ctx, "defaults.cloudinit", "iso_name") nocloudPortStart, _ := r.cfg.GetInt(ctx, "defaults.cloudinit", "nocloud_port_range_start") nocloudPortEnd, _ := r.cfg.GetInt(ctx, "defaults.cloudinit", "nocloud_port_range_end") nocloudMaxRetries, _ := r.cfg.GetInt(ctx, "defaults.cloudinit", "nocloud_max_port_retries") @@ -601,18 +581,8 @@ func (r *VMCreateRequest) Resolve(ctx context.Context) (*ResolvedVMCreateInput, Volumes: vols, ExtraDrives: extraDrives, // Firecracker defaults - LogLevel: logLevel, - LogFilename: logFilename, - SerialOutputFilename: serialOutputFilename, - MetricsFilename: metricsFilename, - APISocketFilename: apiSocketFilename, - PIDFilename: pidFilename, - ConfigFilename: configFilename, - ConsoleSocketFilename: consoleSocketFilename, - ConsolePIDFilename: consolePIDFilename, - VsockFilename: vsockFilename, + LogLevel: logLevel, // Cloud-init defaults - CloudInitISOName: ciIsoName, NocloudPortRangeStart: nocloudPortStart, NocloudPortRangeEnd: nocloudPortEnd, NocloudMaxPortRetries: nocloudMaxRetries, diff --git a/pkg/api/logs.go b/pkg/api/logs.go index 4bd1c40e..78096876 100644 --- a/pkg/api/logs.go +++ b/pkg/api/logs.go @@ -44,7 +44,7 @@ func (op *Operation) LogStream(ctx context.Context, input inputs.LogInput, callb followCtx, cancel := context.WithCancel(ctx) defer cancel() lineCh, errCh := controller.FollowSync(followCtx, resolved.LogType, - resolved.LogFilename, resolved.SerialOutputFilename) + infra.VMFirecrackerLogFilename, infra.VMFirecrackerConsoleLogFilename) for line := range lineCh { if cbErr := callback(line); cbErr != nil { cancel() @@ -57,7 +57,7 @@ func (op *Operation) LogStream(ctx context.Context, input inputs.LogInput, callb return nil } lines, err := controller.Show(ctx, resolved.LogType, resolved.Lines, - resolved.LogFilename, resolved.SerialOutputFilename) + infra.VMFirecrackerLogFilename, infra.VMFirecrackerConsoleLogFilename) if err != nil { return err } @@ -99,7 +99,7 @@ func (op *Operation) LogStreamChannel( defer close(ec) if resolved.Follow { if err := controller.Follow(ctx, resolved.LogType, ch, - resolved.LogFilename, resolved.SerialOutputFilename); err != nil { + infra.VMFirecrackerLogFilename, infra.VMFirecrackerConsoleLogFilename); err != nil { select { case ec <- err: case <-ctx.Done(): @@ -107,7 +107,7 @@ func (op *Operation) LogStreamChannel( } } else { lines, showErr := controller.Show(ctx, resolved.LogType, resolved.Lines, - resolved.LogFilename, resolved.SerialOutputFilename) + infra.VMFirecrackerLogFilename, infra.VMFirecrackerConsoleLogFilename) if showErr != nil { select { case ec <- showErr: diff --git a/pkg/api/snapshot.go b/pkg/api/snapshot.go index 85d9daf7..e430afb0 100644 --- a/pkg/api/snapshot.go +++ b/pkg/api/snapshot.go @@ -2,6 +2,7 @@ package api import ( "context" + "encoding/hex" "fmt" "log/slog" "os" @@ -100,7 +101,7 @@ func (op *Operation) SnapshotCreate( }() // 5. Copy rootfs - rootfsFile := filepath.Join(snapDir, "rootfs.ext4") + rootfsFile := filepath.Join(snapDir, infra.SnapshotRootfsFilename) emitProgress(onProgress, "rootfs", "running", "Copying root filesystem...") if err := infra.CopyFile(vmItem.RootfsPath, rootfsFile); err != nil { cleanup = true @@ -111,7 +112,7 @@ func (op *Operation) SnapshotCreate( // Create phantom symlink so the vmstate captures a snapshot-local path // instead of the source VM's path. The symlink initially points to the // snapshot's own rootfs copy. - phantomPath := filepath.Join(snapDir, "phantom-rootfs.ext4") + phantomPath := filepath.Join(snapDir, "phantom-rootfs.img") if err := os.Symlink(rootfsFile, phantomPath); err != nil { cleanup = true return nil, errs.WrapMsg(errs.CodeSnapshotCreateFailed, @@ -119,8 +120,8 @@ func (op *Operation) SnapshotCreate( } // 6. Firecracker API operations (pause + snapshot + optionally resume) - memFile := filepath.Join(snapDir, "memory") - stateFile := filepath.Join(snapDir, "vmstate") + memFile := filepath.Join(snapDir, infra.SnapshotMemoryFilename) + stateFile := filepath.Join(snapDir, infra.SnapshotStateFilename) if vmItem.Status == model.VMStatusRunning || vmItem.Status == model.VMStatusPaused { emitProgress(onProgress, "snapshot", "running", "Creating Firecracker snapshot...") @@ -236,6 +237,14 @@ func (op *Operation) SnapshotRestore( return nil, errs.WrapMsg(errs.CodeSnapshotRestoreFailed, fmt.Sprintf("snapshot not found: %s", input.SnapshotID), err) } + snapshotPaths, err := resolveManagedSnapshotArtifacts(op.CacheDir, snap.ID) + if err != nil { + return nil, errs.WrapMsg( + errs.CodeSnapshotRestoreFailed, + "snapshot metadata has an invalid managed identity", + err, + ) + } // 2. Enrich snapshot with relations // Skip "network" when an explicit --network override is provided — @@ -349,8 +358,8 @@ func (op *Operation) SnapshotRestore( } // Copy rootfs from snapshot - rootfsPath := filepath.Join(vmDir, "rootfs.ext4") - if err := infra.CopyFile(snap.RootfsFile, rootfsPath); err != nil { + rootfsPath := filepath.Join(vmDir, infra.VMRootfsFilename) + if err := infra.CopyFile(snapshotPaths.rootfs, rootfsPath); err != nil { for _, cvm := range createdVMs { op.cleanupRestoredVM(ctx, cvm) } @@ -449,8 +458,7 @@ func (op *Operation) SnapshotRestore( // We only need a fresh CID (the source VM still holds the old one) // and the new UDS path for the restored VM's directory. if snap.ExtraConfig != nil && snap.ExtraConfig.VsockPort > 0 && vmItem.RootfsPath != "" { - vsockFilename, _ := op.Services.Config.GetString(ctx, "defaults.firecracker", "vsock_filename") - vsockUDSPath := filepath.Join(vmDir, vsockFilename) + vsockUDSPath := filepath.Join(vmDir, infra.VMVsockSocketFilename) // Use the snapshot's original CID — Firecracker preserves the guest // CID from the vmstate (vsock_override only changes the UDS path). @@ -483,14 +491,10 @@ func (op *Operation) SnapshotRestore( } // Set file paths for the new VM's directory - configFilename, _ := op.Services.Config.GetString(ctx, "defaults.firecracker", "config_filename") - apiSocketFilename, _ := op.Services.Config.GetString(ctx, "defaults.firecracker", "api_socket_filename") - logFilename, _ := op.Services.Config.GetString(ctx, "defaults.firecracker", "log_filename") - serialOutputFilename, _ := op.Services.Config.GetString(ctx, "defaults.firecracker", "serial_output_filename") - vmItem.ConfigPath = filepath.Join(vmDir, configFilename) - vmItem.APISocketPath = filepath.Join(vmDir, apiSocketFilename) - logPath := filepath.Join(vmDir, logFilename) - serialPath := filepath.Join(vmDir, serialOutputFilename) + vmItem.ConfigPath = filepath.Join(vmDir, infra.VMFirecrackerConfigFilename) + vmItem.APISocketPath = filepath.Join(vmDir, infra.VMFirecrackerAPISocketFilename) + logPath := filepath.Join(vmDir, infra.VMFirecrackerLogFilename) + serialPath := filepath.Join(vmDir, infra.VMFirecrackerConsoleLogFilename) vmItem.LogPath = &logPath vmItem.SerialOutputPath = &serialPath @@ -499,7 +503,7 @@ func (op *Operation) SnapshotRestore( _ = op.Repos.VM.Upsert(ctx, vmItem) // Respawn Firecracker in snapshot mode (stays paused) - if err := op.vmRespawnFirecracker(ctx, vmItem, true, snap.SnapshotDir); err != nil { + if err := op.vmRespawnFirecracker(ctx, vmItem, true, snapshotPaths.dir); err != nil { // Don't rollback — VM record exists, just log error slog.Error("Failed to respawn Firecracker for snapshot restore", "vm", name, "error", err) @@ -523,18 +527,18 @@ func (op *Operation) SnapshotRestore( // Load snapshot via Firecracker API // Only attempt API load when the snapshot has actual memory/state files. - snapHasState := snap.MemoryFile != "" && snap.StateFile != "" + snapHasState := true if snapHasState { - if _, err := os.Stat(snap.MemoryFile); err != nil { + if _, err := os.Stat(snapshotPaths.memory); err != nil { snapHasState = false - } else if _, err := os.Stat(snap.StateFile); err != nil { + } else if _, err := os.Stat(snapshotPaths.state); err != nil { snapHasState = false } } if vmItem.APISocketPath != "" && snapHasState { // Acquire lock on snapshot dir to serialize concurrent restores // from the same snapshot (prevents phantom symlink races). - lockPath := filepath.Join(snap.SnapshotDir, ".restore.lock") + lockPath := filepath.Join(snapshotPaths.dir, ".restore.lock") lockFile, lkErr := os.OpenFile(lockPath, os.O_CREATE|os.O_RDWR, 0644) if lkErr == nil { _ = syscall.Flock(int(lockFile.Fd()), syscall.LOCK_EX) @@ -542,7 +546,7 @@ func (op *Operation) SnapshotRestore( // Point phantom symlink to this restore's rootfs copy so // LoadSnapshot finds the correct backing file at the vmstate path. - phantomPath := filepath.Join(snap.SnapshotDir, "phantom-rootfs.ext4") + phantomPath := filepath.Join(snapshotPaths.dir, "phantom-rootfs.img") if err := os.Remove(phantomPath); err != nil && !os.IsNotExist(err) { if lockFile != nil { _ = syscall.Flock(int(lockFile.Fd()), syscall.LOCK_UN) @@ -566,8 +570,8 @@ func (op *Operation) SnapshotRestore( vsockUDSPath = vmItem.Vsock.UDSPath } if err := ctrl.SnapshotRestore(ctx, model.SnapshotRestoreConfig{ - MemFile: snap.MemoryFile, - StateFile: snap.StateFile, + MemFile: snapshotPaths.memory, + StateFile: snapshotPaths.state, Resume: input.Resume, NetworkOverrides: map[string]string{"eth0": vmItem.TapDevice}, VsockUDSPath: vsockUDSPath, @@ -608,6 +612,32 @@ func (op *Operation) SnapshotRestore( return createdVMs, nil } +type managedSnapshotArtifacts struct { + dir string + rootfs string + memory string + state string +} + +// resolveManagedSnapshotArtifacts derives the closed snapshot layout from the +// trusted cache root and snapshot identity. Stored file paths are descriptive +// metadata and cannot select restore inputs. +func resolveManagedSnapshotArtifacts(cacheDir, snapshotID string) (*managedSnapshotArtifacts, error) { + if len(snapshotID) != 64 || snapshotID != strings.ToLower(snapshotID) { + return nil, fmt.Errorf("invalid snapshot identity") + } + if _, err := hex.DecodeString(snapshotID); err != nil { + return nil, fmt.Errorf("invalid snapshot identity: %w", err) + } + dir := filepath.Join(cacheDir, "snapshots", snapshotID) + return &managedSnapshotArtifacts{ + dir: dir, + rootfs: filepath.Join(dir, infra.SnapshotRootfsFilename), + memory: filepath.Join(dir, infra.SnapshotMemoryFilename), + state: filepath.Join(dir, infra.SnapshotStateFilename), + }, nil +} + // cleanupRestoredVM cleans up a VM created during snapshot restore on failure. // All errors are best-effort — we try to clean up as much as possible. func (op *Operation) cleanupRestoredVM(ctx context.Context, v *model.VMItem) { @@ -710,12 +740,19 @@ func (op *Operation) SnapshotRemove(ctx context.Context, input inputs.SnapshotIn } for _, snap := range snapshots { - // Remove files on disk - if snap.SnapshotDir != "" { - if err := os.RemoveAll(snap.SnapshotDir); err != nil { - slog.Warn("failed to remove snapshot directory", - "dir", snap.SnapshotDir, "error", err) - } + paths, err := resolveManagedSnapshotArtifacts(op.CacheDir, snap.ID) + if err != nil { + results = append(results, errs.OperationResult{ + Status: "error", Code: string(errs.CodeSnapshotRemoveFailed), + Item: snap, Message: fmt.Sprintf("invalid managed snapshot identity: %v", err), + }) + continue + } + + // Remove only the ID-derived managed snapshot directory. Stored paths + // are display metadata and cannot select a deletion target. + if err := os.RemoveAll(paths.dir); err != nil { + slog.Warn("failed to remove snapshot directory", "dir", paths.dir, "error", err) } // Remove from DB diff --git a/pkg/api/vm.go b/pkg/api/vm.go index 3925b9c4..88b31b70 100644 --- a/pkg/api/vm.go +++ b/pkg/api/vm.go @@ -366,7 +366,7 @@ func (op *Operation) vmBuilderExecute( return fmt.Errorf("failed to resolve necessary dependencies") } // Validate socket path before any expensive operations (cloud-init, resize, etc.). - if apiSocketPath := filepath.Join(builder.vmDir, resolved.APISocketFilename); len(apiSocketPath) >= 108 { + if apiSocketPath := filepath.Join(builder.vmDir, infra.VMFirecrackerAPISocketFilename); len(apiSocketPath) >= 108 { return fmt.Errorf( "VM ID '%s' produces a socket path that is too long (%d chars, max 107). "+ "This is a system limit for Unix domain sockets. Path: %s", @@ -521,7 +521,6 @@ func (op *Operation) vmBuilderExecute( NocloudNetPort: resolved.NocloudNetPort, CloudInitISOPath: resolved.CloudInitISOPath, KeepCloudInitISO: resolved.KeepCloudInitISO, - CloudInitISOName: resolved.CloudInitISOName, NocloudPortRangeStart: resolved.NocloudPortRangeStart, NocloudPortRangeEnd: resolved.NocloudPortRangeEnd, NocloudMaxPortRetries: resolved.NocloudMaxPortRetries, @@ -570,7 +569,6 @@ func (op *Operation) vmBuilderExecute( NocloudNetPort: resolved.NocloudNetPort, CloudInitISOPath: resolved.CloudInitISOPath, KeepCloudInitISO: resolved.KeepCloudInitISO, - CloudInitISOName: resolved.CloudInitISOName, NocloudPortRangeStart: resolved.NocloudPortRangeStart, NocloudPortRangeEnd: resolved.NocloudPortRangeEnd, NocloudMaxPortRetries: resolved.NocloudMaxPortRetries, @@ -616,7 +614,7 @@ func (op *Operation) vmBuilderExecute( if resolved.VsockPort > 0 { builder.vsockPort = resolved.VsockPort builder.vsockToken = crypto.UUIDV4() - builder.vsockUDSPath = filepath.Join(builder.vmDir, resolved.VsockFilename) + builder.vsockUDSPath = filepath.Join(builder.vmDir, infra.VMVsockSocketFilename) if agentBin := vsock.AgentBinary(); len(agentBin) > 0 { if err := backend.InjectAgent(ctx, agentBin, builder.vsockPort, builder.vsockToken); err != nil { provisionErr = fmt.Errorf("inject agent: %w", err) @@ -675,8 +673,7 @@ func (op *Operation) vmBuilderExecute( firecrackerReady = true // Console relay setup (before spawn) if resolved.EnableConsole { - consoleCtrl := console.NewController(builder.vmID, builder.vmDir, builder.name, - resolved.ConsolePIDFilename, resolved.ConsoleSocketFilename) + consoleCtrl := console.NewController(builder.vmID, builder.vmDir, builder.name) ptyFD, ptyErr := consoleCtrl.CreatePTY() if ptyErr != nil { fcErr = fmt.Errorf("console PTY creation failed: %w", ptyErr) @@ -1199,8 +1196,9 @@ func (op *Operation) vmRespawnFirecracker( if v.KernelID == "" || v.Kernel == nil || v.Kernel.Path == "" { return fmt.Errorf("kernel info is required for VM respawn") } - if v.RootfsPath == "" { - return fmt.Errorf("rootfs path is required for VM respawn") + managedRootfsPath := filepath.Join(vmDir, infra.VMRootfsFilename) + if _, err := os.Stat(managedRootfsPath); err != nil { + return fmt.Errorf("managed VM rootfs is unavailable: %w", err) } if v.ImageID != "" && v.Image == nil { return fmt.Errorf("image info is required for VM respawn") @@ -1247,56 +1245,11 @@ func (op *Operation) vmRespawnFirecracker( libnet.FlushARP(ctx, v.Network.Bridge) } logLevel, _ := op.Services.Config.GetString(ctx, "defaults.firecracker", "log_level") - fcPIDFilename, _ := op.Services.Config.GetString(ctx, "defaults.firecracker", "pid_filename") - fcConfig := &model.FirecrackerConfig{ - VMDir: vmDir, - RootfsPath: v.RootfsPath, - BinaryPath: v.Binary.Path, - JailerPath: pair.Jailer.Path, - BinaryVersion: pair.Firecracker.Version, - KernelPath: v.Kernel.Path, - VCPUCount: v.VCPUCount, - MemSizeMiB: v.MemSizeMiB, - CgroupLimits: v.CgroupLimits, - GuestIP: v.IPv4, - GuestMAC: v.MAC, - TapName: v.TapDevice, - NetworkGateway: v.Network.IPv4Gateway, - PCIEnabled: v.PCIEnabled, - NestedVirt: v.NestedVirt, - EnableConsole: v.EnableConsole, - EnableLogging: v.EnableLogging, - EnableMetrics: v.EnableMetrics, - BootArgs: v.BootArgs, - LSMFlags: v.LSMFlags, - SnapshotMode: snapshotMode, - SnapshotDir: snapshotDir, - ImageFSUUID: v.Image.FSUUID, - ImageFSType: v.Image.FSType, - // Full paths from DB (field names match DB column names) - ConfigPath: v.ConfigPath, - APISocketPath: v.APISocketPath, - LogPath: infra.DerefOrZero(v.LogPath), - SerialOutputPath: infra.DerefOrZero(v.SerialOutputPath), - LogLevel: logLevel, - PIDPath: filepath.Join(vmDir, fcPIDFilename), - } - // --- Vsock config --- - if v.Vsock != nil { - fcConfig.Vsock = &model.VsockConfig{ - GuestCID: v.Vsock.GuestCID, - UDSPath: v.Vsock.UDSPath, - } - } - // --- Attach volumes (extra drives) --- - if len(v.Volumes) > 0 { - fcConfig.ExtraDrives = volume.VolumesToDrives(v.Volumes) - } + fcConfig := buildRespawnFirecrackerConfig(v, pair, vmDir, snapshotMode, snapshotDir, logLevel) // --- Console relay setup (before spawn) --- var consoleController *console.Controller if v.EnableConsole { - consoleController = console.NewController(v.ID, vmDir, v.Name, - v.ConsolePIDFilename, v.ConsoleSocketFilename) + consoleController = console.NewController(v.ID, vmDir, v.Name) ptyFD, ptyErr := consoleController.CreatePTY() if ptyErr != nil { slog.Warn("Console PTY creation failed during respawn", "vm", v.Name, "error", ptyErr) @@ -1346,6 +1299,61 @@ func (op *Operation) vmRespawnFirecracker( return nil } +// buildRespawnFirecrackerConfig reconstructs launch paths from the managed VM +// directory. Persisted path columns are descriptive metadata, never authority. +func buildRespawnFirecrackerConfig( + v *model.VMItem, + pair *model.BinaryPair, + vmDir string, + snapshotMode bool, + snapshotDir string, + logLevel string, +) *model.FirecrackerConfig { + config := &model.FirecrackerConfig{ + VMDir: vmDir, + RootfsPath: filepath.Join(vmDir, infra.VMRootfsFilename), + BinaryPath: v.Binary.Path, + JailerPath: pair.Jailer.Path, + BinaryVersion: pair.Firecracker.Version, + KernelPath: v.Kernel.Path, + VCPUCount: v.VCPUCount, + MemSizeMiB: v.MemSizeMiB, + CgroupLimits: v.CgroupLimits, + GuestIP: v.IPv4, + GuestMAC: v.MAC, + TapName: v.TapDevice, + NetworkGateway: v.Network.IPv4Gateway, + PCIEnabled: v.PCIEnabled, + NestedVirt: v.NestedVirt, + EnableConsole: v.EnableConsole, + EnableLogging: v.EnableLogging, + EnableMetrics: v.EnableMetrics, + BootArgs: v.BootArgs, + LSMFlags: v.LSMFlags, + SnapshotMode: snapshotMode, + SnapshotDir: snapshotDir, + ImageFSUUID: v.Image.FSUUID, + ImageFSType: v.Image.FSType, + ConfigPath: filepath.Join(vmDir, infra.VMFirecrackerConfigFilename), + APISocketPath: filepath.Join(vmDir, infra.VMFirecrackerAPISocketFilename), + LogPath: filepath.Join(vmDir, infra.VMFirecrackerLogFilename), + SerialOutputPath: filepath.Join(vmDir, infra.VMFirecrackerConsoleLogFilename), + MetricsPath: filepath.Join(vmDir, infra.VMFirecrackerMetricsFilename), + LogLevel: logLevel, + PIDPath: filepath.Join(vmDir, infra.VMFirecrackerPIDFilename), + } + if v.Vsock != nil { + config.Vsock = &model.VsockConfig{ + GuestCID: v.Vsock.GuestCID, + UDSPath: filepath.Join(vmDir, infra.VMVsockSocketFilename), + } + } + if len(v.Volumes) > 0 { + config.ExtraDrives = volume.VolumesToDrives(v.Volumes) + } + return config +} + func (op *Operation) resolveVMCgroupLimits(ctx context.Context, vcpuCount, memoryMiB int) model.VMCgroupLimits { headroomMiB, _ := op.Services.Config.GetInt(ctx, "defaults.vm", "cgroup_vmm_headroom_mib") cpuWeight, _ := op.Services.Config.GetInt(ctx, "defaults.vm", "cgroup_cpu_weight") @@ -1525,7 +1533,7 @@ func (c *VMCreateBuilder) cloneImage( if fsType == "" { return fmt.Errorf("fsType is required") } - vmRootfsPath := filepath.Join(c.vmDir, "rootfs."+fsType) + vmRootfsPath := filepath.Join(c.vmDir, infra.VMRootfsFilename) if _, err := imageSvc.EnsureCached([]*model.ImageItem{resolved.Image}); err != nil { return fmt.Errorf("ensure cached image: %w", err) } @@ -1590,14 +1598,13 @@ func (c *VMCreateBuilder) buildFirecrackerConfig() *model.FirecrackerConfig { LogLevel: c.resolved.LogLevel, ExtraDrives: c.resolved.ExtraDrives, Writeback: c.resolved.Writeback, - // Full paths constructed from VMDir + resolved filenames. - // Field names match DB column names for respawn compatibility. - ConfigPath: filepath.Join(c.vmDir, c.resolved.ConfigFilename), - APISocketPath: filepath.Join(c.vmDir, c.resolved.APISocketFilename), - LogPath: filepath.Join(c.vmDir, c.resolved.LogFilename), - SerialOutputPath: filepath.Join(c.vmDir, c.resolved.SerialOutputFilename), - MetricsPath: filepath.Join(c.vmDir, c.resolved.MetricsFilename), - PIDPath: filepath.Join(c.vmDir, c.resolved.PIDFilename), + // Fixed managed paths; field names match DB columns used during respawn. + ConfigPath: filepath.Join(c.vmDir, infra.VMFirecrackerConfigFilename), + APISocketPath: filepath.Join(c.vmDir, infra.VMFirecrackerAPISocketFilename), + LogPath: filepath.Join(c.vmDir, infra.VMFirecrackerLogFilename), + SerialOutputPath: filepath.Join(c.vmDir, infra.VMFirecrackerConsoleLogFilename), + MetricsPath: filepath.Join(c.vmDir, infra.VMFirecrackerMetricsFilename), + PIDPath: filepath.Join(c.vmDir, infra.VMFirecrackerPIDFilename), } // Cloud-init info from result (isoPath, nocloudURL) if c.cloudInitResult != nil { @@ -1628,8 +1635,8 @@ func (c *VMCreateBuilder) toVMModel() *model.VMItem { return nil } now := time.Now().Format(time.RFC3339) - logPath := filepath.Join(c.vmDir, c.resolved.LogFilename) - serialPath := filepath.Join(c.vmDir, c.resolved.SerialOutputFilename) + logPath := filepath.Join(c.vmDir, infra.VMFirecrackerLogFilename) + serialPath := filepath.Join(c.vmDir, infra.VMFirecrackerConsoleLogFilename) vm := &model.VMItem{ ID: c.vmID, Name: c.resolved.Name, @@ -1648,8 +1655,8 @@ func (c *VMCreateBuilder) toVMModel() *model.VMItem { MemSizeMiB: c.resolved.MemSizeMib, CgroupLimits: c.resolved.CgroupLimits, DiskSizeMiB: c.resolved.DiskSizeMib, - APISocketPath: filepath.Join(c.vmDir, c.resolved.APISocketFilename), - ConfigPath: filepath.Join(c.vmDir, c.resolved.ConfigFilename), + APISocketPath: filepath.Join(c.vmDir, infra.VMFirecrackerAPISocketFilename), + ConfigPath: filepath.Join(c.vmDir, infra.VMFirecrackerConfigFilename), CloudInitMode: string(c.resolved.CloudInitMode), RootfsPath: c.rootfsPath, RootfsSuffix: c.resolved.Image.FSType, diff --git a/pkg/api/vm_resource_paths_test.go b/pkg/api/vm_resource_paths_test.go new file mode 100644 index 00000000..3d86bf83 --- /dev/null +++ b/pkg/api/vm_resource_paths_test.go @@ -0,0 +1,137 @@ +package api + +import ( + "os" + "path/filepath" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "mvmctl/internal/core/image" + "mvmctl/internal/infra" + "mvmctl/internal/lib/model" + "mvmctl/pkg/api/inputs" +) + +// Rationale: the managed rootfs leaf is part of the privileged resource +// contract. The producer must not derive it from the image filesystem type. +func TestVMCreateBuilderCloneImageUsesFixedRootfsLeaf(t *testing.T) { + t.Setenv(infra.EnvKey("TEMP_DIR"), t.TempDir()) + + sourcePath := filepath.Join(t.TempDir(), "source.ext4") + require.NoError(t, os.WriteFile(sourcePath, []byte("rootfs"), 0600)) + + resolved := &inputs.ResolvedVMCreateInput{Image: &model.ImageItem{ + ID: "fixed-rootfs-test", + Path: sourcePath, + FSType: "ext4", + }} + vmDir := t.TempDir() + builder := &VMCreateBuilder{vmDir: vmDir} + + err := builder.cloneImage(t.Context(), image.NewService(nil), resolved) + require.NoError(t, err) + + wantPath := filepath.Join(vmDir, infra.VMRootfsFilename) + assert.Equal(t, wantPath, builder.rootfsPath) + data, err := os.ReadFile(wantPath) + require.NoError(t, err) + assert.Equal(t, []byte("rootfs"), data) + assert.NoFileExists(t, filepath.Join(vmDir, "rootfs.ext4")) +} + +// Rationale: every Firecracker artifact path passed into the launch boundary +// must be derived from the VM directory and the closed filename vocabulary. +func TestVMCreateBuilderUsesFixedFirecrackerLeaves(t *testing.T) { + vmDir := t.TempDir() + builder := &VMCreateBuilder{ + vmDir: vmDir, + resolved: &inputs.ResolvedVMCreateInput{ + Binary: &model.BinaryItem{}, + Jailer: &model.BinaryItem{}, + Kernel: &model.KernelItem{}, + Image: &model.ImageItem{}, + Network: &model.NetworkItem{}, + }, + } + + config := builder.buildFirecrackerConfig() + require.NotNil(t, config) + + assert.Equal(t, filepath.Join(vmDir, infra.VMFirecrackerConfigFilename), config.ConfigPath) + assert.Equal(t, filepath.Join(vmDir, infra.VMFirecrackerAPISocketFilename), config.APISocketPath) + assert.Equal(t, filepath.Join(vmDir, infra.VMFirecrackerLogFilename), config.LogPath) + assert.Equal(t, filepath.Join(vmDir, infra.VMFirecrackerConsoleLogFilename), config.SerialOutputPath) + assert.Equal(t, filepath.Join(vmDir, infra.VMFirecrackerMetricsFilename), config.MetricsPath) + assert.Equal(t, filepath.Join(vmDir, infra.VMFirecrackerPIDFilename), config.PIDPath) +} + +// Rationale: persisted path columns are display metadata, not launch +// authority. Relaunch must reconstruct every managed path from the VM ID. +func TestBuildRespawnFirecrackerConfigIgnoresPersistedPaths(t *testing.T) { + vmDir := t.TempDir() + logPath := "/user-selected/firecracker.log" + serialPath := "/user-selected/console.log" + vmItem := &model.VMItem{ + RootfsPath: "/user-selected/rootfs.img", + ConfigPath: "/user-selected/config.json", + APISocketPath: "/user-selected/api.socket", + LogPath: &logPath, + SerialOutputPath: &serialPath, + EnableMetrics: true, + Image: &model.ImageItem{}, + Binary: &model.BinaryItem{}, + Kernel: &model.KernelItem{}, + Network: &model.NetworkItem{}, + Vsock: &model.VsockConfigItem{ + GuestCID: 42, + UDSPath: "/user-selected/vsock.socket", + }, + } + pair := &model.BinaryPair{ + Firecracker: &model.BinaryItem{Version: "1.16.0"}, + Jailer: &model.BinaryItem{}, + } + + config := buildRespawnFirecrackerConfig(vmItem, pair, vmDir, false, "", "Info") + require.NotNil(t, config) + + assert.Equal(t, filepath.Join(vmDir, infra.VMRootfsFilename), config.RootfsPath) + assert.Equal(t, filepath.Join(vmDir, infra.VMFirecrackerConfigFilename), config.ConfigPath) + assert.Equal(t, filepath.Join(vmDir, infra.VMFirecrackerAPISocketFilename), config.APISocketPath) + assert.Equal(t, filepath.Join(vmDir, infra.VMFirecrackerLogFilename), config.LogPath) + assert.Equal(t, filepath.Join(vmDir, infra.VMFirecrackerConsoleLogFilename), config.SerialOutputPath) + assert.Equal(t, filepath.Join(vmDir, infra.VMFirecrackerMetricsFilename), config.MetricsPath) + assert.Equal(t, filepath.Join(vmDir, infra.VMFirecrackerPIDFilename), config.PIDPath) + require.NotNil(t, config.Vsock) + assert.Equal(t, filepath.Join(vmDir, infra.VMVsockSocketFilename), config.Vsock.UDSPath) +} + +// Rationale: user-owned snapshot metadata must not select an artifact path; +// exact leaves are derived from the validated snapshot identity. +func TestResolveManagedSnapshotArtifacts(t *testing.T) { + cacheDir := t.TempDir() + snapshotID := "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" + + paths, err := resolveManagedSnapshotArtifacts(cacheDir, snapshotID) + require.NoError(t, err) + + wantDir := filepath.Join(cacheDir, "snapshots", snapshotID) + assert.Equal(t, wantDir, paths.dir) + assert.Equal(t, filepath.Join(wantDir, infra.SnapshotRootfsFilename), paths.rootfs) + assert.Equal(t, filepath.Join(wantDir, infra.SnapshotMemoryFilename), paths.memory) + assert.Equal(t, filepath.Join(wantDir, infra.SnapshotStateFilename), paths.state) +} + +func TestResolveManagedSnapshotArtifactsRejectsInvalidIdentity(t *testing.T) { + for name, snapshotID := range map[string]string{ + "path traversal": "../selected-by-database", + "uppercase hex": "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA", + } { + t.Run(name, func(t *testing.T) { + _, err := resolveManagedSnapshotArtifacts(t.TempDir(), snapshotID) + require.ErrorContains(t, err, "invalid snapshot identity") + }) + } +} From c5f04b59fa2188e2767a88da9d7fe3a579285f8f Mon Sep 17 00:00:00 2001 From: AlanD20 Date: Fri, 28 Aug 2026 22:39:29 +0200 Subject: [PATCH 0047/1457] docs: document fixed managed resource names --- CHANGELOG.md | 16 +++ docs/REFERENCES.md | 16 ++- docs/RESOURCE_MANAGEMENTS.md | 7 +- docs/implementations/SNAPSHOT_DOMAIN.md | 124 +++++++++++++----- docs/improvements/IMPROVEMENTS_008.md | 20 ++- .../MOVING_TO_AGNOSTIC_ARCHITECTURE.md | 4 +- tasks/todo.md | 19 ++- 7 files changed, 155 insertions(+), 51 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 1544018c..d115d0d3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -134,6 +134,17 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Kernel download, build, and import now stage files under the kernel service's configured managed directory and install final artifacts as `kernels/`. Import identity is computed from the receiver-owned staged bytes, so a caller-controlled display name or a source-file replacement cannot select or escape the managed destination. +- VM artifacts now use a compile-time filename vocabulary: `rootfs.img`, `cloud-init.iso`, `firecracker.json`, + `firecracker.log`, `firecracker.console.log`, optional `firecracker.metrics`, and fixed API/vsock/console socket and + PID basenames. VM filesystem type no longer changes the managed rootfs leaf. +- Custom cloud-init ISO paths remain accepted as source data, but the ISO is copied into the VM's fixed + `cloud-init.iso` leaf before launch. Internal console-relay filename flags and filename settings were removed. +- VM relaunch reconstructs rootfs, configuration, output, metrics, API, PID, and vsock paths from the managed VM + directory instead of treating SQLite path columns as launch authority. The transitional privileged Jailer also + rejects alternate manifest basenames and constructs its Firecracker API/config arguments from the compiled names. +- Snapshots now produce and restore `rootfs.img`, `memory`, and `vmstate`. Restore and removal derive the snapshot + directory and leaves from the validated 64-character lowercase snapshot ID rather than stored file-path columns. + The temporary `phantom-rootfs.img` mechanism remains until the private mount-namespace overlay lands. #### Network isolation - Traffic routed between different managed networks is now denied unless an explicit typed service-access policy permits @@ -162,6 +173,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 flow. Self-update remains available for user-owned candidate artifacts. - API consumers can no longer set `output_dir`, `name`, or `output_path` on `KernelPullInput`; managed kernel placement is receiver-owned. The public CLI did not expose equivalent flags. +- Firecracker/cloud-init/console filename settings are removed. Existing `rootfs.`, alternate snapshot leaf, + or custom runtime filename layouts are not adopted; clean-install VMs and snapshots use the fixed v0.3 names. ### Fixed @@ -193,6 +206,9 @@ The following items are release blockers and are intentionally not described abo - Split root-only locks/handshakes beneath `/run/mvmctl/authority` from caller-owned socket/PID output directories beneath `/run/mvmctl/runtime`; stop mounting the whole cache VM directory. Pin persistent config/output leaves individually, truncate enabled `0600` log/console/metrics files only after durable launch registration, and retain them after stop. +- Treat the current fixed-name Jailer checks as transitional defense-in-depth only: Firecracker still reads a + caller-writable configuration from the whole mounted VM directory. The final receiver must consume pinned individual + resources and derive/verify every jail-visible path after durable registration. - Independently verify and atomically install trusted Firecracker/Jailer releases inside the privileged boundary. - Migrate Jailer, loopmount, network, firewall, and supported host mutations to distinct typed privileged actions; remove the public root `mvm run jailer` and `mvm run provision` entry points. diff --git a/docs/REFERENCES.md b/docs/REFERENCES.md index 150e700e..8d2e25be 100644 --- a/docs/REFERENCES.md +++ b/docs/REFERENCES.md @@ -748,20 +748,26 @@ The security model applies to the net cloud-init mode only: ├── volumes/ # Persistent disk volume files ├── vms/ # Per-VM state │ └── / # VM directories named by SHA256 hash -│ ├── rootfs.ext4 (or rootfs.btrfs, rootfs.xfs) +│ ├── rootfs.img +│ ├── cloud-init.iso # ISO mode only │ ├── firecracker.json │ ├── firecracker.log │ ├── firecracker.console.log +│ ├── firecracker.metrics # Metrics enabled only │ ├── firecracker.pid │ ├── firecracker.api.socket -│ ├── firecracker.metrics +│ ├── vsock.sock # Vsock enabled only │ ├── console.sock │ ├── console.pid -│ └── cloud-init/ +│ └── cloud-init/ # Generated seed inputs ├── workflows/ # Workflow state persistence ├── nocloudnet/ # nocloud-net batch server dirs and logs ├── snapshots/ # Snapshot files (mem, vmstate, disk) │ └── / +│ ├── rootfs.img +│ ├── memory +│ ├── vmstate +│ └── phantom-rootfs.img # Transitional until private namespace overlay ├── logs/ # Application log files ├── firecracker-src/ # Firecracker git clone (for building from source) ├── mvmdb.db # SQLite database (canonical asset state) @@ -771,6 +777,10 @@ The security model applies to the net cloud-init mode only: └── ... ``` +The VM and snapshot leaf names above are compiled and are not configurable. `MVM_CACHE_DIR` still selects the managed +cache root. API/vsock/console sockets and PID mirrors remain in the VM cache directory in the transitional Jailer path; +v0.3 release work will move them to `/run/mvmctl/runtime//` and stop mounting the whole VM directory. + Canonical launch assets and jail state are outside the user cache: ```text diff --git a/docs/RESOURCE_MANAGEMENTS.md b/docs/RESOURCE_MANAGEMENTS.md index 0033a462..c7bf50e6 100644 --- a/docs/RESOURCE_MANAGEMENTS.md +++ b/docs/RESOURCE_MANAGEMENTS.md @@ -263,7 +263,8 @@ kernel-firecracker: **Location:** `internal/infra/constants.go` — `OverridableDefaults` map -This map is the single authoritative source for all built-in defaults. Hardcoded values anywhere else in the codebase are a bug. +This map is the authoritative source for user-configurable built-in defaults. Security-sensitive managed artifact +basenames are compile-time constants in the same file and intentionally do not appear in `OverridableDefaults`. ### Structure overview @@ -273,8 +274,8 @@ This map is the single authoritative source for all built-in defaults. Hardcoded | `defaults.network` | Default bridge name, CIDR, NAT enabled | | `defaults.image` | Import format, remote listing limit and cache TTL | | `defaults.kernel` | Default kernel version, build jobs, remote listing limit and cache TTL | -| `defaults.firecracker` | Log filenames, socket filenames, log level, PID filenames | -| `defaults.cloudinit` | ISO name, nocloud-net port range, max retries, kill-after duration | +| `defaults.firecracker` | Firecracker log level | +| `defaults.cloudinit` | nocloud-net port range, max retries, kill-after duration | | `defaults.binary` | Remote version limit | | `defaults.volume` | Cache type (default: `Unsafe`) | | `settings` | General settings: guestfs_enabled, firewall_backend | diff --git a/docs/implementations/SNAPSHOT_DOMAIN.md b/docs/implementations/SNAPSHOT_DOMAIN.md index 2eaaeae5..b747936f 100644 --- a/docs/implementations/SNAPSHOT_DOMAIN.md +++ b/docs/implementations/SNAPSHOT_DOMAIN.md @@ -1,18 +1,65 @@ # Snapshot Functionality +This document explains how mvmctl creates, restores, persists, and removes Firecracker snapshots, including the +transitional behavior that must be replaced before v0.3 release signoff. + +## Table of Contents + +- [Problem](#problem) +- [Architecture](#architecture) + - [Key characteristics](#key-characteristics) +- [Entry point](#entry-point) +- [Happy path: Snapshot create](#happy-path-snapshot-create) + - [1. Resolve and enrich](#1-resolve-and-enrich) + - [2. Generate snapshot ID](#2-generate-snapshot-id) + - [3. Create snapshot directory](#3-create-snapshot-directory) + - [4. Copy rootfs](#4-copy-rootfs) + - [5. Create phantom symlink](#5-create-phantom-symlink) + - [6. Pause VM and patch drive path](#6-pause-vm-and-patch-drive-path) + - [7. Create snapshot](#7-create-snapshot) + - [8. Restore original drive path and resume](#8-restore-original-drive-path-and-resume) + - [9. Insert DB record](#9-insert-db-record) +- [Happy path: Snapshot restore](#happy-path-snapshot-restore) + - [1. Resolve snapshot](#1-resolve-snapshot) + - [2. Load metadata](#2-load-metadata) + - [3. Derive managed artifacts](#3-derive-managed-artifacts) + - [4. For each VM to restore](#4-for-each-vm-to-restore) + - [Phantom symlink](#phantom-symlink) + - [Concurrent restore safety](#concurrent-restore-safety) + - [Vsock override](#vsock-override) + - [Network identity](#network-identity) +- [DB schema](#db-schema) +- [Snapshot config structs](#snapshot-config-structs) +- [Failure modes](#failure-modes) + - [Firecracker constraints on restore](#firecracker-constraints-on-restore) + - [Partial failure on create](#partial-failure-on-create) + - [Stale restore lock](#stale-restore-lock) + - [Snapshot removal does not affect VMs](#snapshot-removal-does-not-affect-vms) + - [Reference counting for delete protection](#reference-counting-for-delete-protection) +- [Key files](#key-files) +- [Design decisions](#design-decisions) + ## Problem Firecracker snapshots are a raw API passthrough. Without a managed snapshot domain, there is no way to list available snapshots, track which VM a snapshot came from, or restore a snapshot into a new VM with proper network identity. The user must specify raw file paths for memory and state dumps with no directory management, no rootfs tracking, and no metadata persistence. ## Architecture -The snapshot domain has its own DB table (`snapshots`), cache directory (`~/.cache/mvm/snapshots//`), CLI commands, and API layer. Snapshots are managed entities — captured atomically, stored in a known location, restorable by name. +The snapshot domain has its own DB table (`snapshots`), cache directory (`~/.cache/mvmctl/snapshots//`), CLI +commands, and API layer. Snapshots are managed entities stored in a known location and restorable by name. Failed +creates remove the partial directory and do not retain a DB row; running-VM disk/memory crash consistency is still +pending as described below. ### Key characteristics -- **No Controller** — snapshots are immutable after creation, so no state machine is needed. The repository handles DB CRUD, and filesystem operations are handled directly by the API layer. -- **Self-contained domain** — the snapshot domain imports `internal/lib/firecracker` to call Pause/CreateSnapshot/Resume/LoadSnapshot directly. No cross-core package imports. -- **Orchestration in API layer** — the API layer's `Operation` methods coordinate VM pausing, rootfs copying, network allocation, and Firecracker spawn. This matches the existing pattern where the API layer orchestrates multiple domains. +- **No snapshot Controller** — the public snapshot lifecycle has no mutable state machine. The repository handles DB + CRUD, while the API layer coordinates filesystem and VM operations. The current restore path does temporarily mutate + `.restore.lock` and `phantom-rootfs.img`; Task 7 removes that transitional mechanism. +- **Self-contained core domain** — `internal/core/snapshot/` owns repository and resolution behavior and does not import + another core domain. +- **Orchestration in API layer** — the API layer coordinates VM pausing, rootfs copying, network allocation, Jailer + launch, and snapshot API calls through the VM controller. This matches the rule that only `pkg/api/` orchestrates + multiple core domains. ## Entry point @@ -24,7 +71,9 @@ Snapshot operations are triggered from the CLI commands in `internal/cli/snapsho - `mvm snapshot inspect ` — calls `op.SnapshotInspect()` - `mvm snapshot rm ` — calls `op.SnapshotRemove()` -The API layer orchestrates all cross-domain operations (VM pause/resume, rootfs copy, network allocation, Firecracker spawn). The snapshot repository in `internal/core/snapshot/` handles DB CRUD. There is no service layer — snapshots are immutable after creation. +The API layer orchestrates all cross-domain operations (VM pause/resume, rootfs copy, network allocation, and +Firecracker spawn). The snapshot repository in `internal/core/snapshot/` handles DB CRUD and resolution. There is no +snapshot service layer. ## Happy path: Snapshot create @@ -38,15 +87,20 @@ A deterministic snapshot ID is generated via `crypto.SnapshotID(sourceVMID, time ### 3. Create snapshot directory -The snapshot cache directory is created at `~/.cache/mvm/snapshots//`. +The snapshot cache directory is created at `~/.cache/mvmctl/snapshots//` (or beneath `MVM_CACHE_DIR`). ### 4. Copy rootfs -The source VM's rootfs is copied to `snapDir/rootfs.ext4` via `infra.CopyFile()` (file-level copy with sparse support). This gives a point-in-time consistent rootfs — the source VM continues running, so referencing the original path risks inconsistency. +The source VM's rootfs is copied to the fixed `snapDir/rootfs.img` leaf via `infra.CopyFile()`. The leaf name is +independent of the guest filesystem type. The current implementation performs this copy before pausing a running VM; +it therefore does not yet guarantee a crash-consistent disk/memory pair. Snapshot crash consistency remains a v0.3 +release blocker. ### 5. Create phantom symlink -A symlink `snapDir/phantom-rootfs.ext4` → `rootfs.ext4` is created. This symlink is critical for making snapshots independent of the source VM's rootfs path. +A transitional symlink `snapDir/phantom-rootfs.img` → `rootfs.img` is created so the captured vmstate can refer to a +snapshot-local backing path. ADR-0016 requires Task 7 to replace this persistent symlink with a private mount-namespace +overlay before v0.3 release signoff. ### 6. Pause VM and patch drive path @@ -76,26 +130,31 @@ The snapshot is resolved by ID (supports prefix matching, errors on ambiguity). Snapshot metadata is loaded from the DB, including kernel/network/binary IDs. -### 3. For each VM to restore +### 3. Derive managed artifacts + +Before creating a VM, restore validates the stored snapshot ID as exactly 64 lowercase hexadecimal characters and +derives `snapshots//{rootfs.img,memory,vmstate}` from the configured cache root. Stored path columns are display +metadata and do not select restore or removal targets. + +### 4. For each VM to restore For each clone (controlled by `--count`, default 1): -1. Generate new VM ID -2. Copy `snapDir/rootfs.ext4` → `vms//rootfs.ext4` -3. Acquire exclusive flock on `snapDir/.restore.lock` (serializes concurrent restores from the same snapshot) -4. Replace `snapDir/phantom-rootfs.ext4` → symlink → new VM's rootfs -5. Create VM record with Stopped status, wired kernel/binary/network -6. Create VM directory and Firecracker config -7. Spawn Firecracker in snapshot mode -8. Call `firecracker.LoadSnapshot()` with mem/state paths, network overrides, and vsock override -9. Release flock on `.restore.lock` +1. Generate the new VM ID and copy `snapDir/rootfs.img` to `vms//rootfs.img`. +2. Allocate the network lease/TAP and persist the stopped VM record with its kernel, image, binary, and network identity. +3. Derive the VM's fixed Firecracker/vsock paths and spawn Firecracker in snapshot mode. +4. Acquire an exclusive flock on `snapDir/.restore.lock` (serializes concurrent restores from the same snapshot). +5. Replace `snapDir/phantom-rootfs.img` with a symlink to the restored VM's jailed `/rootfs` mount. +6. Call `firecracker.LoadSnapshot()` with the fixed memory/state paths, network overrides, and vsock override. +7. Release the flock on `.restore.lock`. ### Phantom symlink The phantom symlink makes snapshots independent of the source VM: -- **During create**: PATCH the running VM's drive to point to the symlink before taking the snapshot. The vmstate captures the snapshot-local symlink path. -- **During restore**: Replace the symlink to point to the new VM's rootfs copy. LoadSnapshot follows the symlink and finds the correct backing file. +- **During create**: PATCH the running VM's drive to point to `phantom-rootfs.img` before taking the snapshot. The + vmstate captures that snapshot-local path. +- **During restore**: replace the symlink so the jailed load resolves the restored VM's `/rootfs` mount. Firecracker's `PUT /snapshot/load` does not support a block-device-path override — the backing file path recorded in the vmstate is used directly. The phantom symlink works around this constraint. @@ -105,7 +164,8 @@ The `.restore.lock` file serializes concurrent `mvm snapshot restore` invocation ### Vsock override -Firecracker's `PUT /snapshot/load` supports `vsock_override` to change the vsock UDS path at load time. Without it, the vmstate's recorded UDS path would collide with the source VM's socket. The vsock guest CID is set via a separate `PUT /vsock` call after LoadSnapshot. +Firecracker's `PUT /snapshot/load` supports `vsock_override` to change the host UDS path at load time. The snapshot's +guest CID is preserved; the implementation does not issue a separate post-load `PUT /vsock`. ### Network identity @@ -121,10 +181,10 @@ The `snapshots` table (in `001_initial_schema.sql`) stores: | `name` | TEXT | User-provided name (defaults to `-`) | | `source_vm_id` | TEXT | Source VM ID | | `source_vm_name` | TEXT | Source VM name (denormalized for display) | -| `snapshot_dir` | TEXT | Absolute path to `cache/snapshots//` | -| `memory_file` | TEXT | Path to memory dump file within snapshot dir | -| `state_file` | TEXT | Path to vmstate file within snapshot dir | -| `rootfs_file` | TEXT | Path to rootfs copy within snapshot dir | +| `snapshot_dir` | TEXT | Derived display path to `cache/snapshots//` | +| `memory_file` | TEXT | Derived display path to the fixed `memory` leaf | +| `state_file` | TEXT | Derived display path to the fixed `vmstate` leaf | +| `rootfs_file` | TEXT | Derived display path to the fixed `rootfs.img` leaf | | `kernel_id` | TEXT | Kernel ID used at snapshot time | | `network_id` | TEXT | Network ID used at snapshot time | | `binary_id` | TEXT | Firecracker binary ID used at snapshot time | @@ -194,20 +254,24 @@ Before deleting a kernel, network, or binary, the API layer checks `snapshotRepo | `internal/core/snapshot/resolver.go` | Entity resolution by identifier | | `internal/lib/model/snapshot.go` | `SnapshotItem`, `SnapshotCreateConfig`, `SnapshotRestoreConfig` | | `pkg/api/snapshot.go` | API orchestration: `SnapshotCreate()`, `SnapshotRestore()`, `SnapshotList()`, `SnapshotInspect()`, `SnapshotRemove()` | -| `pkg/api/inputs/snapshot_input.go` | Input structs: `SnapshotCreateInput`, `SnapshotRestoreInput`, `SnapshotInput` | +| `pkg/api/inputs/snapshot.go` | Input structs: `SnapshotCreateInput`, `SnapshotRestoreInput`, `SnapshotInput` | | `internal/lib/firecracker/client.go` | Firecracker HTTP client: `PauseVM()`, `CreateSnapshot()`, `ResumeVM()`, `LoadSnapshot()` | | `internal/cli/snapshot.go` | Cobra commands: `mvm snapshot create\|ls\|inspect\|restore\|rm` | ## Design decisions -**No Controller — snapshots are immutable.** Snapshots have no state machine (they cannot be modified after creation). A Controller adds complexity for no benefit. The API layer handles orchestration directly. +**No snapshot Controller.** Snapshot metadata has no public state machine. The API layer handles orchestration directly; +the transitional restore lock/symlink mutation is an implementation mechanism scheduled for removal by Task 7. **Phantom symlink over source VM rootfs reference.** The phantom symlink makes the snapshot self-contained and restorable without the source VM. During restore, the symlink is updated to point to the new VM's rootfs copy. The `flock` serializes the update + LoadSnapshot window. **DB over filesystem for config storage.** The Firecracker boot config is stored in the `extra_config` DB column rather than as a `config.json` file. The enricher enriches it at restore time. This avoids file management complexity. -**`infra.CopyFile()` for rootfs copy.** Uses file-level copy with sparse support. A new subprocess call is unnecessary for this operation. +**`infra.CopyFile()` for rootfs copy.** Uses the existing sendfile/userspace fallback and `fdatasync`; a new subprocess +call is unnecessary for this operation. -**Snapshots don't pin images.** A snapshot stores `image_id`, `kernel_id`, `network_id`, and `binary_id` for metadata and enrichment, but the rootfs is copied at snapshot time — the snapshot can be restored without the original image. +**Snapshots copy the rootfs.** A snapshot stores `image_id`, `kernel_id`, `network_id`, and `binary_id` for metadata and +enrichment, but restore reads the snapshot's fixed rootfs copy rather than the original image file. -**Vsock override + separate CID update.** The vsock UDS path is overridable via `vsock_override` on `/snapshot/load`, but the guest CID requires a separate `PUT /vsock` call after load. +**Vsock override preserves the CID.** Restore passes the fixed UDS path through `vsock_override` on `/snapshot/load` and +keeps the guest CID captured in vmstate; it does not issue a separate `PUT /vsock`. diff --git a/docs/improvements/IMPROVEMENTS_008.md b/docs/improvements/IMPROVEMENTS_008.md index c77abc5a..1a07b551 100644 --- a/docs/improvements/IMPROVEMENTS_008.md +++ b/docs/improvements/IMPROVEMENTS_008.md @@ -1,6 +1,6 @@ # Snapshot-Based Instant VM Cloning & Next-Level Optimizations -> **STATUS: Snapshot domain implemented (Phase 2).** +> **STATUS: Snapshot domain implemented (Phase 2), v0.3 hardening incomplete.** > | Phase | Status | > |-------|--------| > | Phase 1: Foundation (huge pages, cgroup v2, boot args) | ⚠️ Partial | @@ -10,7 +10,10 @@ > > The snapshot domain (`internal/core/snapshot/`), CLI commands (`internal/cli/snapshot.go`), API layer (`pkg/api/snapshot.go`), and input validation (`pkg/api/inputs/snapshot.go`) are all implemented. The `SnapshotItem` model uses `SnapshotDir`, `MemoryFile`, `StateFile`, `RootfsFile`, `ImageID`, `ExtraConfig` (as `*SnapshotExtraConfig` struct). Firecracker snapshot/resume API is at `internal/lib/firecracker/client.go`. > -> **Last verified:** 2026-06-27 +> Fixed artifact naming is implemented. Crash-consistent capture, descriptor-pinned resources, and the private +> mount-namespace rootfs overlay remain release work under ADR-0016. +> +> **Last verified:** 2026-08-28 **Phase:** Multi-phase — spans 3-4 milestones **Complexity:** Very High @@ -26,15 +29,18 @@ When `PUT /snapshot/create` is called on a **paused** microVM, Firecracker produ ``` /home/user/.cache/mvmctl/snapshots/my-snapshot/ -├── vm.mem ← Raw dump of ALL guest RAM pages -├── vm.vmstate ← Serde bitcode binary format (vCPU state, device state) -├── rootfs.ext4 ← Frozen rootfs image at snapshot time -└── metadata.json ← Origin, kernel, image, timestamps +├── memory ← Raw dump of ALL guest RAM pages +├── vmstate ← Serde bitcode binary format (vCPU state, device state) +└── rootfs.img ← Fixed managed rootfs leaf; filesystem type is metadata ``` +Snapshot identity and origin metadata are stored in SQLite. The current implementation also carries the transitional +`phantom-rootfs.img` symlink documented in +[SNAPSHOT_DOMAIN.md](../implementations/SNAPSHOT_DOMAIN.md); ADR-0016 removes it through a private namespace overlay. + **The mmap secret (MAP_PRIVATE):** - Pages are NOT read eagerly — they're demand-paged (loaded on first access) -- The original `vm.mem` file is **never modified** — writes go to anonymous COW pages +- The original `memory` file is **never modified** — writes go to anonymous COW pages - **Multiple processes can MAP_PRIVATE the same file simultaneously** — each gets independent COW - Loading the snapshot takes **~3-8ms** (just the mmap syscall + metadata), not the full RAM size diff --git a/docs/improvements/MOVING_TO_AGNOSTIC_ARCHITECTURE.md b/docs/improvements/MOVING_TO_AGNOSTIC_ARCHITECTURE.md index 66861d12..3bfc300e 100644 --- a/docs/improvements/MOVING_TO_AGNOSTIC_ARCHITECTURE.md +++ b/docs/improvements/MOVING_TO_AGNOSTIC_ARCHITECTURE.md @@ -57,7 +57,7 @@ This coupling creates several problems: | `internal/lib/model/firecracker.go` | `FirecrackerConfig` struct — Firecracker-specific fields | Becomes internal to `FirecrackerDriver`. | | `internal/core/vm/controller.go` | Direct `FirecrackerClient` imports for pause/resume/start/snapshot | Controller accepts `VMMDriver` instead. | | `pkg/api/vm.go` | `buildFirecrackerConfig()`, `FirecrackerSpawner` usage | Replace with `buildVMMConfig()`, factory-based driver creation. | -| `internal/infra/constants.go` | `OverridableDefaults["defaults.firecracker"]` (8 filenames + log_level) | Move to `FirecrackerDriver`. | +| `internal/infra/constants.go` | Fixed Firecracker artifact vocabulary plus `defaults.firecracker.log_level` | Move driver-specific fixed names and settings to `FirecrackerDriver`. | | `internal/infra/constants.go` | `FirecrackerGithubReleasesAPIURL`, `FirecrackerGithubDownloadURL` | Move into `FirecrackerDriver`. | | `internal/cli/vm.go` | No `--vmm` or `--vmm-bin` flags | Add `--vmm` + `--vmm-bin` flags. | @@ -385,7 +385,7 @@ func (f *VMMFactory) CreateDriver(vmmType VMMType, vm *model.VM, repo vm.Reposit | Core/VM | `internal/lib/provisioner/` | VMM-aware block device naming | ❌ | | API | `pkg/api/vm.go`, `pkg/api/inputs/*` | VMM-agnostic config building, factory usage | ❌ | | CLI | `internal/cli/vm.go` | `--vmm`/`--vmm-bin` flags, generic help text | ❌ | -| Constants | `internal/infra/constants.go` | Remove Firecracker-specific URLs/filenames | ❌ | +| Constants | `internal/infra/constants.go` | Move Firecracker-specific URLs and fixed artifact names behind the driver | ❌ | | DB | `db/migrations/NNN_vmm_agnostic.sql` | New migration | ❌ | | Services | `internal/service/console/` | Console source abstraction | ❌ | diff --git a/tasks/todo.md b/tasks/todo.md index 11d9c502..60488bb7 100644 --- a/tasks/todo.md +++ b/tasks/todo.md @@ -145,12 +145,19 @@ under `internal/service/jailer/`. No handler, transport, CLI, API, core-domain, - [x] ID-based `volumes/.` storage. - [x] Content-addressed `kernels/` storage with receiver-owned staging; kernel pull inputs cannot select an output directory, name, or path. - - [ ] Fixed persistent `rootfs.img`, cloud-init, Firecracker configuration, log, console-log, and optional metrics - leaves; outputs are caller-owned `0600` one-link regular files truncated only after durable launch registration. - - [ ] Fixed ephemeral API/vsock/console sockets and PID mirrors beneath - `/run/mvmctl/runtime//`; the root-controlled parent and pinned caller-owned VM directory replace the - whole-cache-VM-directory bind. - - [ ] Fixed snapshot leaves and a managed ID-scoped image-provisioning staging subtree. + - [x] Freeze persistent VM producer basenames as `rootfs.img`, `cloud-init.iso`, `firecracker.json`, + `firecracker.log`, `firecracker.console.log`, and optional `firecracker.metrics`; copy custom cloud-init ISO sources + into the fixed managed leaf, rebuild relaunch paths instead of trusting SQLite path columns, and reject alternate + manifest basenames in the transitional Jailer receiver. + - [ ] Make persistent outputs caller-owned `0600` one-link regular files and truncate them only after durable launch + registration. The current launch still creates/truncates outputs before registration. + - [x] Freeze API/vsock/console socket and PID mirror basenames and remove their filename configuration/relay flags. + - [ ] Relocate those ephemeral leaves beneath `/run/mvmctl/runtime//`; the root-controlled parent and + pinned caller-owned VM directory must replace the current whole-cache-VM-directory bind. + - [x] Freeze snapshot producer/restore leaves as `rootfs.img`, `memory`, and `vmstate`; derive restore and remove paths + from the validated snapshot ID rather than SQLite file-path columns. + - [ ] Remove the transitional `phantom-rootfs.img` symlink through the Task 7 namespace overlay and add the managed + ID-scoped image-provisioning staging subtree. - [ ] Define operation-specific non-empty/maximum-size and access-mode policies before coding; a caller- or DB-provided expected size is an equality check, not authorization for an unbounded resource. - [ ] Open user cache/VM/image/kernel/volume/snapshot resources once with beneath/no-symlink/no-magic-link resolution. From 8d6bbfbdbcdd62cca8857c2f32d6a8ec08b94136 Mon Sep 17 00:00:00 2001 From: AlanD20 Date: Fri, 28 Aug 2026 23:21:19 +0200 Subject: [PATCH 0048/1457] feat: pin managed cache roots --- internal/service/jailer/managed_cache.go | 458 +++++++ internal/service/jailer/managed_cache_test.go | 1048 +++++++++++++++++ 2 files changed, 1506 insertions(+) create mode 100644 internal/service/jailer/managed_cache.go create mode 100644 internal/service/jailer/managed_cache_test.go diff --git a/internal/service/jailer/managed_cache.go b/internal/service/jailer/managed_cache.go new file mode 100644 index 00000000..35264180 --- /dev/null +++ b/internal/service/jailer/managed_cache.go @@ -0,0 +1,458 @@ +package jailer + +import ( + "context" + "errors" + "fmt" + "path/filepath" + "strings" + + "golang.org/x/sys/unix" + + "mvmctl/pkg/errs" +) + +const ( + managedCacheRootOpenFlags = unix.O_PATH | unix.O_DIRECTORY | unix.O_CLOEXEC | unix.O_NOFOLLOW + managedCacheOpenFlags = unix.O_PATH | unix.O_DIRECTORY | unix.O_CLOEXEC | unix.O_NOFOLLOW + managedCacheResolveFlags = unix.RESOLVE_BENEATH | unix.RESOLVE_NO_SYMLINKS | unix.RESOLVE_NO_MAGICLINKS + managedCacheRequiredStat = unix.STATX_TYPE | unix.STATX_MODE | unix.STATX_UID | unix.STATX_INO + managedCacheStatxFlags = unix.AT_EMPTY_PATH | unix.AT_NO_AUTOMOUNT | unix.AT_SYMLINK_NOFOLLOW | + unix.AT_STATX_FORCE_SYNC + managedCacheStatxMask = unix.STATX_BASIC_STATS | unix.STATX_MNT_ID | unix.STATX_MNT_ID_UNIQUE + // Linux does not expose ZFS_SUPER_MAGIC through x/sys/unix. + managedCacheZFSSuperMagic int64 = 0x2fc12fc1 + + maxManagedCacheLocatorDepth = 64 +) + +type managedCacheLocator struct { + components []string +} + +type managedCacheIdentity struct { + deviceMajor uint32 + deviceMinor uint32 + inode uint64 + mountID uint64 + mountIDKind managedCacheMountIdentityKind +} + +type managedCacheMountIdentityKind uint8 + +const ( + managedCacheMountIdentityUnavailable managedCacheMountIdentityKind = iota + managedCacheMountIdentityLegacy + managedCacheMountIdentityUnique +) + +type managedCachePolicy struct { + rootPath string + trustedUID uint32 +} + +type managedCacheDeps struct { + open func(context.Context, string, int, uint32) (int, error) + openAt2 func(context.Context, int, string, *unix.OpenHow) (int, error) + statx func(context.Context, int, int, int, *unix.Statx_t) error + statfs func(context.Context, int, *unix.Statfs_t) error + close func(context.Context, int) error +} + +type managedCacheLease struct { + deps managedCacheDeps + cacheFD int + identity managedCacheIdentity + retained []int +} + +type managedCacheComponentRole uint8 + +const ( + managedCacheFilesystemRoot managedCacheComponentRole = iota + managedCacheAncestor + managedCacheRoot +) + +func productionManagedCachePolicy() managedCachePolicy { + return managedCachePolicy{rootPath: "/", trustedUID: 0} +} + +func parseManagedCacheLocator(raw string) (managedCacheLocator, error) { + if strings.ContainsRune(raw, '\x00') { + return managedCacheLocator{}, managedCacheValidationError("managed cache locator contains NUL") + } + if !filepath.IsAbs(raw) { + return managedCacheLocator{}, managedCacheValidationError("managed cache locator must be absolute") + } + if len(raw) >= unix.PathMax { + return managedCacheLocator{}, managedCacheValidationError("managed cache locator exceeds path length limit") + } + if filepath.Clean(raw) != raw { + return managedCacheLocator{}, managedCacheValidationError("managed cache locator must be canonical") + } + if raw == string(filepath.Separator) { + return managedCacheLocator{}, managedCacheValidationError("filesystem root is not a managed cache locator") + } + components := strings.Split(strings.TrimPrefix(raw, string(filepath.Separator)), string(filepath.Separator)) + locator := managedCacheLocator{components: components} + if err := validateManagedCacheLocator(locator); err != nil { + return managedCacheLocator{}, err + } + return locator, nil +} + +func pinManagedCache( + ctx context.Context, + deps managedCacheDeps, + policy managedCachePolicy, + caller instanceCaller, + locator managedCacheLocator, +) (_ *managedCacheLease, returnErr error) { + if err := ctx.Err(); err != nil { + return nil, managedCacheAtomicError("pin managed cache", err) + } + if err := validateInstanceCaller(caller); err != nil { + return nil, err + } + if err := validateManagedCachePolicy(policy); err != nil { + return nil, err + } + if err := validateManagedCacheLocator(locator); err != nil { + return nil, err + } + + retained := make([]int, 0, len(locator.components)+1) + defer func() { + if returnErr == nil { + return + } + if closeErr := closeManagedCacheFDs(ctx, deps, retained); closeErr != nil { + returnErr = appendManagedCacheCleanupError( + returnErr, + "close rejected managed cache descriptors", + closeErr, + ) + } + }() + + rootFD, err := deps.open(ctx, policy.rootPath, managedCacheRootOpenFlags, 0) + if err != nil { + return nil, managedCacheAtomicError("open filesystem root for managed cache", err) + } + retained = append(retained, rootFD) + if _, err := inspectManagedCacheDirectory( + ctx, + deps, + rootFD, + "filesystem root", + managedCacheFilesystemRoot, + policy, + caller, + ); err != nil { + return nil, err + } + + parentFD := rootFD + identity := managedCacheIdentity{} + for index, component := range locator.components { + if err := ctx.Err(); err != nil { + return nil, managedCacheAtomicError("open managed cache component", err) + } + // CRITICAL: The locator may cross onto a dedicated large-filesystem mount. RESOLVE_NO_XDEV is + // intentionally reserved for descendant resource opens beneath the already-pinned cache root. + how := unix.OpenHow{ + Flags: uint64(managedCacheOpenFlags), + Resolve: uint64(managedCacheResolveFlags), + } + fd, openErr := deps.openAt2(ctx, parentFD, component, &how) + if openErr != nil { + return nil, classifyManagedCacheOpenError(component, openErr) + } + retained = append(retained, fd) + role := managedCacheAncestor + if index == len(locator.components)-1 { + role = managedCacheRoot + } + identity, err = inspectManagedCacheDirectory(ctx, deps, fd, component, role, policy, caller) + if err != nil { + return nil, err + } + parentFD = fd + } + + lease := &managedCacheLease{ + deps: deps, + cacheFD: parentFD, + identity: identity, + retained: retained, + } + retained = nil + return lease, nil +} + +func repinManagedCache( + ctx context.Context, + deps managedCacheDeps, + policy managedCachePolicy, + caller instanceCaller, + locator managedCacheLocator, + expected managedCacheIdentity, +) (_ *managedCacheLease, returnErr error) { + lease, err := pinManagedCache(ctx, deps, policy, caller, locator) + if err != nil { + return nil, err + } + if lease.identity != expected { + identityErr := managedCacheAtomicError("managed cache identity does not match registered authority", nil) + return nil, appendManagedCacheCleanupError( + identityErr, + "release mismatched managed cache lease", + lease.Release(context.WithoutCancel(ctx)), + ) + } + return lease, nil +} + +func validateManagedCachePolicy(policy managedCachePolicy) error { + if !filepath.IsAbs(policy.rootPath) || filepath.Clean(policy.rootPath) != policy.rootPath || + strings.ContainsRune(policy.rootPath, '\x00') { + return managedCacheAtomicError("managed cache filesystem root policy is invalid", nil) + } + return nil +} + +func validateManagedCacheLocator(locator managedCacheLocator) error { + if len(locator.components) == 0 { + return managedCacheValidationError("managed cache locator has no components") + } + if len(locator.components) > maxManagedCacheLocatorDepth { + return managedCacheValidationError("managed cache locator exceeds component depth limit") + } + pathBytes := 0 + for _, component := range locator.components { + if component == "" || component == "." || component == ".." || + strings.ContainsRune(component, filepath.Separator) || strings.ContainsRune(component, '\x00') { + return managedCacheValidationError("managed cache locator has unsafe components") + } + if len(component) > unix.NAME_MAX { + return managedCacheValidationError("managed cache locator component exceeds name limit") + } + pathBytes += len(component) + 1 + } + if pathBytes >= unix.PathMax { + return managedCacheValidationError("managed cache locator exceeds path length limit") + } + return nil +} + +func inspectManagedCacheDirectory( + ctx context.Context, + deps managedCacheDeps, + fd int, + name string, + role managedCacheComponentRole, + policy managedCachePolicy, + caller instanceCaller, +) (managedCacheIdentity, error) { + if err := ctx.Err(); err != nil { + return managedCacheIdentity{}, managedCacheAtomicError("inspect managed cache directory "+name, err) + } + var stat unix.Statx_t + if err := deps.statx(ctx, fd, managedCacheStatxFlags, managedCacheStatxMask, &stat); err != nil { + return managedCacheIdentity{}, managedCacheAtomicError("inspect managed cache directory "+name, err) + } + if stat.Mask&managedCacheRequiredStat != managedCacheRequiredStat { + return managedCacheIdentity{}, managedCacheValidationError( + "managed cache directory " + name + " has incomplete stable identity", + ) + } + if stat.Mode&unix.S_IFMT != unix.S_IFDIR { + return managedCacheIdentity{}, managedCacheValidationError( + "managed cache component " + name + " is not a directory", + ) + } + if stat.Attributes_mask&unix.STATX_ATTR_AUTOMOUNT != 0 && + stat.Attributes&unix.STATX_ATTR_AUTOMOUNT != 0 { + return managedCacheIdentity{}, managedCacheValidationError( + "managed cache component " + name + " uses unsupported automount topology", + ) + } + if role == managedCacheRoot { + var filesystem unix.Statfs_t + if err := deps.statfs(ctx, fd, &filesystem); err != nil { + return managedCacheIdentity{}, managedCacheAtomicError( + "inspect managed cache filesystem topology "+name, + err, + ) + } + if !isSupportedManagedCacheFilesystem(filesystem.Type) { + return managedCacheIdentity{}, managedCacheValidationError( + "managed cache root uses unsupported filesystem topology", + ) + } + } + if stat.Mode&0022 != 0 { + return managedCacheIdentity{}, managedCacheValidationError( + "managed cache component " + name + " is writable by another user", + ) + } + switch role { + case managedCacheFilesystemRoot: + if stat.Uid != policy.trustedUID { + return managedCacheIdentity{}, managedCacheValidationError( + "managed cache filesystem root has unexpected ownership", + ) + } + case managedCacheAncestor: + if stat.Uid != policy.trustedUID && stat.Uid != caller.uid { + return managedCacheIdentity{}, managedCacheValidationError( + "managed cache ancestor " + name + " has foreign ownership", + ) + } + case managedCacheRoot: + if stat.Uid != caller.uid { + return managedCacheIdentity{}, managedCacheValidationError( + "managed cache root is not owned by the invoking user", + ) + } + if stat.Mode&0700 != 0700 { + return managedCacheIdentity{}, managedCacheValidationError( + "managed cache root must grant its owner read, write, and traversal access", + ) + } + default: + return managedCacheIdentity{}, managedCacheAtomicError("managed cache component role is invalid", nil) + } + if stat.Ino == 0 || (stat.Dev_major == 0 && stat.Dev_minor == 0) { + return managedCacheIdentity{}, managedCacheValidationError( + "managed cache directory " + name + " has incomplete stable identity", + ) + } + identity := managedCacheIdentity{ + deviceMajor: stat.Dev_major, + deviceMinor: stat.Dev_minor, + inode: stat.Ino, + } + switch { + case stat.Mask&unix.STATX_MNT_ID_UNIQUE != 0: + if stat.Mnt_id == 0 { + return managedCacheIdentity{}, managedCacheValidationError( + "managed cache directory " + name + " has incomplete stable mount identity", + ) + } + identity.mountIDKind = managedCacheMountIdentityUnique + identity.mountID = stat.Mnt_id + case stat.Mask&unix.STATX_MNT_ID != 0: + if stat.Mnt_id == 0 { + return managedCacheIdentity{}, managedCacheValidationError( + "managed cache directory " + name + " has incomplete stable mount identity", + ) + } + identity.mountIDKind = managedCacheMountIdentityLegacy + identity.mountID = stat.Mnt_id + } + return identity, nil +} + +func isSupportedManagedCacheFilesystem(filesystemType int64) bool { + switch filesystemType { + case unix.EXT4_SUPER_MAGIC, + unix.XFS_SUPER_MAGIC, + unix.BTRFS_SUPER_MAGIC, + unix.F2FS_SUPER_MAGIC, + unix.BCACHEFS_SUPER_MAGIC, + unix.TMPFS_MAGIC, + managedCacheZFSSuperMagic: + return true + default: + return false + } +} + +func classifyManagedCacheOpenError(component string, cause error) error { + message := "open managed cache component " + component + if errors.Is(cause, unix.ELOOP) || errors.Is(cause, unix.ENOTDIR) || errors.Is(cause, unix.EXDEV) || + errors.Is(cause, unix.ENOENT) { + return errs.WrapMsg(errs.CodeValidationFailed, message, cause) + } + return managedCacheAtomicError(message, cause) +} + +func (lease *managedCacheLease) Release(ctx context.Context) error { + if lease == nil || len(lease.retained) == 0 { + return nil + } + err := closeManagedCacheFDs(ctx, lease.deps, lease.retained) + lease.retained = nil + lease.cacheFD = -1 + if err != nil { + return managedCacheCleanupError("release managed cache descriptors", err) + } + return nil +} + +func closeManagedCacheFDs(ctx context.Context, deps managedCacheDeps, fds []int) error { + cleanupCtx := context.WithoutCancel(ctx) + var result error + for index := len(fds) - 1; index >= 0; index-- { + if err := deps.close(cleanupCtx, fds[index]); err != nil { + result = errors.Join(result, fmt.Errorf("fd %d: %w", fds[index], err)) + } + } + return result +} + +func appendManagedCacheCleanupError(primary error, message string, cause error) error { + if cause == nil { + return primary + } + if primary == nil { + return cause + } + if domainErr := errs.AsDomainError(primary); domainErr != nil { + domainErr.Message += "; " + message + ": " + cause.Error() + domainErr.Err = errors.Join(domainErr.Err, cause) + if domainErr.Details == nil { + domainErr.Details = make(map[string]any) + } + domainErr.Details["managed_cache_descriptor_cleanup_failed"] = true + return domainErr + } + return managedCacheCleanupError(message+": "+cause.Error(), errors.Join(primary, cause)) +} + +func managedCacheValidationError(message string) *errs.DomainError { + return errs.New(errs.CodeValidationFailed, message) +} + +func managedCacheAtomicError(message string, cause error) *errs.DomainError { + if cause == nil { + return errs.New(errs.CodeVMAtomicFailed, message, errs.WithClass(errs.ClassInternal)) + } + return errs.WrapMsg(errs.CodeVMAtomicFailed, message, cause, errs.WithClass(errs.ClassInternal)) +} + +func managedCacheCleanupError(message string, cause error) *errs.DomainError { + domainErr := managedCacheAtomicError(message, cause) + domainErr.Details = map[string]any{"managed_cache_descriptor_cleanup_failed": true} + return domainErr +} + +func realManagedCacheDeps() managedCacheDeps { + return managedCacheDeps{ + open: func(_ context.Context, path string, flags int, mode uint32) (int, error) { + return unix.Open(path, flags, mode) + }, + openAt2: func(_ context.Context, parentFD int, name string, how *unix.OpenHow) (int, error) { + return unix.Openat2(parentFD, name, how) + }, + statx: func(_ context.Context, fd, flags, mask int, stat *unix.Statx_t) error { + return unix.Statx(fd, "", flags, mask, stat) + }, + statfs: func(_ context.Context, fd int, stat *unix.Statfs_t) error { + return unix.Fstatfs(fd, stat) + }, + close: func(_ context.Context, fd int) error { return unix.Close(fd) }, + } +} diff --git a/internal/service/jailer/managed_cache_test.go b/internal/service/jailer/managed_cache_test.go new file mode 100644 index 00000000..a8210025 --- /dev/null +++ b/internal/service/jailer/managed_cache_test.go @@ -0,0 +1,1048 @@ +package jailer + +import ( + "context" + "errors" + "os" + "path/filepath" + "strings" + "testing" + + "github.com/google/go-cmp/cmp" + "github.com/google/go-cmp/cmp/cmpopts" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + "golang.org/x/sys/unix" + + "mvmctl/pkg/errs" +) + +func TestParseManagedCacheLocator(t *testing.T) { + t.Parallel() + + tests := map[string]struct { + raw string + want []string + wantErr bool + wantErrMsg string + }{ + "absolute custom cache": { + raw: "/mnt/vm storage/mvmctl", + want: []string{"mnt", "vm storage", "mvmctl"}, + }, + "empty": {raw: "", wantErr: true, wantErrMsg: "absolute"}, + "root": {raw: "/", wantErr: true, wantErrMsg: "root"}, + "relative": {raw: "cache/mvmctl", wantErr: true, wantErrMsg: "absolute"}, + "dot component": {raw: "/home/./mvmctl", wantErr: true, wantErrMsg: "canonical"}, + "dot dot component": {raw: "/home/user/../mvmctl", wantErr: true, wantErrMsg: "canonical"}, + "duplicate separator": {raw: "/home//mvmctl", wantErr: true, wantErrMsg: "canonical"}, + "trailing separator": {raw: "/home/mvmctl/", wantErr: true, wantErrMsg: "canonical"}, + "nul": {raw: "/home/mvmctl\x00escape", wantErr: true, wantErrMsg: "NUL"}, + "component exceeds limit": { + raw: "/" + strings.Repeat("a", unix.NAME_MAX+1), + wantErr: true, + wantErrMsg: "component", + }, + } + + for name, tc := range tests { + t.Run(name, func(t *testing.T) { + locator, err := parseManagedCacheLocator(tc.raw) + if tc.wantErr { + require.Error(t, err) + assert.Contains(t, err.Error(), tc.wantErrMsg) + requireManagedCacheDomainErrorCode(t, err, errs.CodeValidationFailed) + return + } + require.NoError(t, err) + if diff := cmp.Diff(tc.want, locator.components); diff != "" { + t.Errorf("managed cache components mismatch (-want +got):\n%s", diff) + } + }) + } +} + +func TestParseManagedCacheLocatorRejectsPathAndDepthBounds(t *testing.T) { + t.Parallel() + + overlong := "/" + strings.Repeat("a", unix.PathMax-1) + _, err := parseManagedCacheLocator(overlong) + require.Error(t, err) + assert.Contains(t, err.Error(), "length") + + deep := "" + for range maxManagedCacheLocatorDepth + 1 { + deep += "/a" + } + _, err = parseManagedCacheLocator(deep) + require.Error(t, err) + assert.Contains(t, err.Error(), "depth") +} + +func TestPinManagedCacheRetainsDescriptorIdentity(t *testing.T) { + t.Parallel() + + fixture := prepareManagedCacheFixture(t) + lease, err := pinManagedCache( + context.Background(), + fixture.deps, + fixture.policy, + fixture.caller, + fixture.locator, + ) + require.NoError(t, err) + require.NotNil(t, lease) + t.Cleanup(func() { require.NoError(t, lease.Release(context.Background())) }) + require.NotEmpty(t, lease.retained) + assert.NotZero(t, lease.identity.inode) + assert.NotZero(t, lease.identity.deviceMajor|lease.identity.deviceMinor) + + var stat unix.Statx_t + require.NoError(t, unix.Statx( + lease.cacheFD, + "", + managedCacheStatxFlags, + managedCacheStatxMask, + &stat, + )) + assert.Equal(t, stat.Ino, lease.identity.inode) + assert.Equal(t, stat.Dev_major, lease.identity.deviceMajor) + assert.Equal(t, stat.Dev_minor, lease.identity.deviceMinor) + switch { + case stat.Mask&unix.STATX_MNT_ID_UNIQUE != 0: + assert.Equal(t, stat.Mnt_id, lease.identity.mountID) + assert.Equal(t, managedCacheMountIdentityUnique, lease.identity.mountIDKind) + case stat.Mask&unix.STATX_MNT_ID != 0: + assert.Equal(t, stat.Mnt_id, lease.identity.mountID) + assert.Equal(t, managedCacheMountIdentityLegacy, lease.identity.mountIDKind) + default: + assert.Zero(t, lease.identity.mountID) + assert.Equal(t, managedCacheMountIdentityUnavailable, lease.identity.mountIDKind) + } + + repinned, err := repinManagedCache( + context.Background(), + fixture.deps, + fixture.policy, + fixture.caller, + fixture.locator, + lease.identity, + ) + require.NoError(t, err) + require.NotNil(t, repinned) + t.Cleanup(func() { require.NoError(t, repinned.Release(context.Background())) }) + if diff := cmp.Diff( + lease.identity, + repinned.identity, + cmpopts.EquateComparable(managedCacheIdentity{}), + ); diff != "" { + t.Errorf("re-pinned cache identity mismatch (-want +got):\n%s", diff) + } +} + +func TestManagedCacheDescriptorSurvivesLocatorReplacement(t *testing.T) { + t.Parallel() + // Rationale: The invoking user can rename its cache path after root validates it. The retained descriptor + // must keep referring to the originally inspected directory, while a later path-based re-pin fails closed. + + fixture := prepareManagedCacheFixture(t) + lease, err := pinManagedCache( + context.Background(), + fixture.deps, + fixture.policy, + fixture.caller, + fixture.locator, + ) + require.NoError(t, err) + require.NotNil(t, lease) + t.Cleanup(func() { require.NoError(t, lease.Release(context.Background())) }) + + detachedPath := fixture.cacheDir + ".detached" + require.NoError(t, os.Rename(fixture.cacheDir, detachedPath)) + require.NoError(t, os.Mkdir(fixture.cacheDir, 0700)) + if os.Geteuid() == 0 { + require.NoError(t, os.Chown(fixture.cacheDir, int(fixture.caller.uid), -1)) + } + + var pinnedStat unix.Stat_t + require.NoError(t, unix.Fstat(lease.cacheFD, &pinnedStat)) + assert.Equal(t, fixture.cacheInode, pinnedStat.Ino) + assert.Equal(t, fixture.cacheInode, lease.identity.inode) + + repinned, err := repinManagedCache( + context.Background(), + fixture.deps, + fixture.policy, + fixture.caller, + fixture.locator, + lease.identity, + ) + require.Error(t, err) + assert.Nil(t, repinned) + assert.Contains(t, err.Error(), "identity") +} + +func TestRepinManagedCacheResistsConcurrentRenameAndSymlinkReplacement(t *testing.T) { + tests := map[string]func(*testing.T, *managedCacheFixture) (string, string){ + "caller-owned ancestor": func(t *testing.T, fixture *managedCacheFixture) (string, string) { + attackerOwner := filepath.Join(fixture.root, "attacker-owner") + require.NoError(t, os.Mkdir(attackerOwner, 0700)) + require.NoError(t, os.Mkdir(filepath.Join(attackerOwner, "mvmctl"), 0700)) + if os.Geteuid() == 0 { + require.NoError(t, os.Chown(attackerOwner, int(fixture.caller.uid), -1)) + require.NoError(t, os.Chown(filepath.Join(attackerOwner, "mvmctl"), int(fixture.caller.uid), -1)) + } + return fixture.ownerDir, attackerOwner + }, + "cache root": func(t *testing.T, fixture *managedCacheFixture) (string, string) { + attackerCache := filepath.Join(fixture.root, "attacker-cache") + require.NoError(t, os.Mkdir(attackerCache, 0700)) + if os.Geteuid() == 0 { + require.NoError(t, os.Chown(attackerCache, int(fixture.caller.uid), -1)) + } + return fixture.cacheDir, attackerCache + }, + } + + for name, prepareReplacement := range tests { + t.Run(name, func(t *testing.T) { + fixture := prepareManagedCacheFixture(t) + targetPath, attackerPath := prepareReplacement(t, &fixture) + assertManagedCacheReplacementRace(t, fixture, targetPath, attackerPath) + }) + } +} + +func assertManagedCacheReplacementRace( + t *testing.T, + fixture managedCacheFixture, + targetPath string, + attackerPath string, +) { + t.Helper() + // Rationale: A malicious caller can race every caller-owned lookup. Re-pin may reject a transient path, but it + // must never return a lease for the replacement directory or follow the replacement symlink. + original, err := pinManagedCache( + context.Background(), + fixture.deps, + fixture.policy, + fixture.caller, + fixture.locator, + ) + require.NoError(t, err) + require.NotNil(t, original) + t.Cleanup(func() { require.NoError(t, original.Release(context.Background())) }) + + detachedPath := targetPath + ".race-detached" + firstReplacement := make(chan struct{}) + continueRace := make(chan struct{}) + stop := make(chan struct{}) + done := make(chan struct{}) + attackerErr := make(chan error, 1) + go func() { + defer close(done) + first := true + for { + select { + case <-stop: + return + default: + } + if renameErr := os.Rename(targetPath, detachedPath); renameErr != nil { + attackerErr <- renameErr + return + } + if symlinkErr := os.Symlink(attackerPath, targetPath); symlinkErr != nil { + attackerErr <- symlinkErr + return + } + if first { + close(firstReplacement) + select { + case <-continueRace: + case <-stop: + return + } + first = false + } + if removeErr := os.Remove(targetPath); removeErr != nil { + attackerErr <- removeErr + return + } + if renameErr := os.Rename(detachedPath, targetPath); renameErr != nil { + attackerErr <- renameErr + return + } + } + }() + stopped := false + stopRace := func() { + if stopped { + return + } + close(stop) + <-done + stopped = true + } + defer stopRace() + <-firstReplacement + + repinned, err := repinManagedCache( + context.Background(), + fixture.deps, + fixture.policy, + fixture.caller, + fixture.locator, + original.identity, + ) + require.Error(t, err) + assert.Nil(t, repinned) + close(continueRace) + + for range 256 { + repinned, repinErr := repinManagedCache( + context.Background(), + fixture.deps, + fixture.policy, + fixture.caller, + fixture.locator, + original.identity, + ) + if repinErr != nil { + continue + } + require.NotNil(t, repinned) + if diff := cmp.Diff( + original.identity, + repinned.identity, + cmpopts.EquateComparable(managedCacheIdentity{}), + ); diff != "" { + t.Errorf("race re-pin identity mismatch (-want +got):\n%s", diff) + } + require.NoError(t, repinned.Release(context.Background())) + } + stopRace() + select { + case err := <-attackerErr: + require.NoError(t, err) + default: + } +} + +func TestPinManagedCacheAcceptsUnavailableMountIdentity(t *testing.T) { + t.Parallel() + // Rationale: ADR-0016 records mount identity where the kernel provides it. Device and inode remain mandatory + // on kernels or filesystems that omit STATX_MNT_ID, and both sides of a later re-pin must agree on that absence. + + fixture := prepareManagedCacheFixture(t) + realStatx := fixture.deps.statx + fixture.deps.statx = func(ctx context.Context, fd, flags, mask int, stat *unix.Statx_t) error { + if err := realStatx(ctx, fd, flags, mask, stat); err != nil { + return err + } + stat.Mask &^= unix.STATX_MNT_ID | unix.STATX_MNT_ID_UNIQUE + stat.Mnt_id = 0 + return nil + } + lease, err := pinManagedCache( + context.Background(), + fixture.deps, + fixture.policy, + fixture.caller, + fixture.locator, + ) + require.NoError(t, err) + require.NotNil(t, lease) + t.Cleanup(func() { require.NoError(t, lease.Release(context.Background())) }) + assert.Zero(t, lease.identity.mountID) + assert.Equal(t, managedCacheMountIdentityUnavailable, lease.identity.mountIDKind) + + repinned, err := repinManagedCache( + context.Background(), + fixture.deps, + fixture.policy, + fixture.caller, + fixture.locator, + lease.identity, + ) + require.NoError(t, err) + require.NotNil(t, repinned) + t.Cleanup(func() { require.NoError(t, repinned.Release(context.Background())) }) +} + +func TestPinManagedCachePrefersUniqueMountIdentityAndFallsBackToLegacy(t *testing.T) { + t.Parallel() + + tests := map[string]struct { + mutate func(*unix.Statx_t) + wantKind managedCacheMountIdentityKind + wantID uint64 + }{ + "unique preferred": { + mutate: func(stat *unix.Statx_t) { + stat.Mask |= unix.STATX_MNT_ID | unix.STATX_MNT_ID_UNIQUE + stat.Mnt_id = 701 + }, + wantKind: managedCacheMountIdentityUnique, + wantID: 701, + }, + "legacy fallback": { + mutate: func(stat *unix.Statx_t) { + stat.Mask &^= unix.STATX_MNT_ID_UNIQUE + stat.Mask |= unix.STATX_MNT_ID + stat.Mnt_id = 702 + }, + wantKind: managedCacheMountIdentityLegacy, + wantID: 702, + }, + } + + for name, tc := range tests { + t.Run(name, func(t *testing.T) { + fixture := prepareManagedCacheFixture(t) + realStatx := fixture.deps.statx + fixture.deps.statx = func(ctx context.Context, fd, flags, mask int, stat *unix.Statx_t) error { + if err := realStatx(ctx, fd, flags, mask, stat); err != nil { + return err + } + if stat.Ino == fixture.cacheInode { + tc.mutate(stat) + } + return nil + } + + lease, err := pinManagedCache( + context.Background(), + fixture.deps, + fixture.policy, + fixture.caller, + fixture.locator, + ) + require.NoError(t, err) + require.NotNil(t, lease) + t.Cleanup(func() { require.NoError(t, lease.Release(context.Background())) }) + assert.Equal(t, tc.wantKind, lease.identity.mountIDKind) + assert.Equal(t, tc.wantID, lease.identity.mountID) + }) + } +} + +func TestRepinManagedCacheRejectsEveryIdentityMismatch(t *testing.T) { + t.Parallel() + + fixture := prepareManagedCacheFixture(t) + lease, err := pinManagedCache( + context.Background(), + fixture.deps, + fixture.policy, + fixture.caller, + fixture.locator, + ) + require.NoError(t, err) + require.NotNil(t, lease) + t.Cleanup(func() { require.NoError(t, lease.Release(context.Background())) }) + + tests := map[string]func(managedCacheIdentity) managedCacheIdentity{ + "device major": func(identity managedCacheIdentity) managedCacheIdentity { + identity.deviceMajor++ + return identity + }, + "device minor": func(identity managedCacheIdentity) managedCacheIdentity { + identity.deviceMinor++ + return identity + }, + "inode": func(identity managedCacheIdentity) managedCacheIdentity { + identity.inode++ + return identity + }, + "mount": func(identity managedCacheIdentity) managedCacheIdentity { + identity.mountID++ + return identity + }, + "mount identity kind": func(identity managedCacheIdentity) managedCacheIdentity { + identity.mountIDKind++ + return identity + }, + } + + for name, mutate := range tests { + t.Run(name, func(t *testing.T) { + repinned, repinErr := repinManagedCache( + context.Background(), + fixture.deps, + fixture.policy, + fixture.caller, + fixture.locator, + mutate(lease.identity), + ) + require.Error(t, repinErr) + assert.Nil(t, repinned) + requireManagedCacheDomainErrorCode(t, repinErr, errs.CodeVMAtomicFailed) + assert.Contains(t, repinErr.Error(), "identity") + }) + } +} + +func TestPinManagedCacheUsesOnlyConstrainedDescriptorResolution(t *testing.T) { + t.Parallel() + + fixture := prepareManagedCacheFixture(t) + realOpen := fixture.deps.open + realOpenAt2 := fixture.deps.openAt2 + realStatx := fixture.deps.statx + realStatfs := fixture.deps.statfs + openedPaths := make([]string, 0, 1) + seenComponents := make([]string, 0, len(fixture.locator.components)) + statxCalls := 0 + statfsCalls := 0 + fixture.deps.open = func(ctx context.Context, path string, flags int, mode uint32) (int, error) { + openedPaths = append(openedPaths, path) + return realOpen(ctx, path, flags, mode) + } + fixture.deps.openAt2 = func( + ctx context.Context, + parentFD int, + name string, + how *unix.OpenHow, + ) (int, error) { + seenComponents = append(seenComponents, name) + assert.Equal(t, uint64(managedCacheOpenFlags), how.Flags) + assert.Equal(t, uint64(managedCacheResolveFlags), how.Resolve) + assert.Zero(t, how.Resolve&unix.RESOLVE_NO_XDEV, + "custom cache roots may reside on a separate filesystem") + return realOpenAt2(ctx, parentFD, name, how) + } + fixture.deps.statx = func(ctx context.Context, fd, flags, mask int, stat *unix.Statx_t) error { + statxCalls++ + assert.Equal(t, managedCacheStatxFlags, flags) + assert.NotZero(t, flags&unix.AT_STATX_FORCE_SYNC) + assert.Zero(t, flags&unix.AT_STATX_DONT_SYNC) + assert.Equal(t, managedCacheStatxMask, mask) + return realStatx(ctx, fd, flags, mask, stat) + } + fixture.deps.statfs = func(ctx context.Context, fd int, stat *unix.Statfs_t) error { + statfsCalls++ + return realStatfs(ctx, fd, stat) + } + + lease, err := pinManagedCache( + context.Background(), + fixture.deps, + fixture.policy, + fixture.caller, + fixture.locator, + ) + require.NoError(t, err) + require.NotNil(t, lease) + require.NoError(t, lease.Release(context.Background())) + if diff := cmp.Diff([]string{fixture.policy.rootPath}, openedPaths); diff != "" { + t.Errorf("pathname opens mismatch (-want +got):\n%s", diff) + } + if diff := cmp.Diff(fixture.locator.components, seenComponents); diff != "" { + t.Errorf("descriptor-relative components mismatch (-want +got):\n%s", diff) + } + assert.Equal(t, len(fixture.locator.components)+1, statxCalls) + assert.Equal(t, 1, statfsCalls, "only the resource-bearing cache mount requires an approved filesystem") +} + +func TestPinManagedCacheRejectsForgedLocatorBeforeFilesystemAccess(t *testing.T) { + t.Parallel() + // Rationale: The typed locator is private but still constructible elsewhere in this package. The privileged + // effect boundary must re-check its invariants instead of depending only on the wire parser's constructor. + + tests := map[string]managedCacheLocator{ + "empty": {}, + "slash in component": {components: []string{"home/caller", "mvmctl"}}, + "dot component": {components: []string{"home", "..", "mvmctl"}}, + "nul component": {components: []string{"home", "caller\x00escape", "mvmctl"}}, + } + + for name, locator := range tests { + t.Run(name, func(t *testing.T) { + fixture := prepareManagedCacheFixture(t) + openCalled := false + fixture.deps.open = func(context.Context, string, int, uint32) (int, error) { + openCalled = true + return -1, errors.New("unexpected open") + } + lease, err := pinManagedCache( + context.Background(), + fixture.deps, + fixture.policy, + fixture.caller, + locator, + ) + require.Error(t, err) + assert.Nil(t, lease) + requireManagedCacheDomainErrorCode(t, err, errs.CodeValidationFailed) + assert.False(t, openCalled) + }) + } +} + +func TestPinManagedCacheRejectsUnsafeComponents(t *testing.T) { + tests := map[string]func(*testing.T, *managedCacheFixture){ + "ancestor symlink": func(t *testing.T, fixture *managedCacheFixture) { + require.NoError(t, os.RemoveAll(fixture.ownerDir)) + require.NoError(t, os.Symlink(fixture.cacheDir, fixture.ownerDir)) + }, + "cache symlink": func(t *testing.T, fixture *managedCacheFixture) { + require.NoError(t, os.Remove(fixture.cacheDir)) + require.NoError(t, os.Symlink(fixture.ownerDir, fixture.cacheDir)) + }, + "cache is regular file": func(t *testing.T, fixture *managedCacheFixture) { + require.NoError(t, os.Remove(fixture.cacheDir)) + require.NoError(t, os.WriteFile(fixture.cacheDir, []byte("not a directory"), 0600)) + }, + "ancestor group writable": func(t *testing.T, fixture *managedCacheFixture) { + require.NoError(t, os.Chmod(fixture.ownerDir, 0770)) + }, + "cache world writable": func(t *testing.T, fixture *managedCacheFixture) { + require.NoError(t, os.Chmod(fixture.cacheDir, 0707)) + }, + "cache not writable": func(t *testing.T, fixture *managedCacheFixture) { + require.NoError(t, os.Chmod(fixture.cacheDir, 0500)) + }, + } + + for name, mutate := range tests { + t.Run(name, func(t *testing.T) { + fixture := prepareManagedCacheFixture(t) + mutate(t, &fixture) + lease, err := pinManagedCache( + context.Background(), + fixture.deps, + fixture.policy, + fixture.caller, + fixture.locator, + ) + require.Error(t, err) + assert.Nil(t, lease) + requireManagedCacheDomainErrorCode(t, err, errs.CodeValidationFailed) + }) + } +} + +func TestPinManagedCacheRejectsForeignOwnerAndIncompleteIdentity(t *testing.T) { + tests := map[string]func(*managedCacheFixture){ + "foreign final owner": func(fixture *managedCacheFixture) { + realStatx := fixture.deps.statx + fixture.deps.statx = func(ctx context.Context, fd, flags, mask int, stat *unix.Statx_t) error { + if err := realStatx(ctx, fd, flags, mask, stat); err != nil { + return err + } + if stat.Ino == fixture.cacheInode { + stat.Uid = fixture.caller.uid + 1 + } + return nil + } + }, + "foreign ancestor owner": func(fixture *managedCacheFixture) { + realStatx := fixture.deps.statx + fixture.deps.statx = func(ctx context.Context, fd, flags, mask int, stat *unix.Statx_t) error { + if err := realStatx(ctx, fd, flags, mask, stat); err != nil { + return err + } + if stat.Ino == fixture.ownerInode { + stat.Uid = fixture.caller.uid + 1 + } + return nil + } + }, + "missing inode identity": func(fixture *managedCacheFixture) { + realStatx := fixture.deps.statx + fixture.deps.statx = func(ctx context.Context, fd, flags, mask int, stat *unix.Statx_t) error { + if err := realStatx(ctx, fd, flags, mask, stat); err != nil { + return err + } + if stat.Ino == fixture.cacheInode { + stat.Mask &^= unix.STATX_INO + } + return nil + } + }, + "automount topology": func(fixture *managedCacheFixture) { + realStatx := fixture.deps.statx + fixture.deps.statx = func(ctx context.Context, fd, flags, mask int, stat *unix.Statx_t) error { + if err := realStatx(ctx, fd, flags, mask, stat); err != nil { + return err + } + if stat.Ino == fixture.cacheInode { + stat.Attributes_mask |= unix.STATX_ATTR_AUTOMOUNT + stat.Attributes |= unix.STATX_ATTR_AUTOMOUNT + } + return nil + } + }, + "present zero unique mount identity": func(fixture *managedCacheFixture) { + realStatx := fixture.deps.statx + fixture.deps.statx = func(ctx context.Context, fd, flags, mask int, stat *unix.Statx_t) error { + if err := realStatx(ctx, fd, flags, mask, stat); err != nil { + return err + } + if stat.Ino == fixture.cacheInode { + stat.Mask |= unix.STATX_MNT_ID_UNIQUE + stat.Mnt_id = 0 + } + return nil + } + }, + "unsupported filesystem topology": func(fixture *managedCacheFixture) { + fixture.deps.statfs = func(context.Context, int, *unix.Statfs_t) error { + return nil + } + }, + } + + for name, mutate := range tests { + t.Run(name, func(t *testing.T) { + fixture := prepareManagedCacheFixture(t) + mutate(&fixture) + lease, err := pinManagedCache( + context.Background(), + fixture.deps, + fixture.policy, + fixture.caller, + fixture.locator, + ) + require.Error(t, err) + assert.Nil(t, lease) + requireManagedCacheDomainErrorCode(t, err, errs.CodeValidationFailed) + }) + } +} + +func TestPinManagedCachePreservesPrimaryErrorWhenCleanupFails(t *testing.T) { + t.Parallel() + // Rationale: A descriptor close failure is partial-state evidence, not permission to replace the primary + // operation's stable DomainError identity. + + fixture := prepareManagedCacheFixture(t) + realOpenAt2 := fixture.deps.openAt2 + realClose := fixture.deps.close + openCalls := 0 + fixture.deps.openAt2 = func( + ctx context.Context, + parentFD int, + name string, + how *unix.OpenHow, + ) (int, error) { + openCalls++ + if openCalls == 2 { + return -1, errors.New("open component failed") + } + return realOpenAt2(ctx, parentFD, name, how) + } + fixture.deps.close = func(ctx context.Context, fd int) error { + closeErr := realClose(ctx, fd) + return errors.Join(closeErr, errors.New("close retained descriptor failed")) + } + + lease, err := pinManagedCache( + context.Background(), + fixture.deps, + fixture.policy, + fixture.caller, + fixture.locator, + ) + require.Error(t, err) + assert.Nil(t, lease) + domainErr := errs.AsDomainError(err) + require.NotNil(t, domainErr) + assert.Equal(t, errs.CodeVMAtomicFailed, domainErr.Code) + assert.Contains(t, domainErr.Message, "open managed cache component") + assert.Contains(t, domainErr.Message, "close rejected managed cache descriptors") + assert.Equal(t, true, domainErr.Details["managed_cache_descriptor_cleanup_failed"]) +} + +func TestManagedCacheLeaseClosesDescriptorsInReverseOrder(t *testing.T) { + t.Parallel() + + fixture := prepareManagedCacheFixture(t) + realOpen := fixture.deps.open + realOpenAt2 := fixture.deps.openAt2 + realClose := fixture.deps.close + opened := make([]int, 0, len(fixture.locator.components)+1) + closed := make([]int, 0, len(fixture.locator.components)+1) + fixture.deps.open = func(ctx context.Context, path string, flags int, mode uint32) (int, error) { + fd, err := realOpen(ctx, path, flags, mode) + if err == nil { + opened = append(opened, fd) + } + return fd, err + } + fixture.deps.openAt2 = func(ctx context.Context, parentFD int, name string, how *unix.OpenHow) (int, error) { + fd, err := realOpenAt2(ctx, parentFD, name, how) + if err == nil { + opened = append(opened, fd) + } + return fd, err + } + fixture.deps.close = func(ctx context.Context, fd int) error { + closed = append(closed, fd) + return realClose(ctx, fd) + } + lease, err := pinManagedCache( + context.Background(), + fixture.deps, + fixture.policy, + fixture.caller, + fixture.locator, + ) + require.NoError(t, err) + require.NotNil(t, lease) + expected := append([]int(nil), opened...) + for left, right := 0, len(expected)-1; left < right; left, right = left+1, right-1 { + expected[left], expected[right] = expected[right], expected[left] + } + + require.NoError(t, lease.Release(context.Background())) + if diff := cmp.Diff(expected, closed); diff != "" { + t.Errorf("descriptor close order mismatch (-want +got):\n%s", diff) + } +} + +func TestPinManagedCacheCancellationAfterAcquisitionClosesDescriptors(t *testing.T) { + t.Parallel() + + fixture := prepareManagedCacheFixture(t) + realOpen := fixture.deps.open + realOpenAt2 := fixture.deps.openAt2 + realClose := fixture.deps.close + opened := make([]int, 0, 2) + closed := make([]int, 0, 2) + cleanupUncancelled := true + ctx, cancel := context.WithCancel(context.Background()) + fixture.deps.open = func(ctx context.Context, path string, flags int, mode uint32) (int, error) { + fd, err := realOpen(ctx, path, flags, mode) + if err == nil { + opened = append(opened, fd) + } + return fd, err + } + fixture.deps.openAt2 = func(ctx context.Context, parentFD int, name string, how *unix.OpenHow) (int, error) { + fd, err := realOpenAt2(ctx, parentFD, name, how) + if err == nil { + opened = append(opened, fd) + cancel() + } + return fd, err + } + fixture.deps.close = func(ctx context.Context, fd int) error { + cleanupUncancelled = cleanupUncancelled && ctx.Err() == nil + closed = append(closed, fd) + return realClose(ctx, fd) + } + + lease, err := pinManagedCache(ctx, fixture.deps, fixture.policy, fixture.caller, fixture.locator) + require.Error(t, err) + assert.Nil(t, lease) + assert.ErrorIs(t, err, context.Canceled) + expected := append([]int(nil), opened...) + for left, right := 0, len(expected)-1; left < right; left, right = left+1, right-1 { + expected[left], expected[right] = expected[right], expected[left] + } + if diff := cmp.Diff(expected, closed); diff != "" { + t.Errorf("cancellation close order mismatch (-want +got):\n%s", diff) + } + assert.True(t, cleanupUncancelled) +} + +func TestManagedCacheFilesystemPolicyIsClosed(t *testing.T) { + t.Parallel() + + allowed := []int64{ + unix.EXT4_SUPER_MAGIC, + unix.XFS_SUPER_MAGIC, + unix.BTRFS_SUPER_MAGIC, + unix.F2FS_SUPER_MAGIC, + unix.BCACHEFS_SUPER_MAGIC, + unix.TMPFS_MAGIC, + managedCacheZFSSuperMagic, + } + for _, filesystemType := range allowed { + assert.True(t, isSupportedManagedCacheFilesystem(filesystemType)) + } + + rejected := []int64{ + unix.FUSE_SUPER_MAGIC, + unix.NFS_SUPER_MAGIC, + unix.CIFS_SUPER_MAGIC, + unix.OVERLAYFS_SUPER_MAGIC, + unix.AUTOFS_SUPER_MAGIC, + } + for _, filesystemType := range rejected { + assert.False(t, isSupportedManagedCacheFilesystem(filesystemType)) + } +} + +func TestPinManagedCacheRequiresOpenat2(t *testing.T) { + t.Parallel() + + fixture := prepareManagedCacheFixture(t) + fixture.deps.openAt2 = func(context.Context, int, string, *unix.OpenHow) (int, error) { + return -1, unix.ENOSYS + } + lease, err := pinManagedCache( + context.Background(), + fixture.deps, + fixture.policy, + fixture.caller, + fixture.locator, + ) + require.Error(t, err) + assert.Nil(t, lease) + requireManagedCacheDomainErrorCode(t, err, errs.CodeVMAtomicFailed) + assert.ErrorIs(t, err, unix.ENOSYS) +} + +func TestManagedCacheLeaseReleaseIsCheckedAndUncancelled(t *testing.T) { + t.Parallel() + + type cleanupContextKey struct{} + fixture := prepareManagedCacheFixture(t) + realClose := fixture.deps.close + contextObserved := false + fixture.deps.close = func(ctx context.Context, fd int) error { + if ctx.Value(cleanupContextKey{}) == "release" && ctx.Err() == nil { + contextObserved = true + } + closeErr := realClose(ctx, fd) + return errors.Join(closeErr, errors.New("release close failed")) + } + lease, err := pinManagedCache( + context.Background(), + fixture.deps, + fixture.policy, + fixture.caller, + fixture.locator, + ) + require.NoError(t, err) + require.NotNil(t, lease) + + ctx, cancel := context.WithCancel(context.WithValue(context.Background(), cleanupContextKey{}, "release")) + cancel() + err = lease.Release(ctx) + require.Error(t, err) + assert.True(t, contextObserved) + domainErr := errs.AsDomainError(err) + require.NotNil(t, domainErr) + assert.Equal(t, errs.CodeVMAtomicFailed, domainErr.Code) + assert.Equal(t, true, domainErr.Details["managed_cache_descriptor_cleanup_failed"]) + assert.Empty(t, lease.retained) + assert.Equal(t, -1, lease.cacheFD) +} + +func TestAppendManagedCacheCleanupErrorPreservesPrimaryDomainMetadata(t *testing.T) { + t.Parallel() + + primary := errs.New( + errs.CodeValidationFailed, + "primary failure", + errs.WithClass(errs.ClassConflict), + errs.WithEntity(testVMID), + errs.WithDetails(map[string]any{"record_registered": true}), + ) + joined := appendManagedCacheCleanupError(primary, "close cache descriptors", errors.New("close failed")) + domainErr := errs.AsDomainError(joined) + require.NotNil(t, domainErr) + assert.Same(t, primary, domainErr) + assert.Equal(t, errs.CodeValidationFailed, domainErr.Code) + assert.Equal(t, errs.ClassConflict, domainErr.Class) + assert.Equal(t, testVMID, domainErr.Entity) + assert.Equal(t, true, domainErr.Details["record_registered"]) + assert.Equal(t, true, domainErr.Details["managed_cache_descriptor_cleanup_failed"]) +} + +func TestPinManagedCacheHonorsPreCancelledContext(t *testing.T) { + t.Parallel() + + fixture := prepareManagedCacheFixture(t) + openCalled := false + fixture.deps.open = func(context.Context, string, int, uint32) (int, error) { + openCalled = true + return -1, errors.New("unexpected open") + } + ctx, cancel := context.WithCancel(context.Background()) + cancel() + + lease, err := pinManagedCache(ctx, fixture.deps, fixture.policy, fixture.caller, fixture.locator) + require.Error(t, err) + assert.Nil(t, lease) + assert.ErrorIs(t, err, context.Canceled) + assert.False(t, openCalled) +} + +type managedCacheFixture struct { + deps managedCacheDeps + policy managedCachePolicy + caller instanceCaller + locator managedCacheLocator + root string + ownerDir string + cacheDir string + ownerInode uint64 + cacheInode uint64 +} + +func prepareManagedCacheFixture(t *testing.T) managedCacheFixture { + t.Helper() + + root := t.TempDir() + require.NoError(t, os.Chmod(root, 0700)) + homeDir := filepath.Join(root, "home") + ownerDir := filepath.Join(homeDir, "caller") + cacheDir := filepath.Join(ownerDir, "mvmctl") + require.NoError(t, os.Mkdir(homeDir, 0755)) + require.NoError(t, os.Mkdir(ownerDir, 0700)) + require.NoError(t, os.Mkdir(cacheDir, 0700)) + + callerUID := uint32(os.Geteuid()) + if callerUID == 0 { + callerUID = testAuthorityUID + } + ownerInode := inodeForManagedCacheTestPath(t, ownerDir) + cacheInode := inodeForManagedCacheTestPath(t, cacheDir) + deps := realManagedCacheDeps() + if callerUID != uint32(os.Geteuid()) { + realStatx := deps.statx + deps.statx = func(ctx context.Context, fd, flags, mask int, stat *unix.Statx_t) error { + if err := realStatx(ctx, fd, flags, mask, stat); err != nil { + return err + } + if stat.Ino == ownerInode || stat.Ino == cacheInode { + stat.Uid = callerUID + } + return nil + } + } + locator, err := parseManagedCacheLocator("/home/caller/mvmctl") + require.NoError(t, err) + return managedCacheFixture{ + deps: deps, + policy: managedCachePolicy{rootPath: root, trustedUID: uint32(os.Geteuid())}, + caller: instanceCaller{uid: callerUID}, + locator: locator, + root: root, + ownerDir: ownerDir, + cacheDir: cacheDir, + ownerInode: ownerInode, + cacheInode: cacheInode, + } +} + +func inodeForManagedCacheTestPath(t *testing.T, path string) uint64 { + t.Helper() + + var stat unix.Stat_t + require.NoError(t, unix.Stat(path, &stat)) + return stat.Ino +} + +func requireManagedCacheDomainErrorCode(t *testing.T, err error, code errs.Code) { + t.Helper() + + domainErr := errs.AsDomainError(err) + require.NotNil(t, domainErr) + assert.Equal(t, code, domainErr.Code) +} From 519da7a38620dfad97f556e2014dd74313a8ed79 Mon Sep 17 00:00:00 2001 From: AlanD20 Date: Fri, 28 Aug 2026 23:23:59 +0200 Subject: [PATCH 0049/1457] docs: record managed cache pinning foundation --- CHANGELOG.md | 25 ++++++++++++++++--- docs/DEPENDENCIES.md | 6 +++++ docs/REFERENCES.md | 6 ++++- ...owned-single-binary-privileged-dispatch.md | 11 ++++++++ tasks/plan.md | 3 +++ tasks/todo.md | 5 ++++ 6 files changed, 51 insertions(+), 5 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index d115d0d3..58fb0c40 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,9 +9,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 > **Security release status:** `0.3.0` is still under development. The canonical Jailer, cgroup policy, routed > service-access policy, root-owned system installer, early privileged-dispatch foundation, and private root-owned VM -> authority substrate described below are implemented. The mandatory per-VM network namespace, nftables-only `traffic` -> policy, exact VM-to-VM `exec` policy, final marker-only sudo policy, and remaining typed privileged operations are not -> implemented yet; see +> authority and managed-cache pinning substrates described below are implemented. The mandatory per-VM network +> namespace, nftables-only `traffic` policy, exact VM-to-VM `exec` policy, final marker-only sudo policy, and remaining +> typed privileged operations are not implemented yet; see > **Security work still pending** before treating this version as release-ready. ### Added @@ -68,6 +68,19 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - This substrate is private and not yet wired into public VM lifecycle operations; that integration remains a release blocker below. +#### Descriptor-pinned managed-cache substrate + +- Added a private typed `MVM_CACHE_DIR` locator and lease under `internal/service/jailer`. Root opens the fixed + filesystem root once, traverses canonical components with beneath/no-symlink/no-magic-link `openat2`, and retains the + complete descriptor chain. +- Caller ownership and safe modes use synchronized metadata. Cache identity records device, inode, and a typed unique, + legacy, or unavailable mount ID; later re-pin rejects any mismatch. +- The final cache root accepts ext2/ext3/ext4, XFS, Btrfs, F2FS, bcachefs, tmpfs, or ZFS. FUSE, network, + overlay/stacked, and automount cache roots fail closed. Ancestor mounts are constrained by descriptor traversal, + owner/mode, and automount checks without unnecessarily requiring the cache filesystem allowlist. +- This is a private foundation only. The privileged request, root-owned instance record, individual resource leaves, + and public VM lifecycle are not wired to it yet and remain release blockers below. + #### Installed release-candidate qualification - For T1/T2, the Python system-test runner now treats `MVM_BINARY` as a distinct compatible outer controller and `MVM_CANDIDATE_BINARY` as the artifact under test. It rejects path, symlink, and hard-link aliases between them and @@ -175,6 +188,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 is receiver-owned. The public CLI did not expose equivalent flags. - Firecracker/cloud-init/console filename settings are removed. Existing `rootfs.`, alternate snapshot leaf, or custom runtime filename layouts are not adopted; clean-install VMs and snapshots use the fixed v0.3 names. +- Custom `MVM_CACHE_DIR` remains supported, but the final v0.3 privileged path requires its resource-bearing + filesystem to be ext2/ext3/ext4, XFS, Btrfs, F2FS, bcachefs, tmpfs, or ZFS. FUSE, remote, overlay/stacked, and + automount cache roots are intentionally unsupported. ### Fixed @@ -202,7 +218,8 @@ The following items are release blockers and are intentionally not described abo process identity, and per-VM lock. - Add root-owned managed-network identity and global capacity authority; user SQLite state must not authorize host-global namespace, link, firewall, process, mount, cgroup, or admission decisions. -- Replace path-validation-then-reopen behavior with descriptor-pinned managed-resource access and private mount namespaces. +- Finish wiring the implemented managed-cache pinning foundation to the privileged protocol and instance record, then + add descriptor-pinned operation-specific resources and private mount namespaces. - Split root-only locks/handshakes beneath `/run/mvmctl/authority` from caller-owned socket/PID output directories beneath `/run/mvmctl/runtime`; stop mounting the whole cache VM directory. Pin persistent config/output leaves individually, truncate enabled `0600` log/console/metrics files only after durable launch registration, and retain them after stop. diff --git a/docs/DEPENDENCIES.md b/docs/DEPENDENCIES.md index a33f503a..de525e60 100644 --- a/docs/DEPENDENCIES.md +++ b/docs/DEPENDENCIES.md @@ -174,6 +174,12 @@ Only needed for `mvm kernel pull --type official --clean-build` (building the of - **Virtualization**: VT-x (Intel) or AMD-V must be enabled in BIOS/UEFI. - **Permissions**: The user must be in the `mvm` group (created by `mvm host init`). +### Managed Cache Storage + +For v0.3 privileged operations, `MVM_CACHE_DIR` must reside on ext2/ext3/ext4, XFS, Btrfs, F2FS, bcachefs, tmpfs, or +ZFS. FUSE, remote, overlay/stacked, and automount cache roots are unsupported. Custom local paths remain supported and +are pinned independently of unrelated ancestor filesystems. + ### Firecracker Jailer `mvm bin pull` downloads and checksum-verifies a matching Firecracker/Jailer release pair. The privileged Jailer service installs the pair into the root-owned trusted store; users do not install or invoke Jailer directly. Source-built pairs remain untrusted and cannot launch production VMs. diff --git a/docs/REFERENCES.md b/docs/REFERENCES.md index 8d2e25be..e878614f 100644 --- a/docs/REFERENCES.md +++ b/docs/REFERENCES.md @@ -727,7 +727,7 @@ The security model applies to the net cloud-init mode only: | Variable | Description | Default | |----------|-------------|---------| -| `MVM_CACHE_DIR` | Override cache directory | `~/.cache/mvmctl` | +| `MVM_CACHE_DIR` | Override the managed cache directory | `~/.cache/mvmctl` | | `MVM_CONFIG_DIR` | Override config directory | `~/.config/mvmctl` | | `MVM_LOG_LEVEL` | Set log verbosity (`DEBUG`, `INFO`, `WARN`, `ERROR`) | `WARN` | | `MVM_ASSET_MIRROR` | Local mirror directory for downloaded assets | (not set) | @@ -781,6 +781,10 @@ The VM and snapshot leaf names above are compiled and are not configurable. `MVM cache root. API/vsock/console sockets and PID mirrors remain in the VM cache directory in the transitional Jailer path; v0.3 release work will move them to `/run/mvmctl/runtime//` and stop mounting the whole VM directory. +The v0.3 privileged path accepts a managed cache root on ext2/ext3/ext4, XFS, Btrfs, F2FS, bcachefs, tmpfs, or ZFS. +FUSE, remote, overlay/stacked, and automount cache roots are unsupported. The descriptor-pinned cache foundation is +implemented, but privileged lifecycle wiring remains release work tracked in `tasks/todo.md`. + Canonical launch assets and jail state are outside the user cache: ```text diff --git a/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md b/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md index 60428445..c6870164 100644 --- a/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md +++ b/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md @@ -235,6 +235,17 @@ owners/modes, and unsupported mount topology, and pins the directory before reso inputs are typed IDs, closed format enums, and presence/access intent. A request never supplies a kernel, rootfs, snapshot, volume, config, PID, socket, log, mount-target, or temporary-file path. +The resource-bearing cache root must use a kernel-backed local filesystem from this closed set: ext2/ext3/ext4, XFS, +Btrfs, F2FS, bcachefs, tmpfs, or ZFS. FUSE, remote filesystems, overlay or other stacked filesystems, and automount cache +roots are rejected. The filesystem allowlist applies to the final pinned cache root, not `/` or unrelated ancestor +mounts. Every ancestor is still opened descriptor-relatively and checked for safe ownership, mode, and automount +topology. Bind, idmapped, shared, or private views of an approved underlying filesystem remain supported because their +authority comes from retained descriptors and exact identity checks, not the pathname or propagation mode. + +Identity inspection requests synchronized `statx` metadata. It prefers `STATX_MNT_ID_UNIQUE`, records the identity kind, +and falls back to ordinary mount ID or explicit unavailability on older kernels. A later comparison includes the kind so +ordinary and unique mount identities are never treated as interchangeable. + This exception is deliberately narrower than a generic path interface. Dropping custom cache roots would break the documented large-filesystem use case, while a persistent root-owned path registry would still need safe pathname resolution after reboot. Instead, each live instance record binds the pinned cache root's stable identity (device, diff --git a/tasks/plan.md b/tasks/plan.md index 1eec483d..ac2767be 100644 --- a/tasks/plan.md +++ b/tasks/plan.md @@ -33,6 +33,9 @@ continue without silently deleting it. protocol failures, and bounded safe `DomainError` preservation. - Private root-owned VM ownership/lifecycle records with strict codecs, descriptor-relative atomic storage, exact release references, global VM-ID claims, and release/index/VM locking. +- Private managed-cache locator and lease foundation with descriptor-relative component traversal, synchronized + owner/mode/identity checks, an explicit local-filesystem policy, retained descriptors, and identity-checked re-pin. + It is not yet carried by a privileged action or bound into the instance record. - ADR-0016 for the single-binary privilege boundary. The action switch is intentionally still empty and the sudoers policy remains transitional. The routed policy, dual diff --git a/tasks/todo.md b/tasks/todo.md index 60488bb7..58a734d9 100644 --- a/tasks/todo.md +++ b/tasks/todo.md @@ -139,6 +139,11 @@ under `internal/service/jailer/`. No handler, transport, CLI, API, core-domain, - [ ] Preserve custom `MVM_CACHE_DIR` through one typed managed-cache locator; pin it from `/`, verify caller ownership, safe components/mode/mount topology, and reject every raw effect path. + - [x] Add the private canonical locator and descriptor lease: component-wise `openat2`, synchronized ownership/mode/ + identity inspection, a closed local-filesystem policy on the final cache root, unique/legacy/unavailable mount-ID + typing, identity-checked re-pin, and checked reverse cleanup. + - [ ] Carry the locator through the privileged request, remove the transitional raw effect paths, and bind the lease + to operation-specific resource descriptors. The private foundation is not yet a production handler. - [ ] Bind the pinned cache device/inode/mount identity into the root-owned instance record and require later operations to re-pin the same identity. - [ ] Land canonical producer names before descriptor pinning: From 924cbede29694851e47cb1cd9870fb122969941a Mon Sep 17 00:00:00 2001 From: AlanD20 Date: Fri, 28 Aug 2026 23:33:03 +0200 Subject: [PATCH 0050/1457] docs: define base launch resource policy --- ...-owned-single-binary-privileged-dispatch.md | 18 ++++++++++++++++++ tasks/todo.md | 6 ++++++ 2 files changed, 24 insertions(+) diff --git a/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md b/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md index c6870164..b80f551d 100644 --- a/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md +++ b/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md @@ -276,6 +276,24 @@ The v0.3 canonical persistent layout is: | Durable image | `images/.zst` | | Image staging | `images/staging//{source.raw,rootfs.img}` | +Base VM launch admission uses a closed receiver policy before any descriptor is handed to the launch implementation. +The request supplies only the VM ID, kernel ID, cloud-init presence, and the expected rootfs size. Root derives every +component and opens it below the pinned cache descriptor with beneath/no-symlink/no-magic-link/no-cross-mount +resolution. The base resource limits are: + +| Resource | Required size | Access and mode policy | +|---|---:|---| +| VM rootfs | 128 MiB through 16 TiB, exactly equal to the bounded expected size | Caller-owned, one-link regular file; owner read/write; no group/world write, execute, or special bits | +| Firecracker configuration | 1 byte through 1 MiB | Caller-owned, one-link regular file; owner read; no group/world write, execute, or special bits | +| Kernel | 1 byte through 1 GiB | Caller-owned, one-link regular file; owner read; execute bits permitted; no group/world write or special bits | +| Cloud-init ISO | 1 byte through 256 MiB | Caller-owned, one-link regular file; owner read; no group/world write, execute, or special bits | + +The expected rootfs size is an equality check inside the receiver's hard range; it never raises the hard maximum. These +checks authorize only the base launch set. Snapshot restore, volume attachment, snapshot creation, output creation, and +image provisioning each require their own closed policy before their descriptors can join a launch or mutation lease. +In particular, a QCOW2 file's physical length is not its virtual capacity, so a future volume policy cannot treat +`statx` size as QCOW2 capacity or invoke an unrestricted image tool as root to compensate. + Sockets and PID mirrors are not persistent cache artifacts. Their host layout is fixed beneath the root-controlled ephemeral runtime tree: diff --git a/tasks/todo.md b/tasks/todo.md index 58a734d9..0ddbdb9b 100644 --- a/tasks/todo.md +++ b/tasks/todo.md @@ -165,6 +165,12 @@ under `internal/service/jailer/`. No handler, transport, CLI, API, core-domain, ID-scoped image-provisioning staging subtree. - [ ] Define operation-specific non-empty/maximum-size and access-mode policies before coding; a caller- or DB-provided expected size is an equality check, not authorization for an unbounded resource. + - [x] Define base VM launch admission: rootfs 128 MiB through 16 TiB with exact bounded size equality, config 1 byte + through 1 MiB, kernel 1 byte through 1 GiB, and optional cloud-init ISO 1 byte through 256 MiB. All are caller-owned, + one-link regular files with resource-specific owner access and no group/world write or special bits. + - [ ] Define snapshot restore memory/state, volume count/format/capacity, writable snapshot creation, persistent output, + and image-provisioning policies in their owning increments. QCOW2 physical length must not be confused with virtual + capacity or inspected by an unrestricted root subprocess. - [ ] Open user cache/VM/image/kernel/volume/snapshot resources once with beneath/no-symlink/no-magic-link resolution. - [ ] Verify caller ownership, file type, fixed basenames, size, and allowed access mode on pinned descriptors. - [ ] Never validate a path and later reopen it by pathname. From a2cac0b254e31e8f00c88ebc96bd2fc6450edc7b Mon Sep 17 00:00:00 2001 From: AlanD20 Date: Fri, 28 Aug 2026 23:38:10 +0200 Subject: [PATCH 0051/1457] feat: validate typed VM launch resources --- .../service/jailer/managed_vm_resources.go | 83 ++++++++ .../jailer/managed_vm_resources_test.go | 185 ++++++++++++++++++ 2 files changed, 268 insertions(+) create mode 100644 internal/service/jailer/managed_vm_resources.go create mode 100644 internal/service/jailer/managed_vm_resources_test.go diff --git a/internal/service/jailer/managed_vm_resources.go b/internal/service/jailer/managed_vm_resources.go new file mode 100644 index 00000000..b1519ed6 --- /dev/null +++ b/internal/service/jailer/managed_vm_resources.go @@ -0,0 +1,83 @@ +package jailer + +import "mvmctl/pkg/errs" + +const ( + managedVMRootfsMinBytes uint64 = 128 << 20 + managedVMRootfsMaxBytes uint64 = 16 << 40 +) + +type vmResourceID string + +type kernelResourceID string + +type cloudInitPresence uint8 + +const ( + cloudInitAbsent cloudInitPresence = iota + 1 + cloudInitPresent +) + +type vmLaunchResourceSpec struct { + vmID vmResourceID + kernelID kernelResourceID + expectedRootfsBytes uint64 + cloudInit cloudInitPresence +} + +func newVMLaunchResourceSpec( + vmID string, + kernelID string, + expectedRootfsBytes uint64, + cloudInit cloudInitPresence, +) (vmLaunchResourceSpec, error) { + spec := vmLaunchResourceSpec{ + vmID: vmResourceID(vmID), + kernelID: kernelResourceID(kernelID), + expectedRootfsBytes: expectedRootfsBytes, + cloudInit: cloudInit, + } + if err := validateVMLaunchResourceSpec(spec); err != nil { + return vmLaunchResourceSpec{}, err + } + return spec, nil +} + +func validateVMLaunchResourceSpec(spec vmLaunchResourceSpec) error { + if !isCanonicalManagedResourceID(string(spec.vmID), 32) { + return managedVMResourceValidationError( + "managed VM ID must be 32 lowercase hexadecimal characters", + ) + } + if !isCanonicalManagedResourceID(string(spec.kernelID), 64) { + return managedVMResourceValidationError( + "managed kernel ID must be 64 lowercase hexadecimal characters", + ) + } + if spec.expectedRootfsBytes < managedVMRootfsMinBytes || + spec.expectedRootfsBytes > managedVMRootfsMaxBytes { + return managedVMResourceValidationError("managed VM rootfs size is outside the supported range") + } + switch spec.cloudInit { + case cloudInitAbsent, cloudInitPresent: + return nil + default: + return managedVMResourceValidationError("managed cloud-init presence is invalid") + } +} + +func isCanonicalManagedResourceID(value string, length int) bool { + if len(value) != length { + return false + } + for index := 0; index < len(value); index++ { + if (value[index] < '0' || value[index] > '9') && (value[index] < 'a' || value[index] > 'f') { + return false + } + } + return true +} + +func managedVMResourceValidationError(message string) *errs.DomainError { + return errs.New(errs.CodeValidationFailed, message) +} diff --git a/internal/service/jailer/managed_vm_resources_test.go b/internal/service/jailer/managed_vm_resources_test.go new file mode 100644 index 00000000..3d12eff3 --- /dev/null +++ b/internal/service/jailer/managed_vm_resources_test.go @@ -0,0 +1,185 @@ +package jailer + +import ( + "strings" + "testing" + + "github.com/google/go-cmp/cmp" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "mvmctl/pkg/errs" +) + +const ( + testManagedVMID = "0123456789abcdef0123456789abcdef" + testManagedKernelID = "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef" + testManagedRootfsMinBytes = uint64(128 << 20) + testManagedRootfsMaxBytes = uint64(16 << 40) +) + +func TestNewVMLaunchResourceSpecAcceptsClosedBaseSelection(t *testing.T) { + t.Parallel() + + tests := []struct { + name string + rootfs uint64 + cloudInit cloudInitPresence + }{ + {name: "minimum rootfs without cloud init", rootfs: testManagedRootfsMinBytes, cloudInit: cloudInitAbsent}, + {name: "maximum rootfs with cloud init", rootfs: testManagedRootfsMaxBytes, cloudInit: cloudInitPresent}, + } + + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + + got, err := newVMLaunchResourceSpec( + testManagedVMID, + testManagedKernelID, + tc.rootfs, + tc.cloudInit, + ) + require.NoError(t, err) + + want := vmLaunchResourceSpec{ + vmID: vmResourceID(testManagedVMID), + kernelID: kernelResourceID(testManagedKernelID), + expectedRootfsBytes: tc.rootfs, + cloudInit: tc.cloudInit, + } + if diff := cmp.Diff(want, got, cmp.AllowUnexported(vmLaunchResourceSpec{})); diff != "" { + t.Errorf("newVMLaunchResourceSpec() mismatch (-want +got):\n%s", diff) + } + }) + } +} + +func TestNewVMLaunchResourceSpecRejectsInvalidSelection(t *testing.T) { + t.Parallel() + + tests := []struct { + name string + vmID string + kernelID string + rootfs uint64 + cloudInit cloudInitPresence + wantMessage string + }{ + { + name: "empty VM ID", + vmID: "", + kernelID: testManagedKernelID, + rootfs: testManagedRootfsMinBytes, + cloudInit: cloudInitAbsent, + wantMessage: "managed VM ID must be 32 lowercase hexadecimal characters", + }, + { + name: "uppercase VM ID", + vmID: strings.ToUpper(testManagedVMID), + kernelID: testManagedKernelID, + rootfs: testManagedRootfsMinBytes, + cloudInit: cloudInitAbsent, + wantMessage: "managed VM ID must be 32 lowercase hexadecimal characters", + }, + { + name: "non-hexadecimal VM ID", + vmID: strings.Repeat("g", 32), + kernelID: testManagedKernelID, + rootfs: testManagedRootfsMinBytes, + cloudInit: cloudInitAbsent, + wantMessage: "managed VM ID must be 32 lowercase hexadecimal characters", + }, + { + name: "short kernel ID", + vmID: testManagedVMID, + kernelID: testManagedKernelID[:63], + rootfs: testManagedRootfsMinBytes, + cloudInit: cloudInitAbsent, + wantMessage: "managed kernel ID must be 64 lowercase hexadecimal characters", + }, + { + name: "uppercase kernel ID", + vmID: testManagedVMID, + kernelID: strings.ToUpper(testManagedKernelID), + rootfs: testManagedRootfsMinBytes, + cloudInit: cloudInitAbsent, + wantMessage: "managed kernel ID must be 64 lowercase hexadecimal characters", + }, + { + name: "rootfs below minimum", + vmID: testManagedVMID, + kernelID: testManagedKernelID, + rootfs: testManagedRootfsMinBytes - 1, + cloudInit: cloudInitAbsent, + wantMessage: "managed VM rootfs size is outside the supported range", + }, + { + name: "rootfs above maximum", + vmID: testManagedVMID, + kernelID: testManagedKernelID, + rootfs: testManagedRootfsMaxBytes + 1, + cloudInit: cloudInitAbsent, + wantMessage: "managed VM rootfs size is outside the supported range", + }, + { + name: "zero cloud-init presence", + vmID: testManagedVMID, + kernelID: testManagedKernelID, + rootfs: testManagedRootfsMinBytes, + cloudInit: 0, + wantMessage: "managed cloud-init presence is invalid", + }, + { + name: "unknown cloud-init presence", + vmID: testManagedVMID, + kernelID: testManagedKernelID, + rootfs: testManagedRootfsMinBytes, + cloudInit: cloudInitPresence(3), + wantMessage: "managed cloud-init presence is invalid", + }, + } + + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + + _, err := newVMLaunchResourceSpec(tc.vmID, tc.kernelID, tc.rootfs, tc.cloudInit) + require.Error(t, err) + domainErr := errs.AsDomainError(err) + require.NotNil(t, domainErr) + assert.Equal(t, errs.CodeValidationFailed, domainErr.Code) + assert.Equal(t, tc.wantMessage, domainErr.Message) + }) + } +} + +func TestValidateVMLaunchResourceSpecRejectsForgedValues(t *testing.T) { + t.Parallel() + + tests := map[string]vmLaunchResourceSpec{ + "zero value": {}, + "forged VM ID": { + vmID: vmResourceID(strings.Repeat("g", 32)), + kernelID: kernelResourceID(testManagedKernelID), + expectedRootfsBytes: testManagedRootfsMinBytes, + cloudInit: cloudInitAbsent, + }, + "forged kernel ID": { + vmID: vmResourceID(testManagedVMID), + kernelID: kernelResourceID(strings.Repeat("g", 64)), + expectedRootfsBytes: testManagedRootfsMinBytes, + cloudInit: cloudInitAbsent, + }, + } + + for name, spec := range tests { + t.Run(name, func(t *testing.T) { + t.Parallel() + + err := validateVMLaunchResourceSpec(spec) + require.Error(t, err) + assert.Equal(t, errs.CodeValidationFailed, errs.AsDomainError(err).Code) + }) + } +} From cecdc5471c9130389c3851450f8b6b774e51c783 Mon Sep 17 00:00:00 2001 From: AlanD20 Date: Sat, 29 Aug 2026 00:11:25 +0200 Subject: [PATCH 0052/1457] feat: pin base VM launch resources --- internal/service/jailer/managed_cache.go | 4 + internal/service/jailer/managed_cache_test.go | 1 + .../service/jailer/managed_vm_resources.go | 436 ++++++++- .../managed_vm_resources_security_test.go | 898 ++++++++++++++++++ .../jailer/managed_vm_resources_test.go | 246 ++++- 5 files changed, 1580 insertions(+), 5 deletions(-) create mode 100644 internal/service/jailer/managed_vm_resources_security_test.go diff --git a/internal/service/jailer/managed_cache.go b/internal/service/jailer/managed_cache.go index 35264180..ab954d9f 100644 --- a/internal/service/jailer/managed_cache.go +++ b/internal/service/jailer/managed_cache.go @@ -62,6 +62,7 @@ type managedCacheDeps struct { type managedCacheLease struct { deps managedCacheDeps cacheFD int + ownerUID uint32 identity managedCacheIdentity retained []int } @@ -184,6 +185,7 @@ func pinManagedCache( lease := &managedCacheLease{ deps: deps, cacheFD: parentFD, + ownerUID: caller.uid, identity: identity, retained: retained, } @@ -386,6 +388,8 @@ func (lease *managedCacheLease) Release(ctx context.Context) error { err := closeManagedCacheFDs(ctx, lease.deps, lease.retained) lease.retained = nil lease.cacheFD = -1 + lease.ownerUID = 0 + lease.identity = managedCacheIdentity{} if err != nil { return managedCacheCleanupError("release managed cache descriptors", err) } diff --git a/internal/service/jailer/managed_cache_test.go b/internal/service/jailer/managed_cache_test.go index a8210025..cb1d32bc 100644 --- a/internal/service/jailer/managed_cache_test.go +++ b/internal/service/jailer/managed_cache_test.go @@ -94,6 +94,7 @@ func TestPinManagedCacheRetainsDescriptorIdentity(t *testing.T) { require.NotNil(t, lease) t.Cleanup(func() { require.NoError(t, lease.Release(context.Background())) }) require.NotEmpty(t, lease.retained) + assert.Equal(t, fixture.caller.uid, lease.ownerUID) assert.NotZero(t, lease.identity.inode) assert.NotZero(t, lease.identity.deviceMajor|lease.identity.deviceMinor) diff --git a/internal/service/jailer/managed_vm_resources.go b/internal/service/jailer/managed_vm_resources.go index b1519ed6..0ceaf2a2 100644 --- a/internal/service/jailer/managed_vm_resources.go +++ b/internal/service/jailer/managed_vm_resources.go @@ -1,10 +1,35 @@ package jailer -import "mvmctl/pkg/errs" +import ( + "context" + "errors" + + "golang.org/x/sys/unix" + + "mvmctl/internal/infra" + "mvmctl/pkg/errs" +) const ( - managedVMRootfsMinBytes uint64 = 128 << 20 - managedVMRootfsMaxBytes uint64 = 16 << 40 + managedVMRootfsMinBytes uint64 = 128 << 20 + managedVMRootfsMaxBytes uint64 = 16 << 40 + managedVMConfigMinBytes uint64 = 1 + managedVMConfigMaxBytes uint64 = 1 << 20 + managedVMKernelMinBytes uint64 = 1 + managedVMKernelMaxBytes uint64 = 1 << 30 + managedVMCloudInitMinBytes uint64 = 1 + managedVMCloudInitMaxBytes uint64 = 256 << 20 + + managedVMResourceDirectoryOpenFlags = unix.O_PATH | unix.O_DIRECTORY | unix.O_CLOEXEC | unix.O_NOFOLLOW + managedVMResourceFileOpenFlags = unix.O_PATH | unix.O_CLOEXEC | unix.O_NOFOLLOW + managedVMResourceResolveFlags = unix.RESOLVE_BENEATH | unix.RESOLVE_NO_SYMLINKS | + unix.RESOLVE_NO_MAGICLINKS | unix.RESOLVE_NO_XDEV + managedVMResourceStatxFlags = unix.AT_EMPTY_PATH | unix.AT_NO_AUTOMOUNT | unix.AT_SYMLINK_NOFOLLOW | + unix.AT_STATX_FORCE_SYNC + managedVMResourceRequiredDirectoryStat = unix.STATX_TYPE | unix.STATX_MODE | unix.STATX_UID | unix.STATX_INO + managedVMResourceRequiredFileStat = managedVMResourceRequiredDirectoryStat | unix.STATX_NLINK | + unix.STATX_SIZE + managedVMResourceStatxMask = unix.STATX_BASIC_STATS ) type vmResourceID string @@ -25,6 +50,26 @@ type vmLaunchResourceSpec struct { cloudInit cloudInitPresence } +type vmLaunchResourceLease struct { + deps managedCacheDeps + cacheIdentity managedCacheIdentity + ownerUID uint32 + rootfsFD int + configFD int + kernelFD int + cloudInitFD int + retained []int +} + +type managedVMRegularFilePolicy struct { + name string + minimumBytes uint64 + maximumBytes uint64 + exactBytes uint64 + requiredOwnerMode uint16 + allowExecute bool +} + func newVMLaunchResourceSpec( vmID string, kernelID string, @@ -66,6 +111,391 @@ func validateVMLaunchResourceSpec(spec vmLaunchResourceSpec) error { } } +func (cache *managedCacheLease) pinVMLaunchResources( + ctx context.Context, + spec vmLaunchResourceSpec, +) (_ *vmLaunchResourceLease, returnErr error) { + if err := validateActiveManagedCacheLease(cache); err != nil { + return nil, err + } + if err := validateVMLaunchResourceSpec(spec); err != nil { + return nil, err + } + if err := ctx.Err(); err != nil { + return nil, managedVMResourceAtomicError("pin managed VM launch resources", err) + } + + opened := make([]int, 0, 7) + defer func() { + if returnErr == nil { + return + } + if closeErr := closeManagedCacheFDs(ctx, cache.deps, opened); closeErr != nil { + returnErr = appendManagedVMResourceCleanupError( + returnErr, + "close rejected managed VM launch resources", + closeErr, + ) + } + }() + + vmsFD, err := openManagedVMResourceDirectory(ctx, cache.deps, cache.cacheFD, "vms", "VM store", cache.ownerUID) + if err != nil { + return nil, err + } + opened = append(opened, vmsFD) + vmFD, err := openManagedVMResourceDirectory( + ctx, + cache.deps, + vmsFD, + string(spec.vmID), + "VM directory", + cache.ownerUID, + ) + if err != nil { + return nil, err + } + opened = append(opened, vmFD) + + rootfsFD, err := openManagedVMResourceFile( + ctx, + cache.deps, + vmFD, + infra.VMRootfsFilename, + managedVMRegularFilePolicy{ + name: "VM rootfs", + minimumBytes: managedVMRootfsMinBytes, + maximumBytes: managedVMRootfsMaxBytes, + exactBytes: spec.expectedRootfsBytes, + requiredOwnerMode: 0600, + }, + cache.ownerUID, + ) + if err != nil { + return nil, err + } + opened = append(opened, rootfsFD) + configFD, err := openManagedVMResourceFile( + ctx, + cache.deps, + vmFD, + infra.VMFirecrackerConfigFilename, + managedVMRegularFilePolicy{ + name: "Firecracker configuration", + minimumBytes: managedVMConfigMinBytes, + maximumBytes: managedVMConfigMaxBytes, + requiredOwnerMode: 0400, + }, + cache.ownerUID, + ) + if err != nil { + return nil, err + } + opened = append(opened, configFD) + + cloudInitFD := -1 + if spec.cloudInit == cloudInitPresent { + cloudInitFD, err = openManagedVMResourceFile( + ctx, + cache.deps, + vmFD, + infra.VMCloudInitISOFilename, + managedVMRegularFilePolicy{ + name: "cloud-init ISO", + minimumBytes: managedVMCloudInitMinBytes, + maximumBytes: managedVMCloudInitMaxBytes, + requiredOwnerMode: 0400, + }, + cache.ownerUID, + ) + if err != nil { + return nil, err + } + opened = append(opened, cloudInitFD) + } + + kernelsFD, err := openManagedVMResourceDirectory( + ctx, + cache.deps, + cache.cacheFD, + "kernels", + "kernel store", + cache.ownerUID, + ) + if err != nil { + return nil, err + } + opened = append(opened, kernelsFD) + kernelFD, err := openManagedVMResourceFile( + ctx, + cache.deps, + kernelsFD, + string(spec.kernelID), + managedVMRegularFilePolicy{ + name: "kernel", + minimumBytes: managedVMKernelMinBytes, + maximumBytes: managedVMKernelMaxBytes, + requiredOwnerMode: 0400, + allowExecute: true, + }, + cache.ownerUID, + ) + if err != nil { + return nil, err + } + opened = append(opened, kernelFD) + + retained := make([]int, 0, len(cache.retained)+len(opened)) + retained = append(retained, cache.retained...) + retained = append(retained, opened...) + lease := &vmLaunchResourceLease{ + deps: cache.deps, + cacheIdentity: cache.identity, + ownerUID: cache.ownerUID, + rootfsFD: rootfsFD, + configFD: configFD, + kernelFD: kernelFD, + cloudInitFD: cloudInitFD, + retained: retained, + } + cache.retained = nil + cache.cacheFD = -1 + cache.ownerUID = 0 + cache.identity = managedCacheIdentity{} + opened = nil + return lease, nil +} + +func validateActiveManagedCacheLease(cache *managedCacheLease) error { + if cache == nil || cache.cacheFD < 0 || cache.ownerUID == 0 || len(cache.retained) == 0 || + cache.deps.openAt2 == nil || cache.deps.statx == nil || cache.deps.close == nil { + return managedVMResourceAtomicError("managed cache lease is not active", nil) + } + return nil +} + +func openManagedVMResourceDirectory( + ctx context.Context, + deps managedCacheDeps, + parentFD int, + component string, + name string, + ownerUID uint32, +) (int, error) { + fd, err := openManagedVMResourceComponent(ctx, deps, parentFD, component, managedVMResourceDirectoryOpenFlags, name) + if err != nil { + return -1, err + } + if err := inspectManagedVMResourceDirectory(ctx, deps, fd, name, ownerUID); err != nil { + if closeErr := deps.close(context.WithoutCancel(ctx), fd); closeErr != nil { + err = appendManagedVMResourceCleanupError(err, "close rejected managed VM directory", closeErr) + } + return -1, err + } + return fd, nil +} + +func openManagedVMResourceFile( + ctx context.Context, + deps managedCacheDeps, + parentFD int, + component string, + policy managedVMRegularFilePolicy, + ownerUID uint32, +) (int, error) { + fd, err := openManagedVMResourceComponent( + ctx, + deps, + parentFD, + component, + managedVMResourceFileOpenFlags, + policy.name, + ) + if err != nil { + return -1, err + } + if err := inspectManagedVMResourceFile(ctx, deps, fd, policy, ownerUID); err != nil { + if closeErr := deps.close(context.WithoutCancel(ctx), fd); closeErr != nil { + err = appendManagedVMResourceCleanupError(err, "close rejected managed VM file", closeErr) + } + return -1, err + } + return fd, nil +} + +func openManagedVMResourceComponent( + ctx context.Context, + deps managedCacheDeps, + parentFD int, + component string, + flags int, + name string, +) (int, error) { + if err := ctx.Err(); err != nil { + return -1, managedVMResourceAtomicError("open managed "+name, err) + } + how := unix.OpenHow{Flags: uint64(flags), Resolve: uint64(managedVMResourceResolveFlags)} + fd, err := deps.openAt2(ctx, parentFD, component, &how) + if err != nil { + return -1, classifyManagedVMResourceOpenError(name, err) + } + return fd, nil +} + +func inspectManagedVMResourceDirectory( + ctx context.Context, + deps managedCacheDeps, + fd int, + name string, + ownerUID uint32, +) error { + stat, err := statManagedVMResource(ctx, deps, fd, name, managedVMResourceRequiredDirectoryStat) + if err != nil { + return err + } + if stat.Mode&unix.S_IFMT != unix.S_IFDIR { + return managedVMResourceValidationError("managed " + name + " is not a directory") + } + if stat.Uid != ownerUID { + return managedVMResourceValidationError("managed " + name + " is not owned by the invoking user") + } + if stat.Mode&0700 != 0700 { + return managedVMResourceValidationError("managed " + name + " must grant owner read, write, and traversal") + } + if stat.Mode&0022 != 0 { + return managedVMResourceValidationError("managed " + name + " is writable by another user") + } + if stat.Mode&(unix.S_ISUID|unix.S_ISGID|unix.S_ISVTX) != 0 { + return managedVMResourceValidationError("managed " + name + " has unsupported special mode bits") + } + return nil +} + +func inspectManagedVMResourceFile( + ctx context.Context, + deps managedCacheDeps, + fd int, + policy managedVMRegularFilePolicy, + ownerUID uint32, +) error { + stat, err := statManagedVMResource(ctx, deps, fd, policy.name, managedVMResourceRequiredFileStat) + if err != nil { + return err + } + if stat.Mode&unix.S_IFMT != unix.S_IFREG { + return managedVMResourceValidationError("managed " + policy.name + " is not a regular file") + } + if stat.Uid != ownerUID { + return managedVMResourceValidationError("managed " + policy.name + " is not owned by the invoking user") + } + if stat.Nlink != 1 { + return managedVMResourceValidationError("managed " + policy.name + " must have exactly one link") + } + if stat.Mode&policy.requiredOwnerMode != policy.requiredOwnerMode { + return managedVMResourceValidationError("managed " + policy.name + " does not grant required owner access") + } + if stat.Mode&0022 != 0 { + return managedVMResourceValidationError("managed " + policy.name + " is writable by another user") + } + if stat.Mode&(unix.S_ISUID|unix.S_ISGID|unix.S_ISVTX) != 0 { + return managedVMResourceValidationError("managed " + policy.name + " has unsupported special mode bits") + } + if !policy.allowExecute && stat.Mode&0111 != 0 { + return managedVMResourceValidationError("managed " + policy.name + " has unsupported execute access") + } + if stat.Size < policy.minimumBytes || stat.Size > policy.maximumBytes { + return managedVMResourceValidationError("managed " + policy.name + " size is outside the supported range") + } + if policy.exactBytes != 0 && stat.Size != policy.exactBytes { + return managedVMResourceValidationError("managed " + policy.name + " size does not match the expected size") + } + return nil +} + +func statManagedVMResource( + ctx context.Context, + deps managedCacheDeps, + fd int, + name string, + requiredMask uint32, +) (unix.Statx_t, error) { + if err := ctx.Err(); err != nil { + return unix.Statx_t{}, managedVMResourceAtomicError("inspect managed "+name, err) + } + var stat unix.Statx_t + if err := deps.statx(ctx, fd, managedVMResourceStatxFlags, managedVMResourceStatxMask, &stat); err != nil { + return unix.Statx_t{}, managedVMResourceAtomicError("inspect managed "+name, err) + } + if stat.Mask&requiredMask != requiredMask || stat.Ino == 0 || (stat.Dev_major == 0 && stat.Dev_minor == 0) { + return unix.Statx_t{}, managedVMResourceValidationError("managed " + name + " has incomplete stable metadata") + } + if stat.Attributes_mask&unix.STATX_ATTR_AUTOMOUNT != 0 && stat.Attributes&unix.STATX_ATTR_AUTOMOUNT != 0 { + return unix.Statx_t{}, managedVMResourceValidationError( + "managed " + name + " uses unsupported automount topology", + ) + } + return stat, nil +} + +func classifyManagedVMResourceOpenError(name string, cause error) error { + message := "open managed " + name + if errors.Is(cause, unix.ELOOP) || errors.Is(cause, unix.ENOTDIR) || errors.Is(cause, unix.EXDEV) || + errors.Is(cause, unix.ENOENT) || errors.Is(cause, unix.EACCES) { + return errs.WrapMsg(errs.CodeValidationFailed, message, cause) + } + return managedVMResourceAtomicError(message, cause) +} + +func (lease *vmLaunchResourceLease) Release(ctx context.Context) error { + if lease == nil || len(lease.retained) == 0 { + return nil + } + err := closeManagedCacheFDs(ctx, lease.deps, lease.retained) + lease.retained = nil + lease.cacheIdentity = managedCacheIdentity{} + lease.ownerUID = 0 + lease.rootfsFD = -1 + lease.configFD = -1 + lease.kernelFD = -1 + lease.cloudInitFD = -1 + if err != nil { + return managedVMResourceCleanupError("release managed VM launch resource descriptors", err) + } + return nil +} + +func appendManagedVMResourceCleanupError(primary error, message string, cause error) error { + if cause == nil { + return primary + } + if primary == nil { + return cause + } + if domainErr := errs.AsDomainError(primary); domainErr != nil { + domainErr.Message += "; " + message + ": " + cause.Error() + domainErr.Err = errors.Join(domainErr.Err, cause) + if domainErr.Details == nil { + domainErr.Details = make(map[string]any) + } + domainErr.Details["managed_vm_resource_descriptor_cleanup_failed"] = true + return domainErr + } + return managedVMResourceCleanupError(message+": "+cause.Error(), errors.Join(primary, cause)) +} + +func managedVMResourceAtomicError(message string, cause error) *errs.DomainError { + if cause == nil { + return errs.New(errs.CodeVMAtomicFailed, message, errs.WithClass(errs.ClassInternal)) + } + return errs.WrapMsg(errs.CodeVMAtomicFailed, message, cause, errs.WithClass(errs.ClassInternal)) +} + +func managedVMResourceCleanupError(message string, cause error) *errs.DomainError { + domainErr := managedVMResourceAtomicError(message, cause) + domainErr.Details = map[string]any{"managed_vm_resource_descriptor_cleanup_failed": true} + return domainErr +} + func isCanonicalManagedResourceID(value string, length int) bool { if len(value) != length { return false diff --git a/internal/service/jailer/managed_vm_resources_security_test.go b/internal/service/jailer/managed_vm_resources_security_test.go new file mode 100644 index 00000000..ea7dc57c --- /dev/null +++ b/internal/service/jailer/managed_vm_resources_security_test.go @@ -0,0 +1,898 @@ +package jailer + +import ( + "context" + "errors" + "os" + "path/filepath" + "runtime" + "sync" + "testing" + + "github.com/google/go-cmp/cmp" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + "golang.org/x/sys/unix" + + "mvmctl/internal/infra" + "mvmctl/pkg/errs" +) + +const ( + testManagedConfigMaxBytes = int64(1 << 20) + testManagedKernelMaxBytes = int64(1 << 30) + testManagedCloudInitMaxBytes = int64(256 << 20) +) + +type managedVMRaceAttackerKind uint8 + +const ( + managedVMRaceVMStore managedVMRaceAttackerKind = iota + 1 + managedVMRaceVMDirectory + managedVMRaceRootfs + managedVMRaceConfig + managedVMRaceCloudInit + managedVMRaceKernelStore + managedVMRaceKernel +) + +func TestPinVMLaunchResourcesRejectsSymlinkAtEveryFixedLevel(t *testing.T) { + t.Parallel() + + tests := []struct { + name string + directory bool + attacker managedVMRaceAttackerKind + path func(managedVMLaunchResourceFixture) string + }{ + {name: "VM store", directory: true, attacker: managedVMRaceVMStore, path: func( + f managedVMLaunchResourceFixture, + ) string { + return filepath.Join(f.cache.cacheDir, "vms") + }}, + {name: "VM directory", directory: true, attacker: managedVMRaceVMDirectory, path: func( + f managedVMLaunchResourceFixture, + ) string { + return filepath.Dir(f.rootfsPath) + }}, + {name: "VM rootfs", attacker: managedVMRaceRootfs, path: func(f managedVMLaunchResourceFixture) string { + return f.rootfsPath + }}, + { + name: "Firecracker configuration", + attacker: managedVMRaceConfig, + path: func(f managedVMLaunchResourceFixture) string { return f.configPath }, + }, + {name: "cloud-init ISO", attacker: managedVMRaceCloudInit, path: func( + f managedVMLaunchResourceFixture, + ) string { + return f.cloudInitPath + }}, + {name: "kernel store", directory: true, attacker: managedVMRaceKernelStore, path: func( + f managedVMLaunchResourceFixture, + ) string { + return filepath.Dir(f.kernelPath) + }}, + {name: "kernel", attacker: managedVMRaceKernel, path: func(f managedVMLaunchResourceFixture) string { + return f.kernelPath + }}, + } + + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + + fixture := prepareManagedVMLaunchResourceFixture(t, cloudInitPresent) + attacker := filepath.Join(fixture.cache.cacheDir, "attacker") + prepareManagedVMResourceRaceAttacker(t, attacker, tc.attacker) + target := tc.path(fixture) + if tc.directory { + require.NoError(t, os.RemoveAll(target)) + require.NoError(t, os.Symlink(attacker, target)) + } else { + require.NoError(t, os.Remove(target)) + require.NoError(t, os.Symlink(attacker, target)) + } + + lease, err := fixture.cacheLease.pinVMLaunchResources( + context.Background(), + mustManagedVMLaunchResourceSpec(t, cloudInitPresent, testManagedRootfsMinBytes), + ) + registerManagedVMResourceLeaseCleanup(t, lease) + assert.Nil(t, lease) + requireManagedVMResourceDomainErrorCode(t, err, errs.CodeValidationFailed) + assert.NotEmpty(t, fixture.cacheLease.retained) + }) + } +} + +func TestPinVMLaunchResourcesRejectsUnsafeBaseResourceMetadata(t *testing.T) { + t.Parallel() + + tests := []struct { + name string + rootfsBytes uint64 + mutate func(*testing.T, managedVMLaunchResourceFixture) + wantMessage string + }{ + { + name: "rootfs FIFO", + mutate: func(t *testing.T, f managedVMLaunchResourceFixture) { + require.NoError(t, os.Remove(f.rootfsPath)) + require.NoError(t, unix.Mkfifo(f.rootfsPath, 0600)) + }, + wantMessage: "not a regular file", + }, + { + name: "configuration directory", + mutate: func(t *testing.T, f managedVMLaunchResourceFixture) { + require.NoError(t, os.Remove(f.configPath)) + require.NoError(t, os.Mkdir(f.configPath, 0700)) + }, + wantMessage: "not a regular file", + }, + { + name: "rootfs hardlink", + mutate: func(t *testing.T, f managedVMLaunchResourceFixture) { + require.NoError(t, os.Link(f.rootfsPath, f.rootfsPath+".alias")) + }, + wantMessage: "exactly one link", + }, + { + name: "rootfs missing owner write", + mutate: func(t *testing.T, f managedVMLaunchResourceFixture) { + require.NoError(t, os.Chmod(f.rootfsPath, 0400)) + }, + wantMessage: "required owner access", + }, + { + name: "rootfs executable", + mutate: func(t *testing.T, f managedVMLaunchResourceFixture) { + require.NoError(t, os.Chmod(f.rootfsPath, 0700)) + }, + wantMessage: "unsupported execute access", + }, + { + name: "configuration missing owner read", + mutate: func(t *testing.T, f managedVMLaunchResourceFixture) { + require.NoError(t, os.Chmod(f.configPath, 0200)) + }, + wantMessage: "required owner access", + }, + { + name: "configuration group writable", + mutate: func(t *testing.T, f managedVMLaunchResourceFixture) { + require.NoError(t, os.Chmod(f.configPath, 0620)) + }, + wantMessage: "writable by another user", + }, + { + name: "configuration executable", + mutate: func(t *testing.T, f managedVMLaunchResourceFixture) { + require.NoError(t, os.Chmod(f.configPath, 0700)) + }, + wantMessage: "unsupported execute access", + }, + { + name: "configuration has setuid bit", + mutate: func(t *testing.T, f managedVMLaunchResourceFixture) { + require.NoError(t, os.Chmod(f.configPath, 0600|os.ModeSetuid)) + }, + wantMessage: "special mode bits", + }, + { + name: "kernel missing owner read", + mutate: func(t *testing.T, f managedVMLaunchResourceFixture) { + require.NoError(t, os.Chmod(f.kernelPath, 0200)) + }, + wantMessage: "required owner access", + }, + { + name: "cloud-init ISO missing owner read", + mutate: func(t *testing.T, f managedVMLaunchResourceFixture) { + require.NoError(t, os.Chmod(f.cloudInitPath, 0200)) + }, + wantMessage: "required owner access", + }, + { + name: "cloud-init ISO executable", + mutate: func(t *testing.T, f managedVMLaunchResourceFixture) { + require.NoError(t, os.Chmod(f.cloudInitPath, 0700)) + }, + wantMessage: "unsupported execute access", + }, + { + name: "VM store missing owner write", + mutate: func(t *testing.T, f managedVMLaunchResourceFixture) { + path := filepath.Join(f.cache.cacheDir, "vms") + require.NoError(t, os.Chmod(path, 0500)) + t.Cleanup(func() { require.NoError(t, os.Chmod(path, 0700)) }) + }, + wantMessage: "owner read, write, and traversal", + }, + { + name: "VM directory has sticky bit", + mutate: func(t *testing.T, f managedVMLaunchResourceFixture) { + require.NoError(t, os.Chmod(filepath.Dir(f.rootfsPath), 0700|os.ModeSticky)) + }, + wantMessage: "special mode bits", + }, + { + name: "empty configuration", + mutate: func(t *testing.T, f managedVMLaunchResourceFixture) { + require.NoError(t, os.Truncate(f.configPath, 0)) + }, + wantMessage: "size is outside", + }, + { + name: "oversized configuration", + mutate: func(t *testing.T, f managedVMLaunchResourceFixture) { + require.NoError(t, os.Truncate(f.configPath, testManagedConfigMaxBytes+1)) + }, + wantMessage: "size is outside", + }, + { + name: "empty kernel", + mutate: func(t *testing.T, f managedVMLaunchResourceFixture) { + require.NoError(t, os.Truncate(f.kernelPath, 0)) + }, + wantMessage: "size is outside", + }, + { + name: "oversized kernel", + mutate: func(t *testing.T, f managedVMLaunchResourceFixture) { + require.NoError(t, os.Truncate(f.kernelPath, testManagedKernelMaxBytes+1)) + }, + wantMessage: "size is outside", + }, + { + name: "empty cloud-init ISO", + mutate: func(t *testing.T, f managedVMLaunchResourceFixture) { + require.NoError(t, os.Truncate(f.cloudInitPath, 0)) + }, + wantMessage: "size is outside", + }, + { + name: "oversized cloud-init ISO", + mutate: func(t *testing.T, f managedVMLaunchResourceFixture) { + require.NoError(t, os.Truncate(f.cloudInitPath, testManagedCloudInitMaxBytes+1)) + }, + wantMessage: "size is outside", + }, + { + name: "rootfs expected size mismatch", + rootfsBytes: testManagedRootfsMinBytes + 4096, + mutate: func(*testing.T, managedVMLaunchResourceFixture) {}, + wantMessage: "does not match the expected size", + }, + { + name: "rootfs below hard minimum", + mutate: func(t *testing.T, f managedVMLaunchResourceFixture) { + require.NoError(t, os.Truncate(f.rootfsPath, int64(testManagedRootfsMinBytes-1))) + }, + wantMessage: "size is outside", + }, + { + name: "rootfs above hard maximum", + rootfsBytes: testManagedRootfsMaxBytes, + mutate: func(t *testing.T, f managedVMLaunchResourceFixture) { + require.NoError(t, os.Truncate(f.rootfsPath, int64(testManagedRootfsMaxBytes+1))) + }, + wantMessage: "size is outside", + }, + } + + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + + fixture := prepareManagedVMLaunchResourceFixture(t, cloudInitPresent) + tc.mutate(t, fixture) + rootfsBytes := tc.rootfsBytes + if rootfsBytes == 0 { + rootfsBytes = testManagedRootfsMinBytes + } + lease, err := fixture.cacheLease.pinVMLaunchResources( + context.Background(), + mustManagedVMLaunchResourceSpec(t, cloudInitPresent, rootfsBytes), + ) + registerManagedVMResourceLeaseCleanup(t, lease) + assert.Nil(t, lease) + requireManagedVMResourceDomainErrorCode(t, err, errs.CodeValidationFailed) + assert.Contains(t, err.Error(), tc.wantMessage) + assert.NotEmpty(t, fixture.cacheLease.retained) + }) + } +} + +func TestPinVMLaunchResourcesAcceptsIndependentSizeBoundaries(t *testing.T) { + t.Parallel() + + tests := []struct { + name string + rootfs int64 + config int64 + kernel int64 + cloudInit int64 + }{ + { + name: "minimum sizes", + rootfs: int64(testManagedRootfsMinBytes), + config: 1, + kernel: 1, + cloudInit: 1, + }, + { + name: "maximum sizes", + rootfs: int64(testManagedRootfsMaxBytes), + config: testManagedConfigMaxBytes, + kernel: testManagedKernelMaxBytes, + cloudInit: testManagedCloudInitMaxBytes, + }, + } + + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + + fixture := prepareManagedVMLaunchResourceFixture(t, cloudInitPresent) + require.NoError(t, os.Truncate(fixture.rootfsPath, tc.rootfs)) + require.NoError(t, os.Truncate(fixture.configPath, tc.config)) + require.NoError(t, os.Truncate(fixture.kernelPath, tc.kernel)) + require.NoError(t, os.Truncate(fixture.cloudInitPath, tc.cloudInit)) + lease, err := fixture.cacheLease.pinVMLaunchResources( + context.Background(), + mustManagedVMLaunchResourceSpec(t, cloudInitPresent, uint64(tc.rootfs)), + ) + registerManagedVMResourceLeaseCleanup(t, lease) + require.NoError(t, err) + require.NotNil(t, lease) + require.NoError(t, lease.Release(context.Background())) + }) + } +} + +func TestPinVMLaunchResourcesRejectsUnstableMetadataAndMountCrossing(t *testing.T) { + t.Parallel() + + tests := []struct { + name string + openErr error + mutate func(*unix.Statx_t) + wantCode errs.Code + }{ + {name: "descendant mount crossing", openErr: unix.EXDEV, wantCode: errs.CodeValidationFailed}, + {name: "unexpected open failure", openErr: unix.EIO, wantCode: errs.CodeVMAtomicFailed}, + {name: "foreign owner", mutate: func(stat *unix.Statx_t) { stat.Uid++ }, wantCode: errs.CodeValidationFailed}, + {name: "incomplete size metadata", mutate: func(stat *unix.Statx_t) { + stat.Mask &^= unix.STATX_SIZE + }, wantCode: errs.CodeValidationFailed}, + {name: "automount resource", mutate: func(stat *unix.Statx_t) { + stat.Attributes_mask |= unix.STATX_ATTR_AUTOMOUNT + stat.Attributes |= unix.STATX_ATTR_AUTOMOUNT + }, wantCode: errs.CodeValidationFailed}, + } + + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + + fixture := prepareManagedVMLaunchResourceFixture(t, cloudInitAbsent) + rootfsInode := inodeForManagedCacheTestPath(t, fixture.rootfsPath) + if tc.openErr != nil { + realOpenAt2 := fixture.cacheLease.deps.openAt2 + fixture.cacheLease.deps.openAt2 = func( + ctx context.Context, + parentFD int, + name string, + how *unix.OpenHow, + ) (int, error) { + if name == infra.VMRootfsFilename { + return -1, tc.openErr + } + return realOpenAt2(ctx, parentFD, name, how) + } + } else { + realStatx := fixture.cacheLease.deps.statx + fixture.cacheLease.deps.statx = func( + ctx context.Context, + fd int, + flags int, + mask int, + stat *unix.Statx_t, + ) error { + if err := realStatx(ctx, fd, flags, mask, stat); err != nil { + return err + } + if stat.Ino == rootfsInode { + tc.mutate(stat) + } + return nil + } + } + + lease, err := fixture.cacheLease.pinVMLaunchResources( + context.Background(), + mustManagedVMLaunchResourceSpec(t, cloudInitAbsent, testManagedRootfsMinBytes), + ) + registerManagedVMResourceLeaseCleanup(t, lease) + assert.Nil(t, lease) + requireManagedVMResourceDomainErrorCode(t, err, tc.wantCode) + assert.NotEmpty(t, fixture.cacheLease.retained) + }) + } +} + +func TestPinVMLaunchResourcesRejectsUnsafeDirectoryMetadata(t *testing.T) { + t.Parallel() + + tests := []struct { + name string + mutate func(*unix.Statx_t) + }{ + {name: "foreign owner", mutate: func(stat *unix.Statx_t) { stat.Uid++ }}, + {name: "incomplete type metadata", mutate: func(stat *unix.Statx_t) { stat.Mask &^= unix.STATX_TYPE }}, + } + + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + + fixture := prepareManagedVMLaunchResourceFixture(t, cloudInitAbsent) + vmsInode := inodeForManagedCacheTestPath(t, filepath.Join(fixture.cache.cacheDir, "vms")) + realStatx := fixture.cacheLease.deps.statx + fixture.cacheLease.deps.statx = func( + ctx context.Context, + fd int, + flags int, + mask int, + stat *unix.Statx_t, + ) error { + if err := realStatx(ctx, fd, flags, mask, stat); err != nil { + return err + } + if stat.Ino == vmsInode { + tc.mutate(stat) + } + return nil + } + + lease, err := fixture.cacheLease.pinVMLaunchResources( + context.Background(), + mustManagedVMLaunchResourceSpec(t, cloudInitAbsent, testManagedRootfsMinBytes), + ) + registerManagedVMResourceLeaseCleanup(t, lease) + assert.Nil(t, lease) + requireManagedVMResourceDomainErrorCode(t, err, errs.CodeValidationFailed) + assert.NotEmpty(t, fixture.cacheLease.retained) + }) + } +} + +func TestPinVMLaunchResourcesLeavesCacheLeaseReusableAfterRejection(t *testing.T) { + t.Parallel() + + fixture := prepareManagedVMLaunchResourceFixture(t, cloudInitAbsent) + require.NoError(t, os.Remove(fixture.configPath)) + spec := mustManagedVMLaunchResourceSpec(t, cloudInitAbsent, testManagedRootfsMinBytes) + + lease, err := fixture.cacheLease.pinVMLaunchResources(context.Background(), spec) + registerManagedVMResourceLeaseCleanup(t, lease) + assert.Nil(t, lease) + requireManagedVMResourceDomainErrorCode(t, err, errs.CodeValidationFailed) + assert.NotEmpty(t, fixture.cacheLease.retained) + + require.NoError(t, os.WriteFile(fixture.configPath, []byte("{}"), 0600)) + lease, err = fixture.cacheLease.pinVMLaunchResources(context.Background(), spec) + registerManagedVMResourceLeaseCleanup(t, lease) + require.NoError(t, err) + require.NotNil(t, lease) + require.NoError(t, lease.Release(context.Background())) +} + +func TestPinVMLaunchResourcesValidatesForgedSpecBeforeFilesystemAccess(t *testing.T) { + t.Parallel() + + fixture := prepareManagedVMLaunchResourceFixture(t, cloudInitAbsent) + var openCalls int + fixture.cacheLease.deps.openAt2 = func(context.Context, int, string, *unix.OpenHow) (int, error) { + openCalls++ + return -1, errors.New("must not open") + } + + lease, err := fixture.cacheLease.pinVMLaunchResources(context.Background(), vmLaunchResourceSpec{}) + registerManagedVMResourceLeaseCleanup(t, lease) + assert.Nil(t, lease) + requireManagedVMResourceDomainErrorCode(t, err, errs.CodeValidationFailed) + assert.Zero(t, openCalls) +} + +func TestVMLaunchResourceLeaseReleasesEveryDescriptorInReverseWithoutCancellation(t *testing.T) { + t.Parallel() + + cache := prepareManagedCacheFixture(t) + createManagedVMLaunchResourceFiles(t, cache, cloudInitPresent) + deps := cache.deps + opened := make([]int, 0, 12) + realOpen := deps.open + deps.open = func(ctx context.Context, path string, flags int, mode uint32) (int, error) { + fd, err := realOpen(ctx, path, flags, mode) + if err == nil { + opened = append(opened, fd) + } + return fd, err + } + realOpenAt2 := deps.openAt2 + deps.openAt2 = func(ctx context.Context, parentFD int, name string, how *unix.OpenHow) (int, error) { + fd, err := realOpenAt2(ctx, parentFD, name, how) + if err == nil { + opened = append(opened, fd) + } + return fd, err + } + + cacheLease, err := pinManagedCache(context.Background(), deps, cache.policy, cache.caller, cache.locator) + require.NoError(t, err) + require.NotNil(t, cacheLease) + t.Cleanup(func() { require.NoError(t, cacheLease.Release(context.Background())) }) + prepareManagedVMResourceOwnershipForTest(cacheLease, cache.caller.uid) + lease, err := cacheLease.pinVMLaunchResources( + context.Background(), + mustManagedVMLaunchResourceSpec(t, cloudInitPresent, testManagedRootfsMinBytes), + ) + registerManagedVMResourceLeaseCleanup(t, lease) + require.NoError(t, err) + require.NotNil(t, lease) + + closed := make([]int, 0, len(opened)) + realClose := lease.deps.close + lease.deps.close = func(ctx context.Context, fd int) error { + assert.NoError(t, ctx.Err()) + closed = append(closed, fd) + return realClose(ctx, fd) + } + cancelled, cancel := context.WithCancel(context.Background()) + cancel() + require.NoError(t, lease.Release(cancelled)) + + wantClosed := make([]int, len(opened)) + for index := range opened { + wantClosed[index] = opened[len(opened)-1-index] + } + if diff := cmp.Diff(wantClosed, closed); diff != "" { + t.Errorf("managed VM resource close order mismatch (-want +got):\n%s", diff) + } + require.NoError(t, lease.Release(context.Background())) + if diff := cmp.Diff(wantClosed, closed); diff != "" { + t.Errorf("idempotent release changed close order (-want +got):\n%s", diff) + } +} + +func TestVMLaunchResourceLeaseReportsCloseFailureAfterClosingAllDescriptors(t *testing.T) { + t.Parallel() + + fixture := prepareManagedVMLaunchResourceFixture(t, cloudInitAbsent) + lease, err := fixture.cacheLease.pinVMLaunchResources( + context.Background(), + mustManagedVMLaunchResourceSpec(t, cloudInitAbsent, testManagedRootfsMinBytes), + ) + registerManagedVMResourceLeaseCleanup(t, lease) + require.NoError(t, err) + require.NotNil(t, lease) + + failedFD := lease.configFD + realClose := lease.deps.close + var closeCalls int + lease.deps.close = func(ctx context.Context, fd int) error { + closeCalls++ + closeErr := realClose(ctx, fd) + if fd == failedFD { + return errors.Join(closeErr, errors.New("injected config close failure")) + } + return closeErr + } + wantCloseCalls := len(lease.retained) + err = lease.Release(context.Background()) + require.Error(t, err) + domainErr := errs.AsDomainError(err) + require.NotNil(t, domainErr) + assert.Equal(t, errs.CodeVMAtomicFailed, domainErr.Code) + assert.Equal(t, errs.ClassInternal, domainErr.Class) + assert.Equal(t, true, domainErr.Details["managed_vm_resource_descriptor_cleanup_failed"]) + assert.Equal(t, wantCloseCalls, closeCalls) + assert.Empty(t, lease.retained) + require.NoError(t, lease.Release(context.Background())) +} + +func TestAppendManagedVMResourceCleanupErrorPreservesPrimaryDomainMetadata(t *testing.T) { + t.Parallel() + + primary := errs.New( + errs.CodeValidationFailed, + "primary failure", + errs.WithClass(errs.ClassConflict), + errs.WithEntity(testManagedVMID), + errs.WithDetails(map[string]any{"record_registered": true}), + ) + joined := appendManagedVMResourceCleanupError(primary, "close resource descriptors", errors.New("close failed")) + domainErr := errs.AsDomainError(joined) + require.NotNil(t, domainErr) + assert.Same(t, primary, domainErr) + assert.Equal(t, errs.CodeValidationFailed, domainErr.Code) + assert.Equal(t, errs.ClassConflict, domainErr.Class) + assert.Equal(t, testManagedVMID, domainErr.Entity) + assert.Equal(t, true, domainErr.Details["record_registered"]) + assert.Equal(t, true, domainErr.Details["managed_vm_resource_descriptor_cleanup_failed"]) +} + +func TestPinVMLaunchResourcesPreservesPrimaryErrorWhenRejectedFileCloseFails(t *testing.T) { + t.Parallel() + + fixture := prepareManagedVMLaunchResourceFixture(t, cloudInitAbsent) + configInode := inodeForManagedCacheTestPath(t, fixture.configPath) + realStatx := fixture.cacheLease.deps.statx + fixture.cacheLease.deps.statx = func( + ctx context.Context, + fd int, + flags int, + mask int, + stat *unix.Statx_t, + ) error { + if err := realStatx(ctx, fd, flags, mask, stat); err != nil { + return err + } + if stat.Ino == configInode { + return errors.New("injected config stat failure") + } + return nil + } + realClose := fixture.cacheLease.deps.close + fixture.cacheLease.deps.close = func(ctx context.Context, fd int) error { + var stat unix.Stat_t + statErr := unix.Fstat(fd, &stat) + closeErr := realClose(ctx, fd) + if statErr == nil && stat.Ino == configInode { + return errors.Join(closeErr, errors.New("injected rejected-file close failure")) + } + return closeErr + } + + lease, err := fixture.cacheLease.pinVMLaunchResources( + context.Background(), + mustManagedVMLaunchResourceSpec(t, cloudInitAbsent, testManagedRootfsMinBytes), + ) + registerManagedVMResourceLeaseCleanup(t, lease) + assert.Nil(t, lease) + require.Error(t, err) + domainErr := errs.AsDomainError(err) + require.NotNil(t, domainErr) + assert.Equal(t, errs.CodeVMAtomicFailed, domainErr.Code) + assert.Equal(t, errs.ClassInternal, domainErr.Class) + assert.Equal(t, true, domainErr.Details["managed_vm_resource_descriptor_cleanup_failed"]) + assert.Contains(t, domainErr.Message, "inspect managed Firecracker configuration") + assert.Contains(t, domainErr.Message, "close rejected managed VM file") + assert.NotEmpty(t, fixture.cacheLease.retained) +} + +func TestPinVMLaunchResourcesClosesPartialAcquisitionWithUncancelledContext(t *testing.T) { + t.Parallel() + + fixture := prepareManagedVMLaunchResourceFixture(t, cloudInitAbsent) + ctx, cancel := context.WithCancel(context.Background()) + realOpenAt2 := fixture.cacheLease.deps.openAt2 + fixture.cacheLease.deps.openAt2 = func( + ctx context.Context, + parentFD int, + name string, + how *unix.OpenHow, + ) (int, error) { + fd, err := realOpenAt2(ctx, parentFD, name, how) + if err == nil && name == "vms" { + cancel() + } + return fd, err + } + realClose := fixture.cacheLease.deps.close + closeContextErrors := make([]error, 0, 1) + fixture.cacheLease.deps.close = func(ctx context.Context, fd int) error { + closeContextErrors = append(closeContextErrors, ctx.Err()) + return realClose(ctx, fd) + } + + lease, err := fixture.cacheLease.pinVMLaunchResources( + ctx, + mustManagedVMLaunchResourceSpec(t, cloudInitAbsent, testManagedRootfsMinBytes), + ) + registerManagedVMResourceLeaseCleanup(t, lease) + assert.Nil(t, lease) + requireManagedVMResourceDomainErrorCode(t, err, errs.CodeVMAtomicFailed) + if diff := cmp.Diff([]error{nil}, closeContextErrors); diff != "" { + t.Errorf("partial acquisition cleanup contexts mismatch (-want +got):\n%s", diff) + } + assert.NotEmpty(t, fixture.cacheLease.retained) +} + +func TestPinVMLaunchResourcesResistsContinuousSymlinkReplacementAtEveryFixedLevel(t *testing.T) { + tests := []struct { + name string + attacker managedVMRaceAttackerKind + path func(managedVMLaunchResourceFixture) string + }{ + {name: "VM store", attacker: managedVMRaceVMStore, path: func(f managedVMLaunchResourceFixture) string { + return filepath.Join(f.cache.cacheDir, "vms") + }}, + {name: "VM directory", attacker: managedVMRaceVMDirectory, path: func(f managedVMLaunchResourceFixture) string { + return filepath.Dir(f.rootfsPath) + }}, + {name: "VM rootfs", attacker: managedVMRaceRootfs, path: func(f managedVMLaunchResourceFixture) string { + return f.rootfsPath + }}, + { + name: "Firecracker configuration", + attacker: managedVMRaceConfig, + path: func(f managedVMLaunchResourceFixture) string { return f.configPath }, + }, + {name: "cloud-init ISO", attacker: managedVMRaceCloudInit, path: func(f managedVMLaunchResourceFixture) string { + return f.cloudInitPath + }}, + {name: "kernel store", attacker: managedVMRaceKernelStore, path: func(f managedVMLaunchResourceFixture) string { + return filepath.Dir(f.kernelPath) + }}, + {name: "kernel", attacker: managedVMRaceKernel, path: func(f managedVMLaunchResourceFixture) string { + return f.kernelPath + }}, + } + + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + fixture := prepareManagedVMLaunchResourceFixture(t, cloudInitPresent) + rootfsInode := inodeForManagedCacheTestPath(t, fixture.rootfsPath) + configInode := inodeForManagedCacheTestPath(t, fixture.configPath) + kernelInode := inodeForManagedCacheTestPath(t, fixture.kernelPath) + cloudInitInode := inodeForManagedCacheTestPath(t, fixture.cloudInitPath) + + target := tc.path(fixture) + good := target + ".race-good" + require.NoError(t, os.Rename(target, good)) + attacker := filepath.Join(fixture.cache.cacheDir, "race-attacker") + prepareManagedVMResourceRaceAttacker(t, attacker, tc.attacker) + + stop := make(chan struct{}) + done := make(chan struct{}) + started := make(chan struct{}) + writerErr := make(chan error, 1) + var startedOnce sync.Once + reportWriterError := func(err error) { + startedOnce.Do(func() { close(started) }) + writerErr <- err + } + go func() { + defer close(done) + for { + select { + case <-stop: + return + default: + } + if err := os.Symlink(attacker, target); err != nil { + reportWriterError(err) + return + } + startedOnce.Do(func() { close(started) }) + runtime.Gosched() + if err := os.Remove(target); err != nil { + reportWriterError(err) + return + } + if err := os.Rename(good, target); err != nil { + reportWriterError(err) + return + } + runtime.Gosched() + if err := os.Rename(target, good); err != nil { + reportWriterError(err) + return + } + } + }() + <-started + var stopOnce sync.Once + stopWriter := func() { + stopOnce.Do(func() { close(stop) }) + <-done + } + t.Cleanup(stopWriter) + + for range 200 { + lease, err := fixture.cacheLease.pinVMLaunchResources( + context.Background(), + mustManagedVMLaunchResourceSpec(t, cloudInitPresent, testManagedRootfsMinBytes), + ) + registerManagedVMResourceLeaseCleanup(t, lease) + if err != nil { + requireManagedVMResourceDomainErrorCode(t, err, errs.CodeValidationFailed) + runtime.Gosched() + continue + } + require.NotNil(t, lease) + assertManagedDescriptorInode(t, lease.rootfsFD, rootfsInode) + assertManagedDescriptorInode(t, lease.configFD, configInode) + assertManagedDescriptorInode(t, lease.kernelFD, kernelInode) + assertManagedDescriptorInode(t, lease.cloudInitFD, cloudInitInode) + require.NoError(t, lease.Release(context.Background())) + break + } + stopWriter() + select { + case err := <-writerErr: + require.NoError(t, err) + default: + } + }) + } +} + +func prepareManagedVMResourceRaceAttacker(t *testing.T, path string, kind managedVMRaceAttackerKind) { + t.Helper() + + switch kind { + case managedVMRaceVMStore: + prepareManagedVMResourceRaceAttackerVMDirectory(t, filepath.Join(path, testManagedVMID)) + case managedVMRaceVMDirectory: + prepareManagedVMResourceRaceAttackerVMDirectory(t, path) + case managedVMRaceRootfs: + require.NoError(t, os.WriteFile(path, nil, 0600)) + require.NoError(t, os.Truncate(path, int64(testManagedRootfsMinBytes))) + case managedVMRaceConfig: + require.NoError(t, os.WriteFile(path, []byte("{}"), 0600)) + case managedVMRaceCloudInit: + require.NoError(t, os.WriteFile(path, []byte("iso"), 0600)) + case managedVMRaceKernelStore: + require.NoError(t, os.Mkdir(path, 0700)) + require.NoError(t, os.WriteFile(filepath.Join(path, testManagedKernelID), []byte("kernel"), 0700)) + case managedVMRaceKernel: + require.NoError(t, os.WriteFile(path, []byte("kernel"), 0700)) + default: + require.FailNow(t, "unknown managed VM race attacker kind") + } +} + +func prepareManagedVMResourceRaceAttackerVMDirectory(t *testing.T, path string) { + t.Helper() + + require.NoError(t, os.MkdirAll(path, 0700)) + rootfs := filepath.Join(path, infra.VMRootfsFilename) + require.NoError(t, os.WriteFile(rootfs, nil, 0600)) + require.NoError(t, os.Truncate(rootfs, int64(testManagedRootfsMinBytes))) + require.NoError(t, os.WriteFile(filepath.Join(path, infra.VMFirecrackerConfigFilename), []byte("{}"), 0600)) + require.NoError(t, os.WriteFile(filepath.Join(path, infra.VMCloudInitISOFilename), []byte("iso"), 0600)) +} + +func registerManagedVMResourceLeaseCleanup(t *testing.T, lease *vmLaunchResourceLease) { + t.Helper() + + if lease == nil { + return + } + t.Cleanup(func() { require.NoError(t, lease.Release(context.Background())) }) +} + +func assertManagedDescriptorInode(t *testing.T, fd int, wantInode uint64) { + t.Helper() + + var stat unix.Stat_t + require.NoError(t, unix.Fstat(fd, &stat)) + assert.Equal(t, wantInode, stat.Ino) +} + +func mustManagedVMLaunchResourceSpec( + t *testing.T, + cloudInit cloudInitPresence, + rootfsBytes uint64, +) vmLaunchResourceSpec { + t.Helper() + + spec, err := newVMLaunchResourceSpec(testManagedVMID, testManagedKernelID, rootfsBytes, cloudInit) + require.NoError(t, err) + return spec +} diff --git a/internal/service/jailer/managed_vm_resources_test.go b/internal/service/jailer/managed_vm_resources_test.go index 3d12eff3..f6aa7b5b 100644 --- a/internal/service/jailer/managed_vm_resources_test.go +++ b/internal/service/jailer/managed_vm_resources_test.go @@ -1,13 +1,20 @@ package jailer import ( + "context" + "errors" + "os" + "path/filepath" "strings" "testing" "github.com/google/go-cmp/cmp" + "github.com/google/go-cmp/cmp/cmpopts" "github.com/stretchr/testify/assert" "github.com/stretchr/testify/require" + "golang.org/x/sys/unix" + "mvmctl/internal/infra" "mvmctl/pkg/errs" ) @@ -178,8 +185,243 @@ func TestValidateVMLaunchResourceSpecRejectsForgedValues(t *testing.T) { t.Parallel() err := validateVMLaunchResourceSpec(spec) - require.Error(t, err) - assert.Equal(t, errs.CodeValidationFailed, errs.AsDomainError(err).Code) + requireManagedVMResourceDomainErrorCode(t, err, errs.CodeValidationFailed) }) } } + +func TestPinVMLaunchResourcesPinsFixedBaseLeaves(t *testing.T) { + t.Parallel() + + tests := []struct { + name string + cloudInit cloudInitPresence + }{ + {name: "without cloud init", cloudInit: cloudInitAbsent}, + {name: "with cloud init", cloudInit: cloudInitPresent}, + } + + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + + fixture := prepareManagedVMLaunchResourceFixture(t, tc.cloudInit) + originalCacheIdentity := fixture.cacheLease.identity + originalRootfsInode := inodeForManagedCacheTestPath(t, fixture.rootfsPath) + + var pathOpenCalls int + fixture.cacheLease.deps.open = func(context.Context, string, int, uint32) (int, error) { + pathOpenCalls++ + return -1, errors.New("managed resources must not reopen a path") + } + type observedOpen struct { + Name string + Flags uint64 + Resolve uint64 + } + observed := make([]observedOpen, 0, 7) + realOpenAt2 := fixture.cacheLease.deps.openAt2 + fixture.cacheLease.deps.openAt2 = func( + ctx context.Context, + parentFD int, + name string, + how *unix.OpenHow, + ) (int, error) { + observed = append(observed, observedOpen{Name: name, Flags: how.Flags, Resolve: how.Resolve}) + return realOpenAt2(ctx, parentFD, name, how) + } + type observedStatx struct { + Flags int + Mask int + } + observedStats := make([]observedStatx, 0, 7) + realStatx := fixture.cacheLease.deps.statx + fixture.cacheLease.deps.statx = func( + ctx context.Context, + fd int, + flags int, + mask int, + stat *unix.Statx_t, + ) error { + observedStats = append(observedStats, observedStatx{Flags: flags, Mask: mask}) + return realStatx(ctx, fd, flags, mask, stat) + } + + spec, err := newVMLaunchResourceSpec( + testManagedVMID, + testManagedKernelID, + testManagedRootfsMinBytes, + tc.cloudInit, + ) + require.NoError(t, err) + lease, err := fixture.cacheLease.pinVMLaunchResources(context.Background(), spec) + require.NoError(t, err) + require.NotNil(t, lease) + t.Cleanup(func() { require.NoError(t, lease.Release(context.Background())) }) + + assert.Zero(t, pathOpenCalls) + directoryFlags := uint64(unix.O_PATH | unix.O_DIRECTORY | unix.O_CLOEXEC | unix.O_NOFOLLOW) + fileFlags := uint64(unix.O_PATH | unix.O_CLOEXEC | unix.O_NOFOLLOW) + resolveFlags := uint64( + unix.RESOLVE_BENEATH | + unix.RESOLVE_NO_SYMLINKS | + unix.RESOLVE_NO_MAGICLINKS | + unix.RESOLVE_NO_XDEV, + ) + wantObserved := []observedOpen{ + {Name: "vms", Flags: directoryFlags, Resolve: resolveFlags}, + {Name: testManagedVMID, Flags: directoryFlags, Resolve: resolveFlags}, + {Name: infra.VMRootfsFilename, Flags: fileFlags, Resolve: resolveFlags}, + {Name: infra.VMFirecrackerConfigFilename, Flags: fileFlags, Resolve: resolveFlags}, + } + if tc.cloudInit == cloudInitPresent { + wantObserved = append(wantObserved, observedOpen{ + Name: infra.VMCloudInitISOFilename, Flags: fileFlags, Resolve: resolveFlags, + }) + } + wantObserved = append(wantObserved, + observedOpen{Name: "kernels", Flags: directoryFlags, Resolve: resolveFlags}, + observedOpen{Name: testManagedKernelID, Flags: fileFlags, Resolve: resolveFlags}, + ) + if diff := cmp.Diff(wantObserved, observed); diff != "" { + t.Errorf("managed VM resource opens mismatch (-want +got):\n%s", diff) + } + wantStatFlags := unix.AT_EMPTY_PATH | unix.AT_NO_AUTOMOUNT | unix.AT_SYMLINK_NOFOLLOW | + unix.AT_STATX_FORCE_SYNC + for _, observedStat := range observedStats { + assert.Equal(t, wantStatFlags, observedStat.Flags) + assert.Equal(t, unix.STATX_BASIC_STATS, observedStat.Mask) + assert.Zero(t, observedStat.Flags&unix.AT_STATX_DONT_SYNC) + } + assert.Len(t, observedStats, len(wantObserved)) + + assert.Equal(t, fixture.cache.caller.uid, lease.ownerUID) + if diff := cmp.Diff( + originalCacheIdentity, + lease.cacheIdentity, + cmpopts.EquateComparable(managedCacheIdentity{}), + ); diff != "" { + t.Errorf("managed cache identity transfer mismatch (-want +got):\n%s", diff) + } + assertManagedDescriptorMatchesPath(t, lease.rootfsFD, fixture.rootfsPath) + assertManagedDescriptorMatchesPath(t, lease.configFD, fixture.configPath) + assertManagedDescriptorMatchesPath(t, lease.kernelFD, fixture.kernelPath) + if tc.cloudInit == cloudInitPresent { + assertManagedDescriptorMatchesPath(t, lease.cloudInitFD, fixture.cloudInitPath) + } else { + assert.Equal(t, -1, lease.cloudInitFD) + } + + movedRootfs := fixture.rootfsPath + ".moved" + require.NoError(t, os.Rename(fixture.rootfsPath, movedRootfs)) + require.NoError(t, os.Symlink(fixture.configPath, fixture.rootfsPath)) + var pinnedStat unix.Stat_t + require.NoError(t, unix.Fstat(lease.rootfsFD, &pinnedStat)) + assert.Equal(t, originalRootfsInode, pinnedStat.Ino) + + assert.Empty(t, fixture.cacheLease.retained) + assert.Equal(t, -1, fixture.cacheLease.cacheFD) + assert.Zero(t, fixture.cacheLease.ownerUID) + }) + } +} + +type managedVMLaunchResourceFixture struct { + cache managedCacheFixture + cacheLease *managedCacheLease + rootfsPath string + configPath string + kernelPath string + cloudInitPath string +} + +func prepareManagedVMLaunchResourceFixture( + t *testing.T, + cloudInit cloudInitPresence, +) managedVMLaunchResourceFixture { + t.Helper() + + cache := prepareManagedCacheFixture(t) + fixture := createManagedVMLaunchResourceFiles(t, cache, cloudInit) + + cacheLease, err := pinManagedCache( + context.Background(), + cache.deps, + cache.policy, + cache.caller, + cache.locator, + ) + require.NoError(t, err) + require.NotNil(t, cacheLease) + t.Cleanup(func() { require.NoError(t, cacheLease.Release(context.Background())) }) + prepareManagedVMResourceOwnershipForTest(cacheLease, cache.caller.uid) + fixture.cacheLease = cacheLease + return fixture +} + +func createManagedVMLaunchResourceFiles( + t *testing.T, + cache managedCacheFixture, + cloudInit cloudInitPresence, +) managedVMLaunchResourceFixture { + t.Helper() + + vmDir := filepath.Join(cache.cacheDir, "vms", testManagedVMID) + kernelDir := filepath.Join(cache.cacheDir, "kernels") + require.NoError(t, os.MkdirAll(vmDir, 0700)) + require.NoError(t, os.Mkdir(kernelDir, 0700)) + + rootfsPath := filepath.Join(vmDir, infra.VMRootfsFilename) + require.NoError(t, os.WriteFile(rootfsPath, nil, 0600)) + require.NoError(t, os.Truncate(rootfsPath, int64(testManagedRootfsMinBytes))) + configPath := filepath.Join(vmDir, infra.VMFirecrackerConfigFilename) + require.NoError(t, os.WriteFile(configPath, []byte("{}"), 0600)) + kernelPath := filepath.Join(kernelDir, testManagedKernelID) + require.NoError(t, os.WriteFile(kernelPath, []byte("kernel"), 0700)) + cloudInitPath := filepath.Join(vmDir, infra.VMCloudInitISOFilename) + if cloudInit == cloudInitPresent { + require.NoError(t, os.WriteFile(cloudInitPath, []byte("iso"), 0600)) + } + + return managedVMLaunchResourceFixture{ + cache: cache, + rootfsPath: rootfsPath, + configPath: configPath, + kernelPath: kernelPath, + cloudInitPath: cloudInitPath, + } +} + +func prepareManagedVMResourceOwnershipForTest(cacheLease *managedCacheLease, ownerUID uint32) { + if ownerUID == uint32(os.Geteuid()) { + return + } + realStatx := cacheLease.deps.statx + cacheLease.deps.statx = func(ctx context.Context, fd, flags, mask int, stat *unix.Statx_t) error { + if err := realStatx(ctx, fd, flags, mask, stat); err != nil { + return err + } + stat.Uid = ownerUID + return nil + } +} + +func assertManagedDescriptorMatchesPath(t *testing.T, fd int, path string) { + t.Helper() + + var descriptorStat unix.Stat_t + require.NoError(t, unix.Fstat(fd, &descriptorStat)) + var pathStat unix.Stat_t + require.NoError(t, unix.Stat(path, &pathStat)) + assert.Equal(t, pathStat.Dev, descriptorStat.Dev) + assert.Equal(t, pathStat.Ino, descriptorStat.Ino) +} + +func requireManagedVMResourceDomainErrorCode(t *testing.T, err error, code errs.Code) { + t.Helper() + + require.Error(t, err) + domainErr := errs.AsDomainError(err) + require.NotNil(t, domainErr) + assert.Equal(t, code, domainErr.Code) +} From e7835889a3850dca6da1c4cbb312c5942a6975be Mon Sep 17 00:00:00 2001 From: AlanD20 Date: Sat, 29 Aug 2026 00:16:42 +0200 Subject: [PATCH 0053/1457] docs: record base launch resource pinning --- CHANGELOG.md | 26 +++++++++++++---- ...owned-single-binary-privileged-dispatch.md | 8 ++++++ tasks/plan.md | 6 +++- tasks/todo.md | 28 ++++++++++++------- 4 files changed, 51 insertions(+), 17 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 58fb0c40..bc0dd461 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,7 +9,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 > **Security release status:** `0.3.0` is still under development. The canonical Jailer, cgroup policy, routed > service-access policy, root-owned system installer, early privileged-dispatch foundation, and private root-owned VM -> authority and managed-cache pinning substrates described below are implemented. The mandatory per-VM network +> authority, managed-cache pinning, and base launch-resource pinning substrates described below are implemented. The +> mandatory per-VM network > namespace, nftables-only `traffic` policy, exact VM-to-VM `exec` policy, final marker-only sudo policy, and remaining > typed privileged operations are not implemented yet; see > **Security work still pending** before treating this version as release-ready. @@ -68,7 +69,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - This substrate is private and not yet wired into public VM lifecycle operations; that integration remains a release blocker below. -#### Descriptor-pinned managed-cache substrate +#### Descriptor-pinned managed-cache and base launch-resource substrates - Added a private typed `MVM_CACHE_DIR` locator and lease under `internal/service/jailer`. Root opens the fixed filesystem root once, traverses canonical components with beneath/no-symlink/no-magic-link `openat2`, and retains the @@ -78,8 +79,19 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - The final cache root accepts ext2/ext3/ext4, XFS, Btrfs, F2FS, bcachefs, tmpfs, or ZFS. FUSE, network, overlay/stacked, and automount cache roots fail closed. Ancestor mounts are constrained by descriptor traversal, owner/mode, and automount checks without unnecessarily requiring the cache filesystem allowlist. -- This is a private foundation only. The privileged request, root-owned instance record, individual resource leaves, - and public VM lifecycle are not wired to it yet and remain release blockers below. +- Added a private typed base launch-resource lease for the fixed VM rootfs, Firecracker configuration, kernel, and + optional cloud-init ISO. It opens every component relative to the pinned cache with beneath/no-symlink/no-magic-link/ + no-cross-mount resolution and never reopens an admitted resource by pathname. +- Base launch admission verifies caller ownership, regular-file type, exactly one hard link, resource-specific owner + access, safe modes, synchronized stable metadata, and hard size limits: rootfs 128 MiB through 16 TiB, configuration + 1 byte through 1 MiB, kernel 1 byte through 1 GiB, and cloud-init ISO 1 byte through 256 MiB. The bounded expected + rootfs size is an equality check, not authority to raise the receiver limit. +- Successful admission transfers the complete cache and resource descriptor chain into one private lease. Rejection + leaves the cache lease reusable; release and failure cleanup attempt every retained descriptor in reverse order with + an uncancelled context and preserve primary `DomainError` metadata on cleanup failure. +- These are private foundations only. The privileged request, cache identity in the root-owned instance record, + Firecracker configuration consumption, mount-namespace consumer, remaining resource classes, and public VM lifecycle + are not wired yet and remain release blockers below. #### Installed release-candidate qualification - For T1/T2, the Python system-test runner now treats `MVM_BINARY` as a distinct compatible outer controller and @@ -218,8 +230,10 @@ The following items are release blockers and are intentionally not described abo process identity, and per-VM lock. - Add root-owned managed-network identity and global capacity authority; user SQLite state must not authorize host-global namespace, link, firewall, process, mount, cgroup, or admission decisions. -- Finish wiring the implemented managed-cache pinning foundation to the privileged protocol and instance record, then - add descriptor-pinned operation-specific resources and private mount namespaces. +- Carry the managed-cache locator and base launch selection through the privileged protocol, bind the cache identity in + the root-owned instance record, and create/consume the implemented descriptor lease inside the root process and + private mount-namespace launch path. Add separate closed descriptor policies for snapshots, volumes, persistent + outputs, and image provisioning. - Split root-only locks/handshakes beneath `/run/mvmctl/authority` from caller-owned socket/PID output directories beneath `/run/mvmctl/runtime`; stop mounting the whole cache VM directory. Pin persistent config/output leaves individually, truncate enabled `0600` log/console/metrics files only after durable launch registration, and retain them after stop. diff --git a/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md b/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md index b80f551d..93ef9478 100644 --- a/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md +++ b/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md @@ -294,6 +294,14 @@ image provisioning each require their own closed policy before their descriptors In particular, a QCOW2 file's physical length is not its virtual capacity, so a future volume policy cannot treat `statx` size as QCOW2 capacity or invoke an unrestricted image tool as root to compensate. +Implementation note (2026-08-29): the private Jailer service now implements this base launch selection and resource +lease. It derives the fixed components from typed IDs, opens them relative to the pinned cache descriptor with the +required no-cross-mount resolution, performs synchronized ownership/type/link/mode/size admission, and transfers the +complete descriptor chain into checked reverse cleanup. The privileged request does not yet carry the cache locator and +base selection, the root-owned instance record does not bind the cache identity, and the mount-namespace launch path +does not consume the root-side lease. The lease is not yet responsible for parsing and constraining the Firecracker +configuration contents. Snapshot, volume, output, and image-provisioning policies remain pending. + Sockets and PID mirrors are not persistent cache artifacts. Their host layout is fixed beneath the root-controlled ephemeral runtime tree: diff --git a/tasks/plan.md b/tasks/plan.md index ac2767be..3068f2d0 100644 --- a/tasks/plan.md +++ b/tasks/plan.md @@ -35,7 +35,11 @@ continue without silently deleting it. release references, global VM-ID claims, and release/index/VM locking. - Private managed-cache locator and lease foundation with descriptor-relative component traversal, synchronized owner/mode/identity checks, an explicit local-filesystem policy, retained descriptors, and identity-checked re-pin. - It is not yet carried by a privileged action or bound into the instance record. +- Private base VM launch-resource selection and lease for fixed rootfs, Firecracker configuration, kernel, and optional + cloud-init leaves, with descriptor-relative no-cross-mount opening, synchronized admission checks, descriptor + transfer, and checked reverse cleanup. + The locator and base selection are not yet carried by a privileged action, cache identity is not bound into the + instance record, and the root-side leases are not consumed by launch. - ADR-0016 for the single-binary privilege boundary. The action switch is intentionally still empty and the sudoers policy remains transitional. The routed policy, dual diff --git a/tasks/todo.md b/tasks/todo.md index 0ddbdb9b..a30ec788 100644 --- a/tasks/todo.md +++ b/tasks/todo.md @@ -142,8 +142,9 @@ under `internal/service/jailer/`. No handler, transport, CLI, API, core-domain, - [x] Add the private canonical locator and descriptor lease: component-wise `openat2`, synchronized ownership/mode/ identity inspection, a closed local-filesystem policy on the final cache root, unique/legacy/unavailable mount-ID typing, identity-checked re-pin, and checked reverse cleanup. - - [ ] Carry the locator through the privileged request, remove the transitional raw effect paths, and bind the lease - to operation-specific resource descriptors. The private foundation is not yet a production handler. + - [ ] Carry the locator and closed base launch selection through the privileged request, remove the transitional raw + effect paths, and consume the private base launch-resource lease. The private foundation is not yet a production + handler. - [ ] Bind the pinned cache device/inode/mount identity into the root-owned instance record and require later operations to re-pin the same identity. - [ ] Land canonical producer names before descriptor pinning: @@ -171,14 +172,21 @@ under `internal/service/jailer/`. No handler, transport, CLI, API, core-domain, - [ ] Define snapshot restore memory/state, volume count/format/capacity, writable snapshot creation, persistent output, and image-provisioning policies in their owning increments. QCOW2 physical length must not be confused with virtual capacity or inspected by an unrestricted root subprocess. -- [ ] Open user cache/VM/image/kernel/volume/snapshot resources once with beneath/no-symlink/no-magic-link resolution. -- [ ] Verify caller ownership, file type, fixed basenames, size, and allowed access mode on pinned descriptors. -- [ ] Never validate a path and later reopen it by pathname. -- [ ] Keep leases private to the owning service package, expose no generic FD/path accessor, and release retained - descriptors in reverse order with checked `context.WithoutCancel` cleanup. -- [ ] Return DomainError partial-state details for close/fsync/cleanup failures without replacing the primary code, - class, entity, or details. -- [ ] Race tests continuously replace every user-owned ancestor and leaf with symlinks/renames. +- [x] Implement the private base launch-resource lease for fixed `vms//rootfs.img`, + `vms//firecracker.json`, optional `vms//cloud-init.iso`, and `kernels/`: component traversal + beneath the pinned cache, `NO_XDEV`, synchronized ownership/type/link/mode/size checks, descriptor transfer, reverse + uncancelled cleanup, primary `DomainError` preservation, and policy-valid continuous replacement races. This lease is + intentionally not wired to a privileged action or launch. +- [ ] Extend open-once beneath/no-symlink/no-magic-link traversal to snapshot, volume, output, and image-provisioning + resources under their separately defined policies. +- [ ] Verify the remaining resource classes' caller ownership, file type, fixed basenames, size/capacity, and allowed + access mode on pinned descriptors. +- [ ] Ensure every remaining resource consumer uses its admitted descriptor without pathname reopen. +- [ ] Keep future resource leases private to the owning service package, expose no generic FD/path accessor, and release + retained descriptors in reverse order with checked `context.WithoutCancel` cleanup. +- [ ] Preserve primary `DomainError` code, class, entity, and details for close/fsync/cleanup failures in every remaining + resource lease. +- [ ] Add policy-valid continuous symlink/rename replacement races for every remaining user-owned ancestor and leaf. - [ ] Task 7 consumes same-package Jailer leases; Task 13 consumes a separate private loopmount image lease. Neither consumer may reconstruct or accept `/proc/self/fd` paths from the request. From 67bdfb2608c014e6832b7ce5b492736b3564c6c6 Mon Sep 17 00:00:00 2001 From: AlanD20 Date: Sat, 29 Aug 2026 00:22:48 +0200 Subject: [PATCH 0054/1457] docs: define trusted release source policy --- ...owned-single-binary-privileged-dispatch.md | 22 +++++++++++++++++++ tasks/todo.md | 3 +++ 2 files changed, 25 insertions(+) diff --git a/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md b/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md index 93ef9478..da6a1b81 100644 --- a/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md +++ b/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md @@ -222,6 +222,28 @@ force-replaced. If upstream provides a stable verifiable signature or provenance mechanism, mvmctl should adopt it. Until then, the fixed HTTPS origin and independently fetched checksum are an explicit supply-chain limitation, not a signature claim. +The receiver derives the source identity from the canonical release slot; none of these values are request fields. The +naming follows Firecracker's +[official binary-installation contract](https://github.com/firecracker-microvm/firecracker/blob/main/docs/getting-started.md#getting-a-firecracker-binary). +For version `` without a leading `v` and architecture `` in `x86_64|aarch64`, the fixed +derivation is: + +| Value | Receiver-derived form | +|---|---| +| Official origin | `https://github.com/firecracker-microvm/firecracker/releases/download` | +| Release tag | `v` | +| Archive | `firecracker-v-.tgz` | +| Checksum sidecar | `firecracker-v-.tgz.sha256.txt` | +| Archive root | `release-v-` | +| Firecracker member | `release-v-/firecracker-v-` | +| Jailer member | `release-v-/jailer-v-` | +| Trusted-store leaves | `firecracker`, `jailer`, and `manifest.json` | + +This table freezes source construction and the two extracted member identities. It does not claim that those are the +only members in the upstream archive: Task 6 separately records and enforces the complete reviewed upstream allowlist +before extraction is accepted. A packaging change must update the reviewed contract and tests; it must not fall back to +basename matching, caller-selected member names, or permissive extraction. + ### Paths, process identity, and runtime state Privileged code treats the normal CLI, arguments, environment, database, cache, manifests, and downloaded content as diff --git a/tasks/todo.md b/tasks/todo.md index a30ec788..ddebf4a2 100644 --- a/tasks/todo.md +++ b/tasks/todo.md @@ -198,6 +198,9 @@ under `internal/service/jailer/`. No handler, transport, CLI, API, core-domain, - [ ] Store exact releases under `/var/lib/mvmctl/binaries//` with fixed Firecracker, Jailer, and strict root-owned manifest leaves. - [ ] Privileged code constructs the fixed official release/checksum locations from validated version and architecture. + - [x] Freeze the official origin, tag, archive, checksum-sidecar, archive-root, exact Firecracker/Jailer member, and + trusted-store leaf derivation. No derived value is accepted from the caller. + - [ ] Implement and verify the private typed source derivation. - [ ] Root obtains the checksum independently through a dedicated bounded HTTPS-only source with proxies disabled. It may consume an exact-length bounded archive stream supplied by the unprivileged client for mirror efficiency, but it accepts no caller path, URL, or checksum and never opens or trusts the user asset mirror as authority. From 5db19678e76b818461591f2bb518b73b46fe29d1 Mon Sep 17 00:00:00 2001 From: AlanD20 Date: Sat, 29 Aug 2026 00:29:19 +0200 Subject: [PATCH 0055/1457] feat: derive trusted release sources --- .../service/jailer/trusted_release_source.go | 54 +++++++++++ .../jailer/trusted_release_source_test.go | 93 +++++++++++++++++++ 2 files changed, 147 insertions(+) create mode 100644 internal/service/jailer/trusted_release_source.go create mode 100644 internal/service/jailer/trusted_release_source_test.go diff --git a/internal/service/jailer/trusted_release_source.go b/internal/service/jailer/trusted_release_source.go new file mode 100644 index 00000000..0cf3268d --- /dev/null +++ b/internal/service/jailer/trusted_release_source.go @@ -0,0 +1,54 @@ +package jailer + +import ( + "fmt" + + "mvmctl/internal/infra" +) + +const ( + trustedReleaseFirecrackerLeaf = "firecracker" + trustedReleaseJailerLeaf = "jailer" + trustedReleaseManifestLeaf = "manifest.json" +) + +type trustedReleaseSource struct { + slot releaseSlot + archiveName string + archiveURL string + checksumURL string + archiveRoot string + firecrackerMember string + jailerMember string +} + +// CRITICAL: Every source identity is derived from the validated release slot. No caller-supplied URL, archive name, or +// member name crosses the privileged release boundary. +func newTrustedReleaseSource(slot releaseSlot) (trustedReleaseSource, error) { + if err := validateReleaseSlotValue(slot); err != nil { + return trustedReleaseSource{}, instanceValidationError(err.Error()) + } + + releaseTag := "v" + slot.version + artifactSuffix := releaseTag + "-" + slot.architecture + firecrackerName := "firecracker-" + artifactSuffix + jailerName := "jailer-" + artifactSuffix + archiveName := firecrackerName + ".tgz" + archiveRoot := "release-" + artifactSuffix + archiveURL := fmt.Sprintf( + "%s/%s/%s", + infra.FirecrackerGithubDownloadURL, + releaseTag, + archiveName, + ) + + return trustedReleaseSource{ + slot: slot, + archiveName: archiveName, + archiveURL: archiveURL, + checksumURL: archiveURL + ".sha256.txt", + archiveRoot: archiveRoot, + firecrackerMember: archiveRoot + "/" + firecrackerName, + jailerMember: archiveRoot + "/" + jailerName, + }, nil +} diff --git a/internal/service/jailer/trusted_release_source_test.go b/internal/service/jailer/trusted_release_source_test.go new file mode 100644 index 00000000..80ccc380 --- /dev/null +++ b/internal/service/jailer/trusted_release_source_test.go @@ -0,0 +1,93 @@ +package jailer + +import ( + "testing" + + "github.com/google/go-cmp/cmp" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "mvmctl/pkg/errs" +) + +func TestNewTrustedReleaseSourceDerivesOfficialIdentity(t *testing.T) { + t.Parallel() + + tests := []struct { + name string + slot releaseSlot + want trustedReleaseSource + }{ + { + name: "x86_64 patch release", + slot: releaseSlot{version: "1.16.1", architecture: "x86_64"}, + want: trustedReleaseSource{ + slot: releaseSlot{version: "1.16.1", architecture: "x86_64"}, + archiveName: "firecracker-v1.16.1-x86_64.tgz", + archiveURL: "https://github.com/firecracker-microvm/firecracker/releases/download/v1.16.1/firecracker-v1.16.1-x86_64.tgz", + checksumURL: "https://github.com/firecracker-microvm/firecracker/releases/download/v1.16.1/firecracker-v1.16.1-x86_64.tgz.sha256.txt", + archiveRoot: "release-v1.16.1-x86_64", + firecrackerMember: "release-v1.16.1-x86_64/firecracker-v1.16.1-x86_64", + jailerMember: "release-v1.16.1-x86_64/jailer-v1.16.1-x86_64", + }, + }, + { + name: "aarch64 prerelease", + slot: releaseSlot{version: "0.22.0-beta5", architecture: "aarch64"}, + want: trustedReleaseSource{ + slot: releaseSlot{version: "0.22.0-beta5", architecture: "aarch64"}, + archiveName: "firecracker-v0.22.0-beta5-aarch64.tgz", + archiveURL: "https://github.com/firecracker-microvm/firecracker/releases/download/v0.22.0-beta5/firecracker-v0.22.0-beta5-aarch64.tgz", + checksumURL: "https://github.com/firecracker-microvm/firecracker/releases/download/v0.22.0-beta5/firecracker-v0.22.0-beta5-aarch64.tgz.sha256.txt", + archiveRoot: "release-v0.22.0-beta5-aarch64", + firecrackerMember: "release-v0.22.0-beta5-aarch64/firecracker-v0.22.0-beta5-aarch64", + jailerMember: "release-v0.22.0-beta5-aarch64/jailer-v0.22.0-beta5-aarch64", + }, + }, + } + + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + + got, err := newTrustedReleaseSource(tc.slot) + require.NoError(t, err) + if diff := cmp.Diff(tc.want, got, cmp.AllowUnexported(trustedReleaseSource{}, releaseSlot{})); diff != "" { + t.Errorf("newTrustedReleaseSource() mismatch (-want +got):\n%s", diff) + } + }) + } +} + +func TestNewTrustedReleaseSourceRejectsInvalidSlot(t *testing.T) { + t.Parallel() + + tests := map[string]releaseSlot{ + "empty version": {version: "", architecture: "x86_64"}, + "leading v": {version: "v1.16.1", architecture: "x86_64"}, + "path-like version": {version: "1.16.1/../../root", architecture: "x86_64"}, + "query-like version": {version: "1.16.1?download=1", architecture: "x86_64"}, + "architecture alias": {version: "1.16.1", architecture: "amd64"}, + "path-like architecture": {version: "1.16.1", architecture: "../x86_64"}, + } + + for name, slot := range tests { + t.Run(name, func(t *testing.T) { + t.Parallel() + + _, err := newTrustedReleaseSource(slot) + require.Error(t, err) + domainErr := errs.AsDomainError(err) + require.NotNil(t, domainErr) + assert.Equal(t, errs.CodeValidationFailed, domainErr.Code) + }) + } +} + +func TestTrustedReleaseStoreLeavesAreFixed(t *testing.T) { + t.Parallel() + + assert.Equal(t, "firecracker", trustedReleaseFirecrackerLeaf) + assert.Equal(t, "jailer", trustedReleaseJailerLeaf) + assert.Equal(t, "manifest.json", trustedReleaseManifestLeaf) +} From 0bdd778d65ec06467f81b7406a31d3ca6f2a95c7 Mon Sep 17 00:00:00 2001 From: AlanD20 Date: Sat, 29 Aug 2026 00:29:59 +0200 Subject: [PATCH 0056/1457] docs: record trusted source derivation --- .../0016-root-owned-single-binary-privileged-dispatch.md | 6 ++++++ tasks/todo.md | 3 ++- 2 files changed, 8 insertions(+), 1 deletion(-) diff --git a/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md b/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md index da6a1b81..99007926 100644 --- a/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md +++ b/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md @@ -244,6 +244,12 @@ only members in the upstream archive: Task 6 separately records and enforces the before extraction is accepted. A packaging change must update the reviewed contract and tests; it must not fall back to basename matching, caller-selected member names, or permissive extraction. +Implementation note (2026-08-29): the private Jailer service derives the exact source identity from a validated +`(version, architecture)` release slot and rejects non-canonical slots before constructing any source value. It is not +yet wired to the privileged request or legacy installer. Independent checksum retrieval, bounded archive transport and +extraction, the complete member allowlist, ELF validation, manifest persistence, and atomic trusted-store installation +remain Task 6 work. + ### Paths, process identity, and runtime state Privileged code treats the normal CLI, arguments, environment, database, cache, manifests, and downloaded content as diff --git a/tasks/todo.md b/tasks/todo.md index ddebf4a2..0d3fdecf 100644 --- a/tasks/todo.md +++ b/tasks/todo.md @@ -200,7 +200,8 @@ under `internal/service/jailer/`. No handler, transport, CLI, API, core-domain, - [ ] Privileged code constructs the fixed official release/checksum locations from validated version and architecture. - [x] Freeze the official origin, tag, archive, checksum-sidecar, archive-root, exact Firecracker/Jailer member, and trusted-store leaf derivation. No derived value is accepted from the caller. - - [ ] Implement and verify the private typed source derivation. + - [x] Implement and verify the private typed source derivation for both architectures and canonical prerelease tags; + reject non-canonical or path-like slots before constructing any source value. - [ ] Root obtains the checksum independently through a dedicated bounded HTTPS-only source with proxies disabled. It may consume an exact-length bounded archive stream supplied by the unprivileged client for mirror efficiency, but it accepts no caller path, URL, or checksum and never opens or trusts the user asset mirror as authority. From 8b410070b826a0c9f331d818a87ab37bdfe518a5 Mon Sep 17 00:00:00 2001 From: AlanD20 Date: Sat, 29 Aug 2026 00:32:46 +0200 Subject: [PATCH 0057/1457] docs: define trusted checksum authority --- .../0016-root-owned-single-binary-privileged-dispatch.md | 9 +++++++++ tasks/todo.md | 3 +++ 2 files changed, 12 insertions(+) diff --git a/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md b/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md index 99007926..4655d08b 100644 --- a/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md +++ b/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md @@ -244,6 +244,15 @@ only members in the upstream archive: Task 6 separately records and enforces the before extraction is accepted. A packaging change must update the reviewed contract and tests; it must not fall back to basename matching, caller-selected member names, or permissive extraction. +Checksum authority uses a dedicated client rather than the ordinary downloader. It performs one retrieval attempt of +the derived sidecar URL with a 15-second total deadline, 5-second dial/TLS/header timeouts, a 16 KiB response-header +limit, transport compression disabled, TLS 1.2 or newer, no proxy function, no response cache, and at most one redirect. +That redirect must remain HTTPS, contain no user information or fragment, and target exactly +`release-assets.githubusercontent.com`. +The response must be HTTP 200 and no more than 256 bytes whether or not it declares a content length. The accepted body +is exactly `<64 lowercase hexadecimal characters>`. Missing or additional +lines, CRLF, alternate filenames, uppercase digests, GNU binary markers, and other whitespace are rejected. + Implementation note (2026-08-29): the private Jailer service derives the exact source identity from a validated `(version, architecture)` release slot and rejects non-canonical slots before constructing any source value. It is not yet wired to the privileged request or legacy installer. Independent checksum retrieval, bounded archive transport and diff --git a/tasks/todo.md b/tasks/todo.md index 0d3fdecf..6d58365b 100644 --- a/tasks/todo.md +++ b/tasks/todo.md @@ -205,6 +205,9 @@ under `internal/service/jailer/`. No handler, transport, CLI, API, core-domain, - [ ] Root obtains the checksum independently through a dedicated bounded HTTPS-only source with proxies disabled. It may consume an exact-length bounded archive stream supplied by the unprivileged client for mirror efficiency, but it accepts no caller path, URL, or checksum and never opens or trusts the user asset mirror as authority. + - [x] Freeze the one-request timeout, transport, redirect, status, body-size, and exact checksum-sidecar grammar + policy. The ordinary downloader and its proxy/cache/retry behavior are outside this trust boundary. + - [ ] Implement and verify the private checksum authority and typed archive digest. - [ ] Bounded extraction rejects traversal, links, devices, duplicates, unexpected members, and size/count overflow. - [ ] Validate the complete reviewed upstream member allowlist while extracting only Firecracker and Jailer; validate their ELF class/machine without executing downloaded code. From 40620d19799c4e83726ccb88ef3716edcd28c552 Mon Sep 17 00:00:00 2001 From: AlanD20 Date: Sat, 29 Aug 2026 00:42:52 +0200 Subject: [PATCH 0058/1457] feat: verify trusted release checksums Add a proxy-free bounded HTTPS authority for receiver-derived Firecracker checksum sidecars. Enforce a one-redirect GitHub asset policy, exact lowercase sha256sum grammar, typed digests, checked response cleanup, and hermetic adversarial tests. --- .../jailer/trusted_release_checksum.go | 218 ++++++++ .../jailer/trusted_release_checksum_test.go | 481 ++++++++++++++++++ .../service/jailer/trusted_release_source.go | 12 + 3 files changed, 711 insertions(+) create mode 100644 internal/service/jailer/trusted_release_checksum.go create mode 100644 internal/service/jailer/trusted_release_checksum_test.go diff --git a/internal/service/jailer/trusted_release_checksum.go b/internal/service/jailer/trusted_release_checksum.go new file mode 100644 index 00000000..93bc378d --- /dev/null +++ b/internal/service/jailer/trusted_release_checksum.go @@ -0,0 +1,218 @@ +package jailer + +import ( + "context" + "crypto/sha256" + "crypto/tls" + "encoding/hex" + "errors" + "fmt" + "io" + "net" + "net/http" + "time" + + "mvmctl/pkg/errs" +) + +const ( + trustedReleaseChecksumMaxBytes = 256 + trustedReleaseChecksumMaxHeaderBytes = 16 * 1024 + trustedReleaseChecksumPhaseTimeout = 5 * time.Second + trustedReleaseChecksumTotalTimeout = 15 * time.Second + trustedReleaseChecksumAssetHost = "release-assets.githubusercontent.com" + trustedReleaseChecksumAccept = "text/plain" + trustedReleaseChecksumUserAgent = "mvmctl-trusted-release/1" +) + +type trustedReleaseArchiveDigest [sha256.Size]byte + +type trustedReleaseChecksumAuthority struct { + client *http.Client +} + +func newTrustedReleaseChecksumAuthority() *trustedReleaseChecksumAuthority { + dialer := &net.Dialer{ + Timeout: trustedReleaseChecksumPhaseTimeout, + KeepAlive: -1, + } + transport := &http.Transport{ + Proxy: nil, + DialContext: dialer.DialContext, + ForceAttemptHTTP2: true, + MaxConnsPerHost: 1, + DisableKeepAlives: true, + DisableCompression: true, + TLSClientConfig: &tls.Config{MinVersion: tls.VersionTLS12}, + TLSHandshakeTimeout: trustedReleaseChecksumPhaseTimeout, + ResponseHeaderTimeout: trustedReleaseChecksumPhaseTimeout, + MaxResponseHeaderBytes: trustedReleaseChecksumMaxHeaderBytes, + } + return &trustedReleaseChecksumAuthority{client: &http.Client{ + Transport: transport, + CheckRedirect: checkTrustedReleaseChecksumRedirect, + Timeout: trustedReleaseChecksumTotalTimeout, + }} +} + +// CRITICAL: The request target is reconstructed and verified before this method performs a network effect. The +// dedicated client has no proxy, cache, decompression, or retry path through which ambient user state can gain authority. +func (authority *trustedReleaseChecksumAuthority) fetch( + ctx context.Context, + source trustedReleaseSource, +) (digest trustedReleaseArchiveDigest, returnErr error) { + if err := validateTrustedReleaseSource(source); err != nil { + return trustedReleaseArchiveDigest{}, err + } + if err := ctx.Err(); err != nil { + return trustedReleaseArchiveDigest{}, trustedReleaseChecksumDownloadError( + "fetch trusted release checksum", + err, + ) + } + request, err := http.NewRequestWithContext(ctx, http.MethodGet, source.checksumURL, nil) + if err != nil { + return trustedReleaseArchiveDigest{}, trustedReleaseChecksumDownloadError( + "create trusted release checksum request", + err, + ) + } + request.Header.Set("Accept", trustedReleaseChecksumAccept) + request.Header.Set("Accept-Encoding", "identity") + request.Header.Set("Cache-Control", "no-store") + request.Header.Set("User-Agent", trustedReleaseChecksumUserAgent) + + response, err := authority.client.Do(request) + if err != nil { + // http.Client closes any non-nil response body returned with a redirect-policy error. + return trustedReleaseArchiveDigest{}, trustedReleaseChecksumDownloadError( + "fetch trusted release checksum", + err, + ) + } + defer func() { + if closeErr := response.Body.Close(); closeErr != nil { + returnErr = appendTrustedReleaseChecksumCloseError(returnErr, closeErr) + } + }() + + if response.StatusCode != http.StatusOK { + return trustedReleaseArchiveDigest{}, trustedReleaseChecksumDownloadError( + fmt.Sprintf("trusted release checksum returned HTTP status %d", response.StatusCode), + nil, + ) + } + if response.ContentLength < -1 || response.ContentLength > trustedReleaseChecksumMaxBytes { + return trustedReleaseArchiveDigest{}, trustedReleaseChecksumDownloadError( + "trusted release checksum response exceeds size limit", + nil, + ) + } + raw, err := io.ReadAll(io.LimitReader(response.Body, trustedReleaseChecksumMaxBytes+1)) + if err != nil { + return trustedReleaseArchiveDigest{}, trustedReleaseChecksumDownloadError( + "read trusted release checksum response", + err, + ) + } + if len(raw) > trustedReleaseChecksumMaxBytes { + return trustedReleaseArchiveDigest{}, trustedReleaseChecksumDownloadError( + "trusted release checksum response exceeds size limit", + nil, + ) + } + + return parseTrustedReleaseChecksum(raw, source) +} + +func checkTrustedReleaseChecksumRedirect(request *http.Request, via []*http.Request) error { + if len(via) != 1 || request.Method != http.MethodGet || request.URL.Scheme != "https" || + request.URL.Host != trustedReleaseChecksumAssetHost || request.URL.User != nil || + request.URL.Fragment != "" || request.URL.RawFragment != "" { + return errs.New( + errs.CodeBinaryUntrusted, + "trusted release checksum redirect is not permitted", + ) + } + return nil +} + +func parseTrustedReleaseChecksum( + raw []byte, + source trustedReleaseSource, +) (trustedReleaseArchiveDigest, error) { + if err := validateTrustedReleaseSource(source); err != nil { + return trustedReleaseArchiveDigest{}, err + } + expectedLength := sha256.Size*2 + 2 + len(source.archiveName) + 1 + if len(raw) > trustedReleaseChecksumMaxBytes || len(raw) != expectedLength { + return trustedReleaseArchiveDigest{}, trustedReleaseChecksumUntrustedError( + "trusted release checksum sidecar has invalid length", + nil, + ) + } + digestEnd := sha256.Size * 2 + if !authorityHashPattern.Match(raw[:digestEnd]) { + return trustedReleaseArchiveDigest{}, trustedReleaseChecksumUntrustedError( + "trusted release checksum digest is not lowercase SHA-256", + nil, + ) + } + if string(raw[digestEnd:digestEnd+2]) != " " { + return trustedReleaseArchiveDigest{}, trustedReleaseChecksumUntrustedError( + "trusted release checksum separator is invalid", + nil, + ) + } + if string(raw[digestEnd+2:len(raw)-1]) != source.archiveName || raw[len(raw)-1] != '\n' { + return trustedReleaseArchiveDigest{}, trustedReleaseChecksumUntrustedError( + "trusted release checksum archive identity is invalid", + nil, + ) + } + + var digest trustedReleaseArchiveDigest + if _, err := hex.Decode(digest[:], raw[:digestEnd]); err != nil { + return trustedReleaseArchiveDigest{}, trustedReleaseChecksumUntrustedError( + "decode trusted release checksum digest", + err, + ) + } + return digest, nil +} + +func trustedReleaseChecksumDownloadError(message string, cause error) *errs.DomainError { + if cause == nil { + return errs.New(errs.CodeDownloadFailed, message, errs.WithClass(errs.ClassInternal)) + } + return errs.WrapMsg( + errs.CodeDownloadFailed, + message, + cause, + errs.WithClass(errs.ClassInternal), + ) +} + +func trustedReleaseChecksumUntrustedError(message string, cause error) *errs.DomainError { + if cause == nil { + return errs.New(errs.CodeBinaryUntrusted, message) + } + return errs.WrapMsg(errs.CodeBinaryUntrusted, message, cause, errs.WithClass(errs.ClassValidation)) +} + +func appendTrustedReleaseChecksumCloseError(primary, cause error) *errs.DomainError { + const message = "close trusted release checksum response" + if primary == nil { + return trustedReleaseChecksumDownloadError(message, cause) + } + domainErr := errs.AsDomainError(primary) + if domainErr == nil { + return trustedReleaseChecksumDownloadError( + primary.Error()+"; "+message+": "+cause.Error(), + errors.Join(primary, cause), + ) + } + domainErr.Message += "; " + message + ": " + cause.Error() + domainErr.Err = errors.Join(domainErr.Err, cause) + return domainErr +} diff --git a/internal/service/jailer/trusted_release_checksum_test.go b/internal/service/jailer/trusted_release_checksum_test.go new file mode 100644 index 00000000..8704a8f3 --- /dev/null +++ b/internal/service/jailer/trusted_release_checksum_test.go @@ -0,0 +1,481 @@ +package jailer + +import ( + "context" + "crypto/tls" + "errors" + "io" + "net/http" + "strings" + "testing" + "time" + + "github.com/google/go-cmp/cmp" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "mvmctl/pkg/errs" +) + +const testTrustedReleaseDigestText = "000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f" + +func TestParseTrustedReleaseChecksumAcceptsExactSidecar(t *testing.T) { + t.Parallel() + + tests := []releaseSlot{ + {version: "1.16.1", architecture: "x86_64"}, + {version: "1.16.1", architecture: "aarch64"}, + } + for _, slot := range tests { + t.Run(slot.architecture, func(t *testing.T) { + t.Parallel() + + source, err := newTrustedReleaseSource(slot) + require.NoError(t, err) + raw := []byte(testTrustedReleaseDigestText + " " + source.archiveName + "\n") + + got, err := parseTrustedReleaseChecksum(raw, source) + require.NoError(t, err) + var want trustedReleaseArchiveDigest + for index := range want { + want[index] = byte(index) + } + if diff := cmp.Diff(want, got); diff != "" { + t.Errorf("parseTrustedReleaseChecksum() mismatch (-want +got):\n%s", diff) + } + }) + } +} + +func TestParseTrustedReleaseChecksumRejectsNonCanonicalSidecar(t *testing.T) { + t.Parallel() + + source, err := newTrustedReleaseSource(releaseSlot{version: "1.16.1", architecture: "x86_64"}) + require.NoError(t, err) + valid := testTrustedReleaseDigestText + " " + source.archiveName + "\n" + + tests := map[string]string{ + "empty": "", + "short digest": testTrustedReleaseDigestText[:62] + " " + source.archiveName + "\n", + "uppercase digest": strings.ToUpper(testTrustedReleaseDigestText) + " " + source.archiveName + "\n", + "non-hex digest": "g" + testTrustedReleaseDigestText[1:] + " " + source.archiveName + "\n", + "single separator": testTrustedReleaseDigestText + " " + source.archiveName + "\n", + "tab separator": testTrustedReleaseDigestText + "\t " + source.archiveName + "\n", + "GNU binary marker": testTrustedReleaseDigestText + " *" + source.archiveName + "\n", + "alternate filename": testTrustedReleaseDigestText + " other.tgz\n", + "missing newline": strings.TrimSuffix(valid, "\n"), + "CRLF": strings.TrimSuffix(valid, "\n") + "\r\n", + "additional line": valid + valid, + "body above hard limit": strings.Repeat("x", trustedReleaseChecksumMaxBytes+1), + } + + for name, raw := range tests { + t.Run(name, func(t *testing.T) { + t.Parallel() + + _, parseErr := parseTrustedReleaseChecksum([]byte(raw), source) + require.Error(t, parseErr) + domainErr := errs.AsDomainError(parseErr) + require.NotNil(t, domainErr) + assert.Equal(t, errs.CodeBinaryUntrusted, domainErr.Code) + }) + } +} + +func TestParseTrustedReleaseChecksumRejectsForgedSource(t *testing.T) { + t.Parallel() + + source, err := newTrustedReleaseSource(releaseSlot{version: "1.16.1", architecture: "x86_64"}) + require.NoError(t, err) + raw := []byte(testTrustedReleaseDigestText + " " + source.archiveName + "\n") + source.archiveName = "attacker-controlled.tgz" + + _, err = parseTrustedReleaseChecksum(raw, source) + require.Error(t, err) + domainErr := errs.AsDomainError(err) + require.NotNil(t, domainErr) + assert.Equal(t, errs.CodeBinaryUntrusted, domainErr.Code) +} + +func TestTrustedReleaseChecksumAuthorityUsesClosedTransportPolicy(t *testing.T) { + t.Parallel() + + authority := newTrustedReleaseChecksumAuthority() + require.NotNil(t, authority) + require.NotNil(t, authority.client) + assert.Equal(t, trustedReleaseChecksumTotalTimeout, authority.client.Timeout) + assert.NotNil(t, authority.client.CheckRedirect) + + transport, ok := authority.client.Transport.(*http.Transport) + require.True(t, ok) + assert.Nil(t, transport.Proxy) + assert.NotNil(t, transport.DialContext) + assert.True(t, transport.DisableCompression) + assert.True(t, transport.DisableKeepAlives) + assert.True(t, transport.ForceAttemptHTTP2) + assert.Equal(t, 1, transport.MaxConnsPerHost) + assert.Equal(t, trustedReleaseChecksumPhaseTimeout, transport.TLSHandshakeTimeout) + assert.Equal(t, trustedReleaseChecksumPhaseTimeout, transport.ResponseHeaderTimeout) + assert.Equal(t, int64(trustedReleaseChecksumMaxHeaderBytes), transport.MaxResponseHeaderBytes) + require.NotNil(t, transport.TLSClientConfig) + assert.Equal(t, uint16(tls.VersionTLS12), transport.TLSClientConfig.MinVersion) +} + +func TestTrustedReleaseChecksumRedirectPolicy(t *testing.T) { + t.Parallel() + + source, err := newTrustedReleaseSource(releaseSlot{version: "1.16.1", architecture: "x86_64"}) + require.NoError(t, err) + initial, err := http.NewRequest(http.MethodGet, source.checksumURL, nil) + require.NoError(t, err) + asset, err := http.NewRequest( + http.MethodGet, + "https://release-assets.githubusercontent.com/github-production-release-asset/checksum?token=opaque", + nil, + ) + require.NoError(t, err) + + tests := []struct { + name string + rawURL string + via []*http.Request + wantErr bool + }{ + { + name: "single HTTPS asset redirect", + rawURL: asset.URL.String(), + via: []*http.Request{initial}, + }, + { + name: "HTTP redirect", + rawURL: "http://release-assets.githubusercontent.com/asset", + via: []*http.Request{initial}, + wantErr: true, + }, + { + name: "foreign host", + rawURL: "https://example.com/asset", + via: []*http.Request{initial}, + wantErr: true, + }, + { + name: "asset host with port", + rawURL: "https://release-assets.githubusercontent.com:443/asset", + via: []*http.Request{initial}, + wantErr: true, + }, + { + name: "userinfo", + rawURL: "https://user@release-assets.githubusercontent.com/asset", + via: []*http.Request{initial}, + wantErr: true, + }, + { + name: "fragment", + rawURL: "https://release-assets.githubusercontent.com/asset#fragment", + via: []*http.Request{initial}, + wantErr: true, + }, + { + name: "second redirect", + rawURL: asset.URL.String(), + via: []*http.Request{initial, asset}, + wantErr: true, + }, + } + + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + + request, requestErr := http.NewRequest(http.MethodGet, tc.rawURL, nil) + require.NoError(t, requestErr) + redirectErr := checkTrustedReleaseChecksumRedirect(request, tc.via) + if tc.wantErr { + assert.Error(t, redirectErr) + return + } + assert.NoError(t, redirectErr) + }) + } +} + +func TestTrustedReleaseChecksumAuthorityFetchesDerivedSidecar(t *testing.T) { + t.Parallel() + + source, err := newTrustedReleaseSource(releaseSlot{version: "1.16.1", architecture: "x86_64"}) + require.NoError(t, err) + body := newTrackingChecksumBody(testTrustedReleaseDigestText+" "+source.archiveName+"\n", nil, nil) + var captured *http.Request + roundTripper := trustedReleaseRoundTripFunc(func(request *http.Request) (*http.Response, error) { + captured = request + return &http.Response{ + StatusCode: http.StatusOK, + ContentLength: int64(len(testTrustedReleaseDigestText + " " + source.archiveName + "\n")), + Body: body, + Request: request, + }, nil + }) + authority := testTrustedReleaseChecksumAuthority(roundTripper) + + got, err := authority.fetch(context.Background(), source) + require.NoError(t, err) + var want trustedReleaseArchiveDigest + for index := range want { + want[index] = byte(index) + } + if diff := cmp.Diff(want, got); diff != "" { + t.Errorf("fetch() digest mismatch (-want +got):\n%s", diff) + } + require.NotNil(t, captured) + assert.Equal(t, http.MethodGet, captured.Method) + assert.Equal(t, source.checksumURL, captured.URL.String()) + assert.Equal(t, trustedReleaseChecksumAccept, captured.Header.Get("Accept")) + assert.Equal(t, "identity", captured.Header.Get("Accept-Encoding")) + assert.Equal(t, "no-store", captured.Header.Get("Cache-Control")) + assert.Equal(t, trustedReleaseChecksumUserAgent, captured.Header.Get("User-Agent")) + assert.Empty(t, captured.Header.Get("Authorization")) + assert.Empty(t, captured.Header.Get("Cookie")) + assert.True(t, body.closed) +} + +func TestTrustedReleaseChecksumAuthorityRejectsForgedSourceBeforeRequest(t *testing.T) { + t.Parallel() + + source, err := newTrustedReleaseSource(releaseSlot{version: "1.16.1", architecture: "x86_64"}) + require.NoError(t, err) + source.checksumURL = "https://example.com/attacker-controlled" + called := false + authority := testTrustedReleaseChecksumAuthority(trustedReleaseRoundTripFunc( + func(*http.Request) (*http.Response, error) { + called = true + return nil, errors.New("request must not run") + }, + )) + + _, err = authority.fetch(context.Background(), source) + require.Error(t, err) + domainErr := errs.AsDomainError(err) + require.NotNil(t, domainErr) + assert.Equal(t, errs.CodeBinaryUntrusted, domainErr.Code) + assert.False(t, called) +} + +func TestTrustedReleaseChecksumAuthorityBoundsResponse(t *testing.T) { + t.Parallel() + + source, err := newTrustedReleaseSource(releaseSlot{version: "1.16.1", architecture: "x86_64"}) + require.NoError(t, err) + valid := testTrustedReleaseDigestText + " " + source.archiveName + "\n" + + tests := []struct { + name string + status int + contentLength int64 + body string + readErr error + closeErr error + wantCode errs.Code + }{ + { + name: "non-OK status", + status: http.StatusNotFound, + contentLength: int64(len("missing")), + body: "missing", + wantCode: errs.CodeDownloadFailed, + }, + { + name: "declared body above limit", + status: http.StatusOK, + contentLength: trustedReleaseChecksumMaxBytes + 1, + wantCode: errs.CodeDownloadFailed, + }, + { + name: "invalid negative content length", + status: http.StatusOK, + contentLength: -2, + wantCode: errs.CodeDownloadFailed, + }, + { + name: "streamed body above limit", + status: http.StatusOK, + contentLength: -1, + body: strings.Repeat("x", trustedReleaseChecksumMaxBytes+1), + wantCode: errs.CodeDownloadFailed, + }, + { + name: "body read failure", + status: http.StatusOK, + contentLength: -1, + readErr: errors.New("read failed"), + wantCode: errs.CodeDownloadFailed, + }, + { + name: "body close failure", + status: http.StatusOK, + contentLength: int64(len(valid)), + body: valid, + closeErr: errors.New("close failed"), + wantCode: errs.CodeDownloadFailed, + }, + { + name: "malformed sidecar", + status: http.StatusOK, + contentLength: int64(len("not-a-checksum\n")), + body: "not-a-checksum\n", + wantCode: errs.CodeBinaryUntrusted, + }, + { + name: "malformed sidecar with close failure", + status: http.StatusOK, + contentLength: int64(len("not-a-checksum\n")), + body: "not-a-checksum\n", + closeErr: errors.New("close failed"), + wantCode: errs.CodeBinaryUntrusted, + }, + } + + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + + body := newTrackingChecksumBody(tc.body, tc.readErr, tc.closeErr) + calls := 0 + authority := testTrustedReleaseChecksumAuthority(trustedReleaseRoundTripFunc( + func(request *http.Request) (*http.Response, error) { + calls++ + return &http.Response{ + StatusCode: tc.status, + ContentLength: tc.contentLength, + Body: body, + Request: request, + }, nil + }, + )) + + _, fetchErr := authority.fetch(context.Background(), source) + require.Error(t, fetchErr) + domainErr := errs.AsDomainError(fetchErr) + require.NotNil(t, domainErr) + assert.Equal(t, tc.wantCode, domainErr.Code) + assert.Equal(t, 1, calls) + assert.True(t, body.closed) + }) + } +} + +func TestTrustedReleaseChecksumAuthorityDoesNotRetryTransportFailure(t *testing.T) { + t.Parallel() + + source, err := newTrustedReleaseSource(releaseSlot{version: "1.16.1", architecture: "x86_64"}) + require.NoError(t, err) + calls := 0 + authority := testTrustedReleaseChecksumAuthority(trustedReleaseRoundTripFunc( + func(*http.Request) (*http.Response, error) { + calls++ + return nil, errors.New("network unavailable") + }, + )) + + _, err = authority.fetch(context.Background(), source) + require.Error(t, err) + domainErr := errs.AsDomainError(err) + require.NotNil(t, domainErr) + assert.Equal(t, errs.CodeDownloadFailed, domainErr.Code) + assert.Equal(t, 1, calls) +} + +func TestTrustedReleaseChecksumAuthorityHonorsCanceledContextBeforeRequest(t *testing.T) { + t.Parallel() + + source, err := newTrustedReleaseSource(releaseSlot{version: "1.16.1", architecture: "x86_64"}) + require.NoError(t, err) + called := false + authority := testTrustedReleaseChecksumAuthority(trustedReleaseRoundTripFunc( + func(*http.Request) (*http.Response, error) { + called = true + return nil, errors.New("request must not run") + }, + )) + ctx, cancel := context.WithCancel(context.Background()) + cancel() + + _, err = authority.fetch(ctx, source) + require.Error(t, err) + assert.ErrorIs(t, err, context.Canceled) + assert.False(t, called) +} + +func TestTrustedReleaseChecksumAuthorityPropagatesInFlightCancellation(t *testing.T) { + t.Parallel() + + source, err := newTrustedReleaseSource(releaseSlot{version: "1.16.1", architecture: "x86_64"}) + require.NoError(t, err) + started := make(chan struct{}) + authority := testTrustedReleaseChecksumAuthority(trustedReleaseRoundTripFunc( + func(request *http.Request) (*http.Response, error) { + close(started) + <-request.Context().Done() + return nil, request.Context().Err() + }, + )) + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + result := make(chan error, 1) + go func() { + _, fetchErr := authority.fetch(ctx, source) + result <- fetchErr + }() + select { + case <-started: + case <-time.After(time.Second): + t.Fatal("checksum request did not enter the transport") + } + cancel() + + select { + case err = <-result: + case <-time.After(time.Second): + t.Fatal("checksum request did not stop after cancellation") + } + require.Error(t, err) + assert.ErrorIs(t, err, context.Canceled) +} + +type trustedReleaseRoundTripFunc func(*http.Request) (*http.Response, error) + +func (roundTrip trustedReleaseRoundTripFunc) RoundTrip(request *http.Request) (*http.Response, error) { + return roundTrip(request) +} + +type trackingChecksumBody struct { + reader io.Reader + readErr error + closeErr error + closed bool +} + +func newTrackingChecksumBody(value string, readErr, closeErr error) *trackingChecksumBody { + return &trackingChecksumBody{reader: strings.NewReader(value), readErr: readErr, closeErr: closeErr} +} + +func (body *trackingChecksumBody) Read(buffer []byte) (int, error) { + count, err := body.reader.Read(buffer) + if err == io.EOF && body.readErr != nil { + return count, body.readErr + } + return count, err +} + +func (body *trackingChecksumBody) Close() error { + body.closed = true + return body.closeErr +} + +func testTrustedReleaseChecksumAuthority(roundTripper http.RoundTripper) *trustedReleaseChecksumAuthority { + return &trustedReleaseChecksumAuthority{client: &http.Client{ + Transport: roundTripper, + CheckRedirect: checkTrustedReleaseChecksumRedirect, + Timeout: 2 * time.Second, + }} +} diff --git a/internal/service/jailer/trusted_release_source.go b/internal/service/jailer/trusted_release_source.go index 0cf3268d..1e5b2482 100644 --- a/internal/service/jailer/trusted_release_source.go +++ b/internal/service/jailer/trusted_release_source.go @@ -4,6 +4,7 @@ import ( "fmt" "mvmctl/internal/infra" + "mvmctl/pkg/errs" ) const ( @@ -52,3 +53,14 @@ func newTrustedReleaseSource(slot releaseSlot) (trustedReleaseSource, error) { jailerMember: archiveRoot + "/" + jailerName, }, nil } + +func validateTrustedReleaseSource(source trustedReleaseSource) error { + expected, err := newTrustedReleaseSource(source.slot) + if err != nil || source != expected { + return errs.New( + errs.CodeBinaryUntrusted, + "trusted release source identity is inconsistent", + ) + } + return nil +} From d77385fae0f1914aaa0851bf137ba6e8983a6b87 Mon Sep 17 00:00:00 2001 From: AlanD20 Date: Sat, 29 Aug 2026 00:43:41 +0200 Subject: [PATCH 0059/1457] docs: record trusted checksum authority --- ...016-root-owned-single-binary-privileged-dispatch.md | 10 ++++++---- tasks/todo.md | 3 ++- 2 files changed, 8 insertions(+), 5 deletions(-) diff --git a/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md b/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md index 4655d08b..5d37ab43 100644 --- a/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md +++ b/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md @@ -254,10 +254,12 @@ is exactly `<64 lowercase hexadecimal characters> Date: Sat, 29 Aug 2026 00:46:43 +0200 Subject: [PATCH 0060/1457] docs: define trusted ELF admission --- .../0016-root-owned-single-binary-privileged-dispatch.md | 6 ++++++ tasks/todo.md | 4 ++++ 2 files changed, 10 insertions(+) diff --git a/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md b/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md index 5d37ab43..67b0fad0 100644 --- a/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md +++ b/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md @@ -219,6 +219,12 @@ class and machine must match the selected architecture without executing downloa release manifest are fsynced and atomically renamed into the trusted store. A referenced release cannot be force-replaced. +ELF admission reads exactly the first 64 bytes of each extracted binary and performs no executable handoff. It requires +the ELF magic, ELF64 class, little-endian encoding, current ident/object versions, System V ABI version 0 with zero ident +padding, `ET_DYN`, a non-zero entry point, a 64-byte ELF header, program headers beginning at byte 64 with 56-byte +entries, and 1 through 64 program headers. The machine is exactly `EM_X86_64` (62) for `x86_64` or `EM_AARCH64` (183) +for `aarch64`. A future upstream format change fails closed until this reviewed policy and its tests are updated. + If upstream provides a stable verifiable signature or provenance mechanism, mvmctl should adopt it. Until then, the fixed HTTPS origin and independently fetched checksum are an explicit supply-chain limitation, not a signature claim. diff --git a/tasks/todo.md b/tasks/todo.md index b3315de7..9f7c00e8 100644 --- a/tasks/todo.md +++ b/tasks/todo.md @@ -212,6 +212,10 @@ under `internal/service/jailer/`. No handler, transport, CLI, API, core-domain, - [ ] Bounded extraction rejects traversal, links, devices, duplicates, unexpected members, and size/count overflow. - [ ] Validate the complete reviewed upstream member allowlist while extracting only Firecracker and Jailer; validate their ELF class/machine without executing downloaded code. + - [ ] Complete the architecture-specific member audit. The existing asset mirror proves the 24-member x86_64 layout + across v1.10.1 and v1.14.2-v1.16.1, but contains no aarch64 archive; do not infer its allowlist from x86_64. + - [x] Freeze the exact bounded ELF64 header-admission policy for x86_64 and aarch64. + - [ ] Implement and verify the private ELF header validator. - [ ] Exact Firecracker/Jailer bytes and a root manifest install atomically and durably. - [ ] Referenced release removal/replacement holds the release lease and fails closed on unreadable records. - [ ] L1 failure injection proves an old complete pair or no pair, never a partial trusted release. From 6c1d34cc3ab0bc8cbe5cd8eb11e523f6a170f46b Mon Sep 17 00:00:00 2001 From: AlanD20 Date: Sat, 29 Aug 2026 00:49:16 +0200 Subject: [PATCH 0061/1457] feat: validate trusted release ELF headers Add a bounded byte-level ELF64 admission check for receiver-selected x86_64 and aarch64 releases. Reject truncated, structurally different, wrong-architecture, and forged-source candidates without loading or executing them. --- .../service/jailer/trusted_release_elf.go | 77 +++++++++++ .../jailer/trusted_release_elf_test.go | 125 ++++++++++++++++++ 2 files changed, 202 insertions(+) create mode 100644 internal/service/jailer/trusted_release_elf.go create mode 100644 internal/service/jailer/trusted_release_elf_test.go diff --git a/internal/service/jailer/trusted_release_elf.go b/internal/service/jailer/trusted_release_elf.go new file mode 100644 index 00000000..bb4a2612 --- /dev/null +++ b/internal/service/jailer/trusted_release_elf.go @@ -0,0 +1,77 @@ +package jailer + +import ( + "debug/elf" + "encoding/binary" + + "mvmctl/pkg/errs" +) + +const ( + trustedReleaseELFHeaderBytes = 64 + trustedReleaseELFProgramHeaderSize = 56 + trustedReleaseELFMaxProgramHeaders = 64 +) + +// CRITICAL: This parser admits only the bounded header shape reviewed in ADR-0016. It never lets untrusted ELF counts +// drive allocation and never loads or executes the candidate bytes. +func validateTrustedReleaseELFHeader(raw []byte, source trustedReleaseSource) error { + if err := validateTrustedReleaseSource(source); err != nil { + return err + } + if len(raw) != trustedReleaseELFHeaderBytes { + return trustedReleaseELFError("trusted release ELF header has invalid length") + } + if raw[0] != 0x7f || raw[1] != 'E' || raw[2] != 'L' || raw[3] != 'F' { + return trustedReleaseELFError("trusted release executable has invalid ELF magic") + } + if elf.Class(raw[4]) != elf.ELFCLASS64 { + return trustedReleaseELFError("trusted release executable is not ELF64") + } + if elf.Data(raw[5]) != elf.ELFDATA2LSB { + return trustedReleaseELFError("trusted release executable is not little-endian ELF") + } + if elf.Version(raw[6]) != elf.EV_CURRENT || elf.OSABI(raw[7]) != elf.ELFOSABI_NONE || raw[8] != 0 { + return trustedReleaseELFError("trusted release executable has unsupported ELF identity") + } + for _, value := range raw[9:16] { + if value != 0 { + return trustedReleaseELFError("trusted release executable has non-zero ELF identity padding") + } + } + if elf.Type(binary.LittleEndian.Uint16(raw[16:18])) != elf.ET_DYN { + return trustedReleaseELFError("trusted release executable is not a position-independent ELF object") + } + machine := elf.Machine(binary.LittleEndian.Uint16(raw[18:20])) + expectedMachine := elf.EM_X86_64 + if source.slot.architecture == "aarch64" { + expectedMachine = elf.EM_AARCH64 + } + if machine != expectedMachine { + return trustedReleaseELFError("trusted release executable architecture does not match release slot") + } + if elf.Version(binary.LittleEndian.Uint32(raw[20:24])) != elf.EV_CURRENT { + return trustedReleaseELFError("trusted release executable has unsupported ELF object version") + } + if binary.LittleEndian.Uint64(raw[24:32]) == 0 { + return trustedReleaseELFError("trusted release executable has no ELF entry point") + } + if binary.LittleEndian.Uint64(raw[32:40]) != trustedReleaseELFHeaderBytes { + return trustedReleaseELFError("trusted release executable has unexpected program-header offset") + } + if binary.LittleEndian.Uint16(raw[52:54]) != trustedReleaseELFHeaderBytes { + return trustedReleaseELFError("trusted release executable has unexpected ELF header size") + } + if binary.LittleEndian.Uint16(raw[54:56]) != trustedReleaseELFProgramHeaderSize { + return trustedReleaseELFError("trusted release executable has unexpected program-header size") + } + programHeaderCount := binary.LittleEndian.Uint16(raw[56:58]) + if programHeaderCount == 0 || programHeaderCount > trustedReleaseELFMaxProgramHeaders { + return trustedReleaseELFError("trusted release executable has unsupported program-header count") + } + return nil +} + +func trustedReleaseELFError(message string) *errs.DomainError { + return errs.New(errs.CodeBinaryUntrusted, message) +} diff --git a/internal/service/jailer/trusted_release_elf_test.go b/internal/service/jailer/trusted_release_elf_test.go new file mode 100644 index 00000000..991c0613 --- /dev/null +++ b/internal/service/jailer/trusted_release_elf_test.go @@ -0,0 +1,125 @@ +package jailer + +import ( + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "mvmctl/pkg/errs" +) + +func TestValidateTrustedReleaseELFHeaderAcceptsSelectedArchitecture(t *testing.T) { + t.Parallel() + + tests := []struct { + name string + architecture string + machineLow byte + }{ + {name: "x86_64", architecture: "x86_64", machineLow: 62}, + {name: "aarch64", architecture: "aarch64", machineLow: 183}, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + + source, err := newTrustedReleaseSource(releaseSlot{ + version: "1.16.1", + architecture: tc.architecture, + }) + require.NoError(t, err) + header := auditedTrustedReleaseELFHeader() + header[18] = tc.machineLow + + assert.NoError(t, validateTrustedReleaseELFHeader(header, source)) + }) + } +} + +// Rationale: Root admits executable bytes before they can become a trusted release. Every field below distinguishes the +// reviewed Firecracker/Jailer ELF shape from a truncated, wrong-architecture, or structurally different object. +func TestValidateTrustedReleaseELFHeaderRejectsUntrustedShape(t *testing.T) { + t.Parallel() + + source, err := newTrustedReleaseSource(releaseSlot{version: "1.16.1", architecture: "x86_64"}) + require.NoError(t, err) + + tests := []struct { + name string + mutate func([]byte) []byte + }{ + {name: "truncated header", mutate: func(value []byte) []byte { return value[:63] }}, + {name: "extended header", mutate: func(value []byte) []byte { return append(value, 0) }}, + {name: "invalid magic", mutate: mutateTrustedReleaseELFByte(0, 0)}, + {name: "ELF32 class", mutate: mutateTrustedReleaseELFByte(4, 1)}, + {name: "big-endian encoding", mutate: mutateTrustedReleaseELFByte(5, 2)}, + {name: "old ident version", mutate: mutateTrustedReleaseELFByte(6, 0)}, + {name: "foreign OS ABI", mutate: mutateTrustedReleaseELFByte(7, 3)}, + {name: "non-zero ABI version", mutate: mutateTrustedReleaseELFByte(8, 1)}, + {name: "non-zero ident padding", mutate: mutateTrustedReleaseELFByte(15, 1)}, + {name: "executable object type", mutate: mutateTrustedReleaseELFByte(16, 2)}, + {name: "aarch64 machine for x86_64 slot", mutate: mutateTrustedReleaseELFByte(18, 183)}, + {name: "old object version", mutate: mutateTrustedReleaseELFByte(20, 0)}, + {name: "zero entry point", mutate: func(value []byte) []byte { + for index := 24; index < 32; index++ { + value[index] = 0 + } + return value + }}, + {name: "wrong program-header offset", mutate: mutateTrustedReleaseELFByte(32, 65)}, + {name: "wrong ELF header size", mutate: mutateTrustedReleaseELFByte(52, 63)}, + {name: "wrong program-header entry size", mutate: mutateTrustedReleaseELFByte(54, 55)}, + {name: "zero program headers", mutate: mutateTrustedReleaseELFByte(56, 0)}, + {name: "too many program headers", mutate: mutateTrustedReleaseELFByte(56, 65)}, + } + + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + + header := tc.mutate(auditedTrustedReleaseELFHeader()) + err := validateTrustedReleaseELFHeader(header, source) + require.Error(t, err) + domainErr := errs.AsDomainError(err) + require.NotNil(t, domainErr) + assert.Equal(t, errs.CodeBinaryUntrusted, domainErr.Code) + }) + } +} + +func TestValidateTrustedReleaseELFHeaderRejectsForgedSource(t *testing.T) { + t.Parallel() + + source, err := newTrustedReleaseSource(releaseSlot{version: "1.16.1", architecture: "x86_64"}) + require.NoError(t, err) + source.archiveRoot = "attacker-controlled" + + err = validateTrustedReleaseELFHeader(auditedTrustedReleaseELFHeader(), source) + require.Error(t, err) + domainErr := errs.AsDomainError(err) + require.NotNil(t, domainErr) + assert.Equal(t, errs.CodeBinaryUntrusted, domainErr.Code) +} + +func mutateTrustedReleaseELFByte(offset int, value byte) func([]byte) []byte { + return func(header []byte) []byte { + header[offset] = value + return header + } +} + +// auditedTrustedReleaseELFHeader is the Firecracker v1.16.1 x86_64 header observed in the configured asset mirror. The +// validator contract is derived from the ELF specification and ADR-0016, not by calling the production parser. +func auditedTrustedReleaseELFHeader() []byte { + return []byte{ + 0x7f, 0x45, 0x4c, 0x46, 0x02, 0x01, 0x01, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x03, 0x00, 0x3e, 0x00, 0x01, 0x00, 0x00, 0x00, + 0x79, 0xe7, 0x02, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0xe0, 0xcb, 0x35, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x40, 0x00, 0x38, 0x00, + 0x0a, 0x00, 0x40, 0x00, 0x1d, 0x00, 0x1c, 0x00, + } +} From d9321cf6ca50d2410d982ed9922666ab610f7dcc Mon Sep 17 00:00:00 2001 From: AlanD20 Date: Sat, 29 Aug 2026 00:50:52 +0200 Subject: [PATCH 0062/1457] docs: record trusted ELF admission --- .../0016-root-owned-single-binary-privileged-dispatch.md | 7 ++++--- tasks/todo.md | 3 ++- 2 files changed, 6 insertions(+), 4 deletions(-) diff --git a/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md b/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md index 67b0fad0..e7a02ca6 100644 --- a/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md +++ b/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md @@ -263,9 +263,10 @@ Implementation note (2026-08-29): the private Jailer service derives the exact s `(version, architecture)` release slot and rejects non-canonical slots before constructing any source value. Its dedicated checksum authority independently fetches the derived sidecar with the closed transport policy above and returns a typed archive digest only after exact grammar validation. Forged private source values fail before a request, -and response cleanup preserves the primary `DomainError` metadata. These foundations are not yet wired to the -privileged request or legacy installer. Bounded archive transport and extraction, the complete member allowlist, ELF -validation, manifest persistence, and atomic trusted-store installation remain Task 6 work. +and response cleanup preserves the primary `DomainError` metadata. The private ELF admission parser validates the +bounded header shape and selected architecture without loading or executing candidate bytes. These foundations are not +yet wired to the privileged request or legacy installer. Bounded archive transport and extraction, the complete member +allowlist, manifest persistence, and atomic trusted-store installation remain Task 6 work. ### Paths, process identity, and runtime state diff --git a/tasks/todo.md b/tasks/todo.md index 9f7c00e8..504347d5 100644 --- a/tasks/todo.md +++ b/tasks/todo.md @@ -215,7 +215,8 @@ under `internal/service/jailer/`. No handler, transport, CLI, API, core-domain, - [ ] Complete the architecture-specific member audit. The existing asset mirror proves the 24-member x86_64 layout across v1.10.1 and v1.14.2-v1.16.1, but contains no aarch64 archive; do not infer its allowlist from x86_64. - [x] Freeze the exact bounded ELF64 header-admission policy for x86_64 and aarch64. - - [ ] Implement and verify the private ELF header validator. + - [x] Implement and verify the private ELF header validator: exact source identity and 64-byte header length, closed + identity/type/machine/program-header policy, no untrusted allocation, and no executable loading or execution. - [ ] Exact Firecracker/Jailer bytes and a root manifest install atomically and durably. - [ ] Referenced release removal/replacement holds the release lease and fails closed on unreadable records. - [ ] L1 failure injection proves an old complete pair or no pair, never a partial trusted release. From 8b1f867447cc518e43f364cdce674f5bf9534999 Mon Sep 17 00:00:00 2001 From: AlanD20 Date: Sat, 29 Aug 2026 00:54:20 +0200 Subject: [PATCH 0063/1457] docs: define trusted release manifest --- ...owned-single-binary-privileged-dispatch.md | 23 +++++++++++++++++++ tasks/plan.md | 2 +- tasks/todo.md | 3 +++ 3 files changed, 27 insertions(+), 1 deletion(-) diff --git a/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md b/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md index e7a02ca6..aefda13d 100644 --- a/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md +++ b/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md @@ -259,6 +259,29 @@ The response must be HTTP 200 and no more than 256 bytes whether or not it decla is exactly `<64 lowercase hexadecimal characters>`. Missing or additional lines, CRLF, alternate filenames, uppercase digests, GNU binary markers, and other whitespace are rejected. +The fixed `manifest.json` leaf uses schema version 1 and is at most 4 KiB. Its closed JSON schema contains exactly the +canonical release slot, the archive SHA-256 digest, and separate Firecracker and Jailer objects containing a SHA-256 +digest and `size_bytes`. All digests are exactly 64 lowercase hexadecimal characters. Each executable size is between +120 bytes (one complete ELF64 header and one program header) and 64 MiB. The current mirrored x86_64 releases use no +more than 3,527,456 bytes for either selected executable, so the upper bound leaves substantial growth room while +preventing a corrupt root manifest from authorizing an unbounded verification pass. An upstream executable exceeding +the bound fails closed until the policy is reviewed; the bound is architecture-neutral and does not substitute for the +still-required aarch64 archive-member audit. + +```json +{ + "schema_version": 1, + "release": {"version": "1.16.1", "architecture": "x86_64"}, + "archive_sha256": "", + "firecracker": {"sha256": "", "size_bytes": 3527456}, + "jailer": {"sha256": "", "size_bytes": 2181264} +} +``` + +Decoding rejects unknown, missing, duplicate, case-conflicting, trailing, wrongly typed, oversized, or out-of-policy +values. The executable hashes in a successfully decoded manifest are the sole source of the release identity supplied +to instance registration; the unprivileged request never supplies them. + Implementation note (2026-08-29): the private Jailer service derives the exact source identity from a validated `(version, architecture)` release slot and rejects non-canonical slots before constructing any source value. Its dedicated checksum authority independently fetches the derived sidecar with the closed transport policy above and diff --git a/tasks/plan.md b/tasks/plan.md index 3068f2d0..03fb79fe 100644 --- a/tasks/plan.md +++ b/tasks/plan.md @@ -233,7 +233,7 @@ and `MVM_ASSET_MIRROR` supplies bytes rather than authority. Extraction validate allowlist and extracts only Firecracker and Jailer. The strict root-owned manifest stores schema version, release slot, archive hash, and each executable's hash and size. -The store is exactly `/var/lib/mvmctl/binaries///{firecracker,jailer,release.json}`. Binaries are +The store is exactly `/var/lib/mvmctl/binaries///{firecracker,jailer,manifest.json}`. Binaries are validated as ELF for the selected architecture without execution. Descriptor-relative atomic install, exchange, and removal use only fixed leaves and preserve an old complete release or expose a new complete release, never a partial pair. Post-commit errors report `release_installed`, `release_replaced`, `release_removed`, `durability_uncertain`, and diff --git a/tasks/todo.md b/tasks/todo.md index 504347d5..9e889e1e 100644 --- a/tasks/todo.md +++ b/tasks/todo.md @@ -218,6 +218,9 @@ under `internal/service/jailer/`. No handler, transport, CLI, API, core-domain, - [x] Implement and verify the private ELF header validator: exact source identity and 64-byte header length, closed identity/type/machine/program-header policy, no untrusted allocation, and no executable loading or execution. - [ ] Exact Firecracker/Jailer bytes and a root manifest install atomically and durably. + - [x] Freeze the strict schema-v1 manifest contract, 4 KiB record bound, typed SHA-256 fields, and 120-byte through + 64 MiB executable-size policy; the fixed store leaf is `manifest.json`. + - [ ] Implement and verify the private manifest codec and release-identity derivation. - [ ] Referenced release removal/replacement holds the release lease and fails closed on unreadable records. - [ ] L1 failure injection proves an old complete pair or no pair, never a partial trusted release. From 36b2771205700af22a26400a8f7b5f59b0135a72 Mon Sep 17 00:00:00 2001 From: AlanD20 Date: Sat, 29 Aug 2026 01:01:59 +0200 Subject: [PATCH 0064/1457] feat: validate trusted release manifests --- .../jailer/trusted_release_manifest.go | 363 ++++++++++++++++++ .../jailer/trusted_release_manifest_test.go | 302 +++++++++++++++ 2 files changed, 665 insertions(+) create mode 100644 internal/service/jailer/trusted_release_manifest.go create mode 100644 internal/service/jailer/trusted_release_manifest_test.go diff --git a/internal/service/jailer/trusted_release_manifest.go b/internal/service/jailer/trusted_release_manifest.go new file mode 100644 index 00000000..ee35a455 --- /dev/null +++ b/internal/service/jailer/trusted_release_manifest.go @@ -0,0 +1,363 @@ +package jailer + +import ( + "bytes" + "crypto/sha256" + "encoding/hex" + "encoding/json" + "fmt" + "io" + "strings" + + "mvmctl/pkg/errs" +) + +const ( + trustedReleaseManifestSchemaVersion = uint32(1) + maxTrustedReleaseManifestBytes = 4 * 1024 + trustedReleaseManifestJSONMaxDepth = 1 + trustedReleaseExecutableMinBytes = uint64( + trustedReleaseELFHeaderBytes + trustedReleaseELFProgramHeaderSize, + ) + trustedReleaseExecutableMaxBytes = uint64(64 * 1024 * 1024) +) + +type trustedReleaseExecutableDigest [sha256.Size]byte + +type trustedReleaseExecutable struct { + digest trustedReleaseExecutableDigest + sizeBytes uint64 +} + +type trustedReleaseManifest struct { + schemaVersion uint32 + slot releaseSlot + archiveDigest trustedReleaseArchiveDigest + firecracker trustedReleaseExecutable + jailer trustedReleaseExecutable +} + +type trustedReleaseManifestWire struct { + SchemaVersion uint32 `json:"schema_version"` + Release trustedReleaseManifestSlotWire `json:"release"` + ArchiveSHA256 string `json:"archive_sha256"` + Firecracker trustedReleaseExecutableWire `json:"firecracker"` + Jailer trustedReleaseExecutableWire `json:"jailer"` +} + +type trustedReleaseManifestSlotWire struct { + Version string `json:"version"` + Architecture string `json:"architecture"` +} + +type trustedReleaseExecutableWire struct { + SHA256 string `json:"sha256"` + SizeBytes uint64 `json:"size_bytes"` +} + +var trustedReleaseManifestObjectFields = map[string]map[string]struct{}{ + "": { + "schema_version": {}, + "release": {}, + "archive_sha256": {}, + "firecracker": {}, + "jailer": {}, + }, + "release": { + "version": {}, + "architecture": {}, + }, + "firecracker": { + "sha256": {}, + "size_bytes": {}, + }, + "jailer": { + "sha256": {}, + "size_bytes": {}, + }, +} + +func encodeTrustedReleaseManifest(manifest trustedReleaseManifest) ([]byte, error) { + if err := validateTrustedReleaseManifest(manifest); err != nil { + return nil, err + } + raw, err := json.Marshal(trustedReleaseManifestToWire(manifest)) + if err != nil { + return nil, trustedReleaseManifestError("encode trusted release manifest", err) + } + if len(raw) > maxTrustedReleaseManifestBytes { + return nil, trustedReleaseManifestError("encoded trusted release manifest exceeds size limit", nil) + } + return raw, nil +} + +func decodeTrustedReleaseManifest(raw []byte) (trustedReleaseManifest, error) { + if len(raw) == 0 { + return trustedReleaseManifest{}, trustedReleaseManifestError("trusted release manifest is empty", nil) + } + if len(raw) > maxTrustedReleaseManifestBytes { + return trustedReleaseManifest{}, trustedReleaseManifestError( + "trusted release manifest exceeds size limit", + nil, + ) + } + if err := validateTrustedReleaseManifestJSON(raw); err != nil { + return trustedReleaseManifest{}, err + } + + decoder := json.NewDecoder(bytes.NewReader(raw)) + decoder.DisallowUnknownFields() + var wire trustedReleaseManifestWire + if err := decoder.Decode(&wire); err != nil { + return trustedReleaseManifest{}, trustedReleaseManifestError("decode trusted release manifest", err) + } + if err := requireTrustedReleaseManifestJSONEOF(decoder); err != nil { + return trustedReleaseManifest{}, trustedReleaseManifestError( + "decode trailing trusted release manifest input", + err, + ) + } + + manifest, err := trustedReleaseManifestFromWire(wire) + if err != nil { + return trustedReleaseManifest{}, err + } + if err := validateTrustedReleaseManifest(manifest); err != nil { + return trustedReleaseManifest{}, err + } + return manifest, nil +} + +func validateTrustedReleaseManifest(manifest trustedReleaseManifest) error { + if manifest.schemaVersion != trustedReleaseManifestSchemaVersion { + return trustedReleaseManifestError("unsupported trusted release manifest schema", nil) + } + if err := validateReleaseSlotValue(manifest.slot); err != nil { + return trustedReleaseManifestError("trusted release manifest slot is invalid", err) + } + if err := validateTrustedReleaseExecutable("Firecracker", manifest.firecracker); err != nil { + return err + } + return validateTrustedReleaseExecutable("Jailer", manifest.jailer) +} + +func validateTrustedReleaseExecutable(name string, executable trustedReleaseExecutable) error { + if executable.sizeBytes < trustedReleaseExecutableMinBytes || + executable.sizeBytes > trustedReleaseExecutableMaxBytes { + return trustedReleaseManifestError( + fmt.Sprintf("trusted release %s size is outside the admitted range", name), + nil, + ) + } + return nil +} + +func (manifest trustedReleaseManifest) releaseIdentity() (releaseIdentity, error) { + if err := validateTrustedReleaseManifest(manifest); err != nil { + return releaseIdentity{}, err + } + identity := releaseIdentity{ + version: manifest.slot.version, + architecture: manifest.slot.architecture, + firecrackerSHA256: hex.EncodeToString(manifest.firecracker.digest[:]), + jailerSHA256: hex.EncodeToString(manifest.jailer.digest[:]), + } + if err := validateReleaseIdentityValue(identity); err != nil { + return releaseIdentity{}, trustedReleaseManifestError(err.Error(), nil) + } + return identity, nil +} + +func validateTrustedReleaseManifestJSON(raw []byte) error { + decoder := json.NewDecoder(bytes.NewReader(raw)) + decoder.UseNumber() + if err := scanTrustedReleaseManifestJSONValue(decoder, "", 0); err != nil { + return trustedReleaseManifestError("decode strict trusted release manifest", err) + } + if err := requireTrustedReleaseManifestJSONEOF(decoder); err != nil { + return trustedReleaseManifestError("decode trailing trusted release manifest input", err) + } + return nil +} + +func scanTrustedReleaseManifestJSONValue(decoder *json.Decoder, objectName string, depth int) error { + if depth > trustedReleaseManifestJSONMaxDepth { + return fmt.Errorf("JSON nesting exceeds %d levels", trustedReleaseManifestJSONMaxDepth) + } + token, err := decoder.Token() + if err != nil { + return err + } + delim, ok := token.(json.Delim) + if !ok || delim != '{' { + return fmt.Errorf("JSON object %q must be an object", objectName) + } + allowed, ok := trustedReleaseManifestObjectFields[objectName] + if !ok { + return fmt.Errorf("unexpected JSON object %q", objectName) + } + + seen := make(map[string]struct{}, len(allowed)) + for decoder.More() { + fieldToken, tokenErr := decoder.Token() + if tokenErr != nil { + return tokenErr + } + field, ok := fieldToken.(string) + if !ok { + return fmt.Errorf("JSON object field is not a string") + } + if _, exists := allowed[field]; !exists { + return fmt.Errorf("unknown JSON field %q", field) + } + for existing := range seen { + if strings.EqualFold(existing, field) { + return fmt.Errorf("duplicate JSON field %q conflicts with %q", field, existing) + } + } + seen[field] = struct{}{} + + switch field { + case "release", "firecracker", "jailer": + if err := scanTrustedReleaseManifestJSONValue(decoder, field, depth+1); err != nil { + return err + } + default: + if err := scanTrustedReleaseManifestJSONScalar(decoder); err != nil { + return err + } + } + } + endToken, err := decoder.Token() + if err != nil { + return err + } + if endToken != json.Delim('}') { + return fmt.Errorf("JSON object %q has an invalid terminator", objectName) + } + if len(seen) != len(allowed) { + return fmt.Errorf("JSON object %q is missing required fields", objectName) + } + return nil +} + +func scanTrustedReleaseManifestJSONScalar(decoder *json.Decoder) error { + token, err := decoder.Token() + if err != nil { + return err + } + if token == nil { + return fmt.Errorf("JSON field must not be null") + } + if _, ok := token.(json.Delim); ok { + return fmt.Errorf("unexpected nested JSON value") + } + return nil +} + +func requireTrustedReleaseManifestJSONEOF(decoder *json.Decoder) error { + var trailing json.RawMessage + err := decoder.Decode(&trailing) + if err == io.EOF { + return nil + } + if err == nil { + return fmt.Errorf("trusted release manifest contains trailing input") + } + return err +} + +func trustedReleaseManifestToWire(manifest trustedReleaseManifest) trustedReleaseManifestWire { + return trustedReleaseManifestWire{ + SchemaVersion: manifest.schemaVersion, + Release: trustedReleaseManifestSlotWire{ + Version: manifest.slot.version, + Architecture: manifest.slot.architecture, + }, + ArchiveSHA256: hex.EncodeToString(manifest.archiveDigest[:]), + Firecracker: trustedReleaseExecutableWire{ + SHA256: hex.EncodeToString(manifest.firecracker.digest[:]), + SizeBytes: manifest.firecracker.sizeBytes, + }, + Jailer: trustedReleaseExecutableWire{ + SHA256: hex.EncodeToString(manifest.jailer.digest[:]), + SizeBytes: manifest.jailer.sizeBytes, + }, + } +} + +func trustedReleaseManifestFromWire(wire trustedReleaseManifestWire) (trustedReleaseManifest, error) { + archiveDigest, err := decodeTrustedReleaseArchiveDigest(wire.ArchiveSHA256) + if err != nil { + return trustedReleaseManifest{}, err + } + firecrackerDigest, err := decodeTrustedReleaseExecutableDigest("Firecracker", wire.Firecracker.SHA256) + if err != nil { + return trustedReleaseManifest{}, err + } + jailerDigest, err := decodeTrustedReleaseExecutableDigest("Jailer", wire.Jailer.SHA256) + if err != nil { + return trustedReleaseManifest{}, err + } + + return trustedReleaseManifest{ + schemaVersion: wire.SchemaVersion, + slot: releaseSlot{ + version: wire.Release.Version, + architecture: wire.Release.Architecture, + }, + archiveDigest: archiveDigest, + firecracker: trustedReleaseExecutable{ + digest: firecrackerDigest, + sizeBytes: wire.Firecracker.SizeBytes, + }, + jailer: trustedReleaseExecutable{ + digest: jailerDigest, + sizeBytes: wire.Jailer.SizeBytes, + }, + }, nil +} + +func decodeTrustedReleaseArchiveDigest(raw string) (trustedReleaseArchiveDigest, error) { + if !authorityHashPattern.MatchString(raw) { + return trustedReleaseArchiveDigest{}, trustedReleaseManifestError( + "trusted release manifest archive digest is not lowercase SHA-256", + nil, + ) + } + var digest trustedReleaseArchiveDigest + if _, err := hex.Decode(digest[:], []byte(raw)); err != nil { + return trustedReleaseArchiveDigest{}, trustedReleaseManifestError( + "decode trusted release manifest archive digest", + err, + ) + } + return digest, nil +} + +func decodeTrustedReleaseExecutableDigest( + name string, + raw string, +) (trustedReleaseExecutableDigest, error) { + if !authorityHashPattern.MatchString(raw) { + return trustedReleaseExecutableDigest{}, trustedReleaseManifestError( + fmt.Sprintf("trusted release manifest %s digest is not lowercase SHA-256", name), + nil, + ) + } + var digest trustedReleaseExecutableDigest + if _, err := hex.Decode(digest[:], []byte(raw)); err != nil { + return trustedReleaseExecutableDigest{}, trustedReleaseManifestError( + fmt.Sprintf("decode trusted release manifest %s digest", name), + err, + ) + } + return digest, nil +} + +func trustedReleaseManifestError(message string, cause error) *errs.DomainError { + if cause == nil { + return errs.New(errs.CodeBinaryUntrusted, message) + } + return errs.WrapMsg(errs.CodeBinaryUntrusted, message, cause, errs.WithClass(errs.ClassValidation)) +} diff --git a/internal/service/jailer/trusted_release_manifest_test.go b/internal/service/jailer/trusted_release_manifest_test.go new file mode 100644 index 00000000..3816762f --- /dev/null +++ b/internal/service/jailer/trusted_release_manifest_test.go @@ -0,0 +1,302 @@ +package jailer + +import ( + "bytes" + "crypto/sha256" + "encoding/hex" + "fmt" + "strings" + "testing" + + "github.com/google/go-cmp/cmp" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "mvmctl/pkg/errs" +) + +func TestTrustedReleaseManifestCodecRoundTrip(t *testing.T) { + t.Parallel() + + tests := []releaseSlot{ + {version: "1.16.1", architecture: "x86_64"}, + {version: "0.22.0-beta5", architecture: "aarch64"}, + } + for _, slot := range tests { + t.Run(slot.architecture, func(t *testing.T) { + t.Parallel() + + want := testTrustedReleaseManifest(slot) + raw, err := encodeTrustedReleaseManifest(want) + require.NoError(t, err) + assert.LessOrEqual(t, len(raw), maxTrustedReleaseManifestBytes) + + got, err := decodeTrustedReleaseManifest(raw) + require.NoError(t, err) + if diff := cmp.Diff( + want, + got, + cmp.AllowUnexported(trustedReleaseManifest{}, releaseSlot{}, trustedReleaseExecutable{}), + ); diff != "" { + t.Errorf("trusted release manifest mismatch (-want +got):\n%s", diff) + } + }) + } +} + +func TestTrustedReleaseManifestCodecUsesCanonicalClosedSchema(t *testing.T) { + t.Parallel() + + manifest := testTrustedReleaseManifest(releaseSlot{version: "1.16.1", architecture: "x86_64"}) + raw, err := encodeTrustedReleaseManifest(manifest) + require.NoError(t, err) + + want := fmt.Sprintf( + `{"schema_version":%d,"release":{"version":"%s","architecture":"%s"},`+ + `"archive_sha256":"%s","firecracker":{"sha256":"%s","size_bytes":%d},`+ + `"jailer":{"sha256":"%s","size_bytes":%d}}`, + manifest.schemaVersion, + manifest.slot.version, + manifest.slot.architecture, + hex.EncodeToString(manifest.archiveDigest[:]), + hex.EncodeToString(manifest.firecracker.digest[:]), + manifest.firecracker.sizeBytes, + hex.EncodeToString(manifest.jailer.digest[:]), + manifest.jailer.sizeBytes, + ) + assert.Equal(t, want, string(raw)) + assert.NotContains(t, string(raw), "url") + assert.NotContains(t, string(raw), "path") + assert.NotContains(t, string(raw), "member") +} + +func TestDecodeTrustedReleaseManifestRejectsMalformedSchema(t *testing.T) { + t.Parallel() + + manifest := testTrustedReleaseManifest(releaseSlot{version: "1.16.1", architecture: "x86_64"}) + validBytes, err := encodeTrustedReleaseManifest(manifest) + require.NoError(t, err) + valid := string(validBytes) + archiveHash := hex.EncodeToString(manifest.archiveDigest[:]) + firecrackerHash := hex.EncodeToString(manifest.firecracker.digest[:]) + + tests := map[string]string{ + "empty": "", + "whitespace_only": " \n\t", + "trailing": valid + `{}`, + "root_array": `[]`, + "unknown_top_level": strings.Replace(valid, `"schema_version":`, `"unknown":1,"schema_version":`, 1), + "case_variant_field": strings.Replace(valid, `"schema_version":`, `"SCHEMA_VERSION":`, 1), + "duplicate_top_level": strings.Replace( + valid, + `"schema_version":`, + `"schema_version":1,"schema_version":`, + 1, + ), + "case_conflicting_field": strings.Replace( + valid, + `"schema_version":`, + `"SCHEMA_VERSION":1,"schema_version":`, + 1, + ), + "missing_top_level": strings.Replace( + valid, + `"archive_sha256":"`+archiveHash+`",`, + "", + 1, + ), + "release_null": strings.Replace( + valid, + `"release":{"version":"1.16.1","architecture":"x86_64"}`, + `"release":null`, + 1, + ), + "release_array": strings.Replace( + valid, + `"release":{"version":"1.16.1","architecture":"x86_64"}`, + `"release":[]`, + 1, + ), + "unknown_release_field": strings.Replace(valid, `"version":`, `"unknown":1,"version":`, 1), + "duplicate_release_field": strings.Replace( + valid, + `"version":`, + `"version":"1.16.1","version":`, + 1, + ), + "missing_release_field": strings.Replace(valid, `"version":"1.16.1",`, "", 1), + "invalid_version": strings.Replace(valid, `"version":"1.16.1"`, `"version":"../1.16.1"`, 1), + "invalid_architecture": strings.Replace(valid, `"architecture":"x86_64"`, `"architecture":"arm64"`, 1), + "executable_null": strings.Replace( + valid, + `"firecracker":{"sha256":"`+firecrackerHash+`","size_bytes":3527456}`, + `"firecracker":null`, + 1, + ), + "unknown_executable_field": strings.Replace( + valid, + `"sha256":"`+firecrackerHash+`"`, + `"unknown":1,"sha256":"`+firecrackerHash+`"`, + 1, + ), + "duplicate_executable_field": strings.Replace( + valid, + `"size_bytes":3527456`, + `"size_bytes":3527456,"size_bytes":3527456`, + 1, + ), + "missing_executable_field": strings.Replace(valid, `,"size_bytes":3527456`, "", 1), + "unsupported_schema": strings.Replace(valid, `"schema_version":1`, `"schema_version":2`, 1), + "schema_boolean": strings.Replace(valid, `"schema_version":1`, `"schema_version":true`, 1), + "uppercase_archive_hash": strings.Replace( + valid, + archiveHash, + strings.ToUpper(archiveHash), + 1, + ), + "short_archive_hash": strings.Replace(valid, archiveHash, archiveHash[:len(archiveHash)-1], 1), + "non_hex_executable_hash": strings.Replace( + valid, + firecrackerHash, + "g"+firecrackerHash[1:], + 1, + ), + "size_below_minimum": strings.Replace( + valid, + `"size_bytes":3527456`, + fmt.Sprintf(`"size_bytes":%d`, trustedReleaseExecutableMinBytes-1), + 1, + ), + "size_above_maximum": strings.Replace( + valid, + `"size_bytes":3527456`, + fmt.Sprintf(`"size_bytes":%d`, trustedReleaseExecutableMaxBytes+1), + 1, + ), + "negative_size": strings.Replace(valid, `"size_bytes":3527456`, `"size_bytes":-1`, 1), + "fractional_size": strings.Replace( + valid, + `"size_bytes":3527456`, + `"size_bytes":3527456.5`, + 1, + ), + "string_size": strings.Replace(valid, `"size_bytes":3527456`, `"size_bytes":"3527456"`, 1), + } + + for name, raw := range tests { + t.Run(name, func(t *testing.T) { + t.Parallel() + + _, decodeErr := decodeTrustedReleaseManifest([]byte(raw)) + require.Error(t, decodeErr) + assert.Equal(t, errs.CodeBinaryUntrusted, errs.AsDomainError(decodeErr).Code) + }) + } +} + +func TestDecodeTrustedReleaseManifestRejectsOversizedInput(t *testing.T) { + t.Parallel() + + _, err := decodeTrustedReleaseManifest([]byte(strings.Repeat(" ", maxTrustedReleaseManifestBytes+1))) + require.Error(t, err) + assert.Equal(t, errs.CodeBinaryUntrusted, errs.AsDomainError(err).Code) +} + +func TestEncodeTrustedReleaseManifestRejectsInvalidTypedValue(t *testing.T) { + t.Parallel() + + tests := map[string]func(*trustedReleaseManifest){ + "unsupported_schema": func(manifest *trustedReleaseManifest) { + manifest.schemaVersion++ + }, + "invalid_version": func(manifest *trustedReleaseManifest) { + manifest.slot.version = "../1.16.1" + }, + "invalid_architecture": func(manifest *trustedReleaseManifest) { + manifest.slot.architecture = "arm64" + }, + "firecracker_too_small": func(manifest *trustedReleaseManifest) { + manifest.firecracker.sizeBytes = trustedReleaseExecutableMinBytes - 1 + }, + "jailer_too_large": func(manifest *trustedReleaseManifest) { + manifest.jailer.sizeBytes = trustedReleaseExecutableMaxBytes + 1 + }, + } + + for name, mutate := range tests { + t.Run(name, func(t *testing.T) { + t.Parallel() + + manifest := testTrustedReleaseManifest(releaseSlot{version: "1.16.1", architecture: "x86_64"}) + mutate(&manifest) + _, err := encodeTrustedReleaseManifest(manifest) + require.Error(t, err) + assert.Equal(t, errs.CodeBinaryUntrusted, errs.AsDomainError(err).Code) + }) + } +} + +func TestTrustedReleaseManifestExecutableSizeBoundaries(t *testing.T) { + t.Parallel() + + want := testTrustedReleaseManifest(releaseSlot{version: "1.16.1", architecture: "x86_64"}) + want.firecracker.sizeBytes = trustedReleaseExecutableMinBytes + want.jailer.sizeBytes = trustedReleaseExecutableMaxBytes + + raw, err := encodeTrustedReleaseManifest(want) + require.NoError(t, err) + padded := append(raw, bytes.Repeat([]byte{' '}, maxTrustedReleaseManifestBytes-len(raw))...) + require.Len(t, padded, maxTrustedReleaseManifestBytes) + got, err := decodeTrustedReleaseManifest(padded) + require.NoError(t, err) + if diff := cmp.Diff( + want, + got, + cmp.AllowUnexported(trustedReleaseManifest{}, releaseSlot{}, trustedReleaseExecutable{}), + ); diff != "" { + t.Errorf("trusted release manifest boundaries mismatch (-want +got):\n%s", diff) + } +} + +func TestTrustedReleaseManifestDerivesInstanceReleaseIdentity(t *testing.T) { + t.Parallel() + + manifest := testTrustedReleaseManifest(releaseSlot{version: "1.16.1", architecture: "x86_64"}) + want := releaseIdentity{ + version: manifest.slot.version, + architecture: manifest.slot.architecture, + firecrackerSHA256: hex.EncodeToString(manifest.firecracker.digest[:]), + jailerSHA256: hex.EncodeToString(manifest.jailer.digest[:]), + } + + got, err := manifest.releaseIdentity() + require.NoError(t, err) + if diff := cmp.Diff(want, got, cmp.AllowUnexported(releaseIdentity{})); diff != "" { + t.Errorf("release identity mismatch (-want +got):\n%s", diff) + } + + manifest.schemaVersion++ + _, err = manifest.releaseIdentity() + require.Error(t, err) + assert.Equal(t, errs.CodeBinaryUntrusted, errs.AsDomainError(err).Code) +} + +func testTrustedReleaseManifest(slot releaseSlot) trustedReleaseManifest { + archiveDigest := sha256.Sum256([]byte("trusted release archive")) + firecrackerDigest := sha256.Sum256([]byte("trusted Firecracker executable")) + jailerDigest := sha256.Sum256([]byte("trusted Jailer executable")) + + return trustedReleaseManifest{ + schemaVersion: trustedReleaseManifestSchemaVersion, + slot: slot, + archiveDigest: trustedReleaseArchiveDigest(archiveDigest), + firecracker: trustedReleaseExecutable{ + digest: trustedReleaseExecutableDigest(firecrackerDigest), + sizeBytes: 3_527_456, + }, + jailer: trustedReleaseExecutable{ + digest: trustedReleaseExecutableDigest(jailerDigest), + sizeBytes: 2_181_264, + }, + } +} From b991bb2f331ac4a8da297f0a90d875293417c0bb Mon Sep 17 00:00:00 2001 From: AlanD20 Date: Sat, 29 Aug 2026 01:02:25 +0200 Subject: [PATCH 0065/1457] docs: record trusted release manifest codec --- .../0016-root-owned-single-binary-privileged-dispatch.md | 8 +++++--- tasks/todo.md | 2 +- 2 files changed, 6 insertions(+), 4 deletions(-) diff --git a/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md b/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md index aefda13d..487b8a7a 100644 --- a/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md +++ b/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md @@ -287,9 +287,11 @@ Implementation note (2026-08-29): the private Jailer service derives the exact s dedicated checksum authority independently fetches the derived sidecar with the closed transport policy above and returns a typed archive digest only after exact grammar validation. Forged private source values fail before a request, and response cleanup preserves the primary `DomainError` metadata. The private ELF admission parser validates the -bounded header shape and selected architecture without loading or executing candidate bytes. These foundations are not -yet wired to the privileged request or legacy installer. Bounded archive transport and extraction, the complete member -allowlist, manifest persistence, and atomic trusted-store installation remain Task 6 work. +bounded header shape and selected architecture without loading or executing candidate bytes. The private manifest codec +enforces the closed schema, record and executable bounds, typed digests, and manifest-derived instance release identity. +These foundations are not yet wired to the privileged request or legacy installer. Bounded archive transport and +extraction, the complete member allowlist, manifest persistence, and atomic trusted-store installation remain Task 6 +work. ### Paths, process identity, and runtime state diff --git a/tasks/todo.md b/tasks/todo.md index 9e889e1e..6c2de6f8 100644 --- a/tasks/todo.md +++ b/tasks/todo.md @@ -220,7 +220,7 @@ under `internal/service/jailer/`. No handler, transport, CLI, API, core-domain, - [ ] Exact Firecracker/Jailer bytes and a root manifest install atomically and durably. - [x] Freeze the strict schema-v1 manifest contract, 4 KiB record bound, typed SHA-256 fields, and 120-byte through 64 MiB executable-size policy; the fixed store leaf is `manifest.json`. - - [ ] Implement and verify the private manifest codec and release-identity derivation. + - [x] Implement and verify the private manifest codec and release-identity derivation. - [ ] Referenced release removal/replacement holds the release lease and fails closed on unreadable records. - [ ] L1 failure injection proves an old complete pair or no pair, never a partial trusted release. From c052996ebef2cdeb0798abada81df8a4cc242013 Mon Sep 17 00:00:00 2001 From: AlanD20 Date: Sat, 29 Aug 2026 01:03:31 +0200 Subject: [PATCH 0066/1457] docs: bind ELF admission to file size --- .../0016-root-owned-single-binary-privileged-dispatch.md | 6 ++++-- tasks/todo.md | 1 + 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md b/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md index 487b8a7a..227c7ecd 100644 --- a/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md +++ b/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md @@ -222,8 +222,10 @@ force-replaced. ELF admission reads exactly the first 64 bytes of each extracted binary and performs no executable handoff. It requires the ELF magic, ELF64 class, little-endian encoding, current ident/object versions, System V ABI version 0 with zero ident padding, `ET_DYN`, a non-zero entry point, a 64-byte ELF header, program headers beginning at byte 64 with 56-byte -entries, and 1 through 64 program headers. The machine is exactly `EM_X86_64` (62) for `x86_64` or `EM_AARCH64` (183) -for `aarch64`. A future upstream format change fails closed until this reviewed policy and its tests are updated. +entries, and 1 through 64 program headers. Admission also receives the exact full-file size, enforces the 120-byte +through 64 MiB executable policy, and requires the complete declared program-header table to fit within that size. The +machine is exactly `EM_X86_64` (62) for `x86_64` or `EM_AARCH64` (183) for `aarch64`. A future upstream format change +fails closed until this reviewed policy and its tests are updated. If upstream provides a stable verifiable signature or provenance mechanism, mvmctl should adopt it. Until then, the fixed HTTPS origin and independently fetched checksum are an explicit supply-chain limitation, not a signature claim. diff --git a/tasks/todo.md b/tasks/todo.md index 6c2de6f8..2c3e5159 100644 --- a/tasks/todo.md +++ b/tasks/todo.md @@ -217,6 +217,7 @@ under `internal/service/jailer/`. No handler, transport, CLI, API, core-domain, - [x] Freeze the exact bounded ELF64 header-admission policy for x86_64 and aarch64. - [x] Implement and verify the private ELF header validator: exact source identity and 64-byte header length, closed identity/type/machine/program-header policy, no untrusted allocation, and no executable loading or execution. + - [ ] Bind ELF admission to the bounded actual file size and reject a truncated declared program-header table. - [ ] Exact Firecracker/Jailer bytes and a root manifest install atomically and durably. - [x] Freeze the strict schema-v1 manifest contract, 4 KiB record bound, typed SHA-256 fields, and 120-byte through 64 MiB executable-size policy; the fixed store leaf is `manifest.json`. From 83f100a5ed7c1ee1a9816a4daad733805fb4b32d Mon Sep 17 00:00:00 2001 From: AlanD20 Date: Sat, 29 Aug 2026 01:05:12 +0200 Subject: [PATCH 0067/1457] fix: bind trusted ELF headers to file size --- .../service/jailer/trusted_release_elf.go | 21 ++++++- .../jailer/trusted_release_elf_test.go | 56 ++++++++++++++++++- .../jailer/trusted_release_manifest.go | 4 -- 3 files changed, 72 insertions(+), 9 deletions(-) diff --git a/internal/service/jailer/trusted_release_elf.go b/internal/service/jailer/trusted_release_elf.go index bb4a2612..018e4485 100644 --- a/internal/service/jailer/trusted_release_elf.go +++ b/internal/service/jailer/trusted_release_elf.go @@ -11,17 +11,28 @@ const ( trustedReleaseELFHeaderBytes = 64 trustedReleaseELFProgramHeaderSize = 56 trustedReleaseELFMaxProgramHeaders = 64 + trustedReleaseExecutableMinBytes = uint64( + trustedReleaseELFHeaderBytes + trustedReleaseELFProgramHeaderSize, + ) + trustedReleaseExecutableMaxBytes = uint64(64 * 1024 * 1024) ) // CRITICAL: This parser admits only the bounded header shape reviewed in ADR-0016. It never lets untrusted ELF counts // drive allocation and never loads or executes the candidate bytes. -func validateTrustedReleaseELFHeader(raw []byte, source trustedReleaseSource) error { +func validateTrustedReleaseELFHeader( + raw []byte, + sizeBytes uint64, + source trustedReleaseSource, +) error { if err := validateTrustedReleaseSource(source); err != nil { return err } if len(raw) != trustedReleaseELFHeaderBytes { return trustedReleaseELFError("trusted release ELF header has invalid length") } + if sizeBytes < trustedReleaseExecutableMinBytes || sizeBytes > trustedReleaseExecutableMaxBytes { + return trustedReleaseELFError("trusted release executable size is outside the admitted range") + } if raw[0] != 0x7f || raw[1] != 'E' || raw[2] != 'L' || raw[3] != 'F' { return trustedReleaseELFError("trusted release executable has invalid ELF magic") } @@ -56,7 +67,8 @@ func validateTrustedReleaseELFHeader(raw []byte, source trustedReleaseSource) er if binary.LittleEndian.Uint64(raw[24:32]) == 0 { return trustedReleaseELFError("trusted release executable has no ELF entry point") } - if binary.LittleEndian.Uint64(raw[32:40]) != trustedReleaseELFHeaderBytes { + programHeaderOffset := binary.LittleEndian.Uint64(raw[32:40]) + if programHeaderOffset != trustedReleaseELFHeaderBytes { return trustedReleaseELFError("trusted release executable has unexpected program-header offset") } if binary.LittleEndian.Uint16(raw[52:54]) != trustedReleaseELFHeaderBytes { @@ -69,6 +81,11 @@ func validateTrustedReleaseELFHeader(raw []byte, source trustedReleaseSource) er if programHeaderCount == 0 || programHeaderCount > trustedReleaseELFMaxProgramHeaders { return trustedReleaseELFError("trusted release executable has unsupported program-header count") } + programHeaderEnd := programHeaderOffset + + uint64(trustedReleaseELFProgramHeaderSize)*uint64(programHeaderCount) + if sizeBytes < programHeaderEnd { + return trustedReleaseELFError("trusted release executable has a truncated program-header table") + } return nil } diff --git a/internal/service/jailer/trusted_release_elf_test.go b/internal/service/jailer/trusted_release_elf_test.go index 991c0613..9c445a41 100644 --- a/internal/service/jailer/trusted_release_elf_test.go +++ b/internal/service/jailer/trusted_release_elf_test.go @@ -32,7 +32,7 @@ func TestValidateTrustedReleaseELFHeaderAcceptsSelectedArchitecture(t *testing.T header := auditedTrustedReleaseELFHeader() header[18] = tc.machineLow - assert.NoError(t, validateTrustedReleaseELFHeader(header, source)) + assert.NoError(t, validateTrustedReleaseELFHeader(header, 3_527_456, source)) }) } } @@ -79,7 +79,7 @@ func TestValidateTrustedReleaseELFHeaderRejectsUntrustedShape(t *testing.T) { t.Parallel() header := tc.mutate(auditedTrustedReleaseELFHeader()) - err := validateTrustedReleaseELFHeader(header, source) + err := validateTrustedReleaseELFHeader(header, 3_527_456, source) require.Error(t, err) domainErr := errs.AsDomainError(err) require.NotNil(t, domainErr) @@ -88,6 +88,56 @@ func TestValidateTrustedReleaseELFHeaderRejectsUntrustedShape(t *testing.T) { } } +// Rationale: Header fields are not meaningful admission bounds unless the complete declared program-header table fits +// within the separately measured file. These cases prove the exact inclusive file-size and table-extent boundaries. +func TestValidateTrustedReleaseELFHeaderBindsActualFileSize(t *testing.T) { + t.Parallel() + + source, err := newTrustedReleaseSource(releaseSlot{version: "1.16.1", architecture: "x86_64"}) + require.NoError(t, err) + + tests := map[string]struct { + sizeBytes uint64 + mutate func([]byte) + wantErr bool + }{ + "below executable minimum": {sizeBytes: 119, wantErr: true}, + "above executable maximum": {sizeBytes: 64*1024*1024 + 1, wantErr: true}, + // CONTRACT: audited header has e_phoff=64, e_phentsize=56, e_phnum=10; table end is byte 624. + "truncated audited table": {sizeBytes: 623, wantErr: true}, + "exact audited table": {sizeBytes: 624}, + // CONTRACT: maximum admitted e_phnum=64; table end is 64 + (56 * 64) = byte 3648. + "truncated maximum table": { + sizeBytes: 3647, + mutate: func(header []byte) { header[56] = 64 }, + wantErr: true, + }, + "exact maximum table": { + sizeBytes: 3648, + mutate: func(header []byte) { header[56] = 64 }, + }, + "exact executable maximum": {sizeBytes: 64 * 1024 * 1024}, + } + + for name, tc := range tests { + t.Run(name, func(t *testing.T) { + t.Parallel() + + header := auditedTrustedReleaseELFHeader() + if tc.mutate != nil { + tc.mutate(header) + } + err := validateTrustedReleaseELFHeader(header, tc.sizeBytes, source) + if tc.wantErr { + require.Error(t, err) + assert.Equal(t, errs.CodeBinaryUntrusted, errs.AsDomainError(err).Code) + return + } + assert.NoError(t, err) + }) + } +} + func TestValidateTrustedReleaseELFHeaderRejectsForgedSource(t *testing.T) { t.Parallel() @@ -95,7 +145,7 @@ func TestValidateTrustedReleaseELFHeaderRejectsForgedSource(t *testing.T) { require.NoError(t, err) source.archiveRoot = "attacker-controlled" - err = validateTrustedReleaseELFHeader(auditedTrustedReleaseELFHeader(), source) + err = validateTrustedReleaseELFHeader(auditedTrustedReleaseELFHeader(), 3_527_456, source) require.Error(t, err) domainErr := errs.AsDomainError(err) require.NotNil(t, domainErr) diff --git a/internal/service/jailer/trusted_release_manifest.go b/internal/service/jailer/trusted_release_manifest.go index ee35a455..81e16cc6 100644 --- a/internal/service/jailer/trusted_release_manifest.go +++ b/internal/service/jailer/trusted_release_manifest.go @@ -16,10 +16,6 @@ const ( trustedReleaseManifestSchemaVersion = uint32(1) maxTrustedReleaseManifestBytes = 4 * 1024 trustedReleaseManifestJSONMaxDepth = 1 - trustedReleaseExecutableMinBytes = uint64( - trustedReleaseELFHeaderBytes + trustedReleaseELFProgramHeaderSize, - ) - trustedReleaseExecutableMaxBytes = uint64(64 * 1024 * 1024) ) type trustedReleaseExecutableDigest [sha256.Size]byte From a71530780ff4346fb734be923d6bfaa2324a02a3 Mon Sep 17 00:00:00 2001 From: AlanD20 Date: Sat, 29 Aug 2026 01:05:32 +0200 Subject: [PATCH 0068/1457] docs: record bounded ELF file admission --- .../adr/0016-root-owned-single-binary-privileged-dispatch.md | 5 +++-- tasks/todo.md | 2 +- 2 files changed, 4 insertions(+), 3 deletions(-) diff --git a/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md b/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md index 227c7ecd..dbe3a590 100644 --- a/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md +++ b/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md @@ -289,8 +289,9 @@ Implementation note (2026-08-29): the private Jailer service derives the exact s dedicated checksum authority independently fetches the derived sidecar with the closed transport policy above and returns a typed archive digest only after exact grammar validation. Forged private source values fail before a request, and response cleanup preserves the primary `DomainError` metadata. The private ELF admission parser validates the -bounded header shape and selected architecture without loading or executing candidate bytes. The private manifest codec -enforces the closed schema, record and executable bounds, typed digests, and manifest-derived instance release identity. +bounded header shape, selected architecture, exact bounded file size, and complete program-header-table extent without +loading or executing candidate bytes. The private manifest codec enforces the closed schema, record and executable +bounds, typed digests, and manifest-derived instance release identity. These foundations are not yet wired to the privileged request or legacy installer. Bounded archive transport and extraction, the complete member allowlist, manifest persistence, and atomic trusted-store installation remain Task 6 work. diff --git a/tasks/todo.md b/tasks/todo.md index 2c3e5159..d006ef86 100644 --- a/tasks/todo.md +++ b/tasks/todo.md @@ -217,7 +217,7 @@ under `internal/service/jailer/`. No handler, transport, CLI, API, core-domain, - [x] Freeze the exact bounded ELF64 header-admission policy for x86_64 and aarch64. - [x] Implement and verify the private ELF header validator: exact source identity and 64-byte header length, closed identity/type/machine/program-header policy, no untrusted allocation, and no executable loading or execution. - - [ ] Bind ELF admission to the bounded actual file size and reject a truncated declared program-header table. + - [x] Bind ELF admission to the bounded actual file size and reject a truncated declared program-header table. - [ ] Exact Firecracker/Jailer bytes and a root manifest install atomically and durably. - [x] Freeze the strict schema-v1 manifest contract, 4 KiB record bound, typed SHA-256 fields, and 120-byte through 64 MiB executable-size policy; the fixed store leaf is `manifest.json`. From 5b2a7ccd8bcdf542dd2181117641032025e0f789 Mon Sep 17 00:00:00 2001 From: AlanD20 Date: Sat, 29 Aug 2026 01:08:25 +0200 Subject: [PATCH 0069/1457] docs: define trusted release store admission --- .../0016-root-owned-single-binary-privileged-dispatch.md | 9 +++++++++ tasks/todo.md | 3 +++ 2 files changed, 12 insertions(+) diff --git a/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md b/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md index dbe3a590..e7ea60ed 100644 --- a/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md +++ b/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md @@ -284,6 +284,15 @@ Decoding rejects unknown, missing, duplicate, case-conflicting, trailing, wrongl values. The executable hashes in a successfully decoded manifest are the sole source of the release identity supplied to instance registration; the unprivileged request never supplies them. +Trusted-store traversal starts at a pinned `/` descriptor and opens every fixed component with `O_NOFOLLOW` relative to +its retained parent. `/`, `/var`, and `/var/lib` must be root-owned directories without group or world write access. +Managed `/var/lib/mvmctl`, `binaries`, architecture, and version directories are exactly `root:root` mode `0700`. +`firecracker` and `jailer` are one-link root-owned regular files of mode `0755`; `manifest.json` is a one-link +root-owned regular file of mode `0600`. A missing architecture or version directory is an ordinary binary-not-found +result. Once the version directory exists, a missing or unsafe fixed leaf is an incomplete/corrupt trusted release and +fails as `binary.untrusted`, never as an absent release. Reads retain the version-directory descriptor, open each leaf +once, compare pre/post-read descriptor identity and metadata, and do not reconstruct a pathname. + Implementation note (2026-08-29): the private Jailer service derives the exact source identity from a validated `(version, architecture)` release slot and rejects non-canonical slots before constructing any source value. Its dedicated checksum authority independently fetches the derived sidecar with the closed transport policy above and diff --git a/tasks/todo.md b/tasks/todo.md index d006ef86..b6426d5d 100644 --- a/tasks/todo.md +++ b/tasks/todo.md @@ -222,6 +222,9 @@ under `internal/service/jailer/`. No handler, transport, CLI, API, core-domain, - [x] Freeze the strict schema-v1 manifest contract, 4 KiB record bound, typed SHA-256 fields, and 120-byte through 64 MiB executable-size policy; the fixed store leaf is `manifest.json`. - [x] Implement and verify the private manifest codec and release-identity derivation. + - [ ] Pin and validate the fixed root/architecture/version directory chain without following links or reopening paths. + - [ ] Read `manifest.json` once through the pinned slot; verify exact metadata, bounded bytes, stable descriptor + identity, strict decoding, and exact slot equality. - [ ] Referenced release removal/replacement holds the release lease and fails closed on unreadable records. - [ ] L1 failure injection proves an old complete pair or no pair, never a partial trusted release. From 84a6624aac93a005e70758b27f3cd9c022cdf5a7 Mon Sep 17 00:00:00 2001 From: AlanD20 Date: Sat, 29 Aug 2026 01:13:55 +0200 Subject: [PATCH 0070/1457] feat: pin trusted release store directories --- .../jailer/trusted_release_store_fs.go | 354 ++++++++++++++++++ .../jailer/trusted_release_store_fs_test.go | 321 ++++++++++++++++ 2 files changed, 675 insertions(+) create mode 100644 internal/service/jailer/trusted_release_store_fs.go create mode 100644 internal/service/jailer/trusted_release_store_fs_test.go diff --git a/internal/service/jailer/trusted_release_store_fs.go b/internal/service/jailer/trusted_release_store_fs.go new file mode 100644 index 00000000..8d505369 --- /dev/null +++ b/internal/service/jailer/trusted_release_store_fs.go @@ -0,0 +1,354 @@ +package jailer + +import ( + "context" + "errors" + "fmt" + "path" + "strings" + + "golang.org/x/sys/unix" + + "mvmctl/internal/infra" + "mvmctl/pkg/errs" +) + +const ( + trustedReleaseStoreDirectoryFlags = unix.O_RDONLY | unix.O_DIRECTORY | unix.O_CLOEXEC | unix.O_NOFOLLOW + trustedReleaseStoreDirectoryMode = uint32(0700) +) + +type trustedReleaseStoreDeps struct { + open func(context.Context, string, int, uint32) (int, error) + openAt func(context.Context, int, string, int, uint32) (int, error) + fstat func(context.Context, int, *unix.Stat_t) error + close func(context.Context, int) error +} + +type trustedReleaseStorePolicy struct { + rootPath string + expectedUID uint32 + expectedGID uint32 +} + +type trustedReleaseStore struct { + deps trustedReleaseStoreDeps + policy trustedReleaseStorePolicy + binariesFD int + retained []int +} + +type trustedReleaseDirectory struct { + deps trustedReleaseStoreDeps + policy trustedReleaseStorePolicy + slot releaseSlot + slotFD int + retained []int +} + +func productionTrustedReleaseStorePolicy() trustedReleaseStorePolicy { + return trustedReleaseStorePolicy{rootPath: "/", expectedUID: 0, expectedGID: 0} +} + +// CRITICAL: Every component is opened once relative to its retained parent. No caller path or reconstructed release +// path participates in trusted-store authorization. +func openTrustedReleaseStoreForRead( + ctx context.Context, + deps trustedReleaseStoreDeps, + policy trustedReleaseStorePolicy, +) (_ *trustedReleaseStore, returnErr error) { + if err := ctx.Err(); err != nil { + return nil, trustedReleaseStoreError("open trusted release store", err) + } + if !path.IsAbs(policy.rootPath) || path.Clean(policy.rootPath) != policy.rootPath { + return nil, trustedReleaseStoreError("trusted release store root policy is not canonical", nil) + } + components, err := trustedReleaseStorePathComponents() + if err != nil { + return nil, err + } + + retained := make([]int, 0, len(components)+1) + defer func() { + if returnErr == nil { + return + } + returnErr = appendTrustedReleaseStoreError( + returnErr, + "close rejected trusted release store", + closeTrustedReleaseStoreFDs(ctx, deps, retained), + ) + }() + + rootFD, err := deps.open(ctx, policy.rootPath, trustedReleaseStoreDirectoryFlags, 0) + if err != nil { + return nil, classifyTrustedReleaseStoreOpenError("filesystem root", err, false) + } + retained = append(retained, rootFD) + if err := verifyTrustedReleaseStoreDirectory(ctx, deps, rootFD, "filesystem root", policy, false); err != nil { + return nil, err + } + + parentFD := rootFD + for index, name := range components { + if err := ctx.Err(); err != nil { + return nil, trustedReleaseStoreError("open trusted release store component", err) + } + fd, openErr := deps.openAt(ctx, parentFD, name, trustedReleaseStoreDirectoryFlags, 0) + if openErr != nil { + return nil, classifyTrustedReleaseStoreOpenError(name, openErr, index >= 2) + } + retained = append(retained, fd) + if err := verifyTrustedReleaseStoreDirectory(ctx, deps, fd, name, policy, index >= 2); err != nil { + return nil, err + } + parentFD = fd + } + + store := &trustedReleaseStore{ + deps: deps, + policy: policy, + binariesFD: parentFD, + retained: retained, + } + retained = nil + return store, nil +} + +func (store *trustedReleaseStore) openInstalledSlot( + ctx context.Context, + slot releaseSlot, +) (_ *trustedReleaseDirectory, returnErr error) { + if store == nil || store.binariesFD < 0 || len(store.retained) == 0 { + return nil, trustedReleaseStoreError("trusted release store is not active", nil) + } + if err := validateReleaseSlotValue(slot); err != nil { + return nil, instanceValidationError(err.Error()) + } + if err := ctx.Err(); err != nil { + return nil, trustedReleaseStoreError("open installed trusted release", err) + } + + retained := make([]int, 0, 2) + defer func() { + if returnErr == nil { + return + } + returnErr = appendTrustedReleaseStoreError( + returnErr, + "close rejected installed trusted release", + closeTrustedReleaseStoreFDs(ctx, store.deps, retained), + ) + }() + + architectureFD, err := store.deps.openAt( + ctx, + store.binariesFD, + slot.architecture, + trustedReleaseStoreDirectoryFlags, + 0, + ) + if err != nil { + return nil, classifyTrustedReleaseStoreOpenError("release architecture", err, true) + } + retained = append(retained, architectureFD) + if err := verifyTrustedReleaseStoreDirectory( + ctx, + store.deps, + architectureFD, + "release architecture", + store.policy, + true, + ); err != nil { + return nil, err + } + + if err := ctx.Err(); err != nil { + return nil, trustedReleaseStoreError("open installed trusted release version", err) + } + slotFD, err := store.deps.openAt( + ctx, + architectureFD, + slot.version, + trustedReleaseStoreDirectoryFlags, + 0, + ) + if err != nil { + return nil, classifyTrustedReleaseStoreOpenError("release version", err, true) + } + retained = append(retained, slotFD) + if err := verifyTrustedReleaseStoreDirectory( + ctx, + store.deps, + slotFD, + "release version", + store.policy, + true, + ); err != nil { + return nil, err + } + + directory := &trustedReleaseDirectory{ + deps: store.deps, + policy: store.policy, + slot: slot, + slotFD: slotFD, + retained: retained, + } + retained = nil + return directory, nil +} + +func trustedReleaseStorePathComponents() ([]string, error) { + if !strings.HasPrefix(infra.TrustedBinaryRoot, "/") || + path.Clean(infra.TrustedBinaryRoot) != infra.TrustedBinaryRoot { + return nil, trustedReleaseStoreError("trusted release store root constant is not canonical", nil) + } + components := strings.Split(strings.TrimPrefix(infra.TrustedBinaryRoot, "/"), "/") + if len(components) != 4 { + return nil, trustedReleaseStoreError("trusted release store root constant has unexpected depth", nil) + } + for _, component := range components { + if component == "" || component == "." || component == ".." { + return nil, trustedReleaseStoreError("trusted release store root constant has an unsafe component", nil) + } + } + return components, nil +} + +func verifyTrustedReleaseStoreDirectory( + ctx context.Context, + deps trustedReleaseStoreDeps, + fd int, + description string, + policy trustedReleaseStorePolicy, + requireManagedMode bool, +) error { + if err := ctx.Err(); err != nil { + return trustedReleaseStoreError("inspect trusted release directory "+description, err) + } + var stat unix.Stat_t + if err := deps.fstat(ctx, fd, &stat); err != nil { + return trustedReleaseStoreError("inspect trusted release directory "+description, err) + } + if stat.Mode&unix.S_IFMT != unix.S_IFDIR { + return trustedReleaseStoreUntrusted("trusted release path "+description+" is not a directory", nil) + } + if stat.Uid != policy.expectedUID { + return trustedReleaseStoreUntrusted("trusted release directory "+description+" has an unsafe owner", nil) + } + if requireManagedMode { + if stat.Gid != policy.expectedGID || stat.Mode&07777 != trustedReleaseStoreDirectoryMode { + return trustedReleaseStoreUntrusted( + "managed trusted release directory "+description+" has unexpected owner or mode", + nil, + ) + } + return nil + } + if stat.Mode&0022 != 0 { + return trustedReleaseStoreUntrusted( + "trusted release directory "+description+" is group or world writable", + nil, + ) + } + return nil +} + +func classifyTrustedReleaseStoreOpenError(description string, cause error, missingIsNotFound bool) error { + if errors.Is(cause, unix.ENOENT) && missingIsNotFound { + return errs.NotFound(errs.CodeBinaryNotFound, "trusted release is not installed") + } + if errors.Is(cause, unix.ELOOP) || errors.Is(cause, unix.ENOTDIR) { + return trustedReleaseStoreUntrusted("trusted release path "+description+" is unsafe", cause) + } + return trustedReleaseStoreError("open trusted release path "+description, cause) +} + +func (store *trustedReleaseStore) Release(ctx context.Context) error { + if store == nil || len(store.retained) == 0 { + return nil + } + err := closeTrustedReleaseStoreFDs(ctx, store.deps, store.retained) + store.retained = nil + store.binariesFD = -1 + if err != nil { + return trustedReleaseStoreError("release trusted release store descriptors", err) + } + return nil +} + +func (directory *trustedReleaseDirectory) Release(ctx context.Context) error { + if directory == nil || len(directory.retained) == 0 { + return nil + } + err := closeTrustedReleaseStoreFDs(ctx, directory.deps, directory.retained) + directory.retained = nil + directory.slotFD = -1 + if err != nil { + return trustedReleaseStoreError("release installed trusted release descriptors", err) + } + return nil +} + +func closeTrustedReleaseStoreFDs(ctx context.Context, deps trustedReleaseStoreDeps, fds []int) error { + cleanupCtx := context.WithoutCancel(ctx) + var result error + for index := len(fds) - 1; index >= 0; index-- { + if err := deps.close(cleanupCtx, fds[index]); err != nil { + result = errors.Join(result, fmt.Errorf("fd %d: %w", fds[index], err)) + } + } + return result +} + +func trustedReleaseStoreError(message string, cause error) *errs.DomainError { + if cause == nil { + return errs.New(errs.CodeBinaryTrustedInstallFailed, message, errs.WithClass(errs.ClassInternal)) + } + return errs.WrapMsg( + errs.CodeBinaryTrustedInstallFailed, + message, + cause, + errs.WithClass(errs.ClassInternal), + ) +} + +func trustedReleaseStoreUntrusted(message string, cause error) *errs.DomainError { + if cause == nil { + return errs.New(errs.CodeBinaryUntrusted, message) + } + return errs.WrapMsg(errs.CodeBinaryUntrusted, message, cause, errs.WithClass(errs.ClassValidation)) +} + +func appendTrustedReleaseStoreError(primary error, message string, cause error) error { + if cause == nil { + return primary + } + if primary == nil { + return trustedReleaseStoreError(message, cause) + } + if domainErr := errs.AsDomainError(primary); domainErr != nil { + domainErr.Message += "; " + message + ": " + cause.Error() + domainErr.Err = errors.Join(domainErr.Err, cause) + return domainErr + } + return trustedReleaseStoreError(message, errors.Join(primary, cause)) +} + +func realTrustedReleaseStoreDeps() trustedReleaseStoreDeps { + return trustedReleaseStoreDeps{ + open: func(_ context.Context, name string, flags int, mode uint32) (int, error) { + return unix.Open(name, flags, mode) + }, + openAt: func(_ context.Context, parentFD int, name string, flags int, mode uint32) (int, error) { + return unix.Openat(parentFD, name, flags, mode) + }, + fstat: func(_ context.Context, fd int, stat *unix.Stat_t) error { + return unix.Fstat(fd, stat) + }, + close: func(_ context.Context, fd int) error { + return unix.Close(fd) + }, + } +} diff --git a/internal/service/jailer/trusted_release_store_fs_test.go b/internal/service/jailer/trusted_release_store_fs_test.go new file mode 100644 index 00000000..25b65ea3 --- /dev/null +++ b/internal/service/jailer/trusted_release_store_fs_test.go @@ -0,0 +1,321 @@ +package jailer + +import ( + "context" + "errors" + "os" + "path/filepath" + "sync" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + "golang.org/x/sys/unix" + + "mvmctl/pkg/errs" +) + +// Rationale: The trusted release path is an authorization boundary. Opening it must pin the exact root-owned directory +// objects once so later release operations cannot be redirected by replacing their pathnames. +func TestOpenTrustedReleaseStorePinsCanonicalSlotDirectories(t *testing.T) { + t.Parallel() + + fixture := newTrustedReleaseStoreFixture(t) + store, err := openTrustedReleaseStoreForRead(t.Context(), fixture.deps, fixture.policy) + require.NoError(t, err) + directory, err := store.openInstalledSlot(t.Context(), fixture.slot) + require.NoError(t, err) + + var before unix.Stat_t + require.NoError(t, unix.Fstat(directory.slotFD, &before)) + fdFlags, err := unix.FcntlInt(uintptr(directory.slotFD), unix.F_GETFD, 0) + require.NoError(t, err) + assert.NotZero(t, fdFlags&unix.FD_CLOEXEC) + movedPath := fixture.slotPath + ".moved" + require.NoError(t, os.Rename(fixture.slotPath, movedPath)) + require.NoError(t, os.Mkdir(fixture.slotPath, 0700)) + var after unix.Stat_t + require.NoError(t, unix.Fstat(directory.slotFD, &after)) + assert.Equal(t, before.Dev, after.Dev) + assert.Equal(t, before.Ino, after.Ino) + assert.NotEqual(t, fileInode(t, fixture.slotPath), after.Ino) + + require.NoError(t, directory.Release(t.Context())) + require.NoError(t, store.Release(t.Context())) +} + +func TestOpenTrustedReleaseStoreRejectsUnexpectedOwnerIdentity(t *testing.T) { + t.Parallel() + + tests := map[string]func(*trustedReleaseStorePolicy){ + "wrong owner UID": func(policy *trustedReleaseStorePolicy) { policy.expectedUID++ }, + "wrong managed GID": func(policy *trustedReleaseStorePolicy) { policy.expectedGID++ }, + } + for name, mutate := range tests { + t.Run(name, func(t *testing.T) { + t.Parallel() + + fixture := newTrustedReleaseStoreFixture(t) + mutate(&fixture.policy) + _, err := openTrustedReleaseStoreForRead(t.Context(), fixture.deps, fixture.policy) + require.Error(t, err) + assert.Equal(t, errs.CodeBinaryUntrusted, errs.AsDomainError(err).Code) + }) + } +} + +// Rationale: O_NOFOLLOW must apply at every level, including the injected test root, or root could traverse an +// attacker-controlled link before reaching otherwise safe-looking managed directories. +func TestOpenTrustedReleaseStoreRejectsSymlinkAtEveryLevel(t *testing.T) { + t.Parallel() + + levels := []string{"root", "var", "lib", "mvmctl", "binaries", "architecture", "version"} + for _, level := range levels { + t.Run(level, func(t *testing.T) { + t.Parallel() + + fixture := newTrustedReleaseStoreFixture(t) + fixture.replaceLevelWithSymlink(t, level) + store, err := openTrustedReleaseStoreForRead(t.Context(), fixture.deps, fixture.policy) + if level == "architecture" || level == "version" { + require.NoError(t, err) + _, err = store.openInstalledSlot(t.Context(), fixture.slot) + require.NoError(t, store.Release(t.Context())) + } + require.Error(t, err) + assert.Equal(t, errs.CodeBinaryUntrusted, errs.AsDomainError(err).Code) + }) + } +} + +func TestOpenTrustedReleaseStoreRejectsUnsafeDirectoryMetadata(t *testing.T) { + t.Parallel() + + tests := map[string]struct { + level string + mode os.FileMode + }{ + "writable ancestor": {level: "var", mode: 0775}, + "permissive managed root": {level: "mvmctl", mode: 0755}, + "wrong binaries mode": {level: "binaries", mode: 0750}, + "wrong architecture mode": {level: "architecture", mode: 0711}, + "wrong version mode": {level: "version", mode: 0755}, + } + for name, tc := range tests { + t.Run(name, func(t *testing.T) { + t.Parallel() + + fixture := newTrustedReleaseStoreFixture(t) + require.NoError(t, os.Chmod(fixture.pathForLevel(tc.level), tc.mode)) + store, err := openTrustedReleaseStoreForRead(t.Context(), fixture.deps, fixture.policy) + if tc.level == "architecture" || tc.level == "version" { + require.NoError(t, err) + _, err = store.openInstalledSlot(t.Context(), fixture.slot) + require.NoError(t, store.Release(t.Context())) + } + require.Error(t, err) + assert.Equal(t, errs.CodeBinaryUntrusted, errs.AsDomainError(err).Code) + }) + } +} + +func TestOpenTrustedReleaseStoreRejectsNonDirectoryComponent(t *testing.T) { + t.Parallel() + + fixture := newTrustedReleaseStoreFixture(t) + require.NoError(t, os.Remove(fixture.slotPath)) + require.NoError(t, os.WriteFile(fixture.slotPath, []byte("not a directory"), 0600)) + + store, err := openTrustedReleaseStoreForRead(t.Context(), fixture.deps, fixture.policy) + require.NoError(t, err) + _, err = store.openInstalledSlot(t.Context(), fixture.slot) + require.Error(t, err) + assert.Equal(t, errs.CodeBinaryUntrusted, errs.AsDomainError(err).Code) + require.NoError(t, store.Release(t.Context())) +} + +func TestOpenTrustedReleaseStoreReportsMissingRelease(t *testing.T) { + t.Parallel() + + tests := []string{"architecture", "version"} + for _, level := range tests { + t.Run(level, func(t *testing.T) { + t.Parallel() + + fixture := newTrustedReleaseStoreFixture(t) + require.NoError(t, os.Remove(fixture.slotPath)) + if level == "architecture" { + require.NoError(t, os.Remove(fixture.architecturePath)) + } + store, err := openTrustedReleaseStoreForRead(t.Context(), fixture.deps, fixture.policy) + require.NoError(t, err) + _, err = store.openInstalledSlot(t.Context(), fixture.slot) + require.Error(t, err) + assert.Equal(t, errs.CodeBinaryNotFound, errs.AsDomainError(err).Code) + require.NoError(t, store.Release(t.Context())) + }) + } +} + +func TestOpenTrustedReleaseStoreRejectsInvalidSlotBeforeFilesystemAccess(t *testing.T) { + t.Parallel() + + fixture := newTrustedReleaseStoreFixture(t) + store, err := openTrustedReleaseStoreForRead(t.Context(), fixture.deps, fixture.policy) + require.NoError(t, err) + _, err = store.openInstalledSlot(t.Context(), releaseSlot{version: "../1.16.1", architecture: "x86_64"}) + require.Error(t, err) + assert.Equal(t, errs.CodeValidationFailed, errs.AsDomainError(err).Code) + require.NoError(t, store.Release(t.Context())) +} + +func TestTrustedReleaseStoreReleaseClosesEachDescriptorOnce(t *testing.T) { + t.Parallel() + + fixture := newTrustedReleaseStoreFixture(t) + var mutex sync.Mutex + closed := make(map[int]int) + realClose := fixture.deps.close + fixture.deps.close = func(ctx context.Context, fd int) error { + mutex.Lock() + closed[fd]++ + mutex.Unlock() + return realClose(ctx, fd) + } + + store, err := openTrustedReleaseStoreForRead(t.Context(), fixture.deps, fixture.policy) + require.NoError(t, err) + directory, err := store.openInstalledSlot(t.Context(), fixture.slot) + require.NoError(t, err) + require.NoError(t, directory.Release(t.Context())) + require.NoError(t, directory.Release(t.Context())) + require.NoError(t, store.Release(t.Context())) + require.NoError(t, store.Release(t.Context())) + + mutex.Lock() + defer mutex.Unlock() + assert.Len(t, closed, 7) + for fd, count := range closed { + assert.Equal(t, 1, count, "descriptor %d close count", fd) + } +} + +func TestOpenTrustedReleaseStoreHonorsCancellation(t *testing.T) { + t.Parallel() + + fixture := newTrustedReleaseStoreFixture(t) + ctx, cancel := context.WithCancel(t.Context()) + cancel() + store, err := openTrustedReleaseStoreForRead(ctx, fixture.deps, fixture.policy) + assert.Nil(t, store) + require.Error(t, err) + assert.ErrorIs(t, err, context.Canceled) + assert.Equal(t, errs.CodeBinaryTrustedInstallFailed, errs.AsDomainError(err).Code) +} + +func TestOpenTrustedReleaseStorePreservesPrimaryErrorWhenCleanupFails(t *testing.T) { + t.Parallel() + + fixture := newTrustedReleaseStoreFixture(t) + require.NoError(t, os.Chmod(fixture.varPath, 0775)) + realClose := fixture.deps.close + fixture.deps.close = func(ctx context.Context, fd int) error { + return errors.Join(realClose(ctx, fd), unix.EIO) + } + + _, err := openTrustedReleaseStoreForRead(t.Context(), fixture.deps, fixture.policy) + require.Error(t, err) + domainErr := errs.AsDomainError(err) + require.NotNil(t, domainErr) + assert.Equal(t, errs.CodeBinaryUntrusted, domainErr.Code) + assert.Contains(t, domainErr.Message, "close rejected trusted release store") + assert.ErrorIs(t, err, unix.EIO) +} + +type trustedReleaseStoreFixture struct { + root string + varPath string + libPath string + mvmctlPath string + binariesPath string + architecturePath string + slotPath string + slot releaseSlot + deps trustedReleaseStoreDeps + policy trustedReleaseStorePolicy +} + +func newTrustedReleaseStoreFixture(t *testing.T) *trustedReleaseStoreFixture { + t.Helper() + + container := t.TempDir() + root := filepath.Join(container, "root") + varPath := filepath.Join(root, "var") + libPath := filepath.Join(varPath, "lib") + mvmctlPath := filepath.Join(libPath, "mvmctl") + binariesPath := filepath.Join(mvmctlPath, "binaries") + architecturePath := filepath.Join(binariesPath, "x86_64") + slotPath := filepath.Join(architecturePath, "1.16.1") + require.NoError(t, os.Mkdir(root, 0700)) + require.NoError(t, os.Mkdir(varPath, 0755)) + require.NoError(t, os.Mkdir(libPath, 0755)) + require.NoError(t, os.Mkdir(mvmctlPath, 0700)) + require.NoError(t, os.Mkdir(binariesPath, 0700)) + require.NoError(t, os.Mkdir(architecturePath, 0700)) + require.NoError(t, os.Mkdir(slotPath, 0700)) + + return &trustedReleaseStoreFixture{ + root: root, + varPath: varPath, + libPath: libPath, + mvmctlPath: mvmctlPath, + binariesPath: binariesPath, + architecturePath: architecturePath, + slotPath: slotPath, + slot: releaseSlot{version: "1.16.1", architecture: "x86_64"}, + deps: realTrustedReleaseStoreDeps(), + policy: trustedReleaseStorePolicy{ + rootPath: root, + expectedUID: uint32(os.Getuid()), + expectedGID: uint32(os.Getgid()), + }, + } +} + +func (fixture *trustedReleaseStoreFixture) pathForLevel(level string) string { + switch level { + case "root": + return fixture.root + case "var": + return fixture.varPath + case "lib": + return fixture.libPath + case "mvmctl": + return fixture.mvmctlPath + case "binaries": + return fixture.binariesPath + case "architecture": + return fixture.architecturePath + case "version": + return fixture.slotPath + default: + panic("unknown trusted release store fixture level: " + level) + } +} + +func (fixture *trustedReleaseStoreFixture) replaceLevelWithSymlink(t *testing.T, level string) { + t.Helper() + + target := fixture.pathForLevel(level) + moved := target + ".real" + require.NoError(t, os.Rename(target, moved)) + require.NoError(t, os.Symlink(moved, target)) +} + +func fileInode(t *testing.T, path string) uint64 { + t.Helper() + + var stat unix.Stat_t + require.NoError(t, unix.Stat(path, &stat)) + return stat.Ino +} From 08618d502572945e0ea70b212aa047391a5590ff Mon Sep 17 00:00:00 2001 From: AlanD20 Date: Sat, 29 Aug 2026 01:18:04 +0200 Subject: [PATCH 0071/1457] feat: read pinned trusted release manifests --- .../jailer/trusted_release_store_fs.go | 4 + .../jailer/trusted_release_store_manifest.go | 169 +++++++++++ .../trusted_release_store_manifest_test.go | 276 ++++++++++++++++++ 3 files changed, 449 insertions(+) create mode 100644 internal/service/jailer/trusted_release_store_manifest.go create mode 100644 internal/service/jailer/trusted_release_store_manifest_test.go diff --git a/internal/service/jailer/trusted_release_store_fs.go b/internal/service/jailer/trusted_release_store_fs.go index 8d505369..7b134d92 100644 --- a/internal/service/jailer/trusted_release_store_fs.go +++ b/internal/service/jailer/trusted_release_store_fs.go @@ -22,6 +22,7 @@ type trustedReleaseStoreDeps struct { open func(context.Context, string, int, uint32) (int, error) openAt func(context.Context, int, string, int, uint32) (int, error) fstat func(context.Context, int, *unix.Stat_t) error + read func(context.Context, int, []byte) (int, error) close func(context.Context, int) error } @@ -347,6 +348,9 @@ func realTrustedReleaseStoreDeps() trustedReleaseStoreDeps { fstat: func(_ context.Context, fd int, stat *unix.Stat_t) error { return unix.Fstat(fd, stat) }, + read: func(_ context.Context, fd int, value []byte) (int, error) { + return unix.Read(fd, value) + }, close: func(_ context.Context, fd int) error { return unix.Close(fd) }, diff --git a/internal/service/jailer/trusted_release_store_manifest.go b/internal/service/jailer/trusted_release_store_manifest.go new file mode 100644 index 00000000..6e02cf5d --- /dev/null +++ b/internal/service/jailer/trusted_release_store_manifest.go @@ -0,0 +1,169 @@ +package jailer + +import ( + "context" + "errors" + "fmt" + "io" + + "golang.org/x/sys/unix" +) + +const ( + trustedReleaseStoreReadFlags = unix.O_RDONLY | unix.O_CLOEXEC | unix.O_NOFOLLOW + trustedReleaseStoreManifestMode = uint32(0600) +) + +func (directory *trustedReleaseDirectory) readManifest( + ctx context.Context, +) (manifest trustedReleaseManifest, returnErr error) { + if directory == nil || directory.slotFD < 0 || len(directory.retained) == 0 { + return trustedReleaseManifest{}, trustedReleaseStoreError( + "installed trusted release directory is not active", + nil, + ) + } + if err := ctx.Err(); err != nil { + return trustedReleaseManifest{}, trustedReleaseStoreError("read trusted release manifest", err) + } + + manifestFD, err := directory.deps.openAt( + ctx, + directory.slotFD, + trustedReleaseManifestLeaf, + trustedReleaseStoreReadFlags, + 0, + ) + if err != nil { + return trustedReleaseManifest{}, classifyTrustedReleaseManifestOpenError(err) + } + defer func() { + if err := directory.deps.close(context.WithoutCancel(ctx), manifestFD); err != nil { + returnErr = appendTrustedReleaseStoreError(returnErr, "close trusted release manifest", err) + } + }() + + before, err := inspectTrustedReleaseManifestDescriptor(ctx, directory, manifestFD) + if err != nil { + return trustedReleaseManifest{}, err + } + raw, err := readTrustedReleaseManifestBytes(ctx, directory.deps, manifestFD) + if err != nil { + return trustedReleaseManifest{}, err + } + after, err := inspectTrustedReleaseManifestDescriptor(ctx, directory, manifestFD) + if err != nil { + return trustedReleaseManifest{}, err + } + if !sameTrustedReleaseManifestStat(before, after) || int64(len(raw)) != after.Size { + return trustedReleaseManifest{}, trustedReleaseStoreUntrusted( + "trusted release manifest changed while being read", + nil, + ) + } + + manifest, err = decodeTrustedReleaseManifest(raw) + if err != nil { + return trustedReleaseManifest{}, err + } + if manifest.slot != directory.slot { + return trustedReleaseManifest{}, trustedReleaseStoreUntrusted( + "trusted release manifest slot does not match its directory", + nil, + ) + } + return manifest, nil +} + +func inspectTrustedReleaseManifestDescriptor( + ctx context.Context, + directory *trustedReleaseDirectory, + fd int, +) (unix.Stat_t, error) { + if err := ctx.Err(); err != nil { + return unix.Stat_t{}, trustedReleaseStoreError("inspect trusted release manifest", err) + } + var stat unix.Stat_t + if err := directory.deps.fstat(ctx, fd, &stat); err != nil { + return unix.Stat_t{}, trustedReleaseStoreError("inspect trusted release manifest", err) + } + if stat.Mode&unix.S_IFMT != unix.S_IFREG { + return unix.Stat_t{}, trustedReleaseStoreUntrusted( + "trusted release manifest is not a regular file", + nil, + ) + } + if stat.Uid != directory.policy.expectedUID || stat.Gid != directory.policy.expectedGID || + stat.Mode&07777 != trustedReleaseStoreManifestMode { + return unix.Stat_t{}, trustedReleaseStoreUntrusted( + "trusted release manifest has unexpected owner or mode", + nil, + ) + } + if stat.Nlink != 1 { + return unix.Stat_t{}, trustedReleaseStoreUntrusted( + "trusted release manifest must have exactly one link", + nil, + ) + } + if stat.Size <= 0 || stat.Size > maxTrustedReleaseManifestBytes { + return unix.Stat_t{}, trustedReleaseStoreUntrusted( + "trusted release manifest size is outside the admitted range", + nil, + ) + } + return stat, nil +} + +func readTrustedReleaseManifestBytes( + ctx context.Context, + deps trustedReleaseStoreDeps, + fd int, +) ([]byte, error) { + buffer := make([]byte, maxTrustedReleaseManifestBytes+1) + total := 0 + for { + if err := ctx.Err(); err != nil { + return nil, trustedReleaseStoreError("read trusted release manifest", err) + } + count, readErr := deps.read(ctx, fd, buffer[total:]) + if count < 0 || count > len(buffer)-total { + return nil, trustedReleaseStoreError( + "read trusted release manifest", + fmt.Errorf("invalid read count %d", count), + ) + } + total += count + if total > maxTrustedReleaseManifestBytes { + return nil, trustedReleaseStoreUntrusted("trusted release manifest exceeds size limit", nil) + } + if readErr != nil { + if errors.Is(readErr, io.EOF) { + return buffer[:total], nil + } + return nil, trustedReleaseStoreError("read trusted release manifest", readErr) + } + if count == 0 { + return buffer[:total], nil + } + } +} + +func sameTrustedReleaseManifestStat(first, second unix.Stat_t) bool { + return first.Dev == second.Dev && + first.Ino == second.Ino && + first.Mode == second.Mode && + first.Nlink == second.Nlink && + first.Uid == second.Uid && + first.Gid == second.Gid && + first.Size == second.Size && + first.Mtim == second.Mtim && + first.Ctim == second.Ctim +} + +func classifyTrustedReleaseManifestOpenError(cause error) error { + if errors.Is(cause, unix.ENOENT) || errors.Is(cause, unix.ELOOP) || errors.Is(cause, unix.ENOTDIR) { + return trustedReleaseStoreUntrusted("trusted release manifest is missing or unsafe", cause) + } + return trustedReleaseStoreError("open trusted release manifest", cause) +} diff --git a/internal/service/jailer/trusted_release_store_manifest_test.go b/internal/service/jailer/trusted_release_store_manifest_test.go new file mode 100644 index 00000000..40d410fa --- /dev/null +++ b/internal/service/jailer/trusted_release_store_manifest_test.go @@ -0,0 +1,276 @@ +package jailer + +import ( + "bytes" + "context" + "errors" + "os" + "path/filepath" + "testing" + + "github.com/google/go-cmp/cmp" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + "golang.org/x/sys/unix" + + "mvmctl/pkg/errs" +) + +// Rationale: The slot descriptor, not its replaceable pathname, selects the root-owned manifest. The decoded manifest +// must exactly match the slot and preserve every field needed to derive executable authority. +func TestTrustedReleaseDirectoryReadsManifestFromPinnedSlot(t *testing.T) { + t.Parallel() + + fixture := newTrustedReleaseStoreFixture(t) + want := testTrustedReleaseManifest(fixture.slot) + writeTrustedReleaseManifestFile(t, fixture.slotPath, want) + store, err := openTrustedReleaseStoreForRead(t.Context(), fixture.deps, fixture.policy) + require.NoError(t, err) + directory, err := store.openInstalledSlot(t.Context(), fixture.slot) + require.NoError(t, err) + + movedPath := fixture.slotPath + ".moved" + require.NoError(t, os.Rename(fixture.slotPath, movedPath)) + require.NoError(t, os.Mkdir(fixture.slotPath, 0700)) + replacement := testTrustedReleaseManifest(fixture.slot) + replacement.firecracker.sizeBytes++ + writeTrustedReleaseManifestFile(t, fixture.slotPath, replacement) + + got, err := directory.readManifest(t.Context()) + require.NoError(t, err) + if diff := cmp.Diff( + want, + got, + cmp.AllowUnexported(trustedReleaseManifest{}, releaseSlot{}, trustedReleaseExecutable{}), + ); diff != "" { + t.Errorf("pinned trusted release manifest mismatch (-want +got):\n%s", diff) + } + require.NoError(t, directory.Release(t.Context())) + require.NoError(t, store.Release(t.Context())) +} + +func TestTrustedReleaseDirectoryRejectsUnsafeOrCorruptManifest(t *testing.T) { + t.Parallel() + + tests := map[string]func(*testing.T, *trustedReleaseStoreFixture){ + "missing": func(_ *testing.T, _ *trustedReleaseStoreFixture) {}, + "symlink": func(t *testing.T, fixture *trustedReleaseStoreFixture) { + target := filepath.Join(filepath.Dir(fixture.slotPath), "attacker-manifest.json") + writeTrustedReleaseManifestFile(t, filepath.Dir(target), testTrustedReleaseManifest(fixture.slot)) + require.NoError(t, os.Rename(filepath.Join(filepath.Dir(target), trustedReleaseManifestLeaf), target)) + require.NoError(t, os.Symlink(target, filepath.Join(fixture.slotPath, trustedReleaseManifestLeaf))) + }, + "directory": func(t *testing.T, fixture *trustedReleaseStoreFixture) { + require.NoError(t, os.Mkdir(filepath.Join(fixture.slotPath, trustedReleaseManifestLeaf), 0600)) + }, + "wrong mode": func(t *testing.T, fixture *trustedReleaseStoreFixture) { + writeTrustedReleaseManifestFile(t, fixture.slotPath, testTrustedReleaseManifest(fixture.slot)) + require.NoError(t, os.Chmod(filepath.Join(fixture.slotPath, trustedReleaseManifestLeaf), 0644)) + }, + "multiple links": func(t *testing.T, fixture *trustedReleaseStoreFixture) { + writeTrustedReleaseManifestFile(t, fixture.slotPath, testTrustedReleaseManifest(fixture.slot)) + manifestPath := filepath.Join(fixture.slotPath, trustedReleaseManifestLeaf) + require.NoError(t, os.Link(manifestPath, manifestPath+".link")) + }, + "empty": func(t *testing.T, fixture *trustedReleaseStoreFixture) { + require.NoError(t, os.WriteFile(filepath.Join(fixture.slotPath, trustedReleaseManifestLeaf), nil, 0600)) + }, + "oversized": func(t *testing.T, fixture *trustedReleaseStoreFixture) { + require.NoError(t, os.WriteFile( + filepath.Join(fixture.slotPath, trustedReleaseManifestLeaf), + bytes.Repeat([]byte{' '}, maxTrustedReleaseManifestBytes+1), + 0600, + )) + }, + "malformed JSON": func(t *testing.T, fixture *trustedReleaseStoreFixture) { + require.NoError(t, os.WriteFile( + filepath.Join(fixture.slotPath, trustedReleaseManifestLeaf), + []byte(`{"schema_version":`), + 0600, + )) + }, + "slot mismatch": func(t *testing.T, fixture *trustedReleaseStoreFixture) { + other := testTrustedReleaseManifest(releaseSlot{version: "1.16.0", architecture: "x86_64"}) + writeTrustedReleaseManifestFile(t, fixture.slotPath, other) + }, + } + + for name, setup := range tests { + t.Run(name, func(t *testing.T) { + t.Parallel() + + fixture := newTrustedReleaseStoreFixture(t) + setup(t, fixture) + store, err := openTrustedReleaseStoreForRead(t.Context(), fixture.deps, fixture.policy) + require.NoError(t, err) + directory, err := store.openInstalledSlot(t.Context(), fixture.slot) + require.NoError(t, err) + _, err = directory.readManifest(t.Context()) + require.Error(t, err) + assert.Equal(t, errs.CodeBinaryUntrusted, errs.AsDomainError(err).Code) + require.NoError(t, directory.Release(t.Context())) + require.NoError(t, store.Release(t.Context())) + }) + } +} + +func TestTrustedReleaseDirectoryRejectsManifestOwnerIdentityMismatch(t *testing.T) { + t.Parallel() + + fixture := newTrustedReleaseStoreFixture(t) + manifest := testTrustedReleaseManifest(fixture.slot) + writeTrustedReleaseManifestFile(t, fixture.slotPath, manifest) + store, err := openTrustedReleaseStoreForRead(t.Context(), fixture.deps, fixture.policy) + require.NoError(t, err) + directory, err := store.openInstalledSlot(t.Context(), fixture.slot) + require.NoError(t, err) + directory.policy.expectedGID++ + + _, err = directory.readManifest(t.Context()) + require.Error(t, err) + assert.Equal(t, errs.CodeBinaryUntrusted, errs.AsDomainError(err).Code) + require.NoError(t, directory.Release(t.Context())) + require.NoError(t, store.Release(t.Context())) +} + +func TestTrustedReleaseDirectoryReadsPartialManifestChunks(t *testing.T) { + t.Parallel() + + fixture := newTrustedReleaseStoreFixture(t) + want := testTrustedReleaseManifest(fixture.slot) + writeTrustedReleaseManifestFile(t, fixture.slotPath, want) + realRead := fixture.deps.read + fixture.deps.read = func(ctx context.Context, fd int, value []byte) (int, error) { + if len(value) > 7 { + value = value[:7] + } + return realRead(ctx, fd, value) + } + store, err := openTrustedReleaseStoreForRead(t.Context(), fixture.deps, fixture.policy) + require.NoError(t, err) + directory, err := store.openInstalledSlot(t.Context(), fixture.slot) + require.NoError(t, err) + + got, err := directory.readManifest(t.Context()) + require.NoError(t, err) + if diff := cmp.Diff( + want, + got, + cmp.AllowUnexported(trustedReleaseManifest{}, releaseSlot{}, trustedReleaseExecutable{}), + ); diff != "" { + t.Errorf("partial-read trusted release manifest mismatch (-want +got):\n%s", diff) + } + require.NoError(t, directory.Release(t.Context())) + require.NoError(t, store.Release(t.Context())) +} + +func TestTrustedReleaseDirectoryRejectsManifestMutationDuringRead(t *testing.T) { + t.Parallel() + + fixture := newTrustedReleaseStoreFixture(t) + writeTrustedReleaseManifestFile(t, fixture.slotPath, testTrustedReleaseManifest(fixture.slot)) + manifestPath := filepath.Join(fixture.slotPath, trustedReleaseManifestLeaf) + realRead := fixture.deps.read + mutated := false + fixture.deps.read = func(ctx context.Context, fd int, value []byte) (int, error) { + count, err := realRead(ctx, fd, value) + if !mutated { + mutated = true + require.NoError(t, os.Chmod(manifestPath, 0644)) + } + return count, err + } + store, err := openTrustedReleaseStoreForRead(t.Context(), fixture.deps, fixture.policy) + require.NoError(t, err) + directory, err := store.openInstalledSlot(t.Context(), fixture.slot) + require.NoError(t, err) + + _, err = directory.readManifest(t.Context()) + require.Error(t, err) + assert.Equal(t, errs.CodeBinaryUntrusted, errs.AsDomainError(err).Code) + require.NoError(t, directory.Release(t.Context())) + require.NoError(t, store.Release(t.Context())) +} + +func TestTrustedReleaseDirectoryManifestReadHonorsCancellation(t *testing.T) { + t.Parallel() + + fixture := newTrustedReleaseStoreFixture(t) + writeTrustedReleaseManifestFile(t, fixture.slotPath, testTrustedReleaseManifest(fixture.slot)) + ctx, cancel := context.WithCancel(t.Context()) + realRead := fixture.deps.read + fixture.deps.read = func(ctx context.Context, fd int, value []byte) (int, error) { + count, err := realRead(ctx, fd, value) + cancel() + return count, err + } + store, err := openTrustedReleaseStoreForRead(ctx, fixture.deps, fixture.policy) + require.NoError(t, err) + directory, err := store.openInstalledSlot(ctx, fixture.slot) + require.NoError(t, err) + + _, err = directory.readManifest(ctx) + require.Error(t, err) + assert.ErrorIs(t, err, context.Canceled) + assert.Equal(t, errs.CodeBinaryTrustedInstallFailed, errs.AsDomainError(err).Code) + require.NoError(t, directory.Release(ctx)) + require.NoError(t, store.Release(ctx)) +} + +func TestTrustedReleaseDirectoryManifestReadPreservesPrimaryErrorWhenCloseFails(t *testing.T) { + t.Parallel() + + fixture := newTrustedReleaseStoreFixture(t) + require.NoError(t, os.WriteFile( + filepath.Join(fixture.slotPath, trustedReleaseManifestLeaf), + []byte(`{"invalid":true}`), + 0600, + )) + store, err := openTrustedReleaseStoreForRead(t.Context(), fixture.deps, fixture.policy) + require.NoError(t, err) + directory, err := store.openInstalledSlot(t.Context(), fixture.slot) + require.NoError(t, err) + realClose := directory.deps.close + directory.deps.close = func(ctx context.Context, fd int) error { + return errors.Join(realClose(ctx, fd), unix.EIO) + } + + _, err = directory.readManifest(t.Context()) + require.Error(t, err) + domainErr := errs.AsDomainError(err) + require.NotNil(t, domainErr) + assert.Equal(t, errs.CodeBinaryUntrusted, domainErr.Code) + assert.Contains(t, domainErr.Message, "close trusted release manifest") + assert.ErrorIs(t, err, unix.EIO) + directory.deps.close = realClose + require.NoError(t, directory.Release(t.Context())) + require.NoError(t, store.Release(t.Context())) +} + +func TestTrustedReleaseDirectoryRejectsInvalidManifestReadCount(t *testing.T) { + t.Parallel() + + fixture := newTrustedReleaseStoreFixture(t) + writeTrustedReleaseManifestFile(t, fixture.slotPath, testTrustedReleaseManifest(fixture.slot)) + store, err := openTrustedReleaseStoreForRead(t.Context(), fixture.deps, fixture.policy) + require.NoError(t, err) + directory, err := store.openInstalledSlot(t.Context(), fixture.slot) + require.NoError(t, err) + directory.deps.read = func(_ context.Context, _ int, value []byte) (int, error) { + return len(value) + 1, nil + } + + _, err = directory.readManifest(t.Context()) + require.Error(t, err) + assert.Equal(t, errs.CodeBinaryTrustedInstallFailed, errs.AsDomainError(err).Code) + require.NoError(t, directory.Release(t.Context())) + require.NoError(t, store.Release(t.Context())) +} + +func writeTrustedReleaseManifestFile(t *testing.T, directory string, manifest trustedReleaseManifest) { + t.Helper() + + raw, err := encodeTrustedReleaseManifest(manifest) + require.NoError(t, err) + require.NoError(t, os.WriteFile(filepath.Join(directory, trustedReleaseManifestLeaf), raw, 0600)) +} From b92129bd5a9b61313be2168ddeb92febbfaaf0a8 Mon Sep 17 00:00:00 2001 From: AlanD20 Date: Sat, 29 Aug 2026 01:19:11 +0200 Subject: [PATCH 0072/1457] docs: record pinned trusted release reads --- .../0016-root-owned-single-binary-privileged-dispatch.md | 8 +++++--- tasks/todo.md | 4 ++-- 2 files changed, 7 insertions(+), 5 deletions(-) diff --git a/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md b/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md index e7ea60ed..c350b665 100644 --- a/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md +++ b/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md @@ -301,9 +301,11 @@ and response cleanup preserves the primary `DomainError` metadata. The private E bounded header shape, selected architecture, exact bounded file size, and complete program-header-table extent without loading or executing candidate bytes. The private manifest codec enforces the closed schema, record and executable bounds, typed digests, and manifest-derived instance release identity. -These foundations are not yet wired to the privileged request or legacy installer. Bounded archive transport and -extraction, the complete member allowlist, manifest persistence, and atomic trusted-store installation remain Task 6 -work. +The private read-side store foundation pins and validates the fixed root/architecture/version directory chain, then +reads `manifest.json` once through the retained slot descriptor with stable metadata and exact slot verification. These +foundations are not yet wired to the privileged request or legacy installer. Bounded archive transport and extraction, +the complete member allowlist, manifest writing, executable verification/pinning, and atomic trusted-store installation +remain Task 6 work. ### Paths, process identity, and runtime state diff --git a/tasks/todo.md b/tasks/todo.md index b6426d5d..8576ea6b 100644 --- a/tasks/todo.md +++ b/tasks/todo.md @@ -222,8 +222,8 @@ under `internal/service/jailer/`. No handler, transport, CLI, API, core-domain, - [x] Freeze the strict schema-v1 manifest contract, 4 KiB record bound, typed SHA-256 fields, and 120-byte through 64 MiB executable-size policy; the fixed store leaf is `manifest.json`. - [x] Implement and verify the private manifest codec and release-identity derivation. - - [ ] Pin and validate the fixed root/architecture/version directory chain without following links or reopening paths. - - [ ] Read `manifest.json` once through the pinned slot; verify exact metadata, bounded bytes, stable descriptor + - [x] Pin and validate the fixed root/architecture/version directory chain without following links or reopening paths. + - [x] Read `manifest.json` once through the pinned slot; verify exact metadata, bounded bytes, stable descriptor identity, strict decoding, and exact slot equality. - [ ] Referenced release removal/replacement holds the release lease and fails closed on unreadable records. - [ ] L1 failure injection proves an old complete pair or no pair, never a partial trusted release. From ca48940c2f58081528b8f12744f83b506e702129 Mon Sep 17 00:00:00 2001 From: AlanD20 Date: Sat, 29 Aug 2026 01:22:46 +0200 Subject: [PATCH 0073/1457] docs: define trusted release executable admission --- ...16-root-owned-single-binary-privileged-dispatch.md | 11 +++++++++++ tasks/todo.md | 3 +++ 2 files changed, 14 insertions(+) diff --git a/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md b/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md index c350b665..46f1662d 100644 --- a/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md +++ b/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md @@ -293,6 +293,17 @@ result. Once the version directory exists, a missing or unsafe fixed leaf is an fails as `binary.untrusted`, never as an absent release. Reads retain the version-directory descriptor, open each leaf once, compare pre/post-read descriptor identity and metadata, and do not reconstruct a pathname. +Executable admission opens the fixed `firecracker` and `jailer` leaves with `O_NOFOLLOW` relative to that retained +version-directory descriptor. It requires each descriptor to identify a one-link `root:root` regular file of exact mode +`0755`, with a size equal to its manifest entry and within the hard executable bound. Verification uses bounded +positioned reads from offset zero with a fixed 32 KiB buffer: it hashes exactly the declared bytes, captures the first +64 bytes for the closed ELF policy, and probes the next byte so truncation and growth fail closed. Positioned reads +leave the descriptor offset unchanged. A second descriptor stat must match device, inode, type and mode, link count, +owner and group, size, modification time, and change time from the first stat. The service retains those same verified +descriptors for the later launch handoff; it never validates one pathname and executes a reopened pathname. Failure to +admit `jailer` closes an already admitted `firecracker`, and cleanup preserves the primary domain error while reporting +any close failure. + Implementation note (2026-08-29): the private Jailer service derives the exact source identity from a validated `(version, architecture)` release slot and rejects non-canonical slots before constructing any source value. Its dedicated checksum authority independently fetches the derived sidecar with the closed transport policy above and diff --git a/tasks/todo.md b/tasks/todo.md index 8576ea6b..f4da6187 100644 --- a/tasks/todo.md +++ b/tasks/todo.md @@ -225,6 +225,9 @@ under `internal/service/jailer/`. No handler, transport, CLI, API, core-domain, - [x] Pin and validate the fixed root/architecture/version directory chain without following links or reopening paths. - [x] Read `manifest.json` once through the pinned slot; verify exact metadata, bounded bytes, stable descriptor identity, strict decoding, and exact slot equality. + - [ ] Pin and verify the fixed `firecracker` and `jailer` leaves through the retained slot descriptor: exact metadata + and manifest size, bounded positioned full-file hashing, closed ELF admission, pre/post-read stability, unchanged + offsets, exact descriptor retention, cancellation, and checked reverse cleanup. - [ ] Referenced release removal/replacement holds the release lease and fails closed on unreadable records. - [ ] L1 failure injection proves an old complete pair or no pair, never a partial trusted release. From 9ba01cabfb72e259eeb2adf16c7d3fdd2703905c Mon Sep 17 00:00:00 2001 From: AlanD20 Date: Sat, 29 Aug 2026 01:32:13 +0200 Subject: [PATCH 0074/1457] feat: pin trusted release executables --- .../trusted_release_store_executables.go | 315 ++++++++++++ .../trusted_release_store_executables_test.go | 457 ++++++++++++++++++ .../jailer/trusted_release_store_fs.go | 4 + .../jailer/trusted_release_store_manifest.go | 2 +- .../trusted_release_store_manifest_test.go | 3 + 5 files changed, 780 insertions(+), 1 deletion(-) create mode 100644 internal/service/jailer/trusted_release_store_executables.go create mode 100644 internal/service/jailer/trusted_release_store_executables_test.go diff --git a/internal/service/jailer/trusted_release_store_executables.go b/internal/service/jailer/trusted_release_store_executables.go new file mode 100644 index 00000000..6ba67735 --- /dev/null +++ b/internal/service/jailer/trusted_release_store_executables.go @@ -0,0 +1,315 @@ +package jailer + +import ( + "context" + "crypto/sha256" + "errors" + "fmt" + "io" + + "golang.org/x/sys/unix" +) + +const ( + trustedReleaseStoreExecutableMode = uint32(0755) + trustedReleaseExecutableReadBufferSize = 32 * 1024 +) + +type trustedReleaseExecutables struct { + deps trustedReleaseStoreDeps + firecrackerFD int + jailerFD int + retained []int +} + +type trustedReleaseExecutablePolicy struct { + name string + leaf string + expected trustedReleaseExecutable +} + +// CRITICAL: Both fixed leaves are opened once through the pinned slot directory. The same descriptors that pass +// metadata, full-content hash, and ELF admission remain retained for the later launch boundary. +func (directory *trustedReleaseDirectory) openExecutables( + ctx context.Context, + manifest trustedReleaseManifest, +) (_ *trustedReleaseExecutables, returnErr error) { + if directory == nil || directory.slotFD < 0 || len(directory.retained) == 0 { + return nil, trustedReleaseStoreError("installed trusted release directory is not active", nil) + } + if err := validateTrustedReleaseManifest(manifest); err != nil { + return nil, err + } + if manifest.slot != directory.slot { + return nil, trustedReleaseStoreUntrusted( + "trusted release executable manifest slot does not match its directory", + nil, + ) + } + if err := ctx.Err(); err != nil { + return nil, trustedReleaseStoreError("open trusted release executables", err) + } + source, err := newTrustedReleaseSource(directory.slot) + if err != nil { + return nil, trustedReleaseStoreUntrusted("trusted release executable source is invalid", err) + } + + retained := make([]int, 0, 2) + defer func() { + if returnErr == nil { + return + } + returnErr = appendTrustedReleaseStoreError( + returnErr, + "close rejected trusted release executables", + closeTrustedReleaseStoreFDs(ctx, directory.deps, retained), + ) + }() + + policies := [...]trustedReleaseExecutablePolicy{ + {name: "Firecracker", leaf: trustedReleaseFirecrackerLeaf, expected: manifest.firecracker}, + {name: "Jailer", leaf: trustedReleaseJailerLeaf, expected: manifest.jailer}, + } + for _, policy := range policies { + fd, openErr := openTrustedReleaseExecutable(ctx, directory, policy, source) + if openErr != nil { + return nil, openErr + } + retained = append(retained, fd) + } + + executables := &trustedReleaseExecutables{ + deps: directory.deps, + firecrackerFD: retained[0], + jailerFD: retained[1], + retained: retained, + } + retained = nil + return executables, nil +} + +func openTrustedReleaseExecutable( + ctx context.Context, + directory *trustedReleaseDirectory, + policy trustedReleaseExecutablePolicy, + source trustedReleaseSource, +) (_ int, returnErr error) { + if err := ctx.Err(); err != nil { + return -1, trustedReleaseStoreError("open trusted release "+policy.name+" executable", err) + } + openedFD, err := directory.deps.openAt(ctx, directory.slotFD, policy.leaf, trustedReleaseStoreReadFlags, 0) + if err != nil { + return -1, classifyTrustedReleaseExecutableOpenError(policy.name, err) + } + defer func() { + if returnErr == nil { + return + } + returnErr = appendTrustedReleaseStoreError( + returnErr, + "close rejected trusted release "+policy.name+" executable", + directory.deps.close(context.WithoutCancel(ctx), openedFD), + ) + }() + + before, err := inspectTrustedReleaseExecutableDescriptor(ctx, directory, openedFD, policy) + if err != nil { + return -1, err + } + header, digest, err := hashTrustedReleaseExecutable(ctx, directory.deps, openedFD, policy) + if err != nil { + return -1, err + } + after, err := inspectTrustedReleaseExecutableDescriptor(ctx, directory, openedFD, policy) + if err != nil { + return -1, err + } + if !sameTrustedReleaseExecutableStat(before, after) { + return -1, trustedReleaseStoreUntrusted( + "trusted release "+policy.name+" executable changed while being read", + nil, + ) + } + if digest != policy.expected.digest { + return -1, trustedReleaseStoreUntrusted( + "trusted release "+policy.name+" executable digest does not match its manifest", + nil, + ) + } + if err := validateTrustedReleaseELFHeader(header[:], policy.expected.sizeBytes, source); err != nil { + return -1, err + } + return openedFD, nil +} + +func inspectTrustedReleaseExecutableDescriptor( + ctx context.Context, + directory *trustedReleaseDirectory, + fd int, + policy trustedReleaseExecutablePolicy, +) (unix.Stat_t, error) { + if err := ctx.Err(); err != nil { + return unix.Stat_t{}, trustedReleaseStoreError( + "inspect trusted release "+policy.name+" executable", + err, + ) + } + var stat unix.Stat_t + if err := directory.deps.fstat(ctx, fd, &stat); err != nil { + return unix.Stat_t{}, trustedReleaseStoreError( + "inspect trusted release "+policy.name+" executable", + err, + ) + } + if stat.Mode&unix.S_IFMT != unix.S_IFREG { + return unix.Stat_t{}, trustedReleaseStoreUntrusted( + "trusted release "+policy.name+" executable is not a regular file", + nil, + ) + } + if stat.Uid != directory.policy.expectedUID || stat.Gid != directory.policy.expectedGID || + stat.Mode&07777 != trustedReleaseStoreExecutableMode { + return unix.Stat_t{}, trustedReleaseStoreUntrusted( + "trusted release "+policy.name+" executable has unexpected owner or mode", + nil, + ) + } + if stat.Nlink != 1 { + return unix.Stat_t{}, trustedReleaseStoreUntrusted( + "trusted release "+policy.name+" executable must have exactly one link", + nil, + ) + } + if stat.Size < int64(trustedReleaseExecutableMinBytes) || + stat.Size > int64(trustedReleaseExecutableMaxBytes) || uint64(stat.Size) != policy.expected.sizeBytes { + return unix.Stat_t{}, trustedReleaseStoreUntrusted( + "trusted release "+policy.name+" executable size does not match its manifest", + nil, + ) + } + return stat, nil +} + +func hashTrustedReleaseExecutable( + ctx context.Context, + deps trustedReleaseStoreDeps, + fd int, + policy trustedReleaseExecutablePolicy, +) ([trustedReleaseELFHeaderBytes]byte, trustedReleaseExecutableDigest, error) { + var header [trustedReleaseELFHeaderBytes]byte + hasher := sha256.New() + buffer := make([]byte, trustedReleaseExecutableReadBufferSize) + var offset uint64 + for offset < policy.expected.sizeBytes { + if err := ctx.Err(); err != nil { + return header, trustedReleaseExecutableDigest{}, trustedReleaseStoreError( + "read trusted release "+policy.name+" executable", + err, + ) + } + remaining := policy.expected.sizeBytes - offset + readSize := len(buffer) + if remaining < uint64(readSize) { + readSize = int(remaining) + } + count, readErr := deps.pread(ctx, fd, buffer[:readSize], int64(offset)) + if count < 0 || count > readSize { + return header, trustedReleaseExecutableDigest{}, trustedReleaseStoreError( + "read trusted release "+policy.name+" executable", + fmt.Errorf("invalid positioned read count %d", count), + ) + } + if count > 0 { + if _, err := hasher.Write(buffer[:count]); err != nil { + return header, trustedReleaseExecutableDigest{}, trustedReleaseStoreError( + "hash trusted release "+policy.name+" executable", + err, + ) + } + if offset < trustedReleaseELFHeaderBytes { + copy(header[offset:], buffer[:count]) + } + offset += uint64(count) + } + if readErr != nil && !errors.Is(readErr, io.EOF) { + return header, trustedReleaseExecutableDigest{}, trustedReleaseStoreError( + "read trusted release "+policy.name+" executable", + readErr, + ) + } + if offset < policy.expected.sizeBytes && (count == 0 || errors.Is(readErr, io.EOF)) { + return header, trustedReleaseExecutableDigest{}, trustedReleaseStoreUntrusted( + "trusted release "+policy.name+" executable is shorter than its manifest", + nil, + ) + } + } + + if err := ctx.Err(); err != nil { + return header, trustedReleaseExecutableDigest{}, trustedReleaseStoreError( + "probe trusted release "+policy.name+" executable length", + err, + ) + } + var probe [1]byte + count, readErr := deps.pread(ctx, fd, probe[:], int64(policy.expected.sizeBytes)) + if count < 0 || count > len(probe) { + return header, trustedReleaseExecutableDigest{}, trustedReleaseStoreError( + "probe trusted release "+policy.name+" executable length", + fmt.Errorf("invalid positioned read count %d", count), + ) + } + if readErr != nil && !errors.Is(readErr, io.EOF) { + return header, trustedReleaseExecutableDigest{}, trustedReleaseStoreError( + "probe trusted release "+policy.name+" executable length", + readErr, + ) + } + if count != 0 { + return header, trustedReleaseExecutableDigest{}, trustedReleaseStoreUntrusted( + "trusted release "+policy.name+" executable is longer than its manifest", + nil, + ) + } + + var digest trustedReleaseExecutableDigest + copy(digest[:], hasher.Sum(nil)) + return header, digest, nil +} + +func sameTrustedReleaseExecutableStat(first, second unix.Stat_t) bool { + return first.Dev == second.Dev && + first.Ino == second.Ino && + first.Mode == second.Mode && + first.Nlink == second.Nlink && + first.Uid == second.Uid && + first.Gid == second.Gid && + first.Size == second.Size && + first.Mtim == second.Mtim && + first.Ctim == second.Ctim +} + +func classifyTrustedReleaseExecutableOpenError(name string, cause error) error { + if errors.Is(cause, unix.ENOENT) || errors.Is(cause, unix.ELOOP) || errors.Is(cause, unix.ENOTDIR) || + errors.Is(cause, unix.ENXIO) { + return trustedReleaseStoreUntrusted( + "trusted release "+name+" executable is missing or unsafe", + cause, + ) + } + return trustedReleaseStoreError("open trusted release "+name+" executable", cause) +} + +func (executables *trustedReleaseExecutables) Release(ctx context.Context) error { + if executables == nil || len(executables.retained) == 0 { + return nil + } + err := closeTrustedReleaseStoreFDs(ctx, executables.deps, executables.retained) + executables.retained = nil + executables.firecrackerFD = -1 + executables.jailerFD = -1 + if err != nil { + return trustedReleaseStoreError("release trusted release executable descriptors", err) + } + return nil +} diff --git a/internal/service/jailer/trusted_release_store_executables_test.go b/internal/service/jailer/trusted_release_store_executables_test.go new file mode 100644 index 00000000..a8fd6ce7 --- /dev/null +++ b/internal/service/jailer/trusted_release_store_executables_test.go @@ -0,0 +1,457 @@ +package jailer + +import ( + "context" + "crypto/sha256" + "errors" + "os" + "path/filepath" + "testing" + + "github.com/google/go-cmp/cmp" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + "golang.org/x/sys/unix" + + "mvmctl/pkg/errs" +) + +const trustedReleaseTestExecutableBytes = 64*1024 + 17 + +// Rationale: launch must use the exact objects whose metadata, complete bytes, and ELF headers were admitted. Path +// replacement after verification must not redirect either retained descriptor, and positioned reads must not alter the +// offsets later inherited by the launch boundary. +func TestTrustedReleaseDirectoryPinsVerifiedExecutables(t *testing.T) { + t.Parallel() + + fixture := newTrustedReleaseExecutableFixture(t) + directory := openTrustedReleaseTestDirectory(t, fixture.store) + executables, err := directory.openExecutables(t.Context(), fixture.manifest) + require.NoError(t, err) + retainedFDs := []int{executables.firecrackerFD, executables.jailerFD} + + before := executableDescriptorInodes(t, executables) + for _, fd := range retainedFDs { + flags, flagErr := unix.FcntlInt(uintptr(fd), unix.F_GETFD, 0) + require.NoError(t, flagErr) + assert.NotZero(t, flags&unix.FD_CLOEXEC) + offset, seekErr := unix.Seek(fd, 0, unix.SEEK_CUR) + require.NoError(t, seekErr) + assert.Equal(t, int64(0), offset) + } + + for _, leaf := range []string{trustedReleaseFirecrackerLeaf, trustedReleaseJailerLeaf} { + candidatePath := filepath.Join(fixture.store.slotPath, leaf) + require.NoError(t, os.Rename(candidatePath, candidatePath+".verified")) + require.NoError(t, os.WriteFile(candidatePath, trustedReleaseTestELF(leaf+" replacement"), 0755)) + } + after := executableDescriptorInodes(t, executables) + if diff := cmp.Diff(before, after); diff != "" { + t.Errorf("retained executable inode mismatch (-before +after):\n%s", diff) + } + assert.NotEqual(t, before[0], fileInode(t, fixture.firecrackerPath)) + assert.NotEqual(t, before[1], fileInode(t, fixture.jailerPath)) + + realClose := executables.deps.close + closed := make(map[int]int, len(retainedFDs)) + executables.deps.close = func(ctx context.Context, fd int) error { + closed[fd]++ + return realClose(ctx, fd) + } + require.NoError(t, executables.Release(t.Context())) + require.NoError(t, executables.Release(t.Context())) + wantClosed := make(map[int]int, len(retainedFDs)) + for _, fd := range retainedFDs { + wantClosed[fd] = 1 + } + if diff := cmp.Diff(wantClosed, closed); diff != "" { + t.Errorf("retained executable close count mismatch (-want +got):\n%s", diff) + } + assert.Equal(t, -1, executables.firecrackerFD) + assert.Equal(t, -1, executables.jailerFD) + assert.Empty(t, executables.retained) +} + +// Rationale: an existing slot is a complete authority record. Either fixed leaf being absent, redirected, incorrectly +// attributed, structurally unsafe, or inconsistent with the manifest must invalidate the whole release. +func TestTrustedReleaseDirectoryRejectsUnsafeExecutableLeaves(t *testing.T) { + t.Parallel() + + tests := map[string]func(*testing.T, string, string, *trustedReleaseExecutable){ + "missing": func(t *testing.T, path, _ string, _ *trustedReleaseExecutable) { + require.NoError(t, os.Remove(path)) + }, + "symlink": func(t *testing.T, path, other string, _ *trustedReleaseExecutable) { + require.NoError(t, os.Remove(path)) + require.NoError(t, os.Symlink(other, path)) + }, + "directory": func(t *testing.T, path, _ string, _ *trustedReleaseExecutable) { + require.NoError(t, os.Remove(path)) + require.NoError(t, os.Mkdir(path, 0755)) + }, + "FIFO": func(t *testing.T, path, _ string, _ *trustedReleaseExecutable) { + require.NoError(t, os.Remove(path)) + require.NoError(t, unix.Mkfifo(path, 0755)) + }, + "wrong mode": func(t *testing.T, path, _ string, _ *trustedReleaseExecutable) { + require.NoError(t, os.Chmod(path, 0700)) + }, + "multiple links": func(t *testing.T, path, _ string, _ *trustedReleaseExecutable) { + require.NoError(t, os.Link(path, path+".link")) + }, + "manifest size mismatch": func(t *testing.T, path, _ string, _ *trustedReleaseExecutable) { + require.NoError(t, os.Truncate(path, trustedReleaseTestExecutableBytes-1)) + }, + "manifest digest mismatch": func(_ *testing.T, _, _ string, entry *trustedReleaseExecutable) { + entry.digest[0] ^= 0xff + }, + "invalid ELF": func(t *testing.T, path, _ string, entry *trustedReleaseExecutable) { + raw := trustedReleaseTestELF("invalid ELF") + raw[0] = 0 + writeTrustedReleaseTestExecutable(t, path, raw, entry) + }, + "wrong ELF architecture": func(t *testing.T, path, _ string, entry *trustedReleaseExecutable) { + raw := trustedReleaseTestELF("wrong architecture") + raw[18] = 183 + writeTrustedReleaseTestExecutable(t, path, raw, entry) + }, + } + + for _, leaf := range []string{trustedReleaseFirecrackerLeaf, trustedReleaseJailerLeaf} { + leaf := leaf + for name, mutate := range tests { + mutate := mutate + t.Run(leaf+"/"+name, func(t *testing.T) { + t.Parallel() + + fixture := newTrustedReleaseExecutableFixture(t) + path, other, entry := fixture.leaf(leaf) + mutate(t, path, other, entry) + directory := openTrustedReleaseTestDirectory(t, fixture.store) + + executables, err := directory.openExecutables(t.Context(), fixture.manifest) + assert.Nil(t, executables) + require.Error(t, err) + domainErr := errs.AsDomainError(err) + require.NotNil(t, domainErr) + assert.Equal(t, errs.CodeBinaryUntrusted, domainErr.Code) + }) + } + } +} + +func TestTrustedReleaseDirectoryRejectsExecutableAuthorityMismatch(t *testing.T) { + t.Parallel() + + tests := map[string]func(*trustedReleaseDirectory, *trustedReleaseManifest){ + "manifest slot": func(_ *trustedReleaseDirectory, manifest *trustedReleaseManifest) { + manifest.slot.version = "1.16.0" + }, + "owner UID": func(directory *trustedReleaseDirectory, _ *trustedReleaseManifest) { + directory.policy.expectedUID++ + }, + "owner GID": func(directory *trustedReleaseDirectory, _ *trustedReleaseManifest) { + directory.policy.expectedGID++ + }, + } + for name, mutate := range tests { + t.Run(name, func(t *testing.T) { + t.Parallel() + + fixture := newTrustedReleaseExecutableFixture(t) + directory := openTrustedReleaseTestDirectory(t, fixture.store) + mutate(directory, &fixture.manifest) + + executables, err := directory.openExecutables(t.Context(), fixture.manifest) + assert.Nil(t, executables) + require.Error(t, err) + assert.Equal(t, errs.CodeBinaryUntrusted, errs.AsDomainError(err).Code) + }) + } +} + +func TestTrustedReleaseDirectoryHashesPartialPositionedReads(t *testing.T) { + t.Parallel() + + fixture := newTrustedReleaseExecutableFixture(t) + realPread := fixture.store.deps.pread + readCalls := 0 + fixture.store.deps.pread = func(ctx context.Context, fd int, value []byte, offset int64) (int, error) { + readCalls++ + if len(value) > 7 { + value = value[:7] + } + return realPread(ctx, fd, value, offset) + } + directory := openTrustedReleaseTestDirectory(t, fixture.store) + + executables, err := directory.openExecutables(t.Context(), fixture.manifest) + require.NoError(t, err) + assert.Greater(t, readCalls, 2*trustedReleaseTestExecutableBytes/7) + for _, fd := range []int{executables.firecrackerFD, executables.jailerFD} { + offset, seekErr := unix.Seek(fd, 0, unix.SEEK_CUR) + require.NoError(t, seekErr) + assert.Equal(t, int64(0), offset) + } + require.NoError(t, executables.Release(t.Context())) +} + +func TestTrustedReleaseDirectoryRejectsExecutableMutationDuringRead(t *testing.T) { + t.Parallel() + + fixture := newTrustedReleaseExecutableFixture(t) + realPread := fixture.store.deps.pread + mutated := false + fixture.store.deps.pread = func(ctx context.Context, fd int, value []byte, offset int64) (int, error) { + count, err := realPread(ctx, fd, value, offset) + if !mutated { + mutated = true + require.NoError(t, os.Chmod(fixture.firecrackerPath, 0700)) + } + return count, err + } + directory := openTrustedReleaseTestDirectory(t, fixture.store) + + executables, err := directory.openExecutables(t.Context(), fixture.manifest) + assert.Nil(t, executables) + require.Error(t, err) + assert.Equal(t, errs.CodeBinaryUntrusted, errs.AsDomainError(err).Code) +} + +func TestTrustedReleaseDirectoryExecutableReadHonorsCancellation(t *testing.T) { + t.Parallel() + + fixture := newTrustedReleaseExecutableFixture(t) + ctx, cancel := context.WithCancel(t.Context()) + realPread := fixture.store.deps.pread + fixture.store.deps.pread = func(ctx context.Context, fd int, value []byte, offset int64) (int, error) { + count, err := realPread(ctx, fd, value, offset) + cancel() + return count, err + } + directory := openTrustedReleaseTestDirectory(t, fixture.store) + realClose := directory.deps.close + closeCount := 0 + cleanupWasCanceled := false + directory.deps.close = func(closeCtx context.Context, fd int) error { + closeCount++ + cleanupWasCanceled = cleanupWasCanceled || closeCtx.Err() != nil + return realClose(closeCtx, fd) + } + + executables, err := directory.openExecutables(ctx, fixture.manifest) + assert.Nil(t, executables) + require.Error(t, err) + assert.ErrorIs(t, err, context.Canceled) + assert.Equal(t, errs.CodeBinaryTrustedInstallFailed, errs.AsDomainError(err).Code) + assert.Equal(t, 1, closeCount) + assert.False(t, cleanupWasCanceled) + directory.deps.close = realClose +} + +func TestTrustedReleaseDirectoryRejectsInvalidExecutableReadCount(t *testing.T) { + t.Parallel() + + fixture := newTrustedReleaseExecutableFixture(t) + fixture.store.deps.pread = func(_ context.Context, _ int, value []byte, _ int64) (int, error) { + return len(value) + 1, nil + } + directory := openTrustedReleaseTestDirectory(t, fixture.store) + + executables, err := directory.openExecutables(t.Context(), fixture.manifest) + assert.Nil(t, executables) + require.Error(t, err) + assert.Equal(t, errs.CodeBinaryTrustedInstallFailed, errs.AsDomainError(err).Code) +} + +func TestTrustedReleaseDirectoryRejectsExecutableReadBoundaryFailures(t *testing.T) { + t.Parallel() + + tests := map[string]struct { + inject func(context.Context, int, []byte, int64) (int, error) + wantCode errs.Code + wantErr error + }{ + "premature EOF": { + inject: func(_ context.Context, _ int, _ []byte, _ int64) (int, error) { + return 0, nil + }, + wantCode: errs.CodeBinaryUntrusted, + }, + "I/O failure": { + inject: func(_ context.Context, _ int, _ []byte, _ int64) (int, error) { + return 0, unix.EIO + }, + wantCode: errs.CodeBinaryTrustedInstallFailed, + wantErr: unix.EIO, + }, + } + for name, tc := range tests { + t.Run(name, func(t *testing.T) { + t.Parallel() + + fixture := newTrustedReleaseExecutableFixture(t) + fixture.store.deps.pread = tc.inject + directory := openTrustedReleaseTestDirectory(t, fixture.store) + + executables, err := directory.openExecutables(t.Context(), fixture.manifest) + assert.Nil(t, executables) + require.Error(t, err) + assert.Equal(t, tc.wantCode, errs.AsDomainError(err).Code) + if tc.wantErr != nil { + assert.ErrorIs(t, err, tc.wantErr) + } + }) + } +} + +func TestTrustedReleaseDirectoryRejectsExecutableGrowthAfterInitialStat(t *testing.T) { + t.Parallel() + + fixture := newTrustedReleaseExecutableFixture(t) + realPread := fixture.store.deps.pread + fixture.store.deps.pread = func(ctx context.Context, fd int, value []byte, offset int64) (int, error) { + if offset == int64(fixture.manifest.firecracker.sizeBytes) { + value[0] = 1 + return 1, nil + } + return realPread(ctx, fd, value, offset) + } + directory := openTrustedReleaseTestDirectory(t, fixture.store) + + executables, err := directory.openExecutables(t.Context(), fixture.manifest) + assert.Nil(t, executables) + require.Error(t, err) + assert.Equal(t, errs.CodeBinaryUntrusted, errs.AsDomainError(err).Code) +} + +// Rationale: rejecting Jailer must unwind it and the already verified Firecracker in reverse order. A cleanup failure +// is diagnostic context and must not replace the security classification that rejected the release. +func TestTrustedReleaseDirectoryPreservesExecutableErrorWhenCleanupFails(t *testing.T) { + t.Parallel() + + fixture := newTrustedReleaseExecutableFixture(t) + require.NoError(t, os.Chmod(fixture.jailerPath, 0700)) + directory := openTrustedReleaseTestDirectory(t, fixture.store) + realOpenAt := directory.deps.openAt + opened := make(map[int]string) + directory.deps.openAt = func(ctx context.Context, parentFD int, name string, flags int, mode uint32) (int, error) { + fd, err := realOpenAt(ctx, parentFD, name, flags, mode) + if err == nil { + opened[fd] = name + } + return fd, err + } + realClose := directory.deps.close + closed := make([]string, 0, 2) + directory.deps.close = func(ctx context.Context, fd int) error { + closed = append(closed, opened[fd]) + closeErr := realClose(ctx, fd) + if opened[fd] == trustedReleaseFirecrackerLeaf { + return errors.Join(closeErr, unix.EIO) + } + return closeErr + } + + executables, err := directory.openExecutables(t.Context(), fixture.manifest) + assert.Nil(t, executables) + require.Error(t, err) + domainErr := errs.AsDomainError(err) + require.NotNil(t, domainErr) + assert.Equal(t, errs.CodeBinaryUntrusted, domainErr.Code) + assert.Contains(t, domainErr.Message, "close rejected trusted release executables") + assert.ErrorIs(t, err, unix.EIO) + if diff := cmp.Diff([]string{trustedReleaseJailerLeaf, trustedReleaseFirecrackerLeaf}, closed); diff != "" { + t.Errorf("executable cleanup order mismatch (-want +got):\n%s", diff) + } + directory.deps.openAt = realOpenAt + directory.deps.close = realClose +} + +type trustedReleaseExecutableFixture struct { + store *trustedReleaseStoreFixture + manifest trustedReleaseManifest + firecrackerPath string + jailerPath string +} + +// newTrustedReleaseExecutableFixture creates both fixed executable leaves and synchronizes their manifest authority. +func newTrustedReleaseExecutableFixture(t *testing.T) *trustedReleaseExecutableFixture { + t.Helper() + + store := newTrustedReleaseStoreFixture(t) + manifest := testTrustedReleaseManifest(store.slot) + firecrackerPath := filepath.Join(store.slotPath, trustedReleaseFirecrackerLeaf) + jailerPath := filepath.Join(store.slotPath, trustedReleaseJailerLeaf) + writeTrustedReleaseTestExecutable(t, firecrackerPath, trustedReleaseTestELF("firecracker"), &manifest.firecracker) + writeTrustedReleaseTestExecutable(t, jailerPath, trustedReleaseTestELF("jailer"), &manifest.jailer) + + return &trustedReleaseExecutableFixture{ + store: store, + manifest: manifest, + firecrackerPath: firecrackerPath, + jailerPath: jailerPath, + } +} + +// leaf selects one fixed candidate path, its peer path, and its corresponding manifest entry for symmetric cases. +func (fixture *trustedReleaseExecutableFixture) leaf( + name string, +) (string, string, *trustedReleaseExecutable) { + if name == trustedReleaseFirecrackerLeaf { + return fixture.firecrackerPath, fixture.jailerPath, &fixture.manifest.firecracker + } + return fixture.jailerPath, fixture.firecrackerPath, &fixture.manifest.jailer +} + +// openTrustedReleaseTestDirectory pins the complete synthetic store chain and registers reverse-order test cleanup. +func openTrustedReleaseTestDirectory( + t *testing.T, + fixture *trustedReleaseStoreFixture, +) *trustedReleaseDirectory { + t.Helper() + + store, err := openTrustedReleaseStoreForRead(t.Context(), fixture.deps, fixture.policy) + require.NoError(t, err) + t.Cleanup(func() { require.NoError(t, store.Release(context.Background())) }) + directory, err := store.openInstalledSlot(t.Context(), fixture.slot) + require.NoError(t, err) + t.Cleanup(func() { require.NoError(t, directory.Release(context.Background())) }) + return directory +} + +// writeTrustedReleaseTestExecutable updates one fixed file and its exact typed manifest size and digest together. +func writeTrustedReleaseTestExecutable( + t *testing.T, + path string, + raw []byte, + entry *trustedReleaseExecutable, +) { + t.Helper() + + require.NoError(t, os.WriteFile(path, raw, 0755)) + digest := sha256.Sum256(raw) + entry.digest = trustedReleaseExecutableDigest(digest) + entry.sizeBytes = uint64(len(raw)) +} + +// trustedReleaseTestELF returns a bounded x86_64 candidate with the independently audited test header. +func trustedReleaseTestELF(label string) []byte { + raw := make([]byte, trustedReleaseTestExecutableBytes) + copy(raw, auditedTrustedReleaseELFHeader()) + copy(raw[trustedReleaseELFHeaderBytes:], label) + return raw +} + +// executableDescriptorInodes captures the two retained identities without consulting their replaceable pathnames. +func executableDescriptorInodes(t *testing.T, executables *trustedReleaseExecutables) []uint64 { + t.Helper() + + result := make([]uint64, 0, 2) + for _, fd := range []int{executables.firecrackerFD, executables.jailerFD} { + var stat unix.Stat_t + require.NoError(t, unix.Fstat(fd, &stat)) + result = append(result, stat.Ino) + } + return result +} diff --git a/internal/service/jailer/trusted_release_store_fs.go b/internal/service/jailer/trusted_release_store_fs.go index 7b134d92..dc39a40d 100644 --- a/internal/service/jailer/trusted_release_store_fs.go +++ b/internal/service/jailer/trusted_release_store_fs.go @@ -23,6 +23,7 @@ type trustedReleaseStoreDeps struct { openAt func(context.Context, int, string, int, uint32) (int, error) fstat func(context.Context, int, *unix.Stat_t) error read func(context.Context, int, []byte) (int, error) + pread func(context.Context, int, []byte, int64) (int, error) close func(context.Context, int) error } @@ -351,6 +352,9 @@ func realTrustedReleaseStoreDeps() trustedReleaseStoreDeps { read: func(_ context.Context, fd int, value []byte) (int, error) { return unix.Read(fd, value) }, + pread: func(_ context.Context, fd int, value []byte, offset int64) (int, error) { + return unix.Pread(fd, value, offset) + }, close: func(_ context.Context, fd int) error { return unix.Close(fd) }, diff --git a/internal/service/jailer/trusted_release_store_manifest.go b/internal/service/jailer/trusted_release_store_manifest.go index 6e02cf5d..280bc264 100644 --- a/internal/service/jailer/trusted_release_store_manifest.go +++ b/internal/service/jailer/trusted_release_store_manifest.go @@ -10,7 +10,7 @@ import ( ) const ( - trustedReleaseStoreReadFlags = unix.O_RDONLY | unix.O_CLOEXEC | unix.O_NOFOLLOW + trustedReleaseStoreReadFlags = unix.O_RDONLY | unix.O_CLOEXEC | unix.O_NOFOLLOW | unix.O_NONBLOCK trustedReleaseStoreManifestMode = uint32(0600) ) diff --git a/internal/service/jailer/trusted_release_store_manifest_test.go b/internal/service/jailer/trusted_release_store_manifest_test.go index 40d410fa..cbc59c6f 100644 --- a/internal/service/jailer/trusted_release_store_manifest_test.go +++ b/internal/service/jailer/trusted_release_store_manifest_test.go @@ -63,6 +63,9 @@ func TestTrustedReleaseDirectoryRejectsUnsafeOrCorruptManifest(t *testing.T) { "directory": func(t *testing.T, fixture *trustedReleaseStoreFixture) { require.NoError(t, os.Mkdir(filepath.Join(fixture.slotPath, trustedReleaseManifestLeaf), 0600)) }, + "FIFO": func(t *testing.T, fixture *trustedReleaseStoreFixture) { + require.NoError(t, unix.Mkfifo(filepath.Join(fixture.slotPath, trustedReleaseManifestLeaf), 0600)) + }, "wrong mode": func(t *testing.T, fixture *trustedReleaseStoreFixture) { writeTrustedReleaseManifestFile(t, fixture.slotPath, testTrustedReleaseManifest(fixture.slot)) require.NoError(t, os.Chmod(filepath.Join(fixture.slotPath, trustedReleaseManifestLeaf), 0644)) From 3a33f0e518aa62c2fbe99e69f7e71c86964a1317 Mon Sep 17 00:00:00 2001 From: AlanD20 Date: Sat, 29 Aug 2026 01:32:50 +0200 Subject: [PATCH 0075/1457] docs: record pinned trusted executables --- ...owned-single-binary-privileged-dispatch.md | 28 ++++++++++--------- tasks/todo.md | 2 +- 2 files changed, 16 insertions(+), 14 deletions(-) diff --git a/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md b/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md index 46f1662d..ed784bf0 100644 --- a/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md +++ b/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md @@ -293,16 +293,17 @@ result. Once the version directory exists, a missing or unsafe fixed leaf is an fails as `binary.untrusted`, never as an absent release. Reads retain the version-directory descriptor, open each leaf once, compare pre/post-read descriptor identity and metadata, and do not reconstruct a pathname. -Executable admission opens the fixed `firecracker` and `jailer` leaves with `O_NOFOLLOW` relative to that retained -version-directory descriptor. It requires each descriptor to identify a one-link `root:root` regular file of exact mode -`0755`, with a size equal to its manifest entry and within the hard executable bound. Verification uses bounded -positioned reads from offset zero with a fixed 32 KiB buffer: it hashes exactly the declared bytes, captures the first -64 bytes for the closed ELF policy, and probes the next byte so truncation and growth fail closed. Positioned reads -leave the descriptor offset unchanged. A second descriptor stat must match device, inode, type and mode, link count, -owner and group, size, modification time, and change time from the first stat. The service retains those same verified -descriptors for the later launch handoff; it never validates one pathname and executes a reopened pathname. Failure to -admit `jailer` closes an already admitted `firecracker`, and cleanup preserves the primary domain error while reporting -any close failure. +Executable admission opens the fixed `firecracker` and `jailer` leaves with `O_NOFOLLOW|O_NONBLOCK` relative to that +retained version-directory descriptor. The nonblocking open prevents an unsafe special-file leaf from stalling root +before type admission. Each descriptor must identify a one-link `root:root` regular file of exact mode `0755`, with a +size equal to its manifest entry and within the hard executable bound. Verification uses bounded positioned reads from +offset zero with a fixed 32 KiB buffer: it hashes exactly the declared bytes, captures the first 64 bytes for the closed +ELF policy, and probes the next byte so truncation and growth fail closed. Positioned reads leave the descriptor offset +unchanged. A second descriptor stat must match device, inode, type and mode, link count, owner and group, size, +modification time, and change time from the first stat. The service retains those same verified descriptors for the +later launch handoff; it never validates one pathname and executes a reopened pathname. Failure to admit `jailer` +closes an already admitted `firecracker`, and cleanup preserves the primary domain error while reporting any close +failure. Implementation note (2026-08-29): the private Jailer service derives the exact source identity from a validated `(version, architecture)` release slot and rejects non-canonical slots before constructing any source value. Its @@ -313,10 +314,11 @@ bounded header shape, selected architecture, exact bounded file size, and comple loading or executing candidate bytes. The private manifest codec enforces the closed schema, record and executable bounds, typed digests, and manifest-derived instance release identity. The private read-side store foundation pins and validates the fixed root/architecture/version directory chain, then -reads `manifest.json` once through the retained slot descriptor with stable metadata and exact slot verification. These +reads `manifest.json` once through the retained slot descriptor with stable metadata and exact slot verification. It +then opens the two fixed executable leaves nonblocking, verifies their exact metadata, manifest sizes and full hashes, +closed ELF policy, read bounds, and pre/post-read stability, and retains the same descriptors at offset zero. These foundations are not yet wired to the privileged request or legacy installer. Bounded archive transport and extraction, -the complete member allowlist, manifest writing, executable verification/pinning, and atomic trusted-store installation -remain Task 6 work. +the complete member allowlist, manifest writing, and atomic trusted-store installation remain Task 6 work. ### Paths, process identity, and runtime state diff --git a/tasks/todo.md b/tasks/todo.md index f4da6187..8e0e5e00 100644 --- a/tasks/todo.md +++ b/tasks/todo.md @@ -225,7 +225,7 @@ under `internal/service/jailer/`. No handler, transport, CLI, API, core-domain, - [x] Pin and validate the fixed root/architecture/version directory chain without following links or reopening paths. - [x] Read `manifest.json` once through the pinned slot; verify exact metadata, bounded bytes, stable descriptor identity, strict decoding, and exact slot equality. - - [ ] Pin and verify the fixed `firecracker` and `jailer` leaves through the retained slot descriptor: exact metadata + - [x] Pin and verify the fixed `firecracker` and `jailer` leaves through the retained slot descriptor: exact metadata and manifest size, bounded positioned full-file hashing, closed ELF admission, pre/post-read stability, unchanged offsets, exact descriptor retention, cancellation, and checked reverse cleanup. - [ ] Referenced release removal/replacement holds the release lease and fails closed on unreadable records. From 81468aa50eea09ea0111f20288b6f53017838acf Mon Sep 17 00:00:00 2001 From: AlanD20 Date: Sat, 29 Aug 2026 01:35:03 +0200 Subject: [PATCH 0076/1457] docs: define prepared trusted release lease --- .../0016-root-owned-single-binary-privileged-dispatch.md | 8 ++++++++ tasks/todo.md | 2 ++ 2 files changed, 10 insertions(+) diff --git a/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md b/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md index ed784bf0..389ecf70 100644 --- a/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md +++ b/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md @@ -305,6 +305,14 @@ later launch handoff; it never validates one pathname and executes a reopened pa closes an already admitted `firecracker`, and cleanup preserves the primary domain error while reporting any close failure. +Installed-release preparation acquires the canonical release-slot lease before opening the trusted store or resolving +`manifest.json`. One private prepared value then owns that lease, the pinned store and version directory, the verified +manifest-derived release identity, and the exact admitted executable descriptors. Preparation failure releases acquired +resources in strict reverse order; successful release closes executables, version traversal, store traversal, and only +then the release-slot lease. Cleanup always uses a cancellation-independent context and preserves the primary domain +error while joining cleanup diagnostics. This slice exposes only preparation and checked release. Launch registration +remains deferred until the process-identity work can transfer the release lease and executable ownership together. + Implementation note (2026-08-29): the private Jailer service derives the exact source identity from a validated `(version, architecture)` release slot and rejects non-canonical slots before constructing any source value. Its dedicated checksum authority independently fetches the derived sidecar with the closed transport policy above and diff --git a/tasks/todo.md b/tasks/todo.md index 8e0e5e00..ff7137a1 100644 --- a/tasks/todo.md +++ b/tasks/todo.md @@ -228,6 +228,8 @@ under `internal/service/jailer/`. No handler, transport, CLI, API, core-domain, - [x] Pin and verify the fixed `firecracker` and `jailer` leaves through the retained slot descriptor: exact metadata and manifest size, bounded positioned full-file hashing, closed ELF admission, pre/post-read stability, unchanged offsets, exact descriptor retention, cancellation, and checked reverse cleanup. + - [ ] Compose the release-slot lease, pinned store and version directory, verified manifest identity, and exact + executable descriptors into one private prepared installed-release lease with strict reverse checked cleanup. - [ ] Referenced release removal/replacement holds the release lease and fails closed on unreadable records. - [ ] L1 failure injection proves an old complete pair or no pair, never a partial trusted release. From ce0671cf9558293f64ed724dbcc4680ad79a22a6 Mon Sep 17 00:00:00 2001 From: AlanD20 Date: Sat, 29 Aug 2026 01:38:56 +0200 Subject: [PATCH 0077/1457] feat: prepare trusted installed releases --- .../jailer/trusted_release_prepared.go | 198 +++++++++++ .../jailer/trusted_release_prepared_test.go | 326 ++++++++++++++++++ 2 files changed, 524 insertions(+) create mode 100644 internal/service/jailer/trusted_release_prepared.go create mode 100644 internal/service/jailer/trusted_release_prepared_test.go diff --git a/internal/service/jailer/trusted_release_prepared.go b/internal/service/jailer/trusted_release_prepared.go new file mode 100644 index 00000000..eb551966 --- /dev/null +++ b/internal/service/jailer/trusted_release_prepared.go @@ -0,0 +1,198 @@ +package jailer + +import ( + "context" + "errors" + + "mvmctl/pkg/errs" +) + +type releaseAuthority struct { + instances *instanceAuthority + storeDeps trustedReleaseStoreDeps + storePolicy trustedReleaseStorePolicy +} + +type preparedRelease struct { + slotLease *releaseSlotLease + store *trustedReleaseStore + directory *trustedReleaseDirectory + manifest trustedReleaseManifest + identity releaseIdentity + executables *trustedReleaseExecutables +} + +func newReleaseAuthority() *releaseAuthority { + return newReleaseAuthorityWithPolicy( + newInstanceAuthority(), + realTrustedReleaseStoreDeps(), + productionTrustedReleaseStorePolicy(), + ) +} + +func newReleaseAuthorityWithPolicy( + instances *instanceAuthority, + storeDeps trustedReleaseStoreDeps, + storePolicy trustedReleaseStorePolicy, +) *releaseAuthority { + return &releaseAuthority{ + instances: instances, + storeDeps: storeDeps, + storePolicy: storePolicy, + } +} + +// CRITICAL: The release-slot lock precedes every trusted-store read. The returned value exclusively owns each object +// selected while holding that lock, so the verified manifest identity and executable descriptors remain one authority. +func (authority *releaseAuthority) prepareInstalled( + ctx context.Context, + slot releaseSlot, +) (_ *preparedRelease, returnErr error) { + if authority == nil || authority.instances == nil { + return nil, trustedReleaseStoreError("release authority is not active", nil) + } + if err := validateReleaseSlotValue(slot); err != nil { + return nil, instanceValidationError(err.Error()) + } + if err := ctx.Err(); err != nil { + return nil, trustedReleaseStoreError("prepare installed trusted release", err) + } + + var slotLease *releaseSlotLease + var store *trustedReleaseStore + var directory *trustedReleaseDirectory + var executables *trustedReleaseExecutables + defer func() { + if returnErr == nil { + return + } + returnErr = releasePreparedReleaseResources( + ctx, + returnErr, + executables, + directory, + store, + slotLease, + "release rejected prepared trusted release", + ) + }() + + var err error + slotLease, err = authority.instances.lockReleaseSlot(ctx, slot) + if err != nil { + return nil, err + } + store, err = openTrustedReleaseStoreForRead(ctx, authority.storeDeps, authority.storePolicy) + if err != nil { + return nil, err + } + directory, err = store.openInstalledSlot(ctx, slot) + if err != nil { + return nil, err + } + manifest, err := directory.readManifest(ctx) + if err != nil { + return nil, err + } + executables, err = directory.openExecutables(ctx, manifest) + if err != nil { + return nil, err + } + identity, err := manifest.releaseIdentity() + if err != nil { + return nil, err + } + + prepared := &preparedRelease{ + slotLease: slotLease, + store: store, + directory: directory, + manifest: manifest, + identity: identity, + executables: executables, + } + slotLease = nil + store = nil + directory = nil + executables = nil + return prepared, nil +} + +func (prepared *preparedRelease) Release(ctx context.Context) error { + if prepared == nil { + return nil + } + err := releasePreparedReleaseResources( + ctx, + nil, + prepared.executables, + prepared.directory, + prepared.store, + prepared.slotLease, + "release prepared trusted release", + ) + prepared.executables = nil + prepared.directory = nil + prepared.store = nil + prepared.slotLease = nil + prepared.manifest = trustedReleaseManifest{} + prepared.identity = releaseIdentity{} + return err +} + +func releasePreparedReleaseResources( + ctx context.Context, + primary error, + executables *trustedReleaseExecutables, + directory *trustedReleaseDirectory, + store *trustedReleaseStore, + slotLease *releaseSlotLease, + description string, +) error { + cleanupCtx := context.WithoutCancel(ctx) + result := primary + if executables != nil { + result = appendPreparedReleaseError( + result, + description+" executable descriptors", + executables.Release(cleanupCtx), + ) + } + if directory != nil { + result = appendPreparedReleaseError( + result, + description+" version directory", + directory.Release(cleanupCtx), + ) + } + if store != nil { + result = appendPreparedReleaseError( + result, + description+" store", + store.Release(cleanupCtx), + ) + } + if slotLease != nil { + result = appendPreparedReleaseError( + result, + description+" slot lease", + slotLease.Release(cleanupCtx), + ) + } + return result +} + +func appendPreparedReleaseError(primary error, message string, cause error) error { + if cause == nil { + return primary + } + if primary == nil { + return cause + } + if domainErr := errs.AsDomainError(primary); domainErr != nil { + domainErr.Message += "; " + message + ": " + cause.Error() + domainErr.Err = errors.Join(domainErr.Err, cause) + return domainErr + } + return trustedReleaseStoreError(message, errors.Join(primary, cause)) +} diff --git a/internal/service/jailer/trusted_release_prepared_test.go b/internal/service/jailer/trusted_release_prepared_test.go new file mode 100644 index 00000000..dfb411b1 --- /dev/null +++ b/internal/service/jailer/trusted_release_prepared_test.go @@ -0,0 +1,326 @@ +package jailer + +import ( + "context" + "errors" + "os" + "path/filepath" + "testing" + "time" + + "github.com/google/go-cmp/cmp" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + "golang.org/x/sys/unix" + + "mvmctl/pkg/errs" +) + +// Rationale: one value must own every authority object selected after the release lock, including the identity derived +// from the verified manifest. It must not expose a state where hashes and retained executable descriptors can diverge. +func TestReleaseAuthorityPreparesInstalledReleaseUnderSlotLease(t *testing.T) { + t.Parallel() + + fixture := newPreparedReleaseFixture(t) + prepared, err := fixture.authority().prepareInstalled(t.Context(), fixture.release.store.slot) + require.NoError(t, err) + wantIdentity, err := fixture.release.manifest.releaseIdentity() + require.NoError(t, err) + if diff := cmp.Diff( + fixture.release.manifest, + prepared.manifest, + cmp.AllowUnexported(trustedReleaseManifest{}, releaseSlot{}, trustedReleaseExecutable{}), + ); diff != "" { + t.Errorf("prepared manifest mismatch (-want +got):\n%s", diff) + } + if diff := cmp.Diff(wantIdentity, prepared.identity, cmp.AllowUnexported(releaseIdentity{})); diff != "" { + t.Errorf("prepared release identity mismatch (-want +got):\n%s", diff) + } + assert.NotNil(t, prepared.slotLease) + assert.NotNil(t, prepared.store) + assert.NotNil(t, prepared.directory) + assert.NotNil(t, prepared.executables) + assert.GreaterOrEqual(t, prepared.executables.firecrackerFD, 0) + assert.GreaterOrEqual(t, prepared.executables.jailerFD, 0) + + require.NoError(t, prepared.Release(t.Context())) + require.NoError(t, prepared.Release(t.Context())) + assert.Nil(t, prepared.slotLease) + assert.Nil(t, prepared.store) + assert.Nil(t, prepared.directory) + assert.Nil(t, prepared.executables) + if diff := cmp.Diff( + trustedReleaseManifest{}, + prepared.manifest, + cmp.AllowUnexported(trustedReleaseManifest{}, releaseSlot{}, trustedReleaseExecutable{}), + ); diff != "" { + t.Errorf("released manifest was not cleared (-want +got):\n%s", diff) + } + if diff := cmp.Diff(releaseIdentity{}, prepared.identity, cmp.AllowUnexported(releaseIdentity{})); diff != "" { + t.Errorf("released identity was not cleared (-want +got):\n%s", diff) + } + assertPreparedReleaseSlotReacquirable(t, fixture.instances, fixture.release.store.slot) +} + +func TestReleaseAuthorityLocksSlotBeforeTrustedStoreResolution(t *testing.T) { + t.Parallel() + + fixture := newPreparedReleaseFixture(t) + lockHeld := false + realFlock := fixture.instances.deps.flock + fixture.instances.deps.flock = func(ctx context.Context, fd int, how int) error { + err := realFlock(ctx, fd, how) + if err == nil && how == unix.LOCK_EX|unix.LOCK_NB { + lockHeld = true + } + if err == nil && how == unix.LOCK_UN { + lockHeld = false + } + return err + } + storeOpenedWhileLocked := false + realOpen := fixture.release.store.deps.open + fixture.release.store.deps.open = func(ctx context.Context, name string, flags int, mode uint32) (int, error) { + storeOpenedWhileLocked = lockHeld + return realOpen(ctx, name, flags, mode) + } + + prepared, err := fixture.authority().prepareInstalled(t.Context(), fixture.release.store.slot) + require.NoError(t, err) + assert.True(t, storeOpenedWhileLocked) + assert.True(t, lockHeld) + require.NoError(t, prepared.Release(t.Context())) + assert.False(t, lockHeld) +} + +func TestReleaseAuthorityPreparationFailureReleasesSlotLease(t *testing.T) { + t.Parallel() + + tests := map[string]struct { + mutate func(*testing.T, *preparedReleaseFixture) + wantCode errs.Code + }{ + "missing manifest": { + mutate: func(t *testing.T, fixture *preparedReleaseFixture) { + require.NoError(t, os.Remove(filepath.Join( + fixture.release.store.slotPath, + trustedReleaseManifestLeaf, + ))) + }, + wantCode: errs.CodeBinaryUntrusted, + }, + "corrupt manifest": { + mutate: func(t *testing.T, fixture *preparedReleaseFixture) { + require.NoError(t, os.WriteFile( + filepath.Join(fixture.release.store.slotPath, trustedReleaseManifestLeaf), + []byte(`{"invalid":true}`), + 0600, + )) + }, + wantCode: errs.CodeBinaryUntrusted, + }, + "unsafe executable": { + mutate: func(t *testing.T, fixture *preparedReleaseFixture) { + require.NoError(t, os.Chmod(fixture.release.firecrackerPath, 0700)) + }, + wantCode: errs.CodeBinaryUntrusted, + }, + } + for name, tc := range tests { + t.Run(name, func(t *testing.T) { + t.Parallel() + + fixture := newPreparedReleaseFixture(t) + tc.mutate(t, fixture) + prepared, err := fixture.authority().prepareInstalled(t.Context(), fixture.release.store.slot) + assert.Nil(t, prepared) + require.Error(t, err) + assert.Equal(t, tc.wantCode, errs.AsDomainError(err).Code) + assertPreparedReleaseSlotReacquirable(t, fixture.instances, fixture.release.store.slot) + }) + } +} + +// Rationale: ownership unwinds opposite acquisition. The release lock is last so no competing install, removal, or +// launch can observe the slot while any pinned release object from this preparation remains active. +func TestPreparedReleaseReleasesResourcesInReverseWithoutCancellation(t *testing.T) { + t.Parallel() + + fixture := newPreparedReleaseFixture(t) + events := make([]string, 0, 10) + fdNames := make(map[int]string) + cleanupWasCanceled := false + realOpen := fixture.release.store.deps.open + fixture.release.store.deps.open = func(ctx context.Context, name string, flags int, mode uint32) (int, error) { + fd, err := realOpen(ctx, name, flags, mode) + if err == nil { + fdNames[fd] = "root" + } + return fd, err + } + realOpenAt := fixture.release.store.deps.openAt + fixture.release.store.deps.openAt = func( + ctx context.Context, + parentFD int, + name string, + flags int, + mode uint32, + ) (int, error) { + fd, err := realOpenAt(ctx, parentFD, name, flags, mode) + if err == nil { + fdNames[fd] = name + } + return fd, err + } + realStoreClose := fixture.release.store.deps.close + fixture.release.store.deps.close = func(ctx context.Context, fd int) error { + cleanupWasCanceled = cleanupWasCanceled || ctx.Err() != nil + events = append(events, "store:"+fdNames[fd]) + return realStoreClose(ctx, fd) + } + realFlock := fixture.instances.deps.flock + fixture.instances.deps.flock = func(ctx context.Context, fd int, how int) error { + if how == unix.LOCK_UN { + cleanupWasCanceled = cleanupWasCanceled || ctx.Err() != nil + events = append(events, "instance:release-lock") + } + return realFlock(ctx, fd, how) + } + + prepared, err := fixture.authority().prepareInstalled(t.Context(), fixture.release.store.slot) + require.NoError(t, err) + events = events[:0] + ctx, cancel := context.WithCancel(t.Context()) + cancel() + require.NoError(t, prepared.Release(ctx)) + + want := []string{ + "store:jailer", + "store:firecracker", + "store:1.16.1", + "store:x86_64", + "store:binaries", + "store:mvmctl", + "store:lib", + "store:var", + "store:root", + "instance:release-lock", + } + if diff := cmp.Diff(want, events); diff != "" { + t.Errorf("prepared release cleanup order mismatch (-want +got):\n%s", diff) + } + assert.False(t, cleanupWasCanceled) +} + +func TestReleaseAuthorityPreparationPreservesPrimaryErrorWhenCleanupFails(t *testing.T) { + t.Parallel() + + fixture := newPreparedReleaseFixture(t) + require.NoError(t, os.WriteFile( + filepath.Join(fixture.release.store.slotPath, trustedReleaseManifestLeaf), + []byte(`{"invalid":true}`), + 0600, + )) + realClose := fixture.release.store.deps.close + fixture.release.store.deps.close = func(ctx context.Context, fd int) error { + return errors.Join(realClose(ctx, fd), unix.EIO) + } + + prepared, err := fixture.authority().prepareInstalled(t.Context(), fixture.release.store.slot) + assert.Nil(t, prepared) + require.Error(t, err) + domainErr := errs.AsDomainError(err) + require.NotNil(t, domainErr) + assert.Equal(t, errs.CodeBinaryUntrusted, domainErr.Code) + assert.ErrorIs(t, err, unix.EIO) + assert.Contains(t, domainErr.Message, "release rejected prepared trusted release") + assertPreparedReleaseSlotReacquirable(t, fixture.instances, fixture.release.store.slot) +} + +func TestReleaseAuthorityPreparationHonorsCancellationAndCleansUp(t *testing.T) { + t.Parallel() + + fixture := newPreparedReleaseFixture(t) + ctx, cancel := context.WithCancel(t.Context()) + realPread := fixture.release.store.deps.pread + fixture.release.store.deps.pread = func(ctx context.Context, fd int, value []byte, offset int64) (int, error) { + count, err := realPread(ctx, fd, value, offset) + cancel() + return count, err + } + cleanupWasCanceled := false + realClose := fixture.release.store.deps.close + fixture.release.store.deps.close = func(ctx context.Context, fd int) error { + cleanupWasCanceled = cleanupWasCanceled || ctx.Err() != nil + return realClose(ctx, fd) + } + + prepared, err := fixture.authority().prepareInstalled(ctx, fixture.release.store.slot) + assert.Nil(t, prepared) + require.Error(t, err) + assert.ErrorIs(t, err, context.Canceled) + assert.Equal(t, errs.CodeBinaryTrustedInstallFailed, errs.AsDomainError(err).Code) + assert.False(t, cleanupWasCanceled) + assertPreparedReleaseSlotReacquirable(t, fixture.instances, fixture.release.store.slot) +} + +func TestPreparedReleaseContinuesCleanupAfterExecutableCloseFailure(t *testing.T) { + t.Parallel() + + fixture := newPreparedReleaseFixture(t) + prepared, err := fixture.authority().prepareInstalled(t.Context(), fixture.release.store.slot) + require.NoError(t, err) + realClose := prepared.executables.deps.close + prepared.executables.deps.close = func(ctx context.Context, fd int) error { + return errors.Join(realClose(ctx, fd), unix.EIO) + } + + err = prepared.Release(t.Context()) + require.Error(t, err) + assert.ErrorIs(t, err, unix.EIO) + assert.Equal(t, errs.CodeBinaryTrustedInstallFailed, errs.AsDomainError(err).Code) + assert.Nil(t, prepared.slotLease) + assert.Nil(t, prepared.store) + assert.Nil(t, prepared.directory) + assert.Nil(t, prepared.executables) + assertPreparedReleaseSlotReacquirable(t, fixture.instances, fixture.release.store.slot) +} + +type preparedReleaseFixture struct { + instances *instanceAuthority + release *trustedReleaseExecutableFixture +} + +// newPreparedReleaseFixture builds independent instance-lock and trusted-store roots and writes one complete manifest. +func newPreparedReleaseFixture(t *testing.T) *preparedReleaseFixture { + t.Helper() + + instances, _ := newTestInstanceAuthority(t) + release := newTrustedReleaseExecutableFixture(t) + writeTrustedReleaseManifestFile(t, release.store.slotPath, release.manifest) + return &preparedReleaseFixture{instances: instances, release: release} +} + +// authority snapshots the fixture's injected dependencies after a test has installed any failure hooks. +func (fixture *preparedReleaseFixture) authority() *releaseAuthority { + return newReleaseAuthorityWithPolicy( + fixture.instances, + fixture.release.store.deps, + fixture.release.store.policy, + ) +} + +// assertPreparedReleaseSlotReacquirable proves failure and release paths do not strand the ordering lock. +func assertPreparedReleaseSlotReacquirable( + t *testing.T, + authority *instanceAuthority, + slot releaseSlot, +) { + t.Helper() + + ctx, cancel := context.WithTimeout(context.Background(), time.Second) + defer cancel() + lease, err := authority.lockReleaseSlot(ctx, slot) + require.NoError(t, err) + require.NoError(t, lease.Release(context.Background())) +} From 3d180524d3df21795ac3f11ed1af2c09a4d69d16 Mon Sep 17 00:00:00 2001 From: AlanD20 Date: Sat, 29 Aug 2026 01:39:23 +0200 Subject: [PATCH 0078/1457] docs: record prepared trusted release lease --- .../0016-root-owned-single-binary-privileged-dispatch.md | 8 +++++--- tasks/plan.md | 8 ++++++++ tasks/todo.md | 2 +- 3 files changed, 14 insertions(+), 4 deletions(-) diff --git a/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md b/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md index 389ecf70..f106924a 100644 --- a/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md +++ b/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md @@ -324,9 +324,11 @@ bounds, typed digests, and manifest-derived instance release identity. The private read-side store foundation pins and validates the fixed root/architecture/version directory chain, then reads `manifest.json` once through the retained slot descriptor with stable metadata and exact slot verification. It then opens the two fixed executable leaves nonblocking, verifies their exact metadata, manifest sizes and full hashes, -closed ELF policy, read bounds, and pre/post-read stability, and retains the same descriptors at offset zero. These -foundations are not yet wired to the privileged request or legacy installer. Bounded archive transport and extraction, -the complete member allowlist, manifest writing, and atomic trusted-store installation remain Task 6 work. +closed ELF policy, read bounds, and pre/post-read stability, and retains the same descriptors at offset zero. The private +release authority now acquires the release-slot lease before those reads and returns one prepared value owning the lease, +pinned store chain, manifest-derived identity, and verified descriptors with checked reverse cleanup. These foundations +are not yet wired to the privileged request or legacy installer. Bounded archive transport and extraction, the complete +member allowlist, manifest writing, and atomic trusted-store installation remain Task 6 work. ### Paths, process identity, and runtime state diff --git a/tasks/plan.md b/tasks/plan.md index 03fb79fe..d7a69476 100644 --- a/tasks/plan.md +++ b/tasks/plan.md @@ -244,6 +244,14 @@ pinned executable descriptors. It alone supplies release hashes to instance regi return a strict versioned response envelope before release install/remove is wired, because generic subprocess errors cannot preserve the partial-state details required for safe retries. +```go +func (a *releaseAuthority) prepareInstalled(ctx context.Context, slot releaseSlot) (*preparedRelease, error) +func (p *preparedRelease) Release(ctx context.Context) error +``` + +Preparation and checked release are implemented first. The later launch slice adds only the typed ownership transfer +that it actually needs; it does not expose raw descriptors, paths, caller-supplied hashes, or a generic operation hook. + ### Network topology and policy ADR-0017 is the authoritative design: diff --git a/tasks/todo.md b/tasks/todo.md index ff7137a1..e5cb79c5 100644 --- a/tasks/todo.md +++ b/tasks/todo.md @@ -228,7 +228,7 @@ under `internal/service/jailer/`. No handler, transport, CLI, API, core-domain, - [x] Pin and verify the fixed `firecracker` and `jailer` leaves through the retained slot descriptor: exact metadata and manifest size, bounded positioned full-file hashing, closed ELF admission, pre/post-read stability, unchanged offsets, exact descriptor retention, cancellation, and checked reverse cleanup. - - [ ] Compose the release-slot lease, pinned store and version directory, verified manifest identity, and exact + - [x] Compose the release-slot lease, pinned store and version directory, verified manifest identity, and exact executable descriptors into one private prepared installed-release lease with strict reverse checked cleanup. - [ ] Referenced release removal/replacement holds the release lease and fails closed on unreadable records. - [ ] L1 failure injection proves an old complete pair or no pair, never a partial trusted release. From cece99f39a9237c9df545f12c49aa61129cca879 Mon Sep 17 00:00:00 2001 From: AlanD20 Date: Sat, 29 Aug 2026 01:42:24 +0200 Subject: [PATCH 0079/1457] docs: define trusted release store creation --- .../0016-root-owned-single-binary-privileged-dispatch.md | 8 ++++++++ tasks/todo.md | 2 ++ 2 files changed, 10 insertions(+) diff --git a/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md b/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md index f106924a..83cd1228 100644 --- a/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md +++ b/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md @@ -293,6 +293,14 @@ result. Once the version directory exists, a missing or unsafe fixed leaf is an fails as `binary.untrusted`, never as an absent release. Reads retain the version-directory descriptor, open each leaf once, compare pre/post-read descriptor identity and metadata, and do not reconstruct a pathname. +Write-side store preparation never creates `/`, `/var`, or `/var/lib`. It may create only the fixed `mvmctl` and +`binaries` components relative to their already pinned parents, using mode `0700`, followed by descriptor-based owner, +group, type, and exact-mode verification. A component created by this call is explicitly set to `root:root 0700`; an +`EEXIST` result is treated as a concurrent creation and is verified without changing its metadata. After a newly +observed component is admitted, both that directory and its parent are fsynced before traversal continues. Symlink, +replacement, unsafe metadata, cancellation, and durability failures close all acquired descriptors in reverse order and +fail closed. This foundation creates no architecture/version slot and writes no archive or executable bytes. + Executable admission opens the fixed `firecracker` and `jailer` leaves with `O_NOFOLLOW|O_NONBLOCK` relative to that retained version-directory descriptor. The nonblocking open prevents an unsafe special-file leaf from stalling root before type admission. Each descriptor must identify a one-link `root:root` regular file of exact mode `0755`, with a diff --git a/tasks/todo.md b/tasks/todo.md index e5cb79c5..f0310ce0 100644 --- a/tasks/todo.md +++ b/tasks/todo.md @@ -219,6 +219,8 @@ under `internal/service/jailer/`. No handler, transport, CLI, API, core-domain, identity/type/machine/program-header policy, no untrusted allocation, and no executable loading or execution. - [x] Bind ELF admission to the bounded actual file size and reject a truncated declared program-header table. - [ ] Exact Firecracker/Jailer bytes and a root manifest install atomically and durably. + - [ ] Create or open only the fixed write-side `mvmctl/binaries` store components relative to pinned safe ancestors; + enforce exact root ownership/mode and fsync newly observed child and parent directories before staging bytes. - [x] Freeze the strict schema-v1 manifest contract, 4 KiB record bound, typed SHA-256 fields, and 120-byte through 64 MiB executable-size policy; the fixed store leaf is `manifest.json`. - [x] Implement and verify the private manifest codec and release-identity derivation. From e3d2e04ab407cbabf96f2d26c3eaec3cf200365a Mon Sep 17 00:00:00 2001 From: AlanD20 Date: Sat, 29 Aug 2026 01:46:16 +0200 Subject: [PATCH 0080/1457] feat: create trusted release store base --- .../jailer/trusted_release_store_fs.go | 155 ++++++++- .../jailer/trusted_release_store_write.go | 30 ++ .../trusted_release_store_write_test.go | 324 ++++++++++++++++++ 3 files changed, 495 insertions(+), 14 deletions(-) create mode 100644 internal/service/jailer/trusted_release_store_write.go create mode 100644 internal/service/jailer/trusted_release_store_write_test.go diff --git a/internal/service/jailer/trusted_release_store_fs.go b/internal/service/jailer/trusted_release_store_fs.go index dc39a40d..c0ba02a1 100644 --- a/internal/service/jailer/trusted_release_store_fs.go +++ b/internal/service/jailer/trusted_release_store_fs.go @@ -19,12 +19,16 @@ const ( ) type trustedReleaseStoreDeps struct { - open func(context.Context, string, int, uint32) (int, error) - openAt func(context.Context, int, string, int, uint32) (int, error) - fstat func(context.Context, int, *unix.Stat_t) error - read func(context.Context, int, []byte) (int, error) - pread func(context.Context, int, []byte, int64) (int, error) - close func(context.Context, int) error + open func(context.Context, string, int, uint32) (int, error) + openAt func(context.Context, int, string, int, uint32) (int, error) + mkdirAt func(context.Context, int, string, uint32) error + fstat func(context.Context, int, *unix.Stat_t) error + fchown func(context.Context, int, int, int) error + fchmod func(context.Context, int, uint32) error + read func(context.Context, int, []byte) (int, error) + pread func(context.Context, int, []byte, int64) (int, error) + fsync func(context.Context, int) error + close func(context.Context, int) error } type trustedReleaseStorePolicy struct { @@ -58,6 +62,15 @@ func openTrustedReleaseStoreForRead( ctx context.Context, deps trustedReleaseStoreDeps, policy trustedReleaseStorePolicy, +) (_ *trustedReleaseStore, returnErr error) { + return openTrustedReleaseStore(ctx, deps, policy, false) +} + +func openTrustedReleaseStore( + ctx context.Context, + deps trustedReleaseStoreDeps, + policy trustedReleaseStorePolicy, + allowCreateManaged bool, ) (_ *trustedReleaseStore, returnErr error) { if err := ctx.Err(); err != nil { return nil, trustedReleaseStoreError("open trusted release store", err) @@ -93,17 +106,20 @@ func openTrustedReleaseStoreForRead( parentFD := rootFD for index, name := range components { - if err := ctx.Err(); err != nil { - return nil, trustedReleaseStoreError("open trusted release store component", err) - } - fd, openErr := deps.openAt(ctx, parentFD, name, trustedReleaseStoreDirectoryFlags, 0) + managed := index >= 2 + fd, openErr := openTrustedReleaseStoreComponent( + ctx, + deps, + parentFD, + name, + policy, + managed, + allowCreateManaged && managed, + ) if openErr != nil { - return nil, classifyTrustedReleaseStoreOpenError(name, openErr, index >= 2) + return nil, openErr } retained = append(retained, fd) - if err := verifyTrustedReleaseStoreDirectory(ctx, deps, fd, name, policy, index >= 2); err != nil { - return nil, err - } parentFD = fd } @@ -117,6 +133,105 @@ func openTrustedReleaseStoreForRead( return store, nil } +func openTrustedReleaseStoreComponent( + ctx context.Context, + deps trustedReleaseStoreDeps, + parentFD int, + name string, + policy trustedReleaseStorePolicy, + managed bool, + allowCreate bool, +) (int, error) { + if err := ctx.Err(); err != nil { + return -1, trustedReleaseStoreError("open trusted release store component "+name, err) + } + fd, err := deps.openAt(ctx, parentFD, name, trustedReleaseStoreDirectoryFlags, 0) + if err == nil { + if verifyErr := verifyTrustedReleaseStoreDirectory(ctx, deps, fd, name, policy, managed); verifyErr != nil { + return -1, closeRejectedTrustedReleaseStoreDirectory(ctx, deps, fd, name, verifyErr) + } + return fd, nil + } + if !errors.Is(err, unix.ENOENT) || !allowCreate { + return -1, classifyTrustedReleaseStoreOpenError(name, err, managed) + } + + mkdirErr := deps.mkdirAt(ctx, parentFD, name, trustedReleaseStoreDirectoryMode) + created := mkdirErr == nil + if mkdirErr != nil && !errors.Is(mkdirErr, unix.EEXIST) { + return -1, trustedReleaseStoreError("create trusted release store directory "+name, mkdirErr) + } + if err := ctx.Err(); err != nil { + return -1, trustedReleaseStoreError("open created trusted release store directory "+name, err) + } + openedFD, err := deps.openAt(ctx, parentFD, name, trustedReleaseStoreDirectoryFlags, 0) + if err != nil { + if errors.Is(err, unix.ELOOP) || errors.Is(err, unix.ENOTDIR) { + return -1, trustedReleaseStoreUntrusted( + "created trusted release store directory "+name+" is unsafe", + err, + ) + } + return -1, trustedReleaseStoreError("open created trusted release store directory "+name, err) + } + if created { + if err := deps.fchown(ctx, openedFD, int(policy.expectedUID), int(policy.expectedGID)); err != nil { + return -1, closeRejectedTrustedReleaseStoreDirectory( + ctx, + deps, + openedFD, + name, + trustedReleaseStoreError("set trusted release store directory owner "+name, err), + ) + } + if err := deps.fchmod(ctx, openedFD, trustedReleaseStoreDirectoryMode); err != nil { + return -1, closeRejectedTrustedReleaseStoreDirectory( + ctx, + deps, + openedFD, + name, + trustedReleaseStoreError("set trusted release store directory mode "+name, err), + ) + } + } + if err := verifyTrustedReleaseStoreDirectory(ctx, deps, openedFD, name, policy, true); err != nil { + return -1, closeRejectedTrustedReleaseStoreDirectory(ctx, deps, openedFD, name, err) + } + if err := deps.fsync(ctx, openedFD); err != nil { + return -1, closeRejectedTrustedReleaseStoreDirectory( + ctx, + deps, + openedFD, + name, + trustedReleaseStoreError("sync created trusted release store directory "+name, err), + ) + } + if err := deps.fsync(ctx, parentFD); err != nil { + return -1, closeRejectedTrustedReleaseStoreDirectory( + ctx, + deps, + openedFD, + name, + trustedReleaseStoreError("sync parent of trusted release store directory "+name, err), + ) + } + return openedFD, nil +} + +func closeRejectedTrustedReleaseStoreDirectory( + ctx context.Context, + deps trustedReleaseStoreDeps, + fd int, + name string, + primary error, +) error { + return appendTrustedReleaseStoreError( + primary, + "close rejected trusted release store directory "+name, + deps.close(context.WithoutCancel(ctx), fd), + ) +} + func (store *trustedReleaseStore) openInstalledSlot( ctx context.Context, slot releaseSlot, @@ -346,15 +461,27 @@ func realTrustedReleaseStoreDeps() trustedReleaseStoreDeps { openAt: func(_ context.Context, parentFD int, name string, flags int, mode uint32) (int, error) { return unix.Openat(parentFD, name, flags, mode) }, + mkdirAt: func(_ context.Context, parentFD int, name string, mode uint32) error { + return unix.Mkdirat(parentFD, name, mode) + }, fstat: func(_ context.Context, fd int, stat *unix.Stat_t) error { return unix.Fstat(fd, stat) }, + fchown: func(_ context.Context, fd, uid, gid int) error { + return unix.Fchown(fd, uid, gid) + }, + fchmod: func(_ context.Context, fd int, mode uint32) error { + return unix.Fchmod(fd, mode) + }, read: func(_ context.Context, fd int, value []byte) (int, error) { return unix.Read(fd, value) }, pread: func(_ context.Context, fd int, value []byte, offset int64) (int, error) { return unix.Pread(fd, value, offset) }, + fsync: func(_ context.Context, fd int) error { + return unix.Fsync(fd) + }, close: func(_ context.Context, fd int) error { return unix.Close(fd) }, diff --git a/internal/service/jailer/trusted_release_store_write.go b/internal/service/jailer/trusted_release_store_write.go new file mode 100644 index 00000000..de16bf94 --- /dev/null +++ b/internal/service/jailer/trusted_release_store_write.go @@ -0,0 +1,30 @@ +package jailer + +import "context" + +type trustedReleaseStoreWriteLease struct { + store *trustedReleaseStore +} + +// CRITICAL: This capability creates only the fixed managed store base. Architecture/version slots and leaves remain +// separate typed operations so a caller cannot turn base preparation into a generic root filesystem write primitive. +func openTrustedReleaseStoreForWrite( + ctx context.Context, + deps trustedReleaseStoreDeps, + policy trustedReleaseStorePolicy, +) (*trustedReleaseStoreWriteLease, error) { + store, err := openTrustedReleaseStore(ctx, deps, policy, true) + if err != nil { + return nil, err + } + return &trustedReleaseStoreWriteLease{store: store}, nil +} + +func (lease *trustedReleaseStoreWriteLease) Release(ctx context.Context) error { + if lease == nil || lease.store == nil { + return nil + } + err := lease.store.Release(ctx) + lease.store = nil + return err +} diff --git a/internal/service/jailer/trusted_release_store_write_test.go b/internal/service/jailer/trusted_release_store_write_test.go new file mode 100644 index 00000000..4942db3f --- /dev/null +++ b/internal/service/jailer/trusted_release_store_write_test.go @@ -0,0 +1,324 @@ +package jailer + +import ( + "context" + "errors" + "os" + "path/filepath" + "sync/atomic" + "testing" + + "github.com/google/go-cmp/cmp" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + "golang.org/x/sys/unix" + + "mvmctl/pkg/errs" +) + +// Rationale: first installation may create only the two fixed managed store components. The write capability must pin +// the exact resulting directories without creating a release architecture or version as an incidental side effect. +func TestOpenTrustedReleaseStoreForWriteCreatesFixedManagedBase(t *testing.T) { + t.Parallel() + + fixture := newTrustedReleaseStoreFixture(t) + require.NoError(t, os.RemoveAll(fixture.mvmctlPath)) + lease, err := openTrustedReleaseStoreForWrite(t.Context(), fixture.deps, fixture.policy) + require.NoError(t, err) + + assertExactMode(t, fixture.mvmctlPath, 0700) + assertExactMode(t, fixture.binariesPath, 0700) + assert.NoDirExists(t, fixture.architecturePath) + var pinned unix.Stat_t + require.NoError(t, unix.Fstat(lease.store.binariesFD, &pinned)) + assert.Equal(t, fileInode(t, fixture.binariesPath), pinned.Ino) + require.NoError(t, lease.Release(t.Context())) + require.NoError(t, lease.Release(t.Context())) +} + +func TestOpenTrustedReleaseStoreForWriteDoesNotCreateAncestors(t *testing.T) { + t.Parallel() + + tests := map[string]string{ + "var": "var", + "lib": "lib", + } + for name, level := range tests { + t.Run(name, func(t *testing.T) { + t.Parallel() + + fixture := newTrustedReleaseStoreFixture(t) + require.NoError(t, os.RemoveAll(fixture.pathForLevel(level))) + lease, err := openTrustedReleaseStoreForWrite(t.Context(), fixture.deps, fixture.policy) + assert.Nil(t, lease) + require.Error(t, err) + assert.Equal(t, errs.CodeBinaryTrustedInstallFailed, errs.AsDomainError(err).Code) + }) + } +} + +func TestOpenTrustedReleaseStoreForWriteRejectsUnsafeManagedBase(t *testing.T) { + t.Parallel() + + tests := map[string]func(*testing.T, *trustedReleaseStoreFixture){ + "mvmctl symlink": func(t *testing.T, fixture *trustedReleaseStoreFixture) { + require.NoError(t, os.RemoveAll(fixture.mvmctlPath)) + target := filepath.Join(fixture.root, "attacker") + require.NoError(t, os.Mkdir(target, 0700)) + require.NoError(t, os.Symlink(target, fixture.mvmctlPath)) + }, + "binaries symlink": func(t *testing.T, fixture *trustedReleaseStoreFixture) { + require.NoError(t, os.RemoveAll(fixture.binariesPath)) + target := filepath.Join(fixture.root, "attacker") + require.NoError(t, os.Mkdir(target, 0700)) + require.NoError(t, os.Symlink(target, fixture.binariesPath)) + }, + "mvmctl wrong mode": func(t *testing.T, fixture *trustedReleaseStoreFixture) { + require.NoError(t, os.Chmod(fixture.mvmctlPath, 0755)) + }, + "binaries wrong mode": func(t *testing.T, fixture *trustedReleaseStoreFixture) { + require.NoError(t, os.Chmod(fixture.binariesPath, 0750)) + }, + } + for name, mutate := range tests { + t.Run(name, func(t *testing.T) { + t.Parallel() + + fixture := newTrustedReleaseStoreFixture(t) + mutate(t, fixture) + lease, err := openTrustedReleaseStoreForWrite(t.Context(), fixture.deps, fixture.policy) + assert.Nil(t, lease) + require.Error(t, err) + assert.Equal(t, errs.CodeBinaryUntrusted, errs.AsDomainError(err).Code) + }) + } +} + +func TestOpenTrustedReleaseStoreForWriteTreatsEEXISTAsConcurrentCreation(t *testing.T) { + t.Parallel() + + fixture := newTrustedReleaseStoreFixture(t) + require.NoError(t, os.RemoveAll(fixture.mvmctlPath)) + realMkdirAt := fixture.deps.mkdirAt + fixture.deps.mkdirAt = func(ctx context.Context, parentFD int, name string, mode uint32) error { + require.NoError(t, realMkdirAt(ctx, parentFD, name, mode)) + return unix.EEXIST + } + fixture.deps.fchown = func(context.Context, int, int, int) error { + return errors.New("must not chown a concurrently created store directory") + } + fixture.deps.fchmod = func(context.Context, int, uint32) error { + return errors.New("must not chmod a concurrently created store directory") + } + + lease, err := openTrustedReleaseStoreForWrite(t.Context(), fixture.deps, fixture.policy) + require.NoError(t, err) + assertExactMode(t, fixture.mvmctlPath, 0700) + assertExactMode(t, fixture.binariesPath, 0700) + require.NoError(t, lease.Release(t.Context())) +} + +func TestOpenTrustedReleaseStoreForWriteSetsCreatedMetadata(t *testing.T) { + t.Parallel() + + fixture := newTrustedReleaseStoreFixture(t) + require.NoError(t, os.RemoveAll(fixture.mvmctlPath)) + realFchown := fixture.deps.fchown + realFchmod := fixture.deps.fchmod + var chownCalls atomic.Int32 + var chmodCalls atomic.Int32 + fixture.deps.fchown = func(ctx context.Context, fd, uid, gid int) error { + chownCalls.Add(1) + assert.Equal(t, int(fixture.policy.expectedUID), uid) + assert.Equal(t, int(fixture.policy.expectedGID), gid) + return realFchown(ctx, fd, uid, gid) + } + fixture.deps.fchmod = func(ctx context.Context, fd int, mode uint32) error { + chmodCalls.Add(1) + assert.Equal(t, trustedReleaseStoreDirectoryMode, mode) + return realFchmod(ctx, fd, mode) + } + + lease, err := openTrustedReleaseStoreForWrite(t.Context(), fixture.deps, fixture.policy) + require.NoError(t, err) + assert.Equal(t, int32(2), chownCalls.Load()) + assert.Equal(t, int32(2), chmodCalls.Load()) + require.NoError(t, lease.Release(t.Context())) +} + +func TestOpenTrustedReleaseStoreForWriteRejectsCreationFailures(t *testing.T) { + t.Parallel() + + tests := map[string]func(*trustedReleaseStoreDeps){ + "mkdir": func(deps *trustedReleaseStoreDeps) { + deps.mkdirAt = func(context.Context, int, string, uint32) error { return unix.EIO } + }, + "open created directory": func(deps *trustedReleaseStoreDeps) { + realOpenAt := deps.openAt + var mvmctlCalls int + deps.openAt = func(ctx context.Context, parentFD int, name string, flags int, mode uint32) (int, error) { + if name == "mvmctl" { + mvmctlCalls++ + if mvmctlCalls == 2 { + return -1, unix.EIO + } + } + return realOpenAt(ctx, parentFD, name, flags, mode) + } + }, + "fchown": func(deps *trustedReleaseStoreDeps) { + deps.fchown = func(context.Context, int, int, int) error { return unix.EIO } + }, + "fchmod": func(deps *trustedReleaseStoreDeps) { + deps.fchmod = func(context.Context, int, uint32) error { return unix.EIO } + }, + "fsync": func(deps *trustedReleaseStoreDeps) { + deps.fsync = func(context.Context, int) error { return unix.EIO } + }, + } + for name, inject := range tests { + t.Run(name, func(t *testing.T) { + t.Parallel() + + fixture := newTrustedReleaseStoreFixture(t) + require.NoError(t, os.RemoveAll(fixture.mvmctlPath)) + inject(&fixture.deps) + lease, err := openTrustedReleaseStoreForWrite(t.Context(), fixture.deps, fixture.policy) + assert.Nil(t, lease) + require.Error(t, err) + assert.ErrorIs(t, err, unix.EIO) + assert.Equal(t, errs.CodeBinaryTrustedInstallFailed, errs.AsDomainError(err).Code) + }) + } +} + +func TestOpenTrustedReleaseStoreForWriteDoesNotRepairConcurrentUnsafeDirectory(t *testing.T) { + t.Parallel() + + fixture := newTrustedReleaseStoreFixture(t) + require.NoError(t, os.RemoveAll(fixture.mvmctlPath)) + realMkdirAt := fixture.deps.mkdirAt + fixture.deps.mkdirAt = func(ctx context.Context, parentFD int, name string, mode uint32) error { + require.NoError(t, realMkdirAt(ctx, parentFD, name, mode)) + if name == "mvmctl" { + require.NoError(t, os.Chmod(fixture.mvmctlPath, 0755)) + } + return unix.EEXIST + } + + lease, err := openTrustedReleaseStoreForWrite(t.Context(), fixture.deps, fixture.policy) + assert.Nil(t, lease) + require.Error(t, err) + assert.Equal(t, errs.CodeBinaryUntrusted, errs.AsDomainError(err).Code) + assertExactMode(t, fixture.mvmctlPath, 0755) +} + +func TestOpenTrustedReleaseStoreForWriteSyncsNewChildAndParent(t *testing.T) { + t.Parallel() + + fixture := newTrustedReleaseStoreFixture(t) + require.NoError(t, os.RemoveAll(fixture.mvmctlPath)) + fdNames := make(map[int]string) + realOpen := fixture.deps.open + fixture.deps.open = func(ctx context.Context, name string, flags int, mode uint32) (int, error) { + fd, err := realOpen(ctx, name, flags, mode) + if err == nil { + fdNames[fd] = "root" + } + return fd, err + } + realOpenAt := fixture.deps.openAt + fixture.deps.openAt = func( + ctx context.Context, + parentFD int, + name string, + flags int, + mode uint32, + ) (int, error) { + fd, err := realOpenAt(ctx, parentFD, name, flags, mode) + if err == nil { + fdNames[fd] = name + } + return fd, err + } + realFsync := fixture.deps.fsync + synced := make([]string, 0, 4) + fixture.deps.fsync = func(ctx context.Context, fd int) error { + synced = append(synced, fdNames[fd]) + return realFsync(ctx, fd) + } + + lease, err := openTrustedReleaseStoreForWrite(t.Context(), fixture.deps, fixture.policy) + require.NoError(t, err) + if diff := cmp.Diff([]string{"mvmctl", "lib", "binaries", "mvmctl"}, synced); diff != "" { + t.Errorf("trusted store creation sync order mismatch (-want +got):\n%s", diff) + } + require.NoError(t, lease.Release(t.Context())) +} + +func TestOpenTrustedReleaseStoreForWriteRejectsReplacementAfterMkdir(t *testing.T) { + t.Parallel() + + fixture := newTrustedReleaseStoreFixture(t) + require.NoError(t, os.RemoveAll(fixture.mvmctlPath)) + realMkdirAt := fixture.deps.mkdirAt + fixture.deps.mkdirAt = func(ctx context.Context, parentFD int, name string, mode uint32) error { + err := realMkdirAt(ctx, parentFD, name, mode) + if err == nil && name == "mvmctl" { + require.NoError(t, os.Rename(fixture.mvmctlPath, fixture.mvmctlPath+".created")) + require.NoError(t, os.Symlink(fixture.mvmctlPath+".created", fixture.mvmctlPath)) + } + return err + } + + lease, err := openTrustedReleaseStoreForWrite(t.Context(), fixture.deps, fixture.policy) + assert.Nil(t, lease) + require.Error(t, err) + assert.Equal(t, errs.CodeBinaryUntrusted, errs.AsDomainError(err).Code) +} + +func TestOpenTrustedReleaseStoreForWriteHonorsCancellationAfterCreation(t *testing.T) { + t.Parallel() + + fixture := newTrustedReleaseStoreFixture(t) + require.NoError(t, os.RemoveAll(fixture.mvmctlPath)) + ctx, cancel := context.WithCancel(t.Context()) + realMkdirAt := fixture.deps.mkdirAt + fixture.deps.mkdirAt = func(ctx context.Context, parentFD int, name string, mode uint32) error { + err := realMkdirAt(ctx, parentFD, name, mode) + cancel() + return err + } + cleanupWasCanceled := false + realClose := fixture.deps.close + fixture.deps.close = func(closeCtx context.Context, fd int) error { + cleanupWasCanceled = cleanupWasCanceled || closeCtx.Err() != nil + return realClose(closeCtx, fd) + } + + lease, err := openTrustedReleaseStoreForWrite(ctx, fixture.deps, fixture.policy) + assert.Nil(t, lease) + require.Error(t, err) + assert.ErrorIs(t, err, context.Canceled) + assert.Equal(t, errs.CodeBinaryTrustedInstallFailed, errs.AsDomainError(err).Code) + assert.False(t, cleanupWasCanceled) +} + +func TestOpenTrustedReleaseStoreForWritePreservesPrimaryErrorWhenCleanupFails(t *testing.T) { + t.Parallel() + + fixture := newTrustedReleaseStoreFixture(t) + require.NoError(t, os.Chmod(fixture.mvmctlPath, 0755)) + realClose := fixture.deps.close + fixture.deps.close = func(ctx context.Context, fd int) error { + return errors.Join(realClose(ctx, fd), unix.EIO) + } + + lease, err := openTrustedReleaseStoreForWrite(t.Context(), fixture.deps, fixture.policy) + assert.Nil(t, lease) + require.Error(t, err) + domainErr := errs.AsDomainError(err) + require.NotNil(t, domainErr) + assert.Equal(t, errs.CodeBinaryUntrusted, domainErr.Code) + assert.ErrorIs(t, err, unix.EIO) +} From e5e6b711ed1de598bac9128d4e779baecf5525b5 Mon Sep 17 00:00:00 2001 From: AlanD20 Date: Sat, 29 Aug 2026 01:46:34 +0200 Subject: [PATCH 0081/1457] docs: record trusted release store creation --- .../0016-root-owned-single-binary-privileged-dispatch.md | 6 ++++-- tasks/todo.md | 2 +- 2 files changed, 5 insertions(+), 3 deletions(-) diff --git a/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md b/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md index 83cd1228..6e3dd1d8 100644 --- a/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md +++ b/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md @@ -335,8 +335,10 @@ then opens the two fixed executable leaves nonblocking, verifies their exact met closed ELF policy, read bounds, and pre/post-read stability, and retains the same descriptors at offset zero. The private release authority now acquires the release-slot lease before those reads and returns one prepared value owning the lease, pinned store chain, manifest-derived identity, and verified descriptors with checked reverse cleanup. These foundations -are not yet wired to the privileged request or legacy installer. Bounded archive transport and extraction, the complete -member allowlist, manifest writing, and atomic trusted-store installation remain Task 6 work. +are not yet wired to the privileged request or legacy installer. The write-side store foundation also creates or opens +only the fixed managed `mvmctl/binaries` base through pinned ancestors, corrects metadata only on directories it created, +and durably syncs each newly observed child and parent. Bounded archive staging and extraction, the complete member +allowlist, architecture/version creation, manifest writing, and atomic trusted-store installation remain Task 6 work. ### Paths, process identity, and runtime state diff --git a/tasks/todo.md b/tasks/todo.md index f0310ce0..7926f1b3 100644 --- a/tasks/todo.md +++ b/tasks/todo.md @@ -219,7 +219,7 @@ under `internal/service/jailer/`. No handler, transport, CLI, API, core-domain, identity/type/machine/program-header policy, no untrusted allocation, and no executable loading or execution. - [x] Bind ELF admission to the bounded actual file size and reject a truncated declared program-header table. - [ ] Exact Firecracker/Jailer bytes and a root manifest install atomically and durably. - - [ ] Create or open only the fixed write-side `mvmctl/binaries` store components relative to pinned safe ancestors; + - [x] Create or open only the fixed write-side `mvmctl/binaries` store components relative to pinned safe ancestors; enforce exact root ownership/mode and fsync newly observed child and parent directories before staging bytes. - [x] Freeze the strict schema-v1 manifest contract, 4 KiB record bound, typed SHA-256 fields, and 120-byte through 64 MiB executable-size policy; the fixed store leaf is `manifest.json`. From 3413d1db6ddc53926f510f051e76ff0833ca0ef2 Mon Sep 17 00:00:00 2001 From: AlanD20 Date: Sat, 29 Aug 2026 01:48:23 +0200 Subject: [PATCH 0082/1457] docs: define anonymous release archive staging --- .../0016-root-owned-single-binary-privileged-dispatch.md | 7 +++++++ tasks/todo.md | 2 ++ 2 files changed, 9 insertions(+) diff --git a/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md b/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md index 6e3dd1d8..86907f17 100644 --- a/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md +++ b/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md @@ -301,6 +301,13 @@ observed component is admitted, both that directory and its parent are fsynced b replacement, unsafe metadata, cancellation, and durability failures close all acquired descriptors in reverse order and fail closed. This foundation creates no architecture/version slot and writes no archive or executable bytes. +Archive staging uses an anonymous regular file created on the pinned trusted-store filesystem with +`O_TMPFILE|O_EXCL|O_RDWR|O_CLOEXEC` relative to the `binaries` descriptor. It has no pathname and cannot later be linked +into the namespace. The descriptor must be `root:root`, exact mode `0600`, link count zero, and initially empty before it +can receive bytes. A filesystem without these anonymous-file semantics fails closed; there is no fallback to `/tmp`, a +visible random pathname, a replayable second stream, or a large in-memory archive. The stage is a private checked lease +and is not itself an installed release or executable authority. + Executable admission opens the fixed `firecracker` and `jailer` leaves with `O_NOFOLLOW|O_NONBLOCK` relative to that retained version-directory descriptor. The nonblocking open prevents an unsafe special-file leaf from stalling root before type admission. Each descriptor must identify a one-link `root:root` regular file of exact mode `0755`, with a diff --git a/tasks/todo.md b/tasks/todo.md index 7926f1b3..6669bf5f 100644 --- a/tasks/todo.md +++ b/tasks/todo.md @@ -221,6 +221,8 @@ under `internal/service/jailer/`. No handler, transport, CLI, API, core-domain, - [ ] Exact Firecracker/Jailer bytes and a root manifest install atomically and durably. - [x] Create or open only the fixed write-side `mvmctl/binaries` store components relative to pinned safe ancestors; enforce exact root ownership/mode and fsync newly observed child and parent directories before staging bytes. + - [ ] Create a private anonymous `O_TMPFILE|O_EXCL` archive-stage lease on the pinned trusted-store filesystem with + exact empty, unlinked, root-owned `0600` metadata and no pathname, memory, or alternate-filesystem fallback. - [x] Freeze the strict schema-v1 manifest contract, 4 KiB record bound, typed SHA-256 fields, and 120-byte through 64 MiB executable-size policy; the fixed store leaf is `manifest.json`. - [x] Implement and verify the private manifest codec and release-identity derivation. From 5dd93c21d733bc193ce89705f15ff221fec28a18 Mon Sep 17 00:00:00 2001 From: AlanD20 Date: Sat, 29 Aug 2026 01:52:56 +0200 Subject: [PATCH 0083/1457] feat: stage trusted release archives anonymously --- .../jailer/trusted_release_archive_stage.go | 109 +++++++ .../trusted_release_archive_stage_test.go | 286 ++++++++++++++++++ 2 files changed, 395 insertions(+) create mode 100644 internal/service/jailer/trusted_release_archive_stage.go create mode 100644 internal/service/jailer/trusted_release_archive_stage_test.go diff --git a/internal/service/jailer/trusted_release_archive_stage.go b/internal/service/jailer/trusted_release_archive_stage.go new file mode 100644 index 00000000..afce7b2d --- /dev/null +++ b/internal/service/jailer/trusted_release_archive_stage.go @@ -0,0 +1,109 @@ +package jailer + +import ( + "context" + + "golang.org/x/sys/unix" +) + +const ( + trustedReleaseArchiveStageFlags = unix.O_TMPFILE | unix.O_EXCL | unix.O_RDWR | unix.O_CLOEXEC + trustedReleaseArchiveStageMode = uint32(0600) +) + +type trustedReleaseArchiveStage struct { + deps trustedReleaseStoreDeps + fd int +} + +// CRITICAL: The stage is anonymous, unlinkable, and created on the pinned trusted-store filesystem. No path is +// generated or accepted, and no fallback can move the bytes outside this descriptor authority. +func (lease *trustedReleaseStoreWriteLease) createArchiveStage( + ctx context.Context, +) (_ *trustedReleaseArchiveStage, returnErr error) { + if lease == nil || lease.store == nil || lease.store.binariesFD < 0 || len(lease.store.retained) == 0 { + return nil, trustedReleaseStoreError("trusted release store write lease is not active", nil) + } + if err := ctx.Err(); err != nil { + return nil, trustedReleaseStoreError("create trusted release archive stage", err) + } + + fd, err := lease.store.deps.openAt( + ctx, + lease.store.binariesFD, + ".", + trustedReleaseArchiveStageFlags, + trustedReleaseArchiveStageMode, + ) + if err != nil { + return nil, trustedReleaseStoreError("create anonymous trusted release archive stage", err) + } + defer func() { + if returnErr == nil { + return + } + returnErr = appendTrustedReleaseStoreError( + returnErr, + "close rejected trusted release archive stage", + lease.store.deps.close(context.WithoutCancel(ctx), fd), + ) + }() + + if err := ctx.Err(); err != nil { + return nil, trustedReleaseStoreError("inspect trusted release archive stage", err) + } + if err := verifyTrustedReleaseArchiveStage(ctx, lease.store, fd, false); err != nil { + return nil, err + } + if err := lease.store.deps.fchown( + ctx, + fd, + int(lease.store.policy.expectedUID), + int(lease.store.policy.expectedGID), + ); err != nil { + return nil, trustedReleaseStoreError("set trusted release archive stage owner", err) + } + if err := lease.store.deps.fchmod(ctx, fd, trustedReleaseArchiveStageMode); err != nil { + return nil, trustedReleaseStoreError("set trusted release archive stage mode", err) + } + if err := verifyTrustedReleaseArchiveStage(ctx, lease.store, fd, true); err != nil { + return nil, err + } + + return &trustedReleaseArchiveStage{deps: lease.store.deps, fd: fd}, nil +} + +func verifyTrustedReleaseArchiveStage( + ctx context.Context, + store *trustedReleaseStore, + fd int, + requireIdentity bool, +) error { + if err := ctx.Err(); err != nil { + return trustedReleaseStoreError("inspect trusted release archive stage", err) + } + var stat unix.Stat_t + if err := store.deps.fstat(ctx, fd, &stat); err != nil { + return trustedReleaseStoreError("inspect trusted release archive stage", err) + } + if stat.Mode&unix.S_IFMT != unix.S_IFREG || stat.Nlink != 0 || stat.Size != 0 { + return trustedReleaseStoreError("trusted release archive stage has unsafe filesystem metadata", nil) + } + if requireIdentity && (stat.Uid != store.policy.expectedUID || stat.Gid != store.policy.expectedGID || + stat.Mode&07777 != trustedReleaseArchiveStageMode) { + return trustedReleaseStoreError("trusted release archive stage has unexpected owner or mode", nil) + } + return nil +} + +func (stage *trustedReleaseArchiveStage) Release(ctx context.Context) error { + if stage == nil || stage.fd < 0 { + return nil + } + err := stage.deps.close(context.WithoutCancel(ctx), stage.fd) + stage.fd = -1 + if err != nil { + return trustedReleaseStoreError("release trusted release archive stage", err) + } + return nil +} diff --git a/internal/service/jailer/trusted_release_archive_stage_test.go b/internal/service/jailer/trusted_release_archive_stage_test.go new file mode 100644 index 00000000..4f83368b --- /dev/null +++ b/internal/service/jailer/trusted_release_archive_stage_test.go @@ -0,0 +1,286 @@ +package jailer + +import ( + "context" + "errors" + "os" + "testing" + + "github.com/google/go-cmp/cmp" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + "golang.org/x/sys/unix" + + "mvmctl/pkg/errs" +) + +// Rationale: archive bytes must be retained on the already trusted filesystem without ever publishing a pathname. The +// O_EXCL form of O_TMPFILE also prevents a later link operation from turning this transport stage into a store leaf. +func TestTrustedReleaseStoreWriteLeaseCreatesAnonymousArchiveStage(t *testing.T) { + t.Parallel() + + fixture := newTrustedReleaseStoreFixture(t) + writer, err := openTrustedReleaseStoreForWrite(t.Context(), fixture.deps, fixture.policy) + require.NoError(t, err) + realOpenAt := writer.store.deps.openAt + var observedName string + var observedFlags int + var observedMode uint32 + writer.store.deps.openAt = func( + ctx context.Context, + parentFD int, + name string, + flags int, + mode uint32, + ) (int, error) { + if name == "." { + observedName = name + observedFlags = flags + observedMode = mode + } + return realOpenAt(ctx, parentFD, name, flags, mode) + } + before := trustedReleaseDirectoryEntryNames(t, fixture.binariesPath) + + stage, err := writer.createArchiveStage(t.Context()) + require.NoError(t, err) + assert.Equal(t, ".", observedName) + assert.Equal(t, trustedReleaseArchiveStageFlags, observedFlags) + assert.Equal(t, trustedReleaseArchiveStageMode, observedMode) + var stat unix.Stat_t + require.NoError(t, unix.Fstat(stage.fd, &stat)) + assert.Equal(t, uint32(unix.S_IFREG)|trustedReleaseArchiveStageMode, stat.Mode) + assert.Equal(t, fixture.policy.expectedUID, stat.Uid) + assert.Equal(t, fixture.policy.expectedGID, stat.Gid) + assert.Equal(t, uint64(0), stat.Nlink) + assert.Equal(t, int64(0), stat.Size) + flags, err := unix.FcntlInt(uintptr(stage.fd), unix.F_GETFD, 0) + require.NoError(t, err) + assert.NotZero(t, flags&unix.FD_CLOEXEC) + offset, err := unix.Seek(stage.fd, 0, unix.SEEK_CUR) + require.NoError(t, err) + assert.Equal(t, int64(0), offset) + after := trustedReleaseDirectoryEntryNames(t, fixture.binariesPath) + if diff := cmp.Diff(before, after); diff != "" { + t.Errorf("anonymous archive stage changed store entries (-before +after):\n%s", diff) + } + + require.NoError(t, writer.Release(t.Context())) + require.NoError(t, unix.Fstat(stage.fd, &stat)) + require.NoError(t, stage.Release(t.Context())) + require.NoError(t, stage.Release(t.Context())) +} + +func TestTrustedReleaseArchiveStageRejectsUnsafeMetadata(t *testing.T) { + t.Parallel() + + tests := map[string]func(*unix.Stat_t){ + "not regular": func(stat *unix.Stat_t) { stat.Mode = uint32(unix.S_IFDIR) | trustedReleaseArchiveStageMode }, + "wrong owner": func(stat *unix.Stat_t) { stat.Uid++ }, + "wrong group": func(stat *unix.Stat_t) { stat.Gid++ }, + "wrong mode": func(stat *unix.Stat_t) { stat.Mode = uint32(unix.S_IFREG) | 0644 }, + "linked": func(stat *unix.Stat_t) { stat.Nlink = 1 }, + "not empty": func(stat *unix.Stat_t) { stat.Size = 1 }, + } + for name, mutate := range tests { + t.Run(name, func(t *testing.T) { + t.Parallel() + + fixture := newTrustedReleaseStoreFixture(t) + writer, err := openTrustedReleaseStoreForWrite(t.Context(), fixture.deps, fixture.policy) + require.NoError(t, err) + t.Cleanup(func() { require.NoError(t, writer.Release(context.Background())) }) + realFstat := writer.store.deps.fstat + writer.store.deps.fstat = func(ctx context.Context, fd int, stat *unix.Stat_t) error { + if err := realFstat(ctx, fd, stat); err != nil { + return err + } + mutate(stat) + return nil + } + + stage, err := writer.createArchiveStage(t.Context()) + assert.Nil(t, stage) + require.Error(t, err) + assert.Equal(t, errs.CodeBinaryTrustedInstallFailed, errs.AsDomainError(err).Code) + }) + } +} + +func TestTrustedReleaseArchiveStageRejectsSetupFailures(t *testing.T) { + t.Parallel() + + tests := map[string]func(*trustedReleaseStoreDeps){ + "anonymous open": func(deps *trustedReleaseStoreDeps) { + realOpenAt := deps.openAt + deps.openAt = func(ctx context.Context, parentFD int, name string, flags int, mode uint32) (int, error) { + if name == "." { + return -1, unix.EOPNOTSUPP + } + return realOpenAt(ctx, parentFD, name, flags, mode) + } + }, + "fchown": func(deps *trustedReleaseStoreDeps) { + deps.fchown = func(context.Context, int, int, int) error { return unix.EIO } + }, + "fchmod": func(deps *trustedReleaseStoreDeps) { + deps.fchmod = func(context.Context, int, uint32) error { return unix.EIO } + }, + "fstat": func(deps *trustedReleaseStoreDeps) { + deps.fstat = func(context.Context, int, *unix.Stat_t) error { return unix.EIO } + }, + } + for name, inject := range tests { + t.Run(name, func(t *testing.T) { + t.Parallel() + + fixture := newTrustedReleaseStoreFixture(t) + writer, err := openTrustedReleaseStoreForWrite(t.Context(), fixture.deps, fixture.policy) + require.NoError(t, err) + t.Cleanup(func() { require.NoError(t, writer.Release(context.Background())) }) + inject(&writer.store.deps) + + stage, err := writer.createArchiveStage(t.Context()) + assert.Nil(t, stage) + require.Error(t, err) + assert.Equal(t, errs.CodeBinaryTrustedInstallFailed, errs.AsDomainError(err).Code) + wantErr := error(unix.EIO) + if name == "anonymous open" { + wantErr = unix.EOPNOTSUPP + } + assert.ErrorIs(t, err, wantErr) + }) + } +} + +func TestTrustedReleaseArchiveStageHonorsCancellationAfterOpen(t *testing.T) { + t.Parallel() + + fixture := newTrustedReleaseStoreFixture(t) + writer, err := openTrustedReleaseStoreForWrite(t.Context(), fixture.deps, fixture.policy) + require.NoError(t, err) + t.Cleanup(func() { require.NoError(t, writer.Release(context.Background())) }) + ctx, cancel := context.WithCancel(t.Context()) + realOpenAt := writer.store.deps.openAt + writer.store.deps.openAt = func( + ctx context.Context, + parentFD int, + name string, + flags int, + mode uint32, + ) (int, error) { + fd, openErr := realOpenAt(ctx, parentFD, name, flags, mode) + if openErr == nil && name == "." { + cancel() + } + return fd, openErr + } + cleanupWasCanceled := false + realClose := writer.store.deps.close + writer.store.deps.close = func(closeCtx context.Context, fd int) error { + cleanupWasCanceled = cleanupWasCanceled || closeCtx.Err() != nil + return realClose(closeCtx, fd) + } + + stage, err := writer.createArchiveStage(ctx) + assert.Nil(t, stage) + require.Error(t, err) + assert.ErrorIs(t, err, context.Canceled) + assert.Equal(t, errs.CodeBinaryTrustedInstallFailed, errs.AsDomainError(err).Code) + assert.False(t, cleanupWasCanceled) +} + +func TestTrustedReleaseArchiveStageRejectsCanceledContextBeforeOpen(t *testing.T) { + t.Parallel() + + fixture := newTrustedReleaseStoreFixture(t) + writer, err := openTrustedReleaseStoreForWrite(t.Context(), fixture.deps, fixture.policy) + require.NoError(t, err) + t.Cleanup(func() { require.NoError(t, writer.Release(context.Background())) }) + ctx, cancel := context.WithCancel(t.Context()) + cancel() + openCalled := false + writer.store.deps.openAt = func(context.Context, int, string, int, uint32) (int, error) { + openCalled = true + return -1, unix.EIO + } + + stage, err := writer.createArchiveStage(ctx) + assert.Nil(t, stage) + require.Error(t, err) + assert.ErrorIs(t, err, context.Canceled) + assert.Equal(t, errs.CodeBinaryTrustedInstallFailed, errs.AsDomainError(err).Code) + assert.False(t, openCalled) +} + +func TestTrustedReleaseArchiveStageRejectsInactiveWriter(t *testing.T) { + t.Parallel() + + var nilWriter *trustedReleaseStoreWriteLease + stage, err := nilWriter.createArchiveStage(t.Context()) + assert.Nil(t, stage) + require.Error(t, err) + assert.Equal(t, errs.CodeBinaryTrustedInstallFailed, errs.AsDomainError(err).Code) + + stage, err = (&trustedReleaseStoreWriteLease{}).createArchiveStage(t.Context()) + assert.Nil(t, stage) + require.Error(t, err) + assert.Equal(t, errs.CodeBinaryTrustedInstallFailed, errs.AsDomainError(err).Code) +} + +func TestTrustedReleaseArchiveStagePreservesPrimaryErrorWhenCloseFails(t *testing.T) { + t.Parallel() + + fixture := newTrustedReleaseStoreFixture(t) + writer, err := openTrustedReleaseStoreForWrite(t.Context(), fixture.deps, fixture.policy) + require.NoError(t, err) + t.Cleanup(func() { require.NoError(t, writer.Release(context.Background())) }) + writer.store.deps.fchmod = func(context.Context, int, uint32) error { return unix.EINVAL } + realClose := writer.store.deps.close + writer.store.deps.close = func(ctx context.Context, fd int) error { + return errors.Join(realClose(ctx, fd), unix.EIO) + } + + stage, err := writer.createArchiveStage(t.Context()) + assert.Nil(t, stage) + require.Error(t, err) + assert.Equal(t, errs.CodeBinaryTrustedInstallFailed, errs.AsDomainError(err).Code) + assert.ErrorIs(t, err, unix.EINVAL) + assert.ErrorIs(t, err, unix.EIO) + writer.store.deps.close = realClose +} + +func TestTrustedReleaseArchiveStageReleaseReportsCloseFailureOnce(t *testing.T) { + t.Parallel() + + fixture := newTrustedReleaseStoreFixture(t) + writer, err := openTrustedReleaseStoreForWrite(t.Context(), fixture.deps, fixture.policy) + require.NoError(t, err) + t.Cleanup(func() { require.NoError(t, writer.Release(context.Background())) }) + stage, err := writer.createArchiveStage(t.Context()) + require.NoError(t, err) + realClose := stage.deps.close + stage.deps.close = func(ctx context.Context, fd int) error { + return errors.Join(realClose(ctx, fd), unix.EIO) + } + + err = stage.Release(t.Context()) + require.Error(t, err) + assert.ErrorIs(t, err, unix.EIO) + assert.Equal(t, errs.CodeBinaryTrustedInstallFailed, errs.AsDomainError(err).Code) + assert.Equal(t, -1, stage.fd) + assert.NoError(t, stage.Release(t.Context())) +} + +// trustedReleaseDirectoryEntryNames returns observable store entries without interpreting any entry metadata. +func trustedReleaseDirectoryEntryNames(t *testing.T, directory string) []string { + t.Helper() + + entries, err := os.ReadDir(directory) + require.NoError(t, err) + names := make([]string, 0, len(entries)) + for _, entry := range entries { + names = append(names, entry.Name()) + } + return names +} From c680d5ed220f067fc18e242f83f4d1b8e5e57199 Mon Sep 17 00:00:00 2001 From: AlanD20 Date: Sat, 29 Aug 2026 01:53:39 +0200 Subject: [PATCH 0084/1457] docs: record anonymous release archive staging --- .../0016-root-owned-single-binary-privileged-dispatch.md | 6 ++++-- tasks/todo.md | 2 +- 2 files changed, 5 insertions(+), 3 deletions(-) diff --git a/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md b/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md index 86907f17..d3fb6e3e 100644 --- a/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md +++ b/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md @@ -344,8 +344,10 @@ release authority now acquires the release-slot lease before those reads and ret pinned store chain, manifest-derived identity, and verified descriptors with checked reverse cleanup. These foundations are not yet wired to the privileged request or legacy installer. The write-side store foundation also creates or opens only the fixed managed `mvmctl/binaries` base through pinned ancestors, corrects metadata only on directories it created, -and durably syncs each newly observed child and parent. Bounded archive staging and extraction, the complete member -allowlist, architecture/version creation, manifest writing, and atomic trusted-store installation remain Task 6 work. +and durably syncs each newly observed child and parent. It can now create and retain the fixed anonymous archive stage +described above, including fail-closed metadata admission, cancellation-independent cleanup, and cleanup-error +preservation. Bounded streaming and digest verification, extraction, the complete member allowlist, +architecture/version creation, manifest writing, and atomic trusted-store installation remain Task 6 work. ### Paths, process identity, and runtime state diff --git a/tasks/todo.md b/tasks/todo.md index 6669bf5f..1a151489 100644 --- a/tasks/todo.md +++ b/tasks/todo.md @@ -221,7 +221,7 @@ under `internal/service/jailer/`. No handler, transport, CLI, API, core-domain, - [ ] Exact Firecracker/Jailer bytes and a root manifest install atomically and durably. - [x] Create or open only the fixed write-side `mvmctl/binaries` store components relative to pinned safe ancestors; enforce exact root ownership/mode and fsync newly observed child and parent directories before staging bytes. - - [ ] Create a private anonymous `O_TMPFILE|O_EXCL` archive-stage lease on the pinned trusted-store filesystem with + - [x] Create a private anonymous `O_TMPFILE|O_EXCL` archive-stage lease on the pinned trusted-store filesystem with exact empty, unlinked, root-owned `0600` metadata and no pathname, memory, or alternate-filesystem fallback. - [x] Freeze the strict schema-v1 manifest contract, 4 KiB record bound, typed SHA-256 fields, and 120-byte through 64 MiB executable-size policy; the fixed store leaf is `manifest.json`. From 4d8461d86e7be715e768a1a38326b7c176e13f12 Mon Sep 17 00:00:00 2001 From: AlanD20 Date: Sat, 29 Aug 2026 01:56:32 +0200 Subject: [PATCH 0085/1457] docs: define trusted archive stream admission --- .../0016-root-owned-single-binary-privileged-dispatch.md | 8 ++++++++ tasks/todo.md | 2 ++ 2 files changed, 10 insertions(+) diff --git a/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md b/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md index d3fb6e3e..76eb572b 100644 --- a/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md +++ b/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md @@ -308,6 +308,14 @@ can receive bytes. A filesystem without these anonymous-file semantics fails clo visible random pathname, a replayable second stream, or a large in-memory archive. The stage is a private checked lease and is not itself an installed release or executable authority. +Caller-supplied archive admission is a single-use transition on that stage. The declared body is 1 byte through +128 MiB; zero bytes select the separate root-fetch path and never represent an empty archive. Admission uses a fixed +32 KiB buffer and monotonically positioned writes so the descriptor offset remains zero. It consumes exactly the +declared bytes, hashes those same bytes once, and requires stream EOF with a one-byte probe; truncation, trailing data, +or disagreement with the independently fetched digest fails as untrusted input. Acceptance requires file `fsync`, +stable descriptor identity, exact post-write size and metadata, and an unchanged zero offset. Once stream consumption +starts, any failure poisons the stage and only checked release remains valid; failed bytes cannot be retried or parsed. + Executable admission opens the fixed `firecracker` and `jailer` leaves with `O_NOFOLLOW|O_NONBLOCK` relative to that retained version-directory descriptor. The nonblocking open prevents an unsafe special-file leaf from stalling root before type admission. Each descriptor must identify a one-link `root:root` regular file of exact mode `0755`, with a diff --git a/tasks/todo.md b/tasks/todo.md index 1a151489..2c84e97a 100644 --- a/tasks/todo.md +++ b/tasks/todo.md @@ -223,6 +223,8 @@ under `internal/service/jailer/`. No handler, transport, CLI, API, core-domain, enforce exact root ownership/mode and fsync newly observed child and parent directories before staging bytes. - [x] Create a private anonymous `O_TMPFILE|O_EXCL` archive-stage lease on the pinned trusted-store filesystem with exact empty, unlinked, root-owned `0600` metadata and no pathname, memory, or alternate-filesystem fallback. + - [ ] Admit one exact caller-supplied 1-byte through 128-MiB stream into that stage with bounded positioned writes, + authoritative SHA-256 comparison, exact EOF, fsync, stable metadata, zero offset, and fail-closed poisoning. - [x] Freeze the strict schema-v1 manifest contract, 4 KiB record bound, typed SHA-256 fields, and 120-byte through 64 MiB executable-size policy; the fixed store leaf is `manifest.json`. - [x] Implement and verify the private manifest codec and release-identity derivation. From 3e8e22fcaee2241dad9ee8fcbcc2456dbc3174bc Mon Sep 17 00:00:00 2001 From: AlanD20 Date: Sat, 29 Aug 2026 02:04:27 +0200 Subject: [PATCH 0086/1457] feat: admit trusted release archive streams Receive one exact nonzero archive body into the anonymous trusted-store stage with bounded positioned writes, independent SHA-256 authority, exact EOF, fsync, stable metadata, and fail-closed single-use state. --- .../jailer/trusted_release_archive_stage.go | 85 ++- .../trusted_release_archive_stage_test.go | 3 + .../jailer/trusted_release_archive_stream.go | 244 +++++++++ .../trusted_release_archive_stream_test.go | 483 ++++++++++++++++++ .../jailer/trusted_release_store_fs.go | 8 + 5 files changed, 808 insertions(+), 15 deletions(-) create mode 100644 internal/service/jailer/trusted_release_archive_stream.go create mode 100644 internal/service/jailer/trusted_release_archive_stream_test.go diff --git a/internal/service/jailer/trusted_release_archive_stage.go b/internal/service/jailer/trusted_release_archive_stage.go index afce7b2d..b5752d4a 100644 --- a/internal/service/jailer/trusted_release_archive_stage.go +++ b/internal/service/jailer/trusted_release_archive_stage.go @@ -12,8 +12,12 @@ const ( ) type trustedReleaseArchiveStage struct { - deps trustedReleaseStoreDeps - fd int + deps trustedReleaseStoreDeps + policy trustedReleaseStorePolicy + fd int + state trustedReleaseArchiveStageState + sizeBytes uint64 + archiveDigest trustedReleaseArchiveDigest } // CRITICAL: The stage is anonymous, unlinkable, and created on the pinned trusted-store filesystem. No path is @@ -52,7 +56,17 @@ func (lease *trustedReleaseStoreWriteLease) createArchiveStage( if err := ctx.Err(); err != nil { return nil, trustedReleaseStoreError("inspect trusted release archive stage", err) } - if err := verifyTrustedReleaseArchiveStage(ctx, lease.store, fd, false); err != nil { + if _, err := inspectTrustedReleaseArchiveStage( + ctx, + lease.store.deps, + lease.store.policy, + fd, + 0, + false, + ); err != nil { + return nil, err + } + if err := requireTrustedReleaseArchiveStageZeroOffset(ctx, lease.store.deps, fd); err != nil { return nil, err } if err := lease.store.deps.fchown( @@ -66,32 +80,70 @@ func (lease *trustedReleaseStoreWriteLease) createArchiveStage( if err := lease.store.deps.fchmod(ctx, fd, trustedReleaseArchiveStageMode); err != nil { return nil, trustedReleaseStoreError("set trusted release archive stage mode", err) } - if err := verifyTrustedReleaseArchiveStage(ctx, lease.store, fd, true); err != nil { + if _, err := inspectTrustedReleaseArchiveStage( + ctx, + lease.store.deps, + lease.store.policy, + fd, + 0, + true, + ); err != nil { return nil, err } - return &trustedReleaseArchiveStage{deps: lease.store.deps, fd: fd}, nil + return &trustedReleaseArchiveStage{ + deps: lease.store.deps, + policy: lease.store.policy, + fd: fd, + state: trustedReleaseArchiveStageEmpty, + }, nil } -func verifyTrustedReleaseArchiveStage( +func inspectTrustedReleaseArchiveStage( ctx context.Context, - store *trustedReleaseStore, + deps trustedReleaseStoreDeps, + policy trustedReleaseStorePolicy, fd int, + expectedSize uint64, requireIdentity bool, -) error { +) (unix.Stat_t, error) { if err := ctx.Err(); err != nil { - return trustedReleaseStoreError("inspect trusted release archive stage", err) + return unix.Stat_t{}, trustedReleaseStoreError("inspect trusted release archive stage", err) } var stat unix.Stat_t - if err := store.deps.fstat(ctx, fd, &stat); err != nil { - return trustedReleaseStoreError("inspect trusted release archive stage", err) + if err := deps.fstat(ctx, fd, &stat); err != nil { + return unix.Stat_t{}, trustedReleaseStoreError("inspect trusted release archive stage", err) } - if stat.Mode&unix.S_IFMT != unix.S_IFREG || stat.Nlink != 0 || stat.Size != 0 { - return trustedReleaseStoreError("trusted release archive stage has unsafe filesystem metadata", nil) + if stat.Mode&unix.S_IFMT != unix.S_IFREG || stat.Nlink != 0 || stat.Size < 0 || uint64(stat.Size) != expectedSize { + return unix.Stat_t{}, trustedReleaseStoreError( + "trusted release archive stage has unsafe filesystem metadata or size", + nil, + ) } - if requireIdentity && (stat.Uid != store.policy.expectedUID || stat.Gid != store.policy.expectedGID || + if requireIdentity && (stat.Uid != policy.expectedUID || stat.Gid != policy.expectedGID || stat.Mode&07777 != trustedReleaseArchiveStageMode) { - return trustedReleaseStoreError("trusted release archive stage has unexpected owner or mode", nil) + return unix.Stat_t{}, trustedReleaseStoreError( + "trusted release archive stage has unexpected owner or mode", + nil, + ) + } + return stat, nil +} + +func requireTrustedReleaseArchiveStageZeroOffset( + ctx context.Context, + deps trustedReleaseStoreDeps, + fd int, +) error { + if err := ctx.Err(); err != nil { + return trustedReleaseStoreError("inspect trusted release archive stage offset", err) + } + offset, err := deps.seek(ctx, fd, 0, unix.SEEK_CUR) + if err != nil { + return trustedReleaseStoreError("inspect trusted release archive stage offset", err) + } + if offset != 0 { + return trustedReleaseStoreError("trusted release archive stage offset is not zero", nil) } return nil } @@ -102,6 +154,9 @@ func (stage *trustedReleaseArchiveStage) Release(ctx context.Context) error { } err := stage.deps.close(context.WithoutCancel(ctx), stage.fd) stage.fd = -1 + stage.state = trustedReleaseArchiveStageFailed + stage.sizeBytes = 0 + stage.archiveDigest = trustedReleaseArchiveDigest{} if err != nil { return trustedReleaseStoreError("release trusted release archive stage", err) } diff --git a/internal/service/jailer/trusted_release_archive_stage_test.go b/internal/service/jailer/trusted_release_archive_stage_test.go index 4f83368b..8d6c6dfe 100644 --- a/internal/service/jailer/trusted_release_archive_stage_test.go +++ b/internal/service/jailer/trusted_release_archive_stage_test.go @@ -129,6 +129,9 @@ func TestTrustedReleaseArchiveStageRejectsSetupFailures(t *testing.T) { "fstat": func(deps *trustedReleaseStoreDeps) { deps.fstat = func(context.Context, int, *unix.Stat_t) error { return unix.EIO } }, + "seek": func(deps *trustedReleaseStoreDeps) { + deps.seek = func(context.Context, int, int64, int) (int64, error) { return -1, unix.EIO } + }, } for name, inject := range tests { t.Run(name, func(t *testing.T) { diff --git a/internal/service/jailer/trusted_release_archive_stream.go b/internal/service/jailer/trusted_release_archive_stream.go new file mode 100644 index 00000000..53afbc7e --- /dev/null +++ b/internal/service/jailer/trusted_release_archive_stream.go @@ -0,0 +1,244 @@ +package jailer + +import ( + "context" + "crypto/sha256" + "errors" + "fmt" + "io" + + "golang.org/x/sys/unix" +) + +const ( + trustedReleaseArchiveMaxBytes = uint64(128 * 1024 * 1024) + trustedReleaseArchiveStreamBufferBytes = 32 * 1024 + trustedReleaseArchiveMaxEmptyReads = 100 +) + +type trustedReleaseArchiveStageState uint8 + +const ( + trustedReleaseArchiveStageEmpty trustedReleaseArchiveStageState = iota + trustedReleaseArchiveStageReady + trustedReleaseArchiveStageFailed +) + +// CRITICAL: Body is the authenticated raw request stream positioned at the payload start, not a pre-limited reader; +// this method owns exact EOF admission. The reader transports bytes but grants no authority. Only the independently +// obtained digest can promote this single-use anonymous stage to ready, and any started failure permanently poisons it. +func (stage *trustedReleaseArchiveStage) receive( + ctx context.Context, + body io.Reader, + declaredBytes uint64, + expectedDigest trustedReleaseArchiveDigest, +) error { + if stage == nil || stage.fd < 0 { + return trustedReleaseStoreError("trusted release archive stage is not active", nil) + } + if stage.state != trustedReleaseArchiveStageEmpty { + return trustedReleaseStoreError("trusted release archive stage is not empty", nil) + } + if declaredBytes == 0 || declaredBytes > trustedReleaseArchiveMaxBytes { + return trustedReleaseStoreUntrusted("trusted release archive stream size is outside policy", nil) + } + if body == nil { + return trustedReleaseStoreError("trusted release archive stream is not available", nil) + } + if err := ctx.Err(); err != nil { + return trustedReleaseStoreError("receive trusted release archive stream", err) + } + + initial, err := inspectTrustedReleaseArchiveStage( + ctx, + stage.deps, + stage.policy, + stage.fd, + 0, + true, + ) + if err != nil { + return err + } + if err := requireTrustedReleaseArchiveStageZeroOffset(ctx, stage.deps, stage.fd); err != nil { + return err + } + stage.state = trustedReleaseArchiveStageFailed + + digest, err := stage.writeAndHashArchive(ctx, body, declaredBytes) + if err != nil { + return err + } + if digest != expectedDigest { + return trustedReleaseStoreUntrusted("trusted release archive digest does not match authority", nil) + } + if err := ctx.Err(); err != nil { + return trustedReleaseStoreError("sync trusted release archive stage", err) + } + if err := stage.deps.fsync(ctx, stage.fd); err != nil { + return trustedReleaseStoreError("sync trusted release archive stage", err) + } + final, err := inspectTrustedReleaseArchiveStage( + ctx, + stage.deps, + stage.policy, + stage.fd, + declaredBytes, + true, + ) + if err != nil { + return err + } + if !sameTrustedReleaseArchiveStageIdentity(initial, final) { + return trustedReleaseStoreError("trusted release archive stage identity changed while receiving bytes", nil) + } + if err := requireTrustedReleaseArchiveStageZeroOffset(ctx, stage.deps, stage.fd); err != nil { + return err + } + + stage.sizeBytes = declaredBytes + stage.archiveDigest = digest + stage.state = trustedReleaseArchiveStageReady + return nil +} + +func (stage *trustedReleaseArchiveStage) writeAndHashArchive( + ctx context.Context, + body io.Reader, + declaredBytes uint64, +) (trustedReleaseArchiveDigest, error) { + hasher := sha256.New() + buffer := make([]byte, trustedReleaseArchiveStreamBufferBytes) + var offset uint64 + emptyReads := 0 + for offset < declaredBytes { + if err := ctx.Err(); err != nil { + return trustedReleaseArchiveDigest{}, trustedReleaseStoreError( + "receive trusted release archive stream", + err, + ) + } + remaining := declaredBytes - offset + readSize := len(buffer) + if remaining < uint64(readSize) { + readSize = int(remaining) + } + count, readErr := body.Read(buffer[:readSize]) + if count < 0 || count > readSize { + return trustedReleaseArchiveDigest{}, trustedReleaseStoreError( + "receive trusted release archive stream", + fmt.Errorf("invalid read count %d", count), + ) + } + if count > 0 { + emptyReads = 0 + if err := ctx.Err(); err != nil { + return trustedReleaseArchiveDigest{}, trustedReleaseStoreError( + "receive trusted release archive stream", + err, + ) + } + if err := stage.writeArchiveChunk(ctx, buffer[:count], offset); err != nil { + return trustedReleaseArchiveDigest{}, err + } + if _, err := hasher.Write(buffer[:count]); err != nil { + return trustedReleaseArchiveDigest{}, trustedReleaseStoreError( + "hash trusted release archive stream", + err, + ) + } + offset += uint64(count) + } else { + emptyReads++ + } + if readErr != nil && !errors.Is(readErr, io.EOF) { + return trustedReleaseArchiveDigest{}, trustedReleaseStoreError( + "receive trusted release archive stream", + readErr, + ) + } + if offset < declaredBytes && errors.Is(readErr, io.EOF) { + return trustedReleaseArchiveDigest{}, trustedReleaseStoreUntrusted( + "trusted release archive stream is shorter than declared", + nil, + ) + } + if count == 0 && readErr == nil && emptyReads >= trustedReleaseArchiveMaxEmptyReads { + return trustedReleaseArchiveDigest{}, trustedReleaseStoreError( + "receive trusted release archive stream", + io.ErrNoProgress, + ) + } + } + + if err := requireTrustedReleaseArchiveEOF(ctx, body); err != nil { + return trustedReleaseArchiveDigest{}, err + } + var digest trustedReleaseArchiveDigest + copy(digest[:], hasher.Sum(nil)) + return digest, nil +} + +func (stage *trustedReleaseArchiveStage) writeArchiveChunk( + ctx context.Context, + value []byte, + offset uint64, +) error { + for written := 0; written < len(value); { + if err := ctx.Err(); err != nil { + return trustedReleaseStoreError("write trusted release archive stage", err) + } + count, writeErr := stage.deps.pwrite(ctx, stage.fd, value[written:], int64(offset)+int64(written)) + if count < 0 || count > len(value)-written { + return trustedReleaseStoreError( + "write trusted release archive stage", + fmt.Errorf("invalid positioned write count %d", count), + ) + } + written += count + if writeErr != nil { + return trustedReleaseStoreError("write trusted release archive stage", writeErr) + } + if count == 0 { + return trustedReleaseStoreError("write trusted release archive stage", io.ErrNoProgress) + } + } + return nil +} + +func requireTrustedReleaseArchiveEOF(ctx context.Context, body io.Reader) error { + var probe [1]byte + for emptyReads := 0; ; emptyReads++ { + if err := ctx.Err(); err != nil { + return trustedReleaseStoreError("probe trusted release archive stream length", err) + } + count, readErr := body.Read(probe[:]) + if count < 0 || count > len(probe) { + return trustedReleaseStoreError( + "probe trusted release archive stream length", + fmt.Errorf("invalid read count %d", count), + ) + } + if count != 0 { + return trustedReleaseStoreUntrusted("trusted release archive stream is longer than declared", nil) + } + if errors.Is(readErr, io.EOF) { + return nil + } + if readErr != nil { + return trustedReleaseStoreError("probe trusted release archive stream length", readErr) + } + if emptyReads+1 >= trustedReleaseArchiveMaxEmptyReads { + return trustedReleaseStoreError("probe trusted release archive stream length", io.ErrNoProgress) + } + } +} + +func sameTrustedReleaseArchiveStageIdentity(first, second unix.Stat_t) bool { + return first.Dev == second.Dev && + first.Ino == second.Ino && + first.Mode == second.Mode && + first.Nlink == second.Nlink && + first.Uid == second.Uid && + first.Gid == second.Gid +} diff --git a/internal/service/jailer/trusted_release_archive_stream_test.go b/internal/service/jailer/trusted_release_archive_stream_test.go new file mode 100644 index 00000000..2bb336b2 --- /dev/null +++ b/internal/service/jailer/trusted_release_archive_stream_test.go @@ -0,0 +1,483 @@ +package jailer + +import ( + "bytes" + "context" + "crypto/sha256" + "io" + "testing" + + "github.com/google/go-cmp/cmp" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + "golang.org/x/sys/unix" + + "mvmctl/pkg/errs" +) + +func TestTrustedReleaseArchiveStageReceivesExactVerifiedStream(t *testing.T) { + t.Parallel() + + stage, writer := newTrustedReleaseArchiveStreamFixture(t) + require.NoError(t, writer.Release(t.Context())) + payload := bytes.Repeat([]byte("trusted archive bytes"), 4097) + digest := trustedReleaseArchiveDigestForTest(payload) + reader := &trustedReleaseArchiveTrackingReader{reader: bytes.NewReader(payload)} + realPwrite := stage.deps.pwrite + var offsets []int64 + var writeSizes []int + stage.deps.pwrite = func(ctx context.Context, fd int, value []byte, offset int64) (int, error) { + offsets = append(offsets, offset) + writeSizes = append(writeSizes, len(value)) + return realPwrite(ctx, fd, value, offset) + } + realFsync := stage.deps.fsync + fsyncCalls := 0 + stage.deps.fsync = func(ctx context.Context, fd int) error { + fsyncCalls++ + return realFsync(ctx, fd) + } + + err := stage.receive(t.Context(), reader, uint64(len(payload)), digest) + require.NoError(t, err) + assert.Equal(t, trustedReleaseArchiveStageReady, stage.state) + assert.Equal(t, uint64(len(payload)), stage.sizeBytes) + assert.Equal(t, digest, stage.archiveDigest) + assert.LessOrEqual(t, reader.maxRequested, trustedReleaseArchiveStreamBufferBytes) + assert.Equal(t, 1, fsyncCalls) + + wantOffsets := make([]int64, 0, len(offsets)) + wantWriteSizes := make([]int, 0, len(writeSizes)) + for offset := 0; offset < len(payload); offset += trustedReleaseArchiveStreamBufferBytes { + wantOffsets = append(wantOffsets, int64(offset)) + remaining := len(payload) - offset + if remaining > trustedReleaseArchiveStreamBufferBytes { + remaining = trustedReleaseArchiveStreamBufferBytes + } + wantWriteSizes = append(wantWriteSizes, remaining) + } + if diff := cmp.Diff(wantOffsets, offsets); diff != "" { + t.Errorf("positioned-write offsets mismatch (-want +got):\n%s", diff) + } + if diff := cmp.Diff(wantWriteSizes, writeSizes); diff != "" { + t.Errorf("positioned-write sizes mismatch (-want +got):\n%s", diff) + } + + stored := make([]byte, len(payload)) + count, err := unix.Pread(stage.fd, stored, 0) + require.NoError(t, err) + assert.Equal(t, len(payload), count) + if diff := cmp.Diff(payload, stored); diff != "" { + t.Errorf("stored archive mismatch (-want +got):\n%s", diff) + } + offset, err := unix.Seek(stage.fd, 0, unix.SEEK_CUR) + require.NoError(t, err) + assert.Equal(t, int64(0), offset) + retryReader := &trustedReleaseArchiveTrackingReader{reader: bytes.NewReader(payload)} + err = stage.receive(t.Context(), retryReader, uint64(len(payload)), digest) + require.Error(t, err) + assert.Equal(t, errs.CodeBinaryTrustedInstallFailed, errs.AsDomainError(err).Code) + assert.Equal(t, 0, retryReader.calls) +} + +func TestTrustedReleaseArchiveStageRejectsStreamSizeOutsidePolicyBeforeRead(t *testing.T) { + t.Parallel() + + tests := map[string]uint64{ + "empty": 0, + "too large": trustedReleaseArchiveMaxBytes + 1, + "overflow": ^uint64(0), + } + for name, declaredBytes := range tests { + t.Run(name, func(t *testing.T) { + t.Parallel() + + stage, _ := newTrustedReleaseArchiveStreamFixture(t) + reader := &trustedReleaseArchiveTrackingReader{reader: bytes.NewReader([]byte("ignored"))} + pwriteCalled := false + stage.deps.pwrite = func(context.Context, int, []byte, int64) (int, error) { + pwriteCalled = true + return 0, unix.EIO + } + + err := stage.receive(t.Context(), reader, declaredBytes, trustedReleaseArchiveDigest{}) + require.Error(t, err) + assert.Equal(t, errs.CodeBinaryUntrusted, errs.AsDomainError(err).Code) + assert.Equal(t, 0, reader.calls) + assert.False(t, pwriteCalled) + assert.Equal(t, trustedReleaseArchiveStageEmpty, stage.state) + }) + } +} + +func TestTrustedReleaseArchiveStageRejectsUntrustedStreamContent(t *testing.T) { + t.Parallel() + + payload := []byte("complete trusted release archive") + digest := trustedReleaseArchiveDigestForTest(payload) + tests := map[string]struct { + body []byte + declaredBytes uint64 + digest trustedReleaseArchiveDigest + }{ + "truncated": { + body: payload, + declaredBytes: uint64(len(payload) + 1), + digest: digest, + }, + "trailing": { + body: append(append([]byte(nil), payload...), '!'), + declaredBytes: uint64(len(payload)), + digest: digest, + }, + "digest mismatch": { + body: payload, + declaredBytes: uint64(len(payload)), + digest: trustedReleaseArchiveDigest{}, + }, + } + for name, test := range tests { + t.Run(name, func(t *testing.T) { + t.Parallel() + + stage, _ := newTrustedReleaseArchiveStreamFixture(t) + fsyncCalled := false + stage.deps.fsync = func(context.Context, int) error { + fsyncCalled = true + return nil + } + + err := stage.receive(t.Context(), bytes.NewReader(test.body), test.declaredBytes, test.digest) + require.Error(t, err) + assert.Equal(t, errs.CodeBinaryUntrusted, errs.AsDomainError(err).Code) + assert.Equal(t, trustedReleaseArchiveStageFailed, stage.state) + assert.False(t, fsyncCalled) + + err = stage.receive( + t.Context(), + bytes.NewReader(payload), + uint64(len(payload)), + digest, + ) + require.Error(t, err) + assert.Equal(t, errs.CodeBinaryTrustedInstallFailed, errs.AsDomainError(err).Code) + }) + } +} + +func TestTrustedReleaseArchiveStageHandlesPartialPositionedWrites(t *testing.T) { + t.Parallel() + + stage, _ := newTrustedReleaseArchiveStreamFixture(t) + payload := bytes.Repeat([]byte("partial write"), 100) + realPwrite := stage.deps.pwrite + stage.deps.pwrite = func(ctx context.Context, fd int, value []byte, offset int64) (int, error) { + if len(value) > 7 { + value = value[:7] + } + return realPwrite(ctx, fd, value, offset) + } + + err := stage.receive( + t.Context(), + bytes.NewReader(payload), + uint64(len(payload)), + trustedReleaseArchiveDigestForTest(payload), + ) + require.NoError(t, err) + assert.Equal(t, trustedReleaseArchiveStageReady, stage.state) + + stored := make([]byte, len(payload)) + count, err := unix.Pread(stage.fd, stored, 0) + require.NoError(t, err) + assert.Equal(t, len(payload), count) + if diff := cmp.Diff(payload, stored); diff != "" { + t.Errorf("stored archive mismatch after partial writes (-want +got):\n%s", diff) + } +} + +func TestTrustedReleaseArchiveStageRejectsStreamAndWriteFailures(t *testing.T) { + t.Parallel() + + payload := []byte("trusted archive bytes") + digest := trustedReleaseArchiveDigestForTest(payload) + tests := map[string]struct { + reader func(context.CancelFunc) io.Reader + inject func(*trustedReleaseArchiveStage) + want error + }{ + "reader error": { + reader: func(context.CancelFunc) io.Reader { + return io.MultiReader(bytes.NewReader(payload[:5]), trustedReleaseArchiveReaderFunc( + func([]byte) (int, error) { return 0, unix.EIO }, + )) + }, + want: unix.EIO, + }, + "reader invalid count": { + reader: func(context.CancelFunc) io.Reader { + return trustedReleaseArchiveReaderFunc(func(value []byte) (int, error) { + return len(value) + 1, nil + }) + }, + }, + "reader no progress": { + reader: func(context.CancelFunc) io.Reader { + return trustedReleaseArchiveReaderFunc(func([]byte) (int, error) { return 0, nil }) + }, + want: io.ErrNoProgress, + }, + "EOF probe no progress": { + reader: func(context.CancelFunc) io.Reader { + firstRead := true + return trustedReleaseArchiveReaderFunc(func(value []byte) (int, error) { + if firstRead { + firstRead = false + return copy(value, payload), nil + } + return 0, nil + }) + }, + want: io.ErrNoProgress, + }, + "canceled after read": { + reader: func(cancel context.CancelFunc) io.Reader { + return trustedReleaseArchiveReaderFunc(func(value []byte) (int, error) { + count := copy(value, payload) + cancel() + return count, nil + }) + }, + inject: func(stage *trustedReleaseArchiveStage) { + stage.deps.pwrite = func(context.Context, int, []byte, int64) (int, error) { + return 0, unix.EFAULT + } + }, + want: context.Canceled, + }, + "positioned write error": { + inject: func(stage *trustedReleaseArchiveStage) { + stage.deps.pwrite = func(context.Context, int, []byte, int64) (int, error) { + return 0, unix.ENOSPC + } + }, + want: unix.ENOSPC, + }, + "positioned write invalid count": { + inject: func(stage *trustedReleaseArchiveStage) { + stage.deps.pwrite = func(_ context.Context, _ int, value []byte, _ int64) (int, error) { + return len(value) + 1, nil + } + }, + }, + "positioned write no progress": { + inject: func(stage *trustedReleaseArchiveStage) { + stage.deps.pwrite = func(context.Context, int, []byte, int64) (int, error) { return 0, nil } + }, + }, + } + for name, test := range tests { + t.Run(name, func(t *testing.T) { + t.Parallel() + + stage, _ := newTrustedReleaseArchiveStreamFixture(t) + ctx, cancel := context.WithCancel(t.Context()) + defer cancel() + if test.inject != nil { + test.inject(stage) + } + reader := io.Reader(bytes.NewReader(payload)) + if test.reader != nil { + reader = test.reader(cancel) + } + + err := stage.receive(ctx, reader, uint64(len(payload)), digest) + require.Error(t, err) + assert.Equal(t, errs.CodeBinaryTrustedInstallFailed, errs.AsDomainError(err).Code) + assert.Equal(t, trustedReleaseArchiveStageFailed, stage.state) + if test.want != nil { + assert.ErrorIs(t, err, test.want) + } + }) + } +} + +func TestTrustedReleaseArchiveStageRejectsAcceptanceFailures(t *testing.T) { + t.Parallel() + + payload := []byte("trusted archive bytes") + digest := trustedReleaseArchiveDigestForTest(payload) + tests := map[string]struct { + inject func(*trustedReleaseArchiveStage) + want error + }{ + "fsync": { + inject: func(stage *trustedReleaseArchiveStage) { + stage.deps.fsync = func(context.Context, int) error { return unix.EIO } + }, + want: unix.EIO, + }, + "final fstat": { + inject: func(stage *trustedReleaseArchiveStage) { + realFstat := stage.deps.fstat + calls := 0 + stage.deps.fstat = func(ctx context.Context, fd int, stat *unix.Stat_t) error { + calls++ + if calls == 2 { + return unix.EIO + } + return realFstat(ctx, fd, stat) + } + }, + want: unix.EIO, + }, + "changed identity": { + inject: func(stage *trustedReleaseArchiveStage) { + realFstat := stage.deps.fstat + calls := 0 + stage.deps.fstat = func(ctx context.Context, fd int, stat *unix.Stat_t) error { + calls++ + if err := realFstat(ctx, fd, stat); err != nil { + return err + } + if calls == 2 { + stat.Ino++ + } + return nil + } + }, + }, + "wrong final size": { + inject: func(stage *trustedReleaseArchiveStage) { + realFstat := stage.deps.fstat + calls := 0 + stage.deps.fstat = func(ctx context.Context, fd int, stat *unix.Stat_t) error { + calls++ + if err := realFstat(ctx, fd, stat); err != nil { + return err + } + if calls == 2 { + stat.Size++ + } + return nil + } + }, + }, + "changed offset": { + inject: func(stage *trustedReleaseArchiveStage) { + realPwrite := stage.deps.pwrite + stage.deps.pwrite = func( + ctx context.Context, + fd int, + value []byte, + offset int64, + ) (int, error) { + count, err := realPwrite(ctx, fd, value, offset) + if err == nil { + _, err = unix.Seek(fd, 1, unix.SEEK_SET) + } + return count, err + } + }, + }, + } + for name, test := range tests { + t.Run(name, func(t *testing.T) { + t.Parallel() + + stage, _ := newTrustedReleaseArchiveStreamFixture(t) + test.inject(stage) + + err := stage.receive(t.Context(), bytes.NewReader(payload), uint64(len(payload)), digest) + require.Error(t, err) + assert.Equal(t, errs.CodeBinaryTrustedInstallFailed, errs.AsDomainError(err).Code) + assert.Equal(t, trustedReleaseArchiveStageFailed, stage.state) + if test.want != nil { + assert.ErrorIs(t, err, test.want) + } + }) + } +} + +func TestTrustedReleaseArchiveStageRejectsCanceledContextBeforeStreamRead(t *testing.T) { + t.Parallel() + + stage, _ := newTrustedReleaseArchiveStreamFixture(t) + ctx, cancel := context.WithCancel(t.Context()) + cancel() + reader := &trustedReleaseArchiveTrackingReader{reader: bytes.NewReader([]byte("ignored"))} + + err := stage.receive(ctx, reader, 1, trustedReleaseArchiveDigest{}) + require.Error(t, err) + assert.ErrorIs(t, err, context.Canceled) + assert.Equal(t, errs.CodeBinaryTrustedInstallFailed, errs.AsDomainError(err).Code) + assert.Equal(t, 0, reader.calls) + assert.Equal(t, trustedReleaseArchiveStageEmpty, stage.state) +} + +func TestTrustedReleaseArchiveStageRejectsNilStreamBeforeTransition(t *testing.T) { + t.Parallel() + + stage, _ := newTrustedReleaseArchiveStreamFixture(t) + err := stage.receive(t.Context(), nil, 1, trustedReleaseArchiveDigest{}) + require.Error(t, err) + assert.Equal(t, errs.CodeBinaryTrustedInstallFailed, errs.AsDomainError(err).Code) + assert.Equal(t, trustedReleaseArchiveStageEmpty, stage.state) +} + +func TestTrustedReleaseArchiveStageRejectsNonzeroInitialOffsetBeforeStreamRead(t *testing.T) { + t.Parallel() + + stage, _ := newTrustedReleaseArchiveStreamFixture(t) + _, err := unix.Seek(stage.fd, 1, unix.SEEK_SET) + require.NoError(t, err) + reader := &trustedReleaseArchiveTrackingReader{reader: bytes.NewReader([]byte("ignored"))} + + err = stage.receive(t.Context(), reader, 1, trustedReleaseArchiveDigest{}) + require.Error(t, err) + assert.Equal(t, errs.CodeBinaryTrustedInstallFailed, errs.AsDomainError(err).Code) + assert.Equal(t, 0, reader.calls) + assert.Equal(t, trustedReleaseArchiveStageEmpty, stage.state) +} + +func newTrustedReleaseArchiveStreamFixture( + t *testing.T, +) (*trustedReleaseArchiveStage, *trustedReleaseStoreWriteLease) { + t.Helper() + + fixture := newTrustedReleaseStoreFixture(t) + writer, err := openTrustedReleaseStoreForWrite(t.Context(), fixture.deps, fixture.policy) + require.NoError(t, err) + t.Cleanup(func() { require.NoError(t, writer.Release(context.Background())) }) + stage, err := writer.createArchiveStage(t.Context()) + require.NoError(t, err) + t.Cleanup(func() { require.NoError(t, stage.Release(context.Background())) }) + return stage, writer +} + +func trustedReleaseArchiveDigestForTest(value []byte) trustedReleaseArchiveDigest { + return trustedReleaseArchiveDigest(sha256.Sum256(value)) +} + +type trustedReleaseArchiveTrackingReader struct { + reader io.Reader + calls int + maxRequested int +} + +func (reader *trustedReleaseArchiveTrackingReader) Read(value []byte) (int, error) { + reader.calls++ + if len(value) > reader.maxRequested { + reader.maxRequested = len(value) + } + return reader.reader.Read(value) +} + +type trustedReleaseArchiveReaderFunc func([]byte) (int, error) + +func (function trustedReleaseArchiveReaderFunc) Read(value []byte) (int, error) { + return function(value) +} + +var _ io.Reader = (*trustedReleaseArchiveTrackingReader)(nil) +var _ io.Reader = trustedReleaseArchiveReaderFunc(nil) diff --git a/internal/service/jailer/trusted_release_store_fs.go b/internal/service/jailer/trusted_release_store_fs.go index c0ba02a1..72abf66b 100644 --- a/internal/service/jailer/trusted_release_store_fs.go +++ b/internal/service/jailer/trusted_release_store_fs.go @@ -27,6 +27,8 @@ type trustedReleaseStoreDeps struct { fchmod func(context.Context, int, uint32) error read func(context.Context, int, []byte) (int, error) pread func(context.Context, int, []byte, int64) (int, error) + pwrite func(context.Context, int, []byte, int64) (int, error) + seek func(context.Context, int, int64, int) (int64, error) fsync func(context.Context, int) error close func(context.Context, int) error } @@ -479,6 +481,12 @@ func realTrustedReleaseStoreDeps() trustedReleaseStoreDeps { pread: func(_ context.Context, fd int, value []byte, offset int64) (int, error) { return unix.Pread(fd, value, offset) }, + pwrite: func(_ context.Context, fd int, value []byte, offset int64) (int, error) { + return unix.Pwrite(fd, value, offset) + }, + seek: func(_ context.Context, fd int, offset int64, whence int) (int64, error) { + return unix.Seek(fd, offset, whence) + }, fsync: func(_ context.Context, fd int) error { return unix.Fsync(fd) }, From ddee9ecaf7d4cc82d41d6d5707a06c21b1f62c0c Mon Sep 17 00:00:00 2001 From: AlanD20 Date: Sat, 29 Aug 2026 02:04:57 +0200 Subject: [PATCH 0087/1457] docs: record trusted archive stream admission --- .../0016-root-owned-single-binary-privileged-dispatch.md | 6 ++++-- tasks/todo.md | 2 +- 2 files changed, 5 insertions(+), 3 deletions(-) diff --git a/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md b/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md index 76eb572b..6044ef60 100644 --- a/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md +++ b/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md @@ -354,8 +354,10 @@ are not yet wired to the privileged request or legacy installer. The write-side only the fixed managed `mvmctl/binaries` base through pinned ancestors, corrects metadata only on directories it created, and durably syncs each newly observed child and parent. It can now create and retain the fixed anonymous archive stage described above, including fail-closed metadata admission, cancellation-independent cleanup, and cleanup-error -preservation. Bounded streaming and digest verification, extraction, the complete member allowlist, -architecture/version creation, manifest writing, and atomic trusted-store installation remain Task 6 work. +preservation. That stage now admits one bounded exact-length caller stream with positioned writes, exact EOF, +independent-digest verification, fsync, stable identity and metadata, zero offset, and permanent poisoning after any +started failure. Root-origin archive fetching, extraction, the complete member allowlist, architecture/version +creation, manifest writing, and atomic trusted-store installation remain Task 6 work. ### Paths, process identity, and runtime state diff --git a/tasks/todo.md b/tasks/todo.md index 2c84e97a..fc479340 100644 --- a/tasks/todo.md +++ b/tasks/todo.md @@ -223,7 +223,7 @@ under `internal/service/jailer/`. No handler, transport, CLI, API, core-domain, enforce exact root ownership/mode and fsync newly observed child and parent directories before staging bytes. - [x] Create a private anonymous `O_TMPFILE|O_EXCL` archive-stage lease on the pinned trusted-store filesystem with exact empty, unlinked, root-owned `0600` metadata and no pathname, memory, or alternate-filesystem fallback. - - [ ] Admit one exact caller-supplied 1-byte through 128-MiB stream into that stage with bounded positioned writes, + - [x] Admit one exact caller-supplied 1-byte through 128-MiB stream into that stage with bounded positioned writes, authoritative SHA-256 comparison, exact EOF, fsync, stable metadata, zero offset, and fail-closed poisoning. - [x] Freeze the strict schema-v1 manifest contract, 4 KiB record bound, typed SHA-256 fields, and 120-byte through 64 MiB executable-size policy; the fixed store leaf is `manifest.json`. From 799de9652658810f49d3a734657ec67639d732be Mon Sep 17 00:00:00 2001 From: AlanD20 Date: Sat, 29 Aug 2026 02:19:19 +0200 Subject: [PATCH 0088/1457] docs: freeze trusted release archive contract --- ...owned-single-binary-privileged-dispatch.md | 69 ++++++++++++++++--- tasks/plan.md | 14 ++++ tasks/todo.md | 8 ++- 3 files changed, 79 insertions(+), 12 deletions(-) diff --git a/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md b/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md index 6044ef60..5bd7036f 100644 --- a/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md +++ b/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md @@ -212,11 +212,11 @@ It hashes the complete archive body against that checksum before parsing it. If fixed archive itself. Root never opens a caller path or accepts a caller URL or checksum. The normal-user client may read `MVM_ASSET_MIRROR`, but those streamed bytes are transport, never release authority. -Extraction occurs in a root-owned temporary directory with bounded compressed/decompressed input, a reviewed allowlist -for the complete upstream archive layout, and rejection of path traversal, symlinks, hardlinks, devices, sparse files, -duplicates, unexpected members, and size/count overflow. Only the exact Firecracker/Jailer pair is extracted. Their ELF -class and machine must match the selected architecture without executing downloaded code. The pair and strict root-owned -release manifest are fsynced and atomically renamed into the trusted store. A referenced release cannot be +Extraction reads the anonymous root-owned archive stage with bounded compressed/decompressed input and validates the +complete reviewed archive contract below before publishing any path. Only the exact Firecracker/Jailer pair is copied +to private root-owned staging descriptors. Their ELF class and machine must match the selected architecture without +executing downloaded code. The pair and strict root-owned release manifest are fsynced and atomically renamed into the +trusted store only after the complete archive and both executables pass admission. A referenced release cannot be force-replaced. ELF admission reads exactly the first 64 bytes of each extracted binary and performs no executable handoff. It requires @@ -248,9 +248,56 @@ derivation is: | Trusted-store leaves | `firecracker`, `jailer`, and `manifest.json` | This table freezes source construction and the two extracted member identities. It does not claim that those are the -only members in the upstream archive: Task 6 separately records and enforces the complete reviewed upstream allowlist -before extraction is accepted. A packaging change must update the reviewed contract and tests; it must not fall back to -basename matching, caller-selected member names, or permissive extraction. +only members in the upstream archive: the complete reviewed allowlist and format contract follow, and Task 6 must +enforce them before extraction is accepted. A packaging change must update the reviewed contract and tests; it must not +fall back to basename matching, caller-selected member names, or permissive extraction. + +The v0.3 archive parser accepts only the separately audited x86_64 releases `1.10.1`, `1.14.2`, `1.14.3`, `1.14.4`, +`1.15.0`, `1.15.1`, `1.16.0`, and `1.16.1`. Source derivation and ELF admission also understand `aarch64`, but that does +not authorize an aarch64 archive: the available asset mirror contains no aarch64 release archive, so extraction for +that architecture fails closed until an architecture-specific member and format audit updates this decision and its +tests. Versions outside the exact audited x86_64 set likewise fail closed rather than assuming that a nearby release +uses the same packaging. + +For an accepted version, let `` be `release-v-x86_64` and `` be +`v-x86_64`. The logical archive contains exactly 24 regular-file members, in any order, with this exact set: + +- `/{SHA256SUMS,LICENSE,THIRD-PARTY,NOTICE}`; +- `/firecracker_spec-v.yaml` and `/seccomp-filter-.json`; +- `/{firecracker,jailer,cpu-template-helper,rebase-snap,seccompiler-bin,snapshot-editor}-`, plus the + corresponding exact `.debug` member for each tool; and +- six CPU-template leaves `/