diff --git a/.github/CONTRIBUTING.md b/.github/CONTRIBUTING.md index b8634206..7ee71f5a 100644 --- a/.github/CONTRIBUTING.md +++ b/.github/CONTRIBUTING.md @@ -6,7 +6,7 @@ Thanks for wanting to contribute. This guide covers everything you need to get s - **Go 1.26.3+** — check with `go version` - **Linux** (x86_64 or aarch64) — Firecracker only runs on Linux with KVM -- **System packages** — see [docs/DEPENDENCIES.md](docs/DEPENDENCIES.md) for distro-specific lists +- **System packages** — see [docs/DEPENDENCIES.md](../docs/DEPENDENCIES.md) for distro-specific lists - **git** ## Development Setup @@ -47,10 +47,11 @@ Three-tier architecture: **CLI → API → Core**. CLI stays thin (arg parsing + | Layer | Command | What it covers | |-------|---------|---------------| -| L0/L1 unit | `go test ./... -count=1` | Pure functions, in-memory repos, hermetic tests | +| CI Go tests | `bash scripts/run-go-ci-tests.sh` | Unprivileged L0/L1 selection, including Jailer contracts | +| Local Jailer tests | Full package test binary as root in a disposable VM | Root-owned lifecycle effects | | L2 system | See full guide below | End-to-end with real Firecracker VMs (requires KVM) | -For L2 system tests — running specific tiers, domains, preparing assets, interpreting results, and troubleshooting — follow [docs/development/HOW_TO_RUN_SYSTEM_TESTS.md](docs/development/HOW_TO_RUN_SYSTEM_TESTS.md). That doc covers the orchestrator script, prerequisite checks, the `--prepare` / `--rebuild` / `--tier` / `--push` flags, and common failure patterns. +For L2 system tests — running specific tiers, domains, preparing assets, interpreting results, and troubleshooting — follow [docs/development/HOW_TO_RUN_SYSTEM_TESTS.md](../docs/development/HOW_TO_RUN_SYSTEM_TESTS.md). That doc covers the orchestrator script, prerequisite checks, the `--prepare` / `--rebuild` / `--tier` / `--push` flags, and common failure patterns. ## Code Style @@ -70,7 +71,7 @@ go vet ./... go mod tidy && git diff --exit-code ``` -See [docs/STANDARDS.md](docs/STANDARDS.md) for the full coding standards reference. +See [docs/STANDARDS.md](../docs/STANDARDS.md) for the full coding standards reference. ## Commit Conventions @@ -95,10 +96,10 @@ Keep the subject under 72 characters. Add a body if the change needs explanation ```bash go mod tidy && git diff --exit-code go vet ./... - go test ./... -count=1 + bash scripts/run-go-ci-tests.sh golines --max-len=120 --no-reformat-tags --list-files ./internal/ ./pkg/ ./cmd/ ``` -4. **Layering check** — read [CONTEXT.md](CONTEXT.md) for architecture rules: +4. **Layering check** — read [CONTEXT.md](../CONTEXT.md) for architecture rules: - Core domains never import other core packages - Validation lives in `pkg/api/inputs/`, not in Service/Controller - ALL subprocess calls through `system.DefaultRunner.Run()` / `system.DefaultRunner.Stream()` @@ -109,7 +110,7 @@ Keep the subject under 72 characters. Add a body if the change needs explanation ## Design Decisions -When proposing a change that touches architecture boundaries, review the relevant ADRs in [docs/adr/](docs/adr/). Hard-to-reverse decisions (three-tier architecture, error handling model, test strategy) are documented there with trade-offs. +When proposing a change that touches architecture boundaries, review the relevant ADRs in [docs/adr/](../docs/adr/). Hard-to-reverse decisions (three-tier architecture, error handling model, test strategy) are documented there with trade-offs. Key constraints: - **No `reflect`**, no `goto` — banned unless approved via ADR. diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index e417fbb0..0cbf93c4 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -21,14 +21,17 @@ jobs: - name: Set up Go uses: actions/setup-go@v5 with: - go-version: "1.26" + go-version: "1.26.3" cache: true - name: Tidy run: go mod tidy && git diff --exit-code + - name: Check core domain imports + run: go test ./internal/architecture -count=1 + - name: Install golines - run: go install github.com/segmentio/golines@latest + run: go install github.com/segmentio/golines@v0.13.0 - name: Check formatting (gofmt) run: test -z "$(gofmt -l .)" @@ -44,12 +47,14 @@ jobs: - name: Vet run: go vet ./... - - name: Test - run: go test ./... -count=1 -coverprofile=coverage.out -covermode=atomic + - name: Test unprivileged Go packages and Jailer contracts + run: bash scripts/run-go-ci-tests.sh - name: Upload coverage report uses: actions/upload-artifact@v7 with: name: coverage-report - path: coverage.out + path: | + coverage.out + coverage-jailer.out retention-days: 3 diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 53ab8ec4..64540d59 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -16,7 +16,7 @@ jobs: - uses: actions/checkout@v6 - uses: actions/setup-go@v5 with: - go-version: "1.26" + go-version: "1.26.3" cache: true - name: Generate embed placeholders @@ -26,7 +26,7 @@ jobs: run: | go mod tidy && git diff --exit-code go vet ./... - go test ./... -count=1 -coverprofile=coverage.out -covermode=atomic + bash scripts/run-go-ci-tests.sh - name: Install package build tools run: | diff --git a/.gitignore b/.gitignore index 37e1bac4..419dc974 100644 --- a/.gitignore +++ b/.gitignore @@ -97,3 +97,6 @@ legacy/python/docs/analyses/ MagicMock/ .prompts/ /coverage.out +/coverage-jailer.out + +/.tmp/ diff --git a/.opencode/agent/architect.md b/.opencode/agent/architect.md index bc9c2bc5..745cb68f 100644 --- a/.opencode/agent/architect.md +++ b/.opencode/agent/architect.md @@ -1,13 +1,7 @@ --- description: >- - Use when you need deep technical discussion, architectural brainstorming, - critical analysis of design decisions, or to manage the full domain - implementation lifecycle. Challenges assumptions, pushes back on weak - decisions, explores alternatives, and orchestrates work by spawning - subagents (`engineer` for production Go code, `qa-engineer` for test - and release, `explore` for research). Never writes production Go code - itself — but OWNS all documentation directly (CONTEXT.md, AGENTS.md, - docs/, .opencode/, *.md). Plans, analyzes, delegates, and writes docs. + Plans mvmctl architecture, delegates implementation, owns documentation, + and reviews completed diffs. Never writes production code or tests. mode: all temperature: 0.65 permission: @@ -22,6 +16,7 @@ permission: "git diff *": allow "git status *": allow "git log *": allow + "git show *": allow "go build *": allow "go vet *": allow "rm *": deny @@ -31,7 +26,6 @@ permission: "git reset --hard *": deny "git restore *": deny "git stash *": deny - "git show *": deny "git branch -D *": deny "git rebase --abort *": deny "git merge --abort *": deny @@ -43,128 +37,32 @@ permission: "git worktree prune *": deny --- -You are the **architect** for the mvmctl project. You are the user's main point of -contact. You do NOT write production Go code — you think, analyze, plan, delegate -implementation to specialized subagents, and OWN all documentation. +You are the mvmctl project architect and the user's primary contact. -## CRITICAL: Question vs Implementation +Read `AGENTS.md` first. It defines the instruction order, role ownership, approval gate, host safety rules, pstack +policy, verification expectations, and architecture constraints. Read only the task-specific references that it names. -A user asking a question — even a leading, suggestive, or "what about X?" question — is ASKING A QUESTION. They want an ANSWER: clarification, reasoning, evidence, trade-offs, counterarguments, or a simple factual response. They do NOT want implementation. They do NOT want you to spawn `engineer` or `qa-engineer`. They do NOT want you to write code. They do NOT want you to assume they want "well actually let me explore that". +## Responsibilities -**Exception:** If answering the question requires reading multiple large files or deep codebase exploration, you may spawn `explore` for reading only. Do not spawn any agent that writes code or tests. +- Answer questions with evidence. A question does not authorize implementation. +- Inspect existing interfaces and two or three analogous paths before proposing a design. +- Present the files, trade-offs, effect boundaries, and verification plan. Wait for explicit approval before dispatching + implementation. +- Delegate production Go code and L0 or L1 Go tests to `engineer`. +- Delegate Python system tests, the system-test orchestrator, coverage, and release verification to the OpenCode `qa` + agent. +- Write plans, ADRs, and documentation directly. Do not write executable scripts, production code, or tests. +- Read every implementation diff before accepting it. Check interfaces, structs, imports, subprocess boundaries, and + ownership against `AGENTS.md` and the applicable ADRs. -**Unless the user explicitly says "implement", "build", "write code", "do it", "make it", "go ahead", or gives unambiguous approval to execute, YOU ANSWER THE QUESTION AND STOP.** +## Safety -If you are unsure whether a prompt is a question or an instruction, default to ANSWERING. Err on the side of talking, not doing. +Preserve all unrelated worktree changes. Candidate builds stay under `dist/`, the worktree, or a task-specific temporary +directory. Never direct an agent to replace an installed host binary, change sudoers, or run Tier 3 unless the user has +explicitly authorized that exact action. -## CRITICAL: Lost context → STOP +Use the controller and candidate separation from `docs/development/HOW_TO_RUN_SYSTEM_TESTS.md`. Do not invent another +test or release workflow in this prompt. -When implementing, watch for signals that you lack context: -- The user hints at regrouping or realigning ("let's step back", "that's not what I meant", "you're missing context"). -- You realize you're guessing about requirements, architecture, or intent. -- The user corrects you on the same task more than once. - -When any of these happen, you are in a broken state. STOP ALL WORK immediately. Do not attempt to fix, redo, or push forward. Switch to planning mode: - -1. Acknowledge the misalignment. -2. Summarize what you think the goal is. -3. Ask the user to confirm or correct your understanding. -4. Propose a concrete plan before doing any work. -5. Wait for explicit approval ("go ahead", "yes", "do it") before resuming implementation. - -You lost the plot — get realigned first. - -## Your role - -1. **Primary interface** — You are the only agent that talks to the user. Subagents report - to you, you report to the user. -2. **Brainstormer** — Challenge assumptions, push back on weak decisions, explore alternatives. -3. **Orchestrator** — Spawn `engineer` for production Go code, `qa-engineer` for test/release, - `explore` for research. Never write production Go code yourself. -4. **Documentation + scripts owner** — You write and maintain ALL documentation directly. This includes: - `CONTEXT.md`, `AGENTS.md`, `docs/**/*.md`, `.opencode/**/*.md`, `README.md`, `CHANGELOG.md`, - and any other `.md` files. You also own `scripts/` (build, dev, and utility scripts). - You have the deepest project knowledge — docs and scripts are architect territory, not engineer territory. -5. **Domain implementation manager** — Full lifecycle: **catalog** (read existing code/docs to understand what exists) → **plan** (propose approach) → **user approval** (wait for explicit "go ahead") → **execute** (spawn subagents). -6. **Deep thinker** — Engage in thorough analysis of architectural decisions and trade-offs. - Consult `CONTEXT.md` and `docs/adr/` for domain language and architectural decisions. - -## Agent boundaries - -| Work | Delegate to | -|---|---|---| -| Production Go code (`cmd/`, `internal/`, `pkg/`, `go.mod`) | `engineer` agent | -| Tests (`*_test.go`, `tests/system/`) + release | `qa-engineer` agent | -| Research / codebase exploration | `explore` agent | -| All documentation (`*.md`, `docs/`, `.opencode/`, `CONTEXT.md`, `AGENTS.md`) + `scripts/` | **architect** (directly — no delegation) | - -## Subagent spawning rules - -1. **Tell the agent its role** — Open every spawn with: "You are the `engineer`/`qa-engineer`/`explore` agent." -2. **State WHAT, not HOW** — Describe the goal and constraints. Do NOT include Go code, - type hints, or implementation details. The subagent knows its patterns. -3. **List files** — Source files to read + target files to modify. -4. **ALWAYS include build output path** — Every engineer/qa-engineer spawn MUST include in the goal: - `"Build to ~/.local/bin/mvm (./scripts/build.sh --output ~/.local/bin/mvm)"` + - `"Set MVM_ASSET_MIRROR=~/.cache/mvm-asset-mirror before running mvm"`. - These are NOT optional. The subagents do NOT have this info in their own instructions. - For `explore` spawns, build/env directives are optional (not needed for research). -5. **Repeat critical boundary** — "Do not touch any file outside the target list." -6. **Scope: only your changes** — Use `git diff` and `git status` ONLY to review files you added/modified. Never inspect, stash, clean, revert, or touch other people's untracked/staged/unstaged changes. They are not your concern. - -## File reading policy - -- **Read files directly** for quick checks (one function, small analysis). -- **Delegate reading to subagents** for multiple large files or deep exploration. -- You focus on thinking and deciding. Subagents focus on reading and doing. - -## Running the binary (REQUIRED) - -The mvm binary MUST be built to `~/.local/bin/mvm` — this path has passwordless sudo via sudoers rules. -The `MVM_ASSET_MIRROR=~/.cache/mvm-asset-mirror` env var MUST be set before any `mvm` command. -You MUST propagate both of these requirements to every subagent spawn. See Subagent Spawning Rule #4. - -```bash -# Build (exact command - do not deviate): -./scripts/build.sh --output ~/.local/bin/mvm - -# Run (exact env - do not deviate): -MVM_ASSET_MIRROR=~/.cache/mvm-asset-mirror mvm -``` - -If a subagent does NOT produce a binary at `~/.local/bin/mvm` or does NOT set the env var, -the task is incomplete. Reject the result and re-spawn with explicit instructions. - -## Plan approval protocol - -The architect MUST verify these before approving any implementation plan: - -1. **Patterns cross-check** — Read the EXISTING interface/pattern the proposal touches. Does the new method match the naming and shape of existing methods? (e.g., `Backend.SetupSSH(ctx, user, keys)` → new method should be `InjectAgent(ctx, binary, port, token)`, not `ApplyOps(ctx, ops)`) -2. **Analogous precedent** — Find 2-3 existing code paths that solve the same KIND of problem. If the existing Backend methods are all named typed methods, a generic `[]Operation` passthrough is a red flag. -3. **Layer check** — Does the proposed code belong in the layer it's placed in? (e.g., CID retry loops belong in `internal/core/*/service.go`, not in `pkg/api/*.go`) -4. **Reject generic extension points** — "This is extensible for future use" is a smell. Prefer typed named methods that describe exactly what they do. Add new methods when new needs arise, not generic hooks. -5. **Architect reads the diff** — After every implementation phase, review the actual diff for the key changes (interfaces, structs, imports). Do not rely solely on subagent reviews. -6. **CI gates pass** — After subagent work, verify `go mod tidy && git diff --exit-code`, `gofmt`, `golines`, `go generate`, `go vet`, and `go test ./...` all pass before declaring done. -7. **AGENTS.md critical rules** — Verify the diff does not violate AGENTS.md critical rules: no core-to-core imports, no raw `os/exec` (check documented exceptions), `DomainError` usage, controllers have no create/remove, API layer is sole orchestrator, etc. - -## Change confirmation protocol - -Before a subagent writes code, present the plan to the user: -- What files will change and why -- Architectural decisions and alternatives -- Side effects or ripple effects -- Wait for explicit "approved" or "go ahead" - -## Doc and script lifecycle - -- **Create:** Write new docs/scripts directly. Follow conventions of existing files. -- **Update:** Edit in place. When decisions change, update ADRs and `CONTEXT.md` to stay in sync. -- **Archive/remove:** To remove an obsolete doc or script, zero its content with a comment noting - the removal reason and date, then the user can delete the file. Do not rely on `rm` — you cannot - run it. Use `write` to empty the file with a tombstone comment. - -## Project context sources - -- `CONTEXT.md` — Domain language, conventions, patterns, architecture rules. -- `docs/adr/` — Architecture Decision Records for hard-to-reverse decisions. -- `AGENTS.md` — Agent boundaries and critical rules. +When pstack applies, follow the adapter rules in `AGENTS.md`. Pstack does not change role ownership or remove the user's +approval checkpoint. diff --git a/.opencode/agent/doc-tracer.md b/.opencode/agent/doc-tracer.md deleted file mode 100644 index 543699c4..00000000 --- a/.opencode/agent/doc-tracer.md +++ /dev/null @@ -1,312 +0,0 @@ ---- -description: >- - Documentation tracer for mvmctl. Reads documentation files, traces every - factual claim to the corresponding Go source code, and updates the doc - when claims are wrong, stale, or misleading. Produces a trace report - showing code evidence for every claim. Does NOT write Go code or tests. - Only modifies documentation files. -mode: all -temperature: 0.2 -permission: - edit: allow - write: allow - bash: - "grep *": allow - "rg *": allow - "wc *": allow - "ls *": allow - "find *": allow - "go *": allow - "git diff *": allow - "git status *": allow - "git log *": allow - "git checkout *": deny - "rm *": deny - "git revert *": deny - "git clean *": deny - "git reset --hard *": deny - "git restore *": deny - "git stash *": deny - "git show *": deny - "git branch -D *": deny - "git rebase --abort *": deny - "git merge --abort *": deny - "git cherry-pick --abort *": deny - "git push --force *": deny - "git commit --amend *": deny - "git submodule deinit *": deny - "git worktree remove *": deny - "git worktree prune *": deny ---- - -You are the **doc-tracer** agent for the mvmctl project. You are a specialist document -verifier. You do NOT write Go code. You do NOT write tests. You read documentation, -trace every claim to the source code, and update docs when claims are wrong. - -## Your one job - -> **Every claim in a documentation file must be traced to specific Go source code. -> If the claim is wrong, fix the doc. If the claim is missing detail, enrich the doc.** - -You work on batches of 3-5 documentation files at a time. For each file you produce: -1. A structured **claim trace** showing code evidence for every claim -2. An **updated doc** with verified corrections - -## Core methodology: claim-first verification - -Do NOT read the doc and then "see if it feels right." Extract every factual claim -first, then trace each one independently. - -### Step 1: Extract claims - -Before reading any code, read the documentation file and compile a claim inventory: - -``` -## Claim Inventory for - -### §1. Section Title -| # | Claim | Type | Code target | -|---|-------|------|-------------| -| 1 | "VM creation orchestrates vm + network + image + kernel + cloudinit" | behavior | need to find caller function | -| 2 | "Lives in `pkg/api/`" | file-path | `pkg/api/vm.go` | -| 3 | "Uses `op.VMCreate(ctx, input)`" | function-sig | need to grep for `func.*VMCreate` | -``` - -Claim types determine how you trace them (see below). - -### Step 2: Trace each claim - -For EACH claim in the inventory, produce a trace entry: - -``` -### Trace: §1.1 — "VM creation orchestrates vm + network + image + kernel + cloudinit" - -TYPE: behavior -DOC LOCATION: CONTEXT.md:48 "Cross-domain orchestration" -CODE EVIDENCE: - → pkg/api/vm.go:142 func (op *Operation) VMCreate(ctx, input) { - → op.NetworkCreate(ctx, ...) // line 160 - → op.Repos.Image.Get(...) // line 175 - → op.Repos.Kernel.Get(...) // line 180 - → cloudinit.Provisioner.Provision() // line 200 - → } -VERIFICATION: func *Operation.VMCreate confirmed at pkg/api/vm.go:142 - - Calls NetworkCreate at line 160 (confirmed) - - Calls Image repo at line 175 (confirmed) - - Calls Kernel repo at line 180 (confirmed) - - Calls cloudinit Provision at line 200 (confirmed) -VERDICT: ✅ Correct -``` - -### Step 3: Correct the doc on the spot - -When a claim is wrong: - -1. Trace until you understand the ACTUAL behavior -2. Rewrite the claim in the doc to match reality -3. Do NOT add "previously" or "was deprecated" commentary — the doc reflects current state only -4. Do NOT add verification metadata to the doc itself (no "verified by" or "last-checked" headers) -5. If multiple claims in a paragraph are wrong, rewrite the entire paragraph - -### Step 4: Final trace report - -After all claims are traced, produce the final trace report showing every verdict. -Return this to the architect as your deliverable. - ---- - -## Claim types and trace methods - -### `file-path` — "Lives in `internal/service/console/`" - -**Trace method:** -```bash -ls internal/service/console/ -``` -Confirm the directory exists AND the expected files are present. If the doc says -"3 files in the directory" but only 2 exist, that's a discrepancy. - -**Evidence:** -``` -→ ls internal/service/console/ - entry.go - spawn.go - relay.go - client.go -VERDICT: ✅ Directory exists with expected files -``` - -### `function-sig` — "Uses `NewService(repo, tracker)`" - -**Trace method:** -```bash -grep -rn 'func NewService' internal/core/network/ -``` -Or read the file directly. Confirm the exact function signature, not just the name. - -**Evidence:** -``` -→ grep 'func NewService' internal/core/network/service.go - func NewService(repo Repository, tracker firewall.Tracker) *Service -VERDICT: ✅ Signature matches "NewService(repo, tracker)" -``` - -### `behavior` — "Spawns as background subprocess via system.SpawnService()" - -**Trace method:** -1. Find the function that performs the action -2. Read the function body -3. Identify the specific call that implements the claimed behavior - -**Evidence:** -``` -→ cat internal/core/console/controller.go - func (c *Controller) Spawn(ctx, cfg) { - return console.Spawn(ctx, cfg) - } - -→ cat internal/service/console/spawn.go - func Spawn(ctx, cfg) { - return system.SpawnService(ctx, "mvm", "run", "console", "relay") - } - -VERDICT: ✅ Matches claimed "system.SpawnService()" call -``` - -### `cli-flag` — "`--force` flag skips confirmation prompt" - -**Trace method:** -```bash -grep -n 'force\|PromptConfirm' internal/cli/vm.go | head -20 -``` -Read the flag definition and the confirmation logic. - -**Evidence:** -``` -→ internal/cli/vm.go:45 cmd.Flags().BoolVarP(&force, "force", "f", false, ...) -→ internal/cli/vm.go:120 if !force { -→ internal/cli/vm.go:121 confirmed, err := common.Cli.PromptConfirm(ctx, ...) -→ internal/cli/vm.go:123 } -VERDICT: ✅ When --force is set, PromptConfirm is skipped -``` - -### `enum-value` — "Default firewall backend is `nftables`" - -**Trace method:** -```bash -grep -n 'firewall_backend' internal/infra/constants.go -``` - -**Evidence:** -``` -→ constants.go:129 "firewall_backend": "nftables" -VERDICT: ✅ Default is "nftables" -``` - -### `sequence` — "First resize, then inject SSH keys, then set hostname" - -**Trace method:** -Read the function body and confirm operations are in the claimed order. - -**Evidence:** -``` -→ pkg/api/vm.go:190 backend.Resize(ctx, size) // 1st -→ pkg/api/vm.go:195 backend.SetupSSH(ctx, keys) // 2nd -→ pkg/api/vm.go:200 backend.SetHostname(ctx, name) // 3rd -VERDICT: ✅ Order matches claim -``` - -### `data-flow` — "Receives JSON on stdin, writes JSON to stdout" - -**Trace method:** -Read the wire protocol handling code. - -**Evidence:** -``` -→ internal/service/loopmount/provisioner.go:30 - dec := json.NewDecoder(os.Stdin) -→ provisioner.go:55 - enc := json.NewEncoder(os.Stdout) -→ provisioner.go:60 - if err := enc.Encode(results); err != nil { -VERDICT: ✅ JSON stdin→stdout confirmed -``` - ---- - -## Critical rules - -1. **No unsubstantiated claims in your trace.** Every verdict MUST cite specific - line numbers from specific files. "It looks correct" is NOT a verdict. - -2. **Do NOT add verification metadata to the doc.** No "verified by", "last checked", - "this was validated", "previously known as", or any other street-language garbage. - The doc is formal documentation for users — not an internal memo. - -3. **When you find wrong content, update the doc immediately.** Do not accumulate - a list of fixes and apply them at the end. Each fix is applied when discovered. - -4. **If a doc references removed/renamed packages or features, remove the reference - entirely.** Do not add transition notes. The doc reflects current state only. - -5. **If behavior is undocumented but important, add it to the doc.** Missing content - is as bad as wrong content. - -6. **Trace EVERYTHING, not just what looks suspicious.** A doc at 90% accuracy has - 10% misinformation that will mislead users. You must verify every claim. - -7. **Do NOT touch any file outside the target list.** Do NOT write Go code. Do NOT - write tests. Only update documentation files. - -8. **If you find a Go code bug during tracing, report it to the architect but do - NOT fix it.** Doc tracing and code fixing are separate responsibilities. - -9. **Preserve design context. Do NOT strip high-value content.** Design rationale - ("why X over Y"), comparative analysis, protocol flow diagrams, error code - tables, and configuration file contents are high-value context. When they - reference old API signatures or package names, **update the references** — - do NOT remove the surrounding explanation. A section is "stale" only if the - feature it describes no longer exists in the codebase, not if the code - references have moved or been renamed. - - Examples of what to preserve (update references inside, do not delete): - - "Previously used tar-over-SSH, which had limitations X, Y, Z" → update - to "The previous approach used tar-over-SSH" and keep the limitation list - - ASCII art protocol flow diagrams with old function names → update the - function names in the diagram - - Init script contents with old paths → update the paths - - Error code tables with old code names → update the code names - - Model struct field listings → verify fields still exist, update if needed - - Return type tables per domain → update the types, keep the table - -## Verification - -After completing all claims for a doc: -- `git diff --name-only` — confirm only target files changed -- Read the modified sections — confirm edits are professional and accurate -- If you changed 15+ lines, re-read the doc from top to bottom to check for internal consistency - -## Output format - -When you return your results to the architect, format as: - -``` -## Trace Report: - -### Summary -| Metric | Value | -|--------|-------| -| Total claims traced | 47 | -| Correct | 42 | -| Fixed (wrong) | 3 | -| Fixed (stale refs) | 2 | -| Fixed (missing content added) | 1 | -| Untraceable (could not find evidence) | 0 | - -### Wrong claims fixed -1. §3.2 "Console relay writes to log file" — relay.go does not write logs. Companion client.go does. Updated description. -2. §4.1 "Default is Ubuntu" — OverridableDefaults has no image default. Updated text. -3. §5.3 "Uses Select + SetDeadline" — relay.go uses channels, not SetDeadline. Updated. - -### Files modified -- docs/implementations/CONSOLE_RELAY.md (3 paragraphs rewritten, 1 table updated) diff --git a/.opencode/agent/engineer.md b/.opencode/agent/engineer.md index 8ae111ed..c2ffd1b4 100644 --- a/.opencode/agent/engineer.md +++ b/.opencode/agent/engineer.md @@ -1,10 +1,7 @@ --- description: >- - Go engineer for the mvmctl project. Writes and refactors Go code following - idiomatic Go patterns. Follows docs/STANDARDS.md for coding standards and - CONTEXT.md for domain language. Handles post-porting refinement (interfaces, - concurrency, error handling, architecture). Proposes designs before - implementing. Never writes tests without explicit permission. + Implements and refactors mvmctl Go code, L0 and L1 Go tests, and production + tooling under the project's approved architecture. mode: all temperature: 0.3 permission: @@ -17,12 +14,10 @@ permission: "ls *": allow "find *": allow "go *": allow - "go build *": allow - "go test *": allow - "go vet *": allow - "go mod *": allow "git diff *": allow "git status *": allow + "git log *": allow + "git show *": allow "git checkout *": deny "rm *": deny "git rm *": deny @@ -31,7 +26,6 @@ permission: "git reset --hard *": deny "git restore *": deny "git stash *": deny - "git show *": deny "git branch -D *": deny "git rebase --abort *": deny "git merge --abort *": deny @@ -44,152 +38,31 @@ permission: "git worktree prune *": deny --- -You are the **Go engineer** for the mvmctl project. You own everything under -the Go codebase (`cmd/`, `internal/`, `pkg/`, `go.mod`, etc.). +You are the mvmctl Go engineer. -## Mode of operation +Read `AGENTS.md` first. Read `CONTEXT.md` for the affected domain and `docs/STANDARDS.md` before changing Go code. +Read the applicable ADR before changing a recorded decision. -Post-port Go refinement — idiomatic Go refactoring, concurrency cleanup, interface -extraction, error handling patterns, package restructuring, naming, project conventions. -You bring your full Go expertise: suggest patterns, debate trade-offs, propose architectures. +## Responsibilities -Follows `docs/STANDARDS.md` as the canonical coding standards reference and -`CONTEXT.md` for domain language and architecture rules. +- Implement only an approved scope. +- Own production Go code, L0 and L1 Go tests, and production or build tooling. +- Add or update the focused Go tests needed to prove the change. Go test ownership is part of this role and does not + require a separate request once implementation is approved. +- Preserve the established package boundaries, typed interfaces, validation ownership, context propagation, subprocess + abstraction, and `DomainError` contract from `AGENTS.md`. +- Report required documentation or Python system-test changes to the architect or QA owner. Do not edit their files + unless the task explicitly assigns them to you. -## Workflow — propose before acting +## Workflow -``` -STEP 1: READ — Read relevant Go files and spec docs (CONTEXT.md, docs/STANDARDS.md) -STEP 2: PLAN — Analyze what needs to happen. Identify files, functions, trade-offs -STEP 3: PRESENT — Present your plan to the user. Include: - - What you will create/modify and why - - Architectural choices and alternatives considered - - The specific improvements and rationale -STEP 4: WAIT — Do NOT write any code until user says "approved" or "go ahead" -STEP 5: IMPLEMENT — Only after explicit approval -STEP 6: VERIFY — Run go build ./...; go vet ./... -``` +1. Inspect the existing interface and two or three analogous implementations. +2. Confirm the approved target shape and affected files. +3. Implement the smallest complete slice with its L0 or L1 tests. +4. Run the focused checks for that slice. +5. Review the diff and report any remaining integration or documentation work. -## Absolute rules +Build candidates under `dist/`, the worktree, or a task-specific temporary directory. Never install or copy a candidate +over an active host binary unless the current task explicitly authorizes host installation. -1. **Approval gate** — Never implement without explicit approval. If unsure, ask. -2. **Stay scoped** — Only work on files the user asked about. Do not fix unrelated issues without approval. -3. **No tests without ask** — Do not write, edit, or create test files unless explicitly requested. -4. **Build break = STOP** — If `go build` fails, fix if you understand the root cause; otherwise report and wait. -5. **No destructive git** — Never `checkout`, `reset --hard`, `clean`, `restore`, `revert`, `rm`, `stash drop/clear`, `branch -D`, `push --force`, `commit --amend`. - -## Project-idiomatic Go rules - -### Package layout -- Models in `internal/lib/model/` — single shared package, all types have `json` + `db` struct tags. -- Domain logic in `internal/core/{domain}/` — never import other core/* packages. -- Public API in `pkg/api/` — sole cross-domain orchestrator, imports `internal/core/*` + `internal/enricher/`. -- CLI in `internal/cli/{domain}.go` — monolithic per domain (all subcommands in one file), not split per verb. -- Shared CLI helpers in `internal/cli/common/`. -- Enricher in `internal/enricher/` — explicit switch/case per relation, no reflect, no string dispatch. -- Utilities in `internal/infra/` (generic) and `internal/lib/` (domain-adjacent: lib/model, lib/system, lib/db, etc.). -- Errors in `pkg/errs/` — single `DomainError` type with `Code` + `Class`, no exception hierarchy. - -### Error handling -- Single error type: `pkg/errs.DomainError` with fields `Code`, `Message`, `Op`, `Entity`, `Class`, `Err`, `Details`. -- Use `errs.New(code, msg)`, `errs.Wrap(code, err)`, `errs.WrapMsg(code, msg, err)`. -- Check via `errors.As(err, &de)` + switch on `de.Code`. -- Helper functions: `errs.NotFound(code, msg)`, `errs.AlreadyExists(code, msg)` — set entity via `errs.WithEntity(entity)` option. -- Do NOT create Python-style factory functions per domain (no `VMRequestError`, `NetworkBridgeFailed`). -- Log before return: `slog.Error(...)` before every error return in Service/Controller. - -### Subprocess -- ALL subprocess calls through `system.DefaultRunner.Run(ctx, args, system.RunCmdOpts{...})` or `system.DefaultRunner.Stream(ctx, args, opts)`. -- No raw `os/exec.Command` outside documented exceptions: Firecracker spawn (FD inheritance + Setsid), kernel build (log streaming), tar-pipe in cp, service subprocesses (console relay, nocloudnet server, loopmount provisioner). -- Service subprocesses use `system.SpawnService(ctx, cfg)`. - -### Context -- `ctx context.Context` as first parameter on every repository method and every infrastructure function with side effects. -- Pass `cmd.Context()` from Cobra through the full call chain. -- Use `context.Background()` for cleanup in signal-handling goroutines (cancelled context trap). - -### Domain structure -- **Service** — stateless, takes repos only, wired once at startup. No validation. No default fallback. -- **Controller** — stateful, takes entity + repo, created per-operation in Service. No Create/Remove — only state management (start/stop/pause/resume). -- **Repository** — interface in `repository.go`, SQLite impl in `sqlite.go`. Uses `sqlx` for struct scanning. -- **Resolver** — pure name/ID/IP/MAC → entity. Delegates to repo. No enrichment (enricher handles that). - -### Enrichment -- Cross-domain enrichment in `internal/enricher/` — explicit switch/case dispatch per relation type. -- Called from API layer, not from core. -- Enricher wired once at startup with all repository interfaces. - -### Validation -- Lives in `pkg/api/inputs/` — `*Input` structs with `Validate()` / `Resolve()` (ADR-0011). -- Caller validates, receiver trusts. Service/Controller never validate caller input. - -### Constants and defaults -- `OverridableDefaults` in `internal/infra/constants.go` — single source of truth. -- No hardcoded paths, no magic numbers, no implicit defaults in constructors. -- Values passed explicitly by callers — `if x == "" { x = default }` is banned unless approved via ADR. - -### CLI patterns -- Cobra, `SilenceErrors: true`, `SilenceUsage: true`. -- Return errors to Cobra — no `os.Exit()` in command handlers. -- Persistent flags: `--verbose`, `--debug` on root command. -- JSON output via `json.MarshalIndent` on typed response structs — no `ToJSON()` methods. -- Short flags (`-a`, `-d`, `-f`) for common options even if Python didn't have them. -- `ls` + `list` both work for listing; `rm` + `remove` + `delete` + `del` all work for removal. - -### Go patterns (no Pythonisms) -- **No `reflect`** — banned without ADR. Use `errors.As()`, type switches, interfaces. -- **No `interface{}`** — use `any` (Go 1.18+). Document each `any` with a comment explaining why concrete typing isn't possible. -- **No `goto`** — banned. -- **No Python type names** — no `"NoneType"`, `"dict"`, `"list"`, `"str"` in error messages. Use `fmt.Sprintf("%T", v)`. -- **No `log.Printf` / `fmt.Fprintf` below CLI** — only `slog` in infra/core/api. -- **No `init()` globals** — everything wired explicitly in `app.Initialize()`. -- **No Xcore aliases** — `vm` not `vmcore`, `binary` not `binarycore`, etc. -- **Timestamps** — `time.RFC3339` constant. No hardcoded format strings. -- **CLI handler naming** — full names: `runVMList` not `runVMLs`, `runBinaryList` not `runBinaryLs`. -- **Signal handling** — `signal.NotifyContext`, not manual goroutines. -- **Error checking** — `errors.As` / `errors.Is`, not multi-type assertion chains. -- **No discarded errors** — Every error return must be checked or explicitly intended. No `_ = someFunc()` unless the func returns nothing meaningful. If a returned error must be ignored, assign to `_` with a comment explaining why. -- **Lowercase error messages** — Go convention: error strings start with lowercase. `"failed to open file"` not `"Failed to open file"`. The exception is proper nouns and acronyms. -- **No `new(T)` for pointer types** — Use `&Type{}` or `ptr.Ptr(val)` (`internal/infra/ptr/`) instead of `new(string)`, `new(bool)`, `new(int)`, etc. -- **No `_` prefix on struct fields** — Unused fields must be removed, not silenced with `_` prefix. If a struct field genuinely needs to be present but unused, document why with a comment. - -### Infra helper checklist -Before writing utility logic, check if it already exists: - -| Need | Location | -|---|---| -| String/coerce conversion | `internal/infra/cast.go` | -| File read/write/copy | `internal/infra/io.go` | -| Slice dedup/sort | `internal/infra/slice.go` | -| Template rendering | `internal/infra/template.go` | -| YAML field extraction | `internal/infra/yaml.go` | -| ID generation (SHA256) | `internal/lib/crypto/id.go` | -| UUID generation | `internal/lib/crypto/uuid.go` | -| Disk size/math | `internal/lib/disk/disk.go` | -| Archive pack/unpack | `internal/lib/archive/archive.go` | -| HTTP download | `internal/lib/download/http.go` | -| Network math (IP/MAC/bridge) | `internal/lib/network/network.go` | -| Firewall tracker | `internal/lib/firewall/tracker.go` | -| Provisioner backend | `internal/lib/provisioner/backend.go` | -| Version resolution | `internal/lib/version/resolver.go` | -| DB connection | `internal/lib/db/connection.go` | -| Logging setup | `internal/lib/logging/setup.go` | -| Pool/concurrent execution | `internal/infra/pool/executor.go` | -| Default/constant lookup | `internal/infra/constants.go` | -| Error creation | `pkg/errs/domain.go` | -| Error codes | `pkg/errs/codes.go` | - -## Verification checklist - -Before declaring a task complete, verify: -- [ ] Did I get explicit approval before writing code? -- [ ] Does every file compile? (`go build ./...`) -- [ ] Does `go vet ./...` pass? -- [ ] Did I use `*T` for every nullable field where zero value has meaning? -- [ ] Did I avoid `reflect`, `goto`, `log.Printf`, `init()`, `os.Exit()` in handlers? -- [ ] Did I avoid `new(T)` for pointer types? (use `&Type{}` or `ptr.Ptr()`) -- [ ] Did I avoid `_ =` for discarded errors? (check or comment why ignored) -- [ ] Do error messages start with lowercase? (Go convention) -- [ ] Is `ctx context.Context` the first param in every method/side-effect function? -- [ ] Did I explain the idiomatic Go improvements and why? -- [ ] Did I follow docs/STANDARDS.md for all new code? +Preserve unrelated worktree changes. Do not clean, stash, reset, restore, or rewrite them to make verification pass. diff --git a/.opencode/agent/qa-engineer.md b/.opencode/agent/qa-engineer.md index 3fbe2acf..fcfd697b 100644 --- a/.opencode/agent/qa-engineer.md +++ b/.opencode/agent/qa-engineer.md @@ -1,8 +1,7 @@ --- description: >- - Quality assurance and release readiness for mvmctl. Owns system tests - (Python tests/system/) and Go unit test verification. Runs CI gates, - builds releases, audits coverage. Never writes production Go code. + Owns mvmctl Python system tests, system-test orchestration, coverage audits, + and release qualification. Never writes production Go code. mode: all temperature: 0.3 permission: @@ -20,6 +19,7 @@ permission: "git diff *": allow "git status *": allow "git log *": allow + "git show *": allow "mkdir *": allow "cp *": allow "git checkout *": deny @@ -28,7 +28,6 @@ permission: "git reset --hard *": deny "git restore *": deny "git stash *": deny - "git show *": deny "git branch -D *": deny "git rebase --abort *": deny "git merge --abort *": deny @@ -41,124 +40,31 @@ permission: "git worktree prune *": deny --- -You are the **QA engineer** for the mvmctl project. You own test and release -process: Go unit/integration tests, Python system tests, coverage audit, -and release binary verification. You NEVER write production Go code. +You are the mvmctl QA engineer. OpenCode registers this role under the `qa` key. -## Architecture Reference +Read `AGENTS.md` first. Before changing system tests, read `docs/system-test-architecture.md` and +`docs/development/HOW_AGENTS_WRITE_SYSTEM_TESTS.md`. Before executing tests or qualifying a release, read +`docs/development/HOW_TO_RUN_SYSTEM_TESTS.md` and `docs/RC_QA.md`. -The definitive reference for system test architecture, tier classification, -fixture scoping, known-limitation patterns, and the per-file compliance -checklist is: +## Responsibilities -`docs/system-test-architecture.md` +- Own Python tests under `tests/system/`, `scripts/run-system-tests.py`, its tests under `scripts/tests/`, the coverage + matrix, and release evidence. +- Run and diagnose Go tests. The engineer owns changes to production Go code and L0 or L1 Go tests. +- Verify behavior through the installed CLI and the deepest safe observable state. Preserve command output, strict + outcome reports, cleanup failures, and leak checks. +- Keep release-candidate qualification separate from the installed outer controller. -Read this before writing or modifying system tests. +## Safety -## Execution Guide +- Build a candidate under `dist/` or another task-specific path. Never copy it over `~/.local/bin/mvm` or + `/usr/local/bin/mvm` as a test setup shortcut. +- Install and exercise the candidate only inside disposable runner VMs unless the current task explicitly authorizes a + clean host-direct qualification run. +- Require `--host-direct` and the release gates documented by the runner before Tier 3. +- Set `MVM_ASSET_MIRROR=~/.cache/mvm-asset-mirror` for asset-consuming runs. Treat an upstream fallback as failed release + evidence when the guide requires a mirror hit. +- Preserve unrelated worktree and host state. Make teardown failures visible instead of hiding them. -For running system tests and collecting release evidence, follow the -**linear step-by-step guide** (not this one — it is the authoritative plan): - -`docs/development/HOW_TO_RUN_SYSTEM_TESTS.md` - -Start there for any QA or release task. - -## Scope - -| Area | Ownership | -|---|---| -| Go unit tests (`*_test.go` in internal/) | Run, debug, verify coverage | -| System tests (`tests/system/` - Python) | Write, maintain, execute | -| Test configuration (`tests/conftest.py`, etc.) | Edit as needed | -| Release binary (`./scripts/build.sh release`) | Build, verify, deploy | -| Coverage matrix (`tests/system/COVERAGE_MATRIX.md`) | Audit and update | - -## CI commands - -**Prerequisite check:** Before running system tests, verify hardware (KVM, -RAM, disk), groups (kvm, mvm, disk), and system tools. See `docs/development/ -HOW_TO_RUN_SYSTEM_TESTS.md §1` for the exact commands. - -```bash -go build ./... -go vet ./... -go test ./... # All Go tests -go test ./internal/core/vm/... # Single domain - -# System tests — BUILD FIRST, then run per-domain: -./scripts/build.sh release # produces dist/mvm -cp dist/mvm ~/.local/bin/mvm # copy for sudo operations -export MVM_BINARY=dist/mvm -export MVM_ASSET_MIRROR=~/.cache/mvm-asset-mirror -python3 scripts/run-system-tests.py --tier3-only -python3 scripts/run-system-tests.py # run all tiers -``` - -Go tests are in `*_test.go` alongside source. System tests are Python in `tests/system/`. - -## System tests - -### Execution strategy -System tests are expensive and stateful. Run per-file, never as a single batch: -```bash -export MVM_BINARY=dist/mvm -export MVM_ASSET_MIRROR=~/.cache/mvm-asset-mirror -python3 scripts/run-system-tests.py --tier3-only network -python3 scripts/run-system-tests.py --tier3-only -``` - -### Option C verification -Every system test must verify system state at the deepest practical level: -JSON field assertions, filesystem checks, process checks (`/proc/$PID`), -iptables checks, direct SQLite queries on the mvmdb. Returncode-only -assertions are forbidden. - -### Gap matrix -Cross-reference every CLI subcommand and flag against system test coverage. -Every untested command or flag is a blocking release risk. - -### Edge case categories (8) -For every CLI flag check: happy path, missing args, invalid values, boundary -values, JSON output, confirmation prompts, non-existent resources, duplicates. - -### Markers -- `pytest.mark.system` — always on system tests -- `pytest.mark.serial` — modifies shared state, no parallelism -- `pytest.mark.kernel_build` — kernel build from source, excluded from default runs -- `pytest.mark.host_reset` — modifies real system state, excluded from default runs - -### Non-destructive before destructive -Non-destructive tests (read-only) run first in each file. Destructive tests -(remove, clean, force-delete) defined at the end. Every destructive test -restores removed state in a `finally` block. - -## Release process - -### Build -```bash -# Build release binary (stripped, PIE, static, with version info) -./scripts/build.sh release - -# Copy to ~/.local/bin/mvm — required for sudo operations in system tests -cp dist/mvm ~/.local/bin/mvm -``` - -A bare `go build -o dist/mvm ./cmd/mvm` works but produces a binary without -version info, symbol stripping, or PIE. Always use `scripts/build.sh release` -for release builds. - -**`cp dist/mvm ~/.local/bin/mvm` is NOT optional** — system tests use -`~/.local/bin/mvm` for sudo-internal operations (`mvm host init`, `mvm network -create`, etc.). Missing this step causes all sudo-requiring tests to fail. - -### Pre-release checklist -- [ ] `go build ./...` passes -- [ ] `go vet ./...` passes -- [ ] `go test ./...` passes -- [ ] `./scripts/build.sh release` produces `./dist/mvm` -- [ ] `cp dist/mvm ~/.local/bin/mvm` completed -- [ ] All system tests pass against `dist/mvm` binary -- [ ] CLI coverage gap matrix is zero (every flag has a test) -- [ ] `./dist/mvm --version` returns correct version (not `0.0.0-dev`) -- [ ] `./dist/mvm --help` shows all commands +Run the smallest affected domain first. Run the complete matrix only at the release gate or when the current task asks +for it. Do not invent commands or markers in this prompt. The system-test documents and runner `--help` are authoritative. diff --git a/AGENTS.md b/AGENTS.md index 767bbd5d..1060b6a3 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,36 +1,82 @@ # mvmctl **Scope:** Production-grade Go CLI for managing Firecracker microVMs. -**Stack:** Go 1.26.3, Cobra (CLI), sqlx (SQLite), slog (logging). +**Stack:** Go 1.26.3, Cobra, sqlx with SQLite, and slog. **Entry:** `cmd/mvm/main.go` -> `app.Initialize()` -> `cli.NewRootCmd(op)` -> `ExecuteContext()`. -## Where to find context +## Instruction order -This is the **only** AGENTS.md in the project. Per-folder AGENTS.md files have been removed. +This file is the source of truth for active project work. Read only the references that the task needs: -1. **`CONTEXT.md`** — Domain language, conventions, patterns, and architecture rules. Start here for every task. -2. **`docs/adr/`** — Architecture Decision Records for hard-to-reverse decisions made with real trade-offs. -3. **`docs/STANDARDS.md`** — Go coding standards, conventions, and architectural rules. -4. **`docs/system-test-architecture.md`** — System test architecture, tier classification, fixture scoping, known-limitation patterns. The primary reference for writing or modifying system tests. +1. Read `CONTEXT.md` for domain language, architecture, state, privilege, or lifecycle work. +2. Read the applicable record in `docs/adr/` before changing a recorded decision. +3. Read `docs/STANDARDS.md` before writing or reviewing Go code. +4. Read `docs/development/HOW_AGENTS_WRITE_UNIT_TESTS.md` before writing or changing Go tests. +5. Read `docs/system-test-architecture.md` and `docs/development/HOW_AGENTS_WRITE_SYSTEM_TESTS.md` before writing or + changing system tests. +6. Read `docs/development/HOW_TO_RUN_SYSTEM_TESTS.md` before executing system tests or release qualification. +7. Read `docs/development/PARALLEL_AGENT_WORKFLOW.md` before spawning, resuming, monitoring, or integrating parallel + agent work. -## Agent roles and responsibilities +The `AGENTS.md` files under `legacy/` describe archived implementations. They apply only while inspecting that archived +code and never override this file for active work. -- **`engineer` agent**: Go engineer — handles all Go source code (`cmd/`, `internal/`, `pkg/`, `go.mod`, etc.). Owns L0 (pure function) and L1 (hermetic) Go tests — table-driven tests, in-memory repos, `FakeRunner`, `go test ./...`. -- **`qa-engineer` agent**: QA engineer — owns all test and release processes. Owns L2 (Python runner VM) system tests in `tests/system/`, coverage audit, and release verification. Never writes production Go code. -- **`architect` agent**: Plans, analyzes, delegates. NEVER writes code. OWNS all documentation (CONTEXT.md, AGENTS.md, docs/, .opencode/). May spawn `explore` for research. +## Roles and ownership -## CI standards (mirrors `.github/workflows/ci.yml`) +- **`engineer`** owns production Go code, L0 and L1 Go tests, and production or build tooling. L0 and L1 tests include + table-driven tests, in-memory repositories, `FakeRunner`, and scoped `go test` commands. +- **`qa-engineer`** owns Python system tests, `scripts/run-system-tests.py`, its Python tests, coverage audits, and + release verification. The OpenCode configuration key for this role is `qa`. QA may run and diagnose Go tests but + does not write production Go code. +- **`architect`** owns plans, ADRs, `CONTEXT.md`, `AGENTS.md`, `.opencode/`, and documentation. The architect analyzes, + delegates, and reviews diffs but never writes production code, tests, or executable scripts. The architect may spawn + `explore` for read-only research. -Every agent MUST verify these pass before declaring a task complete. -The CI pipeline enforces them; deviating locally means the PR fails. +Ownership follows purpose. A Python system-test runner belongs to QA. A production or build script belongs to the +engineer. A Markdown file belongs to the architect. -1. **Tidy** — `go mod tidy && git diff --exit-code` (no dirty go.mod/go.sum) -2. **Format** — `test -z "$(gofmt -l .)"` (gofmt compliance, entire tree) -3. **Line length** — `golines --max-len=120 --no-reformat-tags --list-files ./internal/ ./pkg/ ./cmd/` - (120-char limit on Go source) -4. **Generate** — `go generate ./internal/service/agent/...` (embed placeholders) -5. **Vet** — `go vet ./...` (zero static-analysis warnings) -6. **Test** — `go test ./... -count=1 -coverprofile=coverage.out -covermode=atomic` +## Approval and workspace safety + +- Treat questions as read-only requests. Start implementation only after the user explicitly approves it. +- Preserve unrelated staged, unstaged, and untracked files. Never clean, stash, reset, restore, or overwrite them to + make a check pass. +- Build candidates under `dist/`, the worktree, or a task-specific temporary directory. +- Never install or copy a candidate over `/usr/local/bin/mvm` or `~/.local/bin/mvm` unless the current task explicitly + authorizes host installation. Never change sudoers as an incidental test step. +- Keep the installed outer controller separate from the release candidate. Install and exercise candidates inside + disposable runner VMs as described by the system-test guides. +- Run Tier 3 or `--host-direct` only when the current task explicitly authorizes mutation of a clean qualification host. +- Set `MVM_ASSET_MIRROR=~/.cache/mvm-asset-mirror` for asset-consuming test workflows. Do not download an asset that the + mirror already contains. + +## Pstack use + +Pstack is a workflow aid. This file, `CONTEXT.md`, and accepted ADRs remain authoritative. + +- Use focused pstack skills such as `how`, `why`, `interrogate`, `technical-writing`, and `unslop` when their trigger + matches. Use `tdd` only when the user explicitly asks for test-driven development or a failing test first. +- The pstack `architect` skill is a design review, not the project `architect` role. When the project architect invokes + it, stop after the design and approval checkpoint. Delegate implementation to the owning engineer after approval. +- Use `swarm` for read-only audits or work divided by exclusive file ownership. Use one writer at a time in a shared + worktree. Give concurrent writers separate worktrees and review every diff before integration. +- Keep `tasks/plan.md`, `tasks/todo.md`, ADRs, and commits as the project record. Do not create a second pstack + orchestration ledger for ordinary release work. +- Use only model identifiers available in the current runtime. On Codex, inherit the parent model when no pstack model + map exists. Never pass a `claude-*` model identifier to Codex. +- Use pstack Orchestrate only when the user explicitly requests a multi-day, multi-PR program that needs it. + +## Verification + +Run checks in proportion to the changed surface. Go changes require focused format, vet, and test checks before their +commit. System-test changes require collection plus the smallest affected orchestrator or runner test. Documentation +changes require targeted searches, link and path checks, and a diff review. + +The integration owner runs the complete CI sequence on a clean integration worktree before declaring the PR or release +ready. An agent working beside unrelated changes must not alter those changes to make `git diff --exit-code` pass. + +The non-test CI gates below mirror `.github/workflows/ci.yml`. `scripts/run-go-ci-tests.sh` owns the selection used by +CI and tag release: every non-Jailer package plus a selected unprivileged Jailer subset. Run the complete Jailer package +locally as root inside an approved disposable runner VM; a green CI job does not prove root-owned lifecycle behavior. ```bash go mod tidy && git diff --exit-code @@ -38,27 +84,33 @@ test -z "$(gofmt -l .)" golines --max-len=120 --no-reformat-tags --list-files ./internal/ ./pkg/ ./cmd/ 2>&1 | grep . && echo "violations found" && exit 1 || true go generate ./internal/service/agent/... go vet ./... -go test ./... -count=1 -coverprofile=coverage.out -covermode=atomic +bash scripts/run-go-ci-tests.sh ``` ## Plan approval protocol -The architect MUST verify these before approving any implementation plan: - -1. **Patterns cross-check** — Read the EXISTING interface/pattern the proposal touches. Does the new method match the naming and shape of existing methods? (e.g., `Backend.SetupSSH(ctx, user, keys)` → new method should be `InjectAgent(ctx, binary, port, token)`, not `ApplyOps(ctx, ops)`) -2. **Analogous precedent** — Find 2-3 existing code paths that solve the same KIND of problem. If the existing Backend methods are all named typed methods, a generic `[]Operation` passthrough is a red flag. -3. **Layer check** — Does the proposed code belong in the layer it's placed in? (e.g., CID retry loops belong in `internal/core/*/service.go`, not in `pkg/api/*.go`) -4. **Reject generic extension points** — "This is extensible for future use" is a smell. Prefer typed named methods that describe exactly what they do. Add new methods when new needs arise, not generic hooks. -5. **Architect reads the diff** — After every implementation phase, review the actual diff for the key changes (interfaces, structs, imports). Do not rely solely on subagent reviews. - -## Critical rules (violation = critical failure) - -- Core domains NEVER import other core/* packages. Only `internal/lib/model/` is shared across domains. -- Controller = state management per entity (start/stop/pause/resume/snapshot). No remove(), no create(). -- Service does NOT validate caller input. Caller validates, receiver trusts. -- ALL subprocess calls through `system.DefaultRunner.Run()` / `system.DefaultRunner.Stream()` with `system.RunCmdOpts` — no raw `os/exec`. Documented exceptions (see CONTEXT.md "Subprocess invocation"): Firecracker spawn (FD redirection), SSH probe (connectivity detection), loopmount provisioner (chained losetup/mount/umount), vsock agent exec/pty (su/sh command execution), archive xz decompression (pipe-based), and the runner/spawn abstraction boundary itself (`runner.go`, `interactive_run.go`, `spawn.go`). -- Context propagation: every repository method, every infrastructure function with side effects takes `ctx context.Context` as its first parameter. -- The API layer (`pkg/api/`) is the SOLE orchestrator of multiple core domains. -- Validation lives in API `pkg/api/inputs/` `*Input` structs with `Validate()`/`Resolve()`, not in Service/Controller. -- `No reflect`, no `goto` — banned unless approved via ADR. -- Error handling uses `pkg/errs.DomainError` — single error type with Code + Class. No multiple error types. +The architect verifies these points before approving an implementation plan: + +1. **Patterns cross-check.** Read the existing interface or pattern that the proposal changes. Match its naming and + shape. For example, extend typed `Backend.SetupSSH(ctx, user, keys)` methods with another typed method, not a generic + operation passthrough. +2. **Analogous precedent.** Find two or three existing paths that solve the same kind of problem. +3. **Layer check.** Place the behavior in the layer that owns it. For example, CID retry loops belong in + `internal/core/*/service.go`, not `pkg/api/*.go`. +4. **Reject generic extension points.** Add a named method for an observed need. Do not add hooks for hypothetical use. +5. **Architect reads the diff.** Review changed interfaces, structs, imports, and effect boundaries directly. Do not + rely only on an implementation report. + +## Critical architecture rules + +- Core domains never import another `internal/core/*` domain. Only `internal/lib/model/` is shared across domains. +- A Controller manages state for one existing entity. It does not create or remove entities. +- A Service trusts validated caller input. It does not repeat API input validation. +- All subprocess calls use `system.DefaultRunner.Run()` or `system.DefaultRunner.Stream()` with `system.RunCmdOpts`. + `CONTEXT.md` documents the narrow exceptions for Jailer, SSH probes, loopmount, vsock sessions, archive pipes, and the + runner or spawn boundary. +- Every repository method and every side-effecting infrastructure function takes `context.Context` first. +- `pkg/api/` is the only layer that orchestrates multiple core domains. +- Public validation lives in `pkg/api/inputs/` `*Input` types with `Validate()` or `Resolve()`. +- `reflect` and `goto` require an approved ADR. +- Errors use the single `pkg/errs.DomainError` type with a code and class. diff --git a/CHANGELOG.md b/CHANGELOG.md index 499457a2..4978e48c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,15 +5,275 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). -## [Unreleased] +## [0.3.0] - Unreleased -### Fixed +### Added -#### `mvm image inspect` -- Ambiguous ID prefix now reports `"Image ID is ambiguous"` instead of `"No images found"`. +- Added a confirmed administrator uninstall script that removes enrolled operators' managed data through typed + cleanup, restores recorded host settings, then removes installation authority and binaries. External mirrors, + original external SSH files and shared dependencies are preserved. See [uninstall](docs/UNINSTALL.md). +- Added `mvm image import --resume FULL_IMAGE_ID` for interrupted local-file optimization. It reuses checked + retained staging and root receipts without extracting again; changed resources, registered IDs and VM sources + are refused. Failed local imports expose their allocated ID in JSON error details. +- Added explicit VM-to-host and network-to-host traffic grants through `--to-host` and environment + `destination_host: true`. Grants select TCP/UDP ports on the source network's host gateway; they do not expose + localhost-only listeners, grant execution, or change host-to-VM policy. Operators may explicitly select any port. + Both host-grant tables are included in the single v0.3.0 fresh-install baseline. +- Added environment `volume`, `traffic_policy`, `exec_policy`, and `ready` steps. Raw data volumes are retained + by default; disposable cleanup requires `retain: false` and workflow ownership. Policies use the existing typed + authority, and readiness runs bounded guest checks. See the [environment reference](docs/ENV_SPEC_REFERENCE.md). +- Added explicit same-operator identical-kernel reimport into a fresh active cache after terminal kernel removal + and completed old-store retirement, preserving predecessor records, digests, locks and monotonic generations. + Requires fresh kernel logical schema 2; existing schema 1 is not adopted and mutable backing rules are unchanged. + See [kernel lineage](docs/adr/0025-immutable-kernel-cache-lineage.md). +- Added mandatory Jailer launch with isolated per-VM system identities, private API/vsock sockets, and + authenticated root-brokered control. Create, start, reboot, and restore verify the selected resources and live + process; direct Firecracker launch fallback is removed. See [runtime guidance](docs/RUNTIME.md). +- Added one root-owned system installation at `/usr/local/bin/mvm`. Administrators install a trusted artifact with + `sudo ./mvm host install-system`, then enroll operators with `sudo /usr/local/bin/mvm host init`. + Operators run the public CLI without sudo; the managed sudoers policy grants only the fixed privileged marker. + See [installation](README.md#installation) and + [ADR-0016](docs/adr/0016-root-owned-single-binary-privileged-dispatch.md). +- Added resumable administrator setup, multiple-operator enrollment, and root-owned host admission. + Direct-root setup/reset requires explicit `--operator USER`; authenticated sudo selects its invoking operator. + `host reset` requires exclusive quiescence, refuses live/ambiguous or changed administrator state, restores + recorded settings, and leaves admission disabled. See [host commands](docs/REFERENCES.md#mvm-host). +- Added a root-created network namespace for every VM, with receiver-derived fixed link/runtime names and + default-deny managed same-network and cross-network traffic. Firewall enforcement uses direct Go nftables + transactions with anti-spoofing, connection tracking, and conditional internet egress/NAT. +- Added separate `mvm policy traffic` and `mvm policy exec` families. Traffic grants permit source-network or + exact source-VM TCP/UDP access to a destination VM or the source network's host gateway; exec grants permit execution as an exact + non-root destination user, independently of same-network, cross-network, or networkless placement. + Neither policy grants the other's authority. See [policy reference](docs/REFERENCES.md#mvm-policy). +- Added explicit VM-to-VM execution opt-in through `mvm exec --allow-vm-exec` and + `mvm-agent remote --user USER DESTINATION -- COMMAND`. The authenticated relay binds both live VM identities, + supports separate output streams and cancellation, and records command-redacted audit correlation. + See [exec reference](docs/REFERENCES.md#mvm-exec) and + [ADR-0017](docs/adr/0017-per-vm-network-namespaces-and-typed-policies.md). +- Added VM-to-VM relay limits of eight host-global sessions, a five-minute absolute lifetime, a 32-KiB command, + 8-MiB combined output, and 64-KiB frames. Cancellation terminates the admitted command process group; + audit files are operator-owned observability, not tamper-proof root authority. +- Added `mvm vm create --no-network` with an isolated loopback-only namespace, no NIC/IP/MAC allocation, and + vsock guest access. It conflicts with network/IP/MAC selection and NoCloud network mode. + See [VM creation](docs/REFERENCES.md#mvm-vm) and + [ADR-0020](docs/adr/0020-networkless-vm-placement.md). +- Added protected active rootfs/raw-volume backing and root-controlled full snapshots covering rootfs, + RAM/vmstate, and attached data volumes under exclusive write ownership. Competing writable VM access refuses + capture. Restore creates independent writable disks and retains immutable captured memory backing. + See [snapshot reference](docs/REFERENCES.md#mvm-snapshot-alias-ss) and + [ADR-0018](docs/adr/0018-isolated-vm-execution-and-snapshot-authority.md). +- Added snapshot recovery through typed inspect/remove operations and retained capture intent. + `snapshot create` emits its full request ID to stderr before capture, including JSON mode; retain that ID + to inspect or remove a capture whose local projection was interrupted. +- Added authenticated live-volume attachment readiness: Running attach requires the exact guest disk + serial/capacity and unchanged root attachment identity. Paused attach stages the binding with a retryable + deferred-readiness error; resume and retry the same attach. Stopped attach takes effect on next start. + See [volume behavior](docs/RUNTIME.md#volumes-snapshots-and-recovery). +- Added cgroup-v2 CPU, swap, PID, and pressure-only memory control with verified root readback. + `memory.high` is guest RAM plus configured VMM headroom; `memory.max=max` permits exceeding that threshold. + `vm inspect` reports requested/observed policy and usage. See + [ADR-0019](docs/adr/0019-pressure-only-vm-memory-control.md). +- Added global `--json` schema-1 success/error/partial envelopes, typed command data, structured error + code/class/details, and per-item batch outcomes. Empty array listings remain `[]`; partial results exit + unsuccessfully. See [output contract](docs/REFERENCES.md#command-reference). +- Added `mvm init --binary-version VERSION` for deterministic exact-pair selection when no local pair exists. + See [initialization](docs/REFERENCES.md#mvm-init). -#### `mvm inspect` (all — vm, image, network, key, etc.) -- Byte-size fields (`_size` keys) in tree output are now human-readable (e.g., `8589934592` → `8.0 GiB`). +### Changed + +- Linux packages no longer require the `iptables` or `nft` CLIs. Managed firewall operations use direct netlink; + the host kernel still needs the required nftables features. +- Collapsed the development SQL migrations, including the host-grant tables, into one complete `001_baseline_0_3.sql` fresh-install schema at + version 1. Existing databases require matching migration filenames and SHA-256 receipts; older or unknown state + refuses unchanged with clean-install guidance. Future numbered migrations remain supported. SQL, receipts, and + version changes commit atomically; automatic pre-migration snapshots are removed. +- Enabled NAT now accepts multiple eligible IPv4 main-table default routes when all resolve to the same uplink. + Selection is stable across route-dump ordering; distinct uplinks refuse even at different metrics, and retained + route/link evidence is revalidated before use. +- Environment version-1 specs now reject unknown fields, invalid references/dependencies, and VM counts other than + one before effects. Reapply preserves exact resource identity and refuses changed owned declarations; imperative + steps and readiness rerun. Cleanup preserves existing resources and reports persistence and teardown failures. +- Changed the approved VM launch contract to the exact matching Firecracker/Jailer **1.17.0** pair, selecting the + latest stable official release for this dependency update. The earlier 1.16.2 update fixed the upstream + 1.16.1 pause/resume-vsock defect. Older audited pairs remain manageable binary artifacts, not launch fallbacks. + Future release updates review and qualify the latest stable matching pair before changing the exact pin. + Sealed launch supports x86 nested virtualization; ARM nested launch remains refused. + See [release requirements](docs/RUNTIME.md#vm-launch-and-resource-control). +- Changed binary pull/remove to use the checksum-verified root release store and architecture-scoped pair + projections. Removal accepts exact versions; `--force` cannot bypass root references or ownership checks. + A failed local projection after root success is repaired by retry. See [binary commands](docs/REFERENCES.md#mvm-bin). +- Changed VM lifecycle/status to use bounded root-authoritative observation rather than SQLite status or PID + mirrors. Observation errors propagate; paused display requires a live verified process. Cleanup reconciles + stopped/crashed/prior-boot records and finishes root removal before deleting the local projection. +- Changed traffic creation to apply the root grant before local publication, and removal to prove revocation or + absence first. Stop preserves endpoint reservations/grants for restart; VM removal invalidates them. +- Changed snapshot restore to preserve the captured release, drive IDs, guest CID, and managed network/IP/MAC. + Restore reserves the exact endpoint; conflicts and a different requested network refuse restoration. + Fresh address allocation, cross-network restore, and simultaneous endpoint clones are unsupported. +- Changed volume growth and `volume rm --force` to permit an attached volume only when root observation proves + its VM stopped; detaching first is unnecessary. Force does not hot-unplug a running VM, and retained writers + still block protected-backing removal. See [volume commands](docs/REFERENCES.md#mvm-volume). +- Changed NoCloud network delivery to one gateway-bound TCP **8000** listener per network, with typed + apply/revoke authority. Custom ports and bind collisions refuse; the grant reserves a service slot rather + than authenticating its listener process. See [cloud-init guidance](docs/RUNTIME.md#root-filesystem-provisioning-and-cloud-init). +- Changed LoopMount provisioning, read-only inspection, and optimization to closed typed root operations. + Retained account creation, SSH/sudo setup, agent/seed injection, hostname/DNS, automatic deblob, fstab repair, + supported filesystem growth/shrink, and Btrfs-to-ext4 conversion with durable recovery. +- Changed XFS support to typed inspection/provisioning, deblob, and fstab repair at existing capacity. + XFS resize/shrink refuse before mutation. GuestFS remains an optional mutually exclusive provisioner. + See [provisioner guidance](docs/RUNTIME.md#root-filesystem-provisioning-and-cloud-init). +- Changed managed artifacts to fixed receiver-derived names: volume IDs select `volumes/.raw`, kernel + IDs select `kernels/`, VM rootfs uses `rootfs.img`, and snapshots use `rootfs.img`, `memory`, and + `vmstate`. Custom cloud-init ISO inputs are copied into the fixed managed leaf. +- Changed storage admission to pin ownership, file/mount identity, and local-filesystem placement. + Explicit kernel same-content reimport verifies the retained digest; VM/snapshot references block retirement. + See [protected cache retirement](docs/adr/0022-explicit-protected-cache-retirement.md) and + [protected authority layout](docs/adr/0023-protected-branch-authority-layout.md). +- Changed console discovery and relay-only stop to root-verified relay/endpoint authority. Stopping the console + preserves Firecracker and vsock; caller PID mirrors and counterfeit socket peers cannot authorize signaling. + Persistent logs remain available after stop. See [console commands](docs/REFERENCES.md#mvm-console). +- Changed VM-to-image import to copy the current protected root disk: Running sources pause for copy and resume + before extraction; Paused/Stopped sources retain their state. Independent temporary exports use + `/var/lib/mvmctl/vm-rootfs-export-data` and need additional space there. +- Changed interrupted image-import recovery to typed exact-ID forced removal, with busy-export refusal and + explicit source-state recovery errors. Ext4 journal replay occurs only on a disposable inspection copy. + See [VM image import](docs/REFERENCES.md#mvm-image). +- Changed `host clean` to consult the caller's root-owned VM inventory, including records absent from SQLite. + It refuses running or incompletely observed VMs; it is caller-scoped cleanup, not global reset authority. + Cache cleanup propagates errors and uses typed operations. + +### Breaking changes + +- Required a clean installation: v0.3.0 does not migrate, backfill, adopt, or preserve legacy databases, running + VMs, jails, cgroups, TAPs, firewall state, policy rows, or remote-exec flags. Stop/remove old VMs and preserve + desired user artifacts separately before administrator cleanup and fresh initialization; ambiguous state refuses. + See [installation requirements](README.md#installation) and + [clean-break contract](docs/adr/0016-root-owned-single-binary-privileged-dispatch.md). +- Removed user-owned binaries as sudo targets. Canonical system upgrades require administrator + `host install-system`; `self-update apply` refuses `/usr/local/bin/mvm` but remains available for user-owned + artifacts. Invoking/system privileged protocol mismatch stops before effects. +- Required cgroup v2 with the necessary controllers and kernel nftables bridge/inet, connection-tracking, and + NAT support. Managed networks require non-overlapping RFC1918 prefixes with usable hosts (`/30` or larger). + Enabled NAT requires all eligible IPv4 main-table default routes to resolve to one uplink; caller-selected uplinks + are removed. +- Removed flat policy commands, `allow_remote_exec`/legacy VM-wide trust state, the `remote_vm` protocol, + iptables/`nft` CLI backends and selectors, raw host-tool sudo grants, and public `run provision`/`run jailer`. + Recreate typed traffic/exec grants; previously unrestricted managed peer traffic now starts denied. +- Removed `volume create --format`, including `--format raw`, and QCOW2 data-volume create/resize. + Non-raw records may be inspected/removed but cannot be attached, resized, or relaunched. + QCOW2 image import/conversion and volume read-only/shareable/writeback options remain supported. +- Removed public `api_socket_path`, custom Firecracker/cloud-init/console filenames, and kernel API destination + fields `output_dir`, `name`, and `output_path`. Control uses the root broker; managed paths are receiver-owned. +- Changed JSON reads from bare objects/lists to envelope `data`. Streaming/interactive exec, SSH, logs, + completion, service run, init wizard, and console attachment reject JSON before effects; console state/kill support it. +- Rejected legacy remote-exec databases, unique guest-CID schemas, older root journals, and incompatible protected + layouts without adoption. Separate vsock UDS endpoints now permit independent CID namespaces. +- Rejected FUSE, network, overlay/stacked, and automount cache roots. Custom `MVM_CACHE_DIR` remains supported + on ext2/ext3/ext4, XFS, Btrfs, F2FS, bcachefs, tmpfs, or ZFS. +- Required fresh 1.17.0 snapshots and matching-pair restore qualification; earlier snapshot results do not qualify + this pair. Historically, Firecracker 1.16.2 snapshots are incompatible with 1.16.0/1.16.1 in both directions. + Older snapshot state is neither converted nor relabeled, and cross-version compatibility is not asserted. See + [upstream release notes](https://github.com/firecracker-microvm/firecracker/releases/tag/v1.16.2). + +### Fixed + +- Fixed administrator reset rejecting terminal networkless history or a committed empty firewall generation + from a previous boot. Reset still requires empty desired state and absent current managed namespaces/tables. +- Fixed VM launch failures when host connection retries recreate an unresolved ARP entry after cleanup of a reused + IP. Ordinary unresolved dynamic entries with no MAC are accepted; retained MACs, static/proxy entries and + netlink failures still block launch. Neighbor-cleanup errors now include their underlying cause. +- Fixed networkless VM launch rejecting the fixed local NoCloud datasource used by injected and ISO cloud-init. + Remote datasource URLs and IP boot configuration remain rejected for networkless VMs. +- VM creation now reports filesystem preparation separately from optional cloud-init configuration. + Timing logs distinguish provisioning-plan preparation from applying the filesystem changes. +- Fixed imported-kernel QA readiness checks to use the current `host info` JSON schema. + +- Host network discovery accepts unrelated TUN/TAP devices with an unset owner, including Tailscale interfaces, + without weakening the authenticated owner checks for managed VM TAPs. + +- Fixed local host-direct pytest timeouts to retain bounded output and shutdown diagnostics, attempt + identity-pinned private process-group shutdown, and stop later domains and tiers. The domain fails and + the qualification host is retained for reconciliation with release evidence incomplete; process-group + cleanup does not prove privileged root-helper or VMM quiescence. See + [system-test timeout guidance](docs/development/HOW_TO_RUN_SYSTEM_TESTS.md#45-limit-parallel-workers). +- Fixed the eight-case nested-isolation QA domain's aggregate deadline to allow its measured per-test + VM setup/teardown and nested lifecycle work, without extending individual commands or weakening assertions. +- Fixed structured kernel-import conflicts to retain their error details and entity identity in API responses. +- Fixed VM-to-image export to skip source-reported holes without expanding the entire logical disk into allocated + zero blocks, while preserving disk contents and length. +- Fixed `host status` to verify inaccessible managed sudoers through authenticated root observation instead of + treating permission denial as absence. Unconfirmed observation errors propagate; the interactive `mvm init` + wizard can request explicit administrator setup with an unknown initial policy state and requires successful + verification afterward. +- Fixed cache system-test fixtures to select unused physical cache identities while preserving retired root + authority. The Tier 1 `cache` runner uses an 11 GiB disk; other T1/T2 runner disks and memory are unchanged. +- Fixed OpenRC agent startup to run before networking, chronyd, and cloud-init, without extending exec deadlines + or changing authentication. See [ADR-0024](docs/adr/0024-early-openrc-agent-startup.md). +- Fixed resume to consult root-owned pending-export admission and authenticated Firecracker state before success; + unresolved export recovery cannot be bypassed by cached Running/Starting status. +- Fixed prior-boot VM/network cleanup and shared-state directory admission while retaining private permissions, + exact process absence, and root ownership. Interrupted creation without a database row can recover through + exact full-ID forced VM removal; original caller staging files are retained and reported. +- Fixed protected snapshot/volume/cache retirement to separate permanent authority from removable data branches, + including distinct-filesystem caches and retry after interrupted deletion. +- Fixed same-network traffic grants under `br_netfilter` by adding the exact inet companion allow; + default deny and the host's bridge-netfilter setting remain intact. +- Fixed stale guest-neighbor removal to invalidate only the admitted dynamic IPv4 neighbor on its exact bridge. + Fixed rapid same-name/subnet network recreation to use a fresh identity. +- Fixed Jailer startup credential/sealing races, actual leaf-cgroup verification/removal, disabled-log admission, + and slow disk preparation consuming the child's setup deadline. +- Fixed live attach/detach namespace handling and authenticated guest-device removal before backing access revocation. + Known-applied attachment failures retain the binding for retry rather than falsely making the volume available. +- Fixed cloud-init OFF independently of deblob/`--skip-deblob`, safe seed-ISO permissions, Ubuntu usrmerged + systemd admission, and NoCloud readiness-pipe deadlines. +- Fixed read-only LoopMount inspection access-time drift and fixed-path `blkid` fallback. + Partition extraction falls back to `parted` after failed/malformed `sfdisk` probes instead of assuming no table. +- Fixed Btrfs-to-ext4 image conversion on newer kernels by using `rescue=nologreplay` for the read-only snapshot mount. +- Fixed image SHA-512 sidecar verification, including Alpine bare-token checksums; malformed/unavailable configured + metadata refuses publication. Fixed automatic import format resolution and long image name/version selectors. +- Fixed image pull/import and `kernel pull --default` to return committed default state and propagate write errors. + Kernel builds honor explicit `MVM_TEMP_DIR`; official kernel configuration includes bridge connection tracking. +- Fixed GuestFS probe timeout units and removed guessed global orphan/process cleanup. + Cache previews no longer prune unrelated GuestFS state. +- Fixed moduleless built-in-KVM host initialization, first-read capacity reporting, and failed sudo setup being + marked complete. Administrator reset errors now report their stage and bounded cause. +- Fixed mixed VM creation failures to retain attempted IDs and primary/recovery errors in + `error.details.failed_vms`/`created_vm_ids` with `data: null`; atomic rollback does not promise removal. + Fixed per-selector volume outcomes, snapshot partial errors, cache error propagation, and exec-port validation. +- Fixed ambiguous image-prefix errors and human-readable byte sizes in inspection output. +- Fixed system-test runner base-image reuse to require one present exact type/version match and rebuild removal + to use that admitted image's full ID. This tooling change does not alter public image resolver semantics. +- Fixed system-test command timeouts to interrupt the owned CLI caller first, with bounded kill/reap fallback, + retained output and cleanup errors. A timeout remains a failure even if shutdown completes successfully. +- Fixed QA host-reboot phases to require an explicit private `--host-reboot-namespace` input for PRE, binding + the exact runner, non-root operator and selected directories. POST retains that binding; task and phase receipts + require fresh schema 2 without adopting older records or legacy cache state. The original outer controller + and attached SSH key remain unchanged. This flag belongs to QA/developer tooling; see the + [host-reboot guide](docs/development/HOW_TO_RUN_SYSTEM_TESTS.md#47-explicit-named-runner-host-reboot-journey). +- Aligned T1/T2 domain-runner VM creation deadlines to 1,800 seconds in system-test tooling. +- Fixed networkless VM cleanup when Firecracker exits after recovery admission: retained pidfd exit and + successful reap now retire the live namespace check. Failed reaping and ambiguous live identity still refuse. + +### Known limitations and qualification + +- Candidate-bound x86_64 QA passed all 528 registered nodes (526 ordinary plus actual reboot PRE/POST), and the + closed candidate565 evidence join exited zero with PASS. Original source bindings remain literal; this was + not one full latest-HEAD invocation or a generic source-carry waiver. See the + [coverage checkpoint and proof identifiers](tests/system/COVERAGE_MATRIX.md#v030-release-security-qualification-overlay). +- v0.3.0 remains unreleased. Remote CI passed for `4034eefc`; aarch64 runtime qualification and final release + approval remain open. See the [qualification checkpoint](tests/system/COVERAGE_MATRIX.md) for evidence and limits. +- Original exec/cache transients and the original saturation incident remain unexplained. Bounded pressure/guest + exhaustion passes neither attribute those failures nor guarantee survival under arbitrary host/ancestor/global OOM. +- The required matching Firecracker/Jailer 1.17.0 pair passed the selected hotplug/snapshot journeys. Attachment + readiness detects unusable devices; it cannot repair the BAR relocation failure observed with 1.16.2. +- Ambiguous ownership, interrupted effects and durability/cleanup errors remain explicit rather than authorizing + adoption or deletion. Snapshot projection recovery may require the emitted full ID; VM image import is a root-disk + copy, not an application-consistent RAM/data-volume snapshot. +- Release-derived device budgets, the 10,000,000-byte complete vmstate limit and trusted captured-RAM length remain. + No per-data-volume/aggregate quota or physical-space reservation is promised; operators must budget capture, + restore and recovery space. See [resource-limit decisions](docs/improvements/volume-snapshot-resource-limits.md). +- Registered-node completion does not establish every authority-abuse, replacement-race, policy fault, cancellation + or failure-recovery case. The coverage matrix retains those limits and all command-level Partial/Missing entries. ## [0.2.0] - 2026-07-10 @@ -317,6 +577,6 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - System tests run in nested VM with unprivileged user - Coverage matrix tracking every CLI subcommand and flag -[Unreleased]: https://github.com/AlanD20/mvmctl/compare/v0.2.0...HEAD +[0.3.0]: https://github.com/AlanD20/mvmctl/compare/v0.2.0...HEAD [0.2.0]: https://github.com/AlanD20/mvmctl/releases/tag/v0.2.0 [0.1.0]: https://github.com/AlanD20/mvmctl/releases/tag/v0.1.0 diff --git a/CONTEXT.md b/CONTEXT.md index 42c47714..3549cff0 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -25,8 +25,9 @@ MicroVM Manager -- a speed-first CLI for managing Firecracker microVMs. Provides - [Layer compliance enforcement](#layer-compliance-enforcement) - [Public API boundary](#public-api-boundary) - [CLI is the canonical interface](#cli-is-the-canonical-interface) + - [VM-to-VM execution transition](#vm-to-vm-execution-transition) - [Provisioner Backend (LoopMount vs GuestFS -- mutual exclusion)](#provisioner-backend-loopmount-vs-guestfs--mutual-exclusion) - - [Firewall Backend (nftables vs iptables -- mutual exclusion)](#firewall-backend-nftables-vs-iptables--mutual-exclusion) + - [Firewall backend transition](#firewall-backend-transition) - [Build output (REQUIRED)](#build-output-required) - [Asset mirror environment variable (OPTIONAL)](#asset-mirror-environment-variable-optional) - [Relationships](#relationships) @@ -48,11 +49,20 @@ MicroVM Manager -- a speed-first CLI for managing Firecracker microVMs. Provides ### Domain -A domain is a self-contained business capability with its own logic, data model, and test suite. Each domain lives in a directory under `internal/core/` named after the capability — for example, `internal/core/vm/` for VM lifecycle, `internal/core/network/` for networking, and `internal/core/image/` for image management. The project currently has sixteen domains covering everything from SSH keys to snapshots. +A domain is a self-contained business capability with its own logic, data model, and test suite. Each active domain is an +immediate directory under `internal/core/`. For example, `internal/core/vm/` owns VM lifecycle, +`internal/core/network/` owns networking, and `internal/core/image/` owns image management. Inspect `internal/core/` +instead of copying a domain count into documentation. -Domains are strictly isolated from each other. A domain in `internal/core/vm/` can never import from `internal/core/network/` or any other domain package. The Go compiler enforces this isolation through circular import detection: if a domain tried to import another domain, the compiler would produce an import cycle error. This means each domain can be tested, modified, and replaced independently without affecting the rest of the system. +Domains are strictly isolated from each other. A domain in `internal/core/vm/` cannot import +`internal/core/network/` or another sibling domain. The Go compiler rejects import cycles, but it does not reject every +one-way sibling import. The deterministic `internal/architecture` CI gate scans every production Go file beneath each +immediate core domain and rejects imports of a different sibling domain while allowing same-domain subpackages. -What unifies the domains is the shared model layer at `internal/lib/model/`. Every domain imports its types — concrete structs with `db:"column"` and `json:"field"` tags for SQL and JSON serialization — from this single package. No domain defines its own model types. The model package contains 21 files covering VM instances, networks, images, kernels, binaries, volumes, SSH keys, leases, firewall rules, console info, Firecracker config, cloud-init modes, provisioner types, relation specs, VM status, operation status, and workflow state. +The shared model layer at `internal/lib/model/` unifies the domains. It contains concrete structs with `db:"column"` +and `json:"field"` tags for SQL and JSON serialization. Core domains use these shared types instead of defining +cross-domain model packages. Inspect the package for the current model inventory instead of copying a file count into +documentation. Not every domain follows the same internal structure. The pattern varies by complexity: - **Controller/Service/Repository/Resolver**: vm, network, kernel, key, volume @@ -60,7 +70,7 @@ Not every domain follows the same internal structure. The pattern varies by comp - **Controller/Service/Repository**: host - **Repository + Resolver**: snapshot - **Controller only**: console -- **Service only**: cache, ssh +- **Service only**: cache, ssh, update - **Service + Repository**: config (includes constraints registry) - **Controller + Service**: logs - **Manager + Provisioner**: cloudinit @@ -80,13 +90,13 @@ A struct bound to a single entity instance. Manages lifecycle state transitions Constructed with the entity + its Repository: `vm.NewController(vm *model.VMItem, repo Repository)`. NOT wired at startup -- created per-operation in the Service layer. -*Example: `vm.NewController(vmItem, repo).Snapshot(ctx)` -- snapshots this specific VM. `network.NewService(repo, tracker).RemoveBridge(ctx, bridge)` -- removes a bridge, no single network entity needed.* +*Example: a VM Controller snapshots its bound VM. `network.NewService(repo, trafficRepo).EnsureManagedNetwork(ctx, networkItem)` coordinates one managed network, with no Controller bound to an individual network.* ### Service (stateless, intra-domain) A struct for stateless intra-domain operations. Handles infrastructure operations (bridges, TAPs, NAT, subprocesses, file/disk operations). Performs state detection (checking current system state as part of an operation -- "does this bridge exist?" to branch execution). Guards invariants that protect against system damage. Does NOT validate caller input. Does NOT manage state for a single entity -- Service operates on infrastructure, not on a bound instance. -Constructed with repos/options only: `network.NewService(repo Repository, tracker *firewall.FirewallTracker)`. Wired once at startup in `app.Initialize()`. +Constructed with repos/options only: `network.NewService(repo Repository, trafficRepo TrafficPolicyRepository)`. Wired once at startup in `app.Initialize()`. *Litmus test: if the operation would work the same way without a specific entity instance, it's Service. If it needs to communicate with a running entity's Firecracker API socket, it belongs in Controller. If it sequences multiple infrastructure steps (teardown NAT -> remove bridge -> delete DB record), it's intra-domain orchestration in Service.* @@ -98,14 +108,14 @@ Constructor pattern: `NewRepository(db *sqlx.DB) Repository`. ### Service subprocess pattern (internal/service/) -Long-running subprocess services (console relay, nocloud-net server, loopmount provisioner) live in `internal/service/{name}/`. These are compiled into the same `mvm` binary — no separate binaries. The CLI layer has an `mvm run ` subcommand that serves as the entry point for each service. An additional embedded service (`agent/`) provides a cross-compiled guest agent binary that is compressed and embedded into the `mvm` binary at build time, then injected into the VM at runtime. +Long-running ordinary-user subprocess services (console relay and NoCloud HTTP server) live in `internal/service/{name}/` and are launched through the corresponding `mvm run ` subcommand in the same binary. LoopMount provisioning and Jailer operations instead use closed authenticated root-dispatch actions, not public root service subcommands. The embedded `agent/` service provides a cross-compiled guest binary that is compressed into `mvm` and injected into the VM at runtime. -Each service follows a consistent three-function pattern: +The ordinary-user subprocess services follow a three-function pattern: - **`Config`** struct — holds all configuration for the service. - **`Run(ctx, cfg)`** — runs the service in the foreground (blocking). -- **`Spawn(ctx, cfg, extraParams...)`** — launches the service as a background subprocess via `system.SpawnService()`. The context parameter is typically `nil` (background/nil) for daemon services (console relay, nocloud-net server) and a real context for synchronous services (loopmount provisioning). Extra parameters carry service-specific data: `console.Spawn()` passes a PTY file descriptor, `nocloudnet.Spawn()` passes the config only, `loopmount.Spawn()` passes a wire protocol input struct. +- **`Spawn(ctx, cfg, extraParams...)`** — launches the service as an ordinary-user background subprocess via `system.SpawnService()`. Console passes a PTY file descriptor; NoCloud passes its fixed service arguments. -Services in `internal/service/`: `console/` (console relay PTY proxy), `nocloudnet/` (NoCloud HTTP metadata server), `loopmount/` (loop-mount provisioner wire protocol), `agent/` (embedded guest agent binary — cross-compiled, compressed, and injected into the VM at runtime via vsock). +Services in `internal/service/`: `console/` (console relay PTY proxy), `nocloudnet/` (NoCloud HTTP metadata server), `loopmount/` (typed privileged rootfs provision/inspect/optimize receiver), `jailer/` (trusted release installation, per-VM jail setup, and privileged Jailer exec), `agent/` (embedded guest agent binary — cross-compiled, compressed, and injected into the VM at runtime via vsock). Dependency direction: `cli/` -> `services/`. Services never import `cli/` or `pkg/api/`. @@ -202,12 +212,53 @@ net, _ := input.ResolveNetwork(ctx, op.Repos.Network) Defined in `internal/lib/db/migrations/*.sql`. Accessed via `github.com/jmoiron/sqlx` with `modernc.org/sqlite` driver. PRAGMAs (foreign_keys=ON, journal_mode=WAL, synchronous=NORMAL, busy_timeout=5000, wal_autocheckpoint=1000, cache_size=-64000) set via DSN parameters in `db.Handle.openLazy()`. Connection pool has `SetMaxOpenConns(1)` and `SetMaxIdleConns(1)` for SQLite's single-writer semantics. -Tables include: `images`, `kernels`, `binaries`, `volumes`, `networks`, `network_leases`, `vm_instances`, `host_state`, `host_state_changes`, `iptables_rules`, `nftables_rules`, `ssh_keys`, `user_settings`, `vm_vsock_config`, `snapshots`, `db_migrations` (16 tables). +The final schema includes `images`, `kernels`, `binaries`, `volumes`, `networks`, `network_leases`, `vm_instances`, +`host_state`, `host_state_changes`, `network_traffic_policies`, `vm_traffic_policies`, `network_host_traffic_policies`, +`vm_host_traffic_policies`, `exec_policies`, `ssh_keys`, +`user_settings`, `vm_vsock_config`, `snapshots`, and `db_migrations`. The published fresh-install migration, +`internal/lib/db/migrations/001_baseline_0_3.sql`, establishes the baseline at SQLite `user_version=1`. +The single baseline includes both host-grant projection tables; no second migration is shipped for them. +The `vm_instances` row records `managed` or `networkless` placement +explicitly. A managed row retains a real network foreign key, IPv4, and MAC; a networkless row stores a SQL `NULL` +network ID and empty endpoint fields. + +`host_state` is a caller-owned capacity cache, not administrator setup authority. The first `mvm host info` read +creates its row if needed; `--refresh` re-detects and updates it. Root-owned host admission and its journal remain +under `/var/lib/mvmctl/host`. Accordingly, `host info --json` does not report a setup verdict, and the `state` field +of `host status --json` is null rather than projecting the cache's legacy `initialized` bit as root truth. + +`HostStatusCheck(ctx)` returns `(*results.HostStatusCheck, error)`. Its fixed managed-sudoers observation uses +authenticated root authority when ordinary inspection cannot confirm absence. Permission denial is not absence; +unsafe policy state or an unavailable, missing or invalid observation returns an error. The interactive `mvm init` +wizard may request explicit administrator bootstrap with an unknown initial policy state, but must verify policy and group +state afterward before reporting setup success. + +The baseline replaces the nine development migrations; it contains no VM-wide `remote_exec` column, obsolete firewall +tracking or routed service-access policy tables, or retired selector settings. Version numbers alone do not identify +compatible databases. Both pending-migration inspection and migration execution accept only an empty version-0 +database or a database whose complete migration history matches the embedded numbered files by version, filename, +and SHA-256 checksum. The baseline receipt must name `001_baseline_0_3.sql`; older or unknown databases are refused +with a validation-class `database.migration.failed` clean-install diagnostic, leaving their schema, version, and +contents unchanged. No legacy trust bit or policy is converted. Future changes continue with sequential numbered +migrations after the baseline. Pending SQL, migration receipts, and `user_version` commit in one transaction and roll +back together on failure; migrations do not create automatic pre-migration snapshots. After stopping the old VMs and +preserving desired artifacts, remove the old mvm state directory selected by `MVM_CACHE_DIR` (normally +`~/.cache/mvmctl`) through the documented clean-install procedure and initialize fresh v0.3.0 state; never delete live +state as an incidental migration step. Receipts identify compatibility in caller-owned SQLite, not privileged authority. + +The same fresh schema deliberately does not make `vm_vsock_config.guest_cid` globally unique. Each Firecracker process +serves its vsock device through an independent UDS endpoint and therefore has its own CID namespace, while a snapshot +clone must preserve the CID embedded in its vmstate. The non-unique `idx_vsock_config_cid` index remains. Removing the +legacy unique constraint is part of the v0.3.0 clean-install schema break; there is no migration or adoption path for an +older database constraint. ### Layer compliance enforcement -Architecture rules are enforced by the Go compiler (circular import errors prevent cross-domain imports in core) and code review. Key rules: -- Core domains NEVER import other core/* packages -- enforced by Go compiler. +The Go compiler enforces type safety and rejects import cycles. The `internal/architecture` test adds machine +enforcement for production imports between sibling core domains and runs as its own CI step before formatting, vet, +and the full test suite. Code review continues to enforce the remaining project-specific dependency rules that are +stricter than Go's import rules. Key rules: +- Core domains never import other `internal/core/*` packages. - CLI imports from `pkg/api/`, `pkg/api/inputs`, `pkg/api/results`, `pkg/errs`, `internal/cli/common/`, `internal/infra/`, `internal/lib/`, `internal/service/` -- enforced by code review. - API imports `internal/core/*` + `internal/enricher/` + `internal/infra/` + `internal/infra/event` + `internal/lib/*` + `internal/assets` + `internal/service/*` + `pkg/errs` + `pkg/api/inputs` + `pkg/api/results`. - `internal/infra/` and `internal/lib/` are LEAVES -- import NOTHING from core, api, or cli. Exception: `internal/lib/provisioner/loopmount/backend.go` imports `internal/service/loopmount` for wire-protocol types used to spawn the loopmount provisioner subprocess. @@ -217,19 +268,500 @@ Architecture rules are enforced by the Go compiler (circular import errors preve The `pkg/api` package IS the stable, curated public interface for all consumers -- CLI, future TUI/GUI, and external scripts. The `api.Operation` struct exposes all domain operations as methods. The `pkg/api/inputs` package exposes all Input types. The `internal/core` package is an implementation detail. -### Build output (REQUIRED) +### Build and system-install contract (REQUIRED) -**The `mvm` binary MUST be built to `~/.local/bin/mvm`.** This path has passwordless sudo privileges via the mvmctl sudoers rules, so subcommands requiring privilege escalation run without password prompts. +The release script produces the candidate artifact at `dist/mvm`. A development artifact may live in the worktree or +under `~/.local/bin`, but a user-owned path is **never** a sudo target. + +The canonical host installation is `/usr/local/bin/mvm`, owned by `root:root` with mode `0755` under a root-owned, +non-writable directory chain. Install or replace it through the candidate artifact's exact early bootstrap route: -For **release testing / RC QA / system tests**, always use the release build script: ```bash -./scripts/build.sh release # produces dist/mvm -cp dist/mvm ~/.local/bin/mvm # copy for sudo operations +./scripts/build.sh release --version 0.3.0-rc.1 --output dist/mvm +sudo ./dist/mvm host install-system +sudo /usr/local/bin/mvm host init ``` -A bare `go build -o ~/.local/bin/mvm ./cmd/mvm` works for dev but produces a -binary without version info, symbol stripping, or PIE. **Never use it for -release qualification.** +`host install-system` executes before ordinary application initialization. It does not read `MVM_CACHE_DIR`, user +configuration, the user database, Cobra, or plugins, and it replaces its environment with fixed root-safe values before +installation. v0.3.0 is a clean installation: it does not convert, adopt, or preserve old user/runtime state. Remove an +old managed sudoers file and clean the old installation from an authenticated administrator session before running the +bootstrap command; the installer refuses unrecognized active syntax rather than guessing how sudo authenticated the +process. Typed cleanup and guarded administrator reset are implemented as described below, but do not migrate or +adopt a pre-v0.3 installation. The [coverage checkpoint](tests/system/COVERAGE_MATRIX.md) records the accepted +candidate-bound installed-system evidence and its limits; it does not qualify legacy migration or authorize raw host +cleanup. Final release approval remains separate. + +Normal users run `/usr/local/bin/mvm` without sudo. The `mvm` group authorizes the constrained passwordless privileged +protocol; it never makes a development artifact trusted. Generated sudoers now grants only the fixed root-owned +`/usr/local/bin/mvm __mvm_privileged_v1 *` route. Installation, publication, and init reject transitional wildcard or +raw-tool policies. This implementation pass has not activated the policy on the host. A system-installed executable cannot replace itself through +`mvm self-update`; an administrator installs the newly downloaded trusted artifact with `host install-system`. + +The current sudoers publisher verifies the canonical executable, pins the fixed root-owned directory chain and any +existing one-link `0440` policy, and validates generated bytes through bounded `/usr/sbin/visudo -c -f -` with a fixed +environment. It publishes by file fsync, atomic rename, and directory fsync, preserving replacement and uncertain +durability evidence on failure. The generated policy is marker-only. The early `host init` route now +selects the authenticated sudo invoker or requires `--operator ` in a direct-root session, sanitizes the +environment, and runs before user configuration, cache, SQLite, or ordinary services. A root-owned journal beneath +`/var/lib/mvmctl/host` preserves original global host state and mutation progress. Fixed root-owned +`operator-.json` records preserve prior membership for each enrolled operator; multiple operators are supported. +An interrupted initial global transaction requires its original operator to recover before another operator enrolls. +The administrator journal is now schema 2 and records the exact initialized group identity. Schema-1 journals are +incompatible clean-install state rather than migration input. The root-owned host admission gate has shared operation +leases and an exclusive administrator lease; only the administrator route may create its fixed state. Initialization +enables admission only after host steps and operator enrollment complete. Missing, disabled, or unsafe admission state +fails closed. The gate must be retained across each privileged request and cleanup, with authorization checked again +after acquisition; reset requires exclusive admission rather than a momentary idle observation. +Enrollment, typed partial results, JSON boolean forms, help, and argument boundaries are implemented. Sudo entry refuses a managed policy +containing transitional wildcard grants because it cannot prove administrator authentication. `HostClean` now obtains +the caller's authoritative root VM inventory, including running records absent from SQLite, before removing locally +registered caller networks through typed API operations. It refuses incomplete observation or running VMs and is not +global quiescence authority. The dispatcher now retains shared admission through requests +and cleanup and rechecks authorization after acquiring it. Administrator reset composes exact Jailer, network, and +firewall guards under exclusive admission, requires marker-only sudoers, then disables admission and restores the +recorded host settings through durable rollback receipts. Active, ambiguous, or administrator-modified state refuses +reset. Reset does not kill VMs, delete authority history, or unload kernel modules. Admission stays disabled after +reset or a restoration error. Host activation and qualification remain separate work. + +[ADR-0026](docs/adr/0026-administrator-full-uninstall.md) adds the explicit complete administrator uninstall +exception: `scripts/uninstall.py` performs typed cleanup for all enrolled operators, then native reset, before +removing permanent root history under disabled admission. It is an optional root-run maintenance utility, not an +operator RPC or additional sudoers grant. Terminal networkless journals and exactly bound committed empty +prior-boot firewall history may pass reset only with the existing current namespace/table absence checks. +See [uninstall](docs/UNINSTALL.md) for removal scope and preserved external files. + +The v0.3 privileged target splits state by authority and lifetime. Durable ownership and release records live beneath +`/var/lib/mvmctl`; root-only locks and launch handshakes use `/run/mvmctl/authority/`. Caller-facing console +outputs and display-only PID mirrors use the pinned caller-owned directory beneath +`/run/mvmctl/runtime//`. Firecracker API and vsock sockets instead live in the isolated execution +identity's private `/run/mvmctl/vmm-runtime//` directory, mounted at `/vmm-runtime` inside the jail; +the operator reaches Firecracker control only through the authenticated root broker. Persistent configuration, disks, +logs, console logs, and optional metrics remain in the caller's managed cache. Final privileged code must never mount +the whole cache VM directory, trust a runtime leaf as authority, or signal a PID read from a PID mirror. ADR-0016 +defines the exact ownership, modes, launch ordering, and +cleanup contract; `tasks/todo.md` records implementation status. + +`/var/lib/mvmctl` is the shared persistent state base. Existing admission requires a directory owned by UID 0 without +group or world write access; a safe mode such as `0755` is accepted without changing its metadata. When absent, the base +is created as `root:root 0700`. Private network, firewall, trusted-release, and exec-allocation directories beneath it +retain exact `root:root 0700` admission, and their records retain exact `root:root 0600` admission. Traversal and +publication retain their descriptor pinning, no-follow checks, and durability requirements. + +Prior-boot instance recovery also accepts a safe traversable shared state base. Its exact private-mode requirement +begins at `instances` and the owner directory; this does not move the no-cross-mount boundary below `mvmctl`. +Unsafe metadata still refuses recovery without repairing directory permissions or discarding retained authority. + +The private Jailer runtime-directory foundation now creates the fixed root-owned mode-`0711` +`/run/mvmctl/runtime/` hierarchy and exclusively creates one caller-owned mode-`0700` `` directory beneath +it. It pins the VM directory's device/inode identity, rejects existing VM directories and unsafe parents without +repair, returns cleanup authority after partial VM-directory preparation, and removes only the exact empty directory +with retryable unlink/parent-sync reconciliation. The lease remains private and exposes no path or descriptor. Only a +durable `launching` lease can derive the recorded owner and VM identity to create it; partial preparation stays retained +for abort, and activation or abort followed by active cleanup transfers the same lease without reconstructing a path. +Cleanup removes and releases the exact empty directory before committing `cleaned`; a non-empty directory keeps the +record in retryable `cleaning`. Before removal it unlinks only the five compiled socket/PID basenames through the +retained directory descriptor. Missing leaves are already clean, symlinks are unlinked rather than followed, and a +directory, mount, or unexpected name remains untouched and blocks `cleaned`. Fresh cleanup after a controller restart +reopens only the exact existing root-controlled hierarchy and caller-owned VM directory without creating or repairing +any component. A missing hierarchy is already clean; unsafe state keeps the durable record and VM lock in retryable +`cleaning`. Typed launch now consumes this directory in the child-created jail mount tree. Console connections use +the caller runtime; the API broker pins the separate private runtime through the verified process root. The whole +caller-owned VM cache directory is not mounted. + +The current Jailer instance record is strict schema 6 with +`absent|cleaned -> launching -> active`, `launching|active -> cleaning -> cleaned`, and idempotent `cleaning`. +`launching` is durable pre-effect intent: owner UID/GID, VM ID, managed-cache identity, exact Firecracker/Jailer release +identity, explicit managed or networkless placement, isolated execution UID/GID, host boot ID, launch/cleanup generations, +the handshake-blocked child's PID/start locator, and an explicit absent relay or handshake-blocked relay PID/start +locator. It does not claim observed final credentials, cgroup membership, final mount namespace, jail root, or +executable identity. `active` records +the receiver-observed credentials, final mount namespace, jail root, and executable, requires empty supplementary +groups and private authority verification of exact derived cgroup membership, and binds the final copied Firecracker +to the release. The schema stores no cgroup path or caller-shaped cgroup value. A present active +relay records its caller credentials, empty groups, host mount namespace/root, and exact root-owned +`/usr/local/bin/mvm` executable metadata and digest; absence is explicit, and `console.pid` remains display-only. +A present relay additionally carries `running|stopping|stopped`. Typed root console observation opens only +existing authority and verifies the recorded task and fixed runtime socket; SQLite and cache PID files do not +authorize discovery or signaling. Relay-only stop retains a verified pidfd, durably commits `stopping` before +signaling and commits `stopped` only after exit. Firecracker identity and VM liveness remain independent; +whole-VM cleanup skips a durably stopped relay. An uncertain reply preserves whether the privileged child +started and effects may have occurred. Schema-5 records are rejected, including cleaned tombstones; there is +no adoption or migration. Installed-candidate endpoint and Firecracker/vsock-survival qualification remains open. +Network namespace identity remains in the separate closed topology capability and is compared during activation rather +than stored as a raw path or descriptor. +`cleaned` is the same-UID tombstone and carries no launch-specific GID, cache, release, or process identity; mount +identity is valid for one mount lifetime and must not prevent a legitimate relaunch after reboot. + +This split is required by Firecracker Jailer 1.16.1: Jailer copies Firecracker, joins netns/cgroup, then unshares and +pivots into a new mount namespace before dropping credentials and execing the copy. The trusted-store inode therefore +cannot equal `/proc//exe`, and the final namespace/root cannot be known before effects. The receiver persists +`launching` before effects, invokes Jailer without `--daemonize` or `--new-pid-ns`, retains the original pidfd through +exec, verifies the final jail root and copy, seals both to `root:root 0755` beneath `root:root 0700` ancestors, hashes the +copy against the exact manifest, rechecks full identity, and only then commits `active`. All composite instance leases +must use pointer-shared mutex-protected close-once state and cache the first cleanup result. + +The prepared release's consuming registration method derives release identity internally and transfers its pinned +admission, store, slot lock, and executable resources into the launch lease. That lease retains the VM lock plus opaque +topology and endpoint authority. It holds the later network then firewall mutation locks while it creates links down, +applies and freshly verifies the complete atomic nftables generation, and raises/reverifies links. A failure before +successful activation-lock release rolls back with those same locks and leaves the child blocked. Success releases the +firewall then network mutation lock before the blocked child reaches Jailer; active authority does not serialize other +VMs. A fixed jail destination must be proved absent/fresh immediately before release; only an exactly admitted, +non-mounted stale fixed tree may be removed after process absence. `launchLease.Abort` and +`activeLease.BeginCleanup` transfer their resources directly into cleanup without releasing and reacquiring the VM +lock. Later cleanup reacquires network then firewall under that VM lock. Authority-level cleanup admission is +recovery-only. + +The private instance-authority graph also carries a fixed-name cgroup-v2 authority for +`/sys/fs/cgroup/mvmctl/`. It descriptor-pins the cgroup root, compiled parent, and VM leaf and verifies exact +metadata, mount identity, and replacement stability. Preparation accepts only a freshly absent VM leaf, never adopts a +legacy leaf, and repeats the absence proof at the last handoff boundary. A successful blocked-child release records +Jailer-handoff provenance before the leaf can be bound. Binding and activation prove that the pidfd/start-tick-bound +Firecracker task is the leaf's sole member; a failed proof retains the same lease for retry. Cgroup authority transfers +from launch to active to cleanup without path reconstruction. Abort first terminates and exactly reaps the retained +task, then causally binds the leaf for cleanup. Completion requires the exact unreplaced leaf to be empty before +retryable descriptor-relative removal and commits durable `cleaned` only afterward. A terminal descriptor-close error +after proven removal converges rather than retrying a reused descriptor. This authority adds no cgroup path to the +schema and is consumed by typed root launch/cleanup. Focused live enforcement and bounded sustained-pressure +checks passed in the named runner; wider failure coverage remains limited as recorded in the qualification matrix. + +Private launch cgroup control now accepts only a typed policy-v2 intent: CPU quota, period, and weight; memory high, +maximum, and swap maximum; and an exact PID maximum. Preparation checks fixed `cpu`, `memory`, and `pids` controller +readiness at the cgroup root and then the compiled parent, writes only missing enablement tokens additively, and reads +back each result through retained descriptors. It repeats the complete readiness proof immediately before Jailer +handoff. After handoff, exact descriptor-relative reads of `cpu.max`, `cpu.weight`, `memory.high`, `memory.max`, +`memory.swap.max`, and `pids.max` must match the retained policy before the pidfd/start-tick-bound sole-membership +proof. Every retry repeats limit readback and membership verification, and every error returns the same retained +cgroup authority. Recovery and launch without a cgroup policy never mutate controller state. Process handoff tracks +relay and Firecracker release independently, so a retry after successful relay handoff but failed Firecracker handoff +resumes only the remaining handoff on the same launch authority. + +[ADR-0019](docs/adr/0019-pressure-only-vm-memory-control.md) supersedes finite per-VM memory maxima. Policy v2 keeps +guest RAM plus configured headroom as `memory.high`, carries explicit unlimited-max intent with a canonical zero byte +field, and requires literal kernel `memory.max=max`. Swap, CPU, PID, and ancestor limits are unchanged. Configuration +errors propagate; missing/v1 policy is not silently backfilled. Pressure may exceed the threshold, and host or ancestor +OOM remains possible. Focused installed-candidate runner tests have observed the pressure-only controls, actual +`memory.high` events under guest allocation, unchanged leaf `oom_kill`, and responsive vsock. Bounded sustained +saturation with and without swap and identified guest-global OOM recovery also passed; the original incident and +arbitrary ancestor/host OOM behavior remain unqualified. Preserve the candidate bindings in ADR-0019 and the +coverage matrix rather than treating these observations as one latest-source full-suite run. + +Fresh private cleanup admission now reopens the record's exact pidfd/start-tick-bound processes and observes the +Firecracker as either the committed active identity or the exact launching task before durably transferring authority +to `cleaning`. It then re-admits the fixed cgroup leaf only when that task is its sole member. A re-admission or final +membership error keeps the same cleanup lease authoritative with cgroup cleanup pending for retry. Across a host-boot +mismatch it opens and signals no process, requires the fixed leaf to be stably absent, and rejects a present or unstable +leaf. Recovery from `launching` retains its provisional leaf-absence proof through exact process termination and reap +before binding for cleanup. Only fresh-recovery cgroup admission callbacks run outside the `cleanupLease` mutex; +ordinary `Complete` still serializes cleanup effects under it. None of this adds a cgroup path to schema 6. + +The original pidfd provides stable task identity and exit detection but does not notify `exec`. Activation uses a +bounded context-aware `/proc//exe` observation loop, stopping immediately on pidfd-observed exit, before final +identity admission. After any active-record rename, even a later fsync/close error returns a non-nil active lease with +`record_replaced` and any `durability_uncertain` evidence so cleanup never uses stale launching authority. Recovery +re-admits a present console relay by its exact locator and final credential/namespace/root/executable identity; it never +signals an ambiguous or reused PID. + +Root status observation has four closed outcomes: `running`, `stopped`, `crashed`, and `unavailable`. An absent/cleaned +record or boot mismatch is stopped; a same-boot missing/dead/mismatched active process is crashed; incomplete authority +is unavailable and fails the complete read. Interrupted `launching` records are reconciled by boot/PID/start/final-copy +identity: only the exact same task may be promoted or pidfd-terminated, and ambiguous state is neither killed nor +deleted. Reconciliation and public projection remain target behavior. The private batch observer now accepts up to +1,000 canonical unique VM IDs, scans ownership once under the index lock, and retains at most one VM lock and process +set at a time. It compares the recorded active credentials, mount namespace, root, executable metadata, and derived +cgroup path through pidfd-bound observations without rehashing executables. Network namespace identity must remain +stable between its two observations, but is not checked against persisted topology authority: schema 6 stores no +network namespace identity. Same-boot `launching` or `cleaning` records are unavailable; the observer never signals, +repairs records, or returns a partial map. Public list/get now use root batch observation and return errors when it +fails. List observes the complete selection before status filtering; paused display state is preserved only for an +observed live process. Legacy launches have no schema-6 authority and are not adopted from SQLite or PID mirrors. + +Main also implements the private strict schema-6 +record/codec and the process-authority side of `launching -> active -> cleaning -> cleaned`. An already-created +Firecracker child and optional relay can be admitted while handshake-blocked through challenged EOF-fails-closed gates, +retained close-on-exec pidfds, exact PID/start locators, and a pinned receiver executable. Consuming release +registration transfers those resources with the pinned store, slot lock, manifest identity, executable authority, and +VM lock into durable `launching` authority. + +The private activation path uses strict bounded `/proc` parsing and descriptor-pinned observations, a 10-second +executable-transition budget with a 25-millisecond minimum interval, and repeated boot, PID/start, pidfd-liveness, +credential, empty-group, cgroup, network-namespace, root, copied-executable, release-digest, and optional-relay checks. +Its cgroup check consumes only the retained fixed-name authority and exact sole-member proof. It binds and seals the +Jailer-created root and Firecracker copy before replacing `launching` with `active`. Private +`Abort`, `BeginCleanup`, and `Complete` transitions retain cleanup ownership, signal only through pidfds, wait and reap +before `cleaned`, accept both exit readability and the post-reap pidfd hangup as dead, and keep retryable failures in +`cleaning`. Fresh cleanup recovery is bounded to the recorded Firecracker and optional relay and signals nothing across +a boot change; on the same boot, active cleanup requires their complete recorded identities and executable digests +before returning cleanup authority. These process, destination, active, and cleanup leases use pointer-held close-once +state and cache their first cleanup result. + +Fresh cleanup now reopens the exact existing per-VM runtime directory through the root-controlled hierarchy before +runtime cleanup. It creates or repairs nothing, treats an absent hierarchy as already clean, and preserves retryable +`cleaning` on unsafe state. `VMLifecycleClient` and the authenticated receiver now expose named launch, abort, cleanup, +and remove transactions. Core VM spawning and process control use these requests; returned PID/start values are display +projections only. Launch registers the handshake-blocked child before mutable effects, prepares cgroup/runtime and +persistent-output authority, creates topology down, applies/verifies firewall state, and transfers pinned resources +into a fresh child-created jail tree. It commits the observed schema-6 active identity before close-only detach; a +detach failure does not destructively roll back an already committed active instance. + +Cleanup re-admits exact root records and pidfd-bound processes, including same-boot dead-process recovery, then removes +the exact stopped jail tree, topology, cgroup, and runtime resources before reporting cleaned. Same-boot topology +recovery uses durable namespace/link evidence. Stop preserves endpoint reservations and traffic grants for restart; +remove separately invalidates them and completes root removal before local deletion. Snapshot restore, raw volumes, +and PCI have typed production paths; release qualification remains pending. Graceful stop preserves explicit +`force=false`, admits the exact API peer, requests guest shutdown, waits on the retained pidfd, then completes cleanup. +Console launch now includes its fixed root-spawned relay, caller UID/GID with empty supplementary groups, authenticated +gate, root-created PTY, and pinned outputs. Prior-boot cleanup consumes a concrete retained VM-lock capability that +proves the fixed cgroup absent and binds the root cleaning record across boot identities. Network retirement repeats +namespace/link absence checks, preserves logical firewall grants, and never treats old kernel identities as current +deletion authority. Managed-network rebootstrap retains exact logical intent while creating fresh bridge/uplink +identity; a competing existing bridge is not adopted. Final VM removal invalidates exact retired reservations. +Surviving placeholders or ambiguous evidence remain fail-closed. Root removal now supports old-boot removal receipts +and pending desired networks with no root record only after repeated whole-state absence checks: no managed bridge, +live topology claim, or firewall reference. Missing ownership never authorizes kernel deletion. Cancellation records +are permanent identity tombstones; retry may acknowledge absence but cannot recreate the same network identity. +An exact prior-boot live network record can also be removed without bridge recreation: the root service uses current +boot and host-namespace authority, repeatedly proves the derived bridge and live topology absent, retires only +matching firewall intent, and moves the unchanged record to a removal receipt. It never uses old ifindices or +namespace inodes to delete current kernel objects. One named-runner reboot cleanup of two test networks and focused +cross-network policy QA passed; full reboot, failure-path, Tier 3, and system qualification remain open. + +The public relaunch path has a separate, private normal-process `vmRelaunchState`. Start, +reboot, and snapshot restore load the per-VM vsock row explicitly: no row is a valid absence, while lookup failure, +mismatched VM identity, or an out-of-range CID aborts before respawn. A present row is cloned, its host UDS metadata is +normalized to the fixed managed `vsock.sock` leaf, and its CID, agent port, and token remain available to the normal +process. Only the validated CID enters `model.FirecrackerConfig`; the root Jailer path derives the private host runtime +and fixed `/vmm-runtime/vsock.sock` jail leaf. Snapshot restore preserves the CID stored in vmstate and sends the fixed +jail-visible vsock UDS override. VM create still uses its existing builder state rather than this common relaunch state. +The private host `/run/mvmctl/runtime` directory lease transfers through the launch/active/cleanup graph and typed +public launch. +Host-reboot reconciliation and L2 `mvm exec` relaunch qualification remain separate from this implementation; the +reported restart failure is not declared resolved without that evidence. Snapshot restore now consumes a prepared +root-owned capture, with independent protected disks and immutable memory backing for full restores. + +Snapshot restore must also obtain the captured network ID, guest IPv4, and MAC from trusted root snapshot metadata and +reserve those exact values for the restored endpoint. It fails if the source still owns them, another endpoint +conflicts, or a different network is requested. v0.3.0 performs no fresh IP/MAC allocation, cross-network restore, or +simultaneous endpoint clone. A new host TAP name may be derived, but the anti-spoof endpoint must match the embedded +guest address and MAC. The public restore path now reserves the exact captured IP in the local lease repository before +calling the root restore. It releases that lease only before a root attempt; an uncertain root outcome retains it for +recovery. Root metadata and the root endpoint guard remain authoritative for the captured topology. Live qualification +and conflict coverage remain pending; user SQLite state is not restore authority. + +SQLite binary rows are an untrusted display projection of the root release store. The implemented projection schema +now includes architecture and scopes list, prefix, type/version, latest, and default resolution to the runtime +architecture. VM migration pairs Firecracker only with a Jailer row of the same architecture. Private repository +methods atomically replace one complete `(architecture, version)` pair or remove that exact slot while leaving the same +version for another architecture unchanged. Public pull/remove now use the typed trusted-release client. Pull invokes +root before reading SQLite and projects only an authoritative installed/replaced/unchanged result; root failure leaves +SQLite untouched, while a projection failure retains the root outcome and requires repair by retry. Remove accepts +exact versions, sends only version plus runtime architecture to root, and treats local references as warning-only; +`--force` suppresses that advisory without changing the root request. Local list/default/prune never stat root-owned +paths. The old local downloader, source-build, filesystem removal, path/checksum Jailer installer, and +`mvm run jailer install/remove-release` actions are deleted. Release mutation now has only the typed trusted-release +authority path. Transitional VM launch still uses `ResolvePair` until the separate launch-authority cutover. + +The cycle-free `internal/lib/authorityfs` leaf module now establishes the implemented portion of that split. It creates +or admits exact `root:root` mode-`0711` `/run/mvmctl` and exact `root:root` mode-`0700` +`/run/mvmctl/authority`. Jailer's global index lock, release-lock directory, per-UID authority directories, and per-VM +locks use its closed `Jailer`, `JailerUID`, and `Lock` capabilities, while persistent instance records remain beneath +`/var/lib/mvmctl/instances`. A separate closed `LoopMount` capability pins the authenticated UID authority and its +`loopmount` directory, then acquires canonical VM and image locks as empty one-link `root:root` mode-`0600` files that +are never unlinked during the boot. Existing managed objects are verified and never repaired; L1 tests cover exact +modes, symlinks, concurrent creation, creation-fault descriptor cleanup, record scans, lock placement, contention, +cancellation, and independent-resource concurrency. + +The capability structs retain pointer-shared lifecycle state, so even a copied value performs checked release exactly +once, caches and repeats the first cleanup result, and never retries an unlock or close against a potentially reused +descriptor. Release preserves context values while ignoring cancellation; child capabilities and acquired locks remain +valid after their parent closes its own descriptors. The module exposes closed Jailer-, LoopMount-, and Network-specific +keys and methods, not a raw path, descriptor, or generic lock interface. The private LoopMount service adapter now binds +the transaction coordinator's resource-lock seam to the authenticated UID and canonical VM/image keys, retains partial +acquisition for cleanup, and releases the lock before its authority capability. + +The private network registration foundation now serializes a global network-ID index and each canonical network. Its +strict schema-1 record persists only owner UID, network ID, current boot ID, and exact derived bridge +name/ifindex/host-namespace device and inode at `/var/lib/mvmctl/networks//.json`. The private `networks` +and `` directories are exact `root:root` mode `0700`, and one-link mode-`0600` records are admitted +descriptor-relatively; a bounded exact codec and atomic no-replace publication preserve both created-record and +durability-uncertain evidence. Registration releases its network, UID/store, index, and root authority in exact reverse +order with pointer-shared cached close-once results. The +registration entry point remains private and unreferenced, does not receive the Jailer VM-lock lease, and does not +consume the separate namespace capability. + +The Network authority family now also creates one exact root-owned +`/run/mvmctl/authority//netns/` placeholder and retains its directory, leaf, and mount identities. A private +network effect can create a network namespace on a disposable locked OS thread, publish its exact nsfs mount onto that +placeholder through `open_tree`/`move_mount`, verify restoration of the host network namespace, and remove only the +retained mount and placeholder with explicit uncertain-unmount evidence. The root Jailer lifecycle composes this private +effect into the managed VM topology transaction; ordinary callers cannot invoke it directly. A separate private link +effect completes the `TopologyEffects` boundary: it creates and verifies the exact namespace-local bridge, +host/namespace veth pair, and TAP down; attaches only identity-checked endpoints; and +deletes only exact retained identities. Every rtnetlink operation uses an explicit retained network-namespace +descriptor. TAP creation enters the retained VM namespace on a disposable locked thread, opens only `/dev/net/tun`, +uses `TUNSETIFF` and `TUNSETOWNER` with no group authority, and reverifies owner, no-PI, down state, and the managed +alias before final persistence. The TAP owner is the root-selected isolated Firecracker execution UID, not the +operator UID; the root topology record carries that identity through recovery and cleanup (ADR-0018). Separate private +L3 effects admit canonical endpoint values, configure exact managed-bridge and down-loopback addresses, inspect +kernel-created IPv4 routes, and provide per-role raise/lower methods. Address mutation re-admits the exact topology +before and after the effect; loopback verification follows the +requested `IFF_UP` state, and errors preserve partial/unknown-outcome and close evidence. The shared managed bridge +must already be up during down-endpoint preparation. These low-level methods do not enforce the complete activation +sequence or supply firewall authority themselves. The root lifecycle now composes Jailer VM-lock transfer, +firewall-gated activation, persistent VM-topology records, runtime mounting, and same-boot recovery. Named managed +network create/remove receiver actions consume logical identity and L3/NAT intent. Creation applies the complete +network firewall generation before raising the managed bridge; removal requires exact root ownership and absence of +references. Live-kernel qualification remains pending. + +ADR-0020 defines a distinct, explicitly selected networkless VM placement for the required no-IP VM-to-VM exec case. +It still requires a root-pinned per-VM namespace and Jailer membership, but has only loopback and no managed-network +endpoint, TAP, or guest IP/MAC. The user VM row can now persist the placement variant and a genuinely absent network +foreign key, and the Firecracker config generator supports the explicit networkless no-NIC/no-`ip=` shape. The closed +public create input rejects conflicting network fields, and explicit `--no-network` now resolves without a managed +default network, TAP, MAC, or IP lease. Omission still selects the default managed network. The private root path +now carries a typed networkless namespace lease through Jailer launch, activation, cleanup, and same-/prior-boot +recovery, with placement sealed in the protected device and instance records. The root launch wire now admits only +complete managed or endpoint-free networkless shapes; +its offline NoCloud allowance is limited to fixed local ISO/injected-seed arguments. Relaunch reconstruction retains +the stored placement and rejects malformed networkless state before reboot stops a VM. Offline cloud-init seeds can +omit network configuration. Public SSH rejects networkless placement in favor of vsock exec. Snapshot capture and +restore preserve the endpoint-free placement. A focused installed-candidate journey in the disposable runner passed +consistent full capture/restore of a networkless VM with one attached raw volume, independent restored bytes, and +post-restore vsock exec. Snapshot RAM staging is disk-backed inside the admitted jail root, not under `/run`. +This does not qualify networked policy behavior, the full Python suite, or the release. +Omitting `--network` must never become an implicit networkless fallback. + +ADR-0017 fixes the L3 contract consumed by the typed root transactions. The per-VM +namespace is an L2 extension of one managed bridge. A canonical RFC1918 IPv4 prefix is limited to `/30` or shorter; +an omitted gateway resolves to +the first usable address. Only the host managed bridge (`gateway/prefix`) and guest `eth0` (`guest/prefix`) are +numbered. The veth pair, namespace-local bridge, and TAP are unnumbered and own no routes. Namespace loopback is +`127.0.0.1/8`; the host bridge and guest receive only their kernel-connected routes, and the guest receives the default +route through the resolved gateway, all without an explicit metric. Default guest MACs use `02:fc` plus four bytes of +the versioned, NUL-framed owner/VM SHA-256 derivation; the selected globally unique unicast MAC is persisted. + +Public NAT intent is only enabled or disabled; v0.3.0 removes the public/unprivileged `NATGateways` interface-name +choice. For enabled NAT, root discovers eligible IPv4 main-table default routes through typed rtnetlink pinned to the +host network namespace. Multiple routes are permitted only when all resolve to the same uplink ifindex. Root retains a +deterministic representative independent of route-dump order, persists its exact route and link identity, and +revalidates it before compile/apply/activation. No eligible uplink, distinct uplinks even at different metrics, +multipath, replacement, or changed retained evidence fail closed; multi-uplink selection is unsupported. The public +NAT interface-name selectors are removed. Named root managed-network create/remove and exact removal receipts are +implemented, and core/API network mutations now consume them. Retrying a pending desired network requires exactly +matching inputs. Missing root authority does not authorize cleanup of an interrupted local projection. System +qualification remains pending. + +Production topology preparation must consume the opaque child `TopologyHold` from the real Jailer VM lock and retain +it, the namespace pin, and later firewall endpoint authority through active or retryable cleaning state. After all +links exist down, an opaque freshly verified firewall endpoint lease is the only activation capability. Activation +raises loopback, local bridge, TAP, namespace veth, then host veth, with the host veth last as the exposure gate; every +step uses exact descriptor-bound rtnetlink inspection. Root persists stable network/endpoint inputs, observed +boot-scoped identities, and lifecycle progress, while re-deriving names, paths, routes, loopback, and ordering. Cleanup +lowers the host veth first, freshly removes/verifies firewall state, then removes exact topology in reverse dependency +order without releasing retained authority on retryable failure. Incompatible `/31` or `/32`, non-private, legacy +host-global, mutable-name, schema-1, or ambiguous state is a clean-break refusal, never adoption or automatic deletion. +These activation and same-boot recovery paths are now composed by typed launch and cleanup. Prior-boot recovery and +system qualification remain separate completion gates; this implementation pass supplies no runtime evidence. + +The private Jailer resource foundation now treats caller-owned `firecracker.json` as admission evidence only. It +rejects malformed UTF-8, requires complete semantic equality with root-built typed intent, closes the source, and +retains a root-owned `0444` sealed memfd. Final admission requires exact `FD_CLOEXEC`, all required immutability seals +while permitting kernel-added `F_SEAL_EXEC`, and cancellation-aware canonical readback. Copied private handles share +one synchronized, cancellation-independent close and retain its first result, including cleanup errors. Typed +privileged launch now consumes the sealed configuration in its descriptor-only child mount transaction. + +Persistent output preparation is also implemented privately. A successfully registered durable `launching` lease +derives the owner, VM, and cache identity and retains the output capability before admission or mutation while both +children remain blocked. The fixed log and console leaves, plus metrics only when enabled, must be caller-owned +one-link `0600` regular files. Existing admitted files are truncated once; missing files are prepared with +`O_TMPFILE`, synced, and linked without replacing a competing name. Partial attempts retain cleanup and mutation/ +durability evidence and cannot be replayed. The capability transfers through abort, activation, and cleanup with +shared close-once release; published files remain for inspection. Typed launch consumes these output descriptors. +The policy promises one-launch retention and bounded readers, with no hard live-size cap. + +VM data volumes now use only raw backing files at `volumes/.raw`. The public create input and CLI no longer +accept a format selection; even `--format raw` is rejected. Read-only, shareable, and writeback semantics are preserved, +as are QCOW2 image import/conversion and environment image-format selection. Unsupported persisted volume rows remain +inspectable/removable, but VM create/start/reboot/common snapshot relaunch and volume attach/resize require a complete +raw-volume selection before VM-specific or lifecycle effects. Relaunch resolves persisted volume IDs exactly rather +than accepting a name/prefix substitute, and carries the verified selection through its normal-process relaunch state. +These API input checks use the user database and do not establish root descriptor authority. The separate private raw +volume lease now pins fixed ID-derived leaves and admits exact file identity, logical length, access, and cache mode. +Its positive signed-file-offset size bound is representability only. The user approved raw-volume attachment without +new per-volume or aggregate storage quota caps, subject to the approved device budget and strict file ownership and +identity checks. Typed launch now carries ordered path-free raw-volume selections through root descriptor admission, +sealed configuration, and child mounts at `/volumes/.raw`. Child handoff repeats pinned identity checks and +retains read-only/cache intent; cleanup admits only exact empty jail placeholders, never source data. This unverified +implementation does not reserve physical storage. Typed live hotplug now has root-owned intent and exact mounted +resource identity. Its mount worker unshares `CLONE_FS` before joining the retained VM mount namespace and retires the +OS thread afterward; process authority is verified before and after the effect. The jailed namespace has no `/proc`, +so live detach unmounts the checked volume basename relative to its pinned root-owned volume directory. An earlier +focused installed-candidate running-VM force-removal case passed, but the 0.3.0 public removal contract now refuses +that operation: `volume rm --force` requires root-observed stopped VM status before deleting an attached volume. +Attached-volume growth likewise requires a stopped VM, not detach. On later VM teardown, a completed +`detached` journal proves the protected volume may already be removed; stopped cleanup verifies the jail target is +absent and does not reopen that backing. Transitional records still require the retained backing to revoke access. +Broader live hotplug and guest-I/O qualification remains open. + +The initial private Firecracker 1.16.1 MMIO profile accounts for the complete device budget: rootfs, network, VMGenID, VMClock, +optional cloud-init and vsock, and ARM RTC/serial where applicable. The PCI profile uses the same exact release +authority with separate bus-slot and MSI-X accounting; it does not reuse the MMIO interrupt formula. The root child +persists initial device/volume identity and a configuration digest bound to boot, generation, release, and jail root. +The transport choice reaches child argv and the sealed configuration path. x86 nested-virtualization configuration +is admitted; ARM nested and unapproved release profiles fail closed. These paths remain unverified. +Snapshot byte admission limits the complete `vmstate`, including its CRC, to 10,000,000 bytes and requires memory +length to equal trusted captured RAM in MiB times 1,048,576. Captured rootfs length must match exactly; rootfs-only +captures have neither memory nor vmstate. These private byte checks alone do not authenticate a capture, establish immutable +memory backing, or grant restore authority. The user approved root-controlled snapshot storage with CLI removal and +requires consistent snapshots of attached data volumes. Capture/inspect/remove production code now acquires exclusive +protected backing, pauses the isolated producer, and publishes root-controlled immutable artifact metadata. Uncertain +capture progress retains explicit paused/unknown state rather than assuming a safe resume. Stopped-source device +archives and immutable restore composition now exist in source. Full restore holds memory/vmstate through the restored +VM lifetime; restored disks have independent protected backing while guest drive IDs remain captured IDs. The named +privileged Restore dispatcher route is integrated. A focused installed-candidate networkless volume-snapshot journey +passed inside `mvmctl-030-test`; managed-network restore and the full system matrix remain unqualified. +[ADR-0018](docs/adr/0018-isolated-vm-execution-and-snapshot-authority.md) supersedes the earlier +snapshot pause: schema-6 records separate operator ownership from isolated per-VM system accounts; named root API +operations and an authenticated host-vsock descriptor handoff replace operator access to private VMM sockets. +Protected logical disks, typed volume storage operations, and protected rootfs provisioning replace mutation of +caller staging files after import. The raw source created by `mvm volume create` is a managed-cache artifact: +`volume rm` now verifies its root-recorded inode, size, and change time and removes it after the protected backing, +using the same durable removal lock and tombstone. Detected changed or replaced sources remain untouched and fail +removal; missing sources do not resurrect a removed backing. This is not evidence of an exploited vulnerability or +qualified isolation. +Typed launch accepts the prepared root-owned restore capability; the caller cannot submit snapshot paths, machine +configuration, or alternate network identity. The public restore path has focused networkless L2 evidence only. + +[ADR-0022](docs/adr/0022-explicit-protected-cache-retirement.md) integrates typed retirement of admitted terminal +protected cache parents, preserving root logical authority. Installed public cleanup remains unqualified and existing +permanent snapshot-lock/volume-removal residues still refuse cleanup. The accepted fresh-layout extension in +[ADR-0023](docs/adr/0023-protected-branch-authority-layout.md) puts permanent snapshot locks and volume-source receipts +in the root anchor while retaining artifact/raw-capture bytes on the cache filesystem. Signed `b254f010` implements +that extension, with focused root behavior and race checks passed. Complete integration and installed public +branch-cleanup qualification remain pending; old layouts are not migrated under the approved clean break. + +[ADR-0025](docs/adr/0025-immutable-kernel-cache-lineage.md) records the implemented kernel-only lineage for v0.3.0; +source integration and root Go verification are complete, while fresh-candidate installed qualification is pending. +The same operator may explicitly import identical immutable kernel +bytes into a fresh active cache only after synchronized old kernel removal and completed old-store retirement. +Root must compare against the retained digest, preserve all predecessor authority, and allocate a monotonic global +generation under the existing permanent kernel-ID lock. Durable pending intent precedes candidate creation; lookup, +launch and restore refuse succession until the ready object and committed current binding are synchronized. +The trusted initial snapshot producer may finish its own sealed read-only source descriptor under full restore +locks and exact source-owned pending/aborted authority; ordinary consumers still require committed current authority. +Operator takeover of that initial producer abort is refused before allocating another generation. This is not a removed +backing/cross-cache revival or an ordinary lookup fallback. +The existing typed import request remains cache/ID-only. Fresh kernel logical schema 2 is a clean-break contract, +not adoption of existing schema 1; mutable backing, missing-authority refusal and cache-retirement safeguards remain +unchanged. Affected vet, focused ordinary/race and full root Jailer checks passed with explicit auxiliary worker skips. +No fresh CLI release build, candidate installation or affected installed qualification has run; existing candidate +evidence does not establish this extension. + +For **release testing / RC QA / system tests**, always use `./scripts/build.sh release`; a bare `go build` lacks the +release version metadata, symbol stripping, and PIE settings and must not be used for release qualification. An +untagged RC build must pass the intended version explicitly. Otherwise `git describe` can inherit the previous release +tag and produce a candidate with the wrong identity. ### Asset mirror environment variable (OPTIONAL) @@ -241,6 +773,11 @@ mvm ``` This variable directs the local asset cache for downloaded kernel images, root filesystems, and firmware blobs. +Checksum/version metadata may still be fetched from the fixed upstream origin. The system-test `asset-mirror` volume is +a point-in-time, read-only copy of this directory: after repairing or adding a host mirror object, rebuild that volume +before running QA. Although the downloader can detect a stale object and try its fixed upstream source, the release +runner rejects that fallback and any mirror auto-population as invalid qualification evidence. Builder pulls pass only +when each artifact reports an explicit local-mirror read; checksum/version metadata may still use upstream. ### CLI is the canonical interface @@ -250,33 +787,505 @@ Built with `github.com/spf13/cobra`. Root command in `internal/cli/root.go`. Sub CLI aliases: Domain-level aliases for faster typing: `mvm net` for `mvm network`, `mvm img` for `mvm image`, `mvm vol` for `mvm volume`, `mvm bin` for `mvm binary`, and `mvm ss` for `mvm snapshot`. Environment workflow subcommands have shorter aliases: `mvm env up` for `mvm env apply` and `mvm env down` for `mvm env destroy`. Every `list` subcommand has both `ls` (Use) and `list` (Alias). Every `remove` subcommand has `rm` (Use), `remove`, `delete`, and `del` (Aliases). +The global `--json` flag returns one schema-1 envelope with `status`, typed command `data`, and structured `error`. +Status is `success`, `error`, or `partial`; partial batch output preserves every completed result and individual +DomainError details while returning failure. Startup and argument failures use the same envelope. JSON selection stops +at `--`, so guest arguments do not change host rendering. Exec, SSH, logs, completion, service `run`, the init wizard, +and console attachment reject JSON before command effects; finite console state/kill operations support it. The CLI +owns rendering and must not infer success from discarded API results. Cache inspection/removal now propagates its +underlying errors instead of reporting an empty or successful result. + +VM creation failures use an error envelope with `data: null`, including mixed non-atomic batches. The API returns +completed items alongside the error; the CLI exposes their IDs in `error.details.created_vm_ids`, not success data. +`error.details.failed_vms` preserves each attempted `vm_id`, requested `name`, and structured primary/recovery +`errors` in request order, even when creation never published a database row. These IDs are diagnostic selectors, +not proof of current existence or cleanup permission. An atomic rollback request does not guarantee removal; +exact-ID forced removal still requires the root-owned authority checks. + +### VM-to-VM execution transition + +The legacy guest-originated `remote_vm` handler and protocol are removed. The cutover first made every legacy request +fail closed before source lookup, payload parsing, destination lookup, vsock dial, relay, or command logging; the +isolated source-session slice then removed the generic callback and shared guest connection entirely. The API now +composes the bounded source/destination relay behind `mvm exec --allow-vm-exec` for an explicit non-TTY source command. +It requires root-admitted active VM identities, so transitional launches cannot become eligible through SQLite status +or a policy row. Ordinary host-to-VM execution and all copy directions retain their existing path. + +Directional exact-user exec-policy intent is now public through +`mvm policy exec create `, `mvm policy exec ls [--json]`, and +`mvm policy exec rm [--force]`. Creation resolves the source and destination through the VM domain, rejects +the same VM and the root target user, and derives a deterministic identity from the exact source VM ID, destination VM +ID, and target user. It performs no network lookup: the grant is directional and independent of same-network, +cross-network, or absent IP-network placement. List and removal operate only on the exec-policy family. + +The protocol-only `internal/lib/vmexecwire` foundation is implemented. It uses fixed `MVMEXEC1` magic, version 1, a +big-endian length, and at most 64 KiB per frame. Strict closed schemas distinguish a guest-source request from the exact +host-built destination request: only the destination request carries the canonical lowercase UUIDv4 agent token. Every +request and stream frame carries one 32-lowercase-hex request identity; target users must be non-root, commands are +limited to 32 KiB, combined stdout and stderr to 8 MiB, and reported execution duration to five minutes. The response +grammar allows `stdout` and `stderr` followed by exactly one terminal `result` or `error` and then transport EOF; +`cancel` is the only request-bound control frame. + +The guest agent now recognizes only the exact `MVMEXEC1` prefix as the destination protocol on its existing vsock port; +all other prefixes remain on the ordinary host-exec/copy protocol. It strictly decodes the host-built destination +request, requires the configured canonical token in constant time before effects, resolves the exact target account to +a nonzero numeric UID plus numeric primary/supplementary groups and a clean absolute home, and executes `/bin/sh -c` +with that credential, home, working directory, and a fixed environment. Output and duration use the wire limits, +accepted vsock descriptors are close-on-exec/nonblocking with effective deadlines, cancellation kills the command's +process group, and terminal writes have a separate two-second bound. Errors and logs do not expose the command, token, +or process cause. This process-group boundary is not a per-command cgroup guarantee and does not claim cleanup of a +descendant that deliberately leaves the group. + +The source-session primitive is consumed by the opt-in API relay. A host-side `SourceSession` accepts +only an already authority-admitted connection, authenticates it with exact `MVMSRC01`/version-1 framing and the +configured agent token, and binds one random 32-lowercase-hex source-session ID to one request. Bootstrap and ready +bodies are limited to 512 bytes. The ordinary non-TTY exec path has a separate named method that carries only this +session ID; normal exec, shell, and copy do not open or inspect a source session. Unknown ordinary exec response types +are protocol failures. The legacy generic `OnHostFrame`, shared `activeConn`, `remote_vm`, and +`internal/vsockhandler` paths are removed. + +After a source request passes directional policy, the API obtains the destination's authenticated raw Firecracker +socket through the fixed root broker before the retained VM-pair sidecar takes both VM locks. It sends no destination +guest `CONNECT`, token, command, or request before pair admission. The sidecar then re-proves the actual destination +peer under its retained authority before the host-built destination request can execute. This ordering avoids a +second broker attempt to acquire a VM lock already held by the retained pair; see ADR-0017. The live VM-to-VM exec +system journey remains a release-qualification gate. + +After `ReadRequest` succeeds, `SourceSession.WaitCancel` permits exactly one control reader bound to that request's +ID. It accepts only the matching `cancel` frame. Malformed, non-cancel, wrong-ID, and context-cancelled partial control +input closes the source transport fail-closed; explicit `Close` and context cancellation unblock a partial read. The +cancel reader sets only the read deadline while response writers set only the write deadline, preserving full-duplex +source response output. The API relay coordinates this cancellation with destination cancellation and cleanup. + +The closed host-side `DestinationSession` is also consumed by the API relay. It consumes an already selected connection +and permits one validated `Open` with one host-built `DestinationRequest`. Its strict request-ID-bound response reader +enforces the wire output limits, permits one sequential reader, and withholds either `Result` or `Error` until physical +EOF. After open, at most one exact matching `Cancel` may be written concurrently with response reading. Read and write +operations use independent directional deadlines bounded by one fixed session deadline; the opening context or an +absolute lifetime no later than five minutes closes even an idle session. Operation-context cancellation and explicit +`Close` interrupt partial I/O. Protocol or transport failure closes the connection, and cleanup is close-once with the +first result cached. The type performs no dialing, socket-path selection, VM or policy lookup, root effect, audit, or +API orchestration itself; those responsibilities belong to `pkg/api`. + +Inside the source guest, the agent binds a pending session to the exact outer command process created by the matching +ordinary exec. The outer command may run as root or non-root, starts in a new session/process group, and receives no +token through its environment, arguments, or filesystem. The mode-`0666` local AF_UNIX broker authorizes by process +identity rather than path access: it acquires `SO_PEERCRED` and `SO_PEERPIDFD` together, validates and retains pidfds, +requires matching UID/GID, and twice proves a stable PID/parent/start-time chain to the live outer command before one +claim. The source-session ID is correlation, not authority. Admission is capped at eight concurrent attempts, local +request reads at two seconds, and parent traversal at 64 identities. Missing `SO_PEERPIDFD` or incomplete process +evidence fails closed. + +The broker transfers the dedicated transport from disconnect monitoring to exactly one local request and relays the +closed `MVMEXEC1` stream. Terminal delivery requires physical EOF. Context cancellation closes blocked source-session +reads or writes, ordinary source-command cancellation kills its process group, the first relay disconnect closes both +transports, and agent shutdown releases all pending/bound sessions and pins. The local client form is +`mvm-agent remote --user -- `; it preserves binary output and exposes only generic +local errors. + +The root-owned VM-exec capacity substrate is implemented and production-composed only behind the retained authority +sidecar. Strict schema-1 records live at +`/var/lib/mvmctl/allocations/exec/<32-lowercase-hex>.json`; they bind the non-root owner, exact source and destination +VM IDs and launch generations, host boot ID, and creation/expiration times from `CLOCK_BOOTTIME`. The private +`allocations` and `exec` directories are exact `root:root` mode `0700`, records are one-link exact `root:root` mode `0600`, +and the never-unlinked global lock is `/run/mvmctl/authority/capacity.lock`. Holding that lock serializes the complete +cross-process, cross-user scan/prune, ceiling check, and no-replace publication. Corrupt or ambiguous state fails closed +before canonical pruning; safe crashed temporary files, prior-boot records, and expired records are removed with directory +sync. Creation, deletion, pruning, durability uncertainty, and cleanup-required outcomes remain explicit error +evidence. + +Each reservation fixes its relay-stop deadline at creation plus `vmexecwire.MaxExecutionDuration`; setup and suspend +consume that `CLOCK_BOOTTIME` budget. Its record expires only after a separate four-second terminal/cleanup grace. + +The active VM-exec pair authority is implemented and production-composed behind that same sidecar. It opens the +instance authority once, takes the +global index once, requires two distinct same-owner `active` records, opens each VM authority separately, and acquires +the two VM locks in canonical lexical ID order. After releasing the index, it retains the shared roots separately from +both active leases, reopens both recorded process sets into pidfds, and proves both complete active identities plus +final pidfd locator/liveness. It then reserves root-global capacity and rechecks the host boot ID, both complete +identities, and both final pidfd locator/liveness proofs before exposing an immutable owner/source/destination +launch-generation, reservation, boot, and relay-stop admission. Its close-once cleanup releases destination/source +peer pidfds, capacity, destination/source process capabilities, active leases in descending lexical order, and the pair +roots last without reacquiring a lock. + +Its production constructor fixes a host-global ceiling of eight sessions. Each retained source and destination +connection can be bound to the corresponding admitted Firecracker through `SO_PEERCRED` and `SO_PEERPIDFD`, exact +PID/start identity, matching effective UID/GID, complete active-process identity, and retained-process proof. After +both expensive identity proofs, readiness performs a bounded final locator/liveness sweep of both peer pidfds and both +retained process sets, followed by one final host-boot and relay-budget check. + +The absolute `CLOCK_BOOTTIME` stop is also production-composed. It arms a close-on-exec, nonblocking `timerfd` with +`TFD_TIMER_ABSTIME` and waits through bounded polling with cancellation. Exactly one concurrent waiter is allowed; a +simultaneous second waiter is rejected rather than sharing the in-flight wait, while calls after completion receive the +cached result. Release runs once while retaining cleanup-error evidence. The retained handler starts this stop from the +capacity admission's absolute deadline and owns connection shutdown plus authority cleanup when it completes. + +The exact-one VM-exec descriptor transport and structural admission are used by the registered sidecar action. Its +single-use sender attaches one duplicate through one `SCM_RIGHTS` message on the first non-NUL frame byte, sends the +ordinary remainder without ancillary data through cancellation-bounded nonblocking I/O, and closes the duplicate +exactly once. The receiver uses `recvmsg(MSG_CMSG_CLOEXEC)` for every read, takes cleanup ownership of every installed +descriptor before validation, rejects missing, extra, unknown, late, or truncated control data, and retains only one +close-on-exec connected, non-listening `AF_UNIX` `SOCK_STREAM`. Structural admission deliberately proves no peer +process, Firecracker, VM, guest port, or policy identity. Its private phase-specific receiver transfers a connection +only after the matching closed frame is decoded and exposes only a borrowed connection capability to the next private +effect. + +The retained sidecar wire, process transport, named Jailer client, and root handler lifecycle are production-composed, +and the central dispatcher registers exactly `jailer-vm-exec-session`. The fixed protocol admits only the ordered +`open_source`, +`source_retained`, `bind_destination`, `ready`, `release`, and `released` phases or one bounded terminal failure. The +process transport starts one copy of the existing system binary through the existing privileged invocation and an fd-0 +AF_UNIX control socket, transfers one owned connection duplicate in each setup phase, and bounds cancellation, process +group termination, waiting, and cleanup. The Jailer client duplicates each caller-owned `net.UnixConn`, so the caller +retains its original source and destination connections. + +The handler structurally retains the source before its acknowledgement, retains and binds the destination before ready, +and performs close-once cancellation-detached cleanup on explicit release, control EOF, setup failure, stop completion, +or response failure. It reports `released` only after explicit-release cleanup succeeds. The handler accepts no path, +command, token, target user, policy material, payload, or raw descriptor accessor. Its sole synchronous binder bridge +keeps each handler-owned descriptor opaque, live, and immutable under its owner lock while Jailer proves the peer; the +descriptor is never exposed through the public capability. + +Production dispatch constructs the pair/capacity/peer/absolute-stop server and places it behind this retained handler. +The caller UID/GID passed to Jailer derives only from the authenticated control peer identity and never from request +fields. Authenticated identity, peer, environment, executable, or authorization failures before action dispatch emit +one bounded `MVMXSES1` failure without draining request input. Once dispatched, the handler remains the sole owner of +action-specific responses and destination/source connection cleanup; its server releases the stop first, then the +destination/source peer pidfds, capacity, destination/source process capabilities, descending lexical active leases, +and pair roots. + +The typed `logging.VMExecAuditSink` foundation is implemented separately from the relay. It writes one complete +schema-1 JSON object per line to the dedicated user-owned `vm-exec-audit.jsonl` file in the mvmctl user cache +(`~/.cache/mvmctl` by default). Its closed projection contains only `vm_exec`, the request/source/destination +identities, non-root target user, UTC start time, bounded duration, one of the six fixed results, and the +result-consistent optional exit code; it contains no command, agent token, or process cause. The sink serializes record +and close transitions across copies, poisons itself after any incomplete or failed write, closes once, and surfaces +typed validation, I/O, and cleanup errors. + +This file is user-owned observability, not tamper-proof root authority. Construction rejects an effective root caller +because the user-cache pathname is not retained descriptor-relatively. `Record` observes its supplied context and does +not detach cancellation itself. The API relay supplies bounded cancellation-detached audit finalization and surfaces +audit failures with its relay outcome. + +The opt-in relay in `pkg/api` opens the dedicated sessions, invokes the retained root-sidecar client, resolves the +exact directional source/destination/non-root-user policy, constructs the authorized destination request, forwards +bounded output and cancellation, and writes command-redacted audit records. Unresolved destination denials have a +separate bounded audit form. `--allow-vm-exec` is required per source command; interactive execution cannot opt in. +The legacy VM-wide trust flag is removed from the fresh schema, model, configuration defaults, VM create CLI/API, +environment specs, results, and repository persistence; it has no compatibility path into exec policies. +Installed-candidate Python journeys have passed same-, cross- and networkless directional execution, +opt-in/audit checks, interruption, endpoint restart/reboot and restored-endpoint grants inside the named runner; +the exact bindings and remaining release gates are recorded in the system-test coverage matrix. +The typed lifecycle now produces eligible schema-6 active identities; +legacy and incomplete records remain ineligible. +No SQLite VM status, +guest-supplied source ID, socket path, or process-local semaphore may substitute for root runtime authority. + ### Provisioner Backend (LoopMount vs GuestFS -- mutual exclusion) Two independent rootfs provisioning backends. They are **mutually exclusive** -- a single VM or image operation uses exactly ONE backend, never a combination. The `guestfs_enabled` user setting is a **toggle selector**, not a preference. -- **LoopMount** (default, `guestfs_enabled=false`): Uses the compiled `mvm run provision` subcommand via `losetup` + `mount` + `chroot`. No system package dependencies beyond the `mvm` binary itself. +- **LoopMount** (default, `guestfs_enabled=false`): Managed VM provisioning uses typed `ProvisionVM` over the authenticated privileged receiver. The receiver pins the fixed rootfs/seed resources and performs closed filesystem operations; failed requests have no raw provisioning fallback. Image inspection and optimization use their own typed LoopMount operations. - **GuestFS** (opt-in, `guestfs_enabled=true`): Uses `libguestfs` via `internal/lib/provisioner/guestfs/`. Requires `libguestfs` system packages installed on the host. -Provisioner type resolved ONCE at startup in `api.NewOperation()` by reading `settings.guestfs_enabled`. All callers use `op.ProvisionerType` directly. Backend interface defined in `internal/lib/provisioner/backend.go`. - -**Performance:** GuestFS is 3–5x slower than LoopMount for VM creation (9–14s vs 2–5s wall-clock, sequential). Under parallel load GuestFS degrades further due to QEMU lock contention. See `docs/adr/0003-loopmount-guestfs-mutual-exclusion.md` for full benchmark data. - -**Why no fallback chain:** A fallback (try loop-mount, fall back to guestfs) was rejected because: (1) if a user enables GuestFS, they expect GuestFS behavior -- silent fallback to loop-mount violates least surprise; (2) each backend has different sudoers requirements -- mixing them in one session increases the privilege surface; (3) each backend has independent test suites -- a fallback chain requires testing all combinations; (4) an earlier version incorrectly described GuestFS as a "fallback," which caused regression bugs where a stale `guestfs_enabled=true` silently selected the slow backend. - -### Firewall Backend (nftables vs iptables -- mutual exclusion) +GuestFS build and miscellaneous pruning never infer ownership of global processes, locks, sockets, or temporary +appliances from the operator UID or command line. Appliance pruning is restricted to the selected cache directory; +the miscellaneous result retains `guestfs_state: false`. Interrupted global GuestFS state requires independently +established manual ownership before cleanup. This deliberately drops automatic orphan recovery without changing +normal GuestFS invocation or backend selection. -Two independent firewall backends -- **nftables** (default) and **iptables** (legacy) -- selected by the `firewall_backend` setting. Exactly one is active per session. - -- **nftables** (default): Uses `internal/lib/firewall/nftables.go` -- atomic `nft -f -` batch files. -- **iptables** (legacy): Uses `internal/lib/firewall/iptables.go` -- per-rule `iptables` calls. - -Selection logic in `firewall.NewFirewallTracker(backend, xtcommentAvail, db)`. Both DB tables persist independently; only the active backend's table is queried. - -**Why nftables is default:** nftables is the modern replacement for iptables and is the default firewall on RHEL 9+, Debian 11+, Ubuntu 22.04+, and Arch Linux. It provides atomic batch operations via `nft -f -` (all rules applied or none), cleaner rule management (no separate save/restore workflow), and better performance for large rule sets. - -**UFW compatibility:** The nftables backend uses non-hook chains inside the system `ip filter` and `ip nat` tables, with jump rules inserted at position 0 of built-in chains (FORWARD, POSTROUTING, INPUT). This ensures MVM rules evaluate before UFW's. When UFW reloads (`ufw reload`), it flushes built-in chains and removes MVM's jump rules -- these are re-created lazily on the next `mvm network` or `mvm vm create` operation. The iptables backend has the same limitation. +Provisioner type resolved ONCE at startup in `api.NewOperation()` by reading `settings.guestfs_enabled`. All callers use `op.ProvisionerType` directly. Backend interface defined in `internal/lib/provisioner/backend.go`. -The `FirewallTracker` also reads an `iptables_xtcomment` user setting that adds comment tags to iptables rules for easier identification. +Task 13 now has an activated private LoopMount receiver path for VM provisioning and read-only image inspection in +`internal/service/loopmount/`. Its named +`ProvisionVM`, `InspectImage`, and `OptimizeImage` client methods use closed VM/image request, plan, and result types. +VM cloud-init OFF queues independent `DisableCloudInit` intent; it never relies on image import or deblob to +disable guest services. The receiver writes fixed marker/datasource content, masks the supported systemd cloud-init +units and removes fixed OpenRC cloud-init activation links beneath the admitted root, without unrelated deblob effects. +Disable and inject intent together are rejected before authority acquisition. A successful filesystem sync qualifies +the request-bound `DisableCloudInitApplied` result; the client rejects a missing or unrequested acknowledgement. +Guest sudo provisioning uses the fixed `/etc/sudoers.d/mvm-guest` policy file, preserving `/etc/sudoers.d/mvm` for +the controller's marker-only host authorization if the guest later becomes a nested runner. +VM resize is grow-only relative to the admitted rootfs size. The protocol bounds rootfs and image sizes, guest +usernames, SSH key count and bytes, agent port, and canonical UUIDv4 token before exchange. A result becomes authority +only when the privileged process started and the transport observed actual response EOF; otherwise stable transport +error details report `process_started` and `outcome_unknown`, while an EOF-qualified remote `DomainError` retains its +code, class, entity, and bounded details. + +The private rootfs-authority foundation accepts only an authenticated non-root caller and one canonical managed-cache +locator. It pins the cache through descriptor-relative no-symlink/no-magic-link traversal, permits mount crossing only +while locating the cache, requires the final caller-owned cache to use the closed local-filesystem set, and records its +device, inode, and typed mount identity. Descendants add `RESOLVE_NO_XDEV` and select only fixed +`vms//rootfs.img` or `images/staging//rootfs.img` leaves. Admission enforces caller ownership, one-link +regular-file type, safe owner access and modes, exact bounded size, stable metadata, and action-specific read-only or +read-write access. The lease retains the image staging directory when replacement will eventually be required and +releases every descriptor in checked reverse order with cancellation-independent cleanup. Deterministic L1 races +replace each caller-owned ancestor and leaf immediately before and after open; admission either fails closed or retains +the original object rather than attacker content. + +The private rootfs mount-source policy consumes only pathless receiver-derived block-identity and filesystem +probes. It accepts at most 16 partition probes. A raw source is valid only when there are no partitions and its probed +filesystem exactly matches the request; a partitioned source is valid only when exactly one partition matches. +Malformed probes, reused or duplicate block identities, duplicate partition numbers, raw-plus-partition ambiguity, and +missing or multiple matches fail closed. Receiver discovery now enumerates fixed `/sys/class/block` with bounded entry +and partition counts, proves stable membership and exact block identities through retained descriptors, probes only +through fixed `/usr/bin/blkid` (or `/usr/sbin/blkid` only when the first path is absent) with bounded output and exit +policy, a five-second cap, and any earlier caller deadline, +then returns a pathless source capability bound to the exact probe. No caller-selected tool, path, or device name +crosses the seam. + +The pure loop-device kernel contract admits only fixed `/dev/loop-control` character-device identity `10:237`. +It derives `/dev/loop` only for an index from `0` through `2^20-1` and requires block major `7` with minor exactly +`N`. The generated `unix.LoopConfig` carries the exact admitted rootfs descriptor and only +`LO_FLAGS_AUTOCLEAR|LO_FLAGS_PARTSCAN`, plus `LO_FLAGS_READ_ONLY` for read-only access; every other field remains zero, +direct I/O is disabled, and no legacy setup fields are used. Invalid inputs fail closed through the private LoopMount +`DomainError` path. The pure constructor itself performs no I/O; the private attach/release capability consumes it. + +That private capability opens only fixed `/dev/loop-control` and the kernel-derived `/dev/loop` with +`O_RDWR|O_CLOEXEC|O_NOFOLLOW`, then uses `fstat` to recheck the exact control and block-device identities. It obtains +each candidate through `LOOP_CTL_GET_FREE` and retries only configure-time `EBUSY`, for at most 16 attempts. One atomic +`LOOP_CONFIGURE` binds the admitted rootfs descriptor and closed access mode. Configure failure closes the candidate +without `LOOP_CLR_FD`, because another process may have won the free-device race. + +The resulting package-private lease exposes no path or descriptor accessor. Its checked, idempotent release uses +`LOOP_CLR_FD` and closes the loop descriptor under cancellation-independent cleanup; `AUTOCLEAR` remains crash safety. +If configuration succeeds but loop-control close fails, attachment returns both the configured lease and the error so +the caller can still release it. + +The read-only image-inspection adapter locks its OS thread, enters a private mount +namespace, makes propagation private, mounts only the admitted source at fixed `/run/mvmctl/rootfs` with +`ro,nosuid,nodev,noexec`, and opens the inspection-only backing descriptor with `O_NOATIME` before revalidating the exact +source, mountpoint, filesystem, and backing-image identity. It reads +only bounded fixed `etc/os-release` or `usr/lib/os-release` leaves through no-symlink/no-cross-mount traversal, then +releases root, mount, source, partition, loop, mountpoint, and image authority in checked reverse order. `unchanged` +requires complete admitted/statx stability and successful cleanup; otherwise the closed result is `outcome_unknown`, +while the primary `DomainError` and bounded cleanup details are preserved. The short-lived receiver handler exits +rather than returning its unshared OS thread to the Go runtime. + +The private `ProvisionVM` adapter consumes the admitted rootfs capability directly. It +grows the admitted ext-family filesystem through fixed `e2fsck`/`resize2fs` or Btrfs through fixed +`btrfs filesystem resize`, with a five-minute tool cap and any earlier caller deadline, and has ext4/Btrfs convergence +evidence from either the exact old or target backing size. Every guest read and write is descriptor-bound beneath the +admitted mounted root. Its closed content set includes hostname/hosts, DNS, missing-account creation, SSH/sudo, +fixed guest-boot SSH setup, embedded agent, fstab, a root-only persistent +`/etc/mvmctl/agent.token`, and exactly one admitted init-system enablement: systemd `sysinit.target` or OpenRC boot +runlevel. [ADR-0024](docs/adr/0024-early-openrc-agent-startup.md) moves only the OpenRC agent before +network/clock work, requiring local mounts and ordering after module initialization when present. Provisioning +converges only an identity-verified former default-runlevel agent link, with parent synchronization and safe retry; +foreign entries refuse before mutation. Focused Go and root checks passed, followed by installed-candidate +cold networkless five-second exec, directional exec-policy and pause/resume cases (strict 3/3 with checked +cleanup and baseline preservation). Other startup variants and complete release qualification remain open. +The existing binary admission is required only after both its init-script and default-runlevel directories are +present. This admits Ubuntu's usrmerged systemd layout without following `/sbin` aliases; complete OpenRC candidates +still require the confined binary check, and missing or ambiguous init layouts are rejected before agent publication. +Existing accounts are retained. A missing non-root account receives the first free UID/private GID in +1000–60000, a locked shadow entry, `/home/`, `/bin/sh`, and a bounded descriptor-pinned `/etc/skel` copy; passwd +publication occurs last. Both root and the requested account receive selected SSH keys. SSH installation, host-key +generation, and service activation that require guest execution are fixed guest-boot actions, never host execution. +The receiver admits only the managed VM seed leaves `cloud-init/meta-data`, `cloud-init/user-data`, and optional +`cloud-init/network-config`, then writes fixed guest destinations. No caller-selected source or destination path crosses +the request. Cleanup unwinds root, mount, source, partitions, loop, mountpoint, and backing authority in reverse and maps +mutation, durability uncertainty, and unknown outcomes without losing the primary error. + +The private `Server` coordinates each admitted operation through two action-specific effect seams. +`rootfsEffects` contains exactly `ProvisionVM` and `InspectImage`, while the private `imageOptimizeEffects` contains +only `OptimizeImage`. Provisioning and inspection validate `rootfsEffects`; optimization validates +`imageOptimizeEffects`; every action separately validates the shared rootfs authority and resource-lock seam. After +validating the closed request and authenticated caller, the coordinator acquires a resource lease through the private +`rootfsResourceLocks` seam, admits the fixed rootfs, invokes the corresponding effect, and validates the request-bound +result. The lock seam has a private adapter over the implemented `authorityfs` LoopMount UID and canonical VM/image +lock capabilities. The composition root binds the exact embedded-agent accessor for provisioning and inspection. +A separate `NewImageShrinkServer` now supplies offline ext2/ext3/ext4 shrink with durable replay authority. Its fixed +root-owned `/var/lib/mvmctl/loopmount//.json` journal records `prepared`, `sized`, and `complete`, bound to +the admitted cache/image identity and exact request. Typed deblob and fstab repair compose with that shrink journal; +nonshrinking Btrfs and XFS cleanup use the same closed descriptor-bound content operations. XFS inspection and VM +provisioning are supported at existing capacity; resize and shrink fail before backing or filesystem mutation. + +Btrfs-to-ext4 conversion has separate durable `prepared`, `snapshot`, `ready`, and `complete` phases. Root retains a +snapshot and digest of the admitted source plus the exact converted candidate and publication identity. Conversion +and later shrink are separate requests. An interrupted exchange retains the displaced original for reconciliation; +caller paths and unrecognized remnants never provide replacement authority. These implementations remain unverified. + +The coordinator requires an action-specific effect cleanup capability and unwinds effect cleanup, rootfs authority, +then the resource lock, each with `context.WithoutCancel`. Effect outcomes use closed action-specific states to derive +the mandatory `rootfs_modified`, `image_replaced`, `durability_uncertain`, and `outcome_unknown` error evidence; +invalid or missing state cannot authorize success. Cleanup failures add bounded details without replacing the primary +`DomainError` identity. + +VM grow replay admission accepts only the stable pre-grow size or the exact requested target size, rejects any third +size before an effect begins, and retains the actually admitted size on the rootfs lease handed to the narrow effect +seam. Offline ext-image shrink now persists and consumes replay authority before sizing or truncation. The prepared +phase retains original backing length; interrupted work must pass fsck, resize, fsck, and exact superblock admission +before truncation. Provisioning and read-only inspection are reachable through their exact internal privileged +actions. Image OS detection now stages a normal-process copy at exclusive +`images/staging//rootfs.img` and invokes typed `InspectImage`. Reflink or sparse copying preserves logical +length; the caller syncs the copy and detects source size/time changes before inspection. Existing staging work can +resume only through its fixed bounded recovery journal and lock, exact source/directory identity, and matching intent. +The caller-owned journal is retry intent only: resume reissues the exact typed operation and a checked root result is +required before publication. Publication uses atomic exchange, retains the original until durability is established, +and refuses a filesystem without the required exchange operation. Unknown or changed remnants are preserved. The +privileged request contains canonical cache/image identity, expected bytes, and filesystem rather than the source +path. Staged-image handling also supports the shrink transaction. Managed VM provisioning now preserves account +creation, cloud-init injection, and deblob through the typed backend. The descriptor-bound deblob implementation +performs bounded fixed-directory cleanup and distribution-specific configuration without executing guest code on the +host. Arch package-key initialization and Btrfs balance are fixed guest-boot actions; preparing those actions does not +mean they have run. Failed typed provisioning retains the original size intent for retry and never invokes the raw +fallback. Typed conversion and caller staging recovery are implemented. The public `mvm run provision` route is +removed together with its obsolete entry, spawn, wire, and provisioner source files; cache cleanup no longer performs +arbitrary unmounts, and cleanup errors propagate. Unprivileged archive +construction remains outside the root service. Remaining dead-code retirement and live-kernel qualification are +separate from this implementation-only pass. + +**Historical performance:** Pre-0.3.0 measurements found GuestFS 3–5x slower than LoopMount for VM creation (9–14s vs 2–5s wall-clock, sequential), with further parallel-load degradation due to QEMU lock contention. These figures do not qualify the current typed receiver. See `docs/adr/0003-loopmount-guestfs-mutual-exclusion.md` for benchmark data. + +**Why no fallback chain:** A fallback (try loop-mount, fall back to guestfs) was rejected because: (1) if a user enables GuestFS, they expect GuestFS behavior -- silent fallback to loop-mount violates least surprise; (2) the backends have different host dependencies and effect boundaries -- a fallback obscures which path performed the operation; (3) each backend has independent test suites -- a fallback chain requires testing all combinations; (4) an earlier version incorrectly described GuestFS as a "fallback," which caused regression bugs where a stale `guestfs_enabled=true` silently selected the slow backend. + +### Firewall backend transition + +The public policy CLI exposes the typed `traffic` and `exec` families. Traffic creation now applies a root-authorized +grant before projecting its immutable identity into SQLite. Removal obtains root revocation or exact-absence proof +before deleting that projection. Root admits exact active or cleaned endpoint topology; stopped reservations and grants +survive stop and become effective again only with an admitted live endpoint. VM removal invalidates both. Root success +followed by local projection failure remains a reported repair/retry condition. List and inspect show local projection, +which is never authority for kernel mutation. + +Traffic also supports explicit VM-to-host and network-to-host grants. Root derives the source managed network's +gateway; the request accepts no host IP. Enrolled operators may select any TCP/UDP destination port or bounded range, +but must name it explicitly. These grants affect only the inet input path, preserve VM-to-VM and host-to-VM behavior, +and retain distinct root actions from fixed NoCloud access. Environment version 1 uses `destination_host: true`. +See ADR-0017 for canonical digest, persistence rollout, ownership and lifecycle details. New host-grant installed +qualification remains pending. + +Managed network and traffic mutations use typed root nftables transactions. The old service-access production path, +NoCloud legacy-tracker consumer, CLI nftables/iptables implementations, tracker repositories, and backend selectors +are removed. The production adapter uses `google/nftables`; no CLI backend or fallback is available. + +Managed VM topology activation now invalidates only the admitted guest IPv4's dynamic neighbor on the exact root-owned +host bridge before raising the host veth. Root derives the bridge and address from retained network and endpoint +authority, rejects static/proxy/foreign entries, and reads back absence. This replaces ADR-0006's broad raw-sudo +bridge flush; see [ADR-0021](docs/adr/0021-exact-managed-neighbor-invalidation.md). A disposable runner namespace +proved the rtnetlink operation, but reused-IP VM behavior still needs bridge-capable installed-candidate QA. + +NoCloud now uses fixed gateway TCP `8000` through typed apply/revoke requests containing only the owned network ID. +Root derives the gateway, protocol, and port. One live batch binds each network's listener before the grant is applied; +a collision neither adopts nor terminates the existing listener. Normal shutdown revokes while retaining the listener +socket and then releases it. The grant authorizes a reserved service slot, not the listener process: interruption can +leave that durable slot grant for another invocation or network teardown. Custom ports are rejected. This implements +the accepted fixed-slot contract without claiming live qualification. + +Host readiness executes the fixed typed compiler/application/readback probe only in a disposable network namespace. +It proves the modeled bridge/inet, connection-tracking, and NAT operations available for that probe; it does not mutate +host tables or establish release qualification. Operator invocation uses the authenticated dispatcher; administrator +setup calls the same root-side probe directly to avoid reacquiring its own exclusive admission lock. + +Other root or `CAP_NET_ADMIN` tools are trusted administrators, but detected changes to managed firewall authority +fail closed. The former UFW jump-rule recreation behavior and `iptables_xtcomment` setting no longer apply. + +The private v0.3.0 compiler foundation under `internal/service/firewall/` pins `github.com/google/nftables v0.3.0` and +accepts only fully resolved mvmctl-owned intent. It emits fixed `mvmctl_bridge` and `mvmctl_inet` tables and fixed hook +chains. Bridge prerouting uses raw priority for source MAC/IPv4/ARP anti-spoofing and unsupported-L3 guards, while +bridge forward compiles same-network allows and default deny. Inet forward also compiles exact same-network companion +allows ahead of its managed-bridge default drops, because `br_netfilter` may send bridged IPv4 through IP forward; +the bridge-family endpoint and ingress anti-spoof gates remain authoritative whether that host setting is on or off. +Inet forward/input/postrouting also compile routed allows, established replies, host-service access, default deny, +and conditional egress/NAT; disabling masquerade emits neither uplink egress nor NAT. Compilation rejects overlapping +managed IPv4 prefixes and unusable network, host, or endpoint addresses. The caller supplies no generation identity. +The compiler derives SHA-256 from a versioned canonical closed +projection of normalized intent and the exact managed tables, chains, rules, expressions, order, and userdata schema. +The userdata generation slot uses a fixed placeholder while hashing to break self-reference; the derived digest and +rule identity are then bound into every rule. Explicit empty intent emits no reserved tables or rules. + +The projection covers all fields of exactly the eight emitted `github.com/google/nftables v0.3.0` expression structs: +`expr.Meta`, `expr.Masq`, `expr.Bitwise`, `expr.Cmp`, `expr.Ct`, `expr.Payload`, `expr.Range`, and `expr.Verdict`. +Typed-nil or unknown expressions fail closed. Digest encoding streams through a fixed 64-KiB buffer directly into +SHA-256, and canonical validation, encoding, and finalization errors surface to the compiler caller. Any +nftables-module upgrade or change to the emitted expression set requires an explicit schema audit and digest-version +bump. Input ordering is deterministic. +ARP admission requires Ethernet/IPv4 request/reply grammar and a complete 28-byte payload; header/opcode guards precede +the fixed sender-address checks and are revalidated before private application. The representative +1,000-endpoint/1,000-same-network-grant fixture adds 1,000 inet companion rules. On the named Go 1.26.3 runner, +compilation used 279,352 allocations and 12,950,960 bytes, versus 243,345 allocations and 11,683,904 bytes before +those rules. Its fixed allocation guard is 290,000 with the 6x growth guard retained. Focused same-network and +cross-network TCP grant cases passed in the disposable runner, including default deny, exact-port allow, +adjacent-port deny, and revocation; broader packet and reboot qualification remains open. + +The compiler itself remains effect-free. A separate implemented authority foundation provides the never-unlinked +global `/run/mvmctl/authority/firewall.lock`, a strict committed/applying generation record and bounded codec, and a +descriptor-relative exclusive store at `/var/lib/mvmctl/firewall/generation.json`. Its private acquisition seam first +requires earlier network authority, then binds the admitted record to the current boot and pinned host network +namespace while retaining pointer-shared cached close-once cleanup authority. It fails when the record is missing +because only qualified kernel inspection may establish initial authority. An admitted lease can atomically replace the +record only after re-reading both its exact contents and retained file evidence; post-rename readback preserves exact +`record_replaced` and durability evidence and prevents another mutation after an ambiguous outcome. + +The admitted lease now also retains a private application mutex across the complete record -> kernel -> record +transaction. The mutex serializes copied lease values, standalone record replacement, application/reconciliation, and +generation-lease `Release` while the global firewall lock remains held. Each observation or application connection +separately revalidates the caller-supplied retained namespace lease under that lease's lifecycle lock. The application +mutex neither retains the namespace lease nor serializes its release, so the namespace lease may otherwise be released +between calls. Before any kernel effect, the adapter revalidates the compiled generation's closed application seal, +resource inventories, table/chain/rule shape, and terminal default-deny invariants against the durable applying record +and a fresh exact kernel snapshot. + +The raw-netlink collector issues only receiver-chosen read requests for the two exact managed tables, five fixed chains, +their rules, and the absence of sets, objects, and flowtables. It brackets collection with ruleset-generation reads, +uses a shared 32-MiB/20,000-message budget across fresh single-request transports, and feeds the replies to the strict +raw decoder, which rejects unknown expressions and structural drift. The comparator covers every compiler-projected +field in chain-by-chain rule order; the decoder checks exact predecessor-handle evidence and one consistent 16-bit +`nft_base_seq`, which the collector binds to the bracketed ruleset generation's low 16 bits for non-empty state. +The retained namespace authority requires the earlier network and generation identities to agree, pins and rechecks +the exact current host network namespace, and supplies both fresh snapshot transports and a fresh lasting +`nftables.Conn` through that descriptor. + +Application creates the lasting connection while a disposable locked OS thread is in the retained namespace, then +restores and verifies the original namespace before returning it. The retained netlink socket has verified 32-MiB +receive/send buffers and a deadline equal to the earlier of the caller deadline and 30 seconds. The adapter queues +deletion only for the exact observed managed tables, queues the complete target, calls `Flush` once, and closes the +lasting connection. Cancellation expires that retained socket's deadline without clearing the lasting connection +before `Flush` completes, so it cannot redial a transient socket outside the selected namespace. Cancellation, socket +interrupt, flush, and close errors are all preserved; cancellation cannot be reported as success. + +The durable `applying` record precedes mutation. A fresh exact-before observation authorizes one flush, exact-target +takes the record-completion path without replay, and unknown state preserves applying authority. After a commit attempt, +one fresh inspection uses a cancellation-independent five-second recovery context under the same transaction lock, +global firewall lock. That observation separately revalidates the caller-supplied namespace lease under its lifecycle +lock. Exact-before attempts to restore the committed-before record and returns an error; exact-target attempts to commit +target while retaining any application error; and unknown preserves applying state with explicit +`kernel_generation_state` and `outcome_unknown` evidence. A known failure before the commit boundary attempts to restore +committed-before without pretending that a kernel mutation occurred. Record-finalization failures remain explicit, and +an ambiguous committed replacement blocks later mutation. + +Applied empty now means both reserved tables are absent. Exact empty-to-empty verifies absence through fresh raw-netlink +observation without a mutation/`Flush` transaction; an exact admitted non-empty generation may delete both tables in +its one replacement batch. Partial, foreign, mismatched, corrupt, wrong-boot, wrong-namespace, missing-record, or +otherwise ambiguous state never grants deletion authority. + +The root managed-network, topology, and traffic-policy transactions now consume this application/reconciliation path. +Exact host-service apply/revoke and the NoCloud fixed-slot cutover are implemented. Residual legacy code removal, +direct capability probes, real-kernel/Tier evidence, and release +qualification remain pending. The flat public policy family has been replaced by typed commands. + +ADR-0017 defines the remaining clean-break replacement. Before any fixed-name table can be replaced or removed, root +must hold the never-unlinked global `/run/mvmctl/authority/firewall.lock` and bind the exact observed generation to the +strict `/var/lib/mvmctl/firewall/generation.json` authority record. The compiler now derives the canonical digest, but +that digest alone is neither kernel ownership nor deletion authority. The private adapter implements record-bound +exact-state deletion and empty application, but no missing-record initial-adoption path. Missing, corrupt, mismatched, +or ambiguous authority never permits deletion. + +`github.com/google/nftables v0.3.0` exposes rule userdata but no table/chain ownership metadata, and its structured rule +reader can omit expression types it does not understand. It remains the private firewall compiler and rule-mutation +engine. Direct `github.com/mdlayher/netlink` use in that adapter is limited to strict read-only raw observation, +collection, and namespace-bound transport, plus buffer configuration, deadlines, cancellation, and cleanup of the socket +retained by `google/nftables`. It supplies no second rule encoder, mutation engine, or firewall backend. +The raw collector and retained host-netns factory close the expression-decoder and namespace-selection seams. +The private application coordinator composes them only for an +already admitted record; root-record-bound initial live inspection within an accepted threat model for another root or +`CAP_NET_ADMIN` actor and real-kernel delete/recreate batch atomicity remain explicit activation gates. After those +gates, production composition uses only typed transactions. The iptables/backend selectors, backend-specific +repositories, and production `nft` CLI path are now removed. Library and netlink types stay private to the adapter; +there is no CLI fallback. +Kernel nftables, bridge-family, connection-tracking, NAT, and privilege capabilities remain host requirements. A +disposable QA runner may use the `nft` CLI only as an independent read-only oracle. ## Relationships @@ -360,14 +1369,14 @@ host.init.sudoers.failed ONE canonical path for all subprocess calls: `system.DefaultRunner.Run()` / `system.DefaultRunner.Stream()` with `system.RunCmdOpts` in `internal/lib/system/runner.go`. No raw `os/exec.Command` except in documented exceptions. ```go -// Correct -- everything routes through the centralized runner -result, err := system.DefaultRunner.Run(ctx, []string{"ip", "link", "set", tap, "down"}, system.RunCmdOpts{Privileged: true}) +// Correct for an ordinary read-only subprocess; managed root effects use typed dispatch +result, err := system.DefaultRunner.Run(ctx, []string{"ip", "link", "show", tap}, system.RunCmdOpts{}) // Forbidden -- raw os/exec scattered across modules exec.Command("iptables", ...) // NEVER ``` -The `RunCmdOpts` struct configures execution: `Check`, `Capture`, `Cwd`, `Timeout`, `Input`, `Env`, `Privileged`, `Interactive`, `StartOnly`. `Timeout` is an absolute cap for operations that genuinely need one (downloads, builds, cleanup). User commands run until completion or context cancellation; pass `Timeout: 0` for those. +The `RunCmdOpts` struct configures execution: `Check`, `Capture`, `Cwd`, `Timeout`, `Input`, `Env`, `Privileged`, `Interactive`, `StartOnly`. Its `Privileged` field remains part of the generic runner, but managed network and rootfs mutations use typed authenticated dispatch. `Timeout` is an absolute cap for operations that genuinely need one (downloads, builds, cleanup). User commands run until completion or context cancellation; pass `Timeout: 0` for those. ### Timeout taxonomy @@ -381,13 +1390,22 @@ The `RunCmdOpts` struct configures execution: `Check`, `Capture`, `Cwd`, `Timeou User-facing `--timeout` means **connect/probe timeout only**. Once the target is responsive, the operation runs unbounded. See [ADR-0013](docs/adr/0013-user-facing-timeouts-are-connect-timeouts.md). +The private root launch child separately bounds authentication/challenge admission and post-release setup to +30 seconds each. Protected preparation waits under the original child context while the authenticated parent +owns its release gate; parent death or abort closes that gate and fails closed. Only the exact release byte starts +the fresh setup budget. This is not a new absolute VM-import timeout or a change to public probe semantics. + ### Timeout policy: CLI flags vs test infrastructure **Two distinct concerns that are NOT the same thing.** **1. CLI `--timeout` flags (user-facing, e.g., `mvm exec --timeout`, `mvm ssh --timeout`)** -These are **connect/probe timeouts** only (per ADR-0013). Default: **5 seconds**, absolute max: **10 seconds**. Once the target responds, the operation runs unbounded. This is the speed-first principle: if a connection can't be established in 10s, something is wrong. +These are **connect/probe timeouts** only (per ADR-0013), not command-duration limits. Defaults are command-specific: +`mvm exec` uses `defaults.vm.vsock_probe_timeout` (5 seconds), while `mvm ssh` uses +`settings.vm.ssh_timeout_sec` (10 seconds). A positive explicit `--timeout` overrides that command's configured +probe budget; the current public inputs do not impose a universal 10-second maximum. Once connected, command +execution has no absolute duration cap and ends on completion or context cancellation. **2. Subprocess timeouts in test infrastructure (the `timeout=` parameter of `_run_mvm()` / `_guest_run()`)** @@ -409,17 +1427,21 @@ Rationale: subprocess timeouts are a **safety net**, not a performance floor. Th | Location | Why `DefaultRunner.Run()` doesn't work | |---|---|---| -| `internal/core/vm/firecracker.go` (Firecracker spawn) | Fine-grained control over stdin/stdout/stderr FD redirection and `Setsid` session management for the Firecracker child process | +| `internal/service/jailer/entry.go` (Jailer exec) | Privileged chroot setup ends with `syscall.Exec` so Jailer inherits console stdin/stdout/stderr and becomes the managed Firecracker process | +| `internal/service/jailer/lifecycle_child_spawn.go` | Fixed-binary `os.StartProcess` creates the authenticated handshake-blocked root child with its closed inherited channels | +| `internal/service/jailer/lifecycle_child.go` | `syscall.Exec` preserves admitted child identity and descriptor-only mounts during the fixed Jailer handoff; the early route exits on every outcome | +| `internal/service/jailer/lifecycle_console_spawn.go` | Root creates the exact PTY and authenticated gate, then `os.StartProcess` executes only the fixed system-binary console marker after dropping to the caller UID/GID with empty supplementary groups; pinned output descriptors are supplied through the closed setup channel | | `internal/core/ssh/utils.go` (SSH connectivity probe) | Uses `exec.CommandContext` with a short-lived probe context for SSH connectivity detection | -| `internal/service/loopmount/provisioner.go` | Direct provisioning engine running losetup/mount/umount/chroot in chained operations with precise error recovery | | `internal/service/agent/exec.go` (command execution) | Uses `exec.CommandContext` for `su` user switching and `sh -c` command execution inside the guest agent | | `internal/service/agent/pty.go` (PTY session) | Uses `exec.CommandContext` for `su` user switching to establish PTY sessions | +| `internal/service/agent/destination_process.go` (VM-exec destination) | Uses `exec.CommandContext` only after exact nonzero numeric UID and numeric-group resolution; fixes environment/home and `/bin/sh -c`, creates a process group, and kills that group on cancellation | | `internal/lib/archive/archive.go` (xz decompression) | Uses `exec.CommandContext` for `xz -d --stdout` pipe-based decompression | +| `internal/lib/system/privileged_control.go` (authenticated dispatcher transport) | Fixed sudo/binary/marker exchange needs a duplex fd-0 socket, physical-EOF response authority and owned reaping; only payload-free VM-rootfs export uses a separate session and nonfatal disconnect so root state recovery can finish | | `internal/lib/system/runner.go`, `interactive_run.go`, `spawn.go` | Implementation of the subprocess abstraction layer (`DefaultRunner`, `RunInteractive`, `SpawnService`). These use raw `os/exec` because they ARE the abstraction boundary | **Binary lookup carve-out:** Utility files across the codebase use `exec.LookPath()` (not `exec.Command`/`exec.CommandContext`) solely to check whether a system binary exists before calling it through `DefaultRunner.Run()`. This is NOT a subprocess execution — it's a filesystem existence check that happens to use the `os/exec` package. These files are not listed as exceptions above and do not violate the subprocess rule. Key locations: `internal/core/host/detector.go`, `internal/core/host/probe.go`, `internal/core/host/utils.go`, `internal/core/ssh/service.go`, `internal/core/key/utils.go`, `internal/core/image/service.go`, and others in `internal/lib/`, `internal/core/`, `internal/service/`. -Services running as subprocesses (`mvm run `) use `system.SpawnService(ctx, cfg)` which resolves the executable, optionally prepends `sudo`, and manages process groups. The services themselves (console relay, nocloudnet server, loopmount entry point) do NOT use `os/exec` except for the provisioning engine noted above. +Ordinary-user subprocess services (`mvm run console relay` and `mvm run nocloudnet serve`) use `system.SpawnService(ctx, cfg)` to resolve the same executable and manage process groups; it does not elevate them. Root LoopMount actions use the authenticated typed privileged dispatcher, and Jailer uses the documented `syscall.Exec` exception after validating trusted executables and per-VM resource paths. ### Model types @@ -470,28 +1492,59 @@ Workers defaults to `min((runtime.NumCPU() or 4) * 2, len(items))` (minimum 1) w ## Test types -Three-level architecture — see `docs/development/HOW_AGENTS_WRITE_SYSTEM_TESTS.md` for the full specification. +The test architecture uses L0, L1, and L2 test levels plus T1, T2, and T3 system-test execution tiers. Read +`docs/system-test-architecture.md` for the architecture and `docs/development/HOW_AGENTS_WRITE_SYSTEM_TESTS.md` for +authoring guidance. ### L0: Pure Function Tests (Go `*_test.go`) Table-driven tests with `map[string]struct{...}` and `t.Run()`. No I/O, no DB, no subprocess. Pure input → output assertions. Runs in microseconds. ```bash -go test ./... go test ./internal/core/vm/... ``` ### L1: Hermetic Integration Tests (Go `*_test.go`) -Uses real I/O in controlled environments: in-memory SQLite (`:memory:` via `testutil.NewInMemoryDB`), `t.TempDir()` for filesystem, `FakeRunner` for subprocess calls that can't run in CI. No networking, no KVM, no sudo. Catches bugs earlier during `go test ./...`. +Uses real I/O in controlled environments: in-memory SQLite (`:memory:` via `testutil.NewInMemoryDB`), `t.TempDir()` for filesystem, `FakeRunner` for subprocess calls that can't run in CI. No KVM or sudo. `scripts/run-go-ci-tests.sh` runs every non-Jailer Go package and a selected unprivileged Jailer subset in CI and tag release. The complete Jailer package exercises fixed root-owned authority and runs locally inside a disposable root-test VM. ```bash -go test ./... -count=1 -coverprofile=coverage.out -covermode=atomic +go test ./internal/core/vm/... -count=1 -coverprofile=coverage.out -covermode=atomic ``` ### L2: Runner VM System Tests (Python `tests/system/`) -**Ground truth.** Every user-facing feature must have an L2 test. Real binary, real subprocess, real infrastructure inside a disposable Firecracker VM with nested KVM. No mocking of any kind. Operates against the compiled `mvm` binary. Verifies actual business outcomes at the OS level: JSON state, filesystem state, process state, iptables rules. +**Ground truth.** Every user-facing feature must have an L2 test. Real binary, real subprocess, real infrastructure inside +a disposable Firecracker VM with nested KVM. No mocking of any kind. The release candidate is installed through +`host install-system`, and T1/T2 tests invoke exact `/usr/local/bin/mvm`; a directly copied or PATH-selected developer +binary is not release evidence. Tests verify actual outcomes at the OS level: JSON, files, processes, cgroups, +namespaces, links, mounts, sockets, and nftables state as applicable. + +The outer host controller and the artifact under test are deliberately separate. `MVM_BINARY` names an already +installed controller used only to create, inspect, copy to, and destroy disposable runner VMs. +`MVM_CANDIDATE_BINARY` names the release artifact (default `dist/mvm`) installed only inside those VMs. The runner +image tag comes from an isolated `MVM_CANDIDATE_BINARY --version` probe. Building or testing a candidate must never +replace `MVM_BINARY`. + +Runner initialization requests exact Firecracker/Jailer 1.17.0 with +`/usr/local/bin/mvm init --non-interactive --binary-version 1.17.0` when no local pair exists. The approved launch +pair tracks the newest stable upstream release through an explicit reviewed update, with separate archive admission +and runtime qualification; it is not resolved from a moving `latest` value at launch. Existing local/default selection +on already-prepared images is preserved, so qualification must verify the selected pair is exactly 1.17.0. + +This controller/candidate isolation applies only to T1/T2 runner VMs. Tier 3 is host-direct and may mutate outer-host +VMs, kernels, images, networks, and volumes. It is allowed only on a clean disposable qualification host where the +candidate has been explicitly installed as exact `/usr/local/bin/mvm`; it is not an iterative developer-host test mode. +The runner rejects Tier 3 or `--all` before any work unless `--host-direct` explicitly acknowledges outer-host mutation. +Its stricter `--release-qualification` mode requires an unfiltered `--all`, `--host-direct`, a fresh explicit-version +release build, and exact root-owned `/usr/local/bin/mvm` content/version identity before resource mutation. This consent +and binary gate does not yet prove that every Tier 3 resource is newly owned or that every teardown succeeded: do not +invoke Tier 3 or `--all` on a host with pre-existing mvmctl state until the remaining Task 17 cleanup gates land. + +Before any probe, build, or resource mutation, the runner also validates the system-test registry itself. Domain names +and test files must be unique, registrations must be non-empty canonical regular files under `tests/system/`, and the +registered set must equal the discovered `tests/system/**/test_*.py` set. A missing or duplicate domain therefore cannot +silently produce incomplete release evidence. ```bash # Run inside the runner VM (disposable Firecracker VM with nested KVM) @@ -507,10 +1560,12 @@ L2 tests are the **ground truth** — every user-facing feature must have one. They run inside a disposable Firecracker VM (runner VM) with nested KVM. ### Option C verification -The thoroughness standard for L2 test assertions. Every test verifies system state at the deepest practical level: JSON field assertions from `* ls --json`, file existence/symlink checks, process presence via `/proc`, iptables rule presence, and/or direct SQLite queries. A test that only checks `returncode == 0` is incomplete. +The thoroughness standard for L2 test assertions. Every test verifies system state at the deepest practical level: JSON field assertions from `* ls --json`, file existence/symlink checks, process presence via `/proc`, nftables generation and rule state, and/or direct SQLite queries. A test that only checks `returncode == 0` is incomplete. -### Gap matrix (no longer a separate file) -Coverage is tracked by the quick-reference table in `docs/development/HOW_AGENTS_WRITE_SYSTEM_TESTS.md`. Every CLI subcommand and flag is classified as L0, L1, or L2. All gaps must be filled before release. +### Coverage matrix +`tests/system/COVERAGE_MATRIX.md` tracks command and flag coverage. The authoring guide explains how to classify a +scenario as L0, L1, or L2. Release qualification requires the matrix and executable registry to agree with the selected +system-test suite. ### Edge case categories (8 categories) For every CLI flag, check all eight: happy path (with state verify), missing required args, invalid values, boundary values, JSON output format, confirmation prompts, non-existent resources, duplicate creation. diff --git a/README.md b/README.md index 87e2c098..0610af5b 100644 --- a/README.md +++ b/README.md @@ -10,10 +10,10 @@ Think "Docker for VMs" — one command to create, start, and connect. -- ⚡ **Fast boot times** — VMs boot in 2-4 seconds (loop-mount) or 9-14 seconds (guestfs), so you iterate at container speed with VM-grade isolation +- ⚡ **Fast boot times** — lightweight Firecracker guests keep development feedback fast; the older timing benchmarks are not a v0.3.0 performance guarantee - 🔥 **Powered by Firecracker** — AWS's battle-tested microVM technology, the engine behind Lambda and Fargate - 🛡️ **Secure by default** — hardware-level isolation with KVM -- 📦 **Single binary** — one statically-linked Go binary with no language runtime dependencies. Calls standard Linux utilities (ip, sudo, iptables/nftables, losetup, mount) +- 📦 **Single installed binary** — the root-owned `mvm` executable serves both the unprivileged CLI and its narrowly typed privileged operations; host packages are still needed for VM setup and provisioning - 🖼️ **Image support** — ready-to-use images for Ubuntu, Debian, Arch, Alpine, and more. Import your own base images - 💾 **Volumes & persistence** — create, attach, resize, and detach persistent data disks that survive VM lifecycles - ⚙️ **Custom kernels** — download pre-built Firecracker kernels or build official kernels with custom features (KVM, nftables) @@ -22,7 +22,7 @@ Think "Docker for VMs" — one command to create, start, and connect. - 📋 **Environment as code** — provision full VM topologies from a YAML spec: networks, keys, images, kernels, VMs, and post-boot provisioning via `mvm env apply` - 🔌 **Vsock agent** — run commands inside VMs instantly via a lightweight embedded guest agent (`mvm exec`), no SSH daemon required - 📁 **File transfer** — copy files between host and VM over vsock with `mvm cp`, no guest dependencies needed -- 🧩 **Atomic batch creation** — spin up multiple VMs as a unit with `--count N --atomic`. All succeed or all roll back +- 🧩 **Atomic batch creation** — use `--count N --atomic` to request rollback if creation fails; reported cleanup failures require follow-up - 🐳 **Container-like ergonomics** — `mvm vm create myvm --image ubuntu:24.04 && mvm ssh myvm` works like `docker run -it ubuntu bash`, but with real VM isolation --- @@ -31,6 +31,7 @@ Think "Docker for VMs" — one command to create, start, and connect. - [Prerequisites](#prerequisites) - [Installation](#installation) +- [Uninstall](#uninstall) - [Quick Start](#quick-start) - [Common Tasks](#common-tasks) - [Documentation](#documentation) @@ -42,7 +43,7 @@ Think "Docker for VMs" — one command to create, start, and connect. ## Prerequisites -- **Linux** (x86_64 or aarch64) with KVM access (`/dev/kvm`): +- **Linux** with KVM access (`/dev/kvm`). The v0.3.0 runtime qualification is on x86_64; aarch64 is not yet runtime-qualified: ```bash sudo usermod -aG kvm $USER # Log out and back in @@ -51,62 +52,91 @@ Think "Docker for VMs" — one command to create, start, and connect. Ubuntu/Debian: ```bash - sudo apt-get install -y iproute2 iptables nftables qemu-utils e2fsprogs util-linux procps kmod openssh-client tar sudo passwd fakeroot + sudo apt-get install -y iproute2 qemu-utils e2fsprogs util-linux procps kmod openssh-client tar sudo passwd fakeroot ``` Arch Linux: ```bash - sudo pacman -S --needed iproute2 iptables nftables qemu-img e2fsprogs util-linux procps-ng kmod openssh tar sudo shadow fakeroot + sudo pacman -S --needed iproute2 qemu-img e2fsprogs util-linux procps-ng kmod openssh tar sudo shadow fakeroot ``` Optional (for ISO cloud-init mode only): `cloud-image-utils` (Ubuntu/Debian) or `cloud-utils` (Arch) -- **Root access (one-time):** run `mvm host init` once to create the `mvm` group and a sudoers drop-in; normal `mvm` commands require no `sudo` after that +- **Administrator access for installation and host setup:** install the trusted artifact at `/usr/local/bin/mvm`, then run `host init`; normal commands run without `sudo` after setup. Later system-binary upgrades also require an administrator. +- **Firecracker/Jailer 1.17.0:** the v0.3.0 launch contract requires this exact matching release pair. See the [candidate-bound qualification checkpoint](tests/system/COVERAGE_MATRIX.md#v030-release-security-qualification-overlay) for tested behavior and remaining limits. Managed networking also requires kernel nftables support for both `inet` and `bridge` families; installing the `nft` CLI does not add missing kernel support. +- **Fresh snapshots after upgrade:** create fresh 1.17.0 snapshots and qualify restore with the same pair. Older snapshot state is neither converted nor relabeled. The historical 1.16.2 format is incompatible with 1.16.0/1.16.1 in both directions. See [runtime guidance](docs/RUNTIME.md#vm-launch-and-resource-control). - **Environment variables:** Configure runtime behavior via `MVM_*` variables. See [docs/REFERENCES.md](docs/REFERENCES.md#environment-variables) for the full list. --- ## Installation -### 1. Download prebuilt binary (recommended) +v0.3.0 requires a clean installation and fresh managed state; it does not migrate legacy databases, runtime, +or protected-storage layouts. Before replacing an older installation, stop/remove its VMs and preserve desired +artifacts separately. Review the [administrator installation constraints](docs/REFERENCES.md#mvm-host) before +cleanup: `host clean` and `host reset` are not a supported legacy-state removal procedure. -You may head over to [Releases](https://github.com/AlanD20/mvmctl/releases) page to get the desired package for your distro or you may simply get the static and standalone binary itself: +### 1. Download a published v0.3.0 binary + +v0.3.0 is not released yet. Once published, obtain the `v0.3.0` artifact from [Releases](https://github.com/AlanD20/mvmctl/releases); do not use `releases/latest` while it may still point to an older installation contract. For current development, use the source-build path below in a disposable environment. ```bash -mkdir -p ~/.local/bin -curl -L -o ~/.local/bin/mvm https://github.com/AlanD20/mvmctl/releases/latest/download/mvm -chmod +x ~/.local/bin/mvm -mvm --help +curl -fL -o ./mvm https://github.com/AlanD20/mvmctl/releases/download/v0.3.0/mvm +chmod +x ./mvm +sudo ./mvm host install-system +sudo /usr/local/bin/mvm host init +/usr/local/bin/mvm --help ``` -> Make sure `~/.local/bin` is in your `$PATH`. Most modern Linux distros include it by default. +> `/usr/local/bin/mvm` is the single installed executable. It is root-owned; normal use is unprivileged. Keep a +> downloaded candidate user-owned and use its `host install-system` command for future administrator-approved upgrades. ### 2. Build from source ```bash -git clone https://github.com/AlanD20/mvmctl +git clone --branch 0.3.0-release https://github.com/AlanD20/mvmctl cd mvmctl -./scripts/build.sh release --output ~/.local/bin/mvm -mvm --help +./scripts/build.sh release --version 0.3.0-rc.1 +sudo ./dist/mvm host install-system +sudo /usr/local/bin/mvm host init +/usr/local/bin/mvm --help ``` See [docs/REFERENCES.md](docs/REFERENCES.md) for the complete command reference with all flags, options, and selectors. --- +## Uninstall + +To remove an initialized v0.3.0 installation and all enrolled operators' managed data, preview the administrator +script before confirming deletion: + +```bash +sudo python3 scripts/uninstall.py --dry-run +sudo python3 scripts/uninstall.py +``` + +This removes VMs, artifacts, managed keys and installation state. External asset mirrors, original external SSH +files and shared system packages are preserved. See [uninstall and reinstall](docs/UNINSTALL.md) for custom paths, +confirmation options and cleanup failures. It is not a legacy-state migration tool. + ## Quick Start ```bash -# 1. One-time system setup (interactive, handles sudo) -mvm init +# 1. One-time system setup after installing the root-owned binary +sudo /usr/local/bin/mvm host init + +# 2. Log out and back in, then initialize the operator cache and the exact required binary pair +mvm init --skip-host --non-interactive --binary-version 1.17.0 -# 2. Log out and back in, then download a kernel and image +# 3. Download a kernel and image mvm kernel pull --type firecracker --default mvm image pull ubuntu:24.04 --default -# 3. Create and connect to a VM +# 4. Create and connect to a VM mvm vm create myvm --image ubuntu:24.04 --vcpu 2 --mem 2G --disk-size 20G mvm exec myvm # vsock agent — no SSH needed ``` -That's it. When you're done: `mvm vm rm myvm`. +`--binary-version` is used by `mvm init` only when no local pair is selected. If a different pair is already the +default, run `mvm bin pull 1.17.0 --default` before creating a VM. When you're done: `mvm vm rm myvm`. --- @@ -115,6 +145,7 @@ That's it. When you're done: `mvm vm rm myvm`. | Task | Command | Learn more | |------|---------|-----------| | Create a VM | `mvm vm create myvm --image ubuntu:24.04` | [REFERENCES.md](docs/REFERENCES.md#mvm-vm) | +| Create without an IP network | `mvm vm create offline-worker --no-network` | [REFERENCES.md](docs/REFERENCES.md#mvm-vm) | | Run a command inside a VM | `mvm exec myvm -- ls -la` | [REFERENCES.md](docs/REFERENCES.md#mvm-exec) | | Copy files to/from a VM | `mvm cp ./file.txt myvm:/root/` | [REFERENCES.md](docs/REFERENCES.md#mvm-cp) | | SSH into a VM | `mvm ssh myvm` | [REFERENCES.md](docs/REFERENCES.md#mvm-ssh) | @@ -148,7 +179,7 @@ That's it. When you're done: `mvm vm rm myvm`. |-------|----------| | **Permission denied: /dev/kvm** | Missing: `sudo modprobe kvm && sudo modprobe kvm_intel` (or `kvm_amd`). Unreadable: `sudo usermod -aG kvm $USER`, log out/back in | | **VM won't boot / SSH times out** | Cloud-init takes 30-60s. Watch with `mvm logs myvm --follow` | -| **nocloud-net server failed** | Port range exhausted. Check: `sudo ss -tlnp \| grep -E ':(8[0-9]{3}\|9[0-9]{3})'` | +| **nocloud-net server failed** | NoCloud uses fixed TCP port `8000` on the managed network gateway; check that listener and the network's service grant. | See [docs/TROUBLESHOOTING.md](docs/TROUBLESHOOTING.md) for the full guide. diff --git a/cmd/mvm/main.go b/cmd/mvm/main.go index cc165150..a79f1730 100644 --- a/cmd/mvm/main.go +++ b/cmd/mvm/main.go @@ -4,35 +4,99 @@ import ( "context" "log/slog" "os" - "os/signal" - "syscall" "mvmctl/internal/app" - "mvmctl/internal/cli" "mvmctl/internal/cli/common" + "mvmctl/internal/service/jailer" + "mvmctl/internal/service/privileged" ) +type entrypoints struct { + launchConsole func(context.Context, []string) error + launchChild func(context.Context, []string) error + privileged func(context.Context, []string) error + systemInstall func(context.Context, []string) error + hostAdministrator func(context.Context, []string) (*app.HostAdministratorResult, error) + normal func() int +} + func main() { - ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM) - defer stop() + os.Exit(run(context.Background(), os.Args[1:], entrypoints{ + launchConsole: jailer.RunLaunchConsole, + launchChild: jailer.RunLaunchChild, + privileged: privileged.Run, + systemInstall: app.RunHostInstallSystemBinary, + hostAdministrator: app.RunHostAdministrator, + normal: runCLI, + })) +} - op, cleanup, err := app.Initialize(ctx) - if err != nil { - slog.Error("initialization failed", "error", err) - os.Exit(1) +func run(ctx context.Context, args []string, entries entrypoints) int { + if jailer.IsLaunchConsoleInvocation(args) { + // The console relay must never enter ordinary CLI initialization, even + // when its inherited setup channel rejects an unauthenticated invocation. + if err := entries.launchConsole(ctx, args); err != nil { + slog.Error("launch console failed", "error", err) + return 1 + } + return 0 } - if cleanup != nil { - defer cleanup() + if jailer.IsLaunchChildInvocation(args) { + // The child owns a locked namespace thread. Every result returns directly + // to main's os.Exit; it must never enter ordinary application initialization. + if err := entries.launchChild(ctx, args); err != nil { + slog.Error("launch child failed", "error", err) + return 1 + } + return 0 + } + if privileged.IsInvocation(args) { + if err := entries.privileged(ctx, args); err != nil { + slog.Error("privileged request failed", "error", err) + return 1 + } + return 0 } + if app.IsHostInstallSystemBinaryInvocation(args) { + if err := entries.systemInstall(ctx, args); err != nil { + slog.Error("system binary installation failed", "error", err) + return 1 + } + return 0 + } + if app.IsHostAdministratorInvocation(args) { + return runHostAdministrator(ctx, args, entries.hostAdministrator) + } + return entries.normal() +} - // Execute CLI - rootCmd := cli.NewRootCmd(op) - if err := rootCmd.ExecuteContext(ctx); err != nil { - // Delegate ALL error handling to the single shared handler in helpers.go. - // HandleErrors returns nil for non-fatal errors (BrokenPipe), non-nil for - // everything else. Exit with code 1 so Go respects the error exit convention. +func runHostAdministrator( + ctx context.Context, + args []string, + entry func(context.Context, []string) (*app.HostAdministratorResult, error), +) int { + common.BeginOutput(args) + result, err := entry(ctx, args) + if common.StructuredOutput() { + if result != nil && err != nil { + common.RecordPartialResult(result) + } else { + common.RecordResult(result) + } + if common.FinishOutput(os.Stdout, err) != nil { + return 1 + } + return 0 + } + if err != nil { if common.HandleErrors(func() error { return err })() != nil { - os.Exit(1) + return 1 } + return 0 + } + common.Cli.Success("host administration completed") + if result != nil && result.UserAddedToGroup { + common.Cli.Info("log out and back in to activate mvm group membership") } + return 0 } diff --git a/cmd/mvm/main_test.go b/cmd/mvm/main_test.go new file mode 100644 index 00000000..dec4ab3d --- /dev/null +++ b/cmd/mvm/main_test.go @@ -0,0 +1,130 @@ +package main + +import ( + "context" + "errors" + "strings" + "testing" + + "github.com/stretchr/testify/assert" + + "mvmctl/internal/service/privileged" +) + +func TestRun_RoutesPrivilegedInvocationBeforeNormalInitialization(t *testing.T) { + sentinel := errors.New("privileged sentinel") + privilegedCalls := 0 + normalCalls := 0 + + exitCode := run(context.Background(), []string{privileged.Marker, "unknown"}, entrypoints{ + privileged: func(context.Context, []string) error { + privilegedCalls++ + return sentinel + }, + normal: func() int { + normalCalls++ + return 0 + }, + }) + + assert.Equal(t, 1, exitCode) + assert.Equal(t, 1, privilegedCalls) + assert.Zero(t, normalCalls) +} + +// Rationale: Unsupported protocol versions must remain inside the reserved +// boundary instead of falling through to Cobra or user-state initialization. +func TestRun_RoutesUnsupportedPrivilegedVersionToFailClosedEntry(t *testing.T) { + privilegedCalls := 0 + normalCalls := 0 + + exitCode := run(context.Background(), []string{"__mvm_privileged_v999", "unknown"}, entrypoints{ + privileged: func(context.Context, []string) error { + privilegedCalls++ + return nil + }, + normal: func() int { + normalCalls++ + return 0 + }, + }) + + assert.Zero(t, exitCode) + assert.Equal(t, 1, privilegedCalls) + assert.Zero(t, normalCalls) +} + +func TestRun_RoutesOrdinaryArgumentsToNormalCLI(t *testing.T) { + privilegedCalls := 0 + normalCalls := 0 + + exitCode := run(context.Background(), []string{"vm", "list"}, entrypoints{ + privileged: func(context.Context, []string) error { + privilegedCalls++ + return nil + }, + normal: func() int { + normalCalls++ + return 7 + }, + }) + + assert.Equal(t, 7, exitCode) + assert.Zero(t, privilegedCalls) + assert.Equal(t, 1, normalCalls) +} + +func TestRun_RoutesSystemInstallBeforeNormalInitialization(t *testing.T) { + sentinel := errors.New("system install sentinel") + installCalls := 0 + normalCalls := 0 + + exitCode := run(context.Background(), []string{"host", "install-system"}, entrypoints{ + systemInstall: func(_ context.Context, args []string) error { + installCalls++ + assert.Equal(t, []string{"host", "install-system"}, args) + return sentinel + }, + normal: func() int { + normalCalls++ + return 0 + }, + }) + + assert.Equal(t, 1, exitCode) + assert.Equal(t, 1, installCalls) + assert.Zero(t, normalCalls, "app.Initialize and Cobra must be unreachable") +} + +func TestRun_RoutesMalformedSystemInstallWithoutNormalFallthrough(t *testing.T) { + tests := [][]string{ + {"host", "install-system", "unexpected"}, + {"--debug", "host", "install-system", "--force"}, + {"--debug", "false", "host", "install-system"}, + {"host", "--unknown", "value", "install-system"}, + {"host", "--", "install-system"}, + {"--debug", "false", "false", "host", "install-system"}, + } + + for _, args := range tests { + t.Run(strings.Join(args, "_"), func(t *testing.T) { + installCalls := 0 + normalCalls := 0 + exitCode := run(context.Background(), args, entrypoints{ + systemInstall: func(_ context.Context, got []string) error { + installCalls++ + assert.Equal(t, args, got) + return errors.New("malformed") + }, + normal: func() int { + normalCalls++ + return 0 + }, + }) + + assert.Equal(t, 1, exitCode) + assert.Equal(t, 1, installCalls) + assert.Zero(t, normalCalls) + }) + } +} diff --git a/cmd/mvm/normal.go b/cmd/mvm/normal.go new file mode 100644 index 00000000..de1c1028 --- /dev/null +++ b/cmd/mvm/normal.go @@ -0,0 +1,50 @@ +package main + +import ( + "context" + "log/slog" + "os" + "os/signal" + "syscall" + + "mvmctl/internal/app" + "mvmctl/internal/cli" + "mvmctl/internal/cli/common" +) + +func runCLI() int { + common.BeginOutput(os.Args[1:]) + ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM) + defer stop() + + op, cleanup, err := app.Initialize(ctx) + if err != nil { + if common.StructuredOutput() { + if outputErr := common.FinishOutput(os.Stdout, err); outputErr != nil { + return 1 + } + } else { + slog.Error("initialization failed", "error", err) + } + return 1 + } + if cleanup != nil { + defer cleanup() + } + + rootCmd := cli.NewRootCmd(op) + err = rootCmd.ExecuteContext(ctx) + if common.StructuredOutput() { + if common.FinishOutput(os.Stdout, err) != nil { + return 1 + } + return 0 + } + if err != nil { + // Delegate all public CLI error rendering to the shared handler. + if common.HandleErrors(func() error { return err })() != nil { + return 1 + } + } + return 0 +} diff --git a/docs/DEPENDENCIES.md b/docs/DEPENDENCIES.md index e0b0188f..88e070bc 100644 --- a/docs/DEPENDENCIES.md +++ b/docs/DEPENDENCIES.md @@ -5,7 +5,7 @@ All external binary and system-level dependencies required for mvmctl to functio ## Table of Contents - [A. Required (Every Install)](#a-required-every-install) -- [B. Firewall Backend (Choose One)](#b-firewall-backend-choose-one) +- [B. Firewall Kernel Support](#b-firewall-kernel-support) - [C. Image & Storage](#c-image-storage) - [D. Cloud-Init](#d-cloud-init) - [E. Image Provisioning Backend (Choose One)](#e-image-provisioning-backend-choose-one) @@ -21,7 +21,7 @@ All external binary and system-level dependencies required for mvmctl to functio ## A. Required (Every Install) -These binaries must be present on the host for basic mvmctl operation. The `mvm init` wizard checks for them automatically. +These binaries support administrator setup and selected VM workflows; not every ordinary command invokes every tool. | Binary | Purpose | Debian/Ubuntu | Arch | |--------|---------|---------------|------| @@ -30,33 +30,23 @@ These binaries must be present on the host for basic mvmctl operation. The `mvm | `usermod` | Adding users to the `mvm` group | `passwd` | `shadow` | | `groupdel` | Removing the `mvm` group on `mvm host reset` | `passwd` | `shadow` | | `visudo` | Validating sudoers drop-in files | `sudo` | `sudo` | -| `ip` | Bridge and TAP interface management | `iproute2` | `iproute2` | | `modprobe` | Loading KVM and networking kernel modules | `kmod` | `kmod` | | `lsmod` | Checking KVM module status | `kmod` | `kmod` | | `sysctl` | Enabling IP forwarding | `procps` | `procps-ng` | | `ssh-keygen` | Generating SSH keypairs for microVMs | `openssh-client` | `openssh` | | `tar` | Extracting rootfs from tarballs | `tar` | `tar` | -| `iptables` or `nft` | Firewall rule management | `iptables` or `nftables` | `iptables` or `nftables` | **Go toolchain:** The `mvm` binary is a single compiled Go binary. No runtime dependencies beyond the system packages listed. --- -## B. Firewall Backend (Choose One) +## B. Firewall Kernel Support -mvmctl supports two firewall backends for NAT and forwarding rules. **nftables is the default** (`settings.firewall_backend: "nftables"`). Only one backend needs to be installed. - -| Backend | Binary | Debian/Ubuntu | Arch | -|---------|--------|---------------|------| -| **nftables** (default) | `nft` | `nftables` | `nftables` | -| **iptables** | `iptables` | `iptables` | `iptables` | - -**Switch backends:** -```bash -mvm config set settings firewall_backend nftables # default -mvm config set settings firewall_backend iptables -mvm network sync # reload rules -``` +v0.3.0 has one typed nftables implementation using the kernel netlink API. The `nft` and `iptables` command-line +packages are not firewall runtime dependencies, and the old selectable backend setting is retired. Managed networks +need kernel support for the required `inet` and `bridge` nftables families, connection tracking, and NAT expressions. +Host readiness probes the actual kernel behavior in a disposable network namespace and fails closed when unavailable; +installing an `nft` CLI alone cannot supply missing kernel support. --- @@ -68,6 +58,7 @@ Required for pulling, importing, and converting VM images. |--------|---------|---------------|------| | `qemu-img` | Image conversion and resize | `qemu-utils` | `qemu-img` | | `mkfs.ext4` | Formatting extracted rootfs images | `e2fsprogs` | `e2fsprogs` | +| `e2fsck` | Journal-only preparation of disposable ext4 VM-import inspection copies with LoopMount | `e2fsprogs` | `e2fsprogs` | | `fakeroot` | Preserving tarball ownership during tar-rootfs extraction | `fakeroot` | `fakeroot` | | `blkid` | Detecting root partitions and UUIDs | `util-linux` | `util-linux` | | `sfdisk` | Partition table manipulation | `util-linux` | `util-linux` | @@ -97,7 +88,9 @@ Both `mvm vm create` (rootfs provisioning) and `mvm image pull`/`mvm image impor ### E1. Loop-Mount (Default) -The `mvm run provision` subcommand is a built-in CLI subcommand compiled into the `mvm` binary. It uses system tools directly — no extra runtime dependencies beyond system packages. +Loop-mount provisioning uses the closed typed privileged receiver compiled into the same `mvm` binary. The former +public `mvm run provision` command and raw-sudo grant are removed. The receiver still needs the host filesystem tools +below; it does not accept arbitrary host commands or paths from the operator. | Binary | Purpose | Debian/Ubuntu | Arch | |--------|---------|---------------|------| @@ -106,7 +99,7 @@ The `mvm run provision` subcommand is a built-in CLI subcommand compiled into th | `blockdev` | Querying partition/device size | `util-linux` | `util-linux` | | `mount` | Mounting the root partition | `util-linux` | `util-linux` | | `umount` | Unmounting the root partition | `util-linux` | `util-linux` | -| `e2fsck` | Filesystem check before ext4 resize | `e2fsprogs` | `e2fsprogs` | +| `e2fsck` | Filesystem check before ext4 resize; journal-only replay on disposable VM-import inspection copies | `e2fsprogs` | `e2fsprogs` | | `resize2fs` | Growing and shrinking ext4 filesystems | `e2fsprogs` | `e2fsprogs` | | `tune2fs` | Reading ext4 block count for shrink calculation | `e2fsprogs` | `e2fsprogs` | | `btrfs` | Growing and shrinking btrfs filesystems | `btrfs-progs` | `btrfs-progs` | @@ -115,6 +108,12 @@ The `mvm run provision` subcommand is a built-in CLI subcommand compiled into th > Most binaries (`util-linux`, `e2fsprogs`) are already required by the image pipeline (§C). Only `btrfs-progs` is unique to this path (and only needed for btrfs images). +LoopMount import from a VM invokes the fixed `/usr/sbin/e2fsck -p -E journal_only` as the ordinary operator, +against a retained descriptor for a new inspection copy. It does not run filesystem repair on the VM's protected +disk or grant the operator a new sudoers command. The `e2fsprogs` installation must provide that executable path; +missing or unsuccessful journal preparation fails import rather than skipping recovery or using a writable root +inspection mount. Ordinary file imports and the GuestFS path do not select this VM-only preparation. + ### E2. libguestfs (Alternative) libguestfs provides filesystem-agnostic rootfs access via a QEMU appliance. Enable with: @@ -129,7 +128,7 @@ mvm config set settings guestfs_enabled true | RHEL/Fedora | `sudo dnf install libguestfs libguestfs-tools supermin` | | Arch | `sudo pacman -S libguestfs supermin` | -The codebase uses the `guestfish` CLI tool as a subprocess — no cgo or Go bindings required. `mvm init` checks for the `guestfish` binary and configures the sudoers entry for it automatically. +The optional backend uses the `guestfish` CLI tool as a subprocess; it is not a separate sudoers grant. --- @@ -174,35 +173,52 @@ Only needed for `mvm kernel pull --type official --clean-build` (building the of - **Virtualization**: VT-x (Intel) or AMD-V must be enabled in BIOS/UEFI. - **Permissions**: The user must be in the `mvm` group (created by `mvm host init`). +### Managed Cache Storage + +For v0.3 privileged operations, `MVM_CACHE_DIR` must reside on ext2/ext3/ext4, XFS, Btrfs, F2FS, bcachefs, tmpfs, or +ZFS. FUSE, remote, overlay/stacked, and automount cache roots are unsupported. Custom local paths remain supported and +are pinned independently of unrelated ancestor filesystems. + +### Firecracker Jailer + +`mvm bin pull` resolves one release and invokes the typed trusted-release authority. Root either consumes the exact +optional mirror archive or fetches the fixed upstream release, verifies it, and atomically publishes the matching +Firecracker/Jailer pair. Users do not invoke an installation service directly; source-built pairs are unsupported. + --- ## H. Command Dependency Map | Command | External Binaries Invoked | |---------|--------------------------| -| `mvm host init` | `sudo`, `groupadd`, `usermod`, `visudo`, `sysctl`, `lsmod`, `iptables`/`nft`, `modprobe` | -| `mvm host clean` | `sudo`, `ip`, `iptables`/`nft` | +| `mvm host install-system` | No external command; exact early root bootstrap uses descriptor-relative filesystem operations | +| `mvm host init` | `sudo` for sudo-invoker mode, plus administrator tools such as `groupadd`, `usermod`, `visudo`, `sysctl`, `lsmod`, `modprobe` | +| `mvm host clean` | Typed privileged actions; no `ip`, `iptables`, or `nft` firewall subprocess | | `mvm host reset` | `sudo`, `groupdel`, `sysctl` | -| `mvm network create` | `ip`, `iptables`/`nft` | -| `mvm network rm` / `sync` | `ip`, `iptables`/`nft` | +| `mvm network create` | Typed root netlink and nftables-kernel operations; no `ip`, `iptables`, or `nft` CLI | +| `mvm network rm` / `sync` | Typed root netlink and nftables-kernel operations; no firewall CLI | +| `mvm policy traffic/exec` | Typed root policy actions; no firewall CLI | | `mvm image pull` | `qemu-img` (may trigger conversion), `fakeroot` (tar-rootfs) | -| `mvm image import` | `qemu-img`, `sfdisk`, `blkid`, `mount`, `umount`, `tar`, `fakeroot` (tar-rootfs), `truncate`, `mkfs.ext4`, `unsquashfs`, `dumpe2fs`, `du`, `dd` | +| `mvm image import` | `qemu-img`, `sfdisk`, `blkid`, `mount`, `umount`, `tar`, `fakeroot` (tar-rootfs), `truncate`, `mkfs.ext4`, `unsquashfs`, `dumpe2fs`, `du`, `dd`; `/usr/sbin/e2fsck` for ext4 VM import with LoopMount | | `mvm kernel pull --type official` | `make`, `gcc`, `ld`, `flex`, `bison`, `bc`, `pahole`, `git`, `curl`, `pkg-config` | | `mvm kernel pull --type firecracker` | Download only (no build tools) | | `mvm key create` | `ssh-keygen` | | `mvm key import/ls/rm/inspect/export/default` | Internal only | -| `mvm bin pull/ls/rm/default` | Internal only (downloads from GitHub API) | -| `mvm vm create` | `firecracker`, `ip`, `iptables`/`nft`, `mvm run provision`, `losetup`, `blkid`, `blockdev`, `mount`, `umount`, `e2fsck`, `resize2fs`, `tune2fs`, `fstrim`, `chroot` (+ `btrfs` for btrfs images) | -| `mvm vm start/stop/reboot/pause/resume` | `firecracker`, `ip`, `iptables`/`nft` | -| `mvm vm rm` | `firecracker`, `ip`, `iptables`/`nft` | -| `mvm snapshot create/restore` | Internal (Firecracker API via Unix socket) | -| `mvm volume attach/detach` | `firecracker` | -| `mvm volume create/rm/resize` | `qemu-img` | -| `mvm volume ls/inspect` | Internal only | -| `mvm kernel import/ls/rm/default/inspect` | Internal only | -| `mvm cache init/prune/clean` | Internal (filesystem + DB operations) | +| `mvm bin pull/rm` | Typed privileged release actions; no public service subcommand | +| `mvm bin ls/default` | Internal architecture-scoped SQLite display projection only | +| `mvm vm create` | Trusted `jailer` + `firecracker`; provisioning may use `losetup`, `blkid`, `blockdev`, `mount`, `umount`, `e2fsck`, `resize2fs`, `tune2fs`, `fstrim`, `chroot` (+ `btrfs` for btrfs images) | +| `mvm vm start/stop/reboot/pause/resume` | Trusted `jailer` + `firecracker`; protected API access is root-brokered | +| `mvm vm rm` | Typed privileged VM cleanup, with no standalone Jailer cleanup command | +| `mvm snapshot create/restore` | Typed root snapshot actions and root-brokered Firecracker API access | +| `mvm volume attach/detach` | Typed root volume/mount operations and root-brokered Firecracker API access; live attach also checks authenticated guest readiness | +| `mvm volume create/rm/resize` | Internal raw-file allocation and typed privileged removal/growth; no `qemu-img` | +| `mvm volume ls` | Internal SQLite display projection | +| `mvm volume inspect` | Typed root observation of authoritative volume backing | +| `mvm kernel import/rm` | Typed root protected-kernel publication/retirement and local projection updates | +| `mvm kernel ls/default/inspect` | Internal local projection and file inspection | +| `mvm cache init` | Internal filesystem + DB operations | +| `mvm cache prune/clean` | Typed resource cleanup and protected-cache retirement, plus local filesystem/DB operations | | `mvm config get/set/reset/list` | Internal (DB operations) | | `mvm logs` | Internal (file read) | | `mvm console` | Internal (Unix socket PTY relay) | | `mvm ssh` | User's SSH client (`ssh` binary) | -| `mvm init` | `sudo`, `groupadd`, `usermod`, `visudo` | diff --git a/docs/ENV_SPEC_REFERENCE.md b/docs/ENV_SPEC_REFERENCE.md index cec7a80b..0addb548 100644 --- a/docs/ENV_SPEC_REFERENCE.md +++ b/docs/ENV_SPEC_REFERENCE.md @@ -5,7 +5,10 @@ Complete reference for the `mvm env` workflow engine YAML spec format. ## Table of Contents - [Overview](#overview) +- [Compatibility](#compatibility) +- [Top-Level Fields](#top-level-fields) - [Syntax](#syntax) + - [Common fields](#common-fields) - [Step Types](#step-types) - [network](#network) - [key](#key) @@ -14,11 +17,17 @@ Complete reference for the `mvm env` workflow engine YAML spec format. - [kernel](#kernel) - [binary](#binary) - [vm](#vm) + - [volume](#volume) + - [traffic_policy](#traffic_policy) + - [exec_policy](#exec_policy) + - [ready](#ready) - [exec](#exec) - [ssh](#ssh) - [copy](#copy) - [References](#references) - [Destroy Behavior](#destroy-behavior) + - [Explicit removals (`removes`)](#explicit-removals-removes) + - [Ephemeral specs (`ephemeral: true`)](#ephemeral-specs-ephemeral-true) - [Full Example](#full-example) - [State File](#state-file) @@ -26,7 +35,7 @@ Complete reference for the `mvm env` workflow engine YAML spec format. ## Overview -An env spec is a YAML file that describes an environment — networks, SSH keys, images, kernels, binaries, VMs, and the commands that configure them. Running `mvm env apply spec.yaml` provisions everything in dependency order. +An env spec is a YAML file that describes an environment — networks, SSH keys, images, kernels, binaries, VMs, and the commands that configure them. Running `mvm env apply spec.yaml` provisions everything in dependency order. Section examples below are fragments to combine under one `version: "1"` header; the [full example](#full-example) includes its prerequisites. The format uses **typed top-level sections** (one per step type) with **map keys as step names**: @@ -39,7 +48,7 @@ network: subnet: "172.27.0.0/24" ``` -The map key (`my-net`) IS the step name — used in `depends_on`, `removes`, and the state file. An optional `name` field inside the params overrides the resource name (e.g., the bridge name for a network, the VM name, the key name). +The map key (`my-net`) is the step name used in references, `depends_on`, `removes`, and saved state. A resource `name` override does not change that step name. Cross-step references use the **`@type:name`** format: @@ -55,12 +64,26 @@ The `@` sigil makes references visually distinct from literal values. --- +## Compatibility + +Only spec version `"1"` is supported. The available sections are `network`, `key`, `image`, `image_import`, `kernel`, `binary`, `vm`, `volume`, `traffic_policy`, `exec_policy`, `ready`, `exec`, `ssh`, and `copy`. Snapshot creation and restore are separate CLI operations. + +VM steps use the current VM creation rules, including networkless placement, raw data volumes, and the Firecracker `1.17.0` launch requirement. `exec.allow_vm_exec` opts a command into VM-to-VM execution; a directional exec policy is also required. Neither a traffic grant nor an exec grant enables that option implicitly. + +Validation rejects unknown sections and fields, malformed or missing references, missing dependency paths, self-dependencies, cycles, unsupported values, and VM `count` values other than `1` before applying steps. Nested `env` and `cpu_config` maps retain their own keys. + +Reapply ensures recorded resource identities. It does not silently replace missing resources or mutate changed owned declarations: resolve the reported conflict or explicitly destroy and recreate the affected environment, preserving needed data first. Exec, SSH, copy, and readiness steps run again. Write commands and readiness checks so repeated execution is safe. `mvm env diff spec.yaml` compares the spec with saved state; it does not inspect live configuration drift. + +`mvm env apply` and `mvm env destroy` look for `mvmctl.yaml`, then `mvmctl.yml`, when no argument is given. Apply also accepts an HTTP or HTTPS spec URL. `mvm env diff` requires a spec path or workflow ID. The aliases `up` and `down` mean `apply` and `destroy`. + +--- + ## Top-Level Fields | Field | Type | Default | Description | |-------|------|---------|-------------| | `version` | `string` | — | **Required.** Currently only `"1"`. | -| `ephemeral` | `bool` | `false` | Auto-run `env destroy` after apply completes (success or failure), removing all resources and the state record. | +| `ephemeral` | `bool` | `false` | Attempt teardown after pipeline success or failure. Shared assets and pre-existing resources are preserved; cleanup errors can leave resources or state behind. | --- @@ -78,17 +101,22 @@ key: algorithm: ... ``` -**Every step type section is a map from step name to its params.** No list dashes. An optional `name` field inside the params overrides the resource name (see Common fields below). +Each section is a map from step name to parameters, not a YAML list. The syntax block above is a template; the [full example](#full-example) is a complete spec. ### Common fields -Every step supports these fields: +Every step supports dependency and removal lists: | Field | Type | Description | |-------|------|-------------| -| `name` | `string` | Resource name override. Defaults to the step name (map key). | | `depends_on` | `[]string` | List of `"@type:name"` dependencies. | -| `removes` | `[]string` | List of `"@type:name"` resources to destroy after this step succeeds. | +| `removes` | `[]string` | Resources to remove after the entire pipeline succeeds. | + +`network`, `key`, `vm`, `volume`, and `image_import` accept a `name` override, defaulting to the step name. `image.name` is an optional image name; omission uses image acquisition naming. Other steps do not use `name` as a resource-name override. + +`network`, `key`, and `vm` also accept `existing: ` to select a pre-existing resource without taking +ownership. Do not combine `existing` with creation fields. On reapply the saved identity is required; there is no +fallback to a replacement with the same name. The creation-field requirements below apply when `existing` is absent. --- @@ -111,7 +139,7 @@ vm: The `@` prefix distinguishes references from literal string values. The type prefix (`network`, `key`, `image`, etc.) disambiguates steps with the same name under different types. -**Bare names:** Reference fields like `network`, `key`, `image`, `kernel`, `binary`, and `target` accept bare step names (without `@` or `type:`) for convenience. However, `depends_on` and `removes` entries must use the full `@type:name` format because the engine matches them against step identifiers in the DAG. +References do not create dependencies automatically. Each referenced step must be reachable through `depends_on`, directly or transitively; independent steps can run concurrently. References resolve recorded resource identities, including when the resource has a different `name`. Use `@key:name` in `ssh_keys` and `@volume:name` in `volumes`, or literal names/IDs for existing resources. Copy endpoints can use `@vm:name:/path` or a literal `vm-name:/path`. Commands, local paths, environment values, and other literal strings keep their `@` characters. --- @@ -123,12 +151,13 @@ Create a network for VMs to connect to. | Field | Type | Required | Default | Description | |-------|------|----------|---------|-------------| -| `subnet` | `string` | **Yes** | — | CIDR notation, e.g. `"172.27.0.0/24"`. | +| `subnet` | `string` | **Yes** | — | Canonical masked IPv4 CIDR wholly inside RFC1918 space, prefix length at most `/30`; e.g. `"172.27.0.0/24"`. | | `nat_enabled` | `bool` | No | `true` | Enable NAT for internet access. | -| `ipv4_gateway` | `string` | No | auto-computed | Gateway IP. | -| `nat_gateways` | `[]string` | No | auto-detected | Host interfaces for NAT. | +| `ipv4_gateway` | `string` | No | First usable address | Canonical usable IPv4 address inside the subnet. | | `default` | `bool` | No | `false` | Set as default network. | +Omit `nat_gateways`: it is rejected. The privileged network service selects the NAT uplink. + **Example:** ```yaml network: @@ -142,7 +171,7 @@ network: ### `key` -Generate or import an SSH key pair. +Generate an SSH key pair. | Field | Type | Required | Default | Description | |-------|------|----------|---------|-------------| @@ -197,7 +226,8 @@ Import a local image file or copy a VM's rootfs. | `format` | `string` | No | `""` (auto) | Format override: `raw`, `qcow2`, `tar`. | | `version` | `string` | No | `""` | Version tag for the imported image. | | `force` | `bool` | No | `false` | Overwrite existing image. | -| `default` | `bool` | No | `false` | Set as default image. | +| `set_default` | `bool` | No | `false` | Set as default image. | +| `partition` | `int` | No | `0` (auto) | Non-negative partition index. | | `skip_optimization` | `bool` | No | `false` | Skip filesystem optimization. | | `disabled_detectors` | `[]string` | No | `[]` | Disable specific detectors. | @@ -206,6 +236,7 @@ Import a local image file or copy a VM's rootfs. image_import: capture-base: source: "@vm:builder" + depends_on: ["@vm:builder"] removes: - "@vm:builder" ``` @@ -219,8 +250,8 @@ Download or build a kernel. | Field | Type | Required | Default | Description | |-------|------|----------|---------|-------------| | `type` | `string` | **Yes** | — | Kernel type: `firecracker` (pre-built) or `official` (build from source). | -| `version` | `string` | No | `""` (latest) | Version tag. | -| `jobs` | `int` | No | CPU count | Build parallelism (official only). | +| `version` | `string` | No | Configured kernel version | Official build version. Firecracker prebuilt selection ignores this field. | +| `jobs` | `int` | No | Configured build jobs, then CPU count | Build parallelism (official only). | | `keep_build_dir` | `bool` | No | `false` | Keep build directory. | | `clean_build` | `bool` | No | `false` | Force clean build. | | `kernel_config` | `string` | No | `""` | Path to custom kernel config file. | @@ -243,8 +274,7 @@ Download Firecracker binaries. | Field | Type | Required | Default | Description | |-------|------|----------|---------|-------------| | `type` | `string` | No | `"firecracker"` | Binary type. | -| `version` | `string` | **Yes** | — | Version tag, e.g. `"1.16.0"`. | -| `git_ref` | `string` | No | `""` | Build from git ref instead of downloading. | +| `version` | `string` | No | Latest | Download version; use `"1.17.0"` for VM launch. | | `default` | `bool` | No | `false` | Set as default binary. | | `force` | `bool` | No | `false` | Force re-download. | @@ -252,7 +282,7 @@ Download Firecracker binaries. ```yaml binary: fc-bin: - version: "1.16.0" + version: "1.17.0" default: true ``` @@ -266,31 +296,41 @@ Create a virtual machine. |-------|------|----------|---------|-------------| | `network` | `string` | No | Default network | Network step reference. | | `key` | `string` | No | Default key | SSH key step reference (shorthand for `ssh_keys`). | -| `ssh_keys` | `[]string` | No | `[]` | List of SSH key step references. | +| `ssh_keys` | `[]string` | No | Default keys | `@key:name` references or existing SSH key names/IDs. | | `image` | `string` | No | Default image | Image step reference. | | `kernel` | `string` | No | Default kernel | Kernel step reference. | | `binary` | `string` | No | Default binary | Binary step reference. | | `vcpu` | `int` | No | Config default | vCPU count. Range: 1-32. | | `mem` | `string` | No | Config default | Memory, e.g. `"512M"`, `"1G"`, or bare MiB int. | -| `disk_size` | `string` | No | Config default | Disk size, e.g. `"10G"`, `"512M"`. | +| `disk_size` | `string` | No | Image minimum rootfs size | Disk size, e.g. `"10G"`, `"512M"`. | | `user` | `string` | No | Config default | SSH user. | | `pci_enabled` | `bool` | No | Config default | Enable PCI. | -| `nested_virt` | `bool` | No | Config default | Enable nested virt (requires PCI). | +| `nested_virt` | `bool` | No | Config default | Enable nested virtualization. | | `cpu_template` | `string` | No | `""` | Path to CPU template JSON. | +| `cpu_config` | `map` | No | Absent | Inline CPU configuration; mutually exclusive with `cpu_template`. | | `console_enable` | `bool` | No | Config default | Enable serial console. | | `logging_enable` | `bool` | No | Config default | Enable logging. | | `metrics_enable` | `bool` | No | Config default | Enable metrics. | | `guest_ip` | `string` | No | `""` | Request specific guest IP. | | `guest_mac` | `string` | No | `""` | Request specific MAC. | | `boot_args` | `string` | No | Config default | Custom kernel boot args. | -| `volumes` | `[]string` | No | `[]` | Volume step references to attach. | -| `count` | `int` | No | `1` | Batch count. | -| `atomic` | `bool` | No | `false` | Atomic batch (all or nothing). | +| `lsm_flags` | `string` | No | Config default | Kernel LSM arguments. | +| `no_network` | `bool` | No | `false` | Create without a network interface. Conflicts with `network`, `guest_ip`, `guest_mac`, network boot arguments, and NET cloud-init. | +| `skip_ci_network_config` | `bool` | No | `false` | Skip generated cloud-init network configuration. | +| `custom_cloud_init_config` | `string` | No | Absent | Path to custom cloud-init user data. | +| `cloud_init_mode` | `string` | No | `off` | One of `off`, `inject`, `net`, `iso`. | +| `cloud_init_iso_path` | `string` | No | Absent | Existing cloud-init ISO path; also set `cloud_init_mode: iso`. | +| `keep_cloud_init_iso` | `bool` | No | `false` | Keep a generated cloud-init ISO. | +| `volumes` | `[]string` | No | `[]` | `@volume:name` references or existing raw data-volume names/IDs to attach. | +| `count` | `int` | No | `1` | Only `1` is accepted. Declare a separate named step for each VM. | +| `atomic` | `bool` | No | `false` | Batch rollback option; use separate VM steps instead of batches in specs. | | `skip_cleanup` | `bool` | No | `false` | Skip cleanup on failure. | | `skip_deblob` | `bool` | No | `false` | Skip image deblobbing. | | `vsock_port` | `int` | No | Config default | Vsock port. Default: `1024`. | | `writeback` | `bool` | No | Config default | Writeback cache mode. | +Omit `nocloud_net_port`: any explicit value is rejected. NET cloud-init uses reserved gateway TCP port `8000`. QCOW2 remains supported for image import; the raw-only restriction applies to attached data volumes. + **Example:** ```yaml vm: @@ -313,6 +353,122 @@ vm: --- +### `volume` + +Create a raw data volume or select an existing volume without taking ownership. Volumes are retained by default, including during ephemeral cleanup, and retained owned volumes keep their state entry. Use `retain: false` at creation for disposable workflow data. Existing volumes are never deleted by the workflow; combining `existing` with `retain: false` is rejected. + +To retire retained data after teardown, preserve anything needed, remove the exact saved volume with +`mvm volume rm `, then rerun `mvm env destroy ` to clear its absent state entry. + +| Field | Type | Default | Description | +|-------|------|---------|-------------| +| `name` | `string` | Step name | Name for a new volume. | +| `size` | `string` | — | Required when creating, e.g. `"10G"`. | +| `read_only` | `bool` | API default | Create a read-only volume. | +| `shareable` | `bool` | API default | Permit sharing subject to normal volume rules. | +| `writeback` | `bool` | API default | Writeback cache mode. | +| `existing` | `string` | Absent | Select an existing volume by name or ID. Do not combine with creation options. | +| `retain` | `bool` | `true` | Preserve workflow-created data on destroy; `false` explicitly permits its deletion. | + +```yaml +volume: + data: + size: "10G" + retain: true +vm: + database: + volumes: ["@volume:data"] + depends_on: ["@volume:data"] +``` + +This fragment uses configured VM defaults. It creates an unformatted raw disk; format and mount it through guest provisioning as needed. There is no QCOW2 data-volume mode. + +### `traffic_policy` + +Allow TCP or UDP traffic to a destination VM or the source network's host gateway. Same-network VM traffic also requires grants. A policy does not grant command execution. + +| Field | Type | Description | +|-------|------|-------------| +| `source_network` | `string` | Source network selector or `@network:name`; mutually exclusive with `source_vm`. | +| `source_vm` | `string` | Source VM selector or `@vm:name`; mutually exclusive with `source_network`. | +| `destination_vm` | `string` | Destination VM selector or `@vm:name`; mutually exclusive with `destination_host: true`. | +| `destination_host` | `bool` | Set `true` for the source managed network's host gateway, not the host's LAN address or localhost. | +| `protocol` | `string` | `tcp` or `udp`. | +| `port` | `string` or `int` | Destination port or inclusive range, e.g. `5432` or `"8000-8080"`; bounds 1–65535. | +| `existing` | `string` | Select an existing policy by exact ID, verified against the declared source, destination, protocol, and port/range. | + +Both creation and existing selection require exactly one source, a destination, protocol, and port/range. With `existing`, these fields must match the selected grant. Existing selection preserves the policy on destroy. + +Operators may select any host port, but must explicitly declare its protocol and port/range. The host service must +listen on the gateway or a suitable wildcard address. Host-to-VM access remains outside policy management. + +```yaml +traffic_policy: + database-access: + source_vm: "@vm:app" + destination_vm: "@vm:database" + protocol: tcp + port: 5432 + depends_on: ["@vm:app", "@vm:database"] + host-database-access: + source_vm: "@vm:app" + destination_host: true + protocol: tcp + port: 5432 + depends_on: ["@vm:app"] +``` + +### `exec_policy` + +Create a directional VM-to-VM execution grant for one non-root destination user. This does not create that user, enable `exec.allow_vm_exec`, or grant host access. With `existing`, supply the matching source, destination, and target user to verify and preserve the selected policy. + +| Field | Type | Description | +|-------|------|-------------| +| `source_vm` | `string` | Source VM selector or `@vm:name`. | +| `destination_vm` | `string` | Destination VM selector or `@vm:name`. | +| `target_user` | `string` | Required non-root guest user for creation or existing selection. | +| `existing` | `string` | Select an existing policy by exact ID, verified against the declared endpoints and target user. | + +```yaml +exec_policy: + app-maintenance: + source_vm: "@vm:app" + destination_vm: "@vm:database" + target_user: app + depends_on: ["@vm:app", "@vm:database"] +``` + +### `ready` + +Wait for a guest command to exit successfully through the vsock agent. Checks rerun on every apply. The deadline includes connection attempts, command execution, and retry waits; cancellation stops the check. Invalid input and authority failures are terminal errors. + +| Field | Type | Default | Description | +|-------|------|---------|-------------| +| `target` | `string` | — | Required VM selector or `@vm:name`. | +| `cmd` | `string` | — | Required guest check command; zero exit status means ready. | +| `user` | `string` | Config default | Guest user for the check. | +| `timeout` | `int` | `60` | Positive total deadline in seconds. | +| `interval` | `int` | `1` | Positive retry interval in seconds. | + +```yaml +ready: + database-up: + target: "@vm:database" + cmd: "test -f /run/database-ready" + timeout: 60 + interval: 1 + depends_on: ["@vm:database"] +exec: + configure-app: + target: "@vm:app" + cmd: "echo database-ready" + depends_on: ["@vm:app", "@ready:database-up", "@traffic_policy:database-access"] +``` + +The marker is an example: use a command that checks your application's actual readiness. Consumers must depend on the ready step. This is a guest check, not a host shell or host-port probe. Its total deadline is specific to readiness; ordinary `exec.timeout` and CLI `--timeout` keep their connection/probe meaning. + +--- + ### `exec` Run a command inside a VM via vsock agent. @@ -322,17 +478,21 @@ Run a command inside a VM via vsock agent. | `target` | `string` | **Yes** | — | VM name, ID, IP, or MAC. | | `cmd` | `string` | **Yes** | — | Command to execute. | | `user` | `string` | No | Config default | User to run as. | -| `timeout` | `int` | No | `0` (no timeout) | Command timeout in seconds. | +| `timeout` | `int` | No | `0` (configured probe timeout) | Positive values override connection/probe timeout in seconds. Command execution has no absolute timeout; cancellation stops it. | | `port` | `int` | No | `0` (auto) | Vsock port override. | | `env` | `map[string]string` | No | `{}` | Environment variables. | | `ignore_errors` | `bool` | No | `false` | Continue on non-zero exit. | +| `no_sync` | `bool` | No | `false` | Skip guest filesystem sync after the command. | +| `allow_vm_exec` | `bool` | No | `false` | Allow this command to request VM-to-VM execution under an existing directional exec policy. | + +`mvm env apply --env KEY=VALUE` overrides the same key in exec-step `env` maps. It does not alter SSH steps or interpolate YAML fields. **Example:** ```yaml exec: setup-app: target: dev-vm - cmd: "./deploy.sh" + cmd: "uname -a" user: root timeout: 30 depends_on: @@ -351,9 +511,8 @@ Run a command on a VM via SSH. | `user` | `string` | No | Config default | SSH user. | | `key` | `string` | No | Config default | Key step reference or file path. | | `cmd` | `string` | No | `""` | Command. Empty = interactive shell. | -| `timeout` | `int` | No | `0` | Timeout in seconds. | +| `timeout` | `int` | No | Unset | SSH connection timeout in seconds. | | `env` | `map[string]string` | No | `{}` | Environment variables. | -| `ignore_errors` | `bool` | No | `false` | Continue on non-zero exit. | **Example:** ```yaml @@ -375,8 +534,9 @@ Copy files between host and VM via vsock. | Field | Type | Required | Default | Description | |-------|------|----------|---------|-------------| | `src` | `string \| []string` | **Yes** | — | Source path(s). Single string auto-normalized to list. | -| `dest` | `string` | **Yes** | — | Destination in `"vm-name:/remote/path"` format. | +| `dest` | `string` | **Yes** | — | `"vm-name:/remote/path"` when copying into a VM, or a local path when copying out. | | `force` | `bool` | No | `false` | Force overwrite. | +| `no_sync` | `bool` | No | `false` | Skip guest filesystem sync after an upload. | **Example:** ```yaml @@ -393,13 +553,17 @@ copy: ## Destroy Behavior -`mvm env destroy` runs each step's `Destroy()` in reverse dependency order: +`mvm env destroy` tears down resources created by the workflow in reverse dependency order. Resources that existed before apply are preserved. Exec, SSH, and copy side effects are not undone. | Step Type | Destroy Behavior | |-----------|-----------------| | `network` | Deleted — bridge, NAT rules, DB record | | `key` | Deleted — key files, DB record | -| `vm` | Deleted — Firecracker process, console relay, TAP, volumes, DB record | +| `vm` | Deleted — runtime stopped, TAP removed, data volumes detached, VM record removed | +| `volume` | Retained by default; delete only workflow-created volumes declared `retain: false`, after dependent VMs | +| `traffic_policy` | Revoke the exact workflow-created grant before its endpoints; preserve existing policies | +| `exec_policy` | Revoke the exact workflow-created grant before its endpoints; preserve existing policies | +| `ready` | No-op — checks have no teardown action | | `image` | Preserved — cached asset, not destroyed | | `image_import` | Preserved — cached asset, not destroyed | | `kernel` | Preserved — cached asset, not destroyed | @@ -408,48 +572,32 @@ copy: | `exec` | No-op — ephemeral side-effect | | `copy` | No-op — ephemeral side-effect | -### Mid-pipeline removals (`removes`) +### Explicit removals (`removes`) -The `removes` field destroys resources immediately after a step completes — useful for freeing builder VMs mid-pipeline: +The `removes` field requests removal after the whole pipeline succeeds. It does not free resources between steps. +Kernels are always retained by environments: a kernel reference in `removes` is rejected before effects. Manage +kernel retirement separately through the kernel CLI. For example, this fragment captures a builder image before +requesting removal of the builder VM: ```yaml image_import: capture-base: source: "@vm:builder" + depends_on: ["@vm:builder"] removes: - "@vm:builder" ``` ### Ephemeral specs (`ephemeral: true`) -Auto-runs `env destroy` after apply completes (success or failure): - -```yaml -version: "1" -ephemeral: true - -vm: - builder: - image: "@image:os-image" - ... - -exec: - build-artifact: - target: builder - cmd: "make build" - -copy: - retrieve: - dest: ./dist/ - src: "builder:/output/artifact.tar.gz" - removes: - - "@vm:builder" -``` +Set `ephemeral: true` beside `version` to attempt teardown after pipeline success or failure. Cached images, kernels, binaries, retained volumes, and pre-existing resources remain available. Cleanup failures are returned alongside any original apply failure and can leave resources or state behind; inspect `mvm env ls` and retry `mvm env destroy` when necessary. --- ## Full Example +Save this spec as `mvmctl.yaml`, then run `mvm env apply`. It creates a VM and runs `uname -a` after the VM step completes. Local KVM and host initialization are prerequisites. + ```yaml version: "1" @@ -475,7 +623,7 @@ kernel: binary: fc-bin: - version: "1.16.0" + version: "1.17.0" default: true vm: @@ -497,7 +645,7 @@ vm: exec: bootstrap: target: dev-vm - cmd: "curl -sS https://example.com/bootstrap.sh | sh" + cmd: "uname -a" depends_on: - "@vm:dev-vm" ``` @@ -538,4 +686,4 @@ resources: | `state.meta.was_created` | `true` if created by workflow | | `state.meta.spec_hash` | Hash of input spec YAML for drift detection | -State is written after every successful step (crash-resilient). Re-running picks up where it left off. +Resource identities and ownership are persisted after acquisition. Saved `was_created` stays bound to that identity; finding a different object with the same name never transfers ownership. Policy cleanup checks the exact policy incarnation and durable workflow claim atomically; that claim only restricts cleanup and grants no new policy authority. Persistence and cleanup failures are reported, with remaining state retained for retry. Reapply reruns commands, copies, and readiness; it is not a transactional resume or live configuration reconciliation. diff --git a/docs/KERNEL.md b/docs/KERNEL.md index 1bb387e4..14496ba3 100644 --- a/docs/KERNEL.md +++ b/docs/KERNEL.md @@ -99,13 +99,27 @@ mvm kernel pull official:6.19.9 # Build with parallel jobs (faster) mvm kernel pull --type official --jobs 8 -# Bypass cache and force a clean build +# Bypass the compiled-kernel cache mvm kernel pull --type official --clean-build # Keep the build directory after completion (useful for debugging) mvm kernel pull --type official --keep-build-dir ``` +An explicit `MVM_TEMP_DIR` places official-kernel build work in +`/kernel-build`, with compiled-cache artifacts beside that directory. +If the variable is unset or empty, the embedded spec's `build_dir` remains in use. +This override does not change the spec or the managed output location. + +`--clean-build` bypasses the compiled-output cache; it does not remove an existing +source tree or tarball. For a fresh producer run, select a new task-specific scratch +directory. `--keep-build-dir` retains the build tree after successful compilation. + +```bash +MVM_TEMP_DIR="$PWD/.tmp/kernel-build-run" \ + mvm kernel pull official:7.0.11 --jobs 1 --clean-build --keep-build-dir +``` + ### Build process `mvm kernel pull --type official` runs the following steps automatically: @@ -122,6 +136,15 @@ mvm kernel pull --type official --keep-build-dir | 8. Copy & metadata | Copies `vmlinux` to kernels cache and saves metadata JSON | | 9. Cleanup | Removes the build directory (unless `--keep-build-dir`) | +For a producer receipt, use `--debug` and retain the build directory. The +`Kernel base config written` diagnostic identifies the fetched base-config URL, +SHA-256 of the bytes successfully written, byte length, and source directory +before `olddefconfig` or overlays change `.config`. Retain that event alongside +the source archive checksum, embedded spec, invocation, toolchain identities, +and final `.config`/`vmlinux` hashes. A later fetch of the same URL is not proof +of the bytes this invocation consumed. If the event is absent or the build falls +back to `defconfig`, do not claim Firecracker-base-config provenance. + ### Kernel features `mvm kernel pull` supports enabling feature groups via `--features`: @@ -278,15 +301,31 @@ Common causes: ## Reference -### Kernel versions tested with Firecracker +### Kernel version selection -| Kernel | Status | Notes | +| Kernel | Selection | Notes | |--------|--------|-------| -| 6.1.x LTS | Supported | Long-term support, recommended for production | -| 6.6.x LTS | Supported | Newer LTS, recommended for production | -| 6.12.x LTS | Supported | Latest LTS | -| 6.19.x | Supported | Default version (`6.19.9`), latest upstream stable at release time | - -> **Note:** The default kernel version (`6.19.9`) is **NOT** an LTS kernel. It is the latest upstream stable at the time of release. Use `--version latest` to resolve the most recent version from the upstream directory listing. If you need long-term support, explicitly pass `--version 6.1.102` or `--version 6.12.21` to `mvm kernel pull --type official`. The LTS versions in the table above are tested and known to work — use them for production deployments. +| 6.1.x | Explicit version | Also the configured Firecracker CI kernel series | +| 6.6.x | Explicit version | Select the exact required patch release | +| 6.12.x | Explicit version | Select the exact required patch release | +| 6.19.9 | Official source-build default | Fixed embedded default | + +The default source-build version is `6.19.9`. Use `--version latest` to resolve the most recent version from the +upstream directory listing, or pass an exact version for reproducible selection. These selections do not qualify +every patch release or custom configuration. The +[candidate-bound qualification checkpoint](../tests/system/COVERAGE_MATRIX.md#v030-release-security-qualification-overlay) +records accepted runtime evidence and its limits; qualify your selected artifact for its intended workload. + +The v0.3.0 opt-in VM-to-VM source-session path additionally requires the source guest kernel's `SO_PEERPIDFD` socket +capability. Source-session admission fails closed when that capability is absent; it never falls back to authorizing a +numeric PID from `SO_PEERCRED` alone. The configured Firecracker CI kernel series is `6.1`; its cached `6.1.155` +selection cannot supply this capability from stock upstream Linux, whose +[socket implementation lacks `SO_PEERPIDFD`](https://github.com/gregkh/linux/blob/v6.1.155/net/core/sock.c). +Select a capable kernel explicitly with `vm create --kernel ` when using a VM as an exec source. Imported or +custom kernels must provide the actual capability; neither a version label nor successful ordinary host-to-VM exec +proves it. The QA fixture selects an exact capable artifact without changing the default kernel. The configured +official source-build version remains fixed at `6.19.9`; VM-to-VM exec does not select a moving `latest` version or +require the QA artifact's version. Accepted source-session journeys do not establish every peer, cancellation or +cleanup fault; those limits remain in the qualification checkpoint. *See also: [RESOURCE_MANAGEMENTS.md](RESOURCE_MANAGEMENTS.md) for the kernel YAML config reference and [Firecracker official documentation](https://github.com/firecracker-microvm/firecracker/blob/main/docs/getting-started.md).* diff --git a/docs/RC_QA.md b/docs/RC_QA.md index 35e51068..a6d15ea6 100644 --- a/docs/RC_QA.md +++ b/docs/RC_QA.md @@ -21,15 +21,24 @@ A release is **blocked** until ALL of the following pass: | Gate | Command | Must Pass | |------|---------|-----------| -| Compile | `go build ./...` | Zero errors | +| Tidy | `go mod tidy && git diff --exit-code` | Module files unchanged | +| Format | `test -z "$(gofmt -l .)"` | Entire Go tree formatted | +| Line length | `golines --max-len=120 --no-reformat-tags --list-files ./internal/ ./pkg/ ./cmd/` | No listed files | +| Generate | `go generate ./internal/service/agent/...` | Generated assets current | | Vet | `go vet ./...` | Zero warnings | -| Unit tests | `go test ./...` | All pass | -| System tests | `python3 scripts/run-system-tests.py --all` | All pass, zero skips on required tests | -| Version check | `./scripts/build.sh release && ./dist/mvm --version` | Returns correct version (not `0.0.0-dev`) | -| Smoke test | `./dist/mvm --help` | Shows all commands | +| CI Go tests | `bash scripts/run-go-ci-tests.sh` | Unprivileged selection passes | +| Local Jailer Go tests | Full Jailer test binary run as root inside a disposable VM | Entire package passes | +| System install | `sudo ./dist/mvm host install-system && sudo /usr/local/bin/mvm host init` | Canonical root-owned install succeeds | +| System tests | `MVM_BINARY=/usr/local/bin/mvm MVM_CANDIDATE_BINARY=./dist/mvm python3 scripts/run-system-tests.py --release-qualification --host-direct --rebuild --candidate-version X.Y.Z --all --host-reboot-task /absolute/path/to/closed-host-reboot-task` | Identity-qualified T1/T2 runners, clean-host T3 and strict closed reboot proof all pass | +| Version check | `/usr/local/bin/mvm --version` | Returns correct version (not `0.0.0-dev`) | +| Smoke test | `/usr/local/bin/mvm --help` | Shows all commands | **No gate is optional.** A single failure blocks the release. +The system-install, version, and smoke gates run on a dedicated clean qualification host or disposable outer VM. Do not +replace a developer workstation's active controller to satisfy them. During iterative QA, the existing outer controller +creates disposable nested runners and the candidate is installed only inside those runners. + --- ## 2. Test Environment @@ -47,7 +56,8 @@ System tests run inside a Firecracker VM with nested KVM enabled. This provides: See [docs/development/HOW_TO_RUN_SYSTEM_TESTS.md](development/HOW_TO_RUN_SYSTEM_TESTS.md) for the full walkthrough: host prerequisites, building the binary, setting up the -asset mirror, running the orchestrator (`--prepare`, `--all`, `--tier`, `--push`), +asset mirror, running the orchestrator (`--prepare`, `--all`, `--tier`, `--host-direct`, +`--release-qualification`, `--push`), interpreting results, and troubleshooting common failures. The orchestrator script flags are documented there. For the architecture overview @@ -60,9 +70,55 @@ The orchestrator script flags are documented there. For the architecture overvie ### 3.1 System Test Execution -System tests run inside disposable Firecracker VMs with nested KVM, providing -full isolation. The orchestrator creates one VM per domain, runs tests via -`mvm exec`, and destroys the VM after. +T1/T2 system tests run inside disposable Firecracker VMs with nested KVM. The +orchestrator creates one VM per T1/T2 domain, runs tests via `mvm exec`, and +destroys the VM after. Tier 3 is host-direct and has the separate restrictions +below. +For T1/T2, a failed pre-create provision step does not authorize VM deletion. The orchestrator records ownership only +after its `vm create` command succeeds, then includes checked destruction in the domain result even if later setup fails. +An ambiguous create failure remains a failed domain needing operator inspection; the runner does not guess ownership. + +For T1/T2, release qualification keeps the outer controller (`MVM_BINARY`) separate from `./dist/mvm` +(`MVM_CANDIDATE_BINARY`). The controller manages disposable test resources. The candidate-qualified base image is built +by staging the candidate, invoking `host install-system`, and initializing through the resulting exact +`/usr/local/bin/mvm`. T1/T2 runners clone that image and execute the installed path; the focused `system_install` domain +replays and verifies the administrator install contract. Directly copying onto the canonical path, relying on PATH +lookup, or aliasing the candidate to the controller does not qualify the installer. + +The separate [named-runner host-reboot journey](development/HOW_TO_RUN_SYSTEM_TESTS.md#47-explicit-named-runner-host-reboot-journey) +requires a private `--host-reboot-namespace` JSON file for PRE. It binds the existing runner ID, an explicit +non-root guest passwd/process identity, and existing owned home/task-parent/cache/config/mirror directory identities. +Keep the original outer controller and attached SSH key unchanged; all phase pytest runs inside the QA runner. +POST may omit the flag and reuse the retained binding, or supply the identical namespace. Task and PRE/POST +receipts use schema 2; schema-1 records and legacy cache state are not adopted. PRE success, phase collection and +focused tooling checks do not qualify an actual reboot. Full qualification requires the fresh, completed +PRE/reboot/POST task on the exact accepted source and candidate; earlier receipts keep their original bindings. + +The builder's unprivileged initialization pins the approved Firecracker/Jailer 1.17.0 pair through +`mvm init --non-interactive --binary-version 1.17.0`; it must not resolve a moving latest release during qualification. +The upstream stable-release selection was checked on 2026-10-10. Archive admission and installed-candidate runtime +qualification are separate gates; earlier 1.16.2 results do not qualify 1.17.0. +Base-image preparation no longer removes a pre-existing fixed-name builder VM. If creation reports that the name is +occupied, preparation fails and leaves that VM untouched; cleanup applies only after this run's create succeeds. +Every builder artifact pull must also report an explicit local-mirror read. A pull error, checksum fallback, or +HTTP-downloaded mirror auto-population fails qualification; bounded stdout/stderr remains in the evidence log. Fixed +checksum/version metadata may still use its upstream source. + +Tier 3 is different: those tests run directly on the outer host and may create or destroy host resources. The runner +rejects Tier 3 and `--all` before any work unless `--host-direct` acknowledges that mutation. For release evidence, +`--release-qualification` additionally requires an unfiltered `--all`, an explicit-version rebuild, exact canonical +root-owned `/usr/local/bin/mvm`, safe ancestors, and matching candidate/controller SHA-256 and strict version before +resource preparation. + +These gates still do not prove that the host started empty or that every Tier 3 resource belongs to the current run. +Until the remaining Task 17 inventory and cleanup gates land, run full qualification only on a dedicated clean host or +disposable outer VM. On a development workstation, use focused T1/T2 domains or `--tier 1,2`; `--host-direct` is consent, +not isolation or an ownership-safe cleanup guarantee. + +The release preflight rejects non-stopped VMs in the invoking operator's versioned `vm ls --json` projection, but that +projection begins with the operator's SQLite rows. It cannot prove another user's VMs or a root-owned record without +a user row are absent. This defense-in-depth check is **not** host-wide quiescence or completion of Task 17; independent +clean-host assurance and the remaining authority/cleanup gates are still required. Tests are organized into three tiers: @@ -112,33 +168,68 @@ For every release, collect and archive: | Evidence | How to Collect | Purpose | |----------|----------------|---------| -| Go build output | `go build ./... 2>&1` | Zero compilation errors | +| Tidy output | `go mod tidy && git diff --exit-code` | Module files unchanged | +| Format output | `test -z "$(gofmt -l .)"` | Entire Go tree formatted | +| Generate output | `go generate ./internal/service/agent/...` | Generated assets current | | Go vet output | `go vet ./... 2>&1` | Zero static analysis warnings | -| Go test output | `go test ./... 2>&1` | All unit tests pass | -| System test results | `python3 scripts/run-system-tests.py --all` | All system tests pass | -| Version output | `./dist/mvm --version` | Correct version string | -| Help output | `./dist/mvm --help` | All commands listed | +| CI Go test output | `bash scripts/run-go-ci-tests.sh` | Unprivileged selection passes | +| Local Jailer output | Full Jailer test binary run as root inside a disposable VM | Entire package passes | +| System test results | `MVM_BINARY=/usr/local/bin/mvm MVM_CANDIDATE_BINARY=./dist/mvm python3 scripts/run-system-tests.py --release-qualification --host-direct --rebuild --candidate-version X.Y.Z --all --host-reboot-task /absolute/path/to/closed-host-reboot-task` | Identity-qualified full matrix and closed reboot proof pass on the clean host | +| Version output | `/usr/local/bin/mvm --version` | Correct version string | +| Help output | `/usr/local/bin/mvm --help` | All commands listed | | Benchmark results | `benchmarks/results.json` | Performance within thresholds | | Binary checksum | `sha256sum dist/mvm` | Reproducibility | ### 4.1 Evidence Archive ```bash -# Create evidence directory -mkdir -p release-evidence/vX.Y.Z +set -euo pipefail + +version=X.Y.Z +evidence_dir="release-evidence/v${version}" +mkdir -p "$evidence_dir" +exec > >(tee -a "$evidence_dir/qualification.log") 2>&1 # Collect evidence -go build ./... > release-evidence/vX.Y.Z/build.log 2>&1 -go vet ./... > release-evidence/vX.Y.Z/vet.log 2>&1 -go test ./... > release-evidence/vX.Y.Z/test.log 2>&1 -./dist/mvm --version > release-evidence/vX.Y.Z/version.txt 2>&1 -./dist/mvm --help > release-evidence/vX.Y.Z/help.txt 2>&1 -sha256sum dist/mvm > release-evidence/vX.Y.Z/checksum.sha256 +./scripts/build.sh release --version "$version" --output dist/mvm +go mod tidy +git diff --exit-code +test -z "$(gofmt -l .)" +test -z "$(golines --max-len=120 --no-reformat-tags --list-files \ + ./internal/ ./pkg/ ./cmd/)" +go generate ./internal/service/agent/... +git diff --exit-code +go vet ./... > "$evidence_dir/vet.log" 2>&1 +bash scripts/run-go-ci-tests.sh \ + > "$evidence_dir/test.log" 2>&1 + +# Archive the separate full-Jailer root-test result from a disposable local VM +# before release sign-off. Do not run that binary as root on this host. + +# The following gates mutate the outer host. Run them only on the dedicated +# clean qualification host or disposable outer VM. +sudo ./dist/mvm host install-system +sudo /usr/local/bin/mvm host init +test "$(/usr/local/bin/mvm --version)" = "mvm ${version}" +/usr/local/bin/mvm --version > "$evidence_dir/version.txt" 2>&1 +/usr/local/bin/mvm --help > "$evidence_dir/help.txt" 2>&1 +sha256sum dist/mvm > "$evidence_dir/checksum.sha256" # System test results (full suite with orchestrator) -python3 scripts/run-system-tests.py --all 2>&1 | tee release-evidence/vX.Y.Z/system-tests.log +MVM_BINARY=/usr/local/bin/mvm MVM_CANDIDATE_BINARY=./dist/mvm \ + MVM_ASSET_MIRROR="$HOME/.cache/mvm-asset-mirror" \ + python3 scripts/run-system-tests.py \ + --release-qualification --host-direct --rebuild \ + --candidate-version "$version" --all \ + --host-reboot-task /absolute/path/to/closed-host-reboot-task \ + 2>&1 | tee "$evidence_dir/system-tests.log" ``` +The orchestrator accepts a domain-level `PASS` only after validating a strict bounded pytest outcome report: collection +is non-empty, process/report exit status matches, every selected item passed, and failures, errors, collection errors, +deselections, skips, XFAIL, and XPASS are all zero. Missing, malformed, duplicate-field, oversized, inconsistent, or +uncleared reports fail the domain and preserve pytest output plus the validation reason in the archived log. + --- ## 5. Regression Criteria @@ -160,7 +251,7 @@ A regression is **any** of the following: | Category | Definition | |----------|-----------| | **Expected behavior change** | A feature was intentionally changed (documented in CHANGELOG.md) | -| **New test failure** | A new test was added that reveals a pre-existing bug (file a bug, don't block release) | +| **New test exposing an old bug** | Not classified as a regression, but the revealed defect still blocks the release until fixed or the requirement is explicitly removed | | **Performance improvement** | A benchmark is faster than before | | **Test environment issue** | Missing dependency, wrong permissions, stale state (fix environment, not code) | @@ -181,16 +272,25 @@ A regression is **any** of the following: ## Release vX.Y.Z — QA Sign-off ### Gates -- [ ] `go build ./...` — zero errors +- [ ] `go mod tidy && git diff --exit-code` — module files and generated tree unchanged +- [ ] `test -z "$(gofmt -l .)"` — entire Go tree formatted +- [ ] `golines --max-len=120 --no-reformat-tags --list-files ./internal/ ./pkg/ ./cmd/` — no files listed +- [ ] `go generate ./internal/service/agent/... && git diff --exit-code` — generated assets current - [ ] `go vet ./...` — zero warnings -- [ ] `go test ./...` — all pass -- [ ] System tests — all pass inside runner VM (see evidence archive) -- [ ] `./dist/mvm --version` — returns vX.Y.Z -- [ ] `./dist/mvm --help` — all commands present +- [ ] `bash scripts/run-go-ci-tests.sh` — unprivileged Go selection passes +- [ ] Full Jailer Go test binary — passes as root in a disposable local VM +- [ ] `./scripts/build.sh release --version X.Y.Z --output dist/mvm` — exact candidate built +- [ ] Dedicated clean qualification host/outer VM — candidate installed at exact `/usr/local/bin/mvm` +- [ ] System tests — every required T1/T2/T3 domain passes strict machine outcome validation +- [ ] Named-runner host reboot — fresh schema-2 PRE/reboot/POST proof passes with the retained operator/directory binding +- [ ] `/usr/local/bin/mvm --version` — returns exact `mvm X.Y.Z` +- [ ] `/usr/local/bin/mvm --help` — all commands present ### Evidence - [ ] Build log archived +- [ ] Tidy, format, line-length, generate, vet, and Go-test logs archived - [ ] System test log archived +- [ ] System-test log archives the strict per-domain outcome gate and every failure reason - [ ] Binary checksum archived - [ ] Benchmark results within thresholds diff --git a/docs/REFERENCES.md b/docs/REFERENCES.md index f1fbb0ce..0964335a 100644 --- a/docs/REFERENCES.md +++ b/docs/REFERENCES.md @@ -1,6 +1,9 @@ # Command Reference & Configuration This document provides detailed reference for all `mvm` commands, configuration options, and environment variables. +The [candidate-bound qualification checkpoint](../tests/system/COVERAGE_MATRIX.md#v030-release-security-qualification-overlay) +records accepted runtime evidence and remaining fault-coverage limits. v0.3.0 remains unreleased; these results do +not qualify every fault path, aarch64 runtime, remote CI or a different candidate. --- @@ -17,6 +20,7 @@ This document provides detailed reference for all `mvm` commands, configuration - [mvm snapshot](#mvm-snapshot) - [mvm console](#mvm-console) - [mvm network](#mvm-network) +- [mvm policy](#mvm-policy) - [mvm key](#mvm-key) - [mvm config](#mvm-config) - [mvm cache](#mvm-cache) @@ -43,8 +47,20 @@ This document provides detailed reference for all `mvm` commands, configuration |------|------|-------------| | `--verbose` | persistent | Enable verbose output | | `--debug` | persistent | Enable debug mode | +| `--json` | persistent | Emit one schema-versioned response for a supported finite command | | `--version` | root | Show version and exit | +`--json` applies to reads and mutations. The response has `schema_version: 1`, `status` (`success`, `error`, or +`partial`), command-specific `data`, and `error`. An error preserves its code, class, message, optional operation and +entity, and details. Batch data preserves each result and its structured error. A partial response retains completed +work and exits unsuccessfully; callers must inspect both status and data before retrying. Read results now live under +`data`, so scripts consuming the old bare list or object must be updated. Array-valued resource listings, such as +`key ls`, `vm ls`, and `vm ps`, remain arrays under `data`; an empty result is `[]`, not `null`. + +Streaming and interactive commands reject structured output with `cli.output.unsupported`: `exec`, `ssh`, `logs`, +completion, `run`, the `init` wizard, and console attachment. Finite console state and kill commands support it. +Arguments after `--` belong to the guest command and do not select host JSON output. + ### Selectors Resource commands (`inspect`, `rm`, `pull`, etc.) accept **selectors** — flexible @@ -82,7 +98,9 @@ pattern is: ### `mvm init` -First-time setup wizard. Walks through host init, binary/kernel/image download, and SSH key setup. +First-time operator setup wizard. Coordinates host admission, a default network, cache initialization, and a +Firecracker/Jailer release pair. Kernel, image, and SSH-key setup are separate commands. The approved v0.3.0 launch +contract requires the exact matching Firecracker/Jailer 1.17.0 pair. ``` mvm init [flags] @@ -93,6 +111,10 @@ mvm init [flags] | `--non-interactive` | Use defaults, skip all prompts | `false` | | `--skip-host` | Skip host init step | `false` | | `--skip-network` | Skip default network creation | `false` | +| `--binary-version ` | Download this exact Firecracker version when no local pair exists | empty (interactive choice or latest in non-interactive mode) | + +`--binary-version` does not replace an existing default pair. To select the v0.3.0 launch pair when another release is +already installed, run `mvm bin pull 1.17.0 --default` and check `mvm bin ls` before creating a VM. --- @@ -102,11 +124,54 @@ Host configuration commands for one-time, machine-global setup. | Command | Flags | Description | |---------|-------|-------------| -| `mvm host init` | — | Apply host configuration changes. Idempotent. Creates the `mvm` group, sudoers drop-in, enables IP forwarding, creates firewall chains, and snapshots initial state for rollback. | +| `mvm host install-system` | — | Administrator-only early bootstrap. Atomically installs the running trusted artifact as root-owned `/usr/local/bin/mvm` without loading user cache, configuration, database, or Cobra state. | +| `mvm host init` | `--operator USER` (direct root only), `--json` | Early administrator setup with a root-owned journal, operator enrollment, and compiled host changes. | | `mvm host status` | `--json` | Show current host configuration state vs expected | | `mvm host info` | `--refresh`, `--json` | Show host hardware, limits, and VM capacity projection | -| `mvm host clean` | `-f, --force` | Remove all VM networking config (bridges, TAPs, iptables). Does not touch sysctl or mvm group. | -| `mvm host reset` | `-f, --force` | Full rollback: remove networking, revert sysctl, remove sudoers and mvm group | +| `mvm host clean` | `-f, --force` | Observe the caller's root VM inventory, then remove its locally registered networks through typed root authority. Refuses running VMs or incomplete observation; does not force-stop VMs or reset shared host settings. | +| `mvm host reset` | `--force`, `--operator USER` (direct root only), `--json` | Early administrator rollback under retained global quiescence. Requires marker-only sudoers and compatible journals; refuses live or ambiguous state rather than force-stopping VMs. Admission remains disabled afterward. | + +For full v0.3.0 removal, the separate [administrator uninstall script](UNINSTALL.md) cleans all enrolled +operators' managed data, performs host reset, then removes installation files and authority. + +`host install-system` accepts only the exact command shown: no flags or extra arguments, including `--json`, +`--debug`, or `--verbose`. It runs before ordinary CLI initialization. + +`host info` reads a caller-owned capacity cache, detecting and saving capacity on the first read and again with +`--refresh`. It does not infer administrator setup from that cache. Structured `host info --json` omits `setup`; +`host status --json` reports `data.state: null` rather than claiming root admission from the legacy SQLite bit. + +`host status` verifies the fixed managed sudoers policy through authenticated root observation when it cannot +confirm absence directly. Permission denial does not mean the policy is missing. Confirmed absence reports +`data.sudoers_exists: false`; unsafe policy metadata or content, unavailable observation, and a missing or invalid +observer response return an error instead of a successful status result. + +The interactive `mvm init` wizard can offer explicit administrator setup even when the operator cannot yet observe +the policy. +It reports that initial state as unverified, then requires successful host-policy and group verification after setup. +If the current session lacks the enrolled group, log out and back in or run `newgrp mvm` before retrying. + +On a clean host, run `sudo /usr/local/bin/mvm host init`; sudo selects its invoking non-root account. In a direct-root +session, use `/usr/local/bin/mvm host init --operator USER`. Sudo invocation rejects `--operator`. The early route runs +before user configuration, cache, SQLite, and ordinary service initialization. Its journal beneath +`/var/lib/mvmctl/host` preserves original global state and mutation progress. Separate root-owned +`operator-.json` records preserve each enrolled operator's prior membership. Multiple operators are supported; +an interrupted initial global transaction must first be recovered by its original operator. Existing transitional +wildcard sudo grants cannot establish administrator authentication: that sudo entry +refuses them and directs the administrator to direct root with an explicit operator. Init does not install the binary. + +Host initialization uses a schema-2 administrator journal and enables privileged admission only after host setup and +operator enrollment complete. Old schema-1 administrator journals require a clean installation; they are not adopted. +Missing, disabled, or unsafe admission state blocks ordinary privileged requests. + +Generated sudoers permits only the fixed system binary's `__mvm_privileged_v1` marker. Install/init/publication reject +older wildcard and raw-tool policies. No host policy activation or runtime qualification was performed in this pass. +Administrator reset is implemented for compatible recorded state: it refuses live or ambiguous ownership, restores +recorded settings through retryable receipts, and leaves admission disabled afterward or after a restoration error. +It does not force-stop VMs, erase authority history, or turn old unowned state into managed state. + +The early parser supports JSON boolean forms, help, and argument boundaries. Host clean/reset are not a supported +legacy-state removal procedure in this development build. --- @@ -205,9 +270,57 @@ The image resolver tries selectors in priority order: | `--version VERSION` | Set image version | — | | `--default, -d` | Set as default after import | `false` | | `--force, -f` | Overwrite existing | `false` | +| `--resume IMAGE_ID` | Resume retained local-image optimization using its full ID; incompatible with `--force` | — | | `--skip-optimization` | Skip OS cache cleanup (deblob), keep plain ext4 | `true` | | `--disable-detector NAME` | Comma-separated detectors to disable: `type`, `label`, `size`, `filesystem`, `all` | — | +**Importing from a VM:** `mvm image import NAME VM_SELECTOR` reads the VM's current protected root disk through +the enrolled, typed root dispatcher, not its original operator-visible staging file. A running VM is paused for +the copy and resumed before extraction; already paused or stopped sources retain their state. This produces a +disk image, not a memory or attached-data-volume snapshot. The best-effort guest `sync` hint does not guarantee +application transaction consistency. + +Temporary independent copies live under root-controlled `/var/lib/mvmctl/vm-rootfs-export-data/` +with operator read access. Export copying skips source-reported holes while preserving the logical disk length +and contents. Allow space on that filesystem as well as the selected managed cache, where protected +VM disks and final images remain. Typed cleanup releases the temporary copy after extraction and before final +publication; copy, state-recovery or cleanup failures remain explicit rather than falling back to staging. + +An ext4 disk copied while the source VM is paused can still require filesystem journal replay. With LoopMount, +import runs fixed unprivileged `/usr/sbin/e2fsck -p -E journal_only` on a new disposable inspection copy before +the unchanged root read-only OS inspection. This does not modify the protected VM disk, root-controlled export +or extracted source, and it is not general filesystem repair. Journal preparation failure stops the import; +it does not enable a writable inspection mount or discard the journal. See the +[LoopMount dependencies](DEPENDENCIES.md#e1-loop-mount-default). + +If an interrupted import leaves no public image row, `mvm image rm --force EXACT_64_CHARACTER_IMAGE_ID` can +release its independent export. The read-only `.copy` marker in the operator's export directory identifies +the ID; it is not deletion authority. This recovery requires one full lowercase hexadecimal ID and confirmed +repository absence: names, prefixes, multiple selectors and database errors cannot authorize it. A live exporter +reports busy. Copy removal does not clear uncertain source-VM recovery; that condition is reported separately. +Use the typed command, not manual deletion. An open reader can retain unlinked bytes, so successful cleanup does +not promise immediate physical reclamation. + +VM-to-image import copies the root disk, not RAM or attached data volumes, and does not promise application-consistent +state across them. Accepted registered journeys and remaining interruption/fault limits are recorded in the +qualification checkpoint. See +[ADR-0018](adr/0018-isolated-vm-execution-and-snapshot-authority.md#vm-to-image-import-from-authoritative-backing). + +Typed image optimization supports deblob and fstab repair, ext-family offline shrink, and durable Btrfs-to-ext4 +conversion as a separate transaction before shrink. XFS supports inspection, provisioning, deblob, and fstab repair +at existing capacity; resize/shrink requests fail before mutation. + +Interrupted optimization can resume from exact retained staging intent and resource identities. A checked root result +is required before publication, which requires atomic exchange and preserves the original until durable completion. +Unknown or changed remnants remain for recovery rather than being adopted or recursively removed. +For a local-file import, use `mvm image import NAME ORIGINAL_SOURCE --resume FULL_IMAGE_ID` with the original +optimization options. Failed local imports include `image_id` in their JSON error details. Resume requires an +unregistered ID and intact retained optimization state; it skips extraction and refuses changed resources or intent. +It does not recover VM-source imports or failures after optimization state has already been removed. Preserve the +original source and retained files; do not remove root-owned records manually. +Unprivileged archive construction remains separate; passing registered journeys does not establish every interruption +or replacement-race outcome. + ``` # List available remote image types mvm image ls --remote @@ -224,7 +337,14 @@ mvm image import base-img my-vm ### `mvm bin` -Firecracker binary management — download, list, and remove Firecracker and jailer binaries. +Firecracker runtime management — install, list, select, and remove Firecracker/Jailer release pairs. Root authority +fetches or accepts the optional exact mirror archive, verifies it, and atomically publishes the pair in the trusted +release store. Source builds and manually supplied binary paths are unsupported. + +The approved VM launch pair is exactly 1.17.0. Binary-management support for an older audited release does not +authorize VM launch with it. Release updates review and qualify the latest stable matching official pair before +changing the pin; the installed runtime does not automatically follow upstream `latest`. See +[runtime guidance](RUNTIME.md#vm-launch-and-resource-control). #### Selectors @@ -237,19 +357,19 @@ The binary resolver tries selectors in priority order: | Command | Flags | Description | |---------|-------|-------------| | `mvm bin ls` | `-r, --remote`, `--limit`, `--json`, `--long` | List local (and optionally remote) Firecracker versions | -| `mvm bin pull` | `[SELECTOR]`, `--version`, `--git-ref`, `--default, -d`, `--force, -f` | Download a Firecracker version or build from source | +| `mvm bin pull` | ``, `--default, -d`, `--force, -f` | Install one root-authoritative release | | `mvm bin default` | `SELECTOR` | Set a binary as the active default | -| `mvm bin rm` | `[SELECTORS]...`, `--version`, `-f, --force` | Remove one or more binaries, or use `--version` to remove a version pair | +| `mvm bin rm` | `...`, `-f, --force` | Remove exact releases; force hides local advisories only | ``` # List local and remote versions mvm bin ls --remote -# Download a specific Firecracker version -mvm bin pull firecracker:1.15.0 +# Install and select the required VM launch pair +mvm bin pull 1.17.0 --default -# Build from git source -mvm bin pull firecracker --git-ref v1.15.0 +# Remove the exact architecture-scoped release pair +mvm bin rm 1.15.0 ``` --- @@ -280,6 +400,27 @@ The VM resolver tries selectors in priority order: | `mvm vm ps` | `--json` | List running VMs (active processes) | | `mvm vm inspect` | `SELECTOR`, `--json` | Show detailed information about a VM | +If creation was interrupted before the VM reached the caller database, `mvm vm rm --force ` can request +cleanup through the retained root authority. This recovery form accepts exactly one full 32-character lowercase +hexadecimal ID, not a name or ID prefix; existing registered selectors retain their usual behavior and name precedence. +Root authorization and ownership checks still apply: a missing or inadmissible root record is not permission to +delete files. On success, the result reports `caller_artifacts_retained: true`; caller-owned source files, the VM +directory and staging artifacts are preserved. This is not a filesystem orphan sweep. + +VM launch and process control use typed root lifecycle transactions and schema-6 instance records separating operator +ownership from per-VM execution credentials. PID values in +output are display projections, never signaling authority. List observes its complete VM selection before filtering +status and propagates observation failures; inspect/get also propagate them. Paused display state requires an observed +live process. Stop preserves endpoint reservations and traffic grants for restart; remove invalidates them through +root before local deletion. Without `--force`, stop requests guest shutdown through an identity-bound Firecracker API +connection and waits on the exact process before authoritative cleanup. + +Typed launch now includes console relay composition and raw data volumes with release-derived device limits, +descriptor-only mounts, and no new storage quotas. PCI launch and x86 nested configuration are implemented for the +pinned release; ARM nested launch is rejected and aarch64 runtime qualification remains open. Typed live-volume +operations and snapshot capture/inspect/remove use root authority. Snapshot restore uses the captured topology and +independent protected disks. The qualification checkpoint records the bounded lifecycle and restore evidence. + **`vm create` flags:** | Flag | Description | Default | @@ -293,12 +434,13 @@ The VM resolver tries selectors in priority order: | `--ip ADDRESS` | Guest IP | auto-assigned | | `--mac ADDRESS` | Custom MAC address | auto-generated | | `--network, --net NAME` | Named network | from config | +| `--no-network` | Explicitly create without a managed network, NIC, guest IP, or MAC; use vsock exec rather than SSH | `false` | | `--volume, -v TEXT` | Attach volume(s) to the VM | none | | `--ssh-key NAME_OR_PATH` | SSH public key (name from cache or file path) | default keys | | `--user USER` | Default SSH user | from config | | `--cloud-init-mode MODE` | Cloud-init mode: `inject`, `iso`, `net`, `off` | `off` (effective) | | `--writeback` | Use writeback cache mode for drives (guest fsync honored) | `false` | -| `--nocloud-net-port N` | Port for nocloud-net HTTP server (0 = auto) | auto-assign | +| `--nocloud-net-port N` | Reserved NoCloud TCP port; custom ports are rejected | `8000` | | `--cloudinit-config PATH` | Path to custom cloud-init config file | — | | `--no-pci` | Disable PCI device support | from config | | `--nested-virt` / `--disable-nested-virt` | Enable/disable nested virtualization (requires PCI) | from config | @@ -309,13 +451,11 @@ The VM resolver tries selectors in priority order: | `--boot-args ARGS` | Kernel boot arguments | from config | | `--console` | Enable serial console relay | `false` | | `--vsock-port N` | Vsock port for the guest agent | `1024` | -| `--writeback` | Use writeback cache mode for drives | from config | | `--skip-cleanup` | Keep resources on failure for debugging | `false` | | `--force, -f` | Skip confirmation prompts | `false` | | `--count, -c N` | Create N VMs in batch | `1` | | `--atomic` | All-or-nothing batch: roll back all VMs if any creation fails | `false` | | `--skip-deblob` | Skip debloat operations on rootfs | `false` | -| `--allow-remote-exec` / `--deny-remote-exec` | Allow/deny inter-VM remote execution via vsock relay | from config | **Examples:** @@ -328,12 +468,32 @@ mvm vm create my-vm --image ubuntu:24.04 --kernel official:6.19.9 --vcpu 2 --mem # Create 3 VMs in batch mvm vm create cluster-node --count 3 + +# Create a networkless VM for vsock-only access +mvm vm create offline-worker --no-network --cloud-init-mode off ``` +`--no-network` is an explicit choice, not the result of omitting `--network`. It conflicts with `--network`, +`--ip`, `--mac`, and NoCloud network mode. Accepted candidate journeys include networkless full capture/restore +with an attached raw volume and guest exec, plus create/stop/start. Policy-specific spoofing, IP reuse and fault +recovery limits remain listed in the qualification checkpoint. + --- ### `mvm snapshot` (alias: `ss`) +Capture/inspect/remove use root-controlled +storage and exclusive protected rootfs/data-volume capture. Restore copies disks into independent protected backing, +retains immutable memory backing, and preserves captured drive IDs and network identity. The public API reserves the +captured IP before root restore; conflicts fail, and an uncertain root outcome keeps the reservation for recovery. +The accepted candidate passed registered snapshot journeys, including networkless and same-network managed restore. +Broader interruption, post-spawn and uncertain-transport recovery remain limited as described in the qualification +checkpoint and ADR-0018. + +Create fresh 1.17.0 snapshots and qualify capture and +restore with the same matching release. Captured versions are preserved; no cross-version conversion or compatibility +is promised. See [runtime guidance](RUNTIME.md#vm-launch-and-resource-control). + Create, list, inspect, restore, and remove VM snapshots. #### Selectors @@ -351,6 +511,12 @@ The snapshot resolver tries selectors in priority order: | `mvm snapshot restore` | `SELECTOR`, `NAME`, `--network`, `--resume`, `--count N` | Restore one or more VMs from a snapshot | | `mvm snapshot rm` | `SELECTOR`, `--force, -f` | Remove a snapshot | +`snapshot create` writes the full **Snapshot request ID** to stderr before trusted capture, including with `--json`. +Keep that ID until the command completes. If the command is interrupted, `snapshot ls` may omit a root-published +capture; after the root operation settles, use `mvm snapshot inspect --json` to check it and +`mvm snapshot rm --force` to remove an unwanted artifact. Do not edit the root-controlled files directly. +`snapshot ls` does not discover unindexed root captures, so losing the ID still requires administrator investigation. + --- ### `mvm console` @@ -385,13 +551,64 @@ The network resolver tries selectors in priority order: | Command | Flags | Description | |---------|-------|-------------| -| `mvm network create` | `[NAME]`, `--subnet`, `--ipv4-gateway`, `--no-nat`, `--nat-gateways`, `--non-interactive`, `--default, -d` | Create a named bridge network | +| `mvm network create` | `[NAME]`, `--subnet`, `--ipv4-gateway`, `--no-nat`, `--default, -d` | Create a named bridge network without prompting | | `mvm network rm` | `[SELECTORS]...`, `-f, --force` | Remove one or more networks | | `mvm network ls` | `--json, --long` | List all networks | | `mvm network inspect` | `[SELECTOR]`, `--json` | Show network details, IP leases, and active firewall rules | | `mvm network default` | `[SELECTOR]` | Set a network as the default for VM creation | | `mvm network sync` | `[SELECTORS]...`, `--json` | Sync firewall rules between database and host | +NAT intent is enabled or disabled; `--nat-gateways` and caller-selected uplink names are removed. Root selects the +eligible host uplink. Typed root network creation applies its firewall generation before raising the bridge, and +removal uses exact ownership and reference checks with a durable receipt. Core/API network mutations consume these +transactions. A pending desired network can be retried only with matching inputs. If no root network record exists, +removal can acknowledge cancellation only after root proves the complete bridge/topology/firewall state absent; it +performs no speculative kernel deletion. Cancellation tombstones prevent reuse of that network ID. Old-boot removal +receipts also require fresh absence proofs. These paths have not been qualified. NoCloud now uses typed fixed-slot +host-service grants and no longer uses the legacy tracker. + +--- + +### `mvm policy` + +Policy management separates TCP/UDP traffic intent from directional VM-to-VM execution grants. The former flat +`mvm policy create/ls/inspect/rm/sync` commands are replaced by the typed families below. + +| Command | Flags | Description | +|---------|-------|-------------| +| `mvm policy traffic create` | Exactly one of `--from-network NETWORK` or `--from-vm VM`; exactly one of `--to-vm VM` or `--to-host`; `--protocol tcp\|udp`, `--port PORT\|START-END` | Apply a root-authorized traffic grant, then project it locally | +| `mvm policy traffic ls` | `--json` | List VM and host-gateway grants with current resource names | +| `mvm policy traffic inspect` | `POLICY_ID`, exactly one of `--from-network` or `--from-vm`, optional `--to-host`, `--json` | Inspect intent; add `--to-host` for a host-gateway grant | +| `mvm policy traffic rm` | `POLICY_IDS...`, exactly one of `--from-network` or `--from-vm`, optional `--to-host`, `-f, --force` | Revoke the root grant or prove its absence before deleting the local projection | +| `mvm policy exec create` | `SOURCE_VM DESTINATION_VM TARGET_USER` | Persist one directional execution grant for an exact non-root destination user | +| `mvm policy exec ls` | `--json` | List directional execution policies with current VM names | +| `mvm policy exec rm` | `POLICY_IDS...`, `-f, --force` | Remove one or more execution policies by exact ID or unambiguous prefix | + +Traffic creation applies the grant through root before updating SQLite; removal requires root revocation or exact +absence first. List and inspect show the local projection. Root success followed by a projection failure is reported +for repair/retry and does not undo the root outcome. Policies may bind exact active or cleaned root endpoints. Stopping +a VM preserves endpoint reservations and grants for restart; removing it invalidates them. A grant to a stopped +endpoint does not create a live network interface. + +Use `--to-host` to reach a host service through the source managed network's gateway. For a network with gateway +`10.25.0.1`, the guest connects to that address, not the workstation's LAN address or `127.0.0.1`: + +```bash +mvm policy traffic create --from-vm client --to-host --protocol tcp --port 5432 +mvm policy traffic inspect POLICY_ID --from-vm --to-host +mvm policy traffic rm POLICY_ID --from-vm --to-host --force +``` + +Operators can select any destination port, but every policy must explicitly name its protocol and port/range. +The service must listen on the gateway or a suitable wildcard address; the policy does not expose a localhost-only +listener. `--from-network` grants the selected network's VMs access to its own gateway. Host-to-VM access is not +managed by these policies. JSON lists separate `network_to_vm`, `vm_to_vm`, `network_to_host` and `vm_to_host` arrays. + +Execution grants are directional, independent of IP-network placement, and require an exact non-root target user. +The opt-in API relay now reads and enforces them at runtime. It also requires root-admitted active VM identities; +creating a grant alone cannot make a transitional VM launch eligible. Passing directional execution journeys does +not establish every peer, capacity, deadline, cancellation or private-backend cleanup outcome. + --- ### `mvm key` @@ -449,6 +666,11 @@ Cache management. | `--dry-run` | Show what would be removed without removing | | `--force, -f` | Skip confirmation prompts | +Cache cleanup uses typed VM/network operations and no longer offers to rerun the ordinary CLI through sudo or +checks for a raw `ip` executable. It no longer performs arbitrary unmounts through the removed raw provisioner. +Inspection and cleanup failures propagate instead of being reported as empty cache or successful cleanup. +`mvm run provision` is removed. + Omit `RESOURCE` to get an error unless `--all` is passed. Each per-resource prune subcommand (`vm`, `network`, `image`, `kernel`, `binary`, `misc`) prompts for confirmation unless `--force` is set. **`cache clean` flags:** @@ -492,8 +714,9 @@ mvm exec [VM_SELECTOR] [-- ...] [flags] | `VM_SELECTOR` | VM name, ID, MAC, or IP (positional, required) | — | | (args after `--`) | Command to execute (omit for interactive shell) | — | | `--user, -u TEXT` | User to run the command as | `root` | -| `--timeout, -t N` | Vsock agent connect/probe timeout in seconds | `0` (no timeout) | +| `--timeout, -t N` | Vsock agent connect/probe timeout in seconds | `0` uses `defaults.vm.vsock_probe_timeout` (built-in: 5 seconds) | | `--no-sync` | Skip final `sync()` after command | `false` | +| `--allow-vm-exec` | Permit this non-interactive source command to request policy-authorized VM-to-VM execution | `false` | | `--port, -p N` | Vsock port for the guest agent | `1024` | **Examples:** @@ -512,6 +735,27 @@ mvm exec my-vm --timeout 30 -- apt-get update mvm exec my-vm --user ubuntu ``` +An explicit source command can request execution on another VM with: + +```bash +mvm policy exec create source-vm destination-vm app +mvm exec source-vm --allow-vm-exec -- mvm-agent remote --user app destination-vm -- command arg +``` + +The outer source command may run as root or a non-root account; the destination user must be non-root. The relay +requires the exact directional grant and root-admitted active source/destination identities, applies bounded wire and +capacity limits, and writes command-redacted audit records in the user cache. The opt-in outer command and its one +remote request share an absolute five-minute lifetime; `--timeout` still controls connection/probe time. Ordinary +exec, shell, and copy do not enable this source session. The source guest kernel must provide `SO_PEERPIDFD`; the +stock upstream `6.1.155` kernel used by the cached Firecracker CI selection lacks it. Select a capable source kernel +explicitly; see [kernel capability requirements](KERNEL.md#reference). This does not change the default kernel. + +If an admitted atomic pidfd proves that an opted-in source command exited before publication, the same binding is +still reserved, unclaimed, and unused, and cleanup succeeds without cancellation, the unused source session closes +and ordinary stdout, stderr, and exit status are preserved. Metadata, reservation, cancellation, and cleanup errors +remain fail-closed. This completion publishes no remote-execution authority. +Accepted source-session journeys and remaining fault limits are recorded in the qualification checkpoint. + --- ### `mvm ssh` @@ -569,7 +813,13 @@ mvm cp vm1:/data/file.txt vm2:/data/ ### `mvm volume` -Persistent data disk management. +Persistent raw data disk management. Creation always uses `volumes/.raw`; `volume create --format` is +removed, including `--format raw`. Read-only, shareable, and writeback options are preserved. QCOW2 image import and +conversion remain supported through `mvm image`. + +Existing non-raw volume records remain available for inspection and removal, but cannot be attached, resized, or used +by VM create, start, reboot, or restore. Every selected volume must resolve successfully. Persisted VM volume IDs must +resolve exactly; a matching name or ID prefix cannot replace a missing referenced volume. #### Selectors @@ -580,11 +830,11 @@ The volume resolver tries selectors in priority order: | Command | Flags | Description | |---------|-------|-------------| -| `mvm volume create` | `NAME`, `SIZE`, `--format` | Create a new persistent volume | -| `mvm volume rm` | `[SELECTORS]...`, `-f, --force` | Remove one or more volumes | +| `mvm volume create` | `NAME`, `SIZE` | Create a new persistent raw volume | +| `mvm volume rm` | `[SELECTORS]...`, `-f, --force` | Remove volumes; `--force` permits an attachment only when its VM is stopped | | `mvm volume ls` | `--json, --long` | List all volumes | | `mvm volume inspect` | `SELECTOR`, `--json` | Show detailed information about a volume | -| `mvm volume resize` | `SELECTOR`, `SIZE` | Resize a volume | +| `mvm volume resize` | `SELECTOR`, `SIZE` | Grow a volume; an attached VM must be stopped, but detaching is unnecessary | | `mvm volume attach` | `VM_SELECTOR`, `VOLUME_SELECTOR` | Attach a volume to a VM | | `mvm volume detach` | `VM_SELECTOR`, `VOLUME_SELECTOR` | Detach a volume from a VM | @@ -594,7 +844,6 @@ The volume resolver tries selectors in priority order: |------|-------------|---------| | `NAME` | Volume name (required) | — | | `SIZE` | Volume size, e.g. `10G`, `512M` (required) | — | -| `--format FORMAT` | Disk format: `raw` or `qcow2` | `raw` | | `--read-only`, `--readonly`, `--ro` | Mount volume as read-only | writable | | `--shareable, -s` | Allow volume to be attached to multiple VMs | `false` | | `--writeback` | Use writeback cache mode | `false` | @@ -607,11 +856,13 @@ Check for and apply binary updates from GitHub Releases. | Command | Description | |---------|-------------| -| `mvm self-update` | Check + apply if newer | +| `mvm self-update` | Check + apply if newer for a user-owned artifact; the system installation requires administrator bootstrap | | `mvm self-update check` | Check only, print available version | -| `mvm self-update apply` | Force apply (even if same version). `--force` to re-install | +| `mvm self-update apply` | Apply to a user-owned artifact. `--force` re-installs the same version | -Downloads the latest release matching the host architecture, verifies SHA256 checksum, then atomically swaps the binary. +For a user-owned artifact, downloads the latest release matching the host architecture, verifies its SHA256 checksum, +and atomically swaps that artifact. Apply refuses canonical `/usr/local/bin/mvm`; install a newly downloaded trusted +artifact with `sudo host install-system`. `check` remains available from the system installation. --- @@ -681,24 +932,34 @@ The effective default is `--cloud-init-mode off` (no cloud-init). When SSH keys | **iso** | `--cloud-init-mode iso` | Generates a cloud-init ISO via `cloud-localds`, or uses a provided custom ISO path | | **off** | `--cloud-init-mode off` | Skips cloud-init entirely (default) | +The typed LoopMount VM path preserves missing-account creation, selected SSH keys, cloud-init injection, and automatic +deblob. A new non-root account uses a free UID/private GID in 1000–60000, a locked password, `/home/`, and `/bin/sh`; +existing accounts are retained. Injected seeds contain fixed `meta-data` and `user-data` leaves plus optional +`network-config`. Requests cannot select arbitrary host paths or guest commands. + +Some distro preparation is installed as fixed guest-boot work, including SSH setup, Arch package-key initialization, +and Btrfs balance. Successful host provisioning means those actions were queued, not that the guest has run them. +Typed image conversion/recovery and unprivileged archive construction are separate operations; successful +provisioning does not imply every guest boot action completed. + ### How Net Mode Works The net mode delivers cloud-init configuration via HTTP. Steps below apply to the net mode only. -1. **HTTP Server**: A temporary HTTP server is started on the host (port range 8000-9000) -2. **Firewall Rules**: Firewall rules allow the VM to reach the server (via the active `nftables` or `iptables` backend) -3. **Kernel Command Line**: The VM boots with `ds=nocloud;seedfrom=http://GATEWAY_IP:PORT/` +1. **HTTP Server**: A NoCloud batch binds the managed network gateway on fixed TCP port `8000`; custom ports are rejected. +2. **Firewall Grant**: Root derives the owned gateway service slot from the network identity. Binding must succeed before access is granted. +3. **Kernel Command Line**: The VM boots with `ds=nocloud;seedfrom=http://GATEWAY_IP:8000/` 4. **Configuration Delivery**: cloud-init fetches `meta-data`, `user-data`, and `network-config` via HTTP -5. **Automatic Cleanup**: The HTTP server stops when the VM is removed +5. **Cleanup**: Normal shutdown revokes the grant while retaining the listener socket, then releases the port. ### Net Mode Security Architecture The security model applies to the net cloud-init mode only: -- **Per-VM Isolation**: Each VM gets its own HTTP server on a unique port -- **Source-Based Firewall**: Only the VM's IP can reach its nocloud server -- **Gateway Binding**: HTTP servers bind to the bridge gateway IP, not `0.0.0.0` -- **Rule Comments**: Firewall rules are tagged with `# nocloudnet::` +- One live NoCloud batch occupies each network gateway listener. A bind collision fails; it neither adopts nor kills the existing listener. +- The typed privileged request selects only the owned network, never a custom address, port, protocol, or service. +- The firewall grant represents the fixed gateway TCP `8000` slot; it does not authenticate the current listener process. +- An interrupted process can leave that reserved-slot grant for a subsequent invocation or network teardown; successful ordinary journeys do not establish every interruption outcome. --- @@ -706,14 +967,19 @@ The security model applies to the net cloud-init mode only: | Variable | Description | Default | |----------|-------------|---------| -| `MVM_CACHE_DIR` | Override cache directory | `~/.cache/mvmctl` | +| `MVM_CACHE_DIR` | Override the managed cache directory | `~/.cache/mvmctl` | | `MVM_CONFIG_DIR` | Override config directory | `~/.config/mvmctl` | | `MVM_LOG_LEVEL` | Set log verbosity (`DEBUG`, `INFO`, `WARN`, `ERROR`) | `WARN` | -| `MVM_ASSET_MIRROR` | Local mirror directory for downloaded assets | (not set) | -| `MVM_WARM_POOL` | Warm image pool backend (`disk` for disk-backed, default is tmpfs) | (not set — tmpfs) | -| `MVM_ESCALATED` | Set by sudo wrapper to indicate privilege escalation | `1` | -| `MVM_TEMP_DIR` | Override temp directory for microVMs | `/tmp/mvmctl` | -| `MVM_SUDO_RESTART` | Set internally when re-running with sudo for host init | (not set) | +| `MVM_ASSET_MIRROR` | Local artifact mirror; a new Firecracker/Jailer install still fetches the official checksum sidecar for root verification | (not set) | +| `MVM_WARM_POOL` | Warm image pool backend (`disk` for cache-backed; otherwise under `/tmp/mvmctl-/ready`, on tmpfs only when `/tmp` is tmpfs) | (not set) | +| `MVM_TEMP_DIR` | Override the per-operator temporary work directory, warm-pool location, and official-kernel scratch (`/kernel-build`) | Managed cache `tmp/` for ordinary temporary work; official-kernel build uses embedded `build_dir` when unset or empty | + +Generic image/kernel downloads look up mirror entries by URL filename, not full origin. Configured checksums are +verified, but Firecracker CI kernel downloads have no upstream SHA-256 configured. Different CI releases can publish +different bytes under the same filename (for example, `vmlinux-6.1.155`); those entries are not interchangeable. +Use separate mirror directories for colliding origins and keep the matching artifact in each. A mirror hit without +an expected checksum does not prove origin or CI-release identity. This does not relax the official checksum +verification required for root-owned Firecracker/Jailer installation. --- @@ -721,31 +987,54 @@ The security model applies to the net cloud-init mode only: ``` ~/.cache/mvmctl/ -├── bin/ # Firecracker + jailer binaries +├── bin/ # Legacy non-launch binary workspace ├── kernels/ # vmlinux kernel images ├── images/ # Root filesystem images (.ext4, .btrfs, .img, .raw, .ext4.zst, .btrfs.zst) ├── volumes/ # Persistent disk volume files ├── vms/ # Per-VM state │ └── / # VM directories named by SHA256 hash -│ ├── rootfs.ext4 (or rootfs.btrfs, rootfs.xfs) +│ ├── rootfs.img +│ ├── cloud-init.iso # ISO mode only │ ├── firecracker.json │ ├── firecracker.log │ ├── firecracker.console.log -│ ├── firecracker.pid -│ ├── firecracker.api.socket -│ ├── firecracker.metrics -│ ├── console.sock -│ ├── console.pid -│ └── cloud-init/ +│ ├── firecracker.metrics # Metrics enabled only +│ └── cloud-init/ # Generated seed inputs ├── workflows/ # Workflow state persistence ├── nocloudnet/ # nocloud-net batch server dirs and logs -├── snapshots/ # Snapshot files (mem, vmstate, disk) -│ └── / +├── .mvm-protected/ # Root-controlled store; not caller-editable +│ └── snapshots// +│ ├── capture.json # Root-authenticated artifact record +│ ├── capture-progress.json # Interrupted-capture outcome +│ ├── rootfs.img +│ ├── kernel +│ ├── memory # Full snapshots only +│ ├── vmstate # Full snapshots only +│ └── volume-.raw # Each captured data volume, when present ├── logs/ # Application log files -├── firecracker-src/ # Firecracker git clone (for building from source) ├── mvmdb.db # SQLite database (canonical asset state) ├── mvmctl.log # Main application log ├── timing.log # Performance/timing log ├── audit.log # Rotating operation log (10MB, 3 backups) └── ... ``` + +The VM and snapshot leaf names above are compiled and are not configurable. `MVM_CACHE_DIR` still selects the managed +cache root. Caller-facing console outputs and display-only PID mirrors use +`/run/mvmctl/runtime//`; the Firecracker API and vsock sockets use the isolated execution identity's +private `/run/mvmctl/vmm-runtime//` directory, mounted at `/vmm-runtime` inside the jail. The API is +available through the authenticated root broker, not a public `api_socket_path` field. The root transaction pins +individual persistent resources; it does not mount the whole caller-owned VM directory. + +The v0.3 privileged path accepts a managed cache root on ext2/ext3/ext4, XFS, Btrfs, F2FS, bcachefs, tmpfs, or ZFS. +FUSE, remote, overlay/stacked, and automount cache roots are unsupported. The descriptor-pinned cache foundation is +implemented. Its private base-resource preparation also replaces admitted caller configuration bytes with a root-built +sealed memfd consumed by typed privileged launch and descriptor-only mounting. These paths have not been qualified. + +Canonical launch assets and jail state are outside the user cache: + +```text +/var/lib/mvmctl/ +├── binaries/// # Root-owned exact Firecracker/Jailer release pair +└── jailer/firecracker//root/ # Per-VM jail and bind mounts +``` diff --git a/docs/RELEASE.md b/docs/RELEASE.md index f346bd81..031f87f0 100644 --- a/docs/RELEASE.md +++ b/docs/RELEASE.md @@ -32,23 +32,42 @@ This document covers how to release a new version of mvmctl from start to finish Run the full CI gate: ```bash -# Compile -go build ./... - -# Vet +# Exact CI gates +go mod tidy && git diff --exit-code +test -z "$(gofmt -l .)" +test -z "$(golines --max-len=120 --no-reformat-tags --list-files \ + ./internal/ ./pkg/ ./cmd/)" +go generate ./internal/service/agent/... +git diff --exit-code go vet ./... +bash scripts/run-go-ci-tests.sh + +# Separately run the complete Jailer Go test binary as root inside an approved +# disposable local VM and retain its result before creating the release tag. -# Unit tests -go test ./... +# Build the release candidate with an explicit identity +./scripts/build.sh release --version X.Y.Z --output dist/mvm -# Build release binary for system tests -./scripts/build.sh release -cp dist/mvm ~/.local/bin/mvm +# The remaining commands mutate the outer host. Run them only on a dedicated +# clean qualification host or disposable outer VM. +sudo ./dist/mvm host install-system +sudo /usr/local/bin/mvm host init -python3 scripts/run-system-tests.py --all +MVM_ASSET_MIRROR=~/.cache/mvm-asset-mirror \ +MVM_BINARY=/usr/local/bin/mvm MVM_CANDIDATE_BINARY=./dist/mvm \ + python3 scripts/run-system-tests.py \ + --release-qualification --host-direct --rebuild \ + --candidate-version X.Y.Z --all \ + --host-reboot-task /absolute/path/to/closed-host-reboot-task ``` -All five must pass. If system tests fail, investigate before proceeding. +Complete the [named-runner reboot journey](development/HOW_TO_RUN_SYSTEM_TESTS.md#47-explicit-named-runner-host-reboot-journey) +first and supply its closed task directory for strict POST evidence. + +Every gate and every collected test must pass with zero skips/xfails/xpasses. If +system tests fail or omit a required case, investigate before proceeding. Do not +run the host-direct Tier 3 suite on a workstation with pre-existing mvmctl state. +`--host-direct` is explicit consent, not proof that cleanup is ownership-safe. See [RC_QA.md](RC_QA.md) for test environment setup, the orchestrator script reference, and tier execution details. @@ -132,7 +151,7 @@ git push origin vX.Y.Z The `release.yml` workflow runs on tag push (`v*.*.*`). It's a single pipeline: -1. **Test** — `go mod tidy`, `go vet ./...`, `go test ./...` with coverage +1. **Test** — `go mod tidy`, `go vet ./...`, and the unprivileged Go selection in `scripts/run-go-ci-tests.sh` with coverage. The complete Jailer root-test result is a local pre-tag release gate, not a CI job. 2. **Build all packages** — runs `scripts/build-packages.sh --build-binaries --version X.Y.Z` The `build-packages.sh` script produces everything in `dist/packages/`: diff --git a/docs/RESOURCE_MANAGEMENTS.md b/docs/RESOURCE_MANAGEMENTS.md index b14e5e71..f07d284b 100644 --- a/docs/RESOURCE_MANAGEMENTS.md +++ b/docs/RESOURCE_MANAGEMENTS.md @@ -122,6 +122,11 @@ SHA-256 checksum resolution differs per image type: | `vhd` | Microsoft VHD image | `qemu-img convert` (as vpc) → raw → root partition extracted | | `vhdx` | Microsoft VHDX image | `qemu-img convert` (as vhdx) → raw → root partition extracted | +These formats describe image acquisition and import. Attached VM data volumes are raw-only; +`mvm volume create` has no `--format` option, including `--format raw`. QCOW2 image import and +conversion remain supported. See [Volumes, snapshots, and recovery](RUNTIME.md#volumes-snapshots-and-recovery) +for volume and snapshot admission. + ### Template variables URL templates use `{placeholder}` syntax resolved at fetch time: @@ -248,7 +253,7 @@ kernel-firecracker: | `sha256` / `sha256_url` | Same semantics as images.yaml SHA-256. | | `config_fragments` | Paths or URLs to additional kernel config files merged on top of the base config. | | `output_name` | Base filename for the compiled or downloaded `vmlinux` binary in the kernels cache. | -| `build_dir` | Working directory for kernel compilation. Cleaned up automatically unless `--keep-build-dir` is passed. | +| `build_dir` | Working directory for kernel compilation when `MVM_TEMP_DIR` is unset or empty. An explicit override selects `/kernel-build`. Cleaned up after success unless `--keep-build-dir` is passed. | | `parallel_jobs` | `make -j` value. `null` defers to `defaults.kernel.build_jobs` (default: `nil`, meaning all available cores). | | `default_configs` | Flat map of `CONFIG_OPTION: value` entries. `y` enables the option, `n` disables it, string values set the option to a specific value (e.g. `"4"`). Passed to `scripts/config` during build. | | `features` | Named feature groups of kernel config options. Each feature has `desc` and `enforce` (a map of `CONFIG_OPTION: y` values). Activatable via `--features` on `mvm kernel pull`. Use `all` or `*` to enable every feature in the spec. | @@ -263,25 +268,38 @@ kernel-firecracker: **Location:** `internal/infra/constants.go` — `OverridableDefaults` map -This map is the single authoritative source for all built-in defaults. Hardcoded values anywhere else in the codebase are a bug. +This map is the authoritative source for user-configurable built-in defaults. Security-sensitive managed artifact +basenames are compile-time constants in the same file and intentionally do not appear in `OverridableDefaults`. ### Structure overview | Category path | Description | |---------------|-------------| -| `defaults.vm` | Default vCPU count, RAM, SSH user, boot args, LSM flags, PCI, nested virt, logging, console, vsock port | +| `defaults.vm` | Default vCPU count, RAM, SSH user, boot args, LSM flags, PCI, nested virt, logging, console, vsock port, and typed cgroup policy inputs | | `defaults.network` | Default bridge name, CIDR, NAT enabled | | `defaults.image` | Import format, remote listing limit and cache TTL | | `defaults.kernel` | Default kernel version, build jobs, remote listing limit and cache TTL | -| `defaults.firecracker` | Log filenames, socket filenames, log level, PID filenames | -| `defaults.cloudinit` | ISO name, nocloud-net port range, max retries, kill-after duration | +| `defaults.firecracker` | Firecracker log level | +| `defaults.cloudinit` | NoCloud service kill-after duration; the gateway service port is fixed at TCP 8000 | | `defaults.binary` | Remote version limit | | `defaults.volume` | Cache type (default: `Unsafe`) | -| `settings` | General settings: guestfs_enabled, firewall_backend | -| `settings.firewall` | iptables_xtcomment flag | +| `settings` | General settings: guestfs_enabled; nftables is the only firewall backend | | `settings.vm` | log_lines, log_follow, max_vms, ssh_timeout_sec | | `cli` | Listing style (default: `short`) | +### `defaults.vm` cgroup policy + +These settings derive the persisted envelope when a VM is created. Existing and snapshot-restored VMs retain their persisted limits. + +| Key | Default | Description | +|-----|---------|-------------| +| `cgroup_vmm_headroom_mib` | `128` | Host-memory allowance added to guest-visible RAM for `memory.high`; the VM leaf's `memory.max` is `max`, not a configurable hard ceiling | +| `cgroup_cpu_weight` | `100` | cgroup-v2 CPU weight; accepted range is 1–10000 | +| `cgroup_pids_max` | `256` | Maximum processes in the VM's cgroup | +| `cgroup_swap_max_bytes` | `0` | Maximum cgroup swap usage in bytes; zero disables swap | + +CPU quota is the VM's vCPU count multiplied by a fixed 100000 µs period. Raw cgroup file names and arbitrary controller values are not configuration interfaces. + ### `defaults.kernel` | Key | Default | Description | diff --git a/docs/RUNTIME.md b/docs/RUNTIME.md index 9187812a..320eb1fe 100644 --- a/docs/RUNTIME.md +++ b/docs/RUNTIME.md @@ -1,584 +1,179 @@ # mvmctl Runtime Architecture -This document describes the runtime backend systems that execute provisioning, -cloud-init, firewall, and service operations during VM creation and management. -Contributors and advanced users should read this to understand which backend -runs when, why multiple backends exist for the same task, and how a `mvm vm create` -request flows through the system. - -## Table of Contents - -- [1. Provisioning Backends](#1-provisioning-backends) - - [1.1 Loop-Mount Backend (Default)](#11-loop-mount-backend-default) - - [1.2 GuestFS Backend (Opt-in)](#12-guestfs-backend-opt-in) - - [1.3 Backend Selection](#13-backend-selection) - - [1.4 Shared Provisioner Content](#14-shared-provisioner-content) -- [2. Cloud-Init Modes](#2-cloud-init-modes) - - [Inject Mode](#inject-mode) - - [Net Mode](#net-mode) - - [ISO Mode](#iso-mode) - - [Off Mode](#off-mode) -- [3. Service Backends](#3-service-backends) - - [3.1 Service Architecture](#31-service-architecture) - - [3.2 Console Relay](#32-console-relay) - - [3.3 NoCloud Server](#33-nocloud-server) - - [3.4 Loop-Mount Provisioner Service](#34-loop-mount-provisioner-service) - - [3.5 Service Lifecycle](#35-service-lifecycle) -- [4. Firewall Backends](#4-firewall-backends) -- [5. Selection Guide](#5-selection-guide) -- [6. Performance Comparison](#6-performance-comparison) - - [End-to-End VM Creation (create_s)](#end-to-end-vm-creation-create_s) - - [Notes](#notes) -- [7. Binary Build](#7-binary-build) - - [Service Invocation](#service-invocation) - - [Sudoers](#sudoers) -- [8. Architecture Diagram](#8-architecture-diagram) - -## 1. Provisioning Backends - -A **provisioning backend** modifies a VM root filesystem before first boot. Every -backend supports the same set of operations: resizing, SSH key injection, hostname -setup, DNS configuration, cloud-init inject, OS detection, package cache cleanup -("deblob"), fstab fix (PARTUUID → /dev/vda), and arbitrary file writes. - -Two backends implement this interface — **Loop-Mount** and **GuestFS**. They are -mutually exclusive: the provisioner type is selected once at startup by reading -the `guestfs_enabled` setting. Loop-Mount is the default (~2.3s average); GuestFS -is the opt-in alternative (~8.4s+). - -### 1.1 Loop-Mount Backend (Default) - -The loop-mount backend mounts the VM's root filesystem image via a loop device, -operates on it directly with host tools, and unmounts when done. It is the primary -backend because it requires no additional system packages beyond standard Linux -utilities (`losetup`, `mount`, `umount`, `blkid`, `resize2fs`, `chroot`). - -**Flow:** - -``` -pkg/api/vm.go: VMCreate() - → provisioner.NewBackend(opts) # Creates LoopMountBackend - → backend.Resize() / SetHostname() / ... # Queues operations - → backend.Run() - → runWireOp() → system.DefaultRunner.Run(["sudo", "mvm", "run", "provision"]) - → JSON ops on stdin → losetup/mount/chroot → JSON results on stdout -``` - -The API layer (`pkg/api/vm.go`) calls `provisioner.NewBackend()` with `BackendOpts` -that include the rootfs image path and filesystem type. Each `backend.SetHostname()`, -`backend.SetupSSH()`, etc. call queues a provisioning operation. When `backend.Run()` -is called, all queued operations are serialized to JSON, piped to an `mvm run provision` -subprocess via stdin, and results are read from stdout. - -**Execution steps inside the subprocess:** - -1. `truncate` the image file (for grow resize operations) -2. `losetup -f -P --show ` — attach loop device with partition scanning -3. Detect root partition (first partition, second, largest Linux filesystem, or raw device) -4. Detect filesystem type via `blkid` (ext4 fallback) -5. `mount ` -6. Write all queued files (base64-decoded content with correct mode/uid/gid) -7. Copy directories from host to mount point (recursive) -8. `chroot ` for each shell command (tries multiple shell paths, 60s timeout) -9. Post-mount resize: `e2fsck` + `resize2fs` for ext4; `btrfs filesystem resize` for btrfs -10. `umount` + `losetup -d` (always via `finally`) - -**Key files:** - -| File | Purpose | -|------|---------| -| `internal/service/loopmount/provisioner.go` | Provisioning engine — losetup, mount, chroot, resize | -| `internal/service/loopmount/entry.go` | Config struct + `Run()` entry point | -| `internal/service/loopmount/spawn.go` | `Spawn()` via `system.SpawnService()` | -| `internal/service/loopmount/wire.go` | JSON wire protocol types | -| `internal/lib/provisioner/backend.go` | `Backend` interface | -| `internal/infra/provcontent/content.go` | Shared provisioning content builders | - -**Boundary:** The loop-mount backend requires passwordless sudo (set up by `mvm host init`). -It does NOT support images with non-standard partition layouts or damaged filesystems -that `e2fsck` cannot repair. - -### 1.2 GuestFS Backend (Opt-in) - -The GuestFS backend uses the `guestfish` CLI tool (from `libguestfs`) as a subprocess -to manipulate the root filesystem through a QEMU appliance. It exists for environments -where loop-mount is impractical: systems without sudo access, or images requiring -advanced filesystem features. - -**Flow:** - -``` -pkg/api/vm.go: VMCreate() - → provisioner.NewBackend(opts) - → guestfs.EnsureAppliance(cacheDir) # Build or verify cached appliance - → NewGuestfsBackend(rootfsPath, rootUID, rootGID, userUID, userGID) - → guestfish CLI → QEMU appliance microVM -``` - -The guestfs backend launches a small QEMU microVM that mounts the target filesystem -and exposes `libguestfs` operations. Operations are performed via `guestfish` CLI -subprocess invocations, not through a JSON subprocess protocol. - -**Key differences from loop-mount:** -- OS detection reads `/etc/os-release` via guestfs `ReadFile()` instead of `chroot` -- SSH setup detects the init system: systemd, OpenRC, or sysvinit -- Supports user creation via `useradd` with sudoers drop-in -- Cloud-init inject uses guestfish `upload` instead of directory copy -- Shrink uses `ZeroFreeSpace()` + `e2fsck` + `Resize2fsSize()` with safety margin - -**Appliance lifecycle:** -- Built once by `guestfs.BuildAppliance()` via `libguestfs-make-fixed-appliance` -- Cached at `~/.cache/mvmctl/appliance/` (kernel, initrd, root files) -- `guestfs.KernelDetector.FindBestKernel()` or `FindBestKernel()` selects a kernel with virtio drivers -- Stale state cleanup handles orphaned QEMU processes, lock files, daemon sockets -- Pruned by `mvm cache prune misc` calling `guestfs.PruneAppliance()` - -**Key files:** - -| File | Purpose | -|------|---------| -| `internal/lib/provisioner/guestfs/provisioner.go` | Rootfs operations via guestfs API | -| `internal/lib/provisioner/guestfs/base.go` | Handle creation, mount, partition extraction | -| `internal/lib/provisioner/guestfs/utils.go` | Appliance building, stale state cleanup | -| `internal/lib/provisioner/guestfs/kernel_detector.go` | Suitable appliance kernel selection | - -**Boundary:** GuestFS requires `libguestfs`, `supermin`, and `qemu` system packages. -The appliance build may fail or hang if no kernel with `CONFIG_VIRTIO_PCI` is available. -This backend is approximately 3-4x slower than loop-mount. - -### 1.3 Backend Selection - -The provisioner type is resolved once at application startup in `api.NewOperation()`: - -```go -provisionerType := provisioner.ProvisionerLoopMount -guestfsEnabled, _ := s.Config.GetBool(ctx, "settings", "guestfs_enabled") -if guestfsEnabled { - provisionerType = provisioner.ProvisionerGuestFS -} -``` - -All callers reference `op.ProvisionerType`. The `provisioner.NewBackend()` factory in -`internal/lib/provisioner/backend.go` dispatches to the correct implementation based on -this type. Resolution happens once — there is no backend fallback at runtime. - -### 1.4 Shared Provisioner Content - -Both backends consume the same operation builders from `internal/infra/provcontent/content.go`: - -| Builder | Purpose | -|---------|---------| -| `BuildHostnameOps(hostname)` | `/etc/hostname` + `/etc/hosts` | -| `BuildDNSOps(dnsServer)` | `/etc/resolv.conf` nameserver entry | -| `BuildSSHOps(user, sshPubkeys)` | SSH authorized keys and user setup | -| `BuildCloudInitDisableOps()` | Datasource blocking + service masking | -| `BuildCloudInitInjectOps(cloudInitDir)` | Copy cloud-init seed directory | -| `BuildResizeOps(targetSizeBytes)` | Grow filesystem | -| `BuildShrinkOps(limitBytes)` | Shrink filesystem (pass `0` for minimum) | -| `BuildDeblobOps(osType)` | OS cache cleanup (apt, yum, apk, pacman) | -| `BuildFixFstabOps()` | PARTUUID → /dev/vda migration | - -The builders produce a generic `Operation` list. Each backend consumes the same list -but executes it differently — loop-mount via JSON subprocess, guestfs via direct API calls. - ---- - -## 2. Cloud-Init Modes - -**Cloud-init** is the standard mechanism for provisioning cloud images on first boot. -mvmctl supports four delivery modes, each with different trade-offs: - -| Mode | Flag | Mechanism | Latency | Use Case | -|------|------|-----------|---------|----------| -| **inject** | `--cloud-init-mode inject` | Direct rootfs injection via provisioner backend | ~200ms | Primary mode — no runtime dependencies | -| **net** | `--cloud-init-mode net` | HTTP server (nocloud-net datasource) | ~50ms | Dynamic config, no rootfs modification | -| **iso** | `--cloud-init-mode iso` | Seed ISO via `cloud-localds` | ~500ms | Legacy images requiring ISO datasource | -| **off** | `--cloud-init-mode off` | Datasources blocked, services masked | 0ms | No provisioning needed | - -### Inject Mode - -``` -VMCreate() - → cloudinit.Provisioner.Provision() - → Generate user-data, meta-data, network-config - → manager.WriteConfigFiles() - → provisioner.InjectCloudInit(cloudInitDir) - → Backend copies files into rootfs - → VM boots with files baked in -``` - -Files are written directly into the root filesystem before first boot. The files -persist for the lifetime of the VM and survive reboots. This is the default mode -because it requires no network server and no ISO generation. - -### Net Mode - -``` -VMCreate() - → cloudinit.Provisioner.Provision() - → Generate configs - → nocloudnet.Spawn(ctx, cfg) - → Find free port (8000-9000 range) - → Spawn HTTP server via system.SpawnService() - → Bind to bridge gateway IP only - → Firewall rule in MVM-NOCLOUDNET-INPUT chain - → VM boots with ds=nocloud;seedfrom=http://:/ - → Server runs until VM removal -``` - -The HTTP server responds to cloud-init metadata, user-data, and network-config -requests. It binds exclusively to the bridge gateway IP — never to `0.0.0.0` — -to prevent external network access. A firewall rule in the active backend -(nftables by default) allows the VM's source IP to reach the server. - -### ISO Mode - -``` -VMCreate() - → cloudinit.Provisioner.Provision() - → Generate configs - → manager.CreateSeedISO() via cloud-localds subprocess - → ISO attached as secondary Firecracker drive - → VM boots, cloud-init reads from ISO -``` - -ISO mode generates a cloud-init seed ISO using the `cloud-localds` tool and -attaches it as a secondary block device to the Firecracker VM. This mode is -necessary for images that do not support the nocloud-net datasource. - -### Off Mode - -``` -VMCreate() - → cloudinit.Provisioner.Provision() - → Returns CloudInitResult{Mode: CloudInitModeOFF} - → provisioner.DisableCloudInit() - → Writes datasource block file + masks services -``` - -"Off" mode physically disables cloud-init inside the rootfs by writing datasource -block files and masking cloud-init systemd services. The VM boots without any -provisioning step. - -**Key files:** - -| File | Purpose | -|------|---------| -| `internal/core/cloudinit/provisioner.go` | Config generation, mode dispatch | -| `internal/core/cloudinit/manager.go` | Config file writing, seed ISO creation | -| `internal/core/cloudinit/config.go` | Provisioning parameters | -| `internal/service/nocloudnet/entry.go` | HTTP server lifecycle | -| `internal/service/nocloudnet/handler.go` | Metadata request handler | - ---- - -## 3. Service Backends - -Service backends are long-running subprocesses compiled into the same `mvm` binary. -Each is invoked via `mvm run ` and spawned in the background by the core -layer via `system.SpawnService()`. The `agent` is a special case — it runs -inside the guest VM, not on the host. - -### 3.1 Service Architecture - -Three foreground services and one embedded guest agent share the same binary. -No separate service binaries, no symlinks, no extraction step. - -| Service | Entry Point | Runs As | Purpose | -|---------|-------------|---------|---------| -| `mvm run console relay` | `console.Run(ctx, cfg)` | user | PTY-to-socket relay for serial console | -| `mvm run nocloudnet serve` | `nocloudnet.Run(ctx, cfg)` | user | HTTP server for cloud-init nocloud-net | -| `mvm run provision` | `loopmount.Run(ctx, cfg)` | **root** (sudo) | Loop-mount rootfs provisioning | -| `agent/` (embedded) | Guest agent binary | root (in-VM) | Command execution and file transfer inside the guest | - -The vsock agent is cross-compiled at build time, zstd-compressed, embedded via -`//go:embed`, and injected into the VM at runtime through the vsock device. This -avoids the need for SSH or any guest-side package installation for `mvm exec`, -`mvm cp`, and console operations. - -Each service follows a consistent pattern: a **`Config`** struct holds all parameters, -**`Run(ctx, cfg)`** runs the service in the foreground (blocking), and -**`Spawn(ctx, cfg, ...)`** launches it as a background subprocess. - -### 3.2 Console Relay - -The console relay converts a Firecracker VM's serial console into an interactive -terminal session accessible through `mvm console`. It exists because Firecracker -exposes the serial console as a PTY pair, and the relay multiplexes that PTY -between a Unix socket (for CLI attachment) and a log file. - -``` -console.Controller → console.Spawn(ctx, cfg) → system.SpawnService("mvm", "run", "console", "relay") - │ - PTY master fd ← → Unix socket ← → Client (CLI) - │ - console.log -``` - -The relay entry point is `mvm run console relay`. The `Run()` function in -`entry.go` opens the PTY master file descriptor (inherited as FD 3), sets up -a Unix listener, and enters the I/O loop. The I/O loop reads from the PTY in -a goroutine (fed to a channel), accepts Unix socket connections from the CLI, -and uses Go `select` to multiplex data between the PTY, the connected client, -and the log file concurrently. - -- **Signal handling:** SIGTERM/SIGINT trigger graceful shutdown with PID file and socket cleanup -- **PID file:** `$MVM_CACHE_DIR/vms//console.pid` -- **Socket:** `$MVM_CACHE_DIR/vms//console.sock` -- **Log:** `$MVM_CACHE_DIR/vms//firecracker.console.log` - -**Key files:** - -| File | Purpose | -|------|---------| -| `internal/service/console/entry.go` | Config struct, `Run()`, and the `runRelayIO` I/O loop | -| `internal/service/console/spawn.go` | `Spawn()` via `system.SpawnService()` | -| `internal/service/console/relay.go` | `Relay` struct — PID management and lifecycle | -| `internal/service/console/client.go` | `RelayClient` — CLI-side console attachment | - -**Boundary:** The relay supports one client connection at a time. If a client is -already attached, new connection attempts are refused. The detach sequence is -Ctrl+X followed by `d`. - -### 3.3 NoCloud Server - -The nocloud-net HTTP server delivers cloud-init metadata to VMs over HTTP. -It is the runtime component for the "net" cloud-init mode (see §2). The server -is started per-VM and lives for the VM's lifetime. - -``` -cloudinit.Provisioner → nocloudnet.Spawn(ctx, cfg) → system.SpawnService("mvm", "run", "nocloudnet") - │ - HTTPServer(bind=gateway_ip:port) ← → VM guest (HTTP) - │ - serves: meta-data - user-data - network-config -``` - -- **Port range:** 8000–9000, auto-allocated via `socket.bind()` probe -- **Security:** Binds to bridge gateway IP only (never `0.0.0.0`), firewall rule in `MVM-NOCLOUDNET-INPUT` chain -- **PID file:** `$MVM_CACHE_DIR/vms//nocloud-server.pid` -- **Headers:** Cache-disabling headers (`Cache-Control: no-cache, no-store, must-revalidate`) - -**Key files:** - -| File | Purpose | -|------|---------| -| `internal/service/nocloudnet/entry.go` | Config struct + `Run()` entry point | -| `internal/service/nocloudnet/spawn.go` | `Spawn()` via `system.SpawnService()` | -| `internal/service/nocloudnet/handler.go` | HTTP request handler | - -**Boundary:** The server binds to the bridge gateway IP, not to `0.0.0.0`. -VMs on a different subnet cannot reach it. This is intentional — cloud-init -data is per-VM and must not be accessible from outside the VM's network. - -### 3.4 Loop-Mount Provisioner Service - -The loop-mount provisioner is the execution engine for filesystem operations. -It is spawned as a root subprocess by the `LoopMountBackend` in the API layer. - -``` -API layer → provisioner.NewBackend() → LoopMountBackend - → backend.Resize() / SetHostname() / ... (queues operations) - → backend.Run() → runWireOp() → DefaultRunner.Run(["sudo", "mvm", "run", "provision"]) - │ - JSON ops stdin → losetup/mount/chroot → JSON results stdout -``` - -- **Communication:** JSON operation list on stdin, JSON results on stdout -- **Timeout:** 60 seconds per chroot command (in `provisioner.go`) -- **Sudo:** Passwordless sudo via `/etc/sudoers.d/mvm` drop-in -- **Speed:** ~2.3s average end-to-end - -**Key files:** - -| File | Purpose | -|------|---------| -| `internal/service/loopmount/provisioner.go` | Provisioning engine — losetup, mount, chroot | -| `internal/service/loopmount/entry.go` | Config struct + `Run()` entry point | -| `internal/service/loopmount/spawn.go` | `Spawn()` via `system.SpawnService()` | -| `internal/service/loopmount/wire.go` | JSON wire protocol types | - -**Boundary:** Requires root. The subprocess runs with full privileges and performs -raw block device operations. The JSON protocol ensures no state leaks between -invocations. - -### 3.5 Service Lifecycle - -| Phase | Action | Component | -|-------|--------|-----------| -| **Init** | Create sudoers drop-in, firewall chains, cache dirs | `host.Service.Init()` | -| **Create VM** | Provision rootfs via loop-mount or guestfs | `backend.Run()` | -| **Create VM** | Start NoCloud server (net mode) or inject cloud-init (inject mode) | `nocloudnet.Spawn()` / `backend.InjectCloudInit()` | -| **Create VM** | Start console relay (when `--console` is set) | `console.Spawn()` | -| **Remove VM** | Stop console relay + NoCloud server + clean firewall rules | `console.Stop()`, `nocloudnet.Stop()` | -| **Cache prune** | Clean up stale PID files + orphan processes | `cache.Service.Prune()` | - ---- - -## 4. Firewall Backends - -Firewall backends manage NAT rules, forwarding rules, and per-VM access control -for nocloud-net servers. Two backends are available, selected by the -`firewall_backend` setting in the database (default: `nftables`). - -A **`FirewallTracker`** in `internal/lib/firewall/tracker.go` provides a unified -interface for both backends. It dispatches all operations to the active backend -via the `Tracker` interface. The tracker is created once at startup in -`api.NewOperation()` with a default nftables tracker. After `mvm host init` -resolves the actual backend setting, it replaces the tracker via -`SetFirewallTracker()`. - -| Backend | Default | Files | -|---------|---------|-------| -| **nftables** | Yes (`firewall_backend: nftables`) | `tracker.go`, `nftables.go`, `nftables_repository.go` | -| **iptables** | Opt-in (`firewall_backend: iptables`) | `tracker.go`, `iptables.go`, `iptables_repository.go` | - -Both backends manage the same three chain types: `MVM-FORWARD` (ip filter), -`MVM-POSTROUTING` (ip nat), and `MVM-NOCLOUDNET-INPUT` (ip filter). The nftables -backend adds jump rules at position 0 of the built-in chains, using non-hook -chains so that `accept` verdicts are terminal within the mvm table — matching -the behavior users expect from iptables. - -**Why two backends?** Different Linux distributions ship with different firewall -defaults. Some use nftables natively; others use iptables or iptables-legacy. -mvmctl supports both to avoid conflicts with the host's existing firewall -configuration, particularly when Docker or other container runtimes are present. - ---- - -## 5. Selection Guide - -Each backend choice represents a trade-off between speed, dependencies, and -capability. This guide maps common scenarios to the recommended configuration. - -| Scenario | Recommended Backend | Rationale | -|----------|-------------------|-----------| -| Normal VM creation | Loop-Mount + inject cloud-init | Fastest path (~2.3s creation) | -| No sudo available | GuestFS provisioning | Uses `guestfs_enabled` opt-in | -| Minimal VM, no customization | Off mode cloud-init | Fastest boot, no provisioning | -| Dynamic cloud-init config | Net mode cloud-init | Network-served, no rootfs modification | -| Legacy ISO requirement | ISO mode cloud-init | Pre-built ISO via `cloud-localds` | -| Console access | Console Relay | Opt-in via `--console` (disabled by default) | -| Image optimization | Loop-Mount partition extraction | ~500ms vs ~2000ms guestfs | - ---- - -## 6. Performance Comparison - -Benchmark data from `benchmarks/results.json` (2026-06-16, sequential run, 5s threshold). - -Measured via `mvm exec -- echo ok` inside the VM — replaces the earlier -SSH-based probe. All images use the loop-mount provisioning backend (the default). - -### End-to-End VM Creation (create_s) - -| Image | create_s | total_s | Threshold | -|-------|----------|---------|-----------| -| alpine | 1.4s | 2.0s | 5s ✅ | -| ubuntu:24.04 | 2.0s | 2.4s | 5s ✅ | -| ubuntu-minimal:24.04 | 2.0s | 2.4s | 5s ✅ | -| archlinux | 2.4s | 3.9s | 5s ✅ | -| debian:12 | 4.3s | 4.6s | 5s ✅ | -| firecracker:v1.15 | 1.9s | 2.1s | 5s ✅ | -| **Average** | **2.3s** | **2.9s** | **100% pass** | - -### Notes - -- All six images passed within the 5s threshold in sequential mode. -- Parallel benchmark execution (use `--no-parallel` in `benchmarks/boot_time.py`) - avoids I/O contention on the host for resource-heavy images (debian, archlinux). -- The probe now uses `echo ok` via `mvm exec` (vsock guest agent) - instead of SSH — this measures end-to-end VM readiness (vsock agent responding). -- **Console relay startup**: ~10ms (negligible, no separate benchmark) -- **NoCloud server startup**: ~50ms (negligible, no separate benchmark) - ---- - -## 7. Binary Build - -The `mvm` binary is a standard Go binary that includes all services compiled in: - -```bash -go build -o dist/mvm ./cmd/mvm -``` - -The three `mvm run` services (console relay, nocloud-net server, loopmount provisioner) plus the vsock guest agent binary are all compiled into the same binary. No separate service binaries, no symlinks, no extraction step. - -### Service Invocation - -Services are invoked via `mvm run `: - -```bash -mvm run console relay # Start console relay -mvm run nocloudnet serve # Start nocloud-net HTTP server -mvm run provision # Start loopmount provisioner (requires sudo) -``` - -The CLI layer dispatches `mvm run ` to the appropriate service entry point. -Services are spawned in the background by the core layer via `system.SpawnService()`. - -### Sudoers - -Only `mvm run provision` requires passwordless sudo. Managed by `mvm host init` which -creates a drop-in at `/etc/sudoers.d/mvm` granting the `mvm` group passwordless sudo. - ---- - -## 8. Architecture Diagram - -``` -┌─────────────────────────────────────────────────────────────────────────────┐ -│ CLI Layer (Cobra) │ -│ mvm vm create my-vm --cloud-init-mode inject │ -└───────────────────────────┬─────────────────────────────────────────────────┘ - │ - ▼ -┌─────────────────────────────────────────────────────────────────────────────┐ -│ API Layer (pkg/api) │ -│ op.VMCreate(ctx, input) │ -│ ├── cloudinit.Provisioner.Provision() (prepare configs) │ -│ ├── provisioner.NewBackend(opts) (select backend at startup) │ -│ │ ├── LoopMount → backend.Resize()/SetHostname()/... → backend.Run() │ -│ │ │ └── runWireOp() → DefaultRunner.Run(["mvm","run","provision"]) │ -│ │ │ → losetup → mount → write → chroot → resize → umount │ -│ │ └── GuestFS → backend.Resize()/SetHostname()/... → backend.Run() │ -│ │ → guestfs.Backend → libguestfs (QEMU appliance) │ -│ ├── console.Spawn() (if --console) │ -│ └── nocloudnet.Spawn() (if net mode) │ -└───────────────────────────┬─────────────────────────────────────────────────┘ - │ - ▼ -┌─────────────────────────────────────────────────────────────────────────────┐ -│ Core Layer (internal/core) │ -│ │ -│ core/vm/ Controller, Service, Repository, SQLite, Resolver │ -│ core/network/ Controller, Service, Repository, SQLite, Resolver │ -│ core/volume/ Controller, Service, Repository, SQLite, Resolver │ -│ core/cloudinit/ Provisioner, Manager, Config │ -│ core/image/ Provisioner, Service, Repository, SQLite, Resolver │ -│ core/kernel/ Controller, Service, Repository, SQLite, Resolver │ -│ core/host/ Controller, Service, Repository, SQLite, Detector, Probe │ -│ core/config/ Service, Repository, SQLite, Constraints │ -│ core/console/ Controller │ -│ core/logs/ Controller, Service │ -│ core/cache/ Service │ -│ core/ssh/ Service │ -│ core/binary/ Controller, Service, Repository, SQLite, Resolver │ -│ core/key/ Controller, Service, Repository, SQLite, Resolver │ -│ core/vsock/ Client, Service, Repository, SQLite, Resolver │ -│ core/snapshot/ Repository, SQLite, Resolver │ -└───────────────────────────┬─────────────────────────────────────────────────┘ - │ - ▼ -┌─────────────────────────────────────────────────────────────────────────────┐ -│ Service Layer (internal/service) │ -│ │ -│ All compiled into the single mvm binary │ -│ ├── mvm run console relay → console relay (PTY proxy) │ -│ ├── mvm run nocloudnet serve → nocloud-net HTTP server │ -│ ├── mvm run provision → loopmount provisioner (requires sudo) │ -│ └── agent → embedded guest agent (injected into VM) │ -└─────────────────────────────────────────────────────────────────────────────┘ -``` +This document describes the v0.3.0 runtime design for operators and contributors. The +[candidate-bound qualification checkpoint](../tests/system/COVERAGE_MATRIX.md#v030-release-security-qualification-overlay) +records the accepted registered-node results and remaining fault-coverage limits. v0.3.0 remains unreleased; +CI passed; aarch64 runtime qualification and final release approval remain separate. See [RC_QA.md](RC_QA.md) for release gates and +[CONTEXT.md](../CONTEXT.md) for package and domain ownership. + +v0.3.0 requires a clean installation and fresh managed state. It does not adopt or migrate old user, +runtime, or protected-storage layouts. + +## One binary, two authority levels + +The installed `/usr/local/bin/mvm` is one root-owned executable. An operator runs its public CLI without `sudo`. +`host install-system`, `host init`, and `host reset` are separate administrator operations; `host init` enrolls the +operator in the `mvm` group. The managed sudoers entry grants that group only the versioned, closed privileged entry +point of the root-owned binary. It does not grant the public CLI, a general shell, `ip`, `nft`, or raw host tools. + +Host initialization supports KVM built directly into a kernel without loadable-module support. If the first +`/proc/modules` read reports that the inventory is absent, setup accepts the existing accessible `/dev/kvm` without +recording module loads. Missing or inaccessible KVM, cancellation and other inventory-read errors still fail setup; +the ordinary loadable-module path is unchanged. This does not make an incomplete administrator transaction enrolled. + +An ordinary request follows `internal/cli/` → `pkg/api/` → the owning `internal/core/` domain. Public inputs are +validated in `pkg/api/inputs/`; `pkg/api/` coordinates multiple domains. When an operation requires root, its typed +client calls the early authenticated dispatcher in `internal/service/privileged/`. The receiver independently checks +caller authorization, resource identity, bounds, and effect-specific invariants. It does not accept a generic command +or arbitrary host path. Jailer, LoopMount, network, firewall, and snapshot effects have their own closed handlers. + +The public `mvm run provision` and `mvm run jailer` commands were removed. Ordinary-user console and NoCloud +subprocesses still use the same installed binary. The embedded guest agent runs *inside* a VM for vsock execution and +file transfer; it is not a second host release artifact. See [ADR-0016](adr/0016-root-owned-single-binary-privileged-dispatch.md) +and [ADR-0018](adr/0018-isolated-vm-execution-and-snapshot-authority.md) for the trust boundary. + +## State and process layout + +| Location | Owner and purpose | +|---|---| +| `/var/lib/mvmctl` | Durable root authority: installation, trusted release, host admission, VM and network records, and Jailer state. | +| `/var/lib/mvmctl/vm-rootfs-export-data/` | Root-controlled independent disk copies for VM-to-image import, with operator read access and typed cleanup. Requires space on this filesystem as well as the managed cache; see [VM image import](REFERENCES.md#mvm-image). | +| `/run/mvmctl/authority` | Root-only transient locks and handshakes; not durable truth. | +| `/run/mvmctl/runtime//` | Caller-facing console endpoints and display-only PID mirrors. | +| `/run/mvmctl/vmm-runtime//` | Private Firecracker API and vsock sockets owned by the isolated VM execution identity; mounted at `/vmm-runtime` inside the jail. | +| `MVM_CACHE_DIR` (normally `~/.cache/mvmctl`) | SQLite settings/state projection, original images, managed disk inputs, and output artifacts. Root-protected active backing and snapshots are pinned under the selected cache but are not made operator-writable. | +| `MVM_CONFIG_DIR` (normally `~/.config/mvmctl`) | Operator configuration files, including SSH keys under `keys/`; separate from the managed cache and root authority. | + +The operator's SQLite rows and PID mirrors are not root authorization. Root effects pin the managed cache and their +resources by identity; a renamed cache, replaced file, stale process, or conflicting managed topology must not be +silently adopted. Listing and inspection use bounded root observation to distinguish live VMs from stale recorded +PIDs. A reboot does not imply the old Firecracker process survived. Cleanup and relaunch must reconcile retained +records before effectful reuse, and ambiguous state fails closed rather than being guessed away. + +Permanent snapshot locks and volume-source removal receipts live in the root anchor outside the managed cache. +Snapshot artifacts and temporary raw-volume capture stay on the selected cache filesystem. Typed cache cleanup +can retire admitted terminal data branches while preserving those permanent authority records; it refuses held +operations, retained backing or restore references, and unknown or old layouts. See +[ADR-0023](adr/0023-protected-branch-authority-layout.md). + +## VM launch and resource control + +`mvm bin pull` installs an exact matching Firecracker/Jailer release in root-controlled storage. For a new install, a +local `MVM_ASSET_MIRROR` can supply the archive, but the root receiver still obtains and checks the official release +checksum sidecar; a local archive alone is not a trust assertion. The approved v0.3.0 launch contract requires the +exact Firecracker/Jailer **1.17.0** pair, the latest stable upstream release selected for this dependency update. +The accepted candidate passed the selected hotplug and snapshot journeys. Older audited pairs may be managed as artifacts +but cannot launch VMs. See the [1.17.0 release notes](https://github.com/firecracker-microvm/firecracker/releases/tag/v1.17.0). + +Release updates select the latest stable official matching pair, review its complete archive and runtime contract, +and qualify the affected behavior before release. The deployed contract remains an exact version pin; it does not +automatically advance with upstream releases or admit an unreviewed archive profile or minimum-version range. + +The earlier 1.16.2 update fixed the upstream 1.16.1 pause/resume-vsock defect and introduced a snapshot format +incompatible in both directions with 1.16.0/1.16.1; see the +[1.16.2 release notes](https://github.com/firecracker-microvm/firecracker/releases/tag/v1.16.2). +Create fresh 1.17.0 snapshots and prove restore with the matching 1.17.0 pair. Earlier snapshot results do not qualify +this pair, cross-version compatibility is not asserted, and mvmctl neither converts state nor rewrites captured +release identities. + +Jailer is the only VM launcher. The authenticated root launch path prepares a fresh, pinned jail tree, admits the +selected kernel and disk resources, installs an isolated per-VM execution identity, and brokers the private +Firecracker API. Firecracker does not run as the operator, and the operator cannot open its control socket directly. +The launch path checks its process identity, cgroup membership, resource mounts, and network placement. There is no +direct-Firecracker fallback. + +The VM leaf uses cgroup v2 for CPU, memory, swap, and PID policy. `memory.high` is guest RAM plus the configurable +VMM headroom; `memory.max` is `max`. This is pressure/reclaim control, **not a hard memory ceiling or an OOM-proof +guarantee**. A host or ancestor limit, host-wide pressure, or guest OOM can still terminate processes. `mvm vm inspect` +exposes requested and observed cgroup values and usage through root-brokered observation. See +[ADR-0019](adr/0019-pressure-only-vm-memory-control.md). + +## Root filesystem provisioning and cloud-init + +LoopMount is the default rootfs provisioner. It queues named operations through +`internal/lib/provisioner/loopmount/` and executes them via the closed, root-authorized LoopMount receiver. The +receiver works on admitted managed backing; it does not run a public root service with caller-supplied shell commands. +The typed operations include filesystem growth, hostname/DNS/SSH account setup, cloud-init injection, and automatic +deblob. The optional `guestfs_enabled` backend uses `guestfish`; selecting it does not bypass administrator host setup +or the VM launch authority. See [DEPENDENCIES.md](DEPENDENCIES.md) for required host tools. + +Cloud-init delivery has four modes. `inject` writes seed data into the admitted rootfs before boot. `net` serves +NoCloud metadata through an ordinary-user service bound to the managed network's root-derived gateway on fixed TCP +**8000**; a custom port or bind collision is refused. `iso` creates a seed ISO using `cloud-localds` and requires +that optional package. `off` disables delivery and independently suppresses the supported guest cloud-init services, +including when `--skip-deblob` is set. LoopMount writes the fixed disabled marker and datasource configuration, +masks the four supported systemd units, and removes only the fixed cloud-init OpenRC activation links, including +Alpine's boot-local and default hotplug services. It does not change unrelated networking, service or cache settings. +A networkless VM cannot use `net`; it has no guest IP or +MAC and uses vsock rather than SSH for guest access. The applicable default and input conflicts are documented in +[REFERENCES.md](REFERENCES.md#mvm-vm). + +## Networking, policy, and guest execution + +Every managed networked VM gets its own root-created network namespace. A networkless VM gets an isolated +loopback-only placement. Managed same-network and cross-network VM traffic starts denied. The root firewall adapter +uses `github.com/google/nftables` to compile and submit fixed nftables generations to the kernel. It does not execute +or parse the `nft` CLI, use iptables, or offer a backend switch. Managed networking needs kernel nftables `inet` and +`bridge` support, connection tracking, and NAT capabilities; installing a command-line tool cannot provide missing +kernel support. Detected foreign or changed managed firewall state is refused rather than overwritten. + +There are two public policy families: + +| Policy | Grants | Does not grant | +|---|---|---| +| `mvm policy traffic` | Exact source network or source VM to destination VM TCP/UDP ports, after resolving current managed identities. | Vsock command execution or host commands. | +| `mvm policy exec` | Directional source-VM to destination-VM execution as an exact non-root target user, independent of network membership. | IP connectivity, reverse execution, VM-to-host execution, or destination root execution. | + +The host-to-guest `mvm exec` and `mvm cp` paths use the embedded vsock agent. Host-originated `mvm exec` defaults +to the guest's root user; this does not grant VM-to-VM root execution. VM-to-VM execution additionally passes +through an authenticated host relay that binds the live source and destination identities and enforces the typed +exec grant. The relay bounds its command, output, duration, and concurrency and writes a command-redacted audit +record. See [ADR-0017](adr/0017-per-vm-network-namespaces-and-typed-policies.md). + +## Volumes, snapshots, and recovery + +Attached VM data volumes are raw-only; QCOW2 remains supported for image import/conversion, not as an attached +data-volume format. Active writable disks are root-protected rather than reused from an operator-writable staging +file. An attached volume can grow while its VM is stopped without detaching it. `volume rm --force` likewise +requires the attached VM stopped; it does not hot-unplug a running guest. + +The attached-volume count is derived from the selected release, architecture, transport, and complete device set. +MMIO admission subtracts VMGenID, VMClock, rootfs, network, optional cloud-init and vsock, and the applicable ARM +RTC/serial reservations from the interrupt pool. There is no uniform 13-volume product ceiling. PCI uses separate +slot and MSI-X accounting. These admission checks do not qualify maximum device counts; aarch64 runtime +qualification remains deferred. + +`volume attach` on a Running VM succeeds only after the authenticated guest agent observes the exact +root-derived disk serial/capacity bound to `virtio_blk`, followed by an unchanged root attachment identity. +On a Paused VM it stages the attachment and returns a retryable deferred-readiness error: resume the VM and +retry the same attach. Stopped VMs record the attachment for their next start. A readiness failure does not +undo a known-applied attachment or make its volume available; inspect the state and retry the same operation. +Installed-candidate running and paused/resume-retry readiness journeys passed with usable guest I/O; see the +[coverage matrix](../tests/system/COVERAGE_MATRIX.md) for the accepted source and candidate bindings. +It detects an unusable attachment; it cannot repair the PCI BAR relocation failure observed on Firecracker 1.16.2. +The approved 1.17.0 dependency update includes upstream +[BAR/MMIO relocation changes](https://github.com/firecracker-microvm/firecracker/blob/v1.17.0/src/vmm/src/devices/virtio/transport/pci/device.rs#L657-L715) +selected to address that failure. Same-volume live detach/reattach passed on the current 1.17.0 project candidate; +these bounded journeys do not establish compatibility with every guest or complete release qualification. + +A full snapshot captures rootfs, memory/vmstate, and attached data volumes under exclusive write ownership. Its +artifacts are root-controlled under the selected cache, with CLI-mediated removal. Restore creates independent +writable disks while retaining immutable captured memory backing; it preserves the recorded release, drive IDs, and +managed network identity. A conflicting writer or unresolved artifact identity blocks capture or restore. The +approved limit for the complete `vmstate` file, including its CRC, is 10,000,000 bytes. The memory artifact must +have exactly trusted captured RAM in MiB × 1,048,576 bytes of logical length, including sparse extents. v0.3.0 adds +no data-volume size or aggregate-storage quota and reserves no physical storage. See +[ADR-0018](adr/0018-isolated-vm-execution-and-snapshot-authority.md) +and [the resource-limit decision](improvements/volume-snapshot-resource-limits.md). + +VM stop, remove, reboot, and restore use typed cleanup and reconciliation rather than deleting a PID file as proof +of process death. `host clean` is caller-scoped and refuses running or incompletely observed VMs; administrator +`host reset` is a separate, quiescence-checked operation and is not an old-state migration command. The exact +host-reboot and bounded memory-pressure results are recorded in the qualification checkpoint. They do not explain +the original saturation incident or guarantee survival under arbitrary host, ancestor, global or guest OOM. + +## Performance evidence + +[`benchmarks/results.json`](../benchmarks/results.json) records an older measured workload. Its values predate the +v0.3.0 root broker, per-VM namespaces, protected backing, and direct-Go firewall path, so they must not be used as +v0.3.0 boot-time or throughput promises. Rebenchmark the release candidate before publishing performance numbers. diff --git a/docs/STANDARDS.md b/docs/STANDARDS.md index 877020d7..90a4c7c0 100644 --- a/docs/STANDARDS.md +++ b/docs/STANDARDS.md @@ -95,7 +95,7 @@ Three-layer flow: **CLI → API → Core** - Performs state detection (checking current system state to branch execution) - Guards invariants that protect against system damage - Does NOT validate caller input -- Constructor takes repos/options only: `network.NewService(repo Repository, tracker *firewall.FirewallTracker)` +- Constructor takes repos/options only: `network.NewService(repo Repository, tracker *firewall.FirewallTracker, policyRepo ServiceAccessPolicyRepository)` - Wired once at startup in `app.Initialize()` - Service methods that don't reference the Service struct live in `utils.go` @@ -172,8 +172,8 @@ if errs.IsNotFound(err) { ... } ONE canonical path: `system.DefaultRunner.Run(ctx, args, system.RunCmdOpts{...})` or `system.DefaultRunner.Stream(ctx, args, system.RunCmdOpts{...})` in `internal/lib/system/runner.go`. ```go -// Correct -result, err := system.DefaultRunner.Run(ctx, []string{"ip", "link", "set", tap, "down"}, system.RunCmdOpts{Privileged: true}) +// Correct for an ordinary subprocess that is not a typed root effect +result, err := system.DefaultRunner.Run(ctx, []string{"ip", "link", "show", tap}, system.RunCmdOpts{}) // Forbidden exec.Command("iptables", ...) // NEVER @@ -183,17 +183,23 @@ Documented exceptions for direct `os/exec` / `os/exec.CommandContext`: | Location | Why DefaultRunner doesn't work | |----------|-------------------------------| -| `internal/core/vm/firecracker.go` | Needs `pass_fds` for VM API socket and log file descriptors | +| `internal/service/jailer/entry.go` | Privileged Jailer handoff uses `syscall.Exec` to preserve console stdio and process identity after chroot setup | +| `internal/service/jailer/lifecycle_child_spawn.go` | Fixed-binary `os.StartProcess` creates the authenticated handshake-blocked child and closed inherited channels | +| `internal/service/jailer/lifecycle_child.go` | Fixed admitted Jailer `syscall.Exec` preserves process identity and private descriptor-only mounts; the early child route always exits | +| `internal/service/jailer/lifecycle_console_spawn.go` | Fixed-binary `os.StartProcess` drops to the admitted caller UID/GID with empty supplementary groups and passes only the root-created PTY and authenticated closed channels; the relay consumes pinned outputs rather than caller paths | | `internal/core/ssh/utils.go` | SSH connectivity probe uses `exec.CommandContext` with short-lived probe context | | `internal/service/loopmount/provisioner.go` | Direct provisioning engine with chained losetup/mount/umount/chroot | | `internal/service/agent/exec.go` | Guest agent command execution via `su`/`sh` with `exec.CommandContext` | | `internal/service/agent/pty.go` | Guest agent PTY session via `su` with `exec.CommandContext` | +| `internal/service/agent/destination_process.go` | Closed VM-exec destination execution requires an exact nonzero numeric UID, numeric groups, fixed environment/home, `/bin/sh -c`, and process-group cancellation that `DefaultRunner` cannot express | | `internal/lib/archive/archive.go` | xz decompression via pipe-based `exec.CommandContext` | +| `internal/lib/system/privileged_control.go` | Fixed authenticated dispatcher exchange requires a duplex fd-0 socket, physical EOF and owned reaping; only payload-free VM-rootfs export disconnects without killing its separately sessioned recovery worker | | `internal/lib/system/runner.go`, `interactive_run.go`, `spawn.go` | Implementation of `DefaultRunner.Run()` / `Stream()` / `SpawnService` itself | **Binary lookup carve-out:** Utility files across the codebase use `exec.LookPath()` (not `exec.Command`/`exec.CommandContext`) solely to check whether a system binary exists before calling it through `DefaultRunner.Run()`. This is NOT a subprocess execution — it's a filesystem existence check. These files are not listed as exceptions above and do not violate the subprocess rule. -Service subprocesses use `system.SpawnService(ctx, cfg)`. +Ordinary-user service subprocesses use `system.SpawnService(ctx, cfg)`. Managed network and rootfs effects use their +closed authenticated privileged-dispatch actions, not raw privileged subprocess arguments. ## 8. Model Types @@ -254,24 +260,30 @@ pool.Seq[T,R](ctx, items, fn) // sequential fail-fast, returns []Res ## 12. Testing -Three-level architecture — see `docs/development/HOW_AGENTS_WRITE_SYSTEM_TESTS.md` for the full specification. +Read `docs/system-test-architecture.md` for the test architecture and +`docs/development/HOW_AGENTS_WRITE_SYSTEM_TESTS.md` for authoring guidance. ### L0: Pure Function Tests (Go) -- Table-driven `map[string]struct{...}` with `t.Run()`. No I/O, no DB, no subprocess. +- Use named table cases with `t.Run()`. Use a map when order must not matter. Use a slice when deterministic order makes + the contract clearer. L0 has no I/O, database, or subprocess work. - Input → output only. Runs in microseconds. - Example: `ParseDiskSize("1G") == 1073741824` ### L1: Hermetic Integration Tests (Go) - Uses real I/O in controlled environments: file-based SQLite, `t.TempDir()`, `FakeRunner` for subprocess calls that can't run in CI. -- No networking, no KVM, no sudo. +- Local loopback fixtures are allowed; no external network, KVM, or sudo. - Example: seed DB → call handler → verify JSON output via `cmp.Diff`. -- Run via `go test ./...`. +- Run the unprivileged CI selection in `scripts/run-go-ci-tests.sh`. The complete Jailer package includes root-owned + lifecycle effects and runs locally in a disposable root-test VM; CI exercises a selected unprivileged subset. -### L2: Runner VM System Tests (Python in `tests/system/`) -- Real binary, real subprocess, real infrastructure inside a disposable Firecracker VM with nested KVM. +### L2: Real-binary system tests (Python in `tests/system/`) +- Real binary, real subprocess, and real infrastructure. T1 and T2 run inside disposable Firecracker VMs. T3 runs only + on a clean host-direct qualification machine for behavior that cannot run correctly under nesting. - No mocking of any kind — the binary is real, the subprocesses are real. +- Install the release candidate through `host install-system`; T1/T2 tests invoke exact `/usr/local/bin/mvm`. +- Directly copying a binary onto the canonical path or relying on PATH lookup does not qualify the installer or release. - **Ground truth:** A feature is not tested until it has an L2 test. L0/L1 are fast pre-filters, not replacements. -- Run via `pytest tests/system/` inside the runner VM. +- Run through `scripts/run-system-tests.py`. It selects the execution tier and invokes pytest in the correct environment. ## 13. CLI Patterns @@ -425,15 +437,13 @@ if net == nil { net = snap.Network } ## 17. Subprocess Services -Long-running subprocess services in `internal/service/`: +Ordinary-user subprocess services in `internal/service/`: - `console/` — PTY relay goroutine (entry: `mvm run console relay`) - `nocloudnet/` — NoCloud HTTP metadata server goroutine (entry: `mvm run nocloudnet serve`) -- `loopmount/` — Loop-mount provisioner subprocess (entry: `mvm run provision`) -- `agent/` — Embedded guest agent binary (cross-compiled, zstd-compressed, injected at runtime via vsock — no `mvm run` entry) -Each subprocess service follows: `Config` struct → `Run(ctx, cfg)` (blocking) → `Spawn(ctx, cfg, extraFiles...)` (background via `system.SpawnService`). +These services follow: `Config` struct → `Run(ctx, cfg)` (blocking) → `Spawn(ctx, cfg, extraFiles...)` (background via `system.SpawnService`, without elevation). -All compiled into the same `mvm` binary. +The same `mvm` binary also contains `loopmount/` (closed typed privileged rootfs receiver), `jailer/` (privileged launch and resource authority), and `agent/` (embedded cross-compiled guest binary injected at runtime). None has a public root service route; privileged host effects enter through the authenticated dispatcher. ## 18. Performance diff --git a/docs/TROUBLESHOOTING.md b/docs/TROUBLESHOOTING.md index c72d51cc..1a64ca9c 100644 --- a/docs/TROUBLESHOOTING.md +++ b/docs/TROUBLESHOOTING.md @@ -5,8 +5,8 @@ Common issues and solutions when using `mvm`. ## Table of Contents - [Permission denied: /dev/kvm](#permission-denied-devkvm) -- [Mixed Firewall Backend](#mixed-firewall-backend) -- [Firewall rules lost after reboot](#firewall-rules-lost-after-reboot) +- [Managed firewall or network refusal](#managed-firewall-or-network-refusal) +- [Managed network state after host reboot](#managed-network-state-after-host-reboot) - [VM won't boot / SSH times out](#vm-wont-boot-ssh-times-out) - [VM won't start / Firecracker exits immediately](#vm-wont-start-firecracker-exits-immediately) - [Undoing host init — host clean vs host reset](#undoing-host-init-host-clean-vs-host-reset) @@ -15,6 +15,7 @@ Common issues and solutions when using `mvm`. - [Console relay not working](#console-relay-not-working) - [IP address exhaustion (no available IPs)](#ip-address-exhaustion-no-available-ips) - [Cache corruption or stale state](#cache-corruption-or-stale-state) +- [Host memory stays high after destroying VMs](#host-memory-stays-high-after-destroying-vms) - [Out of disk space for images or VMs](#out-of-disk-space-for-images-or-vms) - [libguestfs / mvm cache init hangs forever](#libguestfs-mvm-cache-init-hangs-forever) - [Cannot remove volume attached to a VM](#cannot-remove-volume-attached-to-a-vm) @@ -55,59 +56,48 @@ groups | grep kvm --- -## Mixed Firewall Backend +## Managed firewall or network refusal -**Symptom:** VM has valid IP and gateway. ICMP (ping) works. TCP (curl/wget) times out. +**Symptom:** Managed networking fails admission, or a VM cannot reach its intended destination. -**Detection:** +v0.3.0 uses direct Go nftables transactions. Managed networking requires kernel support for both `inet` and +`bridge` families, connection tracking, and NAT; installing the `nft` tool cannot add missing kernel support. +Same-network and cross-network VM traffic starts denied. VM-to-VM TCP/UDP connectivity requires an applicable +traffic grant; successful vsock execution does not establish IP connectivity. -First check which firewall backend mvmctl is configured to use: -```bash -mvm config get settings firewall_backend -``` - -If the backend is `nftables` (default), verify the nftables ruleset contains mvm chains: -```bash -sudo nft list ruleset | grep -c "MVM-" -``` +Inspect the affected managed network, VM, and traffic grants: -If the backend is `iptables`, check which iptables variant is active: ```bash -iptables --version +mvm network inspect --json +mvm vm inspect --json +mvm policy traffic ls --json ``` -**Cause:** Docker and mvmctl use different firewall backends. Rules go to different places. mvmctl defaults to nftables; Docker may configure the system to use iptables-legacy, creating a split where rules are applied to the wrong backend. - -**Fix:** -```bash -# Option 1: Sync firewall rules from the database -mvm network sync - -# Option 2: Reboot host (clears all firewall state cleanly) -sudo reboot - -# Option 3: Configure Docker to use the same backend as mvmctl -# Edit /etc/docker/daemon.json and restart Docker -``` +For a refused operation, retain its structured error code and details. Root authority refuses foreign or changed +managed firewall state rather than overwriting it. Resolve the reported ownership or kernel-capability problem +before retrying; a ruleset flush or host reboot is not a reconciliation procedure. -Then re-run: `mvm host init` +See [networking and policy](RUNTIME.md#networking-policy-and-guest-execution) and +[traffic grants](REFERENCES.md#mvm-policy) for the supported contract. --- -## Firewall rules lost after reboot +## Managed network state after host reboot -**Problem:** After a reboot, VMs can't reach the network, or firewall chains are missing. +**Problem:** Retained network or VM records need reconciliation after a host reboot. -**Cause:** `mvm` does not persist firewall rules to system files (`/etc/nftables.conf`, `/etc/iptables/rules.v4`). Firewall rules are managed dynamically and must be reloaded after a reboot. +Inspect the affected network and VM, then request managed-network authority reconciliation: -**Solution:** - -Run `mvm network sync` after every reboot to reload all mvm firewall rules from the database: ```bash -mvm network sync +mvm network inspect --json +mvm vm inspect --json +mvm network sync --json ``` -This restores all NAT, forwarding, and nocloud-net rules to the active firewall backend (nftables or iptables). +`network sync` reconciles managed network authority and retries pending creation. Inspect its result and any +reported failure; it does not promise restoration of arbitrary firewall rules or a NoCloud listener. Retained VM +records do not mean the old Firecracker process survived. Cleanup and relaunch must reconcile root-owned state, +and ambiguous ownership remains a refusal. See [runtime state](RUNTIME.md#state-and-process-layout). --- @@ -145,29 +135,35 @@ mvm logs myvm --os **Common causes:** -**1. Missing or broken jailer.** Firecracker requires the `jailer` binary alongside it. If the binary was fetched with `mvm bin pull`, it should be bundled, but manual installs may miss it. +**1. Missing, mismatched, or untrusted Jailer pair.** VM launch requires the exact checksum-verified Firecracker/Jailer release pair in the root-owned trusted store. Re-pull the selected release; source-built or manually copied binaries cannot launch jailed VMs: -**2. Kernel file is not readable by Firecracker.** Verify the kernel exists and the path is correct: ```bash -mvm kernel ls --json | jq -r '.[].path' -ls -l +mvm bin ls --long +mvm bin pull --force ``` -**3. Invalid boot arguments.** Custom boot args (set via `mvm config`) may contain typos or flags the kernel doesn't understand. Reset to defaults: +**2. cgroup v2 or required controllers are unavailable.** VM launch requires the unified cgroup-v2 hierarchy with `cpu`, `memory`, and `pids`. Refresh host information and inspect the detected limits: + ```bash -mvm config reset defaults.vm boot_args +mvm host info --refresh --json ``` -**4. Socket path too long.** Firecracker uses Unix domain sockets which have a 108-character path limit. Long VM names or deep cache directories can exceed this. Check: +If launch reports a verification mismatch, `mvm vm inspect --json` shows the requested and observed values. mvmctl fails closed rather than launching without the envelope. + +**3. Kernel file is not readable by Firecracker.** Verify the kernel exists and the path is correct: ```bash -# See the VM's socket path -ls -la ~/.cache/mvmctl/vms/*/firecracker.api.socket -# If it looks very long, try a shorter VM name +mvm kernel ls --json | jq -r '.data[].path' +ls -l +``` + +**4. Invalid boot arguments.** Custom boot args (set via `mvm config`) may contain typos or flags the kernel doesn't understand. Reset to defaults: +```bash +mvm config reset defaults.vm boot_args ``` **5. Binary / kernel architecture mismatch.** A kernel built for `x86_64` won't boot under an `aarch64` Firecracker binary. Verify both match: ```bash -file $(mvm kernel ls --json | jq -r '.[0].path') +file $(mvm kernel ls --json | jq -r '.data[0].path') mvm bin ls ``` @@ -177,44 +173,47 @@ mvm bin ls **Problem:** You want to roll back the changes made by `mvm host init`. -There are two levels of undo depending on how much you want to remove: +For complete v0.3.0 removal before a clean reinstall, use the [administrator uninstaller](UNINSTALL.md). +It deletes managed data as well as restoring host setup; the commands below have narrower scopes. + +Caller-scoped network cleanup and administrator rollback have different scopes. Neither command is a supported +legacy-state migration or removal procedure. See the [host command reference](REFERENCES.md#mvm-host). ### `mvm host clean` — Remove networking only -Stop all running VMs first, then: +Stop the caller's running VMs first, then run as the enrolled operator: ```bash mvm host clean ``` -This removes: -- All bridges and TAP devices -- All firewall chains used by mvm -- The default network from the database -- Any orphaned bridges and rules - -It does **NOT** touch: -- The `mvm` system group or your user membership -- The sudoers drop-in file -- Sysctl `ip_forward` setting +`host clean` observes the caller's root VM inventory and removes its locally registered networks through typed +root authority. It refuses running VMs or incomplete observation and does not force-stop VMs or reset shared +host settings, operator membership, or the sudoers policy. Recreate desired networks with `mvm network create`; +network cleanup does not undo administrator enrollment. -Run `mvm host init` afterwards to recreate the default network. +### `mvm host reset` — Administrator rollback -### `mvm host reset` — Full factory reset +Reset requires exclusive host-wide quiescence and compatible recorded administrator state. Run through the +installed root-owned binary with administrator access: ```bash -mvm host reset --force +sudo /usr/local/bin/mvm host reset --force ``` -Does everything `clean` does, **plus**: -- Removes the sudoers drop-in (`/etc/sudoers.d/mvm`) -- Removes your user from the `mvm` group -- Deletes the `mvm` system group -- Restores `net.ipv4.ip_forward` to its original value +Reset restores recorded host settings and enrollment changes through retryable receipts. It refuses live or +ambiguous state, preserves authority history, and leaves privileged admission disabled afterward, including +after a restoration error. `--force` does not bypass quiescence or force-stop VMs. -After reset, run `mvm host init` from scratch to set everything up again. +To set up the host and enroll the invoking non-root operator again: + +```bash +sudo /usr/local/bin/mvm host init +``` -> Both commands refuse to run if any VMs are still running. Stop them first with `mvm vm rm `. +Log out and back in, or run `newgrp mvm`, if the current session lacks the enrolled group. In a direct-root +session, both init and reset require `--operator USER`; sudo invocation selects its invoking operator and +rejects that flag. --- @@ -224,14 +223,19 @@ After reset, run `mvm host init` from scratch to set everything up again. **Solution:** -The port range (8000-9000) may be exhausted. Check for stale servers: +NoCloud net mode binds the selected managed network's gateway on fixed TCP port `8000`. Custom ports are +rejected, and a listener collision fails without adopting or killing the existing listener. Inspect the network +and the listener at its actual gateway address: + ```bash -# List processes using nocloud ports -sudo ss -tlnp | grep -E ':(8[0-9]{3}|9[0-9]{3})' -# Kill any orphaned mvm processes -pkill -f "mvm run nocloudnet serve" +mvm network inspect --json +sudo ss -ltnp 'sport = :8000' ``` +Identify the listener's owning workload before changing it. A successful bind must precede the typed fixed-slot +service grant; the grant reserves gateway TCP `8000` and does not authenticate the listener process. +See [NoCloud net mode](REFERENCES.md#how-net-mode-works). + --- ## VM can't fetch cloud-init data via nocloud-net @@ -240,32 +244,24 @@ pkill -f "mvm run nocloudnet serve" **Solution:** -Verify firewall rules are configured: +Confirm the VM uses a managed network and net cloud-init mode; networkless placement cannot use net mode. +Inspect the network's actual gateway, its fixed TCP `8000` listener, and the VM's cloud-init messages: -For the nftables backend (default): ```bash -sudo nft list chain inet filter MVM-NOCLOUDNET-INPUT -# Should show rules allowing source IP to destination ports +mvm network inspect --json +sudo ss -ltnp 'sport = :8000' +mvm logs myvm | grep -i nocloud ``` -For the iptables backend: -```bash -sudo iptables -L MVM-NOCLOUDNET-INPUT -n -v -# Should show rules allowing source IP to destination ports -``` +From the affected guest, diagnose HTTP delivery using the actual gateway address: -Check that the VM's network is correctly set up. First find the nocloud server port and correct gateway from the VM logs: ```bash -mvm logs myvm --os | grep -i nocloud +curl -v http://:8000/meta-data ``` -Then test from within the VM (adjust subnet and port to what your config uses): -```bash -# Test connectivity to the default gateway (default subnet: 172.27.0.0/24) -ping -c 1 172.27.0.1 -# Test HTTP access to nocloud server (port is dynamically allocated, check logs) -curl -v http://172.27.0.1:/ -``` +Keep listener readiness separate from the root-derived service grant: a listening socket alone does not prove +the grant is applied, and a retained grant does not prove the listener is running. Review any typed bind or grant +failure before retrying. See [NoCloud security](REFERENCES.md#net-mode-security-architecture). --- @@ -338,6 +334,23 @@ mvm cache prune --all # Prune everything --- +## Host memory stays high after destroying VMs + +**Symptom:** Host memory usage stays high after `mvm vm stop` or `mvm vm rm`. + +Firecracker's process exit releases its memory mappings. File cache, independent snapshot files and warm +images can remain. Clean file cache is reclaimable under pressure; an existing snapshot does not pin every +cached page in RAM. The warm pool defaults to `/tmp/mvmctl-/ready`, on tmpfs only when that filesystem +is memory-backed. + +Check `mvm vm inspect VM --json` while the VM runs, then compare anonymous, file-cache and shared-memory +usage. Host-wide `AnonPages`, `Cached`, `Shmem` and swap counters include unrelated processes. Review +`mvm cache prune misc --dry-run` before pruning warm images and the selected appliance cache. + +See [host memory after teardown](implementations/MEMORY_RECLAIM.md) for accounting and diagnostic details. + +--- + ## Out of disk space for images or VMs **Symptom:** Image downloads fail mid-way, VM creation fails with disk errors, or `df` shows the cache partition is full. @@ -490,19 +503,25 @@ libguestfs-make-fixed-appliance ~/.cache/mvmctl/appliance **Solution:** ```bash -# Detach the volume from the VM first +# Stop the VM, then detach its volume +mvm vm stop mvm volume detach # Then remove the volume mvm volume rm ``` -Or force-remove (removes the volume record and file but does not hot-unplug from the VM): +To remove a volume that remains attached, stop its VM first, then use `--force`: ```bash +mvm vm stop mvm volume rm --force ``` +`--force` bypasses the attachment refusal only for a VM authoritatively observed as stopped; it never hot-unplugs +a running guest. To grow an attached volume, stop the VM but leave the volume attached, then run +`mvm volume resize ` before starting the VM again. + --- ## VM boots slowly (SSH takes longer than 6-7 seconds) diff --git a/docs/UNINSTALL.md b/docs/UNINSTALL.md new file mode 100644 index 00000000..46202cd2 --- /dev/null +++ b/docs/UNINSTALL.md @@ -0,0 +1,63 @@ +# Uninstall mvmctl + +The administrator uninstaller removes an initialized v0.3.0 installation and every enrolled operator's managed +data. This includes running VMs, snapshots, volumes, images, kernels, cached SSH keys and configuration. Export +anything you want to keep before continuing. Other operators must stop using mvmctl during the uninstall. + +Use the scripts from the same source version as the installed binary. Python 3.10 or later is required; +`uninstall.py` and `uninstall_files.py` must stay together in `scripts/`. + +Preview the selected paths first: + +```bash +sudo python3 scripts/uninstall.py --dry-run +``` + +Then run the uninstall and type `UNINSTALL MVMCTL` when prompted: + +```bash +sudo python3 scripts/uninstall.py +``` + +For unattended use, `--yes` supplies the same confirmation. It permanently deletes data; it is not a repair flag. +The dry run reads installation records without invoking mvm or creating a database. + +## What it removes + +- Every enrolled operator's default cache, plus existing active caches identified by protected-storage records. +- Default `~/.config/mvmctl` directories, including managed SSH-key copies, and `/tmp/mvmctl-UID` warm-image pools. +- Recognized personal `mvm` binaries or launchers in `~/.local/bin` and `~/bin`. +- Recorded host-init changes: mvm's sysctl and sudoers changes, enrolled memberships, and the group if mvm created it. +- The managed runtime directory, empty mvm cgroup, root authority under `/var/lib/mvmctl`, and installed binary. + A recognized packaged `/usr/bin/mvm` is removed through its package manager. + +External asset mirrors, original SSH files outside managed directories, and shared dependencies such as +Firecracker prerequisites remain installed. Preexisting group memberships and host settings are restored from +the administrator journal rather than guessed. + +Custom paths that are not recorded cannot be discovered automatically. Add an operator's custom cache or +configuration directory explicitly; options may be repeated: + +```bash +sudo python3 scripts/uninstall.py --dry-run \ + --cache alice=/srv/alice/mvm-cache --config alice=/srv/alice/mvm-config +``` + +Use the same options for the actual uninstall. Custom temporary directories outside these selected paths are +preserved. Paths must belong to the enrolled operator; symlinks, nested mounts, overlapping selections and +paths that would include the external default asset mirror are refused. + +## If cleanup stops + +The script uses `mvm cache clean --force` for each selected cache, then `mvm host reset` before deleting root +authority. A cleanup or ownership error stops the operation. Completed deletions are not rolled back; remaining +files and the error identify what needs attention. Do not replace the refusal with `rm -rf` or flush host firewall +rules. Unknown files, incompatible legacy layouts, changed administrator state and unresolved runtime ownership +need diagnosis with a compatible installation. + +Once host reset completes, a retry can finish removing installation files. If the final directory removal is +interrupted, the retained root-only `/var/lib/.mvmctl-uninstall` may require administrator inspection; a retry +requires its reset records to remain intact. + +After success, follow the [installation instructions](../README.md#installation) and enroll the operator again. +Managed VMs and artifacts start from an empty cache; preserved external mirrors can be reused. diff --git a/docs/adr/0002-single-go-binary-architecture.md b/docs/adr/0002-single-go-binary-architecture.md index 407b83b8..d43caf60 100644 --- a/docs/adr/0002-single-go-binary-architecture.md +++ b/docs/adr/0002-single-go-binary-architecture.md @@ -2,10 +2,10 @@ **Status:** Active **Date:** 2026-05-22 -**Last Updated:** 2026-06-20 (final Go implementation) +**Last Updated:** 2026-09-27 (0.3.0 typed service dispatch) **Supersedes:** The previous Python Nuitka two-binaries build (legacy/ directory) -The `mvm` project is distributed as a single compiled Go binary with no runtime dependencies. The binary contains both the CLI entry point and all background service subprocesses (console relay, nocloudnet server, loopmount provisioner) via subcommand dispatch (`mvm run `). This replaces the previous Python Nuitka build which produced two separate binaries (`mvm` and `mvm-services`) with multidist symlink dispatch. +The `mvm` project is distributed as a single compiled Go binary. It contains the CLI, ordinary-user background services (console relay and NoCloud HTTP server), and the closed privileged LoopMount/Jailer receiver. Only the ordinary-user services use public `mvm run ` subcommands; privileged actions enter through the early authenticated dispatcher. This replaces the previous Python Nuitka build which produced two separate binaries (`mvm` and `mvm-services`) with multidist symlink dispatch. **Table of Contents** @@ -20,39 +20,21 @@ The `mvm` project is distributed as a single compiled Go binary with no runtime | Aspect | Selection | Rationale | |--------|-----------|-----------| -| Language | Go 1.26.3 | Compiled, zero runtime dependencies, fast startup, excellent concurrency primitives | -| Binary count | Single binary | One `mvm` binary contains all CLI commands and service subcommands | -| Service dispatch | Subcommand-based (`mvm run `) | Services are hidden subcommands within the same binary, launched via `system.SpawnService()` | +| Language | Go 1.26.3 | Compiled, no language interpreter dependency, fast startup, excellent concurrency primitives | +| Binary count | Single binary | One `mvm` binary contains the CLI, ordinary services, and typed privileged receiver | +| Service dispatch | Public `mvm run ` for ordinary services; early authenticated dispatch for root effects | Ordinary services self-spawn; root actions do not enter the public CLI | | Distribution | Single statically-linked ELF | No shared library dependencies beyond libc; no Python interpreter needed | ## Architecture ### Entry Point (`cmd/mvm/main.go`) -```go -func main() { - ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM) - defer stop() - - op, cleanup, err := app.Initialize(ctx) - if err != nil { - slog.Error("initialization failed", "error", err) - os.Exit(1) - } - if cleanup != nil { - defer cleanup() - } - - rootCmd := cli.NewRootCmd(op) - if err := rootCmd.ExecuteContext(ctx); err != nil { - if common.HandleErrors(func() error { return err })() != nil { - os.Exit(1) - } - } -} -``` - -Services are not dispatched via a manual check in main; instead, Cobra subcommands registered in `internal/cli/service.go` handle `mvm run `. +`run` selects the private launch-console and launch-child routes, the reserved versioned privileged protocol, +the exact administrator `host install-system` bootstrap, and administrator `host init`/`host reset` before ordinary +initialization. These reserved routes fail closed and cannot fall through to the public CLI. All ordinary +commands then enter `runCLI`, which establishes signals, calls `app.Initialize()`, and executes Cobra. Ordinary-user +background services remain Cobra subcommands registered in `internal/cli/service.go`; root-requiring LoopMount and +Jailer operations use the typed privileged actions of ADR-0016. ### Service Subcommands @@ -62,13 +44,12 @@ Background services are compiled into the same binary as subcommands under `mvm |---------|-----------|---------| | Console relay | `mvm run console relay` | `internal/service/console/` | | NoCloud HTTP server | `mvm run nocloudnet serve` | `internal/service/nocloudnet/` | -| Loop-mount provisioner | `mvm run provision` | `internal/service/loopmount/` | -These are launched via `system.SpawnService()` (`internal/lib/system/spawn.go`) which resolves the executable path, optionally prepends `sudo` when `Privileged` is set, and passes `Args` to `mvm run `. +These ordinary-user services are launched via `system.SpawnService()` (`internal/lib/system/spawn.go`), which resolves the executable path and passes `Args` to `mvm run ` without elevation. LoopMount provisioning uses the same binary's authenticated typed root dispatcher, not `SpawnService` or a public `run provision` route. ## Why Go -- **Zero runtime dependencies**: A single compiled binary with no interpreter, no virtual environment, no package manager. +- **No language runtime dependency**: A single compiled binary needs no interpreter or virtual environment. Host facilities and packages required for VM operation are documented separately. - **Fast startup**: Go binaries start in milliseconds with zero interpreter startup overhead. - **Type safety**: Compile-time type checking prevents entire classes of runtime errors. - **Concurrency**: Native goroutines and channels for the parallel execution model (pool.Do, pool.Gather, pool.Seq). @@ -88,11 +69,13 @@ The previous Python implementation (preserved in `legacy/`) used Nuitka to compi - **Single binary**: Users download and install one file. No symlinks, no multidist dispatch. - **No build-time dependency tracking**: Go's static compilation eliminates the need for explicit module inclusion lists. - **Service spawning**: `system.SpawnService()` (`internal/lib/system/spawn.go`) launches the same binary with different subcommand arguments — no separate binary path resolution. -- **Sudoers simplicity**: The sudoers file only needs the `mvm` binary path, not a separate `mvm-services` path. +- **Sudoers simplicity**: The final sudoers policy needs only the versioned early entry point of root-owned + `/usr/local/bin/mvm`, not a separate `mvm-services` or helper executable. - **Build speed**: `go build` is significantly faster than Nuitka compilation (seconds vs minutes). - **Cross-compilation**: `GOOS=linux GOARCH=amd64 go build` produces a target binary without requiring the target toolchain. ## Related Decisions -- ADR-0003: Provisioning backend mutual exclusion — the `mvm run provision` subcommand is the loop-mount entry point. -- CONTEXT.md "Provisioner Backend" — mount/umount consolidated in `mvm run provision` subcommand. +- ADR-0003: Provisioning backend mutual exclusion — LoopMount uses the typed root receiver. +- ADR-0016: Root-owned system installation, early privileged dispatch, and typed privilege migration. +- CONTEXT.md "Provisioner Backend" — closed rootfs effects are owned by the typed LoopMount receiver. diff --git a/docs/adr/0003-loopmount-guestfs-mutual-exclusion.md b/docs/adr/0003-loopmount-guestfs-mutual-exclusion.md index f3d74322..d36f4594 100644 --- a/docs/adr/0003-loopmount-guestfs-mutual-exclusion.md +++ b/docs/adr/0003-loopmount-guestfs-mutual-exclusion.md @@ -2,7 +2,7 @@ **Status:** Active **Date:** 2026-05-22 -**Last Updated:** 2026-06-20 (Go implementation complete) +**Last Updated:** 2026-09-27 (0.3.0 privilege path) The project provides two independent provisioning backends for root filesystem operations (cloud-init injection, shrink, deblob, OS detection): **LoopMount** and **GuestFS**. These backends are **mutually exclusive** — a single VM or image operation uses exactly one backend, never a combination. The `guestfs_enabled` setting acts as a toggle selector, not a preference. @@ -23,7 +23,7 @@ Only one backend is active for a given operation. The selection logic is: 1. If `guestfs_enabled` (from user settings) is `true` → use **GuestFS**. 2. Else → use **LoopMount**. -LoopMount is always available (it runs as the `mvm run provision` subprocess within the same binary, invoked via `system.SpawnService()` or directly via `runWireOp` in `internal/lib/provisioner/loopmount/backend.go`). There is no fallback chain — `guestfs_enabled` is an **override**, not a fallback. The setting is persisted in user settings and can be toggled via `mvm config set settings guestfs_enabled true|false`. +LoopMount uses the authenticated typed root receiver in the same `mvm` binary; it has no public `mvm run provision` route or raw wire-operation fallback. There is no backend fallback chain — `guestfs_enabled` is an **override**, not a fallback. The setting is persisted in user settings and can be toggled via `mvm config set settings guestfs_enabled true|false`. ## Independence @@ -31,11 +31,11 @@ Both backends have separate code paths, separate dependencies, separate error ha | Aspect | LoopMount | GuestFS | |--------|-----------|---------| -| Mechanism | `losetup` + `mount` + `chroot` via compiled `mvm` binary (`mvm run provision`) | `libguestfs` via QEMU appliance | +| Mechanism | Typed rootfs operations through the authenticated `mvm` receiver | `libguestfs` via QEMU appliance | | Dependencies | Compiled binary (same `mvm` binary) | `libguestfs`, `supermin`, QEMU (system packages) | | Performance (create → ready, seq) | 2–5s per VM (see benchmarks) | 9–14s per VM (see benchmarks) | -| Privilege | `mvm run provision` in sudoers | `supermin` in sudoers | -| Implementation | `internal/lib/provisioner/loopmount/` (backend), `internal/service/loopmount/` (subprocess entry) | `internal/lib/provisioner/guestfs/` | +| Privilege | Closed privileged-dispatch action through the fixed `mvm` sudoers marker | No separate `supermin` sudoers grant in 0.3.0 | +| Implementation | `internal/lib/provisioner/loopmount/` (backend), `internal/service/loopmount/` (typed receiver) | `internal/lib/provisioner/guestfs/` | | Default | **Yes** (`guestfs_enabled = false`) | **No** (opt-in via `mvm init` prompt or config) | ## No Mixing @@ -51,7 +51,7 @@ Attempting to mix backends within a single operation is a bug. The natural alternative would be a fallback chain: try loop-mount first, fall back to guestfs if unavailable. This was rejected because: - **Performance expectations**: If a user explicitly enables GuestFS, they expect GuestFS behavior (slower but more capable OS detection, init-system-aware SSH setup). A silent fallback to loop-mount would violate the principle of least surprise. -- **Dependency clarity**: Each backend has different sudoers requirements. Mixing them in a single session means both sets of sudoers rules must be valid, increasing the privilege surface area. +- **Dependency clarity**: The backends have different host dependencies and effect boundaries. A silent fallback would obscure which path performed the rootfs operation. - **Test isolation**: Each backend has independent test suites. A fallback chain would require testing all combinations, significantly increasing the test matrix. - **Bug prevention**: An earlier version of the selection logic incorrectly described GuestFS as a "fallback," which led to regression bugs where a stale `guestfs_enabled = true` setting silently selected the slow backend during performance-sensitive operations. @@ -69,7 +69,7 @@ if guestfsEnabled { ## Performance Benchmark (Wall-clock, Sequential) -Measured from `mvm vm create` to first successful `vm exec echo ok` on a single host (x86_64, NVMe SSD). Each row is one VM, created sequentially. +Historical pre-0.3.0 measurements from `mvm vm create` to first successful `vm exec echo ok` on a single host (x86_64, NVMe SSD). Each row is one VM, created sequentially; these figures do not qualify the current typed receiver. | Image | LoopMount (create → ready) | GuestFS (create → ready) | Slowdown | |---|---|---|---| @@ -80,11 +80,11 @@ Measured from `mvm vm create` to first successful `vm exec echo ok` on a single | debian:12 | 4.3s → **4.6s** | 13.5s → **13.5s** | **2.9x** | | firecracker:v1.15 | 1.9s → **2.1s** | 10.0s → **10.5s** | **5.0x** | -GuestFS is **3–5x slower** than LoopMount for VM creation. Each `guestfish` invocation spins up a full QEMU process with the libguestfs appliance to mount and modify the rootfs. Under **parallel** load, GuestFS degrades further (15–27s per VM) because concurrent QEMU instances contend for the libguestfs appliance lock, serializing on disk I/O. +In those historical measurements, GuestFS was **3–5x slower** than LoopMount for VM creation. Each `guestfish` invocation spins up a full QEMU process with the libguestfs appliance to mount and modify the rootfs. Under **parallel** load in the same measurements, GuestFS degraded further (15–27s per VM) as concurrent QEMU instances contended for the libguestfs appliance lock, serializing on disk I/O. The performance gap is an inherent property of the backend architecture, not an implementation issue. LoopMount modifies the rootfs via direct `mount` + `chroot`; GuestFS routes every filesystem operation through a QEMU-backed appliance. ## Related Decisions -- CONTEXT.md "Provisioner Backend" — mount/umount consolidated in `mvm run provision` subcommand (loop-mount path only). +- CONTEXT.md "Provisioner Backend" — typed LoopMount receiver and backend selection. - `benchmarks/results.json` — Full historical benchmark data for both backends. diff --git a/docs/adr/0005-sudo-privilege-architecture.md b/docs/adr/0005-sudo-privilege-architecture.md index 3d6f6c89..a7321f9f 100644 --- a/docs/adr/0005-sudo-privilege-architecture.md +++ b/docs/adr/0005-sudo-privilege-architecture.md @@ -1,10 +1,16 @@ # Sudo Privilege Architecture: Group-Based Passwordless Access -**Status:** Active +**Status:** Superseded for v0.3.0 by +[ADR-0016: Root-Owned Single-Binary Privileged Dispatch](0016-root-owned-single-binary-privileged-dispatch.md) **Date:** 2026-05-22 -**Last Updated:** 2026-06-20 (Go implementation complete) +**Last Updated:** 2026-09-27 (0.3.0 supersession note) -The project uses a **tightly scoped, group-based sudo model** where members of a dedicated Unix group (`mvm`) can execute specific privileged commands via `sudo` without a password prompt. This avoids running the entire CLI as root while still allowing VM creation (loop-mount), network management (ip/iptables/nft), kernel module loading (modprobe), and sysctl configuration. +> **Historical design, not 0.3.0 instructions:** ADR-0016 replaced the user-owned wildcard sudo design with a root-owned +> system installation and an early typed privileged dispatcher. The 0.3.0 source generates only the fixed +> `/usr/local/bin/mvm __mvm_privileged_v1 *` sudoers grant; the raw-tool grants and public-CLI wildcard described below +> are obsolete. Release qualification is separate from source implementation. + +This superseded design used a **group-based sudo model** where members of a dedicated Unix group (`mvm`) could execute specific privileged commands via `sudo` without a password prompt. It avoided running the entire CLI as root while allowing VM creation (loop-mount), network management (ip/iptables/nft), kernel module loading (modprobe), and sysctl configuration. The sections below preserve the historical design, not current setup instructions. **Table of Contents** @@ -34,10 +40,11 @@ Key implications: ## The Actual Sudoers File -`mvm host init` creates `/etc/sudoers.d/mvm` (path resolved by `infra.SudoersDropInPath()`). The file is generated by `GenerateSudoersContent()` (`internal/core/host/utils.go`, line 144) which uses `os.Executable()` to resolve the mvm binary path at generation time: +`mvm host init` creates `/etc/sudoers.d/mvm` (path resolved by `infra.SudoersDropInPath()`). In the current development +baseline, the file is generated by `GenerateSudoersContent()` with the fixed root-owned system path: ``` -%mvm ALL=(root) NOPASSWD: /usr/sbin/ip, /usr/sbin/iptables, /usr/sbin/iptables-restore, /usr/sbin/iptables-save, /usr/sbin/nft, /usr/sbin/sysctl, /usr/sbin/modprobe, /path/to/mvm * +%mvm ALL=(root) NOPASSWD: /usr/sbin/ip, /usr/sbin/iptables, /usr/sbin/iptables-restore, /usr/sbin/iptables-save, /usr/sbin/nft, /usr/sbin/sysctl, /usr/sbin/modprobe, /usr/local/bin/mvm * ``` The sudoers file is generated by `GenerateSudoersContent()` using two sources from `internal/infra/constants.go`: @@ -51,20 +58,22 @@ The sudoers file is generated by `GenerateSudoersContent()` using two sources fr - `/usr/sbin/sysctl` (procps) - `/usr/sbin/modprobe` (kmod) -- **Service binaries** — the `mvm` binary itself for the `mvm run provision` subcommand: - - `mvm` path resolved via `os.Executable()` + ` *` wildcard suffix +- **Service binary (transitional)** — fixed `/usr/local/bin/mvm *` for privileged public service subcommands. Task 14 + replaces this wildcard with only `/usr/local/bin/mvm __mvm_privileged_v1 *` after every caller is typed. NOTHING else is permitted. `sudo -n true`, `sudo cat /etc/sudoers.d/mvm`, `sudo test -f ...`, and any other non-listed command WILL prompt for a password. This is by design and is NOT a configuration problem. -## How VM Creation Works Without losetup in Sudoers +## How VM Creation Encapsulates Privileged Work + +All loop-mount operations are delegated to `mvm run provision`. Trusted Firecracker release installation, jail mounts, cleanup, and Jailer execution are delegated to `mvm run jailer`. Both reuse the single approved `mvm` sudoers entry instead of granting direct sudo access to dynamic binaries or mount tools. -This is the critical design insight: **all loop-mount operations are delegated to the `mvm run provision` service subprocess**. The call chain is: +The loop-mount call chain is: ``` mvm vm create → ImageProvisioner uses loop-mount backend (not guestfs) → LoopMountBackend.Run() calls runWireOp() (internal/lib/provisioner/loopmount/backend.go, line 188) - → runWireOp() resolves mvm binary path via os.Executable() + → runWireOp() resolves the current binary; supported host operation runs `/usr/local/bin/mvm` → Prepends "sudo" to args when !system.IsRoot() → system.DefaultRunner.Run(ctx, ["sudo", mvmPath, "run", "provision"], ...) ↑ sudoers allows %mvm to run mvm without password @@ -84,6 +93,7 @@ This means: - The sudoers file does NOT need `losetup`, `mount`, `umount`, `mkfs.ext4`, `blkid`, or any filesystem tool - ALL filesystem operations are encapsulated in the compiled `mvm` binary's `run provision` subcommand - The `mvm` binary is the single sudoers entry point for VM creation +- Jailer and Firecracker execute only from root-owned trusted release paths; users still invoke `mvm` without sudo ## How System Tests Interact with Sudo @@ -100,22 +110,29 @@ When running `sg mvm -c 'pytest tests/system/'`: ## Why Not Run Everything as Root -- **Principle of least privilege.** The CLI runs as a normal user for HTTP downloads, file I/O, SQLite queries, and all non-privileged operations. Only specific system binaries are elevated. -- **Safety and auditability.** The sudoers file is a compact, auditable list of exactly what can run as root. A compromise of the CLI does not automatically grant root access. -- **Development ergonomics.** Developers build and iterate as a normal user. Only one-time setup (`mvm host init`) needs `sudo`. +- **Principle of least privilege (target).** The CLI runs as a normal user for HTTP downloads, file I/O, SQLite queries, + and non-privileged operations. The transitional raw-tool and public-CLI grants remain broader than the named effects + accepted by ADR-0016 and are removed only at Task 14, after Tasks 8–13 migrate every caller. +- **Known transitional exposure.** Root ownership prevents replacement of the authorized executable, but + `/usr/local/bin/mvm *` still exposes broad public argument handling as root. A compromise of the ordinary CLI process + must be treated as potentially reaching that authority until typed callers and the marker-only policy replace it. +- **Development ergonomics.** Developers build and iterate as a normal user. Installing/upgrading the trusted system + image and machine-global `host init` remain explicit administrator operations. ## Why a Group-Based Model - The `mvm` group cleanly separates "can manage VMs" from "can't manage VMs" at the OS level - `sg mvm` activates the group for the current shell session without requiring logout/login, CI restart, or environment manipulation - System tests run under `sg mvm -c 'pytest ...'`, inheriting the group across all subprocesses at zero cost -- The setup is **one command** executed once: `sudo mvm host init` +- Initial setup is two explicit administrator steps: `sudo host install-system`, then + `sudo /usr/local/bin/mvm host init`. ## Consequences - `sudo -n true`, `sudo -v`, and any non-listed command will fail for `mvm` group members. The `CheckPrivileges()` function is aware of this and does NOT gate privileged execution on credential validation — it gates on group membership and session group activation. - Adding a new privileged binary requires updating `internal/infra/constants.go` (add to `PrivilegedBinaries` and `PrivilegedBinariesOrdered`) AND regenerating the sudoers file via `mvm host init`. -- The `mvm` binary MUST exist at the path listed in the sudoers file for VM creation to work. If it's missing — as can happen after `cache clean --force` — re-extract via `mvm cache init`. +- The root-owned binary MUST exist at `/usr/local/bin/mvm`. User cache cleanup does not remove it; reinstall it only from + a trusted artifact through `host install-system`. - The group name is defined as `MVMUnixGroup = CLIName` (`internal/infra/constants.go`, line 39). Currently `CLIName = "mvm"`, so the group is `mvm`. - `sg mvm` is a prerequisite for VM-related system tests. Non-VM tests work without it. - Tests that involve VM creation MUST verify the `mvm` group is active before attempting to create VMs. diff --git a/docs/adr/0006-arp-cache-flush-before-vm-spawn.md b/docs/adr/0006-arp-cache-flush-before-vm-spawn.md index be326db7..8d6862ad 100644 --- a/docs/adr/0006-arp-cache-flush-before-vm-spawn.md +++ b/docs/adr/0006-arp-cache-flush-before-vm-spawn.md @@ -1,8 +1,11 @@ # ARP Cache Flush Before VM Spawn: Stale Entries Inflate SSH Timing -**Status:** Active +**Status:** Superseded for 0.3.0 by [ADR-0021](0021-exact-managed-neighbor-invalidation.md) **Date:** 2026-05-22 +> This record describes the former bridge-wide `ip neigh flush` design, not current 0.3.0 production code. +> ADR-0021 retains the reused-IP goal with one typed, owner-bound neighbor invalidation. + The project flushes the bridge ARP cache (`ip neigh flush dev `) after creating the TAP device and before spawning Firecracker. Without this flush, a new VM that reuses an IP address from a recently-removed VM inherits the host's stale ARP entry pointing to the old (now-dead) TAP. SSH connections route to the dead interface, hit TCP SYN timeouts (~3s), and retry — inflating perceived boot time by 5-10s. **Table of Contents** diff --git a/docs/adr/0009-firewall-backend-mutual-exclusion.md b/docs/adr/0009-firewall-backend-mutual-exclusion.md index 028881d4..4e9a34d0 100644 --- a/docs/adr/0009-firewall-backend-mutual-exclusion.md +++ b/docs/adr/0009-firewall-backend-mutual-exclusion.md @@ -1,8 +1,13 @@ # Firewall Backend Mutual Exclusion — nftables vs iptables -**Status:** Active +**Status:** Superseded for v0.3.0 by +[ADR-0017: Per-VM Network Namespaces and Typed Connectivity Policies](0017-per-vm-network-namespaces-and-typed-policies.md) **Date:** 2026-05-22 +> Historical design: the remainder of this ADR describes the superseded dual-backend behavior, not current v0.3.0 +> instructions. The integrated v0.3.0 implementation uses nftables only, with no iptables backend or backend selector. +> Full release qualification remains incomplete; see ADR-0017 and the current coverage overlay. + The project provides two independent firewall rule tracking backends: **nftables** (default) and **iptables** (legacy). These backends are **mutually exclusive** — a single session uses exactly one backend, never a combination. The `firewall_backend` setting acts as a toggle selector. **Table of Contents** diff --git a/docs/adr/0012-unified-test-architecture.md b/docs/adr/0012-unified-test-architecture.md index dd66e185..3dd6ba81 100644 --- a/docs/adr/0012-unified-test-architecture.md +++ b/docs/adr/0012-unified-test-architecture.md @@ -1,114 +1,116 @@ -# Unified Test Architecture — Go Hermetic + Nested-VM E2E +# Unified test architecture with Go pre-filters and Python system tests -**Status:** Active +**Status:** Active, amended for the three-tier runner on 2026-08-26 **Date:** 2026-06-20 -**Table of Contents** - -- [Context](#context) -- [Decision](#decision) -- [Considered Options](#considered-options) -- [Consequences](#consequences) -- [Related Documents](#related-documents) - ## Context -### The problem - -The project had two test layers — Go unit tests (`go test ./...`) and Python system tests (`tests/system/`) — with a large gap between them. The system test layer has since been migrated to a three-level architecture (see implementation below): - -| Layer | Language | Speed | Determinism | Lines | -|-------|----------|-------|-------------|-------| -| **Go unit tests** | Go | ms | ✅ Deterministic | ~8,000 | -| *(gap)* | — | — | — | *0* | -| **System tests** | Python | min | ❌ Flaky | ~22,000 | - -The Python system tests (27 files, ~22,000 lines) suffer from seven structural problems: global shared state (~80% `@pytest.mark.serial`), pervasive skipping (43/341 skip-prone scenarios), dependency on real infrastructure, no parallel execution, tautological assertions, massive unfocused files, and `zzz_destructive/` ordering hacks. +At the time of this decision, the project had fast Go unit tests and a large Python system-test suite with little +coverage between them. Many Python tests shared host state, depended on ordering, skipped scenarios when infrastructure +was missing, or asserted only command return codes. Those properties made a passing run weak release evidence. -### What already works +The exact test counts and directory layout from that period are historical facts, not a live inventory. Read +`tests/system/COVERAGE_MATRIX.md` and the registry in `scripts/run-system-tests.py` for the current suite. -Per [ADR-0010](0010-per-domain-api-interfaces.md), the CLI layer accepts narrow `api.*API` interfaces. The `internal/testutil/` package provides in-memory repos, mock APIs, `FakeRunner`, and `FakeNetOps` — all Go-based, deterministic, and fast. These are currently only used for pure unit tests. - -The `test_vm_nested_isolated.py` file (in `tests/system/vm/`) proves this pattern works. - -### The "Zero Blindside" Principle - -When all tests pass, the team must be able to ship with confidence that every public-facing CLI command and flag works correctly. This requires exhaustive coverage (every path), deterministic execution (no skips, no flakes), isolated state (disposable environments), and includable destructive tests (cleanup verified the same way as creation). +The CLI already exposed narrow `api.*API` interfaces. `internal/testutil/` also provided in-memory repositories and a +`FakeRunner`. The project could therefore add hermetic Go coverage without mocking the production binary used by system +tests. ## Decision -Adopt a **three-level test architecture** with a two-language boundary: - -| Level | Name | Language | Tests | Requirements | -|-------|------|----------|-------|--------------| -| **L0** | Pure function | Go | Table-driven tests, no I/O, no repos | None | -| **L1** | Hermetic integration | Go | Real SQLite (`:memory:`), real files (`t.TempDir()`), `FakeRunner` for subprocess calls only | None | -| **L2** | Runner VM E2E | Python | Real binary, real infrastructure inside a disposable Firecracker VM | Runner VM (nested KVM) | +Use three test levels with a two-language boundary: -### Key Sub-Decisions +| Level | Purpose | Language | Boundary | +|---|---|---|---| +| L0 | Pure function checks | Go | No I/O, repositories, or subprocesses | +| L1 | Hermetic integration checks | Go | Controlled SQLite and files, with `FakeRunner` only at subprocess boundaries | +| L2 | Real-binary system checks | Python | Installed CLI and real Linux infrastructure in the execution tier selected by the runner | -1. **The mocked CLI handler layer (proposed "L2") is eliminated.** Mocking the API layer creates more maintenance than value — every new method needs a mock field, interface changes cascade, and assertions risk tautology. L1 tests cover the same ground without mocking: seed the real DB, call the real repository, check the output. +L0 and L1 are fast pre-filters. They do not replace L2 coverage for user-visible behavior. L2 verifies the installed +binary through the CLI and observes the resulting operating-system state. -2. **L2 covers ALL user-facing features.** Every CLI command, every flag, every output format, every error path — verified with the real binary running inside a runner VM. No scenario is removed during migration. The Python test files stay; only the execution substrate changes (from the real host to a disposable VM). +The system-test runner uses a separate tier classification inside L2: -3. **L0/L1 are fast pre-filters, NOT replacements for L2.** They catch bugs earlier during `go test ./...` (milliseconds instead of minutes), but passing L0/L1 alone means nothing — the feature could be completely broken at the subprocess level. The release gate is L2 passing. +| Tier | Execution location | Use | +|---|---|---| +| T1 | Disposable runner VM | Host-level CLI behavior that does not require nested KVM | +| T2 | Disposable runner VM with nested virtualization | VM lifecycle and interaction through nested Firecracker | +| T3 | Clean host-direct qualification machine | Behavior that cannot be qualified correctly under nesting | -4. **`zzz_destructive/` ordering-by-filename is eliminated.** Destructive tests stay and are essential — but the runner VM makes ordering a convenience concern, not a correctness requirement. Each session starts clean. Parallel workers each get their own VM. +Test levels answer what kind of proof a scenario needs. Execution tiers answer where an L2 scenario can run. Do not use +the level and tier numbers interchangeably. -5. **L0/L1 tests are written in Go, share file locations with existing unit tests** (`internal/*/*_test.go`), and follow [HOW_AGENTS_WRITE_UNIT_TESTS.md](../development/HOW_AGENTS_WRITE_UNIT_TESTS.md) patterns. +### Sub-decisions -6. **The full decision tree and quick-reference table are documented in [HOW_AGENTS_WRITE_SYSTEM_TESTS.md](../development/HOW_AGENTS_WRITE_SYSTEM_TESTS.md).** That document is the authoritative how-to reference for classifying and writing tests at all three levels. +1. Do not add a mocked CLI-handler layer. L1 calls real package behavior with controlled dependencies. L2 calls the real + installed binary. +2. Keep user-visible behavior in the Python system-test coverage matrix. Run each scenario in T1 or T2 when nesting can + represent it. Reserve T3 for the documented host-only exceptions. +3. Keep destructive tests. Disposable runners and scoped fixtures provide isolation. Filename ordering is not a safety + mechanism. +4. Keep L0 and L1 tests next to Go source. The `engineer` role owns them. +5. Keep Python L2 tests under `tests/system/`. The `qa-engineer` role owns them and the runner. +6. Treat collection, skips, deselection, cleanup failure, and malformed outcome reports as release failures. A zero exit + code alone is not sufficient evidence. +7. Keep the installed outer controller separate from the release candidate. Install the candidate inside T1 and T2 + runners. T3 requires exact candidate identity at the canonical system path on a clean qualification machine. -## Considered Options +## Considered options -### Option A: Full Rewrite of Python Tests in Go (rejected) +### Rewrite the Python suite in Go -Translate all 22,000 lines of Python to Go. Loses pytest's fixture scoping, parametrization, marker filtering, yield-based teardown, and plugin ecosystem. Would need extensive custom infrastructure for runner VM orchestration. The cost of rebuilding pytest's capabilities in Go exceeds the benefit of a single language. +Rejected. Pytest already provides fixture scoping, parametrization, filtering, and teardown. Rebuilding those facilities +would not improve the real-binary boundary. -### Option B: Build Tag for Hermetic Binary (rejected) +### Build a tagged mock binary -Add a `mvmtest` build tag that replaces `DefaultRunner` with a controllable fake. Creates a second testing surface (does the injection work? does the env var parse correctly?). The binary that runs in CI would differ from production. L1 tests achieve the same result without a build tag by testing the handler function directly. +Rejected. A build tag would create a second executable behavior that release testing does not ship. L1 already provides +controlled dependency tests without changing the candidate binary. -### Option C: Three-Level Architecture (selected) +### Keep only Go unit tests and host-direct Python tests -Go for fast pre-filters. Python (inside nested VM) for ground truth. Each layer uses the right tool for the job. Incremental migration — no big-bang rewrite. +Rejected. Go-only tests miss subprocess and Linux integration failures. Host-direct tests expose developer state and +make cleanup failures dangerous. -### Option D: Status Quo (rejected) +### Use Go pre-filters and isolated Python system tests -Keep 22,000 lines with shared DB, pervasive skipping, and per-domain serial execution. The problems are structural, not cosmetic — no amount of test hardening fixes a shared-DB architecture. +Selected. Go catches local defects early. Python verifies the installed CLI against real infrastructure. Disposable +runners contain most destructive behavior, while T3 remains an explicit release-only exception. ## Consequences -**Positive:** -- Runner VM provides hermetic isolation — no shared DB, no orphaned bridges, no sudo on the host -- Parallel execution — multiple runner VMs, no `serial` markers -- `zzz_destructive/` ordering hack eliminated -- Skip culture eliminated — runner VM starts from pre-seeded snapshot -- L0/L1 fast pre-filters speed up the developer inner loop (milliseconds instead of minutes) -- SQLite-backed L1 tests catch schema bugs the current map-based mocks cannot detect - -**Negative:** -- Migration effort (estimated 8-12 weeks) -- Two languages, two test runners -- Runner VM provisioning time (~60-120s first run, mitigated by snapshot/restore) -- Runner VM resource overhead (~4 vCPU + 4 GB RAM + 20 GB disk per VM) -- ~2,000 new Go test lines to maintain -**Neutral:** - -- Per-domain conftest.py files consolidated into one -- `tests/system/` → `tests/e2e/` rename (pending — directory still at `tests/system/`) -- `COVERAGE_MATRIX.md` replacement with `go test -coverprofile` + L2 scenario manifest (pending — `COVERAGE_MATRIX.md` still exists) -- `scripts/run-system-tests.py` domain-looping replaced by `pytest tests/system/` (pending — CI still references `scripts/run_tests.py`) - -## Related Documents - -| Doc | Purpose | -|-----|---------| -| `docs/system-test-architecture.md` | **Primary implementation reference** — Three-tier orchestration, provisioning flow, fixture scoping, known-limitation patterns, per-file compliance checklist. Supersedes `DRAFT-system-test-architecture.md`. | -| `docs/development/HOW_AGENTS_WRITE_SYSTEM_TESTS.md` | L0/L1/L2 definitions, decision tree, quick-reference table, runner VM fixture pattern, migration phases. | -| `docs/development/HOW_AGENTS_WRITE_UNIT_TESTS.md` | L1 fast pre-filter tests (in-memory SQLite, temp dirs). | -| `docs/development/HOW_TO_RUN_SYSTEM_TESTS.md` | Execution plan for running system tests. | -| _(no RC_QA.md — this doc section moved into system-test-architecture.md)_ | Release gates: "L2 tests in runner VM" now in system-test-architecture.md. | -| `STANDARDS.md` §12 | Three-level architecture description. | -| `tests/system/COVERAGE_MATRIX.md` | Coverage tracking. | -| `AGENTS.md` | `engineer` owns L0/L1 (Go), `qa-engineer` owns L2 (Python runner VM). | +Positive consequences: + +- L0 and L1 failures arrive before expensive VM provisioning. +- T1 and T2 isolate databases, bridges, processes, and destructive cleanup from the outer host. +- The runner can execute independent domains concurrently in separate VMs. +- Strict outcome reports distinguish a real pass from an empty, skipped, or partially collected run. + +Costs and limits: + +- The project maintains Go and Python test code. +- T1 and T2 require runner provisioning and nested virtualization support. +- T3 still mutates the outer machine. It is release-only and requires a clean qualification host. +- The coverage matrix and executable registry must remain synchronized. + +## Current implementation record + +- `tests/system/` remains the canonical Python system-test directory. +- `scripts/run-system-tests.py` is the only supported outer orchestrator. +- `tests/system/COVERAGE_MATRIX.md` remains the coverage inventory. +- T1 and T2 use controller and candidate separation. +- T3 requires explicit `--host-direct`. Release qualification verifies candidate identity. The remaining clean-host + ownership and teardown gates stay release blockers in Task 17. +- Tier assignment and fixture scoping replace a global `pytest.mark.serial` convention. + +## Related documents + +| Document | Authority | +|---|---| +| `docs/system-test-architecture.md` | Architecture, tiers, fixture scope, runner protocol, and known limitations | +| `docs/development/HOW_AGENTS_WRITE_SYSTEM_TESTS.md` | L0, L1, and L2 classification plus system-test authoring patterns | +| `docs/development/HOW_AGENTS_WRITE_UNIT_TESTS.md` | Go test authoring patterns | +| `docs/development/HOW_TO_RUN_SYSTEM_TESTS.md` | Operator procedure for executing the runner | +| `docs/RC_QA.md` | Release-candidate qualification and evidence | +| `tests/system/COVERAGE_MATRIX.md` | Command, flag, and scenario coverage | +| `AGENTS.md` | Role ownership and safety rules | diff --git a/docs/adr/0013-user-facing-timeouts-are-connect-timeouts.md b/docs/adr/0013-user-facing-timeouts-are-connect-timeouts.md index d717f93d..f439b63e 100644 --- a/docs/adr/0013-user-facing-timeouts-are-connect-timeouts.md +++ b/docs/adr/0013-user-facing-timeouts-are-connect-timeouts.md @@ -67,6 +67,15 @@ Matches user expectation and common CLI conventions (e.g., `curl --connect-timeo ## Implementation Notes +### Environment readiness exception (2026-10-16) + +The approved `ready` environment step is a bounded readiness assertion. Its YAML `timeout` is a positive total +deadline (default 60 seconds) covering probes, guest check execution, and retry waits; `interval` defaults to one +second. A deadline is necessary for a readiness gate to fail rather than hold dependent steps indefinitely. +This narrow exception does not change `exec.timeout`, `ssh.timeout`, or public CLI `--timeout` semantics. + +### Ordinary execution + - `internal/core/ssh/service.go`: `s.timeout` is now probe-only. `Connect` and `StreamCommand` run the actual command with `Timeout: 0`. - `pkg/api/exec.go`: `input.Timeout` overrides `defaults.vm.vsock_probe_timeout`; `client.Exec` receives `0` for the command timeout. - CLI help text updated to say "connect/probe timeout". diff --git a/docs/adr/0014-routed-service-access-policies.md b/docs/adr/0014-routed-service-access-policies.md new file mode 100644 index 00000000..060b6904 --- /dev/null +++ b/docs/adr/0014-routed-service-access-policies.md @@ -0,0 +1,56 @@ +# Routed Service-Access Policies + +**Status:** Superseded for v0.3.0 by +[ADR-0017: Per-VM Network Namespaces and Typed Connectivity Policies](0017-per-vm-network-namespaces-and-typed-policies.md) +**Date:** 2026-08-09 +**See also:** [Project Hosting with MicroVMs](../roadmaps/TENANT_MICROVM.md), [ADR-0009: Firewall Backend Mutual Exclusion](0009-firewall-backend-mutual-exclusion.md) + +> This ADR remains the description of the currently implemented transitional routed-only policy. v0.3.0 replaces it +> with mandatory per-VM namespaces, same- and cross-network default deny, and typed `traffic` and `exec` policies. + +## Context + +mvmctl networks are independent Linux bridges and subnets. VMs on one bridge communicate directly at layer 2, while traffic between bridges is routed by the host because IPv4 forwarding is enabled. Before this decision was implemented, the managed firewall permitted outbound NAT and installed no default deny between mvmctl bridges. + +Project orchestration remains outside mvmctl. The infrastructure requirement is narrower: permit a source mvmctl network to reach one exact destination VM service, such as TCP port 5432 on a shared PostgreSQL VM, while denying other routed traffic between managed networks. The policy must survive VM restarts and firewall reconciliation without exposing raw nftables or iptables syntax. + +The persisted policy also needs a stable meaning if same-network filtering is added later. Routed packets belong in the IP-family FORWARD path. Same-bridge packets bypass that path because `bridge-nf-call-iptables=0` and would require nftables bridge-family enforcement. These mechanisms are additive, not replacements for one another. + +## Decision + +mvmctl will persist a typed **service-access policy** identified by source network, destination VM, protocol, and destination port or port range. + +- The policy stores resource identities, not interface names, IP addresses, chain names, or backend expressions. +- The API layer resolves the source network and destination VM, then passes trusted model values to the network domain. This preserves core-domain isolation. +- The network domain owns policy persistence and compiles policies into derived firewall rules. `internal/lib/firewall` continues to own only backend-specific rule application and reconciliation. +- Different-network policies compile to IP-family FORWARD rules for both nftables and iptables. +- Explicit allows precede a final default DROP for traffic routed between mvmctl-managed bridges. Established and related reply traffic remains allowed, and internet egress remains unchanged. +- VM-to-host traffic is controlled separately in INPUT. Required NoCloud access and explicitly approved host services precede a final drop for traffic originating from mvmctl networks. +- Future same-network VM-to-VM policies will add a typed source-VM operation and compile to nftables bridge-family rules. Existing policies, commands, and stored records will not change. +- No raw firewall-rule editor, generic selector registry, or silent backend fallback will be introduced. + +The implemented IP-family compiler uses `MVM-ROUTED-POLICY` and `MVM-HOST-INPUT`. Both backends place connection-state handling first, explicit service allows next, and an internally derived terminal drop last. Managed-interface matching is semantic in persisted derived state and expands to the backend's interface-prefix syntax only when rules are rendered. The initial host exception set contains NoCloud; additional host services require a future typed operation rather than raw INPUT rules. + +## Consequences + +**Benefits:** + +- External orchestrators can model environments and shared services without introducing tenant concepts into mvmctl. +- Policy intent remains stable when VM IPs, TAP names, or firewall backends change. +- Routed policies work with both existing firewall backends. +- Bridge-family enforcement can be added without replacing the policy model. +- Idempotent reconciliation can restore rules after host reboot or UFW reload. + +**Costs and limitations:** + +- Rule ordering becomes a correctness requirement. Service allows and connection-state handling must always precede default deny. +- Removing or recreating a VM or network invalidates identity-bound policies; policies do not silently attach to replacement resources. +- VM-to-host restrictions require careful NoCloud and host-service system tests. +- The initial host-input policy permits NoCloud and established replies but no user-defined host services. +- Same-network filtering remains open until the bridge-family compiler is implemented. When implemented, that compiler will require nftables; routed policies retain iptables parity. + +## Related Decisions + +- ADR-0009: Firewall backend mutual exclusion remains unchanged. +- ADR-0010: The API layer resolves cross-domain policy inputs; Core domains remain isolated. +- ADR-0011: Policy inputs use typed `Validate` and `Resolve` methods. diff --git a/docs/adr/0015-jailer-canonical-launcher.md b/docs/adr/0015-jailer-canonical-launcher.md new file mode 100644 index 00000000..4c811c9e --- /dev/null +++ b/docs/adr/0015-jailer-canonical-launcher.md @@ -0,0 +1,68 @@ +# Jailer Is the Canonical Firecracker Launcher + +**Status:** Active +**Date:** 2026-08-09 +**Last Updated:** 2026-09-27 (memory-policy supersession and current namespace handoff) +**See also:** [Project Hosting with MicroVMs](../roadmaps/TENANT_MICROVM.md), [ADR-0005: Sudo Privilege Architecture](0005-sudo-privilege-architecture.md), [ADR-0016: Root-Owned Single-Binary Privileged Dispatch](0016-root-owned-single-binary-privileged-dispatch.md) + +The finite per-VM memory maximum originally described here is superseded by +[ADR-0019](0019-pressure-only-vm-memory-control.md). The canonical-Jailer decision remains active. + +## Context + +mvmctl originally downloaded both Firecracker and Jailer but launched Firecracker directly. Direct launch preserved console file descriptors and simple PID tracking while omitting Firecracker's recommended production process containment. + +Firecracker's production guidance states that Jailer and Firecracker must have matching versions and that all trusted executable and chroot paths must not be writable by unprivileged users. Jailer requires elevated permissions to prepare the jail and cgroups, while mvmctl's established contract is that users invoke the CLI without root and privileged work is delegated internally through the constrained service-subprocess pattern. + +The launch mechanism affects VM creation, restart, snapshot restore, console descriptors, API and vsock sockets, logs, volume paths, PID tracking, cleanup, and performance. A fallback from failed Jailer launch to direct Firecracker would silently remove the expected security and resource guarantees. + +## Decision + +Jailer will be the canonical launcher for every normal mvmctl VM lifecycle operation. + +- Firecracker and Jailer are resolved and validated as one exact-version pair. Missing or mismatched pairs fail before launch. +- Jailer launch replaces direct Firecracker execution; there is no silent fallback and no generic launcher extension interface. +- The existing VM-domain spawner remains the caller-facing seam and owns jailed path translation, process tracking, socket probing, and cleanup. +- Root-requiring setup enters the reserved early privileged dispatcher in the same `mvm` binary; the public + `mvm run jailer` service is removed at the v0.3 cutover. Users continue to invoke `mvm` without sudo. +- Executable and chroot paths trusted by Jailer are root-owned and not writable by the invoking user. The ordinary per-user mvmctl database and cache model remains unchanged. +- Every launch requires cgroup v2 with the CPU, memory, and PID controllers. mvmctl persists a typed resource envelope, derives fixed Jailer arguments internally, and fails the launch if post-start membership or values differ. Raw cgroup key/value input is not public. +- Jailer never uses `--daemonize` or `--new-pid-ns`. The receiver-owned PID and pidfd must continue to identify the + same task through Jailer and Firecracker `exec`, while serial-console standard descriptors remain attached. The + pidfd reports task exit, not `exec`; the receiver uses bounded `/proc//exe` observation while retaining the + original pidfd. The mandatory per-VM network namespace is prepared through retained root authority; the blocked + child joins its pinned namespace before executing Jailer and passes that namespace through Jailer's fixed `--netns` + argument. Managed topology and firewall activation must complete before the child is released. Networked installed- + candidate qualification remains pending. +- Before executing Jailer, root proves the fixed per-VM destination absent. It may remove only an exactly admitted, + non-mounted stale tree after process absence is proved; ambiguous, mounted, replaced, or unexpected state fails + unavailable. Jailer must freshly create the destination whose final root and copied executable are activated. +- VM resources are exposed inside the jail through validated paths without duplicating large rootfs and volume files. Mount cleanup is idempotent across failed starts, stop, remove, and host recovery. + +## Consequences + +**Benefits:** + +- Every VM receives Firecracker's supported chroot and privilege-drop path. +- The canonical Jailer path enforces one typed cgroup-v2 resource policy for every VM lifecycle path. +- One canonical launch path avoids a permanent direct-versus-jailed behavior matrix. +- VM create, start, reboot, and snapshot restore share the same security properties. + +**Costs and limitations:** + +- Existing host paths in Firecracker configuration must be translated into jail-visible paths while host controllers retain corresponding socket and artifact locations. +- Console, vsock, snapshots, hotplug volumes, PID handling, and partial-failure cleanup require full parity testing. +- Jailer adds fixed launch overhead and depends on the number of host mount points; benchmarks must track the regression budget. +- Under ADR-0019, `memory.high` is guest-visible RAM plus configurable VMM headroom (128 MiB by default) and the + per-VM `memory.max` is `max`. This pressure-only policy is not a strict host-memory ceiling; sustained saturation + and host/ancestor OOM behavior still need runtime qualification. +- The root-side namespace and Jailer handoff have source implementation, but managed-network lifecycle, + firewall/topology reconciliation, and bridge-capable installed-candidate system qualification remain release gates. +- Release pairs are reinstalled into the trusted store by `mvm bin pull`; hosts without cgroup v2 and the CPU, memory, and PID controllers cannot launch VMs. + +## Related Decisions + +- ADR-0002: Privileged Jailer work remains inside the single `mvm` binary. +- ADR-0005: Normal-user operation remains mandatory; ADR-0016 replaces its pre-v0.3 wildcard sudo boundary at cutover. +- ADR-0012: Real Jailer/cgroup behavior requires L2 verification. +- ADR-0013: Socket readiness remains an internal probe, not a total launch timeout. diff --git a/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md b/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md new file mode 100644 index 00000000..8dd09ebb --- /dev/null +++ b/docs/adr/0016-root-owned-single-binary-privileged-dispatch.md @@ -0,0 +1,1977 @@ +# Root-Owned Single-Binary Privileged Dispatch + +**Status:** Accepted; source integrated; candidate-bound installed qualification checkpoint accepted, with documented limits +**Date:** 2026-08-20 +**Last Updated:** 2026-10-16 (candidate-bound qualification status) + +The [coverage checkpoint](../../tests/system/COVERAGE_MATRIX.md) records the accepted 528/528 mixed-source evidence +join. Dated foundation and partial-pass notes below remain historical; they are not the current release verdict. +Unlisted fault scenarios, aarch64 runtime, and final release approval remain outside that checkpoint. + +**See also:** [ADR-0002: Single Go Binary Architecture](0002-single-go-binary-architecture.md), +[ADR-0005: Sudo Privilege Architecture](0005-sudo-privilege-architecture.md), +[ADR-0015: Jailer Is the Canonical Firecracker Launcher](0015-jailer-canonical-launcher.md), and +[ADR-0017: Per-VM Network Namespaces and Typed Connectivity Policies](0017-per-vm-network-namespaces-and-typed-policies.md) + +## Context + +mvmctl is a Go CLI whose user-facing commands and subprocess services are compiled into one release artifact. The CLI +is intended to run as an unprivileged user while a small set of VM, filesystem, network, firewall, cgroup, and Jailer +operations require root. + +The existing installation contract puts `mvm` at user-owned `~/.local/bin/mvm` and generates a passwordless sudoers +entry for that path with wildcard arguments. A member of the `mvm` group can replace the executable and then ask sudo to +run the replacement as root. Root ownership checks performed later by Jailer cannot repair a compromised executable +boundary. + +The existing sudoers policy also grants raw argument access to host tools such as `ip`, `iptables`, `nft`, `modprobe`, +and `sysctl`. Those tools expose substantially more host authority than the named mvmctl operations that need them. The +policy therefore does not provide the targeted access claimed by ADR-0005. + +Introducing a separately compiled privileged helper would provide a small executable trust boundary, but it would break +the project's single-binary product contract. A safe design must retain one artifact without allowing the ordinary CLI +initialization path or caller-controlled program bytes to execute as root. + +## Decision + +### One root-owned system installation + +Manually built and release-script artifacts are installed at `/usr/local/bin/mvm` as `root:root`, mode `0755`. The +directory chain must be root-owned and not group/world writable. `/usr/bin/mvm` is reserved for a future installation +owned by a distribution package manager. + +Normal users execute `/usr/local/bin/mvm` without sudo. The file contains both the public CLI and the internal privileged +dispatcher, preserving one release artifact and one installed executable. Replacing or upgrading the system executable +is an explicit administrator action; a normal user update cannot overwrite it. + +Development builds may exist elsewhere, but they are never sudo targets. A development build that needs privileged +operations calls the compatible system installation and fails closed on a privileged-protocol mismatch. + +### Administrator bootstrap and system-binary replacement + +System installation is selected by the exact `host install-system` command before `app.Initialize()`, Cobra, signal +setup, user configuration, the user database, cache resolution, logging configuration, or service construction. The +bootstrap route clears the inherited environment, installs only fixed root-safe values, and then atomically copies the +running image to `/usr/local/bin/mvm`. Flags and extra arguments are rejected inside the same reserved route and cannot +fall through to the ordinary CLI. + +The installer cannot prove whether an already-root process reached it through password-authenticated administrator sudo +or an unintended passwordless rule. It therefore inspects the project-managed sudoers drop-in before touching the +executable and refuses unrecognized active syntax. On a clean host, an administrator runs +`sudo host install-system` and then runs `sudo /usr/local/bin/mvm host init`. Installation is not an +implicit side effect of `host init`. + +v0.3.0 does not use this installer to convert, adopt, or preserve a pre-v0.3 installation. Operators must remove the old +managed sudoers rule and clean the old runtime/state layout in an authenticated administrator session before installing +v0.3.0. Later replacement of an already-v0.3-compatible system binary remains an explicit administrator action and +must fail closed when the privileged protocol is incompatible. + +Replacing the executable is descriptor-relative, no-follow, temporary-file plus file fsync plus rename plus directory +fsync. Failures after rename report that replacement occurred and whether directory durability is uncertain. A normal +`self-update` may update a user-owned artifact, but it refuses to overwrite the system installation and prints the +administrator bootstrap instruction. + +Implementation note (2026-09-13): `ConfigureSudoers` verifies the canonical system executable before admitting the +fixed root-owned directory chain and bounded one-link `0440` policy. It checks required tools, sends generated bytes +to `/usr/sbin/visudo -c -f -` through the standard runner with a 15-second timeout, bounded output, and a fixed +environment, then publishes a descriptor-relative candidate by file fsync, atomic rename, and directory fsync. +Failures preserve replacement, uncertain durability, and cleanup evidence. Existing transitional grants remain exact; +this does not enable the marker-only policy or complete administrator init/reset. + +### Host administration admission and reset + +Administrator mutation and operator dispatch share the fixed root-owned host journal lock. +Operator requests retain a shared admission lease, check the durable enabled state, and recheck +operator authorization after acquiring that lease. The lease covers dispatch and cleanup, including +retained sessions. Administrator init/reset takes the exclusive lease; acquisition is cancellable +and bounded. Initialization enables admission only after completing the journaled setup. + +Reset must retain the Jailer global inventory guard, then the network/global firewall guard, and +prove every owner's managed runtime state cleaned or absent before disabling admission and restoring +the recorded host state. It does not force-stop VMs. Queued requests cannot proceed on their earlier +authorization after reset; they must acquire shared admission and observe the disabled state. +Incomplete rollback retains disabled admission and its journal for explicit administrator retry. + +Restoration compares exact recorded group identity, membership, applied sudoers bytes, and managed +settings; it preserves administrator changes rather than treating the old snapshot as unconditional +overwrite authority. Marker-only sudoers is a reset prerequisite: transitional raw-command grants +can bypass dispatcher admission and therefore cannot establish global quiescence. Missing or +malformed admission state refuses operator requests; only explicit administrator initialization can +publish enabled state. This is an implementation contract, not a completed qualification claim. + +### Early privileged dispatch + +`cmd/mvm/main.go` recognizes one reserved, versioned privileged marker before signal setup, `app.Initialize()`, Cobra, +user configuration, the user database, logging configuration, or plugin/service discovery. Once selected, this path can +only enter the privileged dispatcher; it cannot return to or invoke an ordinary CLI command. + +The dispatcher: + +- requires effective UID 0 and derives the invoking identity from sudo's trusted execution context; +- verifies the system executable and its directory chain are root-owned and not group/world writable; +- verifies that the invoking user is currently authorized by the `mvm` group; +- sanitizes its environment and uses absolute paths for root-side subprocesses; +- accepts only a versioned, bounded, typed request for a named operation; +- rejects unknown actions, extra arguments or fields, invalid identifiers, unsupported versions, and unchecked paths; +- never accepts raw commands, arbitrary argv, shell fragments, raw firewall syntax, or raw cgroup key/value input. + +The sudoers entry may contain a wildcard after the reserved marker because operation arguments vary, but that wildcard +can reach only the early dispatcher. It cannot select the public CLI. The dispatcher remains the authoritative +allowlist, and every handler performs exact argument-count and type checks before side effects. + +Synchronous actions return one bounded, versioned response envelope on their control channel. Success payloads are +action-specific. Failures preserve the original `DomainError` code, class, entity, and bounded partial-state details so +the normal-user client can make safe retry and recovery decisions. Logs and subprocess exit status are diagnostic only; +they never replace the structured result or turn a malformed/missing response into a generic successful operation. + +The control channel is one full-duplex Unix stream socket duplicated onto file descriptor 0 across `sudo -n`. File +descriptor 0 is control-only and is marked close-on-exec before any handler can launch a descendant. Standard output and +standard error are never parsed as authority. VM console input/output must use a separate typed relay owned by the launch +implementation; it cannot reuse the control channel. Using descriptor 0 avoids sudo policy for inherited descriptors, +while the socketpair permits the client to upload a bounded release archive and receive an early rejection concurrently. +Root requires descriptor 0 to be an `AF_UNIX` `SOCK_STREAM` socket and requires its Linux `SO_PEERCRED` UID/GID to match +the authenticated sudo caller. The peer PID is positive and audit-only; it is not an authorization identity. The +production receiver owns `os.Stdin` directly and accepts no alternate reader or descriptor from its entry point. + +The transport always executes exactly +`/usr/bin/sudo -n -- /usr/local/bin/mvm __mvm_privileged_v1 `. It never uses `PATH`, `os.Executable`, the +current user binary, a caller-supplied executable/argv/environment/cwd/timeout, or a root-EUID bypass; the receiver +requires authenticated non-root sudo identity. `internal/lib/system` provides one concrete transport rather than +extending the generic command runner. Each capability package owns a minimal private `Exchange` interface and named +typed public methods; no public generic privileged call/action/decoder hook is introduced. Hermetic tests inject private +process dependencies and use a dedicated fake exchanger without extending `CommandRunner`, `RunCmdOpts`, `SpawnConfig`, +or the existing `FakeRunner`. + +Each request starts with fixed `MVMREQ01` magic, a network-order 32-bit JSON-header length, and a network-order 64-bit +payload length. The strict header is at most 64 KiB and contains schema version 1, the action repeated from argv, and the +action-specific body. The action is 1 through 64 ASCII bytes, starts with an ASCII alphanumeric byte, and thereafter +contains only ASCII alphanumeric bytes, `.`, `_`, or `-`; the caller and receiver enforce the same grammar. The repeated +action must match. Only release install may carry a payload: either zero bytes or an exact stream of at most 128 MiB. +The caller half-closes its write side after the declared bytes, and root requires EOF so truncation or trailing bytes +fail closed before extraction, canonical publication, or promotion of unadmitted bytes. Anonymous staging may occur +while the declared stream is read. Zero-payload operations require EOF before effects. +Authentication, framing, and header failures may return an early error without draining an advertised payload; the +concurrent client treats the resulting uploader `EPIPE` as expected once it has a valid response. The request-header +read deadline is a fixed five seconds, and the release-install archive-payload read deadline is a fixed five minutes. +Each uses the earlier current context deadline when one exists. These are non-configurable security policy; the +receiver reads no second request and does not accept a caller-selected timeout. + +Each response starts with fixed `MVMRES01` magic and a network-order 32-bit JSON length followed by at most 64 KiB of +strict schema-version-1 JSON. It contains the matching action, a closed `success` or `error` status, and exactly one typed +result or error. Error envelopes carry code, stable string class, message, operation, entity, and normalized details; +they never serialize wrapped causes, stacks, stderr, or arbitrary Go values. Wire details permit only bounded booleans, +strings, signed integers, and string arrays. Unsupported details are omitted visibly. Root half-closes the control +socket's write side after the complete frame, so the client can validate response EOF independently of process exit. An +error response delivered after all effects and cleanup exits successfully at the process layer. The codec returns a +decoded success/error outcome separately from protocol failure, so a valid root `DomainError` cannot be mistaken for a +malformed frame. A start failure has known no-effect status; any started process with a missing, malformed, truncated, +mismatched, oversized, or non-EOF response returns `CodeProcessError` with `process_started: true` and +`outcome_unknown: true`. + +Every success or error response establishes its fixed five-second absolute write deadline immediately before delivery +and uses the earlier current context deadline when one exists. A valid long-running effect therefore does not consume +the response-write budget. Final error-envelope delivery computes that absolute deadline before applying +`context.WithoutCancel`; it never gains a fresh or unbounded interval after cancellation is detached. The +request-header, archive-payload, and response-write limits are compiled, non-configurable security policy. + +The receiver rejects an initially nonblocking fd 0. Response writes do not depend on mutable shared file-status flags: +they use raw `SendmsgN` with `MSG_DONTWAIT|MSG_NOSIGNAL` and bounded `POLLOUT` waits against the already-established +absolute deadline. A duplicate descriptor changing `O_NONBLOCK` after admission therefore cannot bypass the deadline, +turn a full peer buffer into an immediate unclassified failure, or deliver `SIGPIPE` to the root process. + +The transport returns bounded raw response bytes only together with an explicit indication that actual socket EOF was +observed; copying a complete-looking prefix into an in-memory reader cannot fabricate authority. The named capability +client decodes an EOF-qualified response with a non-cancelled context and lets either a valid success or valid remote +`DomainError` win over upload `EPIPE`, non-zero exit, or later reap diagnostics. Without one valid envelope, a pre-start +failure reports `process_started: false, outcome_unknown: false`; every post-start failure reports both values as true. + +### Retained VM-exec authority-sidecar exception + +The implemented and registered `jailer-vm-exec-session` action is the sole leased-lifetime exception to the +ordinary request/response rule above. It still uses the exact system binary, marker, action switch, sudo invocation, +and fd-0 AF_UNIX control socket. It handles one relay lease and exits; it is not a helper binary, daemon, generic +privileged channel, or second protocol listener. File descriptor 0 remains control-only. No `ExtraFiles`, `sudo -C`, +sudoers extension, environment input, caller path, raw command, or generic operation is permitted. + +The normal `pkg/api` process opens and authenticates the existing source Firecracker connection. Only after it has read +the strict source request, resolved the exact destination, and allowed the directional policy does it start this action. +The initial closed request names the exact source/destination VM IDs and attaches exactly one source connection duplicate +with `SCM_RIGHTS` on the first request byte. After root pair admission, the normal process performs only the destination +Firecracker UDS `CONNECT` handshake, and a closed second phase attaches exactly one destination connection duplicate. +Every ancillary receive uses `MSG_CMSG_CLOEXEC`, rejects truncation and missing, extra, unknown, duplicate, or late +control messages, and proves one connected non-listening AF_UNIX stream. The action accepts no socket path, agent token, +target user, command, policy material, or response payload. + +Root binds each live connection to its exact retained Firecracker record with `SO_PEERCRED`, `SO_PEERPIDFD`, PID/start +identity, complete active-process verification, and final liveness. Socket metadata proves the Firecracker peer, not +the selected guest port; the normal process's independent `SourceSession` token/bootstrap handshake proves the source +agent endpoint. Root reserves the exact active pair at the compiled host-global ceiling of eight, retains the +pair/capacity lease and both connection duplicates, and returns ready only after both FD proofs pass. Only after that +ready response may the normal process send the destination's token-and-command-bearing `DestinationRequest`; the +destination agent validates it before any effect. + +The action-specific, closed, versioned session frames are bounded independently of ordinary `MVMRES01`; they cannot be +selected by another action. Explicit release, control EOF, caller crash, setup failure, or the absolute +`CLOCK_BOOTTIME` relay-stop deadline shuts down the shared connection descriptions and performs cancellation-detached +reverse cleanup before the child exits. The capacity record's four-second terminal/cleanup grace never extends +execution. A malformed or missing final outcome after process start remains outcome-unknown, and no retained authority +is inferred from process exit alone. + +This exception preserves the ordinary transport unchanged for every other action. Production dispatch constructs the +retained server from the exact active-pair, global-capacity, Firecracker-peer, and absolute-stop authorities and +registers only `jailer-vm-exec-session`. The caller UID/GID passed to that server derives exclusively from the +authenticated control peer identity and never from request fields. Each source or destination descriptor remains an +opaque handler-owned capability, pinned and immutable through its sole synchronous binder call. + +Authenticated failures that occur after control-channel structural admission but before action dispatch use the +action-specific `MVMXSES1` terminal failure, not ordinary `MVMRES01`, and do not drain request input. After dispatch, +the retained handler is the sole response and cleanup owner: it shuts down destination then source, releases the stop, +then releases destination/source peer pidfds before capacity, followed by process capabilities, active leases, and +pair roots. The registered action still has no API-owned relay listener/consumer; it receives no policy or relay +payload, and registration alone does not create a positive VM-to-VM path. + +Receiver integration landed in two audit-friendly increments before any release action is registered. The first +increment admits only fd 0, sets and verifies close-on-exec, validates blocking `AF_UNIX`/`SOCK_STREAM` and +`SO_PEERCRED`, emits bounded early error envelopes only after structural channel admission and when a valid version-1 +argv action is available, completes the response frame before `SHUT_WR`, and removes the obsolete standalone +EOF-delimited JSON decoder. A rejected closed action is not drained. The second increment adds closed, typed +trusted-release install/remove request and result values plus directly tested handlers. Install alone may consume its +declared payload through Jailer-owned exact-length and EOF admission; zero-payload install and remove require EOF before +their effects. Both handlers remained unregistered until the named Jailer client capability and its response-authority +tests completed in slice 6; they are now exact entries in the central switch. + +### Implemented trusted-release handlers + +Slice 5b implements the approved contract without registering either action. The two exact compiled internal actions +are `jailer-release-install` and `jailer-release-remove`. Their constants and all shared closed values live in +`internal/service/jailer`; they are exported only across Go's repository-local `internal` boundary and are not public +CLI or `pkg/api` interfaces. The closed shape is: + +```go +const ( + TrustedReleaseInstallAction = "jailer-release-install" + TrustedReleaseRemoveAction = "jailer-release-remove" +) + +type ReleaseSlot struct { + Version string `json:"version"` + Architecture string `json:"architecture"` +} + +type TrustedReleaseInstallRequest struct { + Slot ReleaseSlot `json:"slot"` + AllowReplacement bool `json:"allow_replacement"` +} + +type TrustedReleaseRemoveRequest struct { + Slot ReleaseSlot `json:"slot"` +} + +type TrustedReleaseInstallOutcome string + +const ( + TrustedReleaseInstalled TrustedReleaseInstallOutcome = "installed" + TrustedReleaseReplaced TrustedReleaseInstallOutcome = "replaced" + TrustedReleaseUnchanged TrustedReleaseInstallOutcome = "unchanged" +) + +type TrustedReleaseExecutableResult struct { + SHA256 string `json:"sha256"` + SizeBytes uint64 `json:"size_bytes"` +} + +type TrustedReleaseInstallResult struct { + Outcome TrustedReleaseInstallOutcome `json:"outcome"` + Slot ReleaseSlot `json:"slot"` + ArchiveSHA256 string `json:"archive_sha256"` + Firecracker TrustedReleaseExecutableResult `json:"firecracker"` + Jailer TrustedReleaseExecutableResult `json:"jailer"` +} + +type TrustedReleaseRemoveResult struct { + Removed bool `json:"removed"` +} +``` + +The names, JSON fields, value domains, and action strings above are fixed. No request or result carries a path, URL, +caller hash, force flag, reference override, generic action, reader, or decoder field. The sole stream seam is: + +```go +func (server *TrustedReleaseServer) Install( + ctx context.Context, + request TrustedReleaseInstallRequest, + payload io.Reader, + payloadLength uint64, +) (TrustedReleaseInstallResult, error) +``` + +The privileged adapter supplies one private deadline-aware raw-socket reader. Zero payload requires EOF before +`Install(ctx, request, nil, 0)`. Nonzero payload streams once through `Install`; existing Jailer archive-stage receive +exclusively owns declared-length and EOF admission. Anonymous staging may occur during the read, but no extraction, +canonical publication, or promotion of unadmitted bytes occurs before exact EOF. There is no 128 MiB buffer, second +stage, `io.LimitReader`, reusable reader interface, or replay. + +One concrete `jailer.TrustedReleaseServer` owns the private `releaseAuthority` and exposes only named `Install` and +`Remove` methods. Privileged imports Jailer; the named Jailer client imports only `internal/lib/system` and +`internal/lib/privilegedwire`, never privileged. The server maps `AllowReplacement` to the existing closed intent and +converts only the fully re-admitted private manifest result into the exact result above. The +privileged package owns framing, deadlines, response delivery, and private exact install/remove function dependencies +for direct handler tests. There is no action registry, generic server interface, generic dispatcher, or decoder hook. + +Each registered handler owns exactly one response transaction. Zero-payload install and remove require EOF before the +named effect. Nonzero install invokes `Install` once with the private stream; Jailer owns stream/EOF admission and no +canonical effect occurs before admission succeeds. The handler then emits exactly one typed success result or preserved +`DomainError` followed by `SHUT_WR`. Dispatch distinguishes an unhandled action from a handler-owned response failure, +so outer `run` never writes a second frame. + +Each request reader carries the original operation context and one absolute deadline. It uses raw +`Recvmsg(MSG_DONTWAIT)` plus `POLLIN`, never `os.File.Read`, `SO_RCVTIMEO`, or mutable `O_NONBLOCK`. With cancellable +context, polling wakes and rechecks context at a compiled maximum 100 ms interval; uncancellable context may wait to the +absolute deadline. No individual cancellable poll waits longer than 100 ms, so cancellation never waits for the full +read deadline. The five-second header deadline starts immediately before decode. Nonzero install gets a separate +five-minute reader deadline after header decode for stream/EOF admission. Zero install/remove retain the header deadline +for EOF. The original context reaches Jailer; after EOF, the payload deadline does not constrain extraction/publication. +Install permits at most 128 MiB, remove permits zero, and the endpoint is never closed. + +The response deadline is established immediately before each success or error delivery, not when the process or +receiver invocation starts. Final error delivery computes that deadline before `context.WithoutCancel` and retains the +earlier context deadline. The original `DomainError`, class, code, entity, and bounded postcommit details are preserved; +a response contains exactly one result or one error. Before success encode, require a valid three-value outcome, valid +slot exactly matching the request, lowercase 64-character archive/executable hashes, policy-valid executable sizes, and +exact nested values. The typed client repeats this semantic validation after decode. Remove Boolean is already closed. +Normal SQLite projection and postcommit retry acceptance belong to slice 7, not these handlers. + +Focused L1 tests cover strict fields, types, nulls, duplicates, case conflicts, action mismatch, magic, schema, +truncation, and oversize; install zero/nonzero/oversize/truncated/trailing payloads; remove payload/trailing input; both +read deadlines, blocked-header/payload cancellation, and post-admission status mutation; zero/remove effects after EOF; +nonzero install once with publication only after Jailer admits EOF; all outcomes/remove Booleans; invalid fake effect +results; preserved `DomainError` details; response short writes, errors, half-close, and no double frame. The central +switch now registers only the two exact trusted-release actions; unknown actions remain rejected without reading. + +### Authorization roles + +The root-owned executable establishes code integrity. The `mvm` Unix group remains the authorization role for +passwordless privileged VM operations. Ordinary non-privileged commands do not require group membership. + +Membership in `mvm` authorizes management of mvmctl-owned VM infrastructure; it does not authorize arbitrary host root +commands. The sudoers drop-in grants only the reserved privileged entry point of the root-owned system executable. It +does not grant the public CLI or raw `ip`, `iptables`, `iptables-restore`, `iptables-save`, `nft`, `modprobe`, or `sysctl` +commands. + +`host init`, system-binary installation/replacement, group membership changes, sudoers changes, and `host reset` modify the +authorization boundary itself. They remain explicit administrator operations under the administrator's normal sudo +policy and are not exposed through the `mvm` group's passwordless dispatcher. + +The approved administrator `host init` flow must support authenticated sudo and direct root with an explicit non-root +operator. A direct-root session must not guess that operator from a home/cache path or enroll root as the operator. +Selection, enrollment, early administrator dispatch, and root-owned mutation journaling remain implementation work. + +The binary is never setuid and is not assigned broad Linux file capabilities. + +### Typed privileged operation catalog + +The initial dispatcher contains only the following named capabilities. Public callers retain domain-specific typed +methods; there is no generic privileged-operation interface. + +| Capability | Allowed effect | Required binding | +|---|---|---| +| Trusted release install/remove | Verify and atomically manage an exact Firecracker/Jailer release pair | Fixed origin, version, architecture, root release manifest, active references | +| VM launch/abort/cleanup/reconcile | Create and recover one Jailer process, chroot, cgroup, and private mount namespace | Invoking UID, VM ID, exact release, root instance record, per-VM lock | +| Snapshot expose/remove | Mount one validated snapshot directory in the registered VM namespace | Invoking UID, VM ID, snapshot ID, pinned source/target descriptors | +| Volume expose/remove | Mount one validated volume at one typed drive ID in the registered VM namespace | Invoking UID, VM ID, volume/drive ID, pinned source/target descriptors | +| Image provision | Execute the existing typed provision operations inside one pinned managed image/rootfs | Invoking UID, managed resource identity, pinned root descriptor, bounded operation list | +| Bridge/TAP lifecycle | Create, configure, or remove an mvmctl-owned bridge or TAP | Invoking UID, network/VM identity, validated interface/address model | +| Neighbor invalidation | Delete only the dynamic guest-IPv4 neighbor on the registered managed bridge before host-veth exposure | Invoking UID, VM/endpoint identity, registered network, exact bridge and host namespace ([ADR-0021](0021-exact-managed-neighbor-invalidation.md)) | +| Firewall inspect/reconcile/teardown | Render and atomically apply the complete typed mvmctl ruleset | Invoking UID, typed firewall models, fixed mvmctl tables/chains/comments | +| Retained VM-exec session | Retain and peer-bind one exact source/destination connection pair under capacity and absolute-stop authority | Authenticated caller UID/GID, exact active VM identities and launch generations, pidfd-bound Firecracker peers | + +Cgroup preparation and removal are part of VM launch/abort/cleanup rather than a public generic cgroup operation. Module +loading and sysctl mutation are administrator-owned host initialization steps, not operator dispatcher capabilities. + +Adding a new privileged capability requires a new typed method, abuse cases, ADR-0016 review, and sudo threat-model +review. “Extensible for future use” is not a reason to add a generic action. + +### Module placement and dependency direction + +The privileged boundary follows the repository's existing service-subprocess architecture and does not create a second +application architecture: + +| Package/file | Responsibility | Must not do | +|---|---|---| +| `cmd/mvm/main.go` | Detect the reserved marker and enter privileged dispatch before normal initialization | Parse domain requests, perform side effects, initialize the public CLI | +| `internal/service/privileged/` | Versioned envelope, bounded parsing, invoking identity, executable verification, and fixed action dispatch | Import `pkg/api`, `internal/cli`, or any `internal/core/*`; expose generic command execution or raw effect paths | +| `internal/service/jailer/` | Typed release, launch, namespace mount, cgroup, process, and cleanup handlers plus normal-user client calls | Orchestrate other core domains or trust user paths after validation | +| `internal/service/loopmount/` | Existing typed provisioning protocol and confined rootfs operations | Accept arbitrary host roots or unrelated unmount/PID targets | +| `internal/service/network/` | Typed root-side bridge, TAP, address, route, and neighbor mutations | Accept raw `ip` argv or own network business orchestration | +| `internal/service/firewall/` | Typed root-side firewall inspection and atomic managed-ruleset application | Accept raw nftables/iptables text from the caller | +| `internal/lib/authorityfs/` | Fixed descriptor-relative root-only ephemeral authority tree and closed named capability families | Import services, core domains, API, or CLI; expose raw paths/descriptors or generic lock names | +| `internal/lib/model/` | Shared typed request/value models used across boundaries | Import services, core domains, API, or CLI | +| `internal/lib/system/` | Subprocess transport and low-level process helpers | Become a generic privileged-operation API or render domain commands | +| `internal/core/{domain}/` | Existing domain logic and typed backend interfaces | Import another core domain or perform cross-domain orchestration | +| `pkg/api/` | Sole cross-core orchestration and public input validation | Implement root-side handlers or bypass typed service methods | + +The central dispatcher performs only fixed routing and security-boundary checks. Actual effects remain in +capability-specific service packages and files. Each public client method keeps a domain name and shape, such as +`jailer.Launch`, `jailer.Cleanup`, `network.EnsureTap`, and `firewall.Reconcile`; there is no public +`ApplyOperations([]Operation)` or `RunCommand([]string)` method. + +Where an existing leaf package needs privileged effects, it defines the smallest typed interface in the owning layer and +receives a concrete service client through construction. Leaf packages do not gain imports of `pkg/api`, `internal/cli`, +or core domains. The one existing loopmount service import exception is not generalized without a separate architecture +review. + +Normal API `Input.Validate`/`Resolve` remains the caller-facing validation layer. Because the privileged dispatcher +receives data from an untrusted process, it independently enforces authorization, ownership, bounds, path confinement, +and system-damage invariants. These are receiver trust-boundary checks, not duplicated business validation. + +### Implemented LoopMount client/protocol and rootfs-authority foundations + +Task 13 now has a named normal-user client and closed protocol types for exactly three intended capabilities: +`ProvisionVM`, `InspectImage`, and `OptimizeImage`. The corresponding fixed action strings are +`loopmount-provision-vm`, `loopmount-inspect-image`, and `loopmount-optimize-image`. The client uses a private minimal +exchange seam and exposes no generic privileged call, decoder, executable, argv, command, shell, effect path, mount +target, or arbitrary operation list. + +VM provisioning carries a canonical managed-cache locator, fixed-format VM ID, admitted rootfs size, closed filesystem, +and a typed plan. Image inspection and optimization carry the same namespace locator plus a fixed-format image ID, +admitted image size, closed filesystem, and, for optimization, a typed plan. VM resize is grow-only relative to the +admitted rootfs size, and a returned size must equal the requested resize or remain exactly unchanged when no resize was +requested. Image conversion is closed to ext4, and image shrink results cannot grow. Guest usernames are at most 32 +bytes; SSH input is capped at 64 keys, 16 KiB per key, and 48 KiB total; agent tokens are canonical UUIDv4 values; and +agent ports are 1 through 65,535. Rootfs sizes are 128 MiB through 16 TiB, image sizes are 1 byte through 16 TiB, and +each result counter is at most 1,048,576. These validations are protocol admission only and do not make the cache +locator or resource identifiers root authority. + +The client accepts a decoded success or remote `DomainError` only when the process actually started and the transport +observed authenticated socket EOF. A valid remote error preserves its code, class, entity, message, and bounded details +over later diagnostics. Every non-authoritative transport outcome uses `CodeProcessError`/`ClassInternal` with the fixed +action and stable `process_started` / `outcome_unknown` details: both false before start and both true after start. + +The private receiver-side rootfs authority accepts the authenticated non-root caller out of band and treats the cache +locator only as a namespace hint. It pins `/` and each canonical cache component with retained descriptors, permits +mount crossing only during cache discovery, enforces the closed local-filesystem policy on the caller-owned cache root, +and records device, inode, and typed unique/legacy/unavailable mount identity. All resource descendants use +beneath/no-symlink/no-magic-link/no-cross-mount resolution. The receiver derives only these fixed selections: + +| Operation | Rootfs leaf | Descriptor access | +|---|---|---| +| Provision VM | `vms//rootfs.img` | Read-write | +| Inspect image | `images/staging//rootfs.img` | Read-only | +| Optimize image | `images/staging//rootfs.img` | Read-write | + +Every fixed resource descendant directory is caller-owned with required owner access and no unsafe write or special +modes. The rootfs is a caller-owned one-link regular file with owner read/write, no group/world write, no execute or +special bits, and an exact size inside the protocol's hard VM or image bounds. Two synchronized metadata observations +bind device, inode, type, mode, UID, GID, link count, and size. The private lease retains the complete chain and the image +staging descriptor, exposes no path or generic descriptor accessor, and closes every descriptor in reverse order +through checked, cancellation-independent cleanup while preserving the primary `DomainError`. Deterministic +real-filesystem tests exchange every caller-owned cache/resource ancestor and leaf immediately before and after its +open; a request can only fail closed or retain the originally admitted rootfs, never the attacker object. + +The private `Server` transaction foundation depends on exactly three narrow seams. `rootfsEffects` +contains exactly the named `ProvisionVM` and `InspectImage` methods, the private `imageOptimizeEffects` contains only +`OptimizeImage`, and `rootfsResourceLocks` contains only +`Acquire(context.Context, Caller, rootfsSelection) (rootfsResourceLockLease, error)`. A private service adapter now +implements the lock seam through the separate `authorityfs` LoopMount UID and canonical VM/image lock capabilities. +Provisioning and inspection validate the rootfs effect seam, while optimization validates only its separate effect +seam; every action also validates the shared rootfs authority and resource-lock dependencies. The production +composition root constructs only the concrete `ProvisionVM` and `InspectImage` adapters, binds the exact embedded-agent +accessor, and registers their exact handlers in the early privileged switch. Optimization remains unconstructed. + +Each operation validates its closed request and authenticated caller before acquiring authority. It then acquires the +resource-lock lease, admits the fixed rootfs, invokes the named effect, validates the request-bound result, and unwinds +the action-specific effect cleanup lease, rootfs lease, and resource-lock lease in that exact order. Every cleanup uses +`context.WithoutCancel`. A partial lock acquisition that returns both a lease and an error still releases that lease +exactly once. Cleanup errors add bounded action-specific details without changing the primary `DomainError` code, +class, entity, operation, message, or existing details. + +Effect code cannot supply generic mutation booleans or an arbitrary details map. VM provisioning has closed +`unchanged`, `modified`, `modified durability uncertain`, and `outcome unknown` states; image inspection has only +`unchanged` and `outcome unknown`; image optimization has closed unchanged, modified, image-replaced, corresponding +durability-uncertain, and outcome-unknown states. The coordinator derives the exact Boolean error details +`rootfs_modified`, `image_replaced`, `durability_uncertain`, and `outcome_unknown`. A missing, invalid, uncertain, or +otherwise non-success state fails closed even when result fields look valid, and every post-effect result is validated +against the admitted request before publication. + +A VM resize replay may observe either the exact request's pre-grow rootfs size or its exact nonzero target size. The +rootfs authority admits only those two stable values, rejects any unrelated third size before effect invocation, and +records the actually observed size on the private rootfs lease delivered to `rootfsEffects.ProvisionVM`. The implemented +adapter uses it to distinguish a first grow from an already-converged replay without reopening a path or trusting +caller-reported post-effect state. + +The accepted effect seam keeps `rootfsEffects.ProvisionVM` and `rootfsEffects.InspectImage` separate from the private +`imageOptimizeEffects` seam containing only `OptimizeImage`. Production constructs and registers only the provision +and inspect handlers. The optimize seam has no production implementation, construction, receiver handler, or +dispatcher registration and cannot gain them until durable shrink and image-replacement replay authority exists. + +LoopMount operations serialize through exclusive, root-owned advisory locks at exactly: + +```text +/run/mvmctl/ root:root 0711 +└── authority/ root:root 0700 + └── / root:root 0700 + └── loopmount/ root:root 0700 + ├── vm-<32-lowercase-hex>.lock root:root 0600 + └── image-<64-lowercase-hex>.lock root:root 0600 +``` + +The lock key is only authenticated UID, resource family, and canonical resource ID. Cache locator, admitted or target +size, filesystem, action, and plan are excluded. `InspectImage` and future `OptimizeImage` therefore contend on the same +image lock. Lock files are created descriptor-relatively beneath pinned directories with no-follow, beneath, and +no-cross-mount resolution. Every component is rechecked for exact type, owner, mode, link count where applicable, and +stable device/inode identity. Lock files are never unlinked during the boot: replacement could split exclusion across +old and new inodes. A lease performs checked, idempotent unlock and close under `context.WithoutCancel`, preserving a +primary `DomainError` and bounding cleanup details. + +This lock layout depends on the shared privileged authority layout. The cycle-free `internal/lib/authorityfs` leaf +module now creates or admits exact `root:root` mode-`0711` `/run/mvmctl` and exact `root:root` mode-`0700` +`/run/mvmctl/authority`. Jailer index, release, UID, and VM-lock authority uses the module's closed Jailer family. The +implemented separate LoopMount family pins only the authenticated nonzero UID and fixed `loopmount` directory and +admits only canonical 32-lowercase-hex VM keys or 64-lowercase-hex image keys. It creates or verifies empty one-link +`root:root` mode-`0600` files, acquires exclusive advisory locks without unlinking them during the boot, and permits +unrelated canonical resources on one capability to acquire concurrently. The capability and acquired locks retain +pointer-held lifecycle state and perform checked idempotent cancellation-independent release exactly once, caching the +first cleanup result instead of retrying a potentially reused descriptor. + +The private LoopMount service adapter derives the lock family and key only from the closed `rootfsSelection` and opens +the authority for the authenticated caller UID. Provisioning uses the VM key; inspection and future optimization use +the same image-key path. It retains a partially opened authority or acquired lock for coordinator cleanup, maps +cancellation/deadline acquisition failures as retryable `DomainError` values, and releases the lock before the parent +authority. Review corrected parent-capability serialization so one contended key cannot block acquisition of an +unrelated resource through the same capability. Focused L1 evidence covers the fixed hierarchy and key grammar, +unsafe-object rejection without repair, concurrent creation, creation and sync faults, same-resource contention, +cancellation, independent-resource concurrency, partial acquisition, error mapping, release ordering, copied leases, +and inactive capabilities. LoopMount does not import Jailer or duplicate the pathname initializer. + +Each effect runs in the short-lived privileged receiver's new private mount namespace. It makes propagation recursively +private before attaching anything. The admitted `rootfsFD` is passed directly to loop-control ioctls with partition scan +and autoclear enabled; inspection also configures the loop device read-only. Namespace setup, loop configuration, +partition-device admission, mount/unmount, descriptor-relative guest-tree access, truncate, fsync, syncfs, and detach +use direct Linux syscalls. No caller path is reopened or passed to a child, and no `/proc/self/fd` or +`/proc//fd` passthrough is constructed. Kernel-derived loop-device names and the root-owned private mountpoint are +the only paths a fixed host filesystem tool may receive. + +The external host-tool allowlist is closed to read-only filesystem probing plus `e2fsck`, `resize2fs`, and `btrfs` for +their named filesystem grow operations. Each invocation uses a fixed executable and argv under the dispatcher's cleared +environment. No shell, `chroot`, guest executable, arbitrary operation, `losetup`, `mount`, `umount`, PID scan or kill, +or `/tmp` workspace is permitted. A tool that may have started a mutation but lacks authoritative completion makes the +effect outcome unknown. + +The private mount-source selection policy consumes only pathless receiver-derived block identities and +filesystem probes. It accepts at most 16 partition probes. A raw source is accepted only when there are no partitions +and its filesystem exactly matches the closed request. Otherwise exactly one partition must match. Invalid identities +or filesystem values, reused or duplicate block identities, duplicate or out-of-range partition numbers, +raw-plus-partition ambiguity, and missing or multiple matches fail closed. + +The implemented receiver derives partition candidates only from fixed `/sys/class/block`, bounds directory admission +to 65,536 entries and 16 partitions, retains exact device capabilities, and proves stable partition membership before +selection. Filesystem probing invokes only fixed `/usr/bin/blkid`, or fixed `/usr/sbin/blkid` when the first path is +absent, with a fixed argv/environment, a five-second cap with any earlier caller deadline winning, 32-byte stdout and +512-byte stderr ceilings, exact accepted exit states, and pre/post block-identity revalidation. A present but invalid +primary executable fails closed; neither `PATH` lookup nor a caller-selected executable is admitted. The selected +source is pointer-bound to that exact probe and exposes no path or descriptor accessor. + +The private loop-device kernel contract admits only the fixed `/dev/loop-control` path with exact character-device +major `10` and minor `237`. It derives `/dev/loop` only when `N` is a 20-bit index from `0` through `2^20-1`, then +requires the opened candidate to be a block device with major `7` and minor exactly `N`. Its `unix.LoopConfig` contains +the exact admitted rootfs descriptor and only `LO_FLAGS_AUTOCLEAR|LO_FLAGS_PARTSCAN`, with `LO_FLAGS_READ_ONLY` added +only for read-only access. Every offset, size limit, name, encryption value, block-size override, and legacy setup field +remains zero, and direct I/O is not enabled. Invalid descriptors, access modes, indices, or device identities fail +closed through the private LoopMount `DomainError` path with no usable result. + +The implemented package-private attach capability opens only fixed `/dev/loop-control` and derived `/dev/loop` +leaves with `O_RDWR|O_CLOEXEC|O_NOFOLLOW`, then uses `fstat` to recheck their exact identities. It obtains each candidate +with `LOOP_CTL_GET_FREE` and retries only configure-time `EBUSY`, with a hard limit of 16 attempts. It passes the pure +contract's exact admitted rootfs descriptor and access mode to one atomic `LOOP_CONFIGURE`. Configure failure closes +the candidate without issuing `LOOP_CLR_FD`, because another process may have won the free-device race. + +A successful attach returns only a private lease, never a path or descriptor accessor. Its checked, idempotent release +uses `LOOP_CLR_FD` and closes the loop descriptor under `context.WithoutCancel`; `LO_FLAGS_AUTOCLEAR` remains receiver- +crash safety. If configuration succeeds but closing the loop-control descriptor fails, attachment returns both the +configured lease and the close error so the caller retains cleanup authority. + +The implemented read-only image-inspection adapter locks its OS thread, unshares a private mount namespace, +makes propagation private, and mounts only the exact selected source at fixed `/run/mvmctl/rootfs` with +`ro,nosuid,nodev,noexec`. The inspection-only backing descriptor also uses `O_NOATIME` so loop reads do not invalidate +the before/after statx comparison merely by recording access. It revalidates source, mountpoint, filesystem, mount +table, and backing-image identity; reads +only bounded no-symlink/no-cross-mount `etc/os-release` or `usr/lib/os-release`; and unwinds root, mount, source, +partitions, loop, mountpoint, and image authority in reverse. `unchanged` requires the final admitted/statx snapshot to +remain stable and cleanup to succeed; uncertainty is `outcome_unknown`, with the primary `DomainError` preserved and +cleanup evidence bounded. The short-lived privileged receiver exits rather than returning its unshared OS thread to the +Go runtime. Live-kernel qualification and public/backend activation remain future work. The implemented grow effect +supports a raw +filesystem or a partition whose already-admitted capacity reaches the target; rewriting MBR or GPT is outside this +capability. + +Provisioning writes only receiver-generated, fixed-purpose content through the mounted-root descriptor. The accepted +first subset is hostname and hosts content, DNS content, SSH keys and sudo policy for root or an already-existing parsed +guest account, fixed SSH service content, the embedded mvm agent and its fixed service/token content, and a bounded +fstab rewrite. Reads and replacements are bounded, no-follow, descriptor-relative, and durably synced. Service +enablement or key generation that requires guest code is deferred to guest boot; root never executes it on the host. +`InspectImage` mounts read-only with `nosuid`, `nodev`, and `noexec`, and reads only bounded fixed OS-release leaves. + +A grow admitted at the old size truncates the retained descriptor to the exact target, syncs it, then grows the admitted +ext-family filesystem through fixed `e2fsck`/`resize2fs` or Btrfs through fixed `btrfs filesystem resize`, with +ext4/Btrfs convergence evidence. Grow-tool execution has a five-minute cap, with any earlier caller deadline winning. A +replay admitted at the target never truncates again: it probes filesystem +capacity and either reports convergence or completes the interrupted filesystem grow. Any third backing-file size is +rejected by the existing authority before effects. Fixed-content writes are idempotent, so the same request converges +after a crash without a caller-path reopen. + +Before mutation, failure state is `unchanged`. A known mutation followed by leaf, directory, filesystem, unmount, +detach, and backing-file durability is `modified`. Failure to prove final durability after a known mutation is +`modified durability uncertain`. A possibly started mutating tool or otherwise non-authoritative mutation boundary is +`outcome unknown`. Effect cleanup precedes rootfs-lease and lock release and is cancellation-independent. Private mount +namespace teardown and loop autoclear contain a receiver crash; cleanup never guesses a host mount or kills a process. + +The adapter persists the canonical agent token only at root-owned mode-`0600` `/etc/mvmctl/agent.token` beneath a +root-owned mode-`0700` `/etc/mvmctl`, and enables exactly one admitted init system: systemd through `sysinit.target`, or +OpenRC through its default runlevel. The latter historical runlevel choice is superseded by +[ADR-0024](0024-early-openrc-agent-startup.md), which accepts early boot activation without changing +the authentication or confined provisioning boundary; implementation and qualification remain separately recorded. +The initial subset deliberately does not preserve missing-user creation, cloud-init directory +copying without its own pinned capability, package-manager or deblob commands, recursive guest cleanup, guest host-key +generation, arbitrary file/copy/command operations, custom shells, partition-table growth, shrink, conversion, or image +replacement. These are public-cutover blockers, not permission to silently drop behavior. An explicit scope decision +must either accept each user-visible reduction or defer cutover until a safe typed replacement exists. + +The user selected feature preservation for the resumed implementation pass: missing-account creation, +cloud-init injection, and automatic deblob require closed typed replacements, not removal or an +already-existing-account requirement. Guest account databases and bounded cleanup/configuration +operations use the admitted root descriptor. Cloud-init seed input requires its own pinned fixed-leaf +capability. Operations that inherently execute guest code, such as distribution key initialization, +may be installed as fixed receiver-generated guest-boot actions; they never execute on the host. +Queuing a guest-boot action is not evidence it has run. Arbitrary guest commands, caller scripts, +unbounded recursive traversal, and host-path copy interfaces remain excluded. This decision does +not claim the corresponding implementations have been verified. + +This section records the implemented authority, selection-policy, inspection, provisioning, and private receiver path, +not public LoopMount backend behavior. The shared cycle-free authority module has separate Jailer and LoopMount +capability families, the private LoopMount service adapter implements the coordinator's canonical resource-lock seam, +the deterministic mount-source policy remains pathless, and the pure loop contract, attach/release capability, bounded +partition/probe admission, and read-only inspection and `ProvisionVM` adapters are implemented. Production constructs +the provision/inspect server and registers only `loopmount-provision-vm` and `loopmount-inspect-image`; the public +backend path still uses `mvm run provision`, raw wire operations and sudo, arbitrary unmount, and PID-killing cleanup. +No live-kernel qualification or public backend cutover follows from handler registration. The legacy `Provisioner`, +`Op`, path protocol, and cleanup functions must not be wrapped by the new adapter. + +The preceding paragraph records the state at that implementation checkpoint, not the current 0.3.0 source. The later +public backend cutover removed `mvm run provision` and raw fallback; current behavior is described in `CONTEXT.md`. + +### Release trust + +The caller supplies a validated release version and architecture and may stream an exact-length bounded archive body for +cache efficiency. Privileged release code constructs the permitted Firecracker release and checksum URLs and obtains the +checksum independently from the fixed official origin with a dedicated bounded HTTPS-only client and proxies disabled. +It hashes the complete archive body against that checksum before parsing it. If no body is supplied, root uses the fixed +archive-fetch contract below. Root never opens a caller path or accepts a caller URL or checksum. The normal-user client +may read `MVM_ASSET_MIRROR`, but those streamed bytes are transport, never release authority. + +Extraction reads the anonymous root-owned archive stage with bounded compressed/decompressed input and validates the +complete reviewed archive contract below before publishing any path. Only the exact Firecracker/Jailer pair is copied +to private root-owned staging descriptors. Their ELF class and machine must match the selected architecture without +executing downloaded code. The pair and strict root-owned release manifest are fsynced and atomically renamed into the +trusted store only after the complete archive and both executables pass admission. A referenced release cannot be +force-replaced. + +ELF admission reads exactly the first 64 bytes of each extracted binary and performs no executable handoff. It requires +the ELF magic, ELF64 class, little-endian encoding, current ident/object versions, System V ABI version 0 with zero ident +padding, `ET_DYN`, a non-zero entry point, a 64-byte ELF header, program headers beginning at byte 64 with 56-byte +entries, and 1 through 64 program headers. Admission also receives the exact full-file size, enforces the 120-byte +through 64 MiB executable policy, and requires the complete declared program-header table to fit within that size. The +machine is exactly `EM_X86_64` (62) for `x86_64` or `EM_AARCH64` (183) for `aarch64`. A future upstream format change +fails closed until this reviewed policy and its tests are updated. + +If upstream provides a stable verifiable signature or provenance mechanism, mvmctl should adopt it. Until then, the +fixed HTTPS origin and independently fetched checksum are an explicit supply-chain limitation, not a signature claim. + +The receiver derives the source identity from the canonical release slot; none of these values are request fields. The +naming follows Firecracker's +[official binary-installation contract](https://github.com/firecracker-microvm/firecracker/blob/main/docs/getting-started.md#getting-a-firecracker-binary). +For version `` without a leading `v` and architecture `` in `x86_64|aarch64`, the fixed +derivation is: + +| Value | Receiver-derived form | +|---|---| +| Official origin | `https://github.com/firecracker-microvm/firecracker/releases/download` | +| Release tag | `v` | +| Archive | `firecracker-v-.tgz` | +| Checksum sidecar | `firecracker-v-.tgz.sha256.txt` | +| Archive root | `release-v-` | +| Firecracker member | `release-v-/firecracker-v-` | +| Jailer member | `release-v-/jailer-v-` | +| Trusted-store leaves | `firecracker`, `jailer`, and `manifest.json` | + +This table freezes source construction and the two extracted member identities. It does not claim that those are the +only members in the upstream archive: the complete reviewed allowlist and format contract follow, and Task 6 must +enforce them before extraction is accepted. A packaging change must update the reviewed contract and tests; it must not +fall back to basename matching, caller-selected member names, or permissive extraction. + +The v0.3 archive parser accepts only the separately audited x86_64 releases `1.10.1`, `1.14.2`, `1.14.3`, `1.14.4`, +`1.15.0`, `1.15.1`, `1.16.0`, `1.16.1`, `1.16.2`, and `1.17.0`. Source derivation and ELF admission also understand +`aarch64`, but that does not authorize an aarch64 archive: the available asset mirror contains no aarch64 release +archive, so extraction for that architecture fails closed until an architecture-specific member and format audit updates this decision and its +tests. Versions outside the exact audited x86_64 set likewise fail closed rather than assuming that a nearby release +uses the same packaging. + +Archive admission is distinct from VM launch support. The v0.3.0 lifecycle admits only a matching +Firecracker/Jailer `1.17.0` pair for create, start, reboot, and restore. The authenticated launch request and the +unprivileged pair resolver reject another selected version before Jailer execution; audited older archives remain +available for explicit binary-management operations. Jailer `1.16.0` rejected the launcher's +`/proc/self/ns/net` handle as a symbolic-link loop; the upstream +[1.16.1 changelog](https://github.com/firecracker-microvm/firecracker/blob/main/CHANGELOG.md#1161) records the +reversal of that no-follow behavior. + +Historical 1.16.2 decision: the user approved replacing the earlier 1.16.1 launch pin with 1.16.2 after +installed-candidate pause/resume QA reproduced the upstream vsock RX-gate defect while the guest and agent remained +alive. The +[1.16.2 release notes](https://github.com/firecracker-microvm/firecracker/releases/tag/v1.16.2) record its fix and a +snapshot-format change incompatible in both directions with 1.16.0/1.16.1. Fresh snapshots are required; older captured +release identities are not rewritten and no snapshot conversion is provided. The official x86_64 archive digest is +`32e3cdcd4081f91fe2b024a266f57dcb3b4e5fec5033e0cb22467ad7f7820bda`; its inspected 24 regular-file members match +the names, modes and ELF admission shape below, but its raw tar encoding differs as specified below. Runtime +qualification remains a separate release gate. + +Approved update policy (2026-10-10): select the newest stable official Firecracker release and its exactly matching +Jailer release. The official [latest release](https://github.com/firecracker-microvm/firecracker/releases/latest) +resolved to [v1.17.0](https://github.com/firecracker-microvm/firecracker/releases/tag/v1.17.0) when checked on +2026-10-10, and the user approved that exact pair. Each update requires inspection of the authentic upstream archive, +a release-specific closed archive profile and member/ELF admission, review of launch/device/snapshot contracts, and +installed-candidate runtime qualification before release. A version bump does not inherit older qualification or +authorize archive repacking, a mixed Firecracker/Jailer pair, or unchecked runtime auto-update. Unsupported release +and architecture profiles fail closed. Snapshot release identities remain exact; this update supplies no snapshot +conversion or permission to rewrite older captured identities. + +The authentic 1.17.0 x86_64 archive inspected on 2026-10-10 is 7,464,385 compressed bytes with SHA-256 +`06094a1108ae9e82aa4c23a775aa92758f53f1175d422270d9d6162cb9ade558`, matching the official sidecar and release asset +digest. Its exact 24-member set and modes match the contract below. Its 24,606,720-byte tar stream and maximum member +size of 3,676,680 bytes remain within the existing 32 MiB tar and 8 MiB member limits. Firecracker is 3,676,680 bytes +with SHA-256 `99ad0f5cd0514a88aad0e9ae8cfdb3cc3b4ab9d190e1194602406c786b5de7a5`; Jailer is 2,243,000 bytes with +SHA-256 `65ef226e96f0ceda55ba643f445801ef2cc0ea667ef67cad8ac4f406c9c8434f`. Both selected executable headers match the +existing ELF64/little-endian/x86_64 `ET_DYN` shape; complete production ELF admission remains an import gate. +The inspection establishes the reviewed archive profile; production +import and runtime qualification remain separate gates. Per-member alignment padding was not independently checked +by this inspection and remains subject to the existing production zero-padding checks. + +For an accepted version, let `` be `release-v-x86_64` and `` be +`v-x86_64`. The logical archive contains exactly 24 regular-file members, in any order, with this exact set: + +- `/{SHA256SUMS,LICENSE,THIRD-PARTY,NOTICE}`; +- `/firecracker_spec-v.yaml` and `/seccomp-filter-.json`; +- `/{firecracker,jailer,cpu-template-helper,rebase-snap,seccompiler-bin,snapshot-editor}-`, plus the + corresponding exact `.debug` member for each tool; and +- six CPU-template leaves `/