Skip to content

FM-603: Proxmox template, clone, and snapshot operations behind the destructive review gate - #106

Merged
Andreas-Froyland merged 4 commits into
mainfrom
fm-603-proxmox-destructive-ops
Sep 21, 2026
Merged

Andreas-Froyland merged 4 commits into
mainfrom
fm-603-proxmox-destructive-ops

Conversation

@Andreas-Froyland

@Andreas-Froyland Andreas-Froyland commented Sep 21, 2026

Copy link
Copy Markdown
Member

Closes #105 (FM-603).

Implementation approach (from the issue)

FM-602's lifecycle executor and UPID polling are the foundation; this PR extends both with the destructive-adjacent surface.

What landed

  • Provider (fleet-provider-proxmox): guest_snapshot (create — PVE's vmstate flag carries the include-RAM intent), guest_snapshot_rollback, guest_snapshot_delete (synchronous), guest_clone, guest_convert_template, stop_task, and guest_snapshots (listing, current excluded). The transport now carries Delete and bodies (execute_with_body); the response mapping is shared (status_to_result).
  • Executor kinds (proxmox.guest.snapshot|snapshot-revert|snapshot-delete|clone|template, proxmox.task-cancel): six new CREATABLE_KINDS entries (35 → 41) behind the new proxmox.destructive permission (catalog 39 → 40), catalog-level like the lifecycle kinds.
  • The review gate is structural: POST .../guests/{vmid}/{action}/review renders exactly what will run and returns a token computed as SHA-256 of the canonical request (account, vmid, action, node, params). The create call (.../run) must present that token — recomputed from its own payload, so any parameter change invalidates it. The generic /operations surface refuses these kinds outright (NewOperation.reviewed is set only by the dedicated reviewed endpoint); there is no route-around.
  • Idempotency classification (live-verified): snapshot create is a no-op success when the snapshot exists with the reviewed description and a conflict failure when the description differs; clone refuses when the target VMID already exists (the cluster's truth, never a duplicate); template conversion is a no-op when already a template.
  • Remote task cancellation (deferred from FM-602): proxmox.task-cancel runs stop_task as an authorized, audited operation and reads the outcome back honestly (confirmed/unknown).
  • Compensation by record: a failed clone/snapshot names what ran (UPID, node) in the operation's failure detail; nothing is deleted on failure. ProxmoxDestructiveExecutor re-applies the trust gate before any network call and polls with the same Fleet-owned fixed-interval loop.
  • CLI: fleetctl proxmox snapshot|snapshot-revert|snapshot-delete|clone|template|task-cancel --account --node --vmid — a two-step flow (review → run with the returned token); the action's parameters arrive as JSON on stdin, never argv; --wait works.

Tests

  • Controller e2e through the real worker: review → run → task OK; missing/stale token → 400; a tampered payload with a valid token is refused (what runs is what was reviewed).
  • CLI parse contract tests for all six verbs. cargo xtask verify passes.
  • Live-verified against the integration PVE 9.2 host: a real snapshot of fleet-test-01 created through the review flow (taskState: ok), the idempotent re-run answering the no-op, and the snapshot deleted again through the reviewed flow — the PVE host's snapshot list confirmed both transitions. (One live finding fixed in-PR: PVE's qemu snapshot schema names the RAM flag vmstate, not include_ram.)

Non-goals (respected)

A backup UI; automatic cleanup policy (M7 Lab leases); guest OS provisioning.


Summary by cubic

Adds Proxmox snapshot, clone, template conversion, and task-cancellation operations behind a mandatory review step, so destructive actions run exactly what was reviewed.

Review gate

  • POST .../guests/{vmid}/{action}/review returns a token: a SHA-256 over the canonical request (account, vmid, action, node, params), computed in fleet-application and persisted on the operation.
  • The run call must present that token recomputed from its own payload; any parameter change invalidates it, and the generic /operations surface refuses these kinds outright.
  • The six new operation kinds sit behind a new proxmox.destructive permission.

Operations and CLI

  • fleetctl proxmox snapshot|snapshot-revert|snapshot-delete|clone|template|task-cancel runs the two-step review flow; params arrive as JSON on stdin (read to EOF) while secret inputs stay single-line, and --wait is supported.
  • The provider gains execute_with_body and Delete; failed operations record what ran (UPID, node) without deleting anything.
  • Idempotency: snapshot no-ops on an existing matching snapshot, clone refuses an existing target VMID, template conversion no-ops when already a template.

Written for commit 07bae1e. Summary will update on new commits.

Review in cubic

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

4 issues found across 38 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="crates/fleet-application/src/operation.rs">

<violation number="1" location="crates/fleet-application/src/operation.rs:551">
P1: Any caller of the public `Operations::create` API can set `reviewed: true` and bypass the destructive review gate because this service does not verify a token or capability. Make the reviewed path an unforgeable/privileged API or pass verified review material into this layer and validate it here.</violation>
</file>

<file name="crates/providers/fleet-provider-proxmox/src/lib.rs">

<violation number="1" location="crates/providers/fleet-provider-proxmox/src/lib.rs:1557">
P2: Custom agent: **Flag AI Slop and Fabricated Changes**

`list_qemu_resources` performs the version request through `version_and_resources`, then discards the result with `let _ = &version;`. Clone and template idempotency checks therefore pay for an unrelated network call and can fail on its response; extract a resource-only helper and keep the version-bearing helper only for `discover`.</violation>
</file>

<file name="crates/fleetctl/src/lib.rs">

<violation number="1" location="crates/fleetctl/src/lib.rs:1244">
P3: The new verbs are absent from `usage()`, so unknown-flag and top-level parse errors never advertise the destructive workflow. Add the six `proxmox` forms to the usage text.</violation>
</file>

<file name="crates/fleet-controller/src/proxmox_exec.rs">

<violation number="1" location="crates/fleet-controller/src/proxmox_exec.rs:852">
P2: `timeout_seconds` does not bound the mutating request: `call(...).await` completes before this timer starts, so a slow operation can exceed the reviewed deadline without being counted. Start the timer before invoking `call` and enforce the remaining deadline around that await.</violation>
</file>

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

// dedicated reviewed endpoint, which binds the operation to a
// confirmed review token. A generic-surface create is a
// route-around attempt, refused as malformed.
if !new.reviewed

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1: Any caller of the public Operations::create API can set reviewed: true and bypass the destructive review gate because this service does not verify a token or capability. Make the reviewed path an unforgeable/privileged API or pass verified review material into this layer and validate it here.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At crates/fleet-application/src/operation.rs, line 551:

<comment>Any caller of the public `Operations::create` API can set `reviewed: true` and bypass the destructive review gate because this service does not verify a token or capability. Make the reviewed path an unforgeable/privileged API or pass verified review material into this layer and validate it here.</comment>

<file context>
@@ -525,6 +543,25 @@ impl Operations {
+            // dedicated reviewed endpoint, which binds the operation to a
+            // confirmed review token. A generic-surface create is a
+            // route-around attempt, refused as malformed.
+            if !new.reviewed
+                && (new.kind.starts_with("proxmox.guest.snapshot")
+                    || matches!(
</file context>

Comment thread crates/fleet-api/src/proxmox.rs Outdated
Comment thread crates/fleetctl/src/lib.rs Outdated
Comment thread crates/fleet-controller/src/proxmox_exec.rs
Comment thread crates/providers/fleet-provider-proxmox/src/lib.rs Outdated
let mut resources = Vec::new();
for entry in entries {
if let Ok(Some(resource)) = normalize_resource(&entry) {
let _ = &version;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: Custom agent: Flag AI Slop and Fabricated Changes

list_qemu_resources performs the version request through version_and_resources, then discards the result with let _ = &version;. Clone and template idempotency checks therefore pay for an unrelated network call and can fail on its response; extract a resource-only helper and keep the version-bearing helper only for discover.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At crates/providers/fleet-provider-proxmox/src/lib.rs, line 1557:

<comment>`list_qemu_resources` performs the version request through `version_and_resources`, then discards the result with `let _ = &version;`. Clone and template idempotency checks therefore pay for an unrelated network call and can fail on its response; extract a resource-only helper and keep the version-bearing helper only for `discover`.</comment>

<file context>
@@ -1177,6 +1540,30 @@ impl ProxmoxClient {
+        let mut resources = Vec::new();
+        for entry in entries {
+            if let Ok(Some(resource)) = normalize_resource(&entry) {
+                let _ = &version;
+                resources.push(resource);
+            }
</file context>

Comment thread crates/fleet-controller/tests/proxmox.rs Outdated
}
_ => Err(CliError { message: usage() }),
},
"snapshot" | "snapshot-revert" | "snapshot-delete" | "clone" | "template"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: The new verbs are absent from usage(), so unknown-flag and top-level parse errors never advertise the destructive workflow. Add the six proxmox forms to the usage text.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At crates/fleetctl/src/lib.rs, line 1244:

<comment>The new verbs are absent from `usage()`, so unknown-flag and top-level parse errors never advertise the destructive workflow. Add the six `proxmox` forms to the usage text.</comment>

<file context>
@@ -1223,6 +1241,81 @@ fn parse_proxmox_command(verb: &str, rest: &[&str]) -> Result<Command, CliError>
             }
             _ => Err(CliError { message: usage() }),
         },
+        "snapshot" | "snapshot-revert" | "snapshot-delete" | "clone" | "template"
+        | "task-cancel" => {
+            let mut account_id = None;
</file context>

Comment thread crates/fleetctl/tests/cli.rs
Comment thread crates/fleet-api/src/proxmox.rs Outdated
@Andreas-Froyland

Copy link
Copy Markdown
Member Author

All 19 findings addressed in 56ac300. The most important change: the review gate moved into the application layer and is no longer forgeable.

P1s

  • Forgeable reviewed: true: the bool is gone. NewOperation now carries review_token: Option<String>, and Operations::create recomputes the token itself (SHA-256 over kind + the exact payload bytes, constant-time compared) for every DESTRUCTIVE_KINDS entry — a caller that never reviewed these exact bytes cannot present a matching token, and setting the field arbitrarily fails the comparison. The generic surface's None still refuses outright.
  • Unkeyed token: the token is now the application's review_token_for(kind, payload_json) — computed over the exact operation bytes the run will carry (the timeout is fixed at review time), so what runs is what was reviewed by construction. The review endpoint returns the same material it will execute.
  • CLI run-body null params: follow_review now reuses the reviewed data.params the controller echoed back, guaranteeing byte equality.
  • stop_task wrong endpoint: it now deletes /nodes/{node}/tasks/{upid} (the task itself), not the status subresource.
  • upid_from_data silent sync-success: a non-null value that does not parse as a UPID is now InvalidPayload; None is reserved for the documented synchronous/null response.
  • task-cancel scope: the reviewed UPID's node and target must match the reviewed guest — a task on another node or VMID is a conflict failure.

P2s

  • Stdin truncation: read_stdin_to_eof reads to EOF (multi-line JSON works), matching the apply workflow.
  • Clone preflight LXC gap: the provider's listing now returns qemu + template + lxc resources; the target-VMID conflict check sees every guest.
  • Silently dropped body in the default execute_with_body: the trait now requires implementations; the two test fakes implement it explicitly (their canned fixtures answer by path, documented).
  • Unurlencoded node/snapshot paths: every interpolated component goes through urlencode.
  • Deadline not covering the mutating call: the timer starts before the call and the call runs inside tokio::time::timeout(deadline, …).
  • --wait ignored: ProxmoxDestructive joined the polling matcher.
  • Missing params → durable failure: validate_destructive_params runs at review time; a malformed review is a 400.
  • Post-UPID failures without reconciliation detail: every failure after the UPID exists (status error, cancellation, deadline) now carries upid.raw and the node.

P3s

  • Dead duplicate observe guard removed; the destructive flags' --wait/--timeout/bad-VMID parsing got their own contract tests; the usage line was already present and verified.

Live re-verified against the integration host after the fixes: snapshot create through the review flow → taskState: ok; delete through the reviewed flow → the PVE host's snapshot list confirmed the removal.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2 issues found across 33 files (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="crates/fleet-api/src/proxmox.rs">

<violation number="1" location="crates/fleet-api/src/proxmox.rs:1277">
P2: Custom agent: **Flag AI Slop and Fabricated Changes**

The review and run handlers duplicate the security-critical `kind` and payload construction, so a future edit to either site can silently make the reviewed token cover different bytes than the operation executes. Extract this construction into one shared helper and use it in both handlers.</violation>
</file>

<file name="crates/fleet-controller/src/proxmox_exec.rs">

<violation number="1" location="crates/fleet-controller/src/proxmox_exec.rs:601">
P2: When `template` targets an LXC guest, this lookup now accepts the LXC resource and sends a QEMU template-conversion request instead of rejecting the non-QEMU guest. Filter the selected resource to `qemu` or `qemu-template` before calling `guest_convert_template`.</violation>
</file>

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

}
// The token binds the exact payload the run will carry, so the
// reviewed bytes and the executed bytes are the same by construction.
let kind = if action == "task-cancel" {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: Custom agent: Flag AI Slop and Fabricated Changes

The review and run handlers duplicate the security-critical kind and payload construction, so a future edit to either site can silently make the reviewed token cover different bytes than the operation executes. Extract this construction into one shared helper and use it in both handlers.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At crates/fleet-api/src/proxmox.rs, line 1277:

<comment>The review and run handlers duplicate the security-critical `kind` and payload construction, so a future edit to either site can silently make the reviewed token cover different bytes than the operation executes. Extract this construction into one shared helper and use it in both handlers.</comment>

<file context>
@@ -1229,8 +1272,23 @@ pub async fn review_proxmox_operation(
     }
+    // The token binds the exact payload the run will carry, so the
+    // reviewed bytes and the executed bytes are the same by construction.
+    let kind = if action == "task-cancel" {
+        "proxmox.task-cancel".to_owned()
+    } else {
</file context>

Comment thread crates/fleetctl/src/lib.rs Outdated
// resources are the truth.
let resources = self
.client
.list_guest_resources(request.clone())

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: When template targets an LXC guest, this lookup now accepts the LXC resource and sends a QEMU template-conversion request instead of rejecting the non-QEMU guest. Filter the selected resource to qemu or qemu-template before calling guest_convert_template.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At crates/fleet-controller/src/proxmox_exec.rs, line 601:

<comment>When `template` targets an LXC guest, this lookup now accepts the LXC resource and sends a QEMU template-conversion request instead of rejecting the non-QEMU guest. Filter the selected resource to `qemu` or `qemu-template` before calling `guest_convert_template`.</comment>

<file context>
@@ -595,7 +598,7 @@ impl OperationExecutor for ProxmoxDestructiveExecutor {
                 let resources = self
                     .client
-                    .list_qemu_resources(request.clone())
+                    .list_guest_resources(request.clone())
                     .await
                     .map_err(|error| format!("the resource listing failed: {error}"))?;
</file context>

@Andreas-Froyland

Copy link
Copy Markdown
Member Author

Round-2 re-review: the only new finding was the stdin regression — read_stdin_to_eof had replaced the single-line reader for all stdin inputs, so tailnet configure and proxmox create (single-line secrets) waited for EOF. Fixed in 07bae1e: read_stdin_line is back as the newline-terminating reader for secrets and single-line inputs; the EOF reader is used only for document inputs (the destructive action parameters). All other round-2 comments are the round-1 findings repeated, addressed in 56ac300.

@Andreas-Froyland
Andreas-Froyland merged commit 2312102 into main Sep 21, 2026
12 checks passed
@Andreas-Froyland
Andreas-Froyland deleted the fm-603-proxmox-destructive-ops branch September 21, 2026 12:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

FM-603 — Add Proxmox template, clone, and snapshot operations

1 participant