FM-S08: Proxmox client spike — fallback (reqwest + pinned rustls verifier) chosen

docs/planning/initial-issues.md

@@ -310,7 +310,12 @@ FM-105 moves to M8 with FM-S02. It evaluates Cedar only when authenticated human
 
 ## M4 — Profiles, desired state, and GitOps
 
-**Status: Code-complete (as of 2026-09-19).** Created as epics #5–#8 with issues #89–#92. The wave order is: FM-400 (#89 desired resource schema and composition) → FM-401 (#90 observed state, difference model, and planner) → FM-402 (#91 apply engine) → FM-403 (#92 Git source and GitHub bootstrap). M4 is not on the critical path to the first Lab release; it can proceed independently now that M3 is code-complete. Resolved: FM-403 (#92, PR #96 — the Git source provider with isolated worktrees, validated candidate digests, symlink refusal, and bounded/redacted output; the desired-source use cases gating activation through recorded-candidate verification and a backend-serialized critical section; source.fetch/source.activate executor kinds with a STRICT source storage migration; and the GitHub bootstrap provider with least-permission, expiring tokens; three cubic review rounds addressed). Resolved: FM-402 (#91, PR #95 — the apply engine as pure semantics (approval gate bound to plan identity/order/kind derived from the catalog's risk classification, compensations matching step semantics, post-apply verification) plus a workflow executor with deadline enforcement, cancellation honored before verification, failure detail propagated from inner operations, and payload validation as the last line of defense against generic-surface bypasses; the authz catalog grown to 34 entries; three cubic review rounds addressed). Resolved: FM-401 (#90, PR #94 — the difference model with the documented vocabulary and terminal-state precedence on duplicates, total availability-honest observed-state normalization with per-surface answered flags and bidirectional comparison, and a state-aware planner with dependency ordering and a shared dry-run serializer; three cubic review rounds addressed). Resolved: FM-400 (#89, PR #93 — nine resource kinds joining the FM-005 envelope with typed specs and JsonSchema-derived publication, per-kind if/then schema selection with hoisted \$defs, semantic validation refusing credential-bearing references and non-normalizable remotes before activation, and a pure-function composition engine with provenance, deny-through-traversal, order-independent skill sets, escaped capability identities, and scalar conflicts rejected; two cubic review rounds addressed, with the two remaining migration findings carried as the documented decision from PR #86).
+**Status: Code-complete (as of 2026-09-20).** Created as epics #5–#8 with issues #89–#92. The wave order is: FM-400 (#89 desired resource schema and composition) → FM-401 (#90 observed state, difference model, and planner) → FM-402 (#91 apply engine) → FM-403 (#92 Git source and GitHub bootstrap). M4 was an independent follow-on, not on the critical path to the first Lab release. Resolved: FM-403 (#92, PR #96 — the Git source provider with isolated worktrees, validated candidate digests, symlink refusal, and bounded/redacted output; the desired-source use cases gating activation through recorded-candidate verification and a backend-serialized critical section; source.fetch/source.activate executor kinds with a STRICT source storage migration; and the GitHub bootstrap provider with least-permission, expiring tokens; three cubic review rounds addressed). Resolved: FM-402 (#91, PR #95 — the apply engine as pure semantics (approval gate bound to plan identity/order/kind derived from the catalog's risk classification, compensations matching step semantics, post-apply verification) plus a workflow executor with deadline enforcement, cancellation honored before verification, failure detail propagated from inner operations, and payload validation as the last line of defense against generic-surface bypasses; the authz catalog grown to 34 entries; three cubic review rounds addressed). Resolved: FM-401 (#90, PR #94 — the difference model with the documented vocabulary and terminal-state precedence on duplicates, total availability-honest observed-state normalization with per-surface answered flags and bidirectional comparison, and a state-aware planner with dependency ordering and a shared dry-run serializer; three cubic review rounds addressed). Resolved: FM-400 (#89, PR #93 — nine resource kinds joining the FM-005 envelope with typed specs and JsonSchema-derived publication, per-kind if/then schema selection with hoisted \$defs, semantic validation refusing credential-bearing references and non-normalizable remotes before activation, and a pure-function composition engine with provenance, deny-through-traversal, order-independent skill sets, escaped capability identities, and scalar conflicts rejected; two cubic review rounds addressed, with the two remaining migration findings carried as the documented decision from PR #86). All four issues merged and epics #5–#8 closed; the live end-to-end convergence run remains on the maintainer's acceptance checklist, so the milestone is code-complete rather than complete. Next milestone: M6 (Proxmox), beginning with the fallback implementation the FM-S08 spike chose (FM-600).
+
+### FM-S08 — Spike: Proxmox client (typed crate versus reqwest transport)
+
+**Context:** M6 epic #9 depends on this spike; the ecosystem log's Proxmox section recorded a young, experimental typed crate as the only candidate at plan time.
+**Status:** Done 2026-09-20 (issue #97). Fallback chosen — small `reqwest` transport plus typed provider DTOs. Evidence and decision recorded in [research/ecosystem.md](../research/ecosystem.md#fm-s08-proxmox-client-compatibility-spike) and [spikes.md](spikes.md): the typed crate's TLS surface (`accept_invalid_certs(bool)` only) cannot satisfy fingerprint pinning against PVE's cluster CA without disabling verification, and the fallback's pinned-fingerprint rustls verifier was proven live against the PVE 9.2 integration host (positive and negative case) on the workspace's existing reqwest 0.12 + ring stack. The PVE 8.x leg of the both-majors acceptance criterion is a recorded deviation: no 8.x host is reachable in the integration environment, so 8.x evidence is the endpoint/auth/task-shape documentation from the PVE 8.x API archive, and the live 8.x validation moves to the M6 real-cluster suite.
 
 ### FM-400 — Add the desired resource schema and deterministic composition
 

docs/planning/spikes.md

@@ -37,13 +37,38 @@ Spike outcomes:
   unsafe for Fleet semantics or an explicit non-goal of the first release. The
   purpose-built claim/complete loop over the existing table keeps one database,
   one transaction boundary, and one state machine.
+
 | FM-S04 | How does `fleetd` install as a Windows service, expose a named pipe, and authenticate a local peer at least as strictly as Unix socket peer credentials? | Later Windows in-guest slice | ADR-0003 | Windows `fleetd`/project-readiness epic | Keep Windows support at Proxmox lifecycle and QEMU Guest Agent observation until the broker can be secured |
 | FM-S05 | Is `purple_ssh` reusable as a dependency, as extracted MIT code with attribution, or only as a reference implementation? | M2 | ADR-0005 | FM-201, FM-202 | Direct system OpenSSH invocation using Purple's tested behaviour as a reference only |
 | FM-S06 | Does `skills-manager-cli --json` cover agents, skills, presets, deploy/undeploy, and update status on both supported platforms, and which version range is pinned? | M3 | ADR-0005 | M3 Skills Manager epic | Degrade to detection and status only, and open an upstream contract request |
 | FM-S07 | Which Frogenv commands can run non-interactively with machine-readable output, and which approval ceremonies must stay manual? | M3 | ADR-0005 | M3 Frogenv epic | Report `blocked: manual approval required` and hand off to the operator |
 | FM-S08 | Does the experimental `proxmox-client` crate satisfy authentication, UPID task polling, custom TLS trust and pinning, unknown-field tolerance, and cancellation against PVE 8.x and 9.x? | M6 | ADR-0005 | M6 Proxmox epic | Small `reqwest` transport plus typed provider DTOs, borrowing Purple's parsing patterns |
 | FM-S09 | Which Packer/Proxmox-plugin version range supports modern `.pkr.json`, required template builds, stable machine-readable diagnostics, cancellation, and acceptable redistribution/deployment terms? | M7 | ADR-0005 | Image recipe/build/version epic | Require an operator-installed supported CLI and keep the image-build port available for another implementation |
 
+- **FM-S08 (resolved 2026-09-20, fallback chosen).** The typed
+  `proxmox-client` crate (crates.io, `landrzejewski`, 0.9.2) failed the
+  spike's security gate: its entire TLS surface is
+  `accept_invalid_certs(bool)` — no fingerprint-pinning hook and no custom
+  trust store — while PVE hosts present their own cluster CA, so every
+  working configuration either disables verification (forbidden) or pins
+  outside the crate. Everything else passed: token auth, UPID status/log/stop,
+  broad QEMU/LXC/cluster/storage coverage, and genuinely tolerant decoding of
+  loose/null shapes. Supply-chain weight told the same story: 3 commits, 3
+  releases in one week, zero stars, 591 downloads, one maintainer, and a
+  reqwest 0.13 + aws-lc-rs TLS stack duplicating the workspace's reqwest 0.12
+  + ring. The fallback — a small `reqwest` transport with a custom rustls
+  verifier that pins the leaf certificate's SHA-256 fingerprint and refuses
+  mismatched hosts at the handshake — was proven live against the PVE 9.2
+  integration host, positive and negative case, using the same TLS stack
+  Fleet already standardizes on. Evidence and the rejected alternatives:
+  [research/ecosystem.md](../research/ecosystem.md#fm-s08-proxmox-client-compatibility-spike).
+  The PVE 8.x leg of the both-majors acceptance criterion is a recorded
+  deviation: no 8.x host is reachable in the integration environment, so 8.x
+  evidence is the endpoint/auth/task-shape documentation from the PVE 8.x API
+  archive, and the live 8.x validation moves to the M6 real-cluster suite.
+  The spike's decisive evidence is TLS behavior, which is client-side and
+  version-independent.
+
 ## Rules
 
 - A spike is blocking only for the issues in its **Consumed by** column. Unrelated work in the same milestone proceeds.

docs/research/ecosystem.md

@@ -142,6 +142,29 @@ Findings:
 
 Decision: M6 begins with a compatibility spike against required endpoints and PVE versions. Prefer the typed crate only if authentication, task polling, TLS custom trust/pinning, unknown-field tolerance, cancellation, and endpoint coverage pass. Otherwise keep a small `reqwest` transport plus typed provider DTOs, borrowing Purple's tested parsing patterns. Never accept invalid certificates as the production answer.
 
+#### FM-S08: Proxmox client compatibility spike
+
+Decision (2026-09-20): **fallback chosen — a small `reqwest` transport plus typed provider DTOs with a custom rustls fingerprint-pinning verifier.** The spike tested the leading typed crate against the live PVE 9.2 integration host (`fleet-test-01`'s PVE node, `pve.localdomain`, PVE 9.2.2/repo `b9984c6d90a4bd80`) and inspected its source; the decisive failure is TLS, which is a security gate, not a feature gap.
+
+Evidence from a disposable probe project (outside this repository), Rust 1.98 toolchain:
+
+| Required evidence | Observed result |
+|---|---|
+| Authentication | [`proxmox-client` 0.9.2](https://crates.io/crates/proxmox-client) API-token auth worked live (`PVEAPIToken` header) — version, cluster resources, node list, QEMU list/status, task list, task status, and a real `qmreboot` UPID round-trip all passed. Auth is a builder string; no secret-wrapper integration, but that is adapter work either way. |
+| TLS trust and pinning | **Failed, decisively.** The builder's entire TLS surface is `accept_invalid_certs(bool)`, which maps to reqwest's `danger_accept_invalid_certs`. There is no fingerprint-pinning hook, no custom root store, no `use_preconfigured_tls` passthrough. The PVE host uses its own cluster CA (`PVE Cluster Manager CA`), so system-trust verification fails (`unable to get local issuer certificate`), meaning the only working configurations are "disable verification" — forbidden by the spike rules and the security policy — or nothing. |
+| UPID task polling | Worked: `get_task_status` with `is_running()`/`is_ok()`, plus `stop_task` and `get_task_log`. Gaps: the caller receives a raw `String` UPID and must parse `node`/`type`/`id` out of it itself (the `Upid` newtype has no parsing helpers), and there is no built-in wait/poll helper — Fleet owns the deadline/timeout loop (which is what it wants anyway, to avoid the legacy `waitForTask` wall-clock flake). |
+| Unknown-field tolerance | Good: zero `deny_unknown_fields` across the crate, all response fields `Option`-typed, the `{"data": ...}` envelope unwraps cleanly including `data: null`, and `VmConfig` preserves indexed params (`net0`, `scsi0`, …) in a flattened `extra` map. Live agent responses (loose QGA shapes) parsed fine. |
+| Cancellation | Weak: per-request timeouts and an overall client timeout only; no request-cancellation surface (Fleet owns cancellation by dropping futures, as with any reqwest stack). |
+| Endpoint coverage | Broad — QEMU/LXC lifecycle, snapshots, config, agent (26 agent methods), cluster resources/tasks, storage, tasks, access, pools. Everything M6's first three epics need is present or trivially reachable. |
+| Maintenance and supply chain | Unhealthy for a dependency Fleet must trust: 3 commits ever (all 2026-03-17/18), **zero GitHub stars, 591 total downloads / 29 recent, 3 releases in one week**, one open issue, no README feature list beyond "experimental". MIT (inbound-compatible). Requires reqwest ^0.13.2 whose default TLS is aws-lc-rs — a second TLS backend (the workspace standardizes on reqwest 0.12 + ring) plus a second reqwest major in one graph. `cargo deny check licenses/bans` pass, but the crates.io weight is a single-maintainer zero-adoption snapshot. |
+| Fallback exercised | A probe implemented the fallback's decisive risk — fingerprint pinning over plain reqwest — and passed it live: a custom `rustls::client::danger::ServerCertVerifier` that SHA-256-hashes the leaf certificate and compares it to the pinned fingerprint connected successfully to the real host, and a wrong fingerprint was refused at the TLS handshake (`is_connect() == true`). Verification is never disabled; an unpinned host is refused until an authorized trust step pins it, mirroring the FM-201 SSH TOFU flow. The whole fallback probe used reqwest 0.12 + rustls `ring` — the same TLS stack the tailscale provider already uses — so no new transport dependency class, no duplicated reqwest major, no aws-lc-rs. |
+
+The second candidate inspected, [`proxmox-api` 0.2.0](https://github.com/datdenkikniet/proxmox-api) (schema-generated types, 4.5 MiB of generated code, 32-crate graph, Apache-2.0/MIT, 6 stars), was rejected faster: its default reqwest client hardcodes `danger_accept_invalid_certs(true)`, its Debug output prints the raw API token (no redaction), and its ergonomic surface (typed `VmId(i128)` wrappers, mandatory params structs even for empty GETs, `Option<Vec<_>>` unwrapping) adds friction without solving the same TLS gate. The official `proxmox-rs` workspace crates remain internal/build-time oriented and do not provide a usable PVE automation client.
+
+Consequence for M6: build `fleet-provider-proxmox` on a small `reqwest` 0.12 transport with the pinned-fingerprint rustls verifier (shared trust workflow modeled on FM-201's SSH pin/decide/confirm), typed provider DTOs translating at the boundary, and Fleet-owned UPID parsing, polling loops, and deadlines. Recorded/simulated fixtures plus the dedicated real-cluster suite (epics #10–#12) carry the PVE 8.x/9.x compatibility matrix.
+
+**Recorded deviation against the FM-000 acceptance criterion** ("FM-S08 must produce evidence against both majors; passing on one major is not a pass"): only PVE 9.2.2 was reachable live — the integration environment has a single PVE host and no 8.x node. The 8.x half of the evidence is static, not live: the [PVE 8.x API documentation archive](https://pve.proxmox.com/pve-docs-8/api-viewer/apidoc.js) (verified to be the 8.x generation — it lacks the 9.x-only `sdn/fabrics` endpoints) documents every endpoint the spike exercised, with the same `PVEAPIToken` authentication and `exitstatus` task-status shape. The spike's decisive evidence is TLS behavior, which is client-side and version-independent — the crate's `accept_invalid_certs(bool)` surface cannot pin fingerprints against any PVE major. The client choice is therefore recorded with the deviation named, and the live 8.x leg moves to the M6 real-cluster suite (epics #10–#12), which must validate a PVE 8.x host before Fleet claims 8.x support; that limitation is recorded on epic #9.
+
 ### Rust controller/node stack
 
 Candidate primary sources: [Axum](https://docs.rs/axum/latest/axum/) for HTTP/SSE/WebSocket, [SQLx](https://github.com/launchbadge/sqlx) for compile-checked SQLite/migrations, [Bollard](https://docs.rs/bollard/latest/bollard/) for Docker, [Effectum](https://docs.rs/effectum/latest/effectum/) for an embedded SQLite task queue, and [Cedar](https://github.com/cedar-policy/cedar) for embedded authorization.