Skip to content

@stateless/proxmox/qemu: guestExec and lookup both report success without doing their job #11

Description

@donwlewis

Extension: @stateless/proxmox v2026.06.12.1 — the latest published version
swamp: 20260803.220438.0-sha.5a5f0f92 (both issues reproduced on 20260714.013601 and
again on this build after updating)
Environment: Linux, PVE 9, three-node cluster, guests provisioned by a swamp workflow

I searched guestExec, exit code, exitCode, proxmox, qemu, lookup and data version
before filing; nothing existing covers either. The closest precedent is #1427 (a
workflow-assert step whose CEL fails to evaluate is omitted from --junit, so the XML reports a
clean pass) — the same failure shape in a different subsystem, which is why I think these are
worth treating as one class of problem.

Both methods below signal success based on the PVE API call completing, not on the
intended effect occurring. Workflows built on them report green while doing nothing, which
is materially worse than failing.


1. guestExec discards the guest command's exit code

guestExec reports succeeded whenever the PVE API call succeeds. The exit status of the
command inside the guest is returned as data but never affects the step result, so a command
that fails is a successful step.

Reproducer

swamp model @stateless/proxmox/qemu method run guestExec my-vms \
  --input vmid=<running guest> --input 'command=["bash","-c","exit 42"]'

  Completed  guestExec on my-vms succeeded in 545ms
  exitCode   42

Why it matters

A workflow using guestExec for provisioning cannot fail on a broken command. In a
three-node cluster build this concealed four separate failures, each reported as succeeded:

What actually happened What the workflow reported
An installer step exited 22 after its download returned HTTP 404 succeeded
A verification step printed FAIL: only 1 of 3 nodes ready and exited 1 succeeded
A filesystem-grow step operated on the wrong disk (hard-coded device name) succeeded
A package-install step installed nothing — one package absent, and apt is all-or-nothing succeeded

The run reported 46/46 steps succeeded for what was actually a one-node cluster. Each
failure had to be found by hand afterwards.

Contrast

@swamp/ssh's script method propagates exit codes correctly — a non-zero exit becomes
method_execution_failed and fails the step (confirmed with a throwaway workflow). So the
expected behaviour is already established elsewhere in swamp, and the two transports disagree.

Suggested fix

Fail the step on a non-zero guest exit code. If the current behaviour is load-bearing for some
callers, an opt-in (allowNonZeroExit: true) would preserve it while making the safe case the
default.


2. lookup reports succeeded but never writes a data version

lookup appears to be the method for refreshing a guest's recorded state. It reports
succeeded and writes no data version at all, so a stale or incorrect record cannot be
corrected by it.

Reproducer

Against a guest that is definitely running:

swamp data list my-vms | grep vm-<id>
  vm-<id>  v26  application/json  176B  <date>     # before

swamp model @stateless/proxmox/qemu method run lookup my-vms --input vmid=<id>
  Completed  lookup on my-vms succeeded in 101ms

swamp data list my-vms | grep vm-<id>
  vm-<id>  v26  application/json  176B  <date>     # unchanged — no new version

Why it matters: the record is wrong and cannot be repaired

A VMID in my repo was reused during a rebuild — the original guest was deleted, then a new
guest was created at the same VMID. The stored record froze at the delete and, because
lookup writes nothing, it has stayed that way ever since:

{"vmid":<id>,"node":"<node>","status":"unknown","lastOperation":"lookup",
 "recordedAt":"<T0>","deletedAt":"<T1>","deletedByMethod":"delete"}

Meanwhile the guest at that VMID is up and in service — getConfig against the same model and
VMID returns its live configuration today. So the data model reports a running guest as
deleted, and the method whose job is to refresh that reports success without changing it.

This produced a real false confirmation: querying the model to check whether guests had been
removed returned "deleted" for guests that were running.

Supporting detail

  • status is "unknown" rather than running/stopped, so even the value that was written
    never carried usable state.
  • getConfig on the same model and VMID does write a new version, so this is specific to
    lookup rather than the model being misconfigured.
  • @swamp/method-summary for the run confirms nothing is being swallowed:
    {"method":"lookup","status":"succeeded","dataOutputs":null,"error":null} — no output, no
    error, reported as success.

Suggested fix

lookup should write a data version reflecting observed state, with a real status, and clear
deletedAt when the guest is found to exist. If the guest is genuinely absent it should record
that explicitly rather than leaving a prior record standing.


Upstream repository: https://github.com/stateless/swamp-extensions

Environment

  • Extension: @stateless/proxmox@2026.06.12.1
  • swamp: 20260803.220438.0-sha.5a5f0f92
  • OS: linux (x86_64)
  • Deno: 2.8.3
  • Shell: /usr/bin/zsh

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions