terminate is idempotent on a terminated VM — recorded, and the guess it replaces wedged a real consumer

README.md

@@ -52,9 +52,9 @@ The full documentation site is at [intentius.github.io/m80](https://intentius.gi
 
 ## Status
 
-Released. All 29 operations answer, `/_m80/health` reports 29/29 with nothing pending, and the conformance suite runs 100 checks against fixtures recorded from live AWS.
+Released. All 29 operations answer, `/_m80/health` reports 29/29 with nothing pending, and the conformance suite runs 101 checks against fixtures recorded from live AWS.
 
-Running KubeMicroVM's own 63-case UAT suite against m80 surfaced three issues in the operator — the sharpest a finalizer that never clears after a successful terminate, so a deleted CR hangs forever. All three are filed upstream and acknowledged. None needed an AWS account to find. 50 of the 63 pass, with [every failure accounted for](docs/kubemicrovm.md).
+Running KubeMicroVM's own 63-case UAT suite against m80 surfaced three issues in the operator — the sharpest a finalizer that never clears after a successful terminate, so a deleted CR hangs forever. All three were filed upstream; the finalizer fix shipped in their v1.0.12, verified here the day after. None needed an AWS account to find. 52 of the 63 pass, with [every failure accounted for](docs/kubemicrovm.md), and the suite runs against every commit in CI holding exactly that matrix.
 
 Nothing else like it exists. Verified 2026-07-29 across moto, LocalStack (archived), fakecloud, ministack, floci upstream and forks, and every public repo that codes against the MicroVM API. The service went GA 2026-06-22.
 

conformance/cases/82-errors-conflict.json

@@ -114,6 +114,15 @@
         ]
       }
     },
+    {
+      "name": "terminate-terminated",
+      "operation": "TerminateMicrovm",
+      "method": "DELETE",
+      "path": "/2025-09-09/microvms/${microvmId}",
+      "expect": {
+        "status": 200
+      }
+    },
     {
       "name": "delete-image",
       "operation": "DeleteMicrovmImage",

conformance/fixtures/errors-conflict/terminate-terminated.json

@@ -0,0 +1 @@
+{}

docs/api-surface.md

@@ -78,6 +78,12 @@ Facts the models could not state, learned during the fixture-recording runs.
 | Quota errors carry empty detail | `quotaCode`, `serviceCode`, `resourceId`, and `resourceType` are all present and all `null`. A client cannot branch on which quota was hit; only the message says |
 | Concurrency throttling is masked | The burst never produced `ThrottlingException` or any `ThrottleReason`, including `ConcurrentSnapshotCreateLimitExceeded`. The memory ceiling fires first and hides it. KubeMicroVM's QuotaGuard will meet 402 long before it meets a throttle on a default account |
 
+## Recorded corrections (2026-08-06, the double terminate)
+
+| Fact | Detail |
+|------|--------|
+| A second `TerminateMicrovm` on a `TERMINATED` VM is **200, idempotent** | Empty object, exactly like the first — not the 400 the suspend recording suggested. m80 had extrapolated the 400 onto it, and the guess wedged KubeMicroVM v1.0.12's cleanup loop against behaviour the real service never exhibits ([#83](https://github.com/INTENTIUS/m80/issues/83)). The probe that settled it also re-verified suspend-on-terminated as the recorded 400, so the terminated state refuses *changes* while accepting the one request that asks for what is already true |
+
 ## Recorded corrections (2026-08-01, the VM endpoint)
 
 The per-VM endpoint had never been recorded, because the conformance runner addressed the control plane and signed everything it sent. Giving a step its own `baseURL` and headers ([#42](https://github.com/INTENTIUS/m80/issues/42)) made it reachable, and four of m80's nine guesses were wrong.

docs/conformance.md

@@ -15,7 +15,7 @@ HTTP-level, language-agnostic, pointed at an endpoint URL. No SDK in the suite i
 | Target | Purpose | Where it stands |
 |--------|---------|-----------------|
 | Real AWS | Record fixtures, verify the suite itself. Runs rarely, costs money, needs an account | Recorded 2026-07-29, 07-30 and 08-01 |
-| m80 | The full suite, every operation, every lifecycle path, every error | 100 checks, 0 failures, 29/29 operations |
+| m80 | The full suite, every operation, every lifecycle path, every error | 101 checks, 0 failures, 29/29 operations |
 | floci module | The CFN-sufficient subset, tagged so the narrower scope is explicit rather than a pile of skips | 26 checks, 0 failures at `-tier load-bearing` |
 
 Cases record whether a behaviour is fixture-backed against the real service or documented-only, and coverage against the 29-operation inventory falls out of every run.

docs/index.md

@@ -42,6 +42,6 @@ Then, in whatever order the question arrives in: [scope](scope.md) is what it em
 
 ## Status
 
-Released. All 29 operations answer, `/_m80/health` reports 29/29 with nothing pending, and the conformance suite runs 100 checks against fixtures recorded from live AWS. [KubeMicroVM's own UAT suite](kubemicrovm.md) passes 50 of its 63 cases, with every failure accounted for — three of them bugs in the operator, filed upstream and acknowledged.
+Released. All 29 operations answer, `/_m80/health` reports 29/29 with nothing pending, and the conformance suite runs 101 checks against fixtures recorded from live AWS. [KubeMicroVM's own UAT suite](kubemicrovm.md) runs against every commit in CI and passes 52 of its 63 cases, with every failure accounted for. The suite found three bugs in the operator; the sharpest — a finalizer that never cleared — was fixed upstream in v1.0.12 and verified here the day after.
 
 Nothing else like it exists. Verified 2026-07-29 across moto, LocalStack (archived), fakecloud, ministack, floci upstream and forks, and every public repo that codes against the MicroVM API. The service went GA 2026-06-22.

docs/kubemicrovm.md

@@ -140,34 +140,36 @@ Each of these is a difference between this harness and the EKS run the suite was
 ## Pass matrix
 
 <!-- matrix:start -->
-First recorded 2026-08-01 against m80 v0.1.0 and operator 1.0.11, excluding the performance suite; re-measured 2026-08-06 against v0.4.0 with the identical result. Since #61 landed this is no longer a snapshot: CI runs the whole suite against every commit's build (`.github/workflows/uat.yml`) and holds the run to exactly this matrix in both directions — a new failure fails the build, and a listed failure that starts passing fails it too, until this page and `uat/expected-failures.txt` move with it.
+First recorded 2026-08-01 against m80 v0.1.0 and operator 1.0.11; current record 2026-08-06 against operator chart 1.0.12, whose finalizer fix ([KubeMicroVM#51](https://github.com/codriverlabs/KubeMicroVM/issues/51) — a bug this harness found) moved two cases into the passing column. Since #61 landed this is no longer a snapshot: CI runs the whole suite against every commit's build (`.github/workflows/uat.yml`) and holds the run to exactly this matrix in both directions — a new failure fails the build, and a listed failure that starts passing fails it too, until this page and `uat/expected-failures.txt` move with it.
 
-**50 of 63 cases pass.**
+**52 of 63 cases pass.**
 
 | | Suite | Passed |
 |---|---|---|
 | ⚠️ | 00 Cluster Setup | 6/7 |
-| ⚠️ | 01 Quick Start | 7/9 |
+| ⚠️ | 01 Quick Start | 8/9 |
 | ⚠️ | 02 Rbac | 6/8 |
 | ⚠️ | 03 Networking | 2/5 |
 | ⚠️ | 04 Pod Token Injection | 8/9 |
-| ⚠️ | 05 Replicaset | 5/6 |
+| ✅ | 05 Replicaset | 6/6 |
 | ✅ | 06 Microvm Class | 6/6 |
 | ⚠️ | 07 Drift Autosuspend | 4/5 |
 | ⚠️ | 08 Memory Sizing | 5/6 |
 | ⚠️ | 99 Final Cleanup | 1/2 |
 
-### Why the thirteen fail
+### Why the eleven fail
 
-None is m80 answering differently from real AWS. In two cases m80 answering *exactly* as real AWS is what exposes an operator bug.
+None is m80 answering differently from real AWS.
 
-**Reach the endpoint but not m80 (4).** `QS-07`, `NET-02`, `INJ-08`, `AUTO-02`, all `Token authentication failed`. The token is minted by m80 correctly; the suite then curls `https://<uuid>.lambda-microvm.<region>.on.aws/`, that hostname resolves to real AWS, and AWS rejects an m80-issued token. The call never reaches m80. Recording the endpoint against live AWS in [#42](https://github.com/INTENTIUS/m80/issues/42) confirmed this from the other end: `Token authentication failed` is verbatim what AWS returns for a token that does not match the VM the hostname names. Reaching m80 instead needs wildcard DNS *and* TLS, tracked in [#45](https://github.com/INTENTIUS/m80/issues/45).
+**Reach the endpoint but not m80 (3).** `NET-02`, `INJ-08`, `AUTO-02`, all `Token authentication failed`. The token is minted by m80 correctly; the suite then curls `https://<uuid>.lambda-microvm.<region>.on.aws/`, that hostname resolves to real AWS, and AWS rejects an m80-issued token. The call never reaches m80. Recording the endpoint against live AWS in [#42](https://github.com/INTENTIUS/m80/issues/42) confirmed this from the other end: `Token authentication failed` is verbatim what AWS returns for a token that does not match the VM the hostname names. Reaching m80 instead needs wildcard DNS *and* TLS, tracked in [#45](https://github.com/INTENTIUS/m80/issues/45).
 
-**Lose a race with the operator's resync (4).** `RBAC-05`, `NET-01`, `NET-04`, `MEM-07`, reporting `Endpoint for <vm> did not resolve within 60s`. Not DNS: the UAT polls the CR field `.status.endpointUrl`, which the operator leaves at `PENDING` until its next reconcile. Measured at 61 to 65 seconds against a 60 second allowance. Raising `-build-delay` tenfold moved it four seconds, so the interval is the operator's, and the race is as tight against real AWS.
+**Lose a race with the operator's resync (5).** `QS-07`, `RBAC-05`, `NET-01`, `NET-04`, `MEM-07`, reporting `Endpoint for <vm> did not resolve within 60s`. Not DNS: the UAT polls the CR field `.status.endpointUrl`, which the operator leaves at `PENDING` until its next reconcile. Measured at 61 to 65 seconds against a 60 second allowance. Raising `-build-delay` tenfold moved it four seconds, so the interval is the operator's, and the race is as tight against real AWS. A case on the losing side of this race sometimes gets far enough to fail as the first group instead — the two groups trade members between runs; their union is stable.
 
 **Want an AWS-side identity decision (2).** `Pod Identity Association Exists` has none on k3d. `RBAC-06` expects `not authorized`; m80 accepts and echoes IAM without evaluating it, which [scope.md](scope.md) refuses on purpose.
 
-**Hit an operator bug that m80's fidelity exposes (3).** `RS-06`, `99 Final Cleanup`, and `QS-08`. A deleted MicroVM CR keeps its finalizer forever while the operator logs `Cleaning up …` every ten seconds. The MicroVM is present and correctly `TERMINATED`, with `stateReason: "Success."`. The cleanup path appears to wait for it to *disappear*, and it never does, because terminated MicroVMs stay listed, which is recorded live behaviour. Against real AWS the same CR would hang for as long as AWS retains the VM. Filed as [KubeMicroVM#51](https://github.com/codriverlabs/KubeMicroVM/issues/51), where the maintainers have said they intend to fix it.
+**Inherit the debris (1).** `99 Final Cleanup` counts every resource left behind, and the endpoint-race failures above it abandon theirs — `MicroVMImages still exist` names whichever image `MEM-07` was mid-verifying when its clock ran out. It fails as a consequence, not a cause; it would pass the moment the race group did.
+
+Until chart 1.0.12, a fifth group existed: `QS-08` and `RS-06` (and `99` for this reason rather than debris) hit an operator bug m80's fidelity exposed — a deleted CR whose finalizer never cleared. That is [#51](https://github.com/codriverlabs/KubeMicroVM/issues/51), fixed upstream in v1.0.12, and verifying the fix took one more recording: the operator's new cleanup re-issues the terminate, m80 was answering the retry with an extrapolated 400, and the real service turned out to accept it idempotently ([#83](https://github.com/INTENTIUS/m80/issues/83), recorded 2026-08-06). The emulator was the last thing standing between the fix and the passing column — which is the correct order for an emulator to be wrong in: visibly, and once.
 
 <!-- matrix:end -->
 

internal/vms/handlers.go

@@ -164,9 +164,12 @@ func (h *handlers) list(w http.ResponseWriter, r *http.Request) {
 	api.WriteJSON(w, http.StatusOK, map[string]any{"items": items, "nextToken": nil})
 }
 
-// mutable resolves the VM and rejects the one case the recording pinned down:
-// any state change on a terminated VM is a plain 400 ValidationException, not
-// either modeled conflict type. Suspend, resume and terminate share it.
+// mutable resolves the VM and rejects the case the recording pinned down:
+// a state change on a terminated VM is a plain 400 ValidationException, not
+// either modeled conflict type. Suspend and resume share it. Terminate does
+// NOT — see terminate; a second terminate was recorded 2026-08-06 as a plain
+// 200, and answering it with this 400 wedged a real consumer's cleanup loop
+// (KubeMicroVM v1.0.12, upstream #51) against behaviour AWS never exhibits.
 func (h *handlers) mutable(w http.ResponseWriter, r *http.Request) (*VM, bool) {
 	vm, ok := h.lookup(w, r)
 	if !ok {
@@ -205,12 +208,20 @@ func (h *handlers) resume(w http.ResponseWriter, r *http.Request) {
 	writeAccepted(w)
 }
 
+// terminate is idempotent on a VM that is already terminal, recorded from
+// live AWS 2026-08-06: the second DELETE answers 200 with an empty object,
+// exactly like the first (fixture lifecycle/terminate-terminated). This is
+// the same reasoning unrecorded.md applies to suspend-non-running — a
+// reconciler's retry must not be an error — except here it is no longer
+// reasoning, it is a recording.
 func (h *handlers) terminate(w http.ResponseWriter, r *http.Request) {
-	vm, ok := h.mutable(w, r)
+	vm, ok := h.lookup(w, r)
 	if !ok {
 		return
 	}
-	h.svc.Terminate(vm)
+	if snap := h.svc.Snapshot(vm); !snap.Terminal() {
+		h.svc.Terminate(vm)
+	}
 	writeAccepted(w)
 }
 

internal/vms/vms_test.go

@@ -205,15 +205,19 @@ func TestTerminateWalksThroughTerminating(t *testing.T) {
 }
 
 // Recorded and surprising: a terminal-state mutation is a plain 400
-// ValidationException, not either conflict type the model offers.
+// ValidationException, not either conflict type the model offers. This test
+// used to drive the 400 with a second terminate — which turned out (recorded
+// 2026-08-06) to be the one mutation that answers 200 instead; suspend
+// carries the recorded 400 now, and TestTerminateOnTerminatedVMIsIdempotent200
+// below pins the exception.
 func TestMutatingTerminatedVMIs400Validation(t *testing.T) {
 	h := newHarness(t, stubImages{runnable: true})
 	id, _ := h.run(t)
 	h.clk.Advance(hop)
 	h.do("DELETE", "/2025-09-09/microvms/"+id, nil)
 	h.clk.Advance(hop)
 
-	rec, doc := h.do("DELETE", "/2025-09-09/microvms/"+id, nil)
+	rec, doc := h.do("POST", "/2025-09-09/microvms/"+id+"/suspend", nil)
 	if rec.Code != http.StatusBadRequest {
 		t.Fatalf("status %d, want 400", rec.Code)
 	}
@@ -603,6 +607,32 @@ func TestSuspendAndResumeOnTerminatedVMAre400(t *testing.T) {
 	}
 }
 
+// Recorded 2026-08-06 against live AWS: a second terminate on a TERMINATED
+// VM answers 200 with an empty object, exactly like the first. m80 used to
+// extrapolate the suspend-on-terminated 400 onto it, and that guess wedged
+// KubeMicroVM v1.0.12's cleanup loop — a retry the real service accepts
+// idempotently (upstream #51, m80 #83).
+func TestTerminateOnTerminatedVMIsIdempotent200(t *testing.T) {
+	h := newHarness(t, stubImages{runnable: true})
+	id := h.runIdle(t, 0, 0)
+	h.do("DELETE", "/2025-09-09/microvms/"+id, nil)
+	h.clk.Advance(hop)
+	if got := h.state(t, id); got != StateTerminated {
+		t.Fatalf("state %v, want TERMINATED", got)
+	}
+
+	rec, doc := h.do("DELETE", "/2025-09-09/microvms/"+id, nil)
+	if rec.Code != http.StatusOK {
+		t.Errorf("second terminate: status %d, want 200", rec.Code)
+	}
+	if len(doc) != 0 {
+		t.Errorf("second terminate: body %v, want empty object", doc)
+	}
+	if got := h.state(t, id); got != StateTerminated {
+		t.Errorf("state %v after second terminate, want TERMINATED", got)
+	}
+}
+
 func TestSuspendOnMissingVMIs404(t *testing.T) {
 	h := newHarness(t, stubImages{runnable: true})
 	rec, _ := h.do("POST", "/2025-09-09/microvms/microvm-00000000-0000-0000-0000-000000000000/suspend", nil)

uat/expected-failures.txt

@@ -3,10 +3,14 @@
 # uat/assert-matrix.py holds a run to exactly this set, both directions: a
 # failure not listed here is a regression, and a listed failure that starts
 # passing means this file and docs/kubemicrovm.md's matrix must move with
-# reality. Why each of these fails is accounted for, line by line, in
-# docs/kubemicrovm.md's "Why the thirteen fail".
+# reality. Why each fails is accounted for, line by line, in
+# docs/kubemicrovm.md's failure account.
 #
-# Recorded 2026-08-06 against m80 v0.4.0 (source build) and operator 1.0.11.
+# Recorded 2026-08-06 against m80 (source, with the idempotent-terminate
+# recording) and operator chart 1.0.12 — the release that fixed the
+# finalizer bug m80 found (upstream #51): QS-08 and RS-06 left this list
+# there. 99 Final Cleanup remains, for a new reason the account explains:
+# it inherits debris from the endpoint-race failures above it.
 AUTO-02 Auto-Resume On Traffic
 INJ-08 Token Works To Call MicroVM
 MEM-07 Run VM From 4096 MiB Image
@@ -15,8 +19,6 @@ NET-02 Default Egress Has Internet Access
 NET-04 VPC Egress VM Connects
 Pod Identity Association Exists
 QS-07 Curl Endpoint Returns OK
-QS-08 Teardown Delete VM
 RBAC-05 Authorized SA Gets Token Via Operator
 RBAC-06 Authorized SA Rejected For Different VM
-RS-06 Delete ReplicaSet Terminates All VMs
 Verify No Resources Remaining

uat/run.sh

@@ -27,7 +27,7 @@ CLUSTER="${CLUSTER:-m80-uat}"
 NS="${NS:-kube-microvm}"
 REGION="${REGION:-us-east-1}"
 ACCOUNT_ID="${ACCOUNT_ID:-000000000000}"
-CHART_VERSION="${CHART_VERSION:-1.0.11}"
+CHART_VERSION="${CHART_VERSION:-1.0.12}"
 KUBEMICROVM="${KUBEMICROVM:?set KUBEMICROVM to a KubeMicroVM checkout}"
 RESULTS="${RESULTS:-$(pwd)/uat-results}"
 HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"

uat/up.sh

@@ -38,7 +38,7 @@ if [ "${CLUSTER}" != "m80-uat" ]; then
 fi
 NS="${NS:-kube-microvm}"
 M80_IMAGE="${M80_IMAGE:-ghcr.io/intentius/m80:v0.4.0}"
-CHART_VERSION="${CHART_VERSION:-1.0.11}"
+CHART_VERSION="${CHART_VERSION:-1.0.12}"
 REGION="${REGION:-us-east-1}"
 # m80 defaults to the account memory ceiling recorded from a fresh AWS account:
 # 4096 MiB, which at the 2048 MiB default tier is two concurrent MicroVMs. The
@@ -55,7 +55,14 @@ k3d cluster delete "${CLUSTER}" >/dev/null 2>&1 || true
 HERE="$(cd "$(dirname "$0")" && pwd)"
 (cd "${HERE}/cluster" && npm install --no-audit --no-fund >/dev/null \
     && npx chant build . -o dist/k3d-uat.yaml --format yaml >/dev/null)
-k3d cluster create --config "${HERE}/cluster/dist/k3d-uat.yaml" --wait --timeout 300s >/dev/null
+# The kubeconfig flags repeat what the declared config already says, on
+# purpose: k3d 5.8.3 was observed ignoring `options.kubeconfig` when it
+# comes via --config (twice, across two repos), and every kubectl call
+# below relies on the ambient context being this cluster. Stated as flags,
+# the behaviour is deterministic whatever the config loader does.
+k3d cluster create --config "${HERE}/cluster/dist/k3d-uat.yaml" \
+    --kubeconfig-update-default --kubeconfig-switch-context \
+    --wait --timeout 300s >/dev/null
 
 echo "==> cert-manager (operator webhooks need it; EKS clusters in the upstream flow already have it)"
 helm repo add jetstack https://charts.jetstack.io >/dev/null 2>&1 || true