Upstream KubeMicroVM: chart drops unknown app.envs keys, and the UAT prerequisites are undocumented

[lex00]

Smaller findings from the k3d harness (#18), all in KubeMicroVM rather than m80. Grouped because each is a few lines and none warrants its own upstream issue.

1. The chart silently drops env keys it does not know

helm install ... --set app.envs.AWS_ACCESS_KEY_ID=test

renders nothing. AWS_MICROVM_ENDPOINT and AWS_REGION work because they are in the default values; a new key is dropped without warning. The harness patches credentials in afterwards with kubectl set env, which works but should not be necessary. Silently discarding a --set is the part worth fixing — an error would have been fine.

2. cert-manager is a hard prerequisite and is not documented

The chart declares Certificate and Issuer resources, so a cluster without cert-manager fails the install outright:

resource mapping not found for kind "Certificate" in version "cert-manager.io/v1"
ensure CRDs are installed first

uat/README.md lists Pod Identity, the microvm CLI, and AWS credentials as prerequisites, but not this. EKS clusters in their workflow presumably already have it.

3. The namespace label in the docs does not match the webhook

The admission webhook requires lambda.aws.amazon.com/manage-microvms=true. A reader following the guides can end up applying microvm-enabled, and the rejection message is good but only appears at CR-create time, well after setup.

4. The microvm CLI is published for Linux only

Release assets are microvm-linux-amd64 and microvm-linux-arm64. There is no darwin build, so a macOS developer cannot run the CLI-dependent UAT cases at all. The harness works around it by running the whole suite in a Linux container. Worth either publishing darwin binaries or saying plainly that the UAT requires Linux.

Tasks

• Confirm each against the released chart and current docs
• File upstream, probably as one issue with four points (outward-facing — needs approval)

Gate: filing upstream is outward-facing and needs explicit approval.

[lex00]

Adding a fifth, and it is more serious than the other four — a hang rather than a papercut.

5. Cleanup never gives up when the MicroVM is already gone

If the MicroVMs API returns 404 for the id in status.microVmId, the operator retries cleanup forever and never removes its finalizer. The CR is stuck in Terminating permanently and only kubectl patch to strip the finalizer clears it.

Observed against six CRs at once:

NAME                   STATE         DELETED                FINALIZERS
rs-pool-221542-4hs8x   Terminating   2026-08-01T22:17:18Z   [lambda.aws.amazon.com/microvm-finalizer]
...

with the operator logging, every ten seconds, indefinitely:

INFO [MicroVMReconciler] Cleaning up MicroVM default/rs-pool-221542-4hs8x
INFO [MicroVMReconciler] Cleaning up MicroVM default/rs-pool-221542-4hs8x

GET /2025-09-09/microvms/microvm-2b169838-… returns 404 — the VM genuinely does not exist.

How I hit it, and why it still matters

Honestly: I caused it. m80 holds state in memory, I restarted it to change a flag, and every VM it knew about vanished. That is not something real AWS does.

It still matters, for two reasons. Deleting a resource that is already gone should be success, not an infinite retry — "already absent" is the desired end state, and this is the standard idempotent-delete expectation for a finalizer. And a Kubernetes object that can never be deleted without manual intervention is a bad failure mode whatever provoked it: a namespace containing one will not delete either.

A reconciler that treats 404 on the terminate path as "cleanup complete, drop the finalizer" fixes it.

Note for the m80 side

Nothing to fix here, but worth writing down: restarting m80 orphans every CR that references a VM it had. The harness should tear the cluster down with m80 rather than restart m80 under a live operator — uat/README.md now says so.

[lex00]

Correcting point 5, and it is more serious than I wrote.

I said I caused the finalizer hang by restarting m80 and making the VMs 404. That was true of the instance I looked at, but it is not the only way in — and not the important one.

A clean run, no restarts, reproduces it with the MicroVM present and correctly terminated:

NAME                   STATE         ID                                             FIN
rs-pool-232716-2298j   Terminating   microvm-56f7babd-4b63-4ca3-8247-72b97d6afe65   [lambda.aws.amazon.com/microvm-finalizer]

and the API says, plainly:

{ "state": "TERMINATED", "stateReason": "Success.", "terminatedAt": 1785626906.93 }

The operator keeps logging Cleaning up MicroVM default/rs-pool-232716-2298j every ten seconds regardless, and never removes the finalizer.

Why this matters more than the 404 case

The cleanup path appears to wait for the MicroVM to disappear, not to reach TERMINATED. It never disappears: terminated MicroVMs stay listed — that is recorded live behaviour, and the reason a recorded ListMicrovms fixture can never match a fresh emulator (see docs/lifecycle.md).

So this is not an emulator artefact. m80 is faithfully reproducing what the real service does, and that faithfulness is what exposes the bug. Against real AWS the same CR would hang for as long as AWS retains the terminated VM.

Two ways in, then, and the second is the one to report: a terminate that succeeds leaves a Kubernetes object that cannot be deleted. TERMINATED should be treated as cleanup complete and the finalizer dropped, exactly as 404 should be.

What it costs in the matrix

RS-06 Delete ReplicaSet Terminates All VMs and 99 Final Cleanup — Verify No Resources Remaining, every run. Three of five ReplicaSet VMs were still present seven minutes after deletion.

[lex00]

Filed upstream, split in two so the finalizer bug isn't buried in a docs list:

• MicroVM finalizer never removed after a successful terminate codriverlabs/KubeMicroVM#51 — finalizer never removed after a successful terminate
• Undocumented prerequisites, and the chart drops unknown app.envs keys codriverlabs/KubeMicroVM#52 — undocumented prerequisites, chart drops unknown app.envs keys