Workload container exit is invisible: Pod stays Running and the instance leaks #20

What happened

A workload whose process exits leaves its Pod reported Running forever and its EC2 instance running forever. Nothing observes container state, so Nebula cannot tell a finished workload from a healthy one.

Observed with 30 replicas of config/samples gpu-workload-sample, whose command is:

sh -c "nvidia-smi --query-gpu=... --format=csv || echo 'no nvidia-smi'; sleep 3600"

Timeline from the cluster (all 30 replicas, one hour after start):

Meanwhile kubectl get pods still showed all 30 as 1/1 Running, and all 30 instances stayed running in EC2 — 2+ hours after their only process had exited.

Why

buildUserData emits docker run --rm ... as the last line of user-data (pkg/provider/aws/translate.go:78, :105-108) and the script then ends. When the container exits, nothing runs. The instance idles as running indefinitely.

toState (pkg/provider/aws/aws.go:812) derives instance state from the EC2 instance state alone, so a dead container reads as InstanceRunning, and the vnode poll loop keeps synthesizing a Running Pod status from it.

Two consequences:

1. Wrong status — a completed or crashed workload is indistinguishable from a live one. A crash-looping or instantly-failing container looks perfectly healthy.
2. Leaked instances (the expensive half) — the instance is never reclaimed. 30 idle GPU instances kept billing here.

Not solvable by readiness probes, and not a SandD problem

• Probes don't run. There is no kubelet on a Nebula virtual node; VK synthesizes the entire Pod status from what the provider reports. A readinessProbe on the Pod is ignored data.
• SandD can't report it. The shim (translate.go:178-192) backgrounds sandd, then exec "$@" makes the workload PID 1. sandd is a child of the workload, so it dies in the same instant the container is torn down — it cannot be the thing that reports the exit. (That parent/child direction is correct: the daemon should not outlive the workload it is attached to.)

The gap is host-side, in user-data.

Suggested fix

Have the instance stop itself once docker run returns. This reuses machinery that already exists:

• toState already maps stopped/stopping/terminated → InstanceTerminated
• the poll loop already drives a terminated instance to a terminal Pod
• the owning Deployment then replaces the Pod → a fresh NodeClaim → a fresh instance

That is the delete-and-recreate model the NodeClaim ledger already documents, so no new signalling channel and no container-state observer is needed.

Open design question

The simple version makes any container exit terminal, including exit 0. A workload that legitimately completes would churn a replacement instance instead of settling. Distinguishing the two requires capturing the container's exit code host-side and giving Nebula somewhere to carry it — a larger change. Worth deciding before implementing.

Related

Separately, the SandD controller under-counted live daemons (25 of 30) because its reaper could evict a daemon whose socket was still open. That is a different bug with a fix in flight in the SandD repo (HeartbeatAck → re-register in place); it does not address this issue.