Context
When scaled crashes or its heartbeat KV entry expires past the TTL, the control plane considers the node dead and re-auctions all Ignite workloads to other healthy nodes. This means any Firecracker processes left running on the crashed node are now orphans — their workloads have already been placed elsewhere.
On restart, scaled must clean up these orphans before accepting new work. If it does not, leftover Firecracker processes consume memory and CPU, stale TAP devices occupy subnet indices (now that allocation is dynamic per #70), and vsock CIDs may collide.
Requirements
Ignite (stateless) workloads
On startup, before the launcher accepts new Launch() calls:
- Kill all orphaned Firecracker processes — enumerate processes owned by the jailer UID or matching the jailer cgroup tree, send SIGKILL, and wait for exit.
- Remove all stale TAP devices — enumerate host interfaces matching the
tap* prefix with the 02:53: MAC OUI pattern (or simply all tap prefixed devices owned by the jailer UID), and delete them via netlink.
- Clean up workspace directories — remove
/var/lib/spacescale/microvms/* and /var/lib/spacescale/j/firecracker/* leftover jailer chroots.
- Start the subnet and CID allocators fresh — since all VMs are killed, the allocators start empty at index 0. No state reconstruction needed for Ignite.
This is safe because Ignite workloads are stateless and already re-auctioned. There is no state to preserve.
Machines (persistent) workloads — future, different behavior
Machines are not re-auctioned on node failure. They are persistent and expected to survive host-process restarts. When Machines are implemented:
- Do NOT kill their Firecracker processes on startup
- Instead, reconstruct their state: enumerate surviving VMs, rebuild
ActiveVM records, reclaim their subnet indices and CIDs from host state (TAP addresses, vsock sockets)
- Resume lifecycle management (watchVM) for each recovered Machine
This issue covers Ignite cleanup only. Machine recovery will be a separate issue.
Implementation notes
- The cleanup should run in
scaled preflight (after jailer identity resolution, before NATS connect and auction subscription)
- Prefer cgroup-based enumeration if available (
/sys/fs/cgroup/.../firecracker/) over process table scanning
- The
jailer binary places each VM under a known cgroup path, making enumeration reliable
- Log each killed PID, removed TAP, and cleaned directory at INFO level so crashes are auditable
- If cleanup fails (e.g. permission denied removing a workspace), log a WARN and continue — partial cleanup is better than refusing to start
Context
When
scaledcrashes or its heartbeat KV entry expires past the TTL, the control plane considers the node dead and re-auctions all Ignite workloads to other healthy nodes. This means any Firecracker processes left running on the crashed node are now orphans — their workloads have already been placed elsewhere.On restart,
scaledmust clean up these orphans before accepting new work. If it does not, leftover Firecracker processes consume memory and CPU, stale TAP devices occupy subnet indices (now that allocation is dynamic per #70), and vsock CIDs may collide.Requirements
Ignite (stateless) workloads
On startup, before the launcher accepts new
Launch()calls:tap*prefix with the02:53:MAC OUI pattern (or simply alltapprefixed devices owned by the jailer UID), and delete them via netlink./var/lib/spacescale/microvms/*and/var/lib/spacescale/j/firecracker/*leftover jailer chroots.This is safe because Ignite workloads are stateless and already re-auctioned. There is no state to preserve.
Machines (persistent) workloads — future, different behavior
Machines are not re-auctioned on node failure. They are persistent and expected to survive host-process restarts. When Machines are implemented:
ActiveVMrecords, reclaim their subnet indices and CIDs from host state (TAP addresses, vsock sockets)This issue covers Ignite cleanup only. Machine recovery will be a separate issue.
Implementation notes
scaledpreflight (after jailer identity resolution, before NATS connect and auction subscription)/sys/fs/cgroup/.../firecracker/) over process table scanningjailerbinary places each VM under a known cgroup path, making enumeration reliable