[Lupus]

Summary

On Windows a sandbox is two openvmm.exe processes: the launcher izba records in state.json and an openvmm vm worker that holds the WHP partition, the disk images and the sockets. izba stop verified only the launcher, and only through GetExitCodeProcess — which TerminateProcess satisfies at once, while the kernel-side teardown that releases the handles can hang. Spike #249 caught a worker stuck that way: stop returned 0, deleted state.json and the run dir, status said stopped, and rm --force later failed with a bare Access is denied (os error 5) while the worker still held rw.img — for days.

This PR makes "stopped" mean what it says. One new process-manager primitive, procmgr::tree_survivors, answers which processes of the VMM tree (launcher and descendants) still exist as un-torn-down kernel objects: on Windows the criterion is the process object being signaled (WaitForSingleObject(h, 0)), never the exit code; on Unix it is the root iff pid_alive, i.e. exactly today's gate. It enters the product through the existing liveness::Probes seam, so status, ls, the daemon's stale-state reaper and start's already-running guard all see the same fact.

Behaviour (Windows-visible; Linux normal path unchanged):

• izba status / ls: degraded (vmm process <pid> outlived its launcher and still holds the disks) instead of stopped.
• izba stop: with the launcher gone, first re-sweeps the guarded tree (an orphaned but still-running worker is killed through the same path), polls the whole tree, then refuses (exit 1) while anything survives — naming the pid, saying the disks are held, keeping state.json, and telling the user what to do (Run izba stop <name> again … only a host reboot releases).
• izba rm: without --force reports the degraded state (use force to remove); --force goes through the same sweep + gate and gives the same refusal instead of a raw OS error. A Windows PermissionDenied on the tombstone rename now says a file inside is still open.
• izba start on a stuck sandbox refuses with the same explanation instead of exiting 0 as "already running".
• The recycled-pid hazard of a PPID walk is guarded: a launcher recorded before the current boot has no tree (boot = creation time of pid 4; if unreadable the guard is disabled, never estimated), and when a different process now holds the launcher pid, everything created at or after it is the holder's. The filter is a pure, Linux-tested function.

Changes mapped to the Acceptance Criteria

• Every openvmm.exe of the sandbox (launcher and worker) must be fully gone, not just carrying an exit code, before state.json/run dir removal — procmgr::tree_survivors (unix.rs, windows.rs, new survivors.rs) + stop_locked_with's tree-wide gate after the kill block on both paths (killed now / found already exited).
• stop exits non-zero, names the pid, says the disks are held, says what to do — sandbox::disks_held_error, pinned by assert_eq! on the full text (singular and plural).
• state.json preserved; status not a clean stop — assess rule 2b (Degraded, never Stopped), so reap_stale_stopped keeps the file; pinned by the retry test stop_refuses_when_the_launcher_is_already_gone_but_its_worker_survives.
• rm incl. --force fails with the same explanation, never raw Access is denied — remove_with + rename_for_removal_error; tests for both arms.
• Regression test through a seam/fake, no unkillable process — StuckTeardownProbes/ResweepProbes fake only the survivors verdict; the kill stays real (spawn_sleep + wait_dead). The re-sweep call site is pinned through Probes::sweep_tree_survivors. Windows real-process tests cover the signaled-vs-exit-code distinction, an orphaned worker, the boot/holder guards and the rename hint (they first execute in CI's Windows job).
• Reproduction recipe recorded — docs/spikes/0003-windows-openvmm-worker-teardown-hang.md + hack/spike/stuck-vmm-teardown-check.ps1 + a docs/testing.md §8 pointer.
• Written conclusion on why the guest did not power off and why the worker hangs — spike doc findings 3–4: the worker's last thread sits in a kernel Wait:Executive inside the WHP/vid.sys partition teardown, which TerminateProcess cannot interrupt (outside izba's control; detect and report); the guest's console shows neither the usb detach's disconnect line nor reboot: Power down (printed only after device_shutdown()), pointing at the vhci (usbip) device at shutdown after traffic — filed as Windows/OpenVMM: guest fails to power off with a vhci (usbip) device attached or mid-detach after sustained traffic — stop always escalates to kill #325.
• Normal stop not slower; graceful-then-kill flow and privileges unchanged — one Toolhelp snapshot + zero-timeout waits; no new sleeps on the normal path; PROCESS_QUERY_LIMITED_INFORMATION | SYNCHRONIZE only.

Real-host validation

Validated against the actual stuck worker from the spike (pid 30620, still present on the host): 11/11 checks of the new script pass with the branch's izba.exe — status degraded, stop and rm --force exit 1 with the refusal, state.json kept. The doc records all three runs, including the one that failed because a forged launcher start time is, to the new guards, a recycled pid (the dead launcher's object is still reserved by the worker's handle and reports its real creation time). stop on a truly stuck worker takes ~12 s (the re-sweep's bounded wait) — the price of recovering an orphaned worker that is still running.

Follow-ups filed

• Windows/OpenVMM: guest fails to power off with a vhci (usbip) device attached or mid-detach after sustained traffic — stop always escalates to kill #325 — guest fails to power off with a vhci device attached/mid-detach after traffic (guest side).
• Windows: an auto-spawned izba daemon run inherits the caller's stdout pipe — a piped izba command blocks its consumer until the daemon exits #326 — an auto-spawned izba daemon run inherits the caller's stdout pipe on Windows (found while validating: a piped izba command blocks until the daemon exits).
• Windows: record the boot identity in state.json so the stuck-teardown boot guard survives a clock step-back (follow-up to #319) #327 — record the boot identity in state.json so the boot guard survives a clock step-back.

Known, left as is: a stuck izba start now also tears down that sandbox's daemon-side listeners (fails closed) and the daemon's progress line reads "already up but degraded" just before the error; the supervisor tick still respawns relays for a degraded sandbox (pre-existing).

Testing

• All six gates green on the final tree: cargo test --workspace, cargo clippy --workspace --all-targets -- -D warnings, cargo fmt --check, musl izba-init build, x86_64-pc-windows-gnu check + clippy -D warnings.
• TDD throughout (every new test seen RED first); per-task reviews plus a whole-branch review; the Windows-only tests execute in CI.
• Plan: docs/superpowers/plans/2026-10-08-windows-stuck-vmm-teardown.md.

Closes #319

🤖 Generated with Claude Code

Confidence Score: 5/5

The latest update appears safe to merge; no new blocking issue was established.

Summary

The PR keeps state.json and refuses cleanup while an OpenVMM launcher or worker still holds its resources. The latest update corrects a Windows test's expected result when the boot time cannot be read.

• No new actionable issue was established in this update.
• Earlier fixes use the checked process identity when terminating survivors, read Windows filenames as UTF-16, bound polling by elapsed time, and require opt-in for the destructive check.
• Lupus accepted the remaining same-image ownership risk and deferred it to Windows: record the VMM worker identities in state.json at start so the stuck-teardown walk never infers ownership (follow-up to #319) #329.
• Lupus deferred the clock-step boot check limitation to Windows: record the boot identity in state.json so the stuck-teardown boot guard survives a clock step-back (follow-up to #319) #327. Lupus also acknowledged listener cleanup, the progress message, and relay restarts for degraded sandboxes as known behavior.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Check recorded launcher] --> B{Launcher alive?}
  B -->|Yes| C[Existing running checks]
  B -->|No| D[Check surviving VMM tree]
  D --> E{Any survivor?}
  E -->|No| F[Stopped: cleanup allowed]
  E -->|Yes| G[Degraded: keep state]
  G --> H[Stop sweeps and waits]
  H --> I{Any survivor remains?}
  I -->|Yes| J[Refuse stop and forced removal]
  I -->|No| F

Loading

_{Reviews (12) · Last reviewed commit: "test(procmgr): the pre-boot record is dr..." · Reviewed by Greptile}

[sonarqubecloud]

Quality Gate passed

Issues
2 New issues
0 Accepted issues

Measures
0 Security Hotspots
98.3% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud