Repository navigation
Description
Problem
Maintenance passes and stim gc --dry-run fail with STIM_CLAIM_UNAVAILABLE because the cleanup of a "report child" cannot be verified:
STIM_CLAIM_UNAVAILABLE: Cannot verify cleanup for report child 99108 (its identity was not captured at spawn: ACCESS_DENIED (Process inspection was denied)); its process or descendants were left untouched.
Check whether the report child 99108 (`ps -p 99108`) and its descendants are still running and stop them if they are, then run `stim gc --dry-run` again.
The child was already gone when checked (ps -p 99108 printed nothing). In the daemon the same error ends the maintenance pass, so cleanup does not run on that tick. It happens under load, which is when cleanup is needed.
Evidence
- Stim 2.0.0-next.20261115053519.g626efc0, macOS, one MacBook with many agents building.
stim gc --dry-runon 2026-11-15 exited 1 with the text above after "checking 156 projects for merged or idle worktrees".~/.stim/daemon/daemon.ndjsonhas 22 records with "its identity was not captured at spawn: ACCESS_DENIED" between 2026-11-11 and 2026-11-15: 1 on the 11th, 4 on the 13th, 11 on the 14th, 6 on the 15th. 18 aredaemon_cleanup_unverified("stimd goes on and stays up until the child has stopped") and 4 aredaemon_maintenance_failed.- On 2026-11-14 two of the failed passes fall inside the window in which the disk went from 34 GB to 0.2 GB free (Under load, the Cargo target trim at high and critical waits hours on the deferred size scan #5759).
/bin/pson macOS is setuid root (-rwsr-xr-x root)./usr/sbin/lsofis not.
Cause
Not confirmed; this is an inference from the code and the evidence.
packages/stim-cli/src/exec.ts (around 205-250): when a report child is cancelled or times out, cancel observes it. With a captured identity it uses inspectProcessIdentity. Without one it falls back to process.kill(-pid, 0) and treats only ESRCH as gone; anything else is unknown, and after one second of unknown it refuses with the message above.
A child that is ps runs as root once it has exec'd, because the binary is setuid. From an unprivileged process, inspecting it is denied, so an identity capture that happens after the exec fails with ACCESS_DENIED, and kill(-pid, 0) on its group returns EPERM, not ESRCH. Under load ps is slow enough to hit its timeout, so the cancel path runs with no identity and an answer that can never become "gone" while the child lives, and may stay "unknown" for a moment after it exits. That would explain why it appears under load and why the child is gone by the time a person checks.
Not checked: which report children fail (the record does not name the command), whether the capture races the exec or always fails for ps, and what kill(-pid, 0) returns in each state.
Fix idea
- Record the command of the report child in the refusal and the log record, so the failing tool is known.
- Capture the child's identity before it can change credentials, or treat a child that is a known setuid report tool differently: for example verify it through its own exit (the
closeevent of the spawned process is authoritative for the child itself) and through its process group for descendants. - Treat EPERM from
kill(-pid, 0)as "a process exists that this user may not signal", and decide what that means for a report child that Stim itself spawned and has seen exit.
Whatever is chosen has to keep LLP 0007: a state Stim cannot establish is a refusal, never a silent pass. The question is whether "the child I spawned has exited and its group is empty" can be established here without process inspection.
Activity
janicduplessis commentedon Nov 15, 2026
Claimed: fix/5766-report-child-command
janicduplessis commentedon Nov 15, 2026
Diagnosis (2026-11-15)
Measured on one MacBook, macOS 27, with real spawned children. Not measured on Linux or Windows. The "Cause" section above was partly wrong; this replaces it.
What holds
/bin/psin 200 of 200 runs,/usr/bin/topfor its whole run.xcrun,lsof,du,git,security,system_profilerandsleepcapture normally.top -l 3withtimeoutMs: 300insiderunReportrefuses after about 1.35 s.sleepwith the same timeout is killed normally.What was wrong
process.kill(pid, 0)andprocess.kill(-pid, 0)return success for a running setuid child that the same user spawned, not EPERM. Checked again separately withtop -l 2 -s 1.exitevent, atcloseand afterwards. There is no window in which an exited child still looks alive.Cause as established
A policy path in
reportChild(packages/stim-cli/src/exec.ts). When a report is cancelled (timeout, budget or abort) while an uncaptured child is still alive,cancel()never signals it,observe()maps "group alive" tounknown, and after a fixed 1 s ofunknownit refuses. Under load apsoutlives its deadline by more than 1 s and exits on its own shortly after, which is why it is gone when a person checks.Still unknown
psis the likely one (about a dozen report callers use it) but the records do not name it. fix: name the report child command in a cleanup refusal #5769 adds the command to the refusal and the log.pstakes that long under load.Fix options
exit/closefor the existing cleanup budget (groupExitBudgetMs(REPORT_CHILD_EXIT_CAP_MS), capped at 15 s) instead of 1 s, then apply the existing group check and refuse as today if it is still alive. Covers the observed case, where the child exits by itself. Apsthat hangs past the budget still ends the pass. A cancelled report waits up to the budget instead of failing after 1 s.ChildProcesshandle as the identity proof and SIGKILL the child (and, beforeexitis seen, its group, since it was spawned detached). Covers a hung child too. This is a new signalling rule for a process with no captured identity, so it needs a decision under LLP 0007 and probably an amendment.Going ahead with option 1 after #5769 merges: it changes a wait, not what is signalled or when a state counts as established. Option 2 waits on these decisions:
exitwith live descendants and no record: keep refusing (proposed), or accept the leader's exit for known report tools?janicduplessis commentedon Nov 15, 2026
Claimed: fix/5766-uncaptured-child-wait
janicduplessis commentedon Nov 15, 2026
Measured from the real daemon log (21 pairs of
daemon_cleanup_unverifiedand the followingdaemon_cleanup_settledin the same run): the unverified child was reported stopped 38 to 197 s after the refusal, with eleven between 60.0 and 62.2 s and one at 7227 s. The 60 s cluster reads as the daemon tick, so these are upper bounds on how long the child lived. Six lived more than 60 s. The longer-wait change (#5770) is on hold: Codex showed a 15 s wait can outlast the attached shutdown watchdog, and the log cannot show that children exit within 15 s. Next step is to record the real overrun.janicduplessis commentedon Nov 15, 2026
Correction to my measurement above. I read the emitter:
daemon_cleanup_settledis logged fromcleanups.settle(), which stimd calls once per tick (stimd/main.ts), so the time from refusal to settled is the time to the next tick, not the child's lifetime. The values are upper bounds only. "Six lived more than 60 s" is not shown: ticks ran late on 2026-11-14 (the 7227 s pair spans 23:37 to 01:37 UTC, the long tick recorded on #5759). The earlier statement that the child "exits on its own shortly after" was never measured either. What the log does show: all 21 children were found stopped at a later tick, 15 of them within 63 s of the refusal. Whether the six longer ones (72 to 197 s, and 7227 s) were late ticks or children that lived past a tick, the log cannot say.Also from the log: the four
daemon_maintenance_failedrecords are two failed passes, each logged twice (once indaemon/passes.ts, once instimd/threads.ts), on 2026-11-13 and 11-14. The other refusals were contained without ending a maintenance pass. The installed build predates #5769, so no record names the command yet.So how long the child outlives its cancel is unknown, and that number decides between a longer wait, a caller timeout, and signalling through the handle. Next is one log record with it.
janicduplessis commentedon Nov 15, 2026
Claimed: fix/5766-report-child-overrun
janicduplessis commentedon Nov 15, 2026
#5771 is merged (286452f): the daemon now writes one
daemon_report_child_exitedrecord per refused report child when the child's own exit is seen, withcancelToExitMs,cancelReason,timeoutMsand the command. Not yet observed on a real daemon. From the code: everypscall inside a report has a 5 or 10 s timeout (one retries at 15 s). #5770 (the longer wait) stays on hold until these records show how long such a child outlives its cancel and why it was cancelled.janicduplessis commentedon Nov 16, 2026
Maintainer decision, 2026-11-15: "For rest for with ur recs", in answer to a list of open decisions each carrying a recommendation. The recommendations it accepts for this issue, as they were put to the maintainer:
Scope this sets for the fix:
reportChild.cancel, a child whose identity capture failed and whose ownexitevent has not been delivered is killed with SIGKILL through theChildProcesshandle Stim holds. Nothing is sent to the process group, and nothing is sent onceexithas been seen.#5770 is closed in favour of this.
janicduplessis commentedon Nov 16, 2026
Claimed: fix/5766-kill-uncaptured-child
janicduplessis commentedon Nov 16, 2026
The fix is merged: #5778 as 4b99fbb. On POSIX a cancelled report child whose identity was not captured is now sent SIGKILL from a check-phase callback, only while its exit has not been delivered, and never its group. The full workflow passed on Linux, macOS and Windows on the reviewed head. LLP 0007 has the rule, the libuv and Node source links, and why the signal is deferred: a reap batch can leave a handle looking live for a pid that is already free.
Left open until a real daemon shows it: the installed engine predates #5769, #5771 and this change. Once a build with them runs,
daemon_report_child_exitedrecords withkilledUncaptured: trueand the absence ofdaemon_cleanup_unverifiedforpsare the evidence to close on. Not validated before merge: the kill on Linux (same code, measured on macOS only) and EPERM from a real root-owned child (injected in a test).janicduplessis commentedon Nov 16, 2026
Read-only look at one MacBook: the engine with #5769 and #5771 (g286452f) has run 3.3 hours, with load average up to about 80, and logged no
daemon_cleanup_unverified, nodaemon_maintenance_failedand nodaemon_report_child_exited. So there is still no record naming the command. The 22 earlier refusals came at 14 to 29 per core, far above this window. The kill from #5778 is not in that engine yet.