Skip to content

Maintenance and gc fail under load: cleanup of a setuid report child cannot be verified #5766

Open

Description

@janicduplessis

Problem

Maintenance passes and stim gc --dry-run fail with STIM_CLAIM_UNAVAILABLE because the cleanup of a "report child" cannot be verified:

STIM_CLAIM_UNAVAILABLE: Cannot verify cleanup for report child 99108 (its identity was not captured at spawn: ACCESS_DENIED (Process inspection was denied)); its process or descendants were left untouched.
Check whether the report child 99108 (`ps -p 99108`) and its descendants are still running and stop them if they are, then run `stim gc --dry-run` again.

The child was already gone when checked (ps -p 99108 printed nothing). In the daemon the same error ends the maintenance pass, so cleanup does not run on that tick. It happens under load, which is when cleanup is needed.

Evidence

  • Stim 2.0.0-next.20261115053519.g626efc0, macOS, one MacBook with many agents building.
  • stim gc --dry-run on 2026-11-15 exited 1 with the text above after "checking 156 projects for merged or idle worktrees".
  • ~/.stim/daemon/daemon.ndjson has 22 records with "its identity was not captured at spawn: ACCESS_DENIED" between 2026-11-11 and 2026-11-15: 1 on the 11th, 4 on the 13th, 11 on the 14th, 6 on the 15th. 18 are daemon_cleanup_unverified ("stimd goes on and stays up until the child has stopped") and 4 are daemon_maintenance_failed.
  • On 2026-11-14 two of the failed passes fall inside the window in which the disk went from 34 GB to 0.2 GB free (Under load, the Cargo target trim at high and critical waits hours on the deferred size scan #5759).
  • /bin/ps on macOS is setuid root (-rwsr-xr-x root). /usr/sbin/lsof is not.

Cause

Not confirmed; this is an inference from the code and the evidence.

packages/stim-cli/src/exec.ts (around 205-250): when a report child is cancelled or times out, cancel observes it. With a captured identity it uses inspectProcessIdentity. Without one it falls back to process.kill(-pid, 0) and treats only ESRCH as gone; anything else is unknown, and after one second of unknown it refuses with the message above.

A child that is ps runs as root once it has exec'd, because the binary is setuid. From an unprivileged process, inspecting it is denied, so an identity capture that happens after the exec fails with ACCESS_DENIED, and kill(-pid, 0) on its group returns EPERM, not ESRCH. Under load ps is slow enough to hit its timeout, so the cancel path runs with no identity and an answer that can never become "gone" while the child lives, and may stay "unknown" for a moment after it exits. That would explain why it appears under load and why the child is gone by the time a person checks.

Not checked: which report children fail (the record does not name the command), whether the capture races the exec or always fails for ps, and what kill(-pid, 0) returns in each state.

Fix idea

  • Record the command of the report child in the refusal and the log record, so the failing tool is known.
  • Capture the child's identity before it can change credentials, or treat a child that is a known setuid report tool differently: for example verify it through its own exit (the close event of the spawned process is authoritative for the child itself) and through its process group for descendants.
  • Treat EPERM from kill(-pid, 0) as "a process exists that this user may not signal", and decide what that means for a report child that Stim itself spawned and has seen exit.

Whatever is chosen has to keep LLP 0007: a state Stim cannot establish is a refusal, never a silent pass. The question is whether "the child I spawned has exited and its group is empty" can be established here without process inspection.

Related: #5491, #4155, #5759.

Activity

  1. added
    type: bugBreaks documented or expected behavior
    priority: P1Next up: user-facing defect or unblocks other work
    area: clipackages/stim-cli, including its README and guide
    on Nov 15, 2026
  2. janicduplessis commented on Nov 15, 2026

    @janicduplessis
    CollaboratorAuthor

    Claimed: fix/5766-report-child-command

  3. janicduplessis commented on Nov 15, 2026

    @janicduplessis
    CollaboratorAuthor

    Diagnosis (2026-11-15)

    Measured on one MacBook, macOS 27, with real spawned children. Not measured on Linux or Windows. The "Cause" section above was partly wrong; this replaces it.

    What holds

    • Identity capture fails with ACCESS_DENIED for a setuid child for its whole life, not as a race: /bin/ps in 200 of 200 runs, /usr/bin/top for its whole run. xcrun, lsof, du, git, security, system_profiler and sleep capture normally.
    • The refusal reproduces with the real executor: top -l 3 with timeoutMs: 300 inside runReport refuses after about 1.35 s. sleep with the same timeout is killed normally.

    What was wrong

    • process.kill(pid, 0) and process.kill(-pid, 0) return success for a running setuid child that the same user spawned, not EPERM. Checked again separately with top -l 2 -s 1.
    • Nothing stays unknown after the child exits: both read ESRCH at the exit event, at close and afterwards. There is no window in which an exited child still looks alive.

    Cause as established

    A policy path in reportChild (packages/stim-cli/src/exec.ts). When a report is cancelled (timeout, budget or abort) while an uncaptured child is still alive, cancel() never signals it, observe() maps "group alive" to unknown, and after a fixed 1 s of unknown it refuses. Under load a ps outlives its deadline by more than 1 s and exits on its own shortly after, which is why it is gone when a person checks.

    Still unknown

    Fix options

    1. Wait longer for the child's own exit (no new signalling). For an uncaptured child, wait for its exit/close for the existing cleanup budget (groupExitBudgetMs(REPORT_CHILD_EXIT_CAP_MS), capped at 15 s) instead of 1 s, then apply the existing group check and refuse as today if it is still alive. Covers the observed case, where the child exits by itself. A ps that hangs past the budget still ends the pass. A cancelled report waits up to the budget instead of failing after 1 s.
    2. Signal through the held handle. Treat the unreaped ChildProcess handle as the identity proof and SIGKILL the child (and, before exit is seen, its group, since it was spawned detached). Covers a hung child too. This is a new signalling rule for a process with no captured identity, so it needs a decision under LLP 0007 and probably an amendment.

    Going ahead with option 1 after #5769 merges: it changes a wait, not what is signalled or when a state counts as established. Option 2 waits on these decisions:

    • Is the held handle of an unreaped direct child an identity proof for SIGKILL under LLP 0007?
    • May the group be signalled, or only the child? A root-owned setuid descendant can return EPERM.
    • After the leader's exit with live descendants and no record: keep refusing (proposed), or accept the leader's exit for known report tools?
    • Should a report-only gc or maintenance pass degrade to a warning when one report child cannot be verified, instead of ending the pass?
  4. janicduplessis commented on Nov 15, 2026

    @janicduplessis
    CollaboratorAuthor

    Claimed: fix/5766-uncaptured-child-wait

  5. janicduplessis commented on Nov 15, 2026

    @janicduplessis
    CollaboratorAuthor

    Measured from the real daemon log (21 pairs of daemon_cleanup_unverified and the following daemon_cleanup_settled in the same run): the unverified child was reported stopped 38 to 197 s after the refusal, with eleven between 60.0 and 62.2 s and one at 7227 s. The 60 s cluster reads as the daemon tick, so these are upper bounds on how long the child lived. Six lived more than 60 s. The longer-wait change (#5770) is on hold: Codex showed a 15 s wait can outlast the attached shutdown watchdog, and the log cannot show that children exit within 15 s. Next step is to record the real overrun.

  6. janicduplessis commented on Nov 15, 2026

    @janicduplessis
    CollaboratorAuthor

    Correction to my measurement above. I read the emitter: daemon_cleanup_settled is logged from cleanups.settle(), which stimd calls once per tick (stimd/main.ts), so the time from refusal to settled is the time to the next tick, not the child's lifetime. The values are upper bounds only. "Six lived more than 60 s" is not shown: ticks ran late on 2026-11-14 (the 7227 s pair spans 23:37 to 01:37 UTC, the long tick recorded on #5759). The earlier statement that the child "exits on its own shortly after" was never measured either. What the log does show: all 21 children were found stopped at a later tick, 15 of them within 63 s of the refusal. Whether the six longer ones (72 to 197 s, and 7227 s) were late ticks or children that lived past a tick, the log cannot say.

    Also from the log: the four daemon_maintenance_failed records are two failed passes, each logged twice (once in daemon/passes.ts, once in stimd/threads.ts), on 2026-11-13 and 11-14. The other refusals were contained without ending a maintenance pass. The installed build predates #5769, so no record names the command yet.

    So how long the child outlives its cancel is unknown, and that number decides between a longer wait, a caller timeout, and signalling through the handle. Next is one log record with it.

  7. janicduplessis commented on Nov 15, 2026

    @janicduplessis
    CollaboratorAuthor

    Claimed: fix/5766-report-child-overrun

  8. janicduplessis commented on Nov 15, 2026

    @janicduplessis
    CollaboratorAuthor

    #5771 is merged (286452f): the daemon now writes one daemon_report_child_exited record per refused report child when the child's own exit is seen, with cancelToExitMs, cancelReason, timeoutMs and the command. Not yet observed on a real daemon. From the code: every ps call inside a report has a 5 or 10 s timeout (one retries at 15 s). #5770 (the longer wait) stays on hold until these records show how long such a child outlives its cancel and why it was cancelled.

  9. janicduplessis commented on Nov 16, 2026

    @janicduplessis
    CollaboratorAuthor

    Maintainer decision, 2026-11-15: "For rest for with ur recs", in answer to a list of open decisions each carrying a recommendation. The recommendations it accepts for this issue, as they were put to the maintainer:

    • "May Stim kill a report child whose identity it could not capture, through the handle it holds? Yes, the direct child only, and only before its exit event. The pid cannot have been reused while Stim has not reaped it."
    • "Signal its whole process group too? No. The proof covers only the child."
    • "Child exited but descendants are alive and unrecorded: keep refusing."
    • "Downgrade the refusal to a warning for report-only passes? No. B1 removes the case; revisit only if the new log records show it still happens."
    • "The longer-wait PR fix: wait for an uncaptured report child to exit before refusing #5770: close it if you accept B1. It can outlast the shutdown watchdog and B1 replaces it."

    Scope this sets for the fix:

    • In reportChild.cancel, a child whose identity capture failed and whose own exit event has not been delivered is killed with SIGKILL through the ChildProcess handle Stim holds. Nothing is sent to the process group, and nothing is sent once exit has been seen.
    • Every later step is unchanged: the group check, the "still has descendants" refusal, the refusal when a state cannot be established.
    • LLP 0007 gains the rule and the quote above in the same pull request.
    • Windows and Linux behaviour must be stated from the code, not assumed from the macOS measurements on this issue.

    #5770 is closed in favour of this.

  10. janicduplessis commented on Nov 16, 2026

    @janicduplessis
    CollaboratorAuthor

    Claimed: fix/5766-kill-uncaptured-child

  11. janicduplessis commented on Nov 16, 2026

    @janicduplessis
    CollaboratorAuthor

    The fix is merged: #5778 as 4b99fbb. On POSIX a cancelled report child whose identity was not captured is now sent SIGKILL from a check-phase callback, only while its exit has not been delivered, and never its group. The full workflow passed on Linux, macOS and Windows on the reviewed head. LLP 0007 has the rule, the libuv and Node source links, and why the signal is deferred: a reap batch can leave a handle looking live for a pid that is already free.

    Left open until a real daemon shows it: the installed engine predates #5769, #5771 and this change. Once a build with them runs, daemon_report_child_exited records with killedUncaptured: true and the absence of daemon_cleanup_unverified for ps are the evidence to close on. Not validated before merge: the kill on Linux (same code, measured on macOS only) and EPERM from a real root-owned child (injected in a test).

  12. janicduplessis commented on Nov 16, 2026

    @janicduplessis
    CollaboratorAuthor

    Read-only look at one MacBook: the engine with #5769 and #5771 (g286452f) has run 3.3 hours, with load average up to about 80, and logged no daemon_cleanup_unverified, no daemon_maintenance_failed and no daemon_report_child_exited. So there is still no record naming the command. The 22 earlier refusals came at 14 to 29 per core, far above this window. The kill from #5778 is not in that engine yet.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area: clipackages/stim-cli, including its README and guidepriority: P1Next up: user-facing defect or unblocks other worktype: bugBreaks documented or expected behavior

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions