Skip to content

fix(supervisor): SIGTERM an app's process group on stop, SIGKILL after a 3 s grace - #40

Merged
TeoSlayer merged 1 commit into
fix/stop-app-process-groupfrom
fix/stop-app-graceful
Oct 1, 2026
Merged

TeoSlayer merged 1 commit into
fix/stop-app-process-groupfrom
fix/stop-app-graceful

Conversation

@TeoSlayer

Copy link
Copy Markdown
Contributor

Stacked on #39 (base fix/stop-app-process-group). Merge #39 first; this PR then retargets to main.

Problem

With #39, every stop still SIGKILLs the app: daemon shutdown (StopPlugins), uninstall, and a rescan that replaces the app. The difference is only that the SIGKILL now reaches the whole group. The app never runs its own shutdown.

For io.pilot.wallet that means:

  • In-flight calls are cut between the ledger write and the spend-cap log append.
  • The sqlite ledger is never closed.
  • app.sock is left on disk.

I drove the supervisor's real spawn() against real wallet binaries and stopped them the way the daemon does (ctx cancel):

supervisor wallet exit stop took app.sock wallet logged graceful shutdown complete
#39 (killAppGroup) catalogue 0.3.3 -1 1.9 ms left no
#39 (killAppGroup) 0.4.0 (pilot-protocol/wallet#40) -1 2.4 ms left no
this PR catalogue 0.3.3 0 20.9 ms removed yes
this PR 0.4.0 (pilot-protocol/wallet#40) 0 20.5 ms removed yes

Change

cmd.Cancel is now stopAppGroup(cmd.Process, appStopGrace):

  1. If the app was already reaped (exec can call Cancel after Wait), it returns os.ErrProcessDone and signals nothing.
  2. It SIGTERMs the process group the app leads, falling back to the pid.
  3. It polls every 20 ms for up to appStopGrace (3 s) for the app to exit.
  4. If the app is still running after that, it calls killAppGroup (fix(supervisor): stopping an app stops the processes it started #39), which SIGKILLs the whole group.

The group is only signalled while the app is unreaped (Process.Signal(0) succeeds), so its pid, which is also the group id, is still held. 3 s fits inside the daemon's 5 s pluginStopTimeout (pilotprotocol cmd/daemon/shutdown.go), and apps are stopped in parallel.

Two small related changes:

  • Exit status. A stop now reports the app's real exit status (app=… stopped: exit status 0). Before, it reported wait: context canceled / -1, which also went into the exit audit line.
  • watchSocket. It returns when ctx is already cancelled. An app that removes its own socket on SIGTERM is shutting down normally, and should not be reported as a socket-lost event.

Upgrades still don't overlap. A replacing rescan cancels the old instance, and the new instance's spawn runs reapStale first. reapStale matches the old instance's argv (same binary and socket), SIGTERMs it, and waits up to its own 3 s grace before the new one binds the socket.

Tests (zz7_stop_graceful_test.go)

  • TestStopLetsAppShutDown: the app traps SIGTERM, records it, removes its socket and exits 0. The stop must deliver SIGTERM, finish well under the grace period, remove the socket, return 0, and take the app's subprocess down too.
  • TestStopKillsAppThatIgnoresSIGTERM: the app and its subprocess both ignore SIGTERM. Both must be SIGKILLed once the grace runs out.
    • Without this PR it fails: returned after 1.558ms; expected the 3s grace.
  • TestStopAppGroupAlreadyExited: an already-reaped process gives os.ErrProcessDone and nothing is signalled.
  • fix(supervisor): stopping an app stops the processes it started #39's TestStopKillsAppSubprocesses and TestKillAppGroupFallsBackToThePid still pass.

Where the tests ran:

  • macOS arm64: go test -race ./... passes. The stop, reap and watchSocket tests also pass with -count=3.
  • golang:1.25.13-bookworm, linux/arm64 and linux/amd64, each with and without --init: go test -race ./... passes.

Known limit

A subprocess that ignores SIGTERM survives if its group leader exits within the grace period. Once the leader is reaped, its group id is no longer safe to signal, so this PR does not signal it.

🤖 Generated with Claude Code

…r a 3 s grace

Builds on killAppGroup. Every stop (daemon shutdown, uninstall, a rescan
that replaces the app) still SIGKILLed the app at once, so the app never
ran its shutdown path. For io.pilot.wallet that means in-flight calls are
cut off between the ledger write and the spend-cap log append, the sqlite
ledger is never closed, and app.sock is left on disk.

cmd.Cancel is now stopAppGroup. It SIGTERMs the group the app leads
(falling back to the pid), waits up to appStopGrace (3 s) for the app to
exit, then SIGKILLs the group through killAppGroup. The group is only
signalled while the app is unreaped, so its pid (the group id) is still
held. The daemon's plugin stop budget is 5 s and apps stop in parallel.

Also:
- A stop now reports the app's real exit status instead of "wait:
  context canceled" / -1.
- watchSocket ignores a socket that disappears after the stop began. The
  app removing it on SIGTERM is a normal shutdown, not a "socket-lost"
  event.

Tests (zz7_stop_graceful_test.go). With the Cancel line reverted to
killAppGroup:
- TestStopLetsAppShutDown fails: "app never received SIGTERM", "socket
  ... left behind", "spawn returned -1".
- TestStopKillsAppThatIgnoresSIGTERM fails: "returned after 1.558ms;
  expected the 3s grace".
Both pass with this change. TestStopKillsAppSubprocesses still passes.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@TeoSlayer
TeoSlayer merged commit 30d5632 into fix/stop-app-process-group Oct 1, 2026
3 checks passed
TeoSlayer added a commit that referenced this pull request Oct 1, 2026
…RM first then SIGKILL after 3 s (#39, #40)

* fix(supervisor): stopping an app stops the processes it started

Daemon shutdown (StopPlugins), `pilotctl appstore uninstall` (rescan
cancels the app) and a rescan that replaces an app all stop it by
cancelling the context of its exec.CommandContext. exec's default Cancel
SIGKILLs only the app's own pid, so every process the app had started
was reparented to launchd/init and kept running with nothing left that
would ever stop it (reapStale matches an app's binary and socket in
argv, which a subprocess does not carry).

Apps already lead their own process group (Setpgid), and the other kill
paths (reapStale, watchSocket) already signal the group. The app's
exec.Cmd.Cancel now does the same: SIGKILL to the whole group, falling
back to the pid alone, as before, when the group cannot be signalled.
Cancel runs before Wait reaps the app, so the pid, which is the group
id, is still held and cannot have been reused. A subprocess that moved
itself to its own group or session (setsid) is not touched.

TestStopKillsAppSubprocesses starts an app whose script backgrounds a
`sleep 300`, stops it, and asserts the sleep is gone. It fails on
origin/main on macOS and Linux (verified) and passes with this change.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* test(reap): fake orphans never fork or re-exec

startFakeApp ran `/bin/sh -c 'while :; do sleep 1; done' <argv>`. Each
`sleep` the loop forks carries the shell's argv until its exec, so
reapStale could count a third instance (seen in a Linux container:
"reapStale = [2110 2111 2113], want both pids 2110 and 2111"), and
macOS's /bin/sh shim re-execs bash, during which a scan can miss one.

The fake is now this test binary (APPSTORE_TEST_FAKE_ORPHAN), blocked
reading a pipe the test holds open: its argv is final once Start
returns, and it never forks. Test-only.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* test(supervisor): killAppGroup falls back to the pid when there is no group

Covers the fallback branch: a process that does not lead a process group
is still killed, as exec's default Cancel would.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix(supervisor): SIGTERM an app's process group on stop, SIGKILL after a 3 s grace

Builds on killAppGroup. Every stop (daemon shutdown, uninstall, a rescan
that replaces the app) still SIGKILLed the app at once, so the app never
ran its shutdown path. For io.pilot.wallet that means in-flight calls are
cut off between the ledger write and the spend-cap log append, the sqlite
ledger is never closed, and app.sock is left on disk.

cmd.Cancel is now stopAppGroup. It SIGTERMs the group the app leads
(falling back to the pid), waits up to appStopGrace (3 s) for the app to
exit, then SIGKILLs the group through killAppGroup. The group is only
signalled while the app is unreaped, so its pid (the group id) is still
held. The daemon's plugin stop budget is 5 s and apps stop in parallel.

Also:
- A stop now reports the app's real exit status instead of "wait:
  context canceled" / -1.
- watchSocket ignores a socket that disappears after the stop began. The
  app removing it on SIGTERM is a normal shutdown, not a "socket-lost"
  event.

Tests (zz7_stop_graceful_test.go). With the Cancel line reverted to
killAppGroup:
- TestStopLetsAppShutDown fails: "app never received SIGTERM", "socket
  ... left behind", "spawn returned -1".
- TestStopKillsAppThatIgnoresSIGTERM fails: "returned after 1.558ms;
  expected the 3s grace".
Both pass with this change. TestStopKillsAppSubprocesses still passes.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Teodor Calin <teodor@vulturelabs.io>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants