Repository navigation
fix(supervisor): SIGTERM an app's process group on stop, SIGKILL after a 3 s grace - #40
Merged
Merged
Conversation
…r a 3 s grace Builds on killAppGroup. Every stop (daemon shutdown, uninstall, a rescan that replaces the app) still SIGKILLed the app at once, so the app never ran its shutdown path. For io.pilot.wallet that means in-flight calls are cut off between the ledger write and the spend-cap log append, the sqlite ledger is never closed, and app.sock is left on disk. cmd.Cancel is now stopAppGroup. It SIGTERMs the group the app leads (falling back to the pid), waits up to appStopGrace (3 s) for the app to exit, then SIGKILLs the group through killAppGroup. The group is only signalled while the app is unreaped, so its pid (the group id) is still held. The daemon's plugin stop budget is 5 s and apps stop in parallel. Also: - A stop now reports the app's real exit status instead of "wait: context canceled" / -1. - watchSocket ignores a socket that disappears after the stop began. The app removing it on SIGTERM is a normal shutdown, not a "socket-lost" event. Tests (zz7_stop_graceful_test.go). With the Cancel line reverted to killAppGroup: - TestStopLetsAppShutDown fails: "app never received SIGTERM", "socket ... left behind", "spawn returned -1". - TestStopKillsAppThatIgnoresSIGTERM fails: "returned after 1.558ms; expected the 3s grace". Both pass with this change. TestStopKillsAppSubprocesses still passes. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
TeoSlayer
added a commit
that referenced
this pull request
Oct 1, 2026
…RM first then SIGKILL after 3 s (#39, #40) * fix(supervisor): stopping an app stops the processes it started Daemon shutdown (StopPlugins), `pilotctl appstore uninstall` (rescan cancels the app) and a rescan that replaces an app all stop it by cancelling the context of its exec.CommandContext. exec's default Cancel SIGKILLs only the app's own pid, so every process the app had started was reparented to launchd/init and kept running with nothing left that would ever stop it (reapStale matches an app's binary and socket in argv, which a subprocess does not carry). Apps already lead their own process group (Setpgid), and the other kill paths (reapStale, watchSocket) already signal the group. The app's exec.Cmd.Cancel now does the same: SIGKILL to the whole group, falling back to the pid alone, as before, when the group cannot be signalled. Cancel runs before Wait reaps the app, so the pid, which is the group id, is still held and cannot have been reused. A subprocess that moved itself to its own group or session (setsid) is not touched. TestStopKillsAppSubprocesses starts an app whose script backgrounds a `sleep 300`, stops it, and asserts the sleep is gone. It fails on origin/main on macOS and Linux (verified) and passes with this change. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * test(reap): fake orphans never fork or re-exec startFakeApp ran `/bin/sh -c 'while :; do sleep 1; done' <argv>`. Each `sleep` the loop forks carries the shell's argv until its exec, so reapStale could count a third instance (seen in a Linux container: "reapStale = [2110 2111 2113], want both pids 2110 and 2111"), and macOS's /bin/sh shim re-execs bash, during which a scan can miss one. The fake is now this test binary (APPSTORE_TEST_FAKE_ORPHAN), blocked reading a pipe the test holds open: its argv is final once Start returns, and it never forks. Test-only. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * test(supervisor): killAppGroup falls back to the pid when there is no group Covers the fallback branch: a process that does not lead a process group is still killed, as exec's default Cancel would. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix(supervisor): SIGTERM an app's process group on stop, SIGKILL after a 3 s grace Builds on killAppGroup. Every stop (daemon shutdown, uninstall, a rescan that replaces the app) still SIGKILLed the app at once, so the app never ran its shutdown path. For io.pilot.wallet that means in-flight calls are cut off between the ledger write and the spend-cap log append, the sqlite ledger is never closed, and app.sock is left on disk. cmd.Cancel is now stopAppGroup. It SIGTERMs the group the app leads (falling back to the pid), waits up to appStopGrace (3 s) for the app to exit, then SIGKILLs the group through killAppGroup. The group is only signalled while the app is unreaped, so its pid (the group id) is still held. The daemon's plugin stop budget is 5 s and apps stop in parallel. Also: - A stop now reports the app's real exit status instead of "wait: context canceled" / -1. - watchSocket ignores a socket that disappears after the stop began. The app removing it on SIGTERM is a normal shutdown, not a "socket-lost" event. Tests (zz7_stop_graceful_test.go). With the Cancel line reverted to killAppGroup: - TestStopLetsAppShutDown fails: "app never received SIGTERM", "socket ... left behind", "spawn returned -1". - TestStopKillsAppThatIgnoresSIGTERM fails: "returned after 1.558ms; expected the 3s grace". Both pass with this change. TestStopKillsAppSubprocesses still passes. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Teodor Calin <teodor@vulturelabs.io> Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #39 (base
fix/stop-app-process-group). Merge #39 first; this PR then retargets tomain.Problem
With #39, every stop still SIGKILLs the app: daemon shutdown (
StopPlugins), uninstall, and a rescan that replaces the app. The difference is only that the SIGKILL now reaches the whole group. The app never runs its own shutdown.For
io.pilot.walletthat means:app.sockis left on disk.I drove the supervisor's real
spawn()against real wallet binaries and stopped them the way the daemon does (ctx cancel):graceful shutdown completekillAppGroup)killAppGroup)Change
cmd.Cancelis nowstopAppGroup(cmd.Process, appStopGrace):os.ErrProcessDoneand signals nothing.appStopGrace(3 s) for the app to exit.killAppGroup(fix(supervisor): stopping an app stops the processes it started #39), which SIGKILLs the whole group.The group is only signalled while the app is unreaped (
Process.Signal(0)succeeds), so its pid, which is also the group id, is still held. 3 s fits inside the daemon's 5 spluginStopTimeout(pilotprotocolcmd/daemon/shutdown.go), and apps are stopped in parallel.Two small related changes:
app=… stopped: exit status 0). Before, it reportedwait: context canceled/ -1, which also went into theexitaudit line.watchSocket. It returns when ctx is already cancelled. An app that removes its own socket on SIGTERM is shutting down normally, and should not be reported as asocket-lostevent.Upgrades still don't overlap. A replacing rescan cancels the old instance, and the new instance's spawn runs
reapStalefirst.reapStalematches the old instance's argv (same binary and socket), SIGTERMs it, and waits up to its own 3 s grace before the new one binds the socket.Tests (
zz7_stop_graceful_test.go)TestStopLetsAppShutDown: the app traps SIGTERM, records it, removes its socket and exits 0. The stop must deliver SIGTERM, finish well under the grace period, remove the socket, return 0, and take the app's subprocess down too.killAppGroupit fails:app never received SIGTERM,socket … left behind,spawn returned -1.TestStopKillsAppThatIgnoresSIGTERM: the app and its subprocess both ignore SIGTERM. Both must be SIGKILLed once the grace runs out.returned after 1.558ms; expected the 3s grace.TestStopAppGroupAlreadyExited: an already-reaped process givesos.ErrProcessDoneand nothing is signalled.TestStopKillsAppSubprocessesandTestKillAppGroupFallsBackToThePidstill pass.Where the tests ran:
go test -race ./...passes. The stop, reap and watchSocket tests also pass with-count=3.golang:1.25.13-bookworm, linux/arm64 and linux/amd64, each with and without--init:go test -race ./...passes.Known limit
A subprocess that ignores SIGTERM survives if its group leader exits within the grace period. Once the leader is reaped, its group id is no longer safe to signal, so this PR does not signal it.
🤖 Generated with Claude Code