Skip to content

fix(scaffold): a timed-out cli call gets SIGTERM before SIGKILL - #117

Draft
TeoSlayer wants to merge 1 commit into
mainfrom
fix/aegis-lifecycle
Draft

TeoSlayer wants to merge 1 commit into
mainfrom
fix/aegis-lifecycle

Conversation

@TeoSlayer

@TeoSlayer TeoSlayer commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

Update 2026-09-24: this change is now in both #114 and #116. Only the test is new. #116's head (a0d7665, pushed after this PR) is stacked on #111 and #114's commits 79c4909 and 808243f, and 79c4909 carries this exact cmd.Cancel line. Checked with git show <ref>:internal/scaffold/templates/client_cli.go.tmpl | grep -c 'cmd.Cancel = func': main 0, #111 0, #114 1, #116 1.

This PR's test, zz_cli_cancel_sigterm_e2e_test.go, copied onto #116's head, passes there: --- PASS: TestCLICancelSendsSIGTERMFirstE2E (8.04s). graceful ends at 1.005 s; stubborn gets SIGKILLed at 6.005 s. None of #111, #114 or #116 has a test that covers the method-timeout path: they cover SIGTERM of the adapter mid-call, not a call's context ending.

Suggestion: close this once #116 (or #114) merges, and land the test file on its own as a follow-up if you want that path covered. This PR is not needed for the fix.

Problem

The cli Runner (internal/scaffold/templates/client_cli.go.tmpl) used exec.CommandContext's default cancel. When a call's context ended, the CLI's pid got an immediate SIGKILL. The CLI never got a chance to stop what it had started itself, so those children ran on, reparented to init.

Measured on the published io.pilot.aegis 0.1.3 bundles with aegis.exec {"args":["install-models"]} and a slow curl:

  • Method timeout (60 s). The curl was still running afterwards with ppid=1: darwin-arm64 pid 21800, linux-arm64 pid 30, linux-amd64 pid 39. The call took 65,004 ms, because the orphan held the stderr pipe until WaitDelay ran out.
  • App SIGTERMed with the call in flight. Same result: on darwin/arm64 the curl was left as sleep 600 with ppid=1, 2 of 2 runs.

Fix

cmd.Cancel now sends SIGTERM. The existing WaitDelay (killGrace, 5 s) still escalates to SIGKILL, so a CLI that ignores SIGTERM costs at most 5 s more than before and still can't hang a call. docs/CLI-ADAPTER.md says so.

On macOS this change is required. There is no parent-death signal there, so once the CLI is SIGKILLed, nothing can stop its children. I measured an adapter built from the unpushed fix/miren-lifecycle change (0467324, which waits for running calls on shutdown and sets Pdeathsig on Linux) without this commit, with the patched aegis. SIGTERM mid-call still left the curl with ppid=1 on darwin/arm64 (3/3) and darwin/amd64 (3/3), and the adapter took 5 s to exit. With this commit added: 0 left, and the adapter exits in about 50 ms.

Test

TestCLICancelSendsSIGTERMFirstE2E builds a real cli adapter over a shell script with a 1 s method timeout:

Subtest The script main this branch
graceful traps SIGTERM and kills its own sleep 600 child fails: no SIGTERM, and the child outlives the call passes: SIGTERM seen, child gone, call ends at 1.0 s
stubborn ignores SIGTERM fails: SIGKILL at 1.0 s passes: still SIGKILLed, about 6.0 s after the call started

go vet ./... and go test ./... -count=1 pass on darwin/arm64.

Verified on io.pilot.aegis 0.1.5 (all 4 platforms)

Bundles were built with publish.BuildBundle and a TEST-ONLY key. Nothing was uploaded or signed with a real key. The template was #111 + 0467324 + this commit, and the natives came from pilot-protocol/aegis#4. The lifecycle harness drove the app: darwin/amd64 under Rosetta, Linux in docker --init.

  • install-models past the 60 s timeout: 0 processes left on all four. The call ended at 60,001–60,031 ms, with no WaitDelay overrun.
  • SIGTERM with the call in flight: 3/3 clean on all four.
  • A flagged scan's judge server: 0 left after both SIGTERM and SIGKILL of the app.
  • help, version, status and exec scan-cmd, 200 calls each: 200/200 ok, fd flat.

This branch alone on main (with aegis#4), darwin/arm64: the in-flight SIGTERM case is also 3/3 clean, and the timeout leaves nothing behind.

Aegis 0.1.5 built from each candidate template

Harness measurements, same natives (aegis#4) and TEST-ONLY key. "In-flight" means the app was SIGTERMed while aegis.exec install-models was running, 3 rounds each.

Template darwin/arm64 darwin/amd64 linux/arm64 linux/amd64
#116 alone (0467324) curl left with ppid=1, 3/3; adapter exit 5 s curl left, 3/3
#116 + this commit 0 left, 3/3 0 left, 3/3 0 left, 3/3 0 left, 3/3
#114 head (808243f) 0 left, 3/3 0 left, 3/3 0 left, 3/3 0 left, 3/3

With both #116 + this commit and #114, the other checks passed on all four platforms:

  • the judge after a flagged scan: 0 left after SIGTERM and after SIGKILL;
  • install-models past the 60 s timeout: IPC error, 0 left;
  • orphan without Pdeathsig: SELF-EXIT in 207–319 ms.

Where the two differ:

#116 + this commit #114 head
offline first spawn help 20/20 on all four exits 1 before the socket appears on all four
darwin/arm64 aegis.status ×200, p50 / p95 6.2 / 8.5 ms 41.8 / 296.9 ms, with one guard process per call

Related, not in this PR

🤖 Generated with Claude Code

The cli Runner used exec.CommandContext's default cancel, an immediate
SIGKILL to the CLI's pid. The CLI never got to stop what it had started
itself, so its children ran on, reparented to init. Measured on the
published io.pilot.aegis 0.1.3 bundles: `aegis.exec install-models` with a
slow curl hit the 60 s method timeout, and the curl was still running with
ppid=1 afterwards (darwin/arm64 pid 21800, linux/arm64 pid 30,
linux/amd64 pid 39). The call also took 65 s, because the orphan held the
stderr pipe until WaitDelay ran out.

cmd.Cancel now sends SIGTERM. exec's existing WaitDelay (killGrace, 5 s)
still escalates to SIGKILL, so a CLI that ignores SIGTERM costs at most
5 s more than before and can't hang a call.

TestCLICancelSendsSIGTERMFirstE2E builds a real cli adapter over a script
with a 1 s method timeout:
- graceful: the script traps SIGTERM and kills its own child. On main the
  trap never runs and the child outlives the call; now both happen.
- stubborn: the script ignores SIGTERM. It is still SIGKILLed, about 6 s
  after the call started (1 s timeout + 5 s killGrace).
Both subtests fail on main and pass here (darwin/arm64, 3 runs).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants