Skip to content

Stop the judge server and model download when aegis exits - #4

Merged
TeoSlayer merged 2 commits into
mainfrom
fix/aegis-lifecycle
Oct 1, 2026
Merged

TeoSlayer merged 2 commits into
mainfrom
fix/aegis-lifecycle

Conversation

@TeoSlayer

Copy link
Copy Markdown
Contributor

Problem

aegis starts two long-lived children: the L2 judge (llama-server, on the first judged scan) and curl for install-models (a ~1.8 GB download). Both outlived aegis on every exit path that skips destructors. They were reparented to init and kept running. Measured on the published v0.1.3 natives through the Pilot app (io.pilot.aegis) on darwin/arm64, linux/arm64 and linux/amd64:

Exit path What was left running
Flagged scan: run_scan ends with process::exit(1), which skips IntentJudge::drop The judge server, with ppid=1, still holding the model and its port. It also survived the app's SIGTERM and SIGKILL (darwin-arm64 pid=18840 ppid=1, linux-arm64 pid 38, linux-amd64 pid 45).
SIGTERM / SIGINT / SIGHUP with the default action (a caller's timeout, a supervisor stop, Ctrl-C, a closed terminal) The install-models curl, with ppid=1 (darwin-arm64 pid=21800 ppid=1). A retry started a second curl on the same file.
status, targets, scan with no model installed Each call ran llama-server --version: about 226 ms p50 per status call. Once, 2 of 3 calls hung for more than 15 s, and the stop then left llama-server with ppid=1.
aegis daemon desktop notifications The notifier was never waited for, so the daemon kept one zombie per notification.

main (v0.1.4) has the same code paths. The new tests below fail on it.

Fix

  • children::spawn records the pid of every child aegis starts. An atexit hook, which runs on process::exit and on a normal return from main, sends SIGTERM to every recorded child. So does a SIGTERM/SIGINT/SIGHUP handler, and aegis then dies of the same signal, so callers see the same exit status as before.
    • A signal that is already ignored stays ignored (nohup aegis daemon, aegis daemon & from a script).
    • On Linux each child also gets PR_SET_PDEATHSIG(SIGKILL), which covers SIGKILL of aegis. There is a getppid check for aegis dying between fork and prctl.
    • The stop signals are blocked around fork+record, so a signal can't miss a child that was just forked. They are unblocked again in the child before exec. Without that, the child inherits the blocked mask through the macOS posix_spawn path and can't be stopped with SIGTERM at all. The tests caught this.
  • A judge started by another aegis process is left alone. aegis finds such a server through ~/.aegis/judge.state and never records it, so it keeps running for its owner, as before.
  • The llama-server check is now a PATH lookup, not an exec. With no model installed, aegis also skips judge.state and the health probe.
  • notify() reaps the notifier.
  • Bump version to 0.1.5 is a separate commit, so a v0.1.5 tag on the merge builds natives that report 0.1.5. Drop it if you'd rather version the release separately.

Tests

tests/child_lifecycle.rs runs the real binary with HOME and PATH pointed at a scratch dir that holds a stub llama-server and a stub curl. The stub judge is the test binary re-executed as a tiny HTTP server that answers ATTACK, so the tests need only /bin/sh and sleep. The tests cover:

  • a flagged scan;
  • SIGTERM, SIGINT and SIGHUP during install-models;
  • an ignored SIGHUP;
  • SIGKILL, Linux only;
  • no llama-server process when no model is installed;
  • a shared judge that another process started, which must keep running.
main 27fb1dd this branch
darwin/arm64 4 of 6 fail 6/6 pass, 3 runs
linux/arm64 (static musl build, docker --init and without init) 5 of 7 fail 7/7 pass, 3 runs each
linux/amd64 (same) 7/7 pass, 3 runs each

The tests that pass on main are the stub helper and the shared-judge guard. CI now runs cargo test --release --test child_lifecycle without || true.

Verified through the Pilot app (all 4 platforms)

The natives were built from this branch: cargo build for the two macOS targets, cargo zigbuild for musl on Linux, with --remap-path-prefix so no local paths are embedded. They ran inside io.pilot.aegis 0.1.5 adapter bundles built from app-template with the cli lifecycle fixes, pilot-protocol/app-template#111 plus the SIGTERM-first cancel. The adapter staged the natives over HTTP from a local mirror, and the lifecycle harness drove the app. darwin/amd64 ran under Rosetta; Linux ran in docker --init.

darwin/arm64 darwin/amd64 linux/arm64 linux/amd64
judge after a flagged scan (×5), then SIGTERM / SIGKILL of the app 0 left 0 left 0 left 0 left
install-models past the 60 s method timeout 0 left, 60,001 ms 0 left, 60,007 ms 0 left, 60,002 ms 0 left, 60,031 ms
app SIGTERMed with install-models in flight 3/3 clean 3/3 clean 3/3 clean 3/3 clean
aegis.status ×200 p50 6.2 ms 23 ms (Rosetta) 5.5 ms 42 ms (emulated)
aegis.version / aegis.exec scan-cmd ×200 200/200, fd 7→7 200/200, fd 7→7 200/200 200/200

With the published 0.1.3 native, the in-flight SIGTERM case still leaves the curl behind on darwin/arm64: sleep 600 with ppid=1, 2 of 2 runs.

Not in this PR

  • Judge authentication. The judge server is trusted if /health answers on the port recorded in judge.state (0600). v0.1.4 already fixed the fixed-port :8849 issue. The remaining gap: a process that later binds that recorded port could still answer as the judge.
  • The in-repo Pilot adapter (integrations/pilot/) is a different adapter from the one in the catalogue. The catalogue ships app-template's generated adapter.

🤖 Generated with Claude Code

teovl and others added 2 commits September 24, 2026 13:22
aegis starts two long-lived children: the L2 judge (llama-server, on the
first judged scan) and curl for `install-models` (a ~1.8 GB download).
Both outlived aegis on every exit path that skips destructors, reparented
to init and still running. Measured on the published v0.1.3 natives
(darwin/arm64, linux/arm64, linux/amd64) through the Pilot app:

- A flagged scan ends with process::exit(1), which skipped
  IntentJudge::drop. The judge server stayed up with ppid=1, holding the
  model and its port, and survived the app's SIGTERM and SIGKILL.
- A stop signal (the app's call timeout, a supervisor stop, Ctrl-C, a
  closed terminal) killed aegis with the default action. install-models
  left curl downloading with ppid=1; a retry started a second curl on the
  same file.
- status, targets and scan ran `llama-server --version` on every call,
  even with no model installed: about 0.2 s per call, and once a hang of
  over 15 s whose llama-server was orphaned when the caller gave up.
- The daemon's desktop notifications were never waited for, so it kept
  one zombie per notification.

Changes:

- children::spawn records the pid of every child aegis starts. An atexit
  hook and a SIGTERM/SIGINT/SIGHUP handler SIGTERM the recorded set, then
  aegis dies of the same signal as before. A signal that is already
  ignored (nohup, `aegis daemon &`) stays ignored. On Linux each child
  also gets PR_SET_PDEATHSIG(SIGKILL), which covers SIGKILL of aegis.
  Stop signals are blocked around fork+record, and unblocked again in the
  child before exec (macOS's posix_spawn path would otherwise pass the
  mask on, and the child could not be stopped with SIGTERM).
- A judge server another aegis started (found through judge.state) is
  never recorded, so it keeps running for its owner, as before.
- The llama-server check is a PATH lookup instead of an exec. With no
  model, aegis also skips judge.state and the health probe.
- notify() reaps the notifier.

tests/child_lifecycle.rs runs the real binary against stub llama-server
and curl: a flagged scan, SIGTERM/SIGINT/SIGHUP during install-models, an
ignored SIGHUP, SIGKILL on Linux, no llama-server process without a model,
and a shared judge left running. On main 4 of 6 fail on macOS and 5 of 7
on Linux (the other two are the stub helper and the shared-judge guard);
with this change all pass on darwin/arm64, linux/arm64 and linux/amd64.
CI runs them without `|| true`.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
So a v0.1.5 tag on this commit builds natives that report 0.1.5, and the
Pilot app (io.pilot.aegis) can ship them as 0.1.5.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants