fix(darwin): restart the daemon only under its launchd job, confirm over IPC; restart the updater job

[TeoSlayer]

Problem

On a Mac, pilotctl update (v1.13.9) installed v1.13.10: pilot-daemon, pilotctl and pilot-updater were replaced, with the SLSA and checksum checks passing. But the launchd daemon (network.pilotprotocol.pilot-daemon, KeepAlive SuccessfulExit=false) kept running v1.13.9 until someone ran launchctl kickstart -k by hand. The updater service (network.pilotprotocol.pilot-updater) also kept running its old binary.

Root cause, in two parts:

1. v1.13.9 linked updater v0.2.4. When a release replaced pilot-updater (every release does), v0.2.4's applyUpdate called exitFn(0), which is os.Exit(0). It did this before reaching signalDaemonRestart, so pilotctl update exited silently and never ran launchctl kickstart. v0.2.5 (fix: visible update failures, resumable downloads, systemd-aware Linux restart #49) fixed the exit: RunOnce no longer exits.
2. v0.2.5 still restarted blindly on macOS. It ran launchctl kickstart -k gui/<uid>/network.pilotprotocol.pilot-daemon and reported success as soon as launchctl returned. It did not check that launchd runs this daemon or that the job's program is the replaced binary, and it did not confirm the daemon came back on the new release. It also never moved the updater service onto its new binary. restart_error was also kept after a manual run that did restart the daemon, because the release replaced pilot-updater. On macOS it was never cleared by a later check.

Change (macOS; the Linux path is unchanged apart from the recording fix)

After replacing binaries on darwin, signalDaemonRestartDarwin now does the following (new file launchd.go):

1. Finds the job with launchctl print in gui/<uid>, then user/<uid>. It also finds the legacy com.vulturelabs.pilot-daemon label, and under sudo it uses the SUDO_UID domains first.
2. Restarts only under launchd. It kickstarts only when the job is running and its program is the replaced InstallDir/pilot-daemon (symlinks resolved). Otherwise it leaves the daemon running:
   • No job, or the job is loaded but runs no process: the updater asks the socket. A daemon that answers with an older version was started by hand. It is reported in restart_error with restart it with: pilotctl daemon stop && pilotctl daemon start. If nothing answers, the daemon is not running and its next start uses the new binary, which is not an error.
   • The job runs another binary: reported. Restarting it would not run the new release.
3. launchctl kickstart -k <target>. launchd sends SIGTERM, the daemon's graceful shutdown stops its apps, and launchd starts the job again on the new binary, whatever its KeepAlive setting.
4. Confirms within 60 s. It waits until launchd runs a new pid, then until that process answers IPC info on the job's -socket (default /tmp/pilot.sock) with the installed version from .pilot-version. It uses common/driver, with a 3 s cap per probe. A failed kickstart, a job that does not come back, or a daemon that does not answer with the installed version each goes to restart_error with the command to fix it.
5. Records the result in update-state.json. New fields daemon_restarted_at, daemon_restarted_by (for example launchd gui/501/network.pilotprotocol.pilot-daemon or systemd pilot-daemon.service) and daemon_restarted_version. restart_error is cleared on success.

The updater service. When a one-shot RunOnce (pilotctl update) has replaced pilot-updater, restartUpdaterService kickstarts network.pilotprotocol.pilot-updater. It does this only when the job is running from the replaced binary, and never when the caller is the job itself. It runs after the daemon restart and the .daemon-last-restart write, so the new updater does not restart the daemon a second time. The result goes to updater_restart_error / updater_restarted_at. The loop still exits after replacing itself, and launchd restarts it (KeepAlive true). The loop never kickstarts its own job.

Recording fixes:

• update-state.json writers keep the fields they do not define (see the verification section below).
• A manual run that replaced pilot-updater now records the daemon restart it made.
• On macOS a check clears restart_error once the daemon answers over IPC with the installed version, as Linux already did from /proc. The daemon is asked only while a restart_error is on record, so the hourly loop makes no extra launchctl or IPC calls.

launchctl is still the only external command the updater runs, now print as well as kickstart. TestNoExternalToolDependency now checks every non-test source file.

Tests

• Fake launchd (zz_launchd_test.go). A print/kickstart state machine that renders real launchctl print output, including nested blocks that repeat state/pid, and answers IPC. It covers: kickstart and confirm, a symlinked program, the legacy label in the user domain, the sudo domain, no job (not running / hand-started old / unknown version / already current), job loaded but idle, job running another binary, a refused kickstart, no respawn, never answers, wrong version, Stop() aborting the wait, a self-pid guard, all updater-service cases, and end to end through RunOnce (daemon then updater kickstarted, fields recorded, stale restart_error cleared, no second restart) and through the loop's self-update (it exits, and its successor restarts the daemon).
• Exec-level (zz_launchctl_exec_test.go). Uses the real runCommand/exec path with a shell-script launchctl that is the only thing on PATH, plus the real common/driver IPC probe against a fake daemon socket. Also covers the probe's answering, absent and wedged (timeout) cases.
• TestMain now also stubs the IPC probe, so no test can reach a developer's daemon on /tmp/pilot.sock.
• CI: a new test-macos job runs the suite natively on macOS.
• go test -race ./... passes on macOS arm64 and on Linux (golang:1.25 container, non-root). gosec, with the repo's exclusions, reports nothing.

Clean-runner verification (real launchd)

Run 35979204079 used a temporary verify/launchd-restart branch, now deleted, on clean GitHub macOS runners (3 jobs). Each job:

1. installed the current stable with the official install.sh, which writes and loads both LaunchAgents;
2. downgraded the binaries to v1.13.9 from its checksums.txt-verified release archive and restarted the updater job onto v1.13.9. (install.sh --version v1.13.9 refuses with "integrity anchors disagree", because the manifest anchors are the latest stable's);
3. ran web4 main pilotctl built against this branch: pilotctl update --pin v1.13.10, with real SLSA and checksum verification.

runner	daemon	result
macos-15 (arm64)	launchd job	daemon pid 3966 → 4115, answered v1.13.10 over IPC; updater job pid 3843 → 4121; restart_error empty, restart_needed: false; daemon.log shows the graceful SIGTERM shutdown (shutting down 0 → 1); a second pilotctl update is a no-op (same pid)
macos-15-intel	launchd job	daemon pid 11689 → 11946, answered v1.13.10; updater job pid 11671 → 11957; graceful shutdown logged; second run is a no-op
macos-15 (arm64)	started by hand, no launchd job	daemon left running on v1.13.9 (same pid, not kickstarted, no launchd daemon started); restart_error: daemon on /tmp/pilot.sock left running the old version (v1.13.9): no launchd job network.pilotprotocol.pilot-daemon is loaded in gui/501 or user/501, … restart it with: pilotctl daemon stop && pilotctl daemon start; restart_needed: true; updater job still restarted (pid 3031 → 3141)

This confirms that launchctl kickstart -k on the SuccessfulExit=false daemon job sends SIGTERM, which takes the graceful shutdown path, and that launchd then starts the new binary.

Found by that run, fixed in the second commit. The restarted updater service is v1.13.10's own binary (updater v0.2.5). It rewrote update-state.json 40 ms after pilotctl update and dropped daemon_restarted_* / updater_restarted_at, fields it does not know. Read-merge-write now keeps fields the writer does not define (TestUpdateStatus_KeepsFieldsOfNewerWriters), so from this version on an older writer passes newer fields through. Writers at v0.2.5 and earlier still drop them. That is why the harness reads the restart results from pilotctl's log rather than from those fields.

Rollout note

A release with this updater fixes restarts for updates performed by that release: its pilotctl update, and its updater service after that service restarts on the new binary. Going from v1.13.10 to the next release still runs v0.2.5's blind kickstart, which restarts a launchd daemon but does not confirm it. For web4, bump github.com/pilot-protocol/updater once this is tagged. Proposed tag: v0.2.6 (v0.2.6-beta.1 is the current main).

Do not merge without review.

🤖 Generated with Claude Code

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!