Skip to content

fix(darwin): restart the daemon only under its launchd job, confirm over IPC; restart the updater job #51

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
TeoSlayer merged 3 commits into from
Oct 1, 2026
Merged

fix(darwin): restart the daemon only under its launchd job, confirm over IPC; restart the updater job #51

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
5551971
fix(darwin): restart the daemon only under its launchd job, confirm o…
teovl Sep 24, 2026
7889062
fix(status): keep update-state.json fields this version does not define
teovl Sep 24, 2026
e294faf
Merge branch 'main' into fix/restart-launchd-daemon
TeoSlayer Oct 1, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 17 additions & 0 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,3 +51,20 @@ jobs:
name: coverage-updater
path: updater/coverage.out
retention-days: 30

# The macOS restart path (launchctl, the daemon's IPC socket) runs natively
# here. launchctl and the daemon are fakes: TestMain stubs both, and the
# exec-level test puts a script launchctl alone on PATH.
test-macos:
runs-on: macos-latest
steps:
- uses: actions/checkout@v7

- uses: actions/setup-go@v7
with:
go-version-file: go.mod
cache: true
cache-dependency-path: go.sum

- name: Run tests
run: go test -race ./...
58 changes: 58 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,64 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## [Unreleased]

### Fixed

- **macOS: the daemon is restarted onto the new binaries only when launchd
runs it, and the restart is confirmed.** Before, the updater ran `launchctl
kickstart -k gui/<uid>/network.pilotprotocol.pilot-daemon` blindly and
reported success as soon as launchctl returned. It now reads the job with
`launchctl print` first and kickstarts it only when the job is running and
its program is the `pilot-daemon` that was just replaced. It then waits (up
to 60 s) for launchd to run a new process and for that process to answer
over IPC (`info`) with the installed version. Otherwise the daemon is left
running and `restart_error` says why and how to restart it: no launchd job
runs it (started by hand), the job runs another binary, kickstart failed,
launchd did not start the job again, or the new daemon did not answer with
the installed version in time. A daemon that is not running is not an
error: its next start uses the new binary. The old `com.vulturelabs.*`
label and the `user/<uid>` domain are found too, and under `sudo` the
invoking user's domain (`SUDO_UID`) is used.
- **macOS: a one-shot update (`pilotctl update`) that replaces
`pilot-updater` restarts the updater service** (launchd job
`network.pilotprotocol.pilot-updater`) after the daemon, so the service
runs the new binary instead of the old one until the next login. It is
restarted only when its program is the replaced binary, and never when the
caller is the service itself. The result is recorded in
`updater_restart_error` / `updater_restarted_at`.
- A one-shot update that replaced `pilot-updater` (every release does) now
records the daemon restart it made. Before, `restart_error` from an earlier
run was kept even though this run restarted the daemon.
- On macOS a check clears `restart_error` once the daemon answers over IPC
with the installed version, as Linux already did from `/proc`. Before, it
stayed until the next release. The daemon is only asked while a
`restart_error` is on record.
- Writers of `update-state.json` keep the fields they do not define. The
file is shared by `pilotctl update` and the updater service, which can run
different updater versions. On a macOS runner the freshly restarted
updater service (updater v0.2.5, from v1.13.10) rewrote the file 40 ms
after `pilotctl update` and dropped `daemon_restarted_*` and
`updater_restarted_at`. From this version on, an older writer passes newer
fields through. v0.2.5 and earlier still drop them.

### Added

- `daemon_restarted_at`, `daemon_restarted_by` and `daemon_restarted_version`
in `update-state.json`: when the updater last restarted the daemon onto new
binaries and saw it come back, which service manager did it (`launchd
gui/501/network.pilotprotocol.pilot-daemon`, `systemd pilot-daemon.service`)
and, on macOS, the version the daemon reported.
- `updater_restart_error` and `updater_restarted_at` in `update-state.json`
(see above).

### Changed

- The updater now connects to the daemon's IPC socket (the job's `-socket`
argument, default `/tmp/pilot.sock`) to read its version on macOS. It still
runs no external tool other than `launchctl`, now `print` as well as
`kickstart`.

## [v0.2.5] - 2026-09-24

### Added

- `update-state.json` status file written after every check (`Config.StatusPath`,
Expand Down
80 changes: 61 additions & 19 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,23 +48,33 @@ if st.RestartError != "" {

`RunOnce` returns the check's error. It never exits the calling process. If
the release replaced `pilot-updater`, the daemon is still restarted onto the
new binaries. A running updater service switches to its new binary the next
time its service manager restarts it.
new binaries. On macOS the updater service (launchd job
`network.pilotprotocol.pilot-updater`) is then restarted onto the new
`pilot-updater` as well. On Linux a running updater service switches to its
new binary the next time systemd restarts it.

A `RunOnce` caller writes the status file only when it sets `StatusPath` (or
`StatePath`); otherwise only `LastStatus()` has the result. **`pilotctl
update` does not do this yet.** As of web4 `main` on 2026-09-24,
`cmd/pilotctl/updates.go` sets neither path and ignores the error `RunOnce`
returns. So manual updates are not recorded in `update-state.json`, and a
failed `pilotctl update` still prints success and exits 0. That stays true
until pilotctl passes `StatusPath: ~/.pilot/update-state.json` and checks the
error, which needs a web4 change after this version is released.
`StatePath`); otherwise only `LastStatus()` has the result. `pilotctl update`
(web4 `cmd/pilotctl/updates.go`, since updater v0.2.5) passes
`StatusPath: ~/.pilot/update-state.json` and exits non-zero when `RunOnce`
returns an error.

### Restarting the daemon

After installing new binaries the updater restarts the daemon onto them:

- **macOS:** `launchctl kickstart -k gui/<uid>/network.pilotprotocol.pilot-daemon`.
After installing new binaries the updater restarts the daemon onto them, but
only when a service manager will start it again:

- **macOS:** `launchctl kickstart -k gui/<uid>/network.pilotprotocol.pilot-daemon`,
but only when that launchd job runs the daemon. The updater reads the job
with `launchctl print` and kickstarts it only when it is running and its
program is the `pilot-daemon` that was just replaced. launchd sends the
daemon SIGTERM (its graceful shutdown stops the apps it runs) and starts
the job again on the new binary, whatever its `KeepAlive` policy. The
updater then waits up to 60 s for launchd to run a new process and for that
process to answer over IPC (`info`, on the job's `-socket`) with the
installed version. The old `com.vulturelabs.pilot-daemon` label and the
`user/<uid>` domain are found too; under `sudo` the invoking user's
(`SUDO_UID`) domain is used.
- **Linux:** SIGTERM, but only when systemd will start the daemon again. The
daemon must run as the main program (`ExecStart=`) of a system service whose
unit restarts it after a clean exit. pilot-daemon exits 0 on SIGTERM, so
Expand All @@ -73,6 +83,28 @@ After installing new binaries the updater restarts the daemon onto them:
not run `systemctl`. After the SIGTERM it waits up to 30 s for a new daemon
process on the new binary.

On macOS the daemon is left running the old version, with `restart_error`
saying how to restart it, when:

- no launchd job runs it: it was started by hand (`pilot-daemon ...`) while
the job was not loaded, or the job is loaded but runs no process. The
updater tells this from a daemon answering on the socket with an older
version. When no daemon answers, it is not running and its next start uses
the new binary, which is not an error;
- the job starts another `pilot-daemon` (a second installation), so a
restart would not run the new release.

It is also reported when `launchctl kickstart` fails, when launchd does not
start the job again, or when the new daemon does not answer with the
installed version within 60 s.

When a one-shot `RunOnce` (`pilotctl update`) replaced `pilot-updater`, it
restarts the updater job `network.pilotprotocol.pilot-updater` the same way,
after the daemon and only when the job runs the replaced binary. It never
restarts itself: the loop (`Start`) exits instead, and launchd starts it
again (`KeepAlive`). The result goes to `updater_restart_error` /
`updater_restarted_at`.

On Linux the daemon is left running the old version, with `restart_error`
saying how to restart it, when:

Expand Down Expand Up @@ -121,12 +153,14 @@ set, it goes next to the control file. For a standard install that is
set, nothing is written.

The long-running updater and any one-shot `RunOnce` caller configured with
the same path share the file. `pilotctl update` does not pass one yet (see
above). Each writer takes an advisory lock on `update-state.json.lock` and
the same path share the file, as the updater service and `pilotctl update`
do. Each writer takes an advisory lock on `update-state.json.lock` and
merges its fields into what is already there, so concurrent writers from
different processes do not lose each other's changes. Writes are atomic
(temp file + rename). A writer that cannot get the lock within 5 s writes
anyway and logs a warning.
anyway and logs a warning. A writer keeps the fields it does not define, so
an older updater sharing the file (for example the updater service while a
newer `pilotctl` runs) does not drop fields a newer one wrote.

```json
{
Expand All @@ -142,6 +176,9 @@ anyway and logs a warning.
"latest_version": "v1.13.10",
"last_update_at": "2026-09-01T12:35:40Z",
"last_update_version": "v1.13.9",
"daemon_restarted_at": "2026-09-01T12:35:47Z",
"daemon_restarted_by": "launchd gui/501/network.pilotprotocol.pilot-daemon",
"daemon_restarted_version": "v1.13.9",
"loop_pid": 812,
"loop_started_at": "2026-09-20T08:11:02Z",
"next_check_at": "2026-09-24T11:00:12Z"
Expand All @@ -155,7 +192,9 @@ anyway and logs a warning.
| `consecutive_failures` | Failed checks since the last success. |
| `last_success_at` | The last check that completed without error. |
| `current_version` / `latest_version` | The installed release, and the release it was compared against (the pin, when pinned). |
| `restart_error` | New binaries were installed but the daemon is not running them. Either it was left on the old version because stopping it would have left it down (see [Restarting the daemon](#restarting-the-daemon)) or the restart failed, or it was stopped and did not come back. Says how to restart it. Cleared by the next successful restart, and on Linux by the next check that finds the daemon running the installed binary. |
| `restart_error` | New binaries were installed but the daemon is not running them. Either it was left on the old version because stopping it would have left it down (see [Restarting the daemon](#restarting-the-daemon)) or the restart failed, or it was stopped and did not come back. Says how to restart it. Cleared by the next successful restart, and by the next check that finds the daemon running the installed release (Linux: from `/proc`; macOS: the daemon's IPC `info` version). |
| `daemon_restarted_at`, `daemon_restarted_by`, `daemon_restarted_version` | The last restart the updater made that brought the daemon back on new binaries, the service manager and job that did it, and (macOS) the version the daemon then reported over IPC. |
| `updater_restart_error`, `updater_restarted_at` | macOS only: a one-shot update replaced `pilot-updater` and could not (error) or did (time) restart the updater service's launchd job onto it. |
| `loop_pid`, `loop_started_at` | The long-running updater process. |
| `next_check_at` | When the loop wakes next. It is updated even while auto-update is disabled, so it doubles as a heartbeat. |

Expand Down Expand Up @@ -210,9 +249,11 @@ The SHA-256 match against the attested `checksums.txt` still gates every install

The updater never runs `gh` or any other external tool. It checks the SLSA
provenance of `checksums.txt` in-process with sigstore-go
(`attestation.go`). The only command it runs is `launchctl kickstart`, which
restarts the daemon under launchd on macOS. On Linux it reads `/proc` and the
systemd unit files directly instead of running `systemctl`.
(`attestation.go`). The only command it runs is `launchctl` (`print` and
`kickstart`), which reads and restarts the daemon and updater jobs under
launchd on macOS. It asks the daemon for its version over its IPC socket
directly. On Linux it reads `/proc` and the systemd unit files directly
instead of running `systemctl`.
`TestNoExternalToolDependency` fails the build if another exec is added.

### Nodes stuck on v1.12.2–v1.13.4 (updater v0.2.3)
Expand Down Expand Up @@ -253,6 +294,7 @@ because that switches off provenance checks.
|---|---|
| `updater.go` | `Updater`: check loop, `RunOnce`, release polling, install, daemon restart. |
| `supervisor.go` | Linux restart safety: finds the daemon's systemd service and checks its restart policy before any SIGTERM. |
| `launchd.go` | macOS restart: finds the daemon and updater launchd jobs (`launchctl print`), kickstarts them and confirms the daemon's version over IPC. |
| `download.go` | Resumable asset downloads with idle and total-duration limits. |
| `github.go` | GitHub API calls: optional `GITHUB_TOKEN`, rate-limit errors. |
| `attestation.go` | In-process SLSA provenance verification of `checksums.txt` (sigstore-go). |
Expand Down
Loading
Loading