diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 854a858..1c2d0c6 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -51,3 +51,20 @@ jobs: name: coverage-updater path: updater/coverage.out retention-days: 30 + + # The macOS restart path (launchctl, the daemon's IPC socket) runs natively + # here. launchctl and the daemon are fakes: TestMain stubs both, and the + # exec-level test puts a script launchctl alone on PATH. + test-macos: + runs-on: macos-latest + steps: + - uses: actions/checkout@v7 + + - uses: actions/setup-go@v7 + with: + go-version-file: go.mod + cache: true + cache-dependency-path: go.sum + + - name: Run tests + run: go test -race ./... diff --git a/CHANGELOG.md b/CHANGELOG.md index 29014ef..2ad99b8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,64 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +### Fixed + +- **macOS: the daemon is restarted onto the new binaries only when launchd + runs it, and the restart is confirmed.** Before, the updater ran `launchctl + kickstart -k gui//network.pilotprotocol.pilot-daemon` blindly and + reported success as soon as launchctl returned. It now reads the job with + `launchctl print` first and kickstarts it only when the job is running and + its program is the `pilot-daemon` that was just replaced. It then waits (up + to 60 s) for launchd to run a new process and for that process to answer + over IPC (`info`) with the installed version. Otherwise the daemon is left + running and `restart_error` says why and how to restart it: no launchd job + runs it (started by hand), the job runs another binary, kickstart failed, + launchd did not start the job again, or the new daemon did not answer with + the installed version in time. A daemon that is not running is not an + error: its next start uses the new binary. The old `com.vulturelabs.*` + label and the `user/` domain are found too, and under `sudo` the + invoking user's domain (`SUDO_UID`) is used. +- **macOS: a one-shot update (`pilotctl update`) that replaces + `pilot-updater` restarts the updater service** (launchd job + `network.pilotprotocol.pilot-updater`) after the daemon, so the service + runs the new binary instead of the old one until the next login. It is + restarted only when its program is the replaced binary, and never when the + caller is the service itself. The result is recorded in + `updater_restart_error` / `updater_restarted_at`. +- A one-shot update that replaced `pilot-updater` (every release does) now + records the daemon restart it made. Before, `restart_error` from an earlier + run was kept even though this run restarted the daemon. +- On macOS a check clears `restart_error` once the daemon answers over IPC + with the installed version, as Linux already did from `/proc`. Before, it + stayed until the next release. The daemon is only asked while a + `restart_error` is on record. +- Writers of `update-state.json` keep the fields they do not define. The + file is shared by `pilotctl update` and the updater service, which can run + different updater versions. On a macOS runner the freshly restarted + updater service (updater v0.2.5, from v1.13.10) rewrote the file 40 ms + after `pilotctl update` and dropped `daemon_restarted_*` and + `updater_restarted_at`. From this version on, an older writer passes newer + fields through. v0.2.5 and earlier still drop them. + +### Added + +- `daemon_restarted_at`, `daemon_restarted_by` and `daemon_restarted_version` + in `update-state.json`: when the updater last restarted the daemon onto new + binaries and saw it come back, which service manager did it (`launchd + gui/501/network.pilotprotocol.pilot-daemon`, `systemd pilot-daemon.service`) + and, on macOS, the version the daemon reported. +- `updater_restart_error` and `updater_restarted_at` in `update-state.json` + (see above). + +### Changed + +- The updater now connects to the daemon's IPC socket (the job's `-socket` + argument, default `/tmp/pilot.sock`) to read its version on macOS. It still + runs no external tool other than `launchctl`, now `print` as well as + `kickstart`. + +## [v0.2.5] - 2026-09-24 + ### Added - `update-state.json` status file written after every check (`Config.StatusPath`, diff --git a/README.md b/README.md index 95aab45..6e8b118 100644 --- a/README.md +++ b/README.md @@ -48,23 +48,33 @@ if st.RestartError != "" { `RunOnce` returns the check's error. It never exits the calling process. If the release replaced `pilot-updater`, the daemon is still restarted onto the -new binaries. A running updater service switches to its new binary the next -time its service manager restarts it. +new binaries. On macOS the updater service (launchd job +`network.pilotprotocol.pilot-updater`) is then restarted onto the new +`pilot-updater` as well. On Linux a running updater service switches to its +new binary the next time systemd restarts it. A `RunOnce` caller writes the status file only when it sets `StatusPath` (or -`StatePath`); otherwise only `LastStatus()` has the result. **`pilotctl -update` does not do this yet.** As of web4 `main` on 2026-09-24, -`cmd/pilotctl/updates.go` sets neither path and ignores the error `RunOnce` -returns. So manual updates are not recorded in `update-state.json`, and a -failed `pilotctl update` still prints success and exits 0. That stays true -until pilotctl passes `StatusPath: ~/.pilot/update-state.json` and checks the -error, which needs a web4 change after this version is released. +`StatePath`); otherwise only `LastStatus()` has the result. `pilotctl update` +(web4 `cmd/pilotctl/updates.go`, since updater v0.2.5) passes +`StatusPath: ~/.pilot/update-state.json` and exits non-zero when `RunOnce` +returns an error. ### Restarting the daemon -After installing new binaries the updater restarts the daemon onto them: - -- **macOS:** `launchctl kickstart -k gui//network.pilotprotocol.pilot-daemon`. +After installing new binaries the updater restarts the daemon onto them, but +only when a service manager will start it again: + +- **macOS:** `launchctl kickstart -k gui//network.pilotprotocol.pilot-daemon`, + but only when that launchd job runs the daemon. The updater reads the job + with `launchctl print` and kickstarts it only when it is running and its + program is the `pilot-daemon` that was just replaced. launchd sends the + daemon SIGTERM (its graceful shutdown stops the apps it runs) and starts + the job again on the new binary, whatever its `KeepAlive` policy. The + updater then waits up to 60 s for launchd to run a new process and for that + process to answer over IPC (`info`, on the job's `-socket`) with the + installed version. The old `com.vulturelabs.pilot-daemon` label and the + `user/` domain are found too; under `sudo` the invoking user's + (`SUDO_UID`) domain is used. - **Linux:** SIGTERM, but only when systemd will start the daemon again. The daemon must run as the main program (`ExecStart=`) of a system service whose unit restarts it after a clean exit. pilot-daemon exits 0 on SIGTERM, so @@ -73,6 +83,28 @@ After installing new binaries the updater restarts the daemon onto them: not run `systemctl`. After the SIGTERM it waits up to 30 s for a new daemon process on the new binary. +On macOS the daemon is left running the old version, with `restart_error` +saying how to restart it, when: + +- no launchd job runs it: it was started by hand (`pilot-daemon ...`) while + the job was not loaded, or the job is loaded but runs no process. The + updater tells this from a daemon answering on the socket with an older + version. When no daemon answers, it is not running and its next start uses + the new binary, which is not an error; +- the job starts another `pilot-daemon` (a second installation), so a + restart would not run the new release. + +It is also reported when `launchctl kickstart` fails, when launchd does not +start the job again, or when the new daemon does not answer with the +installed version within 60 s. + +When a one-shot `RunOnce` (`pilotctl update`) replaced `pilot-updater`, it +restarts the updater job `network.pilotprotocol.pilot-updater` the same way, +after the daemon and only when the job runs the replaced binary. It never +restarts itself: the loop (`Start`) exits instead, and launchd starts it +again (`KeepAlive`). The result goes to `updater_restart_error` / +`updater_restarted_at`. + On Linux the daemon is left running the old version, with `restart_error` saying how to restart it, when: @@ -121,12 +153,14 @@ set, it goes next to the control file. For a standard install that is set, nothing is written. The long-running updater and any one-shot `RunOnce` caller configured with -the same path share the file. `pilotctl update` does not pass one yet (see -above). Each writer takes an advisory lock on `update-state.json.lock` and +the same path share the file, as the updater service and `pilotctl update` +do. Each writer takes an advisory lock on `update-state.json.lock` and merges its fields into what is already there, so concurrent writers from different processes do not lose each other's changes. Writes are atomic (temp file + rename). A writer that cannot get the lock within 5 s writes -anyway and logs a warning. +anyway and logs a warning. A writer keeps the fields it does not define, so +an older updater sharing the file (for example the updater service while a +newer `pilotctl` runs) does not drop fields a newer one wrote. ```json { @@ -142,6 +176,9 @@ anyway and logs a warning. "latest_version": "v1.13.10", "last_update_at": "2026-09-01T12:35:40Z", "last_update_version": "v1.13.9", + "daemon_restarted_at": "2026-09-01T12:35:47Z", + "daemon_restarted_by": "launchd gui/501/network.pilotprotocol.pilot-daemon", + "daemon_restarted_version": "v1.13.9", "loop_pid": 812, "loop_started_at": "2026-09-20T08:11:02Z", "next_check_at": "2026-09-24T11:00:12Z" @@ -155,7 +192,9 @@ anyway and logs a warning. | `consecutive_failures` | Failed checks since the last success. | | `last_success_at` | The last check that completed without error. | | `current_version` / `latest_version` | The installed release, and the release it was compared against (the pin, when pinned). | -| `restart_error` | New binaries were installed but the daemon is not running them. Either it was left on the old version because stopping it would have left it down (see [Restarting the daemon](#restarting-the-daemon)) or the restart failed, or it was stopped and did not come back. Says how to restart it. Cleared by the next successful restart, and on Linux by the next check that finds the daemon running the installed binary. | +| `restart_error` | New binaries were installed but the daemon is not running them. Either it was left on the old version because stopping it would have left it down (see [Restarting the daemon](#restarting-the-daemon)) or the restart failed, or it was stopped and did not come back. Says how to restart it. Cleared by the next successful restart, and by the next check that finds the daemon running the installed release (Linux: from `/proc`; macOS: the daemon's IPC `info` version). | +| `daemon_restarted_at`, `daemon_restarted_by`, `daemon_restarted_version` | The last restart the updater made that brought the daemon back on new binaries, the service manager and job that did it, and (macOS) the version the daemon then reported over IPC. | +| `updater_restart_error`, `updater_restarted_at` | macOS only: a one-shot update replaced `pilot-updater` and could not (error) or did (time) restart the updater service's launchd job onto it. | | `loop_pid`, `loop_started_at` | The long-running updater process. | | `next_check_at` | When the loop wakes next. It is updated even while auto-update is disabled, so it doubles as a heartbeat. | @@ -210,9 +249,11 @@ The SHA-256 match against the attested `checksums.txt` still gates every install The updater never runs `gh` or any other external tool. It checks the SLSA provenance of `checksums.txt` in-process with sigstore-go -(`attestation.go`). The only command it runs is `launchctl kickstart`, which -restarts the daemon under launchd on macOS. On Linux it reads `/proc` and the -systemd unit files directly instead of running `systemctl`. +(`attestation.go`). The only command it runs is `launchctl` (`print` and +`kickstart`), which reads and restarts the daemon and updater jobs under +launchd on macOS. It asks the daemon for its version over its IPC socket +directly. On Linux it reads `/proc` and the systemd unit files directly +instead of running `systemctl`. `TestNoExternalToolDependency` fails the build if another exec is added. ### Nodes stuck on v1.12.2–v1.13.4 (updater v0.2.3) @@ -253,6 +294,7 @@ because that switches off provenance checks. |---|---| | `updater.go` | `Updater`: check loop, `RunOnce`, release polling, install, daemon restart. | | `supervisor.go` | Linux restart safety: finds the daemon's systemd service and checks its restart policy before any SIGTERM. | +| `launchd.go` | macOS restart: finds the daemon and updater launchd jobs (`launchctl print`), kickstarts them and confirms the daemon's version over IPC. | | `download.go` | Resumable asset downloads with idle and total-duration limits. | | `github.go` | GitHub API calls: optional `GITHUB_TOKEN`, rate-limit errors. | | `attestation.go` | In-process SLSA provenance verification of `checksums.txt` (sigstore-go). | diff --git a/launchd.go b/launchd.go new file mode 100644 index 0000000..aac3cee --- /dev/null +++ b/launchd.go @@ -0,0 +1,560 @@ +// SPDX-License-Identifier: AGPL-3.0-or-later + +package updater + +import ( + "fmt" + "log/slog" + "os" + "path/filepath" + "strconv" + "strings" + "time" + + "github.com/pilot-protocol/common/driver" +) + +// Restarting the daemon on macOS. +// +// install.sh runs the daemon as the LaunchAgent +// network.pilotprotocol.pilot-daemon (KeepAlive SuccessfulExit=false) and the +// updater as network.pilotprotocol.pilot-updater (KeepAlive true). After an +// update the updater restarts the daemon with +// +// launchctl kickstart -k gui//network.pilotprotocol.pilot-daemon +// +// launchd sends the running daemon SIGTERM (its graceful shutdown stops the +// apps it runs) and starts the job again, which execs the new binary. +// kickstart restarts the job whatever its KeepAlive policy says. +// +// That is only safe, and only useful, when launchd runs this daemon. So the +// updater first reads the job with `launchctl print` and restarts it only when +// the job is running and its program is the binary that was just replaced. +// Afterwards it waits for launchd to run a new process and for that process to +// answer over IPC with the installed version. Otherwise, as on Linux, the +// daemon is left running and restart_error in the status file says why and +// how to restart it: +// +// - no launchd job is loaded (the daemon was started by hand, or not at +// all): a daemon that answers on the socket with an older version is +// reported; one that does not answer is not running, and its next start +// uses the new binary; +// - the job starts another pilot-daemon (a second installation): restarting +// it would not run the new release; +// - the job did not come back, or the new process did not answer with the +// installed version in time. +// +// A one-shot RunOnce (`pilotctl update`) that replaced pilot-updater also +// restarts the updater job, so the updater service runs the new binary too. +// The loop never restarts its own job: it exits and launchd starts it again. + +// Labels of the LaunchAgents install.sh writes, current first. install.sh +// renamed com.vulturelabs.* to network.pilotprotocol.*; pilotctl and the +// uninstaller still accept the old names, and so does the updater. +var ( + daemonLaunchdLabels = []string{"network.pilotprotocol.pilot-daemon", "com.vulturelabs.pilot-daemon"} + updaterLaunchdLabels = []string{"network.pilotprotocol.pilot-updater", "com.vulturelabs.pilot-updater"} +) + +const ( + // defaultLaunchdRestartWait bounds how long the updater waits, after + // `launchctl kickstart -k`, for launchd to start the job again and (for + // the daemon) for the new daemon to answer over IPC. The daemon's + // shutdown takes up to ~5 s, and a daemon that starts installed apps + // first can take well over 10 s to open its socket (`pilotctl daemon + // start` waits 30 s for it). + defaultLaunchdRestartWait = 60 * time.Second + + // daemonProbeTimeout bounds one IPC info call, so a wedged daemon cannot + // hang the updater. + daemonProbeTimeout = 3 * time.Second +) + +// launchdJob is the part of `launchctl print ` the updater uses. +type launchdJob struct { + target string // domain/label, e.g. gui/501/network.pilotprotocol.pilot-daemon + state string // "running", "not running", "spawn scheduled", ... + program string // "program = ..." (the executable launchd starts) + args []string // "arguments = { ... }" + pid int // 0 when no process is running + // lastExit is "last exit code = ..." (or "last exit reason"), for messages. + lastExit string +} + +// programPath is the executable the job starts. +func (j launchdJob) programPath() string { + if j.program != "" { + return j.program + } + if len(j.args) > 0 { + return j.args[0] + } + return "" +} + +// socket is the daemon's IPC socket: the -socket argument of the job, or the +// daemon's default. +func (j launchdJob) socket() string { + return socketFromArgs(j.args) +} + +// describe summarises the job's state for error messages. +func (j launchdJob) describe() string { + s := "state = " + orNone(j.state) + if j.pid > 0 { + s += fmt.Sprintf(", pid = %d", j.pid) + } + if j.lastExit != "" { + s += ", last exit = " + j.lastExit + } + return s +} + +func orNone(s string) string { + if s == "" { + return "unknown" + } + return s +} + +// socketFromArgs returns the value of the daemon's -socket flag in args (Go +// flag syntax: -socket X, --socket X, -socket=X), or the daemon's default +// socket when the flag is absent. +func socketFromArgs(args []string) string { + for i := 1; i < len(args); i++ { + a := args[i] + switch { + case a == "-socket" || a == "--socket": + if i+1 < len(args) { + return args[i+1] + } + case strings.HasPrefix(a, "-socket="): + return strings.TrimPrefix(a, "-socket=") + case strings.HasPrefix(a, "--socket="): + return strings.TrimPrefix(a, "--socket=") + } + } + return driver.DefaultSocketPath() +} + +// parseLaunchdPrint parses the output of `launchctl print /