fix: visible update failures, resumable downloads, systemd-aware Linux restart

[TeoSlayer]

This PR covers the updater part of reliability sweep items 13, 45, 12 and 7. See "Deploy notes" for the release order.

Problem (sweep evidence)

Line numbers are for origin/main / v0.2.4 unless another tag is named.

• Update failures were silent (HYG-05, FIELDLOGS-6).
  • RunOnce() returns nothing (updater.go:127), and errors only go to slog. web4 cmd/pilotctl/updates.go:327 calls u.RunOnce() and then always calls outputOK (:330).
  • Reproduced: pilotctl --json update printed {"status":"ok"} and exited 0. At the same time, stderr logged GitHub API returned 403: ... API rate limit exceeded.
  • update status (updates.go:66-88) shows only the auto-update flag and pilotctl's own compiled version.
  • ~/.pilot/updater.log has 307 hourly failed to apply update … gh CLI required errors, from 2026-07-31T19:08 to 2026-08-16T11:10. That is 16 days stuck on current=v1.9.1 while latest=v1.13.9, and no Pilot surface showed it.
• Downloads had a fixed 30 s total timeout (ci-09).
  • &http.Client{Timeout: 30 * time.Second} (updater.go:102) covers the whole body read in downloadFile (updater.go:486, io.Copy at :508).
  • The v1.13.9 archives are 16,809,908–19,341,023 B, so a download needs a sustained 4.5–5.2 Mbit/s to finish in time.
  • updater.log has 3 context deadline exceeded (... while reading body) errors, 1 checksums timeout and 2 TLS handshake timeouts. After each one, the updater waited a full hour before retrying.
• The releases API call never sent a token (HYG-05).
  • fetchRelease (updater.go:302) calls api.github.com without auth. Only the attestations call used a token.
  • updater.log has 23 API rate limit exceeded 403s (07-31 to 08-19) from VPN egress IPs, which share GitHub's limit of 60 unauthenticated requests per hour.
• Linux daemons were never restarted after an update (found during this work).
  • signalDaemonRestartLinux compares /proc/<pid>/exe to the install path exactly (updater.go:765).
  • replaceBinary renames the new binary over the old one (updater.go:668), after which the link reads <path> (deleted). The daemon is never found, so it keeps running the old binary and nothing records this.
  • The code comment at :749 relies on Restart=on-failure. That setting does not restart the clean exit the daemon makes on SIGTERM.
• RunOnce could exit its caller. When a release replaces pilot-updater, it calls u.exitFn(0) (os.Exit, updater.go:477). pilotctl update then exited 0 without saying anything and never restarted the daemon.
• Nodes on v1.12.2–v1.13.4 are stranded (HYG-04, ci-03).
  • updater v0.2.3 runs exec.LookPath("gh") and fails closed (v0.2.3 updater.go:789-791). web4 pins v0.2.3 in 12 stable tags (2026-06-22 to 07-25).
  • The launchd plist sets no PATH.
  • Verification narrowed this: auto-update was off by default in that window. The nodes affected are the ones that opted in, plus any manual pilotctl update run without gh.
• Go and grpc (ci-04).
  • The module declares go 1.25.12. On the v1.13.9 release binary, binary-mode govulncheck finds 16 standard-library vulnerabilities in the updater.
  • grpc v1.82.1 (GO-2026-6348) fails govulncheck on every PR.

What changed

Errors and status file (item 13): updater.go, status.go

• RunOnce() now returns error (updater.go:173). Failed fetches, bad tags, invalid pins and failed applies are returned, not only logged.
• There is a new update-state.json, written atomically after every check. It holds:
  • last_check_at, last_check_trigger, last_result (up_to_date / updated / failed), last_error
  • consecutive_failures, last_success_at, current_version, latest_version, last_update_*, restart_error
  • the loop heartbeat: loop_pid, loop_started_at and next_check_at. The loop writes these even while auto-update is disabled.
• The path comes from Config.StatusPath. It defaults to the directory of StatePath, so cmd/updater writes ~/.pilot/update-state.json with no code change. ReadStatus(path) and u.LastStatus() expose the file.
• Writers merge with the file on disk under an advisory flock on update-state.json.lock (status.go:145, :185), so processes that share a status path do not lose each other's updates.
  • If the lock is still held after 5 s, the write goes ahead without it and the updater logs this.
  • The lock file is opened read-only, so a lock file created by another user still works.
• RunOnce no longer calls os.Exit on its caller. The loop still exits for its service manager, after it records the update.

Linux daemon restart: supervisor.go, updater.go:836

• The daemon is matched whether /proc/<pid>/exe shows the plain install path or the path with (deleted).
• The updater sends SIGTERM only when the daemon will come back. That means the daemon is the ExecStart= program of a system service that restarts after a clean exit: Restart=always/on-success, with RestartForceExitStatus/RestartPreventExitStatus honoured.
  • The updater finds the unit from /proc/<pid>/cgroup (v1, v2 or hybrid).
  • It then reads the unit file and its drop-ins from the systemd search path. There is no systemctl exec.
• In every other case, the daemon keeps running and restart_error gives the reason and the command that restarts it. The other cases are:
  • pilotctl daemon start in a container, WSL or CI
  • units written by install.sh up to v1.9.0, which say Restart=on-failure
  • user services
  • a daemon running as a child of another service
• After a SIGTERM, the updater waits up to 30 s for a new daemon process on the new binary (supervisor.go:437). If none appears, it records a failure.
• On Linux, the next check that finds the daemon on the installed binary clears restart_error, for example after a manual restart.

Downloads (item 45): download.go

• There is no total timeout any more. An attempt fails after 60 s with no data, and one file is capped at 30 min in total.
• A dropped or stalled transfer resumes with an HTTP Range request, or starts over if the server ignores Range. There are up to 4 attempts; 5xx responses are retried and 4xx are not.
• Stop() aborts a download in progress.
• The TLS handshake timeout goes from 10 s to 20 s.

GitHub token (items 13 and 45): github.go

• The releases API now sends GITHUB_TOKEN / GH_TOKEN, as the attestations call already did. The token only ever goes to api.github.com.
• If GitHub rejects the token with a 401, the call is retried without it.
• Rate-limit errors now say what happened, when the limit resets and how to raise it.

No gh (item 12)

• TestNoExternalToolDependency parses the sources. It fails on any exec other than runCommand (which runs only launchctl) and on any "gh" literal.
• The live attestation test runs with an empty PATH.
• The README explains what stranded nodes must do. It also has a new "Restarting the daemon" section and gives the Linux log location (journalctl -u pilot-updater).

Go, grpc and CI (item 7)

• go 1.25.12 → go 1.25.13.
• grpc v1.82.1 → v1.83.2, which fixes GO-2026-6348. Dependabot's v1.83.1 (chore(deps): bump google.golang.org/grpc from 1.82.1 to 1.83.1 #46) fails dependency review.
• The govulncheck job is pinned to v1.7.0. @latest is v1.8.0, which requires Go 1.26, while setup-go runs with GOTOOLCHAIN=local, so the job failed before it scanned anything. Unpin it when moving to Go 1.26.
• TestMain stubs runCommand. Before this, running the suite on a Mac with Pilot installed ran launchctl kickstart -k on the developer's live daemon.

Compatibility

• Exported API: everything is additive except one signature change.
  • Added: Status, ReadStatus, (*Updater).LastStatus and Config.StatusPath.
  • Changed: RunOnce() now returns error. Statement calls still compile. Only code that stores it as a func() value would break.
  • The review-fix commit (4474271) adds only unexported fields and functions.
• web4: its call sites are unaffected. cmd/pilotctl/updates.go:327 and cmd/updater/main.go:95 build and vet against this branch, and the cmd/updater tests pass. This was checked in a scratch copy with a replace directive.
• New file on disk: writers create update-state.json.lock next to update-state.json. This is expected, and cleanup scripts should not delete it.
• Linux restart behaviour: see "Behaviour change to announce" under Deploy notes.

Tests

All green.

• Local macOS (go1.25.13, GOWORK=off):
  • go vet, go test ./... and go test -race ./... pass. I re-ran them on 4474271 while preparing this PR.
  • Coverage is 90.2% in total. supervisor.go and the status-lock functions are at 88–100%.
• Linux (golang:1.25.13-bookworm container, non-root, as in CI):
  • go vet and go test -race with coverage pass (91.1%).
  • go test -count=3 passes.
  • The new tests also pass 10 times in a row with -race.
• Security scans:
  • gosec v2.28.0 with CI's exclusions: 0 issues. There is one #nosec G115, on an fd-to-int conversion.
  • govulncheck v1.7.0: 0 reachable vulnerabilities.
  • gitleaks: no leaks.
• Real systemd e2e (new env-gated zz_systemd_e2e_test.go, run in a privileged debian:bookworm container with systemd as PID 1):
  • Restart=always restarted the daemon onto the new binary.
  • Restart=on-failure left the daemon active on the old pid and reported the error.
  • A Setsid daemon was left running and the error was reported.
  • With the supervision check bypassed, the last two cases fail, and the post-SIGTERM wait records "did not start it again" instead of a false success.
• New test files:
  • zz_supervisor_test.go: table tests for cgroup forms, restart policies and drop-ins. It also covers end-to-end RunOnce and recoverPendingRestart on the Linux path, clearing a stale restart_error, and the parsers.
  • zz_status_lock_test.go: writers in one process and across processes, the lock held during the write, the lock timeout, a read-only lock file and an unusable lock path.
  • zz_systemd_e2e_test.go.
• Updated tests: restartSpy in zz_status_test.go and TestSignalDaemonRestartLinux_Paths in zz_restart_test.go now use a supervised fake systemd.
• Other test coverage: there are also tests for:
  • resuming after a drop, a stall, a server that ignores Range, and a mismatched Content-Range
  • 4xx vs 5xx handling, and aborting on Stop
  • the token: sent, aliased, the 401 fallback, and never sent to other hosts
  • the rate-limit message
  • the failure streak resetting
  • the loop heartbeat
  • RunOnce not exiting its caller
  • matching the (deleted) exe path
• CI on 4474271: all checks pass: test, race, gosec, govulncheck, gitleaks, dependency-review, CodeQL/analyze, codecov/patch and snyk.
• Known and unrelated: TestReplaceBinary_TmpCreateFails and TestExtractTarGz_DestDirReadOnly fail only when run as root, because root can write to read-only directories. CI runs as non-root.

Review outcome

The review found 3 issues, and all 3 are fixed in 4474271.

• UPD49-1 (high): the first fix SIGTERMed the daemon even when nothing would restart it: pilotctl daemon start hosts, and units with Restart=on-failure. Those nodes went offline while the status said last_result=updated. The updater now sends SIGTERM only under a systemd unit that restarts after a clean exit, as described above.
• UPD49-2 (low): writers in different processes lost each other's status updates. In the repro, 203 of 400 increments were lost. The read-merge-write now holds the flock: 2 Updaters × 200 increments give exactly 400, and 3 child processes plus the parent × 100 give exactly 400.
• UPD49-3 (low): the README and CHANGELOG claimed that pilotctl update writes update-state.json. It does not until web4 passes StatusPath. The docs are corrected.

Deploy notes

• Order:
  1. Merge this PR.
  2. Tag the updater release.
  3. Bump github.com/pilot-protocol/updater in web4, whose go.mod has v0.2.4.
• web4 follow-up in cmd/pilotctl/updates.go:
  • Pass StatusPath: ~/.pilot/update-state.json.
  • Check the error from RunOnce.
  • Show LastStatus().RestartError.
  • Without these, manual updates are still not recorded (UPD49-3).
  • Later: update status should read updater.ReadStatus and warn when next_check_at is stale or loop_pid is dead.
• Behaviour change to announce:
  • Released updaters never found the Linux daemon (see the (deleted) bug above). After this release, daemons under a systemd system unit with Restart=always are restarted onto the new binary.
  • The daemon is still not restarted when it has no systemd supervisor, or when its unit says Restart=on-failure (install.sh up to v1.9.0). The difference is that restart_error now reports this and gives the manual command. Re-running install.sh rewrites the unit with Restart=always.
  • The pre-fix revision of this branch, which SIGTERMed unsupervised daemons, was never released, so no host ran it.
• Other follow-ups (not in this PR):
  • Make the auto-update default in help and docs match install.sh.
  • Set PATH in the plist and unit that install.sh writes.
  • Send a MOTD or registry notice to v1.12.2–v1.13.4 nodes (item 12).

🤖 Generated with Claude Code

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!