From 810d622b9eb599d652fae9e66b864fd4b4e39269 Mon Sep 17 00:00:00 2001 From: Teo Calin Date: Thu, 1 Oct 2026 17:49:50 +0300 Subject: [PATCH 1/2] daemon: warn at startup when running as PID 1 As the entrypoint of a container started without an init, the daemon is handed every orphaned process. The servers its apps start daemonize (redis-server, postgres), so they end up as children of PID 1 and stay zombies after they are stopped: Go only waits for children it started. The daemon does not reap them. A wait for any child also takes the exit status of the apps the supervisor is waiting on (measured: 17 of 300 exec.Cmd.Run calls failed with "waitid: no child processes" next to a wait4(-1) loop). Reaping belongs to an init process, so on Linux the daemon now logs one warning when it is PID 1, and the README's install notes say to use `docker run --init` or tini. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 7 +++++ README.md | 1 + cmd/daemon/main.go | 1 + cmd/daemon/pid1.go | 40 ++++++++++++++++++++++++++ cmd/daemon/zz_pid1_test.go | 57 ++++++++++++++++++++++++++++++++++++++ 5 files changed, 106 insertions(+) create mode 100644 cmd/daemon/pid1.go create mode 100644 cmd/daemon/zz_pid1_test.go diff --git a/CHANGELOG.md b/CHANGELOG.md index fd723e61..0437dc9b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -215,6 +215,13 @@ Detailed per-release notes are on the (redacted) and the hosts it must allow. In a Linux container/VM without systemd the installer runs as root without `PILOT_ALLOW_ROOT` (hosted sandboxes run the agent as root); regular hosts still refuse root. +- **The daemon warns when it is PID 1.** As the entrypoint of a container + started without an init, the daemon is handed every orphaned process, and + the servers its apps start (redis, postgres) stayed as zombies after they + were stopped. The daemon does not reap them — that would also take the exit + status of the apps it supervises — and now logs one warning at startup, on + Linux, saying to run it under an init (`docker run --init`, or tini as the + entrypoint). README: Install. ### Removed - **Hosted control plane client.** The hosted control plane has been retired, diff --git a/README.md b/README.md index fb712397..36d4ca2d 100644 --- a/README.md +++ b/README.md @@ -297,6 +297,7 @@ pilotctl daemon start New installs use transport `auto`: the daemon probes the beacon over UDP once at startup (one round trip when UDP works, at most ~1.5s when it does not) and, when there is no answer but the compat beacon answers over TCP 443 (a TLS request that a live beacon answers with `426 Upgrade Required`), runs in **compat** mode — registry over TLS on `registry.pilotprotocol.network:443`, beacon over WSS on `beacon.pilotprotocol.network:443`. When a proxy is configured and it refuses that check (wrong or stale credentials: `407`, a `403`, or the proxy cannot be reached), it runs compat anyway rather than falling back to direct connections past the proxy, and the registry dial then names the proxy's answer; `pilotctl daemon start` prints the daemon's last proxy error with a hint instead of only "did not become ready". When neither answers and no proxy is involved (no network yet, beacon outage) it stays on UDP, as before. In compat mode the daemon tunnels through `$HTTPS_PROXY` / `$ALL_PROXY` (honoring `$NO_PROXY`), asking the proxy to `CONNECT` by host name, so poisoned local DNS for the Pilot hosts does not matter. `pilotctl`'s own registry commands (`lookup`, the auto-handshake check, `recovery`) follow the daemon: through the proxy when it runs compat (or with an explicit proxy URL), directly when it runs UDP; with no daemon running and no transport configured they try the proxy first and a direct connection second. To skip the UDP probe, pick compat explicitly: `sh -s -- --transport compat`, `pilotctl config --set transport=compat`, or `pilotctl daemon start --transport compat`. - **No root, systemd or launchd needed.** Start the daemon with `pilotctl daemon start` from a shell that has the proxy variables. In a container or VM without systemd the installer also runs as root (the agent user in hosted sandboxes); on a regular host it still refuses root unless `PILOT_ALLOW_ROOT=1`. +- **In a container, give the daemon an init process.** Apps start servers that daemonize (redis, postgres); when one of those stops, the kernel hands it to PID 1 to be reaped, and a `pilot-daemon` that is PID 1 itself does not do that, so each one stays behind as a zombie (`Z` in `ps`). Run the container with `docker run --init` (Compose: `init: true`), or make [tini](https://github.com/krallin/tini) the entrypoint: `ENTRYPOINT ["tini", "--", "pilot-daemon"]`. The daemon logs a warning at startup when it is PID 1. It does not reap orphans itself: doing that in-process would also take the exit status of the apps it supervises. - **Proxy with credentials:** keep them out of `ps` — export `HTTPS_PROXY` / `PILOT_PROXY`, or `pilotctl daemon start --proxy http://user:pass@host:port` (pilotctl hands a URL with credentials to the daemon in its environment, never on its command line). For a systemd/launchd service, which does not see your shell's variables, save it: `pilotctl config --set proxy=http://user:pass@host:port` (config.json is 0600). - **Proxy credentials that rotate** (Meta Muse rotates the credentials in `HTTPS_PROXY` every few minutes): a long-running daemon would keep the ones it started with, and new connections would start failing with `407 Proxy Authentication Required` while old tunnels stay up ("node online, apps broken"). Give the daemon a command that prints the current proxy URL — `proxy_cmd` in config.json, `$PILOT_PROXY_CMD`, `-proxy-cmd` or `pilotctl daemon start --proxy-cmd` — and it re-runs it once 60s have passed and whenever the proxy rejects the credentials (a 407, or Muse's garbled "malformed HTTP status code" answer), retrying that connection once with the fresh credentials (registry, compat beacon and HTTP clients alike, via common's `netproxy` refresh); no restart needed. The apps the daemon starts reach the proxy through a loopback relay in the daemon that adds the current credentials, so they keep working across rotations too. pilotctl's own registry commands use the same command. In a Linux container/VM without systemd whose `HTTPS_PROXY` or `https_proxy` carries credentials, `pilotctl daemon start` hands the daemon a `PILOT_PROXY_CMD` that prints a fresh bash's `$https_proxy` (or `$HTTPS_PROXY` when only that one carries credentials) whenever no `proxy_cmd` is configured, and the installer saves the same command in config.json; elsewhere: `pilotctl config --set proxy_cmd="bash -c 'printf %s \"\$https_proxy\"'"`. Without it, restart the daemon from a fresh shell when it starts failing. - **Precedence** for transport and proxy: command-line flag, then `$PILOT_TRANSPORT` / `$PILOT_PROXY` / `$PILOT_PROXY_CMD`, then `config.json` (`transport`, `proxy`, `proxy_cmd`), then the default (`auto` from pilotctl and the installed systemd/launchd services, via `PILOT_TRANSPORT_DEFAULT=auto`; `udp` for a bare `pilot-daemon`; proxy `auto`). `auto` is never saved in config.json, so reinstalling an older release (`--version`, `pilotctl update --pin`) leaves nothing it cannot read; if you saved it yourself, both rewrite it to `udp` for a daemon that predates it. `-proxy` accepts `auto`, `off` (also `none`, `direct`) or an `http://` / `https://` URL; anything else is an error. Loopback targets (local webhooks, sidecars) are never sent to a proxy. When the registry dial goes through a proxy (compat, or an explicit proxy URL in any transport), the default raw-TCP registry is replaced by `registry.pilotprotocol.network:443` over TLS, since CONNECT proxies carry port 443 only. diff --git a/cmd/daemon/main.go b/cmd/daemon/main.go index a1452089..30600046 100644 --- a/cmd/daemon/main.go +++ b/cmd/daemon/main.go @@ -324,6 +324,7 @@ func main() { installDefaultTransportProxy(proxyResolver, proxyRelay) slog.Info("outbound network", "transport", *transportMode, "proxy", describeProxy(*proxySpec, *transportMode, proxyResolver), "transport_from", transportSrc, "registry", *registryAddr, "registry_tls", *registryTLS) + warnIfPID1(slog.Default()) // Sandbox: validate all configured file paths are under the confinement // root before the daemon touches the filesystem. Network paths are unaffected. diff --git a/cmd/daemon/pid1.go b/cmd/daemon/pid1.go new file mode 100644 index 00000000..a3fa3c85 --- /dev/null +++ b/cmd/daemon/pid1.go @@ -0,0 +1,40 @@ +// SPDX-License-Identifier: AGPL-3.0-or-later + +package main + +import ( + "log/slog" + "os" + "runtime" +) + +// A daemon that is PID 1 (the entrypoint of a container started without an +// init) inherits every orphaned process in the container. Apps start servers +// that daemonize (redis-server, postgres); when one of those exits, it stays +// a zombie until PID 1 waits for it, and Go only waits for the children it +// started itself. The daemon does not reap them: a wait for "any child" also +// takes the exit status of the apps the supervisor is waiting on (os/exec +// then fails with "waitid: no child processes"). Reaping is the job of an +// init process, so the daemon only says so, once, at startup. + +// pid1Warning returns the startup warning for a daemon running as PID 1 on +// Linux, and "" for any other process. +func pid1Warning(goos string, pid int) string { + if goos != "linux" || pid != 1 { + return "" + } + return "the daemon is PID 1: processes that its apps start and leave behind (database servers that daemonize) " + + "will not be reaped and stay as zombies after they exit. Run the daemon under an init process: " + + "`docker run --init`, or tini as the container's entrypoint" +} + +// warnIfPID1 logs pid1Warning for this process. +func warnIfPID1(logger *slog.Logger) { + logPID1Warning(logger, runtime.GOOS, os.Getpid()) +} + +func logPID1Warning(logger *slog.Logger, goos string, pid int) { + if msg := pid1Warning(goos, pid); msg != "" { + logger.Warn(msg) + } +} diff --git a/cmd/daemon/zz_pid1_test.go b/cmd/daemon/zz_pid1_test.go new file mode 100644 index 00000000..cd6112b1 --- /dev/null +++ b/cmd/daemon/zz_pid1_test.go @@ -0,0 +1,57 @@ +// SPDX-License-Identifier: AGPL-3.0-or-later + +package main + +import ( + "bytes" + "log/slog" + "os" + "strings" + "testing" +) + +// The PID 1 warning is for a Linux daemon that is the container's init, and +// for nothing else. +func TestPID1WarningOnlyForLinuxPID1(t *testing.T) { + for _, tc := range []struct { + goos string + pid int + want bool + }{ + {"linux", 1, true}, + {"linux", 2, false}, + {"linux", 7, false}, // what `docker run --init` gives the daemon + {"linux", 0, false}, + {"darwin", 1, false}, + {"windows", 1, false}, + } { + var buf bytes.Buffer + logPID1Warning(slog.New(slog.NewTextHandler(&buf, nil)), tc.goos, tc.pid) + got := buf.String() + if tc.want { + if strings.Count(got, "level=WARN") != 1 { + t.Errorf("%s pid %d: want exactly one warning, got %q", tc.goos, tc.pid, got) + } + for _, part := range []string{"PID 1", "not be reaped", "docker run --init", "tini"} { + if !strings.Contains(got, part) { + t.Errorf("%s pid %d: warning does not mention %q: %q", tc.goos, tc.pid, part, got) + } + } + } else if got != "" { + t.Errorf("%s pid %d: want no warning, got %q", tc.goos, tc.pid, got) + } + } +} + +// The test process is never PID 1 outside a container that runs `go test` as +// its entrypoint, so the real check stays silent here. +func TestWarnIfPID1SilentForOrdinaryProcess(t *testing.T) { + if os.Getpid() == 1 { + t.Skip("running as PID 1") + } + var buf bytes.Buffer + warnIfPID1(slog.New(slog.NewTextHandler(&buf, nil))) + if buf.Len() != 0 { + t.Fatalf("pid %d logged %q", os.Getpid(), buf.String()) + } +} From 090e8ff9814ba85539aaad5f9a30d113926725b7 Mon Sep 17 00:00:00 2001 From: Teo Calin Date: Wed, 7 Oct 2026 16:09:12 +0300 Subject: [PATCH 2/2] daemon: PID 1 warning names the Kubernetes and ECS options; README and changelog wording From review: Kubernetes and Fargate have no --init, so name shareProcessNamespace and initProcessEnabled; say zombies count against the container's PID limit and that no warning does not prove another PID 1 reaps; fix the changelog's tense and a test comment. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 9 +++++---- README.md | 2 +- cmd/daemon/pid1.go | 3 ++- cmd/daemon/zz_pid1_test.go | 2 +- 4 files changed, 9 insertions(+), 7 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 0437dc9b..88f8f37c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -217,11 +217,12 @@ Detailed per-release notes are on the sandboxes run the agent as root); regular hosts still refuse root. - **The daemon warns when it is PID 1.** As the entrypoint of a container started without an init, the daemon is handed every orphaned process, and - the servers its apps start (redis, postgres) stayed as zombies after they - were stopped. The daemon does not reap them — that would also take the exit + the servers its apps start (redis, postgres) stay as zombies after they + are stopped. The daemon does not reap them — that would also take the exit status of the apps it supervises — and now logs one warning at startup, on - Linux, saying to run it under an init (`docker run --init`, or tini as the - entrypoint). README: Install. + Linux, saying to run it under an init (`docker run --init`, tini as the + entrypoint, Kubernetes `shareProcessNamespace`, ECS `initProcessEnabled`). + README: Install. ### Removed - **Hosted control plane client.** The hosted control plane has been retired, diff --git a/README.md b/README.md index 36d4ca2d..61d16296 100644 --- a/README.md +++ b/README.md @@ -297,7 +297,7 @@ pilotctl daemon start New installs use transport `auto`: the daemon probes the beacon over UDP once at startup (one round trip when UDP works, at most ~1.5s when it does not) and, when there is no answer but the compat beacon answers over TCP 443 (a TLS request that a live beacon answers with `426 Upgrade Required`), runs in **compat** mode — registry over TLS on `registry.pilotprotocol.network:443`, beacon over WSS on `beacon.pilotprotocol.network:443`. When a proxy is configured and it refuses that check (wrong or stale credentials: `407`, a `403`, or the proxy cannot be reached), it runs compat anyway rather than falling back to direct connections past the proxy, and the registry dial then names the proxy's answer; `pilotctl daemon start` prints the daemon's last proxy error with a hint instead of only "did not become ready". When neither answers and no proxy is involved (no network yet, beacon outage) it stays on UDP, as before. In compat mode the daemon tunnels through `$HTTPS_PROXY` / `$ALL_PROXY` (honoring `$NO_PROXY`), asking the proxy to `CONNECT` by host name, so poisoned local DNS for the Pilot hosts does not matter. `pilotctl`'s own registry commands (`lookup`, the auto-handshake check, `recovery`) follow the daemon: through the proxy when it runs compat (or with an explicit proxy URL), directly when it runs UDP; with no daemon running and no transport configured they try the proxy first and a direct connection second. To skip the UDP probe, pick compat explicitly: `sh -s -- --transport compat`, `pilotctl config --set transport=compat`, or `pilotctl daemon start --transport compat`. - **No root, systemd or launchd needed.** Start the daemon with `pilotctl daemon start` from a shell that has the proxy variables. In a container or VM without systemd the installer also runs as root (the agent user in hosted sandboxes); on a regular host it still refuses root unless `PILOT_ALLOW_ROOT=1`. -- **In a container, give the daemon an init process.** Apps start servers that daemonize (redis, postgres); when one of those stops, the kernel hands it to PID 1 to be reaped, and a `pilot-daemon` that is PID 1 itself does not do that, so each one stays behind as a zombie (`Z` in `ps`). Run the container with `docker run --init` (Compose: `init: true`), or make [tini](https://github.com/krallin/tini) the entrypoint: `ENTRYPOINT ["tini", "--", "pilot-daemon"]`. The daemon logs a warning at startup when it is PID 1. It does not reap orphans itself: doing that in-process would also take the exit status of the apps it supervises. +- **In a container, give the daemon an init process.** Apps start servers that daemonize (redis, postgres); a daemonized server is handed to PID 1, and when it stops it stays a zombie (`Z` in `ps`) until PID 1 reaps it. A `pilot-daemon` that is PID 1 itself does not, so the zombies accumulate and count against the container's PID limit. Run the container with `docker run --init` (Compose: `init: true`), make [tini](https://github.com/krallin/tini) the entrypoint (`ENTRYPOINT ["tini", "--", "pilot-daemon"]`), or on Kubernetes set `shareProcessNamespace: true` (ECS: `initProcessEnabled`). The daemon logs a warning at startup when it is PID 1; no warning does not prove all is well if some other process without reaping, such as a shell script, is PID 1. It does not reap orphans itself: doing that in-process would also take the exit status of the apps it supervises. - **Proxy with credentials:** keep them out of `ps` — export `HTTPS_PROXY` / `PILOT_PROXY`, or `pilotctl daemon start --proxy http://user:pass@host:port` (pilotctl hands a URL with credentials to the daemon in its environment, never on its command line). For a systemd/launchd service, which does not see your shell's variables, save it: `pilotctl config --set proxy=http://user:pass@host:port` (config.json is 0600). - **Proxy credentials that rotate** (Meta Muse rotates the credentials in `HTTPS_PROXY` every few minutes): a long-running daemon would keep the ones it started with, and new connections would start failing with `407 Proxy Authentication Required` while old tunnels stay up ("node online, apps broken"). Give the daemon a command that prints the current proxy URL — `proxy_cmd` in config.json, `$PILOT_PROXY_CMD`, `-proxy-cmd` or `pilotctl daemon start --proxy-cmd` — and it re-runs it once 60s have passed and whenever the proxy rejects the credentials (a 407, or Muse's garbled "malformed HTTP status code" answer), retrying that connection once with the fresh credentials (registry, compat beacon and HTTP clients alike, via common's `netproxy` refresh); no restart needed. The apps the daemon starts reach the proxy through a loopback relay in the daemon that adds the current credentials, so they keep working across rotations too. pilotctl's own registry commands use the same command. In a Linux container/VM without systemd whose `HTTPS_PROXY` or `https_proxy` carries credentials, `pilotctl daemon start` hands the daemon a `PILOT_PROXY_CMD` that prints a fresh bash's `$https_proxy` (or `$HTTPS_PROXY` when only that one carries credentials) whenever no `proxy_cmd` is configured, and the installer saves the same command in config.json; elsewhere: `pilotctl config --set proxy_cmd="bash -c 'printf %s \"\$https_proxy\"'"`. Without it, restart the daemon from a fresh shell when it starts failing. - **Precedence** for transport and proxy: command-line flag, then `$PILOT_TRANSPORT` / `$PILOT_PROXY` / `$PILOT_PROXY_CMD`, then `config.json` (`transport`, `proxy`, `proxy_cmd`), then the default (`auto` from pilotctl and the installed systemd/launchd services, via `PILOT_TRANSPORT_DEFAULT=auto`; `udp` for a bare `pilot-daemon`; proxy `auto`). `auto` is never saved in config.json, so reinstalling an older release (`--version`, `pilotctl update --pin`) leaves nothing it cannot read; if you saved it yourself, both rewrite it to `udp` for a daemon that predates it. `-proxy` accepts `auto`, `off` (also `none`, `direct`) or an `http://` / `https://` URL; anything else is an error. Loopback targets (local webhooks, sidecars) are never sent to a proxy. When the registry dial goes through a proxy (compat, or an explicit proxy URL in any transport), the default raw-TCP registry is replaced by `registry.pilotprotocol.network:443` over TLS, since CONNECT proxies carry port 443 only. diff --git a/cmd/daemon/pid1.go b/cmd/daemon/pid1.go index a3fa3c85..f5196add 100644 --- a/cmd/daemon/pid1.go +++ b/cmd/daemon/pid1.go @@ -25,7 +25,8 @@ func pid1Warning(goos string, pid int) string { } return "the daemon is PID 1: processes that its apps start and leave behind (database servers that daemonize) " + "will not be reaped and stay as zombies after they exit. Run the daemon under an init process: " + - "`docker run --init`, or tini as the container's entrypoint" + "`docker run --init`, tini as the container's entrypoint, Kubernetes `shareProcessNamespace: true`, " + + "or ECS `initProcessEnabled`" } // warnIfPID1 logs pid1Warning for this process. diff --git a/cmd/daemon/zz_pid1_test.go b/cmd/daemon/zz_pid1_test.go index cd6112b1..e49cec48 100644 --- a/cmd/daemon/zz_pid1_test.go +++ b/cmd/daemon/zz_pid1_test.go @@ -20,7 +20,7 @@ func TestPID1WarningOnlyForLinuxPID1(t *testing.T) { }{ {"linux", 1, true}, {"linux", 2, false}, - {"linux", 7, false}, // what `docker run --init` gives the daemon + {"linux", 7, false}, // under `docker run --init`: any PID other than 1 {"linux", 0, false}, {"darwin", 1, false}, {"windows", 1, false},