daemon: warn at startup when running as PID 1

CHANGELOG.md

@@ -215,6 +215,14 @@ Detailed per-release notes are on the
   (redacted) and the hosts it must allow. In a Linux container/VM without
   systemd the installer runs as root without `PILOT_ALLOW_ROOT` (hosted
   sandboxes run the agent as root); regular hosts still refuse root.
+- **The daemon warns when it is PID 1.** As the entrypoint of a container
+  started without an init, the daemon is handed every orphaned process, and
+  the servers its apps start (redis, postgres) stay as zombies after they
+  are stopped. The daemon does not reap them — that would also take the exit
+  status of the apps it supervises — and now logs one warning at startup, on
+  Linux, saying to run it under an init (`docker run --init`, tini as the
+  entrypoint, Kubernetes `shareProcessNamespace`, ECS `initProcessEnabled`).
+  README: Install.
 
 ### Removed
 - **Hosted control plane client.** The hosted control plane has been retired,

README.md

@@ -297,6 +297,7 @@ pilotctl daemon start
 New installs use transport `auto`: the daemon probes the beacon over UDP once at startup (one round trip when UDP works, at most ~1.5s when it does not) and, when there is no answer but the compat beacon answers over TCP 443 (a TLS request that a live beacon answers with `426 Upgrade Required`), runs in **compat** mode — registry over TLS on `registry.pilotprotocol.network:443`, beacon over WSS on `beacon.pilotprotocol.network:443`. When a proxy is configured and it refuses that check (wrong or stale credentials: `407`, a `403`, or the proxy cannot be reached), it runs compat anyway rather than falling back to direct connections past the proxy, and the registry dial then names the proxy's answer; `pilotctl daemon start` prints the daemon's last proxy error with a hint instead of only "did not become ready". When neither answers and no proxy is involved (no network yet, beacon outage) it stays on UDP, as before. In compat mode the daemon tunnels through `$HTTPS_PROXY` / `$ALL_PROXY` (honoring `$NO_PROXY`), asking the proxy to `CONNECT` by host name, so poisoned local DNS for the Pilot hosts does not matter. `pilotctl`'s own registry commands (`lookup`, the auto-handshake check, `recovery`) follow the daemon: through the proxy when it runs compat (or with an explicit proxy URL), directly when it runs UDP; with no daemon running and no transport configured they try the proxy first and a direct connection second. To skip the UDP probe, pick compat explicitly: `sh -s -- --transport compat`, `pilotctl config --set transport=compat`, or `pilotctl daemon start --transport compat`.
 
 - **No root, systemd or launchd needed.** Start the daemon with `pilotctl daemon start` from a shell that has the proxy variables. In a container or VM without systemd the installer also runs as root (the agent user in hosted sandboxes); on a regular host it still refuses root unless `PILOT_ALLOW_ROOT=1`.
+- **In a container, give the daemon an init process.** Apps start servers that daemonize (redis, postgres); a daemonized server is handed to PID 1, and when it stops it stays a zombie (`Z` in `ps`) until PID 1 reaps it. A `pilot-daemon` that is PID 1 itself does not, so the zombies accumulate and count against the container's PID limit. Run the container with `docker run --init` (Compose: `init: true`), make [tini](https://github.com/krallin/tini) the entrypoint (`ENTRYPOINT ["tini", "--", "pilot-daemon"]`), or on Kubernetes set `shareProcessNamespace: true` (ECS: `initProcessEnabled`). The daemon logs a warning at startup when it is PID 1; no warning does not prove all is well if some other process without reaping, such as a shell script, is PID 1. It does not reap orphans itself: doing that in-process would also take the exit status of the apps it supervises.
 - **Proxy with credentials:** keep them out of `ps` — export `HTTPS_PROXY` / `PILOT_PROXY`, or `pilotctl daemon start --proxy http://user:pass@host:port` (pilotctl hands a URL with credentials to the daemon in its environment, never on its command line). For a systemd/launchd service, which does not see your shell's variables, save it: `pilotctl config --set proxy=http://user:pass@host:port` (config.json is 0600).
 - **Proxy credentials that rotate** (Meta Muse rotates the credentials in `HTTPS_PROXY` every few minutes): a long-running daemon would keep the ones it started with, and new connections would start failing with `407 Proxy Authentication Required` while old tunnels stay up ("node online, apps broken"). Give the daemon a command that prints the current proxy URL — `proxy_cmd` in config.json, `$PILOT_PROXY_CMD`, `-proxy-cmd` or `pilotctl daemon start --proxy-cmd` — and it re-runs it once 60s have passed and whenever the proxy rejects the credentials (a 407, or Muse's garbled "malformed HTTP status code" answer), retrying that connection once with the fresh credentials (registry, compat beacon and HTTP clients alike, via common's `netproxy` refresh); no restart needed. The apps the daemon starts reach the proxy through a loopback relay in the daemon that adds the current credentials, so they keep working across rotations too. pilotctl's own registry commands use the same command. In a Linux container/VM without systemd whose `HTTPS_PROXY` or `https_proxy` carries credentials, `pilotctl daemon start` hands the daemon a `PILOT_PROXY_CMD` that prints a fresh bash's `$https_proxy` (or `$HTTPS_PROXY` when only that one carries credentials) whenever no `proxy_cmd` is configured, and the installer saves the same command in config.json; elsewhere: `pilotctl config --set proxy_cmd="bash -c 'printf %s \"\$https_proxy\"'"`. Without it, restart the daemon from a fresh shell when it starts failing.
 - **Precedence** for transport and proxy: command-line flag, then `$PILOT_TRANSPORT` / `$PILOT_PROXY` / `$PILOT_PROXY_CMD`, then `config.json` (`transport`, `proxy`, `proxy_cmd`), then the default (`auto` from pilotctl and the installed systemd/launchd services, via `PILOT_TRANSPORT_DEFAULT=auto`; `udp` for a bare `pilot-daemon`; proxy `auto`). `auto` is never saved in config.json, so reinstalling an older release (`--version`, `pilotctl update --pin`) leaves nothing it cannot read; if you saved it yourself, both rewrite it to `udp` for a daemon that predates it. `-proxy` accepts `auto`, `off` (also `none`, `direct`) or an `http://` / `https://` URL; anything else is an error. Loopback targets (local webhooks, sidecars) are never sent to a proxy. When the registry dial goes through a proxy (compat, or an explicit proxy URL in any transport), the default raw-TCP registry is replaced by `registry.pilotprotocol.network:443` over TLS, since CONNECT proxies carry port 443 only.

cmd/daemon/main.go

@@ -324,6 +324,7 @@ func main() {
 	installDefaultTransportProxy(proxyResolver, proxyRelay)
 	slog.Info("outbound network", "transport", *transportMode, "proxy", describeProxy(*proxySpec, *transportMode, proxyResolver),
 		"transport_from", transportSrc, "registry", *registryAddr, "registry_tls", *registryTLS)
+	warnIfPID1(slog.Default())
 
 	// Sandbox: validate all configured file paths are under the confinement
 	// root before the daemon touches the filesystem. Network paths are unaffected.

cmd/daemon/pid1.go

@@ -0,0 +1,41 @@
+// SPDX-License-Identifier: AGPL-3.0-or-later
+
+package main
+
+import (
+	"log/slog"
+	"os"
+	"runtime"
+)
+
+// A daemon that is PID 1 (the entrypoint of a container started without an
+// init) inherits every orphaned process in the container. Apps start servers
+// that daemonize (redis-server, postgres); when one of those exits, it stays
+// a zombie until PID 1 waits for it, and Go only waits for the children it
+// started itself. The daemon does not reap them: a wait for "any child" also
+// takes the exit status of the apps the supervisor is waiting on (os/exec
+// then fails with "waitid: no child processes"). Reaping is the job of an
+// init process, so the daemon only says so, once, at startup.
+
+// pid1Warning returns the startup warning for a daemon running as PID 1 on
+// Linux, and "" for any other process.
+func pid1Warning(goos string, pid int) string {
+	if goos != "linux" || pid != 1 {
+		return ""
+	}
+	return "the daemon is PID 1: processes that its apps start and leave behind (database servers that daemonize) " +
+		"will not be reaped and stay as zombies after they exit. Run the daemon under an init process: " +
+		"`docker run --init`, tini as the container's entrypoint, Kubernetes `shareProcessNamespace: true`, " +
+		"or ECS `initProcessEnabled`"
+}
+
+// warnIfPID1 logs pid1Warning for this process.
+func warnIfPID1(logger *slog.Logger) {
+	logPID1Warning(logger, runtime.GOOS, os.Getpid())
+}
+
+func logPID1Warning(logger *slog.Logger, goos string, pid int) {
+	if msg := pid1Warning(goos, pid); msg != "" {
+		logger.Warn(msg)
+	}
+}

cmd/daemon/zz_pid1_test.go

@@ -0,0 +1,57 @@
+// SPDX-License-Identifier: AGPL-3.0-or-later
+
+package main
+
+import (
+	"bytes"
+	"log/slog"
+	"os"
+	"strings"
+	"testing"
+)
+
+// The PID 1 warning is for a Linux daemon that is the container's init, and
+// for nothing else.
+func TestPID1WarningOnlyForLinuxPID1(t *testing.T) {
+	for _, tc := range []struct {
+		goos string
+		pid  int
+		want bool
+	}{
+		{"linux", 1, true},
+		{"linux", 2, false},
+		{"linux", 7, false}, // under `docker run --init`: any PID other than 1
+		{"linux", 0, false},
+		{"darwin", 1, false},
+		{"windows", 1, false},
+	} {
+		var buf bytes.Buffer
+		logPID1Warning(slog.New(slog.NewTextHandler(&buf, nil)), tc.goos, tc.pid)
+		got := buf.String()
+		if tc.want {
+			if strings.Count(got, "level=WARN") != 1 {
+				t.Errorf("%s pid %d: want exactly one warning, got %q", tc.goos, tc.pid, got)
+			}
+			for _, part := range []string{"PID 1", "not be reaped", "docker run --init", "tini"} {
+				if !strings.Contains(got, part) {
+					t.Errorf("%s pid %d: warning does not mention %q: %q", tc.goos, tc.pid, part, got)
+				}
+			}
+		} else if got != "" {
+			t.Errorf("%s pid %d: want no warning, got %q", tc.goos, tc.pid, got)
+		}
+	}
+}
+
+// The test process is never PID 1 outside a container that runs `go test` as
+// its entrypoint, so the real check stays silent here.
+func TestWarnIfPID1SilentForOrdinaryProcess(t *testing.T) {
+	if os.Getpid() == 1 {
+		t.Skip("running as PID 1")
+	}
+	var buf bytes.Buffer
+	warnIfPID1(slog.New(slog.NewTextHandler(&buf, nil)))
+	if buf.Len() != 0 {
+		t.Fatalf("pid %d logged %q", os.Getpid(), buf.String())
+	}
+}