ateapi: ate-api-server is SIGTERMed by its own liveness probe halfway through its store-connect retry budget, then exits instead of retrying

[mayawang]

Expected Behavior

A control plane whose store is temporarily unreachable at startup should do one of two things:

1. Come up, serve its health surface, report not ready, and keep retrying for the life of the
   process. A control plane that cannot reach its store should stay up and say so.
2. At an absolute minimum, be allowed to spend the retry budget it configures for itself.
   postgresConnectTries = 30 at postgresConnectPeriod = 2s is a deliberate 60-second budget.

Either way, a store outage shorter than the budget should not produce a restart.

Actual Behavior

Neither. The process gets ~30 seconds of its 60-second budget, is killed by its own liveness
probe, exits, and enters CrashLoopBackOff — where exponential backoff then keeps it down well past
the point where the store has come back.

Measured on a Postgres install on GKE, store scaled to zero, one replica deleted to force it onto
the boot path:

19:31:06.317  attempt 1  ... lookup postgres.ate-system.svc: no such host
19:31:34.396  attempt 15 ... lookup postgres.ate-system.svc: no such host
19:31:36.192  ERROR Failed to set up persistence backend
              err="setting up PostgreSQL: context canceled"

15 of 30 attempts, and the loop ends in context canceled rather than budget exhaustion. Pod
events:

Liveness probe failed: Get "http://10.x.x.x:9090/healthz": connect: connection refused
Killing — Container ate-api-server failed liveness probe, will be restarted   ×6

Result: CrashLoopBackOff, 7 restarts in 7 minutes, each one Failed to set up persistence backend.

Two properties of this that are worth stating plainly, because both are counter-intuitive:

• The kill is not caused by an unhealthy process. It is caused by there being nothing
  listening on :9090 yet. /healthz is not returning an error; the port is closed.
• The fatal is boot-only. A running replica, on the same cluster during the same outage, rode
  out six minutes with restarts=0 and was serving again 15 s after the store returned. The
  pgx pool simply starts working. So the dangerous event is precisely the one that puts every
  replica on the boot path at once — a node replacement.

Steps to Reproduce the Problem

1. Install on GKE with the PostgreSQL backend (--store-backend=postgres). Confirm both
   ate-api-server replicas are Ready and serving.
2. Take the store away: kubectl -n ate-system scale sts/postgres --replicas=0. Note that nothing
   happens — the running replicas stay Ready=True with restarts=0 and no probe fails. (That is
   the companion defect, #.)
3. Force one replica onto the boot path while the store is down:
   kubectl -n ate-system delete pod <one ate-api-server pod>. This is what a node replacement does
   to every replica simultaneously.
4. Watch the replacement pod: kubectl -n ate-system get pod -w and
   kubectl -n ate-system describe pod <new pod>. It reaches CrashLoopBackOff within ~90 s.
   The events show Liveness probe failed: ... 9090/healthz: connect: connection refused, not a
   probe returning a failure status.
5. Read the container log. Count the Failed to connect to PostgreSQL, retrying... lines: there are
   15, not 30, and the loop terminates with setting up PostgreSQL: context canceled.
6. Bring the store back: kubectl -n ate-system scale sts/postgres --replicas=1. The surviving
   replica serves again within ~15 s with restarts=0. The crash-looping replica comes Ready
   only when its own CrashLoopBackOff timer next fires — for us, at the same second Postgres became
   available, after 7 restarts, but by then the backoff had already reached the tens of seconds and
   is capped at 5 min.

Why CI does not catch it: e2e installs bring Postgres up before or alongside the api-server and
never hold it down across a pod start, so the retry loop is never exercised under a probe.

Specifications

• Version: 4c1b37d (the base of release-0.1-rc) plus three cherry-picks. Confirmed still
  present on main at ea3bdc32 (2026-09-02) by reading the code — see line references below.
  Store-agnostic: this survived the valkey → Postgres migration (60073ecd, Replace ateredis with atepg #940) untouched, and
  the same mechanism produced a much longer outage on a valkey install on 27 Aug.
• Platform: GKE, us-central1-c, cluster provisioned by setup-gcp bootstrap,
  --store-backend=postgres, postgres-0 2/2 with a Bound 500Gi standard-rwo PVC. Reproduced
  2026-08-27.
• Evidence: container logs and pod events from the run above.

---

Root cause

Three things on main at ea3bdc32 that are individually reasonable and jointly fatal.

1. The health surface starts after the store connect. cmd/ateapi/main.go:

• :127 — persistence, err := connectStore(shutdownCtx)
• :129 — serverboot.Fatal(ctx, "Failed to set up persistence backend", err)
• :236 — go serverboot.StartMetricsServer(ctx, serverboot.MetricsServerOptions{...}) with
  EnableHealthz: true at :239

So :9090 binds roughly 110 lines of boot later than the retry loop begins. During the entire retry
window the port is closed.

2. The liveness probe's budget is shorter than the retry budget it is racing.
manifests/ate-install/ate-api-server.yaml sets livenessProbe on /healthz:9090 with
initialDelaySeconds: 10, periodSeconds: 10, and no failureThreshold — so the default 3
applies. First SIGTERM lands at ~30 s from container start. The retry loop wants 60 s.

3. The retry loop is rooted in the shutdown context, so the SIGTERM cancels it.
connectStore(shutdownCtx), where shutdownCtx comes from signal.NotifyContext(ctx, SIGTERM, os.Interrupt). The select in connectPostgresWithRetries (:330) returns ctx.Err() on
cancellation. Hence context canceled at attempt 15 rather than
connect to PostgreSQL after 30 attempts.

Then serverboot.Fatal at :129 exits, the kubelet restarts with backoff (10s, 20s, 40s, 80s,
160s, capped at 300s), and each restart repeats the whole thing.

Why this is worse on main than it was

On the valkey install this was one contributor to a long outage. On main it is the primary
failure path for an ordinary GKE event, because the store got less available at the same time as
this code stayed the same:

manifests/ate-install/postgres.yaml is replicas: 1 on an RWO PVC, with no
PodDisruptionBudget, no topologySpreadConstraints and no node affinity — while ate-api-server
has a PDB and two replicas. #1253 raised the claim to 500Gi and 2–16 CPU, which makes the pod harder
to place on a replacement node. After an ungraceful node loss, GKE's force-detach of the PD alone is
~6 minutes before the volume can attach elsewhere, then reschedule and WAL replay.

Six minutes against a thirty-second window. A node-pool upgrade — on by default in every GKE release
channel — therefore restarts every ate-api-server replica at the exact moment the store is least
available, and every one of them lands on this path. Whole-pool node replacement under load has not
been tested against the Postgres backend, so the true window length is unknown; it is not plausibly
under 30 s.

Interaction with #1196, which is landing on this function now

#1196 (ateapi: add versioned PostgreSQL schema migrations) edits connectPostgresWithRetries
directly. It makes one genuine improvement and one regression to this issue:

• Better: it adds atepg.ErrUnavailable and returns immediately on any error that is not it, so
  a bad DSN or a failed migration fails in one attempt instead of burning the budget. That is the
  right change — but it is the other branch. The store-unavailable path keeps all 30 attempts, the
  same ordering, and the same 40-second ceiling.
• Worse: it moves goose migrations onto the boot critical path, ahead of the health server,
  behind a pg_advisory_xact_lock. Boot gets longer and gains a serialization point: with two
  replicas restarting together, one holds the lock while the other waits — both inside the window
  where nothing is listening on :9090. On an empty schema that is milliseconds; on a real database
  during an upgrade it is not bounded by anything.

The ordering fix should therefore be sequenced with #1196, not against today's main — the
health server has to start before connectStore and before the migration step. It is a handful of
lines in a function that PR already touches.

Candidate fixes

1. Start the health surface before connectStore. Move the StartMetricsServer goroutine (and
   its Readiness) above the store connect, so /healthz answers from the first second of the
   process. This alone converts the failure from "killed at 15 of 30 attempts" to "uses its full
   budget", and it is the two-line version.
2. Do not root the connect retry in the shutdown context — or, better, keep it rooted there but
   let the process survive a store that is merely slow, per fix 3.
3. Retry for the life of the process instead of exiting. Replace serverboot.Fatal on store
   failure with a background reconnect loop, and report the condition through readiness (see
   ateapi: a control plane that cannot reach its store serves 200 on /healthz and /readyz, stays Ready, and fails every RPC #1395 , which needs a reversible readiness predicate for exactly this). This is the fix that
   matches the observed behaviour of a running replica, which already rides out arbitrary
   outages — a booting one should not be strictly less resilient than a running one.
4. Set an explicit failureThreshold and a startupProbe on the manifest, so slow boots are
   distinguished from dead processes. Complementary, not a substitute: it changes the number without
   fixing the ordering.

1 and 3 are the two I would pick. 1 is safe enough to land on its own before the other.

One design constraint, because the obvious fix for #1395 breaks this one: liveness must not
be gated on store reachability. /healthz should mean "the process is alive", start before the store
connect, and never consult the store; /readyz is where store reachability belongs. Gating
/healthz on the store would make this issue strictly worse — the kubelet would then kill a pod that
is correctly waiting.

Related: #1395 (same health surface, opposite phase), #1196 (lands on this function), #636 (the
valkey instance of the same crash loop, closed by deleting the backend rather than by a fix).