ateapi: give the api-server a startup probe covering its store-connect budget

[Maya Wang (mayawang)]

Partially addresses #1394 — the "spend the retry budget it configures for itself" half of that issue's expected behavior. Candidate fix 4 from the issue, done first because it is manifest-only.

The problem, in one line

ate-api-server configures itself a 60s store-connect budget and is killed by its own liveness probe at ~30s, halfway through it.

Why the kill happens

:9090 is bound near the end of boot — after connectStore and, since #1196, after the goose migrations — so nothing answers /healthz for the whole of that window. The liveness probe was the only probe governing it: initialDelaySeconds: 10 then probes at 10s, 20s and 30s, so the default failureThreshold: 3 is met at ~30s. The events show connect: connection refused, not a probe returning failure — the port is closed, the process is fine.

The kill then makes it worse than a plain restart. connectStore(shutdownCtx) roots the retry loop in the signal context, so SIGTERM cancels it mid-budget: the loop reports context canceled at attempt 15 of 30 rather than exhausting the budget, serverboot.Fatal exits, and CrashLoopBackOff holds the pod down well past the point where the store came back.

A running replica rides out the same outage with restarts=0 and serves again ~15s after the store returns. A booting replica was strictly less resilient than a running one, which is backwards.

The change

One startupProbe, 24 × 5s. Liveness is suspended until /healthz answers, so boot gets ~115s before the kubelet intervenes, against a ~60s connect budget.

	before	after
kubelet kills at	~30s	~115s
connect attempts used	15 of 30	30 of 30
failure reported as	context canceled	connect to PostgreSQL after 30 attempts: …

A dead process is still restarted, just on a budget sized for boot rather than one that never was. And because a startup probe also suspends the readiness probe, the pod now stays out of Service endpoints while booting instead of being latched Ready by the zero-value Readiness — a small move in the right direction for #1395 rather than against it, which is why this rather than simply raising failureThreshold.

What this deliberately does not do

• It does not fix ateapi: ate-api-server is SIGTERMed by its own liveness probe halfway through its store-connect retry budget, then exits instead of retrying #1394. The process still Fatals once its own 60s budget is gone. The motivating scenario there is an ungraceful node loss, where GKE's PD force-detach alone is ~6 minutes; this raises tolerance from 30s to 60s, and only "retry for the life of the process, report through readiness" survives six minutes. That fix needs the reversible readiness predicate ateapi: a control plane that cannot reach its store serves 200 on /healthz and /readyz, stays Ready, and fails every RPC #1395 describes.
• It does not bound the migrations. ateapi: add versioned PostgreSQL schema migrations #1196 put goose on the boot critical path behind a pg_advisory_xact_lock, so with two replicas restarting together one waits on the other — inside the window where nothing is listening. On an empty schema that is milliseconds; on a real database during an upgrade it is not bounded by anything, and no probe budget can bound it. Ordering can.
• It does not change the boot ordering. A slow boot and a dead process are still indistinguishable while the port is closed. Moving the health surface ahead of connectStore is candidate fix 1 and the natural follow-up.

Testing

hack/verify-all.sh passes (metrics.sh skipped locally — needs weaver or docker; no metrics change here).

Note for anyone rebasing: make test currently fails on TestExternalVolumeRenders in cmd/ate-setup/internal/demos/counter on a clean cbae8250, unrelated to this change.

[Maya Wang (mayawang)]

Jeremy Alvis (@iplay88keys) tagging you for context rather than as a formal reviewer — GitHub won't let me request you. This sits directly on top of #1196: the "can't bound the migrations" caveat is about the advisory lock you put on the boot path, and I'd like a sanity check that ~115s is the right budget now that goose runs ahead of the health server.