From 941297cf80af60b4345738036e0811a6d772606f Mon Sep 17 00:00:00 2001 From: Maya Wang Date: Wed, 2 Sep 2026 11:53:03 -0700 Subject: [PATCH] ateapi: give the api-server a startup probe covering its store-connect budget The metrics server binds :9090 near the end of boot, after the store connect and the schema migrations, so nothing answers /healthz until both finish. The store connect budgets 60s of its own (30 attempts, 2s apart) and the migrations serialize across replicas on an advisory lock. The liveness probe was the only probe governing that window, and its budget is half as long: initialDelaySeconds 10 then probes at 10s, 20s and 30s, so the third failure kills the container at ~30s. The connect retry loop is rooted in the signal context, so the kill cancels it mid-budget -- the process reports "context canceled" at attempt 15 of 30 and exits into CrashLoopBackOff, where the backoff then holds it down well past the store returning. A running replica rides out the same outage untouched, so a booting one was strictly less resilient than a running one. A startup probe suspends liveness until :9090 answers, giving boot ~115s before the kubelet intervenes. A store slow to return is now waited out for the full configured budget, and exhausting it reports that honestly instead of as a cancellation; a genuinely dead process is still restarted, on a budget sized for boot rather than one that was not. This does not change the boot ordering, so a slow boot is still indistinguishable from a dead process while the port is closed, and an outage outlasting the process's own 60s budget still ends in a restart. Starting the health surface ahead of the store connect, and retrying for the life of the process, are the follow-ups. --- manifests/ate-install/ate-api-server.yaml | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/manifests/ate-install/ate-api-server.yaml b/manifests/ate-install/ate-api-server.yaml index e098bf9b5..aa4db3660 100644 --- a/manifests/ate-install/ate-api-server.yaml +++ b/manifests/ate-install/ate-api-server.yaml @@ -160,6 +160,21 @@ spec: initialDelaySeconds: 5 periodSeconds: 2 failureThreshold: 3 + # :9090 is bound near the end of boot, after the store connect and + # the schema migrations, so nothing answers here until both finish. + # The store connect alone budgets 60s (30 attempts, 2s apart) and the + # migrations serialize across replicas on an advisory lock. Without a + # startup probe the liveness probe below SIGTERMs the container at + # ~30s, halfway through that budget: the retry loop is rooted in the + # signal context, so it reports "context canceled" and the process + # exits into CrashLoopBackOff instead of waiting out a store that is + # merely slow to return. A booting replica is not a dead one. + startupProbe: + httpGet: + path: /healthz + port: 9090 + periodSeconds: 5 + failureThreshold: 24 # /healthz stays 200 while a terminating pod drains; /readyz # turns 503, so liveness and readiness diverge correctly during # shutdown.