ateapi: a control plane that cannot reach its store serves 200 on /healthz and /readyz, stays Ready, and fails every RPC

[mayawang]

Expected Behavior

Readiness should mean "this process can serve". When ate-api-server cannot reach its store it can
serve nothing — every RPC fails — so it should fail /readyz, leave the Service endpoints, and
surface a signal that something above it can act on: an alert, a metric, a Condition, anything.

Liveness should keep succeeding throughout. The process is alive and will recover on its own once
the store returns; restarting it would make things worse, not better (see #).

Actual Behavior

The two probes are indistinguishable, and both are green while the control plane serves nothing.

Store scaled to zero, sampled every 15 s for six minutes:

Baseline	healthz=200 readyz=200 Ready=True restarts=0 rpc=OK
T+15s → T+360s — 24 consecutive samples	healthz=200 readyz=200 Ready=True restarts=0 rpc=FAIL

Six minutes in which the pod is 200 on /healthz, 200 on /readyz, Ready=True to both the kubelet
and its Service, has never restarted, is still receiving traffic — and fails every RPC. Nothing
in Kubernetes has any way to know, and nothing in the control plane says so either.

This is not a slow-detection problem. There is no detection: /readyz does not consult the store
under any circumstances, so the interval is infinite.

What it costs, concretely: on 27 Aug we spent about three hours on an outage of this shape, most of
it looking in the wrong place, because every signal at the top of the stack was green and the only
component visibly misbehaving was the api-server crash loop from # — which is a symptom of
the store being gone, not the cause. A store outage presents as an api-server bug.

Steps to Reproduce the Problem

1. Install on GKE with the PostgreSQL backend and confirm ate-api-server is serving:
   kubectl-ate get atespaces returns.
2. Take the store away: kubectl -n ate-system scale sts/postgres --replicas=0.
3. Poll all four signals every 15 s for six minutes:

   kubectl -n ate-system get pod <ate-api-server pod> -o jsonpath='{.status.containerStatuses[0].ready} {.status.containerStatuses[0].restartCount}'
   curl -s -o /dev/null -w '%{http_code}' localhost:9090/healthz      # via port-forward
   curl -s -o /dev/null -w '%{http_code}' localhost:9090/readyz
   kubectl-ate get atespaces                                          # the actual RPC
   

   Every sample reads Ready=True restarts=0 healthz=200 readyz=200 while the RPC fails. It does
   not change with time — we held it for six minutes; there is nothing to wait for.
4. Confirm the endpoint is still in service:
   kubectl -n ate-system get endpointslices -l kubernetes.io/service-name=ate-api-server. The pod
   is still listed, so it is still being sent traffic.
5. Bring the store back: kubectl -n ate-system scale sts/postgres --replicas=1. RPCs succeed again
   ~15 s later with restarts=0 — the process recovers by itself, which is exactly why liveness
   must not be involved in the fix.

Specifications

• Version: 4c1b37d (base of release-0.1-rc) plus three cherry-picks. Confirmed still
  present on main at ea3bdc32 (2026-09-02) — internal/serverboot/ has one commit since the
  pin (66300d5c, workersync: gate worker registration on ateom readiness #1243) and it does not touch the health surface. Store-agnostic: unchanged by the
  valkey → Postgres migration (60073ecd, Replace ateredis with atepg #940).
• Platform: GKE, us-central1-c, --store-backend=postgres, two ate-api-server replicas.
  Reproduced 2026-08-27.
• Evidence: the 24-sample polling log above. (One sample at T+195s read healthz=000 readyz=000
  — that was the port-forward blipping, not the server; it recovered on the next sample with the
  restart count unchanged.)

---

Root cause

internal/serverboot/serverboot.go on main at ea3bdc32.

1. /healthz is unconditional, by construction. metricsMux at :356:

if opts.EnableHealthz {
    mux.HandleFunc("/healthz", func(w http.ResponseWriter, _ *http.Request) {
        w.WriteHeader(http.StatusOK)
        _, _ = w.Write([]byte("ok"))
    })
}

The doc comment at :319 says so outright: "EnableHealthz adds an always-200 /healthz for liveness
probes, which must keep succeeding while a draining server fails /readyz." That rationale is
correct and this half is arguably working as designed — see the design constraint below.

2. /readyz consults a drain flag and nothing else. readinessHandler (:367) reads
Readiness.Ready(), and Readiness (:299) is flipped only by MarkNotReady, called only from
drainOnShutdown (cmd/ateapi/main.go:242). Nothing else in the process can ever influence it. A
store outage is invisible to it because no code path connects the two.

3. Readiness is a one-way latch, so the obvious fix does not fit the existing type. Its doc
comment (:297): "Calling MarkNotReady flips it permanently to not ready." That is right for
draining and wrong for a store outage, which must be able to recover. Gating readiness on store
reachability needs a reversible predicate — either a new type, or Readiness gaining a separate
recoverable condition alongside the terminal drain latch. This is the reason the fix is not a
one-liner, and worth knowing before someone starts.

4. Nothing else reports it either. There is no store-reachability metric, no log line above
per-RPC errors, and no Condition. The failure is only visible to a caller who tries an RPC and reads
the error.

The repo already argues this position, for a different binary

StartReadinessServer, twenty lines further down at :334, exits the process if its readiness
endpoint cannot bind, and explains why:

a worker whose readiness endpoint cannot come up never turns Ready and never registers, so dying
loudly lets the kubelet restart it instead of leaving a pod that looks alive but can never
receive work.

That is precisely the state ate-api-server sits in for the entire duration of a store outage. The
principle is already written down and already accepted; it just was not applied to the component
where the blast radius is the whole cluster.

Why this is a priority even though it breaks nothing by itself

It is a force multiplier rather than a fault. It does not cause outages — it makes every other
outage cost hours instead of minutes, and it does that without leaving a trace to grep for
afterwards. Two consequences that matter for anyone operating this without the maintainers on hand:

• Every store-side failure presents as an api-server bug, because the api-server crash loop
  (#) is the only visible symptom while the actual cause is silent.
• Nothing can be alerted on. A pod that is Ready=True restarts=0 with no failing probe and no
  metric produces no page, no dashboard change and no event. The first report comes from a user.

Unlike most items of this kind, it also cannot be resolved by documentation. There is no release
note that makes a green health check informative.

The design constraint the two fixes share

The naive fix — make /healthz reflect store reachability — makes # strictly worse: the
kubelet would then kill a pod that is correctly waiting for its store to come back, and a running
replica that today rides out a six-minute outage with restarts=0 would instead crash-loop through
it. The split has to be:

• /healthz = the process is alive. Never store-gated. Starts before the store connect
  (that is #'s fix).
• /readyz = this process can serve. Store-gated, reversibly, in both boot and steady state.

Stated once here because the two issues will otherwise be fixed by two people in two files with
incompatible assumptions.

Candidate fixes

1. Gate /readyz on store reachability with a reversible predicate. A background health check
   against the store (the pgx pool already exposes what is needed) flipping a condition that
   readinessHandler ANDs with the existing drain latch. Removes the pod from Service endpoints
   while it cannot serve, and puts it back automatically. This is the core fix.
2. Export a store-reachability metric and log the transitions. Cheap, independent of 1, and the
   only one of these that helps someone diagnosing a cluster after the fact rather than during.
   Worth doing even if 1 slips.
3. Leave /healthz exactly as it is. Explicitly listed as a fix so that it is a decision on the
   record rather than an omission — see the constraint above.
4. Apply the same treatment to the other binaries before the pattern spreads. atecontroller: serve health probes #844
   (atecontroller: serve health probes, open and unreviewed since 11 Aug) adds
   HealthProbeBindAddress plus liveness on /healthz and readiness on /readyz with no
   AddHealthzCheck or AddReadyzCheck registered — controller-runtime then serves both as
   unconditional 200, reproducing this issue in a second component. It is two lines to fix while the
   PR is still open.

Related: [#1394 ] (the same health surface killing a booting process, and the fix that must land
compatibly with this one), #844 (about to copy the pattern), #636 (the valkey outage this made
unreadable, closed by deleting the backend rather than by a fix).