fix(backend): split liveness from readiness so an ES outage is not a green probe

[Xore]

/healthz returned {"ok": true, "es": <bool>} with ok hardcoded to true, so apiary-backend reported healthy to its own compose healthcheck and to anything else probing it even when it could not reach Elasticsearch at all. The es field was the only real signal in the response and it could not change the status code — a backend with a dead dependency and a backend with a working one were the same 200. The #3283 ingest outage would not have shown up there.

What changes

/livez is "the process is up". It never touches Elasticsearch, and that is the load-bearing part rather than an omission: a probe that can block on a dependency converts that dependency's outage into a restart loop of a container that was never the thing that broke. The image HEALTHCHECK moves to it, with a comment saying why it must not be /readyz.

/readyz is "this can do its job": ES reachable, not red, and no index.blocks.write on any index family this tier writes to. 503 with a reason that names the blocked indices rather than counting them.

/healthz stays as an alias for /livez. The port-test harness, the ops scripts and the image healthcheck have all used that name for years, and breaking a health-probe name is how a stack ends up reporting permanently unhealthy with nothing wrong.

$ curl -s -o /dev/null -w '%{http_code}\n' http://backend-service:8081/livez     # ES write-blocked
200        # container stays up, no restart loop
$ curl -s http://backend-service:8081/readyz
{"ready":false,"reason":"elasticsearch has index.blocks.write set on: dashboard-config-v1. The
flood-stage disk watermark sets this on every index at once, and so does an operator's
`PUT /<index>/_block/write`; this endpoint cannot tell those apart, so check _cat/allocation free
space before concluding which one it is.","cluster":"green","write_blocked":["dashboard-config-v1"]}

Decisions worth reviewing, because each is a way to be wrong quietly

• Yellow stays ready. It means unassigned replicas, the ordinary shape of a replicated cluster mid-rolling-restart. Gating on it would take the backend out of service on every deploy. Red is not ready.
• A missing index is not a block. Every dashboard-owned index is created lazily on its first write, so a fresh cluster correctly reads ready — verified, not assumed.
• Unreachable outranks a block report. A write-block reading taken against a cluster we could not reach is the absence of a fact; reporting it as the cause would send an operator after the wrong thing.
• 5s READINESS_TIMEOUT, not the client's 30s. The shared transport budget is sized for real multi-second queries. A probe that blocks for 30s is indistinguishable from the outage it exists to report. Both probes run concurrently.
• es::WRITE_TARGET_FAMILIES is the complement of EVENT_INDICES, not a copy. A read-only index somebody else owns going write-blocked is that tier's outage, and reporting not-ready for it would train operators to ignore the endpoint. dashboard-* is one wildcard rather than twenty concrete names so it cannot fall out of date the way a hand-copied list can.

Verification

Unit tests cover the decision truth table (the function is pure, so the interesting cases — two probes disagreeing — are reachable without a live cluster). Beyond that, run against a real Elasticsearch 9.5.3 (the deployed major):

Scenario	/livez	/readyz
green, nothing blocked	200	200 write_blocked: []
green, dashboard-config-v1 write-blocked	200	503, names the index
three indices blocked	200	503, all three, sorted
a read-only index (ml-worker-metrics) blocked	200	200 — correctly not gated
Elasticsearch unreachable	200	503, carrying the failing URL
fresh cluster, no dashboard indices yet	200	200 — lazy creation, not a block

cargo test 540 passed / 0 failed; cargo clippy --all-targets -- -D warnings clean.

Also in here

• Es::ping() is now dead and is deleted rather than left as a tempting way to reintroduce this.
• Two comments that described the old probe as a ping() caller are corrected. es.rs's own transport-timeout rationale cited /healthz as the reason a 30s bound was needed, which is no longer true of the probe path — the bound now stands only for the real queries and worker loops that can saturate the search queue.
• /livez and /readyz get their own bounded metrics families, so an operator watching request rates can tell a readiness probe's 503s from an unrelated misshapen path.
• Contract documented in docs/OPERATIONS.md, including what the endpoint deliberately cannot distinguish (flood-stage watermark vs. an operator's _block/write).

Docs linters (check-doc-paths-exist, check-docs-reachable, check-doc-stale-paths, check-backend-boot-contract, check-compose-env-docs) all pass. The test container and process used for verification were removed.

Refs #3317

[github-actions]

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

Scanned Files

None