Readiness design: should /_/ready avoid a live ListBuckets so it survives a sustained backend-throttle tail?

[ton77v]

Following up on #62 (the retry/timeout fix you shipped in v1.14.1 — thank you, it lands the common case cleanly). This is a design question about the sustained-throttle tail, framed against DGP's own readiness-signal goals — your call on the trade-off, not a request for a specific change.

Observation

/_/ready performs a live ListBuckets against the object backend on every probe. When our storage provider (Hetzner hel1) throttles or has a long latency tail, that list 503s ("Backend throttled") and readiness flips to a paging 503 — even though the data plane keeps serving normally (only the probe fails; zero customer-facing impact both times).

We've now hit this twice:

• 2026-07-10 — hel1 degradation, /_/ready 503s at the list timeout, data plane unaffected.
• 2026-07-26 — recurrence, same signature ("Backend throttled" on the list call).

v1.14.1's retry (DGP_READY_RETRIES) + tunable timeout (DGP_READY_TIMEOUT_SECS) absorb a short throttle well. The open question is the sustained case: retry rides out a blip, but a multi-minute provider degradation still flips readiness, because the readiness signal is structurally coupled to a heavyweight, throttle-prone ListBuckets.

The question

Is a live ListBuckets the right primitive for readiness under provider stress, or would something lighter reflect "can we serve?" more faithfully without being hostage to list-throttling — e.g.:

• a cheaper probe like HeadBucket, or
• a short-TTL cached last-known-good, so a transient/sustained list-throttle doesn't immediately downgrade readiness while the data plane is demonstrably fine?

The trade-off (readiness freshness vs. resilience to provider list-throttling) is yours to weigh — you know the readiness contract's intent better than we do. Retry (#62) may well be judged sufficient; if so, a "won't change, retry is the answer" is a perfectly good outcome for us to document.

Evidence available

Happy to share the two incident write-ups (503-at-list-timeout signature + "Backend throttled" backend logs) if useful as input.

🦀 sent by Claude Code

[sscarduzio]

Your read is exactly right, and I've confirmed it against the code. readiness_check (src/api/handlers/status.rs:200) calls engine.list_buckets() on every probe, wrapped in the #62 retry + tunable timeout loop (DGP_READY_TIMEOUT_SECS / DGP_READY_RETRIES). That absorbs a short blip well, but you've put your finger on the structural issue: readiness is coupled to a heavyweight, throttle-prone ListBuckets, so a multi-minute provider degradation flips readiness to a paging 503 even while the data plane keeps serving.

The verdict: worth decoupling — and HeadBucket alone isn't the fix

HeadBucket is cheaper per call, but it's still a live backend round-trip on the same throttled path, so under a sustained provider degradation it flips readiness for the same reason. Swapping the primitive doesn't buy resilience to the case you actually hit twice.

The primitive that does is a short-TTL last-known-good: readiness reports ready as long as the backend has demonstrably answered something recently, and only downgrades once nothing has succeeded within a bounded staleness window. That directly reflects "can we serve?" rather than "did this one heavyweight list just succeed?", which is the question a readiness probe is supposed to answer.

Concretely, what I'm inclined to build

• Keep the existing probe as the refresh path, but record a last_backend_ok_at timestamp on any successful backend interaction — the readiness list and real data-plane ops (a served GET/PUT proves reachability far better than a synthetic list).
• Readiness returns ready if now - last_backend_ok_at < DGP_READY_CACHE_TTL_SECS; it runs the live list only when the cache is cold, and reports not_ready only when nothing has succeeded within the window.
• The TTL is the freshness-vs-resilience knob you named: a genuinely-down backend still fails readiness once the window lapses, but a list-throttle tail with a healthy data plane no longer pages.

I'd ship this opt-in (env-gated, defaulting to today's behavior) so it doesn't change anyone's contract silently. The retry from #62 stays as the inner loop.

If that sounds right to you, I'll take the incident write-ups you offered as test fixtures for the sustained-throttle case. And if you'd rather we treat #62's retry as sufficient and just document the tradeoff, that's a fine outcome too — but given you've hit this twice, I lean toward building the cached path.

[ton77v]

Yes — that sounds right, and it matches where we independently landed one layer down. Plus a third data
point for the thread, and two design questions.

Agreed on the shape

The part we'd underline is your reason for rejecting HeadBucket: it's still a live round-trip on the
same throttled path, so it buys latency, not resilience. We reached the same conclusion in the portal
by the boring route — tightening timeouts on our S3 client made the failure faster, not rarer. Only
remembering the last good answer changes the outcome. Opt-in with today's behaviour as the default is
also the right call; we'd flip it on in staging first and report back.

Third data point: 2026-08-07 ~15:55 UTC, seen from the client side

The two occurrences on record here (2026-07-10, 2026-07-26) are both DGP's own probe. We have one from
the other end of the wire, not yet in this thread:

• When: 2026-08-07T15:55:40Z.
• What: ror-portal's boto3 client, a list_objects_v2 against files.readonlyrest.com, died on a
  gevent SSL read timeout. Customer-visible: the download version list degraded, and a customer waited
  ~64s before getting a wrong answer.
• Duration: under two minutes; it recovered on its own with no intervention.
• Shape: slow-but-serving, not down — exactly the regime this issue is about.

Honest caveat: we have not correlated the two ends. We can't tell from our side whether DGP was
logging Backend throttled in that window, so this is consistent with a hel1 throttle, not proof of
one. If your logs still reach back to 2026-08-07 15:50–16:00 UTC, that would either make this occurrence
#3 of a known pattern or rule it out — both useful.

Two design questions on last_backend_ok_at

1. What counts as a "successful backend interaction"? You mention including real data-plane ops, which
   we think is the load-bearing part. Concretely: do the presigned-URL GETs served through DGP refresh the
   clock? For us those are the overwhelming majority of traffic and the strongest possible evidence of
   reachability — if only the control-plane/probe path refreshes it, a busy proxy can still go unready
   while visibly serving downloads.
2. Does a backend 404 refresh the clock? It should, we'd argue — a NoSuchKey/NoSuchBucket is the
   backend answering, which is precisely what readiness asks. Worth stating explicitly in the
   implementation, because the natural if response.ok reading gets it wrong.

Fixtures — yes, take them

Happy to hand over the write-ups as test fixtures. Ours are in beshu-tech/customer_portal #401 (the
incident) and #402 (the last-known-good fix on the request path); the timings and the exact botocore
timeout/retry arithmetic behind that 64s are recorded there. Say the word on format and we'll extract the
raw traces rather than the prose.

One expectation to set, so nobody's surprised

Cached readiness stops DGP reporting itself unready under a throttle — it doesn't make the backend
answer a client's list faster. So it won't prevent the client-side symptom above, and we're not expecting
it to; that's ours to fix and is already in flight (#402), along with the same decoupling applied to our
own /health probe, which had the identical live-list-per-poll design your verdict here just condemned
(#426). Flagging it only so the two fixes aren't mistaken for one.

🦀 sent by Claude Code

[sscarduzio]

Shipped in 4323031 on main.

What changed

/_/ready keeps its current contract by default. The new knob is DGP_READY_CACHE_TTL_SECS, which defaults to 0 — that is exactly today's behaviour, where the ListBuckets probe must succeed or the node reports not-ready. Nothing changes for anyone who does not opt in.

When you set it above zero and the list probe fails, readiness now degrades in three steps instead of going straight to 503:

1. A cheap HeadBucket reachability check. This is the primitive you suggested. It runs only when the list has already failed, so the healthy path costs no extra round-trip. Either outcome — bucket present or absent — proves the backend answered; only an error or timeout means it did not. If it answers, readiness reports backend: "degraded" and the node keeps serving. This is the direct answer to your incident: the list is throttled, the backend is reachable, the data plane is fine.
2. The last-known-good window. If the cheap probe also fails, readiness accepts a backend call that succeeded within DGP_READY_CACHE_TTL_SECS and reports backend: "cached".
3. The real failure. Past the window, it reports timeout or unreachable and returns 503 as before.

Two properties worth stating explicitly, because they are what stop this from becoming a probe that lies:

• The window only refreshes on a probe that genuinely round-tripped to the backend, so cached decays. A backend that is actually dead still goes not-ready once the window lapses.
• A node that has never reached its backend cannot come up cached-ready on boot.

The #62 retry loop is unchanged and still runs as the inner loop.

Suggested setting

For your Hetzner case, DGP_READY_CACHE_TTL_SECS=300 rides out a five-minute provider degradation without paging, while still pulling a genuinely dead node within five minutes.

The decision logic is a pure function with the full truth table unit-tested (strict mode, throttled-but-reachable, both-probes-down-inside-window, stale-window, never-stamped). Documented in reference/configuration.md.

I have left this issue open until you have had a chance to try it against your instance — if the two incident write-ups you offered show a case this does not cover, that is useful input before we consider changing the default.

[ton77v]

Thanks, will get this armed!

Will see how the things are going for ~ a week to see if it's safe to close