Skip to content

fix: add /readyz Fly check so a dead DB fails health - #4

Closed
BeLazy167 wants to merge 1 commit into
mainfrom
fix/readyz-healthcheck
Closed

fix: add /readyz Fly check so a dead DB fails health#4
BeLazy167 wants to merge 1 commit into
mainfrom
fix/readyz-healthcheck

Conversation

@BeLazy167

Copy link
Copy Markdown
Owner

Problem

/healthz returns a hardcoded {"status":"ok"} and never touches the store (internal/api/server.go:302). It was the only check wired into [checks].

On 2026-09-03 argus-db filled its 10 GB volume. Postgres then crash-looped roughly every 2 seconds:

PANIC: could not write to file "pg_logical/replorigin_checkpoint.tmp": No space left on device
LOG: checkpointer process was terminated by signal 6: Aborted
LOG: all server processes terminated; reinitializing

This ran for three days. Fly reported the app healthy throughout, because /healthz cannot observe the database. The failure surfaced only as "the site loads but nothing works" — the static Next.js shell served 200 while every data fetch failed.

/readyz already existed and pings the pool (server.go:308). It correctly returned 503 for the whole outage. Nothing was watching it.

Change

Adds a second Fly check on /readyz. health keeps liveness semantics on /healthz; ready answers whether the machine can actually serve.

ready uses a 20s grace_period because the pool opens after the listener on boot, and a 30s interval because each probe costs a round trip to Postgres.

Verification

  • fly config validateConfiguration is valid
  • go build ./... and go vet ./... pass
  • /readyz is registered unauthenticated next to /healthz (server.go:133), so the Fly checker reaches it
  • Confirmed against the live outage: /healthz 200 and /readyz 503 while Postgres was down; both 200 after the volume was extended

No Go code changes.

🤖 Generated with Claude Code

https://claude.ai/code/session_01GJehDM4aXz1AkJrTaX2mSL

/healthz returns a constant 200 and never touches the store. It was the
only Fly check, so argus-db filling its volume and crash-looping for three
days still reported the app healthy while every data request failed.

Add a second check on /readyz, which pings the store pool. Longer grace
period covers pool open lagging the listener on boot.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GJehDM4aXz1AkJrTaX2mSL
@BeLazy167

Copy link
Copy Markdown
Owner Author

Wrong repo — reopened as BeLazy167/argus-private#284, where development PRs live.

@BeLazy167 BeLazy167 closed this Sep 4, 2026
@BeLazy167
BeLazy167 deleted the fix/readyz-healthcheck branch September 4, 2026 00:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant