Skip to content

fix(deploy): 10 min healthcheck start grace so autoheal stops killing bot cold starts (#980) - #984

Open
alex-mextner wants to merge 2 commits into
mainfrom
fix/980-autoheal-start-period
Open

alex-mextner wants to merge 2 commits into
mainfrom
fix/980-autoheal-start-period

Conversation

@alex-mextner

Copy link
Copy Markdown
Owner

What

The bot's Docker healthcheck gets a 10-minute start grace (start_period: 10m) and a fast start-period probe (start_interval: 10s), so the odroid's autoheal container stops killing a slow cold start and restarting it from zero.

Why

autoheal (willfarrell/autoheal, AUTOHEAL_CONTAINER_LABEL=all) restarts any container whose health turns unhealthy, and /health answers 503 bot not started until the Telegram start finishes. With start_period: 30s the container turned unhealthy after ~2 min, while cold starts on the loaded host took 95-170 s (2026-11-11, two outages) and 440 s after the hard reset of 2026-11-14. On 2026-11-14 autoheal restarted the bot at 17:25:59, 17:27:43, 17:29:38, 17:31:45 and 17:33:37 UTC; /ready came back only at 17:35:00 UTC (alert down-1794675948).

How

  • docker-compose.yml: start_period: 10m, start_interval: 10s; interval 30 s, timeout 5 s, retries 3 unchanged, so a bot that hangs after it has started is still marked unhealthy within 3 x (30 + 5) = 105 s.
  • test/scripts/bot-healthcheck-grace.test.ts reads the compose file and checks: the grace covers the slowest measured start (440 s), steady-state hang detection stays within 120 s, and start_interval is set (a long grace must not delay a fast start).
  • Runbook: one bullet in the Docker section.

Docker 29.9.0 on the odroid supports start_interval (Engine 25+); docker compose config (2.40.3) renders it.

Verification

  • bun test test/scripts/bot-healthcheck-grace.test.ts: red before the compose change (2 fail), 3 pass after.
  • bun test test/scripts/docker-compose-secrets.test.ts: 16 pass.
  • docker compose config on a copy with an empty .env: renders start_period: 10m0s, start_interval: 10s.
  • Full suite + typecheck on the build server: see the comment below.
  • Review: review diff --staged --task GH-980, one P3 (hang bound ignored timeout) fixed, re-review clean.

Not deployed. Not merged: needs the owner's acceptance (#596).

Closes #980

…l stops killing cold starts (#980)

autoheal on the odroid restarts any unhealthy container, and /health is 503 until the Telegram start finishes. With start_period 30s a cold start on the loaded host (95-170 s on 2026-11-11, 440 s after the 2026-11-14 hard reset) was killed and restarted from zero in a loop: five restarts between 17:25 and 17:34 UTC on 2026-11-14 (alert down-1794675948).

start_period 10m keeps failures during a slow start from counting; start_interval 10s still reports a fast start healthy within seconds. A bot that hangs after starting is still marked unhealthy within retries x (interval + timeout) = 105 s. A regression test reads docker-compose.yml and checks both bounds against the measured starts.

Closes #980
@alex-mextner

Copy link
Copy Markdown
Owner Author

Verification on the build server (ultra@65.21.40.51, server-run, tree of 8ca33d9 + the staged test fix): bun test 9589 pass / 16 skip / 5 fail; tsc --noEmit -p . exit 0. The 5 failures are browser/installer tests that do not read docker-compose.yml: test_odroid_stage_qa_installer.py, test_stage_qa_browser_installer.py, "real browser pool: six isolated acceptance scenarios" (fails in 5 ms: Chromium does not launch on that host), "verify-intent-catalogue passes on the page the preview just wrote", "committed intent page works in actual Chromium". Hosted PR CI test (full suite) passed in 12m22s on 90b08e6; lint, CodeQL, gitleaks, type-bans and the stage image build passed; stage-qa was pending when this was written.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

autoheal kills hypercal-bot mid cold start: /health is 503 until the bot has started (~95 s on a loaded odroid) but start_period is 30 s

1 participant