Repository navigation
fix(deploy): 10 min healthcheck start grace so autoheal stops killing bot cold starts (#980) - #984
Open
alex-mextner wants to merge 2 commits into
Open
alex-mextner wants to merge 2 commits into
alex-mextner wants to merge 2 commits into
Conversation
…l stops killing cold starts (#980) autoheal on the odroid restarts any unhealthy container, and /health is 503 until the Telegram start finishes. With start_period 30s a cold start on the loaded host (95-170 s on 2026-11-11, 440 s after the 2026-11-14 hard reset) was killed and restarted from zero in a loop: five restarts between 17:25 and 17:34 UTC on 2026-11-14 (alert down-1794675948). start_period 10m keeps failures during a slow start from counting; start_interval 10s still reports a fast start healthy within seconds. A bot that hangs after starting is still marked unhealthy within retries x (interval + timeout) = 105 s. A regression test reads docker-compose.yml and checks both bounds against the measured starts. Closes #980
…hang-detection bound (#980)
Owner
Author
|
Verification on the build server (ultra@65.21.40.51, |
2 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The bot's Docker healthcheck gets a 10-minute start grace (
start_period: 10m) and a fast start-period probe (start_interval: 10s), so the odroid'sautohealcontainer stops killing a slow cold start and restarting it from zero.Why
autoheal(willfarrell/autoheal,AUTOHEAL_CONTAINER_LABEL=all) restarts any container whose health turnsunhealthy, and/healthanswers 503bot not starteduntil the Telegram start finishes. Withstart_period: 30sthe container turned unhealthy after ~2 min, while cold starts on the loaded host took 95-170 s (2026-11-11, two outages) and 440 s after the hard reset of 2026-11-14. On 2026-11-14 autoheal restarted the bot at 17:25:59, 17:27:43, 17:29:38, 17:31:45 and 17:33:37 UTC;/readycame back only at 17:35:00 UTC (alert down-1794675948).How
docker-compose.yml:start_period: 10m,start_interval: 10s;interval30 s,timeout5 s,retries3 unchanged, so a bot that hangs after it has started is still marked unhealthy within 3 x (30 + 5) = 105 s.test/scripts/bot-healthcheck-grace.test.tsreads the compose file and checks: the grace covers the slowest measured start (440 s), steady-state hang detection stays within 120 s, andstart_intervalis set (a long grace must not delay a fast start).Docker 29.9.0 on the odroid supports
start_interval(Engine 25+);docker compose config(2.40.3) renders it.Verification
bun test test/scripts/bot-healthcheck-grace.test.ts: red before the compose change (2 fail), 3 pass after.bun test test/scripts/docker-compose-secrets.test.ts: 16 pass.docker compose configon a copy with an empty.env: rendersstart_period: 10m0s,start_interval: 10s.review diff --staged --task GH-980, one P3 (hang bound ignoredtimeout) fixed, re-review clean.Not deployed. Not merged: needs the owner's acceptance (#596).
Closes #980