You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
autoheal kills hypercal-bot mid cold start: /health is 503 until the bot has started (~95 s on a loaded odroid) but start_period is 30 s #980
docker-compose.yml bot healthcheck (curl /health, interval 30s, retries 3, start_period 30s) vs src/web/server.ts livenessFailure, which answers 503 'bot not started' until deps.botStarted. On 2026-11-11 the bot cold-started in ~95 s (web server up 17:44:22, 'Bot started' 17:45:27 UTC) on the shared odroid (load ~9, ~2.3 GB swap); autoheal restarted it at 17:43:32 and again at 17:45:30, 3 s before it finished starting. The third start was healthy at 17:47:06 and /ready recovered (200 ok unverified). Alert down-1794419042. Related: #843
Why (motivation)
Each autoheal kill restarts the 95 s start from zero, so under load the container can loop and the bot stays down far longer than one slow start.
User impact
Everyone who writes to the bot got no replies for about 6 minutes on 2026-11-11 (17:41-17:47 UTC), and can get no replies for longer whenever the bot restarts while the home server is busy. No calendar data was lost.
Cost of inaction
After any restart under load the bot can keep restarting itself and stay silent for users until someone intervenes.
Acceptance criteria
A cold start of up to ~3 min on the loaded host is not restarted by autoheal
A bot that is truly stuck is still restarted by autoheal
Repeat on 2026-11-11 (alert down-1794430912, session hcb-alert-20261111T210252Z). Deployed image 128846b (no deploy since 2026-11-08, RestartCount=0 until 21:01 UTC). Host load 15-20 (swap in use); cold start 100-170 s against start_period 30 s. autoheal killed hypercal-bot at 21:01:38, 21:03:36, 21:06:32 (about 3 s before init finished) and 21:08:41 UTC; /health and /ready were 502/503 "bot not started" through 21:09 UTC, so users got no replies for >8 min. Data intact (DB opened, master key verified, Redis healthy). Second outage on the same day: raising start_period/retries or answering /health 200 before botStarted is needed. Side note, unverified and not investigated: the odroid journal shows repeated hcb-odroid-watchdog.service "start-limit-hit" entries during the incident (timer-driven oneshot).
Third recurrence on 2026-11-14 (alert down-1794675948). Deployed image 90ee425 (no deploy since 2026-11-13 09:08 UTC).
The odroid thrashed from about 16:39 UTC and hard-reset at 17:15:30 UTC (the previous boot's journal ends without a shutdown); boot at 17:16:52 UTC. The first cold start after the reset took 440 s (container start 17:18:30, "Web server started" 17:23:04, "Bot started" 17:25:50 UTC) under I/O pressure some avg10 65-88 %. autoheal restarted hypercal-bot at 17:25:59 (9 s after "Bot started"), 17:27:43, 17:29:38, 17:31:45 and 17:33:37 UTC; the sixth start succeeded once I/O pressure fell to ~17 %, and /ready answered 200 ok (unverified) at 17:35:00 UTC. The bot's own downtime after the host came back was ~18 min, about 10 min of it the restart loop.
autoheal on the odroid: willfarrell/autoheal:1.2.0, AUTOHEAL_CONTAINER_LABEL=all, AUTOHEAL_INTERVAL=10, from the owner's service-failover module (compose project service-failover-autoheal, created 2026-11-10).
Fix in progress: bot healthcheck start_period: 10m + start_interval: 10s, regression test test/scripts/bot-healthcheck-grace.test.ts. PR link follows.
Fix: #984 (start_period 10m, start_interval 10s, regression test test/scripts/bot-healthcheck-grace.test.ts). Hosted CI test passed; open, not merged, not deployed (owner acceptance, #596).
Recurred on 2026-11-15 (alert down-1794768391): autoheal killed the cold-starting bot at 18:49:05 and 18:50:57 UTC (healthcheck unhealthy at 18:48:58 and 18:50:53; a start took ~90 s on the swapping odroid). Outage 18:46:31-18:53:58 UTC. The fix #984 is green (test, stage-qa) and still waits for the owner's acceptance.
More evidence from 2026-11-15 19:35-19:47 UTC (memory pressure after a kodi start, see #977): autoheal restarted the bot at 19:35:45, 19:37:55, 19:41:09, 19:43:15 and 19:46:02, every 2-3 min, each time before the previous cold start had logged "Bot started". Container health log: 19:46:36 exit 7 (connection refused), 19:46:41 "Health check exceeded timeout (5s)", then 0 from 19:47:16. The restart loop lasted ~11 min; the attempt that survived needed 54 s from container start to "Bot started". PR #984 is still open.
Activity
alex-mextner commentedon Nov 11, 2026
Repeat on 2026-11-11 (alert down-1794430912, session hcb-alert-20261111T210252Z). Deployed image 128846b (no deploy since 2026-11-08, RestartCount=0 until 21:01 UTC). Host load 15-20 (swap in use); cold start 100-170 s against start_period 30 s. autoheal killed hypercal-bot at 21:01:38, 21:03:36, 21:06:32 (about 3 s before init finished) and 21:08:41 UTC; /health and /ready were 502/503 "bot not started" through 21:09 UTC, so users got no replies for >8 min. Data intact (DB opened, master key verified, Redis healthy). Second outage on the same day: raising start_period/retries or answering /health 200 before botStarted is needed. Side note, unverified and not investigated: the odroid journal shows repeated hcb-odroid-watchdog.service "start-limit-hit" entries during the incident (timer-driven oneshot).
alex-mextner commentedon Nov 14, 2026
Third recurrence on 2026-11-14 (alert down-1794675948). Deployed image 90ee425 (no deploy since 2026-11-13 09:08 UTC).
The odroid thrashed from about 16:39 UTC and hard-reset at 17:15:30 UTC (the previous boot's journal ends without a shutdown); boot at 17:16:52 UTC. The first cold start after the reset took 440 s (container start 17:18:30, "Web server started" 17:23:04, "Bot started" 17:25:50 UTC) under I/O pressure
some avg1065-88 %. autoheal restartedhypercal-botat 17:25:59 (9 s after "Bot started"), 17:27:43, 17:29:38, 17:31:45 and 17:33:37 UTC; the sixth start succeeded once I/O pressure fell to ~17 %, and/readyanswered200 ok (unverified)at 17:35:00 UTC. The bot's own downtime after the host came back was ~18 min, about 10 min of it the restart loop.autoheal on the odroid:
willfarrell/autoheal:1.2.0,AUTOHEAL_CONTAINER_LABEL=all,AUTOHEAL_INTERVAL=10, from the owner's service-failover module (compose projectservice-failover-autoheal, created 2026-11-10).Fix in progress: bot healthcheck
start_period: 10m+start_interval: 10s, regression testtest/scripts/bot-healthcheck-grace.test.ts. PR link follows.alex-mextner commentedon Nov 14, 2026
Fix: #984 (start_period 10m, start_interval 10s, regression test test/scripts/bot-healthcheck-grace.test.ts). Hosted CI
testpassed; open, not merged, not deployed (owner acceptance, #596).alex-mextner commentedon Nov 15, 2026
Recurred on 2026-11-15 (alert down-1794768391): autoheal killed the cold-starting bot at 18:49:05 and 18:50:57 UTC (healthcheck unhealthy at 18:48:58 and 18:50:53; a start took ~90 s on the swapping odroid). Outage 18:46:31-18:53:58 UTC. The fix #984 is green (test, stage-qa) and still waits for the owner's acceptance.
alex-mextner commentedon Nov 15, 2026
More evidence from 2026-11-15 19:35-19:47 UTC (memory pressure after a kodi start, see #977): autoheal restarted the bot at 19:35:45, 19:37:55, 19:41:09, 19:43:15 and 19:46:02, every 2-3 min, each time before the previous cold start had logged "Bot started". Container health log: 19:46:36 exit 7 (connection refused), 19:46:41 "Health check exceeded timeout (5s)", then 0 from 19:47:16. The restart loop lasted ~11 min; the attempt that survived needed 54 s from container start to "Bot started". PR #984 is still open.