Skip to content

autoheal kills hypercal-bot mid cold start: /health is 503 until the bot has started (~95 s on a loaded odroid) but start_period is 30 s #980

Open

Description

@alex-mextner

What

docker-compose.yml bot healthcheck (curl /health, interval 30s, retries 3, start_period 30s) vs src/web/server.ts livenessFailure, which answers 503 'bot not started' until deps.botStarted. On 2026-11-11 the bot cold-started in ~95 s (web server up 17:44:22, 'Bot started' 17:45:27 UTC) on the shared odroid (load ~9, ~2.3 GB swap); autoheal restarted it at 17:43:32 and again at 17:45:30, 3 s before it finished starting. The third start was healthy at 17:47:06 and /ready recovered (200 ok unverified). Alert down-1794419042. Related: #843

Why (motivation)

Each autoheal kill restarts the 95 s start from zero, so under load the container can loop and the bot stays down far longer than one slow start.

User impact

Everyone who writes to the bot got no replies for about 6 minutes on 2026-11-11 (17:41-17:47 UTC), and can get no replies for longer whenever the bot restarts while the home server is busy. No calendar data was lost.

Cost of inaction

After any restart under load the bot can keep restarting itself and stay silent for users until someone intervenes.

Acceptance criteria

  • A cold start of up to ~3 min on the loaded host is not restarted by autoheal
  • A bot that is truly stuck is still restarted by autoheal

Screenshots

(none)

Deploy

Deployed: no

Links

  • (none)

Activity

  1. alex-mextner commented on Nov 11, 2026

    @alex-mextner
    OwnerAuthor

    Repeat on 2026-11-11 (alert down-1794430912, session hcb-alert-20261111T210252Z). Deployed image 128846b (no deploy since 2026-11-08, RestartCount=0 until 21:01 UTC). Host load 15-20 (swap in use); cold start 100-170 s against start_period 30 s. autoheal killed hypercal-bot at 21:01:38, 21:03:36, 21:06:32 (about 3 s before init finished) and 21:08:41 UTC; /health and /ready were 502/503 "bot not started" through 21:09 UTC, so users got no replies for >8 min. Data intact (DB opened, master key verified, Redis healthy). Second outage on the same day: raising start_period/retries or answering /health 200 before botStarted is needed. Side note, unverified and not investigated: the odroid journal shows repeated hcb-odroid-watchdog.service "start-limit-hit" entries during the incident (timer-driven oneshot).

  2. alex-mextner commented on Nov 14, 2026

    @alex-mextner
    OwnerAuthor

    Third recurrence on 2026-11-14 (alert down-1794675948). Deployed image 90ee425 (no deploy since 2026-11-13 09:08 UTC).

    The odroid thrashed from about 16:39 UTC and hard-reset at 17:15:30 UTC (the previous boot's journal ends without a shutdown); boot at 17:16:52 UTC. The first cold start after the reset took 440 s (container start 17:18:30, "Web server started" 17:23:04, "Bot started" 17:25:50 UTC) under I/O pressure some avg10 65-88 %. autoheal restarted hypercal-bot at 17:25:59 (9 s after "Bot started"), 17:27:43, 17:29:38, 17:31:45 and 17:33:37 UTC; the sixth start succeeded once I/O pressure fell to ~17 %, and /ready answered 200 ok (unverified) at 17:35:00 UTC. The bot's own downtime after the host came back was ~18 min, about 10 min of it the restart loop.

    autoheal on the odroid: willfarrell/autoheal:1.2.0, AUTOHEAL_CONTAINER_LABEL=all, AUTOHEAL_INTERVAL=10, from the owner's service-failover module (compose project service-failover-autoheal, created 2026-11-10).

    Fix in progress: bot healthcheck start_period: 10m + start_interval: 10s, regression test test/scripts/bot-healthcheck-grace.test.ts. PR link follows.

  3. alex-mextner commented on Nov 14, 2026

    @alex-mextner
    OwnerAuthor

    Fix: #984 (start_period 10m, start_interval 10s, regression test test/scripts/bot-healthcheck-grace.test.ts). Hosted CI test passed; open, not merged, not deployed (owner acceptance, #596).

  4. alex-mextner commented on Nov 15, 2026

    @alex-mextner
    OwnerAuthor

    Recurred on 2026-11-15 (alert down-1794768391): autoheal killed the cold-starting bot at 18:49:05 and 18:50:57 UTC (healthcheck unhealthy at 18:48:58 and 18:50:53; a start took ~90 s on the swapping odroid). Outage 18:46:31-18:53:58 UTC. The fix #984 is green (test, stage-qa) and still waits for the owner's acceptance.

  5. alex-mextner commented on Nov 15, 2026

    @alex-mextner
    OwnerAuthor

    More evidence from 2026-11-15 19:35-19:47 UTC (memory pressure after a kodi start, see #977): autoheal restarted the bot at 19:35:45, 19:37:55, 19:41:09, 19:43:15 and 19:46:02, every 2-3 min, each time before the previous cold start had logged "Bot started". Container health log: 19:46:36 exit 7 (connection refused), 19:46:41 "Health check exceeded timeout (5s)", then 0 from 19:47:16. The restart loop lasted ~11 min; the attempt that survived needed 54 s from container start to "Bot started". PR #984 is still open.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions