From 8ca33d9a8dffbde2318813f2279f7b29f2a3889a Mon Sep 17 00:00:00 2001 From: Alex Ultra <111590326+alex-mextner@users.noreply.github.com> Date: Sat, 14 Nov 2026 18:46:49 +0100 Subject: [PATCH 1/2] fix(deploy): give the bot healthcheck a 10 min start grace so autoheal stops killing cold starts (#980) autoheal on the odroid restarts any unhealthy container, and /health is 503 until the Telegram start finishes. With start_period 30s a cold start on the loaded host (95-170 s on 2026-11-11, 440 s after the 2026-11-14 hard reset) was killed and restarted from zero in a loop: five restarts between 17:25 and 17:34 UTC on 2026-11-14 (alert down-1794675948). start_period 10m keeps failures during a slow start from counting; start_interval 10s still reports a fast start healthy within seconds. A bot that hangs after starting is still marked unhealthy within retries x (interval + timeout) = 105 s. A regression test reads docker-compose.yml and checks both bounds against the measured starts. Closes #980 --- docker-compose.yml | 9 ++- docs/reference/deploy-runbook.md | 1 + test/scripts/bot-healthcheck-grace.test.ts | 71 ++++++++++++++++++++++ 3 files changed, 80 insertions(+), 1 deletion(-) create mode 100644 test/scripts/bot-healthcheck-grace.test.ts diff --git a/docker-compose.yml b/docker-compose.yml index 5f5333d48..8527d0991 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -74,12 +74,19 @@ services: volumes: - ./data:/app/data - ./logs:/app/logs + # autoheal on the odroid restarts any container that turns unhealthy, and /health is 503 + # until the Telegram start finishes. Cold starts on the loaded host took 95-170 s + # (2026-11-11) and 440 s after a hard reset (2026-11-14); a 30 s start_period restarted + # them from zero in a loop (#980). Failures inside start_period do not count, start_interval + # still reports healthy within 10 s of a fast start, and a bot that hangs after starting is + # caught by interval x retries as before. healthcheck: test: ["CMD", "curl", "-f", "http://localhost:3311/health"] interval: 30s timeout: 5s retries: 3 - start_period: 30s + start_period: 10m + start_interval: 10s # The home ODROID-N2+ (6 cores, 3.7 GB RAM) hosts this bot since 2026-10-09, shared # with ExpenseSyncBot, HyperSummaryBot, Home Assistant and others: 1.5 CPUs leaves them # headroom. RAM there is overcommitted (2 GB in swap on 2026-10-10, the idle bot down to diff --git a/docs/reference/deploy-runbook.md b/docs/reference/deploy-runbook.md index 6e1c7e5ed..cffdb72d5 100644 --- a/docs/reference/deploy-runbook.md +++ b/docs/reference/deploy-runbook.md @@ -319,6 +319,7 @@ If a release that the override accepted with `rollback-compatible: no` fails ver - `docker compose` requires root (neither `www-data` nor `hcb-runner` is in the docker group). - Resource limits: bot 1G/1.5cpu with a 384M reservation (cgroup `memory.low`), redis 256M/0.5cpu. The odroid has 6 cores and 3.7 GB RAM shared with the neighbours listed above and is overcommitted (on 2026-10-10 about 2 GB sat in swap and the idle bot was down to 71 MB resident); the reservation keeps the bot's working set out of reclaim so a message is not served from swap. Keep the bot well below the whole box. - Bot container memory, measured 2026-11-04 (#963, #800). The image before the 0.3.1-0.3.3 deploys (ff93220a) and the current one (e4185ddf) have the same dependencies, Dockerfile, compose limits and module-load footprint (importing every startup module takes a bun process from 18 MB to 131-133 MB RSS on both, 29 MB of heap), so the jump from the 138-163 MiB seen on the old container is not a code change. A fresh e4185ddf container showed 413 MiB in `docker stats` one minute after start and 449-461 MiB over the next 16 minutes: flat, no growth, while the kernel swapped cold pages out (cgroup anon 405 MB falling to 363 MB, `memory.swap.current` 108 MB at minute 20, `memory.low` 384 MiB). Inside it the `bun` process held 328 MB anon plus 93 MB swapped; the five Chromium processes of the render pool (`playwrightPool.initialize()` starts them at boot) are mostly shared binary pages (the two measured held about 5 MB anon each). A container that has run for 28 h looks small because the host swapped its idle part out (the 71 MB of 2026-10-10 is the same effect): compare containers of the same age. The largest single item was the stress dictionary, 555,461 entries parsed at boot: in a bare bun process it takes RSS from 51 MB to 362 MB at the peak and leaves 190 MB, and production never uses it unless a model calls `lookup_stress` (Silero and the MTProto service tier are off there). It now loads on first use (#963), or at startup when `SILERO_PYTHON_PATH` is set (every Russian voice reply needs it then, and an unreadable file keeps Silero off as before); the load is one synchronous parse, so the first `lookup_stress` after a restart stalls the bot for about 2 s on a fast PC and longer on the odroid. Measure with `docker stats --no-stream hypercal-bot`, `grep -E 'RssAnon|VmSwap' /proc//status` inside the container and `/sys/fs/cgroup/memory.current`, `memory.swap.current`, `memory.low`. +- Bot healthcheck grace (#980): the odroid's `autoheal` container (`willfarrell/autoheal`, `AUTOHEAL_CONTAINER_LABEL=all`, from the owner's service-failover module since 2026-11-10) restarts any container whose health turns `unhealthy`, and `/health` answers 503 `bot not started` until the Telegram start finishes. With `start_period: 30s` a cold start on the loaded host (95-170 s on 2026-11-11, 440 s after the hard reset of 2026-11-14) was killed and started from zero in a loop: four restarts between 17:25 and 17:32 UTC on 2026-11-14. The healthcheck now has `start_period: 10m` (failures inside it do not count) and `start_interval: 10s` (a fast start still turns healthy within seconds, so the deploy health wait is not slower); a bot that hangs after it has started is still caught by `interval` 30 s x `retries` 3. `test/scripts/bot-healthcheck-grace.test.ts` keeps both bounds. - No deploy secrets: the SSH secrets `SSH_HOST`, `SSH_USER`, `SSH_KEY`, `DEPLOY_PATH` are unused since #784. ### Image identity diff --git a/test/scripts/bot-healthcheck-grace.test.ts b/test/scripts/bot-healthcheck-grace.test.ts new file mode 100644 index 000000000..57c180baa --- /dev/null +++ b/test/scripts/bot-healthcheck-grace.test.ts @@ -0,0 +1,71 @@ +// test/scripts/bot-healthcheck-grace.test.ts +// +// The odroid runs willfarrell/autoheal with AUTOHEAL_CONTAINER_LABEL=all: it restarts any +// container whose Docker health status turns `unhealthy`. The bot answers /health with 503 +// "bot not started" until its Telegram start finishes, so a cold start slower than the +// healthcheck grace is killed and starts again from zero. Measured cold starts on the loaded +// host: ~95 s and 100-170 s on 2026-11-11 (restart loops, #980), and 440 s on 2026-11-14 right +// after the host's hard reset (container start 17:18:30 UTC, "Bot started" 17:25:50 UTC, killed +// at 17:25:59, 17:27:43 and 17:29:38 UTC). These tests keep the grace above the slowest +// measured start while a bot that hangs after it has started is still restarted within minutes. +import { describe, expect, test } from 'bun:test'; +import { readFileSync } from 'node:fs'; +import { join } from 'node:path'; +import { z } from 'zod'; + +const COMPOSE_FILE = join(import.meta.dir, '../../docker-compose.yml'); + +const SLOWEST_MEASURED_COLD_START_S = 440; +/** A hung bot that had started must not stay up longer than this before autoheal acts. */ +const STEADY_STATE_DETECTION_LIMIT_S = 120; + +const healthcheckSchema = z.object({ + test: z.array(z.string()), + interval: z.string(), + timeout: z.string(), + retries: z.number().int().positive(), + start_period: z.string(), + start_interval: z.string().optional(), +}); + +const composeSchema = z.object({ + services: z.object({ bot: z.object({ healthcheck: healthcheckSchema }) }), +}); + +/** Compose duration ("30s", "10m", "1m30s") in seconds. */ +function seconds(duration: string): number { + const parts = [...duration.matchAll(/(\d+)(h|m|s)/g)]; + const consumed = parts.map((p) => p[0]).join(''); + if (parts.length === 0 || consumed !== duration) throw new Error(`unsupported duration: ${duration}`); + const unit = { h: 3600, m: 60, s: 1 } as const; + return parts.reduce((total, [, value, u]) => total + Number(value) * unit[u as keyof typeof unit], 0); +} + +function botHealthcheck() { + return composeSchema.parse(Bun.YAML.parse(readFileSync(COMPOSE_FILE, 'utf8'))).services.bot.healthcheck; +} + +describe('bot healthcheck grace under autoheal (#980)', () => { + test('a cold start as slow as the slowest measured one never turns the container unhealthy', () => { + const hc = botHealthcheck(); + // Failures inside start_period do not count; after it, `retries` failures `interval` apart do. + const grace = seconds(hc.start_period) + hc.retries * seconds(hc.interval); + expect(grace).toBeGreaterThan(SLOWEST_MEASURED_COLD_START_S); + // Margin over the measurement itself: a slower host must not loop again at the first excess. + expect(seconds(hc.start_period)).toBeGreaterThanOrEqual(SLOWEST_MEASURED_COLD_START_S); + }); + + test('a started bot that hangs is still marked unhealthy within the steady-state limit', () => { + const hc = botHealthcheck(); + // A hanging probe runs until `timeout`, and Docker waits `interval` after each probe ends. + const worstDetection = hc.retries * (seconds(hc.interval) + seconds(hc.timeout)); + expect(worstDetection).toBeLessThanOrEqual(STEADY_STATE_DETECTION_LIMIT_S); + }); + + test('a long grace does not delay a fast start: the start-period probe runs often', () => { + const hc = botHealthcheck(); + // Without start_interval Docker probes the starting container only every `interval`. + expect(hc.start_interval).toBeDefined(); + expect(seconds(hc.start_interval ?? hc.interval)).toBeLessThanOrEqual(10); + }); +}); From 90b08e6cca759c4a6190f9afc7dd76724c14746d Mon Sep 17 00:00:00 2001 From: Alex Ultra <111590326+alex-mextner@users.noreply.github.com> Date: Sat, 14 Nov 2026 18:47:06 +0100 Subject: [PATCH 2/2] docs(runbook): correct the 2026-11-14 autoheal restart count and the hang-detection bound (#980) --- docs/reference/deploy-runbook.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/deploy-runbook.md b/docs/reference/deploy-runbook.md index cffdb72d5..3be088066 100644 --- a/docs/reference/deploy-runbook.md +++ b/docs/reference/deploy-runbook.md @@ -319,7 +319,7 @@ If a release that the override accepted with `rollback-compatible: no` fails ver - `docker compose` requires root (neither `www-data` nor `hcb-runner` is in the docker group). - Resource limits: bot 1G/1.5cpu with a 384M reservation (cgroup `memory.low`), redis 256M/0.5cpu. The odroid has 6 cores and 3.7 GB RAM shared with the neighbours listed above and is overcommitted (on 2026-10-10 about 2 GB sat in swap and the idle bot was down to 71 MB resident); the reservation keeps the bot's working set out of reclaim so a message is not served from swap. Keep the bot well below the whole box. - Bot container memory, measured 2026-11-04 (#963, #800). The image before the 0.3.1-0.3.3 deploys (ff93220a) and the current one (e4185ddf) have the same dependencies, Dockerfile, compose limits and module-load footprint (importing every startup module takes a bun process from 18 MB to 131-133 MB RSS on both, 29 MB of heap), so the jump from the 138-163 MiB seen on the old container is not a code change. A fresh e4185ddf container showed 413 MiB in `docker stats` one minute after start and 449-461 MiB over the next 16 minutes: flat, no growth, while the kernel swapped cold pages out (cgroup anon 405 MB falling to 363 MB, `memory.swap.current` 108 MB at minute 20, `memory.low` 384 MiB). Inside it the `bun` process held 328 MB anon plus 93 MB swapped; the five Chromium processes of the render pool (`playwrightPool.initialize()` starts them at boot) are mostly shared binary pages (the two measured held about 5 MB anon each). A container that has run for 28 h looks small because the host swapped its idle part out (the 71 MB of 2026-10-10 is the same effect): compare containers of the same age. The largest single item was the stress dictionary, 555,461 entries parsed at boot: in a bare bun process it takes RSS from 51 MB to 362 MB at the peak and leaves 190 MB, and production never uses it unless a model calls `lookup_stress` (Silero and the MTProto service tier are off there). It now loads on first use (#963), or at startup when `SILERO_PYTHON_PATH` is set (every Russian voice reply needs it then, and an unreadable file keeps Silero off as before); the load is one synchronous parse, so the first `lookup_stress` after a restart stalls the bot for about 2 s on a fast PC and longer on the odroid. Measure with `docker stats --no-stream hypercal-bot`, `grep -E 'RssAnon|VmSwap' /proc//status` inside the container and `/sys/fs/cgroup/memory.current`, `memory.swap.current`, `memory.low`. -- Bot healthcheck grace (#980): the odroid's `autoheal` container (`willfarrell/autoheal`, `AUTOHEAL_CONTAINER_LABEL=all`, from the owner's service-failover module since 2026-11-10) restarts any container whose health turns `unhealthy`, and `/health` answers 503 `bot not started` until the Telegram start finishes. With `start_period: 30s` a cold start on the loaded host (95-170 s on 2026-11-11, 440 s after the hard reset of 2026-11-14) was killed and started from zero in a loop: four restarts between 17:25 and 17:32 UTC on 2026-11-14. The healthcheck now has `start_period: 10m` (failures inside it do not count) and `start_interval: 10s` (a fast start still turns healthy within seconds, so the deploy health wait is not slower); a bot that hangs after it has started is still caught by `interval` 30 s x `retries` 3. `test/scripts/bot-healthcheck-grace.test.ts` keeps both bounds. +- Bot healthcheck grace (#980): the odroid's `autoheal` container (`willfarrell/autoheal`, `AUTOHEAL_CONTAINER_LABEL=all`, from the owner's service-failover module since 2026-11-10) restarts any container whose health turns `unhealthy`, and `/health` answers 503 `bot not started` until the Telegram start finishes. With `start_period: 30s` a cold start on the loaded host (95-170 s on 2026-11-11, 440 s after the hard reset of 2026-11-14) was killed and started from zero in a loop: five restarts between 17:25 and 17:34 UTC on 2026-11-14. The healthcheck now has `start_period: 10m` (failures inside it do not count) and `start_interval: 10s` (a fast start still turns healthy within seconds, so the deploy health wait is not slower); a bot that hangs after it has started is still caught within `retries` 3 x (`interval` 30 s + `timeout` 5 s). `test/scripts/bot-healthcheck-grace.test.ts` keeps both bounds. - No deploy secrets: the SSH secrets `SSH_HOST`, `SSH_USER`, `SSH_KEY`, `DEPLOY_PATH` are unused since #784. ### Image identity