Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 8 additions & 1 deletion docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -74,12 +74,19 @@ services:
volumes:
- ./data:/app/data
- ./logs:/app/logs
# autoheal on the odroid restarts any container that turns unhealthy, and /health is 503
# until the Telegram start finishes. Cold starts on the loaded host took 95-170 s
# (2026-11-11) and 440 s after a hard reset (2026-11-14); a 30 s start_period restarted
# them from zero in a loop (#980). Failures inside start_period do not count, start_interval
# still reports healthy within 10 s of a fast start, and a bot that hangs after starting is
# caught by interval x retries as before.
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:3311/health"]
interval: 30s
timeout: 5s
retries: 3
start_period: 30s
start_period: 10m
start_interval: 10s
# The home ODROID-N2+ (6 cores, 3.7 GB RAM) hosts this bot since 2026-10-09, shared
# with ExpenseSyncBot, HyperSummaryBot, Home Assistant and others: 1.5 CPUs leaves them
# headroom. RAM there is overcommitted (2 GB in swap on 2026-10-10, the idle bot down to
Expand Down
1 change: 1 addition & 0 deletions docs/reference/deploy-runbook.md
Original file line number Diff line number Diff line change
Expand Up @@ -319,6 +319,7 @@ If a release that the override accepted with `rollback-compatible: no` fails ver
- `docker compose` requires root (neither `www-data` nor `hcb-runner` is in the docker group).
- Resource limits: bot 1G/1.5cpu with a 384M reservation (cgroup `memory.low`), redis 256M/0.5cpu. The odroid has 6 cores and 3.7 GB RAM shared with the neighbours listed above and is overcommitted (on 2026-10-10 about 2 GB sat in swap and the idle bot was down to 71 MB resident); the reservation keeps the bot's working set out of reclaim so a message is not served from swap. Keep the bot well below the whole box.
- Bot container memory, measured 2026-11-04 (#963, #800). The image before the 0.3.1-0.3.3 deploys (ff93220a) and the current one (e4185ddf) have the same dependencies, Dockerfile, compose limits and module-load footprint (importing every startup module takes a bun process from 18 MB to 131-133 MB RSS on both, 29 MB of heap), so the jump from the 138-163 MiB seen on the old container is not a code change. A fresh e4185ddf container showed 413 MiB in `docker stats` one minute after start and 449-461 MiB over the next 16 minutes: flat, no growth, while the kernel swapped cold pages out (cgroup anon 405 MB falling to 363 MB, `memory.swap.current` 108 MB at minute 20, `memory.low` 384 MiB). Inside it the `bun` process held 328 MB anon plus 93 MB swapped; the five Chromium processes of the render pool (`playwrightPool.initialize()` starts them at boot) are mostly shared binary pages (the two measured held about 5 MB anon each). A container that has run for 28 h looks small because the host swapped its idle part out (the 71 MB of 2026-10-10 is the same effect): compare containers of the same age. The largest single item was the stress dictionary, 555,461 entries parsed at boot: in a bare bun process it takes RSS from 51 MB to 362 MB at the peak and leaves 190 MB, and production never uses it unless a model calls `lookup_stress` (Silero and the MTProto service tier are off there). It now loads on first use (#963), or at startup when `SILERO_PYTHON_PATH` is set (every Russian voice reply needs it then, and an unreadable file keeps Silero off as before); the load is one synchronous parse, so the first `lookup_stress` after a restart stalls the bot for about 2 s on a fast PC and longer on the odroid. Measure with `docker stats --no-stream hypercal-bot`, `grep -E 'RssAnon|VmSwap' /proc/<bun pid>/status` inside the container and `/sys/fs/cgroup/memory.current`, `memory.swap.current`, `memory.low`.
- Bot healthcheck grace (#980): the odroid's `autoheal` container (`willfarrell/autoheal`, `AUTOHEAL_CONTAINER_LABEL=all`, from the owner's service-failover module since 2026-11-10) restarts any container whose health turns `unhealthy`, and `/health` answers 503 `bot not started` until the Telegram start finishes. With `start_period: 30s` a cold start on the loaded host (95-170 s on 2026-11-11, 440 s after the hard reset of 2026-11-14) was killed and started from zero in a loop: five restarts between 17:25 and 17:34 UTC on 2026-11-14. The healthcheck now has `start_period: 10m` (failures inside it do not count) and `start_interval: 10s` (a fast start still turns healthy within seconds, so the deploy health wait is not slower); a bot that hangs after it has started is still caught within `retries` 3 x (`interval` 30 s + `timeout` 5 s). `test/scripts/bot-healthcheck-grace.test.ts` keeps both bounds.
- No deploy secrets: the SSH secrets `SSH_HOST`, `SSH_USER`, `SSH_KEY`, `DEPLOY_PATH` are unused since #784.

### Image identity
Expand Down
71 changes: 71 additions & 0 deletions test/scripts/bot-healthcheck-grace.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
// test/scripts/bot-healthcheck-grace.test.ts
//
// The odroid runs willfarrell/autoheal with AUTOHEAL_CONTAINER_LABEL=all: it restarts any
// container whose Docker health status turns `unhealthy`. The bot answers /health with 503
// "bot not started" until its Telegram start finishes, so a cold start slower than the
// healthcheck grace is killed and starts again from zero. Measured cold starts on the loaded
// host: ~95 s and 100-170 s on 2026-11-11 (restart loops, #980), and 440 s on 2026-11-14 right
// after the host's hard reset (container start 17:18:30 UTC, "Bot started" 17:25:50 UTC, killed
// at 17:25:59, 17:27:43 and 17:29:38 UTC). These tests keep the grace above the slowest
// measured start while a bot that hangs after it has started is still restarted within minutes.
import { describe, expect, test } from 'bun:test';
import { readFileSync } from 'node:fs';
import { join } from 'node:path';
import { z } from 'zod';

const COMPOSE_FILE = join(import.meta.dir, '../../docker-compose.yml');

const SLOWEST_MEASURED_COLD_START_S = 440;
/** A hung bot that had started must not stay up longer than this before autoheal acts. */
const STEADY_STATE_DETECTION_LIMIT_S = 120;

const healthcheckSchema = z.object({
test: z.array(z.string()),
interval: z.string(),
timeout: z.string(),
retries: z.number().int().positive(),
start_period: z.string(),
start_interval: z.string().optional(),
});

const composeSchema = z.object({
services: z.object({ bot: z.object({ healthcheck: healthcheckSchema }) }),
});

/** Compose duration ("30s", "10m", "1m30s") in seconds. */
function seconds(duration: string): number {
const parts = [...duration.matchAll(/(\d+)(h|m|s)/g)];
const consumed = parts.map((p) => p[0]).join('');
if (parts.length === 0 || consumed !== duration) throw new Error(`unsupported duration: ${duration}`);
const unit = { h: 3600, m: 60, s: 1 } as const;
return parts.reduce((total, [, value, u]) => total + Number(value) * unit[u as keyof typeof unit], 0);
}

function botHealthcheck() {
return composeSchema.parse(Bun.YAML.parse(readFileSync(COMPOSE_FILE, 'utf8'))).services.bot.healthcheck;
}

describe('bot healthcheck grace under autoheal (#980)', () => {
test('a cold start as slow as the slowest measured one never turns the container unhealthy', () => {
const hc = botHealthcheck();
// Failures inside start_period do not count; after it, `retries` failures `interval` apart do.
const grace = seconds(hc.start_period) + hc.retries * seconds(hc.interval);
expect(grace).toBeGreaterThan(SLOWEST_MEASURED_COLD_START_S);
// Margin over the measurement itself: a slower host must not loop again at the first excess.
expect(seconds(hc.start_period)).toBeGreaterThanOrEqual(SLOWEST_MEASURED_COLD_START_S);
});

test('a started bot that hangs is still marked unhealthy within the steady-state limit', () => {
const hc = botHealthcheck();
// A hanging probe runs until `timeout`, and Docker waits `interval` after each probe ends.
const worstDetection = hc.retries * (seconds(hc.interval) + seconds(hc.timeout));
expect(worstDetection).toBeLessThanOrEqual(STEADY_STATE_DETECTION_LIMIT_S);
});

test('a long grace does not delay a fast start: the start-period probe runs often', () => {
const hc = botHealthcheck();
// Without start_interval Docker probes the starting container only every `interval`.
expect(hc.start_interval).toBeDefined();
expect(seconds(hc.start_interval ?? hc.interval)).toBeLessThanOrEqual(10);
});
});
Loading