Skip to content

Odroid host memory pressure makes every HyperCalendarBot image render 6-15 s instead of ~1-2 s #800

Open

Description

@alex-mextner

What

Re-filed from the startup tracker, where it was created by mistake (was HYP-1675, deleted 2026-10-20).

On 2026-10-12 the hypercal-bot cgroup had 501 MB in swap (zram + microSD swapfile), the bot Bun process 444 MB swapped, 985k major faults; host had 93 MB free (Home Assistant, ExpenseSyncBot x2, runners share 3.7 GB). The same daily template renders in 0.8-1.8 s warm in the same container, but production renders take 5.9-15.2 s. Decide on more RAM, fewer co-tenants, or memory reservation for hypercal-bot.

Why (motivation)

HYP-1671 widened the render wait to 45 s so images stop being dropped, but the underlying slowness remains.

User impact

People who ask the bot for their day, week or month wait 6-15 seconds for the picture after the text arrives, and sometimes give up before it shows.

Cost of inaction

Renders keep getting slower as host memory pressure grows, and could exceed even the 45 s wait.

Acceptance criteria

  • Production renderTimeMs for daily-agenda stays under 5 s over a week of normal traffic (log evidence), or a documented decision to accept the current latency.
  • The chosen remedy (RAM, co-tenancy or memory reservation) is recorded in docs/reference/deploy-runbook.md with the before/after render times.

Screenshots

(none)

Deploy

Deployed: no

Links

Skipped gates

Activity

  1. alex-mextner commented on Oct 24, 2026

    @alex-mextner
    OwnerAuthor

    Another occurrence, found by the alert watcher session hcb-alert-20261024T222229Z (read-only): alerts 132 and 133, 2026-10-20 18:17:27 and 18:17:28 UTC, image-render jobs 550 and 549, content: Timeout 30000ms exceeded ... waiting until "load" from processRenderJob (src/worker/image-render.queue.ts:47, Playwright's default 30 s on page.setContent(html, { waitUntil: 'load' })).

    The templates in src/worker/templates load no external URL, so a 30 s load wait means Chromium was starved, not waiting on a network. The odroid journal for 20:00-20:30 CEST that day has systemd-journald: Under memory pressure, flushing caches in almost every minute (4 lines at 20:16, 2 at 20:17): the same memory pressure this issue describes. Two jobs failing one second apart (queue concurrency 4) fits a starved browser rather than two bad payloads.

    Nothing since: the current container (started 2026-10-24 21:11 UTC) has logged no render yet. The 30 s render timeout is below the 45 s caller wait from HYP-1671, so it fires first; raising or retrying it is a separate decision from fixing the memory pressure.

  2. alex-mextner commented on Nov 4, 2026

    @alex-mextner
    OwnerAuthor

    Datapoint from unattended session hcb-alert-20261104T105209Z (synthetic alert, no incident). Host odroidn2 at 2026-11-04T10:54:58Z: swap used 2419 of 5976 MB, available RAM 867 of 3761 MB, load 0.93/1.14/1.72, IO PSI full avg10=0.10 avg60=1.16 avg300=6.16, CPU/memory full ~0. Caveat: sampled 10 minutes after the e4185dd deploy (container started 10:44:05Z), so the avg300 IO figure may be deploy-induced. No render job or agent run was examined. Counts only, no user data.

  3. alex-mextner commented on Nov 4, 2026

    @alex-mextner
    OwnerAuthor

    Shipped: merged via #965.
    Merge commit: bbedb86
    This does NOT close the ticket — acceptance still needs proof. Run task read #800 for the outstanding criteria, then task check/task done.

  4. alex-mextner commented on Nov 4, 2026

    @alex-mextner
    OwnerAuthor

    Progress on this ticket (2026-11-04): PR #965 (#963, stress dictionary loaded on first use instead of at every boot) is merged as main bbedb86 and deployed (receipt names it, restarts 0, /health and /ready ok). The bot process holds about 80 MB less anonymous memory than before (247 MB against 328 MB, flat over 15 minutes); details and the numbers are in #963. This ticket stays open: its two criteria need a week of render-time logs and the recorded before/after render times of the chosen remedy (post-merge acceptance recorded for exactly that reason). Follow-up of the PR's review: #966.

  5. alex-mextner commented on Nov 5, 2026

    @alex-mextner
    OwnerAuthor

    Evidence from the unattended session for the synthetic alert alert-142-20261105050615 (no incident; read-only checks, revision 6b7ee71, which contains #965's lazy stress dictionary). Sampled 2026-11-05 about 05:07Z, roughly 4 minutes after the container's cold start at 05:03:32Z:

    • free -m: 3761 MB total, 2799 used, 106 free, 962 available; swap 2559 of 5976 MB in use.
    • /proc/pressure/io full: avg10 0.93, avg60 2.89, avg300 17.28. /proc/pressure/memory full: avg300 2.58 (avg10 0.00). The 5-minute averages cover the cold start; the 10-second ones had already dropped.
    • Render queue now: wait 0, prioritized 0. The failed set still holds 3 jobs from 2026-10-20 (ids 547, 548, 551), failedReason "job stalled more than allowable limit". They outlive the configured removeOnFail: { age: 3600 }; [INFERENCE] BullMQ trims that set only when another job fails, so this is lazy trimming, not a leak.

    So the host still swaps heavily right after a deploy. Not attributed to any request: the alert's requestId is in no log.

  6. alex-mextner commented on Nov 10, 2026

    @alex-mextner
    OwnerAuthor

    Data point from the unattended watcher session hcb-alert-20261110T023129Z (provider-timeout alerts for one turn on 2026-11-10 02:16Z). Host measured at 02:31Z, 15 min AFTER the timeouts, so it is not a time-series: load average 2.08, PSI io some avg300 5.46%/full 4.92%, PSI memory some avg300 0.60%, swap 2278 of 5976 MB used, RAM available 806 of 3761 MB. In the same turn Claude answered in 3.0-3.3 s on this host, so the host did not slow those provider calls; the cause is z.ai's 15 s cap, tracked in #978. Swap use stays as the known state described here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions