You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Odroid host memory pressure makes every HyperCalendarBot image render 6-15 s instead of ~1-2 s #800
Re-filed from the startup tracker, where it was created by mistake (was HYP-1675, deleted 2026-10-20).
On 2026-10-12 the hypercal-bot cgroup had 501 MB in swap (zram + microSD swapfile), the bot Bun process 444 MB swapped, 985k major faults; host had 93 MB free (Home Assistant, ExpenseSyncBot x2, runners share 3.7 GB). The same daily template renders in 0.8-1.8 s warm in the same container, but production renders take 5.9-15.2 s. Decide on more RAM, fewer co-tenants, or memory reservation for hypercal-bot.
Why (motivation)
HYP-1671 widened the render wait to 45 s so images stop being dropped, but the underlying slowness remains.
User impact
People who ask the bot for their day, week or month wait 6-15 seconds for the picture after the text arrives, and sometimes give up before it shows.
Cost of inaction
Renders keep getting slower as host memory pressure grows, and could exceed even the 45 s wait.
Acceptance criteria
Production renderTimeMs for daily-agenda stays under 5 s over a week of normal traffic (log evidence), or a documented decision to accept the current latency.
The chosen remedy (RAM, co-tenancy or memory reservation) is recorded in docs/reference/deploy-runbook.md with the before/after render times.
Another occurrence, found by the alert watcher session hcb-alert-20261024T222229Z (read-only): alerts 132 and 133, 2026-10-20 18:17:27 and 18:17:28 UTC, image-render jobs 550 and 549, content: Timeout 30000ms exceeded ... waiting until "load" from processRenderJob (src/worker/image-render.queue.ts:47, Playwright's default 30 s on page.setContent(html, { waitUntil: 'load' })).
The templates in src/worker/templates load no external URL, so a 30 s load wait means Chromium was starved, not waiting on a network. The odroid journal for 20:00-20:30 CEST that day has systemd-journald: Under memory pressure, flushing caches in almost every minute (4 lines at 20:16, 2 at 20:17): the same memory pressure this issue describes. Two jobs failing one second apart (queue concurrency 4) fits a starved browser rather than two bad payloads.
Nothing since: the current container (started 2026-10-24 21:11 UTC) has logged no render yet. The 30 s render timeout is below the 45 s caller wait from HYP-1671, so it fires first; raising or retrying it is a separate decision from fixing the memory pressure.
Datapoint from unattended session hcb-alert-20261104T105209Z (synthetic alert, no incident). Host odroidn2 at 2026-11-04T10:54:58Z: swap used 2419 of 5976 MB, available RAM 867 of 3761 MB, load 0.93/1.14/1.72, IO PSI full avg10=0.10 avg60=1.16 avg300=6.16, CPU/memory full ~0. Caveat: sampled 10 minutes after the e4185dd deploy (container started 10:44:05Z), so the avg300 IO figure may be deploy-induced. No render job or agent run was examined. Counts only, no user data.
Shipped: merged via #965.
Merge commit: bbedb86
This does NOT close the ticket — acceptance still needs proof. Run task read #800 for the outstanding criteria, then task check/task done.
Progress on this ticket (2026-11-04): PR #965 (#963, stress dictionary loaded on first use instead of at every boot) is merged as main bbedb86 and deployed (receipt names it, restarts 0, /health and /ready ok). The bot process holds about 80 MB less anonymous memory than before (247 MB against 328 MB, flat over 15 minutes); details and the numbers are in #963. This ticket stays open: its two criteria need a week of render-time logs and the recorded before/after render times of the chosen remedy (post-merge acceptance recorded for exactly that reason). Follow-up of the PR's review: #966.
Evidence from the unattended session for the synthetic alert alert-142-20261105050615 (no incident; read-only checks, revision 6b7ee71, which contains #965's lazy stress dictionary). Sampled 2026-11-05 about 05:07Z, roughly 4 minutes after the container's cold start at 05:03:32Z:
free -m: 3761 MB total, 2799 used, 106 free, 962 available; swap 2559 of 5976 MB in use.
/proc/pressure/io full: avg10 0.93, avg60 2.89, avg300 17.28. /proc/pressure/memory full: avg300 2.58 (avg10 0.00). The 5-minute averages cover the cold start; the 10-second ones had already dropped.
Render queue now: wait 0, prioritized 0. The failed set still holds 3 jobs from 2026-10-20 (ids 547, 548, 551), failedReason "job stalled more than allowable limit". They outlive the configured removeOnFail: { age: 3600 }; [INFERENCE] BullMQ trims that set only when another job fails, so this is lazy trimming, not a leak.
So the host still swaps heavily right after a deploy. Not attributed to any request: the alert's requestId is in no log.
Data point from the unattended watcher session hcb-alert-20261110T023129Z (provider-timeout alerts for one turn on 2026-11-10 02:16Z). Host measured at 02:31Z, 15 min AFTER the timeouts, so it is not a time-series: load average 2.08, PSI io some avg300 5.46%/full 4.92%, PSI memory some avg300 0.60%, swap 2278 of 5976 MB used, RAM available 806 of 3761 MB. In the same turn Claude answered in 3.0-3.3 s on this host, so the host did not slow those provider calls; the cause is z.ai's 15 s cap, tracked in #978. Swap use stays as the known state described here.
Activity
alex-mextner commentedon Oct 24, 2026
Another occurrence, found by the alert watcher session hcb-alert-20261024T222229Z (read-only): alerts 132 and 133, 2026-10-20 18:17:27 and 18:17:28 UTC, image-render jobs 550 and 549,
content: Timeout 30000ms exceeded ... waiting until "load"fromprocessRenderJob(src/worker/image-render.queue.ts:47, Playwright's default 30 s onpage.setContent(html, { waitUntil: 'load' })).The templates in
src/worker/templatesload no external URL, so a 30 sloadwait means Chromium was starved, not waiting on a network. The odroid journal for 20:00-20:30 CEST that day hassystemd-journald: Under memory pressure, flushing cachesin almost every minute (4 lines at 20:16, 2 at 20:17): the same memory pressure this issue describes. Two jobs failing one second apart (queue concurrency 4) fits a starved browser rather than two bad payloads.Nothing since: the current container (started 2026-10-24 21:11 UTC) has logged no render yet. The 30 s render timeout is below the 45 s caller wait from HYP-1671, so it fires first; raising or retrying it is a separate decision from fixing the memory pressure.
alex-mextner commentedon Nov 4, 2026
Datapoint from unattended session hcb-alert-20261104T105209Z (synthetic alert, no incident). Host odroidn2 at 2026-11-04T10:54:58Z: swap used 2419 of 5976 MB, available RAM 867 of 3761 MB, load 0.93/1.14/1.72, IO PSI full avg10=0.10 avg60=1.16 avg300=6.16, CPU/memory full ~0. Caveat: sampled 10 minutes after the e4185dd deploy (container started 10:44:05Z), so the avg300 IO figure may be deploy-induced. No render job or agent run was examined. Counts only, no user data.
fix(memory): the 555K-entry stress dictionary loads on first use inst…
alex-mextner commentedon Nov 4, 2026
Shipped: merged via #965.
Merge commit: bbedb86
This does NOT close the ticket — acceptance still needs proof. Run
task read #800for the outstanding criteria, thentask check/task done.alex-mextner commentedon Nov 4, 2026
Progress on this ticket (2026-11-04): PR #965 (#963, stress dictionary loaded on first use instead of at every boot) is merged as main bbedb86 and deployed (receipt names it, restarts 0, /health and /ready ok). The bot process holds about 80 MB less anonymous memory than before (247 MB against 328 MB, flat over 15 minutes); details and the numbers are in #963. This ticket stays open: its two criteria need a week of render-time logs and the recorded before/after render times of the chosen remedy (post-merge acceptance recorded for exactly that reason). Follow-up of the PR's review: #966.
alex-mextner commentedon Nov 5, 2026
Evidence from the unattended session for the synthetic alert alert-142-20261105050615 (no incident; read-only checks, revision 6b7ee71, which contains #965's lazy stress dictionary). Sampled 2026-11-05 about 05:07Z, roughly 4 minutes after the container's cold start at 05:03:32Z:
free -m: 3761 MB total, 2799 used, 106 free, 962 available; swap 2559 of 5976 MB in use./proc/pressure/iofull: avg10 0.93, avg60 2.89, avg300 17.28./proc/pressure/memoryfull: avg300 2.58 (avg10 0.00). The 5-minute averages cover the cold start; the 10-second ones had already dropped.removeOnFail: { age: 3600 }; [INFERENCE] BullMQ trims that set only when another job fails, so this is lazy trimming, not a leak.So the host still swaps heavily right after a deploy. Not attributed to any request: the alert's requestId is in no log.
alex-mextner commentedon Nov 10, 2026
Data point from the unattended watcher session hcb-alert-20261110T023129Z (provider-timeout alerts for one turn on 2026-11-10 02:16Z). Host measured at 02:31Z, 15 min AFTER the timeouts, so it is not a time-series: load average 2.08, PSI io some avg300 5.46%/full 4.92%, PSI memory some avg300 0.60%, swap 2278 of 5976 MB used, RAM available 806 of 3761 MB. In the same turn Claude answered in 3.0-3.3 s on this host, so the host did not slow those provider calls; the cause is z.ai's 15 s cap, tracked in #978. Swap use stays as the known state described here.