You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Odroid host capacity: unlimited wyoming voice stack (2026-11-09 18:18Z) thrashed the 3.7 GB host, dropped cloudflared's tunnel and 502'd /ready for ~7 min (alert down-1794249029) #977
Record the 2026-11-09 18:26-18:33 UTC outage and the recurrence risk: a wyoming voice stack (homeassistant-wyoming-openwakeword-1 + homeassistant-wyoming-satellite-1, mem=0 unlimited, --preload-model, recording 6ch 16kHz audio to /oww-dataset on the root SD card) started at 18:18:32Z on the 3.7 GB odroid that already runs ~30 containers with zram (1.8G) 100% full and ~1G of the swapfile on /mnt/storage in use. PSI at 18:35Z: io some avg300=47%, memory some avg300=39%, load 36.5 on 6 cores at near-zero CPU (reclaim/swap stall). sshd timed out during banner exchange; cloudflared lost QUIC (timeout: no recent network activity) and re-registered from 18:33Z, so the Cloudflare edge 502'd /ready (3 consecutive probe failures -> alert down-1794249029 at 18:30:29Z). hypercal-bot itself was healthy the whole window: image 128846b (main, #974) Up 29h, RestartCount=0, no OOM kill, zero log lines 18:25-18:33 (no traffic reached it); no CI deploy ran after 2026-11-08T13:21Z. Self-recovered by ~18:37Z (load 36->5, /ready 4x200 at 0.08-0.14s); nothing was changed on the server during investigation (read-only). Logins from ubuntu-pc at 18:13Z and 18:33Z: the voice stack was brought up by the owner's own work. Owner decisions recorded, not executed: (1) memory-limit both wyoming containers and move the dataset write off the SD card to /mnt/storage; (2) decide whether the prod bot keeps sharing the 3.7 GB host with the voice, HA and printops stacks (candidate: ubuntu-hyper); (3) optional PSI-based alert so the next thrash pages the owner before the tunnel dies.
Why (motivation)
Alert down-1794249029 fired on a 502 that the bot image did not cause; without a durable record the next host thrash gets re-investigated from scratch and the recurrence risk (unlimited containers on a host with no headroom) stays invisible
User impact
All bot users silent for ~7-10 min (18:26-18:33 UTC): Telegram webhooks failed at the edge (Telegram retries, so most messages arrived late); the bot wrote nothing in the window, no data risk. Same failure shape as #962 but host-starvation-triggered; will recur on the next memory spike. Deployed image was #974, current main.
Cost of inaction
The next voice-stack spike (model reload, HA activity, a deploy) re-thrashes the host and 502s the bot again, and each occurrence costs an investigation session and user-visible downtime
Acceptance criteria
Issue records root cause with the measured evidence (PSI, load, cloudflared QUIC loss, healthy bot container) and the three owner decisions
No production change made by the investigating session (read-only held)
Follow-up found during the completion self-check of the down-1794249029 session: the watcher's launched/ markers show a down-* incident at exactly 03:01 UTC almost every night from 2026-10-26 to 2026-11-07. Root of the cadence: root cron on the odroid runs backup-db.sh at 03:00 UTC daily (backup.log: 'Backup OK … 03-00-01', ~816K gz, no durations logged). The copy itself is seconds; the probe triad fails only when the host has no memory headroom — the two nights after the 2026-11-08 13:21Z reboot ran the same backup with no down incident. Same root condition as this ticket: zram full, ~1G swapfile in use, ~30 containers on 3.7 GB. Owner options to fold into decisions (1)/(3): nice/ionice the backup, log its duration, and/or alert on PSI so the nightly blip and this incident class share one fix — headroom, not per-symptom patches. Alert-delivery integrity check for today's outage (the /admin/alerts/next 'ended abruptly' in cloudflared at 18:33:01Z, the #932 scenario): watcher state.json shows consecutiveProbeFailures=0, downIncidentKey=null, lastProbeFailureAt=18:32:54Z; pending/ is empty and launched/ gained exactly one marker (down-1794249029 = this session) — no alert was consumed and lost.
What actually triggered 18:18:32Z (corrects the report's "voice stack was raised from ubuntu-pc"): nobody deployed the voice stack to the odroid. The satellite+openwakeword pair is the legacy HA Assist stack that has always run on the odroid next to its ReSpeaker 4-mic array; the new AISIS voice work does run on ubuntu-pc. At 19:18:09 local the TV was switched on and the HA automation voice_tv_noise_mode_on (packages/voice_assist_monitor.yaml on the odroid HA config) fired shell_command.voice_restart_services → host-actions bridge (ha-host-actions.py on the odroid host, POST /voice/restart-services) → docker compose up -d --force-recreate wyoming-openwakeword wyoming-satellite (bridge log: POST /voice/restart-services 200 at 19:18:36; dockerd journal: old containers stopped 19:18:16-27, new ones created 19:18:16, single compose traceID). The forced openwakeword model reload on a host with zram 100% full did the rest. The automation did this on every TV on/off; the ubuntu-pc logins at 18:13Z were only the post-SSD-reboot sshfs mount and the wyoming STT tunnel — correlation, not cause.
Fix applied (odroid, all reversible):
wyoming-openwakeword / wyoming-satellite renamed to *-disabled + profiles: ["disabled"] in the odroid's ha.docker-compose.yaml (backup kept next to it, *.bak-20261109-wyoming-off); both containers stopped and removed. Wake word has been non-functional since 2026-09-24 (--threshold 1.0 never fires), so no working feature was lost.
Automations voice_tv_noise_mode_on/off: initial_state: false + comment; reloaded via API (states now off). The two nc -z wyoming-* health sensors in the same package now report stopped instead of erroring every 30 s.
HA wyoming config entry for satellite:10700 disabled (was retry-looping). Untouched: ha-fallback-stt-wyoming-vosk-1 (the ubuntu-pc voice stack still tunnels to it, port 10300), home-lan dns/dhcp, hypercal-bot, cloudflared.
Post-fix: PSI io some avg10 ~7-9 (was 47), load ~1 (was 36), /ready 200 at 0.08-0.13s ×5. Decision (1) from the ticket (memory limits + dataset off SD) is now moot for this stack; it re-applies if the stack is ever re-enabled — the compose comment requires a wakebench-passing model AND a memory limit for that. The mic array is still physically on the odroid: moving it to the PC (spec 140, vad-ctl as the satellite) is the remaining owner action to fully retire voice from this host.
Cleanup addendum (do together with the mic-to-PC move, spec 140): the removal leaves dead config on the odroid — the voice_* shell_commands and the two disabled TV-noise automations in packages/voice_assist_monitor.yaml; the /voice/* allow-list routes in ha-host-actions.py; the respeaker-tuning service in ha.docker-compose.yaml; the wakeword_noise_watchdog.py cron (*/5, monitors a stack that no longer runs); and the data dirs (oww-models, oww-dataset, wyoming-debug). Harmless as-is, but delete them in the same change that retires the mic from this host so the dead wake path cannot be resurrected by accident. Also still open from this ticket: owner decisions (2) host placement and (3) PSI alerting — the nightly 03:00 UTC near-incidents (root cron backup-db.sh on a no-headroom host) share the same fix.
Follow-up bug noticed during the voice wind-down (pre-existing, not caused by the removal): the odroid host-actions bridge GET /health returns 500 (FileNotFoundError from the gnome-screensaver/dbus probe) — it did so on the old code before the /voice routes were stripped, and still does. All live routes (/kodi/, /network/, /projector/, /screen/) answer 200, and the HA command_line sensor that probes the PC-side bridge health is unaffected (different host). Fix when touching the bridge next: make /health degrade to a partial report instead of raising. Also minor leftovers kept as rollback points: two git stashes and branch voice-winddown in the homeassistant repo on the odroid, plus .bak-20261109-* copies of ha-host-actions.py (odroid) and failover.py / runtime.env (PC) — delete after a week of stable voice.
Odroid host-actions bridge: GET /health returned 500 because gdbus is not installed (FileNotFoundError). The probe now degrades to an "unavailable" field; /health and /network/summary return 200. Repo copy of the bridge synced to the deployed version (voice routes removed) and committed.
Rollback leftovers removed: the two voice-winddown git stashes, the bridge backup copy, the ubuntu-pc runtime.env backup and the whole 20261109 backups directory. Other stashes were left untouched. The local voice-winddown branch is merged but could not be deleted from the automation shell (removal guard); it is a one-line manual git branch -d voice-winddown.
ReSpeaker software detach is not possible without root (USB authorized flag is root-owned); physical unplug remains.
Quasar /devices on :31116: one of two Yandex stations (No. 39651) reports connected:false while No. 5183 is connected; this is the station's own cloud connection state, not an aisis defect.
State: 0 voice containers, HA 200, all aisis user units active.
Recurrence on 2026-11-14 (alert down-1794675948), host-side timeline from the odroid journal of the previous boot (times UTC):
15:48:59-15:49:01 the USB disk sda (UAS) aborted ~30 in-flight writes and was reset; the PC probe saw two 502s at 15:48:38-15:49:08 and recovered.
16:38:13 kodi.service started on demand (by ha-host-actions via sudo). 16:39:08 first systemd-journald: Under memory pressure, flushing caches, then repeatedly until the end of the boot; Docker health checks of several containers timed out from 16:40.
~16:59 a transmission download to /mnt/wd500 started (same disk as the Docker overlay rootfs).
17:01:50 first /ready timeout from the PC; 17:05:48 hard down (Cloudflare 502/530, cloudflared tunnel lost). sshd timed out in banner exchange, tailscale ping took 4.5 s while ICMP answered in 1.3 ms.
17:15:30 the journal ends without a shutdown sequence (hard reset; whether the hardware watchdog or a power cycle did it is not established). Boot 17:16:52; I/O pressure some avg10 65-88 % for ~15 min after boot.
Kodi's start is the only new heavy process right before the memory pressure began (55 s earlier); there is no per-process memory record (no atop/sysstat on the host), so the trigger is a correlation, not a measurement. No HCB deploy since 2026-11-13 09:08 UTC.
Recurrence on 2026-11-15 (alert down-1794768391), times UTC, deployed 90ee425 (no deploy since 2026-11-13):
18:25:05 kodi.service started (now bounded: MemoryHigh 900M, MemoryMax 1.2G); 18:26:08 first systemd-journald: Under memory pressure since 17:00.
18:30-18:46 the bot stalled: no log line between 18:30:07 and 18:46:50, BullMQ Missing lock for job repeat:… at 18:43-18:44, dockerd Could not send KILL signal to hypercal-bot at 18:36:32.
Host at 18:48-18:51: load 27 on 6 cores, swap 2.2 GB (zram 1.6/1.8 GB), PSI memory full avg300 36 %, io full avg300 53 %. Largest RSS: Home Assistant ~950 MB; hypercal-bot ~190 MB of its 1 GiB, OOMKilled=false.
Again Kodi's start is the only new heavy process right before the pressure (~1 min); a correlation, not a per-process measurement.
Side effect: Playwright failed to launch in the surviving start, so pictures are off until a restart: #985
Second recurrence on 2026-11-15 (alert-188-20261115194711, google_refresh_lock recovered), times UTC, deployed 90ee425 (no deploy since 2026-11-13):
19:33:13 kodi.service started again on demand (ha-host-actions via sudo); 19:35:09 first systemd-journald: Under memory pressure (24 such lines 19:15-19:50, all after the kodi start).
19:35:49 bot SIGTERM (autoheal); 19:36:15 cloudflared Unable to reach the origin service.
Third recurrence on 2026-11-15 (alert down-1794776555), times UTC, deployed 90ee425 (no deploy since 2026-11-13). Different trigger from the two earlier waves: kodi.service was inactive; the Stage QA run for PR #987 (head 6db890c, run 43759958385) was the trigger.
20:46:50 Stage QA job ran sudo hypercal-stage-qa 6db890c4… on the odroid; 20:49:08 first systemd-journald: Under memory pressure since 20:30 (14 lines to 20:51, again from 21:00).
20:59:30 stage bot container started; 21:02:10 first hypercal-bot health check timed out; 21:02:35 public /ready 502 (3 probes), cloudflared Unable to reach the origin service / context canceled.
21:04-21:06: load 27, CPU 63-83 % iowait, PSI io full avg300 ≈ 61 %, memory full avg300 ≈ 30 %; zram 1.5/1.8 G, swapfile on /mnt/storage 675 M used. Largest cgroups: Home Assistant 1022 M, pm2-www-data 764 M (614 M swapped), hypercal-bot 373 M, odroid-hcb-qa runner 221 M.
autoheal restarted hypercal-bot at 21:02:52 (failed), 21:03:47, 21:06:09, plus printops-api and homeassistant-esphome-1; the bot's shutdown hit Shutdown timeout after 8s (level 60) at 21:03:31. Cold starts during the thrash answered bot not started 503 / connection reset.
21:04:22 stage release failed (evidence /opt/hypercal/stage/evidence/failed-6db890c434ae-2026-11-15_21-04-22/), stage container removed, run ended failure at 21:07:31.
21:08 recovered without intervention: public /ready ok (unverified) 200 ×3, hypercal-bot healthy, PSI io avg10 6 %.
Outage ≈ 21:02-21:08 (≈6 min). Next action: Stage QA should not start (or should abort) while the odroid is already under memory pressure (MemAvailable / PSI gate in stage-qa-run.sh), in addition to PR #984 (autoheal start grace for #980), which is still open and unmerged.
Activity
alex-mextner commentedon Nov 9, 2026
Follow-up found during the completion self-check of the down-1794249029 session: the watcher's launched/ markers show a down-* incident at exactly 03:01 UTC almost every night from 2026-10-26 to 2026-11-07. Root of the cadence: root cron on the odroid runs backup-db.sh at 03:00 UTC daily (backup.log: 'Backup OK … 03-00-01', ~816K gz, no durations logged). The copy itself is seconds; the probe triad fails only when the host has no memory headroom — the two nights after the 2026-11-08 13:21Z reboot ran the same backup with no down incident. Same root condition as this ticket: zram full, ~1G swapfile in use, ~30 containers on 3.7 GB. Owner options to fold into decisions (1)/(3): nice/ionice the backup, log its duration, and/or alert on PSI so the nightly blip and this incident class share one fix — headroom, not per-symptom patches. Alert-delivery integrity check for today's outage (the /admin/alerts/next 'ended abruptly' in cloudflared at 18:33:01Z, the #932 scenario): watcher state.json shows consecutiveProbeFailures=0, downIncidentKey=null, lastProbeFailureAt=18:32:54Z; pending/ is empty and launched/ gained exactly one marker (down-1794249029 = this session) — no alert was consumed and lost.
alex-mextner commentedon Nov 9, 2026
Root cause found and fixed (2026-11-09 ~21:10Z).
What actually triggered 18:18:32Z (corrects the report's "voice stack was raised from ubuntu-pc"): nobody deployed the voice stack to the odroid. The satellite+openwakeword pair is the legacy HA Assist stack that has always run on the odroid next to its ReSpeaker 4-mic array; the new AISIS voice work does run on ubuntu-pc. At 19:18:09 local the TV was switched on and the HA automation
voice_tv_noise_mode_on(packages/voice_assist_monitor.yaml on the odroid HA config) firedshell_command.voice_restart_services→ host-actions bridge (ha-host-actions.pyon the odroid host, POST /voice/restart-services) →docker compose up -d --force-recreate wyoming-openwakeword wyoming-satellite(bridge log:POST /voice/restart-services 200at 19:18:36; dockerd journal: old containers stopped 19:18:16-27, new ones created 19:18:16, single compose traceID). The forced openwakeword model reload on a host with zram 100% full did the rest. The automation did this on every TV on/off; the ubuntu-pc logins at 18:13Z were only the post-SSD-reboot sshfs mount and the wyoming STT tunnel — correlation, not cause.Fix applied (odroid, all reversible):
wyoming-openwakeword/wyoming-satelliterenamed to*-disabled+profiles: ["disabled"]in the odroid's ha.docker-compose.yaml (backup kept next to it,*.bak-20261109-wyoming-off); both containers stopped and removed. Wake word has been non-functional since 2026-09-24 (--threshold 1.0never fires), so no working feature was lost.voice_tv_noise_mode_on/off:initial_state: false+ comment; reloaded via API (states nowoff). The twonc -z wyoming-*health sensors in the same package now reportstoppedinstead of erroring every 30 s.ha-fallback-stt-wyoming-vosk-1(the ubuntu-pc voice stack still tunnels to it, port 10300), home-lan dns/dhcp, hypercal-bot, cloudflared.Post-fix: PSI io some avg10 ~7-9 (was 47), load ~1 (was 36), /ready 200 at 0.08-0.13s ×5. Decision (1) from the ticket (memory limits + dataset off SD) is now moot for this stack; it re-applies if the stack is ever re-enabled — the compose comment requires a wakebench-passing model AND a memory limit for that. The mic array is still physically on the odroid: moving it to the PC (spec 140, vad-ctl as the satellite) is the remaining owner action to fully retire voice from this host.
alex-mextner commentedon Nov 9, 2026
Cleanup addendum (do together with the mic-to-PC move, spec 140): the removal leaves dead config on the odroid — the
voice_*shell_commands and the two disabled TV-noise automations in packages/voice_assist_monitor.yaml; the/voice/*allow-list routes in ha-host-actions.py; therespeaker-tuningservice in ha.docker-compose.yaml; thewakeword_noise_watchdog.pycron (*/5, monitors a stack that no longer runs); and the data dirs (oww-models, oww-dataset, wyoming-debug). Harmless as-is, but delete them in the same change that retires the mic from this host so the dead wake path cannot be resurrected by accident. Also still open from this ticket: owner decisions (2) host placement and (3) PSI alerting — the nightly 03:00 UTC near-incidents (root cron backup-db.sh on a no-headroom host) share the same fix.alex-mextner commentedon Nov 9, 2026
Follow-up bug noticed during the voice wind-down (pre-existing, not caused by the removal): the odroid host-actions bridge
GET /healthreturns 500 (FileNotFoundErrorfrom the gnome-screensaver/dbus probe) — it did so on the old code before the /voice routes were stripped, and still does. All live routes (/kodi/, /network/, /projector/, /screen/) answer 200, and the HA command_line sensor that probes the PC-side bridge health is unaffected (different host). Fix when touching the bridge next: make /health degrade to a partial report instead of raising. Also minor leftovers kept as rollback points: two git stashes and branchvoice-winddownin the homeassistant repo on the odroid, plus.bak-20261109-*copies of ha-host-actions.py (odroid) and failover.py / runtime.env (PC) — delete after a week of stable voice.alex-mextner commentedon Nov 9, 2026
Finish pass after voice wind-down:
git branch -d voice-winddown.alex-mextner commentedon Nov 14, 2026
Recurrence on 2026-11-14 (alert down-1794675948), host-side timeline from the odroid journal of the previous boot (times UTC):
sda(UAS) aborted ~30 in-flight writes and was reset; the PC probe saw two 502s at 15:48:38-15:49:08 and recovered.kodi.servicestarted on demand (by ha-host-actions via sudo). 16:39:08 firstsystemd-journald: Under memory pressure, flushing caches, then repeatedly until the end of the boot; Docker health checks of several containers timed out from 16:40./mnt/wd500started (same disk as the Docker overlay rootfs)./readytimeout from the PC; 17:05:48 hard down (Cloudflare 502/530, cloudflared tunnel lost). sshd timed out in banner exchange,tailscale pingtook 4.5 s while ICMP answered in 1.3 ms.some avg1065-88 % for ~15 min after boot./ready200) after five autoheal restarts, tracked in autoheal kills hypercal-bot mid cold start: /health is 503 until the bot has started (~95 s on a loaded odroid) but start_period is 30 s #980.Kodi's start is the only new heavy process right before the memory pressure began (55 s earlier); there is no per-process memory record (no atop/sysstat on the host), so the trigger is a correlation, not a measurement. No HCB deploy since 2026-11-13 09:08 UTC.
alex-mextner commentedon Nov 15, 2026
Recurrence on 2026-11-15 (alert down-1794768391), times UTC, deployed 90ee425 (no deploy since 2026-11-13):
kodi.servicestarted (now bounded: MemoryHigh 900M, MemoryMax 1.2G); 18:26:08 firstsystemd-journald: Under memory pressuresince 17:00.Missing lock for job repeat:…at 18:43-18:44, dockerdCould not send KILL signalto hypercal-bot at 18:36:32.Again Kodi's start is the only new heavy process right before the pressure (~1 min); a correlation, not a per-process measurement.
Side effect: Playwright failed to launch in the surviving start, so pictures are off until a restart: #985
alex-mextner commentedon Nov 15, 2026
Second recurrence on 2026-11-15 (alert-188-20261115194711, google_refresh_lock recovered), times UTC, deployed 90ee425 (no deploy since 2026-11-13):
kodi.servicestarted again on demand (ha-host-actions via sudo); 19:35:09 firstsystemd-journald: Under memory pressure(24 such lines 19:15-19:50, all after the kodi start).Unable to reach the origin service.AI request metricbetween 19:02:20 and 19:47: no recorded user turn in the outage window.The kodi memory bound alone does not keep the host up: kodi at 332 MB was enough with Home Assistant at ~1 GB and 2.2 GB already in swap.
alex-mextner commentedon Nov 15, 2026
Third recurrence on 2026-11-15 (alert down-1794776555), times UTC, deployed 90ee425 (no deploy since 2026-11-13). Different trigger from the two earlier waves: kodi.service was inactive; the Stage QA run for PR #987 (head 6db890c, run 43759958385) was the trigger.
sudo hypercal-stage-qa 6db890c4…on the odroid; 20:49:08 firstsystemd-journald: Under memory pressuresince 20:30 (14 lines to 20:51, again from 21:00).Unable to reach the origin service/context canceled.Shutdown timeout after 8s(level 60) at 21:03:31. Cold starts during the thrash answeredbot not started503 / connection reset./opt/hypercal/stage/evidence/failed-6db890c434ae-2026-11-15_21-04-22/), stage container removed, run endedfailureat 21:07:31.ok (unverified)200 ×3, hypercal-bot healthy, PSI io avg10 6 %.Outage ≈ 21:02-21:08 (≈6 min). Next action: Stage QA should not start (or should abort) while the odroid is already under memory pressure (MemAvailable / PSI gate in stage-qa-run.sh), in addition to PR #984 (autoheal start grace for #980), which is still open and unmerged.