Skip to content

The odroid's Ethernet link drops every night at 03:00 local for ~85 s: 530 on the public host and a false DOWN incident #848

Open

Description

@alex-mextner

What

Every night at 03:00:07-10 Europe/Berlin the kernel on root@odroidn2 logs 'meson8b-dwmac end0: Link is Down', systemd-networkd 'Lost carrier / DHCP lease lost', then 'Link is Up - 100Mbps' after a 1 Gbps->100 Mbps PHY downshift ('check cabling!'), a second Down at 03:00:40-45 and the final Up at 03:01:11. cloudflared logs 'sendmsg: network is unreachable' for all four edge connections and re-registers at 03:01:34 (02:01:34Z on 2026-10-26). While the tunnel is down hypercal.mextner.com and mextner.com answer 530 (Cloudflare error 1033); both were 200 again by 02:01:59Z. journalctl -k shows 20 Link-is-Down lines = 10 nights, one per night since the host booted on 2026-10-16, none at any other time. The hypercal-bot container is untouched: started 2026-10-25 23:46:06Z, healthy, RestartCount=0, no log line of level >=40 in the last 2 h, loopback /ready 200 when probed at 02:01:47Z (after the link was back; it was not probed during the drop). Nothing on the host explains the drop: the 03:00 cron jobs are the two local SQLite backups (HCB root, ESB www-data; no network use, finished within ~1 s of the start) plus */2 and */5 jobs that also run at other minutes; the drop is 6-10 s after the start with jitter. Open question: a scheduled event on the router/switch side, or the cable/port (the downshift to 100 Mbps after every re-link suggests a bad pair). The link is still at 100 Mbps now. The alert watcher opened its first incident for it at 2026-10-26 02:01:12Z (alert down-1792980012); whether anything probed it on earlier nights is not established.

Why (motivation)

A nightly outage window of ~85 s recurs on a schedule, so it is a defect to remove or to account for, not a transient. Each night it also meets the alert watcher's threshold (3 failed probes x 30 s) and opens one unattended omp investigation session for a bot that is healthy.

User impact

Telegram users and Google webhooks see 530 for ~85 s at 03:00 Europe/Berlin every night (Telegram retries webhooks; no lost message or write was observed in the bot logs for 2026-10-26). The owner gets a false DOWN page and a burned agent session per night.

Cost of inaction

A real outage at 03:00 is indistinguishable from the nightly one; the link stays degraded to 100 Mbps; a deploy or the 03:00 backup that overlaps the window can fail on a lost link; the owner learns to ignore DOWN pages, the same erosion as #843 .

Acceptance criteria

  • The trigger of the nightly 03:00 link drop is identified with evidence (router/switch log or schedule, or a cable/port swap test) and recorded on this issue
  • journalctl -k on root@odroidn2 shows no 'end0: Link is Down' at 03:00 for three consecutive nights and the link negotiates 1000 Mb/s, or the owner records the recurrence as accepted
  • The 03:00 window no longer opens a DOWN incident and an unattended session (cause removed, or HCB_ALERT_DOWN_PROBES / cross-check against a host-level signal decided by the owner), and a real outage still does

Screenshots

(none)

Deploy

Deployed: no

Links

Skipped gates

  • screenshots: ops incident, no UI

Activity

  1. alex-mextner commented on Oct 26, 2026

    @alex-mextner
    OwnerAuthor

    Verified 2026-10-26 02:07Z with journalctl -k on root@odroidn2 (journal starts at boot, 2026-10-16 21:43 CEST): 'end0: Link is Down' occurs exactly twice on each of 10 nights, 2026-10-17 through 2026-10-26, at 03:00:07-10 and 03:00:40-45 local time, and at no other time of day. The wall-clock time did not move when Europe/Berlin left DST on 2026-10-25 (03:00 +02:00 = 01:00Z on 10-24, 03:00 +01:00 = 02:00Z on 10-25 and 10-26), so the trigger follows local time, not UTC. That fits a local-time schedule on the router/switch or on the host, and it rules out a UTC-timed external job. The host's only 03:00-specific jobs are the two SQLite backups, which use no network. The drop starts 6-10 s after cron fires, which a router-side schedule explains at least as well. Not yet checked: the router/switch logs (outside the read-only scope of the alert session), and whether the other services on this host (ExpenseSyncBot, HyperSummaryBot, Home Assistant) were also unreachable in the window; every public name behind the odroid-home tunnel answered 530 (mextner.com confirmed).

  2. alex-mextner commented on Oct 27, 2026

    @alex-mextner
    OwnerAuthor

    Recurrence 2026-10-27 (alert down-1793066420, session hcb-alert-20261027T020120Z), read-only evidence:

    • Link Down 03:00:08 and 03:00:40 local, Up (100 Mbps downshift) 03:00:35 and 03:01:11; cloudflared re-registered 4 connections at 02:02:13Z (03:02:13 local). Public /ready was 530 until then, 200 after; the bot container (image 50c6ba9, restarts=0, healthy) and its logs (no level >= 50) were untouched.
    • Host-side trigger checked and not found: the only jobs scheduled at 03:00 on the odroid are the two local SQLite backups (root /opt/hypercal/scripts/backup-db.sh, www-data ExpenseSyncBot backup-db.sh) plus fake-hwclock-save. The hypercal backup-db.sh is a local VACUUM INTO and logged "Backup OK" at 03:00:02, six seconds before the first drop. The other cron jobs that fire at 03:00 (ha_fail_safe_cron.sh, dataplicity_watchdog.py, wakeword_noise_watchdog.py, storage-sftp/sftp-start.sh, diskbench/collect.sh, armbian-truncate-logs) run every 2-20 minutes, not at 03:00 only, and none of them contains an ethtool/ip link/networkctl/ifdown/reboot/modprobe command (grep). Nothing on the host explains a drop at exactly 03:00 local; the remaining suspects are the router/switch schedule and the cable/port (PHY downshift 1 Gbps -> 100 Mbps; ethtool end0 still says 100Mb/s).
    • Not checked (needs the owner): the router/switch log or schedule at 03:00, and a cable/port swap.
  3. alex-mextner commented on Oct 28, 2026

    @alex-mextner
    OwnerAuthor

    Recurrence 2026-10-28 (alert down-1793152822, session hcb-alert-20261028T020122Z), read-only evidence:

    • end0 Lost carrier 03:00:09 and 03:00:40 local, Gained carrier 03:00:35 and 03:01:12, DHCP re-acquired 03:01:16. cloudflared re-registered all 4 connections at 02:02:54Z (03:02:54 local), about 98 s after the link was back (2026-10-26: ~23 s), so the outage was ~2m45s. Public /ready was 530 until 02:03:42Z probe, 200 after.
    • Bot image 50c6ba9, restarts=0, healthy, no deploy. During the window it logged getaddrinfo ETIMEOUT for api.telegram.org, www.googleapis.com and api.openweathermap.org; morning_agenda jobs 315626/315627 failed, were retried and logged "Notification sent" at 02:01:54Z.
    • Router/switch log and cable/port swap still not checked (needs the owner). Separate finding: the bot token is logged in these errors, see Security: Telegram bot token is written to the container log inside failed-fetch error paths (api.telegram.org ETIMEOUT) #878.
  4. alex-mextner commented on Oct 29, 2026

    @alex-mextner
    OwnerAuthor

    Recurrence on 2026-10-29 (unattended session hcb-alert-20261029T020108Z, watcher alert down-1793239208).

    • end0 kernel: Link Down 02:00:08Z, Up 02:00:35Z (1 Gbps -> 100 Mbps downshift, "check cabling!"), Down 02:00:40Z, Up 02:01:11Z; DHCP re-acquired 02:01:16Z.
    • cloudflared: 16 network is unreachable errors 02:00:30-02:00:55Z, all four connections re-registered 02:01:23Z. /ready returned 530 (error 1033) from the 02:00:08Z probe and 200 again by 02:01:23Z.
    • hypercal-bot untouched (Up 42 min, healthy, 0 restarts, not OOM-killed); deployed revision 0ddc7f2 was verified 01:20Z, 40 min earlier, so no deploy involved.
    • Effect on the bot: 14 level-50 log lines in the window, all getaddrinfo ETIMEOUT (api.telegram.org, www.googleapis.com, api.openweathermap.org). Two morning_agenda jobs (317081, 317082) failed, were retried by the bot and logged "Notification sent" at 02:02:03Z.
    • The watcher raised a false DOWN incident and started an omp session again. Nothing was restarted or changed.

    Still open: the fix is the cable/port/switch or a PHY/EEE setting on the host, and the watcher still cannot tell "tunnel down" from "bot down". Next action unchanged.

  5. alex-mextner commented on Oct 31, 2026

    @alex-mextner
    OwnerAuthor

    Recurrence on 2026-10-31 (unattended session hcb-alert-20261031T020109Z, watcher alert down-1793412008). Same pattern as 2026-10-29; the kernel log shows it nightly (30 end0: Link is Down lines since 2026-10-17, pairs at 03:00:08 and 03:00:40 local).

    • end0: Lost carrier 02:00:08Z, up 02:00:35Z at 100 Mbps (1 Gbps downshift, "check cabling!"), down again 02:00:40Z, up 02:01:12Z, DHCP 02:01:17Z.
    • cloudflared: QUIC timeouts from 02:00:09Z, network is unreachable until 02:00:21Z, all four connections re-registered 02:01:32-02:01:36Z. /ready 530 (error 1033) from the 02:00:08Z probe; 200 ok again by 02:02Z.
    • Bot untouched: Up 37 h, healthy, 0 restarts, not OOM-killed; deployed revision bb7a9d4 (= main HEAD, verified 2026-10-29 13:12Z), so no deploy involved.
    • Effect: two morning_agenda jobs (319986, 319987) failed with getaddrinfo ETIMEOUT api.telegram.org at 02:00:17 and 02:00:47Z and were delivered by the notification retry at 02:01:54/02:01:59Z. Weather for those agendas failed with getaddrinfo ETIMEOUT api.openweathermap.org (02:00:11, 02:00:13, 02:01:00Z).
    • No change made on the server; the fix stays with the owner (cable/port/switch/EEE at the odroid, or whatever runs at 03:00 local on the network).
  6. alex-mextner commented on Nov 1, 2026

    @alex-mextner
    OwnerAuthor

    Recurrence on 2026-11-01 (unattended session hcb-alert-20261101T020126Z, watcher alert down-1793498425). Same pattern as 2026-10-31.

    • end0: Link Down 02:00:10Z, Up 02:00:36Z (1 Gbps -> 100 Mbps downshift), Down 02:00:40Z, Up 02:01:12Z. journalctl -k --since -15d shows the same pair at 03:00 local on every night 2026-10-28..11-01 (30 Link is Down lines in total). ethtool end0: 100 Mb/s now; mmc_rx_crc_error: 2, mmc_rx_align_error: 2.
    • cloudflared lost all connections (QUIC timeout 02:00:08Z, network is unreachable) and re-registered 4 connections at 02:02:09Z; public /ready 530 (error 1033) from the 02:00:25Z probe, 200 ok from the 02:03Z checks.
    • hypercal-bot untouched: Up 2 days, healthy, 0 restarts; revision bb7a9d4 = origin/main HEAD, verified 2026-10-29 13:12Z, so no deploy involved. 14 level-50 lines, all getaddrinfo ETIMEOUT; morning_agenda logIds 2343/2344 logged "Notification sent" at 02:01:58Z after retry.
    • Nothing changed on the server. Router/switch log at 03:00 local and a cable/port swap are still unchecked and need the owner. The watcher still starts a full session for each nightly tunnel drop.
  7. alex-mextner commented on Nov 2, 2026

    @alex-mextner
    OwnerAuthor

    Recurrence on 2026-11-02 (unattended session hcb-alert-20261102T020129Z, watcher alert down-1793584829). Kernel journal (UTC): end0 Link Down 02:00:08, Up 02:00:35 (100 Mbps downshift), Down 02:00:40, Up 02:01:12. Note: dmesg -T on this host is ~70 s off against the journal; use journalctl -k --utc. cloudflared: network is unreachable on all 4 edge connections from 02:00:13Z, re-registered 02:01:39Z. Watcher probes 502/530 from 02:00:29Z, /ready 200 by 02:02Z (3 of 3). hypercal-bot untouched: Up 3 days, healthy, 0 restarts, bb7a9d4 = origin/main HEAD, receipt verified 2026-10-29T13:12Z; no deploy involved. Bot logged 20 level-50 lines (getaddrinfo ETIMEOUT: Telegram, Google, weather) between 02:00:08Z and 02:01:00Z. Nightly pair seen every night 2026-10-28..11-02. Cause of the 03:00 local drop still not established; nothing changed on the server.

  8. alex-mextner commented on Nov 3, 2026

    @alex-mextner
    OwnerAuthor

    Recurrence on 2026-11-03 (unattended session hcb-alert-20261103T020129Z, watcher alert down-1793671228). Same nightly pattern; no deploy involved.

    • Kernel/networkd (UTC): end0 Link Down 02:00:09, Up 02:00:36 (1 Gbps -> 100 Mbps downshift), Down 02:00:41, Up 02:01:12, DHCP 02:01:17. cloudflared: network is unreachable on all four edge connections from 02:00:14, re-registered 4 connections at 02:02:48-50, about 92 s after the link was back. Public /ready 530 from 02:00:09 (watcher saw 502 at 02:00:29, then 530), 200 ok (unverified) at 02:03:00; watcher logged "readiness is back" at 03:03:00 local.
    • Bot untouched: Up 4 h (started 2026-11-02T22:23:35Z), healthy, 0 restarts, not OOM-killed. Revision ff93220 = origin/main HEAD, deploy run 41532503462 succeeded 21:53Z, so no deploy involved. Local /health and /ready 200 throughout. No log line of level >= 40 in the last 2 h; the morning agenda and pull sync ran at 02:00:01-02Z, before the link dropped, so nothing was lost this time.
    • New: the router is not rebooting at 03:00. router status reports uptime 10 d 10 h (last boot about 2026-10-23), DOCSIS locked 24/24 down and 6/8 up, provisioning all Completed; router settings show parental-time has no time-of-day block configured. Yet end0 drops every night since 2026-10-16 (6 carrier events per night in networkd, 19 nights). So the trigger is not a router reboot or a router time-of-day policy; it stays on the link between end0 and the router's LAN port: the cable, the router port or the odroid's RTL8211F PHY.
    • Link health: ethtool end0 100 Mb/s with 1 Gbps downshift ("check cabling!"), EEE enabled, mmc_rx_crc_error: 2, mmc_rx_align_error: 2.

    Not established: what fires at 03:00 local. Next action for the owner: swap the cable and the router LAN port, and look for a device on that LAN segment with a 03:00 job. Nothing on the odroid was changed.

  9. alex-mextner commented on Nov 3, 2026

    @alex-mextner
    OwnerAuthor

    Candidate mitigation, not tried (2026-11-03, read-only observation; changes on the odroid need the owner).

    • Observation: on 2026-11-03 the link was back at 02:01:12Z (DHCP 02:01:17Z), but cloudflared logged nothing between 02:00:37Z and 02:02:48Z and re-registered its four connections only at 02:02:48-50Z, about 92 s after the address was back. Of the ~160 s public outage, roughly 90 s is this reconnect lag, not the link drop. The same lag is in the earlier comments (about 98 s on 2026-10-28, about 23 s on 2026-10-26).
    • [INFERENCE] The silence looks like cloudflared's retry backoff growing while the link was down; I did not read its retry settings or source.
    • Do: a networkd-dispatcher (or systemd path) hook that runs systemctl restart cloudflared when end0 gains carrier. Because: it would cut the outage to the link downtime plus a few seconds, and a shorter outage may also fall under the watcher's 3-probe threshold. Cost: the restart briefly drops every *.mextner.com tunnel connection (ESB, HSB, mextner.com share it), so test it on a normal day before relying on it. It does not fix the 03:00 trigger, so acceptance criterion 1 stays open.
  10. alex-mextner commented on Nov 4, 2026

    @alex-mextner
    OwnerAuthor

    Recurrence on 2026-11-04 (unattended session hcb-alert-20261104T020127Z, watcher alert down-1793757627). Same nightly pattern; no deploy involved.

    • Kernel (UTC): end0 Link Down 02:00:07, Up 02:00:35 (100 Mbps downshift), Down 02:00:40, Up 02:01:11. cloudflared still logged "network is unreachable" at 02:01:19Z and re-registered its four connections at 02:01:42-43Z (vie05 x2, beg01 x2).
    • Public /ready and /health were 200 again by ~02:02Z; local 127.0.0.1:3001/ready = 200 throughout the check.
    • hypercal-bot unchanged: image ff93220 (main HEAD, fix(ai): Claude credential failover with recovery and deadlines (Refs #907) #928), started 2026-11-02T22:23Z, RestartCount 0, healthy, 0 error-level log lines in the last 2 h.
    • journalctl -k shows one Down/Up pair per night on every night 2026-10-17..2026-11-04 (checked 10-17..10-31 by count plus 11-02..11-04 directly).
    • Nothing changed on the server by this session.
  11. alex-mextner commented on Nov 4, 2026

    @alex-mextner
    OwnerAuthor

    2026-11-04 04:33Z to 04:37Z: public host answered Cloudflare 530 (error 1033) for about 4 minutes while the origin was healthy (localhost:3001 /health ok, container up since 03:04Z, no restarts). cloudflared (pid 2109) could not reach the edge over QUIC (UDP timeouts); at 04:37:55Z all four tunnel connections re-registered on their own (protocol=quic, locations vie05 and beg01), and the public /health returned 200 at 04:38Z. Not caused by a deploy; the hcb-alert session 20261104T043612Z saw it and changed nothing. Different time of day than the nightly 03:00 link drop, so QUIC to the edge needs its own look (fallback protocol http2).

  12. alex-mextner commented on Nov 4, 2026

    @alex-mextner
    OwnerAuthor

    Second, different outage on 2026-11-04 (found by watcher session hcb-alert-20261104T043612Z): 530/1033 on hypercal.mextner.com and mextner.com from ~04:33Z to 04:37:55Z (~5 min). It is NOT the 03:00 local window and NOT a link drop: no end0 Link-is-Down in the kernel log in that window.

    • cloudflared: failed to dial to edge with quic: timeout: no recent network activity on all 4 connections, plus lookup region1.v2.argotunnel.com: i/o timeout.
    • From the odroid during the outage: TCP 7844 to the edge, HTTPS to cloudflare.com and ICMP to 1.1.1.1 all worked.
    • It re-registered by itself at 04:37:55Z.
    • Unrelated to the deploy: d0bc076 up since 03:04Z, 0 restarts, localhost:3001 /ready answered throughout.

    [INFERENCE] UDP/QUIC to the edge is unreliable on this uplink; cloudflared --protocol http2 (TCP 7844) might remove this class. Not applied: server change, owner decision.

  13. alex-mextner commented on Nov 5, 2026

    @alex-mextner
    OwnerAuthor

    Recurrence 2026-11-05 (watcher session hcb-alert-20261105T020111Z): the odroid kernel journal shows end0: Link is Down at 02:00:08Z and 02:00:41Z (03:00 local), Up at 02:00:36Z and 02:01:12Z, the same two-flap pattern as 2026-11-03 and 2026-11-04. cloudflared logged sendmsg: network is unreachable 02:01:18-02:01:27Z and re-registered by itself 02:01:30-02:01:32Z (NRestarts=0); public /ready was 530 from 02:00:11Z and 200 again by ~02:03Z. hypercal-bot bbedb86 (= origin/main) healthy, 0 restarts, no level>=40 records in 2h: not a deploy. Next action unchanged: find what drives the 03:00 link reset (cable/port/switch, EEE/ethtool on end0). No server change made.

  14. alex-mextner commented on Nov 6, 2026

    @alex-mextner
    OwnerAuthor

    2026-11-06 02:01Z to 02:02:20Z (03:00 Europe/Berlin): the nightly outage repeated. Public host answered Cloudflare 530 (error 1033) for about 3 minutes; cloudflared reported 'network is unreachable' on all four connections and the bot container saw getaddrinfo timeouts for api.telegram.org, googleapis and openweathermap. Origin stayed healthy (localhost /health and /ready 200, revision 78b6343 = main). Tunnel recovered at 02:02:20Z by itself; two morning_agenda messages were delivered by the retry at 02:01:49Z; no user update arrived in the window. The cloudflared journal shows the same pattern on 6 of 6 nights since 2026-11-01, so it is the link drop this ticket describes, not a deploy and not related to the 2026-11-01 silent reply.

  15. alex-mextner commented on Nov 6, 2026

    @alex-mextner
    OwnerAuthor

    Corroboration from the down-1793930409 session (2026-11-06, read-only investigation): same double flap — networkd end0: Lost carrier 03:00:09 CET, DHCP lease lost, second loss 03:00:41, final Gained carrier 03:01:12; cloudflared lost all 4 QUIC conns (network is unreachable) but tonight only re-registered at 02:02:20Z, so external downtime was ~2.5 min (02:00:09–02:02:40Z) and the PC watcher crossed its 3-probe threshold for the first time (its journal shows prior nights stayed at ≤2 consecutive failures). Bot itself healthy throughout (container up since 15:16Z Nov 5 on 78b63430 = main; localhost /health and /ready 200). User impact tonight: none lost — no Telegram updates in the window; two morning_agenda digests delayed ~2 min and delivered by BullMQ retry (jobs 328723/328724 completed 02:01:49Z).

  16. alex-mextner commented on Nov 8, 2026

    @alex-mextner
    OwnerAuthor

    2026-11-08 evidence (revision 128846b): the nightly link drop recurred at 03:00:05 local (Link is Down, Up at 03:00:33 at 100 Mbps, Down again 03:00:38, Up 03:01:10; carrier_changes counter 5 since boot). It happened inside a separate 10.5 h upstream-DNS outage of the same host (00:49Z-11:24Z, see #975) that started 2 h before the flap and outlasted it by 8 h, so the flap is not its cause. The link also negotiates 1 Gbps then downshifts to 100 Mbps on every Up ('check cabling').

  17. alex-mextner commented on Nov 8, 2026

    @alex-mextner
    OwnerAuthor

    Alert down-1794143298 (2026-11-08): public 530 from 13:08:18Z, outside the nightly 03:00 window. Tailscale shows odroidn2 offline (last handshake 13:07:49Z); ssh and tailscale ping time out; mextner.com and finbot.mextner.com also return 530, so the shared tunnel or host is down, not the bot. The PC's own internet and router work. No deploy near the time (last merge #974, 2026-11-07 23:18Z). Cause not established: no host access. Evidence only; no server changes.

  18. alex-mextner commented on Nov 10, 2026

    @alex-mextner
    OwnerAuthor

    Recurrence 2026-11-10 (watcher session hcb-alert-20261110T020109Z, alert down-1794276009, read-only). networkd: end0: Lost carrier 03:00:08 and 03:00:41 CET, Gained carrier 03:00:36 and 03:01:12 (100 Mbps downshift each time). cloudflared: network is unreachable on all 4 connections, re-registered 02:01:16-02:01:27Z. Public /ready 530 from ~02:00:08Z, 200 at 02:01Z and 02:03Z (~80 s). Bot untouched: image 128846b, started 2026-11-08 13:21Z, restarts=0. Lost carrier appears only at 03:00 on every retained night (34 lines, 2026-10-25..11-10). During the drop the bot logged 20 level-50 getaddrinfo ETIMEOUT records (Telegram, Google, OpenWeather); both morning_agenda jobs retried and logged "Notification sent" at 02:01:51Z, so no job was lost (delivery itself unconfirmed). No new cause established.

  19. alex-mextner commented on Nov 11, 2026

    @alex-mextner
    OwnerAuthor

    Recurrence 2026-11-11 (watcher session hcb-alert-20261111T020115Z, alert down-1794362414, read-only). Revision 128846b, container up since 2026-11-08, 0 restarts, no OOM. Kernel: end0 Link is Down 03:00:09 and 03:00:41 CET, Up 03:00:36 and 03:01:12 (100 Mbps downshift). cloudflared QUIC 'no recent network activity' at 02:00:09Z, re-registered 02:01:22Z. Bot saw 14 DNS/connect errors 02:00:11Z-02:01:43Z. /ready 200 at 02:01:23Z. Same pattern nightly 2026-11-09..11 in the journal. No deploy involved. Still unfixed; needs cable/switch/router-side check.

  20. alex-mextner commented on Nov 12, 2026

    @alex-mextner
    OwnerAuthor

    Recurrence on 2026-11-12 (unattended session hcb-alert-20261112T020117Z, watcher alert down-1794448816). Read-only evidence:

    • end0 kernel: Link Down 02:00:08Z and 02:00:41Z, Up 02:00:36Z and 02:01:12Z (100 Mbps). journalctl -k --since -7d shows exactly two Link is Down lines on each of 2026-11-09, 10, 11 and 12 at 03:00:08-09 and 03:00:40-41 local.
    • cloudflared: 17 network is unreachable errors from 02:00:10Z, DNS server misbehaving at 02:01:08Z, all four connections re-registered 02:01:22-25Z. The first public /ready failure was 02:00:16Z; /ready is 200 again after 02:01:25Z.
    • hypercal-bot untouched: started 2026-11-11T21:08:52Z, restarts=0, not OOM-killed, healthy; image 128846b = main HEAD (fix(ai): the write receipt reads create_event's 'already exists' result as the same event, not a second creation or a failure (Refs #929, #520) #974, 2026-11-08), no deploy. Its last 2 h of logs hold 20 lines, all level 30 (the morning_agenda batch ran at 02:00:03Z, five seconds before the first drop).
    • Nothing changed on the server. Router/switch log and cable/port swap remain unchecked; that decision is still the owner's.
  21. alex-mextner commented on Nov 14, 2026

    @alex-mextner
    OwnerAuthor

    Recurrence 2026-11-14 (unattended session hcb-alert-20261114T020109Z, watcher alert down-1794621609, read-only). Revision 90ee425 (= origin/main HEAD, #982, merged 2026-11-13 10:08 CET), container up 16 h, healthy, no restarts; no deploy in the window.

    • Kernel (UTC): end0 Link Down 02:00:08, Up 02:00:36 (1 Gbps -> 100 Mbps downshift, "check cabling!"), Down 02:00:41, Up 02:01:12. Current speed 100 Mb/s.
    • cloudflared: "no recent network activity" 02:00:08, then "sendmsg: network is unreachable" on all 4 connections; re-registered 02:01:35 (beg01) and 02:01:45 (vie05/vie06/beg01). NRestarts=0.
    • Public /ready: 530 at 02:01:19Z, 200 at ~02:01:50Z; localhost:3001/ready ok throughout.
    • Bot docker logs since 01:55Z: 6 lines, no level 40/50/60. Webhook delivery during the ~100 s window not checked (Telegram retries; unconfirmed).
    • Separate finding during this check: hcb-odroid-watchdog.service is refused by the global service-failover start limit since 2026-11-10, tracked in The odroid's own readiness watchdog runs only ~10 times an hour: the global service-failover start limit refuses its timer starts (start-limit-hit since 2026-11-10) #983.
  22. alex-mextner commented on Nov 16, 2026

    @alex-mextner
    OwnerAuthor

    Recurrence 2026-11-16 (unattended session hcb-alert-20261116T020138Z, watcher alert down-1794794429, read-only).

    • Kernel (UTC): end0 Link Down 02:00:10, Up 02:00:36 (1 Gbps -> 100 Mbps downshift, "check cabling!"), Down 02:00:41, Up 02:01:11.
    • cloudflared (NRestarts 0, active since 2026-11-14): QUIC "no recent network activity" from 02:00:09Z, then "sendmsg: network is unreachable" on all 4 connections until 02:00:57Z; re-registered 4 connections at 02:01:44Z (beg01 x2, vie06 x2).
    • Public /ready: watcher failures 02:00:29Z..02:01:38Z (530); 200 ok at 02:01:45Z and again at the end of the check.
    • hypercal-bot untouched: image 90ee425 (= origin/main HEAD, fix(ai): prioritize Claude 5.5 and keep HF last (UB-178) #982), started 2026-11-15T21:06Z, restarts 0, not OOM-killed, healthy. No deploy involved.
    • Bot impact: 4 level-50 records, all getaddrinfo ETIMEOUT api.telegram.org; 2 morning_agenda jobs failed at 02:00:42Z, were retried and logged "Notification sent" at 02:01:13Z.
    • Separate, already tracked in The odroid's own readiness watchdog runs only ~10 times an hour: the global service-failover start limit refuses its timer starts (start-limit-hit since 2026-11-10) #983: hcb-odroid-watchdog.service is in start-limit-hit (19-20 refusals per hour, 2505 since 2026-11-10), so the odroid-side watchdog was not running during this window either.

    No server change made. Next action unchanged: the cable/switch/router side of the 03:00 local link reset needs the owner.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions