Repository navigation
Description
What
Every night at 03:00:07-10 Europe/Berlin the kernel on root@odroidn2 logs 'meson8b-dwmac end0: Link is Down', systemd-networkd 'Lost carrier / DHCP lease lost', then 'Link is Up - 100Mbps' after a 1 Gbps->100 Mbps PHY downshift ('check cabling!'), a second Down at 03:00:40-45 and the final Up at 03:01:11. cloudflared logs 'sendmsg: network is unreachable' for all four edge connections and re-registers at 03:01:34 (02:01:34Z on 2026-10-26). While the tunnel is down hypercal.mextner.com and mextner.com answer 530 (Cloudflare error 1033); both were 200 again by 02:01:59Z. journalctl -k shows 20 Link-is-Down lines = 10 nights, one per night since the host booted on 2026-10-16, none at any other time. The hypercal-bot container is untouched: started 2026-10-25 23:46:06Z, healthy, RestartCount=0, no log line of level >=40 in the last 2 h, loopback /ready 200 when probed at 02:01:47Z (after the link was back; it was not probed during the drop). Nothing on the host explains the drop: the 03:00 cron jobs are the two local SQLite backups (HCB root, ESB www-data; no network use, finished within ~1 s of the start) plus */2 and */5 jobs that also run at other minutes; the drop is 6-10 s after the start with jitter. Open question: a scheduled event on the router/switch side, or the cable/port (the downshift to 100 Mbps after every re-link suggests a bad pair). The link is still at 100 Mbps now. The alert watcher opened its first incident for it at 2026-10-26 02:01:12Z (alert down-1792980012); whether anything probed it on earlier nights is not established.
Why (motivation)
A nightly outage window of ~85 s recurs on a schedule, so it is a defect to remove or to account for, not a transient. Each night it also meets the alert watcher's threshold (3 failed probes x 30 s) and opens one unattended omp investigation session for a bot that is healthy.
User impact
Telegram users and Google webhooks see 530 for ~85 s at 03:00 Europe/Berlin every night (Telegram retries webhooks; no lost message or write was observed in the bot logs for 2026-10-26). The owner gets a false DOWN page and a burned agent session per night.
Cost of inaction
A real outage at 03:00 is indistinguishable from the nightly one; the link stays degraded to 100 Mbps; a deploy or the 03:00 backup that overlaps the window can fail on a lost link; the owner learns to ignore DOWN pages, the same erosion as #843 .
Acceptance criteria
- The trigger of the nightly 03:00 link drop is identified with evidence (router/switch log or schedule, or a cable/port swap test) and recorded on this issue
- journalctl -k on root@odroidn2 shows no 'end0: Link is Down' at 03:00 for three consecutive nights and the link negotiates 1000 Mb/s, or the owner records the recurrence as accepted
- The 03:00 window no longer opens a DOWN incident and an unattended session (cause removed, or HCB_ALERT_DOWN_PROBES / cross-check against a host-level signal decided by the owner), and a real outage still does
Screenshots
(none)
Deploy
Deployed: no
Links
- Relates to Move the public ingress from hypercal.invntrm.ru (DO droplet) to hypercal.mextner.com (odroid Cloudflare tunnel) #828: Move the public ingress from hypercal.invntrm.ru (DO droplet) to hypercal.mextner.com (odroid Cloudflare tunnel) #828
Skipped gates
- screenshots: ops incident, no UI
Activity
alex-mextner commentedon Oct 26, 2026
Verified 2026-10-26 02:07Z with journalctl -k on root@odroidn2 (journal starts at boot, 2026-10-16 21:43 CEST): 'end0: Link is Down' occurs exactly twice on each of 10 nights, 2026-10-17 through 2026-10-26, at 03:00:07-10 and 03:00:40-45 local time, and at no other time of day. The wall-clock time did not move when Europe/Berlin left DST on 2026-10-25 (03:00 +02:00 = 01:00Z on 10-24, 03:00 +01:00 = 02:00Z on 10-25 and 10-26), so the trigger follows local time, not UTC. That fits a local-time schedule on the router/switch or on the host, and it rules out a UTC-timed external job. The host's only 03:00-specific jobs are the two SQLite backups, which use no network. The drop starts 6-10 s after cron fires, which a router-side schedule explains at least as well. Not yet checked: the router/switch logs (outside the read-only scope of the alert session), and whether the other services on this host (ExpenseSyncBot, HyperSummaryBot, Home Assistant) were also unreachable in the window; every public name behind the odroid-home tunnel answered 530 (mextner.com confirmed).
alex-mextner commentedon Oct 27, 2026
Recurrence 2026-10-27 (alert down-1793066420, session hcb-alert-20261027T020120Z), read-only evidence:
alex-mextner commentedon Oct 28, 2026
Recurrence 2026-10-28 (alert down-1793152822, session hcb-alert-20261028T020122Z), read-only evidence:
alex-mextner commentedon Oct 29, 2026
Recurrence on 2026-10-29 (unattended session hcb-alert-20261029T020108Z, watcher alert down-1793239208).
end0kernel: Link Down 02:00:08Z, Up 02:00:35Z (1 Gbps -> 100 Mbps downshift, "check cabling!"), Down 02:00:40Z, Up 02:01:11Z; DHCP re-acquired 02:01:16Z.network is unreachableerrors 02:00:30-02:00:55Z, all four connections re-registered 02:01:23Z./readyreturned 530 (error 1033) from the 02:00:08Z probe and 200 again by 02:01:23Z.getaddrinfo ETIMEOUT(api.telegram.org, www.googleapis.com, api.openweathermap.org). Two morning_agenda jobs (317081, 317082) failed, were retried by the bot and logged "Notification sent" at 02:02:03Z.Still open: the fix is the cable/port/switch or a PHY/EEE setting on the host, and the watcher still cannot tell "tunnel down" from "bot down". Next action unchanged.
alex-mextner commentedon Oct 31, 2026
Recurrence on 2026-10-31 (unattended session hcb-alert-20261031T020109Z, watcher alert down-1793412008). Same pattern as 2026-10-29; the kernel log shows it nightly (30
end0: Link is Downlines since 2026-10-17, pairs at 03:00:08 and 03:00:40 local).end0: Lost carrier 02:00:08Z, up 02:00:35Z at 100 Mbps (1 Gbps downshift, "check cabling!"), down again 02:00:40Z, up 02:01:12Z, DHCP 02:01:17Z.network is unreachableuntil 02:00:21Z, all four connections re-registered 02:01:32-02:01:36Z./ready530 (error 1033) from the 02:00:08Z probe; 200okagain by 02:02Z.morning_agendajobs (319986, 319987) failed withgetaddrinfo ETIMEOUT api.telegram.orgat 02:00:17 and 02:00:47Z and were delivered by the notification retry at 02:01:54/02:01:59Z. Weather for those agendas failed withgetaddrinfo ETIMEOUT api.openweathermap.org(02:00:11, 02:00:13, 02:01:00Z).alex-mextner commentedon Nov 1, 2026
Recurrence on 2026-11-01 (unattended session hcb-alert-20261101T020126Z, watcher alert down-1793498425). Same pattern as 2026-10-31.
end0: Link Down 02:00:10Z, Up 02:00:36Z (1 Gbps -> 100 Mbps downshift), Down 02:00:40Z, Up 02:01:12Z.journalctl -k --since -15dshows the same pair at 03:00 local on every night 2026-10-28..11-01 (30Link is Downlines in total).ethtool end0: 100 Mb/s now;mmc_rx_crc_error: 2,mmc_rx_align_error: 2.network is unreachable) and re-registered 4 connections at 02:02:09Z; public/ready530 (error 1033) from the 02:00:25Z probe, 200okfrom the 02:03Z checks.getaddrinfo ETIMEOUT; morning_agenda logIds 2343/2344 logged "Notification sent" at 02:01:58Z after retry.alex-mextner commentedon Nov 2, 2026
Recurrence on 2026-11-02 (unattended session hcb-alert-20261102T020129Z, watcher alert down-1793584829). Kernel journal (UTC): end0 Link Down 02:00:08, Up 02:00:35 (100 Mbps downshift), Down 02:00:40, Up 02:01:12. Note:
dmesg -Ton this host is ~70 s off against the journal; usejournalctl -k --utc. cloudflared: network is unreachable on all 4 edge connections from 02:00:13Z, re-registered 02:01:39Z. Watcher probes 502/530 from 02:00:29Z, /ready 200 by 02:02Z (3 of 3). hypercal-bot untouched: Up 3 days, healthy, 0 restarts, bb7a9d4 = origin/main HEAD, receipt verified 2026-10-29T13:12Z; no deploy involved. Bot logged 20 level-50 lines (getaddrinfo ETIMEOUT: Telegram, Google, weather) between 02:00:08Z and 02:01:00Z. Nightly pair seen every night 2026-10-28..11-02. Cause of the 03:00 local drop still not established; nothing changed on the server.alex-mextner commentedon Nov 3, 2026
Recurrence on 2026-11-03 (unattended session hcb-alert-20261103T020129Z, watcher alert down-1793671228). Same nightly pattern; no deploy involved.
end0Link Down 02:00:09, Up 02:00:36 (1 Gbps -> 100 Mbps downshift), Down 02:00:41, Up 02:01:12, DHCP 02:01:17. cloudflared:network is unreachableon all four edge connections from 02:00:14, re-registered 4 connections at 02:02:48-50, about 92 s after the link was back. Public/ready530 from 02:00:09 (watcher saw 502 at 02:00:29, then 530), 200ok (unverified)at 02:03:00; watcher logged "readiness is back" at 03:03:00 local./healthand/ready200 throughout. No log line of level >= 40 in the last 2 h; the morning agenda and pull sync ran at 02:00:01-02Z, before the link dropped, so nothing was lost this time.router statusreports uptime 10 d 10 h (last boot about 2026-10-23), DOCSIS locked 24/24 down and 6/8 up, provisioning all Completed;router settings show parental-timehas no time-of-day block configured. Yetend0drops every night since 2026-10-16 (6 carrier events per night in networkd, 19 nights). So the trigger is not a router reboot or a router time-of-day policy; it stays on the link betweenend0and the router's LAN port: the cable, the router port or the odroid's RTL8211F PHY.ethtool end0100 Mb/s with 1 Gbps downshift ("check cabling!"), EEE enabled,mmc_rx_crc_error: 2,mmc_rx_align_error: 2.Not established: what fires at 03:00 local. Next action for the owner: swap the cable and the router LAN port, and look for a device on that LAN segment with a 03:00 job. Nothing on the odroid was changed.
alex-mextner commentedon Nov 3, 2026
Candidate mitigation, not tried (2026-11-03, read-only observation; changes on the odroid need the owner).
systemctl restart cloudflaredwhenend0gains carrier. Because: it would cut the outage to the link downtime plus a few seconds, and a shorter outage may also fall under the watcher's 3-probe threshold. Cost: the restart briefly drops every*.mextner.comtunnel connection (ESB, HSB, mextner.com share it), so test it on a normal day before relying on it. It does not fix the 03:00 trigger, so acceptance criterion 1 stays open.alex-mextner commentedon Nov 4, 2026
Recurrence on 2026-11-04 (unattended session hcb-alert-20261104T020127Z, watcher alert down-1793757627). Same nightly pattern; no deploy involved.
end0Link Down 02:00:07, Up 02:00:35 (100 Mbps downshift), Down 02:00:40, Up 02:01:11. cloudflared still logged "network is unreachable" at 02:01:19Z and re-registered its four connections at 02:01:42-43Z (vie05 x2, beg01 x2)./readyand/healthwere 200 again by ~02:02Z; local127.0.0.1:3001/ready= 200 throughout the check.journalctl -kshows one Down/Up pair per night on every night 2026-10-17..2026-11-04 (checked 10-17..10-31 by count plus 11-02..11-04 directly).alex-mextner commentedon Nov 4, 2026
2026-11-04 04:33Z to 04:37Z: public host answered Cloudflare 530 (error 1033) for about 4 minutes while the origin was healthy (localhost:3001 /health ok, container up since 03:04Z, no restarts). cloudflared (pid 2109) could not reach the edge over QUIC (UDP timeouts); at 04:37:55Z all four tunnel connections re-registered on their own (protocol=quic, locations vie05 and beg01), and the public /health returned 200 at 04:38Z. Not caused by a deploy; the hcb-alert session 20261104T043612Z saw it and changed nothing. Different time of day than the nightly 03:00 link drop, so QUIC to the edge needs its own look (fallback protocol http2).
alex-mextner commentedon Nov 4, 2026
Second, different outage on 2026-11-04 (found by watcher session hcb-alert-20261104T043612Z): 530/1033 on hypercal.mextner.com and mextner.com from ~04:33Z to 04:37:55Z (~5 min). It is NOT the 03:00 local window and NOT a link drop: no end0 Link-is-Down in the kernel log in that window.
failed to dial to edge with quic: timeout: no recent network activityon all 4 connections, pluslookup region1.v2.argotunnel.com: i/o timeout.[INFERENCE] UDP/QUIC to the edge is unreliable on this uplink;
cloudflared --protocol http2(TCP 7844) might remove this class. Not applied: server change, owner decision.alex-mextner commentedon Nov 5, 2026
Recurrence 2026-11-05 (watcher session hcb-alert-20261105T020111Z): the odroid kernel journal shows
end0: Link is Downat 02:00:08Z and 02:00:41Z (03:00 local), Up at 02:00:36Z and 02:01:12Z, the same two-flap pattern as 2026-11-03 and 2026-11-04. cloudflared loggedsendmsg: network is unreachable02:01:18-02:01:27Z and re-registered by itself 02:01:30-02:01:32Z (NRestarts=0); public /ready was 530 from 02:00:11Z and 200 again by ~02:03Z. hypercal-bot bbedb86 (= origin/main) healthy, 0 restarts, no level>=40 records in 2h: not a deploy. Next action unchanged: find what drives the 03:00 link reset (cable/port/switch, EEE/ethtool on end0). No server change made.alex-mextner commentedon Nov 6, 2026
2026-11-06 02:01Z to 02:02:20Z (03:00 Europe/Berlin): the nightly outage repeated. Public host answered Cloudflare 530 (error 1033) for about 3 minutes; cloudflared reported 'network is unreachable' on all four connections and the bot container saw getaddrinfo timeouts for api.telegram.org, googleapis and openweathermap. Origin stayed healthy (localhost /health and /ready 200, revision 78b6343 = main). Tunnel recovered at 02:02:20Z by itself; two morning_agenda messages were delivered by the retry at 02:01:49Z; no user update arrived in the window. The cloudflared journal shows the same pattern on 6 of 6 nights since 2026-11-01, so it is the link drop this ticket describes, not a deploy and not related to the 2026-11-01 silent reply.
alex-mextner commentedon Nov 6, 2026
Corroboration from the
down-1793930409session (2026-11-06, read-only investigation): same double flap — networkdend0: Lost carrier03:00:09 CET, DHCP lease lost, second loss 03:00:41, finalGained carrier03:01:12; cloudflared lost all 4 QUIC conns (network is unreachable) but tonight only re-registered at 02:02:20Z, so external downtime was ~2.5 min (02:00:09–02:02:40Z) and the PC watcher crossed its 3-probe threshold for the first time (its journal shows prior nights stayed at ≤2 consecutive failures). Bot itself healthy throughout (container up since 15:16Z Nov 5 on78b63430= main; localhost /health and /ready 200). User impact tonight: none lost — no Telegram updates in the window; twomorning_agendadigests delayed ~2 min and delivered by BullMQ retry (jobs 328723/328724 completed 02:01:49Z).alex-mextner commentedon Nov 8, 2026
2026-11-08 evidence (revision 128846b): the nightly link drop recurred at 03:00:05 local (Link is Down, Up at 03:00:33 at 100 Mbps, Down again 03:00:38, Up 03:01:10; carrier_changes counter 5 since boot). It happened inside a separate 10.5 h upstream-DNS outage of the same host (00:49Z-11:24Z, see #975) that started 2 h before the flap and outlasted it by 8 h, so the flap is not its cause. The link also negotiates 1 Gbps then downshifts to 100 Mbps on every Up ('check cabling').
alex-mextner commentedon Nov 8, 2026
Alert down-1794143298 (2026-11-08): public 530 from 13:08:18Z, outside the nightly 03:00 window. Tailscale shows odroidn2 offline (last handshake 13:07:49Z); ssh and tailscale ping time out; mextner.com and finbot.mextner.com also return 530, so the shared tunnel or host is down, not the bot. The PC's own internet and router work. No deploy near the time (last merge #974, 2026-11-07 23:18Z). Cause not established: no host access. Evidence only; no server changes.
alex-mextner commentedon Nov 10, 2026
Recurrence 2026-11-10 (watcher session hcb-alert-20261110T020109Z, alert down-1794276009, read-only). networkd:
end0: Lost carrier03:00:08 and 03:00:41 CET, Gained carrier 03:00:36 and 03:01:12 (100 Mbps downshift each time). cloudflared:network is unreachableon all 4 connections, re-registered 02:01:16-02:01:27Z. Public /ready 530 from ~02:00:08Z, 200 at 02:01Z and 02:03Z (~80 s). Bot untouched: image 128846b, started 2026-11-08 13:21Z, restarts=0.Lost carrierappears only at 03:00 on every retained night (34 lines, 2026-10-25..11-10). During the drop the bot logged 20 level-50 getaddrinfo ETIMEOUT records (Telegram, Google, OpenWeather); both morning_agenda jobs retried and logged "Notification sent" at 02:01:51Z, so no job was lost (delivery itself unconfirmed). No new cause established.alex-mextner commentedon Nov 11, 2026
Recurrence 2026-11-11 (watcher session hcb-alert-20261111T020115Z, alert down-1794362414, read-only). Revision 128846b, container up since 2026-11-08, 0 restarts, no OOM. Kernel: end0 Link is Down 03:00:09 and 03:00:41 CET, Up 03:00:36 and 03:01:12 (100 Mbps downshift). cloudflared QUIC 'no recent network activity' at 02:00:09Z, re-registered 02:01:22Z. Bot saw 14 DNS/connect errors 02:00:11Z-02:01:43Z. /ready 200 at 02:01:23Z. Same pattern nightly 2026-11-09..11 in the journal. No deploy involved. Still unfixed; needs cable/switch/router-side check.
alex-mextner commentedon Nov 12, 2026
Recurrence on 2026-11-12 (unattended session hcb-alert-20261112T020117Z, watcher alert down-1794448816). Read-only evidence:
end0kernel: Link Down 02:00:08Z and 02:00:41Z, Up 02:00:36Z and 02:01:12Z (100 Mbps).journalctl -k --since -7dshows exactly twoLink is Downlines on each of 2026-11-09, 10, 11 and 12 at 03:00:08-09 and 03:00:40-41 local.network is unreachableerrors from 02:00:10Z, DNSserver misbehavingat 02:01:08Z, all four connections re-registered 02:01:22-25Z. The first public /ready failure was 02:00:16Z; /ready is 200 again after 02:01:25Z.alex-mextner commentedon Nov 14, 2026
Recurrence 2026-11-14 (unattended session hcb-alert-20261114T020109Z, watcher alert down-1794621609, read-only). Revision 90ee425 (= origin/main HEAD, #982, merged 2026-11-13 10:08 CET), container up 16 h, healthy, no restarts; no deploy in the window.
end0Link Down 02:00:08, Up 02:00:36 (1 Gbps -> 100 Mbps downshift, "check cabling!"), Down 02:00:41, Up 02:01:12. Current speed 100 Mb/s.alex-mextner commentedon Nov 16, 2026
Recurrence 2026-11-16 (unattended session hcb-alert-20261116T020138Z, watcher alert down-1794794429, read-only).
end0Link Down 02:00:10, Up 02:00:36 (1 Gbps -> 100 Mbps downshift, "check cabling!"), Down 02:00:41, Up 02:01:11.okat 02:01:45Z and again at the end of the check.getaddrinfo ETIMEOUT api.telegram.org; 2 morning_agenda jobs failed at 02:00:42Z, were retried and logged "Notification sent" at 02:01:13Z.hcb-odroid-watchdog.serviceis in start-limit-hit (19-20 refusals per hour, 2505 since 2026-11-10), so the odroid-side watchdog was not running during this window either.No server change made. Next action unchanged: the cable/switch/router side of the 03:00 local link reset needs the owner.