diff --git a/docs/infrastructure/monitoring-and-alerts.md b/docs/infrastructure/monitoring-and-alerts.md index 6424884d..9b48cd5d 100644 --- a/docs/infrastructure/monitoring-and-alerts.md +++ b/docs/infrastructure/monitoring-and-alerts.md @@ -4,6 +4,11 @@ Push-based, near-zero-RAM. No Grafana/Prometheus — a solo operator needs **alerts** (something happened, act now), not dashboards nobody watches at 04:00. All alerts flow through the watchdog's Telegram channel (`/opt/monitoring/telegram.env`). +**The SSOT for everything that reaches the operator's Telegram — alerts, +digests, feedback pings, briefings — is `docs/telegram-notifications.md`**, +enforced by `scripts/ci/check-telegram-registry.sh` (part of `npm run verify`). +This file covers only the box-monitoring senders in that registry. + ## What alerts you | Signal | Mechanism | Installed by | diff --git a/docs/telegram-notifications.md b/docs/telegram-notifications.md new file mode 100644 index 00000000..ec794fb9 --- /dev/null +++ b/docs/telegram-notifications.md @@ -0,0 +1,76 @@ +# Telegram notifications — the registry + +**This file is the SSOT for everything that reaches the operator's Telegram.** +If something pings the phone and is not listed here, that is a bug — and CI +enforces it: `scripts/ci/check-telegram-registry.sh` fails when a file in this +repo sends to Telegram (directly, via `telegram-send.ts`, or via the box's +`lib-alert.sh`) without being registered below. Adding a notification = +adding it here in the same PR. Removing one = removing its row. + +The channel's contract (PR #443/#444): **a message is worth sending only if +the operator must act on it, wants the outcome, or asked for the digest.** +One incident is one message; reminders back off (30m → 2h → 8h → daily); +recoveries name subjects, never keys; N events on one tick digest into one +message; test runs (`ALERT_DRY_RUN=1`) never deliver. + +## Requested updates (the operator asked for these — keep them) + +| What arrives | Trigger | Source | +|---|---|---| +| 📨 New visitor feedback (project, excerpt, link to inbox) | A genuine visitor submits the feedback form (`POST /api/feedback`; AI/synthesizer filings stay silent) | `src/lib/feedback/notify-new.ts` | +| Run outcome: what an agent run concluded (root cause → action → remains) | An orchestration run with `notifyOnClose` closes — includes every incident-dispatch remediation run | `src/lib/orchestration/notify-close.ts`, `src/lib/orchestration/gate-and-close.ts` | +| Morning brief, evening wrap, weekly reflection, Monday digest, email deadlines, financial scan, life scorecard | Loki's scheduled jobs — schedules live in their own SSOT: `/home/openclaw/.openclaw/cron/jobs.json` on bitbaum (edit via Loki, not here) | off-repo: OpenClaw cron | + +## Incidents (something is broken; a fix agent is already dispatched) + +| What arrives | Trigger | Source | +|---|---|---| +| 🔴 DOWN: \ — \ → 🤖 fix agent dispatched | A systemd unit fails and stays failed past the grace window | `scripts/hetzner/install-host-alerts.sh` (notify-failure.sh) | +| ⚙️ STILL FAILING: a; b (digest) / ✅ RECOVERED: a; b (digest) | The 5-min sweep sees a unit failed on two consecutive ticks / recover | `scripts/hetzner/install-host-alerts.sh` (host-check.sh) | +| 🔧 FIXED (no action needed): \ | host-check auto-repaired an unreadable app `.env` and restarted the app | `scripts/hetzner/install-host-alerts.sh` (host-check.sh) | +| 💾 DISK / 🧠 MEM / 🐘 POSTGRES transitions | Resource crosses its hysteresis band on the box | `scripts/hetzner/install-host-alerts.sh` (host-check.sh) | +| 🔴 DOWN: \ (\) → HTTP \ / ✅ RECOVERED (on-box) | An app URL in targets.conf stops answering | `scripts/hetzner/install-watchdog.sh` (watch.sh; also the external dead-man's-switch ping) | +| 🔴 DOWN: \ (\) / ✅ RECOVERED: \ (off-box) | GitHub-side health sweep of every registered app — still reports when bitbaum itself is dead | `.github/workflows/fleet-uptime.yml` (probe: `scripts/hetzner/uptime-sweep.sh`) | +| 🚨 FleetCrown deploy: \ | A fleetcrown deploy fails or rolls back | `scripts/deploy-hetzner.sh` | +| 🧩 \: \ | Deployed reality diverges from the register (wrong port, dead tunnel, inactive unit) | `scripts/hetzner/install-runtime-conformance.sh` | +| 🚫/🧟 agent-work findings | Hourly sweep finds stranded/zombie agent work on the box | `scripts/hetzner/agent-work-check.sh` (installed by `scripts/hetzner/install-agent-work-watch.sh`) | +| 🧹/🚨 DISK GC non-routine outcome | GC ran but the disk is still above the warn mark, or nothing was reclaimable (routine success is journal-only) | `scripts/hetzner/install-disk-gc.sh` | +| Fleet refs audit findings | Deployed refs diverge from expected across the fleet | `scripts/hetzner/install-fleet-refs-audit.sh` | +| ✗ register check findings | Daily 09:15 (laptop): committed apps.conf register vs reality — at most one message per finding per day | `scripts/local/fleet-register-check` | + +## FleetCrown self-checks (the platform watching itself) + +| What arrives | Trigger | Source | +|---|---|---| +| Telemetry stale | Runner telemetry stops arriving | `src/app/api/crons/check-telemetry/route.ts` | +| Runner stalled / wrong version | Box-runner stops claiming or lags the shipped version | `src/app/api/crons/check-runner-stall/route.ts`, `src/app/api/crons/check-runner-version/route.ts` | +| Approvals waiting | Proposed actions sit undecided | `src/app/api/crons/check-pending-approvals/route.ts` | +| Model id rot | A configured model id stops resolving | `src/app/api/crons/check-model-ids/route.ts` | +| Run escalation | A run needs the operator's decision to proceed | `src/db/queries/run-escalations.ts` | +| 📊 Ledger-ready (one-shot) | Run ledger reaches the improver threshold; self-disables after firing | `scripts/hetzner/ledger-ready-gate.sh` | +| 🏷 Rescale window (hcloud) | A bigger Hetzner tier becomes available for migration in fsn1 | `scripts/hetzner/hcloud-availability.sh` | + +## Off-repo senders (registered here, owned elsewhere) + +| Sender | What | Its own SSOT | +|---|---|---| +| Loki (OpenClaw) | Conversation, briefings, reminders — everything conversational | `/home/openclaw/.openclaw/cron/jobs.json` + OpenClaw config on bitbaum | +| orangecat repo | orangecat.ch uptime alerts | `orangecat/.github/workflows/uptime.yml` | + +## Rules for adding a notification + +1. Ask the channel question first: must the operator act, is it an outcome + they are waiting for, or did they ask for the digest? If none — journal, + not Telegram. +2. Route through the shared machinery — the only two blessed send paths: + `lib-alert.sh` on the box (installed by + `scripts/hetzner/install-host-alerts.sh`; use `alert_once` / + `alert_transition` with a human subject) and + `src/lib/actions/telegram-send.ts` in the app (self-only allowlist, + fail-closed). A private `curl` to `api.telegram.org` repeats the + 2026-08-28 register-check double-send and is a reviewable smell. +3. Add the row here, in the same PR. CI (`check-telegram-registry.sh`) makes + the omission fail loudly. +4. Test sends never deliver: `ALERT_DRY_RUN=1` (box), and never bare-run a + sender "to see if it works" — 2026-08-28, a live test rang the phone with + a message about nothing. diff --git a/package.json b/package.json index f1b22cc1..77cdb54f 100644 --- a/package.json +++ b/package.json @@ -22,7 +22,8 @@ "lint": "eslint --cache --cache-location node_modules/.cache/eslint src/ home/", "check:design": "bash scripts/check-design-system.sh", "check:desktop": "bash scripts/check-desktop.sh", - "verify": "npm run check:deploy-ready && tsc --noEmit && npm run lint && npm run check:design && npm run check:desktop && npm run test:unit && npm run test:home && npm run test:deploy-ref-gate && npm run test:not-behind-gate && npm run test:deploy-ready-gate && npm run test:ops", + "verify": "npm run check:deploy-ready && npm run check:telegram-registry && tsc --noEmit && npm run lint && npm run check:design && npm run check:desktop && npm run test:unit && npm run test:home && npm run test:deploy-ref-gate && npm run test:not-behind-gate && npm run test:deploy-ready-gate && npm run test:ops", + "check:telegram-registry": "bash scripts/ci/check-telegram-registry.sh", "smoke": "bash scripts/smoke.sh", "test:home": "bash scripts/test-home.sh", "test:ops": "bash scripts/hetzner/test-host-alerts.sh && bash scripts/hetzner/test-disk-gc.sh && bash scripts/hetzner/test-agent-work-watch.sh && bash scripts/hetzner/test-cron-runner.sh && bash scripts/hetzner/test-ci-gate.sh && bash scripts/hetzner/test-apply-schema.sh && bash scripts/hetzner/test-uptime-sweep.sh", diff --git a/scripts/ci/check-telegram-registry.sh b/scripts/ci/check-telegram-registry.sh new file mode 100755 index 00000000..a46e91b9 --- /dev/null +++ b/scripts/ci/check-telegram-registry.sh @@ -0,0 +1,46 @@ +#!/usr/bin/env bash +# Every Telegram sender must be registered in docs/telegram-notifications.md. +# +# The registry is only an SSOT while nothing can ping the phone without being +# in it. This gate finds every sender in the repo — a direct api.telegram.org +# call, an importer of the app's telegram-send.ts, or a box script that +# sources lib-alert.sh — and fails if its path is not listed in the registry. +# The fix is always the same: add the row (or route the send through shared +# machinery that is already registered). +# +# Registered-but-deleted rows are NOT flagged: a row describing an off-repo +# sender (Loki's cron, orangecat's workflow) has no file here by design. +set -euo pipefail +cd "$(dirname "${BASH_SOURCE[0]}")/../.." + +REGISTRY=docs/telegram-notifications.md +[ -f "$REGISTRY" ] || { echo "✗ $REGISTRY missing — the Telegram registry is gone"; exit 1; } + +# Sender surfaces, tracked files only. Tests and installers' test harnesses are +# excluded: they exercise senders, they are not senders. +senders=$( + { + git grep -l 'api\.telegram\.org' -- ':!docs/' ':!*test*' ':!*.md' 2>/dev/null || true + git grep -l 'actions/telegram-send' -- 'src/' ':!src/lib/actions/telegram-send.ts' ':!*test*' 2>/dev/null || true + git grep -l 'lib-alert\.sh' -- 'scripts/hetzner/' ':!*test*' 2>/dev/null || true + } | sort -u +) + +missing=() +for f in $senders; do + # uptime-sweep.sh only probes; its ALERTING consumer is fleet-uptime.yml. + # It still must appear in the registry (it does, as the probe reference) — + # no special cases: one grep, one rule. + grep -qF "$f" "$REGISTRY" || missing+=("$f") +done + +if [ "${#missing[@]}" -gt 0 ]; then + echo "✗ Telegram sender(s) not registered in $REGISTRY:" + printf ' %s\n' "${missing[@]}" + echo " Every path that can reach the operator's phone must have a row there" + echo " (what arrives, when, from where). Add the row in this PR." + exit 1 +fi + +n=$(printf '%s\n' "$senders" | grep -c . || true) +echo "✓ telegram registry: all ${n} sender file(s) registered in $REGISTRY"