From 6c1bd1df3a4d22d8cfc7f7c36530fcec82f34492 Mon Sep 17 00:00:00 2001 From: Teg79 Date: Thu, 19 Nov 2026 09:37:59 +0100 Subject: [PATCH] docs: get Urbex's alerts on Telegram, Discord or a webhook Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_0182yxsfwRowwnxCfgrQwjnA --- CHANGELOG.md | 2 + docs/cookbook.md | 84 +++++++++++++++++++++++++++++-- docs/known-limitations.md | 2 +- docs/reference/cli.md | 10 +++- docs/reference/credentials.md | 19 +++++++ docs/reference/platform-config.md | 3 ++ docs/status.md | 2 +- 7 files changed, 115 insertions(+), 7 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 4ea7579..a2e5d23 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -30,6 +30,8 @@ Needs **urbex 0.1.0 or later, major 0**. ### Docs +- **Added**: getting Urbex's alerts on Telegram, Discord or a webhook - the cookbook's steps (BotFather, the chat id, Discord's webhook), `observability.alerts` in the platform config, the three credentials, `urbex doctor --test-alerts` (urbforge/urbex-cli#95). + - **Added**: [ADR-0034](docs/decisions/0034-forgejo-lts-forge.md) - Forgejo LTS is the platform's forge, in place of Gitea; the known limitation for platforms installed with 0.1.0, which stay on Gitea diff --git a/docs/cookbook.md b/docs/cookbook.md index f6389fc..a617a0a 100644 --- a/docs/cookbook.md +++ b/docs/cookbook.md @@ -64,6 +64,7 @@ copy of the GitOps repo. - [The reconciler holds applications back](#the-reconciler-holds-applications-back) - [The reconciler is not running](#the-reconciler-is-not-running) - [A disk is filling up, or a machine is running out of memory](#a-disk-is-filling-up-or-a-machine-is-running-out-of-memory) +- [Get the alerts on Telegram or Discord](#get-the-alerts-on-telegram-or-discord) - [An LXC was recreated](#an-lxc-was-recreated) - [Bootstrap was interrupted](#bootstrap-was-interrupted) - [Ansible refuses an LXC: host key changed](#ansible-refuses-an-lxc-host-key-changed) @@ -1185,9 +1186,86 @@ An LXC that stops reporting, stopped or destroyed, raises nothing. more with `resources.memory`; find what grows in *Urbex resources*, per container. -The alerts show in Grafana only (Alerting, Alert rules). To be notified, -add a contact point and a notification policy in Grafana; urbex leaves -them alone. +The alerts show in Grafana (Alerting, Alert rules), and in a chat once +you [get them on Telegram or Discord](#get-the-alerts-on-telegram-or-discord). + +## Get the alerts on Telegram or Discord + +Grafana sends Urbex's alerts - a disk filling up, memory running out, +the reconciler stuck - to a Telegram chat, a Discord channel or any +webhook, with one or several at once. Each message is one alert on one +machine, with a link to it in Grafana: + +``` +FIRING [warning] / on urbex-komodo is 91% full. - service komodo - urbex-komodo +https://grafana-urbex.example.com/alerting/grafana/urbex-disk-filling/view?orgId=1 +``` + +and `RESOLVED [warning] ...` when it clears. Alerts of the same rule and +machine are grouped, and repeated at most every 4 hours while they fire. + +**Telegram.** Create a bot: in Telegram, talk to +[@BotFather](https://t.me/BotFather), send `/newbot`, pick a name; it +answers with the bot's token (`123456789:AA...`). Add the bot to the +group that will get the alerts (or open a chat with it), send a message +there, then read the chat's id: + +```sh +curl -s "https://api.telegram.org/bot/getUpdates" | jq '.result[].message.chat | {id, title}' +``` + +A group's id is negative (`-1001234567890`). Store the token, then +declare the chat in `urbex.platform.yaml`: + +```sh +urbex-op set telegramBotToken +``` + +```yaml +observability: + alerts: + telegram: + chatId: "-1001234567890" +``` + +**Discord.** In the channel's settings, *Integrations* > *Webhooks* > +*New Webhook*, *Copy Webhook URL*. Then: + +```sh +urbex-op set discordWebhookUrl +``` + +```yaml +observability: + alerts: + discord: true +``` + +**Any webhook** (`webhook: true`, the credential `alertsWebhookUrl`) +gets Grafana's JSON for each notification. + +Then apply it to the platform, and send a test message: + +```sh +urbex bootstrap +urbex doctor --test-alerts +``` + +``` +alerts: + ok telegram test message sent + ok discord test message sent +``` + +The token and the URLs are kept encrypted in the observability Stack, +never in clear in the GitOps repo. Use a test chat first: every alert of +the platform goes there. Removing `observability.alerts` and running +`urbex bootstrap` again stops the messages. + +With `observability.alerts`, Urbex owns Grafana's notification policies +(they are provisioned, so Grafana shows them read-only): a contact +point or policy added by hand in Grafana's UI is not used. Without it, +Urbex leaves them alone. ## An LXC was recreated diff --git a/docs/known-limitations.md b/docs/known-limitations.md index ff9e6de..79d1c82 100644 --- a/docs/known-limitations.md +++ b/docs/known-limitations.md @@ -76,7 +76,7 @@ minor release may change how something works (see | Every level of `uses` starts at least 20 s after the one it uses, the time a provider must answer without a failure. | None; it is the price of the check. | | The reconciler acts once a minute, and Komodo refreshes its view of the containers about every 15 seconds, so an application whose provider came back starts a few minutes later. | Deploy its Stack in Komodo to have it at once. | | The reconciler sees a restart only between two runs. A provider that restarts twice within one run counts as one restart, and the crash loop alert comes at the second restart seen, within 10 minutes. | None at present; an alert on every restart of an application is planned. | -| Urbex's alerts - the reconciler's, and those on disk and memory - are only in Grafana, since no contact point is configured, so they notify nobody. | Add a contact point and a notification policy in Grafana ("Alerting", "Contact points"). | +| Urbex's alerts - the reconciler's, and those on disk and memory - notify nobody unless `observability.alerts` names a channel. With it, Urbex owns Grafana's notification policies, so a contact point added by hand isn't used; alerts can't be silenced from the chat. | [Get the alerts on Telegram or Discord](cookbook.md#get-the-alerts-on-telegram-or-discord); silence one in Grafana (Alerting, Silences). | | A platform bootstrapped before the reconciler existed has neither the `urbex-reconcile` Action nor its Grafana alerts. | Run `urbex bootstrap` again with the new CLI; it installs both. | | An application used by others needs an HTTP healthcheck or the `postgres` interface; a provider that speaks another protocol, such as Redis or MQTT, cannot be used yet. | Give it an HTTP healthcheck, if it has one. | | The wait runs `postgres:17-alpine`, pulled on the LXC of every application that uses others, by tag. | None at present; an image of Urbex's own, pinned by digest, is planned. | diff --git a/docs/reference/cli.md b/docs/reference/cli.md index bbb43fb..6b4c6b5 100644 --- a/docs/reference/cli.md +++ b/docs/reference/cli.md @@ -65,12 +65,12 @@ Shared flags: ## `urbex doctor` ``` -urbex doctor [--gitops-repo ] [--output text|json] +urbex doctor [--test-alerts] [--gitops-repo ] [--output text|json] ``` Checks that this machine, the workspace, Proxmox and Cloudflare have what urbex needs, and says what to do about what's missing. It changes -nothing and prints no secret; run it first, and whenever something +nothing (`--test-alerts` sends a test message) and prints no secret; run it first, and whenever something fails for no clear reason. - **tools**: `git`, `ssh`, `terraform`, `ansible-playbook`, `sops`, `age`, @@ -86,6 +86,12 @@ fails for no clear reason. exists. - **cloudflare**, with an account in the config: the token is active and sees the zone of `domain`. +- **alerts**, with `observability.alerts` in the config: each channel's + credential is there. With `--test-alerts`, Grafana sends a test + message through each channel, and each check says whether it went + out, or the channel's error (a wrong bot token, a chat the bot isn't + in). This one check sends something; the others change nothing. See + [get the alerts on Telegram or Discord](../cookbook.md#get-the-alerts-on-telegram-or-discord). It starts with urbex's own version, as [`urbex version`](#urbex-version) prints it. Each check is `ok`, `warn`, `fail` or `skip`; a check that diff --git a/docs/reference/credentials.md b/docs/reference/credentials.md index 463a6c8..e8bd5d3 100644 --- a/docs/reference/credentials.md +++ b/docs/reference/credentials.md @@ -323,6 +323,24 @@ bootstrap` again. https://api.cloudflare.com/client/v4/accounts//tokens/verify | jq .result.status # "active" ``` +## Required with alert channels + +With `observability.alerts` in `urbex.platform.yaml` +([platform config](platform-config.md#fields)), the channels' secrets. +You provide them; `urbex bootstrap` keeps them encrypted in the +observability Stack, and `urbex teardown` leaves them in the file. + +| Credentials file key | Env var | For | How to get it | +|---|---|---|---| +| `telegramBotToken` | `URBEX_TELEGRAM_BOT_TOKEN` | `observability.alerts.telegram` | [@BotFather](https://t.me/BotFather), `/newbot` | +| `discordWebhookUrl` | `URBEX_DISCORD_WEBHOOK_URL` | `observability.alerts.discord: true` | The channel's *Integrations* > *Webhooks* > *Copy Webhook URL* | +| `alertsWebhookUrl` | `URBEX_ALERTS_WEBHOOK_URL` | `observability.alerts.webhook: true` | Your service's URL | + +Store one with `urbex-op set `; [get the alerts on Telegram or +Discord](../cookbook.md#get-the-alerts-on-telegram-or-discord) has the +steps. `urbex doctor` says which one is missing, and `urbex doctor +--test-alerts` sends a test message with them. + ## Declared, but not required by any command yet These exist as fields in `urbex.platform.yaml`/`~/.urbex/credentials.yaml` @@ -351,4 +369,5 @@ item each is waiting on. | age keypair | `URBEX_AGE_KEY` | ✅ yes (encrypts every secret) | | Forgejo/Komodo/Keycloak/Grafana/Technitium passwords, tokens, keys | `URBEX_FORGE_*`, `URBEX_KOMODO_*`, `URBEX_TECHNITIUM_*`, ... | ✅ generated by `urbex bootstrap` | | Cloudflare API token | `URBEX_CLOUDFLARE_TOKEN` | ✅ with Cloudflare configured | +| Telegram bot token, Discord or other webhook URL | `URBEX_TELEGRAM_BOT_TOKEN`, `URBEX_DISCORD_WEBHOOK_URL`, `URBEX_ALERTS_WEBHOOK_URL` | ✅ with the matching alert channel | | Brevo API key | *(none defined)* | ❌ not yet | diff --git a/docs/reference/platform-config.md b/docs/reference/platform-config.md index 12af174..2c0257a 100644 --- a/docs/reference/platform-config.md +++ b/docs/reference/platform-config.md @@ -80,6 +80,9 @@ cloudflare: | `dns.forwarders` | `proxmox.network.dnsServers`, or the gateway | IP addresses Technitium forwards every name outside `dns.zone` to. | | `dns.technitiumUrl` | | ⚠️ Not used: `urbex` finds Technitium itself from the allocated LAN IP. | | `keycloak.baseUrl`, `keycloak.platformRealm` (`platform`) | | ⚠️ Not used yet (SSO). | +| `observability.alerts.telegram.chatId` | none | Send Urbex's alerts to this Telegram chat: a number, negative for a group, or a public channel's `@name`. Needs the credential `telegramBotToken`. See [get the alerts on Telegram or Discord](../cookbook.md#get-the-alerts-on-telegram-or-discord). | +| `observability.alerts.discord` | `false` | Send them to a Discord channel's webhook, the credential `discordWebhookUrl`. | +| `observability.alerts.webhook` | `false` | Post them as Grafana's JSON to the credential `alertsWebhookUrl`. Any of the three, or several; with none, the alerts stay in Grafana. Applied by `urbex bootstrap`; checked by `urbex doctor --test-alerts`. | | `storage.shares..path` | none | A folder of the Proxmox node that applications mount by name: see [shares](#shares). An absolute path that isn't `/` and has no comma, space or tab. | ## Shares diff --git a/docs/status.md b/docs/status.md index 32705c0..dace894 100644 --- a/docs/status.md +++ b/docs/status.md @@ -47,7 +47,7 @@ Proxmox VE 9.2 node, see [What has been validated](#what-has-been-validated). | `urbex status` | Proxmox container presence, allocation ledger and, per service, the version the GitOps repo asks for against what Komodo runs | | **Public endpoints (Cloudflare)** | One tunnel for the platform, `cloudflared` on `urbex-tunnel`; Keycloak public, Forgejo and Komodo behind Cloudflare Access (one-time PIN for listed e-mails), Komodo's webhook listener public; services with `public: true` published per environment; flat or nested hostnames ([ADR-0022](decisions/0022-cloudflare-tunnel-access-workers.md)) | | **Cloudflare Quick Tunnel** | `cloudflare.quickTunnel: true`, without an account, exposes every `public: true` service on its own `https://*.trycloudflare.com` hostname: a `cloudflared` sidecar on the service's LXC, `urbex apply`/`urbex status` reading the hostname live from it. Not a production substitute for ADR-0022 ([ADR-0025](decisions/0025-cloudflare-quick-tunnel.md)) | -| **Logs and resource monitoring** | A Grafana Alloy agent on every LXC - base services and project services - sends every container's logs to Loki, and the LXC's CPU, memory, disk and network and each container's CPU and memory to Prometheus, labelled `kind` (`platform`/`app`/`agent`, from the containers' `urbex.*` Docker labels), `service`, `host`, and `project`/`env` for apps; Grafana comes with both data sources and the *Urbex logs* and *Urbex resources* dashboards, and alerts when an LXC's disk or volume is filling up (85%, 95%) or its memory runs short (under 10% available); `observability.logs: false` stops a project's logs ([ADR-0023](decisions/0023-service-logs-to-loki.md)) | +| **Logs and resource monitoring** | A Grafana Alloy agent on every LXC - base services and project services - sends every container's logs to Loki, and the LXC's CPU, memory, disk and network and each container's CPU and memory to Prometheus, labelled `kind` (`platform`/`app`/`agent`, from the containers' `urbex.*` Docker labels), `service`, `host`, and `project`/`env` for apps; Grafana comes with both data sources and the *Urbex logs* and *Urbex resources* dashboards, and alerts when an LXC's disk or volume is filling up (85%, 95%) or its memory runs short (under 10% available), and sends the alerts to Telegram, Discord or a webhook with `observability.alerts` (`urbex doctor --test-alerts` tests them); `observability.logs: false` stops a project's logs ([ADR-0023](decisions/0023-service-logs-to-loki.md)) | | **Identity (Keycloak)** | A realm per project environment (`-`) with the services' roles and a public OIDC client per frontend; in staging a test user per role (`urbex users staging`) and a password-login client for tests; services with `auth.keycloak` get the issuer and keys ([ADR-0024](decisions/0024-keycloak-realm-per-environment.md)) | | **Internal DNS (Technitium)** | The platform's subdomain (`dns.zone`, default `.`) with a name per platform service, created by `urbex bootstrap`; `urbex apply ` sets one per declared service at its LAN IP (`api.staging.acme-app.urbex.`), `urbex destroy ` removes them; every LXC resolves through Technitium, which forwards other names to `dns.forwarders`; `urbex status` shows every internal name; services, image names, webhooks and tunnel routes reach each other by internal name, and project commands check the configured addresses against Technitium; independent of Cloudflare ([ADR-0027](decisions/0027-service-conventions.md)) | | **Komodo** | Every LXC is a Komodo Server - the base services' too, for monitoring their containers - and Komodo's resources are tagged like Grafana: `platform`, or `app` + project (+ environment) |