Skip to content

feat(observability): send Urbex's alerts to Telegram, Discord or a webhook - #131

Open
Teg79 wants to merge 4 commits into
mainfrom
feat/alerts-telegram-discord
Open

Teg79 wants to merge 4 commits into
mainfrom
feat/alerts-telegram-discord

Conversation

@Teg79

@Teg79 Teg79 commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Urbex's alerts reach a Telegram chat, a Discord channel or a webhook (#95), through Grafana's own integrations: no extra service.

  • internal/platform/config.go: observability.alerts in urbex.platform.yaml: telegram.chatId, discord: true, webhook: true, one or several. The chat id is validated: a number, or a channel's @name.

  • internal/credentials/credentials.go: telegramBotToken, discordWebhookUrl, alertsWebhookUrl (URBEX_TELEGRAM_BOT_TOKEN, ...). They are user-provided, so teardown keeps them.

  • internal/bootstrap/alerts.go (new): generates grafana/provisioning/alerting/urbex-notifications.yml in the observability Stack, with:

    • a contact point urbex, one integration per channel;
    • the notification policy, grouping by alertname, service, host: group_wait 30s, group_interval 5m, repeat_interval 4h.

    The secrets are referenced as $URBEX_ALERTS_*, which Grafana expands from its environment. They are in the Stack's secrets.sops.env, never in clear in the GitOps repo. With no channel the file is removed and nothing changes.

  • The message, one per alert and machine, with resolved messages too:

    FIRING [warning] / on urbex-komodo is 91% full. - service komodo - urbex-komodo
    https://grafana-urbex.example.com/alerting/grafana/urbex-disk-filling/view?orgId=1
    
    • It is inline in each integration, not a named template: Grafana 13's Alertmanager loads provisioned templates up to a minute after it starts, and messages sent before then were empty. The first lab run hit this.
    • A rule Grafana couldn't evaluate (DatasourceError) is named, instead of an empty summary.
  • internal/bootstrap/stacks.go:

    • PlatformStack.Generated: files written beside the embedded ones and hashed with them, so Grafana restarts when they change; a path mapped to "" is removed;
    • with channels, GF_SERVER_ROOT_URL is set: Grafana behind Access with Cloudflare, else its internal name. The alerts' links point there.
  • internal/grafana (new) and internal/doctor/alerts.go: urbex doctor checks each channel's credential. --test-alerts sends a test message through each channel with Grafana 13's receivers API (POST .../receivers/{uid}/test), using the secrets Grafana holds. It reports each channel's error, with a fix for the usual ones (wrong bot token, chat not found).

  • platform/stacks/observability/.../urbex-reconcile.yml: the reconciler's two alerts carry a severity (critical when it doesn't run, warning when it holds applications back).

  • CHANGELOG.md: under Unreleased.

Companion: urbforge/urbex#100.

Why

#95: the alerts only showed in Grafana, so nobody heard about them.

Tests I ran

  • mise run test, mise run lint: pass. New: the provisioning file (valid YAML, no secret in clear, the missing credentials named), the chat id validation, the doctor checks, the Grafana client against a fake of the receivers API.

  • mise run e2e: passed, 24 steps, on f6e8793. The later commits change only the generated message text and two alert labels.

  • Grafana 13.2.3 in podman, with this provisioning:

    • contact point, policy and the 5 rules accepted;
    • a test rule fired and cleared: FIRING, then RESOLVED after group_interval, to Discord and the webhook (pointed at a local listener);
    • the client's test against the live API: Discord and webhook ok, Telegram answered with the bot's real errors.
    • It caught one bug before the lab: Grafana 13 has no grafana-default-email contact point, so a policy routing to it stopped Grafana from starting.
  • On the test node (lab-cubotto/alerts-test.sh), with a test Telegram bot and chat, not the team's group:

    • urbex bootstrap took 161 s;
    • urbex doctor --test-alerts ok;
    • a temporary rule in the urbex folder fired, cleared and was deleted;
    • a second urbex bootstrap changed nothing.

    The first run failed on the empty message (see above); fixed, then passed. Messages received: in a comment below.

  • Not run: a real disk filling up on the lab (the temporary rule takes the same path); Discord against a real channel.

Checklist before review

  • Linked issue, Conventional Commit title
  • Tests added (unit; Grafana 13.2.3 in podman; the lab)
  • Secrets only in credentials and the encrypted Stack secrets, never printed
  • CHANGELOG entry, docs in the companion PR
  • No secrets, tokens or private addresses

Fixes #95

🤖 Generated with Claude Code

https://claude.ai/code/session_0182yxsfwRowwnxCfgrQwjnA

Teg79 and others added 4 commits November 19, 2026 09:04
…--test-alerts

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0182yxsfwRowwnxCfgrQwjnA
…emplate Grafana loads a minute late

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0182yxsfwRowwnxCfgrQwjnA
@Teg79

Teg79 commented Nov 19, 2026

Copy link
Copy Markdown
Contributor Author

Lab run (test node, 2026-11-19, lab-cubotto/alerts-test.sh, a test Telegram bot and chat). Received in the chat, in order (the domain replaced with example.com):

FIRING [info] A test from urbex doctor --test-alerts: this channel gets Urbex's alerts.

FIRING [warning] A test alert of urbex-cli#95: / on urbex-komodo is 91% full (not really). - service komodo - urbex-komodo
https://grafana-urbex.example.com/alerting/grafana/urbex-alerts-test/view?orgId=1

RESOLVED [warning] A test alert of urbex-cli#95: / on urbex-komodo is 91% full (not really). - service komodo - urbex-komodo
https://grafana-urbex.example.com/alerting/grafana/urbex-alerts-test/view?orgId=1

The link is Grafana behind Access (GF_SERVER_ROOT_URL). The script's run: urbex bootstrap 161 s, urbex doctor --test-alerts ok, the temporary rule fired, cleared (RESOLVED at the next group interval) and was deleted, a second urbex bootstrap changed nothing.

@Teg79
Teg79 marked this pull request as ready for review November 19, 2026 14:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(observability): send Urbex's alerts to a Telegram bot or Discord

2 participants