Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,8 @@ Needs **urbex 0.1.0 or later, major 0**.

### Docs

- **Added**: getting Urbex's alerts on Telegram, Discord or a webhook - the cookbook's steps (BotFather, the chat id, Discord's webhook), `observability.alerts` in the platform config, the three credentials, `urbex doctor --test-alerts` (urbforge/urbex-cli#95).

- **Added**: [ADR-0034](docs/decisions/0034-forgejo-lts-forge.md) -
Forgejo LTS is the platform's forge, in place of Gitea; the known
limitation for platforms installed with 0.1.0, which stay on Gitea
Expand Down
84 changes: 81 additions & 3 deletions docs/cookbook.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,7 @@ copy of the GitOps repo.
- [The reconciler holds applications back](#the-reconciler-holds-applications-back)
- [The reconciler is not running](#the-reconciler-is-not-running)
- [A disk is filling up, or a machine is running out of memory](#a-disk-is-filling-up-or-a-machine-is-running-out-of-memory)
- [Get the alerts on Telegram or Discord](#get-the-alerts-on-telegram-or-discord)
- [An LXC was recreated](#an-lxc-was-recreated)
- [Bootstrap was interrupted](#bootstrap-was-interrupted)
- [Ansible refuses an LXC: host key changed](#ansible-refuses-an-lxc-host-key-changed)
Expand Down Expand Up @@ -1185,9 +1186,86 @@ An LXC that stops reporting, stopped or destroyed, raises nothing.
more with `resources.memory`; find what grows in *Urbex resources*,
per container.

The alerts show in Grafana only (Alerting, Alert rules). To be notified,
add a contact point and a notification policy in Grafana; urbex leaves
them alone.
The alerts show in Grafana (Alerting, Alert rules), and in a chat once
you [get them on Telegram or Discord](#get-the-alerts-on-telegram-or-discord).

## Get the alerts on Telegram or Discord

Grafana sends Urbex's alerts - a disk filling up, memory running out,
the reconciler stuck - to a Telegram chat, a Discord channel or any
webhook, with one or several at once. Each message is one alert on one
machine, with a link to it in Grafana:

```
FIRING [warning] / on urbex-komodo is 91% full. - service komodo - urbex-komodo
https://grafana-urbex.example.com/alerting/grafana/urbex-disk-filling/view?orgId=1
```

and `RESOLVED [warning] ...` when it clears. Alerts of the same rule and
machine are grouped, and repeated at most every 4 hours while they fire.

**Telegram.** Create a bot: in Telegram, talk to
[@BotFather](https://t.me/BotFather), send `/newbot`, pick a name; it
answers with the bot's token (`123456789:AA...`). Add the bot to the
group that will get the alerts (or open a chat with it), send a message
there, then read the chat's id:

```sh
curl -s "https://api.telegram.org/bot<token>/getUpdates" | jq '.result[].message.chat | {id, title}'
```

A group's id is negative (`-1001234567890`). Store the token, then
declare the chat in `urbex.platform.yaml`:

```sh
urbex-op set telegramBotToken
```

```yaml
observability:
alerts:
telegram:
chatId: "-1001234567890"
```

**Discord.** In the channel's settings, *Integrations* > *Webhooks* >
*New Webhook*, *Copy Webhook URL*. Then:

```sh
urbex-op set discordWebhookUrl
```

```yaml
observability:
alerts:
discord: true
```

**Any webhook** (`webhook: true`, the credential `alertsWebhookUrl`)
gets Grafana's JSON for each notification.

Then apply it to the platform, and send a test message:

```sh
urbex bootstrap
urbex doctor --test-alerts
```

```
alerts:
ok telegram test message sent
ok discord test message sent
```

The token and the URLs are kept encrypted in the observability Stack,
never in clear in the GitOps repo. Use a test chat first: every alert of
the platform goes there. Removing `observability.alerts` and running
`urbex bootstrap` again stops the messages.

With `observability.alerts`, Urbex owns Grafana's notification policies
(they are provisioned, so Grafana shows them read-only): a contact
point or policy added by hand in Grafana's UI is not used. Without it,
Urbex leaves them alone.

## An LXC was recreated

Expand Down
2 changes: 1 addition & 1 deletion docs/known-limitations.md
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,7 @@ minor release may change how something works (see
| Every level of `uses` starts at least 20 s after the one it uses, the time a provider must answer without a failure. | None; it is the price of the check. |
| The reconciler acts once a minute, and Komodo refreshes its view of the containers about every 15 seconds, so an application whose provider came back starts a few minutes later. | Deploy its Stack in Komodo to have it at once. |
| The reconciler sees a restart only between two runs. A provider that restarts twice within one run counts as one restart, and the crash loop alert comes at the second restart seen, within 10 minutes. | None at present; an alert on every restart of an application is planned. |
| Urbex's alerts - the reconciler's, and those on disk and memory - are only in Grafana, since no contact point is configured, so they notify nobody. | Add a contact point and a notification policy in Grafana ("Alerting", "Contact points"). |
| Urbex's alerts - the reconciler's, and those on disk and memory - notify nobody unless `observability.alerts` names a channel. With it, Urbex owns Grafana's notification policies, so a contact point added by hand isn't used; alerts can't be silenced from the chat. | [Get the alerts on Telegram or Discord](cookbook.md#get-the-alerts-on-telegram-or-discord); silence one in Grafana (Alerting, Silences). |
| A platform bootstrapped before the reconciler existed has neither the `urbex-reconcile` Action nor its Grafana alerts. | Run `urbex bootstrap` again with the new CLI; it installs both. |
| An application used by others needs an HTTP healthcheck or the `postgres` interface; a provider that speaks another protocol, such as Redis or MQTT, cannot be used yet. | Give it an HTTP healthcheck, if it has one. |
| The wait runs `postgres:17-alpine`, pulled on the LXC of every application that uses others, by tag. | None at present; an image of Urbex's own, pinned by digest, is planned. |
Expand Down
10 changes: 8 additions & 2 deletions docs/reference/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -65,12 +65,12 @@ Shared flags:
## `urbex doctor`

```
urbex doctor [--gitops-repo <dir>] [--output text|json]
urbex doctor [--test-alerts] [--gitops-repo <dir>] [--output text|json]
```

Checks that this machine, the workspace, Proxmox and Cloudflare have
what urbex needs, and says what to do about what's missing. It changes
nothing and prints no secret; run it first, and whenever something
nothing (`--test-alerts` sends a test message) and prints no secret; run it first, and whenever something
fails for no clear reason.

- **tools**: `git`, `ssh`, `terraform`, `ansible-playbook`, `sops`, `age`,
Expand All @@ -86,6 +86,12 @@ fails for no clear reason.
exists.
- **cloudflare**, with an account in the config: the token is active and
sees the zone of `domain`.
- **alerts**, with `observability.alerts` in the config: each channel's
credential is there. With `--test-alerts`, Grafana sends a test
message through each channel, and each check says whether it went
out, or the channel's error (a wrong bot token, a chat the bot isn't
in). This one check sends something; the others change nothing. See
[get the alerts on Telegram or Discord](../cookbook.md#get-the-alerts-on-telegram-or-discord).

It starts with urbex's own version, as [`urbex version`](#urbex-version)
prints it. Each check is `ok`, `warn`, `fail` or `skip`; a check that
Expand Down
19 changes: 19 additions & 0 deletions docs/reference/credentials.md
Original file line number Diff line number Diff line change
Expand Up @@ -323,6 +323,24 @@ bootstrap` again.
https://api.cloudflare.com/client/v4/accounts/<account id>/tokens/verify | jq .result.status # "active"
```

## Required with alert channels

With `observability.alerts` in `urbex.platform.yaml`
([platform config](platform-config.md#fields)), the channels' secrets.
You provide them; `urbex bootstrap` keeps them encrypted in the
observability Stack, and `urbex teardown` leaves them in the file.

| Credentials file key | Env var | For | How to get it |
|---|---|---|---|
| `telegramBotToken` | `URBEX_TELEGRAM_BOT_TOKEN` | `observability.alerts.telegram` | [@BotFather](https://t.me/BotFather), `/newbot` |
| `discordWebhookUrl` | `URBEX_DISCORD_WEBHOOK_URL` | `observability.alerts.discord: true` | The channel's *Integrations* > *Webhooks* > *Copy Webhook URL* |
| `alertsWebhookUrl` | `URBEX_ALERTS_WEBHOOK_URL` | `observability.alerts.webhook: true` | Your service's URL |

Store one with `urbex-op set <key>`; [get the alerts on Telegram or
Discord](../cookbook.md#get-the-alerts-on-telegram-or-discord) has the
steps. `urbex doctor` says which one is missing, and `urbex doctor
--test-alerts` sends a test message with them.

## Declared, but not required by any command yet

These exist as fields in `urbex.platform.yaml`/`~/.urbex/credentials.yaml`
Expand Down Expand Up @@ -351,4 +369,5 @@ item each is waiting on.
| age keypair | `URBEX_AGE_KEY` | ✅ yes (encrypts every secret) |
| Forgejo/Komodo/Keycloak/Grafana/Technitium passwords, tokens, keys | `URBEX_FORGE_*`, `URBEX_KOMODO_*`, `URBEX_TECHNITIUM_*`, ... | ✅ generated by `urbex bootstrap` |
| Cloudflare API token | `URBEX_CLOUDFLARE_TOKEN` | ✅ with Cloudflare configured |
| Telegram bot token, Discord or other webhook URL | `URBEX_TELEGRAM_BOT_TOKEN`, `URBEX_DISCORD_WEBHOOK_URL`, `URBEX_ALERTS_WEBHOOK_URL` | ✅ with the matching alert channel |
| Brevo API key | *(none defined)* | ❌ not yet |
3 changes: 3 additions & 0 deletions docs/reference/platform-config.md
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,9 @@ cloudflare:
| `dns.forwarders` | `proxmox.network.dnsServers`, or the gateway | IP addresses Technitium forwards every name outside `dns.zone` to. |
| `dns.technitiumUrl` | | ⚠️ Not used: `urbex` finds Technitium itself from the allocated LAN IP. |
| `keycloak.baseUrl`, `keycloak.platformRealm` (`platform`) | | ⚠️ Not used yet (SSO). |
| `observability.alerts.telegram.chatId` | none | Send Urbex's alerts to this Telegram chat: a number, negative for a group, or a public channel's `@name`. Needs the credential `telegramBotToken`. See [get the alerts on Telegram or Discord](../cookbook.md#get-the-alerts-on-telegram-or-discord). |
| `observability.alerts.discord` | `false` | Send them to a Discord channel's webhook, the credential `discordWebhookUrl`. |
| `observability.alerts.webhook` | `false` | Post them as Grafana's JSON to the credential `alertsWebhookUrl`. Any of the three, or several; with none, the alerts stay in Grafana. Applied by `urbex bootstrap`; checked by `urbex doctor --test-alerts`. |
| `storage.shares.<name>.path` | none | A folder of the Proxmox node that applications mount by name: see [shares](#shares). An absolute path that isn't `/` and has no comma, space or tab. |

## Shares
Expand Down
2 changes: 1 addition & 1 deletion docs/status.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ Proxmox VE 9.2 node, see [What has been validated](#what-has-been-validated).
| `urbex status` | Proxmox container presence, allocation ledger and, per service, the version the GitOps repo asks for against what Komodo runs |
| **Public endpoints (Cloudflare)** | One tunnel for the platform, `cloudflared` on `urbex-tunnel`; Keycloak public, Forgejo and Komodo behind Cloudflare Access (one-time PIN for listed e-mails), Komodo's webhook listener public; services with `public: true` published per environment; flat or nested hostnames ([ADR-0022](decisions/0022-cloudflare-tunnel-access-workers.md)) |
| **Cloudflare Quick Tunnel** | `cloudflare.quickTunnel: true`, without an account, exposes every `public: true` service on its own `https://*.trycloudflare.com` hostname: a `cloudflared` sidecar on the service's LXC, `urbex apply`/`urbex status` reading the hostname live from it. Not a production substitute for ADR-0022 ([ADR-0025](decisions/0025-cloudflare-quick-tunnel.md)) |
| **Logs and resource monitoring** | A Grafana Alloy agent on every LXC - base services and project services - sends every container's logs to Loki, and the LXC's CPU, memory, disk and network and each container's CPU and memory to Prometheus, labelled `kind` (`platform`/`app`/`agent`, from the containers' `urbex.*` Docker labels), `service`, `host`, and `project`/`env` for apps; Grafana comes with both data sources and the *Urbex logs* and *Urbex resources* dashboards, and alerts when an LXC's disk or volume is filling up (85%, 95%) or its memory runs short (under 10% available); `observability.logs: false` stops a project's logs ([ADR-0023](decisions/0023-service-logs-to-loki.md)) |
| **Logs and resource monitoring** | A Grafana Alloy agent on every LXC - base services and project services - sends every container's logs to Loki, and the LXC's CPU, memory, disk and network and each container's CPU and memory to Prometheus, labelled `kind` (`platform`/`app`/`agent`, from the containers' `urbex.*` Docker labels), `service`, `host`, and `project`/`env` for apps; Grafana comes with both data sources and the *Urbex logs* and *Urbex resources* dashboards, and alerts when an LXC's disk or volume is filling up (85%, 95%) or its memory runs short (under 10% available), and sends the alerts to Telegram, Discord or a webhook with `observability.alerts` (`urbex doctor --test-alerts` tests them); `observability.logs: false` stops a project's logs ([ADR-0023](decisions/0023-service-logs-to-loki.md)) |
| **Identity (Keycloak)** | A realm per project environment (`<project>-<env>`) with the services' roles and a public OIDC client per frontend; in staging a test user per role (`urbex users staging`) and a password-login client for tests; services with `auth.keycloak` get the issuer and keys ([ADR-0024](decisions/0024-keycloak-realm-per-environment.md)) |
| **Internal DNS (Technitium)** | The platform's subdomain (`dns.zone`, default `<name>.<domain>`) with a name per platform service, created by `urbex bootstrap`; `urbex apply <env>` sets one per declared service at its LAN IP (`api.staging.acme-app.urbex.<domain>`), `urbex destroy <env>` removes them; every LXC resolves through Technitium, which forwards other names to `dns.forwarders`; `urbex status` shows every internal name; services, image names, webhooks and tunnel routes reach each other by internal name, and project commands check the configured addresses against Technitium; independent of Cloudflare ([ADR-0027](decisions/0027-service-conventions.md)) |
| **Komodo** | Every LXC is a Komodo Server - the base services' too, for monitoring their containers - and Komodo's resources are tagged like Grafana: `platform`, or `app` + project (+ environment) |
Expand Down