diff --git a/CHANGELOG.md b/CHANGELOG.md index 4ea7579..db7452f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -30,6 +30,8 @@ Needs **urbex 0.1.0 or later, major 0**. ### Docs +- **Added**: the Proxmox node's metrics and alerts with an optional read-only token - the token in the credentials reference and getting started, the alerts in the disk recipe, `pve-exporter` in ADR-0027's services (urbforge/urbex-cli#94). + - **Added**: [ADR-0034](docs/decisions/0034-forgejo-lts-forge.md) - Forgejo LTS is the platform's forge, in place of Gitea; the known limitation for platforms installed with 0.1.0, which stay on Gitea diff --git a/docs/cookbook.md b/docs/cookbook.md index f6389fc..f888910 100644 --- a/docs/cookbook.md +++ b/docs/cookbook.md @@ -1166,6 +1166,16 @@ group `urbex-resources`), for the platform's LXCs and the applications': | A disk is filling up | a filesystem above 85% for 10 minutes | `warning` | | A disk is almost full | a filesystem above 95% for 5 minutes | `critical` | | A machine is running out of memory | less than 10% of its memory available for 10 minutes | `warning` | +| The Proxmox node is running out of memory | the node's memory above 90% used for 10 minutes | `warning` | +| A Proxmox storage is filling up | a storage of the node above 85% for 5 minutes | `warning` | +| A Proxmox storage is nearly full | a storage of the node above 95% for 5 minutes | `critical` | + +The last three, and the *Proxmox node* row of the *Urbex resources* +dashboard (CPU, memory used against its total, each storage's use), need +the optional read-only [monitoring token](reference/credentials.md#proxmox-monitoring-token): +the node's memory counts what Proxmox's *Summary* counts, the guests +outside urbex and ZFS's ARC included. They carry `service="proxmox"`, +`node` and `storage`. Every real filesystem counts: the LXC's root disk and its named volumes (`/srv/volumes//`), not `tmpfs`. Each alert carries the diff --git a/docs/decisions/0027-service-conventions.md b/docs/decisions/0027-service-conventions.md index 79f7ac0..bc2930f 100644 --- a/docs/decisions/0027-service-conventions.md +++ b/docs/decisions/0027-service-conventions.md @@ -171,7 +171,7 @@ where their compose files live and in who can repair them: | Kind | What | Komodo tag | Grafana label | |---|---|---|---| -| the platform's services, created by Urbex | Gitea, Komodo, Technitium, Keycloak, Prometheus, Loki, Grafana, the tunnel | `platform` | `kind="platform"` | +| the platform's services, created by Urbex | Gitea, Komodo, Technitium, Keycloak, Prometheus, Loki, Grafana, `pve-exporter`, the tunnel | `platform` | `kind="platform"` | | the users' applications | a project's services, frontends and databases | `app` | `kind="app"` | | technical agents running next to another service | Periphery, Alloy, the Quick Tunnel sidecar ([ADR-0025](0025-cloudflare-quick-tunnel.md)) | `agent` | `kind="agent"` | @@ -199,6 +199,7 @@ where their compose files live and in who can repair them: | Grafana | `urbex-observability` | `grafana.` | behind Access | platform | | Prometheus | `urbex-observability` | `prometheus.` | UI behind Access; remote write internal | platform | | Loki | `urbex-observability` | `loki.` | API behind Access; push internal | platform | +| `pve-exporter` - the Proxmox node's metrics, only with the [monitoring token](../reference/credentials.md#proxmox-monitoring-token) | `urbex-observability` | - (scraped inside the Stack) | none (calls the node's API) | platform | | Tunnel (`cloudflared`) | `urbex-tunnel` | `tunnel.` | none (outbound only) | platform | | Periphery | every LXC | - (connects out to Komodo) | internal | agent | | Alloy | every LXC | - (pushes out) | internal | agent | diff --git a/docs/getting-started.md b/docs/getting-started.md index c28eb29..519160a 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -72,7 +72,9 @@ creating the platform and the environments, releasing, promoting. pveum user token add urbex@pve cli --privsep 0 --comment "urbex" # prints the token once: keep it ``` The token is `urbex@pve!cli=`, with the value the last command - prints. + prints. **Optional:** a second, read-only token puts the node's own + CPU, memory and storages in Grafana, with alerts + ([Proxmox monitoring token](reference/credentials.md#proxmox-monitoring-token)). - **Memory, disk and CPU**: about **6 GiB of free memory** (8 to be comfortable) and **30 GiB of free disk** for the platform and a first app in two environments. Measured on the test node (Proxmox VE 9.2, diff --git a/docs/known-limitations.md b/docs/known-limitations.md index ff9e6de..8bf21ce 100644 --- a/docs/known-limitations.md +++ b/docs/known-limitations.md @@ -116,7 +116,7 @@ minor release may change how something works (see | **The internal zone (`urbex.`) shadows Cloudflare on the LAN**: Technitium answers for every name under it, so a record you created there on Cloudflare doesn't resolve from the LXCs; with `nested` public hostnames, neither do the web frontends' (they have no LXC, so no internal record) ([ADR-0027](decisions/0027-service-conventions.md)). | Keep your own records out of the platform's subdomain, or set `dns.zone`. | | **Without Cloudflare, the token issuer is Keycloak's LAN address** (`KEYCLOAK_ISSUER`): the address your users' browsers reach it at, which the tokens carry. The services fetch the signing keys by internal name. | Configure Cloudflare for a stable, public issuer. | | **The internal names don't resolve on your machine** unless it asks Technitium: images (`git.:3000/...`), the links Forgejo and Komodo generate, and the URLs services are configured with use the internal names ([ADR-0027](decisions/0027-service-conventions.md)). The CLI uses the LAN addresses, after checking them against Technitium. | Have your router forward the zone to Technitium ([cookbook](cookbook.md#resolve-a-service-by-name-on-the-lan)). | -| **Logs and resource metrics only** ([ADR-0023](decisions/0023-service-logs-to-loki.md), [ADR-0027](decisions/0027-service-conventions.md)): the services' own metrics (`observability.metrics`) aren't scraped, per-container metrics are CPU and memory only, the only alerts are the reconciler's and those on the LXCs' disk and memory - not on the Proxmox node's own memory, CPU and storages, nor on a VM - and web frontends' runtime logs are on Cloudflare (their deploy container's are in Loki). | Cloudflare's Workers logs for frontends. | +| **Logs and resource metrics only** ([ADR-0023](decisions/0023-service-logs-to-loki.md), [ADR-0027](decisions/0027-service-conventions.md)): the services' own metrics (`observability.metrics`) aren't scraped, per-container metrics are CPU and memory only, the only alerts are the reconciler's and those on the LXCs' disk and memory - on the Proxmox node's memory and storages only with the optional [monitoring token](reference/credentials.md#proxmox-monitoring-token), never on a VM - and web frontends' runtime logs are on Cloudflare (their deploy container's are in Loki). | Cloudflare's Workers logs for frontends. | | **Loki keeps logs forever**: default single-node configuration, no retention set; the observability LXC's disk fills up over time. | Grow its disk (`terraform/base`), or clean Loki's volume. | | **Grafana's admin password** is the only login (no Keycloak SSO), after Cloudflare Access when reached from outside. | `admin` / `grafanaAdminPassword` from `~/.urbex/credentials.yaml`. | | **Transactional email** (`email.provider: brevo`) is validated only. | - | diff --git a/docs/reference/credentials.md b/docs/reference/credentials.md index 463a6c8..ee9029d 100644 --- a/docs/reference/credentials.md +++ b/docs/reference/credentials.md @@ -91,6 +91,37 @@ they do need the - **Self-signed certificate:** set `proxmox.insecure: true` if the node still uses Proxmox's default certificate. +### Proxmox monitoring token + +Optional. A **read-only** token of the whole node, for the node's CPU, +memory and storages in Grafana (urbex's own token sees only the pool +`urbex`, and Proxmox answers 403 when it asks for the node). + +- **Credentials file key / env var:** `proxmoxMonitorToken` / + `URBEX_PROXMOX_MONITOR_TOKEN`, as `user@realm!name=secret`. +- **Used by:** `urbex bootstrap`, which runs `pve-exporter` in the + observability Stack only when the token is there, with the token in + the Stack's encrypted secrets; `urbex doctor`, which checks it, or says + the node metrics are off without it. +- **Create it**, once, as root on the node: + ```sh + pveum user add urbex-monitor@pve --comment "Urbex: read-only metrics" + pveum acl modify / --users urbex-monitor@pve --roles PVEAuditor + pveum user token add urbex-monitor@pve metrics --privsep 0 --comment "urbex metrics" # prints the token once + ``` + Then `urbex-op set proxmoxMonitorToken` with `urbex-monitor@pve!metrics=`, + and `urbex bootstrap`. +- **What it can do:** `PVEAuditor` only reads (`Sys.Audit`, + `Datastore.Audit`, `VM.Audit`). It sees every guest of the node, those + outside the pool included; Urbex keeps only the node's and the + storages' series, never the guests'. +- **TLS:** `pve-exporter` reaches the node by its address under another + name, so it doesn't verify the node's certificate (Proxmox's own is + self-signed). The token only reads. +- **Turning it off:** remove the key and run `urbex bootstrap`; on the + observability LXC, `docker rm -f urbex-pve-exporter-1` stops an exporter + already running. + ### SSH key - **Where it's set:** `proxmox.sshPublicKey` in `urbex.platform.yaml` @@ -351,4 +382,5 @@ item each is waiting on. | age keypair | `URBEX_AGE_KEY` | ✅ yes (encrypts every secret) | | Forgejo/Komodo/Keycloak/Grafana/Technitium passwords, tokens, keys | `URBEX_FORGE_*`, `URBEX_KOMODO_*`, `URBEX_TECHNITIUM_*`, ... | ✅ generated by `urbex bootstrap` | | Cloudflare API token | `URBEX_CLOUDFLARE_TOKEN` | ✅ with Cloudflare configured | +| Proxmox monitoring token (read-only) | `URBEX_PROXMOX_MONITOR_TOKEN` | ❌ optional: the node's metrics in Grafana | | Brevo API key | *(none defined)* | ❌ not yet | diff --git a/docs/status.md b/docs/status.md index 32705c0..30b2d9f 100644 --- a/docs/status.md +++ b/docs/status.md @@ -71,7 +71,7 @@ Proxmox VE 9.2 node, see [What has been validated](#what-has-been-validated). | Frontend-only projects | A web frontend needs at least one service in the manifest | [0022](decisions/0022-cloudflare-tunnel-access-workers.md) | | Keycloak beyond projects | No `platform` realm (admin SSO), no confidential clients for service-to-service calls | [0008](decisions/0008-keycloak-scope.md), [0024](decisions/0024-keycloak-realm-per-environment.md) | | Terraform state encryption | State is plain JSON in the (private) GitOps repo, not SOPS-encrypted as designed | [0013](decisions/0013-terraform-state-in-gitops-repo.md) | -| Service metrics, alerting | LXC resource metrics are collected, but not the services' own metrics (`observability.metrics` is ignored); no alerts beyond the reconciler's and the LXCs' disk and memory (not the Proxmox node's, not a VM's); Loki has no retention | [0023](decisions/0023-service-logs-to-loki.md) | +| Service metrics, alerting | LXC resource metrics are collected, but not the services' own metrics (`observability.metrics` is ignored); no alerts beyond the reconciler's and the LXCs' disk and memory (the Proxmox node's memory and storages with the optional monitoring token; not a VM's); Loki has no retention | [0023](decisions/0023-service-logs-to-loki.md) | | Mobile frontend deploy | `frontend.type: mobile` (Firebase) is validated, not deployed | - | | Fleet-wide status | `urbex status` is scoped to one project+environment at a time; Komodo's UI is the cross-project view | - | | Per-project Proxmox isolation | New projects should get their own resource pool, a group for their users with minimal permissions, and dedicated technical users; today every LXC goes in the platform-wide `proxmox.pool`, managed with the operator's token | [roadmap](roadmap.md) |