Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,8 @@ Needs **urbex 0.1.0 or later, major 0**.

### Docs

- **Added**: the Proxmox node's metrics and alerts with an optional read-only token - the token in the credentials reference and getting started, the alerts in the disk recipe, `pve-exporter` in ADR-0027's services (urbforge/urbex-cli#94).

- **Added**: [ADR-0034](docs/decisions/0034-forgejo-lts-forge.md) -
Forgejo LTS is the platform's forge, in place of Gitea; the known
limitation for platforms installed with 0.1.0, which stay on Gitea
Expand Down
10 changes: 10 additions & 0 deletions docs/cookbook.md
Original file line number Diff line number Diff line change
Expand Up @@ -1166,6 +1166,16 @@ group `urbex-resources`), for the platform's LXCs and the applications':
| A disk is filling up | a filesystem above 85% for 10 minutes | `warning` |
| A disk is almost full | a filesystem above 95% for 5 minutes | `critical` |
| A machine is running out of memory | less than 10% of its memory available for 10 minutes | `warning` |
| The Proxmox node is running out of memory | the node's memory above 90% used for 10 minutes | `warning` |
| A Proxmox storage is filling up | a storage of the node above 85% for 5 minutes | `warning` |
| A Proxmox storage is nearly full | a storage of the node above 95% for 5 minutes | `critical` |

The last three, and the *Proxmox node* row of the *Urbex resources*
dashboard (CPU, memory used against its total, each storage's use), need
the optional read-only [monitoring token](reference/credentials.md#proxmox-monitoring-token):
the node's memory counts what Proxmox's *Summary* counts, the guests
outside urbex and ZFS's ARC included. They carry `service="proxmox"`,
`node` and `storage`.

Every real filesystem counts: the LXC's root disk and its named volumes
(`/srv/volumes/<service>/<volume>`), not `tmpfs`. Each alert carries the
Expand Down
3 changes: 2 additions & 1 deletion docs/decisions/0027-service-conventions.md
Original file line number Diff line number Diff line change
Expand Up @@ -171,7 +171,7 @@ where their compose files live and in who can repair them:

| Kind | What | Komodo tag | Grafana label |
|---|---|---|---|
| the platform's services, created by Urbex | Gitea, Komodo, Technitium, Keycloak, Prometheus, Loki, Grafana, the tunnel | `platform` | `kind="platform"` |
| the platform's services, created by Urbex | Gitea, Komodo, Technitium, Keycloak, Prometheus, Loki, Grafana, `pve-exporter`, the tunnel | `platform` | `kind="platform"` |
| the users' applications | a project's services, frontends and databases | `app` | `kind="app"` |
| technical agents running next to another service | Periphery, Alloy, the Quick Tunnel sidecar ([ADR-0025](0025-cloudflare-quick-tunnel.md)) | `agent` | `kind="agent"` |

Expand Down Expand Up @@ -199,6 +199,7 @@ where their compose files live and in who can repair them:
| Grafana | `urbex-observability` | `grafana.<zone>` | behind Access | platform |
| Prometheus | `urbex-observability` | `prometheus.<zone>` | UI behind Access; remote write internal | platform |
| Loki | `urbex-observability` | `loki.<zone>` | API behind Access; push internal | platform |
| `pve-exporter` - the Proxmox node's metrics, only with the [monitoring token](../reference/credentials.md#proxmox-monitoring-token) | `urbex-observability` | - (scraped inside the Stack) | none (calls the node's API) | platform |
| Tunnel (`cloudflared`) | `urbex-tunnel` | `tunnel.<zone>` | none (outbound only) | platform |
| Periphery | every LXC | - (connects out to Komodo) | internal | agent |
| Alloy | every LXC | - (pushes out) | internal | agent |
Expand Down
4 changes: 3 additions & 1 deletion docs/getting-started.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,9 @@ creating the platform and the environments, releasing, promoting.
pveum user token add urbex@pve cli --privsep 0 --comment "urbex" # prints the token once: keep it
```
The token is `urbex@pve!cli=<value>`, with the value the last command
prints.
prints. **Optional:** a second, read-only token puts the node's own
CPU, memory and storages in Grafana, with alerts
([Proxmox monitoring token](reference/credentials.md#proxmox-monitoring-token)).
- **Memory, disk and CPU**: about **6 GiB of free memory** (8 to be
comfortable) and **30 GiB of free disk** for the platform and a first
app in two environments. Measured on the test node (Proxmox VE 9.2,
Expand Down
2 changes: 1 addition & 1 deletion docs/known-limitations.md
Original file line number Diff line number Diff line change
Expand Up @@ -116,7 +116,7 @@ minor release may change how something works (see
| **The internal zone (`urbex.<domain>`) shadows Cloudflare on the LAN**: Technitium answers for every name under it, so a record you created there on Cloudflare doesn't resolve from the LXCs; with `nested` public hostnames, neither do the web frontends' (they have no LXC, so no internal record) ([ADR-0027](decisions/0027-service-conventions.md)). | Keep your own records out of the platform's subdomain, or set `dns.zone`. |
| **Without Cloudflare, the token issuer is Keycloak's LAN address** (`KEYCLOAK_ISSUER`): the address your users' browsers reach it at, which the tokens carry. The services fetch the signing keys by internal name. | Configure Cloudflare for a stable, public issuer. |
| **The internal names don't resolve on your machine** unless it asks Technitium: images (`git.<zone>:3000/...`), the links Forgejo and Komodo generate, and the URLs services are configured with use the internal names ([ADR-0027](decisions/0027-service-conventions.md)). The CLI uses the LAN addresses, after checking them against Technitium. | Have your router forward the zone to Technitium ([cookbook](cookbook.md#resolve-a-service-by-name-on-the-lan)). |
| **Logs and resource metrics only** ([ADR-0023](decisions/0023-service-logs-to-loki.md), [ADR-0027](decisions/0027-service-conventions.md)): the services' own metrics (`observability.metrics`) aren't scraped, per-container metrics are CPU and memory only, the only alerts are the reconciler's and those on the LXCs' disk and memory - not on the Proxmox node's own memory, CPU and storages, nor on a VM - and web frontends' runtime logs are on Cloudflare (their deploy container's are in Loki). | Cloudflare's Workers logs for frontends. |
| **Logs and resource metrics only** ([ADR-0023](decisions/0023-service-logs-to-loki.md), [ADR-0027](decisions/0027-service-conventions.md)): the services' own metrics (`observability.metrics`) aren't scraped, per-container metrics are CPU and memory only, the only alerts are the reconciler's and those on the LXCs' disk and memory - on the Proxmox node's memory and storages only with the optional [monitoring token](reference/credentials.md#proxmox-monitoring-token), never on a VM - and web frontends' runtime logs are on Cloudflare (their deploy container's are in Loki). | Cloudflare's Workers logs for frontends. |
| **Loki keeps logs forever**: default single-node configuration, no retention set; the observability LXC's disk fills up over time. | Grow its disk (`terraform/base`), or clean Loki's volume. |
| **Grafana's admin password** is the only login (no Keycloak SSO), after Cloudflare Access when reached from outside. | `admin` / `grafanaAdminPassword` from `~/.urbex/credentials.yaml`. |
| **Transactional email** (`email.provider: brevo`) is validated only. | - |
Expand Down
32 changes: 32 additions & 0 deletions docs/reference/credentials.md
Original file line number Diff line number Diff line change
Expand Up @@ -91,6 +91,37 @@ they do need the
- **Self-signed certificate:** set `proxmox.insecure: true` if the node
still uses Proxmox's default certificate.

### Proxmox monitoring token

Optional. A **read-only** token of the whole node, for the node's CPU,
memory and storages in Grafana (urbex's own token sees only the pool
`urbex`, and Proxmox answers 403 when it asks for the node).

- **Credentials file key / env var:** `proxmoxMonitorToken` /
`URBEX_PROXMOX_MONITOR_TOKEN`, as `user@realm!name=secret`.
- **Used by:** `urbex bootstrap`, which runs `pve-exporter` in the
observability Stack only when the token is there, with the token in
the Stack's encrypted secrets; `urbex doctor`, which checks it, or says
the node metrics are off without it.
- **Create it**, once, as root on the node:
```sh
pveum user add urbex-monitor@pve --comment "Urbex: read-only metrics"
pveum acl modify / --users urbex-monitor@pve --roles PVEAuditor
pveum user token add urbex-monitor@pve metrics --privsep 0 --comment "urbex metrics" # prints the token once
```
Then `urbex-op set proxmoxMonitorToken` with `urbex-monitor@pve!metrics=<value>`,
and `urbex bootstrap`.
- **What it can do:** `PVEAuditor` only reads (`Sys.Audit`,
`Datastore.Audit`, `VM.Audit`). It sees every guest of the node, those
outside the pool included; Urbex keeps only the node's and the
storages' series, never the guests'.
- **TLS:** `pve-exporter` reaches the node by its address under another
name, so it doesn't verify the node's certificate (Proxmox's own is
self-signed). The token only reads.
- **Turning it off:** remove the key and run `urbex bootstrap`; on the
observability LXC, `docker rm -f urbex-pve-exporter-1` stops an exporter
already running.

### SSH key

- **Where it's set:** `proxmox.sshPublicKey` in `urbex.platform.yaml`
Expand Down Expand Up @@ -351,4 +382,5 @@ item each is waiting on.
| age keypair | `URBEX_AGE_KEY` | ✅ yes (encrypts every secret) |
| Forgejo/Komodo/Keycloak/Grafana/Technitium passwords, tokens, keys | `URBEX_FORGE_*`, `URBEX_KOMODO_*`, `URBEX_TECHNITIUM_*`, ... | ✅ generated by `urbex bootstrap` |
| Cloudflare API token | `URBEX_CLOUDFLARE_TOKEN` | ✅ with Cloudflare configured |
| Proxmox monitoring token (read-only) | `URBEX_PROXMOX_MONITOR_TOKEN` | ❌ optional: the node's metrics in Grafana |
| Brevo API key | *(none defined)* | ❌ not yet |
2 changes: 1 addition & 1 deletion docs/status.md
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,7 @@ Proxmox VE 9.2 node, see [What has been validated](#what-has-been-validated).
| Frontend-only projects | A web frontend needs at least one service in the manifest | [0022](decisions/0022-cloudflare-tunnel-access-workers.md) |
| Keycloak beyond projects | No `platform` realm (admin SSO), no confidential clients for service-to-service calls | [0008](decisions/0008-keycloak-scope.md), [0024](decisions/0024-keycloak-realm-per-environment.md) |
| Terraform state encryption | State is plain JSON in the (private) GitOps repo, not SOPS-encrypted as designed | [0013](decisions/0013-terraform-state-in-gitops-repo.md) |
| Service metrics, alerting | LXC resource metrics are collected, but not the services' own metrics (`observability.metrics` is ignored); no alerts beyond the reconciler's and the LXCs' disk and memory (not the Proxmox node's, not a VM's); Loki has no retention | [0023](decisions/0023-service-logs-to-loki.md) |
| Service metrics, alerting | LXC resource metrics are collected, but not the services' own metrics (`observability.metrics` is ignored); no alerts beyond the reconciler's and the LXCs' disk and memory (the Proxmox node's memory and storages with the optional monitoring token; not a VM's); Loki has no retention | [0023](decisions/0023-service-logs-to-loki.md) |
| Mobile frontend deploy | `frontend.type: mobile` (Firebase) is validated, not deployed | - |
| Fleet-wide status | `urbex status` is scoped to one project+environment at a time; Komodo's UI is the cross-project view | - |
| Per-project Proxmox isolation | New projects should get their own resource pool, a group for their users with minimal permissions, and dedicated technical users; today every LXC goes in the platform-wide `proxmox.pool`, managed with the operator's token | [roadmap](roadmap.md) |
Expand Down