From 5d0af76009b532ee26c6f16824994e940ddd7781 Mon Sep 17 00:00:00 2001 From: Brian Miller Date: Sun, 13 Sep 2026 01:30:12 +0000 Subject: [PATCH] feat(nebula): CA rotation, the certificate audit, and pb-nebula v0.3's host flags pb-nebula v0.3.0 grew a rotation trigger and twelve fields; the platform put two routes in front of them. This catches the CLI up to both. STONE NEBULA CA-ROTATE. Three steps, because Nebula verification is mutual and hosts pull their config on their own schedule -- one write carrying both the new trust bundle and the new certificate splits the mesh until propagation finishes. The long help says so at each step rather than only at the parent, because the step someone is about to run is the one they are reading. It is a route because nebula_ca.updateRule is operator-only: permitting the trigger through an update rule means a deny-list over the certificate, the private key and the rest of the CA material, on the record holding the trust anchor for the whole mesh. Same reason as `stone nats account-keys`. STONE NEBULA CERT-AUDIT exists for a different reason -- deciding whether a host certificate still matches its network means parsing a Nebula certificate, which no client can do. The route answers in ids; this resolves them to hostnames, and falls back to the bare id rather than dropping a row, because a host we cannot name is still a host that needs re-signing. No bulk re-issue verb, deliberately. Re-signing moves a fingerprint and the revocation blocklist matches on fingerprints, so a sweep rewrites every peer config in the mesh. The help says to go one at a time. SEVEN NEW HOST FLAGS: --is-relay, --unsafe-networks, --unsafe-routes, --preferred-ranges, --mtu, --tun-device, --renew. is_relay joins the ls columns. ROUTEONLYFIELDS is the new half of the drift guard. The existing tests catch a flag for a field that does not exist -- which is how `nebula-ca --rotate-keys` was found. They cannot catch a flag for a field that DOES exist and that a tenant may not write: that one 404s at the rule layer, on a command that looks like every other update. `nebula_ca.rotate` is exactly that shape now, and the nats_accounts signing-key triggers have been in that position all along with only a prose note guarding them. Verified the guard fails when given a --rotate flag before removing the probe. libraryOwnedFields gains pb-nebula's additions. Worth noting it behaves better than the pb-nats case documented beside it: pb-nebula MIGRATES missing fields, so a database created before v0.3.0 acquires them on the next start rather than never. The floor is v0.3.0, and below it every flag here is a silent no-op -- which the vendored schema cannot catch, because it is a copy of a file that never declared these fields either. ALSO FIXED: every error the CLI printed ended in "(0)". PocketBase sends the status as `status`; PBError only read `code`, which nothing populates. The distinction that matters most -- 400 from a validator versus 404 from an update rule -- was invisible. Both are read now, with the HTTP status as a fallback. Verified against a live platform on pb-nebula v0.3.2: a full prepare -> commit -> finish through the CLI with the out-of-order refusals coming back intact, the audit flagging both hosts after a network widened and clearing per host on --renew, and the new flags round-tripping into the generated config. Co-Authored-By: Claude Opus 5 (1M context) --- .claude/skills/stone/SKILL.md | 20 ++- CHANGELOG.md | 62 +++++++++ CLAUDE.md | 46 ++++++- README.md | 55 +++++++- SKILLS.md | 25 +++- cmd/entity.go | 48 +++++-- cmd/nebula.go | 244 ++++++++++++++++++++++++++++++++++ cmd/schema_drift_test.go | 70 ++++++++++ internal/pb/client.go | 46 ++++++- 9 files changed, 596 insertions(+), 20 deletions(-) create mode 100644 cmd/nebula.go diff --git a/.claude/skills/stone/SKILL.md b/.claude/skills/stone/SKILL.md index ec6a76f..2e29a6e 100644 --- a/.claude/skills/stone/SKILL.md +++ b/.claude/skills/stone/SKILL.md @@ -67,7 +67,7 @@ Verbs `ls / get / create / update / delete / edit` are synthesized from a single | `invite` | yes | full | | `nats-user`, `nats-role`, `nats-import`, `nats-export` | yes | full | | `nebula-network`, `nebula-host` | yes | full | -| `nats-account`, `nebula-ca` | yes | `ls / get / update / edit` only | +| `nats-account`, `nebula-ca` | yes | `ls / get / update / edit` only (and every field is operator-only in practice — see Nebula overlay / `stone nats account-keys`) | | `membership` | no (org relation present, but not auto-filtered) | full | | `organization` | no (gated server-side by `is_operator`) | full | @@ -180,6 +180,20 @@ stone nats sync-context # re-issue per-org creds after key rotation `stone` does **not** manage JetStream consumers — use the `nats` CLI for that. +## Nebula overlay + +Records are ordinary entities. Two operations are routes, under `stone nebula`: + +```sh +stone nebula cert-audit # hosts whose certificate no longer matches their network +stone nebula ca-rotate prepare|commit|finish # roll the org's CA, one step at a time +``` + +- **Never run the three rotation steps back to back.** Nebula verification is mutual and hosts pull their config on their own schedule, so `commit` is only safe once *every* host has fetched what `prepare` published. Running them together is the single write the three-step design exists to avoid, and it splits the mesh until propagation catches up. `prepare` is reversible; `commit` re-signs every active host; `finish` drops the outgoing CA and is refused (by name) while any active host still holds one. +- **`cert-audit` finds hosts that look completely healthy and reach nothing** — active, in date, certificate present, config rendered. pb-nebula signed certificates at `/32` until v0.3.0, and Nebula builds the host's overlay route from the certificate's network. Fix one at a time with `stone nebula-host update --renew`, redeploying each config; never script a sweep, because re-signing moves a fingerprint and the revocation blocklist matches on fingerprints. +- **`--active=false` on a `nebula-host` revokes it across the whole CA**, not just its own network, and takes effect when each *peer's* config is redeployed. Deactivate to revoke — deleting the record leaves the certificate trusted until expiry. +- **`--is-relay` requires `--public-host-port`.** `--unsafe-networks` (on the gateway, signed into its certificate) and `--unsafe-routes` (on each consumer, `via` the gateway's overlay IP) are two halves on two different hosts; setting one alone moves no traffic. + ## Common failure modes and how to react - **`not authenticated. run: stone auth login`** — auth token is missing or expired. Surface to user; they run `stone auth login`. @@ -189,6 +203,8 @@ stone nats sync-context # re-issue per-org creds after key rotation - **`nats-sync: skipped — no membership found for this user+org`** — the authenticated user is an operator on an org they aren't a member of; NATS creds are per-membership. Not a bug. - **HTTP 400 from PocketBase on a relation field** — likely passed something that isn't a 15-char id. Re-look it up with `get --fields id -o json` or `ls -o json`. - **`multiple match "..."`** — the natural key is ambiguous in this org. Use one of the 15-char ids listed in the error. +- **`Invalid CA rotation: ...`** — the rotation steps were run out of order, or `finish` was attempted while a host named in the message still holds an outgoing-CA certificate. Read the message; it says which. There is no force flag. +- **HTTP 404 updating `nebula-ca` or `nats-account`** — both are operator-only for every field. The tenant operations are `stone nebula ca-rotate` and `stone nats account-keys`. ## Things not to do @@ -197,3 +213,5 @@ stone nats sync-context # re-issue per-org creds after key rotation - Don't run `stone apply` against a workspace you didn't `pull` from or hand-author with knowledge of the schema — apply will dutifully create records. - Don't expect `apply` to delete things. It only creates and updates. - Don't bypass `stone org switch` by editing `context.yaml` directly — you'll skip the per-org NATS creds sync. +- Don't script a `--renew` sweep across every host `cert-audit` reports. Each re-issue moves a fingerprint and rewrites every peer's config; do them one at a time, redeploying as you go. +- Don't look for a `--rotate` flag on `nebula-ca`. The field exists but the collection's update rule is operator-only, so a tenant PATCH 404s; `stone nebula ca-rotate` is the way in. diff --git a/CHANGELOG.md b/CHANGELOG.md index 95bd077..998a5ac 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -27,6 +27,62 @@ that period, and this file starts where the versioned releases do. ### Added +- **`stone nebula`**, for the two overlay operations that are not record writes. + + `stone nebula ca-rotate prepare|commit|finish` rolls the organization's Nebula + CA. It is a route (`POST /api/org/nebula-ca/rotate`) because + `nebula_ca.updateRule` is operator-only — permitting the trigger through an + update rule would mean a deny-list over the certificate, the private key and + the rest of the CA material, on the record holding the trust anchor for the + whole mesh. + + Three steps, and the wait between them is the feature. Nebula verification is + mutual — each peer checks the other against its *own* local CA pool, with no + chain and no fallback — and hosts pull their config on their own schedule, so + one write carrying both the new trust bundle and the new certificate splits + the mesh until propagation finishes. `prepare` publishes trust and moves no + issuance, so it is reversible; `commit` switches issuance and re-signs every + active host; `finish` drops the outgoing CA and is refused while any active + host still holds one, naming the host. + + `stone nebula cert-audit` lists active hosts whose certificate no longer + matches their network. pb-nebula signed host certificates at `/32` until + v0.3.0, and Nebula builds a host's overlay route from the network in its + certificate — so such a host reaches no peer while looking entirely healthy: + active, in date, certificate present, config rendered, nothing logged. An + edited `overlay_ip` lands a host here too. It is a route + (`GET /api/org/nebula/cert-audit`) because answering it means parsing a Nebula + certificate, which no client can do. + + Fix one host at a time with `stone nebula-host update --renew`, + redeploying each config as you go. There is deliberately no bulk verb: + re-signing moves a certificate's fingerprint, and a fingerprint is what the + revocation blocklist matches, so a sweep rewrites every peer config in the + mesh. + +- **Seven new `nebula-host` flags**, from pb-nebula v0.3.0: `--is-relay`, + `--unsafe-networks`, `--unsafe-routes`, `--preferred-ranges`, `--mtu`, + `--tun-device` and `--renew`. `is_relay` joins the `ls` columns, which now + badge lighthouse and relay separately — a host can be both. + + `--unsafe-networks` and `--unsafe-routes` are two halves of the same feature + living on *different* hosts: the first is signed into the gateway's + certificate and authorizes it to route that subnet, the second goes on each + host that wants to reach it. Neither derives the other. + + **These require a platform on pb-nebula v0.3.0 or newer** (the platform pins + v0.3.2). Against v0.2.0 they name fields the collection does not have, so + PocketBase discards the write and the command reports success — the same + failure the `message-schema` removal above describes. + +- **`routeOnlyFields` in the schema drift guard.** The existing tests catch a + flag for a field that does not exist. They cannot catch a flag for a field + that *does* exist and that a tenant is not allowed to write — the write simply + 404s at the rule layer, on a command that looks like every other update. That + is the shape `nebula_ca.rotate` would take if anyone added a `--rotate` flag, + and the `nats_accounts` signing-key triggers have been in that position all + along with only a prose note guarding them. + - **`--code` on `organization`,** and `code` in its `ls` columns. The one globally unique identifier in the ecosystem: derived from the name when omitted, immutable once set, and baked into signed NATS account JWTs and @@ -36,6 +92,12 @@ that period, and this file starts where the versioned releases do. ### Fixed +- **Every error the CLI printed ended in `(0)`.** PocketBase sends the status as + `status`; `PBError` only read `code`, which nothing populates. So the number + in every error message was zero, and the distinction that matters most — + a 400 from a validator versus a 404 from an update rule — was invisible. Both + names are read now, with the HTTP status as a fallback. + - **Refreshed the vendored platform schema** (`cmd/testdata/schema.json`), which is what the drift tests check the field table against. It was three platform changes behind. Worth recording that the guard did its job unprompted: told diff --git a/CLAUDE.md b/CLAUDE.md index bbc6365..68fc668 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -73,6 +73,13 @@ intentional — do not "fix" them: - **`nebula-ca.rotate_keys`** — no such field exists, in `schema.json` or in pb-nebula. It was a flag once and did nothing: PocketBase silently drops writes to fields a collection doesn't have, so it reported success every time. +- **`nebula-ca.rotate`** — this one is real (pb-nebula v0.3.0), and is still not + a flag: `nebula_ca.updateRule` is operator-only, so a tenant PATCH 404s. The + three steps live behind `stone nebula ca-rotate`. Same for the material a + rotation produces — `next_certificate`, `previous_certificate`, `rotated_at`. + `routeOnlyFields` in `cmd/schema_drift_test.go` is what stops any of them + becoming a flag later; it covers the `nats_accounts` triggers too, which were + previously only prose. - **`nats-account.rotate_keys` / `add_signing_key` / `remove_signing_key`** — the fields exist, but `nats_accounts.updateRule` is operator-only, so a tenant PATCH 404s. They live behind `stone nats account-keys` instead. @@ -89,13 +96,42 @@ platform repo is normally at `../platform`. ### Custom platform routes -`internal/pb/client.go` has `CallRoute` for the platform's non-collection -endpoints, and `cmd/creds.go` wraps the two that exist: -`POST /api/me/nats-creds/rotate` and `POST /api/org/nats-account/keys`. Both -exist server-side because an API rule cannot express a single-field allowlist. -Neither takes a record id — the target is derived from the caller's identity or +`internal/pb/client.go` has `CallRoute` (POST) and `GetRoute` (GET) for the +platform's non-collection endpoints. Four are wrapped: + +- `cmd/creds.go` — `POST /api/me/nats-creds/rotate` and + `POST /api/org/nats-account/keys`. +- `cmd/nebula.go` — `POST /api/org/nebula-ca/rotate` and + `GET /api/org/nebula/cert-audit`. + +Three of the four exist because an API rule cannot express a single-field +allowlist: permitting one trigger field through an update rule means a deny-list +over every other, which silently opens the moment someone adds a field. None of +them takes a record id — the target is derived from the caller's identity or active organization — so don't add one. +`cert-audit` is the exception and exists for a different reason: deciding whether +a host certificate still matches its network means parsing a Nebula certificate, +which no client can do. It answers in **ids**, and `cmd/nebula.go` resolves them +to hostnames before printing — falling back to the bare id rather than dropping +the row, because a host we cannot name is still a host that needs re-signing. + +### pb-nebula v0.3 and the library-owned fields + +pb-nebula v0.3.0 added twelve fields the platform's `schema.json` does not +declare — the dump predates the release. Unlike the pb-nats case noted in +`cmd/schema_drift_test.go`, pb-nebula **migrates** them (`addMissingFields`), so +a database created earlier acquires them on the next start rather than never. +They are listed in `libraryOwnedFields` so the drift guard can see them. + +**The floor is pb-nebula v0.3.0** (the platform pins v0.3.2). Against v0.2.0 the +two Nebula routes 404, and `--is-relay`, `--unsafe-networks`, `--unsafe-routes`, +`--preferred-ranges`, `--mtu`, `--tun-device` and `--renew` all name fields the +collection does not have — so the write is discarded and the command reports +success. That is precisely the failure the drift guard exists to make loud, and +it cannot catch this one, because the vendored schema is a copy of a file that +never declared them either. + ### Pull / apply (GitOps) `cmd/sync.go`: - `stone pull` writes one YAML file per record into `//.yaml`, where `` is the spec's `LookupKey` value, falling back to `name`, then id. Filename collisions get a `-` suffix; records are pulled sorted by id so the suffix lands on the same record across pulls. Filenames are cosmetic — apply identifies records solely by the `id` field inside the file. Org-scoped collections are filtered by `current_organization`. Server-only fields (`collectionId`, `collectionName`, `created`, `updated`, `expand`) are stripped on read (see `pb.ServerOnlyFields` / `pb.Strip`). diff --git a/README.md b/README.md index cb1712e..cd8266f 100644 --- a/README.md +++ b/README.md @@ -322,6 +322,56 @@ All KV operations — bucket lifecycle and per-key data — live under `stone kv ./stone kv watch twins ``` +## Nebula + +The records are ordinary entities — `stone nebula-ca`, `stone nebula-network`, +`stone nebula-host`. Two operations are not record writes, and live under +`stone nebula`: + +```sh +stone nebula cert-audit # hosts whose certificate no longer matches their network +stone nebula ca-rotate prepare # publish trust in a new CA (reversible) +stone nebula ca-rotate commit # switch issuance, re-sign every active host +stone nebula ca-rotate finish # drop the outgoing CA +``` + +**`ca-rotate` takes three steps and the wait between them is the point.** Nebula +verification is mutual — each peer checks the other against its *own* local CA +pool, with no chain and no fallback — and hosts pull their config whenever they +like. So one write carrying both the new trust bundle and the new certificate +splits the mesh: a host that has fetched presents a new-CA certificate to one +that has not, and the handshake fails in *both* directions until propagation +finishes. `prepare` publishes trust and moves no issuance, so it is fully +reversible. `commit` switches issuance and re-signs every active host, with both +CAs trusted throughout. `finish` drops the outgoing CA and is refused while any +active host still holds a certificate signed by it — the refusal names the host. + +A CA cannot be renewed, only rotated, so start months ahead of the expiry in +`stone nebula-ca ls`, not weeks. + +**`cert-audit` answers a question no client can.** pb-nebula signed host +certificates at `/32` until v0.3.0. Nebula puts a certificate's network straight +onto the tun device and installs a link route for it, so the mask in the +certificate *is* the host's route to the overlay — a `/32` gives a host a route +covering only itself. The certificate verifies, the config renders, the host +starts, the handshake completes, and no packet crosses the mesh. Nothing errors, +which is why you have to ask. Editing a host's `overlay_ip` after issue lands it +here too. + +Nothing is re-signed automatically: re-signing moves a fingerprint, and a +fingerprint is what the revocation blocklist matches, so a sweep would rewrite +every peer config in the mesh. Fix one host at a time, redeploying as you go: + +```sh +stone nebula-host update edge-west --renew +``` + +Requires a platform on **pb-nebula v0.3.0 or newer**. Against v0.2.0 the routes +404 and the newer host flags — `--is-relay`, `--unsafe-networks`, +`--unsafe-routes`, `--preferred-ranges`, `--mtu`, `--tun-device`, `--renew` — +name fields the collection does not have, so PocketBase discards the write and +the command reports success. + ## Limitations - Relation flags (`--type`, `--location`, …) take 15-char PocketBase ids only — @@ -332,9 +382,8 @@ All KV operations — bucket lifecycle and per-key data — live under `stone kv - No JetStream consumer management — the `nats` CLI is better at that. - `nats-account` and `nebula-ca` are **operator-only for every field**. Both `updateRule`s admit no tenant role, so an owner/admin PATCH of any field on - either collection returns 404. The three legitimate tenant key operations live - behind `stone nats account-keys` instead. `nebula_ca` has no rotation trigger - at all — rolling a CA is an operator action. + either collection returns 404. The legitimate tenant operations live behind + routes instead: `stone nats account-keys` and `stone nebula ca-rotate`. - Locations' `floorplan` and organizations' `logo` are file fields; the CLI has no upload path for them. Use the console. diff --git a/SKILLS.md b/SKILLS.md index 3fa3a4c..f19428e 100644 --- a/SKILLS.md +++ b/SKILLS.md @@ -194,12 +194,35 @@ stone nats account-keys remove-signing # last remaining key cannot be stone nats account-keys rotate # EMERGENCY: invalidates every user JWT in the account ``` +## Nebula overlay — two operations that are not record writes + +The records are ordinary entities (`nebula-ca`, `nebula-network`, `nebula-host`). +These two live under `stone nebula` because a PocketBase rule cannot express them. + +```sh +stone nebula cert-audit # active hosts whose certificate no longer matches their network +stone nebula ca-rotate prepare|commit|finish +``` + +Rules a caller must not get wrong: + +- **`ca-rotate` is three steps with a wait between them, and the wait is the feature.** Nebula verification is mutual — each peer checks the other against its *own* local CA pool, with no chain and no fallback — and hosts pull their config on their own schedule. `prepare` publishes trust in the incoming CA and moves no issuance, so it is fully reversible. Only once **every** host has fetched its config does `commit` become safe; it switches issuance and re-signs every active host, with both CAs trusted throughout. `finish` drops the outgoing CA. Never run the three back to back — that is the single write the three-step design exists to avoid, and it splits the mesh for as long as propagation takes. +- **`finish` is interlocked and the refusal is informative.** It is refused while any active host still holds a certificate from the outgoing CA, and the error names the host. Deploy that host's config and retry; do not look for a force flag, there isn't one. +- **A CA cannot be renewed, only rotated.** Check `stone nebula-ca ls` for the expiry and start months ahead, not weeks — the wait in the middle cannot be compressed. +- **`cert-audit` reports a failure that is invisible everywhere else.** pb-nebula signed host certificates at `/32` until v0.3.0; Nebula builds the host's overlay route from the certificate's network, so such a host reaches no peer while looking entirely healthy — active, in date, certificate present, config rendered, no errors. Editing a host's `overlay_ip` after issue has the same effect. +- **Fix stale hosts one at a time**, with `stone nebula-host update --renew`, redeploying each config as you go. Do not script a sweep: re-signing moves a certificate's fingerprint, and a fingerprint is what the revocation blocklist matches, so a bulk re-issue rewrites every peer config in the mesh. +- **`--active=false` on a `nebula-host` is revocation**, and it now reaches every network under the same CA rather than just the host's own — Nebula's trust boundary is the CA. It takes effect when each *peer's* config is redeployed, not instantly. Deactivate to revoke; deleting the record leaves the certificate trusted until it expires, because a fingerprint that is not in the database cannot be blocklisted. +- **`--is-relay` needs `--public-host-port`.** Without it the host listens on an ephemeral port while every peer is handed its overlay IP as a usable path. +- **`--unsafe-networks` and `--unsafe-routes` are two halves on two different hosts.** The first is signed into the *gateway's* certificate and authorizes it to route that subnet; the second goes on every host that wants to reach it, with `via` set to the gateway's overlay IP. Neither derives the other, and setting only one moves no traffic. + +Requires a platform on **pb-nebula v0.3.0+**. Against v0.2.0 the routes 404 and the newer host flags name fields the collection lacks, so the write is discarded and the command still reports success. + ## Known limitations - Relation flags do not resolve names — pass 15-char PocketBase ids only. (Positional record args on `get`/`update`/`delete`/`edit` *do* accept natural keys.) - `apply` does not delete server records absent from the workspace. - No JetStream **consumer** management (use `nats` CLI). -- `nats-account` and `nebula-ca` are **operator-only for every field** — both `updateRule`s admit no tenant role, so an owner/admin PATCH of any field on either returns 404. Tenant key operations live at `stone nats account-keys`. `nebula_ca` has no rotation trigger at all. +- `nats-account` and `nebula-ca` are **operator-only for every field** — both `updateRule`s admit no tenant role, so an owner/admin PATCH of any field on either returns 404. The tenant operations live behind routes: `stone nats account-keys` and `stone nebula ca-rotate`. - File fields have no CLI upload path: `locations.floorplan`, `organizations.logo`. Use the console. ## Configuration files diff --git a/cmd/entity.go b/cmd/entity.go index 76ee6f0..06e069d 100644 --- a/cmd/entity.go +++ b/cmd/entity.go @@ -421,7 +421,7 @@ var entitySpecs = []EntitySpec{ Plural: "nebula-hosts", Collection: "nebula_hosts", OrgScoped: true, - KeyColumns: []string{"hostname", "overlay_ip", "network_id", "is_lighthouse", "active"}, + KeyColumns: []string{"hostname", "overlay_ip", "network_id", "is_lighthouse", "is_relay", "active"}, LookupKey: "hostname", Fields: []Field{ {Name: "email", Type: FString, Required: true, Help: "auth email (required by the auth collection)"}, @@ -431,11 +431,34 @@ var entitySpecs = []EntitySpec{ {Name: "network_id", Type: FID, Required: true, Help: "nebula_networks id"}, {Name: "groups", Type: FJSON, Help: "Nebula firewall groups (JSON array)"}, {Name: "is_lighthouse", Type: FBool, Help: "whether this host is a lighthouse"}, - {Name: "public_host_port", Type: FString, Help: "public address:port (for lighthouses / static peers)"}, + {Name: "is_relay", Type: FBool, Help: "forward traffic for peers that cannot reach each other directly (needs --public-host-port)"}, + {Name: "public_host_port", Type: FString, Help: "public address:port; required for a lighthouse AND for a relay"}, {Name: "firewall_outbound", Type: FJSON, Help: "outbound firewall rules (JSON)"}, {Name: "firewall_inbound", Type: FJSON, Help: "inbound firewall rules (JSON)"}, + + // Gateway routing: two halves on two DIFFERENT hosts, and neither + // derives the other. unsafe_networks is signed INTO the gateway's + // certificate -- Nebula authorizes routing on the certificate, not + // on config -- so setting it re-issues, and the change is inert + // until that host picks the new certificate up. + {Name: "unsafe_networks", Type: FJSON, Help: `subnets THIS host routes to, e.g. ["192.168.1.0/24"] (signed into the certificate)`}, + {Name: "unsafe_routes", Type: FJSON, Help: `subnets reached THROUGH another host, e.g. [{"route":"192.168.1.0/24","via":"10.0.0.7"}]`}, + + // Underlay, not overlay -- which is why IPv6 is accepted here and + // nowhere else in the platform. + {Name: "preferred_ranges", Type: FJSON, Help: `underlay prefixes to favour when a peer has several addresses, e.g. ["192.168.1.0/24"]`}, + + {Name: "mtu", Type: FInt, Help: "MTU override; 0 inherits the default"}, + {Name: "tun_device", Type: FString, Help: "tun interface name override; empty inherits the default"}, + {Name: "validity_years", Type: FInt, Help: "certificate validity in years"}, - {Name: "active", Type: FBool, Help: "whether the host is active"}, + {Name: "active", Type: FBool, Help: "whether the host is active; clearing it BLOCKLISTS the certificate across the CA"}, + + // An action field, not state: pb-nebula re-issues on the false -> + // true edge and resets it in the same save, so it never reads back. + // Exposed because it is the fix for a host whose certificate no + // longer matches its network -- see `stone nebula cert-audit`. + {Name: "renew", Type: FBool, Help: "re-issue this host's certificate now (resets itself; redeploy the config afterwards)"}, }, }, { @@ -450,11 +473,20 @@ var entitySpecs = []EntitySpec{ {Name: "name", Type: FString, Help: "CA name"}, {Name: "validity_years", Type: FInt, Help: "CA cert validity in years (operator-only)"}, {Name: "curve", Type: FString, Help: "elliptic curve, e.g. P256 (operator-only)"}, - // There is no `rotate_keys` here on purpose: nebula_ca has no such - // field, in schema.json or in pb-nebula. The flag used to exist and - // did nothing at all -- PocketBase silently drops writes to fields a - // collection does not have, so it reported success every time. - // Rolling a CA is a platform-operator operation, not a tenant one. + + // There is still no `rotate_keys` here, and there never was a field + // by that name -- the flag that used to exist did nothing at all, + // because PocketBase silently drops writes to fields a collection + // does not have and so reported success every time. + // + // pb-nebula v0.3.0 DID add a real rotation trigger, `rotate`. It is + // deliberately not a flag either, for the reason the nats-account + // key triggers are not: nebula_ca.updateRule is operator-only, so a + // tenant PATCH 404s, and the three steps live behind + // `stone nebula ca-rotate` instead. The same goes for the material + // rotation produces -- next_certificate, previous_certificate, + // rotated_at -- which is server-generated and readable via `get`. + // routeOnlyFields in schema_drift_test.go is what holds that line. }, }, diff --git a/cmd/nebula.go b/cmd/nebula.go new file mode 100644 index 0000000..452b084 --- /dev/null +++ b/cmd/nebula.go @@ -0,0 +1,244 @@ +package cmd + +import ( + "fmt" + "os" + "sort" + + "github.com/spf13/cobra" + "github.com/stone-age-io/stone-cli/internal/ctx" + "github.com/stone-age-io/stone-cli/internal/pb" +) + +// The two Nebula operations that are routes rather than record writes, for the +// same two reasons the NATS ones in creds.go are. +// +// ca-rotate exists because nebula_ca.updateRule is operator-only. Permitting the +// rotation trigger through an update rule would mean a deny-list over the +// certificate, the private key, the incoming and outgoing CA material, the +// expiry and the curve -- and that deny-list would silently open up the moment +// anyone added a field, on the record holding the trust anchor for the whole +// tenant's mesh. +// +// cert-audit exists for a different reason: answering it means parsing a Nebula +// certificate to compare the network it carries against the network the host +// belongs to. No client can do that, so the platform does it and hands back the +// verdict. + +var nebulaCmd = &cobra.Command{ + Use: "nebula", + Short: "Nebula overlay operations that are not record writes", + Long: `Nebula operations the collection API cannot express. + +The records themselves are managed with the entity commands -- nebula-ca, +nebula-network and nebula-host. What lives here is the CA rotation lever, which +nebula_ca.updateRule deliberately does not expose, and the host certificate +audit, which needs a certificate parsed to answer.`, +} + +var nebulaCARotateCmd = &cobra.Command{ + Use: "ca-rotate", + Short: "Rotate your organization's Nebula CA, in three steps", + Long: `Roll the Nebula CA your organization's host certificates chain to. + +Three steps, and the wait between them is the feature. + +Nebula verification is mutual -- each peer checks the other against its OWN local +CA pool, with no chain and no fallback -- and hosts pull their config whenever +they like, with nothing telling the platform when they did. So one write carrying +both the new trust bundle and the new certificate splits the mesh: a host that +has fetched presents a new-CA certificate to one that has not, and the handshake +fails in BOTH directions until propagation finishes. + + prepare mint the incoming CA and publish it as additional trust. + Issuance does not move, no host certificate changes and no + fingerprint moves, so this is fully reversible. + + -- wait here until every host has fetched its config -- + + commit switch issuance to the new CA and re-sign every ACTIVE host. + Both CAs stay trusted, so re-signed and not-yet-re-signed hosts + still talk to each other. Idempotent: re-running it re-signs only + the hosts still on the outgoing CA, which is how you recover from a + partial sweep. + + -- deploy the new config to every active host -- + + finish drop the outgoing CA. Refused while any active host still holds a + certificate signed by it; the error names the host. + +How long to wait in the middle is your judgement and nothing can compress it. +A CA cannot be renewed, only rotated, so start months before expiry rather than +weeks -- see the expiry on: stone nebula-ca ls + +Owner or admin of the active organization. Takes no id: the CA is derived from +your active organization, so this cannot be aimed at another tenant.`, +} + +var nebulaCARotatePrepareCmd = &cobra.Command{ + Use: "prepare", + Short: "Mint the incoming CA and publish trust in it (reversible)", + Args: cobra.NoArgs, + RunE: func(cmd *cobra.Command, args []string) error { return rotateCAStep("prepare") }, +} + +var nebulaCARotateCommitCmd = &cobra.Command{ + Use: "commit", + Short: "Switch issuance to the new CA and re-sign every active host", + Long: `Switch issuance to the prepared CA and re-sign every active host. + +Run this only once every host has fetched the config prepare published -- until +then, some peers do not yet trust the CA the re-signed hosts now present. + +Inactive hosts are deliberately NOT re-signed: an inactive host is revoked, its +fingerprint sits in every peer's blocklist, and re-signing it would publish a new +fingerprint while the old certificate stayed valid.`, + Args: cobra.NoArgs, + RunE: func(cmd *cobra.Command, args []string) error { return rotateCAStep("commit") }, +} + +var nebulaCARotateFinishCmd = &cobra.Command{ + Use: "finish", + Short: "Drop the outgoing CA (refused while any active host still uses it)", + Long: `Drop the outgoing CA from every host's trust bundle. + +Refused while any active host still holds a certificate signed by it, and the +refusal names the host. That interlock is what makes this step safe to expose: +dropping the outgoing CA early does not fail loudly, it just takes that host off +the mesh -- it keeps running, and its peers quietly stop being able to verify it. + +If it refuses, deploy the config to the host it names and try again.`, + Args: cobra.NoArgs, + RunE: func(cmd *cobra.Command, args []string) error { return rotateCAStep("finish") }, +} + +// rotateCAStep posts one step to the rotation route. The CA is derived +// server-side from the caller's active organization, so there is no id to pass. +func rotateCAStep(step string) error { + c, err := ctx.Active(flagContext) + if err != nil { + return err + } + client := newPBClient(c) + + var out struct { + Applied string `json:"applied"` + NebulaCA string `json:"nebula_ca"` + } + // The server refuses an out-of-order step and names the host blocking a + // finish. Those messages are the whole point of the interlock, so they are + // returned as-is rather than replaced with something tidier. + if err := client.CallRoute("/api/org/nebula-ca/rotate", map[string]any{"step": step}, &out); err != nil { + return err + } + + fmt.Printf("%s: applied to CA %s\n", out.Applied, out.NebulaCA) + switch step { + case "prepare": + fmt.Println("every host now trusts the incoming CA. Let them all fetch their config, then: stone nebula ca-rotate commit") + case "commit": + fmt.Println("active hosts have been re-signed. Deploy their configs, then: stone nebula ca-rotate finish") + case "finish": + fmt.Println("the outgoing CA has been dropped. Rotation complete.") + } + return nil +} + +var nebulaCertAuditCmd = &cobra.Command{ + Use: "cert-audit", + Short: "List hosts whose certificate no longer matches their network", + Long: `Report active hosts whose certificate carries the wrong overlay network. + +pb-nebula signed host certificates at /32 until v0.3.0. Nebula does not read a +certificate's network as "this host's address" -- it puts the prefix straight +onto the tun device and installs a link route for it, so the mask in the +certificate IS the host's route to the overlay. A /32 gives a host a route +covering only itself: the certificate verifies, the config renders, the host +starts, the handshake completes, and no packet ever crosses the mesh. + +Nothing errors anywhere, which is why this needs asking for rather than waiting +to be told. A host's overlay_ip being edited after issue lands it here too. + +Nothing is re-signed automatically, and that restraint is deliberate: re-signing +moves a certificate's fingerprint, and a fingerprint is what the revocation +blocklist matches, so a sweep would rewrite every peer config in the mesh. Fix +one host at a time, redeploying each config as you go: + + stone nebula-host update --renew + +Inactive hosts are excluded. They are revoked, and re-signing one would publish a +new fingerprint while the old certificate stayed valid and unblocklisted. + +Owner or admin of the active organization.`, + Args: cobra.NoArgs, + RunE: func(cmd *cobra.Command, args []string) error { + c, err := ctx.Active(flagContext) + if err != nil { + return err + } + client := newPBClient(c) + + var out struct { + Stale []string `json:"stale"` + } + if err := client.GetRoute("/api/org/nebula/cert-audit", &out); err != nil { + return err + } + + if len(out.Stale) == 0 { + fmt.Println("no host certificate is out of step with its network") + return nil + } + + // The route answers in ids, because that is what it can be sure of. An + // id is not what anyone types, so resolve to the hostname and network + // the operator will actually act on -- and fall back to the bare id + // rather than dropping a row if a lookup fails, since a host we cannot + // name is still a host that needs re-signing. + stale := map[string]bool{} + for _, id := range out.Stale { + stale[id] = true + } + + hosts, err := client.ListAll("nebula_hosts", pb.ListOptions{ + Fields: "id,hostname,overlay_ip,network_id", + Sort: "hostname", + }) + if err != nil { + hosts = nil + } + + named := map[string]pb.Record{} + for _, h := range hosts { + if id, _ := h["id"].(string); stale[id] { + named[id] = h + } + } + + rows := make([]pb.Record, 0, len(out.Stale)) + for _, id := range out.Stale { + if h, ok := named[id]; ok { + rows = append(rows, h) + continue + } + rows = append(rows, pb.Record{"id": id, "hostname": "(not readable)"}) + } + sort.Slice(rows, func(i, j int) bool { + a, _ := rows[i]["hostname"].(string) + b, _ := rows[j]["hostname"].(string) + return a < b + }) + + if err := pb.PrintList(os.Stdout, rows, []string{"hostname", "overlay_ip", "network_id"}, resolveOutput()); err != nil { + return err + } + fmt.Fprintf(os.Stderr, "\n%d host(s) need re-issuing: stone nebula-host update --renew\n", len(rows)) + return nil + }, +} + +func init() { + nebulaCARotateCmd.AddCommand(nebulaCARotatePrepareCmd, nebulaCARotateCommitCmd, nebulaCARotateFinishCmd) + nebulaCmd.AddCommand(nebulaCARotateCmd, nebulaCertAuditCmd) + rootCmd.AddCommand(nebulaCmd) +} diff --git a/cmd/schema_drift_test.go b/cmd/schema_drift_test.go index 22f356a..959852a 100644 --- a/cmd/schema_drift_test.go +++ b/cmd/schema_drift_test.go @@ -100,8 +100,78 @@ var deliberatelyOmitted = map[string][]string{ // either. This is the same shape as the system_account_id bug the platform fixed // in v0.2.0. Flagged upstream; the CLI's flags are correct against a current // database. +// +// pb-nebula v0.3.0 is the other case, and it behaves BETTER than the one above. +// It added twelve fields across nebula_hosts and nebula_ca, and unlike pb-nats +// it migrates them: InitializeCollections adds any declared field an existing +// collection is missing (addMissingFields), so a database created before v0.3.0 +// acquires them on the next start rather than never. The platform's schema.json +// still does not declare them -- it is a dump that predates the release, and +// hand-copying a library's field definitions into it is the drift risk the +// platform's own CLAUDE.md warns about -- so they are listed here instead. +// +// The floor is pb-nebula v0.3.0 (the platform pins v0.3.2). Against v0.2.0 every +// flag below is a silent no-op, which is the failure this whole file exists to +// make loud. var libraryOwnedFields = map[string][]string{ "nats_roles": {"allow_response", "allow_response_max", "allow_response_ttl"}, + + // Exposed as flags: the relay role, the two halves of gateway routing, + // underlay path preference, the transport overrides, and the renew trigger. + "nebula_hosts": { + "is_relay", + "unsafe_networks", "unsafe_routes", + "preferred_ranges", + "mtu", "tun_device", + "renew", + }, +} + +// routeOnlyFields are fields that DO exist on a live collection and must never +// become CLI flags, because the collection's updateRule does not let a tenant +// write them -- the operation lives behind a platform route instead. +// +// This is a different failure from the one the two tests above catch, and +// neither of them would see it. A flag here would name a real field, so the +// "collection has no such field" check passes; the write simply 404s at the rule +// layer, on a command that looks exactly like every other update. +// +// It is also the hole that let `nebula-ca --rotate-keys` exist: that flag named +// a field that never existed anywhere, but the same flag naming a field that +// exists-and-is-forbidden would have been just as broken and even quieter. +var routeOnlyFields = map[string]map[string]string{ + "nebula_ca": { + "rotate": "nebula_ca.updateRule is operator-only; use `stone nebula ca-rotate`", + // The material a rotation produces. Server-generated, readable via + // `get`, never typed in. + "next_certificate": "server-generated during rotation", + "next_private_key": "server-generated during rotation, and hidden from the API", + "previous_certificate": "server-generated during rotation", + "rotated_at": "server-generated during rotation", + }, + "nats_accounts": { + "rotate_keys": "nats_accounts.updateRule is operator-only; use `stone nats account-keys rotate`", + "add_signing_key": "operator-only rule; use `stone nats account-keys add-signing`", + "remove_signing_key": "operator-only rule; use `stone nats account-keys remove-signing`", + }, +} + +func TestRouteOnlyFieldsAreNotFlags(t *testing.T) { + for _, spec := range entitySpecs { + forbidden, ok := routeOnlyFields[spec.Collection] + if !ok { + continue + } + t.Run(spec.Name, func(t *testing.T) { + for _, f := range spec.Fields { + if why, bad := forbidden[f.Name]; bad { + t.Errorf("field %q has a --%s flag, but a tenant cannot write it: %s.\n"+ + "The command would look like every other update and 404 at the rule layer.", + f.Name, f.flagName(), why) + } + } + }) + } } func loadVendoredSchema(t *testing.T) map[string]schemaCollection { diff --git a/internal/pb/client.go b/internal/pb/client.go index a9326ed..3d171f8 100644 --- a/internal/pb/client.go +++ b/internal/pb/client.go @@ -277,6 +277,29 @@ func (c *Client) CallRoute(path string, body any, out any) error { return json.NewDecoder(resp.Body).Decode(out) } +// GetRoute is CallRoute for the platform's read-only routes. +// +// Separate rather than a method parameter on CallRoute: every caller of that one +// is performing an action, and its doc comment is about why those actions are +// routes at all. A read is there for a different reason -- the platform's +// certificate audit has to parse a Nebula certificate to answer, which no +// client can do -- and collapsing the two would put an unused body argument in +// front of every read. +func (c *Client) GetRoute(path string, out any) error { + resp, err := c.do(http.MethodGet, path, nil, true) + if err != nil { + return err + } + defer resp.Body.Close() + if err := checkOK(resp); err != nil { + return err + } + if out == nil { + return nil + } + return json.NewDecoder(resp.Body).Decode(out) +} + // BatchOp is one operation in a /api/batch request. type BatchOp struct { Method string `json:"method"` // "POST" | "PATCH" | "DELETE" @@ -406,17 +429,33 @@ func (c *Client) do(method, path string, body io.Reader, needAuth bool) (*http.R } // PBError is the standard error shape PocketBase returns. +// +// The status arrives as `status`, not `code` -- PocketBase's ApiError marshals +// it under that name, and has for every version this CLI has talked to. Reading +// only `code` meant the number in every error message the CLI has ever printed +// was 0, including on the ones where it matters most: a 404 from an update rule +// and a 400 from a validator read identically. Both names are accepted now, and +// checkOK fills in the HTTP status if a server sends neither. type PBError struct { Code int `json:"code"` + Status int `json:"status"` Message string `json:"message"` Data map[string]any `json:"data"` } +// status is whichever the server actually sent. +func (e *PBError) status() int { + if e.Status != 0 { + return e.Status + } + return e.Code +} + func (e *PBError) Error() string { if len(e.Data) > 0 { - return fmt.Sprintf("%s (%d): %s", e.Message, e.Code, formatPBData(e.Data)) + return fmt.Sprintf("%s (%d): %s", e.Message, e.status(), formatPBData(e.Data)) } - return fmt.Sprintf("%s (%d)", e.Message, e.Code) + return fmt.Sprintf("%s (%d)", e.Message, e.status()) } func formatPBData(d map[string]any) string { @@ -434,6 +473,9 @@ func checkOK(resp *http.Response) error { b, _ := io.ReadAll(resp.Body) var pe PBError if err := json.Unmarshal(b, &pe); err == nil && pe.Message != "" { + if pe.status() == 0 { + pe.Status = resp.StatusCode + } return &pe } if len(b) == 0 {