diff --git a/.claude/skills/stone/SKILL.md b/.claude/skills/stone/SKILL.md index ec6a76f..2e29a6e 100644 --- a/.claude/skills/stone/SKILL.md +++ b/.claude/skills/stone/SKILL.md @@ -67,7 +67,7 @@ Verbs `ls / get / create / update / delete / edit` are synthesized from a single | `invite` | yes | full | | `nats-user`, `nats-role`, `nats-import`, `nats-export` | yes | full | | `nebula-network`, `nebula-host` | yes | full | -| `nats-account`, `nebula-ca` | yes | `ls / get / update / edit` only | +| `nats-account`, `nebula-ca` | yes | `ls / get / update / edit` only (and every field is operator-only in practice — see Nebula overlay / `stone nats account-keys`) | | `membership` | no (org relation present, but not auto-filtered) | full | | `organization` | no (gated server-side by `is_operator`) | full | @@ -180,6 +180,20 @@ stone nats sync-context # re-issue per-org creds after key rotation `stone` does **not** manage JetStream consumers — use the `nats` CLI for that. +## Nebula overlay + +Records are ordinary entities. Two operations are routes, under `stone nebula`: + +```sh +stone nebula cert-audit # hosts whose certificate no longer matches their network +stone nebula ca-rotate prepare|commit|finish # roll the org's CA, one step at a time +``` + +- **Never run the three rotation steps back to back.** Nebula verification is mutual and hosts pull their config on their own schedule, so `commit` is only safe once *every* host has fetched what `prepare` published. Running them together is the single write the three-step design exists to avoid, and it splits the mesh until propagation catches up. `prepare` is reversible; `commit` re-signs every active host; `finish` drops the outgoing CA and is refused (by name) while any active host still holds one. +- **`cert-audit` finds hosts that look completely healthy and reach nothing** — active, in date, certificate present, config rendered. pb-nebula signed certificates at `/32` until v0.3.0, and Nebula builds the host's overlay route from the certificate's network. Fix one at a time with `stone nebula-host update --renew`, redeploying each config; never script a sweep, because re-signing moves a fingerprint and the revocation blocklist matches on fingerprints. +- **`--active=false` on a `nebula-host` revokes it across the whole CA**, not just its own network, and takes effect when each *peer's* config is redeployed. Deactivate to revoke — deleting the record leaves the certificate trusted until expiry. +- **`--is-relay` requires `--public-host-port`.** `--unsafe-networks` (on the gateway, signed into its certificate) and `--unsafe-routes` (on each consumer, `via` the gateway's overlay IP) are two halves on two different hosts; setting one alone moves no traffic. + ## Common failure modes and how to react - **`not authenticated. run: stone auth login`** — auth token is missing or expired. Surface to user; they run `stone auth login`. @@ -189,6 +203,8 @@ stone nats sync-context # re-issue per-org creds after key rotation - **`nats-sync: skipped — no membership found for this user+org`** — the authenticated user is an operator on an org they aren't a member of; NATS creds are per-membership. Not a bug. - **HTTP 400 from PocketBase on a relation field** — likely passed something that isn't a 15-char id. Re-look it up with `get --fields id -o json` or `ls -o json`. - **`multiple match "..."`** — the natural key is ambiguous in this org. Use one of the 15-char ids listed in the error. +- **`Invalid CA rotation: ...`** — the rotation steps were run out of order, or `finish` was attempted while a host named in the message still holds an outgoing-CA certificate. Read the message; it says which. There is no force flag. +- **HTTP 404 updating `nebula-ca` or `nats-account`** — both are operator-only for every field. The tenant operations are `stone nebula ca-rotate` and `stone nats account-keys`. ## Things not to do @@ -197,3 +213,5 @@ stone nats sync-context # re-issue per-org creds after key rotation - Don't run `stone apply` against a workspace you didn't `pull` from or hand-author with knowledge of the schema — apply will dutifully create records. - Don't expect `apply` to delete things. It only creates and updates. - Don't bypass `stone org switch` by editing `context.yaml` directly — you'll skip the per-org NATS creds sync. +- Don't script a `--renew` sweep across every host `cert-audit` reports. Each re-issue moves a fingerprint and rewrites every peer's config; do them one at a time, redeploying as you go. +- Don't look for a `--rotate` flag on `nebula-ca`. The field exists but the collection's update rule is operator-only, so a tenant PATCH 404s; `stone nebula ca-rotate` is the way in. diff --git a/CHANGELOG.md b/CHANGELOG.md index 95bd077..998a5ac 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -27,6 +27,62 @@ that period, and this file starts where the versioned releases do. ### Added +- **`stone nebula`**, for the two overlay operations that are not record writes. + + `stone nebula ca-rotate prepare|commit|finish` rolls the organization's Nebula + CA. It is a route (`POST /api/org/nebula-ca/rotate`) because + `nebula_ca.updateRule` is operator-only — permitting the trigger through an + update rule would mean a deny-list over the certificate, the private key and + the rest of the CA material, on the record holding the trust anchor for the + whole mesh. + + Three steps, and the wait between them is the feature. Nebula verification is + mutual — each peer checks the other against its *own* local CA pool, with no + chain and no fallback — and hosts pull their config on their own schedule, so + one write carrying both the new trust bundle and the new certificate splits + the mesh until propagation finishes. `prepare` publishes trust and moves no + issuance, so it is reversible; `commit` switches issuance and re-signs every + active host; `finish` drops the outgoing CA and is refused while any active + host still holds one, naming the host. + + `stone nebula cert-audit` lists active hosts whose certificate no longer + matches their network. pb-nebula signed host certificates at `/32` until + v0.3.0, and Nebula builds a host's overlay route from the network in its + certificate — so such a host reaches no peer while looking entirely healthy: + active, in date, certificate present, config rendered, nothing logged. An + edited `overlay_ip` lands a host here too. It is a route + (`GET /api/org/nebula/cert-audit`) because answering it means parsing a Nebula + certificate, which no client can do. + + Fix one host at a time with `stone nebula-host update --renew`, + redeploying each config as you go. There is deliberately no bulk verb: + re-signing moves a certificate's fingerprint, and a fingerprint is what the + revocation blocklist matches, so a sweep rewrites every peer config in the + mesh. + +- **Seven new `nebula-host` flags**, from pb-nebula v0.3.0: `--is-relay`, + `--unsafe-networks`, `--unsafe-routes`, `--preferred-ranges`, `--mtu`, + `--tun-device` and `--renew`. `is_relay` joins the `ls` columns, which now + badge lighthouse and relay separately — a host can be both. + + `--unsafe-networks` and `--unsafe-routes` are two halves of the same feature + living on *different* hosts: the first is signed into the gateway's + certificate and authorizes it to route that subnet, the second goes on each + host that wants to reach it. Neither derives the other. + + **These require a platform on pb-nebula v0.3.0 or newer** (the platform pins + v0.3.2). Against v0.2.0 they name fields the collection does not have, so + PocketBase discards the write and the command reports success — the same + failure the `message-schema` removal above describes. + +- **`routeOnlyFields` in the schema drift guard.** The existing tests catch a + flag for a field that does not exist. They cannot catch a flag for a field + that *does* exist and that a tenant is not allowed to write — the write simply + 404s at the rule layer, on a command that looks like every other update. That + is the shape `nebula_ca.rotate` would take if anyone added a `--rotate` flag, + and the `nats_accounts` signing-key triggers have been in that position all + along with only a prose note guarding them. + - **`--code` on `organization`,** and `code` in its `ls` columns. The one globally unique identifier in the ecosystem: derived from the name when omitted, immutable once set, and baked into signed NATS account JWTs and @@ -36,6 +92,12 @@ that period, and this file starts where the versioned releases do. ### Fixed +- **Every error the CLI printed ended in `(0)`.** PocketBase sends the status as + `status`; `PBError` only read `code`, which nothing populates. So the number + in every error message was zero, and the distinction that matters most — + a 400 from a validator versus a 404 from an update rule — was invisible. Both + names are read now, with the HTTP status as a fallback. + - **Refreshed the vendored platform schema** (`cmd/testdata/schema.json`), which is what the drift tests check the field table against. It was three platform changes behind. Worth recording that the guard did its job unprompted: told diff --git a/CLAUDE.md b/CLAUDE.md index bbc6365..68fc668 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -73,6 +73,13 @@ intentional — do not "fix" them: - **`nebula-ca.rotate_keys`** — no such field exists, in `schema.json` or in pb-nebula. It was a flag once and did nothing: PocketBase silently drops writes to fields a collection doesn't have, so it reported success every time. +- **`nebula-ca.rotate`** — this one is real (pb-nebula v0.3.0), and is still not + a flag: `nebula_ca.updateRule` is operator-only, so a tenant PATCH 404s. The + three steps live behind `stone nebula ca-rotate`. Same for the material a + rotation produces — `next_certificate`, `previous_certificate`, `rotated_at`. + `routeOnlyFields` in `cmd/schema_drift_test.go` is what stops any of them + becoming a flag later; it covers the `nats_accounts` triggers too, which were + previously only prose. - **`nats-account.rotate_keys` / `add_signing_key` / `remove_signing_key`** — the fields exist, but `nats_accounts.updateRule` is operator-only, so a tenant PATCH 404s. They live behind `stone nats account-keys` instead. @@ -89,13 +96,42 @@ platform repo is normally at `../platform`. ### Custom platform routes -`internal/pb/client.go` has `CallRoute` for the platform's non-collection -endpoints, and `cmd/creds.go` wraps the two that exist: -`POST /api/me/nats-creds/rotate` and `POST /api/org/nats-account/keys`. Both -exist server-side because an API rule cannot express a single-field allowlist. -Neither takes a record id — the target is derived from the caller's identity or +`internal/pb/client.go` has `CallRoute` (POST) and `GetRoute` (GET) for the +platform's non-collection endpoints. Four are wrapped: + +- `cmd/creds.go` — `POST /api/me/nats-creds/rotate` and + `POST /api/org/nats-account/keys`. +- `cmd/nebula.go` — `POST /api/org/nebula-ca/rotate` and + `GET /api/org/nebula/cert-audit`. + +Three of the four exist because an API rule cannot express a single-field +allowlist: permitting one trigger field through an update rule means a deny-list +over every other, which silently opens the moment someone adds a field. None of +them takes a record id — the target is derived from the caller's identity or active organization — so don't add one. +`cert-audit` is the exception and exists for a different reason: deciding whether +a host certificate still matches its network means parsing a Nebula certificate, +which no client can do. It answers in **ids**, and `cmd/nebula.go` resolves them +to hostnames before printing — falling back to the bare id rather than dropping +the row, because a host we cannot name is still a host that needs re-signing. + +### pb-nebula v0.3 and the library-owned fields + +pb-nebula v0.3.0 added twelve fields the platform's `schema.json` does not +declare — the dump predates the release. Unlike the pb-nats case noted in +`cmd/schema_drift_test.go`, pb-nebula **migrates** them (`addMissingFields`), so +a database created earlier acquires them on the next start rather than never. +They are listed in `libraryOwnedFields` so the drift guard can see them. + +**The floor is pb-nebula v0.3.0** (the platform pins v0.3.2). Against v0.2.0 the +two Nebula routes 404, and `--is-relay`, `--unsafe-networks`, `--unsafe-routes`, +`--preferred-ranges`, `--mtu`, `--tun-device` and `--renew` all name fields the +collection does not have — so the write is discarded and the command reports +success. That is precisely the failure the drift guard exists to make loud, and +it cannot catch this one, because the vendored schema is a copy of a file that +never declared them either. + ### Pull / apply (GitOps) `cmd/sync.go`: - `stone pull` writes one YAML file per record into `//.yaml`, where `` is the spec's `LookupKey` value, falling back to `name`, then id. Filename collisions get a `-` suffix; records are pulled sorted by id so the suffix lands on the same record across pulls. Filenames are cosmetic — apply identifies records solely by the `id` field inside the file. Org-scoped collections are filtered by `current_organization`. Server-only fields (`collectionId`, `collectionName`, `created`, `updated`, `expand`) are stripped on read (see `pb.ServerOnlyFields` / `pb.Strip`). diff --git a/README.md b/README.md index cb1712e..cd8266f 100644 --- a/README.md +++ b/README.md @@ -322,6 +322,56 @@ All KV operations — bucket lifecycle and per-key data — live under `stone kv ./stone kv watch twins ``` +## Nebula + +The records are ordinary entities — `stone nebula-ca`, `stone nebula-network`, +`stone nebula-host`. Two operations are not record writes, and live under +`stone nebula`: + +```sh +stone nebula cert-audit # hosts whose certificate no longer matches their network +stone nebula ca-rotate prepare # publish trust in a new CA (reversible) +stone nebula ca-rotate commit # switch issuance, re-sign every active host +stone nebula ca-rotate finish # drop the outgoing CA +``` + +**`ca-rotate` takes three steps and the wait between them is the point.** Nebula +verification is mutual — each peer checks the other against its *own* local CA +pool, with no chain and no fallback — and hosts pull their config whenever they +like. So one write carrying both the new trust bundle and the new certificate +splits the mesh: a host that has fetched presents a new-CA certificate to one +that has not, and the handshake fails in *both* directions until propagation +finishes. `prepare` publishes trust and moves no issuance, so it is fully +reversible. `commit` switches issuance and re-signs every active host, with both +CAs trusted throughout. `finish` drops the outgoing CA and is refused while any +active host still holds a certificate signed by it — the refusal names the host. + +A CA cannot be renewed, only rotated, so start months ahead of the expiry in +`stone nebula-ca ls`, not weeks. + +**`cert-audit` answers a question no client can.** pb-nebula signed host +certificates at `/32` until v0.3.0. Nebula puts a certificate's network straight +onto the tun device and installs a link route for it, so the mask in the +certificate *is* the host's route to the overlay — a `/32` gives a host a route +covering only itself. The certificate verifies, the config renders, the host +starts, the handshake completes, and no packet crosses the mesh. Nothing errors, +which is why you have to ask. Editing a host's `overlay_ip` after issue lands it +here too. + +Nothing is re-signed automatically: re-signing moves a fingerprint, and a +fingerprint is what the revocation blocklist matches, so a sweep would rewrite +every peer config in the mesh. Fix one host at a time, redeploying as you go: + +```sh +stone nebula-host update edge-west --renew +``` + +Requires a platform on **pb-nebula v0.3.0 or newer**. Against v0.2.0 the routes +404 and the newer host flags — `--is-relay`, `--unsafe-networks`, +`--unsafe-routes`, `--preferred-ranges`, `--mtu`, `--tun-device`, `--renew` — +name fields the collection does not have, so PocketBase discards the write and +the command reports success. + ## Limitations - Relation flags (`--type`, `--location`, …) take 15-char PocketBase ids only — @@ -332,9 +382,8 @@ All KV operations — bucket lifecycle and per-key data — live under `stone kv - No JetStream consumer management — the `nats` CLI is better at that. - `nats-account` and `nebula-ca` are **operator-only for every field**. Both `updateRule`s admit no tenant role, so an owner/admin PATCH of any field on - either collection returns 404. The three legitimate tenant key operations live - behind `stone nats account-keys` instead. `nebula_ca` has no rotation trigger - at all — rolling a CA is an operator action. + either collection returns 404. The legitimate tenant operations live behind + routes instead: `stone nats account-keys` and `stone nebula ca-rotate`. - Locations' `floorplan` and organizations' `logo` are file fields; the CLI has no upload path for them. Use the console. diff --git a/SKILLS.md b/SKILLS.md index 3fa3a4c..f19428e 100644 --- a/SKILLS.md +++ b/SKILLS.md @@ -194,12 +194,35 @@ stone nats account-keys remove-signing # last remaining key cannot be stone nats account-keys rotate # EMERGENCY: invalidates every user JWT in the account ``` +## Nebula overlay — two operations that are not record writes + +The records are ordinary entities (`nebula-ca`, `nebula-network`, `nebula-host`). +These two live under `stone nebula` because a PocketBase rule cannot express them. + +```sh +stone nebula cert-audit # active hosts whose certificate no longer matches their network +stone nebula ca-rotate prepare|commit|finish +``` + +Rules a caller must not get wrong: + +- **`ca-rotate` is three steps with a wait between them, and the wait is the feature.** Nebula verification is mutual — each peer checks the other against its *own* local CA pool, with no chain and no fallback — and hosts pull their config on their own schedule. `prepare` publishes trust in the incoming CA and moves no issuance, so it is fully reversible. Only once **every** host has fetched its config does `commit` become safe; it switches issuance and re-signs every active host, with both CAs trusted throughout. `finish` drops the outgoing CA. Never run the three back to back — that is the single write the three-step design exists to avoid, and it splits the mesh for as long as propagation takes. +- **`finish` is interlocked and the refusal is informative.** It is refused while any active host still holds a certificate from the outgoing CA, and the error names the host. Deploy that host's config and retry; do not look for a force flag, there isn't one. +- **A CA cannot be renewed, only rotated.** Check `stone nebula-ca ls` for the expiry and start months ahead, not weeks — the wait in the middle cannot be compressed. +- **`cert-audit` reports a failure that is invisible everywhere else.** pb-nebula signed host certificates at `/32` until v0.3.0; Nebula builds the host's overlay route from the certificate's network, so such a host reaches no peer while looking entirely healthy — active, in date, certificate present, config rendered, no errors. Editing a host's `overlay_ip` after issue has the same effect. +- **Fix stale hosts one at a time**, with `stone nebula-host update --renew`, redeploying each config as you go. Do not script a sweep: re-signing moves a certificate's fingerprint, and a fingerprint is what the revocation blocklist matches, so a bulk re-issue rewrites every peer config in the mesh. +- **`--active=false` on a `nebula-host` is revocation**, and it now reaches every network under the same CA rather than just the host's own — Nebula's trust boundary is the CA. It takes effect when each *peer's* config is redeployed, not instantly. Deactivate to revoke; deleting the record leaves the certificate trusted until it expires, because a fingerprint that is not in the database cannot be blocklisted. +- **`--is-relay` needs `--public-host-port`.** Without it the host listens on an ephemeral port while every peer is handed its overlay IP as a usable path. +- **`--unsafe-networks` and `--unsafe-routes` are two halves on two different hosts.** The first is signed into the *gateway's* certificate and authorizes it to route that subnet; the second goes on every host that wants to reach it, with `via` set to the gateway's overlay IP. Neither derives the other, and setting only one moves no traffic. + +Requires a platform on **pb-nebula v0.3.0+**. Against v0.2.0 the routes 404 and the newer host flags name fields the collection lacks, so the write is discarded and the command still reports success. + ## Known limitations - Relation flags do not resolve names — pass 15-char PocketBase ids only. (Positional record args on `get`/`update`/`delete`/`edit` *do* accept natural keys.) - `apply` does not delete server records absent from the workspace. - No JetStream **consumer** management (use `nats` CLI). -- `nats-account` and `nebula-ca` are **operator-only for every field** — both `updateRule`s admit no tenant role, so an owner/admin PATCH of any field on either returns 404. Tenant key operations live at `stone nats account-keys`. `nebula_ca` has no rotation trigger at all. +- `nats-account` and `nebula-ca` are **operator-only for every field** — both `updateRule`s admit no tenant role, so an owner/admin PATCH of any field on either returns 404. The tenant operations live behind routes: `stone nats account-keys` and `stone nebula ca-rotate`. - File fields have no CLI upload path: `locations.floorplan`, `organizations.logo`. Use the console. ## Configuration files diff --git a/cmd/entity.go b/cmd/entity.go index 76ee6f0..06e069d 100644 --- a/cmd/entity.go +++ b/cmd/entity.go @@ -421,7 +421,7 @@ var entitySpecs = []EntitySpec{ Plural: "nebula-hosts", Collection: "nebula_hosts", OrgScoped: true, - KeyColumns: []string{"hostname", "overlay_ip", "network_id", "is_lighthouse", "active"}, + KeyColumns: []string{"hostname", "overlay_ip", "network_id", "is_lighthouse", "is_relay", "active"}, LookupKey: "hostname", Fields: []Field{ {Name: "email", Type: FString, Required: true, Help: "auth email (required by the auth collection)"}, @@ -431,11 +431,34 @@ var entitySpecs = []EntitySpec{ {Name: "network_id", Type: FID, Required: true, Help: "nebula_networks id"}, {Name: "groups", Type: FJSON, Help: "Nebula firewall groups (JSON array)"}, {Name: "is_lighthouse", Type: FBool, Help: "whether this host is a lighthouse"}, - {Name: "public_host_port", Type: FString, Help: "public address:port (for lighthouses / static peers)"}, + {Name: "is_relay", Type: FBool, Help: "forward traffic for peers that cannot reach each other directly (needs --public-host-port)"}, + {Name: "public_host_port", Type: FString, Help: "public address:port; required for a lighthouse AND for a relay"}, {Name: "firewall_outbound", Type: FJSON, Help: "outbound firewall rules (JSON)"}, {Name: "firewall_inbound", Type: FJSON, Help: "inbound firewall rules (JSON)"}, + + // Gateway routing: two halves on two DIFFERENT hosts, and neither + // derives the other. unsafe_networks is signed INTO the gateway's + // certificate -- Nebula authorizes routing on the certificate, not + // on config -- so setting it re-issues, and the change is inert + // until that host picks the new certificate up. + {Name: "unsafe_networks", Type: FJSON, Help: `subnets THIS host routes to, e.g. ["192.168.1.0/24"] (signed into the certificate)`}, + {Name: "unsafe_routes", Type: FJSON, Help: `subnets reached THROUGH another host, e.g. [{"route":"192.168.1.0/24","via":"10.0.0.7"}]`}, + + // Underlay, not overlay -- which is why IPv6 is accepted here and + // nowhere else in the platform. + {Name: "preferred_ranges", Type: FJSON, Help: `underlay prefixes to favour when a peer has several addresses, e.g. ["192.168.1.0/24"]`}, + + {Name: "mtu", Type: FInt, Help: "MTU override; 0 inherits the default"}, + {Name: "tun_device", Type: FString, Help: "tun interface name override; empty inherits the default"}, + {Name: "validity_years", Type: FInt, Help: "certificate validity in years"}, - {Name: "active", Type: FBool, Help: "whether the host is active"}, + {Name: "active", Type: FBool, Help: "whether the host is active; clearing it BLOCKLISTS the certificate across the CA"}, + + // An action field, not state: pb-nebula re-issues on the false -> + // true edge and resets it in the same save, so it never reads back. + // Exposed because it is the fix for a host whose certificate no + // longer matches its network -- see `stone nebula cert-audit`. + {Name: "renew", Type: FBool, Help: "re-issue this host's certificate now (resets itself; redeploy the config afterwards)"}, }, }, { @@ -450,11 +473,20 @@ var entitySpecs = []EntitySpec{ {Name: "name", Type: FString, Help: "CA name"}, {Name: "validity_years", Type: FInt, Help: "CA cert validity in years (operator-only)"}, {Name: "curve", Type: FString, Help: "elliptic curve, e.g. P256 (operator-only)"}, - // There is no `rotate_keys` here on purpose: nebula_ca has no such - // field, in schema.json or in pb-nebula. The flag used to exist and - // did nothing at all -- PocketBase silently drops writes to fields a - // collection does not have, so it reported success every time. - // Rolling a CA is a platform-operator operation, not a tenant one. + + // There is still no `rotate_keys` here, and there never was a field + // by that name -- the flag that used to exist did nothing at all, + // because PocketBase silently drops writes to fields a collection + // does not have and so reported success every time. + // + // pb-nebula v0.3.0 DID add a real rotation trigger, `rotate`. It is + // deliberately not a flag either, for the reason the nats-account + // key triggers are not: nebula_ca.updateRule is operator-only, so a + // tenant PATCH 404s, and the three steps live behind + // `stone nebula ca-rotate` instead. The same goes for the material + // rotation produces -- next_certificate, previous_certificate, + // rotated_at -- which is server-generated and readable via `get`. + // routeOnlyFields in schema_drift_test.go is what holds that line. }, }, diff --git a/cmd/nebula.go b/cmd/nebula.go new file mode 100644 index 0000000..452b084 --- /dev/null +++ b/cmd/nebula.go @@ -0,0 +1,244 @@ +package cmd + +import ( + "fmt" + "os" + "sort" + + "github.com/spf13/cobra" + "github.com/stone-age-io/stone-cli/internal/ctx" + "github.com/stone-age-io/stone-cli/internal/pb" +) + +// The two Nebula operations that are routes rather than record writes, for the +// same two reasons the NATS ones in creds.go are. +// +// ca-rotate exists because nebula_ca.updateRule is operator-only. Permitting the +// rotation trigger through an update rule would mean a deny-list over the +// certificate, the private key, the incoming and outgoing CA material, the +// expiry and the curve -- and that deny-list would silently open up the moment +// anyone added a field, on the record holding the trust anchor for the whole +// tenant's mesh. +// +// cert-audit exists for a different reason: answering it means parsing a Nebula +// certificate to compare the network it carries against the network the host +// belongs to. No client can do that, so the platform does it and hands back the +// verdict. + +var nebulaCmd = &cobra.Command{ + Use: "nebula", + Short: "Nebula overlay operations that are not record writes", + Long: `Nebula operations the collection API cannot express. + +The records themselves are managed with the entity commands -- nebula-ca, +nebula-network and nebula-host. What lives here is the CA rotation lever, which +nebula_ca.updateRule deliberately does not expose, and the host certificate +audit, which needs a certificate parsed to answer.`, +} + +var nebulaCARotateCmd = &cobra.Command{ + Use: "ca-rotate", + Short: "Rotate your organization's Nebula CA, in three steps", + Long: `Roll the Nebula CA your organization's host certificates chain to. + +Three steps, and the wait between them is the feature. + +Nebula verification is mutual -- each peer checks the other against its OWN local +CA pool, with no chain and no fallback -- and hosts pull their config whenever +they like, with nothing telling the platform when they did. So one write carrying +both the new trust bundle and the new certificate splits the mesh: a host that +has fetched presents a new-CA certificate to one that has not, and the handshake +fails in BOTH directions until propagation finishes. + + prepare mint the incoming CA and publish it as additional trust. + Issuance does not move, no host certificate changes and no + fingerprint moves, so this is fully reversible. + + -- wait here until every host has fetched its config -- + + commit switch issuance to the new CA and re-sign every ACTIVE host. + Both CAs stay trusted, so re-signed and not-yet-re-signed hosts + still talk to each other. Idempotent: re-running it re-signs only + the hosts still on the outgoing CA, which is how you recover from a + partial sweep. + + -- deploy the new config to every active host -- + + finish drop the outgoing CA. Refused while any active host still holds a + certificate signed by it; the error names the host. + +How long to wait in the middle is your judgement and nothing can compress it. +A CA cannot be renewed, only rotated, so start months before expiry rather than +weeks -- see the expiry on: stone nebula-ca ls + +Owner or admin of the active organization. Takes no id: the CA is derived from +your active organization, so this cannot be aimed at another tenant.`, +} + +var nebulaCARotatePrepareCmd = &cobra.Command{ + Use: "prepare", + Short: "Mint the incoming CA and publish trust in it (reversible)", + Args: cobra.NoArgs, + RunE: func(cmd *cobra.Command, args []string) error { return rotateCAStep("prepare") }, +} + +var nebulaCARotateCommitCmd = &cobra.Command{ + Use: "commit", + Short: "Switch issuance to the new CA and re-sign every active host", + Long: `Switch issuance to the prepared CA and re-sign every active host. + +Run this only once every host has fetched the config prepare published -- until +then, some peers do not yet trust the CA the re-signed hosts now present. + +Inactive hosts are deliberately NOT re-signed: an inactive host is revoked, its +fingerprint sits in every peer's blocklist, and re-signing it would publish a new +fingerprint while the old certificate stayed valid.`, + Args: cobra.NoArgs, + RunE: func(cmd *cobra.Command, args []string) error { return rotateCAStep("commit") }, +} + +var nebulaCARotateFinishCmd = &cobra.Command{ + Use: "finish", + Short: "Drop the outgoing CA (refused while any active host still uses it)", + Long: `Drop the outgoing CA from every host's trust bundle. + +Refused while any active host still holds a certificate signed by it, and the +refusal names the host. That interlock is what makes this step safe to expose: +dropping the outgoing CA early does not fail loudly, it just takes that host off +the mesh -- it keeps running, and its peers quietly stop being able to verify it. + +If it refuses, deploy the config to the host it names and try again.`, + Args: cobra.NoArgs, + RunE: func(cmd *cobra.Command, args []string) error { return rotateCAStep("finish") }, +} + +// rotateCAStep posts one step to the rotation route. The CA is derived +// server-side from the caller's active organization, so there is no id to pass. +func rotateCAStep(step string) error { + c, err := ctx.Active(flagContext) + if err != nil { + return err + } + client := newPBClient(c) + + var out struct { + Applied string `json:"applied"` + NebulaCA string `json:"nebula_ca"` + } + // The server refuses an out-of-order step and names the host blocking a + // finish. Those messages are the whole point of the interlock, so they are + // returned as-is rather than replaced with something tidier. + if err := client.CallRoute("/api/org/nebula-ca/rotate", map[string]any{"step": step}, &out); err != nil { + return err + } + + fmt.Printf("%s: applied to CA %s\n", out.Applied, out.NebulaCA) + switch step { + case "prepare": + fmt.Println("every host now trusts the incoming CA. Let them all fetch their config, then: stone nebula ca-rotate commit") + case "commit": + fmt.Println("active hosts have been re-signed. Deploy their configs, then: stone nebula ca-rotate finish") + case "finish": + fmt.Println("the outgoing CA has been dropped. Rotation complete.") + } + return nil +} + +var nebulaCertAuditCmd = &cobra.Command{ + Use: "cert-audit", + Short: "List hosts whose certificate no longer matches their network", + Long: `Report active hosts whose certificate carries the wrong overlay network. + +pb-nebula signed host certificates at /32 until v0.3.0. Nebula does not read a +certificate's network as "this host's address" -- it puts the prefix straight +onto the tun device and installs a link route for it, so the mask in the +certificate IS the host's route to the overlay. A /32 gives a host a route +covering only itself: the certificate verifies, the config renders, the host +starts, the handshake completes, and no packet ever crosses the mesh. + +Nothing errors anywhere, which is why this needs asking for rather than waiting +to be told. A host's overlay_ip being edited after issue lands it here too. + +Nothing is re-signed automatically, and that restraint is deliberate: re-signing +moves a certificate's fingerprint, and a fingerprint is what the revocation +blocklist matches, so a sweep would rewrite every peer config in the mesh. Fix +one host at a time, redeploying each config as you go: + + stone nebula-host update --renew + +Inactive hosts are excluded. They are revoked, and re-signing one would publish a +new fingerprint while the old certificate stayed valid and unblocklisted. + +Owner or admin of the active organization.`, + Args: cobra.NoArgs, + RunE: func(cmd *cobra.Command, args []string) error { + c, err := ctx.Active(flagContext) + if err != nil { + return err + } + client := newPBClient(c) + + var out struct { + Stale []string `json:"stale"` + } + if err := client.GetRoute("/api/org/nebula/cert-audit", &out); err != nil { + return err + } + + if len(out.Stale) == 0 { + fmt.Println("no host certificate is out of step with its network") + return nil + } + + // The route answers in ids, because that is what it can be sure of. An + // id is not what anyone types, so resolve to the hostname and network + // the operator will actually act on -- and fall back to the bare id + // rather than dropping a row if a lookup fails, since a host we cannot + // name is still a host that needs re-signing. + stale := map[string]bool{} + for _, id := range out.Stale { + stale[id] = true + } + + hosts, err := client.ListAll("nebula_hosts", pb.ListOptions{ + Fields: "id,hostname,overlay_ip,network_id", + Sort: "hostname", + }) + if err != nil { + hosts = nil + } + + named := map[string]pb.Record{} + for _, h := range hosts { + if id, _ := h["id"].(string); stale[id] { + named[id] = h + } + } + + rows := make([]pb.Record, 0, len(out.Stale)) + for _, id := range out.Stale { + if h, ok := named[id]; ok { + rows = append(rows, h) + continue + } + rows = append(rows, pb.Record{"id": id, "hostname": "(not readable)"}) + } + sort.Slice(rows, func(i, j int) bool { + a, _ := rows[i]["hostname"].(string) + b, _ := rows[j]["hostname"].(string) + return a < b + }) + + if err := pb.PrintList(os.Stdout, rows, []string{"hostname", "overlay_ip", "network_id"}, resolveOutput()); err != nil { + return err + } + fmt.Fprintf(os.Stderr, "\n%d host(s) need re-issuing: stone nebula-host update --renew\n", len(rows)) + return nil + }, +} + +func init() { + nebulaCARotateCmd.AddCommand(nebulaCARotatePrepareCmd, nebulaCARotateCommitCmd, nebulaCARotateFinishCmd) + nebulaCmd.AddCommand(nebulaCARotateCmd, nebulaCertAuditCmd) + rootCmd.AddCommand(nebulaCmd) +} diff --git a/cmd/schema_drift_test.go b/cmd/schema_drift_test.go index 22f356a..959852a 100644 --- a/cmd/schema_drift_test.go +++ b/cmd/schema_drift_test.go @@ -100,8 +100,78 @@ var deliberatelyOmitted = map[string][]string{ // either. This is the same shape as the system_account_id bug the platform fixed // in v0.2.0. Flagged upstream; the CLI's flags are correct against a current // database. +// +// pb-nebula v0.3.0 is the other case, and it behaves BETTER than the one above. +// It added twelve fields across nebula_hosts and nebula_ca, and unlike pb-nats +// it migrates them: InitializeCollections adds any declared field an existing +// collection is missing (addMissingFields), so a database created before v0.3.0 +// acquires them on the next start rather than never. The platform's schema.json +// still does not declare them -- it is a dump that predates the release, and +// hand-copying a library's field definitions into it is the drift risk the +// platform's own CLAUDE.md warns about -- so they are listed here instead. +// +// The floor is pb-nebula v0.3.0 (the platform pins v0.3.2). Against v0.2.0 every +// flag below is a silent no-op, which is the failure this whole file exists to +// make loud. var libraryOwnedFields = map[string][]string{ "nats_roles": {"allow_response", "allow_response_max", "allow_response_ttl"}, + + // Exposed as flags: the relay role, the two halves of gateway routing, + // underlay path preference, the transport overrides, and the renew trigger. + "nebula_hosts": { + "is_relay", + "unsafe_networks", "unsafe_routes", + "preferred_ranges", + "mtu", "tun_device", + "renew", + }, +} + +// routeOnlyFields are fields that DO exist on a live collection and must never +// become CLI flags, because the collection's updateRule does not let a tenant +// write them -- the operation lives behind a platform route instead. +// +// This is a different failure from the one the two tests above catch, and +// neither of them would see it. A flag here would name a real field, so the +// "collection has no such field" check passes; the write simply 404s at the rule +// layer, on a command that looks exactly like every other update. +// +// It is also the hole that let `nebula-ca --rotate-keys` exist: that flag named +// a field that never existed anywhere, but the same flag naming a field that +// exists-and-is-forbidden would have been just as broken and even quieter. +var routeOnlyFields = map[string]map[string]string{ + "nebula_ca": { + "rotate": "nebula_ca.updateRule is operator-only; use `stone nebula ca-rotate`", + // The material a rotation produces. Server-generated, readable via + // `get`, never typed in. + "next_certificate": "server-generated during rotation", + "next_private_key": "server-generated during rotation, and hidden from the API", + "previous_certificate": "server-generated during rotation", + "rotated_at": "server-generated during rotation", + }, + "nats_accounts": { + "rotate_keys": "nats_accounts.updateRule is operator-only; use `stone nats account-keys rotate`", + "add_signing_key": "operator-only rule; use `stone nats account-keys add-signing`", + "remove_signing_key": "operator-only rule; use `stone nats account-keys remove-signing`", + }, +} + +func TestRouteOnlyFieldsAreNotFlags(t *testing.T) { + for _, spec := range entitySpecs { + forbidden, ok := routeOnlyFields[spec.Collection] + if !ok { + continue + } + t.Run(spec.Name, func(t *testing.T) { + for _, f := range spec.Fields { + if why, bad := forbidden[f.Name]; bad { + t.Errorf("field %q has a --%s flag, but a tenant cannot write it: %s.\n"+ + "The command would look like every other update and 404 at the rule layer.", + f.Name, f.flagName(), why) + } + } + }) + } } func loadVendoredSchema(t *testing.T) map[string]schemaCollection { diff --git a/internal/pb/client.go b/internal/pb/client.go index a9326ed..3d171f8 100644 --- a/internal/pb/client.go +++ b/internal/pb/client.go @@ -277,6 +277,29 @@ func (c *Client) CallRoute(path string, body any, out any) error { return json.NewDecoder(resp.Body).Decode(out) } +// GetRoute is CallRoute for the platform's read-only routes. +// +// Separate rather than a method parameter on CallRoute: every caller of that one +// is performing an action, and its doc comment is about why those actions are +// routes at all. A read is there for a different reason -- the platform's +// certificate audit has to parse a Nebula certificate to answer, which no +// client can do -- and collapsing the two would put an unused body argument in +// front of every read. +func (c *Client) GetRoute(path string, out any) error { + resp, err := c.do(http.MethodGet, path, nil, true) + if err != nil { + return err + } + defer resp.Body.Close() + if err := checkOK(resp); err != nil { + return err + } + if out == nil { + return nil + } + return json.NewDecoder(resp.Body).Decode(out) +} + // BatchOp is one operation in a /api/batch request. type BatchOp struct { Method string `json:"method"` // "POST" | "PATCH" | "DELETE" @@ -406,17 +429,33 @@ func (c *Client) do(method, path string, body io.Reader, needAuth bool) (*http.R } // PBError is the standard error shape PocketBase returns. +// +// The status arrives as `status`, not `code` -- PocketBase's ApiError marshals +// it under that name, and has for every version this CLI has talked to. Reading +// only `code` meant the number in every error message the CLI has ever printed +// was 0, including on the ones where it matters most: a 404 from an update rule +// and a 400 from a validator read identically. Both names are accepted now, and +// checkOK fills in the HTTP status if a server sends neither. type PBError struct { Code int `json:"code"` + Status int `json:"status"` Message string `json:"message"` Data map[string]any `json:"data"` } +// status is whichever the server actually sent. +func (e *PBError) status() int { + if e.Status != 0 { + return e.Status + } + return e.Code +} + func (e *PBError) Error() string { if len(e.Data) > 0 { - return fmt.Sprintf("%s (%d): %s", e.Message, e.Code, formatPBData(e.Data)) + return fmt.Sprintf("%s (%d): %s", e.Message, e.status(), formatPBData(e.Data)) } - return fmt.Sprintf("%s (%d)", e.Message, e.Code) + return fmt.Sprintf("%s (%d)", e.Message, e.status()) } func formatPBData(d map[string]any) string { @@ -434,6 +473,9 @@ func checkOK(resp *http.Response) error { b, _ := io.ReadAll(resp.Body) var pe PBError if err := json.Unmarshal(b, &pe); err == nil && pe.Message != "" { + if pe.status() == 0 { + pe.Status = resp.StatusCode + } return &pe } if len(b) == 0 {