feat(nebula): CA rotation, the certificate audit, and pb-nebula v0.3's host flags

.claude/skills/stone/SKILL.md

@@ -67,7 +67,7 @@ Verbs `ls / get / create / update / delete / edit` are synthesized from a single
 | `invite` | yes | full |
 | `nats-user`, `nats-role`, `nats-import`, `nats-export` | yes | full |
 | `nebula-network`, `nebula-host` | yes | full |
-| `nats-account`, `nebula-ca` | yes | `ls / get / update / edit` only |
+| `nats-account`, `nebula-ca` | yes | `ls / get / update / edit` only (and every field is operator-only in practice — see Nebula overlay / `stone nats account-keys`) |
 | `membership` | no (org relation present, but not auto-filtered) | full |
 | `organization` | no (gated server-side by `is_operator`) | full |
 
@@ -180,6 +180,20 @@ stone nats sync-context        # re-issue per-org creds after key rotation
 
 `stone` does **not** manage JetStream consumers — use the `nats` CLI for that.
 
+## Nebula overlay
+
+Records are ordinary entities. Two operations are routes, under `stone nebula`:
+
+```sh
+stone nebula cert-audit                        # hosts whose certificate no longer matches their network
+stone nebula ca-rotate prepare|commit|finish   # roll the org's CA, one step at a time
+```
+
+- **Never run the three rotation steps back to back.** Nebula verification is mutual and hosts pull their config on their own schedule, so `commit` is only safe once *every* host has fetched what `prepare` published. Running them together is the single write the three-step design exists to avoid, and it splits the mesh until propagation catches up. `prepare` is reversible; `commit` re-signs every active host; `finish` drops the outgoing CA and is refused (by name) while any active host still holds one.
+- **`cert-audit` finds hosts that look completely healthy and reach nothing** — active, in date, certificate present, config rendered. pb-nebula signed certificates at `/32` until v0.3.0, and Nebula builds the host's overlay route from the certificate's network. Fix one at a time with `stone nebula-host update <hostname> --renew`, redeploying each config; never script a sweep, because re-signing moves a fingerprint and the revocation blocklist matches on fingerprints.
+- **`--active=false` on a `nebula-host` revokes it across the whole CA**, not just its own network, and takes effect when each *peer's* config is redeployed. Deactivate to revoke — deleting the record leaves the certificate trusted until expiry.
+- **`--is-relay` requires `--public-host-port`.** `--unsafe-networks` (on the gateway, signed into its certificate) and `--unsafe-routes` (on each consumer, `via` the gateway's overlay IP) are two halves on two different hosts; setting one alone moves no traffic.
+
 ## Common failure modes and how to react
 
 - **`not authenticated. run: stone auth login`** — auth token is missing or expired. Surface to user; they run `stone auth login`.
@@ -189,6 +203,8 @@ stone nats sync-context        # re-issue per-org creds after key rotation
 - **`nats-sync: skipped — no membership found for this user+org`** — the authenticated user is an operator on an org they aren't a member of; NATS creds are per-membership. Not a bug.
 - **HTTP 400 from PocketBase on a relation field** — likely passed something that isn't a 15-char id. Re-look it up with `get <key> --fields id -o json` or `ls -o json`.
 - **`multiple <plural> match <key> "..."`** — the natural key is ambiguous in this org. Use one of the 15-char ids listed in the error.
+- **`Invalid CA rotation: ...`** — the rotation steps were run out of order, or `finish` was attempted while a host named in the message still holds an outgoing-CA certificate. Read the message; it says which. There is no force flag.
+- **HTTP 404 updating `nebula-ca` or `nats-account`** — both are operator-only for every field. The tenant operations are `stone nebula ca-rotate` and `stone nats account-keys`.
 
 ## Things not to do
 
@@ -197,3 +213,5 @@ stone nats sync-context        # re-issue per-org creds after key rotation
 - Don't run `stone apply` against a workspace you didn't `pull` from or hand-author with knowledge of the schema — apply will dutifully create records.
 - Don't expect `apply` to delete things. It only creates and updates.
 - Don't bypass `stone org switch` by editing `context.yaml` directly — you'll skip the per-org NATS creds sync.
+- Don't script a `--renew` sweep across every host `cert-audit` reports. Each re-issue moves a fingerprint and rewrites every peer's config; do them one at a time, redeploying as you go.
+- Don't look for a `--rotate` flag on `nebula-ca`. The field exists but the collection's update rule is operator-only, so a tenant PATCH 404s; `stone nebula ca-rotate` is the way in.

CHANGELOG.md

@@ -27,6 +27,62 @@ that period, and this file starts where the versioned releases do.
 
 ### Added
 
+- **`stone nebula`**, for the two overlay operations that are not record writes.
+
+  `stone nebula ca-rotate prepare|commit|finish` rolls the organization's Nebula
+  CA. It is a route (`POST /api/org/nebula-ca/rotate`) because
+  `nebula_ca.updateRule` is operator-only — permitting the trigger through an
+  update rule would mean a deny-list over the certificate, the private key and
+  the rest of the CA material, on the record holding the trust anchor for the
+  whole mesh.
+
+  Three steps, and the wait between them is the feature. Nebula verification is
+  mutual — each peer checks the other against its *own* local CA pool, with no
+  chain and no fallback — and hosts pull their config on their own schedule, so
+  one write carrying both the new trust bundle and the new certificate splits
+  the mesh until propagation finishes. `prepare` publishes trust and moves no
+  issuance, so it is reversible; `commit` switches issuance and re-signs every
+  active host; `finish` drops the outgoing CA and is refused while any active
+  host still holds one, naming the host.
+
+  `stone nebula cert-audit` lists active hosts whose certificate no longer
+  matches their network. pb-nebula signed host certificates at `/32` until
+  v0.3.0, and Nebula builds a host's overlay route from the network in its
+  certificate — so such a host reaches no peer while looking entirely healthy:
+  active, in date, certificate present, config rendered, nothing logged. An
+  edited `overlay_ip` lands a host here too. It is a route
+  (`GET /api/org/nebula/cert-audit`) because answering it means parsing a Nebula
+  certificate, which no client can do.
+
+  Fix one host at a time with `stone nebula-host update <hostname> --renew`,
+  redeploying each config as you go. There is deliberately no bulk verb:
+  re-signing moves a certificate's fingerprint, and a fingerprint is what the
+  revocation blocklist matches, so a sweep rewrites every peer config in the
+  mesh.
+
+- **Seven new `nebula-host` flags**, from pb-nebula v0.3.0: `--is-relay`,
+  `--unsafe-networks`, `--unsafe-routes`, `--preferred-ranges`, `--mtu`,
+  `--tun-device` and `--renew`. `is_relay` joins the `ls` columns, which now
+  badge lighthouse and relay separately — a host can be both.
+
+  `--unsafe-networks` and `--unsafe-routes` are two halves of the same feature
+  living on *different* hosts: the first is signed into the gateway's
+  certificate and authorizes it to route that subnet, the second goes on each
+  host that wants to reach it. Neither derives the other.
+
+  **These require a platform on pb-nebula v0.3.0 or newer** (the platform pins
+  v0.3.2). Against v0.2.0 they name fields the collection does not have, so
+  PocketBase discards the write and the command reports success — the same
+  failure the `message-schema` removal above describes.
+
+- **`routeOnlyFields` in the schema drift guard.** The existing tests catch a
+  flag for a field that does not exist. They cannot catch a flag for a field
+  that *does* exist and that a tenant is not allowed to write — the write simply
+  404s at the rule layer, on a command that looks like every other update. That
+  is the shape `nebula_ca.rotate` would take if anyone added a `--rotate` flag,
+  and the `nats_accounts` signing-key triggers have been in that position all
+  along with only a prose note guarding them.
+
 - **`--code` on `organization`,** and `code` in its `ls` columns. The one
   globally unique identifier in the ecosystem: derived from the name when
   omitted, immutable once set, and baked into signed NATS account JWTs and
@@ -36,6 +92,12 @@ that period, and this file starts where the versioned releases do.
 
 ### Fixed
 
+- **Every error the CLI printed ended in `(0)`.** PocketBase sends the status as
+  `status`; `PBError` only read `code`, which nothing populates. So the number
+  in every error message was zero, and the distinction that matters most —
+  a 400 from a validator versus a 404 from an update rule — was invisible. Both
+  names are read now, with the HTTP status as a fallback.
+
 - **Refreshed the vendored platform schema** (`cmd/testdata/schema.json`), which
   is what the drift tests check the field table against. It was three platform
   changes behind. Worth recording that the guard did its job unprompted: told

CLAUDE.md

@@ -73,6 +73,13 @@ intentional — do not "fix" them:
 - **`nebula-ca.rotate_keys`** — no such field exists, in `schema.json` or in
   pb-nebula. It was a flag once and did nothing: PocketBase silently drops
   writes to fields a collection doesn't have, so it reported success every time.
+- **`nebula-ca.rotate`** — this one is real (pb-nebula v0.3.0), and is still not
+  a flag: `nebula_ca.updateRule` is operator-only, so a tenant PATCH 404s. The
+  three steps live behind `stone nebula ca-rotate`. Same for the material a
+  rotation produces — `next_certificate`, `previous_certificate`, `rotated_at`.
+  `routeOnlyFields` in `cmd/schema_drift_test.go` is what stops any of them
+  becoming a flag later; it covers the `nats_accounts` triggers too, which were
+  previously only prose.
 - **`nats-account.rotate_keys` / `add_signing_key` / `remove_signing_key`** —
   the fields exist, but `nats_accounts.updateRule` is operator-only, so a tenant
   PATCH 404s. They live behind `stone nats account-keys` instead.
@@ -89,13 +96,42 @@ platform repo is normally at `../platform`.
 
 ### Custom platform routes
 
-`internal/pb/client.go` has `CallRoute` for the platform's non-collection
-endpoints, and `cmd/creds.go` wraps the two that exist:
-`POST /api/me/nats-creds/rotate` and `POST /api/org/nats-account/keys`. Both
-exist server-side because an API rule cannot express a single-field allowlist.
-Neither takes a record id — the target is derived from the caller's identity or
+`internal/pb/client.go` has `CallRoute` (POST) and `GetRoute` (GET) for the
+platform's non-collection endpoints. Four are wrapped:
+
+- `cmd/creds.go` — `POST /api/me/nats-creds/rotate` and
+  `POST /api/org/nats-account/keys`.
+- `cmd/nebula.go` — `POST /api/org/nebula-ca/rotate` and
+  `GET /api/org/nebula/cert-audit`.
+
+Three of the four exist because an API rule cannot express a single-field
+allowlist: permitting one trigger field through an update rule means a deny-list
+over every other, which silently opens the moment someone adds a field. None of
+them takes a record id — the target is derived from the caller's identity or
 active organization — so don't add one.
 
+`cert-audit` is the exception and exists for a different reason: deciding whether
+a host certificate still matches its network means parsing a Nebula certificate,
+which no client can do. It answers in **ids**, and `cmd/nebula.go` resolves them
+to hostnames before printing — falling back to the bare id rather than dropping
+the row, because a host we cannot name is still a host that needs re-signing.
+
+### pb-nebula v0.3 and the library-owned fields
+
+pb-nebula v0.3.0 added twelve fields the platform's `schema.json` does not
+declare — the dump predates the release. Unlike the pb-nats case noted in
+`cmd/schema_drift_test.go`, pb-nebula **migrates** them (`addMissingFields`), so
+a database created earlier acquires them on the next start rather than never.
+They are listed in `libraryOwnedFields` so the drift guard can see them.
+
+**The floor is pb-nebula v0.3.0** (the platform pins v0.3.2). Against v0.2.0 the
+two Nebula routes 404, and `--is-relay`, `--unsafe-networks`, `--unsafe-routes`,
+`--preferred-ranges`, `--mtu`, `--tun-device` and `--renew` all name fields the
+collection does not have — so the write is discarded and the command reports
+success. That is precisely the failure the drift guard exists to make loud, and
+it cannot catch this one, because the vendored schema is a copy of a file that
+never declared them either.
+
 ### Pull / apply (GitOps)
 `cmd/sync.go`:
 - `stone pull` writes one YAML file per record into `<workspace>/<collection>/<key>.yaml`, where `<key>` is the spec's `LookupKey` value, falling back to `name`, then id. Filename collisions get a `-<id>` suffix; records are pulled sorted by id so the suffix lands on the same record across pulls. Filenames are cosmetic — apply identifies records solely by the `id` field inside the file. Org-scoped collections are filtered by `current_organization`. Server-only fields (`collectionId`, `collectionName`, `created`, `updated`, `expand`) are stripped on read (see `pb.ServerOnlyFields` / `pb.Strip`).

README.md

@@ -322,6 +322,56 @@ All KV operations — bucket lifecycle and per-key data — live under `stone kv
 ./stone kv watch twins
 ```
 
+## Nebula
+
+The records are ordinary entities — `stone nebula-ca`, `stone nebula-network`,
+`stone nebula-host`. Two operations are not record writes, and live under
+`stone nebula`:
+
+```sh
+stone nebula cert-audit                 # hosts whose certificate no longer matches their network
+stone nebula ca-rotate prepare          # publish trust in a new CA (reversible)
+stone nebula ca-rotate commit           # switch issuance, re-sign every active host
+stone nebula ca-rotate finish           # drop the outgoing CA
+```
+
+**`ca-rotate` takes three steps and the wait between them is the point.** Nebula
+verification is mutual — each peer checks the other against its *own* local CA
+pool, with no chain and no fallback — and hosts pull their config whenever they
+like. So one write carrying both the new trust bundle and the new certificate
+splits the mesh: a host that has fetched presents a new-CA certificate to one
+that has not, and the handshake fails in *both* directions until propagation
+finishes. `prepare` publishes trust and moves no issuance, so it is fully
+reversible. `commit` switches issuance and re-signs every active host, with both
+CAs trusted throughout. `finish` drops the outgoing CA and is refused while any
+active host still holds a certificate signed by it — the refusal names the host.
+
+A CA cannot be renewed, only rotated, so start months ahead of the expiry in
+`stone nebula-ca ls`, not weeks.
+
+**`cert-audit` answers a question no client can.** pb-nebula signed host
+certificates at `/32` until v0.3.0. Nebula puts a certificate's network straight
+onto the tun device and installs a link route for it, so the mask in the
+certificate *is* the host's route to the overlay — a `/32` gives a host a route
+covering only itself. The certificate verifies, the config renders, the host
+starts, the handshake completes, and no packet crosses the mesh. Nothing errors,
+which is why you have to ask. Editing a host's `overlay_ip` after issue lands it
+here too.
+
+Nothing is re-signed automatically: re-signing moves a fingerprint, and a
+fingerprint is what the revocation blocklist matches, so a sweep would rewrite
+every peer config in the mesh. Fix one host at a time, redeploying as you go:
+
+```sh
+stone nebula-host update edge-west --renew
+```
+
+Requires a platform on **pb-nebula v0.3.0 or newer**. Against v0.2.0 the routes
+404 and the newer host flags — `--is-relay`, `--unsafe-networks`,
+`--unsafe-routes`, `--preferred-ranges`, `--mtu`, `--tun-device`, `--renew` —
+name fields the collection does not have, so PocketBase discards the write and
+the command reports success.
+
 ## Limitations
 
 - Relation flags (`--type`, `--location`, …) take 15-char PocketBase ids only —
@@ -332,9 +382,8 @@ All KV operations — bucket lifecycle and per-key data — live under `stone kv
 - No JetStream consumer management — the `nats` CLI is better at that.
 - `nats-account` and `nebula-ca` are **operator-only for every field**. Both
   `updateRule`s admit no tenant role, so an owner/admin PATCH of any field on
-  either collection returns 404. The three legitimate tenant key operations live
-  behind `stone nats account-keys` instead. `nebula_ca` has no rotation trigger
-  at all — rolling a CA is an operator action.
+  either collection returns 404. The legitimate tenant operations live behind
+  routes instead: `stone nats account-keys` and `stone nebula ca-rotate`.
 - Locations' `floorplan` and organizations' `logo` are file fields; the CLI has
   no upload path for them. Use the console.