diff --git a/docs/agent.md b/docs/agent.md index 45def47..a3ad7bf 100644 --- a/docs/agent.md +++ b/docs/agent.md @@ -1,6 +1,6 @@ # The Agent -The **Stone-Age.io Agent** is a lightweight, NATS-native management and observability daemon designed to run on Windows, Linux, and FreeBSD. It acts as a resilient, outbound-only executor that connects your physical hardware to the Data Plane. +The **Stone-Age.io Agent** is a lightweight, NATS-native management and observability daemon designed to run on Windows, Linux, and FreeBSD. It connects your physical hardware to the Data Plane, and it takes its instructions only over a connection it dialed itself — there is no inbound management API to expose, forward a port to, or firewall. The Agent is what turns a bare server or IoT gateway into a participant in the Data Plane. Once connected, it publishes telemetry that Layer 1 rules can react to, Layer 2 stream processors can aggregate, and Layer 3 tools can archive. See [Platform Layers](./platform-layers.md) for the complete picture. @@ -11,12 +11,12 @@ The Agent is what turns a bare server or IoT gateway into a participant in the D The agent is a single Go binary with zero external dependencies (other than the optional Prometheus exporters). Its design philosophy is simple: **Stay invisible until needed.** - **Lightweight:** Consumes < 50MB of RAM and negligible CPU. -- **Secure:** No listening ports. It initiates all connections outbound via NATS. +- **Secure:** No inbound management API — every instruction arrives over the authenticated NATS connection the Agent dialed outbound, so no port forward is ever needed to manage a device. (It is not, however, "no listening ports": `/ready` and `/metrics` bind `127.0.0.1:9100` unless you set `observability.addr` empty, and a site gateway hosts a NATS server that local devices connect *in* to. See [§6](#6-what-listens-and-what-dials-out).) - **Resilient:** Automatically handles NATS reconnections and backoffs. - **Cross-Platform:** First-class support for Windows Services, Linux systemd, and FreeBSD rc.d. -!!! note "Not to be confused with `leaf-sync`" - The Agent is a **per-Thing** executor — it manages one device or server. `leaf-sync` is a **per-site** config-mirroring agent that bootstraps a NATS leaf node and syncs an org's configuration into local KV. A site often runs both. See [Leaf Nodes](./leaf-nodes.md). +!!! note "It absorbed `leaf-sync`" + There used to be a second binary — `leaf-sync` — that bootstrapped a site's NATS leaf node and mirrored an organization's configuration into local KV. It is gone. Its leaf-node duties moved into the Agent, and the config mirror was dropped rather than moved, because nothing ever read the mirrored rows. A site that runs a leaf node runs **one** binary now, not two. See [Leaf Nodes](./leaf-nodes.md). ### Getting the binary @@ -40,7 +40,9 @@ The per-platform guides in the agent repository (`docs/linux.md`, `docs/windows. The automated provisioning flow is what makes the Agent practical at MSP scale. Instead of manually copying credential files to every device, the Agent authenticates to the Control Plane **as its own Thing** and fetches its NATS credentials itself — then keeps them current for the life of the device. -This is `auth.type: "stone-age"` in the Agent's config. It is deliberately named after the platform rather than after PocketBase: the Agent depends on the Stone-Age.io schema (`things` → `nats_user` → `creds_file`) and on a route the platform defines itself, not on anything generic. +This is `auth.type: "platform"` in the Agent's config, with the platform itself configured in a **top-level `platform:` block**. It is deliberately named after the platform rather than after PocketBase: the Agent depends on the Stone-Age.io schema (`things` → `nats_user` → `creds_file`) and on routes the platform defines itself, not on anything generic. + +The block is top-level rather than nested under `nats.auth` because **three** subsystems read it: the NATS credential lifecycle below, the [Nebula](#4-security-isolation) config source, and the [leaf bootstrap](./leaf-nodes.md#3-get-apimeleaf-config). With it buried under one of them, the other two had to ask "is some other section's type field set to a particular string" instead of "is the block present". ### The Lifecycle of a "Thing": @@ -49,9 +51,9 @@ This is `auth.type: "stone-age"` in the Agent's config. It is deliberately named 3. **Install:** The Agent is installed on the edge device with its `code`, the platform URL, the Thing's login email, and that password in an environment variable — never in the config file. 4. **Bootstrap (first start only):** The Agent authenticates as the Thing against `POST /api/collections/things/auth-with-password?expand=nats_user,location`, verifies the returned record's `code` matches its own config, and writes the `.creds` file from `expand.nats_user.creds_file` with `0600` permissions. It also stores the session token and the credential's revision. 5. **Operation:** The Agent connects to NATS with that `.creds` file and begins publishing. -6. **Upkeep (every 24h by default):** The Agent renews its platform session and adopts a credential the platform has re-minted. Once it holds a session token the Thing's password is optional and can be removed from the device — see §2.2. +6. **Upkeep (`platform.sync_interval`, 24h by default):** The Agent renews its platform session and adopts a credential the platform has re-minted. Once it holds a session token the Thing's password is optional and can be removed from the device — see §2.2. -The Agent fetches **only** its NATS credentials. It does not fetch a NATS URL (`nats.urls` is local config), and it neither fetches nor manages Nebula — a Thing's Nebula config is downloaded from the Console and installed separately. +The Agent does not fetch a NATS URL — `nats.urls` is local config, because what a box can reach is a property of where the box is. With `nebula.enabled` it fetches its **Nebula** config the same way, from the `nebula_host` linked to the same Thing, and re-reads it on `nebula.sync_interval`. That interval is the device's **revocation latency**, not a tuning knob: Nebula has no CRL, so a revoked certificate is only refused once each peer has re-read its own config.
```mermaid @@ -135,6 +137,18 @@ The Agent publishes on a schedule and answers commands on request. Everything it Every telemetry payload carries `code`, `location`, and `ts`, so a message is self-describing to any direct subscriber. +Beyond the bus, the Agent has three capabilities that apply on a box which is also a **site gateway**. Each is independently switchable and none of them is a mode: + +| Capability | Turned on by | What it does | +|---|---|---| +| Leaf bootstrap | `agent -leaf-config` | Writes `nats-leaf.conf` + creds from `GET /api/me/leaf-config` | +| Embedded NATS | `nats.server_config` | Hosts that leaf server in this process | +| Digital twin sync | `twin.enabled` | Relays reported state up, mirrors desired state down | + +There is **no `edge.enabled` key and no gateway flag on the platform**: "gateway" is not a mode the config declares, it is the sum of the capabilities it turns on. A single flag naming the role would be a second control that can disagree with the first. See [Leaf Nodes](./leaf-nodes.md). + +**Local health is not one of them**, though it is often listed beside them. `observability.addr` serves `/ready` and `/metrics` on **every** Agent, defaulted to `127.0.0.1:9100`, because the reason to answer locally has nothing to do with running a leaf node: `cmd.health` travels over NATS, which is the link that breaks, so the box you most want to ask is the one that has just gone quiet. Nebula is the same shape — `nebula.enabled` is a per-device capability, not a gateway one. + !!! note "Heartbeats are deliberately not JetStream" A missed beat is the signal consumers care about, so last-write-wins is the correct semantic — a backlog of stale beats replayed after a reconnect would be actively misleading. This is why the server-side stream must bind `{prefix}.*.telemetry.>` rather than `{prefix}.>`: the heartbeat stays outside the stream by subject construction. @@ -164,7 +178,7 @@ For custom logic, the Agent can execute local scripts or shell commands. ### D. Credential Upkeep -- **`creds_sync` task:** Renews the platform session and adopts a re-minted credential (§2.1). +- **Credential sync:** Renews the platform session and adopts a re-minted credential (§2.1), on `platform.sync_interval`. - **`cmd.rotate_creds`:** Asks the platform to re-mint this Agent's credential, writes it, replies, then reconnects — so a rotation costs no downtime and no site visit. The response reports `changed: false` if the platform handed back the credential the Agent already had. An Agent that is not platform-managed answers with an error rather than timing out. --- @@ -175,7 +189,7 @@ Three mechanisms enforce cryptographic isolation at the edge: - **NKey Authentication:** The Agent signs every NATS connection challenge locally with the nkey seed in its `.creds` file. Be precise about where that key comes from, though: the Control Plane mints the keypair (`pb-nats` generates it and embeds the seed in `creds_file`), so the private key originates on the platform and is *delivered* to the device — it is not generated there. That is exactly why the credential lifecycle in §2 is built the way it is: HTTPS is required, the key is re-transmitted only when it has actually changed, it is written `0600` through an atomic replace, and platform response bodies are never logged. - **Sandboxed Logic:** The Agent does not have "God Mode." Its permissions are restricted by the **NATS Role** assigned to it in the Control Plane. If an Agent is only meant to report temperature, its NATS credentials will physically prevent it from sending a "Restart Server" command. Assigning or changing that role is an **Owner/Admin** action — the `nats_roles` and `nats_users` collections are closed to every role below admin for reads as well as writes, precisely because a role's permission fields are copied verbatim into the JWT the platform signs. -- **Nebula Encryption:** Administrative traffic between your workstation and the host (SSH, for instance) can be encrypted end-to-end via the Nebula mesh, bypassing the public internet entirely. Note this is a property of the *host*, not of the Agent: the Agent does not manage the Nebula interface, and its own NATS traffic is outbound and TLS-protected regardless. +- **Nebula Encryption:** Administrative traffic between your workstation and the host (SSH, for instance) can be encrypted end-to-end via the Nebula mesh, bypassing the public internet entirely. With `nebula.enabled` the Agent runs that mesh host **in-process** and keeps its config current, which is what makes revocation, renewal and CA rotation actually reach the device. A newly applied config that cannot reach a lighthouse is restarted and then rolled back to the last one known to have worked, so a bad config cannot take a fleet off the network. The Agent's own NATS traffic is outbound and TLS-protected either way — the overlay is for everything else. --- @@ -189,15 +203,17 @@ code: "chicago-warehouse-vent-01" # identity token used in NATS subjects location: "chicago-warehouse" # optional, carried in every payload subject_prefix: "agents" +platform: # one home for the platform relationship + url: "https://platform.acme.io" + identity: "chicago-warehouse-vent-01@things.acme.io" + password_env: "AGENT_PLATFORM_PASSWORD" # optional after first boot + sync_interval: "24h" # renews the 7-day platform session + nats: urls: ["nats://nats.acme.io:4222"] auth: - type: "stone-age" + type: "platform" creds_file: "/etc/agent/device.creds" - stone-age: - url: "https://platform.acme.io" - identity: "chicago-warehouse-vent-01@things.acme.io" - password_env: "AGENT_PLATFORM_PASSWORD" # optional after first boot tasks: heartbeat: @@ -210,9 +226,6 @@ tasks: inventory: enabled: true interval: "24h" - creds_sync: - enabled: true - interval: "24h" # renews the 7-day platform session commands: scripts_directory: "/opt/stone-age/scripts" @@ -221,6 +234,46 @@ commands: - "uptime" ``` +A site gateway adds `nats.server_config` and/or `twin.enabled` to the same file — see [Leaf Nodes §5](./leaf-nodes.md#5-deploy-flow). `observability` and `nebula` are not gateway keys and may be set on any device. + Full per-platform installation guides, including how to set `AGENT_PLATFORM_PASSWORD` under systemd, Windows Services, and rc.d, ship in the Agent repository (`docs/credentials.md`). +--- + +## 6. What listens, and what dials out + +This is the table to hand whoever writes the firewall rules. It is also the +correction to a claim that stood in these docs for a long time: the Agent used to +have no listening sockets at all, and that stopped being true when local +readiness and the embedded NATS server arrived. + +| | Direction | Port | When | +|---|---|---|---| +| NATS — telemetry, commands, heartbeats | **outbound** | 4222 (or your hub's) | always | +| Platform HTTPS — credentials, Nebula config, leaf bootstrap | **outbound** | 443 | when the `platform:` block is set | +| Nebula overlay | **outbound** UDP to a lighthouse | the lighthouse's configured port, commonly 4242 | when `nebula.enabled` | +| `/ready` + `/metrics` | **listens** | `127.0.0.1:9100` | unless `observability.addr` is empty | +| Embedded `nats-server` | **listens** | per its own config | only when `nats.server_config` is set | + +The property worth relying on is the first row's direction, not a count of open +sockets: **nothing can instruct an Agent except over the NATS connection it +dialed itself.** A device on a customer network needs no inbound rule and no port +forward to be managed, which is what makes a fleet behind NAT or CGNAT tractable. + +An ordinary Agent binds an **ephemeral** local UDP port for the overlay — `pb-nebula` writes `listen.port: 0` for anything that is not a lighthouse or a relay, since only those two are reached at a fixed address through `static_host_map`. So there is nothing to open inbound for Nebula either. + +Two footnotes that matter in practice: + +- **Loopback is not an authorization boundary** when other users share the box. + `/metrics` carries no per-organization labels by design, but it does carry this + device's health. Set `observability.metrics_token` (accepted as Bearer or as + Basic with any username) before moving `addr` off loopback, or set `addr` empty + and let `cmd.health` over NATS be the only answer. +- **9100 is also node_exporter's default port** on Linux and FreeBSD. If you run + the exporter on the same box — which [§3](#3-capabilities) offers as an + alternative metrics source — move one of the two, or neither will bind + reliably. Windows is unaffected: `windows_exporter` uses 9182. + +--- + The Stone Age Agent turns a raw server or IoT gateway into a managed entity that is secure by default and easy to operate at scale — a first-class participant in the layered platform rather than a bolted-on endpoint. diff --git a/docs/authorization.md b/docs/authorization.md index dc10460..88ae583 100644 --- a/docs/authorization.md +++ b/docs/authorization.md @@ -45,8 +45,8 @@ The authoritative summary. "—" means the API rules reject the operation, not t | Read Things and Locations | ✅ | ✅ | ✅ | ✅ | ✅³ | — | ✅ | | Create / edit Things and Locations | ✅ | ✅ | ✅ | — | — | — | ✅ | | Delete a Thing or Location | ✅ | ✅ | — | — | — | — | ✅ | -| Deactivate / reactivate a Thing or Leaf Node (§4.2) | ✅ | ✅ | — | — | — | — | ✅ | -| Reset a Thing's or Leaf Node's PocketBase password | ✅ | ✅ | — | — | — | — | ✅ | +| Deactivate / reactivate a Thing (§4.2) | ✅ | ✅ | — | — | — | — | ✅ | +| Reset a Thing's PocketBase password | ✅ | ✅ | — | — | — | — | ✅ | | Attach a NATS user / Nebula host to a Thing | ✅ | ✅ | — | — | — | — | ✅ | | Read Thing Types, Operations | ✅ | ✅ | ✅ | ✅ | ✅³ | — | ✅ | | Manage Thing Types, Operations | ✅ | ✅ | — | — | — | — | ✅ | @@ -60,8 +60,6 @@ The authoritative summary. "—" means the API rules reject the operation, not t | Read own linked NATS identity | ✅ | ✅ | ✅ | ✅ | ✅ | — | ✅ | | Rotate own NATS credential (§4) | ✅ | ✅ | ✅ | ✅ | ✅ | — | ✅ | | Revoke someone else's NATS credential | ✅ | ✅ | — | — | — | — | ✅ | -| Read the Leaf Nodes list | ✅ | ✅ | ✅³ | ✅³ | ✅³ | — | ✅ | -| Create / edit / delete Leaf Nodes, reset their credentials | ✅ | ✅ | — | — | — | — | ✅ | | Manage JetStream streams and KV buckets⁴ | ✅ | ✅ | — | — | — | — | — | | Use dashboards | ✅ | ✅ | ✅ | ✅ | ✅⁷ | — | ✅ | | Invite users, manage memberships | ✅ | ✅ | — | — | — | invites only | ✅ | @@ -72,7 +70,7 @@ The authoritative summary. "—" means the API rules reject the operation, not t ¹ **Platform Operator.** `users.is_operator = true` — a flag on the user account, independent of any Membership. See §3. ² The `_superusers` collection bypasses API rules entirely. See §3. -³ **Reads are org-scoped, not role-scoped, and that is deliberate.** The read rules on `things`, `locations`, `thing_types`, `location_types`, `thing_type_operations` and `leaf_nodes` are all `organization = current_organization` with no role branch, so *every* role in an organization — `dashboard` included — can `curl` the whole inventory. What differs between roles is writes, plus which screens the console navigates to: it confines `dashboard` to the Visualizer, and it lists Leaf Nodes to owners and admins only. **That is navigation, not a boundary** — do not read a hidden screen as a denied read. Making one of these an actual boundary means a role branch in `schema.json`, across every one of those collections, with a new failure mode where a relation expansion silently returns nothing. +³ **Reads are org-scoped, not role-scoped, and that is deliberate.** The read rules on `things`, `locations`, `thing_types`, `location_types`, `thing_type_operations` are all `organization = current_organization` with no role branch, so *every* role in an organization — `dashboard` included — can `curl` the whole inventory. What differs between roles is writes, plus which screens the console navigates to: it confines `dashboard` to the Visualizer and shows no inventory screens at all. **That is navigation, not a boundary** — do not read a hidden screen as a denied read. Making one of these an actual boundary means a role branch in `schema.json`, across every one of those collections, with a new failure mode where a relation expansion silently returns nothing. ⁴ JetStream operations run over the browser's own NATS connection, so they are bounded by the caller's **NATS** permissions, not by PocketBase API rules. The console surfaces the views to owners and admins. ⁵ `organizations.deleteRule` keys on the `organizations.owner` **field** — the user recorded as the org's owner, normally the same person who holds the `owner` membership — rather than on the membership role itself. Creating and *editing* the record are Platform-Operator-only; see §3. ⁶ Through `POST /api/org/nats-account/keys`, not by editing the record — `nats_accounts.updateRule` and `nebula_ca.updateRule` are both Platform-Operator-only. `nebula_ca` has no rotation trigger at all, so rolling a CA is a Platform Operator action. @@ -116,7 +114,7 @@ So there is exactly one exception to the owner/admin-only rule on `nats_users`: POST /api/me/nats-creds/rotate ``` -Available to **every role, including `dashboard`**, for callers in the `users`, `things`, and `leaf_nodes` collections. It takes **no id parameter** — it only ever targets the caller's own linked identity, so there is no other identity it could be aimed at. It responds with the identity's id; re-read that record to pick up the new credential. +Available to **every role, including `dashboard`**, for callers in the `users` and `things` collections. It takes **no id parameter** — it only ever targets the caller's own linked identity, so there is no other identity it could be aimed at. It responds with the identity's id; re-read that record to pick up the new credential. It exists as a route rather than a rule branch because **an API rule cannot express a single-field allowlist**. Permitting self-rotation through the update rule would mean asserting `:isset = false` on every *other* writable field — a deny-list that opens up silently the moment someone adds a field. And the field that must stay closed is consequential: `nats_users.publish_permissions` is copied **verbatim** into the JWT the platform signs, so write access to that collection is equivalent to granting NATS permissions. That is why it is owner/admin only. @@ -142,7 +140,7 @@ Like the credential route it takes no record id — the account is derived from ### 4.2 Taking a device out of service -`things.active` and `leaf_nodes.active` are Owner/Admin-only booleans. Clearing one is a **decommission**, not a label change — three things happen in the same operation: +`things.active` is an Owner/Admin-only boolean. Clearing one is a **decommission**, not a label change — three things happen in the same operation: | | What it stops | | :--- | :--- | @@ -152,11 +150,11 @@ Like the credential route it takes no record id — the account is derived from All three are needed, and the reason is worth internalizing before you rely on any similar flag: -> **An `authRule` is evaluated at the authentication endpoint only** — never on a request that arrives carrying an already-issued token. Thing and Leaf Node tokens live **7 days**. So `active = false` on its own would leave a decommissioned device with a working API session for up to a week. Refreshing `tokenKey` is what closes that window, and it is why deactivation is a server-side hook rather than a rule alone. +> **An `authRule` is evaluated at the authentication endpoint only** — never on a request that arrives carrying an already-issued token. Thing tokens live **7 days**. So `active = false` on its own would leave a decommissioned device with a working API session for up to a week. Refreshing `tokenKey` is what closes that window, and it is why deactivation is a server-side hook rather than a rule alone. And the second half: **a device's real capability is its NATS credential, not its PocketBase session.** Blocking API access without revoking the credential leaves the device publishing to the bus. Deactivation does both. -**Reactivating issues a *fresh* NATS credential.** The revocation cutoff embedded in the account JWT is permanent, so the old `.creds` file stays rejected forever — the device needs the new one. For a Thing running the Agent that happens on the next sync; for a Leaf Node, re-run `leaf-sync config`. +**Reactivating issues a *fresh* NATS credential.** The revocation cutoff embedded in the account JWT is permanent, so the old `.creds` file stays rejected forever — the device needs the new one. For a Thing running the Agent that happens on the next sync; on a site gateway, re-run `agent -leaf-config` as well, since the generated config embeds the creds file beside it. > **A flag with nothing enforcing it is worse than no flag**, because someone will trust it during an incident. `nats_users.active` is the cautionary case: `pb-nats` reads it into its model and then consults it nowhere in JWT generation — only `revoke` disconnects anyone. The console therefore no longer exposes it as an editable control; **Revoke** and **Re-enable** on the NATS user's detail view are the real operations. @@ -170,24 +168,28 @@ The practical consequence for MSP deployments: **a tenant admin cannot self-serv --- -## 6. Leaf Nodes and the Edge +## 6. Gateways and the Edge -A Leaf Node authenticates as a record in the `leaf_nodes` collection — "a special Thing" with one NATS identity. Its read surface is deliberately narrow. +**A site gateway is a Thing.** There is no `leaf_nodes` collection any more and no gateway flag — a Thing's [Thing Type](./thing-types.md) already says what it is, and a second marker is a second thing to get wrong. So a gateway's read surface is exactly a Thing's read surface: its own record, the org-scoped inventory collections, and the one NATS identity linked to it. Nothing in `nats_*` or `nebula_*` beyond that. -A leaf-node identity **can** read: - -- its own `leaf_nodes` record, and -- the allowlisted collections it mirrors, within its own organization: `things`, `locations`, `thing_types`, `location_types`, `thing_type_operations`. - -A leaf-node identity reads **nothing** in any `nats_*` or `nebula_*` collection. The four values an edge box cannot derive locally — its own creds, the org's account JWT and public key, and the NATS Operator JWT — come from a dedicated, leaf-node-authenticated route: +The values an edge box cannot derive locally — the org's account JWT and public key, the NATS Operator JWT, and the `$SYS` account JWT and public key — come from a dedicated route: ``` -GET /api/leaf/bootstrap +GET /api/me/leaf-config ``` -It returns eight named fields: `domain`, `code`, `creds`, `account_jwt`, `account_pub`, `operator_jwt`, `sys_account_jwt`, `sys_account_pub`. The server reads the secret-bearing collections with its own privileges and serves named fields, never whole records — so **secret-bearing collections are never exposed to a leaf-node identity**, and the blast radius of a leaked edge credential is those eight values regardless of how the collection rules later evolve. The last two are there because the NATS Operator JWT names a system account and the leaf’s `resolver: MEMORY` has nowhere to fetch it: without the `$SYS` **account** JWT preloaded, `nats-server` dies with `error resolving system account` before JetStream starts. Preloading it grants nothing — connecting as `$SYS` needs a `$SYS` **user** credential, which is never served. `GET /api/leaf/operator-jwt` still exists for older agents but is superseded by `/api/leaf/bootstrap`. +It is bound to `things` and takes **no record id**: the target is the caller's own authenticated record, exactly like `POST /api/me/nats-creds/rotate`. It returns ten named fields: `code`, `domain`, `creds`, `account_jwt`, `account_pub`, `operator_jwt`, `sys_account_jwt`, `sys_account_pub`, `hub_leaf_url`, `hub_domain`. The server reads the secret-bearing collections with its own privileges and serves named fields, never whole records — so **secret-bearing collections are never exposed to a device identity**, and the blast radius of a leaked edge credential is those ten values regardless of how the collection rules later evolve. + +**The route gates on nothing, and that is the interesting part.** Everything it serves is either public trust material — the Operator, account and `$SYS` account JWTs, which every server in the network validates anyway — or the caller's own credential, which it must already hold in order to connect at all. A Thing that will never run a leaf node can call it and learns nothing it could not already read. Adding a permission here would be a gate over data that is not secret, and it would have required inventing the marker field that §1 of [Leaf Nodes](./leaf-nodes.md) explains the absence of. + +!!! note "Why the `$SYS` **account** JWT is safe to serve" + The NATS Operator JWT names a system account, and the leaf's `resolver: MEMORY` has nowhere to fetch it — so without the `$SYS` account JWT preloaded, `nats-server` dies with `error resolving system account` before JetStream starts. Preloading it grants nothing. Connecting **as** `$SYS` needs a `$SYS` **user** credential, which the platform never serves to anything. Those are different objects. + +Account **seeds** and signing keys are never served on this path, and `nats_system_operator` stays superuser-only. + +**Don't re-add a device read branch** to `nats_users` or `nats_accounts` to make some edge feature work — extend the route instead. The point of the route is that the edge's blast radius is a fixed list of named fields rather than a consequence of rules that change for unrelated reasons. -Managing Leaf Node records — creating, editing, deleting, and resetting their credentials via the collection's `manageRule` — is an owner/admin action. The read rule admits any role in the organization, though the console lists them to owners and admins only — see footnote 3 in §2. See [Leaf Nodes](./leaf-nodes.md). +Taking a site out of service is `active` on its Thing (§4.2), like any other device. --- diff --git a/docs/configuration.md b/docs/configuration.md index d6d21b4..67d3f0c 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -222,19 +222,29 @@ default. See [Health & Metrics](./health-metrics.md) for the checks, the metric names and the alert expressions. -### `observability` (leaf-sync only) +### `observability` (the Agent, not this file) -The edge agent's own `/ready` and `/metrics`, in `leaf-sync.yaml`. **Off by default.** +The edge agent's own `/ready` and `/metrics`, in the **Agent's** `config.yaml` — +not in the Control Plane's. **On by default, on loopback.** | Key | Type | Default | Purpose | |---|---|---|---| -| `addr` | string | `""` | Listen address, e.g. `"127.0.0.1:9101"`. Empty creates no listener at all. | -| `metrics_token` | string | `""` | As `metrics.token` above. | +| `addr` | string | `"127.0.0.1:9100"` | Listen address. Set it empty to create no listener at all; the checks still run and still log either way. | +| `metrics_token` | string | `""` | As `metrics.token` above. Set this before moving `addr` off loopback. | | `interval` | string | `"15s"` | Probe interval. | -This is the only place per-site health is visible: the Control Plane cannot read -an organization's `leaf_status` heartbeats, because it holds no credential -inside that account. See [Leaf Nodes](./leaf-nodes.md). +Two things to know about that default. It is **not** a gateway setting — every +Agent serves these, because the reason to answer locally is that `cmd.health` +travels over NATS and NATS is the link that breaks. And **9100 is node_exporter's +own default port** on Linux and FreeBSD, so on a box running both, move one of +them. (`windows_exporter` uses 9182, so Windows is unaffected.) + +This is where per-site health is visible in detail, and it keeps answering with +the WAN down — which is exactly when you want it. The Control Plane cannot +report it: it holds the NATS operator and the `$SYS` account and has no +credential inside any organization's account. (Whether a site's leaf node is +*attached* is a separate, cheaper question, and the console answers that one by +asking the hub directly.) See [Leaf Nodes](./leaf-nodes.md). --- diff --git a/docs/connectivity.md b/docs/connectivity.md index ee83f23..c910306 100644 --- a/docs/connectivity.md +++ b/docs/connectivity.md @@ -55,7 +55,7 @@ For MSPs managing remote customer sites, **Leaf Nodes** are a game changer. A Le Leaf nodes enable **edge deployment of higher layers** too. A rule engine instance running alongside a leaf node continues to evaluate rules against locally-mirrored KV state during a WAN outage. A stream processor at the edge keeps producing aggregates. The whole layered architecture works offline at each site, with changes replicating bidirectionally when connectivity returns. -How the platform models a site and gets that locally-mirrored config to the edge — the `leaf_nodes` entity and the `leaf-sync` agent — is covered in [Leaf Nodes](./leaf-nodes.md). +How the platform models such a site — as an ordinary **Thing**, whose Agent bootstraps and optionally hosts the leaf server — is covered in [Leaf Nodes](./leaf-nodes.md). ### Cross-Account Subject Sharing (Imports & Exports) diff --git a/docs/decisions/0002-organization-code-namespace.md b/docs/decisions/0002-organization-code-namespace.md index 06ba380..61ef69f 100644 --- a/docs/decisions/0002-organization-code-namespace.md +++ b/docs/decisions/0002-organization-code-namespace.md @@ -7,13 +7,15 @@ > The Context and Decision below describe the design as it was argued. All six steps have since shipped; see [As implemented](#as-implemented) for what changed on contact with the code — including one ordering error in this document that would have dropped events. +> **One collection named below no longer exists.** `leaf_nodes` was the precedent this ADR cites for freezing `code`, and it has since been dropped — an edge site is an ordinary Thing now (see [Leaf Nodes](../leaf-nodes.md)). The argument is unaffected: the freeze shipped on every collection listed, `things` included, and `things.code` is the precedent to point at today. The references are left as written, because an ADR records what was argued at the time. + --- ## Context An object in the real world — a door, a card reader, a site — is named several times over. It has a PocketBase id in the platform. It has a different PocketBase id in the helpdesk. It has a `code` that both agree on. It has a NATS subject it publishes under, a KV key its twin lives at, and, if someone screws a label to it, a string on a sticker. -Most of those already agree. `stone` resolves records by code, `leaf-sync` keys KV by the handle `candidateKey` derives, twin keys are `..`, and the helpdesk's machine intakes resolve a payload `thing_code` / `location_code` per `(customer, code)`. The ecosystem federates on `(organization, code)` and has for a while. +Most of those already agree. `stone` resolves records by code, the edge keys KV by the handle `candidateKey` derives, twin keys are `..`, and the helpdesk's machine intakes resolve a payload `thing_code` / `location_code` per `(customer, code)`. The ecosystem federates on `(organization, code)` and has for a while. Two things spoil it. diff --git a/docs/getting-started.md b/docs/getting-started.md index 763460b..d5f3bcb 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -80,9 +80,10 @@ container cannot work out for itself — use the host's real name rather than ### Pre-compiled binary Download for your architecture from the -[Releases page](https://github.com/stone-age-io/platform/releases). Two binaries -are published per platform: `stone-age` (the Control Plane) and `leaf-sync` (the -[edge agent](./leaf-nodes.md), which belongs on edge hardware rather than here). +[Releases page](https://github.com/stone-age-io/platform/releases). One binary +per platform: `stone-age`, the Control Plane. The +[Agent](./agent.md) that belongs on edge hardware releases from +[its own repository](https://github.com/stone-age-io/agent) on its own tags. ### From source diff --git a/docs/health-metrics.md b/docs/health-metrics.md index a65c7ae..5e56255 100644 --- a/docs/health-metrics.md +++ b/docs/health-metrics.md @@ -28,7 +28,7 @@ None of that moves a liveness probe. So the checks here are deliberately about * **A check must be answerable first-hand by the process running it.** -This is the NATS account boundary restated. The Control Plane holds the NATS operator and the `$SYS` account, and it has **no user credential inside any organization's account** — so it cannot read an org's `twin` KV, and it cannot read the `leaf_status` heartbeats `leaf-sync` writes. The console can, because a browser connects as the logged-in user. `leaf-sync` can, because it runs inside the account. +This is the NATS account boundary restated. The Control Plane holds the NATS operator and the `$SYS` account, and it has **no user credential inside any organization's account** — so it cannot read an org's `twin` KV, and it cannot read anything a site reports about itself. The console can, because a browser connects as the logged-in user — which is also how it answers whether a site's leaf node is attached, by asking the bus over that same connection. The Agent can, because it runs inside the account. Do not "improve" a check by minting the platform a credential in a tenant's account. That turns a credential issuer into a data-plane participant in every tenant's bus, which is the one boundary the whole NATS design is built around. Per-site liveness is therefore **absent** from the Control Plane's list on purpose, and lives on the edge instead ([§5](#5-the-edge-agent)). @@ -197,11 +197,11 @@ Plus `stone_age_nats_*` when — and only when — the bus runs in-process via ` Three things to know before building a dashboard on these: -**`stone_age_records{collection="leaf_nodes"}` counts leaf nodes CONFIGURED.** It is not availability, and an alert on it can never fire. Per-site liveness is `leaf_sync_*` on the edge box ([§5](#5-the-edge-agent)). The same caution applies to `things`. +**`stone_age_records{collection="things"}` counts devices CONFIGURED.** It is not availability, and an alert on it can never fire. Per-site liveness is `agent_*` on the edge box ([§5](#5-the-edge-agent)); whether a site's leaf node is *attached* is answered by the console asking the hub, not by anything here ([Leaf Nodes §7](./leaf-nodes.md#7-is-the-site-up)). **There are no per-organization labels, anywhere.** `/metrics` is open by default, and with per-org data reduced to row counts a tenant label would be a customer name attached to an inventory count. That is also why `stone_age_certificate*` reports the *soonest per kind* rather than one series per certificate: a per-host series would need an identifying label to be useful, which is a per-tenant device inventory. -**A collector that fails emits nothing rather than zero.** Zero is a legitimate value here — "no leaf nodes configured" — so reporting it on failure would turn a broken query into a confident wrong answer, and an alert on `== 0` would fire for the wrong reason. The absent series plus a non-zero `stone_age_collector_errors` says what actually happened. +**A collector that fails emits nothing rather than zero.** Zero is a legitimate value here — "no Nebula hosts configured" — so reporting it on failure would turn a broken query into a confident wrong answer, and an alert on `== 0` would fire for the wrong reason. The absent series plus a non-zero `stone_age_collector_errors` says what actually happened. ### The `route` label is a pattern, never a path @@ -215,7 +215,7 @@ With `serve --nats`, the in-process bus's own counters are exported: `stone_age_ With an **external** NATS server these series are absent entirely rather than reported as zero — a quiet bus and an absent one should not look alike. Scrape an external server with `prometheus-nats-exporter`, which reads its monitoring port and reports far more than this ever could. -> `stone_age_nats_leafnode_connections` is a TCP session the server can see. That is not the same fact as a `leaf-sync` heartbeat, which says the edge agent is actually syncing. +> `stone_age_nats_leafnode_connections` counts leaf sessions **across every account**, so it is a capacity number rather than a per-tenant availability signal. A tenant asks about its own sites through `$SYS.REQ.ACCOUNT.PING.CONNZ`, which is scoped to its account — see [Leaf Nodes §7](./leaf-nodes.md#7-is-the-site-up). --- @@ -243,31 +243,35 @@ The check **warns and never fails**, for the same reason an islanded edge warns: ## 5. The Edge Agent -`leaf-sync` serves the same two endpoints from its own registry, under the `leaf_sync` namespace. **This is the only place per-site health is actually visible** — the Control Plane cannot see it, by design ([§1](#1-why-this-exists-at-all)). +The [Agent](./agent.md) serves the same two endpoints from its own registry, under the `agent` namespace. **This is where per-site health is actually visible in detail** — the Control Plane cannot see it, by design ([§1](#1-why-this-exists-at-all)) — and it keeps answering with the WAN down, which is exactly when you want it. `cmd.health` travels over NATS, the link that breaks; the box you most need to ask is the one that has just gone quiet. -It is **off by default**. Set `observability.addr` to enable it: +It is **on by default, on loopback** — `observability.addr` defaults to `127.0.0.1:9100`. Set it empty to serve neither endpoint: ```yaml observability: - addr: "127.0.0.1:9101" # empty (the default) = no listener at all - metrics_token: "" + addr: "127.0.0.1:9100" # the default; "" = no listener at all + metrics_token: "" # set this before moving addr off loopback interval: 15s ``` -Paths are `/ready` and `/metrics` — no `/api` prefix, since this is not the PocketBase router. +Paths are `/ready` and `/metrics` — no `/api` prefix, since this is not the PocketBase router. Empty `addr` serves neither, but the checks still run and still log; a bind failure is logged rather than fatal, because a monitoring port that cannot bind must not stop the agent doing its job. -| Check | Fails when | Notes | +!!! warning "9100 is also node_exporter's port" + On Linux and FreeBSD the default collides with `node_exporter`, which [§3 of the Agent guide](./agent.md#3-capabilities) offers as an alternative metrics source. On a box running both, move one of them — and since a bind failure is only logged, the symptom is a scrape target that quietly never came up rather than a crash. `windows_exporter` uses 9182, so Windows is unaffected. + +| Check | State when it trips | Notes | |---|---|---| -| `nats_local` | the local leaf node is not reachable | The bus the devices on this site actually use. | -| `sync_freshness` | the last sync cycle is too old | Warns before the first cycle completes; skipped when the leaf node has no syncable collections. | -| `sync_errors` | never — warns only | A collection failed to mirror. | -| `hub_uplink` | never — warns only | No outbound leaf connection to the hub. | +| `nats_local` | **fail** | The agent is not connected to the local leaf — the bus the devices on this site actually use. | +| `hub_uplink` | **warn** | No outbound leaf connection to the hub: this site is *islanded*. | -**An islanded edge warns, it does not fail.** Local NATS still works and devices keep running against the mirrored config — that autonomy is the entire reason a leaf node exists, so returning `503` would invert the design and have an orchestrator restart a site that is working exactly as intended. +**An islanded edge warns, it does not fail.** Local NATS still works and devices keep running — that autonomy is the entire reason a leaf node exists, so returning `503` would invert the design and have an orchestrator restart a site that is working exactly as intended. -Metrics: `leaf_sync_nats_connected`, `leaf_sync_nats_connections`, `leaf_sync_hub_uplink_connected`, `leaf_sync_cycles_total`, `leaf_sync_last_cycle_duration_seconds`, `leaf_sync_last_cycle_errors`, `leaf_sync_last_cycle_timestamp_seconds`, `leaf_sync_mirrored_records`, `leaf_sync_jetstream_bytes` — plus the same `leaf_sync_ready` / `_check_state` / `_check_timestamp_seconds` / `_build_info` set. +Metrics: `agent_edge_nats_connected`, `agent_edge_nats_connections`, `agent_edge_hub_uplink_connected`, `agent_edge_jetstream_bytes` — plus the same `agent_ready` / `_check_state` / `_check_timestamp_seconds` / `_build_info` set. -Server-derived series are **omitted** when the leaf's monitoring port is down, rather than reported as zeros. See [Leaf Nodes](./leaf-nodes.md). +The server-derived rows come from the leaf's own **loopback** monitoring port, which is how the edge reads its own server without ever holding a `$SYS` user credential. They are **omitted** when that port is unreachable rather than reported as zeros: zero would claim an islanded site with no devices, which is a far louder statement than "not scraped". The same rule governs the check registry, where `skipped` ranks *below* `ok` — a report that is entirely skipped must not read as a clean bill of health. See [Leaf Nodes](./leaf-nodes.md). + +!!! note "There is no longer a sync-freshness check" + Earlier versions of the edge agent mirrored an organization's config collections into local KV, and had `sync_freshness` and `sync_errors` checks over that loop. The mirror was removed — nothing consumed the mirrored rows — and those two checks went with it. --- @@ -287,6 +291,12 @@ scrape_configs: - targets: ["site-01:9101", "site-02:9101"] ``` +The edge targets are on **9101**, not the default 9100, for two reasons: it keeps +clear of `node_exporter`, and a scrapeable target means `addr` has been moved off +loopback anyway. Do that with `metrics_token` set — these endpoints carry no +per-organization labels, but they do carry a named device's health, and +`/metrics` is open when the token is empty. + The default `metrics_path` is `/metrics`, which is where both binaries serve. Point the same stack at these that you use for [Layer 3](./observability.md) — VictoriaMetrics speaks the Prometheus API, so this is a second `scrape_config`, not a second system. A starting set of alerts: @@ -298,8 +308,9 @@ A starting set of alerts: | Prober wedged | `time() - stone_age_check_timestamp_seconds > 120` | | Certificate expiring | `stone_age_certificate_expiry_seconds - time() < 30 * 86400` | | Database growth | `predict_linear(stone_age_database_size_bytes[6h], 7 * 86400) > ` | -| Edge gone quiet | `time() - leaf_sync_last_cycle_timestamp_seconds > 600` | -| Site islanded | `leaf_sync_hub_uplink_connected == 0` | +| Edge prober wedged | `time() - agent_check_timestamp_seconds > 120` | +| Site islanded | `agent_edge_hub_uplink_connected == 0` | +| Site bus down | `agent_edge_nats_connected == 0` | Note that a **warning** raises no alert here, and should not: `stone_age_ready` stays `1`. Warnings are for a human reading `/api/ready` after a deploy, or for a dashboard panel — not for a pager. diff --git a/docs/index.md b/docs/index.md index 75e2cc0..e3bff17 100644 --- a/docs/index.md +++ b/docs/index.md @@ -105,7 +105,7 @@ These pages, in order, walk through the platform from concept to deployment: 9. **[Thing Types](./thing-types.md)** — The contract layer: how participants on the fabric declare what they publish, subscribe to, request, and reply to. 10. **[Connectivity](./connectivity.md)** — Dive deep into NATS and Nebula configurations (Layer 0). 11. **[The Edge (Agent)](./agent.md)** — Provision and manage lightweight executors on remote hosts. -12. **[Leaf Nodes](./leaf-nodes.md)** — Model a site as a `leaf_nodes` record and mirror its config to the edge with `leaf-sync`. +12. **[Leaf Nodes](./leaf-nodes.md)** — Run a site's own NATS server: how a gateway Thing bootstraps its leaf config, and how you tell whether it is attached. 13. **[Automation](./automation.md)** — Build intelligent routing, scheduled publishing, and stateful alarms with the rule engine (Layer 1). 14. **[Stream Processing](./stream-processing.md)** — Windowed aggregations, joins, and anomaly detection (Layer 2). 15. **[Observability](./observability.md)** — Long-term data storage and historical analysis (Layer 3). diff --git a/docs/leaf-nodes.md b/docs/leaf-nodes.md index 1348bb3..b1916bf 100644 --- a/docs/leaf-nodes.md +++ b/docs/leaf-nodes.md @@ -1,221 +1,186 @@ # Leaf Nodes -A **Leaf Node** is how Stone-Age.io models a customer site that runs its own local NATS server. The transport underneath is a stock NATS leaf node ([Connectivity §1](./connectivity.md#leaf-nodes)) — it dials the hub outbound and gives the site local autonomy when the WAN drops. On its own, though, that leaf node is just an empty pipe: it has a local JetStream domain but knows nothing about *what's at the site* — which Things exist, what their Types can do, which Locations they live in. +A site that runs its own local NATS server keeps working when the WAN does not. The transport underneath is a stock NATS leaf node ([Connectivity §1](./connectivity.md#leaf-nodes)) — it dials the hub outbound and gives the site local autonomy during an outage. -What turns a bare NATS process into a Stone-Age.io **Leaf Node** is **`leaf-sync`**: a small, opinionated edge agent that runs at the site, authenticates as the site's own identity, and mirrors that organization's configuration into the leaf node's local KV. The platform tracks each site as a **`leaf_nodes`** record — "a special Thing" with one NATS identity — so a Leaf Node is a provisioned, RBAC-scoped, audited entity, not a hand-assembled pile of config files. - -This page covers both: the Leaf Node entity and the `leaf-sync` agent that brings it to life. +**The platform models that site as a Thing.** Not a special record type, not a separate collection: a Thing whose [Thing Type](./thing-types.md) says it is a gateway, and whose [Agent](./agent.md) happens to have its leaf-node capabilities turned on. One inventory, one identity model, one set of API rules. > ## "Leaf node" means two related things — keep them straight > | Term | What it is | > | :--- | :--- | > | **NATS leaf node** | The stock `nats-server` running at the site in leaf mode, dialing the hub outbound. A Layer 0 transport primitive — see [Connectivity](./connectivity.md#leaf-nodes). | -> | **Leaf Node** (this page) | The *platform's* model of the site: a PocketBase record with one NATS identity, an optional Nebula host, and an allowlist of config to mirror. Built **on** a NATS leaf node, but adds provisioning, RBAC, and config sync. This is what the UI and PocketBase call a "Leaf Node." | -> | **`leaf-sync`** | The binary that bootstraps the leaf node's config and continuously mirrors central config → local KV. The subject of this page. | -> | **The [Agent](./agent.md)** (`agent`) | A *per-Thing* executor (telemetry, service checks, remote exec). Different binary, different job. A site often runs both: `leaf-sync` keeps the site's config in sync; Agents report on individual devices. | +> | **A gateway** | The *platform's* model of such a site: an ordinary **Thing**, with a Thing Type that says gateway, one NATS identity, and optionally a Nebula host. This page is about how one gets configured and how you tell whether it is up. | +> | **The [Agent](./agent.md)** | The binary on the box. It manages the device *and*, when configured to, bootstraps and hosts the leaf node. One agent, not two. | --- -## 1. More than a NATS leaf node +## 1. There is no gateway flag -A vanilla NATS leaf node is unopinionated by design — it's transport, and nothing more. Stone-Age.io's Leaf Node is the opposite: it's a leaf node with a point of view, and `leaf-sync` is what gives it one. Four things make it *platform-specific* rather than a bare process: +This is the design decision the rest of the page follows from, so it is worth stating plainly: **nothing on the platform marks a Thing as a gateway.** -- **Pre-loaded config.** `leaf-sync` populates the local KV with the org's Thing contract graph and inventory (§4), so the site can resolve what its devices are and what they're allowed to do — offline, with no round-trip to the hub. -- **A managed identity.** The Leaf Node is provisioned with its own NATS user by a server-side hook. You never hand-mint keys; you create a record. -- **A bounded sync scope.** What a site mirrors is an enforced allowlist (§4), not "whatever's in PocketBase." Secret-bearing collections can never reach it. -- **The same control surface as everything else.** Rotating credentials or narrowing the site's NATS role are record edits in the console — not SSH sessions on the edge box. (Those edits *are* audited, but reading `audit_logs` is a Platform Operator action: no tenant role, not even `owner`, can read it — see [Authorization §5](./authorization.md#5-the-audit-log-is-platform-operator-only).) +A `leaf_nodes` collection used to exist — "a special Thing" with its own `domain` column, its own sync allowlist, and its own screens. It was removed, for three separate reasons that happened to arrive together: -The payoff: a Leaf Node *understands its own configuration*, where a plain leaf node only moves bytes. +- **`thing_types` already says what a device is.** A second marker is a second thing to get wrong, and two markers that disagree have no correct interpretation. +- **The `domain` column was a second copy of the code.** It could drift from the code it was derived from, and when it did, the symptom was a site that silently stopped appearing rather than an error. +- **Nothing consumed the mirrored config.** The old agent copied an organization's `things`, `locations` and type collections into the edge's local KV so devices could read them offline — but no rule, no firmware and no tool ever read those rows. It was the reason an edge identity needed read grants spread across the inventory, and it bought nothing. ---- +What is left is one route. Everything an agent needs to stand up a leaf server comes from `GET /api/me/leaf-config`, and **that route gates on nothing** — see §3. -## 2. Why the edge pulls its config +--- -There are two distinct sync planes at a site, and `leaf-sync` owns only one of them. +## 2. Why the edge pulls, rather than the hub pushing -- **Config plane** — PocketBase records → `leaf-sync` (HTTPS pull) → local KV. This exists because the Control Plane is the **NATS Operator** and only ever touches the **SYSTEM account**. It deliberately has no presence inside any tenant's account data plane (see [Architecture §2](./architecture.md#key-properties-of-this-topology)), so it *cannot* push config into an org's buckets. Instead, the site pulls its own config, authenticated as its own identity. -- **Data plane** — live data already in NATS (digital twin state, telemetry) replicates between hub and edge via cross-domain JetStream **mirror/source**, configured by the account's own users. This is ordinary JetStream replication, not `leaf-sync`. +The Control Plane is the **NATS Operator** and only ever touches the **SYSTEM account**. It deliberately has no presence inside any tenant's account data plane (see [Architecture §2](./architecture.md#key-properties-of-this-topology)), so it *cannot* push anything into an organization's buckets, and it cannot read anything out of them either. -The split matters: `leaf-sync` moves *metadata* (the inventory and contracts a site needs to make sense of its traffic), while JetStream moves the *traffic itself*. +So the site pulls, authenticated as itself. Live data — digital twin state, telemetry — replicates separately via cross-domain JetStream mirror and relay, configured by the account's own users. That split is what §5 and §6 are about. --- -## 3. The Leaf Node entity +## 3. `GET /api/me/leaf-config` -Create one in the console under **Leaf Nodes → New**. A server-side hook then provisions its NATS identity automatically — no manual key handling. **Creating, editing, and deleting Leaf Nodes is an Owner/Admin action**; any role in the organization can view the list and detail pages ([Authorization §6](./authorization.md#6-leaf-nodes-and-the-edge)). +Bound to the `things` collection, taking **no record id**: the target is the caller's own authenticated record, exactly like `POST /api/me/nats-creds/rotate`. It returns ten named fields: -| Field | Role | +| Field | What it is | | :--- | :--- | -| `code` | The site's slug. Derives the NATS username and the JetStream domain (`edge-`). Immutable after creation. | -| `domain` | Local JetStream domain, distinct from the hub's, so the site has its own KV/stream namespace. | -| `nats_user` | The site's single NATS identity, minted by the provisioning hook. Shared by the leaf remote, `leaf-sync`, and any local rule engine. | -| `nebula_host` | Optional link to a Nebula host, so the site can also join the overlay mesh. A Leaf Node can have both. | -| `synced_collections` | The subset of the allowlist (§4) this site mirrors. | -| `active` | Owner/Admin switch. Clearing it decommissions the site — see §7. | -| `location`, `metadata` | Optional site context. | +| `code` | The Thing's code | +| `domain` | The leaf's JetStream domain — **the same string as `code`** | +| `creds` | This Thing's own NATS credential | +| `account_jwt`, `account_pub` | The organization's NATS account | +| `operator_jwt` | The NATS Operator | +| `sys_account_jwt`, `sys_account_pub` | The `$SYS` account (see §4) | +| `hub_leaf_url` | Where the leaf remote dials the hub | +| `hub_domain` | The hub's own JetStream domain | -When you create a Leaf Node, the success modal shows its PocketBase credentials **once** — those are what `leaf-sync` authenticates with. If they're lost, an org Admin/Owner can **reset credentials** from the detail view (the new password is shown once); the old one stops working immediately, so update `leaf-sync.yaml` and restart the agent. The detail view mirrors the Thing layout: a **Liveness** card (status, last heartbeat, agent version, per-collection sync counts — see §4.1) and identity on the left; a **Connectivity** card (NATS username/status, role with reassignment, per-user permission overrides, `.creds` download, and Nebula hostname/IP/config) on the right; plus the synced-collection selection. +**There is deliberately no marker, flag or capability check on it.** Everything served is either public trust material — the Operator, account and `$SYS` account JWTs, which every server in the network validates anyway — or the caller's own credential, which it must already hold in order to connect at all. A Thing that will never run a leaf node can call this route and learns nothing it could not already read. A gate would have been a permission over data that is not secret, and it would have needed a marker field to gate on. ---- +What is *not* served: account **seeds**, signing keys, and any `$SYS` **user** credential. The `nats_system_operator` collection stays superuser-only, and a gateway holds no read grant on `nats_users` or `nats_accounts` beyond its own linked identity. The blast radius of a leaked edge credential is those ten values plus that Thing's own access — a fixed list, not a consequence of how those collections' rules later evolve. -## 4. leaf-sync: the edge agent +### The domain is the code, computed and never stored -`leaf-sync` is built from the same repository as the Control Plane but ships as its own lean binary. **The edge never runs PocketBase.** It has two commands: +The platform derives `domain` from `code` at request time. The agent writes it into **both** `server_name` and `jetstream { domain }` in the generated config, and the console matches a leaf's reported `server_name` back to a Thing's `code` to show a site attached (§7) — so those two must not diverge, and the surest way to guarantee that is to have only one of them. -```sh -leaf-sync config # one-shot: write nats-leaf.conf + edge.creds from PocketBase -leaf-sync run # daemon: mirror config collections into local KV -leaf-sync run --nats # ...and run the leaf node itself, in this process -``` +There is no `edge-` prefix and no organization segment in it either. JetStream is already scoped to the account, so the account *is* the namespace; a prefix would be decoration that every consumer then has to strip. -- **`config`** authenticates to PocketBase as the Leaf Node and fetches everything needed to stand the leaf server up from a single dedicated, leaf-node-authenticated route — `GET /api/leaf/bootstrap` — which returns eight named fields: `domain`, `code`, `creds`, `account_jwt`, `account_pub`, `operator_jwt`, `sys_account_jwt`, and `sys_account_pub`. The server reads the secret-bearing collections with its own privileges and hands back named values, never whole records, so the leaf-node identity needs no read grant on `nats_users` or `nats_accounts` at all (§7). It writes `nats-leaf.conf` and the creds file. No NATS connection required — run it *before* the leaf server is up. -- **`run`** connects to the local leaf and, every `sync.interval`, performs a **full reconcile** of each allowed collection: upsert every record into KV bucket ``, then delete keys for records that no longer exist. It is **fail-soft** — on any PocketBase or NATS error it leaves local KV untouched and retries; it never wipes local state. Deletion is driven only by what PocketBase returned, never by whether a write succeeded, so a failed write can never escalate into the key being purged. A *successful but empty* fetch is guarded too: if a collection returns zero records while local KV still holds keys, the purge is skipped for that cycle, so a transient auth or scoping glitch can't wipe the mirror. The mirror is also **self-healing**: a key removed out-of-band — a purged and recreated bucket, a lost store, a manual `nats kv del` — is rewritten on the next cycle rather than staying absent. It shuts down cleanly on `SIGINT`/`SIGTERM`, so it's safe under systemd or Docker. - -After each cycle, `run` also writes a best-effort **liveness heartbeat** (when `nats.hub_domain` is set — see §4.1). - -
-```mermaid -sequenceDiagram - participant LS as leaf-sync (Edge) - participant API as PocketBase (Control) - participant LEAF as Local NATS Leaf - - Note over LS: leaf-sync config (one-shot) - LS->>API: Auth as Leaf Node (email/pass) - API-->>LS: JWT - LS->>API: GET /api/leaf/bootstrap - API-->>LS: domain, code, creds, account JWT + pub,
NATS Operator JWT, $SYS account JWT + pub - LS->>LS: Write nats-leaf.conf + edge.creds - Note over LEAF: nats-server -c nats-leaf.conf - - Note over LS: leaf-sync run (daemon) - loop every sync.interval - LS->>API: List allowed collections (this org) - API-->>LS: Records - LS->>LEAF: Upsert into KV bucket - LS->>LEAF: Delete keys for removed records - end -``` -
- -Records are keyed in KV the same way the Control Plane handles them: by `code`, then `name`, falling back to the PocketBase record id when no stable handle exists. The id always stays inside the stored JSON so relation fields resolve, and server-only noise fields (`collectionId`, `collectionName`, `expand`) are stripped from the value. - -### 4.1 Liveness heartbeat - -A provisioned Leaf Node is only useful if you can tell it's actually running. After each reconcile, `leaf-sync` writes a small **heartbeat** into the org account's **`leaf_status`** KV bucket, keyed by the site's `code`: +--- -```json -{ "code": "warehouse-a", "version": "1.4.0", "ts": "2026-06-08T12:34:56Z", - "interval": "30s", "synced": { "things": 142, "locations": 12 }, "errors": [] } -``` +## 4. Two directives that are not optional -The console reads this bucket over its existing NATS connection and renders each site's status — a dot on the **Leaf Nodes** list and a **Liveness** card on the detail view (agent version, last-beat time, per-collection record counts, and any sync errors). A site is shown **offline** once its last beat is older than three sync intervals. +A generated `nats-leaf.conf` has to satisfy NATS operator-mode validation, and **no string assertion can check that**. Both of the following were missing for months, so the generator produced a file `nats-server` refused to load — invisible, because the only tests were substring checks over the output. -Two things make this a clean, NATS-only signal — no PocketBase write-back, no new server route: +1. **Every leaf remote needs an `account` key** naming the local account. +2. **`resolver_preload` needs the `$SYS` account JWT**, not just the organization's. The Operator JWT names a system account, and `resolver: MEMORY` has nowhere to fetch it — so without it the server dies with `error resolving system account: account missing`, *before JetStream ever starts*. -- **It rides the data plane.** The Leaf Node's own NATS identity writes the beat, and the UI (an account user) reads it. The Control Plane is never involved, consistent with the plane split in §2. -- **The heartbeat targets the *hub's* JetStream domain.** `leaf-sync` is connected to the local leaf (domain `edge-`), so a default KV write would stay local and invisible to the hub. It therefore writes across the leaf link to the hub domain named by `nats.hub_domain` — the same place the digital-twin buckets live. Leave `hub_domain` unset and the heartbeat is simply off. +!!! note "Preloading the `$SYS` **account** JWT grants nothing" + It is public trust material, like the Operator JWT beside it. Connecting *as* `$SYS` requires a `$SYS` **user** credential, which the platform never serves to anything. Those are different objects, and it is worth being precise about which one is which, because the first looks alarming and is not. -The write is **best-effort**: a heartbeat failure (e.g. a WAN outage) is logged and never disturbs the sync loop — and the *absence* of a recent beat is exactly what the console reads as "offline." +The platform's generator now runs its own output through the real `nats-server` config parser in a test. Keep that shape if you touch it — substring checks cannot express "and the server accepts it". --- -## 5. What gets synced +## 5. Deploy flow + +1. In the console, create the site's **Thing**, with a Thing Type that says gateway. Copy the login password from the success dialog — it is shown once. +2. Install the [Agent](./agent.md) on the edge box and configure the platform block: + + ```yaml + code: "s01" + platform: + url: "https://platform.acme.io" + identity: "s01@things.acme.io" + password_env: "AGENT_PLATFORM_PASSWORD" + nats: + urls: ["nats://127.0.0.1:4222"] + auth: + type: "platform" + creds_file: "/etc/agent/device.creds" + twin: + enabled: true # optional, see §6 + observability: + addr: "127.0.0.1:9100" # optional, see §7 + ``` + +3. `agent -leaf-config` → writes `nats-leaf.conf` (0644) and the creds (0600) beside each other. +4. Start the leaf, either way: + - **Two processes (default):** `nats-server -c /etc/agent/nats-leaf.conf` under systemd or Docker. The bus then survives an agent restart, which is what you want when upgrading the agent on a live site. + - **One process:** set `nats.server_config` to that path and the agent hosts the server itself. `nats.urls` must name the port the config listens on; startup refuses the pair if they disagree. +5. `agent -service install && agent -service start`. +6. Point any site-local [rule engine](./automation.md) at the same creds file. + +!!! note "Bootstrapping and running cannot be one invocation" + `-leaf-config` is a one-shot for a structural reason, not a stylistic one. A separately supervised `nats-server` needs its config file to exist *before* it starts — which is before the agent has anything to connect to. -A hard allowlist, enforced both by the server's API rules and by `leaf-sync`: - -``` -things locations thing_types location_types thing_type_operations -``` +--- -This is exactly the **Thing contract graph**: `thing_type` → `thing_type_operation` (see [Thing Types](./thing-types.md)), plus the Things and Locations that instantiate it. With these mirrored locally, a site can resolve the subject any Thing at that site publishes to or listens on **entirely offline**. +## 6. Offline autonomy and the digital twin -A Leaf Node identity can read exactly two things: **its own `leaf_nodes` record**, and **the five allowlisted collections above, within its own organization**. Nothing else. Secret-bearing collections (`nats_users`, `nats_accounts`, `nebula_*`) are never exposed to a Leaf Node identity and can never be synced — the API rules contain no `leaf_nodes` branch for any of them. The bootstrap values an edge box genuinely cannot derive locally (its own creds, the account JWT and public key, the NATS Operator JWT, and the `$SYS` account JWT and public key) come from the dedicated `GET /api/leaf/bootstrap` route instead (§4, §7). The `synced_collections` field on the record selects which of the allowlist a given site actually mirrors. +Once the leaf is up, the rest of the layered platform runs at the edge without the hub: a [rule engine](./automation.md) keeps evaluating site-local reflexes, a [stream processor](./stream-processing.md) keeps producing aggregates, and devices keep publishing to the local leaf, which buffers and forwards once connectivity returns. ---- +With `twin.enabled`, the agent also syncs digital-twin state. **Two buckets, one writer each:** -## 6. Offline autonomy +| Bucket | Written by | Flows | Mechanism | +| :--- | :--- | :--- | :--- | +| `twin` | the device, at the edge | edge → hub | relay | +| `twin_desired` | operators, at the hub | hub → edge | JetStream **mirror** | -Once `leaf-sync` has populated local KV, the rest of the layered platform runs at the edge without the hub: +**One writer per bucket is the whole safety property.** A single bucket written from both ends does not pick a loser on a conflict — it *oscillates*: two concurrent values for one key swap across the link, then swap back, each write generating the next event. Measured at roughly 170,000 writes to a single key in 300 ms before the buckets were split. Encoding the owner in the key (`thing.S01.state.temp`) buys the same safety but taxes every key in firmware, rule configs and widgets, and a mistyped segment silently never syncs. -- A [rule engine](./automation.md) instance reads the mirrored config and live KV and keeps evaluating site-local reflexes during a WAN outage. -- A [stream processor](./stream-processing.md) keeps producing aggregates from local subjects. -- Devices and Agents keep publishing to the local leaf, which buffers and forwards once connectivity returns. +Desired state is a **mirror** because it has exactly one origin, and the edge never writes it — so reads are served locally from last-known values, which is precisely what you want when the link is down. Reported state needs the **relay** because aggregating N sites natively would need N sources all named `KV_twin`, which the client library cannot express; the alternative is a differently-named bucket at every site, which pushes the problem into every rule that reads one. -`leaf-sync`'s full-reconcile loop is what makes this safe: when the link comes back, the next cycle converges local KV to whatever changed centrally, without ever having served stale-but-broken state in the meantime. +Off by default, because it moves data-plane traffic and an upgrade must not silently start doing that. --- -## 7. Security model +## 7. Is the site up? -- **One NATS identity per Leaf Node**, shared by the leaf remote, the rule engine, and `leaf-sync`. The **edge box is the trust boundary**; tenant isolation is the NATS *account* boundary, which the site cannot cross. -- The site only ever holds **public trust material** (NATS Operator JWT, account JWT) plus its own user's creds. It **cannot mint new account users**. -- That trust material is reachable only through `GET /api/leaf/bootstrap`, authenticated as the Leaf Node. The handler reads named fields with the server's own privileges — the NATS Operator collection stays superuser-only, account *seeds* and signing keys are never served, and the leaf-node identity holds **no read grant on `nats_users` or `nats_accounts`**. So the blast radius of a leaked edge credential is those eight values and its allowlisted config, fixed regardless of how those collections' rules later evolve. (`GET /api/leaf/operator-jwt` still exists for older agents; it is superseded by `/api/leaf/bootstrap`.) -- Narrowing a site's blast radius is a record edit — reassign its NATS role or add per-user permission overrides from the Leaf Node's detail view. Both are **Owner/Admin** actions, since they write to `nats_users` and `nats_roles`. -- The Leaf Node's PocketBase password (its `leaf-sync` login) is resettable by an org Admin/Owner from the console — gated by the collection's `manageRule`, so it stays a scoped, audited record action rather than a superuser-only operation. -- **Deactivating a Leaf Node takes the site off the fabric.** `active` is an Owner/Admin boolean, and clearing it does three things at once: `leaf-sync` can no longer authenticate, the session token it already holds is invalidated immediately, and the site's NATS credential is revoked — so the config pull and the leaf remote connection both stop. Reactivating issues a **new** credential; the previous `.creds` stays revoked permanently, so re-run `leaf-sync config` on the box. This is the control to reach for when a site is retired or a box is presumed lost. See [Authorization §4.2](./authorization.md#42-taking-a-device-out-of-service). +Ask NATS, on the Thing's own page in the console. -!!! warning "`active` and the liveness badge answer different questions" - The Liveness card (§4.1) reports whether `leaf-sync` **is** currently talking to the hub. `active` governs whether it **may**. A deactivated site showing "offline" is the expected outcome, not a fault to chase — and an *active* site showing offline is the one worth investigating. +There is **no heartbeat and no status field** behind that badge, and there used to be both. A `leaf_status` KV bucket carried a beat per site and the console rendered it as online/offline. It was removed because a heartbeat travels over the very link whose failure it is meant to report: a missing beat cannot distinguish "edge box down" from "WAN down" from "agent crashed" — three different call-outs behind one red dot. And the Control Plane could never have read one anyway (§2). ---- +The hub, on the other hand, always knows which leaf connections it is holding. So the console asks it, over the browser's own in-account NATS connection: -## 8. Deploy flow +``` +$SYS.REQ.ACCOUNT.PING.CONNZ +``` -1. In the console, create a **Leaf Node**; copy the credentials from the success modal. -2. On the edge box, install `leaf-sync` and a stock `nats-server`, and write `leaf-sync.yaml`: +Entries with `kind: "Leafnode"` are matched by `name` — the leaf's `server_name`, which is the Thing's `code` (§3). - ```yaml - pocketbase: - url: "https://platform.acme.io" - email: "warehouse-a@acme.leaf.local" # from the success modal - password: "••••••••" - nats: - hub_leaf_url: "nats://nats.acme.io:7422" # where the leaf remote dials the hub - local_url: "nats://127.0.0.1:4222" - creds_file: "edge.creds" - hub_domain: "hub" # hub's JetStream domain; enables the heartbeat (§4.1) - sync: - interval: "30s" - ``` +**Each account carries its own `$SYS` subject space.** `$SYS.REQ.ACCOUNT.PING.*` is scoped to the caller's own account and answers for that organization and no other; the operator-wide `$SYS.REQ.SERVER.PING.*` endpoints, which would span every tenant, are not reachable from a tenant credential. The server enforces both halves, and the platform pins them in a test against a real hub with a real leaf attached. - Any key can be overridden by an environment variable: upper-case it, replace dots with underscores, and prefix `LEAF_SYNC_` (e.g. `LEAF_SYNC_POCKETBASE_PASSWORD`). -3. `leaf-sync config` → produces `nats-leaf.conf` + `edge.creds`. -4. `nats-server -c nats-leaf.conf` (under systemd/Docker/your init system). -5. `leaf-sync run` (likewise supervised). -6. Point your local rule engine at `edge.creds` for site-local automation. +!!! warning "In NATS, a publish DENY beats a publish ALLOW" + If that badge ever times out for one organization and not another, this is almost certainly why. A NATS Role carrying `$SYS.>` in its **publish deny** list cannot reach the account-scoped endpoints no matter what its allow list says — and the symptom is a plain request timeout, with the real reason arriving asynchronously on the connection's error handler and never on the request itself. Deny `$SYS.REQ.SERVER.>` instead. -`leaf-sync` does not supervise the other processes — use whatever your platform provides. +The badge has three states, not two, and the third one matters: without a NATS connection the console cannot answer the question at all, so it says **unknown** rather than showing every site as down. -### 8.1 One process instead of two: `run --nats` +!!! note "The empty state is not painted as a fault" + The console cannot tell a gateway that *should* have a leaf from a temperature probe that never will — there is no marker field, by design (§1). So "no leaf node attached" is stated as a fact in neutral colour. It reads as an alarm on a gateway's page and a shrug on a probe's, and the person looking knows which they are looking at. -Steps 4 and 5 can collapse. `leaf-sync run --nats` starts the leaf node **inside the agent process**, from the same `nats-leaf.conf` that step 3 wrote, so an edge site is one supervised service rather than two: +### Per-site health, in detail -```sh -leaf-sync run --nats -``` +The badge answers one question. For the rest — is JetStream filling the disk, how many devices are actually attached, is the uplink down — the agent serves `/ready` and `/metrics` **on the box**, behind `observability.addr`. That is where per-site health can actually be measured, and it keeps answering with the WAN down, which is exactly when you want it. See [Health & Metrics](./health-metrics.md). -Off by default — the two-process topology stays the default, because it is the one that lets you upgrade the agent without interrupting the bus. Two behaviours change when the flag is on, and both are deliberate: +**An islanded site warns; it does not fail.** `hub_uplink` is a warning and still answers 200. Local NATS keeps working and devices keep running — that autonomy is the entire reason a leaf node exists, so reporting it as unready would invert the design. + +--- -- **The NATS server starts before PocketBase is touched at all.** A site whose uplink is down must still come up with a working local bus; that autonomy is the entire point of a leaf node. -- **A failed PocketBase login retries instead of exiting.** Without `--nats`, exiting is correct — the bus is another process and survives. With `--nats`, exiting takes the bus down with it, and a supervisor cycling the pair through a WAN outage means every device on site reconnecting and JetStream recovering its store, in a loop. +## 8. Security model -The cost is binary size: `nats-server` links in either way, so the stripped binary goes from roughly 17 MB to the high twenties. On an edge box that is not a consideration; it is noted so the number is not a surprise. +- **The edge box is the trust boundary.** Tenant isolation is the NATS *account* boundary, which a site cannot cross. One NATS identity per gateway, shared by the leaf remote, the rule engine, and the agent. +- The site holds **public trust material** (Operator JWT, account JWT, `$SYS` account JWT) plus its own user's creds. It **cannot mint new account users**. +- **Taking a site out of service is `active` on its Thing**, Owner/Admin only. Clearing it does three things at once: the agent can no longer authenticate, the session token it already holds is invalidated immediately, and the site's NATS credential is revoked — so the config pull and the leaf remote connection both stop. Reactivating issues a **new** credential; the previous `.creds` stays revoked permanently, so re-run `agent -leaf-config` on the box. See [Authorization §4.2](./authorization.md#42-taking-a-device-out-of-service). +- **Deactivate, do not delete.** Revoking a Nebula certificate requires the certificate to still be in the database so its fingerprint can be published; deleting the record leaves it trusted until it expires. +- The Thing's PocketBase password is resettable by an org Admin/Owner from the console, gated by the collection's `manageRule` — a scoped, audited record action rather than a superuser-only operation. +- Narrowing a site's blast radius is a record edit: reassign its NATS Role or add per-user permission overrides. Both are **Owner/Admin** actions, since they write to `nats_users` and `nats_roles`. -> Build a release binary with the version stamped in (it surfaces in `leaf-sync --version` and in every heartbeat): -> ```sh -> go build -ldflags "-X platform/internal/version.Version=$(git describe --tags --always --dirty)" -o leaf-sync ./cmd/leaf-sync -> ``` -> -> Or download it from the [Releases page](https://github.com/stone-age-io/platform/releases) — `leaf-sync` is published for linux, darwin and windows on amd64 and arm64, already stamped. +!!! warning "`active` and the connectivity badge answer different questions" + The badge (§7) reports whether the site **is** currently attached to the hub. `active` governs whether it **may** be. A deactivated site showing no leaf is the expected outcome, not a fault to chase — and an *active* site showing none is the one worth investigating. --- ## 9. Where to Go Next - **The leaf node transport primitive:** [Connectivity §1 — Leaf Nodes](./connectivity.md#leaf-nodes). -- **The contract graph that gets synced:** [Thing Types](./thing-types.md). -- **Per-Thing edge execution (a different agent):** [The Agent](./agent.md). +- **The agent that does all of this:** [The Agent](./agent.md). +- **The contract graph a site's devices publish against:** [Thing Types](./thing-types.md). - **Site-local rules during outages:** [Automation](./automation.md). - **What an edge identity may read, and who may manage it:** [Authorization & Roles](./authorization.md). +- **Per-site health endpoints:** [Health & Metrics](./health-metrics.md). - **How the planes fit together:** [Architecture](./architecture.md) and [Platform Layers](./platform-layers.md). diff --git a/docs/operations.md b/docs/operations.md index d94b725..bb63b19 100644 --- a/docs/operations.md +++ b/docs/operations.md @@ -16,7 +16,7 @@ Operational state lives in different places, with different owners and different | **Inventory & contracts** — Orgs, Things, Thing Types, Locations, schemas, rules-adjacent config | Control Plane (`pb_data`) | High, but recoverable — re-entry is tedious, not impossible. Also recoverable from a [GitOps workspace](./stone-cli.md#5-declarative-workspaces-pull-apply). | This page (§3), plus `stone pull` workspaces. | | **Live state** — Digital Twin KV, JetStream streams | NATS servers (JetStream storage) | Low to moderate. Twins repopulate from device heartbeats; stream retention is a buffer, not an archive. | JetStream replicas (`replicas: 3` on clustered NATS), stream mirrors. | | **Historical telemetry** | Your Layer 3 TSDB | Your call — it's [BYO](./observability.md). | Your TSDB's own backup tooling. | -| **Edge config mirrors** | Leaf node local KV | None. `leaf-sync` reconverges from the Control Plane on its next cycle. | Nothing needed — see [Leaf Nodes](./leaf-nodes.md). | +| **Edge leaf config** | `nats-leaf.conf` + creds on the edge box | None. Regenerate with `agent -leaf-config` — see [Leaf Nodes](./leaf-nodes.md). | Nothing needed. | The takeaway: **`pb_data` is the crown jewels.** It's also a single directory dominated by one SQLite database, which makes protecting it straightforward. @@ -47,7 +47,7 @@ The Control Plane is a low-traffic metadata store, and its outage is far less dr | Already-issued NATS/Nebula credentials | ✅ Keep working — auth is verified by the cluster, not by PocketBase | | Console login, entity management | ❌ Paused | | Provisioning new Orgs / Things / credentials | ❌ Paused | -| Agent bootstrap and `leaf-sync` config pulls | ⏸️ Paused — both retry; `leaf-sync` is fail-soft and leaves local KV untouched | +| Agent bootstrap and credential sync | ⏸️ Paused — the Agent retries, and an already-provisioned site keeps running on the credential and leaf config it holds | Nothing in that bottom half is latency-critical. So instead of running an HA database topology to protect a metadata store, the platform's answer is **aggressive backups plus a short, rehearsed restore path** (§3–4). With scheduled native backups, S3 offsite copies, and filesystem snapshots, realistic time-to-recovery is minutes — which, for a service whose outage pauses provisioning but not production traffic, is the right trade. @@ -179,7 +179,7 @@ The NATS cluster needs **no changes** — it kept running the whole time, and ev command that fixes it. See [Health & Metrics](./health-metrics.md). - Console login works (Platform Operator user) and the **NATS Status: Connected** indicator is green. - Create a throwaway Thing in a test org — confirms the provisioning hooks and the System Account connection end-to-end. -- `leaf-sync` heartbeats reappear on the Leaf Nodes list within a few sync intervals. +- A gateway Thing's page shows its leaf node attached again. That reading comes from the hub over the console's own NATS connection, so it is a live fact rather than a replayed one. **Rehearse this.** A backup you've never restored is a hypothesis, not a backup. The `pb` migration flow in §5.3 doubles as a restore drill — do it on a schedule, not just before upgrades. @@ -238,7 +238,7 @@ If the upgrade misbehaves, you found out on staging — against your actual sche The Control Plane is the only component with a database and migrations. Everything else upgrades by binary swap, in any order, because the interfaces between components are protocols, not shared code (§6): - **`rule-router`, stream processors, Telegraf** — restart with the new binary; they reconnect to NATS and resume. Stateless per message; durable state is in KV. -- **Agents and `leaf-sync`** — swap and restart; both are designed around reconnection and fail-soft behavior. +- **Agents** — swap and restart; designed around reconnection and fail-soft behaviour. On a site gateway, restarting the agent bounces the leaf server too **if** `nats.server_config` is set; leave it unset and a separately supervised `nats-server` keeps the bus up across the upgrade, which is usually what you want on a live site. - **NATS and Nebula** — stock upstream upgrade procedures; the platform places no constraints beyond theirs (see §6). ### 5.5 Moving the NATS server out of the Control Plane @@ -292,7 +292,6 @@ Stone-Age.io is a set of independent binaries, so the compatibility question is | :--- | :--- | :--- | | **Stone Age Console** (embedded UI) | Ships inside the `stone-age` binary | Always in lockstep with the schema by construction. No version skew is possible. | | **`stone` CLI** | PocketBase REST API + the platform's collections schema; NATS protocol | The REST API is stable upstream PocketBase. The collections schema is the platform's own contract — additive changes don't break the CLI; pre-1.0 breaking schema changes are flagged in release notes. | -| **`leaf-sync`** | PocketBase REST API (its allowlisted collections) + NATS leaf protocol | Mirrors records generically; tolerant of additive schema change. Built from the platform repo, so matching its release to the Control Plane's is the safe default. | | **Agent** | PocketBase auth API (bootstrap only) + NATS protocol | After bootstrap it's a pure NATS client. | | **`rule-router`** | NATS subjects + KV only | Knows nothing about PocketBase. Versioned independently. | | **Stream processors, Telegraf, TSDB, Grafana/Perses** | NATS subjects only | Fully platform-agnostic. The subject contract ([Thing Types](./thing-types.md)) is the only interface. | @@ -301,9 +300,9 @@ Stone-Age.io is a set of independent binaries, so the compatibility question is Practical guidance: -- **Upgrade the Control Plane first** when a release touches the schema — clients (`stone`, `leaf-sync`) tolerate additive changes, and the release notes call out anything that isn't. +- **Upgrade the Control Plane first** when a release touches the schema — clients (`stone`, the Agent) tolerate additive changes, and the release notes call out anything that isn't. - **Layer 1–3 components don't care about platform releases at all.** Their contract is the subject namespace, which is yours to keep stable — see [Connectivity](./connectivity.md). -- **Pre-1.0, pin versions** across `stone-age`, `stone`, and `leaf-sync`, and move them together when the notes mention schema changes. Post-1.0, additive-only within a major version is the rule. +- **Pre-1.0, pin versions** across `stone-age`, `stone`, and the Agent, and move them together when the notes mention schema changes. The Agent releases from its own repository on its own tags, so its version does not track the platform's. Post-1.0, additive-only within a major version is the rule. --- @@ -330,7 +329,7 @@ A condensed pre-flight list for taking a deployment to production: ``` Host certificates count only `active = true` rows, so a decommissioned device’s lapsed certificate does not page anyone. There is no per-organization label: `/metrics` is open by default, and a tenant name beside a certificate inventory is free reconnaissance. Set an alert on the CA series in particular — every host certificate chains to it. -- [ ] **A decommissioning path agreed** — know before you need it that clearing `active` on a Thing or Leaf Node is the control that actually cuts a device off (session killed, credential revoked), and that reactivating issues a **new** `.creds` the device must be given ([Authorization §4.2](./authorization.md#42-taking-a-device-out-of-service)). +- [ ] **A decommissioning path agreed** — know before you need it that clearing `active` on a Thing is the control that actually cuts a device off (session killed, credential revoked), that it applies to a site gateway exactly as to any other device, and that reactivating issues a **new** `.creds` the device must be given ([Authorization §4.2](./authorization.md#42-taking-a-device-out-of-service)). - [ ] **SuperUser reserved** for infrastructure work; day-to-day administration through a Platform Operator user ([Getting Started §2](./getting-started.md#2-initialize-the-control-plane)). - [ ] **Least-privilege role review** — walk each org's memberships and confirm nobody holds more than they need. `admin` is **not** a junior grant: it is identical to `owner` in every API rule, including every credential-bearing collection. Most humans want `member` ([Authorization](./authorization.md)). - [ ] **`./scripts/test-authz.sh` green** on the exact commit you're deploying — the API rules are the only tenancy enforcement in the platform, and the suite is the only thing that checks them. If the release changed a rule, confirm it also shipped a migration (§5.1). diff --git a/docs/platform-ui-entities.md b/docs/platform-ui-entities.md index 76492a6..87c4cdc 100644 --- a/docs/platform-ui-entities.md +++ b/docs/platform-ui-entities.md @@ -23,7 +23,7 @@ A Membership binds a PocketBase User to an Organization. - **Roles (per-organization):** - `Owner`: Full tenant authority. **Identical to `Admin` in every API rule** — the only differences are that an Owner cannot leave their own organization and may delete it. - - `Admin`: Full tenant authority — members and invitations, NATS and Nebula infrastructure, Thing/Location types and contracts, Leaf Nodes, and the identity links on a Thing. + - `Admin`: Full tenant authority — members and invitations, NATS and Nebula infrastructure, Thing/Location types and contracts, and the identity links on a Thing. - `Member`: Creates and edits Things and Locations, and reads the contract collections (Thing Types, Operations). Cannot delete a Thing or Location, cannot attach identities to one, and cannot read the infrastructure collections at all. - `Viewer`: Read-only staff. Browses the inventory screens and uses dashboards, and writes nothing anywhere. Adding it needed no rule change at all — a role that names itself in no write branch is denied by construction. - `Dashboard`: An appliance login for an unattended screen — the Visualizer and its own settings page, nothing else. It holds no write capability, which is exactly why the authorization suite uses it as the probe that proves an allowlist works. @@ -51,7 +51,7 @@ Authorization is enforced **solely** by PocketBase API rules on each collection. ### Self-Service Credential Rotation -Any authenticated identity with a linked NATS user — a `users` membership, a `things` record, or a `leaf_nodes` record — can rotate its own credential: +Any authenticated identity with a linked NATS user — a `users` membership or a `things` record — can rotate its own credential: ``` POST /api/me/nats-creds/rotate @@ -154,7 +154,7 @@ Only the keys present in a desired value are compared, so extra fields a device The same KV buckets are what Layer 1 rules read and write for stateful operations like alarm stacking. See [Architecture §4](./architecture.md#4-the-digital-twin-concept-live-state) for the full model, and [Automation](./automation.md) for the KV-state patterns. -> **Neither the console nor the platform server creates these buckets on its own.** The Control Plane holds the NATS Operator key but has no reach into an organization's own account, so it cannot provision them. Creation is the console's **Initialize** button, or `leaf-sync` at the edge — whichever gets there first defines the bucket, which is why the two retention configurations are kept in step deliberately. +> **Neither the console nor the platform server creates these buckets on its own.** The Control Plane holds the NATS Operator key but has no reach into an organization's own account, so it cannot provision them. Creation is the console's **Initialize** button, or the Agent at the edge — whichever gets there first defines the bucket, which is why the two retention configurations are kept in step deliberately. ### JetStream Streams and KV Buckets