Skip to content

feat!: absorb leaf-sync; an agent can be its site's NATS leaf node - #4

Merged
skeeeon merged 2 commits into
mainfrom
feat/nebula-support
Sep 16, 2026
Merged

skeeeon merged 2 commits into
mainfrom
feat/nebula-support

Conversation

@skeeeon

@skeeeon skeeeon commented Sep 16, 2026 •

Copy link
Copy Markdown
Contributor

The leaf-sync binary that lived in the platform repo is gone; its duties are agent capabilities now. A site that runs a NATS leaf node is an ordinary Thing whose agent has those capabilities turned on.

Pairs with stone-age-io/platform, which drops leaf_nodes and cmd/leaf-sync, and stone-age-io/platform-docs.

Four capabilities, each independently switchable

  • agent -leaf-config — one-shot: authenticate as this agent's Thing, call GET /api/me/leaf-config, write nats-leaf.conf (0644) beside the creds (0600). It has to be separable from running: the usual edge shape is a separately supervised nats-server, and that server needs its config file before it starts — which is before the agent has anything to connect to.
  • nats.server_config — host a nats-server in this process from that file. It points at any nats-server config, not only a generated one, so it is equally how you run a plain embedded broker on a box with no platform. Left empty, systemd or Docker supervises nats-server and the bus survives an agent restart, which is what you want when upgrading the agent on a live site.
  • twin.enabled — relay this site's reported state up, mirror desired state down, so the site keeps deciding locally through a WAN outage.
  • observability.addr — /ready and /metrics on the box. cmd.health travels over NATS, which is the link that breaks: the box you most need to ask is the one whose uplink is down, and that is exactly when it goes quiet.

There is no edge.enabled key, and no gateway flag on the platform either. "Gateway" is not a mode the config declares, it is the sum of the capabilities it turns on — edge.enabled: false beside twin.enabled: true has no correct behaviour. A gateway is just a Thing whose thing_type says so.

What did not come across

The config mirror. leaf-sync copied an organization's inventory collections into local KV and nothing ever read them. Dropping it is also what lets a gateway need no read grant across the inventory.

Its remains were dead code the compiler could not see: kvWriter and its per-collection bucket helper came across with the package, and Go does not flag unused types or methods — only unused locals and imports — so it built clean. Removed, and every other unexported symbol in edge and observe swept for the same shape.

Breaking: the platform relationship moved to a top-level platform: block

auth.type: "stone-age" becomes "platform". Three subsystems read this relationship — the NATS credential lifecycle, the Nebula config source, and the new leaf bootstrap — and buried under nats.auth the other two had to ask "is some other section's type field set to a particular string" instead of "is the block present".

tasks.creds_sync is gone, replaced by platform.sync_interval. Its enabled flag was not a state worth being able to express: an agent told to fetch credentials but not keep them current stops working in seven days.

One test that was not allowed to change

TestBuildLeafConfIsAcceptedByNATSServer moved unchanged apart from its imports. It runs the real generator's output through nats-server's own ProcessConfigFile + NewServer (no ports, no network), and it is the only thing that can catch the two operator-mode directives that were missing for months while substring assertions passed:

  1. every leaf remote needs an account key naming the local account;
  2. resolver_preload needs the $SYS account JWT as well as the org's — the operator JWT names a system account and resolver: MEMORY has nowhere to fetch it, so without it the server dies with error resolving system account: account missing before JetStream starts.

Preloading $SYS's account JWT is public trust material and grants nothing; connecting as $SYS needs a $SYS user credential, which is never served.

Duplicated on purpose

internal/health, internal/metrics and internal/natsd are copies of the platform repo's, not an extracted shared module. Two small copies that drift are cheaper to live with than a third repo to version and keep both consumers pinned to, and the two processes check genuinely different things — the Control Plane cannot answer anything about a site, and the agent cannot answer anything about the fleet. Noted in both CLAUDE.md files so the next person does not "fix" it.

Related: an islanded edge warns, it does not fail. Local NATS still works and devices keep publishing and subscribing locally; that autonomy is why a leaf node exists, so returning 503 would invert the design.

Documentation

New docs/leaf-node.md carries the reasoning that previously existed only in the platform repo's cmd/leaf-sync README — the $SYS preload, the two mandatory directives, one-writer-per-bucket with the number that made the case (~170k writes to a single key in 300ms when both ends wrote one bucket), the relay-vs-mirror asymmetry, and a "What went away" section including the publish-DENY trap.

CLAUDE.md, README.md, CHANGELOG.md, docs/credentials.md, docs/architecture.md, the three platform install guides, docs/nebula*.md and .goreleaser.yaml all follow the platform: block.

The docs pass also caught two user-visible strings that survived the package move: the generated nats-leaf.conf carried a header naming leaf-sync config, a command that no longer exists, on the file an operator reads when something breaks — and twin.go logged leaf-sync: on every warning, one of which told the reader to delete a bucket "to have leaf-sync recreate it".

Verification

Build and vet clean for linux/amd64, windows/amd64, freebsd/amd64 and linux/arm64. TestExampleConfigsParse passes for all three example configs. TestBuildLeafConfIsAcceptedByNATSServer passes.

internal/tasks fails on this Windows box (service-manager access denied, log-path allowlist); git status confirms that package is untouched by this work and the failures are environmental.


Merge with platform: stone-age-io/platform#11 and docs: stone-age-io/platform-docs#6 — the three land together.

🤖 Generated with Claude Code

Four capabilities arrive together, each independently switchable, from the
`leaf-sync` binary that used to live in the platform repo:

- `agent -leaf-config` -- one-shot: authenticate as this agent's Thing, call
  GET /api/me/leaf-config, write nats-leaf.conf (0644) beside the creds
  (0600). It has to be separable from running: the usual edge shape is a
  separately supervised nats-server, and that server needs its config file
  before it starts, which is before the agent has anything to connect to.
- `nats.server_config` -- host a nats-server in this process from that file.
  It points at ANY nats-server config, not only a generated one, so it is
  equally how you run a plain embedded broker on a box with no platform.
- `twin.enabled` -- relay this site's reported state up, mirror desired
  state down, so the site keeps deciding locally through a WAN outage.
- `observability.addr` -- /ready and /metrics on the box. cmd.health travels
  over NATS, which is the link that breaks; the box you most need to ask is
  the one whose uplink is down, and that is when it goes quiet.

There is no `edge.enabled` key and no gateway flag on the platform either.
"Gateway" is not a mode the config declares, it is the sum of the
capabilities it turns on -- `edge.enabled: false` beside `twin.enabled: true`
has no correct behaviour. A gateway is just a Thing whose thing_type says so.

What did NOT come across: the config mirror. leaf-sync copied an org's
inventory collections into local KV and nothing ever read them.

BREAKING: the platform relationship moved to a top-level `platform:` block
and `auth.type: "stone-age"` became `"platform"`. Three subsystems read it --
the NATS credential lifecycle, the Nebula config source, and the new leaf
bootstrap -- and buried under nats.auth the other two had to ask "is some
other section's type field set to a particular string" instead of "is the
block present". `tasks.creds_sync` is gone, replaced by
`platform.sync_interval`: its `enabled` flag was not a state worth being able
to express, since an agent told to fetch credentials but not keep them
current stops working in seven days.

internal/health, internal/metrics and internal/natsd are DUPLICATED from the
platform repo rather than extracted into a shared module. Two small copies
that drift are cheaper than a third repo to version and keep both consumers
pinned to, and the two processes check genuinely different things. Noted in
both CLAUDE.md files so the next person does not "fix" it.

TestBuildLeafConfIsAcceptedByNATSServer moved unchanged apart from imports,
and is the one thing in the move that was not allowed to change: it runs the
generated config through nats-server's own parser, which is the only way to
catch the two operator-mode directives (a leaf remote's `account` key, and
the $SYS account JWT in resolver_preload) that were missing for months
because the tests were substring checks.

Also removes dead code the compiler could not see: the kvWriter type and its
per-collection bucket helper came across with the package and nothing calls
them, since the mirror they served is gone.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
"No listening ports" and "all communication via NATS" were true at 0.1.0 and
have not been true since. Three things changed underneath them: the platform
block talks outbound HTTPS to the Control Plane, `nats.server_config` makes a
gateway host a nats-server that local devices connect in to, and
`observability.addr` defaults to 127.0.0.1:9100 -- so EVERY agent serves
/ready and /metrics unless the key is set empty.

The claim appeared in five places (README, CLAUDE.md, docs/architecture.md and
the firewall step of all three install guides), and the install guides are the
ones that mattered: "Agent only needs outbound NATS connection / No inbound
ports required" is advice someone writes a rule from.

Replaced with the property that IS true and is the one worth relying on:
nothing can instruct the agent except over the NATS connection it dialed
itself, so a fleet behind NAT or CGNAT needs no inbound rule to be managed.
README and the guides now carry a direction-and-port table instead of a slogan.

Two real defects found while checking the claims:

- The example configs told you to uncomment a second top-level `nats:` key to
  set server_config. Doing as instructed is not untidy, it is fatal --
  go-yaml refuses a duplicate mapping key ("mapping key \"nats\" already
  defined at line 1") and the agent will not start. server_config is now a
  commented line inside the one nats: block, in all three examples and in the
  README. Verified by uncommenting it and parsing: it lands at
  nats.server_config. TestExampleConfigsParse only ever read the file as
  shipped, so it could not have caught this.
- observability.addr defaults to 127.0.0.1:9100, which is node_exporter's
  default port on Linux and FreeBSD -- the exporter these same guides offer as
  an alternative metrics source. A bind failure is logged rather than fatal, so
  the symptom is an endpoint that quietly never came up. Documented at every
  point where someone could hit it; the default itself is left alone pending a
  decision on which side should move.

Also: Nebula and local health were filed under "Edge / gateway" in the README.
Neither is a gateway capability -- observability is on for every agent and
nebula.enabled is per-device -- so they now sit in their own sections.

CHANGELOG gains a Changed entry for the new listener, since 0.1.0 advertised
the opposite and firewall rules may have been written against it. The 0.1.0
entry itself is left as written: it recorded what was true at that tag.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@skeeeon
skeeeon merged commit 219ee48 into main Sep 16, 2026
1 check passed
@skeeeon
skeeeon deleted the feat/nebula-support branch September 24, 2026 04:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant