Skip to content

feat(system): add stibnite as an aarch64-darwin build target - #2883

Merged
cameronraysmith merged 5 commits into
mainfrom
fm/vx-stibnite-darwin-builder
Sep 1, 2026
Merged

feat(system): add stibnite as an aarch64-darwin build target#2883
cameronraysmith merged 5 commits into
mainfrom
fm/vx-stibnite-darwin-builder

Conversation

@cameronraysmith

@cameronraysmith cameronraysmith commented Aug 31, 2026

Copy link
Copy Markdown
Owner

Intent

Register stibnite as the fleet's aarch64-darwin build target so magnetite can build darwin derivations, with agent-session access and restricted nix-build access as separately authorized keys, and document the remote-builder versus remote-store distinction. Nothing beyond that.

What this changes

stibnite is declared as the fleet's aarch64-darwin build target, and the two ways of reaching its store are configured and documented as two mechanisms rather than one.

Before this branch there was no nix.buildMachines entry naming stibnite anywhere. The repository held two build-machine entries, both Linux: the rosetta VM and magnetite (modules/machines/darwin/stibnite/default.nix, modules/system/magnetite-builder.nix). magnetite is x86_64-linux and cannot build darwin derivations, so darwin work had no build target at all.

The two mechanisms

modules/system/stibnite-access.nix exports three aspects.

services.stibnite-builder (nixos) emits a nix.buildMachines entry. The caller's daemon copies the input closure out, builds on stibnite, and copies the output closure back, so the result exists in the caller's store. This is for development use: a developer or operator on magnetite who needs an aarch64-darwin result locally, for example building or testing darwin configurations.

services.stibnite-builder.storeUri, materialized at /etc/nix/stibnite-store-uri, is the remote-store path. Under nix build --store ssh-ng://… evaluation stays with the caller, but the remote store is the store, so the derivation and its source inputs must exist there and are copied to stibnite; what is avoided is copying the output closure back. Intended caller: a machine whose store is empty and stays empty, such as an ephemeral runner, a fresh container, or an installer image.

services.stibnite-session is interactive ssh under its own keypair and its own ssh alias. Its authority is described under "Key separation" below, and it is the broader of the two credentials rather than the narrower.

Both build mechanisms speak ssh-ng to the same account through the same alias, which the new check pins so they cannot drift apart. Legacy ssh://, which would run nix-store --serve, is not served.

Values justified rather than copied

  • systems = [ "aarch64-darwin" ]: nix config show extra-platforms on stibnite reports aarch64-darwin and nothing else, so x86_64-darwin would route derivations the machine refuses.
  • maxJobs = 4: measured 18 logical cores (12 performance, 6 efficiency) and 64 GiB, on a laptop in interactive use that commits 12 cores and 48 GiB to the rosetta VM and the same to colima. The remote share is a deliberate minority.
  • speedFactor = 1: inert, because nix compares speed factors only among machines that can build the same system and there is no second darwin machine.
  • supportedFeatures = [ "apple-virt" "big-parallel" ]: taken from nix config show system-features, minus benchmark (timings on an interactively loaded laptop are not measurements) and nixos-test (a Linux sandbox capability nix lists unconditionally).

Key separation

Two clan.core.vars generators on magnetite, stibnite-nix-build and stibnite-agent-session. No plaintext private key material is committed: each private half is committed encrypted under vars/per-machine/magnetite/ alongside its public value, decryptable by magnetite and by the users recorded beside it, and stibnite reads the public halves at evaluation time.

The two keys are separately authorized so each can be revoked or rotated independently. That separation is one-way, and the relation is worth stating the right way round.

The build key is authorized on nixbuild, a non-admin account that exists only to serve the protocol, with restrict,command="…/bin/nix-daemon --stdio". nix-daemon is exactly what an ssh-ng caller would have invoked, so the forced command serves both mechanisms and discards anything else the client sends: no pty, no forwarding, no shell, no other command. That bounds which program the key may run. It says nothing about the authority that program then holds: nixbuild is in nix.settings.trusted-users, which is store-root-equivalent on stibnite, since such a user can cause arbitrary paths to enter the store and influences what the daemon trusts. The grant is deliberate and required, because an untrusted account cannot receive the unsigned store paths a caller evaluated itself; the daemon rejects them as lacking a signature by a trusted key. magnetite's own builder account carries the same grant.

The session key is the broader of the two. It is an unrestricted login as crs58, an admin-group account, and stibnite's trusted-users is [ "root" "@admin" "@wheel" ], so that key already confers build authority and a shell besides. Only the build key is confined to the nix protocol.

One macOS-specific hazard is handled in configuration rather than left to be discovered: sshd gates on the com.apple.access_ssh service ACL, which on stibnite nests only the admin group, so a non-admin build account is refused before its key is consulted and the failure reads as Permission denied (publickey) with a correct key installed. Activation adds nixbuild to that ACL idempotently, under services.stibnite-build-host.authorizeSshAccessGroup for anyone who would rather manage it by hand.

CI is excluded, and by what

CI cannot reach this builder by construction: nixbot.toml sets attribute = "checks.x86_64-linux", so CI never evaluates or requests an aarch64-darwin derivation at all. As an independent second layer, modules/nixos/nixbot.nix and modules/nixos/buildbot.nix each set buildSystems = [ "x86_64-linux" ].

The rationale for keeping it that way is that stibnite is a laptop with no availability guarantee. Routing darwin attributes of pull requests at a machine that is regularly asleep would make its availability a merge gate. Opting in is a small change, and it belongs to a change prepared to accept that coupling or with an always-up darwin builder to point at.

Verification

Selection rationale: the diff is one new module, two machine files, one new check and one docs page, and only magnetite and stibnite import the new aspects, so the covering set is those two configurations plus the new check on both systems. The full fleet check set is not run; no other machine's evaluation reaches the changed files.

Command Result
nix build -L .#checks.aarch64-darwin.darwin-stibnite pass, realizes the stibnite system closure including etc-nixbuild
nix build -L .#checks.aarch64-darwin.stibnite-access-wiring pass
nix build -L .#checks.x86_64-linux.stibnite-access-wiring pass, built on magnetite through the existing remote builder
nix eval --raw .#nixosConfigurations.magnetite.config.system.build.toplevel.drvPath /nix/store/cxvbarhp06rlb4pg2ddiqv7cfy19m918-nixos-system-magnetite-26.11.20260804.85f6261.drv
just docs-format, just docs-lint clean, no fixes applied
nix fmt on the four changed nix files clean

The protocol the forced command names was exercised over ZeroTier against stibnite:

$ nix store info --store 'ssh-ng://crs58@stibnite.zt?remote-program=/nix/store/r9khzd3rsh8scnfs6qdd70xx3m4qzzxk-nix-2.34.8/bin/nix-daemon'
Version: 2.34.8
Trusted: 1

stibnite-access-wiring is the severe check: it asserts a pairing across two machines that neither machine's own evaluation can catch. Swapping the two keys, dropping the forced command, or letting the builder entry and the store URI name different accounts all activate cleanly and are all wrong, and each of those makes the check fail.

A subsequent validation pass re-ran the wiring check, evaluated both composed host configurations, and rendered the documentation page:

nix build --accept-flake-config .#checks.aarch64-darwin.stibnite-access-wiring --print-build-logs
nix eval --accept-flake-config --json .#nixosConfigurations.magnetite.config --apply …
nix eval --accept-flake-config --json .#darwinConfigurations.stibnite.config --apply …

Those confirm that magnetite registers stibnite as an aarch64-darwin ssh-ng builder under nixbuild, and that stibnite grants separate session access alongside a nix-daemon --stdio build key.

Not done, and why

A darwin build dispatched end to end through the new path has not been run. Local evidence proves the composed configuration and the authorization semantics, not a derivation crossing the ssh boundary and returning to magnetite. It needs two operator actions this branch cannot perform:

  1. clan machines update magnetite, which deploys the nix.buildMachines entry, the ssh alias and the private key to /run/secrets/vars/per-machine/magnetite/stibnite-nix-build/key.
  2. just activate on stibnite, in a graphical session, which creates the nixbuild account, installs its authorized-keys file, and adds it to the ssh ACL and to trusted-users. nix-darwin refuses to create a user over ssh without Full Disk Access.

After both, the end-to-end check from magnetite is:

nix store info --store ssh-ng://nixbuild@stibnite-builder
nix build --builders '@/etc/nix/machines' 'nixpkgs#legacyPackages.aarch64-darwin.hello' --system aarch64-darwin
nix build --store "$(cat /etc/nix/stibnite-store-uri)" 'nixpkgs#legacyPackages.aarch64-darwin.hello'

The first two land the closure in magnetite's store; the third leaves it in stibnite's.

No AGENTS.md was added: this repository keeps user-facing docs, development docs and a per-directory README.md tier and no agent-specific instruction files, so the durable knowledge from this work is in packages/docs/src/content/docs/concepts/darwin-build-access.md.

Review record

A validation pass over this branch raised ten findings. Seven were fixed and two were declined; the tenth is the end-to-end limitation recorded above. Review auto-fix was disabled, so each fix required an explicit response rather than being applied silently.

Fixed, all of them prose asserting something the code contradicted:

  • The remote-store description claimed no closure was copied in either direction. Inputs are written to the remote store; only the output copy back is avoided.
  • The comparison table named nixbot and buildbot-nix as the builder's intended callers. The builder is for development use, and those two are excluded.
  • Code comments in modules/system/stibnite-access.nix and modules/machines/nixos/magnetite/default.nix repeated that CI framing.
  • The page implied that restrict plus a forced command bounded the build account's authority, without disclosing that the account is a nix trusted user.
  • The session key was described as "not a build credential" when it is the broader credential of the two.
  • The page stated that no private key material is committed and that private halves never leave magnetite. Encrypted private halves are committed.
  • Remaining comments carrying those last two claims.

Declined, with reasons:

  • Removing services.stibnite-builder.storeUri and its emitted file as beyond scope. The option is a readOnly string computed from the build account, ssh alias and key path, emitted as one line of text. A URI string grants no access; the authorized key does, and that key is part of the intended change.
  • Parameterizing the two key generators to remove duplication. These are clan.core.vars generators whose encrypted secrets are already committed, and generator identity binds a committed secret, so a rename risks invalidating that binding for a cosmetic gain.

@mergify

mergify Bot commented Aug 31, 2026

Copy link
Copy Markdown

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

  • Queue this pull request

@cameronraysmith cameronraysmith changed the title stibnite-access: register stibnite as the fleet's aarch64-darwin build target feat(system): add stibnite as an aarch64-darwin build target Sep 1, 2026
@cameronraysmith
cameronraysmith force-pushed the fm/vx-stibnite-darwin-builder branch from 26c55c5 to cab37f9 Compare September 1, 2026 03:48
…ines

stibnite is the fleet's only aarch64-darwin machine. magnetite is
x86_64-linux and cannot build darwin derivations, and no nix.buildMachines
entry named stibnite existed, so darwin derivations had no build target.

modules/system/stibnite-access.nix exports three aspects. The nixos aspect
services.stibnite-builder computes a nix.buildMachines entry, which
magnetite splices into its own nix.buildMachines; the module never sets
that option itself. The same aspect computes a read-only storeUri and
materializes it at /etc/nix/stibnite-store-uri, for a caller that wants
nix build --store to run the build inside stibnite's store rather than
copying the output closure back. The darwin aspect
services.stibnite-build-host creates the nixbuild account on stibnite and
authorizes the two keys.

The build key is authorized on nixbuild with restrict and a forced
nix-daemon --stdio command, which is the program an ssh-ng caller would
have invoked, so the key can speak the protocol and nothing else. nixbuild
is also placed in nix.settings.trusted-users, which is store-root
equivalent on stibnite. That grant is required rather than incidental: an
untrusted account cannot receive the unsigned store paths a caller
evaluated itself, because the daemon rejects them as lacking a signature
by a trusted key. magnetite's own builder account carries the same grant.
The session key is authorized on crs58 without a forced command and is the
broader of the two credentials, since crs58 is in the admin group and
stibnite's trusted-users includes @admin.

Activation adds nixbuild to the com.apple.access_ssh service ACL. sshd
gates on that ACL, which on stibnite nests only the admin group, so a
non-admin build account is refused before its key is consulted and the
failure reads as Permission denied (publickey) with a correct key
installed.

The builder entry advertises aarch64-darwin alone, because nix config show
extra-platforms on stibnite reports that system and no other. maxJobs is 4
against 18 logical cores and 64 GiB on a machine that is also in
interactive use and commits 12 cores and 48 GiB to the rosetta VM.
…etite

The stibnite-nix-build generator produces the keypair magnetite uses to
reach stibnite's nix daemon. The private half is committed encrypted and
is decryptable by magnetite and by the users recorded alongside it; the
public half is committed in clear, and stibnite's configuration reads it
at evaluation time to authorize the nixbuild account. No plaintext private
key material is committed.
…magnetite

The stibnite-agent-session generator produces the keypair magnetite uses
for interactive ssh to stibnite as crs58. It is a separate generator from
stibnite-nix-build so that either key can be revoked or rotated without
disturbing the other. The private half is committed encrypted and the
public half in clear, as with the build key.
…hines

The wiring spans two machines, and neither machine's own evaluation can
catch a mismatch between them. Swapping the two keys, dropping the forced
command from the build key's authorized-keys entry, or letting magnetite's
buildMachines entry and its /etc/nix/stibnite-store-uri name different
accounts all evaluate and activate cleanly while being wrong.

stibnite-access-wiring evaluates both configurations together and asserts
the pairing: that magnetite's builder entry and store URI name the same
account and ssh alias, that stibnite authorizes the build key with restrict
and the forced nix-daemon --stdio command, and that the build account is a
trusted user. The check is defined for both x86_64-linux and
aarch64-darwin so either machine can run it.
A remote builder and a remote store are routinely treated as one thing.
They differ in where the output closure ends up, which is what decides
between them, so the distinction is written down alongside the wiring
rather than left to be rediscovered.

A remote builder copies input closures out to stibnite, builds there, and
copies the output closure back, so the result exists in the caller's
store. Under nix build --store ssh-ng:// the remote store is the store:
evaluation stays with the caller, the derivation and its source inputs are
written to stibnite, and the output stays there. The page also records the
authority the two keys actually carry, that CI cannot reach this builder
because nixbot.toml sets attribute to checks.x86_64-linux, and the
operator steps that activation still requires on each machine.
@cameronraysmith
cameronraysmith force-pushed the fm/vx-stibnite-darwin-builder branch from cab37f9 to c41729d Compare September 1, 2026 05:37
@cameronraysmith
cameronraysmith changed the base branch from main to fm/vx-magnetite-builder-key-restriction September 1, 2026 05:37
Base automatically changed from fm/vx-magnetite-builder-key-restriction to main September 1, 2026 06:01
@cameronraysmith
cameronraysmith merged commit c41729d into main Sep 1, 2026
8 checks passed
@cameronraysmith
cameronraysmith deleted the fm/vx-stibnite-darwin-builder branch September 1, 2026 06:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant