feat(system): add stibnite as an aarch64-darwin build target - #2883
Merged
Conversation
|
Tick the box to add this pull request to the merge queue (same as
|
cameronraysmith
force-pushed
the
fm/vx-stibnite-darwin-builder
branch
from
September 1, 2026 03:48
26c55c5 to
cab37f9
Compare
…ines stibnite is the fleet's only aarch64-darwin machine. magnetite is x86_64-linux and cannot build darwin derivations, and no nix.buildMachines entry named stibnite existed, so darwin derivations had no build target. modules/system/stibnite-access.nix exports three aspects. The nixos aspect services.stibnite-builder computes a nix.buildMachines entry, which magnetite splices into its own nix.buildMachines; the module never sets that option itself. The same aspect computes a read-only storeUri and materializes it at /etc/nix/stibnite-store-uri, for a caller that wants nix build --store to run the build inside stibnite's store rather than copying the output closure back. The darwin aspect services.stibnite-build-host creates the nixbuild account on stibnite and authorizes the two keys. The build key is authorized on nixbuild with restrict and a forced nix-daemon --stdio command, which is the program an ssh-ng caller would have invoked, so the key can speak the protocol and nothing else. nixbuild is also placed in nix.settings.trusted-users, which is store-root equivalent on stibnite. That grant is required rather than incidental: an untrusted account cannot receive the unsigned store paths a caller evaluated itself, because the daemon rejects them as lacking a signature by a trusted key. magnetite's own builder account carries the same grant. The session key is authorized on crs58 without a forced command and is the broader of the two credentials, since crs58 is in the admin group and stibnite's trusted-users includes @admin. Activation adds nixbuild to the com.apple.access_ssh service ACL. sshd gates on that ACL, which on stibnite nests only the admin group, so a non-admin build account is refused before its key is consulted and the failure reads as Permission denied (publickey) with a correct key installed. The builder entry advertises aarch64-darwin alone, because nix config show extra-platforms on stibnite reports that system and no other. maxJobs is 4 against 18 logical cores and 64 GiB on a machine that is also in interactive use and commits 12 cores and 48 GiB to the rosetta VM.
…etite The stibnite-nix-build generator produces the keypair magnetite uses to reach stibnite's nix daemon. The private half is committed encrypted and is decryptable by magnetite and by the users recorded alongside it; the public half is committed in clear, and stibnite's configuration reads it at evaluation time to authorize the nixbuild account. No plaintext private key material is committed.
…magnetite The stibnite-agent-session generator produces the keypair magnetite uses for interactive ssh to stibnite as crs58. It is a separate generator from stibnite-nix-build so that either key can be revoked or rotated without disturbing the other. The private half is committed encrypted and the public half in clear, as with the build key.
…hines The wiring spans two machines, and neither machine's own evaluation can catch a mismatch between them. Swapping the two keys, dropping the forced command from the build key's authorized-keys entry, or letting magnetite's buildMachines entry and its /etc/nix/stibnite-store-uri name different accounts all evaluate and activate cleanly while being wrong. stibnite-access-wiring evaluates both configurations together and asserts the pairing: that magnetite's builder entry and store URI name the same account and ssh alias, that stibnite authorizes the build key with restrict and the forced nix-daemon --stdio command, and that the build account is a trusted user. The check is defined for both x86_64-linux and aarch64-darwin so either machine can run it.
A remote builder and a remote store are routinely treated as one thing. They differ in where the output closure ends up, which is what decides between them, so the distinction is written down alongside the wiring rather than left to be rediscovered. A remote builder copies input closures out to stibnite, builds there, and copies the output closure back, so the result exists in the caller's store. Under nix build --store ssh-ng:// the remote store is the store: evaluation stays with the caller, the derivation and its source inputs are written to stibnite, and the output stays there. The page also records the authority the two keys actually carry, that CI cannot reach this builder because nixbot.toml sets attribute to checks.x86_64-linux, and the operator steps that activation still requires on each machine.
cameronraysmith
force-pushed
the
fm/vx-stibnite-darwin-builder
branch
from
September 1, 2026 05:37
cab37f9 to
c41729d
Compare
cameronraysmith
changed the base branch from
main
to
fm/vx-magnetite-builder-key-restriction
September 1, 2026 05:37
Base automatically changed from
fm/vx-magnetite-builder-key-restriction
to
main
September 1, 2026 06:01
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Intent
Register stibnite as the fleet's aarch64-darwin build target so magnetite can build darwin derivations, with agent-session access and restricted nix-build access as separately authorized keys, and document the remote-builder versus remote-store distinction. Nothing beyond that.
What this changes
stibnite is declared as the fleet's aarch64-darwin build target, and the two ways of reaching its store are configured and documented as two mechanisms rather than one.
Before this branch there was no
nix.buildMachinesentry naming stibnite anywhere. The repository held two build-machine entries, both Linux: the rosetta VM and magnetite (modules/machines/darwin/stibnite/default.nix,modules/system/magnetite-builder.nix). magnetite is x86_64-linux and cannot build darwin derivations, so darwin work had no build target at all.The two mechanisms
modules/system/stibnite-access.nixexports three aspects.services.stibnite-builder(nixos) emits anix.buildMachinesentry. The caller's daemon copies the input closure out, builds on stibnite, and copies the output closure back, so the result exists in the caller's store. This is for development use: a developer or operator on magnetite who needs an aarch64-darwin result locally, for example building or testing darwin configurations.services.stibnite-builder.storeUri, materialized at/etc/nix/stibnite-store-uri, is the remote-store path. Undernix build --store ssh-ng://…evaluation stays with the caller, but the remote store is the store, so the derivation and its source inputs must exist there and are copied to stibnite; what is avoided is copying the output closure back. Intended caller: a machine whose store is empty and stays empty, such as an ephemeral runner, a fresh container, or an installer image.services.stibnite-sessionis interactive ssh under its own keypair and its own ssh alias. Its authority is described under "Key separation" below, and it is the broader of the two credentials rather than the narrower.Both build mechanisms speak
ssh-ngto the same account through the same alias, which the new check pins so they cannot drift apart. Legacyssh://, which would runnix-store --serve, is not served.Values justified rather than copied
systems = [ "aarch64-darwin" ]:nix config show extra-platformson stibnite reportsaarch64-darwinand nothing else, sox86_64-darwinwould route derivations the machine refuses.maxJobs = 4: measured 18 logical cores (12 performance, 6 efficiency) and 64 GiB, on a laptop in interactive use that commits 12 cores and 48 GiB to the rosetta VM and the same to colima. The remote share is a deliberate minority.speedFactor = 1: inert, because nix compares speed factors only among machines that can build the same system and there is no second darwin machine.supportedFeatures = [ "apple-virt" "big-parallel" ]: taken fromnix config show system-features, minusbenchmark(timings on an interactively loaded laptop are not measurements) andnixos-test(a Linux sandbox capability nix lists unconditionally).Key separation
Two
clan.core.varsgenerators on magnetite,stibnite-nix-buildandstibnite-agent-session. No plaintext private key material is committed: each private half is committed encrypted undervars/per-machine/magnetite/alongside its public value, decryptable by magnetite and by the users recorded beside it, and stibnite reads the public halves at evaluation time.The two keys are separately authorized so each can be revoked or rotated independently. That separation is one-way, and the relation is worth stating the right way round.
The build key is authorized on
nixbuild, a non-admin account that exists only to serve the protocol, withrestrict,command="…/bin/nix-daemon --stdio".nix-daemonis exactly what anssh-ngcaller would have invoked, so the forced command serves both mechanisms and discards anything else the client sends: no pty, no forwarding, no shell, no other command. That bounds which program the key may run. It says nothing about the authority that program then holds:nixbuildis innix.settings.trusted-users, which is store-root-equivalent on stibnite, since such a user can cause arbitrary paths to enter the store and influences what the daemon trusts. The grant is deliberate and required, because an untrusted account cannot receive the unsigned store paths a caller evaluated itself; the daemon rejects them as lacking a signature by a trusted key. magnetite's own builder account carries the same grant.The session key is the broader of the two. It is an unrestricted login as
crs58, an admin-group account, and stibnite'strusted-usersis[ "root" "@admin" "@wheel" ], so that key already confers build authority and a shell besides. Only the build key is confined to the nix protocol.One macOS-specific hazard is handled in configuration rather than left to be discovered: sshd gates on the
com.apple.access_sshservice ACL, which on stibnite nests only the admin group, so a non-admin build account is refused before its key is consulted and the failure reads asPermission denied (publickey)with a correct key installed. Activation addsnixbuildto that ACL idempotently, underservices.stibnite-build-host.authorizeSshAccessGroupfor anyone who would rather manage it by hand.CI is excluded, and by what
CI cannot reach this builder by construction:
nixbot.tomlsetsattribute = "checks.x86_64-linux", so CI never evaluates or requests an aarch64-darwin derivation at all. As an independent second layer,modules/nixos/nixbot.nixandmodules/nixos/buildbot.nixeach setbuildSystems = [ "x86_64-linux" ].The rationale for keeping it that way is that stibnite is a laptop with no availability guarantee. Routing darwin attributes of pull requests at a machine that is regularly asleep would make its availability a merge gate. Opting in is a small change, and it belongs to a change prepared to accept that coupling or with an always-up darwin builder to point at.
Verification
Selection rationale: the diff is one new module, two machine files, one new check and one docs page, and only magnetite and stibnite import the new aspects, so the covering set is those two configurations plus the new check on both systems. The full fleet check set is not run; no other machine's evaluation reaches the changed files.
nix build -L .#checks.aarch64-darwin.darwin-stibniteetc-nixbuildnix build -L .#checks.aarch64-darwin.stibnite-access-wiringnix build -L .#checks.x86_64-linux.stibnite-access-wiringnix eval --raw .#nixosConfigurations.magnetite.config.system.build.toplevel.drvPath/nix/store/cxvbarhp06rlb4pg2ddiqv7cfy19m918-nixos-system-magnetite-26.11.20260804.85f6261.drvjust docs-format,just docs-lintnix fmton the four changed nix filesThe protocol the forced command names was exercised over ZeroTier against stibnite:
stibnite-access-wiringis the severe check: it asserts a pairing across two machines that neither machine's own evaluation can catch. Swapping the two keys, dropping the forced command, or letting the builder entry and the store URI name different accounts all activate cleanly and are all wrong, and each of those makes the check fail.A subsequent validation pass re-ran the wiring check, evaluated both composed host configurations, and rendered the documentation page:
Those confirm that magnetite registers stibnite as an
aarch64-darwinssh-ngbuilder undernixbuild, and that stibnite grants separate session access alongside anix-daemon --stdiobuild key.Not done, and why
A darwin build dispatched end to end through the new path has not been run. Local evidence proves the composed configuration and the authorization semantics, not a derivation crossing the ssh boundary and returning to magnetite. It needs two operator actions this branch cannot perform:
clan machines update magnetite, which deploys thenix.buildMachinesentry, the ssh alias and the private key to/run/secrets/vars/per-machine/magnetite/stibnite-nix-build/key.just activateon stibnite, in a graphical session, which creates thenixbuildaccount, installs its authorized-keys file, and adds it to the ssh ACL and totrusted-users. nix-darwin refuses to create a user over ssh without Full Disk Access.After both, the end-to-end check from magnetite is:
The first two land the closure in magnetite's store; the third leaves it in stibnite's.
No
AGENTS.mdwas added: this repository keeps user-facing docs, development docs and a per-directoryREADME.mdtier and no agent-specific instruction files, so the durable knowledge from this work is inpackages/docs/src/content/docs/concepts/darwin-build-access.md.Review record
A validation pass over this branch raised ten findings. Seven were fixed and two were declined; the tenth is the end-to-end limitation recorded above. Review auto-fix was disabled, so each fix required an explicit response rather than being applied silently.
Fixed, all of them prose asserting something the code contradicted:
modules/system/stibnite-access.nixandmodules/machines/nixos/magnetite/default.nixrepeated that CI framing.restrictplus a forced command bounded the build account's authority, without disclosing that the account is a nix trusted user.Declined, with reasons:
services.stibnite-builder.storeUriand its emitted file as beyond scope. The option is areadOnlystring computed from the build account, ssh alias and key path, emitted as one line of text. A URI string grants no access; the authorized key does, and that key is part of the intended change.clan.core.varsgenerators whose encrypted secrets are already committed, and generator identity binds a committed secret, so a rename risks invalidating that binding for a cosmetic gain.