fleet updates: per-host immutable candidate adoption — stage, gate, switch, probe, roll back

[mecattaf]

Decision from the house-computer inquiry

Stay on NixOS. Add no Proxmox, Kubernetes, K3s, or fourth node for this. The existing three-node fleet is enough:

• coordinator — Strix Halo, daily desktop and primary local-inference node;
• worker — Strix Halo, second inference/compute node;
• nas — 24 GiB, always-on storage/network/service node and the existing build/cache publisher.

The useful finding is narrower: the fleet already builds immutable host closures centrally, but it has no equally good adoption plane. hosts/nas/update-center.nix resolves one immutable main revision, builds coordinator and worker, and pushes their closures to the NAS Attic cache. It deliberately touches no endpoint. Adoption is still a manual nixos-rebuild switch --flake ... on each machine.

Build/publish and activate are correctly separated. Complete that design rather than replacing it.

Desired model

The NAS publishes a signed, immutable candidate per host. Each host independently:

1. discovers the newest eligible candidate;
2. downloads/substitutes it in the background;
3. verifies that exact closure locally;
4. waits until that host is safe to disturb;
5. atomically advances its NixOS system profile and activates it;
6. runs bounded health probes;
7. marks the generation known-good or rolls back locally.

This is a rolling fleet update made from NixOS and systemd primitives. It is not an all-host transaction and it is not a new generic scheduler. One unavailable or busy host must not stop another from updating; one bad host result must not bless or poison the candidate for another host.

Nix already gives the important atomicity: a host switches its system profile from one complete closure to another, and rollback-offline plus retained generations provide a network-independent escape hatch. The missing work is candidate identity, per-host admission, health, and durable reporting.

Candidate contract

Extend the update center with a small manifest per host, published only after the build and Attic push succeed. At minimum:

{
  "schema": 1,
  "host": "coordinator",
  "rev": "github:mecattaf/dotfiles/<rev>",
  "store_path": "/nix/store/...-nixos-system-coordinator-...",
  "built_at": "...",
  "channel": "rolling",
  "requires_reboot": false
}

The exact integrity field can be the signed binary-cache trust plus a hash/closure receipt; do not invent a second package-signing system if Attic/Nix already proves the bytes. What matters is that a device adopts the exact closure the NAS proved and never re-resolves moving main during activation.

Publishing must be per host. If the worker builds but the coordinator does not, the worker candidate may advance while the coordinator pointer remains at its last good revision.

Endpoint state machine

Declare a fleet-wide module with a small system service and timers, host-configurable:

idle
  -> candidate-seen
  -> closure-ready
  -> waiting-for-safe-window
  -> activating
  -> probing
  -> known-good
             \
              -> failed -> rollback -> rolled-back

Persist only compact receipts: candidate revision/store path, discovery/download/activation times, preflight refusal reason, health verdict, previous generation, rollback result. The journal remains the detailed event log; modules/failure-surfacing.nix remains the failure path.

Download/build work and activation are two different units. A machine may warm the new closure while Chrome, Herdr, or an inference job is busy. It must not restart affected services merely because the bytes arrived.

Safe activation gates

The gates are local facts, not a fleet-wide lock:

• do not switch either Strix while its flashnext-lane.target is active;
• do not disturb llama-swap during a live inference request or a dual-node run;
• do not restart Herdr merely because a rolling package set moved; Herdr owns live PTYs and keeps its deliberately controlled pin/restart policy;
• refuse while another activation is running;
• require enough free disk for the new closure while preserving retained known-good generations;
• optionally defer on severe memory or I/O pressure;
• report the exact refusal and retry later instead of turning “busy” into failure.

Use narrow probes/adapters for those facts. Do not make Tally the general fleet-update scheduler again. A Tally lease may be one useful input for a GPU-active check, but update correctness must not depend on the Tally lake or a Cloudflare connection.

Reboots and service restarts

Treat “closure adopted” and “new kernel booted” separately:

• ordinary userspace changes can switch after the local gate passes;
• a kernel/initrd change may install with boot and expose reboot-required without forcing an unattended reboot through an active desktop or agent session;
• define the eventual reboot policy per host rather than silently leaving a new kernel unbooted forever.

The coordinator and worker can follow a fast rolling channel without rebooting in lockstep. The NAS owns routing/storage and currently has a deliberately slower/manual update policy; this issue must make that policy explicit and review whether it still matches the new goal, not silently enroll the NAS in nightly rebooting.

Health and rollback

A successful switch-to-configuration exit is necessary but not a known-good verdict. Define a short host-specific probe set, for example:

• system and user managers are reachable and have no newly failed critical units;
• mesh/LAN reachability still works;
• mounts required by the host are present;
• on Strix nodes, the declared llama-swap service/gateway is healthy when it is meant to be active;
• on the NAS, network, storage, Attic and other explicitly critical services remain healthy.

Probe failure records evidence, activates the previous retained generation locally, probes again, and leaves a loud failure marker. Never garbage-collect the previous/booted/last-known-good generations as part of the update transaction; modules/gc-retention.nix and rollback-offline are the existing recovery foundation.

Freshness without surprise

“Bleeding edge” means candidates are produced frequently and endpoints converge promptly when safe. It does not mean every upstream movement automatically tears down live state.

Define input classes explicitly:

• rolling inputs eligible for routine candidate publication;
• restart-sensitive pins such as Herdr, advanced deliberately even if kept current;
• host policy (rolling, stage-only, or manual) controlling adoption/reboot, not package freshness at build time.

Surface age as a first-class fact: current revision, newest candidate, last attempt, last refusal, last known-good, and whether a reboot is pending. An old node should be visibly old, never silently stale.

Work items

• Specify/version the per-host candidate manifest and atomic pointer publication.
• Refactor update-center so one host's build/push failure does not prevent successful host candidates from advancing.
• Add a fleet update-adopt module: discovery, background substitution, local state/receipts, and timers with jitter.
• Separate closure staging from activation.
• Implement the local safe-to-switch gate with explicit refusal reasons for GPU lanes, inference, concurrent activation, disk and pressure.
• Determine changed-unit/reboot impact without restarting anything during staging.
• Add host-specific post-activation probes and automatic local rollback on a failed verdict.
• Join failure outcomes to the existing failure-marker/journal surface.
• Expose freshness and pending-reboot state to the fleet-status issue.
• Record the NAS adoption/reboot channel explicitly; no accidental nightly NAS reboot.
• Test with one good candidate, one host-specific build failure, one busy-host deferral, one failed health probe, one offline rollback, and one reboot-required candidate.

Acceptance

• coordinator and worker can be on different good revisions without either being considered a broken deployment.
• The NAS can publish a good worker candidate even when the coordinator build fails, and vice versa.
• An endpoint downloads no source and performs no moving-ref evaluation at activation time; it activates the exact published store path.
• A busy inference/TP=2 lane causes a durable deferral, not eviction.
• A normal userspace candidate stages and activates in the background on one idle host without touching the other.
• A bad activation returns that host to its prior retained generation and emits a visible receipt/failure marker.
• Kernel changes become an explicit pending reboot and follow the host's declared reboot policy.
• The update path works on the LAN with Cloudflare/Tally unavailable.
• No Kubernetes/K3s/Proxmox component is introduced.

Relevant surfaces

• hosts/nas/update-center.nix — existing immutable resolve/build/push producer.
• hosts/nas/attic.nix and modules/common.nix — cache and trust.
• modules/gc-retention.nix and modules/rollback-offline.nix — retained recovery.
• modules/failure-surfacing.nix — durable visible failures.
• modules/flashnext-lane.nix and modules/llama-swap.nix — inference disturbance gates.

Superseding context: the old push/all-host fleet deploy was retired in #228. This issue completes the pull model; it does not resurrect it.

September 15 final backlog ruling — verify the current update center

Tom's I39 requests a bounded verification of the existing NAS update-center and actual host adoption, recorded here rather than a new duplicate project. Reconcile this older issue with current deployment before deciding whether any implementation remains. Older model names and topology examples above are historical, not instructions to reinstate them.

• Record the current NAS candidate publication and host-adoption state with concrete evidence for coordinator and worker.
• Identify a demonstrated residual only; retain rollout/recovery evidence.

This verification/any resulting digital work belongs in a scoped Tally flow after Tally readiness. It does not make normal runtime update correctness depend on the cloud SDK, and it does not initiate a fleet switch now.

[mecattaf]

Status 2026-09-14 — implemented and deployed; stays open until the first real unattended adoption.

Proven live: update-center publishes a signed per-host manifest (http://nas:8734/candidates/<host>/), a candidate identical to the running generation is a no-op, and a live downgrade on the worker was refused (local-generation-newer). Private tally inputs are seeded onto the NAS from the coordinator (three trees GC-rooted; a fresh-HOME nix eval on the NAS succeeds). Policies: worker rolling, coordinator stage-only, client manual, NAS not enrolled.

Remaining before close:

1. First clean NAS nightly: ssh root@nas journalctl -u update-center --since today shows seed present: tally-b/tally-lake and no failure.
2. Worker adopts a newer candidate on its own (runbook steps 5–6).
3. DF-354-1: flip the coordinator to rolling, switch, confirm update-adopt-activate.timer, delete the row.
4. Runbook fix: the nix copy publish step needs --no-check-sigs (locally built paths are unsigned); a 404 manifest logs fetch-failed where no-candidate would be clearer.

[mecattaf]

Item 4 of the 2026-09-14 status comment is done in #408 (branch fix/update-adopt-cli-honesty), together with the two neighbouring CLI answers that read as "fine" when they are not. The issue stays open: items 1–3 there are live-adoption milestones, not code.

• --no-check-sigs runbook note — new docs/fleet-candidate-adoption.md covers nix copy --to ssh-ng://root@nas --no-check-sigs <closure> followed by update-center --publish-only, why the flag belongs on that line (a locally built path is unsigned; the daemon honours the flag only for a trusted user), and the three places it is never legitimate.
• A 404 manifest now logs no-candidate, not fetch-failed. curl gets -w %{http_code}, so "the NAS published nothing for this host" and "the NAS did not answer" are different journal lines; both still exit 0. Measured against the live publisher: unpublished host rc 22 / 404, published host rc 0 / 200, unreachable endpoint rc 7 / 000.
• update-adopt status run unprivileged exits 1 instead of printing "state": "idle" for state it never read. /var/lib/update-adopt is 0700 root, so a run as tom now prints only what /run and /nix show, sets state_unreadable, nulls the state-derived fields, and fails. Verified on the coordinator as tom against the real state directory: deployed build rc 0, this branch rc 1.

33 hermetic tests pass in tree and through nix build .#checks.x86_64-linux.update-adopt. nix flake check does not pass on this branch or on main — checks.x86_64-linux.fleet-connectivity fails evaluation on an unrelated model-roster assertion, confirmed pre-existing.

Not deployed: no nixos-rebuild, no home-manager switch. The live coordinator still runs the old build, so the exit-code change lands at the next switch.