docs(notes): separate the three nixbot findings and price the check set

[cameronraysmith]

Documentation only, following #2914 which merged while this was being written. #2914 shipped the nix.settings change and an initial version of the note; this restructures the note around what the later measurements established, and adds the two sections #2914 could not contain because their evidence arrived after it merged.

No code changes. modules/machines/nixos/magnetite/default.nix is untouched by this PR and stays exactly as #2914 landed it.

Why the note is restructured

The original note told one story. There are three independent findings with three different owners, and presenting them as one misattributes two of them.

Finding 1, per-evaluation cost — the evaluator parked on the nix-daemon socket while every attribute's closure resolves against nine substituters. This is the only one that is a code change, and #2914 is it.

Finding 2, global serialization — both build services evaluate one pull request at a time by design (util.MasterLock("nix-eval") and eval_concurrency = 1), and neither is exposed by nixbot's NixOS module at pinned rev 660d7182. Architectural, accepted, not tuned.

Finding 3, queue load — a serial evaluator behind a standing queue of automated pull requests produces the observed wall clock arithmetically. This is a policy lever, not a configuration one.

New: finding 3, with its numbers

Measured: eval-gcroots advanced 171 → 186 between 01:15 and 02:00 — fifteen evaluations in forty-five minutes, ~180s each, separate runs with gaps rather than a retry loop.

The queue is larger than the prompting window suggested, and the count is corrected upward. gh-axi pr list --state open at 02:10: thirty open PRs, twenty-three bot-authored — seventeen vanixiets-flake-updater (2891, 2892, 2895-2909) and six renovate (2613, 2881, 2910-2913) — against seven human-authored. So 23 of 30, not the six of eight the window showed.

On a strictly serial evaluator, one evaluation per automated PR is 4140s ≈ 69 min at the measured mean, or 11270s ≈ 3h08m at build 172's 490s. A single slow-class member beats the whole fast sweep: build 171's 3312s consumed 55 minutes of the only evaluator, and build 168 consumed the full 3600s timeout then failed permanently. That is why PR 2890's check sat in progress for 100 minutes without its own evaluation being slow.

The findings compound rather than coexist. A flake-input bump invalidates the closure and converts warm paths to all-miss, so the automated PRs are each drawn from finding 1's expensive class and metered through finding 2's one-at-a-time queue. Seventeen of them are input bumps by construction. Options are stated as a genuine choice: batch the seventeen (loses per-input bisection), schedule or rate-limit whatever opens them (slower dependency freshness), or accept the queue knowingly — in which case a human PR's check latency is set by bot queue depth at landing and should not be read as a regression.

New: the check set, priced rather than counted

This is the input that makes finding 3 expensive, and it is in our hands rather than the tool's.

117 attributes, matching the service's recorded count: 11 FULL-ACT (6 home-manager activationPackage in home.nix:38; 5 NixOS system.build.toplevel in machines.nix:32), 5 FULL-eval, 12 MODULE, 89 CHEAP.

Measured evaluation cost, warm, --option eval-cache false to match nixbot: home-manager-crs58 9.26s / 2.38 GB / 20.4M thunks; nixos-magnetite 10.78s / 2.85 GB / 27.1M thunks; FULL-eval 2.58-3.62s; MODULE and CHEAP 1.43-1.63s against a 1.45s flake floor every check pays because the eval cache is disabled.

Derivation-closure paths via nix-store -q --requisites, 113 of 117 measurable on darwin: NixOS toplevels 20,065-22,170, home activations 15,338-19,165, every structural check 715. Summed 362,816, distinct union 29,244 — the union is the real fan-out ceiling because the daemon deduplicates per path within a run, and the sum overstates. The eleven FULL-ACT checks are 9.4% of attributes, 57.2% of summed paths, and 13,872 paths reachable only through them: 47.4% of the union. At ~1.27s per all-miss path a fully cold union bounds at ~10.3h; build 172's ~416s wait implies only a few hundred path-equivalents were actually unknown. Dropping FULL-ACT would cap that ceiling at 15,372 paths, −47%.

crs58's home config is evaluated 4x and magnetite's NixOS config 3x. nix-eval-jobs dispatches each attribute independently, so no memoisation is possible across attributes — only reducing what is bound as a check reduces fan-out.

Reducibility is answered in both directions rather than as a recommendation. FULL-ACT asserts that a closure builds, which no module-level evaluation can establish, so it is genuinely full in kind and reducible only in multiplicity: the six home activations occupy four distinct closure shapes, so canary-per-shape with the full set on scheduled runs costs per-user cache fill on PR runs — a tradeoff, not a free win. The five FULL-eval checks should be kept: they assert real cross-module wiring of the emitted configurations at 715-3,121 path closures, and a minimal re-composition would assert a reconstruction instead of the deployed configuration. The repo already holds the cheaper idiom in hm-sops-bridge-assertion-neg and structure/*, so the activation bindings are the outlier and were introduced deliberately for cache fill.

Correction on #2890's devin-worker.nix: it does contain two full home-manager evaluations (lines 76-98 and 125-145, probing aarch64-darwin and x86_64-linux), but they evaluate minimal probe configurations rather than this repository's homeConfigurations, and the check serializes only names, counts and booleans. On the closure-fan-out axis it is a cheap check. The follow-up proposing a third was not located — the PR body, all seven branch commit messages, both comments, the openspec corpus on that branch, and the devin files' own markers were searched. Recorded as not found rather than assumed; if it lives in Linear a pointer would let it be quoted.

Also revised

The storage audit verdict is unchanged and restated with its evidence intact: GC and optimise each fired successfully 63 of 63 retained journal days, tuning commits aca5c708b and e456caf62 are live and holding, /nix is 62.8G of a 250G quota, and storage is firmly not implicated.

The "will this restore 9m41s" answer is sharpened by finding 3: it compresses the wait that is 85% of a healthy evaluation, but it cannot return check latency to 9m41s while twenty-three automated PRs are queued ahead, because latency is queue depth × per-evaluation cost and #2914 addresses only the second factor.

Disclosure

The note now discloses a constraint breach during investigation. The check-set enumeration initially forced import-from-derivation, and this workstation delegates builds to ssh-ng://builder@magnetite, so about ten small derivations (cilium CRD YAML conversions) built there over roughly four minutes before the process was killed and --option builders '' was pinned for every subsequent invocation. No service state was mutated, nothing was cancelled, and no store-wide operation ran, but it was host contact the investigation was told not to make and it consumed a few minutes of the machine the operator was waiting on. Recorded in the note rather than omitted.

Verification

Documentation only, so treefmt and gitleaks are the applicable checks and both pass via pre-commit. No derivation changed: modules/ is untouched by this PR, which git diff --stat origin/main confirms as a single file under docs/notes/. The full check set is deliberately not run, since nothing it covers is modified.

---

Update: queue arithmetic shown, against twenty-three and per push

The automated share was already twenty-three of thirty here, measured independently via gh-axi pr list --state open rather than a truncated listing. What was missing is the working, so the tradeoff read as a judgement rather than as arithmetic. Added in 3ff227061.

One full cycle of the automated set, one evaluation per bot-authored PR, on measured inputs:

23 x 180 s =  4,140 s = 1 h 09 m   at the measured throughput mean
23 x 200 s =  4,600 s = 1 h 17 m   at the fast class (builds 173/176)
23 x 490 s = 11,270 s = 3 h 08 m   at build 172's healthy rate

One slow-class member displaces most of a cycle by itself: substituting build 171's 3,312s for a healthy 490s adds 2,822s / 47 min, and build 168 spent its full 3,600s timeout to produce no verdict at all.

A queue item is one evaluation per PUSH, not per PR, which multiplies that depth rather than adding to it. PR 2890 alone was force-pushed four times. The bot population's push rate was not measured, so it is carried as a parameter rather than asserted:

m = 1:  23 evaluations = 11,270 s =  3 h 08 m
m = 2:  46 evaluations = 22,540 s =  6 h 16 m
m = 4:  92 evaluations = 45,080 s = 12 h 31 m

The measured service rate bounds what the queue can absorb — fifteen evaluations in forty-five minutes is twenty per hour — so any arrival rate above twenty evaluations/hour grows the queue without bound rather than merely lengthening it.

Stated plainly, because it is the consequence worth deciding on: at this depth a human pull request can wait behind hours of automated evaluation. A human PR arriving behind one full automated cycle waits 3h08m at the healthy rate before its own evaluation begins, and 6h16m if each of those PRs is pushed twice. That is why PR 2890's check sat in progress for 100 minutes without its own evaluation ever being slow.

What batching saves, per cycle. Collapsing the flake updater's seventeen PRs into one combined PR takes a cycle from twenty-three evaluations to seven — the six dependency-bot PRs plus the combined one:

before: 23 x 490 s = 11,270 s = 3 h 08 m
after:   7 x 490 s =  3,430 s = 0 h 57 m
saved:  16 x 490 s =  7,840 s = 2 h 11 m per cycle  (69.6% reduction)

At the 180s mean the same collapse saves 16 x 180 = 2,880s / 48 min per cycle, and the saving scales with any push multiplier: 2h11m becomes 4h21m at m = 2.

Inferred and marked as such, because it makes batching cheaper than the evaluation count alone suggests: seventeen separate input bumps each pay their own cold fan-out over overlapping regions of the 29,244-path union, whereas one combined bump pays that union once — so the saving on finding 1's axis exceeds the 17-to-1 ratio in evaluation count.

The shape is deliberately unchanged. Batching and rate-limiting remain policy levers owned outside this repository, presented with their costs — lost per-input bisection, slower dependency freshness — against accepting the queue knowingly, which stays a legitimate answer. Recommendations, not decisions.

[tembo]

The parenthetical lists three distinct path counts (19,165; 15,342; 15,338) and they already account for all six users (2 + 3 + 1), so "four distinct closure shapes" doesn't match the evidence in the same sentence. If the enumeration is complete this should be three; the same "four" is reused in the Recommendations section ("the four distinct home closure shapes") and would need to match.

Suggested change

They are reducible in *multiplicity* rather than in kind: the six home activations occupy only three distinct closure shapes (crs58 and cameron identical at 19,165; christophersmith, janettesmith and raquel identical at 15,342; tara at 15,338), so a canary per shape per pull request with the full set on scheduled or main-branch runs preserves build coverage at a fraction of the per-pull-request exposure.

[mergify]

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

• Queue this pull request