policy: per-target build fan-out — path-selected dispatch on push, scheduled full sweep for badges, compile the tool once, save caches only from main

[zackees]

Lessons from FastLED/fbuild (an embedded build tool that ships ~80 per-board build workflows, each with a README status badge), applied and measured in FastLED/fbuild#1574. I'm proposing them as fleet policy for any repo with a large per-target matrix (boards, platforms, examples).

Problem shapes found

1. Every matrix cell recompiled the same tool. Each of 76 board jobs ran soldr cargo build -p fbuild-cli -p fbuild-daemon, the same board-independent debug build: about 350 s of each ~440 s job, and 33,338 s of runner time per sweep.
2. Branch runs thrashed the cache. One non-main sweep saved 151 entries (79 per-target toolchain caches of ~1 GB each), which took the repo to 35.8 GB. The oldest surviving entry was 25 minutes old, and main's Rust build cache had been evicted. The shared compile then ran at 0/621 zccache hits.
3. Badges froze silently. A badge tracks runs of its own workflow file. When the per-target workflows became workflow_dispatch/workflow_call-only and the sweep called template_build.yml from one matrix, no build-<board>.yml ever ran as its own run again, so all 67 badges stayed stuck on a two-week-old state. Nothing fails when this happens.
4. Every target on every change is wasteful, but taking path triggers away entirely is what caused problem 3.

What worked

A. Compile once, fan out the binary (extends policy-rust "compile once, runners only execute")

One fbuild_bin job builds and uploads the binaries (retention 1 day). The reusable per-target template takes fbuild-artifact + fbuild-run-id inputs and uses actions/download-artifact with run-id + github-token, so it works across runs. With the artifact present, the target job skips setup-soldr, apt and the Rust compile. Running a target workflow standalone keeps a compile fallback.

B. Save base caches only from the default branch (CACHE-003 in practice)

setup-soldr save-cache: ${{ github.ref == 'refs/heads/main' && 'true' || 'false' }}, plus if: always() && github.ref == 'refs/heads/main' on every actions/cache/save. Branches and PRs restore main's copies. Note: setup-soldr's save-cache: auto only suppresses pull_request events. workflow_dispatch/push on a feature branch still save, and with per-target GB-sized caches that's enough to evict everything.

C. A badge-preserving dispatcher, path-selected on push and full on a schedule

One generated workflow (push: [main], schedule, workflow_dispatch):

• plan: a script maps the push diff (github.event.before..sha) to targets. Each target's trigger paths are its own test dir, its family's source paths, and its own workflow file; shared/core paths select only a small set of "core" targets, one per toolchain family. A schedule or manual dispatch selects all targets, and a missing or zero before selects all. The scheduled run is skipped when there were no commits in 24 h.
• fbuild_bin: compile once (A).
• dispatch (permissions: actions: write): gh workflow run <target>.yml --ref $GITHUB_REF_NAME -f fbuild-run-id=$GITHUB_RUN_ID -f checkout_ref=$GITHUB_SHA for each selected target. A workflow_dispatch created by GITHUB_TOKEN is the one event that does start new runs, and each target becomes its own run, so its badge moves.
• It does not call the target workflows as reusable workflows: GitHub caps a workflow file at 20 unique reusable workflows, and exceeding it fails the run with zero jobs and no logs.
• PRs don't run the full target set unless explicitly labelled.

Measurements (FastLED/fbuild)

	before	after
Full sweep runner time	33,338 s (76 jobs, avg 438 s)	5,035 s (80 dispatched runs, avg 54 s, plus a 650 s dispatcher)
Board runs per push to main (replaying 60 real pushes)	4,800 if all targets ran	380 (7%); 14/60 pushes selected 0 targets, 29/60 selected 3
Shared compile cache hit rate while branch sweeps saved	0/621	branch saves are gone (warm-main number pending merge)
Repo cache size	35.8 GB	8.3 GB after purging branch entries
Badge freshness	frozen since the path triggers were removed	refreshed on every relevant push and daily

Proposed rules (candidates)

• FANOUT-001: a matrix or fan-out of N ≥ ~5 target jobs must not compile the same toolchain-independent tool in each cell. Build it once and pass it as an artifact (cross-run capable).
• FANOUT-002: when targets publish per-workflow status badges, each badge's workflow must be triggered as its own run on a schedule (at least daily when there are commits) and on relevant pushes to the default branch. A statically checkable signal: a badge in README → a workflow file whose on: has neither schedule/push nor a dispatcher that gh workflow runs it.
• FANOUT-003: selection on push must be path-derived from a single source of truth (target → paths), with shared paths selecting a bounded core set rather than every target. The full sweep is scheduled, not per-PR. An unknown diff selects all targets and never skips coverage.
• CACHE-00x (sharpening CACHE-003): setup-soldr save-cache and actions/cache/save must be gated on the default branch, not left at auto, whenever non-PR events (dispatch, push to feature branches) can run the job.

Reference implementation: FastLED/fbuild#1574 (ci/select_boards.py, ci/render_workflows.py → nightly-platforms.yml, template_build.yml).

[zackees]

Follow-up measurement from FastLED/fbuild#1574: measure per-target caches against a cold run before keeping them. fbuild's ~1 GB per-board toolchain caches were a net loss: over 50 boards, cold downloads took 1,738 s vs 2,248 s with cache restore + save. Only one family, whose 480 MB upstream archive downloads at a few MB/s, benefits. The cache is now opt-in per family, and the SOT records the measurement that justifies each opt-in. It's keyed on a measured property, in the spirit of RUST-007, not on board names. Also: a restore-keys fallback to any target's cache (fbuild-toolchain-Linux-) restores a GB of mostly wrong toolchain on a miss. Candidate rule: a cache family must be justified by a cold-vs-warm measurement, and restore-keys must not fall back across targets.

[zackees]

Another measured lesson (FastLED/fbuild#1574): unit tests that sleep through production retry backoffs or grace periods. Three fbuild tests cost ~130 s of PR-gate time: exponential backoff 1+2+4+8 s × 5 tests serialized behind a shared guard, a real 30 s "missing owner" grace, and a DNS failure retried with production backoff. Injecting the timing (the code already had a RetryTiming; the tests just didn't use it everywhere), planting the precondition directly, and failing with a non-retryable 4xx took them from 118 s to 2 s locally. Candidate check: flag any unit-test binary whose runtime exceeds a budget ([rust.tests].max-seconds, alongside RUST-012's size budget) using libtest's own finished in output, and have ci-lint name the offending tests (has been running for over 60 seconds is already in the log).

[zackees]

One more from FastLED/fbuild#1574: don't run cargo check before cargo clippy over the same target set. Clippy is a superset of check, and running check first only pre-warms dependency rmeta while re-checking every workspace crate. Measured on fbuild with fresh target dirs and warm zccache: check + clippy took 194.5 s vs 106.4 s for clippy alone. Candidate static rule: flag a job where cargo check and cargo clippy run with the same --workspace/--all-targets/feature/target arguments.