feat: fbuild install, cache observability and split CI caches (#1433)

.github/actions/setup/README.md

@@ -51,6 +51,28 @@ jobs:
           }
 ```
 
+## Split caches (`cache-mode: split`)
+
+For large board matrices, split mode keeps packages shared per platform family and build payloads per board (FastLED/fbuild#1433). It needs an fbuild release with `fbuild install`.
+
+```yaml
+- uses: FastLED/fbuild/.github/actions/setup@main
+  id: fbuild
+  with:
+    cache-mode: split
+    environments: ${{ matrix.board }}
+    cache-key-extra: ${{ hashFiles('platformio.ini') }}
+    save: ${{ github.event_name != 'pull_request' }}
+- run: fbuild build examples/Blink -e ${{ matrix.board }}
+```
+
+In split mode the action:
+
+1. Runs `fbuild install --dry-run --json` to learn each environment's platform, without touching the network.
+2. Restores the packages cache (`toolchains`, `platforms`, `packages`, `libraries`, `archives`, `installed`, `index.sqlite`) by the prefix `fbuild-pkgs-<cache-version>-<os>-<arch>-<family>-`.
+3. Runs `fbuild install` as its own step, then saves the packages cache under that prefix plus the `packages_hash`. It skips the save when the restored key already matches.
+4. Restores the build-payload cache (`core`, `framework-libs`, `library-selection`, the zccache store) keyed by fbuild hash, board and `cache-key-extra`, without falling back to other boards. It is saved at the end of the job unless `save` is `false`.
+
 ## Caching the zccache store
 
 The built-in `cache: true` wiring covers the fbuild package/tool cache rooted at `FBUILD_CACHE_DIR`. If you also want cross-run reuse of zccache's object store, add your own `actions/cache@v5` step for the resolved zccache directory and your project build outputs:
@@ -94,6 +116,11 @@ Use `steps.<id>.outputs.zccache-store-path` inside workflow expressions. The sam
 | `fbuild-version` | `latest` | PyPI version spec. Pin to an exact version (`2.1.16`) for reproducible CI. |
 | `python-version` | `3.12` | Python used to install fbuild. Must be >= 3.9. |
 | `cache` | `true` | Set to `false` to install fbuild without wiring `actions/cache`. |
+| `cache-mode` | `combined` | `combined` keeps one `FBUILD_CACHE_DIR` entry per key. `split` restores a packages cache shared per platform family and a per-board build-payload cache, and runs `fbuild install` as its own step. |
+| `save` | `true` | `false` restores caches without saving; use it on pull requests. |
+| `project-dir` | `.` | Split mode: project passed to `fbuild install`. |
+| `environments` | `""` | Split mode: space-separated environments to provision. Required in split mode. |
+| `board` | `""` | Split mode: name in the build-payload cache key. Defaults to `environments` joined with `_`. |
 | `cache-key-extra` | `""` | String baked into the cache key. Use `hashFiles(...)` over your graph inputs so edits invalidate stale artifacts. |
 | `cache-version` | `v1` | Manual cache bump. Increment when you want to force-invalidate across your matrix. |
 | `cache-dir` | `$RUNNER_TEMP/fbuild-cache` | Override if you need a different cache root. |
@@ -103,7 +130,11 @@ Use `steps.<id>.outputs.zccache-store-path` inside workflow expressions. The sam
 
 | Output | Description |
 |---|---|
-| `cache-hit` | `true` if the cache was restored from a previous run, `false` on miss. |
+| `cache-hit` | Combined mode: `true` if the cache was restored from an exact key match, `false` otherwise. |
+| `platform-family` | Split mode: the platform family the packages cache is shared across. |
+| `packages-hash` | Split mode: `packages_hash` from `fbuild install --json`. |
+| `packages-cache-hit` | Split mode: the restored packages cache key, empty on a miss. |
+| `build-cache-hit` | Split mode: `true` if the build-payload cache was restored from an exact key match. |
 | `cache-dir` | Resolved cache directory path. Useful for diagnostic steps. |
 | `fbuild-hash` | sha256 prefix (16 hex chars) of the installed fbuild wheel's `RECORD` file. Baked into the cache key so any fbuild change, including a re-released wheel at the same version, invalidates stale cache artifacts. |
 | `zccache-store-path` | Resolved zccache object-store directory. The same path is exported to later steps as `ZCCACHE_DIR`, so consumer-managed `actions/cache@v5` blocks can reuse it without guessing platform-specific defaults. |
@@ -117,7 +148,7 @@ Use `steps.<id>.outputs.zccache-store-path` inside workflow expressions. The sam
 5. Installs fbuild from PyPI at the requested version (skipped on install-cache hit). Install uses `pip install --target=$RUNNER_TEMP/fbuild-install` so the cached directory is the entire install surface.
 6. Activates the install dir by appending `bin/` (POSIX) and `Scripts/` (Windows) to `$GITHUB_PATH` and prepending `PYTHONPATH`.
 7. **Computes the installed fbuild's content hash** (sha256 of its dist-info `RECORD`) and bakes it into the **build artifact** cache key. This guarantees the artifact cache is tied to the exact fbuild you're running, not just the PyPI version string, so `latest` is safe and a re-released wheel won't poison the cache.
-8. Restores (and on job-end, saves) the fbuild build artifact cache via `actions/cache@v5`.
+8. Restores (and on job-end, saves) the fbuild build artifact cache via `actions/cache@v5`. With `save: false` it only restores. In `cache-mode: split` this step is replaced by the packages and build-payload caches described in [Split caches](#split-caches-cache-mode-split).
 
 ### Why hash-pinning matters
 

.github/actions/setup/action.yml

@@ -18,6 +18,26 @@ inputs:
     description: "Whether to save/restore the fbuild cache via actions/cache. Set to 'false' to skip caching (install only)."
     required: false
     default: "true"
+  cache-mode:
+    description: "'combined' (default) keeps one FBUILD_CACHE_DIR entry per key. 'split' restores a packages cache shared per platform family plus a per-board build-payload cache, and runs `fbuild install` as its own step (FastLED/fbuild#1433). 'split' needs `environments` and an fbuild release with `fbuild install`."
+    required: false
+    default: "combined"
+  save:
+    description: "Whether to save caches. 'false' only restores; use it on pull requests so PR runs don't add entries of their own to the repository's cache budget."
+    required: false
+    default: "true"
+  project-dir:
+    description: "Project directory passed to `fbuild install` in split mode."
+    required: false
+    default: "."
+  environments:
+    description: "Space-separated environments to provision with `fbuild install` in split mode."
+    required: false
+    default: ""
+  board:
+    description: "Name for the per-board build-payload cache key in split mode. Defaults to `environments` joined with '_'."
+    required: false
+    default: ""
   cache-key-extra:
     description: "Extra string baked into the cache key. Typically hashFiles('platformio.ini', ...) so graph-input changes invalidate stale artifacts."
     required: false
@@ -37,8 +57,8 @@ inputs:
 
 outputs:
   cache-hit:
-    description: "true if the fbuild cache was restored from a previous run, false on miss."
-    value: ${{ steps.fbuild-cache.outputs.cache-hit }}
+    description: "Combined mode: true if the fbuild cache was restored from an exact key match, false otherwise."
+    value: ${{ steps.fbuild-cache.outputs.cache-hit || steps.fbuild-cache-restore-only.outputs.cache-hit }}
   cache-dir:
     description: "Resolved cache directory path (useful for diagnostic steps)."
     value: ${{ steps.resolve-paths.outputs.cache-dir }}
@@ -48,6 +68,18 @@ outputs:
   fbuild-hash:
     description: "sha256 prefix of the installed fbuild wheel's RECORD file. Baked into the cache key so an fbuild upgrade invalidates stale artifacts."
     value: ${{ steps.fbuild-hash.outputs.fbuild-hash }}
+  platform-family:
+    description: "Split mode: the platform family (e.g. 'espressif32') the packages cache is shared across."
+    value: ${{ steps.split-plan.outputs.family }}
+  packages-hash:
+    description: "Split mode: `packages_hash` reported by `fbuild install --json` for the provisioned environments."
+    value: ${{ steps.split-install.outputs.packages-hash }}
+  packages-cache-hit:
+    description: "Split mode: the packages cache key that was restored, or empty on a miss."
+    value: ${{ steps.packages-restore.outputs.cache-matched-key }}
+  build-cache-hit:
+    description: "Split mode: true if the per-board build-payload cache was restored from an exact key match."
+    value: ${{ steps.build-cache.outputs.cache-hit || steps.build-cache-restore-only.outputs.cache-hit }}
 
 runs:
   using: "composite"
@@ -177,8 +209,10 @@ runs:
         echo "fbuild-hash=${FBUILD_HASH}" >> "$GITHUB_OUTPUT"
         echo "Resolved fbuild content hash: ${FBUILD_HASH}"
 
+    # ── Combined mode (default): one FBUILD_CACHE_DIR entry per key ─────────
+
     - name: Restore fbuild cache
-      if: ${{ inputs.cache == 'true' }}
+      if: ${{ inputs.cache == 'true' && inputs.cache-mode != 'split' && inputs.save == 'true' }}
       id: fbuild-cache
       uses: actions/cache@v5
       with:
@@ -187,3 +221,130 @@ runs:
         restore-keys: |
           fbuild-${{ inputs.cache-version }}-${{ runner.os }}-${{ runner.arch }}-${{ steps.fbuild-hash.outputs.fbuild-hash }}-
           fbuild-${{ inputs.cache-version }}-${{ runner.os }}-${{ runner.arch }}-
+
+    - name: Restore fbuild cache (no save)
+      if: ${{ inputs.cache == 'true' && inputs.cache-mode != 'split' && inputs.save != 'true' }}
+      id: fbuild-cache-restore-only
+      uses: actions/cache/restore@v5
+      with:
+        path: ${{ steps.resolve-paths.outputs.cache-dir }}
+        key: fbuild-${{ inputs.cache-version }}-${{ runner.os }}-${{ runner.arch }}-${{ steps.fbuild-hash.outputs.fbuild-hash }}-${{ inputs.cache-key-extra }}
+        restore-keys: |
+          fbuild-${{ inputs.cache-version }}-${{ runner.os }}-${{ runner.arch }}-${{ steps.fbuild-hash.outputs.fbuild-hash }}-
+          fbuild-${{ inputs.cache-version }}-${{ runner.os }}-${{ runner.arch }}-
+
+    # ── Split mode (FastLED/fbuild#1433) ───────────────────────────────────
+    # Packages (toolchains, platforms, frameworks, libraries) are shared per
+    # platform family and saved as soon as `fbuild install` finishes: they do
+    # not change during the build. Build payloads (core objects, framework
+    # library archives, library selection, the zccache store) are per board
+    # and saved at the end of the job. Neither falls back across boards, so a
+    # board's entry never grows into an all-toolchain blob.
+
+    - name: Plan split fbuild caches
+      if: ${{ inputs.cache-mode == 'split' }}
+      id: split-plan
+      shell: bash
+      env:
+        FBUILD_PROJECT_DIR: ${{ inputs.project-dir }}
+        FBUILD_ENVIRONMENTS: ${{ inputs.environments }}
+        FBUILD_BOARD: ${{ inputs.board }}
+      run: |
+        set -euo pipefail
+        read -r -a envs <<< "$FBUILD_ENVIRONMENTS"
+        if [ "${#envs[@]}" -eq 0 ]; then
+          echo "::error::cache-mode: split needs the 'environments' input" >&2
+          exit 1
+        fi
+        env_args=()
+        for env in "${envs[@]}"; do env_args+=(-e "$env"); done
+        # A dry run resolves the package set without touching the network;
+        # it names each env's platform before anything is restored.
+        plan="${RUNNER_TEMP}/fbuild-install-plan.json"
+        fbuild install "$FBUILD_PROJECT_DIR" "${env_args[@]}" --dry-run --json > "$plan"
+        family=$(python -c 'import json,sys; print("-".join(sorted({e["platform"].lower() for e in json.load(open(sys.argv[1]))["environments"]})))' "$plan")
+        board="${FBUILD_BOARD:-$(IFS=_; echo "${envs[*]}")}"
+        echo "family=${family}" >> "$GITHUB_OUTPUT"
+        echo "board=${board}" >> "$GITHUB_OUTPUT"
+        echo "Platform family: ${family}; board payload key: ${board}"
+
+    - name: Restore fbuild packages cache
+      if: ${{ inputs.cache == 'true' && inputs.cache-mode == 'split' }}
+      id: packages-restore
+      uses: actions/cache/restore@v5
+      with:
+        path: |
+          ${{ steps.resolve-paths.outputs.cache-dir }}/toolchains
+          ${{ steps.resolve-paths.outputs.cache-dir }}/platforms
+          ${{ steps.resolve-paths.outputs.cache-dir }}/packages
+          ${{ steps.resolve-paths.outputs.cache-dir }}/libraries
+          ${{ steps.resolve-paths.outputs.cache-dir }}/archives
+          ${{ steps.resolve-paths.outputs.cache-dir }}/installed
+          ${{ steps.resolve-paths.outputs.cache-dir }}/index.sqlite
+        # Saved keys end in the packages_hash, which is unknown until install
+        # runs; restore the newest entry for this platform family by prefix.
+        key: fbuild-pkgs-${{ inputs.cache-version }}-${{ runner.os }}-${{ runner.arch }}-${{ steps.split-plan.outputs.family }}-restore
+        restore-keys: |
+          fbuild-pkgs-${{ inputs.cache-version }}-${{ runner.os }}-${{ runner.arch }}-${{ steps.split-plan.outputs.family }}-
+
+    - name: Install fbuild packages
+      if: ${{ inputs.cache-mode == 'split' }}
+      id: split-install
+      shell: bash
+      env:
+        FBUILD_PROJECT_DIR: ${{ inputs.project-dir }}
+        FBUILD_ENVIRONMENTS: ${{ inputs.environments }}
+        FBUILD_PACKAGES_KEY_PREFIX: fbuild-pkgs-${{ inputs.cache-version }}-${{ runner.os }}-${{ runner.arch }}-${{ steps.split-plan.outputs.family }}-
+      run: |
+        set -euo pipefail
+        read -r -a envs <<< "$FBUILD_ENVIRONMENTS"
+        env_args=()
+        for env in "${envs[@]}"; do env_args+=(-e "$env"); done
+        manifest="${RUNNER_TEMP}/fbuild-install.json"
+        fbuild install "$FBUILD_PROJECT_DIR" "${env_args[@]}" --json > "$manifest"
+        cat "$manifest"
+        hash=$(python -c 'import json,sys; print(json.load(open(sys.argv[1]))["packages_hash"])' "$manifest")
+        echo "packages-hash=${hash}" >> "$GITHUB_OUTPUT"
+        echo "packages-key=${FBUILD_PACKAGES_KEY_PREFIX}${hash}" >> "$GITHUB_OUTPUT"
+
+    - name: Save fbuild packages cache
+      if: ${{ inputs.cache == 'true' && inputs.cache-mode == 'split' && inputs.save == 'true' && steps.packages-restore.outputs.cache-matched-key != steps.split-install.outputs.packages-key }}
+      uses: actions/cache/save@v5
+      with:
+        path: |
+          ${{ steps.resolve-paths.outputs.cache-dir }}/toolchains
+          ${{ steps.resolve-paths.outputs.cache-dir }}/platforms
+          ${{ steps.resolve-paths.outputs.cache-dir }}/packages
+          ${{ steps.resolve-paths.outputs.cache-dir }}/libraries
+          ${{ steps.resolve-paths.outputs.cache-dir }}/archives
+          ${{ steps.resolve-paths.outputs.cache-dir }}/installed
+          ${{ steps.resolve-paths.outputs.cache-dir }}/index.sqlite
+        key: ${{ steps.split-install.outputs.packages-key }}
+
+    - name: Restore fbuild build-payload cache
+      if: ${{ inputs.cache == 'true' && inputs.cache-mode == 'split' && inputs.save == 'true' }}
+      id: build-cache
+      uses: actions/cache@v5
+      with:
+        path: |
+          ${{ steps.resolve-paths.outputs.cache-dir }}/core
+          ${{ steps.resolve-paths.outputs.cache-dir }}/framework-libs
+          ${{ steps.resolve-paths.outputs.cache-dir }}/library-selection
+          ${{ steps.resolve-paths.outputs.zccache-store-path }}
+        key: fbuild-build-${{ inputs.cache-version }}-${{ runner.os }}-${{ runner.arch }}-${{ steps.fbuild-hash.outputs.fbuild-hash }}-${{ steps.split-plan.outputs.board }}-${{ inputs.cache-key-extra }}
+        restore-keys: |
+          fbuild-build-${{ inputs.cache-version }}-${{ runner.os }}-${{ runner.arch }}-${{ steps.fbuild-hash.outputs.fbuild-hash }}-${{ steps.split-plan.outputs.board }}-
+
+    - name: Restore fbuild build-payload cache (no save)
+      if: ${{ inputs.cache == 'true' && inputs.cache-mode == 'split' && inputs.save != 'true' }}
+      id: build-cache-restore-only
+      uses: actions/cache/restore@v5
+      with:
+        path: |
+          ${{ steps.resolve-paths.outputs.cache-dir }}/core
+          ${{ steps.resolve-paths.outputs.cache-dir }}/framework-libs
+          ${{ steps.resolve-paths.outputs.cache-dir }}/library-selection
+          ${{ steps.resolve-paths.outputs.zccache-store-path }}
+        key: fbuild-build-${{ inputs.cache-version }}-${{ runner.os }}-${{ runner.arch }}-${{ steps.fbuild-hash.outputs.fbuild-hash }}-${{ steps.split-plan.outputs.board }}-${{ inputs.cache-key-extra }}
+        restore-keys: |
+          fbuild-build-${{ inputs.cache-version }}-${{ runner.os }}-${{ runner.arch }}-${{ steps.fbuild-hash.outputs.fbuild-hash }}-${{ steps.split-plan.outputs.board }}-

README.md

@@ -183,6 +183,7 @@ These commands extend beyond the PlatformIO workflow surface:
 | `fbuild reset` | Reset a device without flashing it. |
 | `fbuild purge` | Purge downloaded packages or run cache garbage collection. |
 | `fbuild sync` | Resolve `platformio.ini` dependencies into a deterministic lock file. |
+| `fbuild install` | Download an environment's platform, toolchains, framework, tools and `lib_deps` without compiling (`--check` exits 2 when something is missing). |
 | `fbuild daemon` | Manage the background build daemon, locks, and cache. |
 | `fbuild show` | Show daemon logs and other runtime information. |
 | `fbuild device` | List devices and manage device leases. |

agents/docs/commands-reference.md

@@ -51,6 +51,7 @@ help text).
 | `fbuild show` | Show daemon logs or other introspection. | `fbuild help show` |
 | `fbuild device` | List / inspect connected devices the daemon knows about. | `fbuild help device` |
 | `fbuild purge` | Purge cached packages — full purge or LRU-only via `--gc`. | `fbuild help purge` |
+| `fbuild install [project_dir] [-e <env>]... [--all-envs] [--check] [--dry-run] [--json]` | You want an env's platform, toolchains, framework, tools and `lib_deps` downloaded without compiling — e.g. a separate, observable CI step before the build. One line per package (`present` / `fetched` / `would-fetch` / `failed`); `--json` adds a `packages_hash` to key a packages cache on. `--check` and `--dry-run` never touch the network; `--check` exits 2 when anything is missing. Runs in-process, no daemon. | `fbuild help install`, FastLED/fbuild#1433, `docs/reference/cli.md#fbuild-install` |
 | `fbuild lnk` | Manage `.fetch` blob pointers (fetch / verify / add). `.lnk` is still read for pointers written before FastLED/fbuild#1369; FastLED's runtime `.lnk` asset links are a different format and are skipped. | `fbuild help lnk` |
 
 ## Serial-port introspection (FastLED/fbuild#686)

crates/CLAUDE.md

@@ -77,7 +77,7 @@ fbuild-test-support (test utilities) ──────────────
 
 **HTTP API boundary:** CLI sends JSON requests to daemon over HTTP. Build output streams via WebSocket. Serial monitor data streams via `/ws/serial-monitor`. All endpoints match the Python FastAPI daemon's contract.
 
-**Diagnostic subcommand exception:** A small, growing set of `fbuild-cli` subcommands (`clang-tidy`, `clang-query`, `iwyu`, `mcp`, `lnk`, `lib-select`) run in-process and intentionally bypass the daemon. They are read-only diagnostics that don't need build orchestration, so a round-trip through the HTTP API would only add latency. The "thin HTTP client" rule still applies to every command that touches the build pipeline (`build`, `deploy`, `monitor`, `test-emu`, etc.).
+**Diagnostic subcommand exception:** A small, growing set of `fbuild-cli` subcommands (`clang-tidy`, `clang-query`, `iwyu`, `mcp`, `lnk`, `lib-select`) run in-process and intentionally bypass the daemon. They are read-only diagnostics that don't need build orchestration, so a round-trip through the HTTP API would only add latency. `install` also runs in-process: it only resolves and downloads packages into the cache (no compile, no build lock), and CI runs it as its own step before any build (FastLED/fbuild#1433). The "thin HTTP client" rule still applies to every command that touches the build pipeline (`build`, `deploy`, `monitor`, `test-emu`, etc.).
 
 **PyO3 consumer contract:** FastLED imports `SerialMonitor` as a Python context manager with `read_lines()`, `write()`, `write_json_rpc()`. The `fbuild-python` crate must preserve this API exactly.