perf(esp32): compile all stages (lib_deps, framework libs, core, sketch, local libs) in one shared job pool like PlatformIO

[zackees]

Problem

PlatformIO (SCons) builds every translation unit (libraries, framework libraries, core, sketch) as nodes of one dependency graph at -j N, so any ready compile runs. fbuild's ESP32 orchestrator (crates/fbuild-build-esp/src/esp32/orchestrator/build.rs) runs them as serial stages. Each stage ends with a partly idle tail, and the next stage cannot start until it finishes.

#	Stage (build.rs line on main)	Serialization
1	Toolchain check → framework install check → log_toolchain_version (229, 240, 249)	sequential awaits; spawns gcc --version every build
2	lib_deps compile, ensure_libraries (388)	must finish before anything else compiles
3	…inside it, library_manager.rs:130 for lib in &installed	one library at a time; files parallel within a library only
4	LDF / framework library selection (437)	waits for stage 2, but needs only lib_deps sources, not objects
5	Project-as-library (526)	separate serial step
6	Framework libraries (WiFi, FS, SPI…; 544; framework_libs.rs:136 loop)	one library at a time; must finish before core starts
7	Core + sketch (750–776)	the only stage already on a shared pool (#1545), with boot artifacts overlapped (#1548)
8	Local lib/ libraries (826)	runs after core/sketch, not alongside
9	embed_files (851), compile DB (867)	small, serial before link
10	Link → convert/size	matches PlatformIO (the link must wait)

bench/blink has no libraries, so stages 2–6 and 8 are empty and the benchmark does not show this. A FastLED project pays for all of it on every cold build: ~30 FastLED unity TUs one library at a time, then 5 framework libraries one at a time, then core + sketch.

Proposal

Extend the #1545 pattern (compile_phases::compile_core_and_sketch with one Arc<Semaphore>) to the whole build:

1. Resolve first, compile later. Collect every library's sources and include dirs (lib_deps, project-as-lib, LDF-selected framework libs, local libs), then run library selection on sources. It does not need lib_deps objects.
2. One job pool. Submit every TU from every stage to a single FIFO semaphore, capped globally by the zccache per-core gate (perf: let the daemon's zccache run one compiler per core #1546), and ordered by dispatch_rank (C++ first, perf(build): dispatch C++ translation units before C and assembly #1553/fix(build): take compile permits in dispatch order #1558). Within that, put the longest TUs first, e.g. the FastLED unity files.
3. Archive per library as soon as its objects are done. Each archive depends only on its own objects. The link waits on all archives, the core objects and the sketch.
4. Remove the per-library for loops in library_manager.rs and framework_libs.rs, or turn them into producers that feed the pool.
5. Small items: run the toolchain and framework ensure_installed concurrently, and cache the gcc --version probe per toolchain path.

Correctness constraints to keep:

• Per-library flags and include dirs, and library.json build flags.
• Library-selection shadowing rules.
• Archive order on the link line.
• .cmdhash rebuild signatures.
• Fail-fast on the first compile error, with the join/drain soundness in parallel.rs.

Expected impact

For FastLED sketches on ESP32, the cold build should approach one parallel phase bounded by (total TU CPU ÷ cores). Today it is the sum of each stage's makespan plus its idle tail. For blink, nothing changes.

Acceptance

• A FastLED example cold build (e.g. DemoReel100, esp32s3) with FBUILD_PERF_LOG_JSON shows a single compile phase, and a process sampler shows ≥ N-1 compilers busy throughout.
• Before/after wall-time numbers are recorded (median of 3, 4 cores), alongside PlatformIO's phases from the benchmark's pio run -v breakdown (bench: break PlatformIO cold builds into phases for A/B with fbuild #1557).
• The same firmware as before: identical selected libraries and link inputs.

Follow-up to #1537.

[zackees]

Scope update: centralize the scheduler in the engine so every platform gets it

This is not an ESP32-only change. The unified job pool belongs in fbuild-build-engine, in the shared pipeline, and every orchestrator should use it. The ESP32 orchestrator moves onto it too rather than keeping a private scheduler.

Where the serialization lives today

Serialization	Where	Platforms
lib_deps compile before anything else, one library at a time (library_manager.rs:130)	fbuild-library / pipeline/library.rs	all
Project-as-library as a separate serial step after the core/sketch join	pipeline/sequential.rs / compile_project_as_library	all
Framework libraries one at a time, before the core (framework_libs.rs:136)	ESP32 orchestrator	ESP32
Local lib/ libraries after core/sketch (build.rs:826)	ESP32 orchestrator	ESP32 (the shared pipeline already overlaps these)
Library selection waits for the lib_deps objects	ESP32 orchestrator	ESP32
Serial toolchain/framework ensure_installed plus a gcc --version probe on every build	ESP32 orchestrator	ESP32

Plan

1. Engine: pipeline::schedule (new module in fbuild-build-engine). It takes a set of compile units (source, output, flags/overlay, owning target) grouped into targets: core, variant, sketch, each lib_deps library, each framework library, local libraries, and project-as-library.
   • Every unit from every target goes into one FIFO Semaphore, ordered by dispatch_rank and taking permits in dispatch order (perf(build): dispatch C++ translation units before C and assembly #1553/fix(build): take compile permits in dispatch order #1558). zccache's per-core gate (perf: let the daemon's zccache run one compiler per core #1546) stays the global cap.
   • Each target archives as soon as its own units finish. Its output is an ordered object or archive list, and link-line order is preserved.
   • It fails fast on the first error and drains every spawned task, keeping the 'static-borrow soundness rule from parallel.rs.
2. Split resolution from compilation. ensure_libraries resolves and installs libraries and returns their sources, include dirs and flags without compiling them. Library selection runs on sources. The scheduler does all the compiling.
3. pipeline/sequential.rs builds the target set and calls the scheduler. This covers all ~14 shared orchestrators (AVR, ARM families, RP2040, ESP8266, CH32V…) with no per-platform code.
4. ESP32 builds the same target set, framework libraries included, and calls the same scheduler. compile_phases.rs and the per-library loops in framework_libs.rs / library_manager.rs go away.
5. Small items: concurrent ensure_installed, and a cached gcc --version probe keyed by toolchain path.

Testing

Unit tests cover the scheduler completely, in the engine, with a fake Compiler like parallel.rs's TracingCompiler:

• units from different targets overlap inside one permit budget, and the in-flight maximum never exceeds the budget;
• dispatch order holds across targets;
• each archive starts as soon as its own units finish, not when all units finish;
• per-target flags and overlays go to the right units;
• archive and object order on the link line is deterministic and unchanged;
• after a failure, no new units are spawned, every spawned task is drained and every permit is released;
• rebuild-signature skips (.cmdhash) and empty targets are handled;
• the library resolve-without-compile path returns the sources, include dirs and flags the old path compiled.

Light end-to-end checks only on two platforms:

• ESP32-S3: FastLED DemoReel100 cold build (the stages collapse into one compile phase) plus bench/blink.
• AVR: bench/blink -e uno plus one AVR sketch with a lib_deps library.

Each should show the same selected libraries and link inputs as before, with before/after cold times (median of 3, 4 cores).

[zackees]

Done. Every stage compiles in one shared job pool on all platforms, and the sub-issues found along the way are fixed.

PRs

PR	what
#1561	fbuild-library: library compiles draw from a caller-supplied JobGate, with permits taken in dispatch order and fail-stop. ensure_libraries is split into resolve_libraries plus a LibraryCompilePlan that compiles every library concurrently. Also fixes #1560: a stale-archive shortcut meant edited local or symlinked lib_deps never rebuilt.
#1562	All shared-pipeline platforms (AVR, ARM families, RP2040, ESP8266, CH32V…): lib_deps and project-as-library compile on the pipeline's compile_slots, joined with core, variant, sketch and local libraries.
#1563	ESP32: lib_deps, framework libraries (all at once), project-as-library, core + sketch and local libraries run in one tokio::join! on one gate, plus concurrent package install.
#1571	#1564: longest compile units first, using recorded per-unit durations.
#1567	#1566: header farm regression on pioarduino 3.3.11. #include_next headers found beside a quoted includer broke assert.h.
#1570	#1568: library compile temp files were written into the project root. #1569: a failed compile with empty stderr now reports its exit status.

Before / after

Before is main at 4b695966, before this work. After is main at c601d821. Each build was cold (fbuild clean cache plus --clean), pinned to 4 cores, alternating before/after, median of 3. The firmware sha256 is identical before and after for every project.

project	before	after	change
FastLED DemoReel100, ESP32-S3	35.17 s	28.46 s	−19.1%
FastLED Blink, AVR uno	9.38 s	9.05 s	−3.4%
bench/blink ESP32-S3 (no libraries; 6 trials)	3.38 s	3.34 s	no change (noise)
bench/blink uno	0.49 s	0.48 s	no change

The first "after" FastLED ESP32-S3 trial ran with no duration history and took 28.7 s, so almost all of the gain comes from the shared pool itself. The remaining floor is FastLED's single ~17 s fl.fx+.cpp unit plus the link. Library-free sketches (blink) were expected to be unaffected, and they are.