perf(esp32s3): cold build 1.55x slower than PlatformIO — bloated -I list (+55% per TU), serial sketch phase, pre-compile serial overhead

[zackees]

Benchmark: https://fastled.github.io/fbuild/ (fbuild 10.46s vs PlatformIO 6.75s cold, ratio 1.551, stacks matched).

To locate the gap I instrumented both tools on the same box (16-core Linux, identical pins: espressif32@6.13.0, toolchain 8.4.0+2021r2-patch5, framework 3.20017.241212+sha.dcc1105b, fixture bench/blink). PlatformIO has no built-in per-phase timer, so stats came from -v command tracing plus timestamp wrappers installed over both toolchains' g++/gcc/ar/size/objcopy binaries (removed afterwards). fbuild stats from FBUILD_PERF_LOG_JSON. Raw logs: 3 cold trials each, median — pio 4.20s, fbuild 5.71s (1.36x), same shape as CI.

Timelines

PlatformIO (4.20s):

0.00-0.80  SCons eval + LDF scan + ino preprocess      (serial)
0.80-2.45  all 47 compiles, 16-way parallel, cpu-sum 16.2s
2.57-3.84  link                     1.27s
3.84-4.20  size + esptool image     0.36s

fbuild (5.71s):

0.00-1.67  resolve/boot/config, zero compilers running (serial)
1.67-3.51  46 core compiles         (window 1.84s, cpu-sum 21.8s)
3.52-4.10  sketch compile, strictly SERIAAL             0.59s
4.17-5.31  link                     1.14s   (parity)
5.31-5.70  convert/size/image       0.39s   (parity)

Cause 1 — bloated -I list: ~55% slower per compiler invocation (est. 1.5-2.0s on CI)

fbuild passes 385 -I dirs; PlatformIO passes 199 (220 fbuild-only, all existing dirs — SDK component fan-out from get_sdk_include_dirs in crates/fbuild-library/src/library/esp32_framework/sdk_paths.rs:75 plus variant/src/toolchain extras). Toolchain and flags are otherwise identical (fbuild even omits pio's -ggdb, which should make it faster).

2x2 A/B on one TU (FunctionalInterrupt.cpp, 3 reps, min):

command	-I count	time
fbuild flags + fbuild includes	385	0.566s
fbuild flags + pio includes	199	0.357s
pio flags + pio includes	199	0.386s
pio flags + fbuild includes	385	0.596s

Swapping only the include list moves compile time ~+55% in both directions. Whole-build: identical 47-TU cpu-sum goes 16.2s (pio) -> 21.8s (fbuild, --jobs 16). PlatformIO's 199-dir set compiles the entire Arduino core, so it is a proven-sufficient baseline.

Fix: trim the SDK include fan-out to the parsed flags/includes set (what PlatformIO uses), dedupe against toolchain/variant dirs.

Cause 2 — sketch compile is serialized (~0.6s; matches CI compile-sketch: 589ms)

The ESP orchestrator runs strictly sequential phases: compile-core-variant -> core-cache-store -> compile-sketch -> link (crates/fbuild-build-esp/src/esp32/orchestrator/build.rs:742-790). PlatformIO compiles the sketch concurrently with the core files — zero wall cost. fbuild pays a full extra TU slot.

Fix: schedule sketch compile concurrently with the core fan-out (same jobs pool).

Cause 3 — pre-compile serial overhead: 1.67s vs pio's 0.80s (~+0.9s)

• pioarduino-resolve ~150ms and boot-artifacts ~110ms every cold build
• ~0.7s in no perf phase at all: a constant gap between the framework-libs cache and framework core cache log lines (0.21 -> 0.89; also present on warm builds)
• CI-side, Σ phases = 9.73s vs 10.46s total => ~0.73s unphased

Fix: attribute the unphased gap (likely package/cache metadata IO before the PerfTimer starts) and fold it into phases.

1.75 + 0.6 + 0.9 ≈ CI's 3.7s gap.

Ruled out by measurement

• Toolchain/flags: identical versions; flag-only diff favors fbuild (no -ggdb).
• Link: fbuild 1.14s vs pio 1.27s — parity.
• Job oversubscription: default jobs = ncpu*2 (32 on 16 cores) looked suspicious, but a controlled --jobs 16 cold run changed nothing (5.60s vs 5.57s wall).

Side finding (separate issue-worthy)

Local warm rebuilds take 1.8s while CI publishes 142ms. fast-path-check returns in 0.006ms but the full pipeline still runs (resolve 151ms + 46 up-to-date checks ~724ms + esptool ~0.15s). Stable across runs with a hot daemon and unaffected by toolchain mtimes — the fast-path early-return does not engage locally.

[zackees]

Resolved. Final main benchmark: run 36482093821, published at https://fastled.github.io/fbuild/.

ESP32-S3 cold build, fbuild ÷ PlatformIO: 1.551 at the start of this issue → 0.628 now (the 4-core runs for the farm PR measured 0.579–0.641).

	issue opened	now
fbuild cold / PlatformIO cold	10.46 s / 6.75 s (1.551)	2.81 s / 4.48 s (0.628)
fbuild warm	~145 ms	47 ms
fbuild phases	compile ≈5 s, link+image ≈1.2 s, pre-compile ≈0.3 s	compile 2.12 s, link+convert+size 0.63 s, boot artifacts overlapped
PlatformIO phases (new, #1557)	n/a	pre-compile 0.47 s, compile 2.99 s, link 0.80 s, convert+size 0.21 s

The final run landed on a faster runner (PlatformIO took 4.48 s here versus about 6.8 s before), which is why the ratio is the fair comparison.

PRs

• perf(esp32): cut SDK -I fan-out from 385 dirs to PlatformIO's curated list #1543: curated -I list.
• perf: memoize successful path canonicalization (~35k realpath calls per build) #1544: canonicalize memo.
• perf(esp32): compile the sketch concurrently with the framework core #1545: sketch compiled concurrently.
• perf: let the daemon's zccache run one compiler per core #1546: zccache compile cap is now every core. The scrubbed daemon environment had pinned it to cores − 1. This is also the global anti-oversubscription governor.
• perf(esp32): verify the managed esptool once per install, not per build #1547: esptool verified once per install.
• perf(esp32): stage boot artifacts during compile; fix warm fast path without a working python #1548: boot artifacts overlapped with compile, and gen_esp32part runs on the managed Python (fixed the warm fast path, warm fast-path early-return does not engage locally (1.8 s local vs 142 ms CI on the same warm rebuild) #1542).
• perf(esp32): compile through a header farm instead of ~200 SDK -I dirs #1550: header farm, 197 SDK -I dirs → 10 (−30% compile).
• perf(ldf): resolve each include name once per walk (FastLED incremental 25 s -> 7 s) #1551 and perf(ldf): index search-path roots so include misses stop probing every path #1554: the LDF resolver stops stat-probing ~400 paths per include. FastLED incremental builds went from 25 s to 6–7 s.
• fix(esp32): pass PlatformIO's SDK defines on 2.x-layout frameworks #1552: PlatformIO's CPPDEFINES on 2.x frameworks (fix(esp32): 2.x-layout frameworks get none of PlatformIO's CPPDEFINES (ESP_PLATFORM missing; FastLED fails on espressif32@6.13.0) #1549), plus the fast-path fingerprint now covers SDK defines.
• perf(build): dispatch C++ translation units before C and assembly #1553 and fix(build): take compile permits in dispatch order #1558: C++ translation units dispatched first.
• perf(esp32): run the linker under the C locale (-30% link time) #1555: ld runs under LC_ALL=C (−30% link, byte-identical ELF).
• test(esptool): install fake executables by rename to avoid ETXTBSY #1556: test flake fix.
• bench: break PlatformIO cold builds into phases for A/B with fbuild #1557: PlatformIO phase breakdown in the benchmark.