Repository navigation
CI: board build jobs flake with 'daemon did not become healthy after 3 spawn attempts' (2 release dry runs in a row) #1627
Description
Release dry runs for 2.5.32 hit the same daemon-startup flake twice. Each time it was one board-build job; every other board job in the same run passed using the same downloaded fbuild-bin-linux-debug artifact:
- run 37070075494:
Full validation on release commit / Adafruit QT Py M0 / build - run 37072233986:
Full validation on release commit / NXP LPCXpresso804 / build(job 111054725083, step "Build NXP LPCXpresso804 (quick)"). The other LPCXpresso804 job in the same run passed.
ERROR fbuild::output: daemon error: daemon did not become healthy after 3 spawn attempts (10s each). This is a daemon problem, not a defect in the code being built — no compilation was attempted.
running-process broker: direct daemon fallback (failed to connect to broker: No such file or directory (os error 2))
Each flake fails the release gate and needs a manual rerun. Locally the same message appears when another fbuild daemon owns the cache root (root_ownership phase: "another fbuild-daemon owns the cache root (root-owner.lock busy)"), so a CI job may be racing a daemon from an earlier step or a parallel build on the same runner. 3 attempts of 10 s each may also be too short on a loaded runner.
Asks:
- Log the startup phase and the reason for each failed spawn attempt in the CLI error (e.g. lock busy vs. bind failure vs. timeout).
- Make the startup budget robust on CI.
- If the cause is a leftover daemon from an earlier step, attach to it or retire it rather than failing.
Activity
fix(cli): say why the daemon never became healthy (#1627)
fix(cli): say why the daemon never became healthy (#1627) (#1732)
fix(cli): give the daemon a 30s readiness budget on CI (#1627)
fix(cli): give the daemon a 30s readiness budget on CI (#1627)
fix(cli): give the daemon a 30s readiness budget on CI (#1627) (#1734)
zackees commentedon Oct 29, 2026
All three asks from this report have shipped. Closing as addressed; reopen if the flake recurs with a stack that shows none of the new diagnostics firing.
Ask -> landing
2bf95d7d) addedcrates/fbuild-cli/src/daemon_client/startup_log.rs: the CLI records a timestamp before the first spawn, then appends theWARN/ERRORlines the daemon wrote todaemon.logduring those attempts (ANSI-stripped, last 8) to the "did not become healthy" error. A lock-busy vs bind-collision vs panic cause is now visible instead of a silent 30 s.a2913fc0): wall-clock readiness budget is 30 s underCI(10 s elsewhere), overridable withFBUILD_DAEMON_READINESS_TIMEOUT_SECS, and it stretches while the daemon reportsstarting.7a29f1be): after three failed attempts the CLI callsretire_unresponsive_for_recovery, which is fail-closed on process identity (never touches a daemon that answers) and then makes exactly one final spawn attempt. A concurrent caller already adopts a concurrently-started daemon (#1010single-flight), so the "attach" case is covered.Current state
main2ba57549: lazy -> every board job green —Build ESP32 Dev,Build Arduino Uno,Build Teensy 4.1(runs 40874023238 / 40874026753/ 40874030262), plus the full board matrix in #1782'sfullrun. Nodaemon did not become healthyfailure since #1776 merged.zackees commentedon Oct 29, 2026
Correction to a typo in my closing comment: that line should read
main2ba57549— every board job green (Build ESP32 Dev,Build Arduino Uno,Build Teensy 4.1; runs 40874023238, 40874026753, 40874030262).