Repository navigation
fix(daemon): avoid startup stalls when wildcard bind collides without a listener #1700
Description
Context
During FastLED/FastLED#4895 full CI using published FBuild 2.5.43, Linux daemon startup repeatedly failed before any compilation:
- ESP32-C3 run 39751775623: first two sketches (AnalogOutput and Animartrix) fail with
daemon did not become healthy after 3 spawn attempts (10s each); later sketches compile after startup finally succeeds. - Earlier Apollo3 run 39749344685: same behavior.
- Both logs repeatedly report
bind to 0.0.0.0:50830 failed (Address in use (os error 98)); no listener present (stale state), retrying in 500ms.
This is separate from the AVR8JS Node launcher shutdown fix released in 2.5.43. It is a startup/binding failure; the first sketches never reach the compiler.
Verified source at c621e34:
crates/fbuild-daemon/src/main.rs: constructs0.0.0.0:{port}, thenbind_listener_with_retryretries the same address three times. Its diagnostic assumes no listener means stale state.crates/fbuild-paths/src/lib.rs: Linux default daemon port is in 49152..65535, overlapping ephemeral client ports.crates/fbuild-core/src/platform/linux/ipc.rs: listener already sets SO_REUSEADDR and SO_LINGER=0; those options do not solve a live outgoing connection reserving the port.
Local reproduction of the collision mechanism
A pure-loopback Python socket probe on Linux reproduces the exact combination: an outgoing client bound to 127.0.0.2:P, connected to a local test server, gives ECONNREFUSED when probing 127.0.0.1:P, while binding a SO_REUSEADDR listener to 0.0.0.0:P gives EADDRINUSE. Binding the same listener to 127.0.0.1:P succeeds.
Thus "no listener present" does not prove stale daemon/TIME_WAIT state. The CI logs do not include socket-owner snapshots, so an outgoing socket collision is a supported hypothesis, not a proven identification of the CI socket owner.
Proposal
Make local daemon startup robust to wildcard/ephemeral port collisions. Evaluate loopback-only binding (the CLI uses loopback), an appropriate non-ephemeral endpoint policy, or a discoverable fallback port. Confirm any intended remote UI clients before choosing the binding scope. Improve the error so it distinguishes root ownership, a live foreign listener, and a bind collision without a listener.
Acceptance criteria
- RED: add a focused Linux integration repro holding a client socket on the selected port, with no listener at the daemon's loopback endpoint, and demonstrate current startup failure.
- GREEN: the selected fix starts a reachable daemon promptly without waiting for the unrelated connection to disappear or killing its owner.
- Preserve same-version single-flight startup and version/identity endpoint isolation.
- Exercise real foreign listeners and stale TIME_WAIT cases; do not mistake an arbitrary TCP listener for a matching FBuild daemon.
- Run the affected FastLED board job against the fix, showing the first sketches compile instead of failing startup.
Decisions
- Track separately from AVR8JS shutdown: distinct lifecycle stage and independently reproducible failure mechanism.
- Prioritize as reliability work: it causes false compilation failures in a full board sweep.
- Do not kill other processes or delete lock files as recovery; neither identifies the socket owner or fixes endpoint selection.
Activity
zackees commentedon Oct 22, 2026
Additional hosted reproduction: FastLED/FastLED#4877
The same startup failure reproduced with published FBuild 2.5.43 on the ATmega8A board job, independently of the ESP32/Apollo3 targets in the original report:
1a8dce8783fb122ebfc3c77990b3659d5e316b02..venv/bin/fbuildand.venv/bin/fbuild-daemon; the CLI reported version 2.5.43 and endpointhttp://127.0.0.1:50830.The uploaded build summary records:
Each spawn retried the same address three times: 16:42:04.308/04.809/05.311, 16:42:14.833/15.334/15.836, and 16:42:26.851/27.351/27.853 UTC. Additional retries followed; startup finally completed at 16:42:47.469, roughly 43 seconds after the first logged spawn. The failed sketch was not automatically rebuilt. The summary reports no kernel OOM-killer records.
Source confirmation and decisions
Rechecked released source at
c621e34b0db292d44dc9a400d3b678399d8b37ff:crates/fbuild-daemon/src/main.rs:153constructs the wildcard address, andbind_listener_with_retry(line 746) retries that unchanged address while classifying a failed listener probe as stale state. This remains a strong duplicate of this issue, so no additional issue was created.The logs establish the collision/retry failure, not the socket owner's identity; ephemeral outgoing sockets remain a hypothesis. Fix startup/endpoint recovery in FBuild rather than hiding it with sketch changes or killing an unrelated process. The downstream job is being retried for confirmation; a retry result is not yet claimed.
Acceptance evidence to add
Preserve this as a downstream regression example alongside the existing focused Linux socket RED → GREEN criteria. The fix should choose a reachable compatible endpoint/bind strategy or discoverable alternate port promptly, preserve version/identity isolation, and show the first affected sketch compiles instead of exhausting three ten-second startup attempts. Merely increasing retry duration would not address the observed failure mode.
zackees commentedon Oct 22, 2026
Another hosted occurrence now affects Teensy 3.5, so this is also blocking the consolidated FastLED integration PR, not just the earlier AVR/ESP board runs:
129c0f40b756307037e6aa564fa795272addc1b5..venv/bin.daemon did not become healthy after 3 spawn attempts (10s each)and explicitlyno compilation was attempted.bind to 0.0.0.0:50830 failed (Address in use (os error 98)); no listener present (stale state), retrying in 500ms; the CLI pollshttp://127.0.0.1:50830.This adds a distinct ARM/Teensy downstream instance of the same startup defect. The affected job has been retried; no retry outcome or socket-owner identity is claimed. Existing #1700 acceptance criteria apply unchanged: bounded endpoint/startup recovery must allow the first sketch to reach the compiler without waiting for an unrelated port collision to disappear.
fix(daemon): stop calling a listener-less bind collision 'stale state…
fix(daemon): keep the daemon port out of OS ephemeral ranges and bind…
ci: refresh platform-boundary inventory for the #1700 socket2 test
fix(daemon): keep the daemon port out of OS ephemeral ranges and bind…
ci: refresh platform-boundary inventory for the #1700 socket2 test
fix(daemon): keep the daemon port out of OS ephemeral ranges and bind…