Skip to content

node up --background: 10s readiness deadline is shorter than real cold-start times #1427

Description

@khaliqgant

Problem

node up --background gives a cold broker a fixed 10-second readiness window. Real cold-start times measured on two machines exceed it:

Machine Cold start Result
sf-mini (fleet node) 13.420s failedBroker background start did not become ready within 10s (pid: 72139)
Laptop (M-series) 14s succeeded — same run, different side of the race

Same command, same version, opposite outcomes. This is a race against a hard-coded deadline, not a deterministic failure, which is why it presents as flaky.

Observed failure mode

$ agent-relay node up --no-spawn --background
Broker background start did not become ready within 10s (pid: 72139)
Cleaning up failed broker start (pid: …)
Failed to stop half-started broker process (pid: …). Run `agent-relay down --force`
to retry cleanup, or remove `.agentworkforce/relay/` after stopping the process.

The broker itself is fine — a foreground start of the same build on the same machine came up in 13.420s and joined the intended workspace correctly. Only the detached readiness probe gives up early.

Why it matters

The failure is actively misleading. When it fires, it looks like a startup or configuration fault, and the remediation text points at removing .agentworkforce/relay/ — deleting broker state to fix what is only a timeout. During the 2026-08-03 workspace incident this failure was misread as --workspace-key being broken, and several recovery attempts were spent on the wrong cause.

It also leaves a half-started process behind that --force does not always reap, compounding an unrelated orphaned-broker problem.

Suggested fix

Any of these, in rough order of preference:

  1. Wait on a readiness signal rather than a fixed deadline — poll until the broker reports ready or its child process exits, with a generous ceiling.
  2. Make the deadline configurable — a flag plus an env var, so slower machines and CI can raise it.
  3. Raise the default to something above observed cold-start times (30s+) — cheap, but still arbitrary.
  4. Fix the message regardless. On timeout, say the broker may still be starting and suggest agent-relay node status, rather than implying broken state and recommending deletion of .agentworkforce/relay/.

Evidence

Full transcripts in #1425:

Scope

Independent of #1425's workspace-binding work and not fixed by it. #1425 changes detached startup to stop polling when its child exits and to avoid killing a dead PID; neither addresses the deadline being shorter than a normal cold start.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions