Skip to content

fleet spawn:<harness> action reports {spawned:true} without confirming the worker actually survived #1430

Description

@khaliqgant

Bug

Real E2E evidence (E2B fleet-enrollment test, 2026-08-05): a fleet-dispatched `spawn:codex` invocation (`inv_210704094451425280`) was reported by Relaycast as completed with `{spawned:true}`, dispatched to a confirmed-online node (`node_210704025104433152`, status=online/live=true/handlers_live=true). But no agent ever registered for that spawn, and the expected task marker was never produced. `handlerAgentId` on the invocation stayed `null`.

Root cause

`crates/broker/src/runtime/fleet.rs`, `handle_fleet_action_spawn` (around line 456-461):

```rust
// spawn_worker_from_request does not return a result; treat presence of
// the worker as success so the engine's invocation resolves.
if self.workers.workers.contains_key(&name) {
self.reply_action_output(
&invoke.invocation_id,
json!({ "spawned": true, "name": name.as_str() }),
)
.await;
} else {
self.reply_action_error(&invoke.invocation_id, "spawn_failed")
.await;
}
```

This checks only that a registry entry exists — not that the underlying worker process actually survived startup or genuinely registered. This is the same class of bug fixed in #1429 (`WorkerRegistry::spawn_worker` in `worker.rs`, which added a stability window + `try_wait()` confirmation before reporting success for the direct `node agent spawn` CLI path) — but that fix does not cover this separate fleet-action dispatch path (`spawn:` actions invoked via `POST /v1/nodes/:node/actions/:name/invoke`), which apparently reports success based on registry presence alone, without the same confirmation.

Suggested fix

Reuse #1429's confirmation logic (or the underlying `WorkerRegistry::spawn_worker` path, if `spawn_worker_from_request` doesn't already funnel through it) here too, so `handle_fleet_action_spawn` only replies `{spawned:true}` after the worker process is confirmed alive, and replies a real `spawn_failed` error (with exit status / log path, matching #1429's error shape) when it isn't — instead of trusting bare registry-key presence.

Impact

Fleet-orchestrated spawns (used by cloud's `provisionFleetSandboxNode()` / `createFleetE2BRuntime()` / `createFleetDaytonaRuntime()` targeted-spawn proofs, and any external caller of the `spawn:` fleet action) can silently report success for a spawn that never actually produced a working agent.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions