Skip to content

broker: no per-worker-instance identity is exposed, so nothing can prove it reattached to the same process #1454

Description

@khaliqgant

The data already exists. No route returns it.

The broker knows, per worker, exactly when each process was spawned and how many times a given name has been spawned or restarted. It keeps all of it in memory, persists some of it to state.json, and surfaces none of it over HTTP. A client that needs to answer "is this the same worker process I was talking to a moment ago?" is left with the harness pid — a value the OS is free to reuse.

Three separate places already hold the answer:

Where Field Status
crates/broker/src/worker.rs:134 WorkerHandle.spawned_at: Instant in memory, never serialized. WorkerRegistry::list (worker.rs:226-256) omits it.
crates/broker/src/broker.rs:41 PersistedAgent.started_at: Option<u64> absolute unix seconds, already written to state.json on every spawn (runtime/api.rs:604-611) and every restart (maintenance.rs:414). No listen_api route reads BrokerState.agents.
crates/broker/src/metrics.rs:45-47 AgentRecord.spawns / .restarts per-name monotonic counters, incremented at metrics.rs:100 and :114. Already a working generation number. MetricsCollector::to_json (metrics.rs:233) has no caller; /api/metrics uses build_agent_metrics instead, which drops them.

Each is a one-line surfacing job.

Why it matters: reattachment cannot prove sameness

This is an identity problem, not a metrics one. Anything that reconnects to a worker resolves it by name, and a name is not an identity. If a worker died and something else claimed the name, a reconnect that "succeeds" is routing traffic — in the interactive case, a human's keystrokes — into a different process than the one the caller attached to. Restarting the same agent is equally wrong for input safety: keystrokes typed against the old session's context land in a fresh shell.

Concretely, from #1419 / #1453: the drive-attach client now reopens a dead PTY input stream, and before forwarding any input it has to establish that the replacement reached the same worker. What is available to check with:

  • pid / workerPid (worker.rs:242-243, GET /api/spawned) — changes on restart, which is the useful direction, but it is Option<u32> and is None until worker_ready fires (worker_events.rs:849), and OS pids are reusable. A heuristic, not a nonce.
  • uptime_secs (GET /api/metrics, system.rs:92) — resets to 0 on restart, so a decrease proves a new process. But it is relative with 1-second granularity, so proving sameness requires the client to track now - uptime and tolerate drift.
  • sessionId — explicitly useless here. A restart reuses the resolved spec (maintenance.rs:381), so it survives the very event we need to detect.
  • instance_id / run_id / epoch / generation / boot_idabsent on workers entirely. The only instance_id in the crate is the broker's own node-control connection epoch (node_control.rs:1529), which is broker↔engine identity, not per-worker.

Neither the PTY snapshot response (pty_worker.rs:994-1048: {format, rows, cols, cursor, screen, offset}) nor worker_stream frames (worker_events.rs:717-733) carry an instance discriminator either. stream_offset resets to 0 on restart, so a decrease is suggestive, but "same process with no output since" and "new process at offset 0" are indistinguishable.

So #1453 ships a pid comparison that fails closed — it refuses the replacement when identity is unavailable, unreadable, or changed. That is the right behaviour given what exists, but it is defending a real invariant with a value that can collide.

Proposal

Return a per-spawn identity on GET /api/spawned (and therefore in the agents array of GET /api/status). Any one of these closes it:

  1. spawned_at as an absolute timestamp — WorkerHandle.spawned_at already exists; PersistedAgent.started_at already computes the absolute form. Add it to the json! at worker.rs:231-253.
  2. A spawn generation counterMetricsCollector's per-name spawns is already correct and already maintained; it just never reaches a response.
  3. An opaque per-spawn UUID minted at worker construction. Strongest option, and the only one immune to clock changes and counter resets across broker restarts.

(1) or (3) is preferable to (2) alone, since a counter restarts with the broker.

Acceptance criteria

  • A worker's spawn identity is readable over HTTP, and changes whenever the process behind a name changes.
  • It survives a broker restart, or the fact that it does not is documented so clients do not treat a reset as a worker change.
  • A test asserts that respawning the same name yields a different identity, and that a live worker's identity is stable across repeated reads.
  • #1453's reopen gate can then verify sameness rather than compare pids.

Notes

Found while root-causing #1419 (drive attach input flood); the recovery work is #1453 and the transport keepalive gap is #1450. Filing this separately because it is a general identity gap that outlives the PTY bug that surfaced it — anything that reconnects, adopts, or re-registers by name has the same blind spot. Not labelled for dispatch.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions