Skip to content

no-Monitor spawned agents never see queued messages, and spawn can't tell a working agent from a dead one #338

Description

@Salmonellasarduri

Summary — a field incident

A dispatcher sent a task to codex-dev, then spawned it. The spawn "succeeded", the codex came up, claimed its identity via the actas flow — and went idle forever. The task sat unread for 45 minutes while the dispatcher's poll loop watched for a reply that could never come. (The other half of the same incident — the boot prompt arriving mangled — is #336 / #337.)

Post-mortem found three compounding gaps. Versions: incident on 1.1.0; all three still present in 1.1.6 HEAD.

Gap 1 — the actas flow for no-Monitor types ends at the identity claim

scripts/drivers/types/codex/template.md step 5 stops at "Now acting as <name>". There is no "now check your inbox" step, and with no Monitor nothing will ever push a message. --boot-prompt covers the spawn-time handoff, but not messages queued before spawn (our incident: task sent 2 minutes before) or sent after the agent goes idle.

Gap 2 — spawn reports success on a bare window launch

For monitor=no types the readiness wait is skipped and spawn prints success once the terminal opens. An agent that came up broken (e.g. #282's Notepad case, or #336's mangled prompt) is indistinguishable from a working one. grok-build/type.conf documents why the unconditional skip exists — a blanket wait would hang turn/off-mode spawns — so any fix here must be opt-in per type, not a default flip.

Gap 3 — nothing surfaces "sent but never read"

Read state (read_at IS NULL + created_at) is in the DB the whole time, but no command exposes unread-age. Dispatchers can only poll for replies — the one signal a stalled worker will never produce.

Proposal (three independently mergeable pieces, field-tested on our install)

a. templates: inbox drain in the actas flow (no-Monitor types). Add a step after the identity claim: drain inbox.sh <team> <name> now — a spawner may have queued work before you attached — and re-check whenever a task finishes. Additive, no script changes.

b. spawn.sh: opt-in readiness handshake via type.conf. New key (e.g. handshake=actas) for types whose actas flow marks a one-shot ready sentinel (tiny new ready.sh mark|check|clear; spawn consumes the sentinel on observe so it can't masquerade as a live-watcher signal). Types without the key keep today's skip — grok-build semantics untouched. Timeout floor ~300s for these types: we measured codex first-turn at 143–187s, above the 90s default.

c. new scripts/stale.sh — unread-age watchdog. stale.sh <team> [--to <agent>] [--min N]: lists messages still unread after N minutes, exit 2 when any exist. Turns "is the worker slow or dead?" into a one-liner for dispatch loops.

Evidence that the combination works

With a+b+c applied to a live install (Windows 11 / Git Bash): task queued before spawn → spawn codex probe blocks → agent boots, claims identity, marks ready, drains inbox, executes the task and replies → spawn returns status=ready after=187s; stale.sh shows zero unread. Before the change the identical sequence stalled silently.

Happy to split into PRs (a/b/c separately or combined) once there's direction feedback — and to file an ADR for (b) if you'd like, since it adds a type.conf key.


(日本語要約)Monitor を持たない型(codex 等)は spawn 後に「actas で名乗って終わり」でインボックスを見る手順がなく、spawn 側もウィンドウが開いただけで成功報告するため、先に積まれたタスクが誰にも読まれず滞留する事故が実際に起きました(#336/#337 は同じ事故のもう半分です)。提案は 3 点: (a) actas 手順にインボックス排出を追加 / (b) type.conf キーによる opt-in の readiness handshake(grok-build の設計判断は不変) / (c) 未読エージング検知 stale.sh。実機で 3 点併用の E2E を実証済み。方向性の合意後に PR を分割提出します。

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions