Skip to content

fix(codex-bridge): make the turn watchdog an idle timeout, not a fixed ceiling#443

Open
Masashi-Ono0611 wants to merge 2 commits into
fujibee:mainfrom
Masashi-Ono0611:fix/codex-bridge-turn-idle-watchdog
Open

fix(codex-bridge): make the turn watchdog an idle timeout, not a fixed ceiling#443
Masashi-Ono0611 wants to merge 2 commits into
fujibee:mainfrom
Masashi-Ono0611:fix/codex-bridge-turn-idle-watchdog

Conversation

@Masashi-Ono0611

Copy link
Copy Markdown

Summary

startTurnWatchdog() arms a single fixed-duration timer from turn start
(--turn-timeout, default 60s) and never touches it again until the turn
ends. A turn that is genuinely still working — reasoning, running tools,
composing a multi-step reply — for longer than that gets cut off by this
timer before it ever reaches its own send.sh call. onTurnEnded() then
runs exactly as it does on a real completion, so the bridge silently re-arms
as if the turn had ended with nothing to report; whatever it was about to
send never goes out, and the requesting peer sees only silence.

This turns the watchdog into a true idle timeout: any thread-scoped
app-server activity for the active turn re-arms it, so a turn is only ever
cut off after turnTimeout seconds of true silence, regardless of how long
it has been running in total. This preserves the watchdog's actual purpose
(rescue a turn that will never send turn/completed — the app-server does
not reliably send it, see #41) — a turn that is visibly still working is not
that case.

Implementation

  • AppServerClient/WebSocketAppServerClient (both transports) gain an
    onThreadActivity callback, invoked from handleLine() for every
    thread-scoped notification/request — including the many the bridge has no
    specific handler for (reasoning deltas, tool-call/command-output progress,
    etc.), which the existing handlers Map otherwise drops silently before
    dispatch. This is deliberately generic rather than enumerating specific
    event names: the exact set of "the turn is doing something" notifications
    isn't fully known/stable to this bridge, and a generic hook can't miss one.
  • CodexBridge wires onThreadActivity to re-arm startTurnWatchdog() for
    its own active thread/turn.
  • onAgentMessageDelta's own re-arm (an earlier, narrower iteration of this
    fix) is now redundant and removed — the generic hook already covers it.

Observed in production

A Codex bridge turn spent well over 60s diagnosing a GitHub auth/DNS
failure (correctly identifying an invalid gh CLI token and a
network-restricted sandbox, and proposing next steps) but the watchdog fired
before it reached send.sh, so the diagnosis was silently lost and the
requesting peer received nothing.

Test plan

  • tests/test_codex_bridge.bats:

    • "an actively-streaming turn is not cut off by the idle watchdog past
      turn-timeout" (agent-message deltas keep it alive well past the nominal
      timeout, then a real turn/completed ends it — never the watchdog).
    • "non-message turn activity (reasoning/tool-call progress) also re-arms
      the idle watchdog" (same shape, but only item/reasoning/textDelta and
      item/commandExecution/outputDelta — never agentMessage/delta — to
      prove the fix isn't scoped to one specific notification type).
  • bats tests/test_codex_bridge.bats tests/test_codex_bridge_launcher.bats
    47/47 passing (30 pre-existing + 2 new here, plus the launcher suite),
    including the existing watchdog regression tests (Explore a monitor-equivalent delivery mode for Codex #41, Codex bridge: thread stuck in waitingOnApproval deadlocks delivery forever (bridge can't answer approval requests; launcher never replaces the hung bridge) #299) unmodified
    and still green.

  • Reviewed via /review:self-multi-model (Codex + Fugu) before submission:
    Codex's first pass correctly flagged that an earlier iteration of this fix
    only re-armed on agentMessage/delta and missed the multi-step
    reasoning/tool-call case above — addressed by the generic
    onThreadActivity hook in the commit history here, with a dedicated
    regression test.

  • CI green

…d ceiling

startTurnWatchdog() armed a single fixed-duration timer from turn start
(default 60s) and never touched it again until the turn ended. A turn that
was genuinely still working — actively reasoning/tool-calling for longer than
that, e.g. diagnosing a multi-step failure before replying — got cut off by
this timer before it ever reached its own send.sh call. onTurnEnded() then
ran exactly as it does on a real completion, so the bridge silently re-armed
as if the turn had ended with nothing to report; whatever the turn was about
to send never went out.

item/agentMessage/delta notifications already tell the bridge the turn is
producing output. onAgentMessageDelta now re-arms the same watchdog on every
chunk, turning it into a true idle timeout: only turnTimeout seconds of
silence trips it, regardless of how long the turn has been running in total.
This preserves the watchdog's actual purpose (rescue a turn that will never
send turn/completed — the app-server does not reliably send it, see fujibee#41) —
a turn that is visibly still working is not that case.

Observed in production: a Codex bridge turn spent >60s diagnosing a GitHub
auth/DNS failure (correctly identifying an invalid gh CLI token and a
network-restricted sandbox) but the watchdog fired before it reached
send.sh, so the diagnosis was silently lost and the requesting peer saw only
silence.
…ot just message deltas

Review finding (self-multi-model, Codex, P1): the previous commit only
re-armed the watchdog from item/agentMessage/delta, but a Codex turn spends
most of its time in other notification types the bridge has no handler for
— reasoning deltas, tool-call/command-output progress, etc. Those still went
through handleLine() but were silently dropped before dispatch (no handler
registered), so the watchdog kept counting from turn start and could still
fire mid-turn during exactly the kind of multi-step work this fix is meant
to protect.

Added a generic AppServerClient/WebSocketAppServerClient.onThreadActivity
hook that fires for every thread-scoped notification/request in handleLine(),
regardless of whether a specific handler is registered for its method.
CodexBridge wires it to re-arm the watchdog for any activity on its own
active turn. onAgentMessageDelta's specific re-arm is now redundant (the
generic hook already covers it) and has been removed.

New test proves the fix covers non-message activity: a turn that only emits
item/reasoning/textDelta and item/commandExecution/outputDelta (never
agentMessage/delta) still survives past turn-timeout without being cut off.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant