fix(codex-bridge): make the turn watchdog an idle timeout, not a fixed ceiling#443
Open
Masashi-Ono0611 wants to merge 2 commits into
Open
fix(codex-bridge): make the turn watchdog an idle timeout, not a fixed ceiling#443Masashi-Ono0611 wants to merge 2 commits into
Masashi-Ono0611 wants to merge 2 commits into
Conversation
…d ceiling startTurnWatchdog() armed a single fixed-duration timer from turn start (default 60s) and never touched it again until the turn ended. A turn that was genuinely still working — actively reasoning/tool-calling for longer than that, e.g. diagnosing a multi-step failure before replying — got cut off by this timer before it ever reached its own send.sh call. onTurnEnded() then ran exactly as it does on a real completion, so the bridge silently re-armed as if the turn had ended with nothing to report; whatever the turn was about to send never went out. item/agentMessage/delta notifications already tell the bridge the turn is producing output. onAgentMessageDelta now re-arms the same watchdog on every chunk, turning it into a true idle timeout: only turnTimeout seconds of silence trips it, regardless of how long the turn has been running in total. This preserves the watchdog's actual purpose (rescue a turn that will never send turn/completed — the app-server does not reliably send it, see fujibee#41) — a turn that is visibly still working is not that case. Observed in production: a Codex bridge turn spent >60s diagnosing a GitHub auth/DNS failure (correctly identifying an invalid gh CLI token and a network-restricted sandbox) but the watchdog fired before it reached send.sh, so the diagnosis was silently lost and the requesting peer saw only silence.
…ot just message deltas Review finding (self-multi-model, Codex, P1): the previous commit only re-armed the watchdog from item/agentMessage/delta, but a Codex turn spends most of its time in other notification types the bridge has no handler for — reasoning deltas, tool-call/command-output progress, etc. Those still went through handleLine() but were silently dropped before dispatch (no handler registered), so the watchdog kept counting from turn start and could still fire mid-turn during exactly the kind of multi-step work this fix is meant to protect. Added a generic AppServerClient/WebSocketAppServerClient.onThreadActivity hook that fires for every thread-scoped notification/request in handleLine(), regardless of whether a specific handler is registered for its method. CodexBridge wires it to re-arm the watchdog for any activity on its own active turn. onAgentMessageDelta's specific re-arm is now redundant (the generic hook already covers it) and has been removed. New test proves the fix covers non-message activity: a turn that only emits item/reasoning/textDelta and item/commandExecution/outputDelta (never agentMessage/delta) still survives past turn-timeout without being cut off.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
startTurnWatchdog()arms a single fixed-duration timer from turn start(
--turn-timeout, default 60s) and never touches it again until the turnends. A turn that is genuinely still working — reasoning, running tools,
composing a multi-step reply — for longer than that gets cut off by this
timer before it ever reaches its own
send.shcall.onTurnEnded()thenruns exactly as it does on a real completion, so the bridge silently re-arms
as if the turn had ended with nothing to report; whatever it was about to
send never goes out, and the requesting peer sees only silence.
This turns the watchdog into a true idle timeout: any thread-scoped
app-server activity for the active turn re-arms it, so a turn is only ever
cut off after
turnTimeoutseconds of true silence, regardless of how longit has been running in total. This preserves the watchdog's actual purpose
(rescue a turn that will never send
turn/completed— the app-server doesnot reliably send it, see #41) — a turn that is visibly still working is not
that case.
Implementation
AppServerClient/WebSocketAppServerClient(both transports) gain anonThreadActivitycallback, invoked fromhandleLine()for everythread-scoped notification/request — including the many the bridge has no
specific handler for (reasoning deltas, tool-call/command-output progress,
etc.), which the existing
handlersMap otherwise drops silently beforedispatch. This is deliberately generic rather than enumerating specific
event names: the exact set of "the turn is doing something" notifications
isn't fully known/stable to this bridge, and a generic hook can't miss one.
CodexBridgewiresonThreadActivityto re-armstartTurnWatchdog()forits own active thread/turn.
onAgentMessageDelta's own re-arm (an earlier, narrower iteration of thisfix) is now redundant and removed — the generic hook already covers it.
Observed in production
A Codex bridge turn spent well over 60s diagnosing a GitHub auth/DNS
failure (correctly identifying an invalid
ghCLI token and anetwork-restricted sandbox, and proposing next steps) but the watchdog fired
before it reached
send.sh, so the diagnosis was silently lost and therequesting peer received nothing.
Test plan
tests/test_codex_bridge.bats:turn-timeout" (agent-message deltas keep it alive well past the nominal
timeout, then a real
turn/completedends it — never the watchdog).the idle watchdog" (same shape, but only
item/reasoning/textDeltaanditem/commandExecution/outputDelta— neveragentMessage/delta— toprove the fix isn't scoped to one specific notification type).
bats tests/test_codex_bridge.bats tests/test_codex_bridge_launcher.bats—47/47 passing (30 pre-existing + 2 new here, plus the launcher suite),
including the existing watchdog regression tests (Explore a monitor-equivalent delivery mode for Codex #41, Codex bridge: thread stuck in waitingOnApproval deadlocks delivery forever (bridge can't answer approval requests; launcher never replaces the hung bridge) #299) unmodified
and still green.
Reviewed via
/review:self-multi-model(Codex + Fugu) before submission:Codex's first pass correctly flagged that an earlier iteration of this fix
only re-armed on
agentMessage/deltaand missed the multi-stepreasoning/tool-call case above — addressed by the generic
onThreadActivityhook in the commit history here, with a dedicatedregression test.
CI green