fix: time out a daemon RPC call instead of hanging forever - #177
Merged
Merged
Conversation
Rpc.call() in the daemon client (SocketBackend) had no timeout: if the daemon fell behind (its outbox flusher wedged/backlogged behind a slow op under heavy contention, e.g. many processes sharing one git+https bus), a client command like `inbox` or `ack` would wait forever for a response that might never come, with no error and no way to distinguish it from ordinary latency. Every call now fails with a clear error after AGENTCOMM_RPC_TIMEOUT_MS (default 20s) instead. The daemon design keeps client calls off the network path (put/move/moveMany spool locally and flush asynchronously), so a healthy daemon should always answer well within that -- the timeout is sized for "something is stuck," not for git's own round-trip time. Reproduced against a real bus (git+https backend, ~70 concurrent CLI processes sharing one daemon): the daemon's outbox backlogged to 46 entries, and a plain `inbox --json` call hung indefinitely after successfully delivering messages, stuck in the archive/moveMany RPC. Restarting the daemon cleared the immediate backlog; this fix ensures a recurrence fails loudly instead of hanging the caller. New test (test/socket-rpc-timeout.test.ts) drives SocketBackend against a fake daemon that answers `info` (so connect() succeeds) and then goes silent, asserting the client rejects instead of hanging, and that no Timeout handle is left active afterward. Mutation-tested: reverting the fix reproduces the hang (the test never resolves).
yonidavidson
approved these changes
Sep 15, 2026
Merged
yonidavidson
added a commit
that referenced
this pull request
Sep 15, 2026
Daemon reliability, from eran yahav's field report (#177): - #177 a daemon RPC call now times out instead of hanging forever — when the daemon's outbox backlogged under heavy contention on a shared bus, a client command like inbox or ack would wait forever with no output; every call now fails with a clear error after AGENTCOMM_RPC_TIMEOUT_MS (default 20s).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Rpc.call()inSocketBackend(the daemon client) had no timeout: if the daemon fell behind— its outbox flusher backlogged behind a slow op under heavy contention, e.g. many processes
sharing one
git+httpsbus — a client command likeinboxorackwould wait forever for aresponse that might never come. No error, no way to tell a hang apart from ordinary latency.
Every call now fails with a clear error after
AGENTCOMM_RPC_TIMEOUT_MS(default 20s) insteadof hanging. The daemon design already keeps client calls off the network path (
put/move/moveManyspool locally and flush asynchronously), so a healthy daemon should always answerwell within that — the timeout is sized for "something is stuck," not for git's own round-trip
time.
How I found this
Reproduced against a real bus (
git+httpsbackend, ~70 concurrent CLI processes sharing onedaemon across many AI agent sessions working the same repo). The daemon's outbox backlogged to
46 entries, and a plain
inbox --jsoncall hung indefinitely — it successfully deliveredmessages (the
delivercallback fired), then hung inside thearchive/moveManyRPC callwith no output and no way to recover except killing the process.
daemon inforeportedoutbox: 46, flushFailures: 0— no error, just a growing, unresponsive backlog.Restarting the daemon (
daemon stop) cleared the immediate backlog and confirmed a freshdaemon can still reproduce the same class of hang under the same load, since nothing in the
client bounds how long it waits. This PR doesn't address the backlog/throughput question
itself (that's environmental — heavy contention on a shared git remote across many agents) —
it ensures a caller isn't left staring at a frozen terminal forever when it happens, and gets
an actionable error pointing at
AGENTCOMM_RPC_TIMEOUT_MS/--direct.Test plan
test/socket-rpc-timeout.test.ts): drivesSocketBackendagainst a fakedaemon that answers
info(soconnect()succeeds) and then goes silent, asserting theclient rejects instead of hanging, and that no
Timeouthandle is left activeafterward (
process._getActiveHandles()).resolves) rather than failing — restored and confirmed green.
npm run typecheckclean.npm run buildclean.test/daemon.e2e.test.ts(14 tests) passes unchanged — the timeout doesn't affectnormal daemon round-trips.
vitest run: 243 passed, 45 failed — all 45 failures are pre-existing andexclusively in the sqlite-backend suites, caused by
better-sqlite3's native buildbeing incompatible with the Node version available in my sandbox (v26.7.0, very
recent). Unrelated to this change; CI builds natively and should be unaffected.