Skip to content

fix: time out a daemon RPC call instead of hanging forever - #177

Merged
yonidavidson merged 1 commit into
yonidavidson:mainfrom
yahave:fix-socket-rpc-timeout
Sep 15, 2026
Merged

yonidavidson merged 1 commit into
yonidavidson:mainfrom
yahave:fix-socket-rpc-timeout

Conversation

@yahave

@yahave yahave commented Sep 14, 2026

Copy link
Copy Markdown
Contributor

Summary

Rpc.call() in SocketBackend (the daemon client) had no timeout: if the daemon fell behind
— its outbox flusher backlogged behind a slow op under heavy contention, e.g. many processes
sharing one git+https bus — a client command like inbox or ack would wait forever for a
response that might never come. No error, no way to tell a hang apart from ordinary latency.

Every call now fails with a clear error after AGENTCOMM_RPC_TIMEOUT_MS (default 20s) instead
of hanging. The daemon design already keeps client calls off the network path (put/move/
moveMany spool locally and flush asynchronously), so a healthy daemon should always answer
well within that — the timeout is sized for "something is stuck," not for git's own round-trip
time.

How I found this

Reproduced against a real bus (git+https backend, ~70 concurrent CLI processes sharing one
daemon across many AI agent sessions working the same repo). The daemon's outbox backlogged to
46 entries, and a plain inbox --json call hung indefinitely — it successfully delivered
messages (the deliver callback fired), then hung inside the archive/moveMany RPC call
with no output and no way to recover except killing the process. daemon info reported
outbox: 46, flushFailures: 0 — no error, just a growing, unresponsive backlog.

Restarting the daemon (daemon stop) cleared the immediate backlog and confirmed a fresh
daemon can still reproduce the same class of hang under the same load, since nothing in the
client bounds how long it waits. This PR doesn't address the backlog/throughput question
itself (that's environmental — heavy contention on a shared git remote across many agents) —
it ensures a caller isn't left staring at a frozen terminal forever when it happens, and gets
an actionable error pointing at AGENTCOMM_RPC_TIMEOUT_MS/--direct.

Test plan

  • New test (test/socket-rpc-timeout.test.ts): drives SocketBackend against a fake
    daemon that answers info (so connect() succeeds) and then goes silent, asserting the
    client rejects instead of hanging, and that no Timeout handle is left active
    afterward (process._getActiveHandles()).
  • Mutation-tested: reverted the fix locally and confirmed the new test hangs (never
    resolves) rather than failing — restored and confirmed green.
  • npm run typecheck clean.
  • npm run build clean.
  • test/daemon.e2e.test.ts (14 tests) passes unchanged — the timeout doesn't affect
    normal daemon round-trips.
  • Full vitest run: 243 passed, 45 failed — all 45 failures are pre-existing and
    exclusively in the sqlite-backend suites, caused by better-sqlite3's native build
    being incompatible with the Node version available in my sandbox (v26.7.0, very
    recent). Unrelated to this change; CI builds natively and should be unaffected.

Rpc.call() in the daemon client (SocketBackend) had no timeout: if the
daemon fell behind (its outbox flusher wedged/backlogged behind a slow op
under heavy contention, e.g. many processes sharing one git+https bus),
a client command like `inbox` or `ack` would wait forever for a response
that might never come, with no error and no way to distinguish it from
ordinary latency.

Every call now fails with a clear error after AGENTCOMM_RPC_TIMEOUT_MS
(default 20s) instead. The daemon design keeps client calls off the
network path (put/move/moveMany spool locally and flush asynchronously),
so a healthy daemon should always answer well within that -- the timeout
is sized for "something is stuck," not for git's own round-trip time.

Reproduced against a real bus (git+https backend, ~70 concurrent CLI
processes sharing one daemon): the daemon's outbox backlogged to 46
entries, and a plain `inbox --json` call hung indefinitely after
successfully delivering messages, stuck in the archive/moveMany RPC.
Restarting the daemon cleared the immediate backlog; this fix ensures a
recurrence fails loudly instead of hanging the caller.

New test (test/socket-rpc-timeout.test.ts) drives SocketBackend against a
fake daemon that answers `info` (so connect() succeeds) and then goes
silent, asserting the client rejects instead of hanging, and that no
Timeout handle is left active afterward. Mutation-tested: reverting the
fix reproduces the hang (the test never resolves).
@yonidavidson
yonidavidson merged commit e61647f into yonidavidson:main Sep 15, 2026
2 checks passed
yonidavidson added a commit that referenced this pull request Sep 15, 2026
Daemon reliability, from eran yahav's field report (#177):

- #177 a daemon RPC call now times out instead of hanging forever — when
  the daemon's outbox backlogged under heavy contention on a shared bus,
  a client command like inbox or ack would wait forever with no output;
  every call now fails with a clear error after AGENTCOMM_RPC_TIMEOUT_MS
  (default 20s).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants