Skip to content

daemon: info and trusted replies fit one IPC message - #512

Open
TeoSlayer wants to merge 2 commits into
mainfrom
fix/ipc-reply-fits
Open

TeoSlayer wants to merge 2 commits into
mainfrom
fix/ipc-reply-fits

Conversation

@TeoSlayer

Copy link
Copy Markdown
Collaborator

Summary

One IPC message holds at most 1 MB (ipcutil.MaxMessageSize). The info reply lists every peer (~99 bytes each) and the trusted-peers reply every trust record (~129 bytes each), so past about 10,000 peers or 8,000 trust records the reply no longer fits. ipcutil.Write refuses it, writeLoop closes the client's connection, and the command fails with daemon disconnected, along with every later request on that connection.

Seen on the service-agent fleet (read-only check, 2026-10-06):

Agent Peers info reply
coinbase-spot-price 8,056 808,612 bytes (77% of the limit)
github-public 10,909 none: daemon closes the connection

What broke on such a node: pilotctl info, peers, connections, trust (agents hold over 20,000 trust records, so trust failed on all of them), and pilotctl send-message. Its first-contact info fails quietly, but the daemon has closed the connection, so the trust check and dial that follow fail too. The Python responder's post-handshake pilotctl peers check could never succeed there.

Change

  • When a reply would exceed the budget, the daemon sends as many list rows as fit, adds peer_list_truncated / trusted_truncated: true, and leaves every other field intact (peers, connections and the other counts still give the totals). Replies that fit are byte-for-byte unchanged.
  • What is kept: for info, all connection rows, then peers with an open connection first, then the rest in node order. For trust records, the newest first, which is the order pilotctl trust shows them in.
  • pilotctl peers and pilotctl trust say when the daemon's list was cut (text and --json: truncated, plus daemon_peers for peers).

A complete listing on such nodes needs paging or a filtered query, which needs a driver change in common. This PR makes the commands work again without changing the protocol.

Tests

  • TestInfoReplyFitsOneIPCMessage (12,000 peers, one in use): 10,139 rows in 1,048,512 bytes, flagged, in-use peer first. On main it fails with ipcutil.Read: EOF, the same "daemon disconnected".
  • TestTrustedReplyFitsOneIPCMessage (24,000 records): the 8,127 newest, flagged. Fails on main the same way.
  • TestMarshalRowsWithinBudgetKeepsAllThatFit: output ≤ budget, and one more row would not fit.
  • TestCmdPeersReportsCutList, TestCmdTrustReportsCutList.
  • go test ./pkg/... ./cmd/... ./internal/... -short passes.

Conflicts with #510 (both edit handleInfo); whichever merges second needs a small rebase.

🤖 Generated with Claude Code

Teo Calin and others added 2 commits October 6, 2026 02:33
One IPC message holds at most 1 MB. Past about 10,000 peers the info
reply no longer fit, nor did the trusted list past about 8,000 records:
the write failed, the daemon closed the client's connection, and every
command that asked failed with "daemon disconnected" (pilotctl info,
peers, connections, trust, and send-message, whose later requests used
the closed connection). Seen on service agents with 10,909 peers and
over 20,000 trust records.

Send as many rows as fit, peers with an open connection and the newest
trust records first, and mark the list as truncated; the counts keep
the totals. pilotctl peers and trust say when a list was cut.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
runHandler reads with a 500 ms deadline; building a 12,000-row reply
takes 0.6 to 1.1 s under -race, longer on CI runners.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant