Repository navigation
Conversation
One IPC message holds at most 1 MB. Past about 10,000 peers the info reply no longer fit, nor did the trusted list past about 8,000 records: the write failed, the daemon closed the client's connection, and every command that asked failed with "daemon disconnected" (pilotctl info, peers, connections, trust, and send-message, whose later requests used the closed connection). Seen on service agents with 10,909 peers and over 20,000 trust records. Send as many rows as fit, peers with an open connection and the newest trust records first, and mark the list as truncated; the counts keep the totals. pilotctl peers and trust say when a list was cut. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
runHandler reads with a 500 ms deadline; building a 12,000-row reply takes 0.6 to 1.1 s under -race, longer on CI runners. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
One IPC message holds at most 1 MB (
ipcutil.MaxMessageSize). Theinforeply lists every peer (~99 bytes each) and the trusted-peers reply every trust record (~129 bytes each), so past about 10,000 peers or 8,000 trust records the reply no longer fits.ipcutil.Writerefuses it,writeLoopcloses the client's connection, and the command fails withdaemon disconnected, along with every later request on that connection.Seen on the service-agent fleet (read-only check, 2026-10-06):
inforeplyWhat broke on such a node:
pilotctl info,peers,connections,trust(agents hold over 20,000 trust records, sotrustfailed on all of them), andpilotctl send-message. Its first-contactinfofails quietly, but the daemon has closed the connection, so the trust check and dial that follow fail too. The Python responder's post-handshakepilotctl peerscheck could never succeed there.Change
peer_list_truncated/trusted_truncated: true, and leaves every other field intact (peers,connectionsand the other counts still give the totals). Replies that fit are byte-for-byte unchanged.info, all connection rows, then peers with an open connection first, then the rest in node order. For trust records, the newest first, which is the orderpilotctl trustshows them in.pilotctl peersandpilotctl trustsay when the daemon's list was cut (text and--json:truncated, plusdaemon_peersfor peers).A complete listing on such nodes needs paging or a filtered query, which needs a driver change in
common. This PR makes the commands work again without changing the protocol.Tests
TestInfoReplyFitsOneIPCMessage(12,000 peers, one in use): 10,139 rows in 1,048,512 bytes, flagged, in-use peer first. Onmainit fails withipcutil.Read: EOF, the same "daemon disconnected".TestTrustedReplyFitsOneIPCMessage(24,000 records): the 8,127 newest, flagged. Fails onmainthe same way.TestMarshalRowsWithinBudgetKeepsAllThatFit: output ≤ budget, and one more row would not fit.TestCmdPeersReportsCutList,TestCmdTrustReportsCutList.go test ./pkg/... ./cmd/... ./internal/... -shortpasses.Conflicts with #510 (both edit
handleInfo); whichever merges second needs a small rebase.🤖 Generated with Claude Code