Repository navigation
Conversation
…pa-clawbot#2223) loadCanonicalResolvedPaths read the longest observation of each candidate in batches, but a transmission whose longest observation had no stored resolved_path went through bestResolvedPath on its own: one query for that same observation, which the batch had just read, and one for all of its observations. 24% of the observations with a path had no resolved_path on a production database, and a CPU profile of /paths on staging put 65% of loadCanonicalResolvedPaths in that fallback. loadFallbackResolvedPaths reads those transmissions' stored paths in one query per resolvedPathBatchSize transmissions and picks the same observation bestResolvedPath does, through the now shared longestStoredResolvedPath. A failed batch still goes through bestResolvedPath per transmission. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #2223.
Problem
loadCanonicalResolvedPathsreads the resolved path of each candidate's longest observation in batches (#2146). A transmission whose longest observation has no storedresolved_paththen went throughbestResolvedPathon its own, with two queries:fetchResolvedPathForObsre-read that same observation, which the batch had just read;fetchResolvedPathsForTxread all of the transmission's observations.On a production database, 24% of the observations with a path have no
resolved_path, so most/api/nodes/{pubkey}/pathsrequests paid for this once per such transmission.Change
loadFallbackResolvedPathsbatches the fallback. It reads those transmissions' stored paths in one query perresolvedPathBatchSizetransmissions:On a production database the planner runs it as
SEARCH observations USING INDEX idx_observations_tx_ts (transmission_id=?). It then picks the observationbestResolvedPathwould pick: the longestpath_jsonwith a stored path, the first in snapshot order on a tie. The result goes into the resolved-path LRU, as before.longestStoredResolvedPathholds that choice.bestResolvedPathnow uses it as well, so the rule exists in one place.A failed batch still falls back. If the first-stage batch fails, it goes through
bestResolvedPathper transmission, as before. A failed fallback batch does the same for its transmissions.The answer is the same as before, and no cache is added.
Perf
Staging, 9 nodes, requests made from the host itself (
localhost:3001), so no proxy or TLS. Each node 3 times, after one warm-up round:CPU profile, 30 s each, under the same request loop:
handleNodePathsper request: 154 ms before (27.77 s / 180), 107 ms after (27.90 s / 261).bestResolvedPathtook 13.56 s ofloadCanonicalResolvedPathsbefore. The batchedloadFallbackResolvedPathstakes 6.57 s.Caveats:
060679ad…, about 11k transmissions) did not change: 0.455 s against 0.451 s. Its time is elsewhere in the handler, and this PR does not address it.Tests
TestLoadCanonicalResolvedPathsBatchesFallback: 1,001 transmissions whose longest observation has no stored path.TestLoadCanonicalResolvedPathsMatchesBestResolvedPathstill passes. It covers the longest observation without a path (QA #809 §2.1: /api/nodes/{pk}/health.recentPackets[] missing resolved_path #810), ties, nothing stored, an empty stored path, and an observation absent from the snapshot. The LRU and query-count tests also still pass.go vetandgo test .incmd/serverpass.No frontend change.