Skip to content

perf(paths): batch the resolved-path fallback for /nodes/{pk}/paths (#2223) - #2224

Open
efiten wants to merge 1 commit into
Kpa-clawbot:masterfrom
efiten:perf/paths-resolved-path-fallback
Open

efiten wants to merge 1 commit into
Kpa-clawbot:masterfrom
efiten:perf/paths-resolved-path-fallback

Conversation

@efiten

@efiten efiten commented Oct 15, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #2223.

Problem

loadCanonicalResolvedPaths reads the resolved path of each candidate's longest observation in batches (#2146). A transmission whose longest observation has no stored resolved_path then went through bestResolvedPath on its own, with two queries:

  • fetchResolvedPathForObs re-read that same observation, which the batch had just read;
  • fetchResolvedPathsForTx read all of the transmission's observations.

On a production database, 24% of the observations with a path have no resolved_path, so most /api/nodes/{pubkey}/paths requests paid for this once per such transmission.

Change

  • loadFallbackResolvedPaths batches the fallback. It reads those transmissions' stored paths in one query per resolvedPathBatchSize transmissions:

    SELECT id, transmission_id, resolved_path FROM observations
    WHERE resolved_path IS NOT NULL AND transmission_id IN (...)

    On a production database the planner runs it as SEARCH observations USING INDEX idx_observations_tx_ts (transmission_id=?). It then picks the observation bestResolvedPath would pick: the longest path_json with a stored path, the first in snapshot order on a tie. The result goes into the resolved-path LRU, as before.

  • longestStoredResolvedPath holds that choice. bestResolvedPath now uses it as well, so the rule exists in one place.

  • A failed batch still falls back. If the first-stage batch fails, it goes through bestResolvedPath per transmission, as before. A failed fallback batch does the same for its transmissions.

The answer is the same as before, and no cache is added.

Perf

Staging, 9 nodes, requests made from the host itself (localhost:3001), so no proxy or TLS. Each node 3 times, after one warm-up round:

median p90 max total for 27 requests
before 0.174 s 0.604 s 0.866 s 6.95 s
after 0.079 s 0.433 s 0.484 s 3.22 s

CPU profile, 30 s each, under the same request loop:

  • Requests completed: 180 before, 261 after.
  • CPU in handleNodePaths per request: 154 ms before (27.77 s / 180), 107 ms after (27.90 s / 261).
  • Fallback cost: bestResolvedPath took 13.56 s of loadCanonicalResolvedPaths before. The batched loadFallbackResolvedPaths takes 6.57 s.

Caveats:

  • The before run was on a process that had been up longer than the after run, so the two LRU states differ.
  • One node (060679ad…, about 11k transmissions) did not change: 0.455 s against 0.451 s. Its time is elsewhere in the handler, and this PR does not address it.

Tests

  • TestLoadCanonicalResolvedPathsBatchesFallback: 1,001 transmissions whose longest observation has no stored path.
    • On master: 2,002 per-transmission queries.
    • Now: 0 per-transmission queries and 6 batched ones (3 for the longest observations, 3 for the fallback), with every path taken from the shorter observation.
  • Existing tests: TestLoadCanonicalResolvedPathsMatchesBestResolvedPath still passes. It covers the longest observation without a path (QA #809 §2.1: /api/nodes/{pk}/health.recentPackets[] missing resolved_path #810), ties, nothing stored, an empty stored path, and an observation absent from the snapshot. The LRU and query-count tests also still pass.
  • go vet and go test . in cmd/server pass.

No frontend change.

…pa-clawbot#2223)

loadCanonicalResolvedPaths read the longest observation of each candidate
in batches, but a transmission whose longest observation had no stored
resolved_path went through bestResolvedPath on its own: one query for that
same observation, which the batch had just read, and one for all of its
observations. 24% of the observations with a path had no resolved_path on
a production database, and a CPU profile of /paths on staging put 65% of
loadCanonicalResolvedPaths in that fallback.

loadFallbackResolvedPaths reads those transmissions' stored paths in one
query per resolvedPathBatchSize transmissions and picks the same
observation bestResolvedPath does, through the now shared
longestStoredResolvedPath. A failed batch still goes through
bestResolvedPath per transmission.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

perf: /api/nodes/{pubkey}/paths runs two queries per transmission without a stored resolved_path

1 participant