Skip to content

perf(store): key byNode's de-dup sets by transmission ID - #35

Closed
A13xB0 wants to merge 4 commits into
masterfrom
perf/node-sets-by-id
Closed

A13xB0 wants to merge 4 commits into
masterfrom
perf/node-sets-by-id

Conversation

@A13xB0

@A13xB0 A13xB0 commented Oct 11, 2026 •

Copy link
Copy Markdown
Owner

Problem

nodeHashes is the set behind byNode's de-dup: for every node, the transmissions already listed under it. It was keyed by each transmission's 16-character hash string. On a week of a busy network that's about 650k entries and 21.5 MB, four times the size of byNode itself.

Change

  • The sets are keyed by transmission ID. That's the same membership, since a store holds one transmission per hash.
  • Eviction deletes by ID.
  • The health endpoint's region filter matches regional observations by transmission ID.
  • newNodeSets and nodeHasTx wrap the map for callers and tests.

Measured

before after
node sets' live heap (copy of a busy week) 21.5 MB 13.2 MB
fixture, 500k (node, packet) pairs 25.8 MB 18.0 MB

Tests

  • TestNodeSetRetainedHeap: red commit first.
  • TestAddToByNodeDedup.
  • The existing eviction, routes and resolved-index tests updated to the ID-keyed sets.
  • go test ./cmd/server passes.

No config or UI changes.

🤖 Generated with Claude Code

https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H

@A13xB0 A13xB0 added upstream: not submitted Fork only so far; not opened upstream performance Memory or speed improvement live: pre-prod In the image running on CoreScope pre-production labels Oct 11, 2026
A13xB0 and others added 2 commits October 13, 2026 04:12
Red: nodeHashes keeps 25.8MB for 500k (node, packet) pairs, keyed by the
packets' hash strings. newNodeSets and nodeHasTx wrap today's map.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H
nodeHashes is the set behind byNode's de-dup: for every node, the
transmissions already listed under it. It was keyed by each
transmission's 16-character hash string: ~650k entries and 21.5MB on a
busy week. It is now keyed by transmission ID, the same membership (a
store holds one transmission per hash); eviction deletes by ID, and the
health region filter matches regional observations by transmission ID.
nodeHasTx reads the set for callers and tests.

On a copy of that week: the sets' live heap 21.5MB -> 13.2MB. Fixture
(500k pairs): 25.8MB -> 18.0MB.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H
@A13xB0
A13xB0 force-pushed the perf/node-sets-by-id branch from 4bb926b to fcc7451 Compare October 13, 2026 03:26
@A13xB0 A13xB0 added upstream: in review Open as a PR on Kpa-clawbot/CoreScope, waiting on review and removed upstream: not submitted Fork only so far; not opened upstream labels Oct 13, 2026
@A13xB0

A13xB0 commented Oct 13, 2026

Copy link
Copy Markdown
Owner Author

Rebased onto upstream master and submitted as Kpa-clawbot#2199.

A13xB0 and others added 2 commits October 15, 2026 10:09
…ard there

Observation IDs differ from transmission IDs in the fixture, so a filter
keyed by the wrong ID returns nothing and fails.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H
@A13xB0 A13xB0 added upstream: merged Merged into Kpa-clawbot/CoreScope master and removed upstream: in review Open as a PR on Kpa-clawbot/CoreScope, waiting on review labels Oct 18, 2026
@A13xB0

A13xB0 commented Oct 18, 2026

Copy link
Copy Markdown
Owner Author

Merged upstream as Kpa-clawbot#2199; closing.

@A13xB0 A13xB0 closed this Oct 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

live: pre-prod In the image running on CoreScope pre-production performance Memory or speed improvement upstream: merged Merged into Kpa-clawbot/CoreScope master

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant