[A13xB0]

Refs #1870.

Problem

Load, loadChunk and scanAndMergeChunk decode each observation's resolved_path only to collect its non-empty pubkeys. On live data that's a JSON array of pubkeys and nulls, about 420 characters, present on most observations. Decoding into []*string allocated a string and a pointer for every hop, plus two slices.

Change

• resolvedPubkeysFromJSON reads the usual array of escape-free strings and nulls in place. Anything else falls back to decoding, so it returns exactly extractResolvedPubkeys(unmarshalResolvedPath(s)).
• Relay pubkeys repeat across nearly every observation, so each one goes through internStr (from perf(store): share one copy of each observer string across observations #2183) and is kept once. A pubkey seen before is found by substring, without copying, and the indexes it feeds (byNode and friends) share that copy. Interned keys live for the process, evicted nodes' included; the table is bounded at 64k entries.
• The three bulk-load call sites use it.
• Eviction too. resolvedPubkeysForEvictionBatch runs on every eviction batch, about once a minute in steady state, and spent about 130 MB per five minutes on a busy week. It now reads each pubkey in place and de-duplicates each packet's short list by scanning it instead of building a map per packet.
• Other callers of unmarshalResolvedPath are unchanged.

Measured

	before	after
allocations for 3 known pubkeys	13	1
eviction, allocations per evicted packet (3 observations × 4 hops)	51	15
this path's allocations over startup (copy of a busy week)	236 MB	61 MB
server CPU over the run	175.6 s	170.6 s

The startup rows need that dataset and are from a single run. The benchmarks are rerunnable (go test -run X -bench ResolvedPubkeys -benchmem ./cmd/server, medians of 3):

	master	this PR
BenchmarkResolvedPubkeysFromJSON (5 relays + 3 nulls, pool of 2,000)	n/a	619 ns, 128 B, 1 alloc
BenchmarkResolvedPubkeysForEvictionBatch (1,000 packets × 3 observations)	14.3 ms, 3.96 MB, 65,495 allocs	6.7 ms, 1.36 MB, 18,501 allocs

Tests

• TestResolvedPubkeysFromJSONAllocations: red commit first, with the bound tightened in green.
• TestResolvedPubkeysForEvictionBatch: keys, order, and an allocation bound. It has its own red commit.
• TestResolvedPubkeysFromJSONMatchesDecoding compares against decode-then-extract on 26 edge cases and 20,000 random inputs.
• go test ./cmd/server passes.

No config or UI changes.

🤖 Generated with Claude Code

https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H

[efiten]

The scanner and the pubkey interning hold up, and the gains reproduce. One part should go before merge, and the interner fold should happen in this PR.

What I checked

• The scanner. It only takes the fast path for plain, escape-free ASCII strings and nulls; everything else falls back to the old decode. TestResolvedPubkeysFromJSONMatchesDecoding (26 edge cases plus 20,000 random inputs) is good evidence that the two agree.
• The interner.
  • It never stores a substring: a lookup by sub does not copy, and only a new key is cloned.
  • It is bounded at 1<<16 entries.
• Eviction behaviour is unchanged:
  • first-seen order stays;
  • deduplication still works per packet across observations;
  • NULL and empty values are skipped;
  • the SQL fetch still runs outside s.mu.
• Throwaway benchmarks (-benchmem -count=6, medians):
  • One path with 5 pubkeys and 3 nulls, from a pool of 2,000 relays: allocations go from 19 to 1, bytes from 1176 B to 128 B, time from about 6.4 µs to 0.79 µs.
  • resolvedPubkeysForEvictionBatch, 1,000 packets × 3 observations: allocations go from 80,495 to 20,504, bytes from 5.97 MB to 2.18 MB.
  • Both are consistent with your figures.
• Suite: go vet and go test ./cmd/server pass on the head and on the head merged into current master.

Requested changes

1. Drop the sql.RawBytes + unsafe.String part (resolved_index.go:521 and :535).
   • Why it doesn't help: mattn/go-sqlite3 v1.14.52 already allocates a Go string per TEXT column (C.GoStringN). Scanning that into sql.NullString takes it without a copy. Scanning into RawBytes adds a memcpy into database/sql's reused rawbuf, so the comment "rpJSON is the driver's buffer" is not accurate either.
   • Measured: I reverted only that part to NullString on your head. Allocations were the same (20,501 against 20,504 per op), bytes slightly lower, and time not worse.
   • So the unsafe import adds an aliasing risk with no measured gain.
2. Fold pubkeyInterner into internStr now. #2183 has landed, so this is the "whichever lands second" point from your description.
   • Order matters: past its cap, internStr returns s as passed, without cloning (intern.go:44-45). With the unsafe.String view still in eviction, such a returned key would alias the reused rawbuf and be overwritten by the next row. That would put wrong keys in rpBatch and leave stale entries in byNode and the resolved-pubkey index.
   • So item 1 has to come before the fold, or the fold needs a variant that clones on overflow.
   • Your allocation tests should still pass afterwards, since internStr's hit path does not allocate.

Notes, not blocking

1. Interned pubkeys live for the whole process, including keys of nodes that have since been evicted. It is bounded at 64k, so this is acceptable, but worth a sentence in the comment, as intern.go has.
2. A real Benchmark with realistic sizes would make the numbers rerunnable. The startup totals (236 MB to 61 MB, CPU 175.6 s to 170.6 s) need your dataset, and the CPU difference is under 3% from a single run.

[A13xB0]

Review feedback addressed (commit a7f00ad):

1. sql.RawBytes + unsafe.String dropped. Eviction scans resolved_path into sql.NullString again, so the unsafe import and the "driver's buffer" comment are gone.
2. Folded into internStr. pubkeyInterner is removed and scanResolvedPath calls internStr(s[start:i]). Item 1 went first, so a key returned uncloned past internMax is a slice of its own row's string, not of a reused buffer.

Notes:

• Lifetime: the resolvedPubkeysFromJSON comment now says that interned keys live for the process, evicted nodes' keys included, bounded by internMax.
• Benchmarks: BenchmarkResolvedPubkeysFromJSON and BenchmarkResolvedPubkeysForEvictionBatch are in resolved_pubkeys_bench_test.go. For the eviction batch (1,000 packets × 3 observations, medians of 3), master takes 14.3 ms / 3.96 MB / 65,495 allocs, and this PR takes 6.7 ms / 1.36 MB / 18,501 allocs. The table in the description is updated, and says the startup figures are from a single run on my dataset.

The branch is rebased onto current master. go vet, go test ./cmd/server, and -race on the Resolved/Eviction tests pass.

[A13xB0]

Bump

[efiten]

Thanks, both requested changes are in, in the right order. Approving.

What I checked at a7f00ad6

• git range-diff against the series I reviewed: the first four commits are unchanged, a7f00ad6 is the only new one.
• Eviction scans into sql.NullString again (resolved_index.go:520), and unsafe is gone. scanResolvedPath calls internStr(s[start:i]) (neighbor_persist.go:292), so a key returned past the cap is a slice of its own row string, not of a reused buffer. internMax (intern.go:11) bounds the table, and internStr is safe for concurrent callers.
• go vet and gofmt -l are clean; go test -count=1 ./... in cmd/server and -race on the Resolved, Eviction and Intern tests pass.
• Benchmarks on Windows, interleaved base/head, medians of 3, with your eviction benchmark copied onto the base: BenchmarkResolvedPubkeysForEvictionBatch 7.57 ms / 3.96 MB / 65,495 allocs on master against 3.10 ms / 1.36 MB / 18,501 allocs here. Bytes and allocations match your figures exactly; the time ratio is 2.4x. BenchmarkResolvedPubkeysFromJSON: 392-420 ns, 128 B, 1 alloc.
• Mutations on the scanner: dropping the >= 0x80, < 0x20 or trailing-garbage checks, keeping empty strings, disabling eviction dedup, or cloning instead of interning each makes a test fail.

Small points, not blocking

1. Past the cap, a key pins its row. Once the table is full, internStr returns s[start:i] as passed, and that key is kept in nodeHashes/byNode, so it holds the whole ~420-byte row string. The old pubkeyInterner cloned on overflow. Only reachable after 65,536 interned strings; a clone at the call site when the result aliases s, or a sentence in the comment, would settle it.
2. The intern.go comment (:8-10, :23-24) still says the table holds observer IDs, names, IATA codes and directions, "in the hundreds". It now also holds every resolved relay pubkey.
3. Two test gaps. Replacing internStr(s[start:i]) with plain s[start:i] passes every test, so nothing pins "one copy each"; comparing the backing pointer of two calls would. Removing the backslash check at neighbor_persist.go:280 also passes; ["a\\b"] and ["\\"] as edge cases would cover it. Real pubkeys are hex, so neither affects users.

Not checked: the startup figures, which need your dataset.