[dborup]

Summary

Relates to #412.

Move runtime packet/detail/filter/analytics database work outside PacketStore.mu while retaining synchronous cold/expired-cache behavior. A slow SQL request may still delay its caller, but must not create a packet-store reader convoy that holds unrelated ingest writers back.

• Build request-owned response fields and immutable resolved-path inputs under the store lock; enrich from SQL/LRU only after unlocking and before returning.
• Prepare node-name, region/area, observer and topology data before taking the store lock. Coalesce node-cache misses and prevent an invalidated in-flight refresh from republishing stale cache state.
• Pass captured immutable node/prefix snapshots through distance and relay index helpers, rather than warming a cache and then rereading it after invalidation.
• Close ingest cursors before auxiliary metadata queries, including single-connection pools.
• Preserve the fork's hash-migration snapshots, LRU expiry/bounds, canonical/longest-path fallback, filtering, listener-only handling and read-only database separation.

No schema, configuration, frontend, ingestor-binary or workflow changes. Auto-merge remains disabled.

Tests and independent review

• Nineteen deterministic real SQLite-pool scenarios hold the only connection, confirm the request reached SQL, then require an unrelated store writer to complete before SQL is released. Includes packet/hash/observation detail, lists, expanded/grouped views, node filters, cold/expired region/area/node caches, health, subpaths and channels.
• Additional tests cover invalidation while SQL runs, immutable responses during migration/observation append, zero/negative transmission IDs, and captured distance metadata surviving invalidation.
• Baseline detail/cold-region/cold-node cases fail the writer-progress assertion. The candidate passes.
• An AST callgraph guard rejects direct/transitive SQL reached under store locks, with explicit startup exceptions. A separate assertion constrains deferred enrichment to the exact unlock-then-apply closure rather than allowing an unrestricted exception.
• Full server suite: pass (102.720 s).
• Focused authored -race: pass (15.034 s); vet, gofmt and diff checks clean.
• Independent reviewer, not the author: APPROVE, no findings on tree 10adb2bf24a9941e50aca122203799f3caef6b34. Its focused -race execution passes (2.909 s), including all nineteen convoy cases and the invariants above.
• Full npm suite: 257/257 test files pass, with existing platform/tool skips explicitly reported.
• General Chromium regression on an isolated local fixture: 154 passed, 3 fixture-dependent skips matching baseline, no failures.
• The unchanged ingestor baseline passes its complete suite (114.377 s). No staging, demo or production contact.

GitHub/Linux CI is separate evidence and will be reported after the new PR run.

Measured trade-off

Go 1.26.3, macOS arm64 / Apple M5; 30,000 in-memory transmissions. The contention benchmark deliberately holds the SQL pool for 20 ms and uses an actual ingest-style store writer. It is a controlled local scenario, not a deployment latency measurement.

Measurement	Baseline	Candidate
Writer p50 during the controlled SQL hold	~21,090 µs	~4.5 µs
Writer p95	~21,138 µs	~19.4 µs
Uncontended detail median	975 ns	1,110 ns
Uncontended detail allocation	2,000 B / 14 allocations	2,232 B / 19 allocations

The immutable snapshot adds a small uncontended request cost; this is not hidden as an across-the-board speedup. Per-request work stays bounded by the selected result/observations and existing cache bounds.

Lock and staleness contract

Cold/expired SQL-backed caches still refresh synchronously outside the packet-store lock. There is no new stale/temporarily-unfiltered response mode. A request owns its captured metadata snapshot even if invalidation happens later; invalidated SQL cannot republish that snapshot into the cache. Resolved-path enrichment can reflect a newer database state than the store snapshot, as before, but never rereads mutable transmission fields after unlocking.

The legacy startup Load publication section still holds its exclusive store lock around SQL before HTTP serving. Startup/chunk loader methods are explicit guard exceptions because of their nested publication lifecycle; this PR does not claim every SQL call in the entire program is lock-free. Existing outside-lock chunk reads and bounded merge sections are preserved.

EvictStale did not actually perform the SQL described by its old comment on this baseline; its behavior is preserved and the comment corrected. RunEviction keeps its existing outside-lock evidence batch. No broader eviction redesign is included.

Attribution

Historical design reference, adapted rather than cherry-picked wholesale:

Repository: Kpa-clawbot/CoreScope
PR: 2147
Head: 4094d215ce2463a015a53998fdc03c136acc0f10
Merge: f50312767fcb5addf636a2a4d44cc1f40628986d

[dborup]

Review — CS-Coordinator PR#495 — head 8ba8a3a

Verdict: APPROVE with one condition before merge: a staging soak (see below).

Reviewed by code reading plus tests on the merged tree: current master cbbb6057, which includes #494, #498 and #493, merged with this head. The PR's own CI ran on base b6852ba0 (#475), and #494 has since changed store.go too.

What I checked

• Merged tree builds and passes. go vet is clean. The full cmd/server suite passes (66 s). -race on 412|470|165|Coverage|NodeAnalytics|RoutePatterns410|ConcurrentIngest passes, which covers this PR's convoy and AST-guard tests, the server: fatal concurrent map write / heap corruption under concurrent load on linux/amd64 (found in #469) #470/Server: data races between ingest and packet/path readers #475 race regressions and the fix(nodes): calculate analytics over available retained history #494 coverage tests.
• It is consistent with Server: data races between ingest and packet/path readers #475's snapshot pattern. Resolved-path inputs are copied under s.mu into request-owned resolvedPathObservation{id, pathJSON} values, and applyResolvedPathJobs runs only after RUnlock and mutates only the request's own map. bestResolvedPathAt keeps the previous fast-path choice (longest path_json first via LRU/SQL) and fallback choice (longest path that has a stored resolved_path), and writes the LRU under lruMu.
• The deferred enrichment is correct. The defer func(){ s.mu.RUnlock(); s.applyResolvedPathJobs(jobs) }() pattern mutates the returned, request-owned map before control returns to the caller. The shared cache is not touched.
• preparePacketFilters does its SQL and cache work before the store lock, and filterPacketsPrepared runs under it. The grouped page and sort-key snapshot from Server: data races between ingest and packet/path readers #475 are unchanged.
• The AST call-graph guard against SQL under store locks, with explicit startup exceptions, is a good way to keep this invariant from regressing.
• The measured uncontended cost is small: detail 975 → 1,110 ns, +5 allocations. The gain under SQL contention is large: writer p95 dropped from ~21 ms to ~19 µs.

Condition before merge

This is a +1,070-line refactor of hot lock paths, after the race history in #470/#475. Before merging, I recommend a few hours on staging under live ingest, as we did for #494, deploying master + this PR. Check:

• no restarts and no panic, fatal or concurrent map in the logs;
• packets, packet detail, node detail, paths and channels response times no worse than now;
• ingest keeps up (no ingest-buffer drops).

Nits

• The PR text says "GitHub/Linux CI ... will be reported after the new PR run". Please update it with the actual CI result for the head.

[dborup]

Staging soak passed (master cbbb6057 + this PR, about 105 min under live ingest):

• 0 restarts.
• 0 panic, fatal error, concurrent map or race reports.
• 0 SQLITE_BUSY.
• No ingest-buffer drops.
• Memory 1.55 / 2.73 GiB.

Latency was equal or better than before the deploy:

Endpoint	Before	After
grouped packets	0.39 s	0.31 s
packets	0.32 s	0.22 s
channels	2.85 s	0.89 s
stats	0.39 s	0.24 s

Packet detail took 0.24 s. Node paths took 0.56 s warm (2.0 s cold), against 0.92 s on production v0.2.5. The review condition is met; merging.