perf(store): evaluate bounded sharing of observer metadata strings #489

Assessment baseline: fork master b6852ba0876b02426b6200c1c79c7e0ebe3b9389, checked on 16 October 2026. Recheck master and competing work before implementation.

Problem and priority

Medium priority, measurement first: many in-memory observations repeat identical observer ID, name, IATA and direction strings. A bounded intern pool may reduce retained string memory without altering observation semantics.

Upstream reports lower retained heap on its own dataset. Do not assume the same saving or a lower peak RSS for our fork.

Fork adaptation

• Coordinate with local perf(store): avoid eager per-transmission observation dedup maps #411 and perf(store): evaluate on-demand subpath counts instead of permanent indexes #413 before editing store layout or accounting; avoid competing implementations of the same store optimization.
• Use bounded canonical storage with an explicit capacity policy. Clone newly retained strings so a small substring does not pin a large backing buffer. A full pool must preserve correctness, not reject observations.
• Cover every cold-load, chunked-load, merge and live-ingest constructor. Avoid adding intern-pool contention under the main store lock.
• Preserve historical observation metadata, updates/renames and ownership semantics: equal values may share immutable storage, but a metadata change must not mutate older observations.
• Distinguish logical per-observation charges from actual shared retained bytes. Keep trackedBytes, eviction budgets and reported memory meaningful; account for the pool's own bounded retention.
• No schema, API, deployment or workflow changes.
• Do not combine this with decoded-cache field removal: the fork still consumes signatureValid to reject known-invalid ADVERT evidence. That contract must remain intact.

Acceptance criteria

• Test all construction paths, concurrent calls, GC stability, pool saturation, metadata changes and oversized backing buffers; race tests pass.
• Tests establish that sharing survives GC and that saturation safely falls back to normal strings.
• Measure live heap, peak RSS, retained string bytes, startup time and steady-state ingest/reader contention on representative fork data.
• Verify eviction and memory accounting with many observations, unique metadata churn and pool saturation. Document any intentionally conservative accounting.

Source provenance

Repository: Kpa-clawbot/CoreScope
PR: 2183
Title: perf(store): share one copy of each observer string across observations
Head: ff7c8f2b9ac391a86ed1f0683c07df9f55c6b74f
Merge: e53c055d6eb5977f37ae07558240b542c1606400
Changed files:
cmd/server/chunked_load.go
cmd/server/intern.go
cmd/server/observer_strings_intern_test.go
cmd/server/store.go