[dborup]

Relates to #569

Problem

The GPS area filter (originator membership) was applied only by the in-memory packet path. buildTransmissionWhere handled region but not area, so whenever /api/packets (ordinary or groupByHash=true) fell back to SQL for history older than oldestLoaded — or ran store-less against the DB — it returned packets from outside the area, with a total that counted them too.

On a CI-prepared fixture with an SFBOX area and a 1.5 h in-memory window, since=now-6h&area=SFBOX returns total=512 with GRP_TXT/TRACE rows on master, and total=18 (ADVERTs only, equal to a direct SQLite count) with this change.

Plan / what changed

1. SQL predicate (cmd/server/db.go). PacketQuery gains AreaNodes map[string]bool, which is Area resolved to its member pubkeys. The DB layer has no config.Areas, so the caller resolves it. buildTransmissionWhere adds
   t.from_pubkey IN (SELECT value FROM json_each(?)). Both QueryPackets and QueryGroupedPackets use it before COUNT(*) and LIMIT/OFFSET, so totals and pages are filtered, not post-filtered.
2. Same membership in both paths (cmd/server/store.go). The store's ordinary and grouped SQL fallbacks set q.AreaNodes = s.resolveAreaNodes(q.Area). That is the same cached set the in-memory filter reads. The DB-to-set conversion is extracted into areaMemberSet, which resolveAreaNodes now calls.
3. Store-less handler path (cmd/server/routes.go). With no store, handlePackets resolves the area through the same areaMemberSet and cfg.Areas.

Semantics, matched to filterPacketsPrepared

Contract	In-memory	SQL (this PR)
Originator	decoded public_key/pubKey	t.from_pubkey, which the ingestor writes from the decoded ADVERT pubKey (Kpa-clawbot#1143)
No originator (non-ADVERT, ADVERT without a pubkey)	excluded	excluded (NULL/"" never match; "" is dropped from the bound set)
Unknown area key, or an area with no positioned node	resolveAreaNodes → nil → no filter	len(AreaNodes)==0 → no predicate
Nodes without GPS (NULL or 0,0)	never members	same set (GetNodePubkeysInArea)
Polygon / bounding box	geofilter.PassesFilter	same set; geometry is not duplicated in SQL
Area + region and the other filters	cumulative	cumulative (AND)

Known residual (documented, not changed): SQL reads the from_pubkey column, not decoded_json. An ADVERT row whose from_pubkey has not been backfilled yet, or a legacy row that carries only public_key, can match in memory but not in SQL. This is the same Kpa-clawbot#1143 contract that the SQL node and region membership filters already rely on: the ingestor writes from_pubkey at insert time and backfills it. A JSON fallback was measured and rejected (see Performance).

Performance

• One bound parameter regardless of how many nodes are in the area (a JSON array through json_each). There is no IN (?,?,…) growth, no per-row Go loop, and no per-row decoded_json parse.
• EXPLAIN QUERY PLAN for the area predicate: SEARCH t USING COVERING INDEX idx_transmissions_from_pubkey (from_pubkey=?) plus a bloom-filtered json_each list. Together with since/until, the planner drives off the observation-timestamp subquery and checks from_pubkey per candidate rowid. TestAreaFilterSQLPredicateBounded569 asserts the index and no SCAN t.
• A JSON fallback for unbackfilled ADVERTs (OR (from_pubkey IS NULL AND payload_type=4 AND json_extract(...))) turned the plan into a MULTI-INDEX OR that walks every ADVERT through idx_transmissions_payload_type. It was rejected for that reason.
• BenchmarkAreaSQL569: full COUNT + page, 30K ADVERTs from 2K nodes, 1K-member area, -benchtime 20x on a local dev machine:

Query	ordinary	grouped
no filter	0.32 ms	202 ms
since (the shape of every history fallback)	3.09 ms	11.5 ms
since + area	5.51 ms	10.9 ms
area alone, 15K matching rows	34.8 ms	60.6 ms

Every store fallback carries since or until, so the realistic cost is the since + area row. The area-alone row is the worst case for the store-less path.

Tests (written first, red on master)

• TestAreaFilterSQLMatchesMemory569 is an equivalence matrix. It compares the in-memory path against the store SQL fallback and the store-less DB path, for since, until, and since+until windows, ordinary and grouped. Areas: none, polygon, bbox, empty, unknown. Extra filters: none, two regions, type, observer. Pages: limit=50, limit=2, offset=2, and an offset past the end. It asserts identical hashes, order and total.
• TestAreaFilterSQLFallbackContract569 pins absolute results for all three paths. It covers polygon, bbox, area+region, unknown key, empty area and an empty result, so the matrix cannot pass by both sides drifting together.
• TestAreaFilterAcrossCacheBoundary569 adds SQL-only rows older than the real oldestLoaded (inside and outside the area). It checks since, since+until and until windows that cross the boundary, and pages through them with limit=3. It checks totals and the absence of duplicates or outside rows.
• TestAreaFilterSQLPredicateBounded569 checks one predicate and one argument for 5,000 members, no decoded_json, the idx_transmissions_from_pubkey plan, that "" is dropped, and that nil membership adds no predicate.

Red on unchanged master: the first three fail (32 SQL mismatches in the contract test; the in-memory control passes). The predicate test does not compile on master because AreaNodes does not exist.

Eight mutants were applied one at a time, and each is killed (details in the report comment).

Validation

• cd cmd/server && go test ./... passes, and so does cd cmd/ingestor && go test ./.... The new tests also pass under -race.
• sh test-all.sh, node test-frontend-helpers.js and test-e2e-playwright.js were run against a local Go server on a CI-prepared e2e-fixture.db (freshened, migrated, seeded).
• Real Packets UI: with a local server, Area: SF box selected and Last 6 hours chosen (beyond the 1.5 h in-memory window, so the request takes the SQL fallback), the page shows Latest Packets (18) with only ADVERTs. The request is /api/packets?since=…&limit=50000&area=SFBOX&groupByHash=true → total=18.

Scope

• cmd/server stays read-only: SELECTs only, no schema change, and no new index (idx_transmissions_from_pubkey already exists).
• No new map[string]interface{}, no frontend change, no new requests.

🤖 Generated with Claude Code

[dborup]

Rapport — CS-Minimax PR#579 #569 — head e4f2841

Status: Implemented. Draft PR, all CI jobs green on the first run, no reruns. Not merged or marked ready.

Evidence tags: [T] test run by me (local or CI), [A] analysis or reading of code/plans, [K] known or inherited fact, not re-verified here.

Requirement 1 — Area applied in the SQL predicate with in-memory semantics (ordinary and grouped)

Item	Test	Red before	Mutant (killed by)	Evidence
SQL predicate t.from_pubkey IN (SELECT value FROM json_each(?)) in buildTransmissionWhere, so it applies before COUNT and LIMIT/OFFSET	TestAreaFilterSQLMatchesMemory569, TestAreaFilterSQLFallbackContract569	yes: 32 SQL mismatches on master; in-memory control green	M1 predicate disabled (all 4 tests); M8 NOT IN (all 4)	[T]
Store ordinary fallback carries the same resolveAreaNodes set	same + TestAreaFilterAcrossCacheBoundary569	yes	M2 removed (3 tests)	[T]
Store grouped fallback carries the same set	same	yes	M3 removed (3 tests)	[T]
Store-less direct DB path resolves via the shared areaMemberSet	matrix + contract (database mode)	yes	M4 removed (2 tests)	[T]
Unknown or empty area applies no filter (in-memory nil set)	contract (NOPE, SEA → all 12 rows), predicate test	yes (SQL side)	M6 q.Area != "" gate (3 tests)	[T]
Empty originator is never a member (pk == "" in memory; "" is the backfill sentinel)	TestAreaFilterSQLPredicateBounded569	yes (does not compile on master: no AreaNodes)	M5 keep ""	[T]
Nodes without GPS (NULL, 0,0), polygon and bbox	contract: d/e excluded; BEL polygon and BOX bbox	yes	M1/M8	[T]
Combined area+region/type/observer and pagination (offset, past end, empty)	matrix (5 areas × 5 extras × 4 pages × 3 windows × 2 groupings)	yes	M1–M4	[T]

Requirement 2 — Fallback across the cache boundary

Item	Test	Red before	Mutant	Evidence
SQL-only rows older than the real oldestLoaded; since, since+until and until windows; paged limit=3; total 8/2; no outside rows or duplicates	TestAreaFilterAcrossCacheBoundary569	yes	M1, M2, M3, M8	[T]

Requirement 3 — Performance and bounds

Item	Test / measurement	Mutant	Evidence
One bound arg for 5,000 members; no decoded_json in the predicate	TestAreaFilterSQLPredicateBounded569	M7 json_extract predicate	[T]
Plan: SEARCH t USING COVERING INDEX idx_transmissions_from_pubkey, no SCAN t	asserted in the same test; also EXPLAIN on the fixture DB	M7	[T]
A JSON fallback for unbackfilled ADVERTs becomes a MULTI-INDEX OR over every ADVERT, so it was rejected	EXPLAIN on the fixture DB	n/a	[T]/[A]
BenchmarkAreaSQL569 (30K rows, 1K-member area): since 3.09 → since+area 5.51 ms ordinary, 11.5 → 10.9 ms grouped; area alone 34.8 / 60.6 ms	local benchmark	n/a	[T]

Invariants

Item	Evidence
cmd/server read-only: SELECT only, no schema change, no new index	[A]
No new map[string]interface{} (0 in the diff)	[T]
scripts/check-xss-sinks.sh --diff origin/master: clean (no frontend changes)	[T]
Fork guards unchanged: 9 in deploy.yml, 1 in release-fast-path.yml	[T]
No hardcoded colours (no CSS/JS changes)	[A]

Local validation

• cmd/server go test ./... ok, cmd/ingestor go test ./... ok. The new tests also pass under -race. [T]
• sh test-all.sh: 268/268 files. node test-frontend-helpers.js: 719 passed. [T]
• test-e2e-playwright.js against a local Go server on a CI-prepared e2e-fixture.db (freshened, CI seed SQL, migrated, the four seed files): 154/157 passed, 3 skipped by design (flaky-marked / Go perf). [T]
• Real Packets UI: local server, area SFBOX, in-memory window 1.5 h, Last 6 hours selected. The page shows Latest Packets (18), ADVERTs only. The API returns total=18 vs total=512 on a master build with the same DB and config. [T]

CI per job (head e4f2841)

Job	Result
Go Build & Test	pass
Go Server Tests 1/3, 2/3, 3/3	pass
Go Ingestor Tests	pass
Go Libraries & CLI	pass
JavaScript Unit Tests & Lint	pass
Script & Container Unit Checks	pass
Verify Go Test Assignments	pass
Playwright E2E 1/3, 2/3, 3/3	pass
Verify E2E Shards & Coverage	pass
Validate Docker Image / Build & Publish Docker Image	pass
Publish Badges & Summary, Release Artifacts, Deploy Staging	skipped (not applicable to PR)

No reruns were needed (#271 did not occur). [T]

Residuals

1. SQL reads t.from_pubkey; memory reads decoded public_key/pubKey. They can diverge only for an ADVERT whose from_pubkey is not yet backfilled, or a legacy row with only public_key. This is the same bug: pubkey attribution via LIKE-on-JSON is unsound — adversarial spoofing + same-name false positives + perf Kpa-clawbot/CoreScope#1143 contract that the SQL node and region-membership filters already use. The writer sets the column at insert time and backfills it. [A]/[K]
2. The store-less path resolves the area per request without the 30 s cache. That path is only used when no store exists. [A]
3. The benchmark is local; no staging measurement was taken (staging access is out of scope). [A]
4. The local UI validation used a scratch copy of the fixture with transmissions.last_seen refreshed, so the 1.5 h retention window loads a subset. The CI fixture is not modified. [T]

[adminopenclaw8-sketch]

Review — CS-Macmini PR#579 — head e4f2841

Dom: APPROVE med nits

Independent read-only review. Evidence tags: [T] test/measurement I ran myself, [A] my own code analysis, [K] inherited fact I did not re-verify.

Base: the branch sits directly on a38d6820 (head tree == git merge-tree result, one commit, expected commit author). Master moved to 81c798e4 mid-review (#576, an unrelated read-only DSN pragma change at db.go:206); the merge is still clean and I re-ran the suites on the new merged tree. [T]

Findings

#	Sev	Where	Finding
N1	nit	cmd/server/db.go:1078	len(q.AreaNodes) > 0 is a weaker gate than the in-memory areaNodes != nil. A non-nil empty map applies no SQL predicate, while filterPacketsPrepared would exclude every packet. q.AreaNodes != nil would be exactly equivalent (the empty JSON list matches nothing — confirmed: an all-false map already takes that path and returns 0 rows, matching memory). Unreachable today because areaMemberSet is the only producer and returns nil for an empty result, but AreaNodes is an exported field on an exported struct. My mutant MX4 (len(...)>0 → != nil) survives the whole suite, so no test pins the choice. [T]
N2	nit	cmd/server/db.go:1095	In areaMembersJSON, the ok && half of if ok && pk != "" is never exercised — areaMemberSet only ever stores true. Mutant MX3 (dropping ok &&) survives. Harmless defensive code; noting only that it is untested. [T]
N3	nit	cmd/server/routes.go:1855	handleNodes still inlines the same GetNodePubkeysInArea → map[string]bool loop that this PR extracted into areaMemberSet, so the extraction stops one caller short of its own stated rationale ("one membership contract"). Reuse there would be behaviour-preserving (that path gates on areaRequested, so nil and empty-non-nil already behave alike). Out of scope to change here. [A]
O1	obs, pre-existing	cmd/server/db.go:1070	A sibling of #569 survives in the same function: comma-separated region. In memory iataMatchesRegion splits on commas; SQL binds the raw string to obi.iata = ?. region=AAA,BBB → memory total=12, SQL fallback total=0. Confirmed red on unchanged master too, so it is not a regression from this PR. Worth a follow-up issue. It also means the PR body's "Area + region … cumulative" row holds only for a single region code. [T]
O2	obs	report residual 1	The documented from_pubkey-vs-decoded_json residual is real and correctly scoped. The Go decoder emits only pubKey (cmd/ingestor/decoder.go:121), and the writer guard is PayloadADVERT && PubKey != "" (cmd/ingestor/db.go:2793); the public_key branch in filterPacketsPrepared is a legacy Node-era shape with no current writer. The pre-existing test trigger test_from_pubkey_advert (db_test.go:143) mirrors the prod guard exactly, so the fixtures reproduce prod attribution rather than papering over it. [A]/[T]

No blocking finding. No behaviour change beyond the stated purpose: the diff adds one PacketQuery field, one SQL predicate, two store-fallback assignments, one store-less handler branch, and extracts areaMemberSet from resolveAreaNodes with identical nil semantics (err != nil || len(pks) == 0 → nil, same as the old err == nil && len(pks) > 0 guard). [A]

Answers to the review points

SQL predicate has the same semantics as the in-memory path. Yes, and the membership source is literally the same function. Both paths resolve through resolveAreaNodes → areaMemberSet → GetNodePubkeysInArea, so "which coordinates/node" is settled by one code path: NULL lat/lon skipped, (0,0) skipped, polygon and bbox both via geofilter.PassesFilter, no geometry duplicated in SQL. The only contract that is re-expressed rather than shared is the originator (t.from_pubkey vs decoded pubKey) — see O2. Packets with no originator are excluded on both sides (NULL/"" never match; "" is dropped from the bound set). Unknown key / geometry with no positioned node → nil → no filter on both sides, which is the fork's existing contract and is pinned absolutely by the contract test (NOPE and SEA both return all 12 rows). [A]/[T]

Ordinary and grouped, and total. Both QueryPackets and QueryGroupedPackets build w once from buildTransmissionWhere and use it for COUNT(*) and for the paged SELECT, so the predicate lands before COUNT and before LIMIT/OFFSET — not post-filtered. I verified this is load-bearing with my own mutant MX2 (count computed from a query with AreaNodes stripped): killed by all three behavioural tests. Live, across the cache boundary, totals equal the paged row count with no duplicates in every window and grouping I paged through. [T]

I reproduced the equivalence test myself. All four tests green on the merged tree; the three behavioural ones are red on unchanged master with exactly the reported shape — 32 contract mismatches, in-memory control green, and the predicate test does not compile on master (unknown field AreaNodes). [T]

Fallback across the cache boundary — reproduced independently at the HTTP layer. Two servers on the same migrated fixture, packetStore.hotStartupHours=1.5 so oldestLoaded=2026-10-25T02:50:24Z, then since=now-6h (crosses the boundary) with a bbox area of 33 positioned nodes:

	area total/rows	payload types returned	no-area total
master build	500 / 500	10 distinct types	500
this PR	19 / 19	ADVERT only	500

19 equals a direct SQLite count of in-area ADVERTs in that window. Same result ordinary and grouped. Paging with limit=5 over since, since+until and until-only windows: total == pages covered, no duplicates, no out-of-area rows. The until-only window is a second demonstration — master returns one payload_type=5 row (no originator), the PR correctly returns 0. [T]

Real Packets UI. Same two builds, area selected through the real AreaFilter control, 6 h window (beyond the 1.5 h in-memory window, so the request takes the SQL fallback): the PR renders Latest Packets (19) with 19 rows and only Advert badges; the master build renders Latest Packets (500) with six different type badges. Request on both: ?since=…&limit=50000&area=SFBOX&groupByHash=true. [T]

Perf — my own EXPLAIN QUERY PLAN. On the migrated fixture and on a 300 K-row / 4 K-pubkey database I built, with and without ANALYZE, plans identical and stable:

• area alone → SEARCH t USING COVERING INDEX idx_transmissions_from_pubkey (from_pubkey=?) + bloom-filtered json_each list;
• area + since (the shape of every history fallback) → SEARCH t USING INTEGER PRIMARY KEY (rowid=?), driven off the observation-timestamp list, from_pubkey checked per candidate;
• grouped + area + since → same drive, plus the pre-existing correlated subqueries and USE TEMP B-TREE FOR ORDER BY.

No SCAN t in any shape. One bound argument for 5 000 members, no decoded_json in the predicate, no per-row Go loop. Measured at 300 K rows / 2 000 members: area-only COUNT 5 ms; grouped since-only page 279 ms → since+area 101 ms, i.e. the predicate makes the realistic fallback faster by cutting the candidate set, which matches the direction of the author's benchmark. [T]

cmd/server read-only, no SQL under PacketStore.mu (#495). Production hunks add SELECT-only text; no schema change, no new index (idx_transmissions_from_pubkey already exists in dbschema/ingestor migration from_pubkey_v1). Every INSERT in the diff is inside _test.go. In both store fallbacks resolveAreaNodes is called after s.mu.RUnlock() and uses its own areaNodeMu, so no SQL is issued under PacketStore.mu. The new s.cfg.Areas read in handlePackets is unguarded, but handlePackets already dereferences s.cfg.ListLimits earlier and cfgMu is documented to guard only s.cfg.GeoFilter, so this is the existing pattern, not a new race class. [A]

Call-site coverage is complete. The only production callers of db.QueryPackets/db.QueryGroupedPackets are the two store fallbacks and the two s.store == nil branches in handlePackets; the new handler guard is exactly s.store == nil, matching the dispatch. buildPacketWhere (db.go:962) also omits Area, but it has no production callers. [A]

Other invariants. Each acceptance criterion in #569 maps to a test that is red before and green after (see below). No new map[string]interface{} in the diff (0 occurrences). No frontend change, so no hardcoded colours and scripts/check-xss-sinks.sh --diff origin/master is clean ("no public/**/*.{js,html} changes to scan"). Fork guards unchanged at 9 in deploy.yml and 1 in release-fast-path.yml. Commit trailer is Relates to #569 — no closing keyword. [T]/[A]

Tests I ran

On the merged tree against origin/master:

Suite	Result
cmd/server go test ./...	pass (72 s on a38d6820 merge; re-run pass on the 81c798e4 merge)
cmd/ingestor go test ./...	pass (110 s)
sh test-all.sh	268/268 files, 0 failed
node test-frontend-helpers.js	719 passed, 0 failed
test-e2e-playwright.js (local Go server, CI-prepared fixture: freshen → CI seed SQL → corescope-migrate)	154/157 passed, 3 skipped by design
11 packets/filter/region/area E2E + unit suites most exposed to /api/packets	all pass (test-filter-ux-e2e, 1122-packets-filter-ux, 1128-packets-layout, 242-type-exclusions, 147-packets-url-obs, 180-packets-url-modal, 1486-collapse, 1692-packets-init-parallel, 406-packets-live-channel, 531-region-url, 416-region-quick-picks, plus test-area-filter, test-area-nodes-map, test-nodes-geo-scope-filter, test-map-scope-filter)

Red-before on unchanged master: TestAreaFilterSQLMatchesMemory569, TestAreaFilterSQLFallbackContract569 and TestAreaFilterAcrossCacheBoundary569 all FAIL (32 contract mismatches; in-memory control green); TestAreaFilterSQLPredicateBounded569 does not compile. [T]

My own extension of the equivalence matrix. I added ~1 360 further comparisons pairing area with filter dimensions the PR's matrix does not cover — node (same from_pubkey column, 5 variants incl. a no-GPS node and a name lookup), channel, route, excludeTypes (single and multi), hash, multi-value observer, and type+region — across both windows, both groupings, five areas, two pages and both SQL modes. Every comparison matches the in-memory path. The one divergence this surfaced is O1 (comma-separated region), which is red on master too. [T]

CI per job, head e4f28413, run 40159169951 — I checked each job myself: all 18 jobs success except the three legitimately skipped (Release Artifacts, Deploy Staging, Publish Badges). Single run, no reruns; neither known flake (#256, #267) appeared. [T]

My own mutants

#	Mutation	Result
MX1	areaMemberSet returns an empty non-nil map instead of nil when len(pks)==0	killed (matrix + contract) — pins the nil-vs-empty "no filter" contract
MX2	COUNT(*) computed from a PacketQuery with AreaNodes stripped (i.e. post-filtered totals, the failure mode #569 forbids)	killed (all three behavioural tests)
MX3	drop the ok && guard in areaMembersJSON	survives — production-equivalent, see N2
MX4	len(q.AreaNodes) > 0 → q.AreaNodes != nil	survives — production-equivalent today, see N1
MX5	predicate becomes (… IN (…) OR t.from_pubkey IS NULL), leaking originator-less rows	killed (all three) — pins the "no originator → excluded" contract

What I did not verify

• No staging or production access, and no measurement on real production data volumes; my scale numbers come from a 300 K-row database I generated locally and from the committed fixture. [A]
• I did not re-derive the geometry contract itself (geofilter.PassesFilter); I only checked that both paths call the same resolver, so geometry is shared rather than duplicated. [A]
• I did not run the full E2E set shard-by-shard as CI does — I ran the main Playwright suite plus the 15 suites most exposed to /api/packets and relied on my own per-job check of CI run 40159169951 for the rest. [T]/[K]
• I did not exercise the store-less (s.store == nil) path against a live HTTP server; it is covered by the PR's database mode in the Go tests and by my extended matrix, both of which drive it through handlePackets. [T]
• The 30 s resolveAreaNodes cache TTL means a fallback issued within 30 s of a node moving in or out of an area can use a stale membership set — but identically on both paths, since they share the cache, so it is not a divergence. I did not test cache-expiry timing. [A]
• I did not re-run the author's BenchmarkAreaSQL569 numbers; I measured the same query shapes independently instead. [T]