Skip to content

feat(scope-audit): observer region filter for both views - #2171

Merged
efiten merged 6 commits into
Kpa-clawbot:masterfrom
A13xB0:feat/scope-audit-observer-region
Oct 11, 2026
Merged

efiten merged 6 commits into
Kpa-clawbot:masterfrom
A13xB0:feat/scope-audit-observer-region

Conversation

@A13xB0

@A13xB0 A13xB0 commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Builds on the transport view from #2167 and #2169 (both merged); rebased onto master (352a0ccd), so the diff is this PR's six commits only.

The Scope Audit was the one analysis page without the region selector. This adds the shared region filter to it, for both the declared and the transport view. ?region=SFO,SJC means what it means on every other endpoint: count only forwarding heard by observers in those regions, not repeaters located there.

  • API (GET /api/scope-audit, both modes):
    • The region codes are resolved to their observers first (a handful of rows), then the audit's hop scan gets AND o.observer_idx IN (…) on v3, or o.observer_id IN (…) on v2. The 3.5M-row hop scan never joins observers.
    • All means no filter (shared normalizeRegionCodes). Codes with no observer match nothing, instead of silently widening to every observer.
    • Responses echo the normalised codes as region. The declared and transport caches and their singleflight keys now include the region, so each region set is cached separately with the existing TTLs.
    • Declared-region verification (regionEvidence) is deliberately left unfiltered: it corroborates what a repeater declares from its own traffic, which doesn't depend on where a hop was heard.
  • Page: the shared RegionFilter sits with the view and window controls; a change reloads the view, and the selection persists like every other page. The transport view's text box is relabelled Scope (it already sends ?scope= since the rename in feat(scope-audit): transport view of every repeater seen forwarding (#2142, API) #2167), so the page no longer has two controls called Region.
  • Docs: docs/api-spec.md (parameter, both response shapes) and the OpenAPI parameters for the route (mode, scope and region were all missing).

Partial fix for #2142 (its region filter for rollout tracking), together with #2167 and #2169.

Performance

The unfiltered path must not get slower: it's the same query with an empty clause. BenchmarkComputeScopeTransport (from #2167, 1,200 repeaters), the PR it stacks on vs this branch, -count 8:

before after
sec/op 220.3 ms 218.0 ms ~ (p=0.065)
B/op 43.77 MiB 43.70 MiB ~
allocs/op 833.5k 833.5k ~

Filtered requests, measured on staging (a copy of a production database): the first version drove o.observer_idx IN (…) from idx_observations_observer_idx, reading every observation the region's observers ever made. Fixed in 80414242 with a unary + (the same treatment as #2166), so the scan starts from the first_seen window and filters per row. Red test 67e063b1 asserts the query plan against a fixture with the real indexes.

scan, one region over 24h (13 observers) plan time hop rows
o.observer_idx IN (…) observer index 13.44 s 102,114
+o.observer_idx IN (…) first_seen window 0.25 s 102,114
unfiltered first_seen window 0.35 s 225,983

Cold end-to-end requests on staging after the fix: 24h declared, unfiltered 0.84 s · one region 0.48 s (was 16.7 s) · two regions 0.22 s · transport, one region 0.42 s · transport, six regions 0.75 s · 7d, one region 3.46 s. 24h transport, live data: 383 repeaters unfiltered, 204 heard by one country's observers.

Tests

  • Red 629e4f36 (Go): extends the audit fixture with an observers table and observations.observer_idx. Covers the declared view ignoring a scope heard only by another region's observer, the transport view listing only repeaters heard in the region, region=All, per-region caching, a region with no observers, and the echoed codes. Five fail on assertions; region=All passes because the stub ignores the filter, which is the behaviour it pins.
  • Green 6f1b1e1c: the implementation, plus a v2-schema case (observer_id), so both branches of the clause are covered.
  • Red b4933d68 / green 08ecc172 (frontend): apiPath carries the selection in both views and adds nothing when none is selected. The region control on the page is a net-new UI surface.
  • go test (cmd/server) passes. Frontend suites: 193 of 194. The one failure, test-issue-1956-release-routing.js, needs CI's GITHUB_REF_NAME and fails the same way on master locally.

Screenshot

The same transport view with the shared region filter set to SJC: only forwarding heard by San Jose observers counts, so 7 repeaters are listed instead of 44, with their carried scopes counted from those observers alone. Taken against test-fixtures/e2e-fixture.db with fictional scopes.

Scope Audit with the observer region filter on SJC

Browser validation

The branch's server was run locally with 14 configured regions (dropdown layout), checked in headless Chromium at 1280, 900, 700 and 390 px (touch):

check 1280 900 700 390 touch
ticking SJC re-requests with &region=SJC yes yes yes yes
menu fully on screen yes yes yes yes
page wider than viewport no no no no
page errors none none none none

Found and fixed this way: the filter is the right-most control on desktop, and the menu (which opens rightwards) ran 66 px off the page. It now opens leftwards above 640 px. On a phone the controls wrap, the filter starts a line, and the default applies.

SonarLint

SonarQube Community rules (the SonarLint engine) were run on the files this PR touches, and only issues on lines it adds are listed.

  • S3776 on ScopeAuditForwardingHeardIn: complexity 55. The function it generalises, ScopeAuditForwarding, is 54 on master; the region clause adds one point.
  • S3776 on computeScopeAudit: complexity 65, unchanged from master.
  • Both pre-date this PR. Splitting them is better as a refactor PR of its own than mixed into a feature.
  • The page code's findings were fixed in the transport-view PR it stacks on.

Checked against the repo rules

  • Rebased onto master (673442cc). Each red commit was re-run and still fails on its assertions; each green commit passes (TDD.md).
  • make test: all 16 modules pass. Frontend suites: all pass except test-issue-1956-release-routing.js, which needs CI's GITHUB_REF_NAME and fails the same way on master locally (test: make local validation independent of runner environment #2160 addresses it).
  • No private data: test fixtures and examples use the codes the existing tests use (SJC/SFO/LAX, #be-style scopes).
  • No new map[string]interface{}, no hard-coded colours. git add was explicit, and there is one logical change per commit.

🤖 Generated with Claude Code

https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H

@A13xB0
A13xB0 force-pushed the feat/scope-audit-observer-region branch from 8041424 to 949cc4a Compare October 10, 2026 17:22
efiten pushed a commit that referenced this pull request Oct 10, 2026
…2142, API) (#2167)

## Summary

Backend half of #2142 (the page is the stacked PR #2169).

`GET /api/scope-audit` lists only repeaters that have answered a
declared-regions request: 17 of almost 200 nodes on the instance the
request came from. This adds a transport view, `GET
/api/scope-audit?mode=transport&window=…[&scope=X]`, with one row per
repeater **seen forwarding** in the window:

- `transported`: the named region scopes it was seen carrying, each with
packets and first/last seen (most packets first), plus unscoped,
unmatched and ambiguous-hop counts;
- `asked`, `declaredRegions`, `notObserved`, `configState`, `declaredAt`
where a declared answer exists; `asked: false` with null declared fields
where it does not ("not asked" is not "declares nothing");
- with `scope=X` (spelled `be`, `#be` or `BE`): `carriesScope` on every
row and `carrying` / `notCarrying` counts, which is the rollout report
the issue asks for.

`mode=declared` (or no `mode`) is unchanged. An unknown `mode` is a 400.

Partial fix for #2142: this PR provides the API; the page is #2169 and
the observer region filter #2171. The issue should only be considered
done once all three are in.

## How it follows the issue's notes

- **The scan does not change.** It is the same single
`ScopeAuditForwarding` pass, with every repeater and room in `nodes`
(plus any declared target) as a target instead of only the declared
ones. Same attribution rules: FLOOD-family route types only,
`minForwarderHopHexLen` (no 1-byte hops), and a hop whose prefix matches
more than one repeater is credited to none of them.
- **Identity lookup with 1,000+ pubkeys:** `scopeAuditNodeIdentities`
runs one `IN` query; exercised at 1,200 keys in the benchmark below.
- **Row count and response size:** measured below.
- **Cache:** the transport view is cached per window with the declared
view's TTLs (30 s; 5 min for 7d), under the same singleflight group with
a `transport|` key. The scope filter is applied to a copy of the cached
result, so one scan serves every scope. (`scope=`, not `region=`: across
the API `region=` is the observer IATA filter.)
- **Wording:** "transported" is documented as *seen carrying*, not
configured for; the declared side stays separate.

## Performance

`BenchmarkComputeScopeTransport`: 1,200 repeaters (#1975 quotes an
instance with 1,179), 30k flood transmissions in the last 24h on 4-hop
paths of 3-byte prefixes, a third region-scoped:

```
BenchmarkComputeScopeTransport-12   5   219253727 ns/op   425.4 response-KB   1200 rows
```

219 ms per compute, served from cache for 30 s (24h) or 5 min (7d); a
425 KB uncompressed response for 1,200 rows. This is new code with no
"before"; the declared view's numbers are untouched (same scan, same
cache).

### On real data (a staging copy of a production database and its live
feed)

| request | rows | result | time |
|---|---|---|---|
| declared view, 24h | 16 | (unchanged) | |
| `mode=transport`, 24h | **385** repeaters seen forwarding, 16 of them
with a declared answer | | 1.5 s cold |
| `mode=transport&scope=<scope A>`, 7d | 664 | **51 carrying it, 613
not** | 14.4 s cold |
| `mode=transport&scope=%23<SCOPE B>` (`#` and capitals), 7d, same
window now cached | 664 | 237 carrying, 427 not | 0.35 s |

The 7d cold compute is the same full-window hop scan the declared view
already pays for 7d (its own TTL comment measures 16.7 s on a
live-shaped database); like the declared view it is then cached for 5
minutes, and every scope filter is served from that one scan.
- Commit `ceaeb56b` renamed the filter from `region=` to `scope=` after
opening; the real-data figures above were taken before the rename and
are otherwise the same code.

## Tests

- Red commit `2a9b1286`: response types and a `mode=transport` stub that
returns no rows, plus tests. Four fail on the stub:
- repeaters that never answered are listed with `asked: false` and null
declared fields (and the answered one with its declared list and
`notObserved`);
- `scope=fr` / `%23fr` / `FR` marks carriers and non-carriers and counts
them; no scope means no `carriesScope` or counts;
  - an unknown mode is a 400;
- the transport and declared views are cached separately (requesting one
does not serve the other).
- Also added, passing on the stub by construction: a repeater with no
traffic in the window is not listed; the node blacklist applies.
- Green commit `60c669f4`: the implementation, `docs/api-spec.md` (new
`mode` and `scope` parameters, a Transport view section with the
response shape), and the benchmark.
- `make test`: 16 of 16 modules pass, including every existing
scope-audit test.

## Review notes

- Hidden-name and blacklist filtering use the same rules as the declared
view.
- `newTestStoreWithDB` now takes `testing.TB` so the benchmark can use
it (callers unchanged).
- No new `map[string]interface{}`; the response is a named struct.
- No config added.

## SonarLint

SonarQube Community rules (the SonarLint engine) were run on the files
this PR touches, and only issues on lines it adds are listed.

- **Fixed** in `060668b6`: S3776 (cognitive complexity 41 > 15) in the
new `computeScopeTransport`. It is split into target, visibility, row
and sort helpers; the response is the same and the transport tests are
unchanged.

## Checked against the repo rules

- Rebased onto `master` (`673442cc`). Each red commit was re-run and
still fails on its assertions; each green commit passes (TDD.md).
- `make test`: all 16 modules pass. Frontend suites: all pass except
`test-issue-1956-release-routing.js`, which needs CI's `GITHUB_REF_NAME`
and fails the same way on `master` locally (#2160 addresses it).
- No private data: test fixtures and examples use the codes the existing
tests use (SJC/SFO/LAX, `#be`-style scopes).
- No new `map[string]interface{}`, no hard-coded colours. `git add` was
explicit, and there is one logical change per commit.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
efiten pushed a commit that referenced this pull request Oct 10, 2026
…e) (#2169)

> **Requires #2167 to be merged first.** This page calls the transport
view API that #2167 adds (`/api/scope-audit?mode=transport`), so it
cannot be split from it; until #2167 merges this PR also shows its
commits.

## Summary

Page half of #2142. **Stacked on #2167** (the API); review that first.
#2171 adds the observer region filter on top. Partial fix for #2142
until all three are in.

Adds a **Declared / Transport** toggle to `#/scope-audit`. The transport
view calls `?mode=transport` and lists every repeater seen forwarding in
the window:

- **Seen carrying:** one chip per region scope with its packet count
(first/last seen in the tooltip);
- **Declared:** the repeater's declared regions coloured by whether they
were seen carried (the declared view's own chips), or *not asked* when
no declared-regions answer exists. Never asked is worded differently
from declaring nothing;
- **Other traffic:** unscoped, unmatched and ambiguous-hop counts.

A **Region** box (suggestions from the scopes seen) adds a *Carries X*
column, lists the repeaters not yet carrying it first, and summarises
carriers against active non-carriers: the rollout report the issue asks
for. Window, view and region are deep-linked
(`#/scope-audit?window=7d&mode=transport&scope=be`) and survive a
reload. The page says plainly that *transported* means seen carrying,
not configured, and that over a short window a quiet region looks the
same as a missing one. The intro text changes with the view.

## Screenshot

The transport view with scope `be` typed in: every repeater seen
forwarding in the last 24 h, whether it carries `be` (non-carriers
first), the scopes it was seen carrying with counts, and its declared
regions or *not asked*. Taken against `test-fixtures/e2e-fixture.db`
with fictional `be` / `be-van` / `nl` scopes.

![Scope Audit transport
view](https://raw.githubusercontent.com/A13xB0/CoreScope/pr-screenshots/2026-10/scope-audit-transport-view.png)

## Browser validation (staging, real data)

Headless Chromium against a staging copy of a production database with
its live feed:

- `#/scope-audit?window=24h`: the declared view, 16 rows, as before.
- Clicking **Transport**: 385 rows; the hash gains `mode=transport`.
- Typing a scope as `#XX-YY` in capitals: normalised to `xx-yy`; the
hash gains it (now `scope=xx-yy`); summary "385 repeaters seen
forwarding in the last 24h · 16 with a declared-regions answer · 51
carried xx-yy · 334 active but not seen carrying it"; the first rows are
the non-carriers.
- Reloading that hash restores view and region. Switching back to
Declared hides the Region box and restores the declared intro. No page
errors.
- Checked at 1280 px and 390 px wide.

Two layout bugs found this way were fixed before opening: the region box
inherited `width: 100%` from `.nodes-search`, and `display: flex`
overrode the `hidden` attribute on the region bar.

## Tests

`tests/unit/test-frontend-helpers.js` gains 10 cases on rendered markup
and state, using the page's exported internals like the existing
scope-audit tests: hash and API paths (and that a region never leaks
into the declared view's link), region normalisation (`#`, capitals,
empty), *not asked* vs declared chips (with the `*` wildcard),
carried-scope chips with counts and the filtered region marked, the
carries column only with a region, HTML escaping of names, the summary
counts, region suggestions, and the per-view intro. 768 passing in that
file.

Net-new UI surface, so per TDD.md the tests land in this PR rather than
as a preceding red commit. Frontend suites on this branch: 193 of 194;
the one failure, `test-issue-1956-release-routing.js`, needs CI's
`GITHUB_REF_NAME` and fails identically on `master` locally (#2149 and
#2160 address that).

## Review notes

- Colours only through existing CSS variables (no hex literals); the
filtered region is marked with the link colour, not a new status colour.
- The window note's wording is in the same spirit as the declared view's
`windowHonestyNote`.
- No API calls per row: one request per view/window/region, cached
client-side for 30 s like the declared view.

## SonarLint

SonarQube Community rules (the SonarLint engine) were run on the files
this PR touches, and only issues on lines it adds are listed.

- **Fixed** in `df53e770`:
- one `modeQuery` helper for the page hash and the API path (S3358; the
two copies were identical);
  - the carries-scope cell built before the row (S3358);
- `localeCompare` for the region suggestions (S2871), `for…of` and
`.dataset` (S4138, S7761), optional chaining (S6582);
  - object spread in the helper tests (S6661).
- Optional chaining and spread are already used throughout `public/`. No
behaviour change.

## Checked against the repo rules

- Rebased onto `master` (`673442cc`). Each red commit was re-run and
still fails on its assertions; each green commit passes (TDD.md).
- `make test`: all 16 modules pass. Frontend suites: all pass except
`test-issue-1956-release-routing.js`, which needs CI's `GITHUB_REF_NAME`
and fails the same way on `master` locally (#2160 addresses it).
- No private data: test fixtures and examples use the codes the existing
tests use (SJC/SFO/LAX, `#be`-style scopes).
- No new `map[string]interface{}`, no hard-coded colours. `git add` was
explicit, and there is one logical change per commit.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@efiten

efiten commented Oct 10, 2026

Copy link
Copy Markdown
Collaborator

#2167 and #2169 are merged (squashed as 6969063 and deb55dc), together with #2170 (352a0cc). Because of the squash, the commits this branch still carries now conflict with master in docs/api-spec.md, public/scope-audit.css, public/scope-audit.js and tests/unit/test-frontend-helpers.js. Could you rebase onto master so that only the region-filter commits are left? I'll merge it once CI is green.

The re-run of the failed jobs was green on all three E2E shards, so the test-issue-1648-m2-icons-e2e.js failure was the race you described. The cause: theme-refresh re-runs showNodeDetail (live.js) and briefly swaps #nodeDetailContent to "Loading...", and the test calls row.isVisible() right after waiting for the table. A row.waitFor({ state: 'visible' }) would close it; happy to take that PR.

One follow-up I'd like after this merges, under AGENTS.md's "no unbounded data structures". It doesn't block this PR:

scopeAuditCache and scopeTransportCache are now keyed by window + "|" + region, where region is the raw normalizeRegionCodes output. The codes are not sorted, de-duplicated or checked against known regions, and no entry is ever removed. Before this PR each cache had at most 3 keys.

Failure scenario: an unauthenticated client sending region=X1, region=X2, ... gets a new permanent entry per value, and SFO,SJC vs SJC,SFO are two entries. Each cold value pays a full-window meta scan, plus a hop scan when the codes are valid, which takes seconds at 7d.

Could you send a follow-up that canonicalises the codes (sort, de-duplicate, drop unknown codes) and bounds the caches (evict expired entries, or cap the size)?

A13xB0 and others added 6 commits October 10, 2026 23:48
?region=<IATA,...> should count only forwarding heard by observers in
those regions, echo the normalised codes, treat All as no filter, keep
cache entries per region, and match nothing when no observer is there.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H
?region=<IATA,...> resolves the codes to their observers first and adds
the observation's own observer column (observer_idx on v3, observer_id
on v2) to the hop scan, so the 3.5M-row scan never joins observers.
Codes with no observer match nothing. Responses echo the normalised
codes; the declared and transport caches and singleflight keys carry
the region. Declared-region verification stays unfiltered: it
corroborates a declaration, not where a hop was heard.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H
The shared region filter sits with the view and window controls and
applies to both views: the API is asked with ?region= (heard by
observers there), and a change reloads the view. The transport view's
scope box is relabelled Scope, matching its ?scope= parameter, so the
page no longer has two controls called Region. The filter's menu opens
leftwards where it is the right-most control, and rightwards on a phone
where it wraps to the start of a line.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H
On a live-shaped production database the planner drives the region clause
from idx_observations_observer_idx: one region over 24h read every observation
its 13 observers ever made, 13.44s, against 0.25s from the window.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H
+observer column in the region clause, the same unary-plus treatment as
the scope-stats query: SQLite then starts from the window and filters by
observer per row. On a production-sized database, one region over 24h: 13.44s -> 0.25s, same 102,114
hop rows (unfiltered: 0.35s).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H
@A13xB0
A13xB0 force-pushed the feat/scope-audit-observer-region branch from 949cc4a to 6cd10a8 Compare October 10, 2026 22:49
@A13xB0

A13xB0 commented Oct 10, 2026

Copy link
Copy Markdown
Contributor Author

Rebased onto master (352a0ccd), now that #2167 and #2169 are merged. The branch is down to this PR's own six commits, and the merge conflict is gone.

efiten added a commit that referenced this pull request Oct 11, 2026
…-render (#2180)

## What

`test-issue-1648-m2-icons-e2e.js` now asserts the configured-scope row
only after the page's `theme-refresh` re-render has fired and the detail
has rendered again.

## Why

On 2026-10-10 this assertion failed CI on four PRs (#2169, #2171, #2154,
#2164), each time at the first live-map case: `confirmed scope row must
be visible, including an empty scope`. A re-run of the failed jobs
passed every time.

The code path:
- `customize-v2.js:814` dispatches `theme-changed` once
`/api/config/theme` resolves, on every page load.
- `app.js:1309-1315` turns that into `theme-refresh` 300 ms later.
- `live.js:4713` re-runs `showNodeDetail(activeNodeDetailKey)` on
`theme-refresh`, and `showNodeDetail` first sets `#nodeDetailContent` to
"Loading…" (`live.js:2598`). This is the only caller besides the marker
click (`live.js:3090`).
- The test waited for `#nodeDetailContent table` and then called
`row.isVisible()`, which does not wait. A refresh landing between the
two fails the check.

The same race could let the null-scope case pass on the "Loading…"
panel, because `row.count() === 0` holds there too.

The fix uses the same `addInitScript` flag pattern as
`test-e2e-playwright.js:83`.

## Testing

- Locally against a server on the E2E fixture: 5 of 5 runs pass, 27 of
27 checks each.
- **Not reproduced locally.** The unchanged test also passed every local
run: 5 plain, 3 with `/api/config/theme` delayed by 1 s, 5 with 8x CPU
throttling. So the reasoning above comes from the code path and the four
CI failures, not from a local red-then-green.
- Test-only change, no product code touched.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@efiten
efiten merged commit a4ee3a7 into Kpa-clawbot:master Oct 11, 2026
19 of 24 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants