Repository navigation
feat(scope-audit): observer region filter for both views - #2171
Conversation
8041424 to
949cc4a
Compare
…2142, API) (#2167) ## Summary Backend half of #2142 (the page is the stacked PR #2169). `GET /api/scope-audit` lists only repeaters that have answered a declared-regions request: 17 of almost 200 nodes on the instance the request came from. This adds a transport view, `GET /api/scope-audit?mode=transport&window=…[&scope=X]`, with one row per repeater **seen forwarding** in the window: - `transported`: the named region scopes it was seen carrying, each with packets and first/last seen (most packets first), plus unscoped, unmatched and ambiguous-hop counts; - `asked`, `declaredRegions`, `notObserved`, `configState`, `declaredAt` where a declared answer exists; `asked: false` with null declared fields where it does not ("not asked" is not "declares nothing"); - with `scope=X` (spelled `be`, `#be` or `BE`): `carriesScope` on every row and `carrying` / `notCarrying` counts, which is the rollout report the issue asks for. `mode=declared` (or no `mode`) is unchanged. An unknown `mode` is a 400. Partial fix for #2142: this PR provides the API; the page is #2169 and the observer region filter #2171. The issue should only be considered done once all three are in. ## How it follows the issue's notes - **The scan does not change.** It is the same single `ScopeAuditForwarding` pass, with every repeater and room in `nodes` (plus any declared target) as a target instead of only the declared ones. Same attribution rules: FLOOD-family route types only, `minForwarderHopHexLen` (no 1-byte hops), and a hop whose prefix matches more than one repeater is credited to none of them. - **Identity lookup with 1,000+ pubkeys:** `scopeAuditNodeIdentities` runs one `IN` query; exercised at 1,200 keys in the benchmark below. - **Row count and response size:** measured below. - **Cache:** the transport view is cached per window with the declared view's TTLs (30 s; 5 min for 7d), under the same singleflight group with a `transport|` key. The scope filter is applied to a copy of the cached result, so one scan serves every scope. (`scope=`, not `region=`: across the API `region=` is the observer IATA filter.) - **Wording:** "transported" is documented as *seen carrying*, not configured for; the declared side stays separate. ## Performance `BenchmarkComputeScopeTransport`: 1,200 repeaters (#1975 quotes an instance with 1,179), 30k flood transmissions in the last 24h on 4-hop paths of 3-byte prefixes, a third region-scoped: ``` BenchmarkComputeScopeTransport-12 5 219253727 ns/op 425.4 response-KB 1200 rows ``` 219 ms per compute, served from cache for 30 s (24h) or 5 min (7d); a 425 KB uncompressed response for 1,200 rows. This is new code with no "before"; the declared view's numbers are untouched (same scan, same cache). ### On real data (a staging copy of a production database and its live feed) | request | rows | result | time | |---|---|---|---| | declared view, 24h | 16 | (unchanged) | | | `mode=transport`, 24h | **385** repeaters seen forwarding, 16 of them with a declared answer | | 1.5 s cold | | `mode=transport&scope=<scope A>`, 7d | 664 | **51 carrying it, 613 not** | 14.4 s cold | | `mode=transport&scope=%23<SCOPE B>` (`#` and capitals), 7d, same window now cached | 664 | 237 carrying, 427 not | 0.35 s | The 7d cold compute is the same full-window hop scan the declared view already pays for 7d (its own TTL comment measures 16.7 s on a live-shaped database); like the declared view it is then cached for 5 minutes, and every scope filter is served from that one scan. - Commit `ceaeb56b` renamed the filter from `region=` to `scope=` after opening; the real-data figures above were taken before the rename and are otherwise the same code. ## Tests - Red commit `2a9b1286`: response types and a `mode=transport` stub that returns no rows, plus tests. Four fail on the stub: - repeaters that never answered are listed with `asked: false` and null declared fields (and the answered one with its declared list and `notObserved`); - `scope=fr` / `%23fr` / `FR` marks carriers and non-carriers and counts them; no scope means no `carriesScope` or counts; - an unknown mode is a 400; - the transport and declared views are cached separately (requesting one does not serve the other). - Also added, passing on the stub by construction: a repeater with no traffic in the window is not listed; the node blacklist applies. - Green commit `60c669f4`: the implementation, `docs/api-spec.md` (new `mode` and `scope` parameters, a Transport view section with the response shape), and the benchmark. - `make test`: 16 of 16 modules pass, including every existing scope-audit test. ## Review notes - Hidden-name and blacklist filtering use the same rules as the declared view. - `newTestStoreWithDB` now takes `testing.TB` so the benchmark can use it (callers unchanged). - No new `map[string]interface{}`; the response is a named struct. - No config added. ## SonarLint SonarQube Community rules (the SonarLint engine) were run on the files this PR touches, and only issues on lines it adds are listed. - **Fixed** in `060668b6`: S3776 (cognitive complexity 41 > 15) in the new `computeScopeTransport`. It is split into target, visibility, row and sort helpers; the response is the same and the transport tests are unchanged. ## Checked against the repo rules - Rebased onto `master` (`673442cc`). Each red commit was re-run and still fails on its assertions; each green commit passes (TDD.md). - `make test`: all 16 modules pass. Frontend suites: all pass except `test-issue-1956-release-routing.js`, which needs CI's `GITHUB_REF_NAME` and fails the same way on `master` locally (#2160 addresses it). - No private data: test fixtures and examples use the codes the existing tests use (SJC/SFO/LAX, `#be`-style scopes). - No new `map[string]interface{}`, no hard-coded colours. `git add` was explicit, and there is one logical change per commit. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…e) (#2169) > **Requires #2167 to be merged first.** This page calls the transport view API that #2167 adds (`/api/scope-audit?mode=transport`), so it cannot be split from it; until #2167 merges this PR also shows its commits. ## Summary Page half of #2142. **Stacked on #2167** (the API); review that first. #2171 adds the observer region filter on top. Partial fix for #2142 until all three are in. Adds a **Declared / Transport** toggle to `#/scope-audit`. The transport view calls `?mode=transport` and lists every repeater seen forwarding in the window: - **Seen carrying:** one chip per region scope with its packet count (first/last seen in the tooltip); - **Declared:** the repeater's declared regions coloured by whether they were seen carried (the declared view's own chips), or *not asked* when no declared-regions answer exists. Never asked is worded differently from declaring nothing; - **Other traffic:** unscoped, unmatched and ambiguous-hop counts. A **Region** box (suggestions from the scopes seen) adds a *Carries X* column, lists the repeaters not yet carrying it first, and summarises carriers against active non-carriers: the rollout report the issue asks for. Window, view and region are deep-linked (`#/scope-audit?window=7d&mode=transport&scope=be`) and survive a reload. The page says plainly that *transported* means seen carrying, not configured, and that over a short window a quiet region looks the same as a missing one. The intro text changes with the view. ## Screenshot The transport view with scope `be` typed in: every repeater seen forwarding in the last 24 h, whether it carries `be` (non-carriers first), the scopes it was seen carrying with counts, and its declared regions or *not asked*. Taken against `test-fixtures/e2e-fixture.db` with fictional `be` / `be-van` / `nl` scopes.  ## Browser validation (staging, real data) Headless Chromium against a staging copy of a production database with its live feed: - `#/scope-audit?window=24h`: the declared view, 16 rows, as before. - Clicking **Transport**: 385 rows; the hash gains `mode=transport`. - Typing a scope as `#XX-YY` in capitals: normalised to `xx-yy`; the hash gains it (now `scope=xx-yy`); summary "385 repeaters seen forwarding in the last 24h · 16 with a declared-regions answer · 51 carried xx-yy · 334 active but not seen carrying it"; the first rows are the non-carriers. - Reloading that hash restores view and region. Switching back to Declared hides the Region box and restores the declared intro. No page errors. - Checked at 1280 px and 390 px wide. Two layout bugs found this way were fixed before opening: the region box inherited `width: 100%` from `.nodes-search`, and `display: flex` overrode the `hidden` attribute on the region bar. ## Tests `tests/unit/test-frontend-helpers.js` gains 10 cases on rendered markup and state, using the page's exported internals like the existing scope-audit tests: hash and API paths (and that a region never leaks into the declared view's link), region normalisation (`#`, capitals, empty), *not asked* vs declared chips (with the `*` wildcard), carried-scope chips with counts and the filtered region marked, the carries column only with a region, HTML escaping of names, the summary counts, region suggestions, and the per-view intro. 768 passing in that file. Net-new UI surface, so per TDD.md the tests land in this PR rather than as a preceding red commit. Frontend suites on this branch: 193 of 194; the one failure, `test-issue-1956-release-routing.js`, needs CI's `GITHUB_REF_NAME` and fails identically on `master` locally (#2149 and #2160 address that). ## Review notes - Colours only through existing CSS variables (no hex literals); the filtered region is marked with the link colour, not a new status colour. - The window note's wording is in the same spirit as the declared view's `windowHonestyNote`. - No API calls per row: one request per view/window/region, cached client-side for 30 s like the declared view. ## SonarLint SonarQube Community rules (the SonarLint engine) were run on the files this PR touches, and only issues on lines it adds are listed. - **Fixed** in `df53e770`: - one `modeQuery` helper for the page hash and the API path (S3358; the two copies were identical); - the carries-scope cell built before the row (S3358); - `localeCompare` for the region suggestions (S2871), `for…of` and `.dataset` (S4138, S7761), optional chaining (S6582); - object spread in the helper tests (S6661). - Optional chaining and spread are already used throughout `public/`. No behaviour change. ## Checked against the repo rules - Rebased onto `master` (`673442cc`). Each red commit was re-run and still fails on its assertions; each green commit passes (TDD.md). - `make test`: all 16 modules pass. Frontend suites: all pass except `test-issue-1956-release-routing.js`, which needs CI's `GITHUB_REF_NAME` and fails the same way on `master` locally (#2160 addresses it). - No private data: test fixtures and examples use the codes the existing tests use (SJC/SFO/LAX, `#be`-style scopes). - No new `map[string]interface{}`, no hard-coded colours. `git add` was explicit, and there is one logical change per commit. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
#2167 and #2169 are merged (squashed as 6969063 and deb55dc), together with #2170 (352a0cc). Because of the squash, the commits this branch still carries now conflict with master in The re-run of the failed jobs was green on all three E2E shards, so the One follow-up I'd like after this merges, under AGENTS.md's "no unbounded data structures". It doesn't block this PR:
Failure scenario: an unauthenticated client sending Could you send a follow-up that canonicalises the codes (sort, de-duplicate, drop unknown codes) and bounds the caches (evict expired entries, or cap the size)? |
?region=<IATA,...> should count only forwarding heard by observers in those regions, echo the normalised codes, treat All as no filter, keep cache entries per region, and match nothing when no observer is there. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H
?region=<IATA,...> resolves the codes to their observers first and adds the observation's own observer column (observer_idx on v3, observer_id on v2) to the hop scan, so the 3.5M-row scan never joins observers. Codes with no observer match nothing. Responses echo the normalised codes; the declared and transport caches and singleflight keys carry the region. Declared-region verification stays unfiltered: it corroborates a declaration, not where a hop was heard. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H
The shared region filter sits with the view and window controls and applies to both views: the API is asked with ?region= (heard by observers there), and a change reloads the view. The transport view's scope box is relabelled Scope, matching its ?scope= parameter, so the page no longer has two controls called Region. The filter's menu opens leftwards where it is the right-most control, and rightwards on a phone where it wraps to the start of a line. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H
On a live-shaped production database the planner drives the region clause from idx_observations_observer_idx: one region over 24h read every observation its 13 observers ever made, 13.44s, against 0.25s from the window. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H
+observer column in the region clause, the same unary-plus treatment as the scope-stats query: SQLite then starts from the window and filters by observer per row. On a production-sized database, one region over 24h: 13.44s -> 0.25s, same 102,114 hop rows (unfiltered: 0.35s). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H
949cc4a to
6cd10a8
Compare
|
Rebased onto
|
…-render (#2180) ## What `test-issue-1648-m2-icons-e2e.js` now asserts the configured-scope row only after the page's `theme-refresh` re-render has fired and the detail has rendered again. ## Why On 2026-10-10 this assertion failed CI on four PRs (#2169, #2171, #2154, #2164), each time at the first live-map case: `confirmed scope row must be visible, including an empty scope`. A re-run of the failed jobs passed every time. The code path: - `customize-v2.js:814` dispatches `theme-changed` once `/api/config/theme` resolves, on every page load. - `app.js:1309-1315` turns that into `theme-refresh` 300 ms later. - `live.js:4713` re-runs `showNodeDetail(activeNodeDetailKey)` on `theme-refresh`, and `showNodeDetail` first sets `#nodeDetailContent` to "Loading…" (`live.js:2598`). This is the only caller besides the marker click (`live.js:3090`). - The test waited for `#nodeDetailContent table` and then called `row.isVisible()`, which does not wait. A refresh landing between the two fails the check. The same race could let the null-scope case pass on the "Loading…" panel, because `row.count() === 0` holds there too. The fix uses the same `addInitScript` flag pattern as `test-e2e-playwright.js:83`. ## Testing - Locally against a server on the E2E fixture: 5 of 5 runs pass, 27 of 27 checks each. - **Not reproduced locally.** The unchanged test also passed every local run: 5 plain, 3 with `/api/config/theme` delayed by 1 s, 5 with 8x CPU throttling. So the reasoning above comes from the code path and the four CI failures, not from a local red-then-green. - Test-only change, no product code touched. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Summary
Builds on the transport view from #2167 and #2169 (both merged); rebased onto
master(352a0ccd), so the diff is this PR's six commits only.The Scope Audit was the one analysis page without the region selector. This adds the shared region filter to it, for both the declared and the transport view.
?region=SFO,SJCmeans what it means on every other endpoint: count only forwarding heard by observers in those regions, not repeaters located there.GET /api/scope-audit, both modes):AND o.observer_idx IN (…)on v3, oro.observer_id IN (…)on v2. The 3.5M-row hop scan never joinsobservers.Allmeans no filter (sharednormalizeRegionCodes). Codes with no observer match nothing, instead of silently widening to every observer.region. The declared and transport caches and their singleflight keys now include the region, so each region set is cached separately with the existing TTLs.regionEvidence) is deliberately left unfiltered: it corroborates what a repeater declares from its own traffic, which doesn't depend on where a hop was heard.RegionFiltersits with the view and window controls; a change reloads the view, and the selection persists like every other page. The transport view's text box is relabelled Scope (it already sends?scope=since the rename in feat(scope-audit): transport view of every repeater seen forwarding (#2142, API) #2167), so the page no longer has two controls called Region.docs/api-spec.md(parameter, both response shapes) and the OpenAPI parameters for the route (mode,scopeandregionwere all missing).Partial fix for #2142 (its region filter for rollout tracking), together with #2167 and #2169.
Performance
The unfiltered path must not get slower: it's the same query with an empty clause.
BenchmarkComputeScopeTransport(from #2167, 1,200 repeaters), the PR it stacks on vs this branch,-count 8:Filtered requests, measured on staging (a copy of a production database): the first version drove
o.observer_idx IN (…)fromidx_observations_observer_idx, reading every observation the region's observers ever made. Fixed in80414242with a unary+(the same treatment as #2166), so the scan starts from thefirst_seenwindow and filters per row. Red test67e063b1asserts the query plan against a fixture with the real indexes.o.observer_idx IN (…)+o.observer_idx IN (…)first_seenwindowfirst_seenwindowCold end-to-end requests on staging after the fix: 24h declared, unfiltered 0.84 s · one region 0.48 s (was 16.7 s) · two regions 0.22 s · transport, one region 0.42 s · transport, six regions 0.75 s · 7d, one region 3.46 s. 24h transport, live data: 383 repeaters unfiltered, 204 heard by one country's observers.
Tests
629e4f36(Go): extends the audit fixture with anobserverstable andobservations.observer_idx. Covers the declared view ignoring a scope heard only by another region's observer, the transport view listing only repeaters heard in the region,region=All, per-region caching, a region with no observers, and the echoed codes. Five fail on assertions;region=Allpasses because the stub ignores the filter, which is the behaviour it pins.6f1b1e1c: the implementation, plus a v2-schema case (observer_id), so both branches of the clause are covered.b4933d68/ green08ecc172(frontend):apiPathcarries the selection in both views and adds nothing when none is selected. The region control on the page is a net-new UI surface.go test(cmd/server) passes. Frontend suites: 193 of 194. The one failure,test-issue-1956-release-routing.js, needs CI'sGITHUB_REF_NAMEand fails the same way onmasterlocally.Screenshot
The same transport view with the shared region filter set to SJC: only forwarding heard by San Jose observers counts, so 7 repeaters are listed instead of 44, with their carried scopes counted from those observers alone. Taken against
test-fixtures/e2e-fixture.dbwith fictional scopes.Browser validation
The branch's server was run locally with 14 configured regions (dropdown layout), checked in headless Chromium at 1280, 900, 700 and 390 px (touch):
®ion=SJCFound and fixed this way: the filter is the right-most control on desktop, and the menu (which opens rightwards) ran 66 px off the page. It now opens leftwards above 640 px. On a phone the controls wrap, the filter starts a line, and the default applies.
SonarLint
SonarQube Community rules (the SonarLint engine) were run on the files this PR touches, and only issues on lines it adds are listed.
ScopeAuditForwardingHeardIn: complexity 55. The function it generalises,ScopeAuditForwarding, is 54 onmaster; the region clause adds one point.computeScopeAudit: complexity 65, unchanged frommaster.Checked against the repo rules
master(673442cc). Each red commit was re-run and still fails on its assertions; each green commit passes (TDD.md).make test: all 16 modules pass. Frontend suites: all pass excepttest-issue-1956-release-routing.js, which needs CI'sGITHUB_REF_NAMEand fails the same way onmasterlocally (test: make local validation independent of runner environment #2160 addresses it).#be-style scopes).map[string]interface{}, no hard-coded colours.git addwas explicit, and there is one logical change per commit.🤖 Generated with Claude Code
https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H