Repository navigation
feat(scope-audit): transport view on the Scope Audit page (#2142, page) - #2169
Conversation
|
About the red E2E run. Shard 3 failed in I think this is a flake rather than this PR:
I can't re-run the jobs from a fork. Could a maintainer re-run them? If it's worth hardening, a |
…rding (Kpa-clawbot#2142) The scope audit only lists repeaters that have answered a declared-regions request. Kpa-clawbot#2142 asks for a transport view: every repeater seen forwarding in the window, the region scopes it carried, its declared regions where known ("not asked" otherwise), and a region filter for tracking a rollout. Adds ScopeTransportRow / ScopeTransportResponse, a ?mode=transport stub that returns no rows, and tests: repeaters that never answered are listed with asked=false and null declared fields, repeaters with no traffic are left out, region=X (spelled fr, #fr or FR) marks carriers and non-carriers and counts them, the blacklist applies, an unknown mode is a 400, and the two views are cached separately. Four fail on the stub. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H
…pa-clawbot#2142) GET /api/scope-audit?mode=transport lists one row per repeater the forwarding scan attributed traffic to in the window, whether or not it ever answered a declared-regions request: - transported: the named region scopes it was seen carrying, with packet counts and first/last seen (most packets first), plus unscoped, unmatched and ambiguous-hop counts; - asked / declaredRegions / notObserved / configState / declaredAt where a declared answer exists, null where it does not ("not asked"); - region=X (with or without '#', any case) adds carriesRegion per row and carrying / notCarrying counts: the rollout report. Same single ScopeAuditForwarding scan and attribution rules as the declared view (FLOOD-family routes, no 1-byte hops, ambiguous hops credited to nobody), with every repeater and room in nodes as a target. Cached per window with the declared view's TTLs, under the same singleflight group; the region filter is applied to a copy of the cached result. An unknown mode is a 400. Documented in docs/api-spec.md. BenchmarkComputeScopeTransport (1,200 repeaters, 30k flood transmissions in 24h, 4-hop paths): 219 ms per compute, 1,200 rows, 425 KB response. newTestStoreWithDB now takes testing.TB so benchmarks can use it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H
…egion= Across the API, region= is the observer IATA filter (/api/nodes, /api/packets, ...). The transport view used region= for a region SCOPE (be, be-van), which would collide with adding the observer filter to the scope audit. It is now ?scope=, echoed as "scope", with carriesScope on each row; carrying / notCarrying are unchanged. Tests and docs/api-spec.md updated to match. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H
…r, row and sort helpers SonarLint S3776 (cognitive complexity 41 > 15). Same response; the transport tests are unchanged and pass. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H
…t#2142) Adds a Declared / Transport toggle to #/scope-audit. The transport view calls ?mode=transport and lists every repeater seen forwarding in the window: - Seen carrying: one chip per region scope with its packet count; - Declared: the repeater's declared regions coloured by whether they were seen carried (the declared view's chips), or "not asked" when no declared-regions answer exists; - Other traffic: unscoped, unmatched and ambiguous-hop counts. A Region box (suggestions from the scopes seen) adds a Carries column, puts the repeaters not yet carrying the region first, and summarises carriers vs active non-carriers: the rollout report. The window, view and region are in the hash (#/scope-audit?window=7d&mode=transport®ion=be). The page says that "transported" means seen carrying, not configured, and that a quiet region looks like a missing one over a short window. Tests: 9 cases in test-frontend-helpers.js on rendered markup and hash state (net-new UI surface: tests land in the same PR, per TDD.md). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H
Follows the API rename: the page requests ?scope=, deep-links #/scope-audit?…&mode=transport&scope=be, and reads "scope" and "carriesScope" from the response. region= is left for the observer filter. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H
- one modeQuery helper for the page hash and the API path (S3358, and the two copies were identical) - the carries-scope cell is built before the row (S3358) - localeCompare for the region suggestions (S2871), for-of and .dataset (S4138, S7761), optional chaining (S6582) - object spread in the new helper tests (S6661) No behaviour change; the helper tests are unchanged in what they assert. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H
df53e77 to
1fe44fb
Compare
|
Rebased onto current Before pushing, I ran |
…2142, API) (#2167) ## Summary Backend half of #2142 (the page is the stacked PR #2169). `GET /api/scope-audit` lists only repeaters that have answered a declared-regions request: 17 of almost 200 nodes on the instance the request came from. This adds a transport view, `GET /api/scope-audit?mode=transport&window=…[&scope=X]`, with one row per repeater **seen forwarding** in the window: - `transported`: the named region scopes it was seen carrying, each with packets and first/last seen (most packets first), plus unscoped, unmatched and ambiguous-hop counts; - `asked`, `declaredRegions`, `notObserved`, `configState`, `declaredAt` where a declared answer exists; `asked: false` with null declared fields where it does not ("not asked" is not "declares nothing"); - with `scope=X` (spelled `be`, `#be` or `BE`): `carriesScope` on every row and `carrying` / `notCarrying` counts, which is the rollout report the issue asks for. `mode=declared` (or no `mode`) is unchanged. An unknown `mode` is a 400. Partial fix for #2142: this PR provides the API; the page is #2169 and the observer region filter #2171. The issue should only be considered done once all three are in. ## How it follows the issue's notes - **The scan does not change.** It is the same single `ScopeAuditForwarding` pass, with every repeater and room in `nodes` (plus any declared target) as a target instead of only the declared ones. Same attribution rules: FLOOD-family route types only, `minForwarderHopHexLen` (no 1-byte hops), and a hop whose prefix matches more than one repeater is credited to none of them. - **Identity lookup with 1,000+ pubkeys:** `scopeAuditNodeIdentities` runs one `IN` query; exercised at 1,200 keys in the benchmark below. - **Row count and response size:** measured below. - **Cache:** the transport view is cached per window with the declared view's TTLs (30 s; 5 min for 7d), under the same singleflight group with a `transport|` key. The scope filter is applied to a copy of the cached result, so one scan serves every scope. (`scope=`, not `region=`: across the API `region=` is the observer IATA filter.) - **Wording:** "transported" is documented as *seen carrying*, not configured for; the declared side stays separate. ## Performance `BenchmarkComputeScopeTransport`: 1,200 repeaters (#1975 quotes an instance with 1,179), 30k flood transmissions in the last 24h on 4-hop paths of 3-byte prefixes, a third region-scoped: ``` BenchmarkComputeScopeTransport-12 5 219253727 ns/op 425.4 response-KB 1200 rows ``` 219 ms per compute, served from cache for 30 s (24h) or 5 min (7d); a 425 KB uncompressed response for 1,200 rows. This is new code with no "before"; the declared view's numbers are untouched (same scan, same cache). ### On real data (a staging copy of a production database and its live feed) | request | rows | result | time | |---|---|---|---| | declared view, 24h | 16 | (unchanged) | | | `mode=transport`, 24h | **385** repeaters seen forwarding, 16 of them with a declared answer | | 1.5 s cold | | `mode=transport&scope=<scope A>`, 7d | 664 | **51 carrying it, 613 not** | 14.4 s cold | | `mode=transport&scope=%23<SCOPE B>` (`#` and capitals), 7d, same window now cached | 664 | 237 carrying, 427 not | 0.35 s | The 7d cold compute is the same full-window hop scan the declared view already pays for 7d (its own TTL comment measures 16.7 s on a live-shaped database); like the declared view it is then cached for 5 minutes, and every scope filter is served from that one scan. - Commit `ceaeb56b` renamed the filter from `region=` to `scope=` after opening; the real-data figures above were taken before the rename and are otherwise the same code. ## Tests - Red commit `2a9b1286`: response types and a `mode=transport` stub that returns no rows, plus tests. Four fail on the stub: - repeaters that never answered are listed with `asked: false` and null declared fields (and the answered one with its declared list and `notObserved`); - `scope=fr` / `%23fr` / `FR` marks carriers and non-carriers and counts them; no scope means no `carriesScope` or counts; - an unknown mode is a 400; - the transport and declared views are cached separately (requesting one does not serve the other). - Also added, passing on the stub by construction: a repeater with no traffic in the window is not listed; the node blacklist applies. - Green commit `60c669f4`: the implementation, `docs/api-spec.md` (new `mode` and `scope` parameters, a Transport view section with the response shape), and the benchmark. - `make test`: 16 of 16 modules pass, including every existing scope-audit test. ## Review notes - Hidden-name and blacklist filtering use the same rules as the declared view. - `newTestStoreWithDB` now takes `testing.TB` so the benchmark can use it (callers unchanged). - No new `map[string]interface{}`; the response is a named struct. - No config added. ## SonarLint SonarQube Community rules (the SonarLint engine) were run on the files this PR touches, and only issues on lines it adds are listed. - **Fixed** in `060668b6`: S3776 (cognitive complexity 41 > 15) in the new `computeScopeTransport`. It is split into target, visibility, row and sort helpers; the response is the same and the transport tests are unchanged. ## Checked against the repo rules - Rebased onto `master` (`673442cc`). Each red commit was re-run and still fails on its assertions; each green commit passes (TDD.md). - `make test`: all 16 modules pass. Frontend suites: all pass except `test-issue-1956-release-routing.js`, which needs CI's `GITHUB_REF_NAME` and fails the same way on `master` locally (#2160 addresses it). - No private data: test fixtures and examples use the codes the existing tests use (SJC/SFO/LAX, `#be`-style scopes). - No new `map[string]interface{}`, no hard-coded colours. `git add` was explicit, and there is one logical change per commit. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…-render (#2180) ## What `test-issue-1648-m2-icons-e2e.js` now asserts the configured-scope row only after the page's `theme-refresh` re-render has fired and the detail has rendered again. ## Why On 2026-10-10 this assertion failed CI on four PRs (#2169, #2171, #2154, #2164), each time at the first live-map case: `confirmed scope row must be visible, including an empty scope`. A re-run of the failed jobs passed every time. The code path: - `customize-v2.js:814` dispatches `theme-changed` once `/api/config/theme` resolves, on every page load. - `app.js:1309-1315` turns that into `theme-refresh` 300 ms later. - `live.js:4713` re-runs `showNodeDetail(activeNodeDetailKey)` on `theme-refresh`, and `showNodeDetail` first sets `#nodeDetailContent` to "Loading…" (`live.js:2598`). This is the only caller besides the marker click (`live.js:3090`). - The test waited for `#nodeDetailContent table` and then called `row.isVisible()`, which does not wait. A refresh landing between the two fails the check. The same race could let the null-scope case pass on the "Loading…" panel, because `row.count() === 0` holds there too. The fix uses the same `addInitScript` flag pattern as `test-e2e-playwright.js:83`. ## Testing - Locally against a server on the E2E fixture: 5 of 5 runs pass, 27 of 27 checks each. - **Not reproduced locally.** The unchanged test also passed every local run: 5 plain, 3 with `/api/config/theme` delayed by 1 s, 5 with 8x CPU throttling. So the reasoning above comes from the code path and the four CI failures, not from a local red-then-green. - Test-only change, no product code touched. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
## Summary Builds on the transport view from #2167 and #2169 (both merged); rebased onto `master` (`352a0ccd`), so the diff is this PR's six commits only. The Scope Audit was the one analysis page without the region selector. This adds the shared region filter to it, for both the declared and the transport view. `?region=SFO,SJC` means what it means on every other endpoint: count only forwarding **heard by observers in those regions**, not repeaters located there. - **API** (`GET /api/scope-audit`, both modes): - The region codes are resolved to their observers first (a handful of rows), then the audit's hop scan gets `AND o.observer_idx IN (…)` on v3, or `o.observer_id IN (…)` on v2. The 3.5M-row hop scan never joins `observers`. - `All` means no filter (shared `normalizeRegionCodes`). Codes with no observer match nothing, instead of silently widening to every observer. - Responses echo the normalised codes as `region`. The declared and transport caches and their singleflight keys now include the region, so each region set is cached separately with the existing TTLs. - Declared-region verification (`regionEvidence`) is deliberately left unfiltered: it corroborates what a repeater declares from its own traffic, which doesn't depend on where a hop was heard. - **Page:** the shared `RegionFilter` sits with the view and window controls; a change reloads the view, and the selection persists like every other page. The transport view's text box is relabelled **Scope** (it already sends `?scope=` since the rename in #2167), so the page no longer has two controls called Region. - **Docs:** `docs/api-spec.md` (parameter, both response shapes) and the OpenAPI parameters for the route (`mode`, `scope` and `region` were all missing). Partial fix for #2142 (its region filter for rollout tracking), together with #2167 and #2169. ## Performance The unfiltered path must not get slower: it's the same query with an empty clause. `BenchmarkComputeScopeTransport` (from #2167, 1,200 repeaters), the PR it stacks on vs this branch, `-count 8`: | | before | after | | |---|---|---|---| | sec/op | 220.3 ms | 218.0 ms | ~ (p=0.065) | | B/op | 43.77 MiB | 43.70 MiB | ~ | | allocs/op | 833.5k | 833.5k | ~ | **Filtered requests, measured on staging** (a copy of a production database): the first version drove `o.observer_idx IN (…)` from `idx_observations_observer_idx`, reading every observation the region's observers ever made. Fixed in `80414242` with a unary `+` (the same treatment as #2166), so the scan starts from the `first_seen` window and filters per row. Red test `67e063b1` asserts the query plan against a fixture with the real indexes. | scan, one region over 24h (13 observers) | plan | time | hop rows | |---|---|---|---| | `o.observer_idx IN (…)` | observer index | 13.44 s | 102,114 | | `+o.observer_idx IN (…)` | `first_seen` window | **0.25 s** | 102,114 | | unfiltered | `first_seen` window | 0.35 s | 225,983 | Cold end-to-end requests on staging after the fix: 24h declared, unfiltered 0.84 s · one region 0.48 s (was 16.7 s) · two regions 0.22 s · transport, one region 0.42 s · transport, six regions 0.75 s · 7d, one region 3.46 s. 24h transport, live data: 383 repeaters unfiltered, 204 heard by one country's observers. ## Tests - Red `629e4f36` (Go): extends the audit fixture with an `observers` table and `observations.observer_idx`. Covers the declared view ignoring a scope heard only by another region's observer, the transport view listing only repeaters heard in the region, `region=All`, per-region caching, a region with no observers, and the echoed codes. Five fail on assertions; `region=All` passes because the stub ignores the filter, which is the behaviour it pins. - Green `6f1b1e1c`: the implementation, plus a v2-schema case (`observer_id`), so both branches of the clause are covered. - Red `b4933d68` / green `08ecc172` (frontend): `apiPath` carries the selection in both views and adds nothing when none is selected. The region control on the page is a net-new UI surface. - `go test` (cmd/server) passes. Frontend suites: 193 of 194. The one failure, `test-issue-1956-release-routing.js`, needs CI's `GITHUB_REF_NAME` and fails the same way on `master` locally. ## Screenshot The same transport view with the shared region filter set to **SJC**: only forwarding heard by San Jose observers counts, so 7 repeaters are listed instead of 44, with their carried scopes counted from those observers alone. Taken against `test-fixtures/e2e-fixture.db` with fictional scopes.  ## Browser validation The branch's server was run locally with 14 configured regions (dropdown layout), checked in headless Chromium at 1280, 900, 700 and 390 px (touch): | check | 1280 | 900 | 700 | 390 touch | |---|---|---|---|---| | ticking SJC re-requests with `®ion=SJC` | yes | yes | yes | yes | | menu fully on screen | yes | yes | yes | yes | | page wider than viewport | no | no | no | no | | page errors | none | none | none | none | Found and fixed this way: the filter is the right-most control on desktop, and the menu (which opens rightwards) ran 66 px off the page. It now opens leftwards above 640 px. On a phone the controls wrap, the filter starts a line, and the default applies. ## SonarLint SonarQube Community rules (the SonarLint engine) were run on the files this PR touches, and only issues on lines it adds are listed. - **S3776 on `ScopeAuditForwardingHeardIn`:** complexity 55. The function it generalises, `ScopeAuditForwarding`, is 54 on `master`; the region clause adds one point. - **S3776 on `computeScopeAudit`:** complexity 65, unchanged from `master`. - Both pre-date this PR. Splitting them is better as a refactor PR of its own than mixed into a feature. - The page code's findings were fixed in the transport-view PR it stacks on. ## Checked against the repo rules - Rebased onto `master` (`673442cc`). Each red commit was re-run and still fails on its assertions; each green commit passes (TDD.md). - `make test`: all 16 modules pass. Frontend suites: all pass except `test-issue-1956-release-routing.js`, which needs CI's `GITHUB_REF_NAME` and fails the same way on `master` locally (#2160 addresses it). - No private data: test fixtures and examples use the codes the existing tests use (SJC/SFO/LAX, `#be`-style scopes). - No new `map[string]interface{}`, no hard-coded colours. `git add` was explicit, and there is one logical change per commit. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Summary
Page half of #2142. Stacked on #2167 (the API); review that first. #2171 adds the observer region filter on top. Partial fix for #2142 until all three are in.
Adds a Declared / Transport toggle to
#/scope-audit. The transport view calls?mode=transportand lists every repeater seen forwarding in the window:A Region box (suggestions from the scopes seen) adds a Carries X column, lists the repeaters not yet carrying it first, and summarises carriers against active non-carriers: the rollout report the issue asks for. Window, view and region are deep-linked (
#/scope-audit?window=7d&mode=transport&scope=be) and survive a reload. The page says plainly that transported means seen carrying, not configured, and that over a short window a quiet region looks the same as a missing one. The intro text changes with the view.Screenshot
The transport view with scope
betyped in: every repeater seen forwarding in the last 24 h, whether it carriesbe(non-carriers first), the scopes it was seen carrying with counts, and its declared regions or not asked. Taken againsttest-fixtures/e2e-fixture.dbwith fictionalbe/be-van/nlscopes.Browser validation (staging, real data)
Headless Chromium against a staging copy of a production database with its live feed:
#/scope-audit?window=24h: the declared view, 16 rows, as before.mode=transport.#XX-YYin capitals: normalised toxx-yy; the hash gains it (nowscope=xx-yy); summary "385 repeaters seen forwarding in the last 24h · 16 with a declared-regions answer · 51 carried xx-yy · 334 active but not seen carrying it"; the first rows are the non-carriers.Two layout bugs found this way were fixed before opening: the region box inherited
width: 100%from.nodes-search, anddisplay: flexoverrode thehiddenattribute on the region bar.Tests
tests/unit/test-frontend-helpers.jsgains 10 cases on rendered markup and state, using the page's exported internals like the existing scope-audit tests: hash and API paths (and that a region never leaks into the declared view's link), region normalisation (#, capitals, empty), not asked vs declared chips (with the*wildcard), carried-scope chips with counts and the filtered region marked, the carries column only with a region, HTML escaping of names, the summary counts, region suggestions, and the per-view intro. 768 passing in that file.Net-new UI surface, so per TDD.md the tests land in this PR rather than as a preceding red commit. Frontend suites on this branch: 193 of 194; the one failure,
test-issue-1956-release-routing.js, needs CI'sGITHUB_REF_NAMEand fails identically onmasterlocally (#2149 and #2160 address that).Review notes
windowHonestyNote.SonarLint
SonarQube Community rules (the SonarLint engine) were run on the files this PR touches, and only issues on lines it adds are listed.
df53e770:modeQueryhelper for the page hash and the API path (S3358; the two copies were identical);localeComparefor the region suggestions (S2871),for…ofand.dataset(S4138, S7761), optional chaining (S6582);public/. No behaviour change.Checked against the repo rules
master(673442cc). Each red commit was re-run and still fails on its assertions; each green commit passes (TDD.md).make test: all 16 modules pass. Frontend suites: all pass excepttest-issue-1956-release-routing.js, which needs CI'sGITHUB_REF_NAMEand fails the same way onmasterlocally (test: make local validation independent of runner environment #2160 addresses it).#be-style scopes).map[string]interface{}, no hard-coded colours.git addwas explicit, and there is one logical change per commit.🤖 Generated with Claude Code
https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H