[A13xB0]

This PR adds a Data tab (#/data): a full-screen workspace covering every packet of the last few days. Each packet is placed by the area it started in (from GPS and node positions, drawn from a GeoJSON areas file), not by which observer heard it, and by every area whose observers heard it. It works with any deployment's areas. It is off unless data.enabled is set.

Plan: https://claude.ai/artifact/PAJwg4KmQQ7Cp7nSDvyBo9.

This one PR replaces #14 and #15. Each step is its own red/green commit pair, so the steps can be split out again if wanted.

The steps, in commit order

#	Commits	What
1	846b0b6 → 1f59c9f, 4d72542 → b87f636	Areas from GeoJSON. New internal/areamap package. areasFile in config: Polygon or MultiPolygon, holes, layers, smallest area wins, iata fallback, duplicate codes refused. Areas from the file also work in /api/config/areas and the existing area= filter.
2	53b7510 → 2108e69	Where a packet started. New internal/origin package. The start comes from, in order: the advert's GPS, then the sender's node record, then the majority first relay of 2+ bytes (a shared prefix counts only if every holder is in one area), then the zero-hop observer's area, else unknown. For each receiving area it records relayed or across, the relay it was heard from, and the home-area boundary relay.
3	d2877b7 → 3071bac	The index. New internal/dataindex package: one pointer-free 56-byte row per packet per receiving area, with strings interned and hashes stored as uint64. Builds the cube, the summary (matrix, flows, relays) and filtered packet lists, all from memory.
4	360012b → dd7d527	Server and API. The data config block and the background fill from SQLite (read-only, newest chunk first). Every 2 min the last 30 min are re-derived; positions are re-read every 30 min; the whole index is rebuilt nightly. Adds GET /api/data/config, /summary, /cube and /packets, and documents them in openapi.go and docs/api-spec.md.
5	7daecbc, bc314c4	The tab. public/data.js and data.css, plus nav, drawer and phone-menu entries.

The tab

• Period bar for the whole page.
• Rail: filter chips (is / is not, with a count per value), preset filters, bars/lines/area, count/share, X axis (time or any dimension), time grouping, split by, separate charts by, and picked-out series.
• Answer side: tiles, a chart with tooltips, a legend with counts, relay tables, a numbers table, and a who-hears-whom matrix where every cell and heading can be clicked.
• Drill-down: any mark opens its packets in a modal with sortable columns you can add or remove. Clicking a row opens CoreScope's own packet view, and Back returns to the same question.
• Home area: with data.homeArea set, the presets and tiles add leaving/entering for the home area and its boundary relays.
• Phone: chart options fold away under the filters so the chart is on the first screen.

Tests

• Go: the red tests fail on assertions; the green commits pass. scripts/allmod.sh vet and test pass on all 19 modules, including the full server suite. The Dockerfile COPY invariant check passes, and gofmt is clean.
• Frontend: new tests/unit/test-data-tab.js (9 tests, jsdom against faked APIs), added to test-all.sh. All frontend suites pass except test-issue-1956-release-routing.js, which also fails on master because it needs the CI-only GITHUB_REF_NAME.
• Accessibility: /data added to the axe gate. Result: 0 violations across desktop and mobile, dark and light.
• In a browser: checked at 1440 px (dark) and 390 px (light). No console errors and no sideways scroll. Checked the filter editor, lines, share, a dimension on the X axis, separate charts, the 7-day period, a matrix-cell click, full screen with Esc, and drill-down to packet view and back.

Numbers

	Result
Point lookup (areamap)	2.1 µs, 0 allocs
origin.Derive	1.1 µs, 224 B, 2 allocs per packet
7-day cube, worst case (120k rows, every key distinct)	64 ms
Re-deriving 30 minutes (300 packets)	0.7 ms
Real data: 24 h of pre-prod (7,091 packets) loaded locally	indexed in 0.28 s, 10,027 rows
Same 24 h through meshcore.scotmesh.net's peering code vs origin	0 differences in origin, heard and via

Still to do before merge

1. Pre-prod gate. Waits for the current soak to end (~01:07 UTC 11 Oct). Then: scotmesh-preprod with this branch merged, ScotMesh's areas.geojson, and a 24 h soak against these limits:
   • at most 5% more GC per hour than the soak baseline;
   • at most 30 MB more heap.
2. ScotMesh's areas file is built and checked; it lives with pre-prod's config, not in the repo.
   • Areas: SCO, IOI, NEE, CUM, WLS, IOM, plus Rest of England.
   • Sources: ONS boundaries (OGL, contains OS data), Ireland from OSM via geoBoundaries (ODbL), Isle of Man via geoBoundaries (CC BY).
   • Shape: simplified to about 200 m, with a 3 km margin out to sea; 247 KB.

Packets page: area means origin

• Bug fixed: the packets page's area filter kept a packet only when it named its sender, which only adverts do, so picking an area hid every other type.
• Now: with the index running, area= on /api/packets keeps the packets that started in the area, by the same rules as this tab: the advert's position, else the first relay (2+ byte hop), else the observer of an unrelayed packet. Without the index, or for an area it doesn't place in, area= behaves as before.
• Shown: list responses carry origins: {hash: {area, basis}} and the packet detail an origin. The packets table gets an Origin column (code; how it was decided on hover; … for packets newer than the index, — when nothing placed it) and the detail pane an Origin row. The column only appears on servers with areas.
• On pre-prod (24 h): 82% of packets get an origin; area=SCO went from 4.7k packets (adverts only) to 23k of every type. List timings unchanged (0.22–0.33 s).
• Tests: internal/dataindex (StartedIn, OriginsOf), server (data_packet_origin_test.go: filter by origin, origins in list and detail, none without the index), e2e test-packets-origin-e2e.js (6).

Notes

• The Data link always shows, as Scope Audit's does. Where the tab is off, the page says so. Hiding the link when off (the way rx-coverage is injected) can follow if wanted.

Path watch panels

Two new panel types follow the path between two repeaters:

• watch: the share of packets that passed the first repeater and reached the second in the last 24 hours, against its usual range; how many made it; hops; the main route; the daily share as dated bars; the routes taken; and a route history in words.
• watchlist: several pairs, one line each (both ways if asked), tagged when the route changes. A line opens the full watch.
• Route changes, compared day by day against the week before:
  • a repeater that dropped out is quiet if it stopped relaying everywhere, rerouted if it's still busy elsewhere;
  • one that started carrying is a new repeater if first heard in the last 14 days, else sharing the load;
  • the main route moving, or its share moving.
  • Each change needs at least 10 packets behind it (30 for a route change). It stays a flag for 7 days, then moves into the history.
• Adding watches: + Add watch picks repeaters by name, warns when nothing has passed between them, and saves to your copy of the dashboard. A repeater variable kind gets a picker, a built-in Path dashboard works for any pair, and repeater pages link to it with Watch a path.
• Server:
  • internal/pathwatch does the analysis;
  • GET /api/data/path and POST /api/data/paths (up to 24 pairs) read the packet store's path index; resolved paths are read in batches, and only for the packets that reached the second repeater;
  • results are cached for 5 minutes, it's read-only, and returns 404 when the tab is off.
• Tests: package tests, server tests on a seeded store, and tests/e2e/test-data-watch.js (jsdom, 8 cases).
• On pre-prod: a "Links we watch" dashboard in dashboards.json. Six pairs are answered in 1.7 s; Keir Hill's outage on 7 Oct shows as "quiet" on the North link.

🤖 Generated with Claude Code

https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H

[A13xB0]

Dashboards

The Data tab is now a dashboard page rather than one fixed layout. Rebased onto master.

• a7f6b529 (red e0875114): server side. Dashboards come from a file. GET /api/data/dashboards returns the built-ins (Overview, Scopes and channels; generic, needing only an areas file) followed by the operator's, all validated at startup.
• 6503ae27 (red 9fbfa3a6, 15 failing on assertions): the page.
  • Panels: eight types (bars, lines, area, matrix, stat, table, top list, text) on a 12-column grid, each a query over the period's cube.
  • Variables: $variables such as $home fill titles, text and filters, and sit in the address.
  • Editing: edit mode has a rail for the dashboard or the selected panel. Panels drag, resize and use arrow keys, with undo, and the layout never overlaps.
  • Copy-on-edit: built-in and operator dashboards are copied into the browser when edited, never changed in place.
  • Export and import: export gives the JSON. Import validates it with the server's rules first.
  • Share: puts the whole layout in a #/data?d= link, with nothing stored on the server.
  • Drill-down: any figure lists its packets, using the panel's filters plus what was clicked.
• The jsdom test moved to tests/e2e/test-data-tab.js (domIntegration plus its own deploy.yml line), because the unit gate has no node_modules.

Config

"data": {
  "enabled": true,
  "days": 7,
  "layer": "country",
  "homeArea": "SFO",                 // default for an area variable such as $home
  "dashboardsFile": "dashboards.json", // optional: a JSON array of schema-1 dashboards, relative to the config dir
  "defaultDashboard": "cross-border"   // optional: id to open first (default: the first built-in)
}

A dashboard in that file (the same shape as Export):

{ "schema": 1, "id": "cross-border", "title": "Cross-border traffic", "window": "24h",
  "variables": [{ "name": "home", "kind": "area", "label": "Home" }],
  "panels": [
    { "id": "leaving", "title": "Leaving $home", "type": "stat",
      "query": { "filters": [{ "d": "origin", "n": 0, "v": ["$home"] }, { "d": "border", "n": 0, "v": ["crossed"] }] },
      "layout": { "x": 0, "y": 0, "w": 3, "h": 2 } },
    { "id": "where", "title": "Where $home's traffic is heard", "type": "bar",
      "query": { "filters": [{ "d": "origin", "n": 0, "v": ["$home"] }, { "d": "border", "n": 0, "v": ["crossed"] }], "x": "heard", "split": "kind" },
      "layout": { "x": 3, "y": 0, "w": 9, "h": 4 } }
  ] }

Screenshots

An operator dashboard on live data (1440 px, dark):

Edit mode: the rail edits the selected panel, here a copy of the operator dashboard:

The packets behind a figure:

Phone (390 px): panels stack in reading order:

Checks

• go test (cmd/server) passes. All standalone frontend suites pass, and test-data-tab.js passes 19/19.
• Headless Chromium at 1440 px (dark) and 390 px (touch, light): five dashboards, no console errors, no horizontal scroll, and no chart overflowing its panel. Dragging a panel, editing a copy, sharing and drilling down were all exercised.

🤖 Generated with Claude Code

https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H

[A13xB0]

Areas are optional

6ef5df54 (red c1440ae6): with data.enabled and no areas (no areasFile, no areas), the Data tab still runs. It counts by kind, packet type, channel, scope, scope kind and routing.

• API: /api/data/config has places: false, and dimensions leaves out border, origin, heard, through and how. data.homeArea is refused.
• Index: every packet is one row heard in unknown. With areas that row is a placeholder (a packet no placed observer heard); here it is counted. Options.NoPlaces controls this.
• Built-in dashboards: there is a variant without place panels.
  • Overview: packets, flood adverts and messages; kind over time; busiest scopes and channels; packet types by routing.
  • Scopes and channels: scope by kind instead of scope by receiving area.
• Operator dashboards: one with an area variable or a place dimension stops startup, saying it needs areas.
• Editor: offers only the dimensions that exist and no area variables. New panels start on scope and kind, and the packet list leaves out place columns. Import refuses a dashboard that needs areas. A stored or shared one that names them shows "needs areas" in the panel rather than drawing nonsense.

Minimal config:

"data": { "enabled": true, "days": 7 }

Checked: go test for the server, dataindex and origin; all standalone frontend suites; and test-data-tab.js (21/21). Headless Chromium against a server with no areas, at 1440 and 390 px: both built-ins drew with data, with no console errors and no overflow.

🤖 Generated with Claude Code

https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H

[A13xB0]

The memory problem is solved; the Data tab is back on in pre-prod.

The OOM kills came from the packet store being too big on its own, with the startup fill pushing it over the 1.5 GB container limit. These fork PRs cut both:

PR	change
#21	observation dedup without per-packet maps
#22	subpaths counted per request (spTxIndex removed)
#23	one copy of each observer string
#26	relay-filtered candidate lists built once
#27	observation timestamp cache in 8 bytes
#28	bulk loaders stop reading observation raw_hex
#29	row scan targets reused
#30	hop counts without JSON decoding
#31	hop context without a map
#32	resolved_path keys read in place, including eviction
#33	this tab's fill reads raw column bytes (built on this branch)

Measured on a copy of a week of a busy network:

	before	after
peak RSS with the Data tab on	1093 MB	775–827 MB
allocated during startup	5.4 GB	2.5 GB
one fill: allocated	862 MB	581 MB
one fill: time	4.1 s	2.6 s
steady CPU		about 1% of a core

On pre-prod (data.enabled: true, GOMEMLIMIT=1GiB):

• five back-to-back restarts peaked at 836–901 MB, with no OOM kills;
• the index filled in 4.7–6 s each time;
• soak since 01:45 UTC: no kills, and the cube returns 200 every minute.

Removing blocked: memory.

🤖 Generated with Claude Code