Skip to content

feat(regions): explore shared history and observed directed routes - #505

Merged
dborup merged 2 commits into
masterfrom
codex/regions-observed-routes
Oct 17, 2026
Merged

dborup merged 2 commits into
masterfrom
codex/regions-observed-routes

Conversation

@dborup

@dborup dborup commented Oct 16, 2026 •

Copy link
Copy Markdown
Owner

Summary

Add a Connections view to Regions, with two deliberately different evidence classes:

  • Undirected scope pairs weighted by distinct visible identities attributed to both scopes in the existing transmission-level relay history.
  • Directed consecutive relay steps from individual scope-matched flood observations, with distinct-transmission counts and opposite directions kept separate.

Shared history memberships are not labelled as verified forwarding, gateways, cross-scope traffic or direct RF links. Independent steps are not joined into a claimed full route. The existing evidence does not establish a region-change point, so this PR does not invent directional region-to-region crossings.

This is stacked on #503 (codex/regions-health-trends), which builds on #502 and #501. It should not be merged before its base stages. No merge or deployment is included.

Evidence and correctness

  • Inspect each observation's own route and both transport codes; require explicit transport-flood evidence matching the canonical scoped frame. Never promote a planned direct path or fall back to a missing legacy observation header.
  • Reuse stored scope attribution rather than claiming independent authentication of region names. Exclude zero-scope codes, TRACE and adverts.
  • Reset adjacency at unresolved, ambiguous, invalid and listener-only hops; never bridge across observation boundaries.
  • Deduplicate each directed step and scope per transmission across repeated observers and loops. Summing step counts is step appearances, not a globally unique packet count.
  • Use the same transmission first_seen, half-open completed UTC hours and 21-day maximum reconstruction horizon as Timeline/Health. Current visibility, aliases, role changes and archived metadata policies still apply.
  • Keep route endpoint metadata independent of relay-history membership, so an explicit flood observation is not lost merely because its canonical transmission was direct.
  • Show partial/capped evidence and missing coordinates explicitly. Unpositioned steps remain in the table. No observer RSSI/SNR is presented as internal-edge signal quality.

Existing history limitation, explicitly disclosed

Shared pairs reuse the same transmission-level history as Timeline/Health. Its legacy attribution gates on the canonical transmission route and can include planned direct-path candidates when one content transmission has mixed flood/direct observations. It does not independently verify every observation's route/scope variants. This is reachable in the unchanged base and is not a limitation of the new strict directed steps.

The new UI therefore calls these shared relay-history memberships, explains their weaker provenance, and does not present them as guaranteed lower bounds on physically shared relays. Correcting the older history attribution is a separate follow-up; this PR does not silently rewrite the previous stages' statistics.

UI and API

  • Shareable view, period, scope, completed hour and table pages; no automatic playback or unsupported route comparison.
  • One lazy /api/regions/routes bulk fetch; local time/scope selection, keyed paginated tables and a bounded directional map. Latest, Timeline and Health remain available.
  • Named typed OpenAPI response schemas; server remains read-only. No schema, retention, dependency, config or upstream-repository changes.
  • Protocol semantics checked against MeshCore firmware a366955cb2f67b8e6842d4f00d2b6a554dddd88a (Packet.cpp, Packet.h, Mesh.cpp, RegionMap.cpp and packet-format documentation).

Performance

Route evidence is aggregated with the existing background relay snapshot, outside the packet lock after bounded capture; requests do not scan packet history or acquire that lock. Source paths, per-transmission paths, steps, populated hours, endpoint metadata and rendered rows/edges are bounded. Observation transport fields consume existing padding: StoreObs remains 200 bytes on 64-bit.

Apple M5 / Go 1.26.3 / Node 26 measurements (synthetic data):

Workload Measurement
Bounded capture inside the read lock, 30k transmissions / 2k relays 6.62 ms, 13.08 MB allocated
Whole paired background snapshot recomputation, same workload 96.34 ms, 109.64 MB allocated
Warm paired bulk endpoint, same workload 12.13 ms; 10,418,506-byte JSON
Incremental route evidence at 5k rows / 100k hourly cells 8,390,413-byte JSON; 6.83 ms encoding (excludes history/endpoint metadata)
Browser helper: 5k nodes × 32 scopes, 5k steps / 100k cells 57.9 ms step indexing; 39.0 ms initial all-scope selection including frame selection; 13.0 ms selected-scope selection; 0.006 ms cached selection

Aggregation is linear in retained observations/path hops plus bounded sorting; it is not per-node packet scanning. Shared relations use bounded candidate pairs and inverted membership intersections rather than storing a potentially large per-node scope clique. Table pagination/theme redraw reuse the selected interval cache. These are local measurements, not production latency guarantees.

Limits are explicit: 250k captured path variants globally, 128 per transmission, 8,192 bytes per path, 5k published steps, 100k hourly cells and 10k endpoint identities. UI: 1k retained scope pairs, 25 rows per table page, 200 mapped step rows. Display limits/default period are tracked for future customizer exposure; protocol checks are not adjustable thresholds.

Validation

  • Full Go suites: server passed (60.674 s); ingestor passed (112.706 s). Focused protocol/history/route race tests passed.
  • Full npm test: 257 test files passed, no failures. Final unit/UTC/DST-host timezone checks, eslint and XSS preflight passed.
  • Focused Regions browser suite: 36 checks passed on normal and final instrumented frontend, including desktop/mobile screenshots, literal XSS names, route-only metadata/scopes, incomplete/capped evidence, keyed pagination, request reuse and navigation.
  • Full core browser suite passed on normal and final instrumented frontend: 154 passed, 3 existing fixture-dependent skips.
  • A separate, unmocked local API + browser smoke test used synthetic scoped packets: three directed steps, each counted once despite duplicate observations; reverse direction preserved; one shared-scope pair; an unpositioned step retained in the table; only one Regions request while changing scope/period; no page errors. No production/staging data or configuration changed.
  • Both manifests validated: all 167 E2E suites and 2,680 active Go entries are assigned exactly once.
  • Two independent reviews have no open findings. One P2 found during review (stale map theme/dimensions after returning from Health) was reproduced red, fixed centrally for all visible map views, and regression-tested for Connections and Latest.

The stacked branch requires an explicit manual CI run because the automatic PR workflow targets master. Initial full-stack CI passed on e2b14862. After the reviewed provenance-only follow-up, final-head CI run 38906804186 also passed on 53fd65f5ab3bbe8d5cefe76e3ad20783bf6c1d61: all Go/race, ingestor, library/CLI, script/container, JavaScript/lint, three E2E shards, coverage aggregation, Docker validation and the final required gate succeeded. No publishing or deployment was performed.

Local instrumented browser coverage for the final head (isolated to its own coverage captures): Regions routes 92.72% statements / 99.08% lines; Regions map 94.13% / 98.97%; Regions history 90.47% / 97.47%. These are feature-file results, not a claim about whole-application coverage.

@dborup

dborup commented Oct 16, 2026

Copy link
Copy Markdown
Owner Author

Review feedback addressed (commit 53fd65f5)

  1. Rename shared relations to relay-history memberships and table counts to history identities, rather than claiming independently verified forwarding in both scopes.
  2. Add an always-visible provenance explanation: Timeline/Health's inherited transmission-level attribution can include planned direct-path candidates on mixed-route content transmissions. The new directed steps retain their stricter own-observation frame checks.
  3. Remove blanket lower-bound claims for physically shared relays and add browser assertions for the visible qualification. Document legacy attribution repair as a separate correctness follow-up.

This follow-up changes wording/documentation/tests only: no aggregation, protocol, Go, schema or configuration changes. Full local frontend suite passed again (257 files), as did all 36 normal Regions browser checks. Final instrumented checks and CI on the exact new head are running; no merge or deployment has occurred.

@dborup dborup changed the title feat(regions): explore shared relays and observed directed routes feat(regions): explore shared history and observed directed routes Oct 16, 2026
@dborup
dborup changed the base branch from codex/regions-health-trends to master October 17, 2026 11:25
@dborup

dborup commented Oct 17, 2026

Copy link
Copy Markdown
Owner Author

Review — CS-Coordinator PR#505 — head 53fd65f

Verdict: APPROVE with two follow-ups (not blocking, filed separately).

Reviewed against master 9933d05d (#501–#503 are merged, and this PR is retargeted).

Checked

  • Merged tree (master + this head). go vet is clean, the full cmd/server suite passes, and -race on Region|Route|Relay|412|470 passes.
  • Work placement. Directed steps are collected inside the existing relay-snapshot pass (routes.captureAt, built with the snapshot), not per request. Per-observation transport codes are cached in existing StoreObs padding. /api/regions/routes serves the published snapshot.
  • Staging (master-9933d05d-pr505, real data, 48 h retention):
    • /api/regions/routes returns 200 in 0.76–1.0 s, with 5.4 MB raw / 679 KB gzip.
    • 391 step nodes are present.
    • There are 0 panic, fatal, concurrent map or SQLITE_BUSY entries.
  • UI. The Connections view renders a scope-pair table, a directed-step table and 332 map lines. Capped or partial evidence is stated plainly ("Evidence is incomplete or capped; absence cannot be established"). The wording avoids claiming gateways or verified forwarding, which is good.

Follow-ups

  1. The step cap is hit on real data. The response reports steps: 5000, stepsTruncated: true already with 48 h of retained history. meshview-pve keeps 168 h, so it will be truncated even more. The view stays honest, but directed steps are effectively always incomplete on our instances. Consider aggregating step counts per (from, to, scope) across the horizon before capping, a size-based budget, or a higher cap with a measured payload limit.
  2. Memory. Container RSS on staging was ~1.72 GiB with this PR, against ~1.55 GiB on comparable recent deploys. That is within the 2.7 GiB cap, but production hosts are smaller (meshview.dk has 3.7 GiB RAM shared with other services). Measure the snapshot's retained size and add it to the memory diagnostics.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant