From 3124c60733d7dbda7697eed98b9a2aefb497e82f Mon Sep 17 00:00:00 2001 From: efiten Date: Thu, 22 Oct 2026 21:29:12 +0200 Subject: [PATCH 1/2] docs(release-notes): v3.15.0 Notes for v3.15.0 in docs/release-notes/v3.15.0.md and the CHANGELOG entry. 77 commits since v3.14.0, 19 feats, so minor. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 13 ++ docs/release-notes/v3.15.0.md | 216 ++++++++++++++++++++++++++++++++++ 2 files changed, 229 insertions(+) create mode 100644 docs/release-notes/v3.15.0.md diff --git a/CHANGELOG.md b/CHANGELOG.md index e22132db6..da9212568 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,19 @@ ## [Unreleased] +## [3.15.0] - 2026-10-22 + +See [docs/release-notes/v3.15.0.md](docs/release-notes/v3.15.0.md) for the full notes. 77 commits since v3.14.0: 27 fix, 23 perf, 19 feat, 3 test, 2 ci, 2 docs, 1 chore. + +### Highlights +- **Last Heard and node status count only certain evidence** (#2261, #2262, closes #2260) - a shared 1-byte relay hop no longer keeps a silent repeater "Active". New status `uncertain`. **Operator awareness required**: statuses, `last_heard` and offline mails change after the upgrade. +- **`users.db` migrates to schema v6 with user accounts on** (#2174) - a v3.14.x binary refuses it afterwards, so keep the pre-upgrade backup for a rollback. +- **The ingestor runs a key backfill at every start** (#2133) and refreshes planner statistics at most once per 24 h (#2158). +- **Optional LoRa radio settings per node** (#2209, #2210), observer TX airtime (#2208), companion linking (#2174 to #2176), adaptive offline mails (#2262). +- **Packet store memory and request latency** (#2146 and 23 perf commits) - e.g. startup to ready 31 s to 21 s (#2181), `/api/nodes` p95 165.4 to 114.4 ms (#2165). + +No manual migration step. + ## [3.14.0] - 2026-10-08 See [docs/release-notes/v3.14.0.md](docs/release-notes/v3.14.0.md) for the full notes. 23 commits since v3.13.1: 13 fix, 8 feat, 1 ci, 1 docs. diff --git a/docs/release-notes/v3.15.0.md b/docs/release-notes/v3.15.0.md new file mode 100644 index 000000000..00c4ea006 --- /dev/null +++ b/docs/release-notes/v3.15.0.md @@ -0,0 +1,216 @@ +# v3.15.0 + +77 commits since v3.14.0: 27 fix, 23 perf, 19 feat, 3 test, 2 ci, 2 docs (one of them +these notes), 1 chore. Nineteen feats, so minor. + +The headline is node liveness (#2260): a repeater only counts as heard when the +evidence can only be that repeater. The rest is mostly memory and latency work on +the packet store, optional LoRa radio settings per node, and companion linking for +instances with user accounts. + +## Read this before upgrading + +These change what a running instance does without being asked. + +- **Last Heard and node status count only certain evidence** (#2261, closes #2260). + Before, a relay hop counted as "heard" for whichever node the resolver picked, so a + 1-byte hop shared by many repeaters kept a silent one "Active". Now only the node's + own packets and flood relay hops whose prefix matches no other repeater, and no + retired one, count. Adverts with an invalid signature no longer count. + - `last_heard` in `/api/nodes` and the node detail can now differ from `last_seen` + (it was a copy of it). + - A new status `uncertain`: a repeater or room with nothing certain in the silent + window whose shared prefix still shows up in paths. Grey badge, filter buttons on + the nodes list and the map, dimmed on the map like stale. + - Watched-node offline mails now fire for repeaters that were kept green by a + namesake prefix. + - Counts on one staging instance with 1,682 repeaters and rooms, on the day of the change: + 1,497 active, 117 uncertain, 68 stale. No before/after count of the same moment was + taken. Right after a restart the uncertain count starts lower, because the cold + load skips ambiguous relay hops (#1643); it grows with live traffic. +- **Node analytics use only certain data for Availability and Longest Silence** + (#2262). The activity chart is stacked into certain and uncertain. A silence that is + still running now counts. Rates for 30 d and All cover only the data in memory and + the page says from when (#2211): on one staging node the 30 d availability went from + 19.3% to 83.3%, because days before the in-memory span counted as silent. +- **`users.db` migrates to schema v6 at startup when user accounts are on** (#2174). + A v3.14.x binary refuses the file afterwards, so a rollback needs the backup from + before the upgrade (`backups/` next to `users.db`). +- **The ingestor starts a background backfill at every start** (#2133). It rewrites + `enc_*` channel hashes and empty scope names once a matching key is known, and runs + again when the key set grows. It creates three tables in the analyzer database + (`tx_rewrite_feed`, `key_backfill_state`, `key_backfill_done`). The PR gives no + measured cost on a large database. +- **Planner statistics refresh at most once per 24 h** (#2158). A restart within 24 h + of the last refresh runs no `ANALYZE`; a file `.planner-stats-refreshed` is + written next to the database. The first start after the upgrade still runs it. One + `ANALYZE` took 7 min 50 s on a large database. +- **Answers can be older than before, in exchange for latency:** + - `/api/nodes` answers identical queries from a shared cache for up to 15 s + (#2165). A geo-filter, blacklist or hidden-prefix change clears it. Measured on one + deployment: p95 165.4 to 114.4 ms, allocation 1,000 to 594 MB. + - The neighbor graph is only rebuilt in the background, every + `neighborGraph.cacheRecomputeIntervalSeconds` (default 300) (#2222). A rebuild took + 11.1 to 11.4 s on the request path before. + - Server caches past their 30 s TTL serve the old value while one refresh runs + (#2147), and `/api/perf` row counts are up to 60 s old (#2156). +- **Hop names and region placement change:** + - `iataCoords` gains 104 BE, NL, LU, DE and FR codes, and region narrowing now + applies only to the directly heard hop (#2188, closes #2098). + - `/api/nodes?region=` places a node by its zero-hop home regions first, then by the + fewest hops up to `regionFallback.maxHops` (default 3) (#2190, closes #1879). Every + node row gains `region_placement`. + - Transport-direct zero-hop adverts are now detected (#2234, closes #2225). + - The packet detail and the map route view infer 1-byte hops as one whole path; + hops can show as "unlisted" (#2245). + - An observer's own transmissions (`direction='tx'`) no longer count as receptions + in Heard By and the neighbor graph (#2220). Existing neighbor edges keep their + counts. +- **Favorites and home status cards use the role's thresholds** (#2215): a repeater + is degraded after 24 h and silent after 72 h there, as on its node page, instead of + 1 h and 24 h. +- **A geo-filter save on a single-file `config.json` mount now fails with 500** + (#2137). It used to write a copy that the next start discarded. A symlinked + `config.json` is now written through. +- **Hidden tabs stop polling `/api/stats`** (#2162); a visible tab refreshes it at most + once per 15 s. + +## Features + +- **LoRa radio settings per node** (#2209, #2210, closes #1268). A LoRa row on the node + detail (frequency, SF, BW, CR, preset) from the MeshCore map, and a badge plus + `?lora=differs` filter for nodes whose settings differ from nodes within + `rfParams.compareRadiusKm` (50 km). Off until `rfParams.nodesUrl` is set. +- **Observer TX airtime** (#2208, closes #661). A rolling 1 h TX airtime column on the + observer list and a tile on RF Health, with a warning above `dutyCycleWarnPct` + (10%). +- **Adaptive offline threshold for notifications** (#2262). Off by default: + `userManagement.notifications.adaptiveOffline.enabled`. It learns each watched + node's normal silence and mails about 30 minutes into an outage instead of after the + fixed window. +- **External node alerts feed** (#2178, closes #2177): + `userManagement.notifications.externalAlerts` turns an external JSON feed into + `node.external` mails for watched nodes. +- **Companion linking** (#2174, #2175, #2176). With user accounts on, a companion app + links to an account with a device token. `clientRxCoverage.requireLinkedCompanion` + ingests coverage only from linked companions; the account page lists them, with a + "My coverage" view. +- **Coverage gaps** (#2172). The Mobile RX coverage page shows where a companion drove + and received nothing, for instances with `clientRfSamples`. +- **Repeaters known only by a reported name** (#2239). Opt in with + `clientNodeNames.enabled`: names that companions report are stored, used in hop + resolution, and shown with an estimated position. +- **Scope Audit transport view** (#2167, #2169, #2171). Every repeater seen forwarding + a scope, next to the declared view, and an observer region filter for both. +- **Region filter**: `?region=` in the URL works on every page (#2256, closes #2255), + and `regionQuickPicks` adds configured groups to the control (#2170). +- **`?theme=dark|light`** URL parameter for one page load (#2251, closes #2236). +- **Postal as mail provider** for user accounts (#2154): + `userManagement.mail.provider: "postal"`. +- **Rolling hop index for the reach report** (#2214). Opt in with + `retention.hopIndexDays`. One reach scan went from 816 ms to 293 ms in the benchmark; + the index cost about 2 GB on a large database. +- **Docker: run the Go services as an unprivileged user** (#2125, #2235). Set + `RUN_AS_UID` / `RUN_AS_GID`; unset, the services run as root as before. + +## Fixes + +- **Server locks**: no SQL while holding the packet-store lock (#2147); region and + area sets are resolved before taking it (#2247, closes #2219); data races between + ingest and HTTP readers (#2246, closes #2237). +- **Pages**: late responses no longer overwrite a newer page visit on node analytics, + analytics and the account page (#2248, closes #2200, #2218, #2230); the packets node + filter keeps its pick (#2244, closes #2243) and grouped mode re-fetches on an + observer change (#2242, closes #2241); live packets respect the channel filter + (#2192, closes #2191). +- **Analytics**: Packets / Hour is readable on long histories (#2194, closes #2193); + Route Patterns applies `nodeBlacklist` and `hiddenNamePrefixes` (#2189, closes + #2184). +- **Route view**: "N hops" counts only true hops (#2240, closes #2206), and + payload-decoded sender and recipient rows get a badge (#2207). +- **Reach**: the reach page retries a busy (429) report (#2250). +- **Accounts**: a re-registration updates password and name in one statement (#2253, + closes #2228). +- **Ingestor**: an interrupted background migration is recorded as cancelled and + retried (#2161); the background chunk loader sets `observer_iata` (#2185). + +## Performance + +- **Request path** (#2146): the reach scan is driven by the observations time window + (#2155, 14.11 s to 1.93 s on one query); `/paths` reads in batches (#2157, #2224, + closes #2223); battery history uses the primary key (#2156); per-region scope counts + use the `first_seen` index (#2166, closes #2146: 1 h window 3.08 s to 16 ms); the + neighbor graph leaves the request path (#2222, closes #2221). +- **Packet store memory** (#2181 to #2183, #2186, #2187, #2195 to #2199, #2201, #2163, + closes #2148). One deployment: startup to ready 31 s to 21 s (#2181), subpath index + RSS 948 to 767 MB (#2182), heap 581 to 535 MB from shared observer strings (#2183). +- **Ingestor**: live ingestion runs between retention batches (#2153, closes #2151). +- **Opt-in `analytics.pauseWhenIdle`** skips recomputes nobody reads (#2164): 459.7 to + 53.4 MB allocation in the measurement. + +## Tests, CI and docs + +- Frontend coverage from every E2E suite, merged across the shards (#2204, closes + #2203); README badges published to a `badges` branch (#2202, closes #2136). +- Local validation independent of the runner environment (#2160, closes #2159); two + flaky E2E checks settled (#2179, #2180). +- README corrected and the optional features listed (#2205); `SECURITY.md` and OSS + Scanner build files (#2257). + +## Issues closed + +- #661 feat: airtime utilization estimates per observer +- #1268 feat: enrich nodes with LoRa RF params from map.meshcore.io +- #1879 bug: /api/nodes region filter attributes by observer, not repeater home region +- #2098 iata-coords: none of the 42 observers' codes are in the table +- #2136 README badges stale since June: BADGE_PUSH_TOKEN returns 401 +- #2146 Request-path latency grows with DB size +- #2148 perf: avoid allocation-class padding in retained observations +- #2151 perf: let live ingestion run between retention batches +- #2159 test: local validation assumes Actions environment and POSIX file modes +- #2177 Notifications: optional external node alerts feed +- #2184 nodeBlacklist is not applied to Route Patterns +- #2191 Packets page: live WebSocket packets ignore the channel filter +- #2193 Analytics: Packets / Hour chart unreadable on long-retention instances +- #2200 Analytics: older requests can overwrite newer filters or a later page visit +- #2203 Frontend coverage counts only 2 of the E2E suites +- #2206 Route view: DM sender/recipient rows drawn as ordinary hops +- #2218 bug(node analytics): delayed responses overwrite the page after navigation +- #2219 perf: cold region and area lookups still hold the packet-store lock +- #2221 perf: /api/nodes/{pubkey}/neighbors rebuilds the neighbor graph on the request path (~11 s) +- #2223 perf: /api/nodes/{pubkey}/paths runs two queries per transmission +- #2225 bug: zero-hop detection reads the path byte at the wrong offset for transport-direct adverts +- #2228 bug(accounts): make pending registration updates atomic and status-conditional +- #2230 bug(account): delayed session initialization replaces the page after navigation +- #2236 Feature Request: Dark Mode set via URL parameter +- #2237 server: data races between ingest and HTTP readers +- #2241 Packets: observer filter only takes effect after a page reload (grouped mode) +- #2243 Packets: picking a node suggestion sometimes does nothing +- #2255 ?region= in the URL only works on the packets page +- #2260 Node liveness counts ambiguous 1-byte hops: a silent repeater shows "Active" + +Each of these is listed as closing by one of the merged pull requests. + +## Contributors + +Pull request authors: @efiten (33), @A13xB0 (24), @nullrouten0 (6), @n30nex (4), +@anieto (3), @CoderNemesis26 (2), @hermes-gadget (2), @SaarMesh-Bot (2). The +maintainer's own share is 33 of 76 pull requests. + +Issue reporters: @efiten (13), @n30nex (8), @anieto (2), @A13xB0 (1), @dborup (1), +@hermes-gadget (1), @Kpa-clawbot (1), @n1ghtBl00d (1), @SaarMesh-Bot (1). + +No co-authors other than the pull request authors themselves and tooling are credited +in the commits. + +## Upgrade notes + +- No manual migration step. The ingestor creates the backfill tables (#2133) and the + empty `client_node_names` table (#2239) at startup. +- With user accounts on, `users.db` goes to schema v6 at startup (#2174); keep the + pre-upgrade backup if you may roll back. +- Opt-in, off by default: `rfParams.nodesUrl`, `clientNodeNames.enabled`, + `retention.hopIndexDays`, `analytics.pauseWhenIdle`, `regionQuickPicks`, + `userManagement.notifications.adaptiveOffline`, `.externalAlerts`, + `clientRxCoverage.requireLinkedCompanion`, `userManagement.mail.provider: "postal"`, + and `RUN_AS_UID` / `RUN_AS_GID` for Docker. From 316773c27c6a04ca320d5c752947abd1e7cb9d42 Mon Sep 17 00:00:00 2001 From: efiten Date: Thu, 22 Oct 2026 21:37:16 +0200 Subject: [PATCH 2/2] docs(release-notes): v3.15.0 notes written for operators Explain each upgrade change in terms of what an operator sees and what to do, group the features by topic, and keep the numbers that matter. Co-Authored-By: Claude Opus 5.5 --- docs/release-notes/v3.15.0.md | 384 +++++++++++++++++++--------------- 1 file changed, 220 insertions(+), 164 deletions(-) diff --git a/docs/release-notes/v3.15.0.md b/docs/release-notes/v3.15.0.md index 00c4ea006..28212b0ba 100644 --- a/docs/release-notes/v3.15.0.md +++ b/docs/release-notes/v3.15.0.md @@ -1,164 +1,224 @@ # v3.15.0 77 commits since v3.14.0: 27 fix, 23 perf, 19 feat, 3 test, 2 ci, 2 docs (one of them -these notes), 1 chore. Nineteen feats, so minor. +these notes), 1 chore. Nineteen feats, so minor not patch. -The headline is node liveness (#2260): a repeater only counts as heard when the -evidence can only be that repeater. The rest is mostly memory and latency work on -the packet store, optional LoRa radio settings per node, and companion linking for -instances with user accounts. +The main change is how CoreScope decides that a repeater is alive. Until now a +repeater could stay green for days after it went silent, because traffic from +another repeater with the same short prefix was counted for it. That is fixed, and +it changes the status of some of your nodes after the upgrade. The rest of the +release is optional features (LoRa settings per node, observer airtime, companion +linking, smarter offline mails) and a lot of memory and speed work on the server. ## Read this before upgrading -These change what a running instance does without being asked. - -- **Last Heard and node status count only certain evidence** (#2261, closes #2260). - Before, a relay hop counted as "heard" for whichever node the resolver picked, so a - 1-byte hop shared by many repeaters kept a silent one "Active". Now only the node's - own packets and flood relay hops whose prefix matches no other repeater, and no - retired one, count. Adverts with an invalid signature no longer count. - - `last_heard` in `/api/nodes` and the node detail can now differ from `last_seen` - (it was a copy of it). - - A new status `uncertain`: a repeater or room with nothing certain in the silent - window whose shared prefix still shows up in paths. Grey badge, filter buttons on - the nodes list and the map, dimmed on the map like stale. - - Watched-node offline mails now fire for repeaters that were kept green by a - namesake prefix. - - Counts on one staging instance with 1,682 repeaters and rooms, on the day of the change: - 1,497 active, 117 uncertain, 68 stale. No before/after count of the same moment was - taken. Right after a restart the uncertain count starts lower, because the cold - load skips ambiguous relay hops (#1643); it grows with live traffic. -- **Node analytics use only certain data for Availability and Longest Silence** - (#2262). The activity chart is stacked into certain and uncertain. A silence that is - still running now counts. Rates for 30 d and All cover only the data in memory and - the page says from when (#2211): on one staging node the 30 d availability went from - 19.3% to 83.3%, because days before the in-memory span counted as silent. -- **`users.db` migrates to schema v6 at startup when user accounts are on** (#2174). - A v3.14.x binary refuses the file afterwards, so a rollback needs the backup from - before the upgrade (`backups/` next to `users.db`). -- **The ingestor starts a background backfill at every start** (#2133). It rewrites - `enc_*` channel hashes and empty scope names once a matching key is known, and runs - again when the key set grows. It creates three tables in the analyzer database - (`tx_rewrite_feed`, `key_backfill_state`, `key_backfill_done`). The PR gives no - measured cost on a large database. -- **Planner statistics refresh at most once per 24 h** (#2158). A restart within 24 h - of the last refresh runs no `ANALYZE`; a file `.planner-stats-refreshed` is - written next to the database. The first start after the upgrade still runs it. One - `ANALYZE` took 7 min 50 s on a large database. -- **Answers can be older than before, in exchange for latency:** - - `/api/nodes` answers identical queries from a shared cache for up to 15 s - (#2165). A geo-filter, blacklist or hidden-prefix change clears it. Measured on one - deployment: p95 165.4 to 114.4 ms, allocation 1,000 to 594 MB. - - The neighbor graph is only rebuilt in the background, every - `neighborGraph.cacheRecomputeIntervalSeconds` (default 300) (#2222). A rebuild took - 11.1 to 11.4 s on the request path before. - - Server caches past their 30 s TTL serve the old value while one refresh runs - (#2147), and `/api/perf` row counts are up to 60 s old (#2156). -- **Hop names and region placement change:** - - `iataCoords` gains 104 BE, NL, LU, DE and FR codes, and region narrowing now - applies only to the directly heard hop (#2188, closes #2098). - - `/api/nodes?region=` places a node by its zero-hop home regions first, then by the - fewest hops up to `regionFallback.maxHops` (default 3) (#2190, closes #1879). Every - node row gains `region_placement`. - - Transport-direct zero-hop adverts are now detected (#2234, closes #2225). - - The packet detail and the map route view infer 1-byte hops as one whole path; - hops can show as "unlisted" (#2245). - - An observer's own transmissions (`direction='tx'`) no longer count as receptions - in Heard By and the neighbor graph (#2220). Existing neighbor edges keep their - counts. -- **Favorites and home status cards use the role's thresholds** (#2215): a repeater - is degraded after 24 h and silent after 72 h there, as on its node page, instead of - 1 h and 24 h. -- **A geo-filter save on a single-file `config.json` mount now fails with 500** - (#2137). It used to write a copy that the next start discarded. A symlinked - `config.json` is now written through. -- **Hidden tabs stop polling `/api/stats`** (#2162); a visible tab refreshes it at most - once per 15 s. - -## Features - -- **LoRa radio settings per node** (#2209, #2210, closes #1268). A LoRa row on the node - detail (frequency, SF, BW, CR, preset) from the MeshCore map, and a badge plus - `?lora=differs` filter for nodes whose settings differ from nodes within - `rfParams.compareRadiusKm` (50 km). Off until `rfParams.nodesUrl` is set. -- **Observer TX airtime** (#2208, closes #661). A rolling 1 h TX airtime column on the - observer list and a tile on RF Health, with a warning above `dutyCycleWarnPct` - (10%). -- **Adaptive offline threshold for notifications** (#2262). Off by default: - `userManagement.notifications.adaptiveOffline.enabled`. It learns each watched - node's normal silence and mails about 30 minutes into an outage instead of after the - fixed window. -- **External node alerts feed** (#2178, closes #2177): - `userManagement.notifications.externalAlerts` turns an external JSON feed into - `node.external` mails for watched nodes. -- **Companion linking** (#2174, #2175, #2176). With user accounts on, a companion app - links to an account with a device token. `clientRxCoverage.requireLinkedCompanion` - ingests coverage only from linked companions; the account page lists them, with a - "My coverage" view. -- **Coverage gaps** (#2172). The Mobile RX coverage page shows where a companion drove - and received nothing, for instances with `clientRfSamples`. -- **Repeaters known only by a reported name** (#2239). Opt in with - `clientNodeNames.enabled`: names that companions report are stored, used in hop - resolution, and shown with an estimated position. -- **Scope Audit transport view** (#2167, #2169, #2171). Every repeater seen forwarding - a scope, next to the declared view, and an observer region filter for both. -- **Region filter**: `?region=` in the URL works on every page (#2256, closes #2255), - and `regionQuickPicks` adds configured groups to the control (#2170). -- **`?theme=dark|light`** URL parameter for one page load (#2251, closes #2236). -- **Postal as mail provider** for user accounts (#2154): - `userManagement.mail.provider: "postal"`. -- **Rolling hop index for the reach report** (#2214). Opt in with - `retention.hopIndexDays`. One reach scan went from 816 ms to 293 ms in the benchmark; - the index cost about 2 GB on a large database. -- **Docker: run the Go services as an unprivileged user** (#2125, #2235). Set - `RUN_AS_UID` / `RUN_AS_GID`; unset, the services run as root as before. - -## Fixes - -- **Server locks**: no SQL while holding the packet-store lock (#2147); region and - area sets are resolved before taking it (#2247, closes #2219); data races between - ingest and HTTP readers (#2246, closes #2237). -- **Pages**: late responses no longer overwrite a newer page visit on node analytics, - analytics and the account page (#2248, closes #2200, #2218, #2230); the packets node - filter keeps its pick (#2244, closes #2243) and grouped mode re-fetches on an - observer change (#2242, closes #2241); live packets respect the channel filter - (#2192, closes #2191). -- **Analytics**: Packets / Hour is readable on long histories (#2194, closes #2193); - Route Patterns applies `nodeBlacklist` and `hiddenNamePrefixes` (#2189, closes - #2184). -- **Route view**: "N hops" counts only true hops (#2240, closes #2206), and - payload-decoded sender and recipient rows get a badge (#2207). -- **Reach**: the reach page retries a busy (429) report (#2250). -- **Accounts**: a re-registration updates password and name in one statement (#2253, +### Some repeaters change from Active to Uncertain or Stale + +A repeater in a path is only known by the first 1, 2 or 3 bytes of its key. With +1-byte paths, dozens of repeaters share the same prefix, and CoreScope counted a +hop like that as "heard" for whichever repeater it picked. A repeater that had been +off for days could stay Active on traffic that was really another one. + +Since #2261 (closes #2260), a node only counts as heard on evidence that can only +be that node: its own packets, or a relay hop whose prefix no other repeater has. +Adverts with an invalid signature no longer count either. + +What you will see after the upgrade: + +- **A new grey status, Uncertain**, for a repeater or room that sent nothing certain + in its silent window, while its shared prefix still shows up in paths. It may be + that repeater, or another one with the same prefix. The nodes list and the map have + a filter button for it, and the map dims these nodes like stale ones. +- **Last Heard can be older than before** for repeaters that are only heard through a + shared prefix. +- **Offline mails for watched nodes** now go out for repeaters that a namesake kept + green before. The mail says when the node is uncertain rather than gone. +- **Node analytics** split the activity chart into certain and uncertain traffic, and + Availability and Longest Silence use only the certain part (#2262). + +On a staging instance with 1,682 repeaters and rooms, on the day of the change: 1,497 +Active, 117 Uncertain, 68 Stale. We did not count the same instance before the change. + +Right after a restart, fewer nodes show Uncertain and more show Stale. The first load +skips ambiguous relay hops (#1643), so they come back as live traffic arrives. Give +it a few hours before you judge a node. + +Nothing to configure. A node's own packets, such as its adverts, always count, so a +repeater that adverts within its silent window stays Active. If a node you know is +on the air shows Uncertain or Stale, its prefix is shared with another repeater and +its last advert is older than that window. + +### With user accounts on: back up `users.db` first + +`users.db` is migrated to schema version 6 at the first start (#2174). A v3.14 +binary refuses to open it after that. To roll back, you need the copy from before +the upgrade: the daily backups in `backups/` next to `users.db`, or your own. Without +user accounts nothing changes. + +### The ingestor fills in channel and scope names in the background + +When you add a channel key or a region key, packets stored before that keep showing +as an encrypted channel or an unknown scope. From this release the ingestor goes back +over those rows and fills in the name (#2133). It runs at every start and again when +keys are added, and creates three small tables to keep track. The server picks up +the result without a restart. There is no measured duration for a large database +yet. + +### Some answers are a few seconds or minutes older, in return for speed + +- The **nodes list** is shared between identical requests for up to 15 seconds + (#2165). A geo-filter or blacklist change clears it at once. +- The **neighbor graph** is rebuilt in the background every 5 minutes + (`neighborGraph.cacheRecomputeIntervalSeconds`, default 300) instead of while a + page waits (#2222). One rebuild took about 11 seconds before, on a page load. +- Server caches that have expired keep serving the old value while one refresh runs + (#2147). + +### Hop names and regions can shift + +Several changes make hop names and region membership more accurate, so some will +look different after the upgrade: + +- 104 airport codes for Belgium, the Netherlands, Luxembourg, Germany and France were + added to the region table (#2188, closes #2098). Narrowing a hop by the observer's + region is now only applied to the hop the observer heard directly. +- A node is placed in a region by where it is heard zero-hop first, then by the + fewest hops, up to 3 (#2190, closes #1879). Before, it went to every region whose + observers heard it. +- Zero-hop adverts on a transport-direct route are now recognised (#2234, closes + #2225). +- The packet detail and the map route view resolve 1-byte hops as one whole path + instead of hop by hop; a hop that matches no known node shows as "unlisted" + (#2245). +- An observer's own transmissions no longer count as something it received, in Heard + By and in the neighbor graph (#2220). + +### Docker: a geo-filter save on a single-file config mount now fails visibly + +If you mount `config.json` as a single file, saving the geo filter now returns an +error (#2137). Before, it wrote a copy inside the container that the next start threw +away. Mount the directory instead. A symlinked `config.json` is now written through. + +## LoRa settings and airtime + +- **LoRa radio settings per node** (#2209, closes #1268) - the node page shows the + frequency, spreading factor, bandwidth and coding rate the node reports on the + MeshCore map. Off until you set `rfParams.nodesUrl`. +- **Nodes with different settings than their neighbours** (#2210) - a LoRa badge and a + `?lora=differs` filter on the nodes list for a node whose settings differ from the + nodes within 50 km (`rfParams.compareRadiusKm`). Useful to find a repeater that was + set up on the wrong preset. +- **Observer TX airtime** (#2208, closes #661) - a "TX 1h" column on the observers list + and a tile on RF Health show how much of the last hour an observer spent + transmitting, with a warning above 10% (`dutyCycleWarnPct`). + +## Notifications and accounts + +These only apply with user accounts on. + +- **Smarter offline mails** (#2262) - off by default + (`userManagement.notifications.adaptiveOffline`). It learns how long each watched + node is normally silent, and mails about 30 minutes into a real outage instead of + after the fixed window. +- **Alerts from an external feed** (#2178, closes #2177) - point + `userManagement.notifications.externalAlerts` at a JSON feed, and watched nodes in + it get a mail. +- **Link a companion to your account** (#2174, #2175, #2176) - a companion app logs in + with a device token. With `clientRxCoverage.requireLinkedCompanion`, coverage is only + taken from linked companions, and the account page lists your companions with a "My + coverage" view. +- **Postal as mail provider** (#2154) - `userManagement.mail.provider: "postal"`, next + to Brevo. +- Re-registering with a pending account updates password and name together (#2253, closes #2228). -- **Ingestor**: an interrupted background migration is recorded as cancelled and - retried (#2161); the background chunk loader sets `observer_iata` (#2185). - -## Performance - -- **Request path** (#2146): the reach scan is driven by the observations time window - (#2155, 14.11 s to 1.93 s on one query); `/paths` reads in batches (#2157, #2224, - closes #2223); battery history uses the primary key (#2156); per-region scope counts - use the `first_seen` index (#2166, closes #2146: 1 h window 3.08 s to 16 ms); the - neighbor graph leaves the request path (#2222, closes #2221). -- **Packet store memory** (#2181 to #2183, #2186, #2187, #2195 to #2199, #2201, #2163, - closes #2148). One deployment: startup to ready 31 s to 21 s (#2181), subpath index - RSS 948 to 767 MB (#2182), heap 581 to 535 MB from shared observer strings (#2183). -- **Ingestor**: live ingestion runs between retention batches (#2153, closes #2151). -- **Opt-in `analytics.pauseWhenIdle`** skips recomputes nobody reads (#2164): 459.7 to - 53.4 MB allocation in the measurement. + +## Coverage and companions + +- **Where a companion drove and heard nothing** (#2172) - the Mobile RX coverage page + shows gaps next to receptions, for instances with `clientRfSamples`. +- **Repeaters known only by the name a companion reported** (#2239) - opt in with + `clientNodeNames.enabled`. Names that companions report are used in hop resolution + and shown on the map at an estimated position. +- **Faster reach report** (#2214) - opt in with `retention.hopIndexDays`. In the + benchmark one reach scan went from 816 ms to 293 ms; the index took about 2 GB on a + large database. The reach page now retries when the server is busy instead of + showing an error (#2250). + +## Scopes and regions + +- **Scope Audit transport view** (#2167, #2169) - next to what repeaters declare, a + view of every repeater actually seen forwarding a scope. +- **Region filter on Scope Audit** (#2171), for both views. +- **`?region=` in a link works on every page** (#2256, closes #2255) - not only on + packets. Handy for links from a regional website or a Discord post. +- **Region quick picks** (#2170) - `regionQuickPicks` adds groups of regions to the + region control, for example a country. + +## Pages and packets + +- **Pages no longer show an answer for an earlier visit** (#2248, closes #2200, #2218, + #2230) - when you clicked on quickly, a slow answer could replace the page you were + on. Node analytics, analytics and the account page are fixed. +- **Packets** - the node filter keeps your pick on a slow click (#2244, closes #2243), + grouped mode reloads when you change the observer filter (#2242, closes #2241), and + live packets respect the channel filter (#2192, closes #2191). +- **Analytics** - Packets / Hour is readable on long histories (#2194, closes #2193), + and Route Patterns hides blacklisted and hidden nodes (#2189, closes #2184). +- **Node analytics say which period they cover** (#2211) - for 30 days and All, the + rates are computed over the packets in memory, and the page says from when. On one + staging node the 30-day availability went from 19.3% to 83.3%, because days outside + memory counted as silent before. +- **Route view** - "N hops" counts only real hops (#2240, closes #2206), and the sender + and recipient decoded from the payload are marked as such (#2207). +- **Favorites and home cards use the role's thresholds** (#2215) - a repeater is shown + as degraded after 24 hours and silent after 72, the same as on its node page, + instead of after 1 and 24. +- **The region filter matches packets loaded in the background** (#2185) - with + `hotStartupHours`, packets loaded after startup now carry the observer region. +- **`?theme=dark` or `?theme=light`** in a link sets the theme for that page load + (#2251, closes #2236). + +## Faster and lighter + +A large part of this release is work on the request path (#2146) and the packet +store's memory. What you notice: + +- Startup to ready went from 31 to 21 seconds on one instance (#2181). +- The packet store uses less memory: for example 948 to 767 MB for the route + patterns index (#2182) and 581 to 535 MB heap from sharing observer strings (#2183). +- The nodes list: p95 from 165 to 114 ms (#2165). +- Scope statistics for the last hour: 3.08 seconds to 16 ms (#2166). +- One reach query: 14.1 to 1.9 seconds (#2155). +- The ingestor lets live packets through between retention batches (#2153, closes + #2151), and runs the database statistics refresh at most once a day, where one run + took almost 8 minutes (#2158). +- Hidden browser tabs stop polling the server for the navigation counters (#2162). +- Opt-in `analytics.pauseWhenIdle` skips analytics nobody looks at (#2164). + +The other pull requests in this group are #2147, #2156, #2157, #2163, #2186, #2187, +#2195 to #2199, #2201, #2222, #2224, #2246, #2247. + +## Docker and ingestor + +- **Run the services as a normal user** (#2125, #2235) - set `RUN_AS_UID` and + `RUN_AS_GID`. Without them the services run as root, as before. +- An interrupted background migration in the ingestor is retried at the next start + instead of being marked as failed (#2161). ## Tests, CI and docs -- Frontend coverage from every E2E suite, merged across the shards (#2204, closes - #2203); README badges published to a `badges` branch (#2202, closes #2136). -- Local validation independent of the runner environment (#2160, closes #2159); two - flaky E2E checks settled (#2179, #2180). -- README corrected and the optional features listed (#2205); `SECURITY.md` and OSS - Scanner build files (#2257). +- Frontend coverage now counts every E2E suite (#2204, closes #2203), and the README + badges update again (#2202, closes #2136). +- Local test runs no longer depend on the CI environment (#2160, closes #2159), and two + flaky E2E checks are stable (#2179, #2180). +- The README is corrected and lists the optional features (#2205). `SECURITY.md` + explains how to report a vulnerability (#2257). ## Issues closed +29 issues, from the 76 pull requests in this range. Collected from each pull +request's closing references rather than from commit text. + - #661 feat: airtime utilization estimates per observer - #1268 feat: enrich nodes with LoRa RF params from map.meshcore.io - #1879 bug: /api/nodes region filter attributes by observer, not repeater home region @@ -189,28 +249,24 @@ These change what a running instance does without being asked. - #2255 ?region= in the URL only works on the packets page - #2260 Node liveness counts ambiguous 1-byte hops: a silent repeater shows "Active" -Each of these is listed as closing by one of the merged pull requests. - ## Contributors Pull request authors: @efiten (33), @A13xB0 (24), @nullrouten0 (6), @n30nex (4), -@anieto (3), @CoderNemesis26 (2), @hermes-gadget (2), @SaarMesh-Bot (2). The -maintainer's own share is 33 of 76 pull requests. +@anieto (3), @CoderNemesis26 (2), @hermes-gadget (2), @SaarMesh-Bot (2). -Issue reporters: @efiten (13), @n30nex (8), @anieto (2), @A13xB0 (1), @dborup (1), -@hermes-gadget (1), @Kpa-clawbot (1), @n1ghtBl00d (1), @SaarMesh-Bot (1). +No co-authors other than tooling are credited in the commits. -No co-authors other than the pull request authors themselves and tooling are credited -in the commits. +Issues in this release were reported by @efiten (13), @n30nex (8), @anieto (2), +@A13xB0 (1), @dborup (1), @hermes-gadget (1), @Kpa-clawbot (1), @n1ghtBl00d (1), +@SaarMesh-Bot (1). ## Upgrade notes -- No manual migration step. The ingestor creates the backfill tables (#2133) and the - empty `client_node_names` table (#2239) at startup. -- With user accounts on, `users.db` goes to schema v6 at startup (#2174); keep the - pre-upgrade backup if you may roll back. -- Opt-in, off by default: `rfParams.nodesUrl`, `clientNodeNames.enabled`, - `retention.hopIndexDays`, `analytics.pauseWhenIdle`, `regionQuickPicks`, - `userManagement.notifications.adaptiveOffline`, `.externalAlerts`, - `clientRxCoverage.requireLinkedCompanion`, `userManagement.mail.provider: "postal"`, - and `RUN_AS_UID` / `RUN_AS_GID` for Docker. +- No manual migration step. +- With user accounts on, keep a copy of `users.db` from before the upgrade (see above). +- Everything new is off until you set it: `rfParams.nodesUrl`, + `clientNodeNames.enabled`, `retention.hopIndexDays`, `analytics.pauseWhenIdle`, + `regionQuickPicks`, `userManagement.notifications.adaptiveOffline` and + `.externalAlerts`, `clientRxCoverage.requireLinkedCompanion`, + `userManagement.mail.provider: "postal"`, and `RUN_AS_UID` / `RUN_AS_GID` for + Docker. The example values are in `config.example.json`.