diff --git a/CHANGELOG.md b/CHANGELOG.md index e22132db6..da9212568 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,19 @@ ## [Unreleased] +## [3.15.0] - 2026-10-22 + +See [docs/release-notes/v3.15.0.md](docs/release-notes/v3.15.0.md) for the full notes. 77 commits since v3.14.0: 27 fix, 23 perf, 19 feat, 3 test, 2 ci, 2 docs, 1 chore. + +### Highlights +- **Last Heard and node status count only certain evidence** (#2261, #2262, closes #2260) - a shared 1-byte relay hop no longer keeps a silent repeater "Active". New status `uncertain`. **Operator awareness required**: statuses, `last_heard` and offline mails change after the upgrade. +- **`users.db` migrates to schema v6 with user accounts on** (#2174) - a v3.14.x binary refuses it afterwards, so keep the pre-upgrade backup for a rollback. +- **The ingestor runs a key backfill at every start** (#2133) and refreshes planner statistics at most once per 24 h (#2158). +- **Optional LoRa radio settings per node** (#2209, #2210), observer TX airtime (#2208), companion linking (#2174 to #2176), adaptive offline mails (#2262). +- **Packet store memory and request latency** (#2146 and 23 perf commits) - e.g. startup to ready 31 s to 21 s (#2181), `/api/nodes` p95 165.4 to 114.4 ms (#2165). + +No manual migration step. + ## [3.14.0] - 2026-10-08 See [docs/release-notes/v3.14.0.md](docs/release-notes/v3.14.0.md) for the full notes. 23 commits since v3.13.1: 13 fix, 8 feat, 1 ci, 1 docs. diff --git a/docs/release-notes/v3.15.0.md b/docs/release-notes/v3.15.0.md new file mode 100644 index 000000000..28212b0ba --- /dev/null +++ b/docs/release-notes/v3.15.0.md @@ -0,0 +1,272 @@ +# v3.15.0 + +77 commits since v3.14.0: 27 fix, 23 perf, 19 feat, 3 test, 2 ci, 2 docs (one of them +these notes), 1 chore. Nineteen feats, so minor not patch. + +The main change is how CoreScope decides that a repeater is alive. Until now a +repeater could stay green for days after it went silent, because traffic from +another repeater with the same short prefix was counted for it. That is fixed, and +it changes the status of some of your nodes after the upgrade. The rest of the +release is optional features (LoRa settings per node, observer airtime, companion +linking, smarter offline mails) and a lot of memory and speed work on the server. + +## Read this before upgrading + +### Some repeaters change from Active to Uncertain or Stale + +A repeater in a path is only known by the first 1, 2 or 3 bytes of its key. With +1-byte paths, dozens of repeaters share the same prefix, and CoreScope counted a +hop like that as "heard" for whichever repeater it picked. A repeater that had been +off for days could stay Active on traffic that was really another one. + +Since #2261 (closes #2260), a node only counts as heard on evidence that can only +be that node: its own packets, or a relay hop whose prefix no other repeater has. +Adverts with an invalid signature no longer count either. + +What you will see after the upgrade: + +- **A new grey status, Uncertain**, for a repeater or room that sent nothing certain + in its silent window, while its shared prefix still shows up in paths. It may be + that repeater, or another one with the same prefix. The nodes list and the map have + a filter button for it, and the map dims these nodes like stale ones. +- **Last Heard can be older than before** for repeaters that are only heard through a + shared prefix. +- **Offline mails for watched nodes** now go out for repeaters that a namesake kept + green before. The mail says when the node is uncertain rather than gone. +- **Node analytics** split the activity chart into certain and uncertain traffic, and + Availability and Longest Silence use only the certain part (#2262). + +On a staging instance with 1,682 repeaters and rooms, on the day of the change: 1,497 +Active, 117 Uncertain, 68 Stale. We did not count the same instance before the change. + +Right after a restart, fewer nodes show Uncertain and more show Stale. The first load +skips ambiguous relay hops (#1643), so they come back as live traffic arrives. Give +it a few hours before you judge a node. + +Nothing to configure. A node's own packets, such as its adverts, always count, so a +repeater that adverts within its silent window stays Active. If a node you know is +on the air shows Uncertain or Stale, its prefix is shared with another repeater and +its last advert is older than that window. + +### With user accounts on: back up `users.db` first + +`users.db` is migrated to schema version 6 at the first start (#2174). A v3.14 +binary refuses to open it after that. To roll back, you need the copy from before +the upgrade: the daily backups in `backups/` next to `users.db`, or your own. Without +user accounts nothing changes. + +### The ingestor fills in channel and scope names in the background + +When you add a channel key or a region key, packets stored before that keep showing +as an encrypted channel or an unknown scope. From this release the ingestor goes back +over those rows and fills in the name (#2133). It runs at every start and again when +keys are added, and creates three small tables to keep track. The server picks up +the result without a restart. There is no measured duration for a large database +yet. + +### Some answers are a few seconds or minutes older, in return for speed + +- The **nodes list** is shared between identical requests for up to 15 seconds + (#2165). A geo-filter or blacklist change clears it at once. +- The **neighbor graph** is rebuilt in the background every 5 minutes + (`neighborGraph.cacheRecomputeIntervalSeconds`, default 300) instead of while a + page waits (#2222). One rebuild took about 11 seconds before, on a page load. +- Server caches that have expired keep serving the old value while one refresh runs + (#2147). + +### Hop names and regions can shift + +Several changes make hop names and region membership more accurate, so some will +look different after the upgrade: + +- 104 airport codes for Belgium, the Netherlands, Luxembourg, Germany and France were + added to the region table (#2188, closes #2098). Narrowing a hop by the observer's + region is now only applied to the hop the observer heard directly. +- A node is placed in a region by where it is heard zero-hop first, then by the + fewest hops, up to 3 (#2190, closes #1879). Before, it went to every region whose + observers heard it. +- Zero-hop adverts on a transport-direct route are now recognised (#2234, closes + #2225). +- The packet detail and the map route view resolve 1-byte hops as one whole path + instead of hop by hop; a hop that matches no known node shows as "unlisted" + (#2245). +- An observer's own transmissions no longer count as something it received, in Heard + By and in the neighbor graph (#2220). + +### Docker: a geo-filter save on a single-file config mount now fails visibly + +If you mount `config.json` as a single file, saving the geo filter now returns an +error (#2137). Before, it wrote a copy inside the container that the next start threw +away. Mount the directory instead. A symlinked `config.json` is now written through. + +## LoRa settings and airtime + +- **LoRa radio settings per node** (#2209, closes #1268) - the node page shows the + frequency, spreading factor, bandwidth and coding rate the node reports on the + MeshCore map. Off until you set `rfParams.nodesUrl`. +- **Nodes with different settings than their neighbours** (#2210) - a LoRa badge and a + `?lora=differs` filter on the nodes list for a node whose settings differ from the + nodes within 50 km (`rfParams.compareRadiusKm`). Useful to find a repeater that was + set up on the wrong preset. +- **Observer TX airtime** (#2208, closes #661) - a "TX 1h" column on the observers list + and a tile on RF Health show how much of the last hour an observer spent + transmitting, with a warning above 10% (`dutyCycleWarnPct`). + +## Notifications and accounts + +These only apply with user accounts on. + +- **Smarter offline mails** (#2262) - off by default + (`userManagement.notifications.adaptiveOffline`). It learns how long each watched + node is normally silent, and mails about 30 minutes into a real outage instead of + after the fixed window. +- **Alerts from an external feed** (#2178, closes #2177) - point + `userManagement.notifications.externalAlerts` at a JSON feed, and watched nodes in + it get a mail. +- **Link a companion to your account** (#2174, #2175, #2176) - a companion app logs in + with a device token. With `clientRxCoverage.requireLinkedCompanion`, coverage is only + taken from linked companions, and the account page lists your companions with a "My + coverage" view. +- **Postal as mail provider** (#2154) - `userManagement.mail.provider: "postal"`, next + to Brevo. +- Re-registering with a pending account updates password and name together (#2253, + closes #2228). + +## Coverage and companions + +- **Where a companion drove and heard nothing** (#2172) - the Mobile RX coverage page + shows gaps next to receptions, for instances with `clientRfSamples`. +- **Repeaters known only by the name a companion reported** (#2239) - opt in with + `clientNodeNames.enabled`. Names that companions report are used in hop resolution + and shown on the map at an estimated position. +- **Faster reach report** (#2214) - opt in with `retention.hopIndexDays`. In the + benchmark one reach scan went from 816 ms to 293 ms; the index took about 2 GB on a + large database. The reach page now retries when the server is busy instead of + showing an error (#2250). + +## Scopes and regions + +- **Scope Audit transport view** (#2167, #2169) - next to what repeaters declare, a + view of every repeater actually seen forwarding a scope. +- **Region filter on Scope Audit** (#2171), for both views. +- **`?region=` in a link works on every page** (#2256, closes #2255) - not only on + packets. Handy for links from a regional website or a Discord post. +- **Region quick picks** (#2170) - `regionQuickPicks` adds groups of regions to the + region control, for example a country. + +## Pages and packets + +- **Pages no longer show an answer for an earlier visit** (#2248, closes #2200, #2218, + #2230) - when you clicked on quickly, a slow answer could replace the page you were + on. Node analytics, analytics and the account page are fixed. +- **Packets** - the node filter keeps your pick on a slow click (#2244, closes #2243), + grouped mode reloads when you change the observer filter (#2242, closes #2241), and + live packets respect the channel filter (#2192, closes #2191). +- **Analytics** - Packets / Hour is readable on long histories (#2194, closes #2193), + and Route Patterns hides blacklisted and hidden nodes (#2189, closes #2184). +- **Node analytics say which period they cover** (#2211) - for 30 days and All, the + rates are computed over the packets in memory, and the page says from when. On one + staging node the 30-day availability went from 19.3% to 83.3%, because days outside + memory counted as silent before. +- **Route view** - "N hops" counts only real hops (#2240, closes #2206), and the sender + and recipient decoded from the payload are marked as such (#2207). +- **Favorites and home cards use the role's thresholds** (#2215) - a repeater is shown + as degraded after 24 hours and silent after 72, the same as on its node page, + instead of after 1 and 24. +- **The region filter matches packets loaded in the background** (#2185) - with + `hotStartupHours`, packets loaded after startup now carry the observer region. +- **`?theme=dark` or `?theme=light`** in a link sets the theme for that page load + (#2251, closes #2236). + +## Faster and lighter + +A large part of this release is work on the request path (#2146) and the packet +store's memory. What you notice: + +- Startup to ready went from 31 to 21 seconds on one instance (#2181). +- The packet store uses less memory: for example 948 to 767 MB for the route + patterns index (#2182) and 581 to 535 MB heap from sharing observer strings (#2183). +- The nodes list: p95 from 165 to 114 ms (#2165). +- Scope statistics for the last hour: 3.08 seconds to 16 ms (#2166). +- One reach query: 14.1 to 1.9 seconds (#2155). +- The ingestor lets live packets through between retention batches (#2153, closes + #2151), and runs the database statistics refresh at most once a day, where one run + took almost 8 minutes (#2158). +- Hidden browser tabs stop polling the server for the navigation counters (#2162). +- Opt-in `analytics.pauseWhenIdle` skips analytics nobody looks at (#2164). + +The other pull requests in this group are #2147, #2156, #2157, #2163, #2186, #2187, +#2195 to #2199, #2201, #2222, #2224, #2246, #2247. + +## Docker and ingestor + +- **Run the services as a normal user** (#2125, #2235) - set `RUN_AS_UID` and + `RUN_AS_GID`. Without them the services run as root, as before. +- An interrupted background migration in the ingestor is retried at the next start + instead of being marked as failed (#2161). + +## Tests, CI and docs + +- Frontend coverage now counts every E2E suite (#2204, closes #2203), and the README + badges update again (#2202, closes #2136). +- Local test runs no longer depend on the CI environment (#2160, closes #2159), and two + flaky E2E checks are stable (#2179, #2180). +- The README is corrected and lists the optional features (#2205). `SECURITY.md` + explains how to report a vulnerability (#2257). + +## Issues closed + +29 issues, from the 76 pull requests in this range. Collected from each pull +request's closing references rather than from commit text. + +- #661 feat: airtime utilization estimates per observer +- #1268 feat: enrich nodes with LoRa RF params from map.meshcore.io +- #1879 bug: /api/nodes region filter attributes by observer, not repeater home region +- #2098 iata-coords: none of the 42 observers' codes are in the table +- #2136 README badges stale since June: BADGE_PUSH_TOKEN returns 401 +- #2146 Request-path latency grows with DB size +- #2148 perf: avoid allocation-class padding in retained observations +- #2151 perf: let live ingestion run between retention batches +- #2159 test: local validation assumes Actions environment and POSIX file modes +- #2177 Notifications: optional external node alerts feed +- #2184 nodeBlacklist is not applied to Route Patterns +- #2191 Packets page: live WebSocket packets ignore the channel filter +- #2193 Analytics: Packets / Hour chart unreadable on long-retention instances +- #2200 Analytics: older requests can overwrite newer filters or a later page visit +- #2203 Frontend coverage counts only 2 of the E2E suites +- #2206 Route view: DM sender/recipient rows drawn as ordinary hops +- #2218 bug(node analytics): delayed responses overwrite the page after navigation +- #2219 perf: cold region and area lookups still hold the packet-store lock +- #2221 perf: /api/nodes/{pubkey}/neighbors rebuilds the neighbor graph on the request path (~11 s) +- #2223 perf: /api/nodes/{pubkey}/paths runs two queries per transmission +- #2225 bug: zero-hop detection reads the path byte at the wrong offset for transport-direct adverts +- #2228 bug(accounts): make pending registration updates atomic and status-conditional +- #2230 bug(account): delayed session initialization replaces the page after navigation +- #2236 Feature Request: Dark Mode set via URL parameter +- #2237 server: data races between ingest and HTTP readers +- #2241 Packets: observer filter only takes effect after a page reload (grouped mode) +- #2243 Packets: picking a node suggestion sometimes does nothing +- #2255 ?region= in the URL only works on the packets page +- #2260 Node liveness counts ambiguous 1-byte hops: a silent repeater shows "Active" + +## Contributors + +Pull request authors: @efiten (33), @A13xB0 (24), @nullrouten0 (6), @n30nex (4), +@anieto (3), @CoderNemesis26 (2), @hermes-gadget (2), @SaarMesh-Bot (2). + +No co-authors other than tooling are credited in the commits. + +Issues in this release were reported by @efiten (13), @n30nex (8), @anieto (2), +@A13xB0 (1), @dborup (1), @hermes-gadget (1), @Kpa-clawbot (1), @n1ghtBl00d (1), +@SaarMesh-Bot (1). + +## Upgrade notes + +- No manual migration step. +- With user accounts on, keep a copy of `users.db` from before the upgrade (see above). +- Everything new is off until you set it: `rfParams.nodesUrl`, + `clientNodeNames.enabled`, `retention.hopIndexDays`, `analytics.pauseWhenIdle`, + `regionQuickPicks`, `userManagement.notifications.adaptiveOffline` and + `.externalAlerts`, `clientRxCoverage.requireLinkedCompanion`, + `userManagement.mail.provider: "postal"`, and `RUN_AS_UID` / `RUN_AS_GID` for + Docker. The example values are in `config.example.json`.