[efiten]

Fixes #2260 (part 1 of 2: liveness; the adaptive offline threshold follows in a separate PR).

Problem

A repeater silent since 2026-10-18 showed Active — Last relayed 4m ago and Last Heard 36m, and its offline mail never fired. Every hit after that date was the 1-byte hop 67, shared by nine nodes:

• the live resolver's affinity tiebreak (resolvePathForObs) put those transmissions in byNode, which fed stats.lastHeard and LastHeardMap;
• last_relayed folds in the 1-byte prefix bucket on purpose (feat: repeater liveness — detect if repeater is actively relaying traffic #662), and drove the status in roles.js and the notify offline check.

Fix

The store keeps certainHeard per node: the newest transmission that can only be that node.

Evidence	Counts?
Own packet (pubKey, srcPubKey)	✅
Flood relay hop whose prefix, at the packet's path hash size, matches exactly one path-capable node	✅
1-byte hop shared with other repeaters	❌
Direct-route path (hops still to come, Mesh::removeSelfFromPath)	❌
TRACE (path bytes are SNR)	❌
Destination (destPubKey)	❌

A multibyte repeater legitimately appears as a 1-byte hop (the originator picks the hash size, Mesh::routeRecvPacket appends at that size), so only uniqueness decides. The rule is applied in every load path through indexResolvedPathHops and the new indexColdLoadRelayHops: Load, loadChunk, scanAndMergeChunk, live ingest, late observations.

Liveness reads only that:

• /api/nodes and the node detail: last_heard is the newer of last_seen and the certain last heard. Before, it was a plain copy of last_seen.
• health / bulk-health stats.lastHeard.
• LastHeardMap → notify offline check. The separate relay input (lastRelayed, relayTimes, notifyInput.Relayed) is removed.
• roles.js getEffectiveHeardMs: newest known timestamp, without last_relayed. The side pane no longer says "Last relayed"; the Last Relayed row stays (analytics) with an updated tooltip.
• New status uncertain (repeaters and rooms): nothing certain within the silent window, but the shared prefix (last_relayed) still in paths. Grey badge with explanation, an Uncertain filter on the nodes list and the map, dimmed like stale. The offline mail for such a node says only its shared prefix was still seen.
• A prefix also held by a path-capable node in inactive_nodes is not unique; adverts with signatureValid: false are no evidence; /api/nodes/search reports the same last_heard.

Attribution for analytics (byNode, packet counts, last_relayed, relay counters, usefulness) is unchanged.

Performance

Hot path: noteCertainRelays runs per observation on ingest under the write lock: two hex bytes of header parsing plus one prefix-map lookup per resolved hop.

BenchmarkNoteCertainRelays-16   10307640   116.2 ns/op   0 B/op   0 allocs/op   (8-hop 2-byte path, 2,000 repeaters)
BenchmarkLastHeardMap-16            1902   0.77 ms/op                            (5,000 nodes; was 3.7 ms scanning byNode)

Memory: one string per node ever certainly heard, dropped together with the node's byNode list on eviction.

Tests

• certain_heard_loadpaths_test.go: every load path (Load, loadChunk, LoadChunked — each with and without resolved_path — live ingest, late observations) feeds certainHeard; each of the 8 call sites verified red with a nil prefix map.
• certain_heard_test.go: hop width per route/payload type; retired namesake prefix; failed signature; uncertain offline mail; shared vs unique 1-byte vs 2-byte hops; direct/TRACE/nil prefix map; cold-load path; sender vs destination; eviction; lastHeardValue; /api/nodes, detail and health agree.
• Regression for Node liveness (Last Heard, Active status, Notify-me) counts ambiguous 1-byte hops: a silent repeater shows "Active, last relayed 4m ago" #2260: last certain packet on day X, 20 shared-prefix hops after it → LastHeardMap stays at X and node.offline goes bad after 72 h. Verified red with the uniqueness check mutated out.
• Notify: TestEvaluateCertainHeard replaces the relay-input test.
• Frontend: last_relayed alone no longer keeps a repeater or room active; newest timestamp wins (an old _liveSeen can't hide a fresh certain relay).
• cmd/server go test ./... and test-all.sh pass locally, also on a trial merge into a downstream fork with 457 extra commits.

Validation

Ran the server on the freshened e2e fixture: list, detail and health report the same last heard; e.g. a repeater whose last_relayed is 3 min newer than anything certain keeps its certain time. No browser screenshot: Playwright isn't installed on this machine; CI's E2E covers the pages.

Not changed

• The ingestor's relay touch of nodes.last_seen (bug(server): touchRelayLastSeen has been a no-op since the mode=ro refactor — nodes.last_seen never reflects relay activity #1854) resolves a shared hop only when exactly one candidate is adjacent in the neighbor graph; it did not fire in this case. Left as is.
• The lastHeard= list filter still filters last_seen in SQL.

🤖 Generated with Claude Code

[efiten]

Review notes and a measurement on live data

Reviewed commit 1b40d8ac against upstream/master and the firmware (src/Mesh.cpp, src/Packet.h, src/helpers/CommonCLI.cpp). The firmware basis holds: hash size (path_len>>6)+1, flood routes 0/1 vs direct 2/3, a direct path lists the hops still to come (removeSelfFromPath), and a flood relay appends at the originator's size. Tests and the benchmark were not re-run for this review.

1. Nodes with no certain relay fall back to adverts only

After this change a repeater is heard only through its own packets or through hops whose prefix is unique. The firmware default flood.advert.interval is 47 h (examples/simple_repeater/MyMesh.cpp:905) and infraSilentHours is 72, so one missed advert puts such a node at about 94 h of silence.

Measured on a live instance (1678 repeaters and rooms, flood observations of the last 72 h, TRACE excluded, "unique" taken as unique among repeater and room rows in nodes):

Hop width	Share of observations	Hops with a unique prefix
1 byte	25.3 %	0 % (none of the 254 1-byte prefixes is held by exactly one node)
2 bytes	72.6 %	97.2 %
3 bytes	2.1 %	94.8 %

Newest certain evidence per node	Nodes
certain relay < 24 h	1091 (65.0 %)
certain relay 24 to 72 h only	162 (9.7 %)
no certain relay, advert < 24 h	146 (8.7 %)
no certain relay, advert 24 to 72 h	95 (5.7 %)
no certain relay, advert 72 h to 7 d (offline under this PR)	184 (11.0 %)

Multibyte traffic covers most nodes. The 184 are an upper bound for false offline: the data cannot separate dead nodes from nodes that only relay 1-byte traffic. The 241 nodes in the two middle rows depend on adverts alone, so a missed advert takes them offline. This is one 72 h snapshot, network wide.

2. Uniqueness is decided against the nodes table only

certain_heard.go:78, store.go:7371. A repeater that shares a prefix but has never been heard advertising, or was removed by retention while it still relays, is not seen by the check, so its relays count for the other node. The check runs once at ingest and is not re-evaluated when a colliding node appears later.

3. No test covers the wiring in the five load paths

TestNodeEndpointsLastHeardIsCertain seeds certainHeard directly (certain_heard_test.go:196); the other tests call indexResolvedPathHops / indexColdLoadRelayHops themselves. Passing a nil prefix map in Load (store.go:1099/1112), loadChunk (1589/1594), scanAndMergeChunk (chunked_load.go:638/650), live ingest (store.go:3230) or late observations (3544) would leave every test green. One test per path through the real Load or ingest, asserting LastHeardMap, would close that.

Checked without finding a problem: locking (writes under s.mu.Lock, LastHeardMap under RLock), eviction (certainHeard deleted with byNode at store.go:5457), noteCertainHeard keeps the newest time, TRACE and direct routes excluded.

Question

Merge order: this PR on its own moves the 241 advert-only nodes onto a single missed advert. Should #2261 and #2262 land together (with adaptiveOffline still off by default that does not help these nodes), or should the fixed window for nodes without any certain relay stay on the previous evidence until there is a better signal?

[efiten]

Review feedback addressed (commit f7a809d9, amended)

1. Side pane / node page showed the first timestamp present, not the newest. n.last_heard || stats.lastHeard could show an older cached value. Both places and getStatusInfo now take the newest via getEffectiveHeardMs, the same timestamp the status uses. Test: getStatusInfo shows the newest of _lastHeard and last_heard.
2. Adverts whose signature failed counted as certain. indexByNode now skips signatureValid: false (absent = not checked, still counts). signatureValid is taken off decodedCacheDroppedKeys so the cached decode keeps it (one bool per advert). Test: TestIndexByNodeFailedSignatureIsNotCertain.
3. /api/nodes/search still returned last_heard = last_seen. It now folds in the certain last heard like /api/nodes and the detail. Test: TestNodeSearchLastHeardIsCertain.

Not changed, with reasons:

• Ingest-time vs current prefix map (certainHeard vs analytics in Nodes table has dead 'Regions' column — always shows — #26): one rule at two moments. Recording a per-(tx, node) flag to make them identical costs a map entry per byNode pair, tens of MB on large instances. They only diverge when a new repeater with the same prefix appears. Documented on certainFor.
• A shared-contact advert re-sent by another device usually dedups by hash with the original. When the original was already evicted, nodes.last_seen has the same issue today.

cmd/server go test ./... and test-all.sh pass, also on a trial merge into this fork's master.

[efiten]

Follow-up on the review notes above

Read the delta 1b40d8ac..f7a809d9; tests were not re-run, CI is green.

The three changes in f7a809d9 (failed signatures, /api/nodes/search, newest timestamp in the side pane) each come with a test and read correctly.

Still open from the review notes:

1. Advert-only nodes. No change for the 241 nodes that depend on adverts alone.
2. Uniqueness against nodes only. The "Not changed" note covers when the check runs (ingest vs now). It does not cover a colliding repeater that never advertised or was removed by retention while it still relays.
3. Load-path wiring. Still no test that goes through the real Load, loadChunk, scanAndMergeChunk, live ingest or late observations.

Question

For points 1 to 3, which of these do you prefer: address them in this PR, land this PR as is and track them in #2260, or land this PR only together with #2262?

[efiten]

Review notes addressed (commit b5556bdc, amended)

Answers to the question: everything is addressed in this PR; it lands first, #2262 right after.

1. Advert-only nodes (241 / 184). New third status uncertain for repeaters and rooms: nothing that can only be this node within the silent window, but its hop prefix (shared with other repeaters, last_relayed) still appears in paths. It is neither proof of life nor of an outage, so:
   • UI: grey "Uncertain" badge with the explanation "Last certainly heard X — its shared hop prefix was last in a path Y, possibly another node"; an Uncertain button in the status filter on the nodes list and the map; map/live dim it like stale; map legend counts it separately.
   • Notify: still node.offline (an owner should know), but the mail line says why: "offline (nothing that can only be this node; its hop prefix, shared with other repeaters, was last in a path …, possibly another node)". The shared-prefix time only words the mail; it never decides the state.
   • getNodeStatus(role, ms) (legacy form) never returns uncertain. Tests for each case.
2. Uniqueness against nodes only. uniqueHopPrefix now also rejects a prefix held by a path-capable node in inactive_nodes (moved there by retention but may still relay): prefixMap.retired, loaded with the node cache. A repeater that never advertised is unknown to both and cannot be ruled out; that is stated on the function. Test: TestRetiredNamesakeKeepsHopUncertain.
3. Load-path wiring. TestCertainHeardFedByEveryLoadPath goes through the real Load (resolved_path and path_json), loadChunk (both), LoadChunked/scanAndMergeChunk (both), IngestNewFromDB and IngestNewObservations, asserting LastHeardMap. Verified: passing nil as prefix map at each of the 8 call sites in turn fails exactly that path's subtest.

cmd/server go test ./... and test-all.sh pass.

[efiten]

Review of b5556bdc: the retired-prefix rule removes most certain relays on live data

Read the delta f7a809d9..b5556bdc; tests were not re-run. The uncertain status, the mail wording and the load-path test read correctly. One problem blocks the merge.

inactive_nodes still holds nodes that came back

MoveStaleNodes copies rows into inactive_nodes (cmd/ingestor/db.go:2160), and nothing deletes a row there when the node is heard again: there is no DELETE FROM inactive_nodes in cmd/. uniqueHopPrefix rejects any prefix in pm.retired, including one that only the node itself put there. Such a node gets no certain relay at all, at any width.

Measured on a live instance (read-only, same method as the earlier comment: repeater and room rows, flood observations of the last 72 h, TRACE excluded):

• inactive_nodes: 7877 repeater and room rows, of which 838 are also in nodes.
• Hops with a unique prefix:

Hop width	Before b5556bdc	With b5556bdc	Retired rule without the node itself
2 bytes	97.1 %	59.3 %	84.9 %
3 bytes	94.9 %	57.5 %	94.9 %

• Repeaters and rooms with an advert in the last 7 days (1678):

	Before	With b5556bdc	Without the node itself
certain relay < 72 h	1256	629	1206
advert only < 72 h	240	865	290
nothing certain < 72 h	182	184	182

So 627 nodes lose their certain relays, and depend on one advert per 47 h again. TestRetiredNamesakeKeepsHopUncertain has no case where the node itself is in inactive_nodes.

Excluding the node itself (skip retired rows whose public_key is also in nodes, or test the retired set against pk like relayCandidates is) brings it back to 1206. The remaining 50 come from the other 7039 retired rows, many of which may have been silent for months.

Question

Which bound should the retired set get: none beyond excluding the node itself, only rows whose last_seen is within a window (for example 30 days), or only rows still referenced as a hop in the store's packets?

[efiten]

Review feedback addressed (commit 6e956f05)

1. A returned node is no longer its own retired namesake. markRetired skips inactive_nodes rows whose public_key is also in nodes (lower() on both sides). Test: TestReturnedNodeIsNotItsOwnRetiredNamesake (red before the change, green after).

Answer to the question: no further bound for now. The other retired rows keep counting, which costs 50 nodes on live (1256 to 1206) and keeps the protection against a retired repeater that still relays.

Cost: the query runs per node-cache rebuild (every 30 s). On a live database with 12382 inactive_nodes rows it takes about 24 ms against 16 ms before, measured from the sqlite3 CLI including process start.

cmd/server gofmt, vet and go test ./... pass locally. test-all.sh was not run locally (no Python on this machine); the change is Go only.