Skip to content

fix(liveness): count only evidence that can be the node alone (#2260) - #25

Closed
efiten wants to merge 2 commits into
masterfrom
fix/certain-liveness
Closed

efiten wants to merge 2 commits into
masterfrom
fix/certain-liveness

Conversation

@efiten

@efiten efiten commented Oct 22, 2026 •

Copy link
Copy Markdown
Owner

Fixes Kpa-clawbot#2260 (part 1 of 2: liveness; the adaptive offline threshold follows in a separate PR).

Problem

A repeater silent since 2026-10-18 showed Active — Last relayed 4m ago and Last Heard 36m, and its offline mail never fired. Every hit after that date was the 1-byte hop 67, shared by nine nodes:

Fix

The store keeps certainHeard per node: the newest transmission that can only be that node.

Evidence Counts?
Own packet (pubKey, srcPubKey) ✅
Flood relay hop whose prefix, at the packet's path hash size, matches exactly one path-capable node ✅
1-byte hop shared with other repeaters ❌
Direct-route path (hops still to come, Mesh::removeSelfFromPath) ❌
TRACE (path bytes are SNR) ❌
Destination (destPubKey) ❌

A multibyte repeater legitimately appears as a 1-byte hop (the originator picks the hash size, Mesh::routeRecvPacket appends at that size), so only uniqueness decides. The rule is applied in every load path through indexResolvedPathHops and the new indexColdLoadRelayHops: Load, loadChunk, scanAndMergeChunk, live ingest, late observations.

Liveness reads only that:

  • /api/nodes and the node detail: last_heard is the newer of last_seen and the certain last heard. Before, it was a plain copy of last_seen.
  • health / bulk-health stats.lastHeard.
  • LastHeardMap → notify offline check. The separate relay input (lastRelayed, relayTimes, notifyInput.Relayed) is removed.
  • roles.js getEffectiveHeardMs: newest known timestamp, without last_relayed. The side pane no longer says "Last relayed"; the Last Relayed row stays (analytics) with an updated tooltip.
  • New status uncertain (repeaters and rooms): nothing certain within the silent window, but the shared prefix (last_relayed) still in paths. Grey badge with explanation, an Uncertain filter on the nodes list and the map, dimmed like stale. The offline mail for such a node says only its shared prefix was still seen.
  • A prefix also held by a path-capable node in inactive_nodes is not unique; adverts with signatureValid: false are no evidence; /api/nodes/search reports the same last_heard.

Attribution for analytics (byNode, packet counts, last_relayed, relay counters, usefulness) is unchanged.

Performance

Hot path: noteCertainRelays runs per observation on ingest under the write lock: two hex bytes of header parsing plus one prefix-map lookup per resolved hop.

BenchmarkNoteCertainRelays-16   10307640   116.2 ns/op   0 B/op   0 allocs/op   (8-hop 2-byte path, 2,000 repeaters)
BenchmarkLastHeardMap-16            1902   0.77 ms/op                            (5,000 nodes; was 3.7 ms scanning byNode)

Memory: one string per node ever certainly heard, dropped together with the node's byNode list on eviction.

Tests

  • certain_heard_loadpaths_test.go: every load path (Load, loadChunk, LoadChunked — each with and without resolved_path — live ingest, late observations) feeds certainHeard; each of the 8 call sites verified red with a nil prefix map.
  • certain_heard_test.go: hop width per route/payload type; retired namesake prefix; failed signature; uncertain offline mail; shared vs unique 1-byte vs 2-byte hops; direct/TRACE/nil prefix map; cold-load path; sender vs destination; eviction; lastHeardValue; /api/nodes, detail and health agree.
  • Regression for Node liveness (Last Heard, Active status, Notify-me) counts ambiguous 1-byte hops: a silent repeater shows "Active, last relayed 4m ago" Kpa-clawbot/CoreScope#2260: last certain packet on day X, 20 shared-prefix hops after it → LastHeardMap stays at X and node.offline goes bad after 72 h. Verified red with the uniqueness check mutated out.
  • Notify: TestEvaluateCertainHeard replaces the relay-input test.
  • Frontend: last_relayed alone no longer keeps a repeater or room active; newest timestamp wins (an old _liveSeen can't hide a fresh certain relay).
  • cmd/server go test ./... and test-all.sh pass locally, also on a trial merge into a downstream fork with 457 extra commits.

Validation

Ran the server on the freshened e2e fixture: list, detail and health report the same last heard; e.g. a repeater whose last_relayed is 3 min newer than anything certain keeps its certain time. No browser screenshot: Playwright isn't installed on this machine; CI's E2E covers the pages.

Not changed

🤖 Generated with Claude Code

@efiten

efiten commented Oct 22, 2026

Copy link
Copy Markdown
Owner Author

Review feedback addressed (commit f7a809d9, amended)

  1. Side pane / node page showed the first timestamp present, not the newest. n.last_heard || stats.lastHeard could show an older cached value. Both places and getStatusInfo now take the newest via getEffectiveHeardMs, the same timestamp the status uses. Test: getStatusInfo shows the newest of _lastHeard and last_heard.
  2. Adverts whose signature failed counted as certain. indexByNode now skips signatureValid: false (absent = not checked, still counts). signatureValid is taken off decodedCacheDroppedKeys so the cached decode keeps it (one bool per advert). Test: TestIndexByNodeFailedSignatureIsNotCertain.
  3. /api/nodes/search still returned last_heard = last_seen. It now folds in the certain last heard like /api/nodes and the detail. Test: TestNodeSearchLastHeardIsCertain.

Not changed, with reasons:

  • Ingest-time vs current prefix map (certainHeard vs analytics in feat(notify): adaptive offline threshold and certain node analytics (#2260) #26): one rule at two moments. Recording a per-(tx, node) flag to make them identical costs a map entry per byNode pair, tens of MB on large instances. They only diverge when a new repeater with the same prefix appears. Documented on certainFor.
  • A shared-contact advert re-sent by another device usually dedups by hash with the original. When the original was already evicted, nodes.last_seen has the same issue today.

cmd/server go test ./... and test-all.sh pass, also on a trial merge into this fork's master.

…awbot#2260)

A repeater silent since 18 Oct showed "Active, last relayed 4m ago" with
Last Heard 36m, and its offline mail never fired. Every hit after that
date was the 1-byte hop 67, shared by nine nodes: the live resolver's
affinity tiebreak put it in byNode, and last_relayed folds in the 1-byte
prefix bucket on purpose (Kpa-clawbot#662).

The store now keeps certainHeard per node: the newest transmission that
can only be that node. Its own packets (pubKey / srcPubKey) count, and a
flood relay hop counts when its prefix at the packet's path hash size
matches exactly one path-capable node. Direct-route paths (hops still to
come, Mesh::removeSelfFromPath) and TRACE (SNR bytes) never count. The
same rule feeds every load path: Load, loadChunk, scanAndMergeChunk,
live ingest and late observations.

Liveness reads only that:
- /api/nodes and the node detail: last_heard is the newer of last_seen
  and the certain last heard, no longer a copy of last_seen;
- health and bulk-health stats.lastHeard;
- LastHeardMap, so the notify offline check; the separate relay input
  (lastRelayed / relayTimes) is gone;
- roles.js getEffectiveHeardMs: the newest known timestamp, without
  last_relayed; the side pane no longer says "Last relayed".

Attribution for analytics (byNode, packet counts, last_relayed, relay
counters) is unchanged.

Cost: noteCertainRelays 116 ns per observation for 8 hops, 0 allocs;
LastHeardMap 0.77 ms for 5,000 nodes (was 3.7 ms), one map lookup each.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@efiten
efiten force-pushed the fix/certain-liveness branch from f7a809d to b5556bd Compare October 22, 2026 12:22
@efiten

efiten commented Oct 22, 2026

Copy link
Copy Markdown
Owner Author

Review notes addressed (commit b5556bdc, amended)

Answers to the question: everything is addressed in this PR; it lands first, Kpa-clawbot#2262 right after.

  1. Advert-only nodes (241 / 184). New third status uncertain for repeaters and rooms: nothing that can only be this node within the silent window, but its hop prefix (shared with other repeaters, last_relayed) still appears in paths. It is neither proof of life nor of an outage, so:
    • UI: grey "Uncertain" badge with the explanation "Last certainly heard X — its shared hop prefix was last in a path Y, possibly another node"; an Uncertain button in the status filter on the nodes list and the map; map/live dim it like stale; map legend counts it separately.
    • Notify: still node.offline (an owner should know), but the mail line says why: "offline (nothing that can only be this node; its hop prefix, shared with other repeaters, was last in a path …, possibly another node)". The shared-prefix time only words the mail; it never decides the state.
    • getNodeStatus(role, ms) (legacy form) never returns uncertain. Tests for each case.
  2. Uniqueness against nodes only. uniqueHopPrefix now also rejects a prefix held by a path-capable node in inactive_nodes (moved there by retention but may still relay): prefixMap.retired, loaded with the node cache. A repeater that never advertised is unknown to both and cannot be ruled out; that is stated on the function. Test: TestRetiredNamesakeKeepsHopUncertain.
  3. Load-path wiring. TestCertainHeardFedByEveryLoadPath goes through the real Load (resolved_path and path_json), loadChunk (both), LoadChunked/scanAndMergeChunk (both), IngestNewFromDB and IngestNewObservations, asserting LastHeardMap. Verified: passing nil as prefix map at each of the 8 call sites in turn fails exactly that path's subtest.

cmd/server go test ./... and test-all.sh pass.

…lawbot#2260)

Retention copies a node into inactive_nodes and nothing removes the row
when the node is heard again. markRetired loaded those rows too, so such a
node's own prefix counted as a retired namesake and it got no certain
relay at any width. markRetired now skips rows whose public_key is also in
nodes.

On a live instance 838 of the 7877 repeater and room rows in
inactive_nodes are also in nodes; repeaters and rooms with a certain relay
in the last 72 h went from 1256 to 629 with the rule, 1206 with this fix.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@efiten

efiten commented Oct 22, 2026

Copy link
Copy Markdown
Owner Author

Closing: the content of this PR is in fork master through #26 (merge 7de7e3f, master 6f09ced). The fix commit of this branch (6e956f0) went in there as cherry-pick a287de4, so GitHub does not see this PR as merged. Live runs it as v3.14.0-on8ar.13.

Upstream counterpart: Kpa-clawbot#2261.

@efiten efiten closed this Oct 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Node liveness (Last Heard, Active status, Notify-me) counts ambiguous 1-byte hops: a silent repeater shows "Active, last relayed 4m ago"

1 participant