[efiten]

Part 2 of #2260. Builds on #2261 (certain liveness): review/merge that first; until then this diff also shows its commit.

Adaptive offline threshold

userManagement.notifications.adaptiveOffline — off by default:

"notifications": {
  "enabled": true,
  "adaptiveOffline": { "enabled": true, "floorMinutes": 30, "minObservers": 3 }
}

The fixed infraSilentHours (72 h) treats every repeater the same. With this on, a watched node's node.offline window is learned from its own certain traffic (the #2261 rule):

1. Certain packets per hour over the week before its last certain packet. Ending there, not at now, keeps an outage from dragging the rate down until the fixed window returns and the node reads "back online".
2. Empty runs are classified: ≤ 6 h and recurring on another day at the same hours = normal silence (kept; the window is never shorter than the longest + 1 h); any other run = outage, left out of the learning so it neither stretches the window nor blocks learning.
3. λ = 10th percentile of the remaining hourly counts, so quiet nights don't alert.
4. Window = 14/λ hours: silence at rate λ has a < 10⁻⁶ chance (Poisson, ln 10⁶ ≈ 13.8).
5. Clamped to [floorMinutes, the role's silent window].

Kept on the fixed window: λ < 1/hour, under 24 h of history in memory, or a 10th-percentile hour heard by fewer than minObservers observers (one observer dropping out must not read as the node going offline). Silence is measured up to the newest packet in the store, so a stalling feed stops every node's clock. A node stored offline comes back only on evidence from the moment it went offline on. The notifier caches each learned window for 10 minutes and drops unwatched nodes.

For the #2260 repeater (~840 packets/day) the mail would have gone out about 30 minutes into the outage, instead of never (or 72 h with #2261 alone).

Certain node analytics

• Each activityTimeline bucket carries certain: the part that can only be this node (certainFor, the packet-level form of the certainHeard rule). TimeBucket.Certain is omitempty, so other endpoints are unchanged.
• Availability and longest silence use certain buckets only; the longest silence now includes one still running (an outage in progress is the silence worth seeing).
• The chart stacks certain (--accent) and uncertain (--text-muted) — existing theme variables, so no new customizer entry. Descriptions updated.

Performance

CertainActivity walks a week of packets per watched node, at most once per node per 10 minutes, with the read lock taken per node:

BenchmarkCertainActivity-16   12.1 ms/op   (50 watched repeaters × 6,000 packets; ~0.25 ms per lock hold)
BenchmarkNoteCertainRelays-16  121 ns/op   0 allocs

Node analytics adds one certainFor per packet of the requested node (a prefix-map lookup, or the cached decoded JSON for its own packets).

Tests

• adaptiveOfflineAfter: busy repeater hits the floor; 5/hour; quiet nights set the rate; > 10 % empty hours, short history, too few observers → fixed.
• CertainActivity ends at the last certain packet and counts only certain packets — verified red by mutating the window to end at now.
• Notifier end to end: learned window passed → bad; inside → good; too few observers / setting off → fixed 72 h; cache refresh and prune.
• certainFor, longestSilence (incl. running silence), config defaults, /api/nodes/{pk}/analytics splits a shared-prefix bucket.
• Frontend: NodeAnalyticsTimeline.series (new tests/unit/test-node-analytics-certain-timeline.js).
• cmd/server go test ./... and test-all.sh pass locally, also on a trial merge into a downstream fork.

Validation

Server on the freshened e2e fixture: analytics buckets carry certain (e.g. 89 of 90 packets certain for one repeater). No browser screenshot: Playwright isn't installed on this machine.

Later (rule 8)

floorMinutes / minObservers are server config, not theme settings; an admin UI for notification settings would be the place, not the customizer.

🤖 Generated with Claude Code

[efiten]

Review notes

Reviewed the delta 1b40d8ac..000e4585. Tests were not re-run for this review. The measurement of certain relay coverage on live data is on #2261.

1. A stalled feed can still send offline mails for floor-clamped nodes

notify_service.go:52 treats ingest as stale when now.Sub(newest) > 30min; notify_eval.go:138 marks a node offline when now.Sub(last) >= silent, and the default floorMinutes is also 30. A busy repeater (the case that hits the floor) last heard 2 minutes before the feed stops is at 31 min of silence on a tick where the feed is at 29 min, so the mail goes out, followed by a "back online" mail after the resume. positiveOr also accepts floorMinutes: 1.

Two ways to close it: measure node silence against min(now, newestPacket), or enforce floorMinutes >= notifyIngestStaleAfter + tick interval.

2. minObservers counts over the whole learning week

adaptive_offline.go:94-100 collects distinct ObserverID over the week. A node heard 99 % through observer X and once each through Y and Z passes minObservers: 3. If X drops while the rest of ingest stays fresh, IngestStale never trips and the node is mailed offline after 30 min. Counting distinct observers per hour (for example the 10th-percentile hour) would match the intent in the file comment.

3. Smaller

• adaptive_offline.go:52,74,82: now comes from time.Now (local); when the window start binds on now.Add(-retention), fromStr carries the local offset and the string pre-filter against ...Z values drops the first hours. now = now.UTC() at the top of CertainActivity fixes it. Docker defaults to UTC, which limits the impact.
• adaptive_offline.go:84: a legacy "2006-01-02 15:04:05" FirstSeen on the window's first date sorts below fromStr (space < T), so the parsed-time recheck never sees it.
• certain_heard.go:236-260: a node with no certain bucket gets longestSilenceMs 0 (the page shows a dash, not the whole window), silence before the first certain bucket is ignored, and the running silence is measured from the start of the last bucket.
• After an outage longer than about 17 h the zero hours put the 10th percentile at 0, so the node is not learnable and stays on the fixed 72 h for up to a week. Safe, but a repeat outage is reported late.

Checked without finding a problem: percentile index and zero hours included, hour bucketing bounds, units in adaptiveOfflineAfter, lock order (getCachedNodesAndPM before s.mu, no nesting), the cache bounded to watched nodes and reset when disabled, the disabled path returns nil, Certain *int omitempty, colours via --accent / --text-muted, no new map[string]interface{}.

Question

For point 1, which do you prefer: silence measured against the newest ingested packet, or a config floor of at least the stale threshold plus one tick?

[efiten]

Review feedback addressed (commit 7ad251a5, rebased on f7a809d9)

1. False "back online" during a long outage. The learning window is clipped to the store's coverage, so after retention or a restart the learned window could vanish, fall back to 72 h and read a still-dead node as online. nodeOnlineState now keeps a node stored offline until there is evidence from the moment it went offline on (prevAt = the state's ChangedAt). Test: TestOfflineNodeNeedsNewEvidenceToComeBack (verified red with the check removed).
2. Flapping on regular silent hours. The 10th percentile tolerates up to 10 % empty hours, so a node with silent nights could get a 30-minute window. The learned window is now never shorter than the longest run of empty hours in the learning week + 1 h. Test case: regular silent nights lengthen the window.
3. Longest silence missed cases. No certain bucket in the window now reports the whole covered window, and the silence from the window start to the first certain bucket counts. Tests in TestLongestSilence.
4. certainFor also ignores adverts whose signature failed (same rule as Home page content capped at 720px — wastes space on desktop #25).

Not changed, with reasons:

• Mail after 30 min while the UI keeps 72 h. Deliberate for this PR: the mail is the early warning. Learning the UI threshold too is a separate step; I'll note it on Node liveness (Last Heard, Active status, Notify-me) counts ambiguous 1-byte hops: a silent repeater shows "Active, last relayed 4m ago" #2260.
• Use nodeTxsSince instead of walking byNode. That helper walks the whole list as well, so there is nothing to gain.

cmd/server go test ./... and test-all.sh pass, also on a trial merge of both PRs into this fork's master.

[efiten]

Follow-up on the review notes above

Read the delta 000e4585..7ad251a5; tests were not re-run, CI is green.

These changes read correctly:

• The offline-until-new-evidence rule: ChangedAt is only written on a state change (notify_eval.go:286), so prevAt is the moment the node went offline.
• The longest-silence cases.
• The signature rule.

Still open from the review notes:

1. Stalled feed vs floor. notify_service.go:52 is still > 30min against the default floor of 30. The question in the review notes is still open.
2. minObservers still counts distinct observers over the whole week (adaptive_offline.go, certainActivityOf).
3. Window start in local time. fromStr := from.Format(time.RFC3339) is not UTC when from comes from coverageStart(now).
4. Legacy timestamps. tx.FirstSeen < fromStr drops legacy "2006-01-02 15:04:05" values on the window's first date before the parsed-time check. The new comment says that check covers them, but it is never reached for those rows.
5. Outage over about 17 h. The node is still not learnable for up to a week.

New with the longest-zero-run rule

One real outage of N hours in the learning week (N under about 17, above that the node is not learnable) raises the learned window to N+1 h for the rest of that week. A second outage is then mailed after N+1 h instead of the floor.

Question

Which should the zero-run rule cover: every empty run (current), only runs that recur (for example on at least two days), or only runs up to a cap (for example 6 h), with longer ones treated as outages?

[efiten]

Review notes addressed (commit b0a37f2c, rebased on b5556bdc)

Answers to the questions: silence is measured against the newest ingested packet; the zero-run rule covers only runs that recur and are at most 6 h, longer or one-off runs are outages.

1. Stalled feed vs floor. notifyInput.NewestPacket: a node's silence is measured up to min(now, newest packet in the store), so a stalling feed stops every node's clock and nothing is mailed before IngestStale trips, whatever floorMinutes is. Test: TestStalledFeedStopsTheSilenceClock (verified red without the clamp).
2. minObservers. Counted per hour now: certainActivity.HourObservers, and the 10th-percentile hour (of hours with packets) must reach minObservers. A node carried by one observer stays on the fixed window. Test case: one observer carries the node.
3. Window start in local time. CertainActivity converts now and the coverage start to UTC; certainActivity.From is UTC. Test: TestCertainActivityUTCBoundsAndLegacyTimestamps with a +02:00 caller just after local midnight (verified red without it).
4. Legacy timestamps. The pre-filter compares the date only (FirstSeen[:10]); the parsed time decides. Same test, verified red with the old string bound.
5. Outage over ~17 h and the new zero-run question. Empty runs are classified first (classifySilences): a run of ≤ 6 h that recurs (another empty run on a different day covers one of the same hours of the day) is normal silence — it stays in the percentile and the window is never shorter than the longest such run + 1 h. Every other run is an outage and is left out of the learning entirely (percentile, history length, and the zero-run floor). So one outage neither stretches the window for a week nor makes the node unlearnable. Tests: recurring 2 h nightly silence → 3 h; one-off 6 h gap → floor; 30 h outage → still learnable; recurring 8 h gaps → outages; recurring 4 h nightly (17 % empty) → not learnable.

cmd/server go test ./... and test-all.sh pass.

[efiten]

Review of b0a37f2c

Read the delta b5556bdc..b0a37f2c; tests were not re-run.

These read correctly:

• The silence clock stops at the newest packet.
• minObservers is counted per hour, and the timestamps are compared in UTC on the date only.
• classifySilences: worst case about 168 x 168 comparisons per node per refresh, fine.

Two points:

1. Blocking, inherited from fix(liveness): count only evidence that can be the node alone (#2260) #2261. certainFor uses uniqueHopPrefix, so the retired-prefix problem in the fix(liveness): count only evidence that can be the node alone (#2260) #2261 comment above applies here as well. With it, 627 nodes lose their certain relays, and with them the hourly counts the adaptive window learns from.
2. Small. recurs also matches against runs that are themselves outages. One outage of 24 h or more covers every hour of the day, so with it in the week, every short empty run up to 6 h counts as normal silence. Comparing only against runs of at most adaptiveMaxNormalRun hours would avoid that.

[efiten]

Review feedback addressed (commit a287de47, the same change as 6e956f05 on #2261, cherry-picked)

1. Blocking point inherited from fix(liveness): count only evidence that can be the node alone (#2260) #2261: fixed by the same commit, see fix(liveness): count only evidence that can be the node alone (#2260) #2261. cmd/server gofmt, vet and go test ./... pass locally on this branch.

Not changed: the small recurs point (matching against outage runs) stays open for a follow-up.