anomaly: design a backlog signal that marks late-delivered observations (follow-up to #473) #538

Activity

1. dborup-agent

   added 2 commits that reference this issue on Oct 25, 2026

   anomaly: pin the late-delivery limits and the exclusive t + gap bound (…

   ...

   c428a80

   anomaly: state the late-delivery limits (#538)

   ...

   a643321
2. dborup commented on Oct 25, 2026

   

   dborup

   on Oct 25, 2026

   OwnerAuthor

   More actions

   Pre-registration — #538 validation round: rate rule with the late-delivery guard (fixed before any data is seen)

   Posted by the coordinator at the reviewed head of PR #591 (merged as f7e13bc3). No data from V exists yet, and the coordinator has seen no staging data from 2026-10-25 onward (the #474 round 4 blindness also applies).

   Rules of the round

   • Everything below is fixed once posted. Changing a flag value, a rule value, the labelling, a metric or a bar after the data is seen starts a new round, reported separately; it is never a pass of this round.
   • The comment is not edited after posting. A correction is a new comment and starts a new round.

   Tool. cmd/anomaly from this PR at head a6433216e091790ceebd535295c76957e609a127, built with go build -trimpath; the binary sha256 is reported. a6433216e091790ceebd535295c76957e609a127 is the head commit of PR #591 that the last review of the PR reviewed: a commit on feat/538-backlog-late-delivery-signal, not a merge commit and not master. If that commit cannot be built, the round is void.

   1. Data and runs

   Item	Fixed value
   Validation window V	[2026-11-02T00:00Z, 2026-11-09T00:00Z): 7 whole UTC days. V is disjoint from #474 round 4, whose V ends at the latest on 2026-11-01T00:00Z, and starts one day after it, so the export's scan margin (V.start − 30 s − 2 min) reads no row of that round's V either. The snapshot must cover V plus the settle margin plus the scan margin of 2 windows: it is taken after V.end + 30 s + 2 min. There is no shorter fallback. If this text is not posted before 2026-11-02T00:00Z, or the coordinator has seen data from V before posting, the round is void; a new round needs a new posting with a new V.
   Calibration day C	[2026-11-02T00:00Z, 2026-11-03T00:00Z), the first 24 h of V. It is used only for the rate thresholds (§3).
   Evaluation window E	[2026-11-03T00:00Z, 2026-11-09T00:00Z), V minus C, 6 days. All metrics count only episodes activated in E and storms that start in E.
   Snapshot	As in #473 §1: an offline scripts/backup-sqlite.sh copy used as-is when no -wal/-journal file sits beside it, or else sqlite3 -readonly <source> "VACUUM INTO …". Read-only, with -immutable.
   Export (one file, both arms)	corescope-anomaly export -db <snap>/meshcore.db -immutable -start 2026-11-02T00:00:00Z -end 2026-11-09T00:00:00Z -settle 30s -max-rows 50000000 -late-window 1m -late-min-rows 10 -late-share 0.5 -late-gap 2m -out events.jsonl
   Replay, arm A / arm B	corescope-anomaly replay -config round538A.json (or round538B.json) -input events.jsonl -start 2026-11-02T00:00:00Z -end 2026-11-09T00:00:00Z -max-line-bytes 4096 -out reportA.json (or reportB.json)
   Episode	One EventID. Activation is its first transition to active; its end is the last transition to ended (V.end if none).
   Capacity	#457 round 2 §1a, plus a bounded retry. The export runs with -max-rows 50000000. If either export scan fails with more than -max-rows, it runs once more with -max-rows 100000000, the tool's maximum (maxExportRows); no other value is used. If that run also fails with more than -max-rows, the round is void. A failure for more than 4,194,304 (observer, bin) entries also voids the round. Every run and its outcome is reported.

   2. The late-delivery flag (fixed values)

   The definition is the one in this PR:

   • Already seen, with gap 2 min.
   • Late burst: window 1 min (bins k−1..k+1), at least 10 rows, a share of at least 0.5 already seen, and the delays of those spread over at least 2 min.
   • Flag: on the event's first observation; NULL observers are never flagged.
   • Times are observations.timestamp as stored, the time base of the anomaly phase 3: replay CLI, read-only export and blind validation (periodic, rate, new-stream) #457 labels.

   Why these values. All of them come from data before V, and none from staging after 2026-10-24:

   • gap = 2 min is 4× the 30 s settle that the export already treats as live route evidence. Flood propagation and upload jitter sit well inside it.
   • window = 1 min makes the 3-min window hold a delivery burst of seconds, plus its neighbours across a bin edge.
   • min rows = 10 is under a third of the round-2 GRP_TXT threshold (33 per 5 min, anomaly round 2: rate rule with an ingest-backlog guard, blind validation on fresh data #473 issuecomment-6380305586). A burst too small to matter to the rate rule on its own is still flagged when it is clearly late.
   • share = 0.5 is the largest value tested. In the 20-seed synthetic sweep it is the only one that flagged no live event; 0.3 flagged 1 of 8,769, a case with two skewed clocks. That is a property of the sweep, not a protection: two clocks behind by different offsets can make another observer's rows late at any share (limitations below).

   Limitations, stated before the data (synthetic figures from this PR's tests):

   • A common-cause outage is never flagged. When two or more observers drain the same outage together (a shared uplink or broker), each queued row has the other draining observer as a peer within seconds, so no row is already seen. One observer draining a 40-min outage: 25/25 and 40/40 flagged; two or three together: 0. Episodes that such outages cause stay in arm B as in arm A.
   • A queue whose already-seen rows are fewer than half of the burst is not flagged. The sweep's recall was 55 %. For a queue of 40 queue-only GRP_TXT delivered at 5 rows/s, recall is 0/40 after outages of 3–10 min at 6 shared rows/min and after 3–40 min at 1 shared row/min, and 40/40 after 20 and 40 min at 6 shared rows/min.
   • An outage shorter than about 4 min (2·gap) is never flagged.
   • A queue from an observer that overlaps no other observer is not flagged.
   • The draining observer's own live first observations within a window of its delivery are flagged.
   • Share 0.5 does not protect against skewed clocks. Two or more clocks behind by different offsets of at least 2 min can make another observer's rows look late: in a synthetic real storm of 40 GRP_TXT heard by O and, a third each, by observers 3 and 7 min behind, 13 of 40 events were flagged. Such a storm can lose its episode in arm B; the comparative criterion (§6) counts that as a loss.

   3. Rule configs

   • T_ADVERT and T_GRPTXT are computed by the anomaly phase 3: replay CLI, read-only export and blind validation (periodic, rate, new-stream) #457 §3 procedure on C: calibrate.py, unchanged, over all events of C, flagged ones included. Both arms use the same two values, and the coordinator posts them before either replay runs.
   • Arm A is round473A.json from anomaly round 2: rate rule with an ingest-backlog guard, blind validation on fresh data #473 §3, unchanged: the two round-1 rate rules (p457_rate_advert, p457_rate_grptxt) with the same limits and expected.
   • Arm B is arm A with "exclude_ingest_backlog": true on both rules and the new names p538_rate_advert_guard and p538_rate_grptxt_guard. Nothing else differs.
   • Both files are checked with ParseReplayConfig and New on the pinned head, with the placeholders set to 20.

   4. Ground truth

   The #457 round-2 §4 rate labels, unchanged, computed with label457.py. Only V and E change, to the values in §1, and only the rate part is used.

   • Storm bins are count ≥ max(3·B(p,h), B(p,h)+30), and runs separated by one bin are merged.
   • Ingest artifact: ≥ 50 % of a storm's transmissions were first observed by one observer that had no observation in the 10 min before the storm.
   • Counts use all events, flagged ones included. The labels use no detector output and not the flag.
   • The label file holds (payload type, start, stop, artifact) tuples only. Its sha256 is posted before either replay runs.

   5. Matching and metrics

   Per arm, as in #457 round 2 §5 (rate column), unchanged. The recount for arm B counts only lines without "ingest_backlog": true.

   Comparative metrics (they decide, §6). An episode is on a storm, and a storm is detected by an arm, under the #457 round 2 §5 matching rule, unchanged. Over the storms that start in E:

   • N_art and N_real: the number of artifact and non-artifact storms;
   • EA_art and EB_art: the number of arm A's and arm B's episodes that are on an artifact storm;
   • L: the number of non-artifact storms that arm A detects and arm B does not.

   Also reported, outside the bars:

   • per-rule numbers;
   • artifact storms, and each arm's episodes on them;
   • late_delivery.events_flagged and rows_already_seen, in V and in E;
   • per storm, the share of its events that are flagged;
   • the episodes of A without an overlapping episode in B, and the reverse, as counts and tuples;
   • the non-artifact storms that arm B detects and arm A does not.

   6. Pass bars and decision

   All of these must hold, otherwise no-go:

   • recount ≥ threshold at 100 % of activations;
   • precision ≥ 0.70;
   • recall ≥ 0.80;
   • false alarms ≤ 1.0/day;
   • median TTD ≤ 10 min;
   • median stop latency ≤ 20 min.

   Comparative criterion, also required for a go (§5 metrics; the comparator is arm A, the tolerance is zero storms):

   • C1, benefit: N_art ≥ 1 and EB_art < EA_art. Arm B has strictly fewer episodes on artifact storms than arm A.
   • C2, no loss: L = 0. Arm B detects every non-artifact storm that arm A detects. This implies that arm B's recall is at least arm A's.

   How the result is read:

   • Go = arm B passes every bar above and C1 and C2. Arm A's own bars do not decide; arm A is the comparator for C1 and C2.
   • What a go licenses: on this network in E, with the fixed flag values of §2 and the two round-1 rate rules of §3, adding the late-delivery guard reduced episodes on anomaly phase 3: replay CLI, read-only export and blind validation (periodic, rate, new-stream) #457-labelled ingest artifacts compared with the same rules unguarded, lost no non-artifact storm that the unguarded rules detected, and met the anomaly phase 3: replay CLI, read-only export and blind validation (periodic, rate, new-stream) #457 bars. It licenses nothing else: no default or production value; no claim about other rules, windows, flag values or networks; no claim that the flag finds every backlog (recall is not measured against backlogs, only against artifact-storm episodes); and no claim about common-cause outages, which are never flagged (§2).
   • No-go = any bar fails, or C2 fails, or N_art ≥ 1 and EB_art ≥ EA_art.
   • Zero artifact storms in E (N_art = 0), or arm A with no episode on an artifact storm (EA_art = 0), leaves the guard's benefit unmeasured: the round is inconclusive, which counts as a no-go for any claim. C2 and the bars are still reported, as a safety result only.
   • Zero non-artifact storms in E makes the round inconclusive, which counts as a no-go for any claim.
   • A void round (§1: posting after V.start or after data from V was seen, an unbuildable a6433216e091790ceebd535295c76957e609a127, a failed export at -max-rows 100000000, or the bin cap) has no results.
   • No value from this round becomes a default or a production value.

   7. Files

   events.jsonl, both reports and the label file stay with the coordinator. Only aggregates and (payload type, start, stop, artifact) tuples are published.

   🤖 Generated with Claude Code