Repository navigation
Anomaly: periodic rule targets channel spam bots, channel exclusions and injection-based evaluation - #478
Conversation
…ased evaluation Relates to #474, phases 1-3. Library and offline tool only: no ingestor, server, API or UI wiring, no defaults and no thresholds. Phase 1 (target). doc.go and the tool README state the periodic rule's target: an abusive channel bot, not expected automation. #457 round 4 measured wardriving trips, which is the opposite. Phase 2 (detector). PeriodicRule.ExcludeChannels lists channel hashes a periodic rule ignores; an excluded event never reaches the rule, so it forms no pulse, chain or episode and claims no key state. Opt-in, no defaults, validated (canonical, non-zero, no duplicates, bounded), matched on the whole hash (length included), and exposed to the replay JSON as "exclude_channels". 1-byte hash collisions are documented, not hidden. Phase 3 (evaluation). Two new offline subcommands: - inject overlays synthetic bot trains on exported background lines. Ids are "inj-<bot>-<pulse>-<part>", a value no export can produce; a background line already carrying that prefix is refused. File-only (no -db flag, no database), deterministic from the spec seed with an in-tool splitmix64, per-bot streams seeded by name, background copied through byte for byte, merged by data time. The labels file applies the #457 §4 recipe to the emitted pulses. - score measures a report against those labels with the #457 §5 matching, or without labels, which is the false-alarm arm. Nothing injected may ever be written into a database.
Relates to #474. `score -bots <names>` narrows the recall and the medians to the named bots' incidents, so a non-blind sanity-check bot (the September shape, which #130 and #447 were tuned on) can be reported apart from the blind panel without a hand computation afterwards. An episode that matches an unselected incident stays a true positive: the filter must never turn a known match into a false alarm. A filter that selects no labelled incident is an error, not an empty pass.
Relates to #474. The episode struct's id was written and never read, and an unset -bots filter marshalled as null instead of an empty list. No behaviour change.
Rapport — CS-Macmini PR#478 #474 — head b7ee095Status: Phases 1–3 of #474 implemented, 51 of 51 mutants killed, CI green except two pre-existing frontend E2E flakes (rerun once, detail below), and the round-1 pre-registration posted on #474 before any data exists — stopping there, as asked. Evidence tags: [T] a test in this PR, [A] analysis or a reading of existing code and issues, [K] a command run locally (output quoted). Requirements
51 mutants, 51 killed: 13 in the library, 20 in Local verification
CI per job (run 38787561866, head b7ee095)
Run conclusion: success. The two E2E shards were rerun once, as the task allows for a known flake. Pipeline check (synthetic only, no real data)[K] The pre-registered panel injected into an empty background, replayed with
That is a pipeline check, not a validation: it contains no real traffic. The real background has to meet the bars in the pre-registration (false alarms ≤ 0.2/day, recall ≥ 0.80, median time to detect ≤ 12 P, median stop latency ≤ 4 P, median episodes per detected incident ≤ 2). Residues and open points
|
Resolves the two textual conflicts with #477 (#473): - internal/anomaly/detector.go: keep both periodicApplies (#474) and RateRule.counts (#473). periodicApplies now takes *Event, matching counts(e *Event), so the hot path no longer copies the Event per periodic rule per scope (review F7). - cmd/anomaly/README.md: keep both optional-field paragraphs - the periodic rules' end_after_missed_pulses/exclude_channels list and the rate rules' exclude_ingest_backlog - and name all three issues in the intro. #473's pinned behaviour is untouched: backlog.go, export.go, event.go and all three #473 test files are byte-identical to origin/master, and internal/anomaly/testdata/replay_golden.json is byte-identical too. Relates to #474
Review F5. Five mutants survived the first round's suite: - labelIncidents' run length >= min_pulses -> > (nine pulses required): no test had a run of exactly min_pulses. - labelIncidents' gap tolerance <= -> <. - labelIncidents' pulse merge gap >= -> >. - score's periodFits tolerance <= -> <. - score's episode end taking the first "ended" instead of the last. Each now has a test that fails on the mutant and passes on the code: exactly 8 pulses (and 7, which must not label), a gap exactly 30 s off k*P (and one nanosecond further), events exactly 60 s apart (and one nanosecond closer), a period exactly at the tolerance through both periodFits and score, and an episode with two "ended" transitions. Relates to #474
Review F6. Injected lines were trivially distinguishable from exported ones in three ways that the detector does not read but a reader does: - Times carried nanosecond precision, while the export times every line from a whole-second observation time (time.Unix(ts, 0)). Injected event times, and the final_at derived from them, are now truncated to whole seconds, and final_after must be whole seconds exactly as the export's -settle must. - The payload size was constant. The new required spec field payload_bytes_spread draws each event's size uniformly from payload_bytes +/- spread; 0 keeps the old constant size, and a spread must be smaller than payload_bytes so a size can never reach zero. - The README example used the first hop as the whole route; it now shows a real multi-hop route. Truncation moves a time by less than a second towards the past, so the effective offset from the grid is in [-(jitter_max + 1s), jitter_max]; that is documented beside jitter_sd. Relates to #474
…ntially Review F2 (blocking), F3 and F8. F2. Round 1's panel sat on channel bytes a0-a8 and first hops fdff00-fdff08, which no real transmission carries, so every injected bot had its stream key to itself and arm B measured detection in isolation rather than detection inside real traffic. Three changes make a shared-key panel possible and scorable: - New "keys" subcommand. It reads an export and ranks its stream keys by row count, reporting per key the wire fields a bot needs to land on it: the canonical key, the channel hash, the route class and kind, the first hop, the most common whole route and the median payload and wire sizes with a payload spread. The filter is mechanical and restated in the output (requested payload type, a channel hash present, route kind relay, channel not excluded) and every skipped line is counted. It is blind by construction: it reads the export only and counts rows, never runs the detector and reads no report, no labels and no metrics. -max-keys is a hard bound, as the export's -max-rows is. - inject writes each bot's stream key into the labels, derived from the same Event the detector sees. - score matches an episode to an incident on that key when the labels carry one (falling back to the channel hash when they do not), so a different route group on the same busy channel is no longer credited to the bot. New -baseline takes the report of the same window without the injection: an episode it already has on the same key, overlapping in time, is the background's - neither a true positive nor a false positive, counted as baseline_episodes. That is the differential reading a shared key needs. F3. Five clean bots alarm at the earliest pulse the rule allows, so the median sits on that floor and hides the realistic shapes. score now reports the nearest-rank p75 and the maximum beside every median (time to detect, also in units of P, stop latency, episodes per incident), so the tail is visible without reading the per-incident table. F8. The metrics recorded the local path of the labels file; they now record its file name only, and the same for the new baseline report. A path can carry a user or a host name and a metrics file may be shared. Relates to #474
Review F2, F3 and F4. - A "keys" section: what the subcommand is for (a bot on a key no real transmission carries has that key to itself), why it is blind (it reads the export and counts rows, never the detector or a report), the mechanical filter and every field it reports. - The score section states that matching is on the stream key, what -baseline does and why a shared key needs a differential reading, and that the median hides the tail - so every median comes with its p75 and its maximum, and -bots can give the realistic subset its own numbers. - F4: the spec section now says to take the jitter from measurement, not from the rule's tolerance, with #130's measured values (|gap - 300 s| median 3 s, p95 9 s over 85.2 h), the derivation of a 3 s per-pulse sd from them, and the consequence that some pulses of a short-period bot then fall outside a tight tolerance. Scaling the jitter to the rule makes recall conditional on the rule being evaluated and moves the panel whenever the rule moves. - Provenance is file names only (F8). Relates to #474
Rapport — CS-Macmini PR#478 runde-2 — head 7f2f1a8Review feedback addressed (commit Five commits on top of Because the head and the protocol both moved, a new, superseding pre-registration was posted on #474 before any data for its window exists: Evidence tags: [T] I ran it · [A] GitHub API or CI logs · [K] code reading only. 1. F1 (blocking) — merge with #477 (#473)
#473's pinned behaviour is unchanged, byte for byte [T]:
F7 resolved with it. 2. F2 (blocking, protocol) — the panel now shares stream keys with real trafficAgreed, and it was the right call: round 1's bots sat on channel bytes a. New It is blind by construction: it reads the export and counts rows, never runs the detector, and reads no report, no labels and no metrics — so it cannot know which keys would alarm [K]. The filter is mechanical, is restated in its own output and counts every skip: payload type must match How the keys are chosen, mechanically and without seeing results (pre-registration §6a): panel position i takes the key of rank i+1, and nothing else is consulted. b. c. d. The panel is now paired (pre-registration §5/§6). Each of the nine shapes appears twice at the same time on two different keys: The pipeline check in §11 of the new pre-registration shows the effect on a synthetic Poisson background (busiest group 1 message per 10 min): shared keys give median time to detect 13.5 P, p75 16.0 P, max 94.0 P, up to 3 episodes per incident; the isolated twins give 7.0 P, 7.0 P, 35.0 P and up to 2. Recall was 8/8 in both arms on that background [T]. No bar was set from it. 3. F3 (protocol) — the median no longer stands alone
4. F4 (protocol) — jitter from #130's measurements, not from the ruleAgreed. #130 measured a real 5-minute bot over 85.2 h: 5. F5 — the five surviving mutants now dieEach boundary the first round's suite left open has a test that is red on the mutant and green on the code [T]:
6. F6 — injected lines now have the export's shape
7. F8 — provenance is file names only
8. F9 — V starts after the new commentThe new pre-registration's V is the first 7 whole UTC days starting at or after 2026-10-16T00:00:00Z, the first whole UTC day that begins after the comment was posted. The "before any data" claim is now literal, not a statement of intent. TestsGo suites — all with
Node [T]
E2E [T] This PR changes no file outside
New tests (17, in One existing test changed: Mutants22 mutants, one or more per finding, all killed [T]. Each was applied to the pushed tree, the named tests run, and the tree restored.
No mutant is reported for F1 or F7: F1's resolution is textual and is evidenced by the byte-identity table above, and F7's pointer-versus-value change is semantically identical, so no mutant of it can fail a test. M8's kill is through a nil dereference in the following validation case rather than a clean assertion failure; I report that rather than present it as a clean kill. Determinism and offline properties re-checked [T]
CI per jobRun 38803855328 (
All E2E shards were green on attempt 1 as well. The one real failure was in The failure, named and explained. The test compares two whole JSON bodies, one for a hidden hop and one for an unknown hop, and expects them to be identical once the hop byte is normalised. The two sets of rows are seeded from two separate I reproduced it locally on this tree: green at Not verified
|
Round-3 review of PR #478 (#474), tests first. G1: the pre-registered pipeline could not be injected on a realistic key. make_spec.py held wire_bytes at the key's median while drawing the payload from median +/- (p90 - p10)/2, and Event.Validate refuses PayloadBytes > WireBytes - 2, so the first wide draw aborted the run. TestInjectSpreadEnvelopeMustFitTheFrame pins the whole draw envelope against the frame and against MaxPayloadBytes; the two pipeline tests run export -> keys -> make_spec.py -> inject on a block-quantised background and on the package's own synthetic CoreScope fixture. G2: injected sizes were still distinguishable. The keys ranking now has to carry the key's observed (wire, payload) pairs, and inject to draw whole pairs from them. G3: a timing bar over the detected incidents cannot fail on a missed bot. TestScoreSelectedTailCountsAMissAsInfinite shows the panel that passes every round-2 timing bar with four of eight realistic shapes never found, and requires the selected tail that fails it. G4: one test per surviving mutant - the keys tie-break, -top, the nearest-rank percentile boundary, the spread clamp at equality, first/last, truncation towards the past, and both touching ends of the baseline overlap. G5: baseline masking has to be visible per incident. The two round-2 payload-spread cases move with the envelope rule: a spread one below payload_bytes is no longer enough on its own.
G1/G2 of the round-3 review of PR #478 (#474). A frame is header plus path plus payload, so wire - payload is the route's own overhead, while a GRP_TXT payload comes in 16-byte cipher blocks. Round 2 described a bot's sizes as a fixed median wire plus a payload spread, which gets both halves wrong: the drawn payload routinely passed wire - 2 and Event.Validate aborted the run part way through (G1), and where it did fit, the sizes were ones the key never carried, so an injected line was distinguishable on size alone (G2). keys now reports size_mix per key: every observed (wire_bytes, payload_bytes) pair with its count, ordered by wire then payload, with the payload absent for the lines whose length the export did not know. The distinct pairs are bounded like the other per-key histograms and the overflow is reported. inject takes size_mix as an alternative to the scalar sizes and draws one whole pair per event, weighted by count - a binary search over cumulative weights computed once per bot. A single-entry mix consumes no randomness, as a zero spread does, so an existing train's draws do not move. The scalar path keeps working for hand-written specs, but now has to fit: payload_bytes + payload_bytes_spread <= wire_bytes - 2. And every size a bot can draw is checked against the event validator while the spec is read, so an impossible size is a spec error with a message naming the fields, never an abort thousands of lines into a run.
G3 and G5 of the round-3 review of PR #478 (#474). G3. Every median and p75 was taken over the detected incidents only, so no timing bar on them could fail on a bot that was never found: a panel whose four cleanest shapes alarm at the floor and whose four realistic ones are missed reports recall 0.50 and a p75 of one period, and passes the lot. The time to detect is now also reported over every selected incident with a miss ranked above every finite value. A percentile whose rank falls among the misses is infinite; JSON has no infinity, so it is reported as ..._selected_infinite with the value left out, which fails any bar placed on it. incidents_missed counts the misses. G5. A -baseline episode is not scored either way, so an incident whose only episode was absorbed by the background read as a plain miss. Each per-incident row now carries baseline_episodes and baseline_masked, and incidents_baseline_masked counts them, so a no-go caused by background contamination can be told from a detector miss.
…anges make_spec.py moves into the repository. Round 2 kept it only in the pre-registration comment, which is why nothing exercised the one step that was broken: the chain export -> keys -> make_spec.py -> inject on realistic sizes. It is stdlib-only, reads only a window start and a keys ranking, has no clock, and now passes each key's size_mix through verbatim instead of describing the sizes as a median and a spread. The README documents size_mix on both sides, the envelope rule for the scalar sizes, the selected-incident tail with a miss ranked as infinite, and the per-incident baseline flag.
Rapport — CS-Macmini PR#478 runde-3 — head 743d0f3Review feedback addressed (commit Evidence tags: [T] I ran it myself · [A] GitHub API or CI logs · [K] code reading only. 1. G1 (blocking) —
|
| Mutant | Killed by |
|---|---|
N3 keys tie-break reversed |
TestKeysTieBreakIsTheSmallerCanonicalKey |
N11 -top keeps top − 1 |
TestKeysTopReportsExactlyTopKeys |
N6 histPercentile seen > idx → >= |
TestKeysPercentilesAreNearestRank |
N4 spread clamp >= → > |
TestKeysSpreadIsClampedStrictlyBelowTheMedian |
N7 first uses After |
TestKeysFirstAndLastAreTheExtremeTimes |
N8 Truncate → Round |
TestInjectTruncatesTowardsThePast |
N1 markBaseline touching end excluded |
TestScoreBaselineOverlapIsClosedAtBothEnds |
N2 markBaseline touching start excluded |
TestScoreBaselineOverlapIsClosedAtBothEnds |
5. G5 — baseline masking was invisible per incident. Fixed.
Each per-incident row now carries baseline_episodes (the episodes on that incident's own series, overlapping its window, that the plain arm already had) and baseline_masked (not detected, but the background already had an episode here); incidents_baseline_masked counts them. The period check is deliberately not applied to them, because it is not applied to a baseline episode anywhere — that is stated in the field's own comment [K].
TestScoreReportsBaselineMaskingPerIncident pins both the masked incident and a plain miss beside it, so the flag cannot degenerate into "not detected" [T]. And §11 of the new pre-registration demonstrates it end to end, which round 2 could not do (its arm A had no episode at all): with one real 5-minute train added on rank 1's own key, arm A has 1 episode, and in arm B the same run reads recall 6/8 without -baseline and 5/8 with it, baseline_episodes 1, incidents_baseline_masked 1, and b0-90s-clean-shared carries baseline_masked: true instead of reading as a plain miss [T].
6. G6 — CI flake outside this PR. Not fixed here, as instructed.
Agreed and left alone: TestRoutePatterns410_HiddenHopWithTrafficLooksUnknown/observer_blacklist, two separate time.Now() seeds, no cmd/server diff in this PR. It did not recur in this round's run. I filed no issue.
The superseding pre-registration
Posted as issuecomment-6203588711 on #474 at 2026-10-15T22:28:57Z, updated_at = created_at (unedited) [A]. It says "No validation data has been seen", replaces round 2 in full, pins head 743d0f33…, pins make_spec.py's sha256 and a -trimpath binary hash with its go version (round 2's bare-build hash did not reproduce for the reviewer), and starts V at 2026-10-17T00:00:00Z — after the comment, so no part of V existed when it was posted.
One imprecision in it, which I am flagging rather than editing. Its §3 glosses V's start as "the first whole UTC day that begins after this comment was posted". GitHub stamped the comment 2026-10-15T22:28Z, so the first such day is 2026-10-16, not 2026-10-17 — I drafted against a local date of 2026-10-16. The fixed value is unchanged and is the binding one: V starts at or after 2026-10-17T00:00:00Z, which is strictly later than required, so the property the gloss exists to guarantee holds with a full extra day of margin. No window, panel, labelling, metric or bar moves, so this is not a round 4. I recorded it as a no-change note, issuecomment-6203595309 [A].
Tests
New tests: 19, all in cmd/anomaly/review_r3_test.go. Red before green: the file does not compile against the round-2 tree (undefined: sizePair, scoreResult has no field IncidentsMissed), and each behavioural test was then checked against its own mutant below [T].
Two round-2 tests moved with the envelope rule [T]: TestInjectPayloadSpreadIsValidated gains a "wider than the frame allows" case, and its closing positive case changes from spread 34 (which with wire_bytes 60 can reach 69 > 58) to spread 23, whose top is exactly wire_bytes − 2. The old assertion encoded the bug.
Go suites — go vet + -count=1, the anomaly modules also with -race [T]
| Suite | Result |
|---|---|
cmd/anomaly |
ok; -race ok (7.7 s) |
internal/anomaly |
ok; -race ok (10.1 s) |
cmd/server |
ok (54.5 s) |
cmd/ingestor |
ok (107.5 s) |
Node [T]: sh test-all.sh 257 passed, 0 failed. node test-frontend-helpers.js 719 passed, 0 failed.
E2E [T]. Server on port 13841 with the fixture prepared as in CI (freshen, the inline Kpa-clawbot#1486/Kpa-clawbot#1791 seed SQL, corescope-migrate, then seeds 2073, 199, 245, 122), stopped afterwards and the port confirmed free; the fixture was restored with git checkout and the build artefacts removed. This PR has no E2E of its own and touches no served code, so I ran the core suite plus master's two newest UI suites:
| Suite | Result |
|---|---|
test-e2e-playwright.js |
154/157, 3 skipped |
test-issue-480-no-scope-e2e.js |
25/25 |
test-issue-479-airtime-e2e.js |
all passed |
Mutants (18, all killed) [T]
Each mutant was applied to the pushed tree, the named tests run, and the tree restored; the sweep script re-verifies the restore.
| # | Mutant | Finding | Killed by |
|---|---|---|---|
| G1-a | the scalar envelope check is disabled | G1 | TestInjectSpreadEnvelopeMustFitTheFrame |
| G1-b | botStreamKey's error is dropped at spec parse |
G1 | TestInjectSizeMixIsValidatedBeforeAnythingRuns |
| G1-c | only the median payload is validated, not the envelope | G1 | TestInjectSpreadEnvelopeMustFitTheFrame |
| G2-a | keys records the payload as unknown in every pair |
G2 | TestKeysReportsTheObservedSizeDistribution, TestPipelineKeysSpecInjectOnBlockQuantisedSizes |
| G2-b | the size draw always takes the first pair | G2 | TestInjectSizeMixDrawsOnlyObservedPairs |
| G2-c | size_mix orders payloads descending inside one wire size |
G2 | TestKeysReportsTheObservedSizeDistribution |
| G3-a | an infinite selected percentile clamps to the slowest detection | G3 | TestScoreSelectedTailCountsAMissAsInfinite |
| G3-b | the selected percentile ignores the misses in its rank | G3 | TestScoreSelectedTailCountsAMissAsInfinite |
| G5-a | baseline_masked is never set |
G5 | TestScoreReportsBaselineMaskingPerIncident |
| G5-b | per-incident baseline episodes are not counted | G5 | TestScoreReportsBaselineMaskingPerIncident |
| N1 | markBaseline touching end excluded |
G4 | TestScoreBaselineOverlapIsClosedAtBothEnds |
| N2 | markBaseline touching start excluded |
G4 | TestScoreBaselineOverlapIsClosedAtBothEnds |
| N3 | keys tie-break reversed |
G4 | TestKeysTieBreakIsTheSmallerCanonicalKey |
| N4 | spread clamp >= → > |
G4 | TestKeysSpreadIsClampedStrictlyBelowTheMedian |
| N6 | histPercentile seen > idx → >= |
G4 | TestKeysPercentilesAreNearestRank |
| N7 | keys first uses After |
G4 | TestKeysFirstAndLastAreTheExtremeTimes |
| N8 | Truncate → Round |
G4 | TestInjectTruncatesTowardsThePast |
| N11 | -top keeps top − 1 |
G4 | TestKeysTopReportsExactlyTopKeys |
Three of these (G1-a, G1-c, G2-c) survived my first pass, and I report that rather than only the final state. G1-a and G1-c survived because the two size guards are mutually redundant for the spread envelope; I did not delete one — the explicit check carries the diagnostic that round 2 lacked, and the validator check catches bounds the explicit one does not. So the tests were strengthened instead: the envelope error must name payload_bytes + payload_bytes_spread (which only the explicit check produces), and a payload of 180 ± 10 in a 255-byte frame fits wire − 2 but passes MaxPayloadBytes 184 (which only the validator check catches). G2-c survived because every pair in the first fixture had a distinct wire size, so the payload tie-break never decided anything; the fixture now has three pairs sharing one wire size.
§11 pipeline check (synthetic background, no real data) [T]
Reported in full in the pre-registration. 4,945 background lines over 12 route groups and 7 days, GRP_TXT-shaped sizes, the excluded byte 81 the busiest of all; 11 keys after the exclusion; 4,434 injected events, 18 bots, 20 incidents; all four score runs.
| Run | recall | TTD detected med / p75 / max | TTD selected med / p75 | median stop | eps/incident med / max |
|---|---|---|---|---|---|
| A (false alarms) | – | – | – | – | 0 episodes, 0 false alarms |
| B-shared (8 blind) | 6/8 = 0.75 | 7.5 P / 12.0 P / 14.0 P | 8.0 P / 14.0 P | 10 m 41 s | 1 / 3 |
| B-shared realistic (b5/b6/b7) | 3/3 | 8.0 P / 12.0 P / 12.0 P | 8.0 P / 12.0 P | 15 m 12 s | 1 / 3 |
| B-iso (8 blind, control) | 8/8 | 7.0 P / 7.0 P / 115.0 P | 7.0 P / 7.0 P | 15 m 10 s | 1 / 1 |
The two shared misses are the two longest clean periods (b3-600s, b4-900s): key sharing costs recall on the slow shapes while every isolated twin is found. None of the nine isolated keys appears in the full ranking, so §10's absence check passes with 0 contaminated. No bar in §9 is set from any of this.
Determinism and offline properties re-checked [T]
keystwice on the same input: byte-identical.injecttwice with the same spec, seed and background: byte-identicalinj.jsonlandlabels.json(covered byTestInjectIsDeterministic, still green).- A single-entry
size_mixconsumes no randomness, as a zero spread does, so adding it to a bot does not move any other draw of the train [K]. keys,injectandscoreimport no database driver and take no-dbflag;make_spec.pyis stdlib-only with no clock and no network.- No file outside
cmd/anomalydiffers fromorigin/masterin my four commits;git diff --name-only origin/master...HEADexcludingcmd/anomaly/andinternal/anomaly/is empty, sopublic/,cmd/server,cmd/ingestorandtest-e2e-playwright.jsare byte-identical to master. - All four commits are authored and committed by
dborup <kontakt@meshview.dk>; no closing keywords; no force-push. - Nothing touched staging or production, and no API key was used.
CI per job
Run 38816399459 (743d0f33), green in every job on attempt 2 [A].
| Job | Attempt 1 | Attempt 2 |
|---|---|---|
| ✅ Go Libraries & CLI | success | — |
| ✅ Go Server Tests (1/3), (2/3), (3/3) | success | — |
| ✅ Go Ingestor Tests | success | — |
| ✅ JavaScript Unit Tests & Lint | success | — |
| ✅ Script & Container Unit Checks | success | — |
| 🎭 Playwright E2E Tests (1/3) | success | — |
| 🎭 Playwright E2E Tests (2/3) | failure | success |
| 🎭 Playwright E2E Tests (3/3) | success | — |
| 📊 Verify E2E Shards & Coverage | failure (downstream) | success |
| 📋 Verify Go Test Assignments | success | — |
| 🧱 Validate Docker Image | success | — |
| ✅ Go Build & Test | success | — |
| 🏗️ Build & Publish Docker Image | failure (downstream) | success |
| 📦 Release Artifacts, 🚀 Deploy Staging, 📝 Publish Badges | skipped | skipped |
The one failure, named and checked rather than assumed. #382 node stats table rows follow the rounded corners in dark theme in test-e2e-playwright.js, asserting Table must not clip the help tooltip (overflow ) — note the empty computed overflow, which is a style not yet applied rather than a wrong style [A]. It is not #271 and not #301, so I did not treat it as one of the known flakes. It is nonetheless unrelated to this PR, on three independent grounds [T]:
test-e2e-playwright.js,public/andcmd/serverare byte-identical toorigin/masteron this head —git diff --stat origin/master...HEAD -- public/ test-e2e-playwright.js cmd/server/is empty.- This PR's whole diff is
cmd/anomalyandinternal/anomaly, neither of which is imported by any served code. - The same test, both themes, passes locally against a fixture-backed server on port 13841 on this exact tree.
The light-theme twin of the same test passed in the same CI run, which is the usual signature of a per-context style-application race rather than a CSS fault. I reran the failed job once, as instructed, and noted it here. I did not find a tracking issue for it and filed none; the fix (waiting for the stylesheet before reading computed style) belongs in its own PR.
Not verified
- No real data. No export, no snapshot, no staging, no production. Every number above comes from unit tests or from the synthetic §11 pipeline, so the real key-sharing rate, the real size distributions and the real false-alarm rate are all still unknown. G1's original failure on real V keys stays an inference from the protocol's 16-byte cipher blocks, now reproduced on two synthetic backgrounds and on the package's fixture, not on V itself.
- The round-2 reviewer's 16 mutants were not re-run individually, nor round 2's own 22; the 18 above were.
- Only three E2E suites ran locally, not the full CI shard set.
- The
-trimpathbinary hash in §2 is fromgo1.27.0 darwin/arm64; I did not check that a Linux or CI build reproduces it, only that the build is now pinned to-trimpathso it can be. size_mixis bounded at 2048 pairs per key and 4096 per bot. I did not test a key that overflows the bound on real data, only that the overflow is counted.- I did not re-audit the non-size fields of an injected line for distinguishability; that was round 2's work.
Relates to #474
Phases 1–3 of #474: the target definition, the detector-side channel exclusion and the injection-based evaluation tooling. Library and offline tool only — no ingestor, server, API or UI wiring, no defaults, no thresholds, no staging or production access. The pre-registration is posted on #474 before any data exists; the coordinator runs the evaluation later.
Plan (what this PR does, and in which order)
81, where an app posts a position every 75–123 s while someone drives — and the rule failed on recall (0.17).inject(synthetic bot trains over a real background, with ground truth) andscore(the anomaly phase 3: replay CLI, read-only export and blind validation (periodic, rate, new-stream) #457 §5 metrics against that ground truth, or without labels for the false-alarm arm).Phase 1 — target definition
internal/anomaly/doc.gogains a "What the periodic rule is for (#474)" section, andcmd/anomaly/README.mdan "Excluded channels" section. Both say plainly that a target definition, an exclusion list and an evaluation are three different things, and that only the evaluation can support a claim.Phase 2 — channel exclusion (
PeriodicRule.ExcludeChannels)internal/anomaly/testdata/replay_golden.jsonis unchanged.periodicAppliesgates both the per-scope "does any rule apply" pre-check and the rule loop, so an excluded channel forms no pulse, no chain and no episode, and claims no key state in the scope's budget (Stats.Keys).cmd/anomaly, but they touch different rule structs and different subcommands.81and the 2-byte hashes81aband8100are three different entries; a 1-byte entry is never read as the prefix of a wider hash, which is the stanceKey.Stringalready takes. The zero "no channel" value is not a valid entry and never matches.Newclones the slice, so the caller's config and the detector never share a backing array.doc.go, README): excluding the hash of#wardrivingalso excludes every other channel name that hashes to that byte — about 1/256 of all names — including any spam on them."exclude_channels": ["81"], plain lower-case hex, optional.The list starts empty in the repo.
81(#wardriving) is named in the pre-registration, not in any code default, and the owner can add more hashes by editing the config file alone.Phase 3 — injection-based evaluation
injectinj-<bot>-<pulse>-<part>. An export numbers its linestx1,tx2, …, so no real line can carry that marker, and a background line that already starts withinj-is refused rather than mixed in.-dbflag and opens no database: it reads a spec and a JSONL file and writes exactly two new files, neither of which may exist beforehand, and neither of which appears unless the whole run succeeds.math/rand, and each bot's stream is seeded from the spec seed and the bot's name, so adding, removing or reordering bots never changes another bot's events.jitter_maxand applied per pulse (no drift), missed pulses, multi-part pulses (1..N later parts within a delay) and outage holes. The README's example spec is the September bot's documented shape from anomaly: hold a periodic bot as one episode and detect when it stops (evaluate Cooldown, then train tracking) #130 (5 min, ~3 s jitter, ~30 % multi-part with parts up to 55 s behind, ~5 % missed, one 35-minute hole, one stable first hop).seed,payload_type,jitter_sd, the fractions,final_after) are pointers, so omitting one is an error instead of a silent zero. Unknown fields and trailing data are rejected.-labelsapplies the anomaly phase 3: replay CLI, read-only export and blind validation (periodic, rate, new-stream) #457 §4 periodic recipe to the emitted pulses: merge events closer thanpulse_merge_gapinto one pulse, take P as the median pulse gap in[min_period, max_period], and call an incident a maximal run of ≥min_pulsespulses whose every gap is withinmax(tolerance_abs, tolerance_rel·P)of k·P for k ∈ 1..max_harmonic. So a bot with a 35-minute hole at P = 5 min is two incidents, and a 7-pulse bot is none.scoreThe #457 §5 periodic matching, unchanged except that the labels come from
inject:EventID; activation is its first transition toactive, its end the last transition toended, or the report's end if it never ended; only episodes activated inside[-start, -end)are scored;[activation, end]overlapping[incident start, stop + 4P], and a period within the labelling tolerance of k·P for k ∈ 1..max_harmonic. An episode whose activation carries no periodic evidence can never be a true positive;-labels, every episode is one: that is the false-alarm arm, run on background-only traffic;-bots <names>narrows which incidents the recall and the medians cover, so the non-blind sanity-check bot (the September shape, which anomaly: hold a periodic bot as one episode and detect when it stops (evaluate Cooldown, then train tracking) #130 and anomaly: evaluate Cooldown for periodic bot episodes (#130) #447 were tuned on) is reported apart from the blind panel without a hand computation afterwards. It never turns a matched episode into a false alarm, and a filter that selects no labelled incident is an error.Metrics: episodes, true/false positives, false alarms per day, recall, median time to detect (also in units of P), median stop latency, median episodes per detected incident, per-incident detail, and false positives per channel hash.
Measurements from the new tests (not thresholds, not a validation)
On a synthetic 4-hour, 5-minute bot over a sparse synthetic background, with the #474 periodic parameters (
min_pulses 8,burst_gap 15s,jitter_abs 10s,max_missing 2, tracking K = 3, quiet 60 min):The second row is the September-like shape and it matters for the pre-registration: strays and jitter near the tolerance delay re-acquisition, exactly as #130 observed (first alarm 1 h 50 on its noisy fixture). Time to detect is a property of the rule and the bot's shape, not of the scorer; it is logged, not pinned, by the test.
Tests, per requirement
Every point below has a test that was red before the change and green after (
runInject/runScorestubbed out, and the detector-side mutants M1/M2 for phase 2), and at least one mutant:TestPeriodicExcludedChannelProducesNoCandidate,TestPeriodicExclusionCoversEveryListedChannel,TestPeriodicExclusionAtChannelScopeTestPeriodicExclusionMatchesTheWholeHashTestPeriodicExclusionKeepsNoKeyStateTestPeriodicExclusionLeavesOtherRulesAloneTestPeriodicExclusionIsCopiedFromTheCallerConfigExcludeChannelsnot cloned inNewTestPeriodicExclusionNeverMatchesNoChannelTestExcludeChannelsValidationexclude_channels(opt-in, strict hex)TestReplayConfigParsesExcludeChannels,…WithoutExcludeChannelsExcludesNothing,…RejectsBadExcludeChannelsTestReplayCommandMatchesTheLibraryGoldenTestInjectMarksAndMergesWithoutTouchingTheBackgroundTestInjectIsDeterministicTestInjectBotsAreIndependentTestInjectJitterIsClampedAndDoesNotDrift,TestInjectMissedPulses,TestInjectMultipartPulsesStayOnePulseTestInjectLabelsIncidentsFromThePulsesmin_pulsesoff by one; median replaced by the maximumTestInjectWritesOnlyItsTwoFiles,TestInjectMaxInjectedIsEnforced-max-injectednot enforcedTestInjectRefusesPreInjectedBackground,…RefusesUnreadableBackgroundLineTestInjectSpecRejectsMissingAndUnknownFields,TestInjectRequiresExplicitFlagsseedoptional; unknown fields accepted; duplicate bot names acceptedTestInjectLabelsAreKeyFreeAndMarkedSyntheticTestScoreRecallOnAnInjectedBot,…OnANoisyInjectedBotTestScoreFalseAlarmArmWithoutLabels,TestScoreFalsePositivesPerDayTestScoreExcludedChannelDetectsNothingTestScorePeriodCheck,TestScoreOverlapAndChannel,TestScoreWindowAndEpisodeGrouping,TestScoreOnlyTheNamedRule-outsafetyTestScoreRequiresExplicitFlagsAndValidInput,TestScoreWritesOutFileOnce-outoverwrites-botsselects which incidents the bars coverTestScoreBotsFilter51 of 51 mutants killed: 13 in the library, 20 in
inject, 18 inscore.Out of scope / not done here
cmd/serveris untouched.🤖 Generated with Claude Code