diff --git a/cmd/anomaly/README.md b/cmd/anomaly/README.md index 63ec501e8..bdaade2c8 100644 --- a/cmd/anomaly/README.md +++ b/cmd/anomaly/README.md @@ -1,8 +1,11 @@ # corescope-anomaly (experimental, offline) -Offline tool for the `internal/anomaly` study (#457, #473). It is not part of the -server or the ingestor, is not built into any image and has no defaults: -every bound is an explicit flag. +Offline tool for the `internal/anomaly` study (#457, #473, #474). It is not +part of the server or the ingestor, is not built into any image and has no +defaults: every bound is an explicit flag. + +Only `export` reads a database, read-only. Nothing here ever writes to a +CoreScope database: `inject` and `score` read and write plain files. ```sh cd cmd/anomaly && go build -o corescope-anomaly . @@ -147,9 +150,335 @@ corescope-anomaly replay -config config.json -input events.jsonl \ ``` The config is the replay JSON config (`anomaly.ParseReplayConfig`). Periodic -rules accept the optional `end_after_missed_pulses` (train tracking, #447). -Leaving it out means 0, which turns tracking off. Rate rules accept the -optional `exclude_ingest_backlog` (#473). With `true`, the rule ignores lines -with `"ingest_backlog": true`: they are not counted in its window, never -trigger it and do not keep its episodes open. Leaving it out means `false`, -and then the flag changes nothing. +rules accept two optional fields: + +- `end_after_missed_pulses` — train tracking (#447). Leaving it out means 0, + which turns tracking off. +- `exclude_channels` — channel hashes the rule ignores, as plain lower-case + hex (`["81"]`). Leaving it out excludes nothing. See + [Excluded channels](#excluded-channels). + +Rate rules accept the optional `exclude_ingest_backlog` (#473). With `true`, +the rule ignores lines with `"ingest_backlog": true`: they are not counted in +its window, never trigger it and do not keep its episodes open. Leaving it out +means `false`, and then the flag changes nothing. + +## Excluded channels + +The periodic rule's target is an abusive channel bot (#474). Automation that +is *expected* on its channel is not a target: a wardriving app posts a +position every 75-123 s while someone drives, and the #457 round-4 labels +show that is what the rule was measuring. + +`exclude_channels` names the hashes whose automation is expected. An event on +such a channel never reaches that rule, so it forms no pulse, no chain and no +episode, and the channel claims no key state. + +Two properties worth stating plainly: + +- **A channel hash is not a channel.** MeshCore's hash is one byte, so + excluding the hash of `#wardriving` also excludes every other channel name + that hashes to the same byte (about 1/256 of all names), including any spam + on them. That is the price of keying on an opaque 1-byte hash. +- **The match is on the whole hash**, length included: the 1-byte hash `81` + and the 2-byte hash `81ab` are different entries, and a 1-byte entry is + never read as the prefix of a wider hash. + +There are no default exclusions anywhere: an unconfigured rule behaves +exactly as it did before #474, and the replay golden file is unchanged. + +## keys + +Ranks the stream keys of an exported background by row count, so an +injection panel can be placed on keys real traffic actually uses. + +```sh +corescope-anomaly keys -input bg.jsonl -payload-type 5 -top 16 \ + -exclude-channels 81 -max-keys 2000000 -max-line-bytes 65536 -out keys.json +``` + +**Why it exists.** The stream key is payload type + channel hash + route +class + route kind + first hop. A bot on a channel byte and a first hop no +real transmission carries has that key to itself, so the detector sees a +clean train and recall measured on it says little about finding a bot inside +real traffic: a real spam bot on a busy channel shares its key with everyone +else entering through the same repeater. + +**Blind by construction.** `keys` reads the export and counts rows. It never +runs the detector, reads no report, no labels and no metrics, and has no idea +which keys would alarm. The ranking is row counts, ties broken by the +canonical key encoding, so a panel built from its output is chosen +mechanically and before any detection result exists. + +**The filter is mechanical** and is restated in the output, with a counter +per reason a line was skipped: the payload type must be `-payload-type`, the +line must carry a channel hash (the periodic rule's target is a channel bot), +its route kind must be `relay` (only then is the first hop part of the stream +key, so only then can a bot share the key), and its channel must not be in +`-exclude-channels`. + +Per key the output gives the canonical `key`, the `channel`, `route_class`, +`route_kind` and `first_hop` an injection spec needs to land on it, the most +common whole route (`full_route`, ties broken by the smallest encoding), the +key's whole observed size distribution (`size_mix`), the median wire size, +the median payload size and a `payload_bytes_spread` of (p90 − p10)/2, the +row count and the first and last times. Percentiles are nearest-rank and the +spread is clamped strictly below the median. `first` and `last` are the +earliest and latest line on the key, whatever order the input is in. +Per-key size and route variants are bounded and the overflow is reported per +key; `-max-keys` is a hard bound on distinct keys, as the export's +`-max-rows` is on rows. `-top` reports exactly that many keys, so the full +ranking (a large `-top`) is what an absence check should read. + +**`size_mix` is the distribution to inject from.** It is one entry per +distinct `(wire_bytes, payload_bytes)` pair with the number of lines that +carried it, ordered by wire size then payload size; `payload_bytes` is absent +for the lines whose payload length the export did not know. The two sizes are +*not* independent: a frame is header plus path plus payload, so +`wire − payload` is the route's own overhead, while the payload itself comes +in 16-byte cipher blocks. Drawing them separately — a fixed median wire with +a spread payload — both produces frames real traffic never carries and, once +the drawn payload passes `wire − 2`, frames the event validator refuses +outright. So `inject` takes whole observed pairs (see `size_mix` under +Spec). + +The output carries hashes, hop prefixes, sizes, counts and times — what the +export itself carries — and no ids, observers, names or text. + +## inject + +Overlays synthetic channel-bot trains on exported background lines, so recall +can be measured when real spam is rare. It writes two new files and nothing +else. + +```sh +corescope-anomaly inject -spec spec.json -input events.jsonl \ + -out events-injected.jsonl -labels labels.json \ + -max-injected 100000 -max-line-bytes 65536 +``` + +- **Clearly synthetic.** Every injected line's `id` is + `inj---`; an export numbers its lines `tx1`, `tx2`, …, so + no real line can carry that marker. A background line that already starts + with `inj-` is refused, so the marker stays unambiguous. +- **Never written back.** The tool takes no `-db` flag and opens no database. + Injected events exist only in the output file; they must never be imported + into any CoreScope database. +- **File-only and deterministic.** Same spec, same seed, same background → + byte-identical output and labels. The generator is splitmix64 inside this + tool, not `math/rand`, and each bot's stream is seeded from the spec seed + and the bot's name, so adding, removing or reordering bots never changes + another bot's events. +- **Background untouched.** Background lines are copied through byte for + byte, in order; injected lines are merged in by data time, so the output is + a valid replay input. +- `-out` and `-labels` must not exist, and neither appears unless the whole + run succeeds. `-max-injected` is a hard bound. +- The summary (counts and times only) is printed as JSON on stderr. + +### Spec + +Every field is required; there are no defaults. Unknown fields are rejected. +The example is the shape of the September bot documented in #130 (5 min, +~3 s jitter, ~30 % multi-part pulses with later parts up to 55 s behind, ~5 % +single missed pulses, one 35-minute hole, one stable first hop). + +```json +{ + "seed": 20261015, + "final_after": "30s", + "labelling": {"pulse_merge_gap": "60s", "min_period": "1m", "max_period": "1h", + "min_pulses": 8, "max_harmonic": 3, + "tolerance_abs": "30s", "tolerance_rel": 0.1}, + "bots": [{ + "name": "sept-shape", + "channel": "7f", + "payload_type": 5, + "route_class": "flood", + "route_kind": "relay", + "first_hop": "1:fd", + "full_route": "1:fd0a1c", + "size_mix": [{"wire_bytes": 38, "payload_bytes": 35, "count": 70}, + {"wire_bytes": 54, "payload_bytes": 51, "count": 20}, + {"wire_bytes": 70, "payload_bytes": 67, "count": 10}], + "start": "2026-10-16T00:00:00Z", + "stop": "2026-10-19T13:12:00Z", + "period": "5m", + "jitter_sd": "3s", + "jitter_max": "10s", + "missed_fraction": 0.05, + "multipart_fraction": 0.3, + "multipart_max_extra": 2, + "multipart_max_delay": "55s", + "holes": [{"start": "2026-10-17T04:00:00Z", "duration": "35m"}] + }] +} +``` + +- `final_after` is added to each event time to give its `final_at`; set it to + the export's `-settle`, so injected lines are final like the background's. + Like `-settle`, it must be whole seconds. +- **Times are whole seconds.** Every injected event time is truncated to a + whole second, as the export's are (`time.Unix(ts, 0)`), so a fractional + second never gives an injected line away. Truncation moves a time by less + than a second towards the past, so the effective offset from the grid is in + `[-(jitter_max + 1s), jitter_max]`. +- **The sizes are described in exactly one of two ways**, and a spec must + pick one. + - `size_mix` is the realistic one, and what `keys` emits per key: the + observed `(wire_bytes, payload_bytes)` pairs with a count each. One whole + pair is drawn per event, weighted by count, so the injected sizes are + exactly the sizes the key carries — valid by construction, and nothing an + injected line can be recognised by. With `size_mix`, `wire_bytes`, + `payload_bytes` and `payload_bytes_spread` must be left out. + - `wire_bytes` with `payload_bytes` and `payload_bytes_spread` is the older + scalar description, for hand-written specs: a fixed wire size and a + payload drawn uniformly from `payload_bytes ± spread`. 0 keeps the size + constant. It cannot reproduce a real distribution, so the **whole + envelope** is required to fit the frame: the spread must be smaller than + `payload_bytes`, and `payload_bytes + payload_bytes_spread` must be at + most `wire_bytes − 2`, which is what `Event.Validate` allows. + Either way, every size a bot can draw is checked against the event + validator while the spec is read — before a single event, line or file is + produced — so an impossible size is a spec error and never an abort part + way through a run. +- `jitter_sd` is the standard deviation of the offset from the nominal grid + time, clamped to `±jitter_max`. The offset is not cumulative, so the train + does not drift. +- **Take the jitter from measurement, not from the rule.** Scaling + `jitter_sd` to the rule's own per-gap tolerance places every bot inside + that tolerance by construction, which makes recall conditional on the rule + being evaluated and moves the panel whenever the rule moves. #130 measured + a real 5-minute bot over 85.2 h: `|gap − 300 s|` median 3 s, p95 9 s. A gap + offset is the difference of two per-pulse offsets, so its standard + deviation is √2 times the per-pulse one, and p95 9 s on a normal gap offset + gives sd(gap) ≈ 9/1.96 ≈ 4.6 s and a per-pulse sd of ≈ 3.2 s. So + `jitter_sd` 3 s with `jitter_max` 9 s (#130's measured p95) is the + measured shape, for every period, independent of any rule value. Some + pulses of a short-period bot then do fall outside a tight tolerance, which + is the point: it is the rule's job to cope, not the panel's to avoid. +- `missed_fraction` drops pulses other than the first; `multipart_fraction` + gives a pulse 1..`multipart_max_extra` later parts within + `multipart_max_delay`. A hole drops every pulse inside it. +- `channel`, `first_hop`, `full_route`, `payload_type` and the sizes are the + wire fields of a replay line: an injected line carries nothing a real line + does not. `full_route` is a whole observed route, not just the first hop + repeated, so an injected line's route looks like an exported one's. + +### make_spec.py + +`make_spec.py` beside this README is the #474 round-3 panel generator: it +takes the validation window's start and a `keys.json` ranking and prints the +spec for the pre-registered paired panel (nine shapes, each on the key of +rank *i*+1 and on an isolated twin key). It is stdlib-only, reads nothing +else and has no clock, so the spec is a pure function of its two inputs and +can be re-derived from them afterwards. + +It lives in the repository rather than only in the pre-registration comment +so that the whole chain — `export` → `keys` → `make_spec.py` → `inject` — is +exercised by the tool's own tests on a background whose payload sizes are +block-quantised like real GRP_TXT traffic. That chain is where round 2 broke: +the spec it produced could not be injected at all on a realistic key. + +```sh +python3 make_spec.py 2026-10-17T00:00:00Z keys.json > spec.json +``` + +### Labels + +`-labels` holds the ground truth of the run, computed from the emitted pulses +with the `labelling` recipe, which is the #457 §4 periodic recipe: + +1. events closer than `pulse_merge_gap` are one pulse, timed at its first + event; +2. P is the median pulse gap inside `[min_period, max_period]`; +3. an incident is a maximal run of at least `min_pulses` pulses whose every + gap is within `max(tolerance_abs, tolerance_rel * P)` of k·P for some + k in 1..`max_harmonic`. + +So a bot with a long hole is two incidents, and a bot with fewer than +`min_pulses` pulses is none — the ground truth is what the recipe says, not +"one bot, one incident". The file holds channel hashes, times, periods and +counts: no ids, routes, hops, observers, names or text. + +## score + +Measures a replay report against injected labels, or against no labels at +all. + +```sh +corescope-anomaly score -report report.json -labels labels.json \ + -baseline report-a.json \ + -rule p474_periodic -start 2026-10-16T00:00:00Z -end 2026-10-23T00:00:00Z \ + [-bots b0-90s-clean,b1-120s-clean] [-out metrics.json] +``` + +The matching is the #457 §5 periodic rule: + +- An **episode** is one `EventID`. Its activation is its first transition to + `active`; its end is its last transition to `ended`, or the report's end if + it never ended. Only episodes activated inside `[-start, -end)` are scored. +- An episode is a **true positive** when it is the **same series** as the + incident, `[activation, end]` overlaps `[incident start, stop + 4P]`, and + the episode's period is within the labelling tolerance of k·P for some + k in 1..`max_harmonic`. An episode whose activation carries no periodic + evidence can never be a true positive. +- **Same series** means the same stream key when the labels carry one (they + do since #474 round 2), and the same channel hash otherwise. The key is + what makes a panel placed on real traffic's own keys scorable: on a busy + channel several route groups share the 1-byte hash, and matching on the + hash alone would credit any of them to the bot. +- `-baseline` takes the replay report of the **same window without the + injection** (arm A). An episode it already has, on the same key and + overlapping in time, is attributed to the background: it is neither a true + positive nor a false positive, and is counted as `baseline_episodes`. That + is the differential reading a shared key needs — a detection in the + injected arm counts only when it is absent from the plain arm. +- Every other episode is a **false positive**. Without `-labels` every + episode is one, which is the false-alarm arm: run it on background-only + traffic to get false alarms per day. +- `-bots` narrows which incidents the recall and the medians cover - for a + non-blind sanity-check bot, say. It never turns a matched episode into a + false alarm: an episode matching any labelled incident stays a true + positive, selected or not. A filter that selects no incident is an error. + +The metrics are episodes, true and false positives, false alarms per day, +recall, time to detect (also in units of P), stop latency, episodes per +detected incident, the per-incident detail and the false positives per +channel hash. Times to detect and stop latencies are properties of the rule +and the bot's shape, not of the scorer: a noisy train is found much later +than a clean one. + +**The median hides the tail.** A clean train alarms at the earliest pulse +`min_pulses` allows, so a panel of mostly clean bots puts the median on that +floor while the realistic shapes take many times longer. Every median is +therefore reported beside its nearest-rank **p75** and its **maximum**, and +`-bots` can narrow a run to the realistic subset so that subset gets its own +numbers. Read the tail, not only the median. + +**A tail over the detected incidents cannot fail on a missed bot.** The +medians, p75s and maxima above cover the *detected* incidents only, so a +panel whose four cleanest shapes alarm at the floor and whose four realistic +ones are never found reports recall 0.50 and a p75 of one period — a pass of +every timing bar, on a result that found nothing realistic. So the time to +detect is also reported over every **selected** incident with a miss ranked +above every finite value: +`median_time_to_detect_selected` / `p75_time_to_detect_selected`, with +`_periods` beside each. JSON has no infinity, so a percentile whose rank +falls among the misses is reported as `..._selected_infinite: true` with the +value left out — which fails any bar placed on it. `incidents_missed` counts +the selected incidents that were not detected. A bar belongs on the selected +tail; the detected-only tail describes the shapes that were found. + +**Baseline masking is visible per incident.** A `-baseline` episode is not +scored either way, so without a marker an incident whose only episode was +absorbed by the background reads as a plain miss. Each per-incident row +therefore carries `baseline_episodes` (the episodes on the incident's own +series, overlapping its window, that the plain arm already had) and +`baseline_masked` (not detected, but the background already had an episode +here); `incidents_baseline_masked` counts them. That tells a no-go caused by +background contamination apart from a detector miss. + +Provenance (`labels`, `baseline`) is recorded as the file names only, never +the paths they were read from: a path can carry a user or a host name, and a +metrics file may be shared. diff --git a/cmd/anomaly/inject.go b/cmd/anomaly/inject.go new file mode 100644 index 000000000..19019cd54 --- /dev/null +++ b/cmd/anomaly/inject.go @@ -0,0 +1,932 @@ +package main + +import ( + "bufio" + "encoding/hex" + "encoding/json" + "errors" + "flag" + "fmt" + "hash/fnv" + "io" + "math" + "os" + "sort" + "strings" + "time" + + "github.com/meshcore-analyzer/anomaly" +) + +// Issue #474, phase 3: overlay synthetic channel-bot trains on a background +// of real exported events, so recall can be measured when real spam is rare. +// +// The tool is offline and file-only. It opens no database, takes no -db +// flag, touches no network and reads no wall clock: it reads a spec and a +// background JSONL file and writes a new JSONL file plus a labels file. +// Nothing it produces is ever written back to a CoreScope database. +// +// Every injected event is marked: its id is "inj---", a +// value no export can produce (export ids are tx1, tx2, ...), and the tool +// refuses a background line whose id already starts with that prefix. A +// replay report therefore names injected traffic in its TriggerIDs, and the +// labels file says exactly which incidents were injected. + +// injPrefix marks every injected event id. +const injPrefix = "inj-" + +// maxInjectBots and maxInjectHoles bound one spec. +const ( + maxInjectBots = 64 + maxInjectHoles = 64 + // maxInjectSizeMix bounds one bot's size distribution, and + // maxInjectSizeCount one entry's weight, so the cumulative weights + // cannot overflow. + maxInjectSizeMix = 4096 + maxInjectSizeCount = 1 << 32 +) + +// injectSpec is the JSON spec of one injection run. There are no defaults: +// every field must be present, and fields whose zero value is meaningful +// (seed, payload type, jitter, fractions) are pointers so that leaving one +// out is an error rather than a silent zero. +type injectSpec struct { + Seed *uint64 `json:"seed"` + // FinalAfter is added to each injected event time to give its final_at, + // as the export's -settle does for real events. + FinalAfter *dur `json:"final_after"` + Labelling labellingSpec `json:"labelling"` + Bots []injectBot `json:"bots"` +} + +// labellingSpec is the ground-truth recipe applied to the injected pulses +// (#457 §4, periodic): events closer than PulseMergeGap are one pulse, P is +// the median pulse gap inside [MinPeriod, MaxPeriod], and an incident is a +// maximal run of at least MinPulses pulses whose every gap is within +// max(ToleranceAbs, ToleranceRel*P) of k*P for some k in 1..MaxHarmonic. +type labellingSpec struct { + PulseMergeGap dur `json:"pulse_merge_gap"` + MinPeriod dur `json:"min_period"` + MaxPeriod dur `json:"max_period"` + MinPulses int `json:"min_pulses"` + MaxHarmonic int `json:"max_harmonic"` + ToleranceAbs dur `json:"tolerance_abs"` + ToleranceRel float64 `json:"tolerance_rel"` +} + +// injectBot describes one synthetic bot. Times are data times, as in the +// export; the fields that describe the wire are exactly the fields of a +// replay record, so an injected line carries nothing a real line does not. +type injectBot struct { + Name string `json:"name"` + Channel string `json:"channel"` + PayloadType *int `json:"payload_type"` + RouteClass string `json:"route_class"` + RouteKind string `json:"route_kind"` + FirstHop string `json:"first_hop"` + FullRoute string `json:"full_route"` + // The sizes are described in exactly one of two ways, and the spec must + // pick one. + // + // SizeMix is the realistic one, and what `keys` produces: the key's own + // observed (wire_bytes, payload_bytes) pairs with a count each. One + // whole pair is drawn per event, weighted by count. Because every pair + // came off a real line, it is valid by construction and carries the + // route's own wire-minus-payload overhead, so an injected line cannot be + // told from a real one on size. + // + // WireBytes with PayloadBytes and PayloadBytesSpread is the older, + // scalar description, kept for hand-written specs: a fixed wire size and + // a payload drawn uniformly from payload_bytes +/- the spread. It cannot + // reproduce a real distribution - the payload comes in cipher blocks + // while wire - payload is header plus path - so the whole envelope is + // required to fit the frame (payload_bytes + spread <= wire_bytes - 2); + // a spec that could overflow is refused here rather than aborting the + // run on its first wide draw. + WireBytes int `json:"wire_bytes"` + PayloadBytes *int `json:"payload_bytes"` + PayloadBytesSpread *int `json:"payload_bytes_spread"` + SizeMix []sizePair `json:"size_mix"` + Start time.Time `json:"start"` + Stop time.Time `json:"stop"` + Period dur `json:"period"` + // JitterSD is the standard deviation of the per-pulse offset from the + // nominal grid time; JitterMax clamps it. The offset is not cumulative, + // so the train does not drift. + JitterSD *dur `json:"jitter_sd"` + JitterMax dur `json:"jitter_max"` + // MissedFraction is the probability that a pulse other than the first is + // dropped (one missed pulse, as a real bot's). + MissedFraction *float64 `json:"missed_fraction"` + // MultipartFraction is the probability that a pulse is multi-part; + // 1..MultipartMaxExtra later parts then follow it within + // MultipartMaxDelay. + MultipartFraction *float64 `json:"multipart_fraction"` + MultipartMaxExtra *int `json:"multipart_max_extra"` + MultipartMaxDelay dur `json:"multipart_max_delay"` + // Holes are data-time ranges where the bot sends nothing (an outage). + Holes []injectHole `json:"holes"` +} + +type injectHole struct { + Start time.Time `json:"start"` + Duration dur `json:"duration"` +} + +// dur is a JSON duration written and read as a string ("5m0s"), so a spec +// and the labels file it produces round-trip through the same form. +type dur time.Duration + +func (d dur) MarshalJSON() ([]byte, error) { return json.Marshal(time.Duration(d).String()) } + +func (d *dur) UnmarshalJSON(b []byte) error { + var s string + if err := json.Unmarshal(b, &s); err != nil { + return errors.New("durations are strings such as \"5m\"") + } + v, err := time.ParseDuration(s) + if err != nil { + return err + } + *d = dur(v) + return nil +} + +// prng is splitmix64: a fixed, self-contained generator, so an injection is +// reproducible from the spec alone and independent of the Go version's +// math/rand. +type prng struct{ s uint64 } + +func (p *prng) next() uint64 { + p.s += 0x9e3779b97f4a7c15 + z := p.s + z = (z ^ (z >> 30)) * 0xbf58476d1ce4e5b9 + z = (z ^ (z >> 27)) * 0x94d049bb133111eb + return z ^ (z >> 31) +} + +// float returns a uniform value in [0,1). +func (p *prng) float() float64 { return float64(p.next()>>11) / (1 << 53) } + +// normal returns a standard normal value (Box-Muller, cosine branch). +func (p *prng) normal() float64 { + u1 := 1 - p.float() // in (0,1] + u2 := p.float() + return math.Sqrt(-2*math.Log(u1)) * math.Cos(2*math.Pi*u2) +} + +// botPRNG seeds one bot's stream from the spec seed and the bot's name, so +// each bot's output depends only on the seed and its own fields: adding, +// removing or reordering bots never changes another bot's events. +func botPRNG(seed uint64, name string) *prng { + h := fnv.New64a() + _, _ = h.Write([]byte(name)) + p := &prng{s: seed ^ h.Sum64()} + p.next() // discard one value, so a zero seed is not a degenerate start + return p +} + +// botStats counts what one bot produced. +type botStats struct { + Name string `json:"name"` + Channel string `json:"channel"` + // Key is the stream key every one of this bot's events falls on, in the + // library's canonical encoding. The evaluation needs it: a bot placed on + // a key real traffic uses shares that key with the background, and + // scoring on the channel hash alone could then credit a real train to + // the bot (or the other way round). + Key string `json:"key"` + PulsesScheduled int `json:"pulses_scheduled"` + PulsesEmitted int `json:"pulses_emitted"` + PulsesMissed int `json:"pulses_missed"` + PulsesInHole int `json:"pulses_in_hole"` + MultipartPulses int `json:"multipart_pulses"` + Events int `json:"events"` + First time.Time `json:"first,omitempty"` + Last time.Time `json:"last,omitempty"` + Incidents int `json:"incidents"` +} + +// injectSummary is printed on stderr (counts and times only). +type injectSummary struct { + Experimental bool `json:"experimental"` + BackgroundLines int `json:"background_lines"` + BackgroundOutOfOrder int `json:"background_out_of_order"` + InjectedEvents int `json:"injected_events"` + LinesWritten int `json:"lines_written"` + Bots []botStats `json:"bots"` +} + +// Incident is one labelled ground-truth incident: a channel hash and a time +// range, with the bot's period. It holds no key material, no observer +// identity and no text. +type Incident struct { + Bot string `json:"bot"` + Channel string `json:"channel"` + // Key is the bot's stream key (see botStats.Key). score matches on it + // when it is present, so an episode of real traffic on the same channel + // but a different route group is never mistaken for the bot. + Key string `json:"key"` + Start time.Time `json:"start"` + Stop time.Time `json:"stop"` + Period dur `json:"period"` + Pulses int `json:"pulses"` +} + +// InjectLabels is the labels file: the ground truth of one injection run. +type InjectLabels struct { + Experimental bool `json:"experimental"` + Synthetic bool `json:"synthetic"` + Seed uint64 `json:"seed"` + Labelling labellingSpec `json:"labelling"` + Bots []botStats `json:"bots"` + Incidents []Incident `json:"incidents"` +} + +// injEvent is one generated event before it becomes a line. +type injEvent struct { + rec anomaly.ReplayRecord + t int64 // rec.Time in nanoseconds, for sorting + pulse int +} + +func runInject(args []string, stdin io.Reader, stderr io.Writer) int { + fs := flag.NewFlagSet("inject", flag.ContinueOnError) + fs.SetOutput(stderr) + specPath := fs.String("spec", "", "injection spec JSON (required)") + input := fs.String("input", "", "background ReplayRecord JSON Lines, or - for stdin (required; /dev/null for none)") + out := fs.String("out", "", "merged JSON Lines output (required, must not exist)") + labels := fs.String("labels", "", "ground-truth labels JSON (required, must not exist)") + maxInjected := fs.Int("max-injected", 0, "most injected events allowed (required)") + maxLine := fs.Int("max-line-bytes", 0, "longest accepted background line (required)") + if err := fs.Parse(args); err != nil { + return exitUsage + } + usageErr := func(err error) int { + fmt.Fprintf(stderr, "inject: %v\n", err) + return exitUsage + } + if fs.NArg() != 0 { + return usageErr(errors.New("unexpected arguments")) + } + if *specPath == "" || *input == "" || *out == "" || *labels == "" { + return usageErr(errors.New("-spec, -input, -out and -labels are required")) + } + if *maxInjected <= 0 || *maxLine <= 0 { + return usageErr(errors.New("-max-injected and -max-line-bytes must be > 0")) + } + fail := func(err error) int { + fmt.Fprintf(stderr, "inject: %v\n", err) + return exitFail + } + sb, err := os.ReadFile(*specPath) + if err != nil { + return fail(err) + } + spec, err := parseInjectSpec(sb) + if err != nil { + return fail(err) + } + evs, stats, incidents, err := generate(spec, *maxInjected) + if err != nil { + return fail(err) + } + in := stdin + if *input != "-" { + f, err := os.Open(*input) + if err != nil { + return fail(err) + } + defer f.Close() + in = f + } + // Both outputs must be new files, and neither appears unless the whole + // run succeeds. + outF, err := createNew(*out) + if err != nil { + return fail(err) + } + sum, err := mergeLines(in, outF, evs, *maxLine) + if cerr := outF.Close(); err == nil { + err = cerr + } + if err != nil { + os.Remove(*out) + return fail(err) + } + sum.Bots = stats + lb, err := json.MarshalIndent(InjectLabels{Experimental: true, Synthetic: true, Seed: *spec.Seed, + Labelling: spec.Labelling, Bots: stats, Incidents: incidents}, "", " ") + if err == nil { + err = writeNew(*labels, append(lb, '\n')) + } + if err != nil { + os.Remove(*out) + return fail(err) + } + b, err := json.Marshal(sum) + if err != nil { + return fail(err) + } + fmt.Fprintf(stderr, "%s\n", b) + return exitOK +} + +// parseInjectSpec decodes and validates a spec (unknown fields and trailing +// data rejected). +func parseInjectSpec(b []byte) (injectSpec, error) { + var s injectSpec + dec := json.NewDecoder(strings.NewReader(string(b))) + dec.DisallowUnknownFields() + if err := dec.Decode(&s); err != nil { + return injectSpec{}, fmt.Errorf("inject spec: %w", err) + } + if !onlyJSONValue(dec, b) { + return injectSpec{}, errors.New("inject spec: trailing data") + } + if s.Seed == nil { + return injectSpec{}, errors.New("inject spec: seed is required") + } + if s.FinalAfter == nil || *s.FinalAfter < 0 || time.Duration(*s.FinalAfter) > time.Hour || + time.Duration(*s.FinalAfter)%time.Second != 0 { + return injectSpec{}, errors.New("inject spec: final_after is required and must be whole seconds in 0..1h") + } + if err := s.Labelling.validate(); err != nil { + return injectSpec{}, err + } + if len(s.Bots) == 0 || len(s.Bots) > maxInjectBots { + return injectSpec{}, fmt.Errorf("inject spec: need 1..%d bots", maxInjectBots) + } + seen := map[string]bool{} + for i := range s.Bots { + if err := s.Bots[i].validate(); err != nil { + return injectSpec{}, err + } + if seen[s.Bots[i].Name] { + return injectSpec{}, fmt.Errorf("inject spec: duplicate bot name %q", s.Bots[i].Name) + } + seen[s.Bots[i].Name] = true + // Every size the bot can draw has to make a valid event, and the + // check belongs here: a spec that could overflow the frame is + // refused outright rather than aborting a run after thousands of + // lines, which is what round 2 did on a realistic key. + if _, err := botStreamKey(&s.Bots[i], time.Duration(*s.FinalAfter)); err != nil { + return injectSpec{}, err + } + } + return s, nil +} + +func (l labellingSpec) validate() error { + switch { + case time.Duration(l.PulseMergeGap) <= 0: + return errors.New("inject spec: labelling.pulse_merge_gap must be > 0") + case time.Duration(l.MinPeriod) <= 0 || time.Duration(l.MaxPeriod) <= time.Duration(l.MinPeriod): + return errors.New("inject spec: labelling needs 0 < min_period < max_period") + case l.MinPulses < 2: + return errors.New("inject spec: labelling.min_pulses must be >= 2") + case l.MaxHarmonic < 1 || l.MaxHarmonic > 16: + return errors.New("inject spec: labelling.max_harmonic must be in 1..16") + case time.Duration(l.ToleranceAbs) < 0: + return errors.New("inject spec: labelling.tolerance_abs must be >= 0") + case !(l.ToleranceRel >= 0 && l.ToleranceRel <= 1): + return errors.New("inject spec: labelling.tolerance_rel must be in 0..1") + case time.Duration(l.ToleranceAbs) == 0 && l.ToleranceRel == 0: + return errors.New("inject spec: labelling needs a tolerance") + } + return nil +} + +// validBotName allows the characters a rule name allows, so an injected id +// stays printable and greppable. +func validBotName(s string) bool { + if len(s) == 0 || len(s) > 32 { + return false + } + for _, c := range s { + if !(c >= 'a' && c <= 'z' || c >= '0' && c <= '9' || c == '_' || c == '-' || c == '.') { + return false + } + } + return true +} + +func (b *injectBot) validate() error { + name := b.Name + bad := func(what string) error { return fmt.Errorf("inject spec: bot %q: %s", name, what) } + if !validBotName(b.Name) { + return fmt.Errorf("inject spec: bot names must be 1-32 chars of [a-z0-9_.-], got %q", b.Name) + } + if b.Channel == "" || strings.ToLower(b.Channel) != b.Channel { + return bad("channel must be lower-case hex") + } + if _, err := hex.DecodeString(b.Channel); err != nil { + return bad("channel must be hex") + } + switch { + case b.PayloadType == nil: + return bad("payload_type is required") + case b.MissedFraction == nil || *b.MissedFraction < 0 || *b.MissedFraction > 1: + return bad("missed_fraction is required and must be in 0..1") + case b.MultipartFraction == nil || *b.MultipartFraction < 0 || *b.MultipartFraction > 1: + return bad("multipart_fraction is required and must be in 0..1") + case b.MultipartMaxExtra == nil || *b.MultipartMaxExtra < 0 || *b.MultipartMaxExtra > 8: + return bad("multipart_max_extra is required and must be in 0..8") + case b.JitterSD == nil || *b.JitterSD < 0: + return bad("jitter_sd is required and must be >= 0") + case b.SizeMix != nil: + if err := b.validateSizeMix(bad); err != nil { + return err + } + case b.WireBytes <= 0: + return bad("size_mix or wire_bytes is required, and wire_bytes must be > 0") + case b.PayloadBytes != nil && *b.PayloadBytes <= 0: + return bad("payload_bytes must be > 0 when present") + case b.PayloadBytesSpread == nil || *b.PayloadBytesSpread < 0: + return bad("payload_bytes_spread is required and must be >= 0") + case *b.PayloadBytesSpread > 0 && b.PayloadBytes == nil: + return bad("payload_bytes_spread needs payload_bytes") + case b.PayloadBytes != nil && *b.PayloadBytesSpread >= *b.PayloadBytes: + return bad("payload_bytes_spread must be < payload_bytes") + case b.PayloadBytes != nil && *b.PayloadBytes+*b.PayloadBytesSpread > b.WireBytes-2: + // The top of the draw range, not just payload_bytes, has to fit the + // frame: Event.Validate refuses PayloadBytes > WireBytes - 2. + return bad("payload_bytes + payload_bytes_spread must fit the frame (<= wire_bytes - 2)") + } + switch { + case b.Start.IsZero() || b.Stop.IsZero() || !b.Stop.After(b.Start): + return bad("start and stop are required, with stop after start") + case time.Duration(b.Period) <= 0: + return bad("period must be > 0") + case time.Duration(b.JitterMax) < 0 || time.Duration(b.JitterMax) >= time.Duration(b.Period)/2: + return bad("jitter_max must be in [0, period/2)") + case time.Duration(b.MultipartMaxDelay) < 0: + return bad("multipart_max_delay must be >= 0") + case *b.MultipartFraction > 0 && (*b.MultipartMaxExtra == 0 || time.Duration(b.MultipartMaxDelay) == 0): + return bad("multipart_fraction needs multipart_max_extra and multipart_max_delay") + case len(b.Holes) > maxInjectHoles: + return bad("too many holes") + } + for _, h := range b.Holes { + if h.Start.IsZero() || time.Duration(h.Duration) <= 0 { + return bad("every hole needs a start and a duration > 0") + } + } + return nil +} + +// validateSizeMix checks a bot's size distribution. The pairs themselves are +// checked against the event validator in botStreamKey, which sees the bot's +// whole wire shape; here only the shape of the list is checked, plus the +// mutual exclusion with the scalar size fields. +func (b *injectBot) validateSizeMix(bad func(string) error) error { + if b.WireBytes != 0 || b.PayloadBytes != nil || b.PayloadBytesSpread != nil { + return bad("size_mix is the whole size description: leave wire_bytes, payload_bytes and payload_bytes_spread out") + } + if len(b.SizeMix) == 0 || len(b.SizeMix) > maxInjectSizeMix { + return bad(fmt.Sprintf("size_mix needs 1..%d entries", maxInjectSizeMix)) + } + seen := map[[2]int]bool{} + for _, sp := range b.SizeMix { + if sp.Count <= 0 || sp.Count > maxInjectSizeCount { + return bad(fmt.Sprintf("every size_mix entry needs a count in 1..%d", maxInjectSizeCount)) + } + if sp.WireBytes <= 0 { + return bad("every size_mix entry needs wire_bytes > 0") + } + if sp.PayloadBytes != nil && *sp.PayloadBytes < 0 { + return bad("a size_mix payload_bytes must be >= 0") + } + k := [2]int{sp.WireBytes, payloadOrUnknown(sp)} + if seen[k] { + return bad(fmt.Sprintf("size_mix carries the pair (%d, %d) twice", k[0], k[1])) + } + seen[k] = true + } + return nil +} + +// sizeDraw turns one bot's size description into a draw. For a size mix it +// holds the cumulative weights, computed once per bot, so a draw is a binary +// search rather than a walk over the whole distribution. +type sizeDraw struct { + mix []sizePair + cum []int64 + total int64 + // the scalar description + wire int + payload *int + spread int +} + +func newSizeDraw(b *injectBot) sizeDraw { + d := sizeDraw{mix: b.SizeMix, wire: b.WireBytes, payload: b.PayloadBytes} + if b.PayloadBytesSpread != nil { + d.spread = *b.PayloadBytesSpread + } + for _, sp := range b.SizeMix { + d.total += int64(sp.Count) + d.cum = append(d.cum, d.total) + } + return d +} + +// random reports whether a draw consumes a value from the generator. A +// single-entry mix and a zero spread are constant, so they consume none and +// leave every other draw of the train where it was. +func (d sizeDraw) random() bool { + if len(d.mix) > 0 { + return len(d.mix) > 1 + } + return d.payload != nil && d.spread > 0 +} + +// draw returns one event's wire and payload size. It takes exactly one value +// from p when random() is true, and none otherwise. +func (d sizeDraw) draw(p *prng) (int, *int) { + if len(d.mix) > 0 { + i := 0 + if len(d.mix) > 1 { + r := int64(p.float() * float64(d.total)) + i = sort.Search(len(d.cum), func(j int) bool { return d.cum[j] > r }) + if i >= len(d.mix) { + i = len(d.mix) - 1 + } + } + sp := d.mix[i] + if sp.PayloadBytes == nil { + return sp.WireBytes, nil + } + pb := *sp.PayloadBytes + return sp.WireBytes, &pb + } + if d.payload == nil { + return d.wire, nil + } + pb := *d.payload + if d.spread > 0 { + pb += int(p.float()*float64(2*d.spread+1)) - d.spread + } + return d.wire, &pb +} + +// generate builds every injected event, the per-bot counts and the labelled +// incidents. It fails, without producing anything, if the spec would emit +// more than maxInjected events. +func generate(s injectSpec, maxInjected int) ([]injEvent, []botStats, []Incident, error) { + var evs []injEvent + stats := make([]botStats, 0, len(s.Bots)) + var incidents []Incident + for i := range s.Bots { + b := &s.Bots[i] + botEvs, st, err := generateBot(b, *s.Seed, time.Duration(*s.FinalAfter), maxInjected-len(evs)) + if err != nil { + return nil, nil, nil, err + } + inc := labelIncidents(b.Name, b.Channel, st.Key, botEvs, s.Labelling) + st.Incidents = len(inc) + evs = append(evs, botEvs...) + stats = append(stats, st) + incidents = append(incidents, inc...) + } + // One deterministic order for the output: time, then id. + sort.SliceStable(evs, func(i, j int) bool { + if evs[i].t != evs[j].t { + return evs[i].t < evs[j].t + } + return evs[i].rec.ID < evs[j].rec.ID + }) + return evs, stats, incidents, nil +} + +// generateBot draws one bot's pulse train. Draw order per scheduled pulse is +// fixed: the missed-pulse draw (except for the first pulse), the jitter draw, +// the multi-part draw, then one draw per extra part. A pulse dropped by an +// earlier draw consumes no later draw. +func generateBot(b *injectBot, seed uint64, finalAfter time.Duration, budget int) ([]injEvent, botStats, error) { + p := botPRNG(seed, b.Name) + st := botStats{Name: b.Name, Channel: b.Channel} + period := time.Duration(b.Period) + // Every one of a bot's events carries the same wire fields, so the bot + // has exactly one stream key. Derive it once, from the same Event the + // detector would see. + key, err := botStreamKey(b, finalAfter) + if err != nil { + return nil, botStats{}, err + } + st.Key = key + sizes := newSizeDraw(b) + var evs []injEvent + emit := func(pulse, part int, when time.Time) error { + if len(evs) >= budget { + return fmt.Errorf("inject: more than -max-injected events; raise the bound or shorten the spec") + } + // Parity with the export, which times every line from a whole-second + // observation time (time.Unix(ts, 0)) and takes a whole-second + // -settle: an injected line must not be recognisable by a fractional + // second. Truncation moves a time by less than one second towards + // the past, so the effective jitter is in [-(jitter_max + 1s), + // jitter_max]. + when = when.UTC().Truncate(time.Second) + // One size draw per emitted event: a whole (wire, payload) pair from + // the key's observed distribution, or the scalar payload spread. + wire, payload := sizes.draw(p) + rec := anomaly.ReplayRecord{ + ID: fmt.Sprintf("%s%s-%d-%d", injPrefix, b.Name, pulse, part), Time: when, + FinalAt: when.Add(finalAfter), PayloadType: *b.PayloadType, Channel: b.Channel, + RouteClass: b.RouteClass, RouteKind: b.RouteKind, FirstHop: b.FirstHop, FullRoute: b.FullRoute, + WireBytes: wire, PayloadBytes: payload, Traffic: "unclassified", Decryption: "unknown", + } + // An injected line must be exactly as valid as an exported one. Every + // size the draw can produce was checked by botStreamKey before any + // event was built, so this cannot fire on a validated spec. + if _, err := rec.Event(); err != nil { + return fmt.Errorf("inject: bot %q: %w", b.Name, err) + } + evs = append(evs, injEvent{rec: rec, t: rec.Time.UnixNano(), pulse: pulse}) + return nil + } + for i := 0; ; i++ { + nominal := b.Start.Add(time.Duration(i) * period) + if !nominal.Before(b.Stop) { + break + } + st.PulsesScheduled++ + if i > 0 && p.float() < *b.MissedFraction { + st.PulsesMissed++ + continue + } + if inHole(b.Holes, nominal) { + st.PulsesInHole++ + continue + } + off := time.Duration(p.normal() * float64(time.Duration(*b.JitterSD))) + if lim := time.Duration(b.JitterMax); off > lim { + off = lim + } else if off < -lim { + off = -lim + } + when := nominal.Add(off) + if err := emit(i, 0, when); err != nil { + return nil, botStats{}, err + } + st.PulsesEmitted++ + if p.float() < *b.MultipartFraction { + extra := 1 + int(p.float()*float64(*b.MultipartMaxExtra)) + st.MultipartPulses++ + for part := 1; part <= extra; part++ { + d := time.Duration(p.float() * float64(time.Duration(b.MultipartMaxDelay))) + if err := emit(i, part, when.Add(d)); err != nil { + return nil, botStats{}, err + } + } + } + } + st.Events = len(evs) + if len(evs) > 0 { + first, last := evs[0].rec.Time, evs[0].rec.Time + for _, e := range evs { + if e.rec.Time.Before(first) { + first = e.rec.Time + } + if e.rec.Time.After(last) { + last = e.rec.Time + } + } + st.First, st.Last = first, last + } + return evs, st, nil +} + +// botStreamKey returns the canonical stream key of a bot's events, and +// checks that every size the bot can draw makes a valid event. It builds the +// same ReplayRecord emit builds, so a spec whose wire fields do not make a +// valid event fails here - before any event, line or file is produced - +// rather than aborting the run halfway through or silently labelling nothing. +// +// Sizes are not part of a stream key, so every size gives the same key; the +// loop is there to validate, and the equality is asserted rather than +// assumed. +func botStreamKey(b *injectBot, finalAfter time.Duration) (string, error) { + build := func(wire int, payload *int) (string, error) { + rec := anomaly.ReplayRecord{ID: injPrefix + b.Name + "-key", Time: b.Start.UTC().Truncate(time.Second), + PayloadType: *b.PayloadType, Channel: b.Channel, RouteClass: b.RouteClass, RouteKind: b.RouteKind, + FirstHop: b.FirstHop, FullRoute: b.FullRoute, WireBytes: wire, PayloadBytes: payload, + Traffic: "unclassified", Decryption: "unknown"} + rec.FinalAt = rec.Time.Add(finalAfter) + ev, err := rec.Event() + if err != nil { + return "", fmt.Errorf("inject: bot %q: %w", b.Name, err) + } + return anomaly.StreamKeyOf(ev).String(), nil + } + var sizes [][2]*int + if len(b.SizeMix) > 0 { + for _, sp := range b.SizeMix { + wire := sp.WireBytes + sizes = append(sizes, [2]*int{&wire, sp.PayloadBytes}) + } + } else { + // The whole envelope of the scalar draw: the smallest and the largest + // payload it can reach. + wire := b.WireBytes + if b.PayloadBytes == nil { + sizes = append(sizes, [2]*int{&wire, nil}) + } else { + lo, hi := *b.PayloadBytes-*b.PayloadBytesSpread, *b.PayloadBytes+*b.PayloadBytesSpread + sizes = append(sizes, [2]*int{&wire, &lo}, [2]*int{&wire, &hi}) + } + } + key := "" + for _, s := range sizes { + var payload *int + if s[1] != nil { + pb := *s[1] + payload = &pb + } + k, err := build(*s[0], payload) + if err != nil { + return "", err + } + if key == "" { + key = k + } else if k != key { + return "", fmt.Errorf("inject: bot %q: its sizes do not share one stream key", b.Name) + } + } + return key, nil +} + +func inHole(holes []injectHole, t time.Time) bool { + for _, h := range holes { + if !t.Before(h.Start) && t.Before(h.Start.Add(time.Duration(h.Duration))) { + return true + } + } + return false +} + +// labelIncidents applies the labelling recipe to one bot's emitted events. +func labelIncidents(bot, channel, key string, evs []injEvent, l labellingSpec) []Incident { + if len(evs) == 0 { + return nil + } + times := make([]int64, 0, len(evs)) + for _, e := range evs { + times = append(times, e.t) + } + sort.Slice(times, func(i, j int) bool { return times[i] < times[j] }) + // Events closer than the merge gap are one pulse, timed at its first + // event (as the detector's BurstGap does). + pulses := []int64{times[0]} + for _, t := range times[1:] { + if t-pulses[len(pulses)-1] >= int64(time.Duration(l.PulseMergeGap)) { + pulses = append(pulses, t) + } + } + if len(pulses) < 2 { + return nil + } + gaps := make([]int64, 0, len(pulses)-1) + inRange := make([]int64, 0, len(pulses)-1) + for i := 1; i < len(pulses); i++ { + g := pulses[i] - pulses[i-1] + gaps = append(gaps, g) + if g >= int64(time.Duration(l.MinPeriod)) && g <= int64(time.Duration(l.MaxPeriod)) { + inRange = append(inRange, g) + } + } + if len(inRange) == 0 { + return nil + } + period := medianInt64(inRange) + tol := int64(float64(period) * l.ToleranceRel) + if a := int64(time.Duration(l.ToleranceAbs)); a > tol { + tol = a + } + fits := func(g int64) bool { + for k := 1; k <= l.MaxHarmonic; k++ { + d := g - int64(k)*period + if d < 0 { + d = -d + } + if d <= tol { + return true + } + } + return false + } + var out []Incident + // A run is a maximal stretch of pulses whose every gap fits. + runStart := 0 + flush := func(end int) { // pulses[runStart:end+1] is the run + if end-runStart+1 >= l.MinPulses { + out = append(out, Incident{Bot: bot, Channel: channel, Key: key, Start: time.Unix(0, pulses[runStart]).UTC(), + Stop: time.Unix(0, pulses[end]).UTC(), Period: dur(period), Pulses: end - runStart + 1}) + } + } + for i, g := range gaps { + if !fits(g) { + flush(i) + runStart = i + 1 + } + } + flush(len(pulses) - 1) + return out +} + +// medianInt64 returns the median of a non-empty slice: the middle value, or +// the mean of the two middle values (truncated) for an even count. It sorts +// a copy, so the caller's slice order is kept. +func medianInt64(v []int64) int64 { + s := append([]int64(nil), v...) + sort.Slice(s, func(i, j int) bool { return s[i] < s[j] }) + n := len(s) + if n%2 == 1 { + return s[n/2] + } + return s[n/2-1] + (s[n/2]-s[n/2-1])/2 +} + +// backgroundLine is the minimum a background line must tell us: its time, to +// merge on, and its id, to keep the injected marker unambiguous. Unknown +// fields are allowed here: the line is copied through byte for byte. +type backgroundLine struct { + ID string `json:"id"` + Time time.Time `json:"time"` +} + +// mergeLines writes the injected events and the background lines to w in +// data-time order. Background lines are copied verbatim, so the background +// is never reformatted; an injected event goes out before the first +// background line that is not earlier than it. +func mergeLines(r io.Reader, w io.Writer, evs []injEvent, maxLine int) (injectSummary, error) { + sum := injectSummary{Experimental: true, InjectedEvents: len(evs)} + bw := bufio.NewWriter(w) + next := 0 + writeInjected := func(upto int64, all bool) error { + for next < len(evs) && (all || evs[next].t <= upto) { + b, err := json.Marshal(evs[next].rec) + if err != nil { + return err + } + if _, err := bw.Write(append(b, '\n')); err != nil { + return err + } + next++ + sum.LinesWritten++ + } + return nil + } + sc := bufio.NewScanner(r) + limit := maxLine + 2 + initial := 4096 + if limit < initial { + initial = limit + } + sc.Buffer(make([]byte, 0, initial), limit) + var prev int64 + havePrev := false + for sc.Scan() { + line := sc.Bytes() + if len(line) > maxLine { + return sum, errors.New("inject: background line longer than -max-line-bytes") + } + if len(strings.TrimSpace(string(line))) == 0 { + continue + } + sum.BackgroundLines++ + var bl backgroundLine + if err := json.Unmarshal(line, &bl); err != nil || bl.Time.IsZero() { + return sum, fmt.Errorf("inject: background line %d: need a JSON object with an RFC 3339 \"time\"", sum.BackgroundLines) + } + if strings.HasPrefix(bl.ID, injPrefix) { + return sum, fmt.Errorf("inject: background line %d already carries an injected id (%s...): refusing to mix", sum.BackgroundLines, injPrefix) + } + t := bl.Time.UnixNano() + if havePrev && t < prev { + sum.BackgroundOutOfOrder++ + } + prev, havePrev = t, true + if err := writeInjected(t, false); err != nil { + return sum, err + } + if _, err := bw.Write(append(append([]byte(nil), line...), '\n')); err != nil { + return sum, err + } + sum.LinesWritten++ + } + if err := sc.Err(); err != nil { + return sum, fmt.Errorf("inject: background input: %w", err) + } + if err := writeInjected(0, true); err != nil { + return sum, err + } + return sum, bw.Flush() +} + +// onlyJSONValue reports whether the decoder consumed all of b but trailing +// whitespace. +func onlyJSONValue(dec *json.Decoder, b []byte) bool { + return len(strings.TrimSpace(string(b[dec.InputOffset():]))) == 0 +} diff --git a/cmd/anomaly/inject_test.go b/cmd/anomaly/inject_test.go new file mode 100644 index 000000000..563cd6f9a --- /dev/null +++ b/cmd/anomaly/inject_test.go @@ -0,0 +1,596 @@ +package main + +import ( + "bytes" + "encoding/json" + "fmt" + "os" + "path/filepath" + "sort" + "strings" + "testing" + "time" + + "github.com/meshcore-analyzer/anomaly" +) + +// Issue #474, phase 3: injection-based evaluation. Every bot, channel byte +// and time here is synthetic. + +var injT = time.Date(2030, 3, 1, 0, 0, 0, 0, time.UTC) + +func injAt(d time.Duration) string { return injT.Add(d).UTC().Format(time.RFC3339) } + +// specOpts are the knobs the tests vary; everything else is fixed below. +type specOpts struct { + seed uint64 + channel string + period string + stopAfter time.Duration + jitterSD string + missedFraction float64 + multipartFraction float64 + payloadSpread int + holes string // JSON array body, e.g. {"start":..,"duration":"35m"} + extraBots string // JSON for further bots, with a leading comma + leadBots string // JSON for bots before bot-a, with a trailing comma + minPulses int +} + +func (o specOpts) with() specOpts { + if o.channel == "" { + o.channel = "7f" + } + if o.period == "" { + o.period = "5m" + } + if o.stopAfter == 0 { + o.stopAfter = 4 * time.Hour + } + if o.jitterSD == "" { + o.jitterSD = "0s" + } + if o.holes == "" { + o.holes = "[]" + } + if o.minPulses == 0 { + o.minPulses = 8 + } + return o +} + +// specJSON is the one spec shape the tests use, with the September bot's +// shape as its default (5 min, 3 s jitter, multi-part pulses). +func specJSON(o specOpts) string { + o = o.with() + return fmt.Sprintf(`{ + "seed": %d, + "final_after": "30s", + "labelling": {"pulse_merge_gap": "60s", "min_period": "1m", "max_period": "1h", + "min_pulses": %d, "max_harmonic": 3, "tolerance_abs": "30s", "tolerance_rel": 0.1}, + "bots": [%s{ + "name": "bot-a", + "channel": %q, + "payload_type": 5, + "route_class": "flood", + "route_kind": "relay", + "first_hop": "1:fd", + "full_route": "1:fd00", + "wire_bytes": 60, + "payload_bytes": 35, + "payload_bytes_spread": %d, + "start": %q, + "stop": %q, + "period": %q, + "jitter_sd": %q, + "jitter_max": "10s", + "missed_fraction": %v, + "multipart_fraction": %v, + "multipart_max_extra": 2, + "multipart_max_delay": "55s", + "holes": %s + }%s] +}`, o.seed, o.minPulses, o.leadBots, o.channel, o.payloadSpread, injAt(0), injAt(o.stopAfter), o.period, o.jitterSD, + o.missedFraction, o.multipartFraction, o.holes, o.extraBots) +} + +// backgroundJSONL builds a sparse, non-periodic background: n events on +// channel 11 at irregular gaps, through two first hops. +func backgroundJSONL(n int) string { + var b strings.Builder + step := []time.Duration{37 * time.Second, 4 * time.Minute, 71 * time.Second, 13 * time.Minute} + hops := []string{"1:0b", "2:0c01"} + t := injT + for i := 0; i < n; i++ { + t = t.Add(step[i%len(step)]) + pb := 23 + rec := anomaly.ReplayRecord{ID: fmt.Sprintf("tx%d", i+1), Time: t, FinalAt: t.Add(30 * time.Second), + PayloadType: 5, Channel: "11", RouteClass: "flood", RouteKind: "relay", + FirstHop: hops[i%len(hops)], FullRoute: hops[i%len(hops)] + "00", WireBytes: 55, + PayloadBytes: &pb, Traffic: "unclassified", Decryption: "unknown"} + j, err := json.Marshal(rec) + if err != nil { + panic(err) + } + b.Write(j) + b.WriteByte('\n') + } + return b.String() +} + +// injectRun writes the spec and the background into a fresh directory, runs +// inject and returns the output lines, the labels and the summary. +type injectResult struct { + dir string + outPath string + labPath string + lines []string + labels InjectLabels + summary injectSummary + stderr string +} + +func injectInto(t *testing.T, spec, background string, extra ...string) injectResult { + t.Helper() + res, code := tryInject(t, spec, background, extra...) + if code != 0 { + t.Fatalf("inject exit %d: %s", code, res.stderr) + } + return res +} + +func tryInject(t *testing.T, spec, background string, extra ...string) (injectResult, int) { + t.Helper() + dir := t.TempDir() + r := injectResult{dir: dir, outPath: filepath.Join(dir, "events.jsonl"), labPath: filepath.Join(dir, "labels.json")} + specPath := filepath.Join(dir, "spec.json") + bgPath := filepath.Join(dir, "background.jsonl") + if err := os.WriteFile(specPath, []byte(spec), 0o644); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(bgPath, []byte(background), 0o644); err != nil { + t.Fatal(err) + } + args := append([]string{"inject", "-spec", specPath, "-input", bgPath, "-out", r.outPath, + "-labels", r.labPath, "-max-injected", "100000", "-max-line-bytes", "65536"}, extra...) + var stdout, stderr bytes.Buffer + code := run(args, nil, &stdout, &stderr) + r.stderr = stderr.String() + if stdout.Len() != 0 { + t.Fatalf("inject wrote %d bytes to stdout", stdout.Len()) + } + if code != 0 { + return r, code + } + out, err := os.ReadFile(r.outPath) + if err != nil { + t.Fatal(err) + } + r.lines = strings.Split(strings.TrimRight(string(out), "\n"), "\n") + lb, err := os.ReadFile(r.labPath) + if err != nil { + t.Fatal(err) + } + if err := json.Unmarshal(lb, &r.labels); err != nil { + t.Fatal(err) + } + if err := json.Unmarshal([]byte(r.stderr), &r.summary); err != nil { + t.Fatalf("summary %q: %v", r.stderr, err) + } + return r, code +} + +// injectedLines returns the injected lines of a run, parsed. +func injectedLines(t *testing.T, r injectResult) []anomaly.ReplayRecord { + t.Helper() + var out []anomaly.ReplayRecord + for _, l := range r.lines { + var rec anomaly.ReplayRecord + if err := json.Unmarshal([]byte(l), &rec); err != nil { + t.Fatalf("line %q: %v", l, err) + } + if strings.HasPrefix(rec.ID, injPrefix) { + out = append(out, rec) + } + } + return out +} + +// Every injected line is marked, unique, valid as an event, and carries +// exactly the configured wire fields. The background is copied through byte +// for byte, and the merged output is in data-time order. +func TestInjectMarksAndMergesWithoutTouchingTheBackground(t *testing.T) { + bg := backgroundJSONL(40) + r := injectInto(t, specJSON(specOpts{seed: 1, multipartFraction: 0.3}), bg) + ids := map[string]bool{} + var last time.Time + injected, passed := 0, 0 + var bgLines []string + for _, l := range r.lines { + var rec anomaly.ReplayRecord + if err := json.Unmarshal([]byte(l), &rec); err != nil { + t.Fatalf("line %q: %v", l, err) + } + if ids[rec.ID] { + t.Fatalf("duplicate id %s", rec.ID) + } + ids[rec.ID] = true + if rec.Time.Before(last) { + t.Fatalf("output is not in data-time order at %s", rec.ID) + } + last = rec.Time + if !strings.HasPrefix(rec.ID, injPrefix) { + passed++ + bgLines = append(bgLines, l) + continue + } + injected++ + // The marker is the literal "inj---": an export + // numbers its lines tx1, tx2, ..., so no real line can carry it. + if !strings.HasPrefix(rec.ID, "inj-bot-a-") { + t.Fatalf("injected id %q is not marked inj---", rec.ID) + } + if _, err := rec.Event(); err != nil { + t.Fatalf("injected %s is not a valid event: %v", rec.ID, err) + } + if rec.Channel != "7f" || rec.FirstHop != "1:fd" || rec.FullRoute != "1:fd00" || + rec.PayloadType != 5 || rec.WireBytes != 60 || rec.PayloadBytes == nil || *rec.PayloadBytes != 35 { + t.Fatalf("injected %s has unexpected wire fields: %+v", rec.ID, rec) + } + if got := rec.FinalAt.Sub(rec.Time); got != 30*time.Second { + t.Fatalf("injected %s final_at offset %v", rec.ID, got) + } + } + if passed != 40 || injected != r.summary.InjectedEvents || injected < 48 { + t.Fatalf("passed %d background lines, injected %d (summary %d)", passed, injected, r.summary.InjectedEvents) + } + if strings.Join(bgLines, "\n")+"\n" != bg { + t.Fatal("the background lines were not copied through byte for byte") + } + if r.summary.BackgroundLines != 40 || r.summary.LinesWritten != 40+injected || r.summary.BackgroundOutOfOrder != 0 { + t.Fatalf("summary %+v", r.summary) + } +} + +// The same spec, seed and background give byte-identical output and labels. +func TestInjectIsDeterministic(t *testing.T) { + spec := specJSON(specOpts{seed: 7, jitterSD: "3s", missedFraction: 0.05, multipartFraction: 0.3}) + bg := backgroundJSONL(20) + a := injectInto(t, spec, bg) + b := injectInto(t, spec, bg) + ab, _ := os.ReadFile(a.outPath) + bb, _ := os.ReadFile(b.outPath) + if !bytes.Equal(ab, bb) { + t.Fatal("two runs of the same spec differ") + } + al, _ := os.ReadFile(a.labPath) + bl, _ := os.ReadFile(b.labPath) + if !bytes.Equal(al, bl) { + t.Fatal("two runs of the same spec produced different labels") + } + // A different seed moves the pulses but keeps the schedule. + c := injectInto(t, specJSON(specOpts{seed: 8, jitterSD: "3s", missedFraction: 0.05, multipartFraction: 0.3}), bg) + cb, _ := os.ReadFile(c.outPath) + if bytes.Equal(ab, cb) { + t.Fatal("a different seed gave identical output") + } + if a.summary.Bots[0].PulsesScheduled != c.summary.Bots[0].PulsesScheduled { + t.Fatalf("the schedule changed with the seed: %d vs %d", + a.summary.Bots[0].PulsesScheduled, c.summary.Bots[0].PulsesScheduled) + } +} + +// A bot's events depend only on the seed and its own fields, so adding a +// second bot leaves the first one's events untouched. +func TestInjectBotsAreIndependent(t *testing.T) { + botB := `{ + "name": "bot-b", "channel": "22", "payload_type": 5, "route_class": "flood", "route_kind": "relay", + "first_hop": "1:ee", "full_route": "1:ee00", "wire_bytes": 60, "payload_bytes": 35, "payload_bytes_spread": 0, + "start": "` + injAt(0) + `", "stop": "` + injAt(time.Hour) + `", "period": "7m", + "jitter_sd": "2s", "jitter_max": "10s", "missed_fraction": 0, "multipart_fraction": 0, + "multipart_max_extra": 2, "multipart_max_delay": "55s", "holes": []}` + linesOf := func(r injectResult, prefix string) string { + var out []string + for _, rec := range injectedLines(t, r) { + if strings.HasPrefix(rec.ID, prefix) { + b, _ := json.Marshal(rec) + out = append(out, string(b)) + } + } + if len(out) == 0 { + t.Fatalf("no injected lines with prefix %s", prefix) + } + return strings.Join(out, "\n") + } + alone := injectInto(t, specJSON(specOpts{seed: 3, jitterSD: "3s"}), "") + after := injectInto(t, specJSON(specOpts{seed: 3, jitterSD: "3s", extraBots: "," + botB}), "") + // bot-a's events must not depend on its position in the spec either. + before := injectInto(t, specJSON(specOpts{seed: 3, jitterSD: "3s", leadBots: botB + ","}), "") + want := linesOf(alone, injPrefix+"bot-a-") + if got := linesOf(after, injPrefix+"bot-a-"); got != want { + t.Fatal("adding a second bot after bot-a changed bot-a's events") + } + if got := linesOf(before, injPrefix+"bot-a-"); got != want { + t.Fatal("adding a bot before bot-a changed bot-a's events") + } + if len(after.labels.Bots) != 2 || after.labels.Bots[1].Name != "bot-b" { + t.Fatalf("labels %+v", after.labels.Bots) + } +} + +// The labelling recipe is applied to the emitted pulses: a clean train is +// one incident, a hole longer than max_harmonic periods splits it, and a +// train shorter than min_pulses is no incident at all. +func TestInjectLabelsIncidentsFromThePulses(t *testing.T) { + clean := injectInto(t, specJSON(specOpts{seed: 11}), "") + if len(clean.labels.Incidents) != 1 { + t.Fatalf("clean train: %d incidents, want 1", len(clean.labels.Incidents)) + } + inc := clean.labels.Incidents[0] + if inc.Channel != "7f" || inc.Bot != "bot-a" || time.Duration(inc.Period) != 5*time.Minute || inc.Pulses != 48 { + t.Fatalf("incident %+v", inc) + } + if !inc.Start.Equal(injT) || !inc.Stop.Equal(injT.Add(235*time.Minute)) { + t.Fatalf("incident span %s..%s", inc.Start, inc.Stop) + } + // A 35-minute hole is 7 periods: more than max_harmonic, so the gap + // across it fits no k*P and the run is split in two. + holed := injectInto(t, specJSON(specOpts{seed: 11, + holes: `[{"start": "` + injAt(100*time.Minute) + `", "duration": "35m"}]`}), "") + if len(holed.labels.Incidents) != 2 { + t.Fatalf("holed train: %d incidents, want 2: %+v", len(holed.labels.Incidents), holed.labels.Incidents) + } + if holed.summary.Bots[0].PulsesInHole != 7 { + t.Fatalf("pulses in hole: %d, want 7", holed.summary.Bots[0].PulsesInHole) + } + // Seven pulses are fewer than min_pulses 8: no incident, but the events + // are still injected. + short := injectInto(t, specJSON(specOpts{seed: 11, stopAfter: 35 * time.Minute}), "") + if len(short.labels.Incidents) != 0 || short.summary.Bots[0].PulsesEmitted != 7 { + t.Fatalf("short train: %d incidents, %d pulses", len(short.labels.Incidents), short.summary.Bots[0].PulsesEmitted) + } +} + +// Multi-part pulses are extra events inside the merge gap, so they add +// events without adding pulses or incidents. +func TestInjectMultipartPulsesStayOnePulse(t *testing.T) { + r := injectInto(t, specJSON(specOpts{seed: 5, multipartFraction: 1}), "") + st := r.summary.Bots[0] + if st.MultipartPulses != st.PulsesEmitted || st.Events <= st.PulsesEmitted { + t.Fatalf("stats %+v", st) + } + if len(r.labels.Incidents) != 1 || r.labels.Incidents[0].Pulses != st.PulsesEmitted { + t.Fatalf("incidents %+v for %d pulses", r.labels.Incidents, st.PulsesEmitted) + } + // Every extra part lands within multipart_max_delay of its pulse. + byPulse := map[string][]time.Time{} + for _, rec := range injectedLines(t, r) { + p := rec.ID[:strings.LastIndex(rec.ID, "-")] + byPulse[p] = append(byPulse[p], rec.Time) + } + for p, ts := range byPulse { + if len(ts) < 2 { + t.Fatalf("pulse %s has %d parts", p, len(ts)) + } + sort.Slice(ts, func(i, j int) bool { return ts[i].Before(ts[j]) }) + if d := ts[len(ts)-1].Sub(ts[0]); d > 55*time.Second { + t.Fatalf("pulse %s spans %v", p, d) + } + } + // With the fraction at 0 there are no extra parts at all. + none := injectInto(t, specJSON(specOpts{seed: 5}), "") + if st := none.summary.Bots[0]; st.MultipartPulses != 0 || st.Events != st.PulsesEmitted { + t.Fatalf("multipart_fraction 0 produced parts: %+v", st) + } +} + +// missed_fraction drops pulses, and never the first one. +func TestInjectMissedPulses(t *testing.T) { + all := injectInto(t, specJSON(specOpts{seed: 9, missedFraction: 1}), "") + st := all.summary.Bots[0] + if st.PulsesEmitted != 1 || st.PulsesMissed != st.PulsesScheduled-1 { + t.Fatalf("missed_fraction 1: %+v", st) + } + if len(all.labels.Incidents) != 0 { + t.Fatalf("a single pulse is no incident: %+v", all.labels.Incidents) + } + some := injectInto(t, specJSON(specOpts{seed: 9, missedFraction: 0.5}), "") + if st := some.summary.Bots[0]; st.PulsesMissed == 0 || st.PulsesEmitted == st.PulsesScheduled { + t.Fatalf("missed_fraction 0.5: %+v", st) + } +} + +// Jitter stays inside jitter_max and does not accumulate: a pulse is always +// within the clamp of its nominal grid time. +func TestInjectJitterIsClampedAndDoesNotDrift(t *testing.T) { + r := injectInto(t, specJSON(specOpts{seed: 13, jitterSD: "8s"}), "") + seen := false + for _, rec := range injectedLines(t, r) { + var pulse, part int + if _, err := fmt.Sscanf(strings.TrimPrefix(rec.ID, injPrefix+"bot-a-"), "%d-%d", &pulse, &part); err != nil { + t.Fatalf("id %s: %v", rec.ID, err) + } + off := rec.Time.Sub(injT.Add(time.Duration(pulse) * 5 * time.Minute)) + if off > 10*time.Second || off < -10*time.Second { + t.Fatalf("pulse %d is %v off its nominal time", pulse, off) + } + if off != 0 { + seen = true + } + } + if !seen { + t.Fatal("jitter_sd 8s produced no offset at all") + } +} + +// The tool writes only the two files it was given, refuses to overwrite +// either of them, and leaves nothing behind when it fails. +func TestInjectWritesOnlyItsTwoFiles(t *testing.T) { + spec := specJSON(specOpts{seed: 2}) + r := injectInto(t, spec, backgroundJSONL(5)) + ents, err := os.ReadDir(r.dir) + if err != nil { + t.Fatal(err) + } + var names []string + for _, e := range ents { + names = append(names, e.Name()) + } + sort.Strings(names) + want := []string{"background.jsonl", "events.jsonl", "labels.json", "spec.json"} + if strings.Join(names, ",") != strings.Join(want, ",") { + t.Fatalf("directory holds %v", names) + } + // Running again in the same directory refuses to overwrite. + args := []string{"inject", "-spec", filepath.Join(r.dir, "spec.json"), "-input", filepath.Join(r.dir, "background.jsonl"), + "-out", r.outPath, "-labels", r.labPath, "-max-injected", "100000", "-max-line-bytes", "65536"} + var stdout, stderr bytes.Buffer + if code := run(args, nil, &stdout, &stderr); code != exitFail { + t.Fatalf("overwriting -out: exit %d", code) + } + // A labels file that already exists leaves no half-written output. + out2 := filepath.Join(r.dir, "events2.jsonl") + args[6] = out2 + if code := run(args, nil, &stdout, &stderr); code != exitFail { + t.Fatalf("overwriting -labels: exit %d", code) + } + if _, err := os.Stat(out2); !os.IsNotExist(err) { + t.Fatalf("a failed run left %s behind (%v)", out2, err) + } +} + +// -max-injected is a hard bound: exceeding it fails the run and writes +// nothing. +func TestInjectMaxInjectedIsEnforced(t *testing.T) { + r, code := tryInject(t, specJSON(specOpts{seed: 4}), "", "-max-injected", "10") + if code != exitFail { + t.Fatalf("exit %d, want %d", code, exitFail) + } + if !strings.Contains(r.stderr, "max-injected") { + t.Fatalf("stderr %q", r.stderr) + } + if _, err := os.Stat(r.outPath); !os.IsNotExist(err) { + t.Fatal("output file written despite the bound") + } + if _, err := os.Stat(r.labPath); !os.IsNotExist(err) { + t.Fatal("labels file written despite the bound") + } +} + +// A background line that already carries an injected id is refused, so the +// marker stays unambiguous. +func TestInjectRefusesPreInjectedBackground(t *testing.T) { + bg := backgroundJSONL(2) + `{"id":"inj-bot-a-0-0","time":"` + injAt(time.Hour) + + `","payload_type":5,"channel":"7f","route_class":"flood","route_kind":"relay","first_hop":"1:fd","full_route":"1:fd00","wire_bytes":60,"traffic":"unclassified","decryption":"unknown"}` + "\n" + r, code := tryInject(t, specJSON(specOpts{seed: 4}), bg) + if code != exitFail || !strings.Contains(r.stderr, "injected id") { + t.Fatalf("exit %d, stderr %q", code, r.stderr) + } +} + +// A background line without a usable time is an error, not a silently +// misplaced line. +func TestInjectRefusesUnreadableBackgroundLine(t *testing.T) { + for _, bad := range []string{"not json", `{"id":"tx9"}`, `{"id":"tx9","time":"yesterday"}`} { + r, code := tryInject(t, specJSON(specOpts{seed: 4}), bad+"\n") + if code != exitFail { + t.Fatalf("%q: exit %d, stderr %q", bad, code, r.stderr) + } + } +} + +// Required flags are required; -spec, -input, -out, -labels, -max-injected +// and -max-line-bytes all have no default. +func TestInjectRequiresExplicitFlags(t *testing.T) { + dir := t.TempDir() + specPath := filepath.Join(dir, "spec.json") + bgPath := filepath.Join(dir, "bg.jsonl") + if err := os.WriteFile(specPath, []byte(specJSON(specOpts{seed: 1})), 0o644); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(bgPath, nil, 0o644); err != nil { + t.Fatal(err) + } + full := []string{"inject", "-spec", specPath, "-input", bgPath, "-out", filepath.Join(dir, "o.jsonl"), + "-labels", filepath.Join(dir, "l.json"), "-max-injected", "1000", "-max-line-bytes", "4096"} + for i := 1; i < len(full); i += 2 { + args := append(append([]string{}, full[:i]...), full[i+2:]...) + var stdout, stderr bytes.Buffer + if code := run(args, nil, &stdout, &stderr); code != exitUsage { + t.Fatalf("without %s: exit %d", full[i], code) + } + } + var stdout, stderr bytes.Buffer + if code := run(append(append([]string{}, full...), "extra"), nil, &stdout, &stderr); code != exitUsage { + t.Fatalf("with a positional argument: exit %d", code) + } +} + +// The spec has no defaults and no room for anything the tool does not know: +// a missing required field, a field whose zero value would be silently +// assumed, and an unknown field are all errors. +func TestInjectSpecRejectsMissingAndUnknownFields(t *testing.T) { + base := specJSON(specOpts{seed: 1, multipartFraction: 0.3}) + for _, key := range []string{`"seed": 1,`, `"final_after": "30s",`, `"payload_type": 5,`, + `"jitter_sd": "0s",`, `"missed_fraction": 0,`, `"multipart_fraction": 0.3,`, + `"multipart_max_extra": 2,`, `"wire_bytes": 60,`, `"period": "5m",`} { + if !strings.Contains(base, key) { + t.Fatalf("test fixture no longer contains %s", key) + } + if _, code := tryInject(t, strings.Replace(base, key, "", 1), ""); code != exitFail { + t.Errorf("without %s: exit %d", key, code) + } + } + for name, spec := range map[string]string{ + "unknown field": strings.Replace(base, `"seed": 1,`, `"seed": 1, "db": "/tmp/meshcore.db",`, 1), + "trailing data": base + "\n{}", + "no bots": strings.Replace(base, `"bots": [{`, `"bots2": [], "bots": [{`, 1), + "bad channel": strings.Replace(base, `"channel": "7f"`, `"channel": "7F"`, 1), + "bad bot name": strings.Replace(base, `"name": "bot-a"`, `"name": "Bot A"`, 1), + "stop before start": strings.Replace(base, `"stop": "`+injAt(4*time.Hour)+`"`, + `"stop": "`+injAt(-time.Hour)+`"`, 1), + "jitter at half the period": strings.Replace(base, `"jitter_max": "10s"`, `"jitter_max": "150s"`, 1), + "bad labelling": strings.Replace(base, `"min_pulses": 8`, `"min_pulses": 1`, 1), + "duplicate bot": strings.Replace(base, `"holes": [] + }]`, `"holes": [] + }, {"name": "bot-a", "channel": "22", "payload_type": 5, "route_class": "flood", "route_kind": "relay", + "first_hop": "1:ee", "full_route": "1:ee00", "wire_bytes": 60, "payload_bytes": 35, "payload_bytes_spread": 0, + "start": "`+injAt(0)+`", "stop": "`+injAt(time.Hour)+`", "period": "7m", "jitter_sd": "0s", + "jitter_max": "10s", "missed_fraction": 0, "multipart_fraction": 0, "multipart_max_extra": 2, + "multipart_max_delay": "55s", "holes": []}]`, 1), + } { + if _, code := tryInject(t, spec, ""); code != exitFail { + t.Errorf("%s: exit %d", name, code) + } + } +} + +// The labels file carries the ground truth and nothing else: it is marked +// synthetic, and it holds no ids, routes, hops, observers or text. +func TestInjectLabelsAreKeyFreeAndMarkedSynthetic(t *testing.T) { + r := injectInto(t, specJSON(specOpts{seed: 6, multipartFraction: 0.3}), backgroundJSONL(5)) + if !r.labels.Experimental || !r.labels.Synthetic || r.labels.Seed != 6 { + t.Fatalf("labels header %+v", r.labels) + } + raw, err := os.ReadFile(r.labPath) + if err != nil { + t.Fatal(err) + } + for _, banned := range []string{"fd00", "observer", "tx1", injPrefix, "raw_hex", "public_key"} { + if bytes.Contains(raw, []byte(banned)) { + t.Fatalf("labels file carries %q:\n%s", banned, raw) + } + } + // It does carry the channel hash, the times and the period. + if !bytes.Contains(raw, []byte(`"channel": "7f"`)) || !bytes.Contains(raw, []byte(`"period": "5m0s"`)) { + t.Fatalf("labels file is missing the ground truth:\n%s", raw) + } + // Since #474 round 2 it also carries the bot's stream key, which + // scoring needs to tell a bot apart from real traffic on the same + // channel hash. The key holds the payload type, the channel hash, the + // route class and kind and the first hop - fields the export itself + // carries - and nothing else: no whole route, no ids, no observers. + if !bytes.Contains(raw, []byte(`"key": "`+scoreKey+`"`)) { + t.Fatalf("labels file is missing the stream key:\n%s", raw) + } +} diff --git a/cmd/anomaly/keys.go b/cmd/anomaly/keys.go new file mode 100644 index 000000000..08d804fe2 --- /dev/null +++ b/cmd/anomaly/keys.go @@ -0,0 +1,475 @@ +package main + +import ( + "bufio" + "encoding/hex" + "encoding/json" + "errors" + "flag" + "fmt" + "io" + "math" + "os" + "sort" + "strings" + "time" + + "github.com/meshcore-analyzer/anomaly" +) + +// Issue #474, phase 3: rank the stream keys of an exported background by row +// count, so an injection panel can be placed on keys real traffic actually +// uses instead of on keys of its own. +// +// Why this exists. The stream key is payload type + channel hash + route +// class + route kind + first hop. A bot on a channel byte and a first hop +// that no real transmission carries has that key to itself, so the detector +// sees a clean train and recall measured on it says nothing about finding a +// bot inside real traffic: a real spam bot on a busy channel shares its key +// with everyone else entering through the same repeater. +// +// Blind by construction. This command reads the export only and counts rows. +// It never runs the detector, reads no report, no labels and no metrics, and +// has no idea which keys would alarm: the ranking is row counts and nothing +// else, and ties break on the canonical key encoding. So a panel built from +// its output is chosen mechanically and before any detection result exists. +// +// Offline and file-only, like the rest of the tool: no database, no -db +// flag, no network, no clock. + +// maxKeySizeBuckets bounds the distinct payload and wire sizes tracked per +// key, and maxKeyRouteVariants the distinct full routes. Both are counting +// bounds only: a key that exceeds one keeps its count and its times, and the +// overflow is reported per key. +const ( + maxKeySizeBuckets = 512 + maxKeyRouteVariants = 64 + // maxKeySizePairs bounds the distinct (wire, payload) pairs tracked per + // key. It is the distribution an injected bot draws from, so it is kept + // wider than the per-dimension bucket bound; overflow is reported. + maxKeySizePairs = 2048 +) + +// sizePair is one observed (wire_bytes, payload_bytes) pair and the number +// of background lines that carried it. PayloadBytes is absent for the lines +// whose payload length the export did not know. +// +// Why the pair and not two separate figures: the two sizes are not +// independent. A frame is header plus path plus payload, so wire - payload +// is the route's own overhead, while the payload itself comes in cipher +// blocks. Drawing the two separately produces frames real traffic never +// carries - and, when the drawn payload passes wire - 2, frames the event +// validator refuses outright. +type sizePair struct { + WireBytes int `json:"wire_bytes"` + PayloadBytes *int `json:"payload_bytes,omitempty"` + Count int `json:"count"` +} + +// keyStat is one ranked stream key. Every field is derived from the +// background lines on that key: hashes, hop prefixes, sizes, counts and +// times, which is what the export itself carries. No ids, observers, names +// or text. +type keyStat struct { + Rank int `json:"rank"` + Key string `json:"key"` + PayloadType int `json:"payload_type"` + Channel string `json:"channel"` + RouteClass string `json:"route_class"` + RouteKind string `json:"route_kind"` + FirstHop string `json:"first_hop"` + // FullRoute is the most common whole route seen on the key, ties broken + // by the smallest encoding. + FullRoute string `json:"full_route"` + // WireBytesMedian and the payload figures are the realistic sizes an + // injected bot on this key should use. PayloadBytesSpread is + // (p90 - p10) / 2, so payload_bytes +/- spread covers the middle of the + // observed spread; it is clamped below PayloadBytesMedian. + WireBytesMedian int `json:"wire_bytes_median"` + PayloadBytesMedian int `json:"payload_bytes_median,omitempty"` + PayloadBytesSpread int `json:"payload_bytes_spread"` + // SizeMix is the key's whole observed (wire, payload) distribution, one + // entry per distinct pair, ordered by wire size then payload size. This + // is what an injected bot on this key draws its sizes from, so its lines + // cannot be told from the key's real ones by size. + SizeMix []sizePair `json:"size_mix"` + SizeMixDropped int `json:"size_mix_dropped,omitempty"` + Events int `json:"events"` + // SizeBucketsDropped and RouteVariantsDropped count values that did not + // fit the per-key bounds above; they affect the medians and the route + // mode only, never Events. + SizeBucketsDropped int `json:"size_buckets_dropped,omitempty"` + RouteVariantsDropped int `json:"route_variants_dropped,omitempty"` + First time.Time `json:"first"` + Last time.Time `json:"last"` +} + +// keysResult is the output file. +type keysResult struct { + Experimental bool `json:"experimental"` + // PayloadType, ExcludeChannels and the filter below are the mechanical + // selection rule, restated in the output so a panel can be audited + // against it. + PayloadType int `json:"payload_type"` + ExcludeChannels []string `json:"exclude_channels"` + Filter string `json:"filter"` + InputLines int `json:"input_lines"` + Considered int `json:"considered"` + SkippedType int `json:"skipped_payload_type"` + SkippedNoChannel int `json:"skipped_no_channel"` + SkippedNotRelay int `json:"skipped_not_relay"` + SkippedExcluded int `json:"skipped_excluded_channel"` + DistinctKeys int `json:"distinct_keys"` + Top []keyStat `json:"top"` +} + +// keyAgg accumulates one key's lines. +type keyAgg struct { + k anomaly.Key + events int + first, last time.Time + payload, wire map[int]int + // pairs counts the observed (wire, payload) pairs; a payload of -1 is a + // line whose payload length the export did not know. + pairs map[[2]int]int + routes map[string]int + sizeDropped int + pairDropped int + routeDropped int +} + +func runKeys(args []string, stdin io.Reader, stdout, stderr io.Writer) int { + fs := flag.NewFlagSet("keys", flag.ContinueOnError) + fs.SetOutput(stderr) + input := fs.String("input", "", "exported ReplayRecord JSON Lines, or - for stdin (required)") + payloadType := fs.Int("payload-type", -1, "payload type to rank (required)") + top := fs.Int("top", 0, "how many keys to report (required)") + exclude := fs.String("exclude-channels", "", "comma-separated channel hashes to leave out, lower-case hex") + maxKeys := fs.Int("max-keys", 0, "most distinct keys allowed (required, a hard bound)") + maxLine := fs.Int("max-line-bytes", 0, "longest accepted input line (required)") + out := fs.String("out", "", "output file (must not exist); default stdout") + if err := fs.Parse(args); err != nil { + return exitUsage + } + usageErr := func(err error) int { + fmt.Fprintf(stderr, "keys: %v\n", err) + return exitUsage + } + if fs.NArg() != 0 { + return usageErr(errors.New("unexpected arguments")) + } + if *input == "" { + return usageErr(errors.New("-input is required")) + } + if *payloadType < 0 || *payloadType > int(anomaly.MaxPayloadType) { + return usageErr(fmt.Errorf("-payload-type is required and must be in 0..%d", anomaly.MaxPayloadType)) + } + if *top <= 0 || *maxKeys <= 0 || *maxLine <= 0 { + return usageErr(errors.New("-top, -max-keys and -max-line-bytes must be > 0")) + } + excl, err := parseExcludeList(*exclude) + if err != nil { + return usageErr(err) + } + fail := func(err error) int { + fmt.Fprintf(stderr, "keys: %v\n", err) + return exitFail + } + in := stdin + if *input != "-" { + f, err := os.Open(*input) + if err != nil { + return fail(err) + } + defer f.Close() + in = f + } + res, err := rankKeys(in, *payloadType, excl, *top, *maxKeys, *maxLine) + if err != nil { + return fail(err) + } + b, err := json.MarshalIndent(res, "", " ") + if err != nil { + return fail(err) + } + b = append(b, '\n') + if *out == "" { + _, err = stdout.Write(b) + } else { + err = writeNew(*out, b) + } + if err != nil { + return fail(err) + } + return exitOK +} + +// parseExcludeList parses the -exclude-channels list: lower-case hex hashes, +// no duplicates, nothing empty. The same canonical form the rule's +// exclude_channels takes. +func parseExcludeList(s string) ([]string, error) { + out := []string{} + seen := map[string]bool{} + for _, v := range strings.Split(s, ",") { + v = strings.TrimSpace(v) + if v == "" { + continue + } + if strings.ToLower(v) != v { + return nil, errors.New("-exclude-channels wants lower-case hex hashes") + } + b, err := hex.DecodeString(v) + if err != nil || len(b) == 0 { + return nil, fmt.Errorf("-exclude-channels: %q is not a channel hash", v) + } + if _, err := anomaly.NewChannelHash(b); err != nil { + return nil, fmt.Errorf("-exclude-channels: %q: %w", v, err) + } + if seen[v] { + return nil, fmt.Errorf("-exclude-channels: %q twice", v) + } + seen[v] = true + out = append(out, v) + } + sort.Strings(out) + return out, nil +} + +// rankKeys counts the background's stream keys and returns the top ones. +// +// The filter is mechanical and fixed: the line's payload type must be the +// requested one, it must carry a channel hash (the periodic rule's target is +// a channel bot), its route kind must be relay (only then is the first hop +// part of the stream key, so only then can a bot share the key), and its +// channel must not be excluded. +func rankKeys(r io.Reader, payloadType int, exclude []string, top, maxKeys, maxLine int) (keysResult, error) { + res := keysResult{Experimental: true, PayloadType: payloadType, ExcludeChannels: exclude, + Filter: "payload_type == -payload-type, channel hash present, route_kind == relay, channel not excluded", + Top: []keyStat{}} + excluded := map[string]bool{} + for _, v := range exclude { + excluded[v] = true + } + aggs := map[anomaly.Key]*keyAgg{} + sc := bufio.NewScanner(r) + limit := maxLine + 2 + initial := 4096 + if limit < initial { + initial = limit + } + sc.Buffer(make([]byte, 0, initial), limit) + for sc.Scan() { + line := sc.Bytes() + if len(line) > maxLine { + return res, errors.New("input line longer than -max-line-bytes") + } + if len(strings.TrimSpace(string(line))) == 0 { + continue + } + res.InputLines++ + var rec anomaly.ReplayRecord + if err := json.Unmarshal(line, &rec); err != nil { + return res, fmt.Errorf("line %d: %w", res.InputLines, err) + } + if rec.PayloadType != payloadType { + res.SkippedType++ + continue + } + if rec.Channel == "" { + res.SkippedNoChannel++ + continue + } + if rec.RouteKind != "relay" { + res.SkippedNotRelay++ + continue + } + if excluded[rec.Channel] { + res.SkippedExcluded++ + continue + } + ev, err := rec.Event() + if err != nil { + return res, fmt.Errorf("line %d: %w", res.InputLines, err) + } + k := anomaly.StreamKeyOf(ev) + a := aggs[k] + if a == nil { + if len(aggs) >= maxKeys { + return res, errors.New("more than -max-keys distinct keys; raise the bound") + } + a = &keyAgg{k: k, payload: map[int]int{}, wire: map[int]int{}, + pairs: map[[2]int]int{}, routes: map[string]int{}} + aggs[k] = a + } + res.Considered++ + a.events++ + if a.first.IsZero() || rec.Time.Before(a.first) { + a.first = rec.Time.UTC() + } + if rec.Time.After(a.last) { + a.last = rec.Time.UTC() + } + bump(a.wire, rec.WireBytes, maxKeySizeBuckets, &a.sizeDropped) + pb := -1 + if rec.PayloadBytes != nil { + pb = *rec.PayloadBytes + bump(a.payload, pb, maxKeySizeBuckets, &a.sizeDropped) + } + bumpPair(a.pairs, [2]int{rec.WireBytes, pb}, maxKeySizePairs, &a.pairDropped) + bumpStr(a.routes, rec.FullRoute, maxKeyRouteVariants, &a.routeDropped) + } + if err := sc.Err(); err != nil { + return res, fmt.Errorf("input: %w", err) + } + res.DistinctKeys = len(aggs) + ranked := make([]*keyAgg, 0, len(aggs)) + for _, a := range aggs { + ranked = append(ranked, a) + } + // Row count, then the canonical key encoding: a total order that does + // not depend on map iteration. + sort.Slice(ranked, func(i, j int) bool { + if ranked[i].events != ranked[j].events { + return ranked[i].events > ranked[j].events + } + return ranked[i].k.String() < ranked[j].k.String() + }) + if len(ranked) > top { + ranked = ranked[:top] + } + for i, a := range ranked { + res.Top = append(res.Top, a.stat(i+1)) + } + return res, nil +} + +func bump(m map[int]int, v, max int, dropped *int) { + if _, ok := m[v]; !ok && len(m) >= max { + *dropped++ + return + } + m[v]++ +} + +func bumpPair(m map[[2]int]int, v [2]int, max int, dropped *int) { + if _, ok := m[v]; !ok && len(m) >= max { + *dropped++ + return + } + m[v]++ +} + +func bumpStr(m map[string]int, v string, max int, dropped *int) { + if _, ok := m[v]; !ok && len(m) >= max { + *dropped++ + return + } + m[v]++ +} + +func (a *keyAgg) stat(rank int) keyStat { + s := keyStat{Rank: rank, Key: a.k.String(), PayloadType: int(a.k.PayloadType), + Channel: hexChannel(a.k), RouteClass: a.k.RouteClass.String(), RouteKind: a.k.RouteKind.String(), + FirstHop: encodeHop(a.k.FirstHop), Events: a.events, First: a.first, Last: a.last, + SizeBucketsDropped: a.sizeDropped, RouteVariantsDropped: a.routeDropped, + SizeMix: sizeMixOf(a.pairs), SizeMixDropped: a.pairDropped} + s.FullRoute = modeString(a.routes) + s.WireBytesMedian = int(histPercentile(a.wire, 0.5)) + if len(a.payload) > 0 { + med := int(histPercentile(a.payload, 0.5)) + p10 := int(histPercentile(a.payload, 0.1)) + p90 := int(histPercentile(a.payload, 0.9)) + s.PayloadBytesMedian = med + spread := (p90 - p10) / 2 + if spread >= med { + spread = med - 1 + } + if spread < 0 { + spread = 0 + } + s.PayloadBytesSpread = spread + } + return s +} + +// sizeMixOf renders a pair histogram in a fixed order: wire size, then +// payload size, with the unknown payload (-1) first for a given wire size. +// The order is a pure function of the histogram, so it does not depend on +// map iteration. +func sizeMixOf(m map[[2]int]int) []sizePair { + out := make([]sizePair, 0, len(m)) + for v, n := range m { + p := sizePair{WireBytes: v[0], Count: n} + if v[1] >= 0 { + pb := v[1] + p.PayloadBytes = &pb + } + out = append(out, p) + } + sort.Slice(out, func(i, j int) bool { + if out[i].WireBytes != out[j].WireBytes { + return out[i].WireBytes < out[j].WireBytes + } + return payloadOrUnknown(out[i]) < payloadOrUnknown(out[j]) + }) + return out +} + +func payloadOrUnknown(p sizePair) int { + if p.PayloadBytes == nil { + return -1 + } + return *p.PayloadBytes +} + +// encodeHop renders a first hop in the "width:hex" form a replay record and +// an injection spec use. +func encodeHop(h anomaly.Hop) string { + b := h.Bytes() + if len(b) == 0 { + return "" + } + return fmt.Sprintf("%d:%x", len(b), b) +} + +// modeString returns the most common value, ties broken by the smallest. +func modeString(m map[string]int) string { + best, bestN := "", -1 + for v, n := range m { + if n > bestN || (n == bestN && v < best) { + best, bestN = v, n + } + } + return best +} + +// histPercentile is the nearest-rank percentile of a value histogram: the +// value at index ceil(q*n)-1 of the sorted values, clamped into range. It is +// a pure function of the histogram, so it does not depend on map order. +func histPercentile(m map[int]int, q float64) int64 { + n := 0 + vals := make([]int, 0, len(m)) + for v, c := range m { + n += c + vals = append(vals, v) + } + if n == 0 { + return 0 + } + sort.Ints(vals) + idx := int(math.Ceil(q*float64(n))) - 1 + if idx < 0 { + idx = 0 + } + if idx > n-1 { + idx = n - 1 + } + seen := 0 + for _, v := range vals { + seen += m[v] + if seen > idx { + return int64(v) + } + } + return int64(vals[len(vals)-1]) +} diff --git a/cmd/anomaly/main.go b/cmd/anomaly/main.go index c1d7d9f3c..2623c37f9 100644 --- a/cmd/anomaly/main.go +++ b/cmd/anomaly/main.go @@ -5,12 +5,21 @@ // corescope-anomaly export -db PATH [-immutable] -start T -end T -settle D -max-rows N -out FILE // [-backlog-silence D -backlog-span D] // corescope-anomaly replay -config FILE -input FILE|- -start T -end T -max-line-bytes N [-out FILE] +// corescope-anomaly keys -input FILE|- -payload-type N -top N [-exclude-channels LIST] -max-keys N -max-line-bytes N [-out FILE] +// corescope-anomaly inject -spec FILE -input FILE|- -out FILE -labels FILE -max-injected N -max-line-bytes N +// corescope-anomaly score -report FILE [-labels FILE] [-baseline FILE] -rule NAME -start T -end T [-out FILE] // // export reads a CoreScope SQLite database read-only (mode=ro; -immutable for // a static snapshot copy) and writes anomaly.ReplayRecord JSON Lines built // from on-wire fields only, with per-export IDs. replay runs anomaly.Replay -// over such lines and prints the deterministic report. Every bound is an -// explicit flag; there are no defaults. +// over such lines and prints the deterministic report. keys ranks the stream +// keys of such lines by row count, so an injection panel can be placed on +// keys real traffic uses. inject overlays synthetic channel-bot trains +// (clearly marked "inj-..." ids) on such lines and writes the ground truth +// beside them; score measures a replay report against that ground truth +// (#474). Only export reads a database, and +// nothing here ever writes to one. Every bound is an explicit flag; there +// are no defaults. package main import ( @@ -25,6 +34,9 @@ const usage = `usage: corescope-anomaly export -db PATH [-immutable] -start RFC3339 -end RFC3339 -settle DURATION -max-rows N -out FILE [-backlog-silence DURATION -backlog-span DURATION] corescope-anomaly replay -config FILE -input FILE|- -start RFC3339 -end RFC3339 -max-line-bytes N [-out FILE] + corescope-anomaly keys -input FILE|- -payload-type N -top N [-exclude-channels LIST] -max-keys N -max-line-bytes N [-out FILE] + corescope-anomaly inject -spec FILE -input FILE|- -out FILE -labels FILE -max-injected N -max-line-bytes N + corescope-anomaly score -report FILE [-labels FILE] [-baseline FILE] -rule NAME -start RFC3339 -end RFC3339 [-out FILE] ` // Exit codes: 0 success, 1 runtime failure, 2 usage error. @@ -48,6 +60,12 @@ func run(args []string, stdin io.Reader, stdout, stderr io.Writer) int { return runExport(args[1:], stderr) case "replay": return runReplay(args[1:], stdin, stdout, stderr) + case "keys": + return runKeys(args[1:], stdin, stdout, stderr) + case "inject": + return runInject(args[1:], stdin, stderr) + case "score": + return runScore(args[1:], stdout, stderr) } fmt.Fprint(stderr, usage) return exitUsage diff --git a/cmd/anomaly/make_spec.py b/cmd/anomaly/make_spec.py new file mode 100644 index 000000000..80a8a6924 --- /dev/null +++ b/cmd/anomaly/make_spec.py @@ -0,0 +1,89 @@ +# make_spec.py - builds the #474 round-3 injection spec deterministically. +# python3 make_spec.py > spec.json (stdlib only) +# +# The panel is paired. Each shape appears twice at the same time, on two +# different stream keys: +# shared - the key of real background traffic, taken from the keys ranking +# by row count (rank 1..9 in order). This is the arm that +# measures detection inside real traffic. +# isolated - the same channel byte with the 3-byte first hop fdff0i, a key +# no real relayed line carries. This is the round-1 control. +# The two twins never interact: they are different keys. +# +# Sizes come from the key's own observed (wire_bytes, payload_bytes) +# distribution, passed through verbatim as size_mix. Round 2 instead held +# wire_bytes at the key's median while drawing the payload from +# median +/- (p90 - p10) / 2, which has two faults: a real GRP_TXT payload +# comes in 16-byte cipher blocks while wire - payload is only header plus +# path, so the drawn payload routinely exceeded wire - 2 and inject aborted +# on the event validator (#478 review G1); and even where it fitted, the +# drawn sizes were ones the key never carried, so an injected line was +# distinguishable on size alone (G2). Drawing whole observed pairs fixes +# both: every pair came from a real line, so it is valid by construction and +# indistinguishable by size. +# +# Jitter comes from #130's measurements of a real 5-minute bot over 85.2 h +# (|gap - 300 s| median 3 s, p95 9 s), not from the rule's tolerance: a gap +# offset is the difference of two per-pulse offsets, so sd(gap) = sqrt(2) * +# sd(pulse), and p95 9 s gives sd(gap) ~ 9/1.96 ~ 4.6 s and sd(pulse) ~ 3.2 s. +# So every bot gets jitter_sd 3 s and jitter_max 9 s, whatever its period and +# whatever the rule's tolerance is. +import json, sys +from datetime import datetime, timedelta, timezone + +V0 = datetime.strptime(sys.argv[1], "%Y-%m-%dT%H:%M:%SZ").replace(tzinfo=timezone.utc) +RANKING = json.load(open(sys.argv[2])) + + +def iso(t): + return t.strftime("%Y-%m-%dT%H:%M:%SZ") + + +# Panel: (period seconds, shape). Shapes: +# clean = jitter only +# noisy = jitter + 5 % missed pulses + 30 % multi-part pulses (#130 shape) +# sept = noisy plus the one 35-minute outage of the September bot +PANEL = [(90, "clean"), (120, "clean"), (300, "clean"), (600, "clean"), (900, "clean"), + (90, "noisy"), (300, "noisy"), (900, "noisy"), (300, "sept")] +HOURS_EACH, HOURS_APART = 12, 18 +JITTER_SD, JITTER_MAX = 3, 9 # #130, measured +SEED = 474202610170 + +keys = RANKING["top"] +if len(keys) < len(PANEL): + sys.exit("keys ranking has %d keys, need %d" % (len(keys), len(PANEL))) + +bots = [] +for i, (p, shape) in enumerate(PANEL): + k = keys[i] # rank i+1, by row count alone + start = V0 + timedelta(hours=1 + HOURS_APART * i) + stop = start + timedelta(hours=HOURS_EACH) + mpd = min(55, p // 4) + noisy = shape in ("noisy", "sept") + holes = [{"start": iso(start + timedelta(hours=6)), "duration": "35m"}] if shape == "sept" else [] + # The key's observed size distribution, verbatim. inject draws one whole + # (wire, payload) pair per event, weighted by count, so the injected + # sizes are exactly the sizes the key carries. + size_mix = k["size_mix"] + if not size_mix: + sys.exit("rank %d has no observed size pair" % (i + 1)) + for arm, hop in (("shared", k["first_hop"]), ("iso", "3:fdff%02x" % i)): + bots.append({ + "name": "b%d-%ds-%s-%s" % (i, p, shape, arm), + "channel": k["channel"], "payload_type": RANKING["payload_type"], + "route_class": k["route_class"], "route_kind": k["route_kind"], + "first_hop": hop, "full_route": k["full_route"] if arm == "shared" else hop, + "size_mix": size_mix, + "start": iso(start), "stop": iso(stop), "period": "%ds" % p, + "jitter_sd": "%ds" % JITTER_SD, "jitter_max": "%ds" % JITTER_MAX, + "missed_fraction": 0.05 if noisy else 0.0, + "multipart_fraction": 0.3 if noisy else 0.0, + "multipart_max_extra": 2, "multipart_max_delay": "%ds" % mpd, + "holes": holes, + }) + +print(json.dumps({"seed": SEED, "final_after": "30s", + "labelling": {"pulse_merge_gap": "60s", "min_period": "1m", "max_period": "1h", + "min_pulses": 8, "max_harmonic": 3, + "tolerance_abs": "30s", "tolerance_rel": 0.1}, + "bots": bots}, indent=1)) diff --git a/cmd/anomaly/review_r2_test.go b/cmd/anomaly/review_r2_test.go new file mode 100644 index 000000000..485e7c478 --- /dev/null +++ b/cmd/anomaly/review_r2_test.go @@ -0,0 +1,636 @@ +package main + +import ( + "bytes" + "encoding/json" + "fmt" + "os" + "path/filepath" + "strings" + "testing" + "time" + + "github.com/meshcore-analyzer/anomaly" +) + +// Round-2 review of PR #478 (#474): the boundary cases the first round's +// mutants survived, plus the parity and key-sharing properties the +// pre-registration now depends on. Every bot, channel byte and time here is +// synthetic. + +// r2Labelling is the labelling recipe of the pre-registration: a 60 s merge +// gap, P in [1 min, 1 h], k in 1..3 and a tolerance of max(30 s, 0.1 P). +func r2Labelling(minPulses int) labellingSpec { + return labellingSpec{PulseMergeGap: dur(60 * time.Second), MinPeriod: dur(time.Minute), + MaxPeriod: dur(time.Hour), MinPulses: minPulses, MaxHarmonic: 3, + ToleranceAbs: dur(30 * time.Second), ToleranceRel: 0.1} +} + +// r2Pulses builds one injected event per offset, which is all +// labelIncidents reads. +func r2Pulses(offsets ...time.Duration) []injEvent { + evs := make([]injEvent, 0, len(offsets)) + for i, d := range offsets { + evs = append(evs, injEvent{t: injT.Add(d).UnixNano(), pulse: i}) + } + return evs +} + +// grid returns n offsets exactly p apart. +func grid(n int, p time.Duration) []time.Duration { + out := make([]time.Duration, 0, n) + for i := 0; i < n; i++ { + out = append(out, time.Duration(i)*p) + } + return out +} + +// F5: a run of exactly MinPulses pulses is an incident, and one pulse fewer +// is not. Without the exact-boundary case, labelling with ">" instead of +// ">=" (nine pulses required) passes every other test. +func TestLabelIncidentsAtExactlyMinPulses(t *testing.T) { + const p = 300 * time.Second + exact := grid(8, p) + inc := labelIncidents("bot-a", "7f", scoreKey, r2Pulses(exact...), r2Labelling(8)) + if len(inc) != 1 || inc[0].Pulses != 8 { + t.Fatalf("exactly 8 pulses must be one 8-pulse incident: %+v", inc) + } + if got := labelIncidents("bot-a", "7f", scoreKey, r2Pulses(exact[:7]...), r2Labelling(8)); len(got) != 0 { + t.Fatalf("7 pulses must not be an incident: %+v", got) + } +} + +// F5: a gap exactly max(tolerance_abs, tolerance_rel*P) away from k*P is +// inside the tolerance. P = 300 s gives a tolerance of exactly 30 s, so a +// 330 s gap is the boundary and a 331 s gap is outside it. +func TestLabelIncidentsAtTheToleranceBoundary(t *testing.T) { + const p = 300 * time.Second + at := func(last time.Duration) []time.Duration { + return append(grid(7, p), 6*p+last) + } + inc := labelIncidents("bot-a", "7f", scoreKey, r2Pulses(at(330*time.Second)...), r2Labelling(8)) + if len(inc) != 1 || inc[0].Pulses != 8 { + t.Fatalf("a gap exactly at the tolerance must stay in the run: %+v", inc) + } + // One nanosecond further out and the run breaks before the last pulse, + // which leaves 7 pulses: below min_pulses, so no incident at all. + if got := labelIncidents("bot-a", "7f", scoreKey, r2Pulses(at(330*time.Second+1)...), r2Labelling(8)); len(got) != 0 { + t.Fatalf("a gap past the tolerance must break the run: %+v", got) + } +} + +// F5: two events exactly pulse_merge_gap apart are two pulses, not one. The +// merge keeps events strictly closer than the gap together. +func TestLabelIncidentsAtThePulseMergeGapBoundary(t *testing.T) { + l := r2Labelling(2) + inc := labelIncidents("bot-a", "7f", scoreKey, r2Pulses(0, 60*time.Second), l) + if len(inc) != 1 || inc[0].Pulses != 2 || time.Duration(inc[0].Period) != 60*time.Second { + t.Fatalf("events exactly 60 s apart are two pulses: %+v", inc) + } + // One nanosecond closer and they merge, which leaves a single pulse and + // no incident. + if got := labelIncidents("bot-a", "7f", scoreKey, r2Pulses(0, 60*time.Second-1), l); len(got) != 0 { + t.Fatalf("events closer than the gap are one pulse: %+v", got) + } +} + +// r2Cand builds one report candidate. period == 0 means no periodic +// evidence on that candidate. +func r2Cand(id, to, key string, at time.Time, period time.Duration) scoreCandidate { + c := scoreCandidate{EventID: id, Rule: "p474", Kind: "periodic", Key: key, To: to, At: at} + if period > 0 { + c.Periodic = &struct{ Period time.Duration }{Period: period} + } + return c +} + +func r2Report(cands ...scoreCandidate) scoreReport { + return scoreReport{Experimental: true, Start: injT, End: injT.Add(24 * time.Hour), Candidates: cands} +} + +// F5: an episode whose period is exactly the tolerance away from k*P is a +// true positive, not a period mismatch. +func TestScorePeriodFitsAtTheToleranceBoundary(t *testing.T) { + const p = 300 * time.Second + l := r2Labelling(8) + if !periodFits(p+30*time.Second, p, l) { + t.Fatal("a period exactly at the tolerance must fit") + } + if periodFits(p+30*time.Second+1, p, l) { + t.Fatal("a period past the tolerance must not fit") + } + // The same through score: the boundary episode is a true positive and + // raises no period mismatch. + inc := Incident{Bot: "bot-a", Channel: "7f", Key: scoreKey, Start: injT, Stop: injT.Add(time.Hour), + Period: dur(p), Pulses: 8} + lab := InjectLabels{Synthetic: true, Labelling: l, Incidents: []Incident{inc}} + rep := r2Report(r2Cand("e1", "active", scoreKey, injT.Add(30*time.Minute), p+30*time.Second)) + res, err := score(rep, lab, "p474", "labels.json", "", nil, nil, injT, injT.Add(24*time.Hour)) + if err != nil { + t.Fatal(err) + } + if res.TruePositives != 1 || res.PeriodMismatches != 0 || res.IncidentsDetected != 1 { + t.Fatalf("boundary period: %+v", res) + } +} + +// F5: an episode's end is its last transition to ended, not its first. A +// reopened episode that ends twice must be measured to the later end, or the +// stop latency is understated. +func TestScoreTakesTheLastEnded(t *testing.T) { + act := injT.Add(time.Hour) + first := injT.Add(2 * time.Hour) + last := injT.Add(3 * time.Hour) + rep := r2Report( + r2Cand("e1", "active", scoreKey, act, 300*time.Second), + r2Cand("e1", "ended", scoreKey, first, 0), + r2Cand("e1", "ended", scoreKey, last, 0), + ) + eps, noPeriod, err := episodes(rep, "p474", injT, injT.Add(24*time.Hour)) + if err != nil { + t.Fatal(err) + } + if len(eps) != 1 || noPeriod != 0 { + t.Fatalf("episodes %+v (no period %d)", eps, noPeriod) + } + if !eps[0].end.Equal(last) { + t.Fatalf("end %v, want the last ended %v", eps[0].end, last) + } +} + +// F6: the export times every line from a whole-second observation time, so +// an injected line must too. A fractional second would make every injected +// line trivially distinguishable from an exported one. +func TestInjectTimesAreWholeSeconds(t *testing.T) { + r := injectInto(t, specJSON(specOpts{seed: 31, jitterSD: "3s", multipartFraction: 0.5}), + backgroundJSONL(40)) + recs := injectedLines(t, r) + if len(recs) < 20 { + t.Fatalf("only %d injected lines", len(recs)) + } + for _, rec := range recs { + if rec.Time.Nanosecond() != 0 || rec.FinalAt.Nanosecond() != 0 { + t.Fatalf("%s: time %v final_at %v are not whole seconds", rec.ID, rec.Time, rec.FinalAt) + } + } + // The same on the wire: no injected line carries a fractional second. + for _, l := range r.lines { + if !strings.Contains(l, `"id":"`+injPrefix) { + continue + } + for _, f := range []string{`"time":"`, `"final_at":"`} { + i := strings.Index(l, f) + if i < 0 { + t.Fatalf("line %q has no %s", l, f) + } + v := l[i+len(f):] + v = v[:strings.IndexByte(v, '"')] + if strings.Contains(v, ".") { + t.Fatalf("%s %s carries a fractional second in %q", f, v, l) + } + } + } + // The labels are derived from the same whole-second pulses. + for _, inc := range r.labels.Incidents { + if inc.Start.Nanosecond() != 0 || inc.Stop.Nanosecond() != 0 { + t.Fatalf("incident %+v is not on whole seconds", inc) + } + } +} + +// F6: a real bot's payload length varies. payload_bytes_spread draws each +// event's size uniformly from payload_bytes +/- the spread, and a spread of +// 0 keeps the old constant size. +func TestInjectPayloadSizeVariesWithinTheSpread(t *testing.T) { + const base, spread = 35, 6 + r := injectInto(t, specJSON(specOpts{seed: 31, payloadSpread: spread}), backgroundJSONL(40)) + seen := map[int]bool{} + for _, rec := range injectedLines(t, r) { + if rec.PayloadBytes == nil { + t.Fatalf("%s has no payload_bytes", rec.ID) + } + pb := *rec.PayloadBytes + if pb < base-spread || pb > base+spread { + t.Fatalf("%s payload %d outside %d +/- %d", rec.ID, pb, base, spread) + } + seen[pb] = true + } + if len(seen) < 5 { + t.Fatalf("payload size barely varies: %v", seen) + } + flat := injectInto(t, specJSON(specOpts{seed: 31}), backgroundJSONL(40)) + for _, rec := range injectedLines(t, flat) { + if *rec.PayloadBytes != base { + t.Fatalf("spread 0 must keep the size constant, got %d", *rec.PayloadBytes) + } + } +} + +// F6: the spread is a required field with no default, and it may never push +// the payload size to zero or below. +func TestInjectPayloadSpreadIsValidated(t *testing.T) { + bot := func(extra string) string { + return `{"seed":7,"final_after":"30s", + "labelling":{"pulse_merge_gap":"60s","min_period":"1m","max_period":"1h","min_pulses":8, + "max_harmonic":3,"tolerance_abs":"30s","tolerance_rel":0.1}, + "bots":[{"name":"bot-a","channel":"7f","payload_type":5,"route_class":"flood","route_kind":"relay", + "first_hop":"1:fd","full_route":"1:fd0a","wire_bytes":60,` + extra + ` + "start":"` + injAt(0) + `","stop":"` + injAt(time.Hour) + `","period":"5m","jitter_sd":"3s", + "jitter_max":"10s","missed_fraction":0,"multipart_fraction":0,"multipart_max_extra":2, + "multipart_max_delay":"55s","holes":[]}]}` + } + for _, tc := range []struct{ name, extra string }{ + {"missing", `"payload_bytes":35,`}, + {"negative", `"payload_bytes":35,"payload_bytes_spread":-1,`}, + {"as wide as the payload", `"payload_bytes":35,"payload_bytes_spread":35,`}, + {"without a payload size", `"payload_bytes_spread":3,`}, + {"zero payload size", `"payload_bytes":0,"payload_bytes_spread":0,`}, + // Round 3: the top of the draw range has to fit the frame too, not + // only payload_bytes itself (see TestInjectSpreadEnvelopeMustFitTheFrame). + {"wider than the frame allows", `"payload_bytes":35,"payload_bytes_spread":34,`}, + } { + if _, err := parseInjectSpec([]byte(bot(tc.extra))); err == nil { + t.Fatalf("%s: want an error", tc.name) + } + } + // A spread just inside both bounds - below payload_bytes, and with its + // top at exactly wire_bytes - 2 - is legal. + if _, err := parseInjectSpec([]byte(bot(`"payload_bytes":35,"payload_bytes_spread":23,`))); err != nil { + t.Fatalf("a spread whose envelope fits the frame is legal: %v", err) + } +} + +// F6: final_after must be whole seconds, exactly as the export's -settle is. +func TestInjectFinalAfterIsWholeSeconds(t *testing.T) { + spec := strings.Replace(specJSON(specOpts{seed: 7}), `"final_after": "30s"`, `"final_after": "30.5s"`, 1) + if _, err := parseInjectSpec([]byte(spec)); err == nil { + t.Fatal("a fractional final_after must be refused") + } +} + +// keysOn runs the keys command over one background and returns the parsed +// ranking. +func keysOn(t *testing.T, background string, extra ...string) keysResult { + t.Helper() + dir := t.TempDir() + in := filepath.Join(dir, "bg.jsonl") + if err := os.WriteFile(in, []byte(background), 0o644); err != nil { + t.Fatal(err) + } + args := append([]string{"keys", "-input", in, "-payload-type", "5", "-top", "8", + "-max-keys", "1000", "-max-line-bytes", "65536"}, extra...) + var stdout, stderr bytes.Buffer + if code := run(args, nil, &stdout, &stderr); code != 0 { + t.Fatalf("keys exit %d: %s", code, stderr.String()) + } + var res keysResult + if err := json.Unmarshal(stdout.Bytes(), &res); err != nil { + t.Fatalf("keys output %q: %v", stdout.String(), err) + } + return res +} + +// r2Background writes n events spread over the given (channel, first hop, +// full route) series, with the share of each series fixed by weight. +type r2Series struct { + channel, firstHop, fullRoute string + weight int + payload int +} + +func r2Background(series []r2Series, rounds int) string { + var b strings.Builder + t := injT + for i := 0; i < rounds; i++ { + for _, s := range series { + for w := 0; w < s.weight; w++ { + t = t.Add(17 * time.Second) + pb := s.payload + w + rec := anomaly.ReplayRecord{ID: fmt.Sprintf("tx%d", b.Len()+w+1), Time: t, + FinalAt: t.Add(30 * time.Second), PayloadType: 5, Channel: s.channel, + RouteClass: "flood", RouteKind: "relay", FirstHop: s.firstHop, + FullRoute: s.fullRoute, WireBytes: 55, PayloadBytes: &pb, + Traffic: "unclassified", Decryption: "unknown"} + j, err := json.Marshal(rec) + if err != nil { + panic(err) + } + b.Write(j) + b.WriteByte('\n') + } + } + } + return b.String() +} + +// F2: the panel has to sit on stream keys real traffic uses, and those keys +// have to be picked mechanically. keys ranks the background's own stream +// keys by row count - nothing else - and reports the wire fields an +// injection spec needs to land on each one. +func TestKeysRanksBackgroundStreamKeysByRowCount(t *testing.T) { + res := keysOn(t, r2Background([]r2Series{ + {channel: "0b", firstHop: "1:5c", fullRoute: "1:5c0a1c", weight: 5, payload: 30}, + {channel: "0b", firstHop: "2:7a01", fullRoute: "2:7a01", weight: 2, payload: 40}, + {channel: "81", firstHop: "1:5c", fullRoute: "1:5c", weight: 9, payload: 20}, + }, 6), "-exclude-channels", "81") + if res.DistinctKeys != 2 || len(res.Top) != 2 { + t.Fatalf("ranking %+v", res) + } + if res.SkippedExcluded != 54 { + t.Fatalf("the excluded channel must be skipped, got %d", res.SkippedExcluded) + } + top, second := res.Top[0], res.Top[1] + if top.Rank != 1 || top.Events != 30 || top.Channel != "0b" || top.FirstHop != "1:5c" { + t.Fatalf("top key %+v", top) + } + if second.Rank != 2 || second.Events != 12 || second.FirstHop != "2:7a01" { + t.Fatalf("second key %+v", second) + } + // The ranking is the key encoding the report writes, so a spec built + // from it lands on exactly that key. + if top.Key != "v1|s=stream|pt=5|ch=1:0b|rc=flood|rk=relay|hop=1:5c" { + t.Fatalf("top key encoding %q", top.Key) + } + // The sizes and the route are the realistic values for a bot on the + // key: the observed whole route, not the first hop repeated. + if top.FullRoute != "1:5c0a1c" { + t.Fatalf("full route %q", top.FullRoute) + } + if top.PayloadBytesMedian < 30 || top.PayloadBytesMedian > 34 || top.PayloadBytesSpread == 0 { + t.Fatalf("payload figures %+v", top) + } + if top.WireBytesMedian != 55 { + t.Fatalf("wire median %d", top.WireBytesMedian) + } + if top.First.IsZero() || top.Last.Before(top.First) { + t.Fatalf("times %v..%v", top.First, top.Last) + } +} + +// F2: the filter is part of the selection rule, so it is restated in the +// output and every skipped line is counted. A line of another payload type, +// without a channel hash or not relayed cannot share a stream key with a +// channel bot and never enters the ranking. +func TestKeysFilterIsMechanicalAndCounted(t *testing.T) { + lines := []string{ + `{"id":"tx1","time":"2030-03-01T00:00:01Z","payload_type":5,"channel":"0b","route_class":"flood","route_kind":"relay","first_hop":"1:5c","full_route":"1:5c","wire_bytes":55,"traffic":"unclassified","decryption":"unknown"}`, + `{"id":"tx2","time":"2030-03-01T00:00:02Z","payload_type":4,"channel":"0b","route_class":"flood","route_kind":"relay","first_hop":"1:5c","full_route":"1:5c","wire_bytes":55,"traffic":"unclassified","decryption":"unknown"}`, + `{"id":"tx3","time":"2030-03-01T00:00:03Z","payload_type":5,"channel":"","route_class":"flood","route_kind":"relay","first_hop":"1:5c","full_route":"1:5c","wire_bytes":55,"traffic":"unclassified","decryption":"unknown"}`, + `{"id":"tx4","time":"2030-03-01T00:00:04Z","payload_type":5,"channel":"0b","route_class":"direct","route_kind":"direct","first_hop":"","full_route":"","wire_bytes":55,"traffic":"unclassified","decryption":"unknown"}`, + `{"id":"tx5","time":"2030-03-01T00:00:05Z","payload_type":5,"channel":"81","route_class":"flood","route_kind":"relay","first_hop":"1:5c","full_route":"1:5c","wire_bytes":55,"traffic":"unclassified","decryption":"unknown"}`, + } + res := keysOn(t, strings.Join(lines, "\n")+"\n", "-exclude-channels", "81") + if res.InputLines != 5 || res.Considered != 1 || res.DistinctKeys != 1 { + t.Fatalf("counts %+v", res) + } + if res.SkippedType != 1 || res.SkippedNoChannel != 1 || res.SkippedNotRelay != 1 || res.SkippedExcluded != 1 { + t.Fatalf("skip counters %+v", res) + } + if !strings.Contains(res.Filter, "route_kind == relay") || res.PayloadType != 5 { + t.Fatalf("the output must restate the filter: %+v", res) + } + if len(res.ExcludeChannels) != 1 || res.ExcludeChannels[0] != "81" { + t.Fatalf("exclusions %+v", res.ExcludeChannels) + } +} + +// F2: -max-keys is a hard bound, as the export's -max-rows is, so a run +// never silently reports a truncated ranking. +func TestKeysMaxKeysIsAHardBound(t *testing.T) { + bg := r2Background([]r2Series{ + {channel: "0b", firstHop: "1:5c", fullRoute: "1:5c", weight: 1, payload: 30}, + {channel: "0c", firstHop: "1:5d", fullRoute: "1:5d", weight: 1, payload: 30}, + }, 2) + dir := t.TempDir() + in := filepath.Join(dir, "bg.jsonl") + if err := os.WriteFile(in, []byte(bg), 0o644); err != nil { + t.Fatal(err) + } + var stdout, stderr bytes.Buffer + code := run([]string{"keys", "-input", in, "-payload-type", "5", "-top", "8", + "-max-keys", "1", "-max-line-bytes", "65536"}, nil, &stdout, &stderr) + if code != exitFail || !strings.Contains(stderr.String(), "-max-keys") { + t.Fatalf("exit %d, stderr %q", code, stderr.String()) + } + if stdout.Len() != 0 { + t.Fatalf("a failed run must write no ranking, got %q", stdout.String()) + } +} + +// otherHopKey is the same channel hash and payload type as scoreKey, but a +// different first hop: a second route group on one busy channel. +const otherHopKey = "v1|s=stream|pt=5|ch=1:7f|rc=flood|rk=relay|hop=1:ee" + +// F2: once the panel shares a channel with real traffic, matching on the +// channel hash alone is wrong. An episode of a different route group on the +// same channel is a false positive, not a detection of the bot. +func TestScoreMatchesOnTheStreamKeyNotTheChannel(t *testing.T) { + const p = 300 * time.Second + l := r2Labelling(8) + inc := Incident{Bot: "bot-a", Channel: "7f", Key: scoreKey, Start: injT, + Stop: injT.Add(time.Hour), Period: dur(p), Pulses: 8} + lab := InjectLabels{Synthetic: true, Labelling: l, Incidents: []Incident{inc}} + // One episode, same channel hash, same period, overlapping - but + // another route group. + rep := r2Report(r2Cand("e1", "active", otherHopKey, injT.Add(30*time.Minute), p)) + res, err := score(rep, lab, "p474", "labels.json", "", nil, nil, injT, injT.Add(24*time.Hour)) + if err != nil { + t.Fatal(err) + } + if res.IncidentsDetected != 0 || res.Recall != 0 || res.TruePositives != 0 { + t.Fatalf("another route group is not the bot: %+v", res) + } + if res.FalsePositives != 1 || res.PeriodMismatches != 0 { + t.Fatalf("it is a false positive: %+v", res) + } + // On the bot's own key the same episode is the detection. + rep = r2Report(r2Cand("e1", "active", scoreKey, injT.Add(30*time.Minute), p)) + res, err = score(rep, lab, "p474", "labels.json", "", nil, nil, injT, injT.Add(24*time.Hour)) + if err != nil { + t.Fatal(err) + } + if res.IncidentsDetected != 1 || res.TruePositives != 1 || res.FalsePositives != 0 { + t.Fatalf("the bot's own key is the detection: %+v", res) + } + // A labels file from before round 2 carries no key; then the channel + // hash is all there is, and the old behaviour stands. + inc.Key = "" + lab.Incidents = []Incident{inc} + rep = r2Report(r2Cand("e1", "active", otherHopKey, injT.Add(30*time.Minute), p)) + res, err = score(rep, lab, "p474", "labels.json", "", nil, nil, injT, injT.Add(24*time.Hour)) + if err != nil { + t.Fatal(err) + } + if res.IncidentsDetected != 1 { + t.Fatalf("without a key, matching falls back to the channel: %+v", res) + } +} + +// F2: a shared key needs a differential reading. An episode that the +// baseline arm (same window, no injection) already has on the same key and +// overlapping in time is the background's: it is neither a detection of the +// bot nor a false alarm of the injected arm. +func TestScoreBaselineDropsEpisodesPresentWithoutInjection(t *testing.T) { + const p = 300 * time.Second + l := r2Labelling(8) + inc := Incident{Bot: "bot-a", Channel: "7f", Key: scoreKey, Start: injT, + Stop: injT.Add(time.Hour), Period: dur(p), Pulses: 8} + lab := InjectLabels{Synthetic: true, Labelling: l, Incidents: []Incident{inc}} + rep := r2Report( + r2Cand("e1", "active", scoreKey, injT.Add(30*time.Minute), p), + r2Cand("e1", "ended", scoreKey, injT.Add(90*time.Minute), 0), + ) + // Without a baseline the episode is the detection. + res, err := score(rep, lab, "p474", "labels.json", "", nil, nil, injT, injT.Add(24*time.Hour)) + if err != nil { + t.Fatal(err) + } + if res.IncidentsDetected != 1 || res.BaselineEpisodes != 0 { + t.Fatalf("no baseline: %+v", res) + } + // The same episode in the baseline arm, on the same key and overlapping: + // the background already had it, so arm B learns nothing from it. + baseRep := r2Report( + r2Cand("a1", "active", scoreKey, injT.Add(20*time.Minute), p), + r2Cand("a1", "ended", scoreKey, injT.Add(100*time.Minute), 0), + ) + base, _, err := episodes(baseRep, "p474", injT, injT.Add(24*time.Hour)) + if err != nil { + t.Fatal(err) + } + res, err = score(rep, lab, "p474", "labels.json", "report-a.json", base, nil, injT, injT.Add(24*time.Hour)) + if err != nil { + t.Fatal(err) + } + if res.IncidentsDetected != 0 || res.Recall != 0 { + t.Fatalf("a pre-existing episode is not a detection: %+v", res) + } + if res.BaselineEpisodes != 1 || res.TruePositives != 0 || res.FalsePositives != 0 { + t.Fatalf("it is counted on its own: %+v", res) + } + if res.Baseline != "report-a.json" { + t.Fatalf("baseline provenance %q", res.Baseline) + } + // A baseline episode on another key, or one that does not overlap, + // leaves the detection alone. + for _, other := range []scoreReport{ + r2Report(r2Cand("a1", "active", otherHopKey, injT.Add(20*time.Minute), p), + r2Cand("a1", "ended", otherHopKey, injT.Add(100*time.Minute), 0)), + r2Report(r2Cand("a1", "active", scoreKey, injT.Add(3*time.Hour), p), + r2Cand("a1", "ended", scoreKey, injT.Add(4*time.Hour), 0)), + } { + b, _, err := episodes(other, "p474", injT, injT.Add(24*time.Hour)) + if err != nil { + t.Fatal(err) + } + res, err = score(rep, lab, "p474", "labels.json", "report-a.json", b, nil, injT, injT.Add(24*time.Hour)) + if err != nil { + t.Fatal(err) + } + if res.IncidentsDetected != 1 || res.BaselineEpisodes != 0 { + t.Fatalf("unrelated baseline episode changed the result: %+v", res) + } + } +} + +// F2: -baseline works end to end from two report files. +func TestScoreBaselineFlagReadsAReportFile(t *testing.T) { + bg := backgroundJSONL(300) + r := injectInto(t, specJSON(specOpts{seed: 21, jitterSD: "3s"}), bg) + bgPath := filepath.Join(r.dir, "bg.jsonl") + if err := os.WriteFile(bgPath, []byte(bg), 0o644); err != nil { + t.Fatal(err) + } + reportB := replayInto(t, r.dir, "b", replayConfigJSON(""), r.outPath) + reportA := replayInto(t, r.dir, "a", replayConfigJSON(""), bgPath) + res := scoreInto(t, reportB, r.labPath, "-baseline", reportA) + // This background raises no episode of its own, so the baseline drops + // nothing and the injected bot is still found. + if res.Baseline != "a-report.json" || res.BaselineEpisodes != 0 || res.IncidentsDetected != 1 { + t.Fatalf("baseline arm: %+v", res) + } + var stdout, stderr bytes.Buffer + code := run([]string{"score", "-report", reportB, "-labels", r.labPath, "-rule", "p474", + "-baseline", filepath.Join(r.dir, "missing.json"), + "-start", injAt(0), "-end", injAt(24 * time.Hour)}, nil, &stdout, &stderr) + if code != exitFail || stdout.Len() != 0 { + t.Fatalf("a missing baseline must fail: exit %d, stdout %q", code, stdout.String()) + } +} + +// F3: five clean bots alarm at the earliest pulse the rule allows and the +// realistic ones take many times longer, so the median sits on the floor and +// hides them. The p75 and the maximum are reported beside every median. +func TestScoreReportsTheTailBesideTheMedian(t *testing.T) { + const p = 300 * time.Second + l := r2Labelling(8) + var incs []Incident + var cands []scoreCandidate + // Four incidents detected at 1 P and four at 10, 20, 30 and 40 P. + ttds := []time.Duration{p, p, p, p, 10 * p, 20 * p, 30 * p, 40 * p} + for i, ttd := range ttds { + start := injT.Add(time.Duration(i) * 12 * time.Hour) + key := fmt.Sprintf("v1|s=stream|pt=5|ch=1:7f|rc=flood|rk=relay|hop=1:%02x", 0xa0+i) + incs = append(incs, Incident{Bot: fmt.Sprintf("b%d", i), Channel: "7f", Key: key, + Start: start, Stop: start.Add(60 * p), Period: dur(p), Pulses: 8}) + cands = append(cands, + r2Cand(fmt.Sprintf("e%d", i), "active", key, start.Add(ttd), p), + r2Cand(fmt.Sprintf("e%d", i), "ended", key, start.Add(ttd+p), 0)) + } + lab := InjectLabels{Synthetic: true, Labelling: l, Incidents: incs} + rep := scoreReport{Experimental: true, Start: injT, End: injT.Add(120 * time.Hour), Candidates: cands} + res, err := score(rep, lab, "p474", "labels.json", "", nil, nil, injT, injT.Add(120*time.Hour)) + if err != nil { + t.Fatal(err) + } + if res.IncidentsDetected != 8 { + t.Fatalf("detected %d of 8: %+v", res.IncidentsDetected, res) + } + // The median of 1, 1, 1, 1, 10, 20, 30, 40 is 5.5 P; the nearest-rank + // p75 of eight values is the sixth, 20 P, and the maximum is 40 P. + if res.MedianTTDPeriods > 6 { + t.Fatalf("median %.1f P should sit low", res.MedianTTDPeriods) + } + if res.P75TTDPeriods != 20 || res.MaxTTDPeriods != 40 { + t.Fatalf("p75 %.1f P, max %.1f P", res.P75TTDPeriods, res.MaxTTDPeriods) + } + if res.P75TimeToDetect == nil || time.Duration(*res.P75TimeToDetect) != 20*p { + t.Fatalf("p75 time to detect %v", res.P75TimeToDetect) + } + if res.MaxTimeToDetect == nil || time.Duration(*res.MaxTimeToDetect) != 40*p { + t.Fatalf("max time to detect %v", res.MaxTimeToDetect) + } + if res.MaxStopLatency == nil || res.P75StopLatency == nil || res.MaxEpisodesPerIncident != 1 { + t.Fatalf("stop latency and episode tail: %+v", res) + } + // A bar on the noisy subset alone is the other half of the fix: -bots + // narrows the panel, and the medians then describe that subset only. + noisy, err := score(rep, lab, "p474", "labels.json", "", nil, []string{"b4", "b5", "b6", "b7"}, + injT, injT.Add(120*time.Hour)) + if err != nil { + t.Fatal(err) + } + if noisy.IncidentsSelected != 4 || noisy.MedianTTDPeriods != 25 || noisy.MaxTTDPeriods != 40 { + t.Fatalf("noisy subset: selected %d, median %.1f P, max %.1f P", + noisy.IncidentsSelected, noisy.MedianTTDPeriods, noisy.MaxTTDPeriods) + } +} + +// F8: the metrics record the file names of the labels and the baseline, not +// the paths they were read from. A path can carry a user or a host name, and +// a metrics file may be shared. +func TestScoreProvenanceIsFileNamesOnly(t *testing.T) { + r := injectInto(t, specJSON(specOpts{seed: 21, jitterSD: "3s"}), backgroundJSONL(300)) + res := scoreInto(t, replayInto(t, r.dir, "b", replayConfigJSON(""), r.outPath), r.labPath) + if res.Labels != "labels.json" { + t.Fatalf("labels provenance %q must be a bare file name", res.Labels) + } + if strings.ContainsRune(res.Labels, filepath.Separator) || strings.Contains(res.Labels, r.dir) { + t.Fatalf("labels provenance %q leaks a path", res.Labels) + } + // No labels, no provenance. + none := scoreInto(t, replayInto(t, r.dir, "n", replayConfigJSON(""), r.outPath), "") + if none.Labels != "" || none.Baseline != "" { + t.Fatalf("provenance without files: %q / %q", none.Labels, none.Baseline) + } +} diff --git a/cmd/anomaly/review_r3_test.go b/cmd/anomaly/review_r3_test.go new file mode 100644 index 000000000..6ed67f882 --- /dev/null +++ b/cmd/anomaly/review_r3_test.go @@ -0,0 +1,728 @@ +package main + +import ( + "bytes" + "encoding/json" + "fmt" + "os" + "os/exec" + "path/filepath" + "strings" + "testing" + "time" + + "github.com/meshcore-analyzer/anomaly" +) + +// Round-3 review of PR #478 (#474). The round-2 panel could not be injected +// at all on a realistic key: make_spec.py held wire_bytes at the key's +// median while letting the payload size spread, and the event validator +// requires payload <= wire - 2, so the first oversized draw aborted the run +// (G1). The fix draws each event's (wire, payload) pair from the key's own +// observed distribution (G2), which is valid by construction and leaves no +// size an injected line can be recognised by. The rest of this file pins the +// boundaries round 2's mutants survived (G4), the timing statistic that now +// penalises a missed bot (G3) and the per-incident baseline flag (G5). +// +// Every channel byte, hop, size and time here is synthetic. + +// r3Rec is one background line with explicit sizes. +func r3Rec(id string, at time.Time, channel, hop, route string, wire int, payload *int) anomaly.ReplayRecord { + return anomaly.ReplayRecord{ID: id, Time: at, FinalAt: at.Add(30 * time.Second), PayloadType: 5, + Channel: channel, RouteClass: "flood", RouteKind: "relay", FirstHop: hop, FullRoute: route, + WireBytes: wire, PayloadBytes: payload, Traffic: "unclassified", Decryption: "unknown"} +} + +func r3Line(t *testing.T, rec anomaly.ReplayRecord) string { + t.Helper() + b, err := json.Marshal(rec) + if err != nil { + t.Fatal(err) + } + return string(b) + "\n" +} + +// r3Sized writes one line per payload size on one series, with the wire size +// a fixed number of bytes above the payload (header plus path, as a real +// frame has). Times step by a minute, so the lines are in order. +func r3Sized(t *testing.T, channel, hop, route string, payloads []int, headroom int) string { + t.Helper() + var b strings.Builder + for i, pb := range payloads { + pb := pb + b.WriteString(r3Line(t, r3Rec(fmt.Sprintf("tx%d", i+1), injT.Add(time.Duration(i)*time.Minute), + channel, hop, route, pb+headroom, &pb))) + } + return b.String() +} + +// G2/G1: the ranking carries the key's whole observed (wire, payload) +// distribution, with a count per distinct pair. That distribution is what an +// injected bot draws its sizes from, so it has to be in the output, exact, +// and in a fixed order. +func TestKeysReportsTheObservedSizeDistribution(t *testing.T) { + var b strings.Builder + // 35 bytes four times, 51 twice, 67 once, each with a 3-byte frame + // overhead. Two lines share the 38-byte frame with a shorter payload and + // with a payload the export did not know, which is what fixes the order + // inside one wire size. + add := func(i int, wire int, payload *int) { + b.WriteString(r3Line(t, r3Rec(fmt.Sprintf("tx%d", i+1), injT.Add(time.Duration(i)*time.Minute), + "0b", "1:5c", "1:5c0a", wire, payload))) + } + for i, pb := range []int{35, 35, 35, 35, 51, 51, 67} { + pb := pb + add(i, pb+3, &pb) + } + add(7, 40, nil) + add(8, 38, nil) + add(9, 38, intp(30)) + res := keysOn(t, b.String()) + if len(res.Top) != 1 { + t.Fatalf("one key expected, got %d", len(res.Top)) + } + got := res.Top[0].SizeMix + // Ordered by wire size, then by payload size, with the unknown payload + // first inside one wire size. + want := []sizePair{{WireBytes: 38, Count: 1}, {WireBytes: 38, PayloadBytes: intp(30), Count: 1}, + {WireBytes: 38, PayloadBytes: intp(35), Count: 4}, {WireBytes: 40, Count: 1}, + {WireBytes: 54, PayloadBytes: intp(51), Count: 2}, {WireBytes: 70, PayloadBytes: intp(67), Count: 1}} + if len(got) != len(want) { + t.Fatalf("size mix %+v", got) + } + for i := range want { + if got[i].WireBytes != want[i].WireBytes || got[i].Count != want[i].Count || + (got[i].PayloadBytes == nil) != (want[i].PayloadBytes == nil) || + (want[i].PayloadBytes != nil && *got[i].PayloadBytes != *want[i].PayloadBytes) { + t.Fatalf("size mix entry %d: %+v, want %+v", i, got[i], want[i]) + } + } + if res.Top[0].Events != 10 { + t.Fatalf("events %d", res.Top[0].Events) + } +} + +// r3SizeMix renders a size mix as the JSON an injection spec carries. +func r3SizeMix(pairs ...sizePair) string { + b, err := json.Marshal(pairs) + if err != nil { + panic(err) + } + return string(b) +} + +// r3MixSpec is one bot described by a size mix instead of by scalar sizes. +func r3MixSpec(seed uint64, mix string, stopAfter time.Duration) string { + return fmt.Sprintf(`{ + "seed": %d, + "final_after": "30s", + "labelling": {"pulse_merge_gap": "60s", "min_period": "1m", "max_period": "1h", + "min_pulses": 8, "max_harmonic": 3, "tolerance_abs": "30s", "tolerance_rel": 0.1}, + "bots": [{ + "name": "bot-a", "channel": "7f", "payload_type": 5, "route_class": "flood", "route_kind": "relay", + "first_hop": "1:fd", "full_route": "1:fd0a", "size_mix": %s, + "start": %q, "stop": %q, "period": "5m", "jitter_sd": "3s", "jitter_max": "9s", + "missed_fraction": 0, "multipart_fraction": 0, "multipart_max_extra": 2, + "multipart_max_delay": "55s", "holes": [] + }] +}`, seed, mix, injAt(0), injAt(stopAfter)) +} + +// G1/G2: a bot that draws from the key's observed pairs emits only sizes the +// key really carried, in roughly the observed proportions, and every line is +// a valid event - including on a block-quantised distribution whose spread +// is far wider than the frame's own margin, which is what aborted round 2. +func TestInjectSizeMixDrawsOnlyObservedPairs(t *testing.T) { + mix := r3SizeMix(sizePair{WireBytes: 38, PayloadBytes: intp(35), Count: 70}, + sizePair{WireBytes: 54, PayloadBytes: intp(51), Count: 20}, + sizePair{WireBytes: 70, PayloadBytes: intp(67), Count: 10}, + sizePair{WireBytes: 40, Count: 10}) + r := injectInto(t, r3MixSpec(5, mix, 24*time.Hour), backgroundJSONL(40)) + seen := map[string]int{} + lines := injectedLines(t, r) + if len(lines) < 200 { + t.Fatalf("only %d injected lines", len(lines)) + } + for _, rec := range lines { + pb := "none" + if rec.PayloadBytes != nil { + pb = fmt.Sprint(*rec.PayloadBytes) + } + key := fmt.Sprintf("%d/%s", rec.WireBytes, pb) + switch key { + case "38/35", "54/51", "70/67", "40/none": + default: + t.Fatalf("%s carries the unobserved size pair %s", rec.ID, key) + } + seen[key]++ + } + if len(seen) != 4 { + t.Fatalf("only %d of the four observed pairs were drawn: %v", len(seen), seen) + } + // The commonest pair is 70 % of the weight and must dominate; the three + // others are 10-20 % each and must stay well below it. + if seen["38/35"] <= 2*seen["54/51"] || seen["54/51"] <= seen["70/67"] { + t.Fatalf("draw does not follow the observed weights: %v", seen) + } +} + +// G1: a size mix is the whole size description. Mixing it with the scalar +// size fields would leave two sources of truth, and an entry the frame +// cannot hold must be refused by the spec rather than abort the run halfway +// through, which is what round 2 did. +func TestInjectSizeMixIsValidatedBeforeAnythingRuns(t *testing.T) { + for _, tc := range []struct{ name, mix, extra string }{ + {"a pair the frame cannot hold", + r3SizeMix(sizePair{WireBytes: 38, PayloadBytes: intp(37), Count: 1}), ""}, + {"a zero count", r3SizeMix(sizePair{WireBytes: 38, PayloadBytes: intp(35), Count: 0}), ""}, + {"a negative count", r3SizeMix(sizePair{WireBytes: 38, PayloadBytes: intp(35), Count: -1}), ""}, + {"no wire size", r3SizeMix(sizePair{PayloadBytes: intp(35), Count: 1}), ""}, + {"an empty mix", "[]", ""}, + {"beside wire_bytes", r3SizeMix(sizePair{WireBytes: 38, PayloadBytes: intp(35), Count: 1}), + `"wire_bytes": 60,`}, + {"beside payload_bytes", r3SizeMix(sizePair{WireBytes: 38, PayloadBytes: intp(35), Count: 1}), + `"payload_bytes": 35, "payload_bytes_spread": 0,`}, + } { + spec := strings.Replace(r3MixSpec(5, tc.mix, time.Hour), `"size_mix":`, tc.extra+` "size_mix":`, 1) + if _, err := parseInjectSpec([]byte(spec)); err == nil { + t.Fatalf("%s: want an error", tc.name) + } + } + ok := r3MixSpec(5, r3SizeMix(sizePair{WireBytes: 38, PayloadBytes: intp(36), Count: 1}), time.Hour) + if _, err := parseInjectSpec([]byte(ok)); err != nil { + t.Fatalf("a payload exactly wire-2 long is legal: %v", err) + } +} + +// G1: the scalar path keeps its envelope inside the frame too. payload_bytes +// +/- payload_bytes_spread is the whole range a draw can reach, so the top of +// that range - not just payload_bytes - has to fit wire_bytes - 2. Round 2 +// checked the spread against the payload only, so a legal-looking spec +// aborted on its first wide draw. +func TestInjectSpreadEnvelopeMustFitTheFrame(t *testing.T) { + spec := func(payload, spread, wire int) string { + return fmt.Sprintf(`{"seed":7,"final_after":"30s", + "labelling":{"pulse_merge_gap":"60s","min_period":"1m","max_period":"1h","min_pulses":8, + "max_harmonic":3,"tolerance_abs":"30s","tolerance_rel":0.1}, + "bots":[{"name":"bot-a","channel":"7f","payload_type":5,"route_class":"flood","route_kind":"relay", + "first_hop":"1:fd","full_route":"1:fd0a","wire_bytes":%d,"payload_bytes":%d,"payload_bytes_spread":%d, + "start":%q,"stop":%q,"period":"5m","jitter_sd":"0s", + "jitter_max":"10s","missed_fraction":0,"multipart_fraction":0,"multipart_max_extra":2, + "multipart_max_delay":"55s","holes":[]}]}`, wire, payload, spread, injAt(0), injAt(4*time.Hour)) + } + // The real shape of the abort: a key whose median wire is 47 and whose + // payload spread is 24, as the fixture's rank-1 key is. The diagnostic + // names the two fields, because round 2's failure mode was an opaque + // validator error thousands of lines into a run. + _, err := parseInjectSpec([]byte(spec(35, 24, 47))) + if err == nil { + t.Fatal("a spread that can overflow the frame must be refused by the spec") + } + if !strings.Contains(err.Error(), "payload_bytes + payload_bytes_spread") { + t.Fatalf("the error must name the envelope: %v", err) + } + // One byte inside the margin is legal, one byte outside is not. + if _, err := parseInjectSpec([]byte(spec(35, 23, 60))); err != nil { + t.Fatalf("payload_bytes + spread == wire_bytes - 2 is legal: %v", err) + } + if _, err := parseInjectSpec([]byte(spec(35, 24, 60))); err == nil { + t.Fatal("payload_bytes + spread one past wire_bytes - 2 must be refused") + } + // The envelope must clear every bound the event validator has, not only + // the frame: a payload of 180 +/- 10 fits a 255-byte frame but its top + // passes MaxPayloadBytes (184). + if _, err := parseInjectSpec([]byte(spec(180, 10, 255))); err == nil { + t.Fatal("a spread whose top passes MaxPayloadBytes must be refused") + } + if _, err := parseInjectSpec([]byte(spec(180, 4, 255))); err != nil { + t.Fatalf("a spread whose top is exactly MaxPayloadBytes is legal: %v", err) + } + // And no run of a legal spec may ever produce an invalid line. + r := injectInto(t, spec(35, 23, 60), backgroundJSONL(20)) + for _, rec := range injectedLines(t, r) { + if _, err := rec.Event(); err != nil { + t.Fatalf("%s: %v", rec.ID, err) + } + } +} + +// r3BlockBackground is a background shaped like real GRP_TXT traffic: the +// payload comes in 16-byte cipher blocks and the frame adds only a header +// and a one-hop path, so wire - payload is 3 on every line and the size +// spread is many times the frame's own margin. nKeys series, the i-th +// carrying one line more than the next, so the ranking is unambiguous. +func r3BlockBackground(t *testing.T, nKeys, perKey int) string { + t.Helper() + var b strings.Builder + at := injT + n := 0 + for i := 0; i < nKeys; i++ { + hop := fmt.Sprintf("1:%02x", 0x20+i) + for j := 0; j < perKey+(nKeys-i); j++ { + pb := []int{35, 51, 67, 35, 83, 35, 51}[j%7] + at = at.Add(41 * time.Second) + n++ + b.WriteString(r3Line(t, r3Rec(fmt.Sprintf("tx%d", n), at, "0b", hop, hop+"0a", pb+3, &pb))) + } + } + return b.String() +} + +// r3MakeSpec runs the pre-registered make_spec.py over a keys ranking and +// returns the spec it printed. The script is the pre-registration's own, so +// the test exercises what the round will actually run. +func r3MakeSpec(t *testing.T, keysJSON []byte) []byte { + t.Helper() + py, err := exec.LookPath("python3") + if err != nil { + t.Skip("python3 is not installed; make_spec.py cannot be exercised here") + } + dir := t.TempDir() + kp := filepath.Join(dir, "keys.json") + if err := os.WriteFile(kp, keysJSON, 0o644); err != nil { + t.Fatal(err) + } + cmd := exec.Command(py, "make_spec.py", injT.UTC().Format("2006-01-02T15:04:05Z"), kp) + var out, errb bytes.Buffer + cmd.Stdout, cmd.Stderr = &out, &errb + if err := cmd.Run(); err != nil { + t.Fatalf("make_spec.py: %v: %s", err, errb.String()) + } + return out.Bytes() +} + +// G1: the pre-registered pipeline has to run on realistic sizes. keys -> +// make_spec.py -> inject, on a background whose payloads come in 16-byte +// blocks, is exactly the step that aborted in round 2. +func TestPipelineKeysSpecInjectOnBlockQuantisedSizes(t *testing.T) { + bg := r3BlockBackground(t, 10, 40) + dir := t.TempDir() + bgPath := filepath.Join(dir, "bg.jsonl") + if err := os.WriteFile(bgPath, []byte(bg), 0o644); err != nil { + t.Fatal(err) + } + var stdout, stderr bytes.Buffer + if code := run([]string{"keys", "-input", bgPath, "-payload-type", "5", "-top", "16", + "-max-keys", "5000", "-max-line-bytes", "4096"}, nil, &stdout, &stderr); code != 0 { + t.Fatalf("keys exit %d: %s", code, stderr.String()) + } + spec := r3MakeSpec(t, stdout.Bytes()) + specPath, outPath, labPath := filepath.Join(dir, "spec.json"), filepath.Join(dir, "inj.jsonl"), filepath.Join(dir, "labels.json") + if err := os.WriteFile(specPath, spec, 0o644); err != nil { + t.Fatal(err) + } + var istdout, istderr bytes.Buffer + code := run([]string{"inject", "-spec", specPath, "-input", bgPath, "-out", outPath, + "-labels", labPath, "-max-injected", "2000000", "-max-line-bytes", "65536"}, nil, &istdout, &istderr) + if code != 0 { + t.Fatalf("inject exit %d: %s", code, istderr.String()) + } + // Every injected line must be a valid event, and no injected size pair + // may be one the background never carried. + bgPairs := map[string]bool{} + for _, l := range strings.Split(strings.TrimRight(bg, "\n"), "\n") { + var rec anomaly.ReplayRecord + if err := json.Unmarshal([]byte(l), &rec); err != nil { + t.Fatal(err) + } + bgPairs[r3Pair(rec)] = true + } + out, err := os.ReadFile(outPath) + if err != nil { + t.Fatal(err) + } + injected := 0 + for _, l := range strings.Split(strings.TrimRight(string(out), "\n"), "\n") { + var rec anomaly.ReplayRecord + if err := json.Unmarshal([]byte(l), &rec); err != nil { + t.Fatal(err) + } + if !strings.HasPrefix(rec.ID, injPrefix) { + continue + } + injected++ + if _, err := rec.Event(); err != nil { + t.Fatalf("%s: %v", rec.ID, err) + } + if !bgPairs[r3Pair(rec)] { + t.Fatalf("%s carries size pair %s, which the background never had", rec.ID, r3Pair(rec)) + } + } + if injected < 1000 { + t.Fatalf("only %d injected lines", injected) + } +} + +func r3Pair(rec anomaly.ReplayRecord) string { + if rec.PayloadBytes == nil { + return fmt.Sprintf("%d/none", rec.WireBytes) + } + return fmt.Sprintf("%d/%d", rec.WireBytes, *rec.PayloadBytes) +} + +// G1: the whole pre-registered chain, from a database. export -> keys -> +// make_spec.py -> inject on the package's own synthetic CoreScope fixture, +// whose GRP_TXT payloads come in real 16-byte cipher blocks over a one-hop +// path - the size structure that made round 2 abort. +func TestPipelineExportKeysSpecInjectOnTheFixture(t *testing.T) { + f := newFixtureDB(t, false) + // Ten route groups, each a different first hop on one channel byte, with + // 12 to 21 transmissions apiece so the ranking is strict. Payload sizes + // are 3 + 16k bytes, as a GRP_TXT payload is. + n := 0 + for i := 0; i < 10; i++ { + hop := byte(0x20 + i) + for j := 0; j < 12+(10-i); j++ { + n++ + blocks := []int{32, 48, 64, 32, 80, 32, 48}[j%7] + pl := grpPayload(0x0a, blocks) + at := fxAt(time.Duration(n) * 37 * time.Second) + f.tx(fxHash(n), frameHex(1, 5, 1, []byte{hop}, pl), 5, + relay(at, 1, 5, 1, []byte{hop}, pl)) + } + } + dir := t.TempDir() + bgPath := filepath.Join(dir, "bg.jsonl") + var stdout, stderr bytes.Buffer + if code := run([]string{"export", "-db", f.path, "-start", fxT.UTC().Format(time.RFC3339), + "-end", fxAt(24 * time.Hour).UTC().Format(time.RFC3339), "-settle", "30s", + "-max-rows", "100000", "-out", bgPath}, nil, &stdout, &stderr); code != 0 { + t.Fatalf("export exit %d: %s", code, stderr.String()) + } + stdout.Reset() + stderr.Reset() + if code := run([]string{"keys", "-input", bgPath, "-payload-type", "5", "-top", "16", + "-max-keys", "5000000", "-max-line-bytes", "4096"}, nil, &stdout, &stderr); code != 0 { + t.Fatalf("keys exit %d: %s", code, stderr.String()) + } + var ranking keysResult + if err := json.Unmarshal(stdout.Bytes(), &ranking); err != nil { + t.Fatal(err) + } + if len(ranking.Top) < 9 { + t.Fatalf("the fixture must offer at least the panel's nine keys, got %d", len(ranking.Top)) + } + // The size structure that broke round 2 must really be there: the key's + // payload spread has to exceed the frame's own margin. + k := ranking.Top[0] + if k.PayloadBytesSpread <= k.WireBytesMedian-2-k.PayloadBytesMedian { + t.Fatalf("rank 1 is not block-quantised: wire %d, payload %d +/- %d", + k.WireBytesMedian, k.PayloadBytesMedian, k.PayloadBytesSpread) + } + spec := r3MakeSpec(t, stdout.Bytes()) + specPath := filepath.Join(dir, "spec.json") + if err := os.WriteFile(specPath, spec, 0o644); err != nil { + t.Fatal(err) + } + var istdout, istderr bytes.Buffer + code := run([]string{"inject", "-spec", specPath, "-input", bgPath, + "-out", filepath.Join(dir, "inj.jsonl"), "-labels", filepath.Join(dir, "labels.json"), + "-max-injected", "2000000", "-max-line-bytes", "65536"}, nil, &istdout, &istderr) + if code != 0 { + t.Fatalf("inject exit %d: %s", code, istderr.String()) + } + var sum injectSummary + if err := json.Unmarshal(istderr.Bytes(), &sum); err != nil { + t.Fatalf("summary %q: %v", istderr.String(), err) + } + if sum.InjectedEvents < 1000 || len(sum.Bots) != 18 { + t.Fatalf("summary: %d events, %d bots", sum.InjectedEvents, len(sum.Bots)) + } +} + +// G4 (mutant N3): the tie-break is part of the mechanical assignment rule, +// so it has to be pinned. Two keys with the same row count rank by the +// smaller canonical encoding first; reversing that order silently moves +// every panel position. +func TestKeysTieBreakIsTheSmallerCanonicalKey(t *testing.T) { + res := keysOn(t, r2Background([]r2Series{ + {channel: "0b", firstHop: "1:11", fullRoute: "1:11", weight: 3, payload: 30}, + {channel: "0b", firstHop: "1:22", fullRoute: "1:22", weight: 3, payload: 30}, + {channel: "0b", firstHop: "1:33", fullRoute: "1:33", weight: 3, payload: 30}, + }, 4)) + if len(res.Top) != 3 { + t.Fatalf("three tied keys expected, got %d", len(res.Top)) + } + for i := 1; i < len(res.Top); i++ { + if res.Top[i-1].Events != res.Top[i].Events { + t.Fatalf("the counts must tie: %+v", res.Top) + } + if !(res.Top[i-1].Key < res.Top[i].Key) { + t.Fatalf("tie %d: %q must sort before %q", i, res.Top[i-1].Key, res.Top[i].Key) + } + } + if res.Top[0].FirstHop != "1:11" || res.Top[2].FirstHop != "1:33" { + t.Fatalf("tie order: %q .. %q", res.Top[0].FirstHop, res.Top[2].FirstHop) + } +} + +// G4 (mutant N11): -top reports exactly that many keys. The full ranking +// (-top large) is what the pre-registration's absence check for the isolated +// keys reads, so an off-by-one there silently drops a key from the check. +func TestKeysTopReportsExactlyTopKeys(t *testing.T) { + bg := r3BlockBackground(t, 6, 10) + for _, tc := range []struct{ top, want int }{{1, 1}, {3, 3}, {6, 6}, {9, 6}} { + res := keysOn(t, bg, "-top", fmt.Sprint(tc.top)) + if len(res.Top) != tc.want { + t.Fatalf("-top %d reported %d keys, want %d", tc.top, len(res.Top), tc.want) + } + if res.DistinctKeys != 6 { + t.Fatalf("-top %d: distinct keys %d", tc.top, res.DistinctKeys) + } + for i, k := range res.Top { + if k.Rank != i+1 { + t.Fatalf("-top %d: entry %d has rank %d", tc.top, i, k.Rank) + } + } + } +} + +// G4 (mutant N6): the percentiles are nearest-rank over the value +// histogram - the value at index ceil(q*n)-1 - and the boundary of that +// scan is off by one count if the comparison is not strict. Ten distinct +// sizes, one line each, make every percentile a different value. +func TestKeysPercentilesAreNearestRank(t *testing.T) { + payloads := []int{10, 20, 30, 40, 50, 60, 70, 80, 90, 100} + res := keysOn(t, r3Sized(t, "0b", "1:5c", "1:5c0a", payloads, 5)) + if len(res.Top) != 1 { + t.Fatalf("one key expected, got %d", len(res.Top)) + } + k := res.Top[0] + // n = 10: the median is index 4 (50), p10 index 0 (10) and p90 index 8 + // (90), so the spread is (90 - 10) / 2 = 40. A non-strict scan returns + // the value one count earlier: median 40 and p90 80, spread 35. + if k.PayloadBytesMedian != 50 { + t.Fatalf("payload median %d, want 50", k.PayloadBytesMedian) + } + if k.PayloadBytesSpread != 40 { + t.Fatalf("payload spread %d, want 40", k.PayloadBytesSpread) + } + if k.WireBytesMedian != 55 { + t.Fatalf("wire median %d, want 55", k.WireBytesMedian) + } +} + +// G4 (mutant N4): the spread is clamped strictly below the median, because +// payload_bytes +/- spread must stay positive. A raw spread exactly equal to +// the median is the boundary, and only a clamp at equality catches it. +func TestKeysSpreadIsClampedStrictlyBelowTheMedian(t *testing.T) { + // Ten sizes, one line each: p10 = 1, median = 16, p90 = 33, so the raw + // spread is (33 - 1) / 2 = 16, exactly the median. + res := keysOn(t, r3Sized(t, "0b", "1:5c", "1:5c0a", []int{1, 2, 3, 4, 16, 20, 24, 28, 33, 40}, 5)) + k := res.Top[0] + if k.PayloadBytesMedian != 16 { + t.Fatalf("payload median %d, want 16", k.PayloadBytesMedian) + } + if k.PayloadBytesSpread != 15 { + t.Fatalf("spread %d, want the median minus one (15)", k.PayloadBytesSpread) + } +} + +// G4 (mutant N7): first and last are the key's earliest and latest line. The +// export is ordered, but a merged or concatenated background need not be, and +// a first that tracks the maximum reports a window the key never had. +func TestKeysFirstAndLastAreTheExtremeTimes(t *testing.T) { + var b strings.Builder + pb := 35 + for i, d := range []time.Duration{5 * time.Hour, time.Hour, 9 * time.Hour, 3 * time.Hour} { + b.WriteString(r3Line(t, r3Rec(fmt.Sprintf("tx%d", i+1), injT.Add(d), "0b", "1:5c", "1:5c0a", 40, &pb))) + } + k := keysOn(t, b.String()).Top[0] + if !k.First.Equal(injT.Add(time.Hour)) { + t.Fatalf("first %s, want %s", k.First, injT.Add(time.Hour)) + } + if !k.Last.Equal(injT.Add(9 * time.Hour)) { + t.Fatalf("last %s, want %s", k.Last, injT.Add(9*time.Hour)) + } +} + +// G4 (mutant N8): an injected time is truncated, not rounded. The doc says +// truncation moves a time by less than one second towards the past, and the +// effective jitter window stated in the code depends on it; rounding would +// move half the events into the future instead. +func TestInjectTruncatesTowardsThePast(t *testing.T) { + // A start three quarters of a second past the second, with no jitter: + // truncation gives the whole second below it, rounding the one above. + start := injT.Add(750 * time.Millisecond) + spec := fmt.Sprintf(`{"seed":7,"final_after":"30s", + "labelling":{"pulse_merge_gap":"60s","min_period":"1m","max_period":"1h","min_pulses":8, + "max_harmonic":3,"tolerance_abs":"30s","tolerance_rel":0.1}, + "bots":[{"name":"bot-a","channel":"7f","payload_type":5,"route_class":"flood","route_kind":"relay", + "first_hop":"1:fd","full_route":"1:fd0a","wire_bytes":60,"payload_bytes":35,"payload_bytes_spread":0, + "start":%q,"stop":%q,"period":"5m","jitter_sd":"0s", + "jitter_max":"10s","missed_fraction":0,"multipart_fraction":0,"multipart_max_extra":2, + "multipart_max_delay":"55s","holes":[]}]}`, + start.UTC().Format(time.RFC3339Nano), injT.Add(time.Hour).UTC().Format(time.RFC3339Nano)) + r := injectInto(t, spec, backgroundJSONL(10)) + lines := injectedLines(t, r) + if len(lines) == 0 { + t.Fatal("no injected lines") + } + for _, rec := range lines { + if rec.Time.Nanosecond() != 0 { + t.Fatalf("%s is not a whole second: %s", rec.ID, rec.Time) + } + } + if !lines[0].Time.Equal(injT) { + t.Fatalf("first event at %s, want %s (truncated towards the past)", lines[0].Time, injT) + } + if !lines[0].Time.Before(start) { + t.Fatalf("truncation must move the time towards the past: %s is not before %s", lines[0].Time, start) + } +} + +// r3Episode is one arm's episode, written out in full. +func r3Episode(key string, act, end time.Duration, period time.Duration) episode { + return episode{channel: "7f", key: key, activation: injT.Add(act), end: injT.Add(end), + hasPeriod: period > 0, period: period} +} + +// G4 (mutants N1/N2): the baseline overlap is closed at both ends. An +// episode of the background that starts exactly when the injected arm's +// episode ends, or ends exactly when it starts, still overlaps it: a +// half-open test would call such an episode the injection's doing. +func TestScoreBaselineOverlapIsClosedAtBothEnds(t *testing.T) { + const p = 300 * time.Second + for _, tc := range []struct { + name string + baseAct, baseEnd time.Duration + }{ + {"baseline starts exactly when the episode ends", 2 * time.Hour, 3 * time.Hour}, + {"baseline ends exactly when the episode starts", 0, time.Hour}, + } { + eps := []episode{r3Episode(scoreKey, time.Hour, 2*time.Hour, p)} + markBaseline(eps, []episode{r3Episode(scoreKey, tc.baseAct, tc.baseEnd, p)}) + if !eps[0].baseline { + t.Fatalf("%s: a touching baseline episode must still mark it", tc.name) + } + } + // One nanosecond clear of the episode at either end is not an overlap. + for _, tc := range []struct { + name string + baseAct, baseEnd time.Duration + }{ + {"after", 2*time.Hour + 1, 3 * time.Hour}, + {"before", 0, time.Hour - 1}, + } { + eps := []episode{r3Episode(scoreKey, time.Hour, 2*time.Hour, p)} + markBaseline(eps, []episode{r3Episode(scoreKey, tc.baseAct, tc.baseEnd, p)}) + if eps[0].baseline { + t.Fatalf("%s: a baseline episode clear of the window must not mark it", tc.name) + } + } +} + +// r3Panel builds n incidents on n keys, with a detection at the given time +// to detect per incident; a negative time to detect means the incident is +// missed (no episode at all). +func r3Panel(p time.Duration, ttds []float64) (InjectLabels, scoreReport) { + var incs []Incident + var cands []scoreCandidate + for i, mult := range ttds { + start := injT.Add(time.Duration(i) * 12 * time.Hour) + key := fmt.Sprintf("v1|s=stream|pt=5|ch=1:7f|rc=flood|rk=relay|hop=1:%02x", 0xa0+i) + incs = append(incs, Incident{Bot: fmt.Sprintf("b%d", i), Channel: "7f", Key: key, + Start: start, Stop: start.Add(60 * p), Period: dur(p), Pulses: 8}) + if mult < 0 { + continue + } + at := start.Add(time.Duration(mult * float64(p))) + cands = append(cands, r2Cand(fmt.Sprintf("e%d", i), "active", key, at, p), + r2Cand(fmt.Sprintf("e%d", i), "ended", key, at.Add(p), 0)) + } + return InjectLabels{Synthetic: true, Labelling: r2Labelling(8), Incidents: incs}, + scoreReport{Experimental: true, Start: injT, End: injT.Add(240 * time.Hour), Candidates: cands} +} + +// G3: a timing bar over the detected incidents only cannot fail on a missed +// bot. Four clean shapes detected at the floor and four realistic ones missed +// gives recall 0.50 and a p75 of 1 P over the detected subset - a pass of +// every timing bar round 2 had. The selected tail ranks a miss above every +// finite value, so the same panel's p75 is infinite and the bar fails. +func TestScoreSelectedTailCountsAMissAsInfinite(t *testing.T) { + const p = 300 * time.Second + lab, rep := r3Panel(p, []float64{1, 1, 1, 1, -1, -1, -1, -1}) + res, err := score(rep, lab, "p474", "labels.json", "", nil, nil, injT, injT.Add(240*time.Hour)) + if err != nil { + t.Fatal(err) + } + if res.IncidentsSelected != 8 || res.IncidentsDetected != 4 || res.IncidentsMissed != 4 { + t.Fatalf("selected %d, detected %d, missed %d", res.IncidentsSelected, res.IncidentsDetected, res.IncidentsMissed) + } + if res.P75TTDPeriods != 1 { + t.Fatalf("the detected-only p75 is %.1f P; the statistic round 2 had cannot fail", res.P75TTDPeriods) + } + // n = 8 selected: the nearest-rank p75 is the sixth value, and values + // five to eight are misses, so the p75 is infinite. + if !res.P75TTDSelectedInfinite || res.P75TTDSelected != nil || res.P75TTDPeriodsSelected != 0 { + t.Fatalf("p75 over the selected incidents must be infinite: %+v", res) + } + // The median is the fourth value, which is still a detection. + if res.MedianTTDSelectedInfinite || res.MedianTTDSelected == nil || + time.Duration(*res.MedianTTDSelected) != p || res.MedianTTDPeriodsSelected != 1 { + t.Fatalf("median over the selected incidents: %+v", res) + } + // With every incident detected the selected tail is finite and equals + // the detected-only tail. + fullLab, fullRep := r3Panel(p, []float64{1, 1, 1, 1, 10, 20, 30, 40}) + full, err := score(fullRep, fullLab, "p474", "labels.json", "", nil, nil, injT, injT.Add(240*time.Hour)) + if err != nil { + t.Fatal(err) + } + if full.IncidentsMissed != 0 || full.P75TTDSelectedInfinite || full.MedianTTDSelectedInfinite { + t.Fatalf("a complete panel has a finite tail: %+v", full) + } + if full.P75TTDPeriodsSelected != full.P75TTDPeriods || full.MedianTTDPeriodsSelected != 1 { + t.Fatalf("selected %.1f P vs detected %.1f P", full.P75TTDPeriodsSelected, full.P75TTDPeriods) + } + // A miss outside the -bots selection must not enter the statistic. + sub, err := score(rep, lab, "p474", "labels.json", "", nil, []string{"b0", "b1", "b2", "b3"}, + injT, injT.Add(240*time.Hour)) + if err != nil { + t.Fatal(err) + } + if sub.IncidentsMissed != 0 || sub.P75TTDSelectedInfinite || sub.Recall != 1 { + t.Fatalf("-bots subset: missed %d, infinite %v, recall %.2f", + sub.IncidentsMissed, sub.P75TTDSelectedInfinite, sub.Recall) + } +} + +// G5: when the only episode of an incident's own series is absorbed by the +// baseline arm, the per-incident row reads as a plain miss. A no-go caused by +// background contamination has to be distinguishable from a detector miss, so +// the row carries the count and a flag. +func TestScoreReportsBaselineMaskingPerIncident(t *testing.T) { + const p = 300 * time.Second + l := r2Labelling(8) + key := scoreKey + other := "v1|s=stream|pt=5|ch=1:7f|rc=flood|rk=relay|hop=1:ee" + lab := InjectLabels{Synthetic: true, Labelling: l, Incidents: []Incident{ + {Bot: "masked", Channel: "7f", Key: key, Start: injT, Stop: injT.Add(60 * p), Period: dur(p), Pulses: 8}, + {Bot: "plain-miss", Channel: "7f", Key: other, Start: injT, Stop: injT.Add(60 * p), Period: dur(p), Pulses: 8}, + }} + rep := scoreReport{Experimental: true, Start: injT, End: injT.Add(48 * time.Hour), Candidates: []scoreCandidate{ + r2Cand("e1", "active", key, injT.Add(10*time.Minute), p), + r2Cand("e1", "ended", key, injT.Add(2*time.Hour), 0), + }} + base := []episode{r3Episode(key, 0, 24*time.Hour, p)} + res, err := score(rep, lab, "p474", "labels.json", "report-a.json", base, nil, injT, injT.Add(48*time.Hour)) + if err != nil { + t.Fatal(err) + } + if res.BaselineEpisodes != 1 || res.IncidentsDetected != 0 { + t.Fatalf("baseline episodes %d, detected %d", res.BaselineEpisodes, res.IncidentsDetected) + } + if res.IncidentsBaselineMasked != 1 { + t.Fatalf("incidents_baseline_masked %d, want 1", res.IncidentsBaselineMasked) + } + byBot := map[string]incidentScore{} + for _, is := range res.PerIncident { + byBot[is.Bot] = is + } + if m := byBot["masked"]; !m.BaselineMasked || m.BaselineEpisodes != 1 || m.Detected { + t.Fatalf("the masked incident's row: %+v", m) + } + if pm := byBot["plain-miss"]; pm.BaselineMasked || pm.BaselineEpisodes != 0 || pm.Detected { + t.Fatalf("a plain miss must not be flagged as masked: %+v", pm) + } +} diff --git a/cmd/anomaly/score.go b/cmd/anomaly/score.go new file mode 100644 index 000000000..8c1bb89d5 --- /dev/null +++ b/cmd/anomaly/score.go @@ -0,0 +1,739 @@ +package main + +import ( + "bytes" + "encoding/json" + "errors" + "flag" + "fmt" + "io" + "math" + "os" + "path/filepath" + "sort" + "strings" + "time" + + "github.com/meshcore-analyzer/anomaly" +) + +// Issue #474, phase 3: score a replay report of the periodic rule against +// the injected ground truth (or against no ground truth at all, which +// measures false alarms on real background traffic). +// +// The matching and the metrics are the #457 §5 periodic rules, with the +// labels coming from `inject` instead of from a hand-labelled channel: +// +// - An episode is one EventID. Its activation is its first transition to +// active; its end is its last transition to ended, or the report's end if +// it never ended. Only episodes activated inside [-start, -end) count. +// - An episode is a true positive for an incident when they are the same +// series, [activation, end] overlaps [incident start, stop + 4P], and +// the episode's period is within the labelling tolerance of k*P for some +// k in 1..max_harmonic. An episode whose activation carries no periodic +// evidence can never be a true positive. +// - Same series means the same stream key when the incident carries one +// (inject writes it), and the same channel hash otherwise. The key is +// what makes a panel placed on real traffic's own keys scorable: on a +// busy channel, several route groups share the channel hash, and +// matching on the hash alone would credit any of them to the bot. +// - -baseline takes the report of the same window without the injection +// (arm A). An episode that already exists there, on the same key and +// overlapping in time, is attributed to the background: it is neither a +// true positive nor a false positive, and is counted on its own. That is +// the differential reading a shared key needs - a detection in arm B +// counts only when it is absent from arm A. +// - Every other episode is a false positive. With no labels file, every +// episode is a false positive, which is the false-alarm arm. +// - The time-to-detect tail is reported twice: over the detected incidents, +// and over every selected incident with a miss ranked above every finite +// value. Only the second can fail on a bot that was never found. +// - -bots narrows which incidents the recall and the medians cover, for +// example to leave a non-blind sanity-check bot out of the primary +// numbers. It never turns a matched episode into a false positive: an +// episode that matches any labelled incident is a true positive +// whether or not that incident was selected. +// +// Scoring reads files only: a report, a labels file and nothing else. + +// scoreCandidate is the part of a report candidate scoring needs. The report +// is this tool's own output, so unknown fields are ignored rather than +// rejected; the fields below are checked instead. +type scoreCandidate struct { + EventID string + Rule string + Kind string + Key string + To string + At time.Time + Periodic *struct { + Period time.Duration + } `json:",omitempty"` +} + +type scoreReport struct { + Experimental bool + Start, End time.Time + Candidates []scoreCandidate +} + +// episode is one EventID's activation and end. +type episode struct { + channel string + key string + activation time.Time + end time.Time + hasPeriod bool + period time.Duration + matches int + // nearMiss: the episode overlapped an incident on the same series but + // failed the period check. + nearMiss bool + // baseline: an episode on the same key, overlapping in time, is already + // in the -baseline report, so this one is not the injection's doing. + baseline bool +} + +// incidentScore reports one labelled incident's outcome. +type incidentScore struct { + Bot string `json:"bot"` + Channel string `json:"channel"` + Start string `json:"start"` + Stop string `json:"stop"` + Period dur `json:"period"` + Episodes int `json:"episodes"` + Detected bool `json:"detected"` + // BaselineEpisodes counts the episodes on this incident's own series, + // overlapping its window, that the -baseline arm already had. They are + // not scored either way (see markBaseline), so without this count an + // incident whose only episode was absorbed by the background reads as a + // plain miss. The period check is not applied to them, because it is + // not applied to a baseline episode anywhere. + BaselineEpisodes int `json:"baseline_episodes,omitempty"` + // BaselineMasked marks exactly that case: not detected, but the + // background already had an episode here. It tells a no-go caused by + // contamination apart from a detector miss. + BaselineMasked bool `json:"baseline_masked,omitempty"` + TimeToDetect *dur `json:"time_to_detect,omitempty"` + TTDPeriods float64 `json:"time_to_detect_periods,omitempty"` + StopLatency *dur `json:"stop_latency,omitempty"` +} + +// scoreResult is the metrics file. +type scoreResult struct { + Experimental bool `json:"experimental"` + Rule string `json:"rule"` + Start string `json:"start"` + End string `json:"end"` + Days float64 `json:"days"` + // Labels and Baseline are provenance: the file names only, never the + // paths they were read from (a path can carry a user or host name). + Labels string `json:"labels"` + Baseline string `json:"baseline"` + // Bots is the -bots filter, empty when every labelled bot counts. + Bots []string `json:"bots"` + Episodes int `json:"episodes"` + // EpisodesWithoutPeriod counts episodes whose activation candidate + // carried no periodic evidence; they can never be true positives. + EpisodesWithoutPeriod int `json:"episodes_without_period"` + // BaselineEpisodes counts episodes the -baseline report already has on + // the same key, overlapping in time. They are neither true nor false + // positives here: they are the background's, not the injection's. + BaselineEpisodes int `json:"baseline_episodes"` + TruePositives int `json:"true_positives"` + FalsePositives int `json:"false_positives"` + FalseAlarmsPerDay float64 `json:"false_alarms_per_day"` + // PeriodMismatches counts episodes that overlap an incident on the same + // channel but fail the period check (#457 "also reported"). + PeriodMismatches int `json:"period_mismatches"` + // Incidents is every labelled incident; IncidentsSelected is the subset + // the bars cover (all of them unless -bots narrows it). + Incidents int `json:"incidents"` + IncidentsSelected int `json:"incidents_selected"` + IncidentsDetected int `json:"incidents_detected"` + // IncidentsMissed is the selected incidents that were not detected, and + // IncidentsBaselineMasked the subset of those whose own series already + // carried a -baseline episode. + IncidentsMissed int `json:"incidents_missed"` + IncidentsBaselineMasked int `json:"incidents_baseline_masked"` + Recall float64 `json:"recall"` + // Medians over the detected incidents. A median hides the slow tail: + // clean trains alarm at the earliest pulse the rule allows, so half the + // panel can sit at that floor while the realistic shapes take many times + // longer. The p75 and the maximum are reported beside every median so + // the tail is visible without reading the per-incident table. + MedianTimeToDetect *dur `json:"median_time_to_detect,omitempty"` + MedianTTDPeriods float64 `json:"median_time_to_detect_periods,omitempty"` + P75TimeToDetect *dur `json:"p75_time_to_detect,omitempty"` + P75TTDPeriods float64 `json:"p75_time_to_detect_periods,omitempty"` + MaxTimeToDetect *dur `json:"max_time_to_detect,omitempty"` + MaxTTDPeriods float64 `json:"max_time_to_detect_periods,omitempty"` + // The tail over the SELECTED incidents, with a missed incident counting + // as +infinity. + // + // Why both. The medians and p75s above are over the DETECTED incidents + // only, so no timing bar on them can fail on a missed bot: a panel whose + // four cleanest shapes alarm at the floor and whose four realistic ones + // are never found reports recall 0.50 and a p75 of one period, which + // passes every timing bar. Ranking a miss above every finite value fixes + // that - the same panel's p75 is then infinite, and a bar on it fails. + // JSON has no infinity, so an infinite percentile is reported as the + // Infinite flag with the value left out. + MedianTTDSelected *dur `json:"median_time_to_detect_selected,omitempty"` + MedianTTDPeriodsSelected float64 `json:"median_time_to_detect_periods_selected,omitempty"` + MedianTTDSelectedInfinite bool `json:"median_time_to_detect_selected_infinite"` + P75TTDSelected *dur `json:"p75_time_to_detect_selected,omitempty"` + P75TTDPeriodsSelected float64 `json:"p75_time_to_detect_periods_selected,omitempty"` + P75TTDSelectedInfinite bool `json:"p75_time_to_detect_selected_infinite"` + MedianStopLatency *dur `json:"median_stop_latency,omitempty"` + P75StopLatency *dur `json:"p75_stop_latency,omitempty"` + MaxStopLatency *dur `json:"max_stop_latency,omitempty"` + MedianEpisodesPerIncident float64 `json:"median_episodes_per_detected_incident,omitempty"` + MaxEpisodesPerIncident float64 `json:"max_episodes_per_detected_incident,omitempty"` + PerIncident []incidentScore `json:"per_incident"` + // FalsePositivesByChannel lists the false positives per channel hash, + // sorted by hash: channel hashes and counts only. + FalsePositivesByChannel []channelCount `json:"false_positives_by_channel"` +} + +type channelCount struct { + Channel string `json:"channel"` + Count int `json:"count"` +} + +func runScore(args []string, stdout, stderr io.Writer) int { + fs := flag.NewFlagSet("score", flag.ContinueOnError) + fs.SetOutput(stderr) + report := fs.String("report", "", "replay report JSON (required)") + labels := fs.String("labels", "", "injected labels JSON; omitted means no ground truth (false alarms only)") + baseline := fs.String("baseline", "", "replay report of the same window without the injection; episodes it already has on the same key are the background's") + rule := fs.String("rule", "", "periodic rule name to score (required)") + bots := fs.String("bots", "", "comma-separated bot names whose incidents the bars cover; empty means all") + start := fs.String("start", "", "RFC 3339 start of the evaluation window, inclusive (required)") + end := fs.String("end", "", "RFC 3339 end of the evaluation window, exclusive (required)") + out := fs.String("out", "", "metrics file (must not exist); default stdout") + if err := fs.Parse(args); err != nil { + return exitUsage + } + usageErr := func(err error) int { + fmt.Fprintf(stderr, "score: %v\n", err) + return exitUsage + } + if fs.NArg() != 0 { + return usageErr(errors.New("unexpected arguments")) + } + if *report == "" || *rule == "" { + return usageErr(errors.New("-report and -rule are required")) + } + var opt struct{ start, end time.Time } + var err error + if opt.start, err = parseTime("start", *start); err != nil { + return usageErr(err) + } + if opt.end, err = parseTime("end", *end); err != nil { + return usageErr(err) + } + if !opt.end.After(opt.start) { + return usageErr(errors.New("-end must be after -start")) + } + fail := func(err error) int { + fmt.Fprintf(stderr, "score: %v\n", err) + return exitFail + } + rb, err := os.ReadFile(*report) + if err != nil { + return fail(err) + } + rep, err := parseScoreReport(rb) + if err != nil { + return fail(err) + } + var lab InjectLabels + if *labels != "" { + lb, err := os.ReadFile(*labels) + if err != nil { + return fail(err) + } + if lab, err = parseInjectLabels(lb); err != nil { + return fail(err) + } + } + var base []episode + if *baseline != "" { + bb, err := os.ReadFile(*baseline) + if err != nil { + return fail(err) + } + brep, err := parseScoreReport(bb) + if err != nil { + return fail(fmt.Errorf("baseline %w", err)) + } + if base, _, err = episodes(brep, *rule, opt.start, opt.end); err != nil { + return fail(fmt.Errorf("baseline %w", err)) + } + } + want := []string{} + for _, b := range strings.Split(*bots, ",") { + if b = strings.TrimSpace(b); b != "" { + want = append(want, b) + } + } + res, err := score(rep, lab, *rule, *labels, *baseline, base, want, opt.start, opt.end) + if err == nil && len(want) > 0 && res.IncidentsSelected == 0 { + return fail(errors.New("-bots selected no labelled incident")) + } + if err != nil { + return fail(err) + } + b, err := json.MarshalIndent(res, "", " ") + if err != nil { + return fail(err) + } + b = append(b, '\n') + if *out == "" { + _, err = stdout.Write(b) + } else { + err = writeNew(*out, b) + } + if err != nil { + return fail(err) + } + return exitOK +} + +func parseScoreReport(b []byte) (scoreReport, error) { + var r scoreReport + if err := json.Unmarshal(b, &r); err != nil { + return scoreReport{}, fmt.Errorf("report: %w", err) + } + if !r.Experimental || r.Start.IsZero() || r.End.IsZero() || !r.End.After(r.Start) { + return scoreReport{}, errors.New("report: not a replay report (Experimental, Start, End)") + } + for i, c := range r.Candidates { + if c.EventID == "" || c.Rule == "" || c.Kind == "" || c.To == "" || c.At.IsZero() { + return scoreReport{}, fmt.Errorf("report: candidate %d is missing EventID, Rule, Kind, To or At", i) + } + if _, err := anomaly.ParseKey(c.Key); err != nil { + return scoreReport{}, fmt.Errorf("report: candidate %d: %w", i, err) + } + } + return r, nil +} + +func parseInjectLabels(b []byte) (InjectLabels, error) { + var l InjectLabels + dec := json.NewDecoder(bytes.NewReader(b)) + dec.DisallowUnknownFields() + if err := dec.Decode(&l); err != nil { + return InjectLabels{}, fmt.Errorf("labels: %w", err) + } + if !onlyJSONValue(dec, b) { + return InjectLabels{}, errors.New("labels: trailing data") + } + if !l.Synthetic { + return InjectLabels{}, errors.New("labels: not an injected labels file (synthetic)") + } + if err := l.Labelling.validate(); err != nil { + return InjectLabels{}, fmt.Errorf("labels: %w", err) + } + for i, inc := range l.Incidents { + if inc.Channel == "" || inc.Start.IsZero() || inc.Stop.IsZero() || inc.Stop.Before(inc.Start) || time.Duration(inc.Period) <= 0 { + return InjectLabels{}, fmt.Errorf("labels: incident %d is incomplete", i) + } + } + return l, nil +} + +// score computes the metrics. labelsPath and baselinePath are reported as +// provenance, by file name only; baseline, when given, are the episodes of +// the same window without the injection; bots, when not empty, selects which +// bots' incidents the bars cover. +func score(rep scoreReport, lab InjectLabels, rule, labelsPath, baselinePath string, baseline []episode, + bots []string, start, end time.Time) (scoreResult, error) { + eps, noPeriod, err := episodes(rep, rule, start, end) + if err != nil { + return scoreResult{}, err + } + markBaseline(eps, baseline) + days := end.Sub(start).Hours() / 24 + res := scoreResult{Experimental: true, Rule: rule, Start: start.Format(time.RFC3339), + End: end.Format(time.RFC3339), Days: days, Labels: filepath.Base(labelsPath), + Baseline: filepath.Base(baselinePath), Episodes: len(eps), + EpisodesWithoutPeriod: noPeriod, Incidents: len(lab.Incidents), Bots: nonNil(bots), + PerIncident: []incidentScore{}, FalsePositivesByChannel: []channelCount{}} + if labelsPath == "" { + res.Labels = "" + } + if baselinePath == "" { + res.Baseline = "" + } + selected := func(bot string) bool { + if len(bots) == 0 { + return true + } + for _, b := range bots { + if b == bot { + return true + } + } + return false + } + l := lab.Labelling + var ttds, stops []int64 + var perInc []float64 + for _, inc := range lab.Incidents { + p := time.Duration(inc.Period) + is := incidentScore{Bot: inc.Bot, Channel: inc.Channel, Start: inc.Start.Format(time.RFC3339), + Stop: inc.Stop.Format(time.RFC3339), Period: inc.Period} + var firstAct, lastEnd time.Time + for i := range eps { + e := &eps[i] + if !sameSeries(*e, inc) || !overlaps(*e, inc, p) { + continue + } + if e.baseline { + // The background's episode: not scored either way, but + // recorded here so a masked incident is not read as a plain + // miss. + is.BaselineEpisodes++ + continue + } + if !e.hasPeriod { + continue // counted once, as EpisodesWithoutPeriod + } + if !periodFits(e.period, p, l) { + e.nearMiss = true + continue + } + e.matches++ + is.Episodes++ + if firstAct.IsZero() || e.activation.Before(firstAct) { + firstAct = e.activation + } + if lastEnd.IsZero() || e.end.After(lastEnd) { + lastEnd = e.end + } + } + if is.Episodes > 0 { + is.Detected = true + ttd := dur(firstAct.Sub(inc.Start)) + sl := dur(lastEnd.Sub(inc.Stop)) + is.TimeToDetect, is.StopLatency = &ttd, &sl + is.TTDPeriods = float64(firstAct.Sub(inc.Start)) / float64(p) + } + // Only selected incidents enter the recall and the medians; an + // episode that matched an unselected incident is still a match, so + // it never becomes a false positive. + is.BaselineMasked = !is.Detected && is.BaselineEpisodes > 0 + if !selected(inc.Bot) { + continue + } + res.IncidentsSelected++ + if is.Episodes > 0 { + res.IncidentsDetected++ + ttds = append(ttds, int64(*is.TimeToDetect)) + stops = append(stops, int64(*is.StopLatency)) + perInc = append(perInc, float64(is.Episodes)) + } else { + res.IncidentsMissed++ + if is.BaselineMasked { + res.IncidentsBaselineMasked++ + } + } + res.PerIncident = append(res.PerIncident, is) + } + byChannel := map[string]int{} + for _, e := range eps { + if e.baseline { + res.BaselineEpisodes++ + continue + } + if e.matches == 0 && e.nearMiss { + res.PeriodMismatches++ + } + if e.matches > 0 { + res.TruePositives++ + continue + } + res.FalsePositives++ + byChannel[e.channel]++ + } + if days > 0 { + res.FalseAlarmsPerDay = float64(res.FalsePositives) / days + } + if res.IncidentsSelected > 0 { + res.Recall = float64(res.IncidentsDetected) / float64(res.IncidentsSelected) + } + // The selected tail is reported whenever anything is selected, including + // the case where every selected incident was missed and every percentile + // is infinite. + if res.IncidentsSelected > 0 { + tp := ttdPeriods(res.PerIncident) + for _, q := range []struct { + q float64 + d **dur + periods *float64 + infinite *bool + }{ + {0.5, &res.MedianTTDSelected, &res.MedianTTDPeriodsSelected, &res.MedianTTDSelectedInfinite}, + {0.75, &res.P75TTDSelected, &res.P75TTDPeriodsSelected, &res.P75TTDSelectedInfinite}, + } { + v, inf := selectedPercentileInt64(ttds, res.IncidentsMissed, q.q) + if inf { + *q.infinite = true + continue + } + d := dur(v) + *q.d = &d + *q.periods, _ = selectedPercentileFloat(tp, res.IncidentsMissed, q.q) + } + } + if len(ttds) > 0 { + tp := ttdPeriods(res.PerIncident) + m, p75, mx := dur(medianInt64(ttds)), dur(percentileInt64(ttds, 0.75)), dur(percentileInt64(ttds, 1)) + res.MedianTimeToDetect, res.P75TimeToDetect, res.MaxTimeToDetect = &m, &p75, &mx + res.MedianTTDPeriods = medianFloat(tp) + res.P75TTDPeriods = percentileFloat(tp, 0.75) + res.MaxTTDPeriods = percentileFloat(tp, 1) + sm, s75, smx := dur(medianInt64(stops)), dur(percentileInt64(stops, 0.75)), dur(percentileInt64(stops, 1)) + res.MedianStopLatency, res.P75StopLatency, res.MaxStopLatency = &sm, &s75, &smx + res.MedianEpisodesPerIncident = medianFloat(perInc) + res.MaxEpisodesPerIncident = percentileFloat(perInc, 1) + } + for ch, n := range byChannel { + res.FalsePositivesByChannel = append(res.FalsePositivesByChannel, channelCount{Channel: ch, Count: n}) + } + sort.Slice(res.FalsePositivesByChannel, func(i, j int) bool { + return res.FalsePositivesByChannel[i].Channel < res.FalsePositivesByChannel[j].Channel + }) + return res, nil +} + +// ttdPeriods collects the time to detect in units of P over the detected +// incidents. +func ttdPeriods(per []incidentScore) []float64 { + var out []float64 + for _, p := range per { + if p.Detected { + out = append(out, p.TTDPeriods) + } + } + return out +} + +// episodes groups a report's candidates for one rule into episodes, keeping +// those activated inside [start, end). +func episodes(rep scoreReport, rule string, start, end time.Time) ([]episode, int, error) { + type agg struct { + channel string + key string + act, endT time.Time + actSet, endSet bool + period time.Duration + hasPeriod bool + } + order := []string{} + m := map[string]*agg{} + for _, c := range rep.Candidates { + if c.Rule != rule || c.Kind != "periodic" { + continue + } + k, err := anomaly.ParseKey(c.Key) + if err != nil { + return nil, 0, fmt.Errorf("report: %w", err) + } + a := m[c.EventID] + if a == nil { + // ParseKey accepts only the canonical encoding, so k.String() + // is c.Key: the key is comparable as a string. + a = &agg{channel: hexChannel(k), key: k.String()} + m[c.EventID] = a + order = append(order, c.EventID) + } + switch c.To { + case "active": + // The first transition to active is the activation, and its + // candidate carries the periodic evidence that is scored. + if !a.actSet || c.At.Before(a.act) { + a.act, a.actSet = c.At, true + a.hasPeriod = c.Periodic != nil + if c.Periodic != nil { + a.period = c.Periodic.Period + } + } + case "ended": + if !a.endSet || c.At.After(a.endT) { + a.endT, a.endSet = c.At, true + } + } + } + var out []episode + noPeriod := 0 + for _, id := range order { + a := m[id] + if !a.actSet || a.act.Before(start) || !a.act.Before(end) { + continue + } + e := episode{channel: a.channel, key: a.key, activation: a.act, end: a.endT, + hasPeriod: a.hasPeriod, period: a.period} + if !a.endSet { + e.end = rep.End // never ended: the report's end, as #457 §1 says + } + if !e.hasPeriod { + noPeriod++ + } + out = append(out, e) + } + return out, noPeriod, nil +} + +// hexChannel returns a key's channel hash as plain lower-case hex, the form +// the labels file uses ("" when the key carries no channel). +func hexChannel(k anomaly.Key) string { + b := k.Channel.Bytes() + if len(b) == 0 { + return "" + } + return fmt.Sprintf("%x", b) +} + +// sameSeries reports whether an episode and an incident describe the same +// series: the same stream key when the incident carries one, and the same +// channel hash for a labels file written before inject recorded keys. +func sameSeries(e episode, inc Incident) bool { + if inc.Key != "" { + return e.key == inc.Key + } + return e.channel == inc.Channel +} + +// markBaseline flags every episode that the baseline report already has on +// the same key, overlapping in time. Such an episode exists without the +// injection, so it says nothing about the injected bot either way. +func markBaseline(eps, baseline []episode) { + if len(baseline) == 0 { + return + } + byKey := map[string][]episode{} + for _, b := range baseline { + byKey[b.key] = append(byKey[b.key], b) + } + for i := range eps { + for _, b := range byKey[eps[i].key] { + if !b.activation.After(eps[i].end) && !b.end.Before(eps[i].activation) { + eps[i].baseline = true + break + } + } + } +} + +// overlaps reports whether an episode's [activation, end] overlaps the +// incident's [start, stop + 4P]. +func overlaps(e episode, inc Incident, p time.Duration) bool { + return !e.activation.After(inc.Stop.Add(4*p)) && !e.end.Before(inc.Start) +} + +// periodFits reports whether the episode's period is within the labelling +// tolerance of k*P for some k in 1..MaxHarmonic. +func periodFits(got, p time.Duration, l labellingSpec) bool { + tol := time.Duration(float64(p) * l.ToleranceRel) + if a := time.Duration(l.ToleranceAbs); a > tol { + tol = a + } + for k := 1; k <= l.MaxHarmonic; k++ { + d := got - time.Duration(k)*p + if d < 0 { + d = -d + } + if d <= tol { + return true + } + } + return false +} + +// nonNil keeps an empty filter out of the JSON as [] rather than null. +func nonNil(v []string) []string { + if v == nil { + return []string{} + } + return v +} + +func medianFloat(v []float64) float64 { + if len(v) == 0 { + return 0 + } + s := append([]float64(nil), v...) + sort.Float64s(s) + n := len(s) + if n%2 == 1 { + return s[n/2] + } + return (s[n/2-1] + s[n/2]) / 2 +} + +// percentileInt64 and percentileFloat are the nearest-rank percentile: the +// value at index ceil(q*n)-1 of the sorted values, so q = 1 is the maximum +// and q = 0.75 of eight values is the sixth. They sort a copy. +func percentileInt64(v []int64, q float64) int64 { + if len(v) == 0 { + return 0 + } + s := append([]int64(nil), v...) + sort.Slice(s, func(i, j int) bool { return s[i] < s[j] }) + return s[rankIndex(len(s), q)] +} + +func percentileFloat(v []float64, q float64) float64 { + if len(v) == 0 { + return 0 + } + s := append([]float64(nil), v...) + sort.Float64s(s) + return s[rankIndex(len(s), q)] +} + +// selectedPercentileInt64 is the nearest-rank percentile over the selected +// incidents, with each missed incident counting as +infinity: the misses +// rank above every finite value, so the percentile is infinite exactly when +// its rank falls among them. It reports (value, infinite). +func selectedPercentileInt64(detected []int64, missed int, q float64) (int64, bool) { + n := len(detected) + missed + if n == 0 { + return 0, false + } + i := rankIndex(n, q) + if i >= len(detected) { + return 0, true + } + s := append([]int64(nil), detected...) + sort.Slice(s, func(a, b int) bool { return s[a] < s[b] }) + return s[i], false +} + +func selectedPercentileFloat(detected []float64, missed int, q float64) (float64, bool) { + n := len(detected) + missed + if n == 0 { + return 0, false + } + i := rankIndex(n, q) + if i >= len(detected) { + return 0, true + } + s := append([]float64(nil), detected...) + sort.Float64s(s) + return s[i], false +} + +func rankIndex(n int, q float64) int { + i := int(math.Ceil(q*float64(n))) - 1 + if i < 0 { + return 0 + } + if i > n-1 { + return n - 1 + } + return i +} diff --git a/cmd/anomaly/score_test.go b/cmd/anomaly/score_test.go new file mode 100644 index 000000000..dba16c359 --- /dev/null +++ b/cmd/anomaly/score_test.go @@ -0,0 +1,482 @@ +package main + +import ( + "bytes" + "encoding/json" + "fmt" + "os" + "path/filepath" + "strings" + "testing" + "time" +) + +// Issue #474, phase 3: scoring a replay report against injected ground +// truth, and against no ground truth at all (the false-alarm arm). + +const scoreKey = "v1|s=stream|pt=5|ch=1:7f|rc=flood|rk=relay|hop=1:fd" + +// replayConfigJSON is the periodic rule the phase-3 runs use. exclude is +// spliced in, for example `"exclude_channels": ["7f"],`. +func replayConfigJSON(exclude string) string { + return `{ + "limits": {"max_keys_per_scope": 200000, "state_ttl": "48h", "dedup_horizon": "6h", "max_dedup_ids": 2000000, + "reorder_delay": "0s", "max_reorder_buffer": 64, "max_window_entries": 200000, + "max_candidates_per_call": 10000, "max_pending_candidates": 100000}, + "expected": "include", + "periodic": [{"name": "p474", "scope": "stream", "payload_types": [5], + "min_period": "1m", "max_period": "1h", "burst_gap": "15s", "jitter_abs": "10s", "jitter_rel": 0.02, + "max_missing": 2, "min_pulses": 8, "history_len": 32, "min_coverage": 0.6, "max_jitter_fraction": 0.5, + "max_chance": 0.01, "end_after_missed_pulses": 3, ` + exclude + ` + "state": {"confirm_after": "0s", "end_after_quiet": "60m", "cooldown": "0s"}}] +}` +} + +// replayInto runs the replay command over one input file and returns the +// report path. +func replayInto(t *testing.T, dir, name, config, input string) string { + t.Helper() + cfgPath := filepath.Join(dir, name+"-config.json") + if err := os.WriteFile(cfgPath, []byte(config), 0o644); err != nil { + t.Fatal(err) + } + report := filepath.Join(dir, name+"-report.json") + var stdout, stderr bytes.Buffer + code := run([]string{"replay", "-config", cfgPath, "-input", input, + "-start", injAt(0), "-end", injAt(24 * time.Hour), "-max-line-bytes", "65536", + "-out", report}, nil, &stdout, &stderr) + if code != 0 { + t.Fatalf("replay exit %d: %s", code, stderr.String()) + } + return report +} + +// scoreInto runs the score command and returns the parsed metrics. +func scoreInto(t *testing.T, report, labels string, extra ...string) scoreResult { + t.Helper() + args := []string{"score", "-report", report, "-rule", "p474", + "-start", injAt(0), "-end", injAt(24 * time.Hour)} + if labels != "" { + args = append(args, "-labels", labels) + } + var stdout, stderr bytes.Buffer + if code := run(append(args, extra...), nil, &stdout, &stderr); code != 0 { + t.Fatalf("score exit %d: %s", code, stderr.String()) + } + var res scoreResult + if err := json.Unmarshal(stdout.Bytes(), &res); err != nil { + t.Fatalf("metrics %q: %v", stdout.String(), err) + } + return res +} + +// End to end: inject a bot into a background, replay it and score it. The +// injected incident is detected, with the time to detect and the stop +// latency the pre-registration asks for, and the background raises no false +// alarm. +func TestScoreRecallOnAnInjectedBot(t *testing.T) { + bg := backgroundJSONL(300) + r := injectInto(t, specJSON(specOpts{seed: 21, jitterSD: "3s"}), bg) + if len(r.labels.Incidents) != 1 { + t.Fatalf("labels %+v", r.labels.Incidents) + } + report := replayInto(t, r.dir, "b", replayConfigJSON(""), r.outPath) + res := scoreInto(t, report, r.labPath) + if res.Incidents != 1 || res.IncidentsDetected != 1 || res.Recall != 1 { + t.Fatalf("recall: %+v", res) + } + if res.TruePositives == 0 || res.FalsePositives != 0 || res.FalseAlarmsPerDay != 0 { + t.Fatalf("precision side: %+v", res) + } + inc := res.PerIncident[0] + if inc.Channel != "7f" || !inc.Detected || inc.TimeToDetect == nil || inc.StopLatency == nil { + t.Fatalf("per-incident %+v", inc) + } + // The rule needs 8 pulses, so the first alarm cannot come before 7 + // periods; on a clean train it comes at 7 or 8. + if inc.TTDPeriods < 7 || inc.TTDPeriods > 12 { + t.Fatalf("time to detect %v (%.1f P)", time.Duration(*inc.TimeToDetect), inc.TTDPeriods) + } + // Train tracking (K = 3) ends the episode (K+1)*P after the last pulse + // at the latest. + if sl := time.Duration(*inc.StopLatency); sl < 0 || sl > 20*time.Minute { + t.Fatalf("stop latency %v", sl) + } + if res.MedianTimeToDetect == nil || res.MedianStopLatency == nil || res.MedianEpisodesPerIncident < 1 { + t.Fatalf("medians: %+v", res) + } + if res.Days != 1 { + t.Fatalf("days %v", res.Days) + } +} + +// A noisy bot of the September shape (5 % missed pulses, 30 % multi-part) is +// still found and still raises no false alarm, but how long that takes is a +// property of the rule, not of the scorer: this run only pins recall and +// logs the timing, which is what the evaluation measures. +func TestScoreRecallOnANoisyInjectedBot(t *testing.T) { + r := injectInto(t, specJSON(specOpts{seed: 21, jitterSD: "3s", missedFraction: 0.05, + multipartFraction: 0.3}), backgroundJSONL(300)) + res := scoreInto(t, replayInto(t, r.dir, "n", replayConfigJSON(""), r.outPath), r.labPath) + if res.Incidents != 1 || res.IncidentsDetected != 1 || res.FalsePositives != 0 { + t.Fatalf("noisy bot: %+v", res) + } + inc := res.PerIncident[0] + t.Logf("noisy bot: %d episodes, time to detect %v (%.1f P), stop latency %v", + inc.Episodes, time.Duration(*inc.TimeToDetect), inc.TTDPeriods, time.Duration(*inc.StopLatency)) +} + +// The same background without any injection is the false-alarm arm: no +// labels, and every episode would be a false alarm. This background raises +// none. +func TestScoreFalseAlarmArmWithoutLabels(t *testing.T) { + dir := t.TempDir() + bgPath := filepath.Join(dir, "bg.jsonl") + if err := os.WriteFile(bgPath, []byte(backgroundJSONL(300)), 0o644); err != nil { + t.Fatal(err) + } + report := replayInto(t, dir, "a", replayConfigJSON(""), bgPath) + res := scoreInto(t, report, "") + if res.Episodes != 0 || res.FalsePositives != 0 || res.FalseAlarmsPerDay != 0 || res.Incidents != 0 { + t.Fatalf("background raised alarms: %+v", res) + } + if res.Recall != 0 || res.MedianTimeToDetect != nil { + t.Fatalf("no labels must give no recall and no medians: %+v", res) + } +} + +// Phase 2 and phase 3 together: with the bot's channel hash excluded, the +// same injected traffic produces no periodic episode at all. +func TestScoreExcludedChannelDetectsNothing(t *testing.T) { + r := injectInto(t, specJSON(specOpts{seed: 21, jitterSD: "3s"}), backgroundJSONL(300)) + plain := scoreInto(t, replayInto(t, r.dir, "p", replayConfigJSON(""), r.outPath), r.labPath) + if plain.IncidentsDetected != 1 { + t.Fatalf("without exclusions the bot must be detected: %+v", plain) + } + excl := scoreInto(t, replayInto(t, r.dir, "x", replayConfigJSON(`"exclude_channels": ["7f"],`), r.outPath), r.labPath) + if excl.Episodes != 0 || excl.IncidentsDetected != 0 || excl.Recall != 0 { + t.Fatalf("the excluded channel still produced episodes: %+v", excl) + } + // Excluding another channel leaves the bot alone. + other := scoreInto(t, replayInto(t, r.dir, "o", replayConfigJSON(`"exclude_channels": ["11"],`), r.outPath), r.labPath) + if other.IncidentsDetected != 1 { + t.Fatalf("excluding channel 11 changed the bot's detection: %+v", other) + } +} + +// ---- crafted reports: the matching rules ---- + +// craftedReport writes a report with one episode per entry. +type craftedEpisode struct { + id string + key string + active time.Duration // activation, relative to injT + ended time.Duration // 0 = never ended + periodNS int64 // 0 = no periodic evidence on the activation +} + +func craftReport(t *testing.T, dir string, eps ...craftedEpisode) string { + t.Helper() + var cands []string + for _, e := range eps { + key := e.key + if key == "" { + key = scoreKey + } + per := "" + if e.periodNS != 0 { + per = fmt.Sprintf(`,"Periodic":{"Period":%d,"Tolerance":10000000000,"Support":8,"Coverage":1}`, e.periodNS) + } + cands = append(cands, fmt.Sprintf(`{"EventID":%q,"Rule":"p474","Kind":"periodic","Key":%q,`+ + `"From":"suspicious","To":"active","Reason":"confirmed","At":%q,"EpisodeStart":%q,`+ + `"Confidence":"route_observed","Traffic":"unclassified","Coverage":[]%s}`, + e.id, key, injT.Add(e.active).Format(time.RFC3339), injT.Add(e.active).Format(time.RFC3339), per)) + if e.ended != 0 { + cands = append(cands, fmt.Sprintf(`{"EventID":%q,"Rule":"p474","Kind":"periodic","Key":%q,`+ + `"From":"active","To":"ended","Reason":"quiet","At":%q,"EpisodeStart":%q,`+ + `"Confidence":"route_observed","Traffic":"unclassified","Coverage":[]}`, + e.id, key, injT.Add(e.ended).Format(time.RFC3339), injT.Add(e.active).Format(time.RFC3339))) + } + } + body := fmt.Sprintf(`{"Experimental":true,"Start":%q,"End":%q,"Input":{"Lines":1},"Candidates":[%s],"CoverageReduced":false}`, + injT.Format(time.RFC3339), injT.Add(24*time.Hour).Format(time.RFC3339), strings.Join(cands, ",")) + p := filepath.Join(dir, fmt.Sprintf("crafted-%d.json", len(eps)+int(time.Now().UnixNano()%1000))) + if err := os.WriteFile(p, []byte(body), 0o644); err != nil { + t.Fatal(err) + } + return p +} + +// craftLabels writes a labels file with one incident on channel 7f, plus +// any extra incidents given. +func craftLabels(t *testing.T, dir string, start, stop time.Duration, period time.Duration, extra ...Incident) string { + t.Helper() + l := InjectLabels{Experimental: true, Synthetic: true, Seed: 1, + Labelling: labellingSpec{PulseMergeGap: dur(time.Minute), MinPeriod: dur(time.Minute), + MaxPeriod: dur(time.Hour), MinPulses: 8, MaxHarmonic: 3, + ToleranceAbs: dur(30 * time.Second), ToleranceRel: 0.1}, + Incidents: append([]Incident{{Bot: "bot-a", Channel: "7f", Start: injT.Add(start), Stop: injT.Add(stop), + Period: dur(period), Pulses: 20}}, extra...)} + b, err := json.MarshalIndent(l, "", " ") + if err != nil { + t.Fatal(err) + } + p := filepath.Join(dir, fmt.Sprintf("labels-%d.json", time.Now().UnixNano())) + if err := os.WriteFile(p, b, 0o644); err != nil { + t.Fatal(err) + } + return p +} + +// An episode is a true positive only if its period agrees with the +// incident's within the labelling tolerance of k*P, k in 1..max_harmonic. +func TestScorePeriodCheck(t *testing.T) { + dir := t.TempDir() + labels := craftLabels(t, dir, time.Hour, 2*time.Hour, 5*time.Minute) + for _, c := range []struct { + name string + periodNS int64 + tp bool + mismatch int + }{ + {"exact", int64(5 * time.Minute), true, 0}, + {"inside the 30 s tolerance", int64(5*time.Minute + 25*time.Second), true, 0}, + {"outside the tolerance", int64(5*time.Minute + 35*time.Second), false, 1}, + {"second harmonic", int64(10 * time.Minute), true, 0}, + {"third harmonic", int64(15 * time.Minute), true, 0}, + {"fourth harmonic", int64(20 * time.Minute), false, 1}, + {"unrelated", int64(99 * time.Second), false, 1}, + {"no periodic evidence", 0, false, 0}, + } { + report := craftReport(t, dir, craftedEpisode{id: "e1", active: 90 * time.Minute, + ended: 110 * time.Minute, periodNS: c.periodNS}) + res := scoreInto(t, report, labels) + tp := res.TruePositives == 1 + if tp != c.tp || res.PeriodMismatches != c.mismatch { + t.Errorf("%s: tp=%v mismatches=%d, want tp=%v mismatches=%d (%+v)", + c.name, tp, res.PeriodMismatches, c.tp, c.mismatch, res) + } + if c.periodNS == 0 && res.EpisodesWithoutPeriod != 1 { + t.Errorf("%s: episodes_without_period %d", c.name, res.EpisodesWithoutPeriod) + } + } +} + +// The match window is [incident start, stop + 4P] against +// [activation, end], and the channel hash must be equal. +func TestScoreOverlapAndChannel(t *testing.T) { + dir := t.TempDir() + p := 5 * time.Minute + labels := craftLabels(t, dir, time.Hour, 2*time.Hour, p) + for _, c := range []struct { + name string + active, end time.Duration + key string + tp, mismatch bool + }{ + {name: "inside", active: 90 * time.Minute, end: 110 * time.Minute, tp: true}, + {name: "ends just inside the start", active: 30 * time.Minute, end: time.Hour, tp: true}, + {name: "ends before the start", active: 10 * time.Minute, end: 50 * time.Minute}, + {name: "activates at stop + 4P", active: 2*time.Hour + 4*p, end: 3 * time.Hour, tp: true}, + {name: "activates after stop + 4P", active: 2*time.Hour + 4*p + time.Minute, end: 3 * time.Hour}, + {name: "never ended", active: 90 * time.Minute, tp: true}, + {name: "another channel", active: 90 * time.Minute, end: 110 * time.Minute, + key: "v1|s=stream|pt=5|ch=1:11|rc=flood|rk=relay|hop=1:fd"}, + } { + report := craftReport(t, dir, craftedEpisode{id: "e1", key: c.key, active: c.active, + ended: c.end, periodNS: int64(p)}) + res := scoreInto(t, report, labels) + if (res.TruePositives == 1) != c.tp { + t.Errorf("%s: tp %d, want %v (%+v)", c.name, res.TruePositives, c.tp, res) + } + if res.Episodes != 1 { + t.Errorf("%s: %d episodes", c.name, res.Episodes) + } + } +} + +// Only episodes activated inside [-start, -end) are scored, and each +// EventID is one episode however many transitions it has. +func TestScoreWindowAndEpisodeGrouping(t *testing.T) { + dir := t.TempDir() + labels := craftLabels(t, dir, time.Hour, 2*time.Hour, 5*time.Minute) + report := craftReport(t, dir, + craftedEpisode{id: "in", active: 90 * time.Minute, ended: 110 * time.Minute, periodNS: int64(5 * time.Minute)}, + craftedEpisode{id: "late", active: 25 * time.Hour, ended: 26 * time.Hour, periodNS: int64(5 * time.Minute)}, + craftedEpisode{id: "early", active: -time.Hour, ended: 30 * time.Minute, periodNS: int64(5 * time.Minute)}, + ) + res := scoreInto(t, report, labels) + if res.Episodes != 1 || res.TruePositives != 1 || res.PerIncident[0].Episodes != 1 { + t.Fatalf("window: %+v", res) + } + // Two episodes on the same incident are two episodes and one detection. + two := craftReport(t, dir, + craftedEpisode{id: "e1", active: 90 * time.Minute, ended: 100 * time.Minute, periodNS: int64(5 * time.Minute)}, + craftedEpisode{id: "e2", active: 105 * time.Minute, ended: 115 * time.Minute, periodNS: int64(5 * time.Minute)}, + ) + res = scoreInto(t, two, labels) + if res.Episodes != 2 || res.TruePositives != 2 || res.IncidentsDetected != 1 || + res.PerIncident[0].Episodes != 2 || res.MedianEpisodesPerIncident != 2 { + t.Fatalf("fragmentation: %+v", res) + } + // The time to detect is the first matching activation and the stop + // latency the last matching end. + if got := time.Duration(*res.MedianTimeToDetect); got != 30*time.Minute { + t.Fatalf("time to detect %v", got) + } + if got := time.Duration(*res.MedianStopLatency); got != -5*time.Minute { + t.Fatalf("stop latency %v", got) + } +} + +// False positives are counted per day and listed per channel hash. +func TestScoreFalsePositivesPerDay(t *testing.T) { + dir := t.TempDir() + labels := craftLabels(t, dir, time.Hour, 2*time.Hour, 5*time.Minute) + report := craftReport(t, dir, + craftedEpisode{id: "tp", active: 90 * time.Minute, ended: 110 * time.Minute, periodNS: int64(5 * time.Minute)}, + craftedEpisode{id: "fp1", active: 10 * time.Hour, ended: 11 * time.Hour, periodNS: int64(5 * time.Minute)}, + craftedEpisode{id: "fp2", active: 12 * time.Hour, ended: 13 * time.Hour, periodNS: int64(5 * time.Minute), + key: "v1|s=stream|pt=5|ch=1:11|rc=flood|rk=relay|hop=1:fd"}, + ) + res := scoreInto(t, report, labels) + if res.FalsePositives != 2 || res.FalseAlarmsPerDay != 2 { + t.Fatalf("false alarms: %+v", res) + } + if len(res.FalsePositivesByChannel) != 2 || res.FalsePositivesByChannel[0].Channel != "11" || + res.FalsePositivesByChannel[1].Channel != "7f" { + t.Fatalf("per channel: %+v", res.FalsePositivesByChannel) + } +} + +// Another rule's episodes are not scored. +func TestScoreOnlyTheNamedRule(t *testing.T) { + dir := t.TempDir() + report := craftReport(t, dir, craftedEpisode{id: "e1", active: 90 * time.Minute, + ended: 110 * time.Minute, periodNS: int64(5 * time.Minute)}) + b, err := os.ReadFile(report) + if err != nil { + t.Fatal(err) + } + other := filepath.Join(dir, "other.json") + if err := os.WriteFile(other, bytes.ReplaceAll(b, []byte(`"Rule":"p474"`), []byte(`"Rule":"p999"`)), 0o644); err != nil { + t.Fatal(err) + } + if res := scoreInto(t, other, ""); res.Episodes != 0 { + t.Fatalf("another rule was scored: %+v", res) + } + if res := scoreInto(t, report, ""); res.Episodes != 1 { + t.Fatalf("the named rule was not scored: %+v", res) + } +} + +func TestScoreRequiresExplicitFlagsAndValidInput(t *testing.T) { + dir := t.TempDir() + report := craftReport(t, dir, craftedEpisode{id: "e1", active: time.Hour, periodNS: int64(5 * time.Minute)}) + full := []string{"score", "-report", report, "-rule", "p474", "-start", injAt(0), "-end", injAt(24 * time.Hour)} + for i := 1; i < len(full); i += 2 { + args := append(append([]string{}, full[:i]...), full[i+2:]...) + var stdout, stderr bytes.Buffer + if code := run(args, nil, &stdout, &stderr); code != exitUsage { + t.Fatalf("without %s: exit %d", full[i], code) + } + } + var stdout, stderr bytes.Buffer + if code := run(append(append([]string{}, full...), "-end", injAt(-time.Hour)), nil, &stdout, &stderr); code != exitUsage { + t.Fatalf("-end before -start: exit %d", code) + } + // A report that is not a replay report, and a labels file that is not an + // injected labels file, are both errors. + bad := filepath.Join(dir, "bad.json") + for _, body := range []string{`{}`, `{"Experimental":true}`, `not json`, + `{"Experimental":true,"Start":"2030-03-01T00:00:00Z","End":"2030-03-02T00:00:00Z","Candidates":[{"EventID":"e","Rule":"p474","Kind":"periodic","Key":"nonsense","To":"active","At":"2030-03-01T01:00:00Z"}]}`} { + if err := os.WriteFile(bad, []byte(body), 0o644); err != nil { + t.Fatal(err) + } + args := append([]string{}, full...) + args[2] = bad + if code := run(args, nil, &stdout, &stderr); code != exitFail { + t.Fatalf("report %q: exit %d", body, code) + } + } + badLabels := filepath.Join(dir, "badlabels.json") + validLabelling := `"labelling":{"pulse_merge_gap":"60s","min_period":"1m","max_period":"1h","min_pulses":8,"max_harmonic":3,"tolerance_abs":"30s","tolerance_rel":0.1}` + for _, body := range []string{`{}`, + // a complete file that is not marked synthetic: scoring is only + // defined against injected ground truth + `{"experimental":true,"synthetic":false,` + validLabelling + `,"incidents":[]}`, + `{"experimental":true,"synthetic":true,"labelling":{"pulse_merge_gap":"60s","min_period":"1m","max_period":"1h","min_pulses":8,"max_harmonic":3,"tolerance_abs":"30s","tolerance_rel":0.1},"incidents":[{"bot":"b","channel":"","start":"2030-03-01T00:00:00Z","stop":"2030-03-01T01:00:00Z","period":"5m","pulses":9}]}`, + `{"experimental":true,"synthetic":true,"surprise":1}`} { + if err := os.WriteFile(badLabels, []byte(body), 0o644); err != nil { + t.Fatal(err) + } + if code := run(append(append([]string{}, full...), "-labels", badLabels), nil, &stdout, &stderr); code != exitFail { + t.Fatalf("labels %q: exit %d", body, code) + } + } +} + +// -out writes the metrics to a new file and never overwrites one. +func TestScoreWritesOutFileOnce(t *testing.T) { + dir := t.TempDir() + report := craftReport(t, dir, craftedEpisode{id: "e1", active: time.Hour, periodNS: int64(5 * time.Minute)}) + out := filepath.Join(dir, "metrics.json") + args := []string{"score", "-report", report, "-rule", "p474", "-start", injAt(0), + "-end", injAt(24 * time.Hour), "-out", out} + var stdout, stderr bytes.Buffer + if code := run(args, nil, &stdout, &stderr); code != 0 || stdout.Len() != 0 { + t.Fatalf("exit %d, stdout %d bytes: %s", code, stdout.Len(), stderr.String()) + } + var res scoreResult + b, err := os.ReadFile(out) + if err != nil || json.Unmarshal(b, &res) != nil || res.Episodes != 1 { + t.Fatalf("metrics file: %v %s", err, b) + } + if code := run(args, nil, &stdout, &stderr); code != exitFail { + t.Fatal("an existing metrics file was overwritten") + } +} + +// -bots narrows which incidents the bars cover, for the non-blind +// sanity-check bot of the #474 pre-registration. An episode that matches an +// unselected incident stays a true positive: it must never turn into a +// false alarm. +func TestScoreBotsFilter(t *testing.T) { + dir := t.TempDir() + p := 5 * time.Minute + labels := craftLabels(t, dir, time.Hour, 2*time.Hour, p, + Incident{Bot: "sanity", Channel: "11", Start: injT.Add(10 * time.Hour), + Stop: injT.Add(11 * time.Hour), Period: dur(p), Pulses: 20}) + report := craftReport(t, dir, + craftedEpisode{id: "blind", active: 90 * time.Minute, ended: 110 * time.Minute, periodNS: int64(p)}, + craftedEpisode{id: "sanity", active: 10*time.Hour + 30*time.Minute, ended: 11 * time.Hour, + periodNS: int64(p), key: "v1|s=stream|pt=5|ch=1:11|rc=flood|rk=relay|hop=1:fd"}, + ) + all := scoreInto(t, report, labels) + if all.Incidents != 2 || all.IncidentsSelected != 2 || all.IncidentsDetected != 2 || all.Recall != 1 || + all.TruePositives != 2 || all.FalsePositives != 0 { + t.Fatalf("unfiltered: %+v", all) + } + blind := scoreInto(t, report, labels, "-bots", "bot-a") + if blind.Incidents != 2 || blind.IncidentsSelected != 1 || blind.IncidentsDetected != 1 || + blind.Recall != 1 || len(blind.PerIncident) != 1 || blind.PerIncident[0].Bot != "bot-a" { + t.Fatalf("blind arm: %+v", blind) + } + // The sanity bot's episode is still a true positive, not a false alarm. + if blind.TruePositives != 2 || blind.FalsePositives != 0 || blind.FalseAlarmsPerDay != 0 { + t.Fatalf("the unselected incident's episode was counted as a false alarm: %+v", blind) + } + if len(blind.Bots) != 1 || blind.Bots[0] != "bot-a" { + t.Fatalf("the filter is not reported: %+v", blind.Bots) + } + sanity := scoreInto(t, report, labels, "-bots", " sanity ,") + if sanity.IncidentsSelected != 1 || sanity.PerIncident[0].Bot != "sanity" { + t.Fatalf("sanity arm: %+v", sanity) + } + // A filter that selects nothing is an error, not an empty pass. + var stdout, stderr bytes.Buffer + if code := run([]string{"score", "-report", report, "-labels", labels, "-rule", "p474", + "-start", injAt(0), "-end", injAt(24 * time.Hour), "-bots", "nobody"}, nil, &stdout, &stderr); code != exitFail { + t.Fatalf("-bots with an unknown name: exit %d", code) + } +} diff --git a/internal/anomaly/config.go b/internal/anomaly/config.go index 167f2a7eb..47dfd1db6 100644 --- a/internal/anomaly/config.go +++ b/internal/anomaly/config.go @@ -160,6 +160,29 @@ type PeriodicRule struct { // recent (local) pulse rate, over the period search and every pulse // evaluated on the key. MaxChance float64 + // ExcludeChannels lists channel hashes this rule ignores: an event whose + // channel hash is in the list is not offered to this rule at all, so the + // rule never forms a pulse, a chain or an episode on it. It is opt-in and + // has no default: an empty list excludes nothing. + // + // It exists because some channels carry expected automation, where a + // regular pulse train is normal rather than an incident (#474: a + // wardriving app posts positions every 75-123 s while someone drives). + // The library never learns which channel that is; the caller names the + // hashes. + // + // An entry must be a valid, canonical, non-zero ChannelHash, and matching + // is on the whole value, length included: the 1-byte hash 0x81 and the + // 2-byte hash 0x81ab are different entries, as everywhere else in this + // package, and the zero "no channel" value is not a valid entry. + // + // A channel hash is not a channel. MeshCore's hash is one byte, so one + // entry excludes every channel whose name hashes to that byte: excluding + // the hash of #wardriving also excludes the 1/256 of all other channel + // names that collide with it, and any traffic of theirs that this rule + // would otherwise have reported. That is a deliberate, stated cost of + // keying on an opaque 1-byte hash, not something the list can avoid. + ExcludeChannels []ChannelHash // EndAfterMissedPulses opts into train tracking (see track.go): while // the episode is active it ends when this many consecutive expected // pulses are missing, instead of State.EndAfterQuiet after the last @@ -176,6 +199,10 @@ const ( maxMissingLimit = 8 maxRuleNameLen = 64 maxKeysLimit = 10_000_000 + // maxExcludedChannels bounds PeriodicRule.ExcludeChannels. 256 is the + // whole 1-byte hash space MeshCore uses today, so the bound can never + // stop a caller from naming every channel hash it sees. + maxExcludedChannels = 256 ) var errNoRules = errors.New("anomaly: config enables no rule") @@ -353,6 +380,9 @@ func (c *Config) Validate() error { case r.EndAfterMissedPulses < 0 || r.EndAfterMissedPulses > maxMissedPulses: return fmt.Errorf("anomaly: rule %s: EndAfterMissedPulses must be in 0..%d", r.Name, maxMissedPulses) } + if err := validExcludeChannels(r.ExcludeChannels, r.Name); err != nil { + return err + } // A tolerance at or above half the period would let one gap match two // multiples; refuse it. if tolFor(r, r.MinPeriod) >= r.MinPeriod/2 { @@ -376,6 +406,45 @@ func (c *Config) Validate() error { return nil } +// validExcludeChannels checks a rule's exclusion list: every entry is a +// canonical, non-zero channel hash, no entry repeats, and the list stays +// bounded. An empty list is valid and excludes nothing. +func validExcludeChannels(cs []ChannelHash, name string) error { + if len(cs) > maxExcludedChannels { + return fmt.Errorf("anomaly: rule %s: ExcludeChannels must hold at most %d entries", name, maxExcludedChannels) + } + for i, c := range cs { + if c.IsZero() { + return fmt.Errorf("anomaly: rule %s: ExcludeChannels must not hold the zero \"no channel\" value", name) + } + if err := c.canonical(); err != nil { + return fmt.Errorf("anomaly: rule %s: ExcludeChannels: %w", name, err) + } + for _, o := range cs[:i] { + if o == c { + return fmt.Errorf("anomaly: rule %s: ExcludeChannels lists the same channel hash twice", name) + } + } + } + return nil +} + +// excludes reports whether this rule ignores events on channel hash c. +// Matching is on the whole value (length and bytes), so a 1-byte hash is +// never read as the prefix of a wider one, and the zero "no channel" value +// never matches (validExcludeChannels rejects it as an entry). +func (r *PeriodicRule) excludes(c ChannelHash) bool { + if c.IsZero() { + return false + } + for _, x := range r.ExcludeChannels { + if x == c { + return true + } + } + return false +} + func tolFor(r PeriodicRule, period time.Duration) time.Duration { rel := time.Duration(r.JitterRel * float64(period)) if r.JitterAbs > rel { diff --git a/internal/anomaly/detector.go b/internal/anomaly/detector.go index ac77fef1a..9d2baff72 100644 --- a/internal/anomaly/detector.go +++ b/internal/anomaly/detector.go @@ -134,8 +134,9 @@ func New(cfg Config) (*Detector, error) { c.Rate = append([]RateRule(nil), cfg.Rate...) c.NewStream = append([]NewStreamRule(nil), cfg.NewStream...) c.Periodic = append([]PeriodicRule(nil), cfg.Periodic...) - // PayloadTypes are the only slices inside the rules: copy them too, so the - // caller's config and the Detector never share a backing array. + // PayloadTypes, and a periodic rule's ExcludeChannels, are the only + // slices inside the rules: copy them too, so the caller's config and the + // Detector never share a backing array. for i := range c.Rate { c.Rate[i].PayloadTypes = slices.Clone(c.Rate[i].PayloadTypes) } @@ -144,6 +145,7 @@ func New(cfg Config) (*Detector, error) { } for i := range c.Periodic { c.Periodic[i].PayloadTypes = slices.Clone(c.Periodic[i].PayloadTypes) + c.Periodic[i].ExcludeChannels = slices.Clone(c.Periodic[i].ExcludeChannels) } d := &Detector{cfg: c, dedup: make(map[TxID]int64)} for i := range c.Rate { @@ -621,7 +623,7 @@ func (d *Detector) handle(e *Event) { applies = applies || appliesTo(d.cfg.NewStream[i].PayloadTypes, e.PayloadType) } for _, i := range r.periodic { - applies = applies || appliesTo(d.cfg.Periodic[i].PayloadTypes, e.PayloadType) + applies = applies || periodicApplies(&d.cfg.Periodic[i], e) } if !applies { continue @@ -674,7 +676,7 @@ func (d *Detector) handle(e *Event) { } for pi, i := range r.periodic { rule := &d.cfg.Periodic[i] - if appliesTo(rule.PayloadTypes, e.PayloadType) { + if periodicApplies(rule, e) { ps := k.per[pi] before := ps.evaluated ep := &k.eps[slot] @@ -694,6 +696,15 @@ func (d *Detector) handle(e *Event) { } } +// periodicApplies reports whether a periodic rule sees this event: its +// payload type must match, and its channel hash must not be one the rule +// excludes (PeriodicRule.ExcludeChannels). An excluded event never reaches +// the rule, so the rule forms no pulse, chain or episode on that channel and +// an excluded channel alone never claims key state. +func periodicApplies(rule *PeriodicRule, e *Event) bool { + return appliesTo(rule.PayloadTypes, e.PayloadType) && !rule.excludes(e.Channel) +} + // counts reports whether the rule counts e: its payload type, and not a // flagged event for a guarded rule. func (r *RateRule) counts(e *Event) bool { diff --git a/internal/anomaly/doc.go b/internal/anomaly/doc.go index 15258c598..566e6e1df 100644 --- a/internal/anomaly/doc.go +++ b/internal/anomaly/doc.go @@ -44,6 +44,28 @@ // - A global (payload-type) candidate is never attributed to a stream, // channel or route. // +// # What the periodic rule is for (#474) +// +// The periodic rule's target is an abusive channel bot: a sender that floods +// a public or hashtag channel with automated messages. Automation that is +// expected on its channel is NOT a target. The #457 round-4 validation +// measured the opposite: all six labelled periodic incidents were wardriving +// trips, where an app posts positions every 75-123 s while someone drives, +// so the recipe counted normal behaviour as incidents and the rule failed on +// recall (0.17). +// +// PeriodicRule.ExcludeChannels is the library's side of that correction: the +// caller names the channel hashes whose automation is expected, and the rule +// never sees an event on them. It is opt-in and has no default, so an +// unconfigured rule behaves exactly as before. Because a MeshCore channel +// hash is one byte, one entry excludes every channel name that hashes to +// that byte; see PeriodicRule.ExcludeChannels. +// +// Neither the list nor anything else here makes a rule validated. A target +// definition, an exclusion list and an evaluation are three different +// things, and only the evaluation (fresh data, pre-registered bars) can +// support a claim. +// // # Data contract // // The Detector consumes FINAL Events, one per logical transmission, and diff --git a/internal/anomaly/exclude_channels_test.go b/internal/anomaly/exclude_channels_test.go new file mode 100644 index 000000000..ca5872b7c --- /dev/null +++ b/internal/anomaly/exclude_channels_test.go @@ -0,0 +1,289 @@ +package anomaly + +import ( + "testing" + "time" +) + +// Issue #474, phase 2: a periodic rule can ignore expected-automation +// channels (for example #wardriving) through PeriodicRule.ExcludeChannels. +// The option is opt-in: an empty list excludes nothing, which is the +// behaviour every other test in this package pins. +// +// All channel bytes here are synthetic test values. 0x81 is used because the +// #457 round-4 results name it, but nothing in the library knows what a +// channel hash means. + +// perfectTrain is a perfect 12-pulse 5-minute train on one stream, which +// experimentalPeriodic activates on at the 8th pulse. +func perfectTrain(t testing.TB, prefix string, ch byte) []Event { + t.Helper() + return series(t, prefix, t0, 12, 5*time.Minute, nil, ch, 0x22) +} + +// excludingRule returns experimentalPeriodic with the given channel hashes +// excluded. +func excludingRule(t testing.TB, chans ...ChannelHash) PeriodicRule { + t.Helper() + r := experimentalPeriodic() + r.ExcludeChannels = chans + return r +} + +// A train on an excluded channel produces no periodic candidate at all, +// while the same train on another channel still activates. +func TestPeriodicExcludedChannelProducesNoCandidate(t *testing.T) { + rule := excludingRule(t, mustChan(t, 0x81)) + d := newDet(t, Config{Periodic: []PeriodicRule{rule}}) + if cs := run(t, d, perfectTrain(t, "x", 0x81)); len(cs) != 0 { + t.Fatalf("excluded channel produced candidates:\n%s", describe(cs)) + } + d = newDet(t, Config{Periodic: []PeriodicRule{rule}}) + cs := run(t, d, perfectTrain(t, "y", 0x11)) + if len(filter(cs, "per", StateActive)) != 1 { + t.Fatalf("a non-excluded channel must still activate:\n%s", describe(cs)) + } +} + +// The exclusion matches the whole channel hash, length included: excluding +// the 1-byte hash 0x81 does not exclude the 2-byte hash 0x81ab, exactly as +// Key.String keeps a 1-byte value distinct from the prefix of a wider one. +func TestPeriodicExclusionMatchesTheWholeHash(t *testing.T) { + rule := excludingRule(t, mustChan(t, 0x81)) + wide := perfectTrain(t, "w", 0x81) + for i := range wide { + wide[i].Channel = mustChan(t, 0x81, 0xab) + } + cs := run(t, newDet(t, Config{Periodic: []PeriodicRule{rule}}), wide) + if len(filter(cs, "per", StateActive)) != 1 { + t.Fatalf("a 2-byte hash must not be excluded by a 1-byte entry:\n%s", describe(cs)) + } + // The reverse: a 2-byte entry does not exclude the 1-byte hash. + rule = excludingRule(t, mustChan(t, 0x81, 0xab)) + cs = run(t, newDet(t, Config{Periodic: []PeriodicRule{rule}}), perfectTrain(t, "n", 0x81)) + if len(filter(cs, "per", StateActive)) != 1 { + t.Fatalf("a 1-byte hash must not be excluded by a 2-byte entry:\n%s", describe(cs)) + } + // The length is part of the match even when the bytes alone look equal: + // the 2-byte hash 0x8100 is not the 1-byte hash 0x81 followed by + // padding, so a 0x81 entry does not exclude it. + rule = excludingRule(t, mustChan(t, 0x81)) + padded := perfectTrain(t, "z", 0x81) + for i := range padded { + padded[i].Channel = mustChan(t, 0x81, 0x00) + } + cs = run(t, newDet(t, Config{Periodic: []PeriodicRule{rule}}), padded) + if len(filter(cs, "per", StateActive)) != 1 { + t.Fatalf("the 2-byte hash 0x8100 was excluded by the 1-byte entry 0x81:\n%s", describe(cs)) + } +} + +// Every listed channel is excluded, not only the first one. +func TestPeriodicExclusionCoversEveryListedChannel(t *testing.T) { + rule := excludingRule(t, mustChan(t, 0x81), mustChan(t, 0x62)) + for _, ch := range []byte{0x81, 0x62} { + d := newDet(t, Config{Periodic: []PeriodicRule{rule}}) + if cs := run(t, d, perfectTrain(t, "m", ch)); len(cs) != 0 { + t.Fatalf("channel %#x produced candidates:\n%s", ch, describe(cs)) + } + } +} + +// The exclusion belongs to the periodic rule only: rate and new-stream +// rules on the same events are untouched, and so is a second periodic rule +// without the exclusion. +func TestPeriodicExclusionLeavesOtherRulesAlone(t *testing.T) { + other := experimentalPeriodic() + other.Name = "per2" + cfg := Config{ + Rate: []RateRule{rateRule("r", ScopeStream, time.Hour, 8)}, + NewStream: []NewStreamRule{{Name: "ns", Scope: ScopeStream, Window: time.Hour, MinCount: 6, + CensorWindow: 24 * time.Hour, QuietPeriod: 24 * time.Hour, State: quickState()}}, + Periodic: []PeriodicRule{excludingRule(t, mustChan(t, 0x81)), other}, + HistoryStart: t0.Add(-48 * time.Hour), + } + cs := run(t, newDet(t, cfg), perfectTrain(t, "o", 0x81)) + for _, rule := range []string{"r", "ns", "per2"} { + if len(filter(cs, rule, StateActive)) == 0 { + t.Errorf("rule %s did not activate on an excluded channel:\n%s", rule, describe(cs)) + } + } + if a := filter(cs, "per", StateActive); len(a) != 0 { + t.Fatalf("the excluding rule activated:\n%s", describe(cs)) + } +} + +// With only an excluding periodic rule, an event on an excluded channel +// creates no key state at all, so the exclusion also keeps the scope's key +// budget free. +func TestPeriodicExclusionKeepsNoKeyState(t *testing.T) { + d := newDet(t, Config{Periodic: []PeriodicRule{excludingRule(t, mustChan(t, 0x81))}}) + run(t, d, perfectTrain(t, "k", 0x81)) + if n := d.Snapshot().Keys[ScopeStream]; n != 0 { + t.Fatalf("excluded channel kept %d stream keys, want 0", n) + } + d = newDet(t, Config{Periodic: []PeriodicRule{excludingRule(t, mustChan(t, 0x81))}}) + run(t, d, perfectTrain(t, "k2", 0x11)) + if n := d.Snapshot().Keys[ScopeStream]; n != 1 { + t.Fatalf("a non-excluded channel kept %d stream keys, want 1", n) + } +} + +// New copies ExcludeChannels: changing the caller's slice afterwards cannot +// change what the detector excludes, and the detector never writes to it. +func TestPeriodicExclusionIsCopiedFromTheCallerConfig(t *testing.T) { + rule := excludingRule(t, mustChan(t, 0x81)) + cfg := Config{Periodic: []PeriodicRule{rule}, Expected: ExpectedPolicy{Mode: ExpectedInclude}, + Limits: experimentalLimits()} + d, err := New(cfg) + if err != nil { + t.Fatal(err) + } + cfg.Periodic[0].ExcludeChannels[0] = mustChan(t, 0x11) + if cs := run(t, d, perfectTrain(t, "c", 0x81)); len(cs) != 0 { + t.Fatalf("the caller's slice changed what the detector excludes:\n%s", describe(cs)) + } + if got := cfg.Periodic[0].ExcludeChannels[0]; got != mustChan(t, 0x11) { + t.Fatalf("the detector wrote to the caller's slice: %v", got.Bytes()) + } +} + +// An event without a channel (a payload type that carries none) is never +// excluded: the zero "no channel" value is not a 1-byte hash and is not a +// valid list entry. +func TestPeriodicExclusionNeverMatchesNoChannel(t *testing.T) { + rule := excludingRule(t, mustChan(t, 0x00)) + rule.PayloadTypes = []PayloadType{4} + var evs []Event + for i := 0; i < 12; i++ { + evs = append(evs, Event{ID: TxID("a" + string(rune('a'+i))), Time: at(time.Duration(i) * 5 * time.Minute), + PayloadType: 4, RouteClass: RouteClassFlood, RouteKind: RouteKindRelay, + FirstHop: mustHop(t, 0x22), FullRoute: mustRoute(t, 1, 0x22, 0x00), WireBytes: 40}) + } + cs := run(t, newDet(t, Config{Periodic: []PeriodicRule{rule}}), evs) + if len(filter(cs, "per", StateActive)) != 1 { + t.Fatalf("a channel-less train must not be excluded by a 1-byte 0x00 entry:\n%s", describe(cs)) + } +} + +// The exclusion works at channel scope too, where the key is the channel. +func TestPeriodicExclusionAtChannelScope(t *testing.T) { + rule := excludingRule(t, mustChan(t, 0x81)) + rule.Scope = ScopeChannel + d := newDet(t, Config{Periodic: []PeriodicRule{rule}}) + if cs := run(t, d, perfectTrain(t, "s", 0x81)); len(cs) != 0 { + t.Fatalf("channel scope ignored the exclusion:\n%s", describe(cs)) + } +} + +func TestExcludeChannelsValidation(t *testing.T) { + for _, c := range []struct { + name string + list []ChannelHash + ok bool + }{ + {"unset", nil, true}, + {"empty", []ChannelHash{}, true}, + {"one byte", []ChannelHash{mustChan(t, 0x81)}, true}, + {"two entries", []ChannelHash{mustChan(t, 0x81), mustChan(t, 0x62)}, true}, + {"wider hash", []ChannelHash{mustChan(t, 0x81, 0xab, 0xcd, 0xef)}, true}, + {"same byte twice", []ChannelHash{mustChan(t, 0x81), mustChan(t, 0x81)}, false}, + {"no channel", []ChannelHash{{}}, false}, + {"non-canonical", []ChannelHash{nonCanonicalChan(0x81)}, false}, + } { + r := experimentalPeriodic() + r.ExcludeChannels = c.list + err := (&Config{Limits: experimentalLimits(), Expected: ExpectedPolicy{Mode: ExpectedInclude}, + Periodic: []PeriodicRule{r}}).Validate() + if (err == nil) != c.ok { + t.Errorf("%s: err %v, want ok=%v", c.name, err, c.ok) + } + } + // Too many entries are rejected; the whole 1-byte space is not. + full := make([]ChannelHash, 0, maxExcludedChannels+1) + for i := 0; i <= maxExcludedChannels; i++ { + c, err := NewChannelHash([]byte{byte(i % 256), byte(i / 256)}[:1+i/256]) + if err != nil { + t.Fatal(err) + } + full = append(full, c) + } + for _, c := range []struct { + n int + ok bool + }{{maxExcludedChannels, true}, {maxExcludedChannels + 1, false}} { + r := experimentalPeriodic() + r.ExcludeChannels = full[:c.n] + err := (&Config{Limits: experimentalLimits(), Expected: ExpectedPolicy{Mode: ExpectedInclude}, + Periodic: []PeriodicRule{r}}).Validate() + if (err == nil) != c.ok { + t.Errorf("%d entries: err %v, want ok=%v", c.n, err, c.ok) + } + } +} + +// nonCanonicalChan builds a 1-byte hash with a stray byte beyond its length, +// which Key.Validate rejects elsewhere and the exclusion list rejects here. +func nonCanonicalChan(b byte) ChannelHash { + c := ChannelHash{n: 1} + c.b[0], c.b[1] = b, 0xff + return c +} + +// ---- replay JSON config ---- + +func TestReplayConfigParsesExcludeChannels(t *testing.T) { + cfg, err := ParseReplayConfig([]byte(excludeConfigJSON(`"exclude_channels": ["81", "62ab"],`))) + if err != nil { + t.Fatal(err) + } + got := cfg.Periodic[0].ExcludeChannels + if len(got) != 2 || got[0] != mustChan(t, 0x81) || got[1] != mustChan(t, 0x62, 0xab) { + t.Fatalf("exclude_channels parsed as %v", got) + } +} + +// Omitting exclude_channels means no exclusion: the field is opt-in and has +// no default. +func TestReplayConfigWithoutExcludeChannelsExcludesNothing(t *testing.T) { + cfg, err := ParseReplayConfig([]byte(excludeConfigJSON(""))) + if err != nil { + t.Fatal(err) + } + if got := cfg.Periodic[0].ExcludeChannels; got != nil { + t.Fatalf("exclude_channels defaulted to %v", got) + } +} + +func TestReplayConfigRejectsBadExcludeChannels(t *testing.T) { + for name, v := range map[string]string{ + "odd hex": `["8"]`, + "not hex": `["zz"]`, + "empty string": `[""]`, + "too long": `["0102030405"]`, + "upper case": `["AB"]`, + "with prefix": `["0x81"]`, + "duplicate": `["81", "81"]`, + "not a string": `[129]`, + } { + if _, err := ParseReplayConfig([]byte(excludeConfigJSON(`"exclude_channels": ` + v + `,`))); err == nil { + t.Errorf("%s: accepted %s", name, v) + } + } +} + +// excludeConfigJSON is a minimal valid replay config with one periodic rule, +// into which a line such as `"exclude_channels": ["81"],` is spliced. +func excludeConfigJSON(line string) string { + return `{ + "limits": {"max_keys_per_scope": 1000, "state_ttl": "48h", "dedup_horizon": "6h", "max_dedup_ids": 1000, + "reorder_delay": "0s", "max_reorder_buffer": 64, "max_window_entries": 1000, + "max_candidates_per_call": 100, "max_pending_candidates": 1000}, + "expected": "include", + "periodic": [{"name": "p474", "scope": "stream", "payload_types": [5], + "min_period": "1m", "max_period": "1h", "burst_gap": "15s", "jitter_abs": "10s", "jitter_rel": 0.02, + "max_missing": 2, "min_pulses": 8, "history_len": 32, "min_coverage": 0.6, "max_jitter_fraction": 0.5, + "max_chance": 0.01, "end_after_missed_pulses": 3, ` + line + ` + "state": {"confirm_after": "0s", "end_after_quiet": "60m", "cooldown": "0s"}}] +}` +} diff --git a/internal/anomaly/replay.go b/internal/anomaly/replay.go index 8b0968565..019e9433c 100644 --- a/internal/anomaly/replay.go +++ b/internal/anomaly/replay.go @@ -384,8 +384,12 @@ type ReplayConfig struct { MaxChance float64 `json:"max_chance"` // EndAfterMissedPulses is optional: omitted means 0, tracking off // (see PeriodicRule.EndAfterMissedPulses). - EndAfterMissedPulses int `json:"end_after_missed_pulses"` - State stateJSON `json:"state"` + EndAfterMissedPulses int `json:"end_after_missed_pulses"` + // ExcludeChannels is optional: channel hashes this rule ignores, as + // plain lower-case hex of 1 to MaxChannelHashLen bytes ("81"). + // Omitted means none (see PeriodicRule.ExcludeChannels). + ExcludeChannels []string `json:"exclude_channels"` + State stateJSON `json:"state"` } `json:"periodic"` } @@ -451,11 +455,41 @@ func ParseReplayConfig(b []byte) (Config, error) { if err != nil { return Config{}, err } + ex, err := parseExcludeChannels(r.ExcludeChannels) + if err != nil { + return Config{}, err + } cfg.Periodic = append(cfg.Periodic, PeriodicRule{Name: r.Name, Scope: s, PayloadTypes: r.PayloadTypes, - MinPeriod: time.Duration(r.MinPeriod), MaxPeriod: time.Duration(r.MaxPeriod), BurstGap: time.Duration(r.BurstGap), + ExcludeChannels: ex, + MinPeriod: time.Duration(r.MinPeriod), MaxPeriod: time.Duration(r.MaxPeriod), BurstGap: time.Duration(r.BurstGap), JitterAbs: time.Duration(r.JitterAbs), JitterRel: r.JitterRel, MaxMissing: r.MaxMissing, MinPulses: r.MinPulses, HistoryLen: r.HistoryLen, MinCoverage: r.MinCoverage, MaxJitterFraction: r.MaxJitterFraction, MaxChance: r.MaxChance, EndAfterMissedPulses: r.EndAfterMissedPulses, State: r.State.params()}) } return cfg, cfg.Validate() } + +// parseExcludeChannels decodes the JSON form of +// PeriodicRule.ExcludeChannels: plain lower-case hex, one entry per channel +// hash. An empty list gives a nil slice, which excludes nothing. +func parseExcludeChannels(ss []string) ([]ChannelHash, error) { + if len(ss) == 0 { + return nil, nil + } + out := make([]ChannelHash, 0, len(ss)) + for _, s := range ss { + if s == "" || strings.ToLower(s) != s { + return nil, errors.New("anomaly: exclude_channels: want lower-case hex") + } + b, err := hex.DecodeString(s) + if err != nil { + return nil, errors.New("anomaly: exclude_channels: bad hex") + } + c, err := NewChannelHash(b) + if err != nil { + return nil, err + } + out = append(out, c) + } + return out, nil +}