From eed29fd55a64eabd4483e81cd4fbaab17222abff Mon Sep 17 00:00:00 2001 From: dborup Date: Thu, 15 Oct 2026 20:28:42 +0200 Subject: [PATCH 01/12] feat(anomaly): channel-bot target, channel exclusions and injection-based evaluation MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Relates to #474, phases 1-3. Library and offline tool only: no ingestor, server, API or UI wiring, no defaults and no thresholds. Phase 1 (target). doc.go and the tool README state the periodic rule's target: an abusive channel bot, not expected automation. #457 round 4 measured wardriving trips, which is the opposite. Phase 2 (detector). PeriodicRule.ExcludeChannels lists channel hashes a periodic rule ignores; an excluded event never reaches the rule, so it forms no pulse, chain or episode and claims no key state. Opt-in, no defaults, validated (canonical, non-zero, no duplicates, bounded), matched on the whole hash (length included), and exposed to the replay JSON as "exclude_channels". 1-byte hash collisions are documented, not hidden. Phase 3 (evaluation). Two new offline subcommands: - inject overlays synthetic bot trains on exported background lines. Ids are "inj---", a value no export can produce; a background line already carrying that prefix is refused. File-only (no -db flag, no database), deterministic from the spec seed with an in-tool splitmix64, per-bot streams seeded by name, background copied through byte for byte, merged by data time. The labels file applies the #457 §4 recipe to the emitted pulses. - score measures a report against those labels with the #457 §5 matching, or without labels, which is the false-alarm arm. Nothing injected may ever be written into a database. --- cmd/anomaly/README.md | 170 +++++- cmd/anomaly/inject.go | 700 ++++++++++++++++++++++ cmd/anomaly/inject_test.go | 586 ++++++++++++++++++ cmd/anomaly/main.go | 16 +- cmd/anomaly/score.go | 450 ++++++++++++++ cmd/anomaly/score_test.go | 437 ++++++++++++++ internal/anomaly/config.go | 69 +++ internal/anomaly/detector.go | 19 +- internal/anomaly/doc.go | 22 + internal/anomaly/exclude_channels_test.go | 289 +++++++++ internal/anomaly/replay.go | 40 +- 11 files changed, 2785 insertions(+), 13 deletions(-) create mode 100644 cmd/anomaly/inject.go create mode 100644 cmd/anomaly/inject_test.go create mode 100644 cmd/anomaly/score.go create mode 100644 cmd/anomaly/score_test.go create mode 100644 internal/anomaly/exclude_channels_test.go diff --git a/cmd/anomaly/README.md b/cmd/anomaly/README.md index bc7691c5f..029c9fbf4 100644 --- a/cmd/anomaly/README.md +++ b/cmd/anomaly/README.md @@ -1,9 +1,12 @@ # corescope-anomaly (experimental, offline) -Offline tool for the `internal/anomaly` study (#457). It is not part of the -server or the ingestor, is not built into any image and has no defaults: +Offline tool for the `internal/anomaly` study (#457, #474). It is not part of +the server or the ingestor, is not built into any image and has no defaults: every bound is an explicit flag. +Only `export` reads a database, read-only. Nothing here ever writes to a +CoreScope database: `inject` and `score` read and write plain files. + ```sh cd cmd/anomaly && go build -o corescope-anomaly . ``` @@ -110,5 +113,164 @@ corescope-anomaly replay -config config.json -input events.jsonl \ ``` The config is the replay JSON config (`anomaly.ParseReplayConfig`). Periodic -rules accept the optional `end_after_missed_pulses` (train tracking, #447). -Leaving it out means 0, which turns tracking off. +rules accept two optional fields: + +- `end_after_missed_pulses` — train tracking (#447). Leaving it out means 0, + which turns tracking off. +- `exclude_channels` — channel hashes the rule ignores, as plain lower-case + hex (`["81"]`). Leaving it out excludes nothing. See + [Excluded channels](#excluded-channels). + +## Excluded channels + +The periodic rule's target is an abusive channel bot (#474). Automation that +is *expected* on its channel is not a target: a wardriving app posts a +position every 75-123 s while someone drives, and the #457 round-4 labels +show that is what the rule was measuring. + +`exclude_channels` names the hashes whose automation is expected. An event on +such a channel never reaches that rule, so it forms no pulse, no chain and no +episode, and the channel claims no key state. + +Two properties worth stating plainly: + +- **A channel hash is not a channel.** MeshCore's hash is one byte, so + excluding the hash of `#wardriving` also excludes every other channel name + that hashes to the same byte (about 1/256 of all names), including any spam + on them. That is the price of keying on an opaque 1-byte hash. +- **The match is on the whole hash**, length included: the 1-byte hash `81` + and the 2-byte hash `81ab` are different entries, and a 1-byte entry is + never read as the prefix of a wider hash. + +There are no default exclusions anywhere: an unconfigured rule behaves +exactly as it did before #474, and the replay golden file is unchanged. + +## inject + +Overlays synthetic channel-bot trains on exported background lines, so recall +can be measured when real spam is rare. It writes two new files and nothing +else. + +```sh +corescope-anomaly inject -spec spec.json -input events.jsonl \ + -out events-injected.jsonl -labels labels.json \ + -max-injected 100000 -max-line-bytes 65536 +``` + +- **Clearly synthetic.** Every injected line's `id` is + `inj---`; an export numbers its lines `tx1`, `tx2`, …, so + no real line can carry that marker. A background line that already starts + with `inj-` is refused, so the marker stays unambiguous. +- **Never written back.** The tool takes no `-db` flag and opens no database. + Injected events exist only in the output file; they must never be imported + into any CoreScope database. +- **File-only and deterministic.** Same spec, same seed, same background → + byte-identical output and labels. The generator is splitmix64 inside this + tool, not `math/rand`, and each bot's stream is seeded from the spec seed + and the bot's name, so adding, removing or reordering bots never changes + another bot's events. +- **Background untouched.** Background lines are copied through byte for + byte, in order; injected lines are merged in by data time, so the output is + a valid replay input. +- `-out` and `-labels` must not exist, and neither appears unless the whole + run succeeds. `-max-injected` is a hard bound. +- The summary (counts and times only) is printed as JSON on stderr. + +### Spec + +Every field is required; there are no defaults. Unknown fields are rejected. +The example is the shape of the September bot documented in #130 (5 min, +~3 s jitter, ~30 % multi-part pulses with later parts up to 55 s behind, ~5 % +single missed pulses, one 35-minute hole, one stable first hop). + +```json +{ + "seed": 20261015, + "final_after": "30s", + "labelling": {"pulse_merge_gap": "60s", "min_period": "1m", "max_period": "1h", + "min_pulses": 8, "max_harmonic": 3, + "tolerance_abs": "30s", "tolerance_rel": 0.1}, + "bots": [{ + "name": "sept-shape", + "channel": "7f", + "payload_type": 5, + "route_class": "flood", + "route_kind": "relay", + "first_hop": "1:fd", + "full_route": "1:fd00", + "wire_bytes": 60, + "payload_bytes": 35, + "start": "2026-10-16T00:00:00Z", + "stop": "2026-10-19T13:12:00Z", + "period": "5m", + "jitter_sd": "3s", + "jitter_max": "10s", + "missed_fraction": 0.05, + "multipart_fraction": 0.3, + "multipart_max_extra": 2, + "multipart_max_delay": "55s", + "holes": [{"start": "2026-10-17T04:00:00Z", "duration": "35m"}] + }] +} +``` + +- `final_after` is added to each event time to give its `final_at`; set it to + the export's `-settle`, so injected lines are final like the background's. +- `jitter_sd` is the standard deviation of the offset from the nominal grid + time, clamped to `±jitter_max`. The offset is not cumulative, so the train + does not drift. +- `missed_fraction` drops pulses other than the first; `multipart_fraction` + gives a pulse 1..`multipart_max_extra` later parts within + `multipart_max_delay`. A hole drops every pulse inside it. +- `channel`, `first_hop`, `full_route`, `payload_type` and the sizes are the + wire fields of a replay line: an injected line carries nothing a real line + does not. + +### Labels + +`-labels` holds the ground truth of the run, computed from the emitted pulses +with the `labelling` recipe, which is the #457 §4 periodic recipe: + +1. events closer than `pulse_merge_gap` are one pulse, timed at its first + event; +2. P is the median pulse gap inside `[min_period, max_period]`; +3. an incident is a maximal run of at least `min_pulses` pulses whose every + gap is within `max(tolerance_abs, tolerance_rel * P)` of k·P for some + k in 1..`max_harmonic`. + +So a bot with a long hole is two incidents, and a bot with fewer than +`min_pulses` pulses is none — the ground truth is what the recipe says, not +"one bot, one incident". The file holds channel hashes, times, periods and +counts: no ids, routes, hops, observers, names or text. + +## score + +Measures a replay report against injected labels, or against no labels at +all. + +```sh +corescope-anomaly score -report report.json -labels labels.json \ + -rule p474_periodic -start 2026-10-16T00:00:00Z -end 2026-10-23T00:00:00Z \ + [-out metrics.json] +``` + +The matching is the #457 §5 periodic rule: + +- An **episode** is one `EventID`. Its activation is its first transition to + `active`; its end is its last transition to `ended`, or the report's end if + it never ended. Only episodes activated inside `[-start, -end)` are scored. +- An episode is a **true positive** when the channel hashes are equal, + `[activation, end]` overlaps `[incident start, stop + 4P]`, and the + episode's period is within the labelling tolerance of k·P for some + k in 1..`max_harmonic`. An episode whose activation carries no periodic + evidence can never be a true positive. +- Every other episode is a **false positive**. Without `-labels` every + episode is one, which is the false-alarm arm: run it on background-only + traffic to get false alarms per day. + +The metrics are episodes, true and false positives, false alarms per day, +recall, median time to detect (also in units of P), median stop latency, +median episodes per detected incident, the per-incident detail and the false +positives per channel hash. Times to detect and stop latencies are properties +of the rule and the bot's shape, not of the scorer: a noisy train is found +much later than a clean one. diff --git a/cmd/anomaly/inject.go b/cmd/anomaly/inject.go new file mode 100644 index 000000000..cf30730eb --- /dev/null +++ b/cmd/anomaly/inject.go @@ -0,0 +1,700 @@ +package main + +import ( + "bufio" + "encoding/hex" + "encoding/json" + "errors" + "flag" + "fmt" + "hash/fnv" + "io" + "math" + "os" + "sort" + "strings" + "time" + + "github.com/meshcore-analyzer/anomaly" +) + +// Issue #474, phase 3: overlay synthetic channel-bot trains on a background +// of real exported events, so recall can be measured when real spam is rare. +// +// The tool is offline and file-only. It opens no database, takes no -db +// flag, touches no network and reads no wall clock: it reads a spec and a +// background JSONL file and writes a new JSONL file plus a labels file. +// Nothing it produces is ever written back to a CoreScope database. +// +// Every injected event is marked: its id is "inj---", a +// value no export can produce (export ids are tx1, tx2, ...), and the tool +// refuses a background line whose id already starts with that prefix. A +// replay report therefore names injected traffic in its TriggerIDs, and the +// labels file says exactly which incidents were injected. + +// injPrefix marks every injected event id. +const injPrefix = "inj-" + +// maxInjectBots and maxInjectHoles bound one spec. +const ( + maxInjectBots = 64 + maxInjectHoles = 64 +) + +// injectSpec is the JSON spec of one injection run. There are no defaults: +// every field must be present, and fields whose zero value is meaningful +// (seed, payload type, jitter, fractions) are pointers so that leaving one +// out is an error rather than a silent zero. +type injectSpec struct { + Seed *uint64 `json:"seed"` + // FinalAfter is added to each injected event time to give its final_at, + // as the export's -settle does for real events. + FinalAfter *dur `json:"final_after"` + Labelling labellingSpec `json:"labelling"` + Bots []injectBot `json:"bots"` +} + +// labellingSpec is the ground-truth recipe applied to the injected pulses +// (#457 §4, periodic): events closer than PulseMergeGap are one pulse, P is +// the median pulse gap inside [MinPeriod, MaxPeriod], and an incident is a +// maximal run of at least MinPulses pulses whose every gap is within +// max(ToleranceAbs, ToleranceRel*P) of k*P for some k in 1..MaxHarmonic. +type labellingSpec struct { + PulseMergeGap dur `json:"pulse_merge_gap"` + MinPeriod dur `json:"min_period"` + MaxPeriod dur `json:"max_period"` + MinPulses int `json:"min_pulses"` + MaxHarmonic int `json:"max_harmonic"` + ToleranceAbs dur `json:"tolerance_abs"` + ToleranceRel float64 `json:"tolerance_rel"` +} + +// injectBot describes one synthetic bot. Times are data times, as in the +// export; the fields that describe the wire are exactly the fields of a +// replay record, so an injected line carries nothing a real line does not. +type injectBot struct { + Name string `json:"name"` + Channel string `json:"channel"` + PayloadType *int `json:"payload_type"` + RouteClass string `json:"route_class"` + RouteKind string `json:"route_kind"` + FirstHop string `json:"first_hop"` + FullRoute string `json:"full_route"` + WireBytes int `json:"wire_bytes"` + PayloadBytes *int `json:"payload_bytes"` + Start time.Time `json:"start"` + Stop time.Time `json:"stop"` + Period dur `json:"period"` + // JitterSD is the standard deviation of the per-pulse offset from the + // nominal grid time; JitterMax clamps it. The offset is not cumulative, + // so the train does not drift. + JitterSD *dur `json:"jitter_sd"` + JitterMax dur `json:"jitter_max"` + // MissedFraction is the probability that a pulse other than the first is + // dropped (one missed pulse, as a real bot's). + MissedFraction *float64 `json:"missed_fraction"` + // MultipartFraction is the probability that a pulse is multi-part; + // 1..MultipartMaxExtra later parts then follow it within + // MultipartMaxDelay. + MultipartFraction *float64 `json:"multipart_fraction"` + MultipartMaxExtra *int `json:"multipart_max_extra"` + MultipartMaxDelay dur `json:"multipart_max_delay"` + // Holes are data-time ranges where the bot sends nothing (an outage). + Holes []injectHole `json:"holes"` +} + +type injectHole struct { + Start time.Time `json:"start"` + Duration dur `json:"duration"` +} + +// dur is a JSON duration written and read as a string ("5m0s"), so a spec +// and the labels file it produces round-trip through the same form. +type dur time.Duration + +func (d dur) MarshalJSON() ([]byte, error) { return json.Marshal(time.Duration(d).String()) } + +func (d *dur) UnmarshalJSON(b []byte) error { + var s string + if err := json.Unmarshal(b, &s); err != nil { + return errors.New("durations are strings such as \"5m\"") + } + v, err := time.ParseDuration(s) + if err != nil { + return err + } + *d = dur(v) + return nil +} + +// prng is splitmix64: a fixed, self-contained generator, so an injection is +// reproducible from the spec alone and independent of the Go version's +// math/rand. +type prng struct{ s uint64 } + +func (p *prng) next() uint64 { + p.s += 0x9e3779b97f4a7c15 + z := p.s + z = (z ^ (z >> 30)) * 0xbf58476d1ce4e5b9 + z = (z ^ (z >> 27)) * 0x94d049bb133111eb + return z ^ (z >> 31) +} + +// float returns a uniform value in [0,1). +func (p *prng) float() float64 { return float64(p.next()>>11) / (1 << 53) } + +// normal returns a standard normal value (Box-Muller, cosine branch). +func (p *prng) normal() float64 { + u1 := 1 - p.float() // in (0,1] + u2 := p.float() + return math.Sqrt(-2*math.Log(u1)) * math.Cos(2*math.Pi*u2) +} + +// botPRNG seeds one bot's stream from the spec seed and the bot's name, so +// each bot's output depends only on the seed and its own fields: adding, +// removing or reordering bots never changes another bot's events. +func botPRNG(seed uint64, name string) *prng { + h := fnv.New64a() + _, _ = h.Write([]byte(name)) + p := &prng{s: seed ^ h.Sum64()} + p.next() // discard one value, so a zero seed is not a degenerate start + return p +} + +// botStats counts what one bot produced. +type botStats struct { + Name string `json:"name"` + Channel string `json:"channel"` + PulsesScheduled int `json:"pulses_scheduled"` + PulsesEmitted int `json:"pulses_emitted"` + PulsesMissed int `json:"pulses_missed"` + PulsesInHole int `json:"pulses_in_hole"` + MultipartPulses int `json:"multipart_pulses"` + Events int `json:"events"` + First time.Time `json:"first,omitempty"` + Last time.Time `json:"last,omitempty"` + Incidents int `json:"incidents"` +} + +// injectSummary is printed on stderr (counts and times only). +type injectSummary struct { + Experimental bool `json:"experimental"` + BackgroundLines int `json:"background_lines"` + BackgroundOutOfOrder int `json:"background_out_of_order"` + InjectedEvents int `json:"injected_events"` + LinesWritten int `json:"lines_written"` + Bots []botStats `json:"bots"` +} + +// Incident is one labelled ground-truth incident: a channel hash and a time +// range, with the bot's period. It holds no key material, no observer +// identity and no text. +type Incident struct { + Bot string `json:"bot"` + Channel string `json:"channel"` + Start time.Time `json:"start"` + Stop time.Time `json:"stop"` + Period dur `json:"period"` + Pulses int `json:"pulses"` +} + +// InjectLabels is the labels file: the ground truth of one injection run. +type InjectLabels struct { + Experimental bool `json:"experimental"` + Synthetic bool `json:"synthetic"` + Seed uint64 `json:"seed"` + Labelling labellingSpec `json:"labelling"` + Bots []botStats `json:"bots"` + Incidents []Incident `json:"incidents"` +} + +// injEvent is one generated event before it becomes a line. +type injEvent struct { + rec anomaly.ReplayRecord + t int64 // rec.Time in nanoseconds, for sorting + pulse int +} + +func runInject(args []string, stdin io.Reader, stderr io.Writer) int { + fs := flag.NewFlagSet("inject", flag.ContinueOnError) + fs.SetOutput(stderr) + specPath := fs.String("spec", "", "injection spec JSON (required)") + input := fs.String("input", "", "background ReplayRecord JSON Lines, or - for stdin (required; /dev/null for none)") + out := fs.String("out", "", "merged JSON Lines output (required, must not exist)") + labels := fs.String("labels", "", "ground-truth labels JSON (required, must not exist)") + maxInjected := fs.Int("max-injected", 0, "most injected events allowed (required)") + maxLine := fs.Int("max-line-bytes", 0, "longest accepted background line (required)") + if err := fs.Parse(args); err != nil { + return exitUsage + } + usageErr := func(err error) int { + fmt.Fprintf(stderr, "inject: %v\n", err) + return exitUsage + } + if fs.NArg() != 0 { + return usageErr(errors.New("unexpected arguments")) + } + if *specPath == "" || *input == "" || *out == "" || *labels == "" { + return usageErr(errors.New("-spec, -input, -out and -labels are required")) + } + if *maxInjected <= 0 || *maxLine <= 0 { + return usageErr(errors.New("-max-injected and -max-line-bytes must be > 0")) + } + fail := func(err error) int { + fmt.Fprintf(stderr, "inject: %v\n", err) + return exitFail + } + sb, err := os.ReadFile(*specPath) + if err != nil { + return fail(err) + } + spec, err := parseInjectSpec(sb) + if err != nil { + return fail(err) + } + evs, stats, incidents, err := generate(spec, *maxInjected) + if err != nil { + return fail(err) + } + in := stdin + if *input != "-" { + f, err := os.Open(*input) + if err != nil { + return fail(err) + } + defer f.Close() + in = f + } + // Both outputs must be new files, and neither appears unless the whole + // run succeeds. + outF, err := createNew(*out) + if err != nil { + return fail(err) + } + sum, err := mergeLines(in, outF, evs, *maxLine) + if cerr := outF.Close(); err == nil { + err = cerr + } + if err != nil { + os.Remove(*out) + return fail(err) + } + sum.Bots = stats + lb, err := json.MarshalIndent(InjectLabels{Experimental: true, Synthetic: true, Seed: *spec.Seed, + Labelling: spec.Labelling, Bots: stats, Incidents: incidents}, "", " ") + if err == nil { + err = writeNew(*labels, append(lb, '\n')) + } + if err != nil { + os.Remove(*out) + return fail(err) + } + b, err := json.Marshal(sum) + if err != nil { + return fail(err) + } + fmt.Fprintf(stderr, "%s\n", b) + return exitOK +} + +// parseInjectSpec decodes and validates a spec (unknown fields and trailing +// data rejected). +func parseInjectSpec(b []byte) (injectSpec, error) { + var s injectSpec + dec := json.NewDecoder(strings.NewReader(string(b))) + dec.DisallowUnknownFields() + if err := dec.Decode(&s); err != nil { + return injectSpec{}, fmt.Errorf("inject spec: %w", err) + } + if !onlyJSONValue(dec, b) { + return injectSpec{}, errors.New("inject spec: trailing data") + } + if s.Seed == nil { + return injectSpec{}, errors.New("inject spec: seed is required") + } + if s.FinalAfter == nil || *s.FinalAfter < 0 || time.Duration(*s.FinalAfter) > time.Hour { + return injectSpec{}, errors.New("inject spec: final_after is required and must be in 0..1h") + } + if err := s.Labelling.validate(); err != nil { + return injectSpec{}, err + } + if len(s.Bots) == 0 || len(s.Bots) > maxInjectBots { + return injectSpec{}, fmt.Errorf("inject spec: need 1..%d bots", maxInjectBots) + } + seen := map[string]bool{} + for i := range s.Bots { + if err := s.Bots[i].validate(); err != nil { + return injectSpec{}, err + } + if seen[s.Bots[i].Name] { + return injectSpec{}, fmt.Errorf("inject spec: duplicate bot name %q", s.Bots[i].Name) + } + seen[s.Bots[i].Name] = true + } + return s, nil +} + +func (l labellingSpec) validate() error { + switch { + case time.Duration(l.PulseMergeGap) <= 0: + return errors.New("inject spec: labelling.pulse_merge_gap must be > 0") + case time.Duration(l.MinPeriod) <= 0 || time.Duration(l.MaxPeriod) <= time.Duration(l.MinPeriod): + return errors.New("inject spec: labelling needs 0 < min_period < max_period") + case l.MinPulses < 2: + return errors.New("inject spec: labelling.min_pulses must be >= 2") + case l.MaxHarmonic < 1 || l.MaxHarmonic > 16: + return errors.New("inject spec: labelling.max_harmonic must be in 1..16") + case time.Duration(l.ToleranceAbs) < 0: + return errors.New("inject spec: labelling.tolerance_abs must be >= 0") + case !(l.ToleranceRel >= 0 && l.ToleranceRel <= 1): + return errors.New("inject spec: labelling.tolerance_rel must be in 0..1") + case time.Duration(l.ToleranceAbs) == 0 && l.ToleranceRel == 0: + return errors.New("inject spec: labelling needs a tolerance") + } + return nil +} + +// validBotName allows the characters a rule name allows, so an injected id +// stays printable and greppable. +func validBotName(s string) bool { + if len(s) == 0 || len(s) > 32 { + return false + } + for _, c := range s { + if !(c >= 'a' && c <= 'z' || c >= '0' && c <= '9' || c == '_' || c == '-' || c == '.') { + return false + } + } + return true +} + +func (b *injectBot) validate() error { + name := b.Name + bad := func(what string) error { return fmt.Errorf("inject spec: bot %q: %s", name, what) } + if !validBotName(b.Name) { + return fmt.Errorf("inject spec: bot names must be 1-32 chars of [a-z0-9_.-], got %q", b.Name) + } + if b.Channel == "" || strings.ToLower(b.Channel) != b.Channel { + return bad("channel must be lower-case hex") + } + if _, err := hex.DecodeString(b.Channel); err != nil { + return bad("channel must be hex") + } + switch { + case b.PayloadType == nil: + return bad("payload_type is required") + case b.MissedFraction == nil || *b.MissedFraction < 0 || *b.MissedFraction > 1: + return bad("missed_fraction is required and must be in 0..1") + case b.MultipartFraction == nil || *b.MultipartFraction < 0 || *b.MultipartFraction > 1: + return bad("multipart_fraction is required and must be in 0..1") + case b.MultipartMaxExtra == nil || *b.MultipartMaxExtra < 0 || *b.MultipartMaxExtra > 8: + return bad("multipart_max_extra is required and must be in 0..8") + case b.JitterSD == nil || *b.JitterSD < 0: + return bad("jitter_sd is required and must be >= 0") + case b.WireBytes <= 0: + return bad("wire_bytes must be > 0") + case b.Start.IsZero() || b.Stop.IsZero() || !b.Stop.After(b.Start): + return bad("start and stop are required, with stop after start") + case time.Duration(b.Period) <= 0: + return bad("period must be > 0") + case time.Duration(b.JitterMax) < 0 || time.Duration(b.JitterMax) >= time.Duration(b.Period)/2: + return bad("jitter_max must be in [0, period/2)") + case time.Duration(b.MultipartMaxDelay) < 0: + return bad("multipart_max_delay must be >= 0") + case *b.MultipartFraction > 0 && (*b.MultipartMaxExtra == 0 || time.Duration(b.MultipartMaxDelay) == 0): + return bad("multipart_fraction needs multipart_max_extra and multipart_max_delay") + case len(b.Holes) > maxInjectHoles: + return bad("too many holes") + } + for _, h := range b.Holes { + if h.Start.IsZero() || time.Duration(h.Duration) <= 0 { + return bad("every hole needs a start and a duration > 0") + } + } + return nil +} + +// generate builds every injected event, the per-bot counts and the labelled +// incidents. It fails, without producing anything, if the spec would emit +// more than maxInjected events. +func generate(s injectSpec, maxInjected int) ([]injEvent, []botStats, []Incident, error) { + var evs []injEvent + stats := make([]botStats, 0, len(s.Bots)) + var incidents []Incident + for i := range s.Bots { + b := &s.Bots[i] + botEvs, st, err := generateBot(b, *s.Seed, time.Duration(*s.FinalAfter), maxInjected-len(evs)) + if err != nil { + return nil, nil, nil, err + } + inc := labelIncidents(b.Name, b.Channel, botEvs, s.Labelling) + st.Incidents = len(inc) + evs = append(evs, botEvs...) + stats = append(stats, st) + incidents = append(incidents, inc...) + } + // One deterministic order for the output: time, then id. + sort.SliceStable(evs, func(i, j int) bool { + if evs[i].t != evs[j].t { + return evs[i].t < evs[j].t + } + return evs[i].rec.ID < evs[j].rec.ID + }) + return evs, stats, incidents, nil +} + +// generateBot draws one bot's pulse train. Draw order per scheduled pulse is +// fixed: the missed-pulse draw (except for the first pulse), the jitter draw, +// the multi-part draw, then one draw per extra part. A pulse dropped by an +// earlier draw consumes no later draw. +func generateBot(b *injectBot, seed uint64, finalAfter time.Duration, budget int) ([]injEvent, botStats, error) { + p := botPRNG(seed, b.Name) + st := botStats{Name: b.Name, Channel: b.Channel} + period := time.Duration(b.Period) + var evs []injEvent + emit := func(pulse, part int, when time.Time) error { + if len(evs) >= budget { + return fmt.Errorf("inject: more than -max-injected events; raise the bound or shorten the spec") + } + rec := anomaly.ReplayRecord{ + ID: fmt.Sprintf("%s%s-%d-%d", injPrefix, b.Name, pulse, part), Time: when.UTC(), + FinalAt: when.Add(finalAfter).UTC(), PayloadType: *b.PayloadType, Channel: b.Channel, + RouteClass: b.RouteClass, RouteKind: b.RouteKind, FirstHop: b.FirstHop, FullRoute: b.FullRoute, + WireBytes: b.WireBytes, Traffic: "unclassified", Decryption: "unknown", + } + if b.PayloadBytes != nil { + pb := *b.PayloadBytes + rec.PayloadBytes = &pb + } + // An injected line must be exactly as valid as an exported one. + if _, err := rec.Event(); err != nil { + return fmt.Errorf("inject: bot %q: %w", b.Name, err) + } + evs = append(evs, injEvent{rec: rec, t: rec.Time.UnixNano(), pulse: pulse}) + return nil + } + for i := 0; ; i++ { + nominal := b.Start.Add(time.Duration(i) * period) + if !nominal.Before(b.Stop) { + break + } + st.PulsesScheduled++ + if i > 0 && p.float() < *b.MissedFraction { + st.PulsesMissed++ + continue + } + if inHole(b.Holes, nominal) { + st.PulsesInHole++ + continue + } + off := time.Duration(p.normal() * float64(time.Duration(*b.JitterSD))) + if lim := time.Duration(b.JitterMax); off > lim { + off = lim + } else if off < -lim { + off = -lim + } + when := nominal.Add(off) + if err := emit(i, 0, when); err != nil { + return nil, botStats{}, err + } + st.PulsesEmitted++ + if p.float() < *b.MultipartFraction { + extra := 1 + int(p.float()*float64(*b.MultipartMaxExtra)) + st.MultipartPulses++ + for part := 1; part <= extra; part++ { + d := time.Duration(p.float() * float64(time.Duration(b.MultipartMaxDelay))) + if err := emit(i, part, when.Add(d)); err != nil { + return nil, botStats{}, err + } + } + } + } + st.Events = len(evs) + if len(evs) > 0 { + first, last := evs[0].rec.Time, evs[0].rec.Time + for _, e := range evs { + if e.rec.Time.Before(first) { + first = e.rec.Time + } + if e.rec.Time.After(last) { + last = e.rec.Time + } + } + st.First, st.Last = first, last + } + return evs, st, nil +} + +func inHole(holes []injectHole, t time.Time) bool { + for _, h := range holes { + if !t.Before(h.Start) && t.Before(h.Start.Add(time.Duration(h.Duration))) { + return true + } + } + return false +} + +// labelIncidents applies the labelling recipe to one bot's emitted events. +func labelIncidents(bot, channel string, evs []injEvent, l labellingSpec) []Incident { + if len(evs) == 0 { + return nil + } + times := make([]int64, 0, len(evs)) + for _, e := range evs { + times = append(times, e.t) + } + sort.Slice(times, func(i, j int) bool { return times[i] < times[j] }) + // Events closer than the merge gap are one pulse, timed at its first + // event (as the detector's BurstGap does). + pulses := []int64{times[0]} + for _, t := range times[1:] { + if t-pulses[len(pulses)-1] >= int64(time.Duration(l.PulseMergeGap)) { + pulses = append(pulses, t) + } + } + if len(pulses) < 2 { + return nil + } + gaps := make([]int64, 0, len(pulses)-1) + inRange := make([]int64, 0, len(pulses)-1) + for i := 1; i < len(pulses); i++ { + g := pulses[i] - pulses[i-1] + gaps = append(gaps, g) + if g >= int64(time.Duration(l.MinPeriod)) && g <= int64(time.Duration(l.MaxPeriod)) { + inRange = append(inRange, g) + } + } + if len(inRange) == 0 { + return nil + } + period := medianInt64(inRange) + tol := int64(float64(period) * l.ToleranceRel) + if a := int64(time.Duration(l.ToleranceAbs)); a > tol { + tol = a + } + fits := func(g int64) bool { + for k := 1; k <= l.MaxHarmonic; k++ { + d := g - int64(k)*period + if d < 0 { + d = -d + } + if d <= tol { + return true + } + } + return false + } + var out []Incident + // A run is a maximal stretch of pulses whose every gap fits. + runStart := 0 + flush := func(end int) { // pulses[runStart:end+1] is the run + if end-runStart+1 >= l.MinPulses { + out = append(out, Incident{Bot: bot, Channel: channel, Start: time.Unix(0, pulses[runStart]).UTC(), + Stop: time.Unix(0, pulses[end]).UTC(), Period: dur(period), Pulses: end - runStart + 1}) + } + } + for i, g := range gaps { + if !fits(g) { + flush(i) + runStart = i + 1 + } + } + flush(len(pulses) - 1) + return out +} + +// medianInt64 returns the median of a non-empty slice: the middle value, or +// the mean of the two middle values (truncated) for an even count. It sorts +// a copy, so the caller's slice order is kept. +func medianInt64(v []int64) int64 { + s := append([]int64(nil), v...) + sort.Slice(s, func(i, j int) bool { return s[i] < s[j] }) + n := len(s) + if n%2 == 1 { + return s[n/2] + } + return s[n/2-1] + (s[n/2]-s[n/2-1])/2 +} + +// backgroundLine is the minimum a background line must tell us: its time, to +// merge on, and its id, to keep the injected marker unambiguous. Unknown +// fields are allowed here: the line is copied through byte for byte. +type backgroundLine struct { + ID string `json:"id"` + Time time.Time `json:"time"` +} + +// mergeLines writes the injected events and the background lines to w in +// data-time order. Background lines are copied verbatim, so the background +// is never reformatted; an injected event goes out before the first +// background line that is not earlier than it. +func mergeLines(r io.Reader, w io.Writer, evs []injEvent, maxLine int) (injectSummary, error) { + sum := injectSummary{Experimental: true, InjectedEvents: len(evs)} + bw := bufio.NewWriter(w) + next := 0 + writeInjected := func(upto int64, all bool) error { + for next < len(evs) && (all || evs[next].t <= upto) { + b, err := json.Marshal(evs[next].rec) + if err != nil { + return err + } + if _, err := bw.Write(append(b, '\n')); err != nil { + return err + } + next++ + sum.LinesWritten++ + } + return nil + } + sc := bufio.NewScanner(r) + limit := maxLine + 2 + initial := 4096 + if limit < initial { + initial = limit + } + sc.Buffer(make([]byte, 0, initial), limit) + var prev int64 + havePrev := false + for sc.Scan() { + line := sc.Bytes() + if len(line) > maxLine { + return sum, errors.New("inject: background line longer than -max-line-bytes") + } + if len(strings.TrimSpace(string(line))) == 0 { + continue + } + sum.BackgroundLines++ + var bl backgroundLine + if err := json.Unmarshal(line, &bl); err != nil || bl.Time.IsZero() { + return sum, fmt.Errorf("inject: background line %d: need a JSON object with an RFC 3339 \"time\"", sum.BackgroundLines) + } + if strings.HasPrefix(bl.ID, injPrefix) { + return sum, fmt.Errorf("inject: background line %d already carries an injected id (%s...): refusing to mix", sum.BackgroundLines, injPrefix) + } + t := bl.Time.UnixNano() + if havePrev && t < prev { + sum.BackgroundOutOfOrder++ + } + prev, havePrev = t, true + if err := writeInjected(t, false); err != nil { + return sum, err + } + if _, err := bw.Write(append(append([]byte(nil), line...), '\n')); err != nil { + return sum, err + } + sum.LinesWritten++ + } + if err := sc.Err(); err != nil { + return sum, fmt.Errorf("inject: background input: %w", err) + } + if err := writeInjected(0, true); err != nil { + return sum, err + } + return sum, bw.Flush() +} + +// onlyJSONValue reports whether the decoder consumed all of b but trailing +// whitespace. +func onlyJSONValue(dec *json.Decoder, b []byte) bool { + return len(strings.TrimSpace(string(b[dec.InputOffset():]))) == 0 +} diff --git a/cmd/anomaly/inject_test.go b/cmd/anomaly/inject_test.go new file mode 100644 index 000000000..66e57e9ec --- /dev/null +++ b/cmd/anomaly/inject_test.go @@ -0,0 +1,586 @@ +package main + +import ( + "bytes" + "encoding/json" + "fmt" + "os" + "path/filepath" + "sort" + "strings" + "testing" + "time" + + "github.com/meshcore-analyzer/anomaly" +) + +// Issue #474, phase 3: injection-based evaluation. Every bot, channel byte +// and time here is synthetic. + +var injT = time.Date(2030, 3, 1, 0, 0, 0, 0, time.UTC) + +func injAt(d time.Duration) string { return injT.Add(d).UTC().Format(time.RFC3339) } + +// specOpts are the knobs the tests vary; everything else is fixed below. +type specOpts struct { + seed uint64 + channel string + period string + stopAfter time.Duration + jitterSD string + missedFraction float64 + multipartFraction float64 + holes string // JSON array body, e.g. {"start":..,"duration":"35m"} + extraBots string // JSON for further bots, with a leading comma + leadBots string // JSON for bots before bot-a, with a trailing comma + minPulses int +} + +func (o specOpts) with() specOpts { + if o.channel == "" { + o.channel = "7f" + } + if o.period == "" { + o.period = "5m" + } + if o.stopAfter == 0 { + o.stopAfter = 4 * time.Hour + } + if o.jitterSD == "" { + o.jitterSD = "0s" + } + if o.holes == "" { + o.holes = "[]" + } + if o.minPulses == 0 { + o.minPulses = 8 + } + return o +} + +// specJSON is the one spec shape the tests use, with the September bot's +// shape as its default (5 min, 3 s jitter, multi-part pulses). +func specJSON(o specOpts) string { + o = o.with() + return fmt.Sprintf(`{ + "seed": %d, + "final_after": "30s", + "labelling": {"pulse_merge_gap": "60s", "min_period": "1m", "max_period": "1h", + "min_pulses": %d, "max_harmonic": 3, "tolerance_abs": "30s", "tolerance_rel": 0.1}, + "bots": [%s{ + "name": "bot-a", + "channel": %q, + "payload_type": 5, + "route_class": "flood", + "route_kind": "relay", + "first_hop": "1:fd", + "full_route": "1:fd00", + "wire_bytes": 60, + "payload_bytes": 35, + "start": %q, + "stop": %q, + "period": %q, + "jitter_sd": %q, + "jitter_max": "10s", + "missed_fraction": %v, + "multipart_fraction": %v, + "multipart_max_extra": 2, + "multipart_max_delay": "55s", + "holes": %s + }%s] +}`, o.seed, o.minPulses, o.leadBots, o.channel, injAt(0), injAt(o.stopAfter), o.period, o.jitterSD, + o.missedFraction, o.multipartFraction, o.holes, o.extraBots) +} + +// backgroundJSONL builds a sparse, non-periodic background: n events on +// channel 11 at irregular gaps, through two first hops. +func backgroundJSONL(n int) string { + var b strings.Builder + step := []time.Duration{37 * time.Second, 4 * time.Minute, 71 * time.Second, 13 * time.Minute} + hops := []string{"1:0b", "2:0c01"} + t := injT + for i := 0; i < n; i++ { + t = t.Add(step[i%len(step)]) + pb := 23 + rec := anomaly.ReplayRecord{ID: fmt.Sprintf("tx%d", i+1), Time: t, FinalAt: t.Add(30 * time.Second), + PayloadType: 5, Channel: "11", RouteClass: "flood", RouteKind: "relay", + FirstHop: hops[i%len(hops)], FullRoute: hops[i%len(hops)] + "00", WireBytes: 55, + PayloadBytes: &pb, Traffic: "unclassified", Decryption: "unknown"} + j, err := json.Marshal(rec) + if err != nil { + panic(err) + } + b.Write(j) + b.WriteByte('\n') + } + return b.String() +} + +// injectRun writes the spec and the background into a fresh directory, runs +// inject and returns the output lines, the labels and the summary. +type injectResult struct { + dir string + outPath string + labPath string + lines []string + labels InjectLabels + summary injectSummary + stderr string +} + +func injectInto(t *testing.T, spec, background string, extra ...string) injectResult { + t.Helper() + res, code := tryInject(t, spec, background, extra...) + if code != 0 { + t.Fatalf("inject exit %d: %s", code, res.stderr) + } + return res +} + +func tryInject(t *testing.T, spec, background string, extra ...string) (injectResult, int) { + t.Helper() + dir := t.TempDir() + r := injectResult{dir: dir, outPath: filepath.Join(dir, "events.jsonl"), labPath: filepath.Join(dir, "labels.json")} + specPath := filepath.Join(dir, "spec.json") + bgPath := filepath.Join(dir, "background.jsonl") + if err := os.WriteFile(specPath, []byte(spec), 0o644); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(bgPath, []byte(background), 0o644); err != nil { + t.Fatal(err) + } + args := append([]string{"inject", "-spec", specPath, "-input", bgPath, "-out", r.outPath, + "-labels", r.labPath, "-max-injected", "100000", "-max-line-bytes", "65536"}, extra...) + var stdout, stderr bytes.Buffer + code := run(args, nil, &stdout, &stderr) + r.stderr = stderr.String() + if stdout.Len() != 0 { + t.Fatalf("inject wrote %d bytes to stdout", stdout.Len()) + } + if code != 0 { + return r, code + } + out, err := os.ReadFile(r.outPath) + if err != nil { + t.Fatal(err) + } + r.lines = strings.Split(strings.TrimRight(string(out), "\n"), "\n") + lb, err := os.ReadFile(r.labPath) + if err != nil { + t.Fatal(err) + } + if err := json.Unmarshal(lb, &r.labels); err != nil { + t.Fatal(err) + } + if err := json.Unmarshal([]byte(r.stderr), &r.summary); err != nil { + t.Fatalf("summary %q: %v", r.stderr, err) + } + return r, code +} + +// injectedLines returns the injected lines of a run, parsed. +func injectedLines(t *testing.T, r injectResult) []anomaly.ReplayRecord { + t.Helper() + var out []anomaly.ReplayRecord + for _, l := range r.lines { + var rec anomaly.ReplayRecord + if err := json.Unmarshal([]byte(l), &rec); err != nil { + t.Fatalf("line %q: %v", l, err) + } + if strings.HasPrefix(rec.ID, injPrefix) { + out = append(out, rec) + } + } + return out +} + +// Every injected line is marked, unique, valid as an event, and carries +// exactly the configured wire fields. The background is copied through byte +// for byte, and the merged output is in data-time order. +func TestInjectMarksAndMergesWithoutTouchingTheBackground(t *testing.T) { + bg := backgroundJSONL(40) + r := injectInto(t, specJSON(specOpts{seed: 1, multipartFraction: 0.3}), bg) + ids := map[string]bool{} + var last time.Time + injected, passed := 0, 0 + var bgLines []string + for _, l := range r.lines { + var rec anomaly.ReplayRecord + if err := json.Unmarshal([]byte(l), &rec); err != nil { + t.Fatalf("line %q: %v", l, err) + } + if ids[rec.ID] { + t.Fatalf("duplicate id %s", rec.ID) + } + ids[rec.ID] = true + if rec.Time.Before(last) { + t.Fatalf("output is not in data-time order at %s", rec.ID) + } + last = rec.Time + if !strings.HasPrefix(rec.ID, injPrefix) { + passed++ + bgLines = append(bgLines, l) + continue + } + injected++ + // The marker is the literal "inj---": an export + // numbers its lines tx1, tx2, ..., so no real line can carry it. + if !strings.HasPrefix(rec.ID, "inj-bot-a-") { + t.Fatalf("injected id %q is not marked inj---", rec.ID) + } + if _, err := rec.Event(); err != nil { + t.Fatalf("injected %s is not a valid event: %v", rec.ID, err) + } + if rec.Channel != "7f" || rec.FirstHop != "1:fd" || rec.FullRoute != "1:fd00" || + rec.PayloadType != 5 || rec.WireBytes != 60 || rec.PayloadBytes == nil || *rec.PayloadBytes != 35 { + t.Fatalf("injected %s has unexpected wire fields: %+v", rec.ID, rec) + } + if got := rec.FinalAt.Sub(rec.Time); got != 30*time.Second { + t.Fatalf("injected %s final_at offset %v", rec.ID, got) + } + } + if passed != 40 || injected != r.summary.InjectedEvents || injected < 48 { + t.Fatalf("passed %d background lines, injected %d (summary %d)", passed, injected, r.summary.InjectedEvents) + } + if strings.Join(bgLines, "\n")+"\n" != bg { + t.Fatal("the background lines were not copied through byte for byte") + } + if r.summary.BackgroundLines != 40 || r.summary.LinesWritten != 40+injected || r.summary.BackgroundOutOfOrder != 0 { + t.Fatalf("summary %+v", r.summary) + } +} + +// The same spec, seed and background give byte-identical output and labels. +func TestInjectIsDeterministic(t *testing.T) { + spec := specJSON(specOpts{seed: 7, jitterSD: "3s", missedFraction: 0.05, multipartFraction: 0.3}) + bg := backgroundJSONL(20) + a := injectInto(t, spec, bg) + b := injectInto(t, spec, bg) + ab, _ := os.ReadFile(a.outPath) + bb, _ := os.ReadFile(b.outPath) + if !bytes.Equal(ab, bb) { + t.Fatal("two runs of the same spec differ") + } + al, _ := os.ReadFile(a.labPath) + bl, _ := os.ReadFile(b.labPath) + if !bytes.Equal(al, bl) { + t.Fatal("two runs of the same spec produced different labels") + } + // A different seed moves the pulses but keeps the schedule. + c := injectInto(t, specJSON(specOpts{seed: 8, jitterSD: "3s", missedFraction: 0.05, multipartFraction: 0.3}), bg) + cb, _ := os.ReadFile(c.outPath) + if bytes.Equal(ab, cb) { + t.Fatal("a different seed gave identical output") + } + if a.summary.Bots[0].PulsesScheduled != c.summary.Bots[0].PulsesScheduled { + t.Fatalf("the schedule changed with the seed: %d vs %d", + a.summary.Bots[0].PulsesScheduled, c.summary.Bots[0].PulsesScheduled) + } +} + +// A bot's events depend only on the seed and its own fields, so adding a +// second bot leaves the first one's events untouched. +func TestInjectBotsAreIndependent(t *testing.T) { + botB := `{ + "name": "bot-b", "channel": "22", "payload_type": 5, "route_class": "flood", "route_kind": "relay", + "first_hop": "1:ee", "full_route": "1:ee00", "wire_bytes": 60, "payload_bytes": 35, + "start": "` + injAt(0) + `", "stop": "` + injAt(time.Hour) + `", "period": "7m", + "jitter_sd": "2s", "jitter_max": "10s", "missed_fraction": 0, "multipart_fraction": 0, + "multipart_max_extra": 2, "multipart_max_delay": "55s", "holes": []}` + linesOf := func(r injectResult, prefix string) string { + var out []string + for _, rec := range injectedLines(t, r) { + if strings.HasPrefix(rec.ID, prefix) { + b, _ := json.Marshal(rec) + out = append(out, string(b)) + } + } + if len(out) == 0 { + t.Fatalf("no injected lines with prefix %s", prefix) + } + return strings.Join(out, "\n") + } + alone := injectInto(t, specJSON(specOpts{seed: 3, jitterSD: "3s"}), "") + after := injectInto(t, specJSON(specOpts{seed: 3, jitterSD: "3s", extraBots: "," + botB}), "") + // bot-a's events must not depend on its position in the spec either. + before := injectInto(t, specJSON(specOpts{seed: 3, jitterSD: "3s", leadBots: botB + ","}), "") + want := linesOf(alone, injPrefix+"bot-a-") + if got := linesOf(after, injPrefix+"bot-a-"); got != want { + t.Fatal("adding a second bot after bot-a changed bot-a's events") + } + if got := linesOf(before, injPrefix+"bot-a-"); got != want { + t.Fatal("adding a bot before bot-a changed bot-a's events") + } + if len(after.labels.Bots) != 2 || after.labels.Bots[1].Name != "bot-b" { + t.Fatalf("labels %+v", after.labels.Bots) + } +} + +// The labelling recipe is applied to the emitted pulses: a clean train is +// one incident, a hole longer than max_harmonic periods splits it, and a +// train shorter than min_pulses is no incident at all. +func TestInjectLabelsIncidentsFromThePulses(t *testing.T) { + clean := injectInto(t, specJSON(specOpts{seed: 11}), "") + if len(clean.labels.Incidents) != 1 { + t.Fatalf("clean train: %d incidents, want 1", len(clean.labels.Incidents)) + } + inc := clean.labels.Incidents[0] + if inc.Channel != "7f" || inc.Bot != "bot-a" || time.Duration(inc.Period) != 5*time.Minute || inc.Pulses != 48 { + t.Fatalf("incident %+v", inc) + } + if !inc.Start.Equal(injT) || !inc.Stop.Equal(injT.Add(235*time.Minute)) { + t.Fatalf("incident span %s..%s", inc.Start, inc.Stop) + } + // A 35-minute hole is 7 periods: more than max_harmonic, so the gap + // across it fits no k*P and the run is split in two. + holed := injectInto(t, specJSON(specOpts{seed: 11, + holes: `[{"start": "` + injAt(100*time.Minute) + `", "duration": "35m"}]`}), "") + if len(holed.labels.Incidents) != 2 { + t.Fatalf("holed train: %d incidents, want 2: %+v", len(holed.labels.Incidents), holed.labels.Incidents) + } + if holed.summary.Bots[0].PulsesInHole != 7 { + t.Fatalf("pulses in hole: %d, want 7", holed.summary.Bots[0].PulsesInHole) + } + // Seven pulses are fewer than min_pulses 8: no incident, but the events + // are still injected. + short := injectInto(t, specJSON(specOpts{seed: 11, stopAfter: 35 * time.Minute}), "") + if len(short.labels.Incidents) != 0 || short.summary.Bots[0].PulsesEmitted != 7 { + t.Fatalf("short train: %d incidents, %d pulses", len(short.labels.Incidents), short.summary.Bots[0].PulsesEmitted) + } +} + +// Multi-part pulses are extra events inside the merge gap, so they add +// events without adding pulses or incidents. +func TestInjectMultipartPulsesStayOnePulse(t *testing.T) { + r := injectInto(t, specJSON(specOpts{seed: 5, multipartFraction: 1}), "") + st := r.summary.Bots[0] + if st.MultipartPulses != st.PulsesEmitted || st.Events <= st.PulsesEmitted { + t.Fatalf("stats %+v", st) + } + if len(r.labels.Incidents) != 1 || r.labels.Incidents[0].Pulses != st.PulsesEmitted { + t.Fatalf("incidents %+v for %d pulses", r.labels.Incidents, st.PulsesEmitted) + } + // Every extra part lands within multipart_max_delay of its pulse. + byPulse := map[string][]time.Time{} + for _, rec := range injectedLines(t, r) { + p := rec.ID[:strings.LastIndex(rec.ID, "-")] + byPulse[p] = append(byPulse[p], rec.Time) + } + for p, ts := range byPulse { + if len(ts) < 2 { + t.Fatalf("pulse %s has %d parts", p, len(ts)) + } + sort.Slice(ts, func(i, j int) bool { return ts[i].Before(ts[j]) }) + if d := ts[len(ts)-1].Sub(ts[0]); d > 55*time.Second { + t.Fatalf("pulse %s spans %v", p, d) + } + } + // With the fraction at 0 there are no extra parts at all. + none := injectInto(t, specJSON(specOpts{seed: 5}), "") + if st := none.summary.Bots[0]; st.MultipartPulses != 0 || st.Events != st.PulsesEmitted { + t.Fatalf("multipart_fraction 0 produced parts: %+v", st) + } +} + +// missed_fraction drops pulses, and never the first one. +func TestInjectMissedPulses(t *testing.T) { + all := injectInto(t, specJSON(specOpts{seed: 9, missedFraction: 1}), "") + st := all.summary.Bots[0] + if st.PulsesEmitted != 1 || st.PulsesMissed != st.PulsesScheduled-1 { + t.Fatalf("missed_fraction 1: %+v", st) + } + if len(all.labels.Incidents) != 0 { + t.Fatalf("a single pulse is no incident: %+v", all.labels.Incidents) + } + some := injectInto(t, specJSON(specOpts{seed: 9, missedFraction: 0.5}), "") + if st := some.summary.Bots[0]; st.PulsesMissed == 0 || st.PulsesEmitted == st.PulsesScheduled { + t.Fatalf("missed_fraction 0.5: %+v", st) + } +} + +// Jitter stays inside jitter_max and does not accumulate: a pulse is always +// within the clamp of its nominal grid time. +func TestInjectJitterIsClampedAndDoesNotDrift(t *testing.T) { + r := injectInto(t, specJSON(specOpts{seed: 13, jitterSD: "8s"}), "") + seen := false + for _, rec := range injectedLines(t, r) { + var pulse, part int + if _, err := fmt.Sscanf(strings.TrimPrefix(rec.ID, injPrefix+"bot-a-"), "%d-%d", &pulse, &part); err != nil { + t.Fatalf("id %s: %v", rec.ID, err) + } + off := rec.Time.Sub(injT.Add(time.Duration(pulse) * 5 * time.Minute)) + if off > 10*time.Second || off < -10*time.Second { + t.Fatalf("pulse %d is %v off its nominal time", pulse, off) + } + if off != 0 { + seen = true + } + } + if !seen { + t.Fatal("jitter_sd 8s produced no offset at all") + } +} + +// The tool writes only the two files it was given, refuses to overwrite +// either of them, and leaves nothing behind when it fails. +func TestInjectWritesOnlyItsTwoFiles(t *testing.T) { + spec := specJSON(specOpts{seed: 2}) + r := injectInto(t, spec, backgroundJSONL(5)) + ents, err := os.ReadDir(r.dir) + if err != nil { + t.Fatal(err) + } + var names []string + for _, e := range ents { + names = append(names, e.Name()) + } + sort.Strings(names) + want := []string{"background.jsonl", "events.jsonl", "labels.json", "spec.json"} + if strings.Join(names, ",") != strings.Join(want, ",") { + t.Fatalf("directory holds %v", names) + } + // Running again in the same directory refuses to overwrite. + args := []string{"inject", "-spec", filepath.Join(r.dir, "spec.json"), "-input", filepath.Join(r.dir, "background.jsonl"), + "-out", r.outPath, "-labels", r.labPath, "-max-injected", "100000", "-max-line-bytes", "65536"} + var stdout, stderr bytes.Buffer + if code := run(args, nil, &stdout, &stderr); code != exitFail { + t.Fatalf("overwriting -out: exit %d", code) + } + // A labels file that already exists leaves no half-written output. + out2 := filepath.Join(r.dir, "events2.jsonl") + args[6] = out2 + if code := run(args, nil, &stdout, &stderr); code != exitFail { + t.Fatalf("overwriting -labels: exit %d", code) + } + if _, err := os.Stat(out2); !os.IsNotExist(err) { + t.Fatalf("a failed run left %s behind (%v)", out2, err) + } +} + +// -max-injected is a hard bound: exceeding it fails the run and writes +// nothing. +func TestInjectMaxInjectedIsEnforced(t *testing.T) { + r, code := tryInject(t, specJSON(specOpts{seed: 4}), "", "-max-injected", "10") + if code != exitFail { + t.Fatalf("exit %d, want %d", code, exitFail) + } + if !strings.Contains(r.stderr, "max-injected") { + t.Fatalf("stderr %q", r.stderr) + } + if _, err := os.Stat(r.outPath); !os.IsNotExist(err) { + t.Fatal("output file written despite the bound") + } + if _, err := os.Stat(r.labPath); !os.IsNotExist(err) { + t.Fatal("labels file written despite the bound") + } +} + +// A background line that already carries an injected id is refused, so the +// marker stays unambiguous. +func TestInjectRefusesPreInjectedBackground(t *testing.T) { + bg := backgroundJSONL(2) + `{"id":"inj-bot-a-0-0","time":"` + injAt(time.Hour) + + `","payload_type":5,"channel":"7f","route_class":"flood","route_kind":"relay","first_hop":"1:fd","full_route":"1:fd00","wire_bytes":60,"traffic":"unclassified","decryption":"unknown"}` + "\n" + r, code := tryInject(t, specJSON(specOpts{seed: 4}), bg) + if code != exitFail || !strings.Contains(r.stderr, "injected id") { + t.Fatalf("exit %d, stderr %q", code, r.stderr) + } +} + +// A background line without a usable time is an error, not a silently +// misplaced line. +func TestInjectRefusesUnreadableBackgroundLine(t *testing.T) { + for _, bad := range []string{"not json", `{"id":"tx9"}`, `{"id":"tx9","time":"yesterday"}`} { + r, code := tryInject(t, specJSON(specOpts{seed: 4}), bad+"\n") + if code != exitFail { + t.Fatalf("%q: exit %d, stderr %q", bad, code, r.stderr) + } + } +} + +// Required flags are required; -spec, -input, -out, -labels, -max-injected +// and -max-line-bytes all have no default. +func TestInjectRequiresExplicitFlags(t *testing.T) { + dir := t.TempDir() + specPath := filepath.Join(dir, "spec.json") + bgPath := filepath.Join(dir, "bg.jsonl") + if err := os.WriteFile(specPath, []byte(specJSON(specOpts{seed: 1})), 0o644); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(bgPath, nil, 0o644); err != nil { + t.Fatal(err) + } + full := []string{"inject", "-spec", specPath, "-input", bgPath, "-out", filepath.Join(dir, "o.jsonl"), + "-labels", filepath.Join(dir, "l.json"), "-max-injected", "1000", "-max-line-bytes", "4096"} + for i := 1; i < len(full); i += 2 { + args := append(append([]string{}, full[:i]...), full[i+2:]...) + var stdout, stderr bytes.Buffer + if code := run(args, nil, &stdout, &stderr); code != exitUsage { + t.Fatalf("without %s: exit %d", full[i], code) + } + } + var stdout, stderr bytes.Buffer + if code := run(append(append([]string{}, full...), "extra"), nil, &stdout, &stderr); code != exitUsage { + t.Fatalf("with a positional argument: exit %d", code) + } +} + +// The spec has no defaults and no room for anything the tool does not know: +// a missing required field, a field whose zero value would be silently +// assumed, and an unknown field are all errors. +func TestInjectSpecRejectsMissingAndUnknownFields(t *testing.T) { + base := specJSON(specOpts{seed: 1, multipartFraction: 0.3}) + for _, key := range []string{`"seed": 1,`, `"final_after": "30s",`, `"payload_type": 5,`, + `"jitter_sd": "0s",`, `"missed_fraction": 0,`, `"multipart_fraction": 0.3,`, + `"multipart_max_extra": 2,`, `"wire_bytes": 60,`, `"period": "5m",`} { + if !strings.Contains(base, key) { + t.Fatalf("test fixture no longer contains %s", key) + } + if _, code := tryInject(t, strings.Replace(base, key, "", 1), ""); code != exitFail { + t.Errorf("without %s: exit %d", key, code) + } + } + for name, spec := range map[string]string{ + "unknown field": strings.Replace(base, `"seed": 1,`, `"seed": 1, "db": "/tmp/meshcore.db",`, 1), + "trailing data": base + "\n{}", + "no bots": strings.Replace(base, `"bots": [{`, `"bots2": [], "bots": [{`, 1), + "bad channel": strings.Replace(base, `"channel": "7f"`, `"channel": "7F"`, 1), + "bad bot name": strings.Replace(base, `"name": "bot-a"`, `"name": "Bot A"`, 1), + "stop before start": strings.Replace(base, `"stop": "`+injAt(4*time.Hour)+`"`, + `"stop": "`+injAt(-time.Hour)+`"`, 1), + "jitter at half the period": strings.Replace(base, `"jitter_max": "10s"`, `"jitter_max": "150s"`, 1), + "bad labelling": strings.Replace(base, `"min_pulses": 8`, `"min_pulses": 1`, 1), + "duplicate bot": strings.Replace(base, `"holes": [] + }]`, `"holes": [] + }, {"name": "bot-a", "channel": "22", "payload_type": 5, "route_class": "flood", "route_kind": "relay", + "first_hop": "1:ee", "full_route": "1:ee00", "wire_bytes": 60, "payload_bytes": 35, + "start": "`+injAt(0)+`", "stop": "`+injAt(time.Hour)+`", "period": "7m", "jitter_sd": "0s", + "jitter_max": "10s", "missed_fraction": 0, "multipart_fraction": 0, "multipart_max_extra": 2, + "multipart_max_delay": "55s", "holes": []}]`, 1), + } { + if _, code := tryInject(t, spec, ""); code != exitFail { + t.Errorf("%s: exit %d", name, code) + } + } +} + +// The labels file carries the ground truth and nothing else: it is marked +// synthetic, and it holds no ids, routes, hops, observers or text. +func TestInjectLabelsAreKeyFreeAndMarkedSynthetic(t *testing.T) { + r := injectInto(t, specJSON(specOpts{seed: 6, multipartFraction: 0.3}), backgroundJSONL(5)) + if !r.labels.Experimental || !r.labels.Synthetic || r.labels.Seed != 6 { + t.Fatalf("labels header %+v", r.labels) + } + raw, err := os.ReadFile(r.labPath) + if err != nil { + t.Fatal(err) + } + for _, banned := range []string{"1:fd", "fd00", "observer", "tx1", injPrefix, "raw_hex", "public_key"} { + if bytes.Contains(raw, []byte(banned)) { + t.Fatalf("labels file carries %q:\n%s", banned, raw) + } + } + // It does carry the channel hash, the times and the period. + if !bytes.Contains(raw, []byte(`"channel": "7f"`)) || !bytes.Contains(raw, []byte(`"period": "5m0s"`)) { + t.Fatalf("labels file is missing the ground truth:\n%s", raw) + } +} diff --git a/cmd/anomaly/main.go b/cmd/anomaly/main.go index 31e102baa..0a24e8197 100644 --- a/cmd/anomaly/main.go +++ b/cmd/anomaly/main.go @@ -4,12 +4,18 @@ // // corescope-anomaly export -db PATH [-immutable] -start T -end T -settle D -max-rows N -out FILE // corescope-anomaly replay -config FILE -input FILE|- -start T -end T -max-line-bytes N [-out FILE] +// corescope-anomaly inject -spec FILE -input FILE|- -out FILE -labels FILE -max-injected N -max-line-bytes N +// corescope-anomaly score -report FILE [-labels FILE] -rule NAME -start T -end T [-out FILE] // // export reads a CoreScope SQLite database read-only (mode=ro; -immutable for // a static snapshot copy) and writes anomaly.ReplayRecord JSON Lines built // from on-wire fields only, with per-export IDs. replay runs anomaly.Replay -// over such lines and prints the deterministic report. Every bound is an -// explicit flag; there are no defaults. +// over such lines and prints the deterministic report. inject overlays +// synthetic channel-bot trains (clearly marked "inj-..." ids) on such lines +// and writes the ground truth beside them; score measures a replay report +// against that ground truth (#474). Only export reads a database, and +// nothing here ever writes to one. Every bound is an explicit flag; there +// are no defaults. package main import ( @@ -23,6 +29,8 @@ import ( const usage = `usage: corescope-anomaly export -db PATH [-immutable] -start RFC3339 -end RFC3339 -settle DURATION -max-rows N -out FILE corescope-anomaly replay -config FILE -input FILE|- -start RFC3339 -end RFC3339 -max-line-bytes N [-out FILE] + corescope-anomaly inject -spec FILE -input FILE|- -out FILE -labels FILE -max-injected N -max-line-bytes N + corescope-anomaly score -report FILE [-labels FILE] -rule NAME -start RFC3339 -end RFC3339 [-out FILE] ` // Exit codes: 0 success, 1 runtime failure, 2 usage error. @@ -46,6 +54,10 @@ func run(args []string, stdin io.Reader, stdout, stderr io.Writer) int { return runExport(args[1:], stderr) case "replay": return runReplay(args[1:], stdin, stdout, stderr) + case "inject": + return runInject(args[1:], stdin, stderr) + case "score": + return runScore(args[1:], stdout, stderr) } fmt.Fprint(stderr, usage) return exitUsage diff --git a/cmd/anomaly/score.go b/cmd/anomaly/score.go new file mode 100644 index 000000000..33e3d8a0b --- /dev/null +++ b/cmd/anomaly/score.go @@ -0,0 +1,450 @@ +package main + +import ( + "bytes" + "encoding/json" + "errors" + "flag" + "fmt" + "io" + "os" + "sort" + "time" + + "github.com/meshcore-analyzer/anomaly" +) + +// Issue #474, phase 3: score a replay report of the periodic rule against +// the injected ground truth (or against no ground truth at all, which +// measures false alarms on real background traffic). +// +// The matching and the metrics are the #457 §5 periodic rules, with the +// labels coming from `inject` instead of from a hand-labelled channel: +// +// - An episode is one EventID. Its activation is its first transition to +// active; its end is its last transition to ended, or the report's end if +// it never ended. Only episodes activated inside [-start, -end) count. +// - An episode is a true positive for an incident when the channel hashes +// are equal, [activation, end] overlaps [incident start, stop + 4P], and +// the episode's period is within the labelling tolerance of k*P for some +// k in 1..max_harmonic. An episode whose activation carries no periodic +// evidence can never be a true positive. +// - Every other episode is a false positive. With no labels file, every +// episode is a false positive, which is the false-alarm arm. +// +// Scoring reads files only: a report, a labels file and nothing else. + +// scoreCandidate is the part of a report candidate scoring needs. The report +// is this tool's own output, so unknown fields are ignored rather than +// rejected; the fields below are checked instead. +type scoreCandidate struct { + EventID string + Rule string + Kind string + Key string + To string + At time.Time + Periodic *struct { + Period time.Duration + } `json:",omitempty"` +} + +type scoreReport struct { + Experimental bool + Start, End time.Time + Candidates []scoreCandidate +} + +// episode is one EventID's activation and end. +type episode struct { + id string + channel string + activation time.Time + end time.Time + hasPeriod bool + period time.Duration + matches int + // nearMiss: the episode overlapped an incident on the same channel but + // failed the period check. + nearMiss bool +} + +// incidentScore reports one labelled incident's outcome. +type incidentScore struct { + Bot string `json:"bot"` + Channel string `json:"channel"` + Start string `json:"start"` + Stop string `json:"stop"` + Period dur `json:"period"` + Episodes int `json:"episodes"` + Detected bool `json:"detected"` + TimeToDetect *dur `json:"time_to_detect,omitempty"` + TTDPeriods float64 `json:"time_to_detect_periods,omitempty"` + StopLatency *dur `json:"stop_latency,omitempty"` +} + +// scoreResult is the metrics file. +type scoreResult struct { + Experimental bool `json:"experimental"` + Rule string `json:"rule"` + Start string `json:"start"` + End string `json:"end"` + Days float64 `json:"days"` + Labels string `json:"labels"` + Episodes int `json:"episodes"` + // EpisodesWithoutPeriod counts episodes whose activation candidate + // carried no periodic evidence; they can never be true positives. + EpisodesWithoutPeriod int `json:"episodes_without_period"` + TruePositives int `json:"true_positives"` + FalsePositives int `json:"false_positives"` + FalseAlarmsPerDay float64 `json:"false_alarms_per_day"` + // PeriodMismatches counts episodes that overlap an incident on the same + // channel but fail the period check (#457 "also reported"). + PeriodMismatches int `json:"period_mismatches"` + Incidents int `json:"incidents"` + IncidentsDetected int `json:"incidents_detected"` + Recall float64 `json:"recall"` + // Medians over the detected incidents. + MedianTimeToDetect *dur `json:"median_time_to_detect,omitempty"` + MedianTTDPeriods float64 `json:"median_time_to_detect_periods,omitempty"` + MedianStopLatency *dur `json:"median_stop_latency,omitempty"` + MedianEpisodesPerIncident float64 `json:"median_episodes_per_detected_incident,omitempty"` + PerIncident []incidentScore `json:"per_incident"` + // FalsePositivesByChannel lists the false positives per channel hash, + // sorted by hash: channel hashes and counts only. + FalsePositivesByChannel []channelCount `json:"false_positives_by_channel"` +} + +type channelCount struct { + Channel string `json:"channel"` + Count int `json:"count"` +} + +func runScore(args []string, stdout, stderr io.Writer) int { + fs := flag.NewFlagSet("score", flag.ContinueOnError) + fs.SetOutput(stderr) + report := fs.String("report", "", "replay report JSON (required)") + labels := fs.String("labels", "", "injected labels JSON; omitted means no ground truth (false alarms only)") + rule := fs.String("rule", "", "periodic rule name to score (required)") + start := fs.String("start", "", "RFC 3339 start of the evaluation window, inclusive (required)") + end := fs.String("end", "", "RFC 3339 end of the evaluation window, exclusive (required)") + out := fs.String("out", "", "metrics file (must not exist); default stdout") + if err := fs.Parse(args); err != nil { + return exitUsage + } + usageErr := func(err error) int { + fmt.Fprintf(stderr, "score: %v\n", err) + return exitUsage + } + if fs.NArg() != 0 { + return usageErr(errors.New("unexpected arguments")) + } + if *report == "" || *rule == "" { + return usageErr(errors.New("-report and -rule are required")) + } + var opt struct{ start, end time.Time } + var err error + if opt.start, err = parseTime("start", *start); err != nil { + return usageErr(err) + } + if opt.end, err = parseTime("end", *end); err != nil { + return usageErr(err) + } + if !opt.end.After(opt.start) { + return usageErr(errors.New("-end must be after -start")) + } + fail := func(err error) int { + fmt.Fprintf(stderr, "score: %v\n", err) + return exitFail + } + rb, err := os.ReadFile(*report) + if err != nil { + return fail(err) + } + rep, err := parseScoreReport(rb) + if err != nil { + return fail(err) + } + var lab InjectLabels + if *labels != "" { + lb, err := os.ReadFile(*labels) + if err != nil { + return fail(err) + } + if lab, err = parseInjectLabels(lb); err != nil { + return fail(err) + } + } + res, err := score(rep, lab, *rule, *labels, opt.start, opt.end) + if err != nil { + return fail(err) + } + b, err := json.MarshalIndent(res, "", " ") + if err != nil { + return fail(err) + } + b = append(b, '\n') + if *out == "" { + _, err = stdout.Write(b) + } else { + err = writeNew(*out, b) + } + if err != nil { + return fail(err) + } + return exitOK +} + +func parseScoreReport(b []byte) (scoreReport, error) { + var r scoreReport + if err := json.Unmarshal(b, &r); err != nil { + return scoreReport{}, fmt.Errorf("report: %w", err) + } + if !r.Experimental || r.Start.IsZero() || r.End.IsZero() || !r.End.After(r.Start) { + return scoreReport{}, errors.New("report: not a replay report (Experimental, Start, End)") + } + for i, c := range r.Candidates { + if c.EventID == "" || c.Rule == "" || c.Kind == "" || c.To == "" || c.At.IsZero() { + return scoreReport{}, fmt.Errorf("report: candidate %d is missing EventID, Rule, Kind, To or At", i) + } + if _, err := anomaly.ParseKey(c.Key); err != nil { + return scoreReport{}, fmt.Errorf("report: candidate %d: %w", i, err) + } + } + return r, nil +} + +func parseInjectLabels(b []byte) (InjectLabels, error) { + var l InjectLabels + dec := json.NewDecoder(bytes.NewReader(b)) + dec.DisallowUnknownFields() + if err := dec.Decode(&l); err != nil { + return InjectLabels{}, fmt.Errorf("labels: %w", err) + } + if !onlyJSONValue(dec, b) { + return InjectLabels{}, errors.New("labels: trailing data") + } + if !l.Synthetic { + return InjectLabels{}, errors.New("labels: not an injected labels file (synthetic)") + } + if err := l.Labelling.validate(); err != nil { + return InjectLabels{}, fmt.Errorf("labels: %w", err) + } + for i, inc := range l.Incidents { + if inc.Channel == "" || inc.Start.IsZero() || inc.Stop.IsZero() || inc.Stop.Before(inc.Start) || time.Duration(inc.Period) <= 0 { + return InjectLabels{}, fmt.Errorf("labels: incident %d is incomplete", i) + } + } + return l, nil +} + +// score computes the metrics. labelsPath is reported as provenance only. +func score(rep scoreReport, lab InjectLabels, rule, labelsPath string, start, end time.Time) (scoreResult, error) { + eps, noPeriod, err := episodes(rep, rule, start, end) + if err != nil { + return scoreResult{}, err + } + days := end.Sub(start).Hours() / 24 + res := scoreResult{Experimental: true, Rule: rule, Start: start.Format(time.RFC3339), + End: end.Format(time.RFC3339), Days: days, Labels: labelsPath, Episodes: len(eps), + EpisodesWithoutPeriod: noPeriod, Incidents: len(lab.Incidents), PerIncident: []incidentScore{}, + FalsePositivesByChannel: []channelCount{}} + l := lab.Labelling + var ttds, stops []int64 + var perInc []float64 + for _, inc := range lab.Incidents { + p := time.Duration(inc.Period) + is := incidentScore{Bot: inc.Bot, Channel: inc.Channel, Start: inc.Start.Format(time.RFC3339), + Stop: inc.Stop.Format(time.RFC3339), Period: inc.Period} + var firstAct, lastEnd time.Time + for i := range eps { + e := &eps[i] + if e.channel != inc.Channel || !overlaps(*e, inc, p) { + continue + } + if !e.hasPeriod { + continue // counted once, as EpisodesWithoutPeriod + } + if !periodFits(e.period, p, l) { + e.nearMiss = true + continue + } + e.matches++ + is.Episodes++ + if firstAct.IsZero() || e.activation.Before(firstAct) { + firstAct = e.activation + } + if lastEnd.IsZero() || e.end.After(lastEnd) { + lastEnd = e.end + } + } + if is.Episodes > 0 { + is.Detected = true + res.IncidentsDetected++ + ttd := dur(firstAct.Sub(inc.Start)) + sl := dur(lastEnd.Sub(inc.Stop)) + is.TimeToDetect, is.StopLatency = &ttd, &sl + is.TTDPeriods = float64(firstAct.Sub(inc.Start)) / float64(p) + ttds = append(ttds, int64(ttd)) + stops = append(stops, int64(sl)) + perInc = append(perInc, float64(is.Episodes)) + } + res.PerIncident = append(res.PerIncident, is) + } + byChannel := map[string]int{} + for _, e := range eps { + if e.matches == 0 && e.nearMiss { + res.PeriodMismatches++ + } + if e.matches > 0 { + res.TruePositives++ + continue + } + res.FalsePositives++ + byChannel[e.channel]++ + } + if days > 0 { + res.FalseAlarmsPerDay = float64(res.FalsePositives) / days + } + if res.Incidents > 0 { + res.Recall = float64(res.IncidentsDetected) / float64(res.Incidents) + } + if len(ttds) > 0 { + m := dur(medianInt64(ttds)) + res.MedianTimeToDetect = &m + res.MedianTTDPeriods = medianFloat(ttdPeriods(res.PerIncident)) + s := dur(medianInt64(stops)) + res.MedianStopLatency = &s + res.MedianEpisodesPerIncident = medianFloat(perInc) + } + for ch, n := range byChannel { + res.FalsePositivesByChannel = append(res.FalsePositivesByChannel, channelCount{Channel: ch, Count: n}) + } + sort.Slice(res.FalsePositivesByChannel, func(i, j int) bool { + return res.FalsePositivesByChannel[i].Channel < res.FalsePositivesByChannel[j].Channel + }) + return res, nil +} + +// ttdPeriods collects the time to detect in units of P over the detected +// incidents. +func ttdPeriods(per []incidentScore) []float64 { + var out []float64 + for _, p := range per { + if p.Detected { + out = append(out, p.TTDPeriods) + } + } + return out +} + +// episodes groups a report's candidates for one rule into episodes, keeping +// those activated inside [start, end). +func episodes(rep scoreReport, rule string, start, end time.Time) ([]episode, int, error) { + type agg struct { + channel string + act, endT time.Time + actSet, endSet bool + period time.Duration + hasPeriod bool + } + order := []string{} + m := map[string]*agg{} + for _, c := range rep.Candidates { + if c.Rule != rule || c.Kind != "periodic" { + continue + } + k, err := anomaly.ParseKey(c.Key) + if err != nil { + return nil, 0, fmt.Errorf("report: %w", err) + } + a := m[c.EventID] + if a == nil { + a = &agg{channel: hexChannel(k)} + m[c.EventID] = a + order = append(order, c.EventID) + } + switch c.To { + case "active": + // The first transition to active is the activation, and its + // candidate carries the periodic evidence that is scored. + if !a.actSet || c.At.Before(a.act) { + a.act, a.actSet = c.At, true + a.hasPeriod = c.Periodic != nil + if c.Periodic != nil { + a.period = c.Periodic.Period + } + } + case "ended": + if !a.endSet || c.At.After(a.endT) { + a.endT, a.endSet = c.At, true + } + } + } + var out []episode + noPeriod := 0 + for _, id := range order { + a := m[id] + if !a.actSet || a.act.Before(start) || !a.act.Before(end) { + continue + } + e := episode{id: id, channel: a.channel, activation: a.act, end: a.endT, + hasPeriod: a.hasPeriod, period: a.period} + if !a.endSet { + e.end = rep.End // never ended: the report's end, as #457 §1 says + } + if !e.hasPeriod { + noPeriod++ + } + out = append(out, e) + } + return out, noPeriod, nil +} + +// hexChannel returns a key's channel hash as plain lower-case hex, the form +// the labels file uses ("" when the key carries no channel). +func hexChannel(k anomaly.Key) string { + b := k.Channel.Bytes() + if len(b) == 0 { + return "" + } + return fmt.Sprintf("%x", b) +} + +// overlaps reports whether an episode's [activation, end] overlaps the +// incident's [start, stop + 4P]. +func overlaps(e episode, inc Incident, p time.Duration) bool { + return !e.activation.After(inc.Stop.Add(4*p)) && !e.end.Before(inc.Start) +} + +// periodFits reports whether the episode's period is within the labelling +// tolerance of k*P for some k in 1..MaxHarmonic. +func periodFits(got, p time.Duration, l labellingSpec) bool { + tol := time.Duration(float64(p) * l.ToleranceRel) + if a := time.Duration(l.ToleranceAbs); a > tol { + tol = a + } + for k := 1; k <= l.MaxHarmonic; k++ { + d := got - time.Duration(k)*p + if d < 0 { + d = -d + } + if d <= tol { + return true + } + } + return false +} + +func medianFloat(v []float64) float64 { + if len(v) == 0 { + return 0 + } + s := append([]float64(nil), v...) + sort.Float64s(s) + n := len(s) + if n%2 == 1 { + return s[n/2] + } + return (s[n/2-1] + s[n/2]) / 2 +} diff --git a/cmd/anomaly/score_test.go b/cmd/anomaly/score_test.go new file mode 100644 index 000000000..8526ed0d0 --- /dev/null +++ b/cmd/anomaly/score_test.go @@ -0,0 +1,437 @@ +package main + +import ( + "bytes" + "encoding/json" + "fmt" + "os" + "path/filepath" + "strings" + "testing" + "time" +) + +// Issue #474, phase 3: scoring a replay report against injected ground +// truth, and against no ground truth at all (the false-alarm arm). + +const scoreKey = "v1|s=stream|pt=5|ch=1:7f|rc=flood|rk=relay|hop=1:fd" + +// replayConfigJSON is the periodic rule the phase-3 runs use. exclude is +// spliced in, for example `"exclude_channels": ["7f"],`. +func replayConfigJSON(exclude string) string { + return `{ + "limits": {"max_keys_per_scope": 200000, "state_ttl": "48h", "dedup_horizon": "6h", "max_dedup_ids": 2000000, + "reorder_delay": "0s", "max_reorder_buffer": 64, "max_window_entries": 200000, + "max_candidates_per_call": 10000, "max_pending_candidates": 100000}, + "expected": "include", + "periodic": [{"name": "p474", "scope": "stream", "payload_types": [5], + "min_period": "1m", "max_period": "1h", "burst_gap": "15s", "jitter_abs": "10s", "jitter_rel": 0.02, + "max_missing": 2, "min_pulses": 8, "history_len": 32, "min_coverage": 0.6, "max_jitter_fraction": 0.5, + "max_chance": 0.01, "end_after_missed_pulses": 3, ` + exclude + ` + "state": {"confirm_after": "0s", "end_after_quiet": "60m", "cooldown": "0s"}}] +}` +} + +// replayInto runs the replay command over one input file and returns the +// report path. +func replayInto(t *testing.T, dir, name, config, input string) string { + t.Helper() + cfgPath := filepath.Join(dir, name+"-config.json") + if err := os.WriteFile(cfgPath, []byte(config), 0o644); err != nil { + t.Fatal(err) + } + report := filepath.Join(dir, name+"-report.json") + var stdout, stderr bytes.Buffer + code := run([]string{"replay", "-config", cfgPath, "-input", input, + "-start", injAt(0), "-end", injAt(24 * time.Hour), "-max-line-bytes", "65536", + "-out", report}, nil, &stdout, &stderr) + if code != 0 { + t.Fatalf("replay exit %d: %s", code, stderr.String()) + } + return report +} + +// scoreInto runs the score command and returns the parsed metrics. +func scoreInto(t *testing.T, report, labels string, extra ...string) scoreResult { + t.Helper() + args := []string{"score", "-report", report, "-rule", "p474", + "-start", injAt(0), "-end", injAt(24 * time.Hour)} + if labels != "" { + args = append(args, "-labels", labels) + } + var stdout, stderr bytes.Buffer + if code := run(append(args, extra...), nil, &stdout, &stderr); code != 0 { + t.Fatalf("score exit %d: %s", code, stderr.String()) + } + var res scoreResult + if err := json.Unmarshal(stdout.Bytes(), &res); err != nil { + t.Fatalf("metrics %q: %v", stdout.String(), err) + } + return res +} + +// End to end: inject a bot into a background, replay it and score it. The +// injected incident is detected, with the time to detect and the stop +// latency the pre-registration asks for, and the background raises no false +// alarm. +func TestScoreRecallOnAnInjectedBot(t *testing.T) { + bg := backgroundJSONL(300) + r := injectInto(t, specJSON(specOpts{seed: 21, jitterSD: "3s"}), bg) + if len(r.labels.Incidents) != 1 { + t.Fatalf("labels %+v", r.labels.Incidents) + } + report := replayInto(t, r.dir, "b", replayConfigJSON(""), r.outPath) + res := scoreInto(t, report, r.labPath) + if res.Incidents != 1 || res.IncidentsDetected != 1 || res.Recall != 1 { + t.Fatalf("recall: %+v", res) + } + if res.TruePositives == 0 || res.FalsePositives != 0 || res.FalseAlarmsPerDay != 0 { + t.Fatalf("precision side: %+v", res) + } + inc := res.PerIncident[0] + if inc.Channel != "7f" || !inc.Detected || inc.TimeToDetect == nil || inc.StopLatency == nil { + t.Fatalf("per-incident %+v", inc) + } + // The rule needs 8 pulses, so the first alarm cannot come before 7 + // periods; on a clean train it comes at 7 or 8. + if inc.TTDPeriods < 7 || inc.TTDPeriods > 12 { + t.Fatalf("time to detect %v (%.1f P)", time.Duration(*inc.TimeToDetect), inc.TTDPeriods) + } + // Train tracking (K = 3) ends the episode (K+1)*P after the last pulse + // at the latest. + if sl := time.Duration(*inc.StopLatency); sl < 0 || sl > 20*time.Minute { + t.Fatalf("stop latency %v", sl) + } + if res.MedianTimeToDetect == nil || res.MedianStopLatency == nil || res.MedianEpisodesPerIncident < 1 { + t.Fatalf("medians: %+v", res) + } + if res.Days != 1 { + t.Fatalf("days %v", res.Days) + } +} + +// A noisy bot of the September shape (5 % missed pulses, 30 % multi-part) is +// still found and still raises no false alarm, but how long that takes is a +// property of the rule, not of the scorer: this run only pins recall and +// logs the timing, which is what the evaluation measures. +func TestScoreRecallOnANoisyInjectedBot(t *testing.T) { + r := injectInto(t, specJSON(specOpts{seed: 21, jitterSD: "3s", missedFraction: 0.05, + multipartFraction: 0.3}), backgroundJSONL(300)) + res := scoreInto(t, replayInto(t, r.dir, "n", replayConfigJSON(""), r.outPath), r.labPath) + if res.Incidents != 1 || res.IncidentsDetected != 1 || res.FalsePositives != 0 { + t.Fatalf("noisy bot: %+v", res) + } + inc := res.PerIncident[0] + t.Logf("noisy bot: %d episodes, time to detect %v (%.1f P), stop latency %v", + inc.Episodes, time.Duration(*inc.TimeToDetect), inc.TTDPeriods, time.Duration(*inc.StopLatency)) +} + +// The same background without any injection is the false-alarm arm: no +// labels, and every episode would be a false alarm. This background raises +// none. +func TestScoreFalseAlarmArmWithoutLabels(t *testing.T) { + dir := t.TempDir() + bgPath := filepath.Join(dir, "bg.jsonl") + if err := os.WriteFile(bgPath, []byte(backgroundJSONL(300)), 0o644); err != nil { + t.Fatal(err) + } + report := replayInto(t, dir, "a", replayConfigJSON(""), bgPath) + res := scoreInto(t, report, "") + if res.Episodes != 0 || res.FalsePositives != 0 || res.FalseAlarmsPerDay != 0 || res.Incidents != 0 { + t.Fatalf("background raised alarms: %+v", res) + } + if res.Recall != 0 || res.MedianTimeToDetect != nil { + t.Fatalf("no labels must give no recall and no medians: %+v", res) + } +} + +// Phase 2 and phase 3 together: with the bot's channel hash excluded, the +// same injected traffic produces no periodic episode at all. +func TestScoreExcludedChannelDetectsNothing(t *testing.T) { + r := injectInto(t, specJSON(specOpts{seed: 21, jitterSD: "3s"}), backgroundJSONL(300)) + plain := scoreInto(t, replayInto(t, r.dir, "p", replayConfigJSON(""), r.outPath), r.labPath) + if plain.IncidentsDetected != 1 { + t.Fatalf("without exclusions the bot must be detected: %+v", plain) + } + excl := scoreInto(t, replayInto(t, r.dir, "x", replayConfigJSON(`"exclude_channels": ["7f"],`), r.outPath), r.labPath) + if excl.Episodes != 0 || excl.IncidentsDetected != 0 || excl.Recall != 0 { + t.Fatalf("the excluded channel still produced episodes: %+v", excl) + } + // Excluding another channel leaves the bot alone. + other := scoreInto(t, replayInto(t, r.dir, "o", replayConfigJSON(`"exclude_channels": ["11"],`), r.outPath), r.labPath) + if other.IncidentsDetected != 1 { + t.Fatalf("excluding channel 11 changed the bot's detection: %+v", other) + } +} + +// ---- crafted reports: the matching rules ---- + +// craftedReport writes a report with one episode per entry. +type craftedEpisode struct { + id string + key string + active time.Duration // activation, relative to injT + ended time.Duration // 0 = never ended + periodNS int64 // 0 = no periodic evidence on the activation +} + +func craftReport(t *testing.T, dir string, eps ...craftedEpisode) string { + t.Helper() + var cands []string + for _, e := range eps { + key := e.key + if key == "" { + key = scoreKey + } + per := "" + if e.periodNS != 0 { + per = fmt.Sprintf(`,"Periodic":{"Period":%d,"Tolerance":10000000000,"Support":8,"Coverage":1}`, e.periodNS) + } + cands = append(cands, fmt.Sprintf(`{"EventID":%q,"Rule":"p474","Kind":"periodic","Key":%q,`+ + `"From":"suspicious","To":"active","Reason":"confirmed","At":%q,"EpisodeStart":%q,`+ + `"Confidence":"route_observed","Traffic":"unclassified","Coverage":[]%s}`, + e.id, key, injT.Add(e.active).Format(time.RFC3339), injT.Add(e.active).Format(time.RFC3339), per)) + if e.ended != 0 { + cands = append(cands, fmt.Sprintf(`{"EventID":%q,"Rule":"p474","Kind":"periodic","Key":%q,`+ + `"From":"active","To":"ended","Reason":"quiet","At":%q,"EpisodeStart":%q,`+ + `"Confidence":"route_observed","Traffic":"unclassified","Coverage":[]}`, + e.id, key, injT.Add(e.ended).Format(time.RFC3339), injT.Add(e.active).Format(time.RFC3339))) + } + } + body := fmt.Sprintf(`{"Experimental":true,"Start":%q,"End":%q,"Input":{"Lines":1},"Candidates":[%s],"CoverageReduced":false}`, + injT.Format(time.RFC3339), injT.Add(24*time.Hour).Format(time.RFC3339), strings.Join(cands, ",")) + p := filepath.Join(dir, fmt.Sprintf("crafted-%d.json", len(eps)+int(time.Now().UnixNano()%1000))) + if err := os.WriteFile(p, []byte(body), 0o644); err != nil { + t.Fatal(err) + } + return p +} + +// craftLabels writes a labels file with one incident on channel 7f. +func craftLabels(t *testing.T, dir string, start, stop time.Duration, period time.Duration) string { + t.Helper() + l := InjectLabels{Experimental: true, Synthetic: true, Seed: 1, + Labelling: labellingSpec{PulseMergeGap: dur(time.Minute), MinPeriod: dur(time.Minute), + MaxPeriod: dur(time.Hour), MinPulses: 8, MaxHarmonic: 3, + ToleranceAbs: dur(30 * time.Second), ToleranceRel: 0.1}, + Incidents: []Incident{{Bot: "bot-a", Channel: "7f", Start: injT.Add(start), Stop: injT.Add(stop), + Period: dur(period), Pulses: 20}}} + b, err := json.MarshalIndent(l, "", " ") + if err != nil { + t.Fatal(err) + } + p := filepath.Join(dir, fmt.Sprintf("labels-%d.json", time.Now().UnixNano())) + if err := os.WriteFile(p, b, 0o644); err != nil { + t.Fatal(err) + } + return p +} + +// An episode is a true positive only if its period agrees with the +// incident's within the labelling tolerance of k*P, k in 1..max_harmonic. +func TestScorePeriodCheck(t *testing.T) { + dir := t.TempDir() + labels := craftLabels(t, dir, time.Hour, 2*time.Hour, 5*time.Minute) + for _, c := range []struct { + name string + periodNS int64 + tp bool + mismatch int + }{ + {"exact", int64(5 * time.Minute), true, 0}, + {"inside the 30 s tolerance", int64(5*time.Minute + 25*time.Second), true, 0}, + {"outside the tolerance", int64(5*time.Minute + 35*time.Second), false, 1}, + {"second harmonic", int64(10 * time.Minute), true, 0}, + {"third harmonic", int64(15 * time.Minute), true, 0}, + {"fourth harmonic", int64(20 * time.Minute), false, 1}, + {"unrelated", int64(99 * time.Second), false, 1}, + {"no periodic evidence", 0, false, 0}, + } { + report := craftReport(t, dir, craftedEpisode{id: "e1", active: 90 * time.Minute, + ended: 110 * time.Minute, periodNS: c.periodNS}) + res := scoreInto(t, report, labels) + tp := res.TruePositives == 1 + if tp != c.tp || res.PeriodMismatches != c.mismatch { + t.Errorf("%s: tp=%v mismatches=%d, want tp=%v mismatches=%d (%+v)", + c.name, tp, res.PeriodMismatches, c.tp, c.mismatch, res) + } + if c.periodNS == 0 && res.EpisodesWithoutPeriod != 1 { + t.Errorf("%s: episodes_without_period %d", c.name, res.EpisodesWithoutPeriod) + } + } +} + +// The match window is [incident start, stop + 4P] against +// [activation, end], and the channel hash must be equal. +func TestScoreOverlapAndChannel(t *testing.T) { + dir := t.TempDir() + p := 5 * time.Minute + labels := craftLabels(t, dir, time.Hour, 2*time.Hour, p) + for _, c := range []struct { + name string + active, end time.Duration + key string + tp, mismatch bool + }{ + {name: "inside", active: 90 * time.Minute, end: 110 * time.Minute, tp: true}, + {name: "ends just inside the start", active: 30 * time.Minute, end: time.Hour, tp: true}, + {name: "ends before the start", active: 10 * time.Minute, end: 50 * time.Minute}, + {name: "activates at stop + 4P", active: 2*time.Hour + 4*p, end: 3 * time.Hour, tp: true}, + {name: "activates after stop + 4P", active: 2*time.Hour + 4*p + time.Minute, end: 3 * time.Hour}, + {name: "never ended", active: 90 * time.Minute, tp: true}, + {name: "another channel", active: 90 * time.Minute, end: 110 * time.Minute, + key: "v1|s=stream|pt=5|ch=1:11|rc=flood|rk=relay|hop=1:fd"}, + } { + report := craftReport(t, dir, craftedEpisode{id: "e1", key: c.key, active: c.active, + ended: c.end, periodNS: int64(p)}) + res := scoreInto(t, report, labels) + if (res.TruePositives == 1) != c.tp { + t.Errorf("%s: tp %d, want %v (%+v)", c.name, res.TruePositives, c.tp, res) + } + if res.Episodes != 1 { + t.Errorf("%s: %d episodes", c.name, res.Episodes) + } + } +} + +// Only episodes activated inside [-start, -end) are scored, and each +// EventID is one episode however many transitions it has. +func TestScoreWindowAndEpisodeGrouping(t *testing.T) { + dir := t.TempDir() + labels := craftLabels(t, dir, time.Hour, 2*time.Hour, 5*time.Minute) + report := craftReport(t, dir, + craftedEpisode{id: "in", active: 90 * time.Minute, ended: 110 * time.Minute, periodNS: int64(5 * time.Minute)}, + craftedEpisode{id: "late", active: 25 * time.Hour, ended: 26 * time.Hour, periodNS: int64(5 * time.Minute)}, + craftedEpisode{id: "early", active: -time.Hour, ended: 30 * time.Minute, periodNS: int64(5 * time.Minute)}, + ) + res := scoreInto(t, report, labels) + if res.Episodes != 1 || res.TruePositives != 1 || res.PerIncident[0].Episodes != 1 { + t.Fatalf("window: %+v", res) + } + // Two episodes on the same incident are two episodes and one detection. + two := craftReport(t, dir, + craftedEpisode{id: "e1", active: 90 * time.Minute, ended: 100 * time.Minute, periodNS: int64(5 * time.Minute)}, + craftedEpisode{id: "e2", active: 105 * time.Minute, ended: 115 * time.Minute, periodNS: int64(5 * time.Minute)}, + ) + res = scoreInto(t, two, labels) + if res.Episodes != 2 || res.TruePositives != 2 || res.IncidentsDetected != 1 || + res.PerIncident[0].Episodes != 2 || res.MedianEpisodesPerIncident != 2 { + t.Fatalf("fragmentation: %+v", res) + } + // The time to detect is the first matching activation and the stop + // latency the last matching end. + if got := time.Duration(*res.MedianTimeToDetect); got != 30*time.Minute { + t.Fatalf("time to detect %v", got) + } + if got := time.Duration(*res.MedianStopLatency); got != -5*time.Minute { + t.Fatalf("stop latency %v", got) + } +} + +// False positives are counted per day and listed per channel hash. +func TestScoreFalsePositivesPerDay(t *testing.T) { + dir := t.TempDir() + labels := craftLabels(t, dir, time.Hour, 2*time.Hour, 5*time.Minute) + report := craftReport(t, dir, + craftedEpisode{id: "tp", active: 90 * time.Minute, ended: 110 * time.Minute, periodNS: int64(5 * time.Minute)}, + craftedEpisode{id: "fp1", active: 10 * time.Hour, ended: 11 * time.Hour, periodNS: int64(5 * time.Minute)}, + craftedEpisode{id: "fp2", active: 12 * time.Hour, ended: 13 * time.Hour, periodNS: int64(5 * time.Minute), + key: "v1|s=stream|pt=5|ch=1:11|rc=flood|rk=relay|hop=1:fd"}, + ) + res := scoreInto(t, report, labels) + if res.FalsePositives != 2 || res.FalseAlarmsPerDay != 2 { + t.Fatalf("false alarms: %+v", res) + } + if len(res.FalsePositivesByChannel) != 2 || res.FalsePositivesByChannel[0].Channel != "11" || + res.FalsePositivesByChannel[1].Channel != "7f" { + t.Fatalf("per channel: %+v", res.FalsePositivesByChannel) + } +} + +// Another rule's episodes are not scored. +func TestScoreOnlyTheNamedRule(t *testing.T) { + dir := t.TempDir() + report := craftReport(t, dir, craftedEpisode{id: "e1", active: 90 * time.Minute, + ended: 110 * time.Minute, periodNS: int64(5 * time.Minute)}) + b, err := os.ReadFile(report) + if err != nil { + t.Fatal(err) + } + other := filepath.Join(dir, "other.json") + if err := os.WriteFile(other, bytes.ReplaceAll(b, []byte(`"Rule":"p474"`), []byte(`"Rule":"p999"`)), 0o644); err != nil { + t.Fatal(err) + } + if res := scoreInto(t, other, ""); res.Episodes != 0 { + t.Fatalf("another rule was scored: %+v", res) + } + if res := scoreInto(t, report, ""); res.Episodes != 1 { + t.Fatalf("the named rule was not scored: %+v", res) + } +} + +func TestScoreRequiresExplicitFlagsAndValidInput(t *testing.T) { + dir := t.TempDir() + report := craftReport(t, dir, craftedEpisode{id: "e1", active: time.Hour, periodNS: int64(5 * time.Minute)}) + full := []string{"score", "-report", report, "-rule", "p474", "-start", injAt(0), "-end", injAt(24 * time.Hour)} + for i := 1; i < len(full); i += 2 { + args := append(append([]string{}, full[:i]...), full[i+2:]...) + var stdout, stderr bytes.Buffer + if code := run(args, nil, &stdout, &stderr); code != exitUsage { + t.Fatalf("without %s: exit %d", full[i], code) + } + } + var stdout, stderr bytes.Buffer + if code := run(append(append([]string{}, full...), "-end", injAt(-time.Hour)), nil, &stdout, &stderr); code != exitUsage { + t.Fatalf("-end before -start: exit %d", code) + } + // A report that is not a replay report, and a labels file that is not an + // injected labels file, are both errors. + bad := filepath.Join(dir, "bad.json") + for _, body := range []string{`{}`, `{"Experimental":true}`, `not json`, + `{"Experimental":true,"Start":"2030-03-01T00:00:00Z","End":"2030-03-02T00:00:00Z","Candidates":[{"EventID":"e","Rule":"p474","Kind":"periodic","Key":"nonsense","To":"active","At":"2030-03-01T01:00:00Z"}]}`} { + if err := os.WriteFile(bad, []byte(body), 0o644); err != nil { + t.Fatal(err) + } + args := append([]string{}, full...) + args[2] = bad + if code := run(args, nil, &stdout, &stderr); code != exitFail { + t.Fatalf("report %q: exit %d", body, code) + } + } + badLabels := filepath.Join(dir, "badlabels.json") + validLabelling := `"labelling":{"pulse_merge_gap":"60s","min_period":"1m","max_period":"1h","min_pulses":8,"max_harmonic":3,"tolerance_abs":"30s","tolerance_rel":0.1}` + for _, body := range []string{`{}`, + // a complete file that is not marked synthetic: scoring is only + // defined against injected ground truth + `{"experimental":true,"synthetic":false,` + validLabelling + `,"incidents":[]}`, + `{"experimental":true,"synthetic":true,"labelling":{"pulse_merge_gap":"60s","min_period":"1m","max_period":"1h","min_pulses":8,"max_harmonic":3,"tolerance_abs":"30s","tolerance_rel":0.1},"incidents":[{"bot":"b","channel":"","start":"2030-03-01T00:00:00Z","stop":"2030-03-01T01:00:00Z","period":"5m","pulses":9}]}`, + `{"experimental":true,"synthetic":true,"surprise":1}`} { + if err := os.WriteFile(badLabels, []byte(body), 0o644); err != nil { + t.Fatal(err) + } + if code := run(append(append([]string{}, full...), "-labels", badLabels), nil, &stdout, &stderr); code != exitFail { + t.Fatalf("labels %q: exit %d", body, code) + } + } +} + +// -out writes the metrics to a new file and never overwrites one. +func TestScoreWritesOutFileOnce(t *testing.T) { + dir := t.TempDir() + report := craftReport(t, dir, craftedEpisode{id: "e1", active: time.Hour, periodNS: int64(5 * time.Minute)}) + out := filepath.Join(dir, "metrics.json") + args := []string{"score", "-report", report, "-rule", "p474", "-start", injAt(0), + "-end", injAt(24 * time.Hour), "-out", out} + var stdout, stderr bytes.Buffer + if code := run(args, nil, &stdout, &stderr); code != 0 || stdout.Len() != 0 { + t.Fatalf("exit %d, stdout %d bytes: %s", code, stdout.Len(), stderr.String()) + } + var res scoreResult + b, err := os.ReadFile(out) + if err != nil || json.Unmarshal(b, &res) != nil || res.Episodes != 1 { + t.Fatalf("metrics file: %v %s", err, b) + } + if code := run(args, nil, &stdout, &stderr); code != exitFail { + t.Fatal("an existing metrics file was overwritten") + } +} diff --git a/internal/anomaly/config.go b/internal/anomaly/config.go index ab7fac76a..6080a1c3d 100644 --- a/internal/anomaly/config.go +++ b/internal/anomaly/config.go @@ -155,6 +155,29 @@ type PeriodicRule struct { // recent (local) pulse rate, over the period search and every pulse // evaluated on the key. MaxChance float64 + // ExcludeChannels lists channel hashes this rule ignores: an event whose + // channel hash is in the list is not offered to this rule at all, so the + // rule never forms a pulse, a chain or an episode on it. It is opt-in and + // has no default: an empty list excludes nothing. + // + // It exists because some channels carry expected automation, where a + // regular pulse train is normal rather than an incident (#474: a + // wardriving app posts positions every 75-123 s while someone drives). + // The library never learns which channel that is; the caller names the + // hashes. + // + // An entry must be a valid, canonical, non-zero ChannelHash, and matching + // is on the whole value, length included: the 1-byte hash 0x81 and the + // 2-byte hash 0x81ab are different entries, as everywhere else in this + // package, and the zero "no channel" value is not a valid entry. + // + // A channel hash is not a channel. MeshCore's hash is one byte, so one + // entry excludes every channel whose name hashes to that byte: excluding + // the hash of #wardriving also excludes the 1/256 of all other channel + // names that collide with it, and any traffic of theirs that this rule + // would otherwise have reported. That is a deliberate, stated cost of + // keying on an opaque 1-byte hash, not something the list can avoid. + ExcludeChannels []ChannelHash // EndAfterMissedPulses opts into train tracking (see track.go): while // the episode is active it ends when this many consecutive expected // pulses are missing, instead of State.EndAfterQuiet after the last @@ -171,6 +194,10 @@ const ( maxMissingLimit = 8 maxRuleNameLen = 64 maxKeysLimit = 10_000_000 + // maxExcludedChannels bounds PeriodicRule.ExcludeChannels. 256 is the + // whole 1-byte hash space MeshCore uses today, so the bound can never + // stop a caller from naming every channel hash it sees. + maxExcludedChannels = 256 ) var errNoRules = errors.New("anomaly: config enables no rule") @@ -348,6 +375,9 @@ func (c *Config) Validate() error { case r.EndAfterMissedPulses < 0 || r.EndAfterMissedPulses > maxMissedPulses: return fmt.Errorf("anomaly: rule %s: EndAfterMissedPulses must be in 0..%d", r.Name, maxMissedPulses) } + if err := validExcludeChannels(r.ExcludeChannels, r.Name); err != nil { + return err + } // A tolerance at or above half the period would let one gap match two // multiples; refuse it. if tolFor(r, r.MinPeriod) >= r.MinPeriod/2 { @@ -371,6 +401,45 @@ func (c *Config) Validate() error { return nil } +// validExcludeChannels checks a rule's exclusion list: every entry is a +// canonical, non-zero channel hash, no entry repeats, and the list stays +// bounded. An empty list is valid and excludes nothing. +func validExcludeChannels(cs []ChannelHash, name string) error { + if len(cs) > maxExcludedChannels { + return fmt.Errorf("anomaly: rule %s: ExcludeChannels must hold at most %d entries", name, maxExcludedChannels) + } + for i, c := range cs { + if c.IsZero() { + return fmt.Errorf("anomaly: rule %s: ExcludeChannels must not hold the zero \"no channel\" value", name) + } + if err := c.canonical(); err != nil { + return fmt.Errorf("anomaly: rule %s: ExcludeChannels: %w", name, err) + } + for _, o := range cs[:i] { + if o == c { + return fmt.Errorf("anomaly: rule %s: ExcludeChannels lists the same channel hash twice", name) + } + } + } + return nil +} + +// excludes reports whether this rule ignores events on channel hash c. +// Matching is on the whole value (length and bytes), so a 1-byte hash is +// never read as the prefix of a wider one, and the zero "no channel" value +// never matches (validExcludeChannels rejects it as an entry). +func (r *PeriodicRule) excludes(c ChannelHash) bool { + if c.IsZero() { + return false + } + for _, x := range r.ExcludeChannels { + if x == c { + return true + } + } + return false +} + func tolFor(r PeriodicRule, period time.Duration) time.Duration { rel := time.Duration(r.JitterRel * float64(period)) if r.JitterAbs > rel { diff --git a/internal/anomaly/detector.go b/internal/anomaly/detector.go index c64eaca8a..c85395076 100644 --- a/internal/anomaly/detector.go +++ b/internal/anomaly/detector.go @@ -134,8 +134,9 @@ func New(cfg Config) (*Detector, error) { c.Rate = append([]RateRule(nil), cfg.Rate...) c.NewStream = append([]NewStreamRule(nil), cfg.NewStream...) c.Periodic = append([]PeriodicRule(nil), cfg.Periodic...) - // PayloadTypes are the only slices inside the rules: copy them too, so the - // caller's config and the Detector never share a backing array. + // PayloadTypes, and a periodic rule's ExcludeChannels, are the only + // slices inside the rules: copy them too, so the caller's config and the + // Detector never share a backing array. for i := range c.Rate { c.Rate[i].PayloadTypes = slices.Clone(c.Rate[i].PayloadTypes) } @@ -144,6 +145,7 @@ func New(cfg Config) (*Detector, error) { } for i := range c.Periodic { c.Periodic[i].PayloadTypes = slices.Clone(c.Periodic[i].PayloadTypes) + c.Periodic[i].ExcludeChannels = slices.Clone(c.Periodic[i].ExcludeChannels) } d := &Detector{cfg: c, dedup: make(map[TxID]int64)} for i := range c.Rate { @@ -610,7 +612,7 @@ func (d *Detector) handle(e *Event) { applies = applies || appliesTo(d.cfg.NewStream[i].PayloadTypes, e.PayloadType) } for _, i := range r.periodic { - applies = applies || appliesTo(d.cfg.Periodic[i].PayloadTypes, e.PayloadType) + applies = applies || periodicApplies(&d.cfg.Periodic[i], *e) } if !applies { continue @@ -655,7 +657,7 @@ func (d *Detector) handle(e *Event) { } for pi, i := range r.periodic { rule := &d.cfg.Periodic[i] - if appliesTo(rule.PayloadTypes, e.PayloadType) { + if periodicApplies(rule, *e) { ps := k.per[pi] before := ps.evaluated ep := &k.eps[slot] @@ -675,6 +677,15 @@ func (d *Detector) handle(e *Event) { } } +// periodicApplies reports whether a periodic rule sees this event: its +// payload type must match, and its channel hash must not be one the rule +// excludes (PeriodicRule.ExcludeChannels). An excluded event never reaches +// the rule, so the rule forms no pulse, chain or episode on that channel and +// an excluded channel alone never claims key state. +func periodicApplies(rule *PeriodicRule, e Event) bool { + return appliesTo(rule.PayloadTypes, e.PayloadType) && !rule.excludes(e.Channel) +} + // track follows the pulse train of an active periodic episode (track.go). // followed says whether the tracker was armed for this same active episode // before the pulse; sig is the detection signal of the pulse, if ok. diff --git a/internal/anomaly/doc.go b/internal/anomaly/doc.go index 51d5dd3af..371f3d7df 100644 --- a/internal/anomaly/doc.go +++ b/internal/anomaly/doc.go @@ -44,6 +44,28 @@ // - A global (payload-type) candidate is never attributed to a stream, // channel or route. // +// # What the periodic rule is for (#474) +// +// The periodic rule's target is an abusive channel bot: a sender that floods +// a public or hashtag channel with automated messages. Automation that is +// expected on its channel is NOT a target. The #457 round-4 validation +// measured the opposite: all six labelled periodic incidents were wardriving +// trips, where an app posts positions every 75-123 s while someone drives, +// so the recipe counted normal behaviour as incidents and the rule failed on +// recall (0.17). +// +// PeriodicRule.ExcludeChannels is the library's side of that correction: the +// caller names the channel hashes whose automation is expected, and the rule +// never sees an event on them. It is opt-in and has no default, so an +// unconfigured rule behaves exactly as before. Because a MeshCore channel +// hash is one byte, one entry excludes every channel name that hashes to +// that byte; see PeriodicRule.ExcludeChannels. +// +// Neither the list nor anything else here makes a rule validated. A target +// definition, an exclusion list and an evaluation are three different +// things, and only the evaluation (fresh data, pre-registered bars) can +// support a claim. +// // # Data contract // // The Detector consumes FINAL Events, one per logical transmission, and diff --git a/internal/anomaly/exclude_channels_test.go b/internal/anomaly/exclude_channels_test.go new file mode 100644 index 000000000..ca5872b7c --- /dev/null +++ b/internal/anomaly/exclude_channels_test.go @@ -0,0 +1,289 @@ +package anomaly + +import ( + "testing" + "time" +) + +// Issue #474, phase 2: a periodic rule can ignore expected-automation +// channels (for example #wardriving) through PeriodicRule.ExcludeChannels. +// The option is opt-in: an empty list excludes nothing, which is the +// behaviour every other test in this package pins. +// +// All channel bytes here are synthetic test values. 0x81 is used because the +// #457 round-4 results name it, but nothing in the library knows what a +// channel hash means. + +// perfectTrain is a perfect 12-pulse 5-minute train on one stream, which +// experimentalPeriodic activates on at the 8th pulse. +func perfectTrain(t testing.TB, prefix string, ch byte) []Event { + t.Helper() + return series(t, prefix, t0, 12, 5*time.Minute, nil, ch, 0x22) +} + +// excludingRule returns experimentalPeriodic with the given channel hashes +// excluded. +func excludingRule(t testing.TB, chans ...ChannelHash) PeriodicRule { + t.Helper() + r := experimentalPeriodic() + r.ExcludeChannels = chans + return r +} + +// A train on an excluded channel produces no periodic candidate at all, +// while the same train on another channel still activates. +func TestPeriodicExcludedChannelProducesNoCandidate(t *testing.T) { + rule := excludingRule(t, mustChan(t, 0x81)) + d := newDet(t, Config{Periodic: []PeriodicRule{rule}}) + if cs := run(t, d, perfectTrain(t, "x", 0x81)); len(cs) != 0 { + t.Fatalf("excluded channel produced candidates:\n%s", describe(cs)) + } + d = newDet(t, Config{Periodic: []PeriodicRule{rule}}) + cs := run(t, d, perfectTrain(t, "y", 0x11)) + if len(filter(cs, "per", StateActive)) != 1 { + t.Fatalf("a non-excluded channel must still activate:\n%s", describe(cs)) + } +} + +// The exclusion matches the whole channel hash, length included: excluding +// the 1-byte hash 0x81 does not exclude the 2-byte hash 0x81ab, exactly as +// Key.String keeps a 1-byte value distinct from the prefix of a wider one. +func TestPeriodicExclusionMatchesTheWholeHash(t *testing.T) { + rule := excludingRule(t, mustChan(t, 0x81)) + wide := perfectTrain(t, "w", 0x81) + for i := range wide { + wide[i].Channel = mustChan(t, 0x81, 0xab) + } + cs := run(t, newDet(t, Config{Periodic: []PeriodicRule{rule}}), wide) + if len(filter(cs, "per", StateActive)) != 1 { + t.Fatalf("a 2-byte hash must not be excluded by a 1-byte entry:\n%s", describe(cs)) + } + // The reverse: a 2-byte entry does not exclude the 1-byte hash. + rule = excludingRule(t, mustChan(t, 0x81, 0xab)) + cs = run(t, newDet(t, Config{Periodic: []PeriodicRule{rule}}), perfectTrain(t, "n", 0x81)) + if len(filter(cs, "per", StateActive)) != 1 { + t.Fatalf("a 1-byte hash must not be excluded by a 2-byte entry:\n%s", describe(cs)) + } + // The length is part of the match even when the bytes alone look equal: + // the 2-byte hash 0x8100 is not the 1-byte hash 0x81 followed by + // padding, so a 0x81 entry does not exclude it. + rule = excludingRule(t, mustChan(t, 0x81)) + padded := perfectTrain(t, "z", 0x81) + for i := range padded { + padded[i].Channel = mustChan(t, 0x81, 0x00) + } + cs = run(t, newDet(t, Config{Periodic: []PeriodicRule{rule}}), padded) + if len(filter(cs, "per", StateActive)) != 1 { + t.Fatalf("the 2-byte hash 0x8100 was excluded by the 1-byte entry 0x81:\n%s", describe(cs)) + } +} + +// Every listed channel is excluded, not only the first one. +func TestPeriodicExclusionCoversEveryListedChannel(t *testing.T) { + rule := excludingRule(t, mustChan(t, 0x81), mustChan(t, 0x62)) + for _, ch := range []byte{0x81, 0x62} { + d := newDet(t, Config{Periodic: []PeriodicRule{rule}}) + if cs := run(t, d, perfectTrain(t, "m", ch)); len(cs) != 0 { + t.Fatalf("channel %#x produced candidates:\n%s", ch, describe(cs)) + } + } +} + +// The exclusion belongs to the periodic rule only: rate and new-stream +// rules on the same events are untouched, and so is a second periodic rule +// without the exclusion. +func TestPeriodicExclusionLeavesOtherRulesAlone(t *testing.T) { + other := experimentalPeriodic() + other.Name = "per2" + cfg := Config{ + Rate: []RateRule{rateRule("r", ScopeStream, time.Hour, 8)}, + NewStream: []NewStreamRule{{Name: "ns", Scope: ScopeStream, Window: time.Hour, MinCount: 6, + CensorWindow: 24 * time.Hour, QuietPeriod: 24 * time.Hour, State: quickState()}}, + Periodic: []PeriodicRule{excludingRule(t, mustChan(t, 0x81)), other}, + HistoryStart: t0.Add(-48 * time.Hour), + } + cs := run(t, newDet(t, cfg), perfectTrain(t, "o", 0x81)) + for _, rule := range []string{"r", "ns", "per2"} { + if len(filter(cs, rule, StateActive)) == 0 { + t.Errorf("rule %s did not activate on an excluded channel:\n%s", rule, describe(cs)) + } + } + if a := filter(cs, "per", StateActive); len(a) != 0 { + t.Fatalf("the excluding rule activated:\n%s", describe(cs)) + } +} + +// With only an excluding periodic rule, an event on an excluded channel +// creates no key state at all, so the exclusion also keeps the scope's key +// budget free. +func TestPeriodicExclusionKeepsNoKeyState(t *testing.T) { + d := newDet(t, Config{Periodic: []PeriodicRule{excludingRule(t, mustChan(t, 0x81))}}) + run(t, d, perfectTrain(t, "k", 0x81)) + if n := d.Snapshot().Keys[ScopeStream]; n != 0 { + t.Fatalf("excluded channel kept %d stream keys, want 0", n) + } + d = newDet(t, Config{Periodic: []PeriodicRule{excludingRule(t, mustChan(t, 0x81))}}) + run(t, d, perfectTrain(t, "k2", 0x11)) + if n := d.Snapshot().Keys[ScopeStream]; n != 1 { + t.Fatalf("a non-excluded channel kept %d stream keys, want 1", n) + } +} + +// New copies ExcludeChannels: changing the caller's slice afterwards cannot +// change what the detector excludes, and the detector never writes to it. +func TestPeriodicExclusionIsCopiedFromTheCallerConfig(t *testing.T) { + rule := excludingRule(t, mustChan(t, 0x81)) + cfg := Config{Periodic: []PeriodicRule{rule}, Expected: ExpectedPolicy{Mode: ExpectedInclude}, + Limits: experimentalLimits()} + d, err := New(cfg) + if err != nil { + t.Fatal(err) + } + cfg.Periodic[0].ExcludeChannels[0] = mustChan(t, 0x11) + if cs := run(t, d, perfectTrain(t, "c", 0x81)); len(cs) != 0 { + t.Fatalf("the caller's slice changed what the detector excludes:\n%s", describe(cs)) + } + if got := cfg.Periodic[0].ExcludeChannels[0]; got != mustChan(t, 0x11) { + t.Fatalf("the detector wrote to the caller's slice: %v", got.Bytes()) + } +} + +// An event without a channel (a payload type that carries none) is never +// excluded: the zero "no channel" value is not a 1-byte hash and is not a +// valid list entry. +func TestPeriodicExclusionNeverMatchesNoChannel(t *testing.T) { + rule := excludingRule(t, mustChan(t, 0x00)) + rule.PayloadTypes = []PayloadType{4} + var evs []Event + for i := 0; i < 12; i++ { + evs = append(evs, Event{ID: TxID("a" + string(rune('a'+i))), Time: at(time.Duration(i) * 5 * time.Minute), + PayloadType: 4, RouteClass: RouteClassFlood, RouteKind: RouteKindRelay, + FirstHop: mustHop(t, 0x22), FullRoute: mustRoute(t, 1, 0x22, 0x00), WireBytes: 40}) + } + cs := run(t, newDet(t, Config{Periodic: []PeriodicRule{rule}}), evs) + if len(filter(cs, "per", StateActive)) != 1 { + t.Fatalf("a channel-less train must not be excluded by a 1-byte 0x00 entry:\n%s", describe(cs)) + } +} + +// The exclusion works at channel scope too, where the key is the channel. +func TestPeriodicExclusionAtChannelScope(t *testing.T) { + rule := excludingRule(t, mustChan(t, 0x81)) + rule.Scope = ScopeChannel + d := newDet(t, Config{Periodic: []PeriodicRule{rule}}) + if cs := run(t, d, perfectTrain(t, "s", 0x81)); len(cs) != 0 { + t.Fatalf("channel scope ignored the exclusion:\n%s", describe(cs)) + } +} + +func TestExcludeChannelsValidation(t *testing.T) { + for _, c := range []struct { + name string + list []ChannelHash + ok bool + }{ + {"unset", nil, true}, + {"empty", []ChannelHash{}, true}, + {"one byte", []ChannelHash{mustChan(t, 0x81)}, true}, + {"two entries", []ChannelHash{mustChan(t, 0x81), mustChan(t, 0x62)}, true}, + {"wider hash", []ChannelHash{mustChan(t, 0x81, 0xab, 0xcd, 0xef)}, true}, + {"same byte twice", []ChannelHash{mustChan(t, 0x81), mustChan(t, 0x81)}, false}, + {"no channel", []ChannelHash{{}}, false}, + {"non-canonical", []ChannelHash{nonCanonicalChan(0x81)}, false}, + } { + r := experimentalPeriodic() + r.ExcludeChannels = c.list + err := (&Config{Limits: experimentalLimits(), Expected: ExpectedPolicy{Mode: ExpectedInclude}, + Periodic: []PeriodicRule{r}}).Validate() + if (err == nil) != c.ok { + t.Errorf("%s: err %v, want ok=%v", c.name, err, c.ok) + } + } + // Too many entries are rejected; the whole 1-byte space is not. + full := make([]ChannelHash, 0, maxExcludedChannels+1) + for i := 0; i <= maxExcludedChannels; i++ { + c, err := NewChannelHash([]byte{byte(i % 256), byte(i / 256)}[:1+i/256]) + if err != nil { + t.Fatal(err) + } + full = append(full, c) + } + for _, c := range []struct { + n int + ok bool + }{{maxExcludedChannels, true}, {maxExcludedChannels + 1, false}} { + r := experimentalPeriodic() + r.ExcludeChannels = full[:c.n] + err := (&Config{Limits: experimentalLimits(), Expected: ExpectedPolicy{Mode: ExpectedInclude}, + Periodic: []PeriodicRule{r}}).Validate() + if (err == nil) != c.ok { + t.Errorf("%d entries: err %v, want ok=%v", c.n, err, c.ok) + } + } +} + +// nonCanonicalChan builds a 1-byte hash with a stray byte beyond its length, +// which Key.Validate rejects elsewhere and the exclusion list rejects here. +func nonCanonicalChan(b byte) ChannelHash { + c := ChannelHash{n: 1} + c.b[0], c.b[1] = b, 0xff + return c +} + +// ---- replay JSON config ---- + +func TestReplayConfigParsesExcludeChannels(t *testing.T) { + cfg, err := ParseReplayConfig([]byte(excludeConfigJSON(`"exclude_channels": ["81", "62ab"],`))) + if err != nil { + t.Fatal(err) + } + got := cfg.Periodic[0].ExcludeChannels + if len(got) != 2 || got[0] != mustChan(t, 0x81) || got[1] != mustChan(t, 0x62, 0xab) { + t.Fatalf("exclude_channels parsed as %v", got) + } +} + +// Omitting exclude_channels means no exclusion: the field is opt-in and has +// no default. +func TestReplayConfigWithoutExcludeChannelsExcludesNothing(t *testing.T) { + cfg, err := ParseReplayConfig([]byte(excludeConfigJSON(""))) + if err != nil { + t.Fatal(err) + } + if got := cfg.Periodic[0].ExcludeChannels; got != nil { + t.Fatalf("exclude_channels defaulted to %v", got) + } +} + +func TestReplayConfigRejectsBadExcludeChannels(t *testing.T) { + for name, v := range map[string]string{ + "odd hex": `["8"]`, + "not hex": `["zz"]`, + "empty string": `[""]`, + "too long": `["0102030405"]`, + "upper case": `["AB"]`, + "with prefix": `["0x81"]`, + "duplicate": `["81", "81"]`, + "not a string": `[129]`, + } { + if _, err := ParseReplayConfig([]byte(excludeConfigJSON(`"exclude_channels": ` + v + `,`))); err == nil { + t.Errorf("%s: accepted %s", name, v) + } + } +} + +// excludeConfigJSON is a minimal valid replay config with one periodic rule, +// into which a line such as `"exclude_channels": ["81"],` is spliced. +func excludeConfigJSON(line string) string { + return `{ + "limits": {"max_keys_per_scope": 1000, "state_ttl": "48h", "dedup_horizon": "6h", "max_dedup_ids": 1000, + "reorder_delay": "0s", "max_reorder_buffer": 64, "max_window_entries": 1000, + "max_candidates_per_call": 100, "max_pending_candidates": 1000}, + "expected": "include", + "periodic": [{"name": "p474", "scope": "stream", "payload_types": [5], + "min_period": "1m", "max_period": "1h", "burst_gap": "15s", "jitter_abs": "10s", "jitter_rel": 0.02, + "max_missing": 2, "min_pulses": 8, "history_len": 32, "min_coverage": 0.6, "max_jitter_fraction": 0.5, + "max_chance": 0.01, "end_after_missed_pulses": 3, ` + line + ` + "state": {"confirm_after": "0s", "end_after_quiet": "60m", "cooldown": "0s"}}] +}` +} diff --git a/internal/anomaly/replay.go b/internal/anomaly/replay.go index 83f7caa98..d787c379c 100644 --- a/internal/anomaly/replay.go +++ b/internal/anomaly/replay.go @@ -373,8 +373,12 @@ type ReplayConfig struct { MaxChance float64 `json:"max_chance"` // EndAfterMissedPulses is optional: omitted means 0, tracking off // (see PeriodicRule.EndAfterMissedPulses). - EndAfterMissedPulses int `json:"end_after_missed_pulses"` - State stateJSON `json:"state"` + EndAfterMissedPulses int `json:"end_after_missed_pulses"` + // ExcludeChannels is optional: channel hashes this rule ignores, as + // plain lower-case hex of 1 to MaxChannelHashLen bytes ("81"). + // Omitted means none (see PeriodicRule.ExcludeChannels). + ExcludeChannels []string `json:"exclude_channels"` + State stateJSON `json:"state"` } `json:"periodic"` } @@ -440,11 +444,41 @@ func ParseReplayConfig(b []byte) (Config, error) { if err != nil { return Config{}, err } + ex, err := parseExcludeChannels(r.ExcludeChannels) + if err != nil { + return Config{}, err + } cfg.Periodic = append(cfg.Periodic, PeriodicRule{Name: r.Name, Scope: s, PayloadTypes: r.PayloadTypes, - MinPeriod: time.Duration(r.MinPeriod), MaxPeriod: time.Duration(r.MaxPeriod), BurstGap: time.Duration(r.BurstGap), + ExcludeChannels: ex, + MinPeriod: time.Duration(r.MinPeriod), MaxPeriod: time.Duration(r.MaxPeriod), BurstGap: time.Duration(r.BurstGap), JitterAbs: time.Duration(r.JitterAbs), JitterRel: r.JitterRel, MaxMissing: r.MaxMissing, MinPulses: r.MinPulses, HistoryLen: r.HistoryLen, MinCoverage: r.MinCoverage, MaxJitterFraction: r.MaxJitterFraction, MaxChance: r.MaxChance, EndAfterMissedPulses: r.EndAfterMissedPulses, State: r.State.params()}) } return cfg, cfg.Validate() } + +// parseExcludeChannels decodes the JSON form of +// PeriodicRule.ExcludeChannels: plain lower-case hex, one entry per channel +// hash. An empty list gives a nil slice, which excludes nothing. +func parseExcludeChannels(ss []string) ([]ChannelHash, error) { + if len(ss) == 0 { + return nil, nil + } + out := make([]ChannelHash, 0, len(ss)) + for _, s := range ss { + if s == "" || strings.ToLower(s) != s { + return nil, errors.New("anomaly: exclude_channels: want lower-case hex") + } + b, err := hex.DecodeString(s) + if err != nil { + return nil, errors.New("anomaly: exclude_channels: bad hex") + } + c, err := NewChannelHash(b) + if err != nil { + return nil, err + } + out = append(out, c) + } + return out, nil +} From be10178059f2b2c6d877f10927cc74a9001e0972 Mon Sep 17 00:00:00 2001 From: dborup Date: Thu, 15 Oct 2026 20:34:46 +0200 Subject: [PATCH 02/12] feat(anomaly): let score select which bots the bars cover Relates to #474. `score -bots ` narrows the recall and the medians to the named bots' incidents, so a non-blind sanity-check bot (the September shape, which #130 and #447 were tuned on) can be reported apart from the blind panel without a hand computation afterwards. An episode that matches an unselected incident stays a true positive: the filter must never turn a known match into a false alarm. A filter that selects no labelled incident is an error, not an empty pass. --- cmd/anomaly/score.go | 66 ++++++++++++++++++++++++++++++++------- cmd/anomaly/score_test.go | 53 ++++++++++++++++++++++++++++--- 2 files changed, 103 insertions(+), 16 deletions(-) diff --git a/cmd/anomaly/score.go b/cmd/anomaly/score.go index 33e3d8a0b..5b4fe1152 100644 --- a/cmd/anomaly/score.go +++ b/cmd/anomaly/score.go @@ -9,6 +9,7 @@ import ( "io" "os" "sort" + "strings" "time" "github.com/meshcore-analyzer/anomaly" @@ -31,6 +32,11 @@ import ( // evidence can never be a true positive. // - Every other episode is a false positive. With no labels file, every // episode is a false positive, which is the false-alarm arm. +// - -bots narrows which incidents the recall and the medians cover, for +// example to leave a non-blind sanity-check bot out of the primary +// numbers. It never turns a matched episode into a false positive: an +// episode that matches any labelled incident is a true positive +// whether or not that incident was selected. // // Scoring reads files only: a report, a labels file and nothing else. @@ -91,7 +97,9 @@ type scoreResult struct { End string `json:"end"` Days float64 `json:"days"` Labels string `json:"labels"` - Episodes int `json:"episodes"` + // Bots is the -bots filter, empty when every labelled bot counts. + Bots []string `json:"bots"` + Episodes int `json:"episodes"` // EpisodesWithoutPeriod counts episodes whose activation candidate // carried no periodic evidence; they can never be true positives. EpisodesWithoutPeriod int `json:"episodes_without_period"` @@ -100,8 +108,11 @@ type scoreResult struct { FalseAlarmsPerDay float64 `json:"false_alarms_per_day"` // PeriodMismatches counts episodes that overlap an incident on the same // channel but fail the period check (#457 "also reported"). - PeriodMismatches int `json:"period_mismatches"` + PeriodMismatches int `json:"period_mismatches"` + // Incidents is every labelled incident; IncidentsSelected is the subset + // the bars cover (all of them unless -bots narrows it). Incidents int `json:"incidents"` + IncidentsSelected int `json:"incidents_selected"` IncidentsDetected int `json:"incidents_detected"` Recall float64 `json:"recall"` // Medians over the detected incidents. @@ -126,6 +137,7 @@ func runScore(args []string, stdout, stderr io.Writer) int { report := fs.String("report", "", "replay report JSON (required)") labels := fs.String("labels", "", "injected labels JSON; omitted means no ground truth (false alarms only)") rule := fs.String("rule", "", "periodic rule name to score (required)") + bots := fs.String("bots", "", "comma-separated bot names whose incidents the bars cover; empty means all") start := fs.String("start", "", "RFC 3339 start of the evaluation window, inclusive (required)") end := fs.String("end", "", "RFC 3339 end of the evaluation window, exclusive (required)") out := fs.String("out", "", "metrics file (must not exist); default stdout") @@ -175,7 +187,16 @@ func runScore(args []string, stdout, stderr io.Writer) int { return fail(err) } } - res, err := score(rep, lab, *rule, *labels, opt.start, opt.end) + var want []string + for _, b := range strings.Split(*bots, ",") { + if b = strings.TrimSpace(b); b != "" { + want = append(want, b) + } + } + res, err := score(rep, lab, *rule, *labels, want, opt.start, opt.end) + if err == nil && len(want) > 0 && res.IncidentsSelected == 0 { + return fail(errors.New("-bots selected no labelled incident")) + } if err != nil { return fail(err) } @@ -238,8 +259,9 @@ func parseInjectLabels(b []byte) (InjectLabels, error) { return l, nil } -// score computes the metrics. labelsPath is reported as provenance only. -func score(rep scoreReport, lab InjectLabels, rule, labelsPath string, start, end time.Time) (scoreResult, error) { +// score computes the metrics. labelsPath is reported as provenance only; +// bots, when not empty, selects which bots' incidents the bars cover. +func score(rep scoreReport, lab InjectLabels, rule, labelsPath string, bots []string, start, end time.Time) (scoreResult, error) { eps, noPeriod, err := episodes(rep, rule, start, end) if err != nil { return scoreResult{}, err @@ -247,8 +269,19 @@ func score(rep scoreReport, lab InjectLabels, rule, labelsPath string, start, en days := end.Sub(start).Hours() / 24 res := scoreResult{Experimental: true, Rule: rule, Start: start.Format(time.RFC3339), End: end.Format(time.RFC3339), Days: days, Labels: labelsPath, Episodes: len(eps), - EpisodesWithoutPeriod: noPeriod, Incidents: len(lab.Incidents), PerIncident: []incidentScore{}, - FalsePositivesByChannel: []channelCount{}} + EpisodesWithoutPeriod: noPeriod, Incidents: len(lab.Incidents), Bots: bots, + PerIncident: []incidentScore{}, FalsePositivesByChannel: []channelCount{}} + selected := func(bot string) bool { + if len(bots) == 0 { + return true + } + for _, b := range bots { + if b == bot { + return true + } + } + return false + } l := lab.Labelling var ttds, stops []int64 var perInc []float64 @@ -280,13 +313,22 @@ func score(rep scoreReport, lab InjectLabels, rule, labelsPath string, start, en } if is.Episodes > 0 { is.Detected = true - res.IncidentsDetected++ ttd := dur(firstAct.Sub(inc.Start)) sl := dur(lastEnd.Sub(inc.Stop)) is.TimeToDetect, is.StopLatency = &ttd, &sl is.TTDPeriods = float64(firstAct.Sub(inc.Start)) / float64(p) - ttds = append(ttds, int64(ttd)) - stops = append(stops, int64(sl)) + } + // Only selected incidents enter the recall and the medians; an + // episode that matched an unselected incident is still a match, so + // it never becomes a false positive. + if !selected(inc.Bot) { + continue + } + res.IncidentsSelected++ + if is.Episodes > 0 { + res.IncidentsDetected++ + ttds = append(ttds, int64(*is.TimeToDetect)) + stops = append(stops, int64(*is.StopLatency)) perInc = append(perInc, float64(is.Episodes)) } res.PerIncident = append(res.PerIncident, is) @@ -306,8 +348,8 @@ func score(rep scoreReport, lab InjectLabels, rule, labelsPath string, start, en if days > 0 { res.FalseAlarmsPerDay = float64(res.FalsePositives) / days } - if res.Incidents > 0 { - res.Recall = float64(res.IncidentsDetected) / float64(res.Incidents) + if res.IncidentsSelected > 0 { + res.Recall = float64(res.IncidentsDetected) / float64(res.IncidentsSelected) } if len(ttds) > 0 { m := dur(medianInt64(ttds)) diff --git a/cmd/anomaly/score_test.go b/cmd/anomaly/score_test.go index 8526ed0d0..dba16c359 100644 --- a/cmd/anomaly/score_test.go +++ b/cmd/anomaly/score_test.go @@ -207,15 +207,16 @@ func craftReport(t *testing.T, dir string, eps ...craftedEpisode) string { return p } -// craftLabels writes a labels file with one incident on channel 7f. -func craftLabels(t *testing.T, dir string, start, stop time.Duration, period time.Duration) string { +// craftLabels writes a labels file with one incident on channel 7f, plus +// any extra incidents given. +func craftLabels(t *testing.T, dir string, start, stop time.Duration, period time.Duration, extra ...Incident) string { t.Helper() l := InjectLabels{Experimental: true, Synthetic: true, Seed: 1, Labelling: labellingSpec{PulseMergeGap: dur(time.Minute), MinPeriod: dur(time.Minute), MaxPeriod: dur(time.Hour), MinPulses: 8, MaxHarmonic: 3, ToleranceAbs: dur(30 * time.Second), ToleranceRel: 0.1}, - Incidents: []Incident{{Bot: "bot-a", Channel: "7f", Start: injT.Add(start), Stop: injT.Add(stop), - Period: dur(period), Pulses: 20}}} + Incidents: append([]Incident{{Bot: "bot-a", Channel: "7f", Start: injT.Add(start), Stop: injT.Add(stop), + Period: dur(period), Pulses: 20}}, extra...)} b, err := json.MarshalIndent(l, "", " ") if err != nil { t.Fatal(err) @@ -435,3 +436,47 @@ func TestScoreWritesOutFileOnce(t *testing.T) { t.Fatal("an existing metrics file was overwritten") } } + +// -bots narrows which incidents the bars cover, for the non-blind +// sanity-check bot of the #474 pre-registration. An episode that matches an +// unselected incident stays a true positive: it must never turn into a +// false alarm. +func TestScoreBotsFilter(t *testing.T) { + dir := t.TempDir() + p := 5 * time.Minute + labels := craftLabels(t, dir, time.Hour, 2*time.Hour, p, + Incident{Bot: "sanity", Channel: "11", Start: injT.Add(10 * time.Hour), + Stop: injT.Add(11 * time.Hour), Period: dur(p), Pulses: 20}) + report := craftReport(t, dir, + craftedEpisode{id: "blind", active: 90 * time.Minute, ended: 110 * time.Minute, periodNS: int64(p)}, + craftedEpisode{id: "sanity", active: 10*time.Hour + 30*time.Minute, ended: 11 * time.Hour, + periodNS: int64(p), key: "v1|s=stream|pt=5|ch=1:11|rc=flood|rk=relay|hop=1:fd"}, + ) + all := scoreInto(t, report, labels) + if all.Incidents != 2 || all.IncidentsSelected != 2 || all.IncidentsDetected != 2 || all.Recall != 1 || + all.TruePositives != 2 || all.FalsePositives != 0 { + t.Fatalf("unfiltered: %+v", all) + } + blind := scoreInto(t, report, labels, "-bots", "bot-a") + if blind.Incidents != 2 || blind.IncidentsSelected != 1 || blind.IncidentsDetected != 1 || + blind.Recall != 1 || len(blind.PerIncident) != 1 || blind.PerIncident[0].Bot != "bot-a" { + t.Fatalf("blind arm: %+v", blind) + } + // The sanity bot's episode is still a true positive, not a false alarm. + if blind.TruePositives != 2 || blind.FalsePositives != 0 || blind.FalseAlarmsPerDay != 0 { + t.Fatalf("the unselected incident's episode was counted as a false alarm: %+v", blind) + } + if len(blind.Bots) != 1 || blind.Bots[0] != "bot-a" { + t.Fatalf("the filter is not reported: %+v", blind.Bots) + } + sanity := scoreInto(t, report, labels, "-bots", " sanity ,") + if sanity.IncidentsSelected != 1 || sanity.PerIncident[0].Bot != "sanity" { + t.Fatalf("sanity arm: %+v", sanity) + } + // A filter that selects nothing is an error, not an empty pass. + var stdout, stderr bytes.Buffer + if code := run([]string{"score", "-report", report, "-labels", labels, "-rule", "p474", + "-start", injAt(0), "-end", injAt(24 * time.Hour), "-bots", "nobody"}, nil, &stdout, &stderr); code != exitFail { + t.Fatalf("-bots with an unknown name: exit %d", code) + } +} From 3780b28264dc5b5a17337eba6c76c65a1db28da4 Mon Sep 17 00:00:00 2001 From: dborup Date: Thu, 15 Oct 2026 20:35:08 +0200 Subject: [PATCH 03/12] docs(anomaly): document score -bots in the tool README Relates to #474. --- cmd/anomaly/README.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/cmd/anomaly/README.md b/cmd/anomaly/README.md index 029c9fbf4..8d5559766 100644 --- a/cmd/anomaly/README.md +++ b/cmd/anomaly/README.md @@ -251,7 +251,7 @@ all. ```sh corescope-anomaly score -report report.json -labels labels.json \ -rule p474_periodic -start 2026-10-16T00:00:00Z -end 2026-10-23T00:00:00Z \ - [-out metrics.json] + [-bots b0-90s-clean,b1-120s-clean] [-out metrics.json] ``` The matching is the #457 §5 periodic rule: @@ -267,6 +267,10 @@ The matching is the #457 §5 periodic rule: - Every other episode is a **false positive**. Without `-labels` every episode is one, which is the false-alarm arm: run it on background-only traffic to get false alarms per day. +- `-bots` narrows which incidents the recall and the medians cover - for a + non-blind sanity-check bot, say. It never turns a matched episode into a + false alarm: an episode matching any labelled incident stays a true + positive, selected or not. A filter that selects no incident is an error. The metrics are episodes, true and false positives, false alarms per day, recall, median time to detect (also in units of P), median stop latency, From b7ee09529816ea4944081a44471d0c158051e492 Mon Sep 17 00:00:00 2001 From: dborup Date: Thu, 15 Oct 2026 20:40:15 +0200 Subject: [PATCH 04/12] refactor(anomaly): drop a dead field and keep score's bots filter as [] Relates to #474. The episode struct's id was written and never read, and an unset -bots filter marshalled as null instead of an empty list. No behaviour change. --- cmd/anomaly/score.go | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-) diff --git a/cmd/anomaly/score.go b/cmd/anomaly/score.go index 5b4fe1152..5fad6aea5 100644 --- a/cmd/anomaly/score.go +++ b/cmd/anomaly/score.go @@ -63,7 +63,6 @@ type scoreReport struct { // episode is one EventID's activation and end. type episode struct { - id string channel string activation time.Time end time.Time @@ -187,7 +186,7 @@ func runScore(args []string, stdout, stderr io.Writer) int { return fail(err) } } - var want []string + want := []string{} for _, b := range strings.Split(*bots, ",") { if b = strings.TrimSpace(b); b != "" { want = append(want, b) @@ -269,7 +268,7 @@ func score(rep scoreReport, lab InjectLabels, rule, labelsPath string, bots []st days := end.Sub(start).Hours() / 24 res := scoreResult{Experimental: true, Rule: rule, Start: start.Format(time.RFC3339), End: end.Format(time.RFC3339), Days: days, Labels: labelsPath, Episodes: len(eps), - EpisodesWithoutPeriod: noPeriod, Incidents: len(lab.Incidents), Bots: bots, + EpisodesWithoutPeriod: noPeriod, Incidents: len(lab.Incidents), Bots: nonNil(bots), PerIncident: []incidentScore{}, FalsePositivesByChannel: []channelCount{}} selected := func(bot string) bool { if len(bots) == 0 { @@ -430,7 +429,7 @@ func episodes(rep scoreReport, rule string, start, end time.Time) ([]episode, in if !a.actSet || a.act.Before(start) || !a.act.Before(end) { continue } - e := episode{id: id, channel: a.channel, activation: a.act, end: a.endT, + e := episode{channel: a.channel, activation: a.act, end: a.endT, hasPeriod: a.hasPeriod, period: a.period} if !a.endSet { e.end = rep.End // never ended: the report's end, as #457 §1 says @@ -478,6 +477,14 @@ func periodFits(got, p time.Duration, l labellingSpec) bool { return false } +// nonNil keeps an empty filter out of the JSON as [] rather than null. +func nonNil(v []string) []string { + if v == nil { + return []string{} + } + return v +} + func medianFloat(v []float64) float64 { if len(v) == 0 { return 0 From f22169955d0daf33ce201e11eaf6eafcf93334d7 Mon Sep 17 00:00:00 2001 From: dborup Date: Thu, 15 Oct 2026 22:15:52 +0200 Subject: [PATCH 05/12] test(anomaly): pin the boundary cases round 1's mutants survived Review F5. Five mutants survived the first round's suite: - labelIncidents' run length >= min_pulses -> > (nine pulses required): no test had a run of exactly min_pulses. - labelIncidents' gap tolerance <= -> <. - labelIncidents' pulse merge gap >= -> >. - score's periodFits tolerance <= -> <. - score's episode end taking the first "ended" instead of the last. Each now has a test that fails on the mutant and passes on the code: exactly 8 pulses (and 7, which must not label), a gap exactly 30 s off k*P (and one nanosecond further), events exactly 60 s apart (and one nanosecond closer), a period exactly at the tolerance through both periodFits and score, and an episode with two "ended" transitions. Relates to #474 --- cmd/anomaly/review_r2_test.go | 151 ++++++++++++++++++++++++++++++++++ 1 file changed, 151 insertions(+) create mode 100644 cmd/anomaly/review_r2_test.go diff --git a/cmd/anomaly/review_r2_test.go b/cmd/anomaly/review_r2_test.go new file mode 100644 index 000000000..946763195 --- /dev/null +++ b/cmd/anomaly/review_r2_test.go @@ -0,0 +1,151 @@ +package main + +import ( + "testing" + "time" +) + +// Round-2 review of PR #478 (#474): the boundary cases the first round's +// mutants survived, plus the parity and key-sharing properties the +// pre-registration now depends on. Every bot, channel byte and time here is +// synthetic. + +// r2Labelling is the labelling recipe of the pre-registration: a 60 s merge +// gap, P in [1 min, 1 h], k in 1..3 and a tolerance of max(30 s, 0.1 P). +func r2Labelling(minPulses int) labellingSpec { + return labellingSpec{PulseMergeGap: dur(60 * time.Second), MinPeriod: dur(time.Minute), + MaxPeriod: dur(time.Hour), MinPulses: minPulses, MaxHarmonic: 3, + ToleranceAbs: dur(30 * time.Second), ToleranceRel: 0.1} +} + +// r2Pulses builds one injected event per offset, which is all +// labelIncidents reads. +func r2Pulses(offsets ...time.Duration) []injEvent { + evs := make([]injEvent, 0, len(offsets)) + for i, d := range offsets { + evs = append(evs, injEvent{t: injT.Add(d).UnixNano(), pulse: i}) + } + return evs +} + +// grid returns n offsets exactly p apart. +func grid(n int, p time.Duration) []time.Duration { + out := make([]time.Duration, 0, n) + for i := 0; i < n; i++ { + out = append(out, time.Duration(i)*p) + } + return out +} + +// F5: a run of exactly MinPulses pulses is an incident, and one pulse fewer +// is not. Without the exact-boundary case, labelling with ">" instead of +// ">=" (nine pulses required) passes every other test. +func TestLabelIncidentsAtExactlyMinPulses(t *testing.T) { + const p = 300 * time.Second + exact := grid(8, p) + inc := labelIncidents("bot-a", "7f", r2Pulses(exact...), r2Labelling(8)) + if len(inc) != 1 || inc[0].Pulses != 8 { + t.Fatalf("exactly 8 pulses must be one 8-pulse incident: %+v", inc) + } + if got := labelIncidents("bot-a", "7f", r2Pulses(exact[:7]...), r2Labelling(8)); len(got) != 0 { + t.Fatalf("7 pulses must not be an incident: %+v", got) + } +} + +// F5: a gap exactly max(tolerance_abs, tolerance_rel*P) away from k*P is +// inside the tolerance. P = 300 s gives a tolerance of exactly 30 s, so a +// 330 s gap is the boundary and a 331 s gap is outside it. +func TestLabelIncidentsAtTheToleranceBoundary(t *testing.T) { + const p = 300 * time.Second + at := func(last time.Duration) []time.Duration { + return append(grid(7, p), 6*p+last) + } + inc := labelIncidents("bot-a", "7f", r2Pulses(at(330*time.Second)...), r2Labelling(8)) + if len(inc) != 1 || inc[0].Pulses != 8 { + t.Fatalf("a gap exactly at the tolerance must stay in the run: %+v", inc) + } + // One nanosecond further out and the run breaks before the last pulse, + // which leaves 7 pulses: below min_pulses, so no incident at all. + if got := labelIncidents("bot-a", "7f", r2Pulses(at(330*time.Second+1)...), r2Labelling(8)); len(got) != 0 { + t.Fatalf("a gap past the tolerance must break the run: %+v", got) + } +} + +// F5: two events exactly pulse_merge_gap apart are two pulses, not one. The +// merge keeps events strictly closer than the gap together. +func TestLabelIncidentsAtThePulseMergeGapBoundary(t *testing.T) { + l := r2Labelling(2) + inc := labelIncidents("bot-a", "7f", r2Pulses(0, 60*time.Second), l) + if len(inc) != 1 || inc[0].Pulses != 2 || time.Duration(inc[0].Period) != 60*time.Second { + t.Fatalf("events exactly 60 s apart are two pulses: %+v", inc) + } + // One nanosecond closer and they merge, which leaves a single pulse and + // no incident. + if got := labelIncidents("bot-a", "7f", r2Pulses(0, 60*time.Second-1), l); len(got) != 0 { + t.Fatalf("events closer than the gap are one pulse: %+v", got) + } +} + +// r2Cand builds one report candidate. period == 0 means no periodic +// evidence on that candidate. +func r2Cand(id, to, key string, at time.Time, period time.Duration) scoreCandidate { + c := scoreCandidate{EventID: id, Rule: "p474", Kind: "periodic", Key: key, To: to, At: at} + if period > 0 { + c.Periodic = &struct{ Period time.Duration }{Period: period} + } + return c +} + +func r2Report(cands ...scoreCandidate) scoreReport { + return scoreReport{Experimental: true, Start: injT, End: injT.Add(24 * time.Hour), Candidates: cands} +} + +// F5: an episode whose period is exactly the tolerance away from k*P is a +// true positive, not a period mismatch. +func TestScorePeriodFitsAtTheToleranceBoundary(t *testing.T) { + const p = 300 * time.Second + l := r2Labelling(8) + if !periodFits(p+30*time.Second, p, l) { + t.Fatal("a period exactly at the tolerance must fit") + } + if periodFits(p+30*time.Second+1, p, l) { + t.Fatal("a period past the tolerance must not fit") + } + // The same through score: the boundary episode is a true positive and + // raises no period mismatch. + inc := Incident{Bot: "bot-a", Channel: "7f", Start: injT, Stop: injT.Add(time.Hour), + Period: dur(p), Pulses: 8} + lab := InjectLabels{Synthetic: true, Labelling: l, Incidents: []Incident{inc}} + rep := r2Report(r2Cand("e1", "active", scoreKey, injT.Add(30*time.Minute), p+30*time.Second)) + res, err := score(rep, lab, "p474", "labels.json", nil, injT, injT.Add(24*time.Hour)) + if err != nil { + t.Fatal(err) + } + if res.TruePositives != 1 || res.PeriodMismatches != 0 || res.IncidentsDetected != 1 { + t.Fatalf("boundary period: %+v", res) + } +} + +// F5: an episode's end is its last transition to ended, not its first. A +// reopened episode that ends twice must be measured to the later end, or the +// stop latency is understated. +func TestScoreTakesTheLastEnded(t *testing.T) { + act := injT.Add(time.Hour) + first := injT.Add(2 * time.Hour) + last := injT.Add(3 * time.Hour) + rep := r2Report( + r2Cand("e1", "active", scoreKey, act, 300*time.Second), + r2Cand("e1", "ended", scoreKey, first, 0), + r2Cand("e1", "ended", scoreKey, last, 0), + ) + eps, noPeriod, err := episodes(rep, "p474", injT, injT.Add(24*time.Hour)) + if err != nil { + t.Fatal(err) + } + if len(eps) != 1 || noPeriod != 0 { + t.Fatalf("episodes %+v (no period %d)", eps, noPeriod) + } + if !eps[0].end.Equal(last) { + t.Fatalf("end %v, want the last ended %v", eps[0].end, last) + } +} From 8f55c9ee172094041a0b31aa03ddf8593a822fda Mon Sep 17 00:00:00 2001 From: dborup Date: Thu, 15 Oct 2026 22:18:16 +0200 Subject: [PATCH 06/12] fix(anomaly): give injected lines the export's shape Review F6. Injected lines were trivially distinguishable from exported ones in three ways that the detector does not read but a reader does: - Times carried nanosecond precision, while the export times every line from a whole-second observation time (time.Unix(ts, 0)). Injected event times, and the final_at derived from them, are now truncated to whole seconds, and final_after must be whole seconds exactly as the export's -settle must. - The payload size was constant. The new required spec field payload_bytes_spread draws each event's size uniformly from payload_bytes +/- spread; 0 keeps the old constant size, and a spread must be smaller than payload_bytes so a size can never reach zero. - The README example used the first hop as the whole route; it now shows a real multi-hop route. Truncation moves a time by less than a second towards the past, so the effective offset from the grid is in [-(jitter_max + 1s), jitter_max]; that is documented beside jitter_sd. Relates to #474 --- cmd/anomaly/README.md | 16 ++++- cmd/anomaly/inject.go | 59 ++++++++++++++----- cmd/anomaly/inject_test.go | 8 ++- cmd/anomaly/review_r2_test.go | 106 ++++++++++++++++++++++++++++++++++ 4 files changed, 168 insertions(+), 21 deletions(-) diff --git a/cmd/anomaly/README.md b/cmd/anomaly/README.md index a12d5b194..64e0c3996 100644 --- a/cmd/anomaly/README.md +++ b/cmd/anomaly/README.md @@ -239,9 +239,10 @@ single missed pulses, one 35-minute hole, one stable first hop). "route_class": "flood", "route_kind": "relay", "first_hop": "1:fd", - "full_route": "1:fd00", + "full_route": "3:fd0a1c", "wire_bytes": 60, "payload_bytes": 35, + "payload_bytes_spread": 6, "start": "2026-10-16T00:00:00Z", "stop": "2026-10-19T13:12:00Z", "period": "5m", @@ -258,6 +259,16 @@ single missed pulses, one 35-minute hole, one stable first hop). - `final_after` is added to each event time to give its `final_at`; set it to the export's `-settle`, so injected lines are final like the background's. + Like `-settle`, it must be whole seconds. +- **Times are whole seconds.** Every injected event time is truncated to a + whole second, as the export's are (`time.Unix(ts, 0)`), so a fractional + second never gives an injected line away. Truncation moves a time by less + than a second towards the past, so the effective offset from the grid is in + `[-(jitter_max + 1s), jitter_max]`. +- `payload_bytes_spread` draws each event's payload size uniformly from + `payload_bytes ± spread`; a real bot's payload length varies, and a + constant one is another giveaway. 0 keeps the size constant, and the + spread must be smaller than `payload_bytes`. - `jitter_sd` is the standard deviation of the offset from the nominal grid time, clamped to `±jitter_max`. The offset is not cumulative, so the train does not drift. @@ -266,7 +277,8 @@ single missed pulses, one 35-minute hole, one stable first hop). `multipart_max_delay`. A hole drops every pulse inside it. - `channel`, `first_hop`, `full_route`, `payload_type` and the sizes are the wire fields of a replay line: an injected line carries nothing a real line - does not. + does not. `full_route` is a whole observed route, not just the first hop + repeated, so an injected line's route looks like an exported one's. ### Labels diff --git a/cmd/anomaly/inject.go b/cmd/anomaly/inject.go index cf30730eb..e32860bf0 100644 --- a/cmd/anomaly/inject.go +++ b/cmd/anomaly/inject.go @@ -73,18 +73,23 @@ type labellingSpec struct { // export; the fields that describe the wire are exactly the fields of a // replay record, so an injected line carries nothing a real line does not. type injectBot struct { - Name string `json:"name"` - Channel string `json:"channel"` - PayloadType *int `json:"payload_type"` - RouteClass string `json:"route_class"` - RouteKind string `json:"route_kind"` - FirstHop string `json:"first_hop"` - FullRoute string `json:"full_route"` - WireBytes int `json:"wire_bytes"` - PayloadBytes *int `json:"payload_bytes"` - Start time.Time `json:"start"` - Stop time.Time `json:"stop"` - Period dur `json:"period"` + Name string `json:"name"` + Channel string `json:"channel"` + PayloadType *int `json:"payload_type"` + RouteClass string `json:"route_class"` + RouteKind string `json:"route_kind"` + FirstHop string `json:"first_hop"` + FullRoute string `json:"full_route"` + WireBytes int `json:"wire_bytes"` + PayloadBytes *int `json:"payload_bytes"` + // PayloadBytesSpread spreads the payload size: each event's size is + // drawn uniformly from payload_bytes +/- this many bytes. A real bot's + // payload length varies, so a constant one makes an injected line + // trivially distinguishable from an exported one. 0 keeps it constant. + PayloadBytesSpread *int `json:"payload_bytes_spread"` + Start time.Time `json:"start"` + Stop time.Time `json:"stop"` + Period dur `json:"period"` // JitterSD is the standard deviation of the per-pulse offset from the // nominal grid time; JitterMax clamps it. The offset is not cumulative, // so the train does not drift. @@ -312,8 +317,9 @@ func parseInjectSpec(b []byte) (injectSpec, error) { if s.Seed == nil { return injectSpec{}, errors.New("inject spec: seed is required") } - if s.FinalAfter == nil || *s.FinalAfter < 0 || time.Duration(*s.FinalAfter) > time.Hour { - return injectSpec{}, errors.New("inject spec: final_after is required and must be in 0..1h") + if s.FinalAfter == nil || *s.FinalAfter < 0 || time.Duration(*s.FinalAfter) > time.Hour || + time.Duration(*s.FinalAfter)%time.Second != 0 { + return injectSpec{}, errors.New("inject spec: final_after is required and must be whole seconds in 0..1h") } if err := s.Labelling.validate(); err != nil { return injectSpec{}, err @@ -393,6 +399,14 @@ func (b *injectBot) validate() error { return bad("jitter_sd is required and must be >= 0") case b.WireBytes <= 0: return bad("wire_bytes must be > 0") + case b.PayloadBytes != nil && *b.PayloadBytes <= 0: + return bad("payload_bytes must be > 0 when present") + case b.PayloadBytesSpread == nil || *b.PayloadBytesSpread < 0: + return bad("payload_bytes_spread is required and must be >= 0") + case *b.PayloadBytesSpread > 0 && b.PayloadBytes == nil: + return bad("payload_bytes_spread needs payload_bytes") + case b.PayloadBytes != nil && *b.PayloadBytesSpread >= *b.PayloadBytes: + return bad("payload_bytes_spread must be < payload_bytes") case b.Start.IsZero() || b.Stop.IsZero() || !b.Stop.After(b.Start): return bad("start and stop are required, with stop after start") case time.Duration(b.Period) <= 0: @@ -456,14 +470,27 @@ func generateBot(b *injectBot, seed uint64, finalAfter time.Duration, budget int if len(evs) >= budget { return fmt.Errorf("inject: more than -max-injected events; raise the bound or shorten the spec") } + // Parity with the export, which times every line from a whole-second + // observation time (time.Unix(ts, 0)) and takes a whole-second + // -settle: an injected line must not be recognisable by a fractional + // second. Truncation moves a time by less than one second towards + // the past, so the effective jitter is in [-(jitter_max + 1s), + // jitter_max]. + when = when.UTC().Truncate(time.Second) rec := anomaly.ReplayRecord{ - ID: fmt.Sprintf("%s%s-%d-%d", injPrefix, b.Name, pulse, part), Time: when.UTC(), - FinalAt: when.Add(finalAfter).UTC(), PayloadType: *b.PayloadType, Channel: b.Channel, + ID: fmt.Sprintf("%s%s-%d-%d", injPrefix, b.Name, pulse, part), Time: when, + FinalAt: when.Add(finalAfter), PayloadType: *b.PayloadType, Channel: b.Channel, RouteClass: b.RouteClass, RouteKind: b.RouteKind, FirstHop: b.FirstHop, FullRoute: b.FullRoute, WireBytes: b.WireBytes, Traffic: "unclassified", Decryption: "unknown", } if b.PayloadBytes != nil { pb := *b.PayloadBytes + // One payload-size draw per emitted event, taken only when the + // spread is nonzero, so a spread of 0 consumes no randomness and + // leaves every other draw where it was. + if sp := *b.PayloadBytesSpread; sp > 0 { + pb += int(p.float()*float64(2*sp+1)) - sp + } rec.PayloadBytes = &pb } // An injected line must be exactly as valid as an exported one. diff --git a/cmd/anomaly/inject_test.go b/cmd/anomaly/inject_test.go index 66e57e9ec..2d29758f2 100644 --- a/cmd/anomaly/inject_test.go +++ b/cmd/anomaly/inject_test.go @@ -30,6 +30,7 @@ type specOpts struct { jitterSD string missedFraction float64 multipartFraction float64 + payloadSpread int holes string // JSON array body, e.g. {"start":..,"duration":"35m"} extraBots string // JSON for further bots, with a leading comma leadBots string // JSON for bots before bot-a, with a trailing comma @@ -77,6 +78,7 @@ func specJSON(o specOpts) string { "full_route": "1:fd00", "wire_bytes": 60, "payload_bytes": 35, + "payload_bytes_spread": %d, "start": %q, "stop": %q, "period": %q, @@ -88,7 +90,7 @@ func specJSON(o specOpts) string { "multipart_max_delay": "55s", "holes": %s }%s] -}`, o.seed, o.minPulses, o.leadBots, o.channel, injAt(0), injAt(o.stopAfter), o.period, o.jitterSD, +}`, o.seed, o.minPulses, o.leadBots, o.channel, o.payloadSpread, injAt(0), injAt(o.stopAfter), o.period, o.jitterSD, o.missedFraction, o.multipartFraction, o.holes, o.extraBots) } @@ -283,7 +285,7 @@ func TestInjectIsDeterministic(t *testing.T) { func TestInjectBotsAreIndependent(t *testing.T) { botB := `{ "name": "bot-b", "channel": "22", "payload_type": 5, "route_class": "flood", "route_kind": "relay", - "first_hop": "1:ee", "full_route": "1:ee00", "wire_bytes": 60, "payload_bytes": 35, + "first_hop": "1:ee", "full_route": "1:ee00", "wire_bytes": 60, "payload_bytes": 35, "payload_bytes_spread": 0, "start": "` + injAt(0) + `", "stop": "` + injAt(time.Hour) + `", "period": "7m", "jitter_sd": "2s", "jitter_max": "10s", "missed_fraction": 0, "multipart_fraction": 0, "multipart_max_extra": 2, "multipart_max_delay": "55s", "holes": []}` @@ -552,7 +554,7 @@ func TestInjectSpecRejectsMissingAndUnknownFields(t *testing.T) { "duplicate bot": strings.Replace(base, `"holes": [] }]`, `"holes": [] }, {"name": "bot-a", "channel": "22", "payload_type": 5, "route_class": "flood", "route_kind": "relay", - "first_hop": "1:ee", "full_route": "1:ee00", "wire_bytes": 60, "payload_bytes": 35, + "first_hop": "1:ee", "full_route": "1:ee00", "wire_bytes": 60, "payload_bytes": 35, "payload_bytes_spread": 0, "start": "`+injAt(0)+`", "stop": "`+injAt(time.Hour)+`", "period": "7m", "jitter_sd": "0s", "jitter_max": "10s", "missed_fraction": 0, "multipart_fraction": 0, "multipart_max_extra": 2, "multipart_max_delay": "55s", "holes": []}]`, 1), diff --git a/cmd/anomaly/review_r2_test.go b/cmd/anomaly/review_r2_test.go index 946763195..3d35cb790 100644 --- a/cmd/anomaly/review_r2_test.go +++ b/cmd/anomaly/review_r2_test.go @@ -1,6 +1,7 @@ package main import ( + "strings" "testing" "time" ) @@ -149,3 +150,108 @@ func TestScoreTakesTheLastEnded(t *testing.T) { t.Fatalf("end %v, want the last ended %v", eps[0].end, last) } } + +// F6: the export times every line from a whole-second observation time, so +// an injected line must too. A fractional second would make every injected +// line trivially distinguishable from an exported one. +func TestInjectTimesAreWholeSeconds(t *testing.T) { + r := injectInto(t, specJSON(specOpts{seed: 31, jitterSD: "3s", multipartFraction: 0.5}), + backgroundJSONL(40)) + recs := injectedLines(t, r) + if len(recs) < 20 { + t.Fatalf("only %d injected lines", len(recs)) + } + for _, rec := range recs { + if rec.Time.Nanosecond() != 0 || rec.FinalAt.Nanosecond() != 0 { + t.Fatalf("%s: time %v final_at %v are not whole seconds", rec.ID, rec.Time, rec.FinalAt) + } + } + // The same on the wire: no injected line carries a fractional second. + for _, l := range r.lines { + if !strings.Contains(l, `"id":"`+injPrefix) { + continue + } + for _, f := range []string{`"time":"`, `"final_at":"`} { + i := strings.Index(l, f) + if i < 0 { + t.Fatalf("line %q has no %s", l, f) + } + v := l[i+len(f):] + v = v[:strings.IndexByte(v, '"')] + if strings.Contains(v, ".") { + t.Fatalf("%s %s carries a fractional second in %q", f, v, l) + } + } + } + // The labels are derived from the same whole-second pulses. + for _, inc := range r.labels.Incidents { + if inc.Start.Nanosecond() != 0 || inc.Stop.Nanosecond() != 0 { + t.Fatalf("incident %+v is not on whole seconds", inc) + } + } +} + +// F6: a real bot's payload length varies. payload_bytes_spread draws each +// event's size uniformly from payload_bytes +/- the spread, and a spread of +// 0 keeps the old constant size. +func TestInjectPayloadSizeVariesWithinTheSpread(t *testing.T) { + const base, spread = 35, 6 + r := injectInto(t, specJSON(specOpts{seed: 31, payloadSpread: spread}), backgroundJSONL(40)) + seen := map[int]bool{} + for _, rec := range injectedLines(t, r) { + if rec.PayloadBytes == nil { + t.Fatalf("%s has no payload_bytes", rec.ID) + } + pb := *rec.PayloadBytes + if pb < base-spread || pb > base+spread { + t.Fatalf("%s payload %d outside %d +/- %d", rec.ID, pb, base, spread) + } + seen[pb] = true + } + if len(seen) < 5 { + t.Fatalf("payload size barely varies: %v", seen) + } + flat := injectInto(t, specJSON(specOpts{seed: 31}), backgroundJSONL(40)) + for _, rec := range injectedLines(t, flat) { + if *rec.PayloadBytes != base { + t.Fatalf("spread 0 must keep the size constant, got %d", *rec.PayloadBytes) + } + } +} + +// F6: the spread is a required field with no default, and it may never push +// the payload size to zero or below. +func TestInjectPayloadSpreadIsValidated(t *testing.T) { + bot := func(extra string) string { + return `{"seed":7,"final_after":"30s", + "labelling":{"pulse_merge_gap":"60s","min_period":"1m","max_period":"1h","min_pulses":8, + "max_harmonic":3,"tolerance_abs":"30s","tolerance_rel":0.1}, + "bots":[{"name":"bot-a","channel":"7f","payload_type":5,"route_class":"flood","route_kind":"relay", + "first_hop":"1:fd","full_route":"2:fd0a","wire_bytes":60,` + extra + ` + "start":"` + injAt(0) + `","stop":"` + injAt(time.Hour) + `","period":"5m","jitter_sd":"3s", + "jitter_max":"10s","missed_fraction":0,"multipart_fraction":0,"multipart_max_extra":2, + "multipart_max_delay":"55s","holes":[]}]}` + } + for _, tc := range []struct{ name, extra string }{ + {"missing", `"payload_bytes":35,`}, + {"negative", `"payload_bytes":35,"payload_bytes_spread":-1,`}, + {"as wide as the payload", `"payload_bytes":35,"payload_bytes_spread":35,`}, + {"without a payload size", `"payload_bytes_spread":3,`}, + {"zero payload size", `"payload_bytes":0,"payload_bytes_spread":0,`}, + } { + if _, err := parseInjectSpec([]byte(bot(tc.extra))); err == nil { + t.Fatalf("%s: want an error", tc.name) + } + } + if _, err := parseInjectSpec([]byte(bot(`"payload_bytes":35,"payload_bytes_spread":34,`))); err != nil { + t.Fatalf("a spread one below the payload size is legal: %v", err) + } +} + +// F6: final_after must be whole seconds, exactly as the export's -settle is. +func TestInjectFinalAfterIsWholeSeconds(t *testing.T) { + spec := strings.Replace(specJSON(specOpts{seed: 7}), `"final_after": "30s"`, `"final_after": "30.5s"`, 1) + if _, err := parseInjectSpec([]byte(spec)); err == nil { + t.Fatal("a fractional final_after must be refused") + } +} From 98a58fd3c338adaeafff8b8c08fbccd219bfe838 Mon Sep 17 00:00:00 2001 From: dborup Date: Thu, 15 Oct 2026 22:26:43 +0200 Subject: [PATCH 07/12] feat(anomaly): put the panel on real stream keys and score it differentially Review F2 (blocking), F3 and F8. F2. Round 1's panel sat on channel bytes a0-a8 and first hops fdff00-fdff08, which no real transmission carries, so every injected bot had its stream key to itself and arm B measured detection in isolation rather than detection inside real traffic. Three changes make a shared-key panel possible and scorable: - New "keys" subcommand. It reads an export and ranks its stream keys by row count, reporting per key the wire fields a bot needs to land on it: the canonical key, the channel hash, the route class and kind, the first hop, the most common whole route and the median payload and wire sizes with a payload spread. The filter is mechanical and restated in the output (requested payload type, a channel hash present, route kind relay, channel not excluded) and every skipped line is counted. It is blind by construction: it reads the export only and counts rows, never runs the detector and reads no report, no labels and no metrics. -max-keys is a hard bound, as the export's -max-rows is. - inject writes each bot's stream key into the labels, derived from the same Event the detector sees. - score matches an episode to an incident on that key when the labels carry one (falling back to the channel hash when they do not), so a different route group on the same busy channel is no longer credited to the bot. New -baseline takes the report of the same window without the injection: an episode it already has on the same key, overlapping in time, is the background's - neither a true positive nor a false positive, counted as baseline_episodes. That is the differential reading a shared key needs. F3. Five clean bots alarm at the earliest pulse the rule allows, so the median sits on that floor and hides the realistic shapes. score now reports the nearest-rank p75 and the maximum beside every median (time to detect, also in units of P, stop latency, episodes per incident), so the tail is visible without reading the per-incident table. F8. The metrics recorded the local path of the labels file; they now record its file name only, and the same for the new baseline report. A path can carry a user or a host name and a metrics file may be shared. Relates to #474 --- cmd/anomaly/README.md | 2 +- cmd/anomaly/inject.go | 60 ++++- cmd/anomaly/inject_test.go | 10 +- cmd/anomaly/keys.go | 402 ++++++++++++++++++++++++++++++++++ cmd/anomaly/main.go | 18 +- cmd/anomaly/review_r2_test.go | 392 ++++++++++++++++++++++++++++++++- cmd/anomaly/score.go | 182 +++++++++++++-- 7 files changed, 1016 insertions(+), 50 deletions(-) create mode 100644 cmd/anomaly/keys.go diff --git a/cmd/anomaly/README.md b/cmd/anomaly/README.md index 64e0c3996..db32d2a3b 100644 --- a/cmd/anomaly/README.md +++ b/cmd/anomaly/README.md @@ -239,7 +239,7 @@ single missed pulses, one 35-minute hole, one stable first hop). "route_class": "flood", "route_kind": "relay", "first_hop": "1:fd", - "full_route": "3:fd0a1c", + "full_route": "1:fd0a1c", "wire_bytes": 60, "payload_bytes": 35, "payload_bytes_spread": 6, diff --git a/cmd/anomaly/inject.go b/cmd/anomaly/inject.go index e32860bf0..66454e07d 100644 --- a/cmd/anomaly/inject.go +++ b/cmd/anomaly/inject.go @@ -168,8 +168,14 @@ func botPRNG(seed uint64, name string) *prng { // botStats counts what one bot produced. type botStats struct { - Name string `json:"name"` - Channel string `json:"channel"` + Name string `json:"name"` + Channel string `json:"channel"` + // Key is the stream key every one of this bot's events falls on, in the + // library's canonical encoding. The evaluation needs it: a bot placed on + // a key real traffic uses shares that key with the background, and + // scoring on the channel hash alone could then credit a real train to + // the bot (or the other way round). + Key string `json:"key"` PulsesScheduled int `json:"pulses_scheduled"` PulsesEmitted int `json:"pulses_emitted"` PulsesMissed int `json:"pulses_missed"` @@ -195,12 +201,16 @@ type injectSummary struct { // range, with the bot's period. It holds no key material, no observer // identity and no text. type Incident struct { - Bot string `json:"bot"` - Channel string `json:"channel"` - Start time.Time `json:"start"` - Stop time.Time `json:"stop"` - Period dur `json:"period"` - Pulses int `json:"pulses"` + Bot string `json:"bot"` + Channel string `json:"channel"` + // Key is the bot's stream key (see botStats.Key). score matches on it + // when it is present, so an episode of real traffic on the same channel + // but a different route group is never mistaken for the bot. + Key string `json:"key"` + Start time.Time `json:"start"` + Stop time.Time `json:"stop"` + Period dur `json:"period"` + Pulses int `json:"pulses"` } // InjectLabels is the labels file: the ground truth of one injection run. @@ -441,7 +451,7 @@ func generate(s injectSpec, maxInjected int) ([]injEvent, []botStats, []Incident if err != nil { return nil, nil, nil, err } - inc := labelIncidents(b.Name, b.Channel, botEvs, s.Labelling) + inc := labelIncidents(b.Name, b.Channel, st.Key, botEvs, s.Labelling) st.Incidents = len(inc) evs = append(evs, botEvs...) stats = append(stats, st) @@ -465,6 +475,14 @@ func generateBot(b *injectBot, seed uint64, finalAfter time.Duration, budget int p := botPRNG(seed, b.Name) st := botStats{Name: b.Name, Channel: b.Channel} period := time.Duration(b.Period) + // Every one of a bot's events carries the same wire fields, so the bot + // has exactly one stream key. Derive it once, from the same Event the + // detector would see. + key, err := botStreamKey(b, finalAfter) + if err != nil { + return nil, botStats{}, err + } + st.Key = key var evs []injEvent emit := func(pulse, part int, when time.Time) error { if len(evs) >= budget { @@ -552,6 +570,26 @@ func generateBot(b *injectBot, seed uint64, finalAfter time.Duration, budget int return evs, st, nil } +// botStreamKey returns the canonical stream key of a bot's events. It builds +// the same ReplayRecord emit builds, so a spec whose wire fields do not make +// a valid event fails here rather than silently labelling nothing. +func botStreamKey(b *injectBot, finalAfter time.Duration) (string, error) { + rec := anomaly.ReplayRecord{ID: injPrefix + b.Name + "-key", Time: b.Start.UTC().Truncate(time.Second), + PayloadType: *b.PayloadType, Channel: b.Channel, RouteClass: b.RouteClass, RouteKind: b.RouteKind, + FirstHop: b.FirstHop, FullRoute: b.FullRoute, WireBytes: b.WireBytes, + Traffic: "unclassified", Decryption: "unknown"} + rec.FinalAt = rec.Time.Add(finalAfter) + if b.PayloadBytes != nil { + pb := *b.PayloadBytes + rec.PayloadBytes = &pb + } + ev, err := rec.Event() + if err != nil { + return "", fmt.Errorf("inject: bot %q: %w", b.Name, err) + } + return anomaly.StreamKeyOf(ev).String(), nil +} + func inHole(holes []injectHole, t time.Time) bool { for _, h := range holes { if !t.Before(h.Start) && t.Before(h.Start.Add(time.Duration(h.Duration))) { @@ -562,7 +600,7 @@ func inHole(holes []injectHole, t time.Time) bool { } // labelIncidents applies the labelling recipe to one bot's emitted events. -func labelIncidents(bot, channel string, evs []injEvent, l labellingSpec) []Incident { +func labelIncidents(bot, channel, key string, evs []injEvent, l labellingSpec) []Incident { if len(evs) == 0 { return nil } @@ -616,7 +654,7 @@ func labelIncidents(bot, channel string, evs []injEvent, l labellingSpec) []Inci runStart := 0 flush := func(end int) { // pulses[runStart:end+1] is the run if end-runStart+1 >= l.MinPulses { - out = append(out, Incident{Bot: bot, Channel: channel, Start: time.Unix(0, pulses[runStart]).UTC(), + out = append(out, Incident{Bot: bot, Channel: channel, Key: key, Start: time.Unix(0, pulses[runStart]).UTC(), Stop: time.Unix(0, pulses[end]).UTC(), Period: dur(period), Pulses: end - runStart + 1}) } } diff --git a/cmd/anomaly/inject_test.go b/cmd/anomaly/inject_test.go index 2d29758f2..563cd6f9a 100644 --- a/cmd/anomaly/inject_test.go +++ b/cmd/anomaly/inject_test.go @@ -576,7 +576,7 @@ func TestInjectLabelsAreKeyFreeAndMarkedSynthetic(t *testing.T) { if err != nil { t.Fatal(err) } - for _, banned := range []string{"1:fd", "fd00", "observer", "tx1", injPrefix, "raw_hex", "public_key"} { + for _, banned := range []string{"fd00", "observer", "tx1", injPrefix, "raw_hex", "public_key"} { if bytes.Contains(raw, []byte(banned)) { t.Fatalf("labels file carries %q:\n%s", banned, raw) } @@ -585,4 +585,12 @@ func TestInjectLabelsAreKeyFreeAndMarkedSynthetic(t *testing.T) { if !bytes.Contains(raw, []byte(`"channel": "7f"`)) || !bytes.Contains(raw, []byte(`"period": "5m0s"`)) { t.Fatalf("labels file is missing the ground truth:\n%s", raw) } + // Since #474 round 2 it also carries the bot's stream key, which + // scoring needs to tell a bot apart from real traffic on the same + // channel hash. The key holds the payload type, the channel hash, the + // route class and kind and the first hop - fields the export itself + // carries - and nothing else: no whole route, no ids, no observers. + if !bytes.Contains(raw, []byte(`"key": "`+scoreKey+`"`)) { + t.Fatalf("labels file is missing the stream key:\n%s", raw) + } } diff --git a/cmd/anomaly/keys.go b/cmd/anomaly/keys.go new file mode 100644 index 000000000..4c426ec77 --- /dev/null +++ b/cmd/anomaly/keys.go @@ -0,0 +1,402 @@ +package main + +import ( + "bufio" + "encoding/hex" + "encoding/json" + "errors" + "flag" + "fmt" + "io" + "math" + "os" + "sort" + "strings" + "time" + + "github.com/meshcore-analyzer/anomaly" +) + +// Issue #474, phase 3: rank the stream keys of an exported background by row +// count, so an injection panel can be placed on keys real traffic actually +// uses instead of on keys of its own. +// +// Why this exists. The stream key is payload type + channel hash + route +// class + route kind + first hop. A bot on a channel byte and a first hop +// that no real transmission carries has that key to itself, so the detector +// sees a clean train and recall measured on it says nothing about finding a +// bot inside real traffic: a real spam bot on a busy channel shares its key +// with everyone else entering through the same repeater. +// +// Blind by construction. This command reads the export only and counts rows. +// It never runs the detector, reads no report, no labels and no metrics, and +// has no idea which keys would alarm: the ranking is row counts and nothing +// else, and ties break on the canonical key encoding. So a panel built from +// its output is chosen mechanically and before any detection result exists. +// +// Offline and file-only, like the rest of the tool: no database, no -db +// flag, no network, no clock. + +// maxKeySizeBuckets bounds the distinct payload and wire sizes tracked per +// key, and maxKeyRouteVariants the distinct full routes. Both are counting +// bounds only: a key that exceeds one keeps its count and its times, and the +// overflow is reported per key. +const ( + maxKeySizeBuckets = 512 + maxKeyRouteVariants = 64 +) + +// keyStat is one ranked stream key. Every field is derived from the +// background lines on that key: hashes, hop prefixes, sizes, counts and +// times, which is what the export itself carries. No ids, observers, names +// or text. +type keyStat struct { + Rank int `json:"rank"` + Key string `json:"key"` + PayloadType int `json:"payload_type"` + Channel string `json:"channel"` + RouteClass string `json:"route_class"` + RouteKind string `json:"route_kind"` + FirstHop string `json:"first_hop"` + // FullRoute is the most common whole route seen on the key, ties broken + // by the smallest encoding. + FullRoute string `json:"full_route"` + // WireBytesMedian and the payload figures are the realistic sizes an + // injected bot on this key should use. PayloadBytesSpread is + // (p90 - p10) / 2, so payload_bytes +/- spread covers the middle of the + // observed spread; it is clamped below PayloadBytesMedian. + WireBytesMedian int `json:"wire_bytes_median"` + PayloadBytesMedian int `json:"payload_bytes_median,omitempty"` + PayloadBytesSpread int `json:"payload_bytes_spread"` + Events int `json:"events"` + // SizeBucketsDropped and RouteVariantsDropped count values that did not + // fit the per-key bounds above; they affect the medians and the route + // mode only, never Events. + SizeBucketsDropped int `json:"size_buckets_dropped,omitempty"` + RouteVariantsDropped int `json:"route_variants_dropped,omitempty"` + First time.Time `json:"first"` + Last time.Time `json:"last"` +} + +// keysResult is the output file. +type keysResult struct { + Experimental bool `json:"experimental"` + // PayloadType, ExcludeChannels and the filter below are the mechanical + // selection rule, restated in the output so a panel can be audited + // against it. + PayloadType int `json:"payload_type"` + ExcludeChannels []string `json:"exclude_channels"` + Filter string `json:"filter"` + InputLines int `json:"input_lines"` + Considered int `json:"considered"` + SkippedType int `json:"skipped_payload_type"` + SkippedNoChannel int `json:"skipped_no_channel"` + SkippedNotRelay int `json:"skipped_not_relay"` + SkippedExcluded int `json:"skipped_excluded_channel"` + DistinctKeys int `json:"distinct_keys"` + Top []keyStat `json:"top"` +} + +// keyAgg accumulates one key's lines. +type keyAgg struct { + k anomaly.Key + events int + first, last time.Time + payload, wire map[int]int + routes map[string]int + sizeDropped int + routeDropped int +} + +func runKeys(args []string, stdin io.Reader, stdout, stderr io.Writer) int { + fs := flag.NewFlagSet("keys", flag.ContinueOnError) + fs.SetOutput(stderr) + input := fs.String("input", "", "exported ReplayRecord JSON Lines, or - for stdin (required)") + payloadType := fs.Int("payload-type", -1, "payload type to rank (required)") + top := fs.Int("top", 0, "how many keys to report (required)") + exclude := fs.String("exclude-channels", "", "comma-separated channel hashes to leave out, lower-case hex") + maxKeys := fs.Int("max-keys", 0, "most distinct keys allowed (required, a hard bound)") + maxLine := fs.Int("max-line-bytes", 0, "longest accepted input line (required)") + out := fs.String("out", "", "output file (must not exist); default stdout") + if err := fs.Parse(args); err != nil { + return exitUsage + } + usageErr := func(err error) int { + fmt.Fprintf(stderr, "keys: %v\n", err) + return exitUsage + } + if fs.NArg() != 0 { + return usageErr(errors.New("unexpected arguments")) + } + if *input == "" { + return usageErr(errors.New("-input is required")) + } + if *payloadType < 0 || *payloadType > int(anomaly.MaxPayloadType) { + return usageErr(fmt.Errorf("-payload-type is required and must be in 0..%d", anomaly.MaxPayloadType)) + } + if *top <= 0 || *maxKeys <= 0 || *maxLine <= 0 { + return usageErr(errors.New("-top, -max-keys and -max-line-bytes must be > 0")) + } + excl, err := parseExcludeList(*exclude) + if err != nil { + return usageErr(err) + } + fail := func(err error) int { + fmt.Fprintf(stderr, "keys: %v\n", err) + return exitFail + } + in := stdin + if *input != "-" { + f, err := os.Open(*input) + if err != nil { + return fail(err) + } + defer f.Close() + in = f + } + res, err := rankKeys(in, *payloadType, excl, *top, *maxKeys, *maxLine) + if err != nil { + return fail(err) + } + b, err := json.MarshalIndent(res, "", " ") + if err != nil { + return fail(err) + } + b = append(b, '\n') + if *out == "" { + _, err = stdout.Write(b) + } else { + err = writeNew(*out, b) + } + if err != nil { + return fail(err) + } + return exitOK +} + +// parseExcludeList parses the -exclude-channels list: lower-case hex hashes, +// no duplicates, nothing empty. The same canonical form the rule's +// exclude_channels takes. +func parseExcludeList(s string) ([]string, error) { + out := []string{} + seen := map[string]bool{} + for _, v := range strings.Split(s, ",") { + v = strings.TrimSpace(v) + if v == "" { + continue + } + if strings.ToLower(v) != v { + return nil, errors.New("-exclude-channels wants lower-case hex hashes") + } + b, err := hex.DecodeString(v) + if err != nil || len(b) == 0 { + return nil, fmt.Errorf("-exclude-channels: %q is not a channel hash", v) + } + if _, err := anomaly.NewChannelHash(b); err != nil { + return nil, fmt.Errorf("-exclude-channels: %q: %w", v, err) + } + if seen[v] { + return nil, fmt.Errorf("-exclude-channels: %q twice", v) + } + seen[v] = true + out = append(out, v) + } + sort.Strings(out) + return out, nil +} + +// rankKeys counts the background's stream keys and returns the top ones. +// +// The filter is mechanical and fixed: the line's payload type must be the +// requested one, it must carry a channel hash (the periodic rule's target is +// a channel bot), its route kind must be relay (only then is the first hop +// part of the stream key, so only then can a bot share the key), and its +// channel must not be excluded. +func rankKeys(r io.Reader, payloadType int, exclude []string, top, maxKeys, maxLine int) (keysResult, error) { + res := keysResult{Experimental: true, PayloadType: payloadType, ExcludeChannels: exclude, + Filter: "payload_type == -payload-type, channel hash present, route_kind == relay, channel not excluded", + Top: []keyStat{}} + excluded := map[string]bool{} + for _, v := range exclude { + excluded[v] = true + } + aggs := map[anomaly.Key]*keyAgg{} + sc := bufio.NewScanner(r) + limit := maxLine + 2 + initial := 4096 + if limit < initial { + initial = limit + } + sc.Buffer(make([]byte, 0, initial), limit) + for sc.Scan() { + line := sc.Bytes() + if len(line) > maxLine { + return res, errors.New("input line longer than -max-line-bytes") + } + if len(strings.TrimSpace(string(line))) == 0 { + continue + } + res.InputLines++ + var rec anomaly.ReplayRecord + if err := json.Unmarshal(line, &rec); err != nil { + return res, fmt.Errorf("line %d: %w", res.InputLines, err) + } + if rec.PayloadType != payloadType { + res.SkippedType++ + continue + } + if rec.Channel == "" { + res.SkippedNoChannel++ + continue + } + if rec.RouteKind != "relay" { + res.SkippedNotRelay++ + continue + } + if excluded[rec.Channel] { + res.SkippedExcluded++ + continue + } + ev, err := rec.Event() + if err != nil { + return res, fmt.Errorf("line %d: %w", res.InputLines, err) + } + k := anomaly.StreamKeyOf(ev) + a := aggs[k] + if a == nil { + if len(aggs) >= maxKeys { + return res, errors.New("more than -max-keys distinct keys; raise the bound") + } + a = &keyAgg{k: k, payload: map[int]int{}, wire: map[int]int{}, routes: map[string]int{}} + aggs[k] = a + } + res.Considered++ + a.events++ + if a.first.IsZero() || rec.Time.Before(a.first) { + a.first = rec.Time.UTC() + } + if rec.Time.After(a.last) { + a.last = rec.Time.UTC() + } + bump(a.wire, rec.WireBytes, maxKeySizeBuckets, &a.sizeDropped) + if rec.PayloadBytes != nil { + bump(a.payload, *rec.PayloadBytes, maxKeySizeBuckets, &a.sizeDropped) + } + bumpStr(a.routes, rec.FullRoute, maxKeyRouteVariants, &a.routeDropped) + } + if err := sc.Err(); err != nil { + return res, fmt.Errorf("input: %w", err) + } + res.DistinctKeys = len(aggs) + ranked := make([]*keyAgg, 0, len(aggs)) + for _, a := range aggs { + ranked = append(ranked, a) + } + // Row count, then the canonical key encoding: a total order that does + // not depend on map iteration. + sort.Slice(ranked, func(i, j int) bool { + if ranked[i].events != ranked[j].events { + return ranked[i].events > ranked[j].events + } + return ranked[i].k.String() < ranked[j].k.String() + }) + if len(ranked) > top { + ranked = ranked[:top] + } + for i, a := range ranked { + res.Top = append(res.Top, a.stat(i+1)) + } + return res, nil +} + +func bump(m map[int]int, v, max int, dropped *int) { + if _, ok := m[v]; !ok && len(m) >= max { + *dropped++ + return + } + m[v]++ +} + +func bumpStr(m map[string]int, v string, max int, dropped *int) { + if _, ok := m[v]; !ok && len(m) >= max { + *dropped++ + return + } + m[v]++ +} + +func (a *keyAgg) stat(rank int) keyStat { + s := keyStat{Rank: rank, Key: a.k.String(), PayloadType: int(a.k.PayloadType), + Channel: hexChannel(a.k), RouteClass: a.k.RouteClass.String(), RouteKind: a.k.RouteKind.String(), + FirstHop: encodeHop(a.k.FirstHop), Events: a.events, First: a.first, Last: a.last, + SizeBucketsDropped: a.sizeDropped, RouteVariantsDropped: a.routeDropped} + s.FullRoute = modeString(a.routes) + s.WireBytesMedian = int(histPercentile(a.wire, 0.5)) + if len(a.payload) > 0 { + med := int(histPercentile(a.payload, 0.5)) + p10 := int(histPercentile(a.payload, 0.1)) + p90 := int(histPercentile(a.payload, 0.9)) + s.PayloadBytesMedian = med + spread := (p90 - p10) / 2 + if spread >= med { + spread = med - 1 + } + if spread < 0 { + spread = 0 + } + s.PayloadBytesSpread = spread + } + return s +} + +// encodeHop renders a first hop in the "width:hex" form a replay record and +// an injection spec use. +func encodeHop(h anomaly.Hop) string { + b := h.Bytes() + if len(b) == 0 { + return "" + } + return fmt.Sprintf("%d:%x", len(b), b) +} + +// modeString returns the most common value, ties broken by the smallest. +func modeString(m map[string]int) string { + best, bestN := "", -1 + for v, n := range m { + if n > bestN || (n == bestN && v < best) { + best, bestN = v, n + } + } + return best +} + +// histPercentile is the nearest-rank percentile of a value histogram: the +// value at index ceil(q*n)-1 of the sorted values, clamped into range. It is +// a pure function of the histogram, so it does not depend on map order. +func histPercentile(m map[int]int, q float64) int64 { + n := 0 + vals := make([]int, 0, len(m)) + for v, c := range m { + n += c + vals = append(vals, v) + } + if n == 0 { + return 0 + } + sort.Ints(vals) + idx := int(math.Ceil(q*float64(n))) - 1 + if idx < 0 { + idx = 0 + } + if idx > n-1 { + idx = n - 1 + } + seen := 0 + for _, v := range vals { + seen += m[v] + if seen > idx { + return int64(v) + } + } + return int64(vals[len(vals)-1]) +} diff --git a/cmd/anomaly/main.go b/cmd/anomaly/main.go index d730a0110..2623c37f9 100644 --- a/cmd/anomaly/main.go +++ b/cmd/anomaly/main.go @@ -5,16 +5,19 @@ // corescope-anomaly export -db PATH [-immutable] -start T -end T -settle D -max-rows N -out FILE // [-backlog-silence D -backlog-span D] // corescope-anomaly replay -config FILE -input FILE|- -start T -end T -max-line-bytes N [-out FILE] +// corescope-anomaly keys -input FILE|- -payload-type N -top N [-exclude-channels LIST] -max-keys N -max-line-bytes N [-out FILE] // corescope-anomaly inject -spec FILE -input FILE|- -out FILE -labels FILE -max-injected N -max-line-bytes N -// corescope-anomaly score -report FILE [-labels FILE] -rule NAME -start T -end T [-out FILE] +// corescope-anomaly score -report FILE [-labels FILE] [-baseline FILE] -rule NAME -start T -end T [-out FILE] // // export reads a CoreScope SQLite database read-only (mode=ro; -immutable for // a static snapshot copy) and writes anomaly.ReplayRecord JSON Lines built // from on-wire fields only, with per-export IDs. replay runs anomaly.Replay -// over such lines and prints the deterministic report. inject overlays -// synthetic channel-bot trains (clearly marked "inj-..." ids) on such lines -// and writes the ground truth beside them; score measures a replay report -// against that ground truth (#474). Only export reads a database, and +// over such lines and prints the deterministic report. keys ranks the stream +// keys of such lines by row count, so an injection panel can be placed on +// keys real traffic uses. inject overlays synthetic channel-bot trains +// (clearly marked "inj-..." ids) on such lines and writes the ground truth +// beside them; score measures a replay report against that ground truth +// (#474). Only export reads a database, and // nothing here ever writes to one. Every bound is an explicit flag; there // are no defaults. package main @@ -31,8 +34,9 @@ const usage = `usage: corescope-anomaly export -db PATH [-immutable] -start RFC3339 -end RFC3339 -settle DURATION -max-rows N -out FILE [-backlog-silence DURATION -backlog-span DURATION] corescope-anomaly replay -config FILE -input FILE|- -start RFC3339 -end RFC3339 -max-line-bytes N [-out FILE] + corescope-anomaly keys -input FILE|- -payload-type N -top N [-exclude-channels LIST] -max-keys N -max-line-bytes N [-out FILE] corescope-anomaly inject -spec FILE -input FILE|- -out FILE -labels FILE -max-injected N -max-line-bytes N - corescope-anomaly score -report FILE [-labels FILE] -rule NAME -start RFC3339 -end RFC3339 [-out FILE] + corescope-anomaly score -report FILE [-labels FILE] [-baseline FILE] -rule NAME -start RFC3339 -end RFC3339 [-out FILE] ` // Exit codes: 0 success, 1 runtime failure, 2 usage error. @@ -56,6 +60,8 @@ func run(args []string, stdin io.Reader, stdout, stderr io.Writer) int { return runExport(args[1:], stderr) case "replay": return runReplay(args[1:], stdin, stdout, stderr) + case "keys": + return runKeys(args[1:], stdin, stdout, stderr) case "inject": return runInject(args[1:], stdin, stderr) case "score": diff --git a/cmd/anomaly/review_r2_test.go b/cmd/anomaly/review_r2_test.go index 3d35cb790..148d860a9 100644 --- a/cmd/anomaly/review_r2_test.go +++ b/cmd/anomaly/review_r2_test.go @@ -1,9 +1,16 @@ package main import ( + "bytes" + "encoding/json" + "fmt" + "os" + "path/filepath" "strings" "testing" "time" + + "github.com/meshcore-analyzer/anomaly" ) // Round-2 review of PR #478 (#474): the boundary cases the first round's @@ -44,11 +51,11 @@ func grid(n int, p time.Duration) []time.Duration { func TestLabelIncidentsAtExactlyMinPulses(t *testing.T) { const p = 300 * time.Second exact := grid(8, p) - inc := labelIncidents("bot-a", "7f", r2Pulses(exact...), r2Labelling(8)) + inc := labelIncidents("bot-a", "7f", scoreKey, r2Pulses(exact...), r2Labelling(8)) if len(inc) != 1 || inc[0].Pulses != 8 { t.Fatalf("exactly 8 pulses must be one 8-pulse incident: %+v", inc) } - if got := labelIncidents("bot-a", "7f", r2Pulses(exact[:7]...), r2Labelling(8)); len(got) != 0 { + if got := labelIncidents("bot-a", "7f", scoreKey, r2Pulses(exact[:7]...), r2Labelling(8)); len(got) != 0 { t.Fatalf("7 pulses must not be an incident: %+v", got) } } @@ -61,13 +68,13 @@ func TestLabelIncidentsAtTheToleranceBoundary(t *testing.T) { at := func(last time.Duration) []time.Duration { return append(grid(7, p), 6*p+last) } - inc := labelIncidents("bot-a", "7f", r2Pulses(at(330*time.Second)...), r2Labelling(8)) + inc := labelIncidents("bot-a", "7f", scoreKey, r2Pulses(at(330*time.Second)...), r2Labelling(8)) if len(inc) != 1 || inc[0].Pulses != 8 { t.Fatalf("a gap exactly at the tolerance must stay in the run: %+v", inc) } // One nanosecond further out and the run breaks before the last pulse, // which leaves 7 pulses: below min_pulses, so no incident at all. - if got := labelIncidents("bot-a", "7f", r2Pulses(at(330*time.Second+1)...), r2Labelling(8)); len(got) != 0 { + if got := labelIncidents("bot-a", "7f", scoreKey, r2Pulses(at(330*time.Second+1)...), r2Labelling(8)); len(got) != 0 { t.Fatalf("a gap past the tolerance must break the run: %+v", got) } } @@ -76,13 +83,13 @@ func TestLabelIncidentsAtTheToleranceBoundary(t *testing.T) { // merge keeps events strictly closer than the gap together. func TestLabelIncidentsAtThePulseMergeGapBoundary(t *testing.T) { l := r2Labelling(2) - inc := labelIncidents("bot-a", "7f", r2Pulses(0, 60*time.Second), l) + inc := labelIncidents("bot-a", "7f", scoreKey, r2Pulses(0, 60*time.Second), l) if len(inc) != 1 || inc[0].Pulses != 2 || time.Duration(inc[0].Period) != 60*time.Second { t.Fatalf("events exactly 60 s apart are two pulses: %+v", inc) } // One nanosecond closer and they merge, which leaves a single pulse and // no incident. - if got := labelIncidents("bot-a", "7f", r2Pulses(0, 60*time.Second-1), l); len(got) != 0 { + if got := labelIncidents("bot-a", "7f", scoreKey, r2Pulses(0, 60*time.Second-1), l); len(got) != 0 { t.Fatalf("events closer than the gap are one pulse: %+v", got) } } @@ -114,11 +121,11 @@ func TestScorePeriodFitsAtTheToleranceBoundary(t *testing.T) { } // The same through score: the boundary episode is a true positive and // raises no period mismatch. - inc := Incident{Bot: "bot-a", Channel: "7f", Start: injT, Stop: injT.Add(time.Hour), + inc := Incident{Bot: "bot-a", Channel: "7f", Key: scoreKey, Start: injT, Stop: injT.Add(time.Hour), Period: dur(p), Pulses: 8} lab := InjectLabels{Synthetic: true, Labelling: l, Incidents: []Incident{inc}} rep := r2Report(r2Cand("e1", "active", scoreKey, injT.Add(30*time.Minute), p+30*time.Second)) - res, err := score(rep, lab, "p474", "labels.json", nil, injT, injT.Add(24*time.Hour)) + res, err := score(rep, lab, "p474", "labels.json", "", nil, nil, injT, injT.Add(24*time.Hour)) if err != nil { t.Fatal(err) } @@ -227,7 +234,7 @@ func TestInjectPayloadSpreadIsValidated(t *testing.T) { "labelling":{"pulse_merge_gap":"60s","min_period":"1m","max_period":"1h","min_pulses":8, "max_harmonic":3,"tolerance_abs":"30s","tolerance_rel":0.1}, "bots":[{"name":"bot-a","channel":"7f","payload_type":5,"route_class":"flood","route_kind":"relay", - "first_hop":"1:fd","full_route":"2:fd0a","wire_bytes":60,` + extra + ` + "first_hop":"1:fd","full_route":"1:fd0a","wire_bytes":60,` + extra + ` "start":"` + injAt(0) + `","stop":"` + injAt(time.Hour) + `","period":"5m","jitter_sd":"3s", "jitter_max":"10s","missed_fraction":0,"multipart_fraction":0,"multipart_max_extra":2, "multipart_max_delay":"55s","holes":[]}]}` @@ -255,3 +262,370 @@ func TestInjectFinalAfterIsWholeSeconds(t *testing.T) { t.Fatal("a fractional final_after must be refused") } } + +// keysOn runs the keys command over one background and returns the parsed +// ranking. +func keysOn(t *testing.T, background string, extra ...string) keysResult { + t.Helper() + dir := t.TempDir() + in := filepath.Join(dir, "bg.jsonl") + if err := os.WriteFile(in, []byte(background), 0o644); err != nil { + t.Fatal(err) + } + args := append([]string{"keys", "-input", in, "-payload-type", "5", "-top", "8", + "-max-keys", "1000", "-max-line-bytes", "65536"}, extra...) + var stdout, stderr bytes.Buffer + if code := run(args, nil, &stdout, &stderr); code != 0 { + t.Fatalf("keys exit %d: %s", code, stderr.String()) + } + var res keysResult + if err := json.Unmarshal(stdout.Bytes(), &res); err != nil { + t.Fatalf("keys output %q: %v", stdout.String(), err) + } + return res +} + +// r2Background writes n events spread over the given (channel, first hop, +// full route) series, with the share of each series fixed by weight. +type r2Series struct { + channel, firstHop, fullRoute string + weight int + payload int +} + +func r2Background(series []r2Series, rounds int) string { + var b strings.Builder + t := injT + for i := 0; i < rounds; i++ { + for _, s := range series { + for w := 0; w < s.weight; w++ { + t = t.Add(17 * time.Second) + pb := s.payload + w + rec := anomaly.ReplayRecord{ID: fmt.Sprintf("tx%d", b.Len()+w+1), Time: t, + FinalAt: t.Add(30 * time.Second), PayloadType: 5, Channel: s.channel, + RouteClass: "flood", RouteKind: "relay", FirstHop: s.firstHop, + FullRoute: s.fullRoute, WireBytes: 55, PayloadBytes: &pb, + Traffic: "unclassified", Decryption: "unknown"} + j, err := json.Marshal(rec) + if err != nil { + panic(err) + } + b.Write(j) + b.WriteByte('\n') + } + } + } + return b.String() +} + +// F2: the panel has to sit on stream keys real traffic uses, and those keys +// have to be picked mechanically. keys ranks the background's own stream +// keys by row count - nothing else - and reports the wire fields an +// injection spec needs to land on each one. +func TestKeysRanksBackgroundStreamKeysByRowCount(t *testing.T) { + res := keysOn(t, r2Background([]r2Series{ + {channel: "0b", firstHop: "1:5c", fullRoute: "1:5c0a1c", weight: 5, payload: 30}, + {channel: "0b", firstHop: "2:7a01", fullRoute: "2:7a01", weight: 2, payload: 40}, + {channel: "81", firstHop: "1:5c", fullRoute: "1:5c", weight: 9, payload: 20}, + }, 6), "-exclude-channels", "81") + if res.DistinctKeys != 2 || len(res.Top) != 2 { + t.Fatalf("ranking %+v", res) + } + if res.SkippedExcluded != 54 { + t.Fatalf("the excluded channel must be skipped, got %d", res.SkippedExcluded) + } + top, second := res.Top[0], res.Top[1] + if top.Rank != 1 || top.Events != 30 || top.Channel != "0b" || top.FirstHop != "1:5c" { + t.Fatalf("top key %+v", top) + } + if second.Rank != 2 || second.Events != 12 || second.FirstHop != "2:7a01" { + t.Fatalf("second key %+v", second) + } + // The ranking is the key encoding the report writes, so a spec built + // from it lands on exactly that key. + if top.Key != "v1|s=stream|pt=5|ch=1:0b|rc=flood|rk=relay|hop=1:5c" { + t.Fatalf("top key encoding %q", top.Key) + } + // The sizes and the route are the realistic values for a bot on the + // key: the observed whole route, not the first hop repeated. + if top.FullRoute != "1:5c0a1c" { + t.Fatalf("full route %q", top.FullRoute) + } + if top.PayloadBytesMedian < 30 || top.PayloadBytesMedian > 34 || top.PayloadBytesSpread == 0 { + t.Fatalf("payload figures %+v", top) + } + if top.WireBytesMedian != 55 { + t.Fatalf("wire median %d", top.WireBytesMedian) + } + if top.First.IsZero() || top.Last.Before(top.First) { + t.Fatalf("times %v..%v", top.First, top.Last) + } +} + +// F2: the filter is part of the selection rule, so it is restated in the +// output and every skipped line is counted. A line of another payload type, +// without a channel hash or not relayed cannot share a stream key with a +// channel bot and never enters the ranking. +func TestKeysFilterIsMechanicalAndCounted(t *testing.T) { + lines := []string{ + `{"id":"tx1","time":"2030-03-01T00:00:01Z","payload_type":5,"channel":"0b","route_class":"flood","route_kind":"relay","first_hop":"1:5c","full_route":"1:5c","wire_bytes":55,"traffic":"unclassified","decryption":"unknown"}`, + `{"id":"tx2","time":"2030-03-01T00:00:02Z","payload_type":4,"channel":"0b","route_class":"flood","route_kind":"relay","first_hop":"1:5c","full_route":"1:5c","wire_bytes":55,"traffic":"unclassified","decryption":"unknown"}`, + `{"id":"tx3","time":"2030-03-01T00:00:03Z","payload_type":5,"channel":"","route_class":"flood","route_kind":"relay","first_hop":"1:5c","full_route":"1:5c","wire_bytes":55,"traffic":"unclassified","decryption":"unknown"}`, + `{"id":"tx4","time":"2030-03-01T00:00:04Z","payload_type":5,"channel":"0b","route_class":"direct","route_kind":"direct","first_hop":"","full_route":"","wire_bytes":55,"traffic":"unclassified","decryption":"unknown"}`, + `{"id":"tx5","time":"2030-03-01T00:00:05Z","payload_type":5,"channel":"81","route_class":"flood","route_kind":"relay","first_hop":"1:5c","full_route":"1:5c","wire_bytes":55,"traffic":"unclassified","decryption":"unknown"}`, + } + res := keysOn(t, strings.Join(lines, "\n")+"\n", "-exclude-channels", "81") + if res.InputLines != 5 || res.Considered != 1 || res.DistinctKeys != 1 { + t.Fatalf("counts %+v", res) + } + if res.SkippedType != 1 || res.SkippedNoChannel != 1 || res.SkippedNotRelay != 1 || res.SkippedExcluded != 1 { + t.Fatalf("skip counters %+v", res) + } + if !strings.Contains(res.Filter, "route_kind == relay") || res.PayloadType != 5 { + t.Fatalf("the output must restate the filter: %+v", res) + } + if len(res.ExcludeChannels) != 1 || res.ExcludeChannels[0] != "81" { + t.Fatalf("exclusions %+v", res.ExcludeChannels) + } +} + +// F2: -max-keys is a hard bound, as the export's -max-rows is, so a run +// never silently reports a truncated ranking. +func TestKeysMaxKeysIsAHardBound(t *testing.T) { + bg := r2Background([]r2Series{ + {channel: "0b", firstHop: "1:5c", fullRoute: "1:5c", weight: 1, payload: 30}, + {channel: "0c", firstHop: "1:5d", fullRoute: "1:5d", weight: 1, payload: 30}, + }, 2) + dir := t.TempDir() + in := filepath.Join(dir, "bg.jsonl") + if err := os.WriteFile(in, []byte(bg), 0o644); err != nil { + t.Fatal(err) + } + var stdout, stderr bytes.Buffer + code := run([]string{"keys", "-input", in, "-payload-type", "5", "-top", "8", + "-max-keys", "1", "-max-line-bytes", "65536"}, nil, &stdout, &stderr) + if code != exitFail || !strings.Contains(stderr.String(), "-max-keys") { + t.Fatalf("exit %d, stderr %q", code, stderr.String()) + } + if stdout.Len() != 0 { + t.Fatalf("a failed run must write no ranking, got %q", stdout.String()) + } +} + +// otherHopKey is the same channel hash and payload type as scoreKey, but a +// different first hop: a second route group on one busy channel. +const otherHopKey = "v1|s=stream|pt=5|ch=1:7f|rc=flood|rk=relay|hop=1:ee" + +// F2: once the panel shares a channel with real traffic, matching on the +// channel hash alone is wrong. An episode of a different route group on the +// same channel is a false positive, not a detection of the bot. +func TestScoreMatchesOnTheStreamKeyNotTheChannel(t *testing.T) { + const p = 300 * time.Second + l := r2Labelling(8) + inc := Incident{Bot: "bot-a", Channel: "7f", Key: scoreKey, Start: injT, + Stop: injT.Add(time.Hour), Period: dur(p), Pulses: 8} + lab := InjectLabels{Synthetic: true, Labelling: l, Incidents: []Incident{inc}} + // One episode, same channel hash, same period, overlapping - but + // another route group. + rep := r2Report(r2Cand("e1", "active", otherHopKey, injT.Add(30*time.Minute), p)) + res, err := score(rep, lab, "p474", "labels.json", "", nil, nil, injT, injT.Add(24*time.Hour)) + if err != nil { + t.Fatal(err) + } + if res.IncidentsDetected != 0 || res.Recall != 0 || res.TruePositives != 0 { + t.Fatalf("another route group is not the bot: %+v", res) + } + if res.FalsePositives != 1 || res.PeriodMismatches != 0 { + t.Fatalf("it is a false positive: %+v", res) + } + // On the bot's own key the same episode is the detection. + rep = r2Report(r2Cand("e1", "active", scoreKey, injT.Add(30*time.Minute), p)) + res, err = score(rep, lab, "p474", "labels.json", "", nil, nil, injT, injT.Add(24*time.Hour)) + if err != nil { + t.Fatal(err) + } + if res.IncidentsDetected != 1 || res.TruePositives != 1 || res.FalsePositives != 0 { + t.Fatalf("the bot's own key is the detection: %+v", res) + } + // A labels file from before round 2 carries no key; then the channel + // hash is all there is, and the old behaviour stands. + inc.Key = "" + lab.Incidents = []Incident{inc} + rep = r2Report(r2Cand("e1", "active", otherHopKey, injT.Add(30*time.Minute), p)) + res, err = score(rep, lab, "p474", "labels.json", "", nil, nil, injT, injT.Add(24*time.Hour)) + if err != nil { + t.Fatal(err) + } + if res.IncidentsDetected != 1 { + t.Fatalf("without a key, matching falls back to the channel: %+v", res) + } +} + +// F2: a shared key needs a differential reading. An episode that the +// baseline arm (same window, no injection) already has on the same key and +// overlapping in time is the background's: it is neither a detection of the +// bot nor a false alarm of the injected arm. +func TestScoreBaselineDropsEpisodesPresentWithoutInjection(t *testing.T) { + const p = 300 * time.Second + l := r2Labelling(8) + inc := Incident{Bot: "bot-a", Channel: "7f", Key: scoreKey, Start: injT, + Stop: injT.Add(time.Hour), Period: dur(p), Pulses: 8} + lab := InjectLabels{Synthetic: true, Labelling: l, Incidents: []Incident{inc}} + rep := r2Report( + r2Cand("e1", "active", scoreKey, injT.Add(30*time.Minute), p), + r2Cand("e1", "ended", scoreKey, injT.Add(90*time.Minute), 0), + ) + // Without a baseline the episode is the detection. + res, err := score(rep, lab, "p474", "labels.json", "", nil, nil, injT, injT.Add(24*time.Hour)) + if err != nil { + t.Fatal(err) + } + if res.IncidentsDetected != 1 || res.BaselineEpisodes != 0 { + t.Fatalf("no baseline: %+v", res) + } + // The same episode in the baseline arm, on the same key and overlapping: + // the background already had it, so arm B learns nothing from it. + baseRep := r2Report( + r2Cand("a1", "active", scoreKey, injT.Add(20*time.Minute), p), + r2Cand("a1", "ended", scoreKey, injT.Add(100*time.Minute), 0), + ) + base, _, err := episodes(baseRep, "p474", injT, injT.Add(24*time.Hour)) + if err != nil { + t.Fatal(err) + } + res, err = score(rep, lab, "p474", "labels.json", "report-a.json", base, nil, injT, injT.Add(24*time.Hour)) + if err != nil { + t.Fatal(err) + } + if res.IncidentsDetected != 0 || res.Recall != 0 { + t.Fatalf("a pre-existing episode is not a detection: %+v", res) + } + if res.BaselineEpisodes != 1 || res.TruePositives != 0 || res.FalsePositives != 0 { + t.Fatalf("it is counted on its own: %+v", res) + } + if res.Baseline != "report-a.json" { + t.Fatalf("baseline provenance %q", res.Baseline) + } + // A baseline episode on another key, or one that does not overlap, + // leaves the detection alone. + for _, other := range []scoreReport{ + r2Report(r2Cand("a1", "active", otherHopKey, injT.Add(20*time.Minute), p), + r2Cand("a1", "ended", otherHopKey, injT.Add(100*time.Minute), 0)), + r2Report(r2Cand("a1", "active", scoreKey, injT.Add(3*time.Hour), p), + r2Cand("a1", "ended", scoreKey, injT.Add(4*time.Hour), 0)), + } { + b, _, err := episodes(other, "p474", injT, injT.Add(24*time.Hour)) + if err != nil { + t.Fatal(err) + } + res, err = score(rep, lab, "p474", "labels.json", "report-a.json", b, nil, injT, injT.Add(24*time.Hour)) + if err != nil { + t.Fatal(err) + } + if res.IncidentsDetected != 1 || res.BaselineEpisodes != 0 { + t.Fatalf("unrelated baseline episode changed the result: %+v", res) + } + } +} + +// F2: -baseline works end to end from two report files. +func TestScoreBaselineFlagReadsAReportFile(t *testing.T) { + bg := backgroundJSONL(300) + r := injectInto(t, specJSON(specOpts{seed: 21, jitterSD: "3s"}), bg) + bgPath := filepath.Join(r.dir, "bg.jsonl") + if err := os.WriteFile(bgPath, []byte(bg), 0o644); err != nil { + t.Fatal(err) + } + reportB := replayInto(t, r.dir, "b", replayConfigJSON(""), r.outPath) + reportA := replayInto(t, r.dir, "a", replayConfigJSON(""), bgPath) + res := scoreInto(t, reportB, r.labPath, "-baseline", reportA) + // This background raises no episode of its own, so the baseline drops + // nothing and the injected bot is still found. + if res.Baseline != "a-report.json" || res.BaselineEpisodes != 0 || res.IncidentsDetected != 1 { + t.Fatalf("baseline arm: %+v", res) + } + var stdout, stderr bytes.Buffer + code := run([]string{"score", "-report", reportB, "-labels", r.labPath, "-rule", "p474", + "-baseline", filepath.Join(r.dir, "missing.json"), + "-start", injAt(0), "-end", injAt(24 * time.Hour)}, nil, &stdout, &stderr) + if code != exitFail || stdout.Len() != 0 { + t.Fatalf("a missing baseline must fail: exit %d, stdout %q", code, stdout.String()) + } +} + +// F3: five clean bots alarm at the earliest pulse the rule allows and the +// realistic ones take many times longer, so the median sits on the floor and +// hides them. The p75 and the maximum are reported beside every median. +func TestScoreReportsTheTailBesideTheMedian(t *testing.T) { + const p = 300 * time.Second + l := r2Labelling(8) + var incs []Incident + var cands []scoreCandidate + // Four incidents detected at 1 P and four at 10, 20, 30 and 40 P. + ttds := []time.Duration{p, p, p, p, 10 * p, 20 * p, 30 * p, 40 * p} + for i, ttd := range ttds { + start := injT.Add(time.Duration(i) * 12 * time.Hour) + key := fmt.Sprintf("v1|s=stream|pt=5|ch=1:7f|rc=flood|rk=relay|hop=1:%02x", 0xa0+i) + incs = append(incs, Incident{Bot: fmt.Sprintf("b%d", i), Channel: "7f", Key: key, + Start: start, Stop: start.Add(60 * p), Period: dur(p), Pulses: 8}) + cands = append(cands, + r2Cand(fmt.Sprintf("e%d", i), "active", key, start.Add(ttd), p), + r2Cand(fmt.Sprintf("e%d", i), "ended", key, start.Add(ttd+p), 0)) + } + lab := InjectLabels{Synthetic: true, Labelling: l, Incidents: incs} + rep := scoreReport{Experimental: true, Start: injT, End: injT.Add(120 * time.Hour), Candidates: cands} + res, err := score(rep, lab, "p474", "labels.json", "", nil, nil, injT, injT.Add(120*time.Hour)) + if err != nil { + t.Fatal(err) + } + if res.IncidentsDetected != 8 { + t.Fatalf("detected %d of 8: %+v", res.IncidentsDetected, res) + } + // The median of 1, 1, 1, 1, 10, 20, 30, 40 is 5.5 P; the nearest-rank + // p75 of eight values is the sixth, 20 P, and the maximum is 40 P. + if res.MedianTTDPeriods > 6 { + t.Fatalf("median %.1f P should sit low", res.MedianTTDPeriods) + } + if res.P75TTDPeriods != 20 || res.MaxTTDPeriods != 40 { + t.Fatalf("p75 %.1f P, max %.1f P", res.P75TTDPeriods, res.MaxTTDPeriods) + } + if res.P75TimeToDetect == nil || time.Duration(*res.P75TimeToDetect) != 20*p { + t.Fatalf("p75 time to detect %v", res.P75TimeToDetect) + } + if res.MaxTimeToDetect == nil || time.Duration(*res.MaxTimeToDetect) != 40*p { + t.Fatalf("max time to detect %v", res.MaxTimeToDetect) + } + if res.MaxStopLatency == nil || res.P75StopLatency == nil || res.MaxEpisodesPerIncident != 1 { + t.Fatalf("stop latency and episode tail: %+v", res) + } + // A bar on the noisy subset alone is the other half of the fix: -bots + // narrows the panel, and the medians then describe that subset only. + noisy, err := score(rep, lab, "p474", "labels.json", "", nil, []string{"b4", "b5", "b6", "b7"}, + injT, injT.Add(120*time.Hour)) + if err != nil { + t.Fatal(err) + } + if noisy.IncidentsSelected != 4 || noisy.MedianTTDPeriods != 25 || noisy.MaxTTDPeriods != 40 { + t.Fatalf("noisy subset: selected %d, median %.1f P, max %.1f P", + noisy.IncidentsSelected, noisy.MedianTTDPeriods, noisy.MaxTTDPeriods) + } +} + +// F8: the metrics record the file names of the labels and the baseline, not +// the paths they were read from. A path can carry a user or a host name, and +// a metrics file may be shared. +func TestScoreProvenanceIsFileNamesOnly(t *testing.T) { + r := injectInto(t, specJSON(specOpts{seed: 21, jitterSD: "3s"}), backgroundJSONL(300)) + res := scoreInto(t, replayInto(t, r.dir, "b", replayConfigJSON(""), r.outPath), r.labPath) + if res.Labels != "labels.json" { + t.Fatalf("labels provenance %q must be a bare file name", res.Labels) + } + if strings.ContainsRune(res.Labels, filepath.Separator) || strings.Contains(res.Labels, r.dir) { + t.Fatalf("labels provenance %q leaks a path", res.Labels) + } + // No labels, no provenance. + none := scoreInto(t, replayInto(t, r.dir, "n", replayConfigJSON(""), r.outPath), "") + if none.Labels != "" || none.Baseline != "" { + t.Fatalf("provenance without files: %q / %q", none.Labels, none.Baseline) + } +} diff --git a/cmd/anomaly/score.go b/cmd/anomaly/score.go index 5fad6aea5..b7cc1180c 100644 --- a/cmd/anomaly/score.go +++ b/cmd/anomaly/score.go @@ -7,7 +7,9 @@ import ( "flag" "fmt" "io" + "math" "os" + "path/filepath" "sort" "strings" "time" @@ -25,11 +27,22 @@ import ( // - An episode is one EventID. Its activation is its first transition to // active; its end is its last transition to ended, or the report's end if // it never ended. Only episodes activated inside [-start, -end) count. -// - An episode is a true positive for an incident when the channel hashes -// are equal, [activation, end] overlaps [incident start, stop + 4P], and +// - An episode is a true positive for an incident when they are the same +// series, [activation, end] overlaps [incident start, stop + 4P], and // the episode's period is within the labelling tolerance of k*P for some // k in 1..max_harmonic. An episode whose activation carries no periodic // evidence can never be a true positive. +// - Same series means the same stream key when the incident carries one +// (inject writes it), and the same channel hash otherwise. The key is +// what makes a panel placed on real traffic's own keys scorable: on a +// busy channel, several route groups share the channel hash, and +// matching on the hash alone would credit any of them to the bot. +// - -baseline takes the report of the same window without the injection +// (arm A). An episode that already exists there, on the same key and +// overlapping in time, is attributed to the background: it is neither a +// true positive nor a false positive, and is counted on its own. That is +// the differential reading a shared key needs - a detection in arm B +// counts only when it is absent from arm A. // - Every other episode is a false positive. With no labels file, every // episode is a false positive, which is the false-alarm arm. // - -bots narrows which incidents the recall and the medians cover, for @@ -64,14 +77,18 @@ type scoreReport struct { // episode is one EventID's activation and end. type episode struct { channel string + key string activation time.Time end time.Time hasPeriod bool period time.Duration matches int - // nearMiss: the episode overlapped an incident on the same channel but + // nearMiss: the episode overlapped an incident on the same series but // failed the period check. nearMiss bool + // baseline: an episode on the same key, overlapping in time, is already + // in the -baseline report, so this one is not the injection's doing. + baseline bool } // incidentScore reports one labelled incident's outcome. @@ -95,16 +112,23 @@ type scoreResult struct { Start string `json:"start"` End string `json:"end"` Days float64 `json:"days"` - Labels string `json:"labels"` + // Labels and Baseline are provenance: the file names only, never the + // paths they were read from (a path can carry a user or host name). + Labels string `json:"labels"` + Baseline string `json:"baseline"` // Bots is the -bots filter, empty when every labelled bot counts. Bots []string `json:"bots"` Episodes int `json:"episodes"` // EpisodesWithoutPeriod counts episodes whose activation candidate // carried no periodic evidence; they can never be true positives. - EpisodesWithoutPeriod int `json:"episodes_without_period"` - TruePositives int `json:"true_positives"` - FalsePositives int `json:"false_positives"` - FalseAlarmsPerDay float64 `json:"false_alarms_per_day"` + EpisodesWithoutPeriod int `json:"episodes_without_period"` + // BaselineEpisodes counts episodes the -baseline report already has on + // the same key, overlapping in time. They are neither true nor false + // positives here: they are the background's, not the injection's. + BaselineEpisodes int `json:"baseline_episodes"` + TruePositives int `json:"true_positives"` + FalsePositives int `json:"false_positives"` + FalseAlarmsPerDay float64 `json:"false_alarms_per_day"` // PeriodMismatches counts episodes that overlap an incident on the same // channel but fail the period check (#457 "also reported"). PeriodMismatches int `json:"period_mismatches"` @@ -114,11 +138,22 @@ type scoreResult struct { IncidentsSelected int `json:"incidents_selected"` IncidentsDetected int `json:"incidents_detected"` Recall float64 `json:"recall"` - // Medians over the detected incidents. + // Medians over the detected incidents. A median hides the slow tail: + // clean trains alarm at the earliest pulse the rule allows, so half the + // panel can sit at that floor while the realistic shapes take many times + // longer. The p75 and the maximum are reported beside every median so + // the tail is visible without reading the per-incident table. MedianTimeToDetect *dur `json:"median_time_to_detect,omitempty"` MedianTTDPeriods float64 `json:"median_time_to_detect_periods,omitempty"` + P75TimeToDetect *dur `json:"p75_time_to_detect,omitempty"` + P75TTDPeriods float64 `json:"p75_time_to_detect_periods,omitempty"` + MaxTimeToDetect *dur `json:"max_time_to_detect,omitempty"` + MaxTTDPeriods float64 `json:"max_time_to_detect_periods,omitempty"` MedianStopLatency *dur `json:"median_stop_latency,omitempty"` + P75StopLatency *dur `json:"p75_stop_latency,omitempty"` + MaxStopLatency *dur `json:"max_stop_latency,omitempty"` MedianEpisodesPerIncident float64 `json:"median_episodes_per_detected_incident,omitempty"` + MaxEpisodesPerIncident float64 `json:"max_episodes_per_detected_incident,omitempty"` PerIncident []incidentScore `json:"per_incident"` // FalsePositivesByChannel lists the false positives per channel hash, // sorted by hash: channel hashes and counts only. @@ -135,6 +170,7 @@ func runScore(args []string, stdout, stderr io.Writer) int { fs.SetOutput(stderr) report := fs.String("report", "", "replay report JSON (required)") labels := fs.String("labels", "", "injected labels JSON; omitted means no ground truth (false alarms only)") + baseline := fs.String("baseline", "", "replay report of the same window without the injection; episodes it already has on the same key are the background's") rule := fs.String("rule", "", "periodic rule name to score (required)") bots := fs.String("bots", "", "comma-separated bot names whose incidents the bars cover; empty means all") start := fs.String("start", "", "RFC 3339 start of the evaluation window, inclusive (required)") @@ -186,13 +222,27 @@ func runScore(args []string, stdout, stderr io.Writer) int { return fail(err) } } + var base []episode + if *baseline != "" { + bb, err := os.ReadFile(*baseline) + if err != nil { + return fail(err) + } + brep, err := parseScoreReport(bb) + if err != nil { + return fail(fmt.Errorf("baseline %w", err)) + } + if base, _, err = episodes(brep, *rule, opt.start, opt.end); err != nil { + return fail(fmt.Errorf("baseline %w", err)) + } + } want := []string{} for _, b := range strings.Split(*bots, ",") { if b = strings.TrimSpace(b); b != "" { want = append(want, b) } } - res, err := score(rep, lab, *rule, *labels, want, opt.start, opt.end) + res, err := score(rep, lab, *rule, *labels, *baseline, base, want, opt.start, opt.end) if err == nil && len(want) > 0 && res.IncidentsSelected == 0 { return fail(errors.New("-bots selected no labelled incident")) } @@ -258,18 +308,29 @@ func parseInjectLabels(b []byte) (InjectLabels, error) { return l, nil } -// score computes the metrics. labelsPath is reported as provenance only; -// bots, when not empty, selects which bots' incidents the bars cover. -func score(rep scoreReport, lab InjectLabels, rule, labelsPath string, bots []string, start, end time.Time) (scoreResult, error) { +// score computes the metrics. labelsPath and baselinePath are reported as +// provenance, by file name only; baseline, when given, are the episodes of +// the same window without the injection; bots, when not empty, selects which +// bots' incidents the bars cover. +func score(rep scoreReport, lab InjectLabels, rule, labelsPath, baselinePath string, baseline []episode, + bots []string, start, end time.Time) (scoreResult, error) { eps, noPeriod, err := episodes(rep, rule, start, end) if err != nil { return scoreResult{}, err } + markBaseline(eps, baseline) days := end.Sub(start).Hours() / 24 res := scoreResult{Experimental: true, Rule: rule, Start: start.Format(time.RFC3339), - End: end.Format(time.RFC3339), Days: days, Labels: labelsPath, Episodes: len(eps), + End: end.Format(time.RFC3339), Days: days, Labels: filepath.Base(labelsPath), + Baseline: filepath.Base(baselinePath), Episodes: len(eps), EpisodesWithoutPeriod: noPeriod, Incidents: len(lab.Incidents), Bots: nonNil(bots), PerIncident: []incidentScore{}, FalsePositivesByChannel: []channelCount{}} + if labelsPath == "" { + res.Labels = "" + } + if baselinePath == "" { + res.Baseline = "" + } selected := func(bot string) bool { if len(bots) == 0 { return true @@ -291,9 +352,12 @@ func score(rep scoreReport, lab InjectLabels, rule, labelsPath string, bots []st var firstAct, lastEnd time.Time for i := range eps { e := &eps[i] - if e.channel != inc.Channel || !overlaps(*e, inc, p) { + if !sameSeries(*e, inc) || !overlaps(*e, inc, p) { continue } + if e.baseline { + continue // the background's episode, counted once below + } if !e.hasPeriod { continue // counted once, as EpisodesWithoutPeriod } @@ -334,6 +398,10 @@ func score(rep scoreReport, lab InjectLabels, rule, labelsPath string, bots []st } byChannel := map[string]int{} for _, e := range eps { + if e.baseline { + res.BaselineEpisodes++ + continue + } if e.matches == 0 && e.nearMiss { res.PeriodMismatches++ } @@ -351,12 +419,16 @@ func score(rep scoreReport, lab InjectLabels, rule, labelsPath string, bots []st res.Recall = float64(res.IncidentsDetected) / float64(res.IncidentsSelected) } if len(ttds) > 0 { - m := dur(medianInt64(ttds)) - res.MedianTimeToDetect = &m - res.MedianTTDPeriods = medianFloat(ttdPeriods(res.PerIncident)) - s := dur(medianInt64(stops)) - res.MedianStopLatency = &s + tp := ttdPeriods(res.PerIncident) + m, p75, mx := dur(medianInt64(ttds)), dur(percentileInt64(ttds, 0.75)), dur(percentileInt64(ttds, 1)) + res.MedianTimeToDetect, res.P75TimeToDetect, res.MaxTimeToDetect = &m, &p75, &mx + res.MedianTTDPeriods = medianFloat(tp) + res.P75TTDPeriods = percentileFloat(tp, 0.75) + res.MaxTTDPeriods = percentileFloat(tp, 1) + sm, s75, smx := dur(medianInt64(stops)), dur(percentileInt64(stops, 0.75)), dur(percentileInt64(stops, 1)) + res.MedianStopLatency, res.P75StopLatency, res.MaxStopLatency = &sm, &s75, &smx res.MedianEpisodesPerIncident = medianFloat(perInc) + res.MaxEpisodesPerIncident = percentileFloat(perInc, 1) } for ch, n := range byChannel { res.FalsePositivesByChannel = append(res.FalsePositivesByChannel, channelCount{Channel: ch, Count: n}) @@ -384,6 +456,7 @@ func ttdPeriods(per []incidentScore) []float64 { func episodes(rep scoreReport, rule string, start, end time.Time) ([]episode, int, error) { type agg struct { channel string + key string act, endT time.Time actSet, endSet bool period time.Duration @@ -401,7 +474,9 @@ func episodes(rep scoreReport, rule string, start, end time.Time) ([]episode, in } a := m[c.EventID] if a == nil { - a = &agg{channel: hexChannel(k)} + // ParseKey accepts only the canonical encoding, so k.String() + // is c.Key: the key is comparable as a string. + a = &agg{channel: hexChannel(k), key: k.String()} m[c.EventID] = a order = append(order, c.EventID) } @@ -429,7 +504,7 @@ func episodes(rep scoreReport, rule string, start, end time.Time) ([]episode, in if !a.actSet || a.act.Before(start) || !a.act.Before(end) { continue } - e := episode{channel: a.channel, activation: a.act, end: a.endT, + e := episode{channel: a.channel, key: a.key, activation: a.act, end: a.endT, hasPeriod: a.hasPeriod, period: a.period} if !a.endSet { e.end = rep.End // never ended: the report's end, as #457 §1 says @@ -452,6 +527,37 @@ func hexChannel(k anomaly.Key) string { return fmt.Sprintf("%x", b) } +// sameSeries reports whether an episode and an incident describe the same +// series: the same stream key when the incident carries one, and the same +// channel hash for a labels file written before inject recorded keys. +func sameSeries(e episode, inc Incident) bool { + if inc.Key != "" { + return e.key == inc.Key + } + return e.channel == inc.Channel +} + +// markBaseline flags every episode that the baseline report already has on +// the same key, overlapping in time. Such an episode exists without the +// injection, so it says nothing about the injected bot either way. +func markBaseline(eps, baseline []episode) { + if len(baseline) == 0 { + return + } + byKey := map[string][]episode{} + for _, b := range baseline { + byKey[b.key] = append(byKey[b.key], b) + } + for i := range eps { + for _, b := range byKey[eps[i].key] { + if !b.activation.After(eps[i].end) && !b.end.Before(eps[i].activation) { + eps[i].baseline = true + break + } + } + } +} + // overlaps reports whether an episode's [activation, end] overlaps the // incident's [start, stop + 4P]. func overlaps(e episode, inc Incident, p time.Duration) bool { @@ -497,3 +603,35 @@ func medianFloat(v []float64) float64 { } return (s[n/2-1] + s[n/2]) / 2 } + +// percentileInt64 and percentileFloat are the nearest-rank percentile: the +// value at index ceil(q*n)-1 of the sorted values, so q = 1 is the maximum +// and q = 0.75 of eight values is the sixth. They sort a copy. +func percentileInt64(v []int64, q float64) int64 { + if len(v) == 0 { + return 0 + } + s := append([]int64(nil), v...) + sort.Slice(s, func(i, j int) bool { return s[i] < s[j] }) + return s[rankIndex(len(s), q)] +} + +func percentileFloat(v []float64, q float64) float64 { + if len(v) == 0 { + return 0 + } + s := append([]float64(nil), v...) + sort.Float64s(s) + return s[rankIndex(len(s), q)] +} + +func rankIndex(n int, q float64) int { + i := int(math.Ceil(q*float64(n))) - 1 + if i < 0 { + return 0 + } + if i > n-1 { + return n - 1 + } + return i +} From 7f2f1a8d8884f48729ff0999d4612f860514904c Mon Sep 17 00:00:00 2001 From: dborup Date: Thu, 15 Oct 2026 22:27:32 +0200 Subject: [PATCH 08/12] docs(anomaly): document keys, the differential arm and measured jitter Review F2, F3 and F4. - A "keys" section: what the subcommand is for (a bot on a key no real transmission carries has that key to itself), why it is blind (it reads the export and counts rows, never the detector or a report), the mechanical filter and every field it reports. - The score section states that matching is on the stream key, what -baseline does and why a shared key needs a differential reading, and that the median hides the tail - so every median comes with its p75 and its maximum, and -bots can give the realistic subset its own numbers. - F4: the spec section now says to take the jitter from measurement, not from the rule's tolerance, with #130's measured values (|gap - 300 s| median 3 s, p95 9 s over 85.2 h), the derivation of a 3 s per-pulse sd from them, and the consequence that some pulses of a short-period bot then fall outside a tight tolerance. Scaling the jitter to the rule makes recall conditional on the rule being evaluated and moves the panel whenever the rule moves. - Provenance is file names only (F8). Relates to #474 --- cmd/anomaly/README.md | 93 +++++++++++++++++++++++++++++++++++++++---- 1 file changed, 85 insertions(+), 8 deletions(-) diff --git a/cmd/anomaly/README.md b/cmd/anomaly/README.md index db32d2a3b..e89dafb8c 100644 --- a/cmd/anomaly/README.md +++ b/cmd/anomaly/README.md @@ -187,6 +187,48 @@ Two properties worth stating plainly: There are no default exclusions anywhere: an unconfigured rule behaves exactly as it did before #474, and the replay golden file is unchanged. +## keys + +Ranks the stream keys of an exported background by row count, so an +injection panel can be placed on keys real traffic actually uses. + +```sh +corescope-anomaly keys -input bg.jsonl -payload-type 5 -top 16 \ + -exclude-channels 81 -max-keys 2000000 -max-line-bytes 65536 -out keys.json +``` + +**Why it exists.** The stream key is payload type + channel hash + route +class + route kind + first hop. A bot on a channel byte and a first hop no +real transmission carries has that key to itself, so the detector sees a +clean train and recall measured on it says little about finding a bot inside +real traffic: a real spam bot on a busy channel shares its key with everyone +else entering through the same repeater. + +**Blind by construction.** `keys` reads the export and counts rows. It never +runs the detector, reads no report, no labels and no metrics, and has no idea +which keys would alarm. The ranking is row counts, ties broken by the +canonical key encoding, so a panel built from its output is chosen +mechanically and before any detection result exists. + +**The filter is mechanical** and is restated in the output, with a counter +per reason a line was skipped: the payload type must be `-payload-type`, the +line must carry a channel hash (the periodic rule's target is a channel bot), +its route kind must be `relay` (only then is the first hop part of the stream +key, so only then can a bot share the key), and its channel must not be in +`-exclude-channels`. + +Per key the output gives the canonical `key`, the `channel`, `route_class`, +`route_kind` and `first_hop` an injection spec needs to land on it, the most +common whole route (`full_route`, ties broken by the smallest encoding), the +median wire size, the median payload size and a `payload_bytes_spread` of +(p90 − p10)/2, the row count and the first and last times. Percentiles are +nearest-rank. Per-key size and route variants are bounded and the overflow is +reported per key; `-max-keys` is a hard bound on distinct keys, as the +export's `-max-rows` is on rows. + +The output carries hashes, hop prefixes, sizes, counts and times — what the +export itself carries — and no ids, observers, names or text. + ## inject Overlays synthetic channel-bot trains on exported background lines, so recall @@ -272,6 +314,18 @@ single missed pulses, one 35-minute hole, one stable first hop). - `jitter_sd` is the standard deviation of the offset from the nominal grid time, clamped to `±jitter_max`. The offset is not cumulative, so the train does not drift. +- **Take the jitter from measurement, not from the rule.** Scaling + `jitter_sd` to the rule's own per-gap tolerance places every bot inside + that tolerance by construction, which makes recall conditional on the rule + being evaluated and moves the panel whenever the rule moves. #130 measured + a real 5-minute bot over 85.2 h: `|gap − 300 s|` median 3 s, p95 9 s. A gap + offset is the difference of two per-pulse offsets, so its standard + deviation is √2 times the per-pulse one, and p95 9 s on a normal gap offset + gives sd(gap) ≈ 9/1.96 ≈ 4.6 s and a per-pulse sd of ≈ 3.2 s. So + `jitter_sd` 3 s with `jitter_max` 9 s (#130's measured p95) is the + measured shape, for every period, independent of any rule value. Some + pulses of a short-period bot then do fall outside a tight tolerance, which + is the point: it is the rule's job to cope, not the panel's to avoid. - `missed_fraction` drops pulses other than the first; `multipart_fraction` gives a pulse 1..`multipart_max_extra` later parts within `multipart_max_delay`. A hole drops every pulse inside it. @@ -304,6 +358,7 @@ all. ```sh corescope-anomaly score -report report.json -labels labels.json \ + -baseline report-a.json \ -rule p474_periodic -start 2026-10-16T00:00:00Z -end 2026-10-23T00:00:00Z \ [-bots b0-90s-clean,b1-120s-clean] [-out metrics.json] ``` @@ -313,11 +368,22 @@ The matching is the #457 §5 periodic rule: - An **episode** is one `EventID`. Its activation is its first transition to `active`; its end is its last transition to `ended`, or the report's end if it never ended. Only episodes activated inside `[-start, -end)` are scored. -- An episode is a **true positive** when the channel hashes are equal, - `[activation, end]` overlaps `[incident start, stop + 4P]`, and the - episode's period is within the labelling tolerance of k·P for some +- An episode is a **true positive** when it is the **same series** as the + incident, `[activation, end]` overlaps `[incident start, stop + 4P]`, and + the episode's period is within the labelling tolerance of k·P for some k in 1..`max_harmonic`. An episode whose activation carries no periodic evidence can never be a true positive. +- **Same series** means the same stream key when the labels carry one (they + do since #474 round 2), and the same channel hash otherwise. The key is + what makes a panel placed on real traffic's own keys scorable: on a busy + channel several route groups share the 1-byte hash, and matching on the + hash alone would credit any of them to the bot. +- `-baseline` takes the replay report of the **same window without the + injection** (arm A). An episode it already has, on the same key and + overlapping in time, is attributed to the background: it is neither a true + positive nor a false positive, and is counted as `baseline_episodes`. That + is the differential reading a shared key needs — a detection in the + injected arm counts only when it is absent from the plain arm. - Every other episode is a **false positive**. Without `-labels` every episode is one, which is the false-alarm arm: run it on background-only traffic to get false alarms per day. @@ -327,8 +393,19 @@ The matching is the #457 §5 periodic rule: positive, selected or not. A filter that selects no incident is an error. The metrics are episodes, true and false positives, false alarms per day, -recall, median time to detect (also in units of P), median stop latency, -median episodes per detected incident, the per-incident detail and the false -positives per channel hash. Times to detect and stop latencies are properties -of the rule and the bot's shape, not of the scorer: a noisy train is found -much later than a clean one. +recall, time to detect (also in units of P), stop latency, episodes per +detected incident, the per-incident detail and the false positives per +channel hash. Times to detect and stop latencies are properties of the rule +and the bot's shape, not of the scorer: a noisy train is found much later +than a clean one. + +**The median hides the tail.** A clean train alarms at the earliest pulse +`min_pulses` allows, so a panel of mostly clean bots puts the median on that +floor while the realistic shapes take many times longer. Every median is +therefore reported beside its nearest-rank **p75** and its **maximum**, and +`-bots` can narrow a run to the realistic subset so that subset gets its own +numbers. Read the tail, not only the median. + +Provenance (`labels`, `baseline`) is recorded as the file names only, never +the paths they were read from: a path can carry a user or a host name, and a +metrics file may be shared. From 44a2b1372ba7d600b9485cfda83efc6d573009cf Mon Sep 17 00:00:00 2001 From: dborup Date: Fri, 16 Oct 2026 00:19:31 +0200 Subject: [PATCH 09/12] test(anomaly): pin the sizes, the tail and the boundaries round 3 found Round-3 review of PR #478 (#474), tests first. G1: the pre-registered pipeline could not be injected on a realistic key. make_spec.py held wire_bytes at the key's median while drawing the payload from median +/- (p90 - p10)/2, and Event.Validate refuses PayloadBytes > WireBytes - 2, so the first wide draw aborted the run. TestInjectSpreadEnvelopeMustFitTheFrame pins the whole draw envelope against the frame and against MaxPayloadBytes; the two pipeline tests run export -> keys -> make_spec.py -> inject on a block-quantised background and on the package's own synthetic CoreScope fixture. G2: injected sizes were still distinguishable. The keys ranking now has to carry the key's observed (wire, payload) pairs, and inject to draw whole pairs from them. G3: a timing bar over the detected incidents cannot fail on a missed bot. TestScoreSelectedTailCountsAMissAsInfinite shows the panel that passes every round-2 timing bar with four of eight realistic shapes never found, and requires the selected tail that fails it. G4: one test per surviving mutant - the keys tie-break, -top, the nearest-rank percentile boundary, the spread clamp at equality, first/last, truncation towards the past, and both touching ends of the baseline overlap. G5: baseline masking has to be visible per incident. The two round-2 payload-spread cases move with the envelope rule: a spread one below payload_bytes is no longer enough on its own. --- cmd/anomaly/review_r2_test.go | 9 +- cmd/anomaly/review_r3_test.go | 728 ++++++++++++++++++++++++++++++++++ 2 files changed, 735 insertions(+), 2 deletions(-) create mode 100644 cmd/anomaly/review_r3_test.go diff --git a/cmd/anomaly/review_r2_test.go b/cmd/anomaly/review_r2_test.go index 148d860a9..485e7c478 100644 --- a/cmd/anomaly/review_r2_test.go +++ b/cmd/anomaly/review_r2_test.go @@ -245,13 +245,18 @@ func TestInjectPayloadSpreadIsValidated(t *testing.T) { {"as wide as the payload", `"payload_bytes":35,"payload_bytes_spread":35,`}, {"without a payload size", `"payload_bytes_spread":3,`}, {"zero payload size", `"payload_bytes":0,"payload_bytes_spread":0,`}, + // Round 3: the top of the draw range has to fit the frame too, not + // only payload_bytes itself (see TestInjectSpreadEnvelopeMustFitTheFrame). + {"wider than the frame allows", `"payload_bytes":35,"payload_bytes_spread":34,`}, } { if _, err := parseInjectSpec([]byte(bot(tc.extra))); err == nil { t.Fatalf("%s: want an error", tc.name) } } - if _, err := parseInjectSpec([]byte(bot(`"payload_bytes":35,"payload_bytes_spread":34,`))); err != nil { - t.Fatalf("a spread one below the payload size is legal: %v", err) + // A spread just inside both bounds - below payload_bytes, and with its + // top at exactly wire_bytes - 2 - is legal. + if _, err := parseInjectSpec([]byte(bot(`"payload_bytes":35,"payload_bytes_spread":23,`))); err != nil { + t.Fatalf("a spread whose envelope fits the frame is legal: %v", err) } } diff --git a/cmd/anomaly/review_r3_test.go b/cmd/anomaly/review_r3_test.go new file mode 100644 index 000000000..6ed67f882 --- /dev/null +++ b/cmd/anomaly/review_r3_test.go @@ -0,0 +1,728 @@ +package main + +import ( + "bytes" + "encoding/json" + "fmt" + "os" + "os/exec" + "path/filepath" + "strings" + "testing" + "time" + + "github.com/meshcore-analyzer/anomaly" +) + +// Round-3 review of PR #478 (#474). The round-2 panel could not be injected +// at all on a realistic key: make_spec.py held wire_bytes at the key's +// median while letting the payload size spread, and the event validator +// requires payload <= wire - 2, so the first oversized draw aborted the run +// (G1). The fix draws each event's (wire, payload) pair from the key's own +// observed distribution (G2), which is valid by construction and leaves no +// size an injected line can be recognised by. The rest of this file pins the +// boundaries round 2's mutants survived (G4), the timing statistic that now +// penalises a missed bot (G3) and the per-incident baseline flag (G5). +// +// Every channel byte, hop, size and time here is synthetic. + +// r3Rec is one background line with explicit sizes. +func r3Rec(id string, at time.Time, channel, hop, route string, wire int, payload *int) anomaly.ReplayRecord { + return anomaly.ReplayRecord{ID: id, Time: at, FinalAt: at.Add(30 * time.Second), PayloadType: 5, + Channel: channel, RouteClass: "flood", RouteKind: "relay", FirstHop: hop, FullRoute: route, + WireBytes: wire, PayloadBytes: payload, Traffic: "unclassified", Decryption: "unknown"} +} + +func r3Line(t *testing.T, rec anomaly.ReplayRecord) string { + t.Helper() + b, err := json.Marshal(rec) + if err != nil { + t.Fatal(err) + } + return string(b) + "\n" +} + +// r3Sized writes one line per payload size on one series, with the wire size +// a fixed number of bytes above the payload (header plus path, as a real +// frame has). Times step by a minute, so the lines are in order. +func r3Sized(t *testing.T, channel, hop, route string, payloads []int, headroom int) string { + t.Helper() + var b strings.Builder + for i, pb := range payloads { + pb := pb + b.WriteString(r3Line(t, r3Rec(fmt.Sprintf("tx%d", i+1), injT.Add(time.Duration(i)*time.Minute), + channel, hop, route, pb+headroom, &pb))) + } + return b.String() +} + +// G2/G1: the ranking carries the key's whole observed (wire, payload) +// distribution, with a count per distinct pair. That distribution is what an +// injected bot draws its sizes from, so it has to be in the output, exact, +// and in a fixed order. +func TestKeysReportsTheObservedSizeDistribution(t *testing.T) { + var b strings.Builder + // 35 bytes four times, 51 twice, 67 once, each with a 3-byte frame + // overhead. Two lines share the 38-byte frame with a shorter payload and + // with a payload the export did not know, which is what fixes the order + // inside one wire size. + add := func(i int, wire int, payload *int) { + b.WriteString(r3Line(t, r3Rec(fmt.Sprintf("tx%d", i+1), injT.Add(time.Duration(i)*time.Minute), + "0b", "1:5c", "1:5c0a", wire, payload))) + } + for i, pb := range []int{35, 35, 35, 35, 51, 51, 67} { + pb := pb + add(i, pb+3, &pb) + } + add(7, 40, nil) + add(8, 38, nil) + add(9, 38, intp(30)) + res := keysOn(t, b.String()) + if len(res.Top) != 1 { + t.Fatalf("one key expected, got %d", len(res.Top)) + } + got := res.Top[0].SizeMix + // Ordered by wire size, then by payload size, with the unknown payload + // first inside one wire size. + want := []sizePair{{WireBytes: 38, Count: 1}, {WireBytes: 38, PayloadBytes: intp(30), Count: 1}, + {WireBytes: 38, PayloadBytes: intp(35), Count: 4}, {WireBytes: 40, Count: 1}, + {WireBytes: 54, PayloadBytes: intp(51), Count: 2}, {WireBytes: 70, PayloadBytes: intp(67), Count: 1}} + if len(got) != len(want) { + t.Fatalf("size mix %+v", got) + } + for i := range want { + if got[i].WireBytes != want[i].WireBytes || got[i].Count != want[i].Count || + (got[i].PayloadBytes == nil) != (want[i].PayloadBytes == nil) || + (want[i].PayloadBytes != nil && *got[i].PayloadBytes != *want[i].PayloadBytes) { + t.Fatalf("size mix entry %d: %+v, want %+v", i, got[i], want[i]) + } + } + if res.Top[0].Events != 10 { + t.Fatalf("events %d", res.Top[0].Events) + } +} + +// r3SizeMix renders a size mix as the JSON an injection spec carries. +func r3SizeMix(pairs ...sizePair) string { + b, err := json.Marshal(pairs) + if err != nil { + panic(err) + } + return string(b) +} + +// r3MixSpec is one bot described by a size mix instead of by scalar sizes. +func r3MixSpec(seed uint64, mix string, stopAfter time.Duration) string { + return fmt.Sprintf(`{ + "seed": %d, + "final_after": "30s", + "labelling": {"pulse_merge_gap": "60s", "min_period": "1m", "max_period": "1h", + "min_pulses": 8, "max_harmonic": 3, "tolerance_abs": "30s", "tolerance_rel": 0.1}, + "bots": [{ + "name": "bot-a", "channel": "7f", "payload_type": 5, "route_class": "flood", "route_kind": "relay", + "first_hop": "1:fd", "full_route": "1:fd0a", "size_mix": %s, + "start": %q, "stop": %q, "period": "5m", "jitter_sd": "3s", "jitter_max": "9s", + "missed_fraction": 0, "multipart_fraction": 0, "multipart_max_extra": 2, + "multipart_max_delay": "55s", "holes": [] + }] +}`, seed, mix, injAt(0), injAt(stopAfter)) +} + +// G1/G2: a bot that draws from the key's observed pairs emits only sizes the +// key really carried, in roughly the observed proportions, and every line is +// a valid event - including on a block-quantised distribution whose spread +// is far wider than the frame's own margin, which is what aborted round 2. +func TestInjectSizeMixDrawsOnlyObservedPairs(t *testing.T) { + mix := r3SizeMix(sizePair{WireBytes: 38, PayloadBytes: intp(35), Count: 70}, + sizePair{WireBytes: 54, PayloadBytes: intp(51), Count: 20}, + sizePair{WireBytes: 70, PayloadBytes: intp(67), Count: 10}, + sizePair{WireBytes: 40, Count: 10}) + r := injectInto(t, r3MixSpec(5, mix, 24*time.Hour), backgroundJSONL(40)) + seen := map[string]int{} + lines := injectedLines(t, r) + if len(lines) < 200 { + t.Fatalf("only %d injected lines", len(lines)) + } + for _, rec := range lines { + pb := "none" + if rec.PayloadBytes != nil { + pb = fmt.Sprint(*rec.PayloadBytes) + } + key := fmt.Sprintf("%d/%s", rec.WireBytes, pb) + switch key { + case "38/35", "54/51", "70/67", "40/none": + default: + t.Fatalf("%s carries the unobserved size pair %s", rec.ID, key) + } + seen[key]++ + } + if len(seen) != 4 { + t.Fatalf("only %d of the four observed pairs were drawn: %v", len(seen), seen) + } + // The commonest pair is 70 % of the weight and must dominate; the three + // others are 10-20 % each and must stay well below it. + if seen["38/35"] <= 2*seen["54/51"] || seen["54/51"] <= seen["70/67"] { + t.Fatalf("draw does not follow the observed weights: %v", seen) + } +} + +// G1: a size mix is the whole size description. Mixing it with the scalar +// size fields would leave two sources of truth, and an entry the frame +// cannot hold must be refused by the spec rather than abort the run halfway +// through, which is what round 2 did. +func TestInjectSizeMixIsValidatedBeforeAnythingRuns(t *testing.T) { + for _, tc := range []struct{ name, mix, extra string }{ + {"a pair the frame cannot hold", + r3SizeMix(sizePair{WireBytes: 38, PayloadBytes: intp(37), Count: 1}), ""}, + {"a zero count", r3SizeMix(sizePair{WireBytes: 38, PayloadBytes: intp(35), Count: 0}), ""}, + {"a negative count", r3SizeMix(sizePair{WireBytes: 38, PayloadBytes: intp(35), Count: -1}), ""}, + {"no wire size", r3SizeMix(sizePair{PayloadBytes: intp(35), Count: 1}), ""}, + {"an empty mix", "[]", ""}, + {"beside wire_bytes", r3SizeMix(sizePair{WireBytes: 38, PayloadBytes: intp(35), Count: 1}), + `"wire_bytes": 60,`}, + {"beside payload_bytes", r3SizeMix(sizePair{WireBytes: 38, PayloadBytes: intp(35), Count: 1}), + `"payload_bytes": 35, "payload_bytes_spread": 0,`}, + } { + spec := strings.Replace(r3MixSpec(5, tc.mix, time.Hour), `"size_mix":`, tc.extra+` "size_mix":`, 1) + if _, err := parseInjectSpec([]byte(spec)); err == nil { + t.Fatalf("%s: want an error", tc.name) + } + } + ok := r3MixSpec(5, r3SizeMix(sizePair{WireBytes: 38, PayloadBytes: intp(36), Count: 1}), time.Hour) + if _, err := parseInjectSpec([]byte(ok)); err != nil { + t.Fatalf("a payload exactly wire-2 long is legal: %v", err) + } +} + +// G1: the scalar path keeps its envelope inside the frame too. payload_bytes +// +/- payload_bytes_spread is the whole range a draw can reach, so the top of +// that range - not just payload_bytes - has to fit wire_bytes - 2. Round 2 +// checked the spread against the payload only, so a legal-looking spec +// aborted on its first wide draw. +func TestInjectSpreadEnvelopeMustFitTheFrame(t *testing.T) { + spec := func(payload, spread, wire int) string { + return fmt.Sprintf(`{"seed":7,"final_after":"30s", + "labelling":{"pulse_merge_gap":"60s","min_period":"1m","max_period":"1h","min_pulses":8, + "max_harmonic":3,"tolerance_abs":"30s","tolerance_rel":0.1}, + "bots":[{"name":"bot-a","channel":"7f","payload_type":5,"route_class":"flood","route_kind":"relay", + "first_hop":"1:fd","full_route":"1:fd0a","wire_bytes":%d,"payload_bytes":%d,"payload_bytes_spread":%d, + "start":%q,"stop":%q,"period":"5m","jitter_sd":"0s", + "jitter_max":"10s","missed_fraction":0,"multipart_fraction":0,"multipart_max_extra":2, + "multipart_max_delay":"55s","holes":[]}]}`, wire, payload, spread, injAt(0), injAt(4*time.Hour)) + } + // The real shape of the abort: a key whose median wire is 47 and whose + // payload spread is 24, as the fixture's rank-1 key is. The diagnostic + // names the two fields, because round 2's failure mode was an opaque + // validator error thousands of lines into a run. + _, err := parseInjectSpec([]byte(spec(35, 24, 47))) + if err == nil { + t.Fatal("a spread that can overflow the frame must be refused by the spec") + } + if !strings.Contains(err.Error(), "payload_bytes + payload_bytes_spread") { + t.Fatalf("the error must name the envelope: %v", err) + } + // One byte inside the margin is legal, one byte outside is not. + if _, err := parseInjectSpec([]byte(spec(35, 23, 60))); err != nil { + t.Fatalf("payload_bytes + spread == wire_bytes - 2 is legal: %v", err) + } + if _, err := parseInjectSpec([]byte(spec(35, 24, 60))); err == nil { + t.Fatal("payload_bytes + spread one past wire_bytes - 2 must be refused") + } + // The envelope must clear every bound the event validator has, not only + // the frame: a payload of 180 +/- 10 fits a 255-byte frame but its top + // passes MaxPayloadBytes (184). + if _, err := parseInjectSpec([]byte(spec(180, 10, 255))); err == nil { + t.Fatal("a spread whose top passes MaxPayloadBytes must be refused") + } + if _, err := parseInjectSpec([]byte(spec(180, 4, 255))); err != nil { + t.Fatalf("a spread whose top is exactly MaxPayloadBytes is legal: %v", err) + } + // And no run of a legal spec may ever produce an invalid line. + r := injectInto(t, spec(35, 23, 60), backgroundJSONL(20)) + for _, rec := range injectedLines(t, r) { + if _, err := rec.Event(); err != nil { + t.Fatalf("%s: %v", rec.ID, err) + } + } +} + +// r3BlockBackground is a background shaped like real GRP_TXT traffic: the +// payload comes in 16-byte cipher blocks and the frame adds only a header +// and a one-hop path, so wire - payload is 3 on every line and the size +// spread is many times the frame's own margin. nKeys series, the i-th +// carrying one line more than the next, so the ranking is unambiguous. +func r3BlockBackground(t *testing.T, nKeys, perKey int) string { + t.Helper() + var b strings.Builder + at := injT + n := 0 + for i := 0; i < nKeys; i++ { + hop := fmt.Sprintf("1:%02x", 0x20+i) + for j := 0; j < perKey+(nKeys-i); j++ { + pb := []int{35, 51, 67, 35, 83, 35, 51}[j%7] + at = at.Add(41 * time.Second) + n++ + b.WriteString(r3Line(t, r3Rec(fmt.Sprintf("tx%d", n), at, "0b", hop, hop+"0a", pb+3, &pb))) + } + } + return b.String() +} + +// r3MakeSpec runs the pre-registered make_spec.py over a keys ranking and +// returns the spec it printed. The script is the pre-registration's own, so +// the test exercises what the round will actually run. +func r3MakeSpec(t *testing.T, keysJSON []byte) []byte { + t.Helper() + py, err := exec.LookPath("python3") + if err != nil { + t.Skip("python3 is not installed; make_spec.py cannot be exercised here") + } + dir := t.TempDir() + kp := filepath.Join(dir, "keys.json") + if err := os.WriteFile(kp, keysJSON, 0o644); err != nil { + t.Fatal(err) + } + cmd := exec.Command(py, "make_spec.py", injT.UTC().Format("2006-01-02T15:04:05Z"), kp) + var out, errb bytes.Buffer + cmd.Stdout, cmd.Stderr = &out, &errb + if err := cmd.Run(); err != nil { + t.Fatalf("make_spec.py: %v: %s", err, errb.String()) + } + return out.Bytes() +} + +// G1: the pre-registered pipeline has to run on realistic sizes. keys -> +// make_spec.py -> inject, on a background whose payloads come in 16-byte +// blocks, is exactly the step that aborted in round 2. +func TestPipelineKeysSpecInjectOnBlockQuantisedSizes(t *testing.T) { + bg := r3BlockBackground(t, 10, 40) + dir := t.TempDir() + bgPath := filepath.Join(dir, "bg.jsonl") + if err := os.WriteFile(bgPath, []byte(bg), 0o644); err != nil { + t.Fatal(err) + } + var stdout, stderr bytes.Buffer + if code := run([]string{"keys", "-input", bgPath, "-payload-type", "5", "-top", "16", + "-max-keys", "5000", "-max-line-bytes", "4096"}, nil, &stdout, &stderr); code != 0 { + t.Fatalf("keys exit %d: %s", code, stderr.String()) + } + spec := r3MakeSpec(t, stdout.Bytes()) + specPath, outPath, labPath := filepath.Join(dir, "spec.json"), filepath.Join(dir, "inj.jsonl"), filepath.Join(dir, "labels.json") + if err := os.WriteFile(specPath, spec, 0o644); err != nil { + t.Fatal(err) + } + var istdout, istderr bytes.Buffer + code := run([]string{"inject", "-spec", specPath, "-input", bgPath, "-out", outPath, + "-labels", labPath, "-max-injected", "2000000", "-max-line-bytes", "65536"}, nil, &istdout, &istderr) + if code != 0 { + t.Fatalf("inject exit %d: %s", code, istderr.String()) + } + // Every injected line must be a valid event, and no injected size pair + // may be one the background never carried. + bgPairs := map[string]bool{} + for _, l := range strings.Split(strings.TrimRight(bg, "\n"), "\n") { + var rec anomaly.ReplayRecord + if err := json.Unmarshal([]byte(l), &rec); err != nil { + t.Fatal(err) + } + bgPairs[r3Pair(rec)] = true + } + out, err := os.ReadFile(outPath) + if err != nil { + t.Fatal(err) + } + injected := 0 + for _, l := range strings.Split(strings.TrimRight(string(out), "\n"), "\n") { + var rec anomaly.ReplayRecord + if err := json.Unmarshal([]byte(l), &rec); err != nil { + t.Fatal(err) + } + if !strings.HasPrefix(rec.ID, injPrefix) { + continue + } + injected++ + if _, err := rec.Event(); err != nil { + t.Fatalf("%s: %v", rec.ID, err) + } + if !bgPairs[r3Pair(rec)] { + t.Fatalf("%s carries size pair %s, which the background never had", rec.ID, r3Pair(rec)) + } + } + if injected < 1000 { + t.Fatalf("only %d injected lines", injected) + } +} + +func r3Pair(rec anomaly.ReplayRecord) string { + if rec.PayloadBytes == nil { + return fmt.Sprintf("%d/none", rec.WireBytes) + } + return fmt.Sprintf("%d/%d", rec.WireBytes, *rec.PayloadBytes) +} + +// G1: the whole pre-registered chain, from a database. export -> keys -> +// make_spec.py -> inject on the package's own synthetic CoreScope fixture, +// whose GRP_TXT payloads come in real 16-byte cipher blocks over a one-hop +// path - the size structure that made round 2 abort. +func TestPipelineExportKeysSpecInjectOnTheFixture(t *testing.T) { + f := newFixtureDB(t, false) + // Ten route groups, each a different first hop on one channel byte, with + // 12 to 21 transmissions apiece so the ranking is strict. Payload sizes + // are 3 + 16k bytes, as a GRP_TXT payload is. + n := 0 + for i := 0; i < 10; i++ { + hop := byte(0x20 + i) + for j := 0; j < 12+(10-i); j++ { + n++ + blocks := []int{32, 48, 64, 32, 80, 32, 48}[j%7] + pl := grpPayload(0x0a, blocks) + at := fxAt(time.Duration(n) * 37 * time.Second) + f.tx(fxHash(n), frameHex(1, 5, 1, []byte{hop}, pl), 5, + relay(at, 1, 5, 1, []byte{hop}, pl)) + } + } + dir := t.TempDir() + bgPath := filepath.Join(dir, "bg.jsonl") + var stdout, stderr bytes.Buffer + if code := run([]string{"export", "-db", f.path, "-start", fxT.UTC().Format(time.RFC3339), + "-end", fxAt(24 * time.Hour).UTC().Format(time.RFC3339), "-settle", "30s", + "-max-rows", "100000", "-out", bgPath}, nil, &stdout, &stderr); code != 0 { + t.Fatalf("export exit %d: %s", code, stderr.String()) + } + stdout.Reset() + stderr.Reset() + if code := run([]string{"keys", "-input", bgPath, "-payload-type", "5", "-top", "16", + "-max-keys", "5000000", "-max-line-bytes", "4096"}, nil, &stdout, &stderr); code != 0 { + t.Fatalf("keys exit %d: %s", code, stderr.String()) + } + var ranking keysResult + if err := json.Unmarshal(stdout.Bytes(), &ranking); err != nil { + t.Fatal(err) + } + if len(ranking.Top) < 9 { + t.Fatalf("the fixture must offer at least the panel's nine keys, got %d", len(ranking.Top)) + } + // The size structure that broke round 2 must really be there: the key's + // payload spread has to exceed the frame's own margin. + k := ranking.Top[0] + if k.PayloadBytesSpread <= k.WireBytesMedian-2-k.PayloadBytesMedian { + t.Fatalf("rank 1 is not block-quantised: wire %d, payload %d +/- %d", + k.WireBytesMedian, k.PayloadBytesMedian, k.PayloadBytesSpread) + } + spec := r3MakeSpec(t, stdout.Bytes()) + specPath := filepath.Join(dir, "spec.json") + if err := os.WriteFile(specPath, spec, 0o644); err != nil { + t.Fatal(err) + } + var istdout, istderr bytes.Buffer + code := run([]string{"inject", "-spec", specPath, "-input", bgPath, + "-out", filepath.Join(dir, "inj.jsonl"), "-labels", filepath.Join(dir, "labels.json"), + "-max-injected", "2000000", "-max-line-bytes", "65536"}, nil, &istdout, &istderr) + if code != 0 { + t.Fatalf("inject exit %d: %s", code, istderr.String()) + } + var sum injectSummary + if err := json.Unmarshal(istderr.Bytes(), &sum); err != nil { + t.Fatalf("summary %q: %v", istderr.String(), err) + } + if sum.InjectedEvents < 1000 || len(sum.Bots) != 18 { + t.Fatalf("summary: %d events, %d bots", sum.InjectedEvents, len(sum.Bots)) + } +} + +// G4 (mutant N3): the tie-break is part of the mechanical assignment rule, +// so it has to be pinned. Two keys with the same row count rank by the +// smaller canonical encoding first; reversing that order silently moves +// every panel position. +func TestKeysTieBreakIsTheSmallerCanonicalKey(t *testing.T) { + res := keysOn(t, r2Background([]r2Series{ + {channel: "0b", firstHop: "1:11", fullRoute: "1:11", weight: 3, payload: 30}, + {channel: "0b", firstHop: "1:22", fullRoute: "1:22", weight: 3, payload: 30}, + {channel: "0b", firstHop: "1:33", fullRoute: "1:33", weight: 3, payload: 30}, + }, 4)) + if len(res.Top) != 3 { + t.Fatalf("three tied keys expected, got %d", len(res.Top)) + } + for i := 1; i < len(res.Top); i++ { + if res.Top[i-1].Events != res.Top[i].Events { + t.Fatalf("the counts must tie: %+v", res.Top) + } + if !(res.Top[i-1].Key < res.Top[i].Key) { + t.Fatalf("tie %d: %q must sort before %q", i, res.Top[i-1].Key, res.Top[i].Key) + } + } + if res.Top[0].FirstHop != "1:11" || res.Top[2].FirstHop != "1:33" { + t.Fatalf("tie order: %q .. %q", res.Top[0].FirstHop, res.Top[2].FirstHop) + } +} + +// G4 (mutant N11): -top reports exactly that many keys. The full ranking +// (-top large) is what the pre-registration's absence check for the isolated +// keys reads, so an off-by-one there silently drops a key from the check. +func TestKeysTopReportsExactlyTopKeys(t *testing.T) { + bg := r3BlockBackground(t, 6, 10) + for _, tc := range []struct{ top, want int }{{1, 1}, {3, 3}, {6, 6}, {9, 6}} { + res := keysOn(t, bg, "-top", fmt.Sprint(tc.top)) + if len(res.Top) != tc.want { + t.Fatalf("-top %d reported %d keys, want %d", tc.top, len(res.Top), tc.want) + } + if res.DistinctKeys != 6 { + t.Fatalf("-top %d: distinct keys %d", tc.top, res.DistinctKeys) + } + for i, k := range res.Top { + if k.Rank != i+1 { + t.Fatalf("-top %d: entry %d has rank %d", tc.top, i, k.Rank) + } + } + } +} + +// G4 (mutant N6): the percentiles are nearest-rank over the value +// histogram - the value at index ceil(q*n)-1 - and the boundary of that +// scan is off by one count if the comparison is not strict. Ten distinct +// sizes, one line each, make every percentile a different value. +func TestKeysPercentilesAreNearestRank(t *testing.T) { + payloads := []int{10, 20, 30, 40, 50, 60, 70, 80, 90, 100} + res := keysOn(t, r3Sized(t, "0b", "1:5c", "1:5c0a", payloads, 5)) + if len(res.Top) != 1 { + t.Fatalf("one key expected, got %d", len(res.Top)) + } + k := res.Top[0] + // n = 10: the median is index 4 (50), p10 index 0 (10) and p90 index 8 + // (90), so the spread is (90 - 10) / 2 = 40. A non-strict scan returns + // the value one count earlier: median 40 and p90 80, spread 35. + if k.PayloadBytesMedian != 50 { + t.Fatalf("payload median %d, want 50", k.PayloadBytesMedian) + } + if k.PayloadBytesSpread != 40 { + t.Fatalf("payload spread %d, want 40", k.PayloadBytesSpread) + } + if k.WireBytesMedian != 55 { + t.Fatalf("wire median %d, want 55", k.WireBytesMedian) + } +} + +// G4 (mutant N4): the spread is clamped strictly below the median, because +// payload_bytes +/- spread must stay positive. A raw spread exactly equal to +// the median is the boundary, and only a clamp at equality catches it. +func TestKeysSpreadIsClampedStrictlyBelowTheMedian(t *testing.T) { + // Ten sizes, one line each: p10 = 1, median = 16, p90 = 33, so the raw + // spread is (33 - 1) / 2 = 16, exactly the median. + res := keysOn(t, r3Sized(t, "0b", "1:5c", "1:5c0a", []int{1, 2, 3, 4, 16, 20, 24, 28, 33, 40}, 5)) + k := res.Top[0] + if k.PayloadBytesMedian != 16 { + t.Fatalf("payload median %d, want 16", k.PayloadBytesMedian) + } + if k.PayloadBytesSpread != 15 { + t.Fatalf("spread %d, want the median minus one (15)", k.PayloadBytesSpread) + } +} + +// G4 (mutant N7): first and last are the key's earliest and latest line. The +// export is ordered, but a merged or concatenated background need not be, and +// a first that tracks the maximum reports a window the key never had. +func TestKeysFirstAndLastAreTheExtremeTimes(t *testing.T) { + var b strings.Builder + pb := 35 + for i, d := range []time.Duration{5 * time.Hour, time.Hour, 9 * time.Hour, 3 * time.Hour} { + b.WriteString(r3Line(t, r3Rec(fmt.Sprintf("tx%d", i+1), injT.Add(d), "0b", "1:5c", "1:5c0a", 40, &pb))) + } + k := keysOn(t, b.String()).Top[0] + if !k.First.Equal(injT.Add(time.Hour)) { + t.Fatalf("first %s, want %s", k.First, injT.Add(time.Hour)) + } + if !k.Last.Equal(injT.Add(9 * time.Hour)) { + t.Fatalf("last %s, want %s", k.Last, injT.Add(9*time.Hour)) + } +} + +// G4 (mutant N8): an injected time is truncated, not rounded. The doc says +// truncation moves a time by less than one second towards the past, and the +// effective jitter window stated in the code depends on it; rounding would +// move half the events into the future instead. +func TestInjectTruncatesTowardsThePast(t *testing.T) { + // A start three quarters of a second past the second, with no jitter: + // truncation gives the whole second below it, rounding the one above. + start := injT.Add(750 * time.Millisecond) + spec := fmt.Sprintf(`{"seed":7,"final_after":"30s", + "labelling":{"pulse_merge_gap":"60s","min_period":"1m","max_period":"1h","min_pulses":8, + "max_harmonic":3,"tolerance_abs":"30s","tolerance_rel":0.1}, + "bots":[{"name":"bot-a","channel":"7f","payload_type":5,"route_class":"flood","route_kind":"relay", + "first_hop":"1:fd","full_route":"1:fd0a","wire_bytes":60,"payload_bytes":35,"payload_bytes_spread":0, + "start":%q,"stop":%q,"period":"5m","jitter_sd":"0s", + "jitter_max":"10s","missed_fraction":0,"multipart_fraction":0,"multipart_max_extra":2, + "multipart_max_delay":"55s","holes":[]}]}`, + start.UTC().Format(time.RFC3339Nano), injT.Add(time.Hour).UTC().Format(time.RFC3339Nano)) + r := injectInto(t, spec, backgroundJSONL(10)) + lines := injectedLines(t, r) + if len(lines) == 0 { + t.Fatal("no injected lines") + } + for _, rec := range lines { + if rec.Time.Nanosecond() != 0 { + t.Fatalf("%s is not a whole second: %s", rec.ID, rec.Time) + } + } + if !lines[0].Time.Equal(injT) { + t.Fatalf("first event at %s, want %s (truncated towards the past)", lines[0].Time, injT) + } + if !lines[0].Time.Before(start) { + t.Fatalf("truncation must move the time towards the past: %s is not before %s", lines[0].Time, start) + } +} + +// r3Episode is one arm's episode, written out in full. +func r3Episode(key string, act, end time.Duration, period time.Duration) episode { + return episode{channel: "7f", key: key, activation: injT.Add(act), end: injT.Add(end), + hasPeriod: period > 0, period: period} +} + +// G4 (mutants N1/N2): the baseline overlap is closed at both ends. An +// episode of the background that starts exactly when the injected arm's +// episode ends, or ends exactly when it starts, still overlaps it: a +// half-open test would call such an episode the injection's doing. +func TestScoreBaselineOverlapIsClosedAtBothEnds(t *testing.T) { + const p = 300 * time.Second + for _, tc := range []struct { + name string + baseAct, baseEnd time.Duration + }{ + {"baseline starts exactly when the episode ends", 2 * time.Hour, 3 * time.Hour}, + {"baseline ends exactly when the episode starts", 0, time.Hour}, + } { + eps := []episode{r3Episode(scoreKey, time.Hour, 2*time.Hour, p)} + markBaseline(eps, []episode{r3Episode(scoreKey, tc.baseAct, tc.baseEnd, p)}) + if !eps[0].baseline { + t.Fatalf("%s: a touching baseline episode must still mark it", tc.name) + } + } + // One nanosecond clear of the episode at either end is not an overlap. + for _, tc := range []struct { + name string + baseAct, baseEnd time.Duration + }{ + {"after", 2*time.Hour + 1, 3 * time.Hour}, + {"before", 0, time.Hour - 1}, + } { + eps := []episode{r3Episode(scoreKey, time.Hour, 2*time.Hour, p)} + markBaseline(eps, []episode{r3Episode(scoreKey, tc.baseAct, tc.baseEnd, p)}) + if eps[0].baseline { + t.Fatalf("%s: a baseline episode clear of the window must not mark it", tc.name) + } + } +} + +// r3Panel builds n incidents on n keys, with a detection at the given time +// to detect per incident; a negative time to detect means the incident is +// missed (no episode at all). +func r3Panel(p time.Duration, ttds []float64) (InjectLabels, scoreReport) { + var incs []Incident + var cands []scoreCandidate + for i, mult := range ttds { + start := injT.Add(time.Duration(i) * 12 * time.Hour) + key := fmt.Sprintf("v1|s=stream|pt=5|ch=1:7f|rc=flood|rk=relay|hop=1:%02x", 0xa0+i) + incs = append(incs, Incident{Bot: fmt.Sprintf("b%d", i), Channel: "7f", Key: key, + Start: start, Stop: start.Add(60 * p), Period: dur(p), Pulses: 8}) + if mult < 0 { + continue + } + at := start.Add(time.Duration(mult * float64(p))) + cands = append(cands, r2Cand(fmt.Sprintf("e%d", i), "active", key, at, p), + r2Cand(fmt.Sprintf("e%d", i), "ended", key, at.Add(p), 0)) + } + return InjectLabels{Synthetic: true, Labelling: r2Labelling(8), Incidents: incs}, + scoreReport{Experimental: true, Start: injT, End: injT.Add(240 * time.Hour), Candidates: cands} +} + +// G3: a timing bar over the detected incidents only cannot fail on a missed +// bot. Four clean shapes detected at the floor and four realistic ones missed +// gives recall 0.50 and a p75 of 1 P over the detected subset - a pass of +// every timing bar round 2 had. The selected tail ranks a miss above every +// finite value, so the same panel's p75 is infinite and the bar fails. +func TestScoreSelectedTailCountsAMissAsInfinite(t *testing.T) { + const p = 300 * time.Second + lab, rep := r3Panel(p, []float64{1, 1, 1, 1, -1, -1, -1, -1}) + res, err := score(rep, lab, "p474", "labels.json", "", nil, nil, injT, injT.Add(240*time.Hour)) + if err != nil { + t.Fatal(err) + } + if res.IncidentsSelected != 8 || res.IncidentsDetected != 4 || res.IncidentsMissed != 4 { + t.Fatalf("selected %d, detected %d, missed %d", res.IncidentsSelected, res.IncidentsDetected, res.IncidentsMissed) + } + if res.P75TTDPeriods != 1 { + t.Fatalf("the detected-only p75 is %.1f P; the statistic round 2 had cannot fail", res.P75TTDPeriods) + } + // n = 8 selected: the nearest-rank p75 is the sixth value, and values + // five to eight are misses, so the p75 is infinite. + if !res.P75TTDSelectedInfinite || res.P75TTDSelected != nil || res.P75TTDPeriodsSelected != 0 { + t.Fatalf("p75 over the selected incidents must be infinite: %+v", res) + } + // The median is the fourth value, which is still a detection. + if res.MedianTTDSelectedInfinite || res.MedianTTDSelected == nil || + time.Duration(*res.MedianTTDSelected) != p || res.MedianTTDPeriodsSelected != 1 { + t.Fatalf("median over the selected incidents: %+v", res) + } + // With every incident detected the selected tail is finite and equals + // the detected-only tail. + fullLab, fullRep := r3Panel(p, []float64{1, 1, 1, 1, 10, 20, 30, 40}) + full, err := score(fullRep, fullLab, "p474", "labels.json", "", nil, nil, injT, injT.Add(240*time.Hour)) + if err != nil { + t.Fatal(err) + } + if full.IncidentsMissed != 0 || full.P75TTDSelectedInfinite || full.MedianTTDSelectedInfinite { + t.Fatalf("a complete panel has a finite tail: %+v", full) + } + if full.P75TTDPeriodsSelected != full.P75TTDPeriods || full.MedianTTDPeriodsSelected != 1 { + t.Fatalf("selected %.1f P vs detected %.1f P", full.P75TTDPeriodsSelected, full.P75TTDPeriods) + } + // A miss outside the -bots selection must not enter the statistic. + sub, err := score(rep, lab, "p474", "labels.json", "", nil, []string{"b0", "b1", "b2", "b3"}, + injT, injT.Add(240*time.Hour)) + if err != nil { + t.Fatal(err) + } + if sub.IncidentsMissed != 0 || sub.P75TTDSelectedInfinite || sub.Recall != 1 { + t.Fatalf("-bots subset: missed %d, infinite %v, recall %.2f", + sub.IncidentsMissed, sub.P75TTDSelectedInfinite, sub.Recall) + } +} + +// G5: when the only episode of an incident's own series is absorbed by the +// baseline arm, the per-incident row reads as a plain miss. A no-go caused by +// background contamination has to be distinguishable from a detector miss, so +// the row carries the count and a flag. +func TestScoreReportsBaselineMaskingPerIncident(t *testing.T) { + const p = 300 * time.Second + l := r2Labelling(8) + key := scoreKey + other := "v1|s=stream|pt=5|ch=1:7f|rc=flood|rk=relay|hop=1:ee" + lab := InjectLabels{Synthetic: true, Labelling: l, Incidents: []Incident{ + {Bot: "masked", Channel: "7f", Key: key, Start: injT, Stop: injT.Add(60 * p), Period: dur(p), Pulses: 8}, + {Bot: "plain-miss", Channel: "7f", Key: other, Start: injT, Stop: injT.Add(60 * p), Period: dur(p), Pulses: 8}, + }} + rep := scoreReport{Experimental: true, Start: injT, End: injT.Add(48 * time.Hour), Candidates: []scoreCandidate{ + r2Cand("e1", "active", key, injT.Add(10*time.Minute), p), + r2Cand("e1", "ended", key, injT.Add(2*time.Hour), 0), + }} + base := []episode{r3Episode(key, 0, 24*time.Hour, p)} + res, err := score(rep, lab, "p474", "labels.json", "report-a.json", base, nil, injT, injT.Add(48*time.Hour)) + if err != nil { + t.Fatal(err) + } + if res.BaselineEpisodes != 1 || res.IncidentsDetected != 0 { + t.Fatalf("baseline episodes %d, detected %d", res.BaselineEpisodes, res.IncidentsDetected) + } + if res.IncidentsBaselineMasked != 1 { + t.Fatalf("incidents_baseline_masked %d, want 1", res.IncidentsBaselineMasked) + } + byBot := map[string]incidentScore{} + for _, is := range res.PerIncident { + byBot[is.Bot] = is + } + if m := byBot["masked"]; !m.BaselineMasked || m.BaselineEpisodes != 1 || m.Detected { + t.Fatalf("the masked incident's row: %+v", m) + } + if pm := byBot["plain-miss"]; pm.BaselineMasked || pm.BaselineEpisodes != 0 || pm.Detected { + t.Fatalf("a plain miss must not be flagged as masked: %+v", pm) + } +} From 9ea156987cfbfe5632869caa32f505cd68982413 Mon Sep 17 00:00:00 2001 From: dborup Date: Fri, 16 Oct 2026 00:19:43 +0200 Subject: [PATCH 10/12] fix(anomaly): inject the key's own observed size pairs G1/G2 of the round-3 review of PR #478 (#474). A frame is header plus path plus payload, so wire - payload is the route's own overhead, while a GRP_TXT payload comes in 16-byte cipher blocks. Round 2 described a bot's sizes as a fixed median wire plus a payload spread, which gets both halves wrong: the drawn payload routinely passed wire - 2 and Event.Validate aborted the run part way through (G1), and where it did fit, the sizes were ones the key never carried, so an injected line was distinguishable on size alone (G2). keys now reports size_mix per key: every observed (wire_bytes, payload_bytes) pair with its count, ordered by wire then payload, with the payload absent for the lines whose length the export did not know. The distinct pairs are bounded like the other per-key histograms and the overflow is reported. inject takes size_mix as an alternative to the scalar sizes and draws one whole pair per event, weighted by count - a binary search over cumulative weights computed once per bot. A single-entry mix consumes no randomness, as a zero spread does, so an existing train's draws do not move. The scalar path keeps working for hand-written specs, but now has to fit: payload_bytes + payload_bytes_spread <= wire_bytes - 2. And every size a bot can draw is checked against the event validator while the spec is read, so an impossible size is a spec error with a message naming the fields, never an abort thousands of lines into a run. --- cmd/anomaly/inject.go | 259 ++++++++++++++++++++++++++++++++++-------- cmd/anomaly/keys.go | 87 ++++++++++++-- 2 files changed, 293 insertions(+), 53 deletions(-) diff --git a/cmd/anomaly/inject.go b/cmd/anomaly/inject.go index 66454e07d..19019cd54 100644 --- a/cmd/anomaly/inject.go +++ b/cmd/anomaly/inject.go @@ -39,6 +39,11 @@ const injPrefix = "inj-" const ( maxInjectBots = 64 maxInjectHoles = 64 + // maxInjectSizeMix bounds one bot's size distribution, and + // maxInjectSizeCount one entry's weight, so the cumulative weights + // cannot overflow. + maxInjectSizeMix = 4096 + maxInjectSizeCount = 1 << 32 ) // injectSpec is the JSON spec of one injection run. There are no defaults: @@ -73,23 +78,38 @@ type labellingSpec struct { // export; the fields that describe the wire are exactly the fields of a // replay record, so an injected line carries nothing a real line does not. type injectBot struct { - Name string `json:"name"` - Channel string `json:"channel"` - PayloadType *int `json:"payload_type"` - RouteClass string `json:"route_class"` - RouteKind string `json:"route_kind"` - FirstHop string `json:"first_hop"` - FullRoute string `json:"full_route"` - WireBytes int `json:"wire_bytes"` - PayloadBytes *int `json:"payload_bytes"` - // PayloadBytesSpread spreads the payload size: each event's size is - // drawn uniformly from payload_bytes +/- this many bytes. A real bot's - // payload length varies, so a constant one makes an injected line - // trivially distinguishable from an exported one. 0 keeps it constant. - PayloadBytesSpread *int `json:"payload_bytes_spread"` - Start time.Time `json:"start"` - Stop time.Time `json:"stop"` - Period dur `json:"period"` + Name string `json:"name"` + Channel string `json:"channel"` + PayloadType *int `json:"payload_type"` + RouteClass string `json:"route_class"` + RouteKind string `json:"route_kind"` + FirstHop string `json:"first_hop"` + FullRoute string `json:"full_route"` + // The sizes are described in exactly one of two ways, and the spec must + // pick one. + // + // SizeMix is the realistic one, and what `keys` produces: the key's own + // observed (wire_bytes, payload_bytes) pairs with a count each. One + // whole pair is drawn per event, weighted by count. Because every pair + // came off a real line, it is valid by construction and carries the + // route's own wire-minus-payload overhead, so an injected line cannot be + // told from a real one on size. + // + // WireBytes with PayloadBytes and PayloadBytesSpread is the older, + // scalar description, kept for hand-written specs: a fixed wire size and + // a payload drawn uniformly from payload_bytes +/- the spread. It cannot + // reproduce a real distribution - the payload comes in cipher blocks + // while wire - payload is header plus path - so the whole envelope is + // required to fit the frame (payload_bytes + spread <= wire_bytes - 2); + // a spec that could overflow is refused here rather than aborting the + // run on its first wide draw. + WireBytes int `json:"wire_bytes"` + PayloadBytes *int `json:"payload_bytes"` + PayloadBytesSpread *int `json:"payload_bytes_spread"` + SizeMix []sizePair `json:"size_mix"` + Start time.Time `json:"start"` + Stop time.Time `json:"stop"` + Period dur `json:"period"` // JitterSD is the standard deviation of the per-pulse offset from the // nominal grid time; JitterMax clamps it. The offset is not cumulative, // so the train does not drift. @@ -346,6 +366,13 @@ func parseInjectSpec(b []byte) (injectSpec, error) { return injectSpec{}, fmt.Errorf("inject spec: duplicate bot name %q", s.Bots[i].Name) } seen[s.Bots[i].Name] = true + // Every size the bot can draw has to make a valid event, and the + // check belongs here: a spec that could overflow the frame is + // refused outright rather than aborting a run after thousands of + // lines, which is what round 2 did on a realistic key. + if _, err := botStreamKey(&s.Bots[i], time.Duration(*s.FinalAfter)); err != nil { + return injectSpec{}, err + } } return s, nil } @@ -407,8 +434,12 @@ func (b *injectBot) validate() error { return bad("multipart_max_extra is required and must be in 0..8") case b.JitterSD == nil || *b.JitterSD < 0: return bad("jitter_sd is required and must be >= 0") + case b.SizeMix != nil: + if err := b.validateSizeMix(bad); err != nil { + return err + } case b.WireBytes <= 0: - return bad("wire_bytes must be > 0") + return bad("size_mix or wire_bytes is required, and wire_bytes must be > 0") case b.PayloadBytes != nil && *b.PayloadBytes <= 0: return bad("payload_bytes must be > 0 when present") case b.PayloadBytesSpread == nil || *b.PayloadBytesSpread < 0: @@ -417,6 +448,12 @@ func (b *injectBot) validate() error { return bad("payload_bytes_spread needs payload_bytes") case b.PayloadBytes != nil && *b.PayloadBytesSpread >= *b.PayloadBytes: return bad("payload_bytes_spread must be < payload_bytes") + case b.PayloadBytes != nil && *b.PayloadBytes+*b.PayloadBytesSpread > b.WireBytes-2: + // The top of the draw range, not just payload_bytes, has to fit the + // frame: Event.Validate refuses PayloadBytes > WireBytes - 2. + return bad("payload_bytes + payload_bytes_spread must fit the frame (<= wire_bytes - 2)") + } + switch { case b.Start.IsZero() || b.Stop.IsZero() || !b.Stop.After(b.Start): return bad("start and stop are required, with stop after start") case time.Duration(b.Period) <= 0: @@ -438,6 +475,101 @@ func (b *injectBot) validate() error { return nil } +// validateSizeMix checks a bot's size distribution. The pairs themselves are +// checked against the event validator in botStreamKey, which sees the bot's +// whole wire shape; here only the shape of the list is checked, plus the +// mutual exclusion with the scalar size fields. +func (b *injectBot) validateSizeMix(bad func(string) error) error { + if b.WireBytes != 0 || b.PayloadBytes != nil || b.PayloadBytesSpread != nil { + return bad("size_mix is the whole size description: leave wire_bytes, payload_bytes and payload_bytes_spread out") + } + if len(b.SizeMix) == 0 || len(b.SizeMix) > maxInjectSizeMix { + return bad(fmt.Sprintf("size_mix needs 1..%d entries", maxInjectSizeMix)) + } + seen := map[[2]int]bool{} + for _, sp := range b.SizeMix { + if sp.Count <= 0 || sp.Count > maxInjectSizeCount { + return bad(fmt.Sprintf("every size_mix entry needs a count in 1..%d", maxInjectSizeCount)) + } + if sp.WireBytes <= 0 { + return bad("every size_mix entry needs wire_bytes > 0") + } + if sp.PayloadBytes != nil && *sp.PayloadBytes < 0 { + return bad("a size_mix payload_bytes must be >= 0") + } + k := [2]int{sp.WireBytes, payloadOrUnknown(sp)} + if seen[k] { + return bad(fmt.Sprintf("size_mix carries the pair (%d, %d) twice", k[0], k[1])) + } + seen[k] = true + } + return nil +} + +// sizeDraw turns one bot's size description into a draw. For a size mix it +// holds the cumulative weights, computed once per bot, so a draw is a binary +// search rather than a walk over the whole distribution. +type sizeDraw struct { + mix []sizePair + cum []int64 + total int64 + // the scalar description + wire int + payload *int + spread int +} + +func newSizeDraw(b *injectBot) sizeDraw { + d := sizeDraw{mix: b.SizeMix, wire: b.WireBytes, payload: b.PayloadBytes} + if b.PayloadBytesSpread != nil { + d.spread = *b.PayloadBytesSpread + } + for _, sp := range b.SizeMix { + d.total += int64(sp.Count) + d.cum = append(d.cum, d.total) + } + return d +} + +// random reports whether a draw consumes a value from the generator. A +// single-entry mix and a zero spread are constant, so they consume none and +// leave every other draw of the train where it was. +func (d sizeDraw) random() bool { + if len(d.mix) > 0 { + return len(d.mix) > 1 + } + return d.payload != nil && d.spread > 0 +} + +// draw returns one event's wire and payload size. It takes exactly one value +// from p when random() is true, and none otherwise. +func (d sizeDraw) draw(p *prng) (int, *int) { + if len(d.mix) > 0 { + i := 0 + if len(d.mix) > 1 { + r := int64(p.float() * float64(d.total)) + i = sort.Search(len(d.cum), func(j int) bool { return d.cum[j] > r }) + if i >= len(d.mix) { + i = len(d.mix) - 1 + } + } + sp := d.mix[i] + if sp.PayloadBytes == nil { + return sp.WireBytes, nil + } + pb := *sp.PayloadBytes + return sp.WireBytes, &pb + } + if d.payload == nil { + return d.wire, nil + } + pb := *d.payload + if d.spread > 0 { + pb += int(p.float()*float64(2*d.spread+1)) - d.spread + } + return d.wire, &pb +} + // generate builds every injected event, the per-bot counts and the labelled // incidents. It fails, without producing anything, if the spec would emit // more than maxInjected events. @@ -483,6 +615,7 @@ func generateBot(b *injectBot, seed uint64, finalAfter time.Duration, budget int return nil, botStats{}, err } st.Key = key + sizes := newSizeDraw(b) var evs []injEvent emit := func(pulse, part int, when time.Time) error { if len(evs) >= budget { @@ -495,23 +628,18 @@ func generateBot(b *injectBot, seed uint64, finalAfter time.Duration, budget int // the past, so the effective jitter is in [-(jitter_max + 1s), // jitter_max]. when = when.UTC().Truncate(time.Second) + // One size draw per emitted event: a whole (wire, payload) pair from + // the key's observed distribution, or the scalar payload spread. + wire, payload := sizes.draw(p) rec := anomaly.ReplayRecord{ ID: fmt.Sprintf("%s%s-%d-%d", injPrefix, b.Name, pulse, part), Time: when, FinalAt: when.Add(finalAfter), PayloadType: *b.PayloadType, Channel: b.Channel, RouteClass: b.RouteClass, RouteKind: b.RouteKind, FirstHop: b.FirstHop, FullRoute: b.FullRoute, - WireBytes: b.WireBytes, Traffic: "unclassified", Decryption: "unknown", - } - if b.PayloadBytes != nil { - pb := *b.PayloadBytes - // One payload-size draw per emitted event, taken only when the - // spread is nonzero, so a spread of 0 consumes no randomness and - // leaves every other draw where it was. - if sp := *b.PayloadBytesSpread; sp > 0 { - pb += int(p.float()*float64(2*sp+1)) - sp - } - rec.PayloadBytes = &pb + WireBytes: wire, PayloadBytes: payload, Traffic: "unclassified", Decryption: "unknown", } - // An injected line must be exactly as valid as an exported one. + // An injected line must be exactly as valid as an exported one. Every + // size the draw can produce was checked by botStreamKey before any + // event was built, so this cannot fire on a validated spec. if _, err := rec.Event(); err != nil { return fmt.Errorf("inject: bot %q: %w", b.Name, err) } @@ -570,24 +698,63 @@ func generateBot(b *injectBot, seed uint64, finalAfter time.Duration, budget int return evs, st, nil } -// botStreamKey returns the canonical stream key of a bot's events. It builds -// the same ReplayRecord emit builds, so a spec whose wire fields do not make -// a valid event fails here rather than silently labelling nothing. +// botStreamKey returns the canonical stream key of a bot's events, and +// checks that every size the bot can draw makes a valid event. It builds the +// same ReplayRecord emit builds, so a spec whose wire fields do not make a +// valid event fails here - before any event, line or file is produced - +// rather than aborting the run halfway through or silently labelling nothing. +// +// Sizes are not part of a stream key, so every size gives the same key; the +// loop is there to validate, and the equality is asserted rather than +// assumed. func botStreamKey(b *injectBot, finalAfter time.Duration) (string, error) { - rec := anomaly.ReplayRecord{ID: injPrefix + b.Name + "-key", Time: b.Start.UTC().Truncate(time.Second), - PayloadType: *b.PayloadType, Channel: b.Channel, RouteClass: b.RouteClass, RouteKind: b.RouteKind, - FirstHop: b.FirstHop, FullRoute: b.FullRoute, WireBytes: b.WireBytes, - Traffic: "unclassified", Decryption: "unknown"} - rec.FinalAt = rec.Time.Add(finalAfter) - if b.PayloadBytes != nil { - pb := *b.PayloadBytes - rec.PayloadBytes = &pb - } - ev, err := rec.Event() - if err != nil { - return "", fmt.Errorf("inject: bot %q: %w", b.Name, err) + build := func(wire int, payload *int) (string, error) { + rec := anomaly.ReplayRecord{ID: injPrefix + b.Name + "-key", Time: b.Start.UTC().Truncate(time.Second), + PayloadType: *b.PayloadType, Channel: b.Channel, RouteClass: b.RouteClass, RouteKind: b.RouteKind, + FirstHop: b.FirstHop, FullRoute: b.FullRoute, WireBytes: wire, PayloadBytes: payload, + Traffic: "unclassified", Decryption: "unknown"} + rec.FinalAt = rec.Time.Add(finalAfter) + ev, err := rec.Event() + if err != nil { + return "", fmt.Errorf("inject: bot %q: %w", b.Name, err) + } + return anomaly.StreamKeyOf(ev).String(), nil + } + var sizes [][2]*int + if len(b.SizeMix) > 0 { + for _, sp := range b.SizeMix { + wire := sp.WireBytes + sizes = append(sizes, [2]*int{&wire, sp.PayloadBytes}) + } + } else { + // The whole envelope of the scalar draw: the smallest and the largest + // payload it can reach. + wire := b.WireBytes + if b.PayloadBytes == nil { + sizes = append(sizes, [2]*int{&wire, nil}) + } else { + lo, hi := *b.PayloadBytes-*b.PayloadBytesSpread, *b.PayloadBytes+*b.PayloadBytesSpread + sizes = append(sizes, [2]*int{&wire, &lo}, [2]*int{&wire, &hi}) + } + } + key := "" + for _, s := range sizes { + var payload *int + if s[1] != nil { + pb := *s[1] + payload = &pb + } + k, err := build(*s[0], payload) + if err != nil { + return "", err + } + if key == "" { + key = k + } else if k != key { + return "", fmt.Errorf("inject: bot %q: its sizes do not share one stream key", b.Name) + } } - return anomaly.StreamKeyOf(ev).String(), nil + return key, nil } func inHole(holes []injectHole, t time.Time) bool { diff --git a/cmd/anomaly/keys.go b/cmd/anomaly/keys.go index 4c426ec77..08d804fe2 100644 --- a/cmd/anomaly/keys.go +++ b/cmd/anomaly/keys.go @@ -44,8 +44,28 @@ import ( const ( maxKeySizeBuckets = 512 maxKeyRouteVariants = 64 + // maxKeySizePairs bounds the distinct (wire, payload) pairs tracked per + // key. It is the distribution an injected bot draws from, so it is kept + // wider than the per-dimension bucket bound; overflow is reported. + maxKeySizePairs = 2048 ) +// sizePair is one observed (wire_bytes, payload_bytes) pair and the number +// of background lines that carried it. PayloadBytes is absent for the lines +// whose payload length the export did not know. +// +// Why the pair and not two separate figures: the two sizes are not +// independent. A frame is header plus path plus payload, so wire - payload +// is the route's own overhead, while the payload itself comes in cipher +// blocks. Drawing the two separately produces frames real traffic never +// carries - and, when the drawn payload passes wire - 2, frames the event +// validator refuses outright. +type sizePair struct { + WireBytes int `json:"wire_bytes"` + PayloadBytes *int `json:"payload_bytes,omitempty"` + Count int `json:"count"` +} + // keyStat is one ranked stream key. Every field is derived from the // background lines on that key: hashes, hop prefixes, sizes, counts and // times, which is what the export itself carries. No ids, observers, names @@ -68,7 +88,13 @@ type keyStat struct { WireBytesMedian int `json:"wire_bytes_median"` PayloadBytesMedian int `json:"payload_bytes_median,omitempty"` PayloadBytesSpread int `json:"payload_bytes_spread"` - Events int `json:"events"` + // SizeMix is the key's whole observed (wire, payload) distribution, one + // entry per distinct pair, ordered by wire size then payload size. This + // is what an injected bot on this key draws its sizes from, so its lines + // cannot be told from the key's real ones by size. + SizeMix []sizePair `json:"size_mix"` + SizeMixDropped int `json:"size_mix_dropped,omitempty"` + Events int `json:"events"` // SizeBucketsDropped and RouteVariantsDropped count values that did not // fit the per-key bounds above; they affect the medians and the route // mode only, never Events. @@ -103,9 +129,13 @@ type keyAgg struct { events int first, last time.Time payload, wire map[int]int - routes map[string]int - sizeDropped int - routeDropped int + // pairs counts the observed (wire, payload) pairs; a payload of -1 is a + // line whose payload length the export did not know. + pairs map[[2]int]int + routes map[string]int + sizeDropped int + pairDropped int + routeDropped int } func runKeys(args []string, stdin io.Reader, stdout, stderr io.Writer) int { @@ -267,7 +297,8 @@ func rankKeys(r io.Reader, payloadType int, exclude []string, top, maxKeys, maxL if len(aggs) >= maxKeys { return res, errors.New("more than -max-keys distinct keys; raise the bound") } - a = &keyAgg{k: k, payload: map[int]int{}, wire: map[int]int{}, routes: map[string]int{}} + a = &keyAgg{k: k, payload: map[int]int{}, wire: map[int]int{}, + pairs: map[[2]int]int{}, routes: map[string]int{}} aggs[k] = a } res.Considered++ @@ -279,9 +310,12 @@ func rankKeys(r io.Reader, payloadType int, exclude []string, top, maxKeys, maxL a.last = rec.Time.UTC() } bump(a.wire, rec.WireBytes, maxKeySizeBuckets, &a.sizeDropped) + pb := -1 if rec.PayloadBytes != nil { - bump(a.payload, *rec.PayloadBytes, maxKeySizeBuckets, &a.sizeDropped) + pb = *rec.PayloadBytes + bump(a.payload, pb, maxKeySizeBuckets, &a.sizeDropped) } + bumpPair(a.pairs, [2]int{rec.WireBytes, pb}, maxKeySizePairs, &a.pairDropped) bumpStr(a.routes, rec.FullRoute, maxKeyRouteVariants, &a.routeDropped) } if err := sc.Err(); err != nil { @@ -317,6 +351,14 @@ func bump(m map[int]int, v, max int, dropped *int) { m[v]++ } +func bumpPair(m map[[2]int]int, v [2]int, max int, dropped *int) { + if _, ok := m[v]; !ok && len(m) >= max { + *dropped++ + return + } + m[v]++ +} + func bumpStr(m map[string]int, v string, max int, dropped *int) { if _, ok := m[v]; !ok && len(m) >= max { *dropped++ @@ -329,7 +371,8 @@ func (a *keyAgg) stat(rank int) keyStat { s := keyStat{Rank: rank, Key: a.k.String(), PayloadType: int(a.k.PayloadType), Channel: hexChannel(a.k), RouteClass: a.k.RouteClass.String(), RouteKind: a.k.RouteKind.String(), FirstHop: encodeHop(a.k.FirstHop), Events: a.events, First: a.first, Last: a.last, - SizeBucketsDropped: a.sizeDropped, RouteVariantsDropped: a.routeDropped} + SizeBucketsDropped: a.sizeDropped, RouteVariantsDropped: a.routeDropped, + SizeMix: sizeMixOf(a.pairs), SizeMixDropped: a.pairDropped} s.FullRoute = modeString(a.routes) s.WireBytesMedian = int(histPercentile(a.wire, 0.5)) if len(a.payload) > 0 { @@ -349,6 +392,36 @@ func (a *keyAgg) stat(rank int) keyStat { return s } +// sizeMixOf renders a pair histogram in a fixed order: wire size, then +// payload size, with the unknown payload (-1) first for a given wire size. +// The order is a pure function of the histogram, so it does not depend on +// map iteration. +func sizeMixOf(m map[[2]int]int) []sizePair { + out := make([]sizePair, 0, len(m)) + for v, n := range m { + p := sizePair{WireBytes: v[0], Count: n} + if v[1] >= 0 { + pb := v[1] + p.PayloadBytes = &pb + } + out = append(out, p) + } + sort.Slice(out, func(i, j int) bool { + if out[i].WireBytes != out[j].WireBytes { + return out[i].WireBytes < out[j].WireBytes + } + return payloadOrUnknown(out[i]) < payloadOrUnknown(out[j]) + }) + return out +} + +func payloadOrUnknown(p sizePair) int { + if p.PayloadBytes == nil { + return -1 + } + return *p.PayloadBytes +} + // encodeHop renders a first hop in the "width:hex" form a replay record and // an injection spec use. func encodeHop(h anomaly.Hop) string { From d144f4b0924dafa74f9f8c28aa6e93ea1e5835c0 Mon Sep 17 00:00:00 2001 From: dborup Date: Fri, 16 Oct 2026 00:19:56 +0200 Subject: [PATCH 11/12] feat(anomaly): score a missed bot as an infinite time to detect G3 and G5 of the round-3 review of PR #478 (#474). G3. Every median and p75 was taken over the detected incidents only, so no timing bar on them could fail on a bot that was never found: a panel whose four cleanest shapes alarm at the floor and whose four realistic ones are missed reports recall 0.50 and a p75 of one period, and passes the lot. The time to detect is now also reported over every selected incident with a miss ranked above every finite value. A percentile whose rank falls among the misses is infinite; JSON has no infinity, so it is reported as ..._selected_infinite with the value left out, which fails any bar placed on it. incidents_missed counts the misses. G5. A -baseline episode is not scored either way, so an incident whose only episode was absorbed by the background read as a plain miss. Each per-incident row now carries baseline_episodes and baseline_masked, and incidents_baseline_masked counts them, so a no-go caused by background contamination can be told from a detector miss. --- cmd/anomaly/score.go | 144 ++++++++++++++++++++++++++++++++++++------- 1 file changed, 123 insertions(+), 21 deletions(-) diff --git a/cmd/anomaly/score.go b/cmd/anomaly/score.go index b7cc1180c..8c1bb89d5 100644 --- a/cmd/anomaly/score.go +++ b/cmd/anomaly/score.go @@ -45,6 +45,9 @@ import ( // counts only when it is absent from arm A. // - Every other episode is a false positive. With no labels file, every // episode is a false positive, which is the false-alarm arm. +// - The time-to-detect tail is reported twice: over the detected incidents, +// and over every selected incident with a miss ranked above every finite +// value. Only the second can fail on a bot that was never found. // - -bots narrows which incidents the recall and the medians cover, for // example to leave a non-blind sanity-check bot out of the primary // numbers. It never turns a matched episode into a false positive: an @@ -93,16 +96,27 @@ type episode struct { // incidentScore reports one labelled incident's outcome. type incidentScore struct { - Bot string `json:"bot"` - Channel string `json:"channel"` - Start string `json:"start"` - Stop string `json:"stop"` - Period dur `json:"period"` - Episodes int `json:"episodes"` - Detected bool `json:"detected"` - TimeToDetect *dur `json:"time_to_detect,omitempty"` - TTDPeriods float64 `json:"time_to_detect_periods,omitempty"` - StopLatency *dur `json:"stop_latency,omitempty"` + Bot string `json:"bot"` + Channel string `json:"channel"` + Start string `json:"start"` + Stop string `json:"stop"` + Period dur `json:"period"` + Episodes int `json:"episodes"` + Detected bool `json:"detected"` + // BaselineEpisodes counts the episodes on this incident's own series, + // overlapping its window, that the -baseline arm already had. They are + // not scored either way (see markBaseline), so without this count an + // incident whose only episode was absorbed by the background reads as a + // plain miss. The period check is not applied to them, because it is + // not applied to a baseline episode anywhere. + BaselineEpisodes int `json:"baseline_episodes,omitempty"` + // BaselineMasked marks exactly that case: not detected, but the + // background already had an episode here. It tells a no-go caused by + // contamination apart from a detector miss. + BaselineMasked bool `json:"baseline_masked,omitempty"` + TimeToDetect *dur `json:"time_to_detect,omitempty"` + TTDPeriods float64 `json:"time_to_detect_periods,omitempty"` + StopLatency *dur `json:"stop_latency,omitempty"` } // scoreResult is the metrics file. @@ -134,21 +148,43 @@ type scoreResult struct { PeriodMismatches int `json:"period_mismatches"` // Incidents is every labelled incident; IncidentsSelected is the subset // the bars cover (all of them unless -bots narrows it). - Incidents int `json:"incidents"` - IncidentsSelected int `json:"incidents_selected"` - IncidentsDetected int `json:"incidents_detected"` - Recall float64 `json:"recall"` + Incidents int `json:"incidents"` + IncidentsSelected int `json:"incidents_selected"` + IncidentsDetected int `json:"incidents_detected"` + // IncidentsMissed is the selected incidents that were not detected, and + // IncidentsBaselineMasked the subset of those whose own series already + // carried a -baseline episode. + IncidentsMissed int `json:"incidents_missed"` + IncidentsBaselineMasked int `json:"incidents_baseline_masked"` + Recall float64 `json:"recall"` // Medians over the detected incidents. A median hides the slow tail: // clean trains alarm at the earliest pulse the rule allows, so half the // panel can sit at that floor while the realistic shapes take many times // longer. The p75 and the maximum are reported beside every median so // the tail is visible without reading the per-incident table. - MedianTimeToDetect *dur `json:"median_time_to_detect,omitempty"` - MedianTTDPeriods float64 `json:"median_time_to_detect_periods,omitempty"` - P75TimeToDetect *dur `json:"p75_time_to_detect,omitempty"` - P75TTDPeriods float64 `json:"p75_time_to_detect_periods,omitempty"` - MaxTimeToDetect *dur `json:"max_time_to_detect,omitempty"` - MaxTTDPeriods float64 `json:"max_time_to_detect_periods,omitempty"` + MedianTimeToDetect *dur `json:"median_time_to_detect,omitempty"` + MedianTTDPeriods float64 `json:"median_time_to_detect_periods,omitempty"` + P75TimeToDetect *dur `json:"p75_time_to_detect,omitempty"` + P75TTDPeriods float64 `json:"p75_time_to_detect_periods,omitempty"` + MaxTimeToDetect *dur `json:"max_time_to_detect,omitempty"` + MaxTTDPeriods float64 `json:"max_time_to_detect_periods,omitempty"` + // The tail over the SELECTED incidents, with a missed incident counting + // as +infinity. + // + // Why both. The medians and p75s above are over the DETECTED incidents + // only, so no timing bar on them can fail on a missed bot: a panel whose + // four cleanest shapes alarm at the floor and whose four realistic ones + // are never found reports recall 0.50 and a p75 of one period, which + // passes every timing bar. Ranking a miss above every finite value fixes + // that - the same panel's p75 is then infinite, and a bar on it fails. + // JSON has no infinity, so an infinite percentile is reported as the + // Infinite flag with the value left out. + MedianTTDSelected *dur `json:"median_time_to_detect_selected,omitempty"` + MedianTTDPeriodsSelected float64 `json:"median_time_to_detect_periods_selected,omitempty"` + MedianTTDSelectedInfinite bool `json:"median_time_to_detect_selected_infinite"` + P75TTDSelected *dur `json:"p75_time_to_detect_selected,omitempty"` + P75TTDPeriodsSelected float64 `json:"p75_time_to_detect_periods_selected,omitempty"` + P75TTDSelectedInfinite bool `json:"p75_time_to_detect_selected_infinite"` MedianStopLatency *dur `json:"median_stop_latency,omitempty"` P75StopLatency *dur `json:"p75_stop_latency,omitempty"` MaxStopLatency *dur `json:"max_stop_latency,omitempty"` @@ -356,7 +392,11 @@ func score(rep scoreReport, lab InjectLabels, rule, labelsPath, baselinePath str continue } if e.baseline { - continue // the background's episode, counted once below + // The background's episode: not scored either way, but + // recorded here so a masked incident is not read as a plain + // miss. + is.BaselineEpisodes++ + continue } if !e.hasPeriod { continue // counted once, as EpisodesWithoutPeriod @@ -384,6 +424,7 @@ func score(rep scoreReport, lab InjectLabels, rule, labelsPath, baselinePath str // Only selected incidents enter the recall and the medians; an // episode that matched an unselected incident is still a match, so // it never becomes a false positive. + is.BaselineMasked = !is.Detected && is.BaselineEpisodes > 0 if !selected(inc.Bot) { continue } @@ -393,6 +434,11 @@ func score(rep scoreReport, lab InjectLabels, rule, labelsPath, baselinePath str ttds = append(ttds, int64(*is.TimeToDetect)) stops = append(stops, int64(*is.StopLatency)) perInc = append(perInc, float64(is.Episodes)) + } else { + res.IncidentsMissed++ + if is.BaselineMasked { + res.IncidentsBaselineMasked++ + } } res.PerIncident = append(res.PerIncident, is) } @@ -418,6 +464,30 @@ func score(rep scoreReport, lab InjectLabels, rule, labelsPath, baselinePath str if res.IncidentsSelected > 0 { res.Recall = float64(res.IncidentsDetected) / float64(res.IncidentsSelected) } + // The selected tail is reported whenever anything is selected, including + // the case where every selected incident was missed and every percentile + // is infinite. + if res.IncidentsSelected > 0 { + tp := ttdPeriods(res.PerIncident) + for _, q := range []struct { + q float64 + d **dur + periods *float64 + infinite *bool + }{ + {0.5, &res.MedianTTDSelected, &res.MedianTTDPeriodsSelected, &res.MedianTTDSelectedInfinite}, + {0.75, &res.P75TTDSelected, &res.P75TTDPeriodsSelected, &res.P75TTDSelectedInfinite}, + } { + v, inf := selectedPercentileInt64(ttds, res.IncidentsMissed, q.q) + if inf { + *q.infinite = true + continue + } + d := dur(v) + *q.d = &d + *q.periods, _ = selectedPercentileFloat(tp, res.IncidentsMissed, q.q) + } + } if len(ttds) > 0 { tp := ttdPeriods(res.PerIncident) m, p75, mx := dur(medianInt64(ttds)), dur(percentileInt64(ttds, 0.75)), dur(percentileInt64(ttds, 1)) @@ -625,6 +695,38 @@ func percentileFloat(v []float64, q float64) float64 { return s[rankIndex(len(s), q)] } +// selectedPercentileInt64 is the nearest-rank percentile over the selected +// incidents, with each missed incident counting as +infinity: the misses +// rank above every finite value, so the percentile is infinite exactly when +// its rank falls among them. It reports (value, infinite). +func selectedPercentileInt64(detected []int64, missed int, q float64) (int64, bool) { + n := len(detected) + missed + if n == 0 { + return 0, false + } + i := rankIndex(n, q) + if i >= len(detected) { + return 0, true + } + s := append([]int64(nil), detected...) + sort.Slice(s, func(a, b int) bool { return s[a] < s[b] }) + return s[i], false +} + +func selectedPercentileFloat(detected []float64, missed int, q float64) (float64, bool) { + n := len(detected) + missed + if n == 0 { + return 0, false + } + i := rankIndex(n, q) + if i >= len(detected) { + return 0, true + } + s := append([]float64(nil), detected...) + sort.Float64s(s) + return s[i], false +} + func rankIndex(n int, q float64) int { i := int(math.Ceil(q*float64(n))) - 1 if i < 0 { From 743d0f33f6039c2b3e943a3a526304b2312872da Mon Sep 17 00:00:00 2001 From: dborup Date: Fri, 16 Oct 2026 00:19:56 +0200 Subject: [PATCH 12/12] docs(anomaly): commit the panel generator and document the round-3 changes make_spec.py moves into the repository. Round 2 kept it only in the pre-registration comment, which is why nothing exercised the one step that was broken: the chain export -> keys -> make_spec.py -> inject on realistic sizes. It is stdlib-only, reads only a window start and a keys ranking, has no clock, and now passes each key's size_mix through verbatim instead of describing the sizes as a median and a spread. The README documents size_mix on both sides, the envelope rule for the scalar sizes, the selected-incident tail with a miss ranked as infinite, and the per-incident baseline flag. --- cmd/anomaly/README.md | 97 +++++++++++++++++++++++++++++++++++----- cmd/anomaly/make_spec.py | 89 ++++++++++++++++++++++++++++++++++++ 2 files changed, 174 insertions(+), 12 deletions(-) create mode 100644 cmd/anomaly/make_spec.py diff --git a/cmd/anomaly/README.md b/cmd/anomaly/README.md index e89dafb8c..bdaade2c8 100644 --- a/cmd/anomaly/README.md +++ b/cmd/anomaly/README.md @@ -220,11 +220,27 @@ key, so only then can a bot share the key), and its channel must not be in Per key the output gives the canonical `key`, the `channel`, `route_class`, `route_kind` and `first_hop` an injection spec needs to land on it, the most common whole route (`full_route`, ties broken by the smallest encoding), the -median wire size, the median payload size and a `payload_bytes_spread` of -(p90 − p10)/2, the row count and the first and last times. Percentiles are -nearest-rank. Per-key size and route variants are bounded and the overflow is -reported per key; `-max-keys` is a hard bound on distinct keys, as the -export's `-max-rows` is on rows. +key's whole observed size distribution (`size_mix`), the median wire size, +the median payload size and a `payload_bytes_spread` of (p90 − p10)/2, the +row count and the first and last times. Percentiles are nearest-rank and the +spread is clamped strictly below the median. `first` and `last` are the +earliest and latest line on the key, whatever order the input is in. +Per-key size and route variants are bounded and the overflow is reported per +key; `-max-keys` is a hard bound on distinct keys, as the export's +`-max-rows` is on rows. `-top` reports exactly that many keys, so the full +ranking (a large `-top`) is what an absence check should read. + +**`size_mix` is the distribution to inject from.** It is one entry per +distinct `(wire_bytes, payload_bytes)` pair with the number of lines that +carried it, ordered by wire size then payload size; `payload_bytes` is absent +for the lines whose payload length the export did not know. The two sizes are +*not* independent: a frame is header plus path plus payload, so +`wire − payload` is the route's own overhead, while the payload itself comes +in 16-byte cipher blocks. Drawing them separately — a fixed median wire with +a spread payload — both produces frames real traffic never carries and, once +the drawn payload passes `wire − 2`, frames the event validator refuses +outright. So `inject` takes whole observed pairs (see `size_mix` under +Spec). The output carries hashes, hop prefixes, sizes, counts and times — what the export itself carries — and no ids, observers, names or text. @@ -282,9 +298,9 @@ single missed pulses, one 35-minute hole, one stable first hop). "route_kind": "relay", "first_hop": "1:fd", "full_route": "1:fd0a1c", - "wire_bytes": 60, - "payload_bytes": 35, - "payload_bytes_spread": 6, + "size_mix": [{"wire_bytes": 38, "payload_bytes": 35, "count": 70}, + {"wire_bytes": 54, "payload_bytes": 51, "count": 20}, + {"wire_bytes": 70, "payload_bytes": 67, "count": 10}], "start": "2026-10-16T00:00:00Z", "stop": "2026-10-19T13:12:00Z", "period": "5m", @@ -307,10 +323,25 @@ single missed pulses, one 35-minute hole, one stable first hop). second never gives an injected line away. Truncation moves a time by less than a second towards the past, so the effective offset from the grid is in `[-(jitter_max + 1s), jitter_max]`. -- `payload_bytes_spread` draws each event's payload size uniformly from - `payload_bytes ± spread`; a real bot's payload length varies, and a - constant one is another giveaway. 0 keeps the size constant, and the - spread must be smaller than `payload_bytes`. +- **The sizes are described in exactly one of two ways**, and a spec must + pick one. + - `size_mix` is the realistic one, and what `keys` emits per key: the + observed `(wire_bytes, payload_bytes)` pairs with a count each. One whole + pair is drawn per event, weighted by count, so the injected sizes are + exactly the sizes the key carries — valid by construction, and nothing an + injected line can be recognised by. With `size_mix`, `wire_bytes`, + `payload_bytes` and `payload_bytes_spread` must be left out. + - `wire_bytes` with `payload_bytes` and `payload_bytes_spread` is the older + scalar description, for hand-written specs: a fixed wire size and a + payload drawn uniformly from `payload_bytes ± spread`. 0 keeps the size + constant. It cannot reproduce a real distribution, so the **whole + envelope** is required to fit the frame: the spread must be smaller than + `payload_bytes`, and `payload_bytes + payload_bytes_spread` must be at + most `wire_bytes − 2`, which is what `Event.Validate` allows. + Either way, every size a bot can draw is checked against the event + validator while the spec is read — before a single event, line or file is + produced — so an impossible size is a spec error and never an abort part + way through a run. - `jitter_sd` is the standard deviation of the offset from the nominal grid time, clamped to `±jitter_max`. The offset is not cumulative, so the train does not drift. @@ -334,6 +365,25 @@ single missed pulses, one 35-minute hole, one stable first hop). does not. `full_route` is a whole observed route, not just the first hop repeated, so an injected line's route looks like an exported one's. +### make_spec.py + +`make_spec.py` beside this README is the #474 round-3 panel generator: it +takes the validation window's start and a `keys.json` ranking and prints the +spec for the pre-registered paired panel (nine shapes, each on the key of +rank *i*+1 and on an isolated twin key). It is stdlib-only, reads nothing +else and has no clock, so the spec is a pure function of its two inputs and +can be re-derived from them afterwards. + +It lives in the repository rather than only in the pre-registration comment +so that the whole chain — `export` → `keys` → `make_spec.py` → `inject` — is +exercised by the tool's own tests on a background whose payload sizes are +block-quantised like real GRP_TXT traffic. That chain is where round 2 broke: +the spec it produced could not be injected at all on a realistic key. + +```sh +python3 make_spec.py 2026-10-17T00:00:00Z keys.json > spec.json +``` + ### Labels `-labels` holds the ground truth of the run, computed from the emitted pulses @@ -406,6 +456,29 @@ therefore reported beside its nearest-rank **p75** and its **maximum**, and `-bots` can narrow a run to the realistic subset so that subset gets its own numbers. Read the tail, not only the median. +**A tail over the detected incidents cannot fail on a missed bot.** The +medians, p75s and maxima above cover the *detected* incidents only, so a +panel whose four cleanest shapes alarm at the floor and whose four realistic +ones are never found reports recall 0.50 and a p75 of one period — a pass of +every timing bar, on a result that found nothing realistic. So the time to +detect is also reported over every **selected** incident with a miss ranked +above every finite value: +`median_time_to_detect_selected` / `p75_time_to_detect_selected`, with +`_periods` beside each. JSON has no infinity, so a percentile whose rank +falls among the misses is reported as `..._selected_infinite: true` with the +value left out — which fails any bar placed on it. `incidents_missed` counts +the selected incidents that were not detected. A bar belongs on the selected +tail; the detected-only tail describes the shapes that were found. + +**Baseline masking is visible per incident.** A `-baseline` episode is not +scored either way, so without a marker an incident whose only episode was +absorbed by the background reads as a plain miss. Each per-incident row +therefore carries `baseline_episodes` (the episodes on the incident's own +series, overlapping its window, that the plain arm already had) and +`baseline_masked` (not detected, but the background already had an episode +here); `incidents_baseline_masked` counts them. That tells a no-go caused by +background contamination apart from a detector miss. + Provenance (`labels`, `baseline`) is recorded as the file names only, never the paths they were read from: a path can carry a user or a host name, and a metrics file may be shared. diff --git a/cmd/anomaly/make_spec.py b/cmd/anomaly/make_spec.py new file mode 100644 index 000000000..80a8a6924 --- /dev/null +++ b/cmd/anomaly/make_spec.py @@ -0,0 +1,89 @@ +# make_spec.py - builds the #474 round-3 injection spec deterministically. +# python3 make_spec.py > spec.json (stdlib only) +# +# The panel is paired. Each shape appears twice at the same time, on two +# different stream keys: +# shared - the key of real background traffic, taken from the keys ranking +# by row count (rank 1..9 in order). This is the arm that +# measures detection inside real traffic. +# isolated - the same channel byte with the 3-byte first hop fdff0i, a key +# no real relayed line carries. This is the round-1 control. +# The two twins never interact: they are different keys. +# +# Sizes come from the key's own observed (wire_bytes, payload_bytes) +# distribution, passed through verbatim as size_mix. Round 2 instead held +# wire_bytes at the key's median while drawing the payload from +# median +/- (p90 - p10) / 2, which has two faults: a real GRP_TXT payload +# comes in 16-byte cipher blocks while wire - payload is only header plus +# path, so the drawn payload routinely exceeded wire - 2 and inject aborted +# on the event validator (#478 review G1); and even where it fitted, the +# drawn sizes were ones the key never carried, so an injected line was +# distinguishable on size alone (G2). Drawing whole observed pairs fixes +# both: every pair came from a real line, so it is valid by construction and +# indistinguishable by size. +# +# Jitter comes from #130's measurements of a real 5-minute bot over 85.2 h +# (|gap - 300 s| median 3 s, p95 9 s), not from the rule's tolerance: a gap +# offset is the difference of two per-pulse offsets, so sd(gap) = sqrt(2) * +# sd(pulse), and p95 9 s gives sd(gap) ~ 9/1.96 ~ 4.6 s and sd(pulse) ~ 3.2 s. +# So every bot gets jitter_sd 3 s and jitter_max 9 s, whatever its period and +# whatever the rule's tolerance is. +import json, sys +from datetime import datetime, timedelta, timezone + +V0 = datetime.strptime(sys.argv[1], "%Y-%m-%dT%H:%M:%SZ").replace(tzinfo=timezone.utc) +RANKING = json.load(open(sys.argv[2])) + + +def iso(t): + return t.strftime("%Y-%m-%dT%H:%M:%SZ") + + +# Panel: (period seconds, shape). Shapes: +# clean = jitter only +# noisy = jitter + 5 % missed pulses + 30 % multi-part pulses (#130 shape) +# sept = noisy plus the one 35-minute outage of the September bot +PANEL = [(90, "clean"), (120, "clean"), (300, "clean"), (600, "clean"), (900, "clean"), + (90, "noisy"), (300, "noisy"), (900, "noisy"), (300, "sept")] +HOURS_EACH, HOURS_APART = 12, 18 +JITTER_SD, JITTER_MAX = 3, 9 # #130, measured +SEED = 474202610170 + +keys = RANKING["top"] +if len(keys) < len(PANEL): + sys.exit("keys ranking has %d keys, need %d" % (len(keys), len(PANEL))) + +bots = [] +for i, (p, shape) in enumerate(PANEL): + k = keys[i] # rank i+1, by row count alone + start = V0 + timedelta(hours=1 + HOURS_APART * i) + stop = start + timedelta(hours=HOURS_EACH) + mpd = min(55, p // 4) + noisy = shape in ("noisy", "sept") + holes = [{"start": iso(start + timedelta(hours=6)), "duration": "35m"}] if shape == "sept" else [] + # The key's observed size distribution, verbatim. inject draws one whole + # (wire, payload) pair per event, weighted by count, so the injected + # sizes are exactly the sizes the key carries. + size_mix = k["size_mix"] + if not size_mix: + sys.exit("rank %d has no observed size pair" % (i + 1)) + for arm, hop in (("shared", k["first_hop"]), ("iso", "3:fdff%02x" % i)): + bots.append({ + "name": "b%d-%ds-%s-%s" % (i, p, shape, arm), + "channel": k["channel"], "payload_type": RANKING["payload_type"], + "route_class": k["route_class"], "route_kind": k["route_kind"], + "first_hop": hop, "full_route": k["full_route"] if arm == "shared" else hop, + "size_mix": size_mix, + "start": iso(start), "stop": iso(stop), "period": "%ds" % p, + "jitter_sd": "%ds" % JITTER_SD, "jitter_max": "%ds" % JITTER_MAX, + "missed_fraction": 0.05 if noisy else 0.0, + "multipart_fraction": 0.3 if noisy else 0.0, + "multipart_max_extra": 2, "multipart_max_delay": "%ds" % mpd, + "holes": holes, + }) + +print(json.dumps({"seed": SEED, "final_after": "30s", + "labelling": {"pulse_merge_gap": "60s", "min_period": "1m", "max_period": "1h", + "min_pulses": 8, "max_harmonic": 3, + "tolerance_abs": "30s", "tolerance_rel": 0.1}, + "bots": bots}, indent=1))