Skip to content

Six of fifteen built-in rules have no benchmark corpus coverage #36

Description

@blitzcrieg1

python -m cli benchmark reports rules covered 9 against 15 built-in rules.
Six rules have no corpus case at all, so nothing detects it if they silently
stop firing.

The benchmark corpus is the strongest evidence this project has that its
detections are trustworthy. Six rules currently sit outside that evidence.

Uncovered rules

Confirm the exact set by comparing BUILTIN_RULE_IDS in
core/audit/detection/rules.py with the rules that fire during a benchmark run.
As of writing, the covered nine come from the existing 12 attack sessions, so
the gap is roughly the rules relating to approval flow, off-hours activity, and
the swarm and burst family.

Task

Add one attack case per uncovered rule to
apps/orchestrator/tests/fixtures/detection_corpus/, registered in
corpus.yaml.

Each case is a JSONL session that would have made a real operator look twice.
Synthetic sessions that exist only to satisfy the matcher teach the rule nothing
and give false confidence.

Notes

Rules with time semantics need care. Off-hours detection depends on the business
timezone resolution in _business_tz(), and a case that only passes in one
timezone is worse than no case.

If a rule turns out to be genuinely hard to trigger with a realistic session,
that is a finding worth reporting rather than a case worth forcing. A rule that
cannot be provoked by plausible activity may be tuned wrongly.

Acceptance criteria

  • rules covered equals the number of built-in rules
  • Benign case count grows alongside, so coverage does not come at the cost
    of the false-positive claim
  • pytest -q green

Metadata

Metadata

Assignees

No one assigned

    Labels

    help wantedExtra attention is welcome

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions