Skip to content

fix(demo): verify demo pack results in CI - #411

Merged
catinspace-au merged 2 commits into
mainfrom
test/demo-verify
Nov 16, 2026
Merged

catinspace-au merged 2 commits into
mainfrom
test/demo-verify

Conversation

@catinspace-au

Copy link
Copy Markdown
Contributor

Each Published compose leg runs scripts/demo_verify.py after the secrets step. A pack that deploys but lands unparsed rows, dead-letter records, empty dashboards or a silent hunt fails CI. resources//verify.json sets each pack's expected results.

The demo stacks cap a rule at 20 detections a run. Replay loops the same events. Without the cap, a rule wrote its whole window every five minutes. Each pack has two or three hunts, where the base demo had none.

Feodo Tracker replaces Wikipedia in the base demo. verify.json marks Feodo Tracker as waiting on a fetcher and engine that land dump units in their own table. Its rows are not checked yet.

The Spamhaus DROP fetcher is missing. The DROP terms grant no licence to copy the data into this repo and bar the Spamhaus name from marketing material.

2517 tests pass. Every pack's checker passes on a deployed stack. Base demo search check results: HTTP 401 on one run, pass on rerun. The search check reports why when secrets-init gives no login.

Each check fails a deliberately broken run.

The Published compose legs counted rows per source table. That let packs pass with unparsed rows, dead-letter records, empty dashboards or a hunt that never fired. scripts/demo_verify.py runs in every leg after the secrets step and fails on these problems.

It checks that:
- no landed row has error.message or a pipeline_error tag
- fields in resources/<pack>/verify.json are set on at least their ratio of rows
- the intake's own table and dfe.main hold nothing
- every source is deployed in its declared state
- every hunt rule writes detections and each hunt stays under its max_alerts
- every dashboard tile and saved search returns rows, except those marked as intentionally empty in verify.json
- a saved search returns rows as the console login, after its first password change
- replayed fetcher polls land
- a fetcher with no captured poll is skipped with its reason
- dfe.dlq and the five dead-letter topics exist and hold nothing

The demo stacks cap a rule at 20 detections a run. A replay loops the same events, and an uncapped rule wrote its whole window every five minutes. Each pack has two or three hunts. The base demo had none.

- the Feodo Tracker C2 blocklist replaces the base demo's Wikipedia fetcher and has a captured poll. verify.json marks it as waiting on a fetcher and engine that land dump units in their own table.
- demo-syslog-live is removed because it copied demo-syslog-unclaimed over a table that must be empty
- the telco burst tile reads the updates, since the cap limits the detections it read
- demo-telco-prefix-history watches the prefix that flaps
- demo-theatre-vessel-track follows the vessel with the most reports
- demo-theatre-vessel-silent matches a wider window, so it fires within minutes
A source listed as waiting in verify.json disabled the whole dfe.main check. This left cisco-asa and fortinet rows unchecked.

- the check counts other sources' rows in dfe.main
- it fails if the waiting source holds rows, catching a stale waiting entry
- the check runs every saved search as the console login instead of just the first
- the Feodo source drops its replay unit and captured poll because make demo has no demo-replay service
- the evaluating guide says dfe.feodo-c2 starts empty and why
@catinspace-au
catinspace-au merged commit 56aa0dc into main Nov 16, 2026
16 checks passed
@catinspace-au
catinspace-au deleted the test/demo-verify branch November 16, 2026 07:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant