Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 9 additions & 30 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -303,7 +303,7 @@ jobs:
name: Published compose (${{ matrix.pack }})
needs: demo-packs
runs-on: ubuntu-26.04
timeout-minutes: 50
timeout-minutes: 65
strategy:
fail-fast: false
matrix:
Expand Down Expand Up @@ -350,35 +350,6 @@ jobs:
docker compose -f "$FILE" up -d --wait --wait-timeout 1200
curl -fsSL --retry 5 --retry-all-errors -o /dev/null -w 'console answered %{http_code} at %{url_effective}\n' http://localhost:3000/

# Every source with events the pack replays, which need no egress, so a pack whose demo.json lists no sources checks none. Fetcher tables wait on their upstream and are not checked
- name: Land the pack's data
timeout-minutes: 10
working-directory: ${{ env.COMPOSE_DIR }}
run: |
tables=()
for source in $(jq -r '.sources[]' "$GITHUB_WORKSPACE/$PACK_DIR/demo.json"); do
if [ -d "$GITHUB_WORKSPACE/$PACK_DIR/events/$source" ]; then
tables+=("$source")
fi
done
if [ "${#tables[@]}" -eq 0 ]; then
echo "$PACK_DIR replays no source, so no table is checked"
fi
for table in "${tables[@]}"; do
for attempt in $(seq 60); do
rows="$(docker compose -f "$FILE" exec -T clickhouse sh -c ". /run/dfe-secrets/env && clickhouse-client --password \"\$CLICKHOUSE_PASSWORD\" --query 'SELECT count() FROM dfe.\`$table\`'" || echo 0)"
if [ "${rows:-0}" -gt 0 ]; then
echo "dfe.$table holds $rows rows"
break
fi
if [ "$attempt" -eq 60 ]; then
echo "::error::no rows reached dfe.$table"
exit 1
fi
sleep 5
done
done

# The secrets are this runner's own: no service holds one in its environment, ClickHouse refuses its empty default password, and the login the file prints is held for a change
- name: Run on secrets minted here
working-directory: ${{ env.COMPOSE_DIR }}
Expand Down Expand Up @@ -413,6 +384,14 @@ jobs:
with urllib.request.urlopen(request, timeout=30) as answer:
assert json.load(answer).get("password_change_required") is True, "the first login is not held for a change"'

# Runs after the secrets step because the console login check replaces the login's issued password, which that step expects held. One checker proves the pack worked: its replayed sources land parsed rows, the intake and default tables stay empty, every source is deployed, each hunt fires under its ceiling, each dashboard and saved search returns rows, the console login can search, replayed fetcher polls land and the dead-letter queue stays empty. What a pack expects is data in resources/<pack>/verify.json
- name: Verify the pack worked
timeout-minutes: 35
working-directory: ${{ env.COMPOSE_DIR }}
env:
PACK: ${{ matrix.pack }}
run: python3 "$GITHUB_WORKSPACE/scripts/demo_verify.py" --pack "$PACK" --file "$FILE"

- name: Show what did not come up
if: failure()
working-directory: ${{ env.COMPOSE_DIR }}
Expand Down
8 changes: 4 additions & 4 deletions docs/evaluating.md
Original file line number Diff line number Diff line change
Expand Up @@ -173,23 +173,23 @@ The demo is one set of pieces, listed in [`resources/demo/demo.json`](../resourc
| Piece | Where | What it is |
|-------|-------|------------|
| Sources | [`resources/demo/sources/`](../resources/demo/sources/) | One `POST /sources` body per file: the `demo-syslog` intake, the `cisco-asa` and `fortinet` tables, and two fetchers |
| Fetchers | `wikipedia-recentchanges`, `component-releases` | Pull beside the receiver's push. `wikipedia-recentchanges` polls the MediaWiki recent changes API every 5 seconds, asking each window once, with the first poll bringing in the last hour. `component-releases` takes the latest GitHub release of each public DFE component every 15 minutes, to stay inside GitHub's 60 unauthenticated requests an hour |
| Fetchers | `feodo-c2`, `component-releases` | Pull beside the receiver's push. `feodo-c2` takes the abuse.ch Feodo Tracker botnet command-and-control blocklist every 5 minutes, as one snapshot a poll, and until the fetcher and engine land a dump unit in its own table its rows land in `dfe.main`. `component-releases` takes the latest GitHub release of each public DFE component every 15 minutes, to stay inside GitHub's 60 unauthenticated requests an hour |
| Archive | `demo-syslog` has `archive: true` | On a Kafka profile dfe-archiver writes the raw syslog lines to the `archiver-data` volume (`file:///var/lib/dfe-archiver/archive`). dfe-archiver has no retention of its own, so the pack runs a pruner that keeps the archive to 15 minutes and 200 MB. A gRPC profile runs no archiver, so there the Source is created without archiving |
| Console | [`resources/demo/console/`](../resources/demo/console/) | A dashboard per source and one for the stack's health, and saved searches with a live tail for each streamed source. Every name starts `demo-` |
| Tables | `dfe.cisco-asa`, `dfe.fortinet`, `dfe.demo-syslog`, `dfe.component-releases`, `dfe.wikipedia-recentchanges` | `make demo-stop` leaves them, with their rows, because only the engine issues ClickHouse DDL and it never drops a table. The next `make demo` goes on in them, and `make clean` removes them with the volumes |
| Tables | `dfe.cisco-asa`, `dfe.fortinet`, `dfe.demo-syslog`, `dfe.component-releases`, `dfe.feodo-c2` | `make demo-stop` leaves them, with their rows, because only the engine issues ClickHouse DDL and it never drops a table. The next `make demo` goes on in them, and `make clean` removes them with the volumes |

Each device loops over the sample lines in [`resources/demo/events/<source>/`](../resources/demo/events/), which `python3 resources/demo/events/generate.py` regenerates.
The run turns on dfe-receiver's syslog listener (UDP and TCP on port 514) and recreates what that changes. It returns once every device table has gained rows and the archive holds a file, and leaves the stream running:

```console
demo - console demo-cisco-asa: http://localhost:8090/dashboards/<id>
demo - console demo-wikipedia-live: http://localhost:8090/search/<id>?isLive=true
demo - console demo-feodo-c2-all: http://localhost:8090/search/<id>
demo - dfe.cisco-asa: +N rows
demo - dfe.fortinet: +N rows
demo - dfe.demo-syslog: +0 rows
demo - dfe.main: +0 rows
demo - dfe.component-releases: 0 rows
demo - dfe.wikipedia-recentchanges: N rows
demo - dfe.feodo-c2: 0 rows
demo - dfe-archiver archive of demo-syslog: N files
demo - demo-syslog_land/default/2026/11/03/01-0001-<id>.jsonl.zst 77645 bytes
demo - first line of the newest: {"message":"%ASA-6-302014: Teardown TCP connection ...
Expand Down
4 changes: 3 additions & 1 deletion resources/demo-cyber/demo.json
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,9 @@
]
},
"hunts": [
"demo-cyber-drop-sign-in"
"demo-cyber-anonymous-link",
"demo-cyber-drop-sign-in",
"demo-cyber-ransomware"
],
"sources": [
"cisco-asa",
Expand Down
19 changes: 19 additions & 0 deletions resources/demo-cyber/hunts/demo-cyber-anonymous-link.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
# Project: dfe-docker
# File: resources/demo-cyber/hunts/demo-cyber-anonymous-link.yaml
# Purpose: The cyber demo's hunt, running its anonymous-link rule every five minutes
# Language: YAML
#
# License: BUSL-1.1
# Copyright: (c) 2026 HYPERI PTY LIMITED

name: demo-cyber-anonymous-link
display_name: Anonymous links used on Microsoft 365 files
cron: "*/5 * * * *"
# A required label: the runner takes each detection's org from the matched row.
customers:
- default
rules:
- demo-cyber-anonymous-link
global_source_table_name: dfe.o365
global_target_table_name: dfe.detection
checkpoint_timestamp_field: _timestamp_load
19 changes: 19 additions & 0 deletions resources/demo-cyber/hunts/demo-cyber-ransomware.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
# Project: dfe-docker
# File: resources/demo-cyber/hunts/demo-cyber-ransomware.yaml
# Purpose: The cyber demo's hunt, running its ransomware rule every five minutes
# Language: YAML
#
# License: BUSL-1.1
# Copyright: (c) 2026 HYPERI PTY LIMITED

name: demo-cyber-ransomware
display_name: Ransomware behaviour on endpoints
cron: "*/5 * * * *"
# A required label: the runner takes each detection's org from the matched row.
customers:
- default
rules:
- demo-cyber-ransomware
global_source_table_name: dfe.crowdstrike
global_target_table_name: dfe.detection
checkpoint_timestamp_field: _timestamp_load
14 changes: 14 additions & 0 deletions resources/demo-cyber/rules/demo-cyber-anonymous-link.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
# Project: dfe-docker
# File: resources/demo-cyber/rules/demo-cyber-anonymous-link.yaml
# Purpose: The cyber demo's rule: an anonymous sharing link used to open a Microsoft 365 file
# Language: YAML
#
# License: BUSL-1.1
# Copyright: (c) 2026 HYPERI PTY LIMITED

name: demo-cyber-anonymous-link
display_name: Anonymous link used on a Microsoft 365 file
severity: medium
source: o365
source_type: raw
user_sql: "SELECT * FROM dfe.`o365` WHERE event_action = 'AnonymousLinkUsed'"
14 changes: 14 additions & 0 deletions resources/demo-cyber/rules/demo-cyber-ransomware.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
# Project: dfe-docker
# File: resources/demo-cyber/rules/demo-cyber-ransomware.yaml
# Purpose: The cyber demo's rule: a high-severity CrowdStrike Falcon detection of ransomware behaviour
# Language: YAML
#
# License: BUSL-1.1
# Copyright: (c) 2026 HYPERI PTY LIMITED

name: demo-cyber-ransomware
display_name: Ransomware behaviour on an endpoint
severity: critical
source: crowdstrike
source_type: raw
user_sql: "SELECT * FROM dfe.`crowdstrike` WHERE rule_name = 'Ransomware Behaviour' AND event_severity >= 70"
63 changes: 63 additions & 0 deletions resources/demo-cyber/verify.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
{
"hunts": {
"demo-cyber-anonymous-link": {
"max_alerts": 126
},
"demo-cyber-drop-sign-in": {
"max_alerts": 126
},
"demo-cyber-ransomware": {
"max_alerts": 126
}
},
"parsed": {
"cisco-asa": {
"fields": [
"event_action",
"source_ip",
"destination_ip",
"observer_vendor"
],
"min_ratio": 0.9
},
"crowdstrike": {
"fields": [
"event_action",
"user_name",
"source_ip"
],
"min_ratio": 0.99
},
"fortinet": {
"fields": [
"event_action",
"event_outcome",
"source_ip",
"destination_ip",
"source_port",
"network_transport",
"observer_vendor"
],
"min_ratio": 0.99
},
"o365": {
"fields": [
"event_action",
"event_outcome",
"user_name",
"client_ip"
],
"min_ratio": 0.99
},
"okta": {
"fields": [
"event_action",
"event_outcome",
"client_ip",
"user_name"
],
"min_ratio": 0.99
}
},
"zero_ok": []
}
Original file line number Diff line number Diff line change
Expand Up @@ -51,12 +51,12 @@
"config": {
"configType": "sql",
"displayType": "table",
"name": "Peers over 20 withdrawals a minute, from hunt demo-telco-peer-withdrawals",
"name": "Peers over 20 withdrawals a minute",
"source": "bgp-updates",
"sqlTemplate": "SELECT formatDateTime(toStartOfMinute(_timestamp), '%H:%i') AS \"RIS minute (UTC)\",\n concat(splitByChar('-', toString(_json.attrs.source_id))[2], ' (rrc', splitByChar('-', toString(_json.attrs.source_id))[1], ')') AS \"Peer\",\n count() AS \"Withdrawals\"\nFROM dfe.detection\nWHERE hunt_name = 'demo-telco-peer-withdrawals' AND $__timeFilter(_timestamp + INTERVAL 12 HOUR)\nGROUP BY toStartOfMinute(_timestamp), 1, 2\nHAVING \"Withdrawals\" > 20\nORDER BY toStartOfMinute(_timestamp) DESC, 3 DESC\nLIMIT 20"
"sqlTemplate": "SELECT formatDateTime(toStartOfMinute(_timestamp), '%H:%i') AS \"RIS minute (UTC)\",\n concat(splitByChar('-', toString(_json.attrs.source_id))[2], ' (rrc', splitByChar('-', toString(_json.attrs.source_id))[1], ')') AS \"Peer\",\n count() AS \"Withdrawals\"\nFROM dfe.`bgp-updates`\nWHERE toString(_json.type) = 'W' AND $__timeFilter(_timestamp + INTERVAL 12 HOUR)\nGROUP BY toStartOfMinute(_timestamp), 1, 2\nHAVING \"Withdrawals\" > 20\nORDER BY toStartOfMinute(_timestamp) DESC, 3 DESC\nLIMIT 20"
},
"h": 6,
"id": "hunt",
"id": "bursts",
"w": 12,
"x": 0,
"y": 15
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,6 @@
"tags": [
"demo"
],
"where": "toString(_json.attrs.target_prefix) = '172.69.12.0/24'",
"where": "toString(_json.attrs.target_prefix) = '173.82.227.0/24'",
"whereLanguage": "sql"
}
4 changes: 3 additions & 1 deletion resources/demo-telco/demo.json
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,9 @@
]
},
"hunts": [
"demo-telco-peer-withdrawals"
"demo-telco-hss-too-busy",
"demo-telco-peer-withdrawals",
"demo-telco-weak-signal"
],
"sources": [
"bgp-updates",
Expand Down
19 changes: 19 additions & 0 deletions resources/demo-telco/hunts/demo-telco-hss-too-busy.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
# Project: dfe-docker
# File: resources/demo-telco/hunts/demo-telco-hss-too-busy.yaml
# Purpose: The telco demo's hunt, running its HSS overload rule every five minutes
# Language: YAML
#
# License: BUSL-1.1
# Copyright: (c) 2026 HYPERI PTY LIMITED

name: demo-telco-hss-too-busy
display_name: HSS overload on S6a
cron: "*/5 * * * *"
# A required label: the runner takes each detection's org from the matched row.
customers:
- default
rules:
- demo-telco-hss-too-busy
global_source_table_name: dfe.core-probes
global_target_table_name: dfe.detection
checkpoint_timestamp_field: _timestamp_load
19 changes: 19 additions & 0 deletions resources/demo-telco/hunts/demo-telco-weak-signal.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
# Project: dfe-docker
# File: resources/demo-telco/hunts/demo-telco-weak-signal.yaml
# Purpose: The telco demo's hunt, running its weak-signal rule every five minutes
# Language: YAML
#
# License: BUSL-1.1
# Copyright: (c) 2026 HYPERI PTY LIMITED

name: demo-telco-weak-signal
display_name: Cells with a very weak signal
cron: "*/5 * * * *"
# A required label: the runner takes each detection's org from the matched row.
customers:
- default
rules:
- demo-telco-weak-signal
global_source_table_name: dfe.ran-kpis
global_target_table_name: dfe.detection
checkpoint_timestamp_field: _timestamp_load
14 changes: 14 additions & 0 deletions resources/demo-telco/rules/demo-telco-hss-too-busy.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
# Project: dfe-docker
# File: resources/demo-telco/rules/demo-telco-hss-too-busy.yaml
# Purpose: The telco demo's rule: an S6a request the HSS refused with DIAMETER_TOO_BUSY
# Language: YAML
#
# License: BUSL-1.1
# Copyright: (c) 2026 HYPERI PTY LIMITED

name: demo-telco-hss-too-busy
display_name: HSS refused an S6a request as too busy
severity: high
source: core-probes
source_type: raw
user_sql: "SELECT * FROM dfe.`core-probes` WHERE toString(_json.cause) = 'DIAMETER_TOO_BUSY'"
Original file line number Diff line number Diff line change
Expand Up @@ -11,5 +11,5 @@ display_name: BGP withdrawal from a RIS peer
severity: medium
source: bgp-updates
source_type: raw
# A rule matches single rows and may not read another table, so the burst, more than 20 withdrawals in a minute from one peer, is counted where the demo-telco-bgp-peers dashboard reads the detections.
# A rule matches single rows and may not read another table, so the burst, more than 20 withdrawals in a minute from one peer, is counted where the demo-telco-bgp-peers dashboard reads the updates, and each run records at most the engine's per-run detection cap.
user_sql: "SELECT * FROM dfe.`bgp-updates` WHERE toString(_json.type) = 'W'"
14 changes: 14 additions & 0 deletions resources/demo-telco/rules/demo-telco-weak-signal.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
# Project: dfe-docker
# File: resources/demo-telco/rules/demo-telco-weak-signal.yaml
# Purpose: The telco demo's rule: a cell reporting an average RSRP below -102 dBm
# Language: YAML
#
# License: BUSL-1.1
# Copyright: (c) 2026 HYPERI PTY LIMITED

name: demo-telco-weak-signal
display_name: Cell reporting a very weak signal
severity: medium
source: ran-kpis
source_type: raw
user_sql: "SELECT * FROM dfe.`ran-kpis` WHERE toFloat64OrZero(toString(_json.rsrp_dbm)) < -102"
38 changes: 38 additions & 0 deletions resources/demo-telco/verify.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
{
"hunts": {
"demo-telco-hss-too-busy": {
"max_alerts": 126
},
"demo-telco-peer-withdrawals": {
"max_alerts": 126
},
"demo-telco-weak-signal": {
"max_alerts": 126
}
},
"parsed": {
"core-probes": {
"fields": [
"_json.protocol",
"_json.procedure",
"_json.result",
"_json.latency_ms",
"_json.source_node"
],
"min_ratio": 0.99
},
"ran-kpis": {
"fields": [
"_json.cell",
"_json.site",
"_json.rsrp_dbm",
"_json.prb_util_dl_pct",
"_json.dl_throughput_mbps"
],
"min_ratio": 0.99
}
},
"zero_ok": [
"demo-telco-bgp-peers/bursts"
]
}
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,6 @@
"tags": [
"demo"
],
"where": "toString(_json.mmsi) = '209297000'",
"where": "toString(_json.mmsi) = (SELECT toString(_json.mmsi) FROM dfe.`ais-tracks` GROUP BY 1 ORDER BY count() DESC, 1 LIMIT 1)",
"whereLanguage": "sql"
}
1 change: 1 addition & 0 deletions resources/demo-theatre/demo.json
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@
]
},
"hunts": [
"demo-theatre-aircraft-low",
"demo-theatre-vessel-silent"
],
"sources": [
Expand Down
Loading
Loading