Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions .github/workflows/check.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
name: check
on: [push, pull_request]
jobs:
check:
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@v4
- run: pip install pyyaml
- run: python3 tools/check.py
- name: promtool check rules
run: |
docker run --rm -v "$PWD/prometheus:/p:ro" --entrypoint promtool prom/prometheus:v3.5.0 check rules /p/rules/dyode.rules.yml
83 changes: 73 additions & 10 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,17 +1,80 @@
# dyode_monitoring

Monitoring for [DYODE](https://github.com/nc-intezra/dyode_rust) (Rust + lidi):
dashboards, alert rules and templates for the API the output box serves
from DYODE 2.1.0 on.
Monitoring for [DYODE](https://github.com/nc-intezra/dyode_rust) 2.1.0 and
later: Prometheus alert rules, a Grafana dashboard and a Zabbix 7 template,
all reading the **output box's** local API.

Planned contents:
The output box reports both boxes. The input box's status reaches it
through the diode every `lidi.heartbeat` seconds, so nothing here connects
to the input box or the OT network. The most important signal is
`dyode_status_age_seconds{side="in"}`: when it grows, the input box, its
NIC, the diode or the output box's receiver has failed.

| Folder | What |
|---|---|
| `prometheus/` | scrape configs and alert rules (`dyode_status_age_seconds`, rejected and missing files, module restarts) |
| `grafana/` | dashboards for both boxes and the relayed OT metrics |
| `zabbix/` | Zabbix 7.x templates: HTTP agent items on `/v1/status` and `/metrics` |
| `prometheus/scrape.yml` | the scrape job (HTTPS, bearer token, the box's certificate) |
| `prometheus/rules/dyode.rules.yml` | 14 alert rules: silent input box, configuration mismatch, aborted transfers, rejected and missing files, module restarts, disk, ARP entry, API refusals |
| `grafana/dyode-overview.json` | dashboard: link health, throughput, modules, free space, API |
| `zabbix/dyode_by_http.yaml` | Zabbix 7.0 template "DYODE by HTTP": one HTTP agent item on `/v1/status`, dependent items, module discovery for both boxes, triggers |
| `examples/` | real `/metrics` and `/v1/status` from a 2.1.0 loopback run |
| `tools/` | generators for the dashboard and template, and `check.py` |

Everything here reads the output box (IT side). Nothing here connects to
the input box or the OT network: DYODE carries the input box's status, and
relayed OT metrics, through the diode.
## On the output box

```bash
sudo dyode api-cert --name dyode-out --ip 192.168.50.4 # once
sudo dyode api-token add prometheus --scope read # one token per client
sudo dyode api-token add zabbix --scope read
```

and in config.yaml (both boxes carry the same file; only the output box
uses `api.out`):

```yaml
api:
out:
listen: 192.168.50.4:8443
tls: {cert: api.crt, key: api.key}
```

The guided setup (`dyode setup`) does the same on its "Local API" screen.
See [docs/API.md](https://github.com/nc-intezra/dyode_rust/blob/main/docs/API.md).

## Prometheus

1. Copy the output box's `/etc/dyode/api.crt` to `/etc/prometheus/dyode-out.crt`
and the token to `/etc/prometheus/dyode.token`.
2. Add `prometheus/scrape.yml`'s job (keep `job_name: dyode`: the rules use it).
3. Add `prometheus/rules/dyode.rules.yml` to `rule_files` and reload.

## Grafana

Import `grafana/dyode-overview.json` (Dashboards > New > Import) and pick the
Prometheus data source. Variables: job, output box.

## Zabbix 7

1. Import `zabbix/dyode_by_http.yaml` (Data collection > Templates > Import).
2. Create a host for the output box and link "DYODE by HTTP".
3. Set the macros: `{$DYODE.URL}` = `https://192.168.50.4:8443`,
`{$DYODE.TOKEN}` = the zabbix token (secret text).
4. The Zabbix server must trust the box's certificate: put `api.crt` in its
`SSLCALocation`, or turn off "Verify peer" on the "DYODE: Get status" item.

Zabbix 7 can also read `/metrics` with an HTTP agent item and Prometheus
preprocessing, if you prefer to keep one source for both tools.

## Changing things

The dashboard and the template are generated: edit `tools/gen_grafana.py`
or `tools/gen_zabbix.py`, then

```bash
python3 tools/gen_grafana.py grafana/dyode-overview.json
python3 tools/gen_zabbix.py zabbix/dyode_by_http.yaml
python3 tools/check.py # parses everything; checks every metric and
# JSONPath against examples/, and that the
# generated files are current
```

CI runs `check.py` and `promtool check rules`.
48 changes: 48 additions & 0 deletions examples/metrics.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
# HELP dyode_info DYODE version and transport hash of each box (1 when known).
# TYPE dyode_info gauge
dyode_info{side="in",version="2.1.0",transport_hash="ea54ffbf38d4",config="loopback test"} 1
dyode_info{side="out",version="2.1.0",transport_hash="ea54ffbf38d4",config="loopback test"} 1
# HELP dyode_status_age_seconds Seconds since this box's status was taken; for the input box, as seen by the output box.
# TYPE dyode_status_age_seconds gauge
dyode_status_age_seconds{side="in"} 0.6
dyode_status_age_seconds{side="out"} 0
# HELP dyode_status_received 1 when the box's status is known, 0 when the output box has never heard from the input box.
# TYPE dyode_status_received gauge
dyode_status_received{side="in"} 1
dyode_status_received{side="out"} 1
# HELP dyode_uptime_seconds Seconds since DYODE started on the box.
# TYPE dyode_uptime_seconds gauge
dyode_uptime_seconds{side="in"} 6
dyode_uptime_seconds{side="out"} 7
# HELP dyode_config_match 1 when both boxes run the same transport configuration.
# TYPE dyode_config_match gauge
dyode_config_match 1
# HELP dyode_module_up 1 when the module is running, 0 while it restarts.
# TYPE dyode_module_up gauge
dyode_module_up{side="in",module="files",type="folder"} 1
dyode_module_up{side="out",module="files",type="folder"} 1
# HELP dyode_module_restarts_total Times the module stopped and was restarted.
# TYPE dyode_module_restarts_total counter
dyode_module_restarts_total{side="in",module="files"} 0
dyode_module_restarts_total{side="out",module="files"} 0
# HELP dyode_events_total transfer.jsonl events by module and event name.
# TYPE dyode_events_total counter
dyode_events_total{side="in",module="files",event="batch_sent"} 1
dyode_events_total{side="in",module="files",event="file_sent"} 2
dyode_events_total{side="out",module="files",event="batch_received"} 1
dyode_events_total{side="out",module="files",event="file_stored"} 2
dyode_events_total{side="out",module="files",event="manifest_received"} 1
# HELP dyode_bytes_total File bytes sent (input box) or stored (output box).
# TYPE dyode_bytes_total counter
dyode_bytes_total{side="in",module="files",direction="sent"} 2000003
dyode_bytes_total{side="out",module="files",direction="stored"} 2000003
# HELP dyode_last_activity_timestamp_seconds Unix time of the module's last event.
# TYPE dyode_last_activity_timestamp_seconds gauge
dyode_last_activity_timestamp_seconds{side="in",module="files"} 1793160683.934
dyode_last_activity_timestamp_seconds{side="out",module="files"} 1793160684.126
# HELP dyode_free_bytes Free space under a module's folder.
# TYPE dyode_free_bytes gauge
dyode_free_bytes{side="in",module="files",dir="in"} 27251560448
dyode_free_bytes{side="out",module="files",dir="out"} 27251556352
# HELP dyode_counter_total Box-wide counters: lidi transfers aborted, API requests refused, and events without a module.
# TYPE dyode_counter_total counter
73 changes: 73 additions & 0 deletions examples/status.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
{
"boxes": {
"in": {
"config_name": "loopback test",
"counters": {},
"modules": [
{
"bytes_sent": 2000003,
"bytes_stored": 0,
"endpoint": 0,
"events": {
"batch_sent": 1,
"file_sent": 2
},
"free": [
{
"dir": "in",
"free_bytes": 27251560448,
"path": "/srv/dyode/in"
}
],
"last_activity": "2026-10-28T04:11:23.934Z",
"name": "files",
"restarts": 0,
"state": "running",
"type": "folder"
}
],
"received": "2026-10-28T04:11:28.075Z",
"side": "in",
"status_age_s": 0.6,
"time": "2026-10-28T04:11:28.027Z",
"transport_hash": "ea54ffbf38d4",
"uptime_s": 6,
"version": "2.1.0"
},
"out": {
"config_name": "loopback test",
"counters": {},
"modules": [
{
"bytes_sent": 0,
"bytes_stored": 2000003,
"endpoint": 0,
"events": {
"batch_received": 1,
"file_stored": 2,
"manifest_received": 1
},
"free": [
{
"dir": "out",
"free_bytes": 27251552256,
"path": "/srv/dyode/out"
}
],
"last_activity": "2026-10-28T04:11:24.126Z",
"name": "files",
"restarts": 0,
"state": "running",
"type": "folder"
}
],
"side": "out",
"status_age_s": 0,
"time": "2026-10-28T04:11:28.692Z",
"transport_hash": "ea54ffbf38d4",
"uptime_s": 7,
"version": "2.1.0"
}
},
"warnings": []
}
Loading
Loading