From 0450b562ab1ed9ae135aa97f714e280ccadf40c9 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 28 Oct 2026 04:12:11 +0000 Subject: [PATCH] Prometheus rules, Grafana dashboard and Zabbix 7 template for DYODE 2.1.0 All read the output box's local API, which reports both boxes (the input box from the status it sends through the diode). tools/check.py checks every metric and JSONPath against a real /metrics and /v1/status capture, and that the generated dashboard and template are current; CI also runs promtool check rules. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01QXjQsiWnwLMndr5XU9g5LJ --- .github/workflows/check.yml | 12 + README.md | 83 +++- examples/metrics.txt | 48 +++ examples/status.json | 73 ++++ grafana/dyode-overview.json | 691 +++++++++++++++++++++++++++++++ prometheus/rules/dyode.rules.yml | 109 +++++ prometheus/scrape.yml | 17 + tools/check.py | 113 +++++ tools/gen_grafana.py | 103 +++++ tools/gen_zabbix.py | 148 +++++++ zabbix/dyode_by_http.yaml | 557 +++++++++++++++++++++++++ 11 files changed, 1944 insertions(+), 10 deletions(-) create mode 100644 .github/workflows/check.yml create mode 100644 examples/metrics.txt create mode 100644 examples/status.json create mode 100644 grafana/dyode-overview.json create mode 100644 prometheus/rules/dyode.rules.yml create mode 100644 prometheus/scrape.yml create mode 100644 tools/check.py create mode 100644 tools/gen_grafana.py create mode 100644 tools/gen_zabbix.py create mode 100644 zabbix/dyode_by_http.yaml diff --git a/.github/workflows/check.yml b/.github/workflows/check.yml new file mode 100644 index 0000000..a172f29 --- /dev/null +++ b/.github/workflows/check.yml @@ -0,0 +1,12 @@ +name: check +on: [push, pull_request] +jobs: + check: + runs-on: ubuntu-24.04 + steps: + - uses: actions/checkout@v4 + - run: pip install pyyaml + - run: python3 tools/check.py + - name: promtool check rules + run: | + docker run --rm -v "$PWD/prometheus:/p:ro" --entrypoint promtool prom/prometheus:v3.5.0 check rules /p/rules/dyode.rules.yml diff --git a/README.md b/README.md index 5f94962..863c394 100644 --- a/README.md +++ b/README.md @@ -1,17 +1,80 @@ # dyode_monitoring -Monitoring for [DYODE](https://github.com/nc-intezra/dyode_rust) (Rust + lidi): -dashboards, alert rules and templates for the API the output box serves -from DYODE 2.1.0 on. +Monitoring for [DYODE](https://github.com/nc-intezra/dyode_rust) 2.1.0 and +later: Prometheus alert rules, a Grafana dashboard and a Zabbix 7 template, +all reading the **output box's** local API. -Planned contents: +The output box reports both boxes. The input box's status reaches it +through the diode every `lidi.heartbeat` seconds, so nothing here connects +to the input box or the OT network. The most important signal is +`dyode_status_age_seconds{side="in"}`: when it grows, the input box, its +NIC, the diode or the output box's receiver has failed. | Folder | What | |---|---| -| `prometheus/` | scrape configs and alert rules (`dyode_status_age_seconds`, rejected and missing files, module restarts) | -| `grafana/` | dashboards for both boxes and the relayed OT metrics | -| `zabbix/` | Zabbix 7.x templates: HTTP agent items on `/v1/status` and `/metrics` | +| `prometheus/scrape.yml` | the scrape job (HTTPS, bearer token, the box's certificate) | +| `prometheus/rules/dyode.rules.yml` | 14 alert rules: silent input box, configuration mismatch, aborted transfers, rejected and missing files, module restarts, disk, ARP entry, API refusals | +| `grafana/dyode-overview.json` | dashboard: link health, throughput, modules, free space, API | +| `zabbix/dyode_by_http.yaml` | Zabbix 7.0 template "DYODE by HTTP": one HTTP agent item on `/v1/status`, dependent items, module discovery for both boxes, triggers | +| `examples/` | real `/metrics` and `/v1/status` from a 2.1.0 loopback run | +| `tools/` | generators for the dashboard and template, and `check.py` | -Everything here reads the output box (IT side). Nothing here connects to -the input box or the OT network: DYODE carries the input box's status, and -relayed OT metrics, through the diode. +## On the output box + +```bash +sudo dyode api-cert --name dyode-out --ip 192.168.50.4 # once +sudo dyode api-token add prometheus --scope read # one token per client +sudo dyode api-token add zabbix --scope read +``` + +and in config.yaml (both boxes carry the same file; only the output box +uses `api.out`): + +```yaml +api: + out: + listen: 192.168.50.4:8443 + tls: {cert: api.crt, key: api.key} +``` + +The guided setup (`dyode setup`) does the same on its "Local API" screen. +See [docs/API.md](https://github.com/nc-intezra/dyode_rust/blob/main/docs/API.md). + +## Prometheus + +1. Copy the output box's `/etc/dyode/api.crt` to `/etc/prometheus/dyode-out.crt` + and the token to `/etc/prometheus/dyode.token`. +2. Add `prometheus/scrape.yml`'s job (keep `job_name: dyode`: the rules use it). +3. Add `prometheus/rules/dyode.rules.yml` to `rule_files` and reload. + +## Grafana + +Import `grafana/dyode-overview.json` (Dashboards > New > Import) and pick the +Prometheus data source. Variables: job, output box. + +## Zabbix 7 + +1. Import `zabbix/dyode_by_http.yaml` (Data collection > Templates > Import). +2. Create a host for the output box and link "DYODE by HTTP". +3. Set the macros: `{$DYODE.URL}` = `https://192.168.50.4:8443`, + `{$DYODE.TOKEN}` = the zabbix token (secret text). +4. The Zabbix server must trust the box's certificate: put `api.crt` in its + `SSLCALocation`, or turn off "Verify peer" on the "DYODE: Get status" item. + +Zabbix 7 can also read `/metrics` with an HTTP agent item and Prometheus +preprocessing, if you prefer to keep one source for both tools. + +## Changing things + +The dashboard and the template are generated: edit `tools/gen_grafana.py` +or `tools/gen_zabbix.py`, then + +```bash +python3 tools/gen_grafana.py grafana/dyode-overview.json +python3 tools/gen_zabbix.py zabbix/dyode_by_http.yaml +python3 tools/check.py # parses everything; checks every metric and + # JSONPath against examples/, and that the + # generated files are current +``` + +CI runs `check.py` and `promtool check rules`. diff --git a/examples/metrics.txt b/examples/metrics.txt new file mode 100644 index 0000000..ccc6ae0 --- /dev/null +++ b/examples/metrics.txt @@ -0,0 +1,48 @@ +# HELP dyode_info DYODE version and transport hash of each box (1 when known). +# TYPE dyode_info gauge +dyode_info{side="in",version="2.1.0",transport_hash="ea54ffbf38d4",config="loopback test"} 1 +dyode_info{side="out",version="2.1.0",transport_hash="ea54ffbf38d4",config="loopback test"} 1 +# HELP dyode_status_age_seconds Seconds since this box's status was taken; for the input box, as seen by the output box. +# TYPE dyode_status_age_seconds gauge +dyode_status_age_seconds{side="in"} 0.6 +dyode_status_age_seconds{side="out"} 0 +# HELP dyode_status_received 1 when the box's status is known, 0 when the output box has never heard from the input box. +# TYPE dyode_status_received gauge +dyode_status_received{side="in"} 1 +dyode_status_received{side="out"} 1 +# HELP dyode_uptime_seconds Seconds since DYODE started on the box. +# TYPE dyode_uptime_seconds gauge +dyode_uptime_seconds{side="in"} 6 +dyode_uptime_seconds{side="out"} 7 +# HELP dyode_config_match 1 when both boxes run the same transport configuration. +# TYPE dyode_config_match gauge +dyode_config_match 1 +# HELP dyode_module_up 1 when the module is running, 0 while it restarts. +# TYPE dyode_module_up gauge +dyode_module_up{side="in",module="files",type="folder"} 1 +dyode_module_up{side="out",module="files",type="folder"} 1 +# HELP dyode_module_restarts_total Times the module stopped and was restarted. +# TYPE dyode_module_restarts_total counter +dyode_module_restarts_total{side="in",module="files"} 0 +dyode_module_restarts_total{side="out",module="files"} 0 +# HELP dyode_events_total transfer.jsonl events by module and event name. +# TYPE dyode_events_total counter +dyode_events_total{side="in",module="files",event="batch_sent"} 1 +dyode_events_total{side="in",module="files",event="file_sent"} 2 +dyode_events_total{side="out",module="files",event="batch_received"} 1 +dyode_events_total{side="out",module="files",event="file_stored"} 2 +dyode_events_total{side="out",module="files",event="manifest_received"} 1 +# HELP dyode_bytes_total File bytes sent (input box) or stored (output box). +# TYPE dyode_bytes_total counter +dyode_bytes_total{side="in",module="files",direction="sent"} 2000003 +dyode_bytes_total{side="out",module="files",direction="stored"} 2000003 +# HELP dyode_last_activity_timestamp_seconds Unix time of the module's last event. +# TYPE dyode_last_activity_timestamp_seconds gauge +dyode_last_activity_timestamp_seconds{side="in",module="files"} 1793160683.934 +dyode_last_activity_timestamp_seconds{side="out",module="files"} 1793160684.126 +# HELP dyode_free_bytes Free space under a module's folder. +# TYPE dyode_free_bytes gauge +dyode_free_bytes{side="in",module="files",dir="in"} 27251560448 +dyode_free_bytes{side="out",module="files",dir="out"} 27251556352 +# HELP dyode_counter_total Box-wide counters: lidi transfers aborted, API requests refused, and events without a module. +# TYPE dyode_counter_total counter diff --git a/examples/status.json b/examples/status.json new file mode 100644 index 0000000..597ef73 --- /dev/null +++ b/examples/status.json @@ -0,0 +1,73 @@ +{ + "boxes": { + "in": { + "config_name": "loopback test", + "counters": {}, + "modules": [ + { + "bytes_sent": 2000003, + "bytes_stored": 0, + "endpoint": 0, + "events": { + "batch_sent": 1, + "file_sent": 2 + }, + "free": [ + { + "dir": "in", + "free_bytes": 27251560448, + "path": "/srv/dyode/in" + } + ], + "last_activity": "2026-10-28T04:11:23.934Z", + "name": "files", + "restarts": 0, + "state": "running", + "type": "folder" + } + ], + "received": "2026-10-28T04:11:28.075Z", + "side": "in", + "status_age_s": 0.6, + "time": "2026-10-28T04:11:28.027Z", + "transport_hash": "ea54ffbf38d4", + "uptime_s": 6, + "version": "2.1.0" + }, + "out": { + "config_name": "loopback test", + "counters": {}, + "modules": [ + { + "bytes_sent": 0, + "bytes_stored": 2000003, + "endpoint": 0, + "events": { + "batch_received": 1, + "file_stored": 2, + "manifest_received": 1 + }, + "free": [ + { + "dir": "out", + "free_bytes": 27251552256, + "path": "/srv/dyode/out" + } + ], + "last_activity": "2026-10-28T04:11:24.126Z", + "name": "files", + "restarts": 0, + "state": "running", + "type": "folder" + } + ], + "side": "out", + "status_age_s": 0, + "time": "2026-10-28T04:11:28.692Z", + "transport_hash": "ea54ffbf38d4", + "uptime_s": 7, + "version": "2.1.0" + } + }, + "warnings": [] +} diff --git a/grafana/dyode-overview.json b/grafana/dyode-overview.json new file mode 100644 index 0000000..ecc4668 --- /dev/null +++ b/grafana/dyode-overview.json @@ -0,0 +1,691 @@ +{ + "__requires": [ + { + "type": "grafana", + "id": "grafana", + "name": "Grafana", + "version": "10.0.0" + }, + { + "type": "datasource", + "id": "prometheus", + "name": "Prometheus", + "version": "1.0.0" + } + ], + "title": "DYODE overview", + "uid": "dyode-overview", + "description": "Both DYODE boxes, from the output box's /metrics (nc-intezra/dyode_monitoring).", + "tags": [ + "dyode", + "diode" + ], + "timezone": "browser", + "schemaVersion": 39, + "version": 1, + "refresh": "30s", + "time": { + "from": "now-6h", + "to": "now" + }, + "editable": true, + "templating": { + "list": [ + { + "name": "datasource", + "label": "Prometheus", + "type": "datasource", + "query": "prometheus", + "current": {}, + "hide": 0 + }, + { + "name": "job", + "label": "Job", + "type": "query", + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "query": { + "query": "label_values(dyode_info, job)", + "refId": "job" + }, + "definition": "label_values(dyode_info, job)", + "refresh": 1, + "current": {}, + "hide": 0 + }, + { + "name": "instance", + "label": "Output box", + "type": "query", + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "query": { + "query": "label_values(dyode_info{job=\"$job\"}, instance)", + "refId": "inst" + }, + "definition": "label_values(dyode_info{job=\"$job\"}, instance)", + "refresh": 1, + "includeAll": true, + "multi": true, + "current": {}, + "hide": 0 + } + ] + }, + "annotations": { + "list": [] + }, + "panels": [ + { + "id": 1, + "type": "row", + "title": "Link", + "collapsed": false, + "gridPos": { + "x": 0, + "y": 0, + "w": 24, + "h": 1 + }, + "panels": [] + }, + { + "id": 2, + "type": "stat", + "title": "Input box status age", + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "gridPos": { + "x": 0, + "y": 1, + "w": 6, + "h": 4 + }, + "targets": [ + { + "refId": "A", + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "expr": "dyode_status_age_seconds{side=\"in\", job=\"$job\", instance=~\"$instance\"}", + "legendFormat": "{{instance}}" + } + ], + "description": "Seconds since the input box's status arrived through the diode. Over 30 s: input box, diode or link down.", + "fieldConfig": { + "defaults": { + "unit": "s", + "decimals": 0, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "orange", + "value": 15 + }, + { + "color": "red", + "value": 30 + } + ] + } + }, + "overrides": [] + }, + "options": { + "colorMode": "background", + "graphMode": "area", + "reduceOptions": { + "calcs": [ + "lastNotNull" + ] + } + } + }, + { + "id": 3, + "type": "stat", + "title": "Configurations match", + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "gridPos": { + "x": 6, + "y": 1, + "w": 6, + "h": 4 + }, + "targets": [ + { + "refId": "A", + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "expr": "dyode_config_match{job=\"$job\", instance=~\"$instance\"}", + "legendFormat": "{{instance}}" + } + ], + "description": "1 when both boxes run the same transport configuration (lidi parameters and modules).", + "fieldConfig": { + "defaults": { + "mappings": [ + { + "type": "value", + "options": { + "0": { + "text": "DIFFERENT", + "color": "red" + }, + "1": { + "text": "same", + "color": "green" + } + } + } + ], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "red", + "value": null + }, + { + "color": "green", + "value": 1 + } + ] + } + }, + "overrides": [] + }, + "options": { + "colorMode": "background", + "graphMode": "none", + "reduceOptions": { + "calcs": [ + "lastNotNull" + ] + } + } + }, + { + "id": 4, + "type": "stat", + "title": "Transfers aborted (24 h)", + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "gridPos": { + "x": 12, + "y": 1, + "w": 6, + "h": 4 + }, + "targets": [ + { + "refId": "A", + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "expr": "increase(dyode_counter_total{side=\"out\", counter=\"lidi_transfers_aborted\", job=\"$job\", instance=~\"$instance\"}[24h])", + "legendFormat": "{{instance}}" + } + ], + "description": "lidi transfers aborted because blocks were lost in the diode.", + "fieldConfig": { + "defaults": { + "decimals": 0, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "orange", + "value": 1 + } + ] + } + }, + "overrides": [] + }, + "options": { + "colorMode": "background", + "graphMode": "none", + "reduceOptions": { + "calcs": [ + "lastNotNull" + ] + } + } + }, + { + "id": 5, + "type": "stat", + "title": "Versions", + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "gridPos": { + "x": 18, + "y": 1, + "w": 6, + "h": 4 + }, + "targets": [ + { + "refId": "A", + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "expr": "dyode_info{job=\"$job\", instance=~\"$instance\"}", + "legendFormat": "{{side}}: {{version}}" + } + ], + "options": { + "colorMode": "none", + "graphMode": "none", + "textMode": "name", + "reduceOptions": { + "calcs": [ + "lastNotNull" + ] + } + } + }, + { + "id": 6, + "type": "timeseries", + "title": "Input box status age", + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "gridPos": { + "x": 0, + "y": 5, + "w": 24, + "h": 6 + }, + "targets": [ + { + "refId": "A", + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "expr": "dyode_status_age_seconds{side=\"in\", job=\"$job\", instance=~\"$instance\"}", + "legendFormat": "status age" + } + ], + "fieldConfig": { + "defaults": { + "unit": "s", + "custom": { + "thresholdsStyle": { + "mode": "line" + } + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 30 + } + ] + } + }, + "overrides": [] + } + }, + { + "id": 7, + "type": "row", + "title": "Throughput", + "collapsed": false, + "gridPos": { + "x": 0, + "y": 11, + "w": 24, + "h": 1 + }, + "panels": [] + }, + { + "id": 8, + "type": "timeseries", + "title": "File bytes per second", + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "gridPos": { + "x": 0, + "y": 12, + "w": 12, + "h": 8 + }, + "targets": [ + { + "refId": "A", + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "expr": "sum by (side, module) (rate(dyode_bytes_total{job=\"$job\", instance=~\"$instance\"}[5m]))", + "legendFormat": "{{side}} {{module}}" + } + ], + "fieldConfig": { + "defaults": { + "unit": "Bps" + }, + "overrides": [] + } + }, + { + "id": 9, + "type": "timeseries", + "title": "Files per minute", + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "gridPos": { + "x": 12, + "y": 12, + "w": 12, + "h": 8 + }, + "targets": [ + { + "refId": "A", + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "expr": "sum by (side, module, event) (rate(dyode_events_total{event=~\"file_sent|file_stored|file_rejected|file_missing|file_failed\", job=\"$job\", instance=~\"$instance\"}[5m])) * 60", + "legendFormat": "{{side}} {{module}} {{event}}" + } + ], + "fieldConfig": { + "defaults": { + "unit": "short" + }, + "overrides": [] + } + }, + { + "id": 10, + "type": "row", + "title": "Modules", + "collapsed": false, + "gridPos": { + "x": 0, + "y": 20, + "w": 24, + "h": 1 + }, + "panels": [] + }, + { + "id": 11, + "type": "table", + "title": "Modules", + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "gridPos": { + "x": 0, + "y": 21, + "w": 12, + "h": 8 + }, + "targets": [ + { + "refId": "A", + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "expr": "dyode_module_up{job=\"$job\", instance=~\"$instance\"}", + "legendFormat": "", + "instant": true, + "format": "table" + }, + { + "refId": "B", + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "expr": "dyode_module_restarts_total{job=\"$job\", instance=~\"$instance\"}", + "legendFormat": "", + "instant": true, + "format": "table" + } + ], + "transformations": [ + { + "id": "merge", + "options": {} + }, + { + "id": "organize", + "options": { + "excludeByName": { + "Time": true, + "__name__": true, + "job": true, + "instance": true, + "diode": true + }, + "renameByName": { + "Value #A": "up", + "Value #B": "restarts" + } + } + } + ], + "fieldConfig": { + "defaults": {}, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "up" + }, + "properties": [ + { + "id": "mappings", + "value": [ + { + "type": "value", + "options": { + "0": { + "text": "restarting", + "color": "red" + }, + "1": { + "text": "running", + "color": "green" + } + } + } + ] + }, + { + "id": "custom.cellOptions", + "value": { + "type": "color-background" + } + } + ] + } + ] + } + }, + { + "id": 12, + "type": "timeseries", + "title": "Events (rate, per minute)", + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "gridPos": { + "x": 12, + "y": 21, + "w": 12, + "h": 8 + }, + "targets": [ + { + "refId": "A", + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "expr": "sum by (side, module, event) (rate(dyode_events_total{job=\"$job\", instance=~\"$instance\"}[5m])) * 60", + "legendFormat": "{{side}} {{module}} {{event}}" + } + ], + "fieldConfig": { + "defaults": { + "unit": "short" + }, + "overrides": [] + } + }, + { + "id": 13, + "type": "bargauge", + "title": "Free space", + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "gridPos": { + "x": 0, + "y": 29, + "w": 24, + "h": 6 + }, + "targets": [ + { + "refId": "A", + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "expr": "dyode_free_bytes{job=\"$job\", instance=~\"$instance\"}", + "legendFormat": "{{side}} {{module}} {{dir}}" + } + ], + "fieldConfig": { + "defaults": { + "unit": "bytes", + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "red", + "value": null + }, + { + "color": "orange", + "value": 5000000000.0 + }, + { + "color": "green", + "value": 20000000000.0 + } + ] + } + }, + "overrides": [] + }, + "options": { + "orientation": "horizontal", + "displayMode": "gradient", + "reduceOptions": { + "calcs": [ + "lastNotNull" + ] + } + } + }, + { + "id": 14, + "type": "row", + "title": "API", + "collapsed": false, + "gridPos": { + "x": 0, + "y": 35, + "w": 24, + "h": 1 + }, + "panels": [] + }, + { + "id": 15, + "type": "timeseries", + "title": "Refused API requests (per minute)", + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "gridPos": { + "x": 0, + "y": 36, + "w": 24, + "h": 6 + }, + "targets": [ + { + "refId": "A", + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "expr": "sum by (side) (rate(dyode_counter_total{counter=\"api_refused\", job=\"$job\", instance=~\"$instance\"}[5m])) * 60", + "legendFormat": "{{side}}" + } + ], + "fieldConfig": { + "defaults": { + "unit": "short" + }, + "overrides": [] + } + } + ] +} \ No newline at end of file diff --git a/prometheus/rules/dyode.rules.yml b/prometheus/rules/dyode.rules.yml new file mode 100644 index 0000000..6120b61 --- /dev/null +++ b/prometheus/rules/dyode.rules.yml @@ -0,0 +1,109 @@ +# Alert rules for DYODE 2.1.0+ (nc-intezra/dyode_rust, docs/API.md). +# Scrape the OUTPUT box's /metrics: it reports both boxes (side="in" from +# the status the input box sends through the diode). Every series carries +# side="in" or side="out"; module names are the ones in config.yaml. +groups: + - name: dyode + rules: + # ---- the link ----------------------------------------------------- + - alert: DyodeOutputBoxDown + expr: up{job="dyode"} == 0 + for: 2m + labels: {severity: critical} + annotations: + summary: "DYODE output box {{ $labels.instance }} is not answering" + description: "Prometheus cannot scrape the output box's API: DYODE stopped, the box is down, or the token/certificate changed." + + - alert: DyodeInputSilent + expr: dyode_status_age_seconds{side="in"} > 30 or dyode_status_received{side="in"} == 0 + for: 1m + labels: {severity: critical} + annotations: + summary: "No status from the DYODE input box (seen from {{ $labels.instance }})" + description: "The input box's status message has not arrived for over 30 s. The input box, its diode NIC, the diode, or the output box's receiver has failed; nothing is crossing." + + - alert: DyodeConfigMismatch + expr: dyode_config_match == 0 + for: 5m + labels: {severity: critical} + annotations: + summary: "The two DYODE boxes run different configurations" + description: "Their transport hashes differ: lidi parameters or modules are not the same on both boxes, so some traffic is refused. Copy the same config.yaml to both." + + - alert: DyodeTransfersAborted + expr: increase(dyode_counter_total{side="out",counter="lidi_transfers_aborted"}[15m]) > 0 + labels: {severity: warning} + annotations: + summary: "lidi aborted {{ $value | humanize }} transfer(s) in 15 min" + description: "Blocks were lost in the diode (lossy link, receiver too slow, or max_mbps too high). Lost files are named in file_missing events; re-send them from .dyode_sent on the input box." + + - alert: DyodeStaticArpMissing + expr: dyode_static_arp_ok == 0 + for: 2m + labels: {severity: critical} + annotations: + summary: "The input box's static ARP entry is missing" + description: "Without it the input box's own kernel drops everything sent to the output box." + + # ---- modules ------------------------------------------------------- + - alert: DyodeModuleDown + expr: dyode_module_up == 0 + for: 5m + labels: {severity: warning} + annotations: + summary: "DYODE module {{ $labels.module }} ({{ $labels.side }} box) keeps restarting" + + - alert: DyodeModuleFlapping + expr: increase(dyode_module_restarts_total[30m]) > 3 + labels: {severity: warning} + annotations: + summary: "DYODE module {{ $labels.module }} ({{ $labels.side }} box) restarted {{ $value | humanize }} times in 30 min" + + # ---- files --------------------------------------------------------- + - alert: DyodeFilesRejected + expr: increase(dyode_events_total{side="out",event="file_rejected"}[15m]) > 0 + labels: {severity: warning} + annotations: + summary: "{{ $value | humanize }} file(s) rejected by the output box ({{ $labels.module }})" + description: "Incomplete, or not signed with the shared key. See file_rejected records in transfer.jsonl on the output box." + + - alert: DyodeFilesMissing + expr: increase(dyode_events_total{side="out",event="file_missing"}[30m]) > 0 + labels: {severity: warning} + annotations: + summary: "{{ $value | humanize }} file(s) of a batch never arrived ({{ $labels.module }})" + description: "Re-send them by copying them back from .dyode_sent on the input box." + + - alert: DyodeSendFailures + expr: increase(dyode_events_total{side="in",event="file_failed"}[15m]) > 0 + labels: {severity: warning} + annotations: + summary: "The input box failed to send {{ $value | humanize }} file(s) ({{ $labels.module }})" + + - alert: DyodeDiskLow + expr: dyode_free_bytes < 5e9 + for: 10m + labels: {severity: warning} + annotations: + summary: "Under 5 GB free in {{ $labels.module }}'s {{ $labels.dir }} folder ({{ $labels.side }} box)" + + # ---- logs ---------------------------------------------------------- + - alert: DyodeLogsStale + expr: increase(dyode_events_total{side="out",event="logs_stale"}[15m]) > 0 + labels: {severity: warning} + annotations: + summary: "Log stream {{ $labels.module }} went quiet (no heartbeat)" + + - alert: DyodeLogsGap + expr: increase(dyode_events_total{side="out",event="logs_gap"}[15m]) > 0 + labels: {severity: info} + annotations: + summary: "Log lines were lost in the diode ({{ $labels.module }})" + + # ---- the API itself -------------------------------------------------- + - alert: DyodeApiRefusals + expr: increase(dyode_counter_total{counter="api_refused"}[10m]) > 20 + labels: {severity: info} + annotations: + summary: "{{ $value | humanize }} refused API requests on the {{ $labels.side }} box in 10 min" + description: "A client with a wrong or revoked token, or someone probing. See api_refused records in transfer.jsonl." diff --git a/prometheus/scrape.yml b/prometheus/scrape.yml new file mode 100644 index 0000000..85bda6a --- /dev/null +++ b/prometheus/scrape.yml @@ -0,0 +1,17 @@ +# Add to prometheus.yml under scrape_configs. The output box's API serves +# both boxes; scrape the input box too only if your Prometheus is on the OT +# side (it cannot reach the OT side from IT, and does not need to). +scrape_configs: + - job_name: dyode + scheme: https + metrics_path: /metrics + scrape_interval: 15s + authorization: + type: Bearer + credentials_file: /etc/prometheus/dyode.token # from: sudo dyode api-token add prometheus + tls_config: + ca_file: /etc/prometheus/dyode-out.crt # the output box's /etc/dyode/api.crt + # server_name: dyode-out # if you scrape by IP and the cert names a host + static_configs: + - targets: ["192.168.50.4:8443"] # api.out.listen + labels: {diode: plant-a} diff --git a/tools/check.py b/tools/check.py new file mode 100644 index 0000000..0d87e7c --- /dev/null +++ b/tools/check.py @@ -0,0 +1,113 @@ +"""Check that everything here parses and only uses what DYODE serves. + + python3 tools/check.py + +- every YAML/JSON file parses; +- every dyode_* metric in the alert rules and the dashboard exists in + DYODE's /metrics (examples/metrics.txt, captured from a real 2.1.0 run, + plus the few families that only appear when known); +- every JSONPath field the Zabbix template reads exists in /v1/status + (examples/status.json); +- the generated files match their generators (tools/gen_*.py). +""" +import json, re, subprocess, sys, tempfile +from pathlib import Path + +ROOT = Path(__file__).resolve().parent.parent +fail = [] + +def check(ok, msg): + if not ok: + fail.append(msg) + +try: + import yaml +except ImportError: + sys.exit("needs PyYAML (pip install pyyaml)") + +rules = yaml.safe_load((ROOT / "prometheus/rules/dyode.rules.yml").read_text()) +yaml.safe_load((ROOT / "prometheus/scrape.yml").read_text()) +dash = json.loads((ROOT / "grafana/dyode-overview.json").read_text()) +zbx = yaml.safe_load((ROOT / "zabbix/dyode_by_http.yaml").read_text()) +status = json.loads((ROOT / "examples/status.json").read_text()) + +# Families DYODE serves: those in the sample, and those that appear only +# when known (docs/API.md). +served = set(re.findall(r"^# HELP (dyode_\w+)", (ROOT / "examples/metrics.txt").read_text(), re.M)) +served |= {"dyode_static_arp_ok", "dyode_config_match"} +for need in ["dyode_status_age_seconds", "dyode_status_received", "dyode_module_up", "dyode_events_total"]: + check(need in served, f"examples/metrics.txt lacks {need}") + +def metrics_in(expr): + return set(re.findall(r"\b(dyode_\w+)", expr)) + +exprs = [r["expr"] for g in rules["groups"] for r in g["rules"]] +check(len(exprs) >= 10, "too few alert rules") +for r in (r for g in rules["groups"] for r in g["rules"]): + check({"alert", "expr", "labels", "annotations"} <= r.keys(), f"rule {r.get('alert')} incomplete") + check(r["labels"].get("severity") in {"critical", "warning", "info"}, f"rule {r['alert']}: severity") + check(r["expr"].count("(") == r["expr"].count(")"), f"rule {r['alert']}: parentheses") + check(r["expr"].count("{") == r["expr"].count("}"), f"rule {r['alert']}: braces") +for p in dash["panels"]: + for t in p.get("targets", []): + exprs.append(t["expr"]) +for e in exprs: + for m in metrics_in(e): + check(m in served, f"{m} is not served by DYODE (in: {e})") + +# Zabbix: the JSONPaths must lead somewhere in /v1/status. +tmpl = zbx["zabbix_export"]["templates"][0] +check(zbx["zabbix_export"]["version"] == "7.0", "Zabbix export version") +uuids = [] +def walk(o): + if isinstance(o, dict): + if "uuid" in o: + uuids.append(o["uuid"]) + for v in o.values(): + walk(v) + elif isinstance(o, list): + for v in o: + walk(v) +walk(zbx) +check(len(uuids) == len(set(uuids)), "duplicate UUIDs in the Zabbix template") +check(all(re.fullmatch(r"[0-9a-f]{32}", u) for u in uuids), "UUIDs must be 32 hex digits") + +def resolve(path, doc): + """Follow a simple $.a.b.c path (filters stop the walk).""" + cur = doc + for part in path.lstrip("$.").split("."): + if "[" in part or part.endswith("()"): + return True + if not isinstance(cur, dict) or part not in cur: + return False + cur = cur[part] + return True + +keys = {i["key"] for i in tmpl["items"]} +for i in tmpl["items"]: + for step in i.get("preprocessing", []): + if step["type"] == "JSONPATH": + p = step["parameters"][0] + has_default = "error_handler" in step + check(resolve(p, status) or has_default, f"item {i['key']}: {p} not in /v1/status") + if i["type"] == "DEPENDENT": + check(i["master_item"]["key"] in keys, f"item {i['key']}: master missing") +for rule in tmpl["discovery_rules"]: + p = rule["preprocessing"][0]["parameters"][0] + check(resolve(p, status), f"discovery {rule['key']}: {p} not in /v1/status") + for proto in rule["item_prototypes"]: + field = re.search(r"\]\.(\w+)", proto["preprocessing"][0]["parameters"][0]) + mod = (status["boxes"].get("out") or {}).get("modules", [{}])[0] + if field and field.group(1) != "events": + check(field.group(1) in mod, f"prototype {proto['key']}: no field {field.group(1)}") + +# Generated files are up to date. +for gen, out in [("gen_grafana.py", "grafana/dyode-overview.json"), ("gen_zabbix.py", "zabbix/dyode_by_http.yaml")]: + with tempfile.NamedTemporaryFile(suffix=Path(out).suffix) as tmp: + subprocess.run([sys.executable, str(ROOT / "tools" / gen), tmp.name], check=True, capture_output=True) + check(Path(tmp.name).read_text() == (ROOT / out).read_text(), f"{out} is stale: run tools/{gen} {out}") + +if fail: + print("\n".join("FAIL: " + f for f in fail)) + sys.exit(1) +print(f"ok: {len(exprs)} expressions, {len(uuids)} Zabbix objects, {len(served)} metric families") diff --git a/tools/gen_grafana.py b/tools/gen_grafana.py new file mode 100644 index 0000000..9cf27a7 --- /dev/null +++ b/tools/gen_grafana.py @@ -0,0 +1,103 @@ +"""Generate grafana/dyode-overview.json (Grafana 10/11, schemaVersion 39).""" +import json, sys +DS = {"type": "prometheus", "uid": "${datasource}"} +panels = [] +pid = [0] +def P(kind, title, x, y, w, h, targets, **extra): + pid[0] += 1 + p = {"id": pid[0], "type": kind, "title": title, "datasource": DS, + "gridPos": {"x": x, "y": y, "w": w, "h": h}, + "targets": [dict(refId=chr(65 + i), datasource=DS, expr=e, legendFormat=l, **({"instant": True, "format": "table"} if kind == "table" else {})) + for i, (e, l) in enumerate(targets)]} + p.update(extra) + panels.append(p) +def row(title, y): + pid[0] += 1 + panels.append({"id": pid[0], "type": "row", "title": title, "collapsed": False, + "gridPos": {"x": 0, "y": y, "w": 24, "h": 1}, "panels": []}) +sel = 'job="$job", instance=~"$instance"' +thr = lambda steps: {"mode": "absolute", "steps": steps} +g, r, o = "green", "red", "orange" + +row("Link", 0) +P("stat", "Input box status age", 0, 1, 6, 4, + [(f'dyode_status_age_seconds{{side="in", {sel}}}', "{{instance}}")], + description="Seconds since the input box's status arrived through the diode. Over 30 s: input box, diode or link down.", + fieldConfig={"defaults": {"unit": "s", "decimals": 0, "thresholds": thr([{"color": g, "value": None}, {"color": o, "value": 15}, {"color": r, "value": 30}])}, "overrides": []}, + options={"colorMode": "background", "graphMode": "area", "reduceOptions": {"calcs": ["lastNotNull"]}}) +P("stat", "Configurations match", 6, 1, 6, 4, + [(f'dyode_config_match{{{sel}}}', "{{instance}}")], + description="1 when both boxes run the same transport configuration (lidi parameters and modules).", + fieldConfig={"defaults": {"mappings": [{"type": "value", "options": {"0": {"text": "DIFFERENT", "color": r}, "1": {"text": "same", "color": g}}}], + "thresholds": thr([{"color": r, "value": None}, {"color": g, "value": 1}])}, "overrides": []}, + options={"colorMode": "background", "graphMode": "none", "reduceOptions": {"calcs": ["lastNotNull"]}}) +P("stat", "Transfers aborted (24 h)", 12, 1, 6, 4, + [(f'increase(dyode_counter_total{{side="out", counter="lidi_transfers_aborted", {sel}}}[24h])', "{{instance}}")], + description="lidi transfers aborted because blocks were lost in the diode.", + fieldConfig={"defaults": {"decimals": 0, "thresholds": thr([{"color": g, "value": None}, {"color": o, "value": 1}])}, "overrides": []}, + options={"colorMode": "background", "graphMode": "none", "reduceOptions": {"calcs": ["lastNotNull"]}}) +P("stat", "Versions", 18, 1, 6, 4, + [(f'dyode_info{{{sel}}}', "{{side}}: {{version}}")], + options={"colorMode": "none", "graphMode": "none", "textMode": "name", "reduceOptions": {"calcs": ["lastNotNull"]}}) +P("timeseries", "Input box status age", 0, 5, 24, 6, + [(f'dyode_status_age_seconds{{side="in", {sel}}}', "status age")], + fieldConfig={"defaults": {"unit": "s", "custom": {"thresholdsStyle": {"mode": "line"}}, + "thresholds": thr([{"color": g, "value": None}, {"color": r, "value": 30}])}, "overrides": []}) + +row("Throughput", 11) +P("timeseries", "File bytes per second", 0, 12, 12, 8, + [(f'sum by (side, module) (rate(dyode_bytes_total{{{sel}}}[5m]))', "{{side}} {{module}}")], + fieldConfig={"defaults": {"unit": "Bps"}, "overrides": []}) +P("timeseries", "Files per minute", 12, 12, 12, 8, + [(f'sum by (side, module, event) (rate(dyode_events_total{{event=~"file_sent|file_stored|file_rejected|file_missing|file_failed", {sel}}}[5m])) * 60', "{{side}} {{module}} {{event}}")], + fieldConfig={"defaults": {"unit": "short"}, "overrides": []}) + +row("Modules", 20) +P("table", "Modules", 0, 21, 12, 8, + [(f'dyode_module_up{{{sel}}}', ""), (f'dyode_module_restarts_total{{{sel}}}', "")], + transformations=[{"id": "merge", "options": {}}, + {"id": "organize", "options": {"excludeByName": {"Time": True, "__name__": True, "job": True, "instance": True, "diode": True}, + "renameByName": {"Value #A": "up", "Value #B": "restarts"}}}], + fieldConfig={"defaults": {}, "overrides": [{"matcher": {"id": "byName", "options": "up"}, + "properties": [{"id": "mappings", "value": [{"type": "value", "options": {"0": {"text": "restarting", "color": r}, "1": {"text": "running", "color": g}}}]}, + {"id": "custom.cellOptions", "value": {"type": "color-background"}}]}]}) +P("timeseries", "Events (rate, per minute)", 12, 21, 12, 8, + [(f'sum by (side, module, event) (rate(dyode_events_total{{{sel}}}[5m])) * 60', "{{side}} {{module}} {{event}}")], + fieldConfig={"defaults": {"unit": "short"}, "overrides": []}) +P("bargauge", "Free space", 0, 29, 24, 6, + [(f'dyode_free_bytes{{{sel}}}', "{{side}} {{module}} {{dir}}")], + fieldConfig={"defaults": {"unit": "bytes", "thresholds": thr([{"color": r, "value": None}, {"color": o, "value": 5e9}, {"color": g, "value": 2e10}])}, "overrides": []}, + options={"orientation": "horizontal", "displayMode": "gradient", "reduceOptions": {"calcs": ["lastNotNull"]}}) + +row("API", 35) +P("timeseries", "Refused API requests (per minute)", 0, 36, 24, 6, + [(f'sum by (side) (rate(dyode_counter_total{{counter="api_refused", {sel}}}[5m])) * 60', "{{side}}")], + fieldConfig={"defaults": {"unit": "short"}, "overrides": []}) + +dash = { + + "__requires": [{"type": "grafana", "id": "grafana", "name": "Grafana", "version": "10.0.0"}, + {"type": "datasource", "id": "prometheus", "name": "Prometheus", "version": "1.0.0"}], + "title": "DYODE overview", + "uid": "dyode-overview", + "description": "Both DYODE boxes, from the output box's /metrics (nc-intezra/dyode_monitoring).", + "tags": ["dyode", "diode"], + "timezone": "browser", + "schemaVersion": 39, + "version": 1, + "refresh": "30s", + "time": {"from": "now-6h", "to": "now"}, + "editable": True, + "templating": {"list": [ + {"name": "datasource", "label": "Prometheus", "type": "datasource", "query": "prometheus", "current": {}, "hide": 0}, + {"name": "job", "label": "Job", "type": "query", "datasource": DS, "query": {"query": "label_values(dyode_info, job)", "refId": "job"}, + "definition": "label_values(dyode_info, job)", "refresh": 1, "current": {}, "hide": 0}, + {"name": "instance", "label": "Output box", "type": "query", "datasource": DS, + "query": {"query": 'label_values(dyode_info{job="$job"}, instance)', "refId": "inst"}, + "definition": 'label_values(dyode_info{job="$job"}, instance)', "refresh": 1, "includeAll": True, "multi": True, "current": {}, "hide": 0}, + ]}, + "annotations": {"list": []}, + "panels": panels, +} +json.dump(dash, open(sys.argv[1], "w"), indent=2) +print(len(panels), "panels") diff --git a/tools/gen_zabbix.py b/tools/gen_zabbix.py new file mode 100644 index 0000000..2458150 --- /dev/null +++ b/tools/gen_zabbix.py @@ -0,0 +1,148 @@ +"""Generate zabbix/dyode_by_http.yaml: a Zabbix 7.0 template for the DYODE +output box's API (/v1/status). UUIDs are derived from names, so +regenerating gives the same template.""" +import sys, uuid, yaml + +NS = uuid.UUID("6f1c2a52-6b1e-4c43-9d2b-2d1f0a7c0d11") +def U(name): return uuid.uuid5(NS, name).hex +T = "DYODE by HTTP" + +def dep(name, key, path, value_type="FLOAT", units="", error_value=None, triggers=None, desc="", tags=None, extra_pre=None): + pre = [{"type": "JSONPATH", "parameters": [path]}] + if error_value is not None: + pre[0].update({"error_handler": "CUSTOM_VALUE", "error_handler_params": error_value}) + if extra_pre: pre += extra_pre + it = {"uuid": U(key), "name": name, "type": "DEPENDENT", "key": key, "delay": "0", + "value_type": value_type, "description": desc, "preprocessing": pre, + "master_item": {"key": "dyode.status.get"}, + "tags": tags or [{"tag": "component", "value": "dyode"}]} + if units: it["units"] = units + if value_type in ("CHAR", "TEXT"): it["trends"] = "0" + if triggers: it["triggers"] = triggers + return it + +def trig(name, expr, prio, desc="", recovery=None): + t = {"uuid": U("trig:" + name), "expression": expr, "name": name, "priority": prio, "description": desc, + "tags": [{"tag": "scope", "value": "availability"}]} + if recovery: + t["recovery_mode"] = "RECOVERY_EXPRESSION"; t["recovery_expression"] = recovery + return t + +items = [ + {"uuid": U("dyode.status.get"), "name": "DYODE: Get status", "type": "HTTP_AGENT", "key": "dyode.status.get", + "delay": "{$DYODE.INTERVAL}", "history": "0", "value_type": "TEXT", "trends": "0", + "url": "{$DYODE.URL}/v1/status", "timeout": "10s", + "headers": [{"name": "Authorization", "value": "Bearer {$DYODE.TOKEN}"}], + "verify_peer": "YES", "verify_host": "YES", + "description": "/v1/status of the DYODE output box: both boxes (the input box as last received through the diode) and warnings.", + "tags": [{"tag": "component", "value": "raw"}], + "triggers": [trig("DYODE: output box API not answering", f"nodata(/{T}/dyode.status.get,{{$DYODE.NODATA}})=1", "HIGH", + "No /v1/status for {$DYODE.NODATA}: DYODE stopped, the box is down, or the token or certificate changed.")]}, + dep("DYODE: Input box status age", "dyode.in.status_age", "$.boxes.in.status_age_s", units="s", error_value="-1", + desc="Seconds since the input box's status arrived through the diode; -1 when none has arrived yet.", + triggers=[trig("DYODE: no status from the input box", + f"last(/{T}/dyode.in.status_age)>{{$DYODE.IN.SILENT}} or last(/{T}/dyode.in.status_age)<0", "HIGH", + "The input box, its diode NIC, the diode or the output box's receiver has failed: nothing is crossing.")]), + dep("DYODE: Warnings", "dyode.warnings", "$.warnings", value_type="TEXT", + desc="Warnings from the output box, e.g. a silent input box or differing configurations.", + extra_pre=[{"type": "JAVASCRIPT", "parameters": ["var w = JSON.parse(value); return w.length ? w.join('\\n') : '';"]}]), + dep("DYODE: Output box transport hash", "dyode.out.hash", "$.boxes.out.transport_hash", value_type="CHAR", + extra_pre=[{"type": "DISCARD_UNCHANGED_HEARTBEAT", "parameters": ["1h"]}]), + dep("DYODE: Input box transport hash", "dyode.in.hash", "$.boxes.in.transport_hash", value_type="CHAR", error_value="unknown", + extra_pre=[{"type": "DISCARD_UNCHANGED_HEARTBEAT", "parameters": ["1h"]}], + triggers=[trig("DYODE: the boxes run different configurations", + f"last(/{T}/dyode.in.hash)<>last(/{T}/dyode.out.hash) and last(/{T}/dyode.in.hash)<>\"unknown\"", "HIGH", + "Copy the same config.yaml to both boxes.")]), + dep("DYODE: Output box version", "dyode.out.version", "$.boxes.out.version", value_type="CHAR", + extra_pre=[{"type": "DISCARD_UNCHANGED_HEARTBEAT", "parameters": ["1d"]}]), + dep("DYODE: Output box uptime", "dyode.out.uptime", "$.boxes.out.uptime_s", units="uptime", value_type="UNSIGNED"), + dep("DYODE: Input box uptime", "dyode.in.uptime", "$.boxes.in.uptime_s", units="uptime", value_type="UNSIGNED", error_value="0"), + dep("DYODE: Transfers aborted by lidi", "dyode.out.aborted", "$.boxes.out.counters.lidi_transfers_aborted", value_type="UNSIGNED", + error_value="0", extra_pre=[{"type": "SIMPLE_CHANGE", "parameters": [""]}], + desc="Transfers aborted since the last poll (blocks lost in the diode).", + triggers=[trig("DYODE: lidi aborted transfers", f"sum(/{T}/dyode.out.aborted,15m)>0", "WARNING", + "Lost files are named in file_missing events; re-send them from .dyode_sent on the input box.")]), + dep("DYODE: Input box static ARP entry", "dyode.in.arp", "$.boxes.in.static_arp_ok", value_type="UNSIGNED", error_value="1", + extra_pre=[{"type": "BOOL_TO_DECIMAL", "parameters": [""]}], + triggers=[trig("DYODE: input box static ARP entry missing", f"last(/{T}/dyode.in.arp)=0", "HIGH", + "Without it the input box's kernel drops everything sent to the output box.")]), +] + +def module_rule(side): + label = "input" if side == "in" else "output" + key = f"dyode.modules.{side}" + m = "{#MODULE}" + sel = f"$.boxes.{side}.modules[?(@.name=='{m}')]" + def proto(name, k, path, vt="UNSIGNED", units="", error_value="0", extra=None, triggers=None, desc=""): + pre = [{"type": "JSONPATH", "parameters": [path], "error_handler": "CUSTOM_VALUE", "error_handler_params": error_value}] + if extra: pre += extra + p = {"uuid": U(f"{side}:{k}"), "name": name, "type": "DEPENDENT", "key": k, "delay": "0", "value_type": vt, + "description": desc, "preprocessing": pre, "master_item": {"key": "dyode.status.get"}, + "tags": [{"tag": "component", "value": "module"}, {"tag": "module", "value": m}, {"tag": "side", "value": side}]} + if units: p["units"] = units + if vt in ("CHAR", "TEXT"): p["trends"] = "0" + if triggers: p["trigger_prototypes"] = triggers + return p + def tp(name, expr, prio, desc=""): + return {"uuid": U(f"{side}:tp:{name}"), "expression": expr, "name": name, "priority": prio, "description": desc} + protos = [ + proto(f"DYODE {label} module {m}: state", f"dyode.module.state[{side},{m}]", f"{sel}.state.first()", vt="CHAR", error_value="unknown", + triggers=[tp(f"DYODE {label} module {m} is not running", + f"last(/{T}/dyode.module.state[{side},{m}])=\"restarting\"", "AVERAGE")]), + proto(f"DYODE {label} module {m}: restarts", f"dyode.module.restarts[{side},{m}]", f"{sel}.restarts.first()", + triggers=[tp(f"DYODE {label} module {m} keeps restarting", + f"change(/{T}/dyode.module.restarts[{side},{m}])>0", "WARNING")]), + ] + if side == "out": + protos += [ + proto(f"DYODE output module {m}: files rejected", f"dyode.module.rejected[{m}]", f"{sel}.events.file_rejected.first()", + extra=[{"type": "SIMPLE_CHANGE", "parameters": [""]}], + triggers=[tp(f"DYODE output module {m}: files rejected", f"sum(/{T}/dyode.module.rejected[{m}],15m)>0", "WARNING", + "Incomplete or not signed with the shared key; see transfer.jsonl.")]), + proto(f"DYODE output module {m}: files missing", f"dyode.module.missing[{m}]", f"{sel}.events.file_missing.first()", + extra=[{"type": "SIMPLE_CHANGE", "parameters": [""]}], + triggers=[tp(f"DYODE output module {m}: files never arrived", f"sum(/{T}/dyode.module.missing[{m}],30m)>0", "WARNING", + "Re-send them from .dyode_sent on the input box.")]), + proto(f"DYODE output module {m}: bytes stored per second", f"dyode.module.bytes[{side},{m}]", f"{sel}.bytes_stored.first()", + vt="FLOAT", units="Bps", extra=[{"type": "CHANGE_PER_SECOND", "parameters": [""]}]), + ] + else: + protos += [ + proto(f"DYODE input module {m}: bytes sent per second", f"dyode.module.bytes[{side},{m}]", f"{sel}.bytes_sent.first()", + vt="FLOAT", units="Bps", extra=[{"type": "CHANGE_PER_SECOND", "parameters": [""]}]), + ] + return {"uuid": U(key), "name": f"DYODE: {label} box modules", "type": "DEPENDENT", "key": key, "delay": "0", + "master_item": {"key": "dyode.status.get"}, + "lifetime": "7d", + "preprocessing": [{"type": "JSONPATH", "parameters": [f"$.boxes.{side}.modules"], + "error_handler": "CUSTOM_VALUE", "error_handler_params": "[]"}], + "lld_macro_paths": [{"lld_macro": "{#MODULE}", "path": "$.name"}, {"lld_macro": "{#TYPE}", "path": "$.type"}], + "item_prototypes": protos} + +template = { + "uuid": U("template"), "template": T, "name": T, + "description": "DYODE (nc-intezra/dyode_rust) 2.1.0+, through the OUTPUT box's local API (/v1/status).\n" + "Covers both boxes: the input box from the status it sends through the diode.\n" + "Set {$DYODE.URL} (https://ADDRESS:8443) and {$DYODE.TOKEN} (dyode api-token add zabbix).\n" + "The Zabbix server must trust the output box's certificate (its CA path), or set verify_peer off on the master item.\n" + "Generated by tools/gen_zabbix.py in nc-intezra/dyode_monitoring.", + "vendor": {"name": "nc-intezra", "version": "7.0-1"}, + "groups": [{"name": "Templates/Applications"}], + "items": items, + "discovery_rules": [module_rule("in"), module_rule("out")], + "tags": [{"tag": "class", "value": "software"}, {"tag": "target", "value": "dyode"}], + "macros": [ + {"macro": "{$DYODE.URL}", "value": "https://192.168.50.4:8443", "description": "The output box's api.out.listen, with https://"}, + {"macro": "{$DYODE.TOKEN}", "type": "SECRET_TEXT", "description": "A read token from: sudo dyode api-token add zabbix"}, + {"macro": "{$DYODE.INTERVAL}", "value": "30s", "description": "How often /v1/status is read"}, + {"macro": "{$DYODE.NODATA}", "value": "3m", "description": "No answer for this long: API down"}, + {"macro": "{$DYODE.IN.SILENT}", "value": "30", "description": "Seconds without input box status before alerting"}, + ], +} +export = {"zabbix_export": {"version": "7.0", + "template_groups": [{"uuid": U("group"), "name": "Templates/Applications"}], + "templates": [template]}} +class D(yaml.SafeDumper): pass +D.ignore_aliases = lambda *a: True +yaml.dump(export, open(sys.argv[1], "w"), Dumper=D, sort_keys=False, allow_unicode=True, width=1000) +print("items", len(items)) diff --git a/zabbix/dyode_by_http.yaml b/zabbix/dyode_by_http.yaml new file mode 100644 index 0000000..fe7f54b --- /dev/null +++ b/zabbix/dyode_by_http.yaml @@ -0,0 +1,557 @@ +zabbix_export: + version: '7.0' + template_groups: + - uuid: 7ceb6b5d88fb5d7da8c4307f1bf8d652 + name: Templates/Applications + templates: + - uuid: c27d13342c9052bba9f0de9ec0de6f50 + template: DYODE by HTTP + name: DYODE by HTTP + description: 'DYODE (nc-intezra/dyode_rust) 2.1.0+, through the OUTPUT box''s local API (/v1/status). + + Covers both boxes: the input box from the status it sends through the diode. + + Set {$DYODE.URL} (https://ADDRESS:8443) and {$DYODE.TOKEN} (dyode api-token add zabbix). + + The Zabbix server must trust the output box''s certificate (its CA path), or set verify_peer off on the master item. + + Generated by tools/gen_zabbix.py in nc-intezra/dyode_monitoring.' + vendor: + name: nc-intezra + version: 7.0-1 + groups: + - name: Templates/Applications + items: + - uuid: 4459ef3ec77c5b9cb66ad84539698d58 + name: 'DYODE: Get status' + type: HTTP_AGENT + key: dyode.status.get + delay: '{$DYODE.INTERVAL}' + history: '0' + value_type: TEXT + trends: '0' + url: '{$DYODE.URL}/v1/status' + timeout: 10s + headers: + - name: Authorization + value: Bearer {$DYODE.TOKEN} + verify_peer: 'YES' + verify_host: 'YES' + description: '/v1/status of the DYODE output box: both boxes (the input box as last received through the diode) and warnings.' + tags: + - tag: component + value: raw + triggers: + - uuid: f4bd8772b80e5c3594bd77a6f6a4e95f + expression: nodata(/DYODE by HTTP/dyode.status.get,{$DYODE.NODATA})=1 + name: 'DYODE: output box API not answering' + priority: HIGH + description: 'No /v1/status for {$DYODE.NODATA}: DYODE stopped, the box is down, or the token or certificate changed.' + tags: + - tag: scope + value: availability + - uuid: dbbd06bed43d5ef387d00e4782c40b8f + name: 'DYODE: Input box status age' + type: DEPENDENT + key: dyode.in.status_age + delay: '0' + value_type: FLOAT + description: Seconds since the input box's status arrived through the diode; -1 when none has arrived yet. + preprocessing: + - type: JSONPATH + parameters: + - $.boxes.in.status_age_s + error_handler: CUSTOM_VALUE + error_handler_params: '-1' + master_item: + key: dyode.status.get + tags: + - tag: component + value: dyode + units: s + triggers: + - uuid: 8fc0998f5f6d583390cd3fa08862e14d + expression: last(/DYODE by HTTP/dyode.in.status_age)>{$DYODE.IN.SILENT} or last(/DYODE by HTTP/dyode.in.status_age)<0 + name: 'DYODE: no status from the input box' + priority: HIGH + description: 'The input box, its diode NIC, the diode or the output box''s receiver has failed: nothing is crossing.' + tags: + - tag: scope + value: availability + - uuid: 221d5dab285d5c84aaf1bce7454a0b4d + name: 'DYODE: Warnings' + type: DEPENDENT + key: dyode.warnings + delay: '0' + value_type: TEXT + description: Warnings from the output box, e.g. a silent input box or differing configurations. + preprocessing: + - type: JSONPATH + parameters: + - $.warnings + - type: JAVASCRIPT + parameters: + - 'var w = JSON.parse(value); return w.length ? w.join(''\n'') : '''';' + master_item: + key: dyode.status.get + tags: + - tag: component + value: dyode + trends: '0' + - uuid: d61be493bbbf5205afe148ba0d0ffa27 + name: 'DYODE: Output box transport hash' + type: DEPENDENT + key: dyode.out.hash + delay: '0' + value_type: CHAR + description: '' + preprocessing: + - type: JSONPATH + parameters: + - $.boxes.out.transport_hash + - type: DISCARD_UNCHANGED_HEARTBEAT + parameters: + - 1h + master_item: + key: dyode.status.get + tags: + - tag: component + value: dyode + trends: '0' + - uuid: 9e9289127e9d55659f17a82500c9c4e9 + name: 'DYODE: Input box transport hash' + type: DEPENDENT + key: dyode.in.hash + delay: '0' + value_type: CHAR + description: '' + preprocessing: + - type: JSONPATH + parameters: + - $.boxes.in.transport_hash + error_handler: CUSTOM_VALUE + error_handler_params: unknown + - type: DISCARD_UNCHANGED_HEARTBEAT + parameters: + - 1h + master_item: + key: dyode.status.get + tags: + - tag: component + value: dyode + trends: '0' + triggers: + - uuid: ff370714f6565cd78f72bb7d2c9b9ae2 + expression: last(/DYODE by HTTP/dyode.in.hash)<>last(/DYODE by HTTP/dyode.out.hash) and last(/DYODE by HTTP/dyode.in.hash)<>"unknown" + name: 'DYODE: the boxes run different configurations' + priority: HIGH + description: Copy the same config.yaml to both boxes. + tags: + - tag: scope + value: availability + - uuid: 50cf74a2ef4c5291a4be5f1dd7e54eab + name: 'DYODE: Output box version' + type: DEPENDENT + key: dyode.out.version + delay: '0' + value_type: CHAR + description: '' + preprocessing: + - type: JSONPATH + parameters: + - $.boxes.out.version + - type: DISCARD_UNCHANGED_HEARTBEAT + parameters: + - 1d + master_item: + key: dyode.status.get + tags: + - tag: component + value: dyode + trends: '0' + - uuid: da5f477ca8b45cde8d5051e5a3e014c6 + name: 'DYODE: Output box uptime' + type: DEPENDENT + key: dyode.out.uptime + delay: '0' + value_type: UNSIGNED + description: '' + preprocessing: + - type: JSONPATH + parameters: + - $.boxes.out.uptime_s + master_item: + key: dyode.status.get + tags: + - tag: component + value: dyode + units: uptime + - uuid: 5e5db8ed2ca351649fea51f2fda94031 + name: 'DYODE: Input box uptime' + type: DEPENDENT + key: dyode.in.uptime + delay: '0' + value_type: UNSIGNED + description: '' + preprocessing: + - type: JSONPATH + parameters: + - $.boxes.in.uptime_s + error_handler: CUSTOM_VALUE + error_handler_params: '0' + master_item: + key: dyode.status.get + tags: + - tag: component + value: dyode + units: uptime + - uuid: a64b190616315cf095726fa024b45ede + name: 'DYODE: Transfers aborted by lidi' + type: DEPENDENT + key: dyode.out.aborted + delay: '0' + value_type: UNSIGNED + description: Transfers aborted since the last poll (blocks lost in the diode). + preprocessing: + - type: JSONPATH + parameters: + - $.boxes.out.counters.lidi_transfers_aborted + error_handler: CUSTOM_VALUE + error_handler_params: '0' + - type: SIMPLE_CHANGE + parameters: + - '' + master_item: + key: dyode.status.get + tags: + - tag: component + value: dyode + triggers: + - uuid: aafa5fe65d9a543183b298738f47c2df + expression: sum(/DYODE by HTTP/dyode.out.aborted,15m)>0 + name: 'DYODE: lidi aborted transfers' + priority: WARNING + description: Lost files are named in file_missing events; re-send them from .dyode_sent on the input box. + tags: + - tag: scope + value: availability + - uuid: c609d001b2f258cdb99fa4125388e0ae + name: 'DYODE: Input box static ARP entry' + type: DEPENDENT + key: dyode.in.arp + delay: '0' + value_type: UNSIGNED + description: '' + preprocessing: + - type: JSONPATH + parameters: + - $.boxes.in.static_arp_ok + error_handler: CUSTOM_VALUE + error_handler_params: '1' + - type: BOOL_TO_DECIMAL + parameters: + - '' + master_item: + key: dyode.status.get + tags: + - tag: component + value: dyode + triggers: + - uuid: eb17efaf20ae5dfeb7799dae66641372 + expression: last(/DYODE by HTTP/dyode.in.arp)=0 + name: 'DYODE: input box static ARP entry missing' + priority: HIGH + description: Without it the input box's kernel drops everything sent to the output box. + tags: + - tag: scope + value: availability + discovery_rules: + - uuid: 2804aa08bfbd597c8d8db4032f23df53 + name: 'DYODE: input box modules' + type: DEPENDENT + key: dyode.modules.in + delay: '0' + master_item: + key: dyode.status.get + lifetime: 7d + preprocessing: + - type: JSONPATH + parameters: + - $.boxes.in.modules + error_handler: CUSTOM_VALUE + error_handler_params: '[]' + lld_macro_paths: + - lld_macro: '{#MODULE}' + path: $.name + - lld_macro: '{#TYPE}' + path: $.type + item_prototypes: + - uuid: 20c6420c8fc35126ba87a1f75881dbd5 + name: 'DYODE input module {#MODULE}: state' + type: DEPENDENT + key: dyode.module.state[in,{#MODULE}] + delay: '0' + value_type: CHAR + description: '' + preprocessing: + - type: JSONPATH + parameters: + - $.boxes.in.modules[?(@.name=='{#MODULE}')].state.first() + error_handler: CUSTOM_VALUE + error_handler_params: unknown + master_item: + key: dyode.status.get + tags: + - tag: component + value: module + - tag: module + value: '{#MODULE}' + - tag: side + value: in + trends: '0' + trigger_prototypes: + - uuid: 77973587597c598d80482bc6028e5485 + expression: last(/DYODE by HTTP/dyode.module.state[in,{#MODULE}])="restarting" + name: DYODE input module {#MODULE} is not running + priority: AVERAGE + description: '' + - uuid: 309e40b042da5821a9297474ad6d8882 + name: 'DYODE input module {#MODULE}: restarts' + type: DEPENDENT + key: dyode.module.restarts[in,{#MODULE}] + delay: '0' + value_type: UNSIGNED + description: '' + preprocessing: + - type: JSONPATH + parameters: + - $.boxes.in.modules[?(@.name=='{#MODULE}')].restarts.first() + error_handler: CUSTOM_VALUE + error_handler_params: '0' + master_item: + key: dyode.status.get + tags: + - tag: component + value: module + - tag: module + value: '{#MODULE}' + - tag: side + value: in + trigger_prototypes: + - uuid: 57303ff19e4e5e99b51e24e8f6b6df6d + expression: change(/DYODE by HTTP/dyode.module.restarts[in,{#MODULE}])>0 + name: DYODE input module {#MODULE} keeps restarting + priority: WARNING + description: '' + - uuid: bc2c5e05cf7a54ddab09eb1a527776c8 + name: 'DYODE input module {#MODULE}: bytes sent per second' + type: DEPENDENT + key: dyode.module.bytes[in,{#MODULE}] + delay: '0' + value_type: FLOAT + description: '' + preprocessing: + - type: JSONPATH + parameters: + - $.boxes.in.modules[?(@.name=='{#MODULE}')].bytes_sent.first() + error_handler: CUSTOM_VALUE + error_handler_params: '0' + - type: CHANGE_PER_SECOND + parameters: + - '' + master_item: + key: dyode.status.get + tags: + - tag: component + value: module + - tag: module + value: '{#MODULE}' + - tag: side + value: in + units: Bps + - uuid: 165583b653465814ab3d37c6aa05ae91 + name: 'DYODE: output box modules' + type: DEPENDENT + key: dyode.modules.out + delay: '0' + master_item: + key: dyode.status.get + lifetime: 7d + preprocessing: + - type: JSONPATH + parameters: + - $.boxes.out.modules + error_handler: CUSTOM_VALUE + error_handler_params: '[]' + lld_macro_paths: + - lld_macro: '{#MODULE}' + path: $.name + - lld_macro: '{#TYPE}' + path: $.type + item_prototypes: + - uuid: 4619ef3065895880af1856528aa43d40 + name: 'DYODE output module {#MODULE}: state' + type: DEPENDENT + key: dyode.module.state[out,{#MODULE}] + delay: '0' + value_type: CHAR + description: '' + preprocessing: + - type: JSONPATH + parameters: + - $.boxes.out.modules[?(@.name=='{#MODULE}')].state.first() + error_handler: CUSTOM_VALUE + error_handler_params: unknown + master_item: + key: dyode.status.get + tags: + - tag: component + value: module + - tag: module + value: '{#MODULE}' + - tag: side + value: out + trends: '0' + trigger_prototypes: + - uuid: e69b298a069b5da3a158db1793abbed7 + expression: last(/DYODE by HTTP/dyode.module.state[out,{#MODULE}])="restarting" + name: DYODE output module {#MODULE} is not running + priority: AVERAGE + description: '' + - uuid: 706383e1cc7c5fadb3caae8b54515e49 + name: 'DYODE output module {#MODULE}: restarts' + type: DEPENDENT + key: dyode.module.restarts[out,{#MODULE}] + delay: '0' + value_type: UNSIGNED + description: '' + preprocessing: + - type: JSONPATH + parameters: + - $.boxes.out.modules[?(@.name=='{#MODULE}')].restarts.first() + error_handler: CUSTOM_VALUE + error_handler_params: '0' + master_item: + key: dyode.status.get + tags: + - tag: component + value: module + - tag: module + value: '{#MODULE}' + - tag: side + value: out + trigger_prototypes: + - uuid: fcf439199e335079b3668298c962204b + expression: change(/DYODE by HTTP/dyode.module.restarts[out,{#MODULE}])>0 + name: DYODE output module {#MODULE} keeps restarting + priority: WARNING + description: '' + - uuid: a8199b8769f859b6bc312cb950b5098d + name: 'DYODE output module {#MODULE}: files rejected' + type: DEPENDENT + key: dyode.module.rejected[{#MODULE}] + delay: '0' + value_type: UNSIGNED + description: '' + preprocessing: + - type: JSONPATH + parameters: + - $.boxes.out.modules[?(@.name=='{#MODULE}')].events.file_rejected.first() + error_handler: CUSTOM_VALUE + error_handler_params: '0' + - type: SIMPLE_CHANGE + parameters: + - '' + master_item: + key: dyode.status.get + tags: + - tag: component + value: module + - tag: module + value: '{#MODULE}' + - tag: side + value: out + trigger_prototypes: + - uuid: e7d722868bde5aac97e3479eda1bfd04 + expression: sum(/DYODE by HTTP/dyode.module.rejected[{#MODULE}],15m)>0 + name: 'DYODE output module {#MODULE}: files rejected' + priority: WARNING + description: Incomplete or not signed with the shared key; see transfer.jsonl. + - uuid: b6dc917787ae53b0b1a1d254b405c77c + name: 'DYODE output module {#MODULE}: files missing' + type: DEPENDENT + key: dyode.module.missing[{#MODULE}] + delay: '0' + value_type: UNSIGNED + description: '' + preprocessing: + - type: JSONPATH + parameters: + - $.boxes.out.modules[?(@.name=='{#MODULE}')].events.file_missing.first() + error_handler: CUSTOM_VALUE + error_handler_params: '0' + - type: SIMPLE_CHANGE + parameters: + - '' + master_item: + key: dyode.status.get + tags: + - tag: component + value: module + - tag: module + value: '{#MODULE}' + - tag: side + value: out + trigger_prototypes: + - uuid: 078a917bdea6558986a95df1b416c3cd + expression: sum(/DYODE by HTTP/dyode.module.missing[{#MODULE}],30m)>0 + name: 'DYODE output module {#MODULE}: files never arrived' + priority: WARNING + description: Re-send them from .dyode_sent on the input box. + - uuid: 7af1c280c5ab50809b14728ca3e4f993 + name: 'DYODE output module {#MODULE}: bytes stored per second' + type: DEPENDENT + key: dyode.module.bytes[out,{#MODULE}] + delay: '0' + value_type: FLOAT + description: '' + preprocessing: + - type: JSONPATH + parameters: + - $.boxes.out.modules[?(@.name=='{#MODULE}')].bytes_stored.first() + error_handler: CUSTOM_VALUE + error_handler_params: '0' + - type: CHANGE_PER_SECOND + parameters: + - '' + master_item: + key: dyode.status.get + tags: + - tag: component + value: module + - tag: module + value: '{#MODULE}' + - tag: side + value: out + units: Bps + tags: + - tag: class + value: software + - tag: target + value: dyode + macros: + - macro: '{$DYODE.URL}' + value: https://192.168.50.4:8443 + description: The output box's api.out.listen, with https:// + - macro: '{$DYODE.TOKEN}' + type: SECRET_TEXT + description: 'A read token from: sudo dyode api-token add zabbix' + - macro: '{$DYODE.INTERVAL}' + value: 30s + description: How often /v1/status is read + - macro: '{$DYODE.NODATA}' + value: 3m + description: 'No answer for this long: API down' + - macro: '{$DYODE.IN.SILENT}' + value: '30' + description: Seconds without input box status before alerting