From 324de605e93100c4df90ea24253a73bcd748123e Mon Sep 17 00:00:00 2001 From: Moshe Shadmon Date: Sat, 8 Aug 2026 20:16:14 -0700 Subject: [PATCH 01/10] mapping policies --- .../02- Mapping Policy.md | 27 ++++++++++++------- .../01- PLC Mapping.md | 2 -- README.md | 1 - 3 files changed, 17 insertions(+), 13 deletions(-) diff --git a/04- Southbound Interfaces/02- Mapping Policy.md b/04- Southbound Interfaces/02- Mapping Policy.md index c9ef16be..d1b1c365 100644 --- a/04- Southbound Interfaces/02- Mapping Policy.md +++ b/04- Southbound Interfaces/02- Mapping Policy.md @@ -12,7 +12,6 @@ layout: page | 2026-08-08 | Moshe Shadmon | moved from blockchain dir + added content | | ---> - # AnyLog Mapping Policies ## Overview @@ -169,23 +168,31 @@ The humidity message without a `timestamp` intentionally demonstrates the `defau ## 4. `id` -`id` identifies the mapping policy. +The `id` attribute identifies the mapping policy. + +When a policy is added to the metadata, AnyLog automatically generates a unique policy ID based on the hash value of the policy. + +Users can explicitly specify an `id` when creating the policy. A user-defined ID provides a meaningful and predictable identifier that can be used to reference the policy. + +For example: ```json "id": "sensor-policy" ``` -The mapping-policy validator requires an ID. - -When the policy is referenced by a human-readable variable from commands such as: +The following example creates a mapping policy with the ID `my_policy` and inserts it into the local metadata: ```text -policy=!policy_id -``` - -it is often useful to assign the policy ID explicitly rather than relying on an automatically generated identifier. + ---- +blockchain insert where policy = !policy and local = true +``` ## 5. `dbms` diff --git a/04- Southbound Interfaces/04- Industrial Connectors/01- PLC Mapping.md b/04- Southbound Interfaces/04- Industrial Connectors/01- PLC Mapping.md index ab4bc33d..bc3251fb 100644 --- a/04- Southbound Interfaces/04- Industrial Connectors/01- PLC Mapping.md +++ b/04- Southbound Interfaces/04- Industrial Connectors/01- PLC Mapping.md @@ -58,7 +58,6 @@ The connection keywords themselves are protocol-specific — for example, Modbus while OPC-UA and EtherNet/IP use `url`; DNP3 adds `master_id`/`outstation_id`, Modbus adds `device_id`. See each protocol's page for its full keyword table. - * View the data that's accessible via the PLC ```anylog @@ -189,7 +188,6 @@ blockchain insert where policy=!policy_id and local=true and master=!ledger_conn > column, where its value comes from: either lifted straight from the reading (`bring`) or taken positionally > from `params` (`params.0`, `params.1`, ...). - 2. Initiate a `run plc client` - Once the policy is published, reference it by ID on `run plc client` instead of (or alongside) an inline `map`: diff --git a/README.md b/README.md index 64c25dfe..3a939379 100644 --- a/README.md +++ b/README.md @@ -147,7 +147,6 @@ Evey file **must** contain a header change log so when reading it one knows when External destination URL ``` - ## Table of Contents From e578e5f39c61eb2669511a3f6cbaa3fb2bb00561 Mon Sep 17 00:00:00 2001 From: Ori Shadmon Date: Tue, 11 Aug 2026 15:10:07 -0700 Subject: [PATCH 02/10] merge from os-dev Signed-off-by: Ori Shadmon --- .../02- Mapping Policy.md | 26 +++++++------------ .../01- PLC Mapping.md | 8 ------ 2 files changed, 9 insertions(+), 25 deletions(-) diff --git a/04- Southbound Interfaces/02- Mapping Policy.md b/04- Southbound Interfaces/02- Mapping Policy.md index d1b1c365..e277bb40 100644 --- a/04- Southbound Interfaces/02- Mapping Policy.md +++ b/04- Southbound Interfaces/02- Mapping Policy.md @@ -168,32 +168,24 @@ The humidity message without a `timestamp` intentionally demonstrates the `defau ## 4. `id` -The `id` attribute identifies the mapping policy. - -When a policy is added to the metadata, AnyLog automatically generates a unique policy ID based on the hash value of the policy. - -Users can explicitly specify an `id` when creating the policy. A user-defined ID provides a meaningful and predictable identifier that can be used to reference the policy. - -For example: +`id` identifies the mapping policy. ```json "id": "sensor-policy" ``` -The following example creates a mapping policy with the ID `my_policy` and inserts it into the local metadata: +The mapping-policy validator requires an ID. -```text - +When the policy is referenced by a human-readable variable from commands such as: -blockchain insert where policy = !policy and local = true +```text +policy=!policy_id ``` +it is often useful to assign the policy ID explicitly rather than relying on an automatically generated identifier. + +--- + ## 5. `dbms` `dbms` identifies the destination logical database. diff --git a/04- Southbound Interfaces/04- Industrial Connectors/01- PLC Mapping.md b/04- Southbound Interfaces/04- Industrial Connectors/01- PLC Mapping.md index 8ca30ffc..5e8eb61c 100644 --- a/04- Southbound Interfaces/04- Industrial Connectors/01- PLC Mapping.md +++ b/04- Southbound Interfaces/04- Industrial Connectors/01- PLC Mapping.md @@ -54,11 +54,7 @@ The connection keywords themselves are protocol-specific — for example, Modbus while OPC-UA and EtherNet/IP use `url`; DNP3 adds `master_id`/`outstation_id`, Modbus adds `device_id`. See each protocol's page for its full keyword table. -<<<<<<< HEAD -* View the data that's accessible via the PLC -======= * View the data that's accessible via the PLC ->>>>>>> origin/os-dev ```anylog get struct where url = opc.tcp://10.0.0.111:53530/OPCUA/SimulationServer @@ -188,11 +184,7 @@ blockchain insert where policy=!policy_id and local=true and master=!ledger_conn > column, where its value comes from: either lifted straight from the reading (`bring`) or taken positionally > from `params` (`params.0`, `params.1`, ...). -<<<<<<< HEAD -2. Initiate a `run plc client` - Once the policy is published, reference it by ID on `run plc client` instead of (or -======= 2. Initiate a `run plc client` - Once the policy is published, reference it by ID on `run plc client` instead of (or ->>>>>>> origin/os-dev alongside) an inline `map`: ```anylog From 777e68b36a1af9989d5b34138e79b7e1ea9a6bc2 Mon Sep 17 00:00:00 2001 From: Ori Shadmon Date: Tue, 11 Aug 2026 16:26:34 -0700 Subject: [PATCH 03/10] tmp Signed-off-by: Ori Shadmon --- 01- Getting Started/01- Introduction.md | 290 --- 01- Getting Started/02- Prerequisite.md | 114 -- 01- Getting Started/03- install.md | 224 --- 02- Installation & Deployment/01- Install.md | 33 - .../02- Virtualization/01- Docker.md | 277 --- .../02- Installing the VM OVA.md | 239 --- .../02-1 Data Persistence.md | 196 -- .../02- Virtualization/03- Kubernetes.md | 11 - .../99- Third-Party Apps.md | 347 ---- .../03 Kubernetes Networking.md | 82 - .../99-05- AnyLog as a Service.md | 289 --- .../02- Virtualization/99-06- Pip Install.md | 87 - .../03- Orchestrators/01- Open Horizon.md | 246 --- ...02- IBM IEAM (Edge Application Manager).md | 17 - .../03- Orchestrators/03- Barbara.md | 17 - .../04- DELL Distributed Private Cloud.md | 17 - .../03- Orchestrators/05- Zededa.md | 11 - .../04- Cloud Support/01- AWS Deployment.md | 67 - 03- Training & Tutorials/01- Training.md | 57 - .../02- Basic Commands.md | 399 ---- 03- Training & Tutorials/03- Query Data.md | 119 -- .../04- deployment-process.md | 229 --- .../05- deployment-scripts.md | 247 --- 03- Training & Tutorials/06- Nodes.md | 295 --- .../01- Southbound Interfaces.md | 88 - .../02- Direct Connectors/01- REST.md | 133 -- .../02- Message Broker.md | 209 --- .../02- Mapping Policy.md | 1660 ----------------- .../03- Direct Connectors/01- REST.md | 133 -- .../02- Message Broker.md | 209 --- .../03- Industrial Connectors/01- Modbus.md | 157 -- .../03- Industrial Connectors/02- OPC-UA.md | 382 ---- .../03- Industrial Connectors/03- EtherIP.md | 193 -- .../03- Industrial Connectors/04- DNP3.md | 317 ---- ...- DNP3 - Deploying Connector via Script.md | 228 --- .../04-1 DNP3/02- DNP3 - Mapping-Policies.md | 144 -- .../03- DNP3 - TLS test certificates.md | 128 -- .../01- PLC Mapping.md | 213 --- .../04- Industrial Connectors/02- Modbus.md | 157 -- .../04- Industrial Connectors/03- OPC-UA.md | 382 ---- ...- DNP3 - Deploying Connector via Script.md | 228 --- .../04-1 DNP3/02- DNP3 - Mapping-Policies.md | 144 -- .../03- DNP3 - TLS test certificates.md | 128 -- .../04- Industrial Connectors/05- DNP3.md | 317 ---- .../04- Monitoring/01- Node Monitoring.md | 253 --- .../04- Monitoring/02- Syslog.md | 300 --- .../05- Monitoring/01- Node Monitoring.md | 253 --- .../05- Monitoring/02- Syslog.md | 300 --- .../05- RPC & Media Streaming/01- gRPC.md | 193 -- .../02- Video Streaming.md | 206 -- .../06- RPC & Media Streaming/01- gRPC.md | 193 -- .../02- Video Streaming.md | 206 -- .../06- Third-Party/01- node-RED.md | 142 -- .../06- Third-Party/02- Telegraf.md | 257 --- .../06- Third-Party/03- EdgeX.md | 572 ------ .../06- Third-Party/04- Kubearmor.md | 210 --- .../07- Data Ingestion.md | 486 ----- .../07- Third-Party/01- node-RED.md | 142 -- .../07- Third-Party/02- Telegraf.md | 258 --- .../07- Third-Party/03- EdgeX.md | 572 ------ .../07- Third-Party/04- Kubearmor.md | 210 --- .../08- Data Ingestion.md | 486 ----- .../01- Northbound Connectors.md | 78 - .../02- Postman Integration.md | 47 - 05- Northbound Connectors/03- Grafana.md | 297 --- .../04- Postgres Connector (Tableau).md | 165 -- .../05- Microsoft (PowerBI).md | 142 -- 05- Northbound Connectors/06- Google.md | 88 - 05- Northbound Connectors/07- Qlik.md | 110 -- .../08- Data Forwarding.md | 119 -- .../01- Networking & Security.md | 100 - .../02- Network Processing.md | 246 --- .../03- Securing the Network.md | 755 -------- 06- Networking & Security/04- Using REST.md | 146 -- .../05- MQTT Message Broker.md | 810 -------- .../05-1 Kafka Message Client.md | 168 -- .../05-2 Connectors To Data Sources.md | 211 --- .../06- Network/01- Intro Overlay Network.md | 259 --- .../06- Network/02- Nebula.md | 214 --- .../02-1 Nebula Certifications.md | 57 - .../06- Network/03- NGINX.md | 193 -- .../01- Authentication.md | 685 ------- .../02- Authentication-policies.md | 699 ------- .../01- Software TPM.md | 327 ---- .../02- TMP Configuration.md | 16 - 07- CLI/01- CLI.md | 266 --- 07- CLI/02- Background Processes.md | 257 --- 07- CLI/02-1 Nodes.md | 278 --- 07- CLI/03- Get & Set.md | 555 ------ 07- CLI/04- SQL.md | 307 --- 07- CLI/04-1 Notification/01- SMTP.md | 120 -- 07- CLI/04-1 Notification/02- REST.md | 179 -- 07- CLI/04-1 Notification/02-1 Webhooks.md | 126 -- 07- CLI/05- JSON Data Transformation.md | 239 --- 07- CLI/06- Test & Node Status.md | 229 --- 07- CLI/07- Monitoring & Notifications.md | 131 -- ... Conditional Execution and Control Flow.md | 232 --- 07- CLI/09- File Commands.md | 377 ---- 08- Blockchain & Metadata/01- Blockchain.md | 54 - .../02- Policy & Metadata.md | 245 --- 08- Blockchain & Metadata/02-1 ANMP Policy.md | 136 -- .../03- Blockchain Commands.md | 335 ---- .../03-1 Blockchain Full Circle.md | 174 -- .../04- Mapping Policy.md | 250 --- .../05- Unitfied Namespace.md | 375 ---- .../05-1 UNS Custom Dynamic Examples.md | 267 --- .../05-2 UNS Custom Examples.md | 167 -- 09- Data Management/01- Data Management.md | 115 -- 09- Data Management/02- Databases.md | 146 -- .../02-1 Databases/01- SQL Storage.md | 247 --- .../02-1 Databases/02- Blob Storage.md | 92 - .../02-1 Databases/03- NoSQL (MongoDB).md | 179 -- .../02-1 Databases/04- Bucket Storage.md | 211 --- .../02-1 Databases/05- MilvusDB.md | 308 --- 09- Data Management/02-2 Data Aggregations.md | 395 ---- 09- Data Management/03- High Availability.md | 396 ---- 09- Data Management/03-1 HA Support.md | 353 ---- 09- Data Management/04- File Processing.md | 164 -- 09- Data Management/06- Query Profiling.md | 240 --- 09- Data Management/07- Managing OLTP Data.md | 197 -- ...ng and Managing a Non-Time-Series Table.md | 279 --- 10- Edge Data Manager/01- EDM.md | 18 - .../99- [deprecated] Remote CLI .md | 258 --- .../01- LLM Dashboard Generation.md | 476 ----- 11- Extended Services/02- mcpAI.md | 394 ---- .../03- Federated Learning.md | 163 -- .../01- Examples & Use Cases.md | 14 - 13- Support & Troubleshooting/01- FAQ.md | 528 ------ .../02- Troubleshooting.md | 338 ---- .../03- MTU Network Issue.md | 60 - .../01- Docker & K8s Commands.md | 88 - .../04- Third-Party Support/02- MinIO.md | 414 ---- .../04- Third-Party Support/03- MilvusDB.md | 349 ---- .../05- Data Generator.md | 202 -- .../01- AnylogEDF Releases Notes.md | 247 --- .../02- AnylogEDF SOURCE-CHANGELOGS.md | 302 --- ...AnylogEDF DEPLOYMENT_SCRIPTS-CHANGELOGS.md | 122 -- .../04- AnylogEDF DOCKER_COMPOSE-CHANGELOG.md | 121 -- ... AnylogEDF Evaluation License Agreement.md | 129 -- .../02- Privacy Policy.md | 77 - .../03- Notice of Open Source Usage.md | 104 -- .../04- AnylogEDF used OPENSOURCE-NOTICE.md | 91 - ...04-1 AnylogEDF used OPENSOURCE-NOTICE.html | 289 --- .../01 Blockchain (internet) Configuration.md | 83 - .../02 Blockchain example.md | 153 -- .../03 Using Ethereum.md | 161 -- .../01- configuration examples.md | 510 ----- .../C- Reference Materials/02- dictionary.md | 229 --- .../C- Reference Materials/03- sql setup.md | 489 ----- .../C- Reference Materials/04- helpers.md | 116 -- .../05-image mapping.md | 541 ------ .../08- streaming conditions.md | 99 - .../Contributing to Documentation.md | 268 --- .../Python APIs.md | 16 - .../registering pi in the anylog network.md | 86 - .../remote_cli.md | 235 --- .../repeatable_queries.md | 103 - .../test suites.md | 205 -- .../test-suite.md | 368 ---- 159 files changed, 37698 deletions(-) delete mode 100644 01- Getting Started/01- Introduction.md delete mode 100644 01- Getting Started/02- Prerequisite.md delete mode 100644 01- Getting Started/03- install.md delete mode 100644 02- Installation & Deployment/01- Install.md delete mode 100644 02- Installation & Deployment/02- Virtualization/01- Docker.md delete mode 100644 02- Installation & Deployment/02- Virtualization/02- Installing the VM OVA.md delete mode 100644 02- Installation & Deployment/02- Virtualization/02-1 Data Persistence.md delete mode 100644 02- Installation & Deployment/02- Virtualization/03- Kubernetes.md delete mode 100644 02- Installation & Deployment/02- Virtualization/99- Third-Party Apps.md delete mode 100644 02- Installation & Deployment/02- Virtualization/99-02- Kubernetes/03 Kubernetes Networking.md delete mode 100644 02- Installation & Deployment/02- Virtualization/99-05- AnyLog as a Service.md delete mode 100644 02- Installation & Deployment/02- Virtualization/99-06- Pip Install.md delete mode 100644 02- Installation & Deployment/03- Orchestrators/01- Open Horizon.md delete mode 100644 02- Installation & Deployment/03- Orchestrators/02- IBM IEAM (Edge Application Manager).md delete mode 100644 02- Installation & Deployment/03- Orchestrators/03- Barbara.md delete mode 100644 02- Installation & Deployment/03- Orchestrators/04- DELL Distributed Private Cloud.md delete mode 100644 02- Installation & Deployment/03- Orchestrators/05- Zededa.md delete mode 100644 02- Installation & Deployment/04- Cloud Support/01- AWS Deployment.md delete mode 100644 03- Training & Tutorials/01- Training.md delete mode 100644 03- Training & Tutorials/02- Basic Commands.md delete mode 100644 03- Training & Tutorials/03- Query Data.md delete mode 100644 03- Training & Tutorials/04- deployment-process.md delete mode 100644 03- Training & Tutorials/05- deployment-scripts.md delete mode 100644 03- Training & Tutorials/06- Nodes.md delete mode 100644 04- Southbound Interfaces/01- Southbound Interfaces.md delete mode 100644 04- Southbound Interfaces/02- Direct Connectors/01- REST.md delete mode 100644 04- Southbound Interfaces/02- Direct Connectors/02- Message Broker.md delete mode 100644 04- Southbound Interfaces/02- Mapping Policy.md delete mode 100644 04- Southbound Interfaces/03- Direct Connectors/01- REST.md delete mode 100644 04- Southbound Interfaces/03- Direct Connectors/02- Message Broker.md delete mode 100644 04- Southbound Interfaces/03- Industrial Connectors/01- Modbus.md delete mode 100644 04- Southbound Interfaces/03- Industrial Connectors/02- OPC-UA.md delete mode 100644 04- Southbound Interfaces/03- Industrial Connectors/03- EtherIP.md delete mode 100644 04- Southbound Interfaces/03- Industrial Connectors/04- DNP3.md delete mode 100644 04- Southbound Interfaces/03- Industrial Connectors/04-1 DNP3/01- DNP3 - Deploying Connector via Script.md delete mode 100644 04- Southbound Interfaces/03- Industrial Connectors/04-1 DNP3/02- DNP3 - Mapping-Policies.md delete mode 100644 04- Southbound Interfaces/03- Industrial Connectors/04-1 DNP3/03- DNP3 - TLS test certificates.md delete mode 100644 04- Southbound Interfaces/04- Industrial Connectors/01- PLC Mapping.md delete mode 100644 04- Southbound Interfaces/04- Industrial Connectors/02- Modbus.md delete mode 100644 04- Southbound Interfaces/04- Industrial Connectors/03- OPC-UA.md delete mode 100644 04- Southbound Interfaces/04- Industrial Connectors/04-1 DNP3/01- DNP3 - Deploying Connector via Script.md delete mode 100644 04- Southbound Interfaces/04- Industrial Connectors/04-1 DNP3/02- DNP3 - Mapping-Policies.md delete mode 100644 04- Southbound Interfaces/04- Industrial Connectors/04-1 DNP3/03- DNP3 - TLS test certificates.md delete mode 100644 04- Southbound Interfaces/04- Industrial Connectors/05- DNP3.md delete mode 100644 04- Southbound Interfaces/04- Monitoring/01- Node Monitoring.md delete mode 100644 04- Southbound Interfaces/04- Monitoring/02- Syslog.md delete mode 100644 04- Southbound Interfaces/05- Monitoring/01- Node Monitoring.md delete mode 100644 04- Southbound Interfaces/05- Monitoring/02- Syslog.md delete mode 100644 04- Southbound Interfaces/05- RPC & Media Streaming/01- gRPC.md delete mode 100644 04- Southbound Interfaces/05- RPC & Media Streaming/02- Video Streaming.md delete mode 100644 04- Southbound Interfaces/06- RPC & Media Streaming/01- gRPC.md delete mode 100644 04- Southbound Interfaces/06- RPC & Media Streaming/02- Video Streaming.md delete mode 100644 04- Southbound Interfaces/06- Third-Party/01- node-RED.md delete mode 100644 04- Southbound Interfaces/06- Third-Party/02- Telegraf.md delete mode 100644 04- Southbound Interfaces/06- Third-Party/03- EdgeX.md delete mode 100644 04- Southbound Interfaces/06- Third-Party/04- Kubearmor.md delete mode 100644 04- Southbound Interfaces/07- Data Ingestion.md delete mode 100644 04- Southbound Interfaces/07- Third-Party/01- node-RED.md delete mode 100644 04- Southbound Interfaces/07- Third-Party/02- Telegraf.md delete mode 100644 04- Southbound Interfaces/07- Third-Party/03- EdgeX.md delete mode 100644 04- Southbound Interfaces/07- Third-Party/04- Kubearmor.md delete mode 100644 04- Southbound Interfaces/08- Data Ingestion.md delete mode 100644 05- Northbound Connectors/01- Northbound Connectors.md delete mode 100644 05- Northbound Connectors/02- Postman Integration.md delete mode 100644 05- Northbound Connectors/03- Grafana.md delete mode 100644 05- Northbound Connectors/04- Postgres Connector (Tableau).md delete mode 100644 05- Northbound Connectors/05- Microsoft (PowerBI).md delete mode 100644 05- Northbound Connectors/06- Google.md delete mode 100644 05- Northbound Connectors/07- Qlik.md delete mode 100644 05- Northbound Connectors/08- Data Forwarding.md delete mode 100644 06- Networking & Security/01- Networking & Security.md delete mode 100644 06- Networking & Security/02- Network Processing.md delete mode 100644 06- Networking & Security/03- Securing the Network.md delete mode 100644 06- Networking & Security/04- Using REST.md delete mode 100644 06- Networking & Security/05- MQTT Message Broker.md delete mode 100644 06- Networking & Security/05-1 Kafka Message Client.md delete mode 100644 06- Networking & Security/05-2 Connectors To Data Sources.md delete mode 100644 06- Networking & Security/06- Network/01- Intro Overlay Network.md delete mode 100644 06- Networking & Security/06- Network/02- Nebula.md delete mode 100644 06- Networking & Security/06- Network/02-1 Nebula Certifications.md delete mode 100644 06- Networking & Security/06- Network/03- NGINX.md delete mode 100644 06- Networking & Security/07- Security/01- Built-in Authentication/01- Authentication.md delete mode 100644 06- Networking & Security/07- Security/01- Built-in Authentication/02- Authentication-policies.md delete mode 100644 06- Networking & Security/07- Security/02- Trusted Platform Module (TPM)/01- Software TPM.md delete mode 100644 06- Networking & Security/07- Security/02- Trusted Platform Module (TPM)/02- TMP Configuration.md delete mode 100644 07- CLI/01- CLI.md delete mode 100644 07- CLI/02- Background Processes.md delete mode 100644 07- CLI/02-1 Nodes.md delete mode 100644 07- CLI/03- Get & Set.md delete mode 100644 07- CLI/04- SQL.md delete mode 100644 07- CLI/04-1 Notification/01- SMTP.md delete mode 100644 07- CLI/04-1 Notification/02- REST.md delete mode 100644 07- CLI/04-1 Notification/02-1 Webhooks.md delete mode 100644 07- CLI/05- JSON Data Transformation.md delete mode 100644 07- CLI/06- Test & Node Status.md delete mode 100644 07- CLI/07- Monitoring & Notifications.md delete mode 100644 07- CLI/08- Conditional Execution and Control Flow.md delete mode 100644 07- CLI/09- File Commands.md delete mode 100644 08- Blockchain & Metadata/01- Blockchain.md delete mode 100644 08- Blockchain & Metadata/02- Policy & Metadata.md delete mode 100644 08- Blockchain & Metadata/02-1 ANMP Policy.md delete mode 100644 08- Blockchain & Metadata/03- Blockchain Commands.md delete mode 100644 08- Blockchain & Metadata/03-1 Blockchain Full Circle.md delete mode 100644 08- Blockchain & Metadata/04- Mapping Policy.md delete mode 100644 08- Blockchain & Metadata/05- Unitfied Namespace.md delete mode 100644 08- Blockchain & Metadata/05-1 UNS Custom Dynamic Examples.md delete mode 100644 08- Blockchain & Metadata/05-2 UNS Custom Examples.md delete mode 100644 09- Data Management/01- Data Management.md delete mode 100644 09- Data Management/02- Databases.md delete mode 100644 09- Data Management/02-1 Databases/01- SQL Storage.md delete mode 100644 09- Data Management/02-1 Databases/02- Blob Storage.md delete mode 100644 09- Data Management/02-1 Databases/03- NoSQL (MongoDB).md delete mode 100644 09- Data Management/02-1 Databases/04- Bucket Storage.md delete mode 100644 09- Data Management/02-1 Databases/05- MilvusDB.md delete mode 100644 09- Data Management/02-2 Data Aggregations.md delete mode 100644 09- Data Management/03- High Availability.md delete mode 100644 09- Data Management/03-1 HA Support.md delete mode 100644 09- Data Management/04- File Processing.md delete mode 100644 09- Data Management/06- Query Profiling.md delete mode 100644 09- Data Management/07- Managing OLTP Data.md delete mode 100644 09- Data Management/07-1 Creating and Managing a Non-Time-Series Table.md delete mode 100644 10- Edge Data Manager/01- EDM.md delete mode 100644 10- Edge Data Manager/99- [deprecated] Remote CLI .md delete mode 100644 11- Extended Services/01- LLM Dashboard Generation.md delete mode 100644 11- Extended Services/02- mcpAI.md delete mode 100644 11- Extended Services/03- Federated Learning.md delete mode 100644 12- Examples & Use Cases/01- Examples & Use Cases.md delete mode 100644 13- Support & Troubleshooting/01- FAQ.md delete mode 100644 13- Support & Troubleshooting/02- Troubleshooting.md delete mode 100644 13- Support & Troubleshooting/03- MTU Network Issue.md delete mode 100644 13- Support & Troubleshooting/04- Third-Party Support/01- Docker & K8s Commands.md delete mode 100644 13- Support & Troubleshooting/04- Third-Party Support/02- MinIO.md delete mode 100644 13- Support & Troubleshooting/04- Third-Party Support/03- MilvusDB.md delete mode 100644 13- Support & Troubleshooting/05- Data Generator.md delete mode 100644 14- Releases Notes/01- AnylogEDF Releases Notes.md delete mode 100644 14- Releases Notes/02- AnylogEDF SOURCE-CHANGELOGS.md delete mode 100644 14- Releases Notes/03- AnylogEDF DEPLOYMENT_SCRIPTS-CHANGELOGS.md delete mode 100644 14- Releases Notes/04- AnylogEDF DOCKER_COMPOSE-CHANGELOG.md delete mode 100644 15- Appendices/01- Legal & Licensing/01- AnylogEDF Evaluation License Agreement.md delete mode 100644 15- Appendices/01- Legal & Licensing/02- Privacy Policy.md delete mode 100644 15- Appendices/01- Legal & Licensing/03- Notice of Open Source Usage.md delete mode 100644 15- Appendices/01- Legal & Licensing/04- AnylogEDF used OPENSOURCE-NOTICE.md delete mode 100644 15- Appendices/01- Legal & Licensing/04-1 AnylogEDF used OPENSOURCE-NOTICE.html delete mode 100644 99- INTERNAL & DRAFT sections (NOT publicly visible)/B- Blockchain Integration/01 Blockchain (internet) Configuration.md delete mode 100644 99- INTERNAL & DRAFT sections (NOT publicly visible)/B- Blockchain Integration/02 Blockchain example.md delete mode 100644 99- INTERNAL & DRAFT sections (NOT publicly visible)/B- Blockchain Integration/03 Using Ethereum.md delete mode 100644 99- INTERNAL & DRAFT sections (NOT publicly visible)/C- Reference Materials/01- configuration examples.md delete mode 100644 99- INTERNAL & DRAFT sections (NOT publicly visible)/C- Reference Materials/02- dictionary.md delete mode 100644 99- INTERNAL & DRAFT sections (NOT publicly visible)/C- Reference Materials/03- sql setup.md delete mode 100644 99- INTERNAL & DRAFT sections (NOT publicly visible)/C- Reference Materials/04- helpers.md delete mode 100644 99- INTERNAL & DRAFT sections (NOT publicly visible)/C- Reference Materials/05-image mapping.md delete mode 100644 99- INTERNAL & DRAFT sections (NOT publicly visible)/C- Reference Materials/08- streaming conditions.md delete mode 100644 99- INTERNAL & DRAFT sections (NOT publicly visible)/Contributing to Documentation.md delete mode 100644 99- INTERNAL & DRAFT sections (NOT publicly visible)/Python APIs.md delete mode 100644 99- INTERNAL & DRAFT sections (NOT publicly visible)/registering pi in the anylog network.md delete mode 100644 99- INTERNAL & DRAFT sections (NOT publicly visible)/remote_cli.md delete mode 100644 99- INTERNAL & DRAFT sections (NOT publicly visible)/repeatable_queries.md delete mode 100644 99- INTERNAL & DRAFT sections (NOT publicly visible)/test suites.md delete mode 100644 99- INTERNAL & DRAFT sections (NOT publicly visible)/test-suite.md diff --git a/01- Getting Started/01- Introduction.md b/01- Getting Started/01- Introduction.md deleted file mode 100644 index 5b6084e8..00000000 --- a/01- Getting Started/01- Introduction.md +++ /dev/null @@ -1,290 +0,0 @@ ---- -title: Introduction to AnyLog -description: Introduction to AnyLog and the Edge Data Fabric — what it is, core terminology, node types, and high-level architecture. -layout: page ---- - - -Welcome to AnyLog! This guide introduces the platform's architecture, terminology, node types, and the -lifecycle of data as it moves through the network. - -## What is AnyLog? - -**AnyLog** is a **decentralized** network for managing **IoT and time-series data**: rather than centralizing data in -the cloud, it stays at the edge, close to where it's generated. Across that edge footprint, data is also -**distributed** — spread across many Operator nodes, each holding its own localized data lake — and coordinated -through shared metadata and protocols. - -Every node runs the same AnyLog software, though which services are active differs by node type (see -Node Types below). Queries travel **peer-to-peer**: a query sent by a user goes directly to the -relevant Operator node(s), and results come directly back — there's no central broker or hop in between routing the -traffic. - -The architecture consists of two complementary layers: -* **Physical layer** — the Operator nodes where data actually resides, both as structured tables and as the original - raw files. -* **Virtual layer** — the Edge Data Fabric (EDF) connecting those nodes, providing unified, single-point access to - data that's physically spread across all of them. - -Together, these layers create a cloud-like architecture for distributed edge and IoT data — enabling real-time access -without moving data and without locking organizations into a specific cloud, application, or hardware vendor. - -## What is Edge Data Fabric (EDF)? - -AnyLog is built around keeping data at the edge — in localized data lakes close to where it's generated — while still -letting users query across all of them from a single point. The Edge Data Fabric (EDF) is what makes this possible: the -distributed layer that connects those data lakes together without centralizing the actual data. Rather than moving -operational information to one place before it can be analyzed, EDF shares metadata instead — a lightweight index of -what data exists and where it lives. - -In practical terms, this metadata layer tells query nodes where to find the data they need and tells publisher nodes -where to store it. Every node stays autonomous and keeps ownership of its own data, but because they all share the same -metadata, the whole network behaves like one logical system to anyone querying it. - -## EdgeLake vs AnyLog - -EdgeLake is the **open-source, free** version of AnyLog, -distributed by the Linux Foundation. It provides a managed, zero-maintenance experience — most, but not all, of AnyLog's -functionality — and is ideal for teams that want the benefits of edge computing and decentralized data control without -managing infrastructure. - -**EdgeLake offers:** -* Turnkey node deployment at the edge or in the cloud -* Zero-maintenance operation (automatic updates, monitoring, configuration) -* Scalable pricing — starting at **$1 per device/month** -* Real-time SQL and REST API access from any node -* Built-in dashboards and analytics - -| Feature | EdgeLake | AnyLog | -|--------------------------------------------|--------------------|--------------| -| Cost | Free / Open-Source | Subscription | -| Virtual edge layer | ✅ | ✅ | -| Rule engine | ✅ | ✅ | -| Policy-based data management | ✅ | ✅ | -| Node management | ✅ | ✅ | -| Unified APIs, CLIs, Admin UI | ✅ | ✅ | -| Supported IoT connectors | ✅ | ✅ | -| Blockchain abstraction | ✅ | ✅ | -| MCP Integration | limited | ✅ | -| Aggregations | ❌ | ✅ | -| Automated Unified Namespace (UNS) | ❌ | ✅ | -| Security protocol & High Availability (HA) | ❌ | ✅ | -| Publisher node role | ❌ | ✅ | - -**AnyLog (Enterprise)** includes everything in EdgeLake plus: advanced security and authentication, federated data -aggregation and model training, and real-time support with SLA options. - -## Terminology - -| Term | Definition | -|--------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| **Southbound** | Data flowing *in* from devices and sensors, stored into AnyLog/EDF | -| **Northbound** | Queries and results flowing *out* to applications | -| **Metadata** | Descriptive information about the data and nodes in the network — not the data itself | -| **Blockchain** | The mechanism used to store and distribute policies across the network in a consistent, tamper-resistant way. When a blockchain platform isn't in use, the Metadata Manager's local database serves the same role — both documentation and system refer to this repository as "the blockchain" either way. | -| **Policy** | A JSON-structured record stored in the network's metadata, describing things like node configuration, network connectivity, or cluster membership | -| **Services** | Components of AnyLog/EDF that can be started and stopped independently | -| **Nodes / Agents** | Running AnyLog/EDF instances | -| **Containers** | Docker instances running AnyLog/EDF | - -## Node Types - -AnyLog/EDF uses a single codebase across all node types. Except for Operator and Publisher — which are -mutually exclusive on the same node — any node can run any combination of services simultaneously. - -| Node type | Role | Key characteristic | -|----------------------|----------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| **Metadata Manager** | Hosts the network's metadata | Also called the Master Node. Optional — only needed when not using a blockchain platform. One per network (or HA pair). | -| **Operator** | Stores and serves data | Hosts local databases, answers queries, and receives data from southbound connectors or Publishers. | -| **Publisher** | Routes data to Operators | Receives data from devices or connectors, resolves the target Operator from the metadata layer, and forwards the data. Does not store data locally. Cannot run on the same node as an Operator. AnyLog/EDF-specific — not available in EdgeLake. | -| **Query** | Orchestrates distributed queries | Receives SQL from applications, fans the query out to relevant Operators, and returns aggregated results. Any node can serve as a Query node — it's a role, not a dedicated machine. | - -### Metadata Manager - -Also called the **Master Node**. Hosts the network's metadata: node configurations, policies (including network -connectivity details), which cluster/operator a given dataset lives on, Unified Namespace (UNS) definitions, and -scheduler information — everything about the network *except* the data itself. - -The actual data — both the logical database records and the original raw files (e.g. JSON) as they arrived — lives on -the Operator node(s), not the Metadata Manager. - -- **When to use:** Any deployment that does not use a blockchain platform (Optimism, Ethereum, etc.) needs a Metadata - Manager. Using a blockchain instead is optional but removes the single point of failure. -- **Access:** Must be continuously reachable by all nodes in the network. -- **Location:** Cloud or office machine with stable, consistent connectivity. - -### Operator Node - -The data layer of the network. Operator nodes host the actual databases — SQL or NoSQL — where time-series and event -data is stored and indexed, alongside the original raw data as it arrived. They respond directly to queries fanned out -by Query Nodes. - -- **Access:** Must communicate bi-directionally with the Metadata Manager, Query Nodes, and peer Operators within the - same cluster. -- **Location:** Typically at the edge, close to data sources. HA deployments add a cloud-hosted replica in the same - cluster. - -### Publisher Node - -An optional ingestion router. A Publisher accepts data from multiple sensors or devices, looks up the appropriate -Operator for each dataset using the metadata layer, and forwards it. It never writes data to a local database. - -Use a Publisher when a single ingestion point needs to distribute data across multiple clusters, or when you want to -decouple data sources from storage topology. - -> A node cannot run both Operator and Publisher services. Choose one per node. - -- **Access:** Must be able to reach the target Operator node(s). -- **Location:** At the edge, alongside or near the data sources. - -### Query Node - -Accepts SQL queries from external applications — typically via REST — and coordinates execution across the network. It -uses cluster metadata to identify which Operators hold the relevant data, fans the query out in parallel, collects -partial results, and returns a unified response. - -Any node can serve as a Query Node by enabling the REST service and the query thread pool. A dedicated Query Node is -recommended for production workloads handling high query volumes. - -- **Access:** Must have network access to all Operator nodes it may query. -- **Location:** Same network considerations as the Metadata Manager — cloud or office with reliable connectivity. - -### The Cluster - -A **cluster** is a policy on the blockchain, not a running process. It declares that one or more Operator nodes are -collectively responsible for a specific set of tables. Every table in the network belongs to a cluster. This -membership drives: - -- **Query routing** — the Query Node uses the cluster to find which Operators hold the data for a given table, then - sends the query there. -- **HA replication** — when multiple Operators share a cluster, data written to any one of them is automatically - replicated to the others. - -## High-Level Architecture - -### Data Flow Overview - -``` - [ Sensor / Device ] - │ - ▼ - [ Publisher Node ] (optional — distributes data across operators) - │ - ┌────┴────┐ - ▼ ▼ -[ Operator ] [ Operator ] ←──── [ Metadata Manager / Blockchain ] - ▲ ▲ (metadata sync, dotted lines) - └────┬────┘ - │ - [ Query Node ] - │ - ▼ - [ User Application ] -``` - -**Roles at a glance:** -- The **Metadata Manager** holds metadata for the entire network. Metadata is auto-generated as data arrives (node - policies, table definitions, cluster mappings). -- The **Publisher Node** (optional) accepts raw sensor data and routes it to the correct Operator nodes. -- **Operator Nodes** store the actual data. Together they form a virtual data lake. -- The **Query Node** receives requests from applications, uses metadata from the blockchain to locate the data, and - assembles the final result. - -### Traditional vs. AnyLog/EDF Approach - -**Traditional approach:** Data travels from sensors → edge hardware → cloud before it's accessible to applications. -"Real-time" dashboards often carry a significant hidden delay, and accessing edge data typically requires proprietary -software tightly coupled to specific devices. - -**With AnyLog/EDF:** Each edge data server becomes an Operator node, directly part of the queryable network. Multiple -Operator nodes together form a virtual data lake. Applications connect to a single Query node — not to each data -source individually — and AnyLog/EDF handles locating and retrieving the data using blockchain metadata. This removes -the complexity of managing multiple connections, eliminates the need to know where data physically resides, and -dramatically reduces latency. - -### Application-Facing Architecture - -``` - [ Customer Application ] - │ - ▼ - [ Query Node ] - / | \ - ▼ ▼ ▼ -[Edge ] [Edge ] [Cloud / - Op. I] Op. II] Historical Op.] -``` - -The application connects only to the Query node. AnyLog/EDF routes each request to the appropriate operator(s) -automatically, returning a unified result regardless of how many nodes or locations are involved. - -## The Data Lifecycle - -This is the conceptual version of how data moves through AnyLog, from generation to query. A deeper, protocol-by- -protocol walkthrough belongs under **11- Examples & Use Cases/**, not here. - -``` - [ PLC / Sensor ] - │ (MQTT, DNP3, ...) - ▼ - [ Southbound Connector ] (direct — or via a 3rd-party bridge, e.g. Node-RED) - │ - ▼ - [ Operator Node ] - │ generates table (if new) + publishes metadata/policy - ▼ - [ Local Table ] + [ Raw File ] - │ - ▼ - [ Query Node ] - │ - ▼ - [ User / Application ] -``` - -A PLC or other device/sensor generates data and sends it out using one of several protocols — some natively supported -by AnyLog (e.g. MQTT, DNP3), others not (e.g. BACnet). That data reaches AnyLog either directly, through an existing -southbound connector, or indirectly, through a third-party bridge like Node-RED. - -The receiving Operator node ingests the data, converting it from its original format (e.g. JSON) into a SQL table. If -no table definition exists yet for this data, one is generated automatically as part of ingestion — along with any -metadata/policies other nodes need in order to know this data now lives on this cluster. -The data is then stored — both as structured table rows and as the original raw file — and becomes queryable. -The only real "wait" in this pipeline isn't processing delay — it's the Operator's data buffer, which flushes on a -configurable threshold (default: 100MB or 60 seconds, whichever comes first). - -## High Availability (HA) - -AnyLog's availability model is **horizontal**, not vertical: resilience comes from adding more Operator nodes to a -cluster, not from making any single node more redundant on its own. - -Backing up operators is designed to scale with this architecture. You can add as many Operator nodes as needed, the -system will automatically perform hot backups across the cluster. If the primary Operator receiving live data becomes -unavailable, one or more HA Operator nodes — already holding the replicated historical data — can immediately take -over serving it, providing both scalability and resilience. - -Users can also layer on database-level redundancy for **vertical** scaling or additional replication — for example, -PostgreSQL's built-in backup/replication features, or an orchestration layer like Kubernetes. These are supported at -the user's own discretion: they sit outside AnyLog's own HA guarantees unless described in the relevant chapter(s) -covering that specific integration. \ No newline at end of file diff --git a/01- Getting Started/02- Prerequisite.md b/01- Getting Started/02- Prerequisite.md deleted file mode 100644 index c43f2fe0..00000000 --- a/01- Getting Started/02- Prerequisite.md +++ /dev/null @@ -1,114 +0,0 @@ ---- -title: "Prerequisite and setup considerations" -description: "pre-req for AnyLog" -layout: page ---- - - ---- - -# Prerequisite and setup considerations - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
AnyLog Deployment Requirements
FeatureRequirement
Operating SystemLinux (Ubuntu, RedHat, Alpine, Suse)
Windows
Python3.11 – 3.13
Memory footprint100 MB available for AnyLog deployed without Docker
300 MB available for AnyLog deployed with Docker
DatabasesPostgreSQL installed (optional)
SQLite (default, no need to install)
MongoDB installed (only if blob storage is needed)
CPUIntel, ARM, and AMD are supported.
AnyLog can be deployed on a single CPU machine and up to the largest servers (gateways, Raspberry Pi, and all the way to the largest multi-core machines).
StorageAnyLog supports horizontal scaling — nodes (and storage) are added dynamically as needed, reducing scaling complexity. Requirements are based on expected data volume and retention duration on each node. AnyLog supports automated archival and transfer to larger nodes (if needed).
NetworkRequired: a TCP-based network (local TCP-based networks, over the internet, and combinations are supported)
An overlay network is recommended. Most overlay networks can be used transparently. Nebula is used as the default overlay network.
Static IP and 3 ports open and accessible on each node (either via an Overlay Network or without one).
Cloud IntegrationBuilt-in integration using REST, Pub-Sub, and Kafka.
Deployment optionsExecutable (can be deployed as a background process), Docker, or Kubernetes.
- -**Comments**: -* Databases: - - SQLite recommended for smaller nodes and in-memory data. - - PostgreSQL recommended for larger nodes. - - MongoDB used for blob storage. - - Multiple databases can be deployed and used on the same node. - -* Network: - An Overlay network is recommended for the following reasons: - - Isolate the network for security considerations. - - Manage IP and Ports availability. Without an overlay network, users needs to configure and manage availability - of IP and Ports used. - ---- - -## Compatibility Matrix - -| Component | Status | -|--------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| **Python** | 3.11 - 3.13 | -| **Operating System** | No hard requirement - validated on Ubuntu 22.04, Ubuntu 24.04, Windows, and macOS. | -| **Docker** | No hard version requirement. | -| **Kubernetes** | No hard version requirement. | -| **Database versions** | AnyLog connects via standard pip packages rather than maintaining its own version list: `psycopg2` (PostgreSQL), the `pymongo` package (MongoDB), `boto3` (S3), and Python's built-in `sqlite3` (SQLite). Version support therefore tracks whatever each of those packages supports - not a separate AnyLog-maintained matrix. PostgreSQL 12-16 are explicitly guaranteed to work. | -| **Supported connectors** | 🟡 Not yet listed here - needs a confirmed list (see southbound/northbound connector docs). | - -**Recommendation:** For Docker and Kubernetes deployments, use AnyLog's default docker-compose / Kubernetes deployment process rather than a custom setup - the specific Docker/Kubernetes version in use isn't a concern as long as the default deployment process is followed. \ No newline at end of file diff --git a/01- Getting Started/03- install.md b/01- Getting Started/03- install.md deleted file mode 100644 index 5dc52a4c..00000000 --- a/01- Getting Started/03- install.md +++ /dev/null @@ -1,224 +0,0 @@ ---- -title: Installing & Deploying AnyLog -description: How to install, configure, and deploy a 3-node AnyLog network using Docker. -layout: page ---- - - -The directions below provide a zero-touch* quick deployment that does not persist the data — the goal here is just to get -off the ground and get a feel for AnyLog, not to stand up a production-ready system. - -This document covers deploying AnyLog from a single node all the way to a small network consisting of 1 master / metadata -node, 2 operators, and 1 query node. -We start with a single node running the three major services combined: — metadata -management (blockchain database / ledger table), data storage (sensor data coming into AnyLog), and `query`. The Query service provides access to the federated multinode query of data: used for aggregating results from operator(s) into a unified result for the user. -And then we grow that detup into dedicated/separated master, query, and 2 operators, each running on its own node. - -For a more comprehensive deployment, please visit: - -* Docker — includes directions for a persistent, volume-based deployment -* Virtual Machine (OVA) -* Kubernetes - -*A zero-touch deployment is a deployment where everything is automatically defined by the blockchain and default -values. In general, this means not providing any env variables. However, since this is a single codebase that's able to -deploy different services, the user must include the type of AnyLog agent (`NODE_TYPE`), the connection information to -join the network — IP:Port for the Master / metadata node (`LEDGER_CONN`) — and the activation key for -AnyLog (`LICENSE_KEY`). - -## Prerequisites - -### Machine requirements - -| Component | Requirement | -|----------------------|-----------------------------------------------------------------------| -| **Operating System** | Linux (Debian/Ubuntu, RedHat, Alpine, CentOS, Suse) · macOS · Windows | -| **Memory** | 100 MB (without Docker) · 300 MB (with Docker) | -| **CPU** | Intel, ARM, AMD x64. x86 available on request. | -| **Networking** | TCP-based network (local, internet, or hybrid) | - -Recommended minimum for a dev/demo machine: **2 GB RAM, 50 GB disk**. A cloud VM (AWS, DigitalOcean, Linode) works well. - -### Open ports - -The default ports for a single-machine 3-node deployment: - -| Node | TCP | REST | Broker | -|---|---|---|---| -| Master | 32048 | 32049 | — | -| Operator | 32148 | 32149 | 32150 | -| Query | 32348 | 32349 | — | - -If nodes are on separate machines, confirm these ports are accessible between them before deploying. - -## Part 1 — Quick Install - -The following provides a quick installation of a single AnyLog (docker) instance. - -1. Make sure you have Docker and _make_ installed. -2. Request License and Access key -3. Login to Docker - -```shell -docker login -u anyloguser -``` - -4. Start AnyLog Standalone - an instance of AnyLog that contains Master, Operator and Query as a single agent - -```shell -docker run -it --network host \ - -e NODE_TYPE=master-operator \ - -e LICENSE_KEY={LICENSE_KEY} \ ---name anylog-standalone --rm anylogco/anylog-network:2.0.2606 -``` -Please use `make` functionality (in Install) to deploy a persistent / volume-based AnyLog agent. - -5. At this point a single instance of AnyLog is installed on your system - -### Verify the Standalone Instance - -Confirm the node is up and responding: - -```shell -curl -X GET http://127.0.0.1:32149 -``` - -A response confirms the container is running and the REST service is reachable. - -### Node Architecture - -When a node starts, which services come up depends on the node type and how it's configured. For the Standalone -instance above (Master + Operator + Query combined), starting the node brings up: - -* **Communication services** — TCP and REST are always started. The Message Broker only starts on Operator and - Publisher node types. -* **Logical databases** — which ones a node needs depends on its role: - * A **Master / Metadata** node requires the `blockchain` ledger database. - * An **Operator** node requires a logical database to store the actual data, plus an `almgm` database — the - archive hash info database — that stores a hash of each incoming file to prevent duplicate data and support - validation for HA. Local data hashes are stored in `almgm.tsd_info`; hashes for data replicated in from other - operators are stored per-source in `almgm.tsd_[operator ID]`. - * A **Query** node requires the `system_query` database. -* **`run blockchain sync`** — a background service that keeps the node's local copy of the blockchain/metadata in - sync with the rest of the network. -* **Persistence scheduler** *(Operator-specific)* — manages persistence of partitioned data and the original raw - JSON files as they arrive. -* **Monitoring scheduler** *(Operator-specific, enabled by default)* — provides ongoing health/status insight for - the node. - -All of this — and more (e.g. southbound connector services) — can be configured in more detail using a configuration -file rather than passing everything through `docker run` environment variables directly. - -## Part 2 — From Single Agent to Full Network - -The following provides directions on how to deploy a full network -- 1 master, 2 operator, 1 query. -Feel free to skip steps 1&2 if you already have a license. Feel free to skip adding a master node if you'd -like to extend the network with the existing standalone instance from the previous set of directions. - -1. Make sure you have Docker and _make_ installed. -2. Request License and Access key -3. Login to Docker -- this is required on each machine - -```shell -docker login -u anyloguser -``` - -4. Start Master / Metadata node - -```shell -docker run -it --network host \ - -e NODE_TYPE=master \ - -e LICENSE_KEY={LICENSE_KEY} \ ---name anylog-master --rm anylogco/anylog-network:2.0.2606 -``` - -5. Using the command `get connections` (via REST) the value that'll be used as LEDGER_CONN - -```shell -curl -X GET http://127.0.0.1:32049 \ - -H "command: get connections" \ - -H "User-Agent:AnyLog/1.23" -``` - -If you are using a standalone node from step 1 the URL is `http://127.0.0.1:32149`. - -> **Same machine?** If all nodes are running on the same machine, you can use `host.docker.internal` in place of the -> Master's IP for `LEDGER_CONN` — e.g. `LEDGER_CONN=host.docker.internal:32048` — instead of looking up the actual -> host IP. - -6. Start Operator - -```shell -docker run -it --network host \ - -e NODE_TYPE=operator \ - -e LICENSE_KEY={LICENSE_KEY} \ - -e LEDGER_CONN=[IP:TCP_PORT for Master] \ ---name anylog-operator1 --rm anylogco/anylog-network:2.0.2606 -``` - -7. Wait for Operator1 to come up and start Operator 2 - -```shell -docker run -it --network host \ - -e NODE_TYPE=operator \ - -e LICENSE_KEY={LICENSE_KEY} \ - -e LEDGER_CONN=[IP:TCP_PORT for Master] \ ---name anylog-operator2 --rm anylogco/anylog-network:2.0.2606 -``` - -if the second operator resides on the same machine is operator 1 / standalone node then make sure to update the env variable -`ANYLOG_SERVER_PORT`, `ANYLOG_REST_PORT` and `ANYLOG_BROKER_PORT` - -8. Start Query node - -```shell -docker run -it --network host \ - -e NODE_TYPE=query \ - -e LICENSE_KEY={LICENSE_KEY} \ - -e LEDGER_CONN=[IP:TCP_PORT for Master] \ ---name anylog-query --rm anylogco/anylog-network:2.0.2606 -``` - -### Verify the Network - -Once all four nodes are running, confirm each is up and that the network has formed correctly. Run these against any -node's REST port (see the Open ports table) — for example `http://127.0.0.1:32349` for the Query node. - -**Check status** — confirms the node is up and its REST service is reachable: -```shell -curl -X GET http://[ip]:[port] -``` - -**Check node** — confirms the node itself is correctly configured: -```shell -curl -X GET http://[ip]:[port] -H "command: test node" -H "User-Agent: AnyLog/1.23" -``` - -**Check network** — confirms the node can see its peers (Master, Operators, Query): -```shell -curl -X GET http://[ip]:[port] -H "command: test network" -H "User-Agent: AnyLog/1.23" -``` - -If `test network` doesn't show all expected peers, double-check `LEDGER_CONN` was set correctly on each node and that -the required ports (see Open ports) are reachable between them. \ No newline at end of file diff --git a/02- Installation & Deployment/01- Install.md b/02- Installation & Deployment/01- Install.md deleted file mode 100644 index e5a99290..00000000 --- a/02- Installation & Deployment/01- Install.md +++ /dev/null @@ -1,33 +0,0 @@ ---- -title: Installation Overview -description: How to install AnyLog — choose a deployment method (virtualization, orchestrator, or bare metal), review prerequisites, and understand the general install flow before diving into a specific guide. -layout: page ---- - - -# Installing AnyLog - -The following provides directions on how to install AnyLog — whether through an orchestration tool, via -virtualization, or directly on bare metal. - -Please review the prerequisites before getting started. - -* Docker / Podman -* VirtualBox - OVA -* Kubernetes -* IBM's Open Horizon - -## General Directions - -1. Install prerequisites — e.g. `git`, Docker, VirtualBox, etc. -2. Download the appropriate repo with configuration support. -3. Request a license. -4. Update the configuration files. -5. Deploy the AnyLog agent(s). \ No newline at end of file diff --git a/02- Installation & Deployment/02- Virtualization/01- Docker.md b/02- Installation & Deployment/02- Virtualization/01- Docker.md deleted file mode 100644 index 25997bef..00000000 --- a/02- Installation & Deployment/02- Virtualization/01- Docker.md +++ /dev/null @@ -1,277 +0,0 @@ ---- -title: Full Deployment via Docker Compose -description: Deploying AnyLog nodes using AnyLog's official docker-compose repo — installation, configuration, and how LEDGER_CONN and networking work together. -layout: page ---- - - -> This is the full, configuration-driven deployment path, built around AnyLog's official -> docker-compose repo. For a fast, minimal single-command trial -> instead, see Installing & Deploying AnyLog. - -## 1. Install Docker, Make, and Git - -```shell -# Install Docker on Ubuntu -sudo apt-get -y update -sudo apt-get install -y ca-certificates curl -sudo install -m 0755 -d /etc/apt/keyrings -sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc -sudo chmod a+r /etc/apt/keyrings/docker.asc - -echo \ - "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \ - $(. /etc/os-release && echo "${UBUNTU_CODENAME:-$VERSION_CODENAME}") stable" | \ - sudo tee /etc/apt/sources.list.d/docker.list > /dev/null -sudo apt-get -y update - -sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin make git - -# Grant your user permission to use Docker without sudo -USER=`whoami` -sudo groupadd docker -sudo usermod -aG docker ${USER} -newgrp docker -``` - -`make` is optional — every `make` target is a thin wrapper around `deploy.sh`. If `make` isn't available on your -system (e.g. some ARM/Qualcomm hardware), use `bash deploy.sh` directly; behavior is identical either way. - -## 2. Clone the Repository - -```shell -git clone -b os-dev https://github.com/AnyLog-co/docker-compose -cd docker-compose -``` - -AnyLog is a private image repository — request credentials for both your -Docker login and your license key, then log in: - -```shell -docker login -u anyloguser -p [Docker Login Passkey] -``` - -### Repository layout - -``` -docker-compose/ -├── Makefile # Thin wrapper around deploy.sh -├── deploy.sh # Node lifecycle manager — works without make -├── docker-makefiles/ -│ ├── anylog-generic/ -│ │ └── node_configs.env # Config template — copy and customise per node -│ ├── anylog-master/ -│ ├── anylog-operator/ -│ ├── anylog-publisher/ -│ ├── anylog-query/ -│ ├── anylog-standalone-operator/ -│ ├── anylog-standalone-publisher/ -│ ├── build_docker_compose.sh # Generates docker-compose.yaml from configs -│ ├── prep_configs.sh # Pre-flight config validation -│ └── docker-compose-files/ # Generated compose files land here (not committed) -├── license-generator/ # License validation/acceptance service used by `deploy.sh up` -└── support/ # Grafana, PostgreSQL, MongoDB, Remote-GUI -``` - -Both the short form (`operator`) and the full directory name (`anylog-operator`) are accepted everywhere `ANYLOG_TYPE` -is used — both `make` and `deploy.sh` resolve the alias automatically. - -### Container runtime - -`deploy.sh` auto-detects your environment — no configuration needed. It uses `podman` in place of `docker` if -`podman` is installed, and for compose it prefers `podman-compose`, falling back to `docker-compose`, then -`docker compose`, in that order. All the `make`/`bash deploy.sh` commands in this guide work unchanged regardless -of which of these is actually present on your system. - -## 3. Node Types - -Every node type runs the exact same AnyLog image — what differs is which services are enabled. Quick summary: - -| Node Type | Role | -|---|-------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| **Master** (Metadata / `master`) | Hosts the network's metadata — the Metadata Manager described in Introduction to AnyLog| | | -| **Operator** (`operator`) | Stores data and answers queries | -| **Publisher** (`publisher`) | A non-storage node that distributes data from edge sources across Operator nodes | -| **Query** (`query`) | Coordinates distributed queries across Operators | -| **Generic** (`generic`) | A sandbox with only TCP, REST, and Message Broker configured — no storage, no cluster role. Useful for testing connectivity before committing to a real node type. | - -`standalone-operator` and `standalone-publisher` combine Master with an Operator or Publisher, respectively, on a -single agent. - -## 4. Configure Your Node - -For full control over node behavior, copy the generic template and customize it: - -```shell -cp -r docker-makefiles/anylog-generic docker-makefiles/my-operator -``` - -Edit `docker-makefiles/my-operator/node_configs.env`. At minimum, update: - -| Variable | Purpose | -|---|---| -| `NODE_TYPE` | Node role: `generic`, `master`, `operator`, `query`, `publisher`, `standalone-operator`, `standalone-publisher` | -| `NODE_NAME` | Must be unique per node | -| `COMPANY_NAME` | Owner of the node | -| `ANYLOG_SERVER_PORT`, `ANYLOG_REST_PORT` | Must be unique per machine | -| `LEDGER_CONN` | IP:port of the Master/Metadata Manager node — see below | -| `CLUSTER_NAME` | Unique per Operator, unless HA is enabled | -| `LICENSE_KEY` | Optional to set here — see License Key below | -| `DB_USER` / `DB_PASSWD` | Only if using PostgreSQL | - -You'll also want to decide your networking setup (`NETWORK_TYPE`, `OVERLAY_IP`, etc.) — covered in -Understanding the Network Configuration below. - -### License Key - -Every `up` command validates a license before starting the node: - -1. If `LICENSE_KEY` is already set in `node_configs.env`, deployment proceeds immediately — no prompts. -2. If no key is set, and you don't pass one via `--license-key`, you'll be prompted interactively: the license - agreement is displayed, you provide name/email/project and accept, and the accepted key is written back into - `node_configs.env` automatically so future deployments skip the form. - -```shell -# Pass explicitly -make up ANYLOG_TYPE=my-operator LICENSE_KEY="" -bash deploy.sh up --type my-operator --license-key "" - -# Or just set LICENSE_KEY="" directly in node_configs.env beforehand -``` - -## 5. Start the Node - -```shell -# via make -make up ANYLOG_TYPE=my-operator - -# via deploy.sh (no make required) -bash deploy.sh up --type my-operator -``` - -Check it came up: - -```shell -make logs ANYLOG_TYPE=my-operator -bash deploy.sh logs --type my-operator -``` - -Verify it's actually working: - -```shell -make full-test ANYLOG_TYPE=my-operator -bash deploy.sh full-test --type my-operator -``` - -`full-test` runs `test-status` (confirms the process is running), `test-node` (validates node configuration), and -`test-network` (confirms it can see its peers) in sequence — the same three checks from the Quick Install guide, -just wrapped into a single command. - -To deploy a second node of the same type, copy the folder again and update `NODE_NAME`, ports, and `CLUSTER_NAME`: - -```shell -cp -r docker-makefiles/my-operator docker-makefiles/my-operator2 -# edit node_configs.env in my-operator2 first -make up ANYLOG_TYPE=my-operator2 -``` - -## 6. Understanding LEDGER_CONN - -`LEDGER_CONN` is how every non-Master node (Operator, Query, Publisher) finds the Metadata Manager (Master) node — -it's set to that node's **TCP port**, not its REST port: - -```dotenv -LEDGER_CONN=127.0.0.1:32048 -``` - -This is the single most common source of "my node won't join the network" problems, because the correct value -depends entirely on your Docker networking mode (see below): - -- **Host networking (Linux/WSL, `NETWORK_TYPE` empty or `network`)** — containers share the host's network stack - directly, so `127.0.0.1:` (or the Master's real host IP, if on a different machine) works as-is. -- **Port-mapped mode (Windows/macOS, `NETWORK_TYPE=ports`)** — containers do *not* share the host network. `127.0.0.1` - inside one container does not reach another container. You need the **host machine's LAN IP** here instead — set - via `OVERLAY_IP` on the Master, and reference that same IP in `LEDGER_CONN` on every other node. -- **Custom/overlay network (`NETWORK_TYPE=`)** — nodes reach each other by container name or overlay - IP across physical machines. `LEDGER_CONN` should use the Master's overlay IP (or container name, if on the same - named network), not its host IP. - -If `test network` doesn't show all expected peers after deployment, `LEDGER_CONN` pointing at the wrong address is -the first thing to check. - -## 7. Understanding the Network Configuration - -AnyLog's networking is split across two independent layers — getting one right doesn't automatically fix the other. - -### Layer 1 — Docker Network Topology (`NETWORK_TYPE`) - -Controls how the **container** connects to the host and to other containers. Purely a Docker-level concern; it has -no effect on how AnyLog itself resolves or advertises its IP. - -| Value | Behavior | When to use | -|---|---|---| -| *(empty)* | Auto-detect | Recommended default — host mode on Linux/WSL, port-mapped on Windows/macOS | -| `network` | `network_mode: host` | Linux/WSL only | -| `ports` | Explicit port mapping | Windows, macOS, or Linux when host networking isn't available | -| `` | Pre-created external Docker network | Multi-machine deployments (VPN, Swarm overlay) | - -For a multi-host Swarm overlay, create the network once, then point every node at it: - -```shell -docker network create --scope=swarm --attachable -d overlay anylog-net -``` -```dotenv -NETWORK_TYPE=anylog-net -``` - -### Layer 2 — AnyLog Network Identity - -Controls how AnyLog resolves and **advertises its own IP** — the address it registers in the blockchain/metadata -layer and binds its TCP/REST/Broker ports to. Getting this wrong means other nodes can see your node in the -metadata but can't actually reach it. - -| Variable | Purpose | -|---|---| -| `NIC_TYPE` | Resolve the node's IP from a specific network interface (e.g. `eth0`) — useful on multi-NIC hosts | -| `OVERLAY_IP` | Overrides the resolved IP entirely — required whenever the container's internal IP isn't reachable from outside (port-mapped mode, NAT, Swarm overlay) | -| `TCP_BIND` / `REST_BIND` / `BROKER_BIND` | Whether each protocol binds to the resolved/overlay IP specifically (`true`) or all interfaces (`false`) | - -### Quick reference - -| Scenario | `NETWORK_TYPE` | `OVERLAY_IP` | -|---|---|---| -| Linux, single node | *(empty)* | *(empty)* | -| Windows/macOS, single node | `ports` or *(empty)* | host's LAN IP | -| Multi-node, same LAN (Linux) | *(empty)* | *(empty)* | -| Multi-node, VPN/overlay | `` | overlay/VPN IP | -| Multi-node, cloud/NAT | `ports` or `` | public/elastic IP | - -In short: **Layer 1 decides how the container talks to the network at all; Layer 2 decides what address it tells -other nodes to use.** On a single Linux machine you can usually leave both at their defaults. The moment you're on -Windows/macOS or spanning multiple machines, `OVERLAY_IP` (Layer 2) is almost always the setting you need — and it -has to match whatever `LEDGER_CONN` (§4 above) expects on the other end. \ No newline at end of file diff --git a/02- Installation & Deployment/02- Virtualization/02- Installing the VM OVA.md b/02- Installation & Deployment/02- Virtualization/02- Installing the VM OVA.md deleted file mode 100644 index ce2274a1..00000000 --- a/02- Installation & Deployment/02- Virtualization/02- Installing the VM OVA.md +++ /dev/null @@ -1,239 +0,0 @@ ---- -title: Installing AnyLog via OVA -description: Deploy a pre-configured AnyLog demo environment using the OVA virtual machine image. -layout: page ---- - - -The AnyLog Demo OVA is a pre-configured virtual machine image containing a complete, multi-node AnyLog environment. It is designed for demonstrations, training, evaluations, and proof-of-concept deployments — everything runs inside a single VM. - ---- - -## What's included - -| Component | Description | Default port | -|---|---|---| -| **Standalone Node** | Combined master + operator — control plane, query federation, and data storage | `VM_IP:32149` | -| **Operator Node** | Second data ingestion and storage node | `VM_IP:32159` | -| **GUI** | Web-based management UI — query builder, node health, metadata viewer | `http://localhost:31800` | -| **Grafana** | Pre-configured monitoring dashboard | `http://localhost:3000` | - -All components start automatically when the VM boots. - ---- - -## Prerequisites - -- Hypervisor: VMware Workstation, VMware ESXi, or VirtualBox -- Minimum 4 GB RAM, 2 vCPUs, 20 GB disk recommended -- AnyLog license key — get one at https://www.anylog.network/download - ---- - -## Installation - -### 1. Import the OVA - -Import the `.ova` file into your hypervisor. In VMware: **File → Open** and select the file. In VirtualBox: **File → Import Appliance**. - -Boot the VM and log in. The AnyLog scripts are located at `~/AnyLog/`. - -### 2. Make scripts executable - -```bash -chmod +x ~/AnyLog/ALinstall.sh -chmod +x ~/AnyLog/startup.sh -``` - -### 3. Configure the environment (optional) - -Edit `~/AnyLog/ALinstall.env` before installing. At minimum, set your license key: - -```bash -nano ~/AnyLog/ALinstall.env -``` - -Key variables: - -| Variable | Default | Description | -|---|---|---| -| `LICENSE_KEY` | *(blank)* | **Required.** Prompted interactively if left blank. | -| `COMPANY_NAME` | `Anylog-Demo` | Name stamped on node policies in the metadata layer. | -| `TAG` | `pre-develop` | AnyLog Docker image tag / version to deploy. | -| `LEDGER_CONN` | `127.0.0.1:32148` | TCP address of the master node (blockchain ledger). | -| `ENABLE_MQTT` | `true` | Enable MQTT data ingestion. | -| `MQTT_BROKER` | `172.104.228.251` | External MQTT broker address for the demo data feed. | -| `MSG_DBMS` | `new_company` | Database name for MQTT-ingested data. | -| `DEFAULT_DBMS` | `new_company` | Default database name. | -| `NODE_MONITORING` | `true` | Enable node health monitoring. | -| `STORE_MONITORING` | `true` | Persist monitoring metrics to a local database. | -| `SYSLOG_MONITORING` | `true` | Collect syslog messages from the host. | -| `DOCKER_MONITORING` | `true` | Collect Docker container stats. | - -### 4. Install the demo environment - -```bash -cd ~/AnyLog -./ALinstall.sh -d -s install -``` - -The `-d` flag enables demo mode (installs the full pre-configured environment). The `-s` flag starts all nodes immediately after installation. - -Logs are written to `./logs/ALinstall_install_.log`. - -### 5. Verify - -```bash -docker ps -``` - -All containers should be running. Then open the GUI: - -``` -http://localhost:31800 -``` - -Default credentials: -``` -Username: edgelake -Password: edgelake -``` - -> **Change these immediately for any non-demo or networked deployment.** - ---- - -## ALinstall.sh reference - -```bash -./ALinstall.sh [-e env_file] [-n node1,node2,...] [-s] [-k] [-d] -``` - -### Commands - -| Command | Description | -|---|---| -| `install` | Clone docker-compose repo, configure, and optionally start nodes | -| `uninstall` | Stop and remove containers and images | -| `update` | Uninstall then reinstall with current config | -| `start` | Start previously installed nodes | -| `stop` | Stop running nodes | - -### Flags - -| Flag | Description | -|---|---| -| `-d` | Demo mode — installs the full demo environment (overrides `-n`) | -| `-s` | Auto-start nodes after `install` or `update` | -| `-k` | Auto-stop running nodes before `uninstall` or `update` | -| `-n node1,node2` | Target specific node types only | -| `-e path` | Path to a custom environment file (default: `./ALinstall.env`) | - -### Node types - -| Node type | Description | -|---|---| -| `anylog-standalone-operator` | Combined master + operator (used in demo mode) | -| `anylog-master` | Metadata ledger node only | -| `anylog-operator` | Data ingestion and storage node | -| `anylog-query` | Federated SQL query node | -| `anylog-publisher` | Data routing node | -| `anylog-generic` | Generic node — any combination of services | - -### Common examples - -```bash -# Full demo install, auto-start -./ALinstall.sh -d -s install - -# Install specific node types only -./ALinstall.sh -n anylog-master,anylog-query install - -# Install with a custom config file, auto-start -./ALinstall.sh -e /opt/myconfig.env -s install - -# Stop demo nodes -./ALinstall.sh -d stop - -# Restart demo nodes -./ALinstall.sh -d -s start - -# Uninstall a single node -./ALinstall.sh -n anylog-operator uninstall - -# Update all nodes (stop existing, reinstall, restart) -./ALinstall.sh -k -s update -``` - ---- - -## Default dataset - -The demo environment ships with a preloaded dataset: - -- **Database:** `new_company` -- **Table:** `rand_data` - -Data is continuously ingested via an MQTT feed on first launch. You can query it immediately from the GUI or via REST: - -```bash -curl -X GET http://localhost:32149 \ - -H "command: sql new_company format=table \"select * from rand_data limit 10\"" \ - -H "User-Agent: AnyLog/1.23" -``` - ---- - -## Troubleshooting - -**Containers not running:** -```bash -docker ps -a # check all containers including stopped ones -docker logs # view logs for a specific container -``` - -**Restart the demo:** -```bash -./ALinstall.sh -d -s start -``` - -**Reinstall from scratch:** -```bash -./ALinstall.sh -d uninstall -./ALinstall.sh -d -s install -``` - -**Upgrade to a newer version:** -```bash -# Edit ALinstall.env and update TAG to the desired version, then: -./ALinstall.sh -d -k -s update -``` - -**Logs:** -```bash -ls ~/AnyLog/logs/ # list install logs -cat ~/AnyLog/logs/ALinstall_install_*.log -``` - ---- - -## Next steps - -Once the demo environment is running, explore it via the GUI at `http://localhost:31800`, or connect to the standalone node directly: - -- **REST endpoint:** `http://VM_IP:32149` -- **Query node endpoint:** `http://VM_IP:32349` - -To move beyond the demo and deploy a production-grade environment, see: -- Installing AnyLog via Docker -- Installing AnyLog as a Service -- Deployment Scripts \ No newline at end of file diff --git a/02- Installation & Deployment/02- Virtualization/02-1 Data Persistence.md b/02- Installation & Deployment/02- Virtualization/02-1 Data Persistence.md deleted file mode 100644 index abea93a8..00000000 --- a/02- Installation & Deployment/02- Virtualization/02-1 Data Persistence.md +++ /dev/null @@ -1,196 +0,0 @@ ---- -title: Data Persistence -description: AnyLog image and volumes -layout: page ---- - - - -## Persistent Volumes in AnyLog Deployment -A volume is a directory that sits on the physical machine and is associated with one or more Docker instances. -Kubernetes volumes are more of an abstract concept, since persistent data can live either on the machine or in the -cloud (e.g. AWS S3). - -AnyLog "requires" storing certain content generated throughout usage (locally) in order to have a backup for when a -node (physically) resets, or when migrating data from one machine to another. - -In addition to the built-in directories, the `deployment-scripts` directory is downloaded (via `git clone`) locally and used -to help convert the configurations into actual active services and connected logical databases. - -| Volume | Directory (in container) | Usage | -|:---------------------------------:|:--------------------------------:|:--------------------------------------------------------------------------------------------------------------------------------------------------------:| -| `${CONTAINER_NAME}-anylog` | `/app/AnyLog-Network/anylog/` | used for authentication keys | -| `${CONTAINER_NAME}-blockchain` | `/app/AnyLog-Network/blockchain` | directory that contains a copy of the the blockchain (as JSON) file | -| `${CONTAINER_NAME}-data` | `/app/AnyLog-Network/data` | directory that would contain data coming into the node and blob storage (not in database) and data stored in SQLite database (file). | -| `${CONTAINER_NAME}-local-scripts` | `/app/deployment-scripts/` | A copy of deployment-scripts used to initiate + configure the AnyLog agent | - -### Data Directory - -The following provides a breakdown of the different directories under ${CONTAINER_NAME}-data - -```tree -/var/lib/docker/volumes/anylog-node-data/_data -├── archive <-- Archive/Backup of data hosted on the node -│ └── 22 <-- Year of when data came in -│ └── 06 <-- Month of when data came in -│ ├── 05 <-- Day of when data came in -│ ├── 06 -│ ├── 07 -│ ├── 08 -│ ├── 09 -│ └── 10 -├── bkup <-- Data that has been sent to an operator (on publisher node) -├── dbms <-- directory containing SQLite (non-memory) data -├── distr <-- That coming in from other operator nodes on the same cluster -├── error <-- Data files that filed to get processed -├── pem -├── prep <-- Data being prepared to be stored -├── rest <-- Data coming in via REST -├── test <-- Test case -└── watch <-- Data ready to be stored or sent to other operators -``` - -### Deployment Scripts - -Please visit deployment-scripts for more details. - -## Accessing Volumes - -### Docker - -1. Get list of all your volumes -```shell -docker volume ls -<< COMMENT -DRIVER VOLUME NAME -local anylog-node-anylog -local anylog-node-blockchain -local anylog-node-data -local anylog-node-local-scripts -local postgres_pgdata -<< -``` - -2. Using the `inspect` command get the directory path of the volume -```shell -docker volume inspect anylog-node-local-scripts -<< COMMENT -[ - { - "CreatedAt": "2022-07-04T18:11:50Z", - "Driver": "local", - "Labels": {}, - "Mountpoint": "/var/lib/docker/volumes/anylog-node-local-scripts/_data", - "Name": "anylog-node-local-scripts", - "Options": {}, - "Scope": "local" - } -] -<< -``` - -3. Once you know the _Mountpoint_, you can access the content within that volume. Note - Depending on the permissions, -you may need to do a `sudo` command. -```shell -sudo tree /var/lib/docker/volumes/anylog-node-local-scripts/_data -<< COMMENT -/var/lib/docker/volumes/anylog-node-local-scripts/_data -├── README.md -├── create_dir_structure.sh -├── deployment_clean.sh -├── deployment_scripts -│ ├── configure_dbms_almgm.al -│ ├── configure_dbms_blockchain.al -│ ├── configure_dbms_operator.al -│ ├── configure_dbms_system_query.al -│ ├── data_partitioning.al -│ ├── declare_cluster.al -│ ├── declare_generic_policy.al -│ ├── declare_k8s_generic_policy.al -│ ├── declare_k8s_operator.al -│ ├── declare_operator.al -│ ├── deploy_operator.al -│ ├── deploy_publisher.al -│ ├── local_script.al -│ ├── mqtt.al -│ ├── network_configs.al -│ ├── pre_deployment.al -│ ├── run_scheduler.al -│ ├── set_params.al -│ └── validate_policy.al -├── sample_code -│ ├── edgex.al -│ ├── fledge.al -│ └── fledge_old.al -└── start_node.al -``` - -### Kubernetes - -Kubernetes volumes are provisioned through a `PersistentVolumeClaim` rather than named directly like Docker -volumes, so the discovery path looks slightly different: - -1. List the persistent volume claims to find the one backing your node: -```shell -kubectl get pvc -``` - -2. Get the underlying `PersistentVolume` and its actual storage location — this varies by provisioner (local disk, - NFS, cloud block storage, etc.), so the exact path/endpoint shown will depend on your cluster: -```shell -kubectl describe pv -``` - -3. If your provisioner backs onto a local or NFS path reachable from a node, you can inspect it directly the same - way as the Docker `tree` example above. If it's cloud-backed (e.g. an EBS volume or S3 bucket), use that - provider's own tooling instead — there's no local filesystem path to `tree` into. - -> **To verify:** the exact `kubectl` workflow above depends on how AnyLog's Helm charts provision storage -> (StorageClass, provisioner, etc.) — worth confirming against the actual Kubernetes deployment docs rather than -> treating this as authoritative. - -### Via Executable - -Docker lets you access a volume's files directly from the host filesystem, as shown above. With Kubernetes — and -often with Docker too, for convenience — it's simpler to just exec into the running container instead. - -1. Attach to the executable - -```shell -cd docker-compose -make exec ANYLOG_TYPE=[AnyLog agent type] -``` - -> **To verify:** whether `exec` is actually a valid target on this Makefile. The confirmed-current deployment -> guide ("01- Docker.md") lists `up`, `logs`, and `full-test` as targets here — `exec` (with `SERVICE=`, not -> `ANYLOG_TYPE=`) only appears on the separate `docker-compose/support` Makefile. If this Makefile doesn't -> actually have an `exec` target, the equivalent is likely `bash deploy.sh exec --type [AnyLog agent type]`, -> following the same `make`/`deploy.sh` pattern used throughout "01- Docker.md." - -2. Install vim - by default we do not install vi / vim - -```shell -apt-get -y update && apt-get -y install vim -``` - -3. cd into deployment-scripts - -```shell -cd /app/deployment-scripts -``` - -4. using `vim` update the scripts as needed \ No newline at end of file diff --git a/02- Installation & Deployment/02- Virtualization/03- Kubernetes.md b/02- Installation & Deployment/02- Virtualization/03- Kubernetes.md deleted file mode 100644 index 87c4a727..00000000 --- a/02- Installation & Deployment/02- Virtualization/03- Kubernetes.md +++ /dev/null @@ -1,11 +0,0 @@ - - - - -To Be Added \ No newline at end of file diff --git a/02- Installation & Deployment/02- Virtualization/99- Third-Party Apps.md b/02- Installation & Deployment/02- Virtualization/99- Third-Party Apps.md deleted file mode 100644 index effe7f72..00000000 --- a/02- Installation & Deployment/02- Virtualization/99- Third-Party Apps.md +++ /dev/null @@ -1,347 +0,0 @@ ---- -title: Third Party Apps Install -description: The following document provides support on how to use built-in configs (as part of docker-compose)to - seamlessly install third-party applications. -layout: page ---- - - -> This page documents the services managed under `docker-compose/support`: how they're configured, how the Makefile -> drives their lifecycle, and how to add new ones. - -## Overview - -`support/` holds the tooling to configure, generate, and manage the Docker/Podman containers that run alongside AnyLog: - -| Service | Managed by Makefile? | Purpose | -|---|---|---| -| Remote-GUI | ✅ | Web UI for AnyLog | -| Grafana | ✅ | Dashboards / visualization | -| PostgreSQL | ✅ | Relational datastore | -| MongoDB | ✅ | Document datastore | -| Ollama | ❌ (standalone) | Local LLM runtime for MCP function calling | -| Video Inference Models | ❌ (standalone) | CV/ML inference on edge video streams | -| Nebula Overlay Network | ❌ (standalone) | Encrypted peer-to-peer overlay networking | -| Syslog Forwarding | ✅ (special-cased) | Forwards host syslog traffic to the AnyLog broker port | - -### Requirements - -- Docker ≥ 20.10 **or** Podman ≥ 4.0 -- `docker compose` plugin **or** `docker-compose` / `podman-compose` -- Bash ≥ 4.0 (for `docker_compose_builder.sh`) -- `make` - -The Makefile auto-detects whichever container engine and compose command are installed (`podman`/`podman-compose` preferred, falling back to `docker`/`docker-compose`/`docker compose`). - -### Directory Structure - -``` -support/ -├── Makefile # All lifecycle commands -├── README.md # Project readme -├── Ollama.md # Ollama setup guide -├── Video-Inferences.md # Video inference models guide -├── Nebula.md # Nebula overlay network setup guide -├── docker_compose_builder.sh # Generates docker-compose.yml from configs.yaml -├── syslog.sh # Configures host syslog daemon → AnyLog broker port -├── grafana/ -│ ├── configs.yaml -│ └── docker-compose.yml # generated — do not edit by hand -├── mongodb/ -│ ├── configs.yaml -│ └── docker-compose.yml -├── ollama/ -│ ├── configs.yaml -│ ├── docker-compose.yaml -│ ├── docker-compose-gpu.yaml -│ └── ollama-configs.png -├── postgres/ -│ ├── configs.yaml -│ └── docker-compose.yml -└── remote-gui/ - ├── configs.yaml - └── docker-compose.yml -``` - -Each `docker-compose.yml` is auto-generated from its sibling `configs.yaml`, either explicitly via `make dry-run` or automatically on the first `make up` for that service. **Any subdirectory containing a `configs.yaml` is automatically recognized as a service** — no Makefile edits are needed to add new instances. - ---- - -## Quick Start - -```bash -# 1. (Optional) Pre-generate all docker-compose files -make dry-run - -# 2. Start all default services (generates compose files on-demand if missing) -make up - -# 3. Start a specific service -make up SERVICE=grafana - -# 4. List all services the Makefile can see -make list -``` - ---- - -## Config File Format - -Each service directory contains a `configs.yaml` that drives `docker_compose_builder.sh`, with four sections: - -```yaml -GENERAL: - IMAGE: - TAG: - NAME: # also becomes the compose service key - -NETWORK_CONFIGS: - NETWORK_MODE: host | ports # host → network_mode: host; ports → publishes listed ports - PORTS: - - - -ENV_VARS: - KEY: value # passed as environment variables - # comment lines are stripped - -VOLUMES: - volume-name: /container/path # named volumes — declared at top level automatically -``` - -> **`REMOTE_GUI_NIC`** (remote-gui only): when this key is present in `ENV_VARS`, the builder resolves its value as a network interface name at generation time and injects `VITE_API_URL=http://:`. Leave the key absent to skip `VITE_API_URL` entirely. - -All generated compose files also include: - -```yaml -restart: always -stdin_open: true -tty: true -``` - ---- - -## Service Configs - -### Remote-GUI - -| Field | Value | -|---|---| -| Image | `anylogco/remote-gui:1.0.0` | -| Ports | `8080` (backend), `31800` (frontend) | -| Key env vars | `REMOTE_GUI_FE`, `REMOTE_GUI_BE`, `GRAFANA_URL`, `REMOTE_GUI_NIC` | -| Volumes | `image-vol`, `usr-mgm-vol`, `report-configs` | -| Aliases | `remote-gui`, `gui` | - -### Grafana - -| Field | Value | -|---|---| -| Image | `grafana/grafana:latest` | -| Port | `3000` | -| Key env vars | `POSTGRES_USER`, `POSTGRES_PASSWORD`, `POSTGRES_INITDB_ARGS` | -| Volumes | `grafana-data`, `grafana-log`, `grafana-config` | -| Aliases | `grafana` | - -### PostgreSQL - -| Field | Value | -|---|---| -| Image | `postgres:16.0-alpine` | -| Port | `5432` | -| Key env vars | `POSTGRES_USER`, `POSTGRES_PASSWORD`, `POSTGRES_INITDB_ARGS` | -| Volumes | `pgdata` | -| Aliases | `postgres`, `psql` | -| Native client | `psql -U admin -d postgres` (via `make attach`) | - -### MongoDB - -| Field | Value | -|---|---| -| Image | `mongo:latest` | -| Port | `27017` | -| Key env vars | `MONGO_USER`, `MONGO_PASSWORD` | -| Volumes | `mongo-data`, `mongo-configs` | -| Aliases | `mongodb`, `mongo` | -| Native client | `mongosh` (via `make attach`) | - -### Ollama - -Standalone — **not managed by the Makefile**. Lightweight open-source framework for running LLMs locally; AnyLog/EdgeLake use it as the tested model framework for MCP function calling in the Remote-GUI. - -| Field | Value | -|---|---| -| Image | `ollama/ollama:latest` | -| Port | `11434` | -| GPU variant | `ollama/docker-compose-gpu.yaml` (requires NVIDIA Container Toolkit) | -| Default model | `qwen2.5:7b-instruct` | - -→ Full setup guide: `Ollama.md` - -### Video Inference Models - -Standalone — **not managed by the Makefile**. CV/ML inference on edge video streams, feeding results into AnyLog/EdgeLake nodes. - -→ Full setup guide: `Video-Inferences.md` -→ Source repository: `AnyLog-co/AnyLog-Video-Inference-Models` - -### Nebula Overlay Network - -Standalone — **not managed by the Makefile**. Nebula creates an encrypted peer-to-peer mesh across physically separated machines, giving distributed AnyLog/EdgeLake nodes a shared overlay IP space without requiring them to be on the same LAN or VPN. - -| Field | Value | -|---|---| -| Deployed via | `docker compose` on each node | -| Min. nodes | 1 lighthouse + 1 host | -| Overlay port | UDP `4242` | -| Auth | Mutual certificate-based (CA you control) | - -→ Full setup guide: `Nebula.md` -→ Source repository: `oshadmon/nebula-anylog` - -### Syslog Forwarding - -Special-cased in the Makefile: when `SERVICE=syslog` is passed, `setup`, `remove`, and `help` delegate directly to `syslog.sh` instead of the docker-compose lifecycle path. - -`syslog.sh` reads `SYSLOG_MONITORING` and `ANYLOG_BROKER_PORT` directly from a `node_configs.env` file — no extra arguments needed once the node config is set up. Both `setup` and `remove` are **no-ops** when `SYSLOG_MONITORING != "true"`; `setup` is idempotent (safe to call repeatedly). - -| Field | Value | -|---|---| -| Config keys | `SYSLOG_MONITORING` (must be `"true"`), `ANYLOG_BROKER_PORT` | -| Linux | rsyslog drop-in `/etc/rsyslog.d/60-custom-forwarding.conf` (TCP) | -| macOS | `/etc/syslog.conf` append (UDP) — ensure the broker port is open for UDP | -| Default `NODE_CONFIGS` path | `docker-makefiles/anylog-generic/node_configs.env` | - -```bash -# Direct invocation -bash syslog.sh setup [NODE_CONFIGS] -bash syslog.sh remove [NODE_CONFIGS] - -# Via make (SERVICE=syslog intercepts before docker-compose logic) -make setup SERVICE=syslog # prompts for config path -make setup SERVICE=syslog NODE_CONFIGS=../docker-makefiles/anylog-operator/node_configs.env -make remove SERVICE=syslog NODE_CONFIGS=../docker-makefiles/anylog-operator/node_configs.env -``` - -If `NODE_CONFIGS` is not supplied on the command line, `make setup`/`make remove` prompt interactively for the config file path. - ---- - -## `docker_compose_builder.sh` - -Reads a `configs.yaml` and writes a `docker-compose.yml` next to it. - -```bash -# Usage -./docker_compose_builder.sh [config_file] [output_file] - -# Defaults -./docker_compose_builder.sh # configs.yaml → docker-compose.yml -./docker_compose_builder.sh remote-gui/configs.yaml # custom input -./docker_compose_builder.sh remote-gui/configs.yaml out-compose.yml # custom input + output -``` - -### Port Conflict Detection - -Before writing the compose file, the builder checks whether any port in `NETWORK_CONFIGS.PORTS` is already in use, adapting the check to the network mode: - -| `NETWORK_MODE` | How ports are bound | Check method | -|---|---|---| -| `ports` | Docker proxy | `docker ps --format '{{.Ports}}'` | -| `host` | Host OS directly | `ss -tlnp` (falls back to `lsof`) | - -If a conflict is found, the builder prints which container or process holds the port and exits non-zero — the compose file is **not** written. - ---- - -## Makefile Reference - -### Targets - -| Target | Description | -|---|---| -| `dry-run` | Generate `docker-compose.yml` for all discovered services (or `SERVICE=one`) | -| `up` | Start service(s); generates the compose file on-demand if missing | -| `down` | Stop service(s) | -| `clean` | Stop and remove volumes | -| `clean-all` | Stop, remove volumes, and remove the image | -| `logs` | Print container logs — `SERVICE=` required | -| `logs-f` | Follow container logs — `SERVICE=` required | -| `attach` | Attach the service's **native client** (`psql` for postgres*, `mongosh` for mongo*) — `SERVICE=` required. Any other service prints "Unsupported attach option." | -| `exec` | Open a **plain bash shell** in the container — `SERVICE=` required | -| `list` | Print default and all auto-discovered services | -| `setup` | `SERVICE=syslog` only — install the host syslog → AnyLog forwarding rule | -| `remove` | `SERVICE=syslog` only — remove the host syslog → AnyLog forwarding rule | -| `help` | Show usage and the list of targets (delegates to `syslog.sh help` when `SERVICE=syslog`) | - -> **Note on `attach` vs `exec`:** these are two distinct targets. `attach` drops you into the database client that matches the service name pattern (`postgres*` → `psql -U admin -d postgres`, `mongo*` → `mongosh`); `exec` always opens `/bin/bash` regardless of service type. - -Omit `SERVICE` to act on all four default services (`remote-gui`, `grafana`, `postgres`, `mongodb`) for `up`, `down`, `clean`, and `clean-all`. `logs`, `logs-f`, `attach`, and `exec` always require an explicit `SERVICE=`. - -### Service Aliases - -| Service | Accepted values for `SERVICE=` | -|---|---| -| Remote-GUI | `remote-gui`, `gui` | -| Grafana | `grafana` | -| PostgreSQL | `postgres`, `psql` | -| MongoDB | `mongodb`, `mongo` | - -### Custom Instances - -To run a second PostgreSQL or MongoDB instance (e.g. for a different AnyLog node), create a new directory with a `configs.yaml` — no Makefile changes required: - -```bash -cp -r postgres/ postgres-prod/ -# edit postgres-prod/configs.yaml (change NAME, port, volumes as needed) - -make up SERVICE=postgres-prod -make logs SERVICE=postgres-prod -make attach SERVICE=postgres-prod # drops into psql automatically (matches postgres* pattern) -``` - -`make list` shows every directory the Makefile has discovered. - -### Example Commands - -```bash -# Generate compose files -make dry-run # all discovered services -make dry-run SERVICE=remote-gui # one service - -# Lifecycle -make up # start all 4 defaults -make up SERVICE=gui -make up SERVICE=postgres-prod # custom instance -make down SERVICE=grafana -make clean SERVICE=mongo -make clean-all SERVICE=psql - -# Logs -make logs SERVICE=gui # print and exit -make logs-f SERVICE=mongo # follow - -# Native client access (psql / mongosh) -make attach SERVICE=psql # psql -U admin -d postgres -make attach SERVICE=mongo # mongosh -make attach SERVICE=postgres-prod # psql -U admin -d postgres (matches postgres* pattern) - -# Plain bash shell in the container -make exec SERVICE=gui # /bin/bash -make exec SERVICE=psql # /bin/bash -make exec SERVICE=mongo # /bin/bash - -# Syslog forwarding -make setup SERVICE=syslog # prompts for node config path -make setup SERVICE=syslog NODE_CONFIGS=../docker-makefiles/anylog-operator/node_configs.env -make remove SERVICE=syslog NODE_CONFIGS=../docker-makefiles/anylog-operator/node_configs.env - -# Discovery -make list # show default + all discovered services -make help # show usage and target list -``` \ No newline at end of file diff --git a/02- Installation & Deployment/02- Virtualization/99-02- Kubernetes/03 Kubernetes Networking.md b/02- Installation & Deployment/02- Virtualization/99-02- Kubernetes/03 Kubernetes Networking.md deleted file mode 100644 index 9e58fdc6..00000000 --- a/02- Installation & Deployment/02- Virtualization/99-02- Kubernetes/03 Kubernetes Networking.md +++ /dev/null @@ -1,82 +0,0 @@ ---- -title: "Kubernetes Networking" -description: "" -layout: page -source_path: "deployments/D- Networking & Security/03 Kubernetes Networking.md" ---- - - -# Kubernetes Networking - -Kubernetes Orchestrator generates a unique virtual IP each time a node is deployed. - -Since communication between nodes is policy based, AnyLog cannot have a new virtual IP each time a node is deployed. - -As such, the configuration file, for Kubernetes, has a configuration called `KUBERNETES_SERVICE_IP` which should be set -to the service name of the deployment. When using the deployment scripts, -this step is done automatically. - -Once set, the network connectivity will be done with the virtual IP address for local IP. However, on the blockchain the -local_ip will be set to the service name; which other members on the Kubernetes cluster can utilize. - -When deploying with Kubernetes we recommend using Nginx or other proxy service, as well as our -configuration policy. - -Directions for using Kubernetes Cluster Networking can be found here - -## Sample Network Setup -* The External & Virtual IP are used for setting up TCP connections -* The Virtual IP is used for setting the REST connection -* The External & Virtual IP are used for setting up Message Broker connections - -The following chart summarizes the setup: - -| Connection Type | External IP |Local IP | Config Command | -|------------------|-------------|-----------|-------------------------| -| TCP | External IP | Virtual IP | `run tcp server...` | -| REST | Vritul IP | Virtual IP | `run rest server...` | -| TCP | External IP | Virtual IP | `run message broker...` | - -Farther details regarding things like binding and thread count can be found in our networking section - -## Sample Node Policy for Kubernetes -The following provides a basic example of both the configuration policy, as-well-as a (master) node policy. - -Values in the configuration policy are relatively set, that way when a deployment is restarted a new policy will not be -declared for the node due to the changing virtual IP. As for the local IP in the (master) node policy, we assume the -service name will not change. - -The sample JSON for (master) node policy is set to have communication between AnyLog nodes to be set to binding. When -the communication is set to not-binding, the external IP (`73.222.38.13`) will be set to `ip` key in the policy, and the -Kubernetes service IP (`anylog-master-service`) will be set to `local_ip`. - -```json -{"config" : { - "name" : "master-configs", - "company" : "New Company", - "port" : "!anylog_server_port.int", - "external_ip" : "!external_ip", - "ip" : "!ip", - "rest_port" : "!anylog_rest_port.int" -}}, -{"master" : { - "name" : "anylog-master", - "company" : "New Company", - "hostname" : "anylog-master-pod", - "loc" : "0.0, 0.0", - "country" : "Unknown", - "state" : "Unknown", - "city" : "Unknown", - "port" : 32048, - "external_ip" : "73.222.38.13", - "ip" : "anylog-master-service", - "proxy_ip" : "10.0.0.183", -}} -``` diff --git a/02- Installation & Deployment/02- Virtualization/99-05- AnyLog as a Service.md b/02- Installation & Deployment/02- Virtualization/99-05- AnyLog as a Service.md deleted file mode 100644 index e254654a..00000000 --- a/02- Installation & Deployment/02- Virtualization/99-05- AnyLog as a Service.md +++ /dev/null @@ -1,289 +0,0 @@ ---- -title: "AnyLog as a Service" -description: "Running AnyLog as a systemd service instead of via Docker/Kubernetes." -layout: page ---- - - -> ⚠️ **Draft — needs to be recreated / reviewed once we have a confirmed current binary version.** -> This document was merged from two older, mutually inconsistent drafts. The overall flow (run as an -> executable, then convert to a service) is solid, but specific details — binary version, download -> source, and the config-file-generation path — are carried over from stale source material and are -> flagged inline below rather than guessed at. - -# AnyLog as a Service - -AnyLog's standard deployment path is a Docker container, run via Docker or -Kubernetes. Under certain conditions — limited disk space, regulatory constraints, or -network restrictions — you may instead want to install AnyLog (or EdgeLake) directly as a service on the host, -with no container runtime involved. - -When run this way, the CLI interface is disabled — all communication with the node happens over REST once it's -running. - -## Part 1 — Run AnyLog as an Executable - -To start, we'll run everything from `~/Desktop/anylog` — a simple, disposable location for the initial trial run. -Part 2 moves this into a permanent location once you're ready to make it a service. - -1. Download the AnyLog binary. 🟡 *Confirm the current version/filename and download source before using this in - production* — see the changelog above. -```shell -mkdir -p $HOME/Desktop/anylog/ -cd $HOME/Desktop/anylog -wget http://45.33.11.32/anylog_v0.0.0_x86_64 -sudo chmod -R 750 $HOME/Desktop/anylog/anylog_v0.0.0_x86_64 -``` - -2. Clone the deployment scripts, which drive what the node actually does at startup: -```shell -cd $HOME/Desktop/anylog/ -git clone https://github.com/AnyLog-co/deployment-scripts -``` - -3. Build a configuration file. 🟡 *This step is preserved from the source docs as-is — it references an older - docker-compose repo layout (`docker-compose/docker-makefile/anylog-master/`) that doesn't match the current - structure in full-deployment.md. Needs a corrected path before this is trustworthy.* -```shell -cd $HOME/Desktop/anylog/ -git clone https://github.com/AnyLog-co/docker-compose -cat $HOME/Desktop/anylog/docker-compose/docker-makefile/* >> $HOME/Desktop/anylog/anylog_configs.env -``` - - At minimum, `anylog_configs.env` should define: -```dotenv -# file path: $HOME/Desktop/anylog/anylog_configs.env - -#--- Directories --- -ANYLOG_PATH=/home/user -LOCAL_SCRIPTS=/home/user/deployment-scripts/node-deployment -TEST_DIR=/home/user/deployment-scripts/tests - -#--- General --- -# See full-deployment.md's License Key section for how key resolution/prompting works for the -# docker-compose path — the executable path here still just reads this value directly, unvalidated. -LICENSE_KEY="" -NODE_TYPE=generic -NODE_NAME=anylog-node -COMPANY_NAME=New Company -DISABLE_CLI=true -REMOTE_CLI=false - -#--- Networking --- -ANYLOG_SERVER_PORT=32548 -ANYLOG_REST_PORT=32549 -ANYLOG_BROKER_PORT="" - -#--- Database --- -DB_TYPE=sqlite -DB_USER="" -DB_PASSWD="" -DB_IP=127.0.0.1 -DB_PORT=5432 - -#--- Blockchain --- -LEDGER_CONN=127.0.0.1:32048 -``` - -4. Run the executable inside a `screen` session, so it keeps running even after you close the terminal, and you can - reattach to it later to check on it before committing to a full systemd service: -```shell -# start a new named screen session -screen -S anylog - -# inside the session, run AnyLog in the foreground: -$HOME/Desktop/anylog/anylog_v0.0.0_x86_64 process $HOME/Desktop/anylog/deployment-scripts/node-deployment/main.al -``` - - **Detach** without stopping the process: press `Ctrl+A`, then `D`. - - **Reattach** later to check on it: -```shell -# list active sessions -screen -ls - -# reattach -screen -r anylog -``` - - **Stop it** when you're ready to move to Part 2: reattach, then `Ctrl+C` to end the process, then type `exit` to - close the screen session (or run `screen -X -S anylog quit` from outside the session). - -5. Confirm it's up (from a separate terminal, while the screen session is running): -```shell -curl -X GET 127.0.0.1:32549 -``` - -## Part 2 — Convert the Executable into a systemd Service - -1. Make sure the `screen` session from Part 1 is stopped before proceeding — the service will start its own instance. - -2. Move the executable, deployment scripts, and config out of the disposable `Desktop` location into a permanent - one. This example uses `/home/user/services/anylog/` — adjust to your actual username/path convention. -```shell -sudo mkdir -p /home/user/services/anylog -sudo mv $HOME/Desktop/anylog/anylog_v0.0.0_x86_64 /home/user/services/anylog/ -sudo mv $HOME/Desktop/anylog/deployment-scripts /home/user/services/anylog/ -sudo mv $HOME/Desktop/anylog/anylog_configs.env /home/user/services/anylog/ -``` - - Set ownership and permissions on the new location. `anylog_configs.env` contains the license key and database - credentials, so it shouldn't be world-readable; the binary needs to stay executable: -```shell -sudo chown -R root:root /home/user/services/anylog -sudo chmod 750 /home/user/services/anylog -sudo chmod 750 /home/user/services/anylog/anylog_v0.0.0_x86_64 -sudo chmod 640 /home/user/services/anylog/anylog_configs.env -``` - -3. Create the service file: -```ini -# file path: /etc/systemd/system/anylog-service.service -[Unit] -Description=AnyLog Deployment -After=network.target - -[Service] -Type=simple -ExecStart=/home/user/services/anylog/anylog_v0.0.0_x86_64 process /home/user/services/anylog/deployment-scripts/node-deployment/main.al -EnvironmentFile=/home/user/services/anylog/anylog_configs.env -Restart=always -User=root -Group=root - -[Install] -WantedBy=multi-user.target -``` - -4. Enable and start the service: -```shell -sudo systemctl daemon-reload -sudo systemctl enable anylog-service.service -sudo systemctl restart anylog-service.service -``` - -5. Validate: -```shell -sudo systemctl status anylog-service.service - -# get status -curl -X GET 127.0.0.1:32549 - -# get processes -curl -X GET 127.0.0.1:32549 -H "command: get processes" -H "User-Agent: AnyLog/1.23" -``` - -## Alternative: Minimal REST-only Setup - -🟡 **Unverified — kept from the source material as-is.** This variant skips `deployment-scripts` entirely and -configures only TCP + REST via a small `.al` script, intended for nodes deployed 100% through REST calls after -startup. It references a `deployment_script.py` (using the -AnyLog API) that isn't included in either -source document — treat this section as a sketch of the approach, not a verified procedure, until that script is -located or rewritten. - -1. Prepare a startup script that configures only networking: -```anylog -# file path: $HOME/anylog/basic_deployment.al -on error ignore -:prep-instance: -set cli off -set authentication off - -:set-params: -anylog_server_port=32548 -anylog_rest_port=32549 -tcp_bind = false -rest_bind = false -tcp_threads=3 -rest_threads=3 -rest_timeout=30 - -:tcp-conn: -on error goto tcp-conn-error - - -:rest-conn: -on error goto rest-conn-error - - -:end-script: -end script - -:tcp-conn-error: -print "Failed to configure TCP connection" -goto end-script - -:rest-conn-error: -print "Failed to configure REST connection" -goto end-script -``` - -2. Create the service file: -```ini -# file path: /etc/systemd/system/anylog-service.service -[Unit] -Description=My Executable Service -After=network.target - -[Service] -ExecStart=/home/user/anylog/anylog_v0.0.0_x86_64 process /home/user/anylog/basic_deployment.al -ExecStartPost=/usr/bin/python3 /home/user/anylog/deployment_script.py 127.0.0.1:32549 --configs /home/user/anylog/anylog_configs.env -Restart=always -User=root -Group=root - -[Install] -WantedBy=multi-user.target -``` - -3. Start and validate exactly as in Part 2, steps 3–4 above. \ No newline at end of file diff --git a/02- Installation & Deployment/02- Virtualization/99-06- Pip Install.md b/02- Installation & Deployment/02- Virtualization/99-06- Pip Install.md deleted file mode 100644 index 9326c556..00000000 --- a/02- Installation & Deployment/02- Virtualization/99-06- Pip Install.md +++ /dev/null @@ -1,87 +0,0 @@ ---- -title: "AnyLog as a _pip_ Package" -description: "" -layout: page -source_path: "training/advanced/99-06- Pip Install.md" ---- - - -# AnyLog as a _pip_ Package - -## Deployment Process -1. Prerequisites - - **General** - * cython - * ast - * requests - * cryptography - * jwt (for alpine - install py3-jwt) - * pyOpenSSL - * psutil - * python-dateutil - * pytz - - **Database Specific** (optional) - * psycopg2-binary (for PostgresSQL, if you're using SQLite, there's no need to install this) - * pymongo (for storing blobs in MongoDB, alternatively, users can store blobs in files) - - **North / Southbound** (optional) - * paho-mqtt - * kafka-python (for accepting and sending data via Kafka) - - **Utilizing Blockchain instead of Master Node** (optional) - * web3 - * py4j - - **Images & Video Processing** (optional) - * numpy (for alpine - install py3-numpy) - * opencv-python (for alpine - install py3-opencv) - -The script below installs all prerequisites. -```shell -python3 -m pip install --upgrade -r https://raw.githubusercontent.com/AnyLog-co/documentation/master/deployments/Support/requirements.txt -``` - -2. Install AnyLog as a `pip` package - - Note: Use the following Python version: - * python3.10 for Ubuntu and Mac OS X - * Python3.11 for Alpine - -Versions of AnyLog can be found in the Downloads Page - -```shell -# Ubuntu -python3 -m pip install --upgrade http://173.255.254.34:31900/ubuntu/anylog_network-0.0.7-cp310-cp310-linux_x86_64.whl - -# Alpine -python3 -m pip install --upgrade http://173.255.254.34:31900/alpine/anylog_network-0.0.7-cp311-cp311-linux_x86_64.whl - -# Mac OSX -python3 -m pip install --upgrade http://173.255.254.34:31900/macosx/anylog_network-0.0.7-cp310-cp310-macosx_12_0_x86_64.whl -``` - -3. Deploy AnyLog node -```python -import sys -import anylog_node.cmd.user_cmd as user_cmd # import AnyLog Node - -argv = sys.argv -argc = len(argv) - -user_input = user_cmd.UserInput() -user_input.process_input(arguments=argc, arguments_list=argv) # Start AnyLog with CLI -``` - -4. Enable AnyLog (key valid until November 1, 2023) - for a personalized license key, contact us -```anylog -set license where activation_key=01954e0dbfa1b5c1785aed6790a34097c5db148cb78405fd16ae2494045de3e844895851d03e0e599a799d6e6f03cbd2233a5f65a6dfb74832fb1034d5a56d8fa02563061a321da246e7660c4d00b9ea050b5d6fc4c61d7f9d53d58accec0434eb3b0fa98ae9237dfe09a6a75e0c6efcc4bc7860e9e358672b3d93943dbb416c2023-11-01bGuest -``` diff --git a/02- Installation & Deployment/03- Orchestrators/01- Open Horizon.md b/02- Installation & Deployment/03- Orchestrators/01- Open Horizon.md deleted file mode 100644 index d9c3d2de..00000000 --- a/02- Installation & Deployment/03- Orchestrators/01- Open Horizon.md +++ /dev/null @@ -1,246 +0,0 @@ ---- -title: "Open Horizon" -description: "" -layout: page -source_path: "deployments/Open-Horizon/01- Open Horizon.md" ---- - - -# Open Horizon - -Open Horizon is a platform for managing the service software lifecycle of containerized workloads and related machine -learning assets. It enables autonomous management of applications deployed to distributed web scale fleets of edge -computing nodes and devices without requiring on-premise administrators. - -Open Horizon can be used to easily manage and deploy AnyLog node(s) through their interface. -* Open Horizon Website -* IBM Documentation for Open Horizon -* Open Source Documentation -* AnyLog Documentation -* AnyLog Website - -## Requirements -* A physical / virtual machine for each node, as OpenHorizon is unable to deploy more than 1 instance per node -* Machine requirements - -**For 64-bit Intel or AMD device or virtual machine:** - -* 64-bit Intel or AMD device or virtual machine -* An internet connection for your device (wired or Wi-Fi) - -**For Linux on ARM (32-bit):** - -* Hardware requirements - Raspberry Pi 3A+, 3B, 3B+, or 4 (preferred), but also supports A+, B+, 2B, Zero-W, or Zero-WH -* MicroSD flash card (32 GB preferred) -* An Internet connection for your device (wired or Wi-Fi). Note: Some devices can require extra hardware for supporting Wi-Fi. - -## Associating Machine to Open Horizon -The following steps will associate a new machine with the Open Horizon management platform. The process will complete the -following: -* Create an API key -* Install Horizon CLI -* Install Docker -* Validate Open Horizon is working by deploying a _Hello World_ package - -1. On the node Update / Upgrade Node -```shell -for CMD in update upgrade ; do sudo apt-get -y ${CMD} ; done -``` - -2. Create an Open Horizon API Key - -3. Update Environment variables -* In `~/.bashrc` (or `~/.profile` for Alpine) add the following variables - -```shell -export HZN_ORG_ID= -export HZN_EXCHANGE_USER_AUTH="iamapikey:" -export HZN_EXCHANGE_URL= -export HZN_FSS_CSSURL= -``` -* Set Environment variables - -```shell -# For non-Alpine operating systems -source ~/.bashrc - -# For Alpine operating systems -source ~/.profile -``` - -4. Install agent and provide admin privileges - -```shell -curl -u "${HZN_ORG_ID}/${HZN_EXCHANGE_USER_AUTH}" -k -o agent-install.sh ${HZN_FSS_CSSURL}/api/v1/objects/IBM/agent_files/agent-install.sh/data - -chmod +x agent-install.sh - -sudo -s -E ./agent-install.sh -i 'css:' -p IBM/pattern-ibm.helloworld -w '*' -T 120 -``` - -5. Validate helloworld sample edge service is running - -```shell -hzn eventlog list -f - -< If Docker is already installed via _hzn_, however needs permissions to not use root run: ->```shell -> sudo groupadd docker -> sudo usermod -aG docker $(whoami) -> newgrp docker ->``` - -At the end of the process, OpenHorizon should show a new active node -OpenHorizon_node_state.png - -# AnyLog -By deploying AnyLog, users can monitor Distributed Edge Nodes and Data from a single point, without centralizing the data. - -To include AnyLog in your edge deployments, follow a 3 steps process: -1. Request a license key from AnyLog using the following link (once) https://anylog.co/download-anylog/. -2. Update the service definition for each monitored node. -3. Publish the AnyLog-Node Service for each monitored node. - -This process is detailed below. - -## Associate AnyLog Deployment with OpenHorizon - -1. Log into IBM Edge Application Manager - -2. Under _Services_ add an "Edge Device" - -3. Declare _AnyLog_ as a device image - Docker login credentials are received using: AnyLog Downloads - -OpenHorizon_device_info.png - -4. Configure Service variables -* `INIT_TYPE` (**value**: training) - Which AnyLog scripts to use for the deployment -* `LICENSE_KEY` - AnyLog license key -* `NODE_TYPE` (**value**: operator) - which AnyLog node type to deploy (for training purposes we support: _operator_, _query_ and _master_) -* `NODE_NAME` - AnyLog node name -* `COMPANY_NAME` - company the node is associated with -* `ANYLOG_SERVER_PORT` (**value**: 32148) - Port used for communicating between AnyLog nodes -* `ANYLOG_SERVER_PORT` (**value**: 32149) - Port used for communicating used for communicating with an AnyLog node via REST -* `LEDGER_CONN` (**value**: 132.177.125.232:32048) - A remote AnyLog instance used as the "manager" for AnyLog -* `ENABLE_MQTT` (**value**: true) - Enable receiving data from a remote MQTT broker -* `ENABLE_MONITORING` (**value**: true) - Enable monitoring of the Node - -OpenHorizon_serivce_info.png - -5. Under "Deployment Options", Enable _Host Network Mode_ - -OpenHorizon_deployment_options.png - -6. Save changes - you should see "anylog-node" as a published service - -OpenHorizon_published_service.png - -7. Create an AnyLog pattern -OpenHorizon_sample_pattern.png - -## Create AnyLog node as a Service on Open Horizon - -1. Request the AnyLog license key to download AnyLog from the _Docker_ repository using: AnyLog Downloads - -2. Update variables in the `service.definition.json` configuration file at (Operator Node) with the following: - -| Variable | Update with | Default Value | Comments | -| -------------- | --------------------------------------- | ------------ | ------------ | -| INIT_TYPE | training | training | Used to decide which AnyLog scripts to use for the deployment | -| LICENSE_KEY | The Docker Hub key provided by AnyLog | | Request key using AnyLog Downloads | -| NODE_TYPE | operator | operator | A node configured to host data | -| NODE_NAME | [your company name]_operator[node id] | | For example: ibm_operator123 | -| COMPANY_NAME | [your company name] | | For example: ibm | -| LEDGER_CONN | `132.177.125.232:32048` | `132.177.125.232:32048` | The Network ID (the IP and Port of the Master) | - -3. Deploy Node - -> Note, `hzn` is not able to deploy more than a single instance on a given machine - -```shell -# Operator Node -cd ~/service-anylog/deployments/operator/ -hzn register --policy node.policy.json - -# Query Node -cd ~/service-anylog/deployments/query/ -hzn register --policy node.policy.json -``` - -4. Validate node is running - the example is of - -* Validate via `docker log` - -```shell -docker logs c33bd07d4808467d90fc1ef41ef2bff81d6502d5ca0bfb6b97ce614becda42b6-anylog-node - -< - Process Status Details - ---------------|------------|---------------------------------------------------------------------------| - TCP |Running |Listening on: 198.74.50.131:32148, Threads Pool: 6 | - REST |Running |Listening on: 198.74.50.131:32149, Threads Pool: 5, Timeout: 20, SSL: False| - Operator |Running |Cluster Member: True, Using Master: 172.105.4.104:32048, Threads Pool: 3 | - Publisher |Not declared| | - Blockchain Sync|Running |Sync every 30 seconds with master using: 132.177.125.232:32048 | - Scheduler |Running |Schedulers IDs in use: [0 (system)] [1 (user)] | - Distributor |Not declared| | - Blobs Archiver |Running | | - Consumer |Not declared| | - MQTT |Running | | - Message Broker |Running |Listening on: 198.74.50.131:32150, Threads Pool: 5 | - SMTP |Not declared| | - Streamer |Running |Default streaming thresholds are 60 seconds and 10,240 bytes | - Query Pool |Running |Threads Pool: 3 | - Kafka Consumer |Not declared| | - -AL anylog-operator1 > -Subscription ID: 0001 -User: ibglowct -Broker: driver.cloudmqtt.com:18785 -Connection: Connected - - Messages Success Errors Last message time Last error time Last Error - ---------- ---------- ---------- ------------------- ------------------- ---------------------------------- - 0 0 0 - - Subscribed Topics: - Topic QOS DBMS Table Column name Column Type Mapping Function Optional Policies - ----------------|---|------------|----------------|-----------|-----------|-----------------------|--------|--------| - anylogedgex-demo| 0|open_horizon|['[sourceName]']|timestamp |timestamp |now() |False | | - | | | |value |float |['[readings][][value]']|False | | -<< -``` - -* Test Network - -```shell -curl -X GET 127.0.0.1:32149 -H "command: test network" - -<` - -to be completed \ No newline at end of file diff --git a/02- Installation & Deployment/03- Orchestrators/03- Barbara.md b/02- Installation & Deployment/03- Orchestrators/03- Barbara.md deleted file mode 100644 index 214cb9ce..00000000 --- a/02- Installation & Deployment/03- Orchestrators/03- Barbara.md +++ /dev/null @@ -1,17 +0,0 @@ ---- -title: "Deploying AnylogEDF with BARBARA" -description: "" -layout: page -source_path: "" ---- - - - -To Be Added diff --git a/02- Installation & Deployment/03- Orchestrators/04- DELL Distributed Private Cloud.md b/02- Installation & Deployment/03- Orchestrators/04- DELL Distributed Private Cloud.md deleted file mode 100644 index b5a5668f..00000000 --- a/02- Installation & Deployment/03- Orchestrators/04- DELL Distributed Private Cloud.md +++ /dev/null @@ -1,17 +0,0 @@ ---- -title: "Deploying Anylog EDF with DELL (Native Edge) Distributed Private Cloud" -description: "" -layout: page -source_path: "" ---- - - - -To Be Added diff --git a/02- Installation & Deployment/03- Orchestrators/05- Zededa.md b/02- Installation & Deployment/03- Orchestrators/05- Zededa.md deleted file mode 100644 index fb154c8a..00000000 --- a/02- Installation & Deployment/03- Orchestrators/05- Zededa.md +++ /dev/null @@ -1,11 +0,0 @@ - - - - -To Be Added \ No newline at end of file diff --git a/02- Installation & Deployment/04- Cloud Support/01- AWS Deployment.md b/02- Installation & Deployment/04- Cloud Support/01- AWS Deployment.md deleted file mode 100644 index 0e4342d2..00000000 --- a/02- Installation & Deployment/04- Cloud Support/01- AWS Deployment.md +++ /dev/null @@ -1,67 +0,0 @@ ---- -title: "AWS Setup" -description: "support for AWS" -layout: page -source_path: "" ---- - - -The following provides directions for using an EC2 AWS instance(s) to configure and deploy AnyLog. - -Directions for deploying an AnyLog Node - -**Terminology**: -* Node - An AnyLog node that's part of the network (ex. Master, Operator, Query) -* Instance - An AWS EC2 machine - -## Create a new AWS instance -0. Create an account and/or log into AWS - -1. Create a key pair for SSH access into the node. - -2. Create a new security group per node type. - -The image below, shows sample configuration for an AnyLog (Operator) node; this includes _SSH_ access, _TCP_ and _REST_, as well -as _Broker_ if configured for Operator or Publisher node. -* **Master Node** - Default TCP: 32048 | Default REST: 32049 -* **Query Node** - Default TCP: 32348 | Default REST: 32349 -* **Remote-CLI** which usually seats on the same physical machine as the Query node requires port 31800 to be open. -* **Operator Node** - Default TCP: 32148 | Default REST: 32149 | (Default) Broker: 32150 -* **Publisher Node**: Default TCP: 32248 | Default REST: 32249 | (Default) Broker: 32250 -* **Generic / REST Node**: Default TCP: 32548 | Default REST: 32549 | (Default) Broker: 32550 - -image - -3. Create a new EC2 instance that's associated with the corresponding _security group_. - -4. Configure a static IP address - - a. Create an Elastic IP. -This is very important as AWS provides a new IP each time an instance is rebooted (unless _elastic IP_ is configured). - - b. Associate Elastic IP with Instance - -5. Repeat steps 3 and 4 for each new AWS instance. - -When using different regions for the AnyLog nodes, the TCP binding needs to be disabled. - -**Sample Config**: The example shows a subsection of the anylog_configs.env file, with the `TCP_BIND` option disabled. -```.env -... -ANYLOG_SERVER_PORT=32048 -# Port address used by AnyLog's REST protocol -ANYLOG_REST_PORT=32049 -# Port value to be used as an MQTT broker, or some other third-party broker -#ANYLOG_BROKER_PORT= -# A bool value that determines if to bind to a specific IP and Port (a false value binds to all IPs) -TCP_BIND=false -... -``` - diff --git a/03- Training & Tutorials/01- Training.md b/03- Training & Tutorials/01- Training.md deleted file mode 100644 index 8e86d995..00000000 --- a/03- Training & Tutorials/01- Training.md +++ /dev/null @@ -1,57 +0,0 @@ ---- -title: "Training & Tutorials" -description: "A quick run through of the training for AnyLog" -layout: page -source_path: "training/01- Training.md" ---- - -The previous section covered the different ways a user can download and -install AnyLog — whether via Docker / Kubernetes, directly on the machine, via pip or as a service, or using an -orchestration tool like IBM's Open Horizon or ZEDEDA. - -This section covers what happens once the AnyLog agent(s) is installed and running. - -## What's in This Section - -* Basic commands -* Query Data -* Deployment Process -* Deployment Scripts -* Node Configuration Process -* Basic Docker / K8s Commands - -Section 12 provides full examples for deploying a local / demo network from start to finish. - -## The Process - -Recap of the previous section: - -1. `git clone` the docker-compose repo (or another repo that eases the deployment of AnyLog). -2. Update the configuration files in the dotenv. There's a unique dotenv file per AnyLog node type. -3. Start the AnyLog agent. - -Once a user executes `make up` on an AnyLog agent, the following steps occur: - -1. A process (`docker_compose_builder.sh`) takes the different parts of the dotenv configuration and the relative - path of the configuration file to generate a `docker-compose.yml`: - * Container name — same as `NODE_NAME` if provided, else the directory name is used as the container name. - * Container volumes are created if they don't already exist. - * Networking is configured based on user input, falling back to the operating system's defaults if no input is - provided. -2. `docker compose up` is executed. -3. The deployment-scripts repo is downloaded if needed, or reused if it already exists (see the - deployment scripts deep dive for the different sourcing strategies). -4. The AnyLog agent starts inside the docker container. - -What happens inside AnyLog: - -1. Deployment-scripts shell environment parameters are converted into AnyLog params (e.g. `$NODE_NAME` → `!node_name`). -2. Based on the node type, a configuration policy is created if one does not already exist. -3. The configuration policy is deployed: - 1. Network connectivity is defined. - 2. The node policy is defined (if it does not already exist). - 3. Connections to databases are established as needed. - 4. Connections to services such as `blockchain sync` and `run operator` are established, based on the AnyLog - agent type. - -Once that's done, the agent can work independently with the rest of the members in the network. \ No newline at end of file diff --git a/03- Training & Tutorials/02- Basic Commands.md b/03- Training & Tutorials/02- Basic Commands.md deleted file mode 100644 index 257c4a57..00000000 --- a/03- Training & Tutorials/02- Basic Commands.md +++ /dev/null @@ -1,399 +0,0 @@ ---- -title: "The Basic Guided Tour" -description: "a basic understanding of using AnyLog" -layout: page -source_path: "training/02- Basic Commands.md" ---- - - - -# Basic AnyLog Commands - -This document covers the basics of AnyLog commands. These commands can be executed on the CLI (once attached) or via -REST — usually via GET unless stated otherwise. - -The assumption for this document is that the node is already running and configured. - -## REST Formatting - -A quick reference for connecting to AnyLog via REST. A more detailed document can be found here. - -**REST `GET` example** - -```shell -curl -X GET http://[Node ip]:[Node port] \ - -H "command: [AnyLog Command]" \ - -H "AnyLog-Agent: AnyLog/1.23" -``` - -**REST `POST` examples** - -```shell -# GET commands via POST - -curl -X POST http://[Node ip]:[Node port] \ - -H "Content-Type: application/json" \ - -d '{"command": "[AnyLog Command]", "AnyLog-Agent": "AnyLog/1.23" }' - -# Pure POST command — used for non-`get` commands - -curl -X POST http://[Node ip]:[Node port] \ - -H "command: [AnyLog Command]" \ - -H "AnyLog-Agent: AnyLog/1.23" -``` - ---- - -## Help Commands - -The `help` command provides dynamic information on AnyLog commands. - -**List all commands** — type `help` on the CLI: - -```anylog -help -``` - -**List commands by prefix** — for example, `get` is the prefix of a group of commands, listed via `help get`: - -```anylog -help get -help set -help reset -help blockchain -``` - -**List usage and examples for a specific command** — type `help` followed by the command text: - -```shell -help connect dbms -help blockchain insert -help get msg client -``` - -`help` returns the usage, examples, an explanation, and a link to the relevant documentation. For example: - -```anylog -help blockchain get - -Usage: - blockchain get [policy type] [where] [attribute name value pairs] [bring] [bring command variables] - -Explanation: - Get the policies or information from the policies that satisfy the search criteria. - -Examples: - blockchain get * - blockchain get operator where dbms = lsl_demo - blockchain get cluster where table[dbms] = purpleair and table[name] = air_data bring [cluster][id] separator = , - blockchain get operator bring.table [*] [*][name] [*][ip] [*][port] - blockchain get * bring.table.unique [*] - -Index: - ['blockchain'] -``` - ---- - -## Logged Events - -Like an operating system, AnyLog has a built-in logging mechanism that tracks events and errors within the agent. -Each node maintains buffers to record events and errors so that users and applications can retrieve recent events -and errors as they accrue. - -**Command format:** - -```anylog -get [log type] log where format = [format type] and keys = key1 key2 ... -``` - -| Parameter | Description | -|---|---| -| `[log type]` | `event`, `error`, `file`, `query`, or `msg` | -| `format` | Output format — `table` (default) or `json` | -| `key` | One or more keywords to filter logged events by content | - -**Examples:** - -```anylog -get event log where format = json and keys = SQL Error - -get error log where format = json and keys = rest -``` - -### Event Log - -The event log records the events processed on the node, including the AnyLog commands executed and any error -messages. - -```anylog -get event log -``` - -### Error Log - -The error log records recent errors on the node — these could be "human errors" (an invalid command) or internal -issues (a failure to send a message from one machine to another). - -```anylog -get error log -``` - -### Echo Queue - -AnyLog has two print commands — `echo` and `print`. `print` **always** returns output to screen, like Python's -`print`. `echo` does the same thing unless a logging config is enabled (done through the deployment-scripts) — -this lets you run a "print" that's stored in a log rather than only shown on screen. - -```anylog -# Enable echo queue -set echo queue on - -# Use echo queue -echo "hello world" - -# View the echo queue -get echo queue - -# Disable echo queue -set echo queue off -``` - -### Reset the Log - -The `reset` command clears variables and configuration parameters — in this case, it deletes the log entries in -the specified file: - -```anylog -reset [event/error/echo queue] log -``` - ---- - -## Dictionary - -The local dictionary enables hardware abstraction by mapping configuration values — which are specific to the -hardware in use — to generic keys shared across all deployments. Configuration, queries, and AnyLog commands -reference these shared keys, which are translated to the node-specific values. For example, IPs, ports, and file -or directory paths are referenced by key name and translated on each node to the appropriate value. - -Some entries represent default setups and configurations; users can add or modify entries as needed. - -For example, every directory in the default folder structure can be referenced by key, even though the physical -location of each folder may differ per deployment. Using this pattern, the archive directory is referenced via the -key `!archive_dir`, the blobs directory via `!blobs_dir`, and so on — allowing a shared configuration process while -the physical path to each folder can differ per node. - -Users can add any key/value pair needed to support a process — for example, storing values generated by data -ingestion or representing node state. A user could declare a key called `disk_usage` and configure the scheduler to -update its value with the percentage of free space every 15 seconds, then reference that value in processes that -monitor node state. - -The dictionary is also used to construct, maintain, and update policies before their persistent storage in the -shared metadata. - -### Retrieve Dictionary Values - -`get dictionary` returns all dictionary values: - -```anylog -# Table format -get dictionary - -# JSON format -get dictionary where format = json -``` - -A single value is retrieved as follows: - -```anylog -!var_name -``` - -Via REST, specify `get` before the key: - -```shell -curl -X GET http://[Node ip]:[Node port] \ - -H "command: get !var_name" \ - -H "AnyLog-Agent: AnyLog/1.23" -``` - -* On the AnyLog CLI, it's sufficient to reference the value without the `get` keyword. -* System variables are referenced with a dollar sign — for example, `$TMP` or `$HOME`. -* Paths and file names can be referenced by key + relative value. For example, `!local_scripts/node-deployment/main.al` - is transformed on each node into the full path `/app/deployment-scripts/node-deployment/main.al`, using the - physical path associated with the key `!anylog_path`. - -By default, a variable is returned as a string. To return it as a different type (e.g. int or bool), append a -conversion suffix: - -```anylog -AL > !anylog_rest_port - -"2148" - -AL > !anylog_rest_port.int - -2148 - -AL > set tcp_bind = true -AL > !tcp_bind - -"true" - -AL > !tcp_bind.bool - -True -``` - -### Defining a Variable - -To define a variable, set its name followed by `=` and the desired value. When the value is itself an AnyLog -command (e.g. `test`), prefix the assignment with `set` — otherwise, the *result* of the command is stored instead -of the command text. - -```anylog -my_var = 1 -my_var2 = "hello world" -``` - -```anylog -# Stores the result of the query — true / false depending on whether the file exists -is_file = file test !blockchain_dir/blockchain.json - -# Stores the literal command text `file test ...` -set is_file = file test !blockchain_dir/blockchain.json -``` - ---- - -## Networking & Services - -By default, user-defined configurations and the config policy set the network IPs and ports. A more detailed -document can be found here. - -> **Note:** IP addresses and node names shown in the examples below are illustrative placeholders, not a real -> network. - -### Checks Using `get` - -**View connections** — each row shows 3 addresses: External, Internal, and Bind. - -```anylog -get connections -``` - -``` -Type External Address Internal Address Bind Address ---------- ------------------- ------------------- ---------------- -TCP 10.0.0.10:32048 10.0.0.10:32048 10.0.0.10:32048 -REST 10.0.0.10:32049 10.0.0.10:32049 0.0.0.0:32049 -Messaging 10.0.0.10:32250 10.0.0.10:32250 0.0.0.0:32250 -``` - -If the Bind address is `0.0.0.0`, the node replies on any of the machine's IPs for that port. If bound to a specific -IP, AnyLog only accepts messages sent to that address. - -The TCP connection is the internal IP + port AnyLog agents use to communicate with one another — it is **not** -intended for a user to send data through directly; that's what the REST and Message Broker ports are for, covered -here. - -**View services** — lists all services available to enable/disable, and which are currently active. By default, -TCP, REST, and Blockchain Sync should be active. - -```anylog -get processes -``` - -``` -Process Status Details ----------------- ------------ ---------------------------------------------------------------------- -TCP Running Listening on: 10.0.0.10:32148, Threads Pool: 6 -REST Running Listening on: 10.0.0.10:32149, Threads Pool: 6, Timeout: 20, SSL: False -MCP Not declared -Operator Running Cluster Member: True, Using Master: 10.0.0.10:32048, Threads Pool: 3 -Blockchain Sync Running Sync every 120 seconds with master using: 10.0.0.10:32048 -Scheduler Running Schedulers IDs in use: [0 (system)] [1 (user)] -Blobs Archiver Running Flags: dbms = False, folder = True, compress = True, reuse_blobs = True -MQTT Running -MSG Client Pool Not declared -MSG Broker Running Listening on: 10.0.0.10:32150, Threads Pool: 6 -SMTP Not declared -Streamer Running Default streaming thresholds are 60 seconds and 10,240 bytes -UNS Streamer Not declared -Query Pool Running Threads Pool: 3 -Kafka Consumer Not declared -gRPC Not declared -PLC Client Not declared -Pull Processes Running 1 pull processes active -Video Processes Not declared -Publisher Not declared -Distributor Not declared -Consumer Not declared -``` - -**View databases** — lists databases directly accessible on the AnyLog agent. - -```anylog -get databases -``` - -``` -Active DBMS Connections -Logical DBMS Database Type Owner IP:Port Configuration Storage ---------------- ------------- ------- --------------- --------------------------------------------- ---------------------------------------------- -almgm psql system 127.0.0.1:5432 Autocommit On, Fsync on Persistent -blockchain psql system 10.0.0.10:5432 Autocommit On, Fsync on Persistent -monitoring sqlite user Local Autocommit On, Fsync full (after each write) /app/AnyLog-Network/data/dbms/monitoring.dbms -demo_data psql user 127.0.0.1:5432 Autocommit On, Fsync on Persistent -system_query sqlite system Local Autocommit On, RAM, Fsync full (after write) MEMORY -``` - -### Checking Using `test` - -There are two types of tests that can be run on a given node: - -**`test node`** — tests whether the node is configured properly and able to communicate with itself. - -```anylog -test node -``` - -``` -Test Status ------------------------------------------- ---------------------------------------------------------------- -Metadata Version d2ef59e6735872529f81c5c86702db76 -Metadata Test Pass -TCP test using 10.0.0.10:32048 [From Node 10.0.0.10:32048] anylog-master@10.0.0.10:32048 running -REST test using http://10.0.0.10:32049 anylog-master@10.0.0.10:32048 running -``` - -**`test network`** — checks whether the node can communicate with other agents on the blockchain. - -```anylog -test network -``` - -``` -Test Network -[****************************************************************] - -AL anylog-master +> -Address Node Type Node Name Status -------------------- --------- --------------------- ------ -10.0.0.10:32048 master anylog-master + -10.0.0.11:32148 operator operator-demo-1 + -10.0.0.12:32148 operator operator-demo-2 + -10.0.0.13:32148 operator operator-demo-3 + -10.0.0.14:32348 query query-demo-1 + -10.0.0.15:32348 query query-demo-2 + -10.0.0.16:32148 operator operator-demo-4 (backup) -``` - -A `+` status means the nodes can ping one another; a blank status means they cannot. \ No newline at end of file diff --git a/03- Training & Tutorials/03- Query Data.md b/03- Training & Tutorials/03- Query Data.md deleted file mode 100644 index a448fbc2..00000000 --- a/03- Training & Tutorials/03- Query Data.md +++ /dev/null @@ -1,119 +0,0 @@ ---- -title: "Querying the Data" -description: "A quick run through for querying the data" -layout: page -source_path: "training/03- Query Data.md" ---- - - -# Querying the Data - -Once the network is running, there are two types of data sets of interest to the user: - -* **Metadata** — the content stored in the blockchain: information about the agents in the network and their role, - which tables and databases are part of the network, and where that data resides. -* **The actual data** — retrieved by "connecting" from a query node to the node(s) that hold the data, using a SQL - command. - -## Metadata Commands - -**`get data nodes`** — shows a clean view of what data exists and where it resides. - -```anylog -get data nodes -``` - -> **Note:** the example below uses anonymized company/node names and placeholder IPs; a real network's output will -> look the same in structure, just with your own deployment's values. - -``` -Company DBMS Table Cluster ID Cluster Status Node Name Member ID External IP/Port Local IP/Port Main Node Status ---------- ------------ --------------- -------------------------------- -------------- ----------------- --------- ------------------- ------------- ---- ----------- -Acme Co monitoring syslog a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4 active site-operator-1 101 10.0.1.11:32148 + active - f6e5d4c3b2a1f6e5d4c3b2a1f6e5d4c3 active site-operator-2 102 10.0.1.12:32148 + active -Acme Co telemetry device_logs a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4 active site-operator-1 101 10.0.1.11:32148 + active - f6e5d4c3b2a1f6e5d4c3b2a1f6e5d4c3 active site-operator-2 102 10.0.1.12:32148 + active -Contoso monitoring syslog 9a8b7c6d5e4f9a8b7c6d5e4f9a8b7c6d active plant-operator-1 201 10.0.2.21:32148 + active - (backup) active plant-operator-1-bkup 202 10.0.2.22:32148 - active -Contoso process_data turbine_pitch 9a8b7c6d5e4f9a8b7c6d5e4f9a8b7c6d active plant-operator-1 201 10.0.2.21:32148 + active -... -``` - -* **`blockchain get *`** — view all metadata on the blockchain. -* **`blockchain get [policy type]`** — list all metadata of a given policy type (e.g. `operator`, `cluster`, `table`). - -A more thorough set of documentation regarding querying the Blockchain / Metadata can be found here. - ---- - -## Query - -Since data is distributed across the network, prefix the SQL command with `run client ()` so the node knows to send -the request against the network rather than run it locally. The network automatically determines where the data -resides and extracts the results. (The parentheses in `run client ()` are reserved for optionally targeting specific -nodes/clusters — leave them empty to query the whole network.) - -**Supported functionality:** - -| Function | Description | -|---|---| -| raw | Specific column(s), or all columns via `*` | -| `min` | Minimum value of a column | -| `max` | Maximum value of a column | -| `avg` | Average value of a column | -| `count` | Row count | -| `WHERE` | Filter rows by condition | -| `ORDER BY` | Sort results | - -### Examples - -**Raw select:** - -```anylog -run client () sql my_dbms "SELECT timestmap, device_id, temperature, humidity FROM sensor_data" -``` - -``` -timestamp device_id temperature humidity --------------------- ---------- ------------ -------- -2026-07-24 09:00:01 sensor-01 21.4 48.2 -2026-07-24 09:00:01 sensor-02 22.1 46.9 -2026-07-24 09:00:02 sensor-01 21.5 48.0 -``` - -**Aggregates + `WHERE` + `ORDER BY`:** - -```anylog -run client () sql my_dbms "SELECT device_id, min(temperature), max(temperature), avg(temperature), count(*) -FROM sensor_data WHERE device_id = 'sensor-01' ORDER BY device_id" -``` - -``` -device_id min(temperature) max(temperature) avg(temperature) count(*) ----------- ----------------- ----------------- ----------------- -------- -sensor-01 19.8 23.6 21.42 1440 -``` - -### Via REST - -```shell -# cURL GET — note `destination: network` replaces `run client ()` -curl -X GET http://[Node ip]:[Node port] \ - -H 'command: sql my_dbms "SELECT * FROM sensor_data"' \ - -H "AnyLog-Agent: AnyLog/1.23" \ - -H "destination: network" - -# cURL POST -curl -X POST http://[Node ip]:[Node port] \ - -H "Content-Type: application/json" \ - -H "AnyLog-Agent: AnyLog/1.23" \ - -H "destination: network" \ - -d '{"command": "sql my_dbms \"SELECT * FROM sensor_data\""}' -``` - -A more thorough set of documentation regarding querying the SQL can be found here. \ No newline at end of file diff --git a/03- Training & Tutorials/04- deployment-process.md b/03- Training & Tutorials/04- deployment-process.md deleted file mode 100644 index d2938904..00000000 --- a/03- Training & Tutorials/04- deployment-process.md +++ /dev/null @@ -1,229 +0,0 @@ ---- -title: Deployment Integration -description: The process by which events occur and decisions are made at deployment in order to provide the initiation scripts. -layout: page -visibility: public -version: open source -tags: -- install -- integration ---- - - -# Deployment Integration - -The following document describes the relationship between the compiled source code (ie AnyLog agent) and deployment-scripts -which defines which services to run on a given AnyLog agent. - -As such the document covers a few things: - -1. The break between AnyLog and deployment-scripts -2. How to deploy AnyLog (simple) -3. The connection between AnyLog & `deployment-scripts` -4. Overriding the Entry Point -5. Patches & Version Updates - - - -The document discusses AnyLog, but the same logic also works when deploying EdgeLake. - -## The break between AnyLog and deployment-scripts - -A running node is really two independent pieces: - -``` -┌─────────────────────────────────┐ -│ AnyLog / EdgeLake binary │ ← the engine — compiled, never edited -├─────────────────────────────────┤ -│ deployment-scripts (git repo) │ ← the configuration — yours to read, fork, or customize -└─────────────────────────────────┘ -``` - -- **The binary** is the compiled AnyLog/EdgeLake runtime. It knows how to execute commands and `.al` script files, but -it doesn't know anything about *what* to run on its own. -- **`deployment-scripts`** is what tells the binary what to actually do — which services to start, which database -to connect, which policies to publish. It's plain text, not compiled, and it's designed to be read, forked, or edited. - -This split is why you never need to rebuild the AnyLog image just to change how a node behaves — you only need to -change the scripts it's pointed at. - -## Initial Deployment - -1. User clones a repo called `docker-compose` (or `K8s-compose`) containing a set of configuration files. -2. User updates the configuration files as needed. - **Key configs:** - 1. `LICENSE_KEY` - 2. `LEDGER_CONN` - 3. `NODE_TYPE` - - A user can skip the full config file (i.e. skip `make`) and just run `docker run` with the 3 environment variables directly. -3. Deploy AnyLog / EdgeLake using `make` or `docker run`. - -## What Happens in the Docker Container - -AnyLog is the actual source code. To configure its services, AnyLog agents use configuration policies driven by a set of -scripts called **deployment-scripts**. For the internals of how those policies and scripts actually communicate (and -the `process` vs. `thread` execution model behind them), see Deployment Scripts — Integration Reference. - -By default, the docker image ships with the `main` deployment-scripts already baked in. Using the environment params -`DEPLOYMENTS_REPO` and `DEPLOYMENTS_BRANCH`, a user can point at a different set of deployment-scripts instead. - -Resolution happens in two stages — **build time** (compose file generation, driven by `make`) and, for one of the four -options below, **runtime** (inside the running agent container). - -### Stage A — Build script (compose file generation) - -The build script picks one of four modes, in order: - -| # | Condition | Behavior | -|---|-----------|----------| -| 1 | `DEPLOYMENTS_REPO`/`DEPLOYMENTS_BRANCH` unset, **or** explicitly set to the default (`https://github.com/AnyLog-co/deployment-scripts` @ `main`) | **Built-in default.** Enables the `${CONTAINER_NAME}-local-scripts` named volume. On first creation, Docker seeds it from the scripts baked into the image; after that it's just a persistent volume. | -| 2 | `DEPLOYMENTS_REPO` is set and is an existing **local directory** on the host | **Host bind mount.** Compose is rewritten to mount that directory directly at `/app/deployment-scripts`, and the named volume + init-service references are stripped out entirely. Fully static — no cloning, no copying. | -| 3 | `DEPLOYMENTS_REPO` is an `http://` or `https://` **URL** | **Reclone at startup.** No volume is mounted at all at build time; the agent clones the repo itself when it starts (see Stage B below). | -| 4 | `DEPLOYMENTS_REPO` is set but matches none of the above (e.g. a docker image reference) | **Secondary deployment-scripts container.** A one-shot helper service is injected into the compose file: `image: ${DEPLOYMENTS_REPO}:${DEPLOYMENTS_BRANCH}`, which copies `/app/deployment-scripts` out of that image into the shared `${CONTAINER_NAME}-local-scripts` named volume, then exits. The main service waits on it (`condition: service_completed_successfully`) before starting. | - -So if you've simply created your own repo with the same structure, **option 3** (or **option 2**, if it's a local -directory) is usually the one you want — point `DEPLOYMENTS_REPO`/`DEPLOYMENTS_BRANCH` at it and the rest of the -machinery (volumes, cloning) is handled for you. - -> The same build script also has an unrelated block detecting `DOCKER_SOCKET` and setting `DOCKER_GID` (for Docker-in-Docker -> support). That's a separate concern from deployment-scripts selection. - -**Important caveat — this entire table only applies when deploying via `make`.** If a user runs `docker run` directly -instead (skipping `make`, using just the 3 core env vars), none of this build-time logic ever executes. It becomes the -user's responsibility to configure everything themselves: -- volumes (highly recommended) -- branch selection for deployment-scripts, if changing from default at all -- anything else this script would otherwise have handled - -**Why "a pre-existing volume takes precedence"** (the original point 0): Options 1 and 4 both mount the *same* named -volume, `${CONTAINER_NAME}-local-scripts`. Docker does not recreate or repopulate an existing named volume just because -`make` reruns — it's reused as-is. So if that volume already exists from a prior run, it keeps whatever content it has -regardless of what `DEPLOYMENTS_REPO`/`DEPLOYMENTS_BRANCH` currently say. This applies specifically to Options 1 & 4 -(the two modes sharing that volume), not to Options 2 or 3. - -### Stage B — Runtime (inside the agent container, on startup — Option 3 only) - -This clone logic only runs when Stage A resolved to **Option 3** (URL, reclone-at-startup). Options 1, 2, and 4 are all -static by the time the container starts — no git operations happen for them inside the agent. - -```bash -if [[ CURRENT_REMOTE == DEPLOYMENTS_REPO ]] && [[ CURRENT_BRANCH == DEPLOYMENTS_BRANCH ]]; then - # already matches — skip reclone -elif git ls-remote --exit-code --heads "${DEPLOYMENTS_REPO}" "${DEPLOYMENTS_BRANCH}"; then - # branch exists on remote — wipe LOCAL_SCRIPTS and reclone fresh -else - # branch/repo not reachable — error, leave existing LOCAL_SCRIPTS untouched -fi - -if [[ ! -d ${LOCAL_SCRIPTS} ]]; then - # still missing — hard fail, agent won't start -fi -``` - -- If the currently-recorded remote **and** branch already match `DEPLOYMENTS_REPO`/`DEPLOYMENTS_BRANCH` → skip re-cloning, reuse what's on disk. - ⚠️ **This check only compares repo + branch name, not commit hash.** See the Patches section below — this is the mechanism that determines whether a customer picks up a fix automatically. -- Else, if that branch actually exists on the remote → delete the existing `LOCAL_SCRIPTS` directory and clone fresh. -- Else (repo/branch unreachable) → log an error and leave whatever's currently in `LOCAL_SCRIPTS` alone. -- If `LOCAL_SCRIPTS` still doesn't exist after all that → hard fail (`exit 1`). The agent will not start without deployment-scripts present. - -*(The "download the repo as a docker container, `DEPLOYMENTS_BRANCH` as the tag" behavior from the original description -is real — it's Option 4 above, implemented at build time as a helper compose service rather than inside this runtime script.)* - ---- - -## Overriding the Entry Point - -Pointing `DEPLOYMENTS_REPO`/`DEPLOYMENTS_BRANCH` at your own repo (Stage A above) determines *which copy* of -deployment-scripts loads — but the container's `ENTRYPOINT` (`/app/deploy_anylog.sh`) separately hardcodes the path it -runs once that repo is in place: - -```bash -${ANYLOG_PATH}/${APP_NAME} process $ANYLOG_PATH/deployment-scripts/node-deployment/main.al -``` - -This path is fixed in the entrypoint script itself — it doesn't follow `$LOCAL_SCRIPTS`/`$ANYLOG_PATH` the way -`main.al`'s *internal* references do. So if your own repo's entry script isn't named `main.al`, you have two options: - -**Simplest — no override needed:** name (or copy) your custom entry script to `node-deployment/main.al` within your -repo. The entrypoint picks it up with zero further changes, regardless of which Stage A option loaded your repo. - -**If you need a genuinely different filename**, override the container's entrypoint so it calls your script's actual -path instead of the hardcoded one. This bypasses `deploy_anylog.sh`'s other setup steps (OpenBao secret fetch, -version/arch detection, deployment-scripts resolution, the Kubernetes `/etc/hosts` patch, etc.) — only do this if -you've already handled those yourself, or if you maintain your own modified copy of `deploy_anylog.sh` that still does -them before calling your script. - -*Docker Compose:* -```yaml -services: - anylog-node: - image: your-anylog-image - environment: - - ANYLOG_PATH=/app - - APP_NAME=anylog_v1.0.0_x86_64 # or however your image derives this - entrypoint: ["/bin/bash", "-c"] - command: - - > - ${ANYLOG_PATH}/${APP_NAME} process ${ANYLOG_PATH}/deployment-scripts/node-deployment/my_custom_main.al -``` - -*Docker Run:* -```bash -docker run \ - --entrypoint /bin/bash \ - your-anylog-image \ - -c '${ANYLOG_PATH}/${APP_NAME} process ${ANYLOG_PATH}/deployment-scripts/node-deployment/my_custom_main.al' -``` - -Both forms replace `deploy_anylog.sh` entirely for that container, so double-check you don't need anything it would -otherwise have done for you before reaching this line. - ---- - -## Patches & Version Updates - -Standard process for any reported issue: - -1. Understand the issue. -2. Check whether it still exists in the latest code. -3. Fix it. -4. Ask the customer to update to the newer build. - -What "update to the newer build" actually does **depends on where the bug lives**, because a docker container bundles -two different things — the AnyLog/EdgeLake agent binary and the deployment-scripts — and they update through different -mechanisms. - -### A. Bug is in the agent itself -Customer pulls the new image and restarts. The agent code updates immediately, since the image itself is what's being -replaced. Deployment-scripts are untouched by this step — they live in a separate clone/volume/mount, not something baked -fresh into every container restart. - -### B. Bug is in the deployment-scripts -This depends on which of the four build-time options (see Section 2) the customer is running: - -- **Option 1 — default, in-image `main` scripts:** a new image pull brings fixed scripts along *only if* the `${CONTAINER_NAME}-local-scripts` volume doesn't already exist. If it does (i.e. this isn't the customer's first run), the volume is reused as-is and the fix never lands — the customer needs to remove that volume (and ideally the associated configuration policy) so it gets reseeded from the new image. -- **Option 2 — local directory (bind mount):** no clone/copy step exists at all here. The customer has to update that host directory themselves; pulling a new agent image has zero effect on it. -- **Option 3 — URL, reclone at startup:** ⚠️ **if the fix is pushed to the *same* branch the customer is already tracking, a restart will NOT pick it up.** The runtime match check only compares repo + branch strings — it has no concept of "is there a newer commit on this branch." Since repo and branch are unchanged, the agent treats its existing clone as current and skips re-cloning. - - Workarounds until/unless this is fixed: point at a new branch or tag for the patched scripts, manually delete the `LOCAL_SCRIPTS` directory/volume so nothing matches and a reclone is forced, or `docker exec` in and `git pull` manually. -- **Option 4 — secondary deployment-scripts container:** the helper service runs once and copies into the shared named volume, then exits — it doesn't run again on a plain restart. Same volume-reuse caveat as Option 1: if `${CONTAINER_NAME}-local-scripts` already exists, a new image (even with a bumped tag) won't get re-copied in unless the volume is cleared first (or the compose file is regenerated with a config that forces recreation). - -**Practical takeaway for triage:** when a customer reports a deployment-scripts bug, first ask which of the four options -they're in — that determines whether "just pull the new image" is actually sufficient, or whether they also need to clear -the `local-scripts` volume, bump a branch/tag, or manually sync a local directory. And if they're deploying via bare -`docker run` instead of `make`, none of this is automatic in the first place — walk through their setup manually. \ No newline at end of file diff --git a/03- Training & Tutorials/05- deployment-scripts.md b/03- Training & Tutorials/05- deployment-scripts.md deleted file mode 100644 index 44267b10..00000000 --- a/03- Training & Tutorials/05- deployment-scripts.md +++ /dev/null @@ -1,247 +0,0 @@ ---- -title: Deployment Scripts — Integration Reference -description: Internals reference — why deployment-scripts exists, how policies and scripts communicate, and the execution model behind them. -layout: page -visibility: public -version: open source -tags: -- getting-started -- integration ---- - - -# AnyLog / EdgeLake Deployment Scripts — How They Work - -*A reference for understanding the `deployment-scripts` repository: why it exists, how policies and scripts communicate, -and the execution model behind them.* - -This document covers: - -1. Why This Repository Exists -2. The Deployment Flow -3. How Policies and Scripts Actually Communicate -4. `process` vs. `thread` — the Execution Model - -## Why This Repository Exists - -`deployment-scripts` is the **default deployment toolkit for AnyLog and EdgeLake nodes**. It contains the scripts that -set up, configure, and manage a node so that a user doesn't have to hand-write deployment logic every time. - -Two things make this repo work the way it does: - -- **It's downloaded automatically** when a node runs via Docker, Podman, or Kubernetes — the container pulls this repo in and executes it as part of startup. -- **It's config-driven.** A user doesn't write deployment code. They set environment variables and/or a policy, and the scripts in this repo translate that into a running, networked, database-backed AnyLog/EdgeLake node. - -The core idea: separate what an agent should be (declared once, as config) from how that gets built (scripts, written -once, reused everywhere). - -For the full build-time/runtime mechanics of *how* this repo actually gets loaded into a container (default image, host -bind mount, reclone-at-startup, or a secondary container), how to point at your own repo, and how to override the entry -point, see [Deployment Integration](04- deployment-process.md). - -**Repository Structure** -``` -├───node-deployment <- Core deployment orchestration (start here) -│ ├───main.al <- Entry orchestrator: env→params, call policy generator, end-of-script check -│ ├───set_params.al <- Converts $ENV_VARS into !anylog_variables, with defaults -│ ├───connect_blockchain.al <- Connects to blockchain platform (master or optimism) and starts sync -│ ├───local_script.al <- Empty placeholder hook for user custom logic (runs if DEPLOY_LOCAL_SCRIPT=true) -│ ├───blockchain.md <- Docs -│ ├───database <- Database setup and initialization -│ │ ├───deploy_database.al <- Dispatcher: routes to the right DB setup scripts based on !node_type -│ │ ├───configure_dbms_blockchain.al <- Creates the `blockchain` logical db + `ledger` table (master nodes) -│ │ ├───configure_dbms_almgm.al <- Creates the `almgm` logical db + `tsd_info` table (publisher/operator) -│ │ ├───configure_dbms_operator.al <- Connects the operator's default dbms; sets up partitioning if enabled -│ │ ├───configure_dbms_monitoring.al <- Sets up `monitoring` db partitioning (12h partitions, 36h retention) -│ │ ├───configure_dbms_nosql.al <- Connects a NoSQL (Mongo) db and enables the blobs archiver -│ │ ├───configure_blob_storage.al <- Validates blob-storage config, connects storage, enables blobs archiver -│ │ ├───connect_dbms_sql.al <- Generic SQL connection helper (PostgreSQL / SQLite) -│ │ ├───connect_dbms_nosql.al <- Generic NoSQL connection helper (MongoDB) -│ │ ├───connect_dbms_objstore.al <- Object storage connection helper (Akave / S3 / MinIO) -│ │ └───connect_dbms_system_query.al <- Connects the `system_query` db (query / system-query-enabled nodes) -│ └───policies <- Policy declaration scripts (configuration, cluster, node, license, blockchain...) -│ ├───config_policy.al <- THE dispatcher — builds, publishes, and applies the node's full config policy -│ ├───config_policy_networking.al <- Fills in networking fields (ip/port/rest/broker) into the config policy -│ ├───cluster_policy.al <- Declares/publishes the cluster policy (operator nodes) -│ ├───node_policy.al <- Declares/publishes the node's own identity policy -│ ├───validate_node_policy.al <- Checks whether a matching node policy already exists on the blockchain -│ ├───blockchain_policy.al <- Declares a blockchain-info policy (live/optimism blockchain use) -│ ├───license_policy.al <- Declares/activates the license policy (AnyLog only — skipped for EdgeLake) -│ ├───hzn_policy.al <- Declares an Open Horizon (HZN) policy, if HZN_* env vars are set -│ └───publish_policy.al <- Generic helper — signs, validates, publishes any policy to the blockchain -├───data-generator <- Scripts for ingesting sample/test data -├───gRPC <- Sample gRPC connections, protocol defs, compilation utilities -├───sample-scripts <- Scripts for receiving data from third-party applications -├───southbound-industrial <- Sample Ingest from OPC-UA, Modbus, EtherIP, and other industrial protocols -├───southbound-monitoring <- Sample Node/Docker/syslog monitoring collection scripts -├───southbound-video-streaming <- Sample Video stream ingestion scripts -├───test-network-local-scripts <- Scripts used by the local test network -└───aggregations <- Sample Scripts to configure streaming data aggregation -``` - -`main.al` itself only does three things -* convert env vars to params -* call the policy generator -* do an end-of-script check. - -It never directly touches any of the actual deployment-script processes that define databases and/or services for -a given node. Those actions get reached indirectly, because the policy generator (`policies/config_policy.al`) defines and -applies / executes a node-type-specific list of instructions. - -## The Deployment Flow - -Everything starts with one command: - -```bash -# AnyLog -./anylog_node process deployment-scripts/node-deployment/main.al - -# EdgeLake -./edgelake_node process deployment-scripts/node-deployment/main.al -``` - -When main initiates it does the following steps: -1. enables `echo queue` and disables authentication - * `echo queue` is a print like logic that stores content to a `get event log` like structure rather than print to screen. The default deployment process uses it in order to record errors that might not have been caught by the error log - * `authentication` is disabled at start up but maybe enabled at a later point through configuration policies. This is so the node can easily accept an initial copy of the blockchain and set any relationships it may need without needing to worry about public / private keys -2. define relative paths to be used through out (ex. `!local_scripts`) -3. convert environment variables (`-e` in docker) to AnyLog variables (found in dictionary) -4. define the config policy for the node if one does not exists. - -## How Policies and Scripts Actually Communicate - -This is the part that's easy to misread as "the policy contains the deployment code." It doesn't. Here's the real mechanism. - -A **policy** is a piece of declarative JSON, published to the AnyLog blockchain/ledger, that describes what a node *is*. -Note the example below uses AnyLog's own single-quoted policy syntax rather than strict JSON (which requires double -quotes) — this is the format AnyLog commands actually accept, not a typo: - -```json -{'config' : { - 'name' : 'operator-configs', - 'company' : 'My Company', - 'node_type' : 'operator', - 'ip' : '!external_ip', - 'port' : '!anylog_server_port.int', - 'rest_port': '!anylog_rest_port.int', - 'broker_port': '!anylog_broker_port.int', - 'tcp_threads': '!tcp_threads.int', - 'rest_threads': '!rest_threads.int', - 'broker_threads': '!broker_threads.int', - 'script' : [ - 'process !local_scripts/connect_blockchain.al', - 'process !local_scripts/policies/cluster_policy.al', - 'process !local_scripts/policies/node_policy.al', - 'process !local_scripts/database/deploy_database.al', - 'run scheduler 1', - 'run streamer', - 'if !enable_ha == true then run data distributor', - 'if !operator_id and !blockchain_source != master then run operator where ...', - 'process !anylog_path/deployment-scripts/southbound-monitoring/config_monitoring_policy.al', - 'process !anylog_path/deployment-scripts/southbound-industrial/industrial_policy.al', - 'if !deploy_local_script == true then process !local_scripts/node-deployment/local_script.al', - 'if !is_edgelake == false then process !local_scripts/policies/license_policy.al' - ], - 'id' : '2e54c04ce4e1241d41e68cbbd31a2469', - 'ledger' : 'global' -}} -``` - -Notice that within the config policy we are defining the relative network configurations (`'!anylog_server_port.int'`) -as opposed to hardcoded (`32148`), followed by the actual databases and services need for the given instance type. - -The reason the config policy uses relative network configuration is so that the same policy can be reused across multiple -instances of the same type. - -The communication works like this: - -1. **The policy's `script` field is a list of instructions, not implementation.** Each line is either a `process` call to -a `.al` file, or a direct AnyLog command (`run scheduler`, `run streamer`, `run operator where ...`). The choice between -a script and a raw command depends on whether there are dependencies to resolve first — for example, connecting to a -logical database where the user defines the db type — versus a standalone command that's simply required for this type -of AnyLog agent (e.g. `run operator`). - -2. **Every `!variable` referenced in the policy** (`!anylog_server_port`, `!operator_id`, `!enable_ha`, `!deploy_local_script`, etc.) -was already set by `set_params.al` in the previous step, and could be unique per Agent / container. The policy doesn't -compute these values — it just consumes them. - -3. **Conditionals live in the policy, logic lives in the scripts.** Lines like `'if !enable_ha == true then run data distributor'` -mean the policy decides *whether* a capability turns on, but the actual mechanics of what "run data distributor" does are -internal to AnyLog/EdgeLake or defined in the referenced `.al` file — never inlined into the policy itself. - -4. **The policy is data first.** Because it's stored on the blockchain as part of our metadata, ie a ledger record, it needs -to stay small, inspectable, and diffable — a list of strings and key/value pairs, not an executable program. This is -*why* it calls out to scripts rather than embedding them: the ledger holds intent, the filesystem holds mechanism. - -The "communication" between a policy and the scripts is really just **variable substitution + ordered dispatch**: -`set_params.al` populates the variable namespace, the policy reads that namespace to decide what to enable and in what -order, and each `process` line in the policy's `script` array hands control to a dedicated file that knows how to carry -out one piece of that intent (database setup, cluster/node policy declaration, monitoring, industrial ingestion, -licensing). - -## `process` vs. `thread` — the Execution Model - -The reason this repository cares about `process` vs. `thread` at all isn't academic — it matters the moment you go from -*using* the default deployment scripts to *writing or modifying your own* (see -[Deployment Integration](04- deployment-process.md)). Every script you add has to make the same choice every -built-in script makes: does this need to block until it's done, or can it run alongside everything else? - -Both commands do the same basic thing — **run the commands in a script file** — but they differ in *how* that execution -relates to the caller: - -``` -help process - Usage: process [path and file name] - Explanation: Process the commands in the specified file - -help thread - Usage: thread [path and file name] - Explanation: Initiate a new thread to process the commands in the specified file -``` - -| | `process` | `thread` | -|---|---|---| -| Execution | Runs in the **current** execution context | Spawns a **new thread** to run the file | -| Blocking? | Yes — the caller waits for it to finish (synchronous) before moving to the next line | No — the caller continues immediately; the file runs concurrently (asynchronous) | -| Use case | Ordered setup steps where each step depends on the previous one having finished | Independent, long-running, or parallel work that shouldn't block the main flow | - -### Why the deployment scripts use `process` almost everywhere - -Look back at `main.al` and every policy `script` array — they're built almost entirely out of `process` calls. That's -not incidental: **deployment is a strictly ordered dependency chain.** - -- The AnyLog agent cannot define itself as an operator before it knows which cluster the instance would ultimately belong to. -- The AnyLog agent cannot run the database deployment logic before `set_params.al` has populated the variables it references. -- You can't enable monitoring or industrial ingestion meaningfully before the node's core identity (ports, database, -blockchain connection) exists. - -If these were dispatched with `thread`, they'd all fire off concurrently with no guarantee of ordering — and the second -step might run before the first step's variables or policies exist, breaking the deployment. `process` guarantees -"step 2 only starts once step 1 is fully done," which is exactly the contract this repo depends on. - -`thread` becomes the right tool when you *want* concurrency — e.g., running two independent, long-lived monitoring or -ingestion loops side-by-side without one blocking the other. That's a pattern you'd reach for in custom/local scripts -for things like parallel MQTT listeners, rather than in the core deployment sequence itself. - -**Rule of thumb:** if step B needs something step A produced (a variable, a policy, a database), use `process`. If step A -and step B are independent and both need to run continuously/concurrently, `thread` is the right call. - -There's an exception to the rule though — when running a process in a scheduled manner, as the deployment-scripts do -with monitoring, each command runs in its own thread otherwise the CLI would not be accessible to the user most of the time. \ No newline at end of file diff --git a/03- Training & Tutorials/06- Nodes.md b/03- Training & Tutorials/06- Nodes.md deleted file mode 100644 index c9dbb03b..00000000 --- a/03- Training & Tutorials/06- Nodes.md +++ /dev/null @@ -1,295 +0,0 @@ ---- -title: "Nodes: The start-up commands" -description: "a basic understanding of using AnyLog" -layout: page -source_path: "training/06- Nodes.md" ---- - - -The following document provides directions on the different node types and the critical component needed in each one for -them to actually be configured properly. - -> **Note:** in a real deployment, all the commands below are normally generated and run for you automatically — -> driven by your dotenv configuration and the node's configuration policy (see the Deployment Integration and -> Deployment Scripts docs). They're shown here manually so you understand what each node type actually needs -> under the hood. The < > notation is used to merge multiple lines into a single line when executing long commands on -> the CLI. It's used in the examples below for two reasons: -> 1. To demonstrate its usage. -> 2. To keep the output clean and readable. - -## Contents - -1. General Process -2. Common Setup - * Network Setup - * Blockchain Sync -3. The Master / Metadata Node -4. Query Node -5. The Operator Node -6. The Publisher Node - -## General Process - -1. AnyLog agent starts. -2. A configuration policy, based on the node type, is created — if one does not already exist. -3. Network is configured. -4. Logical databases are defined. -5. Node policy gets defined — for operator nodes this includes the cluster policy. -6. Blockchain sync is enabled. -7. Scheduler is enabled. -8. AnyLog-agent-type-specific configs are enabled. - -> **Colocation note:** any combination of AnyLog services / logical databases can reside on the same physical -> machine — **except** `run operator` and `run publisher`, which cannot run on the same node as each other. Keep -> this in mind when planning node placement, before you get into the per-node-type sections below. - -## Common Setup - -Every node type — Metadata Manager (Master), Query, Operator, and Publisher — shares the same network setup and blockchain sync -mechanism (just with different sync intervals). Both are covered here once; the per-node-type sections below only -show what's *additional* for that node type. - -### Network Setup - -```anylog - - - -``` - -> **Exception:** the Query node uses `timeout = 90` instead of `timeout = 30`, to accommodate longer-running queries -> across the network. This is called out again in the Query Node section below. - -### Blockchain Sync - -Every node keeps an accessible local copy of the blockchain — but how *fresh* that copy needs to be depends on the -node type: - -* **Master & Query** — sync every **60–90 seconds**. These nodes route queries and resolve policy in real time, so - they need a near-current view of the ledger. -* **Operator & Publisher** — sync every **~5 minutes**. These nodes primarily ingest and store data; they don't need - as tight a refresh on the ledger to do their job. - -```anylog -# Master / Query - - -# Operator / Publisher - -``` - -## The Master / Metadata Node - -The metadata node is our blockchain emulator, and requires 2 things beyond the common setup -(using the 60–90 second sync interval): - -* `blockchain` Database + `ledger` table -```anylog - - -create table ledger where dbms=blockchain -``` - -AnyLog has built-in table definitions for: -* `blockchain.ledger` — the blockchain emulator storage layer. -* `almgm.tsd_info` — the metadata / hash value record of the data coming in. -* `table` policies on the blockchain. - -## Query Node - -This is a node dedicated to querying data across the network. Beyond the common setup (remember: -`timeout = 90` for network setup, and the 60–90 second sync interval), it needs: - -* `system_query` logical database — where results get aggregated. We recommend using SQLite (in-memory), unless -the Northbound Services have a Postgres direct connection as opposed to using REST. - -```anylog - -``` - -## The Operator Node - -This node is dedicated to storing the actual data coming in from devices and sensors. Beyond the -common setup (using the ~5 minute sync interval), it needs: - -* Define Cluster + Blockchain policy — unlike the other nodes, this is a **must** so that the network knows where the data -resides and how to reach it. - -> **Note:** the example below uses a fictional company/IP for illustration. `True`/`False` (capitalized) is AnyLog's -> own accepted policy syntax, not a typo of JSON's lowercase `true`/`false`. - -```json -{"cluster" : { - "company" : "Acme Co", - "name" : "acme-site-1", - "status" : "active", - "id" : "353495722981c88e3a5e4ffff486075e", - "date" : "2026-07-18T19:49:32.437491Z", - "ledger" : "global" -}}, -{"operator" : { - "name" : "site-operator-1", - "company" : "Acme Co", - "hostname" : "acme-site-1", - "ip" : "10.0.1.11", - "port" : 32148, - "rest_port" : 32149, - "broker_port" : 32150, - "cluster" : "353495722981c88e3a5e4ffff486075e", - "main" : True, - "loc" : "32.7767, 96.7970", - "country" : "US", - "state" : "TX", - "city" : "Dallas", - "id" : "fca91d1eedcb2472a02954be6e276da8", - "date" : "2026-07-18T19:49:37.501204Z", - "member" : 140, - "ledger" : "global" -}} -``` - -* Connect to the logical database where data will ultimately be stored: -```anylog - -``` - -> We recommend defining partitioning of the data for better query performance and data maintenance. - -* Connect to the `almgm.tsd_info` logical database + table. This keeps a record of the files coming in, for HA and to -remove replication of data (based on file hash): -```anylog - -``` - -* Archiver for blob data — this can be used with or without a NoSQL logical database: - -```anylog -# if a NoSQL logical database is set to true, switch True/False between dbms + folder. - - -``` - -* Set buffer size — data buffer size before storing into AnyLog: - -```anylog - - -run streamer -``` - -* Enable HA — ⚠️ **Enterprise feature** — whether a secondary operator -is planned, or a primary already exists, it's good practice to enable HA now. That way, adding an additional operator -later is as simple as joining it to the network: - -```anylog -run data distributor -run data consumer where start_date=-30 days -``` - -* Enable the Operator service: - -```anylog - -``` - -## The Publisher Node - -⚠️ **Enterprise feature** - -This is a unique node type — it acts as a mediator between the device/sensor and the operator nodes, allowing data -from one device to be processed across multiple operators using a round-robin approach, where each file is sent to a -different operator (ideally within the same cluster group). - -**Why use a Publisher?** - -* **Scale** — round-robin spreads load across operators so no single one becomes a bottleneck under high-volume or high-frequency data. -* **Simplicity for the device** — the device only needs to know one destination (the publisher), not the whole cluster topology or which operator is currently healthy. -* **Buffering** — the publisher aggregates incoming messages into files before distributing them, smoothing bursty traffic into steady batches. -* **Policy-based routing** — beyond round-robin, `set data distribution` (see below) can force specific tables to specific operators — e.g. table A always to operator group 1, table B to operator group 2 — rather than spreading evenly. -* **Reduced attack/complexity surface** — only the publisher needs to be reachable by external devices; operators can sit behind it. -* **Cleanup/lifecycle management** — the publisher's `delete_json` / `delete_sql` flags also manage what happens to source files/records after a successful distribution. - -Beyond the common setup (using the ~5 minute sync interval), it needs: - -> **To verify:** if the Publisher receives southbound traffic directly (MQTT/REST push, syslog, etc.) rather than -> picking up files already dropped to disk, it likely also needs `run message broker` — the same as an Operator -> accepting syslog (see the Syslog Integration doc). Confirm whether that step belongs here before publishing. - -* Connect to the `almgm.tsd_info` logical database + table. This keeps a record of the files coming in, for HA and to -remove replication of data (based on file hash): - -```anylog - -``` - -* Run publisher: - -```anylog -# sample file: `my_db.table3.0.0.json` - - -``` - -* Definition on how to distribute the data from the publisher into different operators: - -```anylog - -``` \ No newline at end of file diff --git a/04- Southbound Interfaces/01- Southbound Interfaces.md b/04- Southbound Interfaces/01- Southbound Interfaces.md deleted file mode 100644 index 00cd607e..00000000 --- a/04- Southbound Interfaces/01- Southbound Interfaces.md +++ /dev/null @@ -1,88 +0,0 @@ ---- -title: Data Ingestion (Southbound) -description: Overview of southbound data ingestion in AnyLog — connectors, mapping, file pipeline, and prerequisites. -layout: page ---- - - -AnyLog receives data from edge devices, sensors, and applications through a set of **southbound connectors**. -All connectors ultimately produce JSON files that flow through a common pipeline: Watch Directory → Operator → Local Database. - -This page covers the pipeline, prerequisites, and mapping. For connector-specific configuration see the individual pages -linked below. - -> **See also:** Northbound Interfaces (data egress) — the counterpart to this page, covering how AnyLog forwards -> data onward once it's ingested and stored. - -### The ingestion pipeline - -``` -Data Source - │ - ▼ -Southbound Connector -(MQTT, REST PUT, Kafka, gRPC, PLC/OPC-UA, Syslog, ...) - │ - ▼ -Internal Buffers / Streamer -(aggregates events, flushes on time or volume threshold) - │ - ▼ -Watch Directory (JSON files) - │ - ▼ -Operator -(reads files, maps JSON → SQL, inserts to local DB) - │ - ▼ -Local Database (SQLite / PostgreSQL) -``` - ---- - -## The connectors - -### Built-in Southbound connectors -* REST - * **PUT** — data is stored as-is. - * **POST** — requires a message client to translate the payload before it's stored. -* MQTT / Kafka — utilizes AnyLog's built-in MQTT / Kafka message broker. -* OPC-UA -* Modbus -* DNP3 -* EtherIP -* gRPC -* Video - -### Third-Party Southbound Connectors - -AnyLog focuses on the data management side rather than pulling data directly from every device and protocol. Where -we don't have a built-in connector, we recommend placing a third-party app between the device and AnyLog to bridge -the gap — the app talks to the device/sensor, and forwards the result to AnyLog over REST, MQTT, or Kafka. - -* External MQTT / Kafka -* EdgeX -* Node-RED -* Telegraf - -### Node Monitoring - -Node and container health metrics (CPU, memory, disk, container status) are themselves ingested through this same -southbound pipeline — which is why monitoring configuration lives here rather than under Extended Services. - -* Node Monitoring -* Docker Monitoring -* Syslog - -## Dummy Data - -We provide an MQTT connection to an array of different data sets. For details visit Data Ingestion -or Live Data Generator to run locally. \ No newline at end of file diff --git a/04- Southbound Interfaces/02- Direct Connectors/01- REST.md b/04- Southbound Interfaces/02- Direct Connectors/01- REST.md deleted file mode 100644 index 1f5c0ab9..00000000 --- a/04- Southbound Interfaces/02- Direct Connectors/01- REST.md +++ /dev/null @@ -1,133 +0,0 @@ ---- -title: Publishing data via REST -description: How to insert data into AnyLog via REST — PUT vs. POST, mapping policies, and support commands. -layout: page ---- - - -The following document provides directions on how to insert data via REST. - -## Setting Up the Node - -The `run rest` connection should be enabled by default as part of the configuration policy. - -A more detailed explanation of this command can be found in the network configuration section. - -```anylog - -``` - -## Publishing Data via PUT - -When publishing data into AnyLog via PUT, AnyLog takes the data as-is and stores it into the given database & table -based on the information in the headers. - -**Sample Command**: - -```shell -curl -X PUT http://[Operator IP]:[Operator Port] \ - -H "type: json" \ - -H "dbms: my_db" \ - -H "table: table3" \ - -H "mode: streaming" \ - -H "Content-Type: application/json" \ - -d '[ - {"timestamp": "2026-01-03 10:52:32", "sensor": "temp", "value": 80}, - {"timestamp": "2026-01-03 10:52:32", "sensor": "humidity", "value": 1.2}, - {"timestamp": "2026-01-03 10:52:32", "sensor": "maf", "value": 0.0} - ]' -``` - -The data in this example would be stored under the `my_db` logical database, table `table3`. - -### Header: Mode - streaming vs file - -Data ingested to a local database is organized in files. Each file contains one or more sensor readings (or other type of time series data) organized in a JSON format. -Users adding data with the REST API determines the mode in which data is processed: - -* Using a **File Mode** (the default mode) - a single data file is transferred using the PUT request, the file is registered (in the tsd_info table) and processed independently of other _PUT_ requests. -A File Mode is usually used when the PUT request contains a large amount of data or when the data is not frequently created. - -* Using a **Streaming Mode** - The AnyLog instance receiving the data serves as a buffer that accumulates the data from multiple PUT requests. Upon a threshold, the accumulated data is organized as a file that is processed as a single unit. -A Streaming Mode is usually used when the frequency of data creation is high and the amount of data transferred in each PUT request is low. - -File mode is the default mode. Changing the mode to streaming is by updating the header with the key _mode_ and the value _streaming_. - -**Header options for loading data**: - -| key | value | Explanation | -| ---- | -------| ------------| -| mode | file | The body of the message is JSON data. Database load (on an Operator Node) and data send (on a Publisher Node) are with no wait. File mode is the default behaviour. | -| mode | streaming | The body of the message is JSON data that is buffered in the node. Database load (on an Operator Node) and data send (on a Publisher Node) are based on time and volume thresholds. | - -## Publishing Data via POST - -When publishing data via POST, we can manipulate the data more, since the user is defining the mapping logic for the -table, as opposed to letting AnyLog define it for them. - -**Process**: -1. Define a mapping policy - -> Sample `run msg client` command: -> ```anylog -> broker=rest and user-agent=anylog and -> log=false and topic=( -> name=my-data and -> dbms="bring [dbms]" and -> table="bring [sensor]" and -> column.timestamp.timestamp="bring [timestamp]" and -> column.value.float="bring [value]" -> )> -> ``` - -```shell -curl -X POST http://[Operator IP]:[Operator Port] \ - -H "command: run msg client where broker=rest and user-agent=anylog and log=false and topic=(...)" \ - -H "AnyLog-Agent: AnyLog/1.23" -``` - -2. Publish data - -```shell -curl -X POST http://[Operator IP]:[Operator Port] \ - -H "command: data" \ - -H "topic: my-data" \ - -H "AnyLog-Agent: AnyLog/1.23" \ - -H "Content-Type: application/json" \ - -d '[ - {"dbms": "my_db", "timestamp": "2026-01-03 10:52:32", "sensor": "temp", "value": 80}, - {"dbms": "my_db", "timestamp": "2026-01-03 10:52:32", "sensor": "humidity", "value": 1.2}, - {"dbms": "my_db", "timestamp": "2026-01-03 10:52:32", "sensor": "maf", "value": 0.0} - ]' -``` - -Unlike with REST-PUT, each timestamp/value pair would be stored in its own table, -based on _sensor_, in the `my_db` logical database. - -## Support Commands - -* `get streaming` - statistics on the data flowing - -```anylog -Statistics - Put Put Streaming Streaming Cached Counter Threshold Buffer Threshold Time Left Last Process -DBMS-Table files Rows Calls Rows Rows Immediate Volume(KB) Fill(%) Time(sec) (Sec) HH:MM:SS --------------------------|------|-----|-|---------|---------|------|----------|-----------|--------|----------|---------|------------| -monitoring.docker_insight| 0| 0| | 49,842| 49,842| 5| 0| 10| 27.43| 60| 57|00:00:03 | -``` - -* `get msg client` - Information on messages received by clients subscribed to message brokers. - -* `get operator` & `get publisher` - view the amount of data already processed through the operator or publisher node. diff --git a/04- Southbound Interfaces/02- Direct Connectors/02- Message Broker.md b/04- Southbound Interfaces/02- Direct Connectors/02- Message Broker.md deleted file mode 100644 index d6564c88..00000000 --- a/04- Southbound Interfaces/02- Direct Connectors/02- Message Broker.md +++ /dev/null @@ -1,209 +0,0 @@ ---- -title: "Message Broker" -description: Configure an AnyLog node as a message broker — enable it, subscribe via MQTT or Kafka, register reusable mapping policies, and confirm data lands in the local database. -layout: page -source_path: "background processes.md#message-broker" ---- - - - -As described in the networking section, AnyLog contains a built-in message broker that can be used for MQTT, -Kafka, REST, and other services — simply by defining the message-client mapping logic for the topic. Setting AnyLog -as a message broker is referenced as **Option B** in the Southbound Connectors diagram. - -The Northbound Interfaces cover using an AnyLog agent as a producer; this document covers AnyLog as a consumer. - -This document demonstrates: -1. Configuring an AnyLog node as a broker. -2. Associating published data with a topic. -3. Mapping the data to a table structure (inline, or via a reusable mapping policy). -4. Confirming the data lands in the local database. - ---- - -## Enable the Message Broker - -```anylog - -``` - -The first IP/port pair binds to the external network; the second (optional) pair binds to the local network, if -applicable. - ---- - -## MQTT Message Client - -The MQTT Message client is "identical" to REST POST, except AnyLog is -both the MQTT message broker and MQTT message client. - -When data is published on a broker, it's assigned to a **topic**. An AnyLog node can subscribe to messages published -on a third-party broker, or — if the same node is configured as a broker — to messages published on the AnyLog node -itself, using `run mqtt client`: - -* If subscribing to a third-party broker, provide that broker's IP and port. -* If the same node acts as the broker, set `broker = local` and the process resolves that the data is published - locally. This is also what happens automatically if you set `broker` to this node's own IP and the message - broker's port — AnyLog resolves it to the same thing as `broker = local`. - -**Sample Command**: -```anylog - -``` - -> **To verify:** the previous draft of this example included `user-agent=anylog`, which is the parameter used for -> **REST**-broker mode (`broker=rest and user-agent=anylog`), not local MQTT. I've removed it here since `broker=local` -> shouldn't need it — confirm that's correct before publishing, in case there's an MQTT-specific reason it was there. - -**Subscribing to an external (third-party) broker** instead of AnyLog's own uses the same command, but with the -broker's address in place of `local` — and credentials if the broker requires them: - -```anylog - -``` - -* `broker` — the third-party broker's IP or hostname (required; this is what distinguishes it from `broker=local`). -* `port` — the broker's port (required for third-party brokers). -* `user` / `password` — only needed if the broker requires authentication; omit both for an open broker. - ---- - -## Kafka Message Client - -Similarly to MQTT, AnyLog can also act as a Kafka-like interface for data processing. - -The `run kafka consumer` command subscribes to one or more Kafka topics and maps incoming messages to database -tables using the same column mapping syntax as MQTT. - -| Option | Description | Default | -|---|---|---| -| `ip` | Kafka broker IP | | -| `port` | Kafka broker port | | -| `reset` | Offset policy: `latest` or `earliest` | `latest` | -| `topic` | One or more topics with mapping instructions | | - -```anylog - -``` - -**Example**: -```anylog - -``` - -> **To verify:** fixed a duplicated `and and` in this example. Also worth double-checking: the parameter table above -> lists `ip`/`port` as the Kafka broker's address, with no mention of a `local` shorthand the way MQTT has -> `broker = local`. Confirm `ip = local` is actually supported by `run kafka consumer` before publishing — if not, -> this example needs a real IP/port (or a documented local-broker convention for Kafka specifically). - ---- - -## Registering a Mapping Policy - -Rather than writing the topic's mapping inline on every `run msg client` call, you can register the mapping once as -a **policy** on the blockchain, and reference it by name. This is useful when the same mapping is reused across -multiple subscriptions, or when you want the mapping managed centrally rather than duplicated in each command. - -```anylog -policy_id = telegraf-mapping - - - -blockchain insert where policy=!new_policy and local=true and master=!ledger_conn - -run msg client where broker=local and log=false and topic=(name=my-topic and policy=!new_policy) -``` - -Once inserted, `topic=(name=... and policy=!new_policy)` replaces the inline `dbms=... and table=... and column....` -parameters entirely — the mapping lives in the policy instead. - -> **Fixed two bugs in this example:** (1) `"id"` was hardcoded as `"my-policy"`, ignoring the `policy_id` variable -> set right above it — now it references `!policy_id` so that variable actually does something. (2) The final -> command referenced `policy=!my_policy`, but the policy was defined as `!new_policy` — that variable name was never -> set, so the original example wouldn't have resolved. Now consistent throughout. - ---- - -## Monitor - -```anylog -get msg client -- clients subscribed + messages processed by each -get msg broker -- subscriptions per broker -``` - -For the full connection/config/topic parameter reference, QoS levels, the `bring` command, and debugging options -(`log`, `log_error`, `persist`, `set mqtt debug`), see **Using a Message Broker**. - ---- - -## Confirm It's Working - -Check the streaming buffers (data is held here until flushed): -```anylog -get streaming -``` - -Once buffers flush, query the data directly: -```anylog -run client () sql my_dbms format=table "select timestamp, value from rand_data" -``` diff --git a/04- Southbound Interfaces/02- Mapping Policy.md b/04- Southbound Interfaces/02- Mapping Policy.md deleted file mode 100644 index e277bb40..00000000 --- a/04- Southbound Interfaces/02- Mapping Policy.md +++ /dev/null @@ -1,1660 +0,0 @@ ---- -title: "Mapping Policy" -description: "How mapping policies translate incoming JSON data into table rows — the general model, plus epoch timestamps, blob data, and unknown/dynamic content." -layout: page ---- - -` annotations aren't valid JSON; unquoted `!policy_id`; a missing closing brace in the epoch-timestamp example; a duplicated `readings` key in the "complete" example where the second occurrence should be `schema`); fixed the `run msg client` example that had copied the pre-fix `{user-agent=anylog}` bug and a `nama`/`name` typo from an earlier Message Broker draft; fixed `[device]`→`[deviceName]` and the `dbms` bring-vs-literal mismatch against the EdgeX sample payload; removed a duplicated line in the sample data; fixed a stray H1 mid-document; typo fixes | | - | 2026-08-08 | Moshe Shadmon | moved from blockchain dir + added content | | ----> - -# AnyLog Mapping Policies - -## Overview - -An **AnyLog Mapping Policy** defines how incoming JSON data is mapped to the logical database, table, and columns used by AnyLog. - -Mapping policies synchronize the structure of incoming data with the structure used to store and query that data. They can: - -- Select the destination logical database and table. -- Dynamically derive the database or table from the incoming message. -- Select a list of readings inside a larger JSON message. -- Map source attributes to destination columns. -- Define and normalize column data types. -- Supply default values when source attributes are missing. -- Read values from either an individual reading or the root message. -- Apply transformations to values. -- Create columns dynamically when the source schema is not known in advance. -- Generate multiple rows using parameter sets. -- Apply conditional mapping logic using scripts. -- Store blob or file content separately while maintaining a relational reference. - -A mapping can be defined directly in a `run msg client` command or registered as a policy and referenced by its policy ID. A registered policy is useful when the same mapping is reused by multiple subscriptions or when the mapping should be managed centrally rather than duplicated in commands. - ---- - -## 1. Basic Mapping Policy Structure - -A mapping policy is represented by a root `mapping` object: - -```json -{ - "mapping": { - "id": "sensor-policy", - "dbms": "smart_city", - "table": "sensor_data", - "readings": "", - "schema": { - "timestamp": { - "type": "timestamp", - "bring": "[timestamp]", - "default": "now()" - }, - "value": { - "type": "float", - "bring": "[value]", - "default": null - } - } - } -} -``` - -The two structural elements required by the mapping-policy validator are: - -```text -mapping - ├── id - └── schema -``` - -The destination `dbms` and `table` must also be available when the mapping is executed. They can be supplied by the caller, defined literally in the policy, or derived from the incoming JSON. - ---- - -## 2. Mapping Policy vs. Inline Mapping - -Mapping can be specified directly as part of `run msg client`. - -For example: - -```text - -``` - -When the same mapping is reused, it can instead be registered as a policy and referenced by its ID: - -```text - -``` - -This separates the subscription from the mapping definition and avoids repeating the mapping on every client command. - ---- - -## 3. A Dynamic Routing Example - -Consider the following incoming messages: - -```json -{"dbms": "smart_city", "sensor": "ping_sensor", "timestamp": "2026-07-27T10:00:00Z", "value": 12.4} -{"dbms": "smart_city", "sensor": "ping_sensor", "timestamp": "2026-07-27T10:05:00Z", "value": 13.1} -{"dbms": "smart_city", "sensor": "humidity", "timestamp": "2026-07-27T10:00:00Z", "value": 58.2} -{"dbms": "smart_city", "sensor": "humidity", "value": 60.5} -{"dbms": "smart_city", "sensor": "co2_level", "timestamp": "2026-07-27T09:55:00Z", "value": 412.7} -{"dbms": "smart_city", "sensor": "co2_level", "timestamp": "2026-07-27T10:10:00Z", "value": 418.3} -{"dbms": "monitoring", "sensor": "cpu_temp", "timestamp": "2026-07-27T10:00:00Z", "value": 62.8} -{"dbms": "monitoring", "sensor": "cpu_temp", "timestamp": "2026-07-27T10:15:00Z", "value": 65.0} -``` - -A single mapping policy can route these messages dynamically: - -```json -{ - "mapping": { - "id": "sensor-policy", - "dbms": "bring [dbms]", - "table": "bring [sensor]", - "readings": "", - "schema": { - "timestamp": { - "type": "timestamp", - "default": "now()", - "bring": "[timestamp]" - }, - "value": { - "type": "float", - "default": null, - "bring": "[value]" - } - } - } -} -``` - -The policy creates data for: - -```text -smart_city.ping_sensor -smart_city.humidity -smart_city.co2_level -monitoring.cpu_temp -``` - -This illustrates an important mapping-policy capability: **the database and table do not need to be fixed in the policy**. They can be derived from each incoming message. - -The humidity message without a `timestamp` intentionally demonstrates the `default` mechanism. Because the source timestamp is missing, `"default": "now()"` supplies the current UTC time. - ---- - -# Policy-Level Attributes - -## 4. `id` - -`id` identifies the mapping policy. - -```json -"id": "sensor-policy" -``` - -The mapping-policy validator requires an ID. - -When the policy is referenced by a human-readable variable from commands such as: - -```text -policy=!policy_id -``` - -it is often useful to assign the policy ID explicitly rather than relying on an automatically generated identifier. - ---- - -## 5. `dbms` - -`dbms` identifies the destination logical database. - -A fixed destination can be specified directly: - -```json -"dbms": "industrial_data" -``` - -The database can also be derived from the incoming JSON: - -```json -"dbms": "bring [dbms]" -``` - -Given: - -```json -{ - "dbms": "factory_a" -} -``` - -the destination becomes: - -```text -factory_a -``` - -This allows one mapping policy to route messages to different logical databases. - ---- - -## 6. `table` - -`table` identifies the destination table. - -A fixed table can be specified directly: - -```json -"table": "compressors" -``` - -The table can also be derived dynamically: - -```json -"table": "bring [sensor]" -``` - -Given: - -```json -{ - "sensor": "humidity" -} -``` - -the destination table becomes: - -```text -humidity -``` - -The `dbms` and `table` values are resolved before the rows are generated. - ---- - -## 7. `source` - -`source` identifies the source of the incoming data, such as a device, gateway, application, or other producer. - -Example: - -```json -"source": { - "bring": "[device]" -} -``` - -Given: - -```json -{ - "device": "Pump Station 7" -} -``` - -the resolved source becomes: - -```text -pump_station_7 -``` - -The source string is normalized to lowercase and spaces are replaced with underscores. - ---- - -## 8. `readings` - -The `readings` attribute identifies the part of the incoming JSON that contains the individual records to map. - -### Flat messages - -When the incoming JSON object itself is the reading, `readings` can be empty: - -```json -"readings": "" -``` - -For example: - -```json -{ - "dbms": "smart_city", - "sensor": "ping_sensor", - "timestamp": "2026-07-27T10:00:00Z", - "value": 12.4 -} -``` - -The complete object is processed as one reading. - -If `readings` is omitted, the mapping code likewise treats the complete JSON message as the reading. If the supplied input is already a list, each list entry is processed as a reading. - -### Nested readings - -Consider: - -```json -{ - "dbms": "smart_city", - "sensor": "ping_sensor", - "data": [ - { - "timestamp": "2026-07-27T10:00:00Z", - "value": 12.4 - }, - { - "timestamp": "2026-07-27T10:05:00Z", - "value": 13.1 - } - ] -} -``` - -The policy can specify: - -```json -"readings": "data" -``` - -AnyLog then processes each object in `data` independently. - -The `bring` expressions inside the schema are evaluated relative to each reading: - -```json -"schema": { - "timestamp": { - "type": "timestamp", - "bring": "[timestamp]" - }, - "value": { - "type": "float", - "bring": "[value]" - } -} -``` - -Conceptually: - -```text -Incoming Message - │ - ├── dbms - ├── sensor - │ - └── data - │ - ├── Reading 1 - └── Reading 2 - │ - ▼ - Mapping Schema - │ - ▼ - DB Rows -``` - -> `readings` is a key name, not a `bring` expression. For a JSON key named `data`, use `"readings": "data"`, not `"readings": "[data]"`. - ---- - -# The Schema - -## 9. `schema` - -The `schema` defines the destination columns and the instructions used to obtain their values. - -Example: - -```json -"schema": { - "timestamp": { - "type": "timestamp", - "bring": "[time]" - }, - "temperature": { - "type": "float", - "bring": "[temp]" - }, - "asset": { - "type": "varchar", - "bring": "[name]" - } -} -``` - -Normally, each key in `schema` is the destination column name: - -```text -schema - │ - ├── timestamp - │ ├── type - │ └── bring - │ - ├── temperature - │ ├── type - │ └── bring - │ - └── asset - ├── type - └── bring -``` - -For each reading, AnyLog walks through the schema and constructs an output row. - ---- - -## 10. `type` - -`type` defines the destination column's data type. - -Example: - -```json -"temperature": { - "type": "float", - "bring": "[temperature]" -} -``` - -AnyLog normalizes multiple source or policy type names into a common set of data types. - -### Data Type Normalization - -| Input Type | Normalized Type | -|---|---| -| `str` | `varchar` | -| `string` | `varchar` | -| `char varying` | `varchar` | -| `varchar` | `varchar` | -| `bytestring` | `varchar` | -| `uuid` | `uuid` | -| `bigint` | `bigint` | -| `integer` | `int` | -| `int` | `int` | -| `int16` | `int` | -| `uint16` | `int` | -| `int32` | `int` | -| `uint32` | `int` | -| `sbyte` | `int` | -| `int64` | `bigint` | -| `uint64` | `bigint` | -| `float` | `float` | -| `float64` | `float` | -| `decimal` | `float` | -| `numeric` | `float` | -| `double` | `float` | -| `char` | `char` | -| `character` | `char` | -| `byte` | `char(1)` | -| `bool` | `bool` | -| `boolean` | `bool` | -| `timestamp` | `timestamp` | -| `datetime` | `timestamp` | -| `date` | `date` | -| `time` | `time` | -| `nonetype` | `nonetype` | - -### OPC UA Types - -The type normalizer also recognizes several OPC UA-oriented type names: - -| Input Type | Normalized Type | Description | -|---|---|---| -| `nodeid` | `varchar` | OPC UA NodeId | -| `expandednodeid` | `varchar` | OPC UA ExpandedNodeId | -| `StatusCode` | `int` | OPC UA status code | -| `qualifiedmame` | `varchar` | OPC UA qualified-name value | -| `localizedtext` | `varchar` | OPC UA localized text | -| `variant` | `varchar` | Container for an arbitrary OPC UA data type | -| `datavalue` | `varchar` | Value with status and timestamp information | -| `diagnosticinfo` | `varchar` | Error or diagnostic information | - -> The spelling `qualifiedmame` above reflects the currently supplied type-unifier key. If the implementation changes that key to `qualifiedname`, the documentation should be updated accordingly. - -### Common normalized types - -For most manually written mapping policies, the most common destination types are: - -```text -varchar -uuid -int -bigint -float -char -bool -timestamp -date -time -nonetype -``` - ---- - -## 11. Default Mapping by Column Name - -If a column does not specify `bring`, AnyLog attempts to retrieve a source attribute having the same name as the destination column. - -For example: - -```json -"schema": { - "temperature": { - "type": "float" - } -} -``` - -with: - -```json -{ - "temperature": 72.5 -} -``` - -is effectively equivalent to: - -```json -"temperature": { - "type": "float", - "bring": "[temperature]" -} -``` - -This allows simple mappings to remain concise. - ---- - -## 12. `bring` - -`bring` specifies where a column value should be retrieved from the source JSON. - -Example: - -```json -"temperature": { - "type": "float", - "bring": "[temp]" -} -``` - -Input: - -```json -{ - "temp": 72.5 -} -``` - -Output: - -```json -{ - "temperature": 72.5 -} -``` - -The destination column name and source attribute name do not need to match. - -For example: - -```json -"schema": { - "asset_id": { - "type": "varchar", - "bring": "[device]" - }, - "temperature": { - "type": "float", - "bring": "[measurements][temp]" - } -} -``` - -Conceptually: - -```text -Source JSON Destination - -device --------------------------> asset_id - -measurements - └── temp ---------------------> temperature -``` - -AnyLog parses the `bring` instruction and, when possible, caches the compiled representation for reuse across subsequent rows. - ---- - -## 13. `default` - -`default` supplies a value when `bring` does not return data. - -Example: - -```json -"quality": { - "type": "varchar", - "bring": "[quality]", - "default": "good" -} -``` - -If `quality` is absent, the mapped value becomes: - -```json -{ - "quality": "good" -} -``` - -### Current time - -A particularly useful default is: - -```json -"default": "now()" -``` - -For example: - -```json -"timestamp": { - "type": "timestamp", - "bring": "[timestamp]", - "default": "now()" -} -``` - -When `timestamp` is missing, AnyLog uses the current UTC time. - -A `null` default can also be specified: - -```json -"value": { - "type": "float", - "bring": "[value]", - "default": null -} -``` - ---- - -## 14. `root` - -When `readings` selects a nested list, schema fields normally read from the current item in that list. - -`root: true` changes the lookup context to the complete original JSON message. - -Consider: - -```json -{ - "site": "factory_a", - "readings": [ - { - "temperature": 71.2 - }, - { - "temperature": 71.4 - } - ] -} -``` - -Policy: - -```json -{ - "mapping": { - "id": "temperature-policy", - "dbms": "factory", - "table": "temperature", - "readings": "readings", - "schema": { - "site": { - "type": "varchar", - "bring": "[site]", - "root": true - }, - "temperature": { - "type": "float", - "bring": "[temperature]" - } - } - } -} -``` - -Output: - -```json -{ - "site": "factory_a", - "temperature": 71.2 -} -``` - -and: - -```json -{ - "site": "factory_a", - "temperature": 71.4 -} -``` - -Conceptually: - -```text -Root Message -│ -├── site -----------------------------┐ -│ │ root=true -└── readings │ - │ │ - ├── {temperature: 71.2} ---------┼----> Output Row 1 - │ │ - └── {temperature: 71.4} ---------┼----> Output Row 2 - │ - site copied -----┘ - into each row -``` - ---- - -## 15. `value` - -`value` can identify or derive the source attribute used by a mapping. - -For example: - -```json -"value": "bring [temperature]" -``` - -When `value` contains a `bring [...]` expression, the mapping logic extracts the referenced source attribute and uses it for value retrieval. - -The same mechanism is also used in mapping flows that derive values from PLC attribute names or regular-expression matches. - ---- - -## 16. `apply` - -`apply` transforms a value before it is added to the destination row. - -For regular mapped values, the supplied implementation supports: - -- `epoch_to_datetime` -- `json_dump` - -### `epoch_to_datetime` - -```json -"timestamp": { - "bring": "[origin]", - "default": "now()", - "type": "timestamp", - "apply": "epoch_to_datetime" -} -``` - -This is useful when a device provides a numeric epoch value rather than a formatted timestamp. - -The name `origin` is not special; it is simply the source field used in this example. - -### `json_dump` - -```json -"metadata": { - "type": "varchar", - "bring": "[metadata]", - "apply": "json_dump" -} -``` - -If the source contains: - -```json -{ - "metadata": { - "manufacturer": "ABC", - "model": "X7" - } -} -``` - -the dictionary is serialized into a JSON-formatted string for storage. - ---- - -## 17. Timestamp Handling - -Timestamp columns receive special formatting and validation. - -Example: - -```json -"timestamp": { - "type": "timestamp", - "bring": "[timestamp]" -} -``` - -The mapping implementation handles timestamp values represented as: - -- Date/time strings. -- ISO-formatted strings with timezone offsets. -- Integer values. -- Floating-point values. -- Numeric strings. - -For sources that require an explicit epoch transformation, use: - -```json -"timestamp": { - "type": "timestamp", - "bring": "[origin]", - "default": "now()", - "apply": "epoch_to_datetime" -} -``` - ---- - -# Advanced Schema Features - -## 18. Multiple Mapping Alternatives for a Column - -A schema column can contain a list of mapping definitions instead of one definition. - -Example: - -```json -"temperature": [ - { - "dbms": "factory", - "table": "compressors", - "type": "float", - "bring": "[compressor_temp]" - }, - { - "dbms": "factory", - "table": "pumps", - "type": "float", - "bring": "[pump_temp]" - } -] -``` - -AnyLog evaluates the alternatives in order and skips definitions whose `dbms` or `table` does not match the current destination. - -Conceptually: - -```text -temperature - │ - ├── table = compressors - │ └── bring [compressor_temp] - │ - └── table = pumps - └── bring [pump_temp] -``` - -This allows a schema to contain destination-specific alternatives. - ---- - -## 19. Column-Level `dbms` and `table` - -The `dbms` and `table` attributes inside a column definition restrict that particular mapping option. - -Example: - -```json -"value": [ - { - "dbms": "factory_a", - "table": "temperature", - "type": "float", - "bring": "[temp]" - }, - { - "dbms": "factory_a", - "table": "pressure", - "type": "float", - "bring": "[pressure]" - } -] -``` - -A definition that does not match the current database or table is ignored and the mapper continues to the next alternative. - ---- - -## 20. Unknown and Dynamic Columns - -Mapping policies can create columns dynamically when the complete source schema is not known ahead of time. - -This is useful with sources such as Telegraf, Litmus Edge, and other systems that may provide changing sets of fields. - -There are two common cases. - -### Known keys, types determined dynamically - -If the keys are known but their data types are not: - -```json -"*": { - "type": "*", - "bring": [ - "success", - "tagName", - "value", - "description" - ] -} -``` - -AnyLog retrieves those keys and derives the column types from the source values. - -### Unknown keys and unknown types - -If neither the keys nor their types are known: - -```json -"*": { - "type": "*", - "bring": ["*"] -} -``` - -Given: - -```json -{ - "temperature": 72.5, - "pressure": 101.3, - "rpm": 1450 -} -``` - -the mapper can create: - -```text -temperature -pressure -rpm -``` - -and infer the corresponding data types. - ---- - -## 21. Dynamic Columns from Subobjects - -The `*` mapping can also pull fields from selected subobjects. - -Example: - -```json -"*": { - "type": "*", - "bring": [ - "fields", - "tags" - ] -} -``` - -Input: - -```json -{ - "fields": { - "temperature": 72.5, - "pressure": 101.3 - }, - "tags": { - "site": "A", - "machine": "compressor_7" - } -} -``` - -The generated column names include the parent key: - -```text -fields_temperature -fields_pressure -tags_site -tags_machine -``` - -Conceptually: - -```text -fields.temperature ---> fields_temperature -fields.pressure ---> fields_pressure - -tags.site ---> tags_site -tags.machine ---> tags_machine -``` - ---- - -## 22. `params` - -`params` allows the same schema to generate multiple logical output rows using different parameter sets. - -Example: - -```json -"params": [ - [ - "DelayTimer", - "[DelayTimer.ACC]", - "[DelayTimer.PRE]" - ], - [ - "CycleCounter", - "[CycleCounter.ACC]", - "[CycleCounter.PRE]" - ] -] -``` - -The schema can reference each value using: - -```text -params.0 -params.1 -params.2 -``` - -Example: - -```json -"schema": { - "timestamp": { - "type": "timestamp", - "default": "now()", - "bring": "[timestamp]" - }, - "Monitor_ID": { - "type": "str", - "default": "params.0" - }, - "ACC": { - "type": "int", - "default": null, - "bring": "params.1" - }, - "PRE": { - "type": "int", - "default": null, - "bring": "params.2" - } -} -``` - -For the first parameter set: - -```text -params.0 = DelayTimer -params.1 = [DelayTimer.ACC] -params.2 = [DelayTimer.PRE] -``` - -For the second: - -```text -params.0 = CycleCounter -params.1 = [CycleCounter.ACC] -params.2 = [CycleCounter.PRE] -``` - -The policy is processed once for each parameter list. - -```text - Mapping Schema - │ - ┌───────────────┴───────────────┐ - │ │ - Params Set 1 Params Set 2 - │ │ - DelayTimer CycleCounter - │ │ - ▼ ▼ - Output Row Output Row -``` - -`params.N` substitution is supported in `default`, `bring`, and `value`. - ---- - -## 23. `script` - -A schema entry can contain an AnyLog `script` that is executed before normal column mapping. - -Example structure: - -```json -"some_column": { - "script": "", - "type": "varchar", - "bring": "[value]" -} -``` - -The script can affect mapping flow, including: - -- Skipping the current attribute. -- Skipping the complete event. -- Exiting the script while allowing mapping to continue. -- Switching processing to another imported policy. - -This provides conditional and procedural control over the mapping process. - ---- - -## 24. Script-Only Schema Entries - -Schema names beginning and ending with two underscores are treated as control entries and are not emitted as destination columns. - -For example: - -```json -"__start__": { - "script": "..." -} -``` - -or: - -```json -"__end__": { - "script": "..." -} -``` - -Conceptually: - -```text -schema - │ - ├── __start__ script/control only - │ - ├── timestamp destination column - │ - ├── temperature destination column - │ - └── __end__ script/control only -``` - ---- - -## 25. Changing Mapping Policy During Processing - -Scripts can return a change-policy result. - -When this occurs, the mapping logic can select another policy from an imported-policy dictionary and restart processing with that policy. - -This provides a mechanism for selecting a mapping dynamically based on the content of the incoming event. - -This is an advanced feature and is normally used only when a single incoming stream contains messages requiring substantially different mapping logic. - ---- - -# Blob and File Data - -## 26. Blob Columns - -A schema column can identify source content as blob/file data. - -Example: - -```json -"file": { - "root": true, - "blob": true, - "bring": "[file_content]", - "extension": "jpeg", - "apply": "base64decoding", - "hash": "md5", - "type": "varchar" -} -``` - -When: - -```json -"blob": true -``` - -AnyLog writes the blob content to the configured blob directory. - -The relational row contains the generated file identifier rather than the complete blob content. - -The current implementation recognizes: - -```json -"hash": "md5" -``` - -The hash is used to generate the file name. - -An optional extension can be supplied: - -```json -"extension": "jpeg" -``` - ---- - -## 27. Blob `apply` Operations - -For blob processing, the supplied implementation supports: - -```json -"apply": "base64decoding" -``` - -and: - -```json -"apply": "opencv" -``` - -### Base64 example - -```json -"file": { - "root": true, - "blob": true, - "bring": "[file_content]", - "extension": "jpeg", - "apply": "base64decoding", - "hash": "md5", - "type": "varchar" -} -``` - -The Base64 value is decoded before being written to the blob directory. - -### Updating blob `apply` logic - -Examples of changing the policy: - -```text -set policy new_policy [mapping][schema][file][apply] = "base64decoding" -``` - -or: - -```text -set policy new_policy [mapping][schema][file][apply] = "opencv" -``` - ---- - -# Complete Examples - -## 28. Complete Nested-Reading Example - -Source message: - -```json -{ - "site": "Plant_A", - "gateway": "edge_17", - "readings": [ - { - "time": "2026-08-08T18:30:00Z", - "asset": "compressor_1", - "temp": 72.5, - "pressure": 101.4 - }, - { - "time": "2026-08-08T18:30:01Z", - "asset": "compressor_2", - "temp": 74.1, - "pressure": 102.7 - } - ] -} -``` - -Policy: - -```json -{ - "mapping": { - "id": "compressor_mapping", - "dbms": "industrial", - "table": "compressor_data", - "source": { - "bring": "[gateway]" - }, - "readings": "readings", - "schema": { - "timestamp": { - "type": "timestamp", - "bring": "[time]", - "default": "now()" - }, - "site": { - "type": "varchar", - "bring": "[site]", - "root": true - }, - "asset": { - "type": "varchar", - "bring": "[asset]" - }, - "temperature": { - "type": "float", - "bring": "[temp]" - }, - "pressure": { - "type": "float", - "bring": "[pressure]" - } - } - } -} -``` - -The first reading generates approximately: - -```json -{ - "timestamp": "2026-08-08T18:30:00Z", - "site": "Plant_A", - "asset": "compressor_1", - "temperature": 72.5, - "pressure": 101.4 -} -``` - -The second generates: - -```json -{ - "timestamp": "2026-08-08T18:30:01Z", - "site": "Plant_A", - "asset": "compressor_2", - "temperature": 74.1, - "pressure": 102.7 -} -``` - -`site` is read from the root message because it specifies: - -```json -"root": true -``` - -while the other values are read from each individual item in `readings`. - ---- - -## 29. EdgeX-Style Example - -Sample EdgeX-style input: - -```json -{ - "apiVersion": "v2", - "id": "707564c4-6818-4746-9c54-219a0fd110c6", - "deviceName": "ba-virtual", - "profileName": "BuildingAutomationVirtualDevice", - "sourceName": "AvgTemp", - "origin": 1686087247849269800, - "readings": [ - { - "id": "42700bdd-4525-443f-88dd-22c488011b65", - "origin": 1686087247849269800, - "deviceName": "ba-virtual", - "resourceName": "AvgTemp", - "profileName": "BuildingAutomationVirtualDevice", - "valueType": "Float32", - "units": "°F", - "value": "7.934139e+01" - } - ] -} -``` - -A corresponding mapping policy can be: - -```json -{ - "mapping": { - "id": "full-policy", - "dbms": "edgex_data", - "table": "bring [deviceName]", - "readings": "readings", - "schema": { - "timestamp": { - "type": "timestamp", - "default": "now()", - "bring": "[origin]", - "apply": "epoch_to_datetime" - }, - "*": { - "type": "*", - "bring": [ - "deviceName", - "resourceName", - "profileName", - "valueType", - "units", - "value" - ] - } - } - } -} -``` - -In this example: - -- `dbms` is fixed as `edgex_data`. -- `table` is dynamically derived from `deviceName`. -- `readings` selects the list under the `readings` key. -- `timestamp` is derived from the reading's `origin`. -- `epoch_to_datetime` converts the source epoch value. -- `*` dynamically creates the selected source columns and derives their types. - -If file content exists at the root of the source message, a blob mapping can also be added: - -```json -"file": { - "root": true, - "blob": true, - "bring": "[file_content]", - "extension": "jpeg", - "apply": "base64decoding", - "hash": "md5", - "type": "varchar" -} -``` - ---- - -# Processing Model - -## 30. Mapping Processing Flow - -At a high level: - -```text - Incoming JSON - │ - ▼ - Determine Source - │ - ▼ - Locate Readings - │ - ▼ - Determine DBMS/Table - │ - ▼ - For Each Param Set - │ - ▼ - For Each Reading - │ - ▼ - Apply Schema - │ - ┌────────────┼────────────┐ - │ │ │ - bring default script - │ │ │ - └────────────┼────────────┘ - │ - apply - │ - ▼ - Normalize Type - │ - ▼ - Output Row - │ - ▼ - Destination Table -``` - -When no `params` are defined, the schema is processed once for each reading. - -When `params` are defined, the mapping is repeated for each parameter set and each reading. - ---- - -## 31. Column Processing Flow - -For a normal schema column: - -```text -1. Select applicable column definition - │ - ▼ -2. Apply params.N substitutions, if used - │ - ▼ -3. Execute script, if defined - │ - ▼ -4. Determine column type - │ - ▼ -5. Determine source attribute - │ - ▼ -6. Select reading or root JSON - │ - ▼ -7. Execute bring - │ - ├── value found ───────────────┐ - │ │ - └── no value │ - │ │ - ▼ │ - default │ - │ │ - └──────────────────────┤ - ▼ - Apply function - │ - ▼ - Normalize type - │ - ▼ - Destination value -``` - ---- - -# Reference - -## 32. Policy-Level Attributes - -| Attribute | Purpose | -|---|---| -| `id` | Unique mapping-policy identifier | -| `dbms` | Destination logical database; can be literal or dynamically derived | -| `table` | Destination table; can be literal or dynamically derived | -| `source` | Identifies the data-producing source | -| `readings` | Identifies the key containing the readings to process | -| `params` | Repeats the schema using different parameter sets | -| `schema` | Defines destination columns and mapping instructions | - ---- - -## 33. Schema-Column Attributes - -| Attribute | Purpose | -|---|---| -| `type` | Defines the destination data type | -| `bring` | Retrieves a value from the source JSON | -| `default` | Supplies a value when source data is unavailable | -| `root` | Reads from the root JSON message instead of the current reading | -| `value` | Identifies or derives the source value used for mapping | -| `apply` | Applies a value transformation | -| `script` | Executes conditional or procedural mapping logic | -| `dbms` | Restricts a mapping alternative to a database | -| `table` | Restricts a mapping alternative to a table | -| `blob` | Identifies the value as file/blob data | -| `hash` | Defines the blob hash; the current implementation recognizes `md5` | -| `extension` | Adds an extension to a stored blob file | - ---- - -## 34. Special Schema Constructs - -| Construct | Purpose | -|---|---| -| `"*"` | Dynamically creates columns from source attributes | -| `"__name__"` | Defines a script/control entry that does not become a destination column | -| List of column definitions | Provides alternative mappings for the same destination column | -| `params.N` | Substitutes a value from the active parameter set | - ---- - -## 35. Minimal Policy - -When source attribute names already match the desired destination column names, the policy can be very small: - -```json -{ - "mapping": { - "id": "simple_mapping", - "dbms": "factory", - "table": "sensor_data", - "schema": { - "timestamp": { - "type": "timestamp" - }, - "asset": { - "type": "varchar" - }, - "temperature": { - "type": "float" - } - } - } -} -``` - -Input: - -```json -{ - "timestamp": "2026-08-08T18:30:00Z", - "asset": "compressor_7", - "temperature": 72.5 -} -``` - -Because `bring` is omitted, AnyLog uses the destination column names as the source attribute names: - -```text -Source JSON attribute - │ - │ same name - ▼ -Schema column - │ - ▼ -Destination table -``` - -More advanced attributes are only needed when the source structure, destination structure, routing, or processing logic requires them. diff --git a/04- Southbound Interfaces/03- Direct Connectors/01- REST.md b/04- Southbound Interfaces/03- Direct Connectors/01- REST.md deleted file mode 100644 index 6cc02676..00000000 --- a/04- Southbound Interfaces/03- Direct Connectors/01- REST.md +++ /dev/null @@ -1,133 +0,0 @@ ---- -title: Publishing data via REST -description: How to insert data into AnyLog via REST — PUT vs. POST, mapping policies, and support commands. -layout: page ---- - - -The following document provides directions on how to insert data via REST. - -## Setting Up the Node - -The `run rest` connection should be enabled by default as part of the configuration policy. - -A more detailed explanation of this command can be found in the network configuration section. - -```anylog - -``` - -## Publishing Data via PUT - -When publishing data into AnyLog via PUT, AnyLog takes the data as-is and stores it into the given database & table -based on the information in the headers. - -**Sample Command**: - -```shell -curl -X PUT http://[Operator IP]:[Operator Port] \ - -H "type: json" \ - -H "dbms: my_db" \ - -H "table: table3" \ - -H "mode: streaming" \ - -H "Content-Type: application/json" \ - -d '[ - {"timestamp": "2026-01-03 10:52:32", "sensor": "temp", "value": 80}, - {"timestamp": "2026-01-03 10:52:32", "sensor": "humidity", "value": 1.2}, - {"timestamp": "2026-01-03 10:52:32", "sensor": "maf", "value": 0.0} - ]' -``` - -The data in this example would be stored under the `my_db` logical database, table `table3`. - -### Header: Mode - streaming vs file - -Data ingested to a local database is organized in files. Each file contains one or more sensor readings (or other type of time series data) organized in a JSON format. -Users adding data with the REST API determines the mode in which data is processed: - -* Using a **File Mode** (the default mode) - a single data file is transferred using the PUT request, the file is registered (in the tsd_info table) and processed independently of other _PUT_ requests. -A File Mode is usually used when the PUT request contains a large amount of data or when the data is not frequently created. - -* Using a **Streaming Mode** - The AnyLog instance receiving the data serves as a buffer that accumulates the data from multiple PUT requests. Upon a threshold, the accumulated data is organized as a file that is processed as a single unit. -A Streaming Mode is usually used when the frequency of data creation is high and the amount of data transferred in each PUT request is low. - -File mode is the default mode. Changing the mode to streaming is by updating the header with the key _mode_ and the value _streaming_. - -**Header options for loading data**: - -| key | value | Explanation | -| ---- | -------| ------------| -| mode | file | The body of the message is JSON data. Database load (on an Operator Node) and data send (on a Publisher Node) are with no wait. File mode is the default behaviour. | -| mode | streaming | The body of the message is JSON data that is buffered in the node. Database load (on an Operator Node) and data send (on a Publisher Node) are based on time and volume thresholds. | - -## Publishing Data via POST - -When publishing data via POST, we can manipulate the data more, since the user is defining the mapping logic for the -table, as opposed to letting AnyLog define it for them. - -**Process**: -1. Define a mapping policy - -> Sample `run msg client` command: -> ```anylog -> broker=rest and user-agent=anylog and -> log=false and topic=( -> name=my-data and -> dbms="bring [dbms]" and -> table="bring [sensor]" and -> column.timestamp.timestamp="bring [timestamp]" and -> column.value.float="bring [value]" -> )> -> ``` - -```shell -curl -X POST http://[Operator IP]:[Operator Port] \ - -H "command: run msg client where broker=rest and user-agent=anylog and log=false and topic=(...)" \ - -H "AnyLog-Agent: AnyLog/1.23" -``` - -2. Publish data - -```shell -curl -X POST http://[Operator IP]:[Operator Port] \ - -H "command: data" \ - -H "topic: my-data" \ - -H "AnyLog-Agent: AnyLog/1.23" \ - -H "Content-Type: application/json" \ - -d '[ - {"dbms": "my_db", "timestamp": "2026-01-03 10:52:32", "sensor": "temp", "value": 80}, - {"dbms": "my_db", "timestamp": "2026-01-03 10:52:32", "sensor": "humidity", "value": 1.2}, - {"dbms": "my_db", "timestamp": "2026-01-03 10:52:32", "sensor": "maf", "value": 0.0} - ]' -``` - -Unlike with REST-PUT, each timestamp/value pair would be stored in its own table, -based on _sensor_, in the `my_db` logical database. - -## Support Commands - -* `get streaming` - statistics on the data flowing - -```anylog -Statistics - Put Put Streaming Streaming Cached Counter Threshold Buffer Threshold Time Left Last Process -DBMS-Table files Rows Calls Rows Rows Immediate Volume(KB) Fill(%) Time(sec) (Sec) HH:MM:SS --------------------------|------|-----|-|---------|---------|------|----------|-----------|--------|----------|---------|------------| -monitoring.docker_insight| 0| 0| | 49,842| 49,842| 5| 0| 10| 27.43| 60| 57|00:00:03 | -``` - -* `get msg client` - Information on messages received by clients subscribed to message brokers. - -* `get operator` & `get publisher` - view the amount of data already processed through the operator or publisher node. \ No newline at end of file diff --git a/04- Southbound Interfaces/03- Direct Connectors/02- Message Broker.md b/04- Southbound Interfaces/03- Direct Connectors/02- Message Broker.md deleted file mode 100644 index 05a116cb..00000000 --- a/04- Southbound Interfaces/03- Direct Connectors/02- Message Broker.md +++ /dev/null @@ -1,209 +0,0 @@ ---- -title: "Message Broker" -description: Configure an AnyLog node as a message broker — enable it, subscribe via MQTT or Kafka, register reusable mapping policies, and confirm data lands in the local database. -layout: page -source_path: "background processes.md#message-broker" ---- - - - -As described in the networking section, AnyLog contains a built-in message broker that can be used for MQTT, -Kafka, REST, and other services — simply by defining the message-client mapping logic for the topic. Setting AnyLog -as a message broker is referenced as **Option B** in the Southbound Connectors diagram. - -The Northbound Interfaces cover using an AnyLog agent as a producer; this document covers AnyLog as a consumer. - -This document demonstrates: -1. Configuring an AnyLog node as a broker. -2. Associating published data with a topic. -3. Mapping the data to a table structure (inline, or via a reusable mapping policy). -4. Confirming the data lands in the local database. - ---- - -## Enable the Message Broker - -```anylog - -``` - -The first IP/port pair binds to the external network; the second (optional) pair binds to the local network, if -applicable. - ---- - -## MQTT Message Client - -The MQTT Message client is "identical" to REST POST, except AnyLog is -both the MQTT message broker and MQTT message client. - -When data is published on a broker, it's assigned to a **topic**. An AnyLog node can subscribe to messages published -on a third-party broker, or — if the same node is configured as a broker — to messages published on the AnyLog node -itself, using `run mqtt client`: - -* If subscribing to a third-party broker, provide that broker's IP and port. -* If the same node acts as the broker, set `broker = local` and the process resolves that the data is published - locally. This is also what happens automatically if you set `broker` to this node's own IP and the message - broker's port — AnyLog resolves it to the same thing as `broker = local`. - -**Sample Command**: -```anylog - -``` - -> **To verify:** the previous draft of this example included `user-agent=anylog`, which is the parameter used for -> **REST**-broker mode (`broker=rest and user-agent=anylog`), not local MQTT. I've removed it here since `broker=local` -> shouldn't need it — confirm that's correct before publishing, in case there's an MQTT-specific reason it was there. - -**Subscribing to an external (third-party) broker** instead of AnyLog's own uses the same command, but with the -broker's address in place of `local` — and credentials if the broker requires them: - -```anylog - -``` - -* `broker` — the third-party broker's IP or hostname (required; this is what distinguishes it from `broker=local`). -* `port` — the broker's port (required for third-party brokers). -* `user` / `password` — only needed if the broker requires authentication; omit both for an open broker. - ---- - -## Kafka Message Client - -Similarly to MQTT, AnyLog can also act as a Kafka-like interface for data processing. - -The `run kafka consumer` command subscribes to one or more Kafka topics and maps incoming messages to database -tables using the same column mapping syntax as MQTT. - -| Option | Description | Default | -|---|---|---| -| `ip` | Kafka broker IP | | -| `port` | Kafka broker port | | -| `reset` | Offset policy: `latest` or `earliest` | `latest` | -| `topic` | One or more topics with mapping instructions | | - -```anylog - -``` - -**Example**: -```anylog - -``` - -> **To verify:** fixed a duplicated `and and` in this example. Also worth double-checking: the parameter table above -> lists `ip`/`port` as the Kafka broker's address, with no mention of a `local` shorthand the way MQTT has -> `broker = local`. Confirm `ip = local` is actually supported by `run kafka consumer` before publishing — if not, -> this example needs a real IP/port (or a documented local-broker convention for Kafka specifically). - ---- - -## Registering a Mapping Policy - -Rather than writing the topic's mapping inline on every `run msg client` call, you can register the mapping once as -a **policy** on the blockchain, and reference it by name. This is useful when the same mapping is reused across -multiple subscriptions, or when you want the mapping managed centrally rather than duplicated in each command. - -```anylog -policy_id = telegraf-mapping - - - -blockchain insert where policy=!new_policy and local=true and master=!ledger_conn - -run msg client where broker=local and log=false and topic=(name=my-topic and policy=!new_policy) -``` - -Once inserted, `topic=(name=... and policy=!new_policy)` replaces the inline `dbms=... and table=... and column....` -parameters entirely — the mapping lives in the policy instead. - -> **Fixed two bugs in this example:** (1) `"id"` was hardcoded as `"my-policy"`, ignoring the `policy_id` variable -> set right above it — now it references `!policy_id` so that variable actually does something. (2) The final -> command referenced `policy=!my_policy`, but the policy was defined as `!new_policy` — that variable name was never -> set, so the original example wouldn't have resolved. Now consistent throughout. - ---- - -## Monitor - -```anylog -get msg client -- clients subscribed + messages processed by each -get msg broker -- subscriptions per broker -``` - -For the full connection/config/topic parameter reference, QoS levels, the `bring` command, and debugging options -(`log`, `log_error`, `persist`, `set mqtt debug`), see **Using a Message Broker**. - ---- - -## Confirm It's Working - -Check the streaming buffers (data is held here until flushed): -```anylog -get streaming -``` - -Once buffers flush, query the data directly: -```anylog -run client () sql my_dbms format=table "select timestamp, value from rand_data" -``` \ No newline at end of file diff --git a/04- Southbound Interfaces/03- Industrial Connectors/01- Modbus.md b/04- Southbound Interfaces/03- Industrial Connectors/01- Modbus.md deleted file mode 100644 index a42b1928..00000000 --- a/04- Southbound Interfaces/03- Industrial Connectors/01- Modbus.md +++ /dev/null @@ -1,157 +0,0 @@ ---- -title: Modbus TCP -description: Pull coils, discrete inputs, and holding/input registers from Modbus TCP devices into AnyLog using run plc client. -layout: page -source_path: "02 MODBUS.md" ---- - - - -AnyLog can act as a **Modbus TCP client** (over **hostname** and **port**, typically port **502**). Data is read on a -schedule and streamed into your local operator database as JSON, using the same **`run plc client`** pattern as -OPC-UA and EtherNet/IP. - ---- - -## Prerequisites - -| Requirement | Notes | -|---|---| -| **`pymodbus`** | Must be installed in the AnyLog runtime environment. | - -Declare the target DBMS before streaming, for example: - -```anylog -connect dbms new_company where type = sqlite -``` - ---- - -## Connection and map - -Modbus uses **`hostname`** and **`port`** (not `url`). **`device_id`** is the **Modbus PDU unit id** (often `0` or `1` on TCP). **`map`** is a **JSON array** of points; each object must include a **`name`** (column / logical label) and exactly one address field: - -| Key | Meaning | -|---|---| -| `coil` | Coil (0-based address) | -| `input` | Discrete input | -| `inputRegister` | Input register (single address or list for blocks) | -| `register` | Holding register (single address or list for blocks) | - -**`topic`** is **not** supported on Modbus `run plc client` commands. - ---- - -## One-shot read - -```anylog - -``` - -Alias: **`get modbus values`** (same keywords). - ---- - -## Continuous ingest — wide table (default) - -With **`table = ...`** and **`dbms`**, all points from **`map`** land in **one table**. Each poll inserts **one row**; every object in **`map`** is **one column**: the map **`name`** is the **column name**, and that column stores the **value** read for that point. - -```anylog - -``` - ---- - -## Continuous ingest — dynamic tables (`dynamic = true`) - -Omit **`table`** and omit **`namespace`** for plain dynamic ingest. Each object in **`map`** is written to its **own table**. The table name is derived from the client **`name`** and the map **`name`** (for example, **`fdev10_outside_temperature`** when **`name = fdev10`** and the map entry’s **`name`** is **`outside_temperature`**). Each row includes **`timestamp`**, **`tag`**, and **`value`**. - -```anylog - -``` - ---- - -## Dynamic ingest with UNS (`namespace` + `master_node`) - -With **`dynamic = true`**, you can add a **Unified Namespace** path and a **master node** so Modbus ingest is registered in the UNS alongside your policies and DBMS. **`namespace`** requires **`master_node = [ip:port]`** for policy updates. - -Example (abbreviated): - -```anylog -run plc client where type = modbus and - hostname = 192.168.1.72 and - port = 1502 and - device_id = 1 and - frequency = 5 and - name = fdev11 and - dbms = new_company and - dynamic = true and - master_node = 192.168.1.88:32048 and - namespace = FA9/MID9/DEVICE9 and - map = [{"name":"desk_lamp","coil":0}] -``` - -**Table names** follow the **same pattern** as plain **`dynamic = true`** (client **`name`** plus map **`name`**, e.g. **`fdev11_desk_lamp`** for the example above). Under UNS, the **read value** is usually stored in a **column named like the tag**—the map **`name`** (here **`desk_lamp`**), not a generic **`value`** column. - -With **`namespace`**, table and column layout follow **UNS policies**. **`namespace`** and **`master_node`** drive how -tables are registered in the UNS. See Unified Namespace for background. - ---- - -## Command keywords (summary) - -| Keyword | Required / notes | -|---|---| -| `type` | `modbus` | -| `hostname`, `port` | Modbus TCP target | -| `device_id` | PDU unit id | -| `frequency` | Poll interval | -| `name` | Unique client name | -| `dbms` | Target DBMS | -| `table` | Wide-table ingest; omit with **`dynamic = true`** | -| `dynamic` | `true` for per-map tables or UNS | -| `map` | JSON array of points | -| `namespace` | UNS path (Modbus + **`dynamic = true`** only) | -| `master_node` | Required when **`namespace`** is set | - ---- - -## Related - -- Adding Data to Nodes in the Network -- Unified Namespace -- OPC-UA -- EtherNet/IP diff --git a/04- Southbound Interfaces/03- Industrial Connectors/02- OPC-UA.md b/04- Southbound Interfaces/03- Industrial Connectors/02- OPC-UA.md deleted file mode 100644 index edf54505..00000000 --- a/04- Southbound Interfaces/03- Industrial Connectors/02- OPC-UA.md +++ /dev/null @@ -1,382 +0,0 @@ ---- -title: OPC-UA -description: Configure AnyLog as an OPC-UA client to explore a server, read values, pull data continuously, apply aggregations, and manage tags with policies. -layout: page -source_path: "OPC UA Integration.md" ---- - - -OPC Unified Architecture (OPC UA) is a robust, platform-independent industrial communication protocol with built-in -security (encryption, authentication, and access control), widely used in industrial automation for secure and reliable -data exchange between devices, systems, and applications. Designed as an evolution of the OPC Classic standard, it -supports real-time data access, historical data retrieval, and event notifications, making it ideal for industrial IoT -and Industry 4.0 environments. - -AnyLog can act as an OPC-UA client, pulling data from any OPC-UA server and streaming it into local databases. - ---- - -## Explore the server - -### Get namespaces - -In OPC UA, namespaces organize and uniquely identify nodes in the address space of a server. Each namespace is assigned -a unique index (e.g. `ns=0`, `ns=1`, `ns=2`) used in Node IDs — for example `ns=1;s=TemperatureSensor`. - -```anylog -get opcua namespace where url = [connect string] and user = [username] and password = [password] -``` - -Details: -* `url` — the endpoint of the OPC UA server. -* `user` — the username required by the OPC UA server for access. -* `password` — the password associated with the username. - -Example: -```anylog -get opcua namespace where url = opc.tcp://10.0.0.111:53530/OPCUA/SimulationServer -``` - -### Traverse the address space tree - -The OPC UA tree organizes the server's address space into a hierarchical model resembling a file system. At the root -level, predefined folders like **Objects**, **Types**, and **Views** provide entry points into the address space. The -**Objects** folder contains application-specific nodes (devices, sensors, systems), while the **Types** folder defines -the structure and behavior of nodes (ObjectTypes, VariableTypes, DataTypes). Each node can have child nodes, creating a -parent-child hierarchy. - -The `get opcua struct` command navigates the tree and produces different outputs based on the command variables. The -traversal starts from the root, unless a node is specified to serve as the root. - -```anylog -get opcua struct where url = [connect string] and [options] -``` - -| Option | Description | -|---|---| -| `url` | OPC UA server endpoint. | -| `user` / `password` | Credentials required by the server. | -| `node` | Override the root node by providing a node ID (e.g. `ns=6;s=MyObjectsFolder`). | -| `type` | Filter by node type: `Object`, `Variable`, etc. If not specified, all types are visited. | -| `attributes` | Attribute names to consider, or `*` for all. | -| `class` | Filter the traversal to nodes in the listed class. | -| `depth` | Limit traversal by depth. | -| `limit` | Limit traversal by number of nodes visited. | -| `output` | Target for the output stream (`stdout` or a file name). | -| `append` | If output is a file, a `true` value appends to it (default `false`). | -| `format` | Output format (see below). | -| `validate` | If `true`, reads each node's value to confirm it is readable (see below). | -| `schema` | If `true`, includes the table schema for each tag. | -| `dbms` / `table` | Used when generating `run_client` or `policy` output. | -| `frequency` | Used when generating `run_client` output. | -| `target` | Variables for the `blockchain insert` commands (used with `format = policy`). | - -**Format options:** - -| Format | Output | -|---|---| -| `tree` | OPC-UA tree structure (default). | -| `path` | Full path strings for each node. | -| `stats` | Count of entries per class. | -| `get_value` | Generates get opcua value commands for the visited nodes. | -| `run_client` | Generates run opcua client commands for the visited nodes. | -| `policy` | Generates a policy per tag; combine with `target` for `blockchain insert` commands. | - -**The validate option:** -* The default value is `false`. -* If set to `true`, the value of each considered node is read during traversal. -* If the read fails: - * When `format` is `get_value` or `run_client`, the node is not included. - * In other cases, the output includes a `validate` attribute assigned `success` or `failure`. - * The summary chart includes a counter for the number of nodes that failed to generate a value. - -**Traversal examples:** - -```anylog -# Browse from root, limit 10 nodes -get opcua struct where url = opc.tcp://10.0.0.111:53530/OPCUA/SimulationServer and output = stdout and limit = 10 - -# Direct the output to a file -get opcua struct where url = opc.tcp://10.0.0.111:53530/OPCUA/SimulationServer and output = !prep_dir/opcua_tree.txt and limit = 10 - -# Browse from root, limit by depth -get opcua struct where url = opc.tcp://10.0.0.111:53530/OPCUA/SimulationServer and output = stdout and depth = 4 - -# Browse from a specific node -get opcua struct where url = opc.tcp://10.0.0.111:53530/OPCUA/SimulationServer and output = stdout and node="ns=6;s=MyObjectsFolder" - -# Browse from a specific node, including attribute info -get opcua struct where url = opc.tcp://10.0.0.111:53530/OPCUA/SimulationServer and output = stdout and node="ns=6;s=MyObjectsFolder" and attributes = * - -# Limit by depth and filter by classes "variable" and "object" -get opcua struct where url = opc.tcp://10.0.0.111:53530/OPCUA/SimulationServer and output = stdout and depth = 4 and class = variable and class = object - -# Variables only, generate a get opcua value command -get opcua struct where url = opc.tcp://10.0.0.111:53530/OPCUA/SimulationServer and node="ns=6;s=MyObjectsFolder" and class = variable and format = get_value - -# Variables only, generate a run opcua client command -get opcua struct where url = opc.tcp://10.0.0.111:53530/OPCUA/SimulationServer and node="ns=6;s=MyObjectsFolder" and class = variable and format = run_client and name = opcua_nov and dbms = nov and table = sensor and frequency = 10 and limit = 10 - -# Variables only, output the path of each node -get opcua struct where url = opc.tcp://127.0.0.1:4840/freeopcua/server and format = path and node = "ns=2;s=DeviceSet" and class = variable and dbms = my_dbms - -# Variables only, generate a policy per node -get opcua struct where url = opc.tcp://127.0.0.1:4840/freeopcua/server and format = policy and limit = 100 and node = "ns=2;s=DeviceSet" and class = variable and dbms = my_dbms - -# Variables only, generate blockchain insert commands for each policy, written to a file -get opcua struct where url = opc.tcp://127.0.0.1:4840/freeopcua/server and format = policy and limit = 100 and node = "ns=2;s=DeviceSet" and class = variable and dbms = my_dbms and target = "local = true and master = !master_node" and output = !tmp_dir/my_file.out -``` - ---- - -## Read node values - -Node values are retrieved with the `get plc values` command: - -```anylog -get plc values where type = opcua and url = [connect string] and user = [username] and password = [password] and node = [node id] -``` - -| Option | Description | -|---|---| -| `url` | OPC UA server endpoint. | -| `user` / `password` | Credentials required by the server. | -| `node` | One or more node IDs. | -| `nodes` | A comma-separated list of nodes within square brackets. | -| `include` | Additional attributes returned with the value: `id`, `name`, `source_timestamp`, `server_timestamp`, `status_code`, or `all`. | -| `method` | `collection` (default, a single read pulls all listed nodes) or `individual` (one read per node, used to identify the node causing failures). | -| `failures` | Requires `method = individual`. `false` (default) collects successful and failed reads; `true` collects only failed reads. | - -The `include` attributes: -* `id` — the id of the attribute. -* `name` — the attribute name. -* `source_timestamp` — the timestamp of the value as determined by the data source (e.g. a sensor or device). -* `server_timestamp` — the timestamp assigned by the OPC UA server when the value was received or processed. -* `status_code` — the status of the value (e.g. Good, Bad, Uncertain). - -> Note: if `include` is assigned the keyword `all`, all attributes are included in the output. - -Examples: -```anylog -get plc values where type = opcua and url = opc.tcp://10.0.0.111:53530/OPCUA/SimulationServer and node = "ns=0;i=2257" and node = "ns=0;i=2258" and include = all - -# List format -get plc values where type = opcua and url = opc.tcp://10.0.0.111:53530/OPCUA/SimulationServer and nodes = ["ns=4;s=AirConditioner_1.StateCondition.EventType","ns=4;s=AirConditioner_1.StateCondition.SourceNode"] -``` - ---- - -## Continuous data pull - -Stream data from OPC-UA into the local database continuously: - -```anylog -run plc client where type = opcua and name = [unique name] and url = [connect string] and frequency = [seconds] and dbms = [dbms] and node = [node id] -``` - -| Option | Description | -|---|---| -| `name` | Unique client name. | -| `frequency` | Read frequency in seconds, or in hz (e.g. `10 hz`). | -| `node` / `nodes` | One or more node IDs. | -| `policy` | Use a policy to determine nodes and table (alternative to specifying nodes inline). | -| `table` | The table to host the data (if not specified in a policy). | -| `topic` | Route data through the local broker. | - -Each row is stored with two columns added automatically: -- `timestamp` — the earliest source timestamp of the values considered (if `source_timestamp` is missing, the `server_timestamp` is used). -- `duration` — the number of milliseconds between the earliest and latest timestamp considered in this read. - -Examples: -```anylog -# Individual nodes -run plc client where type = opcua and name = myopcua and url = opc.tcp://10.0.0.111:53530/OPCUA/SimulationServer and frequency = 10 and dbms = nov and table = sensor and node = "ns=0;i=2257" and node = "ns=0;i=2258" - -# Using a node list -run plc client where type = opcua and name = myopcua and url = opc.tcp://10.0.0.111:53530/OPCUA/SimulationServer and frequency = 10 and dbms = nov and table = sensor and nodes = ["ns=0;i=2257","ns=0;i=2258"] -``` - -> Multiple OPC-UA clients can run on the same node simultaneously. - -### Check client status - -```anylog -get plc client -``` - -### Stop a client - -```anylog -exit plc client [client name] -``` - -The client name is the policy ID or `[dbms name].[table name]`. If the client name is `all`, all clients are terminated. - -```anylog -exit plc all -exit plc nov.rig8 -``` - ---- - -## OPC-UA with aggregations - -Aggregation functions summarize streaming data over a time interval, enabling real-time analytics without storing raw -data. See details in the Aggregations section. - -### 1. Identify the time and value column names - -To apply aggregation, identify the names of the time attribute and the value attribute retrieved from the OPC UA -connector. If the AnyLog OPC UA service is used, the time attribute name is `timestamp`, and the value attribute name can -be retrieved with `get plc values` using `include = all` or `include = name`. - -```anylog -get plc values where type = opcua and url = opc.tcp://uademo.prosysopc.com:53530/OPCUA/SimulationServer and node = "ns=3;i=1002" and include = name -``` - -```text -OPCUA Nodes values -name value -------|----------| -random|-0.5909728| -``` - -The call above shows that the column name for `ns=3;i=1002` is `random`. - -### 2. Declare the aggregation - -```anylog -set aggregations where dbms = nov and table = table_2 and time_column = timestamp and value_column = random -``` - -### 3. (Optional) Replace raw data with aggregated data - -```anylog -set aggregations encoding where dbms = nov and table = table_2 and encoding = bounds -``` - -### 4. Validate the aggregation declarations - -```anylog -get aggregations -get aggregations config -``` - -### 5. Start the OPC-UA client - -```anylog - -``` - -### 6. Validate processing - -```anylog -get plc client -get aggregations -get aggregations where dbms = nov and table = table_2 -get streaming -get operator -``` - -### 7. Query aggregated results - -```anylog -run client () sql nov format = table select timestamp::ljust(19), end_interval::ljust(19), min_val, max_val, avg_val, events from bounds_table_2 order by timestamp desc limit 10 -``` - -> **Worth verifying:** unlike every other SQL query example across this doc set, the query text above isn't wrapped -> in quotes (`"select ..."`). This is consistent in both source files, so it isn't something either merge -> introduced — but it's worth confirming this is genuinely valid syntax for `run client () sql` rather than a -> pre-existing typo that was carried through both docs. - ---- - -## Policy-based tag management - -For large OPC-UA deployments, generate a policy for each tag and store it on the blockchain. Policies define the -structure and semantics of the tags (names, data types, relationships) and serve as a mapping between table names and tag -information. This lets AnyLog automatically interpret and organize incoming data, aligning it with the defined structure -for querying, validation, and distribution across the network. - -### Generate the policies - -```anylog -get opcua struct where url = opc.tcp://127.0.0.1:4840/freeopcua/server and format = policy and schema = true and node = "ns=2;s=DeviceSet" and class = variable and dbms = my_dbms and target = "local = true and master = !master_node" and output = !tmp_dir/my_file.out -``` - -The tag policies are stored in the file `!tmp_dir/my_file.out`, in a format like: - -```json -{"tag": {"class": "variable", - "datatype": "Boolean", - "dbms": "my_dbms", - "nodeid": "LS1002H_AlarmSetpoint", - "ns": 2, - "parent": "ALARM_TAGS", - "path": "Root/Objects/DeviceSet/WAGO 750-8210 PFC200 G2 4ETH XTR/Resources/Application/GlobalVars/ALARM_TAGS/LS1002H_AlarmSetpoint", - "table": "t39"}} -``` - -If `schema` is set to `true`, the output includes, for every tag, the table schema associated with the tag: - -```json -{"table": {"name": "t39", - "dbms": "nov", - "create": "CREATE TABLE IF NOT EXISTS t39(row_id SERIAL PRIMARY KEY, insert_timestamp TIMESTAMP NOT NULL DEFAULT NOW(), tsd_name CHAR(3), tsd_id INT, timestamp timestamp not null default now(), value bool ); CREATE INDEX t39_timestamp_index ON t39(timestamp); CREATE INDEX t39_insert_timestamp_index ON t39(insert_timestamp);", - "source": "OPCUA Interface", - "id": "040197b7eed831dddb1b3fd910d86deb", - "date": "2025-04-09T00:09:53.406292Z", - "ledger": "local"}} -``` - -### Load the policies to the metadata - -```anylog -process !tmp_dir/my_file.out -``` - -### Generate the command to read the tag data - -```anylog -get opcua struct where url = opc.tcp://127.0.0.1:4840/freeopcua/server and format = run_client and node = "ns=2;s=DeviceSet" and class = variable and output = !tmp_dir/my_run_cmd.out and dbms = my_dbms and frequency = 3 and name = opcua_nov -``` - -Notes: -* The run opcua client command is stored in the file `!tmp_dir/my_run_cmd.out`. -* The `table` name is not specified, as it is derived from the policies (based on the namespace and node id). - -### Execute the command - -```anylog -process !tmp_dir/my_run_cmd.out -``` - -This pulls the data using OPC-UA and assigns it to the tables according to the info in the policies. \ No newline at end of file diff --git a/04- Southbound Interfaces/03- Industrial Connectors/03- EtherIP.md b/04- Southbound Interfaces/03- Industrial Connectors/03- EtherIP.md deleted file mode 100644 index 2867c6bd..00000000 --- a/04- Southbound Interfaces/03- Industrial Connectors/03- EtherIP.md +++ /dev/null @@ -1,193 +0,0 @@ ---- -title: "EtherNet/IP" -description: "Configure AnyLog as an EtherNet/IP client to pull data from industrial PLCs and controllers continuously." -layout: page -source_path: "03 EtherNet IP.md" ---- - - - -# EtherNet/IP - -EtherNet/IP (Ethernet Industrial Protocol) is an industrial network protocol built on the Common Industrial Protocol (CIP) that enables communication between PLCs, sensors, actuators, and control systems over standard Ethernet. AnyLog can act as an EtherNet/IP client, pulling data from any EtherNet/IP device and streaming it into local databases. - -## The EtherNet/IP Structure - -EtherNet/IP organizes industrial automation data through a set of well-defined CIP (Common Industrial Protocol) objects, -which represent device attributes, configurations, and runtime data. Unlike OPC UA's tree-based model, EtherNet/IP uses a flat, -object-oriented structure where each device exposes standard or vendor-specific classes, instances, and attributes. -These are accessed using CIP messaging over Ethernet. - -Each class (such as Identity Object, Assembly Object, or Connection Object) may contain multiple instances, and each instance -can expose multiple attributes, forming a structured view of the device's capabilities and status. -While the structure is not hierarchical like OPC UA, it provides a standardized way to navigate and interact with device data. - -The `get etherip struct` command explores the structure by querying supported classes and retrieving their instances and attributes. -This provides insight into the connected PLC or device, including program tags and system-level data. - -The EtherNet/IP structure is explored with the following command: - -```anylog -get etherip struct where url = [connect string] and user = [username] and password = [password] and ... -``` -This command enables users to query both system-level and user-defined tags, making it easier to explore and interact with a PLC’s data structure over EtherNet/IP. - -### Command Variables - -| Keyword | Details | -|-------------|----------------------------------------------------------------------------------------------------------------------------------------| -| `url` | The IP address of the target PLC or EtherNet/IP device. | -| `slot` | The slot number of the target controller (used in multi-slot chassis). | -| `user` | Username, if the PLC requires authentication. | -| `password` | Password for authentication. | -| `limit` | Limit the number of tags or objects returned in the response. | -| `prefix` | Limit the tags to a path that satisfies the prefix string. | -| `output` | The target for the output stream (stdout or a file name). | -| `format` | The format of the output (see details below). | -| `target` | The variables in the 'blockchain insert commands'. This option is used with 'format = policy' to generate 'blockchain insert' commands | -| `schema` | A boolean value. If set to True, output includes, for each tag, the table's schema. | -| `frequency` | Specifying the reading frequency in Hz, with **format = run_client**. | -| `name` | Specifying a process name when with **format = run_client** option. | - -Format options: -* tree - the OPC-UA tree structure (default). -* policy - generating a policy representing the tag. If target is specified, output is "blockchain insert" command for every policy. -* get_value - generating a get plc value command with the tags visited in the **get plc struct** command. -* run_client - generating a run plc client command with the tags visited in the **get plc struct** command. - -### Traversal examples - -```anylog -# Browse all tags, show current values -get etherip struct where url = 127.0.0.1 and read = true - -# Generate a get plc values command -get etherip struct where url = 127.0.0.1 and format = get_value - -# Generate a run plc client command -get etherip struct where url = 127.0.0.1 and format = run_client and frequency = 1 and name = etherip_reads and dbms = my_dbms - -# Generate blockchain insert commands for all tags (includes schema) -get etherip struct where url = 127.0.0.1 and format = policy and schema = true \ - and dbms = my_dbms and target = "local = true and master = !master_node" \ - and output = !tmp_dir/my_file.out -``` - ---- - -## Read tag values - -```anylog -get plc values where type = etherip and url = [connect string] and node = [tag name] -``` - -Options: - -| Option | Description | -|---|---| -| `node` | One or more tag names | -| `nodes` | Comma-separated list of tag names in square brackets | - -Examples: -```anylog -get plc values where type = etherip and url = 127.0.0.1 \ - and node = CombinedChlorinatorAI.PV and node = STRUCT.Status - -# List format -get plc values where type = etherip and url = 127.0.0.1 \ - and nodes = ["CombinedChlorinatorAI.PV", "STRUCT.Status"] -``` - -## The Run PLC Client Command - -The command **run plc client*** pulls data from the PLC continuously and streams the data into a database on the local node: -```anylog -run plc client where type = etherip and name = [unique name] and url = [connect string] and frequency = [frequency] and dbms = [dbms name] and table = [table name] and node = [node id]] -``` - -The following tables summarizes the command variables: - -| keyword | Details | -|-----------|----------------------------------------------------------------------------------------------| -| name | A unique connection name. | -| url | The url specifies the endpoint of the OPC UA server. | -| user | the username required by the OPC UA server for access. | -| password | the password associated with the username. | -| frequency | Read frequency in seconds or a fraction of seconds using hz (i.e.: 10 hz). | -| node | ID of one or multiple nodes that their value is retrieved. | -| nodes | Providing a list of nodes, separated by comma, within square brackets. | -| policy | If nodes are not specified on the CLI, the policy determines the nodes and the table to use. | -| dbms | The database to host the data (if not specified in a policy). | -| table | The table to host the data (if not specified in a policy). | -| topic | If data is processed through the local broker. | - -Example 1 - listing individual tags: -```anylog -run plc client where type = etherip and name = etherip_reads and url = 127.0.0.1 and frequency = 1 and dbms = my_dbms and node = FreeChlorinatorAI.PV and node = CombinedChlorinatorAI.PV -``` -Example 2 - providing a list of tags: -```anylog - -``` - -Notes: -1. Multiple clients can be declared on the same node. -2. Each row is added with 2 columns: - * Timestamp - representing the earliest source_timestamp of the values considered (if source_timestamp is missing, the server_timestamp is considered). - * Duration - the number of milliseconds between the earliest timestamp and the latest timestamp that were considered in the values that were retrieved from the PLC. - ---- - -## Policy-based tag management - -For large EtherNet/IP deployments, generate policies for each tag and publish them to the blockchain. This lets AnyLog automatically map incoming data to the correct tables without specifying nodes inline. - -### 1. Generate and publish policies - -```anylog -# Generate policy file (includes schema if schema = true) -get etherip struct where url = 127.0.0.1 and format = policy and schema = true \ - and dbms = my_dbms and target = "local = true and master = !master_node" \ - and output = !tmp_dir/my_file.out - -# Publish to blockchain -process !tmp_dir/my_file.out -``` - -The generated tag policy looks like: -```json -{"tag": { - "protocol": "etherip", - "ns": 0, - "dbms": "my_dbms", - "table": "t101", - "datatype": "boolean", - "node_sid": "BOOL", - "id": "0e17856bdb914cdfe338eff3485ef366", - "date": "2025-05-04T18:07:54.695893Z", - "ledger": "local" -}} -``` - -If `schema = true`, the output also includes a `CREATE TABLE` statement for each tag. - -### 2. Generate and run the data pull command - -```anylog -# Generate run_client command file (table derived from policies — no table= needed) -get etherip struct where url = 127.0.0.1 and format = run_client \ - and frequency = 1 and name = etherip_reads and dbms = my_dbms \ - and output = !tmp_dir/my_run_cmd.out - -# Execute -process !tmp_dir/my_run_cmd.out -``` - -This pulls data continuously and assigns each tag's values to the correct table based on the published policies. diff --git a/04- Southbound Interfaces/03- Industrial Connectors/04- DNP3.md b/04- Southbound Interfaces/03- Industrial Connectors/04- DNP3.md deleted file mode 100644 index 071ea127..00000000 --- a/04- Southbound Interfaces/03- Industrial Connectors/04- DNP3.md +++ /dev/null @@ -1,317 +0,0 @@ ---- -title: DNP3 -description: Pull data from DNP3 outstations into AnyLog as master over TCP or TLS using run plc client. -layout: page -source_path: "06 DNP3.md" ---- - - - -AnyLog can act as a **DNP3 master** over **TCP** or **TLS** (using **hostname** and **port**, default **20000**). -Data is read on a schedule and streamed into your local operator database as JSON, using the same **`run plc client`** -pattern as Modbus, OPC-UA, and EtherNet/IP. - -For a production-style deployment of a DNP3 connection from a standing script (rather than one-off CLI commands), see -Deploying a DNP3 Connector via Script. For -reusing a single point map across multiple outstations instead of repeating it inline, see DNP3 Mapping Policies. - ---- - -## Prerequisites - -| Requirement | Notes | -|---|---| -| **`opendnp3` (yadnp3)** | Must be installed in the AnyLog runtime environment. | - -Declare the target DBMS before streaming, for example: - -```anylog -connect dbms new_company where type = sqlite -``` - ---- - -## Connection and map - -DNP3 uses **`hostname`** and **`port`**. **`master_id`** and **`outstation_id`** are the DNP3 link addresses (defaults: master **1**, outstation **10**). - -**`map`** is a **JSON array** of points. Each object must include: - -| Key | Required | Meaning | -|---|---|---| -| `name` | yes | Column / logical label | -| `index` | yes | Point index on the outstation | -| `type` | one of `type` or `group`+`variation` | Point class (see table below) | -| `group`, `variation` | alternative to `type` | Raw DNP3 group/variation | - -Supported **`type`** values (case-insensitive; spaces/underscores ignored): - -| `type` | DNP3 group / variation | -|---|---| -| `Analog` | 30 / 6 | -| `Binary` | 1 / 2 | -| `BinaryOutputStatus` | 10 / 2 | -| `AnalogOutputStatus` or `AnalogOutput` | 40 / 4 | -| `Counter` | 20 / 6 | -| `DoubleBit` or `DoubleBitBinary` | 3 / 2 | - -`map` can be written inline, as shown throughout this page, or declared once as a reusable policy on the -blockchain and referenced across multiple connections — see DNP3 Mapping Policies. - -Optional **TLS** (all three PEM paths required when `enable_tls = true`): - -| Keyword | Meaning | -|---|---| -| `enable_tls` | Use TLS instead of plain TCP | -| `tls_ca` | CA certificate (PEM file path) | -| `tls_cert` | Client certificate (PEM file path) | -| `tls_key` | Client private key (PEM file path) | - ---- - -## One-shot read - -```anylog - -``` - -Alias: **`get dnp3 values`** (same keywords). - -With TLS (see DNP3 TLS Test Certificates for generating a chain to use here, and DNP3 Out Station Testing below for running a matching test outstation): - -```anylog - -``` - ---- - -## Continuous ingest — wide table (default) - -With **`table = ...`** and **`dbms`**, all points from **`map`** land in **one table**. Each poll inserts **one row**; every object in **`map`** is **one column**: the map **`name`** is the **column name**, and that column stores the value read for that point. - -```anylog - -``` - ---- - -## Continuous ingest — dynamic tables (`dynamic = true`) - -Omit **`table`** and omit **`namespace`** for plain dynamic ingest. Each object in **`map`** is written to its **own table**. The table name is derived from the client **`name`** and the map **`name`** (for example, **`dnp3_rtu_analog_0`** when **`name = dnp3_rtu`** and the map entry's **`name`** is **`analog_0`**). Each row includes **`timestamp`**, **`tag`**, and **`value`**. - -```anylog - -``` - ---- - -## Dynamic ingest with UNS (`namespace` + `master_node`) - -With **`dynamic = true`**, you can add a **Unified Namespace** path and a **master node** so DNP3 ingest is registered in the UNS alongside your policies and DBMS. **`namespace`** requires **`master_node = [ip:port]`** for policy updates. - -```anylog -run plc client where type = dnp3 and - hostname = 127.0.0.1 and - port = 20001 and - master_id = 1 and - outstation_id = 10 and - frequency = 20 and - name = dnp3_uns and - dbms = new_company and - dynamic = true and - master_node = 192.168.1.88:32048 and - namespace = FA9/MID9/DEVICE9 and - map = [{"name":"analog_0","type":"Analog","index":0}] -``` - -**Table names** follow the **same pattern** as plain **`dynamic = true`** (client **`name`** plus map **`name`**, e.g. **`dnp3_uns_analog_0`** for the example above). Under UNS, the **read value** is usually stored in a **column named like the tag**—the map **`name`** (here **`analog_0`**), not a generic **`value`** column. - -With **`namespace`**, table and column layout follow **UNS policies**. **`namespace`** and **`master_node`** drive how tables are registered in the UNS. See Unified Namespace for background. - -For a full example combining this with a scripted deployment and a reusable mapping policy (rather than an inline `map` array), see Deploying a DNP3 Connector via Script. - ---- - -## DNP3 Out Station Testing - -The opendnp3 library includes a demo outstation for lab tests. Build it with demos enabled (TLS optional): - -```bash -git clone --recursive -b release-2.x https://github.com/dnp3/opendnp3.git -cd opendnp3 -mkdir build && cd build -cmake -DDNP3_DEMO=ON -DDNP3_TLS=ON .. # omit -DDNP3_TLS=ON for TCP-only -make -j -``` - -See the OpenDNP3 CMake guide and TLS support (OpenSSL ≥ 1.1.1 required for TLS). - -### Plain TCP outstation - -From the opendnp3 build directory: - -```bash -cd ~/opendnp3/build -./outstation-demo -``` - -The demo listens on **any IP address**, port **20001**, outstation link id **10**, and expects master link id **1**. Use those values in AnyLog: - -```anylog - -``` - -Once started, the demo logs traffic and waits for input to send unsolicited measurement changes: - -```text -Enter one or more measurement changes then press -c = counter, b = binary, d = doublebit, a = analog, o = octet string, 'quit' = exit -``` - -### TLS outstation - -Generate a test CA chain first — see DNP3 TLS Test Certificates for the full script and instructions. Run it in a working directory of your own, outside this repository: - -```bash -mkdir -p ~/dnp3-tls-test && cd ~/dnp3-tls-test -bash create_certificates.sh -``` - -| Side | CA (peer) | Certificate | Private key | -|------|-----------|-------------|-------------| -| AnyLog master | `anylogDNP3ca.cert` | `master1.cert` | `master1.key` | -| Outstation | `anylogDNP3ca.cert` | `outstation1.cert` | `outstation1.key` | - -From the opendnp3 build directory, start the TLS demo with three PEM paths (CA, outstation certificate, outstation private key): - -```bash -cd ~/opendnp3/build -./outstation-tls-demo \ - ~/dnp3-tls-test/anylogDNP3ca.cert \ - ~/dnp3-tls-test/outstation1.cert \ - ~/dnp3-tls-test/outstation1.key -``` - -Same link ids and port as plain TCP (**master_id = 1**, **outstation_id = 10**, port **20001**). AnyLog master uses the **master** certificate files from the same CA chain. - -Example AnyLog one-shot read: - -```anylog - -``` - -More detail, including the full generation script: DNP3 TLS Test Certificates. - -### Third-party simulator - -Another option is a commercial DNP3 outstation simulator, for example the FreyrSCADA DNP3 development bundle. After download, **DNPOutstationSimulator.exe** under the **Simulator** folder can be installed on Windows (32-bit). - ---- - -## Command keywords (summary) - -| Keyword | Required / notes | -|---|---| -| `type` | `dnp3` | -| `hostname`, `port` | DNP3 TCP/TLS target (default port **20000**) | -| `master_id` | Master link address (default **1**) | -| `outstation_id` | Outstation link address (default **10**) | -| `frequency` | Poll interval (seconds) | -| `name` | Unique client name | -| `dbms` | Target DBMS | -| `table` | Wide-table ingest; omit with **`dynamic = true`** | -| `dynamic` | `true` for per-map tables or UNS | -| `map` | JSON array of points, inline or resolved from a mapping policy | -| `namespace` | UNS path (DNP3 + **`dynamic = true`** only) | -| `master_node` | Required when **`namespace`** is set | -| `enable_tls`, `tls_ca`, `tls_cert`, `tls_key` | Optional TLS (all three PEM paths required) | - ---- - -## Related - -- Adding Data to Nodes in the Network -- Unified Namespace -- DNP3 TLS Test Certificates — generating a local CA chain for TLS testing -- Deploying a DNP3 Connector via Script — a production-style `.al` deployment script, parameters, and error handling -- DNP3 Mapping Policies — reusing a point map across multiple connections instead of repeating it inline -- Modbus -- OPC-UA -- EtherNet/IP diff --git a/04- Southbound Interfaces/03- Industrial Connectors/04-1 DNP3/01- DNP3 - Deploying Connector via Script.md b/04- Southbound Interfaces/03- Industrial Connectors/04-1 DNP3/01- DNP3 - Deploying Connector via Script.md deleted file mode 100644 index 11f9bb97..00000000 --- a/04- Southbound Interfaces/03- Industrial Connectors/04-1 DNP3/01- DNP3 - Deploying Connector via Script.md +++ /dev/null @@ -1,228 +0,0 @@ ---- -title: "Deploying a DNP3 Connector via Script" -description: Walking through a production-style .al deployment script for a DNP3 southbound connection — parameters, policy reuse, TLS branching, and error handling -layout: page -source_path: "01- DNP3 - Deploying Connector via Script.md" ---- - - - -# Deploying a DNP3 Connector via Script - -DNP3 documents the `get plc values` / `run plc client` commands directly — typed once on the CLI, or -pasted into a one-off script, with the point `map` written out as a literal JSON array each time. For a -production deployment, it's more common to run DNP3 connections from a standing `.al` script that a node -executes on startup (or on demand), with parameters set at the top and the point map itself stored once as a -reusable blockchain policy rather than duplicated in every script. This page walks through that pattern using a -sample script, `dnp3_connector.al`. - -For the reusable-mapping-policy pattern itself (the `dnp3` policy type this script checks for and creates), see -DNP3 Mapping Policies — Reusing a Schema Across Connections. - -Run the script with: - -```anylog -process !local_scripts/southbound-industrial/dnp3_connector.al -``` - -(adjust the path to wherever the script actually lives in your deployment). - ---- - -## Script structure - -The script is organized as a sequence of labels, executed with `goto`/fall-through, rather than top-to-bottom — -this lets it branch (e.g. TLS vs. plain TCP) and handle errors from a shared sub-script without duplicating -logic. - -### `:set-params:` — connection and identity - -```anylog -on error ignore -:set-params: -client_type=dnp3 -dnp_ip = 192.168.1.88 -dnp_port = 20001 -dnp_master_id = 1 -outstation_id = 10 -dnp_frequency = 20 -dnp_name = plant1 -base_namespace = "FACTORY4/DNP3/SUBSTATION" - -# authentication configs -set enable_tls = false - -# !anylog_dir is accessible as a volume and used to store certifications and access points. -# we can also store the public information on the blockchain so there's no need for persistence of content -tls_ca = !anylog_dir/dnp3_certs/factory_ca.cert -tls_cert = !anylog_dir/dnp3_certs/master1.cert -tls_key = !anylog_dir/dnp3_certs/master1.key -``` - -Everything a given deployment needs to change — target outstation, link IDs, poll frequency, the UNS namespace -this connection is registered under, and whether TLS is used — is set once here. `dnp_name` and -`base_namespace` together are also the key used to look up (or create) the reusable mapping policy in the next -step, so they double as the identity of this particular DNP3 connection, not just a display label. - -Note on TLS paths: `!anylog_dir` is a volume mounted into the AnyLog container, used here to hold certificate -files locally on the node. The comment in the script also flags an alternative worth knowing about — the -*public* half of a certificate can instead be stored on the blockchain as part of a policy, avoiding the need to -persist certificate files on every node that needs them. This script uses the local-file approach; see -DNP3 TLS Test Certificates for generating -a chain to use with `tls_ca`/`tls_cert`/`tls_key`. - -### `:check-policy:` — look for an existing mapping policy - -```anylog -:check-policy: -is_dnp3 = blockchain get dnp3 where namespace = !base_namespace and name=!dnp_name -if not !is_dnp3 then goto prep-policy - -dnp_schema = from !is_dnp3 bring [*][schema] - -goto declare-dnp3 -``` - -If a `dnp3` policy already exists for this `namespace`/`name` pair, its `schema` (the point map) is pulled out -into `!dnp_schema` and the script jumps straight to declaring the connection. If not, it falls through to -`:prep-policy:` to create one. - -### `:prep-policy:` — build a new mapping policy - -```anylog -:prep-policy: - -``` - -This is the point map — the same shape documented as the `map` array in DNP3 — -but declared once, as data, rather than repeated inline in every `run plc client` call. - -### `:publish-policy:` — sign and insert the policy - -```anylog -:publish-policy: -process !local_scripts/node-deployment/policies/publish_policy.al -if not !error_code.int then -do set create_policy = true -goto check-policy - -if !error_code == 1 then goto sign-policy-error -else if !error_code == 2 then goto prepare-policy-error -else if !error_code == 3 then goto declare-policy-error -``` - -This delegates to a shared helper script (`publish_policy.al`) that signs `!new_policy` and inserts it onto the blockchain — -the same helper any connector's deployment script would call, not something DNP3-specific. On success, the script loops -back to `:check-policy:`, which will now find the freshly published policy and pick up `!dnp_schema` from it. A nonzero -`!error_code` routes to one of three error labels depending on which stage failed (signing, preparing, or declaring the -policy) — see Error handling below. - -### `:declare-dnp3:` / `:declare-dnp3-tls:` — start the connection - -```anylog -:declare-dnp3: -on error goto declare-dnp3-err -if !enable_tls == true goto declare-dnp3-tls - -goto end-script - -:declare-dnp3-tls: - -``` - -Both branches are the same `run plc client where type = dnp3 ...` call documented in -DNP3 — Dynamic ingest with UNS; the only difference is -whether the four `enable_tls`/`tls_*` keywords are included. `!dnp_schema` (from either `:check-policy:` or -`:prep-policy:` → `:publish-policy:` → `:check-policy:`) supplies `map`, and `!base_namespace` / -`!dnp_name` supply `namespace` — so the UNS registration and the policy lookup key are the same values, -by design. - -### Error handling - -```anylog -:end-script: -end script - -:terminate-scripts: -exit scripts - -:sign-policy-error: -print "Failed to sign mapping policy" -goto terminate-scripts - -:prepare-policy-error: -print "Failed to prepare mapping policy for publishing on blockchain" -goto terminate-scripts - -:declare-policy-error: -print "Failed to declare mapping policy on blockchain" -goto terminate-scripts - -:declare-dnp3-err: -print "Failed to define connection to DNP3 against" + !dnp_ip + ":" + !dnp_port -goto terminate-scripts -``` - -Each failure path prints a specific message identifying which stage failed (signing, preparing, or declaring the -policy; or establishing the DNP3 connection itself) before terminating the script — useful for diagnosing a -failed deployment from logs without needing to instrument the script further. - ---- - -## Adapting this for your own deployment - -At minimum, change in `:set-params:`: - -- `dnp_ip`, `dnp_port`, `dnp_master_id`, `outstation_id` — to match your actual outstation. -- `dnp_name`, `base_namespace` — pick values that uniquely identify this connection; these double as the lookup - key for the reusable mapping policy. -- The `schema` array in `:prep-policy:` — to match the actual points on your outstation (see - DNP3 — Connection and map for the supported `type` values). -- `enable_tls` and the three `tls_*` paths, if using TLS. - -If you're deploying several identical or near-identical outstations (for example, several substations with the -same point layout), give them the same `base_namespace`/`dnp_name` pattern deliberately, so later deployments hit -the `:check-policy:` fast path and reuse the schema already published by the first one, rather than re-declaring -it. See DNP3 Mapping Policies for more on this reuse pattern. diff --git a/04- Southbound Interfaces/03- Industrial Connectors/04-1 DNP3/02- DNP3 - Mapping-Policies.md b/04- Southbound Interfaces/03- Industrial Connectors/04-1 DNP3/02- DNP3 - Mapping-Policies.md deleted file mode 100644 index 787f3193..00000000 --- a/04- Southbound Interfaces/03- Industrial Connectors/04-1 DNP3/02- DNP3 - Mapping-Policies.md +++ /dev/null @@ -1,144 +0,0 @@ ---- -title: "DNP3 Mapping Policies — Reusing a Schema Across Connections" -description: Storing a DNP3 point map once as a blockchain policy and reusing it across multiple connections, instead of repeating the map JSON inline -layout: page -source_path: "02- DNP3 - Mapping-Policies.md" ---- - - - -# DNP3 Mapping Policies — Reusing a Schema Across Connections - -DNP3 documents `map` as a JSON array supplied directly in the `get plc values` / `run plc client` -command — every example on that page writes the point list out in full, inline. That's the right approach for a -single connection, or for testing. For a deployment with several outstations that share the same, or nearly the -same, point layout — several substations of the same model, for instance — repeating an identical `map` array in -every connection command means the same schema has to be kept in sync by hand in multiple places. - -The alternative: declare the point map once as a policy on the blockchain, and have each connection look it up -by name instead of restating it. - -## The `dnp3` policy type - -The policy has one required shape: - -```json -{ - "dnp3": { - "namespace": "FACTORY4/DNP3/SUBSTATION", - "name": "plant1", - "schema": [ - {"name":"analog_0","type":"Analog","index":0}, - {"name":"binary_0","type":"Binary","index":0}, - {"name":"analog_output_status_0","type":"AnalogOutputStatus","index":0}, - {"name":"binary_output_status_0","type":"BinaryOutputStatus","index":0} - ] - } -} -``` - -- `namespace` + `name` together identify the policy — this pair is the lookup key, and not coincidentally, it's - the same `namespace`/`name` pair used when registering the connection in the UNS (see - DNP3 — Dynamic ingest with UNS). Using the same - values for both isn't required by the platform, but it keeps one connection's identity, UNS registration, and - mapping policy all pointing at the same pair of values instead of three independently-tracked names. -- `schema` is exactly the same array documented as `map` in - DNP3 — Connection and map — the same `name`/`type`/`index` (or - `group`/`variation`) objects, just stored as policy data instead of typed inline. - -## Looking up an existing policy - -```anylog -is_dnp3 = blockchain get dnp3 where namespace = !base_namespace and name = !dnp_name -dnp_schema = from !is_dnp3 bring [*][schema] -``` - -`!dnp_schema` now holds the same JSON array that `map` expects, pulled from whichever policy already matches this -`namespace`/`name` pair — whether that policy was published by this exact connection previously, or by an -earlier, identical outstation deployment reusing the same namespace/name convention. - -## Publishing a new policy - -If no matching policy exists yet, build one and insert it — following the same -sign → insert -pattern used for any other policy type: - -```anylog - -blockchain insert where policy = !new_policy and local = true and master = !ledger_conn -``` - -(A full deployment script wraps this with signing and error handling — see -Deploying a DNP3 Connector via Script -for the complete pattern, including the check-then-create flow that avoids republishing an identical policy on -every run.) - -## Using the resolved schema - -Whether `!dnp_schema` came from an existing policy or was just published, it's used exactly where `map` appears -in the standard `run plc client` command: - -```anylog - -``` - -This is identical to every other `run plc client` example in DNP3 — the only difference is that `map` -is populated from a variable resolved via a policy lookup, rather than written out as a literal array in the -command itself. - -## How this differs from a "mapping policy" - -AnyLog also has a generic **mapping policy** type, covered in -Mapping data to tables, used for REST/MQTT ingestion — those policies -describe a `bring`-based extraction schema (`{"type": "float", "bring": "[reading]"}`) for pulling fields out of -an arbitrary incoming JSON payload. The `dnp3` policy type here is a different, protocol-specific shape: it -describes DNP3 point addresses (`type`/`index`, or `group`/`variation`) on an outstation, not a JSON-payload -extraction rule. The two aren't interchangeable, and a `dnp3` policy isn't referenced via the generic `mapping` -keyword — it's looked up directly with `blockchain get dnp3 where ...`, as shown above. - -## Why bother — the actual benefit - -With several outstations sharing a `dnp3` policy (by using the same `namespace`/`name` convention across -deployments of otherwise-identical equipment): - -- The point schema is edited in exactly one place — updating the policy — rather than in every deployment - script that connects to a matching outstation. -- New deployments of the same equipment type look the schema up instead of needing it re-specified, reducing the - chance of a typo or drift between two supposedly-identical connections. -- The schema itself becomes queryable and auditable via the blockchain, the same as any other policy, rather - than living only inside script files. - -The tradeoff is a layer of indirection: reading a deployment script alone (without also checking the published -policy) won't tell you the actual point map in use. For a one-off connection or a quick test, the inline `map` -array shown throughout DNP3 remains simpler. - -## See also - -- DNP3 — the underlying `get plc values`/`run plc client` command reference -- Deploying a DNP3 Connector via Script — a full deployment - script using this pattern, with TLS branching and error handling \ No newline at end of file diff --git a/04- Southbound Interfaces/03- Industrial Connectors/04-1 DNP3/03- DNP3 - TLS test certificates.md b/04- Southbound Interfaces/03- Industrial Connectors/04-1 DNP3/03- DNP3 - TLS test certificates.md deleted file mode 100644 index 51a02dc5..00000000 --- a/04- Southbound Interfaces/03- Industrial Connectors/04-1 DNP3/03- DNP3 - TLS test certificates.md +++ /dev/null @@ -1,128 +0,0 @@ ---- -title: "DNP3 TLS Test Certificates" -description: Generating a local CA chain for mutual TLS between an AnyLog DNP3 master and an outstation, for testing only -layout: page -source_path: "DNP3-tls-test-certificates.md" ---- - - - -# DNP3 TLS Test Certificates - -This page generates a small, local CA chain for mutual TLS between an AnyLog DNP3 **master** and an -**outstation** — for example, opendnp3's `outstation-demo`, configured for -TLS. It produces a root CA plus one client certificate (the AnyLog master) and two server certificates -(outstations), all signed by that CA. - -**For testing only.** These are self-signed, lab-only credentials — do not reuse them in production, and do not -commit the generated `.cert`/`.key` files to any repository. Run the script below in a working directory of your -own, outside of version control, and regenerate a fresh chain whenever you need one. - -## Generating the chain - -Save the following as `create_certificates.sh` in an empty working directory, then run `bash -create_certificates.sh`. It has no dependencies beyond `openssl`. - -```bash -#!/usr/bin/env bash -# Generate a small CA chain for DNP3 TLS lab tests (AnyLog master <-> outstation). -# -# Usage: -# mkdir -p ~/dnp3-tls-test && cd ~/dnp3-tls-test -# bash create_certificates.sh -# -# AnyLog master (TLS client): anylogDNP3ca.cert, master1.cert, master1.key -# Outstation (TLS server): anylogDNP3ca.cert, outstation1.cert, outstation1.key -# -set -euo pipefail -cd "$(dirname "$0")" - -CA_CERT=anylogDNP3ca.cert -CA_KEY=anylogDNP3ca.key - -echo "=== CA ===" -openssl req -x509 -new -nodes -newkey rsa:2048 \ - -keyout "${CA_KEY}" -sha256 -days 3600 -out "${CA_CERT}" \ - -subj "/C=US/ST=OR/L=Bend/O=AnyLog Certificate Corp" - -echo "=== CSRs ===" -openssl req -newkey rsa:2048 -nodes -keyout master1.key -out master1.csr \ - -subj "/C=US/ST=OR/O=AnyLogDNP3/CN=master1" -openssl req -newkey rsa:2048 -nodes -keyout outstation1.key -out outstation1.csr \ - -subj "/C=US/ST=OR/O=AnyLogDNP3/CN=outstation1" -openssl req -newkey rsa:2048 -nodes -keyout outstation2.key -out outstation2.csr \ - -subj "/C=US/ST=OR/O=AnyLogDNP3/CN=outstation2" - -echo "=== Signed certs ===" -openssl x509 -req -in master1.csr -CA "${CA_CERT}" -CAkey "${CA_KEY}" -CAcreateserial \ - -out master1.cert -days 3600 -sha256 -openssl x509 -req -in outstation1.csr -CA "${CA_CERT}" -CAkey "${CA_KEY}" -CAcreateserial \ - -out outstation1.cert -days 3600 -sha256 -openssl x509 -req -in outstation2.csr -CA "${CA_CERT}" -CAkey "${CA_KEY}" -CAcreateserial \ - -out outstation2.cert -days 3600 -sha256 - -rm -f master1.csr outstation1.csr outstation2.csr *.srl - -echo "=== Verify chain ===" -openssl verify -CAfile "${CA_CERT}" master1.cert outstation1.cert outstation2.cert - -echo "OK: certificates in $(pwd)" -``` - -This produces 8 files in your working directory: - -| File | Role | -|---|---| -| `anylogDNP3ca.cert` / `anylogDNP3ca.key` | Root CA | -| `master1.cert` / `master1.key` | AnyLog master (TLS client) | -| `outstation1.cert` / `outstation1.key` | Primary test outstation (TLS server) | -| `outstation2.cert` / `outstation2.key` | Second outstation (optional) | - -The script's own cleanup (`rm -f ... *.srl`) already removes the intermediate CSRs and OpenSSL's serial file — -so nothing but the certs and keys above is left behind, and there's no separate `.gitignore` to maintain. If you -do generate the chain inside a version-controlled directory for convenience, add `*.cert`, `*.key`, `*.csr`, and -`*.srl` to that repo's `.gitignore` before running the script. - -## Configuring the AnyLog master - -Point the master's TLS settings at your generated files (use the actual path to your working directory): - -| Keyword | Path | -|---|---| -| `tls_ca` | `/path/to/your/working/dir/anylogDNP3ca.cert` | -| `tls_cert` | `/path/to/your/working/dir/master1.cert` | -| `tls_key` | `/path/to/your/working/dir/master1.key` | - -## Configuring the outstation (TLS server) - -| Setting | File | -|---|---| -| Peer / CA | `anylogDNP3ca.cert` | -| Local certificate | `outstation1.cert` | -| Private key | `outstation1.key` | - -With `opendnp3` built at `~/opendnp3/build`, after generating the chain: - -```bash -cd ~/opendnp3/build -./outstation-tls-demo \ - /path/to/your/working/dir/anylogDNP3ca.cert \ - /path/to/your/working/dir/outstation1.cert \ - /path/to/your/working/dir/outstation1.key -``` - -Plain TCP, no TLS: run `./outstation-demo` instead, from the same directory. - -## See also - -- DNP3 — - the main DNP3 connector doc; this page is referenced from its outstation-testing section. \ No newline at end of file diff --git a/04- Southbound Interfaces/04- Industrial Connectors/01- PLC Mapping.md b/04- Southbound Interfaces/04- Industrial Connectors/01- PLC Mapping.md deleted file mode 100644 index 5e8eb61c..00000000 --- a/04- Southbound Interfaces/04- Industrial Connectors/01- PLC Mapping.md +++ /dev/null @@ -1,213 +0,0 @@ ---- -title: PLCs -description: How AnyLog connects to PLCs — supported protocols, the shared command pattern, and how to map incoming reads into tables. -layout: page -source_path: "01 PLC Overview.md" ---- - - - -## How AnyLog Defines a PLC - -A **PLC** (Programmable Logic Controller), is any industrial controller or field device that exposes its data through -one of AnyLog's supported client protocols. AnyLog does not care whether the source is a traditional PLC, an RTU, an -outstation, or a sensor gateway — if it speaks **Modbus TCP**, **OPC-UA**, **EtherNet/IP**, or **DNP3**, AnyLog can act -as a client against it. - -Every PLC client, regardless of protocol, follows the same shape: - -- AnyLog connects to the device over the network (`hostname`/`port` or `url`, depending on the protocol). -- Points on the device — coils, registers, tags, or DNP3 points — are read on a schedule (`frequency`). -- Each read is normalized into JSON and streamed into a local operator database table. - -Because every protocol funnels into that same shape, the table structure, mapping logic, and command patterns -described below are shared across all four — only the connection keywords (Section 3) differ per protocol. - -* Modbus TCP - Reading coils, discrete inputs, and holding/input registers from Modbus TCP devices. -* OPC-UA | Traversing an OPC-UA server's node tree and reading tag values. -* EtherNet/IP | Reading CIP object tags from PLCs and controllers over EtherNet/IP. -* DNP3 | Acting as a DNP3 master against outstations over TCP or TLS. | - -## Standard Command Format: - -Keywords common to every protocol: - -| Keyword | Details | -|---|---| -| `type` | Selects the protocol: `modbus`, `opcua`, `etherip`, or `dnp3`. | -| `name` | A unique client name — also used to derive table names under dynamic ingest. | -| `frequency` | Poll interval (seconds, or a fraction of a second expressed in Hz). | -| `dbms` | Target database for the ingested rows. | -| `table` | Wide-table ingest; omit when using `dynamic = true`. | -| `dynamic` | `true` writes each mapped point to its own table instead of one wide table. | -| `map` / `nodes` | The points to read — a JSON array of point definitions, or a list of tag names depending on protocol. | -| `namespace`, `master_node` | Optional Unified Namespace registration (Modbus and DNP3; requires `dynamic = true`). | - -The connection keywords themselves are protocol-specific — for example, Modbus and DNP3 use `hostname`/`port`, -while OPC-UA and EtherNet/IP use `url`; DNP3 adds `master_id`/`outstation_id`, Modbus adds `device_id`. See each -protocol's page for its full keyword table. - -* View the data that's accessible via the PLC - -```anylog -get struct where url = opc.tcp://10.0.0.111:53530/OPCUA/SimulationServer -``` - -* View current value based on the _map_ of _node_ - -```anylog - and - hostname = 192.168.1.72 and - port = 1502 and - device_id = 1 and - map|nodes = [{"name":"sensor_1","register":0}]> -``` - -* (Continuously) pull content and store into table -```anylog - and - url = [connect string] and - frequency = [seconds] and - dbms = [dbms] and - node = [node id]> -``` - -## Generic Command Pattern - -`run plc client` service, by default converts the _tags_ into column names of the table. -Alternatively, the metadata can be converted into mapping, so data can be store in a more consistent format. - -**Example I**: - -1. A _PLC_ publishes data with tags: `timestamp`, `duration`, `DelayTimer.ACC`, `DelayTimer.PRE`, -`CycleCounter.ACC` and `CycleCounter.PRE`, with data looking like: - -```json -{ - "timestamp": "'2026-08-08T18:23:34.709272Z'", - "duration": 0, - "DelayTimer.ACC": 140, - "DelayTimer.PRE": 2105, - "CycleCounter.ACC": 949, - "CycleCounter.PRE": 2341 -} -``` - -2. Initiate a `run plc client` - -```anylog - -``` - -3. Data would be stored in logical database `my_db` and table `my_data` -```sql -CREATE TABLE my_data ( - row_id SERIAL PRIMARY KEY, - insert_timestamp TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP, - tsd_name character(3), - tsd_id integer, - timestamp TIMESTAMP NOT NULL DEFAULT NOW(), - DelayTimer_ACC INT, - DelayTimer_PRE INT, - CycleCounter_ACC INT, - CycleCounter_PRE INT -); - -# Row -| row_id | insert_timestamp | tsd_name | tsd_id | timestamp | DelayTimer_ACC | DelayTimer_PRE | CycleCounter_ACC | CycleCounter_PRE | -| 1 | 2026-08-08T18:23:35.709272Z | 0 | 0 | 2026-08-08T18:23:34.709272Z | 140 | 2105 | 494 | 2341 | -``` - -That works, but it doesn't scale: every new tag on the device means a new column, and structurally identical -tags (`DelayTimer` and `CycleCounter` are both "a monitor with an ACC and a PRE value") end up as unrelated -columns instead of related rows. - -**Example 2**: A **mapping policy** reshapes the same read into a narrower, repeating structure — one row per monitor instead -of one row per poll: - -1. Define a mapping policy - -```anylog - - -blockchain insert where policy=!policy_id and local=true and master=!ledger_conn -``` -> The policy's `params` list defines each output row as a group of source fields — here, each group is -> `[monitor_id, ACC field, PRE field]`. The `schema` block then defines the table's columns and, for each -> column, where its value comes from: either lifted straight from the reading (`bring`) or taken positionally -> from `params` (`params.0`, `params.1`, ...). - -2. Initiate a `run plc client` - Once the policy is published, reference it by ID on `run plc client` instead of (or -alongside) an inline `map`: - -```anylog -run plc client where type = and and - frequency = and - name = and - dbms = and - nodes= ["DelayTimer.ACC", "DelayTimer.PRE", "CycleCounter.ACC", "CycleCounter.PRE"] - policy = 123 -``` - -3. Data would be stored in logical database `my_db` and table `my_data` - -```sql -CREATE TABLE ( - timestamp DATETIME, - monitor_id VARCHAR, - ACC INT, - PRE INT -); - -# Row -| row_id | insert_timestamp | tsd_name | tsd_id | timestamp | monitor_id | ACC | PRE | -| 1 | 2026-08-08T18:23:35.709272Z | 0 | 0 | 2026-08-08T18:23:34.709272Z | DelayTimer | 140 | 2105 | -| 2 | 2026-08-08T18:23:35.709272Z | 0 | 0 | 2026-08-08T18:23:34.709272Z | CycleCounter | 494 | 2341 | -``` diff --git a/04- Southbound Interfaces/04- Industrial Connectors/02- Modbus.md b/04- Southbound Interfaces/04- Industrial Connectors/02- Modbus.md deleted file mode 100644 index 45390830..00000000 --- a/04- Southbound Interfaces/04- Industrial Connectors/02- Modbus.md +++ /dev/null @@ -1,157 +0,0 @@ ---- -title: Modbus TCP -description: Pull coils, discrete inputs, and holding/input registers from Modbus TCP devices into AnyLog using run plc client. -layout: page -source_path: "02 MODBUS.md" ---- - - - -AnyLog can act as a **Modbus TCP client** (over **hostname** and **port**, typically port **502**). Data is read on a -schedule and streamed into your local operator database as JSON, using the same **`run plc client`** pattern as -OPC-UA and EtherNet/IP. - ---- - -## Prerequisites - -| Requirement | Notes | -|---|---| -| **`pymodbus`** | Must be installed in the AnyLog runtime environment. | - -Declare the target DBMS before streaming, for example: - -```anylog -connect dbms new_company where type = sqlite -``` - ---- - -## Connection and map - -Modbus uses **`hostname`** and **`port`** (not `url`). **`device_id`** is the **Modbus PDU unit id** (often `0` or `1` on TCP). **`map`** is a **JSON array** of points; each object must include a **`name`** (column / logical label) and exactly one address field: - -| Key | Meaning | -|---|---| -| `coil` | Coil (0-based address) | -| `input` | Discrete input | -| `inputRegister` | Input register (single address or list for blocks) | -| `register` | Holding register (single address or list for blocks) | - -**`topic`** is **not** supported on Modbus `run plc client` commands. - ---- - -## One-shot read - -```anylog - -``` - -Alias: **`get modbus values`** (same keywords). - ---- - -## Continuous ingest — wide table (default) - -With **`table = ...`** and **`dbms`**, all points from **`map`** land in **one table**. Each poll inserts **one row**; every object in **`map`** is **one column**: the map **`name`** is the **column name**, and that column stores the **value** read for that point. - -```anylog - -``` - ---- - -## Continuous ingest — dynamic tables (`dynamic = true`) - -Omit **`table`** and omit **`namespace`** for plain dynamic ingest. Each object in **`map`** is written to its **own table**. The table name is derived from the client **`name`** and the map **`name`** (for example, **`fdev10_outside_temperature`** when **`name = fdev10`** and the map entry’s **`name`** is **`outside_temperature`**). Each row includes **`timestamp`**, **`tag`**, and **`value`**. - -```anylog - -``` - ---- - -## Dynamic ingest with UNS (`namespace` + `master_node`) - -With **`dynamic = true`**, you can add a **Unified Namespace** path and a **master node** so Modbus ingest is registered in the UNS alongside your policies and DBMS. **`namespace`** requires **`master_node = [ip:port]`** for policy updates. - -Example (abbreviated): - -```anylog -run plc client where type = modbus and - hostname = 192.168.1.72 and - port = 1502 and - device_id = 1 and - frequency = 5 and - name = fdev11 and - dbms = new_company and - dynamic = true and - master_node = 192.168.1.88:32048 and - namespace = FA9/MID9/DEVICE9 and - map = [{"name":"desk_lamp","coil":0}] -``` - -**Table names** follow the **same pattern** as plain **`dynamic = true`** (client **`name`** plus map **`name`**, e.g. **`fdev11_desk_lamp`** for the example above). Under UNS, the **read value** is usually stored in a **column named like the tag**—the map **`name`** (here **`desk_lamp`**), not a generic **`value`** column. - -With **`namespace`**, table and column layout follow **UNS policies**. **`namespace`** and **`master_node`** drive how -tables are registered in the UNS. See Unified Namespace for background. - ---- - -## Command keywords (summary) - -| Keyword | Required / notes | -|---|---| -| `type` | `modbus` | -| `hostname`, `port` | Modbus TCP target | -| `device_id` | PDU unit id | -| `frequency` | Poll interval | -| `name` | Unique client name | -| `dbms` | Target DBMS | -| `table` | Wide-table ingest; omit with **`dynamic = true`** | -| `dynamic` | `true` for per-map tables or UNS | -| `map` | JSON array of points | -| `namespace` | UNS path (Modbus + **`dynamic = true`** only) | -| `master_node` | Required when **`namespace`** is set | - ---- - -## Related - -- Adding Data to Nodes in the Network -- Unified Namespace -- OPC-UA -- EtherNet/IP \ No newline at end of file diff --git a/04- Southbound Interfaces/04- Industrial Connectors/03- OPC-UA.md b/04- Southbound Interfaces/04- Industrial Connectors/03- OPC-UA.md deleted file mode 100644 index edf54505..00000000 --- a/04- Southbound Interfaces/04- Industrial Connectors/03- OPC-UA.md +++ /dev/null @@ -1,382 +0,0 @@ ---- -title: OPC-UA -description: Configure AnyLog as an OPC-UA client to explore a server, read values, pull data continuously, apply aggregations, and manage tags with policies. -layout: page -source_path: "OPC UA Integration.md" ---- - - -OPC Unified Architecture (OPC UA) is a robust, platform-independent industrial communication protocol with built-in -security (encryption, authentication, and access control), widely used in industrial automation for secure and reliable -data exchange between devices, systems, and applications. Designed as an evolution of the OPC Classic standard, it -supports real-time data access, historical data retrieval, and event notifications, making it ideal for industrial IoT -and Industry 4.0 environments. - -AnyLog can act as an OPC-UA client, pulling data from any OPC-UA server and streaming it into local databases. - ---- - -## Explore the server - -### Get namespaces - -In OPC UA, namespaces organize and uniquely identify nodes in the address space of a server. Each namespace is assigned -a unique index (e.g. `ns=0`, `ns=1`, `ns=2`) used in Node IDs — for example `ns=1;s=TemperatureSensor`. - -```anylog -get opcua namespace where url = [connect string] and user = [username] and password = [password] -``` - -Details: -* `url` — the endpoint of the OPC UA server. -* `user` — the username required by the OPC UA server for access. -* `password` — the password associated with the username. - -Example: -```anylog -get opcua namespace where url = opc.tcp://10.0.0.111:53530/OPCUA/SimulationServer -``` - -### Traverse the address space tree - -The OPC UA tree organizes the server's address space into a hierarchical model resembling a file system. At the root -level, predefined folders like **Objects**, **Types**, and **Views** provide entry points into the address space. The -**Objects** folder contains application-specific nodes (devices, sensors, systems), while the **Types** folder defines -the structure and behavior of nodes (ObjectTypes, VariableTypes, DataTypes). Each node can have child nodes, creating a -parent-child hierarchy. - -The `get opcua struct` command navigates the tree and produces different outputs based on the command variables. The -traversal starts from the root, unless a node is specified to serve as the root. - -```anylog -get opcua struct where url = [connect string] and [options] -``` - -| Option | Description | -|---|---| -| `url` | OPC UA server endpoint. | -| `user` / `password` | Credentials required by the server. | -| `node` | Override the root node by providing a node ID (e.g. `ns=6;s=MyObjectsFolder`). | -| `type` | Filter by node type: `Object`, `Variable`, etc. If not specified, all types are visited. | -| `attributes` | Attribute names to consider, or `*` for all. | -| `class` | Filter the traversal to nodes in the listed class. | -| `depth` | Limit traversal by depth. | -| `limit` | Limit traversal by number of nodes visited. | -| `output` | Target for the output stream (`stdout` or a file name). | -| `append` | If output is a file, a `true` value appends to it (default `false`). | -| `format` | Output format (see below). | -| `validate` | If `true`, reads each node's value to confirm it is readable (see below). | -| `schema` | If `true`, includes the table schema for each tag. | -| `dbms` / `table` | Used when generating `run_client` or `policy` output. | -| `frequency` | Used when generating `run_client` output. | -| `target` | Variables for the `blockchain insert` commands (used with `format = policy`). | - -**Format options:** - -| Format | Output | -|---|---| -| `tree` | OPC-UA tree structure (default). | -| `path` | Full path strings for each node. | -| `stats` | Count of entries per class. | -| `get_value` | Generates get opcua value commands for the visited nodes. | -| `run_client` | Generates run opcua client commands for the visited nodes. | -| `policy` | Generates a policy per tag; combine with `target` for `blockchain insert` commands. | - -**The validate option:** -* The default value is `false`. -* If set to `true`, the value of each considered node is read during traversal. -* If the read fails: - * When `format` is `get_value` or `run_client`, the node is not included. - * In other cases, the output includes a `validate` attribute assigned `success` or `failure`. - * The summary chart includes a counter for the number of nodes that failed to generate a value. - -**Traversal examples:** - -```anylog -# Browse from root, limit 10 nodes -get opcua struct where url = opc.tcp://10.0.0.111:53530/OPCUA/SimulationServer and output = stdout and limit = 10 - -# Direct the output to a file -get opcua struct where url = opc.tcp://10.0.0.111:53530/OPCUA/SimulationServer and output = !prep_dir/opcua_tree.txt and limit = 10 - -# Browse from root, limit by depth -get opcua struct where url = opc.tcp://10.0.0.111:53530/OPCUA/SimulationServer and output = stdout and depth = 4 - -# Browse from a specific node -get opcua struct where url = opc.tcp://10.0.0.111:53530/OPCUA/SimulationServer and output = stdout and node="ns=6;s=MyObjectsFolder" - -# Browse from a specific node, including attribute info -get opcua struct where url = opc.tcp://10.0.0.111:53530/OPCUA/SimulationServer and output = stdout and node="ns=6;s=MyObjectsFolder" and attributes = * - -# Limit by depth and filter by classes "variable" and "object" -get opcua struct where url = opc.tcp://10.0.0.111:53530/OPCUA/SimulationServer and output = stdout and depth = 4 and class = variable and class = object - -# Variables only, generate a get opcua value command -get opcua struct where url = opc.tcp://10.0.0.111:53530/OPCUA/SimulationServer and node="ns=6;s=MyObjectsFolder" and class = variable and format = get_value - -# Variables only, generate a run opcua client command -get opcua struct where url = opc.tcp://10.0.0.111:53530/OPCUA/SimulationServer and node="ns=6;s=MyObjectsFolder" and class = variable and format = run_client and name = opcua_nov and dbms = nov and table = sensor and frequency = 10 and limit = 10 - -# Variables only, output the path of each node -get opcua struct where url = opc.tcp://127.0.0.1:4840/freeopcua/server and format = path and node = "ns=2;s=DeviceSet" and class = variable and dbms = my_dbms - -# Variables only, generate a policy per node -get opcua struct where url = opc.tcp://127.0.0.1:4840/freeopcua/server and format = policy and limit = 100 and node = "ns=2;s=DeviceSet" and class = variable and dbms = my_dbms - -# Variables only, generate blockchain insert commands for each policy, written to a file -get opcua struct where url = opc.tcp://127.0.0.1:4840/freeopcua/server and format = policy and limit = 100 and node = "ns=2;s=DeviceSet" and class = variable and dbms = my_dbms and target = "local = true and master = !master_node" and output = !tmp_dir/my_file.out -``` - ---- - -## Read node values - -Node values are retrieved with the `get plc values` command: - -```anylog -get plc values where type = opcua and url = [connect string] and user = [username] and password = [password] and node = [node id] -``` - -| Option | Description | -|---|---| -| `url` | OPC UA server endpoint. | -| `user` / `password` | Credentials required by the server. | -| `node` | One or more node IDs. | -| `nodes` | A comma-separated list of nodes within square brackets. | -| `include` | Additional attributes returned with the value: `id`, `name`, `source_timestamp`, `server_timestamp`, `status_code`, or `all`. | -| `method` | `collection` (default, a single read pulls all listed nodes) or `individual` (one read per node, used to identify the node causing failures). | -| `failures` | Requires `method = individual`. `false` (default) collects successful and failed reads; `true` collects only failed reads. | - -The `include` attributes: -* `id` — the id of the attribute. -* `name` — the attribute name. -* `source_timestamp` — the timestamp of the value as determined by the data source (e.g. a sensor or device). -* `server_timestamp` — the timestamp assigned by the OPC UA server when the value was received or processed. -* `status_code` — the status of the value (e.g. Good, Bad, Uncertain). - -> Note: if `include` is assigned the keyword `all`, all attributes are included in the output. - -Examples: -```anylog -get plc values where type = opcua and url = opc.tcp://10.0.0.111:53530/OPCUA/SimulationServer and node = "ns=0;i=2257" and node = "ns=0;i=2258" and include = all - -# List format -get plc values where type = opcua and url = opc.tcp://10.0.0.111:53530/OPCUA/SimulationServer and nodes = ["ns=4;s=AirConditioner_1.StateCondition.EventType","ns=4;s=AirConditioner_1.StateCondition.SourceNode"] -``` - ---- - -## Continuous data pull - -Stream data from OPC-UA into the local database continuously: - -```anylog -run plc client where type = opcua and name = [unique name] and url = [connect string] and frequency = [seconds] and dbms = [dbms] and node = [node id] -``` - -| Option | Description | -|---|---| -| `name` | Unique client name. | -| `frequency` | Read frequency in seconds, or in hz (e.g. `10 hz`). | -| `node` / `nodes` | One or more node IDs. | -| `policy` | Use a policy to determine nodes and table (alternative to specifying nodes inline). | -| `table` | The table to host the data (if not specified in a policy). | -| `topic` | Route data through the local broker. | - -Each row is stored with two columns added automatically: -- `timestamp` — the earliest source timestamp of the values considered (if `source_timestamp` is missing, the `server_timestamp` is used). -- `duration` — the number of milliseconds between the earliest and latest timestamp considered in this read. - -Examples: -```anylog -# Individual nodes -run plc client where type = opcua and name = myopcua and url = opc.tcp://10.0.0.111:53530/OPCUA/SimulationServer and frequency = 10 and dbms = nov and table = sensor and node = "ns=0;i=2257" and node = "ns=0;i=2258" - -# Using a node list -run plc client where type = opcua and name = myopcua and url = opc.tcp://10.0.0.111:53530/OPCUA/SimulationServer and frequency = 10 and dbms = nov and table = sensor and nodes = ["ns=0;i=2257","ns=0;i=2258"] -``` - -> Multiple OPC-UA clients can run on the same node simultaneously. - -### Check client status - -```anylog -get plc client -``` - -### Stop a client - -```anylog -exit plc client [client name] -``` - -The client name is the policy ID or `[dbms name].[table name]`. If the client name is `all`, all clients are terminated. - -```anylog -exit plc all -exit plc nov.rig8 -``` - ---- - -## OPC-UA with aggregations - -Aggregation functions summarize streaming data over a time interval, enabling real-time analytics without storing raw -data. See details in the Aggregations section. - -### 1. Identify the time and value column names - -To apply aggregation, identify the names of the time attribute and the value attribute retrieved from the OPC UA -connector. If the AnyLog OPC UA service is used, the time attribute name is `timestamp`, and the value attribute name can -be retrieved with `get plc values` using `include = all` or `include = name`. - -```anylog -get plc values where type = opcua and url = opc.tcp://uademo.prosysopc.com:53530/OPCUA/SimulationServer and node = "ns=3;i=1002" and include = name -``` - -```text -OPCUA Nodes values -name value -------|----------| -random|-0.5909728| -``` - -The call above shows that the column name for `ns=3;i=1002` is `random`. - -### 2. Declare the aggregation - -```anylog -set aggregations where dbms = nov and table = table_2 and time_column = timestamp and value_column = random -``` - -### 3. (Optional) Replace raw data with aggregated data - -```anylog -set aggregations encoding where dbms = nov and table = table_2 and encoding = bounds -``` - -### 4. Validate the aggregation declarations - -```anylog -get aggregations -get aggregations config -``` - -### 5. Start the OPC-UA client - -```anylog - -``` - -### 6. Validate processing - -```anylog -get plc client -get aggregations -get aggregations where dbms = nov and table = table_2 -get streaming -get operator -``` - -### 7. Query aggregated results - -```anylog -run client () sql nov format = table select timestamp::ljust(19), end_interval::ljust(19), min_val, max_val, avg_val, events from bounds_table_2 order by timestamp desc limit 10 -``` - -> **Worth verifying:** unlike every other SQL query example across this doc set, the query text above isn't wrapped -> in quotes (`"select ..."`). This is consistent in both source files, so it isn't something either merge -> introduced — but it's worth confirming this is genuinely valid syntax for `run client () sql` rather than a -> pre-existing typo that was carried through both docs. - ---- - -## Policy-based tag management - -For large OPC-UA deployments, generate a policy for each tag and store it on the blockchain. Policies define the -structure and semantics of the tags (names, data types, relationships) and serve as a mapping between table names and tag -information. This lets AnyLog automatically interpret and organize incoming data, aligning it with the defined structure -for querying, validation, and distribution across the network. - -### Generate the policies - -```anylog -get opcua struct where url = opc.tcp://127.0.0.1:4840/freeopcua/server and format = policy and schema = true and node = "ns=2;s=DeviceSet" and class = variable and dbms = my_dbms and target = "local = true and master = !master_node" and output = !tmp_dir/my_file.out -``` - -The tag policies are stored in the file `!tmp_dir/my_file.out`, in a format like: - -```json -{"tag": {"class": "variable", - "datatype": "Boolean", - "dbms": "my_dbms", - "nodeid": "LS1002H_AlarmSetpoint", - "ns": 2, - "parent": "ALARM_TAGS", - "path": "Root/Objects/DeviceSet/WAGO 750-8210 PFC200 G2 4ETH XTR/Resources/Application/GlobalVars/ALARM_TAGS/LS1002H_AlarmSetpoint", - "table": "t39"}} -``` - -If `schema` is set to `true`, the output includes, for every tag, the table schema associated with the tag: - -```json -{"table": {"name": "t39", - "dbms": "nov", - "create": "CREATE TABLE IF NOT EXISTS t39(row_id SERIAL PRIMARY KEY, insert_timestamp TIMESTAMP NOT NULL DEFAULT NOW(), tsd_name CHAR(3), tsd_id INT, timestamp timestamp not null default now(), value bool ); CREATE INDEX t39_timestamp_index ON t39(timestamp); CREATE INDEX t39_insert_timestamp_index ON t39(insert_timestamp);", - "source": "OPCUA Interface", - "id": "040197b7eed831dddb1b3fd910d86deb", - "date": "2025-04-09T00:09:53.406292Z", - "ledger": "local"}} -``` - -### Load the policies to the metadata - -```anylog -process !tmp_dir/my_file.out -``` - -### Generate the command to read the tag data - -```anylog -get opcua struct where url = opc.tcp://127.0.0.1:4840/freeopcua/server and format = run_client and node = "ns=2;s=DeviceSet" and class = variable and output = !tmp_dir/my_run_cmd.out and dbms = my_dbms and frequency = 3 and name = opcua_nov -``` - -Notes: -* The run opcua client command is stored in the file `!tmp_dir/my_run_cmd.out`. -* The `table` name is not specified, as it is derived from the policies (based on the namespace and node id). - -### Execute the command - -```anylog -process !tmp_dir/my_run_cmd.out -``` - -This pulls the data using OPC-UA and assigns it to the tables according to the info in the policies. \ No newline at end of file diff --git a/04- Southbound Interfaces/04- Industrial Connectors/04-1 DNP3/01- DNP3 - Deploying Connector via Script.md b/04- Southbound Interfaces/04- Industrial Connectors/04-1 DNP3/01- DNP3 - Deploying Connector via Script.md deleted file mode 100644 index 9f99bbee..00000000 --- a/04- Southbound Interfaces/04- Industrial Connectors/04-1 DNP3/01- DNP3 - Deploying Connector via Script.md +++ /dev/null @@ -1,228 +0,0 @@ ---- -title: "Deploying a DNP3 Connector via Script" -description: Walking through a production-style .al deployment script for a DNP3 southbound connection — parameters, policy reuse, TLS branching, and error handling -layout: page -source_path: "01- DNP3 - Deploying Connector via Script.md" ---- - - - -# Deploying a DNP3 Connector via Script - -DNP3 documents the `get plc values` / `run plc client` commands directly — typed once on the CLI, or -pasted into a one-off script, with the point `map` written out as a literal JSON array each time. For a -production deployment, it's more common to run DNP3 connections from a standing `.al` script that a node -executes on startup (or on demand), with parameters set at the top and the point map itself stored once as a -reusable blockchain policy rather than duplicated in every script. This page walks through that pattern using a -sample script, `dnp3_connector.al`. - -For the reusable-mapping-policy pattern itself (the `dnp3` policy type this script checks for and creates), see -DNP3 Mapping Policies — Reusing a Schema Across Connections. - -Run the script with: - -```anylog -process !local_scripts/southbound-industrial/dnp3_connector.al -``` - -(adjust the path to wherever the script actually lives in your deployment). - ---- - -## Script structure - -The script is organized as a sequence of labels, executed with `goto`/fall-through, rather than top-to-bottom — -this lets it branch (e.g. TLS vs. plain TCP) and handle errors from a shared sub-script without duplicating -logic. - -### `:set-params:` — connection and identity - -```anylog -on error ignore -:set-params: -client_type=dnp3 -dnp_ip = 192.168.1.88 -dnp_port = 20001 -dnp_master_id = 1 -outstation_id = 10 -dnp_frequency = 20 -dnp_name = plant1 -base_namespace = "FACTORY4/DNP3/SUBSTATION" - -# authentication configs -set enable_tls = false - -# !anylog_dir is accessible as a volume and used to store certifications and access points. -# we can also store the public information on the blockchain so there's no need for persistence of content -tls_ca = !anylog_dir/dnp3_certs/factory_ca.cert -tls_cert = !anylog_dir/dnp3_certs/master1.cert -tls_key = !anylog_dir/dnp3_certs/master1.key -``` - -Everything a given deployment needs to change — target outstation, link IDs, poll frequency, the UNS namespace -this connection is registered under, and whether TLS is used — is set once here. `dnp_name` and -`base_namespace` together are also the key used to look up (or create) the reusable mapping policy in the next -step, so they double as the identity of this particular DNP3 connection, not just a display label. - -Note on TLS paths: `!anylog_dir` is a volume mounted into the AnyLog container, used here to hold certificate -files locally on the node. The comment in the script also flags an alternative worth knowing about — the -*public* half of a certificate can instead be stored on the blockchain as part of a policy, avoiding the need to -persist certificate files on every node that needs them. This script uses the local-file approach; see -DNP3 TLS Test Certificates for generating -a chain to use with `tls_ca`/`tls_cert`/`tls_key`. - -### `:check-policy:` — look for an existing mapping policy - -```anylog -:check-policy: -is_dnp3 = blockchain get dnp3 where namespace = !base_namespace and name=!dnp_name -if not !is_dnp3 then goto prep-policy - -dnp_schema = from !is_dnp3 bring [*][schema] - -goto declare-dnp3 -``` - -If a `dnp3` policy already exists for this `namespace`/`name` pair, its `schema` (the point map) is pulled out -into `!dnp_schema` and the script jumps straight to declaring the connection. If not, it falls through to -`:prep-policy:` to create one. - -### `:prep-policy:` — build a new mapping policy - -```anylog -:prep-policy: - -``` - -This is the point map — the same shape documented as the `map` array in DNP3 — -but declared once, as data, rather than repeated inline in every `run plc client` call. - -### `:publish-policy:` — sign and insert the policy - -```anylog -:publish-policy: -process !local_scripts/node-deployment/policies/publish_policy.al -if not !error_code.int then -do set create_policy = true -goto check-policy - -if !error_code == 1 then goto sign-policy-error -else if !error_code == 2 then goto prepare-policy-error -else if !error_code == 3 then goto declare-policy-error -``` - -This delegates to a shared helper script (`publish_policy.al`) that signs `!new_policy` and inserts it onto the blockchain — -the same helper any connector's deployment script would call, not something DNP3-specific. On success, the script loops -back to `:check-policy:`, which will now find the freshly published policy and pick up `!dnp_schema` from it. A nonzero -`!error_code` routes to one of three error labels depending on which stage failed (signing, preparing, or declaring the -policy) — see Error handling below. - -### `:declare-dnp3:` / `:declare-dnp3-tls:` — start the connection - -```anylog -:declare-dnp3: -on error goto declare-dnp3-err -if !enable_tls == true goto declare-dnp3-tls - -goto end-script - -:declare-dnp3-tls: - -``` - -Both branches are the same `run plc client where type = dnp3 ...` call documented in -DNP3 — Dynamic ingest with UNS; the only difference is -whether the four `enable_tls`/`tls_*` keywords are included. `!dnp_schema` (from either `:check-policy:` or -`:prep-policy:` → `:publish-policy:` → `:check-policy:`) supplies `map`, and `!base_namespace` / -`!dnp_name` supply `namespace` — so the UNS registration and the policy lookup key are the same values, -by design. - -### Error handling - -```anylog -:end-script: -end script - -:terminate-scripts: -exit scripts - -:sign-policy-error: -print "Failed to sign mapping policy" -goto terminate-scripts - -:prepare-policy-error: -print "Failed to prepare mapping policy for publishing on blockchain" -goto terminate-scripts - -:declare-policy-error: -print "Failed to declare mapping policy on blockchain" -goto terminate-scripts - -:declare-dnp3-err: -print "Failed to define connection to DNP3 against" + !dnp_ip + ":" + !dnp_port -goto terminate-scripts -``` - -Each failure path prints a specific message identifying which stage failed (signing, preparing, or declaring the -policy; or establishing the DNP3 connection itself) before terminating the script — useful for diagnosing a -failed deployment from logs without needing to instrument the script further. - ---- - -## Adapting this for your own deployment - -At minimum, change in `:set-params:`: - -- `dnp_ip`, `dnp_port`, `dnp_master_id`, `outstation_id` — to match your actual outstation. -- `dnp_name`, `base_namespace` — pick values that uniquely identify this connection; these double as the lookup - key for the reusable mapping policy. -- The `schema` array in `:prep-policy:` — to match the actual points on your outstation (see - DNP3 — Connection and map for the supported `type` values). -- `enable_tls` and the three `tls_*` paths, if using TLS. - -If you're deploying several identical or near-identical outstations (for example, several substations with the -same point layout), give them the same `base_namespace`/`dnp_name` pattern deliberately, so later deployments hit -the `:check-policy:` fast path and reuse the schema already published by the first one, rather than re-declaring -it. See DNP3 Mapping Policies for more on this reuse pattern. \ No newline at end of file diff --git a/04- Southbound Interfaces/04- Industrial Connectors/04-1 DNP3/02- DNP3 - Mapping-Policies.md b/04- Southbound Interfaces/04- Industrial Connectors/04-1 DNP3/02- DNP3 - Mapping-Policies.md deleted file mode 100644 index 787f3193..00000000 --- a/04- Southbound Interfaces/04- Industrial Connectors/04-1 DNP3/02- DNP3 - Mapping-Policies.md +++ /dev/null @@ -1,144 +0,0 @@ ---- -title: "DNP3 Mapping Policies — Reusing a Schema Across Connections" -description: Storing a DNP3 point map once as a blockchain policy and reusing it across multiple connections, instead of repeating the map JSON inline -layout: page -source_path: "02- DNP3 - Mapping-Policies.md" ---- - - - -# DNP3 Mapping Policies — Reusing a Schema Across Connections - -DNP3 documents `map` as a JSON array supplied directly in the `get plc values` / `run plc client` -command — every example on that page writes the point list out in full, inline. That's the right approach for a -single connection, or for testing. For a deployment with several outstations that share the same, or nearly the -same, point layout — several substations of the same model, for instance — repeating an identical `map` array in -every connection command means the same schema has to be kept in sync by hand in multiple places. - -The alternative: declare the point map once as a policy on the blockchain, and have each connection look it up -by name instead of restating it. - -## The `dnp3` policy type - -The policy has one required shape: - -```json -{ - "dnp3": { - "namespace": "FACTORY4/DNP3/SUBSTATION", - "name": "plant1", - "schema": [ - {"name":"analog_0","type":"Analog","index":0}, - {"name":"binary_0","type":"Binary","index":0}, - {"name":"analog_output_status_0","type":"AnalogOutputStatus","index":0}, - {"name":"binary_output_status_0","type":"BinaryOutputStatus","index":0} - ] - } -} -``` - -- `namespace` + `name` together identify the policy — this pair is the lookup key, and not coincidentally, it's - the same `namespace`/`name` pair used when registering the connection in the UNS (see - DNP3 — Dynamic ingest with UNS). Using the same - values for both isn't required by the platform, but it keeps one connection's identity, UNS registration, and - mapping policy all pointing at the same pair of values instead of three independently-tracked names. -- `schema` is exactly the same array documented as `map` in - DNP3 — Connection and map — the same `name`/`type`/`index` (or - `group`/`variation`) objects, just stored as policy data instead of typed inline. - -## Looking up an existing policy - -```anylog -is_dnp3 = blockchain get dnp3 where namespace = !base_namespace and name = !dnp_name -dnp_schema = from !is_dnp3 bring [*][schema] -``` - -`!dnp_schema` now holds the same JSON array that `map` expects, pulled from whichever policy already matches this -`namespace`/`name` pair — whether that policy was published by this exact connection previously, or by an -earlier, identical outstation deployment reusing the same namespace/name convention. - -## Publishing a new policy - -If no matching policy exists yet, build one and insert it — following the same -sign → insert -pattern used for any other policy type: - -```anylog - -blockchain insert where policy = !new_policy and local = true and master = !ledger_conn -``` - -(A full deployment script wraps this with signing and error handling — see -Deploying a DNP3 Connector via Script -for the complete pattern, including the check-then-create flow that avoids republishing an identical policy on -every run.) - -## Using the resolved schema - -Whether `!dnp_schema` came from an existing policy or was just published, it's used exactly where `map` appears -in the standard `run plc client` command: - -```anylog - -``` - -This is identical to every other `run plc client` example in DNP3 — the only difference is that `map` -is populated from a variable resolved via a policy lookup, rather than written out as a literal array in the -command itself. - -## How this differs from a "mapping policy" - -AnyLog also has a generic **mapping policy** type, covered in -Mapping data to tables, used for REST/MQTT ingestion — those policies -describe a `bring`-based extraction schema (`{"type": "float", "bring": "[reading]"}`) for pulling fields out of -an arbitrary incoming JSON payload. The `dnp3` policy type here is a different, protocol-specific shape: it -describes DNP3 point addresses (`type`/`index`, or `group`/`variation`) on an outstation, not a JSON-payload -extraction rule. The two aren't interchangeable, and a `dnp3` policy isn't referenced via the generic `mapping` -keyword — it's looked up directly with `blockchain get dnp3 where ...`, as shown above. - -## Why bother — the actual benefit - -With several outstations sharing a `dnp3` policy (by using the same `namespace`/`name` convention across -deployments of otherwise-identical equipment): - -- The point schema is edited in exactly one place — updating the policy — rather than in every deployment - script that connects to a matching outstation. -- New deployments of the same equipment type look the schema up instead of needing it re-specified, reducing the - chance of a typo or drift between two supposedly-identical connections. -- The schema itself becomes queryable and auditable via the blockchain, the same as any other policy, rather - than living only inside script files. - -The tradeoff is a layer of indirection: reading a deployment script alone (without also checking the published -policy) won't tell you the actual point map in use. For a one-off connection or a quick test, the inline `map` -array shown throughout DNP3 remains simpler. - -## See also - -- DNP3 — the underlying `get plc values`/`run plc client` command reference -- Deploying a DNP3 Connector via Script — a full deployment - script using this pattern, with TLS branching and error handling \ No newline at end of file diff --git a/04- Southbound Interfaces/04- Industrial Connectors/04-1 DNP3/03- DNP3 - TLS test certificates.md b/04- Southbound Interfaces/04- Industrial Connectors/04-1 DNP3/03- DNP3 - TLS test certificates.md deleted file mode 100644 index 51a02dc5..00000000 --- a/04- Southbound Interfaces/04- Industrial Connectors/04-1 DNP3/03- DNP3 - TLS test certificates.md +++ /dev/null @@ -1,128 +0,0 @@ ---- -title: "DNP3 TLS Test Certificates" -description: Generating a local CA chain for mutual TLS between an AnyLog DNP3 master and an outstation, for testing only -layout: page -source_path: "DNP3-tls-test-certificates.md" ---- - - - -# DNP3 TLS Test Certificates - -This page generates a small, local CA chain for mutual TLS between an AnyLog DNP3 **master** and an -**outstation** — for example, opendnp3's `outstation-demo`, configured for -TLS. It produces a root CA plus one client certificate (the AnyLog master) and two server certificates -(outstations), all signed by that CA. - -**For testing only.** These are self-signed, lab-only credentials — do not reuse them in production, and do not -commit the generated `.cert`/`.key` files to any repository. Run the script below in a working directory of your -own, outside of version control, and regenerate a fresh chain whenever you need one. - -## Generating the chain - -Save the following as `create_certificates.sh` in an empty working directory, then run `bash -create_certificates.sh`. It has no dependencies beyond `openssl`. - -```bash -#!/usr/bin/env bash -# Generate a small CA chain for DNP3 TLS lab tests (AnyLog master <-> outstation). -# -# Usage: -# mkdir -p ~/dnp3-tls-test && cd ~/dnp3-tls-test -# bash create_certificates.sh -# -# AnyLog master (TLS client): anylogDNP3ca.cert, master1.cert, master1.key -# Outstation (TLS server): anylogDNP3ca.cert, outstation1.cert, outstation1.key -# -set -euo pipefail -cd "$(dirname "$0")" - -CA_CERT=anylogDNP3ca.cert -CA_KEY=anylogDNP3ca.key - -echo "=== CA ===" -openssl req -x509 -new -nodes -newkey rsa:2048 \ - -keyout "${CA_KEY}" -sha256 -days 3600 -out "${CA_CERT}" \ - -subj "/C=US/ST=OR/L=Bend/O=AnyLog Certificate Corp" - -echo "=== CSRs ===" -openssl req -newkey rsa:2048 -nodes -keyout master1.key -out master1.csr \ - -subj "/C=US/ST=OR/O=AnyLogDNP3/CN=master1" -openssl req -newkey rsa:2048 -nodes -keyout outstation1.key -out outstation1.csr \ - -subj "/C=US/ST=OR/O=AnyLogDNP3/CN=outstation1" -openssl req -newkey rsa:2048 -nodes -keyout outstation2.key -out outstation2.csr \ - -subj "/C=US/ST=OR/O=AnyLogDNP3/CN=outstation2" - -echo "=== Signed certs ===" -openssl x509 -req -in master1.csr -CA "${CA_CERT}" -CAkey "${CA_KEY}" -CAcreateserial \ - -out master1.cert -days 3600 -sha256 -openssl x509 -req -in outstation1.csr -CA "${CA_CERT}" -CAkey "${CA_KEY}" -CAcreateserial \ - -out outstation1.cert -days 3600 -sha256 -openssl x509 -req -in outstation2.csr -CA "${CA_CERT}" -CAkey "${CA_KEY}" -CAcreateserial \ - -out outstation2.cert -days 3600 -sha256 - -rm -f master1.csr outstation1.csr outstation2.csr *.srl - -echo "=== Verify chain ===" -openssl verify -CAfile "${CA_CERT}" master1.cert outstation1.cert outstation2.cert - -echo "OK: certificates in $(pwd)" -``` - -This produces 8 files in your working directory: - -| File | Role | -|---|---| -| `anylogDNP3ca.cert` / `anylogDNP3ca.key` | Root CA | -| `master1.cert` / `master1.key` | AnyLog master (TLS client) | -| `outstation1.cert` / `outstation1.key` | Primary test outstation (TLS server) | -| `outstation2.cert` / `outstation2.key` | Second outstation (optional) | - -The script's own cleanup (`rm -f ... *.srl`) already removes the intermediate CSRs and OpenSSL's serial file — -so nothing but the certs and keys above is left behind, and there's no separate `.gitignore` to maintain. If you -do generate the chain inside a version-controlled directory for convenience, add `*.cert`, `*.key`, `*.csr`, and -`*.srl` to that repo's `.gitignore` before running the script. - -## Configuring the AnyLog master - -Point the master's TLS settings at your generated files (use the actual path to your working directory): - -| Keyword | Path | -|---|---| -| `tls_ca` | `/path/to/your/working/dir/anylogDNP3ca.cert` | -| `tls_cert` | `/path/to/your/working/dir/master1.cert` | -| `tls_key` | `/path/to/your/working/dir/master1.key` | - -## Configuring the outstation (TLS server) - -| Setting | File | -|---|---| -| Peer / CA | `anylogDNP3ca.cert` | -| Local certificate | `outstation1.cert` | -| Private key | `outstation1.key` | - -With `opendnp3` built at `~/opendnp3/build`, after generating the chain: - -```bash -cd ~/opendnp3/build -./outstation-tls-demo \ - /path/to/your/working/dir/anylogDNP3ca.cert \ - /path/to/your/working/dir/outstation1.cert \ - /path/to/your/working/dir/outstation1.key -``` - -Plain TCP, no TLS: run `./outstation-demo` instead, from the same directory. - -## See also - -- DNP3 — - the main DNP3 connector doc; this page is referenced from its outstation-testing section. \ No newline at end of file diff --git a/04- Southbound Interfaces/04- Industrial Connectors/05- DNP3.md b/04- Southbound Interfaces/04- Industrial Connectors/05- DNP3.md deleted file mode 100644 index d887eebe..00000000 --- a/04- Southbound Interfaces/04- Industrial Connectors/05- DNP3.md +++ /dev/null @@ -1,317 +0,0 @@ ---- -title: DNP3 -description: Pull data from DNP3 outstations into AnyLog as master over TCP or TLS using run plc client. -layout: page -source_path: "06 DNP3.md" ---- - - - -AnyLog can act as a **DNP3 master** over **TCP** or **TLS** (using **hostname** and **port**, default **20000**). -Data is read on a schedule and streamed into your local operator database as JSON, using the same **`run plc client`** -pattern as Modbus, OPC-UA, and EtherNet/IP. - -For a production-style deployment of a DNP3 connection from a standing script (rather than one-off CLI commands), see -Deploying a DNP3 Connector via Script. For -reusing a single point map across multiple outstations instead of repeating it inline, see DNP3 Mapping Policies. - ---- - -## Prerequisites - -| Requirement | Notes | -|---|---| -| **`opendnp3` (yadnp3)** | Must be installed in the AnyLog runtime environment. | - -Declare the target DBMS before streaming, for example: - -```anylog -connect dbms new_company where type = sqlite -``` - ---- - -## Connection and map - -DNP3 uses **`hostname`** and **`port`**. **`master_id`** and **`outstation_id`** are the DNP3 link addresses (defaults: master **1**, outstation **10**). - -**`map`** is a **JSON array** of points. Each object must include: - -| Key | Required | Meaning | -|---|---|---| -| `name` | yes | Column / logical label | -| `index` | yes | Point index on the outstation | -| `type` | one of `type` or `group`+`variation` | Point class (see table below) | -| `group`, `variation` | alternative to `type` | Raw DNP3 group/variation | - -Supported **`type`** values (case-insensitive; spaces/underscores ignored): - -| `type` | DNP3 group / variation | -|---|---| -| `Analog` | 30 / 6 | -| `Binary` | 1 / 2 | -| `BinaryOutputStatus` | 10 / 2 | -| `AnalogOutputStatus` or `AnalogOutput` | 40 / 4 | -| `Counter` | 20 / 6 | -| `DoubleBit` or `DoubleBitBinary` | 3 / 2 | - -`map` can be written inline, as shown throughout this page, or declared once as a reusable policy on the -blockchain and referenced across multiple connections — see DNP3 Mapping Policies. - -Optional **TLS** (all three PEM paths required when `enable_tls = true`): - -| Keyword | Meaning | -|---|---| -| `enable_tls` | Use TLS instead of plain TCP | -| `tls_ca` | CA certificate (PEM file path) | -| `tls_cert` | Client certificate (PEM file path) | -| `tls_key` | Client private key (PEM file path) | - ---- - -## One-shot read - -```anylog - -``` - -Alias: **`get dnp3 values`** (same keywords). - -With TLS (see DNP3 TLS Test Certificates for generating a chain to use here, and DNP3 Out Station Testing below for running a matching test outstation): - -```anylog - -``` - ---- - -## Continuous ingest — wide table (default) - -With **`table = ...`** and **`dbms`**, all points from **`map`** land in **one table**. Each poll inserts **one row**; every object in **`map`** is **one column**: the map **`name`** is the **column name**, and that column stores the value read for that point. - -```anylog - -``` - ---- - -## Continuous ingest — dynamic tables (`dynamic = true`) - -Omit **`table`** and omit **`namespace`** for plain dynamic ingest. Each object in **`map`** is written to its **own table**. The table name is derived from the client **`name`** and the map **`name`** (for example, **`dnp3_rtu_analog_0`** when **`name = dnp3_rtu`** and the map entry's **`name`** is **`analog_0`**). Each row includes **`timestamp`**, **`tag`**, and **`value`**. - -```anylog - -``` - ---- - -## Dynamic ingest with UNS (`namespace` + `master_node`) - -With **`dynamic = true`**, you can add a **Unified Namespace** path and a **master node** so DNP3 ingest is registered in the UNS alongside your policies and DBMS. **`namespace`** requires **`master_node = [ip:port]`** for policy updates. - -```anylog -run plc client where type = dnp3 and - hostname = 127.0.0.1 and - port = 20001 and - master_id = 1 and - outstation_id = 10 and - frequency = 20 and - name = dnp3_uns and - dbms = new_company and - dynamic = true and - master_node = 192.168.1.88:32048 and - namespace = FA9/MID9/DEVICE9 and - map = [{"name":"analog_0","type":"Analog","index":0}] -``` - -**Table names** follow the **same pattern** as plain **`dynamic = true`** (client **`name`** plus map **`name`**, e.g. **`dnp3_uns_analog_0`** for the example above). Under UNS, the **read value** is usually stored in a **column named like the tag**—the map **`name`** (here **`analog_0`**), not a generic **`value`** column. - -With **`namespace`**, table and column layout follow **UNS policies**. **`namespace`** and **`master_node`** drive how tables are registered in the UNS. See Unified Namespace for background. - -For a full example combining this with a scripted deployment and a reusable mapping policy (rather than an inline `map` array), see Deploying a DNP3 Connector via Script. - ---- - -## DNP3 Out Station Testing - -The opendnp3 library includes a demo outstation for lab tests. Build it with demos enabled (TLS optional): - -```bash -git clone --recursive -b release-2.x https://github.com/dnp3/opendnp3.git -cd opendnp3 -mkdir build && cd build -cmake -DDNP3_DEMO=ON -DDNP3_TLS=ON .. # omit -DDNP3_TLS=ON for TCP-only -make -j -``` - -See the OpenDNP3 CMake guide and TLS support (OpenSSL ≥ 1.1.1 required for TLS). - -### Plain TCP outstation - -From the opendnp3 build directory: - -```bash -cd ~/opendnp3/build -./outstation-demo -``` - -The demo listens on **any IP address**, port **20001**, outstation link id **10**, and expects master link id **1**. Use those values in AnyLog: - -```anylog - -``` - -Once started, the demo logs traffic and waits for input to send unsolicited measurement changes: - -```text -Enter one or more measurement changes then press -c = counter, b = binary, d = doublebit, a = analog, o = octet string, 'quit' = exit -``` - -### TLS outstation - -Generate a test CA chain first — see DNP3 TLS Test Certificates for the full script and instructions. Run it in a working directory of your own, outside this repository: - -```bash -mkdir -p ~/dnp3-tls-test && cd ~/dnp3-tls-test -bash create_certificates.sh -``` - -| Side | CA (peer) | Certificate | Private key | -|------|-----------|-------------|-------------| -| AnyLog master | `anylogDNP3ca.cert` | `master1.cert` | `master1.key` | -| Outstation | `anylogDNP3ca.cert` | `outstation1.cert` | `outstation1.key` | - -From the opendnp3 build directory, start the TLS demo with three PEM paths (CA, outstation certificate, outstation private key): - -```bash -cd ~/opendnp3/build -./outstation-tls-demo \ - ~/dnp3-tls-test/anylogDNP3ca.cert \ - ~/dnp3-tls-test/outstation1.cert \ - ~/dnp3-tls-test/outstation1.key -``` - -Same link ids and port as plain TCP (**master_id = 1**, **outstation_id = 10**, port **20001**). AnyLog master uses the **master** certificate files from the same CA chain. - -Example AnyLog one-shot read: - -```anylog - -``` - -More detail, including the full generation script: DNP3 TLS Test Certificates. - -### Third-party simulator - -Another option is a commercial DNP3 outstation simulator, for example the FreyrSCADA DNP3 development bundle. After download, **DNPOutstationSimulator.exe** under the **Simulator** folder can be installed on Windows (32-bit). - ---- - -## Command keywords (summary) - -| Keyword | Required / notes | -|---|---| -| `type` | `dnp3` | -| `hostname`, `port` | DNP3 TCP/TLS target (default port **20000**) | -| `master_id` | Master link address (default **1**) | -| `outstation_id` | Outstation link address (default **10**) | -| `frequency` | Poll interval (seconds) | -| `name` | Unique client name | -| `dbms` | Target DBMS | -| `table` | Wide-table ingest; omit with **`dynamic = true`** | -| `dynamic` | `true` for per-map tables or UNS | -| `map` | JSON array of points, inline or resolved from a mapping policy | -| `namespace` | UNS path (DNP3 + **`dynamic = true`** only) | -| `master_node` | Required when **`namespace`** is set | -| `enable_tls`, `tls_ca`, `tls_cert`, `tls_key` | Optional TLS (all three PEM paths required) | - ---- - -## Related - -- Adding Data to Nodes in the Network -- Unified Namespace -- DNP3 TLS Test Certificates — generating a local CA chain for TLS testing -- Deploying a DNP3 Connector via Script — a production-style `.al` deployment script, parameters, and error handling -- DNP3 Mapping Policies — reusing a point map across multiple connections instead of repeating it inline -- Modbus -- OPC-UA -- EtherNet/IP \ No newline at end of file diff --git a/04- Southbound Interfaces/04- Monitoring/01- Node Monitoring.md b/04- Southbound Interfaces/04- Monitoring/01- Node Monitoring.md deleted file mode 100644 index 762e5da6..00000000 --- a/04- Southbound Interfaces/04- Monitoring/01- Node Monitoring.md +++ /dev/null @@ -1,253 +0,0 @@ ---- -title: Node Monitoring -description: Collect node and container health metrics and stream them for live viewing via Remote GUI or persistent storage across Operator nodes. -layout: page ---- - - -Each AnyLog node can collect its own health metrics and distribute them in one or both of two ways: - -- **Live view** — push metrics to a Query Node and visualise them in the Remote GUI without storing any data -- **Persistent storage** — stream metrics into an Operator's database for historical queries and dashboards - -The monitoring schedule is deployed as a blockchain policy and activated automatically when `NODE_MONITORING=true` is set in the node configuration. - -> **Pull, not push:** unlike Syslog, where data is *pushed* into AnyLog by an external -> forwarder (rsyslog), node and Docker monitoring are *pulled* on a schedule — the node actively queries its own OS -> and Docker metrics at each interval, rather than waiting for something to send data in. - -**A node generates its own "insight"** — a JSON object describing its current state. Two kinds exist on this page: - -| Insight type | What it captures | Sent to Query Node (live view) | Sent to Operator (archive) | -|---|---|---|---| -| `node_insight` | OS/agent-level metrics — CPU, disk, network, ingestion stats (see below) | ✅ | ✅ | -| `docker_insight` | Container-level stats (see Docker Monitoring) | ❌ | ✅ | - -The reason for that split: -* **Query Node** — a live, in-memory snapshot, almost like running `top` across every node in the network. It only - makes sense for metrics you'd want to glance at *right now*. -* **Operator** — a persistent archive for historical queries, trending, and dashboards. This is where you'd look to - answer "what happened over the last week," not "what's happening this second." - -`docker_insight` is only sent to the Operator archive — there's no live-view path for it (see the Docker Monitoring -section below). - -> **Prerequisite:** any form of monitoring on this page — node insight collection, live view, persistent storage, or -> Docker monitoring below — runs as a scheduled task. The scheduler itself must be enabled, or nothing will fire: -> ```anylog -> run scheduler 1 -> ``` - ---- - -## Collecting node insight - -Each participating node runs a set of scheduled tasks that build a JSON object called `node_insight`. This object is assembled from system metrics collected at two intervals: - -| Metric | AnyLog command | Frequency | -|---|---|---| -| Operator ingestion stats | `get stats where service = operator and topic = summary` | `!monitoring_frequency` | -| Timestamp | `get datetime local now()` | `!monitoring_frequency` | -| Node type | `!node_type` | `!monitoring_frequency` | -| Disk free (%) | `get disk percentage .` | 30 seconds | -| CPU usage (%) | `get node info cpu_percent` | 30 seconds | -| Network packets received | `get node info net_io_counters packets_recv` | 30 seconds | -| Network packets sent | `get node info net_io_counters packets_sent` | 30 seconds | -| Network error count | `errin + errout` | 30 seconds | -| Status | `Active` | 30 seconds | - -The resulting `node_insight` object: - -```json -{ - "timestamp": "2026-03-07 18:41:11", - "node type": "operator", - "Free Space Percent": 72.4, - "CPU Percent": 6.7, - "Packets Recv": 1482930, - "Packets Sent": 983421, - "Network Error": 0, - "status": "Active" -} -``` - -`!monitoring_frequency` is set in the node's `.env` file (default: 60 seconds). Hardware metrics always collect at 30 seconds regardless. - ---- - -## Option 1 — Live view via Remote GUI - -Each node pushes its `node_insight` to the Query Node, which aggregates status from all participating nodes into a single view. The Remote GUI reads from the Query Node to display a live network-wide dashboard — no database required. - -``` -Node A ──┐ -Node B ──┼──► Query Node ──► Remote GUI -Node C ──┘ (aggregator) -``` - -The Query Node destination is resolved automatically from the blockchain: - -```anylog -view_monitoring_dest = blockchain get query bring.ip_port -``` - -Each node then pushes its metrics on a 30-second schedule: - -```anylog -run client (!view_monitoring_dest) monitor operators where info = !node_insight -``` - -The Remote GUI Monitor Node section shows a live table of all nodes pushing metrics to the Query Node: - - - Monitor Node Section in Remote GUI - - -Each row represents one node, showing its name, operational time, elapsed time since last update, new and total rows -ingested, and the hardware metrics — free disk space, CPU usage, network packets, and error count. - -The **Add Threshold Monitor** panel at the top lets you set alert thresholds on any column — for example, alert when -CPU Percent exceeds 80 or Free Space Percent drops below 20. - -**View from the CLI:** -```anylog -get monitored # list all monitored topics -get monitored operators # current status from all nodes -reset monitored operators # clear the aggregated list -``` - -This option requires no storage configuration — metrics are held in memory on the Query Node and reflect the current -state only. - ---- - -## Option 2 — Persistent storage across Operators - -When `STORE_MONITORING=true`, each node streams its `node_insight` into a `monitoring.node_insight` table on an -Operator node, enabling historical queries and Grafana dashboards. - -``` -Node A ──┐ -Node B ──┼──► Operator (monitoring.node_insight) -Node C ──┘ -``` - -**On Operator nodes** — stored locally: -```anylog -stream !node_insight where dbms = monitoring and table = node_insight -``` - -**On non-Operator nodes** — routed to a remote Operator: -```anylog -run client (!monitoring_storage_dest) stream !node_insight where dbms = monitoring and table = node_insight -``` - -The target Operator is resolved automatically: -```anylog -monitoring_storage_dest = blockchain get operator bring.last [*][ip]:[*][port] -``` - -### Table setup - -On the Operator hosting the monitoring data, the `node_insight` table is created automatically on startup: - -```anylog -connect dbms monitoring where type = sqlite -process !anylog_path/deployment-scripts/southbound-monitoring/create_node_monitoring_table.al -``` - -### Querying stored metrics - -```anylog -# Local query on the Operator -sql monitoring format = table "select * from node_insight order by timestamp desc limit 20" - -# Network query from any node -run client () sql monitoring format = table "select timestamp, node_type, cpu_percent, free_space_percent from node_insight where timestamp >= NOW() - 1 hour" -``` - ---- - -## Using both options together - -Options 1 and 2 are not mutually exclusive. Most deployments run both — live view for real-time dashboards and -persistent storage for historical analysis and alerting. - ---- - -## Docker Monitoring - -In addition to node-level health metrics, AnyLog can pull container-level stats directly from Docker using a -**scheduled pull** task — the same general-purpose mechanism used for things like Windows Event Log ingestion. - -Unlike `node_insight` (which can go to both the Query Node for live view and an Operator for archiving — -see Option 1 / Option 2 -above), `docker_insight` is forwarded to an **Operator node only**. There's no live-view equivalent for Docker -metrics — the goal here is archival, not a real-time snapshot. - -> **Prerequisite:** Docker monitoring reads container stats via the Docker socket (typically `/var/run/docker.sock` -> on the host). **To verify:** confirm the exact AnyLog-side configuration for pointing at this path (dictionary -> variable, mount path, or otherwise) before publishing — that detail wasn't provided here and shouldn't be guessed at. - -### Enable the pull - -```anylog -run scheduled pull where name = docker_insights and type = docker and frequency = !docker_frequency and continuous = false and dbms = monitoring and table = docker_insight -``` - -This streams container stats into `monitoring.docker_insight` — the same `monitoring` logical database used for -`node_insight` above, so both can be queried side by side. - -### Command reference - -`run scheduled pull` is a general-purpose command — `type = docker` is one of several supported source types (another -being `eventlog`, shown below): - -``` -AL > help run scheduled pull - -Usage: - run scheduled pull where name = [unique name] and type = [log typ -e] and source = [localhost or IP] and frequency = [in seconds] and dbms = [dbms name] and table = [table name] - -Explanation: - Periodically retrieve data from a specified source (such as Windows Event Log) and insert it into a defined table. - -Examples: - run scheduled pull where name = local_events and type = eventlog -and source = localhost and frequency = 1 and dbms = sensor_data and table = event_log - -Index: - ['streaming', 'api', 'configuration', 'background processes'] -``` - -### Querying Docker stats - -```anylog -run client () sql monitoring format = table "select * from docker_insight order by timestamp desc limit 20" -``` - ---- - -## Configuration - -| Variable | Description | Default | -|---|---|---| -| `NODE_MONITORING` | Enable the monitoring schedule | `true` | -| `STORE_MONITORING` | Enable persistent storage to Operator | `true` | -| `MONITORING_FREQUENCY` | Collection frequency for operator stats | `60 seconds` | -| `DOCKER_FREQUENCY` | Collection frequency for `run scheduled pull` docker stats — used as `!docker_frequency` above | *(to verify — not specified in source material)* | - -The monitoring schedule is stored as a blockchain policy and can be inspected at any time: - -```anylog -blockchain get schedule where id = node-monitoring -``` diff --git a/04- Southbound Interfaces/04- Monitoring/02- Syslog.md b/04- Southbound Interfaces/04- Monitoring/02- Syslog.md deleted file mode 100644 index 1dfcbaa6..00000000 --- a/04- Southbound Interfaces/04- Monitoring/02- Syslog.md +++ /dev/null @@ -1,300 +0,0 @@ ---- -title: Syslog Integration -description: Ingest BSD and IETF syslog messages from Linux, Mac, and network devices directly into AnyLog — prerequisites, node configuration, and running/validating the pipeline. -layout: page -source_path: "07- Southbound Interfaces/Syslog Integration.md" ---- - - - -Syslog is a standardized protocol for sending and -receiving log messages across a network. AnyLog can act as a syslog receiver, accepting messages from any host that -supports TCP syslog output and storing them as queryable time-series data alongside all other data in the network — -letting you monitor and troubleshoot the status of many machines from a single point rather than checking each one -separately. - -> **Prefer automation?** Everything below — the rsyslog forwarding rule and the AnyLog-side message rule — can be set -> up automatically via the dotenv-driven `syslog.sh` script, wired into `make setup SERVICE=syslog`. It reads -> `SYSLOG_MONITORING` and `ANYLOG_BROKER_PORT` from your node's `node_configs.env` and is idempotent (safe to re-run, -> and a no-op if `SYSLOG_MONITORING != "true"`). See the Docker & K8s Commands -> doc for the exact command. The manual walkthrough below is for understanding what that automation is doing, or for -> setting things up by hand. - ---- - -## 1. Prerequisites - -### On the physical machine — install and start rsyslog - -```shell -sudo apt-get -y update -sudo apt -y install rsyslog -sudo service rsyslog start -``` - -> The same steps apply to syslog-ng if preferred over rsyslog. - -Validate rsyslog is running: - -```shell -tail -f /var/log/syslog -``` - -**Expected output:** -``` -Feb 25 02:55:47 localhost systemd[1]: Started User Manager for UID 0. -Feb 25 02:55:47 localhost systemd[1]: Started Session 197 of User root. -Feb 25 02:55:52 localhost systemd-udevd[400]: Network interface NamePolicy= disabled on kernel command line, ignoring. -Feb 25 02:55:53 localhost dbus-daemon[31261]: AppArmor D-Bus mediation is enabled -... -``` - ---- - -## 2. Configure the Node - -### Understand the syslog formats - -Syslog is delivered from each machine in one of two formats — knowing which one you're receiving determines how you'll -set the `syslog` / `format` options on the rule below. - -| Format | Standard | Timestamp | Key fields | -|---|---|---|---| -| BSD | RFC 3164 | `MMM dd hh:mm:ss` | Priority, Timestamp, Hostname, Tag (process + PID), Message | -| IETF | RFC 5424 | ISO 8601 | Priority, Version, Timestamp, Hostname, Application, PID, Message ID, Structured Data, Message | - -
-Field-by-field breakdown - -**BSD format fields** -1. **Priority** — enclosed in angle brackets (`<` and `>`); a numeric value combining facility and severity (e.g. `<34>`). -2. **Timestamp** — immediately follows the priority, typically `MMM dd hh:mm:ss` (e.g. `Jan 12 23:34:56`). -3. **Hostname or IP address** — the name or IP of the device that sent the message. -4. **Tag** — often a process name or application identifier, potentially followed by a process ID in square brackets (e.g. `sshd[3268]`). -5. **Message** — the actual log message text, following the tag. - -**IETF format fields** -1. **Priority** — same as BSD, enclosed in angle brackets. -2. **Version** — a single digit indicating the syslog protocol version (e.g. `1`). -3. **Timestamp** — more precise than BSD, typically ISO 8601. -4. **Hostname** — as in BSD format. -5. **Application** — the name of the application or process generating the message. -6. **Process ID (PID)** — the PID of the process. -7. **Message ID** — a unique identifier for the type of message. -8. **Structured data** — enclosed in square brackets, key-value pairs for additional data. -9. **Message** — the actual log message text. - -
- -### Start the message broker - -The message broker is the TCP listener that receives syslog traffic. AnyLog's regular TCP service is dedicated to -communication *between* AnyLog nodes and cannot be used for external data ingestion — the message broker is the -correct service for receiving data from outside the network. Start it on the operator or publisher node: - -```anylog - -``` - -Check which IP/port to direct syslog output to: - -```anylog -get connections -``` - -### Point rsyslog at AnyLog - -Add the following to the bottom of `/etc/rsyslog.conf`, replacing `DESTINATION_IP` and `DESTINATION_PORT` with the -AnyLog operator/publisher IP and message broker port found above: - -``` -$template remote-incoming-logs, "/var/log/remote/%HOSTNAME%.log" -*.* ?remote-incoming-logs -*.* action(type="omfwd" target="{DESTINATION_IP}" port="{DESTINATION_PORT}" protocol="tcp") -``` - -Restart rsyslog to apply: - -```shell -sudo service rsyslog restart -``` - -> This is exactly the step `syslog.sh setup` automates on Linux (via an rsyslog drop-in) and on macOS (via -> `/etc/syslog.conf`) — see the automation callout above. - -### Set a syslog rule - -Rules tell AnyLog how to route and parse incoming syslog messages: - -```anylog -set msg rule [rule name] if ip = [source IP] and port = [port] and header = [header text] then dbms = [dbms] and table = [table] and syslog = [true/false] and extend = ip and format = [format] and topic = [topic] -``` - -| Option | Required | Description | -|---|---|---| -| `rule name` | ✅ | Unique name for this rule | -| `ip` | — | Source IP to match — omit to match all IPs | -| `port` | — | Source port to match — omit to match all ports | -| `header` | — | Match messages with a specific prefix string (see Example 2 below) | -| `dbms` | ✅ | Target logical database | -| `table` | ✅ | Target table | -| `syslog` | — | `true` — parse as BSD syslog. Set `format = IETF` for RFC 5424 | -| `extend` | — | Add extra fields — `extend = ip` adds the source IP | -| `format` | — | Override the default format: `IETF` for RFC 5424 | -| `topic` | — | Route through the msg-client mapping layer (like MQTT) | -| `structure` | — | `included` — first message event defines the column schema (see Example 3 below) | - -> When `syslog = true`, column names are pre-determined by the format (BSD by default, or `format = IETF`). -> When `syslog` is not set, use `structure = included` so the first event defines the schema instead. - -**Basic example** — accept BSD syslog from this node's own default database/table: - -```anylog -set msg rule syslog_rule if ip = !ip then dbms = new_company and table = syslog and syslog = true -``` - -### Partition & clean up (recommended) - -Syslog volume adds up quickly — partition the table and schedule automatic cleanup so it doesn't grow unbounded: - -```anylog -connect dbms new_company where type=sqlite -partition new_company syslog using insert_timestamp by 12 hours -schedule time = 12 hours and name = "Drop Partition Sync - Syslog" task drop partition where dbms = new_company and table = syslog and keep = 3 -``` - -### More configuration examples - -**Example — Linux `journalctl` via netcat, with a header prefix** - -Pipe `journalctl` output to AnyLog, prefixing each line with a custom header so the rule below can match on it: - -```bash -journalctl --since "${NOW}" | awk '{print "al.sl.header.new_company.syslog", $0}' | nc -w 1 10.0.0.78 7850 -``` - -```anylog -set msg rule my_rule if ip = 10.0.0.50 and header = al.sl.header.new_company.syslog then dbms = new_company and table = syslog and syslog = true -``` - -**Example — Mac syslog with dynamic structure from the first event** - -```bash -(log show --info --start '2024-01-01 16:50:00' --end '2024-12-01 16:51:00' | awk '{print "al.sl", $0}') | nc -w 1 10.0.0.78 7850 -``` - -The first event contains the column headers, which `structure = included` uses to define the schema: - -``` -al.sl Timestamp Thread Type Activity PID TTL -al.sl 2024-01-01 17:51:35.253053-0800 0x4d0c71 Default 0x39223d 482 3 ... -``` - -```anylog -set msg rule my_rule if ip = 10.0.0.251 and header = al.sl then dbms = new_company and table = syslog_mac and structure = included -``` - ---- - -## 3. Run & Validate - -### Trigger test data - -On the monitored machine, run an update/upgrade (or anything else that generates log activity): - -```shell -sudo apt-get -y update -sudo apt-get -y upgrade -``` - -### Confirm the rule is active and receiving data - -```anylog -get msg rules -``` - -**Expected output:** -``` -Name IF IF IF THEN THEN THEN THEN THEN Batches Events Errors Error Msg - Source IP Port Header DBMS Table SysLog Topic Structure ------------|-------------|-----|-------|-----------|------|-------|------|----------|-------|------|------|---------| -syslog_rule|10.0.0.78 |* | |new_company|syslog|True | | | 18| 32| 0| | -``` - -### Query the data - -From a query node: - -```anylog --- row count -run client () sql new_company format=table "select count(*) from syslog" - --- sample rows -run client () sql new_company "select * from syslog limit 10" -``` - -**Sample output** (note the `tsd_name` / `tsd_id` partition-metadata columns AnyLog adds automatically): - -```json -{"Query":[ - {"row_id":1, - "insert_timestamp":"2024-02-25 03:18:35.023262", - "tsd_name":"131", - "tsd_id":4610, - "priority":38, - "timestamp":"2024-02-25 03:17:27.000000", - "hostname":"localhost", - "tag":"sshd[32839]:", - "message":"Invalid user lighthouse from 10.0.0.100 port 45126"}, - {"row_id":2, - "insert_timestamp":"2024-02-25 03:18:35.023262", - "tsd_name":"131", - "tsd_id":4610, - "priority":85, - "timestamp":"2024-02-25 03:17:27.000000", - "hostname":"localhost", - "tag":"sshd[32839]:", - "message":"pam_unix(sshd:auth): check pass; user unknown"} -], -"Statistics":[{"Count": 10, "Time":"00:00:00", "Nodes": 1}]} -``` - -### Manage & debug - -```anylog -get msg rules -- list all active rules and their event counts -reset msg rule [rule name] -- remove a rule -``` - -Enable trace to see the source IP, port, and first 100 bytes of each incoming message: - -```anylog -trace level = 2 run message broker -``` - -**Example trace output:** -``` -[Message Broker Received 1650 Bytes] [Source: 10.0.0.78:1468] [Data: <134>Jan 26 17:30:10 DESKTOP sshd[3268] User login...] -``` - -A single message may contain multiple events — trace only shows the first 100 bytes of each. Use `get msg rules` to -see the actual number of events processed. \ No newline at end of file diff --git a/04- Southbound Interfaces/05- Monitoring/01- Node Monitoring.md b/04- Southbound Interfaces/05- Monitoring/01- Node Monitoring.md deleted file mode 100644 index ede25d01..00000000 --- a/04- Southbound Interfaces/05- Monitoring/01- Node Monitoring.md +++ /dev/null @@ -1,253 +0,0 @@ ---- -title: Node Monitoring -description: Collect node and container health metrics and stream them for live viewing via Remote GUI or persistent storage across Operator nodes. -layout: page ---- - - -Each AnyLog node can collect its own health metrics and distribute them in one or both of two ways: - -- **Live view** — push metrics to a Query Node and visualise them in the Remote GUI without storing any data -- **Persistent storage** — stream metrics into an Operator's database for historical queries and dashboards - -The monitoring schedule is deployed as a blockchain policy and activated automatically when `NODE_MONITORING=true` is set in the node configuration. - -> **Pull, not push:** unlike Syslog, where data is *pushed* into AnyLog by an external -> forwarder (rsyslog), node and Docker monitoring are *pulled* on a schedule — the node actively queries its own OS -> and Docker metrics at each interval, rather than waiting for something to send data in. - -**A node generates its own "insight"** — a JSON object describing its current state. Two kinds exist on this page: - -| Insight type | What it captures | Sent to Query Node (live view) | Sent to Operator (archive) | -|---|---|---|---| -| `node_insight` | OS/agent-level metrics — CPU, disk, network, ingestion stats (see below) | ✅ | ✅ | -| `docker_insight` | Container-level stats (see Docker Monitoring) | ❌ | ✅ | - -The reason for that split: -* **Query Node** — a live, in-memory snapshot, almost like running `top` across every node in the network. It only - makes sense for metrics you'd want to glance at *right now*. -* **Operator** — a persistent archive for historical queries, trending, and dashboards. This is where you'd look to - answer "what happened over the last week," not "what's happening this second." - -`docker_insight` is only sent to the Operator archive — there's no live-view path for it (see the Docker Monitoring -section below). - -> **Prerequisite:** any form of monitoring on this page — node insight collection, live view, persistent storage, or -> Docker monitoring below — runs as a scheduled task. The scheduler itself must be enabled, or nothing will fire: -> ```anylog -> run scheduler 1 -> ``` - ---- - -## Collecting node insight - -Each participating node runs a set of scheduled tasks that build a JSON object called `node_insight`. This object is assembled from system metrics collected at two intervals: - -| Metric | AnyLog command | Frequency | -|---|---|---| -| Operator ingestion stats | `get stats where service = operator and topic = summary` | `!monitoring_frequency` | -| Timestamp | `get datetime local now()` | `!monitoring_frequency` | -| Node type | `!node_type` | `!monitoring_frequency` | -| Disk free (%) | `get disk percentage .` | 30 seconds | -| CPU usage (%) | `get node info cpu_percent` | 30 seconds | -| Network packets received | `get node info net_io_counters packets_recv` | 30 seconds | -| Network packets sent | `get node info net_io_counters packets_sent` | 30 seconds | -| Network error count | `errin + errout` | 30 seconds | -| Status | `Active` | 30 seconds | - -The resulting `node_insight` object: - -```json -{ - "timestamp": "2026-03-07 18:41:11", - "node type": "operator", - "Free Space Percent": 72.4, - "CPU Percent": 6.7, - "Packets Recv": 1482930, - "Packets Sent": 983421, - "Network Error": 0, - "status": "Active" -} -``` - -`!monitoring_frequency` is set in the node's `.env` file (default: 60 seconds). Hardware metrics always collect at 30 seconds regardless. - ---- - -## Option 1 — Live view via Remote GUI - -Each node pushes its `node_insight` to the Query Node, which aggregates status from all participating nodes into a single view. The Remote GUI reads from the Query Node to display a live network-wide dashboard — no database required. - -``` -Node A ──┐ -Node B ──┼──► Query Node ──► Remote GUI -Node C ──┘ (aggregator) -``` - -The Query Node destination is resolved automatically from the blockchain: - -```anylog -view_monitoring_dest = blockchain get query bring.ip_port -``` - -Each node then pushes its metrics on a 30-second schedule: - -```anylog -run client (!view_monitoring_dest) monitor operators where info = !node_insight -``` - -The Remote GUI Monitor Node section shows a live table of all nodes pushing metrics to the Query Node: - - - Monitor Node Section in Remote GUI - - -Each row represents one node, showing its name, operational time, elapsed time since last update, new and total rows -ingested, and the hardware metrics — free disk space, CPU usage, network packets, and error count. - -The **Add Threshold Monitor** panel at the top lets you set alert thresholds on any column — for example, alert when -CPU Percent exceeds 80 or Free Space Percent drops below 20. - -**View from the CLI:** -```anylog -get monitored # list all monitored topics -get monitored operators # current status from all nodes -reset monitored operators # clear the aggregated list -``` - -This option requires no storage configuration — metrics are held in memory on the Query Node and reflect the current -state only. - ---- - -## Option 2 — Persistent storage across Operators - -When `STORE_MONITORING=true`, each node streams its `node_insight` into a `monitoring.node_insight` table on an -Operator node, enabling historical queries and Grafana dashboards. - -``` -Node A ──┐ -Node B ──┼──► Operator (monitoring.node_insight) -Node C ──┘ -``` - -**On Operator nodes** — stored locally: -```anylog -stream !node_insight where dbms = monitoring and table = node_insight -``` - -**On non-Operator nodes** — routed to a remote Operator: -```anylog -run client (!monitoring_storage_dest) stream !node_insight where dbms = monitoring and table = node_insight -``` - -The target Operator is resolved automatically: -```anylog -monitoring_storage_dest = blockchain get operator bring.last [*][ip]:[*][port] -``` - -### Table setup - -On the Operator hosting the monitoring data, the `node_insight` table is created automatically on startup: - -```anylog -connect dbms monitoring where type = sqlite -process !anylog_path/deployment-scripts/southbound-monitoring/create_node_monitoring_table.al -``` - -### Querying stored metrics - -```anylog -# Local query on the Operator -sql monitoring format = table "select * from node_insight order by timestamp desc limit 20" - -# Network query from any node -run client () sql monitoring format = table "select timestamp, node_type, cpu_percent, free_space_percent from node_insight where timestamp >= NOW() - 1 hour" -``` - ---- - -## Using both options together - -Options 1 and 2 are not mutually exclusive. Most deployments run both — live view for real-time dashboards and -persistent storage for historical analysis and alerting. - ---- - -## Docker Monitoring - -In addition to node-level health metrics, AnyLog can pull container-level stats directly from Docker using a -**scheduled pull** task — the same general-purpose mechanism used for things like Windows Event Log ingestion. - -Unlike `node_insight` (which can go to both the Query Node for live view and an Operator for archiving — -see Option 1 / Option 2 -above), `docker_insight` is forwarded to an **Operator node only**. There's no live-view equivalent for Docker -metrics — the goal here is archival, not a real-time snapshot. - -> **Prerequisite:** Docker monitoring reads container stats via the Docker socket (typically `/var/run/docker.sock` -> on the host). **To verify:** confirm the exact AnyLog-side configuration for pointing at this path (dictionary -> variable, mount path, or otherwise) before publishing — that detail wasn't provided here and shouldn't be guessed at. - -### Enable the pull - -```anylog -run scheduled pull where name = docker_insights and type = docker and frequency = !docker_frequency and continuous = false and dbms = monitoring and table = docker_insight -``` - -This streams container stats into `monitoring.docker_insight` — the same `monitoring` logical database used for -`node_insight` above, so both can be queried side by side. - -### Command reference - -`run scheduled pull` is a general-purpose command — `type = docker` is one of several supported source types (another -being `eventlog`, shown below): - -``` -AL > help run scheduled pull - -Usage: - run scheduled pull where name = [unique name] and type = [log typ -e] and source = [localhost or IP] and frequency = [in seconds] and dbms = [dbms name] and table = [table name] - -Explanation: - Periodically retrieve data from a specified source (such as Windows Event Log) and insert it into a defined table. - -Examples: - run scheduled pull where name = local_events and type = eventlog -and source = localhost and frequency = 1 and dbms = sensor_data and table = event_log - -Index: - ['streaming', 'api', 'configuration', 'background processes'] -``` - -### Querying Docker stats - -```anylog -run client () sql monitoring format = table "select * from docker_insight order by timestamp desc limit 20" -``` - ---- - -## Configuration - -| Variable | Description | Default | -|---|---|---| -| `NODE_MONITORING` | Enable the monitoring schedule | `true` | -| `STORE_MONITORING` | Enable persistent storage to Operator | `true` | -| `MONITORING_FREQUENCY` | Collection frequency for operator stats | `60 seconds` | -| `DOCKER_FREQUENCY` | Collection frequency for `run scheduled pull` docker stats — used as `!docker_frequency` above | *(to verify — not specified in source material)* | - -The monitoring schedule is stored as a blockchain policy and can be inspected at any time: - -```anylog -blockchain get schedule where id = node-monitoring -``` \ No newline at end of file diff --git a/04- Southbound Interfaces/05- Monitoring/02- Syslog.md b/04- Southbound Interfaces/05- Monitoring/02- Syslog.md deleted file mode 100644 index 1dfcbaa6..00000000 --- a/04- Southbound Interfaces/05- Monitoring/02- Syslog.md +++ /dev/null @@ -1,300 +0,0 @@ ---- -title: Syslog Integration -description: Ingest BSD and IETF syslog messages from Linux, Mac, and network devices directly into AnyLog — prerequisites, node configuration, and running/validating the pipeline. -layout: page -source_path: "07- Southbound Interfaces/Syslog Integration.md" ---- - - - -Syslog is a standardized protocol for sending and -receiving log messages across a network. AnyLog can act as a syslog receiver, accepting messages from any host that -supports TCP syslog output and storing them as queryable time-series data alongside all other data in the network — -letting you monitor and troubleshoot the status of many machines from a single point rather than checking each one -separately. - -> **Prefer automation?** Everything below — the rsyslog forwarding rule and the AnyLog-side message rule — can be set -> up automatically via the dotenv-driven `syslog.sh` script, wired into `make setup SERVICE=syslog`. It reads -> `SYSLOG_MONITORING` and `ANYLOG_BROKER_PORT` from your node's `node_configs.env` and is idempotent (safe to re-run, -> and a no-op if `SYSLOG_MONITORING != "true"`). See the Docker & K8s Commands -> doc for the exact command. The manual walkthrough below is for understanding what that automation is doing, or for -> setting things up by hand. - ---- - -## 1. Prerequisites - -### On the physical machine — install and start rsyslog - -```shell -sudo apt-get -y update -sudo apt -y install rsyslog -sudo service rsyslog start -``` - -> The same steps apply to syslog-ng if preferred over rsyslog. - -Validate rsyslog is running: - -```shell -tail -f /var/log/syslog -``` - -**Expected output:** -``` -Feb 25 02:55:47 localhost systemd[1]: Started User Manager for UID 0. -Feb 25 02:55:47 localhost systemd[1]: Started Session 197 of User root. -Feb 25 02:55:52 localhost systemd-udevd[400]: Network interface NamePolicy= disabled on kernel command line, ignoring. -Feb 25 02:55:53 localhost dbus-daemon[31261]: AppArmor D-Bus mediation is enabled -... -``` - ---- - -## 2. Configure the Node - -### Understand the syslog formats - -Syslog is delivered from each machine in one of two formats — knowing which one you're receiving determines how you'll -set the `syslog` / `format` options on the rule below. - -| Format | Standard | Timestamp | Key fields | -|---|---|---|---| -| BSD | RFC 3164 | `MMM dd hh:mm:ss` | Priority, Timestamp, Hostname, Tag (process + PID), Message | -| IETF | RFC 5424 | ISO 8601 | Priority, Version, Timestamp, Hostname, Application, PID, Message ID, Structured Data, Message | - -
-Field-by-field breakdown - -**BSD format fields** -1. **Priority** — enclosed in angle brackets (`<` and `>`); a numeric value combining facility and severity (e.g. `<34>`). -2. **Timestamp** — immediately follows the priority, typically `MMM dd hh:mm:ss` (e.g. `Jan 12 23:34:56`). -3. **Hostname or IP address** — the name or IP of the device that sent the message. -4. **Tag** — often a process name or application identifier, potentially followed by a process ID in square brackets (e.g. `sshd[3268]`). -5. **Message** — the actual log message text, following the tag. - -**IETF format fields** -1. **Priority** — same as BSD, enclosed in angle brackets. -2. **Version** — a single digit indicating the syslog protocol version (e.g. `1`). -3. **Timestamp** — more precise than BSD, typically ISO 8601. -4. **Hostname** — as in BSD format. -5. **Application** — the name of the application or process generating the message. -6. **Process ID (PID)** — the PID of the process. -7. **Message ID** — a unique identifier for the type of message. -8. **Structured data** — enclosed in square brackets, key-value pairs for additional data. -9. **Message** — the actual log message text. - -
- -### Start the message broker - -The message broker is the TCP listener that receives syslog traffic. AnyLog's regular TCP service is dedicated to -communication *between* AnyLog nodes and cannot be used for external data ingestion — the message broker is the -correct service for receiving data from outside the network. Start it on the operator or publisher node: - -```anylog - -``` - -Check which IP/port to direct syslog output to: - -```anylog -get connections -``` - -### Point rsyslog at AnyLog - -Add the following to the bottom of `/etc/rsyslog.conf`, replacing `DESTINATION_IP` and `DESTINATION_PORT` with the -AnyLog operator/publisher IP and message broker port found above: - -``` -$template remote-incoming-logs, "/var/log/remote/%HOSTNAME%.log" -*.* ?remote-incoming-logs -*.* action(type="omfwd" target="{DESTINATION_IP}" port="{DESTINATION_PORT}" protocol="tcp") -``` - -Restart rsyslog to apply: - -```shell -sudo service rsyslog restart -``` - -> This is exactly the step `syslog.sh setup` automates on Linux (via an rsyslog drop-in) and on macOS (via -> `/etc/syslog.conf`) — see the automation callout above. - -### Set a syslog rule - -Rules tell AnyLog how to route and parse incoming syslog messages: - -```anylog -set msg rule [rule name] if ip = [source IP] and port = [port] and header = [header text] then dbms = [dbms] and table = [table] and syslog = [true/false] and extend = ip and format = [format] and topic = [topic] -``` - -| Option | Required | Description | -|---|---|---| -| `rule name` | ✅ | Unique name for this rule | -| `ip` | — | Source IP to match — omit to match all IPs | -| `port` | — | Source port to match — omit to match all ports | -| `header` | — | Match messages with a specific prefix string (see Example 2 below) | -| `dbms` | ✅ | Target logical database | -| `table` | ✅ | Target table | -| `syslog` | — | `true` — parse as BSD syslog. Set `format = IETF` for RFC 5424 | -| `extend` | — | Add extra fields — `extend = ip` adds the source IP | -| `format` | — | Override the default format: `IETF` for RFC 5424 | -| `topic` | — | Route through the msg-client mapping layer (like MQTT) | -| `structure` | — | `included` — first message event defines the column schema (see Example 3 below) | - -> When `syslog = true`, column names are pre-determined by the format (BSD by default, or `format = IETF`). -> When `syslog` is not set, use `structure = included` so the first event defines the schema instead. - -**Basic example** — accept BSD syslog from this node's own default database/table: - -```anylog -set msg rule syslog_rule if ip = !ip then dbms = new_company and table = syslog and syslog = true -``` - -### Partition & clean up (recommended) - -Syslog volume adds up quickly — partition the table and schedule automatic cleanup so it doesn't grow unbounded: - -```anylog -connect dbms new_company where type=sqlite -partition new_company syslog using insert_timestamp by 12 hours -schedule time = 12 hours and name = "Drop Partition Sync - Syslog" task drop partition where dbms = new_company and table = syslog and keep = 3 -``` - -### More configuration examples - -**Example — Linux `journalctl` via netcat, with a header prefix** - -Pipe `journalctl` output to AnyLog, prefixing each line with a custom header so the rule below can match on it: - -```bash -journalctl --since "${NOW}" | awk '{print "al.sl.header.new_company.syslog", $0}' | nc -w 1 10.0.0.78 7850 -``` - -```anylog -set msg rule my_rule if ip = 10.0.0.50 and header = al.sl.header.new_company.syslog then dbms = new_company and table = syslog and syslog = true -``` - -**Example — Mac syslog with dynamic structure from the first event** - -```bash -(log show --info --start '2024-01-01 16:50:00' --end '2024-12-01 16:51:00' | awk '{print "al.sl", $0}') | nc -w 1 10.0.0.78 7850 -``` - -The first event contains the column headers, which `structure = included` uses to define the schema: - -``` -al.sl Timestamp Thread Type Activity PID TTL -al.sl 2024-01-01 17:51:35.253053-0800 0x4d0c71 Default 0x39223d 482 3 ... -``` - -```anylog -set msg rule my_rule if ip = 10.0.0.251 and header = al.sl then dbms = new_company and table = syslog_mac and structure = included -``` - ---- - -## 3. Run & Validate - -### Trigger test data - -On the monitored machine, run an update/upgrade (or anything else that generates log activity): - -```shell -sudo apt-get -y update -sudo apt-get -y upgrade -``` - -### Confirm the rule is active and receiving data - -```anylog -get msg rules -``` - -**Expected output:** -``` -Name IF IF IF THEN THEN THEN THEN THEN Batches Events Errors Error Msg - Source IP Port Header DBMS Table SysLog Topic Structure ------------|-------------|-----|-------|-----------|------|-------|------|----------|-------|------|------|---------| -syslog_rule|10.0.0.78 |* | |new_company|syslog|True | | | 18| 32| 0| | -``` - -### Query the data - -From a query node: - -```anylog --- row count -run client () sql new_company format=table "select count(*) from syslog" - --- sample rows -run client () sql new_company "select * from syslog limit 10" -``` - -**Sample output** (note the `tsd_name` / `tsd_id` partition-metadata columns AnyLog adds automatically): - -```json -{"Query":[ - {"row_id":1, - "insert_timestamp":"2024-02-25 03:18:35.023262", - "tsd_name":"131", - "tsd_id":4610, - "priority":38, - "timestamp":"2024-02-25 03:17:27.000000", - "hostname":"localhost", - "tag":"sshd[32839]:", - "message":"Invalid user lighthouse from 10.0.0.100 port 45126"}, - {"row_id":2, - "insert_timestamp":"2024-02-25 03:18:35.023262", - "tsd_name":"131", - "tsd_id":4610, - "priority":85, - "timestamp":"2024-02-25 03:17:27.000000", - "hostname":"localhost", - "tag":"sshd[32839]:", - "message":"pam_unix(sshd:auth): check pass; user unknown"} -], -"Statistics":[{"Count": 10, "Time":"00:00:00", "Nodes": 1}]} -``` - -### Manage & debug - -```anylog -get msg rules -- list all active rules and their event counts -reset msg rule [rule name] -- remove a rule -``` - -Enable trace to see the source IP, port, and first 100 bytes of each incoming message: - -```anylog -trace level = 2 run message broker -``` - -**Example trace output:** -``` -[Message Broker Received 1650 Bytes] [Source: 10.0.0.78:1468] [Data: <134>Jan 26 17:30:10 DESKTOP sshd[3268] User login...] -``` - -A single message may contain multiple events — trace only shows the first 100 bytes of each. Use `get msg rules` to -see the actual number of events processed. \ No newline at end of file diff --git a/04- Southbound Interfaces/05- RPC & Media Streaming/01- gRPC.md b/04- Southbound Interfaces/05- RPC & Media Streaming/01- gRPC.md deleted file mode 100644 index a6a1e9ad..00000000 --- a/04- Southbound Interfaces/05- RPC & Media Streaming/01- gRPC.md +++ /dev/null @@ -1,193 +0,0 @@ ---- -title: gRPC -description: Connect AnyLog as a gRPC client to receive data streams from gRPC servers and map them to local database tables. -layout: page ---- - - -gRPC is Google's open-source RPC framework — efficient, -language-agnostic, and designed for high-throughput streaming. AnyLog connects as a gRPC **client**, receives data -streams from a gRPC server, and maps them to a local database using policies. - ---- - -## Setup - -### 1. Install grpcio-tools - -```bash -python3 -m pip install --upgrade grpcio-tools -``` - -### 2. Create a proto file - -The proto file defines the service, RPC methods, and message types. Example `dummy.proto`: - -```protobuf -syntax = "proto3"; - -package mygrpc; - -service SerializeService { - rpc GetSampleData (Empty) returns (SampleDataResponse); -} - -message Empty {} - -message SampleDataResponse { - repeated string serialized_data = 1; -} -``` - -### 3. Compile the proto file - -Run from the same directory as the `.proto` file: - -```bash -python3 -m grpc_tools.protoc -I. --python_out=. --grpc_python_out=. dummy.proto -``` - -This generates two files (e.g. `dummy_pb2.py` and `dummy_pb2_grpc.py`) that AnyLog uses to communicate with the server. - ---- - -## Starting a gRPC client - -```anylog - -``` - -### Parameter reference - -| Parameter | Required | Description | -|---|---|---| -| `name` | ✅ | Unique ID for this gRPC connection | -| `ip` | ✅ | gRPC server IP | -| `port` | ✅ | gRPC server port | -| `grpc_dir` | ✅ | Directory containing the compiled proto files | -| `proto` | ✅ | Proto base filename (without extension) | -| `function` | ✅ | RPC method to call on the server | -| `request` | ✅ | Request message type name from the proto | -| `response` | ✅ | Response message type name from the proto | -| `service` | ✅ | Service name from the proto | -| `policy` | — | Mapping policy ID for schema mapping | -| `value` | — | Key-value pairs sent in the request message (e.g. `Filter = system and Type = 5.int`) | -| `debug` | — | `true` prints received data to the CLI (default: `false`) | -| `limit` | — | Stop after N events | -| `dbms` | — | Target database (if not provided by policy) | -| `table` | — | Target table (if not provided by policy) | -| `ingest` | — | `false` disables database ingestion — useful for testing (default: `true`) | -| `add_info` | — | Append metadata to each row — see Options for `add_info` below | -| `invoke` | — | Whether to invoke immediately when called by another AnyLog process | - -### Examples - -```anylog - -``` - -```anylog - -``` - -### Options for `add_info` - -When `add_info` is included in the `run grpc client` command, the specified keys are added as extra fields to the -JSON data retrieved from the server. `add_info` can be repeated — once per key you want added: - -| Key | Value added to the JSON | -|---|---| -| `proto` | The name of the proto file | -| `request` | The name of the request message in the proto file | -| `conn` | The IP and port used | - -```anylog - -``` - ---- - -## Monitor and manage - -```anylog -# List all active gRPC clients -get grpc clients -``` - -Example output: -``` -ID Connection Proto Request Message Policy ID Timeouts Data Msg ----------|----------------|---------|----------------|------------------------|--------|--------| -health |10.0.0.251:32769|kubearmor|NonceMessage | | 0| 1254| -kubearmor|10.0.0.251:32769|kubearmor|RequestMessage |kubearmor-system-policy | 0| 0| -``` - -```anylog -# List services offered by the gRPC server (requires server reflection) -get grpc services where conn = [ip:port] - -# Stop a specific client -exit grpc [name] - -# Stop all clients -exit grpc all -``` - ---- - -## gRPC for video inference - -When used with video streaming, the gRPC client connects to a YOLOv5 (or similar) inference server. -See Video Streaming for the full -configuration including `video connect` and `run video stream`. diff --git a/04- Southbound Interfaces/05- RPC & Media Streaming/02- Video Streaming.md b/04- Southbound Interfaces/05- RPC & Media Streaming/02- Video Streaming.md deleted file mode 100644 index 5b3c2788..00000000 --- a/04- Southbound Interfaces/05- RPC & Media Streaming/02- Video Streaming.md +++ /dev/null @@ -1,206 +0,0 @@ ---- -title: Video Streaming -description: Ingest video streams from IP cameras, RTSP feeds, YouTube, and local files — with optional AI inference — into AnyLog. -layout: page ---- - - - -AnyLog can connect to video streams, record segmented clips to a blob database, and optionally run AI inference (e.g. YOLOv5 object detection) via a gRPC server. - ---- - -## Supported protocols - -| Protocol | Latency | Typical use | -|---|---|---| -| RTMP | ~1–2s | Live ingest (OBS, Twitch, YouTube) | -| RTMPS | ~1–2s | Secure RTMP over TLS | -| SRT | ~1–2s | Professional contribution feeds over unreliable networks | -| RTSP | ~1–5s | IP cameras and surveillance | -| HTTP/HTTPS / HLS | ~1–5s | Standard HTTP streams, m3u8 playlists | -| Local file (MP4, MOV, MKV…) | 0s | Stored video files on disk | - ---- - -## Prerequisites - -- MongoDB connected as the blob database (stores video segments) -- PostgreSQL or SQLite connected as the SQL database (stores metadata and detections) -- (Optional) A YOLOv5 inference gRPC server for object detection — see gRPC - ---- - -## Step 1 — Connect databases - -```anylog -# Blob database (MongoDB) - - -# SQL database - -``` - -> **Worth confirming:** both connections above use the same logical dbms name (`customers`) for two different -> physical backends (Mongo for blobs, Postgres for SQL). If AnyLog treats blob storage and SQL storage as separate -> namespaces under one shared logical name, this is correct as written — but if not, this needs two distinct -> logical dbms names instead. Flagging since I can't verify AnyLog's actual behavior here. - ---- - -## Step 2 — Set video variables - -```anylog -video_url = "https://www.youtube.com/watch?v=rnXIjl_Rzy4" # Times Square live -video_host = 127.0.0.1 -video_port = 8888 -video_table = video_table -``` - -Sample stream URLs: -- Abbey Road London: `https://www.youtube.com/watch?v=57w2gYXjRic` -- Times Square: `https://www.youtube.com/watch?v=rnXIjl_Rzy4` - ---- - -## Step 3 — Import the display function - -```anylog -import function where import_name = imshow and lib = external_lib.video_processing.cv2_stream_imshow and method = init_class -set function params where import_name = imshow and param_name = port and param_type = int and param_value = !video_port -set function params where import_name = imshow and param_name = host and param_value = !video_host -``` - ---- - -## Step 4 — Connect to the video stream - -### Without inference - -```anylog -