diff --git a/README.md b/README.md index 0d5a45d2..d31bc0dc 100644 --- a/README.md +++ b/README.md @@ -10,7 +10,7 @@ across industries like manufacturing, utilities, oil & gas, smart cities, retail ## Repository Layout -``` +```tree docker-compose/ ├── Makefile # Thin wrapper around deploy.sh ├── deploy.sh # Node lifecycle manager — works without make @@ -29,6 +29,16 @@ docker-compose/ │ ├── prep_configs.sh # Pre-flight config validation │ ├── docker-compose-template-base.yaml # Linux template (host networking) │ └── docker-compose-template-ports-base.yaml # Mac/Windows template (port mapping) +├── license-generator # code base for licnese verification + storage (Docker) +│ ├── Dockerfile +│ ├── README.md +│ ├── deploy.sh +│ ├── docker-compose.yml +│ ├── license_key.sh +│ ├── requirements.txt +│ └── src + │ ├── __init__.py + │ └── license-server.py └── support/ # External services (Grafana, Postgres, MongoDB, etc.) └── README.md ``` @@ -128,8 +138,8 @@ Variables (make) / Options (deploy.sh): IMAGE / --image Image repository NODE_NAME / --node-name Override container name TEST_CONN / --test-conn ip:port for test commands (default: auto-resolved) - LICENSE_KEY / --license-key License key for deployment (prompts if missing) - PROMPT_LICENSE / --prompt-license Prompt if no saved license is found (default: true) + LICENSE_KEY / --license-key License key (skips acceptance form if set in config file; runs form if passed manually) + PROMPT_LICENSE / --prompt-license Force the acceptance form even when a key is already saved (default: true) ``` ### Important Notes @@ -140,6 +150,52 @@ Variables (make) / Options (deploy.sh): - **License key**: A valid `LICENSE_KEY` is required. [Request one here](https://www.anylog.network/download). +### License Key + +Every `up` command validates a license before starting the node. The behavior depends on where the key comes from. + +**Resolution order** + +1. `--license-key ` flag (or `LICENSE_KEY=` env var / make variable) +2. `LICENSE_KEY=` line in `docker-makefiles//.env` or `node_configs.env` +3. Interactive prompt (if a TTY is available) + +**If the key is found in the config file** — deployment proceeds immediately. No prompts, no registration form, no network call to the license server. + +**If the key comes from manual input** (flag or interactive prompt) — the full acceptance flow runs: +1. Displays `LICENSE.txt` for review +2. Prompts for name, email, and project +3. Asks for explicit `yes/no` acceptance of the license agreement +4. POSTs the registration to the AnyLog license server +5. Writes the accepted key back into the config file (`LICENSE_KEY=""`) so future deployments skip the form + +**Passing the key explicitly** + +```shell +# deploy.sh +bash deploy.sh up --type operator --license-key "" + +# make +make up ANYLOG_TYPE=operator LICENSE_KEY="" +``` + +**Setting the key in the config file** (skips the form on all future runs) + +```dotenv +# docker-makefiles/my-operator/node_configs.env +LICENSE_KEY="" +``` + +**`--prompt-license` / `PROMPT_LICENSE`** + +Forces the acceptance form even when a key is already saved, useful for re-registering under a new name or project: + +```shell +bash deploy.sh up --type operator --prompt-license +make up ANYLOG_TYPE=operator PROMPT_LICENSE=true +``` + + ### Configuration-based Deployment For full control over node behavior, use a `node_configs.env` file. diff --git a/customers/barbara.json b/customers/barbara.json new file mode 100644 index 00000000..31d835e1 --- /dev/null +++ b/customers/barbara.json @@ -0,0 +1,360 @@ +{ + "appName": "anylog-standalone-operator", + "version": "1.4.2604", + "name": "AnyLog Standalone Operator", + "architectures": "x86_64,aarch64", + "type": "marketplace", + "releaseNotes": [ + "Initial AnyLog standalone operator marketplace template" + ], + "templateAppconfig": {}, + "uris": [ + { + "name": "AnyLog REST", + "serviceName": "anylog-standalone-operator", + "pattern": "http://:", + "link": true + } + ], + "services": [ + { + "name": "anylog-standalone-operator", + "ports": [ + { + "description": "TCP port for AnyLog node-to-node protocol", + "name": "ANYLOG_SERVER_PORT", + "readableName": "AnyLog server (TCP) port", + "type": "text", + "default": "32148", + "mandatory": true, + "regex": "^(?:[1-9]\\d{0,3}|[1-5]\\d{4}|6[0-4]\\d{3}|65[0-4]\\d{2}|655[0-2]\\d|6553[0-5])$" + }, + { + "description": "HTTP REST API port for AnyLog", + "name": "ANYLOG_REST_PORT", + "readableName": "AnyLog REST port", + "type": "text", + "default": "32149", + "mandatory": true, + "regex": "^(?:[1-9]\\d{0,3}|[1-5]\\d{4}|6[0-4]\\d{3}|65[0-4]\\d{2}|655[0-2]\\d|6553[0-5])$" + }, + { + "description": "MQTT / broker port used by AnyLog", + "name": "ANYLOG_BROKER_PORT", + "readableName": "AnyLog broker port", + "type": "text", + "default": "32150", + "mandatory": true, + "regex": "^(?:[1-9]\\d{0,3}|[1-5]\\d{4}|6[0-4]\\d{3}|65[0-4]\\d{2}|655[0-2]\\d|6553[0-5])$" + } + ], + "volumes": [], + "env": [ + { + "description": "AnyLog license key (hex payload with trailing JSON metadata)", + "name": "LICENSE_KEY", + "readableName": "License key", + "type": "text", + "default": "", + "mandatory": true, + "regex": "^[0-9a-fA-F]+\\{.*\\}$" + }, + { + "description": "Unique name for this AnyLog instance", + "name": "NODE_NAME", + "readableName": "Node name", + "type": "text", + "default": "anylog-standalone-operator", + "mandatory": true, + "regex": "^[a-zA-Z0-9]([a-zA-Z0-9._-]*[a-zA-Z0-9])?$" + }, + { + "description": "Master ledger endpoint as host:port (IPv4 or hostname)", + "name": "LEDGER_CONN", + "readableName": "Ledger connection", + "type": "text", + "default": "127.0.0.1:32148", + "mandatory": true, + "regex": "^(?:(?:25[0-5]|2[0-4]\\d|1\\d{2}|[1-9]?\\d)(?:\\.(?:25[0-5]|2[0-4]\\d|1\\d{2}|[1-9]?\\d)){3}|[a-zA-Z0-9](?:[a-zA-Z0-9._-]*[a-zA-Z0-9])?):(?:[1-9]\\d{0,3}|[1-5]\\d{4}|6[0-4]\\d{3}|65[0-4]\\d{2}|655[0-2]\\d|6553[0-5])$" + }, + { + "description": "Owner or organization label for this AnyLog instance", + "name": "COMPANY_NAME", + "readableName": "Company name", + "type": "text", + "default": "AnyLog Co.", + "mandatory": true, + "regex": "^[^\\r\\n]{1,256}$" + }, + { + "description": "Physical SQL database backend (sqlite or psql)", + "name": "DB_TYPE", + "readableName": "Database type", + "type": "text", + "default": "sqlite", + "mandatory": true, + "regex": "^(sqlite|psql)$" + }, + { + "description": "Host or IP address of the SQL database", + "name": "DB_IP", + "readableName": "Database host", + "type": "text", + "default": "127.0.0.1", + "mandatory": true, + "regex": "^(?:(?:25[0-5]|2[0-4]\\d|1\\d{2}|[1-9]?\\d)(?:\\.(?:25[0-5]|2[0-4]\\d|1\\d{2}|[1-9]?\\d)){3}|[a-zA-Z0-9](?:[a-zA-Z0-9._-]*[a-zA-Z0-9])?)$" + }, + { + "description": "TCP port of the SQL database", + "name": "DB_PORT", + "readableName": "Database port", + "type": "text", + "default": "5432", + "mandatory": true, + "regex": "^(?:[1-9]\\d{0,3}|[1-5]\\d{4}|6[0-4]\\d{3}|65[0-4]\\d{2}|655[0-2]\\d|6553[0-5])$" + }, + { + "description": "Default logical DBMS name used by the operator", + "name": "DEFAULT_DBMS", + "readableName": "Default DBMS", + "type": "text", + "default": "new_company", + "mandatory": true, + "regex": "^[a-zA-Z0-9_]{1,64}$" + }, + { + "description": "Whether to enable the default MQTT ingestion process", + "name": "ENABLE_MQTT", + "readableName": "Enable MQTT", + "type": "text", + "default": "true", + "mandatory": true, + "regex": "^(true|false)$" + }, + { + "description": "IP address or hostname of the MQTT broker", + "name": "MQTT_BROKER", + "readableName": "MQTT broker host", + "type": "text", + "default": "172.104.228.251", + "mandatory": false, + "regex": "^(?:(?:25[0-5]|2[0-4]\\d|1\\d{2}|[1-9]?\\d)(?:\\.(?:25[0-5]|2[0-4]\\d|1\\d{2}|[1-9]?\\d)){3}|[a-zA-Z0-9](?:[a-zA-Z0-9._-]*[a-zA-Z0-9])?)$" + }, + { + "description": "TCP port of the MQTT broker", + "name": "MQTT_PORT", + "readableName": "MQTT broker port", + "type": "text", + "default": "1883", + "mandatory": false, + "regex": "^(?:[1-9]\\d{0,3}|[1-5]\\d{4}|6[0-4]\\d{3}|65[0-4]\\d{2}|655[0-2]\\d|6553[0-5])$" + }, + { + "description": "Whether to enable MQTT client logging", + "name": "MQTT_LOG", + "readableName": "MQTT logging", + "type": "text", + "default": "false", + "mandatory": true, + "regex": "^(true|false)$" + }, + { + "description": "Username for MQTT broker authentication", + "name": "MQTT_USER", + "readableName": "MQTT user", + "type": "text", + "default": "anyloguser", + "mandatory": false, + "regex": "^[^\\r\\n]{0,128}$" + }, + { + "description": "Password for MQTT broker authentication", + "name": "MQTT_PASSWD", + "readableName": "MQTT password", + "type": "text", + "default": "mqtt4AnyLog!", + "mandatory": false, + "regex": "" + }, + { + "description": "MQTT topic to subscribe for ingested messages", + "name": "MSG_TOPIC", + "readableName": "Message topic", + "type": "text", + "default": "rand-data", + "mandatory": false, + "regex": "^[^\\r\\n]{1,256}$" + }, + { + "description": "Logical database (DBMS) name for MQTT message storage", + "name": "MSG_DBMS", + "readableName": "Message DBMS", + "type": "text", + "default": "mydb", + "mandatory": false, + "regex": "^[a-zA-Z0-9_]{1,64}$" + }, + { + "description": "Target table mapping for MQTT payloads (e.g. bring [table])", + "name": "MSG_TABLE", + "readableName": "Message table", + "type": "text", + "default": "bring [table]", + "mandatory": false, + "regex": "^[^\\r\\n]{1,256}$" + }, + { + "description": "Timestamp column mapping for MQTT payloads", + "name": "MSG_TIMESTAMP_COLUMN", + "readableName": "Message timestamp column", + "type": "text", + "default": "bring [timestamp]", + "mandatory": false, + "regex": "^[^\\r\\n]{1,256}$" + }, + { + "description": "Value column mapping for MQTT payloads", + "name": "MSG_VALUE_COLUMN", + "readableName": "Message value column", + "type": "text", + "default": "bring [value]", + "mandatory": false, + "regex": "^[^\\r\\n]{1,256}$" + }, + { + "description": "SQL type for the value column (e.g. float, int)", + "name": "MSG_VALUE_COLUMN_TYPE", + "readableName": "Message value column type", + "type": "text", + "default": "float", + "mandatory": false, + "regex": "^[a-zA-Z0-9_]{1,32}$" + }, + { + "description": "Whether to enable the OPC-UA client service", + "name": "ENABLE_OPCUA", + "readableName": "Enable OPC-UA", + "type": "text", + "default": "true", + "mandatory": true, + "regex": "^(true|false)$" + }, + { + "description": "OPC-UA server endpoint URL (e.g. opc.tcp://host:4840/...)", + "name": "OPCUA_URL", + "readableName": "OPC-UA URL", + "type": "text", + "default": "opc.tcp://virtualfactory.proveit.services:4842/discovery", + "mandatory": false, + "regex": "^opc\\.tcp://[^\\s\\r\\n]{8,512}$" + }, + { + "description": "OPC-UA node id to read (e.g. ns=2;s=Site1)", + "name": "OPCUA_NODE", + "readableName": "OPC-UA node", + "type": "text", + "default": "ns=2;s=Site1", + "mandatory": false, + "regex": "^[^\\r\\n]{1,512}$" + }, + { + "description": "Polling interval in seconds for OPC-UA reads", + "name": "OPCUA_FREQUENCY", + "readableName": "OPC-UA frequency (seconds)", + "type": "text", + "default": "5", + "mandatory": false, + "regex": "^[1-9][0-9]{0,5}$" + }, + { + "description": "Geographic coordinates for this node (e.g. for Grafana maps)", + "name": "LOCATION", + "readableName": "Location", + "type": "text", + "default": "", + "mandatory": false, + "regex": "^[^\\r\\n]{0,256}$" + }, + { + "description": "Country where this machine is located", + "name": "COUNTRY", + "readableName": "Country", + "type": "text", + "default": "", + "mandatory": false, + "regex": "^[^\\r\\n]{0,128}$" + }, + { + "description": "State or region where this machine is located", + "name": "STATE", + "readableName": "State", + "type": "text", + "default": "", + "mandatory": false, + "regex": "^[^\\r\\n]{0,128}$" + }, + { + "description": "City where this machine is located", + "name": "CITY", + "readableName": "City", + "type": "text", + "default": "", + "mandatory": false, + "regex": "^[^\\r\\n]{0,128}$" + }, + { + "description": "Whether to enable DNS resolution for advertised endpoints", + "name": "ENABLE_DNS", + "readableName": "Enable DNS", + "type": "text", + "default": "false", + "mandatory": true, + "regex": "^(true|false)$" + }, + { + "description": "External DNS server or resolver hint", + "name": "EXTERNAL_DNS", + "readableName": "External DNS", + "type": "text", + "default": "", + "mandatory": false, + "regex": "^[^\\r\\n]{0,256}$" + }, + { + "description": "DNS domain appended to hostnames (e.g. example.com)", + "name": "DNS_DOMAIN", + "readableName": "DNS domain", + "type": "text", + "default": "", + "mandatory": false, + "regex": "^([a-zA-Z0-9._-]{0,253})?$" + }, + { + "description": "Path or URI for local DNS configuration used by the node", + "name": "DNS", + "readableName": "DNS configuration", + "type": "text", + "default": "", + "mandatory": false, + "regex": "^[^\\r\\n]{0,512}$" + }, + { + "description": "Network interface used to resolve the node IP (e.g. eth0, lo); leave empty for default", + "name": "NIC_TYPE", + "readableName": "NIC type", + "type": "text", + "default": "", + "mandatory": false, + "regex": "^([a-zA-Z0-9._-]{1,15})?$" + } + ] + } + ], + "marketDescription": { + "categories": [], + "developer": "AnyLog", + "docker": true, + "longDesc": "AnyLog network node (standalone operator): TCP, REST, and broker ports with core environment configuration.", + "shortDesc": "AnyLog standalone operator node." + } +} diff --git a/deploy.sh b/deploy.sh index 8835ec31..8df53e18 100644 --- a/deploy.sh +++ b/deploy.sh @@ -2,7 +2,7 @@ # deploy.sh — AnyLog node lifecycle manager # Usage: bash deploy.sh [OPTIONS] # Run: bash deploy.sh help -#set -euo pipefail +set -euo pipefail # ────────────────────────────────────────────── # Defaults (override via environment or flags) @@ -162,7 +162,7 @@ cmd_dry_run() { _load_configs if [[ "$IS_MANUAL" == "false" ]]; then echo "Dry Run ${ANYLOG_TYPE} - ${NODE_NAME}" - bash docker-makefiles/prep_configs.sh "${ANYLOG_TYPE}" +# bash docker-makefiles/prep_configs.sh "${ANYLOG_TYPE}" bash docker-makefiles/build_docker_compose.sh "${ANYLOG_TYPE}" "${TAG}" elif [[ "${IS_MANUAL}" == "true" ]]; then local single_file="docker-makefiles/${ANYLOG_TYPE}/node_configs.env" @@ -219,7 +219,6 @@ cmd_up() { fi } - cmd_down() { _check_configs _load_configs @@ -249,6 +248,7 @@ cmd_clean() { echo "Stopping + removing volumes: ${ANYLOG_TYPE}" ${DOCKER_COMPOSE_CMD} -f "${DOCKER_COMPOSE_FILE}" down -v fi + bash docker-makefiles/clean_configs.sh "${ANYLOG_TYPE}" } cmd_clean_all() { @@ -268,6 +268,7 @@ cmd_clean_all() { echo "Stopping + removing volumes + image: ${ANYLOG_TYPE}" ${DOCKER_COMPOSE_CMD} -f "${DOCKER_COMPOSE_FILE}" down -v --rmi all fi + bash docker-makefiles/clean_configs.sh "${ANYLOG_TYPE}" } cmd_logs() { @@ -302,39 +303,33 @@ cmd_license_check() { local env_file="docker-makefiles/${ANYLOG_TYPE}/.env" local single_file="docker-makefiles/${ANYLOG_TYPE}/node_configs.env" local license_arg=() - local accepted_license="" + local license_from_file=false case "${PROMPT_LICENSE}" in true|True|TRUE|1|yes|Yes|YES) PROMPT_LICENSE="true" ;; *) PROMPT_LICENSE="false" ;; esac - if [[ -f ".license_accepted" ]]; then - accepted_license=$(awk -F'|' 'NF >= 5 {print $5; exit}' .license_accepted) - accepted_license="${accepted_license#\"}" ; accepted_license="${accepted_license%\"}" - accepted_license="${accepted_license#\'}" ; accepted_license="${accepted_license%\'}" - fi - - if [[ "${LICENSE_KEY_PROVIDED}" != "true" && -n "${accepted_license}" ]]; then - LICENSE_KEY="${accepted_license}" - export LICENSE_KEY - return 0 - fi - - if [[ "${PROMPT_LICENSE}" == "true" && "${LICENSE_KEY_PROVIDED}" != "true" ]]; then - LICENSE_KEY="" - fi - + # ── Resolve key from file ───────────────────────────────────────── if [[ -z "${LICENSE_KEY}" && -f "${env_file}" ]]; then LICENSE_KEY=$(_get_config_value "$env_file" "LICENSE_KEY") + [[ -n "${LICENSE_KEY}" ]] && license_from_file=true elif [[ -z "${LICENSE_KEY}" && -f "${single_file}" ]]; then LICENSE_KEY=$(_get_config_value "$single_file" "LICENSE_KEY") + [[ -n "${LICENSE_KEY}" ]] && license_from_file=true fi LICENSE_KEY="${LICENSE_KEY#\"}" ; LICENSE_KEY="${LICENSE_KEY%\"}" LICENSE_KEY="${LICENSE_KEY#\'}" ; LICENSE_KEY="${LICENSE_KEY%\'}" - if [[ -z "${LICENSE_KEY}" && "${PROMPT_LICENSE}" != "true" ]] ; then + # ── If key came from file, skip the entire acceptance process ───── + if [[ "${license_from_file}" == "true" && "${LICENSE_KEY_PROVIDED}" != "true" ]]; then + export LICENSE_KEY + return 0 + fi + + # ── No key yet — prompt if possible ────────────────────────────── + if [[ -z "${LICENSE_KEY}" ]]; then if [[ -t 0 || -r /dev/tty ]]; then echo "A license key is required to deploy AnyLog." read -r -p "License Key: " LICENSE_KEY temp.yaml && mv temp.yaml "${COMPOSE_FILE}" else - awk -v cfg="../../docker-makefiles/${NODE_CONFIGS}/node_configs.env" \ + awk -v cfg="../../docker-makefiles/${NODE_CONFIGS}/formatted_node_configs.env" \ '/ env_file:/ {print; print " - " cfg; next}1' \ "${COMPOSE_FILE}" > temp.yaml && mv temp.yaml "${COMPOSE_FILE}" fi @@ -215,7 +219,6 @@ ${INJECT} done fi - # -------- Envsubst & Write Output -------- echo "Generating final docker-compose.yaml..." mkdir -p docker-makefiles/docker-compose-files @@ -223,3 +226,31 @@ OUTPUT_FILE="docker-makefiles/docker-compose-files/${NODE_CONFIGS}-docker-compos envsubst < "${COMPOSE_FILE}" > "$OUTPUT_FILE" rm -rf ${COMPOSE_FILE} ${COMPOSE_FILE}.bak echo "Saved: ${OUTPUT_FILE}" + + + +## Filename: {formatted_node_name}.env (hyphens -> underscores, lowercased) +## Empty-value vars ( VAR="" ) are commented out in the copy. +##FORMATTED_NODE_NAME=$(echo "${NODE_CONFIGS}" | tr '[:upper:]' '[:lower:]' | tr '-' '_') +#SNAPSHOT_DIR="docker-makefiles/${NODE_CONFIGS}" +#SNAPSHOT_FILE="${SNAPSHOT_DIR}/formatted_node_configs.env" +# +#echo "Generating read-only snapshot: ${SNAPSHOT_FILE}" +# +## Collect config files that were used +#if [[ "$MULTI_FILE" == "true" ]]; then +# SNAPSHOT_SOURCES=("${ENV_FILE}" "${BASE_ENV}" "docker-makefiles/${NODE_CONFIGS}/advance_configs.env") +#else +# SNAPSHOT_SOURCES=("${ENV_FILE}") +#fi +# +#{ +# for src in "${SNAPSHOT_SOURCES[@]}"; do +# echo "# ---- $(basename "${src}") ----" +# sed -E 's/^([A-Za-z_][A-Za-z0-9_]*)=""(\s*(#.*)?)$/#\1=""\2/' "${src}" +# echo "" +# done +#} > "${SNAPSHOT_FILE}" +# +#chmod 444 "${SNAPSHOT_FILE}" +#echo "Snapshot saved (read-only): ${SNAPSHOT_FILE}" \ No newline at end of file diff --git a/docker-makefiles/clean_configs.sh b/docker-makefiles/clean_configs.sh new file mode 100644 index 00000000..7f569693 --- /dev/null +++ b/docker-makefiles/clean_configs.sh @@ -0,0 +1,60 @@ +#!/usr/bin/env bash +#set -euo pipefail + +# -------- Helpers -------- +die() { + echo "Error: $1" >&2 + exit "${2:-1}" +} + +sedi() { + local expr="$1" + shift + + if [ "$#" -eq 0 ]; then + echo "sedi: no files provided" >&2 + return 1 + fi + + if sed --version >/dev/null 2>&1; then + sed -i "$expr" "$@" + else + sed -i '' "$expr" "$@" + fi +} + +# -------- Args -------- +NODE_CONFIGS=${1:-anylog-generic} + +# -------- Locate Config Files -------- +if [[ -f "docker-makefiles/${NODE_CONFIGS}/.env" ]] && \ + [[ -f "docker-makefiles/${NODE_CONFIGS}/base_configs.env" ]] && \ + [[ -f "docker-makefiles/${NODE_CONFIGS}/advance_configs.env" ]]; then + CONFIG_FILES=( + "docker-makefiles/${NODE_CONFIGS}/.env" + "docker-makefiles/${NODE_CONFIGS}/base_configs.env" + "docker-makefiles/${NODE_CONFIGS}/advance_configs.env" + ) +elif [[ -f "docker-makefiles/${NODE_CONFIGS}/node_configs.env" ]]; then + CONFIG_FILES=("docker-makefiles/${NODE_CONFIGS}/node_configs.env") +else + die "Missing configuration file(s) for '${NODE_CONFIGS}', cannot continue" +fi + +# -------- Generate Read-Only Snapshot Copy -------- +# Filename: {formatted_node_name}.env (hyphens -> underscores, lowercased) +# Empty-value vars ( VAR="" ) are commented out in the copy. +FORMATTED_NODE_NAME=$(echo "${NODE_CONFIGS}" | tr '[:upper:]' '[:lower:]' | tr '-' '_') +SNAPSHOT_DIR="docker-makefiles/${NODE_CONFIGS}" +SNAPSHOT_FILE="${SNAPSHOT_DIR}/formatted_node_configs.env" + +# -------- Generate Docker Compose File -------- +DOCKER_DIR="docker-makefiles/docker-compose-files" +DOCKER_FILE="${DOCKER_DIR}/${NODE_CONFIGS}-docker-compose.yaml" + +# -------- Remove files -------- +rm -rf ${DOCKER_FILE} "${SNAPSHOT_FILE}" + +if [[ -d "${DOCKER_DIR}" && -z "$(ls -A "${DOCKER_DIR}")" ]]; then + rm -rf "${DOCKER_DIR}" +fi \ No newline at end of file diff --git a/docker-makefiles/docker-compose-template-base.yaml b/docker-makefiles/docker-compose-template-base.yaml index fb45bb0a..ed31a27d 100644 --- a/docker-makefiles/docker-compose-template-base.yaml +++ b/docker-makefiles/docker-compose-template-base.yaml @@ -20,7 +20,7 @@ services: - ${NODE_NAME}-local-scripts:/app/deployment-scripts ${NODE_NAME}: image: ${IMAGE}:${TAG} - user: "anylog:anylog" + user: root pid: "host" group_add: - ${DOCKER_GID} diff --git a/docker-makefiles/prep_configs.sh b/docker-makefiles/prep_configs.sh index f1c230ac..cef76c99 100644 --- a/docker-makefiles/prep_configs.sh +++ b/docker-makefiles/prep_configs.sh @@ -68,30 +68,43 @@ fi if [[ ! -f "${BASE_ENV}" ]]; then echo "Failed to locate file: ${BASE_ENV} — skipping LICENSE_KEY update" else - # If LICENSE_KEY is set in the environment (passed from Makefile), write it - # into the config file so docker compose env_file picks it up too. - if [[ -n "${LICENSE_KEY:-}" ]]; then - if grep -q '^LICENSE_KEY=' "${BASE_ENV}"; then - sedi "s|^LICENSE_KEY=.*|LICENSE_KEY=${LICENSE_KEY}|" "${BASE_ENV}" - echo "LICENSE_KEY written to ${BASE_ENV} from environment" - else - echo "LICENSE_KEY=${LICENSE_KEY}" >> "${BASE_ENV}" - echo "LICENSE_KEY appended to ${BASE_ENV} from environment" - fi + # Extract raw value (everything after LICENSE_KEY=) + CURRENT_LICENSE_KEY=$(sed -n 's/^LICENSE_KEY=//p' "${BASE_ENV}") + + # Strip trailing carriage return and any surrounding quotes + CURRENT_LICENSE_KEY="${CURRENT_LICENSE_KEY%$'\r'}" + CURRENT_LICENSE_KEY="${CURRENT_LICENSE_KEY#\"}" ; CURRENT_LICENSE_KEY="${CURRENT_LICENSE_KEY%\"}" + CURRENT_LICENSE_KEY="${CURRENT_LICENSE_KEY#\'}" ; CURRENT_LICENSE_KEY="${CURRENT_LICENSE_KEY%\'}" + + if [[ -z "${CURRENT_LICENSE_KEY}" ]]; then + echo "No LICENSE_KEY found in ${BASE_ENV} — skipping" else - # Fall back: normalize whatever is already in the file - CURRENT_LICENSE_KEY=$(sed -n 's/^LICENSE_KEY=//p' "${BASE_ENV}") - CURRENT_LICENSE_KEY="${CURRENT_LICENSE_KEY%$'\r'}" - CURRENT_LICENSE_KEY="${CURRENT_LICENSE_KEY#\"}" ; CURRENT_LICENSE_KEY="${CURRENT_LICENSE_KEY%\"}" - CURRENT_LICENSE_KEY="${CURRENT_LICENSE_KEY#\'}" ; CURRENT_LICENSE_KEY="${CURRENT_LICENSE_KEY%\'}" - if [[ -z "${CURRENT_LICENSE_KEY}" ]]; then - echo "No LICENSE_KEY in environment or ${BASE_ENV} — skipping" - else - UPDATED_LICENSE_KEY="${CURRENT_LICENSE_KEY//\'/\"}" - sedi "s|^LICENSE_KEY=.*|LICENSE_KEY=${UPDATED_LICENSE_KEY}|" "${BASE_ENV}" - echo "LICENSE_KEY updated in ${BASE_ENV}" - fi + # Replace any internal single quotes with double quotes + UPDATED_LICENSE_KEY="${CURRENT_LICENSE_KEY//\'/\"}" + sedi "s|^LICENSE_KEY=.*|LICENSE_KEY=${UPDATED_LICENSE_KEY}|" "${BASE_ENV}" + echo "LICENSE_KEY updated in ${BASE_ENV}" fi fi -echo "Config update complete." +# -------- Step 3: Generate Read-Only Snapshot Copy -------- +# Filename: {formatted_node_name}.env (hyphens -> underscores, lowercased) +# Empty-value vars ( VAR="" ) are commented out in the copy. +FORMATTED_NODE_NAME=$(echo "${NODE_CONFIGS}" | tr '[:upper:]' '[:lower:]' | tr '-' '_') +SNAPSHOT_DIR="docker-makefiles/${NODE_CONFIGS}" +SNAPSHOT_FILE="${SNAPSHOT_DIR}/formatted_node_configs.env" + +echo "Generating read-only snapshot: ${SNAPSHOT_FILE}" + +# Concatenate all config files, comment out empty-value lines, write snapshot +{ + for cfg in "${CONFIG_FILES[@]}"; do + echo "# ---- $(basename "${cfg}") ----" + sed -E 's/^([A-Za-z_][A-Za-z0-9_]*)=""(\s*(#.*)?)$/#\1=""\2/' "${cfg}" + echo "" + done +} > "${SNAPSHOT_FILE}" + +chmod 444 "${SNAPSHOT_FILE}" +#echo "Snapshot saved (read-only): ${SNAPSHOT_FILE}" + +echo "Config update complete." \ No newline at end of file diff --git a/docker-makefiles/qualcomm/node_configs.env b/docker-makefiles/qualcomm/node_configs.env new file mode 100644 index 00000000..74aa88e5 --- /dev/null +++ b/docker-makefiles/qualcomm/node_configs.env @@ -0,0 +1,295 @@ +#==== .env ==== +# Basic deployment settings + +# Container image, deployment type +# -> prod: run AnyLog instance +# -> bash: run shell interface with AnyLog installed and params set +INIT_TYPE=prod + +# AnyLog image +IMAGE=anylogco/anylog-network + +# Github repo to use for configuring / deploying node +# The path can either be a Github URL and then the code will automatically clone +# the repo if needed OR a local path for deployment-scripts and then use that instead of cloning into the repositroy +DEPLOYMENTS_REPO=https://github.com/AnyLog-co/deployment-scripts +#DEPLOYMENTS_REPO=/anylog1/deployment-scripts +# Branch associated with git repo +DEPLOYMENTS_BRANCH="pre-develop" + +# specify the docker socket - to be used when deploying docker monitoring +DOCKER_SOCKET="/var/run/docker.sock" + +#==== basic ==== +# Core node configuration - required for all deployments +#--- General --- +# Configuration based deploymnet node deployment type +# Node Types: +# generic: sandbox with only network configured +# master: "blockchain" emulator +# operator: agent dedicated to storing data +# query: agent dedicated to querying data (enable system_query logical database) +# publisher: agent used to distribute data +# master-operator: a combination of master and operator on a single agent +# master-publisher: a combination of master and publisher on a single agent +NODE_TYPE=master-operator + +# Name of the AnyLog instance +NODE_NAME=anylog-standalone-operator +# Owner of the AnyLog instance +COMPANY_NAME=AnyLog Co. + +#--- Networking --- +# Port address used by AnyLog's TCP protocol to communicate with other nodes in the network +ANYLOG_SERVER_PORT=32148 +# Port address used by AnyLog's REST protocol +ANYLOG_REST_PORT=32149 +# Port value to be used as an MQTT broker, or some other third-party broker +ANYLOG_BROKER_PORT=32150 + +#--- Database --- +# Physical database type (sqlite, psql) +DB_TYPE=sqlite +# Database IP address +DB_IP=127.0.0.1 +# Database port number +DB_PORT=5432 +# Whether to set autocommit data +AUTOCOMMIT=true + +#--- NoSQL --- +# Enable external blob storage +BLOBS_STORAGE=false +# Storage backend type - Supported: mongo, s3, akave, minio +BLOB_STORAGE_TYPE=mongo +# Storage endpoint (MongoDB or object storage endpoint) +BLOB_STORAGE_IP=127.0.0.1 +# Storage port (used mainly for MongoDB) +BLOB_STORAGE_PORT=27017 + +#--- System Query --- +# Whether to start system_query logical database +SYSTEM_QUERY=true +# Database type used for system_query logical database +SYSTEM_QUERY_DB=sqlite +# Run system_query using in-memory SQLite. If set to false, will use pre-set database type +MEMORY=true + +#--- Blockchain --- +# TCP connection information for Master Node +LEDGER_CONN=127.0.0.1:32148 +# How often to sync from blockchain +BLOCKCHAIN_SYNC=60 second +# Source of where the blockchain data is coming from (master, optimism) +BLOCKCHAIN_SOURCE=master +# Where the local copy of the blockchain will be stored +BLOCKCHAIN_DESTINATION=file + +#--- Operator --- +# Owner of the cluster +CLUSTER_NAME=anylog-cluster1 +# Logical database name +DEFAULT_DBMS=new_company + +#==== southbound ==== +# Data ingestion protocols (MQTT, OPC-UA, video streaming, monitoring) +#--- Monitoring --- +# Whether to monitor the node or not +NODE_MONITORING=false +# Store monitoring in Operator node(s) +STORE_MONITORING=false +# Accept syslog data from local (Message broker required) +SYSLOG_MONITORING=false +# Docker Monitoring +DOCKER_MONITORING=false + +#--- Video Streaming --- +# Enable or disable video streaming. +# true = stream video, false = do not stream +ENABLE_VIDEO_STREAMING=false +# URL of the video to stream. Can be a YouTube link or other video source. +VIDEO_URL=https://www.youtube.com/watch?v=rnXIjl_Rzy4 +# Port on which the video streaming server will run. +VIDEO_PORT=32800 +# Name of the video stream or service identifier. +VIDEO_NAME=sfstream +# Enable or disable object detections (like YOLO detections) on the video. +# true = enable detections, false = disable +ENABLE_DETECTIONS=false +# Port on which the YOLO model server will run for object detection. +YOLO_MODEL_PORT=50051 + +#--- MQTT --- +# Whether to enable the default MQTT process +ENABLE_MQTT=false + +# IP address of MQTT broker +MQTT_BROKER=172.104.228.251 +# Port associated with MQTT broker +MQTT_PORT=1883 +# Whether to enable MQTT logging process +MQTT_LOG=false + +# Topic to get data for +MSG_TOPIC=rand-data +# Logical database name +MSG_DBMS=mydb +# Table where to store data +MSG_TABLE=bring [table] +# Timestamp column name +MSG_TIMESTAMP_COLUMN=bring [timestamp] +# Value column name +MSG_VALUE_COLUMN=bring [value] +# Column value type +MSG_VALUE_COLUMN_TYPE=float + +#--- OPC-UA --- +# Whether or not to enable to OPC-UA service +ENABLE_OPCUA=true +# OPC-UA URL address (ex. opcua.tcp:;//127.0.0.1:4840) +OPCUA_URL=opc.tcp://virtualfactory.proveit.services:4842/discovery +# Node information the root is located in (ex. ns=2;s=DataSet) +OPCUA_NODE="ns=2;s=Site1" +# How often to pull data from OPC-UA +OPCUA_FREQUENCY=5 + +#==== advanced ==== +# Fine-tuning for experienced users - safe to leave as defaults +#--- Directories --- +# AnyLog Root Path +ANYLOG_PATH=/app +# !local_scripts: ${ANYLOG_HOME}/deployment-scripts/scripts +LOCAL_SCRIPTS=/app/deployment-scripts +# !test_dir: ${ANYLOG_HOME}/deployment-scripts/test +TEST_DIR=/app/deployment-scripts/test + +#--- General --- +# hide blockchain policy - supports all node types except cluster / operator +IS_HIDDEN=false +# Whether to automatically run a local (or personalized) script at the end of the process +DEPLOY_LOCAL_SCRIPT=true +# Coordinates of the machine - used by Grafana to map the network +LOCATION="" +# Country where machine is located +COUNTRY="" +# State where machine is located +STATE="" +# City where machine is located +CITY="" + +#--- Networking --- +# Declare Policy name +CONFIG_NAME="" +# Whether to enable DNS resolution +ENABLE_DNS=false +# NIC type used to resolve the internal IP address (ex. eth0, lo) +NIC_TYPE="" +# Overlay IP address - if set, will replace local IP address when connecting to network +OVERLAY_IP="" +# External DNS resolution +EXTERNAL_DNS="" +# local DNS domain to append to hostname (ex. example.com) +DNS_DOMAIN="" +# Full path for machine's (local) DNS +DNS="" + +# Bind settings - a true value binds to a specific IP and Port (a false value binds to all IPs) +TCP_BIND=true +REST_BIND=false +BROKER_BIND=false + +# Number of concurrent threads per connection protocol +TCP_THREADS=6 +REST_THREADS=6 +BROKER_THREADS=6 + +# Timeout in seconds for REST requests (0 = continuous) +REST_TIMEOUT=30 + +#--- NoSQL --- +# Reuse identical blobs using file hash +BLOBS_REUSE=true +# Compress blobs before storage +BLOBS_COMPRESS=true + +# Folder for storing blobs locally +# Leave empty to use default application path +BLOBS_FOLDER="" + +#--- Object Store --- +# Logical bucket grouping +BUCKET_GROUP="" +# Bucket identifier +BUCKET_ID="" +# Cloud region +BUCKET_REGION="" + +#--- Operator --- +# Operator ID - autogenerated INT if not specified +# This is used to help keep a record of where data was originally stored +MEMBER="" +# Whether of not to enable HA against the cluster +ENABLE_HA=false +# How many days back to sync between nodes +START_DATE=30 +# How many threads to use in the operator process +OPERATOR_THREADS=3 + +#--- Sub processes --- +# Compress JSON and SQL file(s) backup +COMPRESS_FILE=true +# When data comes in write to database immediately, as opposed to waiting for a full buffer +WRITE_IMMEDIATE=true +# If buffer is not full, how long to wait until pushing data through +THRESHOLD_TIME=60 seconds +# Buffer size to reach, at which point data is pushed through +THRESHOLD_VOLUME=10KB +# Number of parallel queries +QUERY_POOL=6 + +#==== secrets ==== +# Sensitive credentials - do not commit to version control +# will be updated to use sercets logic for both docker and k8s +#--- General --- +# AnyLog License Key +LICENSE_KEY=cd554412f2a4ff540e3426a4f1fcb6d50ab8d9e1130e42957b0338ea45f4303fd160a551c5200b0321101abdc9eb44169e1b1c9d5a9492b16cc870619bb5199ae806a1dfadda93e00f02d30f3a5dabc0a6a7bb8f869e37fa565962eba55cb335397a66903d8fb3014ae1e4eea46c104cad8c787a0d5ae45efbf8728cf04cb805{"company":"Guest","expiration":"2026-08-01","type":"beta"} + +#--- Database --- +# Username for SQL database connection +DB_USER=admin +# Password correlated to database user +DB_PASSWD=passwd + +#--- NoSQL --- +# Optional DB authentication +BLOB_STORAGE_USER=admin +BLOB_STORAGE_PASSWORD=passwd + +#--- MQTT --- +# User associated with MQTT broker +MQTT_USER=anyloguser +# Password associated with MQTT user +MQTT_PASSWD=mqtt4AnyLog! + +#--- Object Store --- +# Access credentials +BUCKET_ACCESS_KEY="" +BUCKET_SECRET_KEY="" + +#==== remote-gui ==== +# Companion UI application - deployed alongside the AnyLog node +# Whether to enable Remote-GUI in deployment +ENABLE_REMOTE_GUI=false +# Remote-GUI version Tag +REMOTE_GUI_TAG=1.1.0 +# NIC type to use for Remote-GUI - if not set, uses 127.0.0.1 +REMOTE_GUI_NIC="" +# Remote GUI Frontend Port +REMOTE_GUI_FE=31800 +# Remote GUI Backend Port +REMOTE_GUI_BE=8080 +# Default connection to REST service (IP:Port) if not provided then autogenerated using `REMOTE_GUI_NIC` +REMOTE_CONN="" +# Grafana PATH to reference for Grafana connection +GRAFANA_URL="" + diff --git a/license-generator/license_key.sh b/license-generator/license_key.sh index e7e2fdbf..1b7f8660 100644 --- a/license-generator/license_key.sh +++ b/license-generator/license_key.sh @@ -9,6 +9,7 @@ # # Environment: # LICENSE_URL License server endpoint (default: http://23.239.12.151:8001/api/license-accept) +set -euo pipefail LICENSE_KEY="${1:-}" LICENSE_URL="${LICENSE_URL:-http://23.239.12.151:8001/api/license-accept}" @@ -85,6 +86,7 @@ echo " License Agreement Acceptance Required" echo "========================================" echo "" + # ── Prompt for license key if needed ────────────────────────────────── while true; do if [[ -z "${LICENSE_KEY}" ]]; then diff --git a/setup.cfg b/setup.cfg index 4f692a41..3da1afae 100644 --- a/setup.cfg +++ b/setup.cfg @@ -1,6 +1,6 @@ [metadata] name = docker-compose -version = 1.4.2604 (os-dev - d6daac [2026-05-13 03:05:52]) +version = 1.4.2604 (os-dev - e4a077 [2026-05-21 21:08:29]) author = AnyLog Co. contact = info@anylog.co description = docker-compose configurations for AnyLog deployment diff --git a/support/README.md b/support/README.md index 2b2e4877..5c7a3de0 100644 --- a/support/README.md +++ b/support/README.md @@ -8,6 +8,7 @@ Tooling to configure, generate, and manage the Docker/Podman containers that run * **[Ollama](Ollama.md)** — local LLM runtime for MCP function calling * **[Video Inference Models](https://github.com/AnyLog-co/AnyLog-Video-Inference-Models)** — CV/ML inference on edge video streams * **[Nebula Overlay Network](https://github.com/oshadmon/nebula-anylog)** — encrypted peer-to-peer overlay so AnyLog nodes on separate physical networks communicate as a single unified network +* **[Syslog Forwarding](#syslog-forwarding)** — configure the host syslog daemon to forward all log traffic to the AnyLog broker port ## Requirements @@ -31,6 +32,7 @@ Tooling to configure, generate, and manage the Docker/Podman containers that run - [Ollama](#ollama) - [Video Inference Models](#video-inference-models) - [Nebula Overlay](#nebula-overlay-network) + - [Syslog Forwarding](#syslog-forwarding) - [docker_compose_builder.sh](#docker_compose_buildersh) - [Makefile Reference](#makefile-reference) - [Targets](#targets) @@ -50,6 +52,7 @@ support/ ├── Video-Inferences.md # Video inference models guide ├── Nebula.md # Nebula overlay network setup guide ├── docker_compose_builder.sh # Generates docker-compose.yml from configs.yaml +├── syslog.sh # Configures host syslog daemon → AnyLog broker port ├── grafana/ │ ├── configs.yaml │ └── docker-compose.yml # generated — do not edit by hand @@ -193,6 +196,32 @@ Nebula creates an encrypted peer-to-peer mesh across physically separated machin → Full setup guide: [Nebula.md](Nebula.md) → Source repository: [oshadmon/nebula-anylog](https://github.com/oshadmon/nebula-anylog) +### Syslog Forwarding + +`syslog.sh` configures the host syslog daemon to forward all log traffic to the AnyLog broker port. It reads `SYSLOG_MONITORING` and `ANYLOG_BROKER_PORT` directly from a `node_configs.env` file, so no extra arguments are needed once the node config is set up. + +Both commands are **no-ops** when `SYSLOG_MONITORING != "true"` in the config. `setup` is idempotent — safe to call repeatedly. + +| Field | Value | +|---|---| +| Config keys | `SYSLOG_MONITORING` (must be `"true"`), `ANYLOG_BROKER_PORT` | +| Linux | rsyslog drop-in `/etc/rsyslog.d/60-custom-forwarding.conf` (TCP) | +| macOS | `/etc/syslog.conf` append (UDP) — ensure the broker port is open for UDP | + +```bash +# via deploy.sh +bash syslog.sh setup [NODE_CONFIGS] +bash syslog.sh remove [NODE_CONFIGS] + +# via make (SERVICE=syslog intercepts before docker-compose logic) +make setup SERVICE=syslog # prompts for config path +make setup SERVICE=syslog NODE_CONFIGS=../docker-makefiles/anylog-operator/node_configs.env +make remove SERVICE=syslog NODE_CONFIGS=../docker-makefiles/anylog-operator/node_configs.env +``` + +Default `NODE_CONFIGS` path when not supplied: `docker-makefiles/anylog-generic/node_configs.env`. + + --- ## `docker_compose_builder.sh` @@ -248,6 +277,8 @@ Any subdirectory containing a `configs.yaml` is automatically recognised as a se | `logs-f` | Follow container logs (`SERVICE=` required) | | `exec` | Attach to container shell/client (`SERVICE=` required) | | `list` | Print default and all auto-discovered services | +| `setup` | `SERVICE=syslog` — install host syslog → AnyLog forwarding rule | +| `remove` | `SERVICE=syslog` — remove host syslog → AnyLog forwarding rule | Omit `SERVICE` to act on all four default services (`remote-gui`, `grafana`, `postgres`, `mongodb`). @@ -304,6 +335,11 @@ make exec SERVICE=psql # psql -U postgres ← remove this make exec SERVICE=mongo # mongosh ← remove this make exec SERVICE=postgres-prod # psql -U postgres ← remove this +# Syslog forwarding +make setup SERVICE=syslog # prompts for node config path +make setup SERVICE=syslog NODE_CONFIGS=../docker-makefiles/anylog-operator/node_configs.env +make remove SERVICE=syslog NODE_CONFIGS=../docker-makefiles/anylog-operator/node_configs.env + # Discovery make list # show default + all discovered services ``` \ No newline at end of file