From e573c800c6bafcbabfc7deb74abd95155f13b981 Mon Sep 17 00:00:00 2001 From: Derek Date: Mon, 16 Nov 2026 13:19:29 +1100 Subject: [PATCH 1/2] docs(transform-vrl): correct docs and redraw both diagrams Align the README, docs/ and the filebeat bundle README with the code, scalo-rs 2.14.30 and dfe-infra. - Redraw the architecture diagram as an L3 of the service and the internals diagram as an L4 of the process to match the diagram standard. - `pipeline.name` sets no consumer group or metric label. - Source and sink transports are chosen separately. - The Push retry stops on a permanent failure. - Enrichment lookups use equality indexes on the key columns alone. - The library chart's `reload.*` values roll pods on a credential rotation. - Restore `source.commit_interval_ms` to the inert list. - Restore the scalo line in the README. - Label the filebeat bundle legacy in the README. - Trim config comments in Cargo.toml, .hyperi-ci.yaml, deny.toml and the Makefile to pointers and short reasons. No value changes. --- .hyperi-ci.yaml | 8 +- CONTRIBUTING.md | 30 ++--- Cargo.toml | 16 +-- Makefile | 6 +- README.md | 19 ++-- deny.toml | 4 +- docs/architecture.md | 207 +++++++++++------------------------ docs/internals.md | 136 ++++++++++------------- pipelines/filebeat/README.md | 59 ++++------ 9 files changed, 168 insertions(+), 317 deletions(-) diff --git a/.hyperi-ci.yaml b/.hyperi-ci.yaml index d7eb32e..8ab0b24 100644 --- a/.hyperi-ci.yaml +++ b/.hyperi-ci.yaml @@ -8,23 +8,19 @@ classification: product language: rust -# Quality checks quality: enabled: true # Advisory ignores fed to cargo-audit and osv-scanner. ignore: - # Unreachable, accepted in dfe-infra docs/THREAT-MODEL.md. scalo-rs#281 tracks the move. - tool: osv-scanner id: GHSA-w9wp-h8wv-79jx reason: "CVE-2026-48504, opentelemetry_sdk 0.31.0 via scalo: unreachable, no Baggage propagator registered in dfe-transform-vrl or scalo. Accepted risk per dfe-infra docs/THREAT-MODEL.md. Drop when hyperi-io/scalo-rs#281 moves scalo off opentelemetry 0.31, blocked on metrics-exporter-opentelemetry 0.2.1 requiring opentelemetry ^0.31." - # unsafe-libyaml arrives through the vrl crate's serde_yaml_ng, a normal and a build dependency of it; revisit when vrl moves off serde_yaml_ng. - tool: osv-scanner id: RUSTSEC-2024-0449 reason: "unsafe-libyaml is unmaintained, with no known CVE (RUSTSEC-2023-0075 was fixed in 0.2.10). Only operator-written VRL reaches it on event data: the stdlib's parse_yaml hands libyaml what the event carries, and ingested data cannot add that call. Config YAML parses through scalo's serde-saphyr. Drop when the vrl crate moves off serde_yaml_ng." rust: features: default -# Testing test: enabled: true coverage: true @@ -37,7 +33,6 @@ test: full: required_for_release: true -# Building build: enabled: true strategies: @@ -57,8 +52,7 @@ build: - x86_64-unknown-linux-gnu - aarch64-unknown-linux-gnu -# Release (where the artefacts go). The thin chart is assembled from the -# emitted contract and ships beside the image. +# The thin chart: docs/internals.md, Helm chart. release: enabled: true # Keep `library` at the scalo version in Cargo.toml: the library renders only the contract version its scalo release writes. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 9761c41..b2bfd7e 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -137,16 +137,7 @@ If you have questions about contributing, please open an issue or contact us. ### What this is -A single Rust binary that runs VRL transforms over records in the DFE data path --- Kafka in and out on the `bus` transport, gRPC in and out on the `direct` -transport. Two boundaries get assumed wrongly. First, it is VRL only: a pipeline -needing `lua`, `aggregate`, `dedupe`, `throttle` or `sample` belongs in -dfe-transform-vector, which keeps the Vector subprocess and pays for it. Second, -this crate does not own its own event loop. scalo's `BatchEngine::pipeline` -drives `recv -> process -> send -> release`, and this crate supplies the `process` -closure, the produce sink, the config, the VRL compiler and the enrichment -registry. Reading `src/pipeline.rs` expecting to find the loop is the usual wrong -turn. +A single Rust binary that runs VRL transforms over records in the DFE data path -- Kafka in and out on the `bus` transport, gRPC in and out on the `direct` transport. Two boundaries get assumed wrongly. First, it runs VRL and nothing else: a pipeline needing `lua`, `aggregate`, `dedupe`, `throttle` or `sample` belongs in dfe-transform-vector, which keeps the Vector subprocess and pays for it. Second, this crate does not run its event loop. scalo's `BatchEngine::pipeline` drives `recv -> process -> send -> release`, and this crate supplies the `process` closure, the produce sink, the config, the VRL compiler and the enrichment registry. Reading `src/pipeline.rs` expecting to find the loop is the usual wrong turn. ### Where things live @@ -176,27 +167,24 @@ cargo run --bin dfe-transform-vrl -- emit-dockerfile > Dockerfile # regenerat Green lies here in three ways, and all three are on by default. -`default = []` in `Cargo.toml`, so a bare `cargo nextest run` does not compile the MMDB or SQLite enrichment tests in at all. `.hyperi-ci.yaml` adds `enrichment-mmdb` and `enrichment-sqlite` to both the test and the build feature sets -- the build too, because the Dockerfile only copies the binary, so a default build ships a container that rejects `type: mmdb` and `type: sqlite` tables at startup. +`default = []` in `Cargo.toml`, so a bare `cargo nextest run` does not compile the MMDB or SQLite enrichment tests in at all. `.hyperi-ci.yaml` adds `enrichment-mmdb` and `enrichment-sqlite` to both the test and the build feature sets -- the build as well, because the Dockerfile copies the binary and nothing else, so a default build ships a container that rejects `type: mmdb` and `type: sqlite` tables at startup. -The core-tier e2e tests run on broker containers scalo's test support starts. Without Docker they skip locally and fail only under `CI`, so a local run without Docker is green without having reached a broker. +The core-tier e2e tests run on broker containers scalo's test support starts. Without Docker they skip locally and fail under `CI` alone, so a local run without Docker passes without having reached a broker. -The tests that may reach a broker somebody else owns are `#[ignore]` and run only under `--run-ignored all` or `only`. They take a live broker first when `KAFKA_BROKERS` answers. `TEST_MODE=docker` points the harness at `localhost:19092` with no auth, the port dfe-docker publishes its broker on, and otherwise it reads `KAFKA_BROKERS`, `KAFKA_SECURITY_PROTOCOL` and the `KAFKA_SASL_*` names from the environment or the repo's `.env`. +The tests that may reach a broker somebody else owns are `#[ignore]`. nextest runs them under `--run-ignored all` or `only`, and `cargo test` under `-- --ignored`. They take a live broker first when `KAFKA_BROKERS` answers. `TEST_MODE=docker` points the harness at `localhost:19092` with no auth, the port dfe-docker publishes its broker on, and otherwise it reads `KAFKA_BROKERS`, `KAFKA_SECURITY_PROTOCOL` and the `KAFKA_SASL_*` names from the environment or the repo's `.env`. -One more: the push trigger in `.github/workflows/ci.yml` carries -`paths-ignore: docs/**, **.md`, so a docs-only push runs no jobs. The -`pull_request` trigger has no such filter, so the PR is where a docs change gets -checked. +One more: the push trigger in `.github/workflows/ci.yml` has `paths-ignore: docs/**, **.md`, so a docs-only push runs no jobs. The `pull_request` trigger has no such filter, so the PR is where a docs change gets checked. ### What tends to bite | Don't | Do | Why | |---|---|---| | Hand-edit `Dockerfile`, or commit a chart | Fix `src/deployment.rs::contract()` and regenerate | The Dockerfile is generator output, and `checked_in_dockerfile_matches_emit_dockerfile` fails on a hand edit. The release assembles the chart from the contract, so a committed chart never ships, and `test_every_contract_secret_env_var_reaches_the_config` holds every declared name to the field it spells | -| Bump scalo and leave `release.helm.library` behind | Move `release.helm.library` in `.hyperi-ci.yaml` to the same scalo version | A scalo-service release renders only the contract version its scalo release writes | -| Turn the Kafka lag trigger back on in `contract()` | Leave `KafkaLagTrigger::disabled()`, so the contract scales on CPU and a deployment adds the scaling-pressure trigger | Lag rises when a downstream stage breaks, so a lag trigger adds pods that wait on the same broken stage. The lag trigger also reads `config.kafka.*`, which this app's values do not have | -| Put a scalo section (`metrics`, `logger`, `memory`, `scaling`, `worker_pool`, `batch_processing`, `self_regulation`, `version_check`, `otel_tracing`) in a `config.yaml` the binary finds in its working directory | Pass the file with `--config`, or set the section through the env layer on a **double** underscore | scalo reads the `--config` file as its settings layer but finds other files only by fixed base name (`settings.yaml`, `defaults.yaml`), so a working-directory `config.yaml` reaches the wrapper alone. Full list under [Environment variable overrides](docs/internals.md#environment-variable-overrides) | +| Bump scalo and leave `release.helm.library` behind | Move `release.helm.library` in `.hyperi-ci.yaml` to the same scalo version | A scalo-service release renders the contract version its scalo release writes and no other | +| Turn the Kafka lag trigger back on in `contract()` | Leave `KafkaLagTrigger::disabled()`, so the contract scales on CPU and a deployment adds the scaling-pressure trigger | Lag rises when a downstream stage breaks, so a lag trigger adds pods that wait on the same broken stage. The lag trigger reads `config.kafka.*` as well, which this app's values do not have | +| Put a scalo section (`metrics`, `logger`, `memory`, `scaling`, `worker_pool`, `batch_processing`, `self_regulation`, `version_check`, `otel_tracing`) in a `config.yaml` the binary finds in its working directory | Pass the file with `--config`, or set the section through the env layer on a **double** underscore | scalo reads the `--config` file as its settings layer but finds other files by fixed base name alone (`settings.yaml`, `defaults.yaml`), so a working-directory `config.yaml` reaches the wrapper alone. Full list under [Operating notes](docs/internals.md#operating-notes) | | Trust `pipeline.batch_size`, `pipeline.batch_timeout_ms`, `sink.key_field` or `source.commit_interval_ms` | Size a chunk with `batch_processing.max_chunk_size` | `config::INERT_SETTINGS` -- accepted, validated, reaching nothing. Table under [Reload](docs/internals.md#reload) | -| Call a bare `cargo nextest run` green | Pass `--features enrichment-mmdb,enrichment-sqlite` | `default = []`, so the MMDB and SQLite tests are not compiled in and the run is green without having tested them | +| Call a bare `cargo nextest run` a pass | Pass `--features enrichment-mmdb,enrichment-sqlite` | `default = []`, so the MMDB and SQLite tests are not compiled in and the run passes without having tested them | | Set `sasl.enabled` with an empty username or password | Supply both, or neither | librdkafka's SCRAM check is a NULL check that an empty string passes, so the pod would authenticate against nothing and still report Ready. `validate` refuses it at startup | | Add a serialise path that redacts by field name | Keep the password a `scalo::SensitiveString` | Redaction is by type on every path -- `Debug`, the `config-check` dump, the emitted schema. The figment round-trip has to be wrapped in `expose_during` or a file-sourced password reaches the broker as the literal `***REDACTED***` | | Size the container from steady state when the program is large | Leave headroom for the compile | Compilation scales with the VRL, and below the floor `engine::budget` computes the kernel kills the process mid-compile, seen as an exit-137 restart loop with nothing in the log. `engine::budget` refuses first and names all three numbers | diff --git a/Cargo.toml b/Cargo.toml index 49a9b2b..4d9742f 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -33,7 +33,6 @@ path = "src/bin/pgo_driver.rs" required-features = ["pgo-driver"] [dependencies] -# Async runtime tokio = { version = ">=1.54.1, <2", features = ["full"] } # CancellationToken, scalo's shutdown token type. @@ -74,18 +73,15 @@ scalo = { version = ">=2.14.30, <3", features = [ # Config-schema derivation. Same range as scalo so cargo unifies to one schemars. schemars = { version = ">=1.2.3, <2", default-features = false, features = ["std", "derive", "preserve_order"] } -# VRL — Vector Remap Language (embedded transform engine). Built without its -# env and network functions, so no transform can read the host environment or -# reach the network. `enable_system_functions` stays for `parse_etld`, which the -# filebeat program calls, and the compiler drops its `get_hostname`. +# No env or network functions (docs/internals.md, VRL integration). `enable_system_functions` +# stays for `parse_etld`, and the compiler drops its `get_hostname`. vrl = { version = ">=0.35.0, <0.36", default-features = false, features = [ "compiler", "stdlib-base", "enable_crypto_functions", "enable_system_functions", ] } -# Only for the `TimeZone::Named(Tz::UTC)` constant in the VRL runner — -# same version range as vrl so cargo unifies to one chrono-tz. +# For the `TimeZone::Named(Tz::UTC)` constant in the VRL runner. Same range as vrl so cargo unifies to one chrono-tz. chrono-tz = ">=0.10.4, <0.11" # `Value::Timestamp` is a `chrono::DateTime`. Same range as vrl so cargo unifies to one chrono. chrono = ">=0.4.45, <0.5" @@ -93,30 +89,24 @@ chrono = ">=0.4.45, <0.5" # CLI. No `env` feature: scalo's cascade binds env vars. clap = { version = ">=4.6.7, <5", features = ["derive"] } -# Configuration figment = { version = ">=0.10.19, <0.11", features = ["env"] } # Metrics types (Counter, Gauge, Histogram) returned by scalo MetricsManager metrics = ">=0.24.6, <0.25" -# Serialisation serde = { version = ">=1.0.229, <2", features = ["derive"] } serde_json = ">=1.0.154, <2" sonic-rs = ">=0.5.10, <0.6" -# Error handling thiserror = ">=2.0.21, <3" anyhow = ">=1.0.104, <2" -# Logging and tracing tracing = ">=0.1.44, <0.2" -# Enrichment engine — high-performance lookups rustc-hash = ">=2.1.3, <3" arc-swap = ">=1.9.3, <2" csv = ">=1.4.0, <2" -# Optional enrichment sources maxminddb = { version = ">=0.32.0, <0.33", optional = true } rusqlite = { version = ">=0.41.1, <0.42", features = ["bundled"], optional = true } diff --git a/Makefile b/Makefile index 670c1f2..fc50db1 100644 --- a/Makefile +++ b/Makefile @@ -10,13 +10,13 @@ check: ## Pre-push validation (quality + test) hyperi-ci check -quality: ## Run all quality checks +quality: hyperi-ci run quality -test: ## Run tests +test: hyperi-ci run test -build: ## Build artifacts +build: hyperi-ci run build ci: quality test build ## Run full CI pipeline locally diff --git a/README.md b/README.md index 2e81d1b..d57b448 100644 --- a/README.md +++ b/README.md @@ -9,15 +9,16 @@ Runs your VRL transforms over a DFE deployment's event stream, in-process. - Sits between the land topic and dfe-loader, and runs the VRL (Vector Remap Language) files in a directory, in filename order, against each event. The VRL engine is compiled in, so a transform is a file you change, not a release. - Kafka in and out, or a direct gRPC push on a deployment with no broker. The transform you write is the same on both. -- At-least-once by default: the source is acknowledged only once the sink has the transformed records. `source.acknowledgements.enabled: false` acknowledges at receipt instead. -- Enrichment tables from CSV, JSON, YAML and STIX, read from VRL, plus MMDB and SQLite with the `enrichment-mmdb` and `enrichment-sqlite` build features. A table with a `refresh` setting reloads while the service runs. MMDB loads once at start. +- At-least-once by default: the source is acknowledged after the sink has the transformed records. `source.acknowledgements.enabled: false` acknowledges at receipt instead. +- Enrichment tables from CSV, JSON, YAML and STIX, read from VRL, plus MMDB and SQLite with the `enrichment-mmdb` and `enrichment-sqlite` build features. A table with a `refresh` setting reloads as the service runs. MMDB loads once at start. - The host and network functions (`get_env_var`, `get_hostname`, `http_request`, `dns_lookup`, `reverse_dns`) do not compile, so a transform cannot read the host's environment or reach the network. +- Built on [scalo](https://github.com/hyperi-io/scalo-rs), so config, logs, metrics, health probes and scaling signals work the way they do in every DFE Rust service. -JSON is the only payload format. A record that is not JSON is dropped, counted in `records_error_total{stage="deserialise"}`, and not redelivered. For Vector's own transforms beyond `remap`, use [dfe-transform-vector](https://github.com/hyperi-io/dfe-transform-vector). +JSON is the one payload format. A record that is not JSON is dropped, counted in `records_error_total{stage="deserialise"}`, and not redelivered. For Vector's transforms beyond `remap`, use [dfe-transform-vector](https://github.com/hyperi-io/dfe-transform-vector). ## Quick start -On Linux the build needs `protoc`, a C toolchain, `pkg-config` and librdkafka. scalo compiles its gRPC `.proto` files, `aws-lc-sys` compiles C, and `rdkafka-sys` links librdkafka through pkg-config. It refuses a librdkafka older than the version after the `+` in its `Cargo.lock` entry, and a Debian or Ubuntu package can be older than that, so there [build librdkafka from source](https://github.com/confluentinc/librdkafka#build-from-source) first. +On Linux the build needs `protoc`, a C toolchain, `pkg-config` and librdkafka. scalo compiles its gRPC `.proto` files, `aws-lc-sys` compiles C, and `rdkafka-sys` links librdkafka through pkg-config. It rejects a librdkafka older than the version after the `+` in its `Cargo.lock` entry, and a Debian or Ubuntu package can be older than that, so there [build librdkafka from source](https://github.com/confluentinc/librdkafka#build-from-source) first. ```bash sudo apt-get install -y protobuf-compiler build-essential pkg-config @@ -27,7 +28,7 @@ cp config.example.yaml config.yaml # uncomment source, sink and transforms, ./target/release/dfe-transform-vrl --config config.yaml ``` -`--config` goes before the subcommand, or `config-check` refuses it as an unexpected argument. +`--config` goes before the subcommand, or `config-check` rejects it as an unexpected argument. The smallest config names the source, a directory holding at least one `.vrl` file, and the sink: @@ -56,15 +57,15 @@ A transform file is plain VRL: del(.message) ``` -Health and metrics answer on port 9090: `/livez`, `/readyz` (503 while memory is under pressure or a transport reports itself unhealthy, and 200 once the pipeline runs, an idle one included) and `/metrics`. +Health and metrics answer on port 9090: `/livez`, `/readyz` (503 when memory is under pressure or a transport reports itself unhealthy, and 200 once the pipeline runs, an idle one included) and `/metrics`. Outside the image, set `dlq.file.path` to a writable directory. The default, `/var/spool/scalo/dlq`, is one the image creates, and without a writable one the service logs that the DLQ has no backend and holds any dead letter unacknowledged at its source. ## Configuration -[config.example.yaml](config.example.yaml) is the annotated reference, and [docs/config-schema.yaml](docs/config-schema.yaml) lists every key of the transform's own sections with its type and default. The big dials have flat environment overrides for Kubernetes, `DFE_TRANSFORM_VRL_SOURCE_BROKERS`, `DFE_TRANSFORM_VRL_SINK_TOPIC`, `DFE_TRANSFORM_VRL_TRANSFORMS_DIR` and the SASL pair among them. The whole list, Kafka over TLS with a private CA and the scalo sections that share the file on a double underscore are under [Operating notes](docs/internals.md#operating-notes) in docs/internals.md, and the settings that parse but reach nothing under [Reload](docs/internals.md#reload). +[config.example.yaml](config.example.yaml) is the annotated reference, and [docs/config-schema.yaml](docs/config-schema.yaml) lists every key of the transform's sections with its type and default. The big dials have flat environment overrides for Kubernetes, `DFE_TRANSFORM_VRL_SOURCE_BROKERS`, `DFE_TRANSFORM_VRL_SINK_TOPIC`, `DFE_TRANSFORM_VRL_TRANSFORMS_DIR` and the SASL pair among them. The whole list, Kafka over TLS with a private CA and the scalo sections that share the file on a double underscore are under [Operating notes](docs/internals.md#operating-notes) in docs/internals.md, and the settings that parse but reach nothing under [Reload](docs/internals.md#reload). -`pipelines/filebeat/` ships a pure-VRL port of the DFE 2.1 filebeat templates (Cisco Meraki, IOS, Umbrella) with its `timezones.csv` table. Opt in by pointing `transforms.dir` and `enrichment_tables` at it. dfe-transform-elastic carries the same sources as compiled Rust. +`pipelines/filebeat/` ships a legacy pure-VRL port of the DFE 2.1 filebeat templates (Cisco Meraki, IOS, Umbrella) with its `timezones.csv` table, kept for DFE/XDR 2.0 and 2.1 filebeat feeds. Opt in by pointing `transforms.dir` and `enrichment_tables` at it. Any other feed uses dfe-transform-elastic, which has the same sources as compiled Rust. ## Documentation @@ -77,7 +78,7 @@ The suite's published docs are at , and this reposit ## Help and contributing -How to work in this repo, the commands that prove a change and what tends to bite are in [CONTRIBUTING.md](CONTRIBUTING.md). A question that is not a bug goes in a new issue. +How to work in this repo, the commands that test a change and what tends to bite are in [CONTRIBUTING.md](CONTRIBUTING.md). A question that is not a bug goes in an issue. Open bugs: [this repo](https://github.com/hyperi-io/dfe-transform-vrl/issues?q=is%3Aopen+is%3Aissue+type%3ABug+-label%3Abacklog+-label%3Anext-minor) | [whole DFE suite](https://github.com/search?type=issues&q=is%3Aopen+is%3Aissue+type%3ABug+-label%3Abacklog+-label%3Anext-minor+repo%3Ahyperi-io%2Fdfe-infra+repo%3Ahyperi-io%2Fdfe-docker+repo%3Ahyperi-io%2Fdfe-deploy+repo%3Ahyperi-io%2Fdfe-engine+repo%3Ahyperi-io%2Fdfe-loader+repo%3Ahyperi-io%2Fdfe-receiver+repo%3Ahyperi-io%2Fdfe-fetcher+repo%3Ahyperi-io%2Fdfe-archiver+repo%3Ahyperi-io%2Fdfe-transform-vrl+repo%3Ahyperi-io%2Fdfe-transform-vector+repo%3Ahyperi-io%2Fdfe-transform-elastic+repo%3Ahyperi-io%2Fdfe-ui+repo%3Ahyperi-io%2Fdfe-hyperdx+repo%3Ahyperi-io%2Fscalo-rs+repo%3Ahyperi-io%2Fscalo-py+repo%3Ahyperi-io%2Flogreducer+repo%3Ahyperi-io%2Fdfe-schemas+repo%3Ahyperi-io%2Fvector-vrl+repo%3Ahyperi-io%2Fculvert+repo%3Ahyperi-io%2Fdfe-clickhouse-rs+repo%3Ahyperi-io%2Ffactbook-rs) diff --git a/deny.toml b/deny.toml index f57563f..9dc74cd 100644 --- a/deny.toml +++ b/deny.toml @@ -17,9 +17,7 @@ targets = [ [advisories] unmaintained = "all" db-urls = ["https://github.com/rustsec/advisory-db"] -# Each entry MUST have a comment with: (1) why the advisory is tolerated, -# (2) the upstream-blocker preventing remediation, (3) when to revisit. -# Empty list = nothing tolerated; CI fails on any new advisory. +# Each entry's comment: why it is tolerated, the upstream blocker, when to revisit. ignore = [ # unsafe-libyaml arrives through the vrl crate's serde_yaml_ng, a normal and a build dependency of it; revisit when vrl moves off serde_yaml_ng. { id = "RUSTSEC-2024-0449", reason = "unsafe-libyaml is unmaintained, with no known CVE (RUSTSEC-2023-0075 was fixed in 0.2.10). Only operator-written VRL reaches it on event data: the stdlib's parse_yaml hands libyaml what the event carries, and ingested data cannot add that call. Config YAML parses through scalo's serde-saphyr. Drop when the vrl crate moves off serde_yaml_ng." }, diff --git a/docs/architecture.md b/docs/architecture.md index f198593..09f0d96 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -1,152 +1,94 @@ # Architecture -## The problem the wrapper solves +Why dfe-transform-vrl is built this way and the invariants it keeps. The per-record path, memory budget and reload matrix are in [internals.md](internals.md). -Reshaping events with Vector costs memory DFE cannot afford at volume. +## The problem the wrapper solves -Vector has no configurable memory cap. Its internal buffers grow with -throughput, so a Kubernetes memory limit has to be sized for the worst case -Vector might ever reach rather than the work actually queued. +Vector costs memory DFE cannot afford at volume. -That cost buys access to Vector's non-VRL transforms, which most pipelines never -use. So the answer is to keep VRL and drop Vector. Compile the VRL crate into the -binary, and let the wrapper own the buffers and the wire format. +Vector has no configurable memory cap. Its internal buffers grow with throughput, so a Kubernetes memory limit has to be sized for the worst case Vector might reach rather than the work that is queued. -One Rust binary. It reads records, runs a compiled VRL program over each one, -and writes the results out. Kafka in and out on the `bus` transport, gRPC in and -out on the `direct` transport (`0.0.0.0:6000` listener, default sink endpoint -`http://dfe-loader:6000`). +That cost buys access to Vector's non-VRL transforms, which most pipelines never use. So the answer is to keep VRL and drop Vector. Compile the VRL crate into the binary, and let the wrapper hold the buffers and the wire format. -It is not a general transform host. VRL only. A pipeline needing a -Vector-native transform -- `lua`, `aggregate`, `dedupe`, `throttle`, `sample` -- -belongs in dfe-transform-vector, which keeps the subprocess and pays for it. +One Rust binary. It reads records, runs a compiled VRL program over each one, and writes the results out. Kafka in and out on the `bus` transport, gRPC in and out on the `direct` transport (`0.0.0.0:6000` listener, default sink endpoint `http://dfe-loader:6000`). -It is also not the owner of its own event loop, which is the part that surprises -readers. +It is not a general transform host: a pipeline needing a Vector-native transform -- `lua`, `aggregate`, `dedupe`, `throttle`, `sample` -- belongs in dfe-transform-vector, which keeps the subprocess and pays for it. ## Who owns the loop -The binary is a thin VRL-specific layer over scalo's data-plane runtime. scalo's -`BatchEngine::pipeline` drives `recv -> process -> send -> release`, including -the held source acknowledgement and self-regulation. This crate supplies only the -`process` closure and the produce sink, plus the config, the VRL compiler and the -enrichment registry. +scalo's `BatchEngine::pipeline` drives `recv -> process -> send -> release`, including the held source acknowledgement and self-regulation. This crate supplies the `process` closure and the produce sink, plus the config, the VRL compiler and the enrichment registry. + +This diagram shows the parts inside dfe-transform-vrl that a block of records passes through. ```mermaid -flowchart TB - subgraph SCALO["scalo runtime -- owns the loop"] - DRV["BatchEngine::pipeline
recv -> process -> send -> release"] - POOL["AdaptiveWorkerPool"] - GOV["SelfRegulationGovernor
inbound pause-partitions gate"] +flowchart LR + accTitle: L3 Components -- dfe-transform-vrl + accDescr: Records arrive from the stage in front. scalo's batch engine hands each block to the process closure, which runs the program the VRL compiler built, on scalo's worker pool. The produce sink sends the out-batch to the next stage. + + upstream["Stage in front"] + + subgraph vrl_app["dfe-transform-vrl"] + engine["scalo batch engine"] + compiler["VRL compiler"] + pool["scalo worker pool"] + process[["Process closure"]] + produce["Produce sink"] end - subgraph THIS["this crate -- supplies the work"] - PROC["process closure
deserialise -> VRL -> reserialise"] - SINK["produce sink"] - COMP["VRL compiler + enrichment registry"] - end - DRV -->|WorkBatch of Record| PROC - PROC -->|out-batch| SINK - SINK --> DRV - COMP -.->|compiled Program, once at startup| PROC - POOL -.->|parallel VRL execution| PROC - GOV -.->|pauses intake under memory pressure| DRV + + downstream["Next stage"] + + upstream -->|records| engine + pool -.-> process + compiler -.-> process + engine --> process + process --> produce + produce -->|out-batch| downstream + + classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A + classDef group fill:none,stroke:#64748B,stroke-width:1px,stroke-dasharray:2 4 + class vrl_app group ``` +*Figure 1. L3 Components -- dfe-transform-vrl. Dotted arrows bring what the closure runs with, not records.* Up: top of dfe-transform-vrl. Down: [L4 Detail -- Process closure](internals.md#data-flow). + +scalo holds the batch engine and the worker pool. This crate holds the compiler, the process closure and the produce sink. On the bus the stage in front and the next stage are Kafka topics. On the direct transport the stage in front pushes to this service's listener, and the next stage is another Push listener. scalo's self-regulation governor sits on the engine's intake. + Per block of records: -1. The loop receives a `WorkBatch` from the source. On the bus transport that - is the governed Kafka consumer: the inbound brake is attached, so intake - pauses under memory pressure by pausing partitions -- the member stays in - the consumer group and no rebalance fires. On the direct transport it is the - Push listener, which refuses pushes `UNAVAILABLE` under the same pressure. -2. `process` parses each record's JSON into a VRL `Value`, runs the compiled - program on the worker pool in parallel, and serialises the survivors back to - JSON. +1. The loop receives a `WorkBatch` from the source. On the bus transport that is the governed Kafka consumer: the inbound brake is attached, so intake pauses under memory pressure by pausing partitions -- the member stays in the consumer group and no rebalance fires. On the direct transport it is the Push listener, which rejects pushes `UNAVAILABLE` under the same pressure. +2. `process` parses each record's JSON into a VRL `Value`, runs the compiled program on the worker pool in parallel, and serialises the survivors back to JSON. 3. The loop sends the whole out-batch through `TransportSender::send_batch`. -4. The loop releases the block's source only after the send returns `Ok`: - Kafka commits the offsets, the Push listener answers its sender OK. +4. The loop releases the block's source after the send returns `Ok`: Kafka commits the offsets, the Push listener answers its sender OK. ## Invariants: the delivery contract -These are the rules a reader cannot recover by skimming the code, and each one -is why some part of the code looks the way it does. +**At-least-once is batch-level, not per record.** The source is released after the batch send succeeds. A crash between send and release re-delivers the whole block, so **VRL programs must be idempotent**. There is no exactly-once path and no per-record commit to fall back on. -**At-least-once is batch-level, not per record.** The source is released after -the batch send succeeds. A crash between send and release re-delivers the whole -block, so **VRL programs must be idempotent**. There is no exactly-once path and -no per-record commit to fall back on. - -**A Push is answered only once its records are delivered.** The listener is built armed, so a sender waits for a Kafka delivery report or the next hop's own answer. A failed or refused send is retried until the hold runs out (at most 18 s, less under a shorter sender deadline), then answered `UNAVAILABLE`. The next hop gets 15 s inside that hold: the gRPC sink's deadline, or Kafka's `message.timeout.ms` unless `librdkafka_options` sets it. `source.acknowledgements.enabled: false` answers at receipt, and a crash or failed send then loses what was answered. +**A Push is answered once its records are delivered.** The listener is built armed, so a sender waits for a Kafka delivery report or the next hop's answer. A send that fails for a recoverable reason is retried until the hold runs out (at most 18 s, less under a shorter sender deadline), then answered `UNAVAILABLE`, and a permanent failure stops the loop. The next hop gets at most 15 s inside that hold: the gRPC sink's deadline, or Kafka's `message.timeout.ms` capped at 15 s unless `sink.librdkafka_options` sets it. `source.acknowledgements.enabled: false` answers at receipt, and a crash or failed send then loses what was answered. **Dropped records still ack.** When a VRL `abort` or a transform error removes a record from a block, the block's `commit_tokens` -- the source offsets -- flow through untouched, and the block releases as dropped, not delivered. A block the sink filters out rather than sends releases the same way. A transform that filters most of its input therefore never under-acks its source, and a pipeline cannot stall because a batch emitted fewer records than it consumed. -**The outbound drain is never gated.** Self-regulation brakes intake only. -Applying the same backpressure to the producer would deadlock the pipeline: the -loop cannot drain the sink it is being blocked on. Any future work on -backpressure has to preserve that asymmetry. +**The outbound drain is never gated.** Self-regulation brakes intake alone. Applying the same backpressure to the producer would deadlock the pipeline: the loop cannot drain the sink it is being blocked on. Any future work on backpressure has to keep that asymmetry. ## Invariants: startup and runtime state -**VRL is compiled once, at startup, and is then immutable for the process -lifetime.** Transform files are read in sorted filename order and concatenated -into a single program. Changing a transform means rolling the pod -- hence -`hot_reload: false` and `reload: roll` in the suite manifest. There is no -recompile path at runtime. - -**The compile memory floor is checked before the compiler runs.** Compilation is -the one startup cost that scales with the deployment's own data. Below a certain -container limit the kernel kills the process mid-compile, which an operator sees -as an exit-137 restart loop with nothing in the log naming the cause. -`engine::budget` reads the limit first and refuses with all three numbers -- -program size, limit, required floor -- in one message. Its two constants, a -reserve and a cost per source byte, come from measurement on the shipped image, -and their doc comments name the measured points. - -**Idle is a valid, Ready state.** A transform instance is deployed before its -program exists -- whatever creates it writes the config naming its topics, and -the VRL arrives afterwards. With no topics on the bus, or no program, the app -starts, reports Ready, holds `pipeline_idle` at 1, joins no consumer group, and -logs the reason once. scalo's idle gate re-reads the config when the config file -changes or the process gets a SIGHUP, and a program written into -`transforms.dir` alone does not wake it. In Kubernetes a new program is a change -to the transforms file set, which rolls the pod. - -**Metric names are emitted bare.** The `MetricsManager` namespace is empty -unless `metrics.namespace` is set, and none is by default. A set namespace is -prepended exactly once by the global recorder's prefix layer, so app code passing -a prefixed name doubles it. Apps are told apart by Prometheus label, never by -metric name. - -**Enrichment tables are materialised once and swapped atomically.** Tables load -into `FxHashMap` for O(1) lookup and swap through `ArcSwap`. Background refresh -is opt-in per table and the interval is clamped to a 60 second minimum. Custom -VRL functions (`get_enrichment_table_record`, `find_enrichment_table_records`) -are registered alongside the VRL stdlib, backed by a registry captured at -compile time -- which is why a table cannot appear after startup. +**VRL is compiled once, at startup, and is then immutable for the process lifetime.** Transform files are read in sorted filename order and concatenated into a single program. Changing a transform means rolling the pod -- hence `hot_reload: false` and `reload: roll` in the suite manifest. There is no recompile path at runtime. + +**The compile memory floor is checked before the compiler runs.** Compilation is the one startup cost that scales with the deployment's data. Below a certain container limit the kernel kills the process mid-compile, which an operator sees as an exit-137 restart loop with nothing in the log naming the cause. `engine::budget` reads the limit first and rejects the start with all three numbers -- program size, limit, required floor -- in one message. Its two constants, a reserve and a cost per source byte, come from runs on the shipped image, and their doc comments give the data points. + +**Idle is a valid, Ready state.** A transform instance is deployed before its program exists -- whatever creates it writes the config with its topics, and the VRL arrives afterwards. With no topics on the bus, or no program, the app starts, reports Ready, holds `pipeline_idle` at 1, joins no consumer group, and logs the reason once. scalo's idle gate re-reads the config when the config file changes or the process gets a SIGHUP, and a program written into `transforms.dir` alone does not wake it. In Kubernetes a program change is a change to the transforms file set, which rolls the pod. + +**Metric names are emitted bare.** The `MetricsManager` namespace is empty unless `metrics.namespace` is set, and none is by default. A set namespace is prepended once by the global recorder's prefix layer, so app code passing a prefixed name doubles it. Apps are told apart by Prometheus label, never by metric name. + +**Enrichment tables are materialised once and swapped atomically.** File, STIX and SQLite tables load into an `FxHashMap` keyed on the key columns and swap through `ArcSwap`. A case-sensitive equality lookup on the key columns reads one bucket, and any other lookup scans the rows. MMDB tables use their reader, loaded once. Background refresh is opt-in per table and the interval is clamped to a 60 second minimum. Custom VRL functions (`get_enrichment_table_record`, `find_enrichment_table_records`) are registered beside the VRL stdlib, backed by a registry captured at compile time -- which is why a table cannot appear after startup. ## Two config authorities, one config file -The `--config` file plus `DFE_TRANSFORM_VRL_*` env vars resolve this crate's own -`Config`: `pipeline`, `source`, `sink`, `transforms`, `enrichment_tables`, `dlq`. - -scalo resolves its own sections -- `metrics`, `logger`, `memory`, `scaling`, -`worker_pool`, `batch_processing`, `self_regulation`, `version_check`, -`otel_tracing` -- from a separate cascade, which scalo's `run_app` sets up with -the `--config` file as its settings layer before the wrapper loads its own. So -the mounted config carries both, each authority taking the sections it owns. The -env layer outranks the file for both, and there a scalo section nests on a -double underscore. Only `--config` names the file: `DFE_TRANSFORM_VRL_CONFIG` -reaches neither cascade. A `config.yaml` read from the working directory with no -`--config` reaches the wrapper only, so the wrapper warns at startup for each -section `config::SCALO_CASCADE_SECTIONS` lists, rather than letting the -deployment believe it took. - -A second list, `config::INERT_SETTINGS`, covers fields this crate accepts and -validates but does not act on, each with the reason and the replacement where one -exists. `HotConfig` and `ConfigReloader` re-read and re-validate the file on -change or SIGHUP, so a bad edit logs `Config reload failed, keeping current -config` without a restart, but every field the reloadable subset carries is on -the inert list -- no pipeline behaviour changes on reload. +The `--config` file plus `DFE_TRANSFORM_VRL_*` env vars resolve this crate's `Config`: `pipeline`, `source`, `sink`, `transforms`, `enrichment_tables`, `dlq`. + +scalo resolves its sections -- `metrics`, `logger`, `memory`, `scaling`, `worker_pool`, `batch_processing`, `self_regulation`, `version_check`, `otel_tracing` -- from a separate cascade, which scalo's `run_app` sets up with the `--config` file as its settings layer before the wrapper loads its config. So the mounted config has both, each authority taking the sections it owns. The env layer outranks the file for both, and there a scalo section nests on a double underscore. `--config` is the one way to name the file: `DFE_TRANSFORM_VRL_CONFIG` reaches neither cascade. A `config.yaml` read from the working directory with no `--config` reaches the wrapper alone, so the wrapper warns at startup for each section `config::SCALO_CASCADE_SECTIONS` lists, rather than letting the deployment believe it took. + +A second list, `config::INERT_SETTINGS`, covers fields this crate accepts and validates but does not act on, each with the reason and the replacement where one exists. `HotConfig` and `ConfigReloader` re-read and re-validate the file on change or SIGHUP, so a bad edit logs `Config reload failed, keeping current config` without a restart, but every field in the reloadable subset is on the inert list -- no pipeline behaviour changes on reload. ## Components @@ -162,37 +104,20 @@ the inert list -- no pipeline behaviour changes on reload. | `src/kafka/mod.rs` | The consumer and producer configs for scalo's `KafkaTransport`, and the DFE topic and group names | | `src/deployment.rs` | `contract()` -- the source the Dockerfile and the released chart are generated from | | `src/metrics.rs` | Three metric layers, bare names | -| `pipelines/filebeat/` | Opt-in data bundle, not engine capability | +| `pipelines/filebeat/` | Opt-in legacy data bundle, not engine capability | ## Deployment -The `Dockerfile` is generated from `deployment.rs::contract()` by scalo's generator, and carries a header naming the generator and the contract schema version. `emit-dockerfile` regenerates it, and `checked_in_dockerfile_matches_emit_dockerfile` fails on a hand edit, so a fix belongs in `contract()`. +The `Dockerfile` is generated from `deployment.rs::contract()` by scalo's generator, and has a header naming the generator and the contract schema version. `emit-dockerfile` regenerates it, and `checked_in_dockerfile_matches_emit_dockerfile` fails on a hand edit, so a fix belongs in `contract()`. No chart is committed. At release, hyperi-ci emits the contract with `generate-artefacts` and assembles a thin chart from it on the scalo-service library chart at `release.helm.library`. `dfe-transform-vrl emit-contract` prints the contract: the startup budget, `kubernetes.io/h2c` on the Push port, the Kafka Secret mounted into the per-endpoint SASL names and the shared mechanism name, the DLQ's directory as the one writable path under a read-only root beside the library's scratch `/tmp`, the requests and limits, and the grace period. -The Push port (6000) is gated on `config.source.transport` being `direct` or `grpc`, the two names of the only transport that binds the listener, so the bus default (`bus` or `kafka`) publishes no port nothing answers on. The contract's ScaledObject scales on CPU alone: consumer-group lag rises when a downstream stage breaks, so it is not a trigger. A deployment adds a trigger with the library's `keda.extraTriggers` value. - -In a DFE deployment, dfe-infra's values set the CPU target in every profile. The single, scale and mesh profiles add the scaling-pressure trigger, read through dfe-keda-shim, and slim runs on CPU alone. The engine's [KEDA scaling page](https://github.com/hyperi-io/dfe-engine/blob/main/docs/deployment/keda-scaling.md) covers the triggers for every app. - -### How many instances run +The Push port (6000) is gated on `config.source.transport` being `direct` or `grpc`, the two names of the one transport that binds the listener, so the bus default (`bus` or `kafka`) publishes no port nothing answers on. The contract's ScaledObject scales on CPU alone: consumer-group lag rises when a downstream stage breaks, so it is not a trigger. A deployment adds a trigger with the library's `keda.extraTriggers` value. In a DFE deployment, dfe-infra's values set the CPU target in every profile. The single, scale and mesh profiles add the scaling-pressure trigger, read through dfe-keda-shim, and slim runs on CPU alone. The engine's [KEDA scaling page](https://github.com/hyperi-io/dfe-engine/blob/main/docs/deployment/keda-scaling.md) covers the triggers for every app. -The suite manifest, `dfe-infra/apps.yaml`, is the authority on how this app is deployed, -not this repo. It declares `multiplicity: per_config` -- one deployment per -source config, never a singleton -- with `scale_deployed: true`, both transports, -and a source binding that derives `{source}_land` as the input topic, -`{source}_load` as the output, and `dfe-transform-vrl-{source}` as the consumer -group. Adding a source is a manifest edit, not a change here. +The suite manifest, `dfe-infra/apps.yaml`, is the authority on how many instances run, not this repo. It declares `multiplicity: per_config` -- one deployment per source config, never a singleton -- with `scale_deployed: true`, both transports, and a source binding that derives `{source}_land` as the input topic, `{source}_load` as the output, and `dfe-transform-vrl-{source}` as the consumer group. Adding a source is a manifest edit, not a change here. -## Not covered here +## Known gaps -`docs/internals.md` carries the deeper design detail: the vrl-versus-vector -comparison in full, the per-record parse and serialise path, the memory budget -table, and the field-by-field reload matrix. +Readiness reads scalo's health registry, so a broker refusing the credentials, a missing topic or a consumer that cannot recover takes the pod out of service once the client meets it. But a scalo transport registers healthy before it has reached a broker, so a pod that has never reached its broker still reports Ready. -Two things are known gaps rather than omissions. Readiness reads scalo's health -registry, so a broker refusing the credentials, a missing topic or a consumer -that cannot recover takes the pod out of service once the client meets it. But a -scalo transport registers healthy before it has reached a broker, so a pod whose -broker it has never reached still reports Ready. And rotating the Kafka credential does not roll the -pods, because the chart is generator output and the `checksum/secret` annotation -has to come from scalo. +Rotating the Kafka credential does not roll the pods by default: the env it lands in resolves at pod start, and the library chart's `reload.enabled` and `reload.secrets` values add the Stakater Reloader annotations that roll them, on a cluster that runs Reloader. diff --git a/docs/internals.md b/docs/internals.md index 2c80e0c..77251b2 100644 --- a/docs/internals.md +++ b/docs/internals.md @@ -4,31 +4,43 @@ How a record moves through dfe-transform-vrl, and the settings behind each step. ## Data flow -One deployment runs one transport. On the bus, records come from a Kafka consumer and go to a Kafka producer. On the direct transport, they arrive on a Push listener and go on through a gRPC client to the next stage's listener. +This diagram shows what the process closure does to each block scalo's batch engine hands it. ```mermaid flowchart LR - subgraph SRC["Source"] - KC["Kafka consumer
bus"] - PL["Push listener
direct"] - end - subgraph PROC["process closure"] - PARSE["sonic-rs parse
JSON to Value"] - VRL["compiled VRL program"] - SER["sonic-rs serialise
Value to JSON"] - end - subgraph SNK["Sink"] - KP["Kafka producer
bus"] - GC["gRPC client
direct"] + accTitle: L4 Detail -- Process closure + accDescr: The batch engine hands a block of records to the process closure, the VRL compiler gives it the compiled program and the worker pool gives it threads. Inside, each record is parsed from JSON, run through the VRL program and serialised back to JSON. The out-batch goes to the produce sink. + + engine["scalo batch engine"] + compiler["VRL compiler"] + pool["scalo worker pool"] + + subgraph process["Process closure"] + direction LR + parse["Parse JSON"] --> run["Run VRL program"] --> serialise["Serialise JSON"] end - KC -->|WorkBatch| PARSE - PL -->|WorkBatch| PARSE - PARSE -->|Value| VRL - VRL -->|survivors| SER - SER -->|send_batch| KP - SER -->|send_batch| GC + + produce["Produce sink"] + + engine -->|block| process + compiler -.->|program| process + pool -.->|threads| process + process -->|out-batch| produce + + classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A + classDef group fill:none,stroke:#64748B,stroke-width:1px,stroke-dasharray:2 4 + class process group ``` +*Figure 2. L4 Detail -- Process closure. Dotted arrows bring what the closure runs with, not records.* Up: [L3 Components -- dfe-transform-vrl](architecture.md#who-owns-the-loop). + +`source.transport` and `sink.transport` are chosen apart, so Kafka in and gRPC out, or the reverse, is valid. The closure is the same for every pair: + +| Transport | As the source | As the sink | +|---|---|---| +| `bus` (or `kafka`) | A Kafka consumer | A Kafka producer | +| `direct` (or `grpc`) | A Push listener | A gRPC client, to the next stage's listener | + Per record: 1. `sonic_rs::from_slice` parses the payload straight into a VRL `Value`, which is serde-compatible. A record that is not JSON, or nests deeper than sonic-rs accepts, is dropped. @@ -36,19 +48,19 @@ Per record: 3. `sonic_rs::to_vec` serialises each survivor back to JSON, keyed by the sink topic. 4. The loop sends the whole out-batch with `TransportSender::send_batch`. -Release is batch-level. scalo's batch engine hands a block of records to the transform, sends the result, and releases the block's source offsets once the batch send returns Ok. Under that, scalo's Kafka transport holds each source offset it handed out and commits a partition only up to its lowest unreleased one, so the cumulative commit never passes a record not yet released. +Release is batch-level. scalo's batch engine hands a block of records to the transform, sends the result, and releases the block's source offsets once the batch send returns Ok. Under that, scalo's Kafka transport holds each source offset it handed out and commits a partition up to its lowest unreleased one, so the cumulative commit never passes a record not yet released. If the process crashes between send and release, the whole block is re-consumed and re-processed (at-least-once, not exactly-once). VRL transforms must be idempotent. -JSON is the only payload format. MessagePack, supported in DFE/XDR 2.0 and 2.1, is deprecated in DFE 2.2 and no longer accepted: the JSON path (SIMD parsing with sonic-rs, zstd on the wire) is fast enough that MessagePack gave no CPU saving. A record that does not parse as JSON is removed from its block, counted in `records_error_total{stage="deserialise"}`, logged at debug level with the parse error (sampled), and reported to the security log by reason, with nothing from the record. The block's source is then released `Dropped`, so the record is not redelivered. +JSON is the one payload format. MessagePack, supported in DFE/XDR 2.0 and 2.1, is deprecated in DFE 2.2 and no longer accepted. JSON parses with SIMD in sonic-rs, and the Kafka sink compresses with `sink.compression`, zstd by default. A record that does not parse as JSON is removed from its block, counted in `records_error_total{stage="deserialise"}`, logged at debug level with the parse error (sampled), and reported to the security log by reason, with nothing from the record. The block's source is then released `Dropped`, so the record is not redelivered. ## VRL integration -`compile_vrl` compiles the source once, at startup, against VRL's stdlib less `get_hostname`, plus the two enrichment functions, with the enrichment registry in the compile config. A compile error quotes the line that failed and stops startup. The compiled `Program` is reused for every event -- zero per-event compilation cost. +`compile_vrl` compiles the source once, at startup, against VRL's stdlib less `get_hostname`, plus the two enrichment functions, with the enrichment registry in the compile config. A compile error quotes the line that failed and stops startup. The compiled `Program` is reused for every event, with no per-event compilation. `run_vrl` wraps each event in a `TargetValueRef` with an empty metadata object, and resolves it with UTC as the default timezone, so a zone-less `parse_timestamp` reads the same on every node. `abort` drops the event and counts it as filtered. Any other runtime error drops it and counts it in `records_error_total{stage="transform"}`. A panic inside VRL, which its grok and arithmetic functions can raise on record data, fails that record rather than the process, because the release profile unwinds. -vrl is built without its `enable_env_functions` and `enable_network_functions` features, and `compile_vrl` drops `get_hostname` from the system functions it keeps for `parse_etld`. A program calling `get_env_var`, `get_hostname`, `http_request`, `dns_lookup` or `reverse_dns` fails to compile as a call to an undefined function. `parse_proto` and `encode_proto` stay, and read the descriptor file their literal path names at compile time. `validate_json_schema` stays too, and reads its schema file when the first event reaches the call, keeping it once it compiles. A missing or invalid schema passes compilation and then fails each such event as a transform error. +vrl is built without its `enable_env_functions` and `enable_network_functions` features, and `compile_vrl` drops `get_hostname` from the system functions it keeps for `parse_etld`. A program calling `get_env_var`, `get_hostname`, `http_request`, `dns_lookup` or `reverse_dns` fails to compile as a call to an undefined function. `parse_proto` and `encode_proto` stay, and read the descriptor file their literal path names at compile time. `validate_json_schema` stays as well, and reads its schema file when the first event reaches the call, keeping it once it compiles. A missing or invalid schema passes compilation and then fails each such event as a transform error. Transform files contain raw VRL source code (not Vector YAML): @@ -62,34 +74,22 @@ del(.message) .processed_at = now() ``` -The `.vrl` files in `transforms.dir` load in sorted filename order, then the paths in `transforms.files` in the order given, and they concatenate into a single VRL program. This matches Vector's remap transform behaviour where multiple VRL statements execute sequentially. - -`pipelines/filebeat/` ships a pre-canned, opt-in port of the DFE 2.1 Vector -filebeat templates (Cisco Meraki logs, Cisco IOS, Cisco Umbrella) as one -pure-VRL file plus its `timezones.csv` enrichment table. It is convenience -data, not engine capability: opting in means pointing the standard -`transforms.dir` and `enrichment_tables` config at those files. The engine -carries no filebeat-specific code, and the integration tests -(`tests/integration/filebeat_pipeline.rs`) drive the bundle with the synthetic -cases in `tests/fixtures/filebeat/cases.yaml`, which double as a full-engine -workload. +The `.vrl` files in `transforms.dir` load in sorted filename order, then the paths in `transforms.files` in the order given, and they concatenate into a single VRL program. This matches Vector's remap transform, where multiple VRL statements execute in sequence. -LEGACY: it keeps DFE/XDR 2.0 and 2.1 filebeat feeds working, takes no new modules, and new feeds use `dfe-transform-elastic`. Wiring, routing behaviour, known limitations, and -regeneration tooling (`scripts/filebeat/`) are documented in -[pipelines/filebeat/README.md](../pipelines/filebeat/README.md). +`pipelines/filebeat/` ships a legacy, opt-in port of the DFE 2.1 Vector filebeat templates (Cisco Meraki logs, Cisco IOS, Cisco Umbrella) as one pure-VRL file plus its `timezones.csv` enrichment table. It keeps DFE/XDR 2.0 and 2.1 filebeat feeds working, takes no further modules, and any other feed uses dfe-transform-elastic. To use it, point the standard `transforms.dir` and `enrichment_tables` config at those files. The engine has no filebeat-specific code, and the integration tests (`tests/integration/filebeat_pipeline.rs`) drive the bundle with the synthetic cases in `tests/fixtures/filebeat/cases.yaml`, which double as a full-engine workload. Wiring, routing, limitations and the regeneration tooling (`scripts/filebeat/`) are in [pipelines/filebeat/README.md](../pipelines/filebeat/README.md). ## Memory budget -The wrapper bounds the buffers it owns. [config-schema.yaml](config-schema.yaml) carries the defaults for its own keys, and scalo's `BatchProcessingConfig` the default chunk size. +The wrapper bounds the buffers it holds. [config-schema.yaml](config-schema.yaml) has the defaults for the wrapper's keys, and scalo's `BatchProcessingConfig` the default chunk size. | Component | Configuration | |-----------|--------------| | Consumer prefetch | `source.max_buffer_bytes`, set as librdkafka's `queued.max.messages.kbytes` | | Transform chunk | `batch_processing.max_chunk_size`, a scalo section, from the `--config` file or the env layer | | Producer queue | `sink.max_buffer_bytes`, set as librdkafka's `queue.buffering.max.kbytes` | -| Total pod memory | The container memory limit, whose chart default `src/deployment.rs::contract()` sets | +| Total pod memory | The container memory limit, with its chart default set in `src/deployment.rs::contract()` | -The consumer and producer buffers have to fit within the pod memory limit beside the binary, the runtime, the compiled program and the enrichment tables. A full producer queue comes back from the send as backpressure, and the loop retries that block before it reads another, so intake waits on the sink and librdkafka's prefetch stops at `source.max_buffer_bytes`. Under memory pressure scalo's governor pauses the assigned partitions, and on the direct transport the listener refuses pushes `UNAVAILABLE`. +The consumer and producer buffers have to fit within the pod memory limit beside the binary, the runtime, the compiled program and the enrichment tables. A full producer queue comes back from the send as backpressure, and the loop retries that block before it reads another, so intake waits on the sink. `source.max_buffer_bytes` sets librdkafka's prefetch threshold, clamped and overridable through `source.librdkafka_options`, and a single fetch can go past it. Under memory pressure scalo's governor pauses the assigned partitions, and on the direct transport the listener rejects pushes `UNAVAILABLE`. ## Comparison: dfe-transform-vrl vs dfe-transform-vector @@ -98,7 +98,7 @@ The consumer and producer buffers have to fit within the pod memory limit beside | Transform engine | VRL crate (in-process) | Vector subprocess | | Kafka source/sink | scalo's `KafkaTransport` over rdkafka (wrapper-controlled) | Vector-managed | | Memory control | Bounded consumer and producer buffers, scalo's memory guard | None (Vector unbounded) | -| Supported transforms | VRL only | All Vector transforms | +| Supported transforms | VRL | All Vector transforms | | Container image size | Smaller (Rust binary) | Larger (Rust + Vector) | | Process model | Single process | Wrapper + subprocess | | Startup time | Fast (compile VRL) | Slower (spawn + healthcheck Vector) | @@ -108,28 +108,21 @@ Use dfe-transform-vrl for VRL-only transforms (the common case), memory-constrai ## Reload -`ConfigReloader` (file polling + SIGHUP) re-reads and re-validates the file and -swaps `SharedConfig`. A bad edit logs `Config reload failed, keeping current config` and counts in `config_reloads_total{result="error"}`, and the service keeps running. The reloadable subset carries three fields: +`ConfigReloader` (file polling + SIGHUP) re-reads and re-validates the file and swaps `SharedConfig`. A bad edit logs `Config reload failed, keeping current config` and counts in `config_reloads_total{result="error"}`, and the service keeps running. The reloadable subset has three fields, and the pipeline reads none of them: | Field | What it would control | State | |-------|----------------------|-------| -| `pipeline.batch_size` | Events per transform chunk | not applied -- see below | +| `pipeline.batch_size` | Events per transform chunk | not applied -- the chunk size is `batch_processing.max_chunk_size`, a scalo section | | `pipeline.batch_timeout_ms` | Max wait before flushing partial batch | not applied -- no such timer | | `sink.key_field` | Kafka partition key path | not applied -- scalo-rs#37 | -The pipeline holds the reloadable subset but reads none of it, so a reload -changes no pipeline behaviour. `config::INERT_SETTINGS` is the list, and -the wrapper warns at startup for each one a deployment has moved off its default. The dial that -does size a chunk is `batch_processing.max_chunk_size`, a scalo section. +`config::INERT_SETTINGS` lists those three and `source.commit_interval_ms`, which the production Kafka profile leaves unread because librdkafka auto-commit is off and the engine commits after each sink ack. The wrapper warns at startup for each one a deployment has moved off its default. -The fields below are bound to connections, compiled programs, or server sockets -established at startup. Changing them requires a pod restart (which is the -standard K8s pattern -- ConfigMap changes trigger rolling restart via the -`checksum/config` annotation in the Deployment template). +The fields below are bound to connections, compiled programs, or server sockets set up at startup. Changing them needs a pod restart, and the chart rolls the pods: a config change alters the `checksum/config` annotation on the pod template. | Field | Why | |-------|-----| -| `pipeline.name` | Baked into Kafka group_id, metrics labels, tracing spans | +| `pipeline.name` | The producer's `client.id` and the log fields. The consumer group is `source.group_id` | | `source.*` | rdkafka consumer or Push listener: connection, subscription, auth, TLS, buffers | | `sink.transport`, `sink.endpoint` | The sink built at startup | | `sink.brokers` | rdkafka producer connection | @@ -144,9 +137,7 @@ standard K8s pattern -- ConfigMap changes trigger rolling restart via the ## Operating notes -The transform's own sections are `pipeline`, `source`, `sink`, `transforms`, `enrichment_tables` and `dlq`, and [config-schema.yaml](config-schema.yaml) lists every key in them. Only `--config` names the file: `DFE_TRANSFORM_VRL_CONFIG` reaches neither the wrapper nor scalo, and with no `--config` the wrapper reads `config.yaml` or `config.yml` from the working directory. - -### Environment variable overrides +The transform's sections are `pipeline`, `source`, `sink`, `transforms`, `enrichment_tables` and `dlq`, and [config-schema.yaml](config-schema.yaml) lists every key in them. `--config` is the one way to name the file: `DFE_TRANSFORM_VRL_CONFIG` reaches neither the wrapper nor scalo, and with no `--config` the wrapper reads `config.yaml` or `config.yml` from the working directory. The big dials have flat env var overrides for K8s. These are every flat name `apply_flat_env` in `src/config/loader.rs` reads: @@ -178,14 +169,9 @@ An empty value reads as unset. Every other key of the same sections takes the do The chart mounts the Kafka Secret's `username` and `password` into both endpoint pairs, and its `sasl.mechanism` (`PLAIN`, `SCRAM-SHA-512`) into `KAFKA_SASL_MECHANISM`. Outside the chart the shared `KAFKA_SASL_*` names reach both endpoints, and the per-endpoint names override them. -Either half turns SASL on, an enabled block still missing one once the env -layer has run refuses to start, and the password is redacted on every output -path (`x-scalo-secret` + `writeOnly` in the emitted schema). - -### Kafka over TLS with a private CA +Either half turns SASL on, an enabled block still missing one once the env layer has run stops startup, and the password is redacted on every output path (`x-scalo-secret` + `writeOnly` in the emitted schema). -`tls.ca_cert_file` is a path, and the chart mounts no certificate file. Put the -CA's PEM text in the rendered config instead: +For Kafka over TLS with a private CA: `tls.ca_cert_file` is a path, and the chart mounts no certificate file. Put the CA's PEM text in the rendered config instead: ```yaml config: @@ -199,18 +185,9 @@ config: -----END CERTIFICATE----- ``` -The same two keys go under `sink`. A CA is public, so a ConfigMap can hold it. -Mutual TLS is not supported through the chart, which mounts no client key. KEDA -scales on CPU and opens no Kafka connection, so it needs no CA. - -### scalo's own settings share the config file +The same two keys go under `sink`. A CA is public, so a ConfigMap can hold it. Mutual TLS is not supported through the chart, which mounts no client key. KEDA scales on CPU and opens no Kafka connection, so it needs no CA. -`metrics`, `logger`, `memory`, `scaling`, `worker_pool`, `batch_processing`, -`self_regulation`, `version_check` and `otel_tracing` are resolved by scalo, not -by the wrapper. -scalo reads the file passed with `--config` as its settings layer, once at -startup, so those sections take effect from the mounted `config.yaml` beside the -wrapper's own: +scalo's settings share the config file. `metrics`, `logger`, `memory`, `scaling`, `worker_pool`, `batch_processing`, `self_regulation`, `version_check` and `otel_tracing` are resolved by scalo, not by the wrapper. scalo reads the file given with `--config` as its settings layer, once at startup, so those sections take effect from the mounted `config.yaml` beside the wrapper's: ```yaml scaling: @@ -219,11 +196,9 @@ batch_processing: max_chunk_size: 10000 ``` -A `config.yaml` picked up from the working directory with no `--config` is read -by the wrapper alone, so a scalo section in it changes nothing. The wrapper warns at startup for each section `config::SCALO_CASCADE_SECTIONS` lists, `memory` included. Pass the file with `--config`. +A `config.yaml` picked up from the working directory with no `--config` is read by the wrapper alone, so a scalo section in it changes nothing. The wrapper warns at startup for each section `config::SCALO_CASCADE_SECTIONS` lists, `memory` included. Pass the file with `--config`. -The env layer outranks the file, and there the section nests on a **double** -underscore: +The env layer outranks the file, and there the section nests on a **double** underscore: ```bash METRICS_ADDR=0.0.0.0:9090 # or DFE_TRANSFORM_VRL_METRICS__ADDRESS @@ -235,12 +210,11 @@ DFE_TRANSFORM_VRL_OTEL_TRACING__ENABLED=false # OTLP span export, on by defau DFE_TRANSFORM_VRL_METRICS__OTEL__ENABLED=false # OTLP metric push, on by default ``` -A single underscore (`DFE_TRANSFORM_VRL_METRICS_ADDRESS`) produces a flat key that -matches no section and is ignored, and that spelling also warns. The memory guard is the exception: scalo also reads `DFE_TRANSFORM_VRL_MEMORY_LIMIT_BYTES`, `DFE_TRANSFORM_VRL_MEMORY_PRESSURE_THRESHOLD` and `DFE_TRANSFORM_VRL_MEMORY_CGROUP_HEADROOM` on a single underscore. +A single underscore (`DFE_TRANSFORM_VRL_METRICS_ADDRESS`) produces a flat key that matches no section and is ignored, and that spelling warns as well. The memory guard is the exception: scalo reads `DFE_TRANSFORM_VRL_MEMORY_LIMIT_BYTES`, `DFE_TRANSFORM_VRL_MEMORY_PRESSURE_THRESHOLD` and `DFE_TRANSFORM_VRL_MEMORY_CGROUP_HEADROOM` on a single underscore. ## Helm chart -No chart is committed here. At release, hyperi-ci runs the binary's `generate-artefacts` for the deployment contract and assembles a thin chart from it on the scalo-service library chart, at the version `release.helm.library` names in `.hyperi-ci.yaml`. Keep that version at the scalo version in `Cargo.toml`: a library renders only the contract version its scalo release writes. +No chart is committed here. At release, hyperi-ci runs the binary's `generate-artefacts` for the deployment contract and assembles a thin chart from it on the scalo-service library chart, at the version `release.helm.library` names in `.hyperi-ci.yaml`. Keep that version at the scalo version in `Cargo.toml`: a library renders the contract version its scalo release writes and no other. To see the chart a release would ship, build the binary and run `hyperi-ci chart assemble --binary target/debug/dfe-transform-vrl --image ghcr.io/hyperi-io/dfe-transform-vrl:@sha256: --version `. It prints the chart directory it wrote. diff --git a/pipelines/filebeat/README.md b/pipelines/filebeat/README.md index 20a0704..2d59f23 100644 --- a/pipelines/filebeat/README.md +++ b/pipelines/filebeat/README.md @@ -1,17 +1,10 @@ # Bundled pipeline: filebeat-compat (LEGACY) -A pre-canned, opt-in VRL pipeline that keeps DFE/XDR 2.0 and 2.1 filebeat feeds working -on dfe-transform-vrl. It is pure VRL plus one CSV enrichment table - no -binary capability, no config surface. Use it if you want it, ignore it if -you do not. It also doubles as an integration test workload for the engine -(`tests/integration/filebeat_pipeline.rs`). +A pre-canned, opt-in VRL pipeline that keeps DFE/XDR 2.0 and 2.1 filebeat feeds working on dfe-transform-vrl. It is pure VRL plus one CSV enrichment table -- no binary capability, no config surface. It doubles as an integration test workload for the engine (`tests/integration/filebeat_pipeline.rs`). -> **LEGACY.** Backwards compatibility for DFE/XDR 2.0 and 2.1 filebeat feeds, frozen at the three Cisco modules below. It takes no new modules, fetches nothing from Elastic at build, test or run time, and is regenerated only from our own DFE 2.1 templates. New feeds use `dfe-transform-elastic`, and 2.0 and 2.1 feeds should plan to move there. +> **LEGACY.** Backwards compatibility for DFE/XDR 2.0 and 2.1 filebeat feeds, frozen at the three Cisco modules below. It takes no further modules, fetches nothing from Elastic at build, test or run time, and is regenerated from the DFE 2.1 templates and nothing else. Any other feed uses `dfe-transform-elastic`, and 2.0 and 2.1 feeds should plan to move there. -## What it covers - -Single-file port of the DFE 2.1 Vector templates (dfe-vector-templates -`src/core_templates`, the `14*-transform-filebeat-*` family): +It is a single-file port of the DFE 2.1 Vector templates (dfe-vector-templates `src/core_templates`, the `14*-transform-filebeat-*` family): | DFE 2.1 template | Module | |---|---| @@ -19,11 +12,11 @@ Single-file port of the DFE 2.1 Vector templates (dfe-vector-templates | 141-transform-filebeat-cisco-ios-logs | Cisco IOS syslog | | 143..149-transform-filebeat-cisco-umbrella-*-logs | Cisco Umbrella exports (audit, cloud firewall, dlp, dns, intrusion, ip, proxy) | -The seven umbrella templates were one body split across files (Vector -large-VRL workarounds) differing only in a hardcoded `.log.file.path` seed. -They collapse here into one body behind log-type detection. +The seven umbrella templates were one body split across files (Vector large-VRL workarounds), differing in their pipeline identifier and a hardcoded `.log.file.path` seed. They collapse here into one body behind log-type detection. + +`scripts/filebeat/extract.py` generated it from dfe-vector-templates `origin/main` @ `96eb306`: it checks the seven umbrella bodies match once those two are normalised, before collapsing them, strips geoip, unescapes Vector `$$` and prepends `scripts/filebeat/prologue.vrl`. The tests replay the synthetic lines in `tests/fixtures/filebeat/cases.yaml`. Treat `filebeat.vrl` as generated-then-patched: fixes belong here (it is the shipping artefact), but substantial rework should go back through the template extraction so the umbrella dedupe stays verifiable. -## Opting in +To opt in, point the standard config at the bundle: ```yaml # config.yaml @@ -36,17 +29,15 @@ enrichment_tables: key_columns: ["abbreviation"] ``` -Both files are in this directory. Mount or copy them wherever the pod can -read them. Nothing else is needed - the engine treats this like any other -`transforms.dir`. +Both files are in this directory. Mount or copy them wherever the pod can read them. Nothing else is needed -- the engine treats this like any other `transforms.dir`. -Events carry the DFE 2.1 Kafka fields: `{message, tags, timestamp}`. Two +Events have the DFE 2.1 Kafka fields: `{message, tags, timestamp}`. Two optional event fields drive ios/meraki timezone handling: `._conf.tz_offset` (an offset like `-0600` or an abbreviation from `timezones.csv`) and `._conf.tz_map` (a list of `{tz_short, tz_long}` mappings from log abbreviations to IANA zone names). Without them, meraki still derives `@timestamp` from the log line's epoch, and ios interprets zone-less -timestamps as UTC - the knobs override the zone interpretation, they do +timestamps as UTC -- the knobs override the zone interpretation, they do not gate timestamping. ## How routing works @@ -64,29 +55,31 @@ per event, so this port detects the module first: count: 7 cols = ip, 9 with the timestamp in column 1 = audit, 13 (or 10-12 with the ` (TYPE)` query-type marker in column 6) = dns, 14/16 = cloud firewall, 17 = intrusion when columns 3-4 are numeric gid/sid else - dlp, >=20 = proxy. Every rule but audit also requires a timestamp in + dlp, >=20 = proxy. Every rule but audit needs a timestamp in column 0 5. anything else: tagged `filebeat_unmatched` and passed through unchanged (the 2.1 fan-out silently dropped these) Detection picks the branch, and parsing failures INSIDE the chosen branch still error the event out (dropped and counted), matching the 2.1 -`drop_on_abort` behaviour. Only events no rule matches pass through. +`drop_on_abort` behaviour. Events no rule matches are the ones that pass through. + +## Where it differs -## Deliberate departures from DFE 2.1 +It departs from DFE 2.1 on purpose in four places: - No geoip enrichment. dfe-loader owns geoip in the current pipeline layout, so the `geoip_city`/`geoip_asn` lookups are stripped. - Unmatched events pass through tagged, never silently dropped. - `.log.file.path` is rewritten to `/filebeat/cisco_umbrella/` - in the umbrella branch (2.1 also overwrote it, with `/test/path/`). + in the umbrella branch (2.1 overwrote it as well, with `/test/path/`). - IANA zone names resolve correctly in the ios branch. The 2.1 templates pushed a `tz_map`-mapped name like `Australia/Sydney` into the abbreviation CSV lookup, which always failed and dropped the event. The port routes slash-bearing zone names through `parse_timestamp`'s `timezone:` argument (DST-aware via the tz database). -## Known limitations +Its output differs from elastic's in these places: - `user_agent` classification follows VRL's parser, which differs cosmetically from elastic's ("Mac OSX" vs "Mac OS X"). A @@ -95,20 +88,8 @@ still error the event out (dropped and counted), matching the 2.1 ordered), and elastic preserves raw order. Same parameters, different order. - `url.path` of a URL with no path is `/`, where elastic keeps it empty. - `dns.question.registered_domain` uses the full public-suffix list - including the private section, and elastic uses the ICANN section only. -- Ambiguous timezone abbreviations (the CSV carries duplicates such as - ADT, AMT, AST, CST, IST) never resolve - the 2.1 "ambiguous timezone + including the private section, and elastic uses the ICANN section alone. +- Ambiguous timezone abbreviations (the CSV has duplicates such as + ADT, AMT, AST, CST, IST) never resolve -- the 2.1 "ambiguous timezone abbreviation" path is preserved. - A Cisco IOS timestamp with no year takes the current year. - -## Provenance and regeneration - -Generated from dfe-vector-templates `origin/main` @ `96eb306` by -`scripts/filebeat/extract.py` (verifies the seven umbrella templates are -identical before collapsing them, strips geoip, unescapes Vector `$$`, -prepends `scripts/filebeat/prologue.vrl`). The tests replay the synthetic -lines in `tests/fixtures/filebeat/cases.yaml`. - -Treat `filebeat.vrl` as generated-then-patched: fixes belong here (it is -the shipping artefact), but substantial rework should go back through the -template extraction so the umbrella dedupe stays verifiable. From d844220670af18769a481f7ed8c3b7fcb89c1edf Mon Sep 17 00:00:00 2001 From: Derek Date: Mon, 16 Nov 2026 13:44:19 +1100 Subject: [PATCH 2/2] docs(transform-vrl): say what each diagram shows The intro sentences above the architecture and internals diagrams say what each diagram shows. --- docs/architecture.md | 2 +- docs/internals.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/architecture.md b/docs/architecture.md index 09f0d96..ecd9f77 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -18,7 +18,7 @@ It is not a general transform host: a pipeline needing a Vector-native transform scalo's `BatchEngine::pipeline` drives `recv -> process -> send -> release`, including the held source acknowledgement and self-regulation. This crate supplies the `process` closure and the produce sink, plus the config, the VRL compiler and the enrichment registry. -This diagram shows the parts inside dfe-transform-vrl that a block of records passes through. +The batch engine hands each block to the process closure, which runs the compiled VRL program on the worker pool and passes the out-batch to the produce sink. ```mermaid flowchart LR diff --git a/docs/internals.md b/docs/internals.md index 77251b2..cf1f154 100644 --- a/docs/internals.md +++ b/docs/internals.md @@ -4,7 +4,7 @@ How a record moves through dfe-transform-vrl, and the settings behind each step. ## Data flow -This diagram shows what the process closure does to each block scalo's batch engine hands it. +The process closure parses each record from JSON, runs the compiled VRL program on it and serialises the result back to JSON, on the worker pool's threads. ```mermaid flowchart LR