Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 1 addition & 7 deletions .hyperi-ci.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -8,23 +8,19 @@ classification: product

language: rust

# Quality checks
quality:
enabled: true
# Advisory ignores fed to cargo-audit and osv-scanner.
ignore:
# Unreachable, accepted in dfe-infra docs/THREAT-MODEL.md. scalo-rs#281 tracks the move.
- tool: osv-scanner
id: GHSA-w9wp-h8wv-79jx
reason: "CVE-2026-48504, opentelemetry_sdk 0.31.0 via scalo: unreachable, no Baggage propagator registered in dfe-transform-vrl or scalo. Accepted risk per dfe-infra docs/THREAT-MODEL.md. Drop when hyperi-io/scalo-rs#281 moves scalo off opentelemetry 0.31, blocked on metrics-exporter-opentelemetry 0.2.1 requiring opentelemetry ^0.31."
# unsafe-libyaml arrives through the vrl crate's serde_yaml_ng, a normal and a build dependency of it; revisit when vrl moves off serde_yaml_ng.
- tool: osv-scanner
id: RUSTSEC-2024-0449
reason: "unsafe-libyaml is unmaintained, with no known CVE (RUSTSEC-2023-0075 was fixed in 0.2.10). Only operator-written VRL reaches it on event data: the stdlib's parse_yaml hands libyaml what the event carries, and ingested data cannot add that call. Config YAML parses through scalo's serde-saphyr. Drop when the vrl crate moves off serde_yaml_ng."
rust:
features: default

# Testing
test:
enabled: true
coverage: true
Expand All @@ -37,7 +33,6 @@ test:
full:
required_for_release: true

# Building
build:
enabled: true
strategies:
Expand All @@ -57,8 +52,7 @@ build:
- x86_64-unknown-linux-gnu
- aarch64-unknown-linux-gnu

# Release (where the artefacts go). The thin chart is assembled from the
# emitted contract and ships beside the image.
# The thin chart: docs/internals.md, Helm chart.
release:
enabled: true
# Keep `library` at the scalo version in Cargo.toml: the library renders only the contract version its scalo release writes.
Expand Down
30 changes: 9 additions & 21 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -137,16 +137,7 @@ If you have questions about contributing, please open an issue or contact us.

### What this is

A single Rust binary that runs VRL transforms over records in the DFE data path
-- Kafka in and out on the `bus` transport, gRPC in and out on the `direct`
transport. Two boundaries get assumed wrongly. First, it is VRL only: a pipeline
needing `lua`, `aggregate`, `dedupe`, `throttle` or `sample` belongs in
dfe-transform-vector, which keeps the Vector subprocess and pays for it. Second,
this crate does not own its own event loop. scalo's `BatchEngine::pipeline`
drives `recv -> process -> send -> release`, and this crate supplies the `process`
closure, the produce sink, the config, the VRL compiler and the enrichment
registry. Reading `src/pipeline.rs` expecting to find the loop is the usual wrong
turn.
A single Rust binary that runs VRL transforms over records in the DFE data path -- Kafka in and out on the `bus` transport, gRPC in and out on the `direct` transport. Two boundaries get assumed wrongly. First, it runs VRL and nothing else: a pipeline needing `lua`, `aggregate`, `dedupe`, `throttle` or `sample` belongs in dfe-transform-vector, which keeps the Vector subprocess and pays for it. Second, this crate does not run its event loop. scalo's `BatchEngine::pipeline` drives `recv -> process -> send -> release`, and this crate supplies the `process` closure, the produce sink, the config, the VRL compiler and the enrichment registry. Reading `src/pipeline.rs` expecting to find the loop is the usual wrong turn.

### Where things live

Expand Down Expand Up @@ -176,27 +167,24 @@ cargo run --bin dfe-transform-vrl -- emit-dockerfile > Dockerfile # regenerat

Green lies here in three ways, and all three are on by default.

`default = []` in `Cargo.toml`, so a bare `cargo nextest run` does not compile the MMDB or SQLite enrichment tests in at all. `.hyperi-ci.yaml` adds `enrichment-mmdb` and `enrichment-sqlite` to both the test and the build feature sets -- the build too, because the Dockerfile only copies the binary, so a default build ships a container that rejects `type: mmdb` and `type: sqlite` tables at startup.
`default = []` in `Cargo.toml`, so a bare `cargo nextest run` does not compile the MMDB or SQLite enrichment tests in at all. `.hyperi-ci.yaml` adds `enrichment-mmdb` and `enrichment-sqlite` to both the test and the build feature sets -- the build as well, because the Dockerfile copies the binary and nothing else, so a default build ships a container that rejects `type: mmdb` and `type: sqlite` tables at startup.

The core-tier e2e tests run on broker containers scalo's test support starts. Without Docker they skip locally and fail only under `CI`, so a local run without Docker is green without having reached a broker.
The core-tier e2e tests run on broker containers scalo's test support starts. Without Docker they skip locally and fail under `CI` alone, so a local run without Docker passes without having reached a broker.

The tests that may reach a broker somebody else owns are `#[ignore]` and run only under `--run-ignored all` or `only`. They take a live broker first when `KAFKA_BROKERS` answers. `TEST_MODE=docker` points the harness at `localhost:19092` with no auth, the port dfe-docker publishes its broker on, and otherwise it reads `KAFKA_BROKERS`, `KAFKA_SECURITY_PROTOCOL` and the `KAFKA_SASL_*` names from the environment or the repo's `.env`.
The tests that may reach a broker somebody else owns are `#[ignore]`. nextest runs them under `--run-ignored all` or `only`, and `cargo test` under `-- --ignored`. They take a live broker first when `KAFKA_BROKERS` answers. `TEST_MODE=docker` points the harness at `localhost:19092` with no auth, the port dfe-docker publishes its broker on, and otherwise it reads `KAFKA_BROKERS`, `KAFKA_SECURITY_PROTOCOL` and the `KAFKA_SASL_*` names from the environment or the repo's `.env`.

One more: the push trigger in `.github/workflows/ci.yml` carries
`paths-ignore: docs/**, **.md`, so a docs-only push runs no jobs. The
`pull_request` trigger has no such filter, so the PR is where a docs change gets
checked.
One more: the push trigger in `.github/workflows/ci.yml` has `paths-ignore: docs/**, **.md`, so a docs-only push runs no jobs. The `pull_request` trigger has no such filter, so the PR is where a docs change gets checked.

### What tends to bite

| Don't | Do | Why |
|---|---|---|
| Hand-edit `Dockerfile`, or commit a chart | Fix `src/deployment.rs::contract()` and regenerate | The Dockerfile is generator output, and `checked_in_dockerfile_matches_emit_dockerfile` fails on a hand edit. The release assembles the chart from the contract, so a committed chart never ships, and `test_every_contract_secret_env_var_reaches_the_config` holds every declared name to the field it spells |
| Bump scalo and leave `release.helm.library` behind | Move `release.helm.library` in `.hyperi-ci.yaml` to the same scalo version | A scalo-service release renders only the contract version its scalo release writes |
| Turn the Kafka lag trigger back on in `contract()` | Leave `KafkaLagTrigger::disabled()`, so the contract scales on CPU and a deployment adds the scaling-pressure trigger | Lag rises when a downstream stage breaks, so a lag trigger adds pods that wait on the same broken stage. The lag trigger also reads `config.kafka.*`, which this app's values do not have |
| Put a scalo section (`metrics`, `logger`, `memory`, `scaling`, `worker_pool`, `batch_processing`, `self_regulation`, `version_check`, `otel_tracing`) in a `config.yaml` the binary finds in its working directory | Pass the file with `--config`, or set the section through the env layer on a **double** underscore | scalo reads the `--config` file as its settings layer but finds other files only by fixed base name (`settings.yaml`, `defaults.yaml`), so a working-directory `config.yaml` reaches the wrapper alone. Full list under [Environment variable overrides](docs/internals.md#environment-variable-overrides) |
| Bump scalo and leave `release.helm.library` behind | Move `release.helm.library` in `.hyperi-ci.yaml` to the same scalo version | A scalo-service release renders the contract version its scalo release writes and no other |
| Turn the Kafka lag trigger back on in `contract()` | Leave `KafkaLagTrigger::disabled()`, so the contract scales on CPU and a deployment adds the scaling-pressure trigger | Lag rises when a downstream stage breaks, so a lag trigger adds pods that wait on the same broken stage. The lag trigger reads `config.kafka.*` as well, which this app's values do not have |
| Put a scalo section (`metrics`, `logger`, `memory`, `scaling`, `worker_pool`, `batch_processing`, `self_regulation`, `version_check`, `otel_tracing`) in a `config.yaml` the binary finds in its working directory | Pass the file with `--config`, or set the section through the env layer on a **double** underscore | scalo reads the `--config` file as its settings layer but finds other files by fixed base name alone (`settings.yaml`, `defaults.yaml`), so a working-directory `config.yaml` reaches the wrapper alone. Full list under [Operating notes](docs/internals.md#operating-notes) |
| Trust `pipeline.batch_size`, `pipeline.batch_timeout_ms`, `sink.key_field` or `source.commit_interval_ms` | Size a chunk with `batch_processing.max_chunk_size` | `config::INERT_SETTINGS` -- accepted, validated, reaching nothing. Table under [Reload](docs/internals.md#reload) |
| Call a bare `cargo nextest run` green | Pass `--features enrichment-mmdb,enrichment-sqlite` | `default = []`, so the MMDB and SQLite tests are not compiled in and the run is green without having tested them |
| Call a bare `cargo nextest run` a pass | Pass `--features enrichment-mmdb,enrichment-sqlite` | `default = []`, so the MMDB and SQLite tests are not compiled in and the run passes without having tested them |
| Set `sasl.enabled` with an empty username or password | Supply both, or neither | librdkafka's SCRAM check is a NULL check that an empty string passes, so the pod would authenticate against nothing and still report Ready. `validate` refuses it at startup |
| Add a serialise path that redacts by field name | Keep the password a `scalo::SensitiveString` | Redaction is by type on every path -- `Debug`, the `config-check` dump, the emitted schema. The figment round-trip has to be wrapped in `expose_during` or a file-sourced password reaches the broker as the literal `***REDACTED***` |
| Size the container from steady state when the program is large | Leave headroom for the compile | Compilation scales with the VRL, and below the floor `engine::budget` computes the kernel kills the process mid-compile, seen as an exit-137 restart loop with nothing in the log. `engine::budget` refuses first and names all three numbers |
Expand Down
16 changes: 3 additions & 13 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,6 @@ path = "src/bin/pgo_driver.rs"
required-features = ["pgo-driver"]

[dependencies]
# Async runtime
tokio = { version = ">=1.54.1, <2", features = ["full"] }

# CancellationToken, scalo's shutdown token type.
Expand Down Expand Up @@ -74,49 +73,40 @@ scalo = { version = ">=2.14.30, <3", features = [
# Config-schema derivation. Same range as scalo so cargo unifies to one schemars.
schemars = { version = ">=1.2.3, <2", default-features = false, features = ["std", "derive", "preserve_order"] }

# VRL — Vector Remap Language (embedded transform engine). Built without its
# env and network functions, so no transform can read the host environment or
# reach the network. `enable_system_functions` stays for `parse_etld`, which the
# filebeat program calls, and the compiler drops its `get_hostname`.
# No env or network functions (docs/internals.md, VRL integration). `enable_system_functions`
# stays for `parse_etld`, and the compiler drops its `get_hostname`.
vrl = { version = ">=0.35.0, <0.36", default-features = false, features = [
"compiler",
"stdlib-base",
"enable_crypto_functions",
"enable_system_functions",
] }
# Only for the `TimeZone::Named(Tz::UTC)` constant in the VRL runner —
# same version range as vrl so cargo unifies to one chrono-tz.
# For the `TimeZone::Named(Tz::UTC)` constant in the VRL runner. Same range as vrl so cargo unifies to one chrono-tz.
chrono-tz = ">=0.10.4, <0.11"
# `Value::Timestamp` is a `chrono::DateTime<Utc>`. Same range as vrl so cargo unifies to one chrono.
chrono = ">=0.4.45, <0.5"

# CLI. No `env` feature: scalo's cascade binds env vars.
clap = { version = ">=4.6.7, <5", features = ["derive"] }

# Configuration
figment = { version = ">=0.10.19, <0.11", features = ["env"] }

# Metrics types (Counter, Gauge, Histogram) returned by scalo MetricsManager
metrics = ">=0.24.6, <0.25"

# Serialisation
serde = { version = ">=1.0.229, <2", features = ["derive"] }
serde_json = ">=1.0.154, <2"
sonic-rs = ">=0.5.10, <0.6"

# Error handling
thiserror = ">=2.0.21, <3"
anyhow = ">=1.0.104, <2"

# Logging and tracing
tracing = ">=0.1.44, <0.2"

# Enrichment engine — high-performance lookups
rustc-hash = ">=2.1.3, <3"
arc-swap = ">=1.9.3, <2"
csv = ">=1.4.0, <2"

# Optional enrichment sources
maxminddb = { version = ">=0.32.0, <0.33", optional = true }
rusqlite = { version = ">=0.41.1, <0.42", features = ["bundled"], optional = true }

Expand Down
6 changes: 3 additions & 3 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -10,13 +10,13 @@
check: ## Pre-push validation (quality + test)
hyperi-ci check

quality: ## Run all quality checks
quality:
hyperi-ci run quality

test: ## Run tests
test:
hyperi-ci run test

build: ## Build artifacts
build:
hyperi-ci run build

ci: quality test build ## Run full CI pipeline locally
Loading
Loading