Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 2 additions & 6 deletions .config/nextest.toml
Original file line number Diff line number Diff line change
@@ -1,10 +1,7 @@
# SPDX-License-Identifier: BUSL-1.1
# Copyright (c) 2026 HYPERI PTY LIMITED

# Core is the default selection: unit tests, captured-fixture tests and one
# shared-instance table test per heavy service (integration::shared_services).
# Every other test that needs a real ClickHouse or Kafka is full, which runs
# with `--run-ignored all --ignore-default-filter`. tests/TESTING.md lists both.
# tests/TESTING.md describes the core and full selections.
[profile.default]
default-filter = '''
not (
Expand All @@ -19,8 +16,7 @@ not (
'''
# No retries: a retry keeps a real defect green.
retries = 0
# These tests are legitimately slower than unit tests; warn rather than kill,
# but do not let a hung container run forever.
# Warn every 60 s, and end the test after five periods.
slow-timeout = { period = "60s", terminate-after = 5 }

# Tests that start their own container get a concurrency budget, or cold starts
Expand Down
6 changes: 1 addition & 5 deletions .hyperi-ci.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,6 @@ classification: product

language: rust

# Quality checks
quality:
enabled: true
# osv-scanner does not read deny.toml, so its ignores repeat here.
Expand All @@ -29,7 +28,6 @@ quality:
# Without it the transport-adapter suite is compiled out. helm_contract.rs asserts it stays.
- "default,transport-memory"

# Testing
test:
enabled: true
coverage: true
Expand All @@ -41,7 +39,6 @@ test:
full:
required_for_release: true

# Building
build:
enabled: true
strategies:
Expand All @@ -59,8 +56,7 @@ build:
- x86_64-unknown-linux-gnu
- aarch64-unknown-linux-gnu

# Release (where the artefacts go). The thin chart is assembled from the
# emitted contract and ships beside the image.
# docs/deployment/publishing.md describes the release artefacts.
release:
enabled: true
# Keep `library` at the scalo version in Cargo.toml: the library renders only the contract version its scalo release writes.
Expand Down
4 changes: 1 addition & 3 deletions .tarpaulin.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,4 @@
# Tarpaulin coverage configuration

[default]
# Exclude binary entry point — covered by smoke tests at the integration level.
# main.rs is just CLI bootstrap glue; the meaningful code is in lib.
# main.rs is CLI bootstrap, covered by the smoke tests.
exclude-files = ["src/main.rs"]
16 changes: 8 additions & 8 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -101,8 +101,8 @@ If you have questions about contributing, please open an issue or contact us.
## Run the checks locally before you push

HyperI projects gate every push through CI (lint, format, tests, secret scan,
dependency audit, build). Run the SAME checks locally first so your change lands
green instead of bouncing:
dependency audit, build). Run the SAME checks locally first so your change passes CI
instead of bouncing:

```bash
hyperi-ci check
Expand All @@ -117,7 +117,7 @@ surviving CI.

## CI/CD Workflow

A merge to `main` releases nothing on its own. CI validates the merge, and publishes only when the merge commit carries a `Release: true` trailer (`hyperi-ci push --release` adds it) or someone dispatches a release of `main` (`hyperi-ci release`). A release then runs in order:
A merge to `main` releases nothing by itself. CI validates the merge, and publishes only when the merge commit has a `Release: true` trailer (`hyperi-ci push --release` adds it) or someone dispatches a release of `main` (`hyperi-ci release`). A release then runs in order:

1. semantic-release analyses the commit messages since the last release
2. It picks the next version from the commit types
Expand Down Expand Up @@ -153,20 +153,20 @@ cargo nextest run --features testcontainers # core, Docker for the shared ClickH
cargo nextest run --features testcontainers --run-ignored all --ignore-default-filter # full
```

Green lies three ways. `require_clickhouse!()` returns early, and the test passes, when no ClickHouse is named and the build lacks `--features testcontainers`, or when a named one does not answer outside CI. The e2e and other real-server tests run only in the full tier. And `default = []` compiles a feature-gated test out while the run still reports green. `ci.yml` sets `paths-ignore` for `docs/**` and `**.md`, so a docs-only push runs no jobs, though its pull request still runs CI.
A passing run lies three ways. `require_clickhouse!()` returns early, and the test passes, when no ClickHouse is set and the build lacks `--features testcontainers`, or when a configured one does not answer outside CI. The e2e and other tests that need a server run only in the full tier. And `default = []` compiles a feature-gated test out and the run still reports a pass. `ci.yml` sets `paths-ignore` for `docs/**` and `**.md`, so a docs-only push runs no jobs, though its pull request still runs CI.

### What tends to bite

| Don't | Do | Why |
|-------|----|-----|
| Bump scalo and leave `release.helm.library` behind | Move `release.helm.library` in `.hyperi-ci.yaml` to the same scalo version | The release assembles the chart from the emitted contract on that scalo-service version, and a scalo-service release ships the schema for only the contract version its scalo release writes |
| Gate a test behind a cargo feature without adding it to `.hyperi-ci.yaml` | Add `default,<feature>` to the feature-set list | `default = []` (`Cargo.toml`), so it compiles out of every CI run and CI still reports green. `helm_contract.rs:64` asserts `transport-memory` and `testcontainers` stay listed |
| Gate a test behind a cargo feature without adding it to `.hyperi-ci.yaml` | Add `default,<feature>` to the feature-set list | `default = []` (`Cargo.toml`), so it compiles out of every CI run and CI still reports a pass. `helm_contract.rs` asserts `transport-memory` and `testcontainers` stay listed |
| Inject a nested config key as `DFE_LOADER_SECTION_FIELD` | `DFE_LOADER__SECTION__FIELD` or `DFE_LOADER_SECTION__FIELD` | `__` separates nested keys. A name with single underscores reaches only a key the flat list names (`apply_flat_env` in `src/config/loader.rs`), and anything else matches no field and is dropped. `tests/integration/config_reachability.rs` checks both nested spellings |
| Set `clickhouse.protocol: native` with `insert_format: json_each_row` | `rowbinary` over native, or `http` for JSONEachRow | JSONEachRow is an HTTP body format. `ClickHouseConfig::validate()` refuses the pair by name when the pipeline starts, and `config-check` does not catch it |
| Set `clickhouse.protocol: native` with `insert_format: json_each_row` | `rowbinary` over native, or `http` for JSONEachRow | JSONEachRow is an HTTP body format. `ClickHouseConfig::validate()` rejects the pair by name when the pipeline starts, and `config-check` does not catch it |
| Name `clickhouse.tls.ca_cert_file`, `cert_file`, `key_file` or `skip_verify` | Set `tls.enabled`, mount the CA into the trust store | Only `enabled` reaches a client, and `validate()` rejects the certificate-file and `skip_verify` keys by name |
| Widen the `cel` range past scalo's | Keep it on scalo's range, beside the `cel` entry in `Cargo.toml` | `cel::Program` crosses the scalo boundary. Wider resolves two semver-incompatible `cel` crates and `Program` stops being the same type |
| Take `clickhouse` from git or a `[patch]` | The crates.io release | `deny.toml` allows no git source, and `dfe-clickhouse` builds against the published `clickhouse` only |
| Change a config key and leave `docs/config-schema.*` behind | `dfe-loader config-schema --dir docs` | `test_config_artifacts_do_not_drift` fails on a stale schema, and dfe-engine authors loader configs from the schema the contract carries |
| Change a config key and leave `docs/config-schema.*` behind | `dfe-loader config-schema --dir docs` | `test_config_artifacts_do_not_drift` fails on a stale schema, and dfe-engine authors loader configs from the schema in the contract |

### Where this sits

Expand All @@ -178,6 +178,6 @@ Inbound, declared in `dfe-infra/suite.yaml`:
Outbound, so what a change here can break:

- dfe-loader -> dfe-infra (`image-pin`, lockstep): `dfe-infra/versions.yaml` pins this image and the thin chart released here, and Argo deploys that published chart.
- dfe-loader -> dfe-engine: dfe-engine takes a loader config's defaults, types and descriptions from the `config-schema.json` this repo's contract carries (dfe-engine `src/dfe_engine/appmgmt/contract.py`).
- dfe-loader -> dfe-engine: dfe-engine takes a loader config's defaults, types and descriptions from the `config-schema.json` in this repo's contract (dfe-engine `src/dfe_engine/appmgmt/contract.py`).

Runtime, not build edges: dfe-engine applies the dfe-schemas manifest at boot, creating every destination table before the loader can insert. Producers reach the loader over Kafka or direct gRPC (default topic `main_land`): in the suite dfe-receiver and dfe-fetcher produce `*_land` topics, the transforms produce `*_load`, and dfe-receiver and the transforms push over gRPC.
30 changes: 8 additions & 22 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,6 @@ path = "src/bin/pgo_driver.rs"
required-features = ["pgo-driver"]

[dependencies]
# Async runtime
tokio = { version = ">=1.54.1, <2", features = ["full"] }
tokio-util = { version = ">=0.7.20, <0.8", features = ["rt"] }

Expand All @@ -42,20 +41,16 @@ schemars = { version = ">=1.2.3, <2", default-features = false, features = ["std
# semver-incompatible cels make it two types. In ignoreDeps, so it moves with scalo.
cel = { version = ">=0.14.5, <0.15", features = ["json"] }

# JSON parsing (SIMD-accelerated)
sonic-rs = ">=0.5.10, <0.6"
serde = { version = ">=1.0.229, <2", features = ["derive"] }
serde_json = ">=1.0.154, <2" # Fallback for complex cases
csv = ">=1.4.0, <2" # CSV remap file parsing (ecs-mapper format compatible)
serde_json = ">=1.0.154, <2"
csv = ">=1.4.0, <2"

# Configuration
figment = { version = ">=0.10.19, <0.11", features = ["env"] }
clap = { version = ">=4.6.7, <5", features = ["derive", "env"] }

# Error handling
thiserror = ">=2.0.21, <3"

# Logging & metrics
tracing = ">=0.1.44, <0.2"
metrics = ">=0.24.6, <0.25"

Expand All @@ -80,14 +75,13 @@ hyper-util = { version = ">=0.1.21, <0.2", features = ["client-legacy", "http1",
hyper-rustls = { version = ">=0.27.10, <0.28", default-features = false, features = ["http1", "tls12", "native-tokio", "aws-lc-rs"] }
rustls = { version = ">=0.23.46, <0.24", default-features = false, features = ["std", "tls12", "aws_lc_rs"] }

# Utilities
uuid = { version = ">=1.29.2, <2", features = ["v4", "v7"] }
rustc-hash = ">=2.1.3, <3" # FxHashMap - faster hashing for internal maps
memchr = ">=2.8.3, <3" # SIMD byte scanning for directive parser
rustc-hash = ">=2.1.3, <3"
memchr = ">=2.8.3, <3"
# Kafka auto-discovery filters are regexes in scalo; escape() keeps a DLQ topic
# name literal. Same crate scalo resolves, so cargo unifies to one regex.
regex = ">=1.13.1, <2"
compact_str = ">=0.10.0, <0.11" # Stack-allocated strings for short keys (≤23 bytes)
compact_str = ">=0.10.0, <0.11"
chrono = { version = ">=0.4.45, <0.5", features = ["serde"] }
bytes = ">=1.12.1, <2"
futures = ">=0.3.34, <0.4"
Expand All @@ -99,7 +93,6 @@ indexmap = { version = ">=2.14.2, <3", features = ["serde"] }
rdkafka = { version = ">=0.39.0, <0.40", features = ["dynamic-linking"], optional = true }

[dev-dependencies]
# Testing
criterion = { version = ">=0.8.2, <0.9", features = ["html_reports"] }
proptest = ">=1.12.0, <2"
test-case = ">=3.4.1, <4"
Expand All @@ -115,17 +108,13 @@ scalo = { version = ">=2.14.30, <3", features = ["transport-kafka-test-support"]
# client. Same range as the copy scalo resolves, so cargo unifies to one.
tonic = { version = ">=0.14.9, <0.15", default-features = false }

# Kafka producer for e2e tests (test message injection)
rdkafka = { version = ">=0.39.0, <0.40", features = ["dynamic-linking"] }

# ClickHouse crate -- integration tests need all features
clickhouse = { version = ">=0.15.3, <0.16", features = ["lz4", "time", "rustls-tls", "rustls-tls-native-roots", "inserter"] }
time = ">=0.3.55, <0.4"

# Private CA and server certificates for the TLS ClickHouse tests
rcgen = ">=0.14.10, <0.15"

# Async test utilities
tokio-test = ">=0.4.6, <0.5"

# Sends the loader's tracing output through the test harness's capture. Same
Expand Down Expand Up @@ -244,12 +233,9 @@ duration_suboptimal_units = "allow"
map_unwrap_or = "allow"

[features]
# docs/feature-flags.md describes each feature.
default = []
# Allocator
jemalloc = ["dep:tikv-jemallocator", "dep:tikv-jemalloc-ctl"]
# Transport backends
transport-memory = ["scalo/transport-memory"] # Memory transport for unit tests
# Testing
testcontainers = [] # The tests that run real ClickHouse and Kafka containers (Docker required)
# PGO workload driver (scripts/pgo-workload.sh). Only that binary links rdkafka.
transport-memory = ["scalo/transport-memory"]
testcontainers = []
pgo-driver = ["dep:rdkafka"]
14 changes: 7 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ In a DFE deployment it is the last hop of the core data path, taking what dfe-re
- Columns read from the live table's `system.columns`: matching fields become typed columns and the rest stays in `_json`, with no schema compiled in.
- RowBinary inserts by default, so ClickHouse parses no row text and infers no types. A JSON column's value, `_json` included, still arrives as JSON text for the server to parse.
- Per-table buffers flushed on row count, byte size or age, whichever comes first.
- At-least-once delivery by default: Kafka offsets commit once per flush, never past a row not yet in ClickHouse or the DLQ. `kafka.acknowledgements.enabled: false` commits at receipt instead, and a row only a DLQ could take is dropped and counted in `dfe_loader_rows_lost_total` when no DLQ is working.
- At-least-once delivery by default: Kafka offsets commit once per flush, never past a row not yet in ClickHouse or the DLQ. `kafka.acknowledgements.enabled: false` commits at receipt instead, and a row the DLQ alone could take is dropped and counted in `dfe_loader_rows_lost_total` when no DLQ is working.
- A batch with a bad row is split until only the bad rows are dead-lettered, and a schema drift error refreshes the cached schema and retries.
- Optional GeoIP, reputation and risk columns at load time.

Expand All @@ -38,7 +38,7 @@ cargo build

That prints `[ok] configuration is valid` and the resolved config, and opens no connections. `--config` is a global option, so it goes BEFORE the subcommand -- `config-check --config ...` is rejected by the argument parser.

Moving real data needs a broker or a gRPC sender, and the DFE tables, which the loader never creates. [dfe-docker](https://github.com/hyperi-io/dfe-docker) brings up the whole stack with the tables in place. A running loader serves metrics on `http://localhost:9090/metrics`, with `/livez` and `/readyz` alongside.
Moving data needs a broker or a gRPC sender, and the DFE tables, which the loader never creates. [dfe-docker](https://github.com/hyperi-io/dfe-docker) brings up the whole stack with the tables in place. A running loader serves metrics on `http://localhost:9090/metrics`, with `/livez` and `/readyz` alongside.

## What it does

Expand All @@ -56,18 +56,18 @@ dfe-loader reads records off Kafka or a direct gRPC listener, works out which Cl

JSON is the only payload format. MessagePack, which DFE/XDR 2.0 and 2.1 accepted, was removed in DFE 2.2, and a MessagePack record is dead-lettered as not JSON: the JSON path (SIMD parsing with sonic-rs, zstd on the wire) is fast enough that MessagePack gave no CPU saving.

Failures degrade rather than stall: a batch with a bad row is binary-split so only the bad rows are dead-lettered, a batch whose insert fails is held and retried with backoff while the commit stays below it, and a schema drift error invalidates the cached schema and retries.
A batch with a bad row is binary-split so only the bad rows are dead-lettered. A failed insert's batch is held and retried with backoff, with its offsets uncommitted. A schema drift error invalidates the cached schema before the retry.

## Configuration essentials

One YAML file, named by `--config` or `DFE_LOADER_CONFIG`. With neither set the loader reads `./config.yaml` or `./config.yml` if one exists, but only a named file is hot-reloaded. The chart mounts it at `/etc/dfe/loader.yaml`. `config.example.yaml` is an annotated example, `docs/config-schema.yaml` lists every key with its default, and every setting has a default, so an empty config boots.
One YAML file, from `--config` or `DFE_LOADER_CONFIG`. With neither set the loader reads `./config.yaml` or `./config.yml` if one exists, but only a file given that way is hot-reloaded. The chart mounts it at `/etc/dfe/loader.yaml`. `config.example.yaml` is an annotated example, `docs/config-schema.yaml` lists every key with its default, and every setting has a default, so an empty config boots.

Environment overrides use the `DFE_LOADER` prefix in two forms, and the difference matters: `DFE_LOADER_KAFKA_GROUP_ID` is the flat single-underscore set, a fixed contract with dfe-engine, while `DFE_LOADER__CLICKHOUSE__DATABASE` uses double underscores to nest into any config key.
Environment overrides use the `DFE_LOADER` prefix in two forms, and the difference matters: `DFE_LOADER_KAFKA_GROUP_ID` is the flat single-underscore set, a fixed contract with dfe-engine, and `DFE_LOADER__CLICKHOUSE__DATABASE` uses double underscores to nest into any config key.

| Setting | Default | Note |
|---------|---------|------|
| `clickhouse.hosts` | `localhost:8123` | HTTP port family |
| `clickhouse.protocol` | `http` | `native` (9000, 9440 with TLS) also works, for `rowbinary` inserts |
| `clickhouse.protocol` | `http` | `native` (9000, 9440 with TLS) works for `rowbinary` inserts |
| `clickhouse.database` | `dfe` | The connection database, separate from `routing.default_db` |
| `clickhouse.insert_format` | `rowbinary` | `jsoneachrow` is the fallback |
| `clickhouse.tls.enabled` | unset | The ONLY honoured TLS key. Mount a private CA into the container trust store rather than naming cert files |
Expand Down Expand Up @@ -99,7 +99,7 @@ The suite's published docs are at <https://dfe.docs.hyperi.io>, and this reposit

## Help and contributing

How to work in this repo, the commands that prove a change and what tends to bite are in [CONTRIBUTING.md](CONTRIBUTING.md). A question that is not a bug goes in a new issue.
How to work in this repo, the commands that prove a change and what tends to bite are in [CONTRIBUTING.md](CONTRIBUTING.md). A question that is not a bug goes in an issue.

Open bugs: [this repo](https://github.com/hyperi-io/dfe-loader/issues?q=is%3Aopen+is%3Aissue+type%3ABug+-label%3Abacklog+-label%3Anext-minor) | [whole DFE suite](https://github.com/search?type=issues&q=is%3Aopen+is%3Aissue+type%3ABug+-label%3Abacklog+-label%3Anext-minor+repo%3Ahyperi-io%2Fdfe-infra+repo%3Ahyperi-io%2Fdfe-docker+repo%3Ahyperi-io%2Fdfe-deploy+repo%3Ahyperi-io%2Fdfe-engine+repo%3Ahyperi-io%2Fdfe-loader+repo%3Ahyperi-io%2Fdfe-receiver+repo%3Ahyperi-io%2Fdfe-fetcher+repo%3Ahyperi-io%2Fdfe-archiver+repo%3Ahyperi-io%2Fdfe-transform-vrl+repo%3Ahyperi-io%2Fdfe-transform-vector+repo%3Ahyperi-io%2Fdfe-transform-elastic+repo%3Ahyperi-io%2Fdfe-ui+repo%3Ahyperi-io%2Fdfe-hyperdx+repo%3Ahyperi-io%2Fscalo-rs+repo%3Ahyperi-io%2Fscalo-py+repo%3Ahyperi-io%2Flogreducer+repo%3Ahyperi-io%2Fdfe-schemas+repo%3Ahyperi-io%2Fvector-vrl+repo%3Ahyperi-io%2Fculvert+repo%3Ahyperi-io%2Fdfe-clickhouse-rs+repo%3Ahyperi-io%2Ffactbook-rs)

Expand Down
Loading
Loading