diff --git a/.config/nextest.toml b/.config/nextest.toml index e3fcf06..da66096 100644 --- a/.config/nextest.toml +++ b/.config/nextest.toml @@ -1,10 +1,7 @@ # SPDX-License-Identifier: BUSL-1.1 # Copyright (c) 2026 HYPERI PTY LIMITED -# Core is the default selection: unit tests, captured-fixture tests and one -# shared-instance table test per heavy service (integration::shared_services). -# Every other test that needs a real ClickHouse or Kafka is full, which runs -# with `--run-ignored all --ignore-default-filter`. tests/TESTING.md lists both. +# tests/TESTING.md describes the core and full selections. [profile.default] default-filter = ''' not ( @@ -19,8 +16,7 @@ not ( ''' # No retries: a retry keeps a real defect green. retries = 0 -# These tests are legitimately slower than unit tests; warn rather than kill, -# but do not let a hung container run forever. +# Warn every 60 s, and end the test after five periods. slow-timeout = { period = "60s", terminate-after = 5 } # Tests that start their own container get a concurrency budget, or cold starts diff --git a/.hyperi-ci.yaml b/.hyperi-ci.yaml index ab69f49..86cdca7 100644 --- a/.hyperi-ci.yaml +++ b/.hyperi-ci.yaml @@ -8,7 +8,6 @@ classification: product language: rust -# Quality checks quality: enabled: true # osv-scanner does not read deny.toml, so its ignores repeat here. @@ -29,7 +28,6 @@ quality: # Without it the transport-adapter suite is compiled out. helm_contract.rs asserts it stays. - "default,transport-memory" -# Testing test: enabled: true coverage: true @@ -41,7 +39,6 @@ test: full: required_for_release: true -# Building build: enabled: true strategies: @@ -59,8 +56,7 @@ build: - x86_64-unknown-linux-gnu - aarch64-unknown-linux-gnu -# Release (where the artefacts go). The thin chart is assembled from the -# emitted contract and ships beside the image. +# docs/deployment/publishing.md describes the release artefacts. release: enabled: true # Keep `library` at the scalo version in Cargo.toml: the library renders only the contract version its scalo release writes. diff --git a/.tarpaulin.toml b/.tarpaulin.toml index c904ab3..4715db6 100644 --- a/.tarpaulin.toml +++ b/.tarpaulin.toml @@ -1,6 +1,4 @@ -# Tarpaulin coverage configuration [default] -# Exclude binary entry point — covered by smoke tests at the integration level. -# main.rs is just CLI bootstrap glue; the meaningful code is in lib. +# main.rs is CLI bootstrap, covered by the smoke tests. exclude-files = ["src/main.rs"] diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 8549671..fa1827a 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -101,8 +101,8 @@ If you have questions about contributing, please open an issue or contact us. ## Run the checks locally before you push HyperI projects gate every push through CI (lint, format, tests, secret scan, -dependency audit, build). Run the SAME checks locally first so your change lands -green instead of bouncing: +dependency audit, build). Run the SAME checks locally first so your change passes CI +instead of bouncing: ```bash hyperi-ci check @@ -117,7 +117,7 @@ surviving CI. ## CI/CD Workflow -A merge to `main` releases nothing on its own. CI validates the merge, and publishes only when the merge commit carries a `Release: true` trailer (`hyperi-ci push --release` adds it) or someone dispatches a release of `main` (`hyperi-ci release`). A release then runs in order: +A merge to `main` releases nothing by itself. CI validates the merge, and publishes only when the merge commit has a `Release: true` trailer (`hyperi-ci push --release` adds it) or someone dispatches a release of `main` (`hyperi-ci release`). A release then runs in order: 1. semantic-release analyses the commit messages since the last release 2. It picks the next version from the commit types @@ -153,20 +153,20 @@ cargo nextest run --features testcontainers # core, Docker for the shared ClickH cargo nextest run --features testcontainers --run-ignored all --ignore-default-filter # full ``` -Green lies three ways. `require_clickhouse!()` returns early, and the test passes, when no ClickHouse is named and the build lacks `--features testcontainers`, or when a named one does not answer outside CI. The e2e and other real-server tests run only in the full tier. And `default = []` compiles a feature-gated test out while the run still reports green. `ci.yml` sets `paths-ignore` for `docs/**` and `**.md`, so a docs-only push runs no jobs, though its pull request still runs CI. +A passing run lies three ways. `require_clickhouse!()` returns early, and the test passes, when no ClickHouse is set and the build lacks `--features testcontainers`, or when a configured one does not answer outside CI. The e2e and other tests that need a server run only in the full tier. And `default = []` compiles a feature-gated test out and the run still reports a pass. `ci.yml` sets `paths-ignore` for `docs/**` and `**.md`, so a docs-only push runs no jobs, though its pull request still runs CI. ### What tends to bite | Don't | Do | Why | |-------|----|-----| | Bump scalo and leave `release.helm.library` behind | Move `release.helm.library` in `.hyperi-ci.yaml` to the same scalo version | The release assembles the chart from the emitted contract on that scalo-service version, and a scalo-service release ships the schema for only the contract version its scalo release writes | -| Gate a test behind a cargo feature without adding it to `.hyperi-ci.yaml` | Add `default,` to the feature-set list | `default = []` (`Cargo.toml`), so it compiles out of every CI run and CI still reports green. `helm_contract.rs:64` asserts `transport-memory` and `testcontainers` stay listed | +| Gate a test behind a cargo feature without adding it to `.hyperi-ci.yaml` | Add `default,` to the feature-set list | `default = []` (`Cargo.toml`), so it compiles out of every CI run and CI still reports a pass. `helm_contract.rs` asserts `transport-memory` and `testcontainers` stay listed | | Inject a nested config key as `DFE_LOADER_SECTION_FIELD` | `DFE_LOADER__SECTION__FIELD` or `DFE_LOADER_SECTION__FIELD` | `__` separates nested keys. A name with single underscores reaches only a key the flat list names (`apply_flat_env` in `src/config/loader.rs`), and anything else matches no field and is dropped. `tests/integration/config_reachability.rs` checks both nested spellings | -| Set `clickhouse.protocol: native` with `insert_format: json_each_row` | `rowbinary` over native, or `http` for JSONEachRow | JSONEachRow is an HTTP body format. `ClickHouseConfig::validate()` refuses the pair by name when the pipeline starts, and `config-check` does not catch it | +| Set `clickhouse.protocol: native` with `insert_format: json_each_row` | `rowbinary` over native, or `http` for JSONEachRow | JSONEachRow is an HTTP body format. `ClickHouseConfig::validate()` rejects the pair by name when the pipeline starts, and `config-check` does not catch it | | Name `clickhouse.tls.ca_cert_file`, `cert_file`, `key_file` or `skip_verify` | Set `tls.enabled`, mount the CA into the trust store | Only `enabled` reaches a client, and `validate()` rejects the certificate-file and `skip_verify` keys by name | | Widen the `cel` range past scalo's | Keep it on scalo's range, beside the `cel` entry in `Cargo.toml` | `cel::Program` crosses the scalo boundary. Wider resolves two semver-incompatible `cel` crates and `Program` stops being the same type | | Take `clickhouse` from git or a `[patch]` | The crates.io release | `deny.toml` allows no git source, and `dfe-clickhouse` builds against the published `clickhouse` only | -| Change a config key and leave `docs/config-schema.*` behind | `dfe-loader config-schema --dir docs` | `test_config_artifacts_do_not_drift` fails on a stale schema, and dfe-engine authors loader configs from the schema the contract carries | +| Change a config key and leave `docs/config-schema.*` behind | `dfe-loader config-schema --dir docs` | `test_config_artifacts_do_not_drift` fails on a stale schema, and dfe-engine authors loader configs from the schema in the contract | ### Where this sits @@ -178,6 +178,6 @@ Inbound, declared in `dfe-infra/suite.yaml`: Outbound, so what a change here can break: - dfe-loader -> dfe-infra (`image-pin`, lockstep): `dfe-infra/versions.yaml` pins this image and the thin chart released here, and Argo deploys that published chart. -- dfe-loader -> dfe-engine: dfe-engine takes a loader config's defaults, types and descriptions from the `config-schema.json` this repo's contract carries (dfe-engine `src/dfe_engine/appmgmt/contract.py`). +- dfe-loader -> dfe-engine: dfe-engine takes a loader config's defaults, types and descriptions from the `config-schema.json` in this repo's contract (dfe-engine `src/dfe_engine/appmgmt/contract.py`). Runtime, not build edges: dfe-engine applies the dfe-schemas manifest at boot, creating every destination table before the loader can insert. Producers reach the loader over Kafka or direct gRPC (default topic `main_land`): in the suite dfe-receiver and dfe-fetcher produce `*_land` topics, the transforms produce `*_load`, and dfe-receiver and the transforms push over gRPC. diff --git a/Cargo.toml b/Cargo.toml index 16c895e..4057c27 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -24,7 +24,6 @@ path = "src/bin/pgo_driver.rs" required-features = ["pgo-driver"] [dependencies] -# Async runtime tokio = { version = ">=1.54.1, <2", features = ["full"] } tokio-util = { version = ">=0.7.20, <0.8", features = ["rt"] } @@ -42,20 +41,16 @@ schemars = { version = ">=1.2.3, <2", default-features = false, features = ["std # semver-incompatible cels make it two types. In ignoreDeps, so it moves with scalo. cel = { version = ">=0.14.5, <0.15", features = ["json"] } -# JSON parsing (SIMD-accelerated) sonic-rs = ">=0.5.10, <0.6" serde = { version = ">=1.0.229, <2", features = ["derive"] } -serde_json = ">=1.0.154, <2" # Fallback for complex cases -csv = ">=1.4.0, <2" # CSV remap file parsing (ecs-mapper format compatible) +serde_json = ">=1.0.154, <2" +csv = ">=1.4.0, <2" -# Configuration figment = { version = ">=0.10.19, <0.11", features = ["env"] } clap = { version = ">=4.6.7, <5", features = ["derive", "env"] } -# Error handling thiserror = ">=2.0.21, <3" -# Logging & metrics tracing = ">=0.1.44, <0.2" metrics = ">=0.24.6, <0.25" @@ -80,14 +75,13 @@ hyper-util = { version = ">=0.1.21, <0.2", features = ["client-legacy", "http1", hyper-rustls = { version = ">=0.27.10, <0.28", default-features = false, features = ["http1", "tls12", "native-tokio", "aws-lc-rs"] } rustls = { version = ">=0.23.46, <0.24", default-features = false, features = ["std", "tls12", "aws_lc_rs"] } -# Utilities uuid = { version = ">=1.29.2, <2", features = ["v4", "v7"] } -rustc-hash = ">=2.1.3, <3" # FxHashMap - faster hashing for internal maps -memchr = ">=2.8.3, <3" # SIMD byte scanning for directive parser +rustc-hash = ">=2.1.3, <3" +memchr = ">=2.8.3, <3" # Kafka auto-discovery filters are regexes in scalo; escape() keeps a DLQ topic # name literal. Same crate scalo resolves, so cargo unifies to one regex. regex = ">=1.13.1, <2" -compact_str = ">=0.10.0, <0.11" # Stack-allocated strings for short keys (≤23 bytes) +compact_str = ">=0.10.0, <0.11" chrono = { version = ">=0.4.45, <0.5", features = ["serde"] } bytes = ">=1.12.1, <2" futures = ">=0.3.34, <0.4" @@ -99,7 +93,6 @@ indexmap = { version = ">=2.14.2, <3", features = ["serde"] } rdkafka = { version = ">=0.39.0, <0.40", features = ["dynamic-linking"], optional = true } [dev-dependencies] -# Testing criterion = { version = ">=0.8.2, <0.9", features = ["html_reports"] } proptest = ">=1.12.0, <2" test-case = ">=3.4.1, <4" @@ -115,17 +108,13 @@ scalo = { version = ">=2.14.30, <3", features = ["transport-kafka-test-support"] # client. Same range as the copy scalo resolves, so cargo unifies to one. tonic = { version = ">=0.14.9, <0.15", default-features = false } -# Kafka producer for e2e tests (test message injection) rdkafka = { version = ">=0.39.0, <0.40", features = ["dynamic-linking"] } -# ClickHouse crate -- integration tests need all features clickhouse = { version = ">=0.15.3, <0.16", features = ["lz4", "time", "rustls-tls", "rustls-tls-native-roots", "inserter"] } time = ">=0.3.55, <0.4" -# Private CA and server certificates for the TLS ClickHouse tests rcgen = ">=0.14.10, <0.15" -# Async test utilities tokio-test = ">=0.4.6, <0.5" # Sends the loader's tracing output through the test harness's capture. Same @@ -244,12 +233,9 @@ duration_suboptimal_units = "allow" map_unwrap_or = "allow" [features] +# docs/feature-flags.md describes each feature. default = [] -# Allocator jemalloc = ["dep:tikv-jemallocator", "dep:tikv-jemalloc-ctl"] -# Transport backends -transport-memory = ["scalo/transport-memory"] # Memory transport for unit tests -# Testing -testcontainers = [] # The tests that run real ClickHouse and Kafka containers (Docker required) -# PGO workload driver (scripts/pgo-workload.sh). Only that binary links rdkafka. +transport-memory = ["scalo/transport-memory"] +testcontainers = [] pgo-driver = ["dep:rdkafka"] diff --git a/README.md b/README.md index 2b1c435..b12242a 100644 --- a/README.md +++ b/README.md @@ -23,7 +23,7 @@ In a DFE deployment it is the last hop of the core data path, taking what dfe-re - Columns read from the live table's `system.columns`: matching fields become typed columns and the rest stays in `_json`, with no schema compiled in. - RowBinary inserts by default, so ClickHouse parses no row text and infers no types. A JSON column's value, `_json` included, still arrives as JSON text for the server to parse. - Per-table buffers flushed on row count, byte size or age, whichever comes first. -- At-least-once delivery by default: Kafka offsets commit once per flush, never past a row not yet in ClickHouse or the DLQ. `kafka.acknowledgements.enabled: false` commits at receipt instead, and a row only a DLQ could take is dropped and counted in `dfe_loader_rows_lost_total` when no DLQ is working. +- At-least-once delivery by default: Kafka offsets commit once per flush, never past a row not yet in ClickHouse or the DLQ. `kafka.acknowledgements.enabled: false` commits at receipt instead, and a row the DLQ alone could take is dropped and counted in `dfe_loader_rows_lost_total` when no DLQ is working. - A batch with a bad row is split until only the bad rows are dead-lettered, and a schema drift error refreshes the cached schema and retries. - Optional GeoIP, reputation and risk columns at load time. @@ -38,7 +38,7 @@ cargo build That prints `[ok] configuration is valid` and the resolved config, and opens no connections. `--config` is a global option, so it goes BEFORE the subcommand -- `config-check --config ...` is rejected by the argument parser. -Moving real data needs a broker or a gRPC sender, and the DFE tables, which the loader never creates. [dfe-docker](https://github.com/hyperi-io/dfe-docker) brings up the whole stack with the tables in place. A running loader serves metrics on `http://localhost:9090/metrics`, with `/livez` and `/readyz` alongside. +Moving data needs a broker or a gRPC sender, and the DFE tables, which the loader never creates. [dfe-docker](https://github.com/hyperi-io/dfe-docker) brings up the whole stack with the tables in place. A running loader serves metrics on `http://localhost:9090/metrics`, with `/livez` and `/readyz` alongside. ## What it does @@ -56,18 +56,18 @@ dfe-loader reads records off Kafka or a direct gRPC listener, works out which Cl JSON is the only payload format. MessagePack, which DFE/XDR 2.0 and 2.1 accepted, was removed in DFE 2.2, and a MessagePack record is dead-lettered as not JSON: the JSON path (SIMD parsing with sonic-rs, zstd on the wire) is fast enough that MessagePack gave no CPU saving. -Failures degrade rather than stall: a batch with a bad row is binary-split so only the bad rows are dead-lettered, a batch whose insert fails is held and retried with backoff while the commit stays below it, and a schema drift error invalidates the cached schema and retries. +A batch with a bad row is binary-split so only the bad rows are dead-lettered. A failed insert's batch is held and retried with backoff, with its offsets uncommitted. A schema drift error invalidates the cached schema before the retry. ## Configuration essentials -One YAML file, named by `--config` or `DFE_LOADER_CONFIG`. With neither set the loader reads `./config.yaml` or `./config.yml` if one exists, but only a named file is hot-reloaded. The chart mounts it at `/etc/dfe/loader.yaml`. `config.example.yaml` is an annotated example, `docs/config-schema.yaml` lists every key with its default, and every setting has a default, so an empty config boots. +One YAML file, from `--config` or `DFE_LOADER_CONFIG`. With neither set the loader reads `./config.yaml` or `./config.yml` if one exists, but only a file given that way is hot-reloaded. The chart mounts it at `/etc/dfe/loader.yaml`. `config.example.yaml` is an annotated example, `docs/config-schema.yaml` lists every key with its default, and every setting has a default, so an empty config boots. -Environment overrides use the `DFE_LOADER` prefix in two forms, and the difference matters: `DFE_LOADER_KAFKA_GROUP_ID` is the flat single-underscore set, a fixed contract with dfe-engine, while `DFE_LOADER__CLICKHOUSE__DATABASE` uses double underscores to nest into any config key. +Environment overrides use the `DFE_LOADER` prefix in two forms, and the difference matters: `DFE_LOADER_KAFKA_GROUP_ID` is the flat single-underscore set, a fixed contract with dfe-engine, and `DFE_LOADER__CLICKHOUSE__DATABASE` uses double underscores to nest into any config key. | Setting | Default | Note | |---------|---------|------| | `clickhouse.hosts` | `localhost:8123` | HTTP port family | -| `clickhouse.protocol` | `http` | `native` (9000, 9440 with TLS) also works, for `rowbinary` inserts | +| `clickhouse.protocol` | `http` | `native` (9000, 9440 with TLS) works for `rowbinary` inserts | | `clickhouse.database` | `dfe` | The connection database, separate from `routing.default_db` | | `clickhouse.insert_format` | `rowbinary` | `jsoneachrow` is the fallback | | `clickhouse.tls.enabled` | unset | The ONLY honoured TLS key. Mount a private CA into the container trust store rather than naming cert files | @@ -99,7 +99,7 @@ The suite's published docs are at , and this reposit ## Help and contributing -How to work in this repo, the commands that prove a change and what tends to bite are in [CONTRIBUTING.md](CONTRIBUTING.md). A question that is not a bug goes in a new issue. +How to work in this repo, the commands that prove a change and what tends to bite are in [CONTRIBUTING.md](CONTRIBUTING.md). A question that is not a bug goes in an issue. Open bugs: [this repo](https://github.com/hyperi-io/dfe-loader/issues?q=is%3Aopen+is%3Aissue+type%3ABug+-label%3Abacklog+-label%3Anext-minor) | [whole DFE suite](https://github.com/search?type=issues&q=is%3Aopen+is%3Aissue+type%3ABug+-label%3Abacklog+-label%3Anext-minor+repo%3Ahyperi-io%2Fdfe-infra+repo%3Ahyperi-io%2Fdfe-docker+repo%3Ahyperi-io%2Fdfe-deploy+repo%3Ahyperi-io%2Fdfe-engine+repo%3Ahyperi-io%2Fdfe-loader+repo%3Ahyperi-io%2Fdfe-receiver+repo%3Ahyperi-io%2Fdfe-fetcher+repo%3Ahyperi-io%2Fdfe-archiver+repo%3Ahyperi-io%2Fdfe-transform-vrl+repo%3Ahyperi-io%2Fdfe-transform-vector+repo%3Ahyperi-io%2Fdfe-transform-elastic+repo%3Ahyperi-io%2Fdfe-ui+repo%3Ahyperi-io%2Fdfe-hyperdx+repo%3Ahyperi-io%2Fscalo-rs+repo%3Ahyperi-io%2Fscalo-py+repo%3Ahyperi-io%2Flogreducer+repo%3Ahyperi-io%2Fdfe-schemas+repo%3Ahyperi-io%2Fvector-vrl+repo%3Ahyperi-io%2Fculvert+repo%3Ahyperi-io%2Fdfe-clickhouse-rs+repo%3Ahyperi-io%2Ffactbook-rs) diff --git a/config.dev.yaml b/config.dev.yaml index 9a3b437..433e7be 100644 --- a/config.dev.yaml +++ b/config.dev.yaml @@ -25,15 +25,11 @@ clickhouse: - api_events routing: - # Fields to extract table name (first match wins, dot notation for nested) table_fields: - event_category - tags.event_category - # Database for every org org_routes does not list default_db: common - # Default table if no table_field matches default_table: common - # Optional: map extracted _source values to destination table names source_to_table: auth: auth_events api: api_events @@ -43,21 +39,18 @@ routing: buffer: flush_bytes: 67108864 # 64 MiB, a ceiling for oversized rows: rows trigger first - flush_rows: 1000 # Fewer rows for faster flushes - flush_age_secs: 2 # Faster flushes for dev + flush_rows: 1000 + flush_age_secs: 2 -# LOW MEMORY settings for development memory: - limit_bytes: 67108864 # 64MB fixed cap + limit_bytes: 67108864 pressure_threshold: 0.8 -# Batch processing engine (SIMD parse, pre-route, parallelism) -# Defaults are sensible — only override if tuning. batch_processing: routing_field: "_table" # SIMD pre-route: extract _table before full parse - max_chunk_size: 10000 # Messages per rayon chunk + max_chunk_size: 10000 parse_error_action: dlq # dlq | skip | fail_batch - known_fields: # Fields to pre-extract for fast routing + known_fields: - "_table" - "_timestamp" - "host" @@ -68,8 +61,8 @@ metrics: address: "0.0.0.0:9090" logging: - level: debug # Verbose for dev - format: text # Human-readable for dev + level: debug + format: text timestamp_dq: enabled: true @@ -78,5 +71,5 @@ timestamp_dq: correct_known_bad: true schema: - cache_ttl_secs: 60 # Shorter cache for dev iteration + cache_ttl_secs: 60 refresh_on_error: true diff --git a/deny.toml b/deny.toml index 192ccce..b0d506a 100644 --- a/deny.toml +++ b/deny.toml @@ -1,4 +1,3 @@ -# cargo-deny configuration # Run: cargo deny check [graph] diff --git a/docker-compose.dev.yaml b/docker-compose.dev.yaml index e87fd44..003cf8e 100644 --- a/docker-compose.dev.yaml +++ b/docker-compose.dev.yaml @@ -9,15 +9,14 @@ # docker-compose -f docker-compose.dev.yaml down -v services: - # ClickHouse for testing inserts clickhouse: # Matches CLICKHOUSE_TAG in tests/common/containers.rs and dfe-infra's # services.clickhouse-version. image: clickhouse/clickhouse-server:26.3.42.3@sha256:21d572843e59539c7d100286b6f5a6053c341fe4b33c2d7b74b1ff7cb24c5399 container_name: clickhouse-dev ports: - - "9000:9000" # Native protocol - - "8123:8123" # HTTP protocol + - "9000:9000" + - "8123:8123" environment: CLICKHOUSE_DB: default CLICKHOUSE_USER: default diff --git a/docs/README.md b/docs/README.md index 96cf454..4838e2a 100644 --- a/docs/README.md +++ b/docs/README.md @@ -12,13 +12,7 @@ Loads records from Kafka or a gRPC push into ClickHouse. Point it at a topic, give it a routing rule, and it parses, routes, promotes schema columns, enriches, buffers per table, and inserts -- RowBinary by default so ClickHouse parses no row text, with at-least-once Kafka delivery. -This is the index. Read [architecture.md](architecture.md) for the 10,000-foot -view, [clickhouse/](clickhouse/) for the insert layer, and -[pipeline/](pipeline/) for the hot path. - ---- - -## What dfe-loader does for you +[architecture.md](architecture.md) covers the loader's layers, [clickhouse/](clickhouse/) the insert layer and [pipeline/](pipeline/) the hot path. | You give it | It handles | You don't write | |-------------|------------|-----------------| @@ -27,52 +21,52 @@ view, [clickhouse/](clickhouse/) for the insert layer, and | A ClickHouse table | Schema reflected from `system.columns`, fields promoted to typed columns, the rest kept in `_json` | A schema mapping, an ORM, a migration | | `insert_format` (default RowBinary) | Dynamic `Map` -> RowBinary over HTTP, or Native blocks over the native protocol, with a JSONEachRow fallback over HTTP | A serialiser, a type encoder, a wire format | | `capture_mode` (full/raw_only/json_only/extracted_only) | `_json` and `_raw` population per table, overridable per-table and per-DDL | Per-table capture logic | -| Enrichment toggles (geoip/reputation/risk) | Flat enriched columns injected on the promoted row only | Lookup, caching, CIDR matching | +| Enrichment toggles (geoip/reputation/risk) | Flat enriched columns on the promoted row, never in `_json` | Lookup, caching, CIDR matching | | Nothing extra | Per-table buffering, batch salvage, held-batch retry, schema-cache recovery, one offset commit per flush cycle | The resilience layer | -Everything else in these docs is "and here is how the pieces work". - ---- - ## 10,000-foot view +Every record takes the same six steps whichever transport brought it, and normally leaves as a row in a batched ClickHouse insert or as a dead letter. + ```mermaid flowchart TB - subgraph Ingress["Transport (scalo)"] - K["Kafka"] - G["gRPC"] - end + accTitle: L3 Components -- dfe-loader + accDescr: Records arrive from Kafka topics or gRPC senders. Inside the loader each record is parsed, routed to a table, has its columns extracted, is enriched and is buffered per table, and the buffers are inserted into ClickHouse in batches. Rows ClickHouse refuses for good and records that fail processing go to the DLQ. - subgraph Hot["Hot path (per message)"] - P["JSON parse
sonic-rs -> Arc[u8]"] - R["Route (pre-flatten)
db.table"] - X["Extract
header + schema-promoted cols"] - E["Enrich
GeoIP / reputation / risk"] - B["Per-table buffer
rows + raw + offsets"] - end + kafka[("Kafka topics")] + grpc["gRPC senders"] - subgraph Sink["ClickHouse insert (dfe_clickhouse::dynamic::DynamicInsert)"] - CE["DynamicRow encoder
Map + schema -> binary"] - HTTP["HTTP: insert_formatted_with
FORMAT RowBinaryWithNamesAndTypes"] - NAT["native: insert_native
FORMAT Native"] - JF["HTTP: insert_formatted_with
FORMAT JSONEachRow (fallback)"] + subgraph loader["dfe-loader"] + direction LR + parse["Parse JSON"] --> route[["Route"]] --> extract["Extract columns"] --> enrich[["Enrich"]] --> buffer["Buffer per table"] --> insert[["Insert"]] end - CH[(ClickHouse)] - DLQ["DLQ (Kafka primary, file fallback)"] - - K & G --> P --> R --> X --> E --> B - B -->|RowBinary default| CE - CE --> HTTP --> CH - CE --> NAT --> CH - B -->|insert_format=json_each_row| JF --> CH - B -.->|salvaged bad rows| DLQ - CH -.->|success| Commit["One Kafka commit per flush cycle"] + clickhouse[("ClickHouse")] + dlq[("DLQ")] + + kafka -->|records| loader + grpc -->|records| loader + loader -->|rows| clickhouse + loader -.->|dead letters| dlq + + classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A + classDef store fill:#DDF3EC,stroke:#00785A,stroke-width:1px,color:#0B3327 + classDef async fill:#F6E6EF,stroke:#A8517F,stroke-width:1px,color:#3D1529 + classDef error fill:#FBE3D6,stroke:#C2410C,stroke-width:1px,color:#4A1A05 + classDef ext fill:#EEF0F2,stroke:#6B7280,stroke-width:1px,stroke-dasharray:4 3,color:#1F2937 + classDef group fill:none,stroke:#64748B,stroke-width:1px,stroke-dasharray:2 4 + class kafka async + class grpc ext + class clickhouse store + class dlq error + class loader group ``` ---- +*Figure 1. L3 Components -- dfe-loader. Double-bordered steps have a diagram each.* Up: top of dfe-loader. Down: [L4 Detail -- Loader table choice](pipeline/routing.md), [L4 Detail -- Enrichment](pipeline/enrichment.md), [L4 Detail -- Insert layer](clickhouse/README.md). + +With `kafka.acknowledgements.enabled` (the default) Kafka offsets commit once per flush cycle, never past a row not yet placed. A pre-route filter, a dead letter no DLQ can hold and a revoked partition's rows are the exceptions, each counted, and [operations.md](operations.md) covers them. -## Where things live +The docs, by area: | Area | Doc | Covers | |------|-----|--------| @@ -86,8 +80,4 @@ flowchart TB | Deployment | [deployment/](deployment/) | Container + chart publishing, schemas | | Performance | [performance/](performance/) | Hot-path optimisations, parser-selection history, PGO/BOLT | ---- - -## The client dependency, in one line - dfe-loader inserts through the official `clickhouse` crate from crates.io, with `dfe-clickhouse` reading the server's exception code out of a rejection. The dynamic RowBinary layer the `clickhouse` crate keeps out of scope -- runtime `Map` encoding -- comes from `dfe-clickhouse`'s `dynamic` module. See [clickhouse/clickhouse-ext.md](clickhouse/clickhouse-ext.md). diff --git a/docs/architecture.md b/docs/architecture.md index df65a2d..0ec7a1a 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -19,39 +19,50 @@ promote only schema columns, keep the full payload as a reference-counted ## Layers +The code sits in four layers. Orchestration calls the transform and the insert layers, and both call into the ClickHouse layer, never back up. + ```mermaid -flowchart TB - subgraph L4["L4 -- Orchestration"] - ORC["Orchestrator
wiring, shutdown, hot-reload"] - PROC["MessageProcessor
per-message pipeline"] - end - subgraph L3["L3 -- Transform"] - ROUTE["Router (sonic-rs)"] - EXTRACT["HeaderExtractor"] - ENRICH["Enrichment (geoip/rep/risk)"] +flowchart LR + accTitle: L4 Detail -- Loader layers + accDescr: Orchestration holds the Orchestrator and the MessageProcessor. It calls the transform layer, the Router, the HeaderExtractor and the EnrichmentPipeline, and the buffer and insert layer, the BufferManager and the Inserter. Both call the ClickHouse layer, the query client, the schema cache, the dynamic insert and the ChClient. + + subgraph orchestration["1 Orchestration"] + direction TB + orchestrator["Orchestrator"] ~~~ processor["MessageProcessor"] end - subgraph L2["L2 -- Buffer + insert"] - BUF["BufferManager
per-table rows + raw + offsets"] - INS["Inserter
format dispatch, retry, salvage"] + subgraph transform["2 Transform"] + direction TB + router["Router"] ~~~ extractor["HeaderExtractor"] ~~~ enrichment["EnrichmentPipeline"] end - subgraph L1["L1 -- ClickHouse layer"] - QC["ClickHouseQueryClient
schema, comments, health"] - EXT["dfe_clickhouse::dynamic
ParsedType, encoder, DynamicInsert"] - SC["SchemaCache"] + subgraph insertlayer["3 Buffer and insert"] + direction TB + buffers["BufferManager"] ~~~ inserter["Inserter"] end - subgraph Client["crates.io clickhouse + dfe-clickhouse"] - CLIENT["ChClient: clickhouse::Client (HTTP) or dfe_clickhouse::TcpClient (native)"] - EXC["dfe_clickhouse::ServerException"] + subgraph chlayer["4 ClickHouse layer"] + direction TB + queryclient["ClickHouseQueryClient"] ~~~ schemacache["SchemaCache"] ~~~ dynamic["dfe_clickhouse
dynamic"] ~~~ chclient["ChClient"] end - - ORC --> PROC --> L3 --> BUF --> INS - INS --> EXT --> CLIENT - INS --> QC --> CLIENT - INS --> EXC - EXT --> SC - QC --> SC + orchestration --> transform + orchestration --> insertlayer + transform --> chlayer + insertlayer --> chlayer + + classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A + classDef group fill:none,stroke:#64748B,stroke-width:1px,stroke-dasharray:2 4 + class orchestration,transform,insertlayer,chlayer group ``` +*Figure 1. L4 Detail -- Loader layers. Each layer calls only the layers to its right.* Up: [L3 Components -- dfe-loader](README.md). + +| Component | What it does | +|---|---| +| `Orchestrator` | Wiring, shutdown, hot-reload, the flush cycle and the offset commit | +| `MessageProcessor` | The per-message pass, run on the worker pool | +| `BufferManager` | Per-table rows, raw payloads and offsets | +| `Inserter` | Format dispatch, retry and salvage | +| `ClickHouseQueryClient` | Schema, comment and health queries | +| `ChClient` | The official `clickhouse::Client` over HTTP, or `dfe_clickhouse::TcpClient` over native | + A higher layer depends on lower layers, never the reverse. The insert path's external dependencies are the official `clickhouse` crate's HTTP `Client`, and `dfe-clickhouse` for the native-protocol `TcpClient` and for reading the @@ -61,54 +72,77 @@ server's exception code out of an HTTP rejection so the inserter can classify it The `clickhouse` crate keeps insertion typed (`T: Row`, compile-time schema) -- that is the right upstream default. dfe-loader inserts -`Map` rows whose columns come from `system.columns` at runtime. +`Map` rows with columns from `system.columns` at runtime. That dynamic layer comes from `dfe-clickhouse`'s `dynamic` module, on top of the upstream -`Client`: +`Client`. The rows are encoded against the schema read at runtime, then sent as RowBinary through the official client over HTTP, or as Native blocks through `dfe_clickhouse::TcpClient` over native. ```mermaid flowchart LR - subgraph loader["dfe-loader"] - INS["Inserter"] - subgraph ext["dfe_clickhouse::dynamic"] - PT["ParsedType
runtime CH type parser"] - ENC["DynamicRow encoder
Map -> RowBinary, per column"] - SCH["schema fetch
system.columns -> ColumnDef"] - DI["DynamicInsert
write_map + recovery"] - end - end - subgraph client["clickhouse::Client"] - IFW["insert_formatted_with
FORMAT RowBinaryWithNamesAndTypes | JSONEachRow (HTTP)"] + accTitle: L4 Detail -- Dynamic insert + accDescr: The Inserter hands rows to dfe_clickhouse's dynamic module. It reads the table's schema from system.columns, encodes each row against it with DynamicRow, and DynamicInsert sends the RowBinary through the clickhouse crate's insert_formatted_with. + + inserter["Inserter"] + + subgraph dynamic["dfe_clickhouse::dynamic"] + direction LR + schema["fetch_dynamic_schema"] --> encode["DynamicRow"] --> send["DynamicInsert"] end - INS --> DI - DI --> ENC --> PT - DI --> SCH - DI -->|RowBinary| IFW - INS -->|JSONEachRow fallback| IFW + client["clickhouse::Client
insert_formatted_with"] + + inserter -->|rows| dynamic + dynamic -->|RowBinary| client + + classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A + classDef ext fill:#EEF0F2,stroke:#6B7280,stroke-width:1px,stroke-dasharray:4 3,color:#1F2937 + classDef group fill:none,stroke:#64748B,stroke-width:1px,stroke-dasharray:2 4 + class client ext + class dynamic group ``` +*Figure 2. L4 Detail -- Dynamic insert, over HTTP. The dashed box is the crates.io client.* Up: [L3 Components -- dfe-loader](README.md). + +`ParsedType` parses each column's type string for the encoder. A JSONEachRow insert skips the encoder: `shape_for_type` rewrites the values the server would refuse and the rows go straight to `insert_formatted_with`. + Over HTTP every insert goes through `insert_formatted_with`: `FORMAT RowBinaryWithNamesAndTypes`, one INSERT for each set of defaulted columns the batch's rows supply, or `JSONEachRow`. With `clickhouse.protocol: native` the rows go out as `FORMAT Native` blocks through `TcpClient::insert_native`, and `JSONEachRow` is refused at startup. See [clickhouse/clickhouse-ext.md](clickhouse/clickhouse-ext.md). ## Insert dispatch +A failed RowBinary insert ends one of three ways: held for a later flush, retried against a fresh schema, or split so only its bad rows reach the DLQ. + ```mermaid -flowchart TB - F{"insert_format?"} - RB["DynamicInsert.write_map(s)"] - HTTP["Client.insert_formatted_with
FORMAT RowBinaryWithNamesAndTypes (HTTP)"] - JE["Client.insert_formatted_with
FORMAT JSONEachRow (HTTP)"] - SM{"SchemaMismatch?"} - REC["invalidate schema cache
re-fetch + retry"] - - F -->|row_binary default| RB --> HTTP - F -->|json_each_row| JE - HTTP --> SM - SM -->|yes| REC --> RB - SM -->|no| OK["offsets join the one commit of the flush cycle"] +flowchart LR + accTitle: L4 Detail -- Insert outcome + accDescr: A successful insert lets its offsets join the flush cycle's commit. A timeout, a dropped connection or a server code on the retry list holds the batch. Any other refusal against a changed table is retried against the current schema, and one against an unchanged table goes to salvage and the DLQ. + + ok{"Insert
answered ok?"} + retryable{"Retryable
failure?"} + changed{"Table
changed?"} + commit(["Offsets commit"]) + held(["Batch held"]) + retry(["Retry on
fresh schema"]) + salvage(["Salvage
to DLQ"]) + + ok -->|no| retryable + ok -->|yes| commit + retryable -->|no| changed + retryable -->|yes| held + changed -->|yes| retry + changed -->|no| salvage + + classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A + classDef decision fill:#FCF1D6,stroke:#9A6700,stroke-width:1px,color:#3D2A00 + classDef error fill:#FBE3D6,stroke:#C2410C,stroke-width:1px,color:#4A1A05 + class ok,retryable,changed decision + class salvage error ``` +*Figure 3. L4 Detail -- Insert outcome. Orange is the end that dead-letters rows.* Up: [L3 Components -- dfe-loader](README.md). + +`Inserter::dynamic_verdict` in `src/clickhouse/inserter.rs` makes the call. Retryable means a timeout, a dropped connection or a code on the retry list in `src/clickhouse/error.rs`. A stale or unreadable cached schema is retried on a fresh read without asking the table, and an encoder refusal of a value goes straight to salvage. + The RowBinary path ships `FORMAT RowBinaryWithNamesAndTypes` through -`insert_formatted_with`, and the header's types make the server refuse rows encoded against a type the column no longer has. A refusal is the rows' own only once a fresh read shows +`insert_formatted_with`, and the header's types make the server refuse rows encoded against a type the column no longer has. A refusal is blamed on the rows once a fresh read shows the table unchanged since they were encoded, and otherwise the retry encodes against the current schema. Over native the block declares the types. @@ -118,26 +152,12 @@ over native, never by the words of the message: a parse rejection echoes the offending row back. The loader keeps at most 1 MiB of any error text, because the client reads a whole error body into memory. -## Capture modes - -`capture_mode` decides what gets stored alongside the promoted columns. -Resolution is three-level, highest wins: DDL `@capture_mode` comment > -per-table config > global default. - -| Mode | `_json` | `_raw` | Use | -|------|---------|--------|-----| -| `full` (default) | full payload (JSON type) | the record's own raw field, else NULL | path queries + text search | -| `json_only` | full payload (JSON type) | NULL | path queries without the `_raw` overhead | -| `raw_only` | NULL, unless a directive fills it | full Kafka payload (String) | CH CPU saving, no JSON-type overhead | -| `extracted_only` | NULL, unless a directive fills it | NULL | promoted columns only | - -All four still promote schema columns, and differ only in whether, and where, -the full payload is kept. A directive on `_json` makes it a promoted column under every mode ([pipeline/capture-modes.md](pipeline/capture-modes.md)). Under `full` a JSON record is stored once, in `_json`: `_raw` holds only raw data the source sent with it, such as a syslog line the receiver captured. +`capture_mode` decides whether the full payload is kept beside the promoted columns, as `_json`, `_raw` or neither, resolved per table from the DDL tag, then per-table config, then the global default. Under the default `full` a JSON record is stored once, in `_json`. [pipeline/capture-modes.md](pipeline/capture-modes.md) has the four modes. ## Resilience - A data error binary-splits the batch down to the bad rows, which go to the DLQ, and the rest land. - A flush cycle in which every insert failed opens the circuit gate on the KEDA scaling signal, and the next successful insert closes it. A failed insert never sends its batch to the DLQ. - Drift, a column dropped or retyped by `ALTER`, drops the cached schema, and the retry encodes against a fresh read. An added column breaks no insert, because the INSERT names its columns, so it takes its default until the cached schema refreshes. -- Offsets commit once per cycle, and on each partition stop below the lowest offset not placed yet. A failed batch or a dead letter the DLQ refused is held and retried with jittered backoff, and holds its partition's commit until it lands. A dead letter no DLQ backend can ever hold is dropped and counted instead (`pipeline_dead_letters_dropped_total{reason}`). A held batch whose table ClickHouse has since reported absent is retried against the default table. +- Offsets commit once per cycle, and on each partition stop below the lowest offset not placed yet. A failed batch or a dead letter the DLQ refused is held and retried with jittered backoff, and holds its partition's commit until it lands. A dead letter no DLQ backend can ever hold is dropped and counted instead (`pipeline_dead_letters_dropped_total{reason}`). A held batch is retried against the default table once ClickHouse reports its table absent, a batch bound for `dlq_load.table` aside, which waits for its table. - Rows of a partition a rebalance has moved away are discarded before any write, since the partition's next owner reads them again from the committed offset (`transport_revoke_discarded_total{stage="buffer"}`). See [operations.md](operations.md#rebalances). diff --git a/docs/clickhouse/README.md b/docs/clickhouse/README.md index 004b089..3ddff52 100644 --- a/docs/clickhouse/README.md +++ b/docs/clickhouse/README.md @@ -14,19 +14,41 @@ The dynamic insert path over the official `clickhouse` crate. The loader reflects each table's schema at runtime and encodes `Map` rows straight to binary -- no compile-time `Row` types. +The inserter takes each table's batch from its buffer and sends it through the dynamic insert and the client to ClickHouse, and the rows ClickHouse rejects for good are the ones that leave for the DLQ. + ```mermaid flowchart LR - INS["Inserter"] - DI["DynamicInsert"] - SC["SchemaCache"] - ENC["encoder"] - CL["clickhouse::Client (HTTP)"] - INS --> DI --> ENC - DI --> SC - DI --> CL + accTitle: L4 Detail -- Insert layer + accDescr: The per-table buffer hands batches to the insert layer. Inside, the Inserter passes rows to dfe_clickhouse's DynamicInsert, which writes them through the ChClient. Rows land in ClickHouse, and rows ClickHouse refuses for good go to the DLQ. + + buffer["Buffer per table"] + + subgraph insert["Insert"] + direction LR + inserter["Inserter"] --> dynamic["DynamicInsert"] --> chclient["ChClient"] + end + + clickhouse[("ClickHouse")] + dlq[("DLQ")] + + buffer -->|batches| insert + insert -->|rows| clickhouse + insert -.->|dead letters| dlq + + classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A + classDef store fill:#DDF3EC,stroke:#00785A,stroke-width:1px,color:#0B3327 + classDef error fill:#FBE3D6,stroke:#C2410C,stroke-width:1px,color:#4A1A05 + classDef group fill:none,stroke:#64748B,stroke-width:1px,stroke-dasharray:2 4 + class clickhouse store + class dlq error + class insert group ``` -## Read in this order +*Figure 1. L4 Detail -- Insert layer. The dotted arrow is the salvage path.* Up: [L3 Components -- dfe-loader](../README.md). + +`ChClient` is the official `clickhouse::Client` over HTTP, or `dfe_clickhouse::TcpClient` with `clickhouse.protocol: native`. The schema cache both clients read is in [schema-cache.md](schema-cache.md). + +Read in this order: - [clickhouse-ext.md](clickhouse-ext.md) -- the dynamic insert from `dfe-clickhouse`'s `dynamic` module, and how the inserter reads its failures. @@ -40,6 +62,4 @@ flowchart LR loader reads off table DDL (capture mode, field mapping). - [tls.md](tls.md) -- TLS transport security and private-CA trust. -## The client, in one line - dfe-loader inserts through crates.io `clickhouse` and reads rejection codes with `dfe-clickhouse`, with the dynamic encoding layer from `dfe-clickhouse`'s `dynamic` module. See [clickhouse-ext.md](clickhouse-ext.md). diff --git a/docs/clickhouse/clickhouse-ext.md b/docs/clickhouse/clickhouse-ext.md index c6220e9..78b8ba0 100644 --- a/docs/clickhouse/clickhouse-ext.md +++ b/docs/clickhouse/clickhouse-ext.md @@ -10,7 +10,7 @@ # The dynamic insert -The official `clickhouse` crate keeps insertion typed (`T: Row`). dfe-loader inserts `Map` rows whose columns come from `system.columns`, so it takes the dynamic layer from `dfe-clickhouse`'s `dynamic` module (feature `dynamic`). The loader holds no encoder of its own. +The official `clickhouse` crate keeps insertion typed (`T: Row`). dfe-loader inserts `Map` rows with columns from `system.columns`, so it takes the dynamic layer from `dfe-clickhouse`'s `dynamic` module (feature `dynamic`), and holds no encoder itself. ## What the crate does @@ -23,27 +23,27 @@ The official `clickhouse` crate keeps insertion typed (`T: Row`). dfe-loader ins | `DynamicInsert` | `write_map` / `write_map_with_raw`, then `end` | | `shape_for_type` | The same value rules applied to a row map, for the JSONEachRow path | -Over HTTP an insert is `INSERT ... FORMAT RowBinaryWithNamesAndTypes`, whose header declares each column's cached type, sent in 1 MiB chunks as rows are written. An INSERT names the required columns and the defaulted ones its rows supply, so rows that supply a different set go to an INSERT of their own, up to four open at once. Over the native protocol rows go out as `FORMAT Native` blocks, and a row that supplies a different set of defaulted columns starts a new INSERT. JSON columns are written as text, with `input_format_binary_read_json_as_string=1` over HTTP. +Over HTTP an insert is `INSERT ... FORMAT RowBinaryWithNamesAndTypes`, with a header that declares each column's cached type, sent in 1 MiB chunks as rows are written. An INSERT names the required columns and the defaulted ones its rows supply, so rows that supply a different set go to a separate INSERT, up to four open at once. Over the native protocol rows go out as `FORMAT Native` blocks, and a row that supplies a different set of defaulted columns starts another INSERT. JSON columns are written as text, with `input_format_binary_read_json_as_string=1` over HTTP. The coercions a source value gets on the way: a zoned timestamp lands at the instant it names, a numeric DateTime64 epoch is read by magnitude and rescaled to the column's precision, an IPv4 literal lands in an IPv6 column as `::ffff:a.b.c.d`, a JSON column wraps a root array under `list` and a scalar or non-JSON text under `value`, a `Point` takes every Elasticsearch geo_point form inside its coordinate ranges, and one value lands in an `Array` as a one-element array. A value the column cannot hold (an integer past its width, a decimal past its backing, an undeclared enum value) fails the row rather than being written wrong. `tests/integration/insert_coercions.rs` runs each case through the loader's `Inserter` on every protocol the shared ClickHouse offers. ## How the loader drives it -`Inserter` reads the schema through its own widening read bound and the shared cache, opens the insert with `InsertTimeouts` from `clickhouse.request_timeout_ms` and `clickhouse.insert_timeout_ms`, and sets `SchemaRefresh` from `schema.refresh_on_error`. A failure is read in `Inserter::dynamic_verdict`: +`Inserter` reads the schema through its widening read bound and the shared cache, opens the insert with `InsertTimeouts` from `clickhouse.request_timeout_ms` and `clickhouse.insert_timeout_ms`, and sets `SchemaRefresh` from `schema.refresh_on_error`. A failure is read in `Inserter::dynamic_verdict`: - A timeout or a dropped connection returns the batch to be held. -- A server refusal is read by its code. A retryable code holds the batch. Any other code is the rows' own only once a fresh read shows the table unchanged since they were encoded (`DynamicSchema::same_columns`), and then salvage dead-letters them row by row. Otherwise the table changed under them, and the retry encodes against the current schema. +- A server refusal is read by its code. A retryable code holds the batch. Any other code is blamed on the rows once a fresh read shows the table unchanged since they were encoded (`DynamicSchema::same_columns`), and then salvage dead-letters them row by row. Otherwise the table changed under them, and the retry encodes against the current schema. - A stale, missing or unreadable schema drops the cached schemas and retries. - An encoder refusal (`EncodingError` naming its column) is the payload's, and goes to salvage. - A column type the crate cannot encode is a loader gap, so the rows are held. -Both client crates come from crates.io, with no `[patch]` and no git source. `Cargo.toml` carries the version ranges: +Both client crates come from crates.io, with no `[patch]` and no git source. `Cargo.toml` holds the version ranges: ```toml clickhouse = { version = "...", features = ["lz4", "time", "rustls-tls", "rustls-tls-native-roots", "inserter"] } dfe-clickhouse = { version = "...", default-features = false, features = ["ext", "tls", "dynamic"] } ``` -From `dfe-clickhouse` the loader also takes `ServerException::parse` (`ext`), which reads the code and message out of the `Code: N. DB::Exception: ...` text the HTTP client returns as `Error::BadResponse`, and `TcpClient` with its rustls trust (`tls`, which implies `tcp`) for `clickhouse.protocol: native`. `src/clickhouse/error.rs` classifies every insert rejection by the server's code and keeps at most 1 MiB of any error text. +From `dfe-clickhouse` the loader takes `ServerException::parse` (`ext`), which reads the code and message out of the `Code: N. DB::Exception: ...` text the HTTP client returns as `Error::BadResponse`, and `TcpClient` with its rustls trust (`tls`, which implies `tcp`) for `clickhouse.protocol: native`. `src/clickhouse/error.rs` classifies every insert rejection by the server's code and keeps at most 1 MiB of any error text. -The HTTP client is the loader's own. `build_client` in `src/clickhouse/client_http.rs` hands `clickhouse::Client` a hyper client through `Client::with_http_client`, carrying the pool timer that closes idle connections after 2 s, the connect timeout and HTTPS verified against the system trust store. +The loader builds the HTTP client. `build_client` in `src/clickhouse/client_http.rs` hands `clickhouse::Client` a hyper client through `Client::with_http_client`, carrying the pool timer that closes idle connections after 2 s, the connect timeout and HTTPS verified against the system trust store. diff --git a/docs/clickhouse/ddl-directives.md b/docs/clickhouse/ddl-directives.md index 10d0214..9167c12 100644 --- a/docs/clickhouse/ddl-directives.md +++ b/docs/clickhouse/ddl-directives.md @@ -34,41 +34,51 @@ capture interacts with `_json` and `_raw`, see capture modes in ## Directive resolution +A column reads the paths its `@source` or `@renamed` directive lists, else the field with the column's name, and `@computed` or `@default` fills it when no field is found. + ```mermaid -flowchart TB - COL["Column in reflected schema
+ @-comment directive"] - K{"Which directive?"} - SRC["@source: field
read input, optional fallback"] - REN["@renamed: field
same source-path mapping as @source
read in place of the column name"] - CMP["@computed: expression
evaluate one CEL expression"] - OUT["Value written to output column"] - GEN["@generated: expr
(descriptive label only)"] - CFG["@config: path
(descriptive label only)"] - DEF["ClickHouse DEFAULT/MATERIALIZED/ALIAS
on the column itself"] - - COL --> K - K -->|source| SRC - K -->|renamed| REN - K -->|computed| CMP - SRC --> OUT - REN --> OUT - CMP --> OUT - GEN -.->|describes| DEF - DEF -.-> SKIP["field omitted from insert"] +flowchart LR + accTitle: L4 Detail -- Column directive resolution + accDescr: A column marked skip is left out of the insert. Any other column reads the paths its source or renamed directive lists, else the field named like the column. A field found is written. A column with no field found takes its computed or default value. + + skip{"@skip?"} + read["Listed paths
or column name"] + left(["Left out"]) + found{"Field
found?"} + value(["Field's value"]) + fill(["@computed or
@default value"]) + + skip -->|no| read + skip -->|yes| left + read --> found + found -->|yes| value + found -->|no| fill + + classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A + classDef decision fill:#FCF1D6,stroke:#9A6700,stroke-width:1px,color:#3D2A00 + class skip,found decision ``` -`@generated` and `@config` sit outside the loader's directive dispatch: the -loader parses neither. They are notes an author leaves in the comment, and the -real mechanism sits elsewhere, covered under each heading below. +*Figure 1. L4 Detail -- Column directive resolution. `_org_id` and `_timestamp` read fixed fields and skip this path.* Up: [ClickHouse insert layer](README.md). + +`_timestamp_received` reads the first hop's edge stamp, then falls back to the paths its `@source` lists. -## Overview +`@generated` and `@config` sit outside the loader's directive dispatch: the loader parses neither. They are notes an author leaves in the comment, and the mechanism sits elsewhere, covered under each heading below. A column ClickHouse fills itself, through a DEFAULT, MATERIALIZED or ALIAS clause, can be left out of the row whatever its comment says. -This expression language describes how source data fields map to destination -columns during the **data preparation stage** of the pipeline. It is used in: +### @skip and @default -- dfe-schemas YAML definitions, in a column's `expr` field -- The column COMMENTs those become in the deployed DDL -- Any table definition that needs field mapping documentation +`@skip` leaves the column out of every row the loader builds, `_org_id`, `_timestamp` and `_timestamp_received` aside. `@default: value` fills a column when every path it reads is absent or null: a scalar literal, or `now()`, `uuid()` or `topic_name` resolved per row. Each directive has a config twin in `column_directives`: `global.` for every table and `tables.".".` for one. Config wins over the column comment. + +```yaml +column_directives: + global: + severity: + default: unknown + tables: + "dfe.events": + debug_info: + skip: true +``` ## Expression types @@ -106,10 +116,10 @@ the DDL clause that actually does the filling. The loader still omits the field from the insert, but by a different route: it reads `default_kind` for the column from `system.columns`, and any column -whose kind is `DEFAULT`, `MATERIALIZED`, `ALIAS` or `EPHEMERAL` is marked +with a kind of `DEFAULT`, `MATERIALIZED`, `ALIAS` or `EPHEMERAL` is marked `has_default` and can be left out of the row it sends (the dynamic insert in -`dfe-clickhouse`'s `dynamic` module). dfe-schemas' renderer, whose DDL dfe-engine applies, writes that DEFAULT/MATERIALIZED/ALIAS -clause from the schema's own `default:` and `attribute:` fields -- not by +`dfe-clickhouse`'s `dynamic` module). dfe-schemas' renderer writes the DDL dfe-engine applies, and it writes that DEFAULT/MATERIALIZED/ALIAS +clause from the schema's `default:` and `attribute:` fields -- not by reading the `@generated:` text -- so keep the two in agreement by hand. ```text @@ -129,10 +139,8 @@ reading the `@generated:` text -- so keep the two in agreement by hand. `id` UUID DEFAULT generateUUIDv7() COMMENT '@generated: generateUUIDv7()', ``` -### Capture is not a column directive - `_raw` and `_json` are populated by the loader's capture mode, not by a -per-column `@`-comment directive. Capture mode has its own tag, but it is a +per-column `@`-comment directive. Capture mode has a tag, but it is a **table**-level DDL comment -- `@capture_mode: ` on the `CREATE TABLE` itself, resolved ahead of any per-table or global config (see [capture-modes.md](../pipeline/capture-modes.md)). See @@ -146,10 +154,10 @@ Not a directive the loader parses either -- same drop-as-unknown path as loader config key decides the actual mapping. Nothing reads the comment text to resolve that path: the config key is wired up separately, by the column's special-cased name. `_org_id`, for example, is filled from whichever field -`routing.org_id_field` names, whatever its directive says, and `_tags` from its own `@source` directive, else the first field in `metadata.tags_fields` +`routing.org_id_field` names, whatever its directive says, and `_tags` from its `@source` directive, else the first field in `metadata.tags_fields` (dfe-loader `src/config/pipeline.rs`, `src/routing/router.rs`, `src/transform/extractor.rs`, `src/transform/transformer.rs`). Change the -config path and the mapping changes. Change the `@config:` comment on its own +config path and the mapping changes. Change the `@config:` comment alone and nothing does. ```text @@ -208,7 +216,7 @@ risk come from the loader's `geoip` and `enrichment` config, not from **Semantics:** -- A field the record already carries under the column's name is kept, and the +- A field the record holds under the column's name is kept, and the expression is not run for it. - An expression that does not compile, a function outside the profile included, is logged as a warning and the column is skipped. @@ -217,7 +225,7 @@ risk come from the loader's `geoip` and `enrichment` config, not from column logs one warning with the error. - `dfe_loader_computed_eval_failures_total{table, column, stage}` counts every failure, `stage` being `compile`, `context` or `eval`. -- A `null` result leaves the column empty too. +- A `null` result leaves the column empty as well. The `computed_columns` config section sets or overrides the same expressions: `columns.` for every table, and `overrides.".
".` for @@ -241,7 +249,7 @@ the same `renamed` list, with no transformation of the value. **Semantics:** -- The listed paths are read in place of the column's own name, so a record field named like the column is not read while the directive names others. Only the field-mapping step, which runs after, skips a destination already present. +- The listed paths are read in place of the column's name, so a record field with the column's name is not read when the directive lists others. The field-mapping step, which runs after, is the one that skips a destination the row holds. - With `first()`, fields are tried in order and the first present, non-null one is used. - `_org_id` and `_timestamp` read fixed fields and ignore the directive, so no column comment can point the tenant at a field a sender writes. @@ -285,7 +293,7 @@ host_name` leaves `_source` absent, it does not write the string `host_name`. The loader logs the ignored fallback once per five minutes. `now()`, `uuid()` and `topic_name` are resolved per row in the extractor rather -than at parse time, so they are the three forms `@default:` also accepts as a +than at parse time, so they are the three forms `@default:` accepts as a bare word. ### List operator (`/`) @@ -322,11 +330,11 @@ On the default `json_primary` path the header extractor (`src/transform/extracto | `@source: first(a/b)`, `@renamed: first(a/b)` | Try each path in order, first non-null wins | | `@computed: expr` | Compiled once with `scalo::expression::compile`, evaluated per record | -`@generated` and `@config` are dropped as unknown directives. A `@generated` column is still left out of the insert, by the `system.columns` `default_kind` route described under @generated above. A directive names the input field, and the column name is the output: `@source: timestamp` on `_timestamp` reads `timestamp`. With `metadata.remove_routing_fields` (default `true`) the routing fields are removed after extraction. +`@generated` and `@config` are dropped as unknown directives. A `@generated` column is still left out of the insert, by the `system.columns` `default_kind` route described under @generated above. A directive names the input field, and the column name is the output: `@source: status_code` on `http_status` reads `status_code`. On `legacy_flatten`, `metadata.remove_routing_fields` (default `true`) removes the routing fields after extraction. The default path leaves the payload as it arrived. ## Table-level tags -A table `COMMENT` carries tags in the same `@tag: value` syntax, separated by ` | `. The loader reads one, `@capture_mode`, which overrides the configured capture mode for that table (see [capture-modes.md](../pipeline/capture-modes.md)) and ignores the rest. dfe-schemas writes `@profile` and `@profile_version`, naming the common-header profile the table was built from: +A table `COMMENT` holds tags in the same `@tag: value` syntax, separated by ` | `. The loader reads one, `@capture_mode`, which overrides the configured capture mode for that table (see [capture-modes.md](../pipeline/capture-modes.md)) and ignores the rest. dfe-schemas writes `@profile` and `@profile_version`, naming the common-header profile the table was built from: ```sql CREATE TABLE dfe.main (...) diff --git a/docs/clickhouse/insert-formats.md b/docs/clickhouse/insert-formats.md index 2bc6f59..70f8891 100644 --- a/docs/clickhouse/insert-formats.md +++ b/docs/clickhouse/insert-formats.md @@ -10,41 +10,47 @@ # Insert formats and transports -dfe-loader has two knobs for how rows reach ClickHouse: `clickhouse.protocol` -(how the bytes travel) and `clickhouse.insert_format` (how the rows are -encoded). The default `http + row_binary` is the fast path, and `json_each_row` is -a deliberate fallback. `protocol: native` runs the schema fetch, the queries and -`row_binary` inserts over the native protocol through `dfe_clickhouse::TcpClient`. +dfe-loader has two knobs for how rows reach ClickHouse: `clickhouse.protocol` (how the bytes travel) and `clickhouse.insert_format` (how the rows are encoded). The default `http + rowbinary` is the fast path, and `jsoneachrow` is a deliberate fallback. `protocol: native` runs the schema fetch, the queries and `rowbinary` inserts over the native protocol through `dfe_clickhouse::TcpClient`. + +Three of the four pairs insert, and JSONEachRow over the native protocol stops the pipeline at start. ```mermaid -flowchart TB - ROW["Buffered rows
Map<String,Value> + raw _json"] - F{"insert_format"} - T{"protocol"} - RBH["insert_formatted_with
FORMAT RowBinaryWithNamesAndTypes (HTTP)"] - RBT["TcpClient::insert_native
FORMAT Native blocks (TCP)"] - JE["insert_formatted_with
FORMAT JSONEachRow (HTTP)"] - REJ["refused when the pipeline starts"] - CH[(ClickHouse)] - - ROW --> F - F -->|row_binary| T - T -->|http| RBH --> CH - T -->|native| RBT --> CH - F -->|json_each_row + http| JE --> CH - F -->|json_each_row + native| REJ +flowchart LR + accTitle: L4 Detail -- Insert format and protocol + accDescr: A rowbinary insert goes as RowBinaryWithNamesAndTypes over HTTP or as Native blocks over the native protocol. A jsoneachrow insert goes as JSONEachRow over HTTP, and the pipeline refuses it at start over native. + + format{"insert_format?"} + binproto{"protocol?"} + jsonproto{"protocol?"} + rowbinary(["RowBinaryWithNamesAndTypes
over HTTP"]) + native(["Native blocks
over TCP"]) + jsonrows(["JSONEachRow
over HTTP"]) + refused(["Refused at
pipeline start"]) + + format -->|rowbinary| binproto + format -->|jsoneachrow| jsonproto + binproto -->|http| rowbinary + binproto -->|native| native + jsonproto -->|http| jsonrows + jsonproto -->|native| refused + + classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A + classDef decision fill:#FCF1D6,stroke:#9A6700,stroke-width:1px,color:#3D2A00 + classDef error fill:#FBE3D6,stroke:#C2410C,stroke-width:1px,color:#4A1A05 + class format,binproto,jsonproto decision + class refused error ``` -## The matrix +*Figure 1. L4 Detail -- Insert format and protocol. Orange is the pair the loader rejects.* Up: [ClickHouse insert layer](README.md). | `insert_format` | `protocol = http` | `protocol = native` | |-----------------|--------------------|----------------------| | `row_binary` (default) | `FORMAT RowBinaryWithNamesAndTypes` via `insert_formatted_with` | `FORMAT Native` blocks via `TcpClient::insert_native` | | `json_each_row` | `FORMAT JSONEachRow` via `insert_formatted_with` | refused when the pipeline starts, not by `config-check` | -The rejection is deliberate: JSONEachRow is an HTTP body format, and a native client has no HTTP insert endpoint for it. The guard lives in `ClickHouseConfig::validate()`, which the orchestrator runs before the readiness probe, so a bad combination stops the loader with a clear message rather than at first insert. `config-check` runs `Config::validate` only and passes it. +The rejection is deliberate: JSONEachRow is an HTTP body format, and a native client has no HTTP insert endpoint for it. The guard lives in `ClickHouseConfig::validate()`, which the orchestrator runs before the readiness probe, so a bad combination stops the loader with a clear message rather than at first insert. `config-check` runs `Config::validate` and nothing more, so it passes the pair. -Both protocols check the cached schema against the table. Over HTTP the header declares each column's type and the server refuses a type the column no longer has (code 117). Over native the server answers the INSERT with the table's types, and the insert refuses a difference before sending a row. Either refusal reads as drift. A column whose type `dfe_clickhouse` cannot write fails the insert and the batch is held. +Both protocols check the cached schema against the table. Over HTTP the header declares each column's type and the server rejects a type the column no longer has (code 117). Over native the server answers the INSERT with the table's types, and the insert rejects a difference before sending a row. Either refusal reads as drift. A column of a type `dfe_clickhouse` cannot write fails the insert and the batch is held. ## Why RowBinary is the default @@ -53,12 +59,12 @@ RowBinary ships pre-typed bytes. The server parses no row text and infers no typ JSONEachRow is kept because it is self-describing and forgiving: the server coerces types and tolerates schema drift. That makes it the right tool for diagnostics ("does the row land at all if the server does the typing?") and for -the rare table whose types the dynamic encoder does not yet cover. It is never +the rare table with types the dynamic encoder does not cover. It is never the throughput choice. ## How RowBinary is produced -The dynamic encoder in `dfe-clickhouse`'s `dynamic` module turns one `Map` plus its reflected `ColumnDef` schema into RowBinary, sent in 1 MiB chunks as `INSERT INTO db.table (cols) FORMAT RowBinaryWithNamesAndTypes`. An INSERT names the required columns and the defaulted ones its rows supply, and the server fills the rest from their DEFAULT expressions (`_uuid`, `_timestamp_load`). Rows that supply a different set of defaulted columns go to an INSERT of their own: HTTP keeps up to four open at once, and over native a new set commits the rows before it. A row that supplies no column at all goes as `RowBinaryWithDefaults`. +The dynamic encoder in `dfe-clickhouse`'s `dynamic` module turns one `Map` plus its reflected `ColumnDef` schema into RowBinary, sent in 1 MiB chunks as `INSERT INTO db.table (cols) FORMAT RowBinaryWithNamesAndTypes`. An INSERT names the required columns and the defaulted ones its rows supply, and the server fills the rest from their DEFAULT expressions (`_uuid`, `_timestamp_load`). Rows that supply a different set of defaulted columns go to a separate INSERT: HTTP keeps up to four open at once, and over native a different set commits the rows before it. A row that supplies no column at all goes as `RowBinaryWithDefaults`. A JSON-typed column (including `_json`) is written as a length-prefixed string. On the RowBinary path the loader sets @@ -77,18 +83,18 @@ invalidated so the next attempt re-fetches from `system.columns` and re-encodes. A non-drift error (network, auth) is returned as-is. See [schema-cache.md](schema-cache.md). -The same codes come back for a row the server can never accept, so the server's code decides, never the words beside it: a code on the retry list holds the batch, and any other refusal against a table unchanged since the rows were encoded is the rows' own, so salvage dead-letters the bad row and the rest land. The list and the reason for each group are in [schema-cache.md](schema-cache.md#mismatch-recovery). +The same codes come back for a row the server can never accept, so the server's code decides, never the words beside it: a code on the retry list holds the batch, and any other refusal against a table unchanged since the rows were encoded is blamed on the rows, so salvage dead-letters the bad row and the rest land. The list and the reason for each group are in [schema-cache.md](schema-cache.md#mismatch-recovery). RowBinary is positional, so bytes encoded for a column's old type could parse as other rows. `insert_coercions::type_change_failures`, run on every protocol in the core tier, retypes a column under a cached schema and checks that every row that lands reads back as the value sent. -## Configuring it +The two settings: | Setting | Values | Default | |---------|--------|---------| | `clickhouse.protocol` | `http`, `native` | `http` | | `clickhouse.insert_format` | `rowbinary` (aliases `row_binary`, `binary`), `jsoneachrow` (aliases `json_each_row`, `json`) | `rowbinary` | -The canonical spellings are the ones `docs/config-schema.json` carries, and the aliases parse identically. `native` is refused as a format, because it names a protocol. +The canonical spellings are the ones in `docs/config-schema.json`, and the aliases parse identically. `native` is refused as a format, because it names a protocol. Env form: `DFE_LOADER_CLICKHOUSE__INSERT_FORMAT=json_each_row`. The startup log line `Insert format configured format=...` reports the value that was applied. diff --git a/docs/clickhouse/schema-cache.md b/docs/clickhouse/schema-cache.md index abf7781..8971e83 100644 --- a/docs/clickhouse/schema-cache.md +++ b/docs/clickhouse/schema-cache.md @@ -10,35 +10,46 @@ # Schema reflection and caching -The dynamic insert path needs to know a table's columns and their types before -it can encode a row. dfe-loader never carries a compile-time schema -- it -reflects the live table from `system.columns` and caches the result with a TTL. +The dynamic insert path needs to know a table's columns and their types before it can encode a row. dfe-loader holds no compile-time schema -- it reflects the live table from `system.columns` and caches the result with a TTL. + +A row is encoded against the cached schema until it expires, and with `schema.refresh_on_error` on, the default, a drift error drops the entry so the next attempt reads the table again. ```mermaid -flowchart TB - W["write_map(row)"] - G{"cache hit
and fresh?"} - F["fetch_dynamic_schema
SELECT ... FROM system.columns"] - C["DynamicSchemaCache
(TTL, shared via Arc)"] - ENC["encode against ColumnDef[]"] - INS["insert"] - MM{"drift error?"} - INV["invalidate(table)"] - - W --> G - G -->|yes| ENC - G -->|no| F --> C --> ENC - ENC --> INS --> MM - MM -->|yes| INV --> F - MM -->|no| OK["committed"] +flowchart LR + accTitle: L4 Detail -- Schema cache read and recovery + accDescr: An insert uses the cached schema when it is fresh, else reads system.columns. The rows are encoded and inserted. A drift error invalidates the table's entry so the retry reads the schema again, and any other answer ends the attempt. + + fresh{"Cached and
fresh?"} + cached["Cached schema"] + fetch["Read columns"] + insert["Encode, insert"] + drift{"Drift
error?"} + done(["Attempt ends"]) + invalidate(["Invalidate"]) + + fresh -->|yes| cached + fresh -->|no| fetch + cached --> insert + fetch --> insert + insert --> drift + drift -->|no| done + drift -->|yes| invalidate + + classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A + classDef store fill:#DDF3EC,stroke:#00785A,stroke-width:1px,color:#0B3327 + classDef decision fill:#FCF1D6,stroke:#9A6700,stroke-width:1px,color:#3D2A00 + class fresh,drift decision + class cached store ``` +*Figure 1. L4 Detail -- Schema cache read and recovery, with `schema.refresh_on_error` on.* Up: [ClickHouse insert layer](README.md). + ## Reflection `fetch_dynamic_schema` queries `system.columns` for one table and builds a `DynamicSchema`: the columns in declaration (position) order, each parsed into a structured `ParsedType` (Nullable, LowCardinality, Array, Map, DateTime64, -Decimal, Enum, ...) that drives encoding. It also records, per column, whether +Decimal, Enum, ...) that drives encoding. It records, per column, whether the column has a server-side default -- that is what lets the encoder omit an absent-but-defaulted column (`_uuid`, `_timestamp_load`) and include every required one. See [types.md](types.md). @@ -54,25 +65,19 @@ insert tasks behind an `Arc`. It is lazy, not background-refreshed: - `invalidate(table)` drops one entry, and `invalidate_all` clears the map. Lazy TTL keeps the steady state cheap (no timer threads, no work for idle -tables) while bounding how long a stale schema can persist after a benign +tables) and bounds how long a stale schema can persist after a benign `ALTER`. The TTL is set once when the cache is constructed, from `schema.cache_ttl_secs` (default 300 s). -The loader keeps a second cache of the same read, `SchemaCache` in `src/clickhouse/schema.rs`, which decides the fields the header extractor promotes. Its entries live for `schema.cache_ttl_secs` too, and every 60 s a background task hands each entry within 30 s of expiry to the schema resolver, which re-reads its columns and comments together. +The loader keeps a second cache of the same read, `SchemaCache` in `src/clickhouse/schema.rs`, which decides the fields the header extractor promotes. Its entries live for `schema.cache_ttl_secs` as well, and every 60 s a background task hands each entry within 30 s of expiry to the schema resolver, which re-reads its columns and comments together. ## Mismatch recovery -A TTL alone does not cover a schema change that happens mid-batch. So the insert -path also recovers on error. When ClickHouse rejects an insert with a drift -signal -- a code name such as `TYPE_MISMATCH`, `NO_SUCH_COLUMN` or -`INCORRECT_DATA`, or a drift phrase such as "no such column" or "cannot -parse" -- the table's cache entry is invalidated. The retry re-fetches from -`system.columns` and re-encodes against the current schema. A non-drift error (network, auth, server down) is -returned unchanged, so an outage is not mistaken for schema drift. +A TTL does not cover a schema change that happens mid-batch, so the insert path recovers on error as well. `dfe-clickhouse` reads a drift code, such as `TYPE_MISMATCH`, `NO_SUCH_COLUMN_IN_TABLE` or `INCORRECT_DATA`, as `SchemaMismatch`, and reads the message for drift wording only when the error has no code. The table's cache entry is invalidated, and the retry re-fetches from `system.columns` and re-encodes against the current schema. A non-drift error (network, auth, server down) is returned unchanged, so an outage is not mistaken for schema drift. -A row the server can never accept draws the same signals: an integer no ClickHouse integer type holds, in a JSON column, comes back as code 117 whatever the schema. So the server's code decides, never the words beside it, which echo the row. A code on the retry list in `src/clickhouse/error.rs` (`RETRY_CODES`: a busy or overloaded server, the network, replication and Keeper, the disk or object store under the table, and operator fixes such as a missing table or a refused login) holds the batch for another attempt. Any other code re-reads `system.columns`. A table unchanged since the rows were encoded makes the refusal the rows' own: the insert returns it as permanent, salvage isolates the bad row for the DLQ, and the rest land. A changed table is drift, recovered as above. `TOO_MANY_PARTS` for one INSERT over `max_partitions_per_insert_block` is a refusal, not a retry, because salvage's split is what lands it. A server failure with no code never reaches the DLQ. Only the encoder's own refusal of a value does, since it holds the schema it read the value against. +A row the server can never accept draws the same signals: an integer no ClickHouse integer type holds, in a JSON column, comes back as code 117 whatever the schema. So the server's code decides, never the words beside it, which echo the row. A code on the retry list in `src/clickhouse/error.rs` (`RETRY_CODES`: a busy or overloaded server, the network, replication and Keeper, the disk or object store under the table, and operator fixes such as a missing table or a refused login) holds the batch for another attempt. Any other code re-reads `system.columns`. A table unchanged since the rows were encoded puts the refusal on the rows: the insert returns it as permanent, salvage isolates the bad row for the DLQ, and the rest land. A changed table is drift, recovered as above. `TOO_MANY_PARTS` for one INSERT over `max_partitions_per_insert_block` is a refusal, not a retry, because salvage's split is what lands it. A server failure with no code never reaches the DLQ. The encoder's refusal of a value does, since it holds the schema it read the value against. -A column dropped or retyped by an out-of-band `ALTER` fails one attempt: the rejection clears the stale entry and the retry succeeds against the new schema. An added column fails nothing, since the INSERT names its columns, and it fills once the cached schema refreshes. +A column dropped or retyped by an out-of-band `ALTER` fails one attempt: the rejection clears the stale entry and the retry succeeds against the altered schema. An added column fails nothing, since the INSERT names its columns, and it fills once the cached schema refreshes. `schema.refresh_on_error` (default `true`) switches this recovery. With `false` a drift error leaves both the encoder's entry and the loader's cached schema in diff --git a/docs/clickhouse/tls.md b/docs/clickhouse/tls.md index 0a4495b..96d3169 100644 --- a/docs/clickhouse/tls.md +++ b/docs/clickhouse/tls.md @@ -10,21 +10,34 @@ # TLS and private CAs -With TLS on the loader speaks HTTPS (`clickhouse.protocol: http`, usually 8443) or native TLS (`native`, usually 9440), and either verifies the server's certificate and host name against the container's system trust store. Native TLS trusts the system store only, not a compiled-in root bundle, so both protocols accept the same servers. +With TLS on the loader speaks HTTPS (`clickhouse.protocol: http`, usually 8443) or native TLS (`native`, usually 9440), and either verifies the server's certificate and host name against the container's system trust store. Native TLS trusts the system store alone, not a compiled-in root bundle, so both protocols accept the same servers. + +Turning TLS on is one key, and trust comes from the system store alone, so a private CA is added there rather than set in config. ```mermaid flowchart LR - CFG["clickhouse.tls.enabled"] - STORE["System trust store
(image bundle, or SSL_CERT_FILE / SSL_CERT_DIR)"] - VERIFY["Certificate chain + host name check"] - HTTPS["HTTPS to ClickHouse"] - - CFG --> HTTPS - STORE --> VERIFY - VERIFY --> HTTPS + accTitle: L4 Detail -- TLS trust + accDescr: clickhouse.tls.enabled turns TLS on. The system trust store, the image bundle or the file SSL_CERT_FILE or SSL_CERT_DIR names, supplies the CA certificates. The loader verifies the server's certificate chain and host name against them before it opens HTTPS or native TLS to ClickHouse. + + enabled["clickhouse.tls.enabled"] + store["System trust store"] + verify["Verify chain
and host name"] + conn["HTTPS or
native TLS"] + clickhouse[("ClickHouse")] + + enabled -->|on| verify + store -->|CA certs| verify + verify -->|verified| conn + conn --> clickhouse + + classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A + classDef core fill:#DCEBF7,stroke:#0072B2,stroke-width:2px,color:#0B2A3F + classDef store fill:#DDF3EC,stroke:#00785A,stroke-width:1px,color:#0B3327 + class verify core + class clickhouse store ``` -## Turning TLS on +*Figure 1. L4 Detail -- TLS trust. The store is the image bundle, or what `SSL_CERT_FILE` or `SSL_CERT_DIR` names.* Up: [ClickHouse insert layer](README.md). | Setting | Effect | |---------|--------| @@ -35,13 +48,13 @@ The SNI name and the name checked against the certificate are the host in `click ## Private / internal CAs -A private CA is trusted the way the system trusts it: add it to the image's trust store, or point `SSL_CERT_FILE` at a PEM bundle that holds it (`SSL_CERT_DIR` at a directory of them). Either variable replaces the platform store rather than adding to it, so a bundle that must also trust public CAs has to carry them too. +A private CA is trusted the way the system trusts it: add it to the image's trust store, or point `SSL_CERT_FILE` at a PEM bundle that holds it (`SSL_CERT_DIR` at a directory of them). Either variable replaces the platform store rather than adding to it, so a bundle that must trust public CAs as well has to hold them. A trust store with no readable CA certificate fails the client at startup instead of connecting unverified. `clickhouse.tls.ca_cert_file`, `cert_file`, `key_file` and `skip_verify` are refused by `config-check` and at startup: no client reads them. There is no skip-verify mode and no client-certificate (mTLS) auth. -## Example: private-CA cluster +A private-CA cluster: ```yaml clickhouse: diff --git a/docs/clickhouse/types.md b/docs/clickhouse/types.md index 29f4830..ecebb25 100644 --- a/docs/clickhouse/types.md +++ b/docs/clickhouse/types.md @@ -38,8 +38,8 @@ Sources: |--------|---------| | GA | Generally Available | | Beta | Beta, stable but may change | -| Experimental | Experimental, requires setting flag | -| Deprecated | Deprecated, avoid in new code | +| Experimental | Experimental, needs a setting flag | +| Deprecated | Deprecated, avoid in a schema | --- @@ -103,8 +103,8 @@ Sources: | `DateTime(tz)` | Same + timezone | 1 second | UInt32 + timezone metadata | | `DateTime64(P)` | Wide range | 10^-P seconds | Int64 (scaled) | | `DateTime64(P, tz)` | Same + timezone | 10^-P seconds | Int64 + timezone metadata | -| `Time` | 00:00:00 to 23:59:59 | 1 second | UInt32 | -| `Time64(P)` | High precision time | 10^-P seconds | Int64 | +| `Time` | -999:59:59 to 999:59:59 | 1 second | Int32 (seconds) | +| `Time64(P)` | -999:59:59 to 999:59:59 | 10^-P seconds | Decimal64 (Int64 ticks) | **DateTime64 precision:** @@ -374,19 +374,19 @@ A `Point` is written as two little-endian `Float64`, x then y, where x is the lo | Form | Example | Coordinate order | |------|---------|------------------| -| Object | `{"lat": 41.12, "lon": -71.34}` | named | +| Object | `{"lat": 41.12, "lon": -71.34}` | by key | | GeoJSON | `{"type": "Point", "coordinates": [-71.34, 41.12]}` | lon, lat | | String | `"41.12,-71.34"` | lat, lon | | Array | `[-71.34, 41.12]` | lon, lat | | WKT | `"POINT (-71.34 41.12)"` | lon, lat | -A third coordinate is read and ignored. A geohash, a latitude outside -90 to 90, a longitude outside -180 to 180, or a value no form reads is an encoding error for that row alone: salvage isolates it and it goes to the DLQ with the reason. An absent `Point` is `(0, 0)`, the type's own default. ClickHouse refuses `Nullable(Point)`, so there is no null to write instead. +A third coordinate is read and ignored. A geohash, a latitude outside -90 to 90, a longitude outside -180 to 180, or a value no form reads is an encoding error for that row alone: salvage isolates it and it goes to the DLQ with the reason. An absent `Point` is `(0, 0)`, the type's default. ClickHouse rejects `Nullable(Point)`, so there is no null to write instead. -`Ring`, `LineString`, `Polygon`, `MultiLineString` and `MultiPolygon` are arrays of points on the wire, written from nested arrays whose innermost elements take any form above. An absent one is the empty array. +`Ring`, `LineString`, `Polygon`, `MultiLineString` and `MultiPolygon` are arrays of points on the wire, written from nested arrays with innermost elements in any form above. An absent one is the empty array. ### A single value bound for an Array -Elasticsearch lets any field carry one value or many, so a scalar bound for an `Array(T)` column is written as a one-element array, and T's own rules then decide whether it is valid. `"related_ip": "52.108.0.3"` lands in an `Array(IPv6)` column as `['::ffff:52.108.0.3']`. An object is wrapped only where the element is read from one (`JSON`, `Map`, `Point`). Bound for an array of scalars, it stays an encoding error. A null or absent array is the empty array. For an `Array(Point)` column, a bare `[lon, lat]` is one point, as Elasticsearch reads it. +Elasticsearch lets any field hold one value or many, so a scalar bound for an `Array(T)` column is written as a one-element array, and T's rules then decide whether it is valid. `"related_ip": "52.108.0.3"` lands in an `Array(IPv6)` column as `['::ffff:52.108.0.3']`. An object is wrapped only where the element is read from one (`JSON`, `Map`, `Point`). Bound for an array of scalars, it stays an encoding error. A null or absent array is the empty array. For an `Array(Point)` column, a bare `[lon, lat]` is one point, as Elasticsearch reads it. The rationale for RowBinary as the default: schema-reflected RowBinary leaves the server no row text to parse, cuts ClickHouse CPU, and gives full type support without requiring compile-time schemas. A JSON column's value is still JSON text the server parses. Records stay a `Map` from diff --git a/docs/configuration.md b/docs/configuration.md index 36fd12d..2f3b647 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -10,28 +10,39 @@ # Configuration -dfe-loader reads its config in four layers. Higher layers win. Every setting has a safe default, so a loader started with no config file boots. +dfe-loader reads its config in four layers, each applied over the one before, so a flat ENV variable beats a nested one, which beats the file, which beats the defaults. Every setting has a safe default, so a loader started with no config file boots. ```mermaid -flowchart TB - FLAT["1. Flat ENV (DFE_LOADER_CLICKHOUSE_HOSTS=a:8123,b:8123)"] - ENV["2. Nested ENV (DFE_LOADER_CLICKHOUSE__PROTOCOL=http)"] - YAML["3. The YAML file --config or DFE_LOADER_CONFIG names, else ./config.yaml"] - HARD["4. hard-coded defaults"] - FLAT --> ENV --> YAML --> HARD +flowchart LR + accTitle: L4 Detail -- Config cascade + accDescr: The loader starts from its hard-coded defaults, applies the YAML file over them, then the nested ENV variables, then the flat ENV variables, and validates the result. + + defaults["Hard-coded
defaults"] + file["YAML file"] + nested["Nested ENV"] + flat["Flat ENV"] + config(["Validated config"]) + + defaults --> file --> nested --> flat --> config + + classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A ``` -Nested ENV names are derived from the key: `clickhouse.protocol` -> `DFE_LOADER_CLICKHOUSE__PROTOCOL` (a double underscore for each level of nesting). A nested ENV value is parsed as a number, bool or list where it looks like one, so `DFE_LOADER_BUFFER__FLUSH_ROWS=50000` is a number. A secret field is the exception and is read as the exact string, so a numeric password or MaxMind account ID arrives as written. +*Figure 1. L4 Detail -- Config cascade. Each layer overrides the one to its left.* Up: [dfe-loader docs](README.md). + +| Layer | Example | +|---|---| +| YAML file | the file `--config` or `DFE_LOADER_CONFIG` names, else `./config.yaml` or `./config.yml` | +| Nested ENV | `DFE_LOADER_CLICKHOUSE__PROTOCOL=http` | +| Flat ENV | `DFE_LOADER_CLICKHOUSE_HOSTS=a:8123,b:8123` | -A file that `--config` or `DFE_LOADER_CONFIG` names must exist. A missing one refuses startup rather than running on the defaults. +Nested ENV names are derived from the key: `clickhouse.protocol` -> `DFE_LOADER_CLICKHOUSE__PROTOCOL` (a double underscore for each level of nesting). A nested ENV value is parsed as a number, bool or list where it looks like one, so `DFE_LOADER_BUFFER__FLUSH_ROWS=50000` is a number. A secret field is the exception and is read as the exact string, so a numeric password or MaxMind account ID arrives as written. -A `.env` is read only when `--config` names no file, and then only the one in the working directory, into the environment the two ENV layers read. A `.env` in a parent directory belongs to another project and is never loaded. +A file that `--config` or `DFE_LOADER_CONFIG` names must exist. A missing one stops startup rather than running on the defaults. -scalo's own cascade, which builds the scaling engine KEDA reads, takes the `--config` file and the nested ENV form too (`DFE_LOADER_SCALING__ENABLED` or `DFE_LOADER__SCALING__ENABLED`), so `scaling.enabled` and `scaling.memory_gate_threshold` set either way reach it. +A `.env` is read when `--config` names no file, and then just the one in the working directory, into the environment the two ENV layers read. A `.env` in a parent directory belongs to another project and is never loaded. -> A config value is only useful if the ACTION matches it. The integration tests -> assert the observable outcome (the row written, the transport used), not just -> that the value parsed -- see [the test note](#verifying-config-drives-behaviour). +scalo's cascade, which builds the scaling engine KEDA reads, takes the `--config` file and the nested ENV form as well (`DFE_LOADER_SCALING__ENABLED` or `DFE_LOADER__SCALING__ENABLED`), so `scaling.enabled` and `scaling.memory_gate_threshold` set either way reach it. ## Settings that change behaviour @@ -41,11 +52,11 @@ scalo's own cascade, which builds the scaling engine KEDA reads, takes the `--co |---------|--------|--------| | `clickhouse.protocol` | `http` (default), `native` | HTTP (8123), HTTPS (8443), or the native protocol (9000, 9440 with TLS) through `dfe_clickhouse::TcpClient` | | `clickhouse.insert_format` | `row_binary` (default), `json_each_row` | binary, so the server parses no row text, vs self-describing JSON | -| `clickhouse.hosts` | list | `native` fails over across every host, round-robin. `http` uses the first host only. With TLS, `native` verifies each host against its own name | +| `clickhouse.hosts` | list | `native` fails over across every host, round-robin. `http` uses the first host alone. With TLS, `native` verifies each host against its name | | `clickhouse.database` | string, `dfe` (default) | database the connection opens on, and the one the readiness probe reads `system.tables` and `system.columns` for | `clickhouse.database` is the CONNECTION database and is separate from -`routing.default_db` (also `dfe`), which is the database a routed row lands in. +`routing.default_db` (default `dfe` as well), which is the database a routed row lands in. Both are cascade-overridable like any other setting -- `DFE_LOADER_CLICKHOUSE__DATABASE` or `clickhouse.database` in yaml. @@ -86,7 +97,7 @@ Idle HTTP connections close on the client side after 2 s. | Setting | Values | Effect | |---------|--------|--------| -| `metadata.capture_mode` | `full` (default), `json_only`, `raw_only`, `extracted_only` | `_json` + `_raw` population -- see [architecture.md](architecture.md#capture-modes) | +| `metadata.capture_mode` | `full` (default), `json_only`, `raw_only`, `extracted_only` | `_json` + `_raw` population -- see [pipeline/capture-modes.md](pipeline/capture-modes.md) | | `metadata.table_capture_modes` | map | per-table override of `capture_mode` | | DDL `@capture_mode: ` | comment tag | highest-priority override, per table | | `metadata.capture_raw` | `true` (default), `false` | whether `metadata.raw_source_fields` may fill `_raw` | @@ -98,11 +109,11 @@ Precedence, highest wins: **DDL tag > per-table > global**. | Setting | Effect | |---------|--------| -| `routing.table_fields` | fields whose value sets the table (dot-notation for nested). A null or empty value counts as absent | +| `routing.table_fields` | fields with a value that sets the table (dot-notation for nested). A null or empty value counts as absent | | `routing.default_db` / `default_table` | where a record lands when neither a field nor its topic names a table, or the table it names does not exist | | `routing.org_id_field` / `default_org_id` | where the `_org_id` tenant is read from, and the tenant a record with none lands in (`default`) | | `routing.org_routes` | per-org database routing, keyed on the tenant | -| `routing.external_topics` | topics an external pipeline writes directly. A record there keeps its `_org_id` and `_source` only when they are the source's own or on the topic's allow-list ([routing](pipeline/routing.md#external-pipeline-topics)) | +| `routing.external_topics` | topics an external pipeline writes directly. A record there keeps its `_org_id` and `_source` when they are the source's configured values or on the topic's allow-list, and has them replaced otherwise ([routing](pipeline/routing.md#external-pipeline-topics)) | ### Loading dead letters @@ -111,15 +122,15 @@ Precedence, highest wins: **DDL tag > per-table > global**. | `dlq_load.topics` | `dfe_receiver_dlq`, `dfe_loader_dlq`, `dfe_archiver_dlq`, `dfe_fetcher_dlq`, `dfe_transform_dlq` | Kafka DLQ topics the loader subscribes to as well as its data topics. An empty list loads none | | `dlq_load.table` | `dlq` | the table every record from those topics lands in, whatever it names | -A dead letter carries the `_source` of the record it wraps, so routing it by its fields would send it back to the table that refused it. Each `dlq_load` topic routes to `dlq_load.table` instead, in the database `routing` picks for its tenant. The dlq table's `_json` column reads the entry's own `_json` key (`@source: _json`), so the original record lands there rather than the whole entry. +A dead letter holds the `_source` of the record it wraps, so routing it by its fields would send it back to the table that refused it. Each `dlq_load` topic routes to `dlq_load.table` instead, in the database `routing` picks for its tenant. The dlq table's `_json` column reads the entry's `_json` key (`@source: _json`), so the original record lands there rather than the whole entry. -A dead letter never falls back to the default table. When `dlq_load.table` does not exist, in the default database or a tenant's own, dead letters wait in the pending-schema buffer with their offsets uncommitted, and a batch of them whose insert failed is held for its table. Each retires at `schema.pending_max_age_secs`: a buffered one like any record whose table never resolved, and a held batch counted from when it was first held. +A dead letter never falls back to the default table. When `dlq_load.table` does not exist, in the default database or a tenant's database, dead letters wait in the pending-schema buffer with their offsets uncommitted, and a batch of them with a failed insert is held for its table. Each retires at `schema.pending_max_age_secs`: a buffered one like any record with a table that never resolved, and a held batch counted from when it was first held. -`dlq_load.table` holds dead letters alone. A record from any other topic whose routing names it, through `table_fields`, `source_to_table` or its topic, lands in the default table, as one naming an unknown table does. So `routing.default_table` cannot be `dlq_load.table` while `dlq_load.topics` names a topic. Startup refuses it, and so does a reload, checked against the `dlq_load` the process started with. +`dlq_load.table` holds dead letters alone. A record from any other topic with routing that names it, through `table_fields`, `source_to_table` or its topic, lands in the default table, as one naming an unknown table does. So `routing.default_table` cannot be `dlq_load.table` when `dlq_load.topics` lists a topic. Startup rejects it, and so does a reload, checked against the `dlq_load` the process started with. A record from a `dlq_load` topic that fails again, or retires that way, is dropped, never dead-lettered. It is counted in `pipeline_dead_letters_dropped_total{reason="dlq_load"}` and `dfe_loader_rows_lost_total`. Its offset commits, and the entry stays on the DLQ topic until retention. -Under auto-discovery each topic is included by name. An exclude that matches one, such as scalo's `_dlq$` or the loader's own `routing.dlq.topic`, is narrowed to let exactly those names through, so no other `*_dlq` topic is subscribed. An explicit `kafka.topics` list gains each `dlq_load` topic it lacks. +Under auto-discovery each topic is included by name. An exclude that matches one, such as scalo's `_dlq$` or the loader's `routing.dlq.topic`, is narrowed to let exactly those names through, so no other `*_dlq` topic is subscribed. An explicit `kafka.topics` list has each `dlq_load` topic it lacks added. The gRPC transport reads no Kafka topic, so a gRPC loader loads no dead letters and keeps its file DLQ. @@ -130,7 +141,7 @@ The gRPC transport reads no Kafka topic, so a gRPC loader loads no dead letters | `geoip`, `enrichment.reputation` / `risk_scoring` | inject enriched columns on the promoted row -- see [pipeline/enrichment.md](pipeline/enrichment.md) | | `buffer.flush_rows` / `flush_bytes` / `flush_age_secs` | per-table flush triggers: 20000 rows, the rows' input bytes reaching 64 MiB, or 5 s since the last flush, whichever comes first | -Insert retry, batch salvage and insert concurrency are not settings. Every build binary-splits a batch with a data error down to the bad rows and runs at most 8 inserts at once. Inside one flush a RowBinary insert retries up to 5 times with jittered backoff only to re-read a stale, missing or unreadable schema, and a JSONEachRow insert retries a transient failure the same way. A timeout, a dropped connection or a retryable server code returns the batch at once, and the held batch is retried as described in [operations.md](operations.md). +Insert retry, batch salvage and insert concurrency are not settings. Every build binary-splits a batch with a data error down to the bad rows and runs at most 8 inserts at once. Inside one flush a RowBinary insert retries up to 5 times with jittered backoff, and just to re-read a stale, missing or unreadable schema, and a JSONEachRow insert retries a transient failure the same way. A timeout, a dropped connection or a retryable server code returns the batch at once, and the held batch is retried as described in [operations.md](operations.md). `schema.refresh_on_error` (default `true`) decides whether a RowBinary insert that fails on schema drift drops the table's cached schema. With `false` the stale schema is kept until it expires -- see [clickhouse/schema-cache.md](clickhouse/schema-cache.md#mismatch-recovery). diff --git a/docs/deployment/publishing.md b/docs/deployment/publishing.md index 6fabbd2..266861a 100644 --- a/docs/deployment/publishing.md +++ b/docs/deployment/publishing.md @@ -15,23 +15,39 @@ pushed by CI on every release. dfe-infra's `versions.yaml` pins the image tag, i Both artefacts go to GHCR. The image is `ghcr.io/hyperi-io/dfe-loader` and the chart is pushed to `oci://ghcr.io/hyperi-io/charts`. CI authenticates with `GITHUB_TOKEN`, so no registry secret is needed. +One release build feeds both artefacts: the image wraps the built binary, and the chart comes from the deployment contract that binary emits. + ```mermaid -flowchart TB - REL["Release tag (vX.Y.Z)"] - BIN["CI build
linux/amd64 + linux/arm64 binaries"] - DOCK["Build Dockerfile
wraps pre-built binary"] - PUSH["Push image
version tags"] - EMIT["dfe-loader generate-artefacts
emits the deployment contract"] - HELM["Assemble thin chart on scalo-service
package + push OCI artifact"] - REG[(GHCR)] - - REL --> BIN --> DOCK --> PUSH --> REG - BIN --> EMIT --> HELM --> REG +flowchart LR + accTitle: L4 Detail -- Release artefacts + accDescr: A release request starts a CI build of linux amd64 and arm64 binaries. The Dockerfile wraps the binaries into an image, which is pushed with its version tags. The binary's generate-artefacts emits the deployment contract, from which a thin chart is assembled and pushed. Both land in GHCR. + + tag(["Release request"]) + build["Build binaries"] + image["Build image"] + contract["Emit contract"] + pushimage["Push image tags"] + chart["Assemble and
push chart"] + ghcr[("GHCR")] + + tag --> build + build --> image + build --> contract + image --> pushimage + contract --> chart + pushimage --> ghcr + chart --> ghcr + + classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A + classDef store fill:#DDF3EC,stroke:#00785A,stroke-width:1px,color:#0B3327 + class ghcr store ``` +*Figure 1. L4 Detail -- Release artefacts. A release request is a merge with a `Release: true` trailer or a release dispatch.* Up: [Deployment](README.md). + ## How it works -`.github/workflows/ci.yml` calls hyperi-ci's reusable `rust-ci.yml`. On a release it builds the `Dockerfile` for `linux/amd64` and `linux/arm64` and pushes the image with version tags. It also runs the built binary's `generate-artefacts` to emit the deployment contract, assembles a thin chart from it on the scalo-service library chart, and pushes that. No chart is committed to this repo. The `release:` block in `.hyperi-ci.yaml` turns both on: +`.github/workflows/ci.yml` calls hyperi-ci's reusable `rust-ci.yml`. On a release it builds the `Dockerfile` for `linux/amd64` and `linux/arm64` and pushes the image with version tags. CI runs the built binary's `generate-artefacts` to emit the deployment contract as well, assembles a thin chart from it on the scalo-service library chart, and pushes that. No chart is committed to this repo. The `release:` block in `.hyperi-ci.yaml` turns both on: ```yaml release: @@ -50,9 +66,7 @@ release: `library` moves with the scalo version in `Cargo.toml`: a scalo-service release ships the schema for, and renders, only the contract version its scalo release writes. To see the chart a release would ship, build the binary and run `hyperi-ci chart assemble --binary target/debug/dfe-loader --image ghcr.io/hyperi-io/dfe-loader:@sha256: --version `. It prints the chart directory it wrote. -The `Dockerfile` is generated from the app's deployment contract by scalo and copies in the binary CI has already built. Do not edit it by hand. Regenerate it with `dfe-loader --emit-dockerfile`. `tests/integration/helm_contract.rs` fails when the committed `Dockerfile` drifts from the contract. - -## Tags generated +The `Dockerfile` is generated from the app's deployment contract by scalo and copies in the binary CI built. Do not edit it by hand. Regenerate it with `dfe-loader --emit-dockerfile`. `tests/integration/helm_contract.rs` fails when the committed `Dockerfile` drifts from the contract. For a stable release `v`: @@ -61,7 +75,7 @@ For a stable release `v`: - `ghcr.io/hyperi-io/dfe-loader:sha-` A prerelease (e.g. `v-rc.1`) gets its version tag and `sha-` -only. It never moves `latest`, and nor does a stable release on an older line while a newer one holds it. +only. It never moves `latest`, and nor does a stable release on an older line when a later one holds it. After a release, check the tags: @@ -71,8 +85,6 @@ docker run --rm ghcr.io/hyperi-io/dfe-loader:v --help docker manifest inspect ghcr.io/hyperi-io/dfe-loader:v ``` -## Consumers - - dfe-infra deploys the published chart through Argo CD at the tag and digest `versions.yaml` names. - dfe-docker's compose service takes `${IMAGE_REGISTRY:-ghcr.io/hyperi-io}/dfe-loader:${DFE_LOADER_VERSION}`, which `make stack VERSION=X.Y.Z` pins, and publishes the metrics port on `${DFE_LOADER_PROMETHEUS_PORT:-9091}`. diff --git a/docs/deployment/schemas.md b/docs/deployment/schemas.md index 71c4d6a..0f2390a 100644 --- a/docs/deployment/schemas.md +++ b/docs/deployment/schemas.md @@ -18,17 +18,33 @@ loader by two steps, and nothing here needs a checkout of it. > **Canonical schema documentation:** > [`dfe-schemas`](https://github.com/hyperi-io/dfe-schemas) +The schema reaches the loader through ClickHouse alone: dfe-engine applies the dfe-schemas manifest, and the loader reads the tables that result. + ```mermaid flowchart LR - YAML["dfe-schemas
YAML definitions"] - APPLY["dfe-engine
applies the manifest at boot"] - CH[(ClickHouse system.columns)] - LOADER["dfe-loader
SchemaCache"] - - YAML --> APPLY -->|CREATE / ALTER| CH - CH -->|query on TTL| LOADER + accTitle: L4 Detail -- Where a table's schema comes from + accDescr: dfe-schemas holds the YAML definitions. dfe-engine applies the manifest at boot with CREATE and ALTER statements. The loader reads the result from ClickHouse system.columns into its schema cache, again each time an entry expires. + + yaml["dfe-schemas
YAML"] + engine["dfe-engine"] + clickhouse[("ClickHouse
system.columns")] + loader["dfe-loader
SchemaCache"] + + yaml -->|manifest| engine + engine -->|CREATE, ALTER| clickhouse + clickhouse -->|columns| loader + + classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A + classDef core fill:#DCEBF7,stroke:#0072B2,stroke-width:2px,color:#0B2A3F + classDef store fill:#DDF3EC,stroke:#00785A,stroke-width:1px,color:#0B3327 + classDef ext fill:#EEF0F2,stroke:#6B7280,stroke-width:1px,stroke-dasharray:4 3,color:#1F2937 + class yaml,engine ext + class clickhouse store + class loader core ``` +*Figure 1. L4 Detail -- Where a table's schema comes from. Dashed boxes are other DFE repos.* Up: [Deployment](README.md). + That is why the loader has no submodule and no bundled profile copies: whatever the applier put in the table IS the contract, and a second copy on disk could only disagree with it. @@ -52,7 +68,7 @@ wins over a COMMENT -- the exact resolution order is in |-----------|--------------|--------| | `@skip` | `skip: true` | Omit the column from the insert | | `@default:value` | `default:` | Substitute when the column is null or absent | -| `@source:path` | `renamed:` | Source field path(s) for this column, and a `\|` fallback also fills `default:` | +| `@source:path` | `renamed:` | Source field path(s) for this column, and a `\|` fallback fills `default:` | | `@renamed:path` | `renamed:` | Source field path(s) for this column | | `@computed:expr` | `computed:` | CEL expression producing the value | | `@coerce:category` | `coerce:` | None: the encoder's coercion rules are fixed. The DDL directive logs a warning, the config key is refused | @@ -61,7 +77,7 @@ wins over a COMMENT -- the exact resolution order is in unknown directives (`DIRECTIVE_NAMES` in [`src/column_meta/mod.rs`](../../src/column_meta/mod.rs) lists only the six directives above). What makes the loader omit a value for a `@generated` -column is ClickHouse's own DEFAULT/MATERIALIZED/ALIAS clause on it, reported +column is ClickHouse's DEFAULT/MATERIALIZED/ALIAS clause on it, reported as `default_kind` in `system.columns` -- dfe-schemas' renderer writes that clause from the schema's `default:` and `attribute:` fields, not by parsing the `@generated:` text. `@config: path` documents that a mapping is driven by @@ -70,12 +86,8 @@ nothing reads the comment text itself. See [ddl-directives.md](../clickhouse/ddl-directives.md) for the whole expression language. -## Getting the tables there in the first place - dfe-engine applies the dfe-schemas manifest at boot: it takes the schema lock, renders the manifest for the platform it reaches, and creates what is absent or additively changed. Nothing else creates a DFE table, no Job and no init container. A loader pointed at a database the engine has not applied finds no columns: records wait in the pending-schema buffer, and a missing default table is logged at `warn`. -## Related docs - - [common-header.md](../transport/common-header.md) -- the header columns and what the loader writes into each - [ddl-directives.md](../clickhouse/ddl-directives.md) -- field mapping expression language - [dfe-schemas](https://github.com/hyperi-io/dfe-schemas) -- canonical schema documentation diff --git a/docs/feature-flags.md b/docs/feature-flags.md index 143a8ea..583d87a 100644 --- a/docs/feature-flags.md +++ b/docs/feature-flags.md @@ -10,28 +10,22 @@ # Feature flags -dfe-loader ships as a binary, its library target serving that binary, the tests and the benches, so it carries far fewer features than scalo. The defaults are empty: everything the running loader needs is always compiled, and the flags exist for the allocator, test scaffolding and the PGO build. - -## Loader features +dfe-loader ships as a binary, its library target serving that binary, the tests and the benches, so it has far fewer features than scalo. The defaults are empty: everything the running loader needs is always compiled, and the flags exist for the allocator, test scaffolding and the PGO build. | Feature | Default | Pulls in | Purpose | |---------|---------|----------|---------| | `jemalloc` | off | `tikv-jemallocator`, `tikv-jemalloc-ctl` | jemalloc allocator (DFE policy: jemalloc at every channel, no mimalloc) | | `transport-memory` | off | `scalo/transport-memory` | in-process Memory transport for unit tests | -| `testcontainers` | off | -- | enable the integration tests that run against real ClickHouse and Kafka containers, see `tests/TESTING.md` | +| `testcontainers` | off | -- | enable the integration tests that run against ClickHouse and Kafka containers, see `tests/TESTING.md` | | `pgo-driver` | off | `rdkafka` | build the PGO workload producer binary, leaving the main loader binary unaffected | `default = []`. hyperi-ci adds `jemalloc` to every release-track build, and `.hyperi-ci.yaml` lists `testcontainers`, `jemalloc` and `transport-memory` among the feature sets its tests compile. See [performance/overview.md](performance/overview.md) for what each channel builds. -## The ClickHouse client is not a feature - The loader does not feature-gate its ClickHouse client. It always builds the official `clickhouse` crate from crates.io, with the `lz4`, `time`, `inserter` and rustls TLS features, plus `dfe-clickhouse` with `ext` (the server-exception parser), `tls` (the native-protocol client) and `dynamic` (the dynamic RowBinary insert). Neither is patched or taken from git. See [clickhouse/clickhouse-ext.md](clickhouse/clickhouse-ext.md). -## scalo features - The bulk of the loader's capability surface comes from scalo (config cascade, logging, metrics, health, shutdown, transports, memory guard, CLI, top). Those are selected in the `scalo` dependency entry, pinned to a diff --git a/docs/operations.md b/docs/operations.md index 488f3f8..aacd068 100644 --- a/docs/operations.md +++ b/docs/operations.md @@ -30,7 +30,7 @@ scalo). The default subcommand is `run`. Loader settings take `DFE_LOADER_*` environment variables: flat (`DFE_LOADER_CLICKHOUSE_HOSTS`) or nested on a double underscore -(`DFE_LOADER_CLICKHOUSE__PROTOCOL`). Only scalo's own CLI flags have generic +(`DFE_LOADER_CLICKHOUSE__PROTOCOL`). scalo's CLI flags alone have generic env names (`LOG_LEVEL`, `METRICS_ADDR`). See [configuration.md](configuration.md). @@ -47,13 +47,13 @@ See [clickhouse/insert-formats.md](clickhouse/insert-formats.md). ## Observe -- Prometheus text on `/metrics`: scalo's DFE metric groups, the older names such as `buffer_rows` and `batches_flushed_total` emitted beside them, consumer stats and config-reload counters. On the gRPC transport there are no broker offsets, so `kafka_offsets_committed_total` and the consumer offset counters stay flat while the transport counters move. +- Prometheus text on `/metrics`: scalo's DFE metric groups, the older names such as `buffer_rows` and `batches_flushed_total` emitted beside them, consumer stats and config-reload counters. On the gRPC transport there are no broker offsets, so `kafka_offsets_committed_total` and the consumer offset counters stay flat as the transport counters move. - `dfe-loader metrics-manifest` prints every metric the loader registers, so a dashboard or scaling rule can be written against it rather than a running pod. The per-table series appear on first use and are not in it: `dfe_loader_unknown_table_fallback_total`, `dfe_loader_permanent_reject_total`, `loader_messages_by_table_total`, `loader_json_max_paths_exceeded_total`, `loader_buffer_rows_by_table` and `loader_buffer_bytes_by_table`. - A message carrying several records is split before anything counts it, so `messages_received_total` and `events_per_second` read in records, like `rows_inserted_total`. `dfe_loader_batched_array_*` counts JSON arrays and `dfe_loader_batched_ndjson_*` newline-separated records, so the pair that moves names the producer that batches. - A failing CEL expression leaves its column unset and the record goes on, counted in `dfe_loader_computed_eval_failures_total{table, column, stage}`. See [clickhouse/ddl-directives.md](clickhouse/ddl-directives.md). - `dfe-loader top` renders the metrics as a TUI. `--once` prints one scrape to stdout, `--json` prints it as JSON, and `--filter` takes a name regex. - `/livez` and `/readyz` come from scalo's health pillar. `/readyz` stays 503 until ClickHouse accepts the configured user and answers reads of `system.tables` and `system.columns` for the configured database, and `pipeline_ready` reads 1 once it passes. Each refusal is logged with ClickHouse's code (516 for a wrong password) and retried on the `sink_stack` backoff, doubling from `sink_stack.min_backoff_ms` (100 ms) to `sink_stack.max_backoff_ms` (30 s) plus up to as much again in jitter. Until it passes the loader joins no consumer group and opens no gRPC listener, so a pod with a wrong password takes no partitions from the pods that can write. Kafka and the gRPC peer play no part in readiness. -- A transport that cannot start, such as Kafka with no broker reachable, is retried on the same backoff, logged at `warn` and counted in `pipeline_retries_total{stage="transport_start"}`. A config the transport refuses exits, and one attempt can take the 10 s Kafka metadata timeout, so a shutdown may wait that long. +- A transport that cannot start, such as Kafka with no broker reachable, is retried on the same backoff, logged at `warn` and counted in `pipeline_retries_total{stage="transport_start"}`. A config the transport rejects exits, and one attempt can take the 10 s Kafka metadata timeout, so a shutdown may wait that long. Hot-reload rules are in [configuration.md](configuration.md#hot-reload). @@ -63,25 +63,25 @@ The insert tail degrades rather than stalls: - A data error binary-splits the batch down to the bad rows, which go to the DLQ, and the rest land. - A flush cycle in which every insert failed opens the circuit gate on the scaling signal (`scaling_circuit_open`), which zeroes the pressure KEDA reads, since more pods cannot relieve a dead ClickHouse. The next successful insert closes it. A failed insert never sends its batch to the DLQ. -- A column dropped or retyped by an out-of-band `ALTER` is drift: the cached schema is dropped and the retry re-encodes against a fresh read. A server code on the retry list (a busy server, the network, Keeper, the disk or object store under the table, or an operator fix such as a missing table or grant) holds the batch. Any other code against a table that has not changed is the row's own data, so salvage dead-letters that row. See [clickhouse/schema-cache.md](clickhouse/schema-cache.md). +- A column dropped or retyped by an out-of-band `ALTER` is drift: the cached schema is dropped and the retry re-encodes against a fresh read. A server code on the retry list (a busy server, the network, Keeper, the disk or object store under the table, or an operator fix such as a missing table or grant) holds the batch. Any other code against a table that has not changed is blamed on the row, so salvage dead-letters that row. See [clickhouse/schema-cache.md](clickhouse/schema-cache.md). - Kafka offsets commit once per cycle, after the inserts and the DLQ hand-over. On each partition the commit stops below the lowest offset not placed yet: a buffered row, a row waiting on its schema, a failed batch, or a dead letter the DLQ refused. The loader holds a failed batch or refused dead letter and retries it with jittered backoff until it lands. A dead letter over a Kafka-only DLQ's `message.max.bytes` is dropped and counted in `pipeline_dead_letters_dropped_total{reason="too_large"}`, so it never holds a commit for good. Rows above the floor re-deliver as duplicates after a restart. -- A held batch whose table ClickHouse has since reported absent is retried against the default table and counted in `dfe_loader_unknown_table_fallback_total`. A routing reload that removes the batch's source makes the loader ask about its table at once. Any other dropped table is found gone at its next schema read, up to about `schema.cache_ttl_secs` plus the 60 s refresh interval, and the partition's commit stays below the held batch until then. +- A held batch, other than one bound for `dlq_load.table`, is retried against the default table once ClickHouse reports its table absent, and counted in `dfe_loader_unknown_table_fallback_total`. A routing reload that removes the batch's source makes the loader ask about its table at once. Any other dropped table is found gone at its next schema read, up to about `schema.cache_ttl_secs` plus the 60 s refresh interval, and the partition's commit stays below the held batch until then. Readiness ignores an outage, since flipping it would pull every loader out of its Service at once. `clickhouse_connected` reports it: 1 once ClickHouse answered the startup probe or the latest insert, 0 once an insert went unanswered. On Kafka the held batches' reservations grow until the memory governor pauses the assigned partitions, and on gRPC a held Push is answered `UNAVAILABLE`. ## Rebalances -A roll starts the new pod while the old one still buffers rows, and the rebalance hands some of the old pod's partitions to the new one, which reads them again from the committed offset. Before every write the loader checks the lease each partition's rows were read under. A partition whose lease has ended has its rows discarded -- from the buffers, the pending-schema buffer, the holds and the dead letters -- so only the new owner writes them. Each discarded record counts in `transport_revoke_discarded_total{stage="buffer"}`. +A roll starts the incoming pod as the old one still buffers rows, and the rebalance hands some of the old pod's partitions to the incoming one, which reads them again from the committed offset. Before every write the loader checks the lease each partition's rows were read under. A partition with an ended lease has its rows discarded -- from the buffers, the pending-schema buffer, the holds and the dead letters -- so the next owner alone writes them. Each discarded record counts in `transport_revoke_discarded_total{stage="buffer"}`. With `kafka.acknowledgements.enabled: false` offsets commit at receipt, so the buffered rows are their only copy and nothing is discarded. ## DLQ -Dead-lettered messages go to the DLQ (Kafka first, file fallback, file only on gRPC). Payloads that are not JSON or fail processing, records whose schema never arrived within `schema.pending_max_age_secs`, and salvaged bad rows land there with their reason. A routing miss does not: it lands in the default table. DLQ and consumer errors are debounced in the logs so a burst does not flood. +Dead-lettered messages go to the DLQ (Kafka first, file fallback, file only on gRPC). Payloads that are not JSON or fail processing, records with a schema that never arrived within `schema.pending_max_age_secs`, and salvaged bad rows land there with their reason. A routing miss does not: it lands in the default table. DLQ and consumer errors are debounced in the logs so a burst does not flood. -A JSON record nested deeper than 64 levels is dead-lettered as it arrives, before any stage parses it, with the reason `payload nesting exceeds the maximum parse depth of 64`, and counted on `validation_failures_total{reason="out_of_range"}`. A batch too deep to split goes to the DLQ whole. The bound is the one scalo's parse path uses. +A JSON record nested deeper than 64 levels is dead-lettered as it arrives, before any stage parses it, with the reason `payload nesting exceeds the maximum parse depth of 64`, and counted on `validation_failures_total{reason="out_of_range"}`. A batch nested past that depth goes to the DLQ unsplit. The bound is the one scalo's parse path uses. -A record whose top-level object carries `_org_id` or `_source` more than once, counting a spelling that only decodes to the key, such as one with a unicode escape for the underscore, is dead-lettered the same way on every pipeline mode, with the reason `payload carries the _org_id header key more than once`, and counted on `dfe_loader_duplicate_header_total`. JSON readers disagree on which of the two values wins, so the loader routes neither. A record with no `\` byte and at most one quoted spelling of each key is passed without a scan. +A record with `_org_id` or `_source` more than once in its top-level object, counting a spelling that decodes to the key, such as one with a unicode escape for the underscore, is dead-lettered the same way on every pipeline mode, with the reason `payload carries the _org_id header key more than once`, and counted on `dfe_loader_duplicate_header_total`. JSON readers disagree on which of the two values wins, so the loader routes neither. A record with no `\` byte and at most one quoted spelling of each key goes through without a scan. ## Memory pressure @@ -91,8 +91,8 @@ risking an OOM kill mid-batch. The figure it checks is what the kernel charges the loader. That is the cgroup's `memory.current` (v1 `memory.usage_in_bytes` on an older host), else `VmRSS` from `/proc/self/status`. The cgroup figure is the one the OOM killer compares against `memory.max`, and it counts what jemalloc never sees: librdkafka's allocations, thread stacks and pages jemalloc keeps after a free. Only where neither can be read, such as a macOS dev build, does a `jemalloc` build register jemalloc's live heap instead, so the guard is not left counting only the batch engine's reservations. The `usage_source` field on the `memory guard initialised` log line names the source in force: `cgroup-v2`, `cgroup-v1` or `proc-status` on Linux, `explicit` only in that fallback. -`memory.current` also counts reclaimable page cache, such as the mmap'd MaxMind database or a file DLQ, so the brake can engage a little early. That is the safe direction. +`memory.current` counts reclaimable page cache as well, such as the mmap'd MaxMind database or a file DLQ, so the brake can engage a little early. That is the safe direction. ## Scaling -The contract's ScaledObject scales on CPU and carries a disabled Kafka lag trigger, so raw consumer lag never scales the loader. In a DFE deployment, dfe-infra's values set the CPU target to 70% and add the scaling-pressure trigger (target 70, read through dfe-keda-shim). The cooldown is the contract's 300 s. The slim profile clears the pressure trigger, so slim scales on CPU alone. The engine's [KEDA scaling page](https://github.com/hyperi-io/dfe-engine/blob/main/docs/deployment/keda-scaling.md) covers the triggers for every app. +The contract's ScaledObject scales on CPU and has a disabled Kafka lag trigger, so raw consumer lag never scales the loader. In a DFE deployment, dfe-infra's values set the CPU target to 70% and add the scaling-pressure trigger (target 70, read through dfe-keda-shim). The cooldown is the contract's 300 s. The slim profile clears the pressure trigger, so slim scales on CPU alone. The engine's [KEDA scaling page](https://github.com/hyperi-io/dfe-engine/blob/main/docs/deployment/keda-scaling.md) covers the triggers for every app. diff --git a/docs/performance/history.md b/docs/performance/history.md index c20d473..86d99f7 100644 --- a/docs/performance/history.md +++ b/docs/performance/history.md @@ -16,7 +16,7 @@ parser, which insert encoding, and why. Each entry records a targeted bake-off a For the current insert path and the dynamic encoder, see [../clickhouse/insert-formats.md](../clickhouse/insert-formats.md) and [../clickhouse/clickhouse-ext.md](../clickhouse/clickhouse-ext.md). This doc is -the "why we landed here", not the "how it works now". +the "why we landed here", not the "how it works". ## Why RowBinary is the default @@ -27,7 +27,7 @@ The dynamic encoder turns one `Map` plus its reflected schema int JSONEachRow is retained as a fallback, not a throughput choice. It is self-describing and forgiving -- the server coerces types and tolerates schema drift -- which makes it the right tool for diagnostics ("does the row land at -all if the server does the typing?") and for the rare table whose types the +all if the server does the typing?") and for the rare table with types the dynamic encoder does not yet cover. ### clickhouse-arrow -- trialled, dropped @@ -36,7 +36,7 @@ An early trial used `clickhouse-arrow` (Arrow IPC columnar inserts) as a candidate fast path. It was dropped: RowBinary matched Arrow throughput for this workload because insert time from flush to answer is dominated by the network, not the encode, so the columnar advantage did not show up at the pipeline level. -RowBinary also kept the encoder a single row-wise/per-column path off the same +RowBinary kept the encoder a single row-wise/per-column path off the same reflected schema, with no extra Arrow dependency or schema-mapping layer. ## JSON parser selection @@ -56,20 +56,20 @@ Bench: `benches/bakeoff.rs`. ### Mison -- evaluated, rejected (2025-Q4) -Mison gave a 4-7% throughput improvement over sonic-rs in targeted benchmarks, too little to justify maintaining a separate codebase: it needed a fork with custom Rust bindings. All mison code was deleted in commit `2a7a635`. The bench lived in a separate bake-off repo, not kept, and these results are its only record. +Mison gave a 4-7% throughput improvement over sonic-rs in targeted benchmarks, not enough to justify maintaining a separate codebase: it needed a fork with custom Rust bindings. All mison code was deleted in commit `2a7a635`. The bench lived in a separate bake-off repo, not kept, and these results are its only record. ### simd-json (0.17) -- evaluated, rejected (2026-03-11) **Bench:** a `simdjson_spike` bench, not retained -- flat30 and nested payloads, 15/30 schema columns, batch sizes 100/1K/10K. -**Results (batch=10K, measured).** Extraction: +**Results (batch=10K).** Extraction: | Approach | flat/30col | flat/15col | nested/15col | Notes | |---|---|---|---|---| | `sonic_selective` -- `get_from_slice` x N | 196 ms | 59.7 ms | 179 ms | Was original impl | | `sonic_dom` -- `from_slice` x 1 + `.get()` x N | **71.7 ms** | **58.1 ms** | **42.8 ms** | **Current impl** | -| `simd_dom+clone` -- simd-json + mandatory clone | 48.2 ms | 41.8 ms | 37.6 ms | Requires `Vec` clone | +| `simd_dom+clone` -- simd-json + mandatory clone | 48.2 ms | 41.8 ms | 37.6 ms | Needs a `Vec` clone | Routing (full DOM parse, batch=10K): @@ -89,11 +89,11 @@ Key observations: **Rejection reasons:** -1. A mandatory clone. simd-json requires `&mut [u8]` (in-place string +1. A mandatory clone. simd-json needs `&mut [u8]` (in-place string unescaping), and the DFE pipeline holds payloads as `Arc<[u8]>` for the `_json` splice, so every parse would need `raw.to_vec()`, a full memcpy per message. -2. Too small a net gain. Against `sonic_dom` (single full parse, +2. A net gain below the bar. Against `sonic_dom` (single full parse, same dependency), simd-json was ~28-33% faster for flat extraction and ~12% for nested. Extraction was <10% of total pipeline time (dominated by 40-75 ms network I/O per batch), so the net pipeline improvement was ~2-3%, @@ -103,14 +103,14 @@ Key observations: **Action taken:** `HeaderExtractor` switched from `get_from_slice` x N to `sonic_rs::from_slice` + O(1) hash lookups -- a 2.7-4.2x extraction speedup -(payload/schema dependent), zero new dependencies (commit `7d17a8d`). +(payload/schema dependent), no added dependencies (commit `7d17a8d`). ## Durable design decisions The choices below predate the current module layout but remain the reasons the hot path is built the way it is. -Each destination `db.table` has its own row buffer. An INSERT goes to one table, so a batch holds one table's rows. High-volume tables flush more often, and low-volume tables wait for the age trigger. A table's failed batch holds back the commit of only the partitions its rows came from. Each row is a `Map`, so the buffer layer needs no fixed schema, and rows accumulate only until a flush threshold (rows, bytes or age). +Each destination `db.table` has a separate row buffer. An INSERT goes to one table, so a batch holds one table's rows. High-volume tables flush more often, and low-volume tables wait for the age trigger. A table's failed batch holds back the commit of only the partitions its rows came from. Each row is a `Map`, so the buffer layer needs no fixed schema, and rows accumulate only until a flush threshold (rows, bytes or age). `_json` is never inserted into the promoted `Map`. The payload bytes are copied once into a reference-counted `Arc<[u8]>` and spliced as `_json` at serialise time, never re-serialised. That is why the extractor promotes only the schema's columns instead of flattening the whole payload. diff --git a/docs/performance/overview.md b/docs/performance/overview.md index d7d666f..680f4c2 100644 --- a/docs/performance/overview.md +++ b/docs/performance/overview.md @@ -12,23 +12,33 @@ Performance work splits in two: build-time optimisations (allocator, LTO, PGO, BOLT) that hyperi-ci applies, and runtime tuning (batch sizes) that lives in config. This page covers both, plus the tools for finding where the time goes. +The build-time choices go into the binary once, and the batch settings tune the running loader without a rebuild. + ```mermaid -flowchart TB - subgraph Build["Build-time (hyperi-ci)"] - ALLOC["jemalloc allocator"] - LTO["LTO (thin/fat)"] - PGO["PGO"] - BOLT["BOLT (after PGO)"] - end - subgraph Runtime["Runtime (config)"] - BATCH["Batch sizes
flush_rows / flush_bytes / flush_age_secs"] +flowchart LR + accTitle: L4 Detail -- Where performance is set + accDescr: hyperi-ci builds the binary with jemalloc, LTO, and on release PGO then BOLT. The batch flush settings in config tune the running loader. + + subgraph build["Build-time, hyperi-ci"] + direction LR + jemalloc["jemalloc"] ~~~ lto["LTO"] ~~~ pgo["PGO"] ~~~ bolt["BOLT"] end - BIN["Optimised binary"] - ALLOC & LTO & PGO & BOLT --> BIN - BIN --> RUN["Running loader"] - BATCH --> RUN + + binary["Optimised binary"] + batch["Batch settings
in config"] + loader(["Running loader"]) + + build --> binary + binary --> loader + batch --> loader + + classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A + classDef group fill:none,stroke:#64748B,stroke-width:1px,stroke-dasharray:2 4 + class build group ``` +*Figure 1. L4 Detail -- Where performance is set. BOLT runs on the PGO build.* Up: [Performance](README.md). + ## What CI builds hyperi-ci picks the optimisations by release channel (`src/hyperi_ci/languages/rust/optimize.py` in hyperi-ci): @@ -39,12 +49,10 @@ hyperi-ci picks the optimisations by release channel (`src/hyperi_ci/languages/r | beta | jemalloc | fat | no | | release | jemalloc | fat | PGO when the repo has a workload, BOLT with it unless `build.rust.optimize.bolt.enabled: false` | -This repo's workload is `scripts/pgo-workload.sh`, and `.hyperi-ci.yaml` builds its `pgo-driver` before the workload clock starts. The workload must exercise the real hot path: parse, transform, serialise and insert. Port checks, health probes and startup paths mislead the compiler and can make PGO a net loss. Confirm what shipped from the published artefact, not from the build log. hyperi-ci's guide is `docs/runtime/pgo-bolt.md` in its repo. +This repo's workload is `scripts/pgo-workload.sh`, and `.hyperi-ci.yaml` builds its `pgo-driver` before the workload clock starts. The workload must exercise the hot path a running loader takes: parse, transform, serialise and insert. Port checks, health probes and startup paths mislead the compiler and can make PGO a net loss. Confirm what shipped from the published artefact, not from the build log. hyperi-ci's guide is `docs/runtime/pgo-bolt.md` in its repo. `Cargo.toml` sets `lto = "thin"` and `codegen-units = 1` for `release`, and CI overrides the LTO mode per channel through `CARGO_PROFILE_RELEASE_LTO`, leaving `Cargo.toml` untouched. -## The allocator - The `jemalloc` feature installs `tikv-jemallocator` as the `#[global_allocator]` in `src/main.rs`. `default = []`, so a local build opts in: ```bash @@ -69,7 +77,7 @@ strings target/release/dfe-loader | grep -ciE 'jemalloc|je_mallctl' Insert concurrency is not a setting: at most 8 inserts run at once. RowBinary over HTTP (`FORMAT RowBinaryWithNamesAndTypes`) is the default, and `clickhouse.protocol: native` sends `FORMAT Native` blocks instead. dfe-clickhouse's [insert-formats.md](https://github.com/hyperi-io/dfe-clickhouse-rs/blob/main/docs/insert-formats.md) measures the two against each other. `insert_format: json_each_row` falls back to `FORMAT JSONEachRow` over HTTP. -## Profiling +To find where the time goes: ```bash # CPU, with perf diff --git a/docs/pipeline/README.md b/docs/pipeline/README.md index fc911fa..c345237 100644 --- a/docs/pipeline/README.md +++ b/docs/pipeline/README.md @@ -10,15 +10,36 @@ # Pipeline -The hot path: bytes in, ClickHouse row out, one pass per message, batched per -table on the way out. +The hot path: bytes in, ClickHouse row out. The pass runs per message, and the batching is per table: each table has a separate buffer, flushed as INSERTs of its rows. ```mermaid flowchart LR - M["bytes"] --> P["parse"] --> R["route"] --> X["extract"] - X --> E["enrich"] --> B["buffer"] --> I["insert"] + accTitle: L4 Detail -- Per-message pass, per-table batches + accDescr: Each message goes through one pass that parses, routes, extracts and enriches it. The row lands in the buffer of the table it was routed to, such as dfe.main or dfe.auth. Each table's buffer flushes as INSERTs into ClickHouse. + + pass["Per-message pass"] + + subgraph buffers["Per-table buffers"] + direction TB + main["dfe.main"] ~~~ auth["dfe.auth"] ~~~ other["Each other table"] + end + + insert["INSERTs
per table"] + clickhouse[("ClickHouse")] + + pass -->|rows| buffers + buffers -->|batches| insert + insert --> clickhouse + + classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A + classDef store fill:#DDF3EC,stroke:#00785A,stroke-width:1px,color:#0B3327 + classDef group fill:none,stroke:#64748B,stroke-width:1px,stroke-dasharray:2 4 + class clickhouse store + class buffers group ``` +*Figure 1. L4 Detail -- Per-message pass, per-table batches. Each buffer flushes on a row, byte or age trigger.* Up: [L3 Components -- dfe-loader](../README.md). + ## Docs - [overview.md](overview.md) -- the stages, in order. diff --git a/docs/pipeline/capture-modes.md b/docs/pipeline/capture-modes.md index 2ceb117..5ec1d09 100644 --- a/docs/pipeline/capture-modes.md +++ b/docs/pipeline/capture-modes.md @@ -17,7 +17,7 @@ governs the two payload-retention columns. | Mode | `_json` | `_raw` | Use | |------|---------|--------|-----| -| `full` (default) | full payload (JSON) | the record's own raw field, else NULL | path queries plus text search | +| `full` (default) | full payload (JSON) | a raw field the record sent, else NULL | path queries plus text search | | `json_only` | full payload (JSON) | NULL | path queries, never a `_raw` copy | | `raw_only` | NULL, unless a directive fills it | full source payload (String) | save ClickHouse CPU, no JSON-type overhead | | `extracted_only` | NULL, unless a directive fills it | NULL | promoted columns only | @@ -27,7 +27,7 @@ as a native ClickHouse JSON value (path-based queries). `_raw` is the original bytes as received (the tailed log line, DB row, or raw syslog). A table can keep one, both, or neither. -Under `full`, a JSON record fills `_raw` only from a raw field of its own (`raw` or `_raw`), kept as sent, such as a line the receiver captured. Any other JSON record is kept once, in `_json`, and `_raw` stays NULL. A text field such as `logoriginal` is not copied into `_raw` by default, because `_json` already stores its text. To promote one for a source that needs it, name it in `metadata.raw_source_fields` (first match wins) or in an `@renamed` directive on the `_raw` column. +Under `full`, a JSON record fills `_raw` from a raw field it sent (`raw` or `_raw`), kept as sent, such as a line the receiver captured. Any other JSON record is kept once, in `_json`, and `_raw` stays NULL. A text field such as `logoriginal` is not copied into `_raw` by default, because `_json` stores its text. To promote one for a source that needs it, name it in `metadata.raw_source_fields` (first match wins) or in an `@renamed` directive on the `_raw` column. `metadata.capture_raw` (default `true`) is what lets `raw_source_fields` fill `_raw`. `metadata.capture_json` is read by nothing: `capture_mode` decides `_json`. @@ -35,35 +35,38 @@ A directive on the `_json` column itself, such as `@source: record`, makes `_jso ## Precedence -Capture mode is resolved per table, highest wins: +Capture mode is resolved per table: the table's DDL tag wins, then a per-table config entry, then the global default. ```mermaid -flowchart TB - DDL["DDL @capture_mode comment
(per table, in ClickHouse)"] - TBL["metadata.table_capture_modes
(per table, in config)"] - GLOB["metadata.capture_mode
(global default)"] - DDL -->|if present| WIN["resolved mode"] - TBL -->|else if present| WIN - GLOB -->|else| WIN +flowchart LR + accTitle: L4 Detail -- Capture mode precedence + accDescr: A table whose COMMENT carries a capture_mode tag uses that mode. Otherwise a metadata.table_capture_modes entry for the table decides. Otherwise metadata.capture_mode applies. + + tag{"@capture_mode
tag?"} + entry{"table_capture_modes
entry?"} + tagmode(["The tag's mode"]) + entrymode(["The entry's mode"]) + global(["metadata.capture_mode"]) + + tag -->|no| entry + tag -->|yes| tagmode + entry -->|yes| entrymode + entry -->|no| global + + classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A + classDef decision fill:#FCF1D6,stroke:#9A6700,stroke-width:1px,color:#3D2A00 + class tag,entry decision ``` +*Figure 1. L4 Detail -- Capture mode precedence. The first question answered yes decides.* Up: [Pipeline](README.md). + - A `@capture_mode: ` tag in the table's `COMMENT` wins over any config, since the table owns its capture policy. See [../clickhouse/ddl-directives.md](../clickhouse/ddl-directives.md#table-level-tags). - `metadata.table_capture_modes` maps a `db.table` to a mode. - `metadata.capture_mode` applies to any table without a more specific setting. -## The setting drives the action - -A capture mode is only worth anything if the row that lands matches it. The -processor tests assert the observable action, not just that the config parsed -(`src/pipeline/processor.rs`), on a table with no directive on `_json`: +The processor tests in `src/pipeline/processor.rs` check the row each mode writes, on a table with no directive on `_json`: -- `full` -> the written row has `_json` populated, and `_raw` only when the record carried a raw field of its own. +- `full` -> the written row has `_json` populated, and `_raw` when the record sent a raw field. - `json_only` -> the written row has `_json` populated and `_raw` NULL, always. - `raw_only` -> the written row has `_raw` = the full payload and `_json` NULL. -- `extracted_only` -> the written row has neither `_json` nor `_raw`, only the - promoted columns. - -That is the rule for every config-cascade switch in the loader: the test proves -the cascade setting changes the real outcome (the columns written), so a -"turn it on/off" flag cannot silently no-op. See -[../configuration.md](../configuration.md#verifying-config-drives-behaviour). +- `extracted_only` -> the written row has neither `_json` nor `_raw`, just the promoted columns. diff --git a/docs/pipeline/enrichment.md b/docs/pipeline/enrichment.md index 18e4174..51d5712 100644 --- a/docs/pipeline/enrichment.md +++ b/docs/pipeline/enrichment.md @@ -10,48 +10,58 @@ # Enrichment -Enrichment adds derived columns to a row from its IP fields (`enrichment.ip_fields`, first match wins): geographic data, -reputation flags, and a composite risk score. Each provider is an independent -toggle, off by default. Enriched values are written as flat columns on the -promoted row only -- never folded into `_json`, so the payload stays the payload -and the enrichment stays queryable. +Status: each provider is off by default and turned on by its setting, `geoip.enabled`, `enrichment.reputation.enabled` or `enrichment.risk_scoring.enabled`. Reputation reads operator blocklists alone, so of its flags only `rep_is_malicious` is ever set ([#342](https://github.com/hyperi-io/dfe-loader/issues/342)). + +Enrichment adds columns to a row from the first IP field it has (`enrichment.ip_fields`, default `src_ip`, `client_ip`, `ip`, `source_ip`): GeoIP columns `geo_*`, reputation columns `rep_*` and a risk score in `risk_*`. They are written as flat columns on the promoted row, never folded into `_json`, so the payload stays as it arrived and the enrichment is queryable. + +The lookups key on the first IP field present, and risk scoring weighs what GeoIP and reputation found, so it adds nothing by itself. ```mermaid flowchart LR - IP["IP field(s) on the row"] - GEO["GeoIP
country, city, ASN"] - REP["Reputation
VPN / Tor / proxy / botnet"] - RISK["Risk scoring
weighted composite"] - ROW["promoted row (flat columns)"] - IP --> GEO --> ROW - IP --> REP --> ROW - GEO --> RISK - REP --> RISK --> ROW + accTitle: L4 Detail -- Enrichment + accDescr: The row from column extraction is looked up by its first IP field in GeoIP and in the reputation blocklists. Risk scoring weighs both results. The row goes on to its table's buffer with the geo, rep and risk columns added. + + extract["Extract columns"] + + subgraph lookups["Enrich"] + direction LR + geo["GeoIP"] --> risk["Risk scoring"] + rep["Reputation
blocklists"] --> risk + end + + buffer["Buffer per table"] + + extract -->|row| lookups + lookups -->|row| buffer + + classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A + classDef group fill:none,stroke:#64748B,stroke-width:1px,stroke-dasharray:2 4 + class lookups group ``` +*Figure 1. L4 Detail -- Enrichment. Each provider is off until its setting turns it on.* Up: [L3 Components -- dfe-loader](../README.md). + ## Providers | Provider | Config | What it adds | |----------|--------|--------------| | GeoIP | `geoip` | geographic lookup (country, city, ASN) from a local database under `geoip.auto_download.data_dir` | -| Reputation | `enrichment.reputation` | flags for VPN, Tor, proxy, and botnet membership | +| Reputation | `enrichment.reputation` | `rep_is_malicious` for an address on a file `blocklist_files` names, one IP or CIDR a line. No feed is loaded for the VPN, Tor, proxy and botnet flags, so nothing derives them, and a value the record sent is kept | | Risk scoring | `enrichment.risk_scoring` | weighted composite score from geo + reputation, with a selectable country-risk preset | -Risk scoring is a composite: it needs at least GeoIP or reputation enabled to -have anything to weigh. Enabling it alone does nothing useful, which the config -documents at the field. +Risk scoring runs only when GeoIP or reputation returned a result for the row, so enabling it alone writes nothing. Every value the scorer uses is config under `enrichment.risk_scoring`, with the shipped values as defaults. `config.example.yaml` lists them all. | Key | What it sets | Default | |-----|--------------|---------| -| `preset` | the entry in `presets` whose country lists apply | `global` | +| `preset` | the entry in `presets` with the country lists that apply | `global` | | `presets` | `high` and `medium` country lists by name | `global`, `us_enterprise`, `eu_enterprise`, `apac_enterprise`, `high_security` | | `weights` | `geo`, `reputation`, `privacy`, `threat` share of the composite | 15, 25, 25, 35 | | `scores` | what each signal adds to its component: country class, VPN, proxy, Tor, datacenter, relay, anonymiser count, botnet, malicious, spam, scanner, and the `high_abuse_score` threshold | `config.example.yaml` | | `levels` | lowest composite score of `low`, `medium`, `high`, `critical` | 20, 40, 60, 80 | -A `presets` entry in config replaces the shipped preset of that name, or adds a new one, and the rest stay as shipped. Startup and a reload refuse a `preset` that `presets` does not hold, naming the valid ones, as well as a score over 100 and level bands that do not rise strictly within 1-100. +A `presets` entry in config replaces the shipped preset of that name, or adds one, and the rest stay as shipped. Startup and a reload refuse a `preset` that `presets` does not hold, naming the valid ones, as well as a score over 100 and level bands that do not rise strictly within 1-100. ## Where the GeoIP databases come from @@ -68,17 +78,9 @@ With `geoip.enabled` on, startup provisions the MMDB files through scalo's `geoi The credentials are redacted in logs and config dumps. Supply them through the secrets layer, never as literals. -## Cost and placement - Enrichment runs after extraction and before buffering, on the promoted row. The lookups are cached, so a hot IP is resolved once and reused. Because the columns are flat and additive, a table opts in by declaring the enrichment columns in its schema -- an enriched value is written only where the target table has a column for it, the same promotion rule the rest of the pipeline follows. - -Like every config-cascade switch in the loader, an enrichment toggle is only -meaningful if the row that lands reflects it: enable GeoIP and the geo columns -on the written row are populated, leave it off and they are not. See -[../configuration.md](../configuration.md) and the hot-path context in -[overview.md](overview.md). diff --git a/docs/pipeline/overview.md b/docs/pipeline/overview.md index 852fc53..64e3f36 100644 --- a/docs/pipeline/overview.md +++ b/docs/pipeline/overview.md @@ -10,25 +10,32 @@ # The pipeline -A message becomes a ClickHouse row in one pass. The hot path does the cheap -work per message and defers the expensive work to flush: parse once, promote -only the columns the table declares, keep the full payload as a -reference-counted `Arc<[u8]>`, and splice it in as `_json` only at serialise -time. The tail batches per table and inserts. +A message becomes a ClickHouse row in one pass. The hot path does the cheap work per message and defers the expensive work to flush: parse once, promote only the columns the table declares, keep the full payload as a reference-counted `Arc<[u8]>`, and splice it in as `_json` only at serialise time. The tail batches per table and inserts. + +The pass is eight steps in two stages: the first decides which table a record goes to, and the second builds its row and writes it. ```mermaid -flowchart LR - M["bytes (Kafka/gRPC)"] - S["split batched messages"] - P["JSON parse"] - R["route -> db.table"] - X["extract header + schema columns"] - E["enrich"] - B["per-table buffer"] - I["insert + commit offsets"] - M --> S --> P --> R --> X --> E --> B --> I +flowchart TB + accTitle: L4 Detail -- Loader pipeline + accDescr: Two stages run in order. Read and route takes message bytes, splits batched messages, parses JSON and routes each record to a table. Extract and write extracts the header and schema columns, enriches, buffers per table, then inserts and commits offsets. + + subgraph s1["1 Read and route"] + direction LR + bytes(["Message bytes"]) --> split["Split batches"] --> parse["Parse JSON"] --> route["Route to table"] + end + subgraph s2["2 Extract and write"] + direction LR + extract["Extract columns"] --> enrich["Enrich"] --> buffer["Buffer per table"] --> insert(["Insert and commit"]) + end + s1 --> s2 + + classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A + classDef group fill:none,stroke:#64748B,stroke-width:1px,stroke-dasharray:2 4 + class s1,s2 group ``` +*Figure 1. L4 Detail -- Loader pipeline. Read each stage left to right, then down.* Up: [Pipeline](README.md). + The stages, in order, all in `src/pipeline/` (`MessageProcessor::process` is the per-message entry point): @@ -48,7 +55,7 @@ Each element keeps its source message's topic, partition and offset, so a batch still commits as one unit. What is not a batch of records goes through untouched, so the format check and the DLQ see exactly what arrived: a scalar array, an empty array, a single object however it is formatted, a truncated -tail, and any body whose elements are not all objects. +tail, and any body with elements that are not all objects. The split belongs at the consumer, so a producer's choice of wire format cannot lose records. An array that reaches the processor unsplit is refused as `payload is a JSON array, not a record object` rather than coming back as a ClickHouse encode error on an empty column. @@ -61,7 +68,7 @@ JSON is the only payload format. MessagePack, which DFE/XDR 2.0 and 2.1 accepted ## 2. Route -Routing picks `db.table` from the message before any flattening, using the configured `routing.table_fields` (dot-notation for nested access). No field carrying a value falls back to the topic's source, then `routing.default_table`, and a table ClickHouse says does not exist falls back to `routing.default_table` too. The database is `routing.default_db` unless `routing.org_routes` lists the record's `_org_id` tenant. See [routing.md](routing.md). +Routing picks `db.table` from the message before any flattening, using the configured `routing.table_fields` (dot-notation for nested access). No field carrying a value falls back to the topic's source, then `routing.default_table`, and a table ClickHouse says does not exist falls back to `routing.default_table` as well. The database is `routing.default_db` unless `routing.org_routes` lists the record's `_org_id` tenant. See [routing.md](routing.md). ## 3. Extract @@ -86,7 +93,7 @@ per-table config > global). See [capture-modes.md](capture-modes.md). ## 6. Insert and commit -The encoder reshapes only the values ClickHouse cannot take as sent: epoch vs ISO timestamps, hyphen-less UUIDs, integer IPv4, every geo_point form for a `Point`, and a single value for an `Array` (see [../clickhouse/types.md](../clickhouse/types.md)). Each table's buffer flushes on its own triggers (`flush_rows` / `flush_bytes` / `flush_age_secs`). The batch is encoded and inserted through `dfe-clickhouse`'s `dynamic` module (RowBinary by default). Kafka offsets commit once per flush cycle, and on each partition stop below the lowest offset not placed yet -- at-least-once. Batch salvage isolates a row ClickHouse rejects for good and sends it to the DLQ. A batch that fails for any other reason is held and retried with jittered backoff until it lands. See [../architecture.md](../architecture.md#resilience) and [../clickhouse/insert-formats.md](../clickhouse/insert-formats.md). +The encoder reshapes only the values ClickHouse cannot take as sent: epoch vs ISO timestamps, hyphen-less UUIDs, integer IPv4, every geo_point form for a `Point`, and a single value for an `Array` (see [../clickhouse/types.md](../clickhouse/types.md)). Each table's buffer flushes on its triggers (`flush_rows` / `flush_bytes` / `flush_age_secs`). The batch is encoded and inserted through `dfe-clickhouse`'s `dynamic` module (RowBinary by default). Kafka offsets commit once per flush cycle, and on each partition stop below the lowest offset not placed yet -- at-least-once. Batch salvage isolates a row ClickHouse rejects for good and sends it to the DLQ. A batch that fails for any other reason is held and retried with jittered backoff until it lands. See [../architecture.md](../architecture.md#resilience) and [../clickhouse/insert-formats.md](../clickhouse/insert-formats.md). ## Parallelism diff --git a/docs/pipeline/parallelism.md b/docs/pipeline/parallelism.md index ede1a4e..fffbbc4 100644 --- a/docs/pipeline/parallelism.md +++ b/docs/pipeline/parallelism.md @@ -10,26 +10,40 @@ # Pipeline parallelism -Each received batch goes through two phases in `src/pipeline/orchestrator.rs`. Per-message work with no shared mutable state runs in parallel. Everything that mutates state runs after it, in one thread. +Each received batch goes through two phases in `src/pipeline/orchestrator.rs`: per-message work with no shared mutable state runs in parallel, and everything that mutates state runs after it, in one thread. ```mermaid -flowchart TB - RECV["transport recv (batch)"] - PRE["pre-route (BatchEngine, when routing_field is set)"] - subgraph PAR["PARALLEL -- scalo worker pool"] - MP["MessageProcessor::process(&self, msg)"] +flowchart LR + accTitle: L4 Detail -- Batch phases + accDescr: A received batch passes the BatchEngine's pre-route filters when a routing field is set. The worker pool runs MessageProcessor process on each message in parallel. The BatchCoordinator then walks the results in one thread, pushing each Ok row into its table's buffer, parking a message that waits on its schema and turning any other error into a dead letter. + + recv(["Received batch"]) + preroute["Pre-route filters"] + + subgraph parallel["Parallel"] + process["MessageProcessor
process"] end - RES["Vec<Result<ProcessedMessage>>"] - subgraph SEQ["SEQUENTIAL -- BatchCoordinator"] - OK["Ok: cache entry, mark_pending, buffer push"] - ERR["Err: dead letter, memory release"] + + subgraph sequential["Sequential"] + push["Buffer push"] + dead["Dead letter"] end - RECV --> PRE --> MP --> RES --> OK - RES --> ERR + recv --> preroute + preroute --> process + process -->|Ok| push + process -.->|Err| dead + + classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A + classDef error fill:#FBE3D6,stroke:#C2410C,stroke-width:1px,color:#4A1A05 + classDef group fill:none,stroke:#64748B,stroke-width:1px,stroke-dasharray:2 4 + class dead error + class parallel,sequential group ``` -`MessageProcessor` (`src/pipeline/processor.rs`) holds only shared references, and its `process` method does no I/O and no `.await`. For one message it refuses a payload that does not open a JSON object or array, parses it with sonic-rs, holds an external topic's record to its own tenant, routes it to a table, then extracts and transforms the fields. It goes on to derive the capture settings, apply the field mapping, evaluate computed columns and CEL expressions, and run enrichment. The result is a `ProcessedMessage` (`src/pipeline/types.rs`) or an error. +*Figure 1. L4 Detail -- Batch phases. Pre-route runs only when the BatchEngine has a `routing_field`.* Up: [Pipeline](README.md). + +`MessageProcessor` (`src/pipeline/processor.rs`) holds only shared references, and its `process` method does no I/O and no `.await`. For one message it rejects a payload that does not open a JSON object or array, parses it with sonic-rs, holds an external topic's record to its source's tenant, routes it to a table, then extracts and transforms the fields. It goes on to derive the capture settings, apply the field mapping, evaluate computed columns and CEL expressions, and run enrichment. The result is a `ProcessedMessage` (`src/pipeline/types.rs`) or an error. What it reads is safe to share. `CaptureOverrides::derive_config` (`src/pipeline/capture.rs`) is a pure `&self` method. Enrichment (`src/pipeline/enrichment.rs`) is in-memory GeoIP, reputation and risk lookups, with the GeoIP cache behind a `parking_lot::RwLock`. @@ -41,4 +55,4 @@ The orchestrator runs `process` over the batch on the first pool it has: the `Ba The coordinator is built only after the processor is dropped, so the borrow checker keeps the two phases apart. -The pool and the engine are scalo's `AdaptiveWorkerPool` and `BatchEngine` (scalo-rs `src/worker/`, the `worker-batch` feature). The runtime builds both, and `src/main.rs` hands them to the orchestrator and gives the pool the orchestrator's memory guard. The pool reads the cascade's `worker_pool` section, whose keys and defaults are `WorkerPoolConfig` in scalo-rs `src/worker/config.rs`: thread bounds, the CPU thresholds it grows and shrinks at, a memory pressure cap, the scale interval and the async concurrency limit. [../architecture.md](../architecture.md) covers the layers around this hot path. +The pool and the engine are scalo's `AdaptiveWorkerPool` and `BatchEngine` (scalo-rs `src/worker/`, the `worker-batch` feature). The runtime builds both, and `src/main.rs` hands them to the orchestrator and gives the pool the orchestrator's memory guard. The pool reads the cascade's `worker_pool` section, with keys and defaults in `WorkerPoolConfig` in scalo-rs `src/worker/config.rs`: thread bounds, the CPU thresholds it grows and shrinks at, a memory pressure cap, the scale interval and the async concurrency limit. [../architecture.md](../architecture.md) covers the layers around this hot path. diff --git a/docs/pipeline/routing.md b/docs/pipeline/routing.md index f0f3486..78110df 100644 --- a/docs/pipeline/routing.md +++ b/docs/pipeline/routing.md @@ -10,61 +10,66 @@ # Routing -Routing picks the `db.table` a message lands in, before any flattening, from fields in the message itself. The first hop (dfe-receiver or dfe-fetcher) sets `_source` and `_org_id` with its rules, so the loader reads those two header fields and evaluates no rules of its own. +Routing picks the `db.table` a message lands in, before any flattening, from fields in the message itself. The first hop (dfe-receiver or dfe-fetcher) sets `_source` and `_org_id` with its rules, so the loader reads those two header fields and evaluates no rules itself. + +The table is the first of four answers that holds: a `dlq_load` topic's table, a `table_fields` value, the source a topic names, then `default_table`. ```mermaid -flowchart TB - MSG["parsed message"] - ORG{"org_routes lists the tenant?"} - PIN{"read from a dlq_load topic?"} - DLQTBL["dlq_load.table"] - TBF{"a table_fields value?"} - TOPIC{"topic is {source}_land / {source}_load?"} - DEFDB["default_db"] - ORGDB["the org's database"] - DEFTBL["default_table"] - SRC["the topic's source"] - OUT["db.table"] - - MSG --> ORG - ORG -->|yes| ORGDB --> OUT - ORG -->|no| DEFDB --> OUT - MSG --> PIN - PIN -->|yes| DLQTBL --> OUT - PIN -->|no| TBF - TBF -->|yes| OUT - TBF -->|no| TOPIC - TOPIC -->|yes| SRC --> OUT - TOPIC -->|no| DEFTBL --> OUT +flowchart LR + accTitle: L4 Detail -- Loader table choice + accDescr: A message read from a dlq_load topic goes to dlq_load.table. Otherwise a table_fields value names the table. Otherwise a topic named for a source gives that source's table. Otherwise the message goes to default_table. + + pin{"dlq_load
topic?"} + field{"table_fields
value?"} + topic{"Source
topic?"} + replay(["dlq_load.table"]) + value(["Value's table"]) + source(["Source's table"]) + fallback(["default_table"]) + + pin -->|no| field + pin -->|yes| replay + field -->|no| topic + field -->|yes| value + topic -->|yes| source + topic -->|no| fallback + + classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A + classDef store fill:#DDF3EC,stroke:#00785A,stroke-width:1px,color:#0B3327 + classDef decision fill:#FCF1D6,stroke:#9A6700,stroke-width:1px,color:#3D2A00 + class pin,field,topic decision + class source,fallback,value,replay store ``` +*Figure 1. L4 Detail -- Loader table choice. Green ends are tables.* Up: [L3 Components -- dfe-loader](../README.md). + +The database is the org's database when `org_routes` lists the tenant, else `default_db`. `source_to_table` then maps whichever table was chosen, a `dlq_load` one aside, and a table ClickHouse reports absent falls back to `default_table` as well. + ## Settings | Setting | Effect | |---------|--------| -| `routing.table_fields` | fields whose value selects the table, first one carrying a value wins. Dot-notation reaches nested fields (`event.category`). Default `["_source"]` | +| `routing.table_fields` | fields with a value that selects the table, the first one holding a value winning. Dot-notation reaches nested fields (`event.category`). Default `["_source"]` | | `routing.default_db` | database for every org `org_routes` does not list -- default `dfe` | -| `routing.default_table` | table used when neither a `table_fields` value nor the topic names one, or the table named does not exist -- default `main`, so an unrouted message lands in `dfe.main` | -| `routing.source_to_table` | maps a source value to the table its records land in, `{cisco-ios: network}` say -- default `{}`, so a source lands in the table of its own name | -| `routing.topic_suffixes` | suffixes that mark a topic as one source's own -- default `_land` / `_load`, so `cisco-ios_load` names the source `cisco-ios`. With `kafka.topics` and `kafka.topic_regex` unset, auto-discovery subscribes to the topics ending in one, so an empty list or an empty suffix is refused | +| `routing.default_table` | table used when neither a `table_fields` value nor the topic names one, or the table it names does not exist -- default `main`, so an unrouted message lands in `dfe.main` | +| `routing.source_to_table` | maps a source value to the table its records land in, `{cisco-ios: network}` say -- default `{}`, so a source lands in the table of the same name | +| `routing.topic_suffixes` | suffixes that mark a topic as one source's topic -- default `_land` / `_load`, so `cisco-ios_load` names the source `cisco-ios`. With `kafka.topics` and `kafka.topic_regex` unset, auto-discovery subscribes to the topics ending in one, so an empty list or an empty suffix is refused | | `routing.org_id_field` | field the tenant is read from -- default `_org_id`, the common-header field the first hop stamps | -| `routing.default_org_id` | tenant for a record that carries none -- default `default`, the deployment's default org | -| `routing.org_routes` | orgs that get a database of their own, keyed on the tenant: `[{org_id: acme, database: acme_dfe}]` | +| `routing.default_org_id` | tenant for a record that has none -- default `default`, the deployment's default org | +| `routing.org_routes` | orgs that get a separate database, keyed on the tenant: `[{org_id: acme, database: acme_dfe}]` | | `routing.external_topics` | topics an external pipeline writes straight to, each with the tenant policy below | ## How a field is resolved `table_fields` is an ordered list. The first field carrying a value wins, and the value is the table name. A null or empty value counts as absent, so `{"_source": ""}` falls through to the next field, then the topic, then `default_table`. It never reaches the DLQ. Dot-notation walks nested objects, so `table_fields = ["event.category"]` reads `category` inside the `event` object. -A record read from a `dlq_load` topic skips `table_fields` and lands in `dlq_load.table`. A dead letter carries the `_source` of the record it wraps, which would send it back to the table that refused it. Any other record that resolves to `dlq_load.table` lands in `default_table` instead. See [configuration](../configuration.md#loading-dead-letters). - -## The tenant +A record read from a `dlq_load` topic skips `table_fields` and lands in `dlq_load.table`. A dead letter holds the `_source` of the record it wraps, which would send it back to the table that refused it. Any other record that resolves to `dlq_load.table` lands in `default_table` instead. See [configuration](../configuration.md#loading-dead-letters). The `_org_id` column takes the record's `org_id_field` value and nothing else: no column directive and no sender field such as `org_id` can set it. An absent, null or empty value lands in `default_org_id`, so every row belongs to an org. The database is `default_db` unless `org_routes` lists that tenant. ## External pipeline topics -An external pipeline is a customer's own Vector, Fluentd or Logstash writing a source's Kafka topics directly. No dfe-receiver sits in front, so nothing has stripped a forged `_org_id` or `_source`, and the loader would write the record into whatever tenant and table it names. List the topic under `routing.external_topics` and the loader holds each record to the source's own values: +An external pipeline is a customer's Vector, Fluentd or Logstash writing a source's Kafka topics directly. No dfe-receiver sits in front, so nothing has stripped a forged `_org_id` or `_source`, and the loader would write the record into whatever tenant and table it names. List the topic under `routing.external_topics` and the loader holds each record to the source's configured values: ```yaml routing: @@ -81,15 +86,15 @@ On a listed topic: - `_org_id` absent, null, empty or outside `[org_id] + allowed_org_ids` is set to `org_id`. - `_source` absent, null, empty or outside `[source] + allowed_sources` is set to `source`. - A value inside the list is kept. -- Nothing is rejected or dead-lettered. The record lands, in the source's own tenant and table. +- Nothing is rejected or dead-lettered. The record lands, in the source's configured tenant and table. Each replacement increments `loader_external_org_id_replaced_total{topic}` or `loader_external_source_replaced_total{topic}`, so a pipeline that keeps sending the wrong value shows up as a rate. A topic not listed behaves as before. -The tenant is replaced at `org_id_field`. The source is replaced in `_source` and in every field `metadata.source_fields` and `table_fields` name, so changing those lists does not open a way round the policy. The `_json` and `_raw` captures hold the payload as the sender wrote it, forged values included. The columns carry the replaced ones. +The tenant is replaced at `org_id_field`. The source is replaced in `_source` and in every field `metadata.source_fields` and `table_fields` name, so changing those lists does not open a way round the policy. The `_json` and `_raw` captures hold the payload as the sender wrote it, forged values included. The columns hold the replaced ones. ## A routing edit applies without a restart -The loader watches its config file (`hot_reload.enabled`, on unless set false). On a change it rebuilds the router, the header extractor and the default table from the new `routing` and `metadata`. Startup pre-warms the schemas of the default table and every `source_to_table` target. A reload pre-warms the ones it holds no fresh schema for, and leaves a table already held absent to the recheck described below. A source table the pre-warm finds absent is not recorded absent: the engine writes a source's routing and creates its table together, so the source's first batch resolves the table from the pending-schema buffer instead. An absent default table is reported at once. Rows already buffered keep the table they were routed to, and no offset is committed for them until they land. `routing.dlq` is the exception: the DLQ is opened at startup, so a change there logs a restart warning. The deployment contract lists `routing` in `hot_paths`, so on Kubernetes an edit there reaches the running pods without rolling them. +The loader watches its config file (`hot_reload.enabled`, on unless set false). On a change it rebuilds the router, the header extractor and the default table from the edited `routing` and `metadata`. Startup pre-warms the schemas of the default table and every `source_to_table` target. A reload pre-warms the ones it holds no fresh schema for, and leaves a table held absent to the recheck described below. A source table the pre-warm finds absent is not recorded absent: the engine writes a source's routing and creates its table together, so the source's first batch resolves the table from the pending-schema buffer instead. An absent default table is reported at once. Rows in the buffers keep the table they were routed to, and no offset is committed for them until they land. `routing.dlq` is the exception: the DLQ is opened at startup, so a change there logs a restart warning. The deployment contract lists `routing` in `hot_paths`, so on Kubernetes an edit there reaches the running pods without rolling them. ## The topic names the source when the record does not @@ -101,21 +106,19 @@ a record arriving there with no `table_fields` value routes to `dfe.cisco-ios`. A topic names a source only when one of `routing.topic_suffixes` strips off it AND what is left does not resolve to `default_table`. Both halves matter: `main_land` strips to the landing table, so its records have no -source of their own and keep the default fallback. A topic with no suffix to +source of theirs and keep the default fallback. A topic with no suffix to strip is not `{source}_land` or `{source}_load` and names no source at all. `source_to_table` is applied before the comparison, so a source mapped onto the -landing table is landing traffic too. +landing table is landing traffic as well. Routing by topic covers for the producer but does not hide it: the record still -named no table, so `dfe_loader_routing_field_absent_total` increments and a +gave no table, so `dfe_loader_routing_field_absent_total` increments and a warning names the topic, the table and how many records it stands for, once a minute per topic. -A record whose table ClickHouse confirms does not exist lands in `default_table` instead, counted in `dfe_loader_unknown_table_fallback_total{table}`. The answer is held, then the table is resolved again, so a table created later starts receiving its own data without a pod restart: after 60 seconds, or for a `source_to_table` target the routing names, after 1 second doubling up to 30. The warning names the move into absence and not the state, so a dead source name is announced once rather than once a minute for the life of the deployment. - -The default table is never held absent, because nothing falls back from it. A record routed to it while it does not exist waits in the pending-schema buffer and is resolved again while it waits, until the table exists or the record reaches `schema.pending_max_age_secs` and is retired like any record whose schema never arrived. +A record routed to a table ClickHouse confirms does not exist lands in `default_table` instead, counted in `dfe_loader_unknown_table_fallback_total{table}`. The answer is held, then the table is resolved again, so a table created later starts receiving its data without a pod restart: after 60 seconds, or for a `source_to_table` target the routing names, after 1 second doubling up to 30. The warning names the move into absence and not the state, so a dead source name is announced once rather than once a minute for the life of the deployment. -## The setting drives the action +The default table is never held absent, because nothing falls back from it. A record routed to it before it exists waits in the pending-schema buffer and is resolved again as it waits, until the table exists or the record reaches `schema.pending_max_age_secs` and is retired like any record with a schema that never arrived. Routing is config-cascade behaviour, so it is tested by outcome, not by parse: the processor tests assert the row is actually buffered against the table the diff --git a/docs/transport/README.md b/docs/transport/README.md index 93adf55..a9faea9 100644 --- a/docs/transport/README.md +++ b/docs/transport/README.md @@ -12,20 +12,38 @@ The transports, Kafka and gRPC, are scalo's. The loader does not implement them: it takes what scalo's `Transport` receives and feeds it into its parse stage, then on through routing, extraction and the per-table insert path. Whichever transport delivers a message, the bytes converge on the same hot path: refuse anything not JSON, parse, route to `db.table`, promote schema columns, and buffer for insert. A Memory transport exists behind the `transport-memory` cargo feature for unit tests only. +Each transport has an adapter, and both adapters hand the pipeline the same `KafkaMessage`, so nothing after them knows which transport a record came on. + ```mermaid flowchart LR - subgraph scalo["Transport (scalo)"] - K["Kafka"] - G["gRPC"] + accTitle: L4 Detail -- Transport adapters + accDescr: scalo's KafkaTransport feeds the loader's TransportAdapter and scalo's GrpcTransport feeds the GrpcTransportAdapter. Both adapters turn each record into a KafkaMessage, which goes to the JSON parse step. + + subgraph scalo["scalo transports"] + kafka["KafkaTransport"] + grpc["GrpcTransport"] end - P["JSON parse
(loader hot path)"] - REST["route -> extract -> buffer -> insert"] - K --> P - G --> P - P --> REST + kafkaadapter["TransportAdapter"] + grpcadapter["GrpcTransportAdapter"] + message["KafkaMessage"] + parse["Parse JSON"] + + kafka --> kafkaadapter + grpc --> grpcadapter + kafkaadapter --> message + grpcadapter --> message + message --> parse + + classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A + classDef ext fill:#EEF0F2,stroke:#6B7280,stroke-width:1px,stroke-dasharray:4 3,color:#1F2937 + classDef group fill:none,stroke:#64748B,stroke-width:1px,stroke-dasharray:2 4 + class kafka,grpc ext + class scalo group ``` +*Figure 1. L4 Detail -- Transport adapters. Dashed boxes are scalo's.* Up: [L3 Components -- dfe-loader](../README.md). + ## In this section - [grpc.md](grpc.md) -- the gRPC listener: how it answers a Push, the proto, its config and the deployment contract's port. diff --git a/docs/transport/common-header.md b/docs/transport/common-header.md index c81351e..131ed08 100644 --- a/docs/transport/common-header.md +++ b/docs/transport/common-header.md @@ -1,7 +1,7 @@ |Push| L - R -->|Push| T - T -->|Push| L - L --> CH - L -.->|PushResponse is the answer| R + accTitle: L4 Detail -- gRPC path + accDescr: dfe-receiver pushes to the loader directly or through a transform, which is a gRPC server and client. The loader writes the rows to ClickHouse and answers each Push with a PushResponse. + + subgraph senders["gRPC senders"] + direction LR + receiver["dfe-receiver"] -->|Push| transform["Transform"] + end + + loader["dfe-loader"] + clickhouse[("ClickHouse")] + + senders -->|Push| loader + loader -->|rows| clickhouse + loader -.->|PushResponse| senders + + classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A + classDef core fill:#DCEBF7,stroke:#0072B2,stroke-width:2px,color:#0B2A3F + classDef store fill:#DDF3EC,stroke:#00785A,stroke-width:1px,color:#0B3327 + classDef ext fill:#EEF0F2,stroke:#6B7280,stroke-width:1px,stroke-dasharray:4 3,color:#1F2937 + classDef group fill:none,stroke:#64748B,stroke-width:1px,stroke-dasharray:2 4 + class receiver,transform ext + class loader core + class clickhouse store + class senders group ``` +*Figure 1. L4 Detail -- gRPC path. The dotted arrow is the answer.* Up: [Transport](README.md). + ## How the loader answers a Push The gRPC response is the acknowledgement. There is no protocol on top of it. -With `grpc.acknowledgements.enabled` (the default) the listener answers a Push only once every row built from it is in ClickHouse, in the DLQ with the DLQ's confirmation, or dropped. A failed insert answers `UNAVAILABLE` and the sender retries from its own copy. Rows whose sender waits on them flush at 200 ms (`EARLY_FLUSH_AGE` in `src/pipeline/orchestrator.rs`) rather than at `buffer.flush_age_secs`. A Push still unanswered after `grpc.max_hold_ms`, shortened by the sender's own deadline, is answered `UNAVAILABLE` as well. Its rows may still land, so the retry can write them twice. +With `grpc.acknowledgements.enabled` (the default) the listener answers a Push only once every row built from it is in ClickHouse, in the DLQ with the DLQ's confirmation, or dropped. A failed insert answers `UNAVAILABLE` and the sender retries from its copy. Rows a sender waits on flush at 200 ms (`EARLY_FLUSH_AGE` in `src/pipeline/orchestrator.rs`) rather than at `buffer.flush_age_secs`. A Push still unanswered after `grpc.max_hold_ms`, shortened by the sender's deadline, is answered `UNAVAILABLE` as well. Its rows may still land, so the retry can write them twice. -Rows ClickHouse rejects for good go to the DLQ, and their Push is answered once the DLQ confirms that write on its own (`Dlq::write_confirmed`). A Push whose rows have all landed is answered before that write. A refused write is retried with jittered backoff, 100 ms to 1 s, each retry counted in `pipeline_retries_total{stage="dlq"}`, while the Pushes it holds wait and intake stops. The retry ends 1 s (`HOLD_ANSWER_MARGIN`) before the earliest of their hold deadlines, and they are answered `UNAVAILABLE`, as they are at once when the DLQ or the loader shuts down. Records that fail processing (unparseable, unroutable, a pre-route reject) take the same path. +Rows ClickHouse rejects for good go to the DLQ, and their Push is answered once the DLQ confirms that write (`Dlq::write_confirmed`). A Push with every row landed is answered before that write. A refused write is retried with jittered backoff, 100 ms to 1 s, each retry counted in `pipeline_retries_total{stage="dlq"}`, and the Pushes it holds wait as intake stops. The retry ends 1 s (`HOLD_ANSWER_MARGIN`) before the earliest of their hold deadlines, and they are answered `UNAVAILABLE`, as they are at once when the DLQ or the loader shuts down. Records that fail processing (unparseable, unroutable, a pre-route reject) take the same path. A row no DLQ backend can ever hold is answered as a drop, since a retry would fail the same way at every hop: no working DLQ at all (`reason="dead_letter"`), or larger than a Kafka-only DLQ's ceiling (`reason="too_large"`). It is logged at `error` and counted in `pipeline_dead_letters_dropped_total{reason}` and `dfe_loader_rows_lost_total`. -A message whose table schema is not cached waits in the pending-schema buffer (`src/pipeline/pending_schema.rs`). On gRPC a full buffer (`schema.pending_max_per_table`, `schema.pending_max_total`) stops intake rather than shedding to the DLQ, and a message past `schema.pending_max_age_secs` is answered `UNAVAILABLE`. +A message with a table schema not yet cached waits in the pending-schema buffer (`src/pipeline/pending_schema.rs`). On gRPC a full buffer (`schema.pending_max_per_table`, `schema.pending_max_total`) stops intake rather than shedding to the DLQ, and a message past `schema.pending_max_age_secs` is answered `UNAVAILABLE`. -With `grpc.acknowledgements.enabled: false` a Push is answered once queued and the loader holds the only copy. A failed insert or refused DLQ write is retried while intake stops (`src/pipeline/unsettled.rs`), and a loader stopped while it holds a batch loses it. +With `grpc.acknowledgements.enabled: false` a Push is answered once queued and the loader holds the only copy. A failed insert or refused DLQ write is retried as intake stops (`src/pipeline/unsettled.rs`), and a loader stopped as it holds a batch loses it. -The listener also refuses a Push before it reaches the pipeline (`src/transport/grpc/mod.rs` in scalo-rs): +The listener rejects a Push before it reaches the pipeline in three cases (`src/transport/grpc/mod.rs` in scalo-rs): | Status | When | |---|---| @@ -54,9 +72,9 @@ The listener also refuses a Push before it reaches the pipeline (`src/transport/ ## Proto -The service is scalo's own, defined in scalo-rs `proto/scalo/transport/v1/transport.proto`: +The service is scalo's, defined in scalo-rs `proto/scalo/transport/v1/transport.proto`: -| RPC | What it carries | +| RPC | What it holds | |---|---| | `Push(PushRequest) returns (PushResponse)` | One payload (`bytes`), a `Format` hint and a `metadata` string map | | `RouteBatch(Batch) returns (BatchAck)` | A whole batch of records, payloads as opaque bytes, each with a key, headers, a timestamp and a format hint | @@ -66,7 +84,7 @@ A sender names the topic in the Push metadata key `topic`. The adapter uses it a ## Configuration -The loader's `grpc` block is `GrpcConfig` in `src/config/kafka.rs`, which `GrpcTransportAdapter::new` maps onto scalo's own `GrpcConfig` (scalo-rs `src/transport/grpc/config.rs`). The defaults are in its `Default` impl. +The loader's `grpc` block is `GrpcConfig` in `src/config/kafka.rs`, which `GrpcTransportAdapter::new` maps onto scalo's `GrpcConfig` (scalo-rs `src/transport/grpc/config.rs`). The defaults are in its `Default` impl. | Key | Meaning | |---|---| @@ -79,7 +97,7 @@ The loader's `grpc` block is `GrpcConfig` in `src/config/kafka.rs`, which `GrpcT | `acknowledgements.enabled` | Answer a Push when its rows land (`true`) or when they are queued (`false`) | | `max_hold_ms` | Longest a Push waits for its answer before it is told to retry. Keep it below the sender's send timeout and above the flush delay | -The block carries no TLS settings. +The block has no TLS settings. The deployment contract (`src/config/loader.rs`) declares the `push` port 6000 only when `config.transport` is `grpc`, bound from `grpc.listen`, with `appProtocol: kubernetes.io/h2c`. Liveness and readiness are scalo's `/livez` and `/readyz` on the metrics port, not the gRPC health protocol. The contract's KEDA block comes from `KedaConfig` (`src/config/pipeline.rs`) with the Kafka lag trigger off, because raw consumer-group lag rises when a downstream stage breaks.