Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 7 additions & 1 deletion docs/config-schema.json
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,7 @@
"localhost:9092"
],
"topic": "",
"dlq_topic": "dfe_transform_dlq",
"key_field": "",
"encoding": "json",
"compression": "zstd",
Expand Down Expand Up @@ -468,6 +469,11 @@
"type": "string",
"default": ""
},
"dlq_topic": {
"description": "Topic a record over the producer's size ceiling is dead-lettered to on\nthe bus transport, as a DLQ envelope the loader reads. Unused on direct,\nwhere the bridge's `dlq` holds it, and while `dlq.enabled` is false.",
"type": "string",
"default": "dfe_transform_dlq"
},
"key_field": {
"description": "Event field path for Kafka partition key (e.g., \".org_id\").",
"type": "string",
Expand Down Expand Up @@ -795,7 +801,7 @@
}
},
"DeadLetterConfig": {
"description": "The bridge's dead-letter queue: scalo's `DlqConfig` as written.\n\nThe default writes NDJSON files under `/var/spool/scalo/dlq`, which the\ndeployment contract declares writable. With `enabled: false` such a record\nis dropped and counted on `pipeline_dead_letters_dropped_total`. Only the\ndirect transport's bridge has one: on the bus Vector's own Kafka sink\ndelivers.",
"description": "The bridge's dead-letter queue: scalo's `DlqConfig` as written.\n\nThe default writes NDJSON files under `/var/spool/scalo/dlq`, which the\ndeployment contract declares writable. With `enabled: false` such a record\nis dropped and counted on `pipeline_dead_letters_dropped_total`. Only the\ndirect transport's bridge writes through it. On the bus a record over the\nsink's size ceiling goes to `sink.dlq_topic`, and only `enabled` and\n`max_payload_bytes` apply.",
"$ref": "#/$defs/DlqConfig"
},
"DlqConfig": {
Expand Down
13 changes: 11 additions & 2 deletions docs/config-schema.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -92,6 +92,7 @@ properties:
brokers:
- localhost:9092
topic: ""
dlq_topic: dfe_transform_dlq
key_field: ""
encoding: json
compression: zstd
Expand Down Expand Up @@ -458,6 +459,13 @@ $defs:
the downstream stage picks its table by, so a source keeps its name.
type: string
default: ""
dlq_topic:
description: |-
Topic a record over the producer's size ceiling is dead-lettered to on
the bus transport, as a DLQ envelope the loader reads. Unused on direct,
where the bridge's `dlq` holds it, and while `dlq.enabled` is false.
type: string
default: dfe_transform_dlq
key_field:
description: Event field path for Kafka partition key (e.g., ".org_id").
type: string
Expand Down Expand Up @@ -810,8 +818,9 @@ $defs:
The default writes NDJSON files under `/var/spool/scalo/dlq`, which the
deployment contract declares writable. With `enabled: false` such a record
is dropped and counted on `pipeline_dead_letters_dropped_total`. Only the
direct transport's bridge has one: on the bus Vector's own Kafka sink
delivers.
direct transport's bridge writes through it. On the bus a record over the
sink's size ceiling goes to `sink.dlq_topic`, and only `enabled` and
`max_payload_bytes` apply.
$ref: "#/$defs/DlqConfig"
DlqConfig:
description: Top-level DLQ configuration.
Expand Down
13 changes: 8 additions & 5 deletions docs/internals.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,13 +55,15 @@ transforms:
| `00_global.yaml` | `timezone: UTC`, `data_dir`, the API switch, one `directory` secret backend per credential source |
| `00_source.yaml` | `dfe_source`: a `kafka` source on bus, carrying the librdkafka layers in [librdkafka.md](librdkafka.md), or a `vector` source on `bridge.to_vector` on direct |
| `50_NNN_<file>` | Each transform file, in filename order, re-serialised with its wired `inputs` |
| `90_sink.yaml` | `dfe_sink`: on bus a `kafka` sink behind the `dfe_size_cap` filter, on direct a `vector` sink dialling `bridge.from_vector` |
| `90_sink.yaml` | `dfe_sink`: on bus a `kafka` sink behind the `dfe_size_cap` route, plus `dfe_size_cap_dlq` and `dfe_dlq_sink` for the records it holds back, on direct a `vector` sink dialling `bridge.from_vector` |
| `99_observability.yaml` | `internal_metrics` into a `prometheus_exporter` on `metrics.vector_metrics_address` |
| `.secrets/` | Owner-only files for a SASL credential given as text |

Credentials appear only as references: `SECRET[dfe_source_sasl.username]` for a `secret_dir`, `SECRET[dfe_credentials.source_sasl_username]` for a credential given as text. The assembler rejects a `secret_dir` that lacks its `username` or `password` file.

`dfe_size_cap` drops a record with an encoded length over the producer's `message.max.bytes` less 128 bytes for the key, headers and framing. `message.max.bytes` is scalo's pipeline-wide record ceiling of 16 MiB unless the central profile or `sink.librdkafka_options` sets it. librdkafka's default of 1,000,000 bytes would refuse records the rest of the chain accepts.
`dfe_size_cap` is a `route` that sends a record whose encoded length is over the producer's `message.max.bytes` less 128 bytes for the key, headers and framing out of its `_unmatched` output, and the rest out of `within_cap` to the sink. `message.max.bytes` is scalo's pipeline-wide record ceiling of 16 MiB unless the central profile or `sink.librdkafka_options` sets it. librdkafka's default of 1,000,000 bytes would refuse records the rest of the chain accepts.

While `dlq.enabled` is true, `dfe_size_cap_dlq` remaps each record from `_unmatched` into scalo's DLQ entry and `dfe_dlq_sink` writes it to `sink.dlq_topic` (`dfe_transform_dlq`, the shared topic dfe-loader loads). The entry is `truncated`: `original_bytes` is the record's length, `reason` names it and the ceiling, `destination` is `sink.topic`, and `_raw` is the first bytes as base64 with `_raw_encoding: base64`. The `_org_id`, `_source`, `_timestamp_received` and `_tags` values that fit are copied beside it, and any that do not are named in `header_dropped`. The head and the copied values share `dlq.max_payload_bytes`, 512 KiB by default, held to half of what `message.max.bytes` leaves. `dfe_dlq_sink` uses the main sink's brokers, credentials and librdkafka options, writes JSON with no key, and holds in a 500-event memory buffer that blocks when full, so an unreachable broker back-pressures the source rather than dropping the entry. With `dlq.enabled: false` there is no remap or second sink, and `dfe_size_cap` is a `filter` that drops the record.

The wiring follows these rules:

Expand Down Expand Up @@ -153,9 +155,10 @@ Metric names are bare unless `metrics.namespace` is set, and `dfe-transform-vect
|---|---|
| `records_received_total` | `vector_component_received_events_total{component_id="dfe_source"}` |
| `records_processed_total`, `records_delivered_total` | `vector_component_sent_events_total{component_id="dfe_sink"}` |
| `records_error_total` | `vector_component_discarded_events_total` on `dfe_sink` and on `dfe_size_cap` |
| `records_error_total` | `vector_component_discarded_events_total` on `dfe_sink`, `dfe_size_cap` and `dfe_dlq_sink` |
| `transform_vector_sink_errors_total` | `vector_component_errors_total{component_id="dfe_sink"}` |
| `pipeline_dead_letters_dropped_total{reason="too_large"}` | the growth of `vector_component_discarded_events_total` on `dfe_size_cap` |
| `pipeline_dead_letters_dropped_total{reason="too_large"}` | the growth of `vector_component_discarded_events_total` on `dfe_size_cap` and on `dfe_dlq_sink` |
| Dead letters held by the DLQ topic, with no supervisor counter of their own | `vector_component_sent_events_total{component_id="dfe_dlq_sink"}`, merged with the other `vector_*` series |
| `pipeline_dead_letters_dropped_total{reason="rejected"}` | the growth of `vector_component_discarded_events_total` on `dfe_sink` |

The scrape reads Vector's exporter on scalo's metrics interval and re-registers every sample. A failed scrape is not fatal: it counts `transform_vector_scrape_failures_total`, warns at most once every five minutes, and leaves readiness as it was. A response over 4 MiB is a failed scrape, not a partial merge. A counter is set with `absolute` and carried across Vector restarts: a total that falls is a restarted Vector, and the old count becomes a base. A histogram arrives pre-aggregated, so each `_bucket` registers as a counter with its `le` label. A gauge Vector stops reporting is zeroed after `metrics.vector_metrics_expiry_ticks` scrapes, a counter is forgotten after 240, and at most 50,000 series merge.
Expand All @@ -166,7 +169,7 @@ dfe-infra's values set a CPU target and, on its single, scale and mesh profiles,

With `source.acknowledgements.enabled`, the default, delivery is at-least-once on both transports. On bus the sink's acknowledgements hold Vector's source offsets until delivery, so a crash redelivers from the last committed offset. On direct a push is answered once the next hop has it. With acknowledgements off, both answer at receipt.

On direct a record over the next stage's message-size ceiling goes to scalo's file DLQ, NDJSON under `/var/spool/scalo/dlq` on the contract's `dlq` writable path. With `dlq.enabled: false` it is dropped and counted in `pipeline_dead_letters_dropped_total{reason="too_large"}`. A record Vector rejects for good is dropped and counted under `reason="rejected"`, whatever `dlq` says. On bus no DLQ runs: the size cap's drops count under `too_large` and the Kafka sink's under `rejected`. When no enabled DLQ backend can start, such as a file DLQ with no writable directory off the image, a direct run logs `DLQ enabled but no backend started` and leaves dead letters unacknowledged at their source. Point `dlq.file.path` at a directory you can write.
On direct a record over the next stage's message-size ceiling goes to scalo's file DLQ, NDJSON under `/var/spool/scalo/dlq` on the contract's `dlq` writable path. With `dlq.enabled: false` it is dropped and counted in `pipeline_dead_letters_dropped_total{reason="too_large"}`. A record Vector rejects for good is dropped and counted under `reason="rejected"`, whatever `dlq` says. On bus a record over the producer's ceiling goes to `sink.dlq_topic` (see [The assembled directory](#the-assembled-directory)), counted in `vector_component_sent_events_total{component_id="dfe_dlq_sink"}` and in no drop counter. It counts under `too_large` only when the DLQ sink rejects it for good, or with `dlq.enabled: false`, and the Kafka sink's rejections count under `rejected`. When no enabled DLQ backend can start, such as a file DLQ with no writable directory off the image, a direct run logs `DLQ enabled but no backend started` and leaves dead letters unacknowledged at their source. Point `dlq.file.path` at a directory you can write.

On bus Vector holds the records, bounded by its `buffer` settings and the container's memory limit.

Expand Down
1 change: 1 addition & 0 deletions src/config/assembler.rs
Original file line number Diff line number Diff line change
Expand Up @@ -111,6 +111,7 @@ pub fn assemble_for(config: &Config, output_dir: &Path, published: &Path) -> Res
&config.bridge,
&wiring.sink_inputs,
config.source.acknowledgements.enabled,
&config.dlq.0,
);
write_yaml(output_dir, "90_sink.yaml", &sink_yaml)?;

Expand Down
Loading
Loading