Skip to content

fix(sink): dead-letter records over the producer ceiling on bus - #173

Merged
catinspace-au merged 2 commits into
mainfrom
fix/oversize-to-dlq
Nov 16, 2026
Merged

catinspace-au merged 2 commits into
mainfrom
fix/oversize-to-dlq

Conversation

@catinspace-au

Copy link
Copy Markdown
Contributor

On the bus transport, a filter dropped records over the sink's message.max.bytes and counted them in component_discarded_events_total. That broke at-least-once delivery.

Records within message.max.bytes go to the sink. Larger records go through a remap into scalo's DLQ entry, then to a second Kafka sink on sink.dlq_topic. The default topic is dfe_transform_dlq, which dfe-loader loads.

  • the entry has service, reason, attempts, original_bytes, the first bytes as base64 and the _org_id, _source, _timestamp_received and _tags values that fit
  • the first bytes and copied values share dlq.max_payload_bytes, held to half of what message.max.bytes leaves
  • the DLQ sink takes the main sink's brokers, credentials and librdkafka options
  • it writes JSON with no key
  • it blocks when its buffer is full instead of dropping
  • dlq.enabled false keeps the filter and its drop count
  • vector_component_sent_events_total{component_id="dfe_dlq_sink"} counts the dead letters
  • pipeline_dead_letters_dropped_total{reason="too_large"} counts those the DLQ sink rejected

Tests cover:

  • the generator over json and raw_bytes
  • the remap through vector vrl
  • two e2e cases that read the entry back from the DLQ topic

On the bus transport, a filter dropped records over the sink's message.max.bytes and counted them in component_discarded_events_total. That broke at-least-once delivery.

Records within message.max.bytes go to the sink. Larger records go through a remap into scalo's DLQ entry, then to a second Kafka sink on sink.dlq_topic. The default topic is dfe_transform_dlq, which dfe-loader loads.

- the entry has service, reason, attempts, original_bytes, the first bytes as base64 and the _org_id, _source, _timestamp_received and _tags values that fit
- the first bytes and copied values share dlq.max_payload_bytes, held to half of what message.max.bytes leaves
- the DLQ sink takes the main sink's brokers, credentials and librdkafka options
- it writes JSON with no key
- it blocks when its buffer is full instead of dropping
- dlq.enabled false keeps the filter and its drop count
- vector_component_sent_events_total{component_id="dfe_dlq_sink"} counts the dead letters
- pipeline_dead_letters_dropped_total{reason="too_large"} counts those the DLQ sink rejected

Tests cover:
- the generator over json and raw_bytes
- the remap through vector vrl
- two e2e cases that read the entry back from the DLQ topic
@catinspace-au
catinspace-au merged commit e32c648 into main Nov 16, 2026
23 checks passed
@catinspace-au
catinspace-au deleted the fix/oversize-to-dlq branch November 16, 2026 04:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant