Skip to content

fix(sink): dead-letter events over the per-message budget - #187

Merged
catinspace-au merged 1 commit into
mainfrom
fix/oversize-to-dlq
Nov 16, 2026
Merged

catinspace-au merged 1 commit into
mainfrom
fix/oversize-to-dlq

Conversation

@catinspace-au

Copy link
Copy Markdown
Contributor

Dropping events over sink.max_message_bytes after serialisation broke at-least-once delivery. Enrichment could push a near-cap input over the cap.

Oversize events go to the DLQ:

  • the whole serialised body, keyed to the sink topic, with a reason naming its size and the budget
  • scalo cuts the body to dlq.max_payload_bytes so it fits a Kafka DLQ. It records original_bytes.
  • events_oversize_total still counts each event. The source is released once the DLQ holds it.
  • with dlq.enabled false, oversize events are dropped and counted in pipeline_dead_letters_dropped_total, like non-JSON events

Tests: unit tests in pipeline.rs cover the returned body and the reason. tests/direct_transport.rs pushes an oversize event beside one that fits and reads the file DLQ.

An event over sink.max_message_bytes after serialisation was dropped. It was logged and counted on events_oversize_total. Enrichment grows an event about 1.5 times. A near-cap input could go over the cap and be lost, breaking at-least-once delivery.

Each oversize event goes to the DLQ as its whole serialised body. The reason names its size and the budget. The entry is keyed to the sink topic like the not-JSON dead letters. The DLQ cuts the body to dlq.max_payload_bytes and records original_bytes.

The source is released once the DLQ holds the entry. events_oversize_total still counts each oversize event. With dlq.enabled false, oversize events are dropped and counted in pipeline_dead_letters_dropped_total, like the not-JSON ones.

serialise_events keeps its signature for tests and offline tools. The service calls serialise_events_with_oversize, which hands the oversize bodies back.
@catinspace-au
catinspace-au merged commit 0f8617b into main Nov 16, 2026
26 checks passed
@catinspace-au
catinspace-au deleted the fix/oversize-to-dlq branch November 16, 2026 04:02
catinspace-au added a commit that referenced this pull request Nov 16, 2026
Main's #187 sends an event over sink.max_message_bytes to the DLQ. The architecture page keeps the branch's layout. Its one-record paragraph names serialise_events_with_oversize and the DLQ. The configuration page uses main's DLQ wording and the branch's word choices.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant