Repository navigation
fix(listeners): dead-letter syslog and GELF frames over the size limit - #277
Closed
catinspace-au wants to merge 1 commit into
Closed
catinspace-au wants to merge 1 commit into
catinspace-au wants to merge 1 commit into
Conversation
Syslog TCP or TLS messages over syslog.max_message_size and GELF messages over gelf.max_message_size were dropped without a way to read them back.
The framing codecs return each skipped message as a separate frame with its head (first bytes) and length. The listener dead-letters it through the same DLQ writer used for refused records.
The reason is "frame over <limit> bytes on <listener>". original_bytes is set. receiver_records_dropped_total{reason="oversize"} still counts it.
- The head is capped at 512 KiB, the DLQ's payload cap. Longer frames are marked truncated.
- The connection waits for the DLQ write. This paces oversize floods at the DLQ's write rate. Each codec holds at most one frame's head.
- Unconfirmed DLQ writes are retried until confirmation or shutdown, as with held records.
- A stream that ends or fails inside an oversize frame hands over what arrived.
- With no DLQ configured, the frame is dropped and counted as before.
Tests cover codec heads and lengths across split reads. They check DLQ entries for truncated and whole heads. They cover syslog and GELF listeners writing entries to a file DLQ.
Contributor
Author
|
Syslog and GELF peers are unauthenticated. The receiver keeps unauthenticated oversize bytes out of Kafka (docs/webhook-setup.md:46). Drop and count an oversize frame because it was never accepted. Dead-lettering it would let any peer that reaches the port write a 512 KiB DLQ record per frame. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Syslog TCP or TLS messages over syslog.max_message_size and GELF messages over gelf.max_message_size were dropped without a way to read them back.
The framing codecs return each skipped message as a separate frame with its head (first bytes) and length. The listener dead-letters it through the same DLQ writer used for refused records.
The reason is "frame over bytes on ". original_bytes is set. receiver_records_dropped_total{reason="oversize"} still counts it.
Tests cover codec heads and lengths across split reads. They check DLQ entries for truncated and whole heads. They cover syslog and GELF listeners writing entries to a file DLQ.