Skip to content

fix(listeners): dead-letter syslog and GELF frames over the size limit - #277

Closed
catinspace-au wants to merge 1 commit into
mainfrom
fix/oversize-to-dlq
Closed

catinspace-au wants to merge 1 commit into
mainfrom
fix/oversize-to-dlq

Conversation

@catinspace-au

Copy link
Copy Markdown
Contributor

Syslog TCP or TLS messages over syslog.max_message_size and GELF messages over gelf.max_message_size were dropped without a way to read them back.

The framing codecs return each skipped message as a separate frame with its head (first bytes) and length. The listener dead-letters it through the same DLQ writer used for refused records.

The reason is "frame over bytes on ". original_bytes is set. receiver_records_dropped_total{reason="oversize"} still counts it.

  • The head is capped at 512 KiB, the DLQ's payload cap. Longer frames are marked truncated.
  • The connection waits for the DLQ write. This paces oversize floods at the DLQ's write rate. Each codec holds at most one frame's head.
  • Unconfirmed DLQ writes are retried until confirmation or shutdown, as with held records.
  • A stream that ends or fails inside an oversize frame hands over what arrived.
  • With no DLQ configured, the frame is dropped and counted as before.

Tests cover codec heads and lengths across split reads. They check DLQ entries for truncated and whole heads. They cover syslog and GELF listeners writing entries to a file DLQ.

Syslog TCP or TLS messages over syslog.max_message_size and GELF messages over gelf.max_message_size were dropped without a way to read them back.

The framing codecs return each skipped message as a separate frame with its head (first bytes) and length. The listener dead-letters it through the same DLQ writer used for refused records.

The reason is "frame over <limit> bytes on <listener>". original_bytes is set. receiver_records_dropped_total{reason="oversize"} still counts it.

- The head is capped at 512 KiB, the DLQ's payload cap. Longer frames are marked truncated.
- The connection waits for the DLQ write. This paces oversize floods at the DLQ's write rate. Each codec holds at most one frame's head.
- Unconfirmed DLQ writes are retried until confirmation or shutdown, as with held records.
- A stream that ends or fails inside an oversize frame hands over what arrived.
- With no DLQ configured, the frame is dropped and counted as before.

Tests cover codec heads and lengths across split reads. They check DLQ entries for truncated and whole heads. They cover syslog and GELF listeners writing entries to a file DLQ.
@catinspace-au

Copy link
Copy Markdown
Contributor Author

Syslog and GELF peers are unauthenticated. The receiver keeps unauthenticated oversize bytes out of Kafka (docs/webhook-setup.md:46). Drop and count an oversize frame because it was never accepted. Dead-lettering it would let any peer that reaches the port write a 512 KiB DLQ record per frame.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant