Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion benches/syslog.rs
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ use std::hint::black_box;
use bytes::BytesMut;
use criterion::{BenchmarkId, Criterion, Throughput, criterion_group, criterion_main};
use dfe_receiver::config::RawCapture;
use dfe_receiver::server::oversize::Frame;
use dfe_receiver::server::syslog::convert::syslog_to_json;
use dfe_receiver::server::syslog::framing::SyslogFrameDecoder;
use tokio_util::codec::Decoder;
Expand Down Expand Up @@ -55,7 +56,9 @@ fn framing(c: &mut Criterion) {
let mut buf = BytesMut::from(&stream[..]);
let mut n = 0usize;
while let Ok(Some(frame)) = decoder.decode(&mut buf) {
n += frame.len();
if let Frame::Message(message) = frame {
n += message.len();
}
}
black_box(n)
});
Expand Down
3 changes: 2 additions & 1 deletion benches/syslog_allocs.rs
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ mod common;

use bytes::BytesMut;
use dfe_receiver::config::RawCapture;
use dfe_receiver::server::oversize::Frame;
use dfe_receiver::server::syslog::convert::syslog_to_json;
use dfe_receiver::server::syslog::framing::SyslogFrameDecoder;
use tokio_util::codec::Decoder;
Expand All @@ -42,7 +43,7 @@ fn frames_per_stream(stream: &[u8]) -> (f64, f64) {
let info = allocation_counter::measure(|| {
let mut decoder = SyslogFrameDecoder::default();
while let Ok(Some(frame)) = decoder.decode(&mut buf) {
frames += u64::from(!frame.is_empty());
frames += u64::from(matches!(&frame, Frame::Message(m) if !m.is_empty()));
}
});
let frames = frames.max(1) as f64;
Expand Down
2 changes: 1 addition & 1 deletion docs/hardening.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ Application-level hardening, and cost-conscious fronting on Kubernetes and AWS.
| Dead-letter queue | Global | `routing.dlq` |
| Memory pressure backpressure | 503 / `UNAVAILABLE` on HTTP and gRPC ingest | `DFE_RECEIVER_MEMORY_PRESSURE_THRESHOLD`, `self_regulation.*` (`buffer.*` bounds the destination queues and nothing else) |
| Middleware ordering | RateLimit -> Concurrency -> Timeout -> BodyLimit -> Auth -> Handler, behind the accept loop's IP filter | Cheapest refusal first |
| Frame size validation | GELF 1MB, Syslog 64KB, Fluent 32MB. An oversize syslog or GELF message is skipped to its delimiter and counted (`receiver_records_dropped_total`, reason `oversize`), and its connection goes on. Fluent Forward has no delimiter to resume at, so an oversize frame closes its connection | Per-protocol limits, `syslog.max_message_size`, `gelf.max_message_size` |
| Frame size validation | GELF 1MB, Syslog 64KB, Fluent 32MB. An oversize syslog or GELF message is skipped to its delimiter and counted (`receiver_records_dropped_total`, reason `oversize`), its first bytes go to the DLQ with its length, and its connection goes on. Fluent Forward has no delimiter to resume at, so an oversize frame closes its connection | Per-protocol limits, `syslog.max_message_size`, `gelf.max_message_size` |
| Decoded-size budget | An OTLP export, remote-write request or Vector push past 100,000 records, 1,000,000 messages or 64 MiB of resource, scope, metric or series-label bytes copied into its records, or holding a protobuf group, is refused before it decodes: 400 on HTTP, `INVALID_ARGUMENT` on gRPC. A Fluent Forward message past 1,000,000 values, 100,000 entries or 64 MiB of tag copies closes its connection, found by a scan that reads each byte once. A Splunk HEC request past 100,000 events or raw lines, or 64 MiB of `host`, `source`, `sourcetype` and `index` copied into its raw lines, is refused with a 400 | Hard-coded |
| Zip-bomb rejection | Lumberjack nested compression rejected. A compressed frame is decompressed one inner frame at a time and refused past 64 MiB | Hard-coded |
| Unsettled bytes per connection | Lumberjack settles and ACKs a window in parts past 16 MiB of events | Hard-coded |
Expand Down
2 changes: 1 addition & 1 deletion docs/protocols.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ An RFC 3164 token is the app name when it reads `name:` or `name[pid]:`, with th

RFC 5424 structured data that does not parse, like `[x@1 k=v]`, stays in `message` as sent, and the elements that did parse still land in `structured_data`. A byte order mark at the start of the message and NUL bytes at its end are dropped.

On TCP and TLS a frame is octet-counted (RFC 6587) when it opens with a count, one space and the `<` of a PRI. Anything else is an LF-terminated line, so a line opening with digits, like `2026 Oct 11 ...`, is never read as a count. A message over `syslog.max_message_size` is skipped and counted as dropped (reason `oversize`), and the connection goes on: an octet-counted one by its count, a line up to its LF. A count over `syslog.max_frame_size` (default 256 KiB) is not trusted as a count, and that frame is read as a line.
On TCP and TLS a frame is octet-counted (RFC 6587) when it opens with a count, one space and the `<` of a PRI. Anything else is an LF-terminated line, so a line opening with digits, like `2026 Oct 11 ...`, is never read as a count. A message over `syslog.max_message_size` is skipped and counted as dropped (reason `oversize`), and the connection goes on: an octet-counted one by its count, a line up to its LF. Its first 512 KiB, the DLQ's payload cap, go to the DLQ with the reason `frame over <limit> bytes on syslog TCP` (or `TLS`) and its length in `original_bytes`. The connection waits on the DLQ write, and a write the DLQ does not confirm is offered again. A count over `syslog.max_frame_size` (default 256 KiB) is not trusted as a count, and that frame is read as a line.

Syslog UDP reads every datagram ready on a wake, up to `syslog.udp_batch_size` (default 64), in batched receive calls, and hands them to the pipeline as one batch. Its socket asks for a `syslog.udp_recv_buffer_size` receive buffer (default 8 MiB), and logs a warning when `net.core.rmem_max` grants less. Datagrams the kernel drops for a full buffer are read from `/proc/net/udp` every 10 seconds and counted as dropped, reason `kernel`. Syslog TCP settles every frame one read brought in with a single hold on the pipeline.

Expand Down
72 changes: 71 additions & 1 deletion src/buffer/rejects.rs
Original file line number Diff line number Diff line change
Expand Up @@ -159,8 +159,32 @@ impl Rejects {
/// [`Disposal::Refused`] leaves the record with the caller, to keep and
/// offer again.
pub async fn dispose(&self, topic: &str, payload: &Bytes, reason: &str) -> Disposal {
self.dispose_entry(topic, payload, reason, || entry(topic, payload, reason))
.await
}

/// [`dispose`](Self::dispose) for the first bytes of a frame a listener
/// skipped for its size, `original_bytes` long in all, which has no
/// destination yet.
pub async fn dispose_head(
&self,
head: &Bytes,
original_bytes: usize,
reason: &str,
) -> Disposal {
self.dispose_entry("", head, reason, || cut_entry(head, original_bytes, reason))
.await
}

async fn dispose_entry(
&self,
topic: &str,
payload: &Bytes,
reason: &str,
make_entry: impl FnOnce() -> DlqEntry,
) -> Disposal {
let disposal = match &self.dlq {
Some(dlq) => match dead_letter_confirmed(dlq, entry(topic, payload, reason)).await {
Some(dlq) => match dead_letter_confirmed(dlq, make_entry()).await {
Ok(DeadLetter::Written) => Disposal::DeadLettered,
Ok(DeadLetter::Unwritable) => Disposal::Dropped,
Err(e) => {
Expand Down Expand Up @@ -265,6 +289,14 @@ fn entry(topic: &str, payload: &Bytes, reason: &str) -> DlqEntry {
}
}

/// The DLQ entry for the first bytes of a frame that was `original_bytes` long.
fn cut_entry(head: &Bytes, original_bytes: usize, reason: &str) -> DlqEntry {
let mut entry = DlqEntry::new("receiver", reason, head.to_vec());
entry.truncated = head.len() < original_bytes;
entry.original_bytes = Some(original_bytes);
entry
}

/// DLQ fixtures shared by the buffer tests.
#[cfg(test)]
pub(crate) mod test_dlq {
Expand Down Expand Up @@ -417,6 +449,44 @@ mod tests {
.unwrap()
}

/// The head of a frame a listener skipped reaches the DLQ with the frame's
/// full length, marked cut when the head is shorter, and with no
/// destination to route it by.
#[tokio::test]
async fn the_head_of_a_skipped_frame_lands_in_the_dlq_with_its_length() {
let dir = tempfile::tempdir().unwrap();
let dlq = file_dlq(dir.path());
let metrics = Arc::new(Metrics::default());
let rejects = Rejects::new(Some(Arc::clone(&dlq)), Some(Arc::clone(&metrics)));

let cut = Bytes::from_static(b"<14>first bytes");
for (head, original) in [(&cut, 70_000), (&cut, cut.len())] {
let disposal = rejects
.dispose_head(head, original, "frame over 65536 bytes on syslog tcp")
.await;
assert_eq!(disposal, Disposal::DeadLettered);
}
dlq.flush().await.unwrap();
assert_eq!(metrics.get_messages_dlq(), 2);

let written = std::fs::read_to_string(dir.path().join("receiver/dlq.ndjson")).unwrap();
let entries: Vec<serde_json::Value> = written
.lines()
.map(|line| serde_json::from_str(line).unwrap())
.collect();
assert_eq!(entries.len(), 2, "{written}");
for entry in &entries {
assert_eq!(entry["reason"], "frame over 65536 bytes on syslog tcp");
assert!(entry.get("destination").is_none(), "{entry}");
}
assert_eq!(entries[0]["truncated"], true);
assert_eq!(entries[0]["original_bytes"], 70_000);
assert_eq!(entries[0]["_raw_encoding"], "base64");
assert!(entries[1].get("truncated").is_none(), "{}", entries[1]);
assert_eq!(entries[1]["original_bytes"], cut.len());
assert_eq!(entries[1]["_raw"], "<14>first bytes");
}

/// A refused record reaches the configured DLQ with its reason, its
/// destination and the original bytes, and counts as dead-lettered.
#[tokio::test]
Expand Down
6 changes: 6 additions & 0 deletions src/pipeline/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1497,6 +1497,12 @@ impl PipelineState {
Ok(())
}

/// Where a record refused for good is dead-lettered, for a listener that
/// skips a frame before it is a record.
pub(crate) fn rejects(&self) -> &Rejects {
&self.rejects
}

/// Settle a record a destination refuses for good: dead-letter it, or
/// drop it where no DLQ is configured.
async fn settle_refused(
Expand Down
10 changes: 7 additions & 3 deletions src/server/frame_clock.rs
Original file line number Diff line number Diff line change
Expand Up @@ -247,10 +247,9 @@ mod tests {
use tokio_stream::StreamExt;
use tokio_util::codec::FramedRead;

use bytes::Bytes;

use super::*;
use crate::server::gelf::GelfFrameDecoder;
use crate::server::oversize::Frame;
use crate::server::syslog::framing::SyslogFrameDecoder;

const LIMIT: Duration = Duration::from_millis(500);
Expand All @@ -272,7 +271,7 @@ mod tests {
/// for `hold` before it reads on, until a frame times out or the client closes.
async fn run<D>(codec: D, steps: Vec<Step>, hold: Duration) -> Read
where
D: Framing<Item = Bytes, Error = io::Error>,
D: Framing<Item = Frame, Error = io::Error>,
{
let (mut client, server) = tokio::io::duplex(1024);
let socket = Clocked::new(server, LIMIT, Arc::new(Metrics::default()), "test");
Expand All @@ -296,6 +295,11 @@ mod tests {
read.timed_out = true;
return;
};
// A skipped message is no frame to hold, as before the codec
// handed it over.
let Frame::Message(frame) = frame else {
continue;
};
read.frames
.push(String::from_utf8_lossy(&frame).into_owned());
tokio::time::sleep(hold).await;
Expand Down
Loading
Loading