Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -195,7 +195,7 @@ A record the sink transport would refuse -- over its `message.max.bytes`, or mat

JSON is the only payload format. MessagePack, supported in DFE/XDR 2.0 and 2.1, is deprecated in DFE 2.2 and no longer accepted: the JSON path (SIMD parsing with sonic-rs, zstd on the wire) is fast enough that MessagePack gave no CPU saving.

**One record per event.** `pipeline.rs::serialise_events` gives each event its own payload and `service.rs::send_blocks` sends them via `TransportSender::send_batch`, because dfe-loader parses exactly one JSON document per message ([#67](https://github.com/hyperi-io/dfe-transform-elastic/issues/67)). `sink.max_message_bytes` (default 8,000,000 bytes, the DFE record cap) bounds each payload under the 16 MiB ceiling the producer and dfe-loader's consumer share, and the `send_batch` block under gRPC's 16 MiB ceiling. An event over the budget is dropped and counted on `events_oversize_total`: no broker would take it, and retrying blocks the partition.
**One record per event.** `pipeline.rs::serialise_events_with_oversize` gives each event its own payload and `service.rs::send_blocks` sends them via `TransportSender::send_batch`, because dfe-loader parses exactly one JSON document per message ([#67](https://github.com/hyperi-io/dfe-transform-elastic/issues/67)). `sink.max_message_bytes` (default 8,000,000 bytes, the DFE record cap) bounds each payload under the 16 MiB ceiling the producer and dfe-loader's consumer share, and the `send_batch` block under gRPC's 16 MiB ceiling. An event over the budget is dead-lettered and counted on `events_oversize_total`: no broker would take it, and retrying blocks the partition.

Duplicates are the accepted cost: a retried block re-sends the records that already landed, so every downstream consumer must be idempotent.

Expand Down
2 changes: 1 addition & 1 deletion docs/config-schema.json
Original file line number Diff line number Diff line change
Expand Up @@ -308,7 +308,7 @@
"default": null
},
"max_message_bytes": {
"description": "Ceiling on one outbound record. An event that serialises larger is\ndropped and counted on `events_oversize_total`.\n\nThe default is the DFE record cap, 8,000,000 bytes. Against a broker or\ntopic whose `message.max.bytes` is lower, set this below that limit, or\nthe broker refuses those events instead and they count on\n`send_filtered_dlq_total`. Brokers that carry more can raise it, up to\n16 MiB.",
"description": "Ceiling on one outbound record. An event that serialises larger is\ndead-lettered, with its size and this budget as the reason, and counted\non `events_oversize_total`.\n\nThe default is the DFE record cap, 8,000,000 bytes. Against a broker or\ntopic whose `message.max.bytes` is lower, set this below that limit, or\nthe broker refuses those events instead and they count on\n`send_filtered_dlq_total`. Brokers that carry more can raise it, up to\n16 MiB.",
"type": "integer",
"format": "uint",
"minimum": 0,
Expand Down
3 changes: 2 additions & 1 deletion docs/config-schema.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -324,7 +324,8 @@ $defs:
max_message_bytes:
description: |-
Ceiling on one outbound record. An event that serialises larger is
dropped and counted on `events_oversize_total`.
dead-lettered, with its size and this budget as the reason, and counted
on `events_oversize_total`.

The default is the DFE record cap, 8,000,000 bytes. Against a broker or
topic whose `message.max.bytes` is lower, set this below that limit, or
Expand Down
2 changes: 1 addition & 1 deletion docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,7 @@ On a direct source a push is held at most 18 s, so every send it waits on gives

## `sink.max_message_bytes` caps one outbound record

The default is the DFE record cap, 8,000,000 bytes: dfe-receiver refuses a request body over it, and every broker DFE deploys takes a record that size (Confluent Basic's 8,388,608 limit leaves room for the framing). The producer this service builds allows 16 MiB, so a deployment with larger brokers raises the budget up to that. An event that serialises larger is dropped, logged, counted on `events_oversize_total`, and its source is still acknowledged.
The default is the DFE record cap, 8,000,000 bytes: dfe-receiver refuses a request body over it, and every broker DFE deploys takes a record that size (Confluent Basic's 8,388,608 limit leaves room for the framing). The producer this service builds allows 16 MiB, so a deployment with larger brokers raises the budget up to that. An event that serialises larger is logged, counted on `events_oversize_total` and dead-lettered to the DLQ with its size and the budget as the reason. The DLQ cuts the body to `dlq.max_payload_bytes`, so it keeps `original_bytes` and a truncated head.

Against a broker or topic whose `message.max.bytes` is lower -- a stock Kafka broker takes about 1 MB -- set `sink.max_message_bytes` below that limit. Left at the default, a broker below 8,000,000 bytes refuses the events between the two instead: each is dropped as a dead letter, counted on `send_filtered_dlq_total`, and the partition moves on. Validation refuses a value under 4096 or over 16,777,216.

Expand Down
2 changes: 1 addition & 1 deletion docs/deployment-contract.json
Original file line number Diff line number Diff line change
Expand Up @@ -459,7 +459,7 @@
"default": null
},
"max_message_bytes": {
"description": "Ceiling on one outbound record. An event that serialises larger is\ndropped and counted on `events_oversize_total`.\n\nThe default is the DFE record cap, 8,000,000 bytes. Against a broker or\ntopic whose `message.max.bytes` is lower, set this below that limit, or\nthe broker refuses those events instead and they count on\n`send_filtered_dlq_total`. Brokers that carry more can raise it, up to\n16 MiB.",
"description": "Ceiling on one outbound record. An event that serialises larger is\ndead-lettered, with its size and this budget as the reason, and counted\non `events_oversize_total`.\n\nThe default is the DFE record cap, 8,000,000 bytes. Against a broker or\ntopic whose `message.max.bytes` is lower, set this below that limit, or\nthe broker refuses those events instead and they count on\n`send_filtered_dlq_total`. Brokers that carry more can raise it, up to\n16 MiB.",
"type": "integer",
"format": "uint",
"minimum": 0,
Expand Down
2 changes: 1 addition & 1 deletion docs/metrics-manifest.json
Original file line number Diff line number Diff line change
Expand Up @@ -875,7 +875,7 @@
{
"name": "events_oversize_total",
"type": "counter",
"description": "Events too large for one Kafka record, dropped",
"description": "Events too large for one Kafka record, dead-lettered",
"unit": "",
"labels": [],
"group": "custom"
Expand Down
3 changes: 2 additions & 1 deletion src/config.rs
Original file line number Diff line number Diff line change
Expand Up @@ -326,7 +326,8 @@ pub struct SinkConfig {
pub brokers: Option<Vec<String>>,

/// Ceiling on one outbound record. An event that serialises larger is
/// dropped and counted on `events_oversize_total`.
/// dead-lettered, with its size and this budget as the reason, and counted
/// on `events_oversize_total`.
///
/// The default is the DFE record cap, 8,000,000 bytes. Against a broker or
/// topic whose `message.max.bytes` is lower, set this below that limit, or
Expand Down
6 changes: 3 additions & 3 deletions src/metrics.rs
Original file line number Diff line number Diff line change
Expand Up @@ -82,8 +82,8 @@ pub struct TransformMetrics {
/// or topic ceiling below `sink.max_message_bytes`.
pub send_filtered_dlq: Counter,

/// Events too large for one Kafka record even on their own. Dropped: no
/// broker would ever accept one, and retrying blocks the partition.
/// Events too large for one Kafka record even on their own. Dead-lettered:
/// no broker would ever accept one, and retrying blocks the partition.
pub events_oversize: Counter,

/// Batches whose events did not look like the envelope the config pinned.
Expand Down Expand Up @@ -180,7 +180,7 @@ impl TransformMetrics {
),
events_oversize: manager.counter(
"events_oversize_total",
"Events too large for one Kafka record, dropped",
"Events too large for one Kafka record, dead-lettered",
),
envelope_contradicted: manager.counter(
"envelope_contradicted_total",
Expand Down
93 changes: 71 additions & 22 deletions src/pipeline.rs
Original file line number Diff line number Diff line change
Expand Up @@ -241,6 +241,16 @@ pub(crate) fn elided(value: &str) -> Cow<'_, str> {
/// The dead-letter reason for input that is not JSON.
pub const NOT_JSON: &str = "payload is not JSON";

/// The dead-letter reason for an event that serialised over the per-message
/// budget: names its size and the budget it broke.
#[must_use]
pub fn oversize_reason(bytes: usize, budget: usize) -> String {
format!(
"event serialises to {bytes} bytes, over the {budget} byte per-message budget \
(sink.max_message_bytes)"
)
}

/// What a payload cost to decode.
#[derive(Debug, Default, Clone, PartialEq, Eq)]
pub struct ParseOutcome {
Expand Down Expand Up @@ -353,8 +363,8 @@ pub struct SerialiseOutcome {
/// Events that became an outbound message.
pub serialised: usize,
/// Events that on their own exceed the whole per-message budget. No
/// broker will ever accept one, so it is dropped rather than blocking the
/// partition behind a record that can only fail.
/// broker will ever accept one, so it is dead-lettered rather than
/// blocking the partition behind a record that can only fail.
pub oversize: usize,
/// Events serde refused to serialise. Dropped, on the same footing as an
/// event the transform rejected.
Expand Down Expand Up @@ -394,6 +404,17 @@ pub const fn message_budget(max_bytes: usize) -> usize {
}
}

/// What [`serialise_events_with_oversize`] produced.
#[derive(Debug, Default)]
pub struct Serialised {
/// One payload per event that fits the budget.
pub payloads: Vec<Vec<u8>>,
/// The whole serialised body of each event over the budget, for the DLQ.
pub oversize: Vec<Vec<u8>>,
/// The counts behind both.
pub outcome: SerialiseOutcome,
}

/// Serialise each event into a message payload of its own.
///
/// ONE JSON document per payload, never NDJSON. dfe-loader parses exactly one
Expand All @@ -402,39 +423,52 @@ pub const fn message_budget(max_bytes: usize) -> usize {
/// (hyperi-io/dfe-transform-elastic#67).
///
/// `max_bytes` bounds each payload on its own, because that is what a broker
/// bounds: an event above the budget is dropped rather than blocking the
/// partition behind a record no broker would take.
/// bounds. An event above the budget is returned in [`Serialised::oversize`]
/// for the caller to dead-letter, rather than blocking the partition behind a
/// record no broker would take.
#[must_use]
pub fn serialise_events(events: &[Event], max_bytes: usize) -> (Vec<Vec<u8>>, SerialiseOutcome) {
pub fn serialise_events_with_oversize(events: &[Event], max_bytes: usize) -> Serialised {
let budget = message_budget(max_bytes);
let mut payloads = Vec::with_capacity(events.len());
let mut outcome = SerialiseOutcome::default();
let mut serialised = Serialised {
payloads: Vec::with_capacity(events.len()),
..Serialised::default()
};
// Written once and copied out at its exact length: a payload is held until
// its block is delivered, and a buffer sized by a guess holds the guess.
let mut scratch: Vec<u8> = Vec::with_capacity(1024);

for event in events {
scratch.clear();
if let Err(e) = event.write_json(&mut scratch) {
outcome.failed += 1;
serialised.outcome.failed += 1;
tracing::warn!(error = %e, "event could not be serialised, dropped");
continue;
}
if scratch.len() > budget {
outcome.oversize += 1;
serialised.outcome.oversize += 1;
tracing::error!(
bytes = scratch.len(),
budget,
"event exceeds the per-message budget and cannot be produced, dropped"
"event exceeds the per-message budget and cannot be produced, dead-lettered"
);
serialised.oversize.push(scratch.as_slice().to_vec());
continue;
}

outcome.serialised += 1;
payloads.push(scratch.as_slice().to_vec());
serialised.outcome.serialised += 1;
serialised.payloads.push(scratch.as_slice().to_vec());
}

(payloads, outcome)
serialised
}

/// [`serialise_events_with_oversize`] without the oversize bodies, for tests
/// and offline tools that only count them. The service keeps the bodies and
/// dead-letters them.
#[must_use]
pub fn serialise_events(events: &[Event], max_bytes: usize) -> (Vec<Vec<u8>>, SerialiseOutcome) {
let serialised = serialise_events_with_oversize(events, max_bytes);
(serialised.payloads, serialised.outcome)
}

/// Serialise every event into ONE NDJSON payload, whatever its size.
Expand Down Expand Up @@ -739,9 +773,9 @@ mod tests {
}

/// The shipped budget is the DFE record cap: a payload of 8,000,000 bytes
/// goes out and one byte more is dropped and counted.
/// goes out and one byte more is counted and held back for the DLQ.
#[test]
fn the_shipped_budget_admits_8_000_000_bytes_and_drops_one_more() {
fn the_shipped_budget_admits_8_000_000_bytes_and_holds_back_one_more() {
let budget = crate::config::default_max_message_bytes();
assert_eq!(budget, 8_000_000);
let overhead = serialise_events(&padded(1, 0), budget).0[0].len();
Expand All @@ -755,17 +789,32 @@ mod tests {
assert_eq!((outcome.serialised, outcome.oversize), (0, 1));
}

/// A record no broker can accept is dropped and counted, rather than
/// retried forever behind the rest of the partition.
/// A record no broker can accept is counted and handed back whole for the
/// DLQ, rather than retried forever behind the rest of the partition.
#[test]
fn an_event_larger_than_the_budget_is_dropped_and_counted() {
fn an_event_larger_than_the_budget_is_counted_and_returned_for_the_dlq() {
let mut events = padded(2, 100);
events.extend(padded(1, 20_000));

let (payloads, outcome) = serialise_events(&events, MIN_MESSAGE_BYTES);
assert_eq!(outcome.oversize, 1);
assert_eq!(outcome.serialised, 2);
assert_eq!(payloads.len(), 2);
let serialised = serialise_events_with_oversize(&events, MIN_MESSAGE_BYTES);
assert_eq!(serialised.outcome.oversize, 1);
assert_eq!(serialised.outcome.serialised, 2);
assert_eq!(serialised.payloads.len(), 2);
assert_eq!(serialised.oversize.len(), 1);
assert!(
serialised.oversize[0].len() > 20_000,
"the body was cut before it reached the DLQ"
);
let body: serde_json::Value =
serde_json::from_slice(&serialised.oversize[0]).expect("a whole document");
assert_eq!(body.get("i"), Some(&serde_json::json!(0)));
}

#[test]
fn the_oversize_reason_names_the_size_and_the_budget() {
let reason = oversize_reason(8_000_001, 8_000_000);
assert!(reason.contains("8000001 bytes"), "{reason}");
assert!(reason.contains("8000000 byte"), "{reason}");
}

/// A budget below one ordinary event would drop the entire batch as
Expand Down
44 changes: 31 additions & 13 deletions src/service.rs
Original file line number Diff line number Diff line change
Expand Up @@ -20,12 +20,12 @@
//! downstream must be idempotent, which is the contract every DFE stage carries.
//!
//! An event that cannot be delivered is dropped and counted, never counted
//! delivered: one that will not transform or serialise, one over
//! `sink.max_message_bytes`, and one the sink transport would refuse, which the
//! pipeline screens out before the send. Input that is not JSON is refused as a
//! dead letter. Dead letters, the screened records among them, go to the DLQ
//! `dlq` configures, and the source is released only once the DLQ holds them.
//! With `dlq.enabled: false` they are dropped and counted in
//! delivered: one that will not transform or serialise, and one the sink
//! transport would refuse, which the pipeline screens out before the send. An
//! event over `sink.max_message_bytes` and input that is not JSON are dead
//! letters. Dead letters, the screened records among them, go to the DLQ `dlq`
//! configures, and the source is released only once the DLQ holds them. With
//! `dlq.enabled: false` they are dropped and counted in
//! `pipeline_dead_letters_dropped_total`.

use std::sync::Arc;
Expand All @@ -51,8 +51,8 @@ use crate::agentless::{Agentless, Stopped};
use crate::config::{Config, Transport};
use crate::metrics::TransformMetrics;
use crate::pipeline::{
BatchOutcome, EnvelopeOutcome, NOT_JSON, SerialiseOutcome, message_budget, parse_each,
serialise_events, transform_batch_resolved,
BatchOutcome, EnvelopeOutcome, NOT_JSON, SerialiseOutcome, message_budget, oversize_reason,
parse_each, serialise_events_with_oversize, transform_batch_resolved,
};

/// The longest the loop goes without pushing scaling signals.
Expand Down Expand Up @@ -653,6 +653,8 @@ struct Block {
outcome: BatchOutcome,
envelopes: EnvelopeOutcome,
serialised: SerialiseOutcome,
/// Events over the per-message budget, held for the DLQ.
oversize: Vec<FilteredDlqEntry>,
}

impl Block {
Expand All @@ -666,6 +668,7 @@ impl Block {
outcome: BatchOutcome::default(),
envelopes: EnvelopeOutcome::default(),
serialised: SerialiseOutcome::default(),
oversize: Vec::new(),
}
}
}
Expand All @@ -685,7 +688,8 @@ struct Stage<'a> {

impl Stage<'_> {
/// Parse, transform and serialise one block, returning a record per event
/// that survived and a dead letter per input refused as not JSON.
/// that survived and a dead letter per input refused as not JSON or event
/// over the per-message budget.
///
/// At most `source.batch_size` parsed events are held at once, however many
/// NDJSON lines the block's records carry.
Expand Down Expand Up @@ -735,8 +739,10 @@ impl Stage<'_> {
outcome,
envelopes,
serialised,
oversize,
..
} = block;
dead.extend(oversize);

// One series, counted once. The source transport counts its own wire
// records in `transport_received_*`; these are the events parsed out.
Expand Down Expand Up @@ -816,17 +822,29 @@ impl Stage<'_> {
} => transform_batch_resolved(*transform, resolver, events),
Work::Agentless(agentless) => agentless.transform(events)?,
};
let (payloads, serialised) =
serialise_events(&transformed, self.config.sink.max_message_bytes);
let max_bytes = self.config.sink.max_message_bytes;
let budget = message_budget(max_bytes);
let serialised = serialise_events_with_oversize(&transformed, max_bytes);
drop(transformed);
block.out.extend(
payloads
serialised
.payloads
.into_iter()
.map(|payload| outbound_record(payload, &self.topic)),
);
block.oversize.extend(
serialised
.oversize
.into_iter()
.map(|payload| FilteredDlqEntry {
reason: oversize_reason(payload.len(), budget),
payload,
key: Some(Arc::clone(&self.topic)),
}),
);
block.outcome.absorb(outcome);
block.envelopes.absorb(envelopes);
block.serialised.absorb(serialised);
block.serialised.absorb(serialised.outcome);
Ok(())
}

Expand Down
Loading
Loading