diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 79c2013..cfcdc87 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,24 +1,21 @@ # Contributing -Thank you for considering a contribution to DPP. +DPP welcomes contributions that improve correctness, determinism, maintainability, portability, +documentation, test coverage, and measurable performance within the Community Edition scope. -Community contributions are welcome, especially when they improve correctness, determinism, -maintainability, portability, documentation, test coverage, and measurable performance on the -community-supported feature set. +## Before you start -## Before You Start - -- Read [README.md](README.md) for the product scope and operator-facing contract. -- Read [docs/architecture.md](docs/architecture.md) before changing runtime behavior or ownership - boundaries. -- Read [docs/rfc/README.md](docs/rfc/README.md) for architecture decisions and long-lived design - context. -- Read [benches/README.md](benches/README.md) before making or claiming performance changes. +| Before changing… | Read | +| --- | --- | +| Product behavior or supported features | [README](README.md) | +| Runtime behavior or ownership boundaries | [Architecture](docs/architecture.md) | +| A long-lived design decision | [RFC index](docs/rfc/README.md) | +| Performance, or making a performance claim | [Benchmark contract](benches/README.md) | For larger changes, please open an issue or start a discussion before investing in a large patch. Early alignment is the easiest way to avoid rework. -## Contribution Guidelines +## Contribution guidelines - Preserve ownership boundaries. - Do not introduce a second source of truth. @@ -29,7 +26,7 @@ Early alignment is the easiest way to avoid rework. - Call out hot-path performance risk when touching parser, matcher, pipeline, or writer code. - Mark unsupported assumptions as hypotheses. -## Testing Expectations +## Testing expectations At minimum: @@ -37,13 +34,13 @@ At minimum: - Run additional checks that match the change type. - Include benchmark evidence for hot-path performance claims. -Examples: - -- Parser or matcher changes should include targeted `cargo test` coverage. -- Writer changes should verify output compatibility and shutdown behavior. -- Benchmark harness changes should be validated with `bash -n benches/benchmark.sh` and a dry run. +| Change | Additional validation | +| --- | --- | +| Parser or matcher | Targeted `cargo test` coverage. | +| Writer | Output compatibility and shutdown behavior. | +| Benchmark harness | `bash -n benches/benchmark.sh` and a dry run. | -## Rebuilding a Missing Release Asset +## Rebuilding a missing release asset Once the corrected release workflow is on `main`, rebuild one platform from an existing tag: @@ -51,12 +48,15 @@ Once the corrected release workflow is on `main`, rebuild one platform from an e gh workflow run rust.yml --ref main -f tag=v0.4.0 -f runner=ubuntu-22.04 ``` -This runs the current workflow but checks out the source from `refs/tags/v0.4.0`. It does not -move the tag or overwrite existing release assets. The other runner choices are `ubuntu-24.04` -and `ubuntu-24.04-arm`. Tag pushes still build all three platforms. Re-running an older failed -run uses its original workflow revision, so it does not pick up a workflow fix committed later. +This runs the current workflow but checks out `refs/tags/v0.4.0`. It does not move the tag or +overwrite existing release assets. + +- Runner choices are `ubuntu-22.04`, `ubuntu-24.04`, and `ubuntu-24.04-arm`. +- Tag pushes build all three platforms. +- Re-running an old failed run uses its original workflow revision and does not pick up a later + workflow fix. -## Community and Commercial Scope +## Community and Commercial scope DPP Community Edition and DPP Commercial Edition intentionally have different scopes. diff --git a/README.md b/README.md index 95ffed7..9bdd2d7 100644 --- a/README.md +++ b/README.md @@ -17,27 +17,35 @@ High-performance offline DNS extraction from PCAP to CSV or Parquet.

-DPP is a Rust application for parsing, matching, and exporting DNS query/response traffic from offline PCAP files. It is designed for large captures, bounded parallel processing, and downstream analytics workflows. +DPP is a Rust application for parsing, matching, and exporting DNS query/response traffic from +offline PCAP files. It is designed for large captures, bounded parallel processing, and downstream +analytics workflows. -## Table of Contents +## Contents -- [Overview](#overview) -- [Features](#features) -- [Architecture](#architecture) -- [Documentation](#documentation) -- [Prerequisites](#prerequisites) +- [Overview and features](#overview) - [Build](#build) - [Usage](#usage) +- [Input and matching](#input-and-matching) +- [Output and reports](#output-and-reports) +- [Pseudonymization keys](#create-an-anonymization-key) +- [Synthetic capture generation](#synthetic-capture-generation) - [Example run](#example-run) -- [A simple AWK analysis to measure DNS traffic latency](#a-simple-awk-analysis-to-measure-dns-traffic-latency) -- [Performance Optimization](#performance-optimization) +- [Performance optimization](#performance-optimization) - [Limitations](#limitations) +- [Architecture](#architecture) +- [Documentation](#documentation) - [Commercial Edition](#commercial-edition) - [License](#license) ## Overview -DPP reads offline PCAP files, extracts DNS traffic, matches queries with responses, and writes structured output to CSV or Parquet. The pipeline emphasizes throughput, bounded backpressure, deterministic aggregation, and optional IP pseudonymization. +DPP reads offline captures, pairs DNS queries with responses, and exports transaction records. +It supports bounded parallel processing, deterministic aggregation, and optional IP +pseudonymization. + +Start with the [usage examples](#usage). For implementation details and diagrams, read the +[architecture reference](docs/architecture.md). ## Features @@ -54,33 +62,6 @@ DPP reads offline PCAP files, extracts DNS traffic, matches queries with respons - Graceful `SIGINT`/`SIGTERM` handling that stops intake and drains in-flight work; any still-buffered output tail is discarded before final writer teardown to avoid a skewed partial ending. - Graceful handling of malformed packets and I/O errors. -## Architecture - -The table below is a high-level map of the system. For the canonical architecture reference, -ownership boundaries, and matcher invariants, see [docs/architecture.md](docs/architecture.md). - -| Layer | Components | Responsibility | -|-----------------------------|---------------------------------------------------------------|----------------------------------------------------------------------------------------| -| Configuration and contracts | `src/config.rs`, `src/cli.rs`, `src/record.rs` | Runtime constants, CLI/environment contract, and exported `DnsRecord` schema | -| Input parsing | `PacketParser` | Offline capture input with pure-Rust classic-PCAP parsing, stream-native stdin parsing, and `libpcap` fallback for non-classic file inputs | -| DNS processing | `DnsProcessor`, `pipeline.rs` | Packet routing, DNS decoding, matching, shard ownership, and deterministic aggregation | -| Runtime and orchestration | `src/app.rs`, `src/runtime.rs`, `src/allocator.rs` | App lifecycle, bootstrap, reporting, thread-pool setup, and allocator selection | -| Output pipeline | `src/output.rs`, `src/csv_writer.rs`, `src/parquet_writer.rs` | Async writer lifecycle and CSV/Parquet serialization | -| Memory monitoring | `src/monitor_memory.rs` | Peak RSS tracking with explicit stop/join lifecycle | -| References and benchmarks | `docs/rfc/`, `benches/` | Architecture decisions, benchmark workflow, and harnesses | - -## Documentation - -- Canonical architecture reference: [docs/architecture.md](docs/architecture.md) -- Architecture decision records: [docs/rfc/README.md](docs/rfc/README.md) -- Benchmark workflow: [benches/README.md](benches/README.md) -- Benchmark harness: [benches/benchmark.sh](benches/benchmark.sh) -- Contribution guide: [CONTRIBUTING.md](CONTRIBUTING.md) -- Allocator guide: [docs/allocator-guide.md](docs/allocator-guide.md) -- Allocator benchmarking protocol: [benches/allocator-benchmarking.md](benches/allocator-benchmarking.md) -- Encapsulation handling playbook: [docs/encapsulation-playbook.md](docs/encapsulation-playbook.md) -- Synthetic DNS PCAP generator: [docs/synthetic-pcap-generator.md](docs/synthetic-pcap-generator.md) - ## Prerequisites - [Rust](https://rustup.rs/) 1.98.1 or newer; this repository is pinned by `rust-toolchain.toml`. @@ -120,6 +101,8 @@ safe production build. ## Usage +Examples below assume `dpp` is on `PATH`. After a local build, use `target/release/dpp` instead. + ### Basic examples ```bash @@ -148,13 +131,311 @@ cat input.pcap | dpp - output.csv dpp --report-format json input.pcap output.csv > dpp-summary.json ``` -If `output_filename` is omitted, DPP chooses the default file name from the resolved output format: +### Arguments + +| Argument | Description | +| --- | --- | +| `filename` | Path to the input PCAP file. Use `-` to read a finite offline PCAP stream from `stdin`. | +| `output_filename` | Optional output path; use `-` for CSV stdout. If neither this argument nor `DPP_OUTPUT_FILENAME` is set, the format determines the [default path](#files-and-streams). | + +### Options + +| Option | Description | +| --- | --- | +| `-s, --silent` | Suppress info-level log output | +| `-f, --format ` | Select output format; stdout output is supported only for `csv` | +| `--report-format ` | Select the final process report format; defaults to `text`; `json` cannot be combined with `output_filename = -` | +| `--match-timeout-ms ` | Set the DNS query-response match timeout in milliseconds; allowed range is `1..=5000`, default is `1200` | +| `--monotonic-capture` | Assume globally monotonic packet timestamps, enable batched timeout eviction, and abort if a regression is detected | +| `-t, --threads ` | Cap the CPU execution budget at a positive `N`, never above available CPUs | +| `-b, --bonded ` | Set I/O channel capacity in records; internally rounded up to batched messages of up to `1024` records; `0` uses the safe default bounded capacity | +| `-z, --zstd` | Enable Zstd compression for Parquet output | +| `--v2` | Use Parquet Version 2 | +| `-a, --affinity` | Apply CPU affinity to processing threads | +| `--dns-wire-fast-path` | Enable the optional question-only DNS wire fast path with `hickory` fallback | +| `--max-dns-compression-jumps ` | Maximum compression-pointer jumps per DNS name; defaults to `32`; `0` disables the jump limit | +| `--allow-fragments` | Opt in to matching complete queries with first IPv4 response fragments when the DNS header and question are present; no reassembly | +| `--full-fragments` | Reassemble IPv4 fragments into complete UDP datagrams; also enables `--allow-fragments` for incomplete responses | +| `--anonymize ` | Path to the pseudonymization key file | +| `-h, --help` | Print help | +| `-V, --version` | Print version | + +On macOS, DPP continues without affinity after one warning because the platform does not +provide supported per-core thread pinning through the affinity backend. + +### Environment variables + +| Variable | Description | +| --- | --- | +| `DPP_FILENAME` | Input PCAP path. Use `-` to read a finite offline PCAP stream from `stdin` | +| `DPP_OUTPUT_FILENAME` | Optional output file path; use `-` for CSV stdout output | +| `DPP_FORMAT` | Output format: `csv`, `parquet`, or `pq` | +| `DPP_REPORT_FORMAT` | Final process report format: `text` or `json`; defaults to `text`; `json` cannot be combined with `DPP_OUTPUT_FILENAME=-` | +| `DPP_MATCH_TIMEOUT_MS` | DNS query-response match timeout in milliseconds; allowed range is `1..=5000`, default is `1200` | +| `DPP_MONOTONIC_CAPTURE` | Assume globally monotonic packet timestamps, enable batched timeout eviction, and abort if a regression is detected | +| `DPP_THREADS` | Maximum CPU execution budget when `--threads` is not set; must be a positive integer | +| `DPP_BONDED` | I/O channel capacity in records; internally rounded up to batched messages of up to `1024` records; `0` uses the default bounded capacity | +| `DPP_ZSTD` | Enable Zstd compression for Parquet output | +| `DPP_V2` | Enable Parquet Version 2 | +| `DPP_AFFINITY` | Apply CPU affinity to processing threads | +| `DPP_DNS_WIRE_FAST_PATH` | Enable the optional DNS wire fast path | +| `DPP_MAX_DNS_COMPRESSION_JUMPS` | Maximum compression-pointer jumps per DNS name when `--max-dns-compression-jumps` is not set; defaults to `32`; `0` disables the jump limit | +| `DPP_ALLOW_FRAGMENTS` | Enable first-fragment IPv4 response matching without reassembly. | +| `DPP_FULL_FRAGMENTS` | Enable IPv4 reassembly and first-fragment response matching. | +| `DPP_ANONYMIZE` | Path to the key file used for pseudonymization | +| `DPP_SILENT` | Suppress info-level log output | + +## Input and matching + +### Files and streams + +If neither `output_filename` nor `DPP_OUTPUT_FILENAME` is set, DPP chooses the default file name +from the resolved output format: `dns_output.csv` for `csv` and `dns_output.parquet` for `parquet` or `pq`. Use `-` only when you -want CSV records on stdout. DPP refuses to start when the input and output paths refer to the same +want CSV records on stdout. + +DPP refuses to start when the input and output paths refer to the same file, including hard-link aliases, so the input capture cannot be overwritten. The same protection applies when the output path refers to the anonymization key file. -### Create an anonymization key +| Input or output | Contract | +| --- | --- | +| `filename = -` | Read classic PCAP or PCAPNG from stdin until EOF. This is finite offline input, not live capture. | +| `output_filename = -` | Write CSV records to stdout; suppress non-error logs and the final text report. | +| Stdout restrictions | Reject Parquet output and JSON reports, including `DPP_REPORT_FORMAT=json`. | + +Classic PCAP files and stdin streams use the pure-Rust parser family. PCAPNG stdin uses a +stream-native reader; regular non-classic files use the `libpcap` fallback. Unsupported stdin +magic is rejected without a temporary file or second ingest path. + +### Match identity and retries + +Queries and responses are paired using the original client IP, client port, resolver IP, DNS ID, +QNAME, QTYPE, QCLASS, OPCODE, and VLAN context. DNS QR determines direction, including when both UDP ports are 53. +QCLASS, OPCODE, and VLAN context stay internal and add no output columns. + +VLAN identity contains each tag's TPID and VLAN ID. Priority (PCP) and drop eligibility (DEI) do +not distinguish transactions on the same tagged segment. + +Pending queries with the same identity inside the match window (`1200ms` by default) are +deduplicated to the earliest canonical query. Retries increment a separate counter and do not +produce extra matched or timeout records. + +Default mode retains pending retry timestamps. An earlier timestamp in a later batch can split +pending groups without widening the match window. Finalized transactions are never reopened. +Monotonic mode does not need or allocate retry history. + +**Case matters in Community Edition.** Matcher identity preserves observed presentation-form +QNAME bytes. RFC 4343 permits case-only differences in valid responses, including differences +caused by compression; those pairs may fail to match in DPP. + +This is a deliberate trade-off for offline caching-resolver workloads, not an RFC guarantee. +See the [matcher contract](docs/architecture.md#matcher-contract) for the full identity and ordering +rules. + +### Timestamp order + +In default mode, backwards capture timestamps produce a red warning after processing. Matching +remains deterministic, but pairing quality may degrade. Normalize the capture with Wireshark's +`reordercap` when needed: + +```bash +reordercap input.pcap normalized.pcap +``` + +For globally ordered captures, `--monotonic-capture` or `DPP_MONOTONIC_CAPTURE=1` enables batched +timeout eviction, which can reduce matcher RSS. The first timestamp regression aborts the run; +this optimization must not weaken matching semantics. + +### DNS message validation + +Ordinary QUERY messages with more than one question are rejected under RFC 9619. Responses with +truncated declared answer or authority records are also rejected before matching. + +QNAME limits apply to the decompressed wire form: + +| Limit | Meaning | +| --- | --- | +| 255 octets | Maximum wire QNAME, including label-length octets and terminating root. | +| 1003 bytes | Possible escaped presentation form of a valid wire name; these names stay distinct in matching and export. | + +If any QNAME exceeds the wire limit, DPP rejects the **entire DNS message** before matching or +export. No questions from that message become query or response records. + +The text report labels the counter `DNS messages rejected for oversized QNAME`; JSON uses +`metrics.dns_messages_rejected_oversized_qname`. Both count rejected messages, not questions. + +### IPv4 fragment modes + +| Mode | Input accepted | +| --- | --- | +| Default | Skip IPv4 fragments. | +| `--allow-fragments` / `DPP_ALLOW_FRAGMENTS=1` | Use an observed first response fragment with a complete DNS header and question; no reassembly. | +| `--full-fragments` / `DPP_FULL_FRAGMENTS=1` | Reassemble complete IPv4 UDP datagrams; also enable first-response-fragment fallback. | + +For a response prefix, `response_code` is present only when there are no additional records or +all declared DNS records fit in the prefix. The metric `fragmented_response_prefix_count` (text: +`Accepted first IPv4 response fragments`) counts accepted prefixes, even without a matching query. + +Full reassembly uses the final fragment's (`MF=0`) timestamp. Incomplete responses may fall back +to their first fragment on capacity eviction or EOF, or on match-timeout expiry in monotonic +mode. Fallback uses the first fragment's timestamp. + +Fragmented queries need complete reassembly; +overlapping, inconsistent, or UDP-length-mismatched fragment sets are rejected. + +Duplicate suppression uses bounded history and compares the fragment key and UDP payload bytes: + +- A reused IPv4 ID with different content can form a new datagram, even with an identical first + fragment. +- A fully identical new datagram, or an incomplete one whose observed fragments match a recent + completion, may be suppressed because it is indistinguishable from a duplicate. +- A separate bounded history drops tails after a non-DNS first fragment. If a new DNS datagram + reuses that ID and its tail arrives before its first fragment while the mark is active, the + tail may also be dropped. + +For buffer ownership, validation boundaries, and ordering, see +[IPv4 fragmentation](docs/architecture.md#ipv4-fragmentation). + + +## Output and reports + +### What DPP produces in CSV output + +DPP writes one row per canonical DNS query outcome. + +```csv +request_timestamp,response_timestamp,source_ip,source_port,id,name,query_type,response_code +1774783431482391,1774783431503127,10.0.0.1,53000,4660,example.com,A,No Error +1774783447118904,,10.0.0.2,53001,48879,example.org,A, +``` +In this example: + +- `example.com A` matched a response about 20.736ms later. +- `example.org A` had no matching response within the timeout window; both response fields are + empty. + +### Response fields + +| Output value | Meaning | +| --- | --- | +| Missing `response_timestamp` | No matching response observed inside the configured timeout window. This is the canonical timeout signal. | +| Missing `response_code` on a timeout | No DNS response exists; this does not mean the server returned `ServFail`. | +| Present timestamp, missing code | A first-fragment response prefix matched, but its complete response code was unavailable. | +| `EDNS_BADVERS` | Extended RCODE 16 from EDNS OPT. | +| `TSIG Failure` | Code 16 supplied by a TSIG RR error field. | +| `TYPE`, such as `TYPE65400` | An unknown query type, retaining its wire number. | +| Decimal code, such as `64` | An unknown response code, retaining its distinct value. | + +Missing response fields are empty in CSV and `NULL` in Parquet. Both formats preserve unknown +type/code values rather than collapsing them to an `Unknown` label. + +### JSON summaries + +`--report-format json` or `DPP_REPORT_FORMAT=json` suppresses routine `info`/`warn` reporting and +writes one final summary object to stdout. CSV or Parquet records remain in their own output file. + +The summary includes these processing and matching-quality metrics: + +- `metrics.dns_messages_rejected_oversized_qname` +- `metrics.timed_out_queries` +- `metrics.timed_out_query_ratio` +- `metrics.average_matched_rtt_ms` + +### Interrupted and failed runs + +On `SIGINT` or `SIGTERM`, DPP stops intake and drains accepted work. It skips synthetic timeout +finalization for pending unmatched queries, discards the still-buffered writer tail, and exits. +This policy applies to CSV and Parquet. JSON reports it as `warnings.graceful_signal_shutdown`. + +Stdin shutdown does not require producer EOF. If interruption happens before a capture header +can be read, DPP creates neither an output writer nor a final report. + +A capture read failure after processing starts drains complete accepted batches and flushes +conclusive records as valid partial output, then returns an error. Pending queries do not become +timeout records. If processing failure and a signal coincide, processing failure takes precedence +and buffered output is flushed. + +See [shutdown behavior](docs/architecture.md#output-reporting-and-shutdown) for the lifecycle +contract and error precedence. + + + +### Analyze latency with AWK + +This example uses GNU AWK to calculate timeout counts and RTT percentiles from the timestamp +columns. + +
+Show the AWK command and example output + +```bash +gawk -F',' ' +NR > 1 { + total++ + req = $1 + resp = $2 + + if (req == "") { invalid++; next } + if (resp == "") { timeout++; next } + + d_ms = (resp - req) / 1000.0 + if (d_ms < 0) { invalid++; next } + + ok++ + sum += d_ms + a[ok] = d_ms +} +END { + printf "total_rows: %d\n", total + printf "ok_rows: %d\n", ok + printf "timeout_rows: %d\n", timeout + printf "invalid_rows: %d\n", invalid + printf "timeout_ratio: %.4f%%\n", (total ? 100.0 * timeout / total : 0) + + if (ok == 0) exit 0 + + asort(a) + mean = sum / ok + median = (ok % 2) ? a[(ok + 1) / 2] : (a[ok / 2] + a[ok / 2 + 1]) / 2 + + printf "mean_ms: %.6f\n", mean + printf "median_ms: %.6f\n", median + printf "p50_ms: %.6f\n", pct(a, ok, 50) + printf "p95_ms: %.6f\n", pct(a, ok, 95) + printf "p99_ms: %.6f\n", pct(a, ok, 99) + printf "p99.9_ms: %.6f\n", pct(a, ok, 99.9) +} +function pct(arr, n, p, rank, x) { + x = (p / 100) * n + rank = int(x) + if (x > rank) rank++ + if (rank < 1) rank = 1 + if (rank > n) rank = n + return arr[rank] +} +' dns_output.csv +``` + +Example output for the illustrative capture: + +```text +total_rows: 19646090 +ok_rows: 19630152 +timeout_rows: 15938 +invalid_rows: 0 +timeout_ratio: 0.0811% +mean_ms: 1.303967 +median_ms: 0.021000 +p50_ms: 0.021000 +p95_ms: 0.034000 +p99_ms: 34.479000 +p99.9_ms: 257.322000 +``` + +
+ +## Create an anonymization key `--anonymize` accepts a legacy text key file or a salted v2 key file. DPP derives the internal pseudonymization keys from the file's passphrase and salt. @@ -191,13 +472,16 @@ PY dpp --anonymize anon-v2.key input.pcap output.csv ``` -Notes: +### Key formats and handling - Keep the key file private. Anyone with the same file can reproduce the same pseudonymized output. - A legacy key file contains a non-empty UTF-8 passphrase. A salted v2 file contains the exact binary header `\x89DPP-ANON-KEY-v2\n`, a line of exactly 64 hexadecimal digits representing a 32-byte salt, then a non-empty UTF-8 passphrase. Leading and trailing passphrase whitespace is ignored. Malformed salted files are rejected. + +### Mapping and compatibility + - Rotating the key changes the resulting pseudonymized IP addresses for the same input capture. - Matching and deduplication use the original client IP. Pseudonymization is applied only to the `source_ip` field of finalized output records. IPv4 pseudonymization is a keyed permutation, so @@ -209,154 +493,17 @@ Notes: - The IPv4 Feistel mapping replaces the previous truncated-AES mapping. Even with the same passphrase, exports produced before and after this change cannot be joined by pseudonymized IPv4 `source_ip`. IPv6 pseudonyms remain unchanged. -- If `--anonymize` or `DPP_ANONYMIZE` is configured and the key file is missing, unreadable, or - invalid, DPP exits with an error. It does not silently fall back to pass-through IP addresses. -### Arguments +### Key-loading errors -| Argument | Description | -|-------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `filename` | Path to the input PCAP file. Use `-` to read a finite offline PCAP stream from `stdin`. | -| `output_filename` | Optional output file path. If omitted, DPP writes to `dns_output.csv` for `csv` and to `dns_output.parquet` for `parquet` or `pq`. Use `-` for CSV stdout output. | +If `--anonymize` or `DPP_ANONYMIZE` is configured and the key file is missing, unreadable, or +invalid, DPP exits with an error. It does not silently fall back to pass-through IP addresses. -### Options - -| Option | Description | -|-----------------------------------|---------------------------------------------------------------------------------------------------------------------| -| `-s, --silent` | Suppress info-level log output | -| `-f, --format ` | Select output format; stdout output is supported only for `csv` | -| `--report-format ` | Select the final process report format; defaults to `text`; `json` cannot be combined with `output_filename = -` | -| `--match-timeout-ms ` | Set the DNS query-response match timeout in milliseconds; allowed range is `1..=5000`, default is `1200` | -| `--monotonic-capture` | Assume globally monotonic packet timestamps, enable batched timeout eviction, and abort if a regression is detected | -| `-t, --threads ` | Cap the CPU execution budget at a positive `N`, never above available CPUs | -| `-b, --bonded ` | Set I/O channel capacity in records; internally rounded up to batched messages of up to `1024` records; `0` uses the safe default bounded capacity | -| `-z, --zstd` | Enable Zstd compression for Parquet output | -| `--v2` | Use Parquet Version 2 | -| `-a, --affinity` | Apply CPU affinity to processing threads | -| `--dns-wire-fast-path` | Enable the optional question-only DNS wire fast path with `hickory` fallback | -| `--max-dns-compression-jumps ` | Maximum compression-pointer jumps per DNS name; defaults to `32`; `0` disables the jump limit | -| `--allow-fragments` | Opt in to matching complete queries with first IPv4 response fragments when the DNS header and question are present; no reassembly | -| `--full-fragments` | Reassemble IPv4 fragments into complete UDP datagrams; also enables `--allow-fragments` for incomplete responses | -| `--anonymize ` | Path to the pseudonymization key file | -| `-h, --help` | Print help | -| `-V, --version` | Print version | - -On macOS, DPP continues without affinity after one warning because the platform does not -provide supported per-core thread pinning through the affinity backend. - -### Environment variables - -| Variable | Description | -|--------------------------|---------------------------------------------------------------------------------------------------------------------------| -| `DPP_FILENAME` | Input PCAP path. Use `-` to read a finite offline PCAP stream from `stdin` | -| `DPP_OUTPUT_FILENAME` | Optional output file path; use `-` for CSV stdout output | -| `DPP_FORMAT` | Output format: `csv`, `parquet`, or `pq` | -| `DPP_REPORT_FORMAT` | Final process report format: `text` or `json`; defaults to `text`; `json` cannot be combined with `DPP_OUTPUT_FILENAME=-` | -| `DPP_MATCH_TIMEOUT_MS` | DNS query-response match timeout in milliseconds; allowed range is `1..=5000`, default is `1200` | -| `DPP_MONOTONIC_CAPTURE` | Assume globally monotonic packet timestamps, enable batched timeout eviction, and abort if a regression is detected | -| `DPP_THREADS` | Maximum CPU execution budget when `--threads` is not set; must be a positive integer | -| `DPP_BONDED` | I/O channel capacity in records; internally rounded up to batched messages of up to `1024` records; `0` uses the default bounded capacity | -| `DPP_ZSTD` | Enable Zstd compression for Parquet output | -| `DPP_V2` | Enable Parquet Version 2 | -| `DPP_AFFINITY` | Apply CPU affinity to processing threads | -| `DPP_DNS_WIRE_FAST_PATH` | Enable the optional DNS wire fast path | -| `DPP_MAX_DNS_COMPRESSION_JUMPS` | Maximum compression-pointer jumps per DNS name when `--max-dns-compression-jumps` is not set; defaults to `32`; `0` disables the jump limit | -| `DPP_ANONYMIZE` | Path to the key file used for pseudonymization | -| `DPP_SILENT` | Suppress info-level log output | - -If neither `output_filename` nor `DPP_OUTPUT_FILENAME` is set, DPP chooses the default output path -from the resolved format: `dns_output.csv` for `csv` and `dns_output.parquet` for `parquet` or -`pq`. - -When `output_filename` is `-`, DPP writes CSV records to stdout, suppresses non-error log output, and does not emit -the final text report. `--format parquet`, `--report-format json`, and `DPP_REPORT_FORMAT=json` are rejected in this -mode. - -When `filename` is `-`, DPP reads a finite offline PCAP stream from `stdin` until EOF and then completes the run -normally. This preserves the existing offline-processing contract: `stdin` is not treated as a live capture source. -Classic PCAP streams on `stdin` now use the same pure-Rust parser family as classic file input, and PCAPNG streams -also use a stream-native parser path. For regular files, non-classic formats still fall back to `libpcap`. -Unsupported non-PCAP/non-PCAPNG stream magic on `stdin` is rejected explicitly instead of being redirected through a -hidden temp-file or second ingest path. - -Repeated pending queries that share the same match identity (`id`, `name`, client IP, client port, -resolver IP, `query_type`, QCLASS, and OPCODE) inside the configured match window (`1200ms` by default) are -deduplicated to the earliest canonical query. Deduplicated retries increment a separate counter and -do not emit extra timeout or matched records. In default mode, pending retry timestamps are retained -so an earlier timestamp arriving in a later batch can split pending groups without widening the -match window. Finalized transactions are never reopened. With `--monotonic-capture`, no retry -history is needed or allocated. - -DNS QR determines query/response direction, including valid exchanges with UDP port 53 on both -endpoints. QCLASS, OPCODE and the full VLAN/QinQ tag stack distinguish transactions internally -and do not add output columns. VLAN identity includes each tag's TPID and VLAN ID; changes to -priority (PCP) or drop eligibility (DEI) do not prevent matching within the same tagged segment. -Ordinary DNS QUERY messages with more than one question are rejected under RFC 9619. Responses -whose declared answer or authority records are truncated are also rejected before matching. - -For QNAME matching, DPP preserves the observed presentation-form name bytes and does not lowercase -them before building matcher identity keys. This is a deliberate Community Edition trade-off, not -an RFC-level guarantee: RFC 4343 defines ASCII label comparison as case-insensitive, and a valid -response is allowed to differ from the query's 0x20 casing, including when name compression -reuses label bytes from another wire location. DPP still keeps byte-preserving matcher identity -because that better matches the real behavior we target on offline caching-resolver workloads. As -a result, a query and response that differ only by case may fail to match even on otherwise valid -DNS traffic. - -DPP accepts QNAMEs up to the RFC 1035 limit of 255 octets in decompressed wire form, including -label-length octets and the terminating root octet. Escaping arbitrary label bytes can expand that -valid wire name to as many as 1003 bytes in DPP's presentation form; those expanded names remain -distinct and are matched and exported without substituting an empty name. If any decompressed QNAME -in a DNS message exceeds 255 octets, DPP rejects the entire message before matching or export. The -text report includes `DNS messages rejected for oversized QNAME`; JSON exposes the count as -`metrics.dns_messages_rejected_oversized_qname`. The unit is DNS messages, not individual questions, -and no questions from a rejected message contribute DNS query or response records. - -Timeout records leave response fields empty: - -- `response_timestamp = NULL` in Parquet and an empty field in CSV -- `response_code = NULL` in Parquet and an empty field in CSV - -This means "no matching response was observed inside the configured timeout window". It does not -mean the upstream server actually returned `ServFail`. Downstream timeout detection should key off -the absent `response_timestamp`; `response_code` is absent because no DNS response exists. - -`response_code` preserves DNS extended-RCODE provenance when DPP can observe it. EDNS OPT -extended RCODE `16` is exported as `EDNS_BADVERS`; `TSIG Failure` is used for code `16` only when a -TSIG RR error field supplies that status. - -Unknown query types retain their wire number as `TYPE` (for example, `TYPE65400`), -and unknown response codes use their decimal number (for example, `64`). CSV and Parquet -preserve these values instead of collapsing distinct codes into an `Unknown` label. - -With `--report-format json` or `DPP_REPORT_FORMAT=json`, DPP suppresses routine `info`/`warn` -reporting and emits one final JSON summary object to `stdout`. This keeps CSV or Parquet output in -its own file while making the end-of-run report easy to capture and parse. - -If DPP receives `SIGINT` or `SIGTERM`, it stops accepting new packet batches, drains already -accepted work, skips synthetic timeout finalization for pending unmatched queries, discards any -still-buffered output tail, and then exits. This applies to CSV and Parquet outputs alike. The -final JSON summary reports this through the `warnings.graceful_signal_shutdown` field. When input -comes from stdin, shutdown does not require the stream producer to close its end of the pipe. -If a signal arrives before a capture header can be read, no output writer or final report is -created. - -The final report now also includes basic processing and matching-quality metrics: - -- `metrics.dns_messages_rejected_oversized_qname` -- `metrics.timed_out_queries` -- `metrics.timed_out_query_ratio` -- `metrics.average_matched_rtt_ms` - -If packet timestamps move backwards in capture order, DPP emits a red warning after processing. -Matching remains deterministic, but timestamp-based pairing quality may degrade on such input. -When that warning appears, normalize the capture first with Wireshark's `reordercap`, for example: -`reordercap input.pcap normalized.pcap` - -## Synthetic Capture Generation +## Synthetic capture generation This repository also ships a standalone utility, `dns-pcap-generator`, for producing synthetic DNS classic-PCAP files without needing a source capture at runtime. + The tool lives in the separate workspace crate `tools/dns-pcap-generator`; from the repository root, use `-p dns-pcap-generator`, or run the same commands directly inside that directory without `-p`. @@ -389,27 +536,27 @@ cargo run -p dns-catalog-builder --release -- \ ``` The checked-in `server1-jul-2024` profile is shaped after a representative July 2024 resolver -workload while keeping only sanitized, non-client-specific domains. Its -`fitted-generator.toml` lives under `tools/dns-pcap-generator/profiles/server1-jul-2024`, while +workload while keeping only sanitized, non-client-specific domains. + +Its `fitted-generator.toml` lives under `tools/dns-pcap-generator/profiles/server1-jul-2024`, while the profile's `catalog_path` points back to `tools/dns-pcap-generator/catalog_data.tsv` so the -sanitized catalog remains a single reviewable source of truth. The generator verifies the -referenced catalog digest before generation. See +sanitized catalog remains a single reviewable source of truth. + +The generator verifies the referenced catalog digest before generation. See [docs/synthetic-pcap-generator.md](docs/synthetic-pcap-generator.md) for the full contract and current modeling assumptions. -For captures that are already globally monotonic by timestamp, `--monotonic-capture` or -`DPP_MONOTONIC_CAPTURE=1` enables batched timeout eviction inside the matcher. This can reduce -in-flight matcher RSS, but the run will abort on the first detected timestamp regression so the -optimization does not silently weaken matching semantics. - ## Example run The log below is illustrative. Exact throughput, memory usage, and match counters depend on the hardware, build profile, capture shape, and matcher revision. -```bash +
+Show the illustrative run log + +```text # On Intel(R) Atom(TM) x7425E with power-save profile. # No output file name is specified here, so CSV mode uses the default output path: dns_output.csv. -$ DPP_FILENAME=server1_jul_2024.pcap DPP_FORMAT=csv target/release/dpp --dns-wire-fast-path --monotonic-capture +$ DPP_FILENAME=server1_jul_2024.pcap DPP_FORMAT=csv target/release/dpp --dns-wire-fast-path --monotonic-capture 04:15:01.046 INFO DNS Packet Parser (DPP) community edition, version hash: dbed8b3 04:15:01.046 INFO Git Commit Author: k.mikhailov@dnstele.com 04:15:01.046 INFO Git Timestamp: 2026-03-26T06:06:59.000000000+02:00 @@ -447,131 +594,98 @@ $ DPP_FILENAME=server1_jul_2024.pcap DPP_FORMAT=csv target/release/dpp --dns-wir ``` -### What DPP produces in CSV output +
-DPP writes one row per canonical DNS query outcome. +## Performance optimization -```csv -request_timestamp,response_timestamp,source_ip,source_port,id,name,query_type,response_code -1774783431482391,1774783431503127,10.0.0.1,53000,4660,example.com,A,No Error -1774783447118904,,10.0.0.2,53001,48879,example.org,A, +Build for the target host and benchmark representative captures; throughput depends on hardware, +traffic shape, and output backpressure. + +```bash +RUSTFLAGS='-C target-cpu=native' cargo build --release ``` -What this tells you: -- the first query for example.com A was matched with a response about 20.736 ms later; -- the second query for example.org A had no matching response within the timeout window; -- empty response_timestamp and response_code mean timeout. -- a response_timestamp with an empty response_code means a first-fragment response prefix matched, - but the complete response code was unavailable. +| Control | Guidance | +| --- | --- | +| Benchmark harness | Use [benchmark.sh](benches/benchmark.sh) with `DPP_BENCH_PCAP` or `--pcap`; measure throughput and shutdown tail. | +| `perf` profile | For comparable trusted measurements, use `cargo build --profile perf` or `DPP_BENCH_PROFILE=perf bash benches/benchmark.sh ...`. See the [build caveat](#build). | +| CPU budget | Auto-size from available CPUs, or cap with `--threads` / `DPP_THREADS`. Pipeline selection remains automatic. | +| Question decoder | Opt in with `--dns-wire-fast-path` / `DPP_DNS_WIRE_FAST_PATH=1`; otherwise use the Hickory question decoder. | +| Monotonic capture | Enable batched eviction only for globally ordered captures; regressions abort the run. | +| Allocator | Choose at build time using the [allocator guide](docs/allocator-guide.md) and [comparison protocol](benches/allocator-benchmarking.md). | -### A simple AWK analysis to measure DNS traffic latency -```bash -$ gawk -F',' ' -NR > 1 { - total++ - req = $1 - resp = $2 +Both decoder modes default to **32 compression-pointer jumps per name**. Set +`--max-dns-compression-jumps N` or `DPP_MAX_DNS_COMPRESSION_JUMPS=N` to change the resource limit; +`0` disables it. Cached suffixes count toward the full chain depth. - if (req == "") { invalid++; next } - if (resp == "") { timeout++; next } +Bounds, backward-pointer, overlap, and name-length checks stay active regardless of the jump +limit. Validated suffixes are cached per message to avoid repeated walks. See the +[benchmark contract](benches/README.md) before making performance comparisons. - d_ms = (resp - req) / 1000.0 - if (d_ms < 0) { invalid++; next } - ok++ - sum += d_ms - a[ok] = d_ms -} -END { - printf "total_rows: %d\n", total - printf "ok_rows: %d\n", ok - printf "timeout_rows: %d\n", timeout - printf "invalid_rows: %d\n", invalid - printf "timeout_ratio: %.4f%%\n", (total ? 100.0 * timeout / total : 0) +## Limitations - if (ok == 0) exit 0 +### Capture and protocol coverage - asort(a) - mean = sum / ok - median = (ok % 2) ? a[(ok + 1) / 2] : (a[ok / 2] + a[ok / 2 + 1]) / 2 +| Area | Limit | +| --- | --- | +| DNS transport | UDP port 53 only. | +| Linktype | Ethernet only. VLAN/QinQ TPIDs `0x8100`, `0x88a8`, and `0x9100` are decoded natively; other declared linktypes are rejected. | +| Outer encapsulation | MPLS and other unsupported encapsulations need [preprocessing](docs/encapsulation-playbook.md). | +| PCAPNG | Stream-native stdin support and `libpcap` regular-file fallback; the performance-focused pure-Rust fast path targets classic PCAP. Stdin blocks over 16MiB are rejected before reading the body. | +| IPv6 | Hop-by-Hop, Routing, Destination Options, Authentication, and atomic Fragment headers are traversed. IPv6 reassembly, ESP, and jumbograms are unsupported. | +| Partial IPv4 responses | First-fragment inference cannot establish whether later fragments arrived or validate the full UDP/DNS contents. The response code may be unavailable. | - printf "mean_ms: %.6f\n", mean - printf "median_ms: %.6f\n", median - printf "p50_ms: %.6f\n", pct(a, ok, 50) - printf "p95_ms: %.6f\n", pct(a, ok, 95) - printf "p99_ms: %.6f\n", pct(a, ok, 99) - printf "p99.9_ms: %.6f\n", pct(a, ok, 99.9) -} -function pct(arr, n, p, rank, x) { - x = (p / 100) * n - rank = int(x) - if (x > rank) rank++ - if (rank < 1) rank = 1 - if (rank > n) rank = n - return arr[rank] -} -' dns_output.csv -total_rows: 19646090 -ok_rows: 19630152 -timeout_rows: 15938 -bad_rows: 0 -mean_ms: 1.303967 -median_ms: 0.021000 -p50_ms: 0.021000 -p95_ms: 0.034000 -p99_ms: 34.479000 -p99.9_ms: 257.322000 -``` +Flow identity uses observed IP endpoints. Read [fragment modes](#ipv4-fragment-modes) before opting +into inference or reassembly. +### Ordering, memory, and scaling -## Performance Optimization +- CSV and Parquet records are not guaranteed to be timestamp-sorted. Asynchronous Parquet layout + may also differ byte-for-byte between runs. +- RAM depends on capture size, traffic shape, and backpressure. Larger `--bonded` values increase + peak memory under slow sinks; capacity rounds up to messages of at most `1024` records. +- Full reassembly holds ready packets while fragment sets remain unresolved. The retained backlog + is bounded between batches to 65,536 packets or 64MiB; overflow triggers capacity fallback. + The current input batch and fallback packets may temporarily exceed those bounds. +- Flow affinity and skewed traffic can limit scaling before linear speedup, even with more CPUs. +- Pending duplicate queries are deduplicated within the match window; duplicate responses remain + distinguishable until matched or discarded. Duplicate-heavy workloads can increase matcher RAM. -For best throughput on the target host: +### Interpreting results -```bash -RUSTFLAGS='-C target-cpu=native' cargo build --release -``` +Case-only QNAME differences may prevent a match; see [match identity](#match-identity-and-retries). +Timestamp regressions can reduce pairing quality, and monotonic mode treats them as errors. +Capture failures can leave valid partial output; see [interrupted and failed runs](#interrupted-and-failed-runs). -Additional notes: - -- Use [benches/benchmark.sh](benches/benchmark.sh) with `DPP_BENCH_PCAP` or `--pcap` to measure throughput and shutdown tail on representative input. -- See [benches/README.md](benches/README.md) for the benchmark contract and data-handling notes. -- For apples-to-apples performance testing, prefer the dedicated `perf` build profile: - `cargo build --profile perf` or `DPP_BENCH_PROFILE=perf bash benches/benchmark.sh ...`. -- DPP auto-sizes its execution budget from available CPUs unless `--threads` or `DPP_THREADS` - sets a lower ceiling. The pipeline model is still selected automatically from that budget. -- The parser fast path is opt-in through `--dns-wire-fast-path` or `DPP_DNS_WIRE_FAST_PATH=1`; without that flag DPP uses the legacy `hickory` question decoder. -- Both parser modes limit DNS compression chains to `32` pointer jumps per name by default. Use `--max-dns-compression-jumps N` or `DPP_MAX_DNS_COMPRESSION_JUMPS=N` to change the limit; `0` disables this resource limit. Cached suffixes still count toward the full chain depth. Bounds, backward-pointer, overlap, and existing name-length validation remain active. Checked suffixes are cached within each message to avoid repeatedly walking shared chains. -- `--allow-fragments` or `DPP_ALLOW_FRAGMENTS=1` optionally pairs a query with the observable first IPv4 fragment of a response when its DNS header and complete question are present. `response_code` is populated when there are no additional records or every declared DNS record fits in the first fragment; otherwise it is empty. The JSON metric `fragmented_response_prefix_count` (text: `Accepted first IPv4 response fragments`) counts accepted prefixes, including ones without a matching query. -- `--full-fragments` or `DPP_FULL_FRAGMENTS=1` enables IPv4 reassembly and implicitly enables `--allow-fragments`. Complete fragmented UDP datagrams enter the existing DNS parser with the timestamp of their final IPv4 fragment (`MF=0`). An incomplete first response fragment falls back to the prefix heuristic on capacity eviction or end of input; with `--monotonic-capture`, it can also expire after the configured match timeout. The fallback keeps the first fragment's timestamp. A bounded history of completed datagrams compares the fragment key and UDP payload bytes to suppress repeated complete datagrams and prefix fallback when all observed fragments match a recent completion. A different datagram can still use the same IPv4 ID, including one with an identical first fragment but different later bytes. A fully identical new datagram, or an incomplete one whose observed fragments match a recent completion, cannot be distinguished from a duplicate and may be suppressed. A bounded non-DNS history drops later tails after a non-DNS first fragment; if a new DNS datagram reuses the same IPv4 ID and its tail arrives before its first fragment while that history is active, the tail is indistinguishable and may be dropped. Fragmented queries require complete reassembly. Overlapping, inconsistent, or UDP-length-mismatched fragment sets are rejected. -- For captures normalized with `reordercap`, `--monotonic-capture` can reduce in-flight matcher state by enabling batched timeout eviction. If a timestamp regression is detected, DPP fails the run instead of silently weakening matching semantics. -- Global allocator choice is a build-time concern. See [docs/allocator-guide.md](docs/allocator-guide.md) for the supported allocator matrix and [benches/allocator-benchmarking.md](benches/allocator-benchmarking.md) for the comparison protocol. -## Limitations +## Architecture + +The table below is a high-level map of the system. For the canonical architecture reference, +ownership boundaries, and matcher invariants, see [docs/architecture.md](docs/architecture.md). + +| Layer | Components | Responsibility | +| --- | --- | --- | +| Configuration and contracts | `src/config.rs`, `src/cli.rs`, `src/record.rs` | Runtime constants, CLI/environment contract, and exported `DnsRecord` schema | +| Input parsing | `PacketParser` | Offline capture input with pure-Rust classic-PCAP parsing, stream-native stdin parsing, and `libpcap` fallback for non-classic file inputs | +| DNS processing | `DnsProcessor`, `pipeline.rs` | Packet routing, DNS decoding, matching, shard ownership, and deterministic aggregation | +| Runtime and orchestration | `src/app.rs`, `src/runtime.rs`, `src/allocator.rs` | App lifecycle, bootstrap, reporting, thread-pool setup, and allocator selection | +| Output pipeline | `src/output.rs`, `src/csv_writer.rs`, `src/parquet_writer.rs` | Async writer lifecycle and CSV/Parquet serialization | +| Memory monitoring | `src/monitor_memory.rs` | Peak RSS tracking with explicit stop/join lifecycle | +| References and benchmarks | `docs/rfc/`, `benches/` | Architecture decisions, benchmark workflow, and harnesses | + +## Documentation + +| Task | Reference | +| --- | --- | +| Understand components and processing | [Architecture and diagrams](docs/architecture.md) | +| Read accepted design decisions | [RFC index](docs/rfc/README.md) | +| Contribute a change | [Contribution guide](CONTRIBUTING.md) | +| Run repeatable benchmarks | [Benchmark workflow](benches/README.md) and [harness](benches/benchmark.sh) | +| Select and compare allocators | [Allocator guide](docs/allocator-guide.md) and [benchmark protocol](benches/allocator-benchmarking.md) | +| Prepare unsupported encapsulations | [Encapsulation playbook](docs/encapsulation-playbook.md) | +| Generate test captures | [Synthetic DNS PCAP generator](docs/synthetic-pcap-generator.md) | -- **UDP/53 only:** DPP currently processes DNS traffic over UDP port 53 only. -- By default, IPv4 fragments are skipped. `--allow-fragments` uses only an observed first IPv4 - response fragment when its DNS header and complete question are available. This heuristic cannot - confirm that later fragments arrived or validate the full UDP/DNS contents; the response code - may be unknown if needed records are beyond the first fragment. `--full-fragments` reconstructs - complete IPv4 datagrams before the existing UDP/DNS checks and falls back to the first-response - heuristic for incomplete sets. Neither mode reassembles IPv6 fragments. IPv6 Hop-by-Hop, Routing, - Destination Options, Authentication and atomic Fragment headers are traversed before UDP; ESP - and IPv6 jumbograms are unsupported. Flow identity uses observed IP endpoints. -- Full IPv4 reassembly holds ready packet batches until unresolved fragments are completed or - released, preserving retry and response ordering across batches. This adds a retained backlog - of up to 65,536 packets or 64 MiB of packet payloads; exceeding either bound triggers the - incomplete-fragment capacity fallback. The current input batch adds transient memory on top. -- If capture parsing fails after processing begins, DPP flushes valid partial (not atomic) output - from complete accepted batches and exits with an error; pending queries are not emitted as timeouts. -- **PCAPNG support level:** DPP supports PCAPNG on stream input and via `libpcap` on regular-file fallback paths, but the performance-critical pure-Rust fast path remains focused on classic PCAP. Stdin PCAPNG blocks larger than 16 MiB are rejected before their body is read. -- **Ethernet linktype only:** DPP rejects captures or packets declared with non-Ethernet linktypes instead of interpreting them as Ethernet. Ethernet frames may carry VLAN or QinQ tags (`0x8100`, `0x88a8`, `0x9100`) before IPv4 or IPv6. MPLS and other outer encapsulation layers still require preprocessing; see [docs/encapsulation-playbook.md](docs/encapsulation-playbook.md). -- **Unsorted exported data:** CSV and Parquet outputs are not guaranteed to be timestamp-sorted. -- **Variable RAM usage:** Memory usage depends on capture size, traffic shape, and output backpressure. Larger `--bonded` values increase peak memory usage under slow output sinks because the output channel rounds the requested record backlog up to batched messages of up to `1024` records each. -- **Monotonic-capture mode is explicit:** Batched timeout eviction is available only with `--monotonic-capture` because it depends on globally monotonic packet timestamps. If the capture is not monotonic, DPP aborts and recommends `reordercap`. -- **Scaling ceilings on skewed workloads:** DPP auto-sizes from available CPUs or the configured CPU cap, but flow-affinity ceilings can still limit scaling before linear speedup. -- **Duplicate query handling:** Duplicate in-flight DNS queries and responses are preserved and resolved through deterministic matcher tie-breakers derived from capture order. Duplicate-heavy workloads can still increase matcher memory usage, and Parquet output may vary byte-for-byte because writers remain asynchronous. -- **QNAME casing trade-off:** Matcher identity preserves observed QNAME casing instead of canonicalizing names to lowercase. RFC 4343 allows ASCII-label case-only differences between a query and a valid response, and name compression can contribute to that mismatch. Community Edition still matches on observed bytes because that better reflects the caching-resolver workloads it targets. As a result, names that differ only by case may not match even on protocol-compliant traffic. ## Commercial Edition @@ -579,30 +693,30 @@ DPP Community Edition is focused on deterministic offline DNS processing from PC DPP Commercial Edition extends that foundation with broader capture support, enterprise integrations, compliance-oriented capabilities, and deployment options for production environments. -| Community Edition | Commercial Edition | -|-------------------------------------------------------|-------------------------------------------------------------------------------------------------| -| Offline PCAP processing | Native S3 integration for reading PCAP | -| CSV and single-file Parquet export | Partitioned Parquet datasets, not just single-file export | -| Native Ethernet, VLAN and QinQ decoding | Advanced encapsulation support: MPLS, GRE, VXLAN, ERSPAN, Geneve | -| Local file outputs | Direct enterprise sinks: ClickHouse, Kafka, S3, PostgreSQL outputs | -| Batch-oriented processing | Live/continuous ingestion | -| Deterministic pseudonymization with file-based keying | Commercial anonymization/compliance features | -| Current DNS field and message coverage | Extended DNS protocol coverage | -| Standalone synthetic DNS PCAP generation | Profile extraction, fitting, and validation pipeline for calibrated synthetic DNS traffic | -| Checked-in runtime traffic profiles | Fitted profiles derived from reference captures, with validation reports and tuning artifacts | -| Manual binary-oriented deployment | Containerized delivery for easier deployment in cloud-native environments | -| Built-in runtime reporting | Prometheus/OpenTelemetry metrics | -| CLI-oriented offline workflows | Programmatic DPP library API with observer hooks for canonical per-query telemetry | -| GPL/community distribution | Flexible commercial licensing for both source code and pre-built binaries | -| Self-service benchmarking | Benchmark/tuning help | -| Current parsing and processing stack | An alternative packet parsing and DNS processing stack aimed at broader protocol coverage | - -Commercial licensing and support: `carrier-support@dnstele.com` +| Community Edition | Commercial Edition | +| --- | --- | +| Offline PCAP processing | Native S3 integration for reading PCAP | +| CSV and single-file Parquet export | Partitioned Parquet datasets, not just single-file export | +| Native Ethernet, VLAN and QinQ decoding | Advanced encapsulation support: MPLS, GRE, VXLAN, ERSPAN, Geneve | +| Local file outputs | Direct enterprise sinks: ClickHouse, Kafka, S3, PostgreSQL outputs | +| Batch-oriented processing | Live/continuous ingestion | +| Deterministic pseudonymization with file-based keying | Commercial anonymization/compliance features | +| Current DNS field and message coverage | Extended DNS protocol coverage | +| Standalone synthetic DNS PCAP generation | Profile extraction, fitting, and validation pipeline for calibrated synthetic DNS traffic | +| Checked-in runtime traffic profiles | Fitted profiles derived from reference captures, with validation reports and tuning artifacts | +| Manual binary-oriented deployment | Containerized delivery for easier deployment in cloud-native environments | +| Built-in runtime reporting | Prometheus/OpenTelemetry metrics | +| CLI-oriented offline workflows | Programmatic DPP library API with observer hooks for canonical per-query telemetry | +| GPL/community distribution | Flexible commercial licensing for both source code and pre-built binaries | +| Self-service benchmarking | Benchmark/tuning help | +| Current parsing and processing stack | An alternative packet parsing and DNS processing stack aimed at broader protocol coverage | + +Commercial licensing and support: `carrier-support@dnstele.com` Nameto Oy also offers commercial support for organizations using the Community Edition, including deployment guidance, troubleshooting, benchmarking, and production-readiness assistance. ## License Licensed under GNU GPLv3. -Commercial licensing: `carrier-support@dnstele.com` +Commercial licensing: `carrier-support@dnstele.com` Copyright © 2026 Nameto Oy. All rights reserved. diff --git a/benches/README.md b/benches/README.md index d68fd51..4dd644e 100644 --- a/benches/README.md +++ b/benches/README.md @@ -2,48 +2,36 @@ This directory is the canonical location for repeatable performance measurements. +Use [benchmark.sh](benchmark.sh) for end-to-end measurements, or choose a focused guide below. + ## Files -- `benchmark.sh`: end-to-end benchmark harness for the `dpp` binary. -- `allocator-benchmarking.md`: canonical build-and-measure protocol for global allocator - comparisons. -- `protocol-correctness-audit.md`: final validation, restored throughput and measured memory costs - of the 2026-09-05 protocol and matcher fixes, with earlier experiments identified separately. -- `dns-compression.md`: configurable compression limits, message-local suffix caching, - repeated performance measurements, RR-loop inlining investigation and reproduction instructions. -- `dns-compression-benchmark.py` and `dns-compression/`: synthetic end-to-end and isolated - scanner benchmarks for short names, shared suffixes and deep pointer chains. +| File or directory | Purpose | +| --- | --- | +| [benchmark.sh](benchmark.sh) | End-to-end benchmark harness for the `dpp` binary. | +| [allocator-benchmarking.md](allocator-benchmarking.md) | Canonical build-and-measure protocol for global allocator comparisons. | +| [protocol-correctness-audit.md](protocol-correctness-audit.md) | Final validation, restored throughput and measured memory costs of the 2026-09-05 protocol and matcher fixes; earlier experiments are identified separately. | +| [dns-compression.md](dns-compression.md) | Compression limits, message-local suffix caching, repeated measurements, RR-loop inlining and reproduction instructions. | +| [dns-compression-benchmark.py](dns-compression-benchmark.py) and [dns-compression/](dns-compression/) | Synthetic end-to-end and isolated scanner benchmarks for short names, shared suffixes and deep pointer chains. | ## Safety Contract - The benchmark input PCAP must be provided explicitly with `--pcap` or `DPP_BENCH_PCAP`. - The script does not upload data or fetch remote inputs. - Results are written under the selected output directory, which defaults to `target/benchmarks/...`. -- Benchmark outputs can contain derived data from the input capture, so avoid committing or sharing them unless that is intended for the dataset. +- Benchmark outputs can contain derived data from the input capture. Avoid committing or sharing + them unless that is intended for the dataset. - The script exits non-zero if any benchmarked run fails. -## What It Measures - -`benchmark.sh` records: +## Setup -- build profile -- output format -- execution budget label -- bonded channel capacity -- run number -- whether the run used `--silent` -- whether the row contains full metrics or wall-clock-only metrics -- current git SHA and benchmarked binary path -- `rustc` version and verbose compiler metadata in `metadata.txt` -- total wall-clock time -- processing speed parsed from application logs -- final writer shutdown tail parsed from application logs -- exit code -- log file path -- output file path +For trusted benchmark runs, prefer the dedicated `perf` Cargo profile. It inherits from `release` +but disables overflow checks so benchmark numbers reflect the faster arithmetic path without +changing the default production build. -For `--silent` runs, the script records wall-clock and exit status, but log-derived throughput and -shutdown-tail fields are intentionally left blank. +DPP auto-sizes execution from all available CPUs. The harness does not support a runtime +thread-count sweep. To compare smaller CPU budgets, limit CPU availability externally with your +platform tooling and run the harness once per environment. ## Example @@ -59,10 +47,14 @@ bash benches/benchmark.sh \ Metrics are written to `metrics.csv` inside the chosen benchmark output directory. Run-level provenance is written to `metadata.txt` in the same directory. -Note: DPP now auto-sizes execution from all available CPUs. The harness no longer supports a -runtime thread-count sweep; if you need to compare smaller CPU budgets, limit CPU availability -externally with your platform tooling and run the harness once per environment. +## What It Measures -For trusted benchmark runs, prefer the dedicated `perf` Cargo profile. It inherits from `release` -but disables overflow checks so benchmark numbers reflect the faster arithmetic path without -changing the default production build. +| Category | Recorded fields | +| --- | --- | +| Run configuration | Build profile, output format, execution budget label, bonded channel capacity, run number and whether `--silent` was used. | +| Provenance | Current git SHA, benchmarked binary path, `rustc` version and verbose compiler metadata in `metadata.txt`. | +| Measurements | Total wall-clock time; processing speed and final writer shutdown tail parsed from application logs. | +| Result and artifacts | Exit code, log file path, output file path and whether the row has full metrics or wall-clock-only metrics. | + +For `--silent` runs, the script records wall-clock time and exit status. Log-derived throughput +and shutdown-tail fields are intentionally left blank. diff --git a/benches/allocator-benchmarking.md b/benches/allocator-benchmarking.md index c259dd8..b943310 100644 --- a/benches/allocator-benchmarking.md +++ b/benches/allocator-benchmarking.md @@ -4,10 +4,7 @@ This document records the canonical protocol for comparing DPP global allocator ## Goal -Allocator experiments must answer two questions on the same representative capture: - -- does throughput improve or regress; -- does peak RSS improve or regress. +Compare both throughput and peak RSS on the same representative capture. Allocator changes are meaningful only when compared under the same binary flags, CPU availability, input capture, and output format. @@ -16,15 +13,23 @@ input capture, and output format. Exactly one allocator feature must be enabled at build time. -- default build: `allocator-jemalloc` -- alternative builds: - - `allocator-system` - - `allocator-mimalloc` - - `allocator-tcmalloc` on Linux `x86_64` and Linux `aarch64` only +| Build | Feature | Platform restriction | +| --- | --- | --- | +| Default | `allocator-jemalloc` | — | +| Alternative | `allocator-system` | — | +| Alternative | `allocator-mimalloc` | — | +| Alternative | `allocator-tcmalloc` | Linux `x86_64` and Linux `aarch64` only. | + +### Notes + +`allocator-tcmalloc` starts a background maintenance thread during process bootstrap. +`cargo check --all-features` is intentionally excluded because allocator features are mutually +exclusive. ## Build Matrix -Requires Rust 1.98.1 or newer; this repository is pinned by `rust-toolchain.toml`. +Requires Rust 1.98.1 or newer; the repository is pinned by +[`rust-toolchain.toml`](../rust-toolchain.toml). Build each variant into a separate binary path. @@ -50,11 +55,11 @@ allocator builds and document the chosen baseline. ## Benchmark Protocol 1. Build each allocator variant separately. -2. Run the canonical benchmark harness with the same capture, format set, and bonded value. +2. Run the [canonical benchmark harness](README.md) with the same capture, format set and bonded value. 3. Record throughput, shutdown tail, and peak RSS from logs. 4. Compare outputs for determinism before accepting any allocator change. -Example: +The following example compares jemalloc and mimalloc using the saved binaries: ```bash bash benches/benchmark.sh \ @@ -79,9 +84,3 @@ bash benches/benchmark.sh \ - No correctness regression. - No byte-for-byte CSV drift for the same representative run. - Clear benchmark evidence that the allocator change is worth the operational tradeoff. - -## Notes - -- `allocator-tcmalloc` starts a background maintenance thread during process bootstrap. -- `cargo check --all-features` is intentionally not part of this protocol because allocator features - are mutually exclusive. diff --git a/benches/dns-compression.md b/benches/dns-compression.md index a8f4e6b..fe4f642 100644 --- a/benches/dns-compression.md +++ b/benches/dns-compression.md @@ -1,13 +1,27 @@ # DNS compression validation and performance +This document records compression behavior, benchmark commands and results for the initial +implementation and its later RR-loop inlining fix. The result sections identify the measured +commits; their timings and test counts describe those revisions. + +- [Behavior and validation](#behavior-and-validation) +- [Initial measurement method](#initial-measurement-method-400acc5) +- [Reproduction](#reproduction) +- [Follow-up: RR-loop inlining](#follow-up-rr-loop-inlining-9b329b9) +- [Initial results](#initial-results-400acc5) + ## Behavior and validation +### Compression limits + `--max-dns-compression-jumps` defaults to 32. The environment equivalent is `DPP_MAX_DNS_COMPRESSION_JUMPS`; an explicit CLI value takes precedence. Zero disables the jump-count policy, while backward-pointer, bounds and nonoverlap checks remain mandatory. The policy covers questions, scanned RR owners and TSIG algorithm names, including partial responses and the optional wire fast path's Hickory fallback. +### Decoder ownership and scan paths + The decoder belongs to one immutable DNS message. Cached suffixes retain their encoded segment end, expanded length, pointer depth and first label. Reuse checks the segment end against the new referring name and charges the entire cached @@ -17,7 +31,9 @@ successful walk. No cache survives the message or crosses worker ownership. Short RR-owner walks use a direct scanner: at most eight pointers and 64 literal bytes per referenced segment. Crossing either threshold switches to the cached scanner from the original name offset. These thresholds select an implementation, -not a different acceptance policy. Uncompressed questions in the optional fast +not a different acceptance policy. + +Uncompressed questions in the optional fast path retain a single-pass formatter. Compressed questions are validated before Hickory receives their expanded wire form, avoiding recursive replay of long pointer chains. @@ -26,14 +42,149 @@ The RR section loop and metadata reader are both explicitly inlined. This lets the compiler remove metadata unused by answer/authority scans and retain the scan cursor in registers; see the follow-up investigation below. +### Validation coverage + The tests cover exact jump boundaries, zero/unlimited, warmed-cache depth and overlap checks, literal versus compressed OPT roots, TSIG, partial fragments, malformed and oversized questions, per-message isolation, and bounded traversal -work. The strictly backward, nonoverlapping walker accepts a synthetic raw name +work. + +The strictly backward, nonoverlapping walker accepts a synthetic raw name with 8,193 pointer transitions when unlimited; normal DNS record framing further limits how many such segments fit into a message. The deep benchmark is a valid 64,419-byte DNS response with 4,600 RRs and a maximum depth of 1,170. +## Initial measurement method (`400acc5`) + +The end-to-end baseline is commit `7807751`, immediately before configurable +limits and suffix caching. The measured implementation is `400acc5`. Both binaries +use Rust 1.97.1, the `perf` profile and default jemalloc features, on macOS/aarch64. +`perf` disables overflow checks; these measurements +must not be interpreted as measurements of the production `release` profile. + +`dns-compression-benchmark.py` generates deterministic synthetic PCAPs and `.dns` +messages locally. Each configuration gets an excluded warm-up pair followed by +10 measured pairs, alternating which binary runs first. Builds and other test +processes are stopped during measurement. All `DPP_*` environment overrides are +removed for each child process. + +The harness validates packet, query, response, match and timeout counters on every +run, then compares CSV contents. Parallel output is sorted before comparison. +It records binary SHA-256, fixture dimensions, per-run processing time, wall time, +RSS and checksums. + +The main statistic is the median of the paired +candidate/baseline processing-time ratios; below 1 is faster. The interquartile +range (IQR) describes dispersion, **not** a confidence interval. A single favorable +run is not evidence of an improvement. + +The isolated scanner benchmark uses a fresh message-local decoder per iteration +and a historical uncached walker with the **same** configurable limit. This +separates scanner/cache cost from the end-to-end changes in question parsing and +mandatory validation. The full-application uncached control makes the same +comparison for RR-name skipping while keeping the new question parser (including +its suffix cache) and policy. + +## Reproduction + +### Build and run the end-to-end cases + +Build the baseline and candidate with the same compiler, target and allocator: + +```bash +bench_dir=$(mktemp -d /tmp/dpp-compression.XXXXXX) +mkdir "$bench_dir/baseline" +git archive 7807751 | tar -x -C "$bench_dir/baseline" +cargo build --manifest-path "$bench_dir/baseline/Cargo.toml" \ + --target-dir "$bench_dir/baseline-target" --profile perf --locked --offline +cp "$bench_dir/baseline-target/perf/dpp" "$bench_dir/dpp-before" +cargo build --target-dir "$bench_dir/candidate-target" --profile perf --locked --offline +cp "$bench_dir/candidate-target/perf/dpp" "$bench_dir/dpp-after" + +python3 benches/dns-compression-benchmark.py \ + --baseline "$bench_dir/dpp-before" --candidate "$bench_dir/dpp-after" \ + --outdir "$bench_dir/short" --runs 10 --transactions 1000000 \ + --cases empty,one,two,eight,literal-eight,three-label-eight,chain8-eight \ + --limits 32 --decoders both + +python3 benches/dns-compression-benchmark.py \ + --baseline "$bench_dir/dpp-before" --candidate "$bench_dir/dpp-after" \ + --outdir "$bench_dir/unlimited" --runs 10 --transactions 200000 \ + --cases one,three-label-eight,long-name-eight,chain2-eight --limits 0 --decoders both + +python3 benches/dns-compression-benchmark.py \ + --baseline "$bench_dir/dpp-before" --candidate "$bench_dir/dpp-after" \ + --outdir "$bench_dir/dense" --runs 10 --transactions 200000 \ + --cases distinct-128,suffix-128,chain32-128,flat-4600,deep-4600 \ + --limits 32,0 --decoders both + +python3 benches/dns-compression-benchmark.py \ + --baseline "$bench_dir/dpp-before" --candidate "$bench_dir/dpp-after" \ + --outdir "$bench_dir/staged" --runs 10 --transactions 200000 \ + --cases one,chain32-128,deep-4600 --threads 5 --limits 32,0 --decoders both +``` + +Dense fixtures scale the transaction count by 20 for 128 RRs and by 1,000 for +4,600 RRs. The harness omits configurations whose depth exceeds the requested +nonzero limit, so the deep case runs only with zero. Add `--threads 5` to exercise +the staged pipeline, and use a new output directory for each experiment. + +### Run the isolated scanner + +Build and run the isolated scanner using dependency paths reported by Cargo for +the current compiler, rather than picking arbitrary old `.rlib` files: + +```bash +python3 benches/dns-compression/build-scan.py --output "$bench_dir/dns-compression-scan" +"$bench_dir/dns-compression-scan" "$bench_dir/short" 10 100 > "$bench_dir/scan-short.csv" +"$bench_dir/dns-compression-scan" "$bench_dir/unlimited" 10 100 > "$bench_dir/scan-unlimited.csv" +"$bench_dir/dns-compression-scan" "$bench_dir/dense" 10 100 > "$bench_dir/scan-dense.csv" +``` + +### Run the uncached control + +To reproduce the end-to-end control with uncached RR-name skipping, export the +candidate commit and apply the benchmark-only patch in that separate directory: + +```bash +mkdir "$bench_dir/control" +git archive HEAD | tar -x -C "$bench_dir/control" +git -C "$bench_dir/control" apply "$PWD/benches/dns-compression/uncached-control.patch" +cargo build --manifest-path "$bench_dir/control/Cargo.toml" \ + --target-dir "$bench_dir/control-target" --profile perf --locked --offline +python3 benches/dns-compression-benchmark.py \ + --baseline "$bench_dir/control-target/perf/dpp" --candidate "$bench_dir/dpp-after" \ + --baseline-limit 32 --limits 32 --decoders fast \ + --cases one,two,eight,literal-eight,three-label-eight \ + --outdir "$bench_dir/control-results" --runs 10 --transactions 1000000 +``` + +Use an explicit `--baseline-limit 0` when comparing unlimited configurations with +this control. Leave `--baseline-limit` unset for the historical binary, which +predates the option. The patch is a measurement control, not a production mode. + +### Keep inputs and builds separate + +The scripts use synthetic data only and do not upload anything. Keep raw output +under the selected benchmark directory rather than committing generated PCAPs. +Keep build directories separate: exporting several revisions into one shared +Cargo target directory can leave the top-level executable from the last copy even +when a later build of another copy reports `Fresh`. + +### Recorded checks for the initial implementation + +Final correctness checks on `400acc5` passed: 353 workspace tests, formatting, +`git diff --check`, and Clippy with the existing `too_many_arguments`, +`type_complexity` and `large_enum_variant` allowances. CLI smoke checks cover +default rejection at 33 jumps, explicit 33, zero, environment configuration and +CLI precedence, in both decoder modes. + +Before the final inlining and plain-QNAME +optimizations, a separate randomized comparison of the cached and uncached +validators covered six million calls over 20,000 messages, with cold and warm +caches and limits 0 through 11. The final changes also received an independent +review of cursor handling, error classification and limit equivalence. + ## Follow-up: RR-loop inlining (`9b329b9`) The initial short-message regression was investigated with single-factor builds @@ -48,6 +199,7 @@ Production validation and cache behavior are unchanged by the fix: it adds only The arm64 disassembly of `7807751` contains the RR scan directly inside `decode_response_code_with_sections`. The compiler keeps the scan cursor in a register and removes unused metadata, including TTL, from answer/authority scans. + In `400acc5`, `read_dns_resource_record_meta` became a separate 656-byte function called for every RR. Each call creates a 64-byte stack frame, saves/restores six registers, reads TTL and writes the full metadata result to the stack. The caller @@ -60,7 +212,9 @@ the new decoder, rather than the cost of HashMap lookups on these fixtures. Forcing only the metadata reader inline (F) removed the per-RR calls and dead metadata, but left the section loop as a separate function; timing did not -improve consistently. Forcing both levels inline (G, the selected fix) restored +improve consistently. + +Forcing both levels inline (G, the selected fix) restored the loop within the caller, including a register-held cursor, and produced the measured improvement. Overflow, fixed-header length and full RDATA bounds checks remain in the resulting code. @@ -79,12 +233,16 @@ remain in the resulting code. G also reduced the paired median on eight compressed owners to 0.973 [0.970–0.995] and on two owners to 0.980 [0.956–0.981]. These are screening results, not a claim that each removed instruction accounts for a known fraction -of elapsed time. All controls produced matching CSV contents and counters. +of elapsed time. + +All controls produced matching CSV contents and counters. Workspace tests (353), Clippy with the existing lint allowances, formatting and diff checks passed after applying G. ### Confirmation results +#### Direct comparison with the pre-fix source + Each configuration below has ten alternating pairs and an excluded warm-up pair. The direct causal comparison uses the rebuilt pre-fix source (B0) and production `9b329b9`, both with limit 32 and the optional fast decoder, one @@ -97,7 +255,11 @@ processing thread and one million transactions: | Two compressed owners | 0.237620 | 0.234812 | 0.984 [0.977–0.997] | This confirms approximately 4.2%, 2.9% and 1.6% less processing time respectively -from the selected inlining change. A separate ten-pair comparison against the +from the selected inlining change. + +#### Comparison with the historical decoder + +A separate ten-pair comparison against the historical `7807751` binary, at the same transaction count and candidate limit, shows the remaining differences: @@ -115,8 +277,9 @@ shows the remaining differences: The earlier 3–4% penalties on eight literal or directly compressed answer owners no longer reproduce; their remaining median differences are below 1%, with IQRs -crossing 1. A smaller -0.8–1.4% difference remains on several other cases with IQR above 1. Those costs +crossing 1. + +A smaller 0.8–1.4% difference remains on several other cases with IQR above 1. Those costs have **not** been assigned to a specific instruction or exclusively to the jump limit. The experiment localizes and fixes the main regression, not every last nanosecond of the difference from the historical decoder. @@ -126,6 +289,8 @@ therefore also includes the default decoder, all three RR sections, dense/deep responses and the staged pipeline; the tables below retain the observed differences rather than assuming all workloads benefit. +#### Default decoder + The default decoder's short-message comparisons against `7807751` use one million transactions, one thread and limit 32: @@ -135,6 +300,8 @@ transactions, one thread and limit 32: | Eight literal owners | 0.351446 | 0.318372 | 0.908 [0.897–0.913] | | Eight owners sharing `www.example.com` | 0.450267 | 0.410812 | 0.911 [0.909–0.917] | +#### Dense responses and staged execution + Dense cases use 10,000 transactions for 128 RRs and 200 for 4,600 RRs; staged one-owner runs use 200,000 transactions. Each cell remains a paired ratio to `7807751`, followed by its IQR: @@ -151,8 +318,9 @@ one-owner runs use 200,000 transactions. Each cell remains a paired ratio to The shared-suffix fast case retains an observed +1.9% median with IQR above 1; it is not presented as a demonstrated zero-cost case. Deep-chain improvements -remain about 27-fold with one thread and 15–16-fold with five threads. The final -matrix comprises 29 configurations and 580 measured executions, plus warm-ups; +remain about 27-fold with one thread and 15–16-fold with five threads. + +The final matrix comprises 29 configurations and 580 measured executions, plus warm-ups; every pair matched counters and CSV contents. RSS sampling remains too sparse to compare peaks for these short runs. @@ -177,11 +345,15 @@ other RR sections. The scanner harness now sums all three section counts and has passed a bounds/end-position smoke check on answer, authority and additional fixtures with both 32 and zero limits. +#### Recorded artifacts + The local investigation artifacts are under `/tmp/dpp-compression-ablation`: `manifest.json` records the build variants and source/patch/binary hashes; `final-provenance.json` records the final matrix; `final-*` directories contain the raw samples, checksums and summaries; `final-summary.json` collects all 29 -configurations with absolute times, ratios and sampled RSS. Disassembly excerpts and the analysis +configurations with absolute times, ratios and sampled RSS. + +Disassembly excerpts and the analysis are in `/tmp/dpp-compression-disassembly`. The parser-only harness prepared during the investigation was not used for these conclusions; adding a benchmark caller could itself change the inlining decisions being investigated. @@ -191,123 +363,6 @@ could itself change the inlining decisions being investigated. | Rebuilt pre-fix control B0 | `22cb75b59057420f50a593c9d08b226f88d0df88337bc579dbfdf822eb4dffdd` | | Production `9b329b9` | `70c1d17e3a956bd17de4bd788664b26c42eca5660797a6fd3b9bb050daf079b1` | -## Initial measurement method (`400acc5`) - -The end-to-end baseline is commit `7807751`, immediately before configurable -limits and suffix caching. The measured implementation is `400acc5`. Both binaries -use Rust 1.97.1, the `perf` profile and default jemalloc features, on macOS/aarch64. -`perf` disables overflow checks; these measurements -must not be interpreted as measurements of the production `release` profile. - -`dns-compression-benchmark.py` generates deterministic synthetic PCAPs and `.dns` -messages locally. Each configuration gets an excluded warm-up pair followed by -10 measured pairs, alternating which binary runs first. Builds and other test -processes are stopped during measurement. All `DPP_*` environment overrides are -removed for each child process. - -The harness validates packet, query, response, match and timeout counters on every -run, then compares CSV contents. Parallel output is sorted before comparison. -It records binary SHA-256, fixture dimensions, per-run processing time, wall time, -RSS and checksums. The main statistic is the median of the paired -candidate/baseline processing-time ratios; below 1 is faster. The interquartile -range (IQR) describes dispersion, **not** a confidence interval. A single favorable -run is not evidence of an improvement. - -The isolated scanner benchmark uses a fresh message-local decoder per iteration -and a historical uncached walker with the **same** configurable limit. This -separates scanner/cache cost from the end-to-end changes in question parsing and -mandatory validation. The full-application uncached control makes the same -comparison for RR-name skipping while keeping the new question parser (including -its suffix cache) and policy. - -## Reproduction - -Build the baseline and candidate with the same compiler, target and allocator: - -```bash -bench_dir=$(mktemp -d /tmp/dpp-compression.XXXXXX) -mkdir "$bench_dir/baseline" -git archive 7807751 | tar -x -C "$bench_dir/baseline" -cargo build --manifest-path "$bench_dir/baseline/Cargo.toml" \ - --target-dir "$bench_dir/baseline-target" --profile perf --locked --offline -cp "$bench_dir/baseline-target/perf/dpp" "$bench_dir/dpp-before" -cargo build --target-dir "$bench_dir/candidate-target" --profile perf --locked --offline -cp "$bench_dir/candidate-target/perf/dpp" "$bench_dir/dpp-after" - -python3 benches/dns-compression-benchmark.py \ - --baseline "$bench_dir/dpp-before" --candidate "$bench_dir/dpp-after" \ - --outdir "$bench_dir/short" --runs 10 --transactions 1000000 \ - --cases empty,one,two,eight,literal-eight,three-label-eight,chain8-eight \ - --limits 32 --decoders both - -python3 benches/dns-compression-benchmark.py \ - --baseline "$bench_dir/dpp-before" --candidate "$bench_dir/dpp-after" \ - --outdir "$bench_dir/unlimited" --runs 10 --transactions 200000 \ - --cases one,three-label-eight,long-name-eight,chain2-eight --limits 0 --decoders both - -python3 benches/dns-compression-benchmark.py \ - --baseline "$bench_dir/dpp-before" --candidate "$bench_dir/dpp-after" \ - --outdir "$bench_dir/dense" --runs 10 --transactions 200000 \ - --cases distinct-128,suffix-128,chain32-128,flat-4600,deep-4600 \ - --limits 32,0 --decoders both - -python3 benches/dns-compression-benchmark.py \ - --baseline "$bench_dir/dpp-before" --candidate "$bench_dir/dpp-after" \ - --outdir "$bench_dir/staged" --runs 10 --transactions 200000 \ - --cases one,chain32-128,deep-4600 --threads 5 --limits 32,0 --decoders both -``` - -Dense fixtures scale the transaction count by 20 for 128 RRs and by 1,000 for -4,600 RRs. The harness omits configurations whose depth exceeds the requested -nonzero limit, so the deep case runs only with zero. Add `--threads 5` to exercise -the staged pipeline, and use a new output directory for each experiment. - -Build and run the isolated scanner using dependency paths reported by Cargo for -the current compiler, rather than picking arbitrary old `.rlib` files: - -```bash -python3 benches/dns-compression/build-scan.py --output "$bench_dir/dns-compression-scan" -"$bench_dir/dns-compression-scan" "$bench_dir/short" 10 100 > "$bench_dir/scan-short.csv" -"$bench_dir/dns-compression-scan" "$bench_dir/unlimited" 10 100 > "$bench_dir/scan-unlimited.csv" -"$bench_dir/dns-compression-scan" "$bench_dir/dense" 10 100 > "$bench_dir/scan-dense.csv" -``` - -To reproduce the end-to-end control with uncached RR-name skipping, export the -candidate commit and apply the benchmark-only patch in that separate directory: - -```bash -mkdir "$bench_dir/control" -git archive HEAD | tar -x -C "$bench_dir/control" -git -C "$bench_dir/control" apply "$PWD/benches/dns-compression/uncached-control.patch" -cargo build --manifest-path "$bench_dir/control/Cargo.toml" \ - --target-dir "$bench_dir/control-target" --profile perf --locked --offline -python3 benches/dns-compression-benchmark.py \ - --baseline "$bench_dir/control-target/perf/dpp" --candidate "$bench_dir/dpp-after" \ - --baseline-limit 32 --limits 32 --decoders fast \ - --cases one,two,eight,literal-eight,three-label-eight \ - --outdir "$bench_dir/control-results" --runs 10 --transactions 1000000 -``` - -Use an explicit `--baseline-limit 0` when comparing unlimited configurations with -this control. Leave `--baseline-limit` unset for the historical binary, which -predates the option. The patch is a measurement control, not a production mode. - -The scripts use synthetic data only and do not upload anything. Keep raw output -under the selected benchmark directory rather than committing generated PCAPs. -Keep build directories separate: exporting several revisions into one shared -Cargo target directory can leave the top-level executable from the last copy even -when a later build of another copy reports `Fresh`. - -Final correctness checks on `400acc5` passed: 353 workspace tests, formatting, -`git diff --check`, and Clippy with the existing `too_many_arguments`, -`type_complexity` and `large_enum_variant` allowances. CLI smoke checks cover -default rejection at 33 jumps, explicit 33, zero, environment configuration and -CLI precedence, in both decoder modes. Before the final inlining and plain-QNAME -optimizations, a separate randomized comparison of the cached and uncached -validators covered six million calls over 20,000 messages, with cold and warm -caches and limits 0 through 11. The final changes also received an independent -review of cursor handling, error classification and limit equivalence. - ## Initial results (`400acc5`) ### Short messages against `7807751` @@ -330,6 +385,7 @@ The default decoder uses approximately 6–9% less processing time in this set. The optional fast decoder has a **remaining end-to-end regression** on several small synthetic cases, up to 4.3% for eight literal owners. For that case the median times are 0.267963 s before and 0.278825 s after, per million transactions. + This is not dismissed as noise: its paired IQR is wholly above 1. The complete change includes configurable validation, different question parsing and cache context; the control below isolates the adaptive RR scanner more narrowly. @@ -376,6 +432,7 @@ dense cases use 10,000 transactions for 128 RRs and 200 for 4,600 RRs. The depth-32 workload needs about 30% less processing time. The deep unlimited workload improves from median 1.443625 s to 0.052278 s with the default decoder, and from 1.440767 s to 0.053734 s with the fast decoder (about 27 times faster). + The distinct-owner unlimited/default configuration also has an observed +2.9% paired median with IQR above 1. Other flat dense configurations mostly overlap 1. Their 30–40 ms end-to-end duration limits sensitivity to scanner costs; the @@ -408,6 +465,7 @@ Ten alternating pairs per case/limit, one excluded warm-up pair, 100 ms per sample. Times include skipping the question and all RR owners of one DNS message, with a fresh decoder each time; they exclude question formatting, matching and CSV output. The baseline walker receives the same limit as the candidate. + This historical table uses the scanner harness from `c5fe0f7`, which reads ANCOUNT. The current harness sums ANCOUNT, NSCOUNT and ARCOUNT so generated authority/additional fixtures can also be scanned; that extension received a @@ -432,7 +490,9 @@ correctness smoke check, not a new set of timings for this table. The unlimited deep-case medians are 6.966778 ms and 107.433 microseconds per message, approximately a 65-fold scanner improvement. Most shallow cases also -improve. Two small costs remain measurable: approximately 0.4 ns per empty +improve. + +Two small costs remain measurable: approximately 0.4 ns per empty response at limit 32, and 1.5 ns per response with eight three-label owners. The latter is about 2–3% of isolated scanning. Therefore the result is **not** "zero overhead in every case," even though the same-policy end-to-end control diff --git a/benches/protocol-correctness-audit.md b/benches/protocol-correctness-audit.md index 333618e..1b6f091 100644 --- a/benches/protocol-correctness-audit.md +++ b/benches/protocol-correctness-audit.md @@ -5,9 +5,41 @@ Updated 2026-09-05 for implementation commit `8881539` on The final implementation retains the protocol and matcher corrections and restores throughput on the selected 18-million-packet capture. Default mode slightly exceeds the pre-audit version -in the final comparison; monotonic mode is at approximately the same level. The workspace suite -passes **254 tests**, and full CSV output is byte-identical to the correctness-fixed version in -both modes and with both DNS decoders. Peak RSS is higher, approximately 288–301 MiB in these runs. +in the final comparison; monotonic mode is at approximately the same level. + +The workspace suite passes **254 tests**, and full CSV output is byte-identical to the +correctness-fixed version in both modes and with both DNS decoders. Peak RSS is higher, +approximately 288–301 MiB in these runs. + +All results below describe that implementation and the selected capture. + +- [Measurement method](#measurement-method) +- [Current performance](#current-performance) +- [Validation](#validation) +- [Final implementation](#final-implementation) +- [Earlier measurements and discarded experiments](#earlier-measurements-and-discarded-experiments) + +## Measurement method + +### Input and build + +The explicitly selected input was `synthetic/server1-like.pcap`. All compared binaries were built +with Rust 1.97.1 using `cargo build --profile perf`, default jemalloc and no additional RUSTFLAGS. +This profile inherits release's `opt-level=3`, LTO and one codegen unit, and disables overflow +checks for trusted benchmark input; the production release profile keeps overflow checks enabled. + +### Host and run order + +The host exposed 16 CPUs, giving 14 matcher workers in staged mode. Affinity was disabled, CSV +output went to `/dev/null`, and bounded output settings were identical. Runs alternated reference / +final and final / reference order, with an excluded warmup pair per mode and no subsequent outlier +removal. + +### Timing and memory + +Timings include process startup and shutdown. The capture was read from the filesystem +cache, so these results do not measure cold-storage performance. Peak RSS was collected per child +with `wait4`. See [README.md](README.md) for the benchmark scaffold and explicit-input contract. ## Current performance @@ -16,10 +48,12 @@ are the median of paired reference-time / final-time ratios; values above 1 indi The two reference comparisons are separate measurement series, so their absolute timings and RSS must not be combined as if they were one run. -Against the pre-audit version, `3bb45397baff03a6c7043d2e6cff1508f0161b39`: +### Comparison with the pre-audit version + +Reference: `3bb45397baff03a6c7043d2e6cff1508f0161b39`. | Configuration | Pre-audit median (s) | Final median (s) | Paired speed ratio | Median peak RSS, pre-audit / final (MiB) | -|---|---:|---:|---:|---:| +| --- | ---: | ---: | ---: | ---: | | Default | 0.9807 | 0.9677 | 1.0134 | 217.7 / 298.4 | | Monotonic | 0.9580 | 0.9627 | 0.9962 | 221.3 / 287.9 | @@ -27,10 +61,12 @@ Default throughput is about 1.3% higher in this series. The roughly 0.4% monoton within the observed run-to-run variation: paired speed ratios ranged from 0.9687 to 1.0394. These results establish local recovery on this capture, not a throughput or portability guarantee. -Against the correctness-fixed version before optimization, `2ef34a4`: +### Comparison with the correctness-fixed version + +Reference before optimization: `2ef34a4`. | Configuration | Fixed median (s) | Final median (s) | Paired speed ratio | Median peak RSS, fixed / final (MiB) | -|---|---:|---:|---:|---:| +| --- | ---: | ---: | ---: | ---: | | Default | 1.0273 | 0.9758 | 1.0484 | 209.8 / 301.4 | | Monotonic | 0.9980 | 0.9723 | 1.0290 | 219.0 / 288.4 | @@ -38,35 +74,10 @@ The optimizations improve throughput by about 4.8% and 2.9% respectively in this series. Higher occupied pipeline backlog and allocator retention are hypotheses for the RSS increase; their contributions were not isolated. Queue bounds and payload ownership are unchanged. -## Final implementation - -- Routing metadata is **42 bytes on the measured host**. A checked `u16` offset is relative to - the minimum 42-byte Ethernet/IPv4/UDP prefix. This still represents DNS starts above 65535: - a maximum IPv6 frame ends at byte 65589 and must leave at least 12 DNS bytes. IP/UDP boundaries, - QR direction, QCLASS/OPCODE identity and compression validation remain enforced. -- Flow routing collects the existing tuple's `Hash` writes into a 64-byte stack buffer and - invokes SeaHash once over the bytes. Integer encoding, seeds and the complete digest are - unchanged, preserving shard assignment and output order. If the input exceeds the buffer - capacity, its contents are replayed into the original streaming hasher. No additional persistent - flow state is kept. -- Each pending canonical query owns one optional retry-history pointer. Default mode allocates - history when a pending retry arrives; later retries can grow its vector. Replacing a canonical - query with an earlier attempt regroups that identity's unresolved attempts in `O(n log n)`. - Monotonic mode allocates no retry history. Memory on duplicate-heavy captures therefore depends - on unresolved attempts; this capture's low duplicate rate does not bound that workload. -- PCAPNG stdin retains parser-owned framing, one section-local interface table and exact signed - timestamp conversion, including binary/decimal resolution and offsets. Conversion includes - `u128` division per packet; the reusable block buffer grows with received bytes. Regular-file - PCAPNG still uses libpcap, and classic PCAP retains its pure-Rust streaming path. -- Known exported DNS type/status labels remain borrowed strings; numeric fallback allocates for - unknown values. CSV and Parquet share the same canonical formatting contract. - -`build.rs` was reviewed and left unchanged. It requests executable stack size and supplies build -metadata. Optimization levels, LTO and codegen units belong to `Cargo.toml`; no compiler or linker -settings were changed during performance recovery. - ## Validation +### Commands and coverage + `cargo test --workspace --offline --locked` passes 254 tests. Coverage includes 30 capture-reader tests, 13 DNS parser regressions, a 720-permutation matcher check, both pipeline execution models, and CSV/Parquet preservation of unknown codes. Four differential hashing tests compare complete @@ -83,6 +94,8 @@ cargo clippy --all-targets --no-deps --offline --locked -- \ -A clippy::large_enum_variant ``` +### Counters and CSV equivalence + Every final comparison preserved packet/query/response/duplicate/match/timeout counters and average matched RTT. The selected capture contains 18,006,441 packets, 9,013,611 queries, 13,610 duplicates, 8,992,830 responses and matched pairs, and 7,171 timeouts. @@ -93,26 +106,58 @@ Both decoders produce the same output for a given mode. Default and monotonic mo row ordering by their existing timeout-emission contracts. | Mode | SHA-256, both reference and final | -|---|---| +| --- | --- | | Default | `02db37f51a60a84000b457c47339fb87bd725071c4e08899f3e8b90b430f9fc9` | | Monotonic | `08f435bc3aa241615937e3c08b4c089570d5e3dbe9fb245a46c8e9bd3d680bc0` | -## Measurement method +## Final implementation -The explicitly selected input was `synthetic/server1-like.pcap`. All compared binaries were built -with Rust 1.97.1 using `cargo build --profile perf`, default jemalloc and no additional RUSTFLAGS. -This profile inherits release's `opt-level=3`, LTO and one codegen unit, and disables overflow -checks for trusted benchmark input; the production release profile keeps overflow checks enabled. +### Routing metadata -The host exposed 16 CPUs, giving 14 matcher workers in staged mode. Affinity was disabled, CSV -output went to `/dev/null`, and bounded output settings were identical. Runs alternated reference / -final and final / reference order, with an excluded warmup pair per mode and no subsequent outlier -removal. Timings include process startup and shutdown. The capture was read from the filesystem -cache, so these results do not measure cold-storage performance. Peak RSS was collected per child -with `wait4`. See [README.md](README.md) for the benchmark scaffold and explicit-input contract. +Routing metadata is **42 bytes on the measured host**. A checked `u16` offset is relative to +the minimum 42-byte Ethernet/IPv4/UDP prefix. This still represents DNS starts above 65535: +a maximum IPv6 frame ends at byte 65589 and must leave at least 12 DNS bytes. IP/UDP boundaries, +QR direction, QCLASS/OPCODE identity and compression validation remain enforced. + +### Flow hashing + +Flow routing collects the existing tuple's `Hash` writes into a 64-byte stack buffer and +invokes SeaHash once over the bytes. Integer encoding, seeds and the complete digest are +unchanged, preserving shard assignment and output order. If the input exceeds the buffer +capacity, its contents are replayed into the original streaming hasher. No additional persistent +flow state is kept. + +### Retry history + +Each pending canonical query owns one optional retry-history pointer. Default mode allocates +history when a pending retry arrives; later retries can grow its vector. Replacing a canonical +query with an earlier attempt regroups that identity's unresolved attempts in `O(n log n)`. + +Monotonic mode allocates no retry history. Memory on duplicate-heavy captures therefore depends +on unresolved attempts; this capture's low duplicate rate does not bound that workload. + +### Capture readers + +PCAPNG stdin retains parser-owned framing, one section-local interface table and exact signed +timestamp conversion, including binary/decimal resolution and offsets. Conversion includes +`u128` division per packet; the reusable block buffer grows with received bytes. Regular-file +PCAPNG still uses libpcap, and classic PCAP retains its pure-Rust streaming path. + +### Exported labels + +Known exported DNS type/status labels remain borrowed strings; numeric fallback allocates for +unknown values. CSV and Parquet share the same canonical formatting contract. + +### Build configuration + +`build.rs` was reviewed and left unchanged. It requests executable stack size and supplies build +metadata. Optimization levels, LTO and codegen units belong to `Cargo.toml`; no compiler or linker +settings were changed during performance recovery. ## Earlier measurements and discarded experiments +### Intermediate performance and discarded changes + Before optimization, the correctness fixes showed about 5.6% lower default throughput and 4.8% lower monotonic throughput in the initial comparison. Those figures describe the intermediate `2ef34a4` implementation, not the current version. Compact relative offsets and buffered flow @@ -123,6 +168,8 @@ formatter inlining, a fused borrowed matcher lookup, or a new packed hash input. contains none of those experimental changes. CPU sampling identified packet routing and its streaming hash calls as the limiting stage on this capture. +### Earlier PCAPNG-reader check + An earlier PCAPNG-reader check compared the pre-audit reader with the correctness-fixed reader using the first 1,000,000 packets re-encoded as little-endian Enhanced Packet Blocks. Three pairs measured 0.0780 / 0.0787 seconds, with identical processing counters. The roughly 80 ms runs were diff --git a/docs/allocator-guide.md b/docs/allocator-guide.md index 1255c74..52a314b 100644 --- a/docs/allocator-guide.md +++ b/docs/allocator-guide.md @@ -1,44 +1,31 @@ # Allocator Guide -This document explains how DPP selects its global allocator and how to build alternative allocator -variants. +DPP selects one global allocator at build time through Cargo features. The choice applies to the +whole process; there is no runtime flag or environment override. The default is `tikv-jemallocator`. -## Scope +## Choose an allocator -Allocator choice in DPP is a build-time concern. +Exactly one allocator feature must be enabled. Invalid feature combinations fail the build +explicitly. -- It applies to the whole process. -- It is selected through Cargo features. -- It is not exposed as a runtime flag or environment override. +| Cargo feature | Allocator | Availability | +| --- | --- | --- | +| `allocator-jemalloc` | `tikv-jemallocator` | Default build. | +| `allocator-mimalloc` | `mimalloc` | Alternative build. | +| `allocator-system` | `std::alloc::System` | Alternative build. | +| `allocator-tcmalloc` | `tcmalloc-better` | Linux `x86_64` and Linux `aarch64` only. | -The default build uses `tikv-jemallocator`. - -## Supported Allocators - -Exactly one allocator feature must be enabled. - -- `allocator-jemalloc` - Default build. Uses `tikv-jemallocator`. -- `allocator-mimalloc` - Alternative build. Uses `mimalloc`. -- `allocator-system` - Alternative build. Uses `std::alloc::System`. -- `allocator-tcmalloc` - Alternative build for Linux `x86_64` and Linux `aarch64` only. Uses `tcmalloc-better`. - -Invalid feature combinations fail the build explicitly. - -## Build Commands +## Build a variant Requires Rust 1.98.1 or newer; this repository is pinned by `rust-toolchain.toml`. -Default allocator build: +Default allocator: ```bash cargo build --release ``` -Alternative allocator builds: +Alternative allocators: ```bash cargo build --release --no-default-features --features allocator-system @@ -48,47 +35,33 @@ cargo build --release --no-default-features --features allocator-mimalloc cargo build --release --no-default-features --features allocator-tcmalloc ``` -If you want host-specific code generation during allocator comparisons, keep the same `RUSTFLAGS` -for every build variant. Example: - -```bash -RUSTFLAGS='-C target-cpu=native' cargo build --release -RUSTFLAGS='-C target-cpu=native' cargo build --release --no-default-features --features allocator-mimalloc -``` - -Hypothesis: `target-cpu=native` is the right setting for same-host allocator comparisons. If you are -building portable release binaries, keep the same portable compiler settings across all variants. - -## Runtime Visibility +## Confirm the active allocator -DPP logs the active allocator during startup. Example log line: +DPP logs the active allocator during startup, making benchmark logs and operational reports easier +to interpret. For example: ```text Allocator: tikv-jemallocator ``` -This makes allocator-specific benchmark logs and operational reports easier to interpret. - -## Benchmarking - -Allocator changes must be evaluated on the same representative capture, format, and CPU budget. - -Use the canonical benchmark protocol here: +## Compare allocators -- [benches/allocator-benchmarking.md](../benches/allocator-benchmarking.md) +Use the same representative capture, output format, and CPU budget for every variant. Follow the +[allocator benchmark protocol](../benches/allocator-benchmarking.md) for building separate binaries, +running the harness, comparing metrics, and meeting the correctness bar before accepting a change. -That protocol covers: +Keep the same `RUSTFLAGS` across variants. For host-specific code generation: -- how to build one binary per allocator; -- how to run the benchmark harness fairly; -- what metrics to compare; -- what correctness bar must be met before accepting a change. - -## Design Reference +```bash +RUSTFLAGS='-C target-cpu=native' cargo build --release +RUSTFLAGS='-C target-cpu=native' cargo build --release --no-default-features --features allocator-mimalloc +``` -The accepted architecture contract for allocator selection lives here: +**Hypothesis:** `target-cpu=native` is the right setting for same-host allocator comparisons. For +portable release binaries, keep the same portable compiler settings across all variants. -- [docs/rfc/0003-allocator-selection.md](rfc/0003-allocator-selection.md) +## Design reference -Use this guide for day-to-day builds and comparisons. Use the RFC when reviewing ownership -boundaries or changing the contract. +Use this guide for day-to-day builds and comparisons. The accepted contract for allocator selection +is [RFC 0003](rfc/0003-allocator-selection.md); consult it when reviewing ownership boundaries or +changing the contract. diff --git a/docs/architecture.md b/docs/architecture.md index d6a4b73..b41d226 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -1,458 +1,494 @@ # DPP Architecture -## Scope +DPP Community Edition processes offline PCAP captures into DNS transaction records in CSV or +Parquet. Its only supported matching model is forward-only: the matcher owns all in-flight +query/response state. For a fixed input and configuration, matching decisions and record order +must be deterministic, regardless of worker scheduling. -This document is the canonical architecture reference for the DPP Community Edition. It records the -processing model, ownership boundaries, and matcher invariants that must remain true across bug -fixes, performance work, and future refactors. +This is the canonical reference for processing, ownership, and invariants. The first two sections +give the system overview; later sections describe the contracts and edge cases. -## Processing Model +- [Processing model](#processing-model) +- [Ownership map](#ownership-map) +- [Matcher contract](#matcher-contract) +- [Output, reporting, and shutdown](#output-reporting-and-shutdown) +- [Capture input and packet routing](#capture-input-and-packet-routing) +- [IPv4 fragmentation](#ipv4-fragmentation) +- [DNS validation](#dns-validation) +- [IP pseudonymization](#ip-pseudonymization) +- [Related references](#related-references) -DPP Community Edition has a single supported matching model: +## Processing model -- forward-only matching with deterministic ordering by `(timestamp_micros, packet_ordinal)` -- offline processing from PCAP file input or EOF-terminated stdin PCAP streams into CSV or Parquet output -- optional final JSON run summary through the canonical report-format contract; stdout export mode - is CSV-only, rejects JSON reports, suppresses text reports, treats downstream stdout pipe - closure as graceful CLI termination, suppresses broken-pipe log noise if stderr itself is closed, - and signal-driven shutdown drops any still-buffered output tail before final writer teardown -- optional `--monotonic-capture` mode for globally ordered captures +Input is a PCAP file or an EOF-terminated stdin capture stream. DPP supports DNS over UDP port 53 +on Ethernet, including VLAN and QinQ. File output supports CSV and Parquet; stdout export is CSV-only. +An optional final run summary uses the canonical text/JSON report-format contract. -The matcher is the authoritative owner of in-flight query/response state. Parsing may run in -parallel, but matching decisions must be applied in a deterministic order. +The pipeline has four logical stages: + +| Stage | Responsibility | +| --- | --- | +| Ingest and order | Read capture packets, reject non-Ethernet linktypes, and order packets within each batch. | +| Extract DNS | Parse Ethernet/IP/UDP payloads into query/response candidates. | +| Match | Pair queries and responses, apply timeout windows, and deduplicate retries. | +| Serialize | Stream finalized matched or timed-out records to asynchronous writers. | + +In staged execution, these responsibilities are distributed as follows: ```mermaid flowchart LR - PCAP(["PCAP"]):::input + Input["PCAP file
or stdin"]:::input --> Read["Read bounded
batches"]:::process + Read --> Route["Route
flows"]:::process + Route --> Workers["Decode DNS
and match"]:::process + Workers --> Aggregate["Merge
in order"]:::process + Aggregate --> Write["Write CSV
or Parquet"]:::output + + classDef input fill:#f3e8fd,stroke:#9334e6,color:#1a1a1a + classDef process fill:#e6f4ea,stroke:#34a853,color:#1a1a1a + classDef output fill:#fce8e6,stroke:#ea4335,color:#1a1a1a +``` - subgraph Ingest [" Ingest "] - Parse["Parse\npackets"]:::proc - end +### Execution and parallelism - subgraph Match [" Match "] - Route["Route\nflows"]:::proc - Shards["Shard\nworkers"]:::proc - Agg["Aggregate"]:::proc - end +DPP derives its CPU budget from available CPUs, capped by `--threads` or `DPP_THREADS` when +specified. The execution model is selected automatically from that budget: + +| Model | When used | How work runs | +| --- | --- | --- | +| Staged | Enough CPUs for service roles and shard workers | Reserve two non-worker roles for parsing/routing and aggregation; use the remaining budget for matcher workers. | +| Phase-parallel | Low-core budgets | Process batches in sequence, parallelize shard work within each batch, then collect results in shard order. | + +The key difference is where work overlaps. These two lanes are illustrative; actual shard and +worker counts depend on the CPU budget. Each staged worker owns a range of shards. - subgraph Output [" Output "] - Write["Write\nCSV / Parquet"]:::out +```mermaid +flowchart TB + subgraph Phase["Phase-parallel: one batch at a time"] + direction LR + Batch["Route batch N"] --> Shard0["Shard 0"] + Batch --> Shard1["Shard 1"] + Shard0 --> Emit["Emit in
shard order"] + Shard1 --> Emit + Emit --> Next["Process
batch N+1"] + end + subgraph Staged["Staged: stages overlap across batches"] + direction LR + Router["Route numbered
batches"] -->|"bounded queue"| Worker0["Worker 0"] + Router -->|"bounded queue"| Worker1["Worker 1"] + Worker0 --> Aggregate["Aggregate in
batch order"] + Worker1 --> Aggregate end + Phase ~~~ Staged +``` + +See [RFC 0006](rfc/0006-adaptive-pipeline.md) for the selection policy and thread roles. + +Parallel ingestion and DNS extraction are allowed. Matching may run in parallel only across +independent shards, and **each shard exclusively owns its matcher state**. Query and response +packets in the same client/resolver flow must reach the same worker before full DNS decode. + +Each shard processes candidates in deterministic order, and the aggregator +merges finalized records deterministically before handing them to writers. Scheduler interleaving +and container iteration order must never decide a tie. - Summary(["Summary\nJSON / text"]):::orch +## Ownership map - PCAP --> Parse --> Route --> Shards --> Agg --> Write --> Summary +Each boundary has one authoritative owner. In particular, configuration, exported record schema, +and writer lifecycle must not acquire a second source of truth. - classDef input fill:#f3e8fd,stroke:#9334e6,color:#1a1a1a - classDef proc fill:#e6f4ea,stroke:#34a853,color:#1a1a1a - classDef out fill:#fce8e6,stroke:#ea4335,color:#1a1a1a - classDef orch fill:#fef7e0,stroke:#f9ab00,color:#1a1a1a +This map shows selected responsibilities: solid arrows mean delegation, dotted arrows mean use +of a canonical contract. It is not a call-order or exhaustive dependency diagram. - style Ingest fill:none,stroke:#34a853,stroke-width:1px,stroke-dasharray:4 - style Match fill:none,stroke:#34a853,stroke-width:1px,stroke-dasharray:4 - style Output fill:none,stroke:#ea4335,stroke-width:1px,stroke-dasharray:4 +```mermaid +flowchart TB + Main["main.rs"] --> CLI["cli.rs
resolve arguments"] + Main --> Runtime["runtime.rs
bootstrap helpers"] + Main --> App["app.rs
coordinate the run"] + App --> Runtime + App --> Capture["packet_parser.rs
capture intake"] + App --> DNS["dns_processor.rs
packet processing"] + App --> Output["output.rs
writer lifecycle"] + Output --> Writers["CSV / Parquet
writers"] + Config["config.rs
runtime policy"] -.-> CLI + Config -.-> App + Record["record.rs
DnsRecord schema"] -.-> DNS + Record -.-> Writers ``` -## Pipeline - -The processing pipeline has four logical stages: - -1. Packet ingestion and ordering - Reads packets from an offline PCAP file, rejects captures or packets declared with non-Ethernet - linktypes, and establishes deterministic packet order inside each batch. - -2. DNS extraction - Parses Ethernet/IP/UDP payloads and builds DNS query/response candidates. - -3. Query-response matching - Holds in-flight query/response state, applies timeout windows, deduplicates retries, and emits - exactly one terminal outcome per canonical query. - -4. Record serialization - Streams matched or timed-out DNS records to CSV or Parquet writers. - -In the current multi-threaded implementation, those stages are realized as a bounded batch-prefetch -reader, a lightweight routing stage, shard-owned workers, a deterministic aggregator, and -asynchronous writers. The routing stage performs only cheap L3/L4 extraction, computes a canonical -client/resolver flow key, and hands owned packet batches to the correct worker before full DNS -decode begins. DPP derives its execution budget from available CPUs, capped by `--threads` or -`DPP_THREADS` when specified. On low-core budgets, DPP falls back to the simpler phase-parallel -pipeline so it does not spend too much of the machine on staged-pipeline service roles. In staged -mode, the runtime reserves two non-worker service threads for routing/aggregation and parser work, -and uses the remaining CPU budget for shard workers. Routed DNS packets also carry compact UDP/DNS -metadata into shard workers so the worker path can reuse the first L3/L4 parse instead of -repeating it for full DNS question decoding. - -The staged pipeline waits for every successfully started worker before returning a startup -error. It releases the input and result channels before joining those workers, so cleanup can -finish without an aggregator. During normal teardown it joins every worker -even if an earlier worker returns an error or panics, preserving the first worker error after -any parser-stage error. - -Ethernet decoding also produces one canonical VLAN context: the ordered stack of tag TPIDs and -12-bit VLAN IDs. PCP and DEI are QoS metadata and do not distinguish transactions. This context is -shared by fragment keys and matcher identities, keeping overlapping IP endpoints in different -VLANs separate. Routing still uses the coarser client/resolver endpoint tuple, so different VLANs -with the same endpoints can share a worker without sharing matcher state. Untagged packets allocate -no tag storage; tagged stacks use shared immutable storage and remain internal to processing. - -Flow routing buffers the existing tuple's `Hash` writes on the stack and runs SeaHash once over -the resulting bytes. The integer encoding and digest remain identical to the streaming hasher, -so shard assignment and output order are preserved. If a future tuple encoding exceeds the -buffer, routing replays its bytes into the streaming hasher; this adds no persistent flow state -or second encoding of canonical addresses and ports. - -## Core Components - -- `src/packet_parser.rs` - Reads packets from offline captures and exposes them to the rest of the pipeline. Classic PCAP - files use a pure-Rust streaming reader; regular-file PCAPNG and other non-classic formats - currently fall back to libpcap. EOF-terminated stdin capture streams are also supported through - parser-owned stream-native readers: classic PCAP stdin uses the same pure-Rust fast path family - as classic file input, and PCAPNG stdin owns block framing and one section-local interface table. - PCAPNG uses the dependency's stateless structural validation, while the parser converts raw - high/low timestamp words using the interface's full binary/decimal resolution and signed offset. - Conversion scales the complete counter before rounding to microseconds and saturates only the - final signed timestamp. It never invokes the dependency's stateful timestamp conversion. - Buffers grow with received block bytes rather than untrusted declared lengths, and stdin PCAPNG - blocks over 16 MiB are rejected before their body is read. Stdin probing - needs no temp-file or second ingest owner. Unsupported stdin stream magic is rejected explicitly. - Signal-driven shutdown can interrupt a blocked stdin read without waiting for the producer to - close the stream; capture reading remains parser-owned. Packets already read into a nonempty - batch are handed to the processing pipeline before intake stops. The signal shutdown policy can - still discard records buffered by the output writer. - The pure-Rust classic-PCAP reader still relies on the upstream `pcap-file` `3.0.0-rc1` release - candidate until a stable line with the required functionality is available. - The parser can also enforce globally monotonic capture timestamps for the optional batched - timeout-eviction path; when that contract is enabled, a timestamp regression becomes a hard - processing error instead of a post-run warning. - -- `src/config.rs` - Canonical runtime configuration and processing constants. Output format, report format, batch - sizing, timeout windows, execution-model thresholds, and writer thresholds must have a single - source of truth here. - -- `src/allocator.rs` - Canonical global allocator boundary. Build-time allocator selection is owned here through - mutually exclusive Cargo features and must not leak into CLI, environment parsing, or subsystem - configuration. - -- `src/cli.rs` - Canonical CLI and environment-resolution boundary. Command-line parsing, environment override - precedence, and output-path validation must stay centralized here instead of leaking into runtime - orchestration or module-local helpers. - -- `src/record.rs` - Canonical exported `DnsRecord` contract. Internal matcher discriminators and shard metadata must - never leak into this type. Timeout records currently encode "no response observed in the match - window" by leaving response fields absent. The canonical timeout signal is an absent - `response_timestamp`; `response_code` is absent because no DNS response exists. - -- `src/output.rs` - Writer lifecycle boundary. Owns output control messages and the writer-thread factory. The - handoff channel from processing to output carries batched `DnsRecord` messages and is bounded by - configuration. User-facing capacity remains a record backlog; internally it is rounded up to - batch-message slots. `src/config.rs` owns both the maximum records per output message and the - default queued-record backlog, so the - batched handoff must not become unbounded ambient state. Broken-pipe closure on stdout is treated - as a downstream-close outcome; regular file-write failures remain fatal. - -- `src/monitor_memory.rs` - Optional RSS tracking helper with an explicit stop/join lifecycle. The monitor's 100 ms sampling - wait is interruptible by stop, so join need not wait for the next sample. An in-progress RSS - refresh must still finish before the monitor thread exits. - -- `src/dns_processor.rs` and `src/dns_processor/*` - The DNS processor facade owns packet-to-matcher orchestration and delegates to focused - submodules: `anonymizer.rs` for key loading and deterministic pseudonymization, `parser.rs` for - packet-to-DNS extraction and canonical flow routing metadata, `matcher.rs` for in-flight state - and pairing, `pipeline.rs` for shard-parallel orchestration, and `types.rs` for internal matcher - types. Signal-driven shutdown semantics for pending unmatched queries are also owned here: - interrupted runs must not synthesize timeout tail records from incomplete matcher state. The - staged pipeline may reuse parser-produced UDP/DNS metadata between routing and - shard-local DNS decode, but that reuse must stay within the same ownership boundary so packet - parsing does not gain a second source of truth for IP/port extraction. That metadata owns the - exact DNS byte range validated against IPv4 Total Length or IPv6 Payload Length and then UDP - Length; capture padding and trailing IP payload cannot extend the DNS slice. For an opted-in - first IPv4 response fragment, the DNS slice ends at that fragment's Total Length, and the UDP - Length is checked against the declared datagram size. A compact relative - offset preserves DNS starts above 65535 without widening every packet's routing metadata: the - offset is measured from the minimum Ethernet/IPv4/UDP header length, and construction checks - that it fits. DNS QR determines - direction and the canonical client/resolver flow, including exchanges with UDP port 53 on both - ends. IPv6 Hop-by-Hop, Routing, Destination Options, AH and atomic Fragment headers are traversed - within the declared payload boundary. Non-atomic IPv6 fragments and, by default, fragmented - IPv4 datagrams are skipped. With - `--allow-fragments`, only a first IPv4 response fragment with a complete DNS header and question - can enter the matcher as an inferred response; later fragments and fragmented queries remain - skipped. Its response code is present only when there are no additional records or all declared - DNS records fit in that prefix. - `--full-fragments` additionally enables IPv4 reassembly across packet batches and implies - `--allow-fragments`. The DNS processor owns the reassembly state: a complete datagram enters the - existing UDP/DNS validation path only after the first and last fragments establish its bounds - and every byte is present. Incomplete responses may use the first-fragment inference on capacity - eviction or end of input; with monotonic capture, entries can also expire after the match - timeout. Without monotonic capture, timestamp gaps between fragments of the same datagram - do not expire it; capacity limits and end of input still bound its lifetime. - Fragmented queries require a complete datagram. A complete datagram - uses the final fragment's (`MF=0`) capture timestamp, while inferred responses use the first - fragment's timestamp. While any datagram is unresolved, routing retains its ready packet - batches so a delayed reconstruction cannot arrive after a retry or response that it should - precede. Released packets are sorted together by timestamp and capture ordinal before matching. - The eviction watermark cannot advance beyond either unresolved fragments or retained ready - packets. The additional ready backlog is bounded between batches to 65,536 packets or 64 MiB - of packet payloads; exceeding either bound resolves pending datagrams through the existing - capacity fallback before releasing the backlog. The current input batch and fallback packets - can temporarily exceed that retained-state bound. This buffering is owned by the routing stage - and contains no query/response pairing state. - The reassembler retains a bounded history of completed datagrams and - compares the fragment key and UDP payload bytes before suppressing repeated complete datagrams - or prefix fallback from fragments matching a recent completion. A reused IPv4 ID with different - content can still produce a new datagram, even when its first fragment is identical. An incomplete - new datagram whose observed fragments match a recent completion is indistinguishable from a - duplicate and may be suppressed; the same holds for a fully identical new datagram. Reassembly - state must remain bounded in memory, and IPv6 fragments remain unsupported. A fragmented UDP datagram must - have an IP payload length equal to its UDP Length; mismatched fragment sets are rejected. - When the first fragment arrives after tails, its UDP Length also bounds every retained tail; - a nonfinal fragment cannot reach or exceed that boundary, regardless of arrival order. - A first non-DNS UDP fragment removes any earlier tails for its datagram key. A separate bounded - history drops later non-DNS tails without evicting the completed-DNS history; a new DNS first - fragment with a reused IPv4 ID removes that non-DNS mark. The canonical VLAN context is part of - the fragment key so traffic from different tagged segments is not assembled together, while - priority or drop-eligibility changes do not split a datagram. A new DNS tail - that arrives before its first fragment while the old non-DNS mark is active is indistinguishable - from an old non-DNS tail and may be dropped. - The optional runtime flag - `--dns-wire-fast-path` may enable a custom question-only wire fast path, but `hickory` remains - the semantic fallback for rare DNS messages that the fast path does not accept. Compression - pointers must target prior, nonoverlapping names; enabling the fast path must not weaken that - validation. `--max-dns-compression-jumps` (environment: `DPP_MAX_DNS_COMPRESSION_JUMPS`) - limits each decoded name to 32 pointer transitions by default; `0` disables this resource - limit. The limit applies to questions, scanned RR owners and TSIG algorithm names in both - decoder modes, including partial responses and fast-path fallback. A cached suffix contributes - its full pointer depth to this limit. Even when unlimited, strictly backward, nonoverlapping - segments prevent cycles. This is a resource policy, not a claim that longer chains violate DNS. - One immutable-message name decoder caches fully validated pointer targets and their encoded - segment end, expanded wire length, pointer depth and first label. Every cache hit rechecks the - segment end against the current name's start so a previously valid target cannot authorize an - overlap. Only referenced suffixes are retained; ordinary flat record lists do not grow the cache - per record. RR/TSIG name skipping validates short walks directly (up to eight jumps and 64 - literal bytes per referenced segment), switching to the shared cached walker when either - threshold or the configured jump limit would be exceeded. These thresholds choose a cheaper - validation strategy; they never relax the limit or accept an otherwise invalid name. - Small caches and traversal stacks use inline storage; larger ones allocate within - the message and are bounded by the 14-bit pointer address space. Label materialization skips - pointer-only segments. Compressed questions are passed to Hickory already expanded, avoiding - a second recursive traversal without replacing Hickory's question semantics. Literal-root - validation for OPT remains distinct from a compressed name that expands to root. - The optional fast path keeps a single-pass formatter for uncompressed questions; seeing a - pointer restarts validation at the original name offset through the shared cached decoder. - For ordinary QUERY messages (OPCODE 0), more than one question is rejected under - RFC 9619. A complete response cannot claim answer or authority records that are absent from its DNS - payload, whether or not it has additional records. TSIG status extraction consumes and - bounds-checks the declared Other Data, including the six-byte server time in BADTIME responses. - Both paths accept - decompressed wire QNAMEs up to the RFC 1035 limit of 255 octets, including label-length octets and - the terminating root octet. A valid name can expand to 1003 bytes in escaped presentation form - and must remain distinct through matching and export. If any QNAME exceeds the wire limit, the - parser rejects the whole DNS message before the matcher and increments - `oversized_qname_message_count` once. This counter's unit is rejected DNS messages, not questions; - partial questions from such a message must never reach matcher state or output. The optional - `--monotonic-capture` contract enables batched timeout eviction inside shard-local matcher state, - but only under a strict globally monotonic timestamp assumption. When that contract is active, - the eviction watermark comes from the routed batch maximum timestamp, not from each shard's local - maximum, so sparse shards still retire stale state against the global batch frontier. The - Community Edition scope is currently limited to DNS over UDP port 53. - -- `src/csv_writer.rs` and `src/parquet_writer.rs` - Consume finalized `DnsRecord` values and write them asynchronously. Writers must not become a - second source of truth for exported record schema or pipeline policy. - -- `src/app.rs` - Top-level orchestration layer. Owns the ordered run sequence, process reporting, and shutdown - coordination without taking ownership of canonical configuration or writer internals. Capture - read failures drain and join complete accepted batches, skip unmatched-query finalization, flush - conclusive records as valid partial output, and only then return the processing error. The final - run summary is also where aggregate matching-quality metrics such as timeout ratio and average - matched RTT are derived from authoritative processing counters. Parser rejections caused by an - oversized decompressed QNAME are exposed as - `metrics.dns_messages_rejected_oversized_qname` without changing their DNS-message unit. - -- `src/error.rs` - Canonical top-level error taxonomy for the application, runtime bootstrap, and output lifecycle. - Lower-level hot-path modules may still use focused local error types or `anyhow`, but top-level - orchestration boundaries must map failures into structured categories here. - -- `src/runtime.rs` - Bootstrap and host-runtime boundary. Logger setup, build/system reporting, optional memory - monitoring, signal handling, and Rayon pool creation live here so CLI parsing and app - orchestration do not carry side-effectful runtime bootstrap responsibilities directly. The logger - remains human-readable and suppresses broken-pipe noise when stderr itself is a closed pipe; - machine-readable final JSON reporting is selected through the canonical CLI/config contract and - emitted from the top-level app orchestration layer. - -- `src/main.rs` - Thin entrypoint that delegates to `src/cli.rs`, `src/runtime.rs`, and `src/app.rs`, then returns - the top-level process outcome. - -- `docs/rfc/` - Canonical architecture decision records. `README.md` is the directory index. Current RFCs - cover ownership boundaries (0001), CLI/runtime split (0002), allocator selection (0003), - forward-only matcher determinism (0004), dual-path PCAP parsing (0005), and adaptive - pipeline execution (0006), and packet-storage allocation experiments (0007). - -- `docs/encapsulation-playbook.md` - Operational and engineering guidance for captures with unsupported MPLS or other outer - encapsulation layers before the IP header. Ethernet VLAN and QinQ are decoded natively. - -- `benches/README.md` - Documents the benchmark contract, safety expectations, and result layout for repeatable runs. - -- `benches/benchmark.sh` - Repeatable benchmark scaffold for throughput and shutdown-tail measurements on caller-provided - PCAP inputs. - -- `benches/allocator-benchmarking.md` - Canonical build matrix and measurement protocol for comparing allocator variants. +| Owner | Responsibility | +| --- | --- | +| [`main.rs`](../src/main.rs) | Thin entrypoint: compose CLI, runtime, and app; return the process outcome. | +| [`cli.rs`](../src/cli.rs) | CLI parsing, environment precedence, and output-path validation. | +| [`config.rs`](../src/config.rs) | Runtime policy: output/report formats, batch sizes, timeouts, execution thresholds, and writer thresholds. | +| [`runtime.rs`](../src/runtime.rs) | Logger setup, build/system reporting, signal handling, optional memory monitoring, and Rayon pool creation. | +| [`app.rs`](../src/app.rs) | Ordered run orchestration, final reporting, and shutdown coordination. | +| [`allocator.rs`](../src/allocator.rs) | Global allocator selection through mutually exclusive build-time Cargo features; no CLI or runtime allocator policy. | +| [`packet_parser.rs`](../src/packet_parser.rs) | Capture readers, packet batches, capture timestamps, and interruptible stdin intake. | +| [`dns_processor.rs`](../src/dns_processor.rs) | Packet-to-matcher orchestration and the interrupted-run policy for pending queries. | +| [`record.rs`](../src/record.rs) | Exported `DnsRecord` schema; internal matcher discriminators and sequencing metadata stay out. | +| [`output.rs`](../src/output.rs) | Writer factory, lifecycle, and output control messages. | +| [`csv_writer.rs`](../src/csv_writer.rs), [`parquet_writer.rs`](../src/parquet_writer.rs) | Asynchronous serialization of finalized records; no ownership of schema or pipeline policy. | +| [`monitor_memory.rs`](../src/monitor_memory.rs) | Optional RSS sampling with explicit stop/join. | +| [`error.rs`](../src/error.rs) | Structured top-level application, bootstrap, and output errors. Hot-path modules may use local errors or `anyhow`, mapped at orchestration boundaries. | + +The DNS processor delegates within its boundary: + +| Submodule | Responsibility | +| --- | --- | +| [`parser.rs`](../src/dns_processor/parser.rs) | Packet-to-DNS extraction and canonical flow-routing metadata. | +| [`name_decoder.rs`](../src/dns_processor/name_decoder.rs) | Shared DNS name validation and compression traversal. | +| [`reassembly.rs`](../src/dns_processor/reassembly.rs) | Bounded IPv4 fragment reassembly. | +| [`matcher.rs`](../src/dns_processor/matcher.rs) | Authoritative in-flight query/response state and pairing. | +| [`pipeline.rs`](../src/dns_processor/pipeline.rs) | Shard-parallel routing, processing, and aggregation. | +| [`anonymizer.rs`](../src/dns_processor/anonymizer.rs) | Key loading, derivation, and deterministic IP pseudonymization. | +| [`types.rs`](../src/dns_processor/types.rs) | Internal matcher types. | + +## Matcher contract + +### Identity and routing + +Every query or response candidate has a stable in-flight identity until matched or discarded. +Identity includes: + +- original observed client IP and port, plus resolver IP; +- DNS ID, observed presentation-form QNAME, QTYPE, QCLASS, and OPCODE; +- the full canonical VLAN tag stack. + +Routing uses the coarser client/resolver flow tuple. Different VLANs with the same endpoints can +share a worker, but cannot share matcher state. Resolver identity, QCLASS, OPCODE, and VLAN context +remain internal and do not extend `DnsRecord`. + +The diagram separates **where a packet goes** from **which transaction it belongs to**. VLAN +context contributes to matcher identity, not the flow hash. ```mermaid -flowchart TD - M(["main.rs"]):::entry - - subgraph Entry [" Entry & Bootstrap "] - CLI["cli"]:::entry - CFG["config"]:::entry - RT["runtime"]:::entry - ALLOC["allocator"]:::entry - end +flowchart TB + Packet["Packet metadata"] --> Flow["Canonical flow
observed client IP + port, resolver IP
oriented by DNS QR"] + Packet --> VLAN["Canonical VLAN stack
ordered TPID + VID"] + Flow -->|"hash"| Shard["Shard assignment"] + Shard --> Decode["Full DNS decode"] + Decode --> Identity["Matcher identity
flow + DNS ID + QNAME + QTYPE
QCLASS + OPCODE + VLAN"] + VLAN --> Identity + Identity --> State["Shard-owned
matcher state"] + State -->|"finalize"| Record["DnsRecord
pseudonymize client IP once"] +``` - APP["app"]:::orch +Client-IP pseudonymization happens exactly once, when the matcher constructs a finalized record. +Routing and matching always use the original addresses. - subgraph Core [" Processing "] - DP["dns_processor"]:::proc - PIPE["pipeline"]:::proc - MATCH["matcher"]:::proc - PARSE["parser"]:::proc - ANON["anonymizer"]:::proc - end +**QNAME identity is byte-preserving and case-sensitive.** This is a deliberate Community Edition +trade-off for offline caching-resolver workloads. RFC 4343 defines ASCII label comparison as +case-insensitive, and valid responses may differ in 0x20 casing, including through compression. +Such pairs may fail to match in Community Edition; byte-preserving identity is not a protocol +guarantee. - REC["record"]:::data +### Retries and terminal outcomes - subgraph Out [" Output "] - OUTPUT["output"]:::out - CSV["csv_writer"]:::out - PQ["parquet_writer"]:::out - end +Repeated pending queries with the same identity inside the match-timeout window (`1200ms` by +default) belong to the earliest canonical query. Later duplicates are counted separately and must +not create extra matched or timeout records. Duplicate responses remain distinguishable until +matched or discarded. + +In default mode, the pending canonical query owns retry timestamps needed when an earlier query +arrives across a batch boundary. The matcher regroups unresolved attempts into earliest-first +timeout windows; chains of timestamp regressions must not absorb attempts outside the final +canonical window. Finalized transactions are never reopened. + +Retry history adds one optional pointer per pending query and storage for default-mode retries. +Only earlier-canonical replacements regroup attempts. Monotonic mode allocates no retry history. + +On normal completion, each canonical query has one terminal outcome: matched once or emitted once +as a timeout. Interrupted and failed runs have a separate [shutdown policy](#output-reporting-and-shutdown) +and must not invent timeout records from incomplete matcher state. + +### Ordering and timeout eviction + +For a fixed input PCAP and configuration, matching decisions and emitted record order must be +deterministic. Packet ordering uses `(timestamp_micros, packet_ordinal)`; candidate tie-breaks within +a shard also include `record_ordinal`. Internal sequencing metadata never enters `DnsRecord`. + +Deterministic order does **not** imply globally timestamp-sorted output. Asynchronous output +container layout, such as Parquet row-group boundaries, need not be byte-identical across runs. + +`--monotonic-capture` enables batched timeout eviction under a strict globally monotonic capture +timestamp contract: + +- A timestamp regression is a hard processing error rather than a post-run warning. +- The watermark comes from the routed batch maximum timestamp, not a shard's local maximum, so + sparse shards retire stale state against the global frontier. +- Queries older than `watermark - match_timeout` may be emitted as timeouts; stale responses older + than that threshold may be discarded. +- Unresolved fragments and retained ready packets cap the watermark, as described under + [reassembly ordering](#reassembly-ordering-and-memory). + +See [RFC 0004](rfc/0004-forward-only-matcher.md) for matcher design and trade-offs. + +## Output, reporting, and shutdown + +### Records and writer handoff + +Writers receive only finalized `DnsRecord` values. A timeout means “no response observed in the +match window”: `response_timestamp` is absent, and `response_code` is absent because no DNS +response exists. The absent response timestamp is the canonical timeout signal. + +Processing sends bounded batches of records to output. User-facing channel capacity is a record +backlog, rounded up internally to batch-message slots. `config.rs` owns both the maximum records +per message and the default queued-record backlog; this handoff must remain bounded. + +### Reports and stdout + +`app.rs` derives summary metrics, including timeout ratio and average matched RTT, from +authoritative processing counters. Oversized-QNAME rejections appear as +`metrics.dns_messages_rejected_oversized_qname`, measured in rejected DNS messages. + +Final JSON reporting uses the CLI/config report-format contract. Runtime logging remains +human-readable. Stdout export is CSV-only, rejects JSON reports, and suppresses text reports. +If stderr itself is a closed pipe, the logger suppresses broken-pipe noise. + +### Completion and failure handling + +| Event | Required behavior | +| --- | --- | +| Capture read failure | Drain and join complete accepted batches, skip unmatched-query finalization, flush conclusive records as valid partial output, then return the processing error. | +| Signal-driven shutdown | Interrupt blocked stdin intake without waiting for producer EOF. Hand packets already read into a nonempty batch to processing; do not synthesize unmatched-query timeout tails. Discard still-buffered writer output before final teardown. | +| Downstream stdout pipe closes | Treat broken pipe as a downstream-close outcome and terminate the CLI gracefully. | +| Regular file-write failure | Return a fatal error. | +| Staged worker startup failure | Release input and result channels, then join every successfully started worker before returning the startup error; cleanup must not require an aggregator. | +| Normal staged teardown | Join every worker even if an earlier worker errors or panics. Preserve any parser-stage error first, otherwise the first worker error. | + +If processing failure and a signal coincide, processing failure takes precedence and the writer +flushes already buffered records. + +The optional RSS monitor has an explicit stop/join lifecycle. Stop interrupts its 100ms sampling +wait, so joining need not wait for the next sample; an in-progress RSS refresh must still finish. + +## Capture input and packet routing + +### Capture readers + +Capture reading stays in `packet_parser.rs`, including stdin probing and framing. Stdin requires +neither a temporary file nor a second ingest owner. Unsupported stdin magic and captures or +packets declared with non-Ethernet linktypes are rejected explicitly. + +| Input | Reader | +| --- | --- | +| Classic PCAP file | Pure-Rust streaming reader. | +| Regular-file PCAPNG and other non-classic formats | libpcap fallback. | +| Classic PCAP stdin | Stream-native reader from the same pure-Rust fast-path family. | +| PCAPNG stdin | Parser-owned block framing and one section-local interface table. | + +The pure-Rust classic-PCAP reader relies on `pcap-file` `3.0.0-rc1` until a stable release with the +required functionality is available. + +PCAPNG stdin uses the dependency's stateless structural validation, but **the parser owns timestamp +conversion**. It combines raw high/low timestamp words with the interface's full binary/decimal +resolution and signed offset, scales the complete counter before rounding to microseconds, and +saturates only the final signed timestamp. It never uses the dependency's stateful conversion. + +Buffers grow with received bytes, not untrusted declared lengths. Stdin PCAPNG blocks over 16MiB +are rejected before reading their body. See [RFC 0005](rfc/0005-dual-path-pcap-parsing.md) for reader +selection and monotonic timestamps. + +### Ethernet and VLAN context + +Ethernet decoding produces one canonical VLAN context: the ordered stack of tag TPIDs and 12-bit +VLAN IDs. PCP and DEI are QoS metadata and do not distinguish transactions. Fragment keys and +matcher identities share this context, keeping overlapping endpoints in different VLANs separate. + +Untagged packets allocate no tag storage. Tagged stacks use shared immutable storage and stay +internal to processing. Unsupported outer encapsulation, such as MPLS, is covered by the +[encapsulation playbook](encapsulation-playbook.md). + +### Routing metadata and hashing - M --> CLI & RT & APP - CLI --> CFG - RT --> ALLOC - APP --> CFG & DP & OUTPUT - DP --> PIPE & MATCH & PARSE & ANON - MATCH --> REC - OUTPUT --> REC & CSV & PQ - - classDef entry fill:#e8f0fe,stroke:#4285f4,color:#1a1a1a - classDef orch fill:#fef7e0,stroke:#f9ab00,color:#1a1a1a - classDef proc fill:#e6f4ea,stroke:#34a853,color:#1a1a1a - classDef data fill:#f3e8fd,stroke:#9334e6,color:#1a1a1a - classDef out fill:#fce8e6,stroke:#ea4335,color:#1a1a1a - - style Entry fill:none,stroke:#4285f4,stroke-width:1px,stroke-dasharray:4 - style Core fill:none,stroke:#34a853,stroke-width:1px,stroke-dasharray:4 - style Out fill:none,stroke:#ea4335,stroke-width:1px,stroke-dasharray:4 +Routing performs cheap L3/L4 extraction, computes the canonical client/resolver flow key, and sends +owned batches to workers before full DNS decode. DNS QR determines direction, including when both +UDP ports are 53. + +Compact UDP/DNS metadata lets workers reuse the first parse. It stays within the DNS processor's +ownership boundary so IP/port extraction has one source of truth. The metadata owns the exact +validated DNS byte range: IPv4 Total Length or IPv6 Payload Length bounds it first, then UDP Length. +Capture padding and trailing IP payload cannot extend the DNS slice. + +A compact relative offset supports DNS starts above 65535 without widening every packet's routing +metadata. It is measured from the minimum Ethernet/IPv4/UDP header length and checked to fit during +construction. + +Flow hashing buffers the tuple's existing `Hash` writes on the stack and runs SeaHash once over +those bytes. Integer encoding and digest remain identical to the streaming hasher, preserving +shard assignment and output order. + +If the encoding outgrows the buffer, routing replays its bytes +into the streaming hasher. This adds neither persistent flow state nor another canonical address +and port encoding. + +### IPv6 extension headers + +Hop-by-Hop, Routing, Destination Options, AH, and atomic Fragment headers are traversed within +the declared IPv6 payload boundary. Non-atomic IPv6 fragments are skipped; IPv6 reassembly is +unsupported. + +## IPv4 fragmentation + +### Supported modes + +| Mode | Behavior | +| --- | --- | +| Default | Skip fragmented IPv4 datagrams. | +| `--allow-fragments` | Allow inference from a first IPv4 response fragment containing a complete DNS header and question. Skip later fragments and fragmented queries. | +| `--full-fragments` | Reassemble IPv4 across batches; implies `--allow-fragments`. Fragmented queries require complete reassembly. | + +For an inferred response, the DNS slice ends at the first fragment's Total Length; UDP Length is +checked against the declared datagram size. A response code is present only if there are no +additional records or all declared DNS records fit in the prefix. + +### Reassembly and fallback + +The DNS processor owns reassembly state. A complete datagram reaches the existing UDP/DNS +validation path only after the first and last fragments establish its bounds and every byte is +present. + +The datagram bounds must agree: + +- IP payload length must equal UDP Length; mismatched fragment sets are rejected. +- When the first fragment arrives after tails, its UDP Length also bounds every retained tail. +- A nonfinal fragment must end before that boundary, regardless of arrival order. + +Incomplete responses may fall back to first-fragment inference on capacity eviction or EOF. +With monotonic capture, entries can also expire after the match timeout. Without it, timestamp +gaps between fragments of the same datagram do not expire the entry; capacity and EOF still bound +its lifetime. + +- Complete datagrams use the final fragment's (`MF=0`) capture timestamp. +- Inferred responses use the first fragment's timestamp. + +### Reassembly ordering and memory + +With `--full-fragments`, reassembly and the ready backlog sit before shard-local matching. The +arrows below show packet flow; dotted arrows show events that can resolve pending fragment sets. +Fallback supplies candidate prefixes, which still pass through UDP/DNS validation and may produce +no record. + +```mermaid +flowchart TB + Input["Packet batches"] --> Reassembly["IPv4 reassembly
pending fragment sets"] + Resolve["EOF / capacity pressure
expiry only in monotonic mode"] -.-> Reassembly + Reassembly -->|"unfragmented packets, complete datagrams,
or fallback prefixes"| Ready["Retained ready packets
held while fragments are unresolved"] + Ready -.->|"backlog limit exceeded:
resolve pending sets"| Reassembly + Ready -->|"after pending sets are resolved"| Route["Sort by timestamp and ordinal
validate packet metadata and route"] + Route --> Match["Shard-local DNS decode
and matching"] ``` -## Current Baseline - -The current accepted architecture relies on these structural choices: - -- `src/config.rs`, `src/record.rs`, and `src/output.rs` remain the single sources of truth for - runtime policy, exported record schema, and writer lifecycle. -- `src/cli.rs`, `src/runtime.rs`, and `src/app.rs` split configuration resolution, runtime - bootstrap, and top-level orchestration into explicit boundaries. -- Matching remains forward-only and deterministic. Retry deduplication applies inside the - configured match-timeout window and does not create extra terminal records. -- Staged execution is used only when the available CPU budget can support dedicated routing and - aggregation service threads without starving shard workers. -- Batched timeout eviction is opt-in and valid only under a strict globally monotonic timestamp - assumption. -- Writer threads remain asynchronous and consume only finalized records. - -## Matcher Contract - -The DNS matcher must preserve these invariants: - -- Each observed DNS query or response candidate has a stable in-flight identity until it is matched - or discarded. -- Routing and in-flight matching use the original observed client IP, client port, and resolver IP. - The resolver remains internal; deterministic client-IP pseudonymization is applied exactly once - when the matcher constructs a finalized `DnsRecord`. -- DNS ID, observed QNAME, QTYPE, QCLASS, OPCODE and the full canonical VLAN tag stack also distinguish - in-flight identities. QCLASS, OPCODE, VLAN context and resolver identity remain internal and do - not change the exported record schema. -- Repeated pending queries with the same match identity inside the configured timeout window - (`1200ms` by default) are deduplicated to the earliest canonical query. Later duplicates are - counted separately and must not create extra matched or timeout records. -- In default mode, each pending canonical query owns any retry timestamps needed to regroup - unresolved attempts when an earlier query arrives across a batch boundary. Regrouping uses - earliest-first timeout windows for that identity, so chains of regressions cannot transitively - swallow attempts outside the final canonical window. Already finalized transactions are never - reopened. Monotonic mode allocates no retry history. This adds one optional pointer per pending - query and storage for default-mode retries; only earlier-canonical replacements regroup attempts. -- Match identity preserves the observed presentation-form QNAME bytes instead of lowercasing them. - This is a deliberate Community Edition trade-off, not a protocol guarantee. RFC 4343 defines - ASCII label comparison as case-insensitive, and a valid response is allowed to differ from the - query's 0x20 casing, including when name compression reuses label bytes from another wire - location. Community Edition still keeps byte-preserving identity because that better matches the - real behavior it targets on offline caching-resolver workloads. Pairs that differ only by case - may therefore fail to match even on otherwise valid DNS traffic. -- Batched timeout eviction is valid only when the input capture is globally monotonic by packet - timestamp. In that mode, queries older than `current_watermark - match_timeout` may be emitted - as timeouts during shard processing, and stale responses older than the same threshold may be - discarded from in-flight state. -- Duplicate responses must remain distinguishable in matcher state until they are matched or - discarded. -- For a fixed input PCAP and configuration, matcher decisions and emitted record order must be - deterministic. -- Tie-breaks for equal timestamps must be explicit and stable. Scheduler interleaving and - container iteration order are not valid tie-breaks. -- A query reaches exactly one terminal outcome: matched once or emitted once as a timeout. -- Internal sequencing metadata used only to preserve determinism must not become part of exported - `DnsRecord` output. - -## Parallelism Boundary - -Parallelism is acceptable in packet ingestion and DNS extraction, where work is naturally local to -a packet. Matching state is different: it is a shared, authoritative state machine. Mixing shared -matcher mutation with worker-scheduling order makes results depend on execution timing instead of -input data. - -The required boundary is: - -- Parallel stages may produce candidate records. -- Matching may run in parallel only across independent shards. -- Each shard owns its own in-flight matcher state and must process candidates in deterministic - order. -- Query and response packets that belong to the same client/resolver flow must route to the same - shard worker before full DNS decode. -- Finalized records from shard-local processing are merged back in a deterministic order before - they reach output writers. -- Output writers consume only finalized records. - -## Notes - -- Output records are not guaranteed to be globally sorted by timestamp. -- Writers remain asynchronous. Logical record order can still be deterministic even when output - container layout, such as Parquet row-group boundaries, is not byte-identical across runs. -- IP rewriting is deterministic pseudonymization. It reduces direct exposure of source addresses, - but identical inputs still map to identical outputs. IPv4 uses an eight-round, 32-bit Feistel - permutation with an independently derived AES key, so distinct IPv4 addresses cannot collide. - IPv6 retains its original AES block mapping. The IPv4 mapping changed with this algorithm; - exports from before and after the change cannot be joined by pseudonymized IPv4 address. -- Legacy text key files retain the fixed PBKDF2 salt so existing pseudonyms remain stable across - runs and hosts. The optional salted v2 key-file format supplies a 32-byte salt for both IPv4 and - IPv6 key derivation. Reusing the full file keeps its mapping stable; changing the salt or - passphrase rotates both mappings. Key loading and derivation are owned by `anonymizer.rs`. -- `docs/rfc/` and `benches/` remain the canonical references for accepted architecture decisions - and repeatable benchmark runs. +While any datagram remains unresolved, routing retains ready packet batches. This prevents a +delayed reconstruction from arriving after a retry or response it should precede. Released packets +are sorted together by timestamp and capture ordinal before matching. The eviction watermark cannot +advance beyond unresolved fragments or retained ready packets. + +The additional ready backlog is bounded **between batches** to 65,536 packets or 64MiB of packet +payloads. Exceeding either limit resolves pending datagrams through capacity fallback before +releasing the backlog. The current input batch and fallback packets may temporarily exceed the +retained-state bound. Routing owns this buffer; it contains no query/response pairing state. + +### Duplicate history and reused IPv4 IDs + +Reassembly keeps a bounded history of completed datagrams. Fragment keys and UDP payload bytes +are compared before suppressing repeated complete datagrams or prefix fallback matching a recent +completion. + +A reused IPv4 ID with different content can form a new datagram, even if its first +fragment is identical. An incomplete new datagram whose observed fragments match a recent +completion is indistinguishable from a duplicate and may be suppressed, as may a fully identical +new datagram. + +A first non-DNS UDP fragment removes earlier tails for its key. A separate bounded history drops +later non-DNS tails without evicting completed-DNS history. A new DNS first fragment with a reused +IPv4 ID removes that mark. + +A DNS tail arriving before its first fragment while the old mark is +active is indistinguishable from an old non-DNS tail and may be dropped. + +The canonical VLAN context is part of each fragment key: different tagged segments cannot be +assembled together, while PCP or DEI changes do not split a datagram. All reassembly state must +remain bounded in memory. + +## DNS validation + +### Decoder paths and message boundaries + +`--dns-wire-fast-path` enables an optional custom question-only decoder. Hickory remains the +semantic fallback for messages the fast path does not accept. Both paths preserve these rules: + +- Ordinary QUERY messages (OPCODE 0) with more than one question are rejected under RFC 9619. +- Complete responses cannot claim answer or authority records absent from the payload, whether + or not they contain additional records. +- TSIG status extraction consumes and bounds-checks declared Other Data, including the six-byte + server time in BADTIME responses. +- OPT requires a literal root name; a compressed name that expands to root is distinct. + +### QNAME size + +Both decoder paths accept decompressed wire QNAMEs up to the RFC 1035 limit of **255 octets**, +including label-length octets and the terminating root. A valid name can expand to **1003 bytes** +in escaped presentation form and must remain distinct through matching and export. + +If any QNAME exceeds the wire limit, reject the whole DNS message before matching and increment +`oversized_qname_message_count` once. The counter measures rejected messages, not questions; +partial questions from that message must never enter matcher state or output. + +### Compression safety and resource limits + +Compression pointers must target prior, nonoverlapping names. The fast path must not weaken that +validation. + +`--max-dns-compression-jumps` (`DPP_MAX_DNS_COMPRESSION_JUMPS`) limits each decoded name to 32 pointer +transitions by default; `0` disables the resource limit. It applies to questions, scanned resource +record (RR) owners, and TSIG algorithm names in both decoder modes, including partial responses +and fast-path fallback. Cached suffixes contribute their full pointer depth. + +This is a resource policy, not a claim that longer chains violate DNS. Even with no jump limit, +strictly backward, nonoverlapping segments prevent cycles. + +### Shared name decoder + +One decoder per immutable message caches fully validated pointer targets with their encoded +segment end, expanded wire length, pointer depth, and first label. Every cache hit rechecks the +segment end against the current name's start, so a previously valid target cannot authorize an +overlap. Only referenced suffixes are retained; flat record lists do not grow the cache per record. + +RR/TSIG name skipping validates short walks directly: up to eight jumps and 64 literal bytes per +referenced segment. If either threshold or the configured jump limit would be exceeded, it uses +the shared cached walker. These thresholds choose a cheaper validation strategy; they do not +relax limits or accept invalid names. + +Small caches and traversal stacks use inline storage; larger ones allocate within the message, +bounded by the 14-bit pointer address space. Label materialization skips pointer-only segments. +Compressed questions reach Hickory already expanded, avoiding a second recursive traversal while +retaining Hickory's question semantics. + +The fast path uses a single-pass formatter for uncompressed questions. On seeing a pointer, it +restarts validation at the original name offset through the shared cached decoder. + +## IP pseudonymization + +`anonymizer.rs` owns key loading and derivation. Rewriting reduces direct exposure of source +addresses, but is deterministic: identical inputs still map to identical outputs. + +- **IPv4:** an eight-round, 32-bit Feistel permutation with an independently derived AES key. + Distinct IPv4 addresses cannot collide. This algorithm changed the IPv4 mapping; exports from + before and after that change cannot be joined by pseudonymized IPv4 address. +- **IPv6:** retains its original AES block mapping. + +Legacy text key files retain the fixed PBKDF2 salt, keeping existing pseudonyms stable across runs +and hosts. The optional salted v2 format supplies a 32-byte salt for both IPv4 and IPv6 derivation. +Reusing the full key file preserves its mapping; changing either salt or passphrase rotates both +mappings. + +## Related references + +| Reference | Use it for | +| --- | --- | +| [RFC index](rfc/README.md) | Accepted architecture decisions and their rationale; canonical list of RFCs. | +| [Encapsulation playbook](encapsulation-playbook.md) | Preparing captures with unsupported outer encapsulation. | +| [Benchmark contract](../benches/README.md) | Safety expectations, repeatable runs, and result layout. | +| [Benchmark scaffold](../benches/benchmark.sh) | Throughput and shutdown-tail measurements on caller-provided PCAP inputs. | +| [Allocator benchmarking](../benches/allocator-benchmarking.md) | Canonical allocator build matrix and measurement protocol. | diff --git a/docs/encapsulation-playbook.md b/docs/encapsulation-playbook.md index 9e06bfb..b1fe6d2 100644 --- a/docs/encapsulation-playbook.md +++ b/docs/encapsulation-playbook.md @@ -1,40 +1,49 @@ # Encapsulation Conversion Guide -## Goal +Use this guide to convert captures with MPLS or another unsupported outer encapsulation into flat +Ethernet plus IPv4 or IPv6 captures that DPP can read directly. -This document explains how to convert a capture that contains MPLS or another unsupported outer -encapsulation layer into a flat Ethernet plus IPv4 or IPv6 capture that DPP can read directly. -Ethernet VLAN and QinQ stacks with TPIDs `0x8100`, `0x88a8`, or `0x9100` are decoded natively. -Matching and IPv4 reassembly preserve the complete ordered TPID/VLAN-ID stack, so overlapping -IP endpoints on different tagged segments remain separate. PCP and DEI changes are ignored for -this identity; no VLAN column is added to exported records. +- [Decide whether conversion is needed](#decide-whether-conversion-is-needed) +- [Convert a capture on Linux](#convert-a-capture-on-linux) +- [Validate the output](#validate-the-output) +- [Add native support](#add-native-support) -Target output shape: +## Decide whether conversion is needed -1. Ethernet -2. IPv4 or IPv6 -3. UDP -4. DNS payload +| Capture layers | Action | +| --- | --- | +| Ethernet → IPv4 or IPv6 → UDP → DNS | Read directly; no conversion needed. | +| Ethernet with VLAN or QinQ tags | Read directly; TPIDs `0x8100`, `0x88a8`, and `0x9100` are supported natively. | +| MPLS labels or another unsupported shim between Ethernet and IP | Inspect the capture, then normalize supported labels with the script below. | -If the input capture already has that shape, no conversion is needed. +A typical symptom of unsupported encapsulation is that DPP runs, but query or response counts are +lower than expected because the current fast path does not extract the labeled packets. -## When to Use This Guide +Native VLAN and QinQ handling preserves the complete ordered TPID/VLAN-ID stack in matching and +IPv4 reassembly. Overlapping IP endpoints on different tagged segments remain separate. PCP and +DEI changes do not affect this identity, and exported records do not gain a VLAN column. -Use this guide when the original capture contains any of the following before the IP header: +> The normalization script removes VLAN and QinQ tags as well as MPLS labels. VLAN or +> provider-bridging layers alone do not require conversion. -- MPLS label stacks -- other unsupported shim layers between Ethernet and IP +## Convert a capture on Linux -Typical symptom: +Keep the original capture unchanged for audit and debugging. Write to a separate output file and +treat the normalized capture as derived data; do not overwrite the source in place. -- DPP runs, but query or response counts are lower than expected because unsupported labeled packets - are not being extracted by the current fast path. +### 1. Install the tools -## Linux Workflow +Install `tshark` and Python support: -### 1. Inspect the Capture +```bash +sudo apt-get update +sudo apt-get install -y tshark python3 python3-pip +python3 -m pip install --user scapy +``` + +### 2. Inspect the capture -Use `tshark` to confirm whether the capture contains VLAN, QinQ, MPLS, or other outer layers. +Use `tshark` to check for VLAN, QinQ, MPLS, or other outer layers: ```bash tshark -r input.pcap -q -z io,phs @@ -43,25 +52,22 @@ tshark -r input.pcap -q -z io,phs If you see `mpls` or another unsupported layer before `ip` or `ipv6`, continue with normalization. VLAN and provider-bridging layers alone do not require this step. -### 2. Install the Required Tools - -Install `tshark` and Python support on Linux: - -```bash -sudo apt-get update -sudo apt-get install -y tshark python3 python3-pip -python3 -m pip install --user scapy -``` - If the capture is `pcapng` and a downstream tool expects classic `pcap`, convert it first: ```bash editcap -F libpcap input.pcapng input.pcap ``` -### 3. Normalize the Capture +### 3. Normalize the capture + +The script preserves Ethernet source and destination MAC addresses and packet timestamps. It strips +outer VLAN, QinQ, and MPLS layers, then writes only packets that resolve cleanly to IPv4 or IPv6. -Save the following script as `normalize_encapsulation.py`: +It does not decode or rewrite DNS contents, preserve unsupported non-IP payloads after stripping, +or guarantee support for every proprietary shim layer. If too many packets are skipped, inspect +the protocol hierarchy again for a layer the script cannot handle. + +Save the following as `normalize_encapsulation.py`: ```python #!/usr/bin/env python3 @@ -147,71 +153,50 @@ if __name__ == "__main__": raise SystemExit(main()) ``` -Run it: +Run it with separate input and output paths: ```bash python3 normalize_encapsulation.py input.pcap normalized.pcap ``` -What this does: - -- preserves Ethernet source and destination MAC addresses; -- removes outer VLAN, QinQ, and MPLS layers; -- keeps the original packet timestamp; -- writes only packets that resolve cleanly to IPv4 or IPv6 after stripping the outer labels. - -What it does **not** do: +## Validate the output -- it does not preserve unsupported non-IP payloads after label stripping; -- it does not decode or rewrite DNS contents; -- it does not guarantee support for every possible proprietary shim layer. - -### 4. Validate the Result - -Check that the normalized capture now has the expected protocol hierarchy: +### 1. Check the protocol hierarchy ```bash tshark -r normalized.pcap -q -z io,phs ``` -You should now see the DNS packets under plain IPv4 or IPv6 instead of under outer VLAN or MPLS -labels. +DNS packets should now appear under plain IPv4 or IPv6 instead of outer VLAN or MPLS labels. + +### 2. Run DPP -### 5. Run DPP on the Normalized Capture +For CSV: ```bash target/release/dpp normalized.pcap normalized.csv --format csv ``` -Or for Parquet: +For Parquet: ```bash target/release/dpp normalized.pcap normalized.pq --format pq ``` -## Validation Checklist - -After conversion, validate all of the following: - -- the output capture still preserves the expected packet count order and timestamps; -- DPP query and response counts move in the expected direction; -- representative DNS flows that were previously missing are now present; -- repeated DPP runs on the normalized capture remain deterministic. - -## Safety Notes +### 3. Check the results -- Keep the original capture unchanged for audit and debugging. -- Do not overwrite the source file in place. -- Treat the normalized capture as derived data from the original packet stream. -- If the conversion script skips too many packets, inspect the protocol hierarchy again; the - capture may contain an encapsulation layer that is still unsupported by the script. +- Confirm the expected output packet count, order, and timestamps. +- Check that DPP query and response counts move in the expected direction. +- Inspect representative DNS flows that were previously missing. +- Confirm that repeated DPP runs on the normalized capture remain deterministic. -## If Native Support Is Needed +## Add native support -If normalization is not acceptable operationally and DPP must read those captures directly, native -support belongs in `src/dns_processor/parser.rs` and, for fragmented IPv4, `reassembly.rs`. +If normalization is not acceptable operationally, native support belongs in +[`src/dns_processor/parser.rs`](../src/dns_processor/parser.rs) and, for fragmented IPv4, +[`reassembly.rs`](../src/dns_processor/reassembly.rs). -That change must be accompanied by: +Any such change requires: - unit tests for every newly supported encapsulation layer; - determinism checks on representative captures; diff --git a/docs/rfc/0001-architecture-boundaries.md b/docs/rfc/0001-architecture-boundaries.md index 592e5cc..6e0e08d 100644 --- a/docs/rfc/0001-architecture-boundaries.md +++ b/docs/rfc/0001-architecture-boundaries.md @@ -1,41 +1,40 @@ -# RFC 0001 — Ownership Boundaries and Benchmark Contract +# RFC 0001 — Ownership boundaries and benchmark contract -Status: Accepted -Date: 2025-01-14 +**Status:** Accepted · **Date:** 2025-01-14 ## Why this matters -DPP is a multi-threaded pipeline with distinct stages: packet parsing, flow-based routing, -DNS query/response matching, and serialization to CSV or Parquet. Without explicit ownership -boundaries, responsibility bleeds across modules — configuration gets duplicated, lifecycle -contracts become implicit, and shutdown turns into a guessing game. +DPP has distinct stages for packet parsing, flow routing, DNS query/response matching, and +CSV or Parquet serialization. Explicit ownership boundaries keep configuration in one place +and make lifecycle and shutdown responsibilities clear. -On top of that, the project needs a repeatable benchmark that doesn't depend on local paths -or private PCAP captures sitting on someone's laptop. +The project also needs repeatable benchmarks that do not depend on hardcoded local paths or +private captures. ## Decision Each module owns exactly one responsibility: -| Module | What it owns | -|-----------------------------------|----------------------------------------------------------------------------------------------| -| `src/config.rs` | Single source of truth for runtime policy: batch sizes, timeouts, execution-model thresholds | -| `src/output.rs` | Writer lifecycle and output-channel control messages | -| `src/monitor_memory.rs` | Optional RSS monitoring with explicit stop/join — must not outlive the process | -| `src/app.rs` | Run orchestration, reporting, shutdown coordination | -| `src/main.rs` | Thin entrypoint: wires `cli` + `runtime` + `app`, returns exit code | -| `src/dns_processor/anonymizer.rs` | Key loading, PBKDF2 derivation, deterministic IP pseudonymization | +| Module | Ownership | +| --- | --- | +| [`config.rs`](../../src/config.rs) | Single source of truth for runtime policy: batch sizes, timeouts, and execution-model thresholds | +| [`output.rs`](../../src/output.rs) | Writer lifecycle and output-channel control messages | +| [`monitor_memory.rs`](../../src/monitor_memory.rs) | Optional RSS monitoring with explicit stop/join; must not outlive the process | +| [`app.rs`](../../src/app.rs) | Run orchestration, reporting, and shutdown coordination | +| [`main.rs`](../../src/main.rs) | Thin entrypoint: compose `cli`, `runtime`, and `app`; return the exit code | +| [`dns_processor/anonymizer.rs`](../../src/dns_processor/anonymizer.rs) | Key loading, PBKDF2 derivation, and deterministic IP pseudonymization | -Benchmark scaffolding lives under `benches/` and takes all inputs from the caller. +Benchmark scaffolding lives under [`benches/`](../../benches/README.md) and takes all inputs +from the caller. ## Rationale -- **SSOT** — every policy is defined in one place. Changing the match timeout means editing - `config.rs`, not grepping the whole tree. -- **Explicit lifecycle** — shutdown doesn't depend on drop ordering. `monitor_memory` has - `stop()` + `join()`, the output channel closes via a control message, not by dropping the sender. -- **Reproducible benchmarks** — the script has no idea where PCAP files live until you tell it. - This keeps private data out of the repository by construction. +- **Single source of truth (SSOT):** runtime policy changes, such as match-timeout defaults, + belong in `config.rs`. +- **Explicit lifecycle:** `monitor_memory` uses `stop()` and `join()`; the output channel closes + through a control message. Shutdown does not depend on drop ordering or dropping the sender. +- **Reproducible benchmarks:** callers supply capture paths, keeping private input data out of + the repository. ## Consequences diff --git a/docs/rfc/0002-cli-and-runtime-boundaries.md b/docs/rfc/0002-cli-and-runtime-boundaries.md index 017cbaa..d85f88f 100644 --- a/docs/rfc/0002-cli-and-runtime-boundaries.md +++ b/docs/rfc/0002-cli-and-runtime-boundaries.md @@ -1,45 +1,37 @@ -# RFC 0002 — CLI and Runtime Bootstrap Boundaries +# RFC 0002 — CLI and runtime bootstrap boundaries -Status: Accepted -Date: 2025-02-07 +**Status:** Accepted · **Date:** 2025-02-07 ## Problem -The project used to have a single `src/utils.rs` that did everything: CLI parsing, environment -overrides, output-path validation, logger init, Rayon pool creation, memory-monitor startup, -and duration formatting. It worked, but it was the kind of "works" where adding a new CLI flag -meant touching the same file that sets up the thread pool. +The former `src/utils.rs` combined CLI parsing, environment overrides, output-path validation, +logger initialization, Rayon pool creation, memory-monitor startup, and duration formatting. -The real cost wasn't bugs — it was review friction. Every PR that touched CLI precedence also -touched runtime bootstrap, and vice versa. Ownership was blurry. +This mixed argument resolution with host setup. Changes to CLI precedence and runtime bootstrap +shared the same module, making ownership and review harder to follow. ## Decision Split the responsibilities into three focused modules: -1. **`src/cli.rs`** — CLI parsing, environment-variable precedence, output-path validation. - This is where you go when you need a new flag or want to change how env vars override CLI args. +| Module | Responsibility | Change here when… | +| --- | --- | --- | +| [`cli.rs`](../../src/cli.rs) | CLI parsing, environment-variable precedence, and output-path validation | Adding a flag or changing argument resolution | +| [`runtime.rs`](../../src/runtime.rs) | Logger setup, build/system logging, optional memory monitoring, signal handling, and Rayon pool creation | Changing host setup or bootstrap side effects | +| [`app.rs`](../../src/app.rs) | Ordered run orchestration and reporting | Changing how the pipeline runs or reports results | -2. **`src/runtime.rs`** — Runtime bootstrap: logger setup, build/system info logging, optional - memory monitoring, signal handling, Rayon pool creation. Side-effectful host setup that - shouldn't leak into argument resolution. - -3. **`src/app.rs`** — Ordered run orchestration and reporting. Doesn't parse arguments, doesn't - set up loggers, doesn't create thread pools. Just runs the pipeline and reports results. - -`src/main.rs` composes all three and stays thin. +[`main.rs`](../../src/main.rs) composes all three and stays thin. `app.rs` does not parse +arguments, initialize loggers, or create thread pools. ## Why this split -- CLI precedence has one owner. If someone asks "does the env var override the flag or the other - way around?", the answer is always in `cli.rs`. -- Bootstrap side effects are isolated. Logger init, signal handlers, and thread-pool creation - don't contaminate configuration resolution. -- Unit tests for CLI precedence don't need to reach through a mixed helper module anymore. +- CLI precedence has one owner: `cli.rs`. +- Bootstrap side effects stay separate from configuration resolution. +- CLI precedence can be unit-tested without going through a mixed helper module. ## Consequences -- New CLI flags or environment variables → `src/cli.rs`. -- New bootstrap concerns (logger tweaks, monitoring, pool config) → `src/runtime.rs`. -- If bootstrap grows unwieldy again, decompose *within* `src/runtime.rs` — don't recreate - a catch-all utils module. +- Keep new CLI flags and environment variables in `src/cli.rs`. +- Keep new bootstrap concerns in `src/runtime.rs`. +- If bootstrap needs further decomposition, split it within its existing boundary. Do not + recreate a catch-all utilities module. diff --git a/docs/rfc/0003-allocator-selection.md b/docs/rfc/0003-allocator-selection.md index fe89151..a167bdc 100644 --- a/docs/rfc/0003-allocator-selection.md +++ b/docs/rfc/0003-allocator-selection.md @@ -1,29 +1,28 @@ -# RFC 0003 — Compile-Time Allocator Selection +# RFC 0003 — Compile-time allocator selection -Status: Accepted -Date: 2025-03-21 +**Status:** Accepted · **Date:** 2025-03-21 ## Problem DPP is allocation-heavy: the parser, matcher, and writers all allocate on hot paths. The global -allocator has a measurable impact on throughput, RSS, and cross-thread free behavior. For a while, -the allocator was hardcoded in `src/main.rs` via a direct `jemallocator` dependency — which meant -allocator policy was implicit, hard to benchmark, and tangled with the process entrypoint. +allocator affects throughput, RSS, and cross-thread frees. The allocator was previously hardcoded +in `src/main.rs` through a direct `jemallocator` dependency, making its policy implicit and +harder to benchmark independently of the entrypoint. Allocator choice is fundamentally a build-time decision: it affects the entire binary, must be settled before `main()` runs, and must not drift between CLI flags, env vars, and code. ## Decision -Allocator selection is owned by **`src/allocator.rs`** and configured exclusively through +Allocator selection is owned by [`src/allocator.rs`](../../src/allocator.rs) and configured exclusively through mutually exclusive Cargo features: -| Feature | Allocator | Notes | -|----------------------|----------------------|----------------------------------------------------------------------| -| `allocator-jemalloc` | `tikv-jemallocator` | Default. Good all-round choice for throughput and RSS. | -| `allocator-mimalloc` | `mimalloc` | Alternative with different fragmentation characteristics. | -| `allocator-system` | `std::alloc::System` | Useful as a baseline for benchmarking. | -| `allocator-tcmalloc` | `tcmalloc-better` | Linux x86_64 / aarch64 only. Fails to compile elsewhere — by design. | +| Feature | Allocator | Role or restriction | +| --- | --- | --- | +| `allocator-jemalloc` | `tikv-jemallocator` | Default choice for throughput and RSS | +| `allocator-mimalloc` | `mimalloc` | Alternative with different fragmentation characteristics | +| `allocator-system` | `std::alloc::System` | Benchmark baseline | +| `allocator-tcmalloc` | `tcmalloc-better` | Linux x86_64 / aarch64 only; fails to compile elsewhere | Exactly one feature must be enabled. Invalid combinations fail the build with a clear error. @@ -31,23 +30,21 @@ Exactly one feature must be enabled. Invalid combinations fail the build with a ## Why not a runtime flag? -Because swapping the global allocator at runtime isn't a thing in Rust (and for good reason). -Making it a Cargo feature keeps the decision explicit, auditable, and benchmarkable. You rebuild, -you re-measure — there's no illusion that you can just flip a switch. +The global allocator is selected before `main()` runs. Cargo features make that choice explicit +and reproducible: change the feature, rebuild, and measure the resulting binary. ## Trade-offs -- Allocator changes require a rebuild. That's the point. +- Allocator changes require a rebuild. - `allocator-tcmalloc` intentionally fails on unsupported targets instead of silently falling back. - Surprises in production are worse than a compile error. -- `cargo check --all-features` doesn't work because the features are mutually exclusive. This is - a known ergonomic cost, but the alternative (runtime dispatch or silent feature priority) is worse. +- `cargo check --all-features` cannot pass because the features are mutually exclusive. Each + allocator configuration must be checked separately; there is no silent feature priority. ## Validation When changing allocator configuration: -- `cargo test` with the default feature; -- `cargo check` for each supported alternative; -- throughput + RSS benchmarks via `benches/` — don't assume a new allocator is faster without - measuring. +1. Run `cargo test` with the default feature. +2. Run `cargo check` for each supported alternative. +3. Measure throughput and RSS using the [allocator benchmark protocol](../../benches/allocator-benchmarking.md). + An allocator change needs measured results, not an assumed speedup. diff --git a/docs/rfc/0004-forward-only-matcher.md b/docs/rfc/0004-forward-only-matcher.md index d9ea62f..907f206 100644 --- a/docs/rfc/0004-forward-only-matcher.md +++ b/docs/rfc/0004-forward-only-matcher.md @@ -1,118 +1,150 @@ -# RFC 0004 — Forward-Only Matcher and Determinism Contract +# RFC 0004 — Forward-only matcher and determinism contract -Status: Accepted -Date: 2025-06-03 +**Status:** Accepted · **Date:** 2025-06-03 ## Problem -DNS query/response matching sounds simple until you try to do it in parallel on a 20 GB capture. -The naive approach — a shared hash map protected by a mutex — serializes the entire pipeline on -the matcher. The "clever" approach — lock-free concurrent maps — makes results depend on thread -scheduling, which means the same input can produce different output on different runs. - -Neither is acceptable. DPP needs matching that is both parallel and deterministic. +DPP needs DNS query/response matching that is both parallel and deterministic on large captures. +A shared map protected by a mutex serializes matcher work. Concurrent maps alone do not define a +deterministic processing order: the same input can still produce different results as thread +scheduling changes. ## Decision The matcher is **forward-only**: it processes packets in `(timestamp_micros, packet_ordinal)` -order within each shard and never backtracks. The key design choices: - -1. **Shard-local state.** Each shard owns its own `QueryMap` and `ResponseMap` (both `BTreeMap`- - based). There is no shared mutable matcher state between shards. - -2. **Canonical flow routing.** Before full DNS decode, the routing stage extracts a cheap - `CanonicalFlowKey` (observed client IP, client port and resolver IP, oriented by DNS QR) - and hashes it to a shard index. This includes the valid case where both ports are 53 and - guarantees that a query and its - matching response always land in the same shard. This is a coarser partition than matcher - identity: VLANs with overlapping endpoints may share a worker, but their transaction state - remains distinct. - -3. **Deterministic ordering.** Within a shard, packets are processed in strict - `(timestamp, packet_ordinal, record_ordinal)` order. Tie-breaks are explicit — scheduler - interleaving and container iteration order are not valid tie-breaks. - - With full IPv4 reassembly, routing retains ready packet batches while fragment sets remain - unresolved. A reconstructed datagram can carry an earlier final-fragment timestamp, so merely - limiting timeout eviction would still let a retry or response finalize before that datagram. - Routing releases the retained packets and reconstructed datagrams together for the existing - timestamp/ordinal sort. This also preserves ordering within a batch whose timestamps regress - in default mode; it does not impose global monotonicity on that mode. - - This packet backlog is separate from matcher state and bounded between batches to one packet - batch (65,536 packets) or 64 MiB of payloads. On overflow, pending fragment sets use the existing - capacity fallback before routing releases the backlog. Processing the current input batch and - fallback packets can temporarily exceed those retained-state limits. EOF and interrupted - intake drain accepted ready packets through the existing shutdown policy. The matcher eviction - watermark is capped by both the oldest unresolved fragment and the oldest retained ready packet. - -4. **Retry deduplication.** If a query with the same identity arrives while an earlier one is - still pending inside the match-timeout window (1200 ms by default), the duplicate is counted - but doesn't create a second canonical query. One canonical query → one terminal outcome - (matched or timeout), always. Default mode retains retry timestamps inside that pending query's - payload. If an earlier query arrives across batches, the matcher regroups the identity's - unresolved attempts into earliest-first timeout windows. For example, pending attempts observed - at 3s, 2s and then 1s with a 1.2s window leave canonicals at 1s and 3s, rather than losing the - latter through transitive deduplication. Finalized transactions are never reopened. - - Retry history is allocated only when a duplicate is observed in default mode; monotonic mode - needs none. Regrouping costs `O(n log n)` in the affected identity's unresolved attempts and is - restricted to earlier-canonical replacements. The canonical query payload remains the sole owner - of retry history, with no secondary matcher map. - - Match identity includes the DNS ID, observed name, client IP and port, resolver IP, query - type, query class, opcode and canonical VLAN context. Ethernet decoding owns the ordered - TPID/12-bit-VID stack used by both matcher and fragment keys; PCP and DEI bits are excluded. - Untagged traffic needs no tag allocation, while tagged metadata and keys share immutable tag - storage. Resolver identity, query class, opcode and VLAN context remain internal and are not - added to the exported `DnsRecord` schema. - - The current Community Edition identity key preserves the observed presentation-form QNAME bytes - and does not lowercase them before matching. This is a deliberate Community Edition trade-off, - not a protocol guarantee. RFC 4343 defines ASCII label comparison as case-insensitive, and a - valid response is allowed to differ from the query's 0x20 casing, including when name - compression reuses label bytes from another wire location. Community Edition still keeps - byte-preserving identity because that better matches the real behavior it targets on offline - caching-resolver workloads. Queries and responses that differ only by case may therefore fail - to pair even on otherwise valid DNS traffic. - -5. **Closest-match pairing.** When a response arrives, the matcher finds the pending query with - the closest timestamp (within the timeout window). When a query arrives and a buffered response - already exists, the same closest-timestamp logic applies in reverse. This handles mild - reordering without sacrificing determinism. - -6. **Batched timeout eviction** (opt-in via `--monotonic-capture`). When the capture is globally - monotonic, the matcher can evict stale queries in bulk using the batch-maximum timestamp as a - watermark. The watermark comes from the *routed batch maximum*, not each shard's local maximum, - so sparse shards still retire state against the global frontier. +order within each shard and never backtracks. Each shard owns its state; routing and explicit +ordering keep matching independent of worker scheduling. + +- [State ownership and routing](#state-ownership-and-routing) +- [Deterministic ordering](#deterministic-ordering) +- [Transaction identity](#transaction-identity) +- [Retry deduplication](#retry-deduplication) +- [Closest-match pairing](#closest-match-pairing) +- [Batched timeout eviction](#batched-timeout-eviction) + +### State ownership and routing + +Each shard owns its own `QueryMap` and `ResponseMap`, both based on `BTreeMap`. Shards share no +mutable matcher state. + +Before full DNS decode, routing extracts a `CanonicalFlowKey`: observed client IP, client port, +and resolver IP, oriented by DNS QR. Hashing this key assigns a shard, including when both ports +are 53. A query and its matching response therefore reach the same shard. + +Flow routing is coarser than transaction identity. VLANs with overlapping endpoints may share a +worker, but their transaction state remains distinct. + +### Deterministic ordering + +Within each shard, candidates are processed in strict `(timestamp, packet_ordinal, record_ordinal)` +order. Scheduler interleaving and container iteration order are never valid tie-breaks. + +With full IPv4 reassembly, routing retains ready packet batches while fragment sets remain +unresolved. A reconstructed datagram can carry an earlier final-fragment timestamp. Capping +only timeout eviction would still let a retry or response finalize before that datagram arrives. + +Routing releases retained packets and reconstructed datagrams together for timestamp/ordinal +sorting. This also preserves ordering within a batch whose timestamps regress in default mode; +it does not impose global monotonicity on that mode. + +The ready-packet backlog is separate from matcher state: + +| Constraint or event | Behavior | +| --- | --- | +| Retained-state bound between batches | One packet batch (65,536 packets) or 64 MiB of payloads | +| Either bound exceeded | Resolve pending fragment sets through the existing capacity fallback, then release the backlog | +| Current batch and fallback processing | May temporarily exceed the retained-state bounds | +| EOF or interrupted intake | Drain accepted ready packets under the existing shutdown policy | +| Matcher eviction watermark | Cap at both the oldest unresolved fragment and the oldest retained ready packet | + +### Transaction identity + +Matcher identity includes: + +- DNS ID and observed presentation-form QNAME; +- observed client IP and port, plus resolver IP; +- query type, query class, and opcode; +- canonical VLAN context. + +Ethernet decoding owns the ordered TPID/12-bit-VID stack used by matcher and fragment keys. +PCP and DEI bits are excluded. Untagged traffic needs no tag allocation; tagged metadata and keys +share immutable tag storage. + +Resolver identity, query class, opcode, and VLAN context remain internal. They are not added to +the exported `DnsRecord` schema. + +#### QNAME casing limitation + +Community Edition preserves observed presentation-form QNAME bytes without lowercasing them. +This is a deliberate trade-off for the offline caching-resolver workloads it targets, not a +protocol guarantee. + +RFC 4343 defines ASCII label comparison as case-insensitive. A valid response may differ from +the query's 0x20 casing, including when compression reuses label bytes from another wire location. +Such query/response pairs may fail to match in Community Edition even on otherwise valid DNS +traffic. + +### Retry deduplication + +A repeated query with the same identity inside the match-timeout window (`1200 ms` by default) +is counted as a duplicate while the earlier query remains pending. It does not create a second +canonical query. On normal completion, each canonical query has one terminal outcome: matched +once or emitted once as a timeout. + +Default mode retains retry timestamps in the pending canonical query's payload. If an earlier +query arrives across batches, the matcher regroups that identity's unresolved attempts into +earliest-first timeout windows. Finalized transactions are never reopened. + +For example, attempts observed at 3s, then 2s, then 1s with a 1.2s window leave canonical queries +at 1s and 3s. Transitive deduplication must not absorb the 3s attempt into the earlier window. + +Retry history is allocated only when default mode observes a duplicate. Monotonic mode needs +none. Regrouping costs `O(n log n)` in the affected identity's unresolved attempts and occurs +only for earlier-canonical replacements. The canonical query payload is the sole owner of this +history; there is no secondary matcher map. + +### Closest-match pairing + +When a response arrives, the matcher selects the pending query with the closest timestamp within +the timeout window. When a query arrives and a buffered response already exists, the same logic +applies in reverse. This handles mild reordering while preserving deterministic decisions. + +### Batched timeout eviction + +`--monotonic-capture` opts into bulk eviction of stale queries using a batch-maximum watermark. +The capture must be globally monotonic under the [RFC 0005 contract](0005-dual-path-pcap-parsing.md#monotonic-timestamp-contract). + +The watermark comes from the **routed batch maximum**, not each shard's local maximum. Sparse +shards therefore retire state against the global frontier. ## Invariants These must hold for any valid implementation: -- Each query reaches exactly one terminal outcome: matched once, or emitted once as a timeout. +- On normal completion, each canonical query reaches exactly one terminal outcome: matched once + or emitted once as a timeout. Interrupted and failed runs follow the separate + [shutdown policy](../architecture.md#completion-and-failure-handling). - For a fixed input PCAP and runtime configuration, finalized record values and order are deterministic. Output container layout, such as Parquet row-group boundaries, need not be byte-identical across runs. -- Internal sequencing metadata (`packet_ordinal`, `record_ordinal`) never leaks into the exported +- Internal sequencing metadata (`packet_ordinal`, `record_ordinal`) never enters the exported `DnsRecord`. - Duplicate responses remain distinguishable in matcher state until matched or discarded. -- Community Edition intentionally preserves observed QNAME bytes instead of RFC-4343-style - case-insensitive canonicalization. Case-only query/response mismatches are therefore an accepted - limitation even on otherwise valid DNS traffic. +- QNAME identity preserves observed bytes. Case-only query/response mismatches are an accepted + Community Edition limitation, even on otherwise valid DNS traffic. ## Why BTreeMap and not HashMap -The matcher uses `BTreeMap` keyed on `(identity, timestamp, packet_ordinal, record_ordinal)`. -This gives ordered iteration for free, which is essential for closest-match lookups and -deterministic eviction. A `HashMap` would require sorting on every lookup or eviction pass — -possible, but slower and more error-prone. +The matcher uses `BTreeMap` for identity-indexed query and response state. Each identity has a +timeline ordered by `(timestamp_micros, packet_ordinal, record_ordinal)`. Ordered iteration +supports closest-match lookups and deterministic eviction; a `HashMap` would require additional +sorting for those operations. ## Consequences - The matcher is the authoritative owner of in-flight state. No other module may hold or mutate query/response pairing data. -- Adding a new matching strategy (e.g., bidirectional or streaming) requires a new RFC. -- Timeout records encode "no response observed" by leaving response fields absent. The canonical +- A new matching strategy, such as bidirectional or streaming matching, requires a new RFC. +- Timeout records leave response fields absent because no response was observed. The canonical timeout signal is an absent `response_timestamp`; `response_code` is absent because no DNS response exists. diff --git a/docs/rfc/0005-dual-path-pcap-parsing.md b/docs/rfc/0005-dual-path-pcap-parsing.md index b3c676a..4a8c7fa 100644 --- a/docs/rfc/0005-dual-path-pcap-parsing.md +++ b/docs/rfc/0005-dual-path-pcap-parsing.md @@ -1,173 +1,202 @@ -# RFC 0005 — Dual-Path PCAP Parsing and Monotonic Timestamp Contract +# RFC 0005 — Dual-path PCAP parsing and monotonic timestamp contract -Status: Accepted -Date: 2025-08-18 +**Status:** Accepted · **Date:** 2025-08-18 ## Problem -PCAP comes in two major flavors: classic PCAP (the original tcpdump format) and PCAPNG (the newer, -more featureful format). Most large DNS captures in production are still classic PCAP, but PCAPNG -shows up often enough that ignoring it isn't an option. +DPP must read both classic PCAP and PCAPNG. Most large DNS captures use classic PCAP, where +per-packet `libpcap` FFI calls and opaque buffering add avoidable overhead. A pure-Rust reader +gives DPP more control over this common path. -The challenge is that `libpcap` — the standard C library for reading both formats — carries -overhead that matters at scale: FFI crossings on every packet, less control over buffering, and -an opaque internal state machine. For classic PCAP, we can do better with a pure-Rust reader. -For regular-file PCAPNG and other non-classic formats, we still rely on fallback compatibility -today. For stdin streams, however, probing consumes bytes that cannot be rewound safely on pipes, -so the parser must stay stream-native once it has inspected the magic bytes. +Regular files can retain `libpcap` fallback compatibility. Stdin needs a different approach: +probing consumes bytes that a pipe cannot safely rewind, so the parser must stay stream-native +after inspecting the magic bytes. -Separately, some captures are known to be globally monotonic by timestamp — every packet's -timestamp is ≥ the previous one. When that holds, the matcher can use batch-level watermarks -to evict stale state in bulk instead of checking timeouts per-query. But if the assumption is -wrong, the results are silently incorrect. So the contract needs teeth. +Some captures also have globally non-decreasing timestamps. This permits bulk matcher eviction +using batch-level watermarks, but a timestamp regression would make that eviction unsafe. +Enabling the optimization therefore requires strict validation. ## Decision -### Parsing backends +The parser selects a reader for the input source and format, owns timestamp conversion, and +validates protocol boundaries before handing packets to DNS matching. + +- [Parsing backends](#parsing-backends) +- [PCAPNG timestamp ownership](#pcapng-timestamp-ownership) +- [IP and UDP boundaries](#ip-and-udp-boundaries) +- [DNS QNAME boundary](#dns-qname-boundary) +- [Monotonic timestamp contract](#monotonic-timestamp-contract) -`PacketParser` (in `src/packet_parser.rs`) picks the backend at open time by inspecting the -file's magic bytes when the input source is a regular file: +### Parsing backends -- **Classic PCAP** → pure-Rust streaming reader via `pcap-file`. No FFI, no `libpcap` dependency - on this path. The reader is zero-copy where possible and gives us full control over buffering - and batch construction. +[`PacketParser`](../../src/packet_parser.rs) selects a backend at open time by inspecting the +first four bytes: -- **Everything else** (PCAPNG, modified formats) → `libpcap` fallback via the `pcap` crate. - Correct but slower. This path exists so DPP doesn't reject valid captures — it just won't be - as fast. +| Input source and format | Backend or outcome | +| --- | --- | +| Regular file: classic PCAP | Pure-Rust streaming reader via `pcap-file`; no FFI or `libpcap` on this path | +| Regular file: PCAPNG or another non-classic format | `libpcap` fallback via the `pcap` crate | +| EOF-terminated stdin: classic PCAP | Pure-Rust reader via `pcap-file`, with parser-owned probe-and-replay | +| EOF-terminated stdin: PCAPNG | Parser-owned block framing and section-local interface table; stateless validation via `pcap-file` | +| EOF-terminated stdin: unknown magic | Explicit rejection | -- **EOF-terminated stdin classic PCAP streams** → pure-Rust streaming reader via `pcap-file`, - using parser-owned probe-and-replay so the same stdin byte stream remains the single source of - truth. +The pure-Rust classic reader is zero-copy where possible and owns buffering and batch +construction. The regular-file fallback preserves compatibility at a higher processing cost. -- **EOF-terminated stdin PCAPNG streams** → parser-owned pure-Rust block framing and one - section-local interface table, with stateless block/body/option validation via `pcap-file`. - Once stdin bytes have been inspected, the parser stays stream-native instead of trying to - reopen the stream through a second owner. +Stdin probing replays bytes into the same stream, preserving one input owner. DPP neither reopens +an inspected stdin stream through `libpcap` nor creates a temporary file or hidden second ingest +path for unsupported formats. -- **EOF-terminated stdin streams with unknown magic** → explicit rejection. DPP does not create a - temp file or hidden second ingest path to recover fallback compatibility for unsupported stdin - stream formats. +| Format | Recognized magic values | +| --- | --- | +| Classic PCAP | `0xa1b2c3d4`, `0xd4c3b2a1` | +| Classic PCAP, nanosecond resolution | `0xa1b23c4d`, `0x4d3cb2a1` | +| PCAPNG | `0x0a0d0d0a` | -The detection is a simple 4-byte magic check (`0xa1b2c3d4` or `0xd4c3b2a1` for classic, -`0xa1b23c4d` or `0x4d3cb2a1` for nanosecond-resolution classic, `0x0a0d0d0a` for PCAPNG). For -regular files, everything else goes to `libpcap`. For stdin streams, everything else is rejected. +Other magic values go to `libpcap` for regular files and are rejected for stdin. ### PCAPNG timestamp ownership -The parser reads timestamp high/low words separately in section byte order for both Enhanced -Packet Blocks and legacy Packet Blocks. It applies each interface's `if_tsresol` and signed -`if_tsoffset` exactly once. All seven-bit resolution exponents are accepted; the complete raw -counter is scaled before integer rounding to microseconds, including sub-nanosecond units. -The final timestamp saturates to `i64` only after adding the signed offset. New sections reset -the interface table and may change byte order. +#### Conversion rules + +For both Enhanced Packet Blocks and legacy Packet Blocks, the parser: + +1. Reads the timestamp high/low words separately in section byte order. +2. Applies the interface's `if_tsresol` exactly once, scaling the complete raw counter before + integer rounding to microseconds. This includes sub-nanosecond units and all seven-bit + resolution exponents. +3. Adds the signed `if_tsoffset` exactly once. +4. Saturates the final timestamp to `i64`. + +Each new section resets the interface table and may change byte order. + +#### Dependency boundary -The stateful `pcap-file 3.0.0-rc1` reader is deliberately bypassed: it swaps legacy little-endian +DPP bypasses the stateful `pcap-file 3.0.0-rc1` reader. That reader swaps legacy little-endian timestamp words, mis-scales binary resolutions, ignores offsets, and panics on exponent 30. +The stateless decoder accepts the wire resolution field without the faulty conversion and +remains responsible for structural validation. + The newer `3.0.0-rc.3`, inspected on 2026-09-05, fixes those calculations but still rejects sub-nanosecond resolutions and dates before the Unix epoch. Upgrading alone does not satisfy -this timestamp contract. The existing stateless decoder accepts the wire resolution field -without applying the faulty conversion, and remains responsible for structural validation. +this timestamp contract. + +#### Structural validation and fixtures The parser validates referenced interfaces, captured/original/snap lengths, and uniqueness of -timestamp options. Input buffers grow only as bytes arrive; a huge declared length on a truncated -stream cannot trigger allocation of that entire declared block. Independent manually encoded -fixtures cover both byte orders and packet-block formats, interface offsets, resolution extremes, -section resets, truncation, and malformed lengths/options. Tests must not encode expected -timestamp layouts with the same dependency writer whose decoder they are validating. +timestamp options. Buffers grow only as bytes arrive: a huge declared length on a truncated +stream must not trigger allocation of that entire block. + +Manually encoded fixtures cover both byte orders and packet-block formats, interface offsets, +resolution extremes, section resets, truncation, and malformed lengths/options. Expected layouts +must not be encoded with the same dependency writer whose decoder the test is validating. ### IP and UDP boundaries -The routing stage and shard-local DNS decoder share one `ParsedUdpDnsMeta` value containing the -validated DNS offset and length. IPv4 Total Length or IPv6 Payload Length first bounds the IP -payload; UDP Length then bounds the DNS payload. Bytes outside either declared boundary, including -Ethernet padding and trailing capture bytes, never reach a DNS decoder. - -By default, IPv4 datagrams with the More Fragments flag or a non-zero fragment offset are skipped. -With `--allow-fragments`, a first IPv4 response fragment may be matched when its DNS header and -entire question fit in the fragment. The metadata bounds DNS bytes to the observed fragment and -marks the response as partial. This is an inference from the prefix, not validation of the full UDP -datagram. A response code is emitted only if there are no additional records or all declared DNS -records fit in the prefix. -With `--full-fragments`, DPP also reconstructs complete IPv4 datagrams across packet batches. This -mode implies `--allow-fragments`; incomplete responses may use the prefix heuristic on capacity -eviction or end of input, and on elapsed capture-time timeout with `--monotonic-capture`. -Fragmented queries require complete reassembly. The reassembler keys -fragments by source, destination, protocol, and IPv4 identification, places payload bytes at the -declared eight-octet offsets, and waits for the first fragment, final fragment, and contiguous -coverage before sending a datagram through the existing UDP/DNS parser. Fragmented IP payload -length must equal the declared UDP Length; mismatches are rejected. Complete datagrams carry -the final fragment's (`MF=0`) capture timestamp; prefix fallbacks retain the first fragment's -timestamp. The reassembler bounds incomplete state -to prevent unbounded capture-driven memory growth. Neither mode reassembles IPv6 fragments. -IPv6 extraction traverses Hop-by-Hop, Routing, Destination Options, Authentication and atomic +Routing and shard-local DNS decoding share one `ParsedUdpDnsMeta` value containing the validated +DNS offset and length. IPv4 Total Length or IPv6 Payload Length first bounds the IP payload; +UDP Length then bounds DNS. Ethernet padding and trailing capture bytes outside either boundary +never reach a DNS decoder. + +#### IPv4 fragmentation modes + +| Mode | Accepted fragment processing | +| --- | --- | +| Default | Skip IPv4 datagrams with More Fragments set or a non-zero fragment offset | +| `--allow-fragments` | Allow inference from a first response fragment containing the complete DNS header and question | +| `--full-fragments` | Reconstruct complete IPv4 datagrams across batches; implies `--allow-fragments` | + +With `--allow-fragments`, metadata bounds DNS bytes to the observed fragment and marks the +response as partial. This is inference from a prefix, not validation of the full UDP datagram. +A response code is emitted only if no additional records exist or all declared DNS records fit +in that prefix. + +With `--full-fragments`, incomplete responses may use prefix inference on capacity eviction or +EOF. Elapsed capture-time timeout can also trigger fallback when `--monotonic-capture` is enabled. +Fragmented queries require complete reassembly. + +The reassembler keys fragments by source, destination, protocol, and IPv4 identification. Payload +bytes use the declared eight-octet offsets. A datagram reaches the existing UDP/DNS parser only +when the first fragment, final fragment, and contiguous coverage are present. Its fragmented IP +payload length must equal UDP Length; mismatches are rejected. + +Complete datagrams use the final fragment's (`MF=0`) capture timestamp. Prefix fallbacks retain +the first fragment's timestamp. Incomplete state is bounded to prevent unbounded memory growth. +Neither fragment mode reassembles IPv6. + +#### IPv6 extension headers + +IPv6 extraction traverses Hop-by-Hop, Routing, Destination Options, Authentication, and atomic Fragment headers with per-header bounds checks. Non-atomic fragments require reassembly and are -skipped. The DNS offset can exceed 65535 after a long valid extension chain. Internal metadata -stores a checked `u16` delta from the minimum 42-byte Ethernet/IPv4/UDP prefix and reconstructs -the absolute offset when accessing packet bytes. A maximum IPv6 frame ends at byte 65589 and -must leave at least 12 DNS bytes, so this delta covers the full supported range. The UDP-bounded -DNS length remains `u16`; routing metadata is 42 bytes on the measured macOS ARM64 target. +skipped. + +#### Compact DNS offsets + +A valid IPv6 extension chain can place DNS beyond byte 65535. Internal metadata stores a checked +`u16` delta from the minimum 42-byte Ethernet/IPv4/UDP prefix and reconstructs the absolute offset +when accessing packet bytes. + +A maximum IPv6 frame ends at byte 65589 and must leave at least 12 DNS bytes, so the delta covers +the supported range. The UDP-bounded DNS length remains `u16`. Routing metadata measures 42 bytes +on the tested macOS ARM64 target. ### DNS QNAME boundary -The standard `hickory` DNS question decoder and the optional custom DNS wire fast path enforce the -same RFC 1035 boundary after name decompression. A QNAME may occupy at most 255 wire octets, -including label-length octets and the terminating root octet. Its escaped presentation form can be -as large as 1003 bytes and remains valid input; DPP preserves it for matching and export rather than -replacing it with an empty name. -Both question decoders reject compression pointers that point forward or overlap the current name. -The fast path's fallback must not become a way to accept a message the semantic decoder rejects. -Ordinary QUERY messages (OPCODE 0) with QDCOUNT greater than one are malformed under RFC 9619 and -are rejected by both decoders. Responses that declare answer or authority records must contain -those complete records even when ARCOUNT is zero; the decoder checks their wire boundaries before -the matcher sees the response. - -If any decompressed QNAME exceeds the wire limit, the entire DNS message is rejected before matcher -or writer handoff. The processing counter `oversized_qname_message_count` increments exactly once -for that message, regardless of its question count, and the JSON report exposes the same DNS-message -unit as `metrics.dns_messages_rejected_oversized_qname`. No question from the rejected message may -enter matcher state or output. +#### Shared decoder rules -### Monotonic timestamp contract +The standard `hickory` question decoder and optional custom wire fast path enforce the same +RFC 1035 limit after decompression: at most **255 wire octets**, including label-length octets +and the terminating root. An escaped presentation form can reach **1003 bytes** and remains +valid; DPP preserves it for matching and export. + +Both decoders also enforce these rules: + +- Reject compression pointers that point forward or overlap the current name. Fast-path fallback + must not accept a message the semantic decoder rejects. +- Reject ordinary QUERY messages (OPCODE 0) with QDCOUNT greater than one, as required by RFC 9619. +- Require complete declared answer and authority records even when ARCOUNT is zero. Their wire + boundaries are checked before matcher handoff. + +#### Oversized-name rejection -The `--monotonic-capture` flag opts into a strict invariant: packet timestamps must be globally -non-decreasing. When enabled: +If any decompressed QNAME exceeds the wire limit, reject the entire DNS message before matcher +or writer handoff. No question from it may enter matcher state or output. + +`oversized_qname_message_count` increments exactly once per rejected message, regardless of +question count. The JSON metric `metrics.dns_messages_rejected_oversized_qname` uses the same +DNS-message unit. + +### Monotonic timestamp contract -- `PacketParser` tracks timestamp regressions and **fails hard** on the first one, instead of - logging a warning and continuing. -- The pipeline can use the batch-maximum timestamp as a global eviction watermark (see - RFC 0004), which keeps matcher memory bounded on long captures. +`--monotonic-capture` requires globally non-decreasing packet timestamps: -When disabled (the default): +| Mode | Timestamp regression | Matcher eviction | +| --- | --- | --- | +| Enabled | `PacketParser` fails on the first regression | Use the batch-maximum timestamp as a global eviction watermark, keeping matcher memory bounded on long captures | +| Disabled (default) | Track regressions and report a post-run warning with the first offending sample | Check per-query timeouts at finalization; long captures with many pending queries use more memory | -- Timestamp regressions are tracked and reported as a post-run warning with the first offending - sample. -- The matcher falls back to per-query timeout checks at finalization time — correct but uses - more memory on long captures with many in-flight queries. +The watermark contract is detailed in [RFC 0004](0004-forward-only-matcher.md#batched-timeout-eviction). ## Why not always use libpcap? -Performance. On a 10 GB classic PCAP, the pure-Rust reader is measurably faster because it -avoids per-packet FFI overhead and gives us direct control over read buffering. The difference -is most visible on high-packet-rate captures where the per-packet cost dominates. +On a 10 GB classic PCAP, the pure-Rust reader is measurably faster because it avoids per-packet +FFI and gives DPP control over read buffering. The difference is most visible when high packet +rates make per-packet cost dominant. -The `pcap-file` crate is currently pinned to a `3.0.0-rc1` release candidate because the stable -line doesn't expose the API we need. This is a known dependency risk — if the RC is abandoned, -we'll need to vendor or fork. +`pcap-file` is pinned to `3.0.0-rc1` because the stable line lacks the required API. This is a +dependency risk: if the release candidate is abandoned, DPP will need to vendor or fork it. ## Why fail-fast on monotonic violations? -Because silent data corruption is worse than a crash. If someone passes `--monotonic-capture` on -a capture that isn't actually monotonic, the batched eviction will retire queries too early and -produce incorrect timeout counts. A hard error on the first regression makes the failure obvious -and actionable. +A timestamp regression can cause batched eviction to retire queries too early and produce +incorrect timeout counts. Failing on the first regression makes a violated assumption visible +instead of silently returning incorrect results. ## Consequences -- Adding support for a new capture format means adding a new `PacketBackend` variant, not - changing the parser interface. -- The pure-Rust reader is the performance-critical path. Changes to it should be benchmarked. -- Stdin support stays within the offline-processing contract and now uses parser-owned stream-native - backends for classic PCAP and PCAPNG so stdin probing does not create a second ingest owner. -- Regular-file fallback compatibility for non-classic formats still relies on `libpcap`. -- The `pcap-file` RC dependency should be revisited when a stable release is available. +- Add new capture formats through a new `PacketBackend` variant; keep the parser interface stable. +- Benchmark changes to the pure-Rust reader: it is the performance-critical path. +- Keep stdin within the offline-processing contract, using parser-owned stream-native backends + for classic PCAP and PCAPNG. +- Retain `libpcap` fallback compatibility for non-classic regular files. +- Revisit the `pcap-file` release candidate when a suitable stable release is available. diff --git a/docs/rfc/0006-adaptive-pipeline.md b/docs/rfc/0006-adaptive-pipeline.md index 9fcc250..6ed4a57 100644 --- a/docs/rfc/0006-adaptive-pipeline.md +++ b/docs/rfc/0006-adaptive-pipeline.md @@ -1,90 +1,86 @@ -# RFC 0006 — Adaptive Pipeline: Staged vs Phase-Parallel Execution +# RFC 0006 — Adaptive pipeline: staged vs phase-parallel execution -Status: Accepted -Date: 2025-10-09 +**Status:** Accepted · **Date:** 2025-10-09 ## Problem -DPP's processing pipeline has natural concurrency: packet parsing, flow routing, shard-local -matching, deterministic aggregation, and async output. On a 16-core machine, dedicating threads -to each stage makes sense — the stages overlap in time and the pipeline stays busy. But on a -2-core machine, reserving threads for routing and aggregation leaves zero cores for the actual -matching work. The pipeline stalls instead of speeding up. +DPP can overlap packet parsing, flow routing, shard-local matching, deterministic aggregation, +and asynchronous output. Dedicated stage threads suit a 16-core host. On a 2-core host, reserving +threads for routing and aggregation leaves no CPU budget for matcher workers. -A single execution model doesn't fit both ends of the hardware spectrum. +A single execution model cannot serve both CPU budgets efficiently. ## Decision -DPP picks its execution model automatically based on the available CPU budget: +Select the execution model automatically from the effective CPU budget: + +| Model | CPU budget | Unit of parallel work | +| --- | --- | --- | +| Staged | Enough CPUs for service threads and matcher workers | Pipeline stages overlap across batches | +| Phase-parallel | Too few CPUs for dedicated service threads | Shards run in parallel within one batch | ### Staged pipeline (high-core hosts) -When the CPU budget is large enough to support dedicated service threads without starving -shard workers, DPP runs a **staged pipeline** with explicit thread roles: +The staged pipeline assigns explicit roles: + +| Role | Runs on | Responsibility | +| --- | --- | --- | +| Parsing/routing | `DPP_Parser` thread | Receive raw packet batches, extract each `CanonicalFlowKey`, and route packets by shard index | +| Matching | `DPP_Matcher_N` workers | Own shard ranges, fully decode DNS, and match within each shard | +| Aggregation | Pipeline's parent thread | Collect all worker results, restore batch-sequence order with `PendingBatchBuffer`, and emit finalized record batches | -- **Parser thread** (`DPP_Parser`) — receives raw packet batches, extracts `CanonicalFlowKey` - per packet, routes packets to the correct matcher worker by shard index. -- **Matcher workers** (`DPP_Matcher_N`) — each owns a range of shards. Receives routed batches, - performs full DNS decode, runs shard-local matching. Workers process batches tagged with a - sequence number to preserve global ordering. -- **Aggregator** (runs on the pipeline's parent thread) — collects results from all workers, - reorders them by batch sequence number using a `PendingBatchBuffer`, and emits finalized - records to the output channel as bounded batched record messages in deterministic order. +Routed batches carry sequence numbers, preserving global ordering even when workers finish out +of order. Record batches reach the output channel in deterministic order. -The staged pipeline uses bounded crossbeam channels between stages. Backpressure propagates -naturally: if matcher workers fall behind, the parser blocks on send; if the output channel -is full, the aggregator blocks. Output channel capacity is measured in batched record messages, -derived from the configured queued-record backlog and the per-message record limit owned by -runtime configuration. +Bounded crossbeam channels provide backpressure. If matcher workers fall behind, the routing +thread blocks on send. If output is full, the aggregator blocks. + +Output capacity is measured in batched record messages. Runtime configuration derives that +capacity from the queued-record backlog and the per-message record limit. ### Phase-parallel pipeline (low-core hosts) -When the CPU budget is too small for dedicated service threads, DPP falls back to a simpler -**phase-parallel** model: +Each batch proceeds through three steps: -- A single processing thread receives packet batches. -- Within each batch, shard-local work is parallelized via Rayon's `par_iter`. -- Results are collected and emitted in shard-index order after each batch. +1. A single processing thread receives the packet batch. +2. Rayon's `par_iter` runs shard-local work in parallel. +3. Results are collected and emitted in shard-index order. -This avoids the overhead of dedicated routing and aggregation threads on machines where those -threads would compete with the actual work. +This avoids dedicating routing and aggregation threads on hosts where those threads would +compete with matcher work. ### How the decision is made -The threshold lives in `src/config.rs` as part of the `ExecutionBudget`. The -`uses_staged_pipeline()` method returns `true` when the effective CPU budget is high enough to -reserve service threads (currently 2: one for parsing/routing, one implicitly for aggregation) -and still leave meaningful worker capacity. The exact threshold is a tuned constant, not a -CLI flag — it's an implementation detail, not a user-facing policy. +[`src/config.rs`](../../src/config.rs) owns the threshold through `ExecutionBudget`. +`uses_staged_pipeline()` selects staged execution when the effective budget can reserve service +threads and still leave meaningful worker capacity. + +Currently, two service roles are reserved: parsing/routing and aggregation. The threshold is a +tuned constant, not a CLI flag. ## Key implementation details -- **Shard count** = `available_cpus × MATCHER_SHARD_FACTOR`. Over-sharding relative to worker - count improves load balance when flow sizes are skewed. -- **Worker-to-shard mapping** is a simple range partition: worker `i` owns shards +- **Shard count:** `available_cpus × MATCHER_SHARD_FACTOR`. Over-sharding relative to worker count + improves load balance when flow sizes are skewed. +- **Worker-to-shard mapping:** worker `i` owns the range `[i * shard_count / worker_count, (i+1) * shard_count / worker_count)`. -- **Batch sequence numbers** ensure the aggregator emits results in the same order regardless - of which worker finishes first. The `PendingBatchBuffer` holds out-of-order results and - releases them only when the next expected sequence is complete. -- **Routing metadata reuse** — the parser stage extracts UDP/DNS metadata once during routing - and passes it alongside the packet to the matcher worker, so the worker doesn't repeat L3/L4 - parsing for DNS question decode. +- **Batch ordering:** `PendingBatchBuffer` holds out-of-order results and releases them only when + the next expected batch sequence is complete. +- **Metadata reuse:** routing extracts UDP/DNS metadata once and passes it with each packet. A + matcher worker reuses it instead of repeating L3/L4 parsing before DNS question decode. ## Why the execution model remains automatic -Because the right choice depends on hardware, not on user preference. A user on a 2-core VM -who forces staged mode gets worse performance, not better. The automatic selection removes a -footgun without losing any capability — on high-core machines, you get the staged pipeline -automatically. Operators can cap the CPU budget with `--threads` or `DPP_THREADS`; the pipeline -model is selected from that effective budget. +Selection follows hardware capacity. Forcing staged execution on a 2-core VM adds contention; +sufficiently large budgets already select it automatically. Operators can cap the effective +budget with `--threads` or `DPP_THREADS`. -If future profiling shows the threshold needs tuning, it's a one-line change in `config.rs`. +Profiling may justify tuning the threshold in `config.rs`; it does not require a user-facing +execution-model switch. ## Consequences -- Both paths must produce identical output for the same input. This is the determinism contract - from RFC 0004 — the execution model is an implementation detail, not a semantic difference. -- Performance changes to either path should be benchmarked on both low-core and high-core - machines. -- The `MATCHER_SHARD_FACTOR` and staged-pipeline threshold are tuning constants. Changes should - be backed by benchmark data, not intuition. +- Both paths must produce identical output for the same input, following the + [RFC 0004 determinism contract](0004-forward-only-matcher.md#invariants). +- Benchmark performance changes on both low-core and high-core machines. +- Back changes to `MATCHER_SHARD_FACTOR` or the staged threshold with benchmark data. diff --git a/docs/rfc/0007-packet-storage-allocation.md b/docs/rfc/0007-packet-storage-allocation.md index eade7ae..f1e2575 100644 --- a/docs/rfc/0007-packet-storage-allocation.md +++ b/docs/rfc/0007-packet-storage-allocation.md @@ -1,73 +1,85 @@ -# RFC 0007 - Packet Storage Allocation Experiments +# RFC 0007 — Packet storage allocation experiments -Status: Accepted - -Date: 2026-08-17 +**Status:** Accepted · **Date:** 2026-08-17 ## Problem -Offline capture ingestion copies every borrowed packet payload into its own `Box<[u8]>` before -the batch crosses a thread boundary. Profiles attributed substantial ingestion cost to allocation, -and the DNS decoder also created temporary per-message vectors. This made batch arenas, slabs, and -inline decoder storage plausible optimizations. +Offline capture ingestion copies each borrowed packet payload into its own `Box<[u8]>` before +crossing a thread boundary. Profiles attributed substantial ingestion cost to allocation, while +the DNS decoder also created temporary per-message vectors. Batch arenas, slabs, and inline +decoder storage were therefore plausible optimizations. + +## Decision + +**Keep the current per-packet owned payload.** The accepted decision rejects these tested +replacements under the current pipeline ownership model: + +- contiguous or segmented batch-wide backing; +- chunk-size or capacity-hint tuning of that backing; +- a small/large storage threshold; +- copying routed packets into a second worker-owned arena; +- inline decoded-question storage solely to reduce allocator calls. + +Batch-wide ownership ties every packet's lifetime to the slowest worker, raising peak RSS. +Splitting ownership after routing removes that coupling but adds another payload copy, reducing +throughput, especially for large DNS frames. Tuning allocation shape shifts this trade-off +without removing it. ## Experiments -The experiments used byte-identical baseline and candidate builds on one macOS ARM64 host in -staged mode. Final screens used 20 balanced, randomized pairs per fixture, no outlier removal, -CSV output to `/dev/null`, kernel peak RSS, and byte-identical output verification. The chunk-size -sweep used 10 position-balanced blocks. Results are paired geometric ratios: speed above `1.0` is -better; RSS below `1.0` is better. +### Measurement protocol + +Baseline and candidate builds were compared on one macOS ARM64 host in staged mode. Final +screens used: + +- 20 balanced, randomized pairs per fixture, with no outlier removal; +- CSV output to `/dev/null` and byte-identical output verification; +- kernel peak RSS. + +The chunk-size sweep used 10 position-balanced blocks. Results are paired geometric ratios. +**Speed above `1.0` is better; RSS below `1.0` is better.** + +### Results | Candidate | Dense speed / RSS | Sparse-large speed / RSS | Decisive result | -|-----------|-------------------|--------------------------|-----------------| +| --- | --- | --- | --- | | One contiguous batch arena | `1.009 / 1.428` | `0.459 / 1.151` | Severe sparse regression | | Segmented 1 MiB arena | `1.032 / 1.295` | `0.821 / 1.034` | Dense RSS and sparse throughput regress | | Best chunk sweep result: 64 KiB | `1.054 / 1.348` | `0.954 / 1.293` | No acceptable throughput/RSS point | | Small/large hybrid with worker compaction | `0.980 / 0.968` | `0.928 / 1.093` | Large DNS fell to `0.799 / 1.375` | | Inline one-question decoder storage | `0.999 / 1.082` | `1.020 / 0.990` | Large DNS fell to `0.894 / 1.059` | -The hybrid did reach `1.181x` on small non-DNS packets, confirming that allocation removal can -improve isolated ingestion. It lost that gain once DNS payload ownership crossed into workers. - -Jemalloc statistics also confirmed the local optimization: inline question storage reduced small -allocation requests from `816,043` to `616,028` (`-24.51%`) and size-class-weighted requested bytes -from `394,657,800` to `369,054,944` (`-6.49%`). End-to-end throughput still regressed on the -large-DNS fixture. Allocation-call count is therefore diagnostic evidence, not an acceptance -metric. +The hybrid reached `1.181x` on small non-DNS packets, confirming that fewer allocations can +improve isolated ingestion. That gain disappeared once DNS payload ownership crossed into workers. -## Decision +Inline question storage also reduced jemalloc's allocation requests: -Keep the current per-packet owned payload. Do not replace it with any of the following under the -current pipeline ownership model: +| Measurement | Baseline | Candidate | Change | +| --- | --- | --- | --- | +| Small allocation requests | `816,043` | `616,028` | `-24.51%` | +| Size-class-weighted requested bytes | `394,657,800` | `369,054,944` | `-6.49%` | -- a contiguous or segmented batch-wide backing; -- chunk-size or capacity-hint tuning of that backing; -- a small/large storage threshold; -- copying routed packets into a second worker-owned arena; -- inline decoded-question storage solely to reduce allocator calls. +End-to-end throughput still regressed on the large-DNS fixture. Allocation-call count is +therefore diagnostic evidence, not an acceptance metric. -Batch-wide ownership couples the lifetime of every packet to the slowest worker and raises peak -RSS. Splitting ownership after routing removes that coupling but introduces another payload copy, -which loses throughput, especially for large DNS frames. Tuning allocation shape only moves the -trade-off; it does not remove it. +## Reconsideration gate -## Reconsideration Gate +Reopen this decision only for an architecture that preserves one payload copy without +batch-wide lifetime coupling. -Reopen this decision only for an architecture that preserves one payload copy without batch-wide -lifetime coupling. The credible direction is to classify borrowed capture bytes before ownership -handoff, then copy accepted DNS packets directly into their final worker-owned storage. That moves -routing into or adjacent to ingestion and therefore requires an explicit ownership-boundary design, -not another container substitution. +The proposed direction is to classify borrowed capture bytes before ownership handoff, then +copy accepted DNS packets directly into their final worker-owned storage. This would move +routing into or adjacent to ingestion and requires an explicit ownership-boundary design. +It is a direction to evaluate, not an accepted replacement. -Any replacement must also provide: +Any replacement must satisfy every requirement: -- byte-identical output and counters; -- a positive dense-workload throughput result and no worse than `0.98x` on sparse and large-DNS - workloads, with paired 95% confidence intervals; -- peak RSS no higher than `1.10x` baseline on every workload; -- staged and phase-parallel coverage on representative production captures, including Linux. +| Requirement | Acceptance criterion | +| --- | --- | +| Correctness | Byte-identical output and counters | +| Throughput | Positive dense-workload result; no worse than `0.98x` on sparse and large-DNS workloads, with paired 95% confidence intervals | +| Memory | Peak RSS no higher than `1.10x` baseline on every workload | +| Coverage | Staged and phase-parallel execution on representative production captures, including Linux | -The measurements above are sufficient to reject the tested designs, but they are not portability -claims: they used synthetic fixtures, macOS ARM64, changing background load, and excluded storage -I/O. +The measurements reject the tested designs but do not establish portability. They used +synthetic fixtures on macOS ARM64, changing background load, and no storage I/O. diff --git a/docs/rfc/0008-matcher-state-expiry.md b/docs/rfc/0008-matcher-state-expiry.md index 0f2c56c..98d00cf 100644 --- a/docs/rfc/0008-matcher-state-expiry.md +++ b/docs/rfc/0008-matcher-state-expiry.md @@ -1,81 +1,106 @@ -# RFC 0008 - Matcher State Expiry Experiments +# RFC 0008 — Matcher state expiry experiments -Status: Accepted - -Date: 2026-08-18 +**Status:** Accepted · **Date:** 2026-08-18 ## Problem -In the default non-monotonic mode, entries that find no counterpart remain until EOF. In monotonic -mode the matcher can evict state during processing, but the current `BTreeMap::retain` path visits -the complete pending map for every batch, giving worst-case `O(state * batches)` work. +In default non-monotonic mode, entries without a counterpart remain until EOF. Monotonic mode +can evict state during processing, but `BTreeMap::retain` visits the complete pending map for +each batch: worst-case `O(state * batches)` work. + +Early eviction is unsafe in default mode because later packets may regress to any timestamp. +For example, a query at `1,000 us`, an unrelated high-timestamp packet, and a later matching +response at `1,100 us` must still match. A watermark queue could expire the query too soon. + +## Decision + +**Keep the current matcher state and eviction behavior.** The accepted decision rejects the +tested expiry replacements: + +| Mode | Retained behavior | Rejected replacement | +| --- | --- | --- | +| Default non-monotonic | Retain potentially matchable state until a match or EOF | An expiry queue without a safe bound on future timestamps | +| Monotonic | Exact `retain`-based eviction | The tested `Arc`/`BTreeSet` index, cadence-gated scans, and workload-tuned adaptive activation | -Early eviction is not safe in the default mode. A later capture-order packet may regress to any -timestamp and still match old state. For example, a query at `1,000 us`, an unrelated packet at a -high timestamp, and a later matching response at `1,100 us` must still match. A watermark-based -queue would incorrectly expire the query before reading the response. +Bounding default-mode state requires a new bounded-lateness, preordering, spilling, or +state-limit contract. An expiry queue alone cannot provide that contract. + +Allocation counts or asymptotic improvement alone do not justify a matcher hot-path change. +End-to-end throughput and peak memory are the acceptance metrics. ## Experiments -The monotonic experiment shared each large matcher identity through `Arc`, retained the existing -identity-indexed map as the source of truth, and added an exact `BTreeSet` expiry index keyed by -`(TimelineKey, identity)`. Matches removed expiry entries eagerly; query expirations were sorted -back into the existing deterministic identity order. +### Candidate and measurement protocol + +The monotonic experiment shared each large matcher identity through `Arc`. The existing +identity-indexed map remained the source of truth; an exact `BTreeSet` expiry index used +`(TimelineKey, identity)` keys. Matches eagerly removed expiry entries. Query expirations were +sorted back into the existing deterministic identity order. + +Measurements used separate baseline and candidate release builds on one macOS ARM64 host, +staged execution with 16 available CPUs, and CSV output to `/dev/null`. Main results use 20 +position-balanced pairs and paired geometric ratios. -Measurements used separate baseline and candidate release builds on one macOS ARM64 host in -staged mode with 16 available CPUs. CSV output went to `/dev/null`. The main results use 20 -position-balanced pairs and paired geometric ratios. Speed above `1.0` is better; RSS and retired -instructions below `1.0` are better. +**Speed above `1.0` is better. RSS and retired instructions below `1.0` are better.** + +### Results | Workload | Speed | Peak RSS | Retired instructions | -|----------|-------|----------|----------------------| +| --- | --- | --- | --- | | 2.0 M monotonic query-only packets, 5 s timeout | `1.1096` (`95% CI 1.0766-1.1436`) | `1.0577` (`1.0455-1.0699`) | `1.1159` (`1.1136-1.1183`) | | 18.0 M representative packets, default mode | `0.9995` (`0.9901-1.0089`) | noisy / inconclusive | `1.0036` (`1.0031-1.0042`) | | 2.0 M high-rate matched packets, monotonic mode | `0.9961` (`0.9863-1.0060`) | noisy / inconclusive | `1.0037` (`1.0030-1.0043`) | -An exploratory timeout sweep placed the workload-dependent crossover between roughly 9 and 14 -batches per timeout: the exact index had no reliable gain at 2 seconds, then reached about `1.069x` -at 3 seconds and `1.117x` at 4 seconds. A cadence-gated `retain` reduced scans but deliberately -retained stale state longer and raised observed RSS. An adaptive index avoided common-case costs -only by adding irreversible, workload-specific activation heuristics. - -The large identity is the central ownership constraint: `DnsNameBuf` is 264 bytes and the matcher -identity is approximately 312 bytes on the measured target. A safe eagerly cancellable secondary -index must either share that identity through another allocation or introduce stable handles and -an arena/reverse index. - -On 2026-09-26, the inline `DnsNameBuf` capacity was separately reduced from 255 to 64 bytes; -the size above describes the original experiment. In five alternating paired runs on one macOS -ARM64 host (`perf` profile, 16-CPU budget), the smaller layout improved processing speed by a -paired geometric mean of `1.095x` on an 18.0 M packet synthetic capture and `1.316x` on a 500,612 -query-only subset. Median peak child RSS fell from 307 to 187 MiB and from 407 to 234 MiB, -respectively. CSV output for the query-only subset was byte-identical. These measurements are -specific to the tested host and captures; they do not establish a gain for every QNAME distribution. +An exploratory timeout sweep placed the workload-dependent crossover at roughly 9–14 batches +per timeout. The exact index had no reliable gain at 2 seconds, then reached about `1.069x` at +3 seconds and `1.117x` at 4 seconds. -## Decision +Two alternatives also fell short: -Keep the current matcher state and eviction behavior. +- A cadence-gated `retain` reduced scans but kept stale state longer and raised observed RSS. +- An adaptive index avoided common-case costs only through irreversible, workload-specific + activation heuristics. -- Default non-monotonic mode retains potentially matchable state until a match or EOF. Bounding it - requires a new bounded-lateness, preordering, spilling, or state-limit contract; an expiry queue - alone is not correct. -- Monotonic mode keeps exact `retain`-based eviction. Do not add the tested `Arc`/`BTreeSet` index, - cadence-gated scans, or workload-tuned adaptive activation. -- Allocation counts or asymptotic improvement alone do not justify a matcher hot-path change; - end-to-end throughput and peak memory are acceptance metrics. +### Identity ownership cost -## Reconsideration Gate +The large identity is the central constraint. In the original experiment, `DnsNameBuf` was +264 bytes and matcher identity was approximately 312 bytes on the measured target. A safe, +eagerly cancellable secondary index must either share the identity through another allocation +or introduce stable handles and an arena/reverse index. + +### Later name-layout change + +On 2026-09-26, a separate change reduced inline `DnsNameBuf` capacity from 255 to 64 bytes. +The sizes above describe the original expiry experiment, not the later layout. + +Five alternating paired runs on one macOS ARM64 host used the `perf` profile and a 16-CPU budget: + +| Synthetic capture | Processing speed, paired geometric mean | Median peak child RSS | +| --- | --- | --- | +| 18.0 M packets | `1.095x` | 307 → 187 MiB | +| 500,612 query-only packets | `1.316x` | 407 → 234 MiB | + +CSV output for the query-only subset was byte-identical. These measurements apply to the tested +host and captures; they do not establish a gain for every QNAME distribution. + +## Reconsideration gate Reopen monotonic expiry indexing for a design that stores each identity once and uses compact, -stable generational handles in an eagerly cancellable queue or time wheel. This is an ownership -change, not a container substitution. Reopen default-mode eviction only together with an explicit -contract that provides a safe lower bound for future timestamps. - -Any replacement must preserve byte-identical output and counters, show at least a `1.05x` gain on -the intended large-state workload, keep the lower 95% speed bound at or above `0.99x` on -representative default and matched workloads, and keep the upper 95% peak-RSS ratio at or below -`1.05x`. It must also cover phase-parallel and staged execution, timestamp boundaries, -response-first matching, cancellation, and production-like Linux captures. - -These measurements reject the tested implementations; they are not portability claims. They used -synthetic fixtures, changing background load, and excluded output storage I/O. +stable generational handles in an eagerly cancellable queue or time wheel. This requires an +ownership change, not merely a different container. + +Reopen default-mode eviction only with an explicit contract that provides a safe lower bound +for future timestamps. + +Any replacement must satisfy every requirement: + +| Requirement | Acceptance criterion | +| --- | --- | +| Correctness | Byte-identical output and counters | +| Large-state throughput | At least a `1.05x` gain on the intended workload | +| Other throughput | Lower 95% speed bound at or above `0.99x` on representative default and matched workloads | +| Memory | Upper 95% peak-RSS ratio at or below `1.05x` | +| Coverage | Phase-parallel and staged execution, timestamp boundaries, response-first matching, cancellation, and production-like Linux captures | + +These measurements reject the tested implementations; they do not establish portability. +They used synthetic fixtures, changing background load, and no output storage I/O. diff --git a/docs/rfc/README.md b/docs/rfc/README.md index 5af1b18..7a7bc94 100644 --- a/docs/rfc/README.md +++ b/docs/rfc/README.md @@ -1,25 +1,29 @@ -# RFC Index +# RFC index -This directory holds architecture decision records for DPP. Each RFC captures a specific -design decision — the problem it solves, the chosen approach, and the trade-offs involved. +Each RFC records an architecture decision: the problem, chosen approach, trade-offs, and +conditions for revisiting it. Start with [DPP Architecture](../architecture.md) for the system +overview; use these records to understand why it works that way. ## Ground rules -- If a decision affects ownership boundaries, SSOT, lifecycle contracts, or benchmark policy, - it belongs here. -- `docs/architecture.md` is the high-level architecture reference. RFCs record the *reasoning* - behind that architecture. -- New RFCs use a sequential numeric prefix and a short descriptive slug. +- Record decisions that affect ownership boundaries, a single source of truth (SSOT), lifecycle + contracts, or benchmark policy. +- Keep the high-level architecture in [architecture.md](../architecture.md) and decision + rationale here. +- Give new RFCs the next sequential numeric prefix and a short descriptive slug. ## Current RFCs -| RFC | Title | Scope | -|--------------------------------------------|-------------------------------------------------|----------------------------------------------------------| -| [0001](0001-architecture-boundaries.md) | Ownership Boundaries and Benchmark Contract | Module ownership, SSOT discipline, benchmark scaffolding | -| [0002](0002-cli-and-runtime-boundaries.md) | CLI and Runtime Bootstrap Boundaries | CLI parsing, env precedence, runtime bootstrap split | -| [0003](0003-allocator-selection.md) | Compile-Time Allocator Selection | Build-time allocator features, validation contract | -| [0004](0004-forward-only-matcher.md) | Forward-Only Matcher and Determinism Contract | Shard-local matching, retry dedup, batched eviction | -| [0005](0005-dual-path-pcap-parsing.md) | Dual-Path PCAP Parsing and Monotonic Timestamps | Pure-Rust vs libpcap backends, monotonic contract | -| [0006](0006-adaptive-pipeline.md) | Adaptive Pipeline: Staged vs Phase-Parallel | Automatic execution model selection by CPU budget | -| [0007](0007-packet-storage-allocation.md) | Packet Storage Allocation Experiments | Rejected arena/slab designs and reconsideration gate | -| [0008](0008-matcher-state-expiry.md) | Matcher State Expiry Experiments | Rejected expiry indexes and safe reconsideration gates | +All RFCs below have **Accepted** status. For experiment records, this means the decision to +retain the existing design was accepted; the tested replacements were rejected. + +| RFC | Decision | Read it for | +| --- | --- | --- | +| [0001](0001-architecture-boundaries.md) | Ownership boundaries and benchmark contract | Module ownership, SSOT, and benchmark inputs | +| [0002](0002-cli-and-runtime-boundaries.md) | CLI and runtime bootstrap boundaries | Argument resolution, environment precedence, and host setup | +| [0003](0003-allocator-selection.md) | Compile-time allocator selection | Cargo features, platform limits, and validation | +| [0004](0004-forward-only-matcher.md) | Forward-only matching and determinism | Flow routing, transaction identity, retries, ordering, and eviction | +| [0005](0005-dual-path-pcap-parsing.md) | Capture parsing and monotonic timestamps | Reader selection, protocol boundaries, and timestamp validation | +| [0006](0006-adaptive-pipeline.md) | Adaptive pipeline execution | Staged and phase-parallel models, selected by CPU budget | +| [0007](0007-packet-storage-allocation.md) | Retain per-packet payload ownership | Rejected storage/decoder allocation experiments and reconsideration criteria | +| [0008](0008-matcher-state-expiry.md) | Retain current matcher expiry behavior | Rejected expiry indexes and safe reconsideration criteria | diff --git a/docs/synthetic-pcap-generator.md b/docs/synthetic-pcap-generator.md index 0763805..3f75b6f 100644 --- a/docs/synthetic-pcap-generator.md +++ b/docs/synthetic-pcap-generator.md @@ -1,26 +1,23 @@ # Synthetic DNS PCAP Generator -`dns-pcap-generator` is a standalone utility that emits classic PCAP with synthetic DNS traffic. -It does not need an input capture at runtime. -It lives in the separate workspace crate `tools/dns-pcap-generator`. -It runs from a fitted profile artifact directory. +`dns-pcap-generator` is a standalone utility in `tools/dns-pcap-generator`. It generates classic +PCAP with synthetic DNS traffic from a fitted profile artifact directory; no input capture is +needed at runtime. -## Goal +The checked-in `server1-jul-2024` profile resembles the broad DNS mix in a representative July 2024 +resolver capture. It uses a large positive-domain catalog sourced from filtered real client DNS +traffic, excluding obviously client-specific names and bank domains. -The checked-in `server1-jul-2024` profile is shaped to resemble the broad DNS mix seen in a -representative July 2024 resolver capture, while deliberately excluding obviously client-specific -names and bank domains. +Traffic includes duplicate query retries, unanswered queries, mixed response codes, and multiple +clients talking to one or more resolvers. Output is deterministic for a fixed `--seed`. -The generator also models: +- [Build and generate a capture](#build-and-generate-a-capture) +- [Understand the traffic model](#understand-the-traffic-model) +- [Use fitted profile artifacts](#use-fitted-profile-artifacts) +- [Maintain the domain catalog](#maintain-the-domain-catalog) +- [Sanitization boundary](#sanitization-boundary) -- duplicate query retries -- unanswered queries -- mixed response codes -- multiple clients talking to one or more resolvers -- deterministic output for a fixed `--seed` -- a large checked-in positive-domain catalog sourced from filtered real client DNS traffic - -## Usage +## Build and generate a capture Requires Rust 1.98.1 or newer; this repository is pinned by `rust-toolchain.toml`. @@ -30,7 +27,9 @@ Build the standalone binary: cargo build -p dns-pcap-generator --release --bin dns-pcap-generator ``` -Generate a five-minute synthetic capture: +### Generate by duration + +Generate a five-minute capture with an explicit rate, client pool, resolver pool, and seed: ```bash ./target/release/dns-pcap-generator \ @@ -43,7 +42,9 @@ Generate a five-minute synthetic capture: --seed 42 ``` -Generate an exact number of logical transactions instead of duration-based traffic: +### Generate a fixed transaction count + +Use `--transactions` for an exact number of logical transactions instead of duration-based traffic: ```bash ./target/release/dns-pcap-generator \ @@ -54,8 +55,9 @@ Generate an exact number of logical transactions instead of duration-based traff --resolvers 2 ``` -Generate from the checked-in fitted profile defaults without overriding `qps`, `clients`, or -`resolvers`: +### Use the fitted defaults + +Omitted `--qps`, `--clients`, and `--resolvers` values inherit the fitted profile defaults: ```bash ./target/release/dns-pcap-generator \ @@ -64,6 +66,69 @@ Generate from the checked-in fitted profile defaults without overriding `qps`, ` --transactions 500000 ``` +The profile also sets calibrated duplicate and timeout rates and duplicate retry multiplicity. +Those settings cannot be overridden at runtime, keeping post-DPP behavior anchored to the fitted +profile. + +## Understand the traffic model + +### Packet format and address pools + +Queries go from the synthetic client pool to the synthetic resolver pool. + +| Property | Model | +| --- | --- | +| Output | Classic little-endian PCAP. | +| Packet layers | Ethernet → IPv4 → UDP → DNS. | +| Synthetic clients | `100.64.0.0/10`. | +| Synthetic resolvers | `172.20.0.0/16`. | +| Client limit | `--clients` is validated against the pool's capacity of 4,161,536 distinct synthetic client IPs. | + +### Timing and retries + +Inter-arrival times follow an exponential distribution around `--qps`. Scheduling uses nanosecond +precision with fractional carry, while classic PCAP stores microsecond timestamps. At high QPS, +multiple packets can therefore share a timestamp without imposing a one-microsecond minimum gap on +the generated rate. + +Duplicate retries use the profile's fitted delay for each retry step. Delays are not necessarily +increasing: the first unanswered steps have long backoff, while later fitted or hypothesized steps +can be much shorter. A response follows the last retry. + +Matched response latency is calibrated from the local `server1_jul_2024.csv` distribution. Most +replies arrive in a few dozen microseconds, with a rare long tail and heavier `ServFail` delays. + +The generator streams output and keeps only future scheduled packets in memory, which scales much +better than materializing the whole capture before sorting. + +### DNS answers + +| Successful query | Response contents | +| --- | --- | +| `A`, `AAAA`, `HTTPS`, `SVCB` | Syntactically valid DNS answers. | +| `NS` for the root domain (`.`) | A valid root-server target selected from `a.root-servers.net` through `m.root-servers.net`. | +| `TXT`, `SRV`, `CNAME`, `MX` | May return `NOERROR` with zero answers: an intentional NODATA-style simplification. | + +## Use fitted profile artifacts + +The generator loads `fitted-generator.toml` from `--profile-dir`, verifies the referenced catalog's +digest, and uses the artifact directory as its runtime source of truth. + +The checked-in `server1-jul-2024` profile contains `fitted-generator.toml`. Its `catalog_path` points +to the workspace-level `tools/dns-pcap-generator/catalog_data.tsv`, keeping one reviewable copy of +the sanitized catalog instead of duplicating it inside the profile directory. + +### Validation and errors + +Runtime failures use typed CLI errors, with stable top-level messages and source chains for invalid +arguments and I/O failures. In particular: + +- Catalog rows with zero weights or DNS names longer than 255 wire bytes are rejected. +- Fitted profiles are rejected if `duplicate_max` is below every configured retry count. +- Generation returns an error if a packet timestamp exceeds the classic PCAP 32-bit seconds range. + +## Maintain the domain catalog + Regenerate the checked-in workspace catalog TSV from a local CSV: ```bash @@ -74,73 +139,29 @@ cargo run -p dns-catalog-builder --release -- \ ``` The catalog builder writes and syncs a separate temporary file beside the destination before -replacing the output. Concurrent builds targeting the same path cannot truncate each other's -temporary files; the last successful replacement supplies the complete catalog. Ordinary I/O -errors preserve the previous output and trigger best-effort removal of that build's temporary -file. A process crash or a filesystem cleanup error can leave a temporary file behind. - -The generator loads `fitted-generator.toml` from `--profile-dir`, verifies the referenced -`catalog_data.tsv` digest, and then uses the artifact directory as the runtime source of truth. -The checked-in `server1-jul-2024` profile points its `catalog_path` at the workspace-level -`tools/dns-pcap-generator/catalog_data.tsv`, so the catalog remains a single reviewable source of -truth instead of being duplicated inside the profile directory. Runtime failures are reported -through typed CLI errors, so invalid arguments and I/O failures surface with stable top-level -messages and source chains. -Catalog rows with zero weights or DNS names longer than 255 wire bytes are rejected, as are fitted -profiles whose `duplicate_max` is below every configured retry count. Generation returns an error -if a packet timestamp exceeds the classic PCAP 32-bit seconds range. - -## Model - -- The output is classic little-endian PCAP with Ethernet + IPv4 + UDP + DNS packets. -- Queries go from a synthetic client pool in `100.64.0.0/10` to a synthetic resolver pool in - `172.20.0.0/16`. -- `--clients` is validated against the distinct address capacity of that client pool, which is - 4,161,536 synthetic client IPs. -- Inter-arrival times are sampled from an exponential distribution around the configured `--qps`. - The generator schedules them at nanosecond precision with fractional carry. Classic PCAP stores - microsecond timestamps, so at high QPS multiple packets can share a timestamp without imposing - a one-microsecond minimum gap on the generated rate. -- Duplicate retries use the profile's fitted delay for each retry step. These delays are not - necessarily increasing: the first unanswered steps have long backoff, while later fitted or - hypothesized steps can be much shorter. A response follows the last retry. -- Matched response latency is calibrated from the local `server1_jul_2024.csv` distribution: - most replies land in a few dozen microseconds, with a rare long tail and heavier `ServFail` - delays. -- Successful `A`, `AAAA`, `HTTPS`, and `SVCB` responses include syntactically valid DNS answers. -- Successful `NS` queries for the root domain (`.`) are allowed and return a valid root-server - target selected from the current `a.root-servers.net` through `m.root-servers.net` set. -- `TXT`, `SRV`, `CNAME`, and `MX` questions may return `NOERROR` with zero answers; that is an - intentional NODATA-style simplification. -- The checked-in runtime profile carries calibrated duplicate/timeout rates and duplicate retry - multiplicity. Those knobs are not user-overridable at runtime, which keeps post-DPP behavior - anchored to the fitted profile. -- The positive-domain catalog stays in `tools/dns-pcap-generator/catalog_data.tsv`, and the - checked-in profile references it via `catalog_path`. This avoids a second source of truth for - the sanitized catalog. -- Omitted `--qps`, `--clients`, and `--resolvers` values inherit the fitted profile defaults. - -## Sanitization Boundary +replacing the output: -The checked-in positive-domain catalog is curated and validated to exclude: +- Concurrent builds targeting the same path cannot truncate each other's temporary files. The last + successful replacement supplies the complete catalog. +- Ordinary I/O errors preserve the previous output and trigger best-effort removal of that build's + temporary file. +- A process crash or filesystem cleanup error can leave a temporary file behind. -- `android.clients.*` -- push-courier / Apple-device routing names -- `_dns.resolver.arpa` -- `.local`, `.lan`, `.home.arpa` -- Banking domains -- labels that look like long unique numeric or hex identifiers +When changing the traffic shape, keep the fitted profile, catalog, and tests in sync so the +sanitization boundary remains explicit. -Hypothesis: -The current “client-specific” detector is a conservative heuristic, not a formally complete -classifier. It is designed to block obvious device/user-specific names without requiring access to -the original capture at runtime. +## Sanitization boundary -## Operational Notes +The checked-in positive-domain catalog is curated and validated to exclude: + +| Category | Exclusions | +| --- | --- | +| Client and device routing | `android.clients.*`; push-courier / Apple-device routing names. | +| Resolver discovery | `_dns.resolver.arpa`. | +| Local names | `.local`, `.lan`, `.home.arpa`. | +| Sensitive domain category | Banking domains. | +| Identifier-like labels | Long unique numeric or hex identifiers. | -- The generator streams output and keeps only future scheduled packets in memory, so it scales much - better than materializing the whole capture before sorting. -- The checked-in `server1-jul-2024` profile includes `fitted-generator.toml` and intentionally - reuses the workspace catalog TSV instead of carrying a second copy of the catalog. If the traffic - shape needs to change, keep the fitted profile and the catalog/tests in sync so the sanitization - boundary remains explicit. +**Hypothesis:** The current client-specific detector is a conservative heuristic, not a formally +complete classifier. It is designed to block obvious device/user-specific names without requiring +the original capture at runtime.