Feature/protocol correctness audit

Cargo.toml

@@ -2,7 +2,7 @@
 
 [package]
 name = "dpp"
-version = "0.3.1"
+version = "0.3.2"
 edition = "2024"
 rust-version = "1.97.1"
 build = "build.rs"  # Custom build script to include version information from git
@@ -47,6 +47,7 @@ opt-level = 3
 [dependencies]
 pcap = "2.5"                      # Library for reading pcap files
 pcap-file = "3.0.0-rc1"           # Pure-Rust parser for classic pcap and pcapng files
+byteorder_slice = "3.0"          # Stateless pcapng block decoding without upstream timestamp conversion
 rayon = "1.12"                    # Parallelism library for multithreaded processing
 hickory-proto = { version = "0.26", default-features = false, features = ["std"] } # DNS protocol parsing library
 csv = "1.4"                       # CSV serialization and deserialization

README.md

@@ -245,9 +245,15 @@ Unsupported non-PCAP/non-PCAPNG stream magic on `stdin` is rejected explicitly i
 hidden temp-file or second ingest path.
 
 Repeated pending queries that share the same match identity (`id`, `name`, client IP, client port,
-resolver IP, and `query_type`) inside the configured match window (`1200ms` by default) are
+resolver IP, `query_type`, QCLASS, and OPCODE) inside the configured match window (`1200ms` by default) are
 deduplicated to the earliest canonical query. Deduplicated retries increment a separate counter and
-do not emit extra timeout or matched records.
+do not emit extra timeout or matched records. In default mode, pending retry timestamps are retained
+so an earlier timestamp arriving in a later batch can split pending groups without widening the
+match window. Finalized transactions are never reopened. With `--monotonic-capture`, no retry
+history is needed or allocated.
+
+DNS QR determines query/response direction, including valid exchanges with UDP port 53 on both
+endpoints. QCLASS and OPCODE distinguish transactions internally and do not add output columns.
 
 For QNAME matching, DPP preserves the observed presentation-form name bytes and does not lowercase
 them before building matcher identity keys. This is a deliberate Community Edition trade-off, not
@@ -280,6 +286,10 @@ the absent `response_timestamp`; `response_code` is absent because no DNS respon
 extended RCODE `16` is exported as `EDNS_BADVERS`; `TSIG Failure` is used for code `16` only when a
 TSIG RR error field supplies that status.
 
+Unknown query types retain their wire number as `TYPE<number>` (for example, `TYPE65400`),
+and unknown response codes use their decimal number (for example, `64`). CSV and Parquet
+preserve these values instead of collapsing distinct codes into an `Unknown` label.
+
 With `--report-format json` or `DPP_REPORT_FORMAT=json`, DPP suppresses routine `info`/`warn`
 reporting and emits one final JSON summary object to `stdout`. This keeps CSV or Parquet output in
 its own file while making the end-of-run report easy to capture and parse.
@@ -493,7 +503,9 @@ Additional notes:
 ## Limitations
 
 - **UDP/53 only:** DPP currently processes DNS traffic over UDP port 53 only.
-- DPP does not reassemble IPv4 fragments, so fragmented IPv4 datagrams are skipped.
+- DPP does not reassemble IPv4 or IPv6 fragments. IPv6 Hop-by-Hop, Routing, Destination Options,
+  Authentication and atomic Fragment headers are traversed before UDP; fragments requiring
+  reassembly, ESP and IPv6 jumbograms are unsupported. Flow identity uses observed IP endpoints.
 - If capture parsing fails after processing begins, DPP flushes valid partial (not atomic) output
   from complete accepted batches and exits with an error; pending queries are not emitted as timeouts.
 - **PCAPNG support level:** DPP supports PCAPNG on stream input and via `libpcap` on regular-file fallback paths, but the performance-critical pure-Rust fast path remains focused on classic PCAP.

benches/README.md

@@ -7,6 +7,8 @@ This directory is the canonical location for repeatable performance measurements
 - `benchmark.sh`: end-to-end benchmark harness for the `dpp` binary.
 - `allocator-benchmarking.md`: canonical build-and-measure protocol for global allocator
   comparisons.
+- `protocol-correctness-audit.md`: final validation, restored throughput and measured memory costs
+  of the 2026-09-05 protocol and matcher fixes, with earlier experiments identified separately.
 
 ## Safety Contract
 

benches/protocol-correctness-audit.md

@@ -0,0 +1,130 @@
+# Protocol correctness: final validation and performance
+
+Updated 2026-09-05 for implementation commit `8881539` on
+`feature/protocol-correctness-audit`.
+
+The final implementation retains the protocol and matcher corrections and restores throughput
+on the selected 18-million-packet capture. Default mode slightly exceeds the pre-audit version
+in the final comparison; monotonic mode is at approximately the same level. The workspace suite
+passes **254 tests**, and full CSV output is byte-identical to the correctness-fixed version in
+both modes and with both DNS decoders. Peak RSS is higher, approximately 288–301 MiB in these runs.
+
+## Current performance
+
+Each comparison below used ten position-balanced pairs on one macOS ARM64 host. Speed ratios
+are the median of paired reference-time / final-time ratios; values above 1 indicate improvement.
+The two reference comparisons are separate measurement series, so their absolute timings and RSS
+must not be combined as if they were one run.
+
+Against the pre-audit version, `3bb45397baff03a6c7043d2e6cff1508f0161b39`:
+
+| Configuration | Pre-audit median (s) | Final median (s) | Paired speed ratio | Median peak RSS, pre-audit / final (MiB) |
+|---|---:|---:|---:|---:|
+| Default | 0.9807 | 0.9677 | 1.0134 | 217.7 / 298.4 |
+| Monotonic | 0.9580 | 0.9627 | 0.9962 | 221.3 / 287.9 |
+
+Default throughput is about 1.3% higher in this series. The roughly 0.4% monotonic difference is
+within the observed run-to-run variation: paired speed ratios ranged from 0.9687 to 1.0394.
+These results establish local recovery on this capture, not a throughput or portability guarantee.
+
+Against the correctness-fixed version before optimization, `2ef34a4`:
+
+| Configuration | Fixed median (s) | Final median (s) | Paired speed ratio | Median peak RSS, fixed / final (MiB) |
+|---|---:|---:|---:|---:|
+| Default | 1.0273 | 0.9758 | 1.0484 | 209.8 / 301.4 |
+| Monotonic | 0.9980 | 0.9723 | 1.0290 | 219.0 / 288.4 |
+
+The optimizations improve throughput by about 4.8% and 2.9% respectively in this confirmation
+series. Higher occupied pipeline backlog and allocator retention are hypotheses for the RSS
+increase; their contributions were not isolated. Queue bounds and payload ownership are unchanged.
+
+## Final implementation
+
+- Routing metadata is **42 bytes on the measured host**. A checked `u16` offset is relative to
+  the minimum 42-byte Ethernet/IPv4/UDP prefix. This still represents DNS starts above 65535:
+  a maximum IPv6 frame ends at byte 65589 and must leave at least 12 DNS bytes. IP/UDP boundaries,
+  QR direction, QCLASS/OPCODE identity and compression validation remain enforced.
+- Flow routing collects the existing tuple's `Hash` writes into a 64-byte stack buffer and
+  invokes SeaHash once over the bytes. Integer encoding, seeds and the complete digest are
+  unchanged, preserving shard assignment and output order. If the input exceeds the buffer
+  capacity, its contents are replayed into the original streaming hasher. No additional persistent
+  flow state is kept.
+- Each pending canonical query owns one optional retry-history pointer. Default mode allocates
+  history when a pending retry arrives; later retries can grow its vector. Replacing a canonical
+  query with an earlier attempt regroups that identity's unresolved attempts in `O(n log n)`.
+  Monotonic mode allocates no retry history. Memory on duplicate-heavy captures therefore depends
+  on unresolved attempts; this capture's low duplicate rate does not bound that workload.
+- PCAPNG stdin retains parser-owned framing, one section-local interface table and exact signed
+  timestamp conversion, including binary/decimal resolution and offsets. Conversion includes
+  `u128` division per packet; the reusable block buffer grows with received bytes. Regular-file
+  PCAPNG still uses libpcap, and classic PCAP retains its pure-Rust streaming path.
+- Known exported DNS type/status labels remain borrowed strings; numeric fallback allocates for
+  unknown values. CSV and Parquet share the same canonical formatting contract.
+
+`build.rs` was reviewed and left unchanged. It requests executable stack size and supplies build
+metadata. Optimization levels, LTO and codegen units belong to `Cargo.toml`; no compiler or linker
+settings were changed during performance recovery.
+
+## Validation
+
+`cargo test --workspace --offline --locked` passes 254 tests. Coverage includes 30 capture-reader
+tests, 13 DNS parser regressions, a 720-permutation matcher check, both pipeline execution models,
+and CSV/Parquet preservation of unknown codes. Four differential hashing tests compare complete
+64-bit digests for address families, ports, 4096 deterministic flows, integer encodings, buffer
+boundaries, streaming fallback and repeated `finish` calls.
+
+`cargo fmt --all -- --check` and the CI Clippy command pass:
+
+```sh
+cargo clippy --all-targets --no-deps --offline --locked -- \
+  -D warnings \
+  -A clippy::too_many_arguments \
+  -A clippy::type_complexity \
+  -A clippy::large_enum_variant
+```
+
+Every final comparison preserved packet/query/response/duplicate/match/timeout counters and
+average matched RTT. The selected capture contains 18,006,441 packets, 9,013,611 queries,
+13,610 duplicates, 8,992,830 responses and matched pairs, and 7,171 timeouts.
+
+Full CSV output was compared against `2ef34a4` with each combination of monotonic mode and
+`--dns-wire-fast-path`: 854,848,168 bytes per output, with identical SHA-256 within each pair.
+Both decoders produce the same output for a given mode. Default and monotonic modes have different
+row ordering by their existing timeout-emission contracts.
+
+| Mode | SHA-256, both reference and final |
+|---|---|
+| Default | `02db37f51a60a84000b457c47339fb87bd725071c4e08899f3e8b90b430f9fc9` |
+| Monotonic | `08f435bc3aa241615937e3c08b4c089570d5e3dbe9fb245a46c8e9bd3d680bc0` |
+
+## Measurement method
+
+The explicitly selected input was `synthetic/server1-like.pcap`. All compared binaries were built
+with Rust 1.97.1 using `cargo build --profile perf`, default jemalloc and no additional RUSTFLAGS.
+This profile inherits release's `opt-level=3`, LTO and one codegen unit, and disables overflow
+checks for trusted benchmark input; the production release profile keeps overflow checks enabled.
+
+The host exposed 16 CPUs, giving 14 matcher workers in staged mode. Affinity was disabled, CSV
+output went to `/dev/null`, and bounded output settings were identical. Runs alternated reference /
+final and final / reference order, with an excluded warmup pair per mode and no subsequent outlier
+removal. Timings include process startup and shutdown. The capture was read from the filesystem
+cache, so these results do not measure cold-storage performance. Peak RSS was collected per child
+with `wait4`. See [README.md](README.md) for the benchmark scaffold and explicit-input contract.
+
+## Earlier measurements and discarded experiments
+
+Before optimization, the correctness fixes showed about 5.6% lower default throughput and 4.8%
+lower monotonic throughput in the initial comparison. Those figures describe the intermediate
+`2ef34a4` implementation, not the current version. Compact relative offsets and buffered flow
+hashing were subsequently retained on measurement evidence.
+
+Short four-pair screens did not justify retaining query-helper inlining/cold regrouping, CSV
+formatter inlining, a fused borrowed matcher lookup, or a new packed hash input. The final code
+contains none of those experimental changes. CPU sampling identified packet routing and its
+streaming hash calls as the limiting stage on this capture.
+
+An earlier PCAPNG-reader check compared the pre-audit reader with the correctness-fixed reader
+using the first 1,000,000 packets re-encoded as little-endian Enhanced Packet Blocks. Three pairs
+measured 0.0780 / 0.0787 seconds, with identical processing counters. The roughly 80 ms runs were
+too short to establish a timing change. This is reader-validation history, not a final PCAPNG
+throughput benchmark.

docs/architecture.md

@@ -83,15 +83,26 @@ and uses the remaining CPU budget for shard workers. Routed DNS packets also car
 metadata into shard workers so the worker path can reuse the first L3/L4 parse instead of
 repeating it for full DNS question decoding.
 
+Flow routing buffers the existing tuple's `Hash` writes on the stack and runs SeaHash once over
+the resulting bytes. The integer encoding and digest remain identical to the streaming hasher,
+so shard assignment and output order are preserved. If a future tuple encoding exceeds the
+buffer, routing replays its bytes into the streaming hasher; this adds no persistent flow state
+or second encoding of canonical addresses and ports.
+
 ## Core Components
 
 - `src/packet_parser.rs`
   Reads packets from offline captures and exposes them to the rest of the pipeline. Classic PCAP
   files use a pure-Rust streaming reader; regular-file PCAPNG and other non-classic formats
   currently fall back to libpcap. EOF-terminated stdin capture streams are also supported through
   parser-owned stream-native readers: classic PCAP stdin uses the same pure-Rust fast path family
-  as classic file input, and PCAPNG stdin uses a pure-Rust pcapng reader so stdin probing does not
-  need a temp-file or second ingest owner. Unsupported stdin stream magic is rejected explicitly.
+  as classic file input, and PCAPNG stdin owns block framing and one section-local interface table.
+  PCAPNG uses the dependency's stateless structural validation, while the parser converts raw
+  high/low timestamp words using the interface's full binary/decimal resolution and signed offset.
+  Conversion scales the complete counter before rounding to microseconds and saturates only the
+  final signed timestamp. It never invokes the dependency's stateful timestamp conversion.
+  Buffers grow with received block bytes rather than untrusted declared lengths. Stdin probing
+  needs no temp-file or second ingest owner. Unsupported stdin stream magic is rejected explicitly.
   The pure-Rust classic-PCAP reader still relies on the upstream `pcap-file` `3.0.0-rc1` release
   candidate until a stable line with the required functionality is available.
   The parser can also enforce globally monotonic capture timestamps for the optional batched
@@ -143,10 +154,19 @@ repeating it for full DNS question decoding.
   shard-local DNS decode, but that reuse must stay within the same ownership boundary so packet
   parsing does not gain a second source of truth for IP/port extraction. That metadata owns the
   exact DNS byte range validated against IPv4 Total Length or IPv6 Payload Length and then UDP
-  Length; capture padding and trailing IP payload cannot extend the DNS slice. Fragmented IPv4
-  datagrams are skipped because this boundary has no IP reassembly stage. The optional runtime flag
+  Length; capture padding and trailing IP payload cannot extend the DNS slice. A compact relative
+  offset preserves DNS starts above 65535 without widening every packet's routing metadata: the
+  offset is measured from the minimum Ethernet/IPv4/UDP header length, and construction checks
+  that it fits. DNS QR determines
+  direction and the canonical client/resolver flow, including exchanges with UDP port 53 on both
+  ends. IPv6 Hop-by-Hop, Routing, Destination Options, AH and atomic Fragment headers are traversed
+  within the declared payload boundary. Non-atomic IPv6 and fragmented IPv4 datagrams are skipped
+  because this boundary has no IP reassembly stage. The optional runtime flag
   `--dns-wire-fast-path` may enable a custom question-only wire fast path, but `hickory` remains
-  the semantic fallback for rare DNS messages that the fast path does not accept. Both paths accept
+  the semantic fallback for rare DNS messages that the fast path does not accept. Compression
+  pointers must target prior, nonoverlapping names; enabling the fast path must not weaken that
+  validation. TSIG status extraction consumes and bounds-checks the declared Other Data, including
+  the six-byte server time in BADTIME responses. Both paths accept
   decompressed wire QNAMEs up to the RFC 1035 limit of 255 octets, including label-length octets and
   the terminating root octet. A valid name can expand to 1003 bytes in escaped presentation form
   and must remain distinct through matching and export. If any QNAME exceeds the wire limit, the
@@ -283,9 +303,17 @@ The DNS matcher must preserve these invariants:
 - Routing and in-flight matching use the original observed client IP, client port, and resolver IP.
   The resolver remains internal; deterministic client-IP pseudonymization is applied exactly once
   when the matcher constructs a finalized `DnsRecord`.
+- DNS ID, observed QNAME, QTYPE, QCLASS and OPCODE also distinguish in-flight identities. QCLASS,
+  OPCODE and resolver identity remain internal and do not change the exported record schema.
 - Repeated pending queries with the same match identity inside the configured timeout window
   (`1200ms` by default) are deduplicated to the earliest canonical query. Later duplicates are
   counted separately and must not create extra matched or timeout records.
+- In default mode, each pending canonical query owns any retry timestamps needed to regroup
+  unresolved attempts when an earlier query arrives across a batch boundary. Regrouping uses
+  earliest-first timeout windows for that identity, so chains of regressions cannot transitively
+  swallow attempts outside the final canonical window. Already finalized transactions are never
+  reopened. Monotonic mode allocates no retry history. This adds one optional pointer per pending
+  query and storage for default-mode retries; only earlier-canonical replacements regroup attempts.
 - Match identity preserves the observed presentation-form QNAME bytes instead of lowercasing them.
   This is a deliberate Community Edition trade-off, not a protocol guarantee. RFC 4343 defines
   ASCII label comparison as case-insensitive, and a valid response is allowed to differ from the