Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
50 changes: 25 additions & 25 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -1,24 +1,21 @@
# Contributing

Thank you for considering a contribution to DPP.
DPP welcomes contributions that improve correctness, determinism, maintainability, portability,
documentation, test coverage, and measurable performance within the Community Edition scope.

Community contributions are welcome, especially when they improve correctness, determinism,
maintainability, portability, documentation, test coverage, and measurable performance on the
community-supported feature set.
## Before you start

## Before You Start

- Read [README.md](README.md) for the product scope and operator-facing contract.
- Read [docs/architecture.md](docs/architecture.md) before changing runtime behavior or ownership
boundaries.
- Read [docs/rfc/README.md](docs/rfc/README.md) for architecture decisions and long-lived design
context.
- Read [benches/README.md](benches/README.md) before making or claiming performance changes.
| Before changing… | Read |
| --- | --- |
| Product behavior or supported features | [README](README.md) |
| Runtime behavior or ownership boundaries | [Architecture](docs/architecture.md) |
| A long-lived design decision | [RFC index](docs/rfc/README.md) |
| Performance, or making a performance claim | [Benchmark contract](benches/README.md) |

For larger changes, please open an issue or start a discussion before investing in a large patch.
Early alignment is the easiest way to avoid rework.

## Contribution Guidelines
## Contribution guidelines

- Preserve ownership boundaries.
- Do not introduce a second source of truth.
Expand All @@ -29,34 +26,37 @@ Early alignment is the easiest way to avoid rework.
- Call out hot-path performance risk when touching parser, matcher, pipeline, or writer code.
- Mark unsupported assumptions as hypotheses.

## Testing Expectations
## Testing expectations

At minimum:

- Run targeted tests for the code you touched.
- Run additional checks that match the change type.
- Include benchmark evidence for hot-path performance claims.

Examples:

- Parser or matcher changes should include targeted `cargo test` coverage.
- Writer changes should verify output compatibility and shutdown behavior.
- Benchmark harness changes should be validated with `bash -n benches/benchmark.sh` and a dry run.
| Change | Additional validation |
| --- | --- |
| Parser or matcher | Targeted `cargo test` coverage. |
| Writer | Output compatibility and shutdown behavior. |
| Benchmark harness | `bash -n benches/benchmark.sh` and a dry run. |

## Rebuilding a Missing Release Asset
## Rebuilding a missing release asset

Once the corrected release workflow is on `main`, rebuild one platform from an existing tag:

```bash
gh workflow run rust.yml --ref main -f tag=v0.4.0 -f runner=ubuntu-22.04
```

This runs the current workflow but checks out the source from `refs/tags/v0.4.0`. It does not
move the tag or overwrite existing release assets. The other runner choices are `ubuntu-24.04`
and `ubuntu-24.04-arm`. Tag pushes still build all three platforms. Re-running an older failed
run uses its original workflow revision, so it does not pick up a workflow fix committed later.
This runs the current workflow but checks out `refs/tags/v0.4.0`. It does not move the tag or
overwrite existing release assets.

- Runner choices are `ubuntu-22.04`, `ubuntu-24.04`, and `ubuntu-24.04-arm`.
- Tag pushes build all three platforms.
- Re-running an old failed run uses its original workflow revision and does not pick up a later
workflow fix.

## Community and Commercial Scope
## Community and Commercial scope

DPP Community Edition and DPP Commercial Edition intentionally have different scopes.

Expand Down
756 changes: 435 additions & 321 deletions README.md

Large diffs are not rendered by default.

64 changes: 28 additions & 36 deletions benches/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,48 +2,36 @@

This directory is the canonical location for repeatable performance measurements.

Use [benchmark.sh](benchmark.sh) for end-to-end measurements, or choose a focused guide below.

## Files

- `benchmark.sh`: end-to-end benchmark harness for the `dpp` binary.
- `allocator-benchmarking.md`: canonical build-and-measure protocol for global allocator
comparisons.
- `protocol-correctness-audit.md`: final validation, restored throughput and measured memory costs
of the 2026-09-05 protocol and matcher fixes, with earlier experiments identified separately.
- `dns-compression.md`: configurable compression limits, message-local suffix caching,
repeated performance measurements, RR-loop inlining investigation and reproduction instructions.
- `dns-compression-benchmark.py` and `dns-compression/`: synthetic end-to-end and isolated
scanner benchmarks for short names, shared suffixes and deep pointer chains.
| File or directory | Purpose |
| --- | --- |
| [benchmark.sh](benchmark.sh) | End-to-end benchmark harness for the `dpp` binary. |
| [allocator-benchmarking.md](allocator-benchmarking.md) | Canonical build-and-measure protocol for global allocator comparisons. |
| [protocol-correctness-audit.md](protocol-correctness-audit.md) | Final validation, restored throughput and measured memory costs of the 2026-09-05 protocol and matcher fixes; earlier experiments are identified separately. |
| [dns-compression.md](dns-compression.md) | Compression limits, message-local suffix caching, repeated measurements, RR-loop inlining and reproduction instructions. |
| [dns-compression-benchmark.py](dns-compression-benchmark.py) and [dns-compression/](dns-compression/) | Synthetic end-to-end and isolated scanner benchmarks for short names, shared suffixes and deep pointer chains. |

## Safety Contract

- The benchmark input PCAP must be provided explicitly with `--pcap` or `DPP_BENCH_PCAP`.
- The script does not upload data or fetch remote inputs.
- Results are written under the selected output directory, which defaults to `target/benchmarks/...`.
- Benchmark outputs can contain derived data from the input capture, so avoid committing or sharing them unless that is intended for the dataset.
- Benchmark outputs can contain derived data from the input capture. Avoid committing or sharing
them unless that is intended for the dataset.
- The script exits non-zero if any benchmarked run fails.

## What It Measures

`benchmark.sh` records:
## Setup

- build profile
- output format
- execution budget label
- bonded channel capacity
- run number
- whether the run used `--silent`
- whether the row contains full metrics or wall-clock-only metrics
- current git SHA and benchmarked binary path
- `rustc` version and verbose compiler metadata in `metadata.txt`
- total wall-clock time
- processing speed parsed from application logs
- final writer shutdown tail parsed from application logs
- exit code
- log file path
- output file path
For trusted benchmark runs, prefer the dedicated `perf` Cargo profile. It inherits from `release`
but disables overflow checks so benchmark numbers reflect the faster arithmetic path without
changing the default production build.

For `--silent` runs, the script records wall-clock and exit status, but log-derived throughput and
shutdown-tail fields are intentionally left blank.
DPP auto-sizes execution from all available CPUs. The harness does not support a runtime
thread-count sweep. To compare smaller CPU budgets, limit CPU availability externally with your
platform tooling and run the harness once per environment.

## Example

Expand All @@ -59,10 +47,14 @@ bash benches/benchmark.sh \
Metrics are written to `metrics.csv` inside the chosen benchmark output directory. Run-level
provenance is written to `metadata.txt` in the same directory.

Note: DPP now auto-sizes execution from all available CPUs. The harness no longer supports a
runtime thread-count sweep; if you need to compare smaller CPU budgets, limit CPU availability
externally with your platform tooling and run the harness once per environment.
## What It Measures

For trusted benchmark runs, prefer the dedicated `perf` Cargo profile. It inherits from `release`
but disables overflow checks so benchmark numbers reflect the faster arithmetic path without
changing the default production build.
| Category | Recorded fields |
| --- | --- |
| Run configuration | Build profile, output format, execution budget label, bonded channel capacity, run number and whether `--silent` was used. |
| Provenance | Current git SHA, benchmarked binary path, `rustc` version and verbose compiler metadata in `metadata.txt`. |
| Measurements | Total wall-clock time; processing speed and final writer shutdown tail parsed from application logs. |
| Result and artifacts | Exit code, log file path, output file path and whether the row has full metrics or wall-clock-only metrics. |

For `--silent` runs, the script records wall-clock time and exit status. Log-derived throughput
and shutdown-tail fields are intentionally left blank.
35 changes: 17 additions & 18 deletions benches/allocator-benchmarking.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,7 @@ This document records the canonical protocol for comparing DPP global allocator

## Goal

Allocator experiments must answer two questions on the same representative capture:

- does throughput improve or regress;
- does peak RSS improve or regress.
Compare both throughput and peak RSS on the same representative capture.

Allocator changes are meaningful only when compared under the same binary flags, CPU availability,
input capture, and output format.
Expand All @@ -16,15 +13,23 @@ input capture, and output format.

Exactly one allocator feature must be enabled at build time.

- default build: `allocator-jemalloc`
- alternative builds:
- `allocator-system`
- `allocator-mimalloc`
- `allocator-tcmalloc` on Linux `x86_64` and Linux `aarch64` only
| Build | Feature | Platform restriction |
| --- | --- | --- |
| Default | `allocator-jemalloc` | — |
| Alternative | `allocator-system` | — |
| Alternative | `allocator-mimalloc` | — |
| Alternative | `allocator-tcmalloc` | Linux `x86_64` and Linux `aarch64` only. |

### Notes

`allocator-tcmalloc` starts a background maintenance thread during process bootstrap.
`cargo check --all-features` is intentionally excluded because allocator features are mutually
exclusive.

## Build Matrix

Requires Rust 1.98.1 or newer; this repository is pinned by `rust-toolchain.toml`.
Requires Rust 1.98.1 or newer; the repository is pinned by
[`rust-toolchain.toml`](../rust-toolchain.toml).

Build each variant into a separate binary path.

Expand All @@ -50,11 +55,11 @@ allocator builds and document the chosen baseline.
## Benchmark Protocol

1. Build each allocator variant separately.
2. Run the canonical benchmark harness with the same capture, format set, and bonded value.
2. Run the [canonical benchmark harness](README.md) with the same capture, format set and bonded value.
3. Record throughput, shutdown tail, and peak RSS from logs.
4. Compare outputs for determinism before accepting any allocator change.

Example:
The following example compares jemalloc and mimalloc using the saved binaries:

```bash
bash benches/benchmark.sh \
Expand All @@ -79,9 +84,3 @@ bash benches/benchmark.sh \
- No correctness regression.
- No byte-for-byte CSV drift for the same representative run.
- Clear benchmark evidence that the allocator change is worth the operational tradeoff.

## Notes

- `allocator-tcmalloc` starts a background maintenance thread during process bootstrap.
- `cargo check --all-features` is intentionally not part of this protocol because allocator features
are mutually exclusive.
Loading
Loading