Skip to content

Guarantee monotonic timestamps in rewritten PCAP output (#86) - #87

Merged
sophie-cluml merged 1 commit into
mainfrom
sophie-cluml/issue-86
Jul 29, 2026
Merged

Guarantee monotonic timestamps in rewritten PCAP output (#86)#87
sophie-cluml merged 1 commit into
mainfrom
sophie-cluml/issue-86

Conversation

@sophie-cluml

@sophie-cluml sophie-cluml commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

Summary

rewrite_timestamps overwrote each record's (ts_sec, ts_usec) in place and left the records in capture order, so the output PCAP was monotonic only if the input already was. Bridge/kernel timestamping regularly produces captures where two adjacent records regress by a few hundred µs, which made the compressed-pipeline monotonicity check fail depending on capture timing (as on #85).

The rewriter now owns the invariant:

  • Each record's byte range is collected during the existing raw byte walk, paired with the microsecond value actually written into the record (ts_sec * 1_000_000 + ts_usec) — not the full-precision logical timestamp, so "same sort key" and "same bytes on disk" stay the same predicate.
  • The ranges are stable-sorted by that key (sort_by_key), so records sharing a logical timestamp keep their capture order.
  • The file is reassembled from the original global header, the records in sorted order, and any preserved truncated tail. Reassembly is skipped when the keys are already non-decreasing — a pure optimization that reproduces the buffer byte for byte.
  • The walk still goes through raw bytes rather than the IPv4 parsing path, so non-IPv4 records (ARP, IPv6, LLDP, …) are preserved and reordered along with the rest. incl_len, orig_len, and payloads are never touched.

Side outputs (max logical timestamp, out-of-window info message) are computed during the walk as before; both are order-independent.

The doc comment's unconditional byte-identity claim is updated: byte-identity now holds exactly when the rewritten timestamps are already non-decreasing. An identity mapping over an already-monotonic capture still satisfies that, so the existing byte-identity case is unchanged.

Also adds a monotonicity assertion to the Docker-gated generate_ac0_produces_valid_bundle test so the invariant is checked against a real capture, not only synthetic jitter.

Closes #86

Test plan

  • cargo fmt --check passes
  • cargo clippy --all-targets --all-features -- -D warnings passes with no warnings
  • cargo test passes (388 passed, 16 Docker-gated ignored)
  • A later record with an earlier timestamp is reordered and the output is monotonically non-decreasing
  • Records with equal logical timestamps keep their capture order (three-record tie case)
  • Stability is locked in by a tie-heavy 33-record burst long enough to reach the pattern-defeating quicksort path, where an unstable sort would reorder equal keys
  • An already-monotonic identity rewrite is byte-identical to the input
  • Truncated tail preserved verbatim: partial trailing record header, both with and without reordering
  • Truncated tail preserved verbatim: complete 16-byte header whose payload is cut short (and it does not contribute to max ts)
  • Big-endian captures reorder correctly and stay big-endian
  • Non-IPv4 (IPv6 ethertype) records are preserved and reordered — asserted on raw bytes, since parse_pcap drops them
  • Sort key is the post-rewrite logical timestamp, not the real one (anchored compression reorders a background packet ahead of an execution packet)
  • Heavily jittered capture: every record survives exactly once, file size unchanged, exact output sequence pinned, max ts correct
  • Rewrite stays all-or-nothing — a malformed later record leaves the file's original bytes intact
  • Existing rejection cases still hold: nanosecond-resolution PCAP, pcapng, invalid magic, malformed ts_usec, pre-epoch and past-u32 logical timestamps
  • Max rewritten logical timestamp is still returned for meta.actual_end
  • AC-0 Compressed Pipeline CI job passes, including the "Assert pcap structural invariants" monotonicity check

The rewriter overwrote each record's timestamp in place and left the
records in capture order, so the output was monotonic only if the input
already was. Bridge/kernel timestamping regularly hands us captures that
regress by a few hundred microseconds, which made the compressed-pipeline
monotonicity check fail depending on capture timing.

Collect each record's byte range alongside the microsecond value actually
written, stable-sort by that key, and reassemble the file from the global
header, the sorted records, and any truncated tail. Keying on the stored
value rather than the full-precision logical timestamp keeps "same sort
key" and "same bytes on disk" the same predicate, so ties are observable
in the output; the stable sort then preserves capture order among them.
Reassembly is skipped when the keys are already non-decreasing, which is
a pure optimization.

Byte-identical output is now guaranteed exactly when the rewritten
timestamps are already ordered, rather than unconditionally.

Closes #86
@sophie-cluml

Copy link
Copy Markdown
Contributor Author

[Reviewer Round 1]

No blocking findings.

The implementation matches the issue's requested shape: src/pcap.rs:251 keeps the raw byte walk, src/pcap.rs:292 rewrites only the timestamp fields in the in-memory buffer, and src/pcap.rs:302 records the original byte range for the full record. The sort key at src/pcap.rs:304 is the microsecond value actually written to disk, which is the important detail for the sub-microsecond/tie requirement. The monotonic fast path at src/pcap.rs:318 preserves already-ordered identity captures byte-for-byte, while the reorder path uses stable sort_by_key at src/pcap.rs:329 and reassembles from byte ranges, so equal timestamps keep capture order and non-IPv4 records are not dropped or parsed through the IPv4-only reader.

The truncated-tail handling also looks correct: the walk stops before incomplete record bytes, offset remains the first tail byte, and reassemble appends data[tail_start..] verbatim at src/pcap.rs:353. The existing side effects remain order-independent: max logical timestamp and the out-of-window info flag are computed during the walk before sorting.

The tests are meaningful and would have failed against the previous capture-order behavior. In particular, they cover regressed timestamps, equal-key stability including the tie-heavy case, monotonic identity byte preservation, both truncation shapes, big-endian reorder, raw-byte non-IPv4 preservation, logical-time sorting rather than real-time sorting, record conservation, and all-or-nothing failure behavior. I also ran cargo test pcap::tests::rewrite locally; it passed with 28 tests.

@sophie-cluml

Copy link
Copy Markdown
Contributor Author

[Review Verdict Round 1: APPROVED]

@sophie-cluml
sophie-cluml merged commit e0d67b4 into main Jul 29, 2026
10 checks passed
@sophie-cluml
sophie-cluml deleted the sophie-cluml/issue-86 branch July 29, 2026 05:39
sehkone added a commit that referenced this pull request Aug 4, 2026
`push` matched every branch and `pull_request` matched the same commits,
so both fired on one commit and the whole workflow ran twice. On #87 the
head commit carried two CI runs and 10 check runs -- one full set per
event. They were never a "branch" run and a "merge" run; they were the
same commit checked twice.
Scope `push` to `main`. Merges still run CI.
Drop the `pull_request` base filter as well, so a pull request against
any base is covered rather than only those targeting `main`.
What this gives up is CI on a branch with no pull request open, and
nothing else. `pull_request` checks out the branch already merged into
its base, so merge coverage is unaffected. `workflow_dispatch` replaces
the lost case (`gh workflow run ci.yml --ref <branch>`), and opening the
pull request as a draft works too -- nothing here filters drafts.
Release automation is unaffected: release branches land through pull
requests in this organization, so the `pull_request` trigger still
covers them.

Closes #88
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Guarantee monotonic timestamps in rewritten PCAP output

1 participant