Skip to content

feat: add first end-to-end find slice - #43

Merged
tappe9 merged 51 commits into
mainfrom
feat/first-end-to-end-find
Aug 19, 2026
Merged

tappe9 merged 51 commits into
mainfrom
feat/first-end-to-end-find

Conversation

@tappe9

@tappe9 tappe9 commented Aug 19, 2026 •

Copy link
Copy Markdown
Owner

Summary

Implements the first end-to-end find vertical slice as one dependency-ordered sprint:

  1. Issue [v0.1] Integrate Archive::table_rows with lending row and column APIs #14 — integrate table lookup, TABLE DATA resolution, validated selected-entry seek, custom chunk framing, none/gzip decompression, COPY metadata validation, and borrowed COPY text parsing behind Archive::table_rows.
  2. Issue [v0.1] Implement streaming find_first with OwnedRow and early termination #15 — add byte-oriented find_first, OwnedField, and OwnedRow, materializing only the first matched row and terminating the stream immediately.
  3. Issue [v0.1] Deliver the first end-to-end pgdumpx find CLI slice #16 — add the thin pgdumpx find <FILE> <SCHEMA.TABLE> <COLUMN> <VALUE> CLI adapter using the library production path.

Dependency chain: #14 → #15 → #16. No Issue #17+ API or enforcement work is included.

Closes #14
Closes #15
Closes #16

Public API and CLI

Library

  • Archive::table_rows(schema, table) -> TableRowReader<'_, R>
  • TableRowReader::{columns, column_index, next_row, find_first}
  • borrowed Row<'_> remains the normal lending scan API
  • OwnedField::{Null, Bytes(Vec<u8>)} and OwnedRow are used only when a match must outlive reusable parser storage

CLI

pgdumpx find <FILE> <SCHEMA.TABLE> <COLUMN> <VALUE>
  • match: exit 0, one normalized COPY text record on stdout
  • no match: exit 1, no stdout/stderr
  • usage or runtime failure: exit 2, diagnostic on stderr
  • stdout preserves COPY column order and distinguishes NULL (\N) from an empty field
  • \\, tab, LF, and CR are emitted as canonical COPY escapes; other non-printable/non-ASCII bytes use three-digit octal escapes such as \377
  • query arguments are UTF-8 in v0.1; archive row data remains byte-oriented and is never converted lossily

Red → Green → Refactor

Issue #14

Red

Acceptance tests were added before production code for:

  • official PostgreSQL 18.4 none/gzip fixtures through Archive::table_rows
  • table → TABLE DATA resolution and column order/lookup
  • arbitrary short source reads
  • no complete entry/table-data buffering after one row
  • positional scan when column metadata is unavailable
  • unsupported representation rejection before seek/payload access
  • missing table / missing TABLE DATA typed errors
  • provisional field-count limit
  • parser-consumed byte accounting for none/gzip

The clean Red run passed formatting and failed compilation only because TableRowReader, Archive::table_rows, and TableNotFound were not implemented.

Green

  • extracted the existing validated selected-entry open path into a shared private helper
  • composed EntryDataReader with CopyRowReader in TableRowReader
  • reused open-time TABLE DATA metadata and rejected unsupported representations before seek
  • retained backend-neutral none/gzip streaming and the borrowed lending row API

Refactor

  • added official none/gzip one-byte source-read coverage
  • retained the existing parser-consumed accounting seam without exposing it publicly

Issue #15

Red

Acceptance tests were added first for:

  • first of multiple matches
  • OwnedRow lifetime beyond the reader/archive scope
  • no match as Ok(None)
  • NULL, empty, and non-UTF-8 fields
  • early termination with a large unread tail
  • exact parser-consumed bytes for the first row in none/gzip fixtures

The clean Red run passed formatting and failed compilation only because OwnedField/OwnedRow and find_first were absent.

Green

  • find_first evaluates borrowed rows in order
  • non-matching rows are not materialized
  • only the first matching row is copied into finite, fallibly allocated owned storage
  • the method returns immediately after the first match

Refactor

  • added an official-fixture predicate-call-count test proving one evaluation per visited row and no evaluation after the first match

Issue #16

Red

README/README.ja and 11 CLI integration tests defined the command grammar, stdout normalization, statuses, errors, UTF-8 boundary, non-UTF-8 row handling, and official none/gzip end-to-end behavior before implementation.

The clean Red run passed formatting, Clippy, and Rust 1.85.0 checking; only the new CLI tests failed because the pre-existing main() returned success without performing work.

Green

The CLI now delegates the complete data path to:

Archive::open
  → Archive::table_rows
  → TableRowReader::column_index
  → TableRowReader::find_first
  → normalized row output

It does not reimplement archive parsing, selected-entry validation, decompression, COPY parsing, or filtering.

Refactor

  • buffered stdout and batched contiguous printable bytes
  • explicitly handles flush failures
  • added official-fixture LF/CR normalization coverage

Official PostgreSQL 18.4 evidence

Both committed archive 1.16 fixtures are exercised through production paths:

  • pg18-none-copy-basic.dump
  • pg18-gzip-copy-basic.dump

Evidence includes:

  • identical public table_rows results
  • ordered COPY columns and column lookup
  • byte-oriented NULL/empty/escaped fields
  • one-byte source-read invariance for both compression modes
  • exact parser-consumed byte accounting despite decoder/BufRead read-ahead
  • CLI early, middle, late, repeated, and absent matches
  • normalized tab, backslash, LF, and CR output

Early termination evidence

  • a tracking reader wraps a synthetic table-data entry with the match in the first row and 32,768 trailing rows
  • find_first returns the first match while more than 100,000 archive bytes remain unread
  • an official-fixture predicate counter proves that a middle-row match evaluates rows 1–4 exactly once and never evaluates rows 5–7
  • exact consumed-byte assertions for none/gzip prove accounting remains based on COPY parser consumption rather than underlying decoder read-ahead

Quality gates

Final clean head: c53ef6ef564ebc84596ec0ec06a5a9352201be72

All required gates passed on the PR merge ref:

cargo fmt --check
cargo clippy --workspace --all-targets --all-features -- -D warnings
cargo test --workspace --all-features
RUSTDOCFLAGS="-D warnings" cargo doc --workspace --all-features --no-deps
cargo +1.85.0 check --workspace --all-targets --all-features

Final test inventory includes:

  • 31 library unit tests
  • 14 archive-open tests
  • 5 COPY metadata tests
  • 9 COPY row tests
  • 10 selected-entry streaming tests
  • 2 focused find acceptance tests
  • 10 table-row integration tests
  • 11 CLI find integration tests
  • 1 CLI normalized-output test
  • fixture-manifest, workspace-smoke, and compile-fail lending doctests

No test failures, Clippy warnings, rustdoc warnings, or MSRV failures remain.

Acceptance self-review

Issue #14

  • table lookup resolves the related TABLE DATA entry
  • open-time COPY metadata is reused; no payload metadata reparsing
  • unsupported representations fail before validated seek or row parsing
  • the existing validated offset/block-type/dump-ID path is reused
  • none/gzip remain streaming and backend-neutral publicly
  • arbitrary one-byte reads preserve results for both none/gzip fixtures
  • complete entries and complete table-data are not buffered
  • borrowed Row<'_> lending semantics remain intact
  • COPY NULL and empty fields remain distinct
  • non-UTF-8 logical bytes remain representable
  • COPY column order matches row field order and lookup indices
  • provisional finite allocation limits remain enforced
  • parser-consumed accounting remains independent of decoder read-ahead

Issue #15

  • predicates use byte-oriented borrowed fields
  • column lookup reuses existing metadata
  • only the first match is returned
  • no match is a normal Ok(None) result
  • only the matched row is materialized into OwnedRow
  • owned matches outlive reader/archive storage
  • NULL, empty, and non-UTF-8 fields are preserved
  • the stream is not read to entry end after a match
  • all rows/table-data are never collected
  • the existing consumed-byte seam is preserved
  • no ScanLimits or final public Limits API was introduced

Issue #16

  • CLI argument grammar and UTF-8 boundary are explicit and tested
  • table/column lookup and first-match filtering delegate to the library
  • official none/gzip fixtures exercise the complete end-to-end path
  • match/no-match/error stdout, stderr, and exit statuses are tested
  • invalid table/column, malformed archive, unsupported representation, and unavailable metadata are failures rather than no-match
  • non-UTF-8 row bytes are emitted deterministically without lossy conversion
  • NULL, empty, tab, backslash, LF, and CR output are covered
  • CLI contains no duplicate archive/COPY parser or filter implementation

Safety

  • byte-oriented core design is retained
  • no complete selected entry/table-data buffering was added
  • existing finite provisional allocation policy is retained
  • no project-authored unsafe was added; the library remains #![forbid(unsafe_code)]

Intentionally deferred

This PR does not implement or pre-design:

tappe9 and others added 30 commits August 20, 2026 06:34
@tappe9
tappe9 marked this pull request as ready for review August 19, 2026 23:01
@tappe9
tappe9 merged commit 43804ec into main Aug 19, 2026
2 checks passed
@tappe9
tappe9 deleted the feat/first-end-to-end-find branch August 19, 2026 23:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant