Skip to content

feat: add a parquet source and reader - #204

Merged
SkyeAv merged 4 commits into
mainfrom
add-parquet-source
Sep 30, 2026
Merged

SkyeAv merged 4 commits into
mainfrom
add-parquet-source

Conversation

@SkyeAv

@SkyeAv SkyeAv commented Sep 30, 2026

Copy link
Copy Markdown
Owner

Table sections gain a third source kind, parquet, so .parquet tables can drive builds directly as table inputs shift toward parquet. The kind flows end to end: config model, section pipeline, agent table tooling, docs, and changelog.

Source model

  • New kind token. Files.PARQUET = "parquet" in src/tablassert/enums.py.
  • Parquet(BaseSource) in src/tablassert/models.py adds no format-specific fields: parquet is self-describing (typed columns, real headers, no sheets or delimiters), so extra="forbid" rejects Excel-style sheet/delimiter keys instead of silently ignoring them. Shared rows/row_slice/reindex controls carry over.
  • Union widened. Section.source is now Excel | Text | Parquet (smart union on the kind literal; all three kinds still validate), and _section_source_label accepts the widened union, rendering parquet as the bare path.

Pipeline reader

  • lib.parquet in src/tablassert/lib.py scans via pl.scan_parquet (lazy, native, no new dependency) and renames the file's columns to the positional column_1.. convention in schema order. Why positional: Encoding.excel_style_columns only accepts Excel-style A-ZZ letters for method: column encodings for every source kind, so configs address columns positionally and the reader must match what the headerless csv/excel readers already produce. Accepted tradeoff: a parquet file's own header names are not referenceable from configs, same as CSV/TSV.
  • Wiring. Table._source_ops dispatches exactly one load op per source.kind; PHASE_OF[parquet] is "load" for progress UX; runcache.OP_COST[lib.parquet] = 2 matches the other load ops for snapshot pricing.

Agent

  • Table reads. agent._load_table dispatches .parquet to pl.read_parquet (typed columns as-is; parse failures still collapse to the clear ValueError; unknown suffixes still raise), and _effective_rows_cached counts parquet rows with the same all-null blank-row rule as the Excel branch.
  • Bucketing. TABLE_EXTENSIONS gains .parquet, so is_table_file/is_useful_file accept parquet payloads during fetch. The agent keeps real column names here: it renders previews, it does not author positional configs.

Docs

  • docs/configuration/table.md gains a #### Parquet Source section (field table + YAML example) and the "Specify kind explicitly" callout enumerates the parquet reader.
  • CHANGELOG.md gains an Unreleased Added entry.
  • tests/test_docs_source_of_truth.py binds Parquet via ModelTableBinding(Parquet, ..., "Parquet Source") so the docs field table is checked bidirectionally against the live model like every other config model.

Testing

  • uv run pytest -> 1674 passed, 52 skipped (skips are pre-existing extra/platform gates; the new tests execute by name in the targeted runs below)
  • Targeted across all impacted files: uv run pytest tests/test_enums.py tests/test_cover_models.py tests/test_cli_validation.py tests/test_cover_lib.py tests/test_e2e_smoke.py tests/test_run_cache.py tests/test_cli_progress.py tests/test_cover_agent_core.py tests/test_agent_fetch.py tests/test_docs_examples.py tests/test_docs_source_of_truth.py tests/test_docs_cli_coverage.py -q -> 401 passed
  • uv run ruff check . and uv run ruff format --check . -> clean
  • uv run pyright -> 0 errors, 0 warnings, 0 informations
  • cargo test --manifest-path rust/Cargo.toml -> 127 passed, 11 passed / 1 ignored, 17 passed, 1 passed; 0 failed (no Rust changes)
  • CLI exercise: tablassert validate -s table on a kind: parquet config against a real .parquet file -> exit 0
  • New e2e smoke test_build_pipeline_over_parquet_source builds a KG from a parquet source through a REAL redb and asserts HGNC:1100/HGNC:6871 in the KGX nodes and edges output

Files.PARQUET enum token plus a Parquet(BaseSource) model with no format-specific
fields (parquet is self-describing: typed columns, real headers, no sheets or
delimiters). Section.source now accepts kind: parquet alongside text and excel,
and _section_source_label widens to the new union. Tests cover enum membership,
model parse + extra-field rejection, all-three-kinds Section validation, and the
per-kind diagnostic label.
parquet() scans a parquet file lazily and renames its columns to the positional
column_1.. convention in schema order, matching the headerless csv/excel readers:
method: column encodings accept only Excel-style A-ZZ letters, so configs address
columns positionally and real parquet names are unreachable. _source_ops dispatches
the op for kind: parquet, PHASE_OF labels it load, and runcache OP_COST prices it 2
like the other load ops. A new e2e smoke builds a KG from a parquet source against a
real redb and asserts resolved HGNC CURIEs in the KGX output.
TABLE_EXTENSIONS gains .parquet so fetch bucketing and is_useful_file accept parquet
payloads; _load_table dispatches .parquet to pl.read_parquet (typed columns as-is,
unknown suffixes still raise); _effective_rows_cached counts parquet rows with the
same all-null blank-row rule as the Excel branch; stale Excel | Text union comments
now name Parquet. Tests cover is_table_file, parquet row counting, read_table
rendering, and the direct _load_table dispatch.
docs/configuration/table.md gains a Parquet Source section (field table + example)
matching the Excel/Text shape, with the positional column-addressing note, and the
Specify-kind callout enumerates the parquet reader. CHANGELOG gains an Unreleased
Added entry. The SSOT guard gets a ModelTableBinding(Parquet, ..., 'Parquet Source')
so the new model's docs table is checked bidirectionally against the live pydantic
fields like every other config model.
@coderabbitai

coderabbitai Bot commented Sep 30, 2026

Copy link
Copy Markdown

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: d21e1830-c9f1-4fcd-a5b1-48854450680c


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@SkyeAv
SkyeAv merged commit b567286 into main Sep 30, 2026
5 checks passed
SkyeAv added a commit that referenced this pull request Sep 30, 2026
Cut 19.5.0 and bump the package version in pyproject.toml,
CITATION.cff, and the uv.lock self-entry.

Minor: one new capability since 19.4.0. `parquet` is now a first-class
source kind: table sections accept `source: {kind: parquet, ...}`,
read lazily via `pl.scan_parquet` with columns renamed to the
positional `column_1..` convention (the same contract the headerless
`text`/`excel` readers produce), so `method: column` encodings keep
using Excel-style letters (#204). The source adds no format-specific
fields (parquet is self-describing), and the autonomous agent accepts
`.parquet` payloads as tables: fetch bucketing, previews, and row
counts dispatch on the extension.

Changelog:
- Versioned the Unreleased section as 19.5.0 and added the #204 PR
  link to the entry.

Docs: docs/configuration/table.md and the CHANGELOG entry shipped
with #204; mkdocs builds strict.

Testing:
- uv run pytest -> 1674 passed, 52 skipped
- uv run ruff check . / uv run ruff format --check . -> All checks
  passed! / 101 files already formatted
- uv run pyright -> 0 errors, 0 warnings, 0 informations
- uv lock --check -> Resolved 169 packages
- uv run --no-sync mkdocs build --strict -> built in 0.83 seconds
- cargo test (rust/Cargo.toml) -> 156 passed, 0 failed
- cargo clippy --all-targets -- -D warnings -> clean
- cargo fmt --check -> ok
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant