Repository navigation
feat: add a parquet source and reader - #204
Merged
Merged
Conversation
Files.PARQUET enum token plus a Parquet(BaseSource) model with no format-specific fields (parquet is self-describing: typed columns, real headers, no sheets or delimiters). Section.source now accepts kind: parquet alongside text and excel, and _section_source_label widens to the new union. Tests cover enum membership, model parse + extra-field rejection, all-three-kinds Section validation, and the per-kind diagnostic label.
parquet() scans a parquet file lazily and renames its columns to the positional column_1.. convention in schema order, matching the headerless csv/excel readers: method: column encodings accept only Excel-style A-ZZ letters, so configs address columns positionally and real parquet names are unreachable. _source_ops dispatches the op for kind: parquet, PHASE_OF labels it load, and runcache OP_COST prices it 2 like the other load ops. A new e2e smoke builds a KG from a parquet source against a real redb and asserts resolved HGNC CURIEs in the KGX output.
TABLE_EXTENSIONS gains .parquet so fetch bucketing and is_useful_file accept parquet payloads; _load_table dispatches .parquet to pl.read_parquet (typed columns as-is, unknown suffixes still raise); _effective_rows_cached counts parquet rows with the same all-null blank-row rule as the Excel branch; stale Excel | Text union comments now name Parquet. Tests cover is_table_file, parquet row counting, read_table rendering, and the direct _load_table dispatch.
docs/configuration/table.md gains a Parquet Source section (field table + example) matching the Excel/Text shape, with the positional column-addressing note, and the Specify-kind callout enumerates the parquet reader. CHANGELOG gains an Unreleased Added entry. The SSOT guard gets a ModelTableBinding(Parquet, ..., 'Parquet Source') so the new model's docs table is checked bidirectionally against the live pydantic fields like every other config model.
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Advanced Run ID: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
SkyeAv
added a commit
that referenced
this pull request
Sep 30, 2026
Cut 19.5.0 and bump the package version in pyproject.toml,
CITATION.cff, and the uv.lock self-entry.
Minor: one new capability since 19.4.0. `parquet` is now a first-class
source kind: table sections accept `source: {kind: parquet, ...}`,
read lazily via `pl.scan_parquet` with columns renamed to the
positional `column_1..` convention (the same contract the headerless
`text`/`excel` readers produce), so `method: column` encodings keep
using Excel-style letters (#204). The source adds no format-specific
fields (parquet is self-describing), and the autonomous agent accepts
`.parquet` payloads as tables: fetch bucketing, previews, and row
counts dispatch on the extension.
Changelog:
- Versioned the Unreleased section as 19.5.0 and added the #204 PR
link to the entry.
Docs: docs/configuration/table.md and the CHANGELOG entry shipped
with #204; mkdocs builds strict.
Testing:
- uv run pytest -> 1674 passed, 52 skipped
- uv run ruff check . / uv run ruff format --check . -> All checks
passed! / 101 files already formatted
- uv run pyright -> 0 errors, 0 warnings, 0 informations
- uv lock --check -> Resolved 169 packages
- uv run --no-sync mkdocs build --strict -> built in 0.83 seconds
- cargo test (rust/Cargo.toml) -> 156 passed, 0 failed
- cargo clippy --all-targets -- -D warnings -> clean
- cargo fmt --check -> ok
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Table sections gain a third source kind,
parquet, so.parquettables can drive builds directly as table inputs shift toward parquet. The kind flows end to end: config model, section pipeline, agent table tooling, docs, and changelog.Source model
Files.PARQUET = "parquet"insrc/tablassert/enums.py.Parquet(BaseSource)insrc/tablassert/models.pyadds no format-specific fields: parquet is self-describing (typed columns, real headers, no sheets or delimiters), soextra="forbid"rejects Excel-stylesheet/delimiterkeys instead of silently ignoring them. Sharedrows/row_slice/reindexcontrols carry over.Section.sourceis nowExcel | Text | Parquet(smart union on thekindliteral; all three kinds still validate), and_section_source_labelaccepts the widened union, rendering parquet as the bare path.Pipeline reader
lib.parquetinsrc/tablassert/lib.pyscans viapl.scan_parquet(lazy, native, no new dependency) and renames the file's columns to the positionalcolumn_1..convention in schema order. Why positional:Encoding.excel_style_columnsonly accepts Excel-styleA-ZZletters formethod: columnencodings for every source kind, so configs address columns positionally and the reader must match what the headerlesscsv/excelreaders already produce. Accepted tradeoff: a parquet file's own header names are not referenceable from configs, same as CSV/TSV.Table._source_opsdispatches exactly one load op persource.kind;PHASE_OF[parquet]is"load"for progress UX;runcache.OP_COST[lib.parquet] = 2matches the other load ops for snapshot pricing.Agent
agent._load_tabledispatches.parquettopl.read_parquet(typed columns as-is; parse failures still collapse to the clearValueError; unknown suffixes still raise), and_effective_rows_cachedcounts parquet rows with the same all-null blank-row rule as the Excel branch.TABLE_EXTENSIONSgains.parquet, sois_table_file/is_useful_fileaccept parquet payloads during fetch. The agent keeps real column names here: it renders previews, it does not author positional configs.Docs
docs/configuration/table.mdgains a#### Parquet Sourcesection (field table + YAML example) and the "Specify kind explicitly" callout enumerates the parquet reader.CHANGELOG.mdgains an Unreleased Added entry.tests/test_docs_source_of_truth.pybindsParquetviaModelTableBinding(Parquet, ..., "Parquet Source")so the docs field table is checked bidirectionally against the live model like every other config model.Testing
uv run pytest->1674 passed, 52 skipped(skips are pre-existing extra/platform gates; the new tests execute by name in the targeted runs below)uv run pytest tests/test_enums.py tests/test_cover_models.py tests/test_cli_validation.py tests/test_cover_lib.py tests/test_e2e_smoke.py tests/test_run_cache.py tests/test_cli_progress.py tests/test_cover_agent_core.py tests/test_agent_fetch.py tests/test_docs_examples.py tests/test_docs_source_of_truth.py tests/test_docs_cli_coverage.py -q->401 passeduv run ruff check .anduv run ruff format --check .-> cleanuv run pyright->0 errors, 0 warnings, 0 informationscargo test --manifest-path rust/Cargo.toml->127 passed,11 passed / 1 ignored,17 passed,1 passed; 0 failed (no Rust changes)tablassert validate -s tableon akind: parquetconfig against a real.parquetfile -> exit 0test_build_pipeline_over_parquet_sourcebuilds a KG from a parquet source through a REAL redb and assertsHGNC:1100/HGNC:6871in the KGX nodes and edges output