Optimize attribute parsing and bound Arrow assembly memory (3.0.2)

[iskandr]

gtfparse 3.0.0 repeatedly splits and trims millions of attribute pairs and retains full-file Python values while assembling compact Arrow string columns. Version 3.0.2 captures ordinary double-quoted pairs directly, removes redundant lookups and conversions, and assembles configured Arrow strings in 250,000-row batches. Temporary Python values are released after each batch and column conversion.

Preserves arbitrary attributes, malformed/custom-quoted and pre-split input, repeated/empty values, missing sentinels, first-seen column order, fixed-field collisions, progress/cancellation, aliases/casts/converters, raw/selective reads, configured Python/object strings, dict/optional Polars output and existing stream/validation behavior. Global pandas options stay under caller control. Polars remains optional.

Fixes #91.

Version allocation: the measured candidate was labeled 3.0.1. This optimization shipped independently as 3.0.2 to avoid the concurrent documentation PR’s version; raw records retain their original labels.

Complete production pipelines are measured sequentially in fresh processes with four threads and matching content/order hashes. Reader order rotates and corpus order reverses. Across pandas 3.0.6 and Ensembl/GENCODE 250k-row plain/gzip inputs, three-run medians take 17–25% less time than the 2.9.1 Polars pipeline and 31–36% less time than 3.0.0. pandas 2.3.3 Ensembl medians improve; its GENCODE wall times are mixed and highly variable, so no general speedup is claimed there. Every run and range is retained.

Full-file scaling checks on pandas 3.0.6, one observation per reader:

Full gzip input	Polars 2.9.1	Arrow 3.0.0	Optimized 3.0.2
Ensembl GRCh38.114, 4.12M rows	47.11 s / 6504 MiB	67.63 s / 7199 MiB	36.86 s / 3639 MiB
GENCODE v48 primary, 4.12M rows	43.74 s / 7091 MiB	52.35 s / 6560 MiB	29.88 s / 3698 MiB

These full-file observations take 22–32% less time and 44–48% less peak RSS than Polars. They verify scaling and correctness, rather than repeated full-file throughput. Bounded temporary assembly applies to Arrow string storage; Python/object storage retains compatible construction. This compares complete gtfparse pipelines, not pure pandas-versus-Polars engines. Imports/checksums are outside load time; RSS is captured before validation; OS caches are not flushed and the workstation is shared. The rejected unbatched candidate is retained separately and excluded from final claims.

Validation:

• ./lint.sh and ./test.sh -W error::DeprecationWarning: 213 passed, 97% coverage.
• Base pandas 3.0.6/Arrow 25 and pandas 3.0.6/Arrow 18: 199 passed, 10 optional skips, 96% coverage. Minimum Python 3.9/pandas 2.2.2/Arrow 18: 199 passed, 10 skips, 88% coverage. Minimum optional Polars 0.20.31: 212 passed, one modern-only skip, 97% coverage and the existing native categorical-remapping warning.
• 142 exact full API/dtype comparisons against 3.0.0 across pandas 2.3.3/3.0.6 and all five real fixtures pass.
• New frozen-legacy differential tests exercise malformed/Unicode/whitespace text, custom quotes/sentinels, repetitions, empty values, pre-split rows and column selection. Batch regressions check late columns, missing batches, collisions, configured string storage and global progress counts.
• All 84 final benchmark runs match content/order, including selective reads and explicit Polars output.
• Wheel/source metadata, strict Twine checks and shipped-source lint/tests are verified before submission.

Reproduction and limitations: optimization report, retained measurements, and benchmarks/compare_releases.py.

Release verification (2026-10-06):

• Final push and PR workflows passed: push CI, PR CI, including all runtime/package jobs and coverage flags.
• Merged as a80b0827e982c32409d78aa707d6350374b6555a; ran ./deploy.sh from clean master. Its lint/tests passed: 213 tests, 97% coverage.
• gtfparse 3.0.2 is published on PyPI. Downloaded wheel/source SHA-256 hashes and metadata match PyPI and the clean-master upload artifacts; Polars remains an optional extra.
• Fresh public-index installation of gtfparse==3.0.2 with pandas 3.0.6/Arrow 25 passes pip check, installs/imports no Polars, and passes 20 raw/expanded plain/gzip round trips across all five fixtures. A 250,003-row read verifies the actual batch boundary, late columns, missing prefixes, collisions, repeated attributes and global progress.
• Post-release issue review completed; the workspace remains clean on master.

Next work, grouped by dependency and urgency: within gtfparse, #65 is the next foundational candidate for synthesis policy, followed by complementary completeness flags in #66. #37 is the separate route to a public streaming reader; this release still loads the fixed table. Documentation #86/#92 is already represented by PR #93, with small examples #13/#16 remaining. Downstream, PyEnsembl #445 is the smaller ID-consistency fix, while #447 is the broader unified-annotation work. Datacache’s independent adoption blockers are #88/#89; source identity #92 supports materialization #90, with inventory/pruning #91 and layout/local-input extensions #93/#94 following. These are recommendations from the issue review; no sibling repository changes were made.

[coveralls]

Coverage Report for CI Build 37506258083

Coverage increased (+1.6%) to 97.236%

Details

• Coverage increased (+1.6%) from the base build.
• Patch coverage: No coverable lines changed in this PR.
• 4 coverage regressions across 1 file.

Uncovered Changes

No uncovered changes found.

Coverage Regressions

4 previously-covered lines in 1 file lost coverage.

File	Lines Losing Coverage	Coverage
read_gtf.py	4	97.67%

---

Coverage Stats

Relevant Lines:	398
Covered Lines:	387
Line Coverage:	97.24%
Coverage Strength:	6.77 hits per line

---

💛 - Coveralls