Skip to content

Add GTF parsing progress and fix attribute data loss (2.9.0) - #82

Merged
iskandr merged 1 commit into
masterfrom
feature/80-parsing-progress
Sep 29, 2026
Merged

iskandr merged 1 commit into
masterfrom
feature/80-parsing-progress

Conversation

@iskandr

@iskandr iskandr commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor

Large GTF reads expose no progress hook, and attribute parsing silently truncates values such as "two words" and "part1; part2". Add optional progress_callback(stage, completed, total) hooks and preserve complete quoted values, including apostrophes and semicolons. Release as 2.9.0.

Callbacks report read, attributes, and convert. Attribute expansion reports every 10,000 rows; opaque Polars operations report start/completion with an initially unknown total. The default stays silent with no progress UI dependency. Document the contract and an executable tqdm adapter; propagate cancellation/errors without false completion events.

Also preserve raw attribute text, treat missing attribute fields as empty, and reject invalid result_type before reading input. Retain explicit fix_quotes_columns cleanup as an opt-in compatibility option. Update writer round-trip coverage for quoted values. Embedded unescaped double quotes and literal tabs/newlines remain outside writer guarantees.

Validation:

  • ./lint.sh passes.
  • ./test.sh -W error::DeprecationWarning: 143 passed on Polars 1.44.2 and 1.32.0; 142 passed and one modern-only skip on 1.31.0. Coverage: 96%, attribute parser: 100%.
  • New regression tests exercise callback ordering, filtering/empty results, bounded updates, output types, raw mode, gzip/file/stream input, cancellation, and each parsing fix.
  • All five existing real GTF fixtures produce identical expanded results to 2.8.1. README tqdm example runs successfully.
  • 250,000-row / 69.3 MB repeated Ensembl fixture: all baseline/default/callback outputs match, with 30 callback events. Three interleaved wall-time medians were 6.05s / 4.75s / 4.28s; shared-machine variability (1.34–6.65s overall) prevents a reliable speedup or precise overhead claim.
  • Isolated attribute-expansion benchmark, 100,000 rows with seven interleaved runs per mode: median CPU time 1.1813s without callbacks versus 1.1915s with an event-collecting callback (0.86% bookkeeping overhead). Caller UI work adds its own cost.

Fixes #80. Fixes #23. Fixes #44. Fixes #81. Addresses the invalid-result-type bug and related touched docstrings in #76; leaves its unrelated cleanup open.

Release verification (2026-09-29): merged as 9a281c1. Ran ./deploy.sh from clean master; lint and all 143 tests passed. gtfparse 2.9.0 is published. Both PyPI artifact SHA-256 digests match the local build. Reinstalled the PyPI wheel into an isolated environment and verified every installed package source matches the release wheel. Smoke checks passed with deprecations as errors for progress events, quoted values, missing attributes, raw preservation, invalid result types, and gzip round trips. All five CI matrix jobs and combined coverage checks passed before merge.

Tracked the non-blocking setuptools license-metadata build warning separately in #83. Follow-up gtfparse work can address packaging/cleanup (#83, remaining #76) and stale issue triage (#75) before larger configurable synthesis/validation changes (#65/#66).

@coveralls

Copy link
Copy Markdown

Coverage Report for CI Build 36578137748

Coverage increased (+1.1%) to 96.751%

Details

  • Coverage increased (+1.1%) from the base build.
  • Patch coverage: No coverable lines changed in this PR.
  • 4 coverage regressions across 1 file.

Uncovered Changes

No uncovered changes found.

Coverage Regressions

4 previously-covered lines in 1 file lost coverage.

File Lines Losing Coverage Coverage
read_gtf.py 4 97.12%

Coverage Stats

Coverage Status
Relevant Lines: 277
Covered Lines: 268
Line Coverage: 96.75%
Coverage Strength: 2.9 hits per line

💛 - Coveralls

@iskandr
iskandr merged commit 9a281c1 into master Sep 29, 2026
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants