You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Bound temporary attribute-expansion memory for large GTFs #91
The pandas/Arrow reader expands an entire GTF's raw attributes into Python objects and builds a full-length Python list for every discovered attribute before assembling the final table. With pandas 3's compact Arrow strings, temporary Python values can dominate peak memory even though they are not needed in the final representation.
In the current optimization investigation, complete 4.12-million-row files require roughly 4–6 GiB peak RSS. An unbatched faster parser/Arrow-construction candidate improves load time but produced higher GENCODE RSS than the 3.0.0 observation (about 5.3 GiB versus 4.0 GiB, with another candidate run around 5.8 GiB). These physical-RSS observations vary on the shared workstation; the whole-file Python allocations are directly visible in parse_gtf_and_expand_attributes and expand_attribute_strings.
For configured Arrow string storage, expand and construct attributes in bounded row batches, retaining Arrow buffers rather than all Python strings/lists simultaneously. Preserve first-seen column order, blank values for missing attributes including late-discovered columns and gaps, repeated attributes, fixed-field collisions, global progress/cancellation behavior, column selection and all supported inputs. Keep configured Python/object string behavior compatible and preserve the public expand_attribute_strings result contract.
Verify full-file content, dtypes and peak memory against 3.0.0 and the historical Polars path; keep repeated throughput measurements separate from single-file scaling observations. Related optimization work requested directly by the user after PR #89.
Activity