Skip to content

Bound temporary attribute-expansion memory for large GTFs #91

Closed

Description

@iskandr

The pandas/Arrow reader expands an entire GTF's raw attributes into Python objects and builds a full-length Python list for every discovered attribute before assembling the final table. With pandas 3's compact Arrow strings, temporary Python values can dominate peak memory even though they are not needed in the final representation.

In the current optimization investigation, complete 4.12-million-row files require roughly 4–6 GiB peak RSS. An unbatched faster parser/Arrow-construction candidate improves load time but produced higher GENCODE RSS than the 3.0.0 observation (about 5.3 GiB versus 4.0 GiB, with another candidate run around 5.8 GiB). These physical-RSS observations vary on the shared workstation; the whole-file Python allocations are directly visible in parse_gtf_and_expand_attributes and expand_attribute_strings.

For configured Arrow string storage, expand and construct attributes in bounded row batches, retaining Arrow buffers rather than all Python strings/lists simultaneously. Preserve first-seen column order, blank values for missing attributes including late-discovered columns and gaps, repeated attributes, fixed-field collisions, global progress/cancellation behavior, column selection and all supported inputs. Keep configured Python/object string behavior compatible and preserve the public expand_attribute_strings result contract.

Verify full-file content, dtypes and peak memory against 3.0.0 and the historical Polars path; keep repeated throughput measurements separate from single-file scaling observations. Related optimization work requested directly by the user after PR #89.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions