Build frames from the manager instead of rebuilding them

[marco-mariotti]

Stacked on #171 — this PR targets fix_new, not master. It will be retargeted to master once #171 merges. The diff here is only the two commits on top of that branch.

Every pandas operation that returns a new frame rebuilds it through _constructor, and PyRanges makes that path expensive. A single gr.drop(columns=["X"]) costs four DataFrame constructions and eight manager copies, where one of each will do:

1. pandas builds the result frame from the block manager it just produced;
2. PyRanges.__new__ builds a second one, purely to look at its columns;
3. PyRanges.__init__ builds a third by handing that frame to DataFrame.__init__;
4. RangeFrame.drop builds a fourth, wrapping the result pandas had already rebuilt as a PyRanges.

None of this scales with the data, so it is the same tax on a thousand rows as on a hundred million — it just stops being visible when there is real work to hide behind.

What changed

_constructor_from_mgr on RangeFrame. This is the hook pandas calls when it has a block manager and needs a frame of the caller's class. Its default for a subclass routes through PyRanges(DataFrame(...)); the manager already carries the column index in axes[0], so the required-column check needs no frame at all and the result can be built straight from the manager. Both classes share one implementation and declare what they cannot do without:

class RangeFrame(pd.DataFrame):
    _required_columns: frozenset[str] = frozenset(RANGE_COLS)        # Start, End

class PyRanges(RangeFrame):
    _required_columns: frozenset[str] = frozenset(GENOME_LOC_COLS)   # + Chromosome

Keep the class while the frame still has those columns, hand back a plain DataFrame when it does not — the rule #171 established for PyRanges, now written once and applied to both.

__new__ stops building a frame to read columns. pandas hands it a whole DataFrame on every rebuild, so the columns can be read off that one.

drop, drop_and_return, reindex and copy return what pandas built. After the above, pandas already hands back a frame of the right class, or a DataFrame when the frame lost a required column. Rebuilding either costs two more frames and buys nothing. copy matters more than it looks, because pandas implements head as iloc[:n].copy().

The loci accessor is built on first use. Building from a manager never calls __init__, so _loci cannot be set there alone.

Benchmarks

Microseconds per call at 10^6 rows, median of three interleaved rounds, pandas 3.0.5 on Python 3.13. Verified flat at 10^7 and 10^8 — PyRanges.head is 17.4 / 17.6 / 17.7 µs across the three sizes, because none of this is proportional to the data.

operation	master	#171	this PR	vs master
PyRanges.head	108.0	133.9	17.4	6.2×
PyRanges.reset_index	72.9	84.8	9.3	7.8×
PyRanges.reindex	106.1	129.4	44.2	2.4×
PyRanges.drop	115.7	141.0	55.1	2.1×
PyRanges(df)	26.2	26.2	12.0	2.2×
PyRanges.assign	690.0	667.5	482.5	1.4×
PyRanges.astype	660.7	672.5	595.0	1.1×
RangeFrame.drop	71.4	72.6	51.8	1.4×
RangeFrame.reindex	61.1	63.1	41.5	1.5×
RangeFrame.head	19.3	19.8	12.5	1.5×
RangeFrame[cols]	85.9	87.1	77.9	1.1×

On the middle column: #171 moves the degrade-to-DataFrame decision into _constructor_with_fallback, which builds a DataFrame to inspect its columns on every internal rebuild, costing ~22% on these operations. That is what this PR removes, by making the same decision from axes[0] without building anything. Merged together, the two land about twice as fast as today's master.

Operations whose cost is data movement — copy, boolean masking — are unchanged. A caveat on measuring those: in a tight loop that discards each result, the allocator penalises whichever variant allocates fewer intermediate objects, and that artifact is larger than the effect being measured; it shows in both directions depending on loop shape. Timed single-shot in fresh processes at 10^7 rows, a boolean mask is 57.3 ms on master and 56.2 ms here; at 10^8 rows every variant lands within 1% of the others (581.8 vs 581.9 ms).

Behaviour changes

RangeFrame now survives head, slicing, masking and copy. pandas 2 rebuilt a subclassed frame as the subclass; pandas 3 rebuilds it as a plain DataFrame unless _constructor says otherwise, so rf.head() and rf[cols] had quietly started returning DataFrames while rf.drop() still returned a RangeFrame. They agree again.

Dropping Start or End from a RangeFrame now yields a DataFrame. A RangeFrame without them cannot even be printed — repr raises KeyError — yet rf.drop("Start") built one anyway. It now degrades, exactly as gr.drop("Chromosome") does. This is why test_range_frame_never_degrades from #171 is renamed and extended here: rf.reindex(columns=["Start"]) is a DataFrame now, since a frame without End is not a RangeFrame.

Fixed along the way

• .loci raised AttributeError on any PyRanges that came back from pickle, because _loci was only ever set in __init__, which unpickling skips.
• pr.PyRanges(array, columns=[...]) silently returned a column-less DataFrame, because __new__ checked the columns of a frame built without them.

Risks worth knowing

_constructor_from_mgr and _from_mgr are pandas internals (both present since 2.1). The risk is asymmetric: if _constructor_from_mgr ever disappears, pandas simply stops calling our override and _constructor still handles everything; if _from_mgr disappears, the hook raises. pandas' own _constructor_from_mgr carries a shim for GeoDataFrame, so subclasses overriding this hook is an acknowledged pattern. A test asserting both attributes exist would turn a future pandas bump into a CI failure rather than a user-visible one — happy to add it if you want the belt and braces.

__init__ no longer runs when pandas rebuilds a frame. This is the maintenance trap: anything added to PyRanges.__init__ later will silently not happen for rebuilt frames. _loci was exactly that bug; test_loci_survives_a_rebuild guards it.

Validation is by column name only — the manager's dtypes are not inspected. Same as before, so nothing is lost, but the fast path adds no safety either.

Subclasses of PyRanges are still not preserved end to end. The hook itself now keeps them (MyRanges(...).iloc[:1] is a MyRanges), but 36 ensure_pyranges(...) call sites and _constructor hardcode pr.PyRanges. Unchanged from master; mentioned because the hook makes it look closer to working than it is.

A few pandas paths still call self._constructor(...) directly rather than through the manager hook. Of ~40 operations I instrumented, only astype does, and its overhead over plain pandas is already down from 162 µs to about 5 µs. A manager-based _constructor would take that last bit — measured at 4.7 µs per call — but it means handling every argument shape pandas uses at those call sites. Left out deliberately.

Testing

pyright 0 errors, ruff format and check clean, 136 unit tests, 85 module doctests, 10 tutorial/how-to doctests — all with the versions CI pins (pandas 3.0.5, pandas-stubs 3.0.5.260730, pyright 1.1.406, ruranges 0.2.7).

New tests cover rebuilt frames keeping their class for both PyRanges and RangeFrame, degrading when a required column is gone, and .loci surviving both a rebuild and a pickle round-trip.