Stream transcript aggregation onto a hex lattice and harden Xenium ingest - #6
Merged
Conversation
…gest `SpatialData.aggregate` cannot bin a whole-transcriptome section: it builds one shapely Point per transcript and groups with `observed=False`, materialising the dense `n_spots x n_genes` product regardless of sparsity -- far more than the 500 GB a node has for 18k targets and ~1e9 transcripts. The hex lattice is regular, so membership needs no spatial join. `hex_lattice_params` now owns the single lattice definition; `hex_grid_centroids` materialises it as centroids and the new `assign_points_to_hexes` decides membership in closed form, which is what keeps the two from drifting apart. `bin_points_to_hex_counts` streams `(x, y, gene_code)` batches over that lattice and keeps only non-zero cells, bounding peak memory by one dask partition plus the output. `test_assign_points_to_hexes_matches_shapely_sjoin` pins the result pair-for-pair against the sjoin it replaces, including the duplicates that overlapping spots produce. Aggregating to an all-zero table now raises rather than writing it: that only happens when the lattice and the transcripts disagree, or the gene axis never matched. Alongside it, four fixes for real bundles that failed to convert: - Older "Preview"/"With_Addon" outputs store `cell_id`/`fov_name` as parquet binary, which reaches pyarrow as a bare `object` dtype and makes `to_parquet` raise `ArrowInvalid` on the dummy meta frame before any data is touched. `decode_bytes_columns` re-types those columns to `string`. - Scanners writing OME-TIFFs with `planarconfig=SEPARATE` produce `(1, c, y, x)`, tripping `xenium_aligned_image`'s `shape[-1] == 3` assert. `_aligned_image_dims` probes the lazy array and passes explicit dims. - Bundles naming the alignment `<sample>_he_alignment.csv` went unmatched, leaving the H&E and its tissue contours silently on an Identity transform. The alignment is now parsed once and applied to both, so they cannot end up on different transforms. - Tissue is segmented in the H&E's own pixel grid. The alignment matrix already carries that image's pixel size, so pre-scaling by the Xenium morphology image's `pixel_size` double-applied a scale factor and misplaced the contours. Downloads are now resumable: a leftover `.part` is continued with an HTTP `Range` request instead of refetched from byte 0, and any download whose length the server declares is size-checked before being promoted onto its destination.
Documentation build overview
|
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #6 +/- ##
==========================================
+ Coverage 61.00% 64.43% +3.43%
==========================================
Files 9 9
Lines 900 1119 +219
==========================================
+ Hits 549 721 +172
- Misses 351 398 +47
🚀 New features to boost your workflow:
|
The streaming aggregation imports `scipy.sparse`, the OME-TIFF layout probe imports `dask_image.imread`, and the points schema test imports `pyarrow`. None of the three ship type information, so the mypy pre-commit hook failed with `import-untyped` on all four call sites -- which `pytest` and `ruff` do not catch, since mypy only runs through that hook. Extend the existing `ignore_missing_imports` override, which already covers bioio, geopandas, shapely, and spatialdata, rather than adding `scipy-stubs` and type-checking one of the four differently from the rest.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
SpatialData.aggregatecannot bin a whole-transcriptome section: it builds one shapely Point per transcript and groups withobserved=False, materialising the densen_spots x n_genesproduct regardless of sparsity — far more than the 500 GB a node has for 18k targets and ~1e9 transcripts.Streaming hex aggregation
The hex lattice is regular, so membership needs no spatial join.
hex_lattice_paramsnow owns the single lattice definition;hex_grid_centroidsmaterialises it as centroids and the newassign_points_to_hexesdecides membership in closed form, which is what keeps the two from drifting apart.bin_points_to_hex_countsstreams(x, y, gene_code)batches over that lattice and keeps only non-zero cells, bounding peak memory by one dask partition plus the output.test_assign_points_to_hexes_matches_shapely_sjoinpins the result pair-for-pair against the sjoin it replaces, including the duplicates that overlapping spots produce.max_spots_per_chunkis kept on the public signatures for backwards compatibility and ignored — peak memory no longer depends on how many spots there are.Fixes for real bundles that failed to convert
cell_id/fov_nameas parquet binary, which reaches pyarrow as a bareobjectdtype and makesto_parquetraiseArrowInvalidon the dummy meta frame before any data is touched.decode_bytes_columnsre-types those columns tostring.planarconfig=SEPARATEproduce(1, c, y, x), trippingxenium_aligned_image'sshape[-1] == 3assert._aligned_image_dimsprobes the lazy array and passes explicit dims.<sample>_he_alignment.csvwent unmatched, leaving the H&E and its tissue contours silently on an Identity transform. The alignment is now parsed once and applied to both, so they cannot end up on different transforms.pixel_sizedouble-applied a scale factor and misplaced the contours.Resumable downloads
A leftover
.partis continued with an HTTPRangerequest instead of refetched from byte 0, and any download whose length the server declares is size-checked before being promoted onto its destination. Servers that ignoreRangeanswer200with the whole body, which is detected so a resumed file is never a mix of two responses.Testing
tests/test_utils.pygains coverage for the lattice geometry, the analytic hex membership (against a shapelysjoinreference), the streaming counter, and the bytes-column decoding;tests/test_xenium_io.pyis new and covers the H&E channel-layout probe and the aggregation's table-annotation contract. 47 tests pass;ruff checkandruff format --checkare clean.