Add InstanSeg nucleus segmentation for H&E whole-slide images

[rushin682]

Adds nucleus segmentation on H&E whole-slide images to spatialrefinery, using InstanSeg, and writes the result into a SpatialData zarr.

InstanSeg owns the whole-slide layer we need — tiling, cross-tile label matching, an Otsu tissue prefilter and GeoJSON export — and its runtime requirements (numpy>=1.24, torch>=2.0, no upper bounds) sit inside the environment we already have, so it installs additively without disturbing numpy, pydantic, zarr or spatialdata.

What this adds

File	Purpose
src/spatialrefinery/segmentation/instanseg.py	segment_wsi() — slide → cells.geojson
src/spatialrefinery/segmentation/to_spatialdata.py	geojson_to_spatialdata() — GeoJSON + slide → SpatialData zarr
src/spatialrefinery/segmentation/_compat.py	bridges for three upstream InstanSeg bugs (below)
scripts/instanseg_segment.py	thin CLI wrapper
scripts/geojson_to_spatialdata.py	thin CLI wrapper
slurm/segment_node_worker.sh, segment_slurm.sh	multi-node, multi-GPU batch runner
tests/test_compat.py, tests/test_segmentation.py	24 tests

Logic lives in the package and scripts/ holds thin wrappers, matching the existing scripts/convert_to_ometiff.py pattern.

The two stages exchange a cells.geojson rather than being merged, so it acts as a resume boundary: a failed conversion does not force re-segmentation. Both run in the same interpreter.

python scripts/instanseg_segment.py --wsi-path slide.ome.tif --outdir seg/
python scripts/geojson_to_spatialdata.py \
    --geojson-path seg/slide.ome.tif/cells.geojson \
    --zarr-outdir zarr/ --wsi-path slide.ome.tif --template-adata panel.h5ad

Dependencies

New optional segmentation extra, opt-in because it pulls torch and multi-GB CUDA wheels that most uses of this package do not need. The CI test environment does not install it, so the tests that need instanseg skip rather than fail.

Important

The extra names rasterio and geojson directly rather than taking them via instanseg-torch[io]. That extra also carries zarr>=2.0.0,<3, and installing it silently downgrades zarr, numcodecs and tiffslide, which breaks spatialdata. Its stated reason ("tiffslide doesn't support zarr v3 yet") is stale as of tiffslide 4.0 / Bayer-Group/tiffslide#97.

Three upstream InstanSeg bugs are bridged

Patched at call time in segmentation/_compat.py, never vendored, so deleting that module is all it takes once upstream fixes them.

1. zarr 3. eval_whole_slide_image builds its label canvas with zarr.DirectoryStore, which zarr 3 renamed to zarr.storage.LocalStore. LocalStore is a drop-in for all four operations InstanSeg performs on the store.
2. TiffSlide is never imported. read_slide calls TiffSlide(...) at inference_class.py:236, but every import of that name in the module is function-local, so the module global is unbound and any whole-slide call raises NameError before reading a single tile.
3. The GeoJSON it writes is invalid. The exporter emits a comma after every feature and then closes the array, so output ends ...}},\n] — json.load rejects it, and so does QuPath. Repaired in the artefact rather than worked around at read time, so the published file is valid for any consumer.

Bugs 2 and 3 together mean InstanSeg's whole-slide path cannot have been run end to end upstream. Each bridge is pinned by a test so a future release that changes them fails fast rather than halfway through a multi-minute run.

Sensitivity on pale slides

On weakly haematoxylin-stained slides InstanSeg misses pale nuclei: its per-tile percentile_normalize is a global stretch, so where bright cytoplasm dominates the histogram it cannot lift faint nuclei above the seed threshold (default 0.7).

Two controls, both opt-in and off by default: --clahe applies locally adaptive contrast per tile (to L in LAB, so the H&E hue balance is preserved), and --seed-threshold overrides the model default. CLAHE is injected by wrapping model._to_tensor, the single point every tile passes through in the whole-slide loop, so no fork is needed.

Whole-slide result on a pale kidney H&E (33427 × 11949, mpp 0.2738):

	default	--clahe 2.0 --seed-threshold 0.4
nuclei	67,168	83,554 (+24.4%)
median area	318 px²	339 px²
objects < 20 px²	54	106 (0.13%)
runtime	8m40s	8m52s

The area distribution shifted up at every percentile from 1 to 99, so the extra 16,386 objects are normal-sized nuclei rather than fragments. As an external check, Xenium's own DAPI-based segmentation of this sample has 97,560 cells: the default recovered 69% of that from H&E, CLAHE 86%.

Ruled out along the way: resolution is not the bottleneck (InstanSeg downsamples 1.83× from 0.2738 to its native 0.5 µm; suppressing that gained +2%), and haematoxylin colour deconvolution actively hurts (−54%) — the brightfield model wants true H&E appearance.

Implementation notes

• Outputs are redirected. InstanSeg writes its .zarr and .geojson next to the input file; slides usually live on read-only dataset mounts, so segment_wsi runs the model against a symlink inside the output directory.
• The image element is built lazily from the slide's own pyramid via tifffile's zarr interface, so dask reads only the blocks a write touches — a level-0 plane on the test slide is 33427 × 11949 × 3.
• CRS is cleared before centroids are taken. gpd.read_file tags GeoJSON as EPSG:4326, but these are pixel coordinates; left in place, .centroid is computed against a spherical datum and every nucleus centroid drifts. Pinned by test_centroids_are_planar_pixel_coordinates.

Verification

• CUDA 13 wheels work on the cluster's 595.71.05 driver, Turing sm_75 included.
• Environment stays coherent after install: numpy 2.4.6, pydantic 2.12.5, zarr 3.3.0, spatialdata 0.8.0, anndata 0.13.2 all unchanged.
• pytest → 86 passed locally; 84 passed + 2 skipped in the CI hatch envs on both py3.12 and py3.14. prek run --all-files clean.
• End to end on Xenium_V1_hKidney_nondiseased_section_he_image.ome.tif (867 MB): segmentation 7m49s → 51 MB cells.geojson; conversion 67s → SpatialData zarr. Read back with the expected 5-level image pyramid, 83,554 polygons, and a (83554, 377) table over the Xenium panel.
• Re-run after the final review fixes produced bitwise-identical centroids, geometry areas and var_names, confirming those changes were behaviour-neutral.

slurm/SEGMENTATION_PLAN.md carries the full design notes and measurements.

Notes for review

• uv.lock bumps revision = 2 → 3 and drops 25 exotic-platform wheel entries (riscv64, s390x, ppc64le, Windows free-threaded builds) alongside the new dependencies. This is not hand-editing: restoring the file from main and running uv sync reproduces it byte for byte, so it is simply what current uv generates for this project.
• .gitignore gains explicit negations. /scripts/*.py and /slurm/ are ignored as scratch by design; rather than git add -f, this adds exceptions for the pipeline files so the intent is visible in the repo.
• --template-adata is required but only contributes column names to an all-zero table, which exists so the shapes element carries a SpatialData-valid annotation. Happy to make it optional if that suits downstream use better.
• Rollout caveat: the +24.4% is measured on one slide from one tissue. If staining varies across a cohort, --clahe 2.0 may want tuning per batch before it becomes a default. GPU memory ran ~35 GB with CLAHE versus ~12 GB without — fine on a 46 GB card, but lower --tile-size if fanning out 4 GPUs per node hits OOM.

[codecov]

Codecov Report

❌ Patch coverage is 71.94570% with 62 lines in your changes missing coverage. Please review.
✅ Project coverage is 65.62%. Comparing base (fa9ad74) to head (ffea1f3).

Files with missing lines	Patch %	Lines
src/spatialrefinery/segmentation/instanseg.py	53.62%	32 Missing ⚠️
src/spatialrefinery/segmentation/to_spatialdata.py	85.29%	15 Missing ⚠️
src/spatialrefinery/segmentation/_compat.py	76.92%	9 Missing ⚠️
src/spatialrefinery/segmentation/__init__.py	33.33%	6 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main       #9      +/-   ##
==========================================
+ Coverage   64.43%   65.62%   +1.18%     
==========================================
  Files           9       13       +4     
  Lines        1119     1338     +219     
==========================================
+ Hits          721      878     +157     
- Misses        398      460      +62     

Files with missing lines	Coverage Δ
src/spatialrefinery/__init__.py	80.00% <100.00%> (ø)
src/spatialrefinery/segmentation/__init__.py	33.33% <33.33%> (ø)
src/spatialrefinery/segmentation/_compat.py	76.92% <76.92%> (ø)
src/spatialrefinery/segmentation/to_spatialdata.py	85.29% <85.29%> (ø)
src/spatialrefinery/segmentation/instanseg.py	53.62% <53.62%> (ø)

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.